With all of the excitement (and controversy over the senior men's results) surrounding the 2008 U.S. Figure Skating Championships, is there a chance that the results were biased? A Yale professor of statistics claims that the judging reforms instituted after the 2002 Salt Lake City scandal still make scores susceptible to how judges are chosen, but luckily only in international competitions. U.S. Figure Skating mostly employs rules developed by the International Skating Union, which previously employed the “6.0” judging standard that attracted criticism, but modifies the rules to avoid this problem.
In a professional journal article entitled “Chance, On and Off the Ice,” Assistant Professor Jay Emerson claims the flaw in the judging program cost Russian pairs skaters Maria Petrova and Alexei Tikhonov the silver medal in the 2006 World Figure Skating Championships (they still took home a bronze). Emerson has a fair amount of credibility in analyzing complex systems -- his work on backdating stock options helped The Wall Street Journal win a 2007 Pulitzer prize.
While Emerson employs sophisticated methods to buttress his claims, his thesis is simple. The ISU relies on a randomly selected panel of nine judges from a pool of a dozen (technically, all 12 judges score a competition, but three are randomly thrown out). Different panels are chosen for the short program and the free skate. All selections are made by computer, and there is no allegation that assignments are anything but truly random. Emerson looked at how the scoring would change with a different panel of judges (48,400 combinations in all), and found that the gold medal winners (Pang Qing and Tong Jian from China) would have been unaffected. However, there was a greater chance that Petrova and Tikhonovin would won have the silver medal than the actual medal winners (another Chinese pair, Zhang Dan and Zhang Hao).
Incidentally, the old 6.0 method used a panel of eight judges who were not randomly selected, and the high and low scores were thrown out. This made it easier for judges to collude in their scoring, which is what happened in Salt Lake City. U.S. Figure Skating uses panels of nine judges without random selection; Collusion among nine judges is harder than six, and there is no variation in panel composition between programs that would affect scoring. Here is a less technical explanation than what appears in Emerson's paper.
While these criticisms don't apply to this month's Championships, it still highlights the need for the ISU to continue its work in developing a scoring and judging system that is unequivocally fair. Not that it will happen soon enough to help this year's competitors -- the 2008 World Juniors Championships start on February 25 in Bulgaria, the 2008 World Championships start on March 18 in Sweden, and the 2008 World Synchronized Skating Championships start on March 28 in Hungary.
