James Surowiecki’-s 4 comments at Overcoming Bias (in October 2007), responding to accusations that he got it all wrong about Francis Galton:
—-
James Surowiecki’-s 1st comment:
“-Galton did not even bother to calculate a mean, as he saw his data was clearly not normally distributed. He used the median (of 1207), which was much further off than the mean, but by modern standards clearly the better estimator. It was Karl Pearson in 1924 who calculated the mean.”-
Robin [Hanson], before repeating falsehoods, you might want to go back to the original sources —- or, in this case, to the footnotes to my book. Galton did, in fact, calculate the mean, long before Karl Pearson did. Galton’-s calculation appeared in Nature, Vol. 35, No. 1952 (3/28/07), in a response to letters regarding his original article. One of the correspondents had gone ahead and calculated a mean from the data that Galton had provided in his original piece, and had come up with the number 1196. Galton writes, “-he makes it [the mean] 1196 lb. . . . whereas it should have been 1197 lb.”-
I find the fact that Levy and Peart wrote an entire article about Galton (and, to a lesser extent, about my use of him), and never went back and checked the original sources is astounding in its own right. (They actually wonder in the paper, “-However the new estimate of location came to be part of Surowieki’s account,”- as if the answer isn’-t listed right there in the footnotes.) What makes it even more astounding, though, is that they’-ve written an entire paper about the diffusion of errors by experts who “-pass along false information (wittingly or unwittingly)”- while passing along false information themselves.
It also seems bizarre that Levy and Peart caution, “-The expectation of being careful seems to substitute for actually being careful,”- and yet they were somehow unable to figure out how to spell “-Surowiecki”- correctly. The article is a parody of itself.
I’-m happy to enter into a discussion of whether the median or the mean should be used in aggregating the wisdom of crowds. But whether Galton himself thought the mean or the median was better was and is irrelevant to the argument of my book. I was interested in the story of the ox-weighing competition because it captures, in a single example, just how powerful group judgments can be. Galton did calculate the mean. It was 1197 lbs., and it was 1 lb. away from the actual weight of the ox. The only “-falsehood”- being perpetrated here are the ones Levy and Peart are putting out there, and the ones that you uncritically reprinted.
—-
James Surowiecki’-s 2nd comment:
Here are the links for the letter from Galton, where he reports the mean:
http://galton.org/cgi-bin/search/images/galton/search/essays/pages/galton-1907-ballot-box_1.htm
http://galton.org/cgi-bin/search/images/galton/search/essays/pages/galton-1907-ballot-box_2.htmThere’-s no reason for debate here. Levy and Peart say “-Pearson’s retelling of the ox judging tale apparently served as a starting point for the 2004 popular account of the modern economics of information aggregation, James Surowieki’s Wisdom of Crowds.”- It wasn’-t the starting point. The starting point was Galton’-s own experiment, and his own reporting of the mean in “-The Ballot Box.”- Robin writes: “-Galton did not even bother to calculate a mean.”- He did calculate it, and he did report it. This fact shouldn’-t be listed as an “-addendum”- to the original post. The original post should be rewritten completely —- perhaps along the lines of “-Surowiecki and Galton disagree about which estimate is a better representation of group judgment”- rather than “-Author Misreads Expert”- —- or else scrapped.
—-
James Surowiecki’-s 3rd comment:
I appreciate Levy and Peart admitting their mistake. But they seem not to recognize that their mistake undermines the critique that’-s at the center of their paper. Their paper, they write, is about the misconstruing of Galton’-s experiment. “-A key question,”- they write, “-is whether the tale was changed deliberately (falsified) or whether, not knowing the truth, the retold (and different) tale was passed on unwittingly.”- But the account of Galton’-s experiment was not changed deliberately and was not falsified. It was recounted accurately. Levy and Peart want to use my retelling of the Galton story as evidence of how “-experts pass along false information (wittingly or unwittingly) [and] become part of a process by which errors are diffused.”- But there’-s no false information here, and no diffusion of errors, which rather demolishes their thesis. If they really want to write a paper about how “-experts”- pass along false information, they’-d be better off using themselves as Exhibit A, and tell the story of how they managed to publish such incredibly shoddy work and have prominent economists uncritically link to it.
—-
James Surowiecki’-s 4th comment:
To finish, Levy and Peart insist that their really important point still stands, which is that “-When people quote Galton through Surowiecki, they tell Surowiecki’-s tale, not Galton’-s,”- and that this is a problem because Galton’-s thinking is being misrepresented. But as I said earlier, “-The Wisdom of Crowds”- was not intended to be a discussion of Francis Galton’-s opinions on what’-s the best method to capture group judgment, nor, as far as I know, has anyone who’-s “-Surowiecki’-s tale”- used the Galton example since used it to analyze Galton’-s opinions. People aren’-t quoting the Galton story because they’-re interested in what Galton himself thought about the median vs. mean. They’-re quoting it because they’-re interested in the bigger idea, which is that group judgments (and this is true whether you use the median, the mean, or a method like parimutuel markets) are often exceptionally accurate. Levy and Peart have constructed a straw man —- and, in this case, a straw man based on a falsehood —- and then tried to knock it down.
Robin [Hanson] writes: “-it is ironic that Galton made quite an effort to emphasize and prefer the median, in part because the data did not look like a bell curve, while your retelling focuses on him calculating a mean after checking for a bell curve.”- What’-s ironic about this? He did check for a bell curve, and he did calculate the mean. It’-s the data themselves, not Galton’-s interpretation of them, that I was writing about. (If he hadn’-t calculated the mean, I would have happily told the story with the median, since it was also remarkably accurate, and demonstrated the same point about the wisdom of crowds.)
Finally, on the substantive question, Robin (and Levy and Peart) seem to think that because the distribution of guesses wasn’-t normal, that makes using the mean a mistake. But this is precisely what’-s so interesting: if the group is large enough, even if the distribution isn’-t normal, the mean of a group’-s guesses is nonetheless often exceptionally good.