Superdelegate chicanery notwithstanding, Obama has won his party’-s nomination, so my head-to-head contest between a major pollster (Zogby) and major prediction market (Intrade) will be coming to a close.
Unsurprising to those who know a little bit about the scholarship, economics, and/or track record of prediction markets, the traders of Intrade provided us much better data this election season than the respondents to the Zogby polls.
| Standings | |||||
| Wins | Losses | Ties | Pct | Contender | Avg Eve Prob |
| 7 | 3 | 11 | 59.5% | Intrade | 71.3% |
| 3 | 7 | 11 | 40.5% | Zogby | 40.7% |
| Schedule | ||||||||||
| Score | Date | State | Party | Intrade | Zogby | Winner | Intrade Pct | Zogby Pct | ||
| 7-3-11 | 6-May | IN | Dem | Clinton | 2-way-tie | Clinton | 85% | 42% | ||
| 6-3-11 | 6-May | NC | Dem | Obama | Obama | Obama | 90% | 50% | ||
| 6-3-10 | 22-Apr | PA | Dem | Clinton | Clinton | Clinton | 82% | 47% | ||
| 6-3-9 | 4-Mar | OH | Dem | Clinton | 2-way-tie | Clinton | 70% | 45% | ||
| 5-3-9 | 4-Mar | TX | Dem | Obama | 2-way-tie | Clinton | 57% | 44% | ||
| 5-2-9 | 5-Feb | NJ | Rep | McCain | McCain | McCain | 96% | 52% | ||
| 5-2-8 | 5-Feb | NJ | Dem | Clinton | 2-way-tie | Clinton | 67% | 43% | ||
| 4-2-8 | 5-Feb | NY | Rep | McCain | McCain | McCain | 98% | 53% | ||
| 4-2-7 | 5-Feb | GA | Dem | Obama | Obama | Obama | 96% | 48% | ||
| 4-2-6 | 5-Feb | MO | Dem | Obama | Obama | Obama | 63% | 47% | ||
| 4-2-5 | 5-Feb | CA | Rep | McCain | Romney | McCain | 56% | 40% | ||
| 3-2-5 | 5-Feb | CA | Dem | Obama | Obama | Clinton | 52% | 46% | ||
| 3-2-4 | 29-Jan | FL | Rep | McCain | 2-way-tie | McCain | 51% | 33% | ||
| 2-2-4 | 26-Jan | SC | Dem | Obama | Obama | Obama | 90% | 38% | ||
| 2-2-3 | 19-Jan | SC | Rep | McCain | McCain | McCain | 56% | 29% | ||
| 2-2-2 | 19-Jan | NV | Dem | Obama | Clinton | Clinton | 54% | 42% | ||
| 2-1-2 | 15-Jan | MI | Rep | McCain | 2-way tie | Romney | 54% | 27% | ||
| 2-0-2 | 8-Jan | NH | Dem | Obama | Obama | Clinton | 91% | 39% | ||
| 2-0-1 | 8-Jan | NH | Rep | McCain | McCain | McCain | 82% | 34% | ||
| 2-0-0 | 3-Jan | IA | Dem | Obama | 3-way tie | Obama | 54% | 28% | ||
| 1-0-0 | 3-Jan | IA | Rep | Huckabee | 2-way tie | Huckabee | 53% | 28% | ||
–
Lord willing, I will have another showdown season, but will most likely choose another pollster. Whereas Intrade listed contracts and probabilities for every state primary, Zogby only provided election eve updates for only 21 of the 87 held to date. Rasmussen stands out as a worthy pollster, but I would be happy with any candidate referrals from my readers as well.
Cross-posted from Caveat Bettor.
Previous blog posts by Caveat Bettor:
- Land-Ocean year-to-date temperatures 0.35 Celsius over baseline
- Intrade lists Global Warming Contracts!
- Intrade beats Zogby on Super Tuesday
- Super Tuesday Showdown: Intrade v. Zogby
- The Democrat SC Showdown: Intrade v. Zogby
- Zogby beats Intrade in predicting Nevada caucus winner Clinton.
- The GOP SC and Dem NV Showdown: Intrade v. Zogby
Hi Cav,
Congrats for you work. For your information, I do subscribe to the Memeorandum feed and I saw Gallup popped up there many times in this primary election season. I didn’t see much the other pollsters, as far as I can remember.
PS: Memeorandum, like TechMeMe, computes the bloggers’ links to worthy web pieces —in that, it’s comparable to Google’s PageRank system.
The machine created two posts (#6854 and #6855). This one is the #6855. I deleted the #6854.
–
If you known what happened, tell me, so I can understand whether it’s a bug, or a flaw in the WordPress user interface, misleading the blogger —which is something I have remarked and blogged about here yesterday.
http://www.midasoracle.org/200…..wordpress/
Long time reader, first time poster. Wow, very interesting data, it’s great to see these comparisons. I would love to see the analysis framed a different way, though. The current analysis seems to embrace the idea that a market is “wrong” when it implies a 55% chance of event occurrence and it doesn’t occur. Implicitly condoning this as a relevant interpretation (via win/loss count) serves to promote people who use that same idea against prediction markets, as in articles like this:
Slate: The Fizz-dom of Crowds
–
The market binning strategy tells a fairly different tale…
–
markets in the 50-60 % range:
9
agree with proposition:
5
actual average probability:
56%
–
markets in the 60-90 % range:
6
agree with proposition:
6
actual average probability:
100%
–
markets in the 90-100 % range:
6
agree with proposition:
5
83%
–
Here, you can see the market was mostly “wrong” when it should have been. Conversely, one could argue that the market was actually incorrect in how many times the outcome was successful, even with lower probability estimates in the 60-90 range (although there were too few markets in this data to be sure). There were also too few data points to know if the NH reversal was truly a 1/10 probability or whether the market was overconfident.
–
If the world is to understand prediction markets, it seems that the most informed people (all you fine folks running and contributing to midas oracle) should form a united front in explaining that a market is not wrong when it predicts something with 55% probability and it doesn’t happen. The world needs to understand that this fallacious argument is equivalent to saying any probability calculation can be invalidated by a single event (for example, 67% is still the correct probability for rolling >2 on a die, even after seeing a single roll of a 1).
–
Edit to add paragraph breaks
Great effort and analysis, thanks.
Next time you might try Real Clear Politics or Wikipedia.
Actually, it may not be too late to analyze these for the nomination contest too.
@Jason Carver: “explaining that a market is not wrong when it predicts something with 55% probability and it doesn’t happen”… We have been doing that on the explainer you’ll find on the frontpage, and in each political PM story like the last one.
–
To separate a paragraph, we put a “hyphen”, as you see above. I’ll see look into that issue when I update the “theme” (layout) of this blog.
@David Pennock: Speaking of RCP, Cav, do google The Numbers Guys at the WSJ for an analysis of pollsters and averaged polls.
@Chris Masse: thanks for the tip regarding paragraph breaks.
–
Explainers are useful, but I still think propagating the W/L concept at all is confusing and undoes a lot of the good done by the home page explanations.
Thanks for the great comments Chris, David and Jason.
Jason–you have identified a mismatch in my contest, namely, that the Intrade contracts are “all-or-none” payout derivative, rather than a “spot settlement” payout which would reflect the actual percentage of votes received. If it were the latter, then the mismatch would be eliminated, as the Zogby and Intrade predictions could be matched against the actual percentages.
But because this scenario was not possible, in part because of the complexity of and risks to settlement,
I tried to design an experiment that would measure success of each predicted trial.
Delegates for each state are assigned a pro-rata basis, but everyone is pretty much concerned with which candidate got the most delegates for each state. My experiment attempts to measure for this. The Intrade candidate contract with the highest value for each primary is the predicted winner, while for Zogby, it is the candidate with the highest percentage above the 3% confidence interval stated by Zogby.
What this meant was that a candidate did not need to project over 50% to be the predicted winner of a state contest; merely, the candidate had to project ahead of his or her competition.
Hope this helps, Cav
Cav, thanks for the added info. Interesting stuff!
Hi Cav,
Congrats for you work. For your information, I do subscribe to the Memeorandum feed and I saw Gallup popped up there many times in this primary election season. I didn’t see much the other pollsters, as far as I can remember.
PS: Memeorandum, like TechMeMe, computes the bloggers’ links to worthy web pieces —in that, it’s comparable to Google’s PageRank system.
The machine created two posts (#6854 and #6855). This one is the #6855. I deleted the #6854.
–
If you known what happened, tell me, so I can understand whether it’s a bug, or a flaw in the WordPress user interface, misleading the blogger —which is something I have remarked and blogged about here yesterday.
http://www.midasoracle.org/200…..wordpress/
Long time reader, first time poster. Wow, very interesting data, it’s great to see these comparisons. I would love to see the analysis framed a different way, though. The current analysis seems to embrace the idea that a market is “wrong” when it implies a 55% chance of event occurrence and it doesn’t occur. Implicitly condoning this as a relevant interpretation (via win/loss count) serves to promote people who use that same idea against prediction markets, as in articles like this:
Slate: The Fizz-dom of Crowds
–
The market binning strategy tells a fairly different tale…
–
markets in the 50-60 % range:
9
agree with proposition:
5
actual average probability:
56%
–
markets in the 60-90 % range:
6
agree with proposition:
6
actual average probability:
100%
–
markets in the 90-100 % range:
6
agree with proposition:
5
83%
–
Here, you can see the market was mostly “wrong” when it should have been. Conversely, one could argue that the market was actually incorrect in how many times the outcome was successful, even with lower probability estimates in the 60-90 range (although there were too few markets in this data to be sure). There were also too few data points to know if the NH reversal was truly a 1/10 probability or whether the market was overconfident.
–
If the world is to understand prediction markets, it seems that the most informed people (all you fine folks running and contributing to midas oracle) should form a united front in explaining that a market is not wrong when it predicts something with 55% probability and it doesn’t happen. The world needs to understand that this fallacious argument is equivalent to saying any probability calculation can be invalidated by a single event (for example, 67% is still the correct probability for rolling >2 on a die, even after seeing a single roll of a 1).
–
Edit to add paragraph breaks
Great effort and analysis, thanks.
Next time you might try Real Clear Politics or Wikipedia.
Actually, it may not be too late to analyze these for the nomination contest too.
@Jason Carver: “explaining that a market is not wrong when it predicts something with 55% probability and it doesn’t happen”… We have been doing that on the explainer you’ll find on the frontpage, and in each political PM story like the last one.
–
To separate a paragraph, we put a “hyphen”, as you see above. I’ll see look into that issue when I update the “theme” (layout) of this blog.
@David Pennock: Speaking of RCP, Cav, do google The Numbers Guys at the WSJ for an analysis of pollsters and averaged polls.
@Chris Masse: thanks for the tip regarding paragraph breaks.
–
Explainers are useful, but I still think propagating the W/L concept at all is confusing and undoes a lot of the good done by the home page explanations.
Thanks for the great comments Chris, David and Jason.
Jason–you have identified a mismatch in my contest, namely, that the Intrade contracts are “all-or-none” payout derivative, rather than a “spot settlement” payout which would reflect the actual percentage of votes received. If it were the latter, then the mismatch would be eliminated, as the Zogby and Intrade predictions could be matched against the actual percentages.
But because this scenario was not possible, in part because of the complexity of and risks to settlement,
I tried to design an experiment that would measure success of each predicted trial.
Delegates for each state are assigned a pro-rata basis, but everyone is pretty much concerned with which candidate got the most delegates for each state. My experiment attempts to measure for this. The Intrade candidate contract with the highest value for each primary is the predicted winner, while for Zogby, it is the candidate with the highest percentage above the 3% confidence interval stated by Zogby.
What this meant was that a candidate did not need to project over 50% to be the predicted winner of a state contest; merely, the candidate had to project ahead of his or her competition.
Hope this helps, Cav
Cav, thanks for the added info. Interesting stuff!