In part #2, he speaks about the books he is writing:
SXSW: Nate Silver explains how he approached political forecasting for the 2008 US presidential elections. – [VIDEO]
Reply
Tag Archives: 2008 US presidential elections
Flawed New Hampshire polls = Non-accurate New Hampshire prediction markets
The most comprehensive analysis ever conducted of presidential primary polls:
Via Mister the Great Research Scientist David Pennock –-who is an indispensable element of the field of prediction markets.
As I blogged many times, prediction markets react to polls…- See the addendum below…- – [UPDATE: See also Jed’s comment.] – Prediction markets should not be hyped as crystal balls, but simply as an objective and continuous way to aggregate expectations. So, if you think of it, their social utility is much smaller than what the advocates of the “-idea futures”-, “-wisdom of crowds”- or “-collective intelligence”- concepts told us. Much, much, much, much smaller…- They all make the mistake to put accuracy forward. (By the way, somewhat related to that issue, please go reading the dialog between Robin Hanson and Emile Servan-Schreiber.)
–
Addendum
–
California Institute of Technology economist Charles Plott:
What you’-re doing is collecting bits and pieces of information and aggregating it so we can watch it and understand what people know. People picked this up and called it the “-wisdom of crowds”- and other things, but a lot of that is just hype.
–
New Hampshire – The Democrats
–
The Hillary Clinton event derivative was expired to 100.
–
New Hampshire – The Republicans
–
The John McCain event derivative was expired to 100.
–
No TweetBacks yet. (Be the first to Tweet this post)
The hype is over. The party is over. – Part II
Since the 2008 US presidential election, no more posts on prediction markets at Freakonomics.
Previously: Part I
–
Prediction markets compute facts and expertise quicker that the mass media do.
Political prediction markets react (with a small delay) to political polls —-just like the political experts and the mass media do, too. Hence, in order to discover their true social utility, the prediction markets (which are tools of intelligence) should not be compared to the polls (which are just facts) but to the similar meta intelligence mechanisms (the averaged probabilistic predictions from a large panel of experts, or the averaged probabilistic predictions from the political reporters in the mass media, or else). My bet is that, in complicated situations (such as the 2008 Democratic primary), the prediction markets beat the mass media (in terms of velocity) —-even though the prediction markets are not omniscient and not completely objective (but who is?).
–
You might remember the research article that I have blogged about:
–
Learning in Investment Decisions: Evidence from Prediction Markets and Polls – (PDF file) – David S. Lee and Enrico Moretti – 2008-12-XX
In this paper, we explore how polls and prediction markets interact in the context of the 2008 U.S. Presidential election. We begin by presenting some evidence on the relative predictive power of polls and prediction markers. If almost all of the information that is relevant for predicting electoral outcomes is not captured in polling, then there is little reason to believe that prediction market prices should co-move with contemporaneous polling. If, at the other extreme, there is no useful information beyond what is already summarized by the current polls, then market prices should react to new polling information in a particular way. Using both a random walk and a simple autoregressive model, we find that the latter view appears more consistent with the data. Rather than anticipating significant changes in voter sentiment, the market price appears to be reacting to the release of the polling information.
We then outline and test a more formal model of investor learning. In the model, investors have a prior on the probability of victory of each candidate, and in each period they update this probability after receiving a noisy signal in the form of a poll. This Bayesian model indicates that the market price should be a function of the prior and each of the available signals, with weights reflecting their relative precision. It also indicates that more precise polls (i.e. polls with larger sample size) and earlier polls should have more effect on market prices, everything else constant. The empirical evidence is generally, although not completely, supportive of the predictions of the Bayesian model.
–
–
You might also have watched Emile Servan-Schreiber’-s videos. Emile is a smart man, and those videos are truly instructive.
- In the first part (the lecture), our good doctor Emile Servan-Schreiber sold the usual log lines about the prediction markets —-blah blah blah blah blah.
- In the second part, Emile Servan-Schreiber took questions from the audience in the room. “-Aren’-t political prediction markets just following the polls?”-, asked one guy. Emile’-s answer was long and confused. However, in my view, Emile actually did answer that question (before it was ever asked) in his preceding lecture when, at one point, he made the point that the media were slower than the prediction markets to integrate all the facts about the 2008 Democratic primary, around May 2008. That is the right answer to give to a conference attendee who enquires about prediction markets “-following”- the polls. Both the mass media and the prediction markets do follow the polls (since the polls are facts that can’-t be ignored), during political campaigns. Let’-s compare the prediction markets with the mass media, instead, and let’-s see who’-s quicker to deliver the right intelligence..
–
Lance Fortnow gives a good insight about the relationship between polls and prediction markets (see his last paragraph).
Yesterday the Electoral College delegates voted, 365 for Barack Obama and 173 for John McCain. How did the markets do?
To compare, here is my map the night before the election and the final results. The leaning category had Obama at 364. The markets leaned the wrong way for Missouri and Indiana, their 11 electoral votes canceling each other out. The extra vote for Obama came from a quirk in Nebraska that the Intrade markets didn’-t cover: Nebraska splits their votes based on congressional delegations, one of which went to Obama.
Indiana and Missouri were the most likely Republican and Democratic states to switch sides according to the markets, which mean the markets did very well this year again. Had every state leaned the right way (again), one would wonder if the probabilities in each state had any meaning beyond being above or below 50%.
Many argue the markets just followed the predictions based on polls like Nate Silver’-s fivethirtyeight.com. True to a point, Silver did amazingly well and the markets smartly trusted him. But the markets also did very well in 2004 without Silver. [Chris Masse’s remark: In 2004, Electoral-Vote.com (another poll aggregator) was all the rage.] One can aggregate polls and other information using hours upon hours of analysis or one can just trust the markets to get essentially equally good results with little effort.
–
The polls are facts. Prediction markets are meta to facts. Prediction markets are intelligence tools. Let’-s compare them with similar intelligence tools.
–
Lance Fortnow’-s post attracted an interesting comment from one of his readers:
to provide an exciting collection of political and other prediction markets.
These markets are as much a “-prediction”- tool as a wind vane or outdoor thermometer are. They moved up and down according to the daily trends, with very little insight of the longer place phenomena underlying them.
When the weather was hot (Palin’-s nomination announcement) the market swinged widely towards McCain, while ignoring the cold front on the way here (the economic recession + Palin inexperience).
The value of weather forecast is in telling us things we didn’-t know. We don’-t need to trade securities to believe that if McCain is closing on the polls then his chances of wining are higher (duh!), which is what the markets did. We need sophisticated prediction mechanisms to tell us how the worsening economic conditions, the war in Iraq and Palin ineptitude (which in pre-Couric days wasn’-t as well established) will impact this election, today poll’-s be damned.
Looking at the actions by the republican teams, who were trying to read past the daily trend all the way to November 4th, it is clear that they thought all along they were losing by a fair margin. Because of this is they choose moderate, maverick McCain, went for the Palin hail mary fumble^H^H^H^H^H pass and the put-the-campaign-on-hold move.
A full two weeks before the election the McCain team concluded the election was unwinnable, while the electoral college market was still giving 25-35% odds to McCain.
–
As highlighted in bold, the commenter says two things:
- The prediction markets are just following the polls.
- The prediction markets have a minimal societal value.
–
My replies to his/her points:
- That’-s not the whole truth. The polls are just a set of facts, whereas the prediction markets are intelligence tools that aggregate both facts and expertise. The commenter picks up a simple situation (the 2008 US presidential election) where, indeed, anybody reading the latest polls (highly favorable to Barack Obama) could figure out by himself/herself what the outcome would be (provided the polls wouldn’-t screw it).
- That’-s true in simple situations, but that’-s wrong in complicated situations (such as the 2008 Democratic primary).
–
The emergence of the social utility of the prediction markets will come more clearly to people once we:
- Highlight the complicated situations-
- Code the mass media’-s analysis of those complicated situations, and compare that with the prediction markets.
–
–
APPENDIX:
–
–
–
–
No TweetBacks yet. (Be the first to Tweet this post)
Are prediction markets useful?
According to Alan Abramowitz, John Tierney has been “-greatly exaggerating the accuracy of the betting markets.”- “-They follow the polls. That’s it.”-
–
–
My comment to Alan Abramowitz and John Tierney:
“-They follow the polls. That’s it.”-
Yes, they follow the polls. No, that’-s not it.
Traders also dig the news of the day and make anticipations about the outcome. For instance, towards the end of the 2008 Democratic primary, the polls and the mass media were still giving Hillary Clinton a very good standing, whereas the prediction markets (informed by a bunch of political experts who did the counting of the delegates and super-delegates) were telling us that she was as toasted as Lehman Brothers in the middle of the credit crunch crisis.
Are prediction markets useful? If John Tierney wants to answer this question, he should pick up a prediction market and put it in the social context of that day. Some prediction markets are more useful than others. In the case of the 2008 Democratic primary (a complicated matter), the prediction markets sided with the best informed political experts against the mass media and the polls. So to speak, they were an umpire. In that case, we see the emergence of a social utility. We now have the case for the media citing more the probabilities of the liquid (play-money and/or real-money) prediction markets.
–
Previously: #1 – #2 – #3 – #4 – #5
External Link: Club of Growth
–
Still, as noted, it was a good election for [the] prediction markets and another piece of evidence of their superiority over the pundit[s] (and at least parity with the poll).
Dixit Nigel Eccles in a comment.
–
at least parity with the poll
–
I agree with the above.
–
their superiority over the pundits
–
What documented evidence do you have about that, mister the cocky entrepreneurial Scotsman?
John Tierney linked to that Huffington Post that listed the pundits’- predictions about the total number of electoral votes that each presidential candidate would take. But I disagree with that way of predicting the electoral college and assessing these predictions. With this completely flawed method, if you are damn wrong on a state and damn wrong (in the opposite way) about another state that has the exact same number of electoral votes, then you are a bright genius worth the Nobel prize of forecasting. Gimme a break. Enough with that voodoo way of assessing predictions about the electoral college. Do the assessment state by state.
InTrade and HubDub got lucky that their 2 mistakes (so to speak, in a non-probabilistic way) on Missouri and Indiana (both with 11 electoral votes) canceled themselves perfectly. IT WAS PURE LUCK. If their 2 mistakes had been made in the same direction (say, betting on Obama with the outcome going eventually to McCain), and/or their 2 mistakes had been done on 2 very dissimilar states (say, one with 6 electoral votes and the other one with 27 electoral votes), then we would have had reporters and bloggers bashing the prediction markets for the whole month of November.
–
The Intrade bettors expected Mr. Obama to end up with 364 votes in the Electoral College -one less than he actually got.
My remark to John Tierney:
InTrade got it [almost] spot on because they were wrong on Missouri (which was predicted to go for Obama but went to McCain) and wrong too on Indiana (which was predicted to go for McCain but went to Obama) —and those 2 opposite mistakes canceled themselves because those 2 states have the exact same number of electoral votes (11). Hence, I disagree with your method.
–
–
APPENDIX:
Here’-s a visual post-mortem of the 2008 US presidential elections.
Pay attention to Missouri and Indiana.
–
A) InTrade, on November 5, 2008 (screen shot taken at 2:00 am):
–
–
Prediction Markets &- State Polls, on November 4, 2008:
–
–
B1) Prediction Markets (on November 4, 2008)
–
–
InTrade (screen shot taken at mid-day ET, November 4, 2008):
–
InTrade (screen shot taken in the morning, November 4, 2008):
–
BetFair (screen shot taken in the morning, November 4, 2008):
–
HubDub (screen shot taken in the morning, November 4, 2008):
–
–
–
–
–
B2) State Polls (on November 4, 2008)
–
–
Karl Rove (on November 4, 2008):
–
CNN (on November 4, 2008):
–
Pollster (on November 4, 2008):
–
Electoral-Vote.com (on November 4, 2008):
–
Nate Silver (on November 4, 2008):
–
–
PREDICTION MARKET PROBABILITIES
–
Explainer On Prediction Markets
–
A prediction market is a market for a contract that yields payments based on the outcome of a partially uncertain future event, such as an election. A contract pays $100 only if candidate X wins the election, and $0 otherwise. When the market price of an X contract is $60, the prediction market believes that candidate X has a 60% chance of winning the election. The price of this event derivative represents the imputed perceived likelihood of the partially uncertain event (i.e., its aggregated expected probability). A 60% probability means that, in a series of events each with a 60% probability, the favored outcome is expected to occur 60 times out of 100, and the unfavored outcome is expected to occur 40 times out of 100.
Each prediction exchange organizes its own set of real-money and/or play-money markets, using either a CDA or a MSR mechanism —-with or without an automated market maker.
Prediction markets enable us to attain collective intelligence. Prediction markets produce dynamic, objective probabilistic predictions on the outcomes of future events by aggregating disparate pieces of information that the traders bring when they agree on prices. The event derivative traders are informed by the primary indicators (i.e., the primary sources of information), like the polls, for instance. These informed speculators then execute their transactions based on their anticipations about the future —-anticipations that will be either confirmed or infirmed.
The value of a set of prediction markets consists in the added accuracy that these prediction markets provide relative to the other forecasting mechanisms, times the value of accuracy in improved decisions, minus the cost of maintaining these prediction markets, relative to the cost of the other forecasting mechanisms. According to Robin Hanson, a highly accurate prediction market has little value if some other forecasting mechanism(s) can provide similar accuracy at a lower cost, or if very few substantial decisions are influenced by accurate forecasts on its topic.
–
More Info:
– The Best Resources On Prediction Markets = The Best External Web Links + The Best Midas Oracle Posts
– Prediction Market Science
– The Midas Oracle Explainers On Prediction Markets
– All The Midas Oracle Explainers On Prediction Markets
–
Analysis of Barr and Nader 2008 Intrade contracts
I’-ve used the Bob Barr contracts at Intrade to poke fun at the totally unrealistic expectations of Libertarian Party advocates (a couple times at Midas Oracle), so here’-s a brief (and completely amateur) analysis of those contracts (and Nader contracts), post-election.
You may need to click on each chart to see the whole thing.
The probability of Barr obtaining 1% or more of the vote remained about .4 (40%) throughout the past several months. More optimistic scenarios became more discounted as the election grew nearer, and presumably it became clear Barr would not break through. Even 1% would have been seen as a breakthrough by LP advocates, but in the end Barr obtained only 0.4% of the vote. (Note that obtaining .4% of the vote and a .4 probability of obtaining 1% of the vote are very different things!)
Ralph Nader contracts attracted very little trading, though the 1% or greater contracts gave Nader a 60% chance of obtaining 1% of the vote as late as early October. Nader contracts for 3% and above did not trade at all —- or almost not at all —- Intrade’-s web page table (screenshot below chart) shows a few trades, but no advanced chart or closing price/volume download, and there seemed to be an (unrelated?) possible bug with Nader contract reporting fairly consistently —- last trade prices would not be remembered and reported in the aforementioned table —- or it could be user (me) error/misunderstanding.
Both Barr and Nader contracts were traded heavily (for them) post-election, presumably as traders freed up cash and unwound positions —- for unkown reasons Intrade still has not expired the contracts.
Spurred by comments from David Nolan (scan the page for “-Intrade”- or my name), I also attempted to gauge what traders thought about the average vote percentage candidates would receive across all scenarios —- even a small chance of a genuine breakthrough could make an otherwise hopeless campaign (in the LP’-s case, 9 such presidential campaigns prior to 2008) worthwhile. See below for the average (not most likely!) vote percentage over time each candidate might be expected to receive if the campaign were re-run may times. Assumptions: a floor of .5% (cases in which 1% is not met), very generous given that Barr did not reach even that, and only one LP candidate ever has, if candidate crosses threshold, they do so by .5%, also generous, and if 7% (the highest contract) is crossed, they obtain 7.5% of the vote, slightly ungenerous given the non-impossibility of obtaining a much higher percentage of the vote. Longshot bias should also expected to be at play. I don’-t think these numbers should give LP or other third party advocates any comfort, though I admit my own bias on the matter. The average of all Barr scenarios declined steadily as the election approached, while Nader contracts did not trade until closer to the election, and they both ended at an average of 1% across all scenarios just before the election.
In the end Nader received 0.54% of the vote, beating Barr for (a very distant) third place behind Obama and McCain.
All of the data used above may be found in intrade-2008-barr-nader.zip. The spreadsheet file intrade-2008-barr-nader.ods aggregates everything.
2012
I’-d like to see:
- Fewer contracts (in 2008 there were 7 for Barr and Nader each, from 1% through 7%- actually 8 each counting an electoral vote contract that received almost no attention) might be better, and one testing 0.5% of the vote, which is a more relevant barrier for the LP (only broken once, in 1980) —- maybe 0.5%, 1%, and 3%, or perhaps a single log-scaled contract.
- Minor vote total contracts contingent on nominee —- assuming they had attracted trades, would they have expected the more mainstream for US politics (but less mainstream among libertarians) Bob Barr to outperform other candidates for the LP nomination?
- Even better than contingent nominee/vote total contracts, contingent nominee/some measure of welfare or liberty contracts. Presumably these would show no difference among LP nominees, on the theory that no such nominee will make a difference in the world.
- More trades! Why are libertarians afraid of the market?
Would InTrade or BetFair have done a better job predicting how many people would see the Barack Obama infomercial?
As I disclose on the other blog (Midas Oracle .COM), the HubDub traders wrongly thought that the (Socialist) Barack Obama informercial would have be seen by “-more than 50 million”-. No, no, no, no. It was 33.5 million, according to Nielsen. The HubDub traders were too optimistic.
Would the InTrade or BetFair traders have done a better job? Jesus, that’-s a difficult question. Which boils down (mainly, but not uniquely) to whether some experts on TV business were quoted in the media with some predictions. To investigate that, I have run a Google News search for news articles published before October 30. I haven’-t seen any expert predicting how many viewers would get that infomercial. However, here’-s what I spotted in the New York Times article published in the morning preceding the airing of that infomercial:
Ross Perot, the last presidential candidate to run similar programming, broadcast eight long infomercials to an average of 13 million viewers, with one of them getting 16.5 million viewers.
Hummm…- Obviously, the HubDub traders were too cocky with their “-50 million”- figure…- but should we blame them when, obviously (too), the Barack Obama situation circa 2008 is very different than the Ross Perot situation circa 1992?
The HubDub traders were not informed by the Ross Perot history. They simply made the bet that the Barack Obama infomercial would get as many TV viewers as the Third Presidential debate got (56.5 million). They predicted in a gregarious fashion. They lost.
I don’-t think that the BetFair or InTrade traders would have done better. Do you?
–
How many people will watch Barack Obama’-s primetime address on 10/29?
–
–
2008 US electoral college: What I am betting on.
PollTrack:
–
I like the way they color this electoral college map —-with 5 colors only (simplicity is good). It is very clear and usable, I believe. You can see 6 states in gray (”-too close to call”-). I am heavily betting on Barack Obama for Florida and North Carolina. There will be a good payoff, next Tuesday —-maybe. 
–
–
–
Who will win Florida in the 2008 Presidential Election?
–
Who will win North Carolina in the 2008 Presidential Election?
–
–
Explainer On Prediction Markets
–
A prediction market is a market for a contract that yields payments based on the outcome of a partially uncertain future event, such as an election. A contract pays $100 only if candidate X wins the election, and $0 otherwise. When the market price of an X contract is $60, the prediction market believes that candidate X has a 60% chance of winning the election. The price of this event derivative can be interpreted as the objective probability of the future outcome (i.e., its most statistically accurate forecast). A 60% probability means that, in a series of events each with a 60% probability, then 60 times out of 100, the favored outcome will occur- and 40 times out of 100, the unfavored outcome will occur.
Each prediction exchange organizes its own set of real-money and/or play-money markets, using either a CDA or a MSR mechanism —-with or without an automated market maker.
Prediction markets produce dynamic, objective probabilistic predictions on the outcomes of future events by aggregating disparate pieces of information that the traders bring when they agree on prices. These event derivative traders feed on the primary indicators (i.e., the primary sources of information), like the polls, for instance. (Garbage in, garbage out…- Intelligence in, intelligence out…-) Armed with these bits of information, the speculators then trade based on their anticipations, which will be either confirmed or infirmed. Hence, the prediction markets (which are more than just an information aggregation mechanism) are a meta forecasting tool.
The value of a set of prediction markets consists in the added accuracy that these prediction markets provide relative to the other forecasting mechanisms, times the value of accuracy in improved decisions, minus the cost of maintaining these prediction markets, relative to the cost of the other forecasting mechanisms. According to Robin Hanson, a highly accurate prediction market has little value if some other forecasting mechanism(s) can provide similar accuracy at a lower cost, or if very few substantial decisions are influenced by accurate forecasts on its topic.
–
More Info:
– The Best Resources On Prediction Markets = The Best External Web Links + The Best Midas Oracle Posts
–
More Charts Of Prediction Markets
–