Tag Archives: Predictions

Assessing Probabilistic Predictions 101

Posted on by
1

No Gravatar

Lance Fortnow:

[…] Notice that when we have a surprise victory in a primary, like Clinton in New Hampshire, much of the talk revolves on why the pundits, polls and prediction markets all &#8220-failed.&#8221- Meanwhile in sports when we see a surprise victory, like the New York Giants over Dallas and then again in Green Bay, the focus is on what the Giants did right and the Cowboys and Packers did wrong. Sports fans understand probabilities much better than political junkies—upsets happen occasionally, just as they should.

Previously: Defining Probability in Prediction Markets – by Panos Ipeirotis – 2008

[…] Interestingly enough, such failed predictions are absolutely necessary if we want to take the concept of prediction markets seriously. If the frontrunner in a prediction market was always the winner, then the markets would have been a seriously flawed mechanism. […]

Previously: Can prediction markets be right too often? – by David Pennock – 2006

[…] But this begs another question: didn’t TradeSports call too many states correctly? […] The bottom line is we need more data across many elections to truly test TradeSports’s accuracy and calibration. […] The truth is, I probably just got lucky, and it’s nearly impossible to say whether TradeSports underestimated or overestimated much of anything based on a single election. Such is part of the difficulty of evaluating probabilistic forecasts. […]

Previously: Evaluating probabilistic predictions – by David Pennock – 2006

[…] Their critiques reflect a clear misunderstanding of the nature of probabilistic predictions, as many others have pointed out. Their misunderstanding is perhaps not so surprising. Evaluating probabilistic predictions is a subtle and complex endeavor, and in fact there is no absolute right way to do it. This fact may pose a barrier for the average person to understand and trust (probabilistic) prediction market forecasts. […] In other words, for a predictor to be considered good it must pass the calibration test, but at the same time some very poor or useless predictors may also pass the calibration test. Often a stronger test is needed to truly evaluate the accuracy of probabilistic predictions. […]

THE SILICON ALLEY BLOG COMES TO THE RESCUE OF THE PREDICTION MARKETS.

Posted on by
Reply

No Gravatar

Silicon Alley&#8217-s Jonathan Kennedy:

[…] In denouncing prediction markets as &#8220-wrong,&#8221- however, many pundits miss the point. Prediction markets do not provide accurate predictions of the future. (How could they? They simply represent the consensus guess of a group of people who aren&#8217-t prophets). They merely provide the most-informed guess as to what that future is likely to be.

As numerous &#8220-collective wisdom&#8221- studies have shown, the consensus guess is always better than the majority of the individual guesses that are factored into it (not sometimes&#8211-always). The collective wisdom, moreover, is often more accurate than that of ANY individual. Why? Because the market collectively incorporates far more information than is available to any one individual.

Like the stock market, prediction markets don&#8217-t get it right every time. They do, however, provide a useful window into the collective expectations of others&#8211-one that is often the best available estimate of the future. And they do sometimes get it right. Just as they did with Mr. McCain.

Bravo, mister Jonathan Kennedy.

&#8212-

Take that, Barry Ritholtz. :-D

In an upcoming post, we will review the strengths and weaknesses of these thinly traded prediction markets&#8230-

We are holding our breath, Barry. Hurry up.

The market moved but is it news?

Posted on by
3

No Gravatar

In financial markets there is strong evidence to suggest that news gets priced into markets within 15 minutes of its release and sometimes even more quickly. Recent research into prediction markets suggests that they aren’t nearly as efficient with researchers from University of Pennsylvania showing that prices on IEM can be predicted using public news flow.

Doing a simple analysis of some the key events in the 2008 Presidential Elections against prices on Intrade shows that on discrete events there is a clear relationship between prices and news flow. However over longer periods the relationship is not always clear.

On the 4th of March CBS announced the results of a straw poll conducted at the conservative PAC convention in Washington DC. They picked Romney as their favourite. Romney’s price on Intrade lifted immediately where it stayed for about a week.

Romney price

On the 11th of April the Fred Thompson revealed on Fox News and ABC Radio that he had been diagnosed with non-Hodgkin’s lymphoma nearly three years prior. The New York Times and other publications picked up the story the next day. Looking at his price chart shows he opened on the 12th of April at 19 but then closed at 15. The next day he opened at 11.2 but then closed at 17, as the story died down.

Thompson price

In both these cases, the news stories the media considered to be the important ones correspond with the news flow that traders thought was important.

However, the most interesting market movement of the year must be the Obama August slide. On the first of August Obama opened on Intrade at 35.5 but by the 24th of that month he had slide to 17.2. He continued sliding hitting a rock bottom of 10.7 on the 14th of October.

Obama price

The question is what was the news flow on Obama from the 1st of August to the 24th of August? Analysing the news articles in the New York Times suggests a disconnect between what was reported and how the market was reacting. Obama started August badly with a bungled comment on use of nuclear weapons.

Additionally, his continued line that stabilisation of Iraq had been a ‘complete failure’ may also have cost him some points.

However in sum these news items don’t seem to correlate with an 18 point slide. This could lead us to two possible conclusions:

  1. The New York Times didn’t report the most market sensitive news affecting Obama in August
  2. Obama was over-sold in August and his price did not reflect his true value

Cross-posted from the Hubdub blog.