And he could be right. 
The only way to evaluate accuracy of predictions is with a sufficient group or series of predictions.
I don’-t disagree with that. My previous blog post on the Karl Rove prediction market simply stated that:
- The NewsFutures prediction on Karl Rove happened to be wrong.
- The resignation prediction markets are usually wrong.
- There are different kinds of prediction markets. The resignation prediction markets are of the kind where there are no reliable advanced indicators.
Jed Christiansen and Emile Servan-Schreiber want to deny us the right to say that an individual prediction was inaccurate. I respectfully disagree with that. Other than that, I agree with their general point about using long series and understanding the true nature of probabilities. But, in day-to-day life, we all assess the accuracy of individual predictions. While it’-s not the most important angle, that’-s not something to censor, in my view.
On a related note, Midas Oracle should publish more excerpts of papers that assess long series of prediction markets. We will work on it in the future.
—-
Karl Rove will resign from the White House.
(You will spot that the prediction market was predicting, lately, that the probability for a Karl Rove resignation was only about 20%.)
Previous blog posts by Chris F. Masse:
- Become “friend” with me on Google E-Mail so as to share feed items with me within Google Reader.
- Nigel Eccles’ flawed “vision” about HubDub shows that he hasn’t any.
- How does InTrade deal with insider trading?
- Modern Life
- “The Beacon” is an excellent blog published by The Independent Institute.
- The John Edwards Non-Affair… is making Memeorandum (twice), again.
- Prediction Markets = marketplaces for information trading… and for separating the wheat from the chaff.
Hi, Chris. I’m willing to investigate why we’re disagreeing. Here’s a question in that aim:
You believe that the Karl Rove market was wrong. What value would it have needed to trade it two days ago in order to be “right”?
40%? 50%? 80%? 95%?
I repeat that, mainly, you do a good job of reminding us that we should use long series to assess the prediction markets in general, and that we should really understand the true meaning of one probabilistic prediction.
However, on Midas Oracle, as you have seen, in addition to some postings about the meta-analysis done by Justin Wolfers and Eric Zitzewitz and the like, we also publish dynamic prediction markets (like whether Ms. Clinton will become the next US president). Once those prediction markets have expired, I think it is a public service to our readers to inform them about their expiry. And at this point, it’s all natural to say, oh that prediction market was wrong, or that prediction market was right.
It’s all natural to try to do such accountancy of the expired prediction markets, on an individual basis.
Now if your point is that on Midas Oracle, we should publish more meta-analysis, we should remind more often our readers about what really means a probabilistic prediction, or that we should remind more frequently our readers that what counts is not the individual prediction markets but the long series, yes. But I should have the right to publish information about the expiry of single prediction markets, and make a comment about each of them, and whether they were accurate or not, individually.
For the binaries, a prediction market would be accurate if it has been over 50% in the last time period. From there, there’s a graduation to introduce. The best way is to cite the percentage, so readers see whether the absolute accuracy was weak or strong.
The NewsFutures prediction market on Karl Rove signaled a 80% chance that he would stay at the White House. That was clearly not accurate.
Chris, saying that “something has 20% chance of happening” is very different from saying that it won’t happen. In fact, it is saying that it *might* happen, and that there’s one chance in five that it *will*. In any case, that statement simply can’t be wrong if the event does happen (or even if it doesn’t).
You may wish to propose a rule that, as you put it, “a [binary] prediction market would be accurate if it has been over 50% in the last time period”, and that rule might be easy to understand, and expedient, and easy to apply and draw headline-grabbing black and white conclusions from. But that rule would still violate logic and severely misconstrue the meaning of a probabilistic prediction.
I fully agree with the goal of trying to establish ways to assess the predictiveness of a single binary market’s price, but I don’t think your “for dummies” rule cuts it. Let’s dig deeper and ask around… let’s leverage the mind power of the MO audience !
[…] interpretation of the market can also be tricky. When for instance an event with a low probability estimation happens nonetheless. Does it mean […]