GIGO and prophets, tears and markets

No Gravatar

Prediction markets failed to accurately predict the unexpected effect a few tears had on the New Hampshire primaries- and some analysts rushed to blame the tool and undermine its reliability and applicability. Let me restate some fundamentals and my view, in a snapshot:

  • Markets are not prophets, prophets do not exist.
  • A mechanism&#8217-s forecastability should not be judged against a virtual fool-proof prophet- we&#8217-d better compare it with other existing or widely-used mechanisms and -to my partial and context-bound knowledge- markets outperform all those.
  • Markets are the only tool that intrinsically suggests their probability of failure. If Obama&#8217-s stock is traded at 70 cents, this suggests that there is a 30% probability of Obama losing- I&#8217-d say markets are by character modest and no fanfare has any place in describing their suggestions.
  • Markets are primarily an aggregation/meta mechanism- as such, garbage-in-garbage-out effects are expected to happen, so we&#8217-d need to keep focus on minimizing garbage rather than blaming the market/compiler.
  • Maturity of the mechanism and its use, as long as trading volume (in real-money intrade for example), have not yet reached a fully efficient level (more on this to come soon), but these result in significant profit opportunities, so I expect things to just keep getting better.

cross-posted from my blog

Who did best in explaining the prediction markets to the lynching crowd?

No Gravatar

After the New Hampshire fiasco, 16 18 people came to defend the prediction markets, so far. So far, the best takes are from:

  1. George Tziralis
  2. Robin Hanson
  3. Jonathan Kennedy
  4. and I&#8217-ll give the 4th spot to a combo, mixing takes from John Tierney, Adam Siegel (surprisingly pertinent &#8211-I bet he is on a fish diet, post Christmas :-D ), and Steve Roman.
  5. UPDATE: &#8220-Thrutch&#8220-, Emile Servan-Schreiber and Panos Ipeirotis.

AWOLs (so far): PMIA, AEI-Brookings, InTrade, TradeSports, BetFair, TradeFair, NewsFutures, Emile Servan-Schreiber, Jed Christiansen, Koleman Strumpf, Bo Cowgill, Richard Borghesi, Chris Hibbert, David Perry, Ken Kittlitz, Paul Tetlock, David Pennock, Mike Linksvayer, Brent Stinsky, David Yu, Mark Davis, David Jack, James Surowiecki, Tyler Cowen, Greg Mankiw, Donald Luskin, John Delaney [*], etc.

[*] Steve Bass tells us that John Delaney&#8217-s pre-NH CNBC appearance was awesome. I was up that day, waiting for that CNBC segment, but failed to spot it. If somebody sends me the YouTube link, I&#8217-ll publish it here.

THE SILICON ALLEY BLOG COMES TO THE RESCUE OF THE PREDICTION MARKETS.

No Gravatar

Silicon Alley&#8217-s Jonathan Kennedy:

[…] In denouncing prediction markets as &#8220-wrong,&#8221- however, many pundits miss the point. Prediction markets do not provide accurate predictions of the future. (How could they? They simply represent the consensus guess of a group of people who aren&#8217-t prophets). They merely provide the most-informed guess as to what that future is likely to be.

As numerous &#8220-collective wisdom&#8221- studies have shown, the consensus guess is always better than the majority of the individual guesses that are factored into it (not sometimes&#8211-always). The collective wisdom, moreover, is often more accurate than that of ANY individual. Why? Because the market collectively incorporates far more information than is available to any one individual.

Like the stock market, prediction markets don&#8217-t get it right every time. They do, however, provide a useful window into the collective expectations of others&#8211-one that is often the best available estimate of the future. And they do sometimes get it right. Just as they did with Mr. McCain.

Bravo, mister Jonathan Kennedy.

&#8212-

Take that, Barry Ritholtz. :-D

In an upcoming post, we will review the strengths and weaknesses of these thinly traded prediction markets&#8230-

We are holding our breath, Barry. Hurry up.

Five Reasons the Prediction Market Critics Are Wrong.

No Gravatar

1. It really was an upset – As it has been pointed out elsewhere, the Clinton victory was a surprise to everyone. Favorites can lose. But so what? Ordinarily, that’s not a market flaw or a reason to doubt the odds shown in the market.

Justin Wolfers article in the WSJ had the best summary:

Against this background, it is no exaggeration to term the result truly historic. Not that there haven&#8217-t been more dramatic upsets or come-from-behind wins that carried more significance &#8212- this was just an early primary, albeit a pivotal one. But in terms of unpredictability, or at least the failure of everyone to predict it, it may have no modern match.

Historical comparisons are already being drawn between the New Hampshire primary and the famous 1948 presidential race…Yet the magnitude of the Clinton surprise is arguably even greater&#8230-Thus, Sen. Clinton&#8217-s victory on Tuesday was more surprising than President Truman&#8217-s in 1948.

Given the above, were the Clinton prices on Intrade very far off? It&#8217-s not obvious that they were.

2. Pundits/Critics are NOT traders – If I believe a contract should be trading around 30 and I see it trading at 7, it would make my day. As a trader, seeing a contract that is clearly mispriced is a good thing. Traders who remember the French politician Le Pen’s strong showing in 2002 vs his polls or who read Steve Sailer’s blog should not be surprised that people are dishonest with pollsters. However, to a pundit, an isolated incident of mispricing means the entire concept of prediction markets is faulty.

Since NH results, pundits have been asking, “Are prediction markets flawed?” The traders who make and move the market don’t believe so- they are trading more than ever. In any case, there were no postings on the 7th of January about how wrong the prediction markets are, only after-the-fact postings demonstrating perfect 20/20 hindsight. Traders, not critics, will determine the success of the prediction markets.

Let us not forget that pundits have an agenda too. For some, especially political ones, they need to present themselves as being able to offer insight that no one else has. Since prediction markets allow events to be quantified in real time, the pundits have less to add. This makes critics especially eager to take some of the shine off prediction markets and make themselves look smarter by comparison.

Additionally, there is a contingent of commentators and bloggers with an anti-market bias who delight in seeing any market based tool be wrong. They will be the first to loudly smear PM errors but no where to be found when the market turns out to be right.

3. PMs are not polls – This common mistake is exemplified by this quote from the Chicago Tribune, “The New Hampshire primary was a reminder that prediction markets, where bettors are putting money on the line, can have no more value than opinion polls, where participation costs nothing.” This critic missed the point and doesn&#8217-t realize he is comparing apples and oranges.

Most commentators have focused on the accuracy of the market prices without touching on the underlying purpose of the market: speculation and hedging. Even if the polls are no more accurate than the market, they still can’t be used for trading functions.

4. Regulations have hurt the accuracy and liquidity of PMs – The inconvenience of opening a trading account at Intrade has excluded many Americans from participating. What is the cost of accuracy to the PMs? Surowiecki’s The Wisdom of Crowds lists four factors necessary for a wise crowd: diversity of opinion, independence, decentralization, and aggregation. At least two of these have been highly restricted due to regulations. Even so, the market is usually more accurate than the polls. None of the critics has pointed out that with so many potential traders cut off from trading, the market is surely excluding informed participants.

5. “Serious people who study or work with these markets are not in the &#8216-markets are magic&#8217- camp” – Prediction markets are like other financial markets: fat tails, black swans, bubbles, “manipulations” etc. These are all visible in housing, equities, and fixed income markets as well and no one speculates about the end of those instruments. As Eric Zitzewitz pointed out, the “markets are magic” crowd is just a strawman and not a logical basis to attack prediction markets.

Digg Link:

http://digg.com/business_finance/Top_Five_Reasons&#8230-

Prediction Market Industry Association = useless, so far

No Gravatar

We have witnessed a backlash against the prediction markets just after the Hew Hampshire fiasco. Some bloggers and journalists picked on the prediction markets (InTrade, that is), even though both the polls and the pundits were awfully wrong too. [*] Here are the persons who participated in the pro-PM side of the debate:- Robin Hanson at Overcoming Bias (the best pro-PM piece so far, although his phrasing is a bit too long and a bit too complicated for the average citizen)-

– Justin Wolfers in the Wall Street Journal (who did not convince Felix Salmon, who in tun did not convince me :-D )-

– Chris Masse at Midas Oracle (see Tim Harford&#8217-s new post to discover how irrational Chris Masse really is :-D )-

– Jason Ruspini in a comment here-

– Caveat Bettor on Caveat Bettor

– and John Tierney in the New York Times (a special case I&#8217-ll blog about soon).

[UPDATE: Jonathan Kennedy.]

[UPDATE: Mike Giberson.]

[UPDATE: Eric Zitzewitz.]

[UPDATE: Cass Sunstein.]

[UPDATE: Steve Roman,]

[UPDATE: Nigel Eccles.]

[UPDATE: The Everyday Economist.]

[UPDATE: Adam Siegel of Inkling Markets.]

[UPDATE: George Tziralis.]

[UPDATE: Leighton Vaughan-Williams.]

[UPDATE: Emile Servan-Schreiber of NewsFutures.]

[UPDATE: “Thrutch“.]

[UPDATE: Panos Ipeirotis.]

[UPDATE: Sean Park.]

[UPDATE: Lance Fortnow.]

[UPDATE: Jed Christiansen.]

&#8212-

[*] For why the polls were wrong, see: The New York Times, Zogby, Rasmussen, Gallup&#8230- [Thanks to Emile Servan-Schreiber of NewsFutures for one link.]

The prediction markets deserve a fair trial.

No Gravatar

Niall O&#8217-Connor:

Sadly, from our analysis, the impression that one gets is of a market that is spooked by poll results– driven by stale news- and heavily influenced by gossip and rumour.

Questions:

  1. Could Lord O&#8217-Connor cite the name of a more accurate forecasting tool?
  2. Could Lord O&#8217-Connor publish his own track record at predicting the US and British elections?
  3. Could Lord O&#8217-Connor give one example of an infallible human institution?
  4. Could Lord O&#8217-Connor state publicly whether he believes in knowing the future in advance with 100% accuracy? (If yes, then I&#8217-ll suggest to the CIA to hire him to get Bin Laden.)

Previously: Prediction markets are forecasting tools of convenience that feed on advanced indicators.

Prediction markets are forecasting tools of convenience that feed on advanced indicators.

No Gravatar

Why were the political prediction markets so wrong about Barak Obama and Hillary Clinton in New Hampshire?

&#8230-asks Slate&#8217-s Daniel Gross &#8212-via Mister Usability (Alex Kirtland), who needs to go and get his own gravatar.

So, I&#8217-ve been watching the action in one of the political futures markets this evening, Intrade. And the action in this prediction market has reinforced my opinion that these are less futures markets than immediate-past markets. The price movement tends to respond to conventional wisdom and polling data- it doesn&#8217-t lead them.

Throughout the day and into the early evening, while polls were still open, Democratic investors, mimicking the post-Iowa c.w. and polls, believed Obama was highly likely to be the Democratic nominee. The Obama contract was trading in the lows 70s, meaning investors believed he had a 70 percent chance of being the nominee, while Hillary Clinton contracts were in the 20s. […] At 6 p.m., this market had written Hillary Clinton&#8217-s entire presidential campaign off. At 9:30 p.m., it was calling a dead heat. What caused investors to change their minds so drastically in the space of a couple of hours? A few data points that went against the day&#8217-s prevailing conventional wisdom and polls. […]

See also Niall O&#8217-Connor&#8217-s assessment:

I am looking forward to the post New Hampshire Caucus, when all you prediction market advocates crawl out from under your stones. For the record at one point the market on Intrade and Betfair was suggesting that Obama had a 95% probability of winning the caucas- whilst Intrade had him at 77% to win the nomination.A case perhaps of both the foolery of crowds and, the market biting back.

New Hampshire will go down as the Black Wednesday of prediction markets and unless there is now objective transparent debate (as opposed to the usual biased sabre rattling) – prediction markets will be dead in the water.

My answer to Dan Gross&#8217- legitimate question and to Niall O&#8217-Connor&#8217-s snarky comment:

  1. Prediction markets are forecasting tools of convenience that feed on advanced indicators. When those advanced indicators are wrong, the prediction markets are wrong.
  2. If you prefer the polls or the pundits, your call &#8212-but polls and pundits were also wrong, this time, right? Required reading for mister Niall O&#8217-Connor: &#8220-New Hampshire&#8217-s Polling Fiasco&#8221- + &#8220-Analysis: pundits eat crow&#8220-.
  3. The ultimate forecasting tool would be a way to reverse our psychological arrow of time &#8212-so as to remember the future instead of the past. Only science-fiction writers and some imbecile ( :-D ) believe in that.
  4. The prediction market approach is to stick with the markets, on the long term. Take their successes. Take their failures. Unlike Donald Luskin and Markos Moulitsas, Chris Masse will not turn against the prediction markets when they fail punctually. What counts is the long series.
  5. My first point should be included in the prediction markets approach definition, in my view, but others (like economist Michael Giberson) might have different opinions.
  6. With respect to my first point, I bet that the prediction markets will never replace the polls as the forecasting tool of choice for political analysts &#8212-on that particular point (but not on a myriad of others), I break away from Justin Wolfers&#8217- irrational exuberance and I side with Emile Servan-Schreiber of NewsFutures (my preferred play-money prediction exchange). Prediction market reporting will have a function, indeed (as suggests Justin Wolfers), but not the dominant function.
  7. Going forward, prediction market journalism should emphasize relative accuracy (as opposed to absolute accuracy) &#8212-that is, comparing prediction markets with polls and pundits, which is what Robin Hanson has said from day one. Our good friend Niall O&#8217-Connor has difficulty to compute that, apparently. He should eat more fish. :-D

&#8212-

Justin Wolfers:

In a few years, we may regard the second half of the 20th century as the aberration in which the press used polls rather than markets to track political races,” Justin Wolfers, a business professor at the University of Pennsylvania’s Wharton School, wrote in an e-mail message. “And in the 21st century, we may return to the habits of the early 20th century, reporting on political races through the lens of prediction markets rather than polls.

Emile Servan-Scheiber:

1) The traders themselves are the first to look at the polls to inform their trades. So the polls are here to stay.

2) Our recent experience in Western Europe seems to indicate that the superior accuracy of markets over polls when predicting elections may be a U.S. artifact that isn’t so easily reproducible elsewhere. I’ve discussed this with Forrest Nelson of IEM [Iowa Electronic Markets], and apparently, ever since the Truman-Dewey polling debacle of 1948, U.S. pollsters have adopted a policy of reporting mostly raw numbers rather than projections based on sophisticated secret formulas, so they can’t be accused of manipulating opinion. However, raw numbers are notoriously unreliable when based on small samples, and Western European pollsters never report them, preferring instead to publish projections based on historically-informed statistical formulas. What we’ve observed in France and Holland is that it it’s very hard to beat the accuracy of such projections.

[I don’t make mine Emile Servan-Schreiber’s second point, but that’s a minor.]

&#8212-

InTrade&#8217-s expired prediction markets:

&#8212-

New Hampshire

&#8212-

The Democrats

&#8212-

The Hillary Clinton event derivative was expired to 100.

Dem NH Clinton

Dem NH Obama

Dem NH Edwards

&#8212-

The Republicans

&#8212-

The John McCain event derivative was expired to 100.

Rep NH McCain

Rep NH Romney

Rep NH Huckabee

Rep NH Giuliani

&#8212-

Iowa

&#8212-

The Democrats.

The Barack Obama event derivative was expired to 100.

Dem Iowa Obama

Dem Iowa Clinton

Dem Iowa Edwards

&#8212-

The Republicans

The Mike Huckabee event derivative was expired to 100.

Rep Iowa Huckabee

Rep Iowa omney

Rep Iowa McCain

&#8212-

Source: InTrade

&#8212-

[A more complete prediction market reporting should have included expired contracts from NewsFutures and BetFair. Sorry for that. Note that InTrade-TradeSports is the only exchange to offer a “closed contacts” section.]

&#8212-

NEXT: Prediction Markets 101 + Who did best in explaining the prediction markets to the lynching crowd? + After the New Hampshire fiasco, 16 people came to defend the prediction markets, so far. + The prediction markets deserve a fair trial. + Prediction Markets = the greatest time-saving invention of this century

&#8212-

Felix Salmon slams Wall Street Journals Justin Wolfers -but not Chris Masse.

No Gravatar

Felix Salmon:

[…] And thirdly, if you&#8217-re Justin Wolfers, it&#8217-s probably smart not to make unhedged statements saying that Barack Obama has &#8220-better than a nine-in-ten chance of winning&#8221- the New Hampshire primary. […]

Previously: Prediction markets are forecasting tools of convenience that feed on advanced indicators.

Since Chris must sleep at some time (I think)…

No Gravatar

&#8230- I&#8217-ll alert you to a developing story. [Slate’s Daniel Gross: Why were the political futures markets so wrong about Obama and Clinton?]

Thanks to a friend.

~alex

Prediction Markets as Content, Part 2

No Gravatar

Cross posted from UsableMarkets

Back in April I started talking about how Prediction Markets will be part of many news organizations&#8217- &#8220-citizen-generated&#8221- content strategy going forward.

To quote myself (which seems kind of a rude thing to do, doesn&#8217-t it &#8230-?):

It seems as if no self-respecting news organization can ignore the Web 2.0 movement these days. Many now have some sort of &#8220-wisdom of the crowds&#8221- style content, in addition to RSS feeds, blogs, and so on.

Midas Oracle has covered some of the new relationships that are developing. I recently talked about MarketWatch.

Expect more to happen &#8230- and perhaps quickly, too.

That was nine months ago. Since then we&#8217-ve seen the WSJ, the FT, Reuters, CNN, and others (perhaps everyone can think of a couple or three) begin to dabble in or seriously consider prediction markets. With Inkling and InTrade in the white label prediction market business, the barriers to setting one up are obviously low enough that a certain amount of me-too-ism can easily prevail.

But there is a risk, and those of us who care about the success of the prediction market industry shouldn&#8217-t get too excited about these developments just yet.

First, it remains to be see whether these new prediction markets can attract significant numbers of users. The prediction market industry is already saturated with prediction markets and games. So, despite their powerful brands, I&#8217-m not confident that the FT or the WSJ can attract large followings (although I&#8217-d be happy to be wrong about that).

Ah ha, you may say, we don&#8217-t need a lot of users to generate accurate predictions. The MSR, and automated market makers will help solve the problem. But the problem is not one of generating accurate predictions, but about generating page views. Newspapers (even online) are advertising driven. If you can&#8217-t generate sufficient page views, and you&#8217-re paying too much to manage the prediction market on your site, then it&#8217-s vulnerable to being cut. In fact, I wouldn&#8217-t be surprised if once this US election cycle is over that some of these markets fall away.

And, if the news organizations are really interested in the predictions for predictions sake, they can always simply use someone else&#8217-s.

As always, thanks for listening.
~alex (UsableMarkets)