The Emperor of enterprise prediction markets is naked.
Robin Hanson:
I can confirm that this disinterest is real. For example, when I try to sell firms on internal prediction markets wherein employees forecast things like sales and project completion dates, such firms usually don’t doubt my claims that such forecasts are cheap and more accurate. Nevertheless, they usually aren’t interested.
How do we know the Intrade price was not accurate? Well, the raid wasn’t just executed on a whim. It had been planned for quite some time. Therefore, the true likelihood must have been much higher than the four or five percent chance the market was telling us.
Many people knew of the plans, albeit they were very high ranking, sworn-to-secrecy types. Since the market did not reflect the potential success of the planned raid, either the market was inefficient or the market, in an aggregate sense, did not possess enough information to make a reasonably informed prediction. In this case, I have to believe the market participants (every last one of them) knew next to nothing about the outcome being predicted.
The Intrade market prediction was nothing more than an aggregation of guesses. This is very different from an accurate prediction (based on calibration) that turns out to be wrong.
Markets such as these have no use, whatsoever, in decision-making. The useful information was that gathered by the SEALs and other secret services, and that was the information provided to the real decision-maker, The President. I would argue that these types of markets have no place as betting markets either. There is no way to test the calibration, so we don’t know whether they are “fair” markets (unlike the accurate calibration of horse races and casino games).
In other words, stop wasting our time operating and analyzing these markets. They are never going to be useful.
How do we know, now, that Intrade’-s market price was not an accurate estimate of the probability bin Laden was killed or captured by September 2011? Is an prior estimate of 50 percent likelihood that a tossed coin will come up heads wrong if the coin comes up as “-100 percent”- heads (and not half-heads and half-tails)?
I’-m not buying Chris’-s implied definition of success and failure.
However, one might ask Robin Hanson about what the Intrade market’-s performance implies about the usefulness of his Policy Analysis Market idea.
Note that I was contrasting the InTrade-Bin-Laden failure with the high expectations set by Robin Hanson, Justin Wolfers and James Surowiecki.
Also, other than statisticians, most people don’-t have a probabilistic approach of InTrade’-s predictions. That’-s the big misunderstanding, which is one part of the big fail of the prediction markets.
I already blogged about the big fail of the prediction markets. Here’-s more from the NYT, Eddy Elfenbein, and Barry Ritholtz.
Recently posted to SSRN: FantasySCOTUS: Crowdsourcing a Prediction Market for the Supreme Court, a draft paper by Josh Blackman, Adam Aft, &- Corey Carpenter assessing the accuracy of the Harlan Institute’-s U.S. Supreme Court prediction market, FantasySCOTUS.org. The paper compares and contrasts the accuracy of FantasySCOTUS, which relied on a “-wisdom of the crowd”- approach, with the Supreme Court Forecasting Project, which relied on a computer model of Supreme Court decision making. From the paper’-s abstract:
During the October 2009 Supreme Court term, the 5,000 members made over 11,000 predictions for all 81 cases decided. Based on this data, FantasySCOTUS accurately predicted a majority of the cases, and the top-ranked experts predicted over 75% of the cases correctly. With this combined knowledge, we can now have a method to determine with a degree of certainty how the Justices will decide cases before they do. . . . During the October 2002 Term, the [FantasySCOTUS] Project’s model predicted 75% of the cases correctly, which was more accurate than the [Supreme Court] Forecasting Project’s experts, who only predicted 59.1% of the cases correctly. The FantasySCOTUS experts predicted 64.7% of the cases correctly, surpassing the Forecasting Project’s Experts, though the difference was not statistically significant. The Gold, Silver, and Bronze medalists in FantasySCOTUS scored staggering accuracy rates of 80%, 75% and 72% respectively (an average of 75.7%). The FantasySCOTUS top three experts not only outperformed the Forecasting Project’s experts, but they also slightly outperformed the Project’s model –- 75.7% compared with 75%.
You can download a copy of the draft paper here.
[Crossposted at Agoraphilia, Midas Oracle, and MoneyLaw.]
Justin Wolfers is asked the question, but I would have a different answer than his.
The reason prediction markets are not widely used in business is that their many boosters (Robin Hanson, James Surowiecki, Justin Wolfers, etc.) have exaggerated their usefulness. Just because they are objective in their wisdom does not mean that they are very useful.
Objectivity is over-rated. This is a painful lesson for the handful of young startups who swallowed the prediction market myth. Next step: the dead pool.
I gave a talk on Thursday night at the New York Investing Club meeting.
The basic points:
Gold does well when real rates of return are low. Real rates describe the price of gold much better than inflation alone. This is because real rates reflect the opportunity cost of holding a relatively useless asset. Part of the reason gold seems irrational is that this extrinsic pricing is unintuitive and largely unappreciated.
Gold does well when liquidity, measured for example by LIBOR, is not especially tight.
Sentiment can be predictive with gold.
The “-extrinsic”- way of thinking is natural in the fx world where all trades are two?sided, and the idealized one?sided currency , e.g. Dollar Index, is a weighted average of two?sided rates. Other examples: the Fed Model and Dividend Discount models explicitly tie together the pricing of equities and interest rates. The housing bubble was to some extent already a mispricing of money in the form of interest rates. What was the “right” price for housing given the price of money?
Do people who claim that assets exhibit “irrational” moves have a clear idea of what level of volatility would be “rational”, especially given such cross?influences? I do not endorse the Dividend Discount Model, but no-one can deny that it is a fundamental model, and it predicts higher volatility when rates are low. Given current levels, a 10% one day drop of the market is by no means absurd. The stock market should also have more idiosyncratic volatility when it is driven by “-top-down”- policy, rather than averaging many “-bottom-ups.”-
There is perhaps a “-long-termism”- fallacy. Even if prices change glacially, if you want to maintain a portfolio limited to 30 stocks out of a universe of 6000, it is easy to see how a sensible person might change the “-best ideas”- list with some frequency. The more prices change, the more frequent portfolio changes would be in order from a valuation standpoint. Again, asset prices are not hermetically sealed, one?sided meanings and values. There is always some discount factor or relative valuation at play.
The easiest way to achieve a shock “20 standard deviation” move is to just not mark (or mis?mark) for a while. Deferral of pricing is much more likely than active trading to originate an explosion large enough to affect the underlying economy. Deferral of pricing, not active trading, played a large role in the corporate credit crisis. Social Security and entitlement programs are also “off balance sheet” debt. At least banks failed to predict the future. Governments failed to predict the past. The basic demographic and longevity trend has been apparent since at least the 1960s.
Demographic trends suggest lower real returns than those seen in the 20th century. Do economists have a demographic blind spot?
First, a hearty congratulations to Robert Swagger and Trend Exchange. Along with the Cantor Exchange folks, they have run quite a gauntlet, and although there remains a tremendous obstacle in the form of the Lincoln amendment, I consider these exchanges to have already accomplished a great deal.
In its approval of Trend Exchange and preceding statements, the CFTC has confirmed a very broad definition of “-commodity”- that includes “-event”- contracts. The old debate about whether or not the CFTC has jurisdiction over “-prediction markets”- has been decided for now. Yet, there is considerable dissent within the Commission. Commissioners Chilton and Sommers have expressed disapproval that the Commission did not first address the general questions raised in the 2008 Concept Release. To this point, given the very broad definition of “-commodity,”- it now seems that Intrade and online sports exchanges could be in violation of the Commodity Exchange Act. The Commission does not consider an “-economic purpose test”- in the contract review process, and there is no statutory basis for such a test being used in jurisdictional determinations. Perhaps as a matter of practice, in accordance with the spirit of the Act, the Commission is considering such a test for jurisdicitional questions as I suggested in my Concept Release comments (surprisingly cited by the MPAA group). Otherwise, it seems inconsistent that exchange-traded sports bets, for example, would not also be considered commodities and be subject to the Act.
As a whole, the Commission has apparently decided to defer such questions and focus on specific techniques for ensuring that the new contracts fulfill the Act from the standpoints of manipulation and fair trading. To these ends, the CFTC will require, “-entities and individuals who control a film’s marketing budget, release date or opening screen number to provide the Exchange with information regarding such decisions whenever that entity or individual holds a position of 1,000 or more contracts.”- Additionally, the Commission will require a “-firewall”- within studios and distributors, and has restricted certain employees from trading altogether. These are procedures that I had recommended for event contracts, but they are relatively novel mechanisms in the commodities world. Whether or not the CFTC would agree to support special trading restrictions was the pivotal question in whether the contracts would be approved. I applaud the principled, politically independent thinking of the Commission and the can-do attitude of the Market Oversight Division —- though some headline risk has been assumed here if something should eventually fall through the cracks.
Jacqueline Best’-s The Limits of Transparency is carried by sensible concerns on the tension between efficiency and stability, the tension between promoting employment and stable money, and the political implications of seemingly neutral economic policy. However, while economists, market participants and academics will find much that is thought-provoking here, the book does not actually deliver a compelling critique of transparency.
Much more time is spent promoting institutional arrangements of “-constructive ambiguity”- (e.g. Bretton Woods) than pointing out short-falls with transparency. The problem is that what’-s being argued for under the aegis of “-ambiguity”- is just flexibility, mostly in the form of deferring decisions. (Not to mention that institutional arrangements can also become inflexible.) Clearly ambiguity is desirable when it manifests itself as flexibility, but this does not mean that ambiguity is desirable in and of itself. Best makes some pertinent observations (that are, for instance, relevant to CDS dynamics) but tries to wrap them up in a pleasingly ironic theory that alone threatens to undermine her conclusions. Early on, she tells us that she is “-drawing inspiration from both postmodernist and critical theoretical traditions.”- The thematic emphasis on ambiguity is no doubt influenced by such strains of thought, which stress “-the irreducibility of difference.”-
The early Bretton Woods regime is lauded for its ambiguity and “-continuous deferral”- of adjustments, and this points the the fundamental problem in the analysis. While active trading may increase short-term volatility, the author espouses a philosophy of deferral that is actually more dangerous and more at the cause of the financial crisis than active trading. Deferral of pricing creates the illusion of constancy where there is actually change, and enables longer-term booms and busts that are more pernicious and cause longer-lasting pain and dislocation than the “-intersubjective”- spirals the author warns against.
Take an example: traders push CDS prices up, causing the underlying entity’-s borrowing costs to rise, thus ensuring default. This “-intersubjective”- dynamic would not be possible were there not already a fundamental basis for concern over the entity’-s debt —- if the entity had not already irresponsibly deferred its obligations. Which problem is more fundamental? This of course is exactly what happened in the credit crisis, as banks refused to transparently price their holdings. “-Continuous deferral”- is a recipe for less frequent, perhaps, but almost certainly more intense explosions. Is this kind of deferral not at the heart of the impending entitlement crisis, another kind of “-off balance sheet”- debt? It doesn’-t matter if pricing will never have a perfect technical solution, you still try to price up to the point of ambiguity, which, as Best repeats, is always there anyway.
Lastly, the author holds out the Bretton Woods period as one of unsurpassed prosperity while ignoring the corresponding population boom and other factors contemporaneous with the currency regime. For their sake, Keynesians need to heed the demographic winds more closely.
[Originally posted to Amazon, 2/15/10.]