Tag Archives: accuracy

Super Tuesday = Free money, if you are smarter than the crowd

Posted on by
4

No Gravatar

At Overcoming Bias, Eliezer Yudkowsky invites pundits, partisans, and anyone else with a nascent opinion about the limits of prediction markets to, in effect, put up or shut up. (Though he puts it in somewhat nicer words). Here is a selection, but read the whole thing:

If you think that Hillary is going to do better than the polls on Super Tuesday, and you&#8217-re going to sneer afterward and say that Intrade was &#8220-just tracking the polls&#8221-, buy Hillary now.

If you think that Obama is going to do better than the polls on Super Tuesday, and you&#8217-re going to gloat about how prediction markets didn&#8217-t call this surprise in advance, buy Obama now.

&#8230-

The point is not that prediction markets are a good predictor but that they are the best predictor. &#8230- If prediction markets react to polls, they&#8217-re getting new information, that they didn&#8217-t predict in advance, which happens. Being the best predictor doesn&#8217-t make you omniscient.

Everyone&#8217-s going to find it real easy to make a better prediction afterward, but if you think you can call it in advance, there&#8217-s FREE MONEY GOING NOW.

Buy now, or forever hold your peace.

Fundamentals of Prediction Markets: Probabilities, Prediction Timescale, and Absolute & Relative Accuracy

Posted on by
1

No Gravatar

Jed Christiansen outputs the best explainer on prediction markets I&#8217-ve seen in years. Go read it.

– Fundamentals of Prediction Markets
– Different types of Prediction Markets
– Problem #1 – Understanding Probabilities
– Problem #2 – Prediction timescale
– Problem #3 – Assessing accuracy
– Problem #4 – Compared to what?
– Summary – How have the political prediction markets really performed?

Did the BetFair blog use trading data from InTrade to hint at BetFairs accuracy??

Posted on by
3

No Gravatar

Latest update on the BetFair blog fiasco

&#8212-

I am alerted today that the BetFair blog has updated its infamous Michigan story with a new compound chart bearing a clearer label. It reads now:

Republican nomination – The race so far

I&#8217-ll have some comments, below the chart, but first a technical note. The new chart posted is a 527-KB BMP image. I have replaced it with a 32-KB JPG image. The BetFair blog is not run professionally. Any web publisher knows that images should be reduced to the max. That&#8217-s the ABC of web publishing. (And to add insult to injury, I noted previously the technical bizarrery that the two professor Leighton Vaughan-Williams&#8217-s stories never appeared in the BetFair blog feed.)

For you information, I have updated all my previous blog posts on the topic with an addendum re-publishing this new chart.

Compound chart - BetFair blog fiasco

&#8212-

UPDATED ANALYSIS OF THE BETFAIR BLOG FIASCO:

  1. Professor Leighton Vaughan-Williams should have defined what he means by &#8220-betting markets&#8221-, in his story. In the past (see the addendum of that story), prof Leighton Vaughan-Williams used two types of cocktail &#8212-one including all betting markets (traditional bookmaker odds and exchange odds), one including only the BetFair odds. He should publish an addendum to his story defining exactly what he means by &#8220-betting markets&#8221-, this time.
  2. The BetFair blog editor should not have pasted a BetFair compound chart behind the writer&#8217-s back. It&#8217-s a big no-no in editing. Again, another proof (in a long list) that the Betfair blog is not run professionally.
  3. If a chart were to be inserted on top of LVW&#8217-s story (with his consent, we hope), it should have been the expired chart(s) of the Michigan primary, since that&#8217-s the heart of LVW&#8217-s story.
  4. The fact that the BetFair blog editor pasted (behind LVW&#8217-s back) a BetFair chart lead the readers (like Niall Or&#8217-Connor and me) to conclude that professor Leighton Vaughan-Williams means &#8220-the BetFair betting markets&#8221- when he writes about &#8220-the betting markets&#8221-. This is probably not the case, but nobody knows for sure &#8212-see my point #1 for the need of an explainer on this.
  5. Now, if professor Leighton Vaughan-Williams means &#8220-the BetFair betting market&#8221- (I assign a low probability on this scenario), then the story looks bad. The story is bullish on the fact that the Mitt Romney event derivative (for the Michigan primary) was predictive. The election-day chart that I published yesterday evening (and republished below) shows Mitt Romney being the favorite starting at 3:00 PM EST on election day&#8230- Kind of a stretch to claim victory for the BetFair betting markets. I&#8217-m still waiting for BetFair to send me the full, historical chart on the Michigan primary.
  6. BetFair should publish all expired charts &#8212-just like InTrade-TradeSports are doing. See my new page, re-publishing some important expired prediction market charts. That way, any controversy could be settled more quickly.
  7. With all due respect to him, it looks bad on professor Leighton Vaughan-Williams for giving his writings to a corporate blog where the publisher and editor&#8217-s names are not listed anywhere, and whose overall content quality is feeble &#8212-to say the least. Especially since we read the testimony of a furious Betair blog writer, who described the BetFair blog editor as anonymous, incompetent and tyrannical.
  8. Besides Niall O&#8217-Connor&#8217-s critical comments, professor Leighton Vaughan-Williams&#8217-s story on the BetFair blog has attracted a negative comment, calling his argument &#8220-questionable to say the least&#8220-, and asking (as I am doing on this current post) for more data to be published in an addendum.
  9. It looks bad on the BetFair management for publishing completely crappy stories like that. It damages the BetFair brand. I should tell my readers, though, that the BetFair-TradeFair managers (like Michel Robb, Tony Clare, Mark Davies, David Jack, Robin Marks, etc.) are highly professional, efficient, law-abiding, forward-looking, helpful, ethical, polite, and respectful. It is a real pity that the BetFair blog tarnishes BetFair&#8217-s reputation.
  10. Betair should focus on being a prediction market resource for journalists and bloggers. As of today, they still don&#8217-t provide on their website dynamic charts and expired charts.
  11. As I repeated many times on Midas Oracle, prediction market journalism is hard, complex and costly. It can&#8217-t be done by any living organism (hermaphrodite or not :-D ) simply equipped with a computer and an Internet connection.

&#8212-

As an addendum, I re-publish here the election-day Michigan chart (on the Republican side). As I said, I&#8217-m still waiting for BetFair to send me the full, historical chart. You can see, on this Republican-side chart, Mitt Romney (in red) as the Comeback Kid &#8212-starting at 3:00PM EST on election day (that&#8217-s 8:00 PM, British time, on the chart).

Rep Michigan BetFair

For your information, here&#8217-s what professor Leighton Vaughan-Williams wrote. As I said, an explainer from him is needed to determine whether he means the &#8220-betting markets&#8221- in general (with, or without, BetFair included?) or the &#8220-BetFair betting markets&#8221-.

Professor Leighton Vaughan-Williams on the official BetFair blog:

[…] Those taking the same advice on Tuesday evening [2008-01-15 = date of the Michigan primary] were similarly well rewarded as well-backed Mitt Romney stormed into clear favouritism in the markets and a comfortable victory at the polls. After a blip in the New Hampshire Democratic primary the old certainties – that election favourites tend to win elections – was re-established.

As in the Republican New Hampshire primary, the polls and pundits had declared the race between Senator McCain and Governor Romney as a toss-up while the betting markets pointed to a comfortable victory in both cases for the eventual winners. Once again, in the battle of the polls, pundits and markets, the power of the betting markets to assimilate the collective knowledge and wisdom of the crowd had prevailed. […]

As for the InTrade &#8220-betting markets&#8221-, if that&#8217-s what professor Leighton Vaughan-Williams means (solely, or among others), they show a strong support for Mitt Romney in the last 2 days (which includes election day). Kind of a stretch to claim a victory for the &#8220-betting markets&#8221-. Also, it would be funny to have the (anynomized) InTrade data interpreted on the blog of another exchange (BetFair, a competitor of InTrade-TradeSports) to hint about the alleged strength and accuracy of the BetFair &#8220-betting markets&#8221-. That would be the last drop that breaks the water bucket. Another reason why professor Leighton Vaughan-Williams should come forward to explain what he means by &#8220-betting markets&#8221- in his story. Does he mean the &#8220-InTrade betting markets&#8221-???

(FYI, the Mitt Romney event derivative was expired to 100.)

MI Rep Romney

Psstt&#8230- Sounds like a vertical line is lacking on this chart&#8230- Look at the right end&#8230- Bizarre.

&#8212-

NEXT: No more anonymized trading data, please. State your source(s).

&#8212-

Prediction Market Efficiency vs. Prediction Market Accuracy

Posted on by
1

No Gravatar

Panos Ipeirotis in a comment here:

[W]e should try to separate two things: Market efficiency and market accuracy. Efficiency is the rate in which the market incorporates new information and prevents any arbitrage opportunities. Accuracy is the probability in which the market predicts the correct outcome of an event. The main claim to fame for the [prediction] markets is that they self-report their accuracy, and that “the prices are probabilities”.

We can measure the effectiveness of the market by following the outline discussed above. One axis is the price of the contract at time t before the expiration of the contract and the other axis is the rate in which this event happens. (…60% of the cases the event that trades at 0.6 happens, 30% of the cases the event that trades at 0.3 happens, and so on…). A perfectly accurate market should have a straight line as an outcome when time t gets close to 0. Any deviation of the experimental results indicates an accuracy bias. There are many papers that indicate the favorite-longshot biases in the market (underprice the favorite, overprice the longshots) so there is no need to really repeat this here. An interesting thing is to see how big it can be and still have reasonable accuracy. Furthermore, if we have systematic and robust biases, then we can use a calibration function that will adjust the market prices, compensating for the biases, to reflect real-life probabilities.

Measuring efficiency is a trickier concept. The general definition of efficiency is that “the market immediately incorporates all available information”. Being able to predict price movements indicates inefficiency. Having prices for an event summing up to anything other than 1, indicates inefficiency. However, it is difficult to have a definite proof that the market is efficient. We can only say that “we were not able to spot inefficiencies”. It is very difficult to prove that “the market is efficient”.

The two metrics are, of course, highly connected close to the expiration of the contract. If the market is not efficient, then it will not be accurate, as it will not have had incorporated all the available information, if any material information becomes available just before the expiration of the contract.

Panos Ipeirotis

Why collecting and synthesizing the dispersed available information?

Posted on by
Reply

No Gravatar

Sean Park (after a long, boring introduction to the subject):

[…] The ‘failure’ of New Hampshire was the result of primarily two factors:

  1. It wasn’t a failure. No market is always right. More importantly markets reflect the information available to and the interests of their participants. Basically markets are very efficient mechanisms (I would claim the most efficient) for processing information. No more, no less.
  2. In this particular instance, the probability of the market producing an erroneous forecast was high due to the lack of liquidity. This is a problem of all political markets in the US. Show me a market on the New Hampshire primaries with tens of thousands of participants and millions of dollars traded and I will show you a market that creates more valuable information. BUT it would still on occasion be ’surprised.’

Basically I guess what I’m trying to say is the expectations seem to be set all wrong by many inside the community. I think “prediction markets” – creating markets in information and outcomes is a wonderfully important and valuable thing to do. Equally however I think that anyone that represents such markets as being able to predict the future is a charlatan. What they can do is collect and synthesize powerfully and efficiently all the dispersed available information – using money as the relevance filter. This is very valuable in its own right and is defensible. Promoting prediction markets to true sceptics (ie mainstream American politicians) on the basis that they are a Delphic Oracle is surely a path to certain tears and ultimately is almost guaranteed to fail. [*]

Markets don’t compute unknown unknowns. That doesn’t mean they are useless, just that they have to be understood in context.

[*] How to promote the prediction markets, then? As information collecting tools? Who should use these tools, then? Experts or ignorants? Sean Park does not elaborate further. None of the questions I have asked are answered.

InTrade is no psychic -but what if that bit of truth is systematically said BEFORE, as opposed to AFTER.

Posted on by
Reply

No Gravatar

David Leonhart in his New York Times blog, last week:

The political prediction markets just went through their version of the dot-com bubble. […]

Intrade’s odds have had a very good forecasting record over the last few years, having correctly called every Senate race in 2006, every state in the 2004 presidential election and all but one state in the 2004 Senate races. The odds also correctly called New Hampshire for John McCain this week and now make him the favorite for the Republican nomination- he is given a 38 percent chance, while Rudolph W. Giuliani is given a 29 percent chance.

Intrade’s executives, as well as the academic researchers who study the site, are careful to point out that its contracts provide only odds, not certainties. An outcome that’s given a 20 percent chance of happening should happen 20 percent of the time — not never. […]

The question I asked yesterday was: What would happen if that warning label were to be sticked on InTrade before each election, as opposed to after each predictive debacle? My bet is that, if you suppress the mention of InTrade&#8217-s magical touch, the Irish real-money prediction markets will be far less appealing to people. They want magic. All of the sudden, InTrade is not a psychic anymore, but simply a forecasting tool of convenience for busy people who don&#8217-t want to check the polls in details. This issue is crucial if we want to be able to define what is the &#8220-prediction market approach&#8221- &#8212-as opposed to the &#8220-betting exchange approach&#8221-.

Give me one reason why the political analysts should follow the US primaries thru the prism of the InTrade prediction markets instead of thru the polls. [My question is still unanswered, you will notice. Which shows to you the embarrassment of the prediction market luminaries (or so they think they are).]

Once the true nature of the prediction markets appears more clearly, it becomes evident that they are not tools for the experts, but tools for the ignorants, rather. Which is great, provided that this is said clearly from the start.

Can the prediction markets survive without the over-selling from John Delaney and his little fanboys?

Posted on by
Reply

No Gravatar

Emile Servan-Schreiber:

[…] The classic first line of defense in these cases is to remind people that market “predictions” are really just probabilities, so any one outcome cannot invalidate the approach. The argument is sound and backed up by loads of data. But it would of course be much more convincing if we, as an industry, would remember to show at least as much humility when our market “predictions” appear correct instead. If you’re going to spread the idea that your market called all 50 states in the last U.S. presidential election because each correct outcome was predicted with over 50% chance, then you can’t hide behind probabilities when an 80% prediction comes to naught, as in Obama’s NH collapse. […]

Emile Servan-Schreiber makes a good point &#8212-see also Panos Ipeirotis, in the same vein.

But the over-selling is the reason [*] why InTrade (and not NewsFutures) has managed to infiltrate so many US media. If you suppress the magical touch, then InTrade is just a forecasting tool of convenience &#8212-for those too busy to look at the polls.

Give me one reason why the political analysts should follow InTrade instead of the polls, then?

What is the true nature of the prediction markets? How to use the prediction markets? Who should use the prediction markets? For what benefits? Once you have the answer to these 4 questions, you can tackle the next two problematics: How to market the prediction markets without over-selling them. How to report news thru the prism of the prediction markets while respecting their true probabilistic nature.

Welcome to the version #2 of the prediction market industry. Quite a horse of another color, now.

&#8212-

[*] UPDATE: The over-selling aspect is the topping over the real-money and the liquidity dimensions. The over-selling aspect wraps all that.

Defining Probability in Prediction Markets

Posted on by
5

No Gravatar

The New Hampshire Democratic primary was one of the few(?) events in which prediction markets did not give an &#8220-accurate&#8221- forecast for the winner. In a typical &#8220-accurate&#8221- prediction, the candidate that has the contract with the highest price ends up winning the election.

This result, combined with an increasing interest/hype about the predictive accuracy of prediction markets, generated a huge backslash. Many opponents of prediction markets pointed out the &#8220-failure&#8221- and started questioning the overall concept and the ability of prediction markets to aggregate information.

Interestingly enough, such failed predictions are absolutely necessary if we want to take the concept of prediction markets seriously. If the frontrunner in a prediction market was always the winner, then the markets would have been a seriously flawed mechanism. In such a case, an obvious trading strategy would be to buy the frontrunner&#8217-s contract and then simply wait for the market to expire to get a guaranteed, huge profit. If for example Obama was trading at 66 cents and Clinton at 33 cents (indicating that Obama is twice as likely to be the winner), and the markets were &#8220-always accurate&#8221- then it would make sense to buy Obama&#8217-s contract the day before the election and get $1 back the next day. If this was happening every time, then this would not be an efficient market. This would be a flawed, inefficient market.

In fact, I would like to argue that the late streak of successes of the markets to always pick the winner of the elections lately has been an anomaly, indicating the favorite bias that exists in these markets. The markets were more accurate than they should, according to the trading prices. If the market never fails then the prices do not reflect reality, and the favorite is actually underpriced.

The other point that has been raised in many discussions (mainly from a mainstream audience) is how we can even define probability for an one-time event like the Democratic nomination for the 2008 presidential election. What it means that Clinton has 60% probability of being the nominee and Obama has 40% probability? The common answer is that &#8220-if we repeat the event for many times, 60% of the cases Clinton will be the nominee and 40% of the cases, it will be Obama&#8221-. Even though this is an acceptable answer for someone used to work with probabilities, it makes very little sense for the &#8220-average Joe&#8221- who wants to understand how these markets work. The notion of repeating the nomination process multiple times is an absurd concept.

The discussion brings in mind the ferocious battles between Frequentists and Bayesians for the definition of probability. Bayesians could not accept that we can use a Frequentist approach for defining probabilities for events. &#8220-How can we define the probability of success for an one-time event?&#8221- The Frequentist would approach the prediction market problem by defining a space of events and would say:

After examining prediction markets for many state-level primaries, we observed that 60% of the cases the frontrunners who had a contract priced at 0.60 one day before the election, were actually the winners of the election. In 30% of the cases, the candidates who had a contract priced at 0.30 one day before the election, were actually the winners of the election, and so on.

A Bayesian would criticize such an approach, especially when the sample size of measurement is small, and would point to the need to have an initial belief function, that should be updated as information signals come from the market. Interestingly enough, the two approaches tend to be equivalent in the presence of infinite samples, which is however rarely the case.

Crossposted from my blog

NewsFutures Emile Servan-Schreiber has two lines of defense for the prediction markets.

Posted on by
3

And a slam at the InTrade fanboys:

[&#8230-] The classic first line of defense in these cases is to remind people that market “predictions” are really just probabilities [*], so any one outcome cannot invalidate the approach. The argument is sound and backed up by loads of data. But it would of course be much more convincing if we, as an industry, would remember to show at least as much humility when our market “predictions” appear correct instead. If you’re going to spread the idea that your market called all 50 states in the last U.S. presidential election because each correct outcome was predicted with over 50% chance, then you can’t hide behind probabilities when an 80% prediction comes to naught, as in Obama’s NH collapse. [&#8230-]

Excellent point, my Lord.

[*] Note that Midas Oracle is stuffed with phrases like &#8220-probabilistic predictions expressed in percentages&#8221-, and full of charts showing these probabilities.

Go reading his second point, now.

[&#8230-] capturing the consensus opinion in a much finer and dynamic way than all the amorphous media buzz [&#8230-]

&#8212-

TECHNICAL NOTE:

Because NewsFutures is a strictly hierarchical company, I assume the piece is from EJSS, even though our smart man did not sign it. Bad Karma. Anonymous texts have no weight on the Internet.

On the Internet, nobody knows you’re a dog.

Author Profile&nbsp-Editor and Publisher of Midas Oracle .ORG .NET .COM &#8212- Chris Masse&#8217-s mugshot &#8212- Contact Chris Masse &#8212- Chris Masse&#8217-s LinkedIn profile &#8212- Chris Masse&#8217-s FaceBook profile &#8212- Chris Masse&#8217-s Google profile &#8212- Sophia-Antipolis, France, E.U. Read more from this author&#8230-

Read the previous blog posts by Chris. F. Masse:

  • Good news: The BetFair blog now features a prediction market column. — Bad news: Their columnist is an anonymous writer with long hair… and dubious skills.
  • Once again, a BetFair spin doctor misunderstands the prediction market approach.
  • Grandizer
  • Tss… Tss… Surely, you are joking Doctor Giberson.
  • Comments are still open on Midas Oracle.
  • “I am much more aligned with InTrade than you are, Chris.”
  • And the award for the most technology advanced software vendor goes to… the envelope, please…. QMARKETS in Israel. … [Cheers and applauses in the crowd.]

Prediction Markets 101 – Chapter One: Interpreting The Probabilistic Predictions

Posted on by
Reply

&#8220-Thrutch&#8221-:

Probabilities, Prediction Markets, and Popular Fallacies

With Hillary&#8217-s surprise victory over Obama in the New Hampshire primary, pundits everywhere are decrying the allegedly &#8216-wrong&#8217- odds that prediction markets like Intrade were displaying prior to the announced results. (As just one example, Barry Ritholtz weighs in with his &#8216-explanation&#8217- of : &#8220-Why Opinion Markets Fail&#8220-.)

At one point the betting markets were implying over a 90% probability for Obama to win. Does this mean they were &#8216-wrong&#8217-? No it does not. It is impossible to judge whether a given probability is/was correct based on the outcome of a single event.

A 90% probability simply implies that, if you encounter a series of events each with a 90% probability, then 9 times out of 10, the favored outcome will occur- and 1 time out of 10, the unfavored outcome will occur. Those like Ritholtz who are now calling the prediction markets &#8216-wrong&#8217- are implying the following: if the probability is 90% for an outcome to occur, then that outcome should occur every time. In other words, if the odds are 90% in favor of something &#8212- it should happen 100% of the time! But this is obviously fallacious. If the outcome occurs 100% of the time, then the correct probability to assign to it would be 100% &#8212- not 90%.

To validly assess the accuracy of prediction markets, one needs to aggregate all the situations where the odds were 90%, and then calculate whether the favored outcome indeed occurred 90% of the time. (And do the same with each level of probability.) This &#8212- and only this &#8212- will tell you how accurate prediction markets tend to be.

Barry Ritholtz:

As every good prognosticator knows, if you couch your forecasts in probabilities, the innumeric will never know you were wrong. It&#8217-s a cheap trick for the easily fooled.

Imagine if instead of a &#8220-THE END IS NEAR&#8221- sign, every loon carried a sign that proclaimed:

THERE IS A 57% CHANCE THAT THE END IS NEAR!!!

The fact that this didn&#8217-t happen &#8212- and the 43% probability did &#8212- doesn&#8217-t mean this forecast was accurate. It merely meant that the person had proferred two possibilities and one of those two occurred. But the math remains unverified.

Neat trick: By your definition, PREDICTION MARKETS CAN NEVER BE WRONG, so long as they maintain a 1% possibility of the alternative outcome.

That&#8217-s hardly a satisfying defense&#8230-

Author Profile&nbsp-Editor and Publisher of Midas Oracle .ORG .NET .COM &#8212- Chris Masse&#8217-s mugshot &#8212- Contact Chris Masse &#8212- Chris Masse&#8217-s LinkedIn profile &#8212- Chris Masse&#8217-s FaceBook profile &#8212- Chris Masse&#8217-s Google profile &#8212- Sophia-Antipolis, France, E.U. Read more from this author&#8230-

Read the previous blog posts by Chris. F. Masse:

  • Good news: The BetFair blog now features a prediction market column. — Bad news: Their columnist is an anonymous writer with long hair… and dubious skills.
  • Once again, a BetFair spin doctor misunderstands the prediction market approach.
  • Grandizer
  • Tss… Tss… Surely, you are joking Doctor Giberson.
  • Comments are still open on Midas Oracle.
  • “I am much more aligned with InTrade than you are, Chris.”
  • And the award for the most technology advanced software vendor goes to… the envelope, please…. QMARKETS in Israel. … [Cheers and applauses in the crowd.]