Judging a prediction market's accuracy

Recently we wrote a blog post about the accuracy of prediction markets. As we promised, we follow that up now with more specific data.

Two years ago, Google wrote a blog post about how their internal prediction markets were working. It was an inspiring picture and one that got many people excited about using prediction markets. Now that we've been hosting prediction markets for 2.5 years, we have quite a bit of our own data. Looking at well over 2 million trades and thousands of markets across hundreds of marketplaces, we can easily say Google's impressive results weren't a fluke.

The most popular type of market our market makers create determines the probability of an event happening: Will David Cook win American Idol?, Will Miller Chill still be sold in 6 months, Will a new aircraft design be delivered on time?

To beat the proverbial departed equine, you can't just look at a single outcome of one prediction market question to determine how "accurate" your marketplace is.

When Mrs. Burnette told us in 4th grade that flipping a coin had .50 probability of coming up heads, she didn't stop there. She made us measure it to prove that this was in fact true by measuring the relative frequency of heads coming up. For homework we had to flip the coin over and over and write down the outcome. Flipping a nickel at home I just got: heads tails heads heads tails heads tails tails tails tails heads heads tails. Using our very new skills with fractions we could then measure the probability by computing 6 heads/13 trials = 0.46, pretty close to 0.5.

Mrs. Burnette appeared to know what she was talking about.

So just like flipping a coin, if Inkling told you something has a 15% probability of coming true, you can't just look at one outcome (i.e. one coin flip). You need to look at multiple scenarios where Inkling said something would happen 15% of the time. If those things actually come true, 15% of the time, Inkling is doing well at this.

We plotted a graph a lot like Google showed 2 years ago. Count the number of markets that predicted an event would occur 5% of the time, and see how many of those occurred: almost 5% of them. Count the number of markets that predicted an event would occur 15% of the time, and sure enough 15% of them ended up occurring. And so on. Until we got the graph below.



The green line is what we’d look like if we were perfect: things predicted to happen 15% of the time happen 15% of the time, things predicted to happen 65% of the time happen 65% of the time, etc. Inkling is the black line hugging pretty close to perfect.

Another type of prediction market is one that predicts the numerical outcome of something: What will the population of New York City be in 2010, How many utility patent applications will be filed in the US in 2014?

In this case we’d like to see a plot of what is the value we predicted with what actually happened. We plotted hundred of these markets in the graph below.



The green line is perfect again. We’d be perfect if Inkling said you’d sell 100 units of something, and you sold 100 units. If Inkling said you’d sell 1000 units, you sold 1000 units. The red line is a line of best fit through the data. Not too shabby.

We've discussed several times on this blog and elsewhere the accuracy of a marketplace. There are significant misconceptions about what the results of a prediction market actually mean, especially in the media. Hopefully these graphs reinforce what a prediction market is revealing as this is the first step in using the new information as input to strategic decision making, etc.

How to Kill Bad Projects - Harvard Business Review

A few weeks ago I had the opportunity to sit down in Boston with the Harvard Business Review and discuss how our corporate clients were using prediction markets internally. Lew McCreary, one of their Senior Editors wrote a brief blog post highlighting some of our discussion and touching on a sensitive but valuable use of prediction markets: institutional lie detection.

Harvard Business Review: How to Kill Bad Projects

We've long talked about fighting office politics and bringing transparency to information flow as part of the value proposition of prediction markets. Trying to bring reality or at a minimum understanding the current perception of forecasts and probabilities of risk, etc. ultimately helps with strategic decision making, being more proactive in addressing issues, etc.

The unspoken issue, however, and something we discuss regularly with potential clients is how to deal with the people part of the equation when someone may a) get some egg on their face based on the information being exposed, b) not agree with what the market is saying, or c) try to actively get people to trade a certain way to mask what may really be going on.

When this is raised, we usually respond back with a question of our own: Is it worse for someone to look bad in the short term or to have an entire initiative fail based on unrealistic deadlines and forecasts or unawareness of risks? In other words, does a manager want to know about a problem early, take their lumps, and work to rectify the situation (and be seen as an effective manager who can handle adversity) or do they want to mask problems until it's far too late and risk the possibility of getting fired? Personally, I'd take the first option.

As for the issue of collusion, the key lies in diversity of participants in the markets from different organizational units. Power centers in companies are typically localized to your own kind (unless you're truly one of the big dogs in the company.) A senior person in sales can probably get her minions to go along with her on something but the marketing, product development, and customer support folks who don't answer to this person are likely going to be independent actors who will blunt any concerted effort to manipulate the market. We see this all the time in our public marketplace in questions like: "Who will win Big 10 Player of the Year?" The market may get posted to certain team's discussion boards and you get a predominant number of people from a University taking long positions in their guy. But then other people start to come in who are more rationale players or who are biased towards other players and the market begins to normalize.

Of course a company could always run the market: "Who will try and get others to collude with them to manipulate the market?" and police themselves but this has about a 0% probability of going well, I imagine. :)