Misinterpreting results from prediction markets

There has been a lot of hand wringing the past couple days about how the prediction markets "got it wrong" when it came to the Democratic primary in New Hampshire. Since prediction markets are all about probabilities and not absolutes, using "right" vs. "wrong" in any dialogue and dismissing results as "BS" shows a lack of understanding about how to interpret results.

For example, in the CNN marketplace, Hillary Clinton's closing position in the New Hampshire primary market was $16.40. Barack Obama's closing price was $82.60. The easiest way to interpret these results is to say:

"If we flipped a coin 100 times, Hillary would win 16 times. Barack would win 83 times." So yes, traders were saying there was a high probability Obama would win, but it was never an absolute. Thinking about prediction market results in the context of coin flips is a useful way of staying grounded when interpreting what the market is saying.

If Obama had won we would never be having this dialogue. That said, it's probably useful to come to the forefront now as the awareness of prediction markets widens and more and more businesses consider using prediction markets internally and externally. Unfortunately in the media, talking about chance and probability is a more nuanced discussion than "right" vs. "wrong" which means the discussion is not likely to get the treatment it deserves. But as we tell our existing and potential clients who are considering Inkling for business purposes, the most important role of prediction markets is to give insights in to what people are thinking. Their perceived "accuracy" or ability to predict events occurring in the future is often a tangible benefit proven countless times, but should never be over-promised as a magic crystal ball.

If you do want to judge an Inkling marketplace on how well it's "performing," you can examine all the cases where the marketplace said there was an 80-90% probability of an event occurring, then look at how many times it actually did occur. If the event did occur a vast majority of the time, the marketplace is performing well. Similarly, if traders say there is only a 10-20% chance an event will occur, and indeed those answers are occurring infrequently, the marketplace should be considered successful.

We're in the process of compiling some data and will have a subsequent post shortly looking at these exact figures across all Inkling marketplaces.

As for the New Hampshire primaries, this is the best article we've seen so far about the role of prediction markets:
http://www.alleyinsider.com/2008/01/prediction-markets-blow-it-nh-primaries-confound-expectations.html

Also, the last time this happened on a much smaller scale was the Democratic takeover of Congress in 2006. Bo Cowgill and Cass Sunstein wrote an excellent article in the New Republic addressing certain people's misinterpretation about the market results (TNR took the article down but it's available here):
http://groups.google.com/group/Prediction-Markets/msg/3514dbddf76636f6

CNN Political Market Launches

Yesterday afternoon around 4pm ET, CNN launched the CNN Political Market on the CNN homepage and on the CNN Politics page for the upcoming 2008 U.S. elections. As a company we're certainly excited to work on this project with CNN and as political junkies we're pleased to contribute in a small way to the dialogue this election season.

New York Times Article on Insights at Google From Their Prediction Markets

Our friend at Google, Bo Cowgill, is making headlines with a paper he authored with Justin Wolfers at Penn and Eric W. Zitzewitz at Dartmouth. The paper, instead of focusing on the accuracy of the markets run at Google, discusses what factors influenced trading patterns over time. Perhaps one of the most interesting findings was about an optimism bias in traders:

"The most interesting of these was an optimism bias, which was more pronounced for subjects under the control of Google employees, such as would a project be completed on time or would a particular office be opened. Optimism was more present in the trading of newly hired employees, and was significantly more pronounced on and immediately following the days with Google stock price appreciation. Our optimism results are interesting given the role that optimism is often thought to play in motivation and the success of entrepreneurial firms..."

This mirrors what most of us experience when working at any large organization. Google hires lots of young, bright people out of college, and they surely come in wide-eyed and excited. Even if you're coming in with some experience, walking on to that campus you quickly realize: this is unlike any company...or place...you've ever visited, and I'm not just talking about the free food. It would seem to make sense those people would exhibit a high level of optimism expressed in their trading behavior. And despite the undeniable buzz of the Google campus, I don't think this is true just about Google. There is a certain aura to any large company. They got large for a reason - by being incredibly successful - and one can't help being a little in awe, full of pride, and optimistic when reporting for work.

Then life happens for awhile and you quickly see the contrast between the newbies and grizzly veterans who have had projects cancelled, seen delays, experienced a stab or two in the back, suffered through needless red tape, and lived the trials and tribulations of projects and the politics of large organizations.

The paper's findings seem to not only support the need for a diverse trading pool, but perhaps for more outside participation in a second, external marketplace. A group of trusted people who know you from the outside vs. the inside. One of our clients, for example, a financial services firm, invited all their vendors to trade in markets about the company. Who better to ask about yourself than the people who have to sell to you. They know where you're efficient, they know where it's hard to work with you. They get paid to know your business, but they look at you in a different light and would have no qualms expressing what they think of you (especially if they can be assured anonymity.) Capturing their perspective would likely be very valuable and an interesting contrast to data captured internally.

Bo assured us his paper would be a great read and it is. View it here: http://bocowgill.com/GooglePredictionMarketPaper.pdf

Enterprise 2.0 Predictions for 2008

You see a lot of prediction pieces by journalists at the beginning of a new year. This is one of the better ones I've read (not only because it mentions us :) but it's more realistic about the uptake of certain technologies and isn't too rah rah about anything. Having spent a lot of time on IT initiatives in large companies, bringing about wholesale change or introducing something on an enterprise level, especially when it's not perceived as "mission critical" is a huge chore and several of the points the author makes are more realistic than what you'll hear from other corners of the web.

Specifically about prediction markets, his comments mirror what we have been seeing as well: an incredible amount of interest (marketplaces in our pilot program are in the hundreds now) and more refined thinking on how to use prediction markets to affect real change on business processes. We are also in several conversations with companies about rolling out a marketplace across the entire enterprise -- a level of proliferation not discussed nearly as much the first half of last year.

Here's the blog post:
http://blogs.zdnet.com/Hinchcliffe/?p=157