Defining success of a pilot

A couple weeks ago we ran a one-day workshop for a new client of ours that is about to begin a 6-month pilot for 250 employees.

A common exercise to conduct before embarking on any new project is deciding what the success factors will be. How will we know at the end of the project if we've been successful? What will our metrics be?

When it comes to prediction markets, one obvious success factor is how "accurate" the markets have been. And by accuracy we of course mean the prices of stocks matching the probability of events occurring over time (if markets are regularly saying a particular answer is going to happen 80% of the time, it should actually happen 80% of the time!)

But there are other factors we discussed that should also be used to judge the success of pilots. Among them:

• Good insights from trading data e.g. “executives tend to be consistently overly optimistic in their trading behavior” or "we discovered multiple traders behaving as outliers in a heavily traded market about our quarterly forecast that led us to reconsider our models."
  
• Market results are viewed as valuable input to decision making. In other words, questions have been asked where understanding the probabilities of the answers occurring have proven to be valuable in setting direction or have provided an ROI.
  
• People are actively creating new markets or requesting new markets be run, or even complain when markets are not being cashed out right away. All these behaviors show passion and investment in the marketplace.
  
• People citing the prediction market in the daily discourse of business during status meetings, in the cafeteria, and in company newsletters