Tuesday, February 26, 2008

Benefits of Prediction Markets in IT Project Management

Herbert Remidez from the University of Arkansas recently sent us the final draft of a paper he wrote about the results of some experiments he ran at Acxiom using Inkling Markets. From the abstract of the paper:

Developing obtainable, clear and measurable work expectations early in the project planning process is an important part of successful project management. Converting these expectations into project milestones and communicating openly about progress toward them is crucial to every project’s success. Optimistic estimation biases of IT workers, poor estimating techniques and group politics can hinder communication and decrease the chances of success. A prediction market is a tool that might help project managers overcome these obstacles.

And from the results:

  • Participation in the market was very high (87%) with the average trader making 23.28 trades over the six weeks that the market was open.
  • The market correctly predicted twenty four of the twenty six project milestones (92%).
  • The introduction of the prediction market led to the project manager to clarify and revise an important project milestone before the project started.
  • A survey at the end of the project suggested that the software was easy to learn and promoted discussion and cohesion in the group.
  • The experience was positive for the group and management is exploring the potential use of prediction markets in other business units.
  • The implementation of a prediction market in a setting where market participants had control over the outcome of the contracts contributed to concerns about insider trading.

You can see the entire paper on Scribd:

Using Prediction Markets to Support IT Project Management

Monday, February 25, 2008

O'Reilly Collective Intelligence foo camp

Nate and I just returned from a week in the Bay Area for meetings and had the opportunity to spend the final two days of our trip attending an O'Reilly foo camp. Typically these are held at O'Reilly headquarters north of San Francisco but this time Google offered to sponsor the event so we spent two days shuttling between 6 conference rooms at a Google satellite office.

There have been several foo camp predecessors but this one was focused specifically on collective intelligence. The format of the "camp" is intended to be the "anti-conference" where the first half hour we collaboratively create the 2-day schedule by filling in time slots with sessions we want to lead. I thought this was going to get a little out-of hand even though attendees were invite only and limited to about 50-60 people but it actually turned out pretty well with at least one session each hour that was of interest and people with similar interest/backgrounds putting on sessions together.

Robin Hanson and I signed up to do a session on corporate prediction markets early in the camp which was fairly well attended although I thought we were driven to spend too much time talking about the potential for market manipulation and not enough brainstorming new ideas. Also, while everyone had a general understanding about prediction markets, there was still very little understanding of how they are actually being used so we spent most of the time talking through use cases.

In terms of overall takeaways from the camp, although not directly discussed, there was certainly re-enforcement of the old garbage-in, garbage-out syndrome; that even if you get a "collective" together, if they aren't incented appropriately (and I don't mean prizes) and they don't have relevant knowledge, just because you've put a group together, you're going to get garbage out. This was most evident when we talked about failed collaborative projects involving wikis, but certainly applies to corporate prediction markets as well in terms of the questions you ask, the people you involve, and the information available to the traders to act upon in the market.

Also after attending a data visualization talk it made me think later how we have just been scratching the surface in terms of gaining insight from trading data, market performance, etc. The recently published Google paper delved deeper in to this area - marrying trading data with demographic data but there is an opportunity to go much much further. We have some big things to get done in the next month but I suspect our focus may turn to this area sooner rather than later.

Finally it was noted by one attendee during an end of day recap that in every single session, the topic of incentive and reputation came up. This is clearly an area those working on collective intelligence driven applications must crack in order to be successful. We've done lots of thinking on this too, but we'll save it for another post.

All in all it was a great way to spend a couple days. Even better than the sessions themselves were the conversations we had in the hallways and during breaks and meals. A few people we already knew, but many whose books we've read or blogs we follow we got to meet in person. And of course this is the primary gravitational pull of Tim O'Reilly. His company writes lots of good books, but there probably isn't a bigger node in the technology space to bring together, as he said in his opening remarks, "the people who should know each other."

And on an unrelated note, if you're ever in Mountain View and you need to find a place to work at night, try the Red Rock Cafe. We worked in to the evening multiple nights there and it's fairly work friendly. Big upstairs area with plenty of desks and outlets. Our only complaint: they turned up some pretty bad music way too loud. :)

Wednesday, February 20, 2008

Wolf Blitzer

Admittedly, there is nothing quite like seeing Wolf Blitzer pitch the CNN political prediction market run by software you've created. "Hey, those are our graphs floating across the screen!"


Wednesday, February 13, 2008

Be Lonely in Inkling No Longer

On Monday night we slipped a new link in to the top navigation bar of all marketplaces called "friends." This means that whenever you see a username, you can click on it, then add the person as a "friend." A list will be created on the friends page providing you easy bookmarks to their profiles, their current rank in the marketplace, and their total worth. And of course this doesn't just have to be friendly, it can be competitive too. Have a few office mates or buddies in the offline world you want to compete against? Add them as friends and create your own leader board.

We've also enhanced people's profile pages to include information about when they joined the marketplace, how many trades they've made, how many markets they've created, and their worth. Their latest discussion items in any market are automatically listed along with the markets they are running. And for marketplace administrators, we've added the ability to customize the profiles with up to 10 questions you can ask every trader. For example, ask people what their business unit is, or ask them what book they're currently reading.

This is version 0.1 of friends and profiles in Inkling. Rest assured we have lots of additional ideas we're already hard at work on, but if you have other suggestions of what we should do, we'd love to hear them.

Friday, February 08, 2008

O'Reilly Money:Tech Conference

Earlier this week I had the opportunity to spend some time in New York City at the O'Reilly Money:Tech conference held at the Waldorf Astoria. This was the first time O'Reilly had ever put on a conference like this and I believe most people found it very worth their while.

I participated in a panel about prediction markets along with Justin Wolfers, a Wharton Professor and guest editorialist for the WSJ this election season, Alex Forshaw, a prolific trader on InTrade (and college senior surprisingly,) Jason Jones, a hedgefund manager, and Craig Kaplan from PredictWallStreet. Questions from the audience were fairly typical of the conventional wisdom critiques of prediction markets which the panel dealt with handily. It was especially good to finally meet Justin Wolfers since he has become somewhat of an unofficial spokesman for prediction markets in the mainstream media. We talked about working together on some things in the future which could be quite interesting.

Bo Cowgill from Google also gave a talk on his recently published paper (with Wolfers and Zitzowitz) and the more he is invited to give this talk, the better. There is a real lack of communication right now about the organizational impacts prediction markets have beyond more accurate forecasting and Bo's work is thankfully filling that gap.

Inkling also ran a marketplace in conjunction with the conference, asking a wide range of topics both financial and entertainment driven, which people seemed to enjoy.

And of course the benefit of speaking at a conference is getting to attend other sessions as well. Besides a few of the data analytics talks, I was most interested in the sessions on collaborative stock picking given our side-hobby efforts a year ago with a now defunct application called Worthio. It was liberating to see that similar applications had picked up traction, such as Motley Fool CAPS, Stockpickr, and PredictWallStreet. Another highlight of the conference was Tim O'Reilly's web 2.0 talk. I had heard it before but with a spin towards financial markets it gave some additional food for thought. Finally Jim Cramer starting the conference off in a fireside chat was a treat. About half way through he went in to full-on riff mode about a wide range of companies, even calling one of the platinum sponsors of the conference, Dow Jones (after they surely paid thousands of dollars for the honor,) "worthless." It was classic Cramer.

Thursday, February 07, 2008

ABC7 in San Francisco talks about their experience running a marketplace

It's been close to a year since ABC7-KGO in San Francisco launched their prediction marketplace. They recently did a story recapping their experiences here.

It's not deep on details but I like in the video how they represent the probabilities of things occurring by adding "chance" to the numbers. It makes it clearer what results coming out of the markets really mean vs. a black & white right/wrong.

Also, I didn't realize until recently that Eric Zitzewitz, the Dartmouth/Stanford Professor and prediction market guru was their top trader at over $1M. Nice job, Eric.