Hypermind accuracy over its first 18 months

Hypermind was launched in May 2014. The chart below plots the accuracy of its predictions over the 151 questions and 389 outcomes that have expired at of this writing. All the predictions so far have been about politics, geopolitics, macroeconomics, business issues, and some current events. No sports.

To generate this chart, we proceeded as follows. The data was collected daily: every day at Noon we recorded the latest transaction price on each traded outcome and treated it as a probability for this outcome. These observations were then grouped in 20 probability bins: 1-5%, 6-10%, 11-15%, …, 96-99%. Then, we just plotted the average of the probabilities in each bin against the percentage of the outcomes represented in the bin that actually occurred.

The market is accurate to the extent that the two numbers are well calibrated, ie., that the data points are aligned with the chart’s diagonal. In our case the measure of calibration is .001, meaning that the average difference between the percentage of events actually coming true and the forecast at each level of probability is only about 3.3%.  If we did not know better, we might conclude that reality aligns itself with Hypermind’s predictions.

calibration 151x5 171215

 

2 Comments

  1. @EJSS: It’s a remarkably good calibration graph. But correlation between predicted and realized odds is, I think, a scientifically meaningless measure in this context; after all, perfect correlation could be achieved with severe miscalibration, for example if (reality – 50%) = 0.5 * (prediction – 50%)! Also the large dots at the extremes must reflect either lots of very easy questions, or else very many trades taking place when an outcome is all but certain (right before resolution). I think it is the latter — which suggests a healthy market with a mixture of participants who prefer instant liquidity to find other opportunities to bet on, and other participants who like to capture the “sure thing” money even for a small return on investment. But regardless, it distorts the correlation coefficient.

    Have you tried computing calibration *surfaces* with an additional axis for days-before-resolution?

    Like

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s