Hypermind was launched in May 2014. The chart below plots the accuracy of its predictions over the 151 questions and 389 outcomes that have expired at of this writing. All the predictions so far have been about politics, geopolitics, macroeconomics, business issues, and some current events. No sports.

To generate this chart, we proceeded as follows. The data was collected daily: every day at Noon we recorded the latest transaction price on each traded outcome and treated it as a probability for this outcome. These observations were then grouped in 20 probability bins: 1-5%, 6-10%, 11-15%, …, 96-99%. Then, we just plotted the average of the probabilities in each bin against the percentage of the outcomes represented in the bin that actually occurred.

The market is accurate to the extent that the two numbers are well calibrated, ie., that the data points are aligned with the chart’s diagonal. In our case the measure of calibration is .001, meaning that the average difference between the percentage of events actually coming true and the forecast at each level of probability is only about 3.3%. If we did not know better, we might conclude that reality aligns itself with Hypermind’s predictions.

### Like this:

Like Loading...

*Related*

@EJSS: It’s a remarkably good calibration graph. But correlation between predicted and realized odds is, I think, a scientifically meaningless measure in this context; after all, perfect correlation could be achieved with severe miscalibration, for example if (reality – 50%) = 0.5 * (prediction – 50%)! Also the large dots at the extremes must reflect either lots of very easy questions, or else very many trades taking place when an outcome is all but certain (right before resolution). I think it is the latter — which suggests a healthy market with a mixture of participants who prefer instant liquidity to find other opportunities to bet on, and other participants who like to capture the “sure thing” money even for a small return on investment. But regardless, it distorts the correlation coefficient.

Have you tried computing calibration *surfaces* with an additional axis for days-before-resolution?

LikeLike

[…] in fact, it isn’t unusual at all. The record shows that market predictions in general – and Hypermind’s in particular – are well calibrated, which means that the estimated probabilities finely predict the proportion […]

LikeLike