How Hypermind predicts the future

The Hypermind prediction market is a forecasting contest on geopolitical, business and economic issues. It is designed to consolidate the informed guesses of thousands of brains worldwide into accurate probabilities. For instance, on the eve of 2020, the market judges that Donald Trump is 51% likely to be reelected president of the USA next November, compared to 40% for another man to replace him, or 9% for a woman. (All our predictions are published in real time here.)

USPREZ2020WHO-EN

As this example shows, the prediction market is not a binary oracle that states with certainty what will or won’t happen. Its pronouncements are more subtle. It doesn’t say that the next president of the USA won’t be a woman, it says that there is currently 91% probability that it will be a man.

One naturally wonders whether these probabilities are reliable or pulled out of thin air. If Trump or another man is elected in november, will it mean that the market was overestimating womankind’s chances to seize the White House? Or if a woman does win, will it mean that the market was severely underestimating her chances? To answer these concerns, one would have to rerun the election a hundred times: if a woman won in about 9 runs out of a hundred, then the 9% probability was correctly estimated. But if a woman won much less often, say only 3 times, or too often, say 18 times, then the 9% estimate was, respectively, 3 times too large, or too small by half. Wrong in any case.

Unfortunately, elections or other historical events cannot be rerun, not even once. Does that mean it is impossible to assess the accuracy of probability forecasts? No, there is another way, less ideal but satisfactory nonetheless: consider allthe events that the market has forecasted with the same probability, say 10%, and count how many have actually happened. If about 10% for them did occur, then the 10% probability forecast was correct. Otherwise it wasn’t. Repeating this procedure over all probability levels, from 1% to 99%, gives a complete picture of the forecasting accuracy of the prediction market.

To perform such analysis, lots of data are required. Hypermind has them. Over 5 years, from the market’s launch in the spring of 2014 until the spring of 2019 (when this analysis was performed), it has generated 537,640 probability forecasts about 1,185 possible answers to 400 forecasting questions in the geopolitical, business, and economic domains. In the graph below, each data point answers the question: « What is the proportion of events forecasted with probability pthat actually occurred? »Amazingly, reality seems to literally align itself to the market’s predictions.

calibration-en

This alignment shows the amazing ability of the market’s collective intelligence to discern, not the future itself, but its underlying probabilities. From this we may draw some conclusions:

  1. The future is not determined by the world’s present state. It is fundamentally probabilistic. It is always wrong to think, after the fact, that whatever happened hadto happen. There always was just a probability that it could happen, and another that it might not. It always could have turned out otherwise if the great dice roll in the sky had landed differently.
  2. Collective intelligence, as expressed in a prediction market, is able to accurately discern the probability field that underlies reality.
  3. The forecasts of the Hypermind prediction market are particularly reliable.

To explore these ideas further, in practice:

Graceful degradation in election forecasting

The Washington Post recently published a review of the relative performance of various forecasting methods in the recent US presidential election. Amusingly titled Which election forecast was the most accurate, or rather: The “least wrong”? , the article nicely complements our own post mortem published earlier. Like us it finds that:

  1. Hypermind was one the “least wrong” forecasters of the lot;
  2. Crowd-based methods fared better than statistical poll aggregators.

One take-away is that when all systems failed, human collective foresight failed less than the alternatives. You might call it “graceful degradation”.

Skeptics of crowd wisdom gleefully seize on 2016’s Brexit and Trump forecasting armageddons to argue that our kind can’t predict the future and that it is a hopeless quest at best, a cynical con at worst. That criticism entirely misses the point. Prediction markets have never claimed magical powers to predict everything all the time. That’s just impossible, and the world is better for it. However, the record shows that prediction markets tend to perform better, or fail less, than the alternatives. In that, they help push back the inherent limits of forecasting. That’s all, but it’s remarkable nonetheless.

Why you need collective intelligence in the age of big data

(c) Philippe Andrieu - click to visit the artist's website

There’s an old joke about a someone who has lost his car keys and keeps looking for them under a street light, but with no success. After a while, a policeman finally asks why he doesn’t extend his search elsewhere. “Because that’s where the light is,” answers the man.

The current obsession with Big Data is somewhat reminiscent of this so-called “street light effect” – the tendency to look for answers where they are easiest to look for, not most likely to be found.

In fact, whether or not a big-data search party is likely to discover something useful really depends on the kinds of data that are at hand. Computers are really good at processing data that are well structured: digital, clean, explicit and unambiguous. But when the data are unstructured – analog, noisy, implicit or ambiguous – human brains are better at making sense of them.

Whereas a single human brain, or a modest personal computer, may deal with small data sets of the preferred kind, the “bigger” the data is, the more computing power has to be brought to bear. In the case of structured data, bigger computers will come in handy, but in the case of unstructured data – the kind computers can’t properly deal with – there’s also a hard limit on how much computing power a single human brain can deliver. So the best way to make sense of big unstructured data sets is to tap into the collective intelligence of a multitude of brains.

Big Data vs Collective Intelligence

The best kind of computing power to bring to bear on big data depends on the kind of data that has to be processed. Collective intelligence delivers the best performance when dealing with big unstructured data sets.

When the goal is to peer into the future, statistical big-data approaches are especially brittle, because the data at hand are necessarily rooted in the past. That’s ok when what you are trying to forecast is extremely similar to what has already happened – like a mature product line in a stable market – but it breaks down disgracefully when you are dealing with brand new products or disrupted markets.

Here are just a few examples of situations we have encountered where collective forecasting proves superior to data-driven projections:

Disrupted market: When in the mid-2000 the world-wide demand for dairy products suddenly increased three-fold in the space of a few months, after a decade of stability, dairy product producers could not rely any more on their data-driven forecasting models. Instead, they tapped into the collective forecasting insights of their people on the ground, closest to the customers, to better understand and model the new demand drivers.

New products: A few years ago Lumenogic collaborated with a team of marketing researchers to run a prediction market within a Fortune 100 consumer packaged firm, focusing on new products. When compared to the forecasts issued from the classic data-driven methods, the researchers found that the collective forecasts provide superior results in 67% of the cases, reduce average error by approximately 15 percentage points, and reduce the error range by over 40%.

Political elections: In the past 20 years, prediction markets have become famous for their ability to outperform polls as a means to forecast electoral outcomes.  So much so that a skewer of distinguished economists eventually petitioned the U.S. government to legalize political betting for the benefit of society – which it did recently, to some extent, as evidenced by the recent launch of PredictIt. The big-data camp fought back in the form of poll aggregators, as popularized by statistical wizard Nate Silver, and further enriched by other non-poll data sets such as campaign contributions, ad spend, etc. To no avail. In last november’s U.S. Midterm elections, the collective intelligence of Hypermind’s few hundred (elite) traders outperformed all the big data-driven statistical prediction models put forth by major media organizations. That’s because the wisdom of crowds is able to aggregate a lot of information – unstructured data – about what makes each election unique, whereas this data lies out of the reach of statistical algorithm, however sophisticated.

Despite the current and growing flood digital data – the kind computers and algorithms can deal with – we should not lose sight that the world offers magnitudes more unstructured data – the kind only human brains can collectively make sense of.  So if you ever find yourself searching fruitlessly under that big-data street light, remember that collective intelligence may provide just the night goggles you need to extend your search.