Superforecasting: The Art and Science of Prediction

In this age of big data, one might assume that superforecasters have outstanding numerical abilities and can parse complex calculations to make more accurate predictions than the general population. Tetlock admits that he is yet to encounter a superforecaster who does not have an above-average mathematical ability and the capacity to apply the right calculations to a particular problem. While these traits allow superforecasters to use certain math predictively, the authors argue that the most important superforecasting skill is thinking carefully and applying nuanced judgments.

Another trait of superforecasters is being comfortable with the fact that others team members will come up with different numerical estimates for the same problem. The authors cite the example of Leon Panetta, secretary of defense to President Barack Obama, who was delighted that his intelligence officers differed on whether a new suspect in Pakistan was indeed Osama bin Laden. Panetta thought that certainty, even in this context where a yes/no answer would have been preferable, was impossible and not something to fixate on.

When Obama’s intelligence officers presented their estimates, most were 80% sure that the captive was bin Laden, but others’ confidence was as low as 30%. This caused Obama to think that the chance of the captive being identified correctly was “fifty-fifty,” whereas the CIA’s median estimate was 70% (135). The authors speculate that Obama was troubled by the report’s uncertainty and that he may have even reverted to the state of ignorance prior, where he would ignore the report’s information completely. The authors argue that probability is “deeply counterintuitive” in nature, as a forecast for 70% rain also means a forecast of 30% for no rain (139). Thus, if it does not rain, the forecast is still correct. People are only properly able to grasp both possibilities when the split is closer to 50/50.

Twentieth-century science suggests that uncertainty is an inevitable part of reality and that everything is a “maybe” with different numerical levels of probability. Superforecasters are also probabilistic thinkers, who distinguish between epistemic uncertainty (what is unknown but calculable) and aleatory uncertainty (what can never be known). Superforecasters are granular in their estimation of whether something will happen, appreciating that there is a difference in gradations as subtle as between 3% or 4%. Barbara Mellers’s work has shown that granularity is a key predictor of accuracy; thus, forecasters who use percentage gradations of 5 rather than 10 are more likely to be accurate.

When unusual events occur—such as the Big Bang, the World Trade Center attacks of 9/11, or meeting the love of one’s life in a seemingly random circumstance—humans tend to apply mystical thoughts of divine order or platitudes about things being “meant to be.” By contrast, probabilistic thinkers focus less on the why of such events and more on the how. They see even rare events as the consequence of one of myriad paths that an occurrence could have taken. The authors assert that embracing probabilistic thinking allows for more accurate forecasts.

Another trait of superforecasters is that they are interested in incoming news and willing to update their forecasts in light of changing information as many times as necessary, demonstrating a belief that “a forecast that is updated to reflect the latest available information is likely to be closer to the truth than a forecast that isn’t so informed” (153). Although superforecasters update their prognoses far more frequently than average forecasters, skeptics might suspect that superforecasters excel at predictions only because they constantly absorb news. However, Tetlock and Gardner emphasize that superforecasters’ original forecasts were already 50% better than those of their peers, meaning that not all of superforecasters’ success can be attributed to updating. The authors also clarify that updating one’s forecast to accommodate news is a demanding skill in itself.

The key challenge with updating is to update to the right degree. Forecasters must consider all the factors in the incoming news to weigh development. For example, when Bill Flack learned that the Swiss team analyzing the levels of polonium in Yasser Arafat’s body would be late delivering its results, he considered that the delay might be due to something unrelated to the polonium, such as a technician having skipped a day’s work to nurse a hangover. However, Bill had researched polonium and knew that the substance could make a delayed appearance in a corpse. Thus, learning of the examining team’s delay, Bill changed his forecast from 60% certainty that Arafat had polonium poisoning to 65%. When the Swiss team did find polonium in Arafat’s body, Bill’s Brier score was a modest-seeming 0.36, but he still outperformed the predictions of both the Swiss team and IARPA experts.

Though Bill fared well, updating insufficiently or too much in light of incoming news can derail even the best superforecasters. Often, forecasters stumble into bias after becoming attached to a given forecast, failing to update when the news contradicts their favored narrative. On the other hand, forecasters can become distracted by irrelevant information and over-adjust their forecasts. They thus fall prey to what psychologists call the “dilution effect.”

A key technique for overcoming either bias is to update frequently in response to news, but with subtle gradations. This is the practice of Vancouver native Tim Minto, who stands out among the superforecasters for his frequent updates, which are sometimes as many as 50 before a question closes. However, his average update is only 3.5% at a time. This is crucial, as a few small updates would have risked underreaction, while many bold updates would have made him vulnerable to the opposite affliction. While this is the general rule, there are no fail-safe rules in forecasting—superforecasters know that when they have strayed wildly off the mark and over- or underreacted to incoming news, they are better off returning to their initial guess and recalculating from there.