Thursday, November 3, 2022

Think Tankers Stink at Predictions

Mr. Damien Ma, the co-founder and managing director of the Paulson Institute's in-house think tank MacroPolo, has an anecdote on how bad most think tankers really are at predicting geopolitical events.

Here's what he wrote in a recent piece for the new publication Semafor:

Don’t make predictions about Chinese politics. That’s the lesson from our think-tank’s month-long “fantasy football”-style competition to forecast the “Chinese election” before it concluded last weekend.

Of the more than 1,000 players who played — China specialists and casual observers — not a single person correctly predicted all seven members of the Chinese Communist Party’s new Politburo Standing Committee, the peak of political power in China.

So it turns out the political scientist Phil Tetlock’s longstanding insight holds: On average, expert predictions don’t outperform non-experts’ results.

But maybe it’s not just that expertise didn’t matter, and instead that expertise was neutralized by the paradigm shift that has taken place in China. Basically, expert or not, it was nearly impossible to win our game.

This wasn’t a normal Communist Party congress. The presumed norms that guided and bounded Chinese elite politics fell away. We put too much stock in assumptions about the Party’s commitment to a de facto retirement age, for example, when what counted most was proximity to Xi Jinping himself. Above all else, loyalty and trust were the determinative factors.


An example of a prediction that didn't age well: In 1996, Harry Rowen, who worked at the Hoover Institution, said China will become a democracy in 2015.

Here is one to watch for the future: China expert Oriana Skylar Mastro, a Center Fellow at Stanford University's Freeman Spogli Institute for International Studies and a Non-Resident Senior Fellow at the American Enterprise Institute (AEI), is predicting a 100% chance China will use force against Taiwan in the next five years.

Here is a 2020 RAND Corporation post entitled "How Accurate Were Predictions About the Future?"  Among other things, it discusses RAND's so-called "Delphi Method" which attempts to make effective use of informed intuitive judgement in long-range forecasting.