90% Prediction Accuracy

Brian Wang’s public prediction track record at Metaculus is a Brier Score of 0.079 which is about 90% correct predictions. Brier score is a distance from 100% accuracy. Brier Score evaluates the accuracy of probabilistic predictions.

If you predict that the event A will occur with a probability of 100%, and the event occurs then the Brier score is 0.

If you predict that the event A will occur with a probability of 72%, and the event occurs then the Brier score is 0.078.

If you predict that the event A will occur with a probability of 50%, and the event occurs then the Brier score is 0.25.

Averaging 0.079 means that 18% of the 90% correct predictions were offset by of the wrong predictions.

Superforecasters

Phil Tetlock studied great predictors who he calls Superforecasters and he co-created the Good Judgement project.

Good Judgment co-founder (and Wharton professor) Barb Mellers and INSEAD professor Ville Satopaa summarized the new research findings:

What they found out, as a simple rule of thumb, is that about 50% of the accuracy improvements that they saw going from the regulars to the [Superforecasters] can be attributed to noise reduction. The remaining 25% is information improvement, meaning that they have more information, and that last 25% will be then bias reduction.

What is noise reduction? Ignoring and getting rid of factors that do not matter or actually result in bad predictions. There was an improvement is picking better baseball players for professional baseball teams by focusing on on-base percentage.

What is bias reduction? In forecasting, you might have want a certain result to happen. You go through the prediction process looking for data that confirms the pre-conceived desired result. You can find out you have certain biases by looking at past predictions and seeing if wrong predictions are skewed base upon a bias.If you know or suspect that you have certain biases, then you have to strengthen your assessment of the opposite side of your bias and increase your critical view of the side you favor.

Someone might have a bias against believing that Canada could have a strong economy in the future. They could assume that Canada’s per capita GDP is less than other developed nations. However, if they were to review this belief, they would see that Canada has a higher per capita GDP than Germany, France and the UK. Canada only trails the USA, Australia in countries with a population

Better Predictions

Sports predictions are most commonly done by sports bettors. Bettors who are more accurate do extensive research and handicap games accordingly. By performing smart sports handicapping, they can avoid the risks and increase their winnings more often.

They understand the data of a game so that they can adequately handicap it. The main trick is to know the numbers that can help you tell if a team will win or not. Significant numbers to consider for an NFL game include:

Rushing yards in a game
The number of yards every rush attempt
The Turnover ratio
The number of interceptions a team has per game
The yards met per every pass attempt

Expert handicappers also consider the performance at home and on the road, the first half scoring record, or the difference in yards per point for both the defense and the offense.

Baseball bettors need to look at on base percentage.

Political and Geopolitical Predictions

Bueno de Mesquita has made a slew of uncannily accurate predictions using Game theory. He has made more than 2,000 predictions on subjects ranging from the terrorist threat to America to the peace process in Northern Ireland. Bueno de Mesquita methods focuses on bias reduction.

How does Bueno de Mesquita do this? “You start with a set of assumptions, as you do with anything, but you do it in a formal, mathematical way,” he says. “You break them down as equations and work from there to see what follows logically from those assumptions.” The assumptions he’s talking about concern each actor’s motives. You configure those motives into equations that are, essentially, statements of logic based on a predictive theory of how people with those motives will behave. From there, you begin building your mathematical model. You determine whether the predictive theory holds true by plugging in your data, numbers derived from scales of preferences that you ascribe to each actor based on the various choices they face.

The Prisoner’s Dilemma, a basic in game theory, explains it well: Two burglars are apprehended near the scene of a crime and are interrogated separately by the police. The police know these two goons did it, but they don’t know how, so they offer each one a deal. If they both confess and cooperate, they’ll both get a minor sentence of five years. If neither man confesses, they’ll both get only one year (for having been caught with some of the stolen loot on them). But, and here’s where it gets interesting, if one confesses and the other doesn’t, the one who confesses walks out scot-free while the other will do 10 years. What will they do? Will they trust each other and do what’s obviously in their best interest, which is to not confess? Based on game theory’s assumptions about human nature, the math derived from this dilemma tells you squarely that the two burglars will turn each other in.

Finance and Business Investment Predictions

Venture Capitalists try to predict which startup companies will achieve large success. Successful VCs look at the management team, business plan, business model and the science and technology advantages of the company.