For the past three years, Rich and 3,000 other average people have been quietly making probability estimates about everything from Venezuelan gas subsidies to North Korean politics as part of , an experiment put together by three well-known psychologists and some people inside the intelligence community.
According to one report, the predictions made by the Good Judgment Project are often better even than intelligence analysts with access to classified information, and many of the people involved in the project have been astonished by its success at making accurate predictions.
Brian Wang joined the project and began trading on predictions on Jan 9, 2014.
Getting good results are not just about making good predictions.
* It is about making quick predictions as soon as new propositions appear to help get the early shift from 50% to the direction of the likely result
* it is also about making bigger bets on predictions where results are more certain
* it is also about anticipating shifts in the feelings of the crowd toward predictions
“Everyone has been surprised by these outcomes,” said Philip Tetlock, one of the three psychologists who came up with the idea for the Good Judgment Project. The other two are Barbara Mellers and Don Moore.
For most of his professional career, Tetlock studied the problems associated with expert decision making. His book Expert Political Judgment is considered a classic, and almost everyone in the business of thinking about judgment speaks of it with unqualified awe.
All of his studies brought Tetlock to at least two important conclusions.
First, if you want people to get better at making predictions, you need to keep score of how accurate their predictions turn out to be, so they have concrete feedback.
But also, if you take a large crowd of different people with access to different information and pool their predictions, you will be in much better shape than if you rely on a single very smart person, or even a small group of very smart people.
“The wisdom of crowds is a very important part of this project, and it’s an important driver of accuracy,” Tetlock said.
In fact, Tetlock and his team have even engineered ways to significantly improve the wisdom of the crowd — all of which greatly surprised Jason Matheny, one of the people in the intelligence community who got the experiment started.
“They’ve shown that you can significantly improve the accuracy of geopolitical forecasts, compared to methods that had been the state of the art before this project started,” he said.
What’s so challenging about all of this is the idea that you can get very accurate predictions about geopolitical events without access to secret information. In addition, access to classified information doesn’t automatically and necessarily give you an edge over a smart group of average citizens doing Google searches from their kitchen tables.