Gwern is a research assistant at the Singularity Institute for Artificial Intelligence He has written a useful article about making predictions.
A prediction can be broken up into 3 steps:
1. The specification
* The first issue is simply formulating the prediction. The goal is to make a statement on an objective and easily checkable fact. It is best to state the source that will judge the prediction.
examples – global temperatures will increase by over 1.0 degrees Fahrenheit on average according to the next IPCC report
2. The due-date
3. The probability
Rolling back a prediction to the present day
What does the prediction about the future world imply about the present world?
Every prediction one makes is also a retrodiction: you are claiming that the world is now and in the past on a course towards the future you have picked out of all the possibilities (or not on that course), and on that course to the degree you specified. What does your claim imply about the world as it is now?
Example, Miller predicted 15% for “Within ten years either genetic manipulation or embryo selection will have been used on at least 50% of Chinese babies to increase the babies’ expected intelligence”. This initially seems reasonable: China is a big place with known interests in eugenics. But then we start working backwards – this prediction implies handling >=9 million pregnancies annually, which entails hundreds of thousands of gynecologists, geneticists, lab technicians etc., which all have lead-times measured in years or decades. (It takes a long time to train a doctor even if your standards are low.) And the program must be set up with hundreds of thousands of employees, policies experimented with and implemented, and so on. As matters stand, even in the United States mere SNP genotyping couldn’t be done for 9 million people annually, and genetic sequencing is much more expensive & difficult, and genetic modification is even hairier. If we work backwards, we would expect to see such a program already begun and active as it frantically tries to scale up to handle those millions of cases a year in order to hit Miller’s deadline. But as far as I knows, all the pieces are absent in China as of the day it was predicted; hence, it’s already too late. And then there are the politics; it is a deeply doubtful assertion that the Chinese population would countenance this, given the stress over the One Child policy and the continuing selective abortion crisis. Even if the prediction comes true eventually, it definitely will not come true in time. (The same logic applies to “Within ten years the SAT testing service will require students to take a blood test to prove they are not on cognitive enhancing drugs.”; ~1.65 million test-takers implies scores of thousands of phlebotomists, who do not exist, although in theory they could be trained in under a year – but whence the trainers?)
A second example would be a series of predictions on anti-aging/life-extension. The first and earliest prediction – “By 2025 there will be at least one confirmed person who has lived to 130” – initially seems at least possible (I am optimistic about the approaches suggested by SENS), and so I assigned it a reasonable probability of 3%. But I felt troubled – something about it seemed wrong. So I applied this heuristic: what does the existence of an 130 year-old in 2025 imply about people in 2011? Well, if someone is 130 in 2025, then that implies that are now 116 years old (130−(2025−2011)). Then I looked up the oldest person in the world: Besse Cooper, aged 115 years old. Oops. It’s impossible for the prediction to come true, but because we didn’t think about what it coming true implied about the present world, we made an absurdly high prediction. We can do this for all the other anti-aging predictions; for example “By 2085 there will be at least one confirmed person who has lived to 150” can be rephrased as ‘someone aged 76 now will live to 2085’, which seems implausible except with a technological singularity of some sort
Base rates are easily expressed in terms of frequencies: “of the last Y years, X happened only once, so I will start with 1/Y%”. (“There are 10 candidates for the 2012 Republican nominee, so I will assume 10% until I’ve looked at each candidate more closely.”)
An example: “A Level 7 (Chernobyl/2011 Japan level) nuclear accident will take place by end of 2020”. So the frequency would be 1 in ~20 years, which certainly puts a different face on a prediction spanning 9 years. This gives us a base rate more like ~40%. This is our starting point for asking how much does the rate go down because Fukushima has prompted additional safety improvements or closure of older plants.
NBF – The base rate can also look at the number of incidents per operating plant year.
Breaking predictions down into conjunctions
The prediction “Obama gets reelected and during that time Hillary Clinton brokers the middle east peace deal between Israel and Palestine for the two state solution. This secures her presidency in 2016.”, where the predictor gave it a flabbergasting 80%; before clicking through, the reader is invited to assign probabilities to the following events (and then multiply them to obtain the probability that they will all come true):
1. Barack Obama is re-elected
2. A Middle East peace deal is brokered
3. The peace deal is for a two state solution
4. Hillary Clinton runs in 2016
5. Hillary Clinton is the 2016 Democratic nominee
6. Hillary Clinton is elected
(Sometimes the examples are even more extreme than 6 clauses.) This heuristic is not perfect, as it works best on step-by-step processes where every step must happen. If this is not true, the heuristic will be overly pessi