Super Predictors

Mar 28

Another podcast contemplation today. This time from Hidden Brain, another of my favourite podcasts. The guest on the show was one of the co-authors of a book called "Superforecasting: The Art and Science of Prediction," Phil Tetlock. (Might have to add this to my reading list) Hmm, at least sounds related to data science, right?

The story's actually quite fun. In this case, the US government decided to pit some of their expert analyst teams against others to compare powers of prediction on security related matters, such as the probability that Russia will invade "x" country in the next 18 months. What they found is that Tetlock's team (which apparently included a retired irrigation specialist and former ballroom dancer), ended up doing better in their predictions than the security experts, who had access to classified information that the lay teams did not have. How did they do it?

Bringing in another favourite topic of mine, growth-mindset, they approached these tasks from a persepctive that prediction was a skill that could be learned and improved upon over time. Starting with that premise, here's some of the approaches they employed.

1. Start from the outside-in

Here's how most of us make predictions:
Let's say you're friend asks you whether you think they will fail their statistics final exam tomorrow.
Some of the things you are likely to consider are:

• What you know about your friend's capabilities in statistics
• How much they've studied
• How much sleep they'll get
• Whether they typically do well in testing conditions
• Whether they have anything else going on that might impact their success

But apparently this is to our detriment! It's not that any of these are terrible things to consider, it's just that starting here is not so great.

Here's what people skilled at predition do instead:
Start with a baseline that draws from knowledge of others in similar circumstances.

So, to answer your friend's question, what you should be doing is:

• Consider the average passing rate for people who have taken the statistics final exam
• Discover if there are any personal characteristics shared by your friend that seem to contribute to passing the exam
• Only after you've done all of that, add in some small adjustments for the specifics of your friend's situation (that were mentioned above)

Which leads to the second suggestion they made.

2. Good prediction is degrees of 'maybe'

As stated in the show, when most lay people are making predictions they usually pick from one of three categories:

1. Yes
2. No
3. Maybe

But just as we were to make adjustments to our baseline estimate for our friend's exam success, effective predictors have much more precise measurements within 'maybe' of where the probability of an event lies. Being able to narrow in on the difference in a couple of percentage points for the probability of an event is critical.

3. Make regular adjustments

One of the things they said was that good prediction regularly udpates the model based on small changes. By themselves, these small changes might not seem significant, but especially compounded on each other over time, these can add up.

So in the case of our friend, the next they come and ask us whether we think they will fail their NEXT final, we shouldn't just use the same model that we used for the previous statistics exam. Instead, we should be adjusting our model to incorporate the changes in conditions for this situation.

My takeaways

This really does remind me of some of the conversations that we do have in social services. Findings that are made, and the decisions based on them, are often so static. We complete research, we make findings, we make decisions for action based on these findings. But it is far less common for us to update our research, or test the previous assumptions based on new findings. (I think one of the reasons for this actually has to do with the nature of the type of research that my field of social sciences does, but that might get us too far off track)

So my biggest takeaway: For findings or models to create accurate predictions for the future, they need to be regularly updated.

Though, hearing all of this today, does give me some hope. If we can make predictions about whether Russia WILL invade some country in 18months within some level of probability, surely that means we can do things like model the workings of homelessness in a city to better determine what types of interventions could help people exit homelessness, right?