King's College London

"The What Works Centres: Lessons and Insights from an Evidence Movement", edited by Michael Sanders and Jonathan Breckon, is published today. Find out more about the book.

 

Imagine a game show with a simple premise. In front of you are three doors. Two have goats behind them, one has a car. The host asks you to pick a door – you select door number two. In a twist, the host opens door number three, revealing a goat, and gives you a new choice – to either stick with door number two, or switch to door number one. What do you choose?

 

If you’re anything like most people, you’ll stick with your first choice. It feels safer – you’re not going to experience the regret of losing having reneged on your original choice – and it doesn’t make any difference anyway – it’s 50-50, right?

 

Wrong. This is known as the Monty Hall problem, named after the gameshow presenter who first made use of it. In truth, you have a two thirds chance of winning if you switch, and just a one third chance if you stick with your first choice. You should always switch in this kind of scenario, but the overwhelming majority of people don’t. The truth – that switching is twice as good as sticking – is counterintuitive, even once it’s been demonstrated to us, which takes about five minutes if you’re armed with three cups and a toy car.

 

I’ve taught this example to students around the world; to civil servants, teachers and social workers; to senior leaders in various governments. The results are invariably the same. People’s intuition always runs counter to reality. And here’s the secret – so does mine. I know that the solution is true, because I’ve demonstrated it more than 1,000 times, but I don’t really believe it. That’s because my intuition sucks, and so does yours.

 

I believe in the power of chess as a tool for helping us think better, more logically, about the world. This makes intuitive sense, but isn’t true. A robust evaluation of teaching chess in schools, funded by the Education Endowment Foundation,  showed no effect on attainment. The EEF is one of 13 What Works Centres who, over the last 10 years, have been showing just how wrong our intuition can be, by using the scientific approach that is standard in medicine to look at policy problems.

 

I was involved in an intervention to place social workers in 150 schools across the country. I believed that this would catch challenges early, reducing intervention by children’s services and leading to fewer children being taken into care. The results of that study were published last month and found no effects across any of the study’s several outcome measures. I was wrong – and so were the schools and local authorities, and the department for education, who had hoped and expected it to work.

 

It’s not just my intuition that is off. Achievement for All was a whole school intervention to boost attainment developed by Sonia Blandford, an eminent professor of education with a stellar academic pedigree. She founded a company to take the intervention out to schools, which was financially successful, with a large number of schools paying to transform their schools with the model. Even the EEF gave Achievement for All nearly a million pounds – presumably again in the hope, that it could have an impact. The independent evaluation concluded that yes, Achievement for All made a difference -– it was catastrophic, reducing attainment by the equivalent of two months of schooling. Smart people with a huge variety of expertise looked at Achievement for All and figured it would work – but their intuition was wrong.

 

These are just a few examples of what we’ve learned from 10 years of the What Works network, and there are many, many more. They show a simple, but hard to swallow truth. We are not good at guessing what works, even if we’re experts in the field. If we trust our gut, we’ll end up supporting things that don’t work, or which might even make things worse. The solution – the only solution – is to rigorously test impacts, and to trust the science, not our instincts.

 

Michael Sanders is Professor of Public Policy and Director of the Experimental Government Team at the Policy Institute, King’s College London