King's College London

The promise of evidence-based policy, and in particular the “what works” movement in the UK and elsewhere, is that we will conduct small-scale trials of potential interventions and then scale up the successful ones. However, research shows that many successful interventions, even when low-cost, fail to scale.

John List, a Professor at the University of Chicago and one of the leading producers of field experiments that generate this kind of evidence, has in recent years turned his attention to the science of scaling, asking what needs to be done to achieve the promised impact of successfully trialled interventions.

List’s insights are valuable, but one thing that we observed, particularly when it comes to light-touch interventions inspired by behavioural science, is that scale is nobody’s job.

The independent evaluator or researcher who works on an intervention may be incentivised to move on to the next project; the consultants who developed the intervention might return to business as usual; policymakers might change roles, taking valuable knowledge with them; and practitioners might view the intervention just another task to add to their already punishing schedules.

For these reasons we decided to make scaling our job, using revenue from Exogeneity, a social purpose consultancy, to fund the scaling of a small number of projects where there was a positive evidence base. We undertook the scaling of three projects:

• A collaborative project distributing booklets that provide information on bursaries offered by universities.
• A collaborative project in which young people with good grades receive letters from role models.
• A feedback intervention for social workers in Wales.

Our results have been mixed.

The bursary booklet intervention was successfully scaled to all sixth form colleges in the UK, through our partnership with the Sixth Form Colleges Association, and is now in its third year. We’ve also added a version of the booklet that focuses on the support available to care leavers, which is distributed through Virtual Schools in Local Authorities in England, Scotland and Wales.

The letters from role models intervention has taken longer to get going, because here we are more dependent on others, in particular the Department for Education, who have provided us with the addresses to which the letters need to be sent, as well as permission to use their letterhead. Getting these permissions is time consuming but has ultimately meant that 10,000 letters have been sent out this month.

Finally, the feedback for social workers intervention hasn’t been successful, due to a lack of demand, and a lack of resource on our side to generate that demand.

So, what have we learned in 18 months of scaling? Our key finding is that by diverting even a relatively small amount of effort and funding (around £30,000 to date) to this effort, we’re able to make considerable inroads in scaling evidence-based interventions.

This has been easiest where we have existing relationships, and where we are able to control the bulk of the moving parts – by collecting data on university booklets, paying designers, and working with partners to ship materials. It’s been least successful where we’re a few degrees removed from the end users, and don’t have direct contact with them.

Overall, we’re feeling positive about what we’ve been able to achieve, and what we’ve learned in the process. We’re going to continue with what we’ve succeeded with so far, while expanding our portfolio to include new projects that we think have a decent chance of success.

Michael Sanders is Professor of Public Policy and Director of Experimental Government at King’s College London’s Policy Institute, and Director of Exogeneity.

Vanessa Hirneis is a research associate and doctoral student in the Experimental Government Team and Deputy Director of Exogeneity.