King's College London

This is the seventh instalment of the School for Government’s comment series on the future of evidence-based policy. Check our News & comment page for previous contributions.

The rapidly growing pool of randomised controlled trials and other robust impact evaluations in many domains over the last few years has meant that we now have a much stronger idea of what works, and, just as importantly, what doesn’t for achieving various policy goals.

Many of the most successful what works centres are characterised by long-term funding that has bucked the trend for short termism in budgeting, with funding often handed out a year at a time, or at most over the course of a three-year spending review. Centres with decade-long endowments have been able to focus on the long-term objective of building an evidence base, and following that evidence where it leads.

Although this approach represents a longer-term strategic approach, most individual trials are short, with outcomes rarely collected after more than a year after people begin treatment. This makes a lot of sense – you don’t want to be waiting years and years for results that will help you make spending decisions in the here and now. Schools, for example, must decide each year what to do and how to do it. Waiting three years for results rather than one would mean two years when they were unable to follow the evidence.

In many cases, outcomes in the shortish term are also relevant and important. Consider a curriculum intervention: whether or not a child makes significant progress in a subject over the course of a year when receiving an intervention compared with children not receiving that intervention is probably the most relevant outcome of the intervention, and likely the one a trial will aim to assess. That said, this potentially ignores a host of outcomes that are more salient to an individual’s life outcomes – if they pass their GCSE exams, or get a job, or avoid entering care, within 12 months – all are of important concrete value, particularly to the people funding or delivering the intervention. Why not capture those outcomes as well?

Our research should not stop there, however. The existing what works centres are disproportionately focused on outcomes for young people, and there are good reasons to think that benefits at one point in time might make a difference into the longer term. If someone is more likely to pass their GCSEs, are they more likely to pass their A-levels? Attend higher education? If they get a job, does it persist, and for how long? What happens to their earnings over the medium term?

As is a theme of this series of blogs, the lack of long-term follow up, particularly using administrative data, is an example of dollar bills being left on the sidewalk. We have the tools to cheaply understand much more about a range of interventions – about whether short-term effects attenuate, or long-term effects emerge from null results – but we have not yet fully utilised this resource.