The decline of evidence based policy

There is a brief description of Cartwright and Hardie’s book ‘Evidence-based policy: a practical guide to doing it better’ on Amazon. It starts: “Over the last twenty or so years, it has become standard to require policy makers to base their recommendations on evidence. That is now uncontroversial to the point of triviality--of course, policy should be based on the facts.”

My immediate reaction is to say ‘unless that policy involves the macroeconomics of fiscal policy’, but once you see one area where evidence is ignored, you begin to see many more. Here are just two that I have read about over the last few weeks, courtesy of the UK government. They can better be described as ‘emotion-based policy’, or ‘election-based policy’, and I fear they may now be the rule rather than the exception.

The first involves what is generally known as the ‘bedroom tax’. The policy reduces the housing benefit payable to tenants deemed to be under-occupying their homes. This policy has undoubtedly caused considerable hardship to many of those affected, but it also saves public money. So crucial in any assessment of the desirability of the policy is how much money it saves. Getting such an estimate is complex, because it will depend on a lot of factors, like whether individuals move in response to losing benefit, where they move to, and so on. The UK’s Department of Work and Pensions has a model that calculated savings of £480m in 2013/14, an estimated it published in June 2012.

So the first thing to do in trying to assess the realism of the £480m figure is to look at the model. We cannot do that, as it is not published. However, Rebecca Tunstall of the Centre for Housing Policy at the University of York has managed to obtain some spreadsheets, using Freedom of Information requests. In a report, she looks at some of the assumptions behind the department’s calculations. Many look very questionable, and as Alex Marsh observes here, there seems to be a pattern - the questionable assumptions tend to overestimate the savings involved. The calculations also ignore some of the other financial costs and consequences that an overall assessment of the policy should take into account. Alex Marsh suspects that allowing for these, and taking into account the evidence we are now getting about how people are responding, the savings created by the policy could disappear completely.

If policy was evidence-based, then either the government would be disputing in detail Professor Tunstall’s analysis, or rethinking the policy. Neither are likely to happen for one simple reason. The policy was never evidence based. It was instead inspired by the tabloid led attack on welfare recipients, and the idea that it was unjust that ‘hard working taxpayers’ should fund a spare room for benefit recipients. Here for example is Stephen Glover writing in the Daily Mail: “The notion that many families not on welfare don’t have the luxury of a spare room, and may have to have one or two people in every bedroom, is foreign to the head-in-the-clouds types that proliferate at the BBC.” Now that feeling of unfairness is real enough, but it is not based on evidence either about what impact the policy might have, or whether it will actually save any money.

The second example is the recent ‘clampdown’ announced by the UK Home Secretary on ‘health tourism’. Among the measures included in the government's new immigration bill is a £200 charge on all temporary migrants for using the NHS and a requirement for GPs to check the migration status of new patients. Both policies are aimed at preventing migrants travelling to the UK to seek free healthcare. Conservatives has been particularly keen to attack ‘health tourists’ from other EU countries, thereby clocking two election sensitive issues among potential UKIP voters: migration and the EU.

So what evidence does the UK government have on the scale of this EU health tourism? The answer is that it has none. A classified document had the following very telling phrase: "we consider that these questions place too much emphasis on quantitative evidence". So considerable extra costs are going to be imposed on GPs and other parts of the NHS to help deal with a problem which may be trivial.

However the European Commission has succeeded in getting some evidence on ‘benefit tourism’, which it recently published. As Jonathan Portes reports, approximately 4% of those claiming unemployment benefit (job seekers allowance) in the UK are EU migrants, although they represent well over 5% of those in work. More generally, as the Commission says: “Mobile EU citizens are less likely to receive disability and unemployment benefits in most countries studied.” For much the same reason, the proportional demands made by EU migrants on the UK health service are likely to be less than the native population. They may also choose to avoid the NHS, for a variety of reasons.

Which you might think is a bit embarrassing for a government that is about to create a whole raft of additional bureaucratic costs to deal with this ‘problem’. Well not embarrassing if you read the Daily Telegraph or Daily Mail. Their reading of the Commission report was that it showed “more than 600,000 unemployed European Union migrants are living in Britain at a cost of £1.5 billion to the NHS alone”. Only one slight problem - the true number is 38,000. So how did the Daily Telegraph manage to inflate the true number by a factor of 15? Because they ‘confused’ unemployed with non-employed, where the latter include students (of which there are many), retired people, carers and others with family responsibilities. As Richard Exell says: “Our government may be chumps when it comes to evidence-based policy making, but they can always rely on world-class distortion to see them through.” The report remains on the Telegraph’s website, uncorrected.

I’ve included this last paragraph to make a very simple point. Some commentators tend to argue that we make too much of what happens in the media (see Chris Dillow here for example). I think this is plain wrong. Information is vital. People make judgements based on the information they receive. As one student said to me this week, they knew that the Obama stimulus package had had little impact on the US economy, because they had read this in the FT and the Economist. (For better analysis, follow the links from here.) If people have distorted information (or if those bringing a bit of reality are vilified), politicians have no incentive to base policy on actual evidence. The decline in evidence-based policy, and the declining importance of facts for much of the media, are not unrelated.  


Postscript, 20th October

If a reputable newspaper makes a mistake, it acknowledges that mistake. Another kind of newspaper, it seems, can try shouting down those who pointed out the mistake. That was what the Daily Mail did over its attempt to slur Ralph Miliband as the ‘man who hated Britain’. So today we find the Telegraph doing the same over its reporting that “more than 600,000 unemployed European Union migrants are living in Britain”. Rather than admit that it got that wrong, it instead tries to pretend that it is in fact battling some kind of great conspiracy emanating from Brussels.

The lead article on its website today (19 October) by it chief reporter Robert Mendick could not be a better illustration of the points I made in the last paragraph of my post. The article’s first paragraph says

“The [Commission] study — whose details were first disclosed in The Telegraph — showed that more than 600,000 “non-active” EU migrants were living in the UK at a possible cost to the NHS alone of £1.5 billion a year.”

Here is the opening paragraph by the same reporter in the original (12 October) piece:

“More than 600,000 unemployed European Union migrants are living in Britain at a cost of £1.5 billion to the NHS alone, according to an EU report.”

Virtually identical – except for the little details of replacing unemployed with non-active (the writer puts non-active in quotes, apparently unable to translate this into ordinary language, like students, housewives, retired people), and the insertion of ‘possible’ before cost. No acknowledgement of these changes, but instead an attempt to manufacture another story, the ‘developments’ of which are

(a) some politicians did not like the conclusion of the Commission report. (No, really?)

(b) quotes from an academic at Oxford who says that “There is no problem with the numbers [in the report]. The issue is the interpretation of those numbers.” (Heady stuff!)

(c) the paper has found out that the “independent consultancies who wrote the report were awarded EU contracts worth more than £70 million over six years.” (Ah, the plot thickens.)

(d) a BBC report which attempted to reflect the facts had been accused as being unbalanced by a Conservative minister. (My god, this is serious.)

Then we have: “Evidence of mounting public concern in the EU’s biggest economies over migration emerged in a poll yesterday which showed that the introduction of restrictions on EU migrants’ rights is backed by 83 per cent of Britons..” I wonder where people are getting their information from!

And then there are attempts to discredit opponents. The European commissioner in charge of the department that published the study is quoted in a paragraph that begins: “Mr Andor, a socialist, said: ...”. Later on we have: “It also emerged that one of the main supporters of the report ... was in receipt of more than £600,000 of EU funding for the year ending March 2012 for his think tank.“  Yes, that is Jonathan Portes, director of the National Institute, which has been doing – shock, horror - research funded by the Commission. As Portes points out in the article, it has also been doing research funded by the UK government, but clearly the Commission has managed to exert its evil influence on Mr Portes where the UK government has failed. How can that be? The article helpfully informs us that he was a “former senior economics adviser to the last Labour government “. He was actually a civil servant working under both governments (he has an interesting account of his role advising Norma Lamont in 1992 here), but I guess that is another one of those little details, like the difference between ‘unemployed’ and ‘non-active’.