Eurobarometer, the EU’s official polling agency, is frequently cited in the media and consulted by politicians, think tanks and EU institutions. But these surveys have serious problems.
Information has been granted access to Eurobarometer’s response rates: they show the percentage of people invited to be respondents who actually go on to participate in interviews. According to leading experts in electoral research, the response rates are so low in a number of member states that using the data collected presents substantial problems and is likely to result in a systematic overestimation of public support for the European Union.
The EU seems to be painting an overly flattering picture of itself.
»Eurobarometer’s response rates are so low that they would be highly questionable in a research context,« said Kasper Møller Hansen, Professor of Political Science at the University of Copenhagen, Denmark.
Calling the response rates »horrifying,« he added:
»Ultimately, we run the risk of politicians looking at the Eurobarometer figures and drawing the wrong conclusions: that the public want more EU than they actually do. Or that a referendum can easily be won.«
Of every one hundred Germans who were asked to participate in a Eurobarometer survey in early 2018, only fifteen accepted: a response rate of 15%. Experts consulted by Information estimate that the response rate ought to reach 45-50% before a survey is representative.
In the most recent Eurobarometer survey for which response rates have been calculated, percentages were too low in a number of countries: 14% in Finland, 15% in Germany, 20% in Luxembourg, 22% in Italy, 27% in the United Kingdom, 28% in Denmark, 31% in Greece and France, 33% in Ireland, 34% in Spain, 38% in Latvia, and 40% in Portugal.
Information has seen response rates for five Eurobarometer polls undertaken from the beginning of 2016 to the beginning of 2018. According to the experts, all five pose problems.
»If my hypothesis is correct, then Eurobarometer is systematically underestimating the degree of Euroscepticism,« said Paolo Segatti, professor at the University of Milan.
This is because citizens who are most critical of the EU tend to be less inclined to participate in a lengthy interview about the EU.
»It’s no good at all. You cannot estimate the proportion of Euroscepticism with a response rate of 15%,« said Hermann Schmitt, Emeritus Professor at the Mannheim Centre for European Social Research.
»It’s people with less formal education who are the hardest to get to respond to surveys, so the low response rate affects representativeness.«
In the survey from early 2018, response rates were acceptable in a minority of EU member states: 81% in the Netherlands, 75% in Sweden, 65% in Slovakia, 62% in Romania, 54% in Hungary, 49% in Bulgaria, 47% in Malta and Belgium, and 45% in Slovenia.
Western Europe is seeing a general shift towards falling response rates, with fewer people agreeing to be interviewed. Eurobarometer polls, however, differ from other polls in that they don't use telephone or online interviews, but only conduct interviews with members of the public face-to-face at home. This makes it even more difficult to achieve high response rates.
»Eurobarometer has chosen a difficult and expensive method,« said Kasper Møller Hansen. »After all, very few people are prepared to receive visitors just to sit in the kitchen and answer questions about politics.«
Most other extensive electoral surveys in Europe use a combination of methods.
The Standard Eurobarometer polls have a maximum budget of 51 million euros over four years – nearly 13 million euros per year.
Eurobarometer acknowledges that response rates have dropped, and that »a high response rate is preferable and that the response rate is often used as an indicator of quality of data.«
But they don’t believe that the falling rates disqualify their findings.
»We must also always have in mind that Eurobarometer surveys measure people’s perceptions and feelings about topics, they are not and do not purport to be statistics,« they wrote in an email to Information.
»Unlike other public opinion polls, through Eurobarometer surveys, all EU Member States are covered coherently with the same methodology, the same questions and at the same time.«
Eurobarometer have already taken action to »improve the quality of the whole process, of which response rates are an element.« There are no plans, however, to conduct online or telephone interviews.
