Print Friendly and PDF
only search openDemocracy.net

How can the Tomorrow's Europe poll claim to be representative?

Subjects:
About the author

A freelance writer and editor based in London, J Clive Matthews is Managing Editor of openDemocracy's EU and deliberative democracy blog, dLiberation.

In the real world he has co-authored two books and edited numerous others (ranging in subject-matter from movies to modern Russian politics), been acting editor on a glossy history and travel magazine, editorial consultant for a big name women's magazine, a freelance news editor for AOL UK, worked in both the House of Commons and the European Commission, and contributed to publications as diverse as Starburst and the Times Literary Supplement.

Best known as Nosemonkey online, he has been blogging about British and European politics daily for several years both at his own blog and sites like The Sharpener, General Election 2005 (now defunct), AgoraVox, France 24 and the Washington Post / Newsweek's Postglobal, as well as about movies for the BBC, and has been shortlisted for blog awards by the likes of the Guardian, Deutsche Welle International and the Weblog Awards, amongst others.

The argument for the Tomorrow's Europe poll's representativeness (the first of the three criteria for success) hinges on the claim that it was a "scientifically representative sample" of the population of the European Union. To ensure this scientific representativeness, random sampling was chosen. (Random sampling's benefits lie in simple probability - given a large enough sample, a random selection should produce a representative cross-section of the thing being sampled.)

As such, a random sample of 3,500 people from an EU population of nearly 500 million should end up being fairly representative of the whole*, and a random sample of 400 of those 3,500 should in turn produce a representative sample of that initial sample. Hence the repeated claims by the poll's organisers of creating "the EU in microcosm", and deliberative polling's mastermind, Professor James Fishkin, arguing that "The microcosm chosen by lot embodies political equality because every citizen has an equal random chance to take part". It all sounds fine in theory.

However, the Tomorrow's Europe poll did not take a random sample of 3,500 people from the whole of the EU. Instead, the desire to ensure that all member states were represented meant that what was actually conducted were 27 separate random samples of much smaller numbers, based on the proportion of seats each member state holds in the European Parliament (EP).

So, of the 785 seats in the EP, Germany holds the largest number with 99 - 12.6% of the total - with Malta holding the fewest with just 5 - 0.6% of the total.

12.6% of 3,500 should lead to a random sample of 441 Germans - still a moderate amount, but hardly enough (following even the most basic sample size rules) to provide an overly accurate picture of a population of 82.5 million. 0.6% of 3,500, however, would lead to a random sample of just 21 Maltese, from a population of 404,000.

In turn, 12.6% of 400 should lead to 50.4 Germans among the final group, with 0.6% of 400 leading to 2.4 Maltese.

Yet in Tomorrow's Europe's initial sample, 80 Maltese were polled (2.8%), and only 380 Germans (10.9%). In other words, the proportionality was out - Maltese were over-represented, Germans under-represented.

In the final group only 362 people, rather than 400, attended (figures for the nonattendance of the various nationalities have not been released, but have been estimated at 15%). This included 47 Germans (13%) and 3 Maltese (0.8%), seemingly by happy coincidence rather than design ending up somewhat closer to what the percentages should have been.

But add to this the lack of samples of sufficient size from each of the 27 member states for the benefits of random sampling to kick in, how can the Tomorrow's Europe poll claim to be representative?

* Note, in addition, that 3,500 from 500 million is a very small sample in itself - the regular Eurobarometer EU-wide opinion polls instead survey 1000 people from each member state, a total sample of 27,000.


We encourage anyone to comment, please consult the
oD commenting guidelines if you have any questions.