How can the Tomorrow's Europe poll claim to be representative?

J Clive Matthews
18 October 2007

The argument for the Tomorrow's Europe poll's representativeness (the first of the three criteria for success) hinges on the claim that it was a "scientifically representative sample" of the population of the European Union. To ensure this scientific representativeness, random sampling was chosen. (Random sampling's benefits lie in simple probability - given a large enough sample, a random selection should produce a representative cross-section of the thing being sampled.)

As such, a random sample of 3,500 people from an EU population of nearly 500 million should end up being fairly representative of the whole*, and a random sample of 400 of those 3,500 should in turn produce a representative sample of that initial sample. Hence the repeated claims by the poll's organisers of creating "the EU in microcosm", and deliberative polling's mastermind, Professor James Fishkin, arguing that "The microcosm chosen by lot embodies political equality because every citizen has an equal random chance to take part". It all sounds fine in theory.

However, the Tomorrow's Europe poll did not take a random sample of 3,500 people from the whole of the EU. Instead, the desire to ensure that all member states were represented meant that what was actually conducted were 27 separate random samples of much smaller numbers, based on the proportion of seats each member state holds in the European Parliament (EP).

So, of the 785 seats in the EP, Germany holds the largest number with 99 - 12.6% of the total - with Malta holding the fewest with just 5 - 0.6% of the total.

12.6% of 3,500 should lead to a random sample of 441 Germans - still a moderate amount, but hardly enough (following even the most basic sample size rules) to provide an overly accurate picture of a population of 82.5 million. 0.6% of 3,500, however, would lead to a random sample of just 21 Maltese, from a population of 404,000.

In turn, 12.6% of 400 should lead to 50.4 Germans among the final group, with 0.6% of 400 leading to 2.4 Maltese.

Yet in Tomorrow's Europe's initial sample, 80 Maltese were polled (2.8%), and only 380 Germans (10.9%). In other words, the proportionality was out - Maltese were over-represented, Germans under-represented.

In the final group only 362 people, rather than 400, attended (figures for the nonattendance of the various nationalities have not been released, but have been estimated at 15%). This included 47 Germans (13%) and 3 Maltese (0.8%), seemingly by happy coincidence rather than design ending up somewhat closer to what the percentages should have been.

But add to this the lack of samples of sufficient size from each of the 27 member states for the benefits of random sampling to kick in, how can the Tomorrow's Europe poll claim to be representative?

* Note, in addition, that 3,500 from 500 million is a very small sample in itself - the regular Eurobarometer EU-wide opinion polls instead survey 1000 people from each member state, a total sample of 27,000.

Had enough of ‘alternative facts’? openDemocracy is different Join the conversation: get our weekly email


We encourage anyone to comment, please consult the oD commenting guidelines if you have any questions.
Audio available Bookmark Check Language Close Comments Download Facebook Link Email Newsletter Newsletter Play Print Share Twitter Youtube Search Instagram WhatsApp yourData