Print Friendly and PDF
only search openDemocracy.net

Representativeness: a response

James Clive Mathews has taken issue with the representativeness of the sample. In response to his query, we released the time 1 opinions of the 3,500. Our policy is never to do so prior to a Deliberative Poll because publishing poll results may influence the deliberation. But afterwards there is no harm. The time 1 results are just another poll.

The only way to compare statistically the answers to questions is to compare the means. But means put on a 0 to 1 scale are incomprehensible to journalists and the public so we released percentage breakdowns instead.

Mathews has conducted his comparison by picking out parts of questions (in fact, parts of one selected question only). When we compared the means for all the questions between the 3,500 and the 362, we found that the substantive differences were only 4% of what they could possibly have been. Even a cursory comparison of the answers will show that the differences are mostly small between the 3,500 and the 362.

But yes there are some differences which go in different directions for different substantive issues. Remember that the deliberations were not about whether the EU was a good thing or a bad thing (the one question Mathews picks out) but about what kind of EU there should be on many different dimensions-the extent of the welfare state, enlargement, foreign policy, what levels of decision are appropriate for different kinds of policies. The comparisons for these various issues go in different directions.

I would add that two other comparisons Mathews makes are less relevant. First, he seems to be relying on a Eurobarometer of a different date which offers slightly different results. Opinion on these items will vary from week to week, month to month. And the results we post for the 3,500 and the 362 are both "valid per cent", leaving out the "don't know" responses in both cases (which is standard practice).

However, the Eurobarometer results he cites break out the "don't knows" and he says these can be 20% or more. Please note that 60% approving becomes 75% approving if one leaves out 20% in the denominator. So his comparison to a full scale Eurobarometer is doubly irrelevant. First it is a different survey at a different time. Second it seems to include the don't knows (and it may also have different weightings for different countries, complicating the comparisons).

Mathews also reports on the same table the final end-of-weekend results, But these should not be used to say anything about representativeness. Rather they show the opinion changes from deliberation. So two of his four comparisons really tell us nothing about representativeness. And in any case, his analysis concerns only one question.

No microcosm is perfect. And perhaps the participants liked the EU slightly more than the non-participants. But they changed on many issues in different directions in ways no one I know predicted (note for example the results on enlargement). And the materials and the process that led to those changes was transparently balanced. So to use terms like "brainwashed" is really misleading and inappropriate.


We encourage anyone to comment, please consult the
oD commenting guidelines if you have any questions.