The Poll

Today dLiberation is coming to an end - for the time being at least. I've spent the last 24 hours trying to come up with some kind of neat final post, wrapping up all the various issues we've been covering here over the last few weeks, and laying down a final judgement on the successes and failures of the Tomorrow's Europe deliberative poll.

But, of course, it's just not that simple. I've jotted down (literally) thousands of words today - re-writing, cutting and pasting, editing and starting again from scratch countless times - only to realise that there is no short, simple conclusion here. Because almost all points that could be considered failures with the Tomorrow's Europe poll could equally apply to all other forms of representative democracy.

For the last couple of days, winning over those sceptical of such democratic innovations as deliberative polling (assuming such innovations are valid, of course) has been my chief concern. But, with EU-centred innovations, we also still have the Eurosceptics.

I've already mentioned the dominance of eurosceptics in the online English language debate, largely due to the influence (and traffic-boosting transatlantic links to the closely-knit network of right-wing American blogs) of EU Referendum, independently run by two associates of the highly eurosceptic Bruges Group thinktank, one of whom also used to work for the UK Independence Party, before turning his back on them for being too amateurish (or so I believe).

However, due to the fleeting and superficial coverage of Tomorrow's Europe in the mainstream media - some TV coverage, the occasional short article, but nothing overly in-depth - it will be to the web that most people will look in the weeks and months to come. Amongst the online coverage now will be found EU Referendum's assertion that "what is delivered is a number of findings that are both pointless and irrelevant, except that they will be treated with undue reverence by the EU commission and its lackeys, who will cite them as evidence of what the 'citizens of Europe' think and want."

James Clive Mathews asks why we have the small countries represented. He points out, correctly, that a simple random sample of the EU as a whole might easily leave out the small countries. And he asks, if we are really presenting this as a scientific sample of Europe how can we adjust the sample to ensure representation of the small countries?

The answer turns on how we represent the small countries. By employing stratified random sampling (rather than simple random sampling) we can ensure representation of the small countries and, in theory, actually reduce sampling error. If we have separate strata that are mutually exclusive (in this case separate countries) and we randomly sample from each, we can actually produce a more representative sample of the overall population.

Here we come to the fundamental problem with the EU's drive for democracy - getting true representativeness is seemingly impossible in a body as vast and diverse as the European Union. Even ensuring fair representation for the member states is well nigh impossible - let alone ensuring that smaller than national-level groups are also represented. There will always be complaints: where are the German Turks? Where are the French Basques? Where are the Russian Estonians? Where are the Afro-Carribbean British?

The claim for the Tomorrow's Europe poll was always that it was going to be a "scientific sample" of the whole of the EU. Yet - as with the European Parliament - the forced inclusion of member states with smaller populations in such a sample instantly makes it look odd. Based on population size, a truly scientific random sample of 3,500 people from the whole of the EU should expect (on average) to contain just 2.5 Maltese - yet the Tomorrow's Europe sample, thanks to the member state weighting, contained 80.

From a PR point of view, this is understandable - ensure every member state is represented, the chance for media coverage is greatly increased. But, at the same time, it makes selling the poll as "scientific" rather harder.

A genuinely random poll of 3,500 people from the whole of the EU? Fine. One weighted to ensure the inclusion of someone from every member state? This sounds less random, more likely to contain some selection bias, and therefore less convincing.

My concerns about the statistical representativeness of the Tomorrow's Europe poll remain, despite Professor Fishkin's response - though I accept that my complete lack of knowledge of statistical theory may well be the reason why the sample, to me, doesn't seem quite right (be it for a possible under-representation of eurosceptics or the definite over-representation of people with higher educational qualifications or from smaller EU member states).

Fishkin laid out three criteria for success: was it representative, was it deliberative, and will decision-makers listen? But these criteria leave off the single most important - will the people accept the method? Because the end result of deliberative polling must be to get the people to acknowledge that such polls genuinely do reflect what the situation would be if the people themselves were better informed and more politically engaged. Otherwise the responses will always be similar to those I highlighted from Margot Wallstrom's blog.

So, be it statistically representative or not, the key problem remains - with my apparent confusion merely highlighting the issue. Tomorrow's Europe was designed as an exercise in encouraging participation and engagement. To get people involved, simplicity and transparency is key - both of process and of results. If - after nearly seven weeks spent covering the poll in-depth and questioning key organisers - I still don't quite understand how it all works or whether it should be listened to, what chance have the public as a whole?

James Clive Mathews has taken issue with the representativeness of the sample. In response to his query, we released the time 1 opinions of the 3,500. Our policy is never to do so prior to a Deliberative Poll because publishing poll results may influence the deliberation. But afterwards there is no harm. The time 1 results are just another poll.

The only way to compare statistically the answers to questions is to compare the means. But means put on a 0 to 1 scale are incomprehensible to journalists and the public so we released percentage breakdowns instead.

Mathews has conducted his comparison by picking out parts of questions (in fact, parts of one selected question only). When we compared the means for all the questions between the 3,500 and the 362, we found that the substantive differences were only 4% of what they could possibly have been. Even a cursory comparison of the answers will show that the differences are mostly small between the 3,500 and the 362.

Two weeks after the deliberative poll, and around a month after the initial poll, we now appear to have all the results we need to assess the representativeness of the Tomorrow's Europe poll, with the release of the results of the initialal poll of 3,500 (PDF). This was the group from which the sample of 362 was taken - and so their representativeness should be compared to this initial poll. I've already noted my suspicions about the demographic representativeness - but, of course, what's needed as much as anything in an exercise like this is representativeness of opinion.

So as one of my suspicions was that pro-EU types would be more likely to take part, let's take that as a first point of comparison - especially as this is surely the most easy to measure political opinion when it comes to any aspect of EU politics. As a bonus, support for the EU is one of the things tested in the regular Eurobarometer opinion polls (most recent PDF), and so this makes for an easy point of comparison to a larger sample.

According to some scholars, the European Wide Public Sphere is a phantasm - impossible, misconceived and misguided. It is hard to imagine and only for utopian speculation. Now, however, we have seen and heard it. And for those who were not in the room Oct 12-14, you can go to the website and play the video and see and hear what it would look like.

Vice-President of the European Commission Margot Wallstrom, instigator of the Plan D programme under which Tomorrow's Europe was launched, has had a gander at the results of the poll and put up some thoughts on her blog. She picks out a few interesting results, but what is more interesting is perhaps not the areas that were intended.

First, there's the mention of other past "Plan D" projects (some of which I'd never even heard of, and I generally try to keep up with these things), and the plan to get a sample of 250 participants from all of them to go to Brussels in December and discuss how these events were run and affected them.

Second, however, is the comments the blog post has so far received.

In the latest part of his ongoing series about the role of democracy in the EU, Paul Davies gave the following quote from Aristotle:

"a great state is not the same thing as a state with a large population. But certainly experience also shows that it is difficult and perhaps impossible for a state with too large a population to have good legal government." - Aristotle, Politics, 1326a

It's an assertion that bears much consideration - especially when combined with the language difficulties of the EU.

My initial thinking about the Tomorrow's Europe poll was that the aim was to see what would happen if a representative selection of EU citizens were given access to better knowledge. "What's the point in that?", I thought, "as soon as you make them more knowledgable than the average, they cease to be representative, and so the final findings of the poll will be useless. All you'll prove is that people who know what they're talking about will make different choices to people who don't - which is both obvious and hardly of any use to policy-makers."

Then, on first looking at the Tomorrow's Europe briefing materials, my initial reaction was that they were packed full of differing opinions, but contained little in the way of actual facts. "How," I thought, "can you come to a considered opinion on any given issue without looking at the background information and trying to weigh up the facts? How are these people meant to increase their knowledge and make informed decisions when all they are being exposed to is subjective opinion rather than objective evidence?"

But I think I may well have been missing the point.

One of the key arguments supporting any kind of deliberative democracy is that the process of deliberation leads to an increase in knowledge and understanding - with the natural assumption that the more they improve, the more considered the opinions.

However, as already noted, one of the questions selected by one of the groups for the final Q&A with the experts, after three days of deliberation, was "what is the role of the European Parliament within the EU institutions". With such a fundamental aspect of the way the EU runs still not understood by at least some of the participants on the final day, just how much was their knowledge actually increased?

Well, this was one of the things the poll tested (PDF).

Such was the claim of the organisers of the Tomorrow's Europe poll, and the more I think about it, the more right I think they were.

Because, you see, the more I've been pondering whether or not the 362 people who attended the deliberation can truly be representative of the 500 million people who make up the EU, the more I've started to wonder just what the chances are of any "representative" body from such a large group looking anything like the group as a whole.

While we wait for the results of the initial survey of 3,500 Europeans to be released, to enable comparison with the results of the surveys of the 362 participants in the Tomorrow's Europe deliberation itself, perhaps comparisons with other surveys may be of help. The most obvious, of course, is the regular Eurobarometer surveys, polling the people of Europe on a country-by-country basis - and conducted by the same polling company, TNS Sofres, as conducted the initial survey for Tomorrow's Europe.

However, while comparisons to past opinion polls of EU citizens can help us check the representativeness of the participants this time around, part of the argument for Tomorrow's Europe is that the deliberative process enables the participants to come to more informed decisions. As such, perhaps the real point of comparison should be between the participants and those who take the decisions within the EU? Here, our best option appears to be the Compagnia di San Paolo's European Elites Survey - a poll of MEPs and senior workers at the European Commission and EU Council.

Having been spotted by a friend in the background of Friday night's Newsnight coverage of the Tomorrow's Europe poll, I naturally enough scampered off to make sure that I wasn't to be spotted picking my nose just over the presenter's shoulder (as happened a few years ago when I got rather tipsy on free champagne at the Channel Four Political Awards...). To watch the report, go here, click "watch latest programme", and scroll forward to the 22 minute mark - assuming you're doing it on Monday 22nd.

Picking one's nose is, of course, a disgusting habit. If you spot someone picking their nose, you'll doubtless think rather less of them. But what if it's only that someone tells you that they've seen someone pick their nose - something the accused denies vehemently? When you haven't witnessed it yourself, and have no hard evidence to rely on, who do you believe?

Newsnight asks, at the start of its report on Tomorrow's Europe, "will putting facts before prejudice change what they think?" But throughout the weekend, factual information on the EU was sparse, to say the least.

Nice overview article from one of Britain's leading pro-EU writers, Timothy Garton Ash, on the events of last weekend over at the Guardian. I can't say I spotted him amongst the attendees, but there were quite a few.

Garton Ash seems to have come to similar conclusions about the poll as I am heading towards, however: "More interesting than any result is the experiment itself." The reason? Simply because this was indeed the first time that people from all 27 member states were brought together in one room and allowed to chat amongst themselves in their own languages, all simultaneously interpreted.

To make sense of the results of the Tomorrow's Europe poll, we need:

1) The results for the 3,500 sample

2) The results for the 362 participants before deliberation

3) The results for the 362 participants after deliberation

They're now available. Analysis to follow - but any input much appreciated...

The argument for the Tomorrow's Europe poll's representativeness (the first of the three criteria for success) hinges on the claim that it was a "scientifically representative sample" of the population of the European Union. To ensure this scientific representativeness, random sampling was chosen. (Random sampling's benefits lie in simple probability - given a large enough sample, a random selection should produce a representative cross-section of the thing being sampled.)

As such, a random sample of 3,500 people from an EU population of nearly 500 million should end up being fairly representative of the whole*, and a random sample of 400 of those 3,500 should in turn produce a representative sample of that initial sample. Hence the repeated claims by the poll's organisers of creating "the EU in microcosm", and deliberative polling's mastermind, Professor James Fishkin, arguing that "The microcosm chosen by lot embodies political equality because every citizen has an equal random chance to take part". It all sounds fine in theory.

However, the Tomorrow's Europe poll did not take a random sample of 3,500 people from the whole of the EU. Instead, the desire to ensure that all member states were represented meant that what was actually conducted were 27 separate random samples of much smaller numbers, based on the proportion of seats each member state holds in the European Parliament (EP).