Are PISA scores really that important?

What do PISA scores actually tell us about a country? Judging from data rather than headlines and rhetoric, the answer seems to be 'not a lot'.

Jeremy Fox
11 December 2013

OECD’s Programme for International Student Assessment (PISA) tests 15-year-olds on their knowledge of reading, maths and science. Publication of the latest test results has raised much alarm in the UK - and in other countries that we think of as “advanced” - because our students seem to be flatlining in a world apparently being overtaken by the likes of China and the Asian Tigers. Some of the dismayed rhetoric emanating from the Education Secretary has sounded eerily like an updated version of a terror known in the late nineteenth and a good part of the twentieth centuries as the “yellow peril”, namely that the West was about to be overrun by hoards of ill-intentioned folk from the Far East. Ghengis Khan, thirteenth century founder of the fearsome Mongol Empire, provides the proto-type.

“Do PISA scores matter?” was the introductory topic on BBC’s December 5 edition of Question Time.  All three politicians on the panel concurred that they do, along with that self-important purveyor of prejudice Nick Ferrari (formerly of Fox News, deviser of Topless Darts and the News Bunny - you get the picture).  Professor Mary Beard, sole educator on the panel, found herself alone in questioning the validity of PISA scores; but though she was clearly the most knowledgeable of those present, she was also the least articulate, possibly because she was less accustomed than her fellow-panelists to the soundbite world of instant opinion. Even though her intervention may have lacked impact, however, its thoughtfulness provoked me to dig a little deeper into international educational comparisons.

Finding PISA sceptics proved easier than I expected.  A Huffington Post article by Diane Ravitch, professor of education at New York University dismisses international test scores as worthless. While the US has never been first in the world, nor even near the top, on international tests, Ravitch claims it has done pretty well economically and in terms of scientific invention and business creativity - in fact better on many measures than any other nation. She bases these assertions partly on her own observations but more on a paper by Keith Baker, a retired researcher at the US Department of Education, entitled Are International Tests Worth Anything?

Baker’s paper begins with his main conclusion: “...for the top dozen or so most-advanced nations in the world, standings in the league tables of international tests are worthless.” He then goes onto analyse how well the First International Comparison Study (FIMS) - administered in 1964 to 12-year-olds in 11 countries - predicted national success in the first decade of the 21st century. [1] That he focuses his analysis on the US should not concern us unduly since of the eleven country participants in the 1964 FIMS, US students came second last, just ahead of Sweden.

Politicans wringing their hands (or gloating) over their nation’s PISA performance tend to think in simple terms of economic growth. Baker goes both deeper and wider. His measurements cover Wealth Creation, GDP growth, Productivity, Quality of Life, Democracy, and Creativity; and his results do not simply cast doubt on the validity of international comparisons of student performance, they are in some cases, starkly counterintuitive. He demonstrates convincingly, for example, that the higher a nation’s test score 40 years ago, the worse its subsequent economic performance”.

In an analysis of a more recent PISA exercise incorporating 27 countries, Baker reaches similar conclusions :”...the nations that scored at the PISA average generally outperformed those scoring above or below average….   Mediocre test scores correlate with better, more successful countries than do top scores (or lower scores)”.

Rather than worrying about PISA, Baker argues, policy-makers would do better to concern themselves with issues that UK readers might recognise: “run-down schools in the nation’s inner cites, misdirected parental interference in schools, …. the lack of parental and administrative support for teachers” and assisting students to pursue the dreams and ambitions that really matter: life, liberty and the pursuit of happiness.[2]

Cambridge economist Ha-Joon Chang comes to a remarkably similar conclusion about education in his entertaining 23 things they don’t tell you about capitalism.  In Thing 17, subtitled “More education in itself is not going to make a country richer,” Chang provides a plethora of examples to illustrate the point. Education is valuable, he maintains, not for raising productivity or GDP, but for helping us to “...develop our potentials for living a fulfilling and independent life.”

The table below offers a comparison of selected countries by PISA score, Nobel prizes (as a simple proxy for intellectual creativity) and GDP per capita. In the first of the two Nobel columns, I have counted solely Nobel prizes awarded since the beginning of the century so as to allow for the recent developmental advances that have occurred in the Far East.  The second column accounts for all Nobel prizes as a proportion of each country’s population. Israel and Finland are included on account of their reputation for educational quality.


Clearly the countries with the worst PISA scores are those with the most impressive Nobel record. Equally significant, the correlation between PISA performance and GDP per capita is, as both Baker and Chang suggest, rather weak (less than 0.5). Admittedly, the GDP comparison does not allow for a dynamic element of growth; but it does suggest the enormous distance China still has to travel in order to achieve a GDP per capita level similar to that of the “developed” world. Rather than fearing China’s progress we should be urging her on as well as applauding the achievements of countries like Korea and Taiwan that, in economic terms at least, have already reached the promised land.

Education Secretary Michael Gove has given no sign that he might be aware of the ambiguities inherent in PISA comparisons. What he has evidenced repeatedly - as Seamus Milne outlines in a corruscating critique - is a remarkable degree of political incompetence allied to an uncritical belief in his own genius, characteristics that would make him a figure of fun were his role in public life less important. On hearing him speak, I find myself irresistibly drawn to the thought that he might be an incarnation of Thomas Gradgrind, Dickens’ teeth-grinding, flint-hearted headmaster, for whom all that matters in life are “facts and calculations.”  Gove’s own words suggest he would probably welcome the likeness. Were he alive, would Dickens despair at this re-emergence in the flesh of what he had satirised on the page?

We should not be surprised then that Gove interprets the PISA results as indicating national failure. Rather, he gives the impression of relishing what he considers to be evidence of our scholarly mediocrity because it provides ammunition for his assault on two of his pet hatreds: state education and the Labour Party. Naturally, in the Education Secretary’s smug universe, the excellence of teachers, which he is ever quick to note for fear of an unwelcome riposte from the profession, is entirely a consequence of Tory (meaning his) policies, whereas our PISA results are a product of Labour ineptitude.

Among the current crop of parliamentarians, Gove is far from alone in preferring facile opinion to critical thinking. Today’s politicians would doubtless all claim to be well educated. Yet in the matter of PISA, as in so many other matters, most display a remarkable lack of curiosity about the data they are fed. What might cause us to doubt the quality of our educational system  is their apparent inability to apply their minds to anything not addressed in their Party instructional manual. If at some stage they were trained to think for themselves, they seem to have deposited that faculty at the door of Party Central Office - hoping perhaps to pick it up when they retire and not realising that brainpower left in storage tends to atrophy for lack of exercise.

“In science,” Einstein once remarked, “Imagination is more important than knowledge.” [3] Gove and many of his fellow parliamentarians appear to believe the opposite. Readers may judge which of the two is more likely to be right.


Liked this piece? Please donate to OurKingdom here to help keep us producing independent journalism. Thank you.


[1] He justifies the choice of decade on the grounds that “Today’s world is largely a world created and operated by the now 55-year-old FIMS generation”

[2] It is worth emphasizing that Baker does not claim (and nor do I) that educational standards and scores within nations are not valid and useful.

[3] Interview with George Sylvester Viereck, Saturday Evening Post, October 26, 1929.

Who is bankrolling Britain's democracy? Which groups shape the stories we see in the press; which voices are silenced, and why? Sign up here to find out.


We encourage anyone to comment, please consult the oD commenting guidelines if you have any questions.
Audio available Bookmark Check Language Close Comments Download Facebook Link Email Newsletter Newsletter Play Print Share Twitter Youtube Search Instagram WhatsApp yourData