For many nations, the recent revelation of the Program for International Student Assessment (PISA) results were a call for celebration. For others, they were a sign that their nation might be falling behind—and, perhaps, cause for outright embarrassment.

PISA is a standardized test designed to evaluate the scholastic performance of 15-year-old students in math, reading and science. Ideally, nations will be able to use these results in order to develop better, more comprehensive curricula and learning strategies.

However, this program is not without its flaws, critics claim.

While the results speak to the scholastic achievement, it fails to account for other educational outcomes. Critics suggest that not only are PISA results not enough to determine the quality of education reliably, some argue that such a task might not even be possible.

Svein Sjøberg of the University of Oslo believes that PISA is comparing apples and oranges in most cases. For Sjøberg, the contextual differences between nations trouble PISA’s fundamental assumption: it is possible to create a universal test that validly measures student achievement across the borders of language, culture and curriculum.

As far as problems go, he argues, this is the tip of the iceberg. A perhaps even more important concern lies within how these scores are interpreted—how they might be used to express the success or failure of an entire system that might have other larger problems.

In Vietnam, for instance, there can be little doubt that their recent ranking was an immense success. Vietnam was ranked 17th overall out of 65 nations, beating many larger industrialized nations.

However, Christian Bodewig of the World Bank has called into question the validity of such scores. He argues that there is other relevant data that PISA largely ignores.

Bodewig says that while many of the students participating in the PISA evaluations did perform well, their performance is not a perfect reflection of the state of education in a given nation. The primary reason for this is that enrollment numbers between nations vary enormously and, in poorer nations in particular, this sort of tabulation can be misleading.

In the case of Vietnam, only some 65 percent of school age children are actually enrolled in school. Compare that with the nearly 90% enrollment rate of the US and the picture of Vietnamese education becomes a bit fuzzier.

The Economist reports that the problems for Vietnamese education are legion, ranging from corruption to homogeneity.

So, what do PISA rankings actually tell us?

Professor Svend Kreiner from the University of Copenhagen in Demark, argues that they don’t tell us that much. In fact, his analysis of the PISA testing model suggests that rankings are largely arbitrary and based on what amounts to luck of the draw.

Depending on which questions a particular set of students receive, their global ranking can fluctuate dramatically.

Still, the Organization for Economic Co-operation and Development (OECD) stands by their methodology as the best and most accurate measure of global scholastic achievement available.

It is also clear that participating nations continue to see the value in PISA. Despite its flaws, PISA still helps nations make decisions with regard to the robustness of their systems of education—even if it doesn’t paint a complete picture.

– M. Chase

Sources: The Economist, Sjoberg, Tes Connect, OECD
Photo: Vintage 3D