Measuring the un-measureable
As part of the organizing team of the Inaugural CIES Symposium in Scottsdale, AZ this past November, we were thrilled to continue the debates about Global Learning Metrics (GLMs) at the recent CIES 2017 Conference in Atlanta, GA. CIES 2017 included a number of Presidential Highlighted Sessions. Among these was “Measuring the un-measurable in Global Learning Metrics” featuring four scholars: Karen Mundy (University of Toronto & Global Partnership for Education), Jill Koyama (University of Arizona), Aaron Benavot (University at Albany & UNESCO), and Gustavo Fischman (Arizona State University). Moderated by Iveta Silova (Arizona State University), the participants in this session discussed the merits, possibilities, tensions, and obstacles of GLMs and their influence on education, pedagogical practices, and policy.
Questions such as “What is usually dismissed as too difficult to measure?” and “Should global learning metrics reflect learning outcomes beyond basic numeracy and literacy?” guided the beginning of the debate. Later, Silova challenged the panelists to explain how GLMs could be transparent and explicit in making their underlying epistemologies visible. Silova also asked the panelists to explain their views on whether or not GLMs could originate from the Global South.
Video: Watch the full Presidential Highlighted Session.
In this Presidential Highlighted session, there was neither a consensus about the meaningfulness of current globally linked measures of learning outcomes nor was there a clear vision of how they will evolve in the future. Additionally, it was obvious from both the symposium and the CIES 2017 Presidential Highlighted Session that educational stakeholders disagreed on the purpose of a GLM–for some, it was to be used as a simple tool, helpful for gaining data on student literacy performance, while for others, it had been utilized as an overarching instrument that had been abused to rank countries in ways that did not account for contextual differences.
While the Presidential Highlighted Session at CIES 2017 in Atlanta unsurprisingly concluded without a consensus about GLMs, the four scholars brought attention to ideas to consider when examining the merits and disadvantages of creating and relying upon GLMs. Silova explained in her introduction to the debate that GLMs will continue to expand, and Fischman later expressed concern that GLMs, despite their decontextualized nature and limited capacity to measure what is happening in classrooms, have already driven educational policy and reform efforts. In light of the growth of these metrics and to understand their effects, addressing issues of feasibility is essential.
One of the salient themes in the debate focused on who is involved in the creation of GLMs and how GLMs are and will be applied. While Mundy argued for reinforcing national ownership of metrics, Koyama claimed that linking nations by a single global learning metric was not something that the policy and development communities should be working toward. Perhaps there was some overlap between the two sides, however, when Koyama suggested that because metrics do exist, there should not just be a single one. Rather, diverse and contextualized measures are essential so that the local needs are taken into account, and, for Koyama, these metrics should be applied from the bottom up. Koyama cautioned, however, that the data generated by any metric is not the solution. She also raised the ethical issue of authority when she asked about who gives education stakeholders the power to measure learning outcomes. Koyama problematized the use of nuanced metrics to “rank and diminish some [countries] while elevating others.”
Benavot agreed with Koyama’s argument that metrics should not follow a top-down approach. He concurred with Fischman that metrics should be meaningful and advance pedagogy. However, Benavot observed that the current metrics are not used to inform quality instruction at the classroom level. Instead they are used for policy-making and the distribution of resources. While Mundy argued that GLMs should be used for such distributive justice, Benavot disagreed that a single, cross-nationally linked metric was the correct avenue. He argued that focusing on GLMs distracts from what is happening in classrooms. According to Benavot, if metrics are to be linked, they should be emergent and develop from the local levels, then to the regional and national levels, rather than imposing them from the top down. Despite their dispute about the goal of GLMs, Mundy agreed that metrics should be grounded in national capacity building.
Where does this leave us in the GLM debate? What has been accomplished by holding a symposium dedicated to this topic in November and a follow-up highlighted session four months later? For one, it opened up a dialogue, providing educational stakeholders opportunities to define what they mean when using the term “global learning metric.” It also helped bridge a gap between academia and development to move the conversation in new directions. At the end of the Presidential Highlighted Session, an audience member and Benavot both raised an interesting question that might offer a different angle from which to approach the debate: Is there a universal learning progression that can be defined and measured?