Designing technology for the sake of argument

Collaborative Research: Diagnosing the Argumentation Levels of Grps. (DiALOG)-A Digital Formative Assessment Tool to Support Oral Argumentation in Middle School Science Classrooms

Principal investigator:
Award amount:
Originating sponsor:
National Science Foundation
Grant start date:
September 1, 2016
Grant end date:
August 31, 2020

The challenge:

Reading and writing are integral to every school’s curriculum. But two other communications skills, speaking and listening, are also essential for students to be able to function effectively in the 21st-century workplace, says researcher Bryan Henderson. “Listeners and speakers need to understand what are their contributions and points of view when collaborating with others,” Henderson says. “We may have ideas in our heads that make perfect sense to us, and hence we may think others will understand that thinking just as we do. But often, what might be clear in our minds is not so clear to others.”

Henderson says those skills are rarely addressed with the structured, systematic consideration given to reading and writing when teachers structure their courses. Speaking and listening are certainly common in the classroom but are difficult to assess. Lack of accurate assessment means these skills are unlikely to become part of today’s curricula, which rely on assessment to establish accountability.

“It follows that if we provide teachers with a valid and reliable assessment of speaking and listening, there will likely be more careful, structured, regular opportunities for speaking and listening critically in classrooms,” Henderson says. “But assessing speaking and listening is messy. These skills are fluid and nonlinear, and if utterances are not recorded, they vanish in an instant.”

The approach:

Henderson designed DiALoG — Diagnosing the Argumentation Levels of Groups — a tablet-based tool that assesses multiple dimensions of argumentative speaking and listening skills in real time. Using a touchscreen, teachers follow DiALoG’s prompts and rate eight statements to describe what they just observed in their classroom. A teacher evaluating a peer-to-peer argument, for example, would be prompted, “Students seemed to construct/refine thinking in ways that explicitly built off the contributions of others.”

By moving a finger back and forth across the screen, the teacher can dial in the degree to which an argument in the classroom agrees with each statement. DiALoG instantly generates a scoring breakdown, and, where scores are low, provides real-time prompting for actions the teacher can take to improve speaking and listening. The instant prompts are also linked to Responsive Mini-Lessons, designed activities that further improve the specific skills DiALoG assesses as needing improvement.

Funding for DiALoG began in fall 2016 as part of a joint proposal with the University of California–Berkeley that totals $3 million in funding. Henderson leads the development of the instrument and the analysis of the study. The four-year cycle of the project first will enlist focus group teachers in Arizona and California to employ the instrument and recommend refinements to the platform and the RMLs. Henderson will then launch a multistate, randomized controlled trial among 100 middle school instructors teaching 1-to-2 week science units, some with and some without the DiALoG instrument. After controlling for the science lessons taught by all participating teachers, Henderson will look for differences in how teachers support classroom argumentation with and without DiALoG, and the types and quality of argumentation that result.

Findings and impact:

The project will address two research questions:

  • How can DiALoG be refined to provide a formative assessment tool for oral argumentation that is reliable, practical and useful in middle school classrooms?
  • How does the use of DiALoG affect teacher formative assessment practices around evidence-based argumentation, when implementing science units designed to support oral argumentation?

The key outcome of this work will be a research-informed and field-tested prototype designed to improve the quality of teaching and learning argumentation in middle school science classrooms, usable in different learning environments.