Essay: Designing AI responsibly for youth starts with systems design
This essay, by doctoral student Emmanuel Adeloju, is part of an occasional series about how faculty, staff and students at ASU’s Mary Lou Fulton College for Teaching and Learning Innovation are exploring and integrating AI through research and practice.
______________________________________________________________________________
For most children, an AI chatbot is no longer a novelty. It helps with homework, answers questions at midnight, and talks back like a friend would. What that friendliness does to a developing mind, and whether anyone designing these tools is asking the same question, is what our team at Arizona State University has set out to study.
We are examining how children's interactions with generative AI may influence their social, emotional, and cognitive development, and what those findings could mean for the future design of AI systems intended for young users.
As a doctoral candidate in the Mary Lou Fulton College for Teaching and Learning Innovation, I recently had the opportunity to represent our team at the 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition, one of the largest AI research gatherings in the world. I presented our work at the conference's Humans of Generative AI workshop and shared our findings with an audience of computer vision and AI researchers, bringing an educational perspective to a space where it remains uncommon.
A risk that hides in the design
Our research team is composed of doctoral students in MLFC’s Learning, Literacies and Technologies doctoral program: Lindsey McCaleb, Rebekah Jongewaard, Nicole Oster, and myself. We are advised by MLFC Professor Punya Mishra, who is also director of innovative learning futures at ASU's Learning Engineering Institute..
The workshop was organized to confront a tension at the heart of how AI gets made: that systems are often judged by clean, in-lab evaluations that bear little resemblance to how people actually use them, which is a mismatch that can waste effort at best and cause real harm at worst. As such, the organizers pulled two communities into the same conversation: the AI researchers who build these tools and the human-centered researchers who study their effects.
Through our presentation — From guidance to design: Uncovering socioemotional risks in generative AI systems for education — we pointed out that most of the attention paid to AI safety is on what a model produces, such as hallucination, bias, toxic and misleading output. Those challenges are real and are actively being addressed. But many of the risks that matter most for children surface in quieter places, such as in how a system is designed to interact, where choices about tone and behavior can draw a young user into a kind of bond with the tool. A chatbot can simulate intimacy, encourage a child to lean on it emotionally, subtly reinforce their behavior, and gradually stand in for the human relationships and self-reliance that growing up depends on. This is what we propose also needs attention from the technical community.
What the documents revealed
Hence, we wanted to know whether interaction risks of that kind are even recognized in the guidance meant to govern AI in schools, and what the technical community might learn.
To find out, we focused our research on analyzing a large body of guidance documents on generative AI in K-12 settings, spanning the U.S. Department of Education, school districts in states across the country, and a range of local and international organizations.
It appears that most of the harms that educational AI guidance focus on are the ones the technical community is already addressing and talking about, such as hallucination, bias, and overreliance. The subtler interaction-level risks, such as relational risks, a child forming a one-sided attachment to a chatbot or slowly losing the habit of turning to people, appear far less often. We argue that most guidance are reactive rather than proactive, since they address what these AI systems are already designed to do. Protecting children, therefore, has to begin while these systems are being built, and not after something has gone wrong.
______________________________________________________________________________
Essay by Emmanuel Adeloju, a doctoral candidate with MLFC’s Learning, Literacies and Technologies program. He is also a recipient of the CVPR 2026 Broadening Participation Scholarship Award,
First-person contributions featured in MLFC News reflect the viewpoints and research of individual authors and their teams.