Evaluating STEM program effectiveness

It’s a challenge faced in many STEM-related fields: Not enough students of diverse backgrounds go on to pursue studies and careers in science, technology, engineering and math.

For example, marine sciences students who identify as Black or African American make up less than 2% of graduate students in earth, atmospheric and ocean sciences. This low representation, detailed in research published in Oceanography, underscores a need for more inclusive pathways. One such effort is a National Science Foundation-funded project being led by Hampton University and the nonprofit Black In Marine Science to create programs and curricula aimed at addressing that gap. 

But how can such projects be designed to achieve their desired results? 

Ayesha Boyce, associate professor at Arizona State University’s Mary Lou Fulton Teachers College, is a nationally recognized evaluation scholar and practitioner who specializes in the evaluation of STEM initiatives. In addition to leading the evaluation component of the marine science project (Collaborative Research: Building Racial Equity in Marine Science), Boyce has over 16 years of experience conducting evaluations for projects funded by the NSF, Department of Defense, Department of Education, National Institutes of Health, the Centers for Disease Control and Prevention, Spencer Foundation, and the Howard Hughes Medical Institute. 

To meet the growing demand for talent in STEM fields, funders invest millions in projects designed to increase diversity and broaden participation in STEM through education and workforce development programs. Evaluation helps ensure these initiatives are being designed to increase the number of students from underrepresented backgrounds in these subject areas.

Boyce’s work addresses these goals by embedding cultural responsiveness into the evaluation process, noting that the evaluation of such efforts should be asset-based (focusing on the strengths of students), and incorporate equity and inclusion. It involves examining who controls budget decisions, who joins the decision-making process and how resources are allocated. This approach also raises critical questions about language access, respect for local customs and participants' preferred communication styles.

“Program design carries more weight and credibility when we’re truly responsive to the communities we are trying to reach,” Boyce explained. “By listening closely to their needs, respecting their experiences, and involving them in the process, we not only gain richer insights but also build trust—ensuring the results are meaningful and applicable to those they’re intended to benefit.”

Boyce, who teaches courses on program evaluation and research methodology to MLFTC students, says the academic field of evaluation is relatively young compared to some other fields of social inquiry like psychology and sociology. In 1985, the American Evaluation Association was formed to set standardized criteria around assessing the strengths and weaknesses of programs, policies, personnel, products and organizations to improve their effectiveness. Today, most government or philanthropic funded programs require an external evaluation, according to Boyce, who is currently a member of the AEA Board of Directors.

“The integration of evaluator teams is becoming even more important as part of project design since it creates a foundation for effectively tracking project progress and assessing successful outcomes,” said Boyce. “As a result, we are seeing an increasing professionalization of the field and opportunity to prepare students for these roles.” 

Boyce, who serves as campus advisor for the student organization Black Women in STEM at ASU, currently works with doctoral students in MLFTC’s Educational Policy and Evaluation program and undergraduate students in Barrett, The Honors College at ASU as the Co-Director of the STEM Program Evaluation Lab. She and 10 students just returned from presenting at the 2024 AEA conference in Portland, Oregon.

Photo of ASU STEM Program Evaluation Lab members at the 2024 American Evaluation Association Conference (photo credit: From Ayesha Boyce’s social media channels, and republished with her permission.

At ASU, Boyce has secured over one and a half million dollars in funding to evaluate the effectiveness of STEM education initiatives to broaden participation in STEM. One of her projects includes serving as co-Principal Investigator for EvaluATE, the evaluation technical assistance hub for the NSF’s Advanced Technological Education program. As part of this effort, Boyce is developing an evidence-based toolkit to help evaluators assess ATE projects. Designed for evaluators, project leaders, grant specialists and college administrators, this toolkit supports a comprehensive approach to evaluating equity, diversity, and inclusion in STEM program design.

In addition to her work evaluating external programs, such as EvaluATE and the marine science project, Boyce is collaborating with other faculty groups at ASU. Recently, she collaborated with Margarita Pivovarova, MLFTC associate director of doctoral programs and associate professor, to evaluate an after-school tutoring program for elementary and high schools in Tempe. She is also conducting an evaluation for ASU Vice Provost Amalia Pallares, who created the Embark Fellowship program, to provide underrepresented undergraduate students in STEM training and opportunities to present their research. 

As the newly-designated associate director of strategic partnerships for MLFTC’s Division of Educational Leadership and Innovation, Boyce looks forward to extending her evaluation expertise through partnerships within MLFTC, ASU and beyond.

 “We need scientists and STEM professionals from all backgrounds who can bring their knowledge, ideas and perspectives to address the complexities and decisions that lie ahead in pursuing medical innovations, water security, new energy resources, sustainable agriculture, national defense, and exploring space and the seas,” Boyce said. “The contributions of evaluators help ensure that programs don’t just reach people, but empower them to make a lasting impact on the world.”