The rise of AI promises new solutions to long-standing challenges. It also introduces some challenges of its own. In addition to concerns over privacy, bias and reliability, AI is driving a flood of new products in a broad range of sectors, including education. As options pile up, districts and schools struggle to identify effective solutions amid clever marketing and bold promises.

As a member of the LEARN Network, a federally funded initiative dedicated to supporting the development and scaling of quality educational products and programs, I've collaborated with researchers, developers, practitioners and educational leaders from across the country. Over the years, our team has gained unique insights into why some products succeed, why others fail, and what districts and schools may consider when selecting new tools and programs.

Effective edtech has never — and should never — be designed to replace human relationships with students.

One lesson we’ve learned is that the current wave of AI-powered edtech is not all that different from the products and programs we are used to. Some products show promise; others fall flat. Though options are more plentiful and technology advanced, schools must remain diligent in their selection processes. Based on our work and our conversations with leaders in this space, here are some important questions to ask while searching for an edtech solution in the age of AI.

What Does It Do?

Effective edtech has never — and should never — be designed to replace human relationships with students. In response to the rise of school-based AI programs, policymakers in states like California and Minnesota and organizations like the National Education Association are pushing to ensure that educators remain at the center of education. Quality edtech, whether powered by AI or not, should work to enhance educational effectiveness and efficiency.

One key differentiator decision-makers may consider is between student-facing AI, which students interact with directly, and products and programs designed for practitioners, administrators and other staff. Both uses require unique considerations. For example, for student-facing products, it is essential that developers use guardrails to prevent bias, protect privacy, and ensure reliability. For administrative applications, considerations will likely focus more on whether the edtech increases efficiency while leveraging the expertise of humans.

Does It Have a Solid Evidence Base?

The most critical factor in selecting edtech is its evidence base. Is there research to back its claims? If so, how reliable is that research? As we’ve found in our work, these can be difficult questions to answer.

Edtech only works if it can be implemented effectively. Products can fail if they are too cumbersome, don't fit into staff workflows or don't align with existing programs.

The Every Student Succeeds Act’s (ESSA) tiers of evidence can provide a useful framework for evaluating edtech, describing the varying degrees of research that can underlie a product. Simply meeting an ESSA tier of evidence, however, does not guarantee effectiveness. Products or programs that meet ESSA’s lowest tier, for example, may only be based on evidence-backed strategies or practices. The products themselves may never have been tested. Researchers Mary Bratsch-Hines and Heather Aiken, leaders behind the TRI-Reading App, expanded on the importance of comprehensive evaluation in a recent episode of The SRI Homeroom podcast.

“Sometimes people can claim that they are following the science of reading purely because they are covering the five elements from the National Reading Panel… But how they packaged it together, we don’t necessarily know that the program as a whole will work.” — Mary Bratsch-Hines, Senior Manager for Research and Evaluation at the University of Florida Lastinger Center for Learning.

Edtech that meets ESSA’s highest tiers of evidence, Tiers 1 and 2, have been more rigorously evaluated, with findings cited in repositories like the What Works Clearinghouse.

However, a lack of evidence does not necessarily mean a product is ineffective. Some edtech, particularly in the age of AI, may be too new to boast an extensive research base. In these instances, products should at least present a compelling case and meet a priority need.

Schools may also consider compiling their own body of evidence. Examine developer briefs, speak with peers in similar contexts and, if you decide to move forward, conduct a pilot to generate real, localized evidence. H. Alix Gallagher, director of strategic partnerships for Policy Analysis for California Education (PACE), recently wrote about the piloting process for the LEARN Network blog.

Does It Really Meet Your Needs?

While evidence is crucial, it’s not the only factor that can influence the success or failure of a new product or program. Schools and districts should also carefully consider their goals. Is there a specific challenge you are attempting to address or a set of outcomes you hope to improve? Take some time to investigate your needs, speak with team members and ensure you fully understand the scope of your challenges and their root causes.

Some products claim to offer broad, generalized benefits and improvements for schools, faculty and students. In our conversations with educational leaders, however, we've found that teachers are often unlikely to adopt solutions to problems if they don't perceive them as significant. Ensuring that the technology addresses a recognized need is critical for successful implementation.

Does It Fit Your School Context?

Edtech should be designed to meet the needs of all students, fostering growth and equity through education. When products or programs aren't relevant or accessible to members of the communities they serve, they can inadvertently cause harm.

Edtech only works if it can be implemented effectively. Products can fail if they are too cumbersome, don't fit into staff workflows or don't align with existing programs. Examine your current systems, staff capabilities and capacity to determine if a product is a proper fit.

For example, the developers of A2i, an impactful, broadly scaled tool designed to improve literacy outcomes for young students, partly attributed its success to its integration with various learning management systems. Implementation can suffer when a product requires a radical departure from established processes, duties or expectations.

Districts and schools may also consider the unique needs of their community. Are there any structural, cultural or environmental factors that might limit some members’ access to a new product or limit its effectiveness for the community as a whole? Ensuring that the product fits your specific environment is crucial for its success.

How Was the Product Developed?

Effective edtech often results from extensive discussions, collaborations, revisions and iterations involving a diverse range of stakeholders. At the LEARN Network, we encourage researchers and developers to involve school and district leaders, educators and community members in the design phase of their work. Products that lack sufficient input can struggle with unforeseen challenges upon implementation.

In our recent webinar focused on rural schools and communities, a panel of researchers, practitioners and educational leaders cautioned against “drive-through approaches” to product design, encouraging developers to employ a more inclusive, community-focused approach to development.

Does It Prioritize Equity?

Edtech should be designed to meet the needs of all students, fostering growth and equity through education. When products or programs aren't relevant or accessible to members of the communities they serve, they can inadvertently cause harm.

The rise of AI-powered edtech offers districts and schools a unique opportunity to seek out new tools and programs that are accessible, equitable, and responsive to the needs of diverse learners. In a recent LEARN Network blog, we spoke with leading voices from the Strategic Education Research Partnership (SERP) Institute and SRI about this opportunity and some potential paths forward.

Free LEARN Network resources for researchers, developers and educators:

• The LEARN to Scale Toolkit: a comprehensive resource for researchers and developers based on the Invent-Apply-Transition framework
• Stories of Scaling: a profile series highlighting impactful researchers, entrepreneurs and evidence-based products
• The LEARN Network Blog: articles, podcasts and Q&As featuring experts and thought leaders from across the U.S.
• LEARN Network Research: action-oriented research briefs focused on product development, procurement and more

An effective method for centering equity in the edtech search process is by elevating student voice. Students are self-aware, highly technologically competent and carry unique experiences and perspectives on learning. By involving students in their piloting and procurement processes, districts and schools can help ensure they’re adopting technology that will serve all members of their community.

Decision makers can also prioritize student voice in their edtech search, considering products that have been designed and developed in collaboration with students. LEARN Network researcher Ela Joshi expanded on the value of student voice in this recent podcast.

Artificial intelligence has the potential to power effective new tools and approaches, reducing burdens on schools, fostering equity and inclusion, and helping students overcome long-standing barriers. As we’re seeing in other sectors, however, the letters “AI” are not always indicative of quality.

In their search for the next generation of edtech solutions, we must all avoid flash and continue focusing on fundamentals. By prioritizing evidence, understanding specific needs, ensuring contextual fit, examining how products are developed, emphasizing equity and including diverse voices in the search and selection processes, educational leaders can navigate the complexities of AI-powered edtech and find products that truly lead to better outcomes.

The information reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305N220012 to SRI International. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.