Two recent studies about AI and education have been receiving enormous attention—MIT’s “Your Brain on ChatGPT” and the World Bank’s “Chalkboards to Chatbots.” Initially, I wanted to write a detailed post dissecting both studies, but I held back for two reasons: first, plenty of others have already weighed in, often taking predictable sides either championing or condemning the research; and second, I got something even better—the chance to explore these studies in conversation with my colleagues through our AIR | GPT podcast.
AIR | GPT is a podcast that brings together Caroline Fell Kurban, Liz Kolb, Ruben Puentedura, Helen Crompton, and myself for monthly conversations about AI and education. Our discussions are orchestrated by Emmy Award-winning executive producer Errol St.Clair Smith. The premise remains the same: we meet in a virtual airport lounge each month for unscripted, candid conversations about where AI and education are heading.
In our latest episode, “All Gen AI Studies Are Not Created Equal: What These Two Highly Cited Studies Overlooked, And Why It Matters,” we conducted an autopsy on these two papers that have become lightning rods in education circles.
What struck me most about the response to these studies is how they function as Rorschach tests—those inkblot psychological assessments where what people see reveals more about their own mindset than about the actual image. The MIT study, which suggests ChatGPT use leads to cognitive decline, has been enthusiastically embraced by AI skeptics as validation of their concerns about technology diminishing human capability. Meanwhile, AI advocates have seized on the World Bank study’s claims of dramatic learning gains as proof that generative AI is transformational for education.
In short, if you loved one study you hated the other. Fact for one, fake for the other. And vice-versa.
But here’s the thing: both camps are often overlooking the fundamental methodological problems in favor of confirmation bias. The MIT study had participants write 20-minute essays while often just copying and pasting from ChatGPT—of course they couldn’t recall content they never actually processed. The World Bank study lacked a true control group, mixing AI intervention with additional computer time, teacher attention, and after-school programming, making it impossible to isolate what actually drove the learning gains. At the end of the day it is nothing but a “time on task” study, where those who got more time and support did better than those who did not. Unless controlled for, AI is irrelevant to the discussion.
These reactions tell us more about our pre-existing beliefs about AI in education than about what the research actually demonstrates. Those predisposed to fear AI’s impact on learning gravitate toward the MIT findings while dismissing obvious flaws. Those excited about AI’s potential celebrate the World Bank results while glossing over serious design issues.
What was fun in the discussion was that we went deeper not just into this issue but a range of others, not surprising, given the talented co-hosts I get to hang out with.
You can listen to the episode by following the link above – or all the episodes by following the links below. You can find AIR | GPT on most major podcast platforms.
Below is a list of episodes of AIR | GPT
- The Creators of SAMR, TPACK, Triple E, SETI, and GenAI-U Launch New Podcast With a Review of NoteBook LM
- Para-social Relationships: The Pros and Cons of Falling in Love With Generative AI
- 2024 Mic Drop on the AIR GPT Podcast
- AI Schools Are Catching On: What They Get Wrong, What They Get Right, What They Are Teaching Us About Teaching
- Some Think More Deeply With Gen AI, Some Less: Here’s What’s Behind The Emerging Critical Thinking Divide
- The Science and Art of Teaching Creativity in the Age of Gen AI: What Every Educator Needs to Understand
- The Blindspots in AI Guardrails, Guidelines, and Good Intentions: What We Need, What We Don’t, and Why
- All Gen AI Studies Are Not Created Equal: What These Two Highly Cited Studies Overlooked, Why It Matters