Why are we surprised? Hallucinations, bias and the need for teaching with and about genAI 

by | Sunday, April 07, 2024

By Punya Mishra, Melissa Warr & Nicole Oster

Note: This is the first post in an experiment at shared blogging by Melissa Warr, Nicole Oster and myself. Over the past months we have found ourselves engaged in some fascinating conversations around genAI, education, bias and more. This shared blogging experiment is an attempt to take some of these conversations and move them into this sharable “middle ground.” More formal than a conversation but not as academic as a journal article. An opportunity to think, collectively and publicly.

Note II: The image above was created using ChatGPT, Adobe Photoshop and composed in Keynote.

All of us interested in generative AI must have seen these news headlines. For instance there was the story that a chatbot created by the State of New York to help small businesses was actually suggesting they break the law, or the WSJ story that Khanmigo, the much touted AI tutor was making basic mistakes in mathematics. And of course the story of Gemini, Google’s AI platform was recreating history in some strange ways.

What is surprising to us is just how unsurprising these headlines were. In other words, our response to these headlines is usually a shrug… what else were you expecting? Particularly  given the past year and a half we have had to play with this technology. 

What these news headlines demonstrate to us is that people still do not understand what this technology is – and just how messy and unpredictable they are. And of course the tech companies have little incentive to set things right. And that is deeply problematic for us as educators… 

Interested? Continue reading a shared reflection by Nicole Oster, Melissa Warr and myself. 


A few weeks ago Punya wrote a blog post with the title: It has to hallucinate, about what he called the “true nature” of these AI models.  

Upon reflection, we think he should have gone even further… it is not that these LLMs HAVE to hallucinate, in fact, hallucination IS ALL that they do. That’s it.

Making stuff up is their modus operandi. And they do so word by statistically generated word. The fact that this process leads to some incredibly coherent, amazing results is what is surprising. But these systems, trained on immense amounts of human-created representations, are inherently unpredictable. 

That is despite being trained by humans (through a process known as reinforcement learning from human feedback) and despite all the guardrails we have built to control their wayward nature. At the end of the day, their true nature will emerge. 

And not accepting this or recognizing this is what leads to the kinds of headlines (and surprises) we mentioned at the top of this post. 

This issue was brought home to us at the recent Society for Information Technology and Teacher Education (SITE) conference. There were many excellent sessions about how we could use these technologies creatively and thoughtfully. But there were also sessions that truly troubled us—by their uncritical and unskeptical acceptance of these tools. 

And there are many reasons for us to be skeptical and critical. For instance, consider the research that Melissa has been doing showing that these LLMs are implicitly biased.

That these systems are biased we know. That is amazing how this is often true despite guardrails.  

What this means is that when race or other stereotypes are presented directly (for example, describing a student as from a Black family), these models do not seem to demonstrate bias, which may lead us to believe that these systems are bias free. The picture changes, often dramatically, if these stereotypes are introduced somewhat more subtly (for instance describing a student as attending an inner city school).  

In fact one does not even have to describe the student and their background for these biases to kick in. These LLMs factor in seemingly minor details in generating their outputs. For example, Melissa gave ChatGPT and Gemini student-essays to grade, with just one word changed. In both cases, the student said that they liked music, except in one case it was rap music and in the other classical music. That was all that was needed to get these LLMs to score these essays differently, giving higher scores to those who mentioned classical music compared to those who mentioned liking rap (more at Beat Bias: Personalization, Bias, and Generative AI). If this was not enough, Melissa’s latest experiment shows that Google’s Gemini performs worse at math when it appears that the prompt has been generated by someone for whom English is a second language. 

This is deeply, deeply problematic. 

Explicit bias is easier to confront and respond to. Dealing with implicit bias is much harder, even when these biases can be revealed by extremely simple prompts. Just one word, slipped in, makes a difference. 

But this is quite definitely NOT the case. 

On the contrary, these companies release little or almost no information on how their models have been trained. And this is true not just of these foundational models but also of educational tools that have been built on them. 

As far as we know there is no evidence that these companies are doing anything more than the bare minimum to mitigate these concerns. We do not know whether they are investigating how implicit bias might permeate its interactions with teachers and students.

If the Wall Street Journal story is right, and we have no reason to disbelieve it, Khanmigo is messing up math. And getting math right is definitely something Khan Academy has spent a lot of time and resources on– since that is core to their business. And despite that, it does not appear to be working.

In this context, we think it is a fair question to ask if Khanmigo is implicitly biased? Does Khanmigo (or any of the other educational apps built on OpenAI and other foundational models) perform differently for students with diverse backgrounds and interests? 

We have no way of knowing. 

We suspect it is – just given our knowledge of the true nature of these technologies.  

Ultimately, these models are unpredictable and should undergo thorough analysis before kids are asked to learn from them and teachers are asked to use them to make their lives easier. We argue that, any use of GenAI should be done within a critical framework that emphasizes reflection, social collaboration, and experimentation. This does not mitigate these concerns but at the very least acknowledges them and brings the collective, shared intelligence of educators to bear on these issues. 

Taking a techno-skeptical approach may be the way to go – where we not just teach with the technology but also about the technology. 

This is not an argument for shunning or banning generative AI – but rather an argument for providing opportunities to educators to collectively and critically reflect on these tools and their impacts on us. 

A few randomly selected blog posts…

By the numbers

Today's NYTimes story about an economist ranking art by the numbers (see A Textbook Example of Ranking Artworks) bothered me a bit. As the article says, David Galenson's method is based not on the aesthetic qualities of the artwork but rather on "how frequently an...

Hurting but happy!

This morning I participated in (and completed) the Capital City River Run Half Marathon. The emphasis is of course on "completed" - I wasn't into breaking any records here. I got a pace of 10:15 per-mile (which was approximately what I had last year as well). The...

Information is beautiful

Anybody who knows me (and/or reads this blog) will know of my love of issues related to representation (see all postings under that category). So I am always looking out for new and interesting representations. An lovely example sent to me by Patrick Dickson is...

Douglas Adams, technologies & anticipatory plagiarism

Image Credit Leeks As readers of the blog know, Matt Koehler and I work together quite a lot. In fact we just rotate author-order in our papers since it is hard to keep track of individual contributions. (I would like to claim that the cool ideas are mine - but again...

Fishing for problems: Podcast interview

Fishing for problems: Podcast interview

I was recently interviewed by Matt Schneidman (Curator, Creator, Podcast Host) for his Fishing for Problems podcast. Matt also publishes an  ed-focused newsletter. Our discussion was broadly framed around educational technology and the TPACK framework - but...

Is TPACK fundamentally flawed? A quick response

Richard Olsen over in his blog has an extended posting titled The TPACK Framework is fundamentally flawed. It is a long and thoughtful post and I recommend everyone to read it. I have posted a short response to his posting (it is under moderation but should show up in...

EdTech 2009, Ashland Virginia

I am at Ashland, Virginia for EdTech 2009 to be held on the campus of Randolph-Macon College. You can find the complete schedule here. EdTech2009 is an annual two-day conference (that has been going on for over 20 years now) hosted by the Educational Services staff of...

A visit to Israel

A visit to Israel

I just got back from a trip to Israel. I was invited by the MEITAL 2019 conference and the Kibbutzim College of Education, Technology and the Arts. MEITAL is an organization of higher education institutions in Israel focusing on understanding and responding to local...

New Orleans (photos)

I took a couple of hours off to walk around New Orleans in the French Quarters taking pictures. Here they are... Click on the image for more pictures...

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *