Update September 20, 2024): This article is now published and can be found here:
Warr, M., Oster, N. J., & Isaac, R. (2024). Implicit bias in large language models: Experimental proof and implications for education. Journal of Research on Technology in Education, 1–24. https://doi.org/10.1080/15391523.2024.2395295

I don’t usually post about articles written by other people (however much I may like the study or the authors) but I am making an exception this time – mainly because I believe that this is a critically important piece of research that deserves wider recognition.

In short, this study conducted by Melissa Warr (New Mexico State University), Nicole Oster (Arizona State University) and Roger Isaac (New Mexico State University) provides clear and deeply worrisome evidence that large language models like ChatGPT exhibit racial bias, even when efforts are made to prevent overt discrimination. Melissa and team gave ChatGPT a pedagogically authentic task of evaluating student writing (something we know teachers ARE using the AI for). Essentially they gave identical student writing samples to GPT to evaluate but varied the demographic information provided about the student’s race, class background, and school type. They found that while scores didn’t differ significantly for students labeled “Black” or “White,” there were substantial gaps between students labeled as low-income or attending public schools versus affluent students at elite private schools.

Think about this for a second. When race was explicitly mentioned there was no difference in scores. But just in case you think that these systems are not biased, think again. The bias came roaring back when co-relates of race were used to describe the student.

This is extremely disturbing, since this is a more insidious form of bias. Just as in humans, explicit bias is something we can deal with, but implicit bias is harder to identify and confront.

In other words, these large language models are biased in ways that reflect systemic inequities in society at large – but not explicitly. Though OpenAI has tried to create guardrails against explicit bias, it is relatively easy to uncover these biases. Thus, current “guardrails” against discrimination are limited in their capability to reduce these biases.

These results must give us pause, particularly at a time when we are seeing a significant increase in the use of AI by teachers and other educators. As Melissa and team point out the indiscriminate use of these tools risks exacerbating achievement gaps and harming students from marginalized communities.

Finally, there are some interesting sequence effects that Melissa and team discovered that are difficult to interpret. That is too much to get into in this post (you should read the entire article) but briefly, what it does indicate is that ChatGPT, and other large language models, have “complex alien psychologies” that can be probed and interrogated in ways similar to how we conduct experiments on human cognition. The fact that these alien intelligences can be systematically studied is another important lesson to draw from this groundbreaking work.

The study is currently under review but is available on the Social Science Research Network (SSRN) pre-print archive. Complete citation and abstract given below

Warr, Melissa and Oster, Nicole Jakubczyk and Isaac, Roger, Implicit Bias in Large Language Models: Experimental Proof and Implications for Education (November 6, 2023). Available at SSRN: https://ssrn.com/abstract=4625078 or http://dx.doi.org/10.2139/ssrn.4625078

Abstract: We provide experimental evidence of implicit racial bias in a large language model (specifically ChatGPT) in the context of an authentic educational task and discuss implications for the use of these tools in educational contexts. Specifically, we presented ChatGPT with identical student writing passages alongside various descriptions of student demographics, include race, socioeconomic status, and school type. Results indicated that when directly questioned about race, the model produced higher overall scores than responses to a control prompt, but scores given to student descriptors of Black and White were not significantly different. However, this result belied a subtler form of prejudice that was statistically significant when racial indicators were implied rather than explicitly stated. Additionally, our investigation uncovered subtle sequence effects that suggest the model is attempting to infer user intentions and adapt responses accordingly. The evidence indicates that despite the implementation of guardrails by developers, biases are profoundly embedded in LLMs, reflective of both the training data and societal biases at large. While overt biases can be addressed to some extent, the more ingrained implicit biases present a greater challenge for the application of these technologies in education. It is critical to develop an understanding of the bias embedded in these models and how this bias presents itself in educational contexts before using LLMs to develop personalized learning tools.