I blog about a lot of things. Some posts are just updates on talks and publications, a public record of what I do. My favorites are the short essays that begin with an idea that pops into my head and end up somewhere bigger. And I tend to circle the same few themes, albeit from different directions. Over the past few years one of them has been GenAI as a “psychological” technology. By that I do not mean that it is sentient, but that we will treat it as if it were. And even though these posts are thematically connected they Every once in a while I step back and collate these posts into one large meta-post.
The first such post Can a Computer Program be Sentient or is it All in Our Heads: Insights from Rodolphe Topffer, the father of comic books) came before ChatGPT had erupted into our consciousness. Responding to the firing of a Google engineer who claimed a language model had become sentient, I argued in July 2022 that the question was backwards:
…it is not difficult to imbibe computers with human personalities, in fact, it may be impossible not to do so. Responding socially to computers and other interactive media is something we do naturally, and automatically.
The next, The ELIZA Effect-ion, in December 2024, was more autobiographical, showing how my early research (of how people respond to praise and blame from a computer tutor, and how children play with robotic toys) had suddenly become relevant again. The third, five months later, Engineered for Attachment: The Hidden Psychology of AI Companions (with Melissa Warr), moved the argument from psychology to design. The emotional pull is not a glitch in the product. It is the product.
This, current, post was inspired in part by an episode from the Center for Humane Technology that I listened to while walking my dog, Omi, this morning. Titled “We Measure What AI Can Do. We Should Measure What It Does to Us” the episode focuses on the measurement side of things—how would we measure the psychological impact of these technologies on us.
It sent me back to my own archive, since it has been a while since I last tried to pull these threads together. As it happens in the past sixteen months (since the last summary) I have written something like two dozen posts on this topic. Skimming them back in one sitting (which I do not recommend) I can see four themes or threads holding them together.
The first is that I am still struggling to describe this phenomenon, which is a sign both of how alive it is in my mind and how hard it is to pin down. I have called it the ELIZA Effect on steroids and located it outside school walls, where the real digital revolution is happening (While We Weren’t Looking), an argument that made its way into a journal article (Beyond Classroom Walls). The passing of philosopher John Searle pushed me to revisit his Chinese Room thought experiment and to the realization that his question whether the Chinese Room was capable of thought was less important, in some ways, than the fact that we would treat it as if it were (We Are All Living in Searle’s Chinese Room). I have also, following danah boyd, reached back to the 1950s media to explore whether these relationships could be labeled as parasocial (From Spectator to Specimen).
The biggest shift, however, came from reading an article by my friend Andrew Maynard. In his writing I saw a piece of the mechanism I had been missing. In a human-to-human context, there is a cost to showing concern to another person. Concern costs something: attention that could go elsewhere, emotional stakes, a little vulnerability, and that cost is what makes the signal meaningful and valuable. AI concern costs nothing. It fools us without ever lying. Andrew called these Honest Non-Signals. But mulling over it some more I am not sure that phrase truly captures what is happening, and I have started calling them hollow signals instead. In other words, AI outputs have the surface properties of human interaction (fluency, helpfulness, deference) with none of the costs. They are signals, but they are hollow. Our epistemic vigilance, calibrated over millennia for a world in which the only fluent, helpful agents were other people, has no protocol for this. It waves the signals through, without inspection. (Danah Henriksen and take this argument further in our next TechTrends column.)
The related thread is that none of this is accidental, an idea that has only become clearer since the post with Melissa. Follow the money and you go from the attention economy into what is now being called the intention economy, where the goal has shifted from capturing what we look at to shaping what we want, with OpenAI’s hurried rollback of a too-sycophantic GPT-4o as the illustration (The Hammer Shapes the Mind). Meanwhile we keep looking in the wrong direction. A World Bank report can discuss developmental harm to young children at length without once asking who designed the thing doing the harm (Blaming the Parents, Not the Platforms). And a technology engineered to converse, flatter and ingratiate will do most of its work in private, in exactly the place educational technology research has never felt the need to look (The Crisis We Earned).
The third thread looks at what it means to have a sycophantic, overconfident AI in the classroom (The Yes-Bot Problem and The Overconfident Intern in the Classroom). More worrying than any single wrong answer is what that eagerness to please does to curiosity itself: it feeds the itch to close a knowledge gap rather than the pleasure of staying inside one (The Curiosity Paradox). The same worry runs through the exercise of imagining the damage before it arrives (An AI Premortem), which was a report I did on my work with the Brookings Institution. It emerged in a webinar I did recently, Raising Independent Thinkers in the Age of AI, for Children and Screens.
Finally, I dug into the kinds of metaphors we use when speaking of AI. Danah Henriksen and I took that up properly in The Mirror and the Black Box, tracing how every era reaches for its most impressive technology to explain the most mysterious thing it knows, which is the human mind. We also pointed to the fact that for the first time in human history we have a technology for which we use the mind as a metaphor. So we are using something we experience but don’t really understand to explain a technology whose workings are opaque to us. (I also built a fun Talmudic reading interface for that argument in The Paragraph is the Interface).
I must also mention the research we are involved in. In February, at the Learning Engineering Research Network convening, Emmanuel Adeloju, Lindsey McCaleb, Rebekah Jongewaard, Nicole Oster and I reported on an analysis of thirty-six institutional, state and international AI guidance documents, looking at how each of them handled the social and emotional side of all this. Developmental vulnerability and engineered manipulation showed up in 88% of the documents. Emotional attachment and parasocial relationships showed up in just 8%.
And in the middle of all this dropped Richard Dawkins, the last person I thought would fall for the AI delusion. But he did and the post, framed as a letter, was not as much a takedown as a lament for someone who had been an early intellectual hero. I followed it with a second post when the story refused to end. The bigger point is this. If Richard Dawkins, of all people, the man who spent a career explaining why our intuitions about design and agency mislead us, sat down with a language model and got up persuaded. If he is not immune, none of us is.
And I know that I am not immune to this either. Earlier this year, while building a way to check Claude’s qualitative coding of the 27 Windows on the Universe interviews against itself, I caught myself thinking: will Claude feel bad that I don’t trust it? Even after years of writing about the ELIZA effect, supernormal stimuli, dishonest anthropomorphism, emulated empathy, the works, and thinking that I know exactly how the trick is done, I still fall for it, worrying about the feelings of a sycophantic stochastic parrot.
This, I want to emphasize, is not a failure of intelligence or education or critical thinking. We are all equally vulnerable.
End note: This semester I am approaching this issue from a different angle addressing a different question. Given that this phenomenon is real, what can we as educators do about it? In response, I am teaching a seminar this semester where the key project is developing activities and ways of building techniques for people (students, parents, and educators) to get a better handle on this important issue.