As headlines swirl about AI chatbots misrepresenting Anne Frank (Schools Using AI Emulation of Anne Frank That Urges Kids Not to Blame Anyone for Holocaust) and Apple canceling its AI news summaries due to accuracy concerns (Apple pulls error-prone AI-generated news summaries in its beta iPhone software), I’m reminded of my own recent encounter with AI-generated misinformation.

Mine is a lighter tale, to be sure, but it points to the same fundamental issue—of these technologies being essentially bullshit generators. And the irony, of course, is that as someone who has been preaching this for a while I should have known better.

It started with a creative exploration of the theme “It Takes Two” that I wrote about in this blog post back in June 2023 (It Takes Two: A scientific romp using AI). I was excited to blend science with poetry, and ChatGPT suggested using a form called “Double Dactyl” – which sounded perfect given the theme of twoness. The AI proceeded to generate some delightful science-themed poems about chromosomes, electrodes, and light waves.

I was thrilled! The science was solid, the verses were clever, and it was a great example of the kinds of transdisciplinary genre-mashing AI was capable of.

I loved these examples so much that I included them in a recent chapter I was co-writing for a book on creativity. It wasn’t until one of the editors of the book gently pointed out that none of these poems were technically double dactyls that I realized that I had not done the requisite homework I should have done.

You see, in my excitement, I completely forgot to verify the poetic form.

A quick check revealed that while double dactyls are indeed a real poetic form, they follow specific rules that ChatGPT cheerfully ignored. (For those curious, real double dactyls require specific rhythmic patterns, nonsense words in the first line, and ideally a six-syllable word in line six – none of which appeared in our creative endeavor.)

The AI, in what I now realize was a delightful bit of pattern-matching gone awry, seems to have simply picked a poetic form with “double” in the name to match our theme of “two.” And I, caught up in the excitement of the whole thing, didn’t bother to verify the form.

But here’s the thing – while this was mildly embarrassing (especially for someone who writes about AI literacy), it’s also a small example of a bigger story. The poems themselves are creative and fun, and the science is accurate. The AI did what AI does best: it mashed-up something novel and interesting, based on the prompt it was given. It just didn’t generate what it claimed to generate.

That said, the AI did some remarkably creative things here. If you read the poems (and you should), you will see that it quite cleverly rhymed all the titles of the poems: “Chromosomally,” “Electrochemically,” “Interferentially,” and “Experimentally.” That’s the kind of pattern recognition that makes AI outputs fascinating, even when they don’t quite follow the rules.

So what is the lesson here?

I think it is simple – just as my experiments with optical illusions and AI and the Anne Frank news story demonstrate – these technologies are not ready some of the tasks and roles we are giving them. Which is why the AI school in Arizona is so troubling.

But if we see these tools for what they are – they can be great fun. They are smart but drunk, biased, and supremely confident interns. And they can be great fun to have around. Just don’t trust them completely. For instance, the poems may not be double dactyls, but they’re still enjoyable pieces of scientific verse — a new way of thinking about parallels across scientific disciplines, as a way of thinking creatively. And that is no mean feat.