Those of us who work in and around artificial intelligence often exist in something of a bubble. We talk about vibe coding and hallucination rates as if these concepts are common knowledge. I have often wondered about how much the broader public understands about how these systems work?

I came across a recent survey conducted by the Searchlight Institute and Tavern Research that sheds light on exactly this question. While their published analysis focused on attitudes toward AI they made their full dataset publicly available, that allowed me to jump in and conduct my own analysis. Just to give some context, their August 2025 survey polled 2,301 American adults, weighted to be nationally representative, with a margin of error of ±3.2 percentage points. About 30% of respondents use AI chatbots at least sometimes, while 42% have never touched one. (More about all that here).

1. Turns out most people know what chatbots do..

A full 86% correctly identified that these tools “write and answer questions with text.” Only small fractions chose incorrect answers: 5% thought chatbots mainly play video games, 5% said they control robots, and 4% believed they manage spreadsheets. That said, this accuracy held relatively steady across demographic groups, with some minor deviations, suggesting that the basic concept of a text-based AI assistant has penetrated public awareness.

• How do people think AI learns?

Understanding drops considerably when we move beyond “what” to “how.” When asked how AI systems learn to write, a whopping 61% of people got it wrong, with only 39% selecting the most accurate answer: that AI “read huge amounts of text from books, websites, and articles.” A third of respondents (33%) believed that “programmers wrote down rules for how to write” while  another 23% thought “people typed in answers to millions of questions.”

Education and AI usage mattered. College graduates and frequent AI users were more likely to select the correct answer compared to those without degrees or non-users. Age mattered with older people defaulting to the “programmers wrote rules” while Gen Z respondents were most likely to answer correctly.

• Oops! What Happens When You Ask a Question

Here’s where things get concerning. When asked what AI is actually doing when it responds to a prompt, a whopping 72% got it wrong, with only 28% correctly identifying that the system is “guessing what words should come next based on patterns it learned.” Nearly half of respondents (45%) incorrectly believed AI is “looking up the exact answer in a database,” suggesting they conceptualize chatbots more like search engines than generative systems. Another 21% thought AI follows “a script of prewritten responses,” and 6% believed a human in the background writes each answer, a misconception that was pretty consistent across demographics. Even among regular AI users, only 27% answered correctly, while 44% believed in the database lookup model.

It seems to me that this is a critical issue. If you think AI is “looking up” verified facts rather than generating probabilistic text, you’re likely to place unwarranted trust in its outputs. This misconception may be at the root of many problems with AI misinformation.

• Ooops 2! Half the Public Doesn’t Know AI Can Make Things Up

Only half the public (51%) understands that generative AI systems just make stuff up—which means the other half does not. In fact, when asked about hallucinations, 21% interpreted the term literally, thinking it means “the AI creates images in its mind,” while 18% guessed it refers to the system getting “overwhelmed by too many questions.” College-educated respondents did better than those without degrees and white-collar workers outperformed blue-collar and service workers.

Here is a surprising finding: non-users were slightly more likely than users (52% to 49%) in knowing about hallucinations. Small gap but interesting. Still, the fundamental issue remains that users who don’t grasp that AI can confidently fabricate information may be more vulnerable to accepting false outputs as fact.

Taken together, these findings paint a portrait of a public that grasps AI at a surface level but holds fundamental misconceptions about how these systems work under the hood. Few people understand that chatbot responses emerge from statistical pattern-matching rather than knowledge retrieval or programmed rules. And then there is the persistent “database lookup” misconception which may lead users to place unwarranted trust in AI outputs, assuming the system is retrieving verified facts rather than generating probabilistic text.

We can’t expect everyone to understand the technical details of attention mechanisms or tokenization (and truth be told, I am not entirely sure I fully get that either). But understanding that AI generates rather than retrieves, and that it can confidently produce nonsense, seems a pretty basic bar to cross for the AI age. Clearly we are not there yet.