This is the second in a series of posts about the human side of science, based on interviews with 27 leading cosmologists. The first post told the story of how these transcripts came to exist. This one describes how they were analyzed.

How the analysis was done

I should confess something upfront. Everywhere I go, I’m known as the TPACK guy. It’s a framework I developed for understanding what teachers need to know to intelligently integrate technology in their teaching, and it has followed me, for better or worse, for two decades. I mention this because when I sat down with generative AI to analyze 27 cosmologist interviews about the nature of scientific experience, the AI kept trying to code passages as examples of TPACK.

TPACK has absolutely nothing to do with cosmologists talking about beauty and wonder. But the AI had read enough of my published work to associate me with that framework, and it kept importing it, confidently, into an analysis where it did not belong. Each time, I had to stop the conversation and say: no, that’s not what’s happening here.

This is what working with generative AI is actually like. I’ve taken to describing it as a smart, drunk, sycophantic, biased intern. It has genuine capabilities that extend what I can do but it also makes errors that require constant monitoring. The excitement is real, and so are the limitations. Given that here is what the AI made possible, and my role in that process.

The transcripts had been waiting in a Dropbox folder for twelve years. Earlier this year, I’d been using Claude (Anthropic’s AI) to analyze 2.6 million words of transcripts from the Silver Lining for Learning webinar series. That experience showed me what these tools could actually do with a large corpus. And then I remembered what else I had sitting around.

I had 27 interviews with some of the greatest cosmologists of the late 20^th century, totaling over 300,000 words. I designed a split-sample analysis: Claude read nine interviews with no framework at all, just looking for what stood out. Then, in a fresh session with no memory of the first, it read nine completely different interviews and developed themes from scratch. The themes that emerged independently from both batches were the ones I could trust. A third batch of nine tested whether the framework held. Then the whole corpus was coded against the final set of twelve themes.

Why this elaborate architecture? Because the whole point was to not lead the witness. If you hand an AI your entire dataset and say “find themes,” you get themes — but you have no way of knowing whether they’re robust or artifacts of the model’s pattern-matching habits. The split-sample design forces the themes to earn their place. If the same pattern shows up independently in two separate batches, read by two separate sessions with no shared memory, it’s probably real.

No individual researcher could do this. Not because the reading is hard, but because the design requires genuine independence between passes. A human who has read the first nine interviews cannot unread them. The themes from Batch A will inevitably color what you see in Batch B. The AI could start fresh each time, which meant the cross-validation was real, not performative. I designed the method, chose the batches, made the interpretive calls, pushed back when the AI got things wrong. The AI held 300,000 words in its head and executed an analytical architecture that would have taken a research team months.

But the TPACK episode is a reminder: AI makes errors, and some of them are confident and plausible enough to slip past you if you’re not paying attention. The safeguard isn’t trust. It’s transparency.

I’m calling this approach auditable dialogic inquiry with AI. Each word matters. Dialogic because the understanding emerged through conversation, not from either party alone. Inquiry because it goes beyond coding: it includes framework construction, testing, interpretive challenge, and revision. With AI because naming the interlocutor honestly matters. And auditable because the entire conversation is preserved. Every proposal, every correction, every moment where I said “that contradicts what I know about this data” and the analysis changed course. When the eventual paper is published, the full dialogue will be made public, as I’ve done with the Silver Lining for Learning analysis. Anyone will be able to replay the interpretive process, not just read the conclusions.

The TPACK incident was one kind of error: the AI importing a framework from my other work where it didn’t belong. But the surprises went in the other direction too. The single strongest theme to emerge from the data, scoring higher than any other across all 27 interviews, was cognitive diversity: the discovery that these scientists think in fundamentally different ways. Some visualize in multiple dimensions. Some work entirely in equations. Some think through physical analogies or historical narratives. And most of them have no idea the others are different.

I had just been writing about cognitive diversity. Two blog posts, published months before this analysis, arguing that genuine differences in how minds work are real and consequential, even though the old “learning styles” matching hypothesis was rightly debunked. And then, from a completely different dataset, a different population, a different era, the same finding surfaces as the most robust pattern in the corpus. I didn’t go looking for it. I couldn’t have. The design required open-ended discovery, not hypothesis testing. It found me.

That’s the feeling I want to convey about this process. Not “AI did my research for me.” Not “I used a chatbot as a coding tool.” Something more like what James Gunn, one of the great instrument-builders among these 27 cosmologists, describes when he talks about building his own telescopes: the instrument extends what you can see, but you have to know its limitations intimately, because that knowledge is what tells you what to believe. Gunn trusted his observations precisely because he’d ground the mirrors himself. I trust this analysis precisely because I was in the conversation the whole time, catching the errors, redirecting the inquiry, and watching the patterns emerge from data I’d been carrying around for twelve years. And just in case I’m fooling myself — all of it, the full dialogue, every prompt and response, is available for anyone to review, question, and test.

Lightman and his interviews gave us twenty-seven windows on the universe. This technology gave me one more. The next five posts are what I saw through them, starting with where science begins: in wonder. Coming soon…