Qualitative data can be extremely rich, complex and insightful, allowing us to uncover the complexities of human experiences, perceptions, and interactions.
It can also be complicated and messy. And the way we make sense of it is through the one-two punch of thematic coding and inter-rater reliability. Thematic coding allows us to systematically identify key patterns and themes in the data, and allows us to uncover crucial insights that shape our understanding of the topic at hand. Thematic coding, however, can be subjective. One way of minimizing subjectivity is inter-rater reliability. It’s like a reality check for our coding, making sure we’re not letting our own biases cloud our judgement.
More specifically, Inter-rater reliability is a measure of agreement between two or more raters who are coding or rating the same qualitative data. It is often reported as Cohen’s Kappa, a statistical measure that compares the level of agreement between two raters to the level of agreement expected by chance. It can also be reported more simply by just as a percentage agreement.
Now most qualitative research is done by humans. But with the rise of these larger language models a fair question to ask is:
What role can generative AI play in qualitative research?
So I tried out this little experiment, with an interview that we had conducted with Chris Dede. This was the same interview that I had written about earlier, the one that ended up as both an article and a podcast episode. More details about both with links can be found at: A chat about GPT3 and other forms of alien intelligence
As a test I took the transcript of our conversation with Chris and dumped it into anysummary.app, a website that summarizes websites, articles and more. I then created a custom prompt, essentially saying
Take this interview with Chris Dede. Identify 5 key themes in the interview, explain each of them in 3-5 sentences, and, finally select some quotes from the interview that exemplify these themes.
Below is what it came up with. And I do think this is pretty good. Again this is not coding / thematic analysis at a very fine-grained level, but for a first pass, from a freemium website (I just used the free version) it is not too bad, especially if you compare it to the article we wrote based on our interview.
This makes me wonder, just how these AI tools are going to change qualitative research? Should we be ready to have qualitative data coded by AI coders? How soon before we see journal articles reporting Cohen’s Kappa for data that has been coded by humans AND AI systems?
I am pretty sure the day is not too far off.
Anyway here are the themes, explanations and quotes that the AI system came up with from the Chris Dede Interview:
- Definition of Intelligence: The transcript explores multiple definitions of intelligence, including whether AI possesses intelligence and what constitutes as strong intelligence. Chris Dede notes that “if you told Goldilocks and the Three Bears to generative ai, it would probably now understand in contrast to the 1980s that this is not a factual story” and that “knowledge, information processing, all of those things are deeply embodied in our bodies in the broader cultural matrix, in the broader social matrix.”
- Generative AI: The transcript discusses the concept of generative AI, which is AI that synthesizes things from available resources, resulting in performances that were previously thought to be limited to human beings. Dede explains that “AI does not have that kind of intelligence” and that “generative AI would be completely lost and at sea” without the necessary resources.
- Education and AI: The transcript explores the role of AI in education, including developing new educational systems and policies that support transformation. Dede notes that “it’s an exciting time to be in the era of generative ai, which is a major advance in the field” and that “his interests are in developing new types of educational systems to meet the opportunities and challenges of the 21st century.”
- Science Fiction: The transcript references science fiction as a source of nuanced discussions on intelligence augmentation and AI-human combinations. Dede notes that “the really nuanced discussions of intelligence augmentation IA have come out of science fiction” and that “if you look at Star Trek, the next Generation, you’ve got data, the Android, who is in fact an AI-based machine, and you’ve got the card, the captain, who is the wise human being.”
- Hype Cycles of AI: The transcript acknowledges the cyclical nature of hype surrounding AI and the disappointment that can come with it. Dede notes that he has “lived through many cycles of advances in AI and been impressed by the progress” and that “we rarely never stick to the script” when it comes to utilizing AI in education.
Extended quotes:
- On the definition of intelligence: “if you told Goldilocks and the Three Bears to generative ai, it would probably now understand in contrast to the 1980s that this is not a factual story” and “knowledge, information processing, all of those things are deeply embodied in our bodies in the broader cultural matrix, in the broader social matrix.”
- On generative AI: “AI does not have that kind of intelligence” and “generative AI would be completely lost and at sea” without the necessary resources.
- On education and AI: “it’s an exciting time to be in the era of generative ai, which is a major advance in the field” and “his interests are in developing new types of educational systems to meet the opportunities and challenges of the 21st century.”
- On science fiction: “the really nuanced discussions of intelligence augmentation IA have come out of science fiction” and “if you look at Star Trek, the next Generation, you’ve got data, the Android, who is in fact an AI-based machine, and you’ve got the card, the captain, who is the wise human being.”
- On hype cycles of AI: “I’ve lived through many cycles of advances in AI and been impressed by the progress” and “we rarely never stick to the script” when it comes to utilizing AI in education.
Postscript: Some great resources for next steps here
Interesting blog post. I’m a qualitative researcher myself and I find it really interesting to read about the different ways that AI is being used in qualitative research.