ChatGPT as a blurry jpeg of the web

by | Sunday, February 12, 2023

Ted Chiang is one of the greatest, insightful writers working today. I had written previously about one his short stories in a post titled: Truth of fact and feeling: Unpacking McLuhan (2/3) about his short story The truth of fact and the truth of feeling. (If you haven’t read it – please do so).

In a recent piece in the New Yorker, (ChatGPT is a blurry jpeg of the web) Ted Chiang focuses on ChatGPT, and offers one of the best descriptions of the technology, making an analogy with lossy compressions systems (such as the Jpeg file format that we usually use to compress images). The essay is worth reading in full, but here are some key quotes:

… Think of ChatGPT as a blurry jpeg of all the text on the Web. It retains much of the information on the Web, in the same way that a jpeg retains much of the information of a higher-resolution image, but, if you’re looking for an exact sequence of bits, you won’t find it; all you will ever get is an approximation. But, because the approximation is presented in the form of grammatical text, which ChatGPT excels at creating, it’s usually acceptable. You’re still looking at a blurry jpeg, but the blurriness occurs in a way that doesn’t make the picture as a whole look less sharp.

This analogy makes even more sense when we remember that a common technique used by lossy compression algorithms is interpolation—that is, estimating what’s missing by looking at what’s on either side of the gap. When an image program is displaying a photo and has to reconstruct a pixel that was lost during the compression process, it looks at the nearby pixels and calculates the average. This is what ChatGPT does when it’s prompted to describe, say, losing a sock in the dryer using the style of the Declaration of Independence: it is taking two points in “lexical space” and generating the text that would occupy the location between them. (“When in the Course of human events, it becomes necessary for one to separate his garments from their mates, in order to maintain the cleanliness and order thereof. . . .”) ChatGPT is so good at this form of interpolation that people find it entertaining: they’ve discovered a “blur” tool for paragraphs instead of photos, and are having a blast playing with it.

Given GPT-3’s failure at a subject taught in elementary school, how can we explain the fact that it sometimes appears to perform well at writing college-level essays? Even though large language models often hallucinate, when they’re lucid they sound like they actually understand subjects like economic theory. Perhaps arithmetic is a special case, one for which large language models are poorly suited. Is it possible that, in areas outside addition and subtraction, statistical regularities in text actually do correspond to genuine knowledge of the real world?

I think there’s a simpler explanation. Imagine what it would look like if ChatGPT were a lossless algorithm. If that were the case, it would always answer questions by providing a verbatim quote from a relevant Web page. We would probably regard the software as only a slight improvement over a conventional search engine, and be less impressed by it. The fact that ChatGPT rephrases material from the Web instead of quoting it word for word makes it seem like a student expressing ideas in her own words, rather than simply regurgitating what she’s read; it creates the illusion that ChatGPT understands the material. In human students, rote memorization isn’t an indicator of genuine learning, so ChatGPT’s inability to produce exact quotes from Web pages is precisely what makes us think that it has learned something. When we’re dealing with sequences of words, lossy compression looks smarter than lossless compression.

Do read the actual piece… it is worth your time: ChatGPT is a blurry jpeg of the web

A few randomly selected blog posts…

Common sense in science class

Common sense in science class

Students can sometimes perceive scientific ideas to be in conflict with their common sense. How do we approach such conflicts in the classroom? Do we see these commonsense ideas as being wrong or, at best, misconceived? Alternatively, do we see them as resources and...

A different language

I have always been interested in how we use words to capture intangibles. For instance wine connoisseurs have developed a specialized language (which sadly is quite opaque to me) to explain to each other characteristics of wine. So the words "fruity" and "dry" have...

Guest blogging for Nashworld: TPACK video

Sean Nash over at Nashworld asked me to guest blog for this week while he is out with his students doing some really cool stuff. Here is a link to my posting: A TPACK video mashup!. I end the post with a couple of videos, one a commercial and the other my mashup...

TPACK & More: Presentation at RemoteK12 summit

TPACK & More: Presentation at RemoteK12 summit

REMOTE K12: The Connected Teacher Summit, was a one-day virtual summit hosted by ASU, designed for K-12 teachers and those that support and enable teachers in district public, charter and private schools.  I presented a talk titled: Technology in teaching &...

Buttoning on to a trend

There is an barely interesting article on today's NYTimes.com site by Steven Heller on campaign souvenirs being sold by the three presidential candidates through their websites (read: From Mousepads to Piggy Banks). I thought his earlier columns on the graphic design...

Failure has to be an option

I just read this great interview with Diane Ravitch on Slate.com (The wrong stuff). Diane Ravitch started out under George H.W. Bush as a strong supporter for NCLB (and all that goes with it, educational testing, school choice, charter schools etc. etc. etc.)....

Koehler & Mishra (2005)

One of the important papers in the TPACK sequence is Koehler & Mishra (2005). In this paper we developed and administered a survey to measure the evolution of TPACK as people engaged in a design task. This research complements our previous empirical work (Koehler,...

Have a great 2014!

It is that time of the year... the time for the Mishra/Sawai family new year's video. As tradition has it the video needs to be some kind of a typographical animation, typically a play with words that is synchronized to music, and attempts to incorporate a visually...

Stuff Indian’s Like

After the success of Stuff white people like, can Stuff Indians like be far behind. Check it out... it has the occasional nugget that nails Indians and their behavior.

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *