Against the Pangram Obsession

Why the AI detector's verdicts mean less than you think

People in my bubble seem increasingly obsessed with Pangram checks, but I’m not sure they’ve thought things through. It’s not clear to me what the tool is supposed to prove, or what uses of AI these people object to. Let me show you some basic problems with Pangram obsession, i.e. feeding posts and articles into Pangram and declaring victory for a “100% AI” verdict.

I wrote the first paragraph of this piece—the paragraph you just read—with the help of ChatGPT 5.6 Sol High. I did so to prove a point, but I have to walk you through the whole process to show you why the story of that paragraph raises questions about the meaning of Pangram checks.

I began by typing the basic idea for the paragraph into the chatbox. Here’s what I typed:

please write a short paragraph making the point that people in my bubble seem to be obsessed with Pangram checks but i'm not sure if they have thought this through... what do they think the tool proves and what exactly they object to then say that I'll show some basic problems with feeding posts and articles to pangram and declare victory upon seeing a 100% AI verdict

Obediently, ChatGPT wrote the following paragraph:

People in my bubble seem increasingly obsessed with Pangram checks, but I’m not sure they’ve thought through what the tool is supposed to prove—or what, exactly, they object to. In the next few posts, I’ll show some basic problems with feeding posts and articles into Pangram and declaring victory when it returns a 100% AI verdict.

If you run a Pangram check on the paragraph written by ChatGPT, the verdict will be “100% AI.”

But what does it mean exactly? If you compare my prompt with the AI-generated text, you’ll see that there are very few differences. First of all, the substance is all human. I told ChatGPT what to write. The AI did not add any idea or fact or argument.

But even the text itself is mostly human! The AI tweaked or added only a few words.

Here’s a comparison between the raw text inputted by me (without the directions to ChatGPT) and the AI-generated text:

It’s different, but not that much. Yet for some reason, those few changes left some alien traces that triggered the Pangram detector. I wonder which they might be. “Increasingly”? “Returns”? I have no idea. The fact is that ChatGPT left its fingerprints and Pangram detected them.

What should we make of this? Do we like the AI-generated text? If I fixed typos and punctuation and capital letters, would my raw prompt be better than the AI rewriting? Should we object to this use of AI? Yes or no? And why?

I don’t have solid answers to these questions but brandishing a Pangram verdict as the ultimate proof of intellectual virtue or vice will not help us make progress on these problems.

The story of my paragraph is not finished yet.

I read the AI-generated version and I didn’t like it. It was almost what I wanted to write but not exactly. There was a made-up concept—that I planned to write a series of posts rather than just one post—and a few words or stylistical moves that were not the ones I would normaly choose. So I tweaked the paragraph a little. This is what I changed:

I didn’t change much. I broke up a long sentence, dropped the “In the next few posts,” and tweaked a few other phrases.

Is it now a human text? Well, according to Pangram, now it’s 100% Human:

Again, what should we make of this? My revised version is not that different from the AI-generated one. But the AI-generated text was not that different from my unpunctuated, typed-in-a-flurry prompt! And yet we’ve gone from human prompt to 100%-AI text to 100%-Human text by tweaking just a few words here and there, without changing the substance of the paragraph.

What have these Pangram checks proved? Honestly, I don’t know.

To be sure, in both cases Pangram says that it has low confidence in its verdict because the text is short. Fair enough.

Pangram has a very low false-positives rate. If a 800-word article results in a “100% AI” verdict, it is very likely that it was heavily written with AI. But even in that case, we have no clue about the process. Did the AI simply put one word after another and by doing so left its fingerprints, like in my little experiment? Or did it come up with ideas, facts, examples, memorable phrases, and arguments? We don’t know.

(And, by the way, Pangram is not as impressive with respect to false negatives. I asked ChatGPT to change a third of the text of a previous version of this post, and Pangram still thought it was “100% Human.” It wasn’t. Using word-level Levenshtein distance—a measure to compare texts—the original post had 795 words, and the AI-revised version had 262 edits, which means that 32.96% of the original count was AI-generated.)

Does this mean anything? I don’t know! And, to be clear, I’m not saying that using AI for ideas and arguments or technical knowledge is bad, while using it for “mere” writing is good. I have very tentative, uncertain, and tormented views about all of this. I know a couple of things for sure: I hate bad prose and I love good ideas. The rest is very complicated.

Pangram is an impressive tool, but a tool is a tool, and we should not replace a complicated, inevitably nuanced conversation about AI, writing, and thinking with a tool.

A coda. I asked ChatGPT to give me a clever idea to conclude this post. This is what ChatGPT offered:

Make the point that Pangram checking is becoming a substitute for reading: a critic who cannot explain what is lazy, false, banal, derivative, or intellectually empty in a piece can still paste it into a detector and borrow the authority of a number.

The provocative version is that this is phrenology for prose—a pseudo-scientific purity test used by people who want the prestige of judgment without doing the work of judgment.

I was not impressed. It’s a different flavor of what I already said. But the phrenology analogy seems novel-sounding and kind of cool. Let me add it to the post.

Relying on Pangram to judge a piece of writing is like phrenology for prose: a pseudo-scientific test used by people who want the prestige of judgment without doing the work of judgment. A critic who cannot tell what is lazy or intellectually empty in a piece can still paste it into an AI detector. Who’s lazier now? The writer or the reader?

Well? Which version is better? And what do we even mean by better?

I don’t know. But my version is “100% Human.”