Substack’s AI Detector Is a Vanity Project
A flawed tool, a murky rollout, and one big unanswered question
A COUPLE OF MONTHS AGO, a Google Alert let me know my name was mentioned in a Substack post from a user I hadn’t heard of. The “resistance”-themed account specialized in writing songs and poems and had mentioned me in one of its non-rhyming posts. It got basic facts about my government tenure wrong and made odd conclusions, so I dug deeper. The account posted new content almost daily. Its “poetry” was too long and verged on nonsensical. The pace and the quality made it pretty clear the account was using AI to generate its content.
To my dismay, the account had tens of thousands of subscribers, was monetized, and seemed to be cashing in on the left’s very real desire to support something—anything!—that was taking a stand against Trump. Meanwhile, my own newsletter and others’ sputtered despite putting in real work to write human-generated content on Substack’s so-called “engine of culture.”
I don’t want to read AI slop on Substack—or any other platform where I’m spending hours of my one wild and precious life. I think AI is wildly misogynist—and others have proven it is racist and biased in many other ways that reflect the state of our world. And I’m worried enough about the effects widespread AI use is having on cognitive function that I ban AI use in the graduate-level course I teach.
So I wasn’t necessarily predisposed to dislike Substack’s new partnership with AI-detection tool Pangram. But the tool’s imperfections, along with the platform’s perplexing rollout strategy for the partnership, makes me think this is little more than a vanity project that will allow Substack to claim it is the last human-dominated social media platform.
The Pangram collaboration, which Substack CEO and Co-Founder Chris Best rolled out in The Substack Post on July 21, allows anyone to scan any post, note, or comment on Substack to see if it is fully AI-drafted, AI-assisted, or human-written.
Some writers seem perfectly happy to shrug as an AI tool—itself trained on the stolen works of writers—passes its imperfect judgement on whether our work is also AI-generated. Even though Pangram is widely recognized as best-in-class among AI-detectors, its still makes plenty of mistakes, as Carey Lening writes:
Given that Pangram still gets a false positive 1 out of every 10,000 analyzed posts (and as JHong notes, that’s in controlled lab conditions!), if there are say, 2 million posts a week, that’s 20,000 cases where someone’s content is labeled 100% AI-generated.
Pangram performs worse in the other direction; The Atlantic reported that “Pangram’s false-negative rate, or how frequently the model incorrectly labels text as human, is closer to one-in-70 (although some other assessments say it is more accurate than that).” And after Indicator’s Alexios Mantzarlis conducted his own adversarial test on the tool, he found it fairly simple to get the detector to produce false negatives by making text rhyme or inverting phrases.
Aside from these concerns about accuracy, the way Substack rolled out the feature—and with whom—gives me tremendous pause. Its landmark announcements—the aforementioned piece from Best and a conversation with noted tech bro shill Matt Taibbi—did not address some of the biggest questions users were bound to have about the tool, including:
If I or others scan my text, is it being used to train Pangram’s models? After a lot of folks raised this question, Substack clarified in a post published three days after the launch: “These features do not use publisher content, either through Substack or Pangram, to train generative AI models.”
Is it possible to opt out of this feature? Yes, you can do this on a by-post, by-note basis, only after you scan your own post for AI usage. Substack only introduced this option after user feedback. I worry, though, that if allowing the scans becomes the norm, turning off this feature might be viewed as an admission of guilt. (I’m frankly not sure if I’ll turn it off yet—you’ll find out when I do!)
Further, though users are asking for it, Substack has not allowed users to opt out on a platform level, which seems extremely suspicious particularly when paired with the “first scan, then opt out” flow above. Is the workflow constructed like this because Pangram wants writers to be incentivized to use this “free” tool? What were the terms of the deal? Is Substack paying Pangram on a per-scan basis, or have they paid a flat fee and want to get their money’s worth? Do they want to generate as much data as possible so they can claim they have the “most human-generated content on the internet?”
If your text is marked as AI, does it affect your distribution or ability to monetize content at all? Substack hasn’t really addressed this yet. In Best’s conversation with Taibbi, he says, “Whatever we do I never want Substack to be a cop” and hopes the tool will simply allow users to ascertain if “there’s a trust that’s being violated.”
What about when Pangram gets it wrong? Well, according to the Substack help page about the Pangram collaboration, you, the user, “report [the] detection error,” “enter feedback,” and “it will be reviewed to improve detection quality.” Will the score be reversed? Will it be removed? Will you be paid for the tool’s defamation of your brand, your work, your name? 🤷🏻♀️ We’re left to guess.
And, the most telling question for me: Why did Substack choose Matt Taibbi to flog this rollout? Is Taibbi—a man who has spent the better part of the last four years peddling nonsense about disinformation researchers, willfully misinterpreting quotes, and failing to grasp linear time—who they think the most respected journalist in the space is? Or were they just focused on picking someone who would act as a rubber stamp, ask no hard questions, and wave away inconvenient truths? (For example, in the interview with Best, Taibbi excused Pangram’s error rates by noting that the Pangram guys admit that their tool isn’t foolproof. Gee, I’m sure that’s comforting to a person whose career is being lit on fire by a false accusation of AI-generated writing!) Or perhaps most chillingly, Substack leadership could have been interested in getting ahead of the business implications if Taibbi and his merry band of memelords decided that these labels were somehow “censorship”.
It’s frankly shocking that people like Taibbi don’t have more of a problem with the forced deployment of Pangram to every Substack user. The same set of people who cried “censorship” about fact-checking and disinformation research and tools that rate and label websites for trustworthiness are perfectly fine with a machine running an opaque, probabilistic model being used for deterministic conclusions that could affect people’s reputations and livelihoods. Pangram, they assert, is “designed to raise awareness,” but the same is true of the suite of tools and programs that have been deemed Orwellian. The fact is that Pangram performs a role very similar to Facebook’s maligned fact checkers, except it does it more opaquely.
I don’t hate the idea of a tool like this in general—I was and remain a fan of adding fact-checks and context cards to tweets and Facebook posts, and those efforts slowed the spread of misleading information. But efforts like this need to be rolled out with adequate transparency and must include proper appeals processes. With a tool like Pangram, users should also be educated about its limitations.
Between the rollout opacity and their choice of messenger—a man who uses his brain to write, sure, but uses it to write fantasy under the label of “journalism”—I think this is more of a vanity project than an attempt to safeguard humanity or reality. 🧭
I keep this writing free because access to careful analysis shouldn’t be paywalled, especially when the information landscape is so polluted.
If you find my work useful and want more of it, you can support it with a subscription for just $5 a month or $50 a year.
If you can’t, sharing this newsletter with a friend helps it make an impact. Thank you.