Quinn Gao

On writing humanly

Recently I’ve been heavily engaged in AI-assisted scientific research. While the large language models are generally impressive, their behaviors outside of coding and running research experiments have been getting noticeably worse. Certainly we have all anticipated this outcome, as the AI companies are getting less and less patient and their eyes are less and less on the AI capabilities that are hard to benchmark.

This is what we often call a “misalignment” of goals—the goals of OpenAI and Anthropic are their impending IPOs, and a major part of the user base that they want to cater to (and maybe want to lure into investing) are vibe coders (perhaps transferable to vibe spenders?). While everyone, including me, now finds agentic coding irresistible, there is a major difference between AI-assisted coding and vibe coding, and that difference is to write human-readable code. Similarly, in creative writing we ask for humanly writing; in product designing we ask for human-friendly user experience. But instead, vibe coders/authors/designers do not really care about that—because they don’t read, and they don’t care.

I cannot emphasize this enough. Ever since 2026 we see a gigantic bloom in vibe-coded apps. While you can definitely say “some are more vibe-coded than others,” there are always small details that users find “slop-like” even on the least vibe-coded apps. While different people have different tastes on 2D design, it’s a subjective matter, I won’t argue with that; the most jarring slop that triggers me is always the “writing slop.”

Defensive Writing in GPT Models, a very recent arXiv preprint by Liao et al., discussed the latest reward-hacking tricks by GPT models when they write—”defensive writing,” which makes their writing very un-human. The paper is a good read, if you are willing to; I’ll briefly give my two cents on the ideas I find most interesting.

The reason why defensive writing works is because that makes the language safer and more “rigorous” (a word that recent AI favors) to the eyes of reviewers. Using AI models to grade AI models is the biggest weakness of reinforcement learning that amplifies the reward hacking. The paper found that when the prompt mentions or hints an anticipated reviewer, the reinforcement nature pushes the models to do strategic writing, by all sorts of defensive writing techniques, including but not limited to:

Although human readers will surely vomit at this unreadable slop, AI graders are not tasked to grade “slopness”. Instead, they are tasked to grade “soundness,” which accumulated scoped claims and careful negation can have a very substantial presence in their Attention layers. Attention is NOT all you need.

Can AI differentiate justified, evidence-driven caution and “defensive slop” driven by getting rid of any risks in claims? I believe they can; they are just too heavily trained with reward hacks so they are “sub-consciously” spitting defensive writing. Just like how “small-town swots” (literary translation: small-town exam-problem-solving specialist), generations of Chinese people trained by massively solving (grinding) exam problems, find formulaic success much more appealing than thinking out of the box, even if the question is no longer about correctness. If AI companies do not re-align their models to stop reward-hacking, those models will continue to produce slop in the future.

…Well, that is not a bad thing, really. If the goal is to be able to differentiate human writing with AI-assisted writing (not just AI-generated writing), then the current era is the best time you can quickly find out who used AI assistance. But if your goal is to use AI to assist your writing, then you will get disappointed, as you are going to have a hard time finding a frontier model that is both capable in intelligence and in writing humanly.

#Essays