Arize test finds Opus 5.5 nearly drops em dashes, halves other Claudisms

Anthropic's claim and how it was tested

The Opus 5.5 announcement says, "We’ve made major improvements to the way Opus 5.5 writes and communicates, one of the most common areas of feedback we heard about Opus 5," and that the model "is less likely to use jargon or idiosyncratic phrases." According to the Arize post, Anthropic staff went further: Sholto Douglas wrote "We fixed the writing," and Tom Brown wrote "we fixed the accent."

An Arize author decided to treat that as a testable claim. Using Arize AX, the company's evaluation platform, the author built a tool called claude-compare and ran a standard loop: build a dataset, run an experiment for each model, annotate the output by hand, then build an evaluator from those annotations and run it.

The dataset holds 20 frozen research briefs, each containing notes for a blog post. They cover five genres (opinion, technical explainer, tutorial intro, news analysis, product announcement) and five domains (AI, travel, cooking, games, books). The briefs are written as bullet fragments, so Opus 5's style cannot leak into the input, and the writing model gets no tools and no network.

A controlled comparison

To isolate the model, both Opus 5 and Opus 5.5 received an identical prompt. Effort was pinned to medium, because Opus 5 defaults to high and Opus 5.5 to medium. The harness also checked the served model on every response. Each model ran over the dataset twice, producing four experiments, 80 posts and about 110,000 words.

For ground truth, the author read all 40 Opus 5 posts, about 53,000 words, in a single Google Doc and commented "Claudism" on every phrase that made them wince. That produced 153 flags across 38 of the 40 posts.

Claudisms are moves, not phrases

The flags did not match the author's expectations. A hand-written list of famous Claudisms such as "load-bearing", "delve", "crucially" and "genuinely" matched only 9 of the 153 flags. "Load-bearing" itself appeared just three times in 53,000 words of Opus 5.

What the author had flagged were rhetorical moves. The largest group, about 48 flags, was text announcing that something matters instead of showing why, such as "The interaction matters" or "deserves a moment." Contrast reframes came next at about 30 ("It’s a topology, not a genre."), then verdict intensifiers at about 25 ("the whole point"), signposts at about 20, stock metaphors at about 10 and gotcha framing at about 7. The author's conclusion: "It’s not a phrase problem. It’s a moves problem."

That shaped the evaluator. A first judge that gave each post a single 1 to 5 score was hard to check, so the author switched to a span judge that returns each Claudism as an exact quote in one of six move categories. Its output can be matched against the human flags. Em dashes are counted by plain code. The judge is OpenAI's gpt-6-luna, so Claude does not grade its own family. Results are normalised per 1,000 words, since Opus 5.5 writes about 8% longer.

The results

The em dash result comes from the code evaluator, with no judge involved. Opus 5 used 12.9 em dashes per 1,000 words. Opus 5.5 used two in its entire 57,000 words of output, about 0.05 per 1,000 words, a 99.6 percent drop.

The span judge found 4.79 Claudisms per 1,000 words in Opus 5 against 2.38 in Opus 5.5, a 50 percent drop. The Arize author's verdict is that the writing is better but not fixed.

What it means for users

For readers and editors who spot Claude's output in posts and articles, the most visible marker, the em dash, has effectively disappeared in Opus 5.5 under these test conditions. The subtler habits remain at about half their earlier rate. Because these habits are rhetorical moves rather than fixed wording, the author argues that simple phrase lists will not catch them reliably. The test covers 20 briefs and one reviewer's flags, so it is a single evaluation rather than a definitive measure.