We killed our best feature. The judge that killed it was an AI.

In July 2026 we ran a test designed to kill our own flagship feature, and it worked. This is the writeup, including the part that makes us look worse than the result itself does.

What the feature did

The most common complaint about AI-generated prose is that it tells you things instead of showing them. It reaches for the abstract emotional word — she felt betrayed, he was overwhelmed, the room was tense — where a human writer would have found the concrete detail that produces the feeling in a reader without naming it.

Every AI writing tool will happily fix that for you by rewriting your paragraph. And that is the problem, because the rewritten paragraph is the model's prose, not yours. You have swapped a flat sentence you wrote for a polished sentence you didn't.

So we built something narrower. It found the abstract emotional placeholders in your writing and grounded them in specific, physical detail — while leaving your sentence standing. Your syntax, your clause order, your paragraph rhythm, all untouched. It changed words, not structures. We called it a surgical edit.

That constraint was not a limitation we were working around. It was the entire product. Anyone can rewrite your paragraph; the bet was that writers would rather have their own sentence improved than replaced.

The gate

We wrote the failure condition before running anything, which is the only part of this we would do the same way again. The bar: our surgical edits had to be preferred at least 60% of the time over the obvious alternatives — Sudowrite, and a good general-purpose frontier model given the same passage. Blind, so nobody knew which was which.

We lost in all but one case.

Not close to the bar. Not mixed. The whole-paragraph rewrites won almost every comparison.

Why we did not call it a bug

There is always a comfortable reading of a result like this, and it is how products survive years after their premise has died. It goes: the idea is right, the prompt needs work, give it another iteration.

That reading was not available. The surgical constraint was the thesis. If the version that throws your sentence away wins, then a tool whose only reason to exist is not throwing your sentence away has been told something it cannot engineer around. So we stopped, and the slot the feature occupied went to something else. Before a launch, before a price, before anyone had paid for it — which is the cheapest moment this lesson was ever going to be available.

Now the part we got wrong

The gate was designed for human raters. The verdict came from LLM judges.

We had built a tool to stop AI from flattening human writing, and then we asked an AI to grade whether it had worked. Put that way it is obviously absurd, and it was not obvious at the time.

Fluent, smooth, idiomatic prose is a language model's native register. It is what they are trained to produce and what they recognise as good. Show one a passage that keeps a human's slightly awkward, slightly particular sentence intact, next to a passage rewritten into confident competence, and it will pick the second one — not because it evaluated the trade-off and disagreed with us, but because the second one sounds like it. We put our thesis in front of the one judge structurally guaranteed to vote against it.

So does the verdict stand?

Yes, and we want to be careful about why, because "our test was flawed" is the most convenient sentence in product development and it is usually the beginning of never killing anything.

A gate you re-run until you like the answer is not a gate. We had pre-registered the bar, the comparison and the failure condition. Losing and then discovering a reason the loss might not count is precisely the situation the discipline exists for. So the feature stayed dead.

But there is a second, deeper problem that a human panel would not have fixed, and it is the finding worth carrying away:

Any third party judging anonymous passages will reward polish, because they have nothing at stake in whose voice it is. Voice preservation is worth exactly zero to someone who does not know what the author sounds like.

Neutral human raters would also have preferred the smoother paragraph, for a related reason. The LLM judges made it worse and more systematic, but swapping them for strangers would have answered the same wrong question. The question we cared about — did this keep what was yours? — is unanswerable by anyone who has never read anything of yours.

What changed because of it

The feature is gone. Not paused, not on a roadmap.

The tools that survived analyse rather than rewrite. The two we run today read what you wrote and tell you something about it. That is not a coincidence or a retreat into safety — it is the position this result argues for. The same reasoning runs through why story bibles get ignored: generation is where models are strong and unreliable, checking is where they are boring and dependable.

The next gate is built around the judge. Voice preservation is a different claim from the one that died here, and it is the current bet. Its kill-gate uses authors judging edits to their own manuscripts, blind, against a well-prompted frontier model rather than a competitor picked for being easy to beat. And the question is not "which sounds more like you", because people are agreeable — it is would you publish this as yours?

If you are evaluating any AI writing tool

Ask who judged. Almost every tool on the market claims it preserves your voice, and almost none of them can tell you how that was established. The three questions that separate a measurement from a marketing line:

1. Who did the judging — the vendor, strangers, a model, or writers?
2. Whose text were they judging? If it was not their own, they were grading prose, not voice.
3. Was the failure condition written down before the test, or decided after?

We failed the third question well and the first two badly. That is worth publishing, because a company that only tells you about the tests it won is not telling you about its tests.