We measured our prompts. Here's what changed.

Six months ago we ran an internal decision on our own engagements — every prompt would live in the repo, get a fixture set, get a rubric, and get regression-tested on every merge. No prompt in a config file the eng team couldn’t see; no prompt tuned in a chat window and pasted into production; no prompt shipping without a test.

The decision was mildly unpopular for the first two weeks. Everyone liked it by the end of week four. Here’s what actually happened.

What it caught

Silent regressions from provider updates. Twice in the six months, a model update from one of our vendors changed structured-output behaviour in a way we would not have noticed. Both times, the fixture set failed overnight. Both times, we caught it before a customer did.

Prompts drifting apart. We’d been running two variants of what we thought was the same prompt in two products. The measurement discipline made us diff them. Turned out they’d drifted meaningfully six months before the measurement went in and neither team had noticed. We consolidated to one prompt and cut a small class of confusing behaviour that had been chalked up to “model noise”.

Prompts that had never actually worked. One prompt in a rarely-hit code path had been in production for a year and had always returned plausible-looking output. Turned out its fixture set failed against ground truth 40% of the time — a bug that had been silently degrading a product surface the whole time. We fixed it. Nobody had complained because nobody had been checking.

What it missed

Real-world edge cases the fixture set didn’t have. The rubric only scores what you fixture. We’d curated the fixture set from historical traces, so we’d covered the queries our users had actually asked. Queries the users would ask tomorrow — not so much. Twice we shipped a change that passed the fixture set and caused a mild real-world regression the fixture set couldn’t have caught. Both times, we added the new class of query to the fixture set.

The fixture set is a living thing. It’s not “the eval”. It’s a starting point.

Prompts that were technically correct but stylistically wrong. The rubric was mostly measuring correctness. What it didn’t measure well was tone. A prompt that produced factually correct output in a slightly-off voice would pass; the customer would notice the voice; the customer was right. We added a stylistic rubric — LLM-as-judge, scored against a small “tone reference” set — and it caught the next two.

Prompts that got expensive. Regression tests scored correctness, not cost. A change that improved output quality by two rubric points and doubled the token spend passed the gate. It shouldn’t have. We added a per-fixture cost budget and a delta alert to the CI job.

What the team wouldn’t give back

Two things:

The confidence to change prompts. Before the discipline, changing a prompt in a live product was a bit of a leap. Now it’s not. The gate gives us permission to make small changes often, which turns out to be the compounding practice — most prompt improvements are small and frequent, not large and infrequent.

The written record. Every prompt change is a commit with a message explaining what changed and why. Six months in, that record is where we go when a new engineer joins the team and asks “why is this prompt shaped like this”. The record is the answer.

What we’d do differently

Start the stylistic rubric earlier. We spent three months on correctness-only rubrics and shipped one or two changes that would have been caught earlier by a tone score. Stylistic rubrics feel soft; they matter.

Fixture the failure modes we’re worried about, not the ones we’ve seen. Half the value of the fixture set is in the “adversarial” fixtures — the queries you’re worried about the next version failing on, not the queries the last version failed on. We under-invested in these.

Score cost as a first-class rubric. Correctness is not the whole story. Cost, latency, and quality are the three variables you’re juggling; the rubric should catch a regression in any of them.

So what?

Prompt engineering is engineering. Not “engineering because we’re calling it engineering”. Engineering because it has fixtures, rubrics, version control, regression gates, cost budgets, and a written record.

The discipline is not much overhead. It’s about a day per prompt to set up and thirty seconds per merge to notice. And what it buys is the ability to change prompts confidently, which turns out to be the underlying capability that makes LLM integration a sustainable practice rather than a nervous one.

Related reading

• RAG for the meeting-notes-hostage — the retrieval side of the same discipline.
• A senior engineer’s checklist before letting an agent commit — the code-review equivalent.