§ Case study · LLM integration
Editorial-assist RAG for a global media company.
A retrieval-assisted drafting surface for a 300-strong editorial team — answers cited to the archive, hallucinations effectively eliminated on tested queries.
a global media company
Client
Enterprise
Tier
14 weeks
Duration
2026
Delivered
The context
What they came to us with.
The editorial team was writing on top of a fifteen-year archive that nobody could search meaningfully. A basic RAG prototype had been built by an internal team; it worked well enough on demo queries and badly enough on real ones that senior editors had stopped using it.
They were clear about what "good" meant — every answer had to be cited to source, provenance had to be visible in the drafting surface, and hallucination rate on their own realistic queries had to be below one percent. That last one was the hard bar.
The archive itself was heterogeneous — long-form articles, wire copy, video transcripts, PDF reports scanned in the early 2010s. Chunking was doing damage before retrieval had a chance.
The approach
What we did, in order.
-
Fixture set before framework choice.
Two weeks of shadowing senior editors produced a hundred realistic queries with hand-graded correct answers. That set became the spine of every subsequent decision — no change shipped without a run against it.
-
Chunk with structure in mind.
Article structure, wire dateline, transcript timecodes, and section headings all became first-class signals. Chunking respected the document; retrieval got half its lift here before any embedding choice was made.
-
Hybrid retrieval, then re-rank.
Dense retrieval over the same-model embeddings, sparse retrieval over a well-tuned lexical index, results unioned then re-ranked by a cross-encoder tuned on the fixture set. Cost caps enforced per query.
-
Provenance in the drafting UX.
Every generated sentence was linked to the retrieved passage that supported it, expandable to the source. The editorial team could see what the model had seen and reject anything unsupported before it reached a draft.
-
Grounded generation with a repair loop.
Structured output schema at the sentence level. When the model produced a claim without a supporting citation, a repair pass replaced it or dropped it — with a bounded retry budget so the loop couldn't run away.
How we measured
The bar we held ourselves to.
- Per-query hallucination rate against the fixture set, with a per-topic breakdown.
- Citation-precision (every claim mapped to a supporting passage) and citation-recall (every passage that would have supported the claim was retrieved).
- Editorial approval rate on generated drafts, tracked per editor and per topic.
- Cost-per-query envelope, alerted at threshold and hard-capped in the runtime.
- Regression bars in CI — a merge failed if any topic-class hallucination rate rose more than 0.5pp.
The outcome
Where it landed.
0.4%
Hallucination rate on eval set (from 11%)
4.1×
First-draft time reduction
100%
Answers cited to source
At go-live, hallucination rate on the fixture set was 0.4%, down from 11% on the prototype. First-draft time dropped by more than 4× on the editorial workflows we'd targeted. The most-quoted metric internally was actually citation-precision — the editorial team trusted the surface because every claim was checkable in one click.
The client has since rolled the pattern out to a second editorial unit and is planning a third. The eval spine is theirs; the CI gates are theirs; the retrieval infrastructure runs on their own platform.
Retrospective
What we'd do differently.
- We under-scoped the ingestion pipeline. Two months in, we discovered scanned PDFs from 2011 needed a completely separate OCR pass. Should have caught that in the first week.
- The initial fixture set was too narrow on wire-copy queries. We caught it in eval; a wider baseline would have caught it earlier.
- We'd give provenance a UX designer from day one. The dev-built provenance panel worked; a properly designed one would have been trusted six weeks sooner.
Services used
The moves in this engagement.
S3 · Build
LLM Integration
Chat, RAG, and the wider integration stack — measured, regression-tested, and cost-aware from the first prompt.
S5 · Advise
Technical & Solutions Architecture
A senior architect embedded with your programme — end-to-end design, build-vs-buy calls, vendor evaluation, technical risk review.
Related work
Other engagements in the same shape.
§ Case study · Agentic systems
An agentic ops-triage copilot inside a UK retail bank.
Analyst-in-the-loop triage across three legacy ticketing systems — mean time to first response cut by more than half, without a single new ticketing surface.
§ Case study · AI strategy
An AI opportunity portfolio for a European insurer.
A prioritised opportunity portfolio and an operating model to run it — grounded in what the business could actually build in the next three quarters.
Have something with this shape?
A 30-minute discovery call. We'll say honestly whether the pattern fits.