RAG for the meeting-notes-hostage: retrieval when the corpus is a mess.

You know the demo. Perfect PDF, obvious paragraphs, one clean question, one clean answer. Then you ship RAG onto the actual corpus — the shared drive, the meeting notes, the wire copy, the scanned reports from the 2010s — and quality drops off a cliff.

The temptation, at that point, is to blame the model. Try a bigger one; try a different provider; sprinkle chain-of-thought on the prompt. Ninety times out of a hundred, the model is doing its job and the retrieval is quietly returning the wrong passages. Fix the retrieval and the generation problem often disappears.

This piece is a practitioner’s tour of what “fixing the retrieval” looks like when the corpus is a mess. Real chunking. Hybrid retrieval. Re-ranking. Provenance UX. And the eval spine that ties all of it together.

What you’re actually building

A production RAG system has, at a minimum, five moving parts:

1. Ingestion — turning the corpus into structured, chunked, embedded records.
2. Retrieval — turning a query into a shortlist of candidate passages.
3. Re-ranking — sorting the shortlist by task-specific relevance.
4. Generation — turning the shortlist plus the query into an answer.
5. Provenance UX — showing the user what supported the answer.

Each of the five is capable of dragging the whole thing down. Most RAG programmes I’ve seen concentrate on 2 and 4, spend some energy on 3, and under-invest in 1 and 5. That’s usually the wrong distribution.

Chunking that respects structure

Chunking is where most retrieval failures are born. The naive approach — fixed-size character windows with overlap — throws away every signal the document was trying to give you.

A useful chunking pipeline looks more like a small parser than a splitter:

Recognise document type. A long-form article is not a wire item is not a meeting transcript is not a scanned PDF. Each has a shape; the chunking strategy should know that shape.

Preserve semantic boundaries. Section headings, list items, table rows, transcript turns. Chunks that respect the author’s intended boundaries are chunks that retrieve well.

Attach structural metadata. Chunk = text + section_path + doc_type + timestamp + author + source_url. When the retriever finds a chunk, the downstream stages can reason about where in the document that chunk came from. This is what makes provenance possible.

Deal with the ugly stuff explicitly. Scanned PDFs need OCR before chunking. Meeting transcripts need speaker attribution before chunking. Every “we’ll clean it up in generation” instinct is a bug waiting to happen.

An extra day on the chunking pipeline usually saves a week of retrieval debugging later.

Hybrid retrieval

Dense retrieval — cosine similarity over embeddings — is the default for a reason: it handles paraphrase and synonym gracefully. But it has well-known failure modes on the queries you actually care about:

• Named entities — “What did the 2019 audit say about X?” often needs an exact match on “2019 audit” that dense retrieval blurs.
• Rare terms — technical vocabulary that appears once in the corpus can be under-weighted by the embedding space.
• Numeric queries — “Show me every invoice over £50k” is not what cosine similarity is good at.

The pragmatic answer is hybrid retrieval: run both dense and sparse retrieval, union the results, and let the re-ranker sort them.

dense_hits  = dense_retriever.search(query, k=25)
sparse_hits = bm25_retriever.search(query, k=25)
candidates  = deduplicate(dense_hits + sparse_hits)   # ~40 candidates

BM25 is a mature, cheap, well-understood sparse retriever. Reach for elaborated lexical retrievers when the workload justifies the complexity; don’t reach for them by default.

Re-ranking that pays for itself

Given a shortlist of 40 candidates, a cross-encoder re-ranker scores each against the query with substantially more signal than the retriever alone. The re-ranker is expensive per-call — but it only runs on 40 candidates, not the whole corpus.

Two practical rules from engagements I’ve been on:

Tune the re-ranker on your fixture set. Off-the-shelf cross-encoders often need a small task-specific fine-tune to reach production quality. That fine-tune usually pays for itself in three weeks of retrieval improvement.

Cap re-ranking latency. Cross-encoders are cheap in isolation and expensive at the tail. A ceiling on re-rank latency, with a graceful fallback to the retriever-only ordering, is worth the extra ten lines of code.

The output of the re-ranker is the top-K (usually 3–7) passages that actually reach the model.

Provenance in the UX

Provenance is the reason your users trust the system. It’s also the reason the eval loop works. Both flow from the same design decision: every generated claim links back to the passage that supports it.

The mechanism looks like this:

1. The model produces the answer as a sequence of sentences.
2. For each sentence, a scoring step identifies which retrieved passages support that sentence.
3. The UX renders each sentence with an inline citation to the supporting passage, expandable to the source.

Where a sentence has no supporting passage, the answer is unsupported — regardless of whether it happens to be true. Unsupported sentences get flagged, repaired, or dropped.

The provenance panel in the UX is the difference between a system users trust and a system they check by hand every time.

The eval spine

I’ve written about eval spines more generally in another piece. For RAG specifically, the spine has to score both retrieval and generation separately:

Retrieval scoring. For each fixture query, is the ground-truth passage present in the top-K? At what rank? A retriever that returns the right passage at rank 7 when the model only sees rank 5 has failed even if the “right” passage is technically in the shortlist.

Generation scoring. For each fixture query, given the retrieved shortlist, does the model produce an answer that matches the ground truth? Score both correctness and citation precision. Correctness alone is not enough.

Regression bars on both. A merge that improves generation quality by one point but drops retrieval recall by five is a merge you don’t want.

What “we caught it in eval” looks like

Two examples from real engagements:

An engagement in early 2026 shipped an embedding-model upgrade that improved dense retrieval on the public benchmark by four points and tanked retrieval on the client’s fixture set by nine. The public benchmark hadn’t included any queries shaped like the client’s real workload. The regression fired in CI; the merge was reverted; the retriever was tuned for the specific workload instead of upgraded blindly.

A different engagement caught a chunking regression when a change to transcript ingestion silently dropped speaker attribution. Retrieval still recovered “the right chunk” but the model no longer knew who had said the thing in the chunk, so the answers looked correct and cited the wrong person. Only the citation-precision rubric caught it.

Both of those are failures the naive “score the final answer” eval would have missed.

So what?

Most RAG failures are retrieval failures wearing generation costumes. Fix the retrieval and the generation problem often shrinks to manageable.

The fix is not exotic. Chunk with structure in mind. Retrieve with both dense and sparse. Re-rank with a cross-encoder you’ve tuned on your data. Make provenance visible in the UX so users trust what they’re reading and you can measure what’s actually happening. Build the eval spine early, and score retrieval and generation as separate concerns.

Every one of those is a piece of boring engineering. Boring engineering is what turns RAG demos into RAG products.

Related reading

• We measured our prompts. Here’s what changed. — the same measurement discipline, applied one layer up.
• Evals for agents that use tools, not just tokens — trajectory-level eval for the retrieval-augmented agent case.