Retrieval-augmented generation is easy to stand up and hard to trust. The first version always works on the examples you tested. The problem arrives when someone in the business checks a citation and it does not support the sentence attached to it.
I build retrieval systems where that check passes.
What that involves
- Chunking that respects structure. Splitting on token counts alone severs tables from their headers and clauses from their conditions. How a document is cut determines what can ever be retrieved from it.
- Hybrid search. Dense vectors miss exact identifiers, part numbers and rare terms. Keyword search misses paraphrase. Production systems need both, fused deliberately.
- Reranking. A cross-encoder over the top candidates buys more accuracy per pound than almost anything else, and it is frequently the missing step.
- Evaluation. A retrieval set with known answers, so a change can be measured rather than argued about.
- Freshness. Pipelines that keep the index current, and that tell you when they have not.
Signals you need this
- Answers are fluent, confident and subtly wrong.
- Citations point at real documents that do not say the thing.
- It works in English and falls apart in another language.
- Nobody can say whether last week’s change helped.
- Costs scale with corpus size rather than with usage.
How the work runs
I start by building the evaluation set, because without it every later decision is taste. Then the retrieval path is improved against it in measurable increments, and you get the harness at the end so your team can keep scoring changes after I have gone.
Related writing
Start a conversation
Bring the awkward part. If it is not something I should take on I will say so, and point you at what would help.