Every few weeks someone shows me a retrieval system that “isn’t working,” and almost every time the model is fine. The filing system isn’t.
The feature framing is the bug
Teams reach for RAG the way they’d reach for a date picker: drop it in, wire it up, ship it. But retrieval isn’t a widget you add to a product — it’s the quality of how your knowledge is organized, chunked, labelled, and kept current. The model is just the clerk. A brilliant clerk in a badly organized archive still hands you the wrong folder.
What actually moves the needle
- Chunking that respects meaning, not character counts.
- Metadata you’d trust a human to file by — source, date, authority, scope.
- Freshness as a first-class concern, because a confident answer from a stale doc is worse than no answer.
- Evals on retrieval itself, separate from the generation step, so you know which half is failing.
If you can’t say why the right document should rank first, the model can’t either.
Fix the filing system and the “model problem” usually evaporates. Skip it, and no amount of prompt-tuning will save you.