Your RAG pipeline needs an eval suite before it needs a model
The demo works on ten examples. Production has ten thousand. The thing that gets you across that gap is not a better model — it is a scorecard you trust.
There is a specific failure mode we see over and over. A team builds a retrieval-augmented demo in a fortnight, it works beautifully, everyone is thrilled. Then it meets real users and quietly falls apart, and nobody can say why, because there is no measurement.
The missing piece is almost never the model. It is the evaluation harness.
Build the scorecard first
Before writing a line of pipeline code, assemble a dataset of real questions with known-good answers. A hundred is enough to start. Three hundred is comfortable.
This is tedious work and every team wants to skip it. Do not skip it. Without it you cannot tell whether a change helped, and you will spend months tuning prompts by vibes.
Measure retrieval separately from generation
When an answer is wrong, there are two possible culprits: the retriever fetched the wrong chunk, or the model mishandled the right one. These have completely different fixes.
So score them separately. Retrieval gets recall@k against known-relevant documents. Generation gets scored on faithfulness to the retrieved context, holding retrieval constant.
On the Atlas engagement, our first end-to-end score was mediocre and everyone assumed the model. It was retrieval — chunking was splitting clauses down the middle. A chunking change, not a model change, moved recall from 71% to 94%.
Citations are a feature, not a nicety
Every extraction should point at the exact source span it came from. Two reasons.
First, it makes verification cheap for the user. A reviewer checks a citation in seconds rather than re-reading the document.
Second, it makes your debugging cheap. When something is wrong, you can see immediately whether the model invented it or faithfully reported bad retrieval.
Say "I don't know" loudly
The most valuable behaviour we shipped at Atlas was not an answer. It was surfacing ambiguity as a question instead of resolving it confidently.
Partners told us this was the feature that made them trust the system. A tool that is confidently wrong 4% of the time is worse than useless in a legal context. A tool that flags its own uncertainty is a colleague.