Separate two questions
A retrieval-assisted application has at least two jobs: finding relevant evidence and using it to answer a question. A fluent final response does not prove that the first job succeeded. Evaluation becomes easier when you inspect these stages separately and preserve the retrieved passages beside each test result.
Start by asking whether the available evidence could support an answer at all. Only then evaluate whether the generated answer follows that evidence, communicates uncertainty and respects the user’s access permissions.
Build a small, deliberate test set
Collect questions that represent the actual work: a precise fact, a multi-document comparison, an ambiguous term, a recently changed policy, and a question the knowledge base cannot answer. Include the expected source or an explicit “no supported answer” label. A compact, reviewed set is more useful initially than thousands of automatically generated questions nobody has checked.
Keep separate examples for development and final review. If every prompt adjustment is judged against the same questions, apparent improvement can become familiarity with the test rather than better behaviour.
Inspect the retrieved evidence
Look at the passages returned for each question. Did the necessary source appear? Was it displaced by near-duplicates? Did a heading lose the paragraph that explains it? Chunk boundaries, metadata filters and ranking decisions can create problems that a longer generation prompt cannot repair.
Measure retrieval against the labels you actually have. A relevance score needs a clear definition, and a useful threshold depends on the application. Preserve examples of failures so that a change in aggregate performance can be explained, not merely reported.
Test access and absence
Permission checks belong in the retrieval path. Add test users with different document access and confirm that restricted passages never enter their answer context. Also test deleted, superseded and unavailable material. A correct response built from the wrong user’s document is still a failure.
Questions without sufficient evidence deserve their own evaluation. The application should be able to say what is missing, ask for clarification or direct the user to an appropriate next step. Inventing a confident answer is not a fallback strategy.
Change one layer at a time
Compare experiments with the same data version and test set. Change the chunking strategy, filter, reranker or prompt independently where possible. Record latency and operational cost alongside quality so an improvement remains useful in the real workflow.
Manual inspection is still valuable. Read a selection of improved, unchanged and regressed cases. This reveals whether the system is learning a useful pattern or simply benefiting from one unusually favourable example.
Make the evaluation repeatable
Treat the evaluation set, source snapshot and configuration as versioned artefacts. Run the same checks when documents, models or retrieval settings change. A short report should identify the version, the observed failures and the decision to ship, revise or investigate.
The goal is not a universal score. It is evidence that a particular information pipeline behaves acceptably for a particular set of users and questions.