Why your RAG evaluation is lying to you
Series: Evaluation, honestly — part 1
The three lies
Lie one: the test set only contains answerable questions. If every question in your eval has a grounded answer in the corpus, you are measuring retrieval on easy mode. Real queries are unanswerable, ambiguous, or multi-hop. Fix: label your test set by answerability and report precision on the unanswerable slice separately.
Lie two: the judge and the candidate share a model family. An LLM judge tends to rate its own family generously. Fix: cross-judge every eval with at least one model from a different family, and report agreement.
Lie three: latency is not part of the quality number. A system that answers 2 points better at 3x the latency is a different product, not a better one. Fix: report quality at a latency budget, or publish the Pareto curve.
All three fixes are implemented in Conductor; the numbers in that case study are the result of this methodology.