RAG Evaluation & Regression Testing

What you'll build

Lessons 1 and 3 made retrieval work and made it better; Lesson 4 made it safe. This lesson makes "better" and "safe" measurable. The golden rule of RAG evaluation: if you can't put a number on it, you can't tell when it breaks.

You'll score a pipeline against a small golden set - questions paired with the document that should be retrieved and the keywords a correct answer must contain - on three axes: retrieval recall@k, groundedness, and answer correctness. A question passes only if all three clear their thresholds; the gate passes only if every question passes. Then you'll run a candidate tweak that looks harmless and watch the eval catch the regression it slips in. Everything is offline and dependency-free, implemented identically in Python, Node.js and C#. Pick a language with the selector above; press → to begin.

RAG evaluation: turn "seems good" into a number you can track.

  golden set ─▶ for each question ─▶ retrieve ─▶ answer ─▶ score on 3 axes
  (question +                                              ┌─ recall@k    : gold doc in top-k?
   gold doc +                                              ├─ groundedness: answer terms in context?
   keywords)                                               └─ correctness : expected keywords present?
                                                                    │
                            pass = all three clear their thresholds │
                            gate = every question passes            ▼
        ┌───────────────────────────────────────────────────────────────────────┐
        ▼                                                                         ▼
   BASELINE  (top_k=3, no padding)                      CANDIDATE  (top_k=1, padding on)
   recall 1.00  grounded 1.00  correct 1.00             recall 0.80  grounded 0.61  correct 1.00
   GATE: PASS                                            GATE: FAIL  ← recall + groundedness regressed

A candidate that "looks fine" still drops two tracked numbers. The eval catches
it; an eyeball check (correctness held at 1.00) would not.

What you need

You follow this lesson by reading and running - there's nothing to write. The eval engine, the golden set and the tiny corpus all live in the repo. It is offline and dependency-free (nothing to install for Python; for the other ports you only need Node.js 18+ or the .NET 8 SDK). Run it from the repo root. For Python, bare ./run -l 5 opens the interactive scorecard playground; add demo for the one-shot print-and-exit comparison. The Node and C# ports are one-shot only (no GUI), and the demo word is optional - bare --lang node / --lang csharp runs the same comparison:

$ ./run -l 5                 # Python: interactive scorecard
./run -l 5 demo            # Python: one-shot, print and exit
./run -l 5 --lang node     # Node one-shot  (or: --lang csharp)

The golden set

The golden set is the eval's source of truth: five questions, each tagged with the gold document(s) that should be retrieved and the answer keywords a correct answer must contain, plus the thresholds the gate enforces. All three language ports read this one file, so they score identically. A real eval set is bigger and grows every time a bug teaches you a new question - but the shape is exactly this.

{
  "thresholds": { "groundedness": 0.75, "correctness": 0.5 },
  "questions": [
    {
      "id": "refund-window",
      "question": "how long do refunds take to arrive",
      "gold_docs": ["refund_policy.md"],
      "answer_keywords": ["30", "days", "5", "business"]
    },
    {
      "id": "shipping-speed",
      "question": "how fast is standard shipping",
      "gold_docs": ["shipping_faq.md"],
      "answer_keywords": ["3", "5", "business", "days"]
    },
    {
      "id": "cannot-login",
      "question": "i cannot log in to my account",
      "gold_docs": ["account_help.md"],
      "answer_keywords": ["reset", "password", "sign"]
    },
    {
      "id": "warranty-length",
      "question": "how long is the warranty",
      "gold_docs": ["warranty_terms.md"],
      "answer_keywords": ["12", "month", "warranty", "defects"]
    },
    {
      "id": "reset-password",
      "question": "how do i reset my password",
      "gold_docs": ["password_reset.md"],
      "answer_keywords": ["reset", "password", "sign"]
    }
  ]
}

Retrieve the candidates

Evaluation starts where retrieval does. We reuse the same deterministic keyword retriever from Lesson 4 (distinct-term overlap, minus stopwords) so the same query always pulls the same documents in every language. The first metric will ask a simple question of this step's output: was the gold document in the top-k?

def retrieve(query, docs, top_k):
    """Top-k docs by how many distinct meaningful query terms they contain.
    Deterministic: score desc, then name asc; zero-overlap docs are dropped."""
    q = terms(query)
    scored = []
    for d in docs:
        score = len(q & d["tokens"])
        if score > 0:
            scored.append((score, d))
    scored.sort(key=lambda s: (-s[0], s[1]["name"]))
    return [d for _, d in scored[:top_k]]

function retrieve(query, docs, topK) {
  const q = terms(query);
  const scored = [];
  for (const d of docs) {
    let score = 0;
    for (const t of q) if (d.tokens.has(t)) score++;
    if (score > 0) scored.push({ score, d });
  }
  scored.sort((a, b) => b.score - a.score || cmpOrdinal(a.d.name, b.d.name));
  return scored.slice(0, topK).map((s) => s.d);
}

List<Doc> Retrieve(string query, List<Doc> corpus, int topK)
{
    var q = Terms(query);
    return corpus
        .Select(d => (Doc: d, Score: q.Count(d.TokenSet.Contains)))
        .Where(x => x.Score > 0)
        .OrderByDescending(x => x.Score)
        .ThenBy(x => x.Doc.Name, StringComparer.Ordinal)
        .Take(topK)
        .Select(x => x.Doc)
        .ToList();
}

Score the answer three ways

An offline, deterministic answerer takes the fact from the top retrieved document (optionally padding it with one unsupported sentence - a stand-in for a model that adds a claim no document backs). Then three metrics turn the result into numbers: recall@k (did the gold document come back?), groundedness (what fraction of the answer's terms appear in the retrieved context?), and correctness (what fraction of the expected keywords are present?). Each is a fraction between 0 and 1.

def first_body_line(doc):
    """The first non-heading, non-blank line of a doc (its one fact)."""
    for line in doc["raw"].splitlines():
        line = line.strip()
        if line and not line.startswith("#"):
            return line
    return ""


def answer(retrieved, pad_unsupported=False):
    """Extractive answer: the fact from the top retrieved doc. With
    `pad_unsupported`, append a sentence that is *not* in any document - a
    stand-in for a model that pads its answer with an unsupported claim."""
    if not retrieved:
        return ""
    text = first_body_line(retrieved[0])
    if pad_unsupported:
        text = (text + " " + UNSUPPORTED).strip()
    return text


# --- The three metrics -------------------------------------------------------
def recall_at_k(gold_docs, retrieved):
    """Fraction of the gold documents that made it into the retrieved set."""
    if not gold_docs:
        return 1.0
    names = {d["name"] for d in retrieved}
    hit = sum(1 for g in gold_docs if g in names)
    return hit / len(gold_docs)


def groundedness(answer_text, retrieved):
    """Fraction of the answer's meaningful terms that appear in the retrieved
    context. An unsupported (hallucinated) claim drags this below 1.0."""
    a = terms(answer_text)
    if not a:
        return 1.0
    context = set()
    for d in retrieved:
        context |= terms(d["raw"])
    return len(a & context) / len(a)


def correctness(answer_text, keywords):
    """Fraction of the expected-answer keywords present in the answer."""
    if not keywords:
        return 1.0
    toks = set(tokenize(answer_text))
    hit = sum(1 for k in keywords if k in toks)
    return hit / len(keywords)

function firstBodyLine(doc) {
  for (const raw of doc.raw.split("\n")) {
    const line = raw.trim();
    if (line && !line.startsWith("#")) return line;
  }
  return "";
}

function answer(retrieved, padUnsupported = false) {
  if (retrieved.length === 0) return "";
  let text = firstBodyLine(retrieved[0]);
  if (padUnsupported) text = (text + " " + UNSUPPORTED).trim();
  return text;
}

// --- The three metrics -------------------------------------------------------
function recallAtK(goldDocs, retrieved) {
  if (goldDocs.length === 0) return 1.0;
  const names = new Set(retrieved.map((d) => d.name));
  const hit = goldDocs.filter((g) => names.has(g)).length;
  return hit / goldDocs.length;
}

function intersectCount(a, b) {
  let n = 0;
  for (const t of a) if (b.has(t)) n++;
  return n;
}

function groundedness(answerText, retrieved) {
  const a = terms(answerText);
  if (a.size === 0) return 1.0;
  const context = new Set();
  for (const d of retrieved) for (const t of terms(d.raw)) context.add(t);
  return intersectCount(a, context) / a.size;
}

function correctness(answerText, keywords) {
  if (keywords.length === 0) return 1.0;
  const toks = new Set(tokenize(answerText));
  const hit = keywords.filter((k) => toks.has(k)).length;
  return hit / keywords.length;
}

string FirstBodyLine(Doc doc)
{
    foreach (var raw in doc.Raw.Split('\n'))
    {
        var line = raw.Trim();
        if (line.Length > 0 && !line.StartsWith("#")) return line;
    }
    return "";
}

string Answer(List<Doc> retrieved, bool padUnsupported)
{
    if (retrieved.Count == 0) return "";
    var text = FirstBodyLine(retrieved[0]);
    if (padUnsupported) text = (text + " " + Unsupported).Trim();
    return text;
}

// --- The three metrics -------------------------------------------------------
double RecallAtK(List<string> goldDocs, List<Doc> retrieved)
{
    if (goldDocs.Count == 0) return 1.0;
    var names = retrieved.Select(d => d.Name).ToHashSet();
    var hit = goldDocs.Count(names.Contains);
    return (double)hit / goldDocs.Count;
}

double Groundedness(string answerText, List<Doc> retrieved)
{
    var a = Terms(answerText);
    if (a.Count == 0) return 1.0;
    var context = new HashSet<string>();
    foreach (var d in retrieved) context.UnionWith(Terms(d.Raw));
    return (double)a.Count(context.Contains) / a.Count;
}

double Correctness(string answerText, List<string> keywords)
{
    if (keywords.Count == 0) return 1.0;
    var toks = Tokenize(answerText).ToHashSet();
    var hit = keywords.Count(toks.Contains);
    return (double)hit / keywords.Count;
}

The gate, and the two configs

evaluate runs every golden question under a config and aggregates the scores into means plus a single gate verdict: green only if every question passes. Two configs tell the story - a baseline (top_k=3, no padding) that clears the gate, and a candidate (top_k=1, padding on) that looks like a reasonable tweak but quietly drops two of the numbers.

def evaluate(golden, docs, config):
    """Score every golden question under `config` and aggregate the result.
    `config` = {"name", "top_k", "pad_unsupported"}."""
    thr = golden["thresholds"]
    if not golden["questions"]:
        raise ValueError("golden set has no questions - add at least one to data/golden.json")
    rows = []
    for q in golden["questions"]:
        retrieved = retrieve(q["question"], docs, config["top_k"])
        ans = answer(retrieved, config["pad_unsupported"])
        rec = recall_at_k(q["gold_docs"], retrieved)
        gnd = groundedness(ans, retrieved)
        cor = correctness(ans, q["answer_keywords"])
        passed = rec >= 1.0 and gnd >= thr["groundedness"] and cor >= thr["correctness"]
        rows.append({"id": q["id"], "recall": rec, "groundedness": gnd,
                     "correctness": cor, "passed": passed, "answer": ans})
    n = len(rows)
    agg = {
        "mean_recall": sum(r["recall"] for r in rows) / n,
        "mean_groundedness": sum(r["groundedness"] for r in rows) / n,
        "mean_correctness": sum(r["correctness"] for r in rows) / n,
        "pass_count": sum(1 for r in rows if r["passed"]),
        "total": n,
    }
    return {"config_name": config["name"], "rows": rows, "aggregate": agg,
            "gate_passed": agg["pass_count"] == n}


# Two configs that tell the regression story.
BASELINE = {"name": "baseline", "top_k": 3, "pad_unsupported": False}
CANDIDATE = {"name": "candidate", "top_k": 1, "pad_unsupported": True}

function evaluate(golden, docs, config) {
  const thr = golden.thresholds;
  if (golden.questions.length === 0) {
    throw new Error("golden set has no questions - add at least one to data/golden.json");
  }
  const rows = golden.questions.map((q) => {
    const retrieved = retrieve(q.question, docs, config.top_k);
    const ans = answer(retrieved, config.pad_unsupported);
    const recall = recallAtK(q.gold_docs, retrieved);
    const grounded = groundedness(ans, retrieved);
    const correct = correctness(ans, q.answer_keywords);
    const passed = recall >= 1.0 && grounded >= thr.groundedness && correct >= thr.correctness;
    return { id: q.id, recall, groundedness: grounded, correctness: correct, passed };
  });
  const n = rows.length;
  const sum = (f) => rows.reduce((s, r) => s + f(r), 0);
  const aggregate = {
    mean_recall: sum((r) => r.recall) / n,
    mean_groundedness: sum((r) => r.groundedness) / n,
    mean_correctness: sum((r) => r.correctness) / n,
    pass_count: rows.filter((r) => r.passed).length,
    total: n,
  };
  return { config_name: config.name, rows, aggregate, gate_passed: aggregate.pass_count === n };
}

const BASELINE = { name: "baseline", top_k: 3, pad_unsupported: false };
const CANDIDATE = { name: "candidate", top_k: 1, pad_unsupported: true };

EvalResult Evaluate(GoldenSet g, List<Doc> corpus, Config config)
{
    var thr = g.Thresholds;
    if (g.Questions.Count == 0)
        throw new InvalidOperationException("golden set has no questions - add at least one to data/golden.json");
    var rows = g.Questions.Select(q =>
    {
        var retrieved = Retrieve(q.Text, corpus, config.TopK);
        var ans = Answer(retrieved, config.PadUnsupported);
        var recall = RecallAtK(q.GoldDocs, retrieved);
        var grounded = Groundedness(ans, retrieved);
        var correct = Correctness(ans, q.AnswerKeywords);
        var passed = recall >= 1.0 && grounded >= thr.Groundedness && correct >= thr.Correctness;
        return new Row(q.Id, recall, grounded, correct, passed);
    }).ToList();
    var n = rows.Count;
    var agg = new Aggregate(
        rows.Sum(r => r.Recall) / n,
        rows.Sum(r => r.Groundedness) / n,
        rows.Sum(r => r.Correctness) / n,
        rows.Count(r => r.Passed),
        n);
    return new EvalResult(config.Name, rows, agg, agg.PassCount == n);
}

var baseline = new Config("baseline", 3, false);
var candidate = new Config("candidate", 1, true);

Run the eval

Now run it - this scores the golden set under both configs and prints the scorecards side by side, then a regression summary, and exits - identical output in all three languages. (Bare ./run -l 5 opens the interactive scorecard instead; demo is the one-shot print-and-exit.)

$ ./run -l 5 demo

$ ./run -l 5 --lang node demo

$ ./run -l 5 --lang csharp demo

Read the output

The baseline scorecard is all green - every question clears recall, groundedness and correctness, so the gate passes. The candidate looks like a harmless tweak (fewer documents retrieved, a chattier answer) but the numbers tell the truth: mean recall drops because one question's gold document fell out of the smaller top-k, and mean groundedness falls below its threshold everywhere because the padded sentence is unsupported. Correctness never moved - the answer still contains the right keywords, which is exactly how this regression would sail through a manual eyeball check. The gate flips PASS → FAIL.

Config: baseline  (top_k=3, padding=off)
  id               recall  grounded  correct  result
  refund-window     1.00      1.00     1.00  PASS
  shipping-speed    1.00      1.00     1.00  PASS
  cannot-login      1.00      1.00     1.00  PASS
  warranty-length   1.00      1.00     1.00  PASS
  reset-password    1.00      1.00     1.00  PASS
  Aggregate: recall 1.00  grounded 1.00  correct 1.00   5/5 passed   GATE: PASS

Config: candidate  (top_k=1, padding=on)
  id               recall  grounded  correct  result
  refund-window     1.00      0.63     1.00  FAIL
  shipping-speed    1.00      0.61     1.00  FAIL
  cannot-login      1.00      0.56     1.00  FAIL
  warranty-length   1.00      0.68     1.00  FAIL
  reset-password    0.00      0.56     1.00  FAIL
  Aggregate: recall 0.80  grounded 0.61  correct 1.00   0/5 passed   GATE: FAIL

Regression vs baseline:
  mean recall:       1.00 -> 0.80   (-0.20)
  mean groundedness: 1.00 -> 0.61   (-0.39)   below threshold 0.75
  mean correctness:  1.00 -> 1.00   (+0.00)
  gate:              PASS -> FAIL

A question whose gold document ranks first

Want to experiment? You don't edit any code - the next step opens an interactive scorecard where you type and toggle. Start with a question whose gold document is the top hit, so it survives even a tiny top-k:

how long do refunds take to arrive

A question whose gold document ranks second

This one's gold document sits behind a near-duplicate distractor, so shrinking top_k to 1 drops it from the results - recall falls to 0 even though the answer still reads fine. (Prefer the terminal? ./run -l 5 demo runs both configs and exits.)

how do i reset my password

Confirm it with the test

An offline test pins the lesson's claims: the baseline clears the gate; the candidate fails it; the candidate regresses mean recall and mean groundedness below where the baseline sits while correctness holds steady; groundedness flags the unsupported sentence; and recall drops for the rank-2 question when top_k shrinks. No network, no model.

$ ./run -l 5 test

Tinker in the scorecard - no code editing

This is the payoff: ./run -l 5 (the default) opens an interactive scorecard over the golden set. Leave the box empty to see the whole set scored as one card - the three means and the gate. Pick a question to drill into its three numbers, the keywords it hit or missed, and what each retrieved document contributed. Then move the sliders - top_k, the groundedness and correctness gates - or flip on unsupported padding, and watch a number cross its threshold and the gate flip. Nothing to edit: the controls feed the very same evaluate the demo and test use.

$ ./run -l 5

Try - shrink top_k to 1

Drag the top_k slider down to 1 and watch this question's recall fall to 0: its gold document was the second hit, and a smaller top-k no longer reaches it. The answer still looks reasonable - that's the point.

how do i reset my password

Try - turn on unsupported padding

Leave the box empty for the full scorecard, then flip on unsupported padding. Groundedness drops below its gate for every question even though correctness stays at 1.00 - a fluent, on-topic answer that still smuggles in a claim no document supports.

how long is the warranty

From demo to production

This is a teaching demo; the answerer is a deterministic extractive stand-in so the lesson is reproducible. For real systems:

Keep the golden set, swap the pipeline - point evaluate at your real retriever and model and the metrics still apply. Grow the set - add a question for every bug, so the eval only ever gets stricter. Strengthen the metrics - pair keyword correctness with an LLM-as-judge or embedding similarity, score groundedness against cited spans, and track latency and cost alongside quality. Gate in CI - fail the build when a tracked number drops below threshold, and store the scorecard so you can see trends over time. Fold Lesson 4's untrusted-document handling into the same set so a safety regression also shows up as a number.

What you learned

"Seems good" is not a metric. A golden set plus three scores - recall@k, groundedness and correctness - and a gate turn quality into a number you can track. A candidate that looks fine can still regress a metric, and the eval catches it where an eyeball check would not: correctness held steady while recall and groundedness quietly dropped. Same algorithm in Python, Node.js and C#, byte-identical output, plus a scorecard playground where you watched the gate flip.

Next: Lesson 6 · Repo-aware AI assistant - ground an assistant in your own codebase.