RAG Cookbook

RAG grounds answers in retrieved chunks at query time. Reach for it when the corpus is large or changing and you need citations. Prefer long context for a small stable corpus that fits the window, and fine-tuning for style/format, not facts.

When RAG vs alternatives

Chunking recipes

Overlap rule of thumb: 10–20% of chunk size. Too little loses boundary context; too much inflates the index and duplicates hits.

Hybrid retrieval config

Dense (embeddings) catches paraphrase; sparse (BM25) catches exact IDs, codes and rare terms. Fuse them, then rerank.

retrieval:
  dense:
    model: text-embedding-large
    top_k: 40
  sparse:
    algorithm: bm25
    top_k: 40
  fusion:
    method: reciprocal_rank_fusion   # RRF
    k: 60                            # RRF constant
  rerank:
    model: cross-encoder-reranker
    top_n: 8                         # feed the LLM the top 8
  diversity:
    method: mmr                      # optional, reduce redundancy
    lambda: 0.5

Reciprocal Rank Fusion: score of a doc = Σ over lists of 1 / (k + rank). With k=60, a doc ranked #1 in both lists scores 1/61 + 1/61 ≈ 0.0328; a doc ranked #1 dense but absent in sparse scores 1/61 ≈ 0.0164. RRF needs no score calibration between systems — it ranks by position.

Reranking

A cross-encoder reranker reads the query and each candidate together, giving far better precision than the bi-encoder used for first-stage retrieval — at higher cost. Retrieve broadly (top_k 40–100), rerank to a small top_n (5–10) for the LLM.

Query rewriting

Eval metrics with worked numbers

Separate retrieval quality from generation quality — a good answer can hide bad retrieval and vice-versa.

Worked comparison — swapping in a reranker on a 200-query eval:

Reranking barely moves Recall@5 (same candidates) but lifts MRR and faithfulness sharply by ordering the right chunk first — the LLM sees it earlier.

Debugging playbook

1. Reproduce with the trace. Log the query, rewritten query, retrieved chunk IDs+scores, reranked order, and the final prompt. Most “wrong answer” bugs are visible here.
2. Is the gold chunk retrieved at all? If not in top-k → retrieval problem: check chunking (drift), embeddings (wrong model/dim), or query rewriting.
3. Retrieved but ranked low? Add/adjust the reranker; tune RRF k; verify hybrid weighting.
4. Retrieved and top-ranked but answer wrong? Now it is generation: check the prompt (are chunks actually passed?), context ordering (docs first), and faithfulness (is the model ignoring context?).
5. Confident-but-wrong after a refresh? Suspect the index: re-embedded with a different model, changed chunker, or stale vectors. Re-run retrieval evals.
6. Per-segment check. Break metrics down by document type/source; an aggregate 90% can hide one source at 40%.

Common misconceptions

Scenario walkthrough

A support assistant over 40,000 KB articles (updated weekly) started giving confident, wrong answers this week. Users ask both natural-language questions and exact error codes. What is the FIRST step and the right architecture?

1. FIRST step — pull traces and check whether the gold chunk is retrieved. It surfaced that this week’s re-ingest changed the chunker (drift) — a retrieval problem, not a prompt problem.
2. Chunking — structural/parent-child for articles: child chunks for precise matches, parent sections for context. Version the config.
3. Retrieval — hybrid (dense + BM25): error codes need exact-term matching that dense misses.
4. Reranking — cross-encoder to top-8; the gold article was retrieved but ranked #14 pre-rerank.
5. Eval — recall@k, MRR, faithfulness, per source; gate re-ingests on these.
6. Citations — attach spans so answers are auditable (provenance test).

Rejected alternatives: rewriting the system prompt (the retrieval was broken), fine-tuning on the KB (facts change weekly — RAG’s job), dense-only retrieval (misses error codes), and trusting the aggregate score (masked the failing source).

Key takeaways

• RAG for large/changing corpora + citations; long context for small stable; fine-tuning for style, not facts.
• Chunk to match data shape and query pattern; parent-child balances precise hits with context; version the config.
• Hybrid (dense + BM25) + cross-encoder reranking beats either alone; RRF fuses without score calibration.
• Separate retrieval metrics (recall@k, MRR) from generation metrics (faithfulness, answer relevance), and report per-segment.
• Debug from the trace: gold chunk retrieved? ranked? passed? used? Suspect the index first after a refresh.