Measure retrieval before you try to fix it.
Most RAG work stops at "it returns something". The question nobody answers is whether the right passage was retrieved, how often, how far down the list, and what it cost in context. ragcheck answers those four with numbers, and lets you compare chunking strategies against the same goldset instead of guessing.
$ python -m ragcheck.cli compare corpus
6 documents, 16 questions, embedder hashing-512d-2gram, k=5
strategy chunks hit rate MRR P@k ctx chars
-----------------------------------------------------------
fixed 12 0.812 0.688 0.163 2490
paragraph 62 0.500 0.323 0.100 691
sentence 17 0.875 0.613 0.175 1736
heading 33 0.812 0.612 0.163 925
best hit rate: sentence
still missed by sentence: q4, q15
Paragraph splitting is what most people reach for first. On this corpus it is the worst of the four by a wide margin, and you would never find that out by reading the output and nodding.
Python 3.9 or newer. No required dependencies, including for the tests.
- Language: Python 3.9–3.13, no required dependencies, not even for the tests
- Embedders:
HashingEmbedder(offline, deterministic, the default) andOpenAIEmbedder(real API, same metrics, behind the same interface) - CI: GitHub Actions — ruff, the full suite across all five supported Python versions, and a job that gates the build on actual retrieval quality
- 84 tests, zero dependencies, under 4 seconds — see Tests
- 0 lint findings —
ruff check . - Retrieval quality is CI-gated, not just tested:
ragcheck.cli eval --min-hit-rate 0.8exits non-zero if the sentence strategy's hit rate drops below 0.8 against the example corpus - The
compareoutput above is this repo's own corpus, not a cherry-picked example — run it yourself and it reproduces exactly
flowchart LR
Docs["corpus/*.*\n(txt, md, html)"] --> Ingest["ingest.load_corpus\nnormalise text"]
Ingest --> Chunk["chunk.STRATEGIES[name]\nfixed / paragraph / sentence / heading"]
Chunk --> Embed["embed.Embedder\nHashing (offline) or OpenAI"]
Embed --> Store["store.VectorStore\ncosine search"]
Goldset["goldset.json\n(questions -> documents)"] --> Eval["evaluate.evaluate\nhit rate, MRR, P@k, ctx chars"]
Store --> Eval
Eval --> Report["cli compare / eval / query"]
Every chunk keeps the byte offsets it came from (document.text[start:end] == chunk.text, asserted for every strategy against every document), so a
bad score always traces back to a real span of a real file rather than a
chunking bug hiding behind the numbers.
python -m ragcheck.cli compare corpus
python -m ragcheck.cli eval corpus --strategy sentence
python -m ragcheck.cli query corpus "how are webhooks signed" --k 3
python -m unittest discover -s tests -t .
eval prints every question with the rank its first correct chunk landed at,
then lists what the misses retrieved instead. That list is the actual work: an
average tells you something is wrong, the miss list tells you what.
ok q1 rank 1 How much does the Growth plan cost per month?
ok q2 rank 5 What happens to my allowance if I downgrade my plan?
MISS q4 rank - What is the rate limit for a Growth account?
hit rate is the share of questions where at least one correct chunk made the top k. This is the one to quote, because it answers "how often does the model get anything useful to work with".
MRR is where the first correct chunk ranked. Hit rate can sit at 1.0 while every answer is buried at position five, and MRR is what shows that.
precision@k is the share of retrieved chunks that were relevant.
ctx chars is how much text got handed to the model per question. It exists because precision could not see it, which is covered below.
This is the one design decision the repo turns on.
Chunk ids change the moment you change chunking. A goldset keyed on them can only ever score the chunking it was written against, which makes comparison impossible, which removes the entire reason to have a harness.
So a goldset entry names the document the answer lives in:
{
"id": "q4",
"question": "What is the rate limit for a Growth account?",
"relevant_docs": ["api"],
"answer_contains": ["60 requests per second"]
}answer_contains narrows it without reintroducing the problem. A retrieved
chunk counts only if it comes from the right document and actually contains
the answer. That is the difference between finding the right file and finding
the part of it that answers the question, and it is usually where a pipeline is
quietly failing.
The first version scored whole-document chunks as good retrieval. Precision@k rated them highly, because relevance is binary per chunk: a chunk containing the answer is relevant whether it is 200 characters or 20,000.
A test asserting that giant chunks should score worse failed, correctly. The metric was wrong, not the test. Returning entire documents does work, and it also sends 2.7 times the text on every single call, which you pay for in tokens forever.
So avg_context_chars was added rather than the test softened, and the test
that would have missed it is still in the suite, named as such, to document
why a second cost metric is needed.
HashingEmbedder is term frequency over word unigrams and bigrams, hashed into
fixed buckets with sha1 and L2 normalised. No network, no key, no dependency,
and identical output on every machine forever.
That matters because a test that needs an API key is a test that stops running, and a harness whose numbers are not reproducible cannot be used to argue that a change helped.
It is roughly lexical, so it is weaker than a real embedding model on
paraphrase. OpenAIEmbedder is there for real runs, standard library only, and
the same harness reports the same four metrics over it. If your numbers move
when you swap, that is the harness doing its job.
Write a function, register it in STRATEGIES:
def semantic(doc, threshold=0.8):
...
return chunks
STRATEGIES["semantic"] = semanticcompare picks it up from that dict. Nothing in evaluate.py changes, and the
new strategy is scored against the same goldset as the existing four.
Every chunk carries the offsets it came from, and
document.text[chunk.start:chunk.end] == chunk.text is asserted for every
strategy against every document in the test suite. Once that holds, a bad
retrieval traces back to a real span of a real file. Once it stops holding,
every debugging session after it is wasted.
84 tests, no dependencies, under 4 seconds.
| file | what it holds up |
|---|---|
test_ingest.py |
reading, normalising, and refusing what it cannot read |
test_chunk.py |
the offsets invariant, determinism, per-strategy behaviour |
test_embed_store.py |
reproducible vectors, cosine correctness, ties break on id, OpenAIEmbedder's wire format against a local fake server |
test_evaluate.py |
metrics against hand-worked numbers, goldset validation |
test_discriminates.py |
the harness actually separates good retrieval from bad |
test_cli.py |
end to end through the command a reviewer would run |
test_discriminates.py is the one worth reading. A scoring harness that
returns similar numbers whatever you feed it is worse than no harness, because
it looks like evidence while measuring nothing. Those tests break retrieval in
ways a person would recognise as broken (40 character windows, a 16 bucket
embedding space, one chunk per document) and assert the score notices.
Put your documents in a folder, write a goldset.json beside them, run
compare. Twenty questions is enough to be useful and takes about an hour to
write honestly. Writing them yourself is the point: the questions are where
your knowledge of the domain enters the measurement.
eval --min-hit-rate 0.8 exits non-zero below the threshold, so retrieval
quality can gate a build the same way tests do.
MIT.