reproducible benchmarkdataset v1
actually wins?
Four pipelines, one golden set, identical prompts. Recall, ranking quality, faithfulness, citation validity, abstention, latency and cost — reported with bootstrap confidence intervals instead of vibes.
4
pipelines
40
questions
95%
CIs
What this proves. It answers one question: which retrieval setup is actually better? Four pipelines run on the same 40 questions, with the same prompt and judge, and report quality, latency and cost with confidence intervals — instead of a single cherry-picked answer.
Leaderboard
no completed benchmark yet
Corpus
PDFs stay local; only metadata is committed
No corpus
run ingest
embed
bge-small
rerank
MiniLM-L6
judge
Qwen 3.8
How to read this
the numbers are the product
- Recall@5 is doc-level: did the gold paper make the top 5 chunks?
- Faithfulness decomposes the answer into claims and checks each against retrieved text.
- Citation validity is computed programmatically: every cited id must be in the retrieved set, and an answer with no citations scores 0.
- nDCG@10 uses binary gold-chunk relevance, so a perfect rank-1 gold chunk scores 1.0.