reproducible benchmarkdataset v1

Which retrieval pipeline
actually wins?

Four pipelines, one golden set, identical prompts. Recall, ranking quality, faithfulness, citation validity, abstention, latency and cost — reported with bootstrap confidence intervals instead of vibes.

4

pipelines

40

questions

95%

CIs

What this proves. It answers one question: which retrieval setup is actually better? Four pipelines run on the same 40 questions, with the same prompt and judge, and report quality, latency and cost with confidence intervals — instead of a single cherry-picked answer.

Leaderboard

no completed benchmark yet

Corpus

PDFs stay local; only metadata is committed

No corpus

run ingest

embed

bge-small

rerank

MiniLM-L6

judge

Qwen 3.8

How to read this

the numbers are the product

  • Recall@5 is doc-level: did the gold paper make the top 5 chunks?
  • Faithfulness decomposes the answer into claims and checks each against retrieved text.
  • Citation validity is computed programmatically: every cited id must be in the retrieved set, and an answer with no citations scores 0.
  • nDCG@10 uses binary gold-chunk relevance, so a perfect rank-1 gold chunk scores 1.0.