corpus 26b03ce9a1c2c1d4 · k = 5 · 30 questions

Seven retrievers, the same thirty questions

Recall is the share of expected passages found anywhere in the top 5. MRR is how high the first one sat. Either alone misleads, so both are shown, and every row can be read question by question below.

Retrieval

Configuration	Recall @5	MRR @5	In CI
Dense	0.767	0.603	Gated
BM25	0.600	0.465	Gated
Hybrid · dense + BM25	0.800	0.570	Gated
Full-text	0.400	0.222	Recorded
Hybrid · dense + Full-text	0.767	0.439	Recorded
Rerank over Dense	0.833	0.490	Recorded
Rerank over Hybrid · dense + BM25	0.800	0.451	Recorded

Gated rows are compared with their frozen report on every pull request; any change in any question fails the build. Recorded rows were measured once and published as they came out. The best value in each column is in bold.

Answers over dense retrieval

Citation hit @5: 0.533; 16 of 30 answers cite the expected passage
Over retrieved gold: 0.696; of the 23 whose expected passage was retrieved
Quotes found verbatim: 51/52; 98% of citations resolve to a span

A hit means a citation overlaps the one span the golden set records, so the rate is a lower bound: several misses cite another sentence of the same PEP that also answers the question. Answered by gemini/gemini-3.5-flash (1), groq/openai/gpt-oss-120b (29), prompt 682ec1326b2f.

Question by question

Each cell is the rank at which the expected passage first appeared, 1 to 5; a dash means it was not in the top 5. 4 questions are missed by every configuration: q14, q21, q23, q29.

Question	Dense	BM25	Hybrid BM25	Full-text	Hybrid FT	Rerank dense	Rerank hybrid	Answer
q01What is the maximum line length PEP 8 asks for?	3	1	1	–	2	1	2	hit
q02How many spaces per indentation level does PEP 8 require?	1	1	1	4	1	1	1	hit
q03What does the Zen of Python say about there being one way to do something?	1	1	1	–	1	1	1	hit
q04Which quotes should a docstring use?	1	1	1	1	1	3	3	hit
q05What is the regular expression a Python source encoding declaration must match?	1	1	1	1	1	1	1	hit
q06What is an f-string?	3	1	1	1	1	2	2	hit
q07What does the walrus operator do?	1	2	1	–	3	1	2	hit
q08What does PEP 484 say about runtime enforcement of type hints?	3	–	5	–	3	–	–	hit
q09What is the canonical form of a version's release segment?	1	1	1	–	5	2	2	hit
q10Which table in pyproject.toml declares the build system?	1	2	2	2	2	2	2	hit
q11What is a build backend?	–	–	–	–	–	3	3	n/r
q12What does paramstyle indicate in the DB-API?	1	–	3	–	4	3	3	hit
q13What two methods does the with statement need on a context manager?	1	–	2	–	5	3	3	hit
q14What does yield from do?	–	–	–	–	–	–	–	n/r
q15What is a virtual environment?	1	2	1	3	2	1	1	miss
q16What is a namespace package?	1	–	5	4	1	2	2	miss
q17What is an enumeration?	–	4	4	4	5	1	1	n/r
q18What does PEP 448 generalise?	1	1	1	–	3	1	1	hit
q19What operator does PEP 465 add for matrix multiplication?	2	–	–	–	–	3	3	miss
q20What makes a function a coroutine under PEP 492?	2	–	5	4	2	5	–	miss
q21How is a project name normalised for the simple repository API?	–	–	–	–	–	–	–	n/r
q22Where may underscores appear in a numeric literal?	4	1	1	3	3	2	2	hit
q23What is a protocol class?	–	–	–	–	–	–	–	n/r
q24What does the dataclass decorator generate?	1	5	3	–	4	2	2	hit
q25How does a package declare that it ships type information?	2	–	4	2	1	3	3	hit
q26What does from __future__ import annotations change?	3	2	3	–	4	4	5	miss
q27What problem do context variables solve?	–	1	1	–	–	4	3	n/r
q28What does PEP 585 allow in place of typing.List?	1	2	1	–	2	2	2	miss
q29How is an optional type written under PEP 604?	–	–	–	–	–	–	–	n/r
q30What does a match statement compare a subject against?	1	1	1	1	1	2	2	miss