eval-defect-bench

An agent-evaluation framework can fail silently: a bug in the scoring path doesn't crash, it just quietly returns a plausible-looking verdict that's wrong, the kind of defect nobody notices because nothing about the failure looks like a failure. This benchmark exists to test whether that specific class of bug ("silent verdict degradation") is even detectable by tooling that could plausibly run in CI, before anyone builds a detector and assumes it works.

The benchmark itself is the honest part: 30 real, merged bug-fix commits from google/adk-python's own evaluation module, paired with 30 verified-untouched controls, frozen and scored blind.

Three independent detection approaches were built and run against the frozen 60-record benchmark: a hand-built AST scanner (5 pattern detectors), a local three-model Ollama consensus (llama3.1:8b, gemma2:9b, qwen2.5:7b, majority vote), and a single frontier-model judge scoring an isolated function body blind. Two pass/fail gates were pre-registered before the frontier-judge run: a mechanism-naming gate (EXACT localization at least 40% AND adjudicated false-positive rate at most 20%) and a looser triage gate (balanced accuracy at least 70% AND adjudicated FPR at most 15%).

None of the three detectors passed either gate. That's the headline result, not a footnote, the repository exists to make the negative result, and the benchmark that produced it, reusable by the next attempt instead of something that has to be rediscovered from scratch.

0 / 3, show source for Detectors that passed the pre-registered viability gate

Cited in provenance.md

`README.md:35-38`

verified against source 2026-09-23

Detectors that passed the pre-registered viability gate · mechanism-naming gate (EXACT localization at least 40% AND adjudicated FPR at most 20%) and triage gate (balanced accuracy at least 70% AND adjudicated FPR at most 15%), both pre-registered before the frontier-judge run. Neither was cleared by any baseline

0% recall on 30 held-out positives, never run against the control set, show source for AST scanner (5 hand-built pattern detectors)

Cited in provenance.md

`README.md:25,29-33`

verified against source 2026-09-23

AST scanner (5 hand-built pattern detectors) · its 75% recall on the 4 examples it was hand-built from is not a valid generalization comparison

53.3% recall, 36.7% adjudicated FPR, 58.3% balanced accuracy, d-prime 0.42, show source for Local LLM consensus (llama3.1:8b + gemma2:9b + qwen2.5:7b, majority vote)

Cited in provenance.md

`README.md:26` (leaderboard table)

verified against source 2026-09-23

Local LLM consensus (llama3.1:8b + gemma2:9b + qwen2.5:7b, majority vote)

13.3% recall, 6.7% adjudicated FPR, 53.3% balanced accuracy, d-prime 0.39, show source for Frontier single judge (blind, isolated function body)

Cited in provenance.md

`README.md:27` (leaderboard), `README.md:85-107` ("Verification" section, reproduced output block)

verified against source 2026-09-23

Frontier single judge (blind, isolated function body) · raw numbers (13.3% recall, 10.0% raw FPR, 51.7% balanced accuracy, d-prime 0.17) independently reproduced by transforming the frontier judge's own answers into scripts/score.py's documented input format and re-running it

15 / 30 (50%), show source for In-class taxonomy coverage (silent verdict degradation, the benchmark's actual target class)

Cited in provenance.md

`README.md:109-135` (defect taxonomy table), `README.md:145-157` (keras#23420 merge detail)

measured 2026-09-15

In-class taxonomy coverage (silent verdict degradation, the benchmark's actual target class) · across 3 primary shapes (silent accumulation loss, value not consulted, boundary strips attached data), each cross-referenced against real external google/adk-python PRs/issues; one of those, keras#23420, is a merged, contested, evidence-defended fix in a major framework, landed 2026-09-15

If you're evaluating whether an off-the-shelf or lightly-built detector can catch this class of bug before it ships, this is the reusable ruler: a frozen 60-record benchmark, a standalone scoring script, and three honestly-reported failed attempts to save the next one from repeating them.