tracegauge

Developers using Claude Code (or similar coding agents) have no real visibility into whether a given session used tokens well or burned through them on repeated failed attempts, redundant reads, or a wandering plan. Efficiency claims about coding agents are almost always anecdotal, "it felt fast" or "it felt wasteful", rather than measured against anything.

tracegauge scores a finished session's token economy, trajectory quality (was the sequence of steps purposeful), and deterministic waste, and does it against the developer's own historical baseline rather than an arbitrary universal number, entirely locally, with no data leaving the machine unless the developer explicitly opts in.

The CLI (tes) reads a Claude Code session's JSONL log from ~/.claude/projects, redacts secrets at the moment of ingestion, and computes three independent axes rather than one combined score. Token economy compares the session's token usage against a p25–p75 band (the middle 50% of past sessions) built from the developer's own history for that same kind of task. Trajectory quality is rated by a local LLM judge running Qwen3-30B via Ollama (with an opt-in API path for judging), scoring whether each step in the session was purposeful. Deterministic waste is caught by two rule-based detectors, repeated-failed-retry and redundant-read, each of which attaches the specific turns that prove the finding, rather than asserting waste without evidence.

There is deliberately no composite score: token economy, trajectory quality, and waste each carry their own caveats about where they're valid and where they're not, and collapsing them into one number would hide exactly the honesty that makes each axis useful on its own.

A background watcher plus a dashboard that only listens on localhost automatically scores sessions as they finish, and the CLI also exposes score, monitor, budget, and patterns subcommands directly. Nothing phones home except two explicit, separately-consented opt-ins.

A local ingestion and feature layer feeds three independently-computed axes, no composite score, into a local store, surfaced through a CLI and a localhost-only dashboard.

1. A reference-based, pointwise judge instead of a pairwise one

   Published evidence on LLM judges showed pairwise comparison ("which of these two is better") flips its verdict 35% of the time under small perturbations, versus 9% for pointwise judging ("score this one against a fixed reference") once a reference standard exists. That gap is large enough that it decided the architecture rather than being a footnote.

2. The heuristic-primary design was abandoned after it was actually tested

   Four candidate rule-based waste heuristics were built and checked for inter-annotator agreement, do independent raters applying the same rule agree with each other. Only one of the four cleared the 0.60 bar (it scored κ=0.825); the other three scored 0.15, 0.43, and 0.19. The failing three all needed cross-turn intent inference that no phrase-matching rule can capture, which is what forced the pivot to a hybrid architecture rather than a rewrite of the same rules.

3. The trajectory judge is deliberately a different model family than the agent it judges

   The agent being measured is Claude; the judge runs on Qwen instead. That's a structural choice, not an accident, a judge from the same family as the thing it's scoring creates an obvious incentive for self-enhancement bias (a model rating its own family's outputs more favorably), so the judge is built to be structurally incapable of that.

4. No composite score, by design

   Three labeled signals with their own caveats are more honest and more useful than one number that hides which axis is weak. A single score invites treating a session as "efficient" or "wasteful" overall when in reality one axis might be strong and another might not apply cleanly to that kind of task at all.

5. Local-by-default, enforced by construction

   The dashboard binds to localhost by construction, not by a settings toggle someone could leave misconfigured. Secrets are redacted at ingestion before anything is stored, and every network call the tool can make is a separate, explicit opt-in rather than a bundled default.

643/643 passing, show source for Test suite

Verified source

**Wave 19 correction (2026-07-31):** `CHANGELOG.md:92` ("643/643 tests green, up from 601 in 0.9.0; 43 new tests"), confirmed live via `pytest --collect-only` (643 collected). The prior 601 figure came from `README.md:20` — a scope-limited claim ("The code, the content-free guard, the consent flow, and `tes corpus withdraw` are all built and tested (601 tests green)") describing only the 0.9.0-era community-corpus feature, not the whole suite; it was misread as a whole-suite result. Baselines citation (`README.md:207`) unchanged.

• CHANGELOG.md (CURRENT_STATE.md:82's 601 is a 0.9.0-scoped feature subtotal, not the whole-suite count):92 ↗

commit cd4087e · measured 2026-08-05

Test suite · baselines built from 75 quality-gated sessions across 5 task types

84% strict agreement, 96% top-2, Spearman ρ≈0.79, show source for Judge validation against a reference LLM

Cited in provenance.md

`README.md:67,208`

verified against source 2026-10-10

Judge validation against a reference LLM · honest caveat: there are no human gold labels behind this, it's LLM-vs-LLM agreement, not ground truth

repeated-failed-retry rate ~1.4% in the wild vs. 6.6% in the calibration pool, show source for Generalization: 172 developers, 1,053 real sessions

Cited in provenance.md

`README.md:65,126,210`

verified against source 2026-10-10

Generalization: 172 developers, 1,053 real sessions · the calibration pool is honestly labeled a high-waste outlier, not a representative population

installable from PyPI, show source for Distribution

Cited in provenance.md

Same live mechanism as `adk-tracegauge:pypi` above. **Corrected 2026-08-18:** this row read "Live on PyPI v0.10.0", cited to `pyproject.toml` and a `pip index versions` run on 2026-07-12. The registry now serves **0.12.0, 14 releases** (`pypi.org/pypi/tracegauge/json`, HTTP 200, fetched 2026-08-18), so the page had been two minor versions behind for over a month with nothing comparing the two. Found by putting a live version on the card and noticing it disagreed with the case-study body

measured 2026-08-18

Distribution · pip install tracegauge; the project card carries the version and release count the registry is serving right now

Community-corpus feature held dormant; habit-coach feature unshipped, show source for Two features built and deliberately not shipped live

Cited in provenance.md

`CURRENT_STATE.md` 0.9.0/0.10.0 sections

verified against source 2026-10-10

Two features built and deliberately not shipped live · "a community corpus with zero contributors delivers no user value"; habit-coach recommendations judged too thin at pre-publish review

Four waste-detection heuristics were built, each with a written rubric, and tested the honest way: independent human annotators applied each rule and their agreement was measured with Cohen's kappa (a statistic for inter-rater agreement that corrects for chance). The bar was 0.60. Three of the four heuristics scored 0.15, 0.43, and 0.19, well below it. One case was especially telling: a capable model applying one of the failing heuristics systematically over-fired it on 23% of turns, which ruled out "the annotators are just being noisy" as the explanation. The disagreement was conceptual, not statistical noise.

The postmortem's conclusion was structural rather than incremental: these three concepts need cross-turn intent inference, understanding what a sequence of steps was trying to accomplish, that no phrase-matching rule can capture, no matter how the rubric is worded. As the postmortem put it, any further rubric patch "reduces to 'the annotator decides per instance', which is what κ=0.15 already measures." Patching the rubric a third time wouldn't have fixed anything; it would have just re-measured the same disagreement.

That finding forced the full pivot away from a heuristic-primary design toward the three-layer hybrid that actually shipped: deterministic features for the things rules genuinely can catch, a reference-based LLM judge for the things that need judgment, and human-gold calibration to keep the judge honest.

If you're building any rule-based detector where "this seems right" is currently the only validation, this is the test worth running: measure inter-rater agreement before you trust a heuristic, and be willing to structurally rearchitect around what the measurement actually shows instead of patching the rubric again.