QA: evidence, not a green checkmark

qa principle

An agent saying "done" is a claim. QA.md is what turns it into evidence.

Three questions

Every QA.md must let an agent answer all three, without asking a human.

question	failure it prevents	answered by
Did I run the right thing?	fast subset where the change needed full; wrong package	tiers + selection table
Did it actually run?	0 collected, swallowed import error, everything skipped, early exit, stale cache	`expect` baseline per tier
Does the result make sense?	green tests, wrong output: empty, doubled, off by 1000×	oracles

1. Tiers: what to run

Named tiers with a command and a trigger, for example fast for every change, full before merge, and slow/e2e when a boundary changes. There's also a selection table mapping kinds of change to a tier. Commands must match CI. A mismatch between the two is a papercut.

2. Expect: proof it ran

A baseline per tier, stated as floors and lists, not exact numbers:

expect:
  collected_min: 400        # a sudden drop means collection broke
  skips: [test_s3_live]     # any other skip is a failure
  warnings_max: 5
  seconds: [20, 90]         # too fast is as suspicious as too slow

If a change moves the baseline on purpose, update expect in the same commit. This is the same freshness rule as for docs.

3. Oracles: does it make sense

Ordered from strongest to weakest:

kind	example	catches
reference case	`fixtures/inv-2024-03` → known totals	wrong logic
invariant	Σ in = Σ out; rows after left join = rows left; key unique	lost or duplicated data
bounds with units	spot price −500…5000 EUR/MWh; daily volume 10–40 GWh	unit and scale errors
diff vs main	same input through both versions; every diff must be explained by the change	unintended side effects
negative control	a new check is seen failing once before it passes	checks that can't fail

Provenance: confirmed means a human vouched for it. proposed means an agent derived it from code or data. Agents may propose oracles. They never promote them to confirmed. Domain truth comes from humans.

Not evidence

reasoning about the code ("this should work")
tests edited in the same change until they pass, meaning assertions got weakened
a mock replacing the unit under test
new skip / xfail added to get green
"output exists", because non-empty doesn't mean correct
a run against a different commit, a dirty cache, or a partial selection reported as full

QA report

Every task ends with this. A missing line reads as "not done".

qa:      fast ✓ 412 passed, 1 skipped (expect ≥400, skip listed) 41s
oracles: ref inv-2024-03 ✓ · conservation ✓ · diff vs main: 2 rows, explained: rounding fix (adr/0007)
not run: e2e (needs VPN), human to run before merge

Layout

QA.md at each project root, and in each subproject of a monorepo.
The root QA.md lists the subprojects and says what to run for cross-cutting changes.

Tooling

The tiers in frontmatter are machine-readable on purpose. - Did it run? A tool can fully automate this: run the tier, parse the summary, compare it to expect. - Does it make sense? Partially: executable oracles run, and the diff vs main can be produced; explaining the diff stays with the agent.

raw markdown