Back to Ideas 5 min read

The Failing Test Is the Review

E
EkoHacks Team
·
The Failing Test Is the Review

Encoding the release, part 2 of 8.

Open the commit history of ekohacks/cli and you will see the same shape over and over:

feat: cut stops when the release branch already exists
test: cut stops when the release branch already exists
feat: the nulled gh answers waiting run rounds in order
test: the nulled gh answers waiting run rounds in order

Every behaviour is two commits. The test: commit lands first and it is red — a failing test, committed to main on purpose. The feat: commit follows with the production code that makes it green. Our agent notes state the contract bluntly:

A failing test on main is not a bug. It is the committed specification for the next piece of work. Make it green by writing production code; never edit a committed red test to make it pass, and never commit production code in the same commit as the test that specifies it.

Why work this way? Because the red test is the review artefact. A reviewer looking at a test: commit is reviewing pure intent: here is the behaviour we claim to want, expressed as running code, with no implementation to be seduced by. Arguments about whether the behaviour is right happen at the cheapest possible moment — before the implementation exists. By the time the feat: commit arrives, the only question left is "does this satisfy the committed spec?", and the suite answers that mechanically.

It also keeps the work honest in the other direction. "Never edit a committed red test to make it pass" means the implementation must rise to the specification, not the other way round. And the refactor step keeps its classical meaning: a refactor: commit changes no observable behaviour and therefore needs no new test — the moment a change would alter what a test observes, it is not a refactor, it is a red test waiting to be written.

Honesty requires admitting what this costs. A committed red test is live ammunition. Ours run directly on Node's type stripping — types are removed, not checked, at run time — and we once committed a red test that called an option that did not exist yet: NpmWrapper.create({ cwd }) before cwd was implemented. The option was silently ignored, and the red test ran npm version against our own repository instead of its temporary directory, cheerfully bumping the CLI's own version mid-test. We caught it, reverted it, and wrote the lesson into our agent notes: typecheck the red half before committing it, and point any red test that reaches real infrastructure at throwaway state. Red-first is a discipline, and disciplines have safety rules.

What we get in exchange is a repository where the suite is the specification — not in the aspirational sense that phrase usually carries, but literally: every behaviour the tool has was a reviewed, committed, failing test before it was code. When part 6 of this series adds a brand-new preflight check in five commits, you will be able to watch the specification arrive before the behaviour, in public.

Next: Testing a Release Without Releasing.

E

Written by

EkoHacks Team

More from Ideas

·6 min read

What the Nullable Gave Back

One file, seven behaviours held fixed, the database swapped for a Nullable: about 180 times less time inside the tests, and coverage flat to two decimals.

E
EkoHacks Team
·6 min read

Twenty Six More Tests, Four Fewer Behaviours

Removing the mocks grew the suite from 44 tests to 70 and quietly deleted four behaviours, every one of them a failure path. Test count is not coverage.

E
EkoHacks Team
·6 min read

The Best Coverage Number in the Room

Same commit, same spec, same test count. The mocked suite ran 5.6 times faster, covered 3.5 fewer points of real code, and posted the best branch coverage.

E
EkoHacks Team