Think about the last bug that cost you an afternoon. Not the difficulty of the fix, which was probably one line, but the search: the hours before you knew where to look. Whose code was it in? Was it the library upgrade from Tuesday? Was it your machine?
Now imagine you knew, with certainty, that the entire system was working five minutes ago. The search collapses. Whatever is wrong lives in five minutes of changes, all of them yours, all of them fresh in your memory. You do not debug. You read the error message, look at the four lines you just wrote, and see it.
This is the whole argument for a green trunk, and it is not the argument people usually give. The reason to keep the build passing is not discipline for its own sake, nor tidiness, nor pride. Keeping the build green is what makes the space of possible causes small. Every hour the trunk is red, that space grows to include everything anyone has done since, plus the original breakage, plus every interaction between them.
Speed in software is not typing quickly. It is never having to search a large space.
The agreement, and why it cannot be installed
The uncomfortable part is that this cannot be bought or configured. It is a promise a team makes to each other: we will not leave the trunk broken.
Teams try to skip the promise and buy the outcome, usually by installing a server that turns red and sends email. It does not work, and the way it fails is instructive. One or two people care about the build. They set up the notifications, hoping the visibility will shame everyone into compliance. What happens instead is that the people who did not agree to the practice treat the red as somebody else's alarm, because it is. The one or two people end up fixing every break themselves, resented, exhausted, and increasingly strident. The build ends up more broken than before, because now there is someone whose job it implicitly is.
You cannot impose this practice on a team that has not chosen it. Discuss the trade offs together, decide together, and if the answer is no, then the answer is no and a pipeline will not change it. This is the rare engineering practice where the social step genuinely precedes the technical one.
Roll back before you fix forward
When the build does break, there is a reflex to fix it. The reflex is usually wrong, and the reason is arithmetic rather than machismo.
Fixing forward is a bet: that you understand the failure well enough to correct it in a few minutes. Sometimes you win. When you lose, the trunk stays red while you investigate, and everyone else's search space grows for as long as you are wrong.
Reverting is not a bet. It is a return to a state that provably worked, and it costs one command. Your work is not lost, it is in your branch and in your reflog, and you can take your time understanding what went wrong from the safety of a green trunk. The instinct that reverting is an admission of failure has it backwards. Reverting is the cheap option, and taking the cheap option quickly is what competence looks like from the outside.
This is also why a version control system you cannot revert quickly and confidently is a hazard rather than an inconvenience. If getting back to the last known good state is a research project, you will fix forward every time, because you will have no choice.
The related rule is one people break under pressure: never repair the build by editing the integration machine. If the code worked on your laptop and fails on the runner, the fault is almost always something missing from the repository, a file not added, a dependency not declared, a step missing from the build script. Patch the machine and the build goes green while the repository stays broken, which means the next person to clone it cannot build, and now the failure is invisible and inherited. Fix the repository. Let the machine stay stupid.
A check that cannot fail is not a gate
Which brings us to a discovery in our own pipeline, made while writing this series.
dora.yml runs each check with continue-on-error: true, then converts the recorded outcomes into real failures at the bottom of the file:
- name: Run server tests
id: unit_tests
run: cd server && npm run test
continue-on-error: true
# ... many steps later ...
- name: Fail if tests failed
if: steps.unit_tests.outcome == 'failure'
run: exit 1
That looks like a mistake and is in fact rather elegant. Between the two halves sits a step that posts the run's outcome to our own metrics API, because the platform we are building measures engineering health and eats its own dog food. Broken build rate is one of the things it measures. To record a red run, the job has to survive long enough to report it, and only then die. Deferring the failure is deliberate.
There are five checks with a matching Fail if step: server tests, lint, server type check, web type check, and the behaviour document gate we wrote about in the conflict nobody wrote.
There is a sixth check with no matching step.
- name: Check story coverage doc is current
id: docs_stories
run: cd server && npm run docs:stories:check
continue-on-error: true
Nothing downstream reads steps.docs_stories.outcome. It is absent from the status calculation, absent from the metrics we publish, and absent from the list of things that can turn the job red. The check runs on every commit. It prints its verdict into a log nobody opens. It has never once stopped anything.
So we went and ran it against the trunk, expecting to make a rhetorical point and find the document fine.
$ npx tsx scripts/generate-story-coverage.ts --check
APP-STORIES.md is stale. Run: npm run docs:stories
$ echo $?
1
The document had already rotted, and the drift was the very thing this series opened with. APP-STORIES.md records the line numbers of tests. It first went stale several merges back, when a change moved the developer snapshot tests, and the quota work from the first post then pushed participants.integration.test.ts down by sixty four lines on top of that. Its sibling document, the one with a working gate, was regenerated each time. This one never was. The check saw it every single run, said so, exited with a failure code, and the workflow shrugged and went green.
It is not a gate. It is a comment that consumes a runner.
Looking properly turned up a second one, worse in kind. The web client is typed from the server's OpenAPI specification, and both the spec and the generated types are committed. Change a route's schema without regenerating them and the frontend compiles happily against an API that no longer exists. Nothing checked this. Nothing had ever checked it. And it had already bitten: when the administrator plan route merged, the committed spec described no such endpoint, and stayed that way until the next pull request happened to regenerate it. The drift lasted one merge and healed by luck rather than by design, which is the least reassuring way for a problem to go away.
We are grateful for it, in a backhanded way, because it is such a clean specimen. A check that cannot fail is not a check. It is theatre, and theatre is worse than nothing, because the presence of a step named Check story coverage doc is current is read by every engineer who scans that file as a guarantee that the story coverage doc is current.
The first fix was seven lines and it has since merged, as a pull request rather than a paragraph, which is the point we have made before about turning a convention into a constraint. It went in as two commits, red then green: first the gate on its own, failing honestly against the stale document, and then the regeneration that turned it green. That is the discipline we use for a bug fix, and a pipeline deserves it as much as a function does. The spec check is following behind it.
The lesson generalises past our repository, and it is worth carrying to yours. Go and read your pipeline to the bottom. For every step that reports something, find the line that acts on the report. Where there is no such line, you have a check that is describing your codebase rather than defending it, and you have been trusting it for exactly as long as it has been lying. Then go and look for the artefacts nobody checks at all, the committed schema, the generated client, the lockfile. Those do not even have a step to be suspicious of.
What green has to mean
Underneath all of this sits a definition. Green must mean this commit could be deployed. Not "the tests we happened to run passed". Not "nothing obviously exploded".
Because the instant green means anything less than that, every downstream thing that depends on green, the deploy, the release, the confidence of the next person to pull, is resting on a claim nobody has checked. Our deploy workflow fires only when dora.yml concludes successfully. That is a chain of trust, and it is exactly as strong as the weakest Fail if step at the bottom of a file nobody reads.
That chain, and what it takes to make it worth trusting, is where this series ends.
The practice of continuous integration described in this series is set out in the continuous integration chapter of The Art of Agile Development by James Shore and Shane Warden.
