The previous post ended on a promise. We had found two holes in our pipeline: a documentation gate that could report a failure but never cause one, and, worse, no check whatsoever on the API specification that our web client is typed from. The first fix had merged. The spec check, we said, was following behind it.
It followed. It went red on its first run. And the reason it went red is the most useful thing we have learned this week.
The check
The shape is unremarkable, and that is rather the point. Regenerate the artefact from its source, and fail if the result differs from what is committed:
- name: Check API spec and web types are current
id: api_types
run: |
npm run api:types
git diff --exit-code -- web/src/lib/api/openapi.json web/src/lib/api/generated.d.ts
continue-on-error: true
with the matching failure step at the bottom of the file, where this series has now twice insisted it belongs.
We had tested it before pushing, in both directions, which we now do as a matter of course. On a clean tree it passes. Rename a route's summary without regenerating, and it fails and names the two files that drifted. We were, in the private way one is about a small piece of infrastructure, pleased with it.
The first run
Run npm run api:types
GITHUB_CLIENT_ID/SECRET not set; GitHub login is disabled
Wrote OpenAPI spec to .../web/src/lib/api/openapi.json
diff --git a/web/src/lib/api/generated.d.ts
- "/api/v1/auth/github": {
Read the two halves of that together. The pipeline, generating the specification from the source, produced one that does not contain the GitHub login route. The committed specification does contain it. So the check did what it was built to do and reported a difference.
Nobody had forgotten anything. Here is the code the warning comes from:
const clientId = process.env.GITHUB_CLIENT_ID;
const clientSecret = process.env.GITHUB_CLIENT_SECRET;
if (clientId && clientSecret) {
// register /api/v1/auth/github and its callback
}
The routes exist when the credentials do. The committed specification had been generated on a laptop that had them in its environment file. The pipeline has no environment file at all, so the pipeline generated a smaller, differently shaped, and entirely correct specification of the server as the pipeline is able to run it.
What that actually means
The specification was never a function of the source code. It was a function of the source code and of whose machine last ran the generator.
Sit with the consequence. Two engineers on this team, one with GitHub credentials in their environment file and one without, regenerate the file after an unrelated schema change. They produce two different specifications. Each is correct on the machine that made it. Each is wrong everywhere else. Neither of them has done anything careless, and the committed file simply records whoever pushed most recently.
Now notice where that defect was sitting. It was inside the subject of the gate we had just built. The gate's entire purpose is to detect drift between a generated artefact and the source it is derived from. Its subject was not determined by that source. A drift detector pointed at a nondeterministic artefact does not detect drift. It detects the difference between two environments, and it will report that difference forever, on every run, for reasons that have nothing to do with the thing it was built to protect.
That is how gates die. Not by being removed, but by becoming noisy for a reason nobody can fix in a hurry, until somebody adds continue-on-error and moves on, and then the check is theatre and we are back where the last post started.
Nothing would have revealed this
The specification had been in the repository for months, regenerated by hand many times. Every one of those regenerations was performed on a developer's machine, and every one of them quietly reasserted that developer's environment as the truth. The file was always correct according to the only test anybody applied to it, which was that it looked right and the frontend compiled.
There is no code review that finds this. The diff, on the day the routes first appeared, looked exactly like a diff that should have appeared. There is no test that finds it, because the file agrees with itself. The only instrument that could ever have surfaced it is an attempt to regenerate the artefact somewhere with no environment, and then to insist, mechanically, that the two agree.
That is what a pipeline is. It is not a machine that runs your tests. It is the one participant in your project that has no laptop, no history, no accumulated configuration, and no memory of what used to work. It is the only honest reader of your repository, and this is why a repository that has never been built anywhere else is a repository full of assumptions nobody has met.
The repair, and its smaller cousin
Two lines, before the server is built:
// The spec must describe the whole API surface, not the surface this particular
// environment happens to enable.
process.env.GITHUB_CLIENT_ID ||= 'openapi-export-placeholder';
process.env.GITHUB_CLIENT_SECRET ||= 'openapi-export-placeholder';
No OAuth call is made. The routes are only described. Export with the real credentials or without them and the output is now identical, which is the property the gate needed all along and never had.
There was a second, quieter instance of the same thing in the very same step. Booting the server to read its specification constructs the database client, which refuses to exist without a connection string, although nothing in the export ever opens a connection. So the step needs a DATABASE_URL, and the honest value for it is one that cannot possibly work:
DATABASE_URL: postgresql://unused:unused@127.0.0.1:1/no_connection_is_made
Port one, nothing listening. We checked that the export succeeds against it, which both proves that no connection is made and guarantees a loud failure on the day somebody makes the export depend on a real database without noticing. A fake value that would work if used is a trap. A fake value that cannot work is a test.
The rule underneath
Before you can check that a generated file is current, the generator has to be a pure function of the thing it is generated from. This sounds obvious written down, and we did not know it about our own generator until a machine with no environment told us.
So when you write the check, spend a minute on what the generator reads besides its source. Environment variables. The clock. The network. The order the filesystem hands back a directory. The locale, which will one day sort your keys differently on somebody's machine and produce a diff nobody can explain. Each of those is a way for the artefact to depend on where it was made, and each of them turns a gate into an intermittent alarm.
We nearly shipped a gate like that. The thing that saved us was not care. It was that we let it fail, in the one place that could tell us the truth, and then read what it said instead of making it stop.
The practice of continuous integration described in this series is set out in the continuous integration chapter of The Art of Agile Development by James Shore and Shane Warden.
