Two Hands on One Lever

Triggering deploy...
Failed
Error: A deploy is already in progress for this service
Error: Process completed with exit code 1.

On the morning of 22 July, the Dojo's deploy workflow failed with the message above. Nothing was wrong in production. The commit it was refusing to deploy went live twenty seconds later anyway. No user saw an error, no data was lost, and the site never blinked.

By that afternoon, our own dashboard showed a red build on trunk, the author's green streak was gone, and the failing check was named: tests. The tests had passed.

This is an anatomy of that collision. It is worth writing down precisely because nothing broke. Incidents with damage get studied; incidents without damage get shrugged off, and the shrug is how the same shape returns later with damage attached.

The system, before the incident

The Dojo deploys two services from one repository. The web frontend keeps the platform's own push trigger: every push to main builds it immediately, because a stale or broken frontend build produces no image and therefore no deploy, and a preview build of it touches nothing. The server is different. A preview of the server would boot with the production environment, including the production database, so a branch's migrations would run against real data. Its automatic trigger is switched off, and a workflow deploys it instead: when CI completes green on a main commit, the workflow asks the platform to deploy exactly that commit, guarded by a staleness check and a concurrency group.

The workflow's own header comment says it plainly: this is the only deploy path for the server. At the time of the incident, that sentence had been true for less than a day.

The incident

All times are UTC, 22 July 2026, reconstructed from the platform's activity log and the CI run history.

The manual deploy at 11:17:14 was a person doing what people had done every day before the workflow existed: merge, then reach for the lever. An identical manual deploy had happened at 10:00 the same morning and collided with nothing, because that time the automation arrived four minutes later rather than nineteen seconds. The near miss and the hit were the same act; only the timing differed.

Analysis

Three findings, in decreasing order of comfort.

First, production was correct throughout. The manual deploy built the tip of main after the commit had merged, so the code the workflow wanted live was exactly the code that went live. The refused trigger changed nothing about the outcome, only about who was recorded as having caused it.

Second, the platform's lock behaved correctly. Its guarantee is at most one deploy per service at a time, and its mechanism is refusal. A lock serialises; it does not coordinate. It cannot know that the two contenders wanted the same thing, and it is not its job to know. When both hands pull the lever the lock picks a winner, and the loser's only information is the word no.

Third, the failure was architectural, and it was ours. Two actors held the same authority over the same service with no shared understanding between them beyond that lock. The workflow believed it was the only deploy path because its comment said so. The human's muscle memory predated the comment. Automation that replaces a manual practice does not end that practice; it competes with it, and habits have very low latency.

What the dashboard said, and why that is the interesting part

The Dojo watches its own development. Its Recent Builds panel that afternoon read: twenty builds, two red, trunk health 89 percent, eight green of nine trunk builds. And on the row for 5a87925: status RED, failing checks, tests.

That row makes three claims. Examine each.

The build is red. Under our verdict model this is, strictly, correct behaviour. Any workflow that fails on a commit marks the build red, and a later failure overturns an earlier green, because a check that fails after the green means the build never really was green. The deploy workflow's conclusion arrived through the same webhook as every other workflow's, and the model did what it was designed to do.

Trunk health dropped. This follows from the first claim, and it carried consequences: the overturn reset the author's green streak and reversed the XP the green build had awarded. The gamification layer treated a lost race between two deploy paths as a developer breaking trunk.

The tests failed. This one is false, and its origin is worth tracing, because nobody wrote a line that says blame the tests. The row that stores a verdict has one boolean per check that CI reports. The code that concludes a build sets the tests boolean to false on any red verdict, a shortcut that was harmless when the only workflows able to conclude a build were the ones that run the tests. The UI then, reasonably, names whichever checks reported false. Each decision was locally sensible. Composed, they laundered a deployment orchestration failure into a named accusation against a check that had passed.

The general lesson: an observability system inherits the ontology of its writers. Ours had one category, a verdict on the commit, so a failure that was actually a verdict on the delivery attempt got filed under the only heading that existed. Integration signal and delivery signal are different measurements. A commit can be perfectly integrated and fail to ship; it can also ship while its build burns. Storing both in one boolean guarantees that one of them will eventually lie.

Mitigation, as pseudocode

The root fix is a change of grammar. A deploy trigger phrased as an imperative, start a build now, fails when anyone else is mid sentence. Phrased as a declaration, this commit should be live, it cannot collide, because a rival achieving the same state is success, not failure.

procedure ENSURE-DEPLOYED(service, C):        # C = tested tip of main
    if TIP(main) != C:
        return SKIPPED-STALE                  # a newer commit will bring its own run

    for attempt in 1..N:
        D <- newest deploy of service, in flight or finished

        if D.branch = main and D.created-at >= PUSH-TIME(C):
            wait until D finishes
            if D is live:
                return SUCCESS(adopted = D)   # someone shipped our state; losing
                                              # the race is winning the outcome
        if service is busy:
            sleep(backoff); continue

        TRIGGER(service, C)                   # a refusal here means we lost a
                                              # race that started this instant;
                                              # loop and adopt the winner
    return FAILURE

Two properties are worth stating precisely. The procedure is idempotent: run it twice, or run two of it concurrently, and the system converges on the same state with neither run failing. And it is adoptive: a rival deploy that covers the desired state is counted as this run's success, so the refusal that opened this post becomes an observation, not an error.

The deployment marker follows the same grammar. On the day, the marker was gated on our own trigger step succeeding, so a deploy that reached production without us left a hole in the DORA data that a human had to patch.

if ENSURE-DEPLOYED(service, C) = SUCCESS:     # own or adopted
    POST deployment-marker(C)                 # record the state reached,
                                              # not the actor who reached it

And the telemetry stops filing delivery under integration:

on workflow-run completed for commit C:
    if workflow is a delivery workflow:
        record deployment outcome for C       # its own stream, its own panel
    else:
        conclude build for C, attributing the workflow by name

on concluding a build RED:
    set only the check booleans this workflow actually reports;
    an unexplained red renders as unattributed, never as a named check

The unattributed rendering already exists in our UI; the writer simply never gave it the chance to appear.

What we will change

Four changes, in the order we trust them.

1. The deploy workflow adopts the convergence loop above: bounded retries, and a lost race that covers the same commit is recorded as success.
2. The deployment marker keys on the state reached, not on which actor's step reached it.
3. Delivery workflows stop concluding builds. Their failures land in a deployment stream, where a red mark would have read, accurately, deploy trigger refused, commit already shipping.
4. Policy: nobody touches the manual lever for the server now that the workflow owns it.

The list is ordered by trust deliberately. The fourth item is the one every postmortem writes first and the only one with no enforcement mechanism. We state it, and we do not lean on it; items one to three are what make item four unnecessary rather than merely requested.

Conclusion

A lock refuses; it does not coordinate. An automation that replaces a habit competes with that habit for as long as the habit's hands remember the lever. And a refused command is information about a rival's progress, which a well written trigger treats as an outcome to adopt rather than an error to report.

The quieter finding is about measurement. Nothing in production failed, and still the incident cost a streak, an XP award, and the dashboard's honesty, because one boolean was asked to carry two meanings. We have written before about writes that are safe to run twice and about a gate that went red and was right. This gate went red and was half right: right that a workflow failed, wrong about what that meant, and wrong about whom to blame. The fix is not a greener dashboard. It is a dashboard whose reds mean what they say.