Back to Ideas 6 min read

We Are All Doing Asynchronous Integration

E
EkoHacks Team
·

There is a sentence that will sound wrong for a moment and then will not stop being true. A continuous integration server does not give you continuous integration.

Continuous integration is a practice. It has three parts: bring the trunk into your work every few hours, never leave the trunk broken, and grow your release machinery alongside the product so that shipping is always possible. A server that runs your tests after you push automates the notification. It automates none of the practice. You can own the most elaborate pipeline in the industry and integrate once a fortnight, and many teams do.

Once you see the distinction, you have to look honestly at your own setup, and what you find is uncomfortable, because the tool most of us use nudges us towards the weaker of the two available practices without ever presenting it as a choice.

Two ways to know your integration worked

Synchronous integration. You bring in the trunk, you run the full build on your machine, you push, and then you wait. You watch the build run in the pristine environment. You do not begin the next task. When it goes green, you are integrated and you move on. If it goes red, you are still holding the problem, it is the only thing you are holding, and the change that caused it is the change you were just looking at.

Asynchronous integration. You push and carry on working. Some minutes later a machine tells you whether it worked. If it did not, you must put down the thing you have now started, page the old change back into your head, and fix it.

The second is what a forge gives you by default. Push, and a workflow starts. You do not wait, because nothing is asking you to.

Why the asynchronous version rots

It rots for reasons that are entirely about people and not at all about tooling.

The failure arrives when you have already begun something else. Stopping is expensive, so you finish the new thing first, and the trunk stays red for an hour. During that hour anyone who pulls gets broken code, and cannot tell your breakage from their own. When they run the suite, your failure is sitting there in the output, and their new failure hides in the noise beside it. So they leave theirs too.

Now the build has two problems and no owner, and the cost of fixing it has more than doubled, because the two failures interact and nobody knows which came first. The next person to integrate sees a red trunk and concludes, correctly, that fixing it is not their job. Broken becomes the resting state. Teams reach the point where a red build carries no information at all, and then the pipeline has become an expensive way of generating an email nobody reads.

None of that is inevitable. A disciplined team can integrate asynchronously and keep the trunk green by running the full build locally before pushing and by dropping everything the moment a build breaks. But notice what that discipline consists of: doing, by force of will, everything that synchronous integration would have made unavoidable. The synchronous version does not require more virtue. It requires less, because it does not present you with the opportunity to walk away.

What that means for the Dojo

We have a workflow called dora.yml. It runs the server tests, the web tests, the type checks, the linter, a Playwright smoke suite, and the documentation gates. When it passes on main, a second workflow triggers a deploy. It is a good pipeline and we are fond of it.

It is also asynchronous integration, top to bottom. Nothing in it asks anyone to wait. And reading it closely turns up something sharper. Its push trigger fires on main, develop, and feature/**:

on:
  push:
    branches: [main, develop, 'feature/**']
  pull_request:
    types: [opened, synchronize, reopened, closed]

Our branches are named anastasis/doj-38-phase2b-quota-enforcement. That matches none of those patterns, so pushing a branch runs nothing at all.

Once a pull request exists, the synchronize event takes over and every subsequent push is built, which is a genuine safety net. But the window before that, the hours in which the work is actually being shaped and a mistake is cheapest to find, is unguarded. Whether our feedback arrives early or late is therefore settled by a habit rather than by a pipeline: it depends entirely on how soon we open the pull request. Ours tend to open when the slice is finished, which is the moment feedback is worth least.

The excuse we do not have

Here is where it stops being comfortable.

The reason the practice tolerates asynchronous integration at all is slow builds. If confirming your integration takes half an hour, standing there watching it is untenable, and you are forced into the weaker practice while you work on making the build faster. Twenty minutes is roughly where waiting becomes indefensible. Ten minutes is the target worth defending, because at ten minutes waiting stops being a cost and becomes a pause: you make tea, you talk to your pair about the design, you look up from the keyboard and think about the shape of the thing you are building.

Our last dozen runs of dora.yml took between two minutes forty and four minutes two.

We are inside the budget, with room to spare. The build is not the reason we integrate asynchronously. We integrate asynchronously because it is what the tool did when we were not paying attention.

What synchronous integration would actually look like for us is not exotic. Run the full build locally before pushing, the same checks the pipeline runs, in the same order. Push. Watch the run. Do not start the next slice until it is green. If it goes red, revert first and understand second.

The pristine machine, and why it exists

The one thing the local build cannot do is prove the code works anywhere but on your machine. Forgotten files, packages installed months ago and never declared, environment variables living in your shell: these are invisible from inside. Historically teams kept a spare computer for exactly this, a clean environment where the build had to pass before an integration counted. Our runner is that machine.

It only works if the machine resembles yours. We already do this deliberately, and it is disguised as a landmine: PostgreSQL runs on port 5433 locally, and so the CI service maps it the same way.

services:
  postgres:
    ports:
      - 5433:5432

That line is a small act of respect for the principle. The pristine environment is only useful as an oracle if the difference between it and your laptop is nothing but the pristineness.

When two builds are legitimate

Some checks genuinely cannot fit in ten minutes: load tests, soak tests, deep static analysis. The recognised answer is a two stage build. A fast commit build, synchronous, containing everything needed to prove the software works. A slower secondary build, asynchronous, carrying the rest.

We have this, though we did not plan it as such. Qodana runs as its own workflow, in parallel, and does not gate anything. That is a defensible secondary stage.

The warning attached to the pattern is worth repeating, because it is where the pattern usually gets abused. Most teams reach for a second stage as a place to hide a slow test suite rather than fix it, and a second stage is strictly worse than a fast first one, because the feedback it carries arrives after you have stopped caring. Prefer better tests to more stages. Ours are fast. That was work someone did, and it is why the honest practice is available to us at all.

The next question is what happens when the build does go red, and what it means that one of our checks has been quietly unable to break anything for months.


The practice of continuous integration described in this series is set out in the continuous integration chapter of The Art of Agile Development by James Shore and Shane Warden.

E

Written by

EkoHacks Team

More from Ideas

·6 min read

The Gate Went Red, and It Was Right

We added the missing check on our API specification, pushed it, and it failed on its first run. Not because anyone had forgotten to regenerate the file, but because the file was never a function of the source. It depended on whose environment last produced it, and only a machine with no environment could ever have noticed.

E
EkoHacks Team
·6 min read

Ready to Ship, Not Ready to Release

Most teams have a hidden delay between saying done and being able to ship. The goal is to be technologically ready to release at any moment, even when the feature is half built. Why our deploy workflow ships the commit that was tested rather than the newest one.

E
EkoHacks Team
·6 min read

Never Break the Build

If the software worked five minutes ago, only five minutes of change can be to blame, and debugging turns into reading. That is what a green trunk buys. It is bought by agreement, not by tooling, and one of our own checks turns out to be incapable of failing.

E
EkoHacks Team
EkoHacks

EkoHacks is a coding institute teaching employable software engineering through real projects. We build civic technology and train developers, born in Nigeria, launching in rural Greece. Global tech, local impact.

Stay Tuned

Be the first to hear about new dojos, insights, and opportunities from EkoHacks.