Back to Ideas 6 min read

The Knock and the Axe

E
EkoHacks Team
·
The Knock and the Axe

There is a corner of EkoLite, our small real time backend, that most visitors would walk straight past: the shutdown. On a signal it closes the websocket, then the Mongo connection, then exits, and if any of that hangs past five seconds it exits hard with a code that says so. All of it tested, none of it complicated. Which invites a fair question. The project has no production deployment yet, so where is any of this actually useful?

The answer needs a short detour into what an operating system does when it wants your process gone. It has two instruments, and they could not be less alike.

SIGTERM, the knock

Signals are the oldest messaging system on Unix: tiny numbered notifications the kernel delivers to a process. SIGTERM is the polite one. It means please stop, and crucially, the process is allowed to hear it. You register a handler, and when the signal arrives you get to run code: finish the request you were serving, flush what you were writing, say goodbye to your database, choose your own exit code.

You have already sent this signal thousands of times. Ctrl+C in a terminal sends SIGINT, its interactive sibling. kill <pid> sends SIGTERM by default. docker stop sends SIGTERM. When Kubernetes moves a pod, scales a deployment down, or drains a node, the first thing your container hears is SIGTERM.

The key word in every one of those sentences is first. SIGTERM is not a command, it is an opening offer. The sender starts a clock the moment it knocks.

SIGKILL, the axe

SIGKILL is the other instrument, and the difference is absolute: it cannot be caught, handled, or delayed. The kernel does not deliver it to your process, it removes your process. No handler runs. No flush happens. No exit code of yours is recorded, the process simply stops existing between one instruction and the next.

The two are joined by that clock. docker stop knocks, waits ten seconds, then swings the axe. Kubernetes knocks, waits thirty by default, then swings. So every process that runs under an orchestrator lives with the same contract: you have a fixed window between the knock and the axe, and whatever you wanted to say needs saying inside it.

This is why our shutdown has a five second deadline of its own. If the goodbye hangs, we would rather exit 1 in our own words, inside the window, than be erased mid sentence after it. The polite window is exactly the space in which graceful shutdown exists. Ignore the knock and the axe makes the decision for you.

Why build it before the deployment

Here is the part that answers the fair question. Shutdown is not a thing you bolt on once you have somewhere to deploy. It is a property of running under anything that can stop you, and everything can stop you. kill on a laptop, docker stop in a compose file, a scale down in an orchestrator: every one of them opens with SIGTERM. So shutdown is not something that happens when things go wrong. It is something that happens as part of things going right, dozens of times over the life of a service. Every restart is a shutdown. Building it now, as a feature with tests, is putting the seam where the need already is, so the deploy that arrives later hangs off a shutdown that already works rather than one written in a panic.

And EkoLite holds exactly the kinds of state that make an axe murder expensive:

Live sockets. Clients hold open websocket connections for their subscriptions. Close the socket properly and the client sees a clean close frame and reconnects on its own terms. Die to SIGKILL and every client sees a network fault instead, which is a different and worse code path in every client ever written.

A database mid conversation. Our close order is deliberate: socket first, so no new requests arrive, then Mongo. Reverse it, or skip it, and requests in flight are talking to a database that vanished from under them.

A pipeline with a write back. An app method can resolve an uploaded file, run an analysis script, and write the result back so it streams to every subscriber. A SIGTERM in the middle of that is the genuinely interesting case, because the result is the product. If shutdown hangs there and times out, our exit 1 in the logs is the first breadcrumb when someone asks where their result went. SIGKILL leaves no breadcrumb at all.

An honest gap the work exposed. Writing the shutdown policy made something visible: an analysis child process is not in the close graph. A SIGTERM during a run orphans that child today, and its result never lands. That is not a hole in the new code so much as a question the code can finally ask. Before, what happens to a half finished analysis on deploy was not answerable anywhere. Now there is a tested shutdown policy with an obvious place for the answer, and that is a story for another day.

There is a quieter benefit too. Exit codes are the one sentence a process gets to leave behind, and orchestrators read them. Exit 0, clean stop, nothing to see. Exit 1 with a timestamp, a shutdown gave up, go look. A fleet that shuts down cleanly and says so is what operational maturity looks like from up close, and it is the sort of boring evidence that decides whether people trust you to run their infrastructure.

The move you can reproduce

You do not need an orchestrator to feel the difference between the knock and the axe. Start the server and knock politely:

npm run dev:server        # listens on :3001
# in another shell, or just press Ctrl+C in the server's own
kill -TERM <pid>

Watch the log. It closes the socket, then Mongo, then exits with code 0, in that order, on purpose. Now start it again and swing the axe instead:

kill -KILL <pid>

Nothing. No goodbye, no ordered close, no exit code of its own, not a line in the log. The process was there and then it was not. That gap between the two runs is the whole feature: SIGTERM is a request your code can hear and answer, SIGKILL is a removal your code never sees. Graceful shutdown is not about crashes. It is what your process does with the fixed window it is given, every ordinary time something asks it to stop.

E

Written by

EkoHacks Team

More from Ideas

·6 min read

What the Nullable Gave Back

One file, seven behaviours held fixed, the database swapped for a Nullable: about 180 times less time inside the tests, and coverage flat to two decimals.

E
EkoHacks Team
·6 min read

Twenty Six More Tests, Four Fewer Behaviours

Removing the mocks grew the suite from 44 tests to 70 and quietly deleted four behaviours, every one of them a failure path. Test count is not coverage.

E
EkoHacks Team
·6 min read

The Best Coverage Number in the Room

Same commit, same spec, same test count. The mocked suite ran 5.6 times faster, covered 3.5 fewer points of real code, and posted the best branch coverage.

E
EkoHacks Team