Two weeks ago we published the story of the container that wasn't there: a production outage on the Dojo traced from the outside, a bug report written like evidence, and a European hosting team that shipped five fixes in reply. We ended that post with a confident sentence: the container that wasn't there is gone now.
On 27 July the same impossible error came back.
Loading module from ".../_app/immutable/entry/app.Cfleckv9.js" was blocked
because of a disallowed MIME type ("text/html")
Same signature, worse day. This time the page went fully blank for every visitor, the outage lasted through the working day, and every remedy that saved us in July failed. This is the follow-up, and it earns its place on this blog for one reason above the others: our report contained two diagnoses, and one of them was wrong. What came back corrected us with the same rigour we try to bring, and the correction teaches more than the outage does.
The day the workarounds stopped working
We had merged to main twelve times that day, roughly every forty minutes, which we now know is what exposed the fault. From the outside, the fingerprint looked like July but larger. We probed the entry bundle for five distinct build hashes we had observed in production, twelve to fifteen requests each, seventy two probes in total, and every single one returned 404. Shared chunks, the files whose hashes had survived across builds, kept answering 200. At least eight containers were taking turns answering for a service configured to run exactly one.
We worked through the whole remedy list from July. Suspend and resume. Scaling to two instances and back to one, twice. Two fresh deploys. The stale containers kept answering health checks and kept receiving live traffic. In July, the rescale trick evicted the orphan within minutes; this time nothing reachable through the API touched the things actually serving our users.
So we took the escape route we had never needed before: stood up a fresh service from the same repository, moved dojo.ekohacks.com onto it, and deleted the broken one. Production came back. Then we wrote the second report.
The report, and the sharper questions
We described two problems. Problem one: the builder was emitting internally inconsistent images, because every image we could reach 404ed its own entry files while our local build of the same commit was self-consistent. Problem two: old containers were surviving every deploy, rescale and suspend cycle, which is the July bug again, so the July fix had a hole.
We also asked the question underneath both incidents: is a cache-busting build argument like our BUILD_REV required for correctness on this platform? Because if it is, the build cache is unsafe by default and every customer without the workaround is exposed.
What came back: the orphans were ours
The reply opened by agreeing the platform owned the fault, then did the thing we most respect: it showed its work, and part of that work was showing us where ours went wrong.
The root cause is a sentence we did not see coming. Our pull request preview environments were being given the same internal DNS name as the production service. Every service on the platform gets a stable internal name, and the live route resolves its backend through it. Previews received the production name too, they all sit on the same project network, and Docker's DNS happily round-robined across every container holding the name. Every open PR preview silently joined the production pool and served a share of real traffic on our domain, each from its own branch build. We had a lot of open PRs that week. The eight orphans were our own previews.
Two lifecycle bugs kept them alive. Preview teardown looked containers up in an in-memory map, so any restart of the platform's agent emptied the map and teardown became a silent success that removed nothing. And a preview deploy that finished after its own teardown flipped the record back to active and registered a container nothing would ever clean up again, a race that fired hardest on exactly our workload of many PRs pushed often.
And here is why July's fix missed it. That fix taught every deploy to reap stray containers carrying the service's identity, and it deliberately skipped anything labelled as a preview, on the reasonable assumption that previews were a separate, self-contained world. The aliasing bug made that assumption false. The preview containers were invisible to precisely the mechanism built to catch them, which is also why suspend, rescale and redeploy all failed us: every one of those operations acts on production containers, and the things serving our traffic were previews.
The correction: we measured the pool, and called it the image
Our first diagnosis was wrong, and the way it was wrong is the part we will be teaching in the Dojo for years.
Seventy two probes, zero hits, on five different build hashes. We read that as an image that was missing its own files, layers from different builds mixed into one artefact. The reply pointed out the flaw with a single observation: a corrupt image fails deterministically. The same file 404s on every request, from every client, forever. Our failures varied by which container answered. Per-build entry files 404ed because only one container in a pool of eight had any given build's entry chunk. Shared chunks answered 200 because they existed in all of them. That is the routing signature, the very one we had diagnosed correctly ourselves in July, wearing more containers.
The zero out of seventy two still bothered us, because a fair round-robin across eight containers should have landed roughly one probe in eight on the right one. The explanation is the kind of detail you only get from the people who run the proxy: the edge pools and reuses upstream connections, and DNS ordering is sampled only when a new connection opens. A probe run gets pinned to a small subset of the pool. Our seventy two requests were sampling perhaps two or three containers, and the one holding the hash we probed for was simply outside the pinned set.
The lesson generalises to every system with a load balancer in front of it. A probe measures the path it travels, and through a shared front door that path is the pool. If you want to fingerprint an artefact, you must reach the artefact; if all you can reach is the pool, then variance across requests is your signal, and a uniform answer from a varying pool is the one result you should refuse to trust. We had written in July that a fingerprint turns a complaint into a report. The amendment from this incident: label your fingerprint with the thing it actually identifies.
It is worth saying plainly why being corrected felt fine. We had published numbers precise enough to be re-run, so the correction was cheap, specific and verifiable. The engineer re-ran our own probe against the replacement service and posted the healthy result in their reply: fifteen fetches, one entry bundle, eight straight 200s of application/javascript. Evidence invites correction, and correction is the fastest way to be right by the end. That is the deal we teach, and this thread is the cleanest example of it we own.
The answers, and the fixes
Every question got a direct answer. The cache-busting question closed cleanly: BUILD_REV is optional. Since July every build is pinned to the exact commit being deployed, checked out detached and verified before building, and a COPY layer is keyed on a checksum of the files it copies, so changed source can never match a cached layer. The one-second fully-cached deploys we still see on main are the healthy case now, a squash merge producing a tree identical to the branch build the platform already made. We keep our BUILD_REV for explicitness, and we keep it knowing it is a comfort.
Three structural fixes shipped on their side. Previews now get their own namespaced DNS name and can never hold the production one, so a leaked preview stays out of the production pool by construction. Teardown resolves containers by their Docker label, durable state that survives any restart. And a deploy that completes after its preview was destroyed re-issues the teardown for the container it just booted.
One more thing came out of our report. The platform swept every host for this class of orphan and found them running for other customers too, one container quietly alive for a month, and removed them all. That is the second time an afternoon of our evidence has fixed something for every team on the platform.
Fixing the mechanism and fixing the class
The July fix was real, and it addressed exactly what the July evidence showed: stray containers carrying the service's identity. The class of problem was wider than the mechanism found, and the gap sat behind an assumption that looked safe. This is a pattern every codebase we teach in has somewhere: the fix that cleans up the instances, and the fix that removes the shared name so the collision has nowhere to happen. The second kind costs more thought and is the only kind that ends a saga. Production and preview containers on our platform now share nothing they could collide on, which is why this post can be a follow-up rather than a chapter.
We stand by every word of the July post about building on European infrastructure, and this incident is the test of it, because conviction is easy between outages. The second failure is the moment the easy conversation about migrating becomes genuinely tempting. We sent evidence instead, again, and got back a root cause, a correction, a platform-wide sweep and a structural fix, again. The relationship compounds precisely because both sides keep showing their work.
The container that wasn't there is gone. The previews that served production have their own names now. And the seventy two probes that measured the wrong thing have a permanent place in our syllabus, right next to the lesson they paid for: measure first, then make sure of what your measurement is touching.
