The Fix Was Merged, but the Numbers Were Still Wrong

We fixed a real bug in how the dashboard scores developers: the Impact Score was double counting rework, First Attempt Success divided by the wrong denominator, and we added a new reworkedLines signal. The fix was reviewed and merged to main. And the dashboards in production still showed the old, wrong numbers. Nothing was broken about the fix. The lesson lives in the gap between merged and live and correct, and it has two halves.

Half one: derived state does not fix itself

A developer snapshot is not a fact we store, it is a fact we compute. Every snapshot is calculated from the pull requests underneath it: merge rate, rework, size, and the Impact Score on top. We then write the result into a DeveloperSnapshot row so the dashboard can read it back fast, without recomputing on every page load.

That storing is the catch. A snapshot row is a cache of a calculation. When you change the calculation you change what new snapshots will say. You do not touch the rows already sitting in the table. They were written by the old formula and they keep the old numbers until something rewrites them.

So "the formula is fixed in the code" and "the stored numbers are right" are two different states of the world, and the first does not imply the second. Anything derived, whether a snapshot, a cached total, a denormalised count, or a search index, carries this property: change the recipe and you have to rebuild the thing the recipe made. We have a script for exactly that, recompute-snapshots.ts, which rebuilds every snapshot from its pull requests.

Half two: production is not your laptop

Knowing we needed a recompute, the naive plan is "run the script against prod." Scoping it turned up three reasons that does not work, and each is a way production quietly differs from local:

• The server was not even on the new build. Our server is manual deploy, and its last deploy predated the merge. The new reworkedLines column does not exist in the production database until a deploy runs prisma migrate deploy. Recompute first and there is no column to write into, and it would run the old code anyway.
• The script was not in the image. The Dockerfile copied prisma, tsconfig and src, but not scripts/. The recompute script was simply not present in the container, so there was nothing to run.
• The database is internal only. Production's DB is not reachable from a laptop; its connection string points at an internal host. Running the script locally would mean briefly opening external access and handling the DB password, exactly the sort of temporary hole you forget to close.

None of these is exotic. They are the ordinary difference between the machine you wrote the code on and the environment it actually runs in.

The decision

Run the maintenance inside the platform, on boot, behind a flag. The server already runs migrate deploy, then seed, then start on every deploy. We added one step: when RECOMPUTE_SNAPSHOTS=1, run the recompute after the migration and before the server starts. To ship the fix you set the flag, deploy once (which migrates and recomputes against the internal DB), then set the flag back.

Three properties made this the right shape:

• It runs where the data lives. No external access, no password on a laptop, no hole to close afterwards.
• It is idempotent. The recompute rebuilds from source and the XP awards are award once, so running it twice changes nothing the second time. A leftover flag costs boot time, not correctness. Idempotence is what makes a data rebuild safe to run, and safe to run again when you are not sure the first one finished.
• It is repeatable. The next metric change pulls the same lever. We did not build a one off, we built the mechanism we will use again.

The lesson

• Derived data is a cache, and caches go stale on purpose. When you change a formula, ask "what did the old formula already write down, and who rebuilds it?" If the answer is "nobody," you have shipped half a fix.
• Trace the path from merged to live. Merged to main is not deployed; deployed is not migrated; migrated is not recomputed. Each arrow is a step that can be missing. Walk them before you call it done.
• Make maintenance safe and repeatable. Run it where the data lives, make it idempotent, and put it behind a flag you can pull again, rather than a heroic one time script run by hand with the production password in your shell history.

The fix was the easy part. Getting it to change what a user actually sees, without leaving a door open behind you, is the craft.