We had a scoreboard we were pleased with. The mocked suite: 44 tests, 1.31 seconds. The rewritten sociable suite: 70 tests, 13.50 seconds. Slower, obviously, because it talks to a real database, but twenty six tests richer. More thorough. That was the reading, and we published it.
Then we pulled the actual scenario titles out of both suites and compared them line by line, because a count is not a list. The reading did not survive.
Twenty six is a net, and nets hide things
The plus twenty six is two movements in opposite directions. A new WakaTime subsystem, untested before, brought thirty three tests with it. The files that existed in both versions went the other way, from 44 to 37.
That minus seven contains two completely different things, and only one of them is fine.
Some of it is legitimate collapse. The old suite had four assertions shaped like should route push events to handlePushEvent, each checking that a function had been called on a mock. The rewrite replaced them with assertions about what actually happened: dispatches push events to the push handler which records a GitMetric. Once you assert the effect, "was it called" is a question about your own test scaffolding rather than about the system. Collapsing those is right.
The rest is loss.
The four that vanished
| Behaviour | In the mocked suite | After the rewrite |
|---|---|---|
| 500 when an event handler throws | should return 500 if handler throws error | gone |
| 500 when Qodana processing throws | should return 500 if processQodanaReport throws | gone |
| 400 when the event header is missing | should return 400 for missing event header | gone |
| A report with issues earns no clean scan XP | should handle payload with issues and fixes | gone |
We checked each one against the rewritten source in case it had survived under a different name. None had. The first three are error paths. The fourth is an ordinary path we simply stopped asserting: the new tests check the clean report thoroughly and never construct a dirty one.
Nobody decided these behaviours had stopped mattering. There was no meeting. They went because the new style made them expensive.
Mocks are very good at exactly one thing
This is the part we would rather not have found, and the part most worth saying out loud.
Every behaviour on that list requires the system to do something other than succeed. With a mock, arranging that is one line: tell the stub to throw. Without a mock you have to construct a collaborator that genuinely fails at the right moment, or build a real fixture in a genuinely dirty state, and that is an afternoon rather than a line.
So injecting failure is free under mocks and deliberate work under everything else. Any honest migration has to budget for it. If you do not consciously reinstate the failure behaviours, the migration deletes them for you, and the test count goes up while it happens, which is the cruel part. The number that would normally reassure you is the number that hides it.
What it did to the comparison
It also meant our scoreboard was never comparing like with like. Two suites that do not cover the same behaviours cannot tell you what a technique costs. So we stopped comparing suites and started comparing implementations of a fixed specification: a verified union of 71 behaviours, being every behaviour the sociable suite really covered, plus the four that had been dropped, reinstated.
Then we built the mocked implementation of that same specification and measured it on the same commit and the same host. All four reinstated behaviours went green, three of them with a single mockRejectedValueOnce each, which is the finding restating itself in the most literal possible way.
| Mocked, matched | Sociable | |
|---|---|---|
| Behaviours | 71 | 67 plus 3 harness tests |
| Wall clock, mean of ten | 2.29 s | 12.80 s |
The corrected gap is 5.6 times, where the confounded scoreboard had implied roughly ten. The entire remaining difference is the database.
What we take from it
Test count is not coverage of behaviour, and neither is a coverage percentage, which the first post in this series showed rising as the tests touched less of the system. The only instrument that caught this was reading the scenario titles, by hand, and asking of each one: which behaviour is this, and does the other suite have it.
That is a slow instrument. It is also the one that found four missing behaviours in a suite whose headline numbers were all moving the right way.
Next in this series: what happens to those 12.80 seconds when the database goes behind a Nullable, and whether the fidelity survives it.
