We have been telling cohorts to stop mocking for two years, which is easy to say and hard to prove. So last winter we started measuring our own codebase instead of asserting things about everyone else's, and we wrote the numbers down as we went, including the ones we did not like.
The cleanest measurement is a pair. Two branches sit on the same production commit, cover the same behaviour specification, and run almost the same number of tests, 71 against 70. One uses vi.mock at the collaborator boundary. The other runs sociably against a real database. Nothing else differs. Whatever separates them is the technique and only the technique.
What came out
| Mocked | Sociable | |
|---|---|---|
| Tests | 71 | 70 |
| Wall clock | 2.29 s | 12.80 s |
| Line coverage | 14.95% | 18.47% |
| Function coverage | 65.95% | 72.72% |
| Branch coverage | 79.16% | 74.19% |
Read the first three rows and the story is the one you expect. Mocks are 5.6 times faster and execute 3.5 fewer points of the codebase. The mechanism is not subtle. The mocked push and check run tests replace the repository service, the Prisma client and the XP service with stubs. Those functions never run, so they never count. A mock is, by definition, a decision not to execute the thing you are mocking, and a coverage tool is a machine for noticing exactly that.
Now read the last row again.
The number that flatters
The mocked suite posts the highest branch coverage of every condition we measured. A team reading that off a dashboard would conclude their tests were in excellent health, and they would be badly wrong.
Branch coverage only measures branches inside code that executes. The mocked suite runs a small, richly branched slice of the system, the parsing, the pure logic, the handler control flow, and it runs that slice very thoroughly indeed. Meanwhile most of the data layer never runs at all, so its branches are not scored as missed, they are simply absent from the question. Restrict the exam to the chapter you revised and your mark goes up.
This is the part worth taking to your own project. A single high coverage headline can rise as your tests touch less of the system, because the metric measures thoroughness within whatever ran, and mocking is how you shrink what runs. Line and function coverage told the truer story for us. One number on its own told a flattering lie.
The trade nobody advertises
Put the two halves together and the shape is uncomfortable: mocks are much faster because they do not run your code, and they cover less because they do not run your code. Speed and executed fidelity are not independent dials you can tune separately. For mocks they move together, in opposite directions from the one you want.
That is the honest case for mocks, incidentally. If a suite exists to give you a fast red or green on pure logic, mocking the world around it is a rational trade. The failure is not choosing speed. The failure is choosing speed and then reading the coverage number as though you had chosen fidelity.
What we did about it
The Nullable pattern is a claim that you can have both: infrastructure wrapped in a thin class that knows how to build a version of itself with only its lowest level switched off. The database, the clock and the network answer to the test, and every line of your own code between the call and the boundary runs for real.
We have been converting the dojo to it, commit by commit, with a measurement point at each step. Today's suite is 55 test files, 644 tests, and 95.98% line coverage, against the 35 tests and 9.34% we started with. That is the good half of the sentence.
The other half: it takes about four minutes. The suite grew slower as it grew truer, which is not the outcome the pattern promises, and we have not yet done the work of separating "narrow integration tests are inherently slower" from "we have been careless about where the database is touched". Getting it back to seconds without giving back the fidelity is the next thing we measure, and when we have the number, it will go in the notebook whichever way it falls.
The number for today
Same code, same specification, same test count: 5.6 times faster, 3.5 points less of the codebase executed, and the best looking branch coverage in the room belonging to the suite that ran the least.
Next in this series: the twenty six tests we gained on the way out of mocks, and the four behaviours we lost without noticing.
We teach this conversion as a four week course, Testing Without Mocks in TypeScript, two evenings a week, on your own codebase, with the measurements above handed over so you know what it costs before you spend it.
