The Small Things the Crawler Notices

Everything so far in this series has been about whether a crawler can find a page and what it is told when it does. This post is about what it notices once it is there, which is mostly the same thing a reader notices: how long the page takes, and what happens when the address is wrong. Both are measurable from a laptop, which is why day zero had numbers for them.

What day zero said, and what it got wrong

Lighthouse on the twenty first of August, against the live site, mobile emulation: the home page scored 77 for performance with its largest element arriving after 4.5 seconds; the Dojos page scored 75 with 8.9 seconds. The post and curriculum pages were fine, in the high nineties.

Day zero also said the site loaded four font families from Google at seven weights each, and that two more families sat in the repository unused. The first part was true. The second was wrong, and we only found out by trying to delete them: the folder was ignored by git, so those two families had never been in the repository or in any build. They were on one laptop, costing nothing but disk. We are correcting the record because the series promised to.

Then we read the Lighthouse reports properly rather than skimming their scores, and the fonts turned out not to be the problem either. A browser only fetches the faces a page actually uses, so the five unused families cost a few kilobytes of declarations, not the 241 kilobytes we had written down. What cost the time was the images. The Dojos hero was a 452 kilobyte PNG of a photograph. Beside it, a 430 kilobyte thumbnail and a 413 kilobyte section image. The home page hero was a 376 kilobyte JPEG. The largest element on the Dojos page was that PNG, and the report said the page had not told the browser to fetch it first.

The change

Seven images, converted once to WebP at quality 80:

dojo-hero.png               451 KB ->   43 KB
home-hero.png               306 KB ->   35 KB
how-dojo-works.jpg          412 KB ->   58 KB
thumbnail.jpg               429 KB ->   59 KB
why-dojo-exist.jpg          471 KB ->   65 KB
why-workshop-exists.jpg     375 KB ->   45 KB
workshop-hero.jpg           286 KB ->   27 KB
total                      2734 KB ->  335 KB

The two heroes that are the largest element on their pages gained fetchpriority="high" and explicit dimensions, so the browser starts them first and reserves their space. An eighth image, a 630 kilobyte file nothing referenced, was deleted. We looked at each conversion side by side before committing it; at this size and quality the difference is not visible.

The fonts were trimmed anyway, because a declaration nothing uses is a lie about what the site is made of. The site uses two families, Outfit for headings and Inter for text, at five weights, upright only. The configuration now says exactly that, the output went from thirty seven font files to nineteen, and four tokens that named fonts nothing used, one of them still called montserrat for a family that had left the site months ago, are gone along with the forty seven classes that referenced it.

Two smaller things. The site had no error page of its own, so a wrong address showed the framework's default screen with none of our navigation on it. It now shows a page that looks like ours, says what probably happened, and offers the three places a lost reader most likely wants, with a noindex so the error itself is never a search result. And the example environment file still listed six variables for a content system we removed in June; it now lists the one variable the site uses.

One more the build found for us. With the draft post kept out of the build since post three, the crawler reported a dead link: the seventh part of an older series pointed at the eighth, which is the draft. The link is now plain text until the post exists. A link to a page that does not exist is the smallest thing a crawler notices, and it notices every time.

What the first measurement found

We deployed, measured, and read the reports again. The Dojos page had gone from 75 to 96 and its largest element from 8.9 seconds to 2.3. The home page had gone from 77 to 81. Its largest element was now its headline, plain text, and the report said the text was waiting on two things: the document itself, and a font file of 140 kilobytes.

That was a surprise, because we had just trimmed the fonts. What we had trimmed was the list of families. What we had not looked at was the files, and the files were the wrong kind. The font provider we had been using delivered both faces as WOFF, one file per weight, the whole character set, around 140 kilobytes each. The newer WOFF2 format, cut into language subsets, delivers the same face at 14 to 24 kilobytes a file, and a browser fetches only the subsets a page uses. We switched the provider and limited the subsets to Latin and Greek.

The same report showed a 201 kilobyte data file on the home page. The home page shows three post cards. To draw them it had been querying every post with every field, which includes the full text of every post, and shipping the result to the browser as the page's payload: 652 kilobytes on disk, 201 over the wire, to render three titles. The query now selects the card fields and nothing else. The payload is 31 kilobytes. The listing page and the post page made the same query for the same reason and got the same fix.

Neither of these was a small thing, and we found them only because we measured after the first change rather than declaring it done. That is the habit the series is about, and here it paid for itself within the hour.

The numbers

Lighthouse, mobile emulation, against the live site. Day zero was a single run per page; today we ran each page three times and report the median, because the runs vary by up to a second on the same page and we wanted to know that before we claimed anything.

Two pages got better by a lot, one got better, and two got worse on the largest element while shipping a third of the bytes. The post and the curriculum page both have text as their largest element, and on both the report attributes the delay to rendering rather than downloading: the text now waits on six small font files where it waited on three larger ones, and each request costs a round trip on a slow connection even when the bytes are fewer. We are reporting that as it is. The next move on those two pages is to preload the one heading face they need, and we will say what it did when we have done it rather than now.

What we will not do is pick the runs that flatter us. The three runs of the home page scored 90, 94 and 81. The median is the number in the table.

The gotcha, for the record

While we were in the configuration we left a comment in place that earns its keep. On the sixth of June, when the individual workshop pages went away, a rule was added to send any old sub address back to the Dojos page: "/workshops/**": { redirect: "/workshops" }. It looped. The same day it was changed to a single star, and it looped again. The client side matcher treats the bare /workshops as matching /workshops/*, so the page redirected to itself until the browser gave up with a 500. On the seventh the rule was removed and a comment went in its place saying why, and that a redirect must always point at a different prefix. The comment has outlived three rewrites of that file, which is the right fate for a landmine.

The number for today: the Dojos page, 8.9 seconds to 3.2, and two pages we made slower and said so.