This is the one post in the series that describes work which shipped before the series began. We are telling it first because everything that follows is built on it, and because it ends with the defect the previous post found.
The blog that was not there
On the sixth of June we moved our blog and legal pages off a hosted CMS and into the repository as markdown files. The framework we use reads those files at build time and loads them into a small database, and every page that shows content queries that database. On a laptop it worked the first time. The listing showed the posts, each post rendered, the legal pages rendered.
We deployed. The listing was empty. Every post returned 404. Every legal page returned 404.
The cause took a while to see because nothing was broken in the usual sense. Our host renders pages inside a serverless function, and the database the build had produced was not inside that function at the moment a request arrived. The build had made it, the function could not see it. So a page that asked the database for a post got nothing, and the page behaved correctly for nothing: an empty list, a 404.
The fix was to stop asking at request time. If the database only exists at build time, render the pages at build time, when it is there, and ship the HTML. The framework calls this prerendering, and the configuration was short:
nitro: {
preset: 'netlify',
prerender: {
crawlLinks: true,
routes: ['/', '/blog'],
ignore: ['/styleguide'],
failOnError: false,
},
},
Start at the home page and the listing, follow every link the crawler finds in the HTML, write each page to a file. Thirty six routes came out, the blog listed its posts, the posts and legal pages rendered. Committed, deployed, fixed.
The posts that only existed on page one
A month later, while adding four posts, we found an older one returning 404 in production. We opened the listing on a laptop, paged through to it, clicked, and there it was. Live, 404.
The listing paginates on the client. The HTML of /blog contains the first page of cards and nothing else; the rest are fetched when a reader presses next. The build's crawler does not press next. It reads HTML and follows links, so it found the first page of posts, rendered them, and stopped. Any post that had scrolled off the first page was never linked from anything the crawler could read, so it was never rendered, so in production it was not there.
That is the title of this post and the principle under it. A page that the build does not produce does not exist. It does not matter that the source file is in the repository, that the route is defined, that the page renders on a laptop. If the process that produces the site never produced that page, the site does not have it. And a crawler that only follows links will only produce the pages that something links to, which makes the existence of a page depend on the accident of what happened to be on page one.
The fix on the eighth of July was to stop depending on links at all for the one thing we could enumerate:
const blogDir = join(dirname(fileURLToPath(import.meta.url)), "content", "blog");
const blogRoutes = readdirSync(blogDir)
.filter((f) => f.endsWith(".md"))
.map((f) => `/blog/${f.replace(/\.md$/, "")}`);
// later
routes: ['/', '/blog', '/curriculum', '/edtech-accelerator', ...blogRoutes],
Read the directory, turn every markdown file into a route, hand the whole list to the prerenderer. The crawler still runs for the pages that have links, but the posts no longer need one. Today the build renders every post and every page, one hundred and thirty two routes including the data files that go with them, whether or not anything points at them.
That list is the seed of this series. The same enumeration that decides which pages exist will go on to produce the sitemap, and it is the reason the sitemap can be trusted: it is not a second list someone maintains, it is the one list the build already uses.
The twin
Here is the part we found out on the twenty second of August, from the outside.
The prerenderer writes each page as an index.html inside a folder: dist/story/index.html, dist/blog/landmines/index.html. Our host sees a folder and prefers the folder's address, which ends in a slash. Ask it for /story and it answers with a 301 to /story/. Ask for /blog/landmines and it sends you to /blog/landmines/. We checked all sixty two routes on the live site: sixty answer with a redirect before they answer with a page. The two that do not are the home page, which is the root folder, and one post that had not yet deployed.
Every link on the site, in the navigation, the footer, and every card in the feed, points at the form without the slash. So every click inside the site is a redirect followed by a page. It costs a round trip each time, which a reader will not notice. A search engine notices. It now holds two addresses for each page, sees the site linking to the one that redirects, and has no canonical tag to tell it which one we mean. Search Console shows eighteen posts indexed under the address with the slash, which nothing on our site links to, and seven under the address without it, which the server redirects. We cannot see from our side why it chose differently for those seven, and it does not matter: the point is that we gave it a choice.
None of this was visible from the laptop. The development server serves /story as /story, so our own audit the day before, which ran against it, recorded a clean 200 on every route. The live host, with the same files, redirects. The fix that made the pages exist also decided what their addresses were, and we did not notice we had made that decision until a crawler told us.
What it leaves us
The prerender stays. It is the right fix for a database that is only present at build time, and the enumeration is the right fix for a crawler that cannot press next. What it leaves is a question the next post answers with code: one address per page, declared on the page itself, with the other form redirecting to it, and a check that runs against the live site after a deploy and fails if any route answers with a 301.
The number for today: sixty of our sixty two addresses answer with a redirect first. The next post is about making that zero.
