The previous post ended with a number: sixty of our sixty two addresses answered with a redirect before they answered with a page. This post is the change that made it zero, and the check that will tell us if it ever stops being zero.
The shape of the defect
Our build writes each page to a file. Until today it wrote story/index.html, a folder with an index inside. Our host, seeing a folder, treats the folder's address as the real one, which ends in a slash, and answers a request for /story with a 301 to /story/. Every link we have ever written points at /story. So a reader who clicks the navigation gets a redirect, then the page. A crawler that follows the same link gets a redirect, then a page that does not say which of its two addresses is the one to keep.
Written out, that is three separate faults wearing one symptom.
The build produced a file shape the host interprets differently from how we link. No page declared a canonical address, so the host's preference and our links disagreed with nothing to settle it. And two old addresses from before the site moved, /privacy-policy and /terms-of-service, had been given redirect rules that never fired, because the build had also written an HTML file at each old address, a meta refresh stub, and a static file wins over a redirect rule on our host. So an old link to the privacy policy went to a redirect, then a stub page, then to the policy, and a search engine recorded a 200 at an address that should have said 301.
The change
Three edits in one commit, because they are one fix.
First, tell the prerenderer to write flat files:
prerender: {
autoSubfolderIndex: false,
crawlLinks: true,
routes: ['/', '/blog', '/curriculum', '/edtech-accelerator', ...blogRoutes],
ignore: ['/styleguide', '/privacy-policy', '/terms-of-service'],
failOnError: false,
},
story/index.html becomes story.html. The host serves a flat file at the address without the slash and, it turns out, normalises the slash form back to it with a 301. The same option, reversed, reverses which address the host prefers. The two old legal addresses join the ignore list so the build stops writing a stub where a redirect rule should be the only thing there.
Second, make the old addresses move permanently. The framework's default for a redirect rule is a 307, which the host's adapter writes as a temporary 302. A page that moved a year ago is not temporarily elsewhere:
routeRules: {
"/posts/**": { redirect: { to: "/blog/**", statusCode: 301 } },
"/terms-of-service": { redirect: { to: "/legal/terms-of-service", statusCode: 301 } },
"/privacy-policy": { redirect: { to: "/legal/privacy-policy", statusCode: 301 } },
},
Third, let every page say its own address. We have one composable that every page calls for its title and description, so the canonical tag went there, derived from the route, with no trailing slash and the production origin even on a preview:
export const SITE_URL = "https://ekohacks.com";
export function canonicalUrl(path: string) {
const trimmed = path.length > 1 ? path.replace(/\/+$/, "") : "/";
return trimmed === "/" ? `${SITE_URL}/` : `${SITE_URL}${trimmed}`;
}
// inside useSiteMetadata
useHead({ link: [{ rel: "canonical", href: canonical }] });
A canonical tag is the page stating which address it considers itself to be. With one form served and the other redirecting to it, the tag is redundant, which is the point: the server, the links and the page now agree, and if any one of them drifts the other two still say the same thing.
The check
A fix you cannot run again is a fix you will lose. The host's behaviour is not something a unit test can see, so the check runs against the live site:
npm run check:live
It reads the route list the same way the build does, from the page files and the content folder, so there is no second list to maintain. For every route it asks the site once, with redirects disabled, and fails if the answer is anything other than 200, if the page has no canonical tag, or if the canonical tag names a different address. Then it asks the three old addresses and fails unless each answers 301 to its new home.
Run against production on the morning of the twenty second of August, before the change:
https://ekohacks.com: 67 addresses checked, 67 wrong
/blog answered 301 -> /blog/
/contact answered 301 -> /contact/
/ has no canonical tag
...
Sixty seven of sixty seven. Sixty redirect hops, the home page and six posts that answered 200 but declared no canonical, and the three old addresses answering with the wrong status.
We did not push that commit on the strength of reasoning about the host. We built it, deployed the build to a throwaway preview address on the same host, and ran the check against the preview:
https://6a89a306b14d3a488057413b--ekohack.netlify.app: 67 addresses checked, 0 wrong
/story answered 200. /story/ answered 301 to /story. /privacy-policy answered 301 straight to /legal/privacy-policy. Only then did it go to production, where the same command found one thing wrong:
https://ekohacks.com: 67 addresses checked, 1 wrong
/blog/two-hands-on-one-lever answered 404
That is a post sitting in the repository, written and not yet committed, so the build never produced it, so it does not exist. The previous post's lesson, caught by the check on its first day. It will answer 200 when it ships, and until then the check is right to complain.
What this does not do yet
Google still holds eighteen of our posts under the address with the slash. Those addresses now answer 301, which is the correct signal, and the index will follow it at its own pace, weeks rather than hours. We ran the inspection again after the deploy and, as expected, nothing had moved: the index reports what it last crawled, and it has not been back yet. We will report when it has, and if a page drops out of the index while it reads us again, that goes in the report too.
It also does not help the thirty six routes Google has never seen, because a page that answers perfectly to a request nobody makes is still unknown. That needs a list of everything that exists, handed to the crawler, which is the next post.
The number for today: sixty seven addresses checked on production, sixty redirect hops gone, one page that does not exist yet.
