Can Google find your page? robots.txt, sitemaps, canonicals and 404s

Everything you improve on your website is wasted effort if Google cannot reach, read or index the page. That sounds obvious, but it is the most common cause of “we do not show up in Google” that we come across: not the copy, not the competition, but one line in a file nobody remembered.

This article walks through the technical conditions a page has to meet before anything else matters: HTTPS, the noindex directive, robots.txt, the sitemap, the canonical URL, the HTTP status, 404 pages and redirects.

HTTPS

A site without a valid certificate is labelled “Not secure” by browsers, and that costs you visitors before they have read a word. Google also confirms that HTTPS is a lightweight ranking signal. A certificate is free with nearly every host (Let’s Encrypt) and installing it is usually one button.

What matters is the second step, which is often skipped: redirect all HTTP traffic to HTTPS. Otherwise your site exists in two versions side by side, which confuses visitors and search engines alike.

The noindex directive

One line in the source can keep your whole page out of Google:

<meta name="robots" content="noindex">

This is the most damaging mistake there is, and it happens more often than you would think: it stays behind after a site goes live, because during the build it was there on purpose. While that line is present, nothing else matters. Check it on your most important pages, and check the X-Robots-Tag in the server headers too, because the same instruction can sit there without appearing in the source.

robots.txt

This file lives at yourdomain.com/robots.txt and tells crawlers which parts of your site they may fetch. Your site works fine without one, but with it you can do two useful things: keep crawlers away from areas that do not matter (internal search results, filter pages with endless combinations) and point them at your sitemap.

Where it goes wrong is this line:

Disallow: /

That asks every crawler to stay off your entire site. It belongs in a staging environment and nowhere else. Be careful about blocking your CSS and JavaScript too: Google renders your page the way a browser does, and cannot judge it properly when the styling is blocked.

One misunderstanding is persistent: robots.txt does not keep a page out of the search results. It keeps crawlers away, but a blocked page can still end up in the index through links from elsewhere. If you genuinely want a page out of Google, use noindex and do not block it in robots.txt, or Google can never read that instruction. Google covers this in its robots.txt documentation.

The XML sitemap

A sitemap is a list of the addresses you want indexed. Google finds pages by following links; the sitemap is the shortcut, and it counts for most on new pages and on pages with few links pointing at them.

Nearly every CMS generates one automatically. Check that it exists (usually /sitemap.xml or /sitemap_index.xml), that it only lists pages you actually want indexed, and that it is referenced in your robots.txt. Submit it once in Google Search Console and in Bing Webmaster Tools.

The canonical URL

The same page is often reachable at several addresses: with and without www, with and without a trailing slash, with a tracking parameter attached. To you that is one page; to a search engine it is four.

The canonical is the link element that says which address is the original:

<link rel="canonical" href="https://yourdomain.com/services/roof-repair/">

Duplicate content is not a penalty, as Google has said explicitly. Without a canonical, Google simply has to choose which version it indexes, and that choice does not always fall the way you would want. One line removes the guesswork.

The HTTP status

Every page returns a status code. 200 means “here is the page”. 301 means “moved, permanently”. 404 means “does not exist”. 500 means “something is broken”. A page you want indexed should return 200, every time.

404 pages and soft 404s

An address that does not exist should return a 404. That sounds obvious, but many sites redirect such a request to the home page with a tidy 200 attached. Google calls that a soft 404, and the consequence is that unlimited junk addresses can enter the index, using crawl budget that should have gone to your real pages.

Make a 404 page that does something, too: a search field, links to your main pages and the normal navigation. A dead-end error message costs you the visitor.

Redirects

One redirect is normal and harmless, for example HTTP to HTTPS or non-www to www. Chains of three or four are not: each step costs time before the first byte arrives, and the visitor is waiting. Check now and then that old redirects have not been stacked on each other, and point them straight at the final destination.

What you can do today

• Open yourdomain.com/robots.txt and check there is no Disallow: /.
• View the source of your most important page and search for “noindex”.
• Check that your sitemap exists and has been submitted in Search Console.
• Type an address that does not exist and see whether you get a real 404 with a usable page.
• Check that http:// redirects cleanly to https://.

Want all of this checked in one go? The free SEO check runs through these points and shows you what was found on your page for each of them.