Learning center

Why Two Files You'll Never See Decide How Much of Your Site Google Actually Finds

A sitemap is a direct list of every page you want Google to know about. Without one, Google is left to discover your pages the slow way — by stumbling onto links, if it finds them at all.

Why Two Files You'll Never See Decide How Much of Your Site Google Actually Finds

Your audit checks for two small files: a sitemap.xml (a structured list of your site's real, indexable pages) and a robots.txt (a file that tells search engine crawlers what they're allowed to access, and typically points to the sitemap). Neither is visible to a homeowner browsing your site — both matter to how completely Google can find and index it.

13%of sites return an error instead of a valid robots.txt file, based on large-scale crawl analysisRobots.txt crawl study, 2026
23%of sites have pages that don't link to their own XML sitemap from robots.txt, missing an easy signal boostRobots.txt crawl study, 2026

"A Beginners Guide To Sitemaps and Robots txt" — The Stuff You Need To Know About SEO on YouTube (third-party video)

What each file actually does

A sitemap.xml lists every real page on your site you want indexed — your homepage, service pages, city pages, blog posts — so Google doesn't have to rely purely on finding internal links to each one. A robots.txt is mostly a set of crawl instructions ("don't bother crawling this folder"), and it usually also points directly to your sitemap's location, connecting the two.

Small sites rarely have a "crawl budget" problem — the real risk is a discovery gap

For a handful of pages, Google's crawl budget (how much of a site it bothers crawling) is almost never the limiting factor for a small contractor site. The real risk without a sitemap is a plain discovery gap: a new city page or service page you just added sits undiscovered longer than it needs to, simply because nothing told Google it existed yet.

It's a five-minute technical fix, not a content project

Unlike most SEO work, this isn't writing or design — it's a small, mostly-automated technical file most website platforms can generate on their own once configured. If it's missing, it usually means it was simply never set up, not that it's actively difficult to add.

Want us to just fix this for you?

Free audit shows you exactly what's costing you jobs on your own site — no obligation.

Get my free audit →

Related

FAQ

Do I need a sitemap if my site only has a few pages?

It still helps — it's a direct, explicit list rather than relying on Google to find every page through links, and it costs nothing to maintain once set up.

What does robots.txt actually control?

Which parts of your site search engine crawlers are allowed to access, and (usually) where to find your sitemap. It doesn't control rankings directly.

Is this something I can set up myself?

On most modern website platforms, yes — many generate both files automatically once basic SEO settings are configured. If both are missing, it typically means that setup step was skipped.