CMSKite
DashboardStart free

Free tool · robots.txt and sitemap

robots.txt and sitemap checker

Type your site’s address. We read its robots.txt and sitemap the way a search engine would, and tell you whether your blog posts are listed.

What it checks

  1. robots.txt is reachable

    We fetch /robots.txt from your site’s address, following up to five redirects as Google does. A 200 is what you want. A 404 is allowed (Google then assumes nothing is blocked); a server error makes Google pause crawling.

  2. It names a sitemap

    We look for Sitemap: lines. Google takes only full addresses there, like Sitemap: https://example.com/sitemap.xml. With none, we try /sitemap.xml.

  3. The sitemap loads and lists pages

    We read up to three sitemaps from robots.txt and, for a sitemap index, up to five of the sitemaps inside it, blog ones first, and count the addresses in each.

  4. Your blog is in it

    We look for addresses with /blog in their path, like /blog/my-post, and show a few.

It is a quick health check, not a full audit: it does not test every address in the sitemap, and it does not read Disallow rules. For that, use the robots.txt report and the Sitemaps report in Google Search Console.

What a healthy setup looks like

robots.txt · plain
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/blog/sitemap.xml
  • Every sitemap you have is listed, each on its own Sitemap: line with its full address. There is no limit on how many.
  • Each sitemap holds at most 50,000 addresses or 50 MB uncompressed. A bigger site splits it and lists the parts in a sitemap index.
  • Addresses in the sitemap are full and final: https://, the same host as the site, no redirects, the pages you want found.
  • Listing a sitemap helps; it does not force indexing. Google treats it as a hint. Submitting it in Search Console as well shows you which pages were read.

What we fetch, and from where

The check runs on our server, as CMSKiteSiteCheck/1.0. It reads only public addresses on the standard web ports, at most eight files per check, 512 KB of robots.txt and 5 MB of each sitemap, and stops after 20 seconds. We do not store the results. Each visitor can run ten checks an hour.

Sources

Questions

Do I need a robots.txt at all?
No. Without one, crawlers assume they may read everything. It is still the easiest place to tell every search engine where your sitemap is, so most sites have one.
My blog is on another domain or path. Is “no /blog found” a problem?
Not if your posts live somewhere else, like /news or blog.example.com. The check only looks for /blog in the path. Check that the addresses you do use are listed.
The sitemap is in robots.txt. Do I still submit it in Search Console?
It is worth it. Google finds it either way, but Search Console then reports how many of its pages were read and indexed, and any errors.
Why can’t it check my site on localhost or a private network?
For safety, the checker only reads public internet addresses on ports 80 and 443. A site search engines can reach is a site it can check.

Related

Checked