Updated
CMSKiteBot
CMSKiteBot is the crawler of CMSKite, a blogging platform. It visits a website only after the site’s owner has connected it to CMSKite and proved they own it in Google Search Console, and it reads those pages to tell that owner about SEO problems: broken pages, wrong canonicals, missing titles. It identifies itself as “CMSKiteBot/1.0 (+https://cmskite.com/bot)”.
Which sites it visits
Only websites whose owners added them to a CMSKite project and connected a Google Search Console property that matches the site. Search Console verification is Google’s proof of ownership, so CMSKiteBot never crawls a site on someone else’s say-so.
When the verified property is a URL prefix on a sub-path (for example https://example.com/blog/), only pages under that path are fetched. Blogs hosted on CMSKite itself are never crawled: we already have their content.
If you see CMSKiteBot on a site you did not connect, somebody with access to that site’s Search Console did. Write to us (below) and we will look into it.
What it fetches
- robots.txt, once at the start of each run.
- Up to four sitemap files, from robots.txt or /sitemap.xml, to learn which pages exist.
- The HTML of pages listed in your sitemap or reported by Search Console. Links on a page are counted, not followed, and images, scripts and stylesheets are never downloaded.
- At most 1 MB of each page, with 15 seconds to answer.
From each page it reads the status code, redirects, canonical, robots meta tags and X-Robots-Tag, title, description, h1, language, link counts and the types of structured data. Nothing is submitted, no forms are filled in, and nothing is signed into.
How often and how fast
- At most 200 pages per site per day, in one run of at most 10 minutes.
- At most one request per second to your server, redirects and sitemaps included.
- Slower if your robots.txt asks: a Crawl-delay is honoured up to 10 seconds between requests.
A site is crawled at most about once a day. If robots.txt answers with a server error or not at all, nothing is crawled that day.
robots.txt and robots tags
CMSKiteBot follows robots.txt as Google does (RFC 9309). It obeys a group for “User-agent: CMSKiteBot” when there is one, and otherwise the “User-agent: *” group. Allow and Disallow, wildcards and the $ end anchor all work, and the longest matching rule wins. A robots.txt that answers 404 means everything is allowed.
Robots meta tags and X-Robots-Tag headers addressed to all bots or to “cmskitebot” are respected: noindex and nofollow are reported to the site owner as such.
How to slow it down or block it
To keep CMSKiteBot off your whole site, add this to robots.txt:
User-agent: CMSKiteBot Disallow: /
Only slower
To keep it but space its requests out (in seconds, up to 10):
User-agent: CMSKiteBot Crawl-delay: 5
Contact
If CMSKiteBot misbehaves on your site, or you want it to stop and robots.txt is not an option, write to cmskite.com@gmail.com with your domain. A site owner can also stop it at any time by disconnecting Search Console from their CMSKite project.
See also the Privacy Policy.