robots.txt and sitemap checker online
A single Disallow: / line in robots.txt can drop a site from Google. Enter an address and check whether the site is open to search bots and whether a sitemap is listed.
What the robots.txt checker does
The checker downloads /robots.txt from the site root and reads its rules the way a search bot does: which bots each group of rules applies to, which paths are disallowed and which sitemaps are listed. Its main job is to catch the most expensive mistake, a Disallow: / rule for all bots or for Googlebot, which gradually removes the whole site from search.
This usually happens when a staging site goes live with its blocking rules still in place, or when an SEO or security plugin writes rules nobody checked. The full check takes off 4 points for it, one of its largest penalties.
How to read the result
- Site for Googlebot: open means Googlebot may crawl the homepage; BLOCKED means a Disallow: / rule applies to it.
- Disallow rules: the paths closed to bots. Closing /wp-admin/ or a cart is normal; closing folders with CSS or JS can hurt how Google sees your pages.
- Sitemap in robots.txt: a Sitemap: line helps every search engine find your pages, not only those where you submitted the sitemap by hand.
- No robots.txt file: not an error, everything is allowed by default.
Related errors and guides
Frequently asked questions
- Does Disallow remove a page from Google?
- No. It stops crawling, but a blocked page can still appear in results if other sites link to it. To keep a page out of the index, allow crawling and add a noindex tag instead.
- Is robots.txt a security measure?
- No. It is public and only polite bots follow it. Listing secret paths in it actually points people to them.
- How quickly does Google notice a change?
- Google usually caches robots.txt for up to 24 hours. After removing a block, request indexing of your key pages in Search Console to speed things up.