Your robots.txt file and sitemap.xml are two cornerstones of technical SEO. A single misplaced Disallow rule can inadvertently block entire sections of your website from search engines, while a missing sitemap slows down the discovery of new pages. This tool checks both files in seconds, giving you an instant picture of what is allowed, what is blocked, and where common mistakes are hiding.
How to use the tool
Enter your website address in the field above and click "Check". The tool automatically fetches /robots.txt and /sitemap.xml, parses their contents, and returns a structured breakdown. No registration or API keys required.
- Paste your site URL — with or without the protocol, the tool will handle it.
- Hit the check button and wait 2–5 seconds for the results.
- Review the robots.txt contents, any directives found, and the sitemap status.
- Fix the issues found and re-run the check to confirm everything is clean.
What the results show
The tool displays the contents of robots.txt and highlights the key elements:
- User-agent — which crawlers the rule block applies to (Googlebot, Bingbot, * for all).
- Disallow / Allow — paths that are blocked or explicitly permitted for crawling.
- Sitemap reference — whether a sitemap path is declared inside robots.txt and whether the sitemap.xml file itself is actually reachable at the domain.
- File availability — whether the server returns a healthy response (200) for both robots.txt and sitemap.xml.
If robots.txt is missing or returns a 404, search engines treat the entire site as open for crawling — not always a problem, but worth knowing.
Frequently asked questions
What does Disallow: / mean?
A forward slash / after Disallow blocks the specified user-agent from crawling your entire website. This is one of the most common accidental mistakes — developers enable it during staging and forget to remove it before going live.
Is sitemap.xml required for SEO?
Technically no, but practically yes. Without a sitemap, search engine crawlers can only discover pages by following internal links — this is especially limiting for large sites or newly launched ones. Both Google and Bing officially recommend submitting an XML sitemap, and referencing it in robots.txt ensures crawlers find it immediately on their first visit.
Does blocking a page in robots.txt remove it from search results?
Not necessarily. Disallow prevents crawling, but it does not guarantee removal from the index. If external websites link to a blocked page, search engines may still index it as a URL — just without any content. To fully exclude a page from search results, use a noindex meta tag, either alongside Disallow or instead of it.
How often should I check my robots.txt?
After every significant structural change: migrating to a new CMS, redesigning the site, adding new sections, or switching to HTTPS. It is also worth running a check whenever Google Search Console or Bing Webmaster Tools surfaces crawl coverage warnings — a misconfigured robots.txt is often the silent culprit.
Want to go deeper? Read our complete guide to robots.txt and sitemap.xml — with real-world configuration examples for the most popular CMS platforms.