Check any site's robots.txt and test whether a path is allowed for Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, or any user-agent. Returns the matched rule, the surrounding rule group, and any sitemap declarations found in the file.
Free. No signup. Rate-limited to 30 checks per IP per hour. Every fetch is server-side and SSRF-guarded — internal IPs, loopback, and bare hostnames are rejected before the request leaves our network.
After a deploy, when traffic suddenly drops. A copy-pasted Disallow: /from staging or a CMS upgrade that overwrites the production robots.txt is one of the most common causes of “Google stopped ranking us overnight”. Run the checker against / with User-agent set to Googlebot — if it says Disallowed, that's your bug.
When AI search engines stop citing you. Many sites added a “no AI training” robots.txt block in 2023–2024 without realising the same rule blocks the runtime citation crawlers. Test User-agent ChatGPT-User and OAI-SearchBotagainst /. If they're Disallowed, you're opted out of ChatGPT search citations.
During a domain migration.Subdomain robots.txt files don't inherit from the apex. After moving from www.example.com to example.com, both hosts need a valid robots.txt or one of them silently blocks crawling. Check both.
A robots.txt checker fetches the robots.txt file at the root of a domain, parses the directives, and tells you whether a specific path is allowed or disallowed for a specific crawler. It is the first thing to run when a page that should rank suddenly disappears from Google — a wrong Disallow rule is one of the most common causes.
Enter your domain and the path you want to test (e.g. /), set User-agent to Googlebot, and click Test. The tool fetches the live robots.txt, finds the rule group that applies to Googlebot, and reports whether the path is Allow or Disallow — including which specific rule won the precedence match.
It depends on your robots.txt. The checker has GPTBot and other AI bots in the User-agent dropdown — pick one and run the test against / to see whether the entire site is disallowed. Many sites accidentally block AI crawlers because they copy-pasted a "no AI training" rule that also blocks the runtime citation crawlers (ChatGPT-User, OAI-SearchBot, PerplexityBot). Result: the site cannot be cited by ChatGPT search, Perplexity, Claude, or Google AI Overviews.
Google picks the most specific (longest-matching) rule for the given path. If an Allow and a Disallow are equally specific, Allow wins. The checker applies the same rule and shows you which directive actually decides whether the URL is crawlable.
Yes. robots.txt applies only to the exact host from which it is served. www.example.com/robots.txt does not govern shop.example.com — each subdomain needs its own file at /robots.txt.
No. robots.txt only blocks crawling. A URL blocked there can still appear in Google's search results if other sites link to it — just without a description. To prevent indexing, use a noindex meta tag on the page itself. Do not block a noindex page in robots.txt or Googlebot will never see the noindex directive.
A Seoxpert scan checks robots.txt plus 230+ other signals — SEO, security, performance, AI / GEO citability, content gaps. Free first scan.
Sitemap checker · Full coverage · How to get cited by ChatGPT