Robots.txt Tester
Check whether a robots.txt file lets Googlebot, Bingbot or AI crawlers like GPTBot fetch your pages. Test many URLs at once and see the exact line that decides each one.
The robots.txt file, with the lines that decided a result highlighted
How this robots.txt checker decides
It applies the rules exactly as Google documents them and as RFC 9309 specifies:
- One group per crawler. A crawler obeys the group that names its user-agent token, such as
User-agent: Googlebot. Only if no group names it does it fall back toUser-agent: *. The two are never combined, so a Googlebot group that leaves out a rule from*lets Googlebot crawl that path. - The longest matching rule wins.
Allow: /shop/salebeatsDisallow: /shopfor/shop/sale-items, because it is longer. When an Allow and a Disallow match with the same length, Allow wins. - Wildcards.
*matches any run of characters and$anchors the end of the URL, soDisallow: /*.pdf$blocks every PDF. - Paths are case-sensitive.
Disallow: /Admindoes not block/admin. - Missing or broken files. A robots.txt that returns 404 allows everything. One that returns a 5xx error or times out makes Google stop crawling the whole site until it recovers.
AI crawlers you can test
These are user-agent tokens that their operators document as following robots.txt. Blocking them does not affect your Google or Bing rankings.
| Token | Operator | What blocking it does |
|---|---|---|
GPTBot | OpenAI | Keeps pages out of data used to train OpenAI's models. |
OAI-SearchBot | OpenAI | Stops pages being shown in ChatGPT search results. |
ClaudeBot | Anthropic | Keeps pages out of data used to train Anthropic's models. |
Google-Extended | Stops content being used to train and ground Gemini. It doesn't affect Google Search. | |
PerplexityBot | Perplexity | Stops pages being indexed for Perplexity's answers. |
CCBot | Common Crawl | Keeps pages out of the open Common Crawl dataset, which many AI models are trained on. |
Applebot-Extended | Apple | Stops content being used to train Apple's AI models. Applebot still crawls for Siri and Spotlight. |
Questions
Does Google Search Console still have a robots.txt tester?
No. Google retired the legacy robots.txt Tester in December 2023 and replaced it with the robots.txt report, which shows the files Google fetched and any errors in them, but can't test a URL against a rule. This tool fills that gap, for any site, including ones you don't own.
Does robots.txt stop a page from being indexed?
No. robots.txt controls crawling, not indexing. A blocked URL can still appear in Google if other pages link to it, just without a description. To keep a page out of search results, allow it to be crawled and add a noindex tag. Check both with the Indexability Checker, and read robots.txt vs noindex for when to use which.
How do I block GPTBot and other AI crawlers?
Add a group for each crawler, for example User-agent: GPTBot followed by Disallow: / on the next line. Several tokens can share one group by listing a User-agent line for each above the rules. Then test the result here with the AI crawlers selected.
Why is my URL allowed when I disallowed it?
The usual causes are a separate group for that crawler (it then ignores the * group entirely), a longer Allow rule that matches, a difference in letter case, or a path that doesn't start the way the rule does. The highlighted line in the results shows which rule actually decided it.
Can I test a robots.txt before publishing it?
Yes. Choose "Paste robots.txt", paste your draft and test your URLs against it. Nothing is fetched from your site in that mode.