SEO tools
Who is allowed to crawl what?
Fetch a site’s robots.txt or paste one, see every problem line by line, and test any URL for Googlebot, Bingbot or AI crawlers.
Sites are fetched through Imbuz’s own fetch service so your browser can read them; nothing is stored. A fetched file appears under Paste or edit, so you can try changes.
Test a URL
About this tool
Google’s rules, applied the way Google applies them
A single typo in robots.txt can hide a whole site, and Google retired its robots.txt tester in 2023. This parses the file the way RFC 9309 and Google describe — groups by user-agent, the most specific group wins, the longest matching rule wins, Allow wins a tie, * and $ work as wildcards — flags lines crawlers will ignore, and lets you test any URL for any crawler, showing the exact line that decided it. It also shows at a glance which AI crawlers can read your site.
- Free, no sign-up
- Nothing stored
- Paste or fetch
A real tester
Pick Googlebot, Bingbot, GPTBot, ClaudeBot or any user-agent, enter a URL, and see Allowed or Blocked with the line responsible.
Lint that explains
Typos like Dissalow, rules before any User-agent, full URLs instead of paths, retired directives like noindex — each with what crawlers actually do.
AI crawlers at a glance
A table of the main AI training and search crawlers and whether each can read your homepage.
Step by step
How to test robots.txt
Enter your domain
Type example.com and its /robots.txt is fetched — or paste a draft before you publish it.
Read the problems
Each issue names the line, says what crawlers do with it, and how to fix it.
Test a URL
Choose a crawler and a URL to see whether it is allowed, and which rule decided.
Get the most out of it
Tips for the best results
- Robots.txt is not noindexBlocking a URL stops crawling, not indexing — a blocked page can still appear in results. Use a noindex tag to keep it out.
- Never block CSS or JavaScriptGoogle renders pages. If it cannot load your styles and scripts it cannot judge the page properly.
- Paths are case-sensitiveDisallow: /Admin does not block /admin.
- Test changes before publishingPaste the new version here and test the URLs that matter before it goes live.
Keep going
More free SEO tools
Questions
Questions about this tool
Pasted text never leaves your browser. Addresses you enter are fetched through Imbuz’s own fetch service, and nothing is stored. More in the general FAQ.
How do I test if a URL is blocked by robots.txt?
Enter the site’s domain to fetch its robots.txt, then in the tester choose the crawler — Googlebot, for example — and type the URL or path. The result says Allowed or Blocked and shows the rule that decided it.
Does robots.txt stop a page from being indexed?
No. It stops crawling. Google can still index a blocked URL from links pointing at it, showing it without a description. To keep a page out of results, allow crawling and add a noindex meta tag or X-Robots-Tag header.
How does Google handle conflicting Allow and Disallow rules?
The most specific rule wins — the one with the longest matching path. If an Allow and a Disallow match with the same length, Allow wins. The tester applies exactly that logic.
How do I block AI crawlers like GPTBot?
Add a group for each crawler, such as “User-agent: GPTBot” followed by “Disallow: /”. The AI crawler table shows which are currently allowed. Blocking Google-Extended stops Gemini training without affecting Google Search.
What happens if robots.txt returns an error?
A 404 or other 4xx means there are no restrictions, so everything is crawlable. A 5xx server error makes Google stop crawling the site until robots.txt is reachable again — the validator flags this.