Robots.txt Generator
Build a valid robots.txt in a minute: choose what crawlers may visit, block the AI crawlers you don't want, and point search engines at your sitemap. Test the result against real URLs before you upload it.
How to use your robots.txt
- Choose what to block. Most sites need very little: admin areas, internal search results, and URL parameters that create endless variations, such as sorting and filters.
- Press Test this file to check real URLs against it in the Robots.txt Tester, for Googlebot and any AI crawler.
- Upload it as
robots.txtto the root of your site, so it loads athttps://example.com/robots.txt. It only applies to the host and protocol it is served from. - In WordPress, Shopify and most site builders you edit robots.txt in the SEO settings rather than uploading a file.
Rules that trip people up
- robots.txt doesn't remove pages from Google. It stops crawling; a blocked URL can still be indexed from links elsewhere. Use
noindexto keep a page out, and leave it crawlable so Google can see the tag. - The longest matching rule wins.
Allow: /shop/sale/overridesDisallow: /shop/for that folder, in either order. - Paths are case-sensitive.
Disallow: /Admin/doesn't block/admin/. - A crawler follows only its own group. If you add rules for
Googlebot, Googlebot ignores the*group entirely, so repeat anything it should still obey. - Don't block CSS and JavaScript. Google needs them to render the page.
Questions
Do I need a robots.txt file?
No. Without one, everything may be crawled, which is fine for most small sites. A robots.txt is useful to list your sitemap and to keep crawlers out of areas that waste their time.
Should I block AI crawlers?
That's a business decision. Blocking training crawlers such as GPTBot, ClaudeBot and Google-Extended keeps your content out of future model training without affecting Google or Bing rankings. Blocking OAI-SearchBot or PerplexityBot also means those AI search tools won't cite your pages, which can cost visits.
Does Google respect Crawl-delay?
No. Google ignores it and sets its crawl rate from how your server responds. Bing and some other crawlers do follow it.
What do * and $ mean in a path?
* matches any characters, so /*?sort= blocks any URL with a sort parameter. $ marks the end of the URL, so /*.pdf$ blocks URLs ending in .pdf. Google, Bing and most major crawlers support both.