Blocked by robots.txt: what it means and how to fix it

What the status means

Before crawling a URL, Googlebot reads /robots.txt on that host. If a Disallow rule matches the URL's path, Googlebot doesn't fetch the page. Search Console then lists it in the Page indexing report under Blocked by robots.txt.

Two things follow from that:

• Google can't read the page, so it can't see its content, its links or any noindex tag on it.
• Blocking crawling is not the same as blocking indexing. If other pages link to the URL, Google can still list it, without a description. That case has its own status, covered below.

Is the block intended?

Look at the example URLs first. Many sites block these on purpose, and for them the status is fine:

• cart, checkout and account pages;
• internal search results (/search, ?s=);
• filter and sort combinations that create endless URL variants;
• admin areas, such as WordPress's /wp-admin/.

It's a problem when the list includes pages you want in search: products, posts, categories, or whole sections. It's also a problem when it includes CSS or JavaScript files that Google needs to render your pages.

Find the rule that blocks it

Robots.txt rules are matched by the most specific (longest) path, and Allow wins a tie. Wildcards (*) and the end anchor ($) make it easy to block more than you meant, so test instead of reading by eye:

Enter the site and paste the blocked URLs from Search Console. The tester shows, for each URL and each crawler, whether it's allowed and the exact line number of the rule that decides it.

Common culprits:

Disallow: /            # blocks the whole site, often left over from staging
Disallow: /*?          # blocks every URL with a query string
Disallow: /blog        # also blocks /blog-post-1 and /blogging/
Disallow: /*.js$       # stops Google rendering your pages

Also check that the file you're editing is the one being served. Each host and protocol has its own robots.txt, so shop.example.com/robots.txt is separate from www.example.com/robots.txt.

How to fix it

1. Remove or narrow the rule. Delete the line, or make it more precise. To keep a folder blocked but open one page inside it, add a more specific Allow:

   User-agent: *
   Disallow: /private/
   Allow: /private/press-kit/

2. Check the result in the Robots.txt Tester, using the pasted-file mode before you publish, so a typo can't block the whole site.
3. Publish and wait. Google usually caches robots.txt for up to 24 hours. Search Console's robots.txt report (Settings → robots.txt) shows the version Google last fetched and lets you request a recrawl of the file.
4. Request indexing for your most important URLs with URL Inspection, then click Validate fix in the report.

If you blocked the URL to keep it out of search, robots.txt is the wrong tool. Use noindex and allow crawling so Google can see the tag. The difference is explained in Robots.txt vs noindex.

Indexed, though blocked by robots.txt

This warning means Google indexed the URL without crawling it, usually because other pages link to it. The search result appears without a description.

• If the page should be in search, remove the blocking rule. Google can then crawl it and show a proper result.
• If it should not be in search, remove the blocking rule and add noindex. Google needs to crawl the page to see the noindex; once it drops out, you can block it again if you also want to save crawling. For urgent cases, Search Console's Removals tool hides a URL for about six months while the fix takes effect.

WordPress and Shopify

WordPress serves a virtual robots.txt unless a real file exists in the site root. A physical robots.txt file replaces it, and SEO plugins such as Yoast and Rank Math have their own robots.txt editors. Check all three places, and check that Settings → Reading → Discourage search engines from indexing this site is off on a live site.

Shopify generates robots.txt for every store. Its default rules block cart, checkout, account, internal search and some filtered collection URLs, so seeing those under "Blocked by robots.txt" is normal. To change the rules, edit the robots.txt.liquid template in your theme's code; Shopify recommends keeping the default rules and adding to them.

Questions