Robots.txt vs noindex: how to keep a page out of Google
They sound like two ways of doing the same thing, but they do different jobs. Mixing them up is one of the most common reasons a page either stays in Google when it shouldn't, or vanishes when it shouldn't.
The short answer
robots.txt Disallow | noindex | |
|---|---|---|
| Controls | Crawling: whether a bot may fetch the URL | Indexing: whether the page may appear in results |
| Where it goes | One file at /robots.txt | A meta tag in the page, or an X-Robots-Tag header |
| Can the URL still show in Google? | Yes, if other pages link to it, without a description | No, once Google has recrawled the page |
| Best for | Saving crawl budget on endless or useless URLs | Keeping a real page out of search results |
What robots.txt does
A Disallow rule tells crawlers not to fetch matching URLs:
User-agent: * Disallow: /search Disallow: /cart/
Google won't download those pages, so it never sees what is on them. But it can still index the bare URL when other pages link to it. It then appears in results with a note that no information is available. robots.txt is a crawl control, not a privacy or indexing control.
What noindex does
A noindex directive tells search engines to leave the page out of results. Put it in the page's <head>:
<meta name="robots" content="noindex">
or send it as an HTTP header, which also works for PDFs and other files:
X-Robots-Tag: noindex
Google has to crawl the page to read the directive. Once it does, the page drops out of results, usually within days to weeks depending on how often it's crawled.
Why using both backfires
This is the trap: if you add noindex to a page and block it in robots.txt, Google can't fetch the page, so it never sees the noindex. A page that was already indexed can then stay in results indefinitely.
noindex. Only once it has dropped out can you block it in robots.txt, if you also want to save crawling.Which one to use
- Thank-you pages, internal search results, thin tag archives:
noindex, so they stay crawlable but out of results. - Endless filter and sort combinations, calendars, session URLs: robots.txt, to stop crawlers wasting time on them.
- Staging sites and private content: neither is enough. Use password protection, which returns a 401 or 403.
- Duplicates of another page: a canonical tag pointing at the main version, rather than either of these.
- AI crawlers: robots.txt, with a group for each bot such as
User-agent: GPTBot.
How to check a page
To see what a robots.txt file allows, for Googlebot and any other crawler:
To check every signal on a list of pages at once (robots.txt, meta robots, X-Robots-Tag, redirects and canonicals), use the Noindex & Indexability Checker. It flags the robots.txt-plus-noindex trap automatically, because a blocked page shows as blocked whatever its tags say.
Questions
Does noindex in robots.txt still work?
No. Google stopped supporting Noindex: lines in robots.txt in September 2019. Use a meta robots tag or an X-Robots-Tag header instead.
How long does noindex take to remove a page?
Until Google next crawls the page, which can be days for popular pages and weeks for rarely visited ones. For an urgent case on a site you own, the Removals tool in Search Console hides a URL for about six months while the noindex takes effect.
Should I noindex pagination pages?
Usually not. Page 2, 3 and so on of a category help search engines find the products or posts listed there. Leave them indexable, with each one self-canonical.