Bulk Indexability Checker
Find out whether Google is allowed to index each URL, and if it isn't, exactly which signal is stopping it.
| # | URL | Verdict | Why | Status | Canonical | Robots directives |
|---|
What gets checked
For every URL, the checker fetches the page and its site's robots.txt the way Googlebot would, then looks at each signal that can keep a page out of Google:
| Signal | Effect |
|---|---|
robots.txt Disallow | Google can't crawl the page. It is matched with Google's rules: the Googlebot group takes priority over *, the longest matching rule wins, and * and $ wildcards are supported. |
<meta name="robots"> / googlebot | noindex or none removes the page from results. |
X-Robots-Tag header | Same as the meta tag, but sent as an HTTP header. Directives aimed at other crawlers (bingbot: noindex) are ignored, as Google does. |
| Redirects | The URL itself won't be indexed. Google indexes the destination. |
| 4xx / 5xx status | The page isn't indexed, and an indexed page is dropped once Google sees the error. |
rel="canonical" | If it points to a different URL, Google will usually index that URL instead. Read from both the HTML and the Link header. |
Reading the verdicts
- Indexable Nothing stops Google indexing the page. Whether it does depends on quality, internal links and crawl budget.
- Canonicalised The page names another URL as canonical. That's correct for duplicates, and a problem if it's your main page.
- Not indexable At least one signal blocks indexing. The reason is listed.
- Check No hard block, but something looks wrong, such as conflicting canonicals or a robots.txt that returns an error.
- Unknown The page couldn't be fetched at all.
Questions
Why don't you check whether a URL is in Google's index?
There is no reliable, permitted way to do it for arbitrary sites. Scraping site: searches breaks Google's terms, gets blocked quickly and often gives wrong answers. The real answer is in Google Search Console's URL Inspection tool, for sites you own. This checker tells you whether anything on your side is preventing indexing, which is usually the part you can fix.
A page is blocked by robots.txt. Can it still show up in Google?
Yes. robots.txt blocks crawling, not indexing. If other pages link to a blocked URL, Google can index the bare URL without its content. To keep a page out of results, allow crawling and use noindex, because Google can't see a noindex tag on a page it isn't allowed to fetch.
Does it render JavaScript?
No. It reads the HTML your server sends. If a JavaScript framework adds or changes the robots meta tag after load, Google may see something different once it renders the page. Server-rendered tags are the reliable kind.
Is "Fetch as Googlebot" really Googlebot?
No. It sends Googlebot's user-agent string from Cloudflare's network. Sites that verify Googlebot by IP address may treat it differently from the real crawler.