Diagnostic Technical SEO
robots.txt blocks the noindex it was meant to enforce
You added noindex to a page and blocked it in robots.txt for good measure. Weeks later it is still in the results, listed without a description, and Search Console says Indexed, though blocked by robots.txt. The two rules cancelled each other out. Disallow prevents the crawl, so the crawler never fetches the page and never sees the noindex. Belt and braces removed the belt.
robots.txt controls crawling. noindex controls indexing. A URL that is blocked from crawling can still be indexed from links pointing at it — Google just has no content to show, hence the missing snippet.
To remove a page: allow the crawl and serve noindex. Once it has dropped out of the index you can block it again, if you still want to.
The problem in plain words
The two mechanisms sound like they do the same job, and stacking them feels safer than picking one. It is the specific combination that fails, so the more cautious you are, the more likely you are to hit it.
The same trap catches resources. Blocking your CSS or JavaScript directory stops Google rendering the page as a visitor sees it, which affects how the page is assessed. That block is usually years old and made for a reason that no longer exists.
Why it happens
Crawling and indexing are separate stages and the controls live in different places. One is a file at the root of the host, the other is a tag in the document. Nothing cross-checks them, and the failure is silent for weeks because it only shows up as a page that will not leave the index.
Google states the dependency explicitly: for the noindex rule to work, the page must not be blocked by robots.txt and must otherwise be reachable by the crawler. It is documented, and it is still the most common way a removal fails.
Matching is not first-match-wins. Google uses the most specific rule — the longest matching path — and on a tie, Allow beats Disallow. Reading the file top to bottom gives you the wrong answer, which is how people conclude a page is allowed when it is not.
The file has limits. Google parses the first 500 KiB and ignores the rest, so a generated robots.txt that has grown for years may have rules that are simply not read.
How to fix it
Decide which outcome you actually want
Out of the index: allow the crawl, serve noindex. Off the crawl budget but indexing is fine: Disallow alone. Gone entirely and urgently: remove the page and return 410, plus the Search Console removal tool for the short term.
Test every noindex page against the live robots.txt
This is what the script does: for each URL, it evaluates the robots.txt rules the way Google does — longest match wins, Allow wins ties — and reports any page that is both blocked and carries a noindex.
Unblock, wait for the recrawl, then reblock if you want
The crawler has to fetch the page once to learn it should be dropped. That is a crawl cycle, not an instant change; days to weeks depending on the URL.
Check you are not blocking CSS or JS
An old Disallow: /assets/ or /static/ stops the page rendering the way a visitor sees it. The script flags these separately because the fix is different: you almost always just remove the rule.
Confirm robots.txt is under 500 KiB
Anything past that is not parsed. If your file is generated and large, the rules you care about may be in the part Google never reads.
How to check it worked
Confirm the page is fetchable and still says noindex:
curl -sI https://example.com/private/ | grep -i x-robots-tag
curl -s https://example.com/private/ | grep -i 'name="robots"'Then use the URL Inspection tool in Search Console, which reports crawl permission and the indexing decision as two separate lines — which is the distinction the whole problem turns on.
The full code
The script fetches robots.txt, implements Google's matching rules (longest match wins, Allow breaks ties), and tests each URL you give it. It reports the conflict that matters — blocked and noindex — plus blocked CSS and JS, and warns if the file exceeds the parsed size.
"""Find pages that are both Disallowed in robots.txt and marked noindex.
Those two rules cancel: the crawler never fetches the page, so it never reads the
noindex, and the URL can stay indexed without a snippet indefinitely.
"""
import argparse
import logging
import re
import sys
from urllib.parse import urlsplit
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("robots_conflict")
MAX_ROBOTS_BYTES = 500 * 1024 # Google parses the first 500 KiB
ASSET = re.compile(r"\.(css|js|mjs|woff2?|svg|png|jpe?g|webp)$", re.I)
def parse_robots(text, agent="*"):
"""Return the (allow, disallow) path lists for one user-agent group.
A specific group wins over * entirely -- Google does not merge them.
"""
groups, current = {}, None
for raw in text.splitlines():
line = raw.split("#", 1)[0].strip()
if not line or ":" not in line:
continue
field, _, value = line.partition(":")
field, value = field.strip().lower(), value.strip()
if field == "user-agent":
current = value.lower()
groups.setdefault(current, {"allow": [], "disallow": []})
elif field in ("allow", "disallow") and current is not None:
groups[current][field].append(value)
g = groups.get(agent.lower()) or groups.get("*") or {"allow": [], "disallow": []}
return g["allow"], g["disallow"]
def _match_len(pattern, path):
"""Length of the match, or -1. Supports * and $ as Google does."""
if pattern == "":
return -1
rx = "^" + re.escape(pattern).replace(r"\*", ".*").replace(r"\$", "$")
return len(pattern) if re.match(rx, path) else -1
def is_blocked(path, allow, disallow):
"""Google's rule: the longest matching path wins; Allow breaks a tie.
Reading the file top to bottom gives the wrong answer, which is how people
conclude a page is allowed when it is not.
"""
best_allow = max((_match_len(p, path) for p in allow), default=-1)
best_disallow = max((_match_len(p, path) for p in disallow), default=-1)
if best_disallow < 0:
return False
return best_disallow > best_allow
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--site", required=True, help="https://example.com")
ap.add_argument("--url", nargs="+", required=True)
ap.add_argument("--agent", default="Googlebot")
args = ap.parse_args()
s = requests.Session()
r = s.get(args.site.rstrip("/") + "/robots.txt", timeout=30)
if len(r.content) > MAX_ROBOTS_BYTES:
log.warning("robots.txt is over 500 KiB -- Google ignores the rest of the file")
allow, disallow = parse_robots(r.text, args.agent)
log.info("%d allow rule(s), %d disallow rule(s)", len(allow), len(disallow))
conflicts = 0
for url in args.url:
path = urlsplit(url).path or "/"
blocked = is_blocked(path, allow, disallow)
if blocked and ASSET.search(path):
log.warning("%s is a blocked asset -- Google cannot render the page as a "
"visitor sees it", url)
continue
if not blocked:
log.info("%s crawlable", url)
continue
page = s.get(url, timeout=30)
meta = re.search(r'<meta[^>]+name=["\']robots["\'][^>]*>', page.text, re.I)
noindex = bool(meta and "noindex" in meta.group(0).lower()) or \
"noindex" in page.headers.get("X-Robots-Tag", "").lower()
if noindex:
conflicts += 1
log.error("%s is BLOCKED and marked noindex -- the crawler never reads the "
"noindex. Unblock it, wait for the recrawl, then reblock.", url)
else:
log.info("%s blocked, no noindex -- may still be indexed without a snippet", url)
return 1 if conflicts else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find pages that are both Disallowed in robots.txt and marked noindex.
*
* Those two rules cancel: the crawler never fetches the page, so it never reads the
* noindex, and the URL can stay indexed without a snippet indefinitely.
*/
const MAX_ROBOTS_BYTES = 500 * 1024; // Google parses the first 500 KiB
const ASSET = /\.(css|js|mjs|woff2?|svg|png|jpe?g|webp)$/i;
/**
* Return { allow, disallow } for one user-agent group.
* A specific group wins over * entirely -- Google does not merge them.
*/
export function parseRobots(text, agent = '*') {
const groups = {};
let current = null;
for (const raw of text.split(/\r?\n/)) {
const line = raw.split('#')[0].trim();
if (!line || !line.includes(':')) continue;
const [field, ...rest] = line.split(':');
const key = field.trim().toLowerCase();
const value = rest.join(':').trim();
if (key === 'user-agent') {
current = value.toLowerCase();
groups[current] ??= { allow: [], disallow: [] };
} else if ((key === 'allow' || key === 'disallow') && current !== null) {
groups[current][key].push(value);
}
}
return groups[agent.toLowerCase()] ?? groups['*'] ?? { allow: [], disallow: [] };
}
const matchLen = (pattern, path) => {
if (pattern === '') return -1;
const rx = new RegExp(`^${pattern.replace(/[.+^${}()|[\]\\]/g, '\\$&')
.replace(/\*/g, '.*').replace(/\\\$$/, '$')}`);
return rx.test(path) ? pattern.length : -1;
};
/**
* Google's rule: the longest matching path wins; Allow breaks a tie. Reading the
* file top to bottom gives the wrong answer.
*/
export function isBlocked(path, allow, disallow) {
const bestAllow = Math.max(-1, ...allow.map((p) => matchLen(p, path)));
const bestDisallow = Math.max(-1, ...disallow.map((p) => matchLen(p, path)));
if (bestDisallow < 0) return false;
return bestDisallow > bestAllow;
}
async function main() {
const arg = (n) => process.argv[process.argv.indexOf(n) + 1];
const site = arg('--site');
const ui = process.argv.indexOf('--url');
const urls = process.argv.slice(ui + 1).filter((a) => !a.startsWith('--'));
const agent = process.argv.includes('--agent') ? arg('--agent') : 'Googlebot';
const res = await fetch(`${site.replace(/\/$/, '')}/robots.txt`);
const text = await res.text();
if (Buffer.byteLength(text) > MAX_ROBOTS_BYTES) {
console.warn('robots.txt is over 500 KiB -- Google ignores the rest of the file');
}
const { allow, disallow } = parseRobots(text, agent);
console.log(`${allow.length} allow rule(s), ${disallow.length} disallow rule(s)`);
let conflicts = 0;
for (const url of urls) {
const path = new URL(url).pathname || '/';
const blocked = isBlocked(path, allow, disallow);
if (blocked && ASSET.test(path)) {
console.warn(`${url} is a blocked asset -- Google cannot render the page as a visitor sees it`);
continue;
}
if (!blocked) { console.log(`${url} crawlable`); continue; }
const page = await fetch(url);
const body = await page.text();
const meta = body.match(/<meta[^>]+name=["']robots["'][^>]*>/i);
const noindex = Boolean(meta && meta[0].toLowerCase().includes('noindex'))
|| (page.headers.get('x-robots-tag') ?? '').toLowerCase().includes('noindex');
if (noindex) {
conflicts += 1;
console.error(`${url} is BLOCKED and marked noindex -- the crawler never reads the `
+ 'noindex. Unblock it, wait for the recrawl, then reblock.');
} else {
console.log(`${url} blocked, no noindex -- may still be indexed without a snippet`);
}
}
process.exit(conflicts ? 1 : 0);
}
if (import.meta.url === `file://${process.argv[1]}`) main();
Add a test
These tests encode the matching rule that trips people up. A specific Allow under a broad Disallow wins because it is longer, not because of where it sits in the file — and an equal-length tie goes to Allow.
from robots_conflict import is_blocked, parse_robots
TXT = """
User-agent: *
Disallow: /private/
Allow: /private/public-page
User-agent: Googlebot
Disallow: /admin/
"""
def test_a_specific_agent_group_wins_over_star():
"""Google does not merge groups; the specific one replaces the wildcard."""
allow, disallow = parse_robots(TXT, "Googlebot")
assert disallow == ["/admin/"]
def test_the_star_group_is_used_for_an_unknown_agent():
allow, disallow = parse_robots(TXT, "SomeOtherBot")
assert disallow == ["/private/"]
def test_an_unmatched_path_is_allowed():
assert not is_blocked("/about", [], ["/private/"])
def test_a_disallowed_path_is_blocked():
assert is_blocked("/private/x", [], ["/private/"])
def test_a_longer_allow_beats_a_shorter_disallow():
"""Position in the file is irrelevant; length decides."""
assert not is_blocked("/private/public-page", ["/private/public-page"], ["/private/"])
def test_a_tie_goes_to_allow():
assert not is_blocked("/x/", ["/x/"], ["/x/"])
def test_an_empty_disallow_blocks_nothing():
assert not is_blocked("/anything", [], [""])
def test_a_wildcard_pattern_matches():
assert is_blocked("/search?q=1", [], ["/*?"])
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { isBlocked, parseRobots } from './robots-conflict.mjs';
const TXT = `
User-agent: *
Disallow: /private/
Allow: /private/public-page
User-agent: Googlebot
Disallow: /admin/
`;
test('a specific agent group wins over *', () => {
assert.deepEqual(parseRobots(TXT, 'Googlebot').disallow, ['/admin/']);
});
test('the * group is used for an unknown agent', () => {
assert.deepEqual(parseRobots(TXT, 'SomeOtherBot').disallow, ['/private/']);
});
test('a disallowed path is blocked', () => {
assert.equal(isBlocked('/private/x', [], ['/private/']), true);
});
test('a longer allow beats a shorter disallow', () => {
assert.equal(isBlocked('/private/public-page', ['/private/public-page'], ['/private/']), false);
});
test('a tie goes to allow', () => {
assert.equal(isBlocked('/x/', ['/x/'], ['/x/']), false);
});
test('an empty disallow blocks nothing', () => {
assert.equal(isBlocked('/anything', [], ['']), false);
});
FAQ
Why is my noindex page still in Google?
Most often because robots.txt blocks it. The crawler never fetches the page, so it never reads the noindex. Allow the crawl, wait for a recrawl, and it will drop out — then you can block it again if you want.
What does 'Indexed, though blocked by robots.txt' mean?
Google learned the URL exists from links pointing at it, but was not allowed to fetch it. It can list the URL without a description, because it has no content to show.
Does robots.txt stop a page being indexed?
No. It stops crawling. Indexing is controlled by noindex, which requires a crawl to be read. They are separate stages, which is exactly why stacking both rules fails.
How does Google resolve conflicting rules?
The most specific rule wins — the longest matching path — and on a tie, Allow beats Disallow. Position in the file does not matter, so reading top to bottom gives the wrong answer.
Is there a size limit on robots.txt?
Google parses the first 500 KiB and ignores the rest. A large generated file may have rules that are simply never read.
Should I block my CSS and JavaScript?
No. Blocking them stops Google rendering the page the way a visitor sees it. These rules are usually years old and made for a reason that no longer applies.
Related field notes
- A sitemap listing URLs that redirect, 404 or say noindex
- A canonical pointing at staging or a redirect
- Technical SEO field notes
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Robots.txt specification — Google Search Central
- Introduction to robots.txt — Google Search Central
- Block search indexing with noindex — Google Search Central
- Page indexing report — Google Search Console Help
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.