Diagnostic Technical SEO

a missing page that returns 200 instead of 404

Search Console is reporting Soft 404 on pages that look fine, or your index count is far higher than the number of pages you actually have. Somewhere a catch-all is answering every request with 200 OK and a page that says “not found” in the body. To a crawler that is a real page with thin content, so it gets crawled, judged and sometimes indexed.

No API key needed Python and Node.js Probes real URLs

The short answer

Request a URL that certainly does not exist and read the status line. If it is 200, your not-found page is lying to crawlers.

Serve a real 404, or 410 for something you deliberately removed. Both are correct; 410 signals permanence and tends to be dropped a little faster. What matters is that the status line and the page content agree.

The problem in plain words

It looks right to a person. The page says the thing was not found, so nobody thinks to check the status code — and browsers do not show it. Only a crawler, or a curl -I, notices the disagreement.

The cost is real: crawl budget spent on URLs that do not exist, thin pages competing with your actual content, and an index count that stops meaning anything. On a large site with a parameterised catch-all, the set of fake URLs is effectively unbounded.

Why it happens

Client-side routing decides after the response. The server sends 200 and the shell, and JavaScript works out later that the route is unknown. The status code was already committed before anyone knew the page did not exist.

Catch-all rules are written for the happy path. A rewrite that sends everything to index.html is what makes deep links work on a static host. The same rule sends every typo there too.

Redirecting to the homepage is the same bug wearing a hat. A missing page that 302s to / also reports success for something that does not exist, and Google treats it as a soft 404 too. It is a common well-meant fix that makes the problem harder to see.

An empty page is also a soft 404. A category with no products, a search with no results, a profile that was emptied — these return 200 honestly and still have nothing on them. The classification is about substance, not just status.

How to fix it

Probe a URL that cannot exist

Add a random segment to a real path. The status line is the whole answer.

curl -sI https://example.com/definitely-not-a-real-page-9f3a | head -1
# HTTP/2 404   <- correct
# HTTP/2 200   <- soft 404

Probe several shapes, not one

A missing top-level path, a missing path under a real section, and a missing item with a real-looking ID often go through different handlers. Sites commonly get one right and two wrong.

Fix the status at the layer that owns the route

For server-rendered apps, return the status from the route handler. For static hosts, configure a 404 document rather than a catch-all rewrite to the index. For client-side routing, prerender or serve a real 404 for unknown paths — JavaScript cannot change a status code that has already been sent.

Use 410 for things you removed on purpose

404 means “not here”; 410 means “gone, do not come back”. Both work, and 410 tends to be dropped from the index slightly faster.

Do not redirect missing pages to the homepage

It reports success for something that does not exist, and it wastes the visitor's time too — they wanted a specific thing and got a front page with no explanation.

How to check it worked

Re-probe and check both the status and the body:

curl -si https://example.com/definitely-not-a-real-page-9f3a | head -1
curl -s  https://example.com/definitely-not-a-real-page-9f3a | grep -io 'not found' | head -1

A correct not-found page returns 404 and says so. Then use URL Inspection on a URL Search Console flagged, and confirm it now reports the 404.

The full code

The script probes each path you give it plus a generated nonsense URL under each, and classifies the response by status, redirect behaviour and body length together. The classification is a pure function, because the interesting cases — a 200 with not-found text, a redirect to the homepage, a 200 with almost no content — are judgement calls that deserve to be visible and tested rather than buried in a request loop.

soft_404_probe.py

"""Find URLs that report success for a page that does not exist.

Browsers do not show status codes, so a not-found page that returns 200 looks
correct to everyone except a crawler. Google calls it a soft 404.
"""
import argparse
import logging
import re
import sys
from urllib.parse import urlsplit

import requests

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("soft_404_probe")

NOT_FOUND_TEXT = re.compile(
    r"\b(not found|doesn.t exist|does not exist|no longer available|page missing|404)\b", re.I)
THIN_BYTES = 2_000  # visible text below this is thin enough to be worth reporting


def classify(status, final_path, requested_path, visible_text):
    """Pure decision function for one probe.

    Returns (is_problem, message). The judgement calls live here on purpose: a 200
    carrying not-found text and a redirect to the homepage are both soft 404s, and
    both look like success from a request loop.
    """
    said_missing = bool(NOT_FOUND_TEXT.search(visible_text[:4000]))
    if status in (404, 410):
        return False, f"{status} -- correct"
    if status >= 500:
        return True, f"{status} -- server error, not a 404; crawlers retry these"
    if 300 <= status < 400 or final_path != requested_path:
        if final_path in ("/", "", "/index.html"):
            return True, (f"redirects to the homepage -- a soft 404. Return 404 or 410 "
                          "so the crawler knows the URL is dead.")
        return True, f"redirects to {final_path} -- reports success for a missing page"
    if status == 200 and said_missing:
        return True, ("200 with not-found text -- a soft 404. The status line and the "
                      "page content disagree.")
    if status == 200 and len(visible_text.strip()) < THIN_BYTES:
        return True, (f"200 with {len(visible_text.strip())} bytes of text -- thin enough "
                      "that Google may treat it as a soft 404")
    return False, f"{status} -- looks like a real page"


def visible(html):
    """Crude text extraction. Good enough to tell 'empty' from 'has content'."""
    body = re.sub(r"(?is)<(script|style|template)[^>]*>.*?</\1>", " ", html)
    return re.sub(r"\s+", " ", re.sub(r"(?s)<[^>]+>", " ", body))


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--url", nargs="+", required=True,
                    help="real section URLs; a nonsense child is probed under each")
    ap.add_argument("--nonce", default="zz-not-a-real-page-9f3a",
                    help="segment appended to build a URL that cannot exist")
    args = ap.parse_args()

    s = requests.Session()
    s.headers.update({"User-Agent": "soft-404-probe/1.0"})

    problems = 0
    for base in args.url:
        probe = base.rstrip("/") + "/" + args.nonce
        try:
            r = s.get(probe, timeout=30, allow_redirects=True)
        except requests.RequestException as e:
            log.error("%s -- request failed: %s", probe, e.__class__.__name__)
            problems += 1
            continue
        status = r.history[0].status_code if r.history else r.status_code
        bad, msg = classify(status, urlsplit(r.url).path,
                            urlsplit(probe).path, visible(r.text))
        (log.error if bad else log.info)("%s -- %s", probe, msg)
        problems += bool(bad)
    log.info("%d problem(s)", problems)
    return 1 if problems else 0


if __name__ == "__main__":
    sys.exit(main())

soft-404-probe.mjs

/**
 * Find URLs that report success for a page that does not exist.
 *
 * Browsers do not show status codes, so a not-found page returning 200 looks correct
 * to everyone except a crawler. Google calls it a soft 404.
 */
const NOT_FOUND_TEXT =
  /\b(not found|doesn.t exist|does not exist|no longer available|page missing|404)\b/i;
const THIN_BYTES = 2000;

/**
 * Pure decision function for one probe. Returns [isProblem, message].
 * The judgement calls live here: a 200 carrying not-found text and a redirect to the
 * homepage are both soft 404s, and both look like success from a request loop.
 */
export function classify(status, finalPath, requestedPath, visibleText) {
  const saidMissing = NOT_FOUND_TEXT.test(visibleText.slice(0, 4000));
  if (status === 404 || status === 410) return [false, `${status} -- correct`];
  if (status >= 500) return [true, `${status} -- server error, not a 404; crawlers retry these`];
  if ((status >= 300 && status < 400) || finalPath !== requestedPath) {
    if (['/', '', '/index.html'].includes(finalPath)) {
      return [true, 'redirects to the homepage -- a soft 404. Return 404 or 410 so the '
        + 'crawler knows the URL is dead.'];
    }
    return [true, `redirects to ${finalPath} -- reports success for a missing page`];
  }
  if (status === 200 && saidMissing) {
    return [true, '200 with not-found text -- a soft 404. The status line and the page '
      + 'content disagree.'];
  }
  if (status === 200 && visibleText.trim().length < THIN_BYTES) {
    return [true, `200 with ${visibleText.trim().length} bytes of text -- thin enough that `
      + 'Google may treat it as a soft 404'];
  }
  return [false, `${status} -- looks like a real page`];
}

/** Crude text extraction. Good enough to tell 'empty' from 'has content'. */
export const visible = (html) => html
  .replace(/<(script|style|template)[^>]*>[\s\S]*?<\/\1>/gi, ' ')
  .replace(/<[^>]+>/g, ' ')
  .replace(/\s+/g, ' ');

async function main() {
  const ui = process.argv.indexOf('--url');
  const bases = process.argv.slice(ui + 1).filter((a) => !a.startsWith('--'));
  const nonce = process.argv.includes('--nonce')
    ? process.argv[process.argv.indexOf('--nonce') + 1] : 'zz-not-a-real-page-9f3a';

  let problems = 0;
  for (const base of bases) {
    const probe = `${base.replace(/\/$/, '')}/${nonce}`;
    let r;
    try { r = await fetch(probe, { redirect: 'follow' }); }
    catch (e) { console.error(`${probe} -- request failed: ${e.name}`); problems += 1; continue; }
    const text = visible(await r.text());
    const status = r.redirected ? 302 : r.status;
    const [bad, msg] = classify(status, new URL(r.url).pathname, new URL(probe).pathname, text);
    (bad ? console.error : console.log)(`${probe} -- ${msg}`);
    if (bad) problems += 1;
  }
  console.log(`${problems} problem(s)`);
  process.exit(problems ? 1 : 0);
}

if (import.meta.url === `file://${process.argv[1]}`) main();

Add a test

The tests hold the line on what counts as correct. A real 404 with a short body is fine — a not-found page is supposed to be short — so the thin-content rule must not fire on it, or the check reports every well-behaved site as broken.

test_soft_404_probe.py

from soft_404_probe import classify, visible

LONG = "real content " * 400


def test_a_real_404_is_correct():
    bad, msg = classify(404, "/missing", "/missing", "Not found")
    assert not bad and "correct" in msg


def test_a_410_is_correct():
    assert not classify(410, "/gone", "/gone", "Gone")[0]


def test_a_short_404_body_is_not_flagged_as_thin():
    """A not-found page is supposed to be short. Flagging it reports every good site."""
    assert not classify(404, "/missing", "/missing", "Not found")[0]


def test_200_with_not_found_text_is_a_soft_404():
    bad, msg = classify(200, "/missing", "/missing", "Sorry, page not found. " + LONG)
    assert bad and "soft 404" in msg


def test_a_redirect_to_the_homepage_is_a_soft_404():
    bad, msg = classify(302, "/", "/missing", LONG)
    assert bad and "homepage" in msg


def test_a_thin_200_is_reported():
    bad, msg = classify(200, "/empty", "/empty", "   ")
    assert bad and "thin" in msg


def test_a_real_page_passes():
    assert not classify(200, "/real", "/real", LONG)[0]


def test_a_500_is_not_treated_as_a_404():
    bad, msg = classify(500, "/x", "/x", "")
    assert bad and "server error" in msg


def test_visible_strips_scripts():
    assert "alert" not in visible("<script>alert(1)</script><p>hi</p>")

soft-404-probe.test.mjs

import { test } from 'node:test';
import assert from 'node:assert/strict';
import { classify, visible } from './soft-404-probe.mjs';

const LONG = 'real content '.repeat(400);

test('a real 404 is correct', () => {
  const [bad, msg] = classify(404, '/missing', '/missing', 'Not found');
  assert.equal(bad, false);
  assert.ok(msg.includes('correct'));
});

test('a short 404 body is not flagged as thin', () => {
  assert.equal(classify(404, '/missing', '/missing', 'Not found')[0], false);
});

test('200 with not-found text is a soft 404', () => {
  const [bad, msg] = classify(200, '/missing', '/missing', `Sorry, page not found. ${LONG}`);
  assert.equal(bad, true);
  assert.ok(msg.includes('soft 404'));
});

test('a redirect to the homepage is a soft 404', () => {
  const [bad, msg] = classify(302, '/', '/missing', LONG);
  assert.equal(bad, true);
  assert.ok(msg.includes('homepage'));
});

test('a real page passes', () => {
  assert.equal(classify(200, '/real', '/real', LONG)[0], false);
});

test('visible strips scripts', () => {
  assert.ok(!visible('<script>alert(1)</script><p>hi</p>').includes('alert'));
});

FAQ

What is a soft 404?

A URL that returns 200 OK for a page that does not exist, or that has effectively no content. To a crawler it is a real page with thin content, so it gets crawled, assessed and sometimes indexed.

Why does my SPA return 200 for missing routes?

The server sends the shell before anything knows the route is unknown, and JavaScript decides afterwards. A status code cannot be changed once it has been sent — the fix has to happen at the server or host layer.

Should I use 404 or 410?

Both are correct. 404 means not here; 410 means gone and not coming back. Use 410 for something you deliberately removed — it tends to be dropped from the index slightly faster.

Is redirecting missing pages to the homepage a good fix?

No. It is the same problem in a different form: you are reporting success for a URL that does not exist, and Google treats it as a soft 404. It also wastes the visitor's time.

Can a page that returns 200 honestly still be a soft 404?

Yes. An empty category, a search with no results, or a profile with nothing on it all return 200 truthfully and still have nothing to show. The classification is about substance, not just the status line.

Does this waste crawl budget?

Yes, and unboundedly on a site with a parameterised catch-all — every typo becomes a crawlable URL. That is the practical reason to fix it even if nothing is being indexed.

Related field notes

Sources

Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.

Stuck on a tricky one?

If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.