Diagnostic URL Rewrites

Disabling rewrite setting still creates duplicate rewrites

You went into Catalog, Search Engine Optimization and set Generate category/product URL Rewrites to No, expecting products to keep one clean URL each. Instead a product assigned to a category still ends up with two rows in url_rewrite: a plain product URL and a category-prefixed one, as if the setting did nothing. Here is why the toggle only stops half the story, and a small script that finds the duplicate pairs over REST and reports them for repair.

Python and Node.js Magento REST API Safe by default (dry run)

Printed paper — Photo by Kelly Sikkema on Unsplash

The short answer

The catalog/seo/generate_category_product_rewrites setting only gates the CategoryProcessUrlRewriteSavingObserver and CanonicalUrlRewriteGenerator path, which appends a category segment to a product's canonical URL when a category is saved. It does not gate the separate ProductProcessUrlRewriteSavingObserver, which independently writes a plain, non-category url_rewrite row on catalog_product_save_after every time the product itself is saved. Because the two generators fire from separate observers on separate save events, and the config check is applied inconsistently between them, a product assigned to a category still gets one autogenerated product-only rewrite and one category/product rewrite even with the setting set to No. This is a confirmed core bug, tracked as magento/magento2 issue 38317 and issue 39070, not a merchant misconfiguration. Run a small Python or Node.js script that resolves each product's SKU over REST, groups its rewrite rows by store and target path, and flags the pairs where one request_path is category-prefixed and one is not. Full code, tests, and sources are below.

The problem in plain words

Magento generates a product's URL rewrite in two separate places. When you save a product, ProductProcessUrlRewriteSavingObserver writes a plain rewrite for it, something like green-shirt.html. When you save a category that the product belongs to, a different path through CategoryProcessUrlRewriteSavingObserver and CanonicalUrlRewriteGenerator can also write a category-prefixed rewrite for the same product, something like mens/shirts/green-shirt.html.

The Generate category/product URL Rewrites setting is supposed to be the single switch that turns the second kind off. Turn it to No, and you would expect every product to keep just its plain rewrite. What actually happens is that the setting is only checked inside the category-save code path. The product-save code path that writes the plain rewrite never looks at the setting at all, because it does not need to since it was never generating a category rewrite in the first place. So both writes happen on their own schedule, from their own observer, and a product that belongs to a category ends up with two live rows in url_rewrite no matter what the setting says.

Two independent observers write two independent rewrite rows. Only one of them checks the setting, and it checks it too late to stop the other.

Why it happens

The behavior traces back to how Magento split the rewrite generation logic across two unrelated save events, both of which can produce a rewrite for the same product:

Saving a product fires catalog_product_save_after, which the ProductProcessUrlRewriteSavingObserver handles by writing a plain, non-category product rewrite. This path has no reason to check generate_category_product_rewrites, because it never generated a category-prefixed path to begin with.
Saving a category fires a separate event chain, handled by CategoryProcessUrlRewriteSavingObserver, which calls into CanonicalUrlRewriteGenerator to build the category-prefixed rewrite for every product assigned to that category. This is the path meant to respect the setting.
Because these are two different observers reacting to two different save actions, disabling the setting on one path does nothing to the other. A product saved after being assigned to a category, or a category resaved with the setting off, can still trigger the canonical generator depending on the order operations run in, reproduced across Magento 2.4.7-beta2 through 2.4.7-p2 in magento/magento2 issue 38317 and issue 39070.
Related community reports describe the same class of problem from the other direction, category URL rewrites failing to regenerate consistently after changing SEO config settings, in magento/magento2 issue 17585.

The result is confusing because the Admin setting looks authoritative. It has one clear label, one Yes or No toggle, and a description that reads like it controls exactly this. Store owners turn it off, confirm it saved, and still find category-path rewrites showing up in the rewrite grid for products they never expected to have one. See the citations at the end for the exact reports.

The key insight

Do not treat this as a data problem you can vacuum away with a delete query. It is core generator behavior, and the extra row will come back on the next product or category save while the bug is unpatched. What a script can safely do is detect the duplicate pairs by comparing rewrite rows for the same product, same store, and same underlying target, then report which one is the redundant category-path row. Deleting is a separate, explicit, human-approved step, never the default action.

The fix, as a flow

The script never assumes it knows which row is safe to remove. It resolves each product id to a SKU over REST, collects that product's rewrite rows grouped by store and target path, and only flags a pair as a duplicate when the setting is confirmed off and one request_path is longer or category-prefixed compared to the other. Everything it finds is reported. A guarded delete only runs when a human passes an explicit apply flag, and it never touches the row that matches the product's current plain url_key, since that is the one serving live traffic.

The script always reports what it finds. It only performs a guarded delete of the redundant category-path row when a human explicitly passes an apply flag, and never touches the row currently serving live traffic.

Build it step by step

Get an admin bearer token

Call POST /rest/V1/integration/admin/token with your admin username and password, or use a preconfigured integration token. Either way you end up with a bearer token you send as Authorization: Bearer <token> on every call. Keep the token and the store URL in environment variables, never in the file.

setup (shell)

pip install requests

export MAGENTO_URL="https://your-store.example.com"
export MAGENTO_ADMIN_TOKEN="your admin bearer token"
export DRY_RUN="true"   # start safe, change to false only with --apply

setup (shell)

// Node 18+ has fetch built in, no dependencies needed

export MAGENTO_URL="https://your-store.example.com"
export MAGENTO_ADMIN_TOKEN="your admin bearer token"
export DRY_RUN="true"   // start safe, change to false only with --apply

Talk to the Magento REST API

Every call goes to {MAGENTO_URL}/rest/V1 with your token in the Authorization header. A small helper sends the request and raises on a non success status, and we reuse it to resolve the SKU for each product id and to read the store's config.

step2.py

import os, requests

MAGENTO_URL = os.environ["MAGENTO_URL"].rstrip("/")
TOKEN = os.environ["MAGENTO_ADMIN_TOKEN"]
HEADERS = {"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"}

def api_get(path, params=None):
    r = requests.get(f"{MAGENTO_URL}/rest/V1{path}", headers=HEADERS, params=params or {}, timeout=30)
    r.raise_for_status()
    return r.json()

step2.js

const MAGENTO_URL = (process.env.MAGENTO_URL || "").replace(/\/+$/, "");
const TOKEN = process.env.MAGENTO_ADMIN_TOKEN;
const HEADERS = { Authorization: `Bearer ${TOKEN}`, "Content-Type": "application/json" };

async function apiGet(path, params = {}) {
  const url = new URL(`${MAGENTO_URL}/rest/V1${path}`);
  for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
  const res = await fetch(url, { headers: HEADERS });
  if (!res.ok) throw new Error(`Magento ${res.status}`);
  return res.json();
}

Resolve the SKU for each product id

Core REST has no direct search endpoint over url_rewrite, so the script starts from the product id you want to check. GET /rest/V1/products filtered by entity_id equal to that id resolves the SKU. From there the rewrite rows for that product are read from your read-only export or admin-exposed rewrite list, since url_rewrite itself has no public REST endpoint either.

step3.py

def resolve_sku_by_entity_id(product_id):
    params = {
        "searchCriteria[filterGroups][0][filters][0][field]": "entity_id",
        "searchCriteria[filterGroups][0][filters][0][value]": product_id,
        "searchCriteria[filterGroups][0][filters][0][conditionType]": "eq",
    }
    result = api_get("/products", params)
    items = result.get("items", [])
    return items[0]["sku"] if items else None

def generate_category_product_rewrites_enabled():
    # Not exposed on a public REST GET; read via admin config export or CLI
    # (bin/magento config:show catalog/seo/generate_category_product_rewrites)
    # and pass the boolean into the pure function below.
    return os.environ.get("GENERATE_CATEGORY_PRODUCT_REWRITES", "false").lower() == "true"

step3.js

async function resolveSkuByEntityId(productId) {
  const params = {
    "searchCriteria[filterGroups][0][filters][0][field]": "entity_id",
    "searchCriteria[filterGroups][0][filters][0][value]": productId,
    "searchCriteria[filterGroups][0][filters][0][conditionType]": "eq",
  };
  const result = await apiGet("/products", params);
  const items = result.items || [];
  return items.length ? items[0].sku : null;
}

function generateCategoryProductRewritesEnabled() {
  // Not exposed on a public REST GET; read via admin config export or CLI
  // (bin/magento config:show catalog/seo/generate_category_product_rewrites)
  // and pass the boolean into the pure function below.
  return (process.env.GENERATE_CATEGORY_PRODUCT_REWRITES || "false").toLowerCase() === "true";
}

Decide, with one pure function

Keep the decision in its own function that takes the rewrite rows for a store, the product id, and whether the setting is enabled, and returns the duplicate pairs it finds. It filters to entity_type equal to product and the matching entity_id, groups by store_id, and within each group looks for rows that share the same target_path where one request_path has more path segments than the other. When the setting is off and more than one such row exists, it marks the shorter, non-category path to keep and the longer, category-containing path to remove. No network calls, so it is easy to test with fixture arrays.

decide.py

def find_duplicate_product_rewrites(rows, product_id, generate_category_rewrites_enabled):
    if generate_category_rewrites_enabled:
        return []

    product_rows = [
        r for r in rows
        if r.get("entity_type") == "product" and r.get("entity_id") == product_id
    ]

    by_store = {}
    for row in product_rows:
        by_store.setdefault(row["store_id"], []).append(row)

    pairs = []
    for store_id, store_rows in by_store.items():
        by_target = {}
        for row in store_rows:
            by_target.setdefault(row["target_path"], []).append(row)
        for target_path, target_rows in by_target.items():
            if len(target_rows) <= 1:
                continue
            ordered = sorted(target_rows, key=lambda r: r["request_path"].count("/"))
            keep_row = ordered[0]
            for remove_row in ordered[1:]:
                if remove_row["request_path"].count("/") > keep_row["request_path"].count("/"):
                    pairs.append({
                        "keep": keep_row["url_rewrite_id"],
                        "remove": remove_row["url_rewrite_id"],
                        "reason": "duplicate-despite-disabled-setting",
                    })
    return pairs

decide.js

export function findDuplicateProductRewrites(rows, productId, generateCategoryRewritesEnabled) {
  if (generateCategoryRewritesEnabled) return [];

  const productRows = rows.filter(
    (r) => r.entity_type === "product" && r.entity_id === productId
  );

  const byStore = new Map();
  for (const row of productRows) {
    if (!byStore.has(row.store_id)) byStore.set(row.store_id, []);
    byStore.get(row.store_id).push(row);
  }

  const pairs = [];
  for (const [, storeRows] of byStore) {
    const byTarget = new Map();
    for (const row of storeRows) {
      if (!byTarget.has(row.target_path)) byTarget.set(row.target_path, []);
      byTarget.get(row.target_path).push(row);
    }
    for (const [, targetRows] of byTarget) {
      if (targetRows.length <= 1) continue;
      const segments = (path) => (path.match(/\//g) || []).length;
      const ordered = [...targetRows].sort((a, b) => segments(a.request_path) - segments(b.request_path));
      const keepRow = ordered[0];
      for (const removeRow of ordered.slice(1)) {
        if (segments(removeRow.request_path) > segments(keepRow.request_path)) {
          pairs.push({
            keep: keepRow.url_rewrite_id,
            remove: removeRow.url_rewrite_id,
            reason: "duplicate-despite-disabled-setting",
          });
        }
      }
    }
  }
  return pairs;
}

Report first, delete only with an explicit apply flag

By default the script only logs what it found: the product id, the url_rewrite_id to keep, the one it would remove, and why. It recommends applying the corrective core patch for issue 38317 and issue 39070, or adding a small after-plugin on Magento\CatalogUrlRewrite\Model\Category\CanonicalUrlRewriteGenerator::generate that drops the redundant row. Only when --apply is explicitly passed does it perform a guarded delete of the flagged row, and it always double-checks that row is not the one matching the product's current plain url_key, since that is the row serving live traffic.

apply.py

def is_live_traffic_row(row, current_url_key):
    # Never remove the plain row matching the product's current url_key.
    # Compare the whole request_path, not just its last segment, so a
    # category-prefixed path is never mistaken for the plain one.
    return row["request_path"].rstrip("/") == f"{current_url_key}.html"

def report_pair(product_id, sku, pair):
    logging.getLogger("disabled_rewrite_setting_still_duplicates").warning(
        "Product id=%s sku=%s: url_rewrite_id=%s is redundant next to keep=%s (%s)",
        product_id, sku, pair["remove"], pair["keep"], pair["reason"],
    )

apply.js

function isLiveTrafficRow(row, currentUrlKey) {
  // Never remove the plain row matching the product's current url_key.
  // Compare the whole request_path, not just its last segment, so a
  // category-prefixed path is never mistaken for the plain one.
  return row.request_path.replace(/\/+$/, "") === `${currentUrlKey}.html`;
}

function reportPair(productId, sku, pair) {
  console.warn(
    `Product id=${productId} sku=${sku}: url_rewrite_id=${pair.remove} is redundant next to keep=${pair.keep} (${pair.reason})`
  );
}

Wire it together with a dry run guard

The loop resolves each product id to a SKU, reads its rewrite rows, runs the pure function, and reports every pair it finds. With DRY_RUN=true, the default, nothing is written. Only when a human runs the script with both DRY_RUN=false and --apply does it delete the flagged row through an admin action, always skipping any row that matches the product's current url_key.

Run it safe

Always start with DRY_RUN=true and read the report first. This is core generator behavior, not a data entry mistake, so a delete without --apply should never happen, and the row serving live traffic should never be touched even with --apply set.

The full code

Here is the complete script in one file for each language. It reads settings from the environment, resolves each product id to a SKU, groups its rewrite rows, flags the duplicate pairs with the pure function, and only performs a guarded delete of the redundant row when explicitly told to.

Get this script on GitHub Follow @allanninal Python and Node.js, with tests. Dry run by default. One of 59 Magento fixes, free and open source.

disabled_rewrite_setting_still_duplicates.py

"""Detect Magento products that still get a category/product url_rewrite
duplicate even though catalog/seo/generate_category_product_rewrites is off.

This is a confirmed core bug (magento/magento2 issues 38317 and 39070), not a
misconfiguration: ProductProcessUrlRewriteSavingObserver writes a plain
product rewrite on every product save regardless of the setting, while
CategoryProcessUrlRewriteSavingObserver and CanonicalUrlRewriteGenerator can
still write a category-prefixed rewrite for the same product on a category
save. This script resolves each product id to a SKU over REST, reads that
product's url_rewrite rows from a read-only export or admin-exposed rewrite
list (url_rewrite has no public REST search endpoint), flags the duplicate
pairs with a pure function, and only performs a guarded delete of the
redundant row when --apply is explicitly passed. Report only by default.
"""
import os
import sys
import logging
import requests

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("disabled_rewrite_setting_still_duplicates")

MAGENTO_URL = os.environ["MAGENTO_URL"].rstrip("/")
TOKEN = os.environ["MAGENTO_ADMIN_TOKEN"]
HEADERS = {"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"}
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"
GENERATE_CATEGORY_PRODUCT_REWRITES = os.environ.get(
    "GENERATE_CATEGORY_PRODUCT_REWRITES", "false"
).lower() == "true"
PRODUCT_IDS = [
    int(p) for p in os.environ.get("PRODUCT_IDS", "").split(",") if p.strip()
]


def api_get(path, params=None):
    r = requests.get(f"{MAGENTO_URL}/rest/V1{path}", headers=HEADERS, params=params or {}, timeout=30)
    r.raise_for_status()
    return r.json()


def api_delete(path):
    r = requests.delete(f"{MAGENTO_URL}/rest/V1{path}", headers=HEADERS, timeout=30)
    r.raise_for_status()
    return r.json()


def resolve_sku_by_entity_id(product_id):
    params = {
        "searchCriteria[filterGroups][0][filters][0][field]": "entity_id",
        "searchCriteria[filterGroups][0][filters][0][value]": product_id,
        "searchCriteria[filterGroups][0][filters][0][conditionType]": "eq",
    }
    result = api_get("/products", params)
    items = result.get("items", [])
    return items[0] if items else None


def custom_attr(attrs, code, default=None):
    for a in attrs or []:
        if a.get("attribute_code") == code:
            return a.get("value")
    return default


def read_url_rewrite_rows_for_product(product_id):
    """Read this product's url_rewrite rows from a read-only export or
    admin-exposed rewrite list. url_rewrite has no public REST search
    endpoint, so this is produced by an Admin grid export or a read-only
    DB query, keyed on entity_type='product' and entity_id=product_id."""
    import csv
    path = os.environ.get("URL_REWRITE_EXPORT_CSV", "url_rewrite_export.csv")
    rows = []
    with open(path, newline="", encoding="utf-8") as f:
        for row in csv.DictReader(f):
            if row["entity_type"] != "product" or int(row["entity_id"]) != product_id:
                continue
            rows.append({
                "url_rewrite_id": int(row["url_rewrite_id"]),
                "entity_type": row["entity_type"],
                "entity_id": int(row["entity_id"]),
                "request_path": row["request_path"],
                "target_path": row["target_path"],
                "store_id": int(row["store_id"]),
                "is_autogenerated": int(row.get("is_autogenerated", 0) or 0),
            })
    return rows


def find_duplicate_product_rewrites(rows, product_id, generate_category_rewrites_enabled):
    if generate_category_rewrites_enabled:
        return []

    product_rows = [
        r for r in rows
        if r.get("entity_type") == "product" and r.get("entity_id") == product_id
    ]

    by_store = {}
    for row in product_rows:
        by_store.setdefault(row["store_id"], []).append(row)

    pairs = []
    for store_id, store_rows in by_store.items():
        by_target = {}
        for row in store_rows:
            by_target.setdefault(row["target_path"], []).append(row)
        for target_path, target_rows in by_target.items():
            if len(target_rows) <= 1:
                continue
            ordered = sorted(target_rows, key=lambda r: r["request_path"].count("/"))
            keep_row = ordered[0]
            for remove_row in ordered[1:]:
                if remove_row["request_path"].count("/") > keep_row["request_path"].count("/"):
                    pairs.append({
                        "keep": keep_row["url_rewrite_id"],
                        "remove": remove_row["url_rewrite_id"],
                        "reason": "duplicate-despite-disabled-setting",
                    })
    return pairs


def is_live_traffic_row(row, current_url_key):
    if not current_url_key:
        return False
    return row["request_path"].rstrip("/") == f"{current_url_key}.html"


def run():
    apply_deletes = "--apply" in sys.argv
    total_pairs = 0

    for product_id in PRODUCT_IDS:
        product = resolve_sku_by_entity_id(product_id)
        if not product:
            log.info("Product id=%s not found, skipping", product_id)
            continue
        sku = product["sku"]
        current_url_key = custom_attr(product.get("custom_attributes"), "url_key", "")

        rows = read_url_rewrite_rows_for_product(product_id)
        pairs = find_duplicate_product_rewrites(rows, product_id, GENERATE_CATEGORY_PRODUCT_REWRITES)

        for pair in pairs:
            total_pairs += 1
            log.warning(
                "Product id=%s sku=%s: url_rewrite_id=%s is redundant next to keep=%s (%s)",
                product_id, sku, pair["remove"], pair["keep"], pair["reason"],
            )
            remove_row = next(r for r in rows if r["url_rewrite_id"] == pair["remove"])
            if is_live_traffic_row(remove_row, current_url_key):
                log.info("Skipping delete: row matches the product's current url_key, serving live traffic")
                continue
            if not apply_deletes or DRY_RUN:
                log.info(
                    "Recommend: apply the core patch for issue 38317/39070, or add an "
                    "after-plugin on CanonicalUrlRewriteGenerator::generate to drop this row. "
                    "Would delete url_rewrite_id=%s (pass --apply and DRY_RUN=false to delete)",
                    pair["remove"],
                )
                continue
            api_delete(f"/url-rewrites/{pair['remove']}")
            log.info("Deleted redundant url_rewrite_id=%s", pair["remove"])

    log.info("Done. %d duplicate pair(s) found across %d product(s).", total_pairs, len(PRODUCT_IDS))


if __name__ == "__main__":
    run()

disabled-rewrite-setting-still-duplicates.js

/**
 * Detect Magento products that still get a category/product url_rewrite
 * duplicate even though catalog/seo/generate_category_product_rewrites is off.
 *
 * This is a confirmed core bug (magento/magento2 issues 38317 and 39070), not
 * a misconfiguration: ProductProcessUrlRewriteSavingObserver writes a plain
 * product rewrite on every product save regardless of the setting, while
 * CategoryProcessUrlRewriteSavingObserver and CanonicalUrlRewriteGenerator can
 * still write a category-prefixed rewrite for the same product on a category
 * save. This script resolves each product id to a SKU over REST, reads that
 * product's url_rewrite rows from a read-only export, flags duplicate pairs
 * with a pure function, and only performs a guarded delete of the redundant
 * row when --apply is explicitly passed. Report only by default.
 *
 * Guide: https://www.allanninal.dev/magento/disabled-rewrite-setting-still-duplicates/
 */
import { pathToFileURL } from "node:url";
import { readFile } from "node:fs/promises";

const MAGENTO_URL = (process.env.MAGENTO_URL || "https://example.test").replace(/\/+$/, "");
const TOKEN = process.env.MAGENTO_ADMIN_TOKEN || "dummy-token";
const HEADERS = { Authorization: `Bearer ${TOKEN}`, "Content-Type": "application/json" };
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
const GENERATE_CATEGORY_PRODUCT_REWRITES =
  (process.env.GENERATE_CATEGORY_PRODUCT_REWRITES || "false").toLowerCase() === "true";
const PRODUCT_IDS = (process.env.PRODUCT_IDS || "")
  .split(",")
  .map((p) => p.trim())
  .filter(Boolean)
  .map(Number);

export function findDuplicateProductRewrites(rows, productId, generateCategoryRewritesEnabled) {
  if (generateCategoryRewritesEnabled) return [];

  const productRows = rows.filter(
    (r) => r.entity_type === "product" && r.entity_id === productId
  );

  const byStore = new Map();
  for (const row of productRows) {
    if (!byStore.has(row.store_id)) byStore.set(row.store_id, []);
    byStore.get(row.store_id).push(row);
  }

  const pairs = [];
  for (const [, storeRows] of byStore) {
    const byTarget = new Map();
    for (const row of storeRows) {
      if (!byTarget.has(row.target_path)) byTarget.set(row.target_path, []);
      byTarget.get(row.target_path).push(row);
    }
    for (const [, targetRows] of byTarget) {
      if (targetRows.length <= 1) continue;
      const segments = (path) => (path.match(/\//g) || []).length;
      const ordered = [...targetRows].sort((a, b) => segments(a.request_path) - segments(b.request_path));
      const keepRow = ordered[0];
      for (const removeRow of ordered.slice(1)) {
        if (segments(removeRow.request_path) > segments(keepRow.request_path)) {
          pairs.push({
            keep: keepRow.url_rewrite_id,
            remove: removeRow.url_rewrite_id,
            reason: "duplicate-despite-disabled-setting",
          });
        }
      }
    }
  }
  return pairs;
}

function isLiveTrafficRow(row, currentUrlKey) {
  if (!currentUrlKey) return false;
  return row.request_path.replace(/\/+$/, "") === `${currentUrlKey}.html`;
}

async function apiGet(path, params = {}) {
  const url = new URL(`${MAGENTO_URL}/rest/V1${path}`);
  for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
  const res = await fetch(url, { headers: HEADERS });
  if (!res.ok) throw new Error(`Magento ${res.status}`);
  return res.json();
}

async function apiDelete(path) {
  const res = await fetch(`${MAGENTO_URL}/rest/V1${path}`, { method: "DELETE", headers: HEADERS });
  if (!res.ok) throw new Error(`Magento ${res.status}`);
  return res.json();
}

async function resolveSkuByEntityId(productId) {
  const params = {
    "searchCriteria[filterGroups][0][filters][0][field]": "entity_id",
    "searchCriteria[filterGroups][0][filters][0][value]": productId,
    "searchCriteria[filterGroups][0][filters][0][conditionType]": "eq",
  };
  const result = await apiGet("/products", params);
  const items = result.items || [];
  return items.length ? items[0] : null;
}

function customAttr(attrs, code, fallback = null) {
  for (const a of attrs || []) {
    if (a.attribute_code === code) return a.value;
  }
  return fallback;
}

async function readUrlRewriteRowsForProduct(productId) {
  // Read this product's url_rewrite rows from a read-only export (CSV
  // columns: url_rewrite_id, entity_type, entity_id, request_path,
  // target_path, store_id, is_autogenerated). url_rewrite has no public
  // REST search endpoint.
  const path = process.env.URL_REWRITE_EXPORT_CSV || "url_rewrite_export.csv";
  const text = await readFile(path, "utf-8");
  const [headerLine, ...lines] = text.trim().split("\n");
  const headers = headerLine.split(",");
  return lines
    .filter(Boolean)
    .map((line) => {
      const cells = line.split(",");
      const row = {};
      headers.forEach((h, i) => (row[h] = cells[i]));
      return row;
    })
    .filter((row) => row.entity_type === "product" && Number(row.entity_id) === productId)
    .map((row) => ({
      url_rewrite_id: Number(row.url_rewrite_id),
      entity_type: row.entity_type,
      entity_id: Number(row.entity_id),
      request_path: row.request_path,
      target_path: row.target_path,
      store_id: Number(row.store_id),
      is_autogenerated: Number(row.is_autogenerated || 0),
    }));
}

export async function run() {
  const applyDeletes = process.argv.includes("--apply");
  let totalPairs = 0;

  for (const productId of PRODUCT_IDS) {
    const product = await resolveSkuByEntityId(productId);
    if (!product) {
      console.log(`Product id=${productId} not found, skipping`);
      continue;
    }
    const sku = product.sku;
    const currentUrlKey = customAttr(product.custom_attributes, "url_key", "");

    const rows = await readUrlRewriteRowsForProduct(productId);
    const pairs = findDuplicateProductRewrites(rows, productId, GENERATE_CATEGORY_PRODUCT_REWRITES);

    for (const pair of pairs) {
      totalPairs++;
      console.warn(
        `Product id=${productId} sku=${sku}: url_rewrite_id=${pair.remove} is redundant next to keep=${pair.keep} (${pair.reason})`
      );
      const removeRow = rows.find((r) => r.url_rewrite_id === pair.remove);
      if (isLiveTrafficRow(removeRow, currentUrlKey)) {
        console.log("Skipping delete: row matches the product's current url_key, serving live traffic");
        continue;
      }
      if (!applyDeletes || DRY_RUN) {
        console.log(
          `Recommend: apply the core patch for issue 38317/39070, or add an after-plugin on ` +
          `CanonicalUrlRewriteGenerator::generate to drop this row. Would delete url_rewrite_id=${pair.remove} ` +
          `(pass --apply and DRY_RUN=false to delete)`
        );
        continue;
      }
      await apiDelete(`/url-rewrites/${pair.remove}`);
      console.log(`Deleted redundant url_rewrite_id=${pair.remove}`);
    }
  }

  console.log(`Done. ${totalPairs} duplicate pair(s) found across ${PRODUCT_IDS.length} product(s).`);
}

if (import.meta.url === pathToFileURL(process.argv[1]).href) {
  run().catch((err) => { console.error(err); process.exit(1); });
}

Add a test

The pairing rule is the part most worth testing, because it decides which row a human is told to delete on a live storefront. Because find_duplicate_product_rewrites is pure, the test needs no network and no Magento store. It just feeds in fixture arrays and checks the answer.

test_disabled_rewrite_duplicates.py

from disabled_rewrite_setting_still_duplicates import find_duplicate_product_rewrites


def row(**over):
    base = {
        "url_rewrite_id": 1,
        "entity_type": "product",
        "entity_id": 42,
        "request_path": "green-shirt.html",
        "target_path": "catalog/product/view/id/42",
        "store_id": 1,
        "is_autogenerated": 1,
    }
    base.update(over)
    return base


def test_no_pairs_when_setting_enabled():
    rows = [
        row(url_rewrite_id=1, request_path="green-shirt.html"),
        row(url_rewrite_id=2, request_path="mens/shirts/green-shirt.html"),
    ]
    assert find_duplicate_product_rewrites(rows, 42, True) == []


def test_no_pairs_when_only_one_row_per_store():
    rows = [row(url_rewrite_id=1, request_path="green-shirt.html")]
    assert find_duplicate_product_rewrites(rows, 42, False) == []


def test_flags_pair_when_setting_disabled_and_two_rows_share_target():
    rows = [
        row(url_rewrite_id=1, request_path="green-shirt.html"),
        row(url_rewrite_id=2, request_path="mens/shirts/green-shirt.html"),
    ]
    result = find_duplicate_product_rewrites(rows, 42, False)
    assert result == [
        {"keep": 1, "remove": 2, "reason": "duplicate-despite-disabled-setting"}
    ]


def test_ignores_rows_for_a_different_product():
    rows = [
        row(url_rewrite_id=1, entity_id=42, request_path="green-shirt.html"),
        row(url_rewrite_id=2, entity_id=99, request_path="blue-shirt.html"),
    ]
    assert find_duplicate_product_rewrites(rows, 42, False) == []


def test_ignores_rows_for_a_different_entity_type():
    rows = [
        row(url_rewrite_id=1, entity_type="product", request_path="green-shirt.html"),
        row(url_rewrite_id=2, entity_type="category", request_path="mens/shirts.html", target_path="catalog/category/view/id/7"),
    ]
    assert find_duplicate_product_rewrites(rows, 42, False) == []


def test_same_target_different_store_is_not_paired():
    rows = [
        row(url_rewrite_id=1, request_path="green-shirt.html", store_id=1),
        row(url_rewrite_id=2, request_path="mens/shirts/green-shirt.html", store_id=2),
    ]
    assert find_duplicate_product_rewrites(rows, 42, False) == []


def test_different_target_path_is_not_paired():
    rows = [
        row(url_rewrite_id=1, request_path="green-shirt.html", target_path="catalog/product/view/id/42"),
        row(url_rewrite_id=2, request_path="mens/shirts/red-shirt.html", target_path="catalog/product/view/id/99"),
    ]
    assert find_duplicate_product_rewrites(rows, 42, False) == []

duplicates.test.js

import { test } from "node:test";
import assert from "node:assert/strict";
import { findDuplicateProductRewrites } from "./disabled-rewrite-setting-still-duplicates.js";

const row = (over = {}) => ({
  url_rewrite_id: 1,
  entity_type: "product",
  entity_id: 42,
  request_path: "green-shirt.html",
  target_path: "catalog/product/view/id/42",
  store_id: 1,
  is_autogenerated: 1,
  ...over,
});

test("no pairs when setting enabled", () => {
  const rows = [
    row({ url_rewrite_id: 1, request_path: "green-shirt.html" }),
    row({ url_rewrite_id: 2, request_path: "mens/shirts/green-shirt.html" }),
  ];
  assert.deepEqual(findDuplicateProductRewrites(rows, 42, true), []);
});

test("no pairs when only one row per store", () => {
  const rows = [row({ url_rewrite_id: 1, request_path: "green-shirt.html" })];
  assert.deepEqual(findDuplicateProductRewrites(rows, 42, false), []);
});

test("flags pair when setting disabled and two rows share target", () => {
  const rows = [
    row({ url_rewrite_id: 1, request_path: "green-shirt.html" }),
    row({ url_rewrite_id: 2, request_path: "mens/shirts/green-shirt.html" }),
  ];
  const result = findDuplicateProductRewrites(rows, 42, false);
  assert.deepEqual(result, [{ keep: 1, remove: 2, reason: "duplicate-despite-disabled-setting" }]);
});

test("ignores rows for a different product", () => {
  const rows = [
    row({ url_rewrite_id: 1, entity_id: 42, request_path: "green-shirt.html" }),
    row({ url_rewrite_id: 2, entity_id: 99, request_path: "blue-shirt.html" }),
  ];
  assert.deepEqual(findDuplicateProductRewrites(rows, 42, false), []);
});

test("ignores rows for a different entity type", () => {
  const rows = [
    row({ url_rewrite_id: 1, entity_type: "product", request_path: "green-shirt.html" }),
    row({
      url_rewrite_id: 2,
      entity_type: "category",
      request_path: "mens/shirts.html",
      target_path: "catalog/category/view/id/7",
    }),
  ];
  assert.deepEqual(findDuplicateProductRewrites(rows, 42, false), []);
});

test("same target different store is not paired", () => {
  const rows = [
    row({ url_rewrite_id: 1, request_path: "green-shirt.html", store_id: 1 }),
    row({ url_rewrite_id: 2, request_path: "mens/shirts/green-shirt.html", store_id: 2 }),
  ];
  assert.deepEqual(findDuplicateProductRewrites(rows, 42, false), []);
});

test("different target path is not paired", () => {
  const rows = [
    row({ url_rewrite_id: 1, request_path: "green-shirt.html", target_path: "catalog/product/view/id/42" }),
    row({ url_rewrite_id: 2, request_path: "mens/shirts/red-shirt.html", target_path: "catalog/product/view/id/99" }),
  ];
  assert.deepEqual(findDuplicateProductRewrites(rows, 42, false), []);
});

Case studies

Multi-store catalog

The setting was off in every scope and duplicates still appeared

A multi-store Magento 2.4.7-p1 catalog turned Generate category/product URL Rewrites to No at the default scope and confirmed it inherited down to every store view. Reindexing did not help. Products in more than one category still had two live rewrites each, one plain and one category-prefixed, matching the exact pattern in magento/magento2 issue 38317.

Running the detector against a rewrite export for a sample of product ids confirmed the pattern store by store: the setting really was off everywhere, and the pairs were still there. That confirmation was what got the team to stop hunting for a config mistake and instead track the patch status for the core issue.

SEO audit

A crawl turned up two indexable URLs for the same product

An SEO audit flagged that a chunk of products had two indexable URLs competing for the same content, which search engines were treating as near duplicate pages. The store's setting had been off for months, so the team assumed it was a leftover from before the setting existed.

The detector matched each pair back to a live product id and confirmed both rows were still being generated fresh on every save, not leftover data. That let the team scope the fix correctly: add the after-plugin on CanonicalUrlRewriteGenerator::generate rather than trying to clean the table once and hope it stayed clean.

What good looks like

After running this against your product catalog, you get a clear list of which products have a redundant category-path rewrite despite the setting being off, with the exact url_rewrite_id to keep and the one that is extra. Nothing gets deleted until a human passes an explicit apply flag, and the row serving live traffic is never touched. From there the real fix is either the core patch for issue 38317 and issue 39070, or a small plugin that stops the duplicate from being written in the first place.

FAQ

Why does a product still get a category URL rewrite after I set Generate category/product URL Rewrites to No?

That setting only gates the CanonicalUrlRewriteGenerator path that appends a category segment to a product's canonical URL when a category is saved. It does not gate the separate ProductProcessUrlRewriteSavingObserver that writes a plain product url_rewrite row on every product save. Both generators run from separate observers on separate save events, and the config check is applied inconsistently between them, so a product assigned to a category ends up with both a plain rewrite and a category rewrite even with the setting off. This is a confirmed core bug, not a misconfiguration.

Can a script safely delete the extra url_rewrite row with the setting disabled?

Not automatically. Deleting the wrong row breaks either the plain URL or the category URL for live traffic, and Magento regenerates the extra row on the next product or category save while the core bug is unpatched. The safe approach is to detect and report the duplicate pairs, and only perform a guarded delete of the non-canonical row when a human explicitly passes an apply flag, never touching the row currently serving live traffic.

Is this a known Magento bug or a store configuration mistake?

It is a confirmed core bug, tracked as magento/magento2 issue 38317 and issue 39070, reproduced on Magento 2.4.7-beta2 through 2.4.7-p2. The fix is either applying the corrective core patch or adding a small plugin or observer that removes the redundant category-path rewrite after the generators run. It is not something a store admin misconfigured.

Related field notes

Citations

On the problem:

magento/magento2: Set Generate "category/product" URL Rewrites to "no" has no effect. github.com/magento/magento2/issues/38317
magento/magento2: Category Rewrites are generating even if config is disabled. github.com/magento/magento2/issues/39070
magento/magento2: Category URL rewrites not generated after changes to seo config settings. github.com/magento/magento2/issues/17585

On the solution:

Adobe Commerce/Magento 2 Admin User Guide: URL Rewrites. experienceleague.adobe.com url-rewrite
Adobe Commerce/Magento 2 docs: Search Engine Optimization configuration (catalog/seo). experienceleague.adobe.com catalog config
Adobe Commerce/Magento 2 REST API reference: Products endpoint. developer.adobe.com quick-reference

Stuck on a tricky one?

If you have a problem in Magento catalog data, URL rewrites, cron, or MSI stock that you would rather hand off, this is the kind of work I do. Message me and we can work through it together.

Contact me on LinkedIn

Did this clear up your rewrite confusion?

If this saved you from chasing a config setting that was never the real cause, you can buy me a coffee. It is the best way to keep these field notes free and growing.

Buy me a coffee on Ko-fi

Back to all Magento field notes