Diagnostic Indexing
Products vanish mid reindex on delete then insert cycle
A support ticket says a product was on the site an hour ago and is gone now, but nobody touched it in the admin. The product is still enabled, still in stock, still sitting in the catalog. What changed is the price index. Magento's price indexer rebuilds its table by deleting a batch of rows and then inserting them again, and for a moment in between, that batch of products simply is not there. Here is why that gap exists and a small script that tells the harmless version of this apart from a real disappearance.
The catalog_product_price indexer rebuilds catalog_product_index_price by paging through eligible products and issuing a delete then insert for each batch, either against the live table when the indexer runs on save, or against catalog_product_index_price_replica in Update by Schedule mode, which is later swapped in. With small batch sizes and a catalog carrying many disabled or out of stock SKUs, a product's row can be deleted in one batch and not reinserted until a later one, so a category page, layered navigation, or a search hitting the price index during that gap sees the product as absent even though it is still in the catalog. Because this is CLI and DB territory, not a writable REST resource, a script cannot detect it by reading the index table directly. Instead it polls GET /rest/V1/products for the enabled and visible SKU set before, during, and after a known reindex window, checks GET /rest/V1/indexer for catalog_product_price status, and reports which SKUs vanished and whether they came back. Full code, tests, and a dry run guard are below.
The problem in plain words
Magento does not compute a product's storefront price on the fly for every listing page. It keeps a flat table, catalog_product_index_price, that already holds the final price for every product across every customer group and website, so category pages and layered navigation can read one row instead of running the whole EAV and price rule pipeline each time. That table has to be rebuilt whenever prices, rules, or catalog data change, and that rebuild is the reindex.
The rebuild does not rewrite the whole table in one transaction. It pages through eligible products in batches, and for each batch it deletes the existing rows for those product ids and inserts the freshly computed ones, sometimes as a delete followed by an insert, sometimes as an insert or replace after a delete. If the indexer runs in the legacy on-save mode, or a forced full reindex runs against the live table on a split database or a high traffic setup, that delete and insert happens directly against the table storefront queries are reading from right now. For the short window between the delete and the matching insert, any query touching that batch of products, a category listing, layered nav, or a search or collection built on the price index, simply does not find them. The product has not been removed from the catalog. It has been removed from the index, for a moment, by the rebuild itself.
Why it happens
- The
catalog_product_priceindexer, whether it runs on save or is invoked byindexer:reindex, paginates eligible products and processes them in batches, issuing a delete for the batch's rows before writing the fresh ones. - A catalog with many disabled or out of stock SKUs means the indexer has to page through and skip more rows to fill each batch, which stretches the delete-to-insert gap and makes it more likely a request lands inside it.
- In on-save or legacy mode, that delete and insert happen against the live
catalog_product_index_pricetable, the same one storefront category, layered nav, and some search or collection queries read from right now. - In Update by Schedule mode the build writes to
catalog_product_index_price_replicaand only swaps table names when the whole build finishes, which is safer, but any code path that reads the live table mid swap, or a forced full reindex run synchronously against a live table during peak traffic, can still hit an equivalent gap. - Smaller batch sizes mean more delete and insert cycles for the same catalog, which is exactly the setting most likely to be turned down on constrained hosting, making the problem worse where it is least expected.
This is a well documented pattern rather than a one-off bug: it is filed against Magento 2 itself as products intermittently missing during a price reindex, and store operators have written up the same symptom while working on reindex performance. See the citations at the end for the exact reports.
You cannot fix this from the REST API, because the root cause lives in the CLI-only reindex and cron pipeline, not in any endpoint you can call or write to. What you can do over REST is catch it in the act: pull the enabled and visible SKU set before a reindex window, again during it, and again after, and cross reference against the indexer's own reported status. If the SKUs that vanished all come back once the indexer finishes, that is the batching race working exactly as documented, self healing, and not something to chase as a bug. If they do not come back, that is a different problem entirely, a real removal or disabling, and it needs a human to look at the product record before anyone touches it.
The fix, as a flow
The script never reindexes or touches the database. It watches the catalog from the outside, across a reindex window, and sorts what it sees into two very different buckets: an expected timing gap that will resolve on its own, or a real loss that needs a person.
Build it step by step
Get an admin bearer token
Call the token endpoint once with an admin user, or use a long lived integration token. Keep the base URL and the token in environment variables, never in the file, and leave DRY_RUN on so nothing is ever written by mistake, since this script does not repair anything by itself anyway.
pip install requests
export MAGENTO_URL="https://your-store.example.com"
export MAGENTO_ADMIN_TOKEN="your admin bearer token"
export DRY_RUN="true" # this script only reports; kept for consistency// Node 18+ has fetch built in, no dependencies needed
export MAGENTO_URL="https://your-store.example.com"
export MAGENTO_ADMIN_TOKEN="your admin bearer token"
export DRY_RUN="true" // this script only reports; kept for consistencyList enabled and visible SKUs
Call GET /rest/V1/products filtered to status equal to 1 and visibility matching one of catalog, search, or both, paging with searchCriteria[pageSize] and [currentPage] until every SKU is collected. This is the EAV source of truth for what should be visible, independent of the price index.
import os, requests
MAGENTO_URL = os.environ["MAGENTO_URL"].rstrip("/")
TOKEN = os.environ["MAGENTO_ADMIN_TOKEN"]
PAGE_SIZE = int(os.environ.get("PAGE_SIZE", "100"))
def enabled_visible_skus():
"""Full pageSize iteration over enabled, visible products. Returns a list of SKUs."""
skus = []
page = 1
while True:
params = {
"searchCriteria[filterGroups][0][filters][0][field]": "status",
"searchCriteria[filterGroups][0][filters][0][value]": "1",
"searchCriteria[filterGroups][0][filters][0][conditionType]": "eq",
"searchCriteria[filterGroups][1][filters][0][field]": "visibility",
"searchCriteria[filterGroups][1][filters][0][value]": "2,3,4",
"searchCriteria[filterGroups][1][filters][0][conditionType]": "in",
"searchCriteria[pageSize]": PAGE_SIZE,
"searchCriteria[currentPage]": page,
}
r = requests.get(
f"{MAGENTO_URL}/rest/V1/products",
params=params,
headers={"Authorization": f"Bearer {TOKEN}"},
timeout=30,
)
r.raise_for_status()
body = r.json()
items = body.get("items", [])
skus.extend(item["sku"] for item in items)
if len(items) < PAGE_SIZE:
return skus
page += 1const MAGENTO_URL = (process.env.MAGENTO_URL || "").replace(/\/$/, "");
const TOKEN = process.env.MAGENTO_ADMIN_TOKEN;
const PAGE_SIZE = Number(process.env.PAGE_SIZE || 100);
async function enabledVisibleSkus() {
// Full pageSize iteration over enabled, visible products. Returns an array of SKUs.
const skus = [];
let page = 1;
while (true) {
const params = new URLSearchParams({
"searchCriteria[filterGroups][0][filters][0][field]": "status",
"searchCriteria[filterGroups][0][filters][0][value]": "1",
"searchCriteria[filterGroups][0][filters][0][conditionType]": "eq",
"searchCriteria[filterGroups][1][filters][0][field]": "visibility",
"searchCriteria[filterGroups][1][filters][0][value]": "2,3,4",
"searchCriteria[filterGroups][1][filters][0][conditionType]": "in",
"searchCriteria[pageSize]": String(PAGE_SIZE),
"searchCriteria[currentPage]": String(page),
});
const res = await fetch(`${MAGENTO_URL}/rest/V1/products?${params}`, {
headers: { Authorization: `Bearer ${TOKEN}` },
});
if (!res.ok) throw new Error(`Magento ${res.status}`);
const body = await res.json();
const items = body.items || [];
for (const item of items) skus.push(item.sku);
if (items.length < PAGE_SIZE) return skus;
page++;
}
}Read the indexer status
Call GET /rest/V1/indexer and pick out the entry whose indexer_id is catalog_product_price. Its status field reads valid, invalid, or processing, and that timestamped status is what lets the decision function tell a race apart from a real problem.
def price_indexer_status():
r = requests.get(
f"{MAGENTO_URL}/rest/V1/indexer",
headers={"Authorization": f"Bearer {TOKEN}"},
timeout=30,
)
r.raise_for_status()
for row in r.json():
if row.get("indexer_id") == "catalog_product_price":
return {"code": row["indexer_id"], "status": row.get("status", "")}
return {"code": "catalog_product_price", "status": "unknown"}async function priceIndexerStatus() {
const res = await fetch(`${MAGENTO_URL}/rest/V1/indexer`, {
headers: { Authorization: `Bearer ${TOKEN}` },
});
if (!res.ok) throw new Error(`Magento ${res.status}`);
const rows = await res.json();
const row = rows.find((r) => r.indexer_id === "catalog_product_price");
return { code: "catalog_product_price", status: row ? row.status || "" : "unknown" };
}Decide, with one pure function
Compute what went missing between the before and during snapshots, and what is still missing once the after snapshot is in. If the gap closed and the indexer was actively reindexing, that is the expected batching race. If SKUs never came back, that is a real loss and needs a human, not a script, to look at the product.
def decide_reindex_anomaly(before_skus, during_skus, after_skus, indexer_status):
before_set, during_set, after_set = set(before_skus), set(during_skus), set(after_skus)
missing = before_set - during_set
still_missing_after = missing - after_set
if not missing:
return {
"isTransientDropDetected": False,
"missingDuringWindow": [],
"falsePositive": len(before_skus) != len(after_skus),
"recommendation": "ok",
}
reindexing = indexer_status.get("code") == "catalog_product_price" and indexer_status.get("status") in ("processing", "invalid")
if not still_missing_after and reindexing:
return {
"isTransientDropDetected": True,
"missingDuringWindow": sorted(missing),
"falsePositive": False,
"recommendation": "flag_transient_index_gap",
}
if still_missing_after:
return {
"isTransientDropDetected": False,
"missingDuringWindow": sorted(missing),
"falsePositive": False,
"recommendation": "flag_permanent_loss",
}
return {
"isTransientDropDetected": False,
"missingDuringWindow": sorted(missing),
"falsePositive": True,
"recommendation": "ok",
}export function decideReindexAnomaly(beforeSkus, duringSkus, afterSkus, indexerStatus) {
const beforeSet = new Set(beforeSkus);
const duringSet = new Set(duringSkus);
const afterSet = new Set(afterSkus);
const missing = [...beforeSet].filter((sku) => !duringSet.has(sku));
const stillMissingAfter = missing.filter((sku) => !afterSet.has(sku));
if (missing.length === 0) {
return {
isTransientDropDetected: false,
missingDuringWindow: [],
falsePositive: beforeSkus.length !== afterSkus.length,
recommendation: "ok",
};
}
const reindexing =
indexerStatus.code === "catalog_product_price" &&
(indexerStatus.status === "processing" || indexerStatus.status === "invalid");
if (stillMissingAfter.length === 0 && reindexing) {
return {
isTransientDropDetected: true,
missingDuringWindow: missing.sort(),
falsePositive: false,
recommendation: "flag_transient_index_gap",
};
}
if (stillMissingAfter.length > 0) {
return {
isTransientDropDetected: false,
missingDuringWindow: missing.sort(),
falsePositive: false,
recommendation: "flag_permanent_loss",
};
}
return {
isTransientDropDetected: false,
missingDuringWindow: missing.sort(),
falsePositive: true,
recommendation: "ok",
};
}Poll a reindex window and report
Take a before snapshot, wait for the window to include a reindex, take a during snapshot and the indexer status, take an after snapshot, and hand all four to the decision function. Log the recommendation. The script never calls a mutating endpoint for a transient gap, and only ever suggests a human-confirmed PUT /rest/V1/products/{sku} when the loss is permanent.
This script is a detector, not a repair tool. It never writes to a product from a transient finding. If it reports flag_permanent_loss, treat that as a lead for a person to check the product's status and visibility by hand, then decide whether to restore it, not as an instruction to auto-enable anything.
The full code
Here is the complete script in one file for each language. It reads settings from the environment, logs what it finds, and never mutates a product from this detection path, only reporting the SKUs, the timestamps, and the indexer status so a human can act on a genuine loss.
"""Detect the transient product-vanishing gap in Magento's catalog_product_price reindex.
Reindex/cron and direct index-table access are CLI/DB-only, so this detects the symptom
over REST: it records the enabled and visible SKU set before, during, and after a known
reindex window, cross references indexer status, and reports whether a drop is the
expected self-healing batching race or a genuine, still-missing product. It never calls
a mutating endpoint for a transient gap. Safe to run again and again.
"""
import os
import time
import logging
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("reindex_anomaly")
MAGENTO_URL = os.environ["MAGENTO_URL"].rstrip("/")
TOKEN = os.environ["MAGENTO_ADMIN_TOKEN"]
PAGE_SIZE = int(os.environ.get("PAGE_SIZE", "100"))
POLL_INTERVAL_SECONDS = float(os.environ.get("POLL_INTERVAL_SECONDS", "5"))
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"
def enabled_visible_skus():
"""Full pageSize iteration over enabled, visible products. Returns a list of SKUs."""
skus = []
page = 1
while True:
params = {
"searchCriteria[filterGroups][0][filters][0][field]": "status",
"searchCriteria[filterGroups][0][filters][0][value]": "1",
"searchCriteria[filterGroups][0][filters][0][conditionType]": "eq",
"searchCriteria[filterGroups][1][filters][0][field]": "visibility",
"searchCriteria[filterGroups][1][filters][0][value]": "2,3,4",
"searchCriteria[filterGroups][1][filters][0][conditionType]": "in",
"searchCriteria[pageSize]": PAGE_SIZE,
"searchCriteria[currentPage]": page,
}
r = requests.get(
f"{MAGENTO_URL}/rest/V1/products",
params=params,
headers={"Authorization": f"Bearer {TOKEN}"},
timeout=30,
)
r.raise_for_status()
body = r.json()
items = body.get("items", [])
skus.extend(item["sku"] for item in items)
if len(items) < PAGE_SIZE:
return skus
page += 1
def price_indexer_status():
r = requests.get(
f"{MAGENTO_URL}/rest/V1/indexer",
headers={"Authorization": f"Bearer {TOKEN}"},
timeout=30,
)
r.raise_for_status()
for row in r.json():
if row.get("indexer_id") == "catalog_product_price":
return {"code": row["indexer_id"], "status": row.get("status", "")}
return {"code": "catalog_product_price", "status": "unknown"}
def decide_reindex_anomaly(before_skus, during_skus, after_skus, indexer_status):
before_set, during_set, after_set = set(before_skus), set(during_skus), set(after_skus)
missing = before_set - during_set
still_missing_after = missing - after_set
if not missing:
return {
"isTransientDropDetected": False,
"missingDuringWindow": [],
"falsePositive": len(before_skus) != len(after_skus),
"recommendation": "ok",
}
reindexing = indexer_status.get("code") == "catalog_product_price" and indexer_status.get("status") in ("processing", "invalid")
if not still_missing_after and reindexing:
return {
"isTransientDropDetected": True,
"missingDuringWindow": sorted(missing),
"falsePositive": False,
"recommendation": "flag_transient_index_gap",
}
if still_missing_after:
return {
"isTransientDropDetected": False,
"missingDuringWindow": sorted(missing),
"falsePositive": False,
"recommendation": "flag_permanent_loss",
}
return {
"isTransientDropDetected": False,
"missingDuringWindow": sorted(missing),
"falsePositive": True,
"recommendation": "ok",
}
def run():
log.info("Recording before snapshot.")
before_skus = enabled_visible_skus()
log.info("Waiting %ss to bracket the reindex window.", POLL_INTERVAL_SECONDS)
time.sleep(POLL_INTERVAL_SECONDS)
log.info("Recording during snapshot and indexer status.")
during_skus = enabled_visible_skus()
indexer_status = price_indexer_status()
log.info("Waiting %ss for the reindex to finish before the after snapshot.", POLL_INTERVAL_SECONDS)
time.sleep(POLL_INTERVAL_SECONDS)
log.info("Recording after snapshot.")
after_skus = enabled_visible_skus()
result = decide_reindex_anomaly(before_skus, during_skus, after_skus, indexer_status)
if result["recommendation"] == "ok":
log.info("No anomaly detected. %d before, %d after.", len(before_skus), len(after_skus))
elif result["recommendation"] == "flag_transient_index_gap":
log.warning(
"Transient index gap detected during catalog_product_price reindex. %d SKU(s) dipped and returned: %s",
len(result["missingDuringWindow"]), result["missingDuringWindow"],
)
log.warning("This is expected, self healing batching behavior. No write performed. DRY_RUN=%s", DRY_RUN)
elif result["recommendation"] == "flag_permanent_loss":
log.error(
"%d SKU(s) missing during the window and still missing after it finished: %s",
len(result["missingDuringWindow"]), result["missingDuringWindow"],
)
log.error("This looks like a real removal, not a reindex race. A human should confirm before any product is re-enabled. No write performed.")
return result
if __name__ == "__main__":
run()/**
* Detect the transient product-vanishing gap in Magento's catalog_product_price reindex.
*
* Reindex/cron and direct index-table access are CLI/DB-only, so this detects the
* symptom over REST: it records the enabled and visible SKU set before, during, and
* after a known reindex window, cross references indexer status, and reports whether
* a drop is the expected self-healing batching race or a genuine, still-missing product.
* It never calls a mutating endpoint for a transient gap. Safe to run again and again.
*
* Guide: https://www.allanninal.dev/magento/products-vanish-during-price-reindex/
*/
import { pathToFileURL } from "node:url";
const MAGENTO_URL = (process.env.MAGENTO_URL || "https://demo.example.com").replace(/\/$/, "");
const TOKEN = process.env.MAGENTO_ADMIN_TOKEN || "token_dummy";
const PAGE_SIZE = Number(process.env.PAGE_SIZE || 100);
const POLL_INTERVAL_SECONDS = Number(process.env.POLL_INTERVAL_SECONDS || 5);
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
export function decideReindexAnomaly(beforeSkus, duringSkus, afterSkus, indexerStatus) {
const beforeSet = new Set(beforeSkus);
const duringSet = new Set(duringSkus);
const afterSet = new Set(afterSkus);
const missing = [...beforeSet].filter((sku) => !duringSet.has(sku));
const stillMissingAfter = missing.filter((sku) => !afterSet.has(sku));
if (missing.length === 0) {
return {
isTransientDropDetected: false,
missingDuringWindow: [],
falsePositive: beforeSkus.length !== afterSkus.length,
recommendation: "ok",
};
}
const reindexing =
indexerStatus.code === "catalog_product_price" &&
(indexerStatus.status === "processing" || indexerStatus.status === "invalid");
if (stillMissingAfter.length === 0 && reindexing) {
return {
isTransientDropDetected: true,
missingDuringWindow: missing.sort(),
falsePositive: false,
recommendation: "flag_transient_index_gap",
};
}
if (stillMissingAfter.length > 0) {
return {
isTransientDropDetected: false,
missingDuringWindow: missing.sort(),
falsePositive: false,
recommendation: "flag_permanent_loss",
};
}
return {
isTransientDropDetected: false,
missingDuringWindow: missing.sort(),
falsePositive: true,
recommendation: "ok",
};
}
async function enabledVisibleSkus() {
const skus = [];
let page = 1;
while (true) {
const params = new URLSearchParams({
"searchCriteria[filterGroups][0][filters][0][field]": "status",
"searchCriteria[filterGroups][0][filters][0][value]": "1",
"searchCriteria[filterGroups][0][filters][0][conditionType]": "eq",
"searchCriteria[filterGroups][1][filters][0][field]": "visibility",
"searchCriteria[filterGroups][1][filters][0][value]": "2,3,4",
"searchCriteria[filterGroups][1][filters][0][conditionType]": "in",
"searchCriteria[pageSize]": String(PAGE_SIZE),
"searchCriteria[currentPage]": String(page),
});
const res = await fetch(`${MAGENTO_URL}/rest/V1/products?${params}`, {
headers: { Authorization: `Bearer ${TOKEN}` },
});
if (!res.ok) throw new Error(`Magento ${res.status}`);
const body = await res.json();
const items = body.items || [];
for (const item of items) skus.push(item.sku);
if (items.length < PAGE_SIZE) return skus;
page++;
}
}
async function priceIndexerStatus() {
const res = await fetch(`${MAGENTO_URL}/rest/V1/indexer`, {
headers: { Authorization: `Bearer ${TOKEN}` },
});
if (!res.ok) throw new Error(`Magento ${res.status}`);
const rows = await res.json();
const row = rows.find((r) => r.indexer_id === "catalog_product_price");
return { code: "catalog_product_price", status: row ? row.status || "" : "unknown" };
}
function sleep(seconds) {
return new Promise((resolve) => setTimeout(resolve, seconds * 1000));
}
export async function run() {
console.log("Recording before snapshot.");
const beforeSkus = await enabledVisibleSkus();
console.log(`Waiting ${POLL_INTERVAL_SECONDS}s to bracket the reindex window.`);
await sleep(POLL_INTERVAL_SECONDS);
console.log("Recording during snapshot and indexer status.");
const duringSkus = await enabledVisibleSkus();
const indexerStatus = await priceIndexerStatus();
console.log(`Waiting ${POLL_INTERVAL_SECONDS}s for the reindex to finish before the after snapshot.`);
await sleep(POLL_INTERVAL_SECONDS);
console.log("Recording after snapshot.");
const afterSkus = await enabledVisibleSkus();
const result = decideReindexAnomaly(beforeSkus, duringSkus, afterSkus, indexerStatus);
if (result.recommendation === "ok") {
console.log(`No anomaly detected. ${beforeSkus.length} before, ${afterSkus.length} after.`);
} else if (result.recommendation === "flag_transient_index_gap") {
console.warn(
`Transient index gap detected during catalog_product_price reindex. ${result.missingDuringWindow.length} SKU(s) dipped and returned: ${result.missingDuringWindow.join(", ")}`
);
console.warn(`This is expected, self healing batching behavior. No write performed. DRY_RUN=${DRY_RUN}`);
} else if (result.recommendation === "flag_permanent_loss") {
console.error(
`${result.missingDuringWindow.length} SKU(s) missing during the window and still missing after it finished: ${result.missingDuringWindow.join(", ")}`
);
console.error("This looks like a real removal, not a reindex race. A human should confirm before any product is re-enabled. No write performed.");
}
return result;
}
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((err) => { console.error(err); process.exit(1); });
}Add a test
The decision rule is the part most worth testing, because it is what tells an ops team whether to shrug or to escalate. Because decide_reindex_anomaly is pure, taking plain arrays and a status object, the test needs no Magento store and no network.
from reindex_anomaly import decide_reindex_anomaly
PROCESSING = {"code": "catalog_product_price", "status": "processing"}
INVALID = {"code": "catalog_product_price", "status": "invalid"}
VALID = {"code": "catalog_product_price", "status": "valid"}
def test_transient_gap_when_missing_returns_and_indexer_processing():
before = ["sku-1", "sku-2", "sku-3"]
during = ["sku-1", "sku-3"]
after = ["sku-1", "sku-2", "sku-3"]
result = decide_reindex_anomaly(before, during, after, PROCESSING)
assert result["isTransientDropDetected"] is True
assert result["missingDuringWindow"] == ["sku-2"]
assert result["recommendation"] == "flag_transient_index_gap"
def test_permanent_loss_when_sku_never_returns():
before = ["sku-1", "sku-2"]
during = ["sku-1"]
after = ["sku-1"]
result = decide_reindex_anomaly(before, during, after, PROCESSING)
assert result["isTransientDropDetected"] is False
assert result["missingDuringWindow"] == ["sku-2"]
assert result["recommendation"] == "flag_permanent_loss"
def test_permanent_loss_even_if_indexer_says_valid():
before = ["sku-1", "sku-2"]
during = ["sku-1"]
after = ["sku-1"]
result = decide_reindex_anomaly(before, during, after, VALID)
assert result["recommendation"] == "flag_permanent_loss"
def test_ok_when_nothing_missing_and_counts_match():
before = ["sku-1", "sku-2"]
result = decide_reindex_anomaly(before, before, before, VALID)
assert result["recommendation"] == "ok"
assert result["falsePositive"] is False
def test_false_positive_when_nothing_missing_but_counts_differ():
before = ["sku-1", "sku-2"]
after = ["sku-1", "sku-2", "sku-3"]
result = decide_reindex_anomaly(before, before, after, VALID)
assert result["recommendation"] == "ok"
assert result["falsePositive"] is True
def test_transient_gap_detected_when_indexer_status_invalid():
before = ["sku-1", "sku-2"]
during = ["sku-1"]
after = ["sku-1", "sku-2"]
result = decide_reindex_anomaly(before, during, after, INVALID)
assert result["recommendation"] == "flag_transient_index_gap"import { test } from "node:test";
import assert from "node:assert/strict";
import { decideReindexAnomaly } from "./reindex-anomaly.js";
const PROCESSING = { code: "catalog_product_price", status: "processing" };
const INVALID = { code: "catalog_product_price", status: "invalid" };
const VALID = { code: "catalog_product_price", status: "valid" };
test("transient gap when missing returns and indexer processing", () => {
const before = ["sku-1", "sku-2", "sku-3"];
const during = ["sku-1", "sku-3"];
const after = ["sku-1", "sku-2", "sku-3"];
const result = decideReindexAnomaly(before, during, after, PROCESSING);
assert.equal(result.isTransientDropDetected, true);
assert.deepEqual(result.missingDuringWindow, ["sku-2"]);
assert.equal(result.recommendation, "flag_transient_index_gap");
});
test("permanent loss when sku never returns", () => {
const before = ["sku-1", "sku-2"];
const during = ["sku-1"];
const after = ["sku-1"];
const result = decideReindexAnomaly(before, during, after, PROCESSING);
assert.equal(result.isTransientDropDetected, false);
assert.deepEqual(result.missingDuringWindow, ["sku-2"]);
assert.equal(result.recommendation, "flag_permanent_loss");
});
test("permanent loss even if indexer says valid", () => {
const before = ["sku-1", "sku-2"];
const during = ["sku-1"];
const after = ["sku-1"];
const result = decideReindexAnomaly(before, during, after, VALID);
assert.equal(result.recommendation, "flag_permanent_loss");
});
test("ok when nothing missing and counts match", () => {
const before = ["sku-1", "sku-2"];
const result = decideReindexAnomaly(before, before, before, VALID);
assert.equal(result.recommendation, "ok");
assert.equal(result.falsePositive, false);
});
test("false positive when nothing missing but counts differ", () => {
const before = ["sku-1", "sku-2"];
const after = ["sku-1", "sku-2", "sku-3"];
const result = decideReindexAnomaly(before, before, after, VALID);
assert.equal(result.recommendation, "ok");
assert.equal(result.falsePositive, true);
});
test("transient gap detected when indexer status invalid", () => {
const before = ["sku-1", "sku-2"];
const during = ["sku-1"];
const after = ["sku-1", "sku-2"];
const result = decideReindexAnomaly(before, during, after, INVALID);
assert.equal(result.recommendation, "flag_transient_index_gap");
});Case studies
The furniture importer with thousands of discontinued SKUs
A furniture importer kept years of discontinued SKUs disabled rather than deleted, for historical order references. Every price rule edit triggered an on-save reindex that had to page past thousands of disabled rows to reach the active ones, stretching every batch's delete-to-insert gap, and support kept getting tickets that a product had vanished for no reason.
Running the detector across a rule edit showed the same handful of SKUs dipping and returning within seconds, every time, with the indexer status reading processing throughout. That confirmed it was the batching race, not a data problem, and the team switched the price indexer to Update by Schedule so shoppers stopped seeing it at all.
The store that tuned batch size down for a slow host
A store on constrained hosting had turned the indexer batch size down to ease database load during reindex. That fixed the load problem but multiplied the number of delete and insert cycles for the same catalog, and a few products started disappearing from category pages right after every price update, always for a few seconds.
The team ran the detector during a scheduled price update and got flag_transient_index_gap every time, with the same SKUs returning within the after snapshot. Knowing it was self healing, they raised the batch size back up on a beefier reindex window instead of chasing a phantom bug, and the dips got shorter.
After this runs across a reindex window, a support ticket about a vanishing product stops being a mystery. Either the SKU dipped and returned while the indexer was processing, which is the expected batching race and needs no code change, or it is still missing afterward, which is flagged clearly for a human to check the product's real status before anyone touches it. No product gets auto re-enabled from a guess.
FAQ
Why do products disappear from my Magento store during a reindex?
The catalog_product_price indexer rebuilds catalog_product_index_price in batches, and each batch deletes the rows for the products in that batch before it inserts their fresh rows. When a batch is small and the catalog has many disabled or out of stock SKUs to skip over, a product can be deleted from the index and not reinserted until a later batch. Any storefront query that reads the price index, such as a category listing or layered navigation, can hit that gap and show the product as gone even though it still exists in the catalog data.
Does switching to Update by Schedule fix the vanishing products problem?
It removes the worst version of the problem. In Update by Schedule mode Magento builds the new index into catalog_product_index_price_replica and only swaps the table names in when the build finishes, so storefront queries keep reading the old, complete table for the whole rebuild. The same delete then insert batching still happens inside the replica build, but shoppers never see it because they are not reading the replica table while it is being written.
How can I tell a transient reindex gap apart from a product that is actually gone?
Record the set of enabled and visible SKUs before a reindex window, again during it, and again after it finishes, using GET /rest/V1/products with pageSize iteration so you get exact SKU lists rather than just counts. If SKUs that were missing during the window are back afterward, and the indexer status for catalog_product_price was processing or invalid at the time, that is the expected, self healing race and needs no fix. If SKUs are still missing after the window closes, that is a real data problem, most likely a disabled or hidden product, and needs a human to confirm before anyone changes its status.
Related field notes
Citations
On the problem:
- Magento 2 GitHub Issues: catalog_product_index_price reindexing occasionally resulting in missing products on site. github.com/magento/magento2/issues/35616
- How we reduced the Magento 2 price reindex process from hours to minutes, including the products and prices disappearing on frontend symptom. royandre.medium.com
- Magento Forums: Products not showing and re-indexing. community.magento.com products not showing and re-indexing
On the solution:
- Adobe Commerce: Manage the indexers, including Update by Schedule and indexer:set-mode. experienceleague.adobe.com manage-indexers
- Adobe Commerce Admin: Index management. experienceleague.adobe.com index-management
- Adobe Commerce developer docs: Indexing, Commerce PHP Extensions. developer.adobe.com commerce indexing
Stuck on a tricky one?
If you have a problem in Magento or Adobe Commerce indexing, catalog, orders, or inventory that you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this save you a false alarm?
If this helped you tell a harmless reindex race apart from a real problem, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi