Diagnostic Indexing
Products flap out of category or search during scheduled indexing
A merchandiser swears a product was in the category five minutes ago. A shopper searches for a SKU that was findable this morning and gets zero results. Nobody touched the product. What actually happened is a scheduled reindex cycle briefly deleted the search or category index entries for that product before rebuilding them, and the storefront was queried in that gap. Here is why Magento's own scheduled indexing causes this, and a small script that detects and reports the flapping instead of guessing.
When an indexer is set to Update by Schedule, a cron job called indexer_update_all_views reads the changed entity IDs out of a changelog (*_cl) table and processes them in batches. For catalogsearch_fulltext, each batch first deletes the existing search documents for those product IDs and then recreates them, so between the delete and the recreate a storefront query can return zero or partial results for those products. Because catalogsearch_fulltext also depends on the price, category, and stock indexers, a product can flap out of a category listing too if a dependent indexer has not caught up yet. This is transient and self-healing, not corrupt data. Poll the category and search endpoints during a reindex window, diff the results against a known good baseline, and flag any SKU that goes missing and comes back within one or two cron cycles. Full code, tests, and a dry run guard are below.
The problem in plain words
Magento's scheduled indexing exists so a big catalog update does not lock the storefront while every index rebuilds inline. Instead, a database trigger writes the changed product ID into a changelog table the moment something relevant changes, and a cron job comes along roughly once a minute and processes whatever piled up since the last run.
For the full text search index, that processing is not an update in place. Magento deletes the old search documents for the changed IDs, then builds and inserts the new ones. That is fast, usually well under a second per batch, but it is not atomic from the storefront's point of view. If a shopper's search request lands in that small window, the product simply is not there yet. The same kind of gap can happen with category listings when a dependent indexer such as price or stock has not finished its own pass, so the category page temporarily disagrees with what is actually assigned.
Why it happens
- Update by Schedule mode tracks changed entity IDs in changelog (
*_cl) tables written by database triggers, and a cron job periodically reads and clears them through Mview. - For
catalogsearch_fulltext, both the MySQL and Elasticsearch or OpenSearch engines process a batch by deleting the existing search documents for the changed IDs and then re-creating and re-inserting them, which is not one atomic step from the storefront's perspective. catalogsearch_fulltextalso depends on the price, category and product, and stock indexers. Scheduled mode does not guarantee those finish in a fixed order relative to each other, so a product can flap out of a category listing (catalog_category_product) even when the search index itself is fine.- The default cron interval means
indexer_update_all_viewsruns roughly once a minute, so the missing window is usually short, but under a large catalog update or a slow batch it can stretch out. - None of this is visible through the REST API. The changelog tables and
mview_stateare database internals with no public endpoint, so the only way to see the effect is to compare what the storefront actually returns against a known good baseline.
A missing product during a reindex window is not necessarily a bug. It only becomes worth escalating if it stays missing across several cron cycles in a row, which points at a stuck or overloaded indexer rather than a normal delete then recreate cycle. So the useful script is not one that tries to fix the index, since there is nothing safe to write there. It is one that tells the difference between a product that flapped and self healed, and one that is stuck.
The fix, as a flow
We cannot write our way out of a transient indexing state, and we should not try. Instead the script polls the storefront-equivalent product, category, and search endpoints on a short interval, diffs each response against a baseline of enabled and visible SKUs, and classifies anything missing as either flapping, which is expected to self heal within a cycle or two, or stuck, which is worth telling a human about.
Build it step by step
Get an admin bearer token
Authenticate against POST /rest/V1/integration/admin/token with an admin username and password, or use a preconfigured integration token. Either way you end up with a bearer token you send as Authorization: Bearer <token> on every call. Keep the token and store URL in environment variables.
pip install requests
export MAGENTO_URL="https://your-store.example.com"
export MAGENTO_ADMIN_TOKEN="your admin bearer token"
export DRY_RUN="true" # start safe, report only// Node 18+ has fetch built in, no dependencies needed
export MAGENTO_URL="https://your-store.example.com"
export MAGENTO_ADMIN_TOKEN="your admin bearer token"
export DRY_RUN="true" // start safe, report onlyFetch the baseline of enabled, visible products
Ask /rest/V1/products for products with status equal to 1 using searchCriteria filters. This is the set of SKUs and entity IDs that should be findable on the storefront. Everything the script checks later is compared back against this list.
import os, requests
MAGENTO_URL = os.environ["MAGENTO_URL"].rstrip("/")
TOKEN = os.environ["MAGENTO_ADMIN_TOKEN"]
def get(path, params=None):
r = requests.get(
f"{MAGENTO_URL}/rest/V1/{path}",
params=params or {},
headers={"Authorization": f"Bearer {TOKEN}"},
timeout=30,
)
r.raise_for_status()
return r.json()
def baseline_skus():
params = {
"searchCriteria[filterGroups][0][filters][0][field]": "status",
"searchCriteria[filterGroups][0][filters][0][value]": "1",
"searchCriteria[filterGroups][0][filters][0][conditionType]": "eq",
"searchCriteria[pageSize]": "200",
}
data = get("products", params)
return {item["sku"] for item in data.get("items", [])}const MAGENTO_URL = (process.env.MAGENTO_URL || "").replace(/\/$/, "");
const TOKEN = process.env.MAGENTO_ADMIN_TOKEN;
async function get(path, params = {}) {
const url = new URL(`${MAGENTO_URL}/rest/V1/${path}`);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
const res = await fetch(url, { headers: { Authorization: `Bearer ${TOKEN}` } });
if (!res.ok) throw new Error(`Magento ${res.status}`);
return res.json();
}
async function baselineSkus() {
const data = await get("products", {
"searchCriteria[filterGroups][0][filters][0][field]": "status",
"searchCriteria[filterGroups][0][filters][0][value]": "1",
"searchCriteria[filterGroups][0][filters][0][conditionType]": "eq",
"searchCriteria[pageSize]": "200",
});
return new Set((data.items || []).map((item) => item.sku));
}Poll category products and a storefront-equivalent search
For each category you care about, call /rest/V1/categories/{id}/products to see the currently assigned SKUs. There is no public REST endpoint for catalogsearch_fulltext itself, so approximate the storefront quick search with a name like-filter against /rest/V1/products. Poll both every 5 to 10 seconds during a known reindex window.
def category_skus(category_id):
data = get(f"categories/{category_id}/products")
return {row["sku"] for row in data}
def search_skus(name_like):
params = {
"searchCriteria[filterGroups][0][filters][0][field]": "name",
"searchCriteria[filterGroups][0][filters][0][value]": f"%{name_like}%",
"searchCriteria[filterGroups][0][filters][0][conditionType]": "like",
"searchCriteria[pageSize]": "200",
}
data = get("products", params)
return {item["sku"] for item in data.get("items", [])}async function categorySkus(categoryId) {
const data = await get(`categories/${categoryId}/products`);
return new Set((data || []).map((row) => row.sku));
}
async function searchSkus(nameLike) {
const data = await get("products", {
"searchCriteria[filterGroups][0][filters][0][field]": "name",
"searchCriteria[filterGroups][0][filters][0][value]": `%${nameLike}%`,
"searchCriteria[filterGroups][0][filters][0][conditionType]": "like",
"searchCriteria[pageSize]": "200",
});
return new Set((data.items || []).map((item) => item.sku));
}Decide, with one pure function
The decision does not need any network access. Given the baseline set, the current category and search sets, the set that was already missing on the previous poll, and the current timestamp, it works out which SKUs are missing right now, and classifies each missing SKU as flapping if its absence is bounded to roughly a cron cycle or two, or stuck if it has been missing for three or more cycles running.
def is_product_flapping(baseline_skus, current_category_skus, current_search_skus,
previous_missing, now_ts, cron_interval_sec=60):
missing_from_category = baseline_skus - current_category_skus
missing_from_search = baseline_skus - current_search_skus
missing_now = missing_from_category | missing_from_search
flapping = set()
stuck = set()
for sku in missing_now:
first_seen_ts = previous_missing.get(sku, now_ts)
missing_for = now_ts - first_seen_ts
if missing_for > cron_interval_sec * 3:
stuck.add(sku)
else:
flapping.add(sku)
return {
"flapping": flapping,
"stuck": stuck,
"missing_from_category": missing_from_category,
"missing_from_search": missing_from_search,
}export function isProductFlapping(baselineSkus, currentCategorySkus, currentSearchSkus,
previousMissing, nowTs, cronIntervalSec = 60) {
const missingFromCategory = new Set([...baselineSkus].filter((s) => !currentCategorySkus.has(s)));
const missingFromSearch = new Set([...baselineSkus].filter((s) => !currentSearchSkus.has(s)));
const missingNow = new Set([...missingFromCategory, ...missingFromSearch]);
const flapping = new Set();
const stuck = new Set();
for (const sku of missingNow) {
const firstSeenTs = previousMissing.has(sku) ? previousMissing.get(sku) : nowTs;
const missingFor = nowTs - firstSeenTs;
if (missingFor > cronIntervalSec * 3) stuck.add(sku);
else flapping.add(sku);
}
return { flapping, stuck, missingFromCategory, missingFromSearch };
}Track how long each SKU has been missing
The pure function needs to know when each currently-missing SKU was first seen missing, not just whether it is missing right now. The polling loop keeps a small dictionary of SKU to first-missing timestamp, carrying forward entries for SKUs still missing and dropping entries for SKUs that reappeared, which is what lets the classification tell a two cycle flap apart from a stuck reindex.
def advance_missing_tracker(previous_missing, missing_now, now_ts):
updated = {}
for sku in missing_now:
updated[sku] = previous_missing.get(sku, now_ts)
return updatedexport function advanceMissingTracker(previousMissing, missingNow, nowTs) {
const updated = new Map();
for (const sku of missingNow) {
updated.set(sku, previousMissing.has(sku) ? previousMissing.get(sku) : nowTs);
}
return updated;
}Wire it together with a dry run guard
The loop polls once, runs the decision function, logs any flapping or stuck SKUs, and updates the tracker for the next poll. There is nothing safe to write to fix the index itself, so DRY_RUN guards even the one optional workaround: a no-op PUT /rest/V1/products/{sku} that re-affirms the existing status and visibility, which nudges the product back into the next changelog batch. Leave DRY_RUN=true to only log and report; a stuck SKU should go to an admin who runs bin/magento indexer:show-mode and bin/magento indexer:reindex catalogsearch_fulltext catalog_category_product.
This script never deletes or recreates index data and never guesses at a fix. It only reports. The one write it can make, guarded behind an explicit un-flag of DRY_RUN, is a no-op re-affirmation of a product's existing status, which is safe because it changes nothing about the product itself.
The full code
Here is the complete script in one file for each language. It reads settings from the environment, polls the baseline, category, and search endpoints, runs the pure decision function, logs flapping and stuck SKUs, and only touches the API to report unless a human explicitly turns off dry run.
"""Detect Magento 2 products that flap out of category or search results during
a scheduled reindex, and tell flapping (transient, self healing) apart from
stuck (worth escalating). Never writes to the index. Run on a schedule during
a known reindex window. Safe to run again and again.
"""
import os
import time
import logging
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("flag_flapping_products")
MAGENTO_URL = os.environ["MAGENTO_URL"].rstrip("/")
TOKEN = os.environ["MAGENTO_ADMIN_TOKEN"]
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"
CATEGORY_IDS = [c for c in os.environ.get("WATCH_CATEGORY_IDS", "").split(",") if c]
SEARCH_TERMS = [t for t in os.environ.get("WATCH_SEARCH_TERMS", "").split(",") if t]
POLL_INTERVAL_SEC = float(os.environ.get("POLL_INTERVAL_SEC", "8"))
POLL_COUNT = int(os.environ.get("POLL_COUNT", "6"))
CRON_INTERVAL_SEC = int(os.environ.get("CRON_INTERVAL_SEC", "60"))
def get(path, params=None):
r = requests.get(
f"{MAGENTO_URL}/rest/V1/{path}",
params=params or {},
headers={"Authorization": f"Bearer {TOKEN}"},
timeout=30,
)
r.raise_for_status()
return r.json()
def put(path, body):
r = requests.put(
f"{MAGENTO_URL}/rest/V1/{path}",
json=body,
headers={"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"},
timeout=30,
)
r.raise_for_status()
return r.json()
def baseline_skus():
params = {
"searchCriteria[filterGroups][0][filters][0][field]": "status",
"searchCriteria[filterGroups][0][filters][0][value]": "1",
"searchCriteria[filterGroups][0][filters][0][conditionType]": "eq",
"searchCriteria[pageSize]": "200",
}
data = get("products", params)
return {item["sku"] for item in data.get("items", [])}
def category_skus(category_id):
data = get(f"categories/{category_id}/products")
return {row["sku"] for row in data}
def search_skus(name_like):
params = {
"searchCriteria[filterGroups][0][filters][0][field]": "name",
"searchCriteria[filterGroups][0][filters][0][value]": f"%{name_like}%",
"searchCriteria[filterGroups][0][filters][0][conditionType]": "like",
"searchCriteria[pageSize]": "200",
}
data = get("products", params)
return {item["sku"] for item in data.get("items", [])}
def is_product_flapping(baseline_skus, current_category_skus, current_search_skus,
previous_missing, now_ts, cron_interval_sec=60):
missing_from_category = baseline_skus - current_category_skus
missing_from_search = baseline_skus - current_search_skus
missing_now = missing_from_category | missing_from_search
flapping = set()
stuck = set()
for sku in missing_now:
first_seen_ts = previous_missing.get(sku, now_ts)
missing_for = now_ts - first_seen_ts
if missing_for > cron_interval_sec * 3:
stuck.add(sku)
else:
flapping.add(sku)
return {
"flapping": flapping,
"stuck": stuck,
"missing_from_category": missing_from_category,
"missing_from_search": missing_from_search,
}
def advance_missing_tracker(previous_missing, missing_now, now_ts):
updated = {}
for sku in missing_now:
updated[sku] = previous_missing.get(sku, now_ts)
return updated
def reaffirm_product(sku, status):
body = {"product": {"sku": sku, "status": status}}
return put(f"products/{sku}", body)
def run():
baseline = baseline_skus()
log.info("Baseline has %d enabled, visible product(s).", len(baseline))
previous_missing = {}
stuck_reported = set()
for poll_n in range(POLL_COUNT):
now_ts = time.time()
current_category = set()
for category_id in CATEGORY_IDS:
current_category |= category_skus(category_id)
current_search = set()
for term in SEARCH_TERMS:
current_search |= search_skus(term)
# If no categories or terms were configured, treat that side as fully present
# so the check does not falsely flag everything as missing.
current_category_effective = current_category if CATEGORY_IDS else set(baseline)
current_search_effective = current_search if SEARCH_TERMS else set(baseline)
result = is_product_flapping(
baseline, current_category_effective, current_search_effective,
previous_missing, now_ts, CRON_INTERVAL_SEC,
)
for sku in result["flapping"]:
log.info("Poll %d: %s is flapping (transient, expected to self heal).", poll_n, sku)
for sku in result["stuck"]:
if sku in stuck_reported:
continue
log.warning("Poll %d: %s is stuck missing for over %ds. Check indexer:show-mode "
"and reindex catalogsearch_fulltext catalog_category_product.",
poll_n, sku, CRON_INTERVAL_SEC * 3)
stuck_reported.add(sku)
if not DRY_RUN:
log.info("DRY_RUN is off: re-affirming %s status=1 as a no-op workaround.", sku)
reaffirm_product(sku, 1)
missing_now = result["missing_from_category"] | result["missing_from_search"]
previous_missing = advance_missing_tracker(previous_missing, missing_now, now_ts)
if poll_n < POLL_COUNT - 1:
time.sleep(POLL_INTERVAL_SEC)
log.info("Done. %d SKU(s) stuck across the polling window.", len(stuck_reported))
if __name__ == "__main__":
run()/**
* Detect Magento 2 products that flap out of category or search results during
* a scheduled reindex, and tell flapping (transient, self healing) apart from
* stuck (worth escalating). Never writes to the index. Run on a schedule during
* a known reindex window. Safe to run again and again.
*
* Guide: https://www.allanninal.dev/magento/products-flap-during-scheduled-indexing/
*/
import { pathToFileURL } from "node:url";
const MAGENTO_URL = (process.env.MAGENTO_URL || "https://demo.example.com").replace(/\/$/, "");
const TOKEN = process.env.MAGENTO_ADMIN_TOKEN || "token_dummy";
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
const CATEGORY_IDS = (process.env.WATCH_CATEGORY_IDS || "").split(",").filter(Boolean);
const SEARCH_TERMS = (process.env.WATCH_SEARCH_TERMS || "").split(",").filter(Boolean);
const POLL_INTERVAL_SEC = Number(process.env.POLL_INTERVAL_SEC || 8);
const POLL_COUNT = Number(process.env.POLL_COUNT || 6);
const CRON_INTERVAL_SEC = Number(process.env.CRON_INTERVAL_SEC || 60);
export function isProductFlapping(baselineSkus, currentCategorySkus, currentSearchSkus,
previousMissing, nowTs, cronIntervalSec = 60) {
const missingFromCategory = new Set([...baselineSkus].filter((s) => !currentCategorySkus.has(s)));
const missingFromSearch = new Set([...baselineSkus].filter((s) => !currentSearchSkus.has(s)));
const missingNow = new Set([...missingFromCategory, ...missingFromSearch]);
const flapping = new Set();
const stuck = new Set();
for (const sku of missingNow) {
const firstSeenTs = previousMissing.has(sku) ? previousMissing.get(sku) : nowTs;
const missingFor = nowTs - firstSeenTs;
if (missingFor > cronIntervalSec * 3) stuck.add(sku);
else flapping.add(sku);
}
return { flapping, stuck, missingFromCategory, missingFromSearch };
}
export function advanceMissingTracker(previousMissing, missingNow, nowTs) {
const updated = new Map();
for (const sku of missingNow) {
updated.set(sku, previousMissing.has(sku) ? previousMissing.get(sku) : nowTs);
}
return updated;
}
async function get(path, params = {}) {
const url = new URL(`${MAGENTO_URL}/rest/V1/${path}`);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
const res = await fetch(url, { headers: { Authorization: `Bearer ${TOKEN}` } });
if (!res.ok) throw new Error(`Magento ${res.status}`);
return res.json();
}
async function put(path, body) {
const res = await fetch(`${MAGENTO_URL}/rest/V1/${path}`, {
method: "PUT",
headers: { Authorization: `Bearer ${TOKEN}`, "Content-Type": "application/json" },
body: JSON.stringify(body),
});
if (!res.ok) throw new Error(`Magento ${res.status}`);
return res.json();
}
async function baselineSkus() {
const data = await get("products", {
"searchCriteria[filterGroups][0][filters][0][field]": "status",
"searchCriteria[filterGroups][0][filters][0][value]": "1",
"searchCriteria[filterGroups][0][filters][0][conditionType]": "eq",
"searchCriteria[pageSize]": "200",
});
return new Set((data.items || []).map((item) => item.sku));
}
async function categorySkus(categoryId) {
const data = await get(`categories/${categoryId}/products`);
return new Set((data || []).map((row) => row.sku));
}
async function searchSkus(nameLike) {
const data = await get("products", {
"searchCriteria[filterGroups][0][filters][0][field]": "name",
"searchCriteria[filterGroups][0][filters][0][value]": `%${nameLike}%`,
"searchCriteria[filterGroups][0][filters][0][conditionType]": "like",
"searchCriteria[pageSize]": "200",
});
return new Set((data.items || []).map((item) => item.sku));
}
async function reaffirmProduct(sku, status) {
return put(`products/${sku}`, { product: { sku, status } });
}
function sleep(ms) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function run() {
const baseline = await baselineSkus();
console.log(`Baseline has ${baseline.size} enabled, visible product(s).`);
let previousMissing = new Map();
const stuckReported = new Set();
for (let pollN = 0; pollN < POLL_COUNT; pollN++) {
const nowTs = Date.now() / 1000;
let currentCategory = new Set();
for (const categoryId of CATEGORY_IDS) {
const skus = await categorySkus(categoryId);
currentCategory = new Set([...currentCategory, ...skus]);
}
let currentSearch = new Set();
for (const term of SEARCH_TERMS) {
const skus = await searchSkus(term);
currentSearch = new Set([...currentSearch, ...skus]);
}
// If no categories or terms were configured, treat that side as fully present
// so the check does not falsely flag everything as missing.
const currentCategoryEffective = CATEGORY_IDS.length ? currentCategory : new Set(baseline);
const currentSearchEffective = SEARCH_TERMS.length ? currentSearch : new Set(baseline);
const result = isProductFlapping(
baseline, currentCategoryEffective, currentSearchEffective,
previousMissing, nowTs, CRON_INTERVAL_SEC,
);
for (const sku of result.flapping) {
console.log(`Poll ${pollN}: ${sku} is flapping (transient, expected to self heal).`);
}
for (const sku of result.stuck) {
if (stuckReported.has(sku)) continue;
console.warn(`Poll ${pollN}: ${sku} is stuck missing for over ${CRON_INTERVAL_SEC * 3}s. ` +
`Check indexer:show-mode and reindex catalogsearch_fulltext catalog_category_product.`);
stuckReported.add(sku);
if (!DRY_RUN) {
console.log(`DRY_RUN is off: re-affirming ${sku} status=1 as a no-op workaround.`);
await reaffirmProduct(sku, 1);
}
}
const missingNow = new Set([...result.missingFromCategory, ...result.missingFromSearch]);
previousMissing = advanceMissingTracker(previousMissing, missingNow, nowTs);
if (pollN < POLL_COUNT - 1) await sleep(POLL_INTERVAL_SEC * 1000);
}
console.log(`Done. ${stuckReported.size} SKU(s) stuck across the polling window.`);
}
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((err) => { console.error(err); process.exit(1); });
}Add a test
The classification logic is the part worth testing, because it decides whether a report says "ignore this, it will heal" or "escalate this." Because is_product_flapping is pure, plain sets and timestamps stand in for API responses, and no Magento store or network call is needed.
from flag_flapping_products import is_product_flapping, advance_missing_tracker
BASELINE = {"sku-a", "sku-b", "sku-c"}
def test_no_missing_when_everything_present():
result = is_product_flapping(BASELINE, BASELINE, BASELINE, {}, 1000, 60)
assert result["flapping"] == set()
assert result["stuck"] == set()
def test_new_miss_is_flapping_not_stuck():
current_category = BASELINE - {"sku-a"}
result = is_product_flapping(BASELINE, current_category, BASELINE, {}, 1000, 60)
assert result["flapping"] == {"sku-a"}
assert result["stuck"] == set()
def test_recently_missing_stays_flapping():
previous_missing = {"sku-a": 1000}
current_category = BASELINE - {"sku-a"}
result = is_product_flapping(BASELINE, current_category, BASELINE, previous_missing, 1090, 60)
assert result["flapping"] == {"sku-a"}
assert result["stuck"] == set()
def test_missing_past_three_cycles_is_stuck():
previous_missing = {"sku-a": 1000}
current_category = BASELINE - {"sku-a"}
result = is_product_flapping(BASELINE, current_category, BASELINE, previous_missing, 1000 + 181, 60)
assert result["stuck"] == {"sku-a"}
assert result["flapping"] == set()
def test_missing_from_search_is_tracked_separately():
current_search = BASELINE - {"sku-b"}
result = is_product_flapping(BASELINE, BASELINE, current_search, {}, 1000, 60)
assert result["missing_from_search"] == {"sku-b"}
assert result["missing_from_category"] == set()
def test_advance_missing_tracker_keeps_first_seen_ts():
previous_missing = {"sku-a": 500}
updated = advance_missing_tracker(previous_missing, {"sku-a", "sku-b"}, 900)
assert updated["sku-a"] == 500
assert updated["sku-b"] == 900
def test_advance_missing_tracker_drops_recovered_skus():
previous_missing = {"sku-a": 500, "sku-b": 600}
updated = advance_missing_tracker(previous_missing, {"sku-a"}, 900)
assert "sku-b" not in updatedimport { test } from "node:test";
import assert from "node:assert/strict";
import { isProductFlapping, advanceMissingTracker } from "./flag-flapping-products.js";
const BASELINE = new Set(["sku-a", "sku-b", "sku-c"]);
test("no missing when everything present", () => {
const result = isProductFlapping(BASELINE, BASELINE, BASELINE, new Map(), 1000, 60);
assert.equal(result.flapping.size, 0);
assert.equal(result.stuck.size, 0);
});
test("new miss is flapping not stuck", () => {
const currentCategory = new Set([...BASELINE].filter((s) => s !== "sku-a"));
const result = isProductFlapping(BASELINE, currentCategory, BASELINE, new Map(), 1000, 60);
assert.deepEqual([...result.flapping], ["sku-a"]);
assert.equal(result.stuck.size, 0);
});
test("recently missing stays flapping", () => {
const previousMissing = new Map([["sku-a", 1000]]);
const currentCategory = new Set([...BASELINE].filter((s) => s !== "sku-a"));
const result = isProductFlapping(BASELINE, currentCategory, BASELINE, previousMissing, 1090, 60);
assert.deepEqual([...result.flapping], ["sku-a"]);
assert.equal(result.stuck.size, 0);
});
test("missing past three cycles is stuck", () => {
const previousMissing = new Map([["sku-a", 1000]]);
const currentCategory = new Set([...BASELINE].filter((s) => s !== "sku-a"));
const result = isProductFlapping(BASELINE, currentCategory, BASELINE, previousMissing, 1000 + 181, 60);
assert.deepEqual([...result.stuck], ["sku-a"]);
assert.equal(result.flapping.size, 0);
});
test("missing from search is tracked separately", () => {
const currentSearch = new Set([...BASELINE].filter((s) => s !== "sku-b"));
const result = isProductFlapping(BASELINE, BASELINE, currentSearch, new Map(), 1000, 60);
assert.deepEqual([...result.missingFromSearch], ["sku-b"]);
assert.equal(result.missingFromCategory.size, 0);
});
test("advanceMissingTracker keeps first seen timestamp", () => {
const previousMissing = new Map([["sku-a", 500]]);
const updated = advanceMissingTracker(previousMissing, new Set(["sku-a", "sku-b"]), 900);
assert.equal(updated.get("sku-a"), 500);
assert.equal(updated.get("sku-b"), 900);
});
test("advanceMissingTracker drops recovered skus", () => {
const previousMissing = new Map([["sku-a", 500], ["sku-b", 600]]);
const updated = advanceMissingTracker(previousMissing, new Set(["sku-a"]), 900);
assert.equal(updated.has("sku-b"), false);
});Case studies
The support ticket that fixed itself
A merchant ran a bulk price update on four hundred SKUs right before a flash sale. Within a minute, three support tickets came in saying products had vanished from a category page. By the time anyone opened the admin, all three products were back, and nobody could reproduce it.
Running the poller during the next bulk update confirmed it: each product was missing for one, sometimes two, cron cycles and then reappeared on its own. Every case classified as flapping, never stuck. The team stopped opening emergency tickets for it and instead watches the report during big catalog updates.
The SKU that would not come back
A larger catalog with a slow Elasticsearch cluster saw the same kind of disappearance, but one SKU stayed missing from search for over ten minutes while everything else recovered within a minute. The team almost dismissed it as normal flapping.
The poller's stuck classification caught it after the third consecutive missing poll and logged a clear escalation instead of another quiet flap entry. An admin ran bin/magento indexer:reindex catalogsearch_fulltext by hand and the product came back immediately, pointing at a batch that had silently failed rather than a normal cycle.
After this runs during reindex windows, a vanished product is either a logged, expected flap that resolves on its own within a cycle or two, or a clearly flagged stuck SKU with a specific CLI command to run next. Nobody re-saves random products hoping it helps, and nobody escalates a normal delete then recreate cycle as an emergency.
FAQ
Why do products briefly disappear from category or search pages in Magento 2?
When an indexer runs Update by Schedule, a cron job reads the changelog table for changed products and rebuilds their index entries in batches. For catalogsearch_fulltext the batch process deletes the existing search documents for those product IDs before it recreates them, so for a short window the storefront query returns nothing for those products. They reappear once the batch finishes.
Is this a bug I should report or a sign my Magento store is broken?
No. This is expected behavior of Magento's own scheduled indexing mechanism, not corrupted data. It is documented in Magento core issues and forum threads. The correct response is to detect and measure it, and if it is frequent or long lasting, tune indexer settings, not to try to patch the database.
Can I fix flapping products with a REST API call?
Not directly. The flap is produced by Magento's own indexer and mview tables, which are CLI and database internals with no REST endpoint. The safe REST based workaround is a no-op PUT to the product that re-affirms its status and visibility, which forces it back into the next changelog batch. The real remediation, such as changing indexer mode or batch size, requires bin/magento CLI access.
Related field notes
Citations
On the problem:
- Magento 2 GitHub Issues: Products randomly disappear from category view on storefront when partial indexation is running. github.com/magento/magento2/issues/35248
- Magento 2 GitHub Issues: dependencies of the catalogsearch_fulltext index in scheduled mode. github.com/magento/magento2/issues/22885
- Magento Forums: Update by schedule and reindex problem. community.magento.com Update by schedule and reindex problem
On the solution:
- Adobe Commerce: Manage the indexers, including indexer:show-mode and indexer:reindex. experienceleague.adobe.com manage-indexers
- Adobe Commerce PHP Extensions: Indexing, changelog tables, and Mview. developer.adobe.com/commerce/php/development/components/indexing
- Adobe Commerce Web APIs: the Products endpoint used for searchCriteria filtering. developer.adobe.com/commerce/webapi/rest/modules/products
Stuck on a tricky one?
If you have a problem in Magento 2 or Adobe Commerce indexing, cron, MSI stock, or order sync that you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this stop a false alarm?
If this saved you an emergency ticket over a product that was going to heal itself, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi