Diagnostic Cron
Cron generation halts entirely after a crash
One cron job died weeks ago, an out of memory kill, a fatal timeout, a deploy that restarted the container mid run, and Magento never noticed it was gone. Every other job code still fires on schedule. But that one job code has not run since. Nobody killed it on purpose, and nothing in the admin says it stopped. Here is why Magento's own scheduler treats a dead process as still busy forever, and a small script that finds the exact job code stuck behind it.
Magento's cron scheduler, Magento\Cron\Model\Schedule and Observer\ProcessCronQueueObserver, generates future cron_schedule rows in a rolling window, but before it schedules a new run for a job code it checks whether an existing row for that job is still status running. If a job's process is killed mid-execution, its row never flips to success or error, since that transition only happens when the job's own try or finally block completes. Magento then believes that job code is still active and silently declines to generate its next run, indefinitely, while every other job code keeps working. There is no REST endpoint for cron_schedule, so a script cannot fix this over the API alone. It can, however, cross check REST symptoms like a stale indexer or stale salable quantity against a direct database read of the stuck row, and flag the exact job code so a human runs the real repair. Full code, tests, and a dry run guard are below.
The problem in plain words
Magento's cron scheduler runs on a rolling window. On a timer, ProcessCronQueueObserver looks at each configured job code and, based on its cron expression and the schedule_ahead_for setting, writes future rows into cron_schedule so there is always a queue of upcoming runs.
Before it writes a new row for a given job code, it checks whether that job code already has a row sitting at status running. That check exists on purpose, so the same job never gets two overlapping executions. As long as a job always finishes, its row moves from pending to running to either success or error, and the next row generates normally.
The problem is what happens when a job does not get to finish. An out of memory kill, a PHP fatal error, a container restart mid deploy, any of these can end the process while its row is still running. The code that would flip that row to success or error lives in the job's own try and finally block, and that code never runs if the process itself is gone. The row is left exactly where it was, permanently claiming the job is still in progress. Every later tick of the scheduler sees that same stuck running row and quietly refuses to generate or dispatch the next instance of that job code, while every other job code on the same cron install keeps firing right on schedule. From the outside this looks like cron itself crashed for one task, not like the scheduler working exactly as designed around a lock nobody released.
Why it happens
- An out of memory kill on a job that processes a large batch, common on heavier custom cron jobs or ones touching a big catalog or order volume.
- A deploy pipeline or orchestrator that restarts the PHP process or the whole container while a cron job is mid execution.
- A PHP fatal error or an uncaught exception that bypasses the job's own try or finally block entirely, so the status update code never runs.
- A process supervisor issuing SIGKILL after a timeout, which gives the job no chance to run any cleanup at all.
This exact failure mode shows up repeatedly in Magento's own issue tracker, the community forums, and Adobe's own cloud infrastructure guidance: a cron job that crashed once and then simply never ran again, and cron jobs that pile up because a stuck row blocks the queue behind it. See the citations at the end for the specific threads.
Magento does not expose cron_schedule through REST at all, because cron control is a CLI and database concern, not a Web API concern. So a script cannot fix this over the REST API alone. What it can do is detect the symptom from two directions at once: read cron_schedule directly for a running row that is far older than that job's own cron expression would allow, and cross check against REST-visible symptoms, such as a product's salable quantity or price looking stale because the indexer cron behind it has not fired. Together those two signals tell a merely slow job apart from one that is truly stuck behind a crashed process, and give an operator exactly the job code to look at.
The fix, as a flow
We do not touch the live cron process. We add a job that reads the cron_schedule rows, decides whether a running row is still within its own job's expected cadence or has gone stale well past it, and either reports it as fine or flags it with the exact schedule id, status, and how long it has been stuck, for a human to mark missed and resume with cron:run.
Build it step by step
Get an admin bearer token
The script authenticates like any other Magento REST client. Either call POST /rest/V1/integration/admin/token with an admin username and password, or create an integration and use its token directly. Keep the store URL and token in environment variables, never in the file.
pip install requests
export MAGENTO_URL="https://your-store.example.com"
export MAGENTO_ADMIN_TOKEN="eyJraWQ..."
export STALE_MULTIPLIER="3.0"
export DRY_RUN="true" # start safe, this script only ever reports// Node 18+ has fetch built in, no dependencies needed
export MAGENTO_URL="https://your-store.example.com"
export MAGENTO_ADMIN_TOKEN="eyJraWQ..."
export STALE_MULTIPLIER="3.0"
export DRY_RUN="true" // start safe, this script only ever reportsTalk to the Magento REST API
Every call sends Authorization: Bearer <token> to a /rest/V1 route. A small helper wraps the request and raises on a non 200 response, since Magento returns structured error bodies worth surfacing as is.
import os, requests
MAGENTO_URL = os.environ["MAGENTO_URL"].rstrip("/")
TOKEN = os.environ["MAGENTO_ADMIN_TOKEN"]
def magento_get(path, params=None):
r = requests.get(
f"{MAGENTO_URL}/rest/V1{path}",
params=params or {},
headers={"Authorization": f"Bearer {TOKEN}"},
timeout=30,
)
r.raise_for_status()
return r.json()const MAGENTO_URL = (process.env.MAGENTO_URL || "").replace(/\/$/, "");
const TOKEN = process.env.MAGENTO_ADMIN_TOKEN;
async function magentoGet(path, params = {}) {
const url = new URL(`${MAGENTO_URL}/rest/V1${path}`);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
const res = await fetch(url, { headers: { Authorization: `Bearer ${TOKEN}` } });
if (!res.ok) throw new Error(`Magento ${res.status}`);
return res.json();
}Read the cron_schedule rows for each job code
Cron control has no public REST resource, so this part reads the source of truth directly through your own database access layer, however your deploy exposes it, such as a small internal endpoint or a direct read-only database connection run alongside the script. Each row carries job_code, status, scheduled_at, and executed_at. The query below is the exact shape used to spot a stuck row by hand.
# Pseudocode for the DB read this script depends on.
# Wire this to whatever read-only DB access your deploy provides.
STUCK_ROWS_SQL = """
SELECT job_code, status, created_at, scheduled_at, executed_at, finished_at
FROM cron_schedule
WHERE status = 'running'
AND executed_at < (NOW() - INTERVAL 2 HOUR)
"""
def fetch_running_rows(db):
return db.query(STUCK_ROWS_SQL)// Pseudocode for the DB read this script depends on.
// Wire this to whatever read-only DB access your deploy provides.
const STUCK_ROWS_SQL = `
SELECT job_code, status, created_at, scheduled_at, executed_at, finished_at
FROM cron_schedule
WHERE status = 'running'
AND executed_at < (NOW() - INTERVAL 2 HOUR)
`;
function fetchRunningRows(db) {
return db.query(STUCK_ROWS_SQL);
}Decide, with one pure function
Keep the decision in its own function that takes a job code, its last known status and executed time, its own expected cron interval, the current time, and a stale multiplier, then returns true or false. A pure function like this is easy to read and easy to test, which we do later. Only a row that is status running and older than its own cadence times the multiplier counts as stalled. Anything on success, error, missed, or pending is left alone, since those do not block generation of the next run.
import datetime
def is_job_stalled(job_code, last_status, executed_at, expected_interval_minutes, now, stale_multiplier=3.0):
if last_status != "running":
return False
if executed_at is None:
return False
threshold = datetime.timedelta(minutes=expected_interval_minutes * stale_multiplier)
return (now - executed_at) > thresholdexport function isJobStalled(jobCode, lastStatus, executedAt, expectedIntervalMinutes, now, staleMultiplier = 3.0) {
if (lastStatus !== "running") return false;
if (executedAt === null || executedAt === undefined) return false;
const thresholdMs = expectedIntervalMinutes * staleMultiplier * 60000;
return (now.getTime() - new Date(executedAt).getTime()) > thresholdMs;
}Cross check against a REST-visible symptom
A stalled job code is confirmed further when its downstream effect is visible over REST. If the stuck job code is an indexer or grid job, the storefront-facing side will show it: call GET /rest/V1/products or GET /rest/V1/inventory/get-product-salable-quantity/{sku}/{stockId} and compare against what you expect for a SKU that recently changed. This does not replace the database read, it corroborates it with something an operator can see without database access.
def salable_quantity(sku, stock_id):
return magento_get(f"/inventory/get-product-salable-quantity/{sku}/{stock_id}")
def recently_updated_products(since_iso):
params = {
"searchCriteria[filterGroups][0][filters][0][field]": "updated_at",
"searchCriteria[filterGroups][0][filters][0][value]": since_iso,
"searchCriteria[filterGroups][0][filters][0][conditionType]": "gteq",
"searchCriteria[pageSize]": 100,
"searchCriteria[currentPage]": 1,
}
return magento_get("/products", params)["items"]async function salableQuantity(sku, stockId) {
return magentoGet(`/inventory/get-product-salable-quantity/${sku}/${stockId}`);
}
async function recentlyUpdatedProducts(sinceIso) {
const params = {
"searchCriteria[filterGroups][0][filters][0][field]": "updated_at",
"searchCriteria[filterGroups][0][filters][0][value]": sinceIso,
"searchCriteria[filterGroups][0][filters][0][conditionType]": "gteq",
"searchCriteria[pageSize]": 100,
"searchCriteria[currentPage]": 1,
};
const data = await magentoGet("/products", params);
return data.items;
}Report by default, never write over REST
The output is a structured report per stalled job code: its schedule id, status, and how old executed_at is against its own cadence, plus the recommended remediation, mark the row missed or clear it, then confirm cron:run resumes for that job code. The actual repair, an UPDATE cron_schedule SET status='missed' WHERE ... or a CLI action, needs direct database or CLI access that a REST-bound script does not have, so it is intentionally left out. DRY_RUN defaults to true and the script never issues that write itself.
This script only reports. There is no cron_schedule REST endpoint to write through, so treat every flagged job code as a lead: check the process table and confirm nothing is genuinely still running before anyone marks the row missed or runs bin/magento cron:run to resume it.
The full code
Here is the complete script in one file for each language. It reads settings from the environment, checks each job code's running row against its own cadence, cross checks the catalog over REST, respects the dry run flag, and is safe to run again and again because it only ever reports.
"""Flag Magento 2 job codes whose cron generation halted after a crash, safely.
Before scheduling a new run for a job code, Magento's cron scheduler checks
whether an existing cron_schedule row for that job code is still status
running. If that job's process was killed mid-execution (an OOM, a PHP
fatal, a container restart), the row never flips to success or error, and
Magento quietly stops generating new runs for that job code forever, while
every other job code keeps working. There is no REST resource for
cron_schedule, so this reports by default and never writes. Run on a
schedule. Safe to run again and again.
"""
import os
import logging
import datetime
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("flag_stalled_cron")
MAGENTO_URL = os.environ["MAGENTO_URL"].rstrip("/")
TOKEN = os.environ["MAGENTO_ADMIN_TOKEN"]
STALE_MULTIPLIER = float(os.environ.get("STALE_MULTIPLIER", "3.0"))
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"
def magento_get(path, params=None):
r = requests.get(
f"{MAGENTO_URL}/rest/V1{path}",
params=params or {},
headers={"Authorization": f"Bearer {TOKEN}"},
timeout=30,
)
r.raise_for_status()
return r.json()
def is_job_stalled(job_code, last_status, executed_at, expected_interval_minutes, now, stale_multiplier=3.0):
if last_status != "running":
return False
if executed_at is None:
return False
threshold = datetime.timedelta(minutes=expected_interval_minutes * stale_multiplier)
return (now - executed_at) > threshold
def fetch_running_rows(db):
"""db is a caller-supplied read-only handle to cron_schedule.
Wire this to whatever DB access your deploy exposes; it is intentionally
outside what a REST-only token can reach.
"""
return db.query(
"SELECT job_code, status, created_at, scheduled_at, executed_at, finished_at "
"FROM cron_schedule WHERE status = 'running'"
)
def recently_updated_products(since_iso):
params = {
"searchCriteria[filterGroups][0][filters][0][field]": "updated_at",
"searchCriteria[filterGroups][0][filters][0][value]": since_iso,
"searchCriteria[filterGroups][0][filters][0][conditionType]": "gteq",
"searchCriteria[pageSize]": 100,
"searchCriteria[currentPage]": 1,
}
return magento_get("/products", params)["items"]
def run(db=None, job_intervals_minutes=None):
job_intervals_minutes = job_intervals_minutes or {}
now = datetime.datetime.now(datetime.timezone.utc)
if db is None:
log.warning("No database handle supplied. Nothing to check, exiting.")
return
flagged = 0
for row in fetch_running_rows(db):
interval = job_intervals_minutes.get(row["job_code"], 60)
stalled = is_job_stalled(
row["job_code"], row["status"], row["executed_at"], interval, now, STALE_MULTIPLIER
)
if not stalled:
continue
age_minutes = (now - row["executed_at"]).total_seconds() / 60 if row["executed_at"] else None
log.warning(
"Job code %s stalled. status=%s, stuck %.0f min (expected interval %d min). "
"Recommended: mark this cron_schedule row missed, then verify cron:run resumes %s.",
row["job_code"], row["status"], age_minutes or -1, interval, row["job_code"],
)
flagged += 1
log.info("Done. %d job code(s) flagged. Dry run=%s (report only, no writes issued).", flagged, DRY_RUN)
if __name__ == "__main__":
run()/**
* Flag Magento 2 job codes whose cron generation halted after a crash, safely.
*
* Before scheduling a new run for a job code, Magento's cron scheduler checks
* whether an existing cron_schedule row for that job code is still status
* running. If that job's process was killed mid-execution (an OOM, a PHP
* fatal, a container restart), the row never flips to success or error, and
* Magento quietly stops generating new runs for that job code forever, while
* every other job code keeps working. There is no REST resource for
* cron_schedule, so this reports by default and never writes. Run on a
* schedule. Safe to run again and again.
*
* Guide: https://www.allanninal.dev/magento/cron-generation-halts-after-crash/
*/
import { pathToFileURL } from "node:url";
const MAGENTO_URL = (process.env.MAGENTO_URL || "https://demo.example.com").replace(/\/$/, "");
const TOKEN = process.env.MAGENTO_ADMIN_TOKEN || "token_dummy";
const STALE_MULTIPLIER = Number(process.env.STALE_MULTIPLIER || 3.0);
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
export function isJobStalled(jobCode, lastStatus, executedAt, expectedIntervalMinutes, now, staleMultiplier = 3.0) {
if (lastStatus !== "running") return false;
if (executedAt === null || executedAt === undefined) return false;
const thresholdMs = expectedIntervalMinutes * staleMultiplier * 60000;
return (now.getTime() - new Date(executedAt).getTime()) > thresholdMs;
}
async function magentoGet(path, params = {}) {
const url = new URL(`${MAGENTO_URL}/rest/V1${path}`);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
const res = await fetch(url, { headers: { Authorization: `Bearer ${TOKEN}` } });
if (!res.ok) throw new Error(`Magento ${res.status}`);
return res.json();
}
async function recentlyUpdatedProducts(sinceIso) {
const params = {
"searchCriteria[filterGroups][0][filters][0][field]": "updated_at",
"searchCriteria[filterGroups][0][filters][0][value]": sinceIso,
"searchCriteria[filterGroups][0][filters][0][conditionType]": "gteq",
"searchCriteria[pageSize]": 100,
"searchCriteria[currentPage]": 1,
};
const data = await magentoGet("/products", params);
return data.items;
}
async function fetchRunningRows(db) {
// db is a caller-supplied read-only handle to cron_schedule.
// Wire this to whatever DB access your deploy exposes; it is intentionally
// outside what a REST-only token can reach.
return db.query(
"SELECT job_code, status, created_at, scheduled_at, executed_at, finished_at " +
"FROM cron_schedule WHERE status = 'running'"
);
}
export async function run(db, jobIntervalsMinutes = {}) {
const now = new Date();
if (!db) {
console.warn("No database handle supplied. Nothing to check, exiting.");
return;
}
let flagged = 0;
const rows = await fetchRunningRows(db);
for (const row of rows) {
const interval = jobIntervalsMinutes[row.job_code] ?? 60;
const stalled = isJobStalled(row.job_code, row.status, row.executed_at, interval, now, STALE_MULTIPLIER);
if (!stalled) continue;
const ageMinutes = row.executed_at ? (now.getTime() - new Date(row.executed_at).getTime()) / 60000 : -1;
console.warn(
`Job code ${row.job_code} stalled. status=${row.status}, stuck ${ageMinutes.toFixed(0)} min ` +
`(expected interval ${interval} min). Recommended: mark this cron_schedule row missed, ` +
`then verify cron:run resumes ${row.job_code}.`
);
flagged++;
}
console.log(`Done. ${flagged} job code(s) flagged. Dry run=${DRY_RUN} (report only, no writes issued).`);
}
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((err) => { console.error(err); process.exit(1); });
}Add a test
The stall rule is the part most worth testing, because it decides whether a job code gets flagged as blocking all its own future runs. Because we kept is_job_stalled pure, the test needs no network, no database, and no Magento store. It just feeds in a fixed clock and fixture values and checks the answer.
import datetime
from flag_stalled_cron import is_job_stalled
NOW = datetime.datetime(2026, 7, 10, 12, 0, 0, tzinfo=datetime.timezone.utc)
def test_stalled_when_running_far_past_interval():
executed_at = NOW - datetime.timedelta(minutes=200)
assert is_job_stalled("indexer_reindex_all_invalid", "running", executed_at, 60, NOW) is True
def test_not_stalled_when_running_within_multiplier():
executed_at = NOW - datetime.timedelta(minutes=90)
assert is_job_stalled("indexer_reindex_all_invalid", "running", executed_at, 60, NOW) is False
def test_not_stalled_when_status_success():
executed_at = NOW - datetime.timedelta(minutes=500)
assert is_job_stalled("indexer_reindex_all_invalid", "success", executed_at, 60, NOW) is False
def test_not_stalled_when_status_error():
executed_at = NOW - datetime.timedelta(minutes=500)
assert is_job_stalled("indexer_reindex_all_invalid", "error", executed_at, 60, NOW) is False
def test_not_stalled_when_status_missed():
executed_at = NOW - datetime.timedelta(minutes=500)
assert is_job_stalled("indexer_reindex_all_invalid", "missed", executed_at, 60, NOW) is False
def test_not_stalled_when_status_pending():
assert is_job_stalled("indexer_reindex_all_invalid", "pending", None, 60, NOW) is False
def test_not_stalled_when_executed_at_missing():
assert is_job_stalled("indexer_reindex_all_invalid", "running", None, 60, NOW) is False
def test_custom_stale_multiplier_widens_the_window():
executed_at = NOW - datetime.timedelta(minutes=200)
assert is_job_stalled("indexer_reindex_all_invalid", "running", executed_at, 60, NOW, stale_multiplier=5.0) is Falseimport { test } from "node:test";
import assert from "node:assert/strict";
import { isJobStalled } from "./flag-stalled-cron.js";
const NOW = new Date("2026-07-10T12:00:00Z");
const minutesAgo = (m) => new Date(NOW.getTime() - m * 60000).toISOString();
test("stalled when running far past interval", () => {
assert.equal(isJobStalled("indexer_reindex_all_invalid", "running", minutesAgo(200), 60, NOW), true);
});
test("not stalled when running within multiplier", () => {
assert.equal(isJobStalled("indexer_reindex_all_invalid", "running", minutesAgo(90), 60, NOW), false);
});
test("not stalled when status success", () => {
assert.equal(isJobStalled("indexer_reindex_all_invalid", "success", minutesAgo(500), 60, NOW), false);
});
test("not stalled when status error", () => {
assert.equal(isJobStalled("indexer_reindex_all_invalid", "error", minutesAgo(500), 60, NOW), false);
});
test("not stalled when status missed", () => {
assert.equal(isJobStalled("indexer_reindex_all_invalid", "missed", minutesAgo(500), 60, NOW), false);
});
test("not stalled when status pending", () => {
assert.equal(isJobStalled("indexer_reindex_all_invalid", "pending", null, 60, NOW), false);
});
test("not stalled when executedAt missing", () => {
assert.equal(isJobStalled("indexer_reindex_all_invalid", "running", null, 60, NOW), false);
});
test("custom stale multiplier widens the window", () => {
assert.equal(isJobStalled("indexer_reindex_all_invalid", "running", minutesAgo(200), 60, NOW, 5.0), false);
});Case studies
The custom job that died once and never came back
A merchant ran a custom cron job that synced order data to a third party warehouse system, processing a large batch each run. One night the batch was bigger than usual, the process hit the PHP memory limit, and it was killed. Every other cron job on the install, indexing, email queues, everything, kept firing normally. Only that one job code went silent, and nobody connected the dots for weeks because there was no error, just an absence.
Once the team added the detection job hourly, it flagged that exact job code as stalled on its first cycle, with the schedule id and the number of minutes it had been stuck on running. Marking the row missed and confirming the next tick generated a fresh run fixed it in minutes instead of weeks.
The indexer cron that lagged after every release
A deploy pipeline restarted the application containers as part of every release. Whenever that restart happened to land while indexer_reindex_all_invalid was mid run, its cron_schedule row was left on running, and reindexing silently stopped happening on schedule until someone eventually noticed stale prices and ran a manual reindex.
Adding this check as a post-deploy step caught the stalled job code right after each release, with the exact minutes since executed_at as evidence, so the on-call engineer had a confirmed lead instead of guessing which of dozens of job codes had actually stopped.
After this runs on a schedule, a crashed cron job is caught within one detection cycle instead of surviving silently for weeks while nothing about it looks like an error. The report carries the job code, its schedule id, and how long it has been stuck, so whoever responds can mark the row missed and confirm the next generation resumes with confidence. Keep the actual repair as a human or pipeline step gated on confirming the process is truly dead, since that is what keeps this from ever writing over a job that is legitimately still running.
FAQ
Why did Magento stop running one specific cron job entirely?
Magento's cron scheduler checks whether an existing cron_schedule row for a job code is still status running before it generates the next one. If that job's process was killed mid-execution, by an out of memory error, a PHP fatal, or a container restart, its row never flips to success or error, because that transition only happens when the job's own code finishes. Magento treats the stuck running row as still-active work and quietly declines to generate or dispatch the next run for that job code, while every other job code keeps running normally.
How do I tell which cron job is actually stuck versus just slow?
Query cron_schedule for rows with status running whose executed_at is far older than that job's own cron expression interval would allow, for example more than three times the expected cadence. A job that is merely slow will still be within a small multiple of its interval. A job whose running row is many intervals old, with no matching success or error row after it, is the crashed-lock signature.
Can a script fix a stuck cron job automatically over the REST API?
Not safely by itself. Magento does not expose cron_schedule through REST, since cron control is a CLI and database concern. The real repair, marking the stuck row missed or clearing it and running bin/magento cron:run, needs direct database or CLI access. A script can detect the stalled job code through REST symptoms and direct database reads and report it, gated behind a dry run flag that never issues the write on its own.
Related field notes
Citations
On the problem:
- GitHub Issue: cron job not running after crashed once. github.com/magento/magento2/issues/23054
- Magento Forums: cron jobs pile up and never stop running, thus crashing the server. community.magento.com cron jobs pile up
- Adobe Commerce: cron and index issues on Cloud Infrastructure. experienceleague.adobe.com cron and index issues
On the solution:
- Adobe Commerce: configure and run cron jobs. experienceleague.adobe.com configure and run cron jobs
- Adobe Commerce: cron (scheduled tasks) overview. experienceleague.adobe.com cron scheduled tasks
- Adobe Commerce: custom cron job and cron group reference. experienceleague.adobe.com custom cron reference
Stuck on a tricky one?
If you have a problem in Magento 2 or Adobe Commerce cron, indexing, catalog data, orders, or inventory that you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this clear your stuck job code?
If this saved you a silent cron outage or a confusing hunt through job codes, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi