Diagnostic Workflows & Background Jobs
Scheduled jobs stop firing after long uptime
Everything ran fine for days. Then, sometime after a long stretch of uptime, every scheduled job in your Medusa store just stops. No crash in the logs, no error anywhere, the process is still up and answering requests. It just quietly stopped ticking. Here is why one stuck workflow step can silently freeze the entire scheduler, and a small monitor that catches it early instead of finding out from a customer.
Medusa v2's Redis workflow engine, @medusajs/medusa/workflow-engine-redis, runs every scheduled job as a BullMQ job on one shared queue, and by default jobWorkerOptions.concurrency is 1. If a scheduled job's workflow step hangs, for example an unbounded stream read, an await on a promise that never settles, or a stalled external call with no timeout, that single execution never completes and never fails out. It just sits in the only worker slot forever. Every later cron tick for every scheduled job in that queue is enqueued but never dequeued, so the whole scheduler looks like it silently died after some uptime. Nothing times the stuck job out on its own, so the only fix in practice is restarting the worker process. Give a heartbeat job a last_run_at marker readable over the Admin API, poll it, and compare the gap against the schedule times a tolerance to catch the stall early. Full code, tests, and a dry run guard are below.
The problem in plain words
A Medusa v2 scheduled job is a function and a cron schedule that Medusa wraps in a workflow and re-invokes on a timer. When you run it in production behind @medusajs/medusa/workflow-engine-redis, that timer does not fire the job directly. It enqueues a BullMQ job onto a queue, and a worker pulls jobs off that queue and executes them.
The queue has a worker pool, and by default that pool has room for exactly one job at a time. That is fine as long as every job finishes quickly. But if one scheduled job's workflow step hangs, an unbounded stream read that never closes, an await on a promise nothing ever resolves, a call to an external API with no timeout that never comes back, that execution never completes and never throws. BullMQ has no idea it is stuck. It is not crashed, it is not errored, it is just sitting there occupying the only worker slot, forever.
Why it happens
None of this is one single bug. It is a small set of defaults that combine badly the first time a workflow step misbehaves:
- The Redis workflow engine puts every scheduled job on one shared BullMQ queue, and
jobWorkerOptions.concurrencydefaults to 1, so only one scheduled job execution can run at any moment across your entire store. - Nothing in BullMQ or in Medusa's workflow engine puts a timeout on a workflow step by default. A step that awaits something that never settles just sits there. It is not failed, it is not retried, it does not emit an error event. It is simply still running, as far as the queue is concerned.
- Because concurrency is 1, that one stuck job blocks the worker from ever picking up the next job in the queue, including every future tick of every other scheduled job you have registered, not just the one that hung.
- There is no Admin API route to inspect or kill an individual BullMQ job from outside the process, so once a store is in this state, restarting the worker is the only way out, and that is exactly the workaround documented in medusajs/medusa issue #14889.
- This is easy to miss for a long time, because the symptom looks like nothing at all. The server keeps answering HTTP requests fine, since request handling is a separate code path from the job queue. The only sign is that things the scheduled jobs were supposed to do quietly stop happening, hours or days after the store was actually deployed.
This is a common source of confusion because the failure has no error to grep for. The scheduler was working. Uptime kept climbing. Then, at some point nobody can pin down after the fact, it stopped, and the only trace is silence. See the citations at the end for the exact issues and docs.
You cannot ask Medusa or BullMQ whether the scheduler is stuck, and there is no safe way to unstick it from outside the process. So the right shape for a fix is a heartbeat, not a repair. A dedicated, trivial scheduled job writes a timestamp somewhere you can read over the Admin API on every tick. If the gap since that timestamp grows past what the job's own schedule allows, times a tolerance, the scheduler has stalled, whether or not you can see why from outside. Detection has to be this indirect, because the alternative is waiting for a customer to notice a report never generated or an email that never sent.
The fix, as a flow
We do not touch the stuck workflow step and we do not try to reach into BullMQ. A heartbeat scheduled job writes last_run_at into a small piece of metadata every time it ticks. A separate monitor script polls that timestamp, computes how stale it is against the heartbeat's own cron schedule and a tolerance multiplier, and alerts when the scheduler looks stalled. It never tries to restart anything itself.
Build it step by step
Register a heartbeat scheduled job
Add a tiny scheduled job in src/jobs that does one thing: write the current time as ISO into metadata.heartbeat_last_run_at on a low-traffic record, such as a dedicated stock location, using the Store Module or a direct Admin API write. Keep it minimal on purpose so it is never the thing that hangs. Its only job is to prove the scheduler is still ticking.
pip install requests
export MEDUSA_BACKEND_URL="http://localhost:9000"
export MEDUSA_ADMIN_EMAIL="admin@example.com"
export MEDUSA_ADMIN_PASSWORD="supersecret"
export HEARTBEAT_CRON="*/5 * * * *" # must match the heartbeat job's own schedule
export TOLERANCE_MULTIPLIER="3"
export DRY_RUN="true" # flag only, there is no safe automatic repair// Node 18+ has fetch built in, no dependencies needed
export MEDUSA_BACKEND_URL="http://localhost:9000"
export MEDUSA_ADMIN_EMAIL="admin@example.com"
export MEDUSA_ADMIN_PASSWORD="supersecret"
export HEARTBEAT_CRON="*/5 * * * *" // must match the heartbeat job's own schedule
export TOLERANCE_MULTIPLIER="3"
export DRY_RUN="true" // flag only, there is no safe automatic repairAuthenticate against the Admin API
Exchange the admin email and password for a JWT once, then send it as a Bearer token on every request. The monitor is a read-only client, it never needs write scopes beyond what the heartbeat job itself already has.
import os, requests
BASE_URL = os.environ.get("MEDUSA_BACKEND_URL", "http://localhost:9000")
EMAIL = os.environ["MEDUSA_ADMIN_EMAIL"]
PASSWORD = os.environ["MEDUSA_ADMIN_PASSWORD"]
def get_token():
r = requests.post(
f"{BASE_URL}/auth/user/emailpass",
json={"email": EMAIL, "password": PASSWORD},
timeout=30,
)
r.raise_for_status()
return r.json()["token"]const BASE_URL = process.env.MEDUSA_BACKEND_URL || "http://localhost:9000";
const EMAIL = process.env.MEDUSA_ADMIN_EMAIL;
const PASSWORD = process.env.MEDUSA_ADMIN_PASSWORD;
async function getToken() {
const res = await fetch(`${BASE_URL}/auth/user/emailpass`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ email: EMAIL, password: PASSWORD }),
});
if (!res.ok) throw new Error(`Medusa auth ${res.status}`);
const body = await res.json();
return body.token;
}Read the heartbeat timestamp
Poll the record the heartbeat job writes to. A dedicated stock location's metadata works without any custom route, or expose a small GET /admin/heartbeat route if you would rather not repurpose an existing resource. Either way, all the script needs back is one ISO timestamp.
def get_heartbeat_last_run_at(token, stock_location_id):
headers = {"Authorization": f"Bearer {token}"}
r = requests.get(
f"{BASE_URL}/admin/stock-locations",
params={"fields": "id,metadata", "limit": 1},
headers=headers,
timeout=30,
)
r.raise_for_status()
locations = r.json()["stock_locations"]
for loc in locations:
if loc["id"] == stock_location_id:
iso = (loc.get("metadata") or {}).get("heartbeat_last_run_at")
return iso
return Noneasync function getHeartbeatLastRunAt(token, stockLocationId) {
const res = await fetch(
`${BASE_URL}/admin/stock-locations?fields=id,metadata&limit=1`,
{ headers: { Authorization: `Bearer ${token}` } }
);
if (!res.ok) throw new Error(`Medusa ${res.status}`);
const body = await res.json();
const loc = body.stock_locations.find((l) => l.id === stockLocationId);
return loc?.metadata?.heartbeat_last_run_at ?? null;
}Decide, with one pure function
Keep the decision in a function with no I/O at all. It takes the last heartbeat timestamp, the current time, the heartbeat job's own cron schedule, and a tolerance multiplier, works out the schedule's expected interval, and returns true only when the gap since the last heartbeat has passed that interval times the tolerance. A minute-level cron parser covers */N * * * * and every other standard field Medusa jobs use, so this stays correct without a dependency on an external cron library.
def is_scheduler_stalled(last_run_at, now, cron_schedule, tolerance_multiplier=3):
"""Pure: no I/O. Returns True iff the gap since last_run_at exceeds the
schedule's expected interval times tolerance_multiplier."""
interval_ms = expected_interval_ms(cron_schedule, last_run_at)
gap_ms = (now - last_run_at).total_seconds() * 1000
return gap_ms > interval_ms * tolerance_multiplierexport function isSchedulerStalled(lastRunAt, now, cronSchedule, toleranceMultiplier = 3) {
// Pure: no I/O. True iff the gap since lastRunAt exceeds the schedule's
// expected interval times toleranceMultiplier.
const intervalMs = expectedIntervalMs(cronSchedule, lastRunAt);
const gapMs = now.getTime() - lastRunAt.getTime();
return gapMs > intervalMs * toleranceMultiplier;
}Flag, do not repair
There is no /admin/jobs/:id/kill route and no /admin/scheduler/restart route in Medusa v2, so there is no safe Admin API action that unblocks a hung BullMQ worker slot from outside the process. When the pure function says the scheduler is stalled, the script's only job is to say so loudly: log it, and optionally call a push notification or post to an ops webhook, recommending an operator restart the worker instance running MEDUSA_WORKER_MODE=worker. DRY_RUN exists here purely to distinguish "just log" from "also call the alert webhook," since there is nothing this script can safely write to Medusa itself.
def alert_stalled_scheduler(gap_minutes, webhook_url=None):
message = (
f"Medusa scheduler looks stalled. No heartbeat for {gap_minutes:.1f} minutes. "
"A workflow step is likely stuck occupying the only BullMQ worker slot "
"(jobWorkerOptions.concurrency=1). Restart the worker process "
"(MEDUSA_WORKER_MODE=worker) to recover. Consider raising concurrency and "
"adding step-level timeouts to prevent this recurring."
)
if webhook_url:
requests.post(webhook_url, json={"text": message}, timeout=15)
return messageasync function alertStalledScheduler(gapMinutes, webhookUrl) {
const message =
`Medusa scheduler looks stalled. No heartbeat for ${gapMinutes.toFixed(1)} minutes. ` +
"A workflow step is likely stuck occupying the only BullMQ worker slot " +
"(jobWorkerOptions.concurrency=1). Restart the worker process " +
"(MEDUSA_WORKER_MODE=worker) to recover. Consider raising concurrency and " +
"adding step-level timeouts to prevent this recurring.";
if (webhookUrl) {
await fetch(webhookUrl, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ text: message }),
});
}
return message;
}Wire it together on a polling interval
The loop authenticates, reads the heartbeat timestamp, runs it through the pure decision function against the current time, and alerts when stalled. Run this on its own schedule, ideally outside the Medusa process itself, such as a small cron job or a monitoring service, since if the whole app is somehow down there is nothing left in-process to run this check.
This script never writes anything to Medusa and never tries to touch the stuck job or the queue. DRY_RUN=true only logs locally. Setting DRY_RUN=false additionally calls the alert webhook, since posting to an external ops channel is the only side effect this script is allowed to have. The actual fix is always a human restarting the worker process.
The full code
Here is the complete script in one file for each language. It authenticates, polls the heartbeat timestamp, runs the pure isSchedulerStalled decision against the heartbeat job's own cron schedule, and alerts an operator when the gap has grown past the tolerance, without ever attempting to touch the stuck job itself.
"""Detect a stalled Medusa v2 scheduler caused by a hung workflow step
occupying the only BullMQ worker slot (jobWorkerOptions.concurrency=1
by default on @medusajs/medusa/workflow-engine-redis). There is no
Admin API route that can kill a stuck job or restart the scheduler,
so this only flags the stall and alerts an operator to restart the
worker process. DRY_RUN=true only logs locally; DRY_RUN=false also
calls the alert webhook. Never writes anything to Medusa itself.
"""
import os
import logging
from datetime import datetime, timedelta, timezone
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("check_scheduler_heartbeat")
BASE_URL = os.environ.get("MEDUSA_BACKEND_URL", "http://localhost:9000")
EMAIL = os.environ.get("MEDUSA_ADMIN_EMAIL", "admin@example.com")
PASSWORD = os.environ.get("MEDUSA_ADMIN_PASSWORD", "supersecret")
STOCK_LOCATION_ID = os.environ.get("HEARTBEAT_STOCK_LOCATION_ID", "sloc_heartbeat")
HEARTBEAT_CRON = os.environ.get("HEARTBEAT_CRON", "*/5 * * * *")
TOLERANCE_MULTIPLIER = float(os.environ.get("TOLERANCE_MULTIPLIER", "3"))
ALERT_WEBHOOK_URL = os.environ.get("ALERT_WEBHOOK_URL")
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"
def get_token():
r = requests.post(
f"{BASE_URL}/auth/user/emailpass",
json={"email": EMAIL, "password": PASSWORD},
timeout=30,
)
r.raise_for_status()
return r.json()["token"]
def get_heartbeat_last_run_at(token, stock_location_id):
headers = {"Authorization": f"Bearer {token}"}
r = requests.get(
f"{BASE_URL}/admin/stock-locations",
params={"fields": "id,metadata", "limit": 100},
headers=headers,
timeout=30,
)
r.raise_for_status()
locations = r.json()["stock_locations"]
for loc in locations:
if loc["id"] == stock_location_id:
return (loc.get("metadata") or {}).get("heartbeat_last_run_at")
return None
def _parse_field(field, lo, hi):
"""Parse one cron field (*, N, N-M, N,M, */S) into a set of allowed ints."""
if field == "*":
return set(range(lo, hi + 1))
values = set()
for part in field.split(","):
if part.startswith("*/"):
values.update(range(lo, hi + 1, int(part[2:])))
elif "-" in part:
a, b = part.split("-")
values.update(range(int(a), int(b) + 1))
else:
values.add(int(part))
return values
def _parse_cron(cron_expression):
minute, hour, dom, month, dow = cron_expression.strip().split()
return {
"minute": _parse_field(minute, 0, 59),
"hour": _parse_field(hour, 0, 23),
"dom": _parse_field(dom, 1, 31),
"month": _parse_field(month, 1, 12),
"dow": _parse_field(dow, 0, 6),
}
def _matches(dt, spec):
if dt.minute not in spec["minute"] or dt.hour not in spec["hour"] or dt.month not in spec["month"]:
return False
dom_ok = dt.day in spec["dom"]
dow_ok = (dt.weekday() + 1) % 7 in spec["dow"]
if spec["dom"] != set(range(1, 32)) and spec["dow"] != set(range(0, 7)):
return dom_ok or dow_ok
return dom_ok and dow_ok
def _next_run_after(cron_expression, after, limit_minutes=527040):
"""Smallest minute-aligned datetime strictly after `after` matching the cron spec."""
spec = _parse_cron(cron_expression)
cursor = after.replace(second=0, microsecond=0) + timedelta(minutes=1)
for _ in range(limit_minutes):
if _matches(cursor, spec):
return cursor
cursor += timedelta(minutes=1)
raise RuntimeError("No matching run found within search window")
def expected_interval_ms(cron_expression, anchor):
"""The gap, in ms, between two consecutive matches of cron_expression near anchor."""
first_run = _next_run_after(cron_expression, anchor)
second_run = _next_run_after(cron_expression, first_run)
return (second_run - first_run).total_seconds() * 1000
def is_scheduler_stalled(last_run_at, now, cron_schedule, tolerance_multiplier=3):
"""Pure: no I/O. Returns True iff the gap since last_run_at exceeds the
schedule's expected interval times tolerance_multiplier."""
interval_ms = expected_interval_ms(cron_schedule, last_run_at)
gap_ms = (now - last_run_at).total_seconds() * 1000
return gap_ms > interval_ms * tolerance_multiplier
def alert_stalled_scheduler(gap_minutes, webhook_url=None):
message = (
f"Medusa scheduler looks stalled. No heartbeat for {gap_minutes:.1f} minutes. "
"A workflow step is likely stuck occupying the only BullMQ worker slot "
"(jobWorkerOptions.concurrency=1). Restart the worker process "
"(MEDUSA_WORKER_MODE=worker) to recover. Consider raising concurrency and "
"adding step-level timeouts to prevent this recurring."
)
if webhook_url:
requests.post(webhook_url, json={"text": message}, timeout=15)
return message
def run():
token = get_token()
last_run_iso = get_heartbeat_last_run_at(token, STOCK_LOCATION_ID)
now = datetime.now(timezone.utc)
if last_run_iso is None:
log.warning("No heartbeat recorded yet at all. Treating as stalled.")
message = alert_stalled_scheduler(float("inf"), ALERT_WEBHOOK_URL if not DRY_RUN else None)
log.warning(message)
return
last_run_at = datetime.fromisoformat(last_run_iso.replace("Z", "+00:00"))
stalled = is_scheduler_stalled(last_run_at, now, HEARTBEAT_CRON, TOLERANCE_MULTIPLIER)
gap_minutes = (now - last_run_at).total_seconds() / 60
if not stalled:
log.info("Scheduler healthy. Last heartbeat %.1f minute(s) ago.", gap_minutes)
return
log.warning("Scheduler stalled. Last heartbeat %.1f minute(s) ago.", gap_minutes)
message = alert_stalled_scheduler(gap_minutes, ALERT_WEBHOOK_URL if not DRY_RUN else None)
log.warning(message)
if __name__ == "__main__":
run()/**
* Detect a stalled Medusa v2 scheduler caused by a hung workflow step
* occupying the only BullMQ worker slot (jobWorkerOptions.concurrency=1
* by default on @medusajs/medusa/workflow-engine-redis). There is no
* Admin API route that can kill a stuck job or restart the scheduler,
* so this only flags the stall and alerts an operator to restart the
* worker process. DRY_RUN=true only logs locally; DRY_RUN=false also
* calls the alert webhook. Never writes anything to Medusa itself.
*/
import { pathToFileURL } from "node:url";
const BASE_URL = process.env.MEDUSA_BACKEND_URL || "http://localhost:9000";
const EMAIL = process.env.MEDUSA_ADMIN_EMAIL || "admin@example.com";
const PASSWORD = process.env.MEDUSA_ADMIN_PASSWORD || "supersecret";
const STOCK_LOCATION_ID = process.env.HEARTBEAT_STOCK_LOCATION_ID || "sloc_heartbeat";
const HEARTBEAT_CRON = process.env.HEARTBEAT_CRON || "*/5 * * * *";
const TOLERANCE_MULTIPLIER = Number(process.env.TOLERANCE_MULTIPLIER || 3);
const ALERT_WEBHOOK_URL = process.env.ALERT_WEBHOOK_URL || "";
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
function parseField(field, lo, hi) {
if (field === "*") return new Set(Array.from({ length: hi - lo + 1 }, (_, i) => lo + i));
const values = new Set();
for (const part of field.split(",")) {
if (part.startsWith("*/")) {
const step = Number(part.slice(2));
for (let v = lo; v <= hi; v += step) values.add(v);
} else if (part.includes("-")) {
const [a, b] = part.split("-").map(Number);
for (let v = a; v <= b; v++) values.add(v);
} else {
values.add(Number(part));
}
}
return values;
}
function parseCron(cronExpression) {
const [minute, hour, dom, month, dow] = cronExpression.trim().split(/\s+/);
return {
minute: parseField(minute, 0, 59),
hour: parseField(hour, 0, 23),
dom: parseField(dom, 1, 31),
month: parseField(month, 1, 12),
dow: parseField(dow, 0, 6),
};
}
function fullRange(lo, hi) {
return new Set(Array.from({ length: hi - lo + 1 }, (_, i) => lo + i));
}
function matches(date, spec) {
if (!spec.minute.has(date.getUTCMinutes())) return false;
if (!spec.hour.has(date.getUTCHours())) return false;
if (!spec.month.has(date.getUTCMonth() + 1)) return false;
const domOk = spec.dom.has(date.getUTCDate());
const dowOk = spec.dow.has(date.getUTCDay());
const domIsFull = spec.dom.size === fullRange(1, 31).size;
const dowIsFull = spec.dow.size === fullRange(0, 6).size;
if (!domIsFull && !dowIsFull) return domOk || dowOk;
return domOk && dowOk;
}
function nextRunAfter(cronExpression, after, limitMinutes = 527040) {
// Smallest minute-aligned date strictly after `after` matching the cron spec.
const spec = parseCron(cronExpression);
const cursor = new Date(after.getTime());
cursor.setUTCSeconds(0, 0);
cursor.setUTCMinutes(cursor.getUTCMinutes() + 1);
for (let i = 0; i < limitMinutes; i++) {
if (matches(cursor, spec)) return new Date(cursor.getTime());
cursor.setUTCMinutes(cursor.getUTCMinutes() + 1);
}
throw new Error("No matching run found within search window");
}
export function expectedIntervalMs(cronExpression, anchor) {
// The gap, in ms, between two consecutive matches of cronExpression near anchor.
const firstRun = nextRunAfter(cronExpression, anchor);
const secondRun = nextRunAfter(cronExpression, firstRun);
return secondRun.getTime() - firstRun.getTime();
}
export function isSchedulerStalled(lastRunAt, now, cronSchedule, toleranceMultiplier = 3) {
// Pure: no I/O. True iff the gap since lastRunAt exceeds the schedule's
// expected interval times toleranceMultiplier.
const intervalMs = expectedIntervalMs(cronSchedule, lastRunAt);
const gapMs = now.getTime() - lastRunAt.getTime();
return gapMs > intervalMs * toleranceMultiplier;
}
async function getToken() {
const res = await fetch(`${BASE_URL}/auth/user/emailpass`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ email: EMAIL, password: PASSWORD }),
});
if (!res.ok) throw new Error(`Medusa auth ${res.status}`);
const body = await res.json();
return body.token;
}
async function getHeartbeatLastRunAt(token, stockLocationId) {
const res = await fetch(
`${BASE_URL}/admin/stock-locations?fields=id,metadata&limit=100`,
{ headers: { Authorization: `Bearer ${token}` } }
);
if (!res.ok) throw new Error(`Medusa ${res.status}`);
const body = await res.json();
const loc = body.stock_locations.find((l) => l.id === stockLocationId);
return loc?.metadata?.heartbeat_last_run_at ?? null;
}
async function alertStalledScheduler(gapMinutes, webhookUrl) {
const message =
`Medusa scheduler looks stalled. No heartbeat for ${gapMinutes.toFixed(1)} minutes. ` +
"A workflow step is likely stuck occupying the only BullMQ worker slot " +
"(jobWorkerOptions.concurrency=1). Restart the worker process " +
"(MEDUSA_WORKER_MODE=worker) to recover. Consider raising concurrency and " +
"adding step-level timeouts to prevent this recurring.";
if (webhookUrl) {
await fetch(webhookUrl, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ text: message }),
});
}
return message;
}
export async function run() {
const token = await getToken();
const lastRunIso = await getHeartbeatLastRunAt(token, STOCK_LOCATION_ID);
const now = new Date();
if (!lastRunIso) {
console.warn("No heartbeat recorded yet at all. Treating as stalled.");
const message = await alertStalledScheduler(Infinity, DRY_RUN ? "" : ALERT_WEBHOOK_URL);
console.warn(message);
return;
}
const lastRunAt = new Date(lastRunIso);
const stalled = isSchedulerStalled(lastRunAt, now, HEARTBEAT_CRON, TOLERANCE_MULTIPLIER);
const gapMinutes = (now.getTime() - lastRunAt.getTime()) / 60000;
if (!stalled) {
console.log(`Scheduler healthy. Last heartbeat ${gapMinutes.toFixed(1)} minute(s) ago.`);
return;
}
console.warn(`Scheduler stalled. Last heartbeat ${gapMinutes.toFixed(1)} minute(s) ago.`);
const message = await alertStalledScheduler(gapMinutes, DRY_RUN ? "" : ALERT_WEBHOOK_URL);
console.warn(message);
}
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((err) => { console.error(err); process.exit(1); });
}Add a test
The function worth testing is isSchedulerStalled. It is pure, no network and no Medusa instance, just date math against a tiny cron parser, so the tests feed in a fixed clock, a fixed schedule, and a last run timestamp, then check the answer.
from datetime import datetime, timedelta, timezone
from check_scheduler_heartbeat import is_scheduler_stalled
NOW = datetime(2026, 7, 10, 0, 0, 0, tzinfo=timezone.utc)
EVERY_MINUTE = "* * * * *"
def test_not_stalled_when_heartbeat_is_recent():
last_run = NOW - timedelta(seconds=30)
assert is_scheduler_stalled(last_run, NOW, EVERY_MINUTE, 3) is False
def test_not_stalled_right_at_the_tolerance_boundary():
# 60_000ms interval * 3 tolerance = 180s; 179s of silence is still healthy
last_run = NOW - timedelta(seconds=179)
assert is_scheduler_stalled(last_run, NOW, EVERY_MINUTE, 3) is False
def test_stalled_after_twenty_minutes_of_silence_with_default_tolerance():
last_run = NOW - timedelta(minutes=20)
assert is_scheduler_stalled(last_run, NOW, EVERY_MINUTE, 3) is True
def test_higher_tolerance_delays_the_stalled_verdict():
last_run = NOW - timedelta(minutes=4)
assert is_scheduler_stalled(last_run, NOW, EVERY_MINUTE, 3) is True
assert is_scheduler_stalled(last_run, NOW, EVERY_MINUTE, 10) is False
def test_works_with_a_five_minute_schedule():
every_five = "*/5 * * * *"
healthy = NOW - timedelta(minutes=6)
stalled = NOW - timedelta(minutes=20)
assert is_scheduler_stalled(healthy, NOW, every_five, 3) is False
assert is_scheduler_stalled(stalled, NOW, every_five, 3) is Trueimport { test } from "node:test";
import assert from "node:assert/strict";
import { isSchedulerStalled } from "./check-scheduler-heartbeat.js";
const NOW = new Date("2026-07-10T00:00:00Z");
const EVERY_MINUTE = "* * * * *";
test("not stalled when heartbeat is recent", () => {
const lastRun = new Date(NOW.getTime() - 30_000);
assert.equal(isSchedulerStalled(lastRun, NOW, EVERY_MINUTE, 3), false);
});
test("not stalled right at the tolerance boundary", () => {
// 60_000ms interval * 3 tolerance = 180s; 179s of silence is still healthy
const lastRun = new Date(NOW.getTime() - 179_000);
assert.equal(isSchedulerStalled(lastRun, NOW, EVERY_MINUTE, 3), false);
});
test("stalled after twenty minutes of silence with default tolerance", () => {
const lastRun = new Date(NOW.getTime() - 20 * 60_000);
assert.equal(isSchedulerStalled(lastRun, NOW, EVERY_MINUTE, 3), true);
});
test("higher tolerance delays the stalled verdict", () => {
const lastRun = new Date(NOW.getTime() - 4 * 60_000);
assert.equal(isSchedulerStalled(lastRun, NOW, EVERY_MINUTE, 3), true);
assert.equal(isSchedulerStalled(lastRun, NOW, EVERY_MINUTE, 10), false);
});
test("works with a five minute schedule", () => {
const everyFive = "*/5 * * * *";
const healthy = new Date(NOW.getTime() - 6 * 60_000);
const stalled = new Date(NOW.getTime() - 20 * 60_000);
assert.equal(isSchedulerStalled(healthy, NOW, everyFive, 3), false);
assert.equal(isSchedulerStalled(stalled, NOW, everyFive, 3), true);
});Case studies
The nightly report job that hung on a payment provider outage
A store ran a nightly reporting job that called out to a payment provider's reporting endpoint with no timeout set on the request. One night the provider's endpoint accepted the connection but never responded and never closed it. The job's workflow step just sat there, waiting forever.
Every scheduled job after that, price syncs, abandoned cart reminders, inventory reconciliation, silently stopped firing, because the one worker slot never freed up. Nobody noticed for four days, until a marketing report came back empty. A heartbeat monitor polling every ten minutes would have flagged the stall within thirty minutes of the first missed tick.
The CSV export job that never saw end of stream
A custom scheduled job piped a large product export to a remote storage bucket using a stream that, under a specific network condition, never emitted its final end event. The step awaited the stream's completion with no fallback timeout, so it hung indefinitely once that condition hit.
The team had already deployed the heartbeat pattern from this note as a matter of habit. The monitor's alert fired within one polling cycle of the tolerance being exceeded, an operator restarted the worker process, and the fix that followed added a hard timeout around the stream await so the same stall could not recur.
A heartbeat job this small should never be the thing that hangs, so a stalled heartbeat is strong evidence something else in the queue is stuck, not a false alarm. The monitor never guesses at a fix and never touches the queue, it only tells an operator, quickly, that a restart of the worker process is needed. The durable fix lives in the Medusa config itself: raise jobWorkerOptions.concurrency above 1 so one hung step cannot block every other scheduled job, and wrap the specific workflow step that can hang, the stream read, the unbounded await, the untimed external call, in its own timeout so it fails loudly instead of sitting forever.
FAQ
Why do all my Medusa scheduled jobs stop firing after the app has been up for a while?
Medusa v2's Redis workflow engine runs scheduled jobs as BullMQ jobs on one shared queue with a default concurrency of 1. If any scheduled job's workflow step hangs and never completes or fails, that single execution occupies the only worker slot forever. Every later cron tick, for every scheduled job on that queue, gets enqueued but never dequeued, so the whole scheduler looks like it silently died.
How do I detect that Medusa's scheduler has stalled?
Give a lightweight heartbeat job a body that writes a last_run_at timestamp somewhere readable over the Admin API, such as metadata on a low-traffic stock location. Poll that timestamp on an interval, compare the gap against the heartbeat job's own cron schedule times a tolerance multiplier, and treat the scheduler as stalled once the gap exceeds that threshold.
Can a script automatically fix a stalled Medusa scheduler?
No. There is no Admin API route that kills a stuck BullMQ job or restarts the scheduler from outside the process, so the only safe action for a script is to flag the stall and alert an operator to restart the worker process. The durable fix is raising jobWorkerOptions.concurrency and adding step-level timeouts to the workflow step that can hang.
Related field notes
Citations
On the problem:
- medusajs/medusa GitHub issue #14889: Medusa stops executing jobs after some point. github.com/medusajs/medusa/issues/14889
- medusajs/medusa GitHub issue #8422: Scheduled Jobs fail when @medusajs/workflow-engine-redis is used. github.com/medusajs/medusa/issues/8422
- Medusa Documentation: Scheduled Job Not Running on Schedule. docs.medusajs.com/resources/troubleshooting/scheduled-job-not-running
On the solution:
- Medusa Documentation: Scheduled Jobs. docs.medusajs.com/learn/fundamentals/scheduled-jobs
- Medusa Documentation: Redis Workflow Engine Module. docs.medusajs.com/resources/infrastructure-modules/workflow-engine/redis
- Medusa Documentation: Workflow Engine Module. docs.medusajs.com/resources/infrastructure-modules/workflow-engine
Stuck on a tricky one?
If you have a problem in Medusa pricing, inventory, orders, promotions, or workflows that you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this catch a silent stall before a customer did?
If this saved you from a scheduler that quietly died for days, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi