Diagnostic Workflows & Background Jobs
Scheduled jobs run sporadically on Redis backed deploys
The job in src/jobs works every time on your laptop. In production, split across a server tier and a worker tier the way Railway and similar hosts expect, it runs most ticks fine and then, every so often, just fails. Redis ends up holding a failed BullMQ job with a stacktrace that says Workflow with id "<job-id>" not found, and nobody touched the code. Here is why the Redis backed workflow engine lets this happen, and a script that reads the queue directly and tells you which jobs are actually broken.
In Medusa v2, scheduled jobs run through @medusajs/workflow-engine-redis, which is backed by BullMQ queues in Redis. Before PR #11740, the module used one shared queue for both workflow transactions and scheduled jobs, so a server-mode instance, which never loads job or workflow definitions, could still dequeue a scheduled-job entry and throw Error: Workflow with id "<job-id>" not found, since only worker-mode instances register those workflows. Whichever instance happens to grab that tick decides whether the run fails, which is exactly the "runs fine locally, sporadic on a split server and worker deploy" symptom. A related cause, tracked in issue #14889, is a hanging or unbounded step inside a job's workflow, such as an unclosed stream, blocking the single default BullMQ worker, since jobWorkerOptions concurrency defaults to 1, stalling the whole job queue until the process restarts. Because Medusa has no Admin API for the job queue, detection connects to Redis directly with ioredis and bullmq's Queue class and inspects each job's state, stacktrace, and attempt count. Full code, tests, and a dry run guard are below.
The problem in plain words
A scheduled job in Medusa v2 is a file in src/jobs that exports a handler and a config.schedule cron string. Under the hood, that handler is wrapped in a workflow, and the Redis backed workflow engine module, @medusajs/workflow-engine-redis, is what actually queues the tick and hands it to a worker to execute using BullMQ.
The trouble starts on a deploy that splits the process in two, one instance answering HTTP as a server, one running background work as a worker, which is exactly how Railway and similar hosts recommend you scale Medusa. Before PR #11740, both the workflow transactions and the scheduled jobs shared one BullMQ queue. A server-mode instance never loads job or workflow definitions at all, since that is not its job. But it can still be the one that dequeues an entry from the shared queue when a tick fires, and when it tries to run a workflow it never registered, it throws Error: Workflow with id "<job-id>" not found. Whether that happens depends on which instance happens to pick up the job at that exact moment, so most ticks succeed, quietly, on the worker instance, and every so often one lands on the server instance instead and fails. That is the sporadic part.
Why it happens
A few distinct causes stack up to produce this same sporadic symptom:
- Prior to PR #11740,
@medusajs/workflow-engine-redisused one shared queue for both workflow transactions and scheduled jobs, so aserver-mode instance, which never registers job or workflow definitions, could still dequeue a scheduled-job entry. Onlyworker-mode instances register those workflows, so the server-mode dequeue throwsError: Workflow with id "<job-id>" not found. - Whether a given tick fails depends entirely on which instance's BullMQ worker happens to pick it up first, which is exactly why the failure comes and goes instead of happening every time, as reported in issue #11286 for Railway-style split deploys.
- A separate root cause, issue #14889, is a hanging or unbounded step inside a job's workflow, such as an unclosed stream, blocking the single default BullMQ worker.
jobWorkerOptionsconcurrency defaults to 1, so one stuck job stalls the entire job queue until the process is restarted. - Issue #8422 documents scheduled jobs failing specifically once the Redis workflow engine is in use, which is the general category this shared-queue bug and the hanging-step bug both fall under.
- The failed attempt does not vanish. It is persisted in Redis as a BullMQ job carrying its own stacktrace, so the evidence is there, it is just not visible anywhere in the Medusa Admin UI or API.
This is a common source of confusion because the code deployed is correct, the job definition is correct, and most runs succeed, so the instinct is to assume a flaky network or an unrelated timeout. See the citations at the end for the exact issues, the PR, and the docs.
Medusa gives you no Admin API route for the workflow or job queue, so you have to go to the same place the workflow engine itself reads and writes: Redis. Connect with ioredis and bullmq's own Queue class, using the same redisUrl or redisOptions the backend is configured with, and call getJobs for the failed, active, delayed, and waiting states. Each job's stacktrace, attemptsMade, and timestamps tell you exactly what happened without needing an execution history API that does not exist. Classify before you touch anything: a job failing with Workflow with id .* not found is the pre-#11740-style cross-mode dequeue bug, a job stuck active past its own step timeout with no finishedOn is the hanging-step bug from issue #14889, and only those two categories are candidates for a guarded cleanup.
The fix, as a flow
We do not touch the live scheduler and we do not blindly re-run anything. We connect straight to Redis with the same connection settings the Medusa backend uses, pull the job queue's entries across every relevant state, run each one through a pure classifier, and only under an explicit DRY_RUN=false flag remove the entries that are safely, provably orphaned.
Build it step by step
Point the script at the same Redis your backend uses
The job queue lives in the Redis instance your Medusa backend already points at for @medusajs/workflow-engine-redis. Use the exact same URL. Also set the step timeout you expect your workflow steps to respect, so a stuck job can be told apart from one that is merely slow.
pip install redis
export REDIS_URL="redis://localhost:6379"
export JOB_QUEUE_NAME="medusa-job-queue"
export STEP_TIMEOUT_MS="60000" # how long a step should ever take
export DRY_RUN="true" # start safe, only logs and classifiesnpm install ioredis bullmq
export REDIS_URL="redis://localhost:6379"
export JOB_QUEUE_NAME="medusa-job-queue"
export STEP_TIMEOUT_MS="60000" // how long a step should ever take
export DRY_RUN="true" // start safe, only logs and classifiesOpen the BullMQ job queue directly in Redis
PR #11740 split the transaction queue and the job queue apart, so scheduled jobs now live in their own dedicated jobQueueName, following the bull:<queueName> key convention BullMQ itself uses. Connect once and reuse the same queue handle for every read.
import os
import redis
REDIS_URL = os.environ.get("REDIS_URL", "redis://localhost:6379")
JOB_QUEUE_NAME = os.environ.get("JOB_QUEUE_NAME", "medusa-job-queue")
def get_redis_client():
return redis.Redis.from_url(REDIS_URL, decode_responses=True)
def queue_key(suffix):
# BullMQ's own key convention: bull:<queueName>:<suffix>
return f"bull:{JOB_QUEUE_NAME}:{suffix}"import IORedis from "ioredis";
import { Queue } from "bullmq";
const REDIS_URL = process.env.REDIS_URL || "redis://localhost:6379";
const JOB_QUEUE_NAME = process.env.JOB_QUEUE_NAME || "medusa-job-queue";
function getConnection() {
return new IORedis(REDIS_URL, { maxRetriesPerRequest: null });
}
function openJobQueue(connection) {
// PR #11740 split this out as a dedicated jobQueueName, separate
// from the workflow transaction queueName.
return new Queue(JOB_QUEUE_NAME, { connection });
}Pull jobs across every state that matters
A stuck or failed scheduled job can be sitting as failed, active, delayed, or waiting. Fetch all four in one pass so nothing is missed, and read back the fields the classifier needs: id, timestamps, attemptsMade, opts, and failedReason.
import json
STATES = ["failed", "active", "delayed", "waiting"]
def fetch_jobs(client):
"""Read BullMQ job hashes directly from Redis for every relevant state."""
jobs = []
for state in STATES:
job_ids = client.zrange(queue_key(state), 0, -1) if state in ("delayed",) \
else client.lrange(queue_key(state), 0, -1)
for job_id in job_ids:
data = client.hgetall(queue_key(job_id))
if not data:
continue
opts = json.loads(data.get("opts", "{}"))
jobs.append({
"id": data.get("name", job_id),
"timestamp": int(data.get("timestamp", 0)),
"processedOn": int(data["processedOn"]) if data.get("processedOn") else None,
"finishedOn": int(data["finishedOn"]) if data.get("finishedOn") else None,
"failedReason": data.get("failedReason"),
"attemptsMade": int(data.get("attemptsMade", 0)),
"opts": opts,
})
return jobsconst STATES = ["failed", "active", "delayed", "waiting"];
async function fetchJobs(jobQueue) {
// bullmq's own Queue.getJobs reads the same hashes this script would
// otherwise have to parse by hand from Redis.
const bullJobs = await jobQueue.getJobs(STATES);
return bullJobs.map((job) => ({
id: job.name || String(job.id),
timestamp: job.timestamp,
processedOn: job.processedOn ?? undefined,
finishedOn: job.finishedOn ?? undefined,
failedReason: job.failedReason,
attemptsMade: job.attemptsMade,
opts: { attempts: job.opts?.attempts },
}));
}Decide, with one pure function
Keep the classification in a function with no Redis, no network, and a clock passed in as an argument. Given a job and the current time, it returns one of five outcomes. This is the part we test in isolation, and it is the only place the sporadic failure gets turned into a clear label.
import re
NOT_FOUND_RE = re.compile(r"Workflow with id .* not found")
def classify_job(job, now, step_timeout_ms):
"""Pure: no I/O. job is a dict already read from Redis/BullMQ."""
failed_reason = job.get("failedReason")
if failed_reason and NOT_FOUND_RE.search(failed_reason):
return "orphaned-not-found"
processed_on = job.get("processedOn")
finished_on = job.get("finishedOn")
if processed_on is not None and finished_on is None and (now - processed_on) > step_timeout_ms:
return "stuck-active"
attempts_made = job.get("attemptsMade", 0)
attempts_allowed = (job.get("opts") or {}).get("attempts") or 1
if finished_on is None and attempts_made >= attempts_allowed and failed_reason:
return "exhausted-retries"
timestamp = job.get("timestamp")
if processed_on is None and timestamp is not None and (now - timestamp) > step_timeout_ms:
return "pending-too-long"
return "healthy"const NOT_FOUND_RE = /Workflow with id .* not found/;
export function classifyJob(job, now, stepTimeoutMs) {
// Pure: no I/O. job is a plain object already read from Redis/BullMQ.
const failedReason = job.failedReason;
if (failedReason && NOT_FOUND_RE.test(failedReason)) {
return "orphaned-not-found";
}
const { processedOn, finishedOn } = job;
if (processedOn !== undefined && finishedOn === undefined && now - processedOn > stepTimeoutMs) {
return "stuck-active";
}
const attemptsMade = job.attemptsMade ?? 0;
const attemptsAllowed = job.opts?.attempts ?? 1;
if (finishedOn === undefined && attemptsMade >= attemptsAllowed && failedReason) {
return "exhausted-retries";
}
if (processedOn === undefined && job.timestamp !== undefined && now - job.timestamp > stepTimeoutMs) {
return "pending-too-long";
}
return "healthy";
}Report every classified job, and only clean up what is provably orphaned
Log the id, the classification, the failure reason, and the matching schedule from src/jobs/<name>.ts for every job that is not healthy. Only when DRY_RUN is explicitly false, remove entries classified as orphaned-not-found or exhausted-retries, since those two are the ones that are safe to discard without risking a duplicate side effect. A stuck-active or pending-too-long job is left alone and reported, because removing it does not fix a hung process, it just hides the symptom.
REMOVABLE = {"orphaned-not-found", "exhausted-retries"}
def remove_job(client, job_id_key):
"""Delete the BullMQ job hash for a genuinely orphaned entry. Never call
this on stuck-active or pending-too-long jobs, since removing those
hides a hang instead of fixing it."""
client.delete(job_id_key)const REMOVABLE = new Set(["orphaned-not-found", "exhausted-retries"]);
async function removeJob(bullJob) {
// Delete a genuinely orphaned BullMQ job. Never call this on
// stuck-active or pending-too-long jobs, since removing those hides
// a hang instead of fixing it.
await bullJob.remove();
}Wire it together with a dry run guard
The loop ties every piece together. Leave DRY_RUN on for the first runs so you only see the classification report. Once you have confirmed which jobs are truly orphaned and, separately, confirmed your deploy is on a Medusa version with PR #11740 merged, with jobWorkerOptions.concurrency above 1 and explicit step timeouts, switch DRY_RUN off to let the script remove the orphaned entries.
Never let this script re-invoke a workflow automatically. Marking a job as reviewed is not the same as confirming its side effects did not partially apply, so a manual re-trigger by an operator, using medusa exec against the underlying workflow, is the only safe way to replay one. Always verify the upstream fixes first: PR #11740 merged, jobWorkerOptions.concurrency greater than 1, and explicit timeouts on every step.
The full code
Here is the complete script in one file for each language. It connects to the same Redis your Medusa backend uses, reads the job queue across every relevant BullMQ state, classifies each job with a pure function, always reports what it finds, and only under an explicit DRY_RUN=false removes entries that are genuinely orphaned.
"""Classify Medusa v2 scheduled job entries sitting in the Redis backed
BullMQ job queue, because a server-mode instance dequeued a job from a
shared queue before PR #11740 and threw Workflow with id not found, or
a hanging step stalled the single default BullMQ worker (issue #14889).
DRY_RUN=true only reports the classified jobs. Only removes entries
classified as genuinely orphaned, never re-runs a workflow.
"""
import os
import re
import json
import time
import logging
import redis
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("classify_stuck_jobs")
REDIS_URL = os.environ.get("REDIS_URL", "redis://localhost:6379")
JOB_QUEUE_NAME = os.environ.get("JOB_QUEUE_NAME", "medusa-job-queue")
STEP_TIMEOUT_MS = int(os.environ.get("STEP_TIMEOUT_MS", "60000"))
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"
STATES = ["failed", "active", "delayed", "waiting"]
NOT_FOUND_RE = re.compile(r"Workflow with id .* not found")
REMOVABLE = {"orphaned-not-found", "exhausted-retries"}
def get_redis_client():
return redis.Redis.from_url(REDIS_URL, decode_responses=True)
def queue_key(suffix):
# BullMQ's own key convention: bull:<queueName>:<suffix>
return f"bull:{JOB_QUEUE_NAME}:{suffix}"
def fetch_jobs(client):
"""Read BullMQ job hashes directly from Redis for every relevant state."""
jobs = []
for state in STATES:
job_ids = client.zrange(queue_key(state), 0, -1) if state == "delayed" \
else client.lrange(queue_key(state), 0, -1)
for job_id in job_ids:
data = client.hgetall(queue_key(job_id))
if not data:
continue
opts = json.loads(data.get("opts", "{}"))
jobs.append({
"redis_key": queue_key(job_id),
"id": data.get("name", job_id),
"timestamp": int(data.get("timestamp", 0)),
"processedOn": int(data["processedOn"]) if data.get("processedOn") else None,
"finishedOn": int(data["finishedOn"]) if data.get("finishedOn") else None,
"failedReason": data.get("failedReason"),
"attemptsMade": int(data.get("attemptsMade", 0)),
"opts": opts,
})
return jobs
def classify_job(job, now, step_timeout_ms):
"""Pure: no I/O. job is a dict already read from Redis/BullMQ."""
failed_reason = job.get("failedReason")
if failed_reason and NOT_FOUND_RE.search(failed_reason):
return "orphaned-not-found"
processed_on = job.get("processedOn")
finished_on = job.get("finishedOn")
if processed_on is not None and finished_on is None and (now - processed_on) > step_timeout_ms:
return "stuck-active"
attempts_made = job.get("attemptsMade", 0)
attempts_allowed = (job.get("opts") or {}).get("attempts") or 1
if finished_on is None and attempts_made >= attempts_allowed and failed_reason:
return "exhausted-retries"
timestamp = job.get("timestamp")
if processed_on is None and timestamp is not None and (now - timestamp) > step_timeout_ms:
return "pending-too-long"
return "healthy"
def remove_job(client, redis_key):
"""Delete the BullMQ job hash for a genuinely orphaned entry. Never call
this on stuck-active or pending-too-long jobs, since removing those
hides a hang instead of fixing it."""
client.delete(redis_key)
def run():
client = get_redis_client()
now = int(time.time() * 1000)
jobs = fetch_jobs(client)
flagged = []
for job in jobs:
classification = classify_job(job, now, STEP_TIMEOUT_MS)
if classification == "healthy":
continue
flagged.append((job, classification))
log.warning(
"Job %s classified %s. failedReason=%s attemptsMade=%s/%s",
job["id"], classification, job.get("failedReason"),
job.get("attemptsMade"), (job.get("opts") or {}).get("attempts"),
)
if not flagged:
log.info("No stuck or orphaned jobs across %d entr(y/ies).", len(jobs))
return
if not DRY_RUN:
for job, classification in flagged:
if classification in REMOVABLE:
log.info("Removing orphaned job %s (%s).", job["id"], classification)
remove_job(client, job["redis_key"])
log.info("Done. %d job(s) %s.", len(flagged), "to review" if DRY_RUN else "processed")
if __name__ == "__main__":
run()/**
* Classify Medusa v2 scheduled job entries sitting in the Redis backed
* BullMQ job queue, because a server-mode instance dequeued a job from a
* shared queue before PR #11740 and threw Workflow with id not found, or
* a hanging step stalled the single default BullMQ worker (issue #14889).
* DRY_RUN=true only reports the classified jobs. Only removes entries
* classified as genuinely orphaned, never re-runs a workflow.
*/
import { pathToFileURL } from "node:url";
import IORedis from "ioredis";
import { Queue } from "bullmq";
const REDIS_URL = process.env.REDIS_URL || "redis://localhost:6379";
const JOB_QUEUE_NAME = process.env.JOB_QUEUE_NAME || "medusa-job-queue";
const STEP_TIMEOUT_MS = Number(process.env.STEP_TIMEOUT_MS || 60000);
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
const STATES = ["failed", "active", "delayed", "waiting"];
const NOT_FOUND_RE = /Workflow with id .* not found/;
const REMOVABLE = new Set(["orphaned-not-found", "exhausted-retries"]);
export function classifyJob(job, now, stepTimeoutMs) {
// Pure: no I/O. job is a plain object already read from Redis/BullMQ.
const failedReason = job.failedReason;
if (failedReason && NOT_FOUND_RE.test(failedReason)) {
return "orphaned-not-found";
}
const { processedOn, finishedOn } = job;
if (processedOn !== undefined && finishedOn === undefined && now - processedOn > stepTimeoutMs) {
return "stuck-active";
}
const attemptsMade = job.attemptsMade ?? 0;
const attemptsAllowed = job.opts?.attempts ?? 1;
if (finishedOn === undefined && attemptsMade >= attemptsAllowed && failedReason) {
return "exhausted-retries";
}
if (processedOn === undefined && job.timestamp !== undefined && now - job.timestamp > stepTimeoutMs) {
return "pending-too-long";
}
return "healthy";
}
function getConnection() {
return new IORedis(REDIS_URL, { maxRetriesPerRequest: null });
}
function openJobQueue(connection) {
// PR #11740 split this out as a dedicated jobQueueName, separate
// from the workflow transaction queueName.
return new Queue(JOB_QUEUE_NAME, { connection });
}
async function fetchJobs(jobQueue) {
const bullJobs = await jobQueue.getJobs(STATES);
return bullJobs.map((job) => ({
bullJob: job,
id: job.name || String(job.id),
timestamp: job.timestamp,
processedOn: job.processedOn ?? undefined,
finishedOn: job.finishedOn ?? undefined,
failedReason: job.failedReason,
attemptsMade: job.attemptsMade,
opts: { attempts: job.opts?.attempts },
}));
}
async function removeJob(bullJob) {
// Delete a genuinely orphaned BullMQ job. Never call this on
// stuck-active or pending-too-long jobs, since removing those hides
// a hang instead of fixing it.
await bullJob.remove();
}
export async function run() {
const connection = getConnection();
const jobQueue = openJobQueue(connection);
const now = Date.now();
try {
const jobs = await fetchJobs(jobQueue);
const flagged = [];
for (const job of jobs) {
const classification = classifyJob(job, now, STEP_TIMEOUT_MS);
if (classification === "healthy") continue;
flagged.push({ job, classification });
console.warn(
`Job ${job.id} classified ${classification}. failedReason=${job.failedReason} attemptsMade=${job.attemptsMade}/${job.opts.attempts}`
);
}
if (flagged.length === 0) {
console.log(`No stuck or orphaned jobs across ${jobs.length} entr(y/ies).`);
return;
}
if (!DRY_RUN) {
for (const { job, classification } of flagged) {
if (REMOVABLE.has(classification)) {
console.log(`Removing orphaned job ${job.id} (${classification}).`);
await removeJob(job.bullJob);
}
}
}
console.log(`Done. ${flagged.length} job(s) ${DRY_RUN ? "to review" : "processed"}.`);
} finally {
await jobQueue.close();
connection.disconnect();
}
}
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((err) => { console.error(err); process.exit(1); });
}Add a test
The function worth testing is classify_job. It is pure, no Redis and no network, just plain job objects and a fixed clock, so the tests check every branch of the classification directly.
from classify_stuck_jobs import classify_job
NOW = 1_800_000_000_000
STEP_TIMEOUT_MS = 60_000
def job(**over):
base = {
"id": "job-transaction-sync-job",
"timestamp": NOW - 1_000,
"processedOn": None,
"finishedOn": None,
"failedReason": None,
"attemptsMade": 0,
"opts": {"attempts": 3},
}
base.update(over)
return base
def test_healthy_when_freshly_queued():
assert classify_job(job(), NOW, STEP_TIMEOUT_MS) == "healthy"
def test_orphaned_not_found_from_failed_reason():
j = job(failedReason='Error: Workflow with id "job-transaction-sync-job" not found')
assert classify_job(j, NOW, STEP_TIMEOUT_MS) == "orphaned-not-found"
def test_stuck_active_when_processed_but_not_finished_past_timeout():
j = job(processedOn=NOW - STEP_TIMEOUT_MS - 1)
assert classify_job(j, NOW, STEP_TIMEOUT_MS) == "stuck-active"
def test_not_stuck_active_when_within_timeout():
j = job(processedOn=NOW - 1_000)
assert classify_job(j, NOW, STEP_TIMEOUT_MS) == "healthy"
def test_exhausted_retries_when_attempts_used_up():
j = job(attemptsMade=3, opts={"attempts": 3}, failedReason="boom")
assert classify_job(j, NOW, STEP_TIMEOUT_MS) == "exhausted-retries"
def test_not_exhausted_when_finished_even_if_attempts_high():
j = job(attemptsMade=3, opts={"attempts": 3}, failedReason="boom", finishedOn=NOW)
assert classify_job(j, NOW, STEP_TIMEOUT_MS) == "healthy"
def test_pending_too_long_when_never_processed():
j = job(timestamp=NOW - STEP_TIMEOUT_MS - 1)
assert classify_job(j, NOW, STEP_TIMEOUT_MS) == "pending-too-long"
def test_not_found_takes_priority_over_other_signals():
j = job(
failedReason='Workflow with id "x" not found',
processedOn=NOW - STEP_TIMEOUT_MS - 1,
attemptsMade=3,
opts={"attempts": 3},
)
assert classify_job(j, NOW, STEP_TIMEOUT_MS) == "orphaned-not-found"import { test } from "node:test";
import assert from "node:assert/strict";
import { classifyJob } from "./classify-stuck-jobs.js";
const NOW = 1_800_000_000_000;
const STEP_TIMEOUT_MS = 60_000;
const job = (over = {}) => ({
id: "job-transaction-sync-job",
timestamp: NOW - 1_000,
processedOn: undefined,
finishedOn: undefined,
failedReason: undefined,
attemptsMade: 0,
opts: { attempts: 3 },
...over,
});
test("healthy when freshly queued", () => {
assert.equal(classifyJob(job(), NOW, STEP_TIMEOUT_MS), "healthy");
});
test("orphaned-not-found from failedReason", () => {
const j = job({ failedReason: 'Error: Workflow with id "job-transaction-sync-job" not found' });
assert.equal(classifyJob(j, NOW, STEP_TIMEOUT_MS), "orphaned-not-found");
});
test("stuck-active when processed but not finished past timeout", () => {
const j = job({ processedOn: NOW - STEP_TIMEOUT_MS - 1 });
assert.equal(classifyJob(j, NOW, STEP_TIMEOUT_MS), "stuck-active");
});
test("not stuck-active when within timeout", () => {
const j = job({ processedOn: NOW - 1_000 });
assert.equal(classifyJob(j, NOW, STEP_TIMEOUT_MS), "healthy");
});
test("exhausted-retries when attempts used up", () => {
const j = job({ attemptsMade: 3, opts: { attempts: 3 }, failedReason: "boom" });
assert.equal(classifyJob(j, NOW, STEP_TIMEOUT_MS), "exhausted-retries");
});
test("not exhausted when finished even if attempts high", () => {
const j = job({ attemptsMade: 3, opts: { attempts: 3 }, failedReason: "boom", finishedOn: NOW });
assert.equal(classifyJob(j, NOW, STEP_TIMEOUT_MS), "healthy");
});
test("pending-too-long when never processed", () => {
const j = job({ timestamp: NOW - STEP_TIMEOUT_MS - 1 });
assert.equal(classifyJob(j, NOW, STEP_TIMEOUT_MS), "pending-too-long");
});
test("not-found takes priority over other signals", () => {
const j = job({
failedReason: 'Workflow with id "x" not found',
processedOn: NOW - STEP_TIMEOUT_MS - 1,
attemptsMade: 3,
opts: { attempts: 3 },
});
assert.equal(classifyJob(j, NOW, STEP_TIMEOUT_MS), "orphaned-not-found");
});Case studies
The nightly sync that failed one night in five
A team ran their Medusa backend on Railway with two services, one set to server mode for the storefront API, one set to worker mode for background jobs, following the standard worker mode guidance. A nightly inventory sync job was scheduled at 2am. Most nights it completed on time. On some nights, seemingly at random, it failed with no output beyond a stacktrace no one was looking at.
Connecting to their Redis instance directly with the script here showed several past entries in the failed state whose failedReason matched Workflow with id "inventory-sync-job" not found. Every one of those failures traced back to a tick that had been dequeued by the server-mode service instead of the worker-mode one, exactly the pre-#11740 shared-queue behavior, and upgrading to a version with that fix merged stopped it from recurring.
The job queue that quietly stopped ticking
A store's scheduled export job opened a file stream to write a report and never closed it on one edge case input. With jobWorkerOptions concurrency left at its default of 1, that one hung job blocked every other scheduled job behind it in the same queue, and none of them ran again until the process was restarted.
Running the classifier against the job queue showed a single job stuck in the active state with no finishedOn, its processedOn timestamp far older than the step timeout, which the script labeled stuck-active. The team left that entry alone as instructed, fixed the stream handling and set an explicit step timeout, and raised jobWorkerOptions.concurrency above 1 so a single hung job could no longer stall the whole queue.
Run this classifier whenever a scheduled job's effects look inconsistent, and trust its report over guesswork. It never re-invokes a workflow and never removes a job unless that job is provably orphaned, so it can never double-run a side effect or paper over a real hang. The durable fix stays at the infrastructure level: confirm your Medusa version has PR #11740 merged so the job queue is separate from the transaction queue, set jobWorkerOptions.concurrency above 1, and give every workflow step an explicit timeout so one bad step can never again stall the entire schedule.
FAQ
Why do my Medusa scheduled jobs only fail sometimes after I moved to Redis?
Before a fix landed in PR #11740, workflow-engine-redis used one shared BullMQ queue for both workflow transactions and scheduled jobs. On a split deploy, a server-mode instance never loads job or workflow definitions, but it can still dequeue a scheduled-job entry from that shared queue, and then it throws Workflow with id not found because only worker-mode instances register those workflows. Which instance happens to grab the job each tick decides whether that run fails, which is exactly the sporadic pattern.
How do I check whether a Medusa scheduled job actually failed in Redis?
Medusa exposes no Admin API route for the workflow or job queue, so you connect to the same Redis instance directly with ioredis and bullmq, open the Queue by name, and call getJobs with the failed, active, delayed, and waiting states. Each job carries an id that matches the job's registered name, timestamps to compute how stuck it is, attemptsMade against opts.attempts, and a stacktrace array that holds the Workflow with id not found error or a hung-step timeout.
Is it safe to automatically remove or re-run a stuck scheduled job from Redis?
Not by default. Re-running a workflow blindly can double-run side effects such as inventory sync or order updates, so the safe pattern is to classify each job and only report it. An opt-in DRY_RUN=false cleanup can remove entries that are genuinely orphaned, meaning the not found failure or exhausted retries, while everything else is left for a human to confirm before manually re-invoking the workflow.
Related field notes
Citations
On the problem:
- medusajs/medusa GitHub issue #11286: Scheduled jobs run sporadically when deployed to Railway, and Redis instance stores failed job with error stacktrace. github.com/medusajs/medusa/issues/11286
- medusajs/medusa GitHub issue #8422: Scheduled Jobs fail when @medusajs/workflow-engine-redis is used. github.com/medusajs/medusa/issues/8422
- medusajs/medusa GitHub issue #14889: Medusa stops executing jobs after some point. github.com/medusajs/medusa/issues/14889
On the solution:
- Medusa Documentation: Redis Workflow Engine Module. docs.medusajs.com/resources/infrastructure-modules/workflow-engine/redis
- Medusa Documentation: Scheduled Jobs. docs.medusajs.com/learn/fundamentals/scheduled-jobs
- Medusa Documentation: Locking Operations in Workflows. docs.medusajs.com/learn/fundamentals/workflows/locks
Stuck on a tricky one?
If you have a problem in Medusa pricing, inventory, orders, promotions, or workflows that you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this explain a job that failed for no reason?
If this saved you from chasing a phantom Redis or network issue, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi