Diagnostic Events & Notifications
Events fire before subscribers finish loading on boot
A redeploy or a horizontal-scale restart brings a Medusa instance back up, and for a brief window right after boot, an event that was already sitting in Redis gets processed with zero subscribers attached. No error, no retry, no trace. The webhook that should have fired never fires, the notification that should have gone out never goes out, and BullMQ has already marked the job complete. Here is why the Redis event bus can race ahead of your own subscribers, and a script that finds the gap in your boot logs and reports it safely.
In Medusa v2's Redis event bus, the event-bus-redis module and its BullMQ worker are instantiated and start consuming queued jobs as soon as the module loader resolves, but custom subscribers in src/subscribers are registered by a separate, later loader phase. If events were already queued in Redis before or during that window, for example from before a redeploy or a horizontal-scale restart, the worker dequeues them immediately, logs Processing <event> which has 0 subscribers, and marks the job complete, permanently losing that delivery. There is no subscriber-aware retry for a job BullMQ considers successfully processed. This is confirmed as a real bug in medusajs/medusa#10822. Detection is log correlation right after a restart, comparing the timestamp of each 0 subscribers line against the subscriber loader's own done marker. Full code, tests, and a dry run guard are below.
The problem in plain words
When a Medusa instance boots, several loader phases run in sequence to wire up the modules, the API routes, the workflows, and the subscribers. The Redis event bus is one of the earliest modules to come online, because it needs to be ready before almost anything else can emit an event. As soon as its loader resolves, its BullMQ worker attaches to the queue and starts pulling jobs, whether or not anything is listening for them yet.
Custom subscribers, the files you write in src/subscribers that call subscribe() for a given event name, are registered later, in their own loader phase. Between the moment the event bus worker starts consuming and the moment the subscriber loader finishes registering every handler, there is a real window where a dequeued job has nowhere to go. If Redis already had events queued from before the restart, perhaps because the previous process was mid-flight when it was killed for a deploy, or a second instance came up faster than expected during a scale-out, the worker happily processes them anyway. It logs that the event has zero subscribers, and because BullMQ only tracks whether the job function threw or completed, not whether any handler actually ran, it marks the job a success. The event is gone. Nothing will ever retry it.
Why it happens
This is a real ordering bug in the module loading sequence, not a configuration mistake in your project. A few things make it worse in production:
- The event-bus-redis module and its BullMQ worker are instantiated and begin consuming as soon as the module loader resolves, which happens well before the loader phase that registers
src/subscribersfiles finishes. - A redeploy or a horizontal-scale restart is exactly the moment this matters most, since Redis may already be holding events queued by the previous process, or by a sibling instance, that get dequeued the instant the new worker attaches.
- BullMQ has no concept of "zero subscribers ran," it only tracks whether the job function completed without throwing. Medusa's event bus does not throw in that case, it logs a warning and returns, so the job is marked successfully processed.
- Because the job is marked complete, there is no subscriber-aware retry mechanism to fall back on. The event does not go to a dead letter queue, it does not get flagged anywhere, it is simply gone.
- This is confirmed and tracked as
medusajs/medusa#10822: "Events should not be processed until all subscribers are loaded," and related community reports describe subscribers that appear to silently not fire around deploys and restarts.
This is a common source of confusion because everything else about the deploy looks completely healthy. The new instance boots, the health check passes, the admin and storefront routes all respond, and the only symptom is a handful of missing side effects clustered right around the restart timestamp. See the citations at the end for the exact issue and docs.
You cannot ask Medusa or Redis "which events were dropped," because BullMQ already recorded those jobs as complete. Detection has to happen by correlating your own boot log: find the timestamp where the subscriber loader finished registering every handler, then find every Processing <event> which has 0 subscribers line, and anything timestamped before the loader finished is a confirmed gap, not a guess. Repair is the same story as detection, careful and manual by default. Re-emitting an event can duplicate side effects, so the fix defaults to reporting the gap under DRY_RUN=true and only re-publishes when an operator has confirmed the handler is idempotent.
The fix, as a flow
We do not touch the event bus or try to reorder the loaders ourselves. We parse the boot log for the subscriber-loader-done marker and every 0 subscribers line, keep only the ones that happened strictly before the loader finished, and either report each gap or, under an explicit flag, re-emit the event from fresh Admin API state.
Build it step by step
Get an admin session and the base URL
Point the script at your Medusa backend and an admin user with rights to read the entities your subscribers act on. Exchange the email and password for a JWT once, then send it as a Bearer token on every admin call. Keep everything in environment variables, never hardcoded.
pip install requests
export MEDUSA_BACKEND_URL="http://localhost:9000"
export MEDUSA_ADMIN_EMAIL="admin@example.com"
export MEDUSA_ADMIN_PASSWORD="supersecret"
export BOOT_LOG_PATH="/var/log/medusa/boot.log"
export DRY_RUN="true" # start safe, only reports the confirmed gapsnpm install @medusajs/js-sdk
export MEDUSA_BACKEND_URL="http://localhost:9000"
export MEDUSA_ADMIN_EMAIL="admin@example.com"
export MEDUSA_ADMIN_PASSWORD="supersecret"
export BOOT_LOG_PATH="/var/log/medusa/boot.log"
export DRY_RUN="true" // start safe, only reports the confirmed gapsAuthenticate against the Admin API
Both languages exchange credentials for a token the same way. The Python version talks to the REST route directly with requests. The Node version uses the official @medusajs/js-sdk, which wraps the same login call.
import os, requests
BASE_URL = os.environ.get("MEDUSA_BACKEND_URL", "http://localhost:9000")
EMAIL = os.environ["MEDUSA_ADMIN_EMAIL"]
PASSWORD = os.environ["MEDUSA_ADMIN_PASSWORD"]
def get_token():
r = requests.post(
f"{BASE_URL}/auth/user/emailpass",
json={"email": EMAIL, "password": PASSWORD},
timeout=30,
)
r.raise_for_status()
return r.json()["token"]import Medusa from "@medusajs/js-sdk";
const BASE_URL = process.env.MEDUSA_BACKEND_URL || "http://localhost:9000";
const EMAIL = process.env.MEDUSA_ADMIN_EMAIL;
const PASSWORD = process.env.MEDUSA_ADMIN_PASSWORD;
const sdk = new Medusa({ baseUrl: BASE_URL, auth: { type: "jwt" } });
async function login() {
await sdk.auth.login("user", "emailpass", { email: EMAIL, password: PASSWORD });
return sdk;
}Parse the boot log for the two markers that matter
Medusa logs each subscriber file it registers, then a done marker once the loaders/subscribers phase finishes. The event-bus-redis processor logs a distinct line, Processing <event.name> which has 0 subscribers, whenever it dequeues a job with nothing listening. Read the log once per restart and pull out both kinds of line with their timestamps.
import re
from datetime import datetime, timezone
LOG_LINE = re.compile(r"^\[(?P<ts>[^\]]+)\]\s+(?P<msg>.*)$")
ZERO_SUB = re.compile(r"Processing\s+(?P<event>\S+)\s+which has 0 subscribers")
LOADER_DONE = re.compile(r"subscribers loaded", re.IGNORECASE)
def _to_epoch_ms(ts):
return datetime.fromisoformat(ts.replace("Z", "+00:00")).timestamp() * 1000
def parse_boot_log(path):
"""Read the boot log once. Returns (bootLog, subscriberLoaderDoneAtMs)."""
boot_log = []
loader_done_at_ms = None
with open(path, "r", encoding="utf-8") as f:
for line in f:
m = LOG_LINE.match(line.strip())
if not m:
continue
at_ms = _to_epoch_ms(m.group("ts"))
msg = m.group("msg")
zero = ZERO_SUB.search(msg)
if zero:
boot_log.append({"event": zero.group("event"), "atMs": at_ms})
elif LOADER_DONE.search(msg) and loader_done_at_ms is None:
loader_done_at_ms = at_ms
return boot_log, loader_done_at_msimport { readFileSync } from "node:fs";
const LOG_LINE = /^\[(?<ts>[^\]]+)\]\s+(?<msg>.*)$/;
const ZERO_SUB = /Processing\s+(?<event>\S+)\s+which has 0 subscribers/;
const LOADER_DONE = /subscribers loaded/i;
function toEpochMs(ts) {
return Date.parse(ts);
}
export function parseBootLog(path) {
// Read the boot log once. Returns { bootLog, subscriberLoaderDoneAtMs }.
const text = readFileSync(path, "utf-8");
const bootLog = [];
let subscriberLoaderDoneAtMs = null;
for (const raw of text.split("\n")) {
const line = raw.trim();
const m = line.match(LOG_LINE);
if (!m) continue;
const atMs = toEpochMs(m.groups.ts);
const msg = m.groups.msg;
const zero = msg.match(ZERO_SUB);
if (zero) {
bootLog.push({ event: zero.groups.event, atMs });
} else if (LOADER_DONE.test(msg) && subscriberLoaderDoneAtMs === null) {
subscriberLoaderDoneAtMs = atMs;
}
}
return { bootLog, subscriberLoaderDoneAtMs };
}Decide, with one pure function
Keep the decision in a function with no I/O. It takes the parsed boot log entries and the subscriber-loader-done timestamp, and returns every event that was processed strictly before the loader finished, each with how big the gap was. This is the exact question the whole fix is trying to answer, and it is fully testable with plain arrays and numbers, no log file and no Medusa instance required.
def find_missed_event_windows(boot_log, subscriber_loader_done_at_ms):
"""Pure: no I/O. boot_log is a list of {"event", "atMs"} dicts already parsed
from lines like "Processing <eventName> which has 0 subscribers".
An event was missed iff it was processed strictly before the subscriber
loader finished registering handlers."""
return [
{"event": e["event"], "atMs": e["atMs"], "gapMs": subscriber_loader_done_at_ms - e["atMs"]}
for e in boot_log
if e["atMs"] < subscriber_loader_done_at_ms
]export function findMissedEventWindows(bootLog, subscriberLoaderDoneAtMs) {
// Pure: no I/O. bootLog is a plain array of { event, atMs } already parsed
// from lines like "Processing <eventName> which has 0 subscribers".
// An event was missed iff it was processed strictly before the subscriber
// loader finished registering handlers.
return bootLog
.filter((e) => e.atMs < subscriberLoaderDoneAtMs)
.map((e) => ({ event: e.event, atMs: e.atMs, gapMs: subscriberLoaderDoneAtMs - e.atMs }));
}Cross-check business impact, then repair under a dry run guard
A confirmed gap is not yet proof of business impact on its own, so cross-check it against the Admin API. For an order.placed gap, diff /admin/orders around the restart timestamp against /admin/notifications for the same window, any order with no matching notification record is the observable symptom. Only with DRY_RUN=false, and only once the operator has confirmed the handler is idempotent, re-publish the event through the Event Module using data pulled fresh from the Admin API, never the original stale payload.
def orders_missing_notifications(token, restart_iso):
"""Diff orders created since the restart against notifications sent since
the restart, for order.placed style gaps. Returns order ids with no
matching notification record."""
headers = {"Authorization": f"Bearer {token}"}
orders = requests.get(
f"{BASE_URL}/admin/orders",
params={"fields": "id,status,*fulfillments,*payment_collection",
"created_at[$gte]": restart_iso},
headers=headers, timeout=30,
).json()["orders"]
notifications = requests.get(
f"{BASE_URL}/admin/notifications",
params={"fields": "id,to,template,data",
"created_at[$gte]": restart_iso},
headers=headers, timeout=30,
).json()["notifications"]
notified_order_ids = {n["data"].get("id") for n in notifications if n.get("data")}
return [o["id"] for o in orders if o["id"] not in notified_order_ids]
def reemit_order_placed(token, order_id):
"""Only called when DRY_RUN=false and the operator confirmed the handler
is idempotent. Sources fresh payload from the Admin API, not the
original stale event."""
headers = {"Authorization": f"Bearer {token}"}
order = requests.get(
f"{BASE_URL}/admin/orders/{order_id}",
params={"fields": "id,*items,*customer"},
headers=headers, timeout=30,
).json()["order"]
# In the Medusa backend process itself:
# const eventModuleService = container.resolve(Modules.EVENT)
# await eventModuleService.emit({ name: "order.placed", data: order })
return orderasync function ordersMissingNotifications(sdk, restartIso) {
// Diff orders created since the restart against notifications sent since
// the restart, for order.placed style gaps. Returns order ids with no
// matching notification record.
const { orders } = await sdk.client.fetch("/admin/orders", {
query: { fields: "id,status,*fulfillments,*payment_collection", "created_at[$gte]": restartIso },
});
const { notifications } = await sdk.client.fetch("/admin/notifications", {
query: { fields: "id,to,template,data", "created_at[$gte]": restartIso },
});
const notifiedOrderIds = new Set(notifications.map((n) => n.data?.id).filter(Boolean));
return orders.filter((o) => !notifiedOrderIds.has(o.id)).map((o) => o.id);
}
async function reemitOrderPlaced(sdk, orderId) {
// Only called when DRY_RUN=false and the operator confirmed the handler
// is idempotent. Sources fresh payload from the Admin API, not the
// original stale event.
const { order } = await sdk.client.fetch(`/admin/orders/${orderId}`, {
query: { fields: "id,*items,*customer" },
});
// In the Medusa backend process itself, inside a workflow:
// import { emitEventStep } from "@medusajs/medusa/core-flows"
// emitEventStep({ eventName: "order.placed", data: order })
return order;
}Wire it together with a dry run guard
The run loop parses the log once, finds every confirmed gap with the pure function, and reports each one with its event name and gap size. Leave DRY_RUN on for every early run, since the safe default is to only log and report for manual review. Only switch it off, per event, once you have both confirmed business impact through the Admin API cross-check and verified the handler tolerates being called twice.
Never auto-re-emit blindly. A duplicate order.placed can mean a duplicate confirmation email or a duplicate webhook call to a fulfillment partner. Keep DRY_RUN=true as the default, cross-check impact against the Admin API before touching anything, and only re-publish an event with data pulled fresh from the entity's current state, never the original stale payload that might now be out of date.
The full code
Here is the complete script in one file for each language. It parses the boot log, finds every event processed before the subscriber loader finished with a pure function, and either reports the batch or, once DRY_RUN is off, cross-checks business impact and re-emits from fresh Admin API state.
"""Find Medusa v2 events that the Redis event bus processed with 0
subscribers because the event-bus-redis module's BullMQ worker started
consuming before the later src/subscribers loader phase finished
registering handlers, typically right after a redeploy or a
horizontal-scale restart. Never auto-re-emits by default. DRY_RUN=true
only reports the confirmed gaps found in the boot log. Repair only
re-publishes an event under DRY_RUN=false, using data pulled fresh from
the Admin API, and only once the operator has confirmed the handler is
idempotent.
"""
import os
import re
import logging
from datetime import datetime
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("find_missed_events")
BASE_URL = os.environ.get("MEDUSA_BACKEND_URL", "http://localhost:9000")
EMAIL = os.environ.get("MEDUSA_ADMIN_EMAIL", "admin@example.com")
PASSWORD = os.environ.get("MEDUSA_ADMIN_PASSWORD", "supersecret")
BOOT_LOG_PATH = os.environ.get("BOOT_LOG_PATH", "/var/log/medusa/boot.log")
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"
LOG_LINE = re.compile(r"^\[(?P<ts>[^\]]+)\]\s+(?P<msg>.*)$")
ZERO_SUB = re.compile(r"Processing\s+(?P<event>\S+)\s+which has 0 subscribers")
LOADER_DONE = re.compile(r"subscribers loaded", re.IGNORECASE)
def get_token():
r = requests.post(
f"{BASE_URL}/auth/user/emailpass",
json={"email": EMAIL, "password": PASSWORD},
timeout=30,
)
r.raise_for_status()
return r.json()["token"]
def _to_epoch_ms(ts):
return datetime.fromisoformat(ts.replace("Z", "+00:00")).timestamp() * 1000
def parse_boot_log(path):
"""Read the boot log once. Returns (bootLog, subscriberLoaderDoneAtMs)."""
boot_log = []
loader_done_at_ms = None
with open(path, "r", encoding="utf-8") as f:
for line in f:
m = LOG_LINE.match(line.strip())
if not m:
continue
at_ms = _to_epoch_ms(m.group("ts"))
msg = m.group("msg")
zero = ZERO_SUB.search(msg)
if zero:
boot_log.append({"event": zero.group("event"), "atMs": at_ms})
elif LOADER_DONE.search(msg) and loader_done_at_ms is None:
loader_done_at_ms = at_ms
return boot_log, loader_done_at_ms
def find_missed_event_windows(boot_log, subscriber_loader_done_at_ms):
"""Pure: no I/O. boot_log is a list of {"event", "atMs"} dicts already parsed
from lines like "Processing <eventName> which has 0 subscribers".
An event was missed iff it was processed strictly before the subscriber
loader finished registering handlers."""
return [
{"event": e["event"], "atMs": e["atMs"], "gapMs": subscriber_loader_done_at_ms - e["atMs"]}
for e in boot_log
if e["atMs"] < subscriber_loader_done_at_ms
]
def orders_missing_notifications(token, restart_iso):
"""Diff orders created since the restart against notifications sent since
the restart, for order.placed style gaps. Returns order ids with no
matching notification record."""
headers = {"Authorization": f"Bearer {token}"}
orders = requests.get(
f"{BASE_URL}/admin/orders",
params={"fields": "id,status,*fulfillments,*payment_collection",
"created_at[$gte]": restart_iso},
headers=headers, timeout=30,
).json()["orders"]
notifications = requests.get(
f"{BASE_URL}/admin/notifications",
params={"fields": "id,to,template,data",
"created_at[$gte]": restart_iso},
headers=headers, timeout=30,
).json()["notifications"]
notified_order_ids = {n["data"].get("id") for n in notifications if n.get("data")}
return [o["id"] for o in orders if o["id"] not in notified_order_ids]
def reemit_order_placed(token, order_id):
"""Only called when DRY_RUN=false and the operator confirmed the handler
is idempotent. Sources fresh payload from the Admin API, not the
original stale event."""
headers = {"Authorization": f"Bearer {token}"}
order = requests.get(
f"{BASE_URL}/admin/orders/{order_id}",
params={"fields": "id,*items,*customer"},
headers=headers, timeout=30,
).json()["order"]
# In the Medusa backend process itself:
# const eventModuleService = container.resolve(Modules.EVENT)
# await eventModuleService.emit({ name: "order.placed", data: order })
return order
def run():
boot_log, loader_done_at_ms = parse_boot_log(BOOT_LOG_PATH)
if loader_done_at_ms is None:
log.warning("Subscriber loader done marker not found in %s. Nothing to compare.", BOOT_LOG_PATH)
return
missed = find_missed_event_windows(boot_log, loader_done_at_ms)
if not missed:
log.info("No confirmed gaps. %d event(s) processed, all after the subscriber loader finished.", len(boot_log))
return
for item in missed:
log.warning(
"Event %s processed %.0f ms before subscribers finished loading. Confirmed gap.",
item["event"], item["gapMs"],
)
if not DRY_RUN:
token = get_token()
restart_iso = datetime.utcfromtimestamp(min(e["atMs"] for e in missed) / 1000).isoformat() + "Z"
order_ids = orders_missing_notifications(token, restart_iso)
for order_id in order_ids:
log.info("Order %s has no matching notification. Re-emitting order.placed.", order_id)
reemit_order_placed(token, order_id)
log.info("Done. %d event(s) %s.", len(missed), "to review" if DRY_RUN else "reported and cross-checked")
if __name__ == "__main__":
run()/**
* Find Medusa v2 events that the Redis event bus processed with 0
* subscribers because the event-bus-redis module's BullMQ worker started
* consuming before the later src/subscribers loader phase finished
* registering handlers, typically right after a redeploy or a
* horizontal-scale restart. Never auto-re-emits by default. DRY_RUN=true
* only reports the confirmed gaps found in the boot log. Repair only
* re-publishes an event under DRY_RUN=false, using data pulled fresh from
* the Admin API, and only once the operator has confirmed the handler is
* idempotent.
*/
import { readFileSync } from "node:fs";
import { pathToFileURL } from "node:url";
const BASE_URL = process.env.MEDUSA_BACKEND_URL || "http://localhost:9000";
const EMAIL = process.env.MEDUSA_ADMIN_EMAIL || "admin@example.com";
const PASSWORD = process.env.MEDUSA_ADMIN_PASSWORD || "supersecret";
const BOOT_LOG_PATH = process.env.BOOT_LOG_PATH || "/var/log/medusa/boot.log";
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
const LOG_LINE = /^\[(?<ts>[^\]]+)\]\s+(?<msg>.*)$/;
const ZERO_SUB = /Processing\s+(?<event>\S+)\s+which has 0 subscribers/;
const LOADER_DONE = /subscribers loaded/i;
function toEpochMs(ts) {
return Date.parse(ts);
}
export function parseBootLog(path) {
// Read the boot log once. Returns { bootLog, subscriberLoaderDoneAtMs }.
const text = readFileSync(path, "utf-8");
const bootLog = [];
let subscriberLoaderDoneAtMs = null;
for (const raw of text.split("\n")) {
const line = raw.trim();
const m = line.match(LOG_LINE);
if (!m) continue;
const atMs = toEpochMs(m.groups.ts);
const msg = m.groups.msg;
const zero = msg.match(ZERO_SUB);
if (zero) {
bootLog.push({ event: zero.groups.event, atMs });
} else if (LOADER_DONE.test(msg) && subscriberLoaderDoneAtMs === null) {
subscriberLoaderDoneAtMs = atMs;
}
}
return { bootLog, subscriberLoaderDoneAtMs };
}
export function findMissedEventWindows(bootLog, subscriberLoaderDoneAtMs) {
// Pure: no I/O. bootLog is a plain array of { event, atMs } already parsed
// from lines like "Processing <eventName> which has 0 subscribers".
// An event was missed iff it was processed strictly before the subscriber
// loader finished registering handlers.
return bootLog
.filter((e) => e.atMs < subscriberLoaderDoneAtMs)
.map((e) => ({ event: e.event, atMs: e.atMs, gapMs: subscriberLoaderDoneAtMs - e.atMs }));
}
async function login() {
const { default: Medusa } = await import("@medusajs/js-sdk");
const sdk = new Medusa({ baseUrl: BASE_URL, auth: { type: "jwt" } });
await sdk.auth.login("user", "emailpass", { email: EMAIL, password: PASSWORD });
return sdk;
}
async function ordersMissingNotifications(sdk, restartIso) {
// Diff orders created since the restart against notifications sent since
// the restart, for order.placed style gaps. Returns order ids with no
// matching notification record.
const { orders } = await sdk.client.fetch("/admin/orders", {
query: { fields: "id,status,*fulfillments,*payment_collection", "created_at[$gte]": restartIso },
});
const { notifications } = await sdk.client.fetch("/admin/notifications", {
query: { fields: "id,to,template,data", "created_at[$gte]": restartIso },
});
const notifiedOrderIds = new Set(notifications.map((n) => n.data?.id).filter(Boolean));
return orders.filter((o) => !notifiedOrderIds.has(o.id)).map((o) => o.id);
}
async function reemitOrderPlaced(sdk, orderId) {
// Only called when DRY_RUN=false and the operator confirmed the handler
// is idempotent. Sources fresh payload from the Admin API, not the
// original stale event.
const { order } = await sdk.client.fetch(`/admin/orders/${orderId}`, {
query: { fields: "id,*items,*customer" },
});
// In the Medusa backend process itself, inside a workflow:
// import { emitEventStep } from "@medusajs/medusa/core-flows"
// emitEventStep({ eventName: "order.placed", data: order })
return order;
}
export async function run() {
const { bootLog, subscriberLoaderDoneAtMs } = parseBootLog(BOOT_LOG_PATH);
if (subscriberLoaderDoneAtMs === null) {
console.warn(`Subscriber loader done marker not found in ${BOOT_LOG_PATH}. Nothing to compare.`);
return;
}
const missed = findMissedEventWindows(bootLog, subscriberLoaderDoneAtMs);
if (missed.length === 0) {
console.log(`No confirmed gaps. ${bootLog.length} event(s) processed, all after the subscriber loader finished.`);
return;
}
for (const item of missed) {
console.warn(`Event ${item.event} processed ${Math.round(item.gapMs)} ms before subscribers finished loading. Confirmed gap.`);
}
if (!DRY_RUN) {
const sdk = await login();
const restartIso = new Date(Math.min(...missed.map((e) => e.atMs))).toISOString();
const orderIds = await ordersMissingNotifications(sdk, restartIso);
for (const orderId of orderIds) {
console.log(`Order ${orderId} has no matching notification. Re-emitting order.placed.`);
await reemitOrderPlaced(sdk, orderId);
}
}
console.log(`Done. ${missed.length} event(s) ${DRY_RUN ? "to review" : "reported and cross-checked"}.`);
}
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((err) => { console.error(err); process.exit(1); });
}Add a test
The function worth testing is the one that decides the outcome, find_missed_event_windows. It is pure, no log file and no network, just a comparison of timestamps already parsed into plain records, so the tests feed in a small boot log and a loader-done time and check the answer.
from find_missed_events import find_missed_event_windows
LOADER_DONE_AT_MS = 1_000_000
def entry(**over):
base = {"event": "order.placed", "atMs": 900_000}
base.update(over)
return base
def test_event_before_loader_done_is_missed():
result = find_missed_event_windows([entry()], LOADER_DONE_AT_MS)
assert len(result) == 1
assert result[0]["event"] == "order.placed"
assert result[0]["gapMs"] == 100_000
def test_event_after_loader_done_is_not_missed():
result = find_missed_event_windows([entry(atMs=1_100_000)], LOADER_DONE_AT_MS)
assert result == []
def test_event_exactly_at_loader_done_is_not_missed():
result = find_missed_event_windows([entry(atMs=LOADER_DONE_AT_MS)], LOADER_DONE_AT_MS)
assert result == []
def test_handles_multiple_events_independently():
early = entry(event="cart.completed", atMs=500_000)
late = entry(event="customer.created", atMs=1_200_000)
result = find_missed_event_windows([entry(), early, late], LOADER_DONE_AT_MS)
events = [r["event"] for r in result]
assert events == ["order.placed", "cart.completed"]
def test_empty_boot_log_returns_empty():
assert find_missed_event_windows([], LOADER_DONE_AT_MS) == []
def test_gap_ms_matches_the_difference_exactly():
result = find_missed_event_windows([entry(atMs=250_000)], LOADER_DONE_AT_MS)
assert result[0]["gapMs"] == 750_000import { test } from "node:test";
import assert from "node:assert/strict";
import { findMissedEventWindows } from "./find-missed-events.js";
const LOADER_DONE_AT_MS = 1_000_000;
const entry = (over = {}) => ({ event: "order.placed", atMs: 900_000, ...over });
test("event before loader done is missed", () => {
const result = findMissedEventWindows([entry()], LOADER_DONE_AT_MS);
assert.equal(result.length, 1);
assert.equal(result[0].event, "order.placed");
assert.equal(result[0].gapMs, 100_000);
});
test("event after loader done is not missed", () => {
const result = findMissedEventWindows([entry({ atMs: 1_100_000 })], LOADER_DONE_AT_MS);
assert.deepEqual(result, []);
});
test("event exactly at loader done is not missed", () => {
const result = findMissedEventWindows([entry({ atMs: LOADER_DONE_AT_MS })], LOADER_DONE_AT_MS);
assert.deepEqual(result, []);
});
test("handles multiple events independently", () => {
const early = entry({ event: "cart.completed", atMs: 500_000 });
const late = entry({ event: "customer.created", atMs: 1_200_000 });
const result = findMissedEventWindows([entry(), early, late], LOADER_DONE_AT_MS);
assert.deepEqual(result.map((r) => r.event), ["order.placed", "cart.completed"]);
});
test("empty boot log returns empty", () => {
assert.deepEqual(findMissedEventWindows([], LOADER_DONE_AT_MS), []);
});
test("gapMs matches the difference exactly", () => {
const result = findMissedEventWindows([entry({ atMs: 250_000 })], LOADER_DONE_AT_MS);
assert.equal(result[0].gapMs, 750_000);
});Case studies
The confirmation emails that never went out
A store rolled out a routine backend redeploy during a moderately busy hour. The old process was killed mid-request while a few checkouts were completing, leaving a handful of order.placed events queued in Redis. The new process came up healthy within seconds, passed its health check, and started serving traffic normally.
Three customers never received an order confirmation email. Nothing in the deploy pipeline flagged an issue, since every route responded correctly and no error was thrown anywhere. Correlating the new instance's boot log found three Processing order.placed which has 0 subscribers lines, all timestamped seconds before the subscribers loaded marker, exactly matching the three missing emails once cross-checked against /admin/notifications.
The new replica that raced its own siblings
A traffic spike triggered an autoscaler to add a second Medusa instance sharing the same Redis event bus as the first. The new replica's event-bus-redis module attached to the queue and began pulling jobs within a second of boot, while its own subscriber loader was still a few hundred milliseconds from finishing.
A handful of webhook-triggering events landed on the new replica during that window and were processed with zero subscribers before being marked complete. The team now runs this script against every instance's boot log immediately after any scale-out event, confirming there were exactly two missed events in the crossover window, both surfaced for manual review rather than silently lost.
Run this right after every redeploy or scale-out event, against the new instance's boot log. It never guesses, it only confirms a gap by comparing two timestamps that are already in your own logs, and it never re-emits anything unless you explicitly turn off DRY_RUN after confirming both business impact and handler idempotency. The durable fix stays structural: set WORKER_MODE=worker only on instances that come up after subscriber loaders resolve, or track Medusa issue #10822 for the upstream fix that gates event-bus-redis's queue consumption behind the subscriber-loader-done signal.
FAQ
Why does Medusa log 0 subscribers for an event right after a restart?
In Medusa v2's Redis event bus, the event-bus-redis module and its BullMQ worker start consuming queued jobs as soon as the module loader resolves, but custom subscribers in src/subscribers are registered by a separate, later loader phase. If an event was already queued in Redis before that later phase finishes, the worker dequeues it early, logs Processing which has 0 subscribers, and marks the job complete, so no subscriber ever runs for it.
How do I detect events that were lost to this boot race condition?
There is no API that shows missing events after the fact, since BullMQ marks the job complete even when zero subscribers ran. The practical method is log correlation right after a restart: capture the process start time, find the timestamp where the subscriber loader finishes registering handlers, then grep the same window for 0 subscribers lines. Any such line timestamped before the loader finished is a confirmed gap, which you can then cross-check against the Admin API, for example diffing /admin/orders against /admin/notifications for the affected window.
Is it safe to automatically re-emit a missed Medusa event?
Not by default, because re-emitting can duplicate side effects like emails or webhook calls if the original handler actually ran partially or if it is not idempotent. The safe pattern is DRY_RUN=true by default, which only logs and reports the gap for manual review. Only with DRY_RUN=false, after confirming the handler is idempotent, should you re-publish the event through the Event Module using fresh data pulled from the current Admin API state of the affected entity, not the original stale payload.
Related field notes
Citations
On the problem:
- medusajs/medusa GitHub issue #10822: event-bus-redis processor executes event before subscribers are loaded. github.com/medusajs/medusa/issues/10822
- Medusa Documentation: Subscribers Not Working, troubleshooting. docs.medusajs.com/resources/troubleshooting/subscribers/not-working
- medusajs/medusa GitHub issue #7850: events in Redis event bus not triggering subscribers (notification service). github.com/medusajs/medusa/issues/7850
On the solution:
- Medusa Documentation: Emit Workflow and Service Events. docs.medusajs.com/learn/fundamentals/events-and-subscribers/emit-event
- Medusa Documentation: emitEventStep, Core Workflows Reference. docs.medusajs.com/resources/references/medusa-workflows/steps/emitEventStep
- Medusa Documentation: Events and Subscribers. docs.medusajs.com/learn/fundamentals/events-and-subscribers
Stuck on a tricky one?
If you have a problem in Medusa pricing, inventory, orders, promotions, or workflows that you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this catch a silently lost event?
If this saved you from a swallowed notification or a webhook that never fired, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi