Reconciler Webhooks & Events
Queued events become permanently failed while app disabled
You disable an app, or Saleor's circuit breaker auto-disables it after a run of failed deliveries, expecting things to just pause. Then you re-enable it and new events flow fine, but a chunk of orders and other events from the disabled window are simply gone. Not retried, not queued, gone. Here is why Saleor fails those queued events instead of holding them, and a script that finds and safely retries the ones still worth saving.
Saleor's webhook dispatcher checks Webhook.isActive and the parent app's isActive at the moment a queued event is popped off the Celery task queue, not at the moment it was enqueued. Disable an app, whether by hand or through the circuit breaker after repeated delivery failures, and any event already sitting in the queue still gets picked up, sees the webhook or app is inactive, and is written to EventDelivery with status: FAILED, a terminal state with no attempt made and no automatic retry. Re-enabling the app only resumes delivery for events enqueued after that point. Run a small Python or Node.js script that lists the app's webhooks, pulls the FAILED deliveries that fall inside the disabled window, and retries each recoverable one with eventDeliveryRetry before EVENT_PAYLOAD_DELETE_PERIOD, 14 days by default, purges the payload. Full code, tests, and a dry run guard are below.
The problem in plain words
An app in Saleor has webhooks, and each webhook has queued events waiting for a Celery worker to actually deliver them. When you disable the app, or it disables itself through the circuit breaker after too many delivery failures, the natural assumption is that the queue just stops moving until you turn it back on.
That is not what happens. The events that were already sitting in the queue before the disable are still popped off by a worker. The worker checks whether the webhook and its app are active right then, at delivery time, not back when the event was enqueued. They are not active, so the worker writes an EventDelivery row with status: FAILED and moves on. No delivery attempt is made, no retry is scheduled, and the event is done. Turning the app back on does not touch that row. It only lets fresh events queue and deliver normally from that point forward.
Why it happens
- Saleor's webhook dispatcher reads
Webhook.isActiveand the parentApp.isActiveat task execution time, the moment a Celery worker pulls the event off the queue, not when the event was originally enqueued. - Disabling an app happens instantly from the merchant's side, but the queue can hold events from seconds or minutes earlier that have not been popped yet, and those get caught by the now-inactive check.
- Saleor's circuit breaker can auto-disable an app after a run of delivery failures to protect it from a struggling endpoint, and any events still queued at that moment fail the same way as a manual disable, often silently, since nobody was watching the toggle.
FAILEDis a terminalEventDeliveryStatusEnumvalue in Saleor's own schema. There is no background job that walks failed deliveries and reschedules them once the webhook becomes active again.
None of this raises an error a merchant would notice. The app dashboard shows active again, new webhooks fire, and the only trace of the gap is a set of EventDelivery rows sitting at FAILED for a window that is easy to miss unless you go looking. See the citations at the end for the exact docs and schema.
You cannot fix this by toggling the app faster or waiting for Saleor to notice. There is no automatic replay for a queued event that failed because the app was inactive when it was popped. The only reliable path is to find the app's disabled window, pull every FAILED EventDelivery created inside it, and retry each one yourself with eventDeliveryRetry while its payload still exists, since EVENT_PAYLOAD_DELETE_PERIOD purges it after 14 days by default.
The fix, as a flow
The script does not touch how webhooks fire. It lists the app and its webhooks, pulls the deliveries with status: FAILED, keeps only the ones whose createdAt falls inside the app's disabled-to-re-enabled window, and for each one checks whether the payload is still there. If it is, it retries. If the payload was already purged past the retention period, it flags that delivery for manual reconciliation instead, since nothing can safely replay it anymore.
Build it step by step
Get a token with MANAGE_APPS
Create a staff account token with tokenCreate, or use the app's own token, either way it needs the MANAGE_APPS permission to read app and webhook data and to call eventDeliveryRetry. Keep the API URL and token in environment variables, never in the file.
pip install requests
export SALEOR_API_URL="https://store.saleor.cloud/graphql/"
export SALEOR_AUTH_TOKEN="your-app-or-staff-token"
export APP_ID="gid://saleor/App/1"
export APP_DISABLED_AT="2026-06-25T09:00:00Z"
export APP_REENABLED_AT="2026-06-25T11:30:00Z"
export DRY_RUN="true" # start safe, change to false to write// Node 18+ has fetch built in, no dependencies needed
export SALEOR_API_URL="https://store.saleor.cloud/graphql/"
export SALEOR_AUTH_TOKEN="your-app-or-staff-token"
export APP_ID="gid://saleor/App/1"
export APP_DISABLED_AT="2026-06-25T09:00:00Z"
export APP_REENABLED_AT="2026-06-25T11:30:00Z"
export DRY_RUN="true" // start safe, change to false to writeTalk to the Saleor GraphQL API
Saleor is one GraphQL endpoint. Every call is a POST with a JSON body of {query, variables} and an Authorization: Bearer <token> header. A small helper sends a query and returns the data, raising if Saleor reports errors.
import os, requests
API_URL = os.environ["SALEOR_API_URL"]
TOKEN = os.environ["SALEOR_AUTH_TOKEN"]
def gql(query, variables=None):
r = requests.post(
API_URL,
json={"query": query, "variables": variables or {}},
headers={"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"},
timeout=30,
)
r.raise_for_status()
body = r.json()
if body.get("errors"):
raise RuntimeError(body["errors"])
return body["data"]const API_URL = process.env.SALEOR_API_URL;
const TOKEN = process.env.SALEOR_AUTH_TOKEN;
async function gql(query, variables = {}) {
const res = await fetch(API_URL, {
method: "POST",
headers: { Authorization: `Bearer ${TOKEN}`, "Content-Type": "application/json" },
body: JSON.stringify({ query, variables }),
});
if (!res.ok) throw new Error(`Saleor ${res.status}`);
const body = await res.json();
if (body.errors) throw new Error(JSON.stringify(body.errors));
return body.data;
}List the app's webhooks, then pull FAILED deliveries per webhook
Ask for the app and its webhooks first, then for each webhook page through eventDeliveries filtered to status: FAILED, reading back the id, timestamp, event type, and the payload from its recent delivery attempts so you know whether it is still there.
APP_WEBHOOKS_QUERY = """
query($appId: ID!) {
app(id: $appId) {
id
name
isActive
webhooks { id name isActive targetUrl }
}
}"""
FAILED_DELIVERIES_QUERY = """
query($webhookId: ID!, $after: String) {
webhook(id: $webhookId) {
eventDeliveries(first: 100, after: $after, filter: { status: FAILED }) {
pageInfo { hasNextPage endCursor }
edges {
node {
id
createdAt
eventType
status
payload
}
}
}
}
}"""
def app_webhooks(app_id):
data = gql(APP_WEBHOOKS_QUERY, {"appId": app_id})["app"]
return data["webhooks"] if data else []
def failed_deliveries(webhook_id):
cursor = None
while True:
data = gql(FAILED_DELIVERIES_QUERY, {"webhookId": webhook_id, "after": cursor})["webhook"]
for edge in data["eventDeliveries"]["edges"]:
yield edge["node"]
page = data["eventDeliveries"]["pageInfo"]
if not page["hasNextPage"]:
return
cursor = page["endCursor"]const APP_WEBHOOKS_QUERY = `
query($appId: ID!) {
app(id: $appId) {
id
name
isActive
webhooks { id name isActive targetUrl }
}
}`;
const FAILED_DELIVERIES_QUERY = `
query($webhookId: ID!, $after: String) {
webhook(id: $webhookId) {
eventDeliveries(first: 100, after: $after, filter: { status: FAILED }) {
pageInfo { hasNextPage endCursor }
edges {
node {
id
createdAt
eventType
status
payload
}
}
}
}
}`;
async function appWebhooks(appId) {
const data = (await gql(APP_WEBHOOKS_QUERY, { appId })).app;
return data ? data.webhooks : [];
}
async function* failedDeliveries(webhookId) {
let cursor = null;
while (true) {
const data = (await gql(FAILED_DELIVERIES_QUERY, { webhookId, after: cursor })).webhook;
for (const edge of data.eventDeliveries.edges) yield edge.node;
const page = data.eventDeliveries.pageInfo;
if (!page.hasNextPage) return;
cursor = page.endCursor;
}
}Decide, with one pure function
Keep the decision in its own function that takes a delivery, the disabled and re-enabled timestamps, the current time, and a retention window, and returns whether to retry, flag it as unrecoverable, or skip it. No I/O, so it is easy to test. Anything outside the disabled window or not FAILED is skipped. Inside the window, a null payload or one older than the retention period is unrecoverable. Everything else is safe to retry.
import datetime
RETENTION_DAYS_DEFAULT = 14
def _parse(iso):
return datetime.datetime.fromisoformat(iso.replace("Z", "+00:00"))
def classify_dropped_deliveries(deliveries, app_disabled_at, app_reenabled_at, now, retention_days=RETENTION_DAYS_DEFAULT):
window_start = _parse(app_disabled_at)
window_end = _parse(app_reenabled_at) if app_reenabled_at else _parse(now)
now_dt = _parse(now)
results = []
for d in deliveries:
if d["status"] != "FAILED":
results.append({"id": d["id"], "action": "SKIP"})
continue
created = _parse(d["createdAt"])
if created < window_start or created > window_end:
results.append({"id": d["id"], "action": "SKIP"})
continue
age_days = (now_dt - created).total_seconds() / 86400
if d.get("payload") is None or age_days > retention_days:
results.append({"id": d["id"], "action": "FLAG_UNRECOVERABLE"})
continue
results.append({"id": d["id"], "action": "RETRY"})
return resultsexport const RETENTION_DAYS_DEFAULT = 14;
export function classifyDroppedDeliveries(deliveries, appDisabledAt, appReenabledAt, now, retentionDays = RETENTION_DAYS_DEFAULT) {
const windowStart = new Date(appDisabledAt);
const windowEnd = new Date(appReenabledAt || now);
const nowDate = new Date(now);
return deliveries.map((d) => {
if (d.status !== "FAILED") return { id: d.id, action: "SKIP" };
const created = new Date(d.createdAt);
if (created < windowStart || created > windowEnd) return { id: d.id, action: "SKIP" };
const ageDays = (nowDate - created) / 86400000;
if (d.payload === null || d.payload === undefined || ageDays > retentionDays) {
return { id: d.id, action: "FLAG_UNRECOVERABLE" };
}
return { id: d.id, action: "RETRY" };
});
}Retry the recoverable deliveries one at a time
Saleor has no bulk-replay mutation, so the repair is a loop that calls eventDeliveryRetry with one delivery id at a time, reading back userErrors after each call. Anything classified as FLAG_UNRECOVERABLE is only logged, never retried, since its payload cannot be replayed.
RETRY_MUTATION = """
mutation($id: ID!) {
eventDeliveryRetry(id: $id) {
delivery { id status }
errors { field message code }
}
}"""
def retry_delivery(delivery_id):
result = gql(RETRY_MUTATION, {"id": delivery_id})["eventDeliveryRetry"]
if result["errors"]:
raise RuntimeError(result["errors"])
return result["delivery"]["status"]const RETRY_MUTATION = `
mutation($id: ID!) {
eventDeliveryRetry(id: $id) {
delivery { id status }
errors { field message code }
}
}`;
async function retryDelivery(deliveryId) {
const result = (await gql(RETRY_MUTATION, { id: deliveryId })).eventDeliveryRetry;
if (result.errors.length) throw new Error(JSON.stringify(result.errors));
return result.delivery.status;
}Wire it together with a dry run guard
The loop ties every piece together. Under DRY_RUN=true, the default, the script only prints the delivery ids and their target webhook and event type, split into what it would retry and what it would flag as unrecoverable. Read the output, agree with it, then switch DRY_RUN off to let it call eventDeliveryRetry for real. Run it once right after you notice a disable window, before the 14-day retention clock runs out.
Always start with DRY_RUN=true and read the printed list before writing anything. Never guess at a delivery whose payload is already null, past EVENT_PAYLOAD_DELETE_PERIOD, flag it for manual reconciliation, such as replaying the underlying business event through a re-sync script, instead of forcing a retry that Saleor cannot fulfill.
The full code
Here is the complete script in one file for each language. It reads settings from the environment, lists the app's webhooks, pulls FAILED deliveries in the disabled window, classifies each one, logs what it does, and respects the dry run flag before calling eventDeliveryRetry.
"""Find and retry Saleor EventDelivery rows that were marked FAILED because
the webhook or its app was disabled at the moment a queued event was popped
off the Celery task queue.
Saleor's dispatcher checks Webhook.isActive and App.isActive at task
execution time, not when the event was enqueued. Disabling an app, by hand
or through the circuit breaker after repeated delivery failures, does not
pause the queue: anything already queued still gets popped, sees the
webhook or app inactive, and is written to EventDelivery with status FAILED,
a terminal state with no attempt made and no automatic retry. Re-enabling
the app only resumes delivery for events enqueued after that point.
Saleor has no bulk-replay mutation, so repair is a guarded, per-delivery
retry loop over eventDeliveryRetry. Under DRY_RUN=true (the default) it
only prints the ids it would retry or flag. When DRY_RUN=false it calls
eventDeliveryRetry for real, but only for deliveries whose payload is
still inside EVENT_PAYLOAD_DELETE_PERIOD (14 days by default); anything
older or already purged is flagged for manual reconciliation instead.
Run it once per disabled window. Safe to run again and again.
"""
import os
import datetime
import logging
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("retry_dropped_events")
API_URL = os.environ["SALEOR_API_URL"]
TOKEN = os.environ["SALEOR_AUTH_TOKEN"]
APP_ID = os.environ.get("APP_ID", "gid://saleor/App/1")
APP_DISABLED_AT = os.environ.get("APP_DISABLED_AT", "1970-01-01T00:00:00Z")
APP_REENABLED_AT = os.environ.get("APP_REENABLED_AT") or None
RETENTION_DAYS = int(os.environ.get("EVENT_PAYLOAD_RETENTION_DAYS", "14"))
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"
APP_WEBHOOKS_QUERY = """
query($appId: ID!) {
app(id: $appId) {
id
name
isActive
webhooks { id name isActive targetUrl }
}
}"""
FAILED_DELIVERIES_QUERY = """
query($webhookId: ID!, $after: String) {
webhook(id: $webhookId) {
eventDeliveries(first: 100, after: $after, filter: { status: FAILED }) {
pageInfo { hasNextPage endCursor }
edges {
node {
id
createdAt
eventType
status
payload
}
}
}
}
}"""
RETRY_MUTATION = """
mutation($id: ID!) {
eventDeliveryRetry(id: $id) {
delivery { id status }
errors { field message code }
}
}"""
def gql(query, variables=None):
r = requests.post(
API_URL,
json={"query": query, "variables": variables or {}},
headers={"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"},
timeout=30,
)
r.raise_for_status()
body = r.json()
if body.get("errors"):
raise RuntimeError(body["errors"])
return body["data"]
def _parse(iso):
return datetime.datetime.fromisoformat(iso.replace("Z", "+00:00"))
def classify_dropped_deliveries(deliveries, app_disabled_at, app_reenabled_at, now, retention_days=RETENTION_DAYS):
window_start = _parse(app_disabled_at)
window_end = _parse(app_reenabled_at) if app_reenabled_at else _parse(now)
now_dt = _parse(now)
results = []
for d in deliveries:
if d["status"] != "FAILED":
results.append({"id": d["id"], "action": "SKIP"})
continue
created = _parse(d["createdAt"])
if created < window_start or created > window_end:
results.append({"id": d["id"], "action": "SKIP"})
continue
age_days = (now_dt - created).total_seconds() / 86400
if d.get("payload") is None or age_days > retention_days:
results.append({"id": d["id"], "action": "FLAG_UNRECOVERABLE"})
continue
results.append({"id": d["id"], "action": "RETRY"})
return results
def app_webhooks(app_id):
data = gql(APP_WEBHOOKS_QUERY, {"appId": app_id})["app"]
return data["webhooks"] if data else []
def failed_deliveries(webhook_id):
cursor = None
while True:
data = gql(FAILED_DELIVERIES_QUERY, {"webhookId": webhook_id, "after": cursor})["webhook"]
for edge in data["eventDeliveries"]["edges"]:
yield edge["node"]
page = data["eventDeliveries"]["pageInfo"]
if not page["hasNextPage"]:
return
cursor = page["endCursor"]
def retry_delivery(delivery_id):
result = gql(RETRY_MUTATION, {"id": delivery_id})["eventDeliveryRetry"]
if result["errors"]:
raise RuntimeError(result["errors"])
return result["delivery"]["status"]
def run():
now_iso = datetime.datetime.now(datetime.timezone.utc).isoformat()
webhooks = app_webhooks(APP_ID)
retried = 0
flagged = 0
for webhook in webhooks:
deliveries = list(failed_deliveries(webhook["id"]))
decisions = classify_dropped_deliveries(deliveries, APP_DISABLED_AT, APP_REENABLED_AT, now_iso, RETENTION_DAYS)
by_id = {d["id"]: d for d in deliveries}
for decision in decisions:
if decision["action"] == "SKIP":
continue
delivery = by_id[decision["id"]]
if decision["action"] == "FLAG_UNRECOVERABLE":
log.warning(
"UNRECOVERABLE webhook=%s eventType=%s id=%s createdAt=%s (payload purged past retention)",
webhook["name"], delivery["eventType"], delivery["id"], delivery["createdAt"],
)
flagged += 1
continue
log.info(
"RETRY webhook=%s eventType=%s id=%s createdAt=%s %s",
webhook["name"], delivery["eventType"], delivery["id"], delivery["createdAt"],
"would retry" if DRY_RUN else "retrying",
)
if not DRY_RUN:
retry_delivery(delivery["id"])
retried += 1
log.info(
"Done. %d delivery(ies) %s, %d flagged unrecoverable.",
retried, "to retry" if DRY_RUN else "retried", flagged,
)
if __name__ == "__main__":
run()/**
* Find and retry Saleor EventDelivery rows that were marked FAILED because
* the webhook or its app was disabled at the moment a queued event was
* popped off the Celery task queue.
*
* Saleor's dispatcher checks Webhook.isActive and App.isActive at task
* execution time, not when the event was enqueued. Disabling an app, by
* hand or through the circuit breaker after repeated delivery failures,
* does not pause the queue: anything already queued still gets popped,
* sees the webhook or app inactive, and is written to EventDelivery with
* status FAILED, a terminal state with no attempt made and no automatic
* retry. Re-enabling the app only resumes delivery for events enqueued
* after that point.
*
* Saleor has no bulk-replay mutation, so repair is a guarded, per-delivery
* retry loop over eventDeliveryRetry. Under DRY_RUN=true (the default) it
* only prints the ids it would retry or flag. When DRY_RUN=false it calls
* eventDeliveryRetry for real, but only for deliveries whose payload is
* still inside EVENT_PAYLOAD_DELETE_PERIOD (14 days by default); anything
* older or already purged is flagged for manual reconciliation instead.
* Run it once per disabled window.
*
* Guide: https://www.allanninal.dev/saleor/queued-events-fail-while-app-disabled/
*/
import { pathToFileURL } from "node:url";
const API_URL = process.env.SALEOR_API_URL || "https://store.saleor.cloud/graphql/";
const TOKEN = process.env.SALEOR_AUTH_TOKEN || "dummy-token";
const APP_ID = process.env.APP_ID || "gid://saleor/App/1";
const APP_DISABLED_AT = process.env.APP_DISABLED_AT || new Date().toISOString();
const APP_REENABLED_AT = process.env.APP_REENABLED_AT || null;
const RETENTION_DAYS = Number(process.env.EVENT_PAYLOAD_RETENTION_DAYS || 14);
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
export const RETENTION_DAYS_DEFAULT = 14;
export function classifyDroppedDeliveries(deliveries, appDisabledAt, appReenabledAt, now, retentionDays = RETENTION_DAYS_DEFAULT) {
const windowStart = new Date(appDisabledAt);
const windowEnd = new Date(appReenabledAt || now);
const nowDate = new Date(now);
return deliveries.map((d) => {
if (d.status !== "FAILED") return { id: d.id, action: "SKIP" };
const created = new Date(d.createdAt);
if (created < windowStart || created > windowEnd) return { id: d.id, action: "SKIP" };
const ageDays = (nowDate - created) / 86400000;
if (d.payload === null || d.payload === undefined || ageDays > retentionDays) {
return { id: d.id, action: "FLAG_UNRECOVERABLE" };
}
return { id: d.id, action: "RETRY" };
});
}
async function gql(query, variables = {}) {
const res = await fetch(API_URL, {
method: "POST",
headers: { Authorization: `Bearer ${TOKEN}`, "Content-Type": "application/json" },
body: JSON.stringify({ query, variables }),
});
if (!res.ok) throw new Error(`Saleor ${res.status}`);
const body = await res.json();
if (body.errors) throw new Error(JSON.stringify(body.errors));
return body.data;
}
const APP_WEBHOOKS_QUERY = `
query($appId: ID!) {
app(id: $appId) {
id
name
isActive
webhooks { id name isActive targetUrl }
}
}`;
const FAILED_DELIVERIES_QUERY = `
query($webhookId: ID!, $after: String) {
webhook(id: $webhookId) {
eventDeliveries(first: 100, after: $after, filter: { status: FAILED }) {
pageInfo { hasNextPage endCursor }
edges {
node {
id
createdAt
eventType
status
payload
}
}
}
}
}`;
const RETRY_MUTATION = `
mutation($id: ID!) {
eventDeliveryRetry(id: $id) {
delivery { id status }
errors { field message code }
}
}`;
async function appWebhooks(appId) {
const data = (await gql(APP_WEBHOOKS_QUERY, { appId })).app;
return data ? data.webhooks : [];
}
async function* failedDeliveries(webhookId) {
let cursor = null;
while (true) {
const data = (await gql(FAILED_DELIVERIES_QUERY, { webhookId, after: cursor })).webhook;
for (const edge of data.eventDeliveries.edges) yield edge.node;
const page = data.eventDeliveries.pageInfo;
if (!page.hasNextPage) return;
cursor = page.endCursor;
}
}
async function retryDelivery(deliveryId) {
const result = (await gql(RETRY_MUTATION, { id: deliveryId })).eventDeliveryRetry;
if (result.errors.length) throw new Error(JSON.stringify(result.errors));
return result.delivery.status;
}
export async function run() {
const nowIso = new Date().toISOString();
const webhooks = await appWebhooks(APP_ID);
let retried = 0;
let flagged = 0;
for (const webhook of webhooks) {
const deliveries = [];
for await (const node of failedDeliveries(webhook.id)) deliveries.push(node);
const decisions = classifyDroppedDeliveries(deliveries, APP_DISABLED_AT, APP_REENABLED_AT, nowIso, RETENTION_DAYS);
const byId = new Map(deliveries.map((d) => [d.id, d]));
for (const decision of decisions) {
if (decision.action === "SKIP") continue;
const delivery = byId.get(decision.id);
if (decision.action === "FLAG_UNRECOVERABLE") {
console.warn(
`UNRECOVERABLE webhook=${webhook.name} eventType=${delivery.eventType} id=${delivery.id} createdAt=${delivery.createdAt} (payload purged past retention)`
);
flagged++;
continue;
}
console.log(
`RETRY webhook=${webhook.name} eventType=${delivery.eventType} id=${delivery.id} createdAt=${delivery.createdAt} ${DRY_RUN ? "would retry" : "retrying"}`
);
if (!DRY_RUN) await retryDelivery(delivery.id);
retried++;
}
}
console.log(`Done. ${retried} delivery(ies) ${DRY_RUN ? "to retry" : "retried"}, ${flagged} flagged unrecoverable.`);
}
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((err) => { console.error(err); process.exit(1); });
}Add a test
The decision rule is the part most worth testing, because it decides which failed deliveries get a real retry versus a flag you cannot undo cheaply. Because classify_dropped_deliveries is pure, the test needs no network and no Saleor account. It just feeds in plain records with a fixed clock and checks the answer.
from retry_dropped_events import classify_dropped_deliveries
DISABLED_AT = "2026-06-25T09:00:00Z"
REENABLED_AT = "2026-06-25T11:30:00Z"
NOW = "2026-06-30T09:00:00Z"
def delivery(**over):
base = {
"id": "gid://saleor/EventDelivery/1",
"createdAt": "2026-06-25T10:00:00Z",
"status": "FAILED",
"eventType": "ORDER_CREATED",
"payload": "{\\"id\\": \\"gid://saleor/Order/1\\"}",
}
base.update(over)
return base
def test_before_window_is_skipped():
d = delivery(createdAt="2026-06-25T08:00:00Z")
result = classify_dropped_deliveries([d], DISABLED_AT, REENABLED_AT, NOW)
assert result == [{"id": d["id"], "action": "SKIP"}]
def test_inside_window_with_payload_is_retried():
d = delivery()
result = classify_dropped_deliveries([d], DISABLED_AT, REENABLED_AT, NOW)
assert result == [{"id": d["id"], "action": "RETRY"}]
def test_inside_window_past_retention_is_unrecoverable():
old_now = "2026-07-15T09:00:00Z" # more than 14 days after createdAt
d = delivery()
result = classify_dropped_deliveries([d], DISABLED_AT, REENABLED_AT, old_now)
assert result == [{"id": d["id"], "action": "FLAG_UNRECOVERABLE"}]
def test_inside_window_null_payload_is_unrecoverable():
d = delivery(payload=None)
result = classify_dropped_deliveries([d], DISABLED_AT, REENABLED_AT, NOW)
assert result == [{"id": d["id"], "action": "FLAG_UNRECOVERABLE"}]
def test_success_status_inside_window_is_skipped():
d = delivery(status="SUCCESS")
result = classify_dropped_deliveries([d], DISABLED_AT, REENABLED_AT, NOW)
assert result == [{"id": d["id"], "action": "SKIP"}]
def test_pending_status_inside_window_is_skipped():
d = delivery(status="PENDING")
result = classify_dropped_deliveries([d], DISABLED_AT, REENABLED_AT, NOW)
assert result == [{"id": d["id"], "action": "SKIP"}]
def test_after_window_is_skipped():
d = delivery(createdAt="2026-06-25T12:00:00Z")
result = classify_dropped_deliveries([d], DISABLED_AT, REENABLED_AT, NOW)
assert result == [{"id": d["id"], "action": "SKIP"}]
def test_no_reenabled_at_uses_now_as_window_end():
d = delivery(createdAt="2026-06-29T09:00:00Z")
result = classify_dropped_deliveries([d], DISABLED_AT, None, NOW)
assert result == [{"id": d["id"], "action": "RETRY"}]import { test } from "node:test";
import assert from "node:assert/strict";
import { classifyDroppedDeliveries } from "./retry-dropped-events.js";
const DISABLED_AT = "2026-06-25T09:00:00Z";
const REENABLED_AT = "2026-06-25T11:30:00Z";
const NOW = "2026-06-30T09:00:00Z";
const delivery = (over = {}) => ({
id: "gid://saleor/EventDelivery/1",
createdAt: "2026-06-25T10:00:00Z",
status: "FAILED",
eventType: "ORDER_CREATED",
payload: '{"id": "gid://saleor/Order/1"}',
...over,
});
test("before window is skipped", () => {
const d = delivery({ createdAt: "2026-06-25T08:00:00Z" });
const result = classifyDroppedDeliveries([d], DISABLED_AT, REENABLED_AT, NOW);
assert.deepEqual(result, [{ id: d.id, action: "SKIP" }]);
});
test("inside window with payload is retried", () => {
const d = delivery();
const result = classifyDroppedDeliveries([d], DISABLED_AT, REENABLED_AT, NOW);
assert.deepEqual(result, [{ id: d.id, action: "RETRY" }]);
});
test("inside window past retention is unrecoverable", () => {
const oldNow = "2026-07-15T09:00:00Z"; // more than 14 days after createdAt
const d = delivery();
const result = classifyDroppedDeliveries([d], DISABLED_AT, REENABLED_AT, oldNow);
assert.deepEqual(result, [{ id: d.id, action: "FLAG_UNRECOVERABLE" }]);
});
test("inside window null payload is unrecoverable", () => {
const d = delivery({ payload: null });
const result = classifyDroppedDeliveries([d], DISABLED_AT, REENABLED_AT, NOW);
assert.deepEqual(result, [{ id: d.id, action: "FLAG_UNRECOVERABLE" }]);
});
test("success status inside window is skipped", () => {
const d = delivery({ status: "SUCCESS" });
const result = classifyDroppedDeliveries([d], DISABLED_AT, REENABLED_AT, NOW);
assert.deepEqual(result, [{ id: d.id, action: "SKIP" }]);
});
test("pending status inside window is skipped", () => {
const d = delivery({ status: "PENDING" });
const result = classifyDroppedDeliveries([d], DISABLED_AT, REENABLED_AT, NOW);
assert.deepEqual(result, [{ id: d.id, action: "SKIP" }]);
});
test("after window is skipped", () => {
const d = delivery({ createdAt: "2026-06-25T12:00:00Z" });
const result = classifyDroppedDeliveries([d], DISABLED_AT, REENABLED_AT, NOW);
assert.deepEqual(result, [{ id: d.id, action: "SKIP" }]);
});
test("no reenabledAt uses now as window end", () => {
const d = delivery({ createdAt: "2026-06-29T09:00:00Z" });
const result = classifyDroppedDeliveries([d], DISABLED_AT, null, NOW);
assert.deepEqual(result, [{ id: d.id, action: "RETRY" }]);
});Case studies
A struggling endpoint took down two hours of order events
A fulfillment integration's endpoint had a brief outage during a deploy. Saleor's circuit breaker auto-disabled the app after enough failed deliveries in a row, which was the right call to stop hammering a dead endpoint. But the events already queued when the breaker tripped were popped shortly after, saw the app inactive, and were marked FAILED with nothing scheduled to retry them.
The team re-enabled the app once the deploy finished, and everything looked normal until reconciliation showed a two-hour gap of missing order events. Running the reconciler against that exact window found every FAILED delivery still inside the retention period and retried them cleanly, with the two truly stale ones flagged for a manual re-sync.
A maintenance window quietly dropped in-flight fulfillment events
A merchant disabled a shipping app for twenty minutes to swap API credentials, assuming events would just wait. Several fulfillment events that were already queued before the toggle got picked up during that window, saw the app inactive, and failed permanently. Re-enabling the app only let new events through, so shipment notifications for a handful of orders never reached the carrier integration.
A dry run right after the maintenance window listed exactly which deliveries had failed for that reason, all comfortably inside the 14-day retention period, and a live run retried every one of them, closing the gap with no manual order-by-order digging.
After this runs once for every disable window, whether it was a manual toggle or the circuit breaker tripping, a queued event that Saleor silently failed gets a real second chance instead of vanishing. The team gets the exact webhook, event type, and delivery id it touched, a clean flag for anything past the 14-day payload retention that genuinely cannot be recovered, and confidence that re-enabling an app is not secretly leaving a gap behind.
FAQ
Why did my Saleor webhook deliveries fail right when I disabled the app?
Saleor checks whether the webhook and its parent app are active at the moment a queued event is popped off the Celery task queue, not at the moment it was enqueued. Any event that was already sitting in the queue when you disabled the app is still picked up by a worker, sees the webhook or app is inactive, and is written to EventDelivery with status FAILED, a terminal state with no attempt made and no automatic retry scheduled.
Does re-enabling the app automatically retry the events that failed while it was disabled?
No. Re-enabling the app only resumes delivery for events enqueued after that point. Events already marked FAILED during the disabled window are never replayed automatically, so you have to find them and retry each one manually with the eventDeliveryRetry mutation before EVENT_PAYLOAD_DELETE_PERIOD, 14 days by default, purges their stored payload.
Can I bulk-retry every failed Saleor event delivery at once?
Saleor has no bulk-replay mutation. Repair is a guarded, per-delivery loop that calls eventDeliveryRetry with one delivery id at a time. Run it behind a dry run flag that only lists the ids and target webhook and event type it would retry, skip any delivery whose payload is already null past the retention period, and flag those for manual reconciliation instead.
Related field notes
Citations
On the problem:
- Saleor Commerce Documentation: Webhooks Troubleshooting. docs.saleor.io/developer/extending/webhooks/troubleshooting
- Saleor Commerce Documentation: Webhooks Overview. docs.saleor.io/developer/extending/webhooks/overview
- Webhooks Events Subscription. github.com/saleor/saleor/issues/2084
On the solution:
- Saleor GraphQL Schema: EventDelivery, EventDeliveryAttempt, EventDeliveryStatusEnum. raw.githubusercontent.com/saleor/saleor/main/saleor/graphql/schema.graphql
- Saleor Commerce Documentation: How to Update App Webhooks. docs.saleor.io/developer/extending/apps/updating-app-webhooks
- Saleor Commerce Documentation: Webhooks Troubleshooting. docs.saleor.io/developer/extending/webhooks/troubleshooting
Stuck on a tricky one?
If you have a problem in Saleor checkout, stock, channels, or fulfillment that you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this recover a dropped event for you?
If this saved you from silently missing orders or fulfillment events during an app outage, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi