Repair Webhooks & Events
Webhook deliveries stuck failed past retry limit
Your endpoint had a bad five minutes. A deploy, a cold start, a database blip, something transient. Saleor tried to deliver the event, failed, backed off, and tried again a few more times. Then it stopped trying. The EventDelivery sits there marked FAILED forever, and nothing in Saleor will ever touch it again on its own. Here is why Saleor gives up for good and a small script that finds the ones worth a second chance.
Saleor's send_webhook_request Celery task, in saleor/plugins/webhook/tasks.py, retries an async webhook delivery with retry_backoff=10 and retry_kwargs={"max_retries": 5}, roughly 10 times 2 to the n seconds of delay per attempt. Once the fifth retry also fails, Celery stops rescheduling the task entirely and Saleor persists the EventDelivery's final status as FAILED. There is no scheduled job that comes back for it later. Run a small Python or Node.js script that lists FAILED deliveries with their attempts, keeps only the ones that already hit 5 attempts and are old enough that Saleor's own backoff window has definitely closed, and for those calls the eventDeliveryRetry mutation one at a time, but only when the last few attempts look transient rather than a permanently broken endpoint. Full code, tests, and a dry run guard are below.
The problem in plain words
When Saleor fires an async event such as ORDER_CREATED or PRODUCT_UPDATED, it hands the delivery to a Celery task, send_webhook_request, which POSTs the payload to your app's target URL. If your endpoint answers with a 5xx status, times out, or is simply unreachable, Saleor does not give up right away. It retries, with the delay between attempts growing each time.
But that patience has a hard ceiling. After the fifth retry also fails, Celery's own max_retries setting kicks in and the task is never rescheduled again. Saleor writes the final status on the EventDelivery as FAILED and moves on. If your endpoint was only down for the length of a deploy, a cold start, or a brief 5xx, the event that would have told your system about that order or that product update is simply abandoned. Nobody retries it unless a human notices and clicks retry in the dashboard, or a script calls the retry mutation. Left alone, these failed deliveries just accumulate until Saleor's 14 day EVENT_PAYLOAD_DELETE_PERIOD quietly purges the payload.
Why it happens
- The
send_webhook_requestCelery task insaleor/plugins/webhook/tasks.pyis configured withretry_backoff=10andretry_kwargs={"max_retries": 5}, which works out to roughly 10, 20, 40, 80, 160 seconds of delay across the 5 attempts, capping out around the 320 second mark. - Once the fifth retry also fails, Celery's own retry accounting stops the task from being rescheduled again. This is normal Celery behavior, not a bug, but Saleor never layers a second, longer-horizon retry job on top of it.
- Saleor persists the delivery's final status as FAILED on the
EventDeliveryrecord and there is no cron, no periodic task, nothing built in that comes back to check whether the endpoint recovered later. - The only ways a FAILED delivery ever moves again are a human clicking retry in the Dashboard, a call to the
eventDeliveryRetrymutation, or the same event type firing fresh from a new state change, none of which happen unless someone or something notices.
None of this raises an alert anywhere in the store. The order still gets created, the product still updates, and the only casualty is the notification that was supposed to reach your app. Left unmonitored, a brief outage on your side quietly turns into a growing pile of dead deliveries that nobody is looking at.
A FAILED delivery past 5 attempts is not "still trying," it is "already gave up." Saleor will not touch it again, so the only way it moves is eventDeliveryRetry or a fresh event. But not every FAILED delivery deserves that retry. One whose last few attempts all show a 5xx or a timeout is telling you the endpoint is still broken, and retrying it just burns another 5 attempt cycle for nothing. The safe rule is to retry the ones that look like a passing blip, and flag the ones that look like a dead endpoint for a human to actually fix.
The fix, as a flow
The script runs on a schedule. It lists each webhook's FAILED deliveries with their recent attempts, decides which ones are genuinely stuck past Saleor's own retry ceiling, and splits those into two piles: ones whose failures look transient, which get a single guarded call to eventDeliveryRetry, and ones whose endpoint looks consistently dead, which are only reported so a human can fix the target URL. Anything still within Saleor's own retry window is left alone so the script never races Saleor's own Celery task.
Build it step by step
Get an app token with read and write access to webhooks
Create an app in Saleor Dashboard under Configuration, Apps, and give it permission to manage webhooks and their event deliveries, or exchange staff credentials with tokenCreate. Keep the API URL and token in environment variables, never in the file.
pip install requests
export SALEOR_API_URL="https://store.saleor.cloud/graphql/"
export SALEOR_AUTH_TOKEN="your-app-or-staff-token"
export STALE_AFTER_MINUTES="60"
export DRY_RUN="true" # start safe, this script never writes without it off// Node 18+ has fetch built in, no dependencies needed
export SALEOR_API_URL="https://store.saleor.cloud/graphql/"
export SALEOR_AUTH_TOKEN="your-app-or-staff-token"
export STALE_AFTER_MINUTES="60"
export DRY_RUN="true" // start safe, this script never writes without it offTalk to the Saleor GraphQL API
Saleor is one GraphQL endpoint. Every call is a POST with a JSON body of {query, variables} and an Authorization: Bearer <token> header. A small helper sends a query and returns the data, raising if Saleor reports errors.
import os, requests
API_URL = os.environ["SALEOR_API_URL"]
TOKEN = os.environ["SALEOR_AUTH_TOKEN"]
def gql(query, variables=None):
r = requests.post(
API_URL,
json={"query": query, "variables": variables or {}},
headers={"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"},
timeout=30,
)
r.raise_for_status()
body = r.json()
if body.get("errors"):
raise RuntimeError(body["errors"])
return body["data"]const API_URL = process.env.SALEOR_API_URL;
const TOKEN = process.env.SALEOR_AUTH_TOKEN;
async function gql(query, variables = {}) {
const res = await fetch(API_URL, {
method: "POST",
headers: { Authorization: `Bearer ${TOKEN}`, "Content-Type": "application/json" },
body: JSON.stringify({ query, variables }),
});
if (!res.ok) throw new Error(`Saleor ${res.status}`);
const body = await res.json();
if (body.errors) throw new Error(JSON.stringify(body.errors));
return body.data;
}List the webhooks and their FAILED deliveries
Ask each webhook for its eventDeliveries filtered to status: FAILED, sorted newest first, and read back the last few attempts with their responseStatusCode. That is everything the decision needs: how many attempts already happened and whether the recent ones look like a timeout or a broken endpoint.
FAILED_DELIVERIES_QUERY = """
query($cursor: String) {
webhooks(first: 50) {
edges {
node {
id
name
isActive
targetUrl
eventDeliveries(first: 100, filter: { status: FAILED },
sortBy: { field: CREATED_AT, direction: DESC }) {
edges {
node {
id
eventType
createdAt
status
attempts(first: 5, sortBy: { field: CREATED_AT, direction: DESC }) {
edges { node { id createdAt status duration responseStatusCode taskId } }
}
}
}
}
}
}
}
}"""
def failed_deliveries_by_webhook():
data = gql(FAILED_DELIVERIES_QUERY)["webhooks"]
for edge in data["edges"]:
webhook = edge["node"]
deliveries = [d["node"] for d in webhook["eventDeliveries"]["edges"]]
yield webhook, deliveriesconst FAILED_DELIVERIES_QUERY = `
query {
webhooks(first: 50) {
edges {
node {
id
name
isActive
targetUrl
eventDeliveries(first: 100, filter: { status: FAILED },
sortBy: { field: CREATED_AT, direction: DESC }) {
edges {
node {
id
eventType
createdAt
status
attempts(first: 5, sortBy: { field: CREATED_AT, direction: DESC }) {
edges { node { id createdAt status duration responseStatusCode taskId } }
}
}
}
}
}
}
}
}`;
async function* failedDeliveriesByWebhook() {
const data = (await gql(FAILED_DELIVERIES_QUERY)).webhooks;
for (const edge of data.edges) {
const webhook = edge.node;
const deliveries = webhook.eventDeliveries.edges.map((d) => d.node);
yield { webhook, deliveries };
}
}Decide, with one pure function
Keep the decision in its own function that takes the plain delivery shape (status, createdAt, attempts) and the current time, and returns one of three actions: SKIP, RETRY, or FLAG_DEAD_ENDPOINT. It defaults to Celery's own max_retries=5 and a one hour staleness window, well past the roughly 320 second ceiling of the exponential backoff, so it never races Saleor's own retries still in flight.
DEFAULT_MAX_RETRIES = 5
DEFAULT_STALE_AFTER_MS = 3600000 # 1 hour
def decide_stale_failed_retries(deliveries, now_iso, opts=None):
opts = opts or {}
max_retries = opts.get("maxRetries", DEFAULT_MAX_RETRIES)
stale_after_ms = opts.get("staleAfterMs", DEFAULT_STALE_AFTER_MS)
now_ms = _parse_iso_ms(now_iso)
decisions = []
for delivery in deliveries:
if delivery.get("status") != "FAILED":
decisions.append({"id": delivery["id"], "action": "SKIP", "reason": "not-failed"})
continue
attempts = delivery.get("attempts") or []
attempt_count = len(attempts)
last_attempt_at = delivery.get("createdAt")
if attempts:
last_attempt_at = max(a["createdAt"] for a in attempts)
age_ms = now_ms - _parse_iso_ms(last_attempt_at)
if attempt_count < max_retries and age_ms < stale_after_ms:
decisions.append({"id": delivery["id"], "action": "SKIP", "reason": "still-within-retry-window"})
continue
if attempt_count >= max_retries and age_ms >= stale_after_ms:
recent = attempts[:max_retries]
all_dead = all(_looks_dead(a) for a in recent) if recent else False
if all_dead:
decisions.append({"id": delivery["id"], "action": "FLAG_DEAD_ENDPOINT",
"reason": "endpoint-repeatedly-unreachable"})
else:
decisions.append({"id": delivery["id"], "action": "RETRY",
"reason": "stale-failed-past-retry-limit-transient-error"})
continue
decisions.append({"id": delivery["id"], "action": "SKIP",
"reason": "recently-exhausted-wait-for-staleness-window"})
return decisions
def _looks_dead(attempt):
code = attempt.get("responseStatusCode")
return code is None or code >= 500
def _parse_iso_ms(iso):
import datetime
return datetime.datetime.fromisoformat(iso.replace("Z", "+00:00")).timestamp() * 1000const DEFAULT_MAX_RETRIES = 5;
const DEFAULT_STALE_AFTER_MS = 3600000; // 1 hour
export function decideStaleFailedRetries(deliveries, nowIso, opts = {}) {
const maxRetries = opts.maxRetries ?? DEFAULT_MAX_RETRIES;
const staleAfterMs = opts.staleAfterMs ?? DEFAULT_STALE_AFTER_MS;
const nowMs = Date.parse(nowIso);
return deliveries.map((delivery) => {
if (delivery.status !== "FAILED") {
return { id: delivery.id, action: "SKIP", reason: "not-failed" };
}
const attempts = delivery.attempts || [];
const attemptCount = attempts.length;
const lastAttemptAt = attempts.length
? attempts.reduce((max, a) => (a.createdAt > max ? a.createdAt : max), attempts[0].createdAt)
: delivery.createdAt;
const ageMs = nowMs - Date.parse(lastAttemptAt);
if (attemptCount < maxRetries && ageMs < staleAfterMs) {
return { id: delivery.id, action: "SKIP", reason: "still-within-retry-window" };
}
if (attemptCount >= maxRetries && ageMs >= staleAfterMs) {
const recent = attempts.slice(0, maxRetries);
const allDead = recent.length > 0 && recent.every(looksDead);
return allDead
? { id: delivery.id, action: "FLAG_DEAD_ENDPOINT", reason: "endpoint-repeatedly-unreachable" }
: { id: delivery.id, action: "RETRY", reason: "stale-failed-past-retry-limit-transient-error" };
}
return { id: delivery.id, action: "SKIP", reason: "recently-exhausted-wait-for-staleness-window" };
});
}
function looksDead(attempt) {
const code = attempt.responseStatusCode;
return code === null || code === undefined || code >= 500;
}Retry the transient ones, one at a time
For every delivery classified RETRY, call eventDeliveryRetry with its id, check that errors is empty, then re-poll the delivery shortly after to confirm it actually left FAILED. Never loop this blindly across every FAILED delivery, since retrying a dead endpoint just wastes another backoff cycle.
RETRY_MUTATION = """
mutation($id: ID!) {
eventDeliveryRetry(id: $id) {
delivery { id status }
errors { field code message }
}
}"""
def retry_delivery(delivery_id):
result = gql(RETRY_MUTATION, {"id": delivery_id})["eventDeliveryRetry"]
if result["errors"]:
raise RuntimeError(result["errors"])
return result["delivery"]["status"]const RETRY_MUTATION = `
mutation($id: ID!) {
eventDeliveryRetry(id: $id) {
delivery { id status }
errors { field code message }
}
}`;
async function retryDelivery(deliveryId) {
const result = (await gql(RETRY_MUTATION, { id: deliveryId })).eventDeliveryRetry;
if (result.errors.length) throw new Error(JSON.stringify(result.errors));
return result.delivery.status;
}Wire it together with a dry run guard
The loop reads every webhook's FAILED deliveries, runs each through the decision function, and for RETRY it either logs what it would do or actually retries when DRY_RUN=false. Deliveries classified FLAG_DEAD_ENDPOINT are always reported, never retried, since Saleor has no safe way to edit a webhook's targetUrl automatically, that judgment call belongs to a human.
Always start with DRY_RUN=true, retry one delivery at a time, and never loop blindly across every FAILED delivery. A dead endpoint stays dead until a human fixes the target URL, so the script's job is to report it clearly, not to keep hammering it.
The full code
Here is the complete script in one file for each language. It reads settings from the environment, lists FAILED deliveries with their attempts, classifies each one, retries the transient failures behind a dry run, and reports dead endpoints for a human to fix.
"""Find Saleor webhook EventDeliveries stuck FAILED past Celery's retry limit
and retry only the ones that look worth retrying.
send_webhook_request (saleor/plugins/webhook/tasks.py) retries an async
delivery with retry_backoff=10 and retry_kwargs={"max_retries": 5}, roughly
10 * 2^n seconds of delay across 5 attempts. Once the fifth retry also
fails, Celery never reschedules the task again and the EventDelivery is
persisted as FAILED for good. Nothing in Saleor resurrects it later.
Under DRY_RUN=true (the default) this only reports what it would do. When
DRY_RUN=false it calls eventDeliveryRetry once per delivery classified
RETRY, and re-polls its status to confirm it left FAILED. Deliveries whose
recent attempts all look like a dead endpoint (5xx or no response code)
are only ever reported, never retried. Run on a schedule. Safe to run
again and again.
"""
import os
import time
import logging
import datetime
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("retry_stale_failed_deliveries")
API_URL = os.environ["SALEOR_API_URL"]
TOKEN = os.environ["SALEOR_AUTH_TOKEN"]
STALE_AFTER_MINUTES = float(os.environ.get("STALE_AFTER_MINUTES", "60"))
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"
DEFAULT_MAX_RETRIES = 5
DEFAULT_STALE_AFTER_MS = 3600000 # 1 hour
FAILED_DELIVERIES_QUERY = """
query {
webhooks(first: 50) {
edges {
node {
id
name
isActive
targetUrl
eventDeliveries(first: 100, filter: { status: FAILED },
sortBy: { field: CREATED_AT, direction: DESC }) {
edges {
node {
id
eventType
createdAt
status
attempts(first: 5, sortBy: { field: CREATED_AT, direction: DESC }) {
edges { node { id createdAt status duration responseStatusCode taskId } }
}
}
}
}
}
}
}
}"""
DELIVERY_STATUS_QUERY = """
query($id: ID!) {
eventDelivery(id: $id) { id status }
}"""
RETRY_MUTATION = """
mutation($id: ID!) {
eventDeliveryRetry(id: $id) {
delivery { id status }
errors { field code message }
}
}"""
def gql(query, variables=None):
r = requests.post(
API_URL,
json={"query": query, "variables": variables or {}},
headers={"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"},
timeout=30,
)
r.raise_for_status()
body = r.json()
if body.get("errors"):
raise RuntimeError(body["errors"])
return body["data"]
def _parse_iso_ms(iso):
return datetime.datetime.fromisoformat(iso.replace("Z", "+00:00")).timestamp() * 1000
def _looks_dead(attempt):
code = attempt.get("responseStatusCode")
return code is None or code >= 500
def decide_stale_failed_retries(deliveries, now_iso, opts=None):
opts = opts or {}
max_retries = opts.get("maxRetries", DEFAULT_MAX_RETRIES)
stale_after_ms = opts.get("staleAfterMs", DEFAULT_STALE_AFTER_MS)
now_ms = _parse_iso_ms(now_iso)
decisions = []
for delivery in deliveries:
if delivery.get("status") != "FAILED":
decisions.append({"id": delivery["id"], "action": "SKIP", "reason": "not-failed"})
continue
attempts = delivery.get("attempts") or []
attempt_count = len(attempts)
last_attempt_at = delivery.get("createdAt")
if attempts:
last_attempt_at = max(a["createdAt"] for a in attempts)
age_ms = now_ms - _parse_iso_ms(last_attempt_at)
if attempt_count < max_retries and age_ms < stale_after_ms:
decisions.append({"id": delivery["id"], "action": "SKIP", "reason": "still-within-retry-window"})
continue
if attempt_count >= max_retries and age_ms >= stale_after_ms:
recent = attempts[:max_retries]
all_dead = all(_looks_dead(a) for a in recent) if recent else False
if all_dead:
decisions.append({"id": delivery["id"], "action": "FLAG_DEAD_ENDPOINT",
"reason": "endpoint-repeatedly-unreachable"})
else:
decisions.append({"id": delivery["id"], "action": "RETRY",
"reason": "stale-failed-past-retry-limit-transient-error"})
continue
decisions.append({"id": delivery["id"], "action": "SKIP",
"reason": "recently-exhausted-wait-for-staleness-window"})
return decisions
def failed_deliveries_by_webhook():
data = gql(FAILED_DELIVERIES_QUERY)["webhooks"]
for edge in data["edges"]:
webhook = edge["node"]
deliveries = [d["node"] for d in webhook["eventDeliveries"]["edges"]]
yield webhook, deliveries
def retry_delivery(delivery_id):
result = gql(RETRY_MUTATION, {"id": delivery_id})["eventDeliveryRetry"]
if result["errors"]:
raise RuntimeError(result["errors"])
return result["delivery"]["status"]
def confirm_left_failed(delivery_id):
data = gql(DELIVERY_STATUS_QUERY, {"id": delivery_id})["eventDelivery"]
return data["status"] if data else None
def run():
now_iso = datetime.datetime.now(datetime.timezone.utc).isoformat()
opts = {"staleAfterMs": STALE_AFTER_MINUTES * 60 * 1000}
retried = 0
flagged = 0
for webhook, deliveries in failed_deliveries_by_webhook():
decisions = decide_stale_failed_retries(deliveries, now_iso, opts)
by_id = {d["id"]: d for d in deliveries}
for decision in decisions:
if decision["action"] == "SKIP":
continue
delivery = by_id[decision["id"]]
if decision["action"] == "FLAG_DEAD_ENDPOINT":
codes = [a.get("responseStatusCode") for a in (delivery.get("attempts") or [])]
log.warning(
"DEAD ENDPOINT webhook=%s (%s) delivery=%s eventType=%s recent_codes=%s",
webhook["name"], webhook["targetUrl"], delivery["id"], delivery["eventType"], codes,
)
flagged += 1
continue
if decision["action"] == "RETRY":
log.info(
"[DRY RUN] would retry delivery %s (%s, %s)" if DRY_RUN else "Retrying delivery %s (%s, %s)",
delivery["id"], delivery["eventType"], webhook["name"],
)
if not DRY_RUN:
retry_delivery(delivery["id"])
time.sleep(2)
status = confirm_left_failed(delivery["id"])
log.info("Delivery %s status after retry: %s", delivery["id"], status)
retried += 1
log.info(
"Done. %d delivery(ies) %s, %d dead endpoint(s) flagged for review.",
retried, "to retry" if DRY_RUN else "retried", flagged,
)
if __name__ == "__main__":
run()/**
* Find Saleor webhook EventDeliveries stuck FAILED past Celery's retry limit
* and retry only the ones that look worth retrying.
*
* send_webhook_request (saleor/plugins/webhook/tasks.py) retries an async
* delivery with retry_backoff=10 and retry_kwargs={"max_retries": 5}, roughly
* 10 * 2^n seconds of delay across 5 attempts. Once the fifth retry also
* fails, Celery never reschedules the task again and the EventDelivery is
* persisted as FAILED for good. Nothing in Saleor resurrects it later.
*
* Under DRY_RUN=true (the default) this only reports what it would do. When
* DRY_RUN=false it calls eventDeliveryRetry once per delivery classified
* RETRY, and re-polls its status to confirm it left FAILED. Deliveries whose
* recent attempts all look like a dead endpoint (5xx or no response code)
* are only ever reported, never retried. Run on a schedule.
*
* Guide: https://www.allanninal.dev/saleor/webhook-deliveries-stuck-failed/
*/
import { pathToFileURL } from "node:url";
const API_URL = process.env.SALEOR_API_URL || "https://store.saleor.cloud/graphql/";
const TOKEN = process.env.SALEOR_AUTH_TOKEN || "dummy-token";
const STALE_AFTER_MINUTES = Number(process.env.STALE_AFTER_MINUTES || 60);
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
const DEFAULT_MAX_RETRIES = 5;
const DEFAULT_STALE_AFTER_MS = 3600000; // 1 hour
export function decideStaleFailedRetries(deliveries, nowIso, opts = {}) {
const maxRetries = opts.maxRetries ?? DEFAULT_MAX_RETRIES;
const staleAfterMs = opts.staleAfterMs ?? DEFAULT_STALE_AFTER_MS;
const nowMs = Date.parse(nowIso);
return deliveries.map((delivery) => {
if (delivery.status !== "FAILED") {
return { id: delivery.id, action: "SKIP", reason: "not-failed" };
}
const attempts = delivery.attempts || [];
const attemptCount = attempts.length;
const lastAttemptAt = attempts.length
? attempts.reduce((max, a) => (a.createdAt > max ? a.createdAt : max), attempts[0].createdAt)
: delivery.createdAt;
const ageMs = nowMs - Date.parse(lastAttemptAt);
if (attemptCount < maxRetries && ageMs < staleAfterMs) {
return { id: delivery.id, action: "SKIP", reason: "still-within-retry-window" };
}
if (attemptCount >= maxRetries && ageMs >= staleAfterMs) {
const recent = attempts.slice(0, maxRetries);
const allDead = recent.length > 0 && recent.every(looksDead);
return allDead
? { id: delivery.id, action: "FLAG_DEAD_ENDPOINT", reason: "endpoint-repeatedly-unreachable" }
: { id: delivery.id, action: "RETRY", reason: "stale-failed-past-retry-limit-transient-error" };
}
return { id: delivery.id, action: "SKIP", reason: "recently-exhausted-wait-for-staleness-window" };
});
}
function looksDead(attempt) {
const code = attempt.responseStatusCode;
return code === null || code === undefined || code >= 500;
}
async function gql(query, variables = {}) {
const res = await fetch(API_URL, {
method: "POST",
headers: { Authorization: `Bearer ${TOKEN}`, "Content-Type": "application/json" },
body: JSON.stringify({ query, variables }),
});
if (!res.ok) throw new Error(`Saleor ${res.status}`);
const body = await res.json();
if (body.errors) throw new Error(JSON.stringify(body.errors));
return body.data;
}
const FAILED_DELIVERIES_QUERY = `
query {
webhooks(first: 50) {
edges {
node {
id
name
isActive
targetUrl
eventDeliveries(first: 100, filter: { status: FAILED },
sortBy: { field: CREATED_AT, direction: DESC }) {
edges {
node {
id
eventType
createdAt
status
attempts(first: 5, sortBy: { field: CREATED_AT, direction: DESC }) {
edges { node { id createdAt status duration responseStatusCode taskId } }
}
}
}
}
}
}
}
}`;
const DELIVERY_STATUS_QUERY = `
query($id: ID!) {
eventDelivery(id: $id) { id status }
}`;
const RETRY_MUTATION = `
mutation($id: ID!) {
eventDeliveryRetry(id: $id) {
delivery { id status }
errors { field code message }
}
}`;
async function* failedDeliveriesByWebhook() {
const data = (await gql(FAILED_DELIVERIES_QUERY)).webhooks;
for (const edge of data.edges) {
const webhook = edge.node;
const deliveries = webhook.eventDeliveries.edges.map((d) => d.node);
yield { webhook, deliveries };
}
}
async function retryDelivery(deliveryId) {
const result = (await gql(RETRY_MUTATION, { id: deliveryId })).eventDeliveryRetry;
if (result.errors.length) throw new Error(JSON.stringify(result.errors));
return result.delivery.status;
}
async function confirmLeftFailed(deliveryId) {
const data = (await gql(DELIVERY_STATUS_QUERY, { id: deliveryId })).eventDelivery;
return data ? data.status : null;
}
function sleep(ms) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export async function run() {
const nowIso = new Date().toISOString();
const opts = { staleAfterMs: STALE_AFTER_MINUTES * 60 * 1000 };
let retried = 0;
let flagged = 0;
for await (const { webhook, deliveries } of failedDeliveriesByWebhook()) {
const decisions = decideStaleFailedRetries(deliveries, nowIso, opts);
const byId = new Map(deliveries.map((d) => [d.id, d]));
for (const decision of decisions) {
if (decision.action === "SKIP") continue;
const delivery = byId.get(decision.id);
if (decision.action === "FLAG_DEAD_ENDPOINT") {
const codes = (delivery.attempts || []).map((a) => a.responseStatusCode);
console.warn(
`DEAD ENDPOINT webhook=${webhook.name} (${webhook.targetUrl}) delivery=${delivery.id} eventType=${delivery.eventType} recent_codes=${JSON.stringify(codes)}`
);
flagged++;
continue;
}
if (decision.action === "RETRY") {
console.log(
DRY_RUN
? `[DRY RUN] would retry delivery ${delivery.id} (${delivery.eventType}, ${webhook.name})`
: `Retrying delivery ${delivery.id} (${delivery.eventType}, ${webhook.name})`
);
if (!DRY_RUN) {
await retryDelivery(delivery.id);
await sleep(2000);
const status = await confirmLeftFailed(delivery.id);
console.log(`Delivery ${delivery.id} status after retry: ${status}`);
}
retried++;
}
}
}
console.log(
`Done. ${retried} delivery(ies) ${DRY_RUN ? "to retry" : "retried"}, ${flagged} dead endpoint(s) flagged for review.`
);
}
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((err) => { console.error(err); process.exit(1); });
}Add a test
The decision rule is the part most worth testing, because it decides which failed deliveries get retried, which get flagged as a dead endpoint, and which get left alone. Because decide_stale_failed_retries is pure, the test needs no network and no Saleor account. It just feeds in plain objects and checks the answer.
from retry_stale_failed_deliveries import decide_stale_failed_retries
NOW = "2026-07-10T02:00:00Z" # 2 hours after the attempts below
def attempt(**over):
base = {"createdAt": "2026-07-10T00:00:00Z", "responseStatusCode": 503}
base.update(over)
return base
def delivery(**over):
base = {
"id": "gid://saleor/EventDelivery/1",
"status": "FAILED",
"createdAt": "2026-07-09T23:55:00Z",
"attempts": [attempt() for _ in range(5)],
}
base.update(over)
return base
def test_skip_when_not_failed():
result = decide_stale_failed_retries([delivery(status="SUCCESS")], NOW)
assert result[0]["action"] == "SKIP"
assert result[0]["reason"] == "not-failed"
def test_skip_when_still_within_retry_window():
d = delivery(attempts=[attempt(createdAt="2026-07-10T01:59:00Z")])
result = decide_stale_failed_retries([d], NOW)
assert result[0]["action"] == "SKIP"
assert result[0]["reason"] == "still-within-retry-window"
def test_retry_when_stale_and_transient_error():
# A 429 (rate limited) is not treated as a dead endpoint, unlike a 5xx.
d = delivery(attempts=[attempt(responseStatusCode=429) for _ in range(5)])
result = decide_stale_failed_retries([d], NOW)
assert result[0]["action"] == "RETRY"
assert result[0]["reason"] == "stale-failed-past-retry-limit-transient-error"
def test_retry_when_mixed_status_codes_not_all_dead():
attempts = [attempt(responseStatusCode=200), *[attempt(responseStatusCode=503) for _ in range(4)]]
d = delivery(attempts=attempts)
result = decide_stale_failed_retries([d], NOW)
assert result[0]["action"] == "RETRY"
def test_flag_dead_endpoint_when_all_recent_attempts_are_5xx():
d = delivery(attempts=[attempt(responseStatusCode=500) for _ in range(5)])
result = decide_stale_failed_retries([d], NOW)
assert result[0]["action"] == "FLAG_DEAD_ENDPOINT"
assert result[0]["reason"] == "endpoint-repeatedly-unreachable"
def test_flag_dead_endpoint_when_attempts_have_no_response_code():
d = delivery(attempts=[attempt(responseStatusCode=None) for _ in range(5)])
result = decide_stale_failed_retries([d], NOW)
assert result[0]["action"] == "FLAG_DEAD_ENDPOINT"
def test_skip_when_recently_exhausted_but_not_yet_stale():
d = delivery(attempts=[attempt(createdAt="2026-07-10T01:45:00Z") for _ in range(5)])
result = decide_stale_failed_retries([d], NOW, {"staleAfterMs": 3600000})
assert result[0]["action"] == "SKIP"
assert result[0]["reason"] == "recently-exhausted-wait-for-staleness-window"
def test_skip_when_fewer_than_max_retries_and_not_stale():
d = delivery(attempts=[attempt(createdAt="2026-07-10T01:59:30Z")])
result = decide_stale_failed_retries([d], NOW)
assert result[0]["action"] == "SKIP"
assert result[0]["reason"] == "still-within-retry-window"
def test_uses_delivery_created_at_when_no_attempts():
# No attempts means attemptCount (0) < maxRetries, so it has not exhausted
# Saleor's retry budget yet, even though the delivery itself is old.
d = delivery(attempts=[], createdAt="2026-07-10T00:00:00Z")
result = decide_stale_failed_retries([d], NOW)
assert result[0]["action"] == "SKIP"
assert result[0]["reason"] == "recently-exhausted-wait-for-staleness-window"import { test } from "node:test";
import assert from "node:assert/strict";
import { decideStaleFailedRetries } from "./retry-stale-failed-deliveries.js";
const NOW = "2026-07-10T02:00:00Z"; // 2 hours after the attempts below
const attempt = (over = {}) => ({ createdAt: "2026-07-10T00:00:00Z", responseStatusCode: 429, ...over });
const delivery = (over = {}) => ({
id: "gid://saleor/EventDelivery/1",
status: "FAILED",
createdAt: "2026-07-09T23:55:00Z",
attempts: Array.from({ length: 5 }, () => attempt()),
...over,
});
test("skip when not failed", () => {
const result = decideStaleFailedRetries([delivery({ status: "SUCCESS" })], NOW);
assert.equal(result[0].action, "SKIP");
assert.equal(result[0].reason, "not-failed");
});
test("skip when still within retry window", () => {
const d = delivery({ attempts: [attempt({ createdAt: "2026-07-10T01:59:00Z" })] });
const result = decideStaleFailedRetries([d], NOW);
assert.equal(result[0].action, "SKIP");
assert.equal(result[0].reason, "still-within-retry-window");
});
test("retry when stale and transient error", () => {
const d = delivery({ attempts: Array.from({ length: 5 }, () => attempt({ responseStatusCode: 429 })) });
const result = decideStaleFailedRetries([d], NOW);
assert.equal(result[0].action, "RETRY");
assert.equal(result[0].reason, "stale-failed-past-retry-limit-transient-error");
});
test("retry when mixed status codes not all dead", () => {
const attempts = [attempt({ responseStatusCode: 200 }), ...Array.from({ length: 4 }, () => attempt({ responseStatusCode: 503 }))];
const d = delivery({ attempts });
const result = decideStaleFailedRetries([d], NOW);
assert.equal(result[0].action, "RETRY");
});
test("flag dead endpoint when all recent attempts are 5xx", () => {
const d = delivery({ attempts: Array.from({ length: 5 }, () => attempt({ responseStatusCode: 500 })) });
const result = decideStaleFailedRetries([d], NOW);
assert.equal(result[0].action, "FLAG_DEAD_ENDPOINT");
assert.equal(result[0].reason, "endpoint-repeatedly-unreachable");
});
test("flag dead endpoint when attempts have no response code", () => {
const d = delivery({ attempts: Array.from({ length: 5 }, () => attempt({ responseStatusCode: null })) });
const result = decideStaleFailedRetries([d], NOW);
assert.equal(result[0].action, "FLAG_DEAD_ENDPOINT");
});
test("skip when recently exhausted but not yet stale", () => {
const d = delivery({ attempts: Array.from({ length: 5 }, () => attempt({ createdAt: "2026-07-10T01:45:00Z" })) });
const result = decideStaleFailedRetries([d], NOW, { staleAfterMs: 3600000 });
assert.equal(result[0].action, "SKIP");
assert.equal(result[0].reason, "recently-exhausted-wait-for-staleness-window");
});
test("skip when fewer than max retries and not stale", () => {
const d = delivery({ attempts: [attempt({ createdAt: "2026-07-10T01:59:30Z" })] });
const result = decideStaleFailedRetries([d], NOW);
assert.equal(result[0].action, "SKIP");
assert.equal(result[0].reason, "still-within-retry-window");
});
test("uses delivery createdAt when no attempts", () => {
const d = delivery({ attempts: [], createdAt: "2026-07-10T00:00:00Z" });
const result = decideStaleFailedRetries([d], NOW);
assert.equal(result[0].action, "SKIP");
assert.equal(result[0].reason, "recently-exhausted-wait-for-staleness-window");
});Case studies
A ten minute deploy quietly dropped a day of order events
A fulfillment app's webhook endpoint went down for about ten minutes during a routine deploy. Saleor's Celery task retried on its usual backoff, the endpoint came back up well within that window, but a handful of deliveries landed exactly on attempts that hit the dead window and exhausted their 5 retries before the app was healthy again. Those ORDER_CREATED events sat FAILED for two days before anyone noticed orders that never made it to the fulfillment queue.
Running the script hourly caught those deliveries as stale and transient, since the endpoint's most recent attempts elsewhere were already succeeding again, and eventDeliveryRetry cleared the backlog the same hour it was found instead of after a support ticket.
A misconfigured URL kept failing the same way for a week
A team rotated their webhook receiver to a new subdomain but forgot to update the targetUrl on one Saleor webhook. Every delivery failed with a connection timeout, retried five times, and piled up as FAILED, hundreds of them by the end of the week, with nobody watching that particular webhook's health.
The script flagged the webhook itself, since every one of the last five attempts on every stale delivery showed a timeout with no response code, and reported the exact stale target URL instead of quietly burning retry cycles against a URL that was never going to answer. Fixing the URL and letting the next scheduled events fire cleared the queue for good.
After this runs on a schedule, a webhook delivery that Saleor gave up on for a genuinely transient reason gets a real second chance within the hour, instead of sitting FAILED until the payload ages out. A webhook whose endpoint is actually broken gets surfaced with its target URL and the exact failure pattern, so a human fixes the real problem instead of a script quietly retrying a dead address forever.
FAQ
Why does a Saleor webhook delivery stay FAILED forever?
Saleor's send_webhook_request Celery task retries an async delivery with retry_backoff=10 and max_retries=5, roughly 10 times 2 to the n seconds of delay per attempt. Once the fifth retry also fails, Celery stops rescheduling the task entirely and the EventDelivery is persisted as FAILED for good. There is no scheduled job in Saleor that comes back and resurrects it later, so it sits there until a human or a script retries it, or until the 14 day EVENT_PAYLOAD_DELETE_PERIOD purges it.
How do I find webhook deliveries that are stuck past the retry limit?
Query webhooks and their eventDeliveries filtered to status FAILED, and for each one look at its attempts. If the delivery already has 5 or more attempts and the most recent attempt is older than about an hour, well past the roughly 320 second ceiling of the exponential backoff, Saleor has already given up and will not retry it again on its own. That combination of attempt count and staleness is what marks a delivery as stuck rather than still in flight.
Is it safe to bulk retry every FAILED webhook delivery?
No. A delivery whose last several attempts all show a 5xx status code or a timeout points at an endpoint that is still broken, and calling eventDeliveryRetry on it just burns another 5 attempt backoff cycle for nothing. Only retry deliveries where the failure looks transient, flag the ones with a consistently broken endpoint for a human to fix the URL, and always retry one at a time behind a dry run so you can see exactly what the script is about to touch.
Related field notes
Citations
On the problem:
- Saleor Commerce Documentation: Webhooks Troubleshooting. docs.saleor.io/developer/extending/webhooks/troubleshooting
- Saleor Commerce Documentation: Asynchronous Events. docs.saleor.io/developer/extending/webhooks/asynchronous-events
- saleor/saleor: send_webhook_request task, retry_backoff and max_retries. github.com/saleor/saleor/blob/.../saleor/plugins/webhook/tasks.py
On the solution:
- Saleor Commerce Documentation: Webhooks Troubleshooting. docs.saleor.io/developer/extending/webhooks/troubleshooting
- Saleor Commerce Documentation: API reference Overview. docs.saleor.io/api-reference/
- Saleor Commerce Documentation: Webhooks Overview. docs.saleor.io/developer/extending/webhooks/overview
Stuck on a tricky one?
If you have a problem in Saleor checkout, stock, channels, or fulfillment that you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this clear a pile of stuck deliveries?
If this saved you from a silent gap between Saleor and your app, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi