Diagnostic Webhooks
Order webhooks stop firing entirely with no surfaced error
Orders keep coming in, the storefront looks fine, and then you notice your fulfillment queue or inventory sync has not heard from BigCommerce in hours. Nothing in the control panel says a webhook died. BigCommerce quietly disables a subscription after it keeps failing to deliver, and separately can blocklist your whole receiving domain for a few minutes at a time if its short-term success rate dips, and neither event shows up as an alert anywhere you would normally look. Here is why that happens and a small script that finds the gap and flags it instead of guessing.
BigCommerce retries a failing webhook destination on a backoff schedule for roughly 48 hours, and if it still is not returning HTTP 200, it permanently flips that subscription's is_active to false, emailing only the address on file for the subscribing app. Separately, once a destination domain has received 100 or more requests, BigCommerce tracks a rolling 2 minute success and failure ratio and will blocklist that whole domain for 3 minutes if the success rate drops below 90 percent, which looks like everything stopped even though the hook still shows is_active:true. Neither mechanism raises a dashboard alert. Pull recent orders with GET /v2/orders, pull your subscriptions with GET /v3/hooks, and diff both against your own webhook receiver log to catch the gap. Full detection code, tests, and a guarded (dry run) repair path are below.
The problem in plain words
A BigCommerce webhook subscription is a promise, not a guarantee. When your endpoint stops answering with a 200, either because it is down, redeployed with a broken route, behind an expired certificate, or rejecting the payload for an auth reason, BigCommerce does not just give up on the next event. It retries the delivery on a backoff schedule for roughly 48 hours. If your endpoint still has not recovered by the end of that window, BigCommerce permanently disables the subscription by setting is_active to false.
The only notice of that deactivation is an email, sent to whatever address is on file for the app that created the subscription, not to the store owner, not to the control panel, and not to any webhook of your own. If that inbox is unmonitored, or the subscribing app is one of several integrations on the store, the disabling can go unnoticed for days. Meanwhile orders, customers, and everything else in the store keep being created completely normally. Nothing on the storefront or in the order list looks wrong.
There is a second, separate failure mode that is even harder to see. BigCommerce tracks delivery success against each destination domain in a rolling 2 minute window, and once it has sent 100 or more requests to that domain, it will blocklist the entire domain for 3 minutes if the success rate in that window drops below 90 percent. During that 3 minute window, deliveries to every hook pointed at that domain fail, even ones with a completely healthy is_active:true record, which reads identically to a wider outage from the receiving side.
Why it happens
BigCommerce is protecting itself and its delivery infrastructure from endpoints that cannot keep up, but the way it does that leaves store owners with no visible signal. The common paths into this:
- Your receiving endpoint redeploys with a broken route, an expired TLS certificate, or a changed auth check, and every delivery for roughly the next 48 hours comes back as a non-200, until BigCommerce permanently sets
is_activetofalseon that subscription. - The deactivation email goes to the address on file for the app that created the hook, not to the store owner, so if that app is a third-party integration or an old internal tool, nobody who can fix the receiver ever sees the notice.
- Your destination domain receives 100 or more webhook requests in a short window and its rolling success rate dips below 90 percent, for example during a deploy, a database failover, or a brief spike in 5xx responses, triggering a 3 minute domain-wide blocklist that fails hooks which still show
is_active:true. - A hook has sat unused or already disabled for around 90 days and gets pruned entirely, so even a well-intentioned reactivation attempt on an old hook id can silently do nothing.
In every one of these cases the store's control panel looks completely normal, because there is no webhook-of-webhooks and no dashboard banner for either mechanism. The only ground truth is comparing what orders actually happened against what your own systems actually received. See the citations at the end for the exact support threads and docs.
is_active:true on a hook record is not proof that hook is delivering. It only proves BigCommerce has not yet permanently disabled it. The reliable check is to compare three things: the timestamped orders that actually happened, the webhook deliveries your own receiver actually logged, and each hook's current is_active and updated_at. A hook with is_active:false is a clear deactivation. A hook that still reads is_active:true but has a delivery gap larger than your normal latency is the quieter, domain-blocklisting kind of failure, and both need a human, not an automatic flip back to on.
The fix, as a flow
We do not touch the live checkout or order flow. We add a job that pulls recent orders, pulls the store's current hook subscriptions, diffs both against the store's own webhook receiver log, and turns any mismatch into a named finding instead of a silent gap.
Build it step by step
Get a store hash and an API access token
Create an API account in your BigCommerce control panel under Settings, API, or use the store's existing app credentials. Grant it Orders (read) and Webhooks scope so it can list orders and hooks and, later, perform a guarded reactivation. You need the store hash from your control panel URL and the access token, sent on every call as the X-Auth-Token header. Keep both in environment variables, never in the file.
pip install requests
export BIGCOMMERCE_STORE_HASH="abc123"
export BIGCOMMERCE_ACCESS_TOKEN="..."
export LOOKBACK_DAYS="1"
export STALE_AFTER_MINUTES="30"
export DRY_RUN="true" # start safe, change to false to write// Node 18+ has fetch built in, no dependencies needed
export BIGCOMMERCE_STORE_HASH="abc123"
export BIGCOMMERCE_ACCESS_TOKEN="..."
export LOOKBACK_DAYS="1"
export STALE_AFTER_MINUTES="30"
export DRY_RUN="true" // start safe, change to false to writeTalk to the V2 orders and V3 hooks REST APIs
Orders live under https://api.bigcommerce.com/stores/{store_hash}/v2/, hooks live under the V3 base and wrap results in {data, meta.pagination}. The same token in the X-Auth-Token header works for both. A small helper handles GET and PUT and raises on a non-2xx response.
import os, requests
STORE_HASH = os.environ["BIGCOMMERCE_STORE_HASH"]
ACCESS_TOKEN = os.environ["BIGCOMMERCE_ACCESS_TOKEN"]
API_BASE_V2 = f"https://api.bigcommerce.com/stores/{STORE_HASH}/v2"
API_BASE_V3 = f"https://api.bigcommerce.com/stores/{STORE_HASH}/v3"
HEADERS = {
"X-Auth-Token": ACCESS_TOKEN,
"Content-Type": "application/json",
"Accept": "application/json",
}
def bc_get(base, path, params=None):
r = requests.get(f"{base}{path}", headers=HEADERS, params=params or {}, timeout=30)
r.raise_for_status()
return r.json() if r.text else []
def bc_put(base, path, body):
r = requests.put(f"{base}{path}", headers=HEADERS, json=body, timeout=30)
r.raise_for_status()
return r.json()const STORE_HASH = process.env.BIGCOMMERCE_STORE_HASH;
const ACCESS_TOKEN = process.env.BIGCOMMERCE_ACCESS_TOKEN;
const API_BASE_V2 = `https://api.bigcommerce.com/stores/${STORE_HASH}/v2`;
const API_BASE_V3 = `https://api.bigcommerce.com/stores/${STORE_HASH}/v3`;
const HEADERS = {
"X-Auth-Token": ACCESS_TOKEN,
"Content-Type": "application/json",
Accept: "application/json",
};
async function bcGet(base, path, params = {}) {
const url = new URL(`${base}${path}`);
for (const [key, value] of Object.entries(params)) url.searchParams.set(key, value);
const res = await fetch(url, { headers: HEADERS });
if (!res.ok) throw new Error(`BigCommerce ${res.status}`);
const text = await res.text();
return text ? JSON.parse(text) : [];
}
async function bcPut(base, path, body) {
const res = await fetch(`${base}${path}`, { method: "PUT", headers: HEADERS, body: JSON.stringify(body) });
if (!res.ok) throw new Error(`BigCommerce ${res.status}`);
return res.json();
}Pull the three inputs the decision needs
Call GET /v2/orders?min_date_created={ISO date}&sort=date_created:desc for a timestamped ground-truth list of order events. Call GET /v3/hooks (optionally ?is_active=true) for every subscription's id, scope, destination, is_active, and updated_at. And read your own webhook receiver log table for the timestamps you actually got, grouped by scope.
def recent_orders(lookback_days):
orders = []
page = 1
while True:
batch = bc_get(API_BASE_V2, "/orders", {
"min_date_created": f"-{lookback_days} days",
"sort": "date_created:desc",
"page": page,
"limit": 50,
})
if not batch:
return orders
orders.extend(batch)
page += 1
def current_hooks():
result = bc_get(API_BASE_V3, "/hooks", {"limit": 250})
return result.get("data", []) if isinstance(result, dict) else resultasync function recentOrders(lookbackDays) {
const orders = [];
let page = 1;
while (true) {
const batch = await bcGet(API_BASE_V2, "/orders", {
min_date_created: `-${lookbackDays} days`,
sort: "date_created:desc",
page,
limit: 50,
});
if (!batch.length) return orders;
orders.push(...batch);
page += 1;
}
}
async function currentHooks() {
const result = await bcGet(API_BASE_V3, "/hooks", { limit: 250 });
return Array.isArray(result) ? result : result.data || [];
}Decide, with one pure function
Keep the decision in its own function that takes the order timestamps, your webhook log timestamps grouped by scope, the raw hook records, and the current time, and returns a list of findings. Any hook matching store/order/* or store/customer/* with is_active:false is a deactivation. Any hook still is_active:true whose scope's last received delivery is older than the newest matching order event, by more than the stale threshold, is a silent delivery gap.
from datetime import datetime, timedelta, timezone
def _parse(ts):
return datetime.fromisoformat(ts.replace("Z", "+00:00"))
def _scope_matches(scope):
return scope.startswith("store/order/") or scope.startswith("store/customer/")
def detect_webhook_gap(order_timestamps, webhook_log_timestamps, hook_records, now, stale_after_minutes=30):
findings = []
now_dt = _parse(now)
latest_order = max((_parse(t) for t in order_timestamps), default=None)
for hook in hook_records:
scope = hook.get("scope", "")
if not _scope_matches(scope):
continue
if not hook.get("is_active", True):
findings.append({
"hook_id": hook.get("id"), "scope": scope,
"destination": hook.get("destination"), "is_active": False,
"reason": "deactivated",
})
continue
if latest_order is None:
continue
log_times = webhook_log_timestamps.get(scope, [])
last_received = max((_parse(t) for t in log_times), default=None)
gap_reference = last_received or latest_order
stale_cutoff = now_dt - timedelta(minutes=stale_after_minutes)
if latest_order > gap_reference and gap_reference < stale_cutoff:
findings.append({
"hook_id": hook.get("id"), "scope": scope,
"destination": hook.get("destination"), "is_active": True,
"reason": "stale_no_recent_delivery",
})
return findingsfunction scopeMatches(scope) {
return scope.startsWith("store/order/") || scope.startsWith("store/customer/");
}
export function detectWebhookGap(orderTimestamps, webhookLogTimestamps, hookRecords, now, staleAfterMinutes = 30) {
const findings = [];
const nowMs = new Date(now).getTime();
const orderMs = orderTimestamps.map((t) => new Date(t).getTime());
const latestOrder = orderMs.length ? Math.max(...orderMs) : null;
for (const hook of hookRecords) {
const scope = hook.scope || "";
if (!scopeMatches(scope)) continue;
if (hook.is_active === false) {
findings.push({ hook_id: hook.id, scope, destination: hook.destination, is_active: false, reason: "deactivated" });
continue;
}
if (latestOrder === null) continue;
const logTimes = (webhookLogTimestamps[scope] || []).map((t) => new Date(t).getTime());
const lastReceived = logTimes.length ? Math.max(...logTimes) : null;
const gapReference = lastReceived ?? latestOrder;
const staleCutoff = nowMs - staleAfterMinutes * 60 * 1000;
if (latestOrder > gapReference && gapReference < staleCutoff) {
findings.push({ hook_id: hook.id, scope, destination: hook.destination, is_active: true, reason: "stale_no_recent_delivery" });
}
}
return findings;
}Report the finding, do not auto-fix it
Every finding names the hook id, scope, destination, and reason. That is the whole output on a normal run. Reactivating is deliberately a separate, manual-gated step, because the receiver's underlying problem is still there until someone confirms otherwise.
def report_finding(finding):
log.warning(
"webhook gap: hook_id=%s scope=%s destination=%s is_active=%s reason=%s",
finding["hook_id"], finding["scope"], finding["destination"],
finding["is_active"], finding["reason"],
)function reportFinding(finding) {
console.warn(
`webhook gap: hook_id=${finding.hook_id} scope=${finding.scope} destination=${finding.destination} is_active=${finding.is_active} reason=${finding.reason}`
);
}Wire it together with a dry run guard on reactivation
The loop pulls all three inputs, runs the pure function, and reports every finding. Only after a manual HEAD or GET check confirms the destination now returns a healthy response should reactivation run, and even then it goes through PUT /v3/hooks/{hook_id} with {"is_active": true} behind DRY_RUN, which logs the intended call and skips it unless DRY_RUN=false. If a hook has been disabled long enough to risk the 90 day pruning window, recreate it fresh with POST /v3/hooks instead of trusting the old id.
Never flip is_active back to true automatically. Confirm the destination is actually healthy first, keep DRY_RUN=true until you have reviewed the findings, and treat a hook disabled for a long stretch as a candidate for a fresh POST /v3/hooks rather than a reactivated old one.
The full code
Here is the complete script in one file for each language. It reads settings from the environment, pulls the three inputs, runs the pure gap detector, logs every finding, and only attempts a guarded, dry-run-respecting reactivation when explicitly told to.
View this code on GitHub Full runnable folder with tests in the bigcommerce-fixes repo.
"""Detect BigCommerce order webhooks that stopped firing with no surfaced error.
BigCommerce retries a failing webhook destination on a backoff schedule for
roughly 48 hours, then permanently sets is_active to false on that subscription,
emailing only the address on file for the subscribing app. Separately, once a
destination domain has received 100 or more requests, BigCommerce tracks a
rolling 2 minute success and failure ratio and blocklists the whole domain for
3 minutes if the success rate drops below 90 percent, which can fail deliveries
even on a hook that still reads is_active:true. Neither mechanism raises a
dashboard alert. This job pulls recent orders, the store's current hook
subscriptions, and the store's own webhook receiver log, and reports any hook
that is deactivated or has gone stale with no recent delivery. It never
auto-reactivates; repair is a separate, guarded, dry-run-respecting step.
Guide: https://www.allanninal.dev/bigcommerce/order-webhooks-stop-firing-silently/
"""
import os
import logging
from datetime import datetime, timedelta
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("detect_webhook_gap")
STORE_HASH = os.environ["BIGCOMMERCE_STORE_HASH"]
ACCESS_TOKEN = os.environ["BIGCOMMERCE_ACCESS_TOKEN"]
API_BASE_V2 = f"https://api.bigcommerce.com/stores/{STORE_HASH}/v2"
API_BASE_V3 = f"https://api.bigcommerce.com/stores/{STORE_HASH}/v3"
LOOKBACK_DAYS = int(os.environ.get("LOOKBACK_DAYS", "1"))
STALE_AFTER_MINUTES = int(os.environ.get("STALE_AFTER_MINUTES", "30"))
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"
HEADERS = {
"X-Auth-Token": ACCESS_TOKEN,
"Content-Type": "application/json",
"Accept": "application/json",
}
def bc_get(base, path, params=None):
r = requests.get(f"{base}{path}", headers=HEADERS, params=params or {}, timeout=30)
r.raise_for_status()
if not r.text:
return []
return r.json()
def bc_put(base, path, body):
r = requests.put(f"{base}{path}", headers=HEADERS, json=body, timeout=30)
r.raise_for_status()
return r.json()
def _parse(ts):
return datetime.fromisoformat(ts.replace("Z", "+00:00"))
def _scope_matches(scope):
return scope.startswith("store/order/") or scope.startswith("store/customer/")
def detect_webhook_gap(order_timestamps, webhook_log_timestamps, hook_records, now, stale_after_minutes=30):
"""Pure decision. No network, no side effects.
order_timestamps: ISO8601 date_created/date_modified values from GET /v2/orders.
webhook_log_timestamps: {scope: [received_at ISO8601, ...]} from the store's
own webhook receiver log.
hook_records: raw items from GET /v3/hooks, each with id, scope, destination,
is_active, updated_at.
now: ISO8601 current time, injected for testability.
stale_after_minutes: threshold beyond normal delivery latency before a
still-active hook counts as stale.
Returns a list of finding dicts: {hook_id, scope, destination, is_active, reason}
with reason in {"deactivated", "stale_no_recent_delivery"}.
"""
findings = []
now_dt = _parse(now)
latest_order = max((_parse(t) for t in order_timestamps), default=None)
for hook in hook_records:
scope = hook.get("scope", "")
if not _scope_matches(scope):
continue
if not hook.get("is_active", True):
findings.append({
"hook_id": hook.get("id"),
"scope": scope,
"destination": hook.get("destination"),
"is_active": False,
"reason": "deactivated",
})
continue
if latest_order is None:
continue
log_times = webhook_log_timestamps.get(scope, [])
last_received = max((_parse(t) for t in log_times), default=None)
gap_reference = last_received or latest_order
stale_cutoff = now_dt - timedelta(minutes=stale_after_minutes)
if latest_order > gap_reference and gap_reference < stale_cutoff:
findings.append({
"hook_id": hook.get("id"),
"scope": scope,
"destination": hook.get("destination"),
"is_active": True,
"reason": "stale_no_recent_delivery",
})
return findings
def recent_orders(lookback_days):
"""Page through recent orders as a timestamped ground-truth event list."""
orders = []
page = 1
while True:
batch = bc_get(API_BASE_V2, "/orders", {
"min_date_created": f"-{lookback_days} days",
"sort": "date_created:desc",
"page": page,
"limit": 50,
})
if not batch:
return orders
orders.extend(batch)
page += 1
def current_hooks():
result = bc_get(API_BASE_V3, "/hooks", {"limit": 250})
return result.get("data", []) if isinstance(result, dict) else result
def load_webhook_log_timestamps():
"""Read your own webhook receiver log table, grouped by scope.
Replace this with a real query against your receiver's storage. Left as a
stub here since the log table is store-specific infrastructure.
"""
return {}
def reactivate_hook(hook_id):
"""Guarded reactivation. Only call this after confirming the destination
is healthy again. Always wrapped in DRY_RUN.
"""
if DRY_RUN:
log.info("DRY_RUN: would PUT /v3/hooks/%s {'is_active': True}", hook_id)
return None
return bc_put(API_BASE_V3, f"/hooks/{hook_id}", {"is_active": True})
def run():
orders = recent_orders(LOOKBACK_DAYS)
order_timestamps = [o.get("date_modified") or o.get("date_created") for o in orders if o.get("date_created")]
hooks = current_hooks()
webhook_log_timestamps = load_webhook_log_timestamps()
now = datetime.utcnow().isoformat() + "Z"
findings = detect_webhook_gap(order_timestamps, webhook_log_timestamps, hooks, now, STALE_AFTER_MINUTES)
for finding in findings:
log.warning(
"webhook gap: hook_id=%s scope=%s destination=%s is_active=%s reason=%s",
finding["hook_id"], finding["scope"], finding["destination"],
finding["is_active"], finding["reason"],
)
log.info("Done. %d order(s) checked, %d hook finding(s).", len(orders), len(findings))
return findings
if __name__ == "__main__":
run()/**
* Detect BigCommerce order webhooks that stopped firing with no surfaced error.
*
* BigCommerce retries a failing webhook destination on a backoff schedule for
* roughly 48 hours, then permanently sets is_active to false on that
* subscription, emailing only the address on file for the subscribing app.
* Separately, once a destination domain has received 100 or more requests,
* BigCommerce tracks a rolling 2 minute success and failure ratio and
* blocklists the whole domain for 3 minutes if the success rate drops below
* 90 percent, which can fail deliveries even on a hook that still reads
* is_active:true. Neither mechanism raises a dashboard alert. This job pulls
* recent orders, the store's current hook subscriptions, and the store's own
* webhook receiver log, and reports any hook that is deactivated or has gone
* stale with no recent delivery. It never auto-reactivates; repair is a
* separate, guarded, dry-run-respecting step.
*
* Guide: https://www.allanninal.dev/bigcommerce/order-webhooks-stop-firing-silently/
*/
import { pathToFileURL } from "node:url";
const STORE_HASH = process.env.BIGCOMMERCE_STORE_HASH || "example_hash";
const ACCESS_TOKEN = process.env.BIGCOMMERCE_ACCESS_TOKEN || "bc_dummy";
const API_BASE_V2 = `https://api.bigcommerce.com/stores/${STORE_HASH}/v2`;
const API_BASE_V3 = `https://api.bigcommerce.com/stores/${STORE_HASH}/v3`;
const LOOKBACK_DAYS = Number(process.env.LOOKBACK_DAYS || 1);
const STALE_AFTER_MINUTES = Number(process.env.STALE_AFTER_MINUTES || 30);
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
const HEADERS = {
"X-Auth-Token": ACCESS_TOKEN,
"Content-Type": "application/json",
Accept: "application/json",
};
function scopeMatches(scope) {
return scope.startsWith("store/order/") || scope.startsWith("store/customer/");
}
/**
* Pure decision. No network, no side effects.
*
* orderTimestamps: ISO8601 date_created/date_modified values from GET /v2/orders.
* webhookLogTimestamps: {scope: [receivedAt ISO8601, ...]} from the store's own
* webhook receiver log.
* hookRecords: raw items from GET /v3/hooks, each with id, scope, destination,
* is_active, updated_at.
* now: ISO8601 current time, injected for testability.
* staleAfterMinutes: threshold beyond normal delivery latency before a
* still-active hook counts as stale.
*
* Returns a list of finding objects: {hook_id, scope, destination, is_active, reason}
* with reason in "deactivated" or "stale_no_recent_delivery".
*/
export function detectWebhookGap(orderTimestamps, webhookLogTimestamps, hookRecords, now, staleAfterMinutes = 30) {
const findings = [];
const nowMs = new Date(now).getTime();
const orderMs = orderTimestamps.map((t) => new Date(t).getTime()).filter((t) => Number.isFinite(t));
const latestOrder = orderMs.length ? Math.max(...orderMs) : null;
for (const hook of hookRecords) {
const scope = hook.scope || "";
if (!scopeMatches(scope)) continue;
if (hook.is_active === false) {
findings.push({
hook_id: hook.id,
scope,
destination: hook.destination,
is_active: false,
reason: "deactivated",
});
continue;
}
if (latestOrder === null) continue;
const logTimes = (webhookLogTimestamps[scope] || [])
.map((t) => new Date(t).getTime())
.filter((t) => Number.isFinite(t));
const lastReceived = logTimes.length ? Math.max(...logTimes) : null;
const gapReference = lastReceived ?? latestOrder;
const staleCutoff = nowMs - staleAfterMinutes * 60 * 1000;
if (latestOrder > gapReference && gapReference < staleCutoff) {
findings.push({
hook_id: hook.id,
scope,
destination: hook.destination,
is_active: true,
reason: "stale_no_recent_delivery",
});
}
}
return findings;
}
async function bcGet(base, path, params = {}) {
const url = new URL(`${base}${path}`);
for (const [key, value] of Object.entries(params)) {
if (value !== undefined && value !== null) url.searchParams.set(key, value);
}
const res = await fetch(url, { headers: HEADERS });
if (!res.ok) throw new Error(`BigCommerce ${res.status}`);
const text = await res.text();
return text ? JSON.parse(text) : [];
}
async function bcPut(base, path, body) {
const res = await fetch(`${base}${path}`, {
method: "PUT",
headers: HEADERS,
body: JSON.stringify(body),
});
if (!res.ok) throw new Error(`BigCommerce ${res.status}`);
return res.json();
}
async function recentOrders(lookbackDays) {
const orders = [];
let page = 1;
while (true) {
const batch = await bcGet(API_BASE_V2, "/orders", {
min_date_created: `-${lookbackDays} days`,
sort: "date_created:desc",
page,
limit: 50,
});
if (!batch.length) return orders;
orders.push(...batch);
page += 1;
}
}
async function currentHooks() {
const result = await bcGet(API_BASE_V3, "/hooks", { limit: 250 });
return Array.isArray(result) ? result : result.data || [];
}
function loadWebhookLogTimestamps() {
// Replace this with a real query against your receiver's storage. Left as
// a stub here since the log table is store-specific infrastructure.
return {};
}
async function reactivateHook(hookId) {
if (DRY_RUN) {
console.log(`DRY_RUN: would PUT /v3/hooks/${hookId} {"is_active": true}`);
return null;
}
return bcPut(API_BASE_V3, `/hooks/${hookId}`, { is_active: true });
}
export async function run() {
const orders = await recentOrders(LOOKBACK_DAYS);
const orderTimestamps = orders.map((o) => o.date_modified || o.date_created).filter(Boolean);
const hooks = await currentHooks();
const webhookLogTimestamps = loadWebhookLogTimestamps();
const now = new Date().toISOString();
const findings = detectWebhookGap(orderTimestamps, webhookLogTimestamps, hooks, now, STALE_AFTER_MINUTES);
for (const finding of findings) {
console.warn(
`webhook gap: hook_id=${finding.hook_id} scope=${finding.scope} destination=${finding.destination} is_active=${finding.is_active} reason=${finding.reason}`
);
}
console.log(`Done. ${orders.length} order(s) checked, ${findings.length} hook finding(s).`);
return findings;
}
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((err) => { console.error(err); process.exit(1); });
}Add a test
The gap detector is the part most worth testing, because it decides whether a real hook gets reported as broken. Because detect_webhook_gap takes only plain values, including an injected now, and returns a plain list, the test needs no network and no BigCommerce store. It just feeds in plain objects and checks the findings.
from detect_webhook_gap import detect_webhook_gap
NOW = "2026-07-10T12:00:00Z"
def hook(id_=1, scope="store/order/statusUpdated", is_active=True, destination="https://example.com/hooks"):
return {"id": id_, "scope": scope, "destination": destination, "is_active": is_active, "updated_at": NOW}
def test_no_findings_when_everything_is_current():
orders = ["2026-07-10T11:55:00Z"]
log = {"store/order/statusUpdated": ["2026-07-10T11:56:00Z"]}
assert detect_webhook_gap(orders, log, [hook()], NOW) == []
def test_flags_deactivated_hook_regardless_of_delivery_log():
orders = ["2026-07-10T11:55:00Z"]
log = {"store/order/statusUpdated": ["2026-07-10T11:56:00Z"]}
findings = detect_webhook_gap(orders, log, [hook(is_active=False)], NOW)
assert len(findings) == 1
assert findings[0]["reason"] == "deactivated"
assert findings[0]["is_active"] is False
def test_flags_stale_active_hook_with_no_recent_delivery():
orders = ["2026-07-10T11:55:00Z"]
log = {"store/order/statusUpdated": ["2026-07-10T10:00:00Z"]}
findings = detect_webhook_gap(orders, log, [hook()], NOW, stale_after_minutes=30)
assert len(findings) == 1
assert findings[0]["reason"] == "stale_no_recent_delivery"
def test_no_finding_when_gap_is_within_stale_threshold():
orders = ["2026-07-10T11:55:00Z"]
log = {"store/order/statusUpdated": ["2026-07-10T11:45:00Z"]}
assert detect_webhook_gap(orders, log, [hook()], NOW, stale_after_minutes=30) == []
def test_ignores_hooks_outside_order_and_customer_scope():
orders = ["2026-07-10T11:55:00Z"]
log = {}
findings = detect_webhook_gap(orders, log, [hook(scope="store/product/updated", is_active=False)], NOW)
assert findings == []
def test_no_findings_when_there_are_no_orders_at_all():
assert detect_webhook_gap([], {}, [hook()], NOW) == []import { test } from "node:test";
import assert from "node:assert/strict";
import { detectWebhookGap } from "./detect-webhook-gap.js";
const NOW = "2026-07-10T12:00:00Z";
const hook = ({ id = 1, scope = "store/order/statusUpdated", isActive = true, destination = "https://example.com/hooks" } = {}) => ({
id, scope, destination, is_active: isActive, updated_at: NOW,
});
test("no findings when everything is current", () => {
const orders = ["2026-07-10T11:55:00Z"];
const log = { "store/order/statusUpdated": ["2026-07-10T11:56:00Z"] };
assert.deepEqual(detectWebhookGap(orders, log, [hook()], NOW), []);
});
test("flags deactivated hook regardless of delivery log", () => {
const orders = ["2026-07-10T11:55:00Z"];
const log = { "store/order/statusUpdated": ["2026-07-10T11:56:00Z"] };
const findings = detectWebhookGap(orders, log, [hook({ isActive: false })], NOW);
assert.equal(findings.length, 1);
assert.equal(findings[0].reason, "deactivated");
assert.equal(findings[0].is_active, false);
});
test("flags stale active hook with no recent delivery", () => {
const orders = ["2026-07-10T11:55:00Z"];
const log = { "store/order/statusUpdated": ["2026-07-10T10:00:00Z"] };
const findings = detectWebhookGap(orders, log, [hook()], NOW, 30);
assert.equal(findings.length, 1);
assert.equal(findings[0].reason, "stale_no_recent_delivery");
});
test("no finding when gap is within stale threshold", () => {
const orders = ["2026-07-10T11:55:00Z"];
const log = { "store/order/statusUpdated": ["2026-07-10T11:45:00Z"] };
assert.deepEqual(detectWebhookGap(orders, log, [hook()], NOW, 30), []);
});
test("ignores hooks outside order and customer scope", () => {
const orders = ["2026-07-10T11:55:00Z"];
const findings = detectWebhookGap(orders, {}, [hook({ scope: "store/product/updated", isActive: false })], NOW);
assert.deepEqual(findings, []);
});
test("no findings when there are no orders at all", () => {
assert.deepEqual(detectWebhookGap([], {}, [hook()], NOW), []);
});Case studies
The integration that broke its own auth check on deploy
A fulfillment integration redeployed its webhook receiver with a stricter signature check, and a header mismatch meant every delivery started failing. Nobody noticed for two days, since orders kept flowing and the store's admin showed nothing unusual. Then the deactivation email landed in an inbox nobody read regularly, and by the time someone found it, the fulfillment queue was a full day behind.
Now a scheduled check compares order timestamps against the webhook receiver log every 15 minutes. The first run after the auth check broke immediately reported a stale_no_recent_delivery finding, hours before the 48 hour deactivation would have even kicked in.
The brief outage that looked like a bigger one
During a routine deploy, a receiver returned 5xx for about ninety seconds while new pods came up. That was enough to drop the domain's rolling success rate below 90 percent, and BigCommerce blocklisted the whole domain for 3 minutes. Every hook pointed at that domain, still showing is_active true, failed to deliver during the window, and the on-call engineer briefly assumed a full outage.
The gap detector flagged the affected scopes as stale_no_recent_delivery with is_active still true, which matched the domain blocklisting pattern rather than a deactivation, so the team knew immediately it was a transient issue and not a broken subscription needing repair.
After this runs on a schedule, a deactivated hook or a stale, silently-blocked one is a named finding within minutes, not a mystery discovered days later from a backed-up fulfillment queue. Nothing gets reactivated automatically. A human confirms the receiver is healthy first, and only a long-disabled hook gets recreated fresh rather than trusted as-is.
FAQ
Why did my BigCommerce order webhooks just stop, with nothing in the control panel?
BigCommerce retries a failing webhook destination on a backoff schedule for roughly 48 hours and then permanently flips that subscription's is_active to false. The only notice is an email to the address on file for the subscribing app, nothing appears in the store control panel, so orders keep being created normally while the hook silently stops delivering.
My hooks still show is_active true, so why are events missing?
BigCommerce also tracks a rolling 2 minute success and failure ratio per destination domain once it has sent 100 or more requests, and will blocklist the whole domain for 3 minutes if the success rate drops below 90 percent. During that window deliveries fail even though the hook record itself still reads is_active true, which looks identical to everything just stopping.
Is it safe to just flip is_active back to true on a deactivated hook?
No. The underlying delivery failure, a bad endpoint, expired auth, or a TLS or certificate issue, is still unresolved, so reactivating blindly just restarts the same failure loop. Confirm the destination returns a healthy response first, then reactivate with a guarded PUT wrapped in a dry run, and if the hook has been disabled long enough to risk 90 day pruning, recreate it fresh with POST instead of trusting the old record.
Related field notes
Citations
On the problem:
- BigCommerce Support: how to avoid webhooks deactivation, it happens periodically. support.bigcommerce.com how to avoid webhooks deactivation
- BigCommerce Support: webhooks stopped firing today. support.bigcommerce.com webhooks stopped firing today
- BigCommerce Support: order webhook not firing. support.bigcommerce.com order webhook not firing
On the solution:
- BigCommerce Developer Center: webhooks overview, retry schedule, and deactivation. developer.bigcommerce.com webhooks overview
- BigCommerce Docs: webhooks overview. docs.bigcommerce.com webhooks overview
- BigCommerce Developer Center: webhooks admin, the /v3/hooks endpoints. developer.bigcommerce.com webhooks admin
Stuck on a tricky one?
If you have a problem in BigCommerce orders, payments, webhooks, inventory, or fulfillment that you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this catch a silent webhook gap?
If this saved you from a mystery fulfillment backlog or caught a deactivated hook before it cost you a day, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi