Diagnostic Webhooks
Webhook domain blocklisted after low delivery success ratio
Deliveries to a domain that was working fine a minute ago suddenly stop, for every hook registered on that host, not just the one that was failing. BigCommerce tracks a rolling success ratio per destination domain, and once it dips below 90 percent it blocklists the whole domain for 3 minutes. There is no API call to lift that block early. Here is why it happens, how to detect it from your own request log, and the one safe repair that is left once the dust settles.
BigCommerce's webhook dispatcher computes a rolling success versus failure ratio per destination domain over a sliding 2-minute window, but only starts evaluating it once at least 100 requests have been sent to that domain within the window. If the ratio falls below 90 percent, usually because the receiving endpoint is slow, returning non-200 responses, or intermittently down, BigCommerce blocklists the entire domain for 3 minutes, not just the failing hook. Because the block is domain-scoped, one flaky path like /webhooks/orders can starve delivery to an unrelated, healthy hook such as /webhooks/inventory on the same host. There is no safe API call to lift the block early or force a redelivery, it self-expires and BigCommerce requeues automatically. Run a small Python or Node.js script that lists hooks with GET /v3/hooks, correlates them against your own app's request log to compute rolling success ratios per domain, flags any domain at risk of blocklisting and any hook BigCommerce already deactivated, and only ever writes PUT /v3/hooks/{hook_id} with {"is_active": true} once the destination is confirmed healthy again. Full code, tests, and a dry run guard are below.
The problem in plain words
BigCommerce fans out webhook deliveries to whatever destinations a store has registered under /v3/hooks. To protect itself from hammering a destination that is clearly struggling, the dispatcher watches a rolling 2-minute window of delivery attempts per destination domain. Once that window has seen at least 100 requests, it checks the success ratio. If fewer than 90 percent came back as a successful delivery, the entire domain, every hook pointed at it, gets blocklisted for 3 minutes.
The part that catches people off guard is the scope of the block. It is not per hook and not per path, it is per domain. If a store has one hook on /webhooks/orders that is timing out because the receiving app is overloaded, and a completely separate, perfectly healthy hook on /webhooks/inventory on that same host, both stop receiving deliveries for the next 3 minutes. The inventory hook did nothing wrong. It just shares a domain with the hook that did.
If the underlying instability on the endpoint does not clear up, a second and unrelated failure path can also kick in: BigCommerce separately tracks delivery attempts per hook and, after repeated failures over roughly 48 hours or 11 retries, deactivates that specific hook by flipping is_active to false. That one is permanent until someone re-enables it. The 3-minute domain block and the 48-hour deactivation are two different mechanisms, but a persistently flaky endpoint can trigger both.
Why it happens
BigCommerce does not evaluate every hook in isolation while deciding whether to keep delivering. A few things line up to produce this behavior:
- The dispatcher tracks success and failure per destination domain, not per registered hook, so the accounting that decides "block or don't block" is coarser than the accounting a store owner usually thinks about.
- The ratio only kicks in once the domain has seen at least 100 requests inside the current rolling 2-minute window. A domain with light traffic can have a terrible day and never trip the rule, while a high-traffic domain can trip it from a short burst of slowness.
- A receiving endpoint that is slow enough to time out, or that occasionally returns a non-200 status during a deploy or a database hiccup, counts as a failure toward that same domain-wide ratio, even if most of its own hooks are fine.
- If the instability behind the low ratio does not clear up, the separate per-hook failure count keeps climbing in the background, and once it crosses the roughly 48-hour or 11-retry threshold, BigCommerce deactivates that specific hook permanently by setting
is_activeto false, independent of the 3-minute domain block.
BigCommerce does not expose a delivery-log or success-rate endpoint for webhooks, so none of this shows up by simply listing hooks. See the citations at the end for the official webhook docs and the support thread where merchants ask how to avoid the periodic deactivation.
There is nothing in the BigCommerce API to prevent or lift this block, because it is not a bug, it is the platform protecting itself and every other store sharing that infrastructure from a slow or broken destination. The only thing worth automating is detection: correlate GET /v3/hooks against your own receiving app's request log to compute rolling success ratios per domain, flag any domain trending toward the 90 percent floor before it trips, and separately flag any hook that already got auto-deactivated with is_active: false. The one safe write left is re-enabling a hook once its endpoint is confirmed healthy again.
The fix, as a flow
We do not fight the domain block, it is not ours to lift. We add a job that lists registered hooks, pulls matching request-log entries for each destination domain, computes the rolling success ratio, and reports anything at risk or already deactivated, only ever writing back to re-enable a hook that BigCommerce turned off and that we have confirmed is healthy again.
Build it step by step
Get a store hash and an API access token
Create an API account in your BigCommerce control panel under Settings, API, or use the store's existing app credentials. Grant it Webhooks (modify) scope so it can list hooks and, when needed, re-enable one. You need the store hash from your control panel URL and the access token, sent on every call as the X-Auth-Token header. Keep both in environment variables, never in the file.
pip install requests
export BIGCOMMERCE_STORE_HASH="abc123"
export BIGCOMMERCE_ACCESS_TOKEN="..."
export MIN_SAMPLE="100"
export SUCCESS_THRESHOLD="0.90"
export DRY_RUN="true" # start safe, change to false to write// Node 18+ has fetch built in, no dependencies needed
export BIGCOMMERCE_STORE_HASH="abc123"
export BIGCOMMERCE_ACCESS_TOKEN="..."
export MIN_SAMPLE="100"
export SUCCESS_THRESHOLD="0.90"
export DRY_RUN="true" // start safe, change to false to writeTalk to the V3 Hooks REST API
Every call goes to https://api.bigcommerce.com/stores/{store_hash}/v3/ with the token in the X-Auth-Token header. A small helper handles GET and PUT and raises on a non-2xx response, and paginates through meta.pagination since /v3/hooks wraps its results in a {data, meta} envelope.
import os, requests
STORE_HASH = os.environ["BIGCOMMERCE_STORE_HASH"]
ACCESS_TOKEN = os.environ["BIGCOMMERCE_ACCESS_TOKEN"]
API_BASE = f"https://api.bigcommerce.com/stores/{STORE_HASH}/v3"
HEADERS = {
"X-Auth-Token": ACCESS_TOKEN,
"Content-Type": "application/json",
"Accept": "application/json",
}
def bc_get(path, params=None):
r = requests.get(f"{API_BASE}{path}", headers=HEADERS, params=params or {}, timeout=30)
r.raise_for_status()
return r.json()
def bc_put(path, body):
r = requests.put(f"{API_BASE}{path}", headers=HEADERS, json=body, timeout=30)
r.raise_for_status()
return r.json()const STORE_HASH = process.env.BIGCOMMERCE_STORE_HASH;
const ACCESS_TOKEN = process.env.BIGCOMMERCE_ACCESS_TOKEN;
const API_BASE = `https://api.bigcommerce.com/stores/${STORE_HASH}/v3`;
const HEADERS = {
"X-Auth-Token": ACCESS_TOKEN,
"Content-Type": "application/json",
Accept: "application/json",
};
async function bcGet(path, params = {}) {
const url = new URL(`${API_BASE}${path}`);
for (const [key, value] of Object.entries(params)) url.searchParams.set(key, value);
const res = await fetch(url, { headers: HEADERS });
if (!res.ok) throw new Error(`BigCommerce ${res.status}`);
return res.json();
}
async function bcPut(path, body) {
const res = await fetch(`${API_BASE}${path}`, { method: "PUT", headers: HEADERS, body: JSON.stringify(body) });
if (!res.ok) throw new Error(`BigCommerce ${res.status}`);
return res.json();
}List every registered hook, paginated
Call GET /v3/hooks, paging through meta.pagination, and read id, destination, scope, is_active, and updated_at for each. BigCommerce has no delivery-log or success-rate endpoint, so this list only tells us what is registered and what BigCommerce itself already deactivated. Everything about the rolling success ratio has to come from your own app's request log.
def list_hooks():
page = 1
while True:
payload = bc_get("/hooks", {"page": page, "limit": 50})
hooks = payload.get("data") or []
if not hooks:
return
for hook in hooks:
yield hook
pagination = (payload.get("meta") or {}).get("pagination") or {}
if page >= (pagination.get("total_pages") or page):
return
page += 1async function* listHooks() {
let page = 1;
while (true) {
const payload = await bcGet("/hooks", { page, limit: 50 });
const hooks = payload.data || [];
if (!hooks.length) return;
for (const hook of hooks) yield hook;
const pagination = (payload.meta || {}).pagination || {};
if (page >= (pagination.total_pages || page)) return;
page += 1;
}
}Decide, with one pure function
Keep the evaluation in its own function that takes the delivery attempts recorded for a single rolling 2-minute window, one entry per attempt with its domain and status code, and returns the total and success ratio per domain, along with whether that domain is at risk of the block. It only starts judging a domain once it has seen at least 100 requests, exactly matching BigCommerce's own rule, so a quiet domain never gets falsely flagged just because a couple of requests failed.
def evaluate_webhook_health(window_requests, min_sample=100, threshold=0.90):
by_domain = {}
for entry in window_requests or []:
domain = entry["domain"]
bucket = by_domain.setdefault(domain, {"total": 0, "success": 0})
bucket["total"] += 1
if 200 <= entry["status_code"] < 300:
bucket["success"] += 1
results = {}
for domain, bucket in by_domain.items():
total = bucket["total"]
if total < min_sample:
results[domain] = {
"domain": domain, "total": total,
"success_ratio": None, "at_risk": False,
}
continue
ratio = bucket["success"] / total
results[domain] = {
"domain": domain, "total": total,
"success_ratio": ratio, "at_risk": ratio < threshold,
}
return resultsexport function evaluateWebhookHealth(windowRequests, minSample = 100, threshold = 0.90) {
const byDomain = new Map();
for (const entry of windowRequests || []) {
const domain = entry.domain;
const bucket = byDomain.get(domain) || { total: 0, success: 0 };
bucket.total += 1;
if (entry.status_code >= 200 && entry.status_code < 300) bucket.success += 1;
byDomain.set(domain, bucket);
}
const results = {};
for (const [domain, bucket] of byDomain.entries()) {
if (bucket.total < minSample) {
results[domain] = { domain, total: bucket.total, success_ratio: null, at_risk: false };
continue;
}
const ratio = bucket.success / bucket.total;
results[domain] = { domain, total: bucket.total, success_ratio: ratio, at_risk: ratio < threshold };
}
return results;
}Report, never repair, the domain block
There is no API call that lifts a domain blocklist or forces BigCommerce to redeliver missed events. The block self-expires in 3 minutes and BigCommerce requeues automatically. So for every domain the pure function marks at_risk, emit an alert with the domain, the computed success ratio, and the window timestamps. That is the entire action, log it loudly enough that a human notices before the 90 percent floor is crossed.
def report_at_risk_domains(health_by_domain, window_start, window_end, log):
for result in health_by_domain.values():
if not result["at_risk"]:
continue
log.warning(
"Domain %s at risk of blocklisting. success_ratio=%.3f total=%d window=%s..%s",
result["domain"], result["success_ratio"], result["total"],
window_start, window_end,
)function reportAtRiskDomains(healthByDomain, windowStart, windowEnd) {
for (const result of Object.values(healthByDomain)) {
if (!result.at_risk) continue;
console.warn(
`Domain ${result.domain} at risk of blocklisting. ` +
`success_ratio=${result.success_ratio.toFixed(3)} total=${result.total} ` +
`window=${windowStart}..${windowEnd}`
);
}
}Re-enable a deactivated hook, only after a health check, guarded by dry run
When GET /v3/hooks shows is_active: false, that hook already crossed the separate 48-hour or 11-retry exhaustion threshold and BigCommerce turned it off for good. The only corrective write is PUT /v3/hooks/{hook_id} with {"is_active": true}, and it should only run after a synthetic health-check request to the destination comes back 200. With DRY_RUN=true, log the intended PUT and skip the network call.
def health_check_ok(destination):
try:
r = requests.get(destination, timeout=10)
return r.status_code == 200
except requests.RequestException:
return False
def reenable_hook(hook_id, destination, dry_run, log):
if not health_check_ok(destination):
log.warning("Skipping re-enable for hook %s, health check failed.", hook_id)
return False
if dry_run:
log.info("DRY_RUN: would PUT /hooks/%s {'is_active': True}", hook_id)
return True
bc_put(f"/hooks/{hook_id}", {"is_active": True})
log.info("Re-enabled hook %s after passing health check.", hook_id)
return Trueasync function healthCheckOk(destination) {
try {
const res = await fetch(destination, { method: "GET" });
return res.status === 200;
} catch {
return false;
}
}
async function reenableHook(hookId, destination, dryRun) {
if (!(await healthCheckOk(destination))) {
console.warn(`Skipping re-enable for hook ${hookId}, health check failed.`);
return false;
}
if (dryRun) {
console.log(`DRY_RUN: would PUT /hooks/${hookId} {"is_active": true}`);
return true;
}
await bcPut(`/hooks/${hookId}`, { is_active: true });
console.log(`Re-enabled hook ${hookId} after passing health check.`);
return true;
}Always start with DRY_RUN=true. Never write to a hook other than flipping is_active back to true, and never do even that without a passing synthetic health check first. There is no API call that lifts a domain blocklist, and trying to force one is not a real operation, it is a no-op at best and a misleading log entry at worst.
The full code
Here is the complete script in one file for each language. It reads settings from the environment, paginates /v3/hooks, evaluates rolling success ratios from a supplied request log, reports every domain at risk and every hook already deactivated, and only ever writes to re-enable a hook once its endpoint passes a live health check, respecting the dry run flag throughout.
View this code on GitHub Full runnable folder with tests in the bigcommerce-fixes repo.
"""Detect a BigCommerce webhook domain at risk of being blocklisted.
BigCommerce's webhook dispatcher tracks a rolling success versus failure
ratio per destination domain over a sliding 2-minute window, evaluated only
once at least 100 requests have landed in that window. If the ratio drops
below 90 percent, typically because the receiving endpoint is slow,
returning non-200s, or intermittently down, BigCommerce blocklists the
entire domain for 3 minutes, not just the failing hook. Because the block
is domain scoped, one flaky path (for example /webhooks/orders) can starve
delivery to an unrelated healthy hook (for example /webhooks/inventory) on
the same host. If the instability persists, the same webhook can also hit
the separate 48-hour / 11-retry exhaustion path and get permanently
deactivated (is_active=false).
There is no safe API call to lift a domain blocklist or force a redelivery,
the 3-minute block self-expires and BigCommerce requeues automatically, so
this script never tries. It lists registered hooks with GET /v3/hooks,
correlates them against your own app's request log to compute rolling
success ratios per domain, reports any domain at risk and any hook already
deactivated, and makes exactly one kind of write: re-enabling a hook with
PUT /v3/hooks/{hook_id} and {"is_active": true}, and only after a synthetic
health-check request to the destination returns 200. Guarded by DRY_RUN.
Guide: https://www.allanninal.dev/bigcommerce/webhook-domain-blocklisted-low-success-ratio/
"""
import os
import logging
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("webhook_domain_health")
STORE_HASH = os.environ["BIGCOMMERCE_STORE_HASH"]
ACCESS_TOKEN = os.environ["BIGCOMMERCE_ACCESS_TOKEN"]
API_BASE = f"https://api.bigcommerce.com/stores/{STORE_HASH}/v3"
MIN_SAMPLE = int(os.environ.get("MIN_SAMPLE", "100"))
SUCCESS_THRESHOLD = float(os.environ.get("SUCCESS_THRESHOLD", "0.90"))
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"
HEADERS = {
"X-Auth-Token": ACCESS_TOKEN,
"Content-Type": "application/json",
"Accept": "application/json",
}
def bc_get(path, params=None):
r = requests.get(f"{API_BASE}{path}", headers=HEADERS, params=params or {}, timeout=30)
r.raise_for_status()
return r.json()
def bc_put(path, body):
r = requests.put(f"{API_BASE}{path}", headers=HEADERS, json=body, timeout=30)
r.raise_for_status()
return r.json()
def evaluate_webhook_health(window_requests: list, min_sample: int = 100, threshold: float = 0.90) -> dict:
"""Pure. No network, no side effects.
window_requests: list of {"timestamp": float, "domain": str, "status_code": int}
entries for a single rolling 2-minute window, one per delivery attempt.
Returns, per domain, {"domain": str, "total": int, "success_ratio": float | None, "at_risk": bool}.
success_ratio is None (and at_risk False) when total < min_sample, matching
BigCommerce's rule that the ratio is not evaluated until 100 requests are seen.
at_risk is True only when total >= min_sample and success_ratio < threshold.
"""
by_domain = {}
for entry in window_requests or []:
domain = entry["domain"]
bucket = by_domain.setdefault(domain, {"total": 0, "success": 0})
bucket["total"] += 1
if 200 <= entry["status_code"] < 300:
bucket["success"] += 1
results = {}
for domain, bucket in by_domain.items():
total = bucket["total"]
if total < min_sample:
results[domain] = {
"domain": domain,
"total": total,
"success_ratio": None,
"at_risk": False,
}
continue
ratio = bucket["success"] / total
results[domain] = {
"domain": domain,
"total": total,
"success_ratio": ratio,
"at_risk": ratio < threshold,
}
return results
def list_hooks():
"""Page through every registered hook via GET /v3/hooks."""
page = 1
while True:
payload = bc_get("/hooks", {"page": page, "limit": 50})
hooks = payload.get("data") or []
if not hooks:
return
for hook in hooks:
yield hook
pagination = (payload.get("meta") or {}).get("pagination") or {}
if page >= (pagination.get("total_pages") or page):
return
page += 1
def health_check_ok(destination):
try:
r = requests.get(destination, timeout=10)
return r.status_code == 200
except requests.RequestException:
return False
def reenable_hook(hook_id, destination):
"""The only safe write: flip is_active back to True, and only when healthy."""
if not health_check_ok(destination):
log.warning("Skipping re-enable for hook %s, health check failed.", hook_id)
return False
if DRY_RUN:
log.info("DRY_RUN: would PUT /hooks/%s {'is_active': True}", hook_id)
return True
bc_put(f"/hooks/{hook_id}", {"is_active": True})
log.info("Re-enabled hook %s after passing health check.", hook_id)
return True
def fetch_recent_request_log():
"""Placeholder for your app's own request log lookup.
BigCommerce exposes no delivery-log or success-rate endpoint, so this
must come from wherever your receiving app records each webhook
request's timestamp, destination domain, and response status code.
Replace this with a real query against your logs or metrics store.
"""
return []
def run():
window_requests = fetch_recent_request_log()
health_by_domain = evaluate_webhook_health(window_requests, MIN_SAMPLE, SUCCESS_THRESHOLD)
at_risk_count = 0
for result in health_by_domain.values():
if not result["at_risk"]:
continue
at_risk_count += 1
log.warning(
"Domain %s at risk of blocklisting. success_ratio=%.3f total=%d",
result["domain"], result["success_ratio"], result["total"],
)
reenabled = 0
for hook in list_hooks():
if hook.get("is_active"):
continue
hook_id = hook["id"]
destination = hook.get("destination")
log.warning(
"Hook %s is deactivated (is_active=false). destination=%s updated_at=%s",
hook_id, destination, hook.get("updated_at"),
)
if destination and reenable_hook(hook_id, destination):
reenabled += 1
log.info(
"Done. %d domain(s) at risk, %d hook(s) %s.",
at_risk_count, reenabled, "to re-enable" if DRY_RUN else "re-enabled",
)
if __name__ == "__main__":
run()/**
* Detect a BigCommerce webhook domain at risk of being blocklisted.
*
* BigCommerce's webhook dispatcher tracks a rolling success versus failure
* ratio per destination domain over a sliding 2-minute window, evaluated
* only once at least 100 requests have landed in that window. If the ratio
* drops below 90 percent, typically because the receiving endpoint is slow,
* returning non-200s, or intermittently down, BigCommerce blocklists the
* entire domain for 3 minutes, not just the failing hook. Because the block
* is domain scoped, one flaky path (for example /webhooks/orders) can starve
* delivery to an unrelated healthy hook (for example /webhooks/inventory) on
* the same host. If the instability persists, the same webhook can also hit
* the separate 48-hour / 11-retry exhaustion path and get permanently
* deactivated (is_active=false).
*
* There is no safe API call to lift a domain blocklist or force a
* redelivery, the 3-minute block self-expires and BigCommerce requeues
* automatically, so this script never tries. It lists registered hooks with
* GET /v3/hooks, correlates them against your own app's request log to
* compute rolling success ratios per domain, reports any domain at risk and
* any hook already deactivated, and makes exactly one kind of write:
* re-enabling a hook with PUT /v3/hooks/{hook_id} and {"is_active": true},
* and only after a synthetic health-check request to the destination
* returns 200. Guarded by DRY_RUN.
*
* Guide: https://www.allanninal.dev/bigcommerce/webhook-domain-blocklisted-low-success-ratio/
*/
import { pathToFileURL } from "node:url";
const STORE_HASH = process.env.BIGCOMMERCE_STORE_HASH || "example_hash";
const ACCESS_TOKEN = process.env.BIGCOMMERCE_ACCESS_TOKEN || "bc_dummy";
const API_BASE = `https://api.bigcommerce.com/stores/${STORE_HASH}/v3`;
const MIN_SAMPLE = Number(process.env.MIN_SAMPLE || 100);
const SUCCESS_THRESHOLD = Number(process.env.SUCCESS_THRESHOLD || 0.90);
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
const HEADERS = {
"X-Auth-Token": ACCESS_TOKEN,
"Content-Type": "application/json",
Accept: "application/json",
};
/**
* Pure. No network, no side effects.
*
* windowRequests: list of {timestamp, domain, status_code} entries for a
* single rolling 2-minute window, one per delivery attempt.
* Returns, per domain, {domain, total, success_ratio, at_risk}.
* success_ratio is null (and at_risk false) when total < minSample, matching
* BigCommerce's rule that the ratio is not evaluated until 100 requests are
* seen. at_risk is true only when total >= minSample and success_ratio < threshold.
*/
export function evaluateWebhookHealth(windowRequests, minSample = 100, threshold = 0.90) {
const byDomain = new Map();
for (const entry of windowRequests || []) {
const domain = entry.domain;
const bucket = byDomain.get(domain) || { total: 0, success: 0 };
bucket.total += 1;
if (entry.status_code >= 200 && entry.status_code < 300) bucket.success += 1;
byDomain.set(domain, bucket);
}
const results = {};
for (const [domain, bucket] of byDomain.entries()) {
if (bucket.total < minSample) {
results[domain] = { domain, total: bucket.total, success_ratio: null, at_risk: false };
continue;
}
const ratio = bucket.success / bucket.total;
results[domain] = { domain, total: bucket.total, success_ratio: ratio, at_risk: ratio < threshold };
}
return results;
}
async function bcGet(path, params = {}) {
const url = new URL(`${API_BASE}${path}`);
for (const [key, value] of Object.entries(params)) {
if (value !== undefined && value !== null) url.searchParams.set(key, value);
}
const res = await fetch(url, { headers: HEADERS });
if (!res.ok) throw new Error(`BigCommerce ${res.status}`);
return res.json();
}
async function bcPut(path, body) {
const res = await fetch(`${API_BASE}${path}`, {
method: "PUT",
headers: HEADERS,
body: JSON.stringify(body),
});
if (!res.ok) throw new Error(`BigCommerce ${res.status}`);
return res.json();
}
async function* listHooks() {
let page = 1;
while (true) {
const payload = await bcGet("/hooks", { page, limit: 50 });
const hooks = payload.data || [];
if (!hooks.length) return;
for (const hook of hooks) yield hook;
const pagination = (payload.meta || {}).pagination || {};
if (page >= (pagination.total_pages || page)) return;
page += 1;
}
}
async function healthCheckOk(destination) {
try {
const res = await fetch(destination, { method: "GET" });
return res.status === 200;
} catch {
return false;
}
}
async function reenableHook(hookId, destination) {
if (!(await healthCheckOk(destination))) {
console.warn(`Skipping re-enable for hook ${hookId}, health check failed.`);
return false;
}
if (DRY_RUN) {
console.log(`DRY_RUN: would PUT /hooks/${hookId} {"is_active": true}`);
return true;
}
await bcPut(`/hooks/${hookId}`, { is_active: true });
console.log(`Re-enabled hook ${hookId} after passing health check.`);
return true;
}
/**
* Placeholder for your app's own request log lookup. BigCommerce exposes no
* delivery-log or success-rate endpoint, so this must come from wherever
* your receiving app records each webhook request's timestamp, destination
* domain, and response status code. Replace with a real query.
*/
async function fetchRecentRequestLog() {
return [];
}
export async function run() {
const windowRequests = await fetchRecentRequestLog();
const healthByDomain = evaluateWebhookHealth(windowRequests, MIN_SAMPLE, SUCCESS_THRESHOLD);
let atRiskCount = 0;
for (const result of Object.values(healthByDomain)) {
if (!result.at_risk) continue;
atRiskCount += 1;
console.warn(
`Domain ${result.domain} at risk of blocklisting. ` +
`success_ratio=${result.success_ratio.toFixed(3)} total=${result.total}`
);
}
let reenabled = 0;
for await (const hook of listHooks()) {
if (hook.is_active) continue;
const hookId = hook.id;
const destination = hook.destination;
console.warn(
`Hook ${hookId} is deactivated (is_active=false). destination=${destination} updated_at=${hook.updated_at}`
);
if (destination && (await reenableHook(hookId, destination))) {
reenabled += 1;
}
}
console.log(
`Done. ${atRiskCount} domain(s) at risk, ${reenabled} hook(s) ${DRY_RUN ? "to re-enable" : "re-enabled"}.`
);
}
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((err) => { console.error(err); process.exit(1); });
}Add a test
The decision rule worth testing is the rolling-window evaluation, because it decides whether a domain gets flagged as at risk. Because evaluate_webhook_health takes only a plain list of dictionaries and returns a plain dictionary, the test needs no network and no BigCommerce store. It just feeds in plain request-log entries and checks the answer.
from webhook_domain_health import evaluate_webhook_health
def make_requests(domain, total, failures):
entries = []
for i in range(total):
status = 500 if i < failures else 200
entries.append({"timestamp": float(i), "domain": domain, "status_code": status})
return entries
def test_domain_below_sample_size_is_not_evaluated():
requests_ = make_requests("shop-a.example.com", 40, 40)
result = evaluate_webhook_health(requests_)
entry = result["shop-a.example.com"]
assert entry["total"] == 40
assert entry["success_ratio"] is None
assert entry["at_risk"] is False
def test_domain_at_or_above_sample_with_low_ratio_is_at_risk():
requests_ = make_requests("shop-b.example.com", 100, 15)
result = evaluate_webhook_health(requests_)
entry = result["shop-b.example.com"]
assert entry["total"] == 100
assert entry["success_ratio"] == 0.85
assert entry["at_risk"] is True
def test_domain_at_sample_with_healthy_ratio_is_not_at_risk():
requests_ = make_requests("shop-c.example.com", 120, 5)
result = evaluate_webhook_health(requests_)
entry = result["shop-c.example.com"]
assert entry["total"] == 120
assert round(entry["success_ratio"], 4) == round((120 - 5) / 120, 4)
assert entry["at_risk"] is False
def test_ratio_exactly_at_threshold_is_not_at_risk():
requests_ = make_requests("shop-d.example.com", 100, 10)
result = evaluate_webhook_health(requests_)
entry = result["shop-d.example.com"]
assert entry["success_ratio"] == 0.90
assert entry["at_risk"] is False
def test_multiple_domains_are_evaluated_independently():
healthy = make_requests("healthy.example.com", 100, 2)
flaky = make_requests("flaky.example.com", 150, 40)
result = evaluate_webhook_health(healthy + flaky)
assert result["healthy.example.com"]["at_risk"] is False
assert result["flaky.example.com"]["at_risk"] is True
def test_empty_input_returns_empty_dict():
assert evaluate_webhook_health([]) == {}import { test } from "node:test";
import assert from "node:assert/strict";
import { evaluateWebhookHealth } from "./webhook-domain-health.js";
function makeRequests(domain, total, failures) {
const entries = [];
for (let i = 0; i < total; i++) {
const status = i < failures ? 500 : 200;
entries.push({ timestamp: i, domain, status_code: status });
}
return entries;
}
test("domain below sample size is not evaluated", () => {
const result = evaluateWebhookHealth(makeRequests("shop-a.example.com", 40, 40));
const entry = result["shop-a.example.com"];
assert.equal(entry.total, 40);
assert.equal(entry.success_ratio, null);
assert.equal(entry.at_risk, false);
});
test("domain at or above sample with low ratio is at risk", () => {
const result = evaluateWebhookHealth(makeRequests("shop-b.example.com", 100, 15));
const entry = result["shop-b.example.com"];
assert.equal(entry.total, 100);
assert.equal(entry.success_ratio, 0.85);
assert.equal(entry.at_risk, true);
});
test("domain at sample with healthy ratio is not at risk", () => {
const result = evaluateWebhookHealth(makeRequests("shop-c.example.com", 120, 5));
const entry = result["shop-c.example.com"];
assert.equal(entry.total, 120);
assert.equal(Math.round(entry.success_ratio * 10000) / 10000, Math.round(((120 - 5) / 120) * 10000) / 10000);
assert.equal(entry.at_risk, false);
});
test("ratio exactly at threshold is not at risk", () => {
const result = evaluateWebhookHealth(makeRequests("shop-d.example.com", 100, 10));
const entry = result["shop-d.example.com"];
assert.equal(entry.success_ratio, 0.90);
assert.equal(entry.at_risk, false);
});
test("multiple domains are evaluated independently", () => {
const healthy = makeRequests("healthy.example.com", 100, 2);
const flaky = makeRequests("flaky.example.com", 150, 40);
const result = evaluateWebhookHealth([...healthy, ...flaky]);
assert.equal(result["healthy.example.com"].at_risk, false);
assert.equal(result["flaky.example.com"].at_risk, true);
});
test("empty input returns empty object", () => {
assert.deepEqual(evaluateWebhookHealth([]), {});
});Case studies
The store whose inventory hook went quiet during an orders incident
A merchant's order-processing service started timing out during a deploy, and the team assumed only the orders webhook would be affected. Instead their inventory sync, on a completely separate path but the same domain, silently stopped receiving deliveries for a few minutes at a time throughout the incident, and nobody connected the two until they compared timestamps.
Once they started correlating GET /v3/hooks against their own access logs bucketed into rolling 2-minute windows, the pattern was obvious: every time the orders path's failures pushed the domain-wide ratio under 90 percent, the inventory path went quiet too. The fix was not code, it was recognizing the domain-scoped nature of the block and fixing the actual slow endpoint instead of chasing a phantom inventory bug.
The hook that outlived its 3-minute blocks until it did not
A smaller integration had an endpoint that flaked out just often enough to trip the 3-minute domain block a few times a week, always recovering once the block expired. Nobody was watching it, so nobody noticed the underlying instability never actually got fixed, it just kept happening to clear before anyone looked.
Eventually the same flakiness crossed the separate 48-hour and 11-retry exhaustion threshold, and BigCommerce flipped is_active to false on that specific hook. A scan against updated_at and is_active caught it within the hour. After confirming the endpoint held up under a synthetic health check, the team ran the script with DRY_RUN=false to flip it back on, this time with monitoring in place.
A domain trending toward the 90 percent floor gets flagged before it ever gets blocklisted, with the exact ratio and window so whoever owns the receiving endpoint can look at the right minutes instead of guessing. If a hook does get auto-deactivated, it is caught quickly and only ever re-enabled once a live health check confirms the destination is actually fixed, never blindly. And nobody wastes time trying to call an API to lift a 3-minute block that was already expiring on its own.
FAQ
Why did BigCommerce stop delivering webhooks to a domain that was working fine an hour ago?
BigCommerce's webhook dispatcher calculates a rolling success versus failure ratio per destination domain over a sliding 2-minute window, but only once at least 100 requests have been sent to that domain in the window. If the ratio drops below 90 percent, typically because the receiving endpoint is slow, returning non-200 responses, or intermittently down, BigCommerce blocklists the entire domain for 3 minutes. Because the block is domain-scoped, every hook on that host stops receiving deliveries, not just the one that was failing.
Can one broken webhook path take down a healthy one on the same domain?
Yes. Blocking happens per destination domain, not per registered hook or path. If /webhooks/orders on a host is flaky enough to drag the domain-wide success ratio below 90 percent, deliveries to an unrelated and perfectly healthy /webhooks/inventory endpoint on that same host are starved for the 3-minute block too, since BigCommerce is not distinguishing between paths, only the domain.
Is there an API call to lift the blocklist or force BigCommerce to redeliver the missed events?
No. The 3-minute domain blocklist self-expires and BigCommerce automatically requeues the affected events once it does, so there is no safe endpoint to call to lift it early or force a redelivery. The only corrective API write available is re-enabling a hook that got separately auto-deactivated (is_active set to false after repeated failures) with PUT /v3/hooks/{hook_id} and {"is_active": true}, and only after confirming the destination is healthy again.
Related field notes
Citations
On the problem:
- BigCommerce Developer Center: webhooks overview, delivery, and retry behavior. developer.bigcommerce.com webhooks overview
- BigCommerce Support: how to avoid webhooks deactivation, it happens periodically. support.bigcommerce.com how to avoid webhooks deactivation
- Hookdeck: guide to BigCommerce webhooks, features and best practices. hookdeck.com BigCommerce webhooks guide
On the solution:
- BigCommerce API Reference: the Webhooks v3 endpoints, including GET and PUT /v3/hooks. docs.bigcommerce.com webhooks v3
- BigCommerce Docs: webhooks overview and pagination via meta.pagination. docs.bigcommerce.com webhooks overview
- BigCommerce Developer Center: manage webhooks, including re-enabling a single hook. developer.bigcommerce.com manage webhooks
Stuck on a tricky one?
If you have a problem in BigCommerce orders, payments, webhooks, inventory, or fulfillment that you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this save you from chasing a phantom bug?
If this helped you spot a domain-wide block before it hit, or explained why an unrelated hook went quiet, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi