Diagnostic Stripe
reused idempotency keys hit 409 idempotency_key_in_use
The keys are being sent. That is what makes this different from payment requests with no key at all, where the header is simply absent. Here every request carries one, and the same one keeps turning up on requests that are not the same operation — so under load a slice of checkouts fails with 409 idempotency_key_in_use, and a day later the protection stops working entirely and the duplicates come back.
Page GET /v1/events across the retained window and group the events by request.idempotency_key, ignoring nulls. A key that appears against more than one distinct request.id was not deduplicated: both requests really executed. Inside 24 hours that is two concurrent operations sharing a key, which is the 409. Beyond 24 hours it is a key that was pruned and then reused, which is a duplicate rather than a replay.
Then look at the shape of the keys themselves. Anything that reads as a customer ID, a PaymentIntent ID, a user ID, a bare integer, a date or an email address is derived from something that repeats, and it will collide as soon as that thing comes round again.
The problem in plain words
Two different failures come out of the same mistake, and they arrive weeks apart. The first is a 409 Conflict reading There is currently another in-progress request using this Idempotent Key. It only happens under concurrency, so it is rare in staging and clustered at your busiest hour in production, which makes it look like a capacity problem. The second is a 400 idempotency_error: Keys for idempotent requests can only be used with the same parameters they were first used with. That one names the cause precisely and is still usually read as a Stripe quirk.
The third outcome is the one nobody sees. Keys are pruned after roughly 24 hours. A key derived from a customer ID protects a retry at 10:00 and does nothing at all for the same customer the next morning, because Stripe has forgotten it and starts a genuinely new operation. The protection appears to work for as long as anybody tests it, and fails on exactly the timescale nobody tests.
Why it happens
Stripe saves the result after execution begins, not before. Two requests carrying one key that arrive close enough together both get past that point, so neither can replay the other. One of them gets the 409. It is a retryable error, and the correct retry uses the same key — but the reason it happened is that the key is shared between operations that should have had their own.
Keys live for about 24 hours. After that they are pruned, and a request carrying a pruned key is brand new. Any key derived from something long-lived — a customer, an order, a cart — is fine for hours and useless for days, which is why the duplicates look uncorrelated with the retries.
Same key plus different parameters is an error, not a replay. Reusing a key across two operations that differ at all returns idempotency_error. So a key derived from an order ID breaks the moment one order creates both a customer and a PaymentIntent, or the moment a failed payment is retried with a different amount.
The evidence is in the events, and it needs two conditions. Every event carries the request that caused it, including its idempotency_key. One key on one request ID is normal. One key on several request IDs means the deduplication did not happen, and the spread between their timestamps says which of the two failures you have.
The fix, as a flow
The script groups events by idempotency key and measures the spread between the first and the last, because the same reused key is a 409 inside 24 hours and a silent duplicate outside it.
How to fix it
Page the window and group by key
GET /v1/events?limit=100 with starting_after. Skip events where request.idempotency_key is null: Stripe-initiated events never had a key, and unkeyed API requests are a different problem with a different fix.
Count distinct request ids per key
One key on one request.id is a key doing its job. Two or more means both requests executed, so nothing was deduplicated. That is the finding, and it is a fact rather than an inference.
Measure the spread between the timestamps
More than 86400 seconds between the first and last event for a key puts the reuse either side of the pruning window. That is a duplicate created rather than a conflict returned, and it is the more expensive of the two.
Read the shape of the key
A key that starts cus_, pi_ or user_, or that parses as a bare integer or a date, or that contains an @, is derived from something that repeats. It has not necessarily collided yet; it will. Personal identifiers are also simply not acceptable as keys.
Generate one key per logical operation and persist it
A v4 UUID, made once when the operation is first attempted, written next to the operation record, and reused byte for byte on every retry of that exact request. Keys cap at 255 characters. On a 409, back off and retry with the same key rather than minting a new one.
How to check it worked
Re-run the script after the deploy. Because the window is rolling, the old keys stay in the sample for 30 days; the number to watch is that no key created since the deploy appears twice.
python3 stripe_idempotency_key_reuse.py --days 1
# 412 key(s) sampled, 0 reused, 0 derived from something that repeatsThe full code
One paginated GET against /v1/events — a restricted key with read access to Events is enough. Two pure functions: one reads the shape of a key string, the other turns a key's tally into a verdict. Keeping them apart matters because a key can be perfectly unique so far and still be built out of something that will repeat next Tuesday, and that is a finding the counts alone cannot produce.
"""Report Stripe idempotency keys reused across more than one request.
Read only. One paginated GET and no writes: give this a RESTRICTED key with read
access to Events. The repair is printed, never performed, because this script
holds a credential to a live payments account.
"""
import argparse
import logging
import os
import re
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("stripe_idempotency_key_reuse")
API = "https://api.stripe.com/v1"
# Stripe prunes saved idempotency results after roughly 24 hours. A key seen
# either side of that gap was not replayed; it started a new operation.
PRUNE_WINDOW = 86400
MAX_KEY_LEN = 255
OBJECT_ID = re.compile(r"^(cus_|pi_|ch_|sub_|in_|seti_|user[-_])", re.I)
UUID4 = re.compile(
r"^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$", re.I)
ISO_DATE = re.compile(r"^\d{4}-\d{2}-\d{2}")
def key_shape(key):
"""What a key string is built out of. Pure.
Returns (shape, description). Anything but "uuid" and "opaque" is derived
from something that comes round again, whether or not it has collided yet.
"""
if key is None or key == "":
return ("missing", "no key at all")
k = str(key)
if len(k) > MAX_KEY_LEN:
return ("over-long",
"%d characters, over the %d limit" % (len(k), MAX_KEY_LEN))
if "@" in k:
return ("personal",
"an email address, which repeats and should not be sent as a key")
if UUID4.match(k):
return ("uuid", "a v4 uuid")
if OBJECT_ID.match(k):
return ("object-id", "an object id, which repeats every time that object "
"is used again")
if k.isdigit():
return ("integer", "a bare integer, which repeats")
if ISO_DATE.match(k):
return ("date", "a date, which repeats for every operation that day")
return ("opaque", "not obviously derived from anything that repeats")
def verdict(key, request_ids, spread_seconds):
"""Classify one key's tally. Pure, so the thresholds can be tested.
`request_ids` is the number of distinct request ids carrying this key, and
`spread_seconds` the gap between its first and last event.
"""
shape, described = key_shape(key)
if request_ids > 1 and spread_seconds > PRUNE_WINDOW:
return ("pruned",
"%d distinct requests, %d seconds apart. Stripe forgets a key "
"after about %d seconds, so the later one started a fresh "
"operation and created a duplicate rather than replaying."
% (request_ids, spread_seconds, PRUNE_WINDOW))
if request_ids > 1:
return ("concurrent",
"%d distinct requests inside the window Stripe remembers the "
"key. Both executed, so the key is shared between operations "
"rather than unique to one. Under load this returns 409 "
"idempotency_key_in_use." % request_ids)
if shape not in ("uuid", "opaque"):
return ("derived",
"one request so far, but the key is %s" % described)
return ("unique", "one request, and the key is %s" % described)
def get(session, path, **params):
r = session.get(API + path, params=params, timeout=30)
if r.status_code == 401:
raise SystemExit("401 from Stripe: the key is wrong, or is for the other mode")
r.raise_for_status()
return r.json()
def keys_seen(session, since, limit):
"""Per-key distinct request ids and the timestamp spread. Keyed requests only."""
seen = {}
total = 0
params = {"limit": 100, "created[gte]": int(since)}
while True:
page = get(session, "/events", **params)
data = page.get("data", [])
for ev in data:
total += 1
req = ev.get("request")
if not isinstance(req, dict):
continue # Stripe-initiated, or an old bare-string request field
key = req.get("idempotency_key")
if not key:
continue # unkeyed requests are a different problem
row = seen.setdefault(key, {"ids": set(), "first": None, "last": None})
if req.get("id"):
row["ids"].add(req["id"])
created = ev.get("created")
if created is not None:
row["first"] = created if row["first"] is None else min(row["first"], created)
row["last"] = created if row["last"] is None else max(row["last"], created)
if not data or not page.get("has_more") or total >= limit:
break
params["starting_after"] = data[-1]["id"]
return seen, total
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--days", type=int, default=30,
help="how far back to sample, up to the 30-day retention limit")
ap.add_argument("--max-events", type=int, default=5000,
help="stop paginating after this many events")
args = ap.parse_args()
key = os.environ.get("STRIPE_API_KEY")
if not key:
log.error("set STRIPE_API_KEY (use a restricted, read-only key)")
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + key})
since = time.time() - args.days * 86400
seen, total = keys_seen(s, since, args.max_events)
log.info("sampled %d event(s) over %d day(s)", total, args.days)
reused = derived = 0
for k in sorted(seen):
row = seen[k]
spread = (row["last"] or 0) - (row["first"] or 0)
state, detail = verdict(k, len(row["ids"]) or 1, spread)
if state == "unique":
continue
line = "%-11s %-40s %s" % (state, k[:40], detail)
if state == "derived":
derived += 1
log.info(line)
else:
reused += 1
log.warning(line)
if reused or derived:
log.warning(" repair: one fresh v4 uuid per logical operation, made when "
"the operation is first attempted")
log.warning(" persist it next to the operation record and resend it "
"unchanged for every retry of that exact request")
log.warning(" on 409 idempotency_key_in_use, back off and retry with the "
"same key rather than minting a new one")
log.warning(" never derive a key from a customer id, an order id, a date "
"or an email address; keys cap at %d characters", MAX_KEY_LEN)
log.info("%d key(s) sampled, %d reused, %d derived from something that repeats",
len(seen), reused, derived)
return 1 if reused else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Report Stripe idempotency keys reused across more than one request.
*
* Read only. One paginated GET and no writes: give this a RESTRICTED key with
* read access to Events. The repair is printed, never performed.
*/
const API = 'https://api.stripe.com/v1';
// Stripe prunes saved idempotency results after roughly 24 hours. A key seen
// either side of that gap was not replayed; it started a new operation.
export const PRUNE_WINDOW = 86400;
const MAX_KEY_LEN = 255;
const OBJECT_ID = /^(cus_|pi_|ch_|sub_|in_|seti_|user[-_])/i;
const UUID4 =
/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i;
const ISO_DATE = /^\d{4}-\d{2}-\d{2}/;
/**
* What a key string is built out of. Pure. Returns [shape, description].
* Anything but 'uuid' and 'opaque' is derived from something that comes round
* again, whether or not it has collided yet.
*/
export function keyShape(key) {
if (key === null || key === undefined || key === '') {
return ['missing', 'no key at all'];
}
const k = String(key);
if (k.length > MAX_KEY_LEN) {
return ['over-long', `${k.length} characters, over the ${MAX_KEY_LEN} limit`];
}
if (k.includes('@')) {
return ['personal',
'an email address, which repeats and should not be sent as a key'];
}
if (UUID4.test(k)) return ['uuid', 'a v4 uuid'];
if (OBJECT_ID.test(k)) {
return ['object-id',
'an object id, which repeats every time that object is used again'];
}
if (/^\d+$/.test(k)) return ['integer', 'a bare integer, which repeats'];
if (ISO_DATE.test(k)) {
return ['date', 'a date, which repeats for every operation that day'];
}
return ['opaque', 'not obviously derived from anything that repeats'];
}
/**
* Classify one key's tally. Pure, so the thresholds can be tested.
* `requestIds` is the number of distinct request ids carrying this key.
*/
export function verdict(key, requestIds, spreadSeconds) {
const [shape, described] = keyShape(key);
if (requestIds > 1 && spreadSeconds > PRUNE_WINDOW) {
return ['pruned',
`${requestIds} distinct requests, ${spreadSeconds} seconds apart. Stripe ` +
`forgets a key after about ${PRUNE_WINDOW} seconds, so the later one ` +
'started a fresh operation and created a duplicate rather than replaying.'];
}
if (requestIds > 1) {
return ['concurrent',
`${requestIds} distinct requests inside the window Stripe remembers the ` +
'key. Both executed, so the key is shared between operations rather than ' +
'unique to one. Under load this returns 409 idempotency_key_in_use.'];
}
if (shape !== 'uuid' && shape !== 'opaque') {
return ['derived', `one request so far, but the key is ${described}`];
}
return ['unique', `one request, and the key is ${described}`];
}
async function get(key, path, params = {}) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
const res = await fetch(url, { headers: { Authorization: `Bearer ${key}` } });
if (res.status === 401) {
throw new Error('401 from Stripe: the key is wrong, or is for the other mode');
}
if (!res.ok) throw new Error(`${res.status} from ${url.pathname}`);
return res.json();
}
export async function keysSeen(key, since, limit = 5000) {
const seen = new Map();
let total = 0;
const params = { limit: 100, 'created[gte]': Math.floor(since) };
for (;;) {
const page = await get(key, '/events', params);
const data = page.data ?? [];
for (const ev of data) {
total += 1;
const req = ev.request;
if (!req || typeof req !== 'object') continue;
const idem = req.idempotency_key;
if (!idem) continue; // unkeyed requests are a different problem
if (!seen.has(idem)) seen.set(idem, { ids: new Set(), first: null, last: null });
const row = seen.get(idem);
if (req.id) row.ids.add(req.id);
const created = ev.created;
if (created !== null && created !== undefined) {
row.first = row.first === null ? created : Math.min(row.first, created);
row.last = row.last === null ? created : Math.max(row.last, created);
}
}
if (data.length === 0 || !page.has_more || total >= limit) break;
params.starting_after = data[data.length - 1].id;
}
return { seen, total };
}
async function main() {
const apiKey = process.env.STRIPE_API_KEY;
if (!apiKey) {
console.error('set STRIPE_API_KEY (use a restricted, read-only key)');
process.exitCode = 2;
return;
}
const days = Number(process.env.DAYS ?? 30);
const since = Date.now() / 1000 - days * 86400;
const { seen, total } = await keysSeen(apiKey, since);
console.log(`sampled ${total} event(s) over ${days} day(s)`);
let reused = 0;
let derived = 0;
for (const k of [...seen.keys()].sort()) {
const row = seen.get(k);
const spread = (row.last ?? 0) - (row.first ?? 0);
const [state, detail] = verdict(k, row.ids.size || 1, spread);
if (state === 'unique') continue;
const line = `${state.padEnd(11)} ${k.slice(0, 40).padEnd(40)} ${detail}`;
if (state === 'derived') { derived += 1; console.log(line); }
else { reused += 1; console.warn(line); }
}
if (reused || derived) {
console.warn(' repair: one fresh v4 uuid per logical operation, made when ' +
'the operation is first attempted');
console.warn(' persist it next to the operation record and resend it ' +
'unchanged for every retry of that exact request');
console.warn(' on 409 idempotency_key_in_use, back off and retry with the ' +
'same key rather than minting a new one');
console.warn(' never derive a key from a customer id, an order id, a date ' +
`or an email address; keys cap at ${MAX_KEY_LEN} characters`);
}
console.log(`${seen.size} key(s) sampled, ${reused} reused, ${derived} derived ` +
'from something that repeats');
process.exitCode = reused ? 1 : 0;
}
// Only run when invoked directly. The test file imports this module, and without
// the guard main() would run there too, fail on the missing key, and set a
// non-zero exit code that fails the whole test file even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The first two tests separate the two failures, which matter differently. Inside the pruning window a shared key produces a 409 that a caller can retry: noisy, survivable. Outside it, the same shared key produces a second real charge and no error at all. The gap between the timestamps is the only thing in the data that tells them apart, so it is worth pinning that the threshold is read the right way round.
from stripe_idempotency_key_reuse import key_shape, verdict
UUID = "6f9619ff-8b86-4d01-b42d-00cf4fc964ff"
def test_two_requests_inside_the_window_is_the_409():
state, detail = verdict(UUID, 2, 30)
assert state == "concurrent"
assert "409" in detail
def test_two_requests_a_day_apart_is_a_duplicate_not_a_conflict():
# Past the pruning window Stripe has forgotten the key, so the second
# request is genuinely new and no error is returned at all.
state, detail = verdict(UUID, 2, 108000)
assert state == "pruned"
assert "86400" in detail
def test_a_key_built_from_a_customer_id_is_flagged_before_it_collides():
assert key_shape("cus_Nc1mzuAyRlKmGT")[0] == "object-id"
state, _ = verdict("cus_Nc1mzuAyRlKmGT", 1, 0)
assert state == "derived"
def test_an_email_address_is_never_an_acceptable_key():
shape, described = key_shape("ada@example.com")
assert shape == "personal"
assert "email" in described
assert verdict("ada@example.com", 1, 0)[0] == "derived"
def test_a_uuid_on_one_request_is_clean():
assert key_shape(UUID)[0] == "uuid"
assert verdict(UUID, 1, 0)[0] == "unique"
assert key_shape("2026-08-30")[0] == "date"
assert key_shape("41231")[0] == "integer"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { keyShape, verdict } from './stripe-idempotency-key-reuse.mjs';
const UUID = '6f9619ff-8b86-4d01-b42d-00cf4fc964ff';
test('two requests inside the window is the 409', () => {
const [state, detail] = verdict(UUID, 2, 30);
assert.equal(state, 'concurrent');
assert.match(detail, /409/);
});
test('two requests a day apart is a duplicate not a conflict', () => {
const [state, detail] = verdict(UUID, 2, 108000);
assert.equal(state, 'pruned');
assert.match(detail, /86400/);
});
test('a key built from a customer id is flagged before it collides', () => {
assert.equal(keyShape('cus_Nc1mzuAyRlKmGT')[0], 'object-id');
assert.equal(verdict('cus_Nc1mzuAyRlKmGT', 1, 0)[0], 'derived');
});
test('an email address is never an acceptable key', () => {
const [shape, described] = keyShape('ada@example.com');
assert.equal(shape, 'personal');
assert.match(described, /email/);
assert.equal(verdict('ada@example.com', 1, 0)[0], 'derived');
});
test('a uuid on one request is clean', () => {
assert.equal(keyShape(UUID)[0], 'uuid');
assert.equal(verdict(UUID, 1, 0)[0], 'unique');
assert.equal(keyShape('2026-08-30')[0], 'date');
assert.equal(keyShape('41231')[0], 'integer');
});
FAQ
Is reusing an idempotency key ever correct?
Yes, for a retry of the exact same request with the exact same parameters. That is the entire point of the key. What this check finds is a key reused for a different operation, or reused so long after the first that Stripe has already forgotten it.
What does 409 idempotency_key_in_use actually mean?
Two requests carrying the same key arrived close enough together that neither result had been saved yet. Stripe saves the response only after execution begins, so there was nothing to replay. It is retryable, and the retry should carry the same key.
Why do duplicates come back after a day?
Because the key was pruned. Saved idempotency results live for roughly 24 hours. A key derived from something long-lived, like a customer or an order, protects retries within that day and does nothing at all the next morning, when Stripe treats it as a key it has never seen.
What is the difference between the 409 and idempotency_error?
The 409 is a timing collision: same key, two in-flight requests. The idempotency_error is a semantic one: same key, different parameters. The first says your key is shared across concurrent operations; the second says it is shared across different ones.
How is this different from sending no key at all?
A missing key means every retry executes, always. A reused key means deduplication works right up until the moment two requests collide or the key is pruned, so it fails intermittently and looks like load rather than like a bug. The detection is different too: this one groups by the key, the other one looks for its absence.
Related field notes
- Payment-creating requests carry no idempotency key
- One customer has several Customer records
- Two endpoints share one URL, so every event is handled twice
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Idempotent requests — Stripe API reference
- Errors — Stripe API reference
- The event object — Stripe API reference
- List all events — Stripe API reference
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.