Diagnostic Stripe
report runs past data_available_end return short data
The month-end report succeeded. The totals are lower than the Dashboard by a few thousand, and re-running the identical report the next morning produces a larger number. Nothing errored either time, which is the part that makes this take a day to find: a report that is short does not look any different from a report that is complete.
Read GET /v1/reporting/report_types and take data_available_start and data_available_end for the type you run. Then read your runs with GET /v1/reporting/report_runs and compare each one's parameters.interval_end against that data_available_end. Anything past it was answered with whatever Stripe had finalized at the time, and the run still says succeeded.
Also flag the availability window itself: a data_available_end more than about 36 hours old is Stripe's pipeline being late, not your job being wrong, and the correct response is to defer rather than to retry.
The problem in plain words
Every report type carries a window of data Stripe has finalized, exposed as data_available_start and data_available_end on the report type object. That window lags real time, because the underlying daily snapshot has to be built: for the Sigma-backed types it is typically ready in the early afternoon UTC for the previous UTC day.
A job that asks for an interval_end beyond that point does not get an error explaining the situation. It gets a successful run containing the part Stripe could answer. The file arrives, the pipeline loads it, the dashboard renders, and the only symptom is that a number is smaller than it should be — in a report where numbers are supposed to be smaller some months.
The reason it is so persistent is that it is intermittent by construction. A job that runs at 02:00 UTC asking for "up to midnight" is racing a boundary that moves. Most days it loses by a little and nobody can tell. At month-end, when someone compares against the Dashboard, it loses visibly, and the natural next step — re-run it — produces a different, larger answer that makes the whole system look untrustworthy rather than late.
Why it happens
Truncation is a success, not an error. This is the whole problem in one sentence. Every guard rail people build for the Reports API watches status, and status is succeeded. There is no field on the run that says "this is short", so the only way to know is to have checked the availability window before asking, or to check it afterwards against what you asked for.
The window moves during the job. data_available_end advances as Stripe finalizes more data, so a check written as "fetch the type, then create the run" is correct, and one written as "create the run, then explain the result" is guesswork. Reading availability after the fact, as this script does, can only prove a run was short — never that it was complete — because the window has since moved forward.
Timezones make it worse in a way that survives review. Report intervals are unix timestamps and the finalization boundary is a UTC day. A job computing "yesterday" in a local timezone asks for a window that is an hour or several past the UTC boundary, and it will do that correctly for years until the availability lag changes slightly.
Report type versions are not interchangeable. balance.summary.1 and balance.summary.2 accept different parameters and emit different columns. A job that resolves the type by prefix, or that was written against whichever version was current, can start failing or silently changing shape. Pin the version you depend on and read its availability, not the family's.
The fix, as a flow
The script compares what each run asked for against the window Stripe had finalized, because a short report and a complete one look identical from every field on the run object.
How to fix it
Read the availability window for the exact type you use
GET /v1/reporting/report_types?limit=100 lists every type with its data_available_start and data_available_end. Match on the full id including the version suffix. This is the number your job should be gating on, and almost certainly is not.
Compare every recent run's interval against it
For each run, read parameters.interval_start and parameters.interval_end and compare against the matching type. An interval_end past data_available_end means the run was answered with less than it asked for. Because availability only moves forward, anything this check flags today was definitely short when it ran.
Flag the runs sitting on the edge, not just the ones past it
A run whose interval_end lands within an hour of data_available_end is a coin flip that happened to land right. It will be flagged as truncated on some other night. Treating the edge as a warning is what turns this from a recurring mystery into a scheduling fix.
Check whether the window itself has gone stale
If data_available_end is more than about 36 hours behind now, the problem is upstream and no interval you choose will help. Defer the job and say so in the alert; retrying against an availability window that has not moved just produces more short reports.
Gate the job instead of scheduling it hopefully
The repair is a read before the write: GET /v1/reporting/report_types/{type_id}, and only POST /v1/reporting/report_runs when data_available_end >= interval_end. Otherwise wait and try later. This turns a silently short report into a late one, which is a problem finance can actually see.
How to check it worked
Re-run the script. Every run in the window should sit fully inside the type's availability window, with room to spare rather than on the boundary.
python3 stripe_report_interval.py --days 14
# clear 14 run(s) checked, all fully inside the available windowThe full code
Two GETs and no writes — a restricted key with read access to Reports is enough. The comparison between a requested interval and an availability window is a pure function, which matters more here than usual: the whole failure is an off-by-a-few-hours that nothing in the API will complain about, so the boundary rules are the check, and they are tested directly.
"""Report Stripe report runs whose interval reached past the finalized data window.
Read only. Two GETs and no writes: give this a RESTRICTED key with read access to
Reports. The repair is printed, never performed, because this script holds a
credential to a live payments account.
"""
import argparse
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("stripe_report_interval")
API = "https://api.stripe.com/v1"
# An interval_end landing this close to data_available_end is not safely covered:
# it is the same request that gets truncated on a night the pipeline runs late.
EDGE_HOURS = 1.0
# Past this, the availability window itself is the problem, not the interval.
STALE_HOURS = 36.0
def interval_state(interval_start, interval_end, available_start, available_end):
"""Compare one requested interval against a report type's availability window.
Pure, so the boundary rules can be tested without a network. All four values
are unix seconds; the availability values may be None on a type that has
never produced data. Returns (state, detail).
"""
if available_end is None or interval_end is None:
return ("unknown",
"no data_available_end or no interval_end to compare; the run "
"cannot be judged either way")
if interval_end > available_end:
short = (interval_end - available_end) / 3600.0
return ("truncated",
"interval_end is %.1f hour(s) past data_available_end; the run "
"succeeded and returned less than it asked for" % short)
if (interval_start is not None and available_start is not None
and interval_start < available_start):
early = (available_start - interval_start) / 3600.0
return ("before_window",
"interval_start is %.1f hour(s) before data_available_start; the "
"earliest part of the range does not exist" % early)
margin = (available_end - interval_end) / 3600.0
if margin < EDGE_HOURS:
return ("at_edge",
"interval_end is only %.2f hour(s) inside data_available_end; this "
"run was a coin flip and will be short on a slower night" % margin)
return ("covered",
"fully inside the available window, with %.1f hour(s) to spare" % margin)
def freshness_state(available_end_age_hours):
"""Judge the availability window itself, independently of any run. Pure."""
if available_end_age_hours is None:
return ("unknown", "the report type reports no data_available_end")
if available_end_age_hours >= STALE_HOURS:
return ("stale",
"data_available_end is %.1f hour(s) behind now; Stripe has not "
"finalized recent data, so defer rather than retry"
% available_end_age_hours)
return ("fresh",
"data_available_end is %.1f hour(s) behind now, which is normal lag"
% available_end_age_hours)
def get(session, path, params):
r = session.get(API + path, params=params, timeout=30)
if r.status_code == 401:
raise SystemExit("401 from Stripe: the key is wrong, or is for the other mode")
if r.status_code == 403:
raise SystemExit("403 from Stripe: the restricted key has no read access to "
"this resource")
r.raise_for_status()
return r.json()
def page_all(session, path, params, cap=2000):
out = []
params = dict(params)
while True:
page = get(session, path, params)
data = page.get("data", [])
out.extend(data)
if not data or not page.get("has_more") or len(out) >= cap:
return out
params["starting_after"] = data[-1]["id"]
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--days", type=int, default=14,
help="how far back to read report runs")
ap.add_argument("--report-type",
help="only check runs of this exact type id, version included")
args = ap.parse_args()
key = os.environ.get("STRIPE_API_KEY")
if not key:
log.error("set STRIPE_API_KEY (use a restricted, read-only key)")
return 2
s = requests.Session()
s.headers.update({"Authorization": "Bearer " + key})
now = time.time()
types = {t["id"]: t for t in page_all(s, "/reporting/report_types", {"limit": 100})}
runs = page_all(s, "/reporting/report_runs",
{"limit": 100, "created[gte]": int(now - args.days * 86400)})
if args.report_type:
runs = [r for r in runs if r.get("report_type") == args.report_type]
bad = 0
stale_types = set()
for r in runs:
rt = types.get(r.get("report_type"), {})
params = r.get("parameters") or {}
state, detail = interval_state(params.get("interval_start"),
params.get("interval_end"),
rt.get("data_available_start"),
rt.get("data_available_end"))
if state in ("truncated", "before_window", "at_edge"):
bad += 1
log.warning(" %-13s %s %s %s", state, r.get("id"),
r.get("report_type"), detail)
end = rt.get("data_available_end")
age = None if end is None else (now - end) / 3600.0
if freshness_state(age)[0] == "stale":
stale_types.add(r.get("report_type"))
for t in sorted(stale_types):
end = types.get(t, {}).get("data_available_end")
age = None if end is None else (now - end) / 3600.0
log.warning(" %-13s %s %s", "stale-window", t, freshness_state(age)[1])
if not bad and not stale_types:
log.info("%-11s %d run(s) checked, all fully inside the available window",
"clear", len(runs))
return 0
log.warning("%-11s %d of %d run(s) reached past what Stripe had finalized",
"short", bad, len(runs))
log.warning(" availability only moves forward, so anything flagged here was "
"definitely short when it ran")
log.warning(" gate the job on the type before creating the run:")
log.warning(" GET %s/reporting/report_types/<type_id> "
"(create only while data_available_end >= interval_end)", API)
log.warning(" and pin the version you depend on, e.g. balance.summary.1 rather "
"than whichever is current")
return 1
if __name__ == "__main__":
sys.exit(main())
/**
* Report Stripe report runs whose interval reached past the finalized data window.
*
* Read only. Two GETs and no writes: give this a RESTRICTED key with read access
* to Reports. The repair is printed, never performed.
*/
const API = 'https://api.stripe.com/v1';
// An interval_end this close to data_available_end is not safely covered.
export const EDGE_HOURS = 1.0;
// Past this, the availability window itself is the problem.
export const STALE_HOURS = 36.0;
/**
* Compare one requested interval against a report type's availability window.
* Pure, so the boundary rules can be tested without a network.
*/
export function intervalState(intervalStart, intervalEnd, availableStart, availableEnd) {
const missing = (v) => v === null || v === undefined;
if (missing(availableEnd) || missing(intervalEnd)) {
return ['unknown',
'no data_available_end or no interval_end to compare; the run cannot be ' +
'judged either way'];
}
if (intervalEnd > availableEnd) {
const short = (intervalEnd - availableEnd) / 3600;
return ['truncated',
`interval_end is ${short.toFixed(1)} hour(s) past data_available_end; the ` +
'run succeeded and returned less than it asked for'];
}
if (!missing(intervalStart) && !missing(availableStart) && intervalStart < availableStart) {
const early = (availableStart - intervalStart) / 3600;
return ['before_window',
`interval_start is ${early.toFixed(1)} hour(s) before data_available_start; ` +
'the earliest part of the range does not exist'];
}
const margin = (availableEnd - intervalEnd) / 3600;
if (margin < EDGE_HOURS) {
return ['at_edge',
`interval_end is only ${margin.toFixed(2)} hour(s) inside data_available_end; ` +
'this run was a coin flip and will be short on a slower night'];
}
return ['covered',
`fully inside the available window, with ${margin.toFixed(1)} hour(s) to spare`];
}
/** Judge the availability window itself, independently of any run. Pure. */
export function freshnessState(availableEndAgeHours) {
if (availableEndAgeHours === null || availableEndAgeHours === undefined) {
return ['unknown', 'the report type reports no data_available_end'];
}
if (availableEndAgeHours >= STALE_HOURS) {
return ['stale',
`data_available_end is ${availableEndAgeHours.toFixed(1)} hour(s) behind now; ` +
'Stripe has not finalized recent data, so defer rather than retry'];
}
return ['fresh',
`data_available_end is ${availableEndAgeHours.toFixed(1)} hour(s) behind now, ` +
'which is normal lag'];
}
async function get(key, path, params) {
const url = new URL(API + path);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
const res = await fetch(url, { headers: { Authorization: `Bearer ${key}` } });
if (res.status === 401) {
throw new Error('401 from Stripe: the key is wrong, or is for the other mode');
}
if (res.status === 403) {
throw new Error('403 from Stripe: the restricted key has no read access here');
}
if (!res.ok) throw new Error(`${res.status} from ${url.pathname}`);
return res.json();
}
async function pageAll(key, path, params, cap = 2000) {
const out = [];
const p = { ...params };
for (;;) {
const page = await get(key, path, p);
const data = page.data ?? [];
out.push(...data);
if (data.length === 0 || !page.has_more || out.length >= cap) return out;
p.starting_after = data[data.length - 1].id;
}
}
async function main() {
const key = process.env.STRIPE_API_KEY;
if (!key) {
console.error('set STRIPE_API_KEY (use a restricted, read-only key)');
process.exitCode = 2;
return;
}
const days = Number(process.env.DAYS ?? 14);
const now = Date.now() / 1000;
const types = new Map();
for (const t of await pageAll(key, '/reporting/report_types', { limit: 100 })) {
types.set(t.id, t);
}
const runs = await pageAll(key, '/reporting/report_runs',
{ limit: 100, 'created[gte]': Math.floor(now - days * 86400) });
let bad = 0;
const staleTypes = new Set();
for (const r of runs) {
const rt = types.get(r.report_type) ?? {};
const params = r.parameters ?? {};
const [state, detail] = intervalState(params.interval_start, params.interval_end,
rt.data_available_start, rt.data_available_end);
if (['truncated', 'before_window', 'at_edge'].includes(state)) {
bad += 1;
console.warn(` ${state.padEnd(13)} ${r.id} ${r.report_type} ${detail}`);
}
const end = rt.data_available_end;
const age = end === undefined || end === null ? null : (now - end) / 3600;
if (freshnessState(age)[0] === 'stale') staleTypes.add(r.report_type);
}
for (const t of [...staleTypes].sort()) {
const end = (types.get(t) ?? {}).data_available_end;
const age = end === undefined || end === null ? null : (now - end) / 3600;
console.warn(` stale-window ${t} ${freshnessState(age)[1]}`);
}
if (!bad && staleTypes.size === 0) {
console.log(`clear ${runs.length} run(s) checked, all fully inside the ` +
'available window');
return;
}
console.warn(`short ${bad} of ${runs.length} run(s) reached past what Stripe ` +
'had finalized');
console.warn(' availability only moves forward, so anything flagged here was ' +
'definitely short when it ran');
console.warn(' gate the job on the type before creating the run:');
console.warn(` GET ${API}/reporting/report_types/<type_id> ` +
'(create only while data_available_end >= interval_end)');
console.warn(' and pin the version you depend on, e.g. balance.summary.1 rather ' +
'than whichever is current');
process.exitCode = 1;
}
// Only run when invoked directly, so importing this module from the test file
// does not fire main() and fail the suite on the missing key.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
Everything here is a boundary, because the failure is a boundary: one hour on the wrong side of data_available_end and a successful report is short. The tests pin the exact comparison, including the case that gets written wrong most often — an interval landing exactly on the edge, which is covered but is not safe.
from stripe_report_interval import freshness_state, interval_state
DAY = 86400
# A fixed availability window: data finalized up to this timestamp.
AVAIL_END = 1_756_000_000
AVAIL_START = AVAIL_END - 90 * DAY
def test_interval_inside_the_window_is_covered():
state, detail = interval_state(AVAIL_END - 2 * DAY, AVAIL_END - DAY,
AVAIL_START, AVAIL_END)
assert state == "covered"
assert "24.0 hour" in detail
def test_one_hour_past_availability_is_truncated_not_an_error():
state, detail = interval_state(AVAIL_END - DAY, AVAIL_END + 3600,
AVAIL_START, AVAIL_END)
assert state == "truncated"
assert "1.0 hour(s) past" in detail
assert "succeeded" in detail
def test_landing_exactly_on_the_edge_is_a_warning_not_a_pass():
# interval_end == data_available_end is not truncated, but it is the request
# that gets truncated the night Stripe finalizes an hour later than usual.
assert interval_state(AVAIL_START, AVAIL_END, AVAIL_START, AVAIL_END)[0] == "at_edge"
assert interval_state(AVAIL_START, AVAIL_END - 1800,
AVAIL_START, AVAIL_END)[0] == "at_edge"
# A full hour of margin is the boundary, and the boundary counts as covered.
assert interval_state(AVAIL_START, AVAIL_END - 3600,
AVAIL_START, AVAIL_END)[0] == "covered"
def test_start_before_the_window_is_reported_separately():
state, _ = interval_state(AVAIL_START - DAY, AVAIL_END - DAY,
AVAIL_START, AVAIL_END)
assert state == "before_window"
def test_a_stale_window_is_stripes_problem_not_the_intervals():
assert freshness_state(12.0)[0] == "fresh"
assert freshness_state(35.9)[0] == "fresh"
state, detail = freshness_state(36.0)
assert state == "stale"
assert "defer rather than retry" in detail
assert freshness_state(None)[0] == "unknown"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { freshnessState, intervalState } from './stripe-report-interval.mjs';
const DAY = 86400;
const AVAIL_END = 1756000000;
const AVAIL_START = AVAIL_END - 90 * DAY;
test('interval inside the window is covered', () => {
const [state, detail] = intervalState(AVAIL_END - 2 * DAY, AVAIL_END - DAY,
AVAIL_START, AVAIL_END);
assert.equal(state, 'covered');
assert.match(detail, /24\.0 hour/);
});
test('one hour past availability is truncated, not an error', () => {
const [state, detail] = intervalState(AVAIL_END - DAY, AVAIL_END + 3600,
AVAIL_START, AVAIL_END);
assert.equal(state, 'truncated');
assert.match(detail, /1\.0 hour\(s\) past/);
assert.match(detail, /succeeded/);
});
test('landing exactly on the edge is a warning, not a pass', () => {
assert.equal(intervalState(AVAIL_START, AVAIL_END, AVAIL_START, AVAIL_END)[0],
'at_edge');
assert.equal(intervalState(AVAIL_START, AVAIL_END - 1800, AVAIL_START, AVAIL_END)[0],
'at_edge');
// A full hour of margin is the boundary, and the boundary counts as covered.
assert.equal(intervalState(AVAIL_START, AVAIL_END - 3600, AVAIL_START, AVAIL_END)[0],
'covered');
});
test('start before the window is reported separately', () => {
assert.equal(intervalState(AVAIL_START - DAY, AVAIL_END - DAY,
AVAIL_START, AVAIL_END)[0], 'before_window');
});
test('a stale window is Stripe problem, not the interval', () => {
assert.equal(freshnessState(12)[0], 'fresh');
assert.equal(freshnessState(35.9)[0], 'fresh');
const [state, detail] = freshnessState(36);
assert.equal(state, 'stale');
assert.match(detail, /defer rather than retry/);
assert.equal(freshnessState(null)[0], 'unknown');
});
FAQ
Why does Stripe not error when I ask for data it does not have?
Because the request is legitimate: the report type has a window of finalized data and Stripe answers with the part of your interval that falls inside it. The run reaches status succeeded with a real file attached. There is no field on the run marking it as short, which is why this has to be checked against data_available_end rather than against the run.
How far behind real time is data_available_end?
It lags, and the size of the lag depends on the report type and on the day. The daily snapshot behind the Sigma-backed types is typically ready in the early afternoon UTC for the previous UTC day. Do not hard-code a number: read data_available_end before creating the run and gate on it.
Can this script prove a report was complete?
No, and it does not claim to. Availability only moves forward, so comparing a past run against today's window can only prove a run was short. Proving completeness requires reading data_available_end before the run, which is exactly what the repair asks you to start doing.
Why flag runs that land exactly on the boundary?
Because they are the same job that gets truncated on a slower night. An interval_end within an hour of data_available_end succeeded by luck. Flagging the edge turns a report that is intermittently short into a scheduling change you make once.
Does the report type version matter?
Yes. Versions such as balance.summary.1 and balance.summary.2 accept different parameters and emit different schemas, and each carries its own availability window. Pin the exact type id you depend on, and read the availability for that id rather than for the family.
Related field notes
- A report run fails after the 200 and no CSV lands
- Sigma scheduled query runs time out and email nothing
- Payout reconciliation reports are unavailable
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- The report type object — Stripe API reference
- The report run object — Stripe API reference
- Reports — Stripe Docs
- Schedule queries — Stripe Docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.