Diagnostic LLM APIs
Every response is stored and no endpoint will list them
The question arrives from legal, not from engineering, and it is one sentence long: what customer data are we currently holding on the model provider's side. Everybody assumes somebody knows. Nobody does. The application logs what it sent and what came back, the provider stores the same thing independently, and when you go looking for the endpoint that lists what is stored there so you can answer the sentence, there is not one. Not for responses, not for conversations. There is no query. There is only whatever ids you happened to write down.
Start by accepting the limitation, because it shapes everything else. Neither /v1/responses nor /v1/conversations has a list endpoint. Both resources are reachable only by an id you already hold. So this is a probe over your own records rather than an audit of your account, and the script says so on every run instead of implying coverage it does not have.
Then read the retention wording carefully, because it runs the opposite way to how it is usually quoted. The conversation-state guide says response objects are saved for 30 days by default; the data-retention page says stored response data will be kept for at least 30 days. For a thread that has to keep resolving, thirty days is a deadline. For a question about what you are holding, it is a floor. A response you have not deleted is a response you are still holding, and the API will not tell you for how much longer.
With a project read key, call GET /v1/responses/{response_id} for each recorded id. A 200 is the only evidence a response was stored, because the Response object does not echo store back — that parameter exists on the create request and nowhere in the retrieved object. Read created_at, status and conversation, and grade the age against your own retention rule rather than against the platform's.
Conversations are the other half and they are worse. Their documented retention is until deleted, they are excluded from the zero-data-retention column of the same table, and every turn appends items indefinitely, so a long thread is both a data-holding problem and a rising input bill on every subsequent turn. GET /v1/conversations/{conversation_id}/items counts them, 100 per page.
Two repairs, and the second one is a trap. DELETE /v1/responses/{response_id} removes a response. DELETE /v1/conversations/{conversation_id} removes a conversation and its items are not deleted with it, so a sweep that only deletes conversations leaves the transcripts behind. The script prints both, in that order, and executes neither.
The problem in plain words
Nobody chose this. store defaults to true, because that is what makes previous_response_id threading and background mode work, and defaults do not appear in code review. So every prompt and every completion your application has ever produced is persisted on the provider's side, including the ones that carried a support ticket, a pasted credential, a medical detail or a retrieved internal document, and there was never a moment where somebody decided that.
The absence of a list endpoint is what turns a policy question into an impossible one. You cannot ask what is stored. You can only ask about an id, and you can only ask about the ids you recorded, and the ids you recorded are a subset of the ids you created because logging is best-effort and log retention is shorter than data retention. The gap between those two sets is invisible by construction: the objects in it are still there, still readable by anyone with the key, and permanently unenumerable.
Conversations make the same problem worse along a second axis. They have no expiry at all — the retention table says until deleted — and they grow monotonically, one item per turn, forever. A thread that has been going for six months is a transcript nobody has read, held indefinitely, and it is also the reason that thread's input tokens have been climbing every week. The data-holding problem and the cost problem are the same object.
And there is a sting in the deletion path that catches the obvious cleanup script. Deleting a conversation does not delete the items inside it. A sweep written by somebody reasonable, that walks a list of conversation ids and deletes each one, leaves every message body in place and produces a report saying the cleanup succeeded.
Why it happens
The limitation is the note, so the script is built around it rather than apologising for it. There is no list endpoint for either resource. That is not an inconvenience to route around, it is the fact that determines what an honest answer looks like: a coverage statement printed on every run, ids counted next to findings, and a first repair that is not a deletion at all but a column. If you are not recording response and conversation ids with a created-at, no script can ever answer the legal question, and that is the finding.
"At least 30 days" is a floor, and reading it as a deadline inverts the conclusion. The two OpenAI pages phrase it differently and the weaker one is the one a retention question needs. A thread note treats thirty days as the moment something disappears and computes runway from it. This note treats it as the minimum time something persists, so an object older than your own policy is a finding regardless of what the platform intends to do next, and the output says which of the two readings it is using.
A 200 is the only evidence of storage, because store is write-only. The parameter appears in the create request and is absent from the retrieved Response object; there is no flag to read back. So the script never reports a store value, and a 404 gets a sentence naming both of its causes — created with store: false, or already aged out — rather than being resolved into whichever one is more convenient.
Deleting a conversation does not delete its items, and that is documented. This is the single most expensive detail in the note, because the natural cleanup is exactly wrong: a script that iterates conversation ids and deletes them reports success and removes nothing of substance. The repair is printed in the order it has to be done — items first, conversation second — and there is a test that fails if the wording ever stops saying so.
A long thread is two findings that happen to share an object. Item count past a cap is a data-holding problem and a per-turn cost problem at once: every future turn on that conversation carries the accumulated items as input. The script grades the count separately from the idleness, because a busy thread with four thousand items and a dead thread with forty are fixed by different things — a summarise-and-restart in the first case, a sweep in the second.
This is not the chain note and the code proves it. The published previous-response-id-chain-broken walks parents upward and asks whether the next turn will resolve. This one never follows a parent: the row it builds from a retrieved response deliberately has no previous_response_id key in it, and a test asserts that. Same GET, opposite question, and the retention number even means the opposite thing in each.
The fix, as a flow
Retention here was never chosen. Storing a response is the default, which means every prompt and every completion is persisted server side unless something explicitly opted out, and the objects that thread them together have no expiry at all. The hard part is not the policy, it is that neither collection can be listed. There is no query that answers what am I holding, so the script probes the ids you recorded and says plainly that its coverage is your logs and not your account.
How to fix it
Export the ids you have, and notice which ones you do not
One id per line, mixed. Anything beginning resp_ is probed as a response, anything beginning conv_ as a conversation, and anything else is reported as unroutable rather than guessed at. If this file is short because your logs roll over faster than the provider's retention, that gap is the first thing the run tells you.
Use a project read key
Stored responses and conversations are project data, so an admin key is the wrong credential. Every call is a GET: the response, the conversation, and the conversation's items. Nothing is created, nothing is deleted, and no completion is generated.
Grade responses against your policy, not the platform's
--policy-days is your retention rule, defaulting to 30. A response still readable past it is a finding whatever the platform intends, because the documented guarantee is at least 30 days and therefore says nothing about when the object goes away.
Count conversation items, 100 at a time
GET /v1/conversations/{conversation_id}/items?limit=100, paged on after. The count answers the retention question and the cost question at once, and --max-items sets where a thread stops being long and starts being a problem.
Take the repair in the order it is printed
Responses: DELETE /v1/responses/{response_id}, and store: false on calls carrying regulated data. Conversations: delete the items first, because deleting the conversation does not delete them, then the conversation. None of it is executed here.
How to check it worked
Delete one conversation properly — items first, then the conversation — and re-run. That id should move to not-retained, and the total item count in the run should fall by the number of items you removed rather than by zero, which is the check that tells you the sweep did what you think. The number to keep an eye on across runs is not the findings count but the coverage line: as your own id ledger improves, the number of ids probed should rise, and a run that probes more and finds proportionally fewer problems is the only shape that means progress.
python3 openai_stored_state_probe.py --records stored-ids.txt --policy-days 30
# 412 id(s) supplied: 388 response(s), 22 conversation(s), 2 unroutable
# coverage: neither /v1/responses nor /v1/conversations has a list endpoint,
# so this is your records and not your account
# retained-past-policy resp_a19: still readable 94.2 day(s) after creation,
# past your 30 day policy. Retention is documented as at
# least 30 days, so that is a floor and not a deadline
# repair: DELETE /v1/responses/{response_id} for what you no longer need,
# and pass store false on calls carrying regulated data.
# items-outlive-response resp_b40: 4.1 day(s) old and inside your policy, but
# its items were added to conversation conv_x1, which is
# retained until deleted
# not-retained resp_c02: nothing is stored under this id. It was
# created with store false, or it has already aged out
# thread-unbounded conv_x1: 4,182 item(s) and no TTL, so every turn on
# this thread carries them as input
# repair: start a fresh conversation seeded with a summary once a thread
# gets long, so input tokens stop compounding.
# repair: delete the items first with DELETE /v1/conversations/
# {conversation_id}/items/{item_id}. Deleting the conversation does
# not delete its items.
# thread-idle conv_y7: last item 211.4 day(s) ago, past your 30 day
# policy, and conversations are retained until deleted
# unrecognised-id legacy-7742: neither a resp_ nor a conv_ id, so it was
# not probed
# 412 supplied, 410 probed, 3 finding(s)The full code
Three kinds of GET and seven pure functions. parse_records, which routes ids by prefix and keeps what it could not route rather than dropping it; response_row, which reduces a retrieved response to five fields and deliberately has no previous_response_id in it, because following a parent is the other note's job; item_totals, which folds a conversation's items into a count and two timestamps; age_days, taking the clock as an argument so retention is testable without one; grade_response and grade_conversation, which grade against your policy rather than the platform's and treat a 404 as one fact with two named causes; and repair_lines, which prints the conversation cleanup in the only order that works.
"""Probe recorded response and conversation ids for retention and volume.
Read only. GET /v1/responses/{id}, GET /v1/conversations/{id} and
GET /v1/conversations/{id}/items. Nothing is created and nothing is deleted.
Neither /v1/responses nor /v1/conversations has a list endpoint, so there is no
way to enumerate what is stored. This probes the ids you recorded and prints a
coverage statement every run rather than implying it audited an account.
Two retention facts, both documented, both easy to quote the wrong way round.
Stored response data is kept for AT LEAST 30 days, which is a floor rather than
a deadline: an object you have not deleted is one you are still holding.
Conversations are retained UNTIL DELETED and their items are not deleted when
the conversation is.
The Response object does not echo `store` back, so a 200 is the only evidence
that a response was stored and a 404 has two causes, both of which are named.
This never follows previous_response_id. Walking a chain to see whether the
next turn resolves is a different question and a different script.
"""
import argparse
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_stored_state_probe")
RESPONSES_URL = "https://api.openai.com/v1/responses"
CONVERSATIONS_URL = "https://api.openai.com/v1/conversations"
# Documented as "at least 30 days" for stored response data, and "until
# deleted" for conversations. Only the first is a number, and it is a floor.
RESPONSE_RETENTION_FLOOR_DAYS = 30
ITEM_PAGE = 100
FINDINGS = ("retained-past-policy", "items-outlive-response", "thread-unbounded",
"thread-idle", "probe-unreadable")
def parse_records(text):
"""Route recorded ids by prefix. Pure. What cannot be routed is kept.
Dropping an unroutable id would quietly shrink the denominator in a note
whose whole subject is how little of the account it can see.
"""
out = {"responses": [], "conversations": [], "unrecognised": []}
seen = set()
for line in str(text or "").splitlines():
item = line.split("#", 1)[0].strip()
if not item or item in seen:
continue
seen.add(item)
if item.startswith("resp_"):
out["responses"].append(item)
elif item.startswith("conv_"):
out["conversations"].append(item)
else:
out["unrecognised"].append(item)
return out
def response_row(body):
"""One retrieved response, reduced. Pure. Five fields and no chain.
There is deliberately no previous_response_id here. Walking upward from a
response to its parent answers whether a thread still resolves, which is a
different note; this one asks how old the object is and what it is attached
to. There is no `store` field either, because the object does not carry one.
"""
body = body if isinstance(body, dict) else {}
conversation = body.get("conversation")
if isinstance(conversation, dict):
conversation = conversation.get("id")
try:
created = int(body.get("created_at") or 0)
except (TypeError, ValueError):
created = 0
metadata = body.get("metadata")
return {"id": str(body.get("id") or ""),
"created_at": max(0, created),
"status": str(body.get("status") or ""),
"conversation": str(conversation or ""),
"metadata_keys": len(metadata) if isinstance(metadata, dict) else 0}
def item_totals(items):
"""Count and the two timestamps that bound a thread. Pure."""
stamps = []
for item in items or []:
if not isinstance(item, dict):
continue
try:
at = int(item.get("created_at") or 0)
except (TypeError, ValueError):
at = 0
if at > 0:
stamps.append(at)
return {"count": len(items or []),
"oldest": min(stamps) if stamps else 0,
"newest": max(stamps) if stamps else 0}
def age_days(when, now):
"""Age in days. Pure. The clock is an argument. None when undatable."""
try:
at, ref = int(when), int(now)
except (TypeError, ValueError):
return None
return (ref - at) / 86400.0 if at > 0 else None
def grade_response(row, status, now, policy_days):
"""Grade one stored response against YOUR policy. Pure."""
if status == 404:
return ("not-retained",
"nothing is stored under this id. It was created with store "
"false, or it has already aged out")
if status != 200:
return ("probe-unreadable",
"HTTP %s, so nothing about this id was established" % status)
age = age_days((row or {}).get("created_at"), now)
if age is None:
return ("undatable",
"stored, but it carried no usable created_at, so its age "
"cannot be graded")
conversation = str((row or {}).get("conversation") or "")
if age > float(policy_days):
tail = ("" if not conversation else
", and its items were added to conversation %s, which is "
"retained until deleted" % conversation)
return ("retained-past-policy",
"still readable %.1f day(s) after creation, past your %d day "
"policy. Retention is documented as at least %d days, so that "
"is a floor and not a deadline%s"
% (age, int(policy_days), RESPONSE_RETENTION_FLOOR_DAYS, tail))
if conversation:
return ("items-outlive-response",
"%.1f day(s) old and inside your policy, but its items were "
"added to conversation %s, which is retained until deleted"
% (age, conversation))
return ("within-policy",
"stored, %.1f day(s) old, inside your %d day policy"
% (age, int(policy_days)))
def grade_conversation(row, totals, status, now, policy_days, max_items):
"""Grade one conversation on volume first, then on idleness. Pure."""
if status == 404:
return ("not-retained",
"no conversation under this id, so it has already been deleted")
if status != 200:
return ("probe-unreadable",
"HTTP %s, so nothing about this id was established" % status)
totals = totals or {"count": 0, "oldest": 0, "newest": 0}
if int(totals.get("count") or 0) > int(max_items):
return ("thread-unbounded",
"%d item(s) and no TTL, so every turn on this thread carries "
"them as input" % int(totals["count"]))
idle = age_days(totals.get("newest"), now)
if idle is not None and idle > float(policy_days):
return ("thread-idle",
"last item %.1f day(s) ago, past your %d day policy, and "
"conversations are retained until deleted"
% (idle, int(policy_days)))
if idle is None:
return ("thread-undatable",
"%d item(s), none of which carried a usable created_at"
% int(totals.get("count") or 0))
return ("thread-within-policy",
"%d item(s), last active %.1f day(s) ago"
% (int(totals.get("count") or 0), idle))
def coverage_note(records):
"""The sentence that has to appear on every run. Pure."""
records = records or {}
return ("%d id(s) supplied: %d response(s), %d conversation(s), %d "
"unroutable. Neither /v1/responses nor /v1/conversations has a "
"list endpoint, so this is your records and not your account"
% (sum(len(records.get(k) or []) for k in
("responses", "conversations", "unrecognised")),
len(records.get("responses") or []),
len(records.get("conversations") or []),
len(records.get("unrecognised") or [])))
def repair_lines(state):
"""The repair for one verdict. Pure. Printed, never performed."""
items_first = ("delete the items first with DELETE /v1/conversations/"
"{conversation_id}/items/{item_id}, then the conversation. "
"Deleting the conversation does not delete its items.")
if state == "retained-past-policy":
return ["DELETE /v1/responses/{response_id} for what you no longer "
"need, and pass store false on calls carrying regulated data.",
"keep an id ledger with a created_at. It is the only inventory "
"that can exist, because neither collection can be listed."]
if state == "items-outlive-response":
return ["deleting the response is not enough here. " + items_first]
if state == "thread-unbounded":
return ["start a fresh conversation seeded with a summary once a "
"thread gets long, so input tokens stop compounding.",
items_first]
if state == "thread-idle":
return [items_first]
if state == "probe-unreadable":
return ["the key could not read this id. Check that it belongs to the "
"project that created the object before concluding anything "
"about retention."]
if state == "unrecognised-id":
return ["route it by hand, or drop it. An id this script cannot "
"classify is a hole in a coverage figure that is already "
"bounded by your own records."]
return []
def get_json(url, key, params=None, timeout=30):
"""One GET. Returns (status, body). A 404 is the answer, not an error."""
try:
r = requests.get(url, params=params or {},
headers={"Authorization": "Bearer " + key},
timeout=timeout)
except requests.RequestException as exc:
log.debug("GET %s failed: %s", url, exc)
return (None, None)
try:
return (r.status_code, r.json())
except ValueError:
return (r.status_code, None)
def walk_items(conversation_id, key, max_pages):
"""Page a conversation's items on `after`. Returns (items, complete)."""
url = "%s/%s/items" % (CONVERSATIONS_URL, conversation_id)
items, cursor, pages = [], None, 0
while pages < max_pages:
params = {"limit": ITEM_PAGE, "order": "asc"}
if cursor:
params["after"] = cursor
status, body = get_json(url, key, params)
if status != 200 or not isinstance(body, dict):
return (items, False)
data = body.get("data") or []
pages += 1
items.extend(data)
if not data or body.get("has_more") is False:
return (items, True)
cursor = data[-1].get("id")
if not cursor:
return (items, True)
return (items, False)
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--records", required=True,
help="file of recorded resp_ and conv_ ids, one per line")
ap.add_argument("--policy-days", type=int, default=30,
help="your own retention rule, not the platform's")
ap.add_argument("--max-items", type=int, default=500,
help="item count at which a thread stops being long")
ap.add_argument("--max-item-pages", type=int, default=50,
help="page cap when counting one conversation's items")
args = ap.parse_args()
key = os.environ.get("OPENAI_API_KEY")
if not key:
log.error("set OPENAI_API_KEY to a project key set to Read Only. Every "
"call is a GET of a response, a conversation or its items")
return 2
try:
with open(args.records, "r", encoding="utf-8") as fh:
records = parse_records(fh.read())
except OSError as exc:
log.error("could not read %s: %s", args.records, exc)
return 2
probed = len(records["responses"]) + len(records["conversations"])
if not probed:
log.error("no resp_ or conv_ ids in %s. Neither collection can be "
"listed, so the ids have to come from your own records",
args.records)
return 2
now = int(time.time())
log.info("%s", coverage_note(records))
findings = 0
for rid in records["responses"]:
status, body = get_json("%s/%s" % (RESPONSES_URL, rid), key)
row = response_row(body)
state, detail = grade_response(row, status, now, args.policy_days)
emit = log.warning if state in FINDINGS else log.info
emit("%-22s %s: %s", state, rid, detail)
for line in repair_lines(state):
emit(" repair: %s", line)
if state in FINDINGS:
findings += 1
for cid in records["conversations"]:
status, body = get_json("%s/%s" % (CONVERSATIONS_URL, cid), key)
totals = None
if status == 200:
items, complete = walk_items(cid, key, args.max_item_pages)
totals = item_totals(items)
if not complete:
log.warning("%-22s %s: the item listing stopped early, so %d "
"is a floor", "items-incomplete", cid,
totals["count"])
state, detail = grade_conversation(body, totals, status, now,
args.policy_days, args.max_items)
emit = log.warning if state in FINDINGS else log.info
emit("%-22s %s: %s", state, cid, detail)
for line in repair_lines(state):
emit(" repair: %s", line)
if state in FINDINGS:
findings += 1
for other in records["unrecognised"]:
log.info("%-22s %s: neither a resp_ nor a conv_ id, so it was not "
"probed", "unrecognised-id", other)
log.info("%d supplied, %d probed, %d finding(s)",
probed + len(records["unrecognised"]), probed, findings)
return 1 if findings else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Probe recorded response and conversation ids for retention and volume.
*
* Read only. GET /v1/responses/{id}, GET /v1/conversations/{id} and
* GET /v1/conversations/{id}/items. Nothing is created or deleted.
*
* Neither collection has a list endpoint, so this probes the ids you recorded
* and prints a coverage statement every run.
*
* Stored response data is kept for AT LEAST 30 days, which is a floor rather
* than a deadline. Conversations are retained UNTIL DELETED, and their items
* are not deleted when the conversation is.
*
* This never follows previous_response_id: whether a thread still resolves is
* a different question and a different script.
*/
import { readFile } from 'node:fs/promises';
const RESPONSES_URL = 'https://api.openai.com/v1/responses';
const CONVERSATIONS_URL = 'https://api.openai.com/v1/conversations';
export const RESPONSE_RETENTION_FLOOR_DAYS = 30;
const ITEM_PAGE = 100;
const FINDINGS = new Set(['retained-past-policy', 'items-outlive-response',
'thread-unbounded', 'thread-idle', 'probe-unreadable']);
/** Route recorded ids by prefix. Pure. What cannot be routed is kept. */
export function parseRecords(text) {
const out = { responses: [], conversations: [], unrecognised: [] };
const seen = new Set();
for (const line of String(text ?? '').split('\n')) {
const item = line.split('#')[0].trim();
if (!item || seen.has(item)) continue;
seen.add(item);
if (item.startsWith('resp_')) out.responses.push(item);
else if (item.startsWith('conv_')) out.conversations.push(item);
else out.unrecognised.push(item);
}
return out;
}
/** One retrieved response, reduced. Pure. Five fields and no chain. */
export function responseRow(body) {
const row = (body && typeof body === 'object') ? body : {};
const conversation = (row.conversation && typeof row.conversation === 'object')
? row.conversation.id : row.conversation;
const created = Number(row.created_at ?? 0);
const metadata = row.metadata;
return {
id: String(row.id ?? ''),
created_at: Number.isFinite(created) ? Math.max(0, Math.trunc(created)) : 0,
status: String(row.status ?? ''),
conversation: String(conversation ?? ''),
metadata_keys: (metadata && typeof metadata === 'object')
? Object.keys(metadata).length : 0,
};
}
/** Count and the two timestamps that bound a thread. Pure. */
export function itemTotals(items) {
const stamps = [];
for (const item of items ?? []) {
if (!item || typeof item !== 'object') continue;
const at = Number(item.created_at ?? 0);
if (Number.isFinite(at) && at > 0) stamps.push(Math.trunc(at));
}
return {
count: (items ?? []).length,
oldest: stamps.length ? Math.min(...stamps) : 0,
newest: stamps.length ? Math.max(...stamps) : 0,
};
}
/** Age in days. Pure. The clock is an argument. Null when undatable. */
export function ageDays(when, now) {
const at = Number(when);
const ref = Number(now);
if (!Number.isFinite(at) || !Number.isFinite(ref) || at <= 0) return null;
return (ref - at) / 86400;
}
/** Grade one stored response against YOUR policy. Pure. */
export function gradeResponse(row, status, now, policyDays) {
if (status === 404) {
return ['not-retained',
'nothing is stored under this id. It was created with store false, or it '
+ 'has already aged out'];
}
if (status !== 200) {
return ['probe-unreadable',
`HTTP ${status}, so nothing about this id was established`];
}
const age = ageDays((row ?? {}).created_at, now);
if (age === null) {
return ['undatable',
'stored, but it carried no usable created_at, so its age cannot be graded'];
}
const conversation = String((row ?? {}).conversation ?? '');
if (age > Number(policyDays)) {
const tail = conversation
? `, and its items were added to conversation ${conversation}, which is `
+ 'retained until deleted'
: '';
return ['retained-past-policy',
`still readable ${age.toFixed(1)} day(s) after creation, past your `
+ `${Math.trunc(policyDays)} day policy. Retention is documented as at least `
+ `${RESPONSE_RETENTION_FLOOR_DAYS} days, so that is a floor and not a `
+ `deadline${tail}`];
}
if (conversation) {
return ['items-outlive-response',
`${age.toFixed(1)} day(s) old and inside your policy, but its items were `
+ `added to conversation ${conversation}, which is retained until deleted`];
}
return ['within-policy',
`stored, ${age.toFixed(1)} day(s) old, inside your ${Math.trunc(policyDays)} `
+ 'day policy'];
}
/** Grade one conversation on volume first, then on idleness. Pure. */
export function gradeConversation(row, totals, status, now, policyDays, maxItems) {
if (status === 404) {
return ['not-retained',
'no conversation under this id, so it has already been deleted'];
}
if (status !== 200) {
return ['probe-unreadable',
`HTTP ${status}, so nothing about this id was established`];
}
const tot = totals ?? { count: 0, oldest: 0, newest: 0 };
if (Number(tot.count ?? 0) > Number(maxItems)) {
return ['thread-unbounded',
`${Number(tot.count)} item(s) and no TTL, so every turn on this thread `
+ 'carries them as input'];
}
const idle = ageDays(tot.newest, now);
if (idle !== null && idle > Number(policyDays)) {
return ['thread-idle',
`last item ${idle.toFixed(1)} day(s) ago, past your ${Math.trunc(policyDays)} `
+ 'day policy, and conversations are retained until deleted'];
}
if (idle === null) {
return ['thread-undatable',
`${Number(tot.count ?? 0)} item(s), none of which carried a usable created_at`];
}
return ['thread-within-policy',
`${Number(tot.count ?? 0)} item(s), last active ${idle.toFixed(1)} day(s) ago`];
}
/** The sentence that has to appear on every run. Pure. */
export function coverageNote(records) {
const r = records ?? {};
const responses = (r.responses ?? []).length;
const conversations = (r.conversations ?? []).length;
const unrecognised = (r.unrecognised ?? []).length;
return `${responses + conversations + unrecognised} id(s) supplied: `
+ `${responses} response(s), ${conversations} conversation(s), `
+ `${unrecognised} unroutable. Neither /v1/responses nor /v1/conversations `
+ 'has a list endpoint, so this is your records and not your account';
}
/** The repair for one verdict. Pure. Printed, never performed. */
export function repairLines(state) {
const itemsFirst = 'delete the items first with DELETE /v1/conversations/'
+ '{conversation_id}/items/{item_id}, then the conversation. Deleting the '
+ 'conversation does not delete its items.';
if (state === 'retained-past-policy') {
return ['DELETE /v1/responses/{response_id} for what you no longer need, and '
+ 'pass store false on calls carrying regulated data.',
'keep an id ledger with a created_at. It is the only inventory that can '
+ 'exist, because neither collection can be listed.'];
}
if (state === 'items-outlive-response') {
return [`deleting the response is not enough here. ${itemsFirst}`];
}
if (state === 'thread-unbounded') {
return ['start a fresh conversation seeded with a summary once a thread gets '
+ 'long, so input tokens stop compounding.', itemsFirst];
}
if (state === 'thread-idle') return [itemsFirst];
if (state === 'probe-unreadable') {
return ['the key could not read this id. Check that it belongs to the project '
+ 'that created the object before concluding anything about retention.'];
}
if (state === 'unrecognised-id') {
return ['route it by hand, or drop it. An id this script cannot classify is a '
+ 'hole in a coverage figure that is already bounded by your own records.'];
}
return [];
}
async function getJson(url, key, params) {
const target = new URL(url);
for (const [k, v] of Object.entries(params ?? {})) {
target.searchParams.set(k, String(v));
}
try {
const res = await fetch(target, { headers: { Authorization: `Bearer ${key}` } });
const body = await res.json().catch(() => null);
return [res.status, body];
} catch {
return [null, null];
}
}
async function walkItems(conversationId, key, maxPages) {
const url = `${CONVERSATIONS_URL}/${conversationId}/items`;
const items = [];
let cursor = null;
let pages = 0;
while (pages < maxPages) {
const params = { limit: ITEM_PAGE, order: 'asc' };
if (cursor) params.after = cursor;
const [status, body] = await getJson(url, key, params);
if (status !== 200 || !body || typeof body !== 'object') return [items, false];
const data = body.data ?? [];
pages += 1;
items.push(...data);
if (!data.length || body.has_more === false) return [items, true];
cursor = data[data.length - 1]?.id;
if (!cursor) return [items, true];
}
return [items, false];
}
function args(argv) {
const out = { policyDays: 30, maxItems: 500, maxItemPages: 50 };
for (let i = 0; i < argv.length; i += 1) {
if (argv[i] === '--records') out.records = argv[i += 1];
else if (argv[i] === '--policy-days') out.policyDays = Number(argv[i += 1]);
else if (argv[i] === '--max-items') out.maxItems = Number(argv[i += 1]);
else if (argv[i] === '--max-item-pages') out.maxItemPages = Number(argv[i += 1]);
}
return out;
}
async function main() {
const opts = args(process.argv.slice(2));
const key = process.env.OPENAI_API_KEY;
if (!key) {
console.error('set OPENAI_API_KEY to a project key set to Read Only. Every '
+ 'call is a GET of a response, a conversation or its items');
process.exitCode = 2;
return;
}
if (!opts.records) {
console.error('usage: --records <file> [--policy-days 30] [--max-items 500]');
process.exitCode = 2;
return;
}
let records;
try {
records = parseRecords(await readFile(opts.records, 'utf8'));
} catch (err) {
console.error(`could not read ${opts.records}: ${err.message}`);
process.exitCode = 2;
return;
}
const probed = records.responses.length + records.conversations.length;
if (!probed) {
console.error(`no resp_ or conv_ ids in ${opts.records}. Neither collection `
+ 'can be listed, so the ids have to come from your own records');
process.exitCode = 2;
return;
}
const now = Math.trunc(Date.now() / 1000);
console.log(coverageNote(records));
let findings = 0;
for (const id of records.responses) {
const [status, body] = await getJson(`${RESPONSES_URL}/${id}`, key);
const [state, detail] = gradeResponse(responseRow(body), status, now,
opts.policyDays);
console.log(`${state.padEnd(22)} ${id}: ${detail}`);
for (const line of repairLines(state)) console.log(` repair: ${line}`);
if (FINDINGS.has(state)) findings += 1;
}
for (const id of records.conversations) {
const [status, body] = await getJson(`${CONVERSATIONS_URL}/${id}`, key);
let totals = null;
if (status === 200) {
const [items, complete] = await walkItems(id, key, opts.maxItemPages);
totals = itemTotals(items);
if (!complete) {
console.log(`${'items-incomplete'.padEnd(22)} ${id}: the item listing `
+ `stopped early, so ${totals.count} is a floor`);
}
}
const [state, detail] = gradeConversation(body, totals, status, now,
opts.policyDays, opts.maxItems);
console.log(`${state.padEnd(22)} ${id}: ${detail}`);
for (const line of repairLines(state)) console.log(` repair: ${line}`);
if (FINDINGS.has(state)) findings += 1;
}
for (const other of records.unrecognised) {
console.log(`${'unrecognised-id'.padEnd(22)} ${other}: neither a resp_ nor a `
+ 'conv_ id, so it was not probed');
}
console.log(`${probed + records.unrecognised.length} supplied, ${probed} probed, `
+ `${findings} finding(s)`);
process.exitCode = findings ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) await main();
Add a test
The second test is the one that keeps this note off the chain note's ground, so it is worth reading first: the row built from a retrieved response must contain no previous_response_id and no store field, the first because following a parent is a different question and the second because the object does not carry one. Around it, the retention reading in both directions — an object past your own policy is a finding and the wording says floor rather than deadline, while a 404 gets both of its causes named instead of one. Then the conversation trap, which is the most expensive detail here: deleting a conversation does not delete its items, and the repair has to say so in the order the work has to be done. Then volume and idleness graded as two separate states on one object. And last the coverage sentence, asserted to appear whatever the run found, because a probe over your own records can never speak for the account.
from openai_stored_state_probe import (RESPONSE_RETENTION_FLOOR_DAYS, age_days,
coverage_note, grade_conversation,
grade_response, item_totals,
parse_records, repair_lines,
response_row)
NOW = 1_800_000_000
DAY = 86400
def resp(rid, days_old, conversation=None):
return response_row({"id": rid, "object": "response", "status": "completed",
"created_at": NOW - int(days_old * DAY),
"conversation": ({"id": conversation} if conversation
else None),
"metadata": {"tenant": "acme"}})
def items(n, newest_days_old=1.0):
return [{"id": "msg_%d" % i, "type": "message",
"created_at": NOW - int((newest_days_old + n - i - 1) * DAY)}
for i in range(n)]
def test_retention_is_read_as_a_floor_and_a_404_keeps_both_its_causes():
state, detail = grade_response(resp("resp_a19", 94.2), 200, NOW, 30)
assert state == "retained-past-policy"
assert "still readable 94.2 day(s)" in detail
assert "past your 30 day policy" in detail
assert "at least %d days" % RESPONSE_RETENTION_FLOOR_DAYS in detail
assert "a floor and not a deadline" in detail
lines = repair_lines(state)
assert any("DELETE /v1/responses/{response_id}" in line for line in lines)
assert any("id ledger" in line for line in lines)
# A 404 is one fact with two causes and the script names both.
gone, gone_detail = grade_response(None, 404, NOW, 30)
assert gone == "not-retained"
assert "store false" in gone_detail and "aged out" in gone_detail
assert repair_lines(gone) == []
assert grade_response(resp("resp_z1", 1.0), 403, NOW, 30)[0] == "probe-unreadable"
def test_the_row_has_no_chain_in_it_and_no_store_flag():
row = response_row({"id": "resp_b40", "created_at": 1_700_000_000,
"previous_response_id": "resp_a01", "status": "completed",
"conversation": {"id": "conv_x1"},
"metadata": {"tenant": "acme", "env": "prod"}})
assert row == {"id": "resp_b40", "created_at": 1_700_000_000,
"status": "completed", "conversation": "conv_x1",
"metadata_keys": 2}
# Following a parent is the other note. Reading store back is impossible.
assert "previous_response_id" not in row
assert "store" not in row and "stored" not in row
assert response_row(None)["id"] == ""
assert response_row({"created_at": "nonsense"})["created_at"] == 0
assert response_row({"metadata": "nope"})["metadata_keys"] == 0
def test_deleting_the_conversation_does_not_delete_its_items():
state, detail = grade_response(resp("resp_b40", 4.1, "conv_x1"), 200, NOW, 30)
assert state == "items-outlive-response"
assert "inside your policy" in detail
assert "conv_x1" in detail and "retained until deleted" in detail
lines = repair_lines(state)
assert any("not enough here" in line for line in lines)
assert any("items/{item_id}" in line for line in lines)
assert any("does not delete its items" in line for line in lines)
# And the same warning rides along when the response is also over policy.
over = grade_response(resp("resp_c11", 91.0, "conv_x1"), 200, NOW, 30)
assert over[0] == "retained-past-policy"
assert "retained until deleted" in over[1]
def test_volume_and_idleness_are_two_findings_on_one_object():
busy = item_totals(items(4182, newest_days_old=0.5))
assert busy["count"] == 4182 and busy["newest"] > busy["oldest"]
state, detail = grade_conversation({"id": "conv_x1"}, busy, 200, NOW, 30, 500)
assert state == "thread-unbounded"
assert "4182 item(s) and no TTL" in detail
assert any("seeded with a summary" in line for line in repair_lines(state))
# A small thread nobody has touched is the other finding entirely.
idle = item_totals(items(12, newest_days_old=211.4))
state, detail = grade_conversation({"id": "conv_y7"}, idle, 200, NOW, 30, 500)
assert state == "thread-idle"
assert "211.4 day(s) ago" in detail
assert "retained until deleted" in detail
# Busy and recent is neither.
fine = item_totals(items(12, newest_days_old=1.0))
assert grade_conversation({}, fine, 200, NOW, 30, 500)[0] == "thread-within-policy"
assert grade_conversation(None, None, 404, NOW, 30, 500)[0] == "not-retained"
def test_ids_are_routed_by_prefix_and_what_cannot_be_routed_is_kept():
records = parse_records("resp_a19\n\n# exported 2026-08-31\nconv_x1\n"
"resp_a19\nlegacy-7742 # old schema\n \nconv_y7\n")
assert records["responses"] == ["resp_a19"]
assert records["conversations"] == ["conv_x1", "conv_y7"]
assert records["unrecognised"] == ["legacy-7742"]
assert parse_records("") == {"responses": [], "conversations": [],
"unrecognised": []}
assert any("hole in a coverage figure" in line
for line in repair_lines("unrecognised-id"))
def test_the_coverage_sentence_is_printed_whatever_the_run_found():
note = coverage_note({"responses": ["resp_a19"] * 388,
"conversations": ["conv_x1"] * 22,
"unrecognised": ["x", "y"]})
assert "412 id(s) supplied" in note
assert "388 response(s), 22 conversation(s), 2 unroutable" in note
assert "has a list endpoint" in note
assert "your records and not your account" in note
# Even an empty run says it, because the limitation is not a result.
assert "has a list endpoint" in coverage_note({})
assert "0 id(s) supplied" in coverage_note(None)
assert age_days(0, NOW) is None and age_days("x", NOW) is None
assert item_totals(None) == {"count": 0, "oldest": 0, "newest": 0}
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { RESPONSE_RETENTION_FLOOR_DAYS, ageDays, coverageNote, gradeConversation,
gradeResponse, itemTotals, parseRecords, repairLines,
responseRow } from './openai-stored-state-probe.mjs';
const NOW = 1_800_000_000;
const DAY = 86400;
const resp = (id, daysOld, conversation = null) => responseRow({
id, object: 'response', status: 'completed',
created_at: NOW - Math.trunc(daysOld * DAY),
conversation: conversation ? { id: conversation } : null,
metadata: { tenant: 'acme' },
});
const items = (n, newestDaysOld = 1) => Array.from({ length: n }, (_, i) => ({
id: `msg_${i}`, type: 'message',
created_at: NOW - Math.trunc((newestDaysOld + n - i - 1) * DAY),
}));
test('retention is read as a floor and a 404 keeps both its causes', () => {
const [state, detail] = gradeResponse(resp('resp_a19', 94.2), 200, NOW, 30);
assert.equal(state, 'retained-past-policy');
assert.ok(detail.includes('still readable 94.2 day(s)'));
assert.ok(detail.includes('past your 30 day policy'));
assert.ok(detail.includes(`at least ${RESPONSE_RETENTION_FLOOR_DAYS} days`));
assert.ok(detail.includes('a floor and not a deadline'));
const lines = repairLines(state);
assert.ok(lines.some((l) => l.includes('DELETE /v1/responses/{response_id}')));
assert.ok(lines.some((l) => l.includes('id ledger')));
const [gone, goneDetail] = gradeResponse(null, 404, NOW, 30);
assert.equal(gone, 'not-retained');
assert.ok(goneDetail.includes('store false') && goneDetail.includes('aged out'));
assert.deepEqual(repairLines(gone), []);
assert.equal(gradeResponse(resp('resp_z1', 1), 403, NOW, 30)[0], 'probe-unreadable');
});
test('the row has no chain in it and no store flag', () => {
const row = responseRow({ id: 'resp_b40', created_at: 1700000000,
previous_response_id: 'resp_a01', status: 'completed',
conversation: { id: 'conv_x1' },
metadata: { tenant: 'acme', env: 'prod' } });
assert.deepEqual(row, { id: 'resp_b40', created_at: 1700000000,
status: 'completed', conversation: 'conv_x1',
metadata_keys: 2 });
assert.ok(!('previous_response_id' in row));
assert.ok(!('store' in row) && !('stored' in row));
assert.equal(responseRow(null).id, '');
assert.equal(responseRow({ created_at: 'nonsense' }).created_at, 0);
assert.equal(responseRow({ metadata: 'nope' }).metadata_keys, 0);
});
test('deleting the conversation does not delete its items', () => {
const [state, detail] = gradeResponse(resp('resp_b40', 4.1, 'conv_x1'), 200, NOW, 30);
assert.equal(state, 'items-outlive-response');
assert.ok(detail.includes('inside your policy'));
assert.ok(detail.includes('conv_x1') && detail.includes('retained until deleted'));
const lines = repairLines(state);
assert.ok(lines.some((l) => l.includes('not enough here')));
assert.ok(lines.some((l) => l.includes('items/{item_id}')));
assert.ok(lines.some((l) => l.includes('does not delete its items')));
const over = gradeResponse(resp('resp_c11', 91, 'conv_x1'), 200, NOW, 30);
assert.equal(over[0], 'retained-past-policy');
assert.ok(over[1].includes('retained until deleted'));
});
test('volume and idleness are two findings on one object', () => {
const busy = itemTotals(items(4182, 0.5));
assert.equal(busy.count, 4182);
assert.ok(busy.newest > busy.oldest);
const [state, detail] = gradeConversation({ id: 'conv_x1' }, busy, 200, NOW, 30, 500);
assert.equal(state, 'thread-unbounded');
assert.ok(detail.includes('4182 item(s) and no TTL'));
assert.ok(repairLines(state).some((l) => l.includes('seeded with a summary')));
const idle = itemTotals(items(12, 211.4));
const [idleState, idleDetail] = gradeConversation({ id: 'conv_y7' }, idle, 200,
NOW, 30, 500);
assert.equal(idleState, 'thread-idle');
assert.ok(idleDetail.includes('211.4 day(s) ago'));
assert.ok(idleDetail.includes('retained until deleted'));
const fine = itemTotals(items(12, 1));
assert.equal(gradeConversation({}, fine, 200, NOW, 30, 500)[0], 'thread-within-policy');
assert.equal(gradeConversation(null, null, 404, NOW, 30, 500)[0], 'not-retained');
});
test('ids are routed by prefix and what cannot be routed is kept', () => {
const records = parseRecords('resp_a19\n\n# exported 2026-08-31\nconv_x1\n'
+ 'resp_a19\nlegacy-7742 # old schema\n \nconv_y7\n');
assert.deepEqual(records.responses, ['resp_a19']);
assert.deepEqual(records.conversations, ['conv_x1', 'conv_y7']);
assert.deepEqual(records.unrecognised, ['legacy-7742']);
assert.deepEqual(parseRecords(''), { responses: [], conversations: [], unrecognised: [] });
assert.ok(repairLines('unrecognised-id').some((l) => l.includes('hole in a coverage figure')));
});
test('the coverage sentence is printed whatever the run found', () => {
const note = coverageNote({ responses: Array(388).fill('resp_a19'),
conversations: Array(22).fill('conv_x1'),
unrecognised: ['x', 'y'] });
assert.ok(note.includes('412 id(s) supplied'));
assert.ok(note.includes('388 response(s), 22 conversation(s), 2 unroutable'));
assert.ok(note.includes('has a list endpoint'));
assert.ok(note.includes('your records and not your account'));
assert.ok(coverageNote({}).includes('has a list endpoint'));
assert.ok(coverageNote(null).includes('0 id(s) supplied'));
assert.equal(ageDays(0, NOW), null);
assert.equal(ageDays('x', NOW), null);
assert.deepEqual(itemTotals(null), { count: 0, oldest: 0, newest: 0 });
});
FAQ
Can I get a list of everything we have stored?
No, and that is the fact this note is built on rather than a gap in the script. Neither /v1/responses nor /v1/conversations has a list endpoint; both are reachable only by an id you already hold. So the honest answer to a retention question is bounded by your own records, the script prints that sentence on every run next to the number of ids it probed, and the first repair for an application that keeps no ids is to start keeping them with a created-at. That column is the only inventory that can exist.
Is retention 30 days or not?
It depends which page you read, and the difference matters here. The conversation-state guide says response objects are saved for 30 days by default. The data-retention page says stored response data will be kept for at least 30 days. For a note about whether a thread will still resolve next week, the first reading is the useful one and 30 days is a deadline. For a note about what you are holding, the second is the operative one and 30 days is a floor: an object you have not deleted is one you are still holding, and nothing tells you for how much longer. Conversations are simpler and worse: the retention table says until deleted.
Why not just read the store field back off the response?
Because there is not one. store appears in the create request and does not appear anywhere in the retrieved Response object, so there is nothing to read back. That is why a 200 is treated as the only evidence that a response was stored, and why a 404 comes back with both of its causes named (created with store false, or already aged out) instead of being resolved into whichever is more convenient. The script never prints a store value it did not receive.
We deleted all our old conversations. Are we clean?
Probably not, and this is the most expensive detail in the note. Deleting a conversation does not delete the items inside it, and that behaviour is documented on the delete call itself. So a sweep that walks conversation ids and deletes each one removes the container and leaves every message body in place, while reporting complete success. The repair here is printed in the order the work has to happen: delete the items first, then the conversation.
How is this different from the note about previous_response_id?
Same GET, opposite question, and the retention number even means the opposite thing in each. That note walks a chain of parents upward and asks whether the next turn on a thread will still resolve, so for it thirty days is a deadline after which a link vanishes and the finding is a broken conversation. This one never follows a parent at all (the row it builds has no previous_response_id in it, and there is a test that fails if one appears) and it asks how old the objects are, how many items a thread has accumulated, and whether either is past a rule you wrote down.
Related field notes
- The same GET read for whether a thread still resolves tomorrow
- The retention posture that was configured, versus what is held
- The same store read as one number against a ceiling
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Your data, including the retention table
- Conversation state — OpenAI platform docs
- Responses — OpenAI API reference
- Conversations — OpenAI API reference
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.