Diagnostic LLM APIs
system_fingerprint moved and seed stopped reproducing
The golden-file suite has been green for eight months. It pins seed, it pins temperature to zero, it asserts on the exact string the model returned the day the fixtures were recorded, and everybody agreed at the time that this was a reasonable way to test an LLM because it worked. This morning forty of them fail. The diffs are trivial — a comma, a reordered clause, one adjective — and nothing in the repository changed. The commit that broke them is not in your repository.
Read the fingerprints you already have. Chat completions created with store: true can be listed: GET /v1/chat/completions?limit=100&order=asc with a project key set to Read Only, optionally filtered by model or by metadata. Each stored completion carries system_fingerprint, which represents the backend configuration the model ran with and exists precisely so that it can be used together with seed to tell when a backend change has happened.
Group by model, order by created, and look for the day the value changed. Two distinct fingerprints for one model id inside your window means every seed-keyed cache entry and every golden file that spans that day is describing a backend that no longer exists. The script prints the transition, both values and the timestamp.
This script does not send a canary completion, and that is a deliberate design decision rather than a limitation. The obvious version of this check posts a fixed prompt and diffs the fingerprint that comes back. That generates, it bills, and this section does not do it. It is also the weaker reading: a canary describes the backend that served one request at the moment you ran it, while your stored completions describe the backend that served the traffic you are actually worried about.
There is an honest failure mode and the script reports it as a finding rather than as silence. On many current models system_fingerprint comes back null or absent. Where every stored completion for a model has no fingerprint, the signal does not exist, you cannot detect a backend change through it even in principle, and the only correct conclusion is to stop building reproducibility on seed for that model.
The problem in plain words
seed is documented as best effort. The exact words are that the system will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result. It is not a guarantee, it has never been one, and the parameter is described as a beta feature. Determinism was always conditional on the backend staying still.
system_fingerprint is the field that tells you whether it did. It represents the backend configuration the model runs with, and its documented purpose is to be used in conjunction with the seed request parameter to understand when backend changes have been made that might impact determinism. When that string changes, the same seed and the same prompt can produce different output, and nothing else announces it: no version bump, no deprecation, no header on your requests, and no entry in any log you own.
What breaks is everything downstream that treated identical output as a contract. Golden-file tests fail with meaningless diffs. A response cache keyed on a hash of (prompt, seed, params) stops hitting, which is expensive and quiet. An evaluation harness that compares a new prompt against a stored baseline now compares two things that differ for two reasons and cannot separate them. And a snapshot test in CI turns into a flaky test, which is worse than a failing one, because the team learns to re-record it.
The migration angle is the sharp end. Moving onto a reasoning model removes the sampling knobs entirely, so a strategy built on seed plus temperature: 0 does not survive the move. And the Responses API object carries neither seed nor system_fingerprint at all, so a team that migrated off Chat Completions did not weaken this signal, they deleted it.
Why it happens
The canary is the obvious design and it is the wrong one twice over. Sending a fixed prompt to compare fingerprints generates output and bills for it, which this section does not do. But even with a budget for it, it answers a narrower question: a canary tells you the configuration that served one request from one host at one moment, while the stored completions tell you which configurations served the requests whose outputs you cached, tested against and shipped. The evidence you want is already sitting in the account.
This works because stored chat completions can be listed, and stored responses cannot. That asymmetry is the whole reason this note has a script. /v1/responses has no list endpoint, so every note in this section that reads a stored response takes a file of ids from your own logs. GET /v1/chat/completions is a real listing with model, metadata, after, limit and order, so the sample is whatever your application stored rather than whatever you happened to write down.
A missing fingerprint is a finding, not a pass. The field is optional and on a good deal of current traffic it is null. A script that only alarmed on a change would report a clean run for a model whose determinism signal is entirely absent, which is the most misleading output it could produce. So absence gets its own verdict, its own repair, and the plainest wording in the script: this cannot be detected here, stop depending on it.
Two fingerprints in a window is not always a clean switchover. If the values interleave — A, then B, then A again — you are being served by a fleet running more than one configuration at once, and two calls made the same afternoon can land on different backends. That is a materially different situation from a single dated change, so the script distinguishes them, because the repair for one is to re-record baselines and the repair for the other is to stop believing in reproducibility altogether.
Stability inside your window is not a guarantee outside it. When one fingerprint holds across the whole sample, the script says that the parameter is behaving as documented so far, and says explicitly that this is not a promise. The documented contract is best effort. A run of good luck is not a change in that contract, and a script that printed a green tick here would be teaching the wrong lesson.
The fix, as a flow
The obvious check here is a canary completion whose fingerprint you diff against a stored baseline, and this section does not send completions. What replaced it is stronger anyway: chat completions stored with store true can be listed, unlike stored responses, so the fingerprints are already sitting on the traffic that actually mattered. Group them by model, put them in order, and read the day the value changed.
How to fix it
Use a project key set to Read Only
Everything here is a GET of /v1/chat/completions. No completion is created, nothing is stored, nothing is deleted, and no admin key is needed: this is project-scoped data rather than organization usage.
Check that anything is stored at all
Only completions created with store: true appear in the listing. An empty list is one of the more useful outcomes: it means this question cannot currently be answered from the API, and the first repair is to store a sample of traffic, or to accept that reproducibility has no evidence behind it. A Zero Data Retention organization will never have anything here, and that is a decision rather than a bug.
List the window, oldest first
limit=100, order=asc, paginating on after, optionally narrowed with --model or a metadata filter if your application tags its calls. Ordering ascending matters: the finding is the moment a value changed, and that is only readable in sequence.
Group by model and read the transitions
Per model id, the script reports the number of completions, how many carry a fingerprint, the distinct values, and every point where consecutive values differ, with the timestamp. It also says whether the values switched once or interleaved, because those are different problems.
Take the repair out of the test suite, not out of the API
There is nothing to fix on the platform. Pin the model snapshot rather than a floating alias, assert on structure and semantics instead of exact strings, record system_fingerprint next to every golden file so a change explains a diff rather than failing a build, and if you need true determinism, cache your own responses.
How to check it worked
Re-run after the baselines are re-recorded. The transition does not disappear, because it is a fact about the platform's history rather than about your code; what changes is that your fixtures now sit entirely on one side of it. The check worth automating is the fingerprint one: run this weekly against the last seven days, and treat a new transition as a scheduled task to re-record baselines rather than as a broken build on a Tuesday morning.
python3 openai_fingerprint_drift.py --days 30
# gpt-5.6-sol 412 stored, 412 with a fingerprint, 2 distinct
# fp_9c1a44d2e8 -> fp_7be03f1a55 at 2026-08-14T02:11:07Z
# fingerprint-moved gpt-5.6-sol ran under 2 backend configurations in this
# window, switching once
# measured: distinct system_fingerprint values on completions you already made
# inferred: that output recorded before the switch is not reproducible after it
# repair: stop using seed as a cache key or a test oracle. Assert on structure
# and semantics, and record system_fingerprint beside every baseline.
# gpt-5.6-terra 88 stored, 0 with a fingerprint, 0 distinct
# fingerprint-absent no stored completion on gpt-5.6-terra carries a
# system_fingerprint, so a backend change cannot be
# detected here even in principle
# repair: do not build reproducibility on seed for this model. There is no
# signal to alarm on, so cache your own responses instead.
# 2 finding(s)The full code
One paged GET and six pure functions. flatten, which turns pages into rows and coerces a missing fingerprint to an empty string rather than letting None reach a comparison; within, which applies the window against a cutoff passed in, so the function is testable without a clock; by_model, which groups and sorts by created because a transition is only visible in order; transitions, which returns every point where consecutive non-empty values differ, with the timestamp; interleaved, which separates one dated switchover from a fleet serving two configurations at once; and verdict, which treats an absent fingerprint as a finding rather than as a clean run.
"""Find the day system_fingerprint moved, using completions you already stored.
Read only, and it sends nothing at all. One paged GET of /v1/chat/completions,
which lists chat completions your application created with store set to true.
No completion is created here, which is why this reads somebody else's stored
traffic rather than posting a canary of its own: a canary would generate, would
bill, and would only describe the backend that served one request at the moment
the script ran.
system_fingerprint represents the backend configuration the model runs with and
exists to be read alongside seed, which is documented as best effort rather than
a guarantee. Two distinct values for one model inside the window means any
seed-keyed cache entry or golden file spanning that point is void.
The field is optional. Where it comes back empty on every stored completion for
a model, that is reported as a finding, because a determinism signal you cannot
read is not a determinism signal.
"""
import argparse
import logging
import os
import sys
import time
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("openai_fingerprint_drift")
LIST_URL = "https://api.openai.com/v1/chat/completions"
MEASURED = ("measured: distinct system_fingerprint values on completions you "
"already made")
INFERRED = ("inferred: that output recorded before the switch is not "
"reproducible after it")
FINDINGS = ("fingerprint-moved", "fingerprint-absent", "nothing-stored")
def iso(ts):
"""A UTC timestamp string. Pure. Empty for anything unusable."""
try:
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(int(ts)))
except (TypeError, ValueError, OSError):
return ""
def flatten(pages):
"""Rows from listing pages. Pure. A missing fingerprint becomes "".
Coerced rather than passed through: None and an absent key both mean "no
fingerprint here", and letting either reach a comparison would make an
absent value look like a distinct one.
"""
rows = []
for page in pages or []:
for item in (page or {}).get("data") or []:
if not isinstance(item, dict):
continue
try:
created = int(item.get("created") or 0)
except (TypeError, ValueError):
created = 0
rows.append({"id": str(item.get("id") or ""),
"created": created,
"model": str(item.get("model") or "(unknown)"),
"fingerprint": str(item.get("system_fingerprint") or "")})
return rows
def within(rows, cutoff):
"""Rows created at or after cutoff. Pure. The clock is passed in."""
if not cutoff:
return list(rows or [])
return [r for r in rows or [] if int(r.get("created") or 0) >= int(cutoff)]
def by_model(rows):
"""{model: [row, ...]} sorted by created. Pure. Order is the whole finding."""
grouped = {}
for row in rows or []:
grouped.setdefault(row.get("model") or "(unknown)", []).append(row)
for model in grouped:
grouped[model].sort(key=lambda r: (int(r.get("created") or 0),
r.get("id") or ""))
return grouped
def transitions(rows):
"""[(created, old, new)] where consecutive fingerprints differ. Pure."""
out = []
previous = ""
for row in rows or []:
current = str(row.get("fingerprint") or "")
if not current:
continue
if previous and current != previous:
out.append((int(row.get("created") or 0), previous, current))
previous = current
return out
def interleaved(rows):
"""True when a fingerprint reappears after another one. Pure.
One dated switchover and a fleet serving two configurations at once look
identical in a set of distinct values and are different problems: the first
invalidates baselines recorded before a date, the second invalidates the
idea that two calls this afternoon agree with each other.
"""
runs = []
for row in rows or []:
current = str(row.get("fingerprint") or "")
if not current:
continue
if not runs or runs[-1] != current:
runs.append(current)
return len(runs) > len(set(runs))
def verdict(model, rows):
"""Grade one model. Pure. Returns (state, detail)."""
rows = list(rows or [])
with_fp = [r for r in rows if r.get("fingerprint")]
distinct = sorted({r["fingerprint"] for r in with_fp})
if not rows:
return ("nothing-stored",
"no stored completions for %s in this window" % model)
if not with_fp:
return ("fingerprint-absent",
"no stored completion on %s carries a system_fingerprint, so a "
"backend change cannot be detected here even in principle"
% model)
if len(distinct) == 1 and len(with_fp) == 1:
return ("single-observation",
"one stored completion on %s carries a fingerprint, which is a "
"reading and not a comparison" % model)
if len(distinct) == 1:
return ("fingerprint-stable",
"%s ran under one backend configuration across %d stored "
"completions. seed is documented as best effort, so this is "
"the parameter behaving rather than a guarantee"
% (model, len(with_fp)))
shape = ("interleaving, so more than one configuration is being served at "
"once" if interleaved(rows) else "switching once")
return ("fingerprint-moved",
"%s ran under %d backend configurations in this window, %s"
% (model, len(distinct), shape))
def repair_lines(state, mixed=False):
"""The repair for one verdict. Pure. Printed, never performed."""
if state == "fingerprint-moved":
lines = ["stop using seed as a cache key or a test oracle. Assert on "
"structure and semantics, and record system_fingerprint beside "
"every baseline so a change explains a diff instead of failing "
"a build.",
"pin the model snapshot rather than a floating alias, so at "
"least the weights are not a second moving part."]
if mixed:
lines.append("the values interleave rather than switching once, so "
"two calls made minutes apart can land on different "
"configurations. Re-recording baselines will not fix "
"that; only caching your own responses will.")
return lines
if state == "fingerprint-absent":
return ["do not build reproducibility on seed for this model. There is "
"no signal to alarm on, so cache your own responses instead.",
"if a test needs stability, freeze the response in the fixture "
"rather than asking the platform to reproduce it."]
if state == "nothing-stored":
return ["nothing was stored, so this question cannot be answered from "
"the API. Set store: true on a sample of traffic, or accept "
"that reproducibility has no evidence behind it.",
"note that the Responses API object carries neither seed nor "
"system_fingerprint, and /v1/responses cannot be listed, so a "
"migration onto it removes this reading entirely."]
if state == "fingerprint-stable":
return ["nothing to do today. Keep this run on a schedule: the value "
"held across the window, which is best effort holding, not a "
"promise that it will."]
if state == "single-observation":
return ["store more traffic or widen the window. One fingerprint is a "
"reading, and this note needs two to say anything."]
return []
def fetch(key, model=None, metadata=None, timeout=30):
"""Paged GET of the stored chat completions. Returns (pages, error)."""
pages = []
params = {"limit": 100, "order": "asc"}
if model:
params["model"] = model
for pair in metadata or []:
name, _, value = str(pair).partition("=")
if name and value:
params["metadata[%s]" % name.strip()] = value.strip()
headers = {"Authorization": "Bearer " + key}
for _ in range(200):
try:
r = requests.get(LIST_URL, headers=headers, params=params,
timeout=timeout)
except requests.RequestException as exc:
return (pages, "request failed: %s" % exc)
if r.status_code != 200:
return (pages, "HTTP %d %s" % (r.status_code, (r.text or "")[:160]))
body = r.json()
pages.append(body)
if not body.get("has_more") or not body.get("last_id"):
break
params["after"] = body["last_id"]
return (pages, None)
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--days", type=int, default=30,
help="how far back to read, in days")
ap.add_argument("--model", help="narrow the listing to one model id")
ap.add_argument("--metadata", action="append", default=[],
help="key=value filter, if your calls are tagged")
args = ap.parse_args()
key = os.environ.get("OPENAI_API_KEY")
if not key:
log.error("set OPENAI_API_KEY to a project key set to Read Only. It is "
"used for one paged GET of /v1/chat/completions")
return 2
pages, err = fetch(key, args.model, args.metadata)
if err:
log.error("%s", err)
return 2
cutoff = int(time.time()) - args.days * 86400
rows = within(flatten(pages), cutoff)
grouped = by_model(rows)
findings = 0
if not rows:
state, detail = verdict("(any model)", [])
log.warning("%-20s %s", state, detail)
for line in repair_lines(state):
log.warning(" repair: %s", line)
log.info("1 finding(s)")
return 1
for model in sorted(grouped):
entries = grouped[model]
with_fp = [r for r in entries if r.get("fingerprint")]
distinct = sorted({r["fingerprint"] for r in with_fp})
log.info("%-20s %d stored, %d with a fingerprint, %d distinct",
model, len(entries), len(with_fp), len(distinct))
for created, old, new in transitions(entries):
log.warning(" %s -> %s at %s", old, new, iso(created))
state, detail = verdict(model, entries)
emit = log.warning if state in FINDINGS else log.info
emit("%-20s %s", state, detail)
if state == "fingerprint-moved":
emit(" %s", MEASURED)
emit(" %s", INFERRED)
for line in repair_lines(state, interleaved(entries)):
emit(" repair: %s", line)
if state in FINDINGS:
findings += 1
log.info("%d finding(s)", findings)
return 1 if findings else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Find the day system_fingerprint moved, using completions you already stored.
*
* Read only, and it sends nothing at all. One paged GET of
* /v1/chat/completions, which lists chat completions created with store set to
* true. No canary is posted: that would generate and bill, and it would only
* describe the backend serving one request at the moment the script ran.
*
* The field is optional. Where it is empty on every stored completion for a
* model, that is a finding, because a signal you cannot read is not a signal.
*/
const LIST_URL = 'https://api.openai.com/v1/chat/completions';
export const MEASURED =
'measured: distinct system_fingerprint values on completions you already made';
const FINDINGS = new Set(['fingerprint-moved', 'fingerprint-absent', 'nothing-stored']);
/** A UTC timestamp string. Pure. Empty for anything unusable. */
export function iso(ts) {
const seconds = Number(ts);
if (!Number.isFinite(seconds) || seconds <= 0) return '';
return `${new Date(seconds * 1000).toISOString().slice(0, 19)}Z`;
}
/** Rows from listing pages. Pure. A missing fingerprint becomes ''. */
export function flatten(pages) {
const rows = [];
for (const page of pages ?? []) {
for (const item of page?.data ?? []) {
if (!item || typeof item !== 'object') continue;
const created = Number(item.created ?? 0);
rows.push({
id: String(item.id ?? ''),
created: Number.isFinite(created) ? Math.trunc(created) : 0,
model: String(item.model ?? '(unknown)'),
fingerprint: String(item.system_fingerprint ?? ''),
});
}
}
return rows;
}
/** Rows created at or after cutoff. Pure. The clock is passed in. */
export function within(rows, cutoff) {
if (!cutoff) return [...(rows ?? [])];
return (rows ?? []).filter((r) => Number(r?.created ?? 0) >= Number(cutoff));
}
/** {model: [row]} sorted by created. Pure. Order is the whole finding. */
export function byModel(rows) {
const grouped = {};
for (const row of rows ?? []) {
const model = row?.model || '(unknown)';
(grouped[model] ??= []).push(row);
}
for (const model of Object.keys(grouped)) {
grouped[model].sort((a, b) => (a.created - b.created)
|| String(a.id).localeCompare(String(b.id)));
}
return grouped;
}
/** [[created, old, new]] where consecutive fingerprints differ. Pure. */
export function transitions(rows) {
const out = [];
let previous = '';
for (const row of rows ?? []) {
const current = String(row?.fingerprint ?? '');
if (!current) continue;
if (previous && current !== previous) {
out.push([Math.trunc(Number(row.created ?? 0)), previous, current]);
}
previous = current;
}
return out;
}
/** True when a fingerprint reappears after another one. Pure. */
export function interleaved(rows) {
const runs = [];
for (const row of rows ?? []) {
const current = String(row?.fingerprint ?? '');
if (!current) continue;
if (!runs.length || runs[runs.length - 1] !== current) runs.push(current);
}
return runs.length > new Set(runs).size;
}
/** Grade one model. Pure. Returns [state, detail]. */
export function verdict(model, rows) {
const list = [...(rows ?? [])];
const withFp = list.filter((r) => r?.fingerprint);
const distinct = [...new Set(withFp.map((r) => r.fingerprint))].sort();
if (!list.length) {
return ['nothing-stored', `no stored completions for ${model} in this window`];
}
if (!withFp.length) {
return ['fingerprint-absent',
`no stored completion on ${model} carries a system_fingerprint, so a `
+ 'backend change cannot be detected here even in principle'];
}
if (distinct.length === 1 && withFp.length === 1) {
return ['single-observation',
`one stored completion on ${model} carries a fingerprint, which is a `
+ 'reading and not a comparison'];
}
if (distinct.length === 1) {
return ['fingerprint-stable',
`${model} ran under one backend configuration across ${withFp.length} `
+ 'stored completions. seed is documented as best effort, so this is the '
+ 'parameter behaving rather than a guarantee'];
}
const shape = interleaved(list)
? 'interleaving, so more than one configuration is being served at once'
: 'switching once';
return ['fingerprint-moved',
`${model} ran under ${distinct.length} backend configurations in this `
+ `window, ${shape}`];
}
/** The repair for one verdict. Pure. Printed, never performed. */
export function repairLines(state, mixed = false) {
if (state === 'fingerprint-moved') {
const lines = ['stop using seed as a cache key or a test oracle. Assert on '
+ 'structure and semantics, and record system_fingerprint beside every '
+ 'baseline so a change explains a diff instead of failing a build.',
'pin the model snapshot rather than a floating alias, so at least the '
+ 'weights are not a second moving part.'];
if (mixed) {
lines.push('the values interleave rather than switching once, so two calls '
+ 'made minutes apart can land on different configurations. Re-recording '
+ 'baselines will not fix that; only caching your own responses will.');
}
return lines;
}
if (state === 'fingerprint-absent') {
return ['do not build reproducibility on seed for this model. There is no '
+ 'signal to alarm on, so cache your own responses instead.',
'if a test needs stability, freeze the response in the fixture rather than '
+ 'asking the platform to reproduce it.'];
}
if (state === 'nothing-stored') {
return ['nothing was stored, so this question cannot be answered from the '
+ 'API. Set store: true on a sample of traffic, or accept that '
+ 'reproducibility has no evidence behind it.',
'note that the Responses API object carries neither seed nor '
+ 'system_fingerprint, and /v1/responses cannot be listed, so a migration '
+ 'onto it removes this reading entirely.'];
}
if (state === 'fingerprint-stable') {
return ['nothing to do today. Keep this run on a schedule: the value held '
+ 'across the window, which is best effort holding, not a promise that it '
+ 'will.'];
}
if (state === 'single-observation') {
return ['store more traffic or widen the window. One fingerprint is a '
+ 'reading, and this note needs two to say anything.'];
}
return [];
}
async function fetchPages(key, model, metadata) {
const pages = [];
const params = new URLSearchParams({ limit: '100', order: 'asc' });
if (model) params.set('model', model);
for (const pair of metadata ?? []) {
const at = String(pair).indexOf('=');
if (at > 0) {
params.set(`metadata[${pair.slice(0, at).trim()}]`, pair.slice(at + 1).trim());
}
}
for (let page = 0; page < 200; page += 1) {
const url = `${LIST_URL}?${params.toString()}`;
let res;
try {
res = await fetch(url, { headers: { Authorization: `Bearer ${key}` } });
} catch (err) {
return [pages, `request failed: ${err.message}`];
}
if (res.status !== 200) {
return [pages, `HTTP ${res.status} ${(await res.text()).slice(0, 160)}`];
}
const body = await res.json();
pages.push(body);
if (!body.has_more || !body.last_id) break;
params.set('after', body.last_id);
}
return [pages, null];
}
function args(argv) {
const out = { days: 30, metadata: [] };
for (let i = 0; i < argv.length; i += 1) {
if (argv[i] === '--days') out.days = Number.parseInt(argv[i += 1], 10);
else if (argv[i] === '--model') out.model = argv[i += 1];
else if (argv[i] === '--metadata') out.metadata.push(argv[i += 1]);
}
return out;
}
async function main() {
const opts = args(process.argv.slice(2));
const key = process.env.OPENAI_API_KEY;
if (!key) {
console.error('set OPENAI_API_KEY to a project key set to Read Only. It is '
+ 'used for one paged GET of /v1/chat/completions');
process.exitCode = 2;
return;
}
const [pages, err] = await fetchPages(key, opts.model, opts.metadata);
if (err) {
console.error(err);
process.exitCode = 2;
return;
}
const cutoff = Math.trunc(Date.now() / 1000) - (opts.days || 30) * 86400;
const rows = within(flatten(pages), cutoff);
const grouped = byModel(rows);
let findings = 0;
if (!rows.length) {
const [state, detail] = verdict('(any model)', []);
console.log(`${state.padEnd(20)} ${detail}`);
for (const line of repairLines(state)) console.log(` repair: ${line}`);
console.log('1 finding(s)');
process.exitCode = 1;
return;
}
for (const model of Object.keys(grouped).sort()) {
const entries = grouped[model];
const withFp = entries.filter((r) => r.fingerprint);
const distinct = new Set(withFp.map((r) => r.fingerprint));
console.log(`${model.padEnd(20)} ${entries.length} stored, ${withFp.length} `
+ `with a fingerprint, ${distinct.size} distinct`);
for (const [created, old, next] of transitions(entries)) {
console.log(` ${old} -> ${next} at ${iso(created)}`);
}
const [state, detail] = verdict(model, entries);
console.log(`${state.padEnd(20)} ${detail}`);
if (state === 'fingerprint-moved') {
console.log(` ${MEASURED}`);
console.log(' inferred: that output recorded before the switch is not '
+ 'reproducible after it');
}
for (const line of repairLines(state, interleaved(entries))) {
console.log(` repair: ${line}`);
}
if (FINDINGS.has(state)) findings += 1;
}
console.log(`${findings} finding(s)`);
process.exitCode = findings ? 1 : 0;
}
if (import.meta.url === `file://${process.argv[1]}`) await main();
Add a test
The first test is the note itself: two fingerprints on one model, in order, produce fingerprint-moved and a transition carrying both values and the timestamp of the first completion under the new one. The second is the outcome that most scripts would get wrong — every stored completion with a null fingerprint must be a finding named fingerprint-absent, never a quiet pass, and its repair has to say that the signal does not exist rather than that nothing changed. The third separates one dated switchover from an interleaved fleet, because the first is repaired by re-recording baselines and the second is not repaired at all. Then the single observation, which refuses to conclude. Then the empty listing, which must point at store and at the Responses API by name. And last, ordering: rows arriving out of sequence must be sorted by created before any transition is read off them, since an unsorted list invents transitions that never happened.
from openai_fingerprint_drift import (by_model, flatten, interleaved, iso,
repair_lines, transitions, verdict, within)
DAY = 86400
def page(*rows):
return {"object": "list", "data": list(rows), "has_more": False}
def completion(cid, created, model, fingerprint):
return {"id": cid, "object": "chat.completion", "created": created,
"model": model, "system_fingerprint": fingerprint}
def test_two_fingerprints_in_order_are_the_finding_with_a_date():
rows = flatten([page(completion("c_1", 1000, "gpt-5.6-sol", "fp_aa11"),
completion("c_2", 2000, "gpt-5.6-sol", "fp_aa11"),
completion("c_3", 3000, "gpt-5.6-sol", "fp_bb22"))])
entries = by_model(rows)["gpt-5.6-sol"]
assert transitions(entries) == [(3000, "fp_aa11", "fp_bb22")]
state, detail = verdict("gpt-5.6-sol", entries)
assert state == "fingerprint-moved"
assert "2 backend configurations" in detail and "switching once" in detail
assert iso(3000) == "1970-01-01T00:50:00Z"
assert any("not a test oracle" in line or "test oracle" in line
for line in repair_lines(state))
def test_an_absent_fingerprint_is_a_finding_and_never_a_quiet_pass():
rows = flatten([page(completion("c_1", 1000, "gpt-5.6-terra", None),
completion("c_2", 2000, "gpt-5.6-terra", ""))])
entries = by_model(rows)["gpt-5.6-terra"]
assert transitions(entries) == []
state, detail = verdict("gpt-5.6-terra", entries)
assert state == "fingerprint-absent"
assert "even in principle" in detail
lines = repair_lines(state)
assert any("no signal to alarm on" in line for line in lines)
# Never phrased as stability. Nothing was observed.
assert not any("stable" in line for line in lines)
def test_an_interleaved_fleet_is_separated_from_one_dated_switchover():
mixed = [{"fingerprint": "fp_aa11"}, {"fingerprint": "fp_bb22"},
{"fingerprint": "fp_aa11"}]
once = [{"fingerprint": "fp_aa11"}, {"fingerprint": "fp_aa11"},
{"fingerprint": "fp_bb22"}]
assert interleaved(mixed) and not interleaved(once)
state, detail = verdict("gpt-5.6-sol", mixed)
assert state == "fingerprint-moved"
assert "more than one configuration is being served at once" in detail
assert any("minutes apart" in line for line in repair_lines(state, True))
assert not any("minutes apart" in line for line in repair_lines(state, False))
def test_one_fingerprint_is_a_reading_rather_than_a_comparison():
single = [{"fingerprint": "fp_aa11"}]
assert verdict("gpt-5.6-sol", single)[0] == "single-observation"
steady = [{"fingerprint": "fp_aa11"}] * 40
state, detail = verdict("gpt-5.6-sol", steady)
assert state == "fingerprint-stable"
assert "best effort" in detail
assert any("not a promise" in line for line in repair_lines(state))
def test_an_empty_listing_points_at_store_and_at_the_responses_api():
state, detail = verdict("(any model)", [])
assert state == "nothing-stored"
assert "no stored completions" in detail
lines = repair_lines(state)
assert any("store: true" in line for line in lines)
assert any("Responses API" in line and "cannot be listed" in line
for line in lines)
def test_rows_are_ordered_before_transitions_are_read_off_them():
rows = flatten([page(completion("c_2", 2000, "m", "fp_aa11"),
completion("c_3", 3000, "m", "fp_bb22"),
completion("c_1", 1000, "m", "fp_aa11"))])
# Unsorted, this reads as two transitions. Sorted, it is one.
assert len(transitions(rows)) == 2
assert transitions(by_model(rows)["m"]) == [(3000, "fp_aa11", "fp_bb22")]
assert len(within(by_model(rows)["m"], 1500)) == 2
assert within(rows, 0) == rows
assert iso("nonsense") == "" and iso(None) == ""
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { byModel, flatten, interleaved, iso, repairLines, transitions, verdict,
within } from './openai-fingerprint-drift.mjs';
const page = (...rows) => ({ object: 'list', data: rows, has_more: false });
const completion = (id, created, model, system_fingerprint) =>
({ id, object: 'chat.completion', created, model, system_fingerprint });
test('two fingerprints in order are the finding with a date', () => {
const rows = flatten([page(completion('c_1', 1000, 'gpt-5.6-sol', 'fp_aa11'),
completion('c_2', 2000, 'gpt-5.6-sol', 'fp_aa11'),
completion('c_3', 3000, 'gpt-5.6-sol', 'fp_bb22'))]);
const entries = byModel(rows)['gpt-5.6-sol'];
assert.deepEqual(transitions(entries), [[3000, 'fp_aa11', 'fp_bb22']]);
const [state, detail] = verdict('gpt-5.6-sol', entries);
assert.equal(state, 'fingerprint-moved');
assert.ok(detail.includes('2 backend configurations'));
assert.ok(detail.includes('switching once'));
assert.equal(iso(3000), '1970-01-01T00:50:00Z');
assert.ok(repairLines(state).some((l) => l.includes('test oracle')));
});
test('an absent fingerprint is a finding and never a quiet pass', () => {
const rows = flatten([page(completion('c_1', 1000, 'gpt-5.6-terra', null),
completion('c_2', 2000, 'gpt-5.6-terra', ''))]);
const entries = byModel(rows)['gpt-5.6-terra'];
assert.deepEqual(transitions(entries), []);
const [state, detail] = verdict('gpt-5.6-terra', entries);
assert.equal(state, 'fingerprint-absent');
assert.ok(detail.includes('even in principle'));
const lines = repairLines(state);
assert.ok(lines.some((l) => l.includes('no signal to alarm on')));
assert.ok(!lines.some((l) => l.includes('stable')));
});
test('an interleaved fleet is separated from one dated switchover', () => {
const mixed = [{ fingerprint: 'fp_aa11' }, { fingerprint: 'fp_bb22' },
{ fingerprint: 'fp_aa11' }];
const once = [{ fingerprint: 'fp_aa11' }, { fingerprint: 'fp_aa11' },
{ fingerprint: 'fp_bb22' }];
assert.ok(interleaved(mixed));
assert.ok(!interleaved(once));
const [state, detail] = verdict('gpt-5.6-sol', mixed);
assert.equal(state, 'fingerprint-moved');
assert.ok(detail.includes('more than one configuration is being served at once'));
assert.ok(repairLines(state, true).some((l) => l.includes('minutes apart')));
assert.ok(!repairLines(state, false).some((l) => l.includes('minutes apart')));
});
test('one fingerprint is a reading rather than a comparison', () => {
assert.equal(verdict('gpt-5.6-sol', [{ fingerprint: 'fp_aa11' }])[0],
'single-observation');
const steady = Array.from({ length: 40 }, () => ({ fingerprint: 'fp_aa11' }));
const [state, detail] = verdict('gpt-5.6-sol', steady);
assert.equal(state, 'fingerprint-stable');
assert.ok(detail.includes('best effort'));
assert.ok(repairLines(state).some((l) => l.includes('not a promise')));
});
test('an empty listing points at store and at the responses api', () => {
const [state, detail] = verdict('(any model)', []);
assert.equal(state, 'nothing-stored');
assert.ok(detail.includes('no stored completions'));
const lines = repairLines(state);
assert.ok(lines.some((l) => l.includes('store: true')));
assert.ok(lines.some((l) => l.includes('Responses API') && l.includes('cannot be listed')));
});
test('rows are ordered before transitions are read off them', () => {
const rows = flatten([page(completion('c_2', 2000, 'm', 'fp_aa11'),
completion('c_3', 3000, 'm', 'fp_bb22'),
completion('c_1', 1000, 'm', 'fp_aa11'))]);
assert.equal(transitions(rows).length, 2);
assert.deepEqual(transitions(byModel(rows).m), [[3000, 'fp_aa11', 'fp_bb22']]);
assert.equal(within(byModel(rows).m, 1500).length, 2);
assert.deepEqual(within(rows, 0), rows);
assert.equal(iso('nonsense'), '');
assert.equal(iso(null), '');
});
FAQ
Why not just send one request and compare the fingerprint?
Because that generates a completion and bills for it, and no script in this section does that. It is also the weaker measurement. A canary tells you which backend configuration served one request at the moment you ran the script; your stored completions tell you which configurations served the traffic whose outputs you cached, tested against and shipped. The evidence is already in the account, it covers the window you care about, and reading it costs nothing.
Our completions are not stored. Can we still check this?
Not through the API, and the script says so as a finding rather than pretending the run was clean. Listing only returns chat completions created with store set to true, and a Zero Data Retention organization will never have any. The options are to store a representative sample of traffic, or to accept that reproducibility has no evidence behind it and stop depending on it. Both are decisions worth making explicitly.
We moved to the Responses API. What is the equivalent?
There is not one, and that is the honest answer. The Response object carries no system_fingerprint, and /v1/responses has no list endpoint, so neither half of this reading survives the migration. If reproducibility matters to you, that is an argument for keeping a stored sample on Chat Completions, or for caching your own responses so determinism is a property of your infrastructure rather than a favour from someone else's.
The fingerprint is null on our model. Is that a bug?
No. The field is optional, and it is empty on a good deal of current traffic. What it means is specific and worth being blunt about: there is no signal, so a backend change cannot be detected here even in principle, and any determinism strategy for that model is running without instrumentation. The script reports it as a finding for that reason. A run that called it stable would be claiming an observation it never made.
Does pinning a model snapshot fix this?
It removes one of the two moving parts, not both. Pinning stops the weights changing under a floating alias, which is a separate published note and a good idea regardless. The fingerprint describes the backend configuration the model runs with, which includes infrastructure beyond the weights, so it can move under a pinned snapshot as well. Pin the snapshot, record the fingerprint next to every baseline, and assert on structure rather than on exact strings.
Related field notes
- The model string in your config, and what it resolves to today
- The other step change that lines up with the day a model id moved
- The other number that stopped meaning what it meant last quarter
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- Chat Completions — OpenAI API reference
- Reproducible outputs with the seed parameter — OpenAI Cookbook
- Models — OpenAI API reference
- Your data — OpenAI platform docs
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.