Diagnostic LLM APIs
a floating model alias silently changes model under you
No error, no deploy, no incident. The evals were 91% on Thursday and 87% on Monday, the mean output length moved, the prompt cache hit rate dropped a few points and the bill went up slightly. Everyone looks at the prompt, which did not change, and at the retrieval corpus, which did not change either. The model changed: the config names an alias, and an alias is a pointer, not a model.
GET /v1/models/{id} for every model string in your config and compare the id that comes back with the one you sent. If they differ, the string is an alias and the returned value is the snapshot it resolves to today — pin that instead.
Two things this check is not. It is not a retirement check: nothing here has an end date, and an alias that moves is the opposite failure to an id that disappears. And from the 4.6 generation onward the dateless id is the snapshot, so appending a date to it is a 404 rather than a tightening.
The problem in plain words
Alias drift is the only failure in this group that produces no error at any point. There is no status code, no missing field, no log line. The application keeps working; it just starts doing slightly different work, and every measurement that would show it — eval scores, output length, cache hit rate, cost per request — moves by an amount small enough to be attributed to noise.
What makes it expensive is where the investigation goes. Nothing in your repository changed, so the search starts in the places that did: data, traffic mix, a dependency bump, a customer's new usage pattern. Days go into that before anybody asks the one question that resolves it, which is what the model string actually resolves to. And because the answer is a moving target, reproducing the old behaviour later requires knowing which snapshot you were on, which nobody recorded.
Why it happens
An alias is a convenience, and its convenience is the problem. For models released before the 4.6 generation the undated string is a pointer: claude-sonnet-4-5 resolves to claude-sonnet-4-5-20250929, claude-haiku-4-5 to claude-haiku-4-5-20251001, claude-opus-4-5 to claude-opus-4-5-20251101. The string is stable; what serves it is not.
The naming convention changed, so the rule you learned is now half wrong. From 4.6 on, the dateless id is itself the pinned snapshot. So claude-opus-4-6, claude-sonnet-4-6, claude-opus-4-8 and their siblings need no date, and adding one produces a 404 for an id that never existed. "Always pin by appending the date" is now a way to break production, which is why this check reads the resolution rather than pattern-matching the string.
The resolution is only knowable by asking. The Models API resolves an alias to a model id, and the returned id is the one that will appear in response.model and in the Admin usage report. Nothing warns you when the pointer moves; the only way to notice is to have recorded what it pointed at before.
Pinning is not the end of the work. A pinned snapshot has a retirement date, which is what the other half of this cluster is about. Pinning trades an invisible failure for a scheduled one, which is a good trade only if something is reading the schedule.
The fix, as a flow
The only question this script asks is what the string resolves to, because the shape of the name stopped answering it: before the 4.6 generation a dateless id was a pointer, and from 4.6 on it is the snapshot itself.
How to fix it
List the model strings your code actually sends
Same collection as any model audit: configs, environment variables, default arguments, fallback branches, batch bodies. An alias in a rarely-exercised path drifts just as much, and is even harder to attribute afterwards.
Ask the API what each one resolves to
GET https://api.anthropic.com/v1/models/{id} with x-api-key and anthropic-version: 2023-06-01. Compare the returned id with the string you sent. Different means alias; identical means the string is already a snapshot.
Read the identical case correctly
A string that resolves to itself and carries no date suffix is a 4.6-or-later id, which is a pinned snapshot in its own right. Do not "fix" it by appending a date. That is a distinct outcome in the report for a reason: the obvious remediation is the one that breaks it.
Record the resolution, not just the finding
Store today's mapping from alias to snapshot alongside your eval results. Without it, a future regression cannot be attributed to a model change, and the previous snapshot cannot be re-pinned to confirm the theory.
Pin, then put the new id on the retirement check
Write the resolved snapshot into the config. Then run the retirement check against it, because a pinned id is one with a date attached, and the whole point of pinning is to know when the change is coming rather than to avoid it forever.
How to check it worked
Re-run after pinning. Every string should resolve to itself, and the only remaining states should be pinned ones.
python3 anthropic_alias_pinning_audit.py --model claude-sonnet-4-6 --model claude-haiku-4-5-20251001
# 2 id(s) checked, 0 unpinned alias(es)The full code
One GET per model string, with a workspace key, and no writes. The classifier compares what you asked for with what came back and takes the current date only to say how old the resolved snapshot is — a snapshot created last week behind an alias you have been calling for a year is the drift, stated as plainly as this check can state it.
"""Report Claude model strings that are aliases rather than pinned snapshots.
Read only. One GET per model string and nothing else: give this a workspace API
key. The repair is printed, never performed, because this script holds a
credential that can spend real money on inference.
"""
import argparse
import datetime as dt
import logging
import os
import re
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("anthropic_alias_pinning_audit")
API = "https://api.anthropic.com/v1"
VERSION = "2023-06-01"
# A trailing -YYYYMMDD. Used only to describe an id, never to decide whether it
# is pinned: that answer comes from the API, because from the 4.6 generation on
# a dateless id is itself a snapshot and pattern-matching gets it backwards.
DATED = re.compile(r"-\d{8}$")
BAD = ("alias", "not-found", "unreadable")
def parse_created(value):
"""Read created_at into a date, or None.
The field is RFC 3339 with a trailing Z, which date.fromisoformat will not
accept before Python 3.11, so the timestamp is cut at the T rather than
parsed whole.
"""
raw = str(value or "").strip()
if not raw:
return None
try:
return dt.date.fromisoformat(raw.split("T")[0])
except ValueError:
return None
def verdict(requested, model, today):
"""Compare a model string with what GET /v1/models/{id} resolved it to.
`model` is the returned object, or None for a 404. Pure, and `today` is
passed in so the age of the resolved snapshot is testable at a fixed date.
Returns (state, detail).
"""
requested = str(requested or "").strip()
if not requested:
return ("unreadable", "empty model string")
if model is None:
return ("not-found",
"404 not_found_error: nothing resolves this id. If a date "
"suffix was appended to a 4.6-or-later id, remove it: those "
"ids are already snapshots and the dated form never existed.")
resolved = str(model.get("id") or "").strip()
if not resolved:
return ("unreadable", "the model object came back with no id")
created = parse_created(model.get("created_at"))
age = ("" if created is None else
" The snapshot behind it was created %s, %d day(s) ago."
% (created.isoformat(), (today - created).days))
if resolved != requested:
return ("alias",
"an alias: it resolves to %s today, and the pointer moves "
"without a deploy or an error.%s Pin %s."
% (resolved, age, resolved))
if DATED.search(requested):
return ("pinned", "a dated snapshot; it resolves to itself.%s" % (age,))
return ("pinned-dateless",
"already a pinned snapshot even though it carries no date: from the "
"4.6 generation on, the dateless id is the snapshot. Do not append "
"a date to it, that id does not exist.%s" % (age,))
def get_model(session, model_id):
"""The model object for one id, or None when the API returns 404."""
r = session.get("%s/models/%s" % (API, model_id), timeout=30)
if r.status_code == 404:
return None
if r.status_code in (401, 403):
raise SystemExit("%d from Anthropic: check ANTHROPIC_API_KEY; an Admin "
"key cannot read the models list" % r.status_code)
r.raise_for_status()
return r.json()
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--model", action="append", default=[],
help="a model string found in your code; repeatable")
ap.add_argument("--from-file",
help="file of model strings, one per line, # for comments")
args = ap.parse_args()
wanted = list(args.model)
if args.from_file:
with open(args.from_file, "r", encoding="utf-8") as fh:
for line in fh:
line = line.split("#", 1)[0].strip()
if line:
wanted.append(line)
wanted = list(dict.fromkeys(wanted))
if not wanted:
log.error("give at least one --model, or a --from-file list")
return 2
key = os.environ.get("ANTHROPIC_API_KEY")
if not key:
log.error("set ANTHROPIC_API_KEY (a workspace key; this script only "
"sends GET requests)")
return 2
session = requests.Session()
session.headers.update({"x-api-key": key, "anthropic-version": VERSION})
today = dt.date.today()
unpinned = 0
for model_id in wanted:
state, detail = verdict(model_id, get_model(session, model_id), today)
line = "%-15s %s %s" % (state, model_id, detail)
if state not in BAD:
log.info(line)
continue
if state == "alias":
unpinned += 1
log.warning(line)
if state == "alias":
log.warning(" repair: write the resolved snapshot into the config "
"in place of the alias, record today's mapping beside "
"your eval results, then check the new id's retirement "
"date")
log.info("%d id(s) checked, %d unpinned alias(es)", len(wanted), unpinned)
return 1 if unpinned else 0
if __name__ == "__main__":
sys.exit(main())
/**
* Report Claude model strings that are aliases rather than pinned snapshots.
*
* Read only. One GET per model string and nothing else: give this a workspace
* API key. The repair is printed, never performed.
*/
import { readFileSync } from 'node:fs';
const API = 'https://api.anthropic.com/v1';
const VERSION = '2023-06-01';
const DAY = 86400000;
// A trailing -YYYYMMDD. Used only to describe an id, never to decide whether it
// is pinned: that answer comes from the API, because from the 4.6 generation on
// a dateless id is itself a snapshot and pattern-matching gets it backwards.
const DATED = /-\d{8}$/;
const BAD = ['alias', 'not-found', 'unreadable'];
/**
* Read created_at into a UTC date, or null. The field is RFC 3339, and only the
* date part is used.
*/
export function parseCreated(value) {
const raw = String(value ?? '').trim().split('T')[0];
if (!/^\d{4}-\d{2}-\d{2}$/.test(raw)) return null;
const ms = Date.parse(`${raw}T00:00:00Z`);
return Number.isNaN(ms) ? null : new Date(ms);
}
/**
* Compare a model string with what GET /v1/models/{id} resolved it to. `model`
* is the returned object, or null for a 404. Pure, and `today` is passed in so
* the age of the resolved snapshot is testable at a fixed date. Returns
* [state, detail].
*/
export function verdict(requested, model, today) {
const asked = String(requested ?? '').trim();
if (!asked) return ['unreadable', 'empty model string'];
if (model === null || model === undefined) {
return ['not-found',
'404 not_found_error: nothing resolves this id. If a date suffix was ' +
'appended to a 4.6-or-later id, remove it: those ids are already ' +
'snapshots and the dated form never existed.'];
}
const resolved = String(model.id ?? '').trim();
if (!resolved) return ['unreadable', 'the model object came back with no id'];
const created = parseCreated(model.created_at);
const age = created === null ? ''
: ` The snapshot behind it was created ${created.toISOString().slice(0, 10)}, ` +
`${Math.round((today.getTime() - created.getTime()) / DAY)} day(s) ago.`;
if (resolved !== asked) {
return ['alias',
`an alias: it resolves to ${resolved} today, and the pointer moves ` +
`without a deploy or an error.${age} Pin ${resolved}.`];
}
if (DATED.test(asked)) {
return ['pinned', `a dated snapshot; it resolves to itself.${age}`];
}
return ['pinned-dateless',
'already a pinned snapshot even though it carries no date: from the 4.6 ' +
'generation on, the dateless id is the snapshot. Do not append a date to ' +
`it, that id does not exist.${age}`];
}
/** The model object for one id, or null when the API returns 404. */
export async function getModel(key, modelId) {
const res = await fetch(`${API}/models/${modelId}`, {
headers: { 'x-api-key': key, 'anthropic-version': VERSION },
});
if (res.status === 404) return null;
if (res.status === 401 || res.status === 403) {
throw new Error(`${res.status} from Anthropic: check ANTHROPIC_API_KEY; an ` +
'Admin key cannot read the models list');
}
if (!res.ok) throw new Error(`${res.status} from /models/${modelId}`);
return res.json();
}
function readIds(argv) {
const ids = [];
argv.forEach((arg, i) => {
if (arg === '--model' && argv[i + 1]) ids.push(argv[i + 1]);
if (arg === '--from-file' && argv[i + 1]) {
for (const line of readFileSync(argv[i + 1], 'utf8').split('\n')) {
const trimmed = line.split('#')[0].trim();
if (trimmed) ids.push(trimmed);
}
}
});
return [...new Set(ids)];
}
async function main() {
const wanted = readIds(process.argv);
if (wanted.length === 0) {
console.error('give at least one --model, or a --from-file list');
process.exitCode = 2;
return;
}
const key = process.env.ANTHROPIC_API_KEY;
if (!key) {
console.error('set ANTHROPIC_API_KEY (a workspace key; this script only ' +
'sends GET requests)');
process.exitCode = 2;
return;
}
const today = new Date(`${new Date().toISOString().slice(0, 10)}T00:00:00Z`);
let unpinned = 0;
for (const modelId of wanted) {
const [state, detail] = verdict(modelId, await getModel(key, modelId), today);
const line = `${state.padEnd(15)} ${modelId} ${detail}`;
if (!BAD.includes(state)) { console.log(line); continue; }
if (state === 'alias') unpinned += 1;
console.warn(line);
if (state === 'alias') {
console.warn(' repair: write the resolved snapshot into the config in ' +
"place of the alias, record today's mapping beside your eval results, " +
"then check the new id's retirement date");
}
}
console.log(`${wanted.length} id(s) checked, ${unpinned} unpinned alias(es)`);
process.exitCode = unpinned ? 1 : 0;
}
// Only run when invoked directly. The test file imports this module, and without
// the guard main() would run there too, fail on the missing key, and set a
// non-zero exit code that fails the whole test file even as every test passes.
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The resolution is handed to the classifier, so every case runs offline. The test that matters most is the dateless one: a 4.6-or-later id resolves to itself and is already pinned, and a script that decides pinning by looking for a date suffix would tell you to append one and hand you a 404 in exchange for a working config.
import datetime as dt
from anthropic_alias_pinning_audit import parse_created, verdict
TODAY = dt.date(2026, 8, 30)
def model(model_id, created="2025-09-29T00:00:00Z"):
return {"id": model_id, "created_at": created, "type": "model"}
def test_a_string_that_resolves_to_something_else_is_an_alias():
state, detail = verdict("claude-sonnet-4-5",
model("claude-sonnet-4-5-20250929"), TODAY)
assert state == "alias"
assert "resolves to claude-sonnet-4-5-20250929" in detail
assert "Pin claude-sonnet-4-5-20250929" in detail
def test_a_dated_id_that_resolves_to_itself_is_pinned():
state, detail = verdict("claude-haiku-4-5-20251001",
model("claude-haiku-4-5-20251001"), TODAY)
assert state == "pinned"
assert "resolves to itself" in detail
def test_a_dateless_id_that_resolves_to_itself_is_also_pinned():
# The trap: appending a date to a 4.6-or-later id gives a 404, so the check
# has to read the resolution rather than look for a date suffix.
state, detail = verdict("claude-opus-4-8", model("claude-opus-4-8"), TODAY)
assert state == "pinned-dateless"
assert "Do not append a date" in detail
def test_a_404_says_what_probably_caused_it():
state, detail = verdict("claude-opus-4-8-20260601", None, TODAY)
assert state == "not-found"
assert "remove it" in detail
def test_the_age_of_the_resolved_snapshot_is_measured_from_the_date_passed_in():
assert parse_created("2025-09-29T00:00:00Z") == dt.date(2025, 9, 29)
assert parse_created("") is None
assert parse_created("last autumn") is None
detail = verdict("claude-sonnet-4-5", model("claude-sonnet-4-5-20250929"),
TODAY)[1]
assert "335 day(s) ago" in detail
def test_a_missing_created_at_drops_the_age_rather_than_inventing_one():
state, detail = verdict("claude-sonnet-4-5",
{"id": "claude-sonnet-4-5-20250929"}, TODAY)
assert state == "alias"
assert "day(s) ago" not in detail
def test_an_empty_string_or_a_headless_object_is_unreadable():
assert verdict("", model("x"), TODAY)[0] == "unreadable"
assert verdict("claude-opus-4-8", {"created_at": "x"}, TODAY)[0] == "unreadable"
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { parseCreated, verdict } from './anthropic-alias-pinning-audit.mjs';
const TODAY = new Date('2026-08-30T00:00:00Z');
const model = (id, created = '2025-09-29T00:00:00Z') =>
({ id, created_at: created, type: 'model' });
test('a string that resolves to something else is an alias', () => {
const [state, detail] = verdict('claude-sonnet-4-5',
model('claude-sonnet-4-5-20250929'), TODAY);
assert.equal(state, 'alias');
assert.match(detail, /resolves to claude-sonnet-4-5-20250929/);
assert.match(detail, /Pin claude-sonnet-4-5-20250929/);
});
test('a dated id that resolves to itself is pinned', () => {
const [state, detail] = verdict('claude-haiku-4-5-20251001',
model('claude-haiku-4-5-20251001'), TODAY);
assert.equal(state, 'pinned');
assert.match(detail, /resolves to itself/);
});
test('a dateless id that resolves to itself is also pinned', () => {
const [state, detail] = verdict('claude-opus-4-8', model('claude-opus-4-8'), TODAY);
assert.equal(state, 'pinned-dateless');
assert.match(detail, /Do not append a date/);
});
test('a 404 says what probably caused it', () => {
const [state, detail] = verdict('claude-opus-4-8-20260601', null, TODAY);
assert.equal(state, 'not-found');
assert.match(detail, /remove it/);
});
test('the age of the resolved snapshot is measured from the date passed in', () => {
assert.equal(parseCreated('2025-09-29T00:00:00Z').toISOString().slice(0, 10),
'2025-09-29');
assert.equal(parseCreated(''), null);
assert.equal(parseCreated('last autumn'), null);
const [, detail] = verdict('claude-sonnet-4-5',
model('claude-sonnet-4-5-20250929'), TODAY);
assert.match(detail, /335 day\(s\) ago/);
});
test('a missing created_at drops the age rather than inventing one', () => {
const [state, detail] = verdict('claude-sonnet-4-5',
{ id: 'claude-sonnet-4-5-20250929' }, TODAY);
assert.equal(state, 'alias');
assert.ok(!/day\(s\) ago/.test(detail));
});
test('an empty string or a headless object is unreadable', () => {
assert.equal(verdict('', model('x'), TODAY)[0], 'unreadable');
assert.equal(verdict('claude-opus-4-8', { created_at: 'x' }, TODAY)[0],
'unreadable');
});
FAQ
How do I know whether a model string is an alias?
Ask. GET /v1/models/{id} returns the model it resolves to, and if that id differs from the string you sent, the string is an alias. Guessing from the shape of the name does not work any more: before the 4.6 generation a dateless id was an alias, and from 4.6 on it is the snapshot itself.
Should I append a date to claude-opus-4-6 to pin it?
No. That id is already a pinned snapshot, and the dated form does not exist, so appending a date returns a 404. This is the most common way the pinning advice gets misapplied, which is why the script gives the dateless-but-pinned case a state of its own instead of quietly calling it fine.
What does an alias moving actually look like in production?
Nothing, at first. There is no error and no deploy. Output length and token counts shift, prompt cache hit rates drop as the cached prefix stops matching, eval scores move a few points, and cost per request changes slightly. Every one of those is individually dismissible as noise, which is why it is usually found weeks later.
If I pin, do I stop having to think about model changes?
You change which problem you have. A pinned snapshot cannot drift, but it does have a retirement date, so it will eventually stop working on a day somebody can look up in advance. That is the trade: an invisible failure exchanged for a scheduled one, and it is only a good trade if the schedule is being read.
Which credential does this need?
A workspace API key, because the Models API lives on the data plane and an Admin key cannot reach it. The script issues GET requests only, and the Admin API is the wrong tool here even though it is the read-only one, which is the sort of inversion worth knowing before you go looking for a safer key.
Related field notes
- A model id past its published shutdown date
- A model still in production retiring in under 90 days
- A retired model id still sitting in the code
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.