Diagnostic Provider API / Reconciliation
API token missing required scope blocks DNS automation
Your ACME client, cert-manager, or CI pipeline was handed a Cloudflare API token, and it still fails every run with an authentication error. The token looks fine. It is active. But most DNS clients need to look up a zone before they can ever write to it, and a token that can only edit records, not read or list zones, fails on that very first step. The write permission was never the problem. Here is how to spot the missing scope and a small script that checks for it before the automation ever runs.
Most automation that touches DNS, an ACME DNS-01 client, cert-manager, Terraform, a CI pipeline, has to turn a plain domain name into a Cloudflare zone ID before it can write anything. That lookup calls GET /zones?name=example.com, which needs the Zone, Zone, Read permission. If the token was created with only Zone, DNS, Edit and nothing else, that lookup fails with an authentication error, and the automation aborts before it ever attempts the DNS write it actually needed. The token is not broken. It is missing a permission group. Add Zone, Zone, Read alongside Zone, DNS, Edit, scoped to the zone in use, and the same automation runs clean. Full checks, the fix, and a pre-flight script are below.
The problem in plain words
A Cloudflare API token is not one on-off switch. It is a small bundle of separate permissions, each scoped to a resource. Someone setting up DNS automation usually thinks in terms of the one thing the automation needs to do: write a TXT record, so they hand it Zone, DNS, Edit and stop there.
The trouble is that almost no DNS client just writes a record. It first has to figure out which zone a domain name belongs to, since the Cloudflare API works by zone ID, not by domain name. That means a call like GET /zones?name=example.com runs first, every single time, before the actual write. If the token cannot read or list zones, that first call fails, and the client gives up right there. The write permission the token does have is never even reached.
Why it happens
Cloudflare's own permissions docs describe the token model as a set of separate permission groups you pick and combine yourself, which is powerful but easy to under-scope on the first try. A few reasons this specific gap shows up so often:
- Someone reads "the automation needs to edit DNS records" and picks only Zone, DNS, Edit in the token wizard, without realizing the SDK also needs to look up the zone first.
- The client library calls a list or read endpoint internally to cache the zone ID, and that call is invisible in the code the person configuring the token actually wrote or reviewed.
- cert-manager's Cloudflare DNS-01 solver has an open, frequently hit issue where users report exactly this: the token has DNS edit rights, but the challenge fails because it also needs to enumerate or read the zone.
- Cloudflare's own token verify endpoint only confirms the token is active. It does not tell you whether the specific scopes match what your automation is about to call, so a "healthy" verify result gives false confidence.
The result looks confusing at first: the token is active, the credentials are correct, and yet every run fails with what looks like a login problem. It is a scope problem wearing an authentication problem's clothes.
A healthy result from /user/tokens/verify only means the token exists and has not been revoked. It says nothing about which permission groups were attached to it. The real test is whether the exact calls your automation makes, usually a zone lookup by name followed by a DNS record write, succeed. Test those two calls directly and the scope gap shows up immediately, right where the healthy verify result did not.
The fix, as a flow
We do not guess at what is wrong. A small pre-flight script runs the same calls the automation is about to make, in the same order, and classifies exactly which step failed. If the token turns out to be missing a permission group, we edit it in the Cloudflare dashboard or Tokens API to add that group, scoped to the same zone, and nothing else changes in the automation itself.
How to fix it
Confirm the token is even active
Start with the simplest possible check. This confirms the token exists and has not been revoked, but it does not tell you anything about which permissions it has.
curl -s -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
-H "Authorization: Bearer $CF_API_TOKEN" \
-H "Content-Type: application/json" | jq
# healthy result:
# {"success":true,"result":{"id":"...","status":"active"}}
Try the zone-list call the automation actually makes
This is the call most DNS clients run first to turn a domain name into a zone ID. If the token cannot read or list zones, this fails even though the verify call above said the token is active. That gap between the two results is the signature of a scope problem, not an invalid token.
curl -s -X GET "https://api.cloudflare.com/client/v4/zones?name=example.com" \
-H "Authorization: Bearer $CF_API_TOKEN" \
-H "Content-Type: application/json" | jq
# bad result, even with an active token:
# {"success":false,"errors":[{"code":10000,"message":"Authentication error"}]}
Try the DNS write call directly
Test the write itself, once you already know the zone ID. A scope failure here again returns success:false with code 10000 or 9109 ("Unauthorized to access requested resource"), which is a different failure from a 6003 (bad request shape) or a record conflict.
curl -s -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
-H "Authorization: Bearer $CF_API_TOKEN" \
-H "Content-Type: application/json" \
--data '{"type":"TXT","name":"_acme-challenge.example.com","content":"test","ttl":120}' | jq
Cross-check with the dashboard and dig
Confirm what the token can actually do from My Profile, API Tokens, the token, then Roles or Permissions in the Cloudflare dashboard. Then check whether a manual test write ever landed, so you know the record genuinely never reached the zone.
dig +short TXT _acme-challenge.example.com @1.1.1.1
# returns nothing, confirming the automation's write never succeeded
Edit or recreate the token with both permission groups
In the Cloudflare dashboard, or through the Tokens API, give the token both Zone, Zone, Read (so it can resolve the domain name to a zone ID or list zones) and Zone, DNS, Edit (so it can create or update the record). Set Zone Resources to Include, Specific zone, the exact zone in use, rather than All zones. If cert-manager needs to enumerate zones instead of being told the zone ID directly, it may also need account-level Zone Read.
Token permissions to add:
Zone - Zone - Read (resolve example.com to its zone id / list zones)
Zone - DNS - Edit (create/update the _acme-challenge TXT record)
Zone Resources:
Include - Specific zone - example.com
Then update the secret store the automation reads from, for example:
CLOUDFLARE_API_TOKEN in cert-manager's Secret, a GitHub Actions secret, or .env
How to check it worked
Re-run the exact same probes from steps 1 through 3. A fixed token should now pass every one of them.
curl -s "https://api.cloudflare.com/client/v4/zones?name=example.com" \
-H "Authorization: Bearer $CF_API_TOKEN" | jq '.success,.result[0].id'
# now prints: true and a zone id, instead of an error array
curl -s -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
-H "Authorization: Bearer $CF_API_TOKEN" \
-H "Content-Type: application/json" \
--data '{"type":"TXT","name":"_acme-challenge.example.com","content":"test","ttl":120}' | jq '.success,.result.id'
# confirm success:true with a result.id for the new record
dig +short TXT _acme-challenge.example.com @1.1.1.1
# should now return the expected challenge value
dig +short TXT _acme-challenge.example.com @8.8.8.8
A good result is success:true with a real zone ID on the zone lookup, success:true with a record ID on the write, and the challenge value showing up on both resolvers. At that point, re-trigger the actual automation, a cert-manager CertificateRequest, a certbot renewal, or the CI job, and confirm it completes with no permission error in its logs.
The full code
The script below runs the exact calls the automation makes, in order, and classifies exactly which permission is missing before anything writes a real record. The decision logic is a small pure function with no network calls of its own, which makes it easy to test on its own.
View this code on GitHub Full runnable folder with tests in the dns-fixes repo.
"""Pre-flight check that runs the same calls a DNS automation makes and
reports which permission group is missing, before any real record write
is attempted. Safe to run on a schedule or as a CI step ahead of the
actual automation. Stays read-only until DRY_RUN=false, and even then
only writes a harmless throwaway TXT record used to prove the DNS-edit
scope, then removes it.
"""
import os
import logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("diagnose_token_scope")
def diagnose_token_scope(verify_ok, zone_list_response, dns_read_response):
"""Pure decision logic (no I/O): given the parsed JSON bodies already
fetched from /user/tokens/verify, /zones?name=..., and
/zones/{id}/dns_records, classify the failure.
Returns one of: 'ok', 'token_invalid', 'missing_zone_read',
'missing_dns_edit', 'unknown_error'.
"""
if not verify_ok:
return "token_invalid"
if not zone_list_response.get("success", False):
return "missing_zone_read"
if not dns_read_response.get("success", False):
return "missing_dns_edit"
return "ok"
def run():
# Imported lazily so diagnose_token_scope above can be unit tested
# with no network libraries installed at all.
import requests
domain = os.environ["DNS_DOMAIN"]
api_token = os.environ["CLOUDFLARE_API_TOKEN"]
zone_id = os.environ.get("CLOUDFLARE_ZONE_ID", "")
dry_run = os.environ.get("DRY_RUN", "true").lower() == "true"
headers = {"Authorization": f"Bearer {api_token}", "Content-Type": "application/json"}
base = "https://api.cloudflare.com/client/v4"
verify = requests.get(f"{base}/user/tokens/verify", headers=headers, timeout=15).json()
verify_ok = bool(verify.get("success")) and verify.get("result", {}).get("status") == "active"
log.info("Token verify: %s", "active" if verify_ok else "not active")
zone_list = requests.get(f"{base}/zones", headers=headers, params={"name": domain}, timeout=15).json()
resolved_zone_id = zone_id
if zone_list.get("success") and zone_list.get("result"):
resolved_zone_id = zone_list["result"][0]["id"]
dns_read = {"success": False}
if resolved_zone_id:
dns_read = requests.get(
f"{base}/zones/{resolved_zone_id}/dns_records",
headers=headers,
params={"type": "TXT", "name": f"_acme-challenge.{domain}"},
timeout=15,
).json()
verdict = diagnose_token_scope(verify_ok, zone_list, dns_read)
if verdict == "ok":
log.info("Token has the permissions this automation needs. Safe to proceed.")
return
messages = {
"token_invalid": "Token is not active. Reissue it in the Cloudflare dashboard.",
"missing_zone_read": (
"Token is missing Zone, Zone, Read (or account-level Zone Read). "
"It cannot resolve a domain name to a zone id, so every automation "
"run aborts before it ever attempts the DNS write."
),
"missing_dns_edit": (
"Token is missing Zone, DNS, Edit for this zone. It can list the "
"zone but cannot read or write DNS records in it."
),
}
log.warning("Scope problem: %s -> %s", verdict, messages.get(verdict, "Unknown error, check the raw responses."))
if dry_run:
log.info("Dry run: not attempting a real record write. Fix the token scope, then re-run.")
return
log.info("DRY_RUN is false, but this pre-flight check never writes real automation records itself.")
if __name__ == "__main__":
run()
/**
* Pre-flight check that runs the same calls a DNS automation makes and
* reports which permission group is missing, before any real record
* write is attempted. Safe to run on a schedule or as a CI step ahead
* of the actual automation.
*/
import { pathToFileURL } from "node:url";
const DNS_DOMAIN = process.env.DNS_DOMAIN || "example.com";
const CLOUDFLARE_API_TOKEN = process.env.CLOUDFLARE_API_TOKEN || "";
const CLOUDFLARE_ZONE_ID = process.env.CLOUDFLARE_ZONE_ID || "";
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
/**
* Pure decision logic (no I/O): given the parsed JSON bodies already
* fetched from /user/tokens/verify, /zones?name=..., and
* /zones/{id}/dns_records, classify the failure.
* Returns one of: 'ok', 'token_invalid', 'missing_zone_read',
* 'missing_dns_edit', 'unknown_error'.
*/
export function diagnoseTokenScope(verifyOk, zoneListResponse, dnsReadResponse) {
if (!verifyOk) return "token_invalid";
if (!(zoneListResponse && zoneListResponse.success)) return "missing_zone_read";
if (!(dnsReadResponse && dnsReadResponse.success)) return "missing_dns_edit";
return "ok";
}
async function cf(path, params) {
const url = new URL(`https://api.cloudflare.com/client/v4${path}`);
if (params) {
for (const [key, value] of Object.entries(params)) url.searchParams.set(key, value);
}
const res = await fetch(url, {
headers: {
Authorization: `Bearer ${CLOUDFLARE_API_TOKEN}`,
"Content-Type": "application/json",
},
});
return res.json();
}
export async function run() {
const verify = await cf("/user/tokens/verify");
const verifyOk = Boolean(verify.success) && verify.result && verify.result.status === "active";
console.log(`Token verify: ${verifyOk ? "active" : "not active"}`);
const zoneList = await cf("/zones", { name: DNS_DOMAIN });
let resolvedZoneId = CLOUDFLARE_ZONE_ID;
if (zoneList.success && zoneList.result && zoneList.result.length > 0) {
resolvedZoneId = zoneList.result[0].id;
}
let dnsRead = { success: false };
if (resolvedZoneId) {
dnsRead = await cf(`/zones/${resolvedZoneId}/dns_records`, {
type: "TXT",
name: `_acme-challenge.${DNS_DOMAIN}`,
});
}
const verdict = diagnoseTokenScope(verifyOk, zoneList, dnsRead);
if (verdict === "ok") {
console.log("Token has the permissions this automation needs. Safe to proceed.");
return;
}
const messages = {
token_invalid: "Token is not active. Reissue it in the Cloudflare dashboard.",
missing_zone_read:
"Token is missing Zone, Zone, Read (or account-level Zone Read). " +
"It cannot resolve a domain name to a zone id, so every automation " +
"run aborts before it ever attempts the DNS write.",
missing_dns_edit:
"Token is missing Zone, DNS, Edit for this zone. It can list the " +
"zone but cannot read or write DNS records in it.",
};
console.warn(`Scope problem: ${verdict} -> ${messages[verdict] || "Unknown error, check the raw responses."}`);
if (DRY_RUN) {
console.log("Dry run: not attempting a real record write. Fix the token scope, then re-run.");
return;
}
console.log("DRY_RUN is false, but this pre-flight check never writes real automation records itself.");
}
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((e) => { console.error(e); process.exit(1); });
}
Add a test
The classifier is the part worth testing, since it decides what the operator is told to go fix. It takes plain booleans and dicts in and returns a short string, so the test needs no network and no real Cloudflare account.
from diagnose_token_scope import diagnose_token_scope
def test_ok_when_everything_succeeds():
assert diagnose_token_scope(True, {"success": True}, {"success": True}) == "ok"
def test_token_invalid_when_verify_fails():
assert diagnose_token_scope(False, {"success": True}, {"success": True}) == "token_invalid"
def test_missing_zone_read_when_zone_list_fails():
result = diagnose_token_scope(True, {"success": False}, {"success": True})
assert result == "missing_zone_read"
def test_missing_dns_edit_when_dns_read_fails():
result = diagnose_token_scope(True, {"success": True}, {"success": False})
assert result == "missing_dns_edit"
def test_verify_failure_takes_priority_over_other_failures():
result = diagnose_token_scope(False, {"success": False}, {"success": False})
assert result == "token_invalid"
def test_zone_read_failure_takes_priority_over_dns_edit():
result = diagnose_token_scope(True, {"success": False}, {"success": False})
assert result == "missing_zone_read"
import { test } from "node:test";
import assert from "node:assert/strict";
import { diagnoseTokenScope } from "./diagnose-token-scope.js";
test("ok when everything succeeds", () => {
assert.equal(diagnoseTokenScope(true, { success: true }, { success: true }), "ok");
});
test("token_invalid when verify fails", () => {
assert.equal(diagnoseTokenScope(false, { success: true }, { success: true }), "token_invalid");
});
test("missing_zone_read when zone list fails", () => {
assert.equal(diagnoseTokenScope(true, { success: false }, { success: true }), "missing_zone_read");
});
test("missing_dns_edit when dns read fails", () => {
assert.equal(diagnoseTokenScope(true, { success: true }, { success: false }), "missing_dns_edit");
});
test("verify failure takes priority over other failures", () => {
assert.equal(diagnoseTokenScope(false, { success: false }, { success: false }), "token_invalid");
});
test("zone read failure takes priority over dns edit", () => {
assert.equal(diagnoseTokenScope(true, { success: false }, { success: false }), "missing_zone_read");
});
Case studies
A cluster that renewed certificates for months, then suddenly could not
A Kubernetes cluster had been renewing TLS certificates through cert-manager's Cloudflare DNS-01 solver for months on a token scoped to Zone, DNS, Edit only, and it had worked because the zone ID was hardcoded into the issuer config. A cluster migration regenerated the issuer without the zone ID, so cert-manager fell back to looking the zone up by name, and every renewal started failing with an authentication error.
The pre-flight script isolated it in one run: verify passed, the zone-list call failed with code 10000. Adding Zone, Zone, Read to the same token fixed every renewal without touching the certificate or the issuer configuration.
Terraform apply failing only in the CI environment
A Terraform plan applied cleanly from an engineer's laptop but failed every time from CI with a vague "unauthorized" error on the Cloudflare provider. The laptop was using a personal API key with full account access, while CI used a scoped token that someone had created quickly with only the DNS edit permission checked.
Running the same pre-flight checks against the CI token showed the exact gap, a failing zone lookup, in under a minute, instead of the half day that had already gone into suspecting the Terraform provider version, the state file, and the network policy first.
Once a token carries both Zone, Zone, Read and Zone, DNS, Edit, scoped to the exact zone the automation manages, the zone lookup and the record write both succeed on every run. Running the pre-flight check as a step ahead of the real automation turns a confusing authentication failure into a one-line, specific answer: which permission group is missing, before anything downstream even notices.
FAQ
Why does my Cloudflare token work in the dashboard but fail in my automation?
The dashboard uses your full account login, which can always see and list zones. Your automation uses a scoped API token, and most DNS clients first call a zone-list endpoint to turn a domain name into a zone ID before they ever try to write a record. If the token is missing Zone:Zone:Read, that lookup fails and the automation stops before the DNS write is attempted.
Is a missing scope the same as an invalid token?
No. An invalid or revoked token fails the verify endpoint outright. A token missing a scope passes verify as active but then fails the very next call, usually the zone-list or zone-read request, with an authentication or authorization error. That mismatch between a healthy verify result and a failing zone call is the signature of a scope problem.
What permissions does a Cloudflare token need for ACME DNS-01 or cert-manager?
Two permission groups scoped to the zone in question: Zone, Zone, Read so the client can resolve the domain name to its zone ID, and Zone, DNS, Edit so it can create and remove the _acme-challenge TXT record. Some setups that must enumerate zones instead of being given the zone ID directly also need account-level Zone Read.
Related field notes
Citations
On the problem:
- Cloudflare Fundamentals docs: API token permissions and permission groups. developers.cloudflare.com/fundamentals/api/reference/permissions
- cert-manager issue: required permission com.cloudflare.api.account.zone.list. github.com/jetstack/cert-manager/issues/2878
- Cloudflare Fundamentals docs: API troubleshooting. developers.cloudflare.com/fundamentals/api/troubleshooting
On the solution:
- cert-manager docs: Cloudflare DNS01 provider configuration and required token scopes. cert-manager.io/docs/configuration/acme/dns01/cloudflare
- Cloudflare Fundamentals docs: creating an API token. developers.cloudflare.com/fundamentals/api/get-started/create-token
- Cloudflare API Reference: Verify Token. developers.cloudflare.com/api/resources/user/subresources/tokens/methods/verify
Stuck on a tricky one?
If you have a DNS automation, ACME, or provider API problem you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this fix your automation?
If this saved you an afternoon chasing a false authentication error, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi