Skip to content

Diagnostic GitHub API

The org's IP allow list refuses the runner, not the token

The job fails in CI and passes on the laptop. Same repository, same endpoint, same token — the engineer copies the token out of the secret store and runs the identical command locally to prove it, and it works. So the token is fine, and the code is fine, and the only thing left is the API, which is up. Meanwhile anonymous calls to the organization's public repositories keep answering 200 from the same runner, which is taken as proof that the network is fine too. It is not the token and not the network. The organization restricts which source addresses may reach it, and the runner's address has never been on the list.

Read-only token Python and Node.js Tests included
An online store on a laptop
Photo by charlesdeluvio on Unsplash
The short answer

An organization IP allow list is a check on where the request came from, applied to a credential that is otherwise perfectly good. Ephemeral CI egress addresses and serverless NAT pools are rarely on it, which is why the laptop succeeds and the runner does not.

Read the body of the 403. This refusal is the only one on the API that names an IP address: Although you appear to have the correct authorization credentials, the ACME organization has an IP allow list enabled, and 203.0.113.9 is not permitted to access this resource. That address is the one GitHub actually saw, which is better evidence than any echo service, because it is the address the rule was applied to. Where you hold admin:org, the list itself is readable through the GraphQL ipAllowListEntries field and can be compared against that address directly.

The problem in plain words

What makes this expensive is that every cheap experiment exonerates the thing that is broken. Copy the token to a laptop: it works, so the token is good. Curl a public repository from the runner: it answers, so egress is good. Read the token's scopes: they are what they always were. Each of those is a true observation and none of them touches the actual rule, because the rule only applies to the organization's own resources and only to the address the request arrives from.

Then the search goes somewhere unhelpful. A 403 is the status a missing permission produces, so somebody widens the token, and the refusal is unchanged. Somebody swaps a classic token for a fine-grained one, and it is unchanged again. Somebody moves the job to a different runner pool and it starts working, which looks like the fix but is really the first correct reading anybody has taken: that pool's egress happens to be allowed.

The other reason it hides is that the change is usually not yours. Somebody in the organization turned on an allow list, or added an entry and removed a broader one, and the integration is not mentioned in that work at all. From inside the failing job there is no event, no deprecation notice and no header. There is a sentence in a response body that most clients throw away in favour of the status code.

403 on therunnerone organizationToken triedlocallyand it worksScopes widenedsame refusalPublic repoprobedanswers 200Address neverallowedon the org listand another credential is minted
Every reading in this chain is true and none of them touches the rule. The variable being changed is not the variable.

Why it happens

The check runs against the connection, not the credential. An allow list narrows the set of source addresses permitted to reach the organization's resources. Passing it is a precondition; it is not something a token can carry. That is why widening scopes changes nothing and why the same token is simultaneously allowed and refused, depending only on which machine it is used from.

The refusal names the address, and that is the whole diagnostic. Every other 403 on this API describes a rule; this one describes a rule and then tells you the value that failed it. A runner behind a NAT pool does not know its own egress address, and teams routinely paste the address of the wrong gateway into a change request. GitHub is telling you, in the error, which address the decision was made about. Compare that against the CIDRs you believe you egress from before you ask anybody to add anything.

It is a different 403 from the other four, and it says so. The section's User-Agent note owns the sort of a 403 into its causes from the response body, and this is a fifth entry in that sort rather than a second copy of it. Primary quota exhaustion says API rate limit exceeded and comes with x-ratelimit-remaining: 0. A secondary limit says secondary rate limit and usually carries retry-after. The User-Agent rule names the header. A missing permission names the resource and nothing else. This one, alone, contains a dotted quad or a colonned IPv6 address, which is a signal you can key on without reading English.

Which list judges you depends on what kind of credential it is. A GitHub App can maintain its own allow list, and where an organization has enabled the setting, the App's ranges are contributed automatically — but that covers installation tokens. A user-to-server token acts for a person, so it is judged against the organization's own list even when the App's ranges are allowed. That distinction accounts for the confusing case where an App's background sync works and the same App's interactive calls do not.

The rule is org state and needs org access to read. A repository-scoped read-only token sees the effect, never the rule. This is a real blind spot and the script reports it as one: without admin:org-class access, the entries are unreadable and the finding rests on the refusal body alone, which is enough to name the cause and not enough to say which entry would have covered you. Say that plainly rather than printing a confident diff of an empty list.

The fix, as a flow

The whole diagnosis is one probe read twice over: once for its status and once for its body. A refusal from this API can be four things, and exactly one of them contains an IP address, so the script sorts on the presence of an address rather than on a sentence GitHub can reword. That address is also the finding, because it is the one GitHub applied the rule to, which is better evidence than any echo service about what this job really egresses through. The list itself is read only where the token is allowed to read it, and where it is not, the script says the rule is unreadable rather than implying it is empty.

One probe, body and headerssorted before anything is believedBody names an addressthe source address, not the tokenEntry exists but is offswitch it on, do not add a secondQuota or secondary limita different 403 with its own repairCovered and still refusedlook at which org the call named
Four causes, one status code. Only the top row names an address, which is what makes the sort survive a reworded sentence.

How to fix it

Run it from the machine that is failing

This is the one diagnostic in the section where the location of the process is part of the input. Run the script on the runner, in the function, inside the container — wherever the 403 happens. Run it on a laptop and it will cheerfully report that everything is fine, which is exactly the misleading result that sent you here.

Sort the 403 before believing anything about it

The script probes one organization-scoped path and classifies the refusal from the body and the headers: quota, secondary limit, the User-Agent rule, a permission, or the allow list. Only the last of those contains an IP address, so the classification does not depend on matching a sentence GitHub may reword.

Take the address out of GitHub's own error

When the classification is the allow list, the script lifts the address out of the message and prints it as the address GitHub saw. Pass --egress 203.0.113.0/24 with the ranges you believe your jobs leave from and it does the containment arithmetic locally: if the observed address is outside every range you declared, your egress assumption is wrong and adding those ranges to the allow list would not have helped.

Read the list itself, if your token is allowed to

With --org-allow-list and an admin:org-class token, the script runs one GraphQL query for ipAllowListEnabledSetting and the entries, and reports whether the observed address is covered, covered by an entry that is switched off, or not covered at all. The query is refused before it is sent if the document contains a mutation. Without that access the script says the list is unreadable rather than implying it is empty.

Pair it with the reading from the machine that works

A script cannot be in two places at once, so pass the other machine's status with --seen-elsewhere 200. A 403 here beside a 200 there, with the same token, is the network path stated as a fact rather than a hunch, and it is the sentence to put in the change request. The repair the script prints is a request for somebody with organization admin to add a range, or for the job to be moved behind a fixed-address gateway that is already allowed. It adds nothing itself.

How to check it worked

Once the range is added, the same probe from the same runner stops being refused and the observed address is reported as covered.

GITHUB_TOKEN=$READ_ONLY_TOKEN python3 github_ip_allow_list.py acme \
    --egress 198.51.100.0/24 --seen-elsewhere 200
# read cost: 2 REST request(s) against the core hourly quota, 0 GraphQL point(s)
# probe: GET /orgs/acme/repos?per_page=1 -> HTTP 403
# refusal: ip-allow-list — the body names an IP address, which no other 403 on
#   this API does. This is a check on the source address, not on the token.
# address GitHub saw: 203.0.113.9
# egress-assumption-wrong: the address GitHub saw is outside every range you
#   declared (198.51.100.0/24), so adding those ranges would not have helped.
# paired reading: network-path — refused here, 200 elsewhere, same token.
# repair: ask an owner of acme to add 203.0.113.9/32, or the runner pool's
#   documented egress range, to the organization IP allow list. Nothing here
#   adds it: that is an organization setting and this script only reads.

The full code

The live part is two GETs and, only if you ask for it, one GraphQL query. Everything that decides anything is pure: the classifier that sorts a 403 by what its body contains, the scan that lifts an address out of a sentence without a regular expression to get wrong, the CIDR arithmetic, and the pairing of two readings taken from two machines. That matters here more than usual, because the interesting inputs are refusals you cannot conveniently reproduce — you would need to be on the blocked runner to produce one, and the tests need to hold a dozen of them at once.

Get this script on GitHub Follow @allanninal Python and Node.js, with tests. Read only, it never writes. One of 101 GitHub API fixes, free and open source.
github_ip_allow_list.py
"""Say whether a 403 came from an organization IP allow list.

Read only. Two GETs, plus one optional GraphQL query, which is a read that
happens to travel over the same verb a write would. Nothing is added to any
allow list: an allow-list entry is organization state, so the script compares
what is there against the address GitHub saw and prints the request for
somebody with organization admin to make.

The point of the note: this refusal is a check on the source address, not on
the credential. The identical token succeeds from a laptop and is refused from
an ephemeral CI runner, which is why every experiment that varies the token
comes back clean.

What this can and cannot see: the refusal body names the address GitHub applied
the rule to, which is stronger evidence than any echo service. The list itself
is organization state and needs admin:org-class access; without it this script
reports the effect and says the rule is unreadable rather than pretending an
unreadable list is an empty one.

Environment:

    GITHUB_TOKEN    the same read-only token the failing job holds
"""
import argparse
import json
import logging
import os
import sys

import requests

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("github_ip_allow_list")

API = "https://api.github.com"
UA = "github-ip-allow-list/1.0"

# Sentences the four other causes of a 403 put in the body. Matched in
# lowercase and only as corroboration: the decisive test for an allow-list
# refusal is that the body contains an IP address, which none of the others do,
# so a reworded sentence cannot silently break the classification.
QUOTA_MARKERS = ("api rate limit exceeded",)
SECONDARY_MARKERS = ("secondary rate limit",)
USER_AGENT_MARKERS = ("user-agent", "user agent")
ALLOW_LIST_MARKERS = ("ip allow list", "not permitted to access this resource")

# Read for ipAllowListEnabledSetting and the entries. One query, one point, and
# it is refused before it is sent if the document stops being a read.
ALLOW_LIST_QUERY = """
query($login: String!) {
  organization(login: $login) {
    ipAllowListEnabledSetting
    ipAllowListForInstalledAppsEnabledSetting
    ipAllowListEntries(first: 100) {
      nodes { allowListValue isActive name }
    }
  }
}
"""

# Which list judges which credential. An App-managed allow list contributes the
# App's ranges for installation tokens; a user-to-server token acts for a
# person and is judged against the organization's own list regardless.
TOKEN_PREFIXES = (
    ("github_pat_", "fine-grained PAT"),
    ("ghp_", "classic PAT"),
    ("gho_", "OAuth user token"),
    ("ghu_", "App user-to-server token"),
    ("ghs_", "App installation token"),
    ("ghr_", "App refresh token"),
)

APP_MANAGED_APPLIES = ("App installation token",)


def read_cost(with_allow_list=False):
    """(REST requests, GraphQL points) this run will spend. Pure.

    Printed before anything is spent. The GraphQL half is counted separately
    because it comes out of a different budget and a reader deciding whether to
    pass --org-allow-list is deciding about that budget, not this one.
    """
    return (2, 1 if with_allow_list else 0)


def token_kind(token):
    """Name the credential from its prefix. Pure; nothing leaves the machine."""
    value = (token or "").strip()
    for prefix, name in TOKEN_PREFIXES:
        if value.startswith(prefix):
            return name
    return "unknown"


def list_that_applies(kind):
    """Which allow list judges this credential. Pure. (which, detail).

    The case that confuses people: an App with its own allow list keeps its
    background sync working while the same App's interactive calls are refused,
    because those calls carry a user-to-server token and are judged against the
    organization's list.
    """
    if kind in APP_MANAGED_APPLIES:
        return ("org-list-plus-app-managed",
                "an installation token is judged against the organization's "
                "list, and where the organization has enabled the App-managed "
                "setting the App's own ranges are contributed to it "
                "automatically.")
    if kind == "App user-to-server token":
        return ("org-list-only",
                "a user-to-server token acts for a person, so it is judged "
                "against the organization's own list even when the App's "
                "ranges are allowed. An App whose background sync works and "
                "whose interactive calls do not is this exact case.")
    return ("org-list-only",
            "this credential carries no App identity, so only the "
            "organization's own allow list applies to it.")


def looks_like_ipv4(text):
    """Four dot-separated numbers in 0..255. Pure. No regular expression.

    Written as arithmetic rather than a pattern because the input is a
    human-readable sentence and the parts that matter -- a trailing full stop,
    a leading bracket -- are easier to strip than to describe in a pattern.
    """
    parts = str(text or "").split(".")
    if len(parts) != 4:
        return False
    for part in parts:
        if not part.isdigit() or len(part) > 3:
            return False
        if int(part) > 255:
            return False
    return True


def looks_like_ipv6(text):
    """A rough IPv6 test: colons, hex groups only. Pure.

    Rough on purpose. The script never does arithmetic on an IPv6 address; it
    only needs to recognise one well enough to report it and to say that the
    containment check was not run.
    """
    value = str(text or "")
    if value.count(":") < 2:
        return False
    for group in value.split(":"):
        if group == "":
            continue
        if len(group) > 4:
            return False
        for ch in group.lower():
            if ch not in "0123456789abcdef":
                return False
    return True


def address_in_message(message):
    """The address GitHub says it saw, or None. Pure.

    Tokenised rather than matched, so punctuation attached to the address --
    the full stop that ends the sentence, a comma, a closing bracket -- does
    not have to be anticipated in a pattern.
    """
    for raw in str(message or "").split():
        candidate = raw.strip(".,;:()[]<>\"'")
        if looks_like_ipv4(candidate) or looks_like_ipv6(candidate):
            return candidate
    return None


def header_value(headers, name):
    """Case-insensitive header read against a plain dict. Pure."""
    if not isinstance(headers, dict):
        return None
    wanted = str(name).lower()
    for key, value in headers.items():
        if str(key).lower() == wanted:
            return value
    return None


def classify_refusal(status, body_text, headers=None):
    """Sort one refusal into its cause. Pure. (state, detail).

    The decisive test for this note's cause is structural: the body contains an
    IP address. Every other 403 on this API describes a rule and names no
    address, so the classification survives GitHub rewording the sentence.
    """
    text = str(body_text or "").lower()
    if int(status or 0) not in (401, 403, 429):
        return ("not-a-refusal",
                "HTTP %s is not a refusal, so there is nothing here to sort."
                % status)
    if any(m in text for m in SECONDARY_MARKERS):
        return ("secondary-limit",
                "the body names a secondary rate limit. Wait for retry-after "
                "and slow down; no allow-list entry is involved.")
    if any(m in text for m in QUOTA_MARKERS) or header_value(
            headers, "x-ratelimit-remaining") in ("0", 0):
        return ("primary-quota-exhausted",
                "primary quota is spent. x-ratelimit-reset says when it "
                "returns, and the address is not the problem.")
    if address_in_message(body_text) is not None:
        return ("ip-allow-list",
                "the body names an IP address, which no other 403 on this API "
                "does. This is a check on where the request came from, not on "
                "what it carried.")
    if any(m in text for m in ALLOW_LIST_MARKERS):
        return ("ip-allow-list-unaddressed",
                "the body reads like an allow-list refusal but names no "
                "address. Treat the cause as the allow list and get the "
                "egress address another way.")
    if any(m in text for m in USER_AGENT_MARKERS):
        return ("user-agent-rule",
                "the body names the User-Agent header. That check runs before "
                "authentication and has its own note.")
    if int(status) == 401:
        return ("credential-rejected",
                "401 means the credential itself was not accepted. An allow "
                "list refuses with 403 and a body that names an address.")
    return ("permission-or-role",
            "no rule named itself in the body, which is what a missing "
            "permission or too low a repository role looks like.")


def ipv4_to_int(text):
    """Dotted quad to an integer, or None. Pure."""
    if not looks_like_ipv4(text):
        return None
    total = 0
    for part in str(text).split("."):
        total = (total << 8) + int(part)
    return total


def cidr_contains(cidr, address):
    """Is this address inside this CIDR. Pure. True, False or None.

    None means "not evaluated" -- an IPv6 entry, or something this script does
    not parse -- and it is deliberately not False. Reporting an unevaluated
    entry as a miss is how a script tells somebody their address is not covered
    when the entry covering it was simply one it could not read.
    """
    value = str(cidr or "").strip()
    if not value:
        return None
    if "/" in value:
        net, _, bits = value.partition("/")
        if not bits.isdigit():
            return None
        prefix = int(bits)
    else:
        net, prefix = value, 32
    if ":" in net or ":" in str(address or ""):
        return None
    left, right = ipv4_to_int(net), ipv4_to_int(address)
    if left is None or right is None or prefix > 32:
        return None
    if prefix == 0:
        return True
    mask = (0xFFFFFFFF << (32 - prefix)) & 0xFFFFFFFF
    return (left & mask) == (right & mask)


def covered_by(entries, address):
    """Does any active entry cover this address. Pure. (state, entry).

    Ordered so that an inactive entry that would have covered the address is
    reported as its own state. That is the most useful finding the list can
    produce: somebody already added the range and then switched it off, and a
    plain "not covered" would send you to add it a second time.
    """
    if address is None:
        return ("address-unknown", None)
    if not entries:
        return ("no-entries", None)
    inactive = None
    unevaluated = False
    for entry in entries:
        hit = cidr_contains(entry.get("value"), address)
        if hit is None:
            unevaluated = True
            continue
        if hit and entry.get("active"):
            return ("covered", entry)
        if hit:
            inactive = inactive or entry
    if inactive is not None:
        return ("covered-but-inactive", inactive)
    if unevaluated:
        return ("not-covered-some-unevaluated", None)
    return ("not-covered", None)


def egress_assumption(declared, address):
    """Compare the ranges you believe you leave from against the real one. Pure."""
    if address is None:
        return ("address-unknown",
                "no address was reported, so there is nothing to compare your "
                "declared egress against.")
    if not declared:
        return ("nothing-declared",
                "pass --egress with the ranges you believe this job leaves "
                "from and this becomes a check rather than a reading.")
    for cidr in declared:
        if cidr_contains(cidr, address) is True:
            return ("egress-as-expected",
                    "the address GitHub saw is inside %s, so your egress "
                    "assumption holds and the range simply is not allowed "
                    "yet." % cidr)
    return ("egress-assumption-wrong",
            "the address GitHub saw is outside every range you declared (%s), "
            "so adding those ranges would not have helped. Find out what this "
            "job really egresses through before asking for a change."
            % ", ".join(declared))


def paired_reading(status_here, status_elsewhere):
    """Two readings of the same call from two machines. Pure. (state, detail)."""
    here = int(status_here or 0)
    there = None if status_elsewhere is None else int(status_elsewhere)
    if there is None:
        return ("single-reading",
                "only this machine was read. Pass --seen-elsewhere with the "
                "status the same token gets from a machine that works and the "
                "network path stops being a hunch.")
    if here == 403 and there == 200:
        return ("network-path",
                "the same token is refused here and accepted there, so the "
                "difference is the source address and nothing else.")
    if here == 403 and there == 403:
        return ("refused-everywhere",
                "both addresses are refused. Either the allow list covers "
                "neither, or the cause is the credential after all.")
    if here == 200:
        return ("no-refusal",
                "this machine was not refused, so there is nothing to explain "
                "from here. Run this on the machine that fails.")
    return ("inconclusive",
            "the pair of statuses does not describe an allow list. Sort the "
            "refusal by its body first.")


def words(document):
    """Bare words in a GraphQL document. Pure. No regular expression."""
    out, current = [], ""
    for ch in str(document or ""):
        if ch.isalnum() or ch == "_":
            current += ch
        else:
            if current:
                out.append(current.lower())
            current = ""
    if current:
        out.append(current.lower())
    return out


def refuses_mutation(document):
    """Would this document change something. Pure.

    The GraphQL endpoint takes reads and writes over one verb, so the guard
    lives beside the sender rather than in a comment. This script sends one
    constant document; the guard exists so that editing the constant cannot
    quietly turn a read-only tool into a writing one.
    """
    banned = {"mutation", "subscription"}
    return bool(banned.intersection(words(document)))


def allow_list_from_graphql(body):
    """Normalise the GraphQL answer. Pure. (setting, apps_setting, entries, note)."""
    if not isinstance(body, dict):
        return (None, None, [], "no readable GraphQL body came back.")
    errors = body.get("errors") or []
    org = ((body.get("data") or {}).get("organization")) or {}
    if not org:
        detail = errors[0].get("message") if errors and isinstance(errors[0], dict) else ""
        return (None, None, [],
                "the organization block was not returned: %s. Reading an IP "
                "allow list needs admin:org-class access, so an unreadable "
                "list here means your token, not an empty list."
                % (detail or "no organization in the response"))
    nodes = ((org.get("ipAllowListEntries") or {}).get("nodes")) or []
    entries = []
    for node in nodes:
        if not isinstance(node, dict):
            continue
        entries.append({"value": node.get("allowListValue"),
                        "active": bool(node.get("isActive")),
                        "name": node.get("name") or ""})
    return (org.get("ipAllowListEnabledSetting"),
            org.get("ipAllowListForInstalledAppsEnabledSetting"),
            entries,
            "read %d entr(y/ies)." % len(entries))


def verdict(refusal_state, coverage_state, setting):
    """The finding, in one state. Pure. (state, detail)."""
    if refusal_state not in ("ip-allow-list", "ip-allow-list-unaddressed"):
        return (refusal_state,
                "this refusal is not an allow-list refusal, so the rest of "
                "this script is not about your problem.")
    if str(setting or "").upper() == "DISABLED":
        return ("allow-list-disabled",
                "the organization reports the allow list as disabled, which "
                "does not agree with the refusal. Check that you read the "
                "same organization the failing call was made against.")
    if coverage_state == "covered-but-inactive":
        return ("entry-exists-but-is-off",
                "an entry covering this address exists and is switched off. "
                "Somebody already did the work; it just is not active.")
    if coverage_state == "covered":
        return ("covered-yet-refused",
                "an active entry covers this address, so either the refusal "
                "predates the entry or the call was against a different "
                "organization. Re-run the probe before escalating.")
    if coverage_state in ("not-covered", "not-covered-some-unevaluated"):
        return ("address-not-covered",
                "no active entry covers the address GitHub saw. This is the "
                "ordinary case and the repair is one entry.")
    return ("rule-unreadable",
            "the refusal is an allow-list refusal and the list itself could "
            "not be read, which needs admin:org-class access. The cause is "
            "established; the entry that would have covered you is not.")


def repair(state, address, org):
    """The sentence a reader has to act on. Pure. Nothing here is executed."""
    if state == "entry-exists-but-is-off":
        return ("ask an owner of %s to switch the existing entry back on. "
                "Adding a second entry for the same range will not help while "
                "the first one is inactive." % org)
    if state in ("address-not-covered", "rule-unreadable"):
        return ("ask an owner of %s to add %s, or the documented egress range "
                "of this runner pool, to the organization IP allow list. For "
                "a GitHub App, enabling the App-managed allow list contributes "
                "its ranges for installation tokens. Nothing here adds "
                "anything." % (org, (address or "this job's egress range") +
                               ("/32" if address and "." in address else "")))
    if state == "covered-yet-refused":
        return ("re-run the probe from this machine. A covered address that "
                "is still refused usually means the reading and the refusal "
                "came from different places or different organizations.")
    if state == "allow-list-disabled":
        return ("confirm which organization the failing call names. A "
                "disabled list cannot produce this refusal.")
    return ("sort the refusal by its body before doing anything about "
            "addresses. This script found no allow-list refusal to repair.")


def get(session, path):
    """One GET. Returns the response object."""
    response = session.get(API + path, timeout=30)
    if response.status_code == 401:
        log.warning("401 from GitHub on %s: the credential itself was not "
                    "accepted, which is a different note", path)
    return response


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("org", help="the organization the calls are refused by")
    parser.add_argument("--path",
                        help="the org-scoped path to probe, default "
                             "/orgs/{org}/repos?per_page=1")
    parser.add_argument("--egress", action="append", default=[],
                        help="a CIDR you believe this job leaves from; repeatable")
    parser.add_argument("--seen-elsewhere", type=int,
                        help="the status the same token gets from a machine "
                             "that works")
    parser.add_argument("--org-allow-list", action="store_true",
                        help="read the list itself; needs admin:org-class access")
    args = parser.parse_args()

    token = os.environ.get("GITHUB_TOKEN")
    if not token:
        log.error("set GITHUB_TOKEN (the same read-only token the failing job holds)")
        return 2

    rest, points = read_cost(args.org_allow_list)
    log.info("read cost: %d REST request(s) against the core hourly quota, "
             "%d GraphQL point(s)", rest, points)

    session = requests.Session()
    session.headers.update({
        "Authorization": "Bearer " + token,
        "Accept": "application/vnd.github+json",
        "X-GitHub-Api-Version": "2022-11-28",
        # GitHub refuses requests with no User-Agent before it looks at auth.
        "User-Agent": UA,
    })

    kind = token_kind(token)
    which_list, which_detail = list_that_applies(kind)
    log.info("token: %s. %s: %s", kind, which_list, which_detail)

    path = args.path or ("/orgs/%s/repos?per_page=1" % args.org)
    probe = get(session, path)
    log.info("probe: GET %s -> HTTP %s", path, probe.status_code)
    body_text = probe.text or ""
    refusal_state, refusal_detail = classify_refusal(
        probe.status_code, body_text, dict(probe.headers))
    log.info("refusal: %s. %s", refusal_state, refusal_detail)

    address = address_in_message(body_text)
    if address:
        log.info("address GitHub saw: %s", address)

    egress_state, egress_detail = egress_assumption(args.egress, address)
    log.info("%s: %s", egress_state, egress_detail)

    setting, apps_setting, entries, note = None, None, [], "not read"
    if args.org_allow_list:
        if refuses_mutation(ALLOW_LIST_QUERY):
            log.error("the allow-list document is not a read; refusing to send it")
            return 2
        graph = session.post(API + "/graphql",
                             json={"query": ALLOW_LIST_QUERY,
                                   "variables": {"login": args.org}},
                             timeout=30)
        try:
            payload = graph.json()
        except ValueError:
            payload = None
        setting, apps_setting, entries, note = allow_list_from_graphql(payload)
        log.info("allow list: setting=%s, apps=%s, %s", setting, apps_setting, note)

    coverage_state, entry = covered_by(entries, address) if args.org_allow_list \
        else ("rule-unread", None)
    if args.org_allow_list:
        log.info("coverage: %s%s", coverage_state,
                 "" if entry is None else " by %r" % entry.get("value"))

    paired_state, paired_detail = paired_reading(probe.status_code, args.seen_elsewhere)
    log.info("paired reading: %s. %s", paired_state, paired_detail)

    state, detail = verdict(refusal_state, coverage_state, setting)
    log.info("%s: %s", state, detail)
    log.info("repair: %s", repair(state, address, args.org))

    print(json.dumps({
        "organization": args.org,
        "probe_path": path,
        "probe_status": probe.status_code,
        "token_kind": kind,
        "list_that_applies": which_list,
        "refusal_state": refusal_state,
        "address_github_saw": address,
        "declared_egress": args.egress,
        "egress_state": egress_state,
        "allow_list_setting": setting,
        "app_managed_setting": apps_setting,
        "entries_read": len(entries),
        "coverage_state": coverage_state,
        "paired_state": paired_state,
        "state": state,
        "detail": detail,
        "repair": repair(state, address, args.org),
    }, indent=2, default=str))
    return 1 if state in ("address-not-covered", "entry-exists-but-is-off",
                          "rule-unreadable") else 0


if __name__ == "__main__":
    sys.exit(main())
github-ip-allow-list.mjs
/**
 * Say whether a 403 came from an organization IP allow list.
 *
 * Read only. Two GETs, plus one optional GraphQL query, which is a read that
 * happens to travel over the same verb a write would. Nothing is added to any
 * allow list: the script compares what is there against the address GitHub
 * saw and prints the request for an organization owner to make.
 *
 * The refusal is a check on the source address, not on the credential, which
 * is why the identical token succeeds from a laptop and fails on a runner.
 *
 * Environment:
 *   GITHUB_TOKEN       the same read-only token the failing job holds
 *   GITHUB_ORG         the organization the calls are refused by
 *   GITHUB_PROBE_PATH  optional org-scoped path to probe
 *   GITHUB_EGRESS      optional comma-separated CIDRs you believe you use
 *   GITHUB_ELSEWHERE   optional status the same token gets from a good machine
 *   GITHUB_READ_LIST   set to 1 to read the list itself (needs admin:org)
 */
const API = 'https://api.github.com';
const UA = 'github-ip-allow-list/1.0';

/** Sentences the other causes of a 403 put in the body. Corroboration only. */
export const QUOTA_MARKERS = ['api rate limit exceeded'];
export const SECONDARY_MARKERS = ['secondary rate limit'];
export const USER_AGENT_MARKERS = ['user-agent', 'user agent'];
export const ALLOW_LIST_MARKERS = ['ip allow list', 'not permitted to access this resource'];

/** One query, one point, refused before it is sent if it stops being a read. */
export const ALLOW_LIST_QUERY = `
query($login: String!) {
  organization(login: $login) {
    ipAllowListEnabledSetting
    ipAllowListForInstalledAppsEnabledSetting
    ipAllowListEntries(first: 100) {
      nodes { allowListValue isActive name }
    }
  }
}
`;

export const TOKEN_PREFIXES = [
  ['github_pat_', 'fine-grained PAT'],
  ['ghp_', 'classic PAT'],
  ['gho_', 'OAuth user token'],
  ['ghu_', 'App user-to-server token'],
  ['ghs_', 'App installation token'],
  ['ghr_', 'App refresh token'],
];

export const APP_MANAGED_APPLIES = ['App installation token'];

/** [REST requests, GraphQL points] this run will spend. Pure. */
export function readCost(withAllowList = false) {
  return [2, withAllowList ? 1 : 0];
}

/** Name the credential from its prefix. Pure. */
export function tokenKind(token) {
  const value = String(token ?? '').trim();
  for (const [prefix, name] of TOKEN_PREFIXES) {
    if (value.startsWith(prefix)) return name;
  }
  return 'unknown';
}

/** Which allow list judges this credential. Pure. [which, detail]. */
export function listThatApplies(kind) {
  if (APP_MANAGED_APPLIES.includes(kind)) {
    return ['org-list-plus-app-managed', "an installation token is judged "
      + "against the organization's list, and where the organization has "
      + "enabled the App-managed setting the App's own ranges are contributed "
      + 'to it automatically.'];
  }
  if (kind === 'App user-to-server token') {
    return ['org-list-only', 'a user-to-server token acts for a person, so it '
      + "is judged against the organization's own list even when the App's "
      + 'ranges are allowed. An App whose background sync works and whose '
      + 'interactive calls do not is this exact case.'];
  }
  return ['org-list-only', 'this credential carries no App identity, so only '
    + "the organization's own allow list applies to it."];
}

/** Four dot-separated numbers in 0..255. Pure. No regular expression. */
export function looksLikeIpv4(text) {
  const parts = String(text ?? '').split('.');
  if (parts.length !== 4) return false;
  for (const part of parts) {
    if (part.length === 0 || part.length > 3) return false;
    for (const ch of part) if (ch < '0' || ch > '9') return false;
    if (Number(part) > 255) return false;
  }
  return true;
}

/** A rough IPv6 test. The script never does arithmetic on one. Pure. */
export function looksLikeIpv6(text) {
  const value = String(text ?? '');
  if ((value.match(/:/g) || []).length < 2) return false;
  for (const group of value.split(':')) {
    if (group === '') continue;
    if (group.length > 4) return false;
    for (const ch of group.toLowerCase()) {
      if (!'0123456789abcdef'.includes(ch)) return false;
    }
  }
  return true;
}

/** The address GitHub says it saw, or null. Pure. Tokenised, not matched. */
export function addressInMessage(message) {
  for (const raw of String(message ?? '').split(/\s+/)) {
    let candidate = raw;
    while (candidate.length && '.,;:()[]<>"\''.includes(candidate[candidate.length - 1])) {
      candidate = candidate.slice(0, -1);
    }
    while (candidate.length && '.,;:()[]<>"\''.includes(candidate[0])) {
      candidate = candidate.slice(1);
    }
    if (looksLikeIpv4(candidate) || looksLikeIpv6(candidate)) return candidate;
  }
  return null;
}

/** Case-insensitive header read against a plain object. Pure. */
export function headerValue(headers, name) {
  if (!headers || typeof headers !== 'object') return null;
  const wanted = String(name).toLowerCase();
  for (const key of Object.keys(headers)) {
    if (key.toLowerCase() === wanted) return headers[key];
  }
  return null;
}

/** Sort one refusal into its cause. Pure. [state, detail]. */
export function classifyRefusal(status, bodyText, headers = null) {
  const text = String(bodyText ?? '').toLowerCase();
  const code = Number(status) || 0;
  if (![401, 403, 429].includes(code)) {
    return ['not-a-refusal', `HTTP ${status} is not a refusal, so there is `
      + 'nothing here to sort.'];
  }
  if (SECONDARY_MARKERS.some((m) => text.includes(m))) {
    return ['secondary-limit', 'the body names a secondary rate limit. Wait '
      + 'for retry-after and slow down; no allow-list entry is involved.'];
  }
  const remaining = headerValue(headers, 'x-ratelimit-remaining');
  if (QUOTA_MARKERS.some((m) => text.includes(m)) || String(remaining) === '0') {
    return ['primary-quota-exhausted', 'primary quota is spent. '
      + 'x-ratelimit-reset says when it returns, and the address is not the '
      + 'problem.'];
  }
  if (addressInMessage(bodyText) !== null) {
    return ['ip-allow-list', 'the body names an IP address, which no other 403 '
      + 'on this API does. This is a check on where the request came from, not '
      + 'on what it carried.'];
  }
  if (ALLOW_LIST_MARKERS.some((m) => text.includes(m))) {
    return ['ip-allow-list-unaddressed', 'the body reads like an allow-list '
      + 'refusal but names no address. Treat the cause as the allow list and '
      + 'get the egress address another way.'];
  }
  if (USER_AGENT_MARKERS.some((m) => text.includes(m))) {
    return ['user-agent-rule', 'the body names the User-Agent header. That '
      + 'check runs before authentication and has its own note.'];
  }
  if (code === 401) {
    return ['credential-rejected', '401 means the credential itself was not '
      + 'accepted. An allow list refuses with 403 and a body that names an '
      + 'address.'];
  }
  return ['permission-or-role', 'no rule named itself in the body, which is '
    + 'what a missing permission or too low a repository role looks like.'];
}

/** Dotted quad to a number, or null. Pure. */
export function ipv4ToInt(text) {
  if (!looksLikeIpv4(text)) return null;
  let total = 0;
  for (const part of String(text).split('.')) total = total * 256 + Number(part);
  return total;
}

/** Is this address inside this CIDR. Pure. true, false or null for unevaluated. */
export function cidrContains(cidr, address) {
  const value = String(cidr ?? '').trim();
  if (!value) return null;
  let net = value;
  let prefix = 32;
  if (value.includes('/')) {
    const cut = value.indexOf('/');
    net = value.slice(0, cut);
    const bits = value.slice(cut + 1);
    if (!bits.length || [...bits].some((c) => c < '0' || c > '9')) return null;
    prefix = Number(bits);
  }
  if (net.includes(':') || String(address ?? '').includes(':')) return null;
  const left = ipv4ToInt(net);
  const right = ipv4ToInt(address);
  if (left === null || right === null || prefix > 32) return null;
  if (prefix === 0) return true;
  const mask = prefix === 32 ? 0xFFFFFFFF : (0xFFFFFFFF - (2 ** (32 - prefix) - 1));
  return (left & mask) >>> 0 === (right & mask) >>> 0;
}

/** Does any active entry cover this address. Pure. [state, entry]. */
export function coveredBy(entries, address) {
  if (address === null || address === undefined) return ['address-unknown', null];
  if (!entries || entries.length === 0) return ['no-entries', null];
  let inactive = null;
  let unevaluated = false;
  for (const entry of entries) {
    const hit = cidrContains(entry.value, address);
    if (hit === null) { unevaluated = true; continue; }
    if (hit && entry.active) return ['covered', entry];
    if (hit && !inactive) inactive = entry;
  }
  if (inactive) return ['covered-but-inactive', inactive];
  if (unevaluated) return ['not-covered-some-unevaluated', null];
  return ['not-covered', null];
}

/** Declared egress against the address GitHub really saw. Pure. */
export function egressAssumption(declared, address) {
  if (address === null || address === undefined) {
    return ['address-unknown', 'no address was reported, so there is nothing '
      + 'to compare your declared egress against.'];
  }
  if (!declared || declared.length === 0) {
    return ['nothing-declared', 'declare the ranges you believe this job '
      + 'leaves from and this becomes a check rather than a reading.'];
  }
  for (const cidr of declared) {
    if (cidrContains(cidr, address) === true) {
      return ['egress-as-expected', `the address GitHub saw is inside ${cidr}, `
        + 'so your egress assumption holds and the range simply is not allowed '
        + 'yet.'];
    }
  }
  return ['egress-assumption-wrong', 'the address GitHub saw is outside every '
    + `range you declared (${declared.join(', ')}), so adding those ranges `
    + 'would not have helped. Find out what this job really egresses through '
    + 'before asking for a change.'];
}

/** Two readings of the same call from two machines. Pure. [state, detail]. */
export function pairedReading(statusHere, statusElsewhere) {
  const here = Number(statusHere) || 0;
  const there = (statusElsewhere === null || statusElsewhere === undefined)
    ? null : Number(statusElsewhere);
  if (there === null) {
    return ['single-reading', 'only this machine was read. Supply the status '
      + 'the same token gets from a machine that works and the network path '
      + 'stops being a hunch.'];
  }
  if (here === 403 && there === 200) {
    return ['network-path', 'the same token is refused here and accepted '
      + 'there, so the difference is the source address and nothing else.'];
  }
  if (here === 403 && there === 403) {
    return ['refused-everywhere', 'both addresses are refused. Either the '
      + 'allow list covers neither, or the cause is the credential after all.'];
  }
  if (here === 200) {
    return ['no-refusal', 'this machine was not refused, so there is nothing '
      + 'to explain from here. Run this on the machine that fails.'];
  }
  return ['inconclusive', 'the pair of statuses does not describe an allow '
    + 'list. Sort the refusal by its body first.'];
}

/** Bare words in a GraphQL document. Pure. */
export function words(document) {
  const out = [];
  let current = '';
  for (const ch of String(document ?? '')) {
    if (/[A-Za-z0-9_]/.test(ch)) current += ch;
    else { if (current) out.push(current.toLowerCase()); current = ''; }
  }
  if (current) out.push(current.toLowerCase());
  return out;
}

/** Would this document change something. Pure. */
export function refusesMutation(document) {
  const banned = ['mutation', 'subscription'];
  return words(document).some((w) => banned.includes(w));
}

/** Normalise the GraphQL answer. Pure. [setting, appsSetting, entries, note]. */
export function allowListFromGraphql(body) {
  if (!body || typeof body !== 'object') {
    return [null, null, [], 'no readable GraphQL body came back.'];
  }
  const errors = body.errors || [];
  const org = (body.data && body.data.organization) || null;
  if (!org) {
    const detail = errors.length && errors[0] && errors[0].message
      ? errors[0].message : 'no organization in the response';
    return [null, null, [], `the organization block was not returned: ${detail}. `
      + 'Reading an IP allow list needs admin:org-class access, so an '
      + 'unreadable list here means your token, not an empty list.'];
  }
  const nodes = (org.ipAllowListEntries && org.ipAllowListEntries.nodes) || [];
  const entries = nodes.filter((n) => n && typeof n === 'object').map((n) => ({
    value: n.allowListValue,
    active: Boolean(n.isActive),
    name: n.name || '',
  }));
  return [org.ipAllowListEnabledSetting,
    org.ipAllowListForInstalledAppsEnabledSetting,
    entries,
    `read ${entries.length} entries.`];
}

/** The finding, in one state. Pure. [state, detail]. */
export function verdict(refusalState, coverageState, setting) {
  if (!['ip-allow-list', 'ip-allow-list-unaddressed'].includes(refusalState)) {
    return [refusalState, 'this refusal is not an allow-list refusal, so the '
      + 'rest of this script is not about your problem.'];
  }
  if (String(setting ?? '').toUpperCase() === 'DISABLED') {
    return ['allow-list-disabled', 'the organization reports the allow list as '
      + 'disabled, which does not agree with the refusal. Check that you read '
      + 'the same organization the failing call was made against.'];
  }
  if (coverageState === 'covered-but-inactive') {
    return ['entry-exists-but-is-off', 'an entry covering this address exists '
      + 'and is switched off. Somebody already did the work; it just is not '
      + 'active.'];
  }
  if (coverageState === 'covered') {
    return ['covered-yet-refused', 'an active entry covers this address, so '
      + 'either the refusal predates the entry or the call was against a '
      + 'different organization. Re-run the probe before escalating.'];
  }
  if (['not-covered', 'not-covered-some-unevaluated'].includes(coverageState)) {
    return ['address-not-covered', 'no active entry covers the address GitHub '
      + 'saw. This is the ordinary case and the repair is one entry.'];
  }
  return ['rule-unreadable', 'the refusal is an allow-list refusal and the list '
    + 'itself could not be read, which needs admin:org-class access. The cause '
    + 'is established; the entry that would have covered you is not.'];
}

/** The sentence a reader has to act on. Pure. Nothing here is executed. */
export function repair(state, address, org) {
  if (state === 'entry-exists-but-is-off') {
    return `ask an owner of ${org} to switch the existing entry back on. Adding `
      + 'a second entry for the same range will not help while the first one is '
      + 'inactive.';
  }
  if (['address-not-covered', 'rule-unreadable'].includes(state)) {
    const what = address ? `${address}${address.includes('.') ? '/32' : ''}`
      : "this job's egress range";
    return `ask an owner of ${org} to add ${what}, or the documented egress `
      + 'range of this runner pool, to the organization IP allow list. For a '
      + 'GitHub App, enabling the App-managed allow list contributes its ranges '
      + 'for installation tokens. Nothing here adds anything.';
  }
  if (state === 'covered-yet-refused') {
    return 're-run the probe from this machine. A covered address that is still '
      + 'refused usually means the reading and the refusal came from different '
      + 'places or different organizations.';
  }
  if (state === 'allow-list-disabled') {
    return 'confirm which organization the failing call names. A disabled list '
      + 'cannot produce this refusal.';
  }
  return 'sort the refusal by its body before doing anything about addresses. '
    + 'This script found no allow-list refusal to repair.';
}

function headers(token) {
  return {
    Authorization: `Bearer ${token}`,
    Accept: 'application/vnd.github+json',
    'X-GitHub-Api-Version': '2022-11-28',
    'Content-Type': 'application/json',
    'User-Agent': UA,
  };
}

async function main() {
  const token = process.env.GITHUB_TOKEN;
  const org = process.env.GITHUB_ORG;
  if (!token || !org) {
    console.error('set GITHUB_TOKEN (read-only is enough) and GITHUB_ORG');
    process.exitCode = 2;
    return;
  }
  const readList = process.env.GITHUB_READ_LIST === '1';
  const declared = (process.env.GITHUB_EGRESS || '').split(',')
    .map((s) => s.trim()).filter(Boolean);
  const elsewhere = process.env.GITHUB_ELSEWHERE
    ? Number(process.env.GITHUB_ELSEWHERE) : null;
  const [rest, points] = readCost(readList);
  console.log(`read cost: ${rest} REST request(s) against the core hourly quota, `
    + `${points} GraphQL point(s)`);

  const kind = tokenKind(token);
  const [whichList, whichDetail] = listThatApplies(kind);
  console.log(`token: ${kind}. ${whichList}: ${whichDetail}`);

  const path = process.env.GITHUB_PROBE_PATH || `/orgs/${org}/repos?per_page=1`;
  const probe = await fetch(`${API}${path}`, { headers: headers(token) });
  const bodyText = await probe.text();
  console.log(`probe: GET ${path} -> HTTP ${probe.status}`);

  const headerBag = {};
  probe.headers.forEach((value, key) => { headerBag[key] = value; });
  const [refusalState, refusalDetail] = classifyRefusal(probe.status, bodyText, headerBag);
  console.log(`refusal: ${refusalState}. ${refusalDetail}`);

  const address = addressInMessage(bodyText);
  if (address) console.log(`address GitHub saw: ${address}`);
  const [egressState, egressDetail] = egressAssumption(declared, address);
  console.log(`${egressState}: ${egressDetail}`);

  let setting = null;
  let appsSetting = null;
  let entries = [];
  let coverageState = 'rule-unread';
  if (readList) {
    if (refusesMutation(ALLOW_LIST_QUERY)) {
      console.error('the allow-list document is not a read; refusing to send it');
      process.exitCode = 2;
      return;
    }
    const graph = await fetch(`${API}/graphql`, {
      // A GraphQL query is a read. This is only how the document travels, and
      // refusesMutation() has already rejected anything that is not a read.
      method: 'POST',
      headers: headers(token),
      body: JSON.stringify({ query: ALLOW_LIST_QUERY, variables: { login: org } }),
    });
    let payload = null;
    try { payload = await graph.json(); } catch { payload = null; }
    [setting, appsSetting, entries] = allowListFromGraphql(payload);
    [coverageState] = coveredBy(entries, address);
    console.log(`allow list: setting=${setting}, apps=${appsSetting}, `
      + `entries=${entries.length}, coverage=${coverageState}`);
  }

  const [pairedState, pairedDetail] = pairedReading(probe.status, elsewhere);
  console.log(`paired reading: ${pairedState}. ${pairedDetail}`);
  const [state, detail] = verdict(refusalState, coverageState, setting);
  console.log(`${state}: ${detail}`);
  console.log(`repair: ${repair(state, address, org)}`);

  console.log(JSON.stringify({
    organization: org,
    probe_path: path,
    probe_status: probe.status,
    token_kind: kind,
    list_that_applies: whichList,
    refusal_state: refusalState,
    address_github_saw: address,
    declared_egress: declared,
    egress_state: egressState,
    allow_list_setting: setting,
    entries_read: entries.length,
    coverage_state: coverageState,
    paired_state: pairedState,
    state,
    detail,
    repair: repair(state, address, org),
  }, null, 2));
  process.exitCode = ['address-not-covered', 'entry-exists-but-is-off',
    'rule-unreadable'].includes(state) ? 1 : 0;
}

// Guarded so importing this file from the test runner does not start a run.
if (import.meta.url === `file://${process.argv[1]}`) {
  main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}

Add a test

The suite is mostly recorded refusals, because the refusal you need to handle is the one you cannot conveniently produce: you would have to be on the blocked runner to get it. So the bodies are held as fixtures and the classifier is asserted against all five causes, including the one where the sentence has been reworded and the classification survives because it keys on the presence of an address rather than on the English. Then the address scan against a sentence with a full stop attached to the address, the CIDR arithmetic including the boundary cases people get wrong, and the deliberate three-valued result where an unevaluated entry is not reported as a miss. The last group is the note's headline: refused here, 200 there, same token.

test_github_ip_allow_list.py
from github_ip_allow_list import (
    ALLOW_LIST_QUERY, address_in_message, allow_list_from_graphql,
    cidr_contains, classify_refusal, covered_by, egress_assumption,
    ipv4_to_int, list_that_applies, looks_like_ipv4, looks_like_ipv6,
    paired_reading, read_cost, refuses_mutation, repair, token_kind, verdict,
)

ALLOW_LIST_BODY = (
    "Although you appear to have the correct authorization credentials, the "
    "ACME organization has an IP allow list enabled, and 203.0.113.9 is not "
    "permitted to access this resource."
)
UA_BODY = ("Request forbidden by administrative rules. Please make sure your "
           "request has a User-Agent header.")
QUOTA_BODY = "API rate limit exceeded for user ID 12345."
SECONDARY_BODY = "You have exceeded a secondary rate limit."
PERMISSION_BODY = "Resource not accessible by integration"

ENTRIES = [
    {"value": "198.51.100.0/24", "active": True, "name": "office"},
    {"value": "203.0.113.0/24", "active": False, "name": "old ci"},
    {"value": "2001:db8::/32", "active": True, "name": "ipv6 office"},
]


def test_the_allow_list_refusal_is_the_only_one_naming_an_address():
    state, detail = classify_refusal(403, ALLOW_LIST_BODY, {})
    assert state == "ip-allow-list"
    assert "names an IP address" in detail
    assert classify_refusal(403, UA_BODY, {})[0] == "user-agent-rule"
    assert classify_refusal(403, PERMISSION_BODY, {})[0] == "permission-or-role"


def test_quota_and_secondary_limits_are_sorted_out_first():
    assert classify_refusal(403, QUOTA_BODY, {})[0] == "primary-quota-exhausted"
    assert classify_refusal(429, SECONDARY_BODY, {})[0] == "secondary-limit"
    # The header is enough on its own, because a proxy can replace the body.
    assert classify_refusal(403, "", {"X-RateLimit-Remaining": "0"})[0] == (
        "primary-quota-exhausted")


def test_a_reworded_allow_list_message_still_classifies():
    # The English is corroboration. The address is the signal, so GitHub can
    # rewrite the sentence without this becoming a permission problem.
    reworded = "Access from 198.51.100.77 is blocked by policy for this org."
    assert classify_refusal(403, reworded, {})[0] == "ip-allow-list"


def test_an_allow_list_message_with_no_address_is_kept_apart():
    state, _ = classify_refusal(403, "This org has an IP allow list enabled.", {})
    assert state == "ip-allow-list-unaddressed"


def test_a_200_is_not_a_refusal_to_sort():
    assert classify_refusal(200, "[]", {})[0] == "not-a-refusal"
    assert classify_refusal(401, "Bad credentials", {})[0] == "credential-rejected"


def test_the_address_survives_the_full_stop_at_the_end_of_the_sentence():
    assert address_in_message(ALLOW_LIST_BODY) == "203.0.113.9"
    assert address_in_message("from (2001:db8::1) today") == "2001:db8::1"
    assert address_in_message("no address at all here") is None
    # A version number is four groups but not four bytes.
    assert address_in_message("version 1.2.3.400 shipped") is None


def test_what_an_address_looks_like():
    assert looks_like_ipv4("203.0.113.9") is True
    assert looks_like_ipv4("203.0.113") is False
    assert looks_like_ipv4("203.0.113.256") is False
    assert looks_like_ipv6("2001:db8::1") is True
    assert looks_like_ipv6("203.0.113.9") is False


def test_cidr_arithmetic_at_the_edges():
    assert cidr_contains("203.0.113.0/24", "203.0.113.9") is True
    assert cidr_contains("203.0.113.0/24", "203.0.114.9") is False
    assert cidr_contains("203.0.113.9", "203.0.113.9") is True
    assert cidr_contains("0.0.0.0/0", "8.8.8.8") is True
    assert ipv4_to_int("0.0.0.1") == 1


def test_an_unevaluated_entry_is_none_and_not_false():
    # Reporting an IPv6 entry as a miss would tell somebody their address is
    # uncovered when the entry covering it was one this script cannot read.
    assert cidr_contains("2001:db8::/32", "203.0.113.9") is None
    assert cidr_contains("not-a-cidr", "203.0.113.9") is None
    assert cidr_contains("203.0.113.0/xx", "203.0.113.9") is None


def test_an_entry_that_exists_but_is_switched_off_is_its_own_finding():
    state, entry = covered_by(ENTRIES, "203.0.113.9")
    assert state == "covered-but-inactive"
    assert entry["name"] == "old ci"
    assert verdict("ip-allow-list", state, "ENABLED")[0] == "entry-exists-but-is-off"


def test_coverage_reports_the_entries_it_could_not_evaluate():
    state, entry = covered_by(ENTRIES, "192.0.2.5")
    assert state == "not-covered-some-unevaluated"
    assert entry is None
    assert covered_by(ENTRIES, "198.51.100.4")[0] == "covered"
    assert covered_by([], "198.51.100.4")[0] == "no-entries"


def test_a_wrong_egress_assumption_is_named_before_anybody_files_a_ticket():
    state, detail = egress_assumption(["198.51.100.0/24"], "203.0.113.9")
    assert state == "egress-assumption-wrong"
    assert "would not have helped" in detail
    assert egress_assumption(["203.0.113.0/24"], "203.0.113.9")[0] == (
        "egress-as-expected")
    assert egress_assumption([], "203.0.113.9")[0] == "nothing-declared"


def test_the_pair_of_readings_is_the_headline():
    state, detail = paired_reading(403, 200)
    assert state == "network-path"
    assert "source address" in detail
    assert paired_reading(403, 403)[0] == "refused-everywhere"
    assert paired_reading(403, None)[0] == "single-reading"
    assert paired_reading(200, 200)[0] == "no-refusal"


def test_an_installation_token_and_a_user_token_are_judged_differently():
    which, _ = list_that_applies("App installation token")
    assert which == "org-list-plus-app-managed"
    which, detail = list_that_applies("App user-to-server token")
    assert which == "org-list-only"
    assert "background sync works" in detail
    assert token_kind("ghs_x") == "App installation token"
    assert token_kind("ghu_x") == "App user-to-server token"


def test_an_unreadable_list_is_not_an_empty_one():
    body = {"data": {"organization": None},
            "errors": [{"message": "Resource not accessible"}]}
    setting, apps, entries, note = allow_list_from_graphql(body)
    assert setting is None and entries == []
    assert "admin:org" in note
    state, detail = verdict("ip-allow-list", "rule-unread", None)
    assert state == "rule-unreadable"
    assert "cause is established" in detail


def test_the_entries_are_normalised_off_the_graphql_shape():
    body = {"data": {"organization": {
        "ipAllowListEnabledSetting": "ENABLED",
        "ipAllowListForInstalledAppsEnabledSetting": "DISABLED",
        "ipAllowListEntries": {"nodes": [
            {"allowListValue": "198.51.100.0/24", "isActive": True, "name": "office"},
        ]}}}}
    setting, apps, entries, _ = allow_list_from_graphql(body)
    assert setting == "ENABLED" and apps == "DISABLED"
    assert entries == [{"value": "198.51.100.0/24", "active": True, "name": "office"}]


def test_the_query_this_script_sends_is_a_read():
    assert refuses_mutation(ALLOW_LIST_QUERY) is False
    assert refuses_mutation("mutation M { createIpAllowListEntry { id } }") is True
    assert refuses_mutation("subscription S { x }") is True


def test_the_repair_asks_a_human_and_adds_nothing():
    fix = repair("address-not-covered", "203.0.113.9", "acme")
    assert "ask an owner of acme" in fix
    assert "203.0.113.9/32" in fix
    assert "adds anything" in fix
    assert "switch the existing entry back on" in repair(
        "entry-exists-but-is-off", "203.0.113.9", "acme")


def test_the_two_budgets_are_counted_separately():
    assert read_cost() == (2, 0)
    assert read_cost(True) == (2, 1)
github-ip-allow-list.test.mjs
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
  ALLOW_LIST_QUERY, addressInMessage, allowListFromGraphql, cidrContains,
  classifyRefusal, coveredBy, egressAssumption, ipv4ToInt, listThatApplies,
  looksLikeIpv4, looksLikeIpv6, pairedReading, readCost, refusesMutation,
  repair, tokenKind, verdict,
} from './github-ip-allow-list.mjs';

const ALLOW_LIST_BODY = 'Although you appear to have the correct authorization '
  + 'credentials, the ACME organization has an IP allow list enabled, and '
  + '203.0.113.9 is not permitted to access this resource.';
const UA_BODY = 'Request forbidden by administrative rules. Please make sure '
  + 'your request has a User-Agent header.';

const ENTRIES = [
  { value: '198.51.100.0/24', active: true, name: 'office' },
  { value: '203.0.113.0/24', active: false, name: 'old ci' },
  { value: '2001:db8::/32', active: true, name: 'ipv6 office' },
];

test('the allow list refusal is the only one naming an address', () => {
  const [state, detail] = classifyRefusal(403, ALLOW_LIST_BODY, {});
  assert.equal(state, 'ip-allow-list');
  assert.match(detail, /names an IP address/);
  assert.equal(classifyRefusal(403, UA_BODY, {})[0], 'user-agent-rule');
  assert.equal(
    classifyRefusal(403, 'Resource not accessible by integration', {})[0],
    'permission-or-role',
  );
});

test('quota and secondary limits are sorted out first', () => {
  assert.equal(
    classifyRefusal(403, 'API rate limit exceeded for user ID 1.', {})[0],
    'primary-quota-exhausted',
  );
  assert.equal(
    classifyRefusal(429, 'You have exceeded a secondary rate limit.', {})[0],
    'secondary-limit',
  );
  assert.equal(
    classifyRefusal(403, '', { 'X-RateLimit-Remaining': '0' })[0],
    'primary-quota-exhausted',
  );
});

test('a reworded allow list message still classifies', () => {
  const reworded = 'Access from 198.51.100.77 is blocked by policy for this org.';
  assert.equal(classifyRefusal(403, reworded, {})[0], 'ip-allow-list');
});

test('an allow list message with no address is kept apart', () => {
  assert.equal(
    classifyRefusal(403, 'This org has an IP allow list enabled.', {})[0],
    'ip-allow-list-unaddressed',
  );
});

test('the address survives the full stop at the end of the sentence', () => {
  assert.equal(addressInMessage(ALLOW_LIST_BODY), '203.0.113.9');
  assert.equal(addressInMessage('from (2001:db8::1) today'), '2001:db8::1');
  assert.equal(addressInMessage('no address at all here'), null);
  assert.equal(addressInMessage('version 1.2.3.400 shipped'), null);
});

test('what an address looks like', () => {
  assert.equal(looksLikeIpv4('203.0.113.9'), true);
  assert.equal(looksLikeIpv4('203.0.113.256'), false);
  assert.equal(looksLikeIpv6('2001:db8::1'), true);
  assert.equal(looksLikeIpv6('203.0.113.9'), false);
});

test('cidr arithmetic at the edges', () => {
  assert.equal(cidrContains('203.0.113.0/24', '203.0.113.9'), true);
  assert.equal(cidrContains('203.0.113.0/24', '203.0.114.9'), false);
  assert.equal(cidrContains('203.0.113.9', '203.0.113.9'), true);
  assert.equal(cidrContains('0.0.0.0/0', '8.8.8.8'), true);
  assert.equal(ipv4ToInt('0.0.0.1'), 1);
});

test('an unevaluated entry is null and not false', () => {
  assert.equal(cidrContains('2001:db8::/32', '203.0.113.9'), null);
  assert.equal(cidrContains('not-a-cidr', '203.0.113.9'), null);
  assert.equal(cidrContains('203.0.113.0/xx', '203.0.113.9'), null);
});

test('an entry that exists but is switched off is its own finding', () => {
  const [state, entry] = coveredBy(ENTRIES, '203.0.113.9');
  assert.equal(state, 'covered-but-inactive');
  assert.equal(entry.name, 'old ci');
  assert.equal(verdict('ip-allow-list', state, 'ENABLED')[0], 'entry-exists-but-is-off');
});

test('coverage reports the entries it could not evaluate', () => {
  assert.equal(coveredBy(ENTRIES, '192.0.2.5')[0], 'not-covered-some-unevaluated');
  assert.equal(coveredBy(ENTRIES, '198.51.100.4')[0], 'covered');
  assert.equal(coveredBy([], '198.51.100.4')[0], 'no-entries');
});

test('a wrong egress assumption is named before anybody files a ticket', () => {
  const [state, detail] = egressAssumption(['198.51.100.0/24'], '203.0.113.9');
  assert.equal(state, 'egress-assumption-wrong');
  assert.match(detail, /would not have helped/);
  assert.equal(egressAssumption(['203.0.113.0/24'], '203.0.113.9')[0], 'egress-as-expected');
  assert.equal(egressAssumption([], '203.0.113.9')[0], 'nothing-declared');
});

test('the pair of readings is the headline', () => {
  const [state, detail] = pairedReading(403, 200);
  assert.equal(state, 'network-path');
  assert.match(detail, /source address/);
  assert.equal(pairedReading(403, 403)[0], 'refused-everywhere');
  assert.equal(pairedReading(403, null)[0], 'single-reading');
  assert.equal(pairedReading(200, 200)[0], 'no-refusal');
});

test('an installation token and a user token are judged differently', () => {
  assert.equal(listThatApplies('App installation token')[0], 'org-list-plus-app-managed');
  const [which, detail] = listThatApplies('App user-to-server token');
  assert.equal(which, 'org-list-only');
  assert.match(detail, /background sync works/);
  assert.equal(tokenKind('ghs_x'), 'App installation token');
  assert.equal(tokenKind('ghu_x'), 'App user-to-server token');
});

test('an unreadable list is not an empty one', () => {
  const [setting, , entries, note] = allowListFromGraphql({
    data: { organization: null },
    errors: [{ message: 'Resource not accessible' }],
  });
  assert.equal(setting, null);
  assert.deepEqual(entries, []);
  assert.match(note, /admin:org/);
  assert.equal(verdict('ip-allow-list', 'rule-unread', null)[0], 'rule-unreadable');
});

test('the entries are normalised off the graphql shape', () => {
  const [setting, apps, entries] = allowListFromGraphql({
    data: {
      organization: {
        ipAllowListEnabledSetting: 'ENABLED',
        ipAllowListForInstalledAppsEnabledSetting: 'DISABLED',
        ipAllowListEntries: {
          nodes: [{ allowListValue: '198.51.100.0/24', isActive: true, name: 'office' }],
        },
      },
    },
  });
  assert.equal(setting, 'ENABLED');
  assert.equal(apps, 'DISABLED');
  assert.deepEqual(entries, [{ value: '198.51.100.0/24', active: true, name: 'office' }]);
});

test('the query this script sends is a read', () => {
  assert.equal(refusesMutation(ALLOW_LIST_QUERY), false);
  assert.equal(refusesMutation('mutation M { createIpAllowListEntry { id } }'), true);
  assert.equal(refusesMutation('subscription S { x }'), true);
});

test('the repair asks a human and adds nothing', () => {
  const fix = repair('address-not-covered', '203.0.113.9', 'acme');
  assert.match(fix, /ask an owner of acme/);
  assert.match(fix, /203\.0\.113\.9\/32/);
  assert.match(fix, /adds anything/);
});

test('the two budgets are counted separately', () => {
  assert.deepEqual(readCost(), [2, 0]);
  assert.deepEqual(readCost(true), [2, 1]);
});

FAQ

The same token works on my laptop. Does that not prove the token is fine?

It proves the token is fine <em>from your laptop</em>. An IP allow list is a check on the connection, not on the credential, so the two readings are answering different questions and only look like the same experiment. That is why this is the one diagnostic in the section whose result depends on which machine you run it from: the whole finding is the difference between two locations holding one token. Take the reading on the runner, and pass the laptop's status in so the pair is stated rather than remembered.

Anonymous requests from the same runner still work. How?

An organization allow list governs access to that organization's resources. Unauthenticated reads of a public repository are not access to the organization in that sense, so they keep answering 200 from an address the organization refuses. It is a genuinely confusing signal, because it makes the network look healthy from exactly the machine that is being refused, and it is why people conclude the problem must be the credential.

How is this different from the User-Agent 403 that also names its own rule?

Both name a rule in the body, and that note owns the general sort of a 403 into its causes. This is a fifth cause with a signal none of the others has: an IP address in the message. The script keys on that structurally rather than on the wording, so a rewritten sentence still classifies. The repairs have nothing in common either — one is a header your client sends, the other is an entry somebody with organization admin has to add.

Our GitHub App has an allow list of its own and calls are still refused. Why?

Check which token the failing call carries. The App-managed allow list contributes the App's ranges for <em>installation</em> tokens where the organization has enabled that setting. A user-to-server token acts for a person and is judged against the organization's own list regardless, so an App can have a background sync that works perfectly and interactive calls that are refused, from the same process on the same machine.

Can the script add the range for me if I give it an admin token?

No. Nothing in this section writes, and this is a case where that rule earns its keep: an allow-list entry is a security control, adding one is a decision about who may reach the organization, and a diagnostic tool is not the thing that should make it. The script reads the list, names the address that was refused, says which entry would have covered it, and prints the request for an owner to action.

Related field notes

Sources

Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.

Stuck on a tricky one?

If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.