Diagnostic Slack
The repo manifest, the live manifest and the granted scopes
The reinstall was supposed to be routine. A new workspace, the same app, the same install link everyone has used for a year. Twenty minutes later half the app is broken: it cannot read the channel list, it cannot see reactions, and the two event subscriptions that drive the whole product are gone.
Nobody deployed anything. The repository has not changed in six weeks. What changed is that the reinstall applied the manifest that was live, and the live manifest has been quietly different from the one in the repository since a Sunday in March, when somebody fixed an outage through the web UI at two in the morning and never came back to write it down.
Slack app configuration is editable in two places that do not reconcile: the web UI and the App Manifest API. A UI edit does not touch your repository, and the next apps.manifest.update from CI does not preserve the UI edit — it replaces the document. Neither side warns you. The divergence is invisible until something applies one version over the other, which is usually a reinstall, and by then the scopes or the subscriptions have already gone.
So make it visible on a schedule. apps.manifest.export?app_id=A... returns the live manifest as JSON with an app configuration token. Normalise both documents — sort the keys, sort the arrays — and diff them path by path, so a reordered scope list is not reported as a change and a genuinely missing scope is. Report what moved: which paths exist only in the repo, which exist only on the live app, and which exist in both with different values.
Then add the column that almost nobody checks. The scopes on the installed token are a third list, readable from the X-OAuth-Scopes response header on any Web API call, and it can differ from both of the others. A scope added to the live app is not granted until somebody reinstalls; a scope removed from the live app stays in the token until somebody reinstalls. Three lists, drifting independently, and only one of them is what your code is actually allowed to do today.
The problem in plain words
The two editors have opposite failure modes and both are silent. The web UI is fast, correct, and available at two in the morning during an incident, which is exactly when it gets used and exactly when nobody is writing anything down. The API is authoritative, repeatable and version controlled, and it clobbers: apps.manifest.update replaces the manifest wholesale, so the emergency scope somebody added in March vanishes the next time CI runs, without CI having any idea it removed anything.
What makes the drift so hard to notice is that nothing applies it. The live manifest is the app's configuration and it is already in effect; the repo manifest is a file. They can disagree for a year with no symptom at all, because the only events that reconcile them are a manifest update from CI and an install or reinstall. Both are rare, both happen under time pressure, and both look like routine operations right up to the moment a scope disappears.
The scope column adds a third timeline that nobody has a mental model for. Adding reactions:read to the live app does not give the running token reactions:read — the token was minted at install time and carries what it carried then. So a manifest can be correct, a deploy can be green, and the API can still answer missing_scope, because the grant is a fourth thing that only moves when a human clicks Allow. Teams read the manifest, see the scope listed, and go looking for the bug somewhere else entirely.
And then there is ownership. Once the repo and the app disagree, nobody can say which is right, because both are plausible: the repo was reviewed, and the app is what has actually been running for eight months. The resolution is always a judgement call made under pressure with incomplete memory of who changed what in March, and the usual outcome is that somebody picks one and finds out which parts of the other one mattered afterwards.
Why it happens
Normalisation before comparison is what makes this a tool rather than a nuisance. Slack does not promise key order and does not promise array order, so a naive text diff of two exported manifests reports thirty differences of which zero matter. Sorting keys and sorting arrays first means a difference in the output is a difference in the app. Without that step the check gets ignored inside a fortnight, which is the same as not having it.
Diff by path, not by line. The output that is useful in an incident is oauth_config.scopes.bot changed: reactions:read removed, not a hunk of JSON with a minus in front of it. So the script flattens both documents into dotted paths first, keeps arrays of scalars whole so a scope list is one finding rather than six, and keys arrays of objects by their own identity — features.slash_commands[/deploy].url tells you which command moved, where slash_commands[2].url tells you nothing at all if somebody reordered them.
Not every difference deserves the same alarm. A changed description or a bumped manifest version is noise; a changed scope list, event subscription, interactivity setting or Socket Mode flag is the app behaving differently. The script grades each path and counts them separately, so a build step can fail on the second category without failing every time somebody fixes a typo in the app's blurb.
The third list is the one that explains the errors you are actually seeing. X-OAuth-Scopes comes back on any Web API response and is the definitive statement of what the installed token may do. Held against the repo and the live manifest it produces seven distinct states, and three of them are actionable in different ways: a scope in the repo and nowhere else was never deployed, a scope in the repo and the app but not the token needs a reinstall, and a scope in the token that no longer appears in the app will be lost the next time anybody reinstalls. That last one is the trap that started this note.
The script reads and refuses to reconcile. apps.manifest.export is a read. apps.manifest.update is a write, it replaces the whole document, and a tool that offered to “just sync it” would be capable of deleting the March fix during the incident it was called in to explain. The repair is printed as a decision for a person: pick an authority, backport the difference, then automate the direction you chose.
When the configuration token is dead, the answer is “not assessed”. That credential lives twelve hours and rotates through its own endpoint, and every manifest check in this note depends on it. A run that could not read the live manifest reports exactly that, rather than an empty diff that reads like agreement.
The fix, as a flow
This is the only chain in the batch with a loop under it, and the loop is the whole failure: every emergency fix made through the web UI moves the two documents further apart, while nobody deploys anything and nothing complains. The red arrow is early, at the point where the two editors fail to reconcile, and the two steps after it are simply time passing. The fix branch sorts scopes rather than paths, because the third list is the one that explains the errors you are actually seeing, and the fourth row is the one that decides whether anybody keeps the check: a reordered array is not a finding.
How to fix it
Normalise both documents before comparing anything
normalise walks the manifest recursively, sorts dictionary keys and sorts every array by its canonical serialisation. Slack promises no ordering, so this is the step that stops the check reporting thirty differences that are not differences. Run it on the repo document and the exported document alike.
Flatten to paths so the report names the field
flatten turns a manifest into {path: value}. Arrays of scalars stay whole, so a scope list is one row rather than fifteen. Arrays of objects are keyed by the object's own identity — a slash command by its command, a shortcut by its callback id — so a reordering is not a finding and features.slash_commands[/deploy].url says which one moved.
Diff the two, path by path
diff_manifests returns every path where the two documents disagree, marked only-in-repo, only-in-live or changed, with both values. For the array rows, set_delta reduces a changed scope list to what was added and what was removed, which is the sentence that goes in the incident notes.
Separate what matters from what does not
severity grades each path load-bearing, cosmetic or volatile. Scopes, event subscriptions, interactivity, Socket Mode, redirect URLs, commands and shortcuts change what the app can do. Descriptions and manifest metadata do not. A CI gate that fails on the first category is one somebody keeps; one that fails on all three is one somebody deletes.
Add the granted scopes as a third column
scope_triangle takes the repo scopes, the live scopes and the X-OAuth-Scopes header from any Web API response, and sorts the union into seven states. The interesting ones are not-yet-granted (deployed, never reinstalled), ui-only (added through the web UI and installed, and the repo has never heard of it) and stale-grant (the token still carries it and nothing declares it any more).
Pick an authority and automate that direction
The printed repair is a decision, not a command: make the repository authoritative, run the update from CI on every change, and add this diff as a build step. Where an emergency UI edit is unavoidable, exporting the manifest back into the repository is part of closing the incident. The script never offers to sync for you, because syncing in the wrong direction is how the March fix disappears.
How to check it worked
Backport the difference, commit the exported manifest, and run it again in CI. The number to watch is the load-bearing count; cosmetic drift can stay non-fatal indefinitely.
python3 slack_manifest_drift.py --repo-manifest manifest.json --app-id A05DEPB0T
# repo manifest.json 142 path(s)
# live A05DEPB0T 139 path(s)
# drift drifted 4 load-bearing, 1 cosmetic, 1 volatile
# path oauth_config.scopes.bot changed load-bearing
# added: none removed: reactions:read, groups:read
# path settings.event_subscriptions.bot_events changed load-bearing
# added: none removed: reaction_added, member_joined_channel
# path settings.socket_mode_enabled changed load-bearing
# repo: False live: True
# path features.slash_commands[/deploy].url changed load-bearing
# repo: https://ops.example.com/cmd live: https://a1b2.ngrok-free.app/cmd
# path display_information.description changed cosmetic
# path _metadata.minor_version only-in-live volatile
# scopes aligned 11
# scopes not-yet-granted chat:write.public - in the repo and the app, not the token
# scopes stale-grant files:read - the token holds it and nothing declares it
# verdict the deployed app is not the app in source control
# repair: decide which document is authoritative before changing either one
# repair: backport the live differences into the repo, then run apps.manifest.update
# from CI on every change and fail the build on load-bearing driftThe full code
This is a small diff engine and almost all of the value is in the two steps before the comparison. normalise sorts keys and arrays so ordering is never a finding, and flatten reduces both documents to dotted paths with arrays of scalars kept whole and arrays of objects keyed by their own identity. After that diff_manifests is six lines. scope_triangle is the part that turns a configuration diff into an explanation of today's errors, because the token's grant is a third list and it moves on a different clock from either document.
"""Diff the manifest in your repository against the one Slack is running.
Read only. apps.manifest.export and auth.test are reads; the repository
manifest is read from disk. apps.manifest.update - the method that would
reconcile the two - is a write, it replaces the whole document, and this script
never calls it. Syncing in the wrong direction is how the emergency fix
somebody made through the web UI in March disappears, so the repair is printed
as a decision for a person.
Three lists can disagree and they move on different clocks: the manifest in the
repository, the manifest live on the app, and the scopes the installed token
was actually granted. The last one only changes when a human reinstalls.
"""
import argparse
import json
import logging
import os
import sys
import requests
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("slack_manifest_drift")
API = "https://slack.com/api/"
# Fields whose own identity is a better key than their position, so a
# reordering is not reported as a change and the path says which item moved.
LIST_KEYS = ("command", "callback_id", "action_id", "name", "type", "domain")
# Paths where a difference changes what the app can do.
LOAD_BEARING = (
"oauth_config.scopes", "oauth_config.redirect_urls",
"settings.event_subscriptions", "settings.interactivity",
"settings.socket_mode_enabled", "settings.org_deploy_enabled",
"settings.token_rotation_enabled", "settings.allowed_ip_address_ranges",
"features.slash_commands", "features.shortcuts", "features.bot_user",
"features.unfurl_domains", "features.app_home", "features.workflow_steps",
)
# Paths that differ for reasons nobody chose, and should never fail a build.
VOLATILE = ("_metadata",)
def _canonical(value):
"""A stable string for any JSON value, for ordering. Pure."""
return json.dumps(value, sort_keys=True, default=str)
def normalise(doc):
"""Sort every key and every array so two manifests compare by value. Pure.
Slack promises no key order and no array order, so a naive text diff of two
exports reports differences that are not differences. Doing this first is
what makes the output worth reading, and a check nobody trusts is the same
as no check.
"""
if isinstance(doc, dict):
return {k: normalise(doc[k]) for k in sorted(doc)}
if isinstance(doc, list):
return sorted((normalise(item) for item in doc), key=_canonical)
return doc
def _item_key(item, index):
"""Name a list item by its own identity where it has one. Pure."""
if isinstance(item, dict):
for key in LIST_KEYS:
if item.get(key):
return str(item[key])
return str(index)
def flatten(doc, prefix=""):
"""Reduce a manifest to {dotted path: value}. Pure.
Arrays of scalars are kept whole as a tuple, because "the scope list
changed" is one finding and "scopes[3] changed" is six. Arrays of objects
are keyed by the item's own identity, so slash_commands[/deploy].url names
the command that moved rather than a position that means nothing after a
reorder.
"""
out = {}
if isinstance(doc, dict):
if not doc and prefix:
out[prefix] = {}
for key in sorted(doc):
path = "%s.%s" % (prefix, key) if prefix else str(key)
out.update(flatten(doc[key], path))
return out
if isinstance(doc, list):
if all(not isinstance(item, (dict, list)) for item in doc):
out[prefix or "."] = tuple(doc)
return out
for index, item in enumerate(doc):
out.update(flatten(item, "%s[%s]" % (prefix, _item_key(item, index))))
return out
out[prefix or "."] = doc
return out
def diff_manifests(repo, live):
"""What moved between the repository manifest and the live one? Pure.
Returns [(path, state, repo_value, live_value)] sorted by path, with state
only-in-repo, only-in-live or changed. Both documents are normalised first,
so ordering never appears here.
"""
left = flatten(normalise(repo or {}))
right = flatten(normalise(live or {}))
out = []
for path in sorted(set(left) | set(right)):
if path not in right:
out.append((path, "only-in-repo", left[path], None))
elif path not in left:
out.append((path, "only-in-live", None, right[path]))
elif left[path] != right[path]:
out.append((path, "changed", left[path], right[path]))
return out
def set_delta(before, after):
"""For an array that changed, what came and what went. Pure.
Returns (added, removed). This is the line that goes in the incident notes:
a scope list that "changed" is not actionable, and "reactions:read was
removed" is.
"""
left = set(before or ())
right = set(after or ())
return (sorted(right - left), sorted(left - right))
def severity(path):
"""How much does a difference at this path matter? Pure.
Returns load-bearing, cosmetic or volatile. A gate that fails on
load-bearing drift is one a team keeps; a gate that fails because somebody
fixed a typo in the app description is one they delete in a fortnight.
"""
text = str(path or "")
for prefix in VOLATILE:
if text == prefix or text.startswith(prefix + ".") or text.startswith(prefix + "["):
return "volatile"
for prefix in LOAD_BEARING:
if text == prefix or text.startswith(prefix + ".") or text.startswith(prefix + "["):
return "load-bearing"
return "cosmetic"
def drift_summary(diffs):
"""One state and three counts for the whole comparison. Pure.
Returns (state, counts) with state aligned, cosmetic-only or drifted.
"""
counts = {"load-bearing": 0, "cosmetic": 0, "volatile": 0}
for path, _state, _left, _right in diffs or []:
counts[severity(path)] += 1
if not any(counts.values()):
return ("aligned", counts)
if counts["load-bearing"]:
return ("drifted", counts)
return ("cosmetic-only", counts)
def scope_triangle(repo_scopes, live_scopes, granted_scopes):
"""Three scope lists that drift independently. Pure.
Returns {state: [scope, ...]} over the union of all three. The states exist
because each combination has a different repair:
aligned in all three. Nothing to do.
repo-only declared in the repo and nowhere else: the change was
never deployed to the app at all.
not-yet-granted in the repo and on the app, not in the token. Somebody
has to reinstall before the code can use it.
ui-only on the app and in the token, absent from the repo:
added through the web UI, and the next update from CI
will remove it.
ui-only-ungranted on the app only. Added in the UI, never installed, and
invisible to both the repo and the running code.
removed-live in the repo and the token, gone from the app. The
token still works and the next reinstall drops it.
stale-grant in the token alone. Nothing declares it any more.
"""
repo = {str(s) for s in (repo_scopes or [])}
live = {str(s) for s in (live_scopes or [])}
granted = {str(s) for s in (granted_scopes or [])}
states = {"aligned": [], "repo-only": [], "not-yet-granted": [], "ui-only": [],
"ui-only-ungranted": [], "removed-live": [], "stale-grant": []}
table = {
(True, True, True): "aligned",
(True, False, False): "repo-only",
(True, True, False): "not-yet-granted",
(False, True, True): "ui-only",
(False, True, False): "ui-only-ungranted",
(True, False, True): "removed-live",
(False, False, True): "stale-grant",
}
for scope in sorted(repo | live | granted):
key = (scope in repo, scope in live, scope in granted)
states[table[key]].append(scope)
return states
def split_scopes(header):
"""Read the X-OAuth-Scopes response header into a list. Pure."""
return [part.strip() for part in str(header or "").split(",") if part.strip()]
def bot_scopes(manifest):
"""The bot scopes declared by a manifest. Pure."""
oauth = (manifest or {}).get("oauth_config") or {}
return list((oauth.get("scopes") or {}).get("bot") or [])
def load_live(args):
"""Read the live manifest. A read: export returns it and changes nothing."""
if args.live_manifest:
return json.loads(open(args.live_manifest, encoding="utf-8").read())
token = os.environ.get(args.config_token_env)
if not token or not args.app_id:
log.warning("live unavailable set %s and --app-id, or pass "
"--live-manifest", args.config_token_env)
return None
body = requests.get(API + "apps.manifest.export",
headers={"Authorization": "Bearer " + token},
params={"app_id": args.app_id}, timeout=30).json()
if body.get("ok") is not True:
log.error("live unavailable %s; every manifest finding below is not "
"assessed rather than clean", body.get("error"))
return None
return body.get("manifest") or {}
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--repo-manifest", required=True,
help="path to the manifest checked into your repository")
ap.add_argument("--live-manifest", default="",
help="path to an already exported manifest, instead of exporting")
ap.add_argument("--config-token-env", default="SLACK_CONFIG_TOKEN",
help="environment variable holding an app configuration token")
ap.add_argument("--app-id", default="", help="app id, for the manifest read")
ap.add_argument("--token-env", default="SLACK_BOT_TOKEN",
help="environment variable holding a bot token, to read the grant")
ap.add_argument("--granted", default="",
help="comma separated granted scopes, instead of reading the header")
args = ap.parse_args()
repo = json.loads(open(args.repo_manifest, encoding="utf-8").read())
live = load_live(args)
log.info("repo %-16s %d path(s)", args.repo_manifest,
len(flatten(normalise(repo))))
if live is None:
log.warning("verdict not assessed the live manifest could not be read, "
"so no comparison was made")
return 2
log.info("live %-16s %d path(s)", args.app_id or args.live_manifest,
len(flatten(normalise(live))))
diffs = diff_manifests(repo, live)
state, counts = drift_summary(diffs)
(log.warning if state == "drifted" else log.info)(
"drift %-16s %d load-bearing, %d cosmetic, %d volatile", state,
counts["load-bearing"], counts["cosmetic"], counts["volatile"])
for path, how, left, right in diffs:
grade = severity(path)
(log.warning if grade == "load-bearing" else log.info)(
"path %-40s %-13s %s", path, how, grade)
if isinstance(left, tuple) or isinstance(right, tuple):
added, removed = set_delta(left or (), right or ())
log.info(" added: %s removed: %s",
", ".join(added) or "none", ", ".join(removed) or "none")
else:
log.info(" repo: %s live: %s", left, right)
granted = split_scopes(args.granted)
if not granted:
token = os.environ.get(args.token_env)
if token:
reply = requests.get(API + "auth.test",
headers={"Authorization": "Bearer " + token},
timeout=30)
granted = split_scopes(reply.headers.get("x-oauth-scopes"))
else:
log.warning("scopes set %s or pass --granted to read the third column",
args.token_env)
triangle = scope_triangle(bot_scopes(repo), bot_scopes(live), granted)
for name in ("aligned", "repo-only", "not-yet-granted", "ui-only",
"ui-only-ungranted", "removed-live", "stale-grant"):
items = triangle[name]
if not items:
continue
if name == "aligned":
log.info("scopes %-16s %d", name, len(items))
else:
log.warning("scopes %-16s %s", name, ", ".join(items))
if state == "drifted" or any(triangle[n] for n in
("repo-only", "ui-only", "removed-live",
"stale-grant")):
log.warning("verdict the deployed app is not the app in source control")
log.warning(" repair: decide which document is authoritative before changing "
"either one; this script will not choose for you")
log.warning(" repair: backport the differences, then run apps.manifest.update "
"from CI on every change and fail the build on load-bearing drift")
log.warning(" repair: reinstall after any scope change, because the grant on "
"the installed token moves only when a human clicks Allow")
return 1
log.info("verdict aligned the repo, the app and the grant agree")
return 0
if __name__ == "__main__":
sys.exit(main())
/**
* Diff the manifest in your repository against the one Slack is running.
*
* Read only. apps.manifest.export and auth.test are reads; the repository
* manifest is read from disk. apps.manifest.update - the method that would
* reconcile the two - is a write that replaces the whole document, and this
* script never calls it.
*
* Three lists can disagree and they move on different clocks: the manifest in
* the repository, the manifest live on the app, and the scopes the installed
* token was actually granted.
*/
import { readFileSync } from 'node:fs';
const API = 'https://slack.com/api/';
// Fields whose own identity is a better key than their position.
export const LIST_KEYS = ['command', 'callback_id', 'action_id', 'name', 'type',
'domain'];
// Paths where a difference changes what the app can do.
export const LOAD_BEARING = [
'oauth_config.scopes', 'oauth_config.redirect_urls',
'settings.event_subscriptions', 'settings.interactivity',
'settings.socket_mode_enabled', 'settings.org_deploy_enabled',
'settings.token_rotation_enabled', 'settings.allowed_ip_address_ranges',
'features.slash_commands', 'features.shortcuts', 'features.bot_user',
'features.unfurl_domains', 'features.app_home', 'features.workflow_steps',
];
// Paths that differ for reasons nobody chose.
export const VOLATILE = ['_metadata'];
function canonical(value) {
if (value === null || typeof value !== 'object') return JSON.stringify(value) ?? 'null';
if (Array.isArray(value)) return `[${value.map(canonical).join(',')}]`;
return `{${Object.keys(value).sort().map((k) => `${JSON.stringify(k)}:${canonical(value[k])}`).join(',')}}`;
}
/**
* Sort every key and every array so two manifests compare by value. Pure.
* Slack promises no ordering, so this is what makes the diff worth reading.
*/
export function normalise(doc) {
if (Array.isArray(doc)) {
return doc.map(normalise).sort((a, b) => (canonical(a) < canonical(b) ? -1 : 1));
}
if (doc !== null && typeof doc === 'object') {
const out = {};
for (const key of Object.keys(doc).sort()) out[key] = normalise(doc[key]);
return out;
}
return doc;
}
function itemKey(item, index) {
if (item !== null && typeof item === 'object' && !Array.isArray(item)) {
for (const key of LIST_KEYS) if (item[key]) return String(item[key]);
}
return String(index);
}
/**
* Reduce a manifest to {dotted path: value}. Pure.
* Arrays of scalars stay whole; arrays of objects are keyed by identity.
*/
export function flatten(doc, prefix = '') {
const out = {};
if (Array.isArray(doc)) {
if (doc.every((item) => item === null || typeof item !== 'object')) {
out[prefix || '.'] = [...doc];
return out;
}
doc.forEach((item, index) => {
Object.assign(out, flatten(item, `${prefix}[${itemKey(item, index)}]`));
});
return out;
}
if (doc !== null && typeof doc === 'object') {
const keys = Object.keys(doc).sort();
if (!keys.length && prefix) out[prefix] = {};
for (const key of keys) {
Object.assign(out, flatten(doc[key], prefix ? `${prefix}.${key}` : String(key)));
}
return out;
}
out[prefix || '.'] = doc;
return out;
}
/**
* What moved between the repository manifest and the live one? Pure.
* Returns [[path, state, repoValue, liveValue], ...] sorted by path.
*/
export function diffManifests(repo, live) {
const left = flatten(normalise(repo ?? {}));
const right = flatten(normalise(live ?? {}));
const paths = [...new Set([...Object.keys(left), ...Object.keys(right)])].sort();
const out = [];
for (const path of paths) {
const inLeft = Object.prototype.hasOwnProperty.call(left, path);
const inRight = Object.prototype.hasOwnProperty.call(right, path);
if (!inRight) out.push([path, 'only-in-repo', left[path], null]);
else if (!inLeft) out.push([path, 'only-in-live', null, right[path]]);
else if (canonical(left[path]) !== canonical(right[path])) {
out.push([path, 'changed', left[path], right[path]]);
}
}
return out;
}
/**
* For an array that changed, what came and what went. Pure.
* Returns [added, removed].
*/
export function setDelta(before, after) {
const left = new Set(before ?? []);
const right = new Set(after ?? []);
return [
[...right].filter((v) => !left.has(v)).sort(),
[...left].filter((v) => !right.has(v)).sort(),
];
}
/**
* How much does a difference at this path matter? Pure.
* Returns load-bearing, cosmetic or volatile.
*/
export function severity(path) {
const text = String(path ?? '');
const under = (prefix) => text === prefix || text.startsWith(`${prefix}.`)
|| text.startsWith(`${prefix}[`);
if (VOLATILE.some(under)) return 'volatile';
if (LOAD_BEARING.some(under)) return 'load-bearing';
return 'cosmetic';
}
/**
* One state and three counts for the whole comparison. Pure.
* Returns [state, counts]; aligned, cosmetic-only, drifted.
*/
export function driftSummary(diffs) {
const counts = { 'load-bearing': 0, cosmetic: 0, volatile: 0 };
for (const [path] of diffs ?? []) counts[severity(path)] += 1;
const total = counts['load-bearing'] + counts.cosmetic + counts.volatile;
if (!total) return ['aligned', counts];
if (counts['load-bearing']) return ['drifted', counts];
return ['cosmetic-only', counts];
}
/**
* Three scope lists that drift independently. Pure.
* Returns {state: [scope, ...]} over the union of all three.
*/
export function scopeTriangle(repoScopes, liveScopes, grantedScopes) {
const repo = new Set((repoScopes ?? []).map(String));
const live = new Set((liveScopes ?? []).map(String));
const granted = new Set((grantedScopes ?? []).map(String));
const states = {
aligned: [],
'repo-only': [],
'not-yet-granted': [],
'ui-only': [],
'ui-only-ungranted': [],
'removed-live': [],
'stale-grant': [],
};
const table = {
'true,true,true': 'aligned',
'true,false,false': 'repo-only',
'true,true,false': 'not-yet-granted',
'false,true,true': 'ui-only',
'false,true,false': 'ui-only-ungranted',
'true,false,true': 'removed-live',
'false,false,true': 'stale-grant',
};
const union = [...new Set([...repo, ...live, ...granted])].sort();
for (const scope of union) {
const key = `${repo.has(scope)},${live.has(scope)},${granted.has(scope)}`;
states[table[key]].push(scope);
}
return states;
}
/** Read the X-OAuth-Scopes response header into a list. Pure. */
export function splitScopes(header) {
return String(header ?? '').split(',').map((p) => p.trim()).filter(Boolean);
}
/** The bot scopes declared by a manifest. Pure. */
export function botScopes(manifest) {
const oauth = (manifest ?? {}).oauth_config ?? {};
return [...((oauth.scopes ?? {}).bot ?? [])];
}
function arg(args, name, fallback = null) {
const i = args.indexOf(name);
return i === -1 ? fallback : args[i + 1];
}
async function loadLive(args) {
const path = arg(args, '--live-manifest', '');
if (path) return JSON.parse(readFileSync(path, 'utf8'));
const tokenEnv = arg(args, '--config-token-env', 'SLACK_CONFIG_TOKEN');
const appId = arg(args, '--app-id', '');
const token = process.env[tokenEnv];
if (!token || !appId) {
console.warn(`live unavailable set ${tokenEnv} and --app-id, or pass `
+ '--live-manifest');
return null;
}
const params = new URLSearchParams({ app_id: appId });
const body = await (await fetch(`${API}apps.manifest.export?${params}`,
{ headers: { Authorization: `Bearer ${token}` } })).json();
if (body.ok !== true) {
console.error(`live unavailable ${body.error}; every manifest finding `
+ 'below is not assessed rather than clean');
return null;
}
return body.manifest ?? {};
}
async function main() {
const args = process.argv.slice(2);
const repoPath = arg(args, '--repo-manifest', '');
if (!repoPath) {
console.error('pass --repo-manifest with the manifest checked into your repository');
process.exitCode = 2;
return;
}
const repo = JSON.parse(readFileSync(repoPath, 'utf8'));
const live = await loadLive(args);
console.log(`repo ${repoPath.padEnd(16)} `
+ `${Object.keys(flatten(normalise(repo))).length} path(s)`);
if (live === null) {
console.warn('verdict not assessed the live manifest could not be read, so '
+ 'no comparison was made');
process.exitCode = 2;
return;
}
console.log(`live ${String(arg(args, '--app-id', '') || arg(args, '--live-manifest', '')).padEnd(16)} `
+ `${Object.keys(flatten(normalise(live))).length} path(s)`);
const diffs = diffManifests(repo, live);
const [state, counts] = driftSummary(diffs);
const head = `drift ${state.padEnd(16)} ${counts['load-bearing']} load-bearing, `
+ `${counts.cosmetic} cosmetic, ${counts.volatile} volatile`;
if (state === 'drifted') console.warn(head); else console.log(head);
for (const [path, how, left, right] of diffs) {
const grade = severity(path);
const row = `path ${path.padEnd(40)} ${how.padEnd(13)} ${grade}`;
if (grade === 'load-bearing') console.warn(row); else console.log(row);
if (Array.isArray(left) || Array.isArray(right)) {
const [added, removed] = setDelta(left ?? [], right ?? []);
console.log(` added: ${added.join(', ') || 'none'} `
+ `removed: ${removed.join(', ') || 'none'}`);
} else {
console.log(` repo: ${left} live: ${right}`);
}
}
let granted = splitScopes(arg(args, '--granted', ''));
if (!granted.length) {
const tokenEnv = arg(args, '--token-env', 'SLACK_BOT_TOKEN');
const token = process.env[tokenEnv];
if (token) {
const reply = await fetch(`${API}auth.test`,
{ headers: { Authorization: `Bearer ${token}` } });
granted = splitScopes(reply.headers.get('x-oauth-scopes'));
} else {
console.warn(`scopes set ${tokenEnv} or pass --granted to read the third `
+ 'column');
}
}
const triangle = scopeTriangle(botScopes(repo), botScopes(live), granted);
for (const name of ['aligned', 'repo-only', 'not-yet-granted', 'ui-only',
'ui-only-ungranted', 'removed-live', 'stale-grant']) {
const items = triangle[name];
if (!items.length) continue;
if (name === 'aligned') console.log(`scopes ${name.padEnd(16)} ${items.length}`);
else console.warn(`scopes ${name.padEnd(16)} ${items.join(', ')}`);
}
const scopeFindings = ['repo-only', 'ui-only', 'removed-live', 'stale-grant']
.some((name) => triangle[name].length);
if (state === 'drifted' || scopeFindings) {
console.warn('verdict the deployed app is not the app in source control');
console.warn(' repair: decide which document is authoritative before changing '
+ 'either one; this script will not choose for you');
console.warn(' repair: backport the differences, then run apps.manifest.update '
+ 'from CI on every change and fail the build on load-bearing drift');
console.warn(' repair: reinstall after any scope change, because the grant on the '
+ 'installed token moves only when a human clicks Allow');
process.exitCode = 1;
return;
}
console.log('verdict aligned the repo, the app and the grant agree');
}
// Only run when invoked directly, so importing this module in the tests does
// not execute main().
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}
Add a test
The first two tests are the ones that decide whether anybody keeps this check: a reordered scope list and a reordered set of slash commands must produce no findings at all, because a diff that cries wolf on ordering is deleted within a fortnight. After that the tests pin the identity keying (a command is named by its command, not its index), the array rows staying whole so one changed scope list is one row, and every one of the seven states in the scope triangle — including stale-grant, the one that explains why a token still works for a scope the app no longer declares.
from slack_manifest_drift import (
bot_scopes, diff_manifests, drift_summary, flatten, normalise, scope_triangle,
set_delta, severity, split_scopes,
)
def app(scopes=("chat:write", "channels:read"), events=("app_mention",),
socket=False, commands=(), description="the release bot"):
return {
"_metadata": {"major_version": 1, "minor_version": 1},
"display_information": {"name": "releasebot", "description": description},
"oauth_config": {"scopes": {"bot": list(scopes)}},
"settings": {
"socket_mode_enabled": socket,
"event_subscriptions": {"bot_events": list(events)},
},
"features": {"slash_commands": list(commands)},
}
def test_a_reordered_scope_list_is_not_a_difference():
left = app(scopes=("chat:write", "channels:read"))
right = app(scopes=("channels:read", "chat:write"))
assert diff_manifests(left, right) == []
def test_reordered_slash_commands_are_not_a_difference_either():
one = {"command": "/deploy", "url": "https://ops.example.com/a"}
two = {"command": "/rollback", "url": "https://ops.example.com/b"}
assert diff_manifests(app(commands=(one, two)), app(commands=(two, one))) == []
def test_a_command_is_named_by_its_command_and_not_its_index():
paths = flatten(normalise(app(commands=(
{"command": "/deploy", "url": "https://ops.example.com/a"},))))
assert "features.slash_commands[/deploy].url" in paths
def test_a_missing_scope_is_one_row_and_not_six():
diffs = diff_manifests(app(), app(scopes=("chat:write",)))
assert len(diffs) == 1
path, state, left, right = diffs[0]
assert path == "oauth_config.scopes.bot"
assert state == "changed"
assert set_delta(left, right) == ([], ["channels:read"])
def test_a_path_only_the_live_app_has_is_reported_as_such():
live = app()
live["settings"]["interactivity"] = {"is_enabled": True}
diffs = dict((d[0], d[1]) for d in diff_manifests(app(), live))
assert diffs["settings.interactivity.is_enabled"] == "only-in-live"
def test_a_path_only_the_repo_has_is_reported_as_such():
repo = app()
repo["settings"]["interactivity"] = {"is_enabled": True}
diffs = dict((d[0], d[1]) for d in diff_manifests(repo, app()))
assert diffs["settings.interactivity.is_enabled"] == "only-in-repo"
def test_scalar_values_are_compared_by_value():
diffs = diff_manifests(app(socket=False), app(socket=True))
assert diffs == [("settings.socket_mode_enabled", "changed", False, True)]
def test_scope_and_event_and_socket_paths_are_load_bearing():
for path in ("oauth_config.scopes.bot",
"settings.event_subscriptions.bot_events",
"settings.socket_mode_enabled",
"features.slash_commands[/deploy].url",
"settings.interactivity.request_url"):
assert severity(path) == "load-bearing"
def test_descriptions_are_cosmetic_and_metadata_is_volatile():
assert severity("display_information.description") == "cosmetic"
assert severity("_metadata.minor_version") == "volatile"
def test_a_prefix_match_does_not_leak_across_names():
assert severity("settings.socket_mode_enabled_at") == "cosmetic"
def test_the_summary_separates_the_gate_from_the_noise():
state, counts = drift_summary([("display_information.description", "changed",
"a", "b")])
assert state == "cosmetic-only"
assert counts["load-bearing"] == 0
assert drift_summary([])[0] == "aligned"
assert drift_summary([("oauth_config.scopes.bot", "changed", (), ())])[0] == "drifted"
def test_the_three_lists_agreeing_is_the_only_quiet_state():
tri = scope_triangle(["chat:write"], ["chat:write"], ["chat:write"])
assert tri["aligned"] == ["chat:write"]
assert all(not tri[k] for k in tri if k != "aligned")
def test_a_scope_only_in_the_repo_was_never_deployed():
tri = scope_triangle(["reactions:read"], [], [])
assert tri["repo-only"] == ["reactions:read"]
def test_a_scope_deployed_and_never_reinstalled_needs_a_human():
tri = scope_triangle(["reactions:read"], ["reactions:read"], [])
assert tri["not-yet-granted"] == ["reactions:read"]
def test_a_scope_added_through_the_web_ui_is_named_as_such():
tri = scope_triangle([], ["files:read"], ["files:read"])
assert tri["ui-only"] == ["files:read"]
assert scope_triangle([], ["files:read"], [])["ui-only-ungranted"] == ["files:read"]
def test_a_scope_removed_from_the_app_still_works_until_a_reinstall():
tri = scope_triangle(["groups:read"], [], ["groups:read"])
assert tri["removed-live"] == ["groups:read"]
def test_a_grant_nothing_declares_any_more_is_a_stale_grant():
tri = scope_triangle([], [], ["im:read"])
assert tri["stale-grant"] == ["im:read"]
def test_the_grant_header_is_read_from_a_comma_separated_string():
assert split_scopes("chat:write, channels:read ,") == ["chat:write", "channels:read"]
assert split_scopes(None) == []
def test_the_bot_scopes_are_read_out_of_the_manifest_shape():
assert bot_scopes(app(scopes=("chat:write",))) == ["chat:write"]
assert bot_scopes({}) == []
def test_normalise_leaves_scalars_alone_and_sorts_everything_else():
assert normalise({"b": [3, 1, 2], "a": 1}) == {"a": 1, "b": [1, 2, 3]}
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
botScopes, diffManifests, driftSummary, flatten, normalise, scopeTriangle, setDelta,
severity, splitScopes,
} from './slack-manifest-drift.mjs';
function app({
scopes = ['chat:write', 'channels:read'], events = ['app_mention'], socket = false,
commands = [], description = 'the release bot',
} = {}) {
return {
_metadata: { major_version: 1, minor_version: 1 },
display_information: { name: 'releasebot', description },
oauth_config: { scopes: { bot: [...scopes] } },
settings: {
socket_mode_enabled: socket,
event_subscriptions: { bot_events: [...events] },
},
features: { slash_commands: [...commands] },
};
}
test('a reordered scope list is not a difference', () => {
const left = app({ scopes: ['chat:write', 'channels:read'] });
const right = app({ scopes: ['channels:read', 'chat:write'] });
assert.deepEqual(diffManifests(left, right), []);
});
test('reordered slash commands are not a difference either', () => {
const one = { command: '/deploy', url: 'https://ops.example.com/a' };
const two = { command: '/rollback', url: 'https://ops.example.com/b' };
assert.deepEqual(
diffManifests(app({ commands: [one, two] }), app({ commands: [two, one] })), []);
});
test('a command is named by its command and not its index', () => {
const paths = flatten(normalise(app({
commands: [{ command: '/deploy', url: 'https://ops.example.com/a' }],
})));
assert.ok('features.slash_commands[/deploy].url' in paths);
});
test('a missing scope is one row and not six', () => {
const diffs = diffManifests(app(), app({ scopes: ['chat:write'] }));
assert.equal(diffs.length, 1);
const [path, state, left, right] = diffs[0];
assert.equal(path, 'oauth_config.scopes.bot');
assert.equal(state, 'changed');
assert.deepEqual(setDelta(left, right), [[], ['channels:read']]);
});
test('a path only the live app has is reported as such', () => {
const live = app();
live.settings.interactivity = { is_enabled: true };
const diffs = Object.fromEntries(diffManifests(app(), live).map((d) => [d[0], d[1]]));
assert.equal(diffs['settings.interactivity.is_enabled'], 'only-in-live');
});
test('a path only the repo has is reported as such', () => {
const repo = app();
repo.settings.interactivity = { is_enabled: true };
const diffs = Object.fromEntries(diffManifests(repo, app()).map((d) => [d[0], d[1]]));
assert.equal(diffs['settings.interactivity.is_enabled'], 'only-in-repo');
});
test('scalar values are compared by value', () => {
assert.deepEqual(diffManifests(app({ socket: false }), app({ socket: true })),
[['settings.socket_mode_enabled', 'changed', false, true]]);
});
test('scope and event and socket paths are load bearing', () => {
for (const path of ['oauth_config.scopes.bot',
'settings.event_subscriptions.bot_events',
'settings.socket_mode_enabled',
'features.slash_commands[/deploy].url',
'settings.interactivity.request_url']) {
assert.equal(severity(path), 'load-bearing');
}
});
test('descriptions are cosmetic and metadata is volatile', () => {
assert.equal(severity('display_information.description'), 'cosmetic');
assert.equal(severity('_metadata.minor_version'), 'volatile');
});
test('a prefix match does not leak across names', () => {
assert.equal(severity('settings.socket_mode_enabled_at'), 'cosmetic');
});
test('the summary separates the gate from the noise', () => {
const [state, counts] = driftSummary([['display_information.description', 'changed',
'a', 'b']]);
assert.equal(state, 'cosmetic-only');
assert.equal(counts['load-bearing'], 0);
assert.equal(driftSummary([])[0], 'aligned');
assert.equal(driftSummary([['oauth_config.scopes.bot', 'changed', [], []]])[0],
'drifted');
});
test('the three lists agreeing is the only quiet state', () => {
const tri = scopeTriangle(['chat:write'], ['chat:write'], ['chat:write']);
assert.deepEqual(tri.aligned, ['chat:write']);
for (const key of Object.keys(tri)) {
if (key !== 'aligned') assert.deepEqual(tri[key], []);
}
});
test('a scope only in the repo was never deployed', () => {
assert.deepEqual(scopeTriangle(['reactions:read'], [], [])['repo-only'],
['reactions:read']);
});
test('a scope deployed and never reinstalled needs a human', () => {
assert.deepEqual(
scopeTriangle(['reactions:read'], ['reactions:read'], [])['not-yet-granted'],
['reactions:read']);
});
test('a scope added through the web ui is named as such', () => {
assert.deepEqual(scopeTriangle([], ['files:read'], ['files:read'])['ui-only'],
['files:read']);
assert.deepEqual(scopeTriangle([], ['files:read'], [])['ui-only-ungranted'],
['files:read']);
});
test('a scope removed from the app still works until a reinstall', () => {
assert.deepEqual(scopeTriangle(['groups:read'], [], ['groups:read'])['removed-live'],
['groups:read']);
});
test('a grant nothing declares any more is a stale grant', () => {
assert.deepEqual(scopeTriangle([], [], ['im:read'])['stale-grant'], ['im:read']);
});
test('the grant header is read from a comma separated string', () => {
assert.deepEqual(splitScopes('chat:write, channels:read ,'),
['chat:write', 'channels:read']);
assert.deepEqual(splitScopes(null), []);
});
test('the bot scopes are read out of the manifest shape', () => {
assert.deepEqual(botScopes(app({ scopes: ['chat:write'] })), ['chat:write']);
assert.deepEqual(botScopes({}), []);
});
test('normalise leaves scalars alone and sorts everything else', () => {
assert.deepEqual(normalise({ b: [3, 1, 2], a: 1 }), { a: 1, b: [1, 2, 3] });
});
FAQ
Which one is authoritative, the repo or the live app?
Whichever you decide, and the decision has to come before the next change rather than during the next incident. The workable arrangement is repo-authoritative: the manifest is reviewed like code, CI applies it with apps.manifest.update, and this diff runs as a build step so drift fails the build. That only holds if emergency UI edits are followed by exporting the manifest back into the repository as part of closing the incident, because otherwise the next CI run silently deletes the fix that stopped the outage.
Why does this script refuse to sync the two for me?
Because syncing means calling apps.manifest.update, which is a write and which replaces the entire document rather than merging into it. Run in the wrong direction it removes the change somebody made through the UI at two in the morning, and the moment you are most likely to run it is the moment you understand that change least. This section's scripts read and print; the reconciliation is a decision with consequences, and it belongs to a person who knows which half is the fix.
Our manifest matches, so why is the API still returning missing_scope?
Because the token's grant is a third list. Scopes take effect at install time: adding one to the app changes what will be requested next time somebody installs, not what the existing token may do. Until a human reinstalls and clicks Allow, the running token carries exactly what it carried before. That state is not-yet-granted in this script's output, and it is the single most common reason a correct-looking manifest coexists with a scope error.
What is a stale grant, and is it dangerous?
It is a scope the installed token still carries that no longer appears anywhere in the app's configuration, usually because somebody removed it from the manifest without reinstalling. It is not dangerous today and it is a trap tomorrow: the code still works, so nobody notices, and the capability disappears the moment anyone reinstalls the app for any reason. A workspace migration or a routine reauthorisation then looks like it broke the app, when what it actually did was apply a change made months earlier.
Should this run in CI or on a schedule?
Both, for different reasons. In CI it catches the case where somebody edits the repository manifest and expects that to be enough. On a schedule, daily, it catches the case CI cannot see at all: a UI edit made after the last build, which will sit undetected until the next deploy overwrites it. Note the check needs an app configuration token, which lives twelve hours, so a scheduled run has to rotate its refresh token rather than store an access token.
Related field notes
- the twelve-hour credential this whole check depends on
- the error a not-yet-granted scope produces at runtime
- the scopes in the grant that nothing ever calls
Sources
Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.
- apps.manifest.export method reference - Slack Docs
- apps.manifest.update method reference - Slack Docs
- App manifests - Slack Docs
- bolt-js #2437: the deployed manifest and the checked-in one diverge
If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.