Skip to content

Diagnostic Slack

no_service: the incoming webhook died with its install

The deploy notifications stopped. Nobody noticed for five weeks, because the absence of a message is not a message, and a channel that has gone quiet looks exactly like a channel where nothing has gone wrong. When somebody finally checks, the build server is still calling the same URL it has called for two years, the URL still looks perfectly well formed, and Slack has been answering 404 with the four words no_service into a curl whose exit status nothing has ever read.

The webhook did not expire. Somebody left the company, or somebody reinstalled the app, and the URL died at that moment. There is nothing in the URL that records this, and there is nothing in your build server that could have found out.

Read-only token Python and Node.js Tests included
A laptop on a desk
Photo by Markus Winkler on Unsplash
The short answer

An incoming webhook URL is bound to one app installation, one authorising user and one channel. Uninstall the app, revoke the authorisation, delete the webhook in app configuration, or deactivate the user who clicked Allow, and the URL is dead permanently. It answers 404 with the plain text body no_service or no_active_hooks, or 401 invalid_token, and it will answer that forever. Reinstalling the app does not resurrect it: a reinstall mints a new URL.

The detail that catches people is the fourth cause. A webhook dies with the human who authorised it, and a bot token does not. So an app can be perfectly healthy — auth.test answers ok: true, chat.postMessage works, the bot is in every channel it should be — while three webhook URLs pasted into three unrelated systems have all been dead since an offboarding nobody connected to Slack.

Do not send anything to the URL to find out. A webhook URL has exactly one verb and no read side: no status endpoint, no probe, no metadata call. Anything you send is a delivery, and a delivery lands in a real channel in front of real people, who then have to be told what it was. The script below establishes all of this without sending: from the shape of the stored URL, from your install records, from users.info on the authorising user, and from the bound channel's own history showing what the webhook has and has not delivered.

The problem in plain words

The reason this note is about an inventory rather than about a URL is that almost nobody has an inventory. A webhook URL is created by clicking through an install, copied out of a browser once, and pasted into whatever needed it: a build server, a monitoring rule, a cron job on a box that predates the current team, a Zapier step, a colleague's shell alias. It is a bearer credential with no name, no owner, no expiry and no listing endpoint, and the moment it leaves the browser Slack has no idea where it went and neither do you.

That is survivable while the URLs work. It stops being survivable at the moment one dies, because the failure has no reader. A webhook that fails answers with a real HTTP status — genuinely unusual on Slack, where almost everything is 200 with ok: false in the body — and the status arrives in the one class of client least likely to check it. curl exits 0 on a 404 unless you passed --fail. A shell step in CI goes green. A monitoring rule that alerts through the dead webhook cannot alert you that the webhook is dead.

The four ways to kill one are worth separating, because they have different blast radii and different repairs. Uninstalling the app kills every webhook the app ever issued in that workspace, all at once, which at least fails loudly enough to be noticed. Revoking one authorisation kills the webhooks minted under it. Deleting the webhook from app configuration kills that one URL. And deactivating the authorising user kills the webhooks they authorised, quietly, on a date chosen by HR, while every other part of the app carries on working perfectly.

Then there is the slow version, which is worse. Nothing has died at all: the app is installed, the user is active, the URL is fine, and the system that used to send through it was decommissioned eighteen months ago. The URL is still valid, still a credential, still in a config file in a repository somewhere, and still capable of posting into your workspace as your app if it ever leaks. A dead webhook is an outage. A live orphaned webhook is a standing grant to whoever finds it.

A webhook ismintedpasted into abuild serverTwo years ofalertsand nobody ownsthe URLThe installerleavesor the app isreinstalled404 no_serviceinto a curlnothing checksThe channelgoes quietand quiet reads ascalm
The URL is a bearer credential with no expiry stamped on it and no owner recorded anywhere. It looks exactly the same the day it stops working as it did the day it was minted.

Why it happens

Nothing here sends anything to a webhook URL, and that is not squeamishness. A webhook has no read side. Every request to it that Slack accepts is a message posted into somebody's channel: an empty payload is refused, a test payload is delivered, and either way somebody now has to explain it. This is the one surface in this section where a diagnostic that "just tries it" is not a write against a database, it is a write in front of an audience. So the script reads the URL's shape, the install records, the authorising user and the bound channel, and reaches the same verdict without an audience.

The shape of the URL is worth reading before anything else, because half of the bad rows are not webhooks. A stored value that is not hooks.slack.com/services/T…/B…/… is something else: a workflow trigger URL, a Web API URL somebody filed in the wrong column, a truncated paste, a URL that picked up a trailing quote from a YAML file. Those fail identically to a dead webhook and are fixed completely differently, so they are sorted out first.

The team ID is in the URL, and it is a free join key. The first path segment of a webhook URL is the workspace it was issued for. Holding that against the workspaces you actually hold installs for finds the rows that belong to a workspace you no longer serve — a customer that churned, a test workspace, a Grid workspace that was split off — without any call at all.

The authorising user is the join everyone forgets. users.info on the person who clicked Allow costs one read and answers the question that no amount of staring at the app's health will answer, because the app is healthy. This is also why the verdict for that case says so explicitly rather than just saying dead: the repair is a reinstall by somebody who is not leaving, and ideally a move to a bot token, which has no human attached to it.

The bound channel's history is the only positive evidence available. Everything else in this check is an absence. conversations.history on the channel the webhook posts into says when an app last delivered anything there, which turns "we think this still works" into a date. A channel where no app has ever posted is a webhook that has delivered nothing in the sample read, which for a deploy notifier is the whole finding.

The inventory gaps are reported as findings in their own right. A row with no recorded owner cannot be rotated, because a rotation has nowhere to land. A row with no recorded authorising user cannot be watched for the one event that kills it. The same URL under two owners means a single rotation breaks a system nobody remembered. None of these are outages today and all of them are the reason the next one takes five weeks to find.

The fix, as a flow

This chain runs on a calendar rather than on a request, because the two halves of the fault are months apart. Somebody leaves, or an app is reinstalled, and the URL that a build server has been holding for two years stops working at that moment. Nothing tells the build server. The red step is not an error anybody read: it is a 404 landing in a curl command whose exit status nothing checks, in a pipeline that goes green. The fix branch sorts an inventory rather than a token, because the question is never really whether one URL works. It is which systems hold a URL at all, and whether the install behind each one is still standing.

Inventory beside installsshape, owner, install, historyInstall already revokedthe URL answers no_serviceInstaller deactivatedthe bot token still worksIssued for another teaman inventory row from elsewhereNo system recordeda rotation has nowhere to landDelivered this weekinstall live, channel receiving
Nothing is sent to any URL to reach this. The shape of the URL, the state of the install, the installing user and what the bound channel has actually received are all readable, and together they say more than a delivery attempt would.

How to fix it

Write down where the URLs are, once

The inventory is a JSON list, one row per place a webhook URL is pasted: the URL, the system using it, the authorising user, the bound channel and the state of the install. If you do not have one, building it is the work — the script cannot discover a URL sitting in a Jenkins credential store. Everything after this step is mechanical.

Sort the rows that are not webhooks out first

parse_webhook_url reads the shape and answers services, workflow-trigger, not-a-webhook or unusable. A trailing quote picked up from YAML, a Web API URL in the wrong column and a workflow trigger URL all fail like a dead webhook and are none of them fixed like one.

Check the workspace in the URL against the workspaces you serve

The first path segment is the team the webhook was issued for. A row whose team is not one you hold an install for is an orphan from a workspace you no longer serve, and it is reported before any network call is made, because the URL already said so.

Read the authorising user, not the app

One users.info per distinct authoriser. deleted: true is the finding that nothing else in your monitoring will produce, because the bot token is fine, the app is installed, and the webhook has been dead since the leaving date.

Date the last delivery from the channel, not from your logs

conversations.history on the bound channel, then last_webhook_delivery for the most recent message an app posted. Your own logs record what you sent; the channel records what arrived. When those disagree, the channel is right.

Move the ones that matter off webhooks entirely

The repair the script prints for dead-installer is not "reinstall". A bot token and chat.postMessage survive the installing user leaving, post to any channel the bot is in rather than one fixed room, and answer with a structured error you can read. Webhooks are the right tool for exactly one thing: a message into one known channel from a system that cannot hold a token.

How to check it worked

Fix the rows, run it again, and read the last column. Every row should say live, and the gaps section should be empty.

python3 slack_webhook_inventory.py --inventory webhooks.json --stale-days 14
# anchor     T04AB12CD      auth.test names the workspace this token lives in
# webhook    live           ci/deploy-notify -> hooks.slack.com/services/T04AB/B071P/********
#                           an app delivered into the bound channel 1 day(s) ago
# webhook    dead-installer alerts/pagerduty -> hooks.slack.com/services/T04AB/B05KQ/********
#                           the authorising user is deactivated; a webhook dies with its
#                           authoriser even though the bot token keeps working
# webhook    never-delivered ops/nightly-backup -> hooks.slack.com/services/T04AB/B02WE/********
#                           nothing in the bound channel came from an app in the sample read
# webhook    unusable-url   legacy/status-page -> <not-a-webhook at api.slack.com>
# gap        no-owner       one row records no system, so a rotation has nowhere to land
# verdict    3 finding(s)
#   repair: reinstall to mint a new URL, and move the alerting paths to a bot token
#   note:   nothing was sent to any webhook URL to establish this

The full code

Three reads and no sends. parse_webhook_url and redact_webhook_url are pure and never let a secret reach the output; webhook_verdict is a pure decision over one inventory row and the facts the reads established, in a fixed precedence so that a revoked install is never reported as merely silent; last_webhook_delivery turns a page of channel history into a number of days; inventory_gaps and duplicate_owners report the bookkeeping that makes the next outage slow to find. There is no code path in this file that sends anything to hooks.slack.com.

Get this script on GitHub Follow @allanninal Python and Node.js, with tests. Read only, it never writes. One of 108 Slack fixes, free and open source.
slack_webhook_inventory.py
"""Hold an inventory of incoming webhook URLs against the installs behind them.

Read only, and one thing is refused outright. An incoming webhook URL exists
for exactly one purpose: to have a message sent to it. There is no status
endpoint, no HEAD, no "is this alive" call. The only way to ask Slack directly
whether a webhook still works is to deliver a message through it, into a real
channel, in front of real people, who then have to be told what the message
was. So nothing in this file sends anything to hooks.slack.com.

Everything is established instead from the shape of the stored URL, from your
own install records, from users.info on the authorising user, and from the
bound channel's history showing what the webhook has and has not delivered.

Three reads: auth.test once, users.info once per distinct authorising user,
conversations.history once per distinct bound channel. No webhook URL is ever
printed in full - a webhook URL is a bearer credential, and anyone holding one
can post into your workspace as your app.
"""
import argparse
import json
import logging
import os
import sys
import time
from urllib.parse import urlsplit

import requests

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("slack_webhook_inventory")

API = "https://slack.com/api/"
HOOK_HOST = "hooks.slack.com"

# Install states an inventory row can carry that already mean the webhook is
# gone. Slack does not revoke a webhook on its own: it dies with the
# installation, with the authorisation, or with the human who granted it.
DEAD_INSTALL_STATES = ("revoked", "uninstalled", "token_revoked", "deleted")


def parse_webhook_url(url):
    """Read a webhook URL's shape without contacting anything. Pure.

    Returns (kind, parts, notes). Kinds:

      services          hooks.slack.com/services/T.../B.../secret, the real thing.
      workflow-trigger  a Workflow Builder trigger URL, which is a different
                        surface with a different lifecycle and different repairs.
      not-a-webhook     a URL, but not one Slack delivers webhooks through.
      unusable          not a URL at all, which is what a trailing quote picked
                        up from a YAML file leaves behind.

    parts never contains the secret segment, only its length, so that a caller
    cannot accidentally log one by logging this.
    """
    text = str(url or "").strip()
    if not text:
        return ("unusable", {}, ["empty"])
    try:
        split = urlsplit(text if "://" in text else "https://" + text)
        host = (split.hostname or "").lower()
    except ValueError:
        return ("unusable", {}, ["unparseable"])
    if not host:
        return ("unusable", {}, ["unparseable"])
    notes = []
    if (split.scheme or "https").lower() != "https":
        notes.append("insecure-scheme")
    segments = [s for s in (split.path or "").split("/") if s]
    if host != HOOK_HOST:
        return ("not-a-webhook", {"host": host},
                notes + ["slack-but-not-hooks" if host.endswith("slack.com")
                         else "foreign-host"])
    if segments[:1] == ["services"]:
        if len(segments) != 4:
            return ("unusable", {"host": host}, notes + ["truncated-path"])
        team, hook, secret = segments[1], segments[2], segments[3]
        if not team.startswith(("T", "E")):
            notes.append("odd-team-segment")
        if not hook.startswith("B"):
            notes.append("odd-hook-segment")
        if len(secret) < 20:
            notes.append("short-secret")
        return ("services",
                {"host": host, "team": team, "hook": hook, "secret_len": len(secret)},
                notes)
    if segments[:1] in (["triggers"], ["workflows"]):
        return ("workflow-trigger",
                {"host": host, "team": segments[1] if len(segments) > 1 else ""},
                notes + ["workflow-trigger"])
    return ("not-a-webhook", {"host": host}, notes + ["unknown-path"])


def redact_webhook_url(url):
    """A form of the URL safe to print, log, paste and screenshot. Pure.

    The secret segment never appears, and neither does enough of the other two
    to reconstruct anything. Everything this script prints goes through here.
    """
    kind, parts, _notes = parse_webhook_url(url)
    if kind == "services":
        return "%s/services/%s/%s/%s" % (parts["host"], parts["team"][:5],
                                         parts["hook"][:5], "*" * 8)
    if kind == "unusable":
        return "<unusable value>"
    return "<%s at %s>" % (kind, parts.get("host", "unknown"))


def last_webhook_delivery(messages, now_ts):
    """Days since an app last posted into this channel. Pure.

    Returns None when nothing in the sample came from an app at all, which is
    a stronger statement than "old": it means this channel has no record of the
    webhook ever having delivered within the window that was read.
    """
    best = None
    for m in messages or []:
        if not isinstance(m, dict):
            continue
        if not (m.get("bot_id") or m.get("app_id") or m.get("subtype") == "bot_message"):
            continue
        try:
            ts = float(m.get("ts") or 0)
        except (TypeError, ValueError):
            continue
        if ts and (best is None or ts > best):
            best = ts
    if best is None:
        return None
    return max(0, int((float(now_ts) - best) // 86400))


def webhook_verdict(record, facts):
    """One inventory row, judged against what the reads established. Pure.

    facts: installer_deleted (True, False or None), last_delivery_days (int or
    None), stale_days (int), known_teams (list of team ids you hold installs
    for; empty disables the check).

    The precedence is deliberate and runs from permanent to circumstantial. A
    revoked install reported as merely "silent" would send somebody to look at
    the sending system, which is working perfectly and cannot be fixed.
    """
    kind, parts, _notes = parse_webhook_url(record.get("url"))
    if kind != "services":
        return ("unusable-url",
                "the stored value is not an incoming webhook URL; it reads as %s" % kind)
    state = str(record.get("install_state") or "").strip().lower()
    if state in DEAD_INSTALL_STATES:
        return ("dead-install",
                "the installation behind this webhook is %s, so the URL answers "
                "no_service and a reinstall mints a different one" % state)
    if facts.get("installer_deleted") is True:
        return ("dead-installer",
                "the authorising user is deactivated; a webhook dies with the human "
                "who granted it even though the bot token keeps working")
    known = [str(t) for t in (facts.get("known_teams") or []) if t]
    if known and parts["team"] not in known:
        return ("foreign-workspace",
                "the URL was issued for %s, which is not a workspace this inventory "
                "holds an install for" % parts["team"])
    days = facts.get("last_delivery_days")
    stale = int(facts.get("stale_days") or 30)
    if days is None:
        return ("never-delivered",
                "no app posted into the bound channel anywhere in the sample read, so "
                "this webhook has delivered nothing within that window")
    if int(days) > stale:
        return ("silent",
                "the last app message in the bound channel is %d days old, past the "
                "%d day window" % (int(days), stale))
    return ("live", "an app delivered into the bound channel %d day(s) ago" % int(days))


def inventory_gaps(record):
    """The bookkeeping that makes the next outage slow to find. Pure.

    None of these is an outage today. Every one of them is a reason the outage
    that does happen takes five weeks to attribute to Slack.
    """
    out = []
    if not str(record.get("used_by") or "").strip():
        out.append(("no-owner", "no system is recorded as holding this URL, so a "
                                "rotation has nowhere to land"))
    if not str(record.get("installed_by") or "").strip():
        out.append(("no-installer", "no authorising user is recorded, so the one "
                                    "event that silently kills this URL cannot be "
                                    "watched for"))
    if not str(record.get("channel_id") or "").strip():
        out.append(("no-channel", "no bound channel is recorded, so nothing can be "
                                  "read back to show what was delivered"))
    return out


def duplicate_owners(records):
    """Which URLs are pasted into more than one system? Pure.

    Returns [(redacted, [owner, ...]), ...]. Sharing one URL between systems
    is not wrong in itself, and it is the reason a rotation that looked like a
    one line change took a system down that nobody had connected to it.
    """
    by_url = {}
    order = []
    for r in records or []:
        url = str((r or {}).get("url") or "").strip()
        if not url:
            continue
        owner = str(r.get("used_by") or "").strip() or "unrecorded"
        if url not in by_url:
            by_url[url] = []
            order.append(url)
        if owner not in by_url[url]:
            by_url[url].append(owner)
    return [(redact_webhook_url(u), by_url[u]) for u in order if len(by_url[u]) > 1]


def read(session, method, token, params=None):
    """One GET. Slack answers 200 for failures too, so the body is the answer."""
    r = session.get(API + method, headers={"Authorization": "Bearer " + token},
                    params=params or {}, timeout=30)
    try:
        return r.json()
    except ValueError:
        return {"ok": False, "error": "unparseable_body"}


def main():
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("--inventory", required=True,
                    help="JSON list of rows: url, used_by, installed_by, "
                         "channel_id, install_state")
    ap.add_argument("--token-env", default="SLACK_BOT_TOKEN",
                    help="environment variable holding a read scoped bot token")
    ap.add_argument("--stale-days", type=int, default=30,
                    help="how old the last delivery may be before it is a finding")
    ap.add_argument("--limit", type=int, default=200,
                    help="messages of channel history to read per bound channel")
    ap.add_argument("--now", type=float, default=0.0,
                    help="epoch seconds to measure against, for a repeatable run")
    args = ap.parse_args()

    with open(args.inventory, encoding="utf-8") as fh:
        records = json.load(fh)
    now_ts = args.now or time.time()
    findings = 0

    token = os.environ.get(args.token_env)
    session = requests.Session()
    known_teams = []
    if token:
        who = read(session, "auth.test", token)
        if who.get("ok") is True:
            known_teams = [who.get("team_id")]
            log.info("anchor     %-14s auth.test names the workspace this token "
                     "lives in", who.get("team_id"))
        else:
            log.warning("anchor     unavailable    auth.test answered ok: false, "
                        "error=%s", who.get("error"))
    else:
        log.warning("anchor     skipped        set %s to read the authorising users "
                    "and the bound channels", args.token_env)

    installer_state = {}
    channel_days = {}
    if token:
        for uid in sorted({str(r.get("installed_by") or "").strip()
                           for r in records} - {""}):
            body = read(session, "users.info", token, {"user": uid})
            installer_state[uid] = (bool((body.get("user") or {}).get("deleted"))
                                    if body.get("ok") is True else None)
        for cid in sorted({str(r.get("channel_id") or "").strip()
                           for r in records} - {""}):
            body = read(session, "conversations.history", token,
                        {"channel": cid, "limit": args.limit})
            if body.get("ok") is True:
                channel_days[cid] = last_webhook_delivery(body.get("messages"), now_ts)
            else:
                log.warning("channel    %-14s conversations.history answered %s",
                            cid, body.get("error"))

    for record in records:
        facts = {
            "installer_deleted": installer_state.get(
                str(record.get("installed_by") or "").strip()),
            "last_delivery_days": channel_days.get(
                str(record.get("channel_id") or "").strip()),
            "stale_days": args.stale_days,
            "known_teams": known_teams,
        }
        state, detail = webhook_verdict(record, facts)
        owner = str(record.get("used_by") or "unrecorded")
        emit = log.info if state == "live" else log.warning
        emit("webhook    %-14s %s -> %s", state, owner,
             redact_webhook_url(record.get("url")))
        emit("                          %s", detail)
        if state != "live":
            findings += 1
        for code, why in inventory_gaps(record):
            log.warning("gap        %-14s %s: %s", code, owner, why)
            findings += 1

    for redacted, owners in duplicate_owners(records):
        log.warning("shared     %-14s %s is held by %s", "duplicate-url", redacted,
                    ", ".join(owners))
        findings += 1

    if not findings:
        log.info("verdict    clean          every row resolves to a live install and "
                 "a channel that is receiving")
        return 0
    log.warning("verdict    %d finding(s)", findings)
    log.warning("  repair: a dead webhook cannot be revived; reinstall the app to mint "
                "a new URL and update every system holding the old one")
    log.warning("  repair: move anything that matters to a bot token and "
                "chat.postMessage, which survives the authorising user leaving")
    log.warning("  repair: keep the inventory, with an owner per row, because Slack "
                "has no method that lists where your webhook URLs went")
    log.warning("  note:   nothing was sent to any webhook URL to establish this; a "
                "webhook has no read side and every request to one is a message")
    return 1


if __name__ == "__main__":
    sys.exit(main())
slack-webhook-inventory.mjs
/**
 * Hold an inventory of incoming webhook URLs against the installs behind them.
 *
 * Read only, and one thing is refused outright. An incoming webhook URL exists
 * for exactly one purpose: to have a message sent to it. There is no status
 * endpoint and no "is this alive" call, so the only way to ask Slack directly
 * whether a webhook works is to deliver a message into a real channel in front
 * of real people. Nothing here sends anything to hooks.slack.com.
 *
 * Three reads: auth.test once, users.info once per distinct authorising user,
 * conversations.history once per distinct bound channel. No webhook URL is
 * ever printed in full.
 */
import { readFile } from 'node:fs/promises';

const API = 'https://slack.com/api/';
const HOOK_HOST = 'hooks.slack.com';

// Install states that already mean the webhook is gone. Slack does not revoke
// a webhook on its own: it dies with the install, the authorisation, or the
// human who granted it.
export const DEAD_INSTALL_STATES = ['revoked', 'uninstalled', 'token_revoked',
  'deleted'];

/**
 * Read a webhook URL's shape without contacting anything. Pure.
 * Returns [kind, parts, notes]; services, workflow-trigger, not-a-webhook,
 * unusable. parts never carries the secret segment, only its length.
 */
export function parseWebhookUrl(url) {
  const text = String(url ?? '').trim();
  if (!text) return ['unusable', {}, ['empty']];
  let parsed;
  try {
    parsed = new URL(text.includes('://') ? text : `https://${text}`);
  } catch {
    return ['unusable', {}, ['unparseable']];
  }
  const host = parsed.hostname.toLowerCase();
  if (!host) return ['unusable', {}, ['unparseable']];
  const notes = [];
  if (parsed.protocol.replace(':', '').toLowerCase() !== 'https') {
    notes.push('insecure-scheme');
  }
  const segments = parsed.pathname.split('/').filter(Boolean);
  if (host !== HOOK_HOST) {
    notes.push(host.endsWith('slack.com') ? 'slack-but-not-hooks' : 'foreign-host');
    return ['not-a-webhook', { host }, notes];
  }
  if (segments[0] === 'services') {
    if (segments.length !== 4) {
      notes.push('truncated-path');
      return ['unusable', { host }, notes];
    }
    const [, team, hook, secret] = segments;
    if (!(team.startsWith('T') || team.startsWith('E'))) notes.push('odd-team-segment');
    if (!hook.startsWith('B')) notes.push('odd-hook-segment');
    if (secret.length < 20) notes.push('short-secret');
    return ['services', { host, team, hook, secretLen: secret.length }, notes];
  }
  if (segments[0] === 'triggers' || segments[0] === 'workflows') {
    notes.push('workflow-trigger');
    return ['workflow-trigger', { host, team: segments[1] ?? '' }, notes];
  }
  notes.push('unknown-path');
  return ['not-a-webhook', { host }, notes];
}

/** A form of the URL safe to print, log, paste and screenshot. Pure. */
export function redactWebhookUrl(url) {
  const [kind, parts] = parseWebhookUrl(url);
  if (kind === 'services') {
    return `${parts.host}/services/${parts.team.slice(0, 5)}/`
      + `${parts.hook.slice(0, 5)}/********`;
  }
  if (kind === 'unusable') return '<unusable value>';
  return `<${kind} at ${parts.host ?? 'unknown'}>`;
}

/**
 * Days since an app last posted into this channel. Pure.
 * null means nothing in the sample came from an app at all.
 */
export function lastWebhookDelivery(messages, nowTs) {
  let best = null;
  for (const m of messages ?? []) {
    if (!m || typeof m !== 'object') continue;
    if (!(m.bot_id || m.app_id || m.subtype === 'bot_message')) continue;
    const ts = Number(m.ts ?? 0);
    if (!Number.isFinite(ts) || !ts) continue;
    if (best === null || ts > best) best = ts;
  }
  if (best === null) return null;
  return Math.max(0, Math.floor((Number(nowTs) - best) / 86400));
}

/**
 * One inventory row, judged against what the reads established. Pure.
 * The precedence runs from permanent to circumstantial on purpose.
 */
export function webhookVerdict(record, facts) {
  const [kind, parts] = parseWebhookUrl(record?.url);
  if (kind !== 'services') {
    return ['unusable-url',
      `the stored value is not an incoming webhook URL; it reads as ${kind}`];
  }
  const state = String(record?.install_state ?? '').trim().toLowerCase();
  if (DEAD_INSTALL_STATES.includes(state)) {
    return ['dead-install', `the installation behind this webhook is ${state}, so the `
      + 'URL answers no_service and a reinstall mints a different one'];
  }
  if (facts?.installerDeleted === true) {
    return ['dead-installer', 'the authorising user is deactivated; a webhook dies '
      + 'with the human who granted it even though the bot token keeps working'];
  }
  const known = (facts?.knownTeams ?? []).filter(Boolean).map(String);
  if (known.length && !known.includes(parts.team)) {
    return ['foreign-workspace', `the URL was issued for ${parts.team}, which is not a `
      + 'workspace this inventory holds an install for'];
  }
  const days = facts?.lastDeliveryDays;
  const stale = Number(facts?.staleDays ?? 30);
  if (days === null || days === undefined) {
    return ['never-delivered', 'no app posted into the bound channel anywhere in the '
      + 'sample read, so this webhook has delivered nothing within that window'];
  }
  if (Number(days) > stale) {
    return ['silent', `the last app message in the bound channel is ${Number(days)} `
      + `days old, past the ${stale} day window`];
  }
  return ['live', `an app delivered into the bound channel ${Number(days)} day(s) ago`];
}

/** The bookkeeping that makes the next outage slow to find. Pure. */
export function inventoryGaps(record) {
  const out = [];
  if (!String(record?.used_by ?? '').trim()) {
    out.push(['no-owner', 'no system is recorded as holding this URL, so a rotation '
      + 'has nowhere to land']);
  }
  if (!String(record?.installed_by ?? '').trim()) {
    out.push(['no-installer', 'no authorising user is recorded, so the one event that '
      + 'silently kills this URL cannot be watched for']);
  }
  if (!String(record?.channel_id ?? '').trim()) {
    out.push(['no-channel', 'no bound channel is recorded, so nothing can be read back '
      + 'to show what was delivered']);
  }
  return out;
}

/** Which URLs are pasted into more than one system? Pure. */
export function duplicateOwners(records) {
  const byUrl = new Map();
  for (const r of records ?? []) {
    const url = String(r?.url ?? '').trim();
    if (!url) continue;
    const owner = String(r?.used_by ?? '').trim() || 'unrecorded';
    if (!byUrl.has(url)) byUrl.set(url, []);
    if (!byUrl.get(url).includes(owner)) byUrl.get(url).push(owner);
  }
  return [...byUrl.entries()]
    .filter(([, owners]) => owners.length > 1)
    .map(([url, owners]) => [redactWebhookUrl(url), owners]);
}

function arg(args, name, fallback = '') {
  const i = args.indexOf(name);
  return i === -1 ? fallback : args[i + 1];
}

async function read(method, token, params = {}) {
  const qs = new URLSearchParams(params).toString();
  const r = await fetch(`${API}${method}${qs ? `?${qs}` : ''}`,
    { headers: { Authorization: `Bearer ${token}` } });
  try {
    return await r.json();
  } catch {
    return { ok: false, error: 'unparseable_body' };
  }
}

async function main() {
  const args = process.argv.slice(2);
  const records = JSON.parse(await readFile(arg(args, '--inventory'), 'utf8'));
  const tokenEnv = arg(args, '--token-env', 'SLACK_BOT_TOKEN');
  const staleDays = Number(arg(args, '--stale-days', '30'));
  const limit = arg(args, '--limit', '200');
  const nowTs = Number(arg(args, '--now', '0')) || Date.now() / 1000;
  const token = process.env[tokenEnv];
  let findings = 0;
  let knownTeams = [];

  if (token) {
    const who = await read('auth.test', token);
    if (who.ok === true) {
      knownTeams = [who.team_id];
      console.log(`anchor     ${String(who.team_id).padEnd(14)} auth.test names the `
        + 'workspace this token lives in');
    } else {
      console.warn('anchor     unavailable    auth.test answered ok: false, '
        + `error=${who.error}`);
    }
  } else {
    console.warn(`anchor     skipped        set ${tokenEnv} to read the authorising `
      + 'users and the bound channels');
  }

  const installerState = new Map();
  const channelDays = new Map();
  if (token) {
    const users = [...new Set(records.map((r) => String(r.installed_by ?? '').trim()))]
      .filter(Boolean).sort();
    for (const uid of users) {
      const body = await read('users.info', token, { user: uid });
      installerState.set(uid, body.ok === true ? Boolean(body.user?.deleted) : null);
    }
    const channels = [...new Set(records.map((r) => String(r.channel_id ?? '').trim()))]
      .filter(Boolean).sort();
    for (const cid of channels) {
      const body = await read('conversations.history', token,
        { channel: cid, limit });
      if (body.ok === true) {
        channelDays.set(cid, lastWebhookDelivery(body.messages, nowTs));
      } else {
        console.warn(`channel    ${cid.padEnd(14)} conversations.history answered `
          + `${body.error}`);
      }
    }
  }

  for (const record of records) {
    const uid = String(record.installed_by ?? '').trim();
    const cid = String(record.channel_id ?? '').trim();
    const facts = {
      installerDeleted: installerState.has(uid) ? installerState.get(uid) : undefined,
      lastDeliveryDays: channelDays.has(cid) ? channelDays.get(cid) : undefined,
      staleDays,
      knownTeams,
    };
    const [state, detail] = webhookVerdict(record, facts);
    const owner = String(record.used_by ?? 'unrecorded');
    const emit = state === 'live' ? console.log : console.warn;
    emit(`webhook    ${state.padEnd(14)} ${owner} -> ${redactWebhookUrl(record.url)}`);
    emit(`                          ${detail}`);
    if (state !== 'live') findings += 1;
    for (const [code, why] of inventoryGaps(record)) {
      console.warn(`gap        ${code.padEnd(14)} ${owner}: ${why}`);
      findings += 1;
    }
  }

  for (const [redacted, owners] of duplicateOwners(records)) {
    console.warn(`shared     duplicate-url  ${redacted} is held by ${owners.join(', ')}`);
    findings += 1;
  }

  if (!findings) {
    console.log('verdict    clean          every row resolves to a live install and a '
      + 'channel that is receiving');
    return;
  }
  console.warn(`verdict    ${findings} finding(s)`);
  console.warn('  repair: a dead webhook cannot be revived; reinstall the app to mint a '
    + 'new URL and update every system holding the old one');
  console.warn('  repair: move anything that matters to a bot token and '
    + 'chat.postMessage, which survives the authorising user leaving');
  console.warn('  repair: keep the inventory, with an owner per row, because Slack has '
    + 'no method that lists where your webhook URLs went');
  console.warn('  note:   nothing was sent to any webhook URL to establish this; a '
    + 'webhook has no read side and every request to one is a message');
  process.exitCode = 1;
}

// Only run when invoked directly, so importing this module in the tests does
// not execute main().
if (import.meta.url === `file://${process.argv[1]}`) {
  main().catch((err) => { console.error(err.message); process.exitCode = 2; });
}

Add a test

Every URL in these fixtures is obviously fake and the secret segment is deliberately far too short to be a real one, which the shape reader notices and says so. The assertion that matters most is the redaction test: it asserts the secret string does not appear anywhere in the output of the only function this script prints URLs through, because a diagnostic that leaks the credential it is diagnosing is worse than no diagnostic. After that it is the precedence in webhook_verdict — a revoked install outranks a deactivated authoriser, which outranks a foreign workspace, which outranks silence — and the dead-installer case, which is the whole reason this note exists.

test_slack_webhook_inventory.py
from slack_webhook_inventory import (
    duplicate_owners, inventory_gaps, last_webhook_delivery, parse_webhook_url,
    redact_webhook_url, webhook_verdict,
)

SECRET = "notarealsecret"
HOOK = "https://hooks.slack.com/services/T00FAKE/B00FAKE/" + SECRET
NOW = 1_700_000_000.0
DAY = 86400.0


def row(**kw):
    base = {"url": HOOK, "used_by": "ci/deploy", "installed_by": "U00FAKE",
            "channel_id": "C00FAKE01", "install_state": "active"}
    base.update(kw)
    return base


def facts(**kw):
    base = {"installer_deleted": False, "last_delivery_days": 1, "stale_days": 30,
            "known_teams": ["T00FAKE"]}
    base.update(kw)
    return base


def test_a_real_shaped_webhook_url_is_read_as_services():
    kind, parts, _notes = parse_webhook_url(HOOK)
    assert kind == "services"
    assert parts["team"] == "T00FAKE"
    assert parts["hook"] == "B00FAKE"


def test_the_secret_segment_is_measured_and_never_carried():
    _kind, parts, notes = parse_webhook_url(HOOK)
    assert parts["secret_len"] == len(SECRET)
    assert SECRET not in str(parts)
    assert "short-secret" in notes


def test_redaction_never_reproduces_the_secret():
    redacted = redact_webhook_url(HOOK)
    assert SECRET not in redacted
    assert "hooks.slack.com/services/" in redacted
    assert redacted.endswith("*" * 8)


def test_a_workflow_trigger_url_is_a_different_surface_and_says_so():
    kind, _parts, _notes = parse_webhook_url(
        "https://hooks.slack.com/triggers/T00FAKE/1234/abcd")
    assert kind == "workflow-trigger"


def test_a_web_api_url_filed_in_the_wrong_column_is_not_a_webhook():
    kind, _parts, notes = parse_webhook_url("https://slack.com/api/chat.postMessage")
    assert kind == "not-a-webhook"
    assert "slack-but-not-hooks" in notes


def test_a_truncated_paste_is_unusable_rather_than_dead():
    kind, _parts, notes = parse_webhook_url("https://hooks.slack.com/services/T00FAKE")
    assert kind == "unusable"
    assert "truncated-path" in notes


def test_an_empty_or_unparseable_value_never_reaches_a_verdict():
    assert parse_webhook_url("")[0] == "unusable"
    assert redact_webhook_url("") == "<unusable value>"


def test_the_most_recent_app_message_sets_the_age():
    messages = [{"ts": str(NOW - 3 * DAY), "bot_id": "B00FAKE"},
                {"ts": str(NOW - 9 * DAY), "bot_id": "B00FAKE"}]
    assert last_webhook_delivery(messages, NOW) == 3


def test_messages_from_humans_do_not_count_as_a_delivery():
    messages = [{"ts": str(NOW - DAY), "user": "U00FAKE", "text": "morning"}]
    assert last_webhook_delivery(messages, NOW) is None


def test_an_empty_history_is_none_rather_than_zero():
    assert last_webhook_delivery([], NOW) is None


def test_a_revoked_install_outranks_every_other_explanation():
    state, detail = webhook_verdict(row(install_state="revoked"),
                                    facts(installer_deleted=True,
                                          last_delivery_days=None))
    assert state == "dead-install"
    assert "no_service" in detail


def test_a_deactivated_authoriser_kills_the_url_while_the_bot_token_lives():
    state, detail = webhook_verdict(row(), facts(installer_deleted=True))
    assert state == "dead-installer"
    assert "bot token" in detail


def test_a_url_issued_for_another_workspace_is_named_before_any_silence():
    state, detail = webhook_verdict(row(), facts(known_teams=["T00OTHER"],
                                                 last_delivery_days=None))
    assert state == "foreign-workspace"
    assert "T00FAKE" in detail


def test_a_channel_no_app_has_posted_in_is_never_delivered():
    assert webhook_verdict(row(), facts(last_delivery_days=None))[0] == "never-delivered"


def test_a_delivery_older_than_the_window_is_silent_and_names_the_window():
    state, detail = webhook_verdict(row(), facts(last_delivery_days=45, stale_days=14))
    assert state == "silent"
    assert "14" in detail


def test_a_recent_delivery_into_the_bound_channel_is_live():
    assert webhook_verdict(row(), facts(last_delivery_days=0))[0] == "live"


def test_a_row_holding_something_that_is_not_a_webhook_never_reaches_liveness():
    state, _detail = webhook_verdict(row(url="https://example.com/hook"), facts())
    assert state == "unusable-url"


def test_the_gaps_that_make_the_next_outage_slow_are_each_named():
    codes = [c for c, _w in inventory_gaps({"url": HOOK})]
    assert codes == ["no-owner", "no-installer", "no-channel"]
    assert inventory_gaps(row()) == []


def test_one_url_held_by_two_systems_is_reported_once_and_redacted():
    shared = duplicate_owners([row(used_by="ci/deploy"), row(used_by="ops/cron"),
                               row(url=HOOK + "x", used_by="alerts")])
    assert len(shared) == 1
    redacted, owners = shared[0]
    assert owners == ["ci/deploy", "ops/cron"]
    assert SECRET not in redacted
slack-webhook-inventory.test.mjs
import { test } from 'node:test';
import assert from 'node:assert/strict';
import {
  duplicateOwners, inventoryGaps, lastWebhookDelivery, parseWebhookUrl,
  redactWebhookUrl, webhookVerdict,
} from './slack-webhook-inventory.mjs';

const SECRET = 'notarealsecret';
const HOOK = `https://hooks.slack.com/services/T00FAKE/B00FAKE/${SECRET}`;
const NOW = 1700000000;
const DAY = 86400;

const row = (kw = {}) => ({
  url: HOOK,
  used_by: 'ci/deploy',
  installed_by: 'U00FAKE',
  channel_id: 'C00FAKE01',
  install_state: 'active',
  ...kw,
});

const facts = (kw = {}) => ({
  installerDeleted: false,
  lastDeliveryDays: 1,
  staleDays: 30,
  knownTeams: ['T00FAKE'],
  ...kw,
});

test('a real shaped webhook url is read as services', () => {
  const [kind, parts] = parseWebhookUrl(HOOK);
  assert.equal(kind, 'services');
  assert.equal(parts.team, 'T00FAKE');
  assert.equal(parts.hook, 'B00FAKE');
});

test('the secret segment is measured and never carried', () => {
  const [, parts, notes] = parseWebhookUrl(HOOK);
  assert.equal(parts.secretLen, SECRET.length);
  assert.equal(JSON.stringify(parts).includes(SECRET), false);
  assert.equal(notes.includes('short-secret'), true);
});

test('redaction never reproduces the secret', () => {
  const redacted = redactWebhookUrl(HOOK);
  assert.equal(redacted.includes(SECRET), false);
  assert.equal(redacted.includes('hooks.slack.com/services/'), true);
  assert.equal(redacted.endsWith('********'), true);
});

test('a workflow trigger url is a different surface and says so', () => {
  assert.equal(
    parseWebhookUrl('https://hooks.slack.com/triggers/T00FAKE/1234/abcd')[0],
    'workflow-trigger');
});

test('a web api url filed in the wrong column is not a webhook', () => {
  const [kind, , notes] = parseWebhookUrl('https://slack.com/api/chat.postMessage');
  assert.equal(kind, 'not-a-webhook');
  assert.equal(notes.includes('slack-but-not-hooks'), true);
});

test('a truncated paste is unusable rather than dead', () => {
  const [kind, , notes] = parseWebhookUrl('https://hooks.slack.com/services/T00FAKE');
  assert.equal(kind, 'unusable');
  assert.equal(notes.includes('truncated-path'), true);
});

test('an empty or unparseable value never reaches a verdict', () => {
  assert.equal(parseWebhookUrl('')[0], 'unusable');
  assert.equal(redactWebhookUrl(''), '<unusable value>');
});

test('the most recent app message sets the age', () => {
  const messages = [{ ts: String(NOW - 3 * DAY), bot_id: 'B00FAKE' },
    { ts: String(NOW - 9 * DAY), bot_id: 'B00FAKE' }];
  assert.equal(lastWebhookDelivery(messages, NOW), 3);
});

test('messages from humans do not count as a delivery', () => {
  const messages = [{ ts: String(NOW - DAY), user: 'U00FAKE', text: 'morning' }];
  assert.equal(lastWebhookDelivery(messages, NOW), null);
});

test('an empty history is null rather than zero', () => {
  assert.equal(lastWebhookDelivery([], NOW), null);
});

test('a revoked install outranks every other explanation', () => {
  const [state, detail] = webhookVerdict(row({ install_state: 'revoked' }),
    facts({ installerDeleted: true, lastDeliveryDays: null }));
  assert.equal(state, 'dead-install');
  assert.equal(detail.includes('no_service'), true);
});

test('a deactivated authoriser kills the url while the bot token lives', () => {
  const [state, detail] = webhookVerdict(row(), facts({ installerDeleted: true }));
  assert.equal(state, 'dead-installer');
  assert.equal(detail.includes('bot token'), true);
});

test('a url issued for another workspace is named before any silence', () => {
  const [state, detail] = webhookVerdict(row(),
    facts({ knownTeams: ['T00OTHER'], lastDeliveryDays: null }));
  assert.equal(state, 'foreign-workspace');
  assert.equal(detail.includes('T00FAKE'), true);
});

test('a channel no app has posted in is never delivered', () => {
  assert.equal(webhookVerdict(row(), facts({ lastDeliveryDays: null }))[0],
    'never-delivered');
});

test('a delivery older than the window is silent and names the window', () => {
  const [state, detail] = webhookVerdict(row(),
    facts({ lastDeliveryDays: 45, staleDays: 14 }));
  assert.equal(state, 'silent');
  assert.equal(detail.includes('14'), true);
});

test('a recent delivery into the bound channel is live', () => {
  assert.equal(webhookVerdict(row(), facts({ lastDeliveryDays: 0 }))[0], 'live');
});

test('a row holding something that is not a webhook never reaches liveness', () => {
  assert.equal(webhookVerdict(row({ url: 'https://example.com/hook' }), facts())[0],
    'unusable-url');
});

test('the gaps that make the next outage slow are each named', () => {
  assert.deepEqual(inventoryGaps({ url: HOOK }).map(([c]) => c),
    ['no-owner', 'no-installer', 'no-channel']);
  assert.deepEqual(inventoryGaps(row()), []);
});

test('one url held by two systems is reported once and redacted', () => {
  const shared = duplicateOwners([row({ used_by: 'ci/deploy' }),
    row({ used_by: 'ops/cron' }), row({ url: `${HOOK}x`, used_by: 'alerts' })]);
  assert.equal(shared.length, 1);
  const [redacted, owners] = shared[0];
  assert.deepEqual(owners, ['ci/deploy', 'ops/cron']);
  assert.equal(redacted.includes(SECRET), false);
});

FAQ

Can I not just send a test message to the URL and see what comes back?

You can, and it is the one thing this note asks you not to do. A webhook URL has no read side at all: no status endpoint, no HEAD, no metadata call. Every request Slack accepts is a message delivered into a real channel, so a probe is a write with an audience, and somebody then has to explain the test message to the people who saw it. Sending an empty body does not help either, because it is refused with no_text and tells you nothing you did not already know. Everything the probe would establish is available from the URL's shape, your install records, users.info on the authorising user, and the bound channel's own history.

Our app is installed and the bot token works. How can the webhook be dead?

Because they have different lifetimes. A bot token belongs to the installation and survives the person who created it; an incoming webhook belongs to one authorisation by one human. Deactivate that human and the webhook dies while auth.test keeps answering ok: true, chat.postMessage keeps working and every dashboard you have stays green. That is the dead-installer verdict, and it is the reason this check reads users.info on the authoriser rather than reading the health of the app.

Will reinstalling the app bring the old URL back?

No. A reinstall mints a new webhook URL, and the old one stays dead forever. That is why the repair is not a one line change: every system holding the old URL has to be updated, which is exactly the moment teams discover they do not know how many systems that is. Keeping an inventory with an owner per row is the cheap part of this note and the part that pays for itself.

What is the difference between no_service, no_active_hooks and invalid_token?

They are three ways of saying the URL will never work again, and the distinction matters less than people hope. no_service and no_active_hooks come back as 404 and mean the webhook behind this URL no longer exists: uninstalled, revoked, deleted or authorised by someone who is gone. invalid_token comes back as 401 and usually means the URL itself is wrong rather than dead, which in practice means truncated by a config file, re-wrapped by an email client, or copied with a trailing character. The first two are fixed by reinstalling; the third is fixed by checking what you stored, which is what the shape reader here does before anything else.

A webhook nobody uses any more is harmless, surely?

It is the opposite of harmless. A dead webhook is an outage that has already happened. A live webhook that nothing uses is a standing bearer credential sitting in a config file, a wiki page or an old repository, and anyone who finds it can post into your workspace as your app, indefinitely, with no audit trail pointing at them. That is why the no-owner gap is reported as a finding rather than as a note: a URL nobody claims is a URL nobody will revoke.

Related field notes

Sources

Every figure in this note is traced to one of these. Prices are list rates and change — check them for your own region before acting.

Stuck on a tricky one?

If your setup is misbehaving in a way this note does not cover, message me on LinkedIn with what you are seeing.