Skip to content

Runbooks

Runbooks for the credentials the engine holds, what the Sentinel App needs on the GitHub side, switching issue filing on, and how to read what the pipeline reports about itself. Deploying the receiver itself is covered in infra/README.md; the one-time Microsoft and Sentry setup lives in Microsoft setup and Sentry setup.

Credential Where it lives Purpose
/sentinel/github-app-private-key SSM SecureString, receiver The Sentinel App’s private key, read by the receiver at cold start; it mints the scoped installation token the receiver uses to open each pull request and to make each issue write
SENTRY_WEBHOOK_SECRET, SENTRY_API_TOKEN, BOT_CLIENT_SECRET, ROUTINE_TRIGGER_TOKEN SSM SecureStrings, receiver Webhook verification, Sentry API access, the bot identity, and the routine fire token
Installation tokens Nowhere: minted per pull request or per issue operation inside the receiver, spent in the same invocation, never returned or stored Lets the receiver push one branch and open one pull request (contents: write and pull_requests: write), or make one issue operation’s write, downscoped to that operation (OP_PERMISSIONS); scoped to the target repo, one hour
Callback capability tokens DynamoDB, hashed, one per dispatch Bearer credential the fix session presents to /autofix-result; scoped to a single dispatch, not reusable
Tracking tokens DynamoDB, hashed, one per delivered result while issue filing is on for its environment Bearer credential a session presents to /issues; scoped to one alert and its thread, dead at its grant’s expires_at (30 minutes after findings, or 90 when the result also earned an autofix grant)
The operator’s GitHub connection The cloud platform, outside the session; never in this repository or the parameter store Authenticates the session’s repository fetch and its issue reads through the platform’s built-in GitHub tools; the routine’s tool policy withholds every write tool, so it is used for reads only
/sentinel/ntfy-topic SSM SecureString, alarm forwarder (optional) The ntfy topic alarms are published to, read only by the forwarder and only when aws.alarm_ntfy_server is set. On a public ntfy server the topic name is the only access control, so it is held like a credential
SENTINEL_BANNED_PATTERNS Actions secret, engine repo The deployment-identifying ban patterns tests/test_public_tree.py enforces; not a credential, but held like one because publishing the list would index what it protects

The routine trigger token is never stored in this repo; keep the only other copy in your password manager.

The receiver mints every GitHub credential itself, for autofix pull requests and for issue writes alike, so the App’s configuration is the only thing standing between a passing gate and a pull request, or between a session’s issue operation and the repository:

  • The App must be installed on the target repository, the one named by target_repo. Minting looks that installation up by repository, so an App installed on the organization but not on this repository fails at the lookup.
  • For autofix, the installation must grant contents: write and pull requests: write. The receiver downscopes each fix’s token to exactly those two, and a token can never hold a permission the installation lacks, so a missing one produces a token that cannot push or cannot open the pull request.
  • For issue filing, the installation must also grant Issues: read and write (create, comment, reopen, labels, the issue type on create, and issue-field values), organization Projects: read and write (adding a new issue to the board issues.project_number names), and organization Issue fields: read-only (the field and option names set_fields resolves). Each issue operation mints its own token, downscoped to what that operation needs (OP_PERMISSIONS in receiver/github_app.py), and none carries contents, pull requests, workflows or administration. Organization permissions reach every board in the organization; the named operations are what confine the App’s board writes to the one configured.
  • GitHub holds a permission change until an organization owner accepts it on the installation. Until then the installation keeps its old permissions, autofix is unaffected, and every issue operation fails at the mint with a 502.
  • Do not grant workflows. It is excluded from every minted token on purpose (why), and granting it at the installation widens the App’s own reach for no benefit here.
  • autofix.app_id in the config file must be the App’s numeric id, and /sentinel/github-app-private-key must be one of that App’s active private keys. Issue filing uses the same App and key, even with autofix off.

The routine’s environment reaches GitHub for reads only

Section titled “The routine’s environment reaches GitHub for reads only”

The session reaches GitHub three ways, and all three are reads: the investigation’s repository fetch, the fix phase’s fetch of the base branch, and the issue step’s reads through the platform’s built-in GitHub tools. It sends its findings, its fix, and its issue operations to the receiver, and the receiver makes every GitHub write. So the routine’s environment needs the same GitHub reach the investigation already has, the repository fetch, and no more: the tool reads need no egress change, because the built-in tools reach GitHub through the platform’s own GitHub proxy, the path the repository fetch already takes. If fixes are failing while findings keep arriving, the pattern points at the receiver’s side, the App configuration above or the AUTOFIX_FAILED lines below, before it points at egress.

The cloud platform gives every routine session a built-in GitHub server whose tools act as the operator. The investigator uses one of them, issue_read, to open the target repository’s issues. It does not search with that server: the platform removes the server’s search tools from routine sessions and points them at gh api instead, which carries the same operator credential and can write as easily as it reads. The investigator searches through the receiver’s search operation, which runs as the App with a read-only token. Every other tool of the server, search_issues and every tool that writes among them, is withheld by the routine’s own tool settings. prompts/routine-tools.yaml is the canonical copy of those settings: read_allowlist holds the one read, and disallowed_tools holds every other tool of the server, each with the mcp__github__ prefix. tests/test_routine_tools.py keeps the list sorted, disjoint from the read, and complete against a floor of write tools and the server’s issue searches, and fails if the investigator prompt names a withheld tool.

Both routines carry the same policy: the investigator, and the probe routine recorded as trigger.probe_routine_id. That keeps a probe run a faithful test of the investigator’s settings.

Applying it. The public routines API cannot read or write a routine’s tool settings. Apply them through routine tooling (a Claude Code session’s routine get and update), or on the routine’s page at claude.ai when it shows both fields. Either way:

  1. Read the routine.
  2. Set job_config.ccr.session_context.disallowed_tools to the file’s disallowed_tools, and append the read, prefixed (mcp__github__issue_read), to job_config.ccr.session_context.allowed_tools, keeping every entry already there except mcp__github__search_issues, which earlier versions of the file allowed and which must now come out. Allowing the read outright keeps it from waiting on the session’s permission checker mid-run.
  3. Send the whole job_config back. The update treats job_config as one field, so a fragment can replace the stored prompt with nothing.
  4. Read the routine again and compare both tool lists and the stored prompt with the files. Any difference is a partial update: apply again.

Then do the same for the probe routine.

Regenerating the list. Step 5 of prompts/probe.md lists every GitHub tool the probe’s session can see. A name beyond issue_read is a tool the list does not know about: add it to disallowed_tools in sorted order, fold in any new names from the upstream github-mcp-server tool catalog at the same time, run the tests, apply the file to both routines, and probe again until the probe sees only that read. Re-run the probe whenever either routine’s prompt or tool settings change.

Reading the probe. Read steps 4 to 6 of the probe’s transcript against this table, and the receiver’s log for the /findings/probe line.

The probe shows Verdict Next
4b: the receiver logged the POST, and the transcript’s repository and commit match it The investigator’s delivery path works from this environment Pass
4b: the transcript shows the POST attempted and failing, and the receiver logged nothing Egress to the receiver is blocked Check the environment’s allowed hosts
5a lists exactly issue_read, and 5c made no call The list applies Pass
5a lists add_issue_comment, and the session’s permission settings refused 5c The list applies at call time, not at listing Pass, and record that it applied at call time
5a lists names beyond issue_read, but not add_issue_comment The list applies; the extras are tools it lacks Add them, apply again, probe again
5c got a response from GitHub (404 or 422) The list does not apply to the built-in tools The fallback below
5c was refused only by another safety check Unproven: that check is judgment, not configuration The fallback below
5b: issue_read was refused The read path is broken Keep issue filing off: the session could find candidates but never open one to judge it
6c: either token variable looks like a real credential The session holds a GitHub credential it is meant not to have Remove it from the environment’s variables
6d: git ls-remote failed while 6b reached GitHub The fetch the fix phase bases a fix on is broken Keep autofix off until it reads
Any step declined on the session’s own judgment That step is unproven, not failed Read the stated reason and probe again; reword the step if the reason points at the prompt

When the list cannot be set or does not apply. Issue filing still ships, on the prompt’s prohibition, which forbids every GitHub write whatever tool would make it. Keep the list configured wherever the routine accepts it, keep it in prompts/routine-tools.yaml, and apply it again at every prompt update: it is harmless while inert, takes effect the day the platform applies it, and every probe checks it again. The security model says so plainly until a probe passes.

A tool the platform adds between probes is not in the list until the next probe finds it. The prompt’s prohibition covers that tool meanwhile, which is prose rather than configuration.

Runbook: rotating the GitHub App private key

Section titled “Runbook: rotating the GitHub App private key”
  1. Generate a second private key in the GitHub App settings. GitHub Apps allow two active keys at once, so this rotation is zero-downtime: the old key keeps authenticating while the new one rolls out.
  2. Update the key in both places it lives. Set AWS_REGION to the region you deployed into first; scripts/put-parameters.sh and the receiver both default to us-east-1, but nothing requires it.
    • Your password manager: update the stored copy manually.
    • SSM parameter (read by the receiver at cold start). scripts/put-parameters.sh writes all five receiver secrets together and its put() helper hard-exits the moment any of them is unset, so it is not a safe tool for rotating this one key alone unless you also have the other four ready to supply in the same run. Update the parameter directly instead:
      Terminal window
      aws ssm put-parameter \
      --name /sentinel/github-app-private-key \
      --type SecureString \
      --value "$GITHUB_APP_PRIVATE_KEY" \
      --overwrite \
      --region "$AWS_REGION"
  3. Confirm the new key is actually the one in place, not just that GitHub still accepts requests. Both keys stay valid until you delete one, so a successful mint proves nothing about which key produced it:
    • SSM: aws ssm get-parameter --name /sentinel/github-app-private-key --with-decryption --region "$AWS_REGION" --query Parameter.Value --output text, and diff the output against the new key file.
    • The receiver caches its App client per container, so a warm Lambda keeps using the key it read at cold start. Wait for a cold start, or force one with a no-op config deploy, before reading anything into the next autofix outcome.
  4. Only once both checks in step 3 pass, delete the old key from the App settings.

Two gates decide what this repository is willing to hand a stranger, and they cover different things.

tests/test_public_tree.py bans specific strings, and gitleaks (secret-scan.yml) matches credential shapes. Both catch what someone already thought to describe, both are deterministic, and both run locally before anything is pushed for any clone that ran pre-commit install (see CONTRIBUTING.md). On a public repository every pushed branch is published, so that local hook is the only gate that fires before content becomes public; CI is the backstop for clones without it.

Nothing automated reads a change for prose that describes a deployment without using a listed string. That is the author’s to catch, by reading the diff before pushing, and the security model lists what to read for.

The banned strings themselves are split in two. Generic shapes live in the test file; the literals that identify a real deployment live in an untracked local file, config/banned.local.txt, because a tracked ban list is an index of everything it protects. CI materializes that file from the SENTINEL_BANNED_PATTERNS secret and sets SENTINEL_REQUIRE_BANNED_LOCAL, so a vanished secret fails the suite rather than quietly narrowing the scan. After editing the local file, mirror it back:

Terminal window
gh secret set SENTINEL_BANNED_PATTERNS < config/banned.local.txt

A fork’s pull request runs without the secret and tests the generic patterns only, which is the same posture a stranger’s clone gets: the push to the default branch after the merge is where the full list applies.

Every way the pipeline can fail at its job raises a CloudWatch alarm on the stack’s alarm topic, which emails aws.alarm_email and, when configured, is forwarded to ntfy. Each alarm fires on a single occurrence in five minutes, and each log-derived one counts a marker the receiver writes at the point of failure, so the receiver’s log group, filtered on the marker, shows which alert it was.

Alarm Marker What failed Where to look
ReceiverErrors (the Lambda Errors metric) The receiver raised. Teams alerts may be silent The receiver’s log group around the alarm time
ReceiverDeliveryFailures ALERT_DELIVERY_FAILED A card or a thread reply could not be posted to Teams The line names the issue and what was being posted
ReceiverPayloadUnusable PAYLOAD_UNUSABLE Sentry sent a correctly signed alert the receiver could not parse. A payload shape change fails every alert this way The line carries the parse error; compare it with Sentry’s current webhook payload
ReceiverEnqueueFailed ENQUEUE_FAILED An alert card posted, but its investigation row was never written, so nothing will investigate it The line names the issue; replay the alert once the cause is fixed
ReceiverFireRejected FIRE_REJECTED The routine API refused to start a session (or the payload was over its cap), and the batch’s rows were marked failed The FIRE_UNCLASSIFIED line just before it carries the API’s response
ReceiverFireStalled FIRE_STALLED Fires have failed with a retryable error (a 5xx, or the API unreachable) for over an hour, so nothing is being investigated. Logged once per unbroken run of failures The transient fire failure warnings since the time the line names
ReceiverWindowExhausted SUBSCRIPTION_WINDOW_EXHAUSTED The Claude subscription’s usage window is spent, so investigations are being skipped Usage on the subscription; nothing fires until the window resets
ReceiverInvestigationTimedOut INVESTIGATION_TIMED_OUT A fired session never reported before investigation.deadline_seconds, and the thread got the fallback reply The session’s own transcript, from the routine’s run history
ReceiverInvestigationFailed INVESTIGATION_FAILED A session reported failed for an issue, with its reason on the line. A partial result is not under it The reason on the line, then the session’s transcript
ReceiverFindingsRejected FINDINGS_REJECTED A findings document failed validation: the stored prompt and the receiver disagree Updating the investigator routine
ReceiverAutofixFailures AUTOFIX_FAILED A fix attempt failed or never called back Reading autofix in the logs
ReceiverIssueTrackingFailures ISSUE_TRACKING_FAILED Filing, matching or searching issues failed, or an alert was left untracked Reading issue tracking in the logs
AlarmForwarderErrors (the forwarder’s Errors metric) The forwarder could not publish to ntfy. The email still arrives The forwarder’s own log group

Deliberate holds are never alarmed: a paused routine, the daily fire cap, and a 429 that names a short Retry-After (the platform’s daily run cap) are each the operator’s budget at work, and an alarm on them would teach people to ignore the rest. Construct ids carry a generated suffix after the names above.

The alarm email is easy to miss. With aws.alarm_ntfy_server set (see Configure and deploy), the stack adds a small forwarder function subscribed to the alarm topic, which publishes every alarm, as it moves into ALARM, to an ntfy topic at high priority, tagged sentinel. Anything subscribed to the topic receives it: the ntfy phone app, or a desktop subscriber that raises its own alert.

Each message carries the alarm’s description as its title and body, its name, and when it fired. Nothing else: no Sentry data, no thread or issue link.

To set it up:

  1. Generate a topic name that cannot be guessed, for example openssl rand -hex 16. On a public ntfy server the name is the only access control: anyone who has it can read the messages and publish to the topic.
  2. Write it to /sentinel/ntfy-topic by running scripts/put-parameters.sh with NTFY_TOPIC set alongside the other secrets, or with a single aws ssm put-parameter --type SecureString.
  3. Set aws.alarm_ntfy_server and deploy.
  4. Subscribe your devices to the topic, then test the path by setting one alarm’s state by hand: aws cloudwatch set-alarm-state --alarm-name <name> --state-value ALARM --state-reason "forwarder test". The next evaluation returns it to OK.

To rotate the topic, generate a new one, write it to the parameter, and move every subscriber to it. The forwarder reads the parameter once per container, so a new topic is picked up within minutes as containers recycle; redeploying picks it up at once. To turn forwarding off, remove aws.alarm_ntfy_server and deploy.

These are the three markers to filter on. They cover the gate’s decisions and every outcome that reaches a thread, but they are not everything autofix logs: see the unmarked lines below the table.

Marker Means
AUTOFIX_DECLINED The gate said no, with the reason on the same line. Not a fault: this is the normal outcome for most findings
AUTOFIX_DISPATCHED A grant was staged, with its dispatch id
AUTOFIX_FAILED Something went wrong: a crashed gate, a session reporting failed, a fix_ready payload that broke a rule, a pull request the receiver could not open, or a deadline passing with nothing settled. The rest of the line says which

A filter built from the table above will miss several things, and one of them matters operationally. Rejections at /autofix-result are unmarked warnings: a request with no usable bearer token, one naming an unknown dispatch or presenting the wrong token, and one carrying a status outside the accepted set each log a plain rejected autofix callback ... warning. So repeated authentication failures against the callback endpoint, which is the shape a probe of that endpoint would take, produce nothing any of the four markers would catch. Search the callback route’s own wording, not the markers, when that is the question.

Also unmarked: a callback arriving for a dispatch the sweep already settled logs an already settled; ignoring notice at info, which is normal and means the deadline won a race. And a completion reply that cannot be posted to the thread carries the receiver’s general delivery-failure marker rather than an autofix one, so a chat outage during an autofix reply is found under that marker. The App client’s own failures log with their own wording too, each naming the underlying error. The App client’s token mint and pull-request sequence both name the repository they were working against.

AUTOFIX_FAILED ... session reported failure

Section titled “AUTOFIX_FAILED ... session reported failure”

The session stopped the grant itself and reported failed. That callback carries no reason, so the line ends there, and the reason lives only in the session’s transcript: find the routine run that started just before the matching AUTOFIX_DISPATCHED line and read how its fix phase ended.

The usual cause is step 8d’s test. The session wrote a fix but could not run its test, either because the routine’s environment lacks a dependency or because the session never found where the environment installed it. The first means making the target repository install it, through a SessionStart hook in that repository or the environment’s setup script. The second is what the prompt’s instruction to search for installed dependencies covers, so a transcript showing a session that gave up without searching is a reason to make the environment put its dependencies where a session looks first, such as on PATH. Once the cause is fixed, replay the alert with --reset (see the replay runbook below) to get the pull request.

The session sent a fix_ready body that broke one of the receiver’s rules, and the line names the rule: both or neither of files and patch, a path under .github/ or matching autofix.exclude_paths (on either side of a rename), a path that is not plain and repository-relative, more than twenty files, a symlink or submodule mode, more than 512 KB of whole-file content or a patch over 4 MB, a patch that is not plain git diff output (the line says where it stopped reading), a title that is empty or wraps, or a base commit that is not a full SHA. The record is settled as failed, the thread has been told, and no GitHub call was made.

The one worth acting on is a path rejection, because it means the session decided a fix needed a file the operator ruled out. Read the transcript to see whether the exclusion is right or the finding was steered; either way the pull request did not happen, which is the design.

The payload passed and the record was claimed, but the base tree refused the fix, and the line names the file and the reason. It is the session’s fix that is wrong here, not the GitHub App, so nothing on the App checklist applies:

  • differs at base_sha from the version the patch was made against, does not match base_sha at line N, or does not reproduce blob ...: the patch was not taken against the base commit it names. The session diffed against some other commit, or edited the patch by hand. The receiver applies hunks only where they say and checks every result against git’s own blob id, so this never lands as a near miss.
  • does not exist at base_sha or already exists at base_sha: the same mismatch, seen as a path the patch edits that is not there, or creates that already is.
  • is a symlink at base_sha, is a submodule at base_sha, is a directory at base_sha, or runs through ... which is not a directory: the fix would replace or write through something that is not a regular file. A pure rename carries no mode in a patch, so only the base tree can catch it.
  • more than the ... bytes the receiver rewrites: the files the patch modifies are larger in total than the receiver will read and upload in one invocation. GitHub builds blobs only from whole content, so a small change to a large file still costs the file’s full size here.

No branch was created for any of these. The session’s response carried the same reason, and the transcript shows how it made the patch.

AUTOFIX_FAILED ... pull request not opened

Section titled “AUTOFIX_FAILED ... pull request not opened”

The payload passed, the record was claimed, and one of the GitHub calls that build the branch and open the pull request did not succeed. The App client’s own line, just above, names the failing call and the underlying error. Work through the App checklist above: the wrong autofix.app_id, a private key that no longer matches an active key, an App not installed on the target repository, or an installation missing one of the two permissions all land here. A branch that already exists (a replayed dispatch id) lands here too, as does a base commit that is not on the base branch (the session checked out something other than the branch it was told to; the App client’s line says so) and a sequence that ran out of its time budget.

A failure after the branch was created leaves that branch behind with no pull request on it. It is inert, and its name carries the dispatch id that failed:

Terminal window
git ls-remote --heads origin 'autofix/*'

lists them, and once you have confirmed no pull request references a branch, git push origin --delete <branch> removes it. The receiver never deletes branches itself; that is a destructive path it does not need.

The receiver claimed a record and never settled it, which means the invocation died between its first GitHub call and its last. The sweep fails the record after three minutes and the thread reads that the receiver did not finish opening the pull request. Check the receiver’s own error log for the invocation, then the branch listing above, since the branch may exist without its pull request.

If a batch instead reports a crash before any of this, the cause is usually the App private key parameter being unreadable at cold start rather than GitHub itself.

Issue filing logs one marker, ISSUE_TRACKING_FAILED, and the IssueTrackingFailures alarm fires on a single line of it in five minutes. The rest of the line names the alert and the case:

Line Means
... <short id> grant not written: <error> The receiver could not write a tracking grant while delivering findings. The findings reply still posted, but the session got no grant for that result, so nothing can file it
... <short id> issue op rejected: <rule> A session sent an operation that broke one of the proxy’s rules, named on the line. Like FINDINGS_REJECTED, it means the prompt and the receiver disagree
... <short id> <op> failed: <step> A GitHub call did not succeed, and the step says which: token mint failed, read the issue, list the labels, create the issue, post the comment, reopen the issue, read the issue fields, field <name> is not single-select (a configured field of another type), set the field values, read the field values back, field values did not persist, read the project, add the issue to the project, or search the issues (for a search, a rate limit included). The session may retry once
... <short id> <op> rejected: <reason> GitHub refused the request and wrote nothing (GitHub refused the issue as sent, usually an issue type it does not recognize), or set_fields named an option no field offers
... <short id> <op> #<n> landed with warnings: <what> The write happened, but a label or the type is not on the issue as sent: a label the repository does not define, dropped before sending, or one GitHub dropped on its own
... <short id> <op> crashed: <error> The receiver raised before the operation’s GitHub write returned, so nothing was written. The operation’s record is failed and the alert’s tracking claim released, so a retry can run
... <short id> <op> crashed after its GitHub write landed: <error> The write happened, then the receiver raised while recording it. The record and the claim are left as they are, so the write is never repeated: a retry is answered from a record that settled done, and finishes marking the alert tracked and posting its link if the crash came first; one that did not settle is refused as in progress. If the alert never shows as tracked, the backstop’s notice at expiry says filing may not have finished
... <short id> <op> landed on #<n> after the grant closed An operation finished just after the backstop expired its grant. The issue exists, and the thread got its link after the backstop’s notice
... <short id> left untracked at grant expiry The backstop: the grant expired with no issue filed, commented on or reopened
... <short id> may not have finished filing at grant expiry The backstop found an operation that claimed the alert and never finished marking it tracked (it died mid-operation, or crashed after its write and was never retried), so an issue may exist

Refusals carry no marker. A well-formed operation that a cap or a state forbids is answered 409 and logged at warning as issue op for <short id> refused: <reason>: a per-alert or daily cap reached, the alert already tracked, an issue that is not in the target repository or is a pull request, a reopen of an open issue, an operation still in progress, or an environment no longer in issues.environments. If a refusal leaves the alert untracked, the backstop logs the marker once, when the grant expires. Authentication failures at /issues are plain rejected issue op ... warnings, the same shape as the callback route’s. A search whose terms GitHub could not parse is answered 400 and logged at warning as issue search for <short id> refused by GitHub; the session rewords it.

Checking that sessions search. Every search that worked logs issue search for <short id> found <n> issues at info. A filed alert with no such line before its file_issue never searched, and a file_issue carrying search_unavailable true after such a line means the session judged every search failed when one worked. To prove the whole path, replay an alert that already has a tracking issue (a repeat of the same Sentry issue at a new release, sent as replaying an alert describes): the session should log at least two searches, and its tracking outcome should be a duplicate comment on the existing issue, not a new one.

The two backstop notices are what the alert’s thread reads when its grant expires with nothing tracked: No GitHub issue was filed for this alert, so it is tracked only in this thread., or, when an operation died mid-filing, Sentinel may not have finished filing this alert in GitHub; check the repository for an issue before filing one. Either is followed by The duplicate search was also unavailable for this alert, so an existing issue may already describe it. when the session’s duplicate search had failed. A grant whose result opened a pull request, or whose fix attempt failed on the receiver’s side, is released without a notice: the pull request, or the AUTOFIX_FAILED line and its own thread reply, already says what happened.

A thread reply or a notice that cannot be posted carries the receiver’s general delivery-failure marker (issue tracking reply for ..., tracking notice for ...), not this one. A grant with no thread to reply in (its alert card never posted) skips the reply with a plain no thread to reply in for ... warning, and the backstop skips its notice the same way (no thread to post the tracking notice in for ...), still logging the marker.

Runbook: replaying an alert the receiver has already seen

Section titled “Runbook: replaying an alert the receiver has already seen”

A repeat alert on an unchanged release finds the investigation row and stops before any Sentry call: it posts a pointer to the earlier thread, or retries once if that attempt failed, and never re-runs a delivered investigation. Which repeats arrive at all depends on the alert rules you configured: a new-issue rule fires once per issue per environment, while frequency, regression, and reappearance rules re-fire for a known issue. So when you need the receiver to process a known issue again, after fixing something in the pipeline for example, sign a webhook body and post it yourself:

Terminal window
python -m scripts.replay_alert --issue 1000000007 --environment prod --dry-run

The body is fixtures/sentry-webhook-alert.json with the target issue’s fields written over it from its stored alert:<environment> row, so you can only replay an issue the receiver has already posted. The signature comes from receiver.handler.sign_body, the same function /sentry verifies against, and the webhook secret is read from SSM rather than the environment so it never lands in shell history.

Start with --dry-run. It prints the exact body and target and sends nothing. A live replay posts a real card to that environment’s real Teams channel.

Two rows make a replay a no-op for a repeat, and --reset deletes both before posting:

  • investigation:<env>#<release> stops the enqueue, so no session fires and no findings arrive.
  • autofix:<env>#<release> stops the autofix gate one stage later, as “already attempted for this release”.

Each delete is conditional on the row existing, and the script prints deleted: or absent: per row. That distinction is the point: an unconditional delete-item against a mistyped sort key succeeds silently and reads as a successful clear. The alert:<environment> row is never touched, because it is the replay’s own input.

Without --reset, a repeat issue posts a fresh card that points at the earlier thread, or retries a failed investigation if its retry budget is unspent. That is the right mode for a freshly staged error that has no rows yet.

Runbook: updating the investigator routine

Section titled “Runbook: updating the investigator routine”

A routine holds one prompt and its own tool settings, and the receiver’s contract with the prompt changes whenever steps 6 to 8 of prompts/investigator.md do. The public routines API cannot read or write a routine’s configuration, so the update happens through routine tooling (a Claude Code session’s routine get and update) or on the routine’s own page at claude.ai:

  1. Substitute both placeholders in the repository file: the receiver’s origin for REPLACE_WITH_FUNCTION_URL, and the paths in issues.conventions, separated by a comma and a space, for REPLACE_WITH_ISSUE_CONVENTIONS (the word none when that list is empty).
  2. Apply the substituted prompt and the tool policy from prompts/routine-tools.yaml together, as the tool policy section describes.
  3. Read the routine back and compare its prompt with the substituted file and both tool lists with prompts/routine-tools.yaml. A prompt update sends the whole job config again, so the read-back covers the two tool lists as well as the prompt. Any difference is a partial update: apply again.
  4. Write down the engine commit the prompt came from, the UTC date, and whether the read-back matched. Nothing compares the routine’s copy with the repository on a schedule, so that record is how a later reader learns which commit the routine carries. Every later prompt or tool-list change repeats it.

After an update, a FINDINGS_SCHEMA_OUTDATED schema_version <n> warning in the receiver’s log group means a session posted findings in a schema older than the one prompts/investigator.md emits, so it ran an older prompt. The receiver delivers those findings as usual, without the status-line entries their schema lacks, so no finding is lost, and the marker is counted rather than alarmed. What an older schema does cost is autofix: a document from before intent was rated (v2 or v3) fails the gate closed with the reason intent not rated, so a routine on an older prompt earns no fix attempts until it is updated. It is expected only for sessions that were already running when the prompt was swapped.

When a change adds a findings schema version, or an /issues operation the prompt calls, deploy the receiver before updating the prompt. A receiver that does not know an operation rejects it as a rule break, which for the search means every alert is filed as unsearched. A receiver that does not know the new version rejects the whole document, which leaves the batch for its deadline to fail into the thread; a receiver that knows it accepts the older version the routine still sends in the meantime. Once those have finished, a new one means the routine still holds an older prompt: compare its prompt against the substituted file again.

The conventions paths live in the pasted prompt, not in the receiver. Moving or renaming a conventions file in the target repository therefore takes two steps: change issues.conventions, then update the routine again.

The probe routine keeps its own prompt, prompts/probe.md, with only REPLACE_WITH_FUNCTION_URL to substitute, and it carries the same tool policy as the investigator, so a tool-policy change is applied and read back on both routines.

The prompts in prompts/ are templates. REPLACE_WITH_FUNCTION_URL stands in for the receiver’s Function URL and must be substituted with the real origin when the prompt is written to the routine. The session cross-checks that origin against the SENTINEL_RECEIVER_URL environment variable, so a prompt written with the placeholder left in place fails the check and posts nothing. The investigator prompt also carries REPLACE_WITH_ISSUE_CONVENTIONS, which stands in for the paths in issues.conventions, separated by a comma and a space, or the word none when that list is empty. The session reads those files for its urgency rating and its issue drafts, so a placeholder left in place names a file that does not exist: the session falls back to the rubric’s default priority meanings and files issues with no type, labels, fields, or board placement.

The routines API exposes no configuration read, so the routine’s repository binding and connector set can only be verified by firing a session that reports on itself (prompts/probe.md, reporting to /findings/probe; scripts/probe_routine.py fires it). Connector drift is not detected automatically: re-run the probe whenever the routine or the account’s connectors change.

Give the probe its own routine. A routine holds one prompt at a time, so a single routine cannot be both. Create a second routine carrying prompts/probe.md and leave the first on prompts/investigator.md. Record the probe’s id as trigger.probe_routine_id and put its trigger token in the parameter store under the key trigger.probe_token_ref names (probe-trigger-token by default; the config holds the key name, never the token), then:

Terminal window
scripts/probe_routine.py

Read the session transcript alongside the receiver’s log group for the /findings/probe POST. The probe’s pass condition is agreement between the two: a POST with no matching transcript proves nothing, which is why handle_probe can be unauthenticated.

The probe calls the receiver, the health check and then the POST, before it reads anything from GitHub. A session that read an issue first had its health check blocked by its own safety checks as exfiltration, and then declined the POST as well, so keep those two requests ahead of the GitHub-tools step whenever the prompt changes.

The probe does not try a push or send a credential of its own, and an unattended session declines both as reconnaissance, taking neighbouring steps with them. Neither would tell you anything the platform’s documentation does not: its GitHub proxy substitutes your GitHub credential for the proxy-injected placeholder, and it does not limit which branches a push can update. The session never pushes, so what bounds a push it should not make is the prompt’s prohibition plus branch protection or rulesets on GitHub, which no probe can change. Each step the probe keeps says what you learn from it and what counts as a pass; keep that true of any step you add.

A trigger token authorises exactly one routine. The investigator’s token cannot fire the probe, and the API answers with 401 Token is not authorized for this routine, which names the routine and so reads like a wrong id when the id is fine. Each routine needs its own token recorded separately.

Running one routine instead of two costs you a swap, and the restore is the part that bites. If you have not created a probe routine, probing means replacing the investigator’s prompt with prompts/probe.md, firing ROUTINE_ID=... ROUTINE_TRIGGER_TOKEN=... scripts/probe_routine.py, reading the result, and then putting prompts/investigator.md back. Until you restore it, every live alert fires a probe session instead of an investigation, and those alerts get the deadline sweep’s “did not report back” reply rather than findings. A second routine removes that window entirely, which is why it is the recommended shape.

Firing the investigator’s own routine with this script does not run a probe. The script sends {"probe": true}, the investigator sees a payload missing all five required fields (project, issue_ids, release, batch_id, reply_token), and correctly refuses to interpret it, reporting a malformed fire instead. That is the investigator behaving as designed, not a probe result.

A malformed-fire report is still useful even though it is not a probe. It confirms the receiver URL the routine actually holds, that egress reaches that host, and that /findings rejects an unauthenticated POST, without touching the prompt at all. When all you need is “does the routine still point at the right receiver”, fire the script and read the report rather than doing the swap.

Issue filing ships off, and merging the engine change that carries it activates nothing: the receiver runs only what cdk deploy ships, the routine runs only the prompt pasted into it, and issues.enabled: false keeps the /issues route answering 404 and every findings response’s tracking null. Switching it on is a sequence of operator steps, each checked before the next:

  1. Deploy the receiver with the flag still off. The receiver must accept findings schema v3 before the routine sends it, or every v3 findings POST is rejected and each thread gets the deadline reply instead of its findings. Check: /health answers ok, and the next findings delivery posts as before with no FINDINGS_REJECTED line.
  2. Update the App’s permissions as What the Sentinel App needs lists, and have an organization owner accept the request. This step is independent of the others and can run any time before step 4. Check: the installation page lists the new permissions with no pending request.
  3. Update the investigator routine. Set issues.conventions first, because the paste writes those paths into the prompt, and confirm each path exists on every branch your releases are cut from. Then follow updating the investigator routine, and re-run the configuration probe. Check: a replayed alert’s findings card shows its urgency, and the receiver’s log shows no request to /issues, which proves the issue step stays dormant while the flag is off.
  4. Turn filing on for one environment. Set issues.enabled: true, issues.environments to that one environment, issues.proxy_url, and issues.issue_types, issues.fields and issues.project_number as the target repository uses them, then deploy with the same cdk deploy. assert_ready runs at synth, so a missing or placeholder value stops the deploy before anything ships. Check: replay alerts in that environment until each path has happened once (a new issue, a comment on an open duplicate, a reopen of a closed one, a new issue linked to a related one, and a create with the duplicate search unavailable), and read each issue and each thread reply. If any path fails, set the flag back off, fix, and re-run it and every path after it.
  5. Soak, then widen. Leave it on for the first environment until several organic outcomes have been read end to end, then add the remaining environments to issues.environments and deploy.

Rollback works stage by stage, and the order matters: the routine’s prompt goes back before the receiver does.

Undo How Notes
Filing issues.enabled: false, or drop the environment from issues.environments, then cdk deploy The durable stop. The route re-checks both on every call, so a session holding a grant is refused at once. Issues already filed stay; close a misfiled one by hand as a duplicate or not planned
Emergency stop Remove Issues write from the App’s permissions Takes effect at once with no approval and stops every issue write at the token mint, without touching autofix. Restore it once the flag is off
Emergency stop Revert the routine’s prompt Stops new sessions from running the issue step at all
The routine Paste the previous engine commit’s substituted prompt, read it back, record it Do this before rolling the receiver back
The receiver Check out the previous engine commit and cdk deploy Only while the routine runs the previous prompt: a routine still sending schema v3 gets every findings POST rejected
The App’s permissions Remove Issues write, Projects write and Issue fields read Only after the flag is off, or live operations fail at the mint

The receiver cannot delete or close anything, so no rollback touches an issue already filed: each one describes a real error.

To fall back to Sentry’s stock Teams cards without touching AWS at all, run scripts/migrate_rules.py rollback (see Sentry setup). The alert table is retained on stack delete, so cdk destroy leaves posted-message history intact.