Runbooks
Runbooks for the credentials the engine holds, what the Sentinel App needs on the GitHub side, switching issue filing on, and how to read what the pipeline reports about itself. Deploying the receiver itself is covered in infra/README.md; the one-time Microsoft and Sentry setup lives in Microsoft setup and Sentry setup.
Credential inventory
Section titled “Credential inventory”| Credential | Where it lives | Purpose |
|---|---|---|
/sentinel/github-app-private-key |
SSM SecureString, receiver | The Sentinel App’s private key, read by the receiver at cold start; it mints the scoped installation token the receiver uses to open each pull request and to make each issue write |
SENTRY_WEBHOOK_SECRET, SENTRY_API_TOKEN, BOT_CLIENT_SECRET, ROUTINE_TRIGGER_TOKEN |
SSM SecureStrings, receiver | Webhook verification, Sentry API access, the bot identity, and the routine fire token |
| Installation tokens | Nowhere: minted per pull request or per issue operation inside the receiver, spent in the same invocation, never returned or stored | Lets the receiver push one branch and open one pull request (contents: write and pull_requests: write), or make one issue operation’s write, downscoped to that operation (OP_PERMISSIONS); scoped to the target repo, one hour |
| Callback capability tokens | DynamoDB, hashed, one per dispatch | Bearer credential the fix session presents to /autofix-result; scoped to a single dispatch, not reusable |
| Tracking tokens | DynamoDB, hashed, one per delivered result while issue filing is on for its environment | Bearer credential a session presents to /issues; scoped to one alert and its thread, dead at its grant’s expires_at (30 minutes after findings, or 90 when the result also earned an autofix grant) |
| The operator’s GitHub connection | The cloud platform, outside the session; never in this repository or the parameter store | Authenticates the session’s repository fetch and its issue reads through the platform’s built-in GitHub tools; the routine’s tool policy withholds every write tool, so it is used for reads only |
/sentinel/ntfy-topic |
SSM SecureString, alarm forwarder (optional) | The ntfy topic alarms are published to, read only by the forwarder and only when aws.alarm_ntfy_server is set. On a public ntfy server the topic name is the only access control, so it is held like a credential |
SENTINEL_BANNED_PATTERNS |
Actions secret, engine repo | The deployment-identifying ban patterns tests/test_public_tree.py enforces; not a credential, but held like one because publishing the list would index what it protects |
The routine trigger token is never stored in this repo; keep the only other copy in your password manager.
What the Sentinel App needs
Section titled “What the Sentinel App needs”The receiver mints every GitHub credential itself, for autofix pull requests and for issue writes alike, so the App’s configuration is the only thing standing between a passing gate and a pull request, or between a session’s issue operation and the repository:
- The App must be installed on the target repository, the one named by
target_repo. Minting looks that installation up by repository, so an App installed on the organization but not on this repository fails at the lookup. - For autofix, the installation must grant contents: write and pull requests: write. The receiver downscopes each fix’s token to exactly those two, and a token can never hold a permission the installation lacks, so a missing one produces a token that cannot push or cannot open the pull request.
- For issue filing, the installation must also grant Issues: read and write (create, comment, reopen, labels, the issue type on create, and issue-field values), organization Projects: read and write (adding a new issue to the board
issues.project_numbernames), and organization Issue fields: read-only (the field and option namesset_fieldsresolves). Each issue operation mints its own token, downscoped to what that operation needs (OP_PERMISSIONSinreceiver/github_app.py), and none carries contents, pull requests, workflows or administration. Organization permissions reach every board in the organization; the named operations are what confine the App’s board writes to the one configured. - GitHub holds a permission change until an organization owner accepts it on the installation. Until then the installation keeps its old permissions, autofix is unaffected, and every issue operation fails at the mint with a 502.
- Do not grant
workflows. It is excluded from every minted token on purpose (why), and granting it at the installation widens the App’s own reach for no benefit here. autofix.app_idin the config file must be the App’s numeric id, and/sentinel/github-app-private-keymust be one of that App’s active private keys. Issue filing uses the same App and key, even with autofix off.
The routine’s environment reaches GitHub for reads only
Section titled “The routine’s environment reaches GitHub for reads only”The session reaches GitHub three ways, and all three are reads: the investigation’s
repository fetch, the fix phase’s fetch of the base branch, and the issue step’s reads
through the platform’s built-in GitHub tools. It sends its findings, its fix, and its
issue operations to the receiver, and the receiver makes every GitHub write. So the
routine’s environment needs the same GitHub reach the investigation already has, the
repository fetch, and no more: the tool reads need no egress change, because the
built-in tools reach GitHub through the platform’s own GitHub proxy, the path the
repository fetch already takes. If fixes are failing while findings keep arriving, the
pattern points at the receiver’s side, the App configuration above or the
AUTOFIX_FAILED lines below, before it points at egress.
The routine’s GitHub tool policy
Section titled “The routine’s GitHub tool policy”The cloud platform gives every routine session a built-in GitHub server whose tools act as the operator. The investigator uses one of them, issue_read, to open the target repository’s issues. It does not search with that server: the platform removes the server’s search tools from routine sessions and points them at gh api instead, which carries the same operator credential and can write as easily as it reads. The investigator searches through the receiver’s search operation, which runs as the App with a read-only token. Every other tool of the server, search_issues and every tool that writes among them, is withheld by the routine’s own tool settings. prompts/routine-tools.yaml is the canonical copy of those settings: read_allowlist holds the one read, and disallowed_tools holds every other tool of the server, each with the mcp__github__ prefix. tests/test_routine_tools.py keeps the list sorted, disjoint from the read, and complete against a floor of write tools and the server’s issue searches, and fails if the investigator prompt names a withheld tool.
Both routines carry the same policy: the investigator, and the probe routine recorded as trigger.probe_routine_id. That keeps a probe run a faithful test of the investigator’s settings.
Applying it. The public routines API cannot read or write a routine’s tool settings. Apply them through routine tooling (a Claude Code session’s routine get and update), or on the routine’s page at claude.ai when it shows both fields. Either way:
- Read the routine.
- Set
job_config.ccr.session_context.disallowed_toolsto the file’sdisallowed_tools, and append the read, prefixed (mcp__github__issue_read), tojob_config.ccr.session_context.allowed_tools, keeping every entry already there exceptmcp__github__search_issues, which earlier versions of the file allowed and which must now come out. Allowing the read outright keeps it from waiting on the session’s permission checker mid-run. - Send the whole
job_configback. The update treatsjob_configas one field, so a fragment can replace the stored prompt with nothing. - Read the routine again and compare both tool lists and the stored prompt with the files. Any difference is a partial update: apply again.
Then do the same for the probe routine.
Regenerating the list. Step 5 of prompts/probe.md lists every GitHub tool the probe’s session can see. A name beyond issue_read is a tool the list does not know about: add it to disallowed_tools in sorted order, fold in any new names from the upstream github-mcp-server tool catalog at the same time, run the tests, apply the file to both routines, and probe again until the probe sees only that read. Re-run the probe whenever either routine’s prompt or tool settings change.
Reading the probe. Read steps 4 to 6 of the probe’s transcript against this table, and the receiver’s log for the /findings/probe line.
| The probe shows | Verdict | Next |
|---|---|---|
| 4b: the receiver logged the POST, and the transcript’s repository and commit match it | The investigator’s delivery path works from this environment | Pass |
| 4b: the transcript shows the POST attempted and failing, and the receiver logged nothing | Egress to the receiver is blocked | Check the environment’s allowed hosts |
5a lists exactly issue_read, and 5c made no call |
The list applies | Pass |
5a lists add_issue_comment, and the session’s permission settings refused 5c |
The list applies at call time, not at listing | Pass, and record that it applied at call time |
5a lists names beyond issue_read, but not add_issue_comment |
The list applies; the extras are tools it lacks | Add them, apply again, probe again |
| 5c got a response from GitHub (404 or 422) | The list does not apply to the built-in tools | The fallback below |
| 5c was refused only by another safety check | Unproven: that check is judgment, not configuration | The fallback below |
5b: issue_read was refused |
The read path is broken | Keep issue filing off: the session could find candidates but never open one to judge it |
| 6c: either token variable looks like a real credential | The session holds a GitHub credential it is meant not to have | Remove it from the environment’s variables |
6d: git ls-remote failed while 6b reached GitHub |
The fetch the fix phase bases a fix on is broken | Keep autofix off until it reads |
| Any step declined on the session’s own judgment | That step is unproven, not failed | Read the stated reason and probe again; reword the step if the reason points at the prompt |
When the list cannot be set or does not apply. Issue filing still ships, on the prompt’s prohibition, which forbids every GitHub write whatever tool would make it. Keep the list configured wherever the routine accepts it, keep it in prompts/routine-tools.yaml, and apply it again at every prompt update: it is harmless while inert, takes effect the day the platform applies it, and every probe checks it again. The security model says so plainly until a probe passes.
A tool the platform adds between probes is not in the list until the next probe finds it. The prompt’s prohibition covers that tool meanwhile, which is prose rather than configuration.
Runbook: rotating the GitHub App private key
Section titled “Runbook: rotating the GitHub App private key”- Generate a second private key in the GitHub App settings. GitHub Apps allow two active keys at once, so this rotation is zero-downtime: the old key keeps authenticating while the new one rolls out.
- Update the key in both places it lives. Set
AWS_REGIONto the region you deployed into first;scripts/put-parameters.shand the receiver both default tous-east-1, but nothing requires it.- Your password manager: update the stored copy manually.
- SSM parameter (read by the receiver at cold start).
scripts/put-parameters.shwrites all five receiver secrets together and itsput()helper hard-exits the moment any of them is unset, so it is not a safe tool for rotating this one key alone unless you also have the other four ready to supply in the same run. Update the parameter directly instead:Terminal window aws ssm put-parameter \--name /sentinel/github-app-private-key \--type SecureString \--value "$GITHUB_APP_PRIVATE_KEY" \--overwrite \--region "$AWS_REGION"
- Confirm the new key is actually the one in place, not just that GitHub still accepts requests. Both keys stay valid until you delete one, so a successful mint proves nothing about which key produced it:
- SSM:
aws ssm get-parameter --name /sentinel/github-app-private-key --with-decryption --region "$AWS_REGION" --query Parameter.Value --output text, and diff the output against the new key file. - The receiver caches its App client per container, so a warm Lambda keeps using the key it read at cold start. Wait for a cold start, or force one with a no-op config deploy, before reading anything into the next autofix outcome.
- SSM:
- Only once both checks in step 3 pass, delete the old key from the App settings.
The disclosure gates
Section titled “The disclosure gates”Two gates decide what this repository is willing to hand a stranger, and they cover different things.
tests/test_public_tree.py bans specific strings, and gitleaks (secret-scan.yml) matches credential shapes. Both catch what someone already thought to describe, both are deterministic, and both run locally before anything is pushed for any clone that ran pre-commit install (see CONTRIBUTING.md). On a public repository every pushed branch is published, so that local hook is the only gate that fires before content becomes public; CI is the backstop for clones without it.
Nothing automated reads a change for prose that describes a deployment without using a listed string. That is the author’s to catch, by reading the diff before pushing, and the security model lists what to read for.
The banned strings themselves are split in two. Generic shapes live in the test file; the literals that identify a real deployment live in an untracked local file, config/banned.local.txt, because a tracked ban list is an index of everything it protects. CI materializes that file from the SENTINEL_BANNED_PATTERNS secret and sets SENTINEL_REQUIRE_BANNED_LOCAL, so a vanished secret fails the suite rather than quietly narrowing the scan. After editing the local file, mirror it back:
gh secret set SENTINEL_BANNED_PATTERNS < config/banned.local.txtA fork’s pull request runs without the secret and tests the generic patterns only, which is the same posture a stranger’s clone gets: the push to the default branch after the merge is where the full list applies.
Alarms
Section titled “Alarms”Every way the pipeline can fail at its job raises a CloudWatch alarm on the stack’s alarm topic, which emails aws.alarm_email and, when configured, is forwarded to ntfy. Each alarm fires on a single occurrence in five minutes, and each log-derived one counts a marker the receiver writes at the point of failure, so the receiver’s log group, filtered on the marker, shows which alert it was.
| Alarm | Marker | What failed | Where to look |
|---|---|---|---|
ReceiverErrors |
(the Lambda Errors metric) |
The receiver raised. Teams alerts may be silent | The receiver’s log group around the alarm time |
ReceiverDeliveryFailures |
ALERT_DELIVERY_FAILED |
A card or a thread reply could not be posted to Teams | The line names the issue and what was being posted |
ReceiverPayloadUnusable |
PAYLOAD_UNUSABLE |
Sentry sent a correctly signed alert the receiver could not parse. A payload shape change fails every alert this way | The line carries the parse error; compare it with Sentry’s current webhook payload |
ReceiverEnqueueFailed |
ENQUEUE_FAILED |
An alert card posted, but its investigation row was never written, so nothing will investigate it | The line names the issue; replay the alert once the cause is fixed |
ReceiverFireRejected |
FIRE_REJECTED |
The routine API refused to start a session (or the payload was over its cap), and the batch’s rows were marked failed | The FIRE_UNCLASSIFIED line just before it carries the API’s response |
ReceiverFireStalled |
FIRE_STALLED |
Fires have failed with a retryable error (a 5xx, or the API unreachable) for over an hour, so nothing is being investigated. Logged once per unbroken run of failures | The transient fire failure warnings since the time the line names |
ReceiverWindowExhausted |
SUBSCRIPTION_WINDOW_EXHAUSTED |
The Claude subscription’s usage window is spent, so investigations are being skipped | Usage on the subscription; nothing fires until the window resets |
ReceiverInvestigationTimedOut |
INVESTIGATION_TIMED_OUT |
A fired session never reported before investigation.deadline_seconds, and the thread got the fallback reply |
The session’s own transcript, from the routine’s run history |
ReceiverInvestigationFailed |
INVESTIGATION_FAILED |
A session reported failed for an issue, with its reason on the line. A partial result is not under it |
The reason on the line, then the session’s transcript |
ReceiverFindingsRejected |
FINDINGS_REJECTED |
A findings document failed validation: the stored prompt and the receiver disagree | Updating the investigator routine |
ReceiverAutofixFailures |
AUTOFIX_FAILED |
A fix attempt failed or never called back | Reading autofix in the logs |
ReceiverIssueTrackingFailures |
ISSUE_TRACKING_FAILED |
Filing, matching or searching issues failed, or an alert was left untracked | Reading issue tracking in the logs |
AlarmForwarderErrors |
(the forwarder’s Errors metric) |
The forwarder could not publish to ntfy. The email still arrives | The forwarder’s own log group |
Deliberate holds are never alarmed: a paused routine, the daily fire cap, and a 429 that names a short Retry-After (the platform’s daily run cap) are each the operator’s budget at work, and an alarm on them would teach people to ignore the rest. Construct ids carry a generated suffix after the names above.
Forwarding alarms to ntfy
Section titled “Forwarding alarms to ntfy”The alarm email is easy to miss. With aws.alarm_ntfy_server set (see Configure and deploy), the stack adds a small forwarder function subscribed to the alarm topic, which publishes every alarm, as it moves into ALARM, to an ntfy topic at high priority, tagged sentinel. Anything subscribed to the topic receives it: the ntfy phone app, or a desktop subscriber that raises its own alert.
Each message carries the alarm’s description as its title and body, its name, and when it fired. Nothing else: no Sentry data, no thread or issue link.
To set it up:
- Generate a topic name that cannot be guessed, for example
openssl rand -hex 16. On a public ntfy server the name is the only access control: anyone who has it can read the messages and publish to the topic. - Write it to
/sentinel/ntfy-topicby runningscripts/put-parameters.shwithNTFY_TOPICset alongside the other secrets, or with a singleaws ssm put-parameter --type SecureString. - Set
aws.alarm_ntfy_serverand deploy. - Subscribe your devices to the topic, then test the path by setting one alarm’s state by hand:
aws cloudwatch set-alarm-state --alarm-name <name> --state-value ALARM --state-reason "forwarder test". The next evaluation returns it toOK.
To rotate the topic, generate a new one, write it to the parameter, and move every subscriber to it. The forwarder reads the parameter once per container, so a new topic is picked up within minutes as containers recycle; redeploying picks it up at once. To turn forwarding off, remove aws.alarm_ntfy_server and deploy.
Reading autofix in the logs
Section titled “Reading autofix in the logs”These are the three markers to filter on. They cover the gate’s decisions and every outcome that reaches a thread, but they are not everything autofix logs: see the unmarked lines below the table.
| Marker | Means |
|---|---|
AUTOFIX_DECLINED |
The gate said no, with the reason on the same line. Not a fault: this is the normal outcome for most findings |
AUTOFIX_DISPATCHED |
A grant was staged, with its dispatch id |
AUTOFIX_FAILED |
Something went wrong: a crashed gate, a session reporting failed, a fix_ready payload that broke a rule, a pull request the receiver could not open, or a deadline passing with nothing settled. The rest of the line says which |
The lines that carry no marker
Section titled “The lines that carry no marker”A filter built from the table above will miss several things, and one of them matters operationally. Rejections at /autofix-result are unmarked warnings: a request with no usable bearer token, one naming an unknown dispatch or presenting the wrong token, and one carrying a status outside the accepted set each log a plain rejected autofix callback ... warning. So repeated authentication failures against the callback endpoint, which is the shape a probe of that endpoint would take, produce nothing any of the four markers would catch. Search the callback route’s own wording, not the markers, when that is the question.
Also unmarked: a callback arriving for a dispatch the sweep already settled logs an already settled; ignoring notice at info, which is normal and means the deadline won a race. And a completion reply that cannot be posted to the thread carries the receiver’s general delivery-failure marker rather than an autofix one, so a chat outage during an autofix reply is found under that marker. The App client’s own failures log with their own wording too, each naming the underlying error. The App client’s token mint and pull-request sequence both name the repository they were working against.
AUTOFIX_FAILED ... session reported failure
Section titled “AUTOFIX_FAILED ... session reported failure”The session stopped the grant itself and reported failed. That callback carries no reason, so the line ends there, and the reason lives only in the session’s transcript: find the routine run that started just before the matching AUTOFIX_DISPATCHED line and read how its fix phase ended.
The usual cause is step 8d’s test. The session wrote a fix but could not run its test, either because the routine’s environment lacks a dependency or because the session never found where the environment installed it. The first means making the target repository install it, through a SessionStart hook in that repository or the environment’s setup script. The second is what the prompt’s instruction to search for installed dependencies covers, so a transcript showing a session that gave up without searching is a reason to make the environment put its dependencies where a session looks first, such as on PATH. Once the cause is fixed, replay the alert with --reset (see the replay runbook below) to get the pull request.
AUTOFIX_FAILED ... rejected fix payload
Section titled “AUTOFIX_FAILED ... rejected fix payload”The session sent a fix_ready body that broke one of the receiver’s rules, and the line names the rule: both or neither of files and patch, a path under .github/ or matching autofix.exclude_paths (on either side of a rename), a path that is not plain and repository-relative, more than twenty files, a symlink or submodule mode, more than 512 KB of whole-file content or a patch over 4 MB, a patch that is not plain git diff output (the line says where it stopped reading), a title that is empty or wraps, or a base commit that is not a full SHA. The record is settled as failed, the thread has been told, and no GitHub call was made.
The one worth acting on is a path rejection, because it means the session decided a fix needed a file the operator ruled out. Read the transcript to see whether the exclusion is right or the finding was steered; either way the pull request did not happen, which is the design.
AUTOFIX_FAILED ... at base_sha
Section titled “AUTOFIX_FAILED ... at base_sha”The payload passed and the record was claimed, but the base tree refused the fix, and the line names the file and the reason. It is the session’s fix that is wrong here, not the GitHub App, so nothing on the App checklist applies:
differs at base_sha from the version the patch was made against,does not match base_sha at line N, ordoes not reproduce blob ...: the patch was not taken against the base commit it names. The session diffed against some other commit, or edited the patch by hand. The receiver applies hunks only where they say and checks every result against git’s own blob id, so this never lands as a near miss.does not exist at base_shaoralready exists at base_sha: the same mismatch, seen as a path the patch edits that is not there, or creates that already is.is a symlink at base_sha,is a submodule at base_sha,is a directory at base_sha, orruns through ... which is not a directory: the fix would replace or write through something that is not a regular file. A pure rename carries no mode in a patch, so only the base tree can catch it.more than the ... bytes the receiver rewrites: the files the patch modifies are larger in total than the receiver will read and upload in one invocation. GitHub builds blobs only from whole content, so a small change to a large file still costs the file’s full size here.
No branch was created for any of these. The session’s response carried the same reason, and the transcript shows how it made the patch.
AUTOFIX_FAILED ... pull request not opened
Section titled “AUTOFIX_FAILED ... pull request not opened”The payload passed, the record was claimed, and one of the GitHub calls that build the branch and open the pull request did not succeed. The App client’s own line, just above, names the failing call and the underlying error. Work through the App checklist above: the wrong autofix.app_id, a private key that no longer matches an active key, an App not installed on the target repository, or an installation missing one of the two permissions all land here. A branch that already exists (a replayed dispatch id) lands here too, as does a base commit that is not on the base branch (the session checked out something other than the branch it was told to; the App client’s line says so) and a sequence that ran out of its time budget.
A failure after the branch was created leaves that branch behind with no pull request on it. It is inert, and its name carries the dispatch id that failed:
git ls-remote --heads origin 'autofix/*'lists them, and once you have confirmed no pull request references a branch, git push origin --delete <branch> removes it. The receiver never deletes branches itself; that is a destructive path it does not need.
AUTOFIX_FAILED ... opening timeout
Section titled “AUTOFIX_FAILED ... opening timeout”The receiver claimed a record and never settled it, which means the invocation died between its first GitHub call and its last. The sweep fails the record after three minutes and the thread reads that the receiver did not finish opening the pull request. Check the receiver’s own error log for the invocation, then the branch listing above, since the branch may exist without its pull request.
If a batch instead reports a crash before any of this, the cause is usually the App private key parameter being unreadable at cold start rather than GitHub itself.
Reading issue tracking in the logs
Section titled “Reading issue tracking in the logs”Issue filing logs one marker, ISSUE_TRACKING_FAILED, and the IssueTrackingFailures alarm fires on a single line of it in five minutes. The rest of the line names the alert and the case:
| Line | Means |
|---|---|
... <short id> grant not written: <error> |
The receiver could not write a tracking grant while delivering findings. The findings reply still posted, but the session got no grant for that result, so nothing can file it |
... <short id> issue op rejected: <rule> |
A session sent an operation that broke one of the proxy’s rules, named on the line. Like FINDINGS_REJECTED, it means the prompt and the receiver disagree |
... <short id> <op> failed: <step> |
A GitHub call did not succeed, and the step says which: token mint failed, read the issue, list the labels, create the issue, post the comment, reopen the issue, read the issue fields, field <name> is not single-select (a configured field of another type), set the field values, read the field values back, field values did not persist, read the project, add the issue to the project, or search the issues (for a search, a rate limit included). The session may retry once |
... <short id> <op> rejected: <reason> |
GitHub refused the request and wrote nothing (GitHub refused the issue as sent, usually an issue type it does not recognize), or set_fields named an option no field offers |
... <short id> <op> #<n> landed with warnings: <what> |
The write happened, but a label or the type is not on the issue as sent: a label the repository does not define, dropped before sending, or one GitHub dropped on its own |
... <short id> <op> crashed: <error> |
The receiver raised before the operation’s GitHub write returned, so nothing was written. The operation’s record is failed and the alert’s tracking claim released, so a retry can run |
... <short id> <op> crashed after its GitHub write landed: <error> |
The write happened, then the receiver raised while recording it. The record and the claim are left as they are, so the write is never repeated: a retry is answered from a record that settled done, and finishes marking the alert tracked and posting its link if the crash came first; one that did not settle is refused as in progress. If the alert never shows as tracked, the backstop’s notice at expiry says filing may not have finished |
... <short id> <op> landed on #<n> after the grant closed |
An operation finished just after the backstop expired its grant. The issue exists, and the thread got its link after the backstop’s notice |
... <short id> left untracked at grant expiry |
The backstop: the grant expired with no issue filed, commented on or reopened |
... <short id> may not have finished filing at grant expiry |
The backstop found an operation that claimed the alert and never finished marking it tracked (it died mid-operation, or crashed after its write and was never retried), so an issue may exist |
Refusals carry no marker. A well-formed operation that a cap or a state forbids is answered 409 and logged at warning as issue op for <short id> refused: <reason>: a per-alert or daily cap reached, the alert already tracked, an issue that is not in the target repository or is a pull request, a reopen of an open issue, an operation still in progress, or an environment no longer in issues.environments. If a refusal leaves the alert untracked, the backstop logs the marker once, when the grant expires. Authentication failures at /issues are plain rejected issue op ... warnings, the same shape as the callback route’s. A search whose terms GitHub could not parse is answered 400 and logged at warning as issue search for <short id> refused by GitHub; the session rewords it.
Checking that sessions search. Every search that worked logs issue search for <short id> found <n> issues at info. A filed alert with no such line before its file_issue never searched, and a file_issue carrying search_unavailable true after such a line means the session judged every search failed when one worked. To prove the whole path, replay an alert that already has a tracking issue (a repeat of the same Sentry issue at a new release, sent as replaying an alert describes): the session should log at least two searches, and its tracking outcome should be a duplicate comment on the existing issue, not a new one.
The two backstop notices are what the alert’s thread reads when its grant expires with nothing tracked: No GitHub issue was filed for this alert, so it is tracked only in this thread., or, when an operation died mid-filing, Sentinel may not have finished filing this alert in GitHub; check the repository for an issue before filing one. Either is followed by The duplicate search was also unavailable for this alert, so an existing issue may already describe it. when the session’s duplicate search had failed. A grant whose result opened a pull request, or whose fix attempt failed on the receiver’s side, is released without a notice: the pull request, or the AUTOFIX_FAILED line and its own thread reply, already says what happened.
A thread reply or a notice that cannot be posted carries the receiver’s general delivery-failure marker (issue tracking reply for ..., tracking notice for ...), not this one. A grant with no thread to reply in (its alert card never posted) skips the reply with a plain no thread to reply in for ... warning, and the backstop skips its notice the same way (no thread to post the tracking notice in for ...), still logging the marker.
Runbook: replaying an alert the receiver has already seen
Section titled “Runbook: replaying an alert the receiver has already seen”A repeat alert on an unchanged release finds the investigation row and stops before any Sentry call: it posts a pointer to the earlier thread, or retries once if that attempt failed, and never re-runs a delivered investigation. Which repeats arrive at all depends on the alert rules you configured: a new-issue rule fires once per issue per environment, while frequency, regression, and reappearance rules re-fire for a known issue. So when you need the receiver to process a known issue again, after fixing something in the pipeline for example, sign a webhook body and post it yourself:
python -m scripts.replay_alert --issue 1000000007 --environment prod --dry-runThe body is fixtures/sentry-webhook-alert.json with the target issue’s fields written over it from its stored alert:<environment> row, so you can only replay an issue the receiver has already posted. The signature comes from receiver.handler.sign_body, the same function /sentry verifies against, and the webhook secret is read from SSM rather than the environment so it never lands in shell history.
Start with --dry-run. It prints the exact body and target and sends nothing. A live replay posts a real card to that environment’s real Teams channel.
Two rows make a replay a no-op for a repeat, and --reset deletes both before posting:
investigation:<env>#<release>stops the enqueue, so no session fires and no findings arrive.autofix:<env>#<release>stops the autofix gate one stage later, as “already attempted for this release”.
Each delete is conditional on the row existing, and the script prints deleted: or absent: per row. That distinction is the point: an unconditional delete-item against a mistyped sort key succeeds silently and reads as a successful clear. The alert:<environment> row is never touched, because it is the replay’s own input.
Without --reset, a repeat issue posts a fresh card that points at the earlier thread, or retries a failed investigation if its retry budget is unspent. That is the right mode for a freshly staged error that has no rows yet.
Runbook: updating the investigator routine
Section titled “Runbook: updating the investigator routine”A routine holds one prompt and its own tool settings, and the receiver’s contract with the prompt changes whenever steps 6 to 8 of prompts/investigator.md do. The public routines API cannot read or write a routine’s configuration, so the update happens through routine tooling (a Claude Code session’s routine get and update) or on the routine’s own page at claude.ai:
- Substitute both placeholders in the repository file: the receiver’s origin for
REPLACE_WITH_FUNCTION_URL, and the paths inissues.conventions, separated by a comma and a space, forREPLACE_WITH_ISSUE_CONVENTIONS(the wordnonewhen that list is empty). - Apply the substituted prompt and the tool policy from
prompts/routine-tools.yamltogether, as the tool policy section describes. - Read the routine back and compare its prompt with the substituted file and both tool lists with
prompts/routine-tools.yaml. A prompt update sends the whole job config again, so the read-back covers the two tool lists as well as the prompt. Any difference is a partial update: apply again. - Write down the engine commit the prompt came from, the UTC date, and whether the read-back matched. Nothing compares the routine’s copy with the repository on a schedule, so that record is how a later reader learns which commit the routine carries. Every later prompt or tool-list change repeats it.
After an update, a FINDINGS_SCHEMA_OUTDATED schema_version <n> warning in the receiver’s log group means a session posted findings in a schema older than the one prompts/investigator.md emits, so it ran an older prompt. The receiver delivers those findings as usual, without the status-line entries their schema lacks, so no finding is lost, and the marker is counted rather than alarmed. What an older schema does cost is autofix: a document from before intent was rated (v2 or v3) fails the gate closed with the reason intent not rated, so a routine on an older prompt earns no fix attempts until it is updated. It is expected only for sessions that were already running when the prompt was swapped.
When a change adds a findings schema version, or an /issues operation the prompt calls, deploy the receiver before updating the prompt. A receiver that does not know an operation rejects it as a rule break, which for the search means every alert is filed as unsearched. A receiver that does not know the new version rejects the whole document, which leaves the batch for its deadline to fail into the thread; a receiver that knows it accepts the older version the routine still sends in the meantime. Once those have finished, a new one means the routine still holds an older prompt: compare its prompt against the substituted file again.
The conventions paths live in the pasted prompt, not in the receiver. Moving or renaming a conventions file in the target repository therefore takes two steps: change issues.conventions, then update the routine again.
The probe routine keeps its own prompt, prompts/probe.md, with only REPLACE_WITH_FUNCTION_URL to substitute, and it carries the same tool policy as the investigator, so a tool-policy change is applied and read back on both routines.
Configuration probe
Section titled “Configuration probe”The prompts in prompts/ are templates. REPLACE_WITH_FUNCTION_URL stands in
for the receiver’s Function URL and must be substituted with the real origin
when the prompt is written to the routine. The session cross-checks that
origin against the SENTINEL_RECEIVER_URL environment variable, so a prompt
written with the placeholder left in place fails the check and posts nothing.
The investigator prompt also carries REPLACE_WITH_ISSUE_CONVENTIONS, which
stands in for the paths in issues.conventions, separated by a comma and a
space, or the word none when that list is empty. The session reads those
files for its urgency rating and its issue drafts, so a placeholder left in
place names a file that does not exist: the session falls back to the
rubric’s default priority meanings and files issues with no type, labels,
fields, or board placement.
The routines API exposes no configuration read, so the routine’s repository binding and connector set can only be verified by firing a session that reports on itself (prompts/probe.md, reporting to /findings/probe; scripts/probe_routine.py fires it). Connector drift is not detected automatically: re-run the probe whenever the routine or the account’s connectors change.
Give the probe its own routine. A routine holds one prompt at a time, so a single routine cannot be both. Create a second routine carrying prompts/probe.md and leave the first on prompts/investigator.md. Record the probe’s id as trigger.probe_routine_id and put its trigger token in the parameter store under the key trigger.probe_token_ref names (probe-trigger-token by default; the config holds the key name, never the token), then:
scripts/probe_routine.pyRead the session transcript alongside the receiver’s log group for the /findings/probe POST. The probe’s pass condition is agreement between the two: a POST with no matching transcript proves nothing, which is why handle_probe can be unauthenticated.
The probe calls the receiver, the health check and then the POST, before it reads anything from GitHub. A session that read an issue first had its health check blocked by its own safety checks as exfiltration, and then declined the POST as well, so keep those two requests ahead of the GitHub-tools step whenever the prompt changes.
The probe does not try a push or send a credential of its own, and an unattended session declines both as reconnaissance, taking neighbouring steps with them. Neither would tell you anything the platform’s documentation does not: its GitHub proxy substitutes your GitHub credential for the proxy-injected placeholder, and it does not limit which branches a push can update. The session never pushes, so what bounds a push it should not make is the prompt’s prohibition plus branch protection or rulesets on GitHub, which no probe can change. Each step the probe keeps says what you learn from it and what counts as a pass; keep that true of any step you add.
A trigger token authorises exactly one routine. The investigator’s token cannot fire the probe, and the API answers with 401 Token is not authorized for this routine, which names the routine and so reads like a wrong id when the id is fine. Each routine needs its own token recorded separately.
Running one routine instead of two costs you a swap, and the restore is the part that bites. If you have not created a probe routine, probing means replacing the investigator’s prompt with prompts/probe.md, firing ROUTINE_ID=... ROUTINE_TRIGGER_TOKEN=... scripts/probe_routine.py, reading the result, and then putting prompts/investigator.md back. Until you restore it, every live alert fires a probe session instead of an investigation, and those alerts get the deadline sweep’s “did not report back” reply rather than findings. A second routine removes that window entirely, which is why it is the recommended shape.
Firing the investigator’s own routine with this script does not run a probe. The script sends {"probe": true}, the investigator sees a payload missing all five required fields (project, issue_ids, release, batch_id, reply_token), and correctly refuses to interpret it, reporting a malformed fire instead. That is the investigator behaving as designed, not a probe result.
A malformed-fire report is still useful even though it is not a probe. It confirms the receiver URL the routine actually holds, that egress reaches that host, and that /findings rejects an unauthenticated POST, without touching the prompt at all. When all you need is “does the routine still point at the right receiver”, fire the script and read the report rather than doing the swap.
Runbook: enabling issue filing
Section titled “Runbook: enabling issue filing”Issue filing ships off, and merging the engine change that carries it activates nothing: the receiver runs only what cdk deploy ships, the routine runs only the prompt pasted into it, and issues.enabled: false keeps the /issues route answering 404 and every findings response’s tracking null. Switching it on is a sequence of operator steps, each checked before the next:
- Deploy the receiver with the flag still off. The receiver must accept findings schema v3 before the routine sends it, or every v3 findings POST is rejected and each thread gets the deadline reply instead of its findings. Check:
/healthanswersok, and the next findings delivery posts as before with noFINDINGS_REJECTEDline. - Update the App’s permissions as What the Sentinel App needs lists, and have an organization owner accept the request. This step is independent of the others and can run any time before step 4. Check: the installation page lists the new permissions with no pending request.
- Update the investigator routine. Set
issues.conventionsfirst, because the paste writes those paths into the prompt, and confirm each path exists on every branch your releases are cut from. Then follow updating the investigator routine, and re-run the configuration probe. Check: a replayed alert’s findings card shows its urgency, and the receiver’s log shows no request to/issues, which proves the issue step stays dormant while the flag is off. - Turn filing on for one environment. Set
issues.enabled: true,issues.environmentsto that one environment,issues.proxy_url, andissues.issue_types,issues.fieldsandissues.project_numberas the target repository uses them, then deploy with the samecdk deploy.assert_readyruns at synth, so a missing or placeholder value stops the deploy before anything ships. Check: replay alerts in that environment until each path has happened once (a new issue, a comment on an open duplicate, a reopen of a closed one, a new issue linked to a related one, and a create with the duplicate search unavailable), and read each issue and each thread reply. If any path fails, set the flag back off, fix, and re-run it and every path after it. - Soak, then widen. Leave it on for the first environment until several organic outcomes have been read end to end, then add the remaining environments to
issues.environmentsand deploy.
Rollback works stage by stage, and the order matters: the routine’s prompt goes back before the receiver does.
| Undo | How | Notes |
|---|---|---|
| Filing | issues.enabled: false, or drop the environment from issues.environments, then cdk deploy |
The durable stop. The route re-checks both on every call, so a session holding a grant is refused at once. Issues already filed stay; close a misfiled one by hand as a duplicate or not planned |
| Emergency stop | Remove Issues write from the App’s permissions | Takes effect at once with no approval and stops every issue write at the token mint, without touching autofix. Restore it once the flag is off |
| Emergency stop | Revert the routine’s prompt | Stops new sessions from running the issue step at all |
| The routine | Paste the previous engine commit’s substituted prompt, read it back, record it | Do this before rolling the receiver back |
| The receiver | Check out the previous engine commit and cdk deploy |
Only while the routine runs the previous prompt: a routine still sending schema v3 gets every findings POST rejected |
| The App’s permissions | Remove Issues write, Projects write and Issue fields read | Only after the flag is off, or live operations fail at the mint |
The receiver cannot delete or close anything, so no rollback touches an issue already filed: each one describes a real error.
Falling back to stock Sentry alerts
Section titled “Falling back to stock Sentry alerts”To fall back to Sentry’s stock Teams cards without touching AWS at all, run scripts/migrate_rules.py rollback (see Sentry setup). The alert table is retained on stack delete, so cdk destroy leaves posted-message history intact.