Files
pwap/ops/README.md
T
SatoshiQaziMuhammed 7c9fbfb4b2 ci: one approval per deploy, not two (#30)
All four deploy jobs carried `environment: production`, which reads as the
stricter design — GitHub holds each one until approved. It does not behave that
way. Approval requests are batched by eligibility, and these jobs are never
eligible together: deploy-supabase waits only on the notification while the rest
wait on image builds. So a run asks twice, minutes apart, and nothing announces
the second round.

That happened on 2026-07-31. The first approval shipped the Supabase functions
and migrations; the run went back to waiting and stayed there. For as long as it
sat, the schema had moved and the app serving it had not.

Deployment protection now lives on a single `approve-deploy` job that the four
deploy jobs depend on. One approval releases the whole set.

That trades GitHub's own per-job enforcement for a dependency edge, and an edge
can be dropped by a later edit. ops/check-deploy-gate.py makes it a guarantee
again: it fails CI if a deploy job does not depend on the gate, if a second job
declares the environment and brings the split back, or if a job uses always()
without asserting the gate succeeded — which looks correct and silently
un-holds it. Verified against all three by mutating the workflow.

Also documented in ops/README.md: a run waiting on approval cannot be stopped
with `gh run cancel`; its pending deployment must be rejected. And that
bump-version still pushes to main before the gate — left alone, but written
down rather than left to be rediscovered.
2026-08-01 00:55:13 -07:00

129 lines
5.4 KiB
Markdown

# ops
Operational configuration that lives outside the application code.
## `apply-repo-settings.sh`
The repository's GitHub protections, as code.
Branch protection and the `production` environment are configured through the
GitHub API. Nothing in the repository reflects them, so by default they are
invisible here, unversioned, and gone without a trace if someone removes them —
which undercuts the point, since both exist precisely so that nothing reaches
`main` or production unreviewed.
```bash
./ops/apply-repo-settings.sh # apply the desired state
./ops/apply-repo-settings.sh --check # report drift, change nothing
```
Idempotent: each call PUTs the full desired state, so repeated runs converge.
Requires `gh` authenticated with admin rights on the repo.
### What it enforces
**Branch protection on `main`**
| setting | value | why |
|---|---|---|
| required check | `CI Gate ✅` | aggregate job — fails if web, backend or security-audit did not succeed |
| strict | true | branch must be up to date with `main` before merging |
| approvals | 1 | no unreviewed code on `main` |
| dismiss stale reviews | true | an approval does not carry over to new commits |
| force pushes | blocked | history cannot be rewritten |
| deletions | blocked | `main` cannot be deleted |
| conversation resolution | required | review comments cannot be merged past |
Only `CI Gate ✅` is listed as a required check, deliberately. It runs with
`if: always()` and inspects the results of the jobs it needs, so it already
covers them. Listing each job here as well would mean editing this file whenever
one is renamed — and a rename would then quietly drop a requirement instead of
failing loudly.
`enforce_admins` stays `false` on purpose: an admin needs a way through during a
real incident. It is an escape hatch, not the normal path — rare, visible, and
recorded in the branch's protection history.
**`production` environment**
Deployment approval is an environment protection rule, not something the workflow
implements. GitHub holds the run until a reviewer approves it:
- no runner is occupied while waiting
- the window is 30 days, so a missed notification costs nothing
- who approved which SHA is recorded in the deployment history
- approval state survives a runner restart
Exactly one job in `quality-gate.yml` opts in with `environment: production`:
`approve-deploy`. The four deploy jobs are held by depending on it.
`notify-deploy-pending` only sends a Telegram message pointing at the run; it
cannot block anything.
This replaced a job that polled `/tmp/pexsec-gates` for 30 minutes while holding
a runner slot. On 2026-07-30 that window expired unseen and cancelled a deploy —
a merged migration never reached the database and the only trace was a failed
job.
### Why one gate job and not `environment:` on each deploy
The first version put `environment: production` on all four deploy jobs, which
reads as the more direct design — GitHub itself holds each one. It does not
behave that way.
GitHub opens an approval request for the jobs that are *eligible at that moment*,
and these four never are. `deploy-supabase` waits only on the notification;
the rest wait on image builds that take minutes. So a run asks twice. On
2026-07-31 the first approval shipped the Supabase functions and migrations, and
the run then went back to waiting — unnoticed, because the notification had
already been sent and answered. The schema had moved and the app serving it had
not. Nobody is told about the second round; you have to be looking at the run.
One gate job removes the split by construction: everything that deploys hangs off
a single approval, so there is only ever one to give.
The cost is that approval is now enforced by a dependency edge rather than by
GitHub. `ops/check-deploy-gate.py` restores the guarantee — it fails CI if a
`deploy-*` job does not depend on `approve-deploy`, if a second job declares the
environment (which brings the split back), or if a job uses `always()` without
asserting the gate succeeded, which silently un-holds it. The `Deploy gate
wiring` job runs it on every PR.
```bash
python3 ops/check-deploy-gate.py
```
### Known: the version bump lands before approval
`bump-version` commits `web/package.json` to `main` and pushes it before the gate
— so rejecting a deploy leaves a bumped version for a release that never shipped.
It is a number in a file, `[skip ci]`, and touches no server. Left as is rather
than changed quietly; moving it behind `approve-deploy` is a one-line change if
that is preferred.
### Stopping a run that is waiting for approval
`gh run cancel` does not work on a run held at an environment gate — it reports
success and the run stays `waiting`, still holding the queue behind it. The
pending deployment has to be **rejected**:
```bash
gh api repos/:owner/:repo/actions/runs/<RUN_ID>/pending_deployments \
-X POST -f state=rejected -f comment='superseded' \
-F 'environment_ids[]=<ENV_ID>'
```
Get `<ENV_ID>` from `gh api repos/:owner/:repo/actions/runs/<RUN_ID>/pending_deployments`.
This came up on 2026-07-31: a stale run sat in `waiting` and blocked the deploy
of the run behind it until it was rejected.
### Reviewer
`REVIEWER_ID` is a numeric user id rather than a login, because the API takes ids
and a login rename would silently break the rule instead of erroring.
Override any of it via environment variables:
```bash
REPO=owner/repo BRANCH=main REVIEWER_ID=123456 ./ops/apply-repo-settings.sh
```