ci: one approval per deploy, not two (#30)

All four deploy jobs carried `environment: production`, which reads as the
stricter design — GitHub holds each one until approved. It does not behave that
way. Approval requests are batched by eligibility, and these jobs are never
eligible together: deploy-supabase waits only on the notification while the rest
wait on image builds. So a run asks twice, minutes apart, and nothing announces
the second round.

That happened on 2026-07-31. The first approval shipped the Supabase functions
and migrations; the run went back to waiting and stayed there. For as long as it
sat, the schema had moved and the app serving it had not.

Deployment protection now lives on a single `approve-deploy` job that the four
deploy jobs depend on. One approval releases the whole set.

That trades GitHub's own per-job enforcement for a dependency edge, and an edge
can be dropped by a later edit. ops/check-deploy-gate.py makes it a guarantee
again: it fails CI if a deploy job does not depend on the gate, if a second job
declares the environment and brings the split back, or if a job uses always()
without asserting the gate succeeded — which looks correct and silently
un-holds it. Verified against all three by mutating the workflow.

Also documented in ops/README.md: a run waiting on approval cannot be stopped
with `gh run cancel`; its pending deployment must be rejected. And that
bump-version still pushes to main before the gate — left alone, but written
down rather than left to be rediscovered.

.github/workflows/quality-gate.yml

@@ -345,6 +345,38 @@ jobs:
 
           echo "Approver notified; deployment waits on the 'production' environment."
 
+  # ========================================
+  # APPROVAL GATE — ONE JOB, ONE APPROVAL
+  # ========================================
+  # Every deploy hangs off this job. It exists so that a run needs exactly one
+  # approval, not several.
+  #
+  # It used to be that all four deploy jobs carried `environment: production`
+  # themselves. GitHub opens an approval request for whichever jobs are eligible
+  # at that moment, and these four are not eligible at the same moment —
+  # deploy-supabase only waits on the notification, while the rest wait on image
+  # builds that take minutes. So the run asked twice, several minutes apart. On
+  # 2026-07-31 the first approval shipped Supabase and the run then sat waiting
+  # again, unnoticed: the schema had moved and the app serving it had not.
+  #
+  # Deployment protection lives here now and nowhere else, so approving once
+  # releases the whole set. The deploy jobs are held by their dependency on this
+  # job instead of by their own environment — workflow-guard enforces that edge
+  # so a new deploy job cannot quietly skip the gate.
+  #
+  # Waiting for approval costs no runner: GitHub holds the job before dispatch.
+  approve-deploy:
+    name: Await deploy approval
+    runs-on: pwap-runner
+    environment: production
+    needs: [notify-deploy-pending]
+    timeout-minutes: 5
+
+    steps:
+      - name: Record approval
+        run: |
+          echo "Deploy of ${GITHUB_SHA:0:7} approved. Releasing deploy jobs."
+
   # ========================================
   # VERSION BUMP (RUNS BEFORE BOTH DEPLOYS)
   # ========================================
@@ -397,10 +429,10 @@ jobs:
   deploy-app:
     name: Deploy app.pezkuwichain.io
     runs-on: pwap-runner
-    environment: production
+    needs: [approve-deploy, notify-deploy-pending, bump-version, build-image]
-    needs: [notify-deploy-pending, bump-version, build-image]
     if: |
       always() &&
+      needs.approve-deploy.result == 'success' &&
       needs.notify-deploy-pending.result == 'success' &&
       ((github.event_name == 'push' && needs.build-image.result == 'success' && needs.bump-version.result == 'success') ||
        (github.event_name == 'workflow_dispatch' && github.event.inputs.rollback_to != ''))
@@ -567,10 +599,10 @@ jobs:
   deploy-pex:
     name: Deploy pex.mom
     runs-on: pwap-runner
-    environment: production
+    needs: [approve-deploy, notify-deploy-pending, bump-version, build-image]
-    needs: [notify-deploy-pending, bump-version, build-image]
     if: |
       always() &&
+      needs.approve-deploy.result == 'success' &&
       needs.notify-deploy-pending.result == 'success' &&
       ((github.event_name == 'push' && needs.build-image.result == 'success' && needs.bump-version.result == 'success') ||
        (github.event_name == 'workflow_dispatch' && github.event.inputs.rollback_to != ''))
@@ -763,10 +795,10 @@ jobs:
   deploy-backend:
     name: Deploy Backend (indexer)
     runs-on: pwap-runner
-    environment: production
+    needs: [approve-deploy, notify-deploy-pending, build-image-backend, backend-cfg]
-    needs: [notify-deploy-pending, build-image-backend, backend-cfg]
     if: |
       always() &&
+      needs.approve-deploy.result == 'success' &&
       needs.backend-cfg.outputs.configured == 'true' &&
       (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/')) &&
       needs.notify-deploy-pending.result == 'success' &&
@@ -928,10 +960,10 @@ jobs:
   deploy-supabase:
     name: Deploy Supabase (functions + migrations)
     runs-on: pwap-runner
-    environment: production
+    needs: [approve-deploy, notify-deploy-pending]
-    needs: [notify-deploy-pending]
     if: |
       always() &&
+      needs.approve-deploy.result == 'success' &&
       needs.notify-deploy-pending.result == 'success' &&
       github.event_name == 'push' && github.ref == 'refs/heads/main'
     permissions:
@@ -1168,10 +1200,24 @@ jobs:
             exit 1
           fi
 
+  # Deploy approval is enforced by a dependency edge (see approve-deploy), and an
+  # edge can be dropped by a later edit without anyone noticing until a deploy
+  # goes out unapproved. This job is what keeps that from being possible: it
+  # fails the PR, not production.
+  workflow-guard:
+    name: Deploy gate wiring
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Every deploy job must sit behind the approval gate
+        run: python3 ops/check-deploy-gate.py
+
   ci-gate:
     name: CI Gate ✅
     runs-on: pwap-runner
-    needs: [web, backend, security-audit, migration-test]
+    needs: [web, backend, security-audit, migration-test, workflow-guard]
     if: always()
 
     steps:

ops/README.md

@@ -54,7 +54,8 @@ implements. GitHub holds the run until a reviewer approves it:
 - who approved which SHA is recorded in the deployment history
 - approval state survives a runner restart
 
-The deploy jobs in `quality-gate.yml` opt in with `environment: production`.
+Exactly one job in `quality-gate.yml` opts in with `environment: production`:
+`approve-deploy`. The four deploy jobs are held by depending on it.
 `notify-deploy-pending` only sends a Telegram message pointing at the run; it
 cannot block anything.
 
@@ -63,6 +64,58 @@ a runner slot. On 2026-07-30 that window expired unseen and cancelled a deploy
 a merged migration never reached the database and the only trace was a failed
 job.
 
+### Why one gate job and not `environment:` on each deploy
+
+The first version put `environment: production` on all four deploy jobs, which
+reads as the more direct design — GitHub itself holds each one. It does not
+behave that way.
+
+GitHub opens an approval request for the jobs that are *eligible at that moment*,
+and these four never are. `deploy-supabase` waits only on the notification;
+the rest wait on image builds that take minutes. So a run asks twice. On
+2026-07-31 the first approval shipped the Supabase functions and migrations, and
+the run then went back to waiting — unnoticed, because the notification had
+already been sent and answered. The schema had moved and the app serving it had
+not. Nobody is told about the second round; you have to be looking at the run.
+
+One gate job removes the split by construction: everything that deploys hangs off
+a single approval, so there is only ever one to give.
+
+The cost is that approval is now enforced by a dependency edge rather than by
+GitHub. `ops/check-deploy-gate.py` restores the guarantee — it fails CI if a
+`deploy-*` job does not depend on `approve-deploy`, if a second job declares the
+environment (which brings the split back), or if a job uses `always()` without
+asserting the gate succeeded, which silently un-holds it. The `Deploy gate
+wiring` job runs it on every PR.
+
+```bash
+python3 ops/check-deploy-gate.py
+```
+
+### Known: the version bump lands before approval
+
+`bump-version` commits `web/package.json` to `main` and pushes it before the gate
+— so rejecting a deploy leaves a bumped version for a release that never shipped.
+It is a number in a file, `[skip ci]`, and touches no server. Left as is rather
+than changed quietly; moving it behind `approve-deploy` is a one-line change if
+that is preferred.
+
+### Stopping a run that is waiting for approval
+
+`gh run cancel` does not work on a run held at an environment gate — it reports
+success and the run stays `waiting`, still holding the queue behind it. The
+pending deployment has to be **rejected**:
+
+```bash
+gh api repos/:owner/:repo/actions/runs/<RUN_ID>/pending_deployments \
+  -X POST -f state=rejected -f comment='superseded' \
+  -F 'environment_ids[]=<ENV_ID>'
+```
+
+Get `<ENV_ID>` from `gh api repos/:owner/:repo/actions/runs/<RUN_ID>/pending_deployments`.
+This came up on 2026-07-31: a stale run sat in `waiting` and blocked the deploy
+of the run behind it until it was rejected.
+
 ### Reviewer
 
 `REVIEWER_ID` is a numeric user id rather than a login, because the API takes ids

ops/check-deploy-gate.py

@@ -0,0 +1,104 @@
+#!/usr/bin/env python3
+"""Assert that nothing deploys without passing the approval gate.
+
+Approval used to be enforced by GitHub itself: every deploy job carried
+`environment: production`, so GitHub held each one until a human approved. That
+had to change, because GitHub batches approval requests by eligibility and the
+deploy jobs are not eligible at the same moment — a single run asked for
+approval twice, minutes apart, and one approval shipped only part of it.
+
+The gate is now a single job, `approve-deploy`, and the deploy jobs are held by
+depending on it. That is deterministic, but it moves the guarantee from
+"GitHub enforces it" to "the dependency graph says so" — and a graph can be
+edited. This script is what makes it a guarantee again: it fails CI if a deploy
+job is not behind the gate, or if a second job quietly grows its own
+environment and reintroduces the split.
+
+Run: python3 ops/check-deploy-gate.py [workflow.yml ...]
+"""
+
+import sys
+import yaml
+
+DEFAULT_WORKFLOWS = [".github/workflows/quality-gate.yml"]
+GATE = "approve-deploy"
+ENVIRONMENT = "production"
+
+
+def needs_of(job):
+    needs = job.get("needs", [])
+    return [needs] if isinstance(needs, str) else list(needs)
+
+
+def environment_of(job):
+    env = job.get("environment")
+    if isinstance(env, dict):
+        return env.get("name")
+    return env
+
+
+def check(path):
+    with open(path) as fh:
+        workflow = yaml.safe_load(fh)
+
+    jobs = workflow.get("jobs", {})
+    problems = []
+
+    gated = [
+        name
+        for name, job in jobs.items()
+        if environment_of(job) == ENVIRONMENT
+    ]
+
+    if gated != [GATE]:
+        problems.append(
+            f"exactly one job may declare `environment: {ENVIRONMENT}`, and it must be "
+            f"`{GATE}`; found {gated or 'none'}. More than one splits the run into "
+            f"separate approval rounds — the failure this design exists to prevent."
+        )
+
+    if GATE not in jobs:
+        problems.append(f"`{GATE}` is missing; nothing holds the deploys.")
+        return problems
+
+    for name, job in jobs.items():
+        if not name.startswith("deploy-") or name == GATE:
+            continue
+        if GATE not in needs_of(job):
+            problems.append(
+                f"`{name}` does not list `{GATE}` in its needs, so it would deploy "
+                f"without approval."
+            )
+            continue
+        condition = str(job.get("if", ""))
+        # `always()` overrides the implicit "all needs succeeded" rule, so the
+        # dependency alone stops holding and the condition has to say so.
+        if "always()" in condition and f"needs.{GATE}.result == 'success'" not in condition:
+            problems.append(
+                f"`{name}` uses always() but does not require "
+                f"needs.{GATE}.result == 'success' — with always(), depending on the "
+                f"gate no longer waits for it to pass."
+            )
+
+    return problems
+
+
+def main(argv):
+    paths = argv[1:] or DEFAULT_WORKFLOWS
+    failed = False
+
+    for path in paths:
+        problems = check(path)
+        if problems:
+            failed = True
+            print(f"{path}:")
+            for problem in problems:
+                print(f"  - {problem}")
+        else:
+            print(f"{path}: every deploy job is behind `{GATE}`.")
+
+    return 1 if failed else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))