ci: widen the deploy approval window from 30 minutes to 6 hours

#25 was merged, the approval request went out at 22:14, nobody saw it in time,
and at 22:44 the gate cancelled the deploy. Nothing shipped and the only trace
was a failed job — the repair migration sat merged but unapplied.

Thirty minutes assumes the approver is at their phone when the request lands. A
meeting or a night's sleep loses the deploy, and the failure mode is silent
unless someone goes looking at Actions.

Six hours, with the deadline now stated in the Telegram message itself so the
approver knows what they are working against. Job timeout-minutes is set to 370
explicitly: without it the job relies on GitHub's 6h default, which would kill
the job at almost exactly the moment the script is still waiting.

Worth being honest about the tradeoff: this job holds a pwap-runner slot for the
entire wait, so six hours of waiting is six hours that runner cannot do anything
else. The right fix is a GitHub Environment with required reviewers — that holds
no runner while it waits and allows up to 30 days. This loop is a hand-rolled
stand-in for that mechanism; widening it buys time but does not remove the cost.
This commit is contained in:
2026-07-30 16:23:38 -07:00
parent c4b22d0a14
commit 6b368d9735
+15 -4
View File
@@ -307,6 +307,9 @@ jobs:
name: Telegram deploy approval name: Telegram deploy approval
runs-on: pwap-runner runs-on: pwap-runner
needs: [web, security-audit] needs: [web, security-audit]
# Must exceed the in-script wait below, otherwise the job is killed first and
# the approval window is silently shorter than advertised.
timeout-minutes: 370
if: github.ref == 'refs/heads/main' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') if: github.ref == 'refs/heads/main' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch')
steps: steps:
@@ -331,7 +334,7 @@ jobs:
-d "{ -d "{
\"chat_id\": \"${CEO_CHAT_ID}\", \"chat_id\": \"${CEO_CHAT_ID}\",
\"parse_mode\": \"Markdown\", \"parse_mode\": \"Markdown\",
\"text\": \"🚀 *pwap/web Deploy Approval*\\n\\n\`${SHORT}\` — ${ACTOR}\\n\\n_${SAFE_MSG}_\\n\\nTargets: app.pezkuwichain.io + pex.mom\", \"text\": \"🚀 *pwap/web Deploy Approval*\\n\\n\`${SHORT}\` — ${ACTOR}\\n\\n_${SAFE_MSG}_\\n\\nTargets: app.pezkuwichain.io + pex.mom\\n\\n_Approve within 6 hours or the deploy is cancelled._\",
\"reply_markup\": { \"reply_markup\": {
\"inline_keyboard\": [[ \"inline_keyboard\": [[
{\"text\": \"✅ Approve\", \"callback_data\": \"deploy_approve:${SHORT}\"}, {\"text\": \"✅ Approve\", \"callback_data\": \"deploy_approve:${SHORT}\"},
@@ -340,8 +343,16 @@ jobs:
} }
}" }"
echo "Waiting for Telegram approval (max 30 min)..." # 30 minutes meant an approval request sent at a bad moment expired
TIMEOUT=1800 # unseen and the deploy cancelled itself with nothing shipped.
#
# NOTE: this job holds a pwap-runner slot for the whole wait. Six hours
# of waiting is six hours that runner cannot do anything else. The
# proper fix is a GitHub Environment with required reviewers, which
# holds no runner at all while it waits; this loop is a hand-rolled
# stand-in for that.
echo "Waiting for Telegram approval (max 6 hours)..."
TIMEOUT=21600
ELAPSED=0 ELAPSED=0
while [ $ELAPSED -lt $TIMEOUT ]; do while [ $ELAPSED -lt $TIMEOUT ]; do
if [ -f "$GATE_DIR/$SHORT" ]; then if [ -f "$GATE_DIR/$SHORT" ]; then
@@ -358,7 +369,7 @@ jobs:
sleep 10 sleep 10
ELAPSED=$((ELAPSED + 10)) ELAPSED=$((ELAPSED + 10))
done done
echo "No approval received within 30 minutes — deploy cancelled." echo "No approval received within 6 hours — deploy cancelled."
exit 1 exit 1
# ======================================== # ========================================