From 6b368d97353a52b2f40fd9c814696161f7af24bd Mon Sep 17 00:00:00 2001 From: Satoshi Qazi Muhammed Date: Thu, 30 Jul 2026 16:23:38 -0700 Subject: [PATCH] ci: widen the deploy approval window from 30 minutes to 6 hours MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #25 was merged, the approval request went out at 22:14, nobody saw it in time, and at 22:44 the gate cancelled the deploy. Nothing shipped and the only trace was a failed job — the repair migration sat merged but unapplied. Thirty minutes assumes the approver is at their phone when the request lands. A meeting or a night's sleep loses the deploy, and the failure mode is silent unless someone goes looking at Actions. Six hours, with the deadline now stated in the Telegram message itself so the approver knows what they are working against. Job timeout-minutes is set to 370 explicitly: without it the job relies on GitHub's 6h default, which would kill the job at almost exactly the moment the script is still waiting. Worth being honest about the tradeoff: this job holds a pwap-runner slot for the entire wait, so six hours of waiting is six hours that runner cannot do anything else. The right fix is a GitHub Environment with required reviewers — that holds no runner while it waits and allows up to 30 days. This loop is a hand-rolled stand-in for that mechanism; widening it buys time but does not remove the cost. --- .github/workflows/quality-gate.yml | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/.github/workflows/quality-gate.yml b/.github/workflows/quality-gate.yml index 573f8538..d8596290 100644 --- a/.github/workflows/quality-gate.yml +++ b/.github/workflows/quality-gate.yml @@ -307,6 +307,9 @@ jobs: name: Telegram deploy approval runs-on: pwap-runner needs: [web, security-audit] + # Must exceed the in-script wait below, otherwise the job is killed first and + # the approval window is silently shorter than advertised. + timeout-minutes: 370 if: github.ref == 'refs/heads/main' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') steps: @@ -331,7 +334,7 @@ jobs: -d "{ \"chat_id\": \"${CEO_CHAT_ID}\", \"parse_mode\": \"Markdown\", - \"text\": \"🚀 *pwap/web Deploy Approval*\\n\\n\`${SHORT}\` — ${ACTOR}\\n\\n_${SAFE_MSG}_\\n\\nTargets: app.pezkuwichain.io + pex.mom\", + \"text\": \"🚀 *pwap/web Deploy Approval*\\n\\n\`${SHORT}\` — ${ACTOR}\\n\\n_${SAFE_MSG}_\\n\\nTargets: app.pezkuwichain.io + pex.mom\\n\\n_Approve within 6 hours or the deploy is cancelled._\", \"reply_markup\": { \"inline_keyboard\": [[ {\"text\": \"✅ Approve\", \"callback_data\": \"deploy_approve:${SHORT}\"}, @@ -340,8 +343,16 @@ jobs: } }" - echo "Waiting for Telegram approval (max 30 min)..." - TIMEOUT=1800 + # 30 minutes meant an approval request sent at a bad moment expired + # unseen and the deploy cancelled itself with nothing shipped. + # + # NOTE: this job holds a pwap-runner slot for the whole wait. Six hours + # of waiting is six hours that runner cannot do anything else. The + # proper fix is a GitHub Environment with required reviewers, which + # holds no runner at all while it waits; this loop is a hand-rolled + # stand-in for that. + echo "Waiting for Telegram approval (max 6 hours)..." + TIMEOUT=21600 ELAPSED=0 while [ $ELAPSED -lt $TIMEOUT ]; do if [ -f "$GATE_DIR/$SHORT" ]; then @@ -358,7 +369,7 @@ jobs: sleep 10 ELAPSED=$((ELAPSED + 10)) done - echo "No approval received within 30 minutes — deploy cancelled." + echo "No approval received within 6 hours — deploy cancelled." exit 1 # ========================================