diff --git a/docs/RELEASE.md b/docs/RELEASE.md index 19b182ae..c385a207 100644 --- a/docs/RELEASE.md +++ b/docs/RELEASE.md @@ -54,10 +54,14 @@ trigger it with `workflow_dispatch` and wait for it; do not rely only on the fas one in the release. 8. **Watch**: confirm blocks are still produced and finalized after the upgrade. -## Pending bundle (next mainnet spec_version) +## Shipped: 2026-07-10 mainnet upgrade (spec_version 1_020_010 / 1_020_011) -These repo changes are ready and green but **not yet deployed**; they ship together in -one coordinated upgrade after the heavy suite is fully green and reviewed: +Deployed to all three teyrchains (relay, Asset Hub, People chain), each verified against +its own `System::LastRuntimeUpgrade` post-upgrade. Full account of what was found and +fixed along the way, including two release-pipeline bugs caught by independent +verification before anything reached mainnet: [docs/audits/2026-07-mainnet-upgrade.md](audits/2026-07-mainnet-upgrade.md). + +The bundle: - Token symbol standardized to **HEZ** (metadata-hash, claims `Prefix`, chain-spec). - Claims statements repointed to **statement.pex.network** (hash-pinned; see @@ -74,9 +78,14 @@ one coordinated upgrade after the heavy suite is fully green and reviewed: (`DisputeWindow`) before taking effect, disputable by any Council member and slashable by `RootOrDiwanOrTechnical` on confirmed fraud. Root/XCM-Transact submissions are unaffected (chain-authenticated, not a personal key). - - Fixed `MONTH_IN_BLOCKS`/`HOUR_IN_BLOCKS` in `staking-score`: was silently assuming - 6s blocks (10/min) against this chain's actual 12s (5/min), doubling every - duration-tier threshold in production. + - Fixed `MONTH_IN_BLOCKS`/`HOUR_IN_BLOCKS` in `staking-score`: an earlier revision + had switched these to assume 12s blocks (5/min), citing an unused same-named + constant from a different crate. `people-pezkuwichain` actually wires its + `HOURS`/`DAYS`/`SlotDuration` from a 6s-block source — confirmed via live mainnet + empirical block time, the real import path, and a try-runtime dry-run that + panicked with an exact 2x slot mismatch under the wrong assumption. Every + duration-tier threshold was filling twice as fast as intended; restored to 10 + blocks/min (6s), matching reality. - `staking-score`'s weights are now real benchmark output (previously manual estimates); `presale::withdraw_funds` had no benchmark case at all until commit `dc6d29e` — real weight pending a benchmark run. diff --git a/docs/audits/2026-07-mainnet-upgrade.md b/docs/audits/2026-07-mainnet-upgrade.md new file mode 100644 index 00000000..fea7dd89 --- /dev/null +++ b/docs/audits/2026-07-mainnet-upgrade.md @@ -0,0 +1,198 @@ +# PezkuwiChain mainnet runtime upgrade — audit report + +**Scope:** pezkuwi-DKS relay, Asset Hub, and People chain +**Window:** 2026-07-08 – 2026-07-10 +**Findings:** 6 (0 unresolved) +**Fund impact:** none — zero loss, zero irreversible error +**Status:** all three chains upgraded and independently verified + +## Executive summary + +This is a full account of a routine mainnet runtime upgrade that became a +multi-stage engineering exercise: six real defects were found across the +runtime source, the release pipeline, and the verification tooling itself — +three pre-existing in the deployed runtime, two in the CI/release +infrastructure, and one in this audit's own first verification attempt. Every +defect was caught before it reached mainnet funds, because each step of +[RELEASE.md](../RELEASE.md)'s procedure was independently re-verified rather +than trusted on the first pass. No user funds were lost or put at risk at any +point. + +## Final chain status + +Verified against on-chain `System::LastRuntimeUpgrade`, not RPC display alone +(finding 6, below, is exactly why that distinction matters). + +| Chain | Before | After | Notes | +|---|---|---|---| +| Relay | `1_020_007` | `1_020_010` | Clean on first attempt | +| Asset Hub | `1_020_008` | `1_020_010` | First attempt was a silent no-op (finding 6); corrected and reapplied | +| People chain | `1_020_009` | `1_020_011` | Clean on first attempt | + +## Procedure followed + +[`docs/RELEASE.md`](../RELEASE.md)'s 8-step fund-critical runbook: + +1. **Freeze & bump** — change set frozen on `main`; `spec_version` bumped per runtime. +2. **Heavy suite** — full test suite, workspace clippy, and benches to green before any tag. +3. **Live-state dry run** — migrations rehearsed with `try-runtime` against a snapshot of real mainnet state. +4. **Tag & build** — a signed tag triggers a reproducible WASM build, hash, and SLSA build-provenance attestation. +5. **Independent reproduction** — the exact release artifact rebuilt from scratch, on separate hardware, before it is ever trusted. +6. **On-chain authorization** — `system.authorizeUpgrade(hash)` then `system.applyAuthorizedUpgrade(wasm)`, gated on explicit human sign-off per chain. +7. **Post-upgrade verification** — confirmed against the chain's own execution record, not just its RPC display. +8. **Watch** — confirmed blocks kept producing and finalizing on all three chains after each step. + +## Findings + +In the order each was discovered; severity reflects what would have happened +had it shipped unnoticed. + +### 1. Staking-tier durations assumed the wrong block time — Medium + +**What:** The `staking-score` pallet's tier-duration constants +(`MONTH_IN_BLOCKS`, `HOUR_IN_BLOCKS`) had been switched to assume 12-second +blocks (5/min) in an earlier revision, citing an unused same-named constant +from a different crate. `people-pezkuwichain` actually wires its +`HOURS`/`DAYS`/`SlotDuration` from a 6-second-block source. + +**Impact:** Every staking duration tier filled twice as fast as intended — a +silent, compounding economic drift, not a crash risk. + +**Confirmed via:** live mainnet empirical block time, the actual import path +in `people-pezkuwichain/src/lib.rs`, and a `try-runtime` live-state dry-run +that panicked with an exact 2x slot mismatch under the wrong assumption. + +**Fix:** restored to 10 blocks/min (6s), matching reality (commit `16cbe65`). + +### 2. Council pallet's storage version was never stamped, next to a migration that could have deleted it — High + +**What:** `Council`'s on-chain storage version had been stuck at its genesis +value since deployment — the version-bump hook never ran. Sitting next to it +in the runtime's migration list was a `RemovePallet` entry, +almost certainly inherited unmodified from an upstream reference chain where +no council pallet was active under that name. + +**Impact:** `Council` is a live, actively-configured pallet on this chain. Had +that removal migration ever executed, it would have deleted a governance +pallet currently in use. + +**Fix:** replaced the removal with a no-op `VersionedMigration<0,4,...>` that +only stamps the correct storage version; deleted the removal entry entirely +(commit `0fd85f6`). + +### 3. 12.49M HEZ sat in undecodable "ghost" holds from a removed pallet — High + +**What:** Twenty-five accounts carried a `Balances::Holds` entry tagged with a +hold-reason discriminant that no longer corresponds to any pallet in the +current runtime — left behind when an old staking pallet was removed without +releasing its holds. Any code path that tried to decode these accounts' holds +would panic. + +**Impact:** a latent crash risk on any future storage-decoding operation +touching these accounts, and 12.49M HEZ nominally "locked" against nothing. + +**Investigation:** pulled the live `RuntimeHoldReason` enum directly from +on-chain metadata and hand-decoded the raw SCALE bytes for all 25 accounts. +Most had little or no real reserved balance behind the stale record — the +funds had already found their way loose through normal channels over time, +leaving only a bookkeeping ghost. + +**Fix:** a migration that releases exactly `min(claimed, actually-reserved)` +per account and always clears the stale record — never invents balance that +isn't really there (commit `8c9fb93`). Dry-run against a full live-state +snapshot: 0 errors, all 63 pallets' try-state checks passed. + +### 4. Release pipeline could publish a test-only WASM variant as the production relay runtime — High + +**What:** the relay runtime's `build.rs` always compiles two WASM blobs side +by side: the real one, and a `fast-runtime`-feature devnet variant with +shortened governance and staking periods, meant only for local test networks. +The release workflow's artifact-selection step (`find ... | head -1`) picked +whichever file its directory listing happened to return first — not a +reliable distinction between the two. + +**Impact:** on the first release build of this upgrade, the picker selected +the devnet variant. Its hash was computed, signed, and staged as the official +relay artifact. Had it gone unnoticed past the independent-reproduction step, +authorizing it on-chain would have silently swapped mainnet's governance and +staking timers for local-test values. + +**Caught by:** step 5 of the procedure (independent, from-scratch rebuild), +which surfaced a hash mismatch against the published artifact once the +correct file was identified by hand. + +**Fix:** the picker now explicitly excludes the devnet variant and fails +loudly, rather than guessing, if more than one candidate remains (PR #27). +Never published, never authorized — caught entirely inside the verification +stage. + +### 5. Release builds were not yet reproducible across the CI fleet — Medium + +**What:** these builds are pinned-toolchain reproducible, not yet fully +hermetic (a known, already-documented limitation pending upstream `srtool` +support for the pinned Rust version). The self-hosted build fleet doesn't +share one filesystem layout, and the absolute checkout path measurably +changes the compiled WASM's bytes. Two idle runners building the identical +commit produced different hashes. + +**Impact:** not a correctness bug in the shipped code, but a real +trust-and-verifiability gap: an outside party rebuilding at a different path +than the one that happened to build the release would see a mismatched hash +and could reasonably suspect tampering that wasn't there. + +**Fix:** release builds are now pinned to one specific, named machine, making +the published hash reliably reproducible until hermetic builds are available +(PR #28). + +### 6. Asset Hub's version number was never bumped — its first upgrade attempt did nothing — High + +**What:** the runtime bundle for this release rebuilt and redeployed Asset +Hub's WASM, but its source-level `spec_version` was left unchanged. The +on-chain code was successfully replaced — but because the chain's own +version bookkeeping saw no version change, it correctly ran no migrations +and left its execution behavior untouched. + +**Impact:** none to funds or safety — the swap was between two builds the +chain considered identical. But the upgrade was, in practice, a no-op. + +**Caught by:** post-upgrade verification checking the chain's own execution +record (`System::LastRuntimeUpgrade`) rather than trusting that a successful +transaction implies a successful upgrade. + +**Fix:** version bumped in source (PR #29), rebuilt, independently +re-verified, and reapplied as a second, corrected on-chain transaction. +Confirmed genuine on the second attempt via the same strict check. + +## Independent verification, chain by chain + +Every published hash was rebuilt from the tagged source and matched, byte +for byte, before being trusted. + +| Runtime | Published blake2-256 | Rebuilt independently | On-chain result | +|---|---|---|---| +| Relay (`pezkuwichain-runtime`) | `0x00399247…7494` | Match | Confirmed live — `1_020_010` | +| Asset Hub (corrected build) | `0xf2084e9f…7b5e` | Match | Confirmed live — `1_020_010` | +| People chain | `0x0b7d4e34…5a5` | Match | Confirmed live — `1_020_011` | + +## Methodology note + +A recurring pattern across this audit: the convenient check and the correct +check gave different answers more than once. RPC endpoints and quick version +queries reflect what a node *believes*, which can lag or mislead; the +chain's own storage — the executed record, not its self-report — was +treated as the only ground truth whenever the two were in tension. This +discipline is what surfaced findings 4 and 6, both of which a faster, more +trusting process would have missed. + +The mainnet Sudo key used to authorize each upgrade is held offline and was +verified against the chain's own `Sudo::Key` storage before any transaction +was signed with it — not assumed correct from a previously recorded +reference, which in this case had itself gone stale. + +A secondary, host-specific pitfall is worth recording for future +verification passes: on at least one build host, the `root` account's +default Rust toolchain differed from the pinned one the CI service account +actually uses, producing a wrong-toolchain (not wrong-path) hash mismatch +during a rebuild attempt. Verification builds on any host should confirm +`rustc --version` matches `rust-toolchain.toml` for the account actually +running the build, not assume it from another account on the same machine.