pezkuwi-sdk/docs/audits/2026-07-mainnet-upgrade.md

PezkuwiChain mainnet runtime upgrade — audit report

Scope: pezkuwi-DKS relay, Asset Hub, and People chain Window: 2026-07-08 – 2026-07-10 Findings: 6 (0 unresolved) Fund impact: none — zero loss, zero irreversible error Status: all three chains upgraded and independently verified

Executive summary

This is a full account of a routine mainnet runtime upgrade that became a multi-stage engineering exercise: six real defects were found across the runtime source, the release pipeline, and the verification tooling itself — three pre-existing in the deployed runtime, two in the CI/release infrastructure, and one in this audit's own first verification attempt. Every defect was caught before it reached mainnet funds, because each step of RELEASE.md's procedure was independently re-verified rather than trusted on the first pass. No user funds were lost or put at risk at any point.

Final chain status

Verified against on-chain System::LastRuntimeUpgrade, not RPC display alone (finding 6, below, is exactly why that distinction matters).

Chain	Before	After	Notes
Relay	1_020_007	1_020_010	Clean on first attempt
Asset Hub	1_020_008	1_020_010	First attempt was a silent no-op (finding 6); corrected and reapplied
People chain	1_020_009	1_020_011	Clean on first attempt

Procedure followed

docs/RELEASE.md's 8-step fund-critical runbook:

1. Freeze & bump — change set frozen on main; spec_version bumped per runtime.
2. Heavy suite — full test suite, workspace clippy, and benches to green before any tag.
3. Live-state dry run — migrations rehearsed with try-runtime against a snapshot of real mainnet state.
4. Tag & build — a signed tag triggers a reproducible WASM build, hash, and SLSA build-provenance attestation.
5. Independent reproduction — the exact release artifact rebuilt from scratch, on separate hardware, before it is ever trusted.
6. On-chain authorization — system.authorizeUpgrade(hash) then system.applyAuthorizedUpgrade(wasm), gated on explicit human sign-off per chain.
7. Post-upgrade verification — confirmed against the chain's own execution record, not just its RPC display.
8. Watch — confirmed blocks kept producing and finalizing on all three chains after each step.

Findings

In the order each was discovered; severity reflects what would have happened had it shipped unnoticed.

1. Staking-tier durations assumed the wrong block time — Medium

What: The staking-score pallet's tier-duration constants (MONTH_IN_BLOCKS, HOUR_IN_BLOCKS) had been switched to assume 12-second blocks (5/min) in an earlier revision, citing an unused same-named constant from a different crate. people-pezkuwichain actually wires its HOURS/DAYS/SlotDuration from a 6-second-block source.

Impact: Every staking duration tier filled twice as fast as intended — a silent, compounding economic drift, not a crash risk.

Confirmed via: live mainnet empirical block time, the actual import path in people-pezkuwichain/src/lib.rs, and a try-runtime live-state dry-run that panicked with an exact 2x slot mismatch under the wrong assumption.

Fix: restored to 10 blocks/min (6s), matching reality (commit 16cbe65).

2. Council pallet's storage version was never stamped, next to a migration that could have deleted it — High

What: Council's on-chain storage version had been stuck at its genesis value since deployment — the version-bump hook never ran. Sitting next to it in the runtime's migration list was a RemovePallet<CouncilPalletName> entry, almost certainly inherited unmodified from an upstream reference chain where no council pallet was active under that name.

Impact: Council is a live, actively-configured pallet on this chain. Had that removal migration ever executed, it would have deleted a governance pallet currently in use.

Fix: replaced the removal with a no-op VersionedMigration<0,4,...> that only stamps the correct storage version; deleted the removal entry entirely (commit 0fd85f6).

3. 12.49M HEZ sat in undecodable "ghost" holds from a removed pallet — High

What: Twenty-five accounts carried a Balances::Holds entry tagged with a hold-reason discriminant that no longer corresponds to any pallet in the current runtime — left behind when an old staking pallet was removed without releasing its holds. Any code path that tried to decode these accounts' holds would panic.

Impact: a latent crash risk on any future storage-decoding operation touching these accounts, and 12.49M HEZ nominally "locked" against nothing.

Investigation: pulled the live RuntimeHoldReason enum directly from on-chain metadata and hand-decoded the raw SCALE bytes for all 25 accounts. Most had little or no real reserved balance behind the stale record — the funds had already found their way loose through normal channels over time, leaving only a bookkeeping ghost.

Fix: a migration that releases exactly min(claimed, actually-reserved) per account and always clears the stale record — never invents balance that isn't really there (commit 8c9fb93). Dry-run against a full live-state snapshot: 0 errors, all 63 pallets' try-state checks passed.

4. Release pipeline could publish a test-only WASM variant as the production relay runtime — High

What: the relay runtime's build.rs always compiles two WASM blobs side by side: the real one, and a fast-runtime-feature devnet variant with shortened governance and staking periods, meant only for local test networks. The release workflow's artifact-selection step (find ... | head -1) picked whichever file its directory listing happened to return first — not a reliable distinction between the two.

Impact: on the first release build of this upgrade, the picker selected the devnet variant. Its hash was computed, signed, and staged as the official relay artifact. Had it gone unnoticed past the independent-reproduction step, authorizing it on-chain would have silently swapped mainnet's governance and staking timers for local-test values.

Caught by: step 5 of the procedure (independent, from-scratch rebuild), which surfaced a hash mismatch against the published artifact once the correct file was identified by hand.

Fix: the picker now explicitly excludes the devnet variant and fails loudly, rather than guessing, if more than one candidate remains (PR #27). Never published, never authorized — caught entirely inside the verification stage.

5. Release builds were not yet reproducible across the CI fleet — Medium

What: these builds are pinned-toolchain reproducible, not yet fully hermetic (a known, already-documented limitation pending upstream srtool support for the pinned Rust version). The self-hosted build fleet doesn't share one filesystem layout, and the absolute checkout path measurably changes the compiled WASM's bytes. Two idle runners building the identical commit produced different hashes.

Impact: not a correctness bug in the shipped code, but a real trust-and-verifiability gap: an outside party rebuilding at a different path than the one that happened to build the release would see a mismatched hash and could reasonably suspect tampering that wasn't there.

Fix: release builds are now pinned to one specific, named machine, making the published hash reliably reproducible until hermetic builds are available (PR #28).

6. Asset Hub's version number was never bumped — its first upgrade attempt did nothing — High

What: the runtime bundle for this release rebuilt and redeployed Asset Hub's WASM, but its source-level spec_version was left unchanged. The on-chain code was successfully replaced — but because the chain's own version bookkeeping saw no version change, it correctly ran no migrations and left its execution behavior untouched.

Impact: none to funds or safety — the swap was between two builds the chain considered identical. But the upgrade was, in practice, a no-op.

Caught by: post-upgrade verification checking the chain's own execution record (System::LastRuntimeUpgrade) rather than trusting that a successful transaction implies a successful upgrade.

Fix: version bumped in source (PR #29), rebuilt, independently re-verified, and reapplied as a second, corrected on-chain transaction. Confirmed genuine on the second attempt via the same strict check.

Independent verification, chain by chain

Every published hash was rebuilt from the tagged source and matched, byte for byte, before being trusted.

Runtime	Published blake2-256	Rebuilt independently	On-chain result
Relay (pezkuwichain-runtime)	0x00399247…7494	Match	Confirmed live — 1_020_010
Asset Hub (corrected build)	0xf2084e9f…7b5e	Match	Confirmed live — 1_020_010
People chain	0x0b7d4e34…5a5	Match	Confirmed live — 1_020_011

Methodology note

A recurring pattern across this audit: the convenient check and the correct check gave different answers more than once. RPC endpoints and quick version queries reflect what a node believes, which can lag or mislead; the chain's own storage — the executed record, not its self-report — was treated as the only ground truth whenever the two were in tension. This discipline is what surfaced findings 4 and 6, both of which a faster, more trusting process would have missed.

The mainnet Sudo key used to authorize each upgrade is held offline and was verified against the chain's own Sudo::Key storage before any transaction was signed with it — not assumed correct from a previously recorded reference, which in this case had itself gone stale.

A secondary, host-specific pitfall is worth recording for future verification passes: on at least one build host, the root account's default Rust toolchain differed from the pinned one the CI service account actually uses, producing a wrong-toolchain (not wrong-path) hash mismatch during a rebuild attempt. Verification builds on any host should confirm rustc --version matches rust-toolchain.toml for the account actually running the build, not assume it from another account on the same machine.