2026-10-26 21:49:54 +00:00 · 2026-07-24 11:42:50 -07:00
1 parent 82543bdfab
commit d285d59108
@@ -1,191 +1,110 @@
# Release
# Release & runtime-upgrade runbook
The outputs of a stable release are:
PezkuwiChain holds real user funds. A runtime upgrade is the single most dangerous
operation we perform — a bad migration or a wrong WASM can corrupt state or brick the
chain. This runbook is the required, repeatable process. Do not skip steps.
- The binaries:
   - `pezkuwi`
   - `pezkuwi-execute-worker`
   - `pezkuwi-prepare-worker`
   - `pezkuwi-teyrchain`
   - `pezkuwi-omni-node`
   - `chain-spec-builder`
   - `frame-omni-bencher`
## What a release produces
   built for the `x86_64-unknown-linux-gnu` and `aarch64-apple-darwin` targets. The gpg signatures and sha256 checksums.
- **Runtime WASM blobs** for the deployable runtimes (relay `pezkuwichain`, system
  teyrchains `asset-hub-pezkuwichain`, `people-pezkuwichain`), built by
  `.github/workflows/release.yml`.
- For each blob: the **blake2-256 hash** (the value `authorizeUpgrade` commits to),
  the sha256, and an **SLSA build-provenance attestation**.
- A **draft GitHub release** (human review before publishing a fund-critical artifact).
- The runtimes for Zagros and its system teyrchains.
- The new crate versions published to `crates.io`.
- Debian package for the Pezkuwi binary.
- Docker images for:
   - `pezkuwi` (includes `pezkuwi-execute-worker` & `pezkuwi-prepare-worker`)
   - `pezkuwi-teyrchain`
   - `pezkuwi-omni-node`
   - `chain-spec-builder`
The node binary is built from the same tag with the pinned toolchain.
## CI tiers (what gates what)
# Timeline
`Stable` releases are scheduled on a quarterly basis, usually by the end of the last month of each quarter. The exact
 schedule can be found on the [Release Registry](https://github.com/pezkuwichain/release-registry/).It is possible to
 subscribe to a [calendar link](https://raw.githubusercontent.com/pezkuwichain/release-registry/main/releases-v1.ics)
 to have it in your personal calendar.
| Tier | When | Gates |
|---|---|---|
| Fast gate (`check`, `quick-checks`, `security`) | every push/PR | compile, 6-runtime WASM build, **try-runtime**, fmt, supply-chain (cargo-deny), zepter/taplo |
| Heavy suite (`test`) | nightly + `workflow_dispatch` | full nextest tests, workspace clippy, benches, doc-tests |
| Release (`release`) | tag `v*`/`runtime-*` + `workflow_dispatch` | builds WASM, hashes, attestation, draft release |
Each stable release is supported for a period of one year from its first release. For example, `Pezkuwi stable2412`
was released on `2024-12-17` and its end of life is set to `2025-12-16`.
Before any mainnet upgrade the **heavy suite must be green on the release commit** —
trigger it with `workflow_dispatch` and wait for it; do not rely only on the fast gate.
During this period, each stable release is updated with patch releases, which are scheduled on a monthly basis
and contain fixes for any bugs that may be found.
## spec_version policy
ℹ️ Note: the binaries and runtimes (if needed) are only provided for the latest `stable` release, for the previous
releases only the crates.io release takes place.
- `spec_name` is the chain identity and **must never change** for a live chain.
- `spec_version` is a single monotonically increasing integer (schema `M_mmm_ppp`;
  current relay `1_020_007`). Bump it on **every** runtime change that ships — nodes use
  it to detect the upgrade.
- Bump `transaction_version` only when extrinsic encoding/call indices change.
- `impl_version` is informational.
This three month period between `stable` releases includes a 1.5 month QA period. This means that for each upcoming
`stable` release, the branch from which that release will be made is created 1.5 months before the release date.
This time is used to test the upcoming release candidate and find any issues that may arise with publishing crates
to crates.io, binary artifacts and template synchronisation before the final release. The findings should be fixed
 and backported to the release branch. During this time, multiple release candidates may be built and rolled out.
## Runtime-upgrade procedure (mainnet, fund-critical)
# Setup
1. **Freeze the change set** on `main`; fast gate green.
2. **Bump `spec_version`** (+ `transaction_version` if call encoding changed) in the
   affected runtime(s).
3. **Run the heavy suite** (`workflow_dispatch` → `test.yml`) on that commit → all green.
4. **try-runtime against live state** — dry-run the migrations on a snapshot of the live
   chain and confirm they succeed and post-upgrade invariants hold, *before* touching
   mainnet. (Pre-upgrade gate.)
5. **Tag** `runtime-<name>-vX` → `release.yml` builds the WASM and publishes the
   **blake2-256** hash + sha256 + attestation as a draft release.
6. **Reproduce & verify**: independently rebuild the tag with the pinned toolchain
   (`rust-toolchain.toml`) + committed `Cargo.lock`; confirm the blake2-256 matches.
   Publish the release only after the hash is confirmed.
7. **Authorize on-chain**: governance/sudo `system.authorizeUpgrade(<blake2-256>)`, then
   `system.applyAuthorizedUpgrade(<wasm>)`. The hash authorized on-chain must equal the
   one in the release.
8. **Watch**: confirm blocks are still produced and finalized after the upgrade.
We have two types of branches related to the releases: `master` and `stableYYMM`:
- `master` is the main development branch where normal Pull Requests are opened. Developers need to mostly only care
about this branch.
- `stableYYMM` branch contains a version of the code that is ready to be released. Each `stableYYMM` branch corresponds
to the corresponding stable release, which is in the maintenance or in a QA period. Its contents should be always
audited. Merging to it is restricted to [Backports](#backports).
## Pending bundle (next mainnet spec_version)
# Versioning
These repo changes are ready and green but **not yet deployed**; they ship together in
one coordinated upgrade after the heavy suite is fully green and reviewed:
We are releasing multiple different things from this repository in one release, but we don't want to use the same
version for everything. Thus, in the following we explain the versioning story for the crates, node and Zagros.
- Token symbol standardized to **HEZ** (metadata-hash, claims `Prefix`, chain-spec).
- Claims statements repointed to **statement.pex.network** (hash-pinned; see
  [statement/README.md](statement/README.md)).
- `tiki` pallet: 12 new functional/professional roles.
- RuntimeVersion identity cleanup (dropped `parity-*` / `westmint`).
- **Security-audit fix set** (AI-assisted internal review; commits `14cbf1a`, `9afd99a`
  on `main`) — 11 findings closed across `identity-kyc`, `messaging`, `perwerde`,
  `presale`, `tiki`, `welati`, plus the relay `SendXcmOrigin` narrowing in
  `xcm_config.rs`. No storage migration required (all changes are new storage items or
  Config/logic-only).
  - `staking-score`: noter submissions now require a posted bond
    (`register_as_noter`/`unregister_as_noter`) and sit in a 1-hour dispute window
    (`DisputeWindow`) before taking effect, disputable by any Council member and
    slashable by `RootOrDiwanOrTechnical` on confirmed fraud. Root/XCM-Transact
    submissions are unaffected (chain-authenticated, not a personal key).
  - Fixed `MONTH_IN_BLOCKS`/`HOUR_IN_BLOCKS` in `staking-score`: was silently assuming
    6s blocks (10/min) against this chain's actual 12s (5/min), doubling every
    duration-tier threshold in production.
  - `staking-score`'s weights are now real benchmark output (previously manual
    estimates); `presale::withdraw_funds` had no benchmark case at all until
    commit `dc6d29e` — real weight pending a benchmark run.
- Dependency-security cleanup: `rand`/`tar`/`rpassword` patched (commit `2cbf2b3`);
  `wasmtime` (2 critical, aarch64-specific sandbox-escape CVEs; our fleet is x86_64)
  and a handful of other client-networking deps (`hickory-proto`, `rustls-webpki`,
  `yamux`) remain — each pinned by a deeper Substrate/Polkadot-SDK dependency and
  needing a coordinated major-version bump plus real node testing. Tracked as a
  separate task, not part of this bundle.
To easily refer to a release, it shall be named by its date in the form `Pezkuwi stableYYMM`. Patches to stable releases
are tagged in the form of `Pezkuwi stableYYMM-PATCH`, with `PATCH` ranging from 1 to 99. For example, the fourth patch
to `Pezkuwi stable2409` would be `Pezkuwi stable2409-4`.
**Note on Zagros testnet validation:** the live "Zagros Testnet" chain (217.77.6.126)
runs the `pezkuwichain`/`asset-hub-pezkuwichain`/`people-pezkuwichain` runtime family
(spec_name confirmed via live RPC), not the `zagros`/`asset-hub-zagros`/`people-zagros`
family. The latter is a CI-checked-only reference implementation (never released — see
`release.yml`), left over from before this repo's clean-room rewrite. Bump
`spec_version` and build WASM from the `*-pezkuwichain` crates when validating a bundle
on Zagros before it ships to mainnet; the `*-zagros` crates are not deployed anywhere.
## Crate
## Changing a claim statement
We try to follow [SemVer 2.0.0](https://semver.org/) as best as possible for versioning our crates. The definitions of
`major`, `minor` and `patch` version for Rust crates are slightly altered from their standard for pre `1.0.0` versions.
Quoting [rust-lang.org](https://doc.rust-lang.org/cargo/reference/semver.html):
A statement edit changes its hash → changes the runtime → is a runtime upgrade and
invalidates prior signatures. Procedure: edit `docs/statement/*.html`, redeploy to
`statement.pex.network`, recompute sha256, update `StatementKind::to_text()`, bump
`spec_version`, ship. The repo copy, the served copy, and the runtime hash must stay
identical.
>Initial development releases starting with "0.y.z" can treat changes in "y" as a major release, and "z" as a minor
release. "0.0.z" releases are always major changes. This is because Cargo uses the convention that only changes in the
left-most non-zero component are considered incompatible.
## Reproducibility note
SemVer requires a piece of software to first declare a public API. The public API of the Pezkuwi SDK
is hereby declared as the sum of all crates' public APIs.
Inductively, the public API of our library crates is declared as all public items that are neither:
- Inside a `__private` module
- Documented as "unstable" or "experimental" in the first line of docs
- Bear `unstable` or `experimental` in their absolute path
## Node
The versioning of the Pezkuwi node is done most of the time by only incrementing the `minor` version. The `major`
version is only bumped for special releases and the `patch` is used for a patch release that happens every month and
fixes found issues. The node version is not following SemVer. This means that the version doesn't express if there are
any breaking changes in the CLI interface or similar. The node version is declared
in the [`NODE_VERSION`](https://docs.pezkuwichain.io/sdk/master/polkadot_pez_node_primitives/constant.NODE_VERSION.html)
variable.
## Zagros
For the Westene testnet, in addition to incrementing the `Cargo.toml` version we also increment the `spec_version` and
sometimes the `transaction_version`. The spec version is also following the node version. Its schema is: `M_mmm_ppp` and
for example `1_002_000` is the node release `1.2.0`. This versioning has no further meaning, and is only done to map
from an on chain `spec_version` easily to the release in this repository.
# Backports
**From `master` to `stable`**
Backports in this direction can be anything that is audited and either a `minor` or a `patch` bump.
See [BACKPORT.md](./BACKPORT.md) for more explanation. [Security fixes](#bug-and-security-fix)
should be prioritized over additions or improvements. Crates that are declared as internal API can
also have `major` version bumps through backports.
**From `stable` to `master`**
Backports to `master` only happen after a `stable` or `patch` release has been made for the current stable release,
and include node version and spec version bumps, plus reorganizing the prdoc files
(they should go into the appropriate release folder under the [prdoc](./proc) folder).
This is done by the release team to keep things organized. Developers need not to care about such backports.
# Processes
The following processes are necessary to actualize our releases. Each process has a *Cadence* on which it must execute
and a *Responsible* that is responsible for autonomously doing so and reporting back any error in the *RelEng: Pezkuwi
Release Coordination* Matrix channel. All processes should be automated as much as possible.
## Crate Bumping
Cadence: currently every 3 months for new `stable` releases and monthly for existing `stables`.
Responsible: Developer that opened the Pull Request.
Following SemVer isn't easy, but there exists [a guide](https://doc.rust-lang.org/cargo/reference/semver.html) in the
Rust documentation that explains the small details on when to bump what. This process is supported with a CI check that
utilizes [`cargo-semver-checks`](https://github.com/obi1kenobi/cargo-semver-checks).
### Steps
1. Developer opens a Pull Request with changed crates against `master`.
2. They mention the type of the bump of all changed crates according to SemVer in the prdoc file attached to the PR.
 Note that this includes any crates that expose the changed behaviour in their *public API* and also transitive dependencies
 for whom the same rule applies.
3. The bump itself happens during the release and is done by a release engineer using the
[Parity-Publish](https://github.com/paritytech/parity-publish) tool.
## Stable Release
Cadence: every 3 months for new `stable` releases and monthly for existing `stables`. Responsible: Release Team.
### Steps to execute a new stable binary release
From the main Pezkuwi-sdk repository in the paritytech org:
1. On the cut-off date, create a new branch with the name `satbleYYMM`
  using combined [Branch-off stable/tag rc flow](/.github/workflows/release-10_branchoff-stable.yml)
2. Create a new rc tag from the stable branch using combined
   [Branch-off stable/tag rc flow](/.github/workflows/release-10_branchoff-stable.yml)
ℹ️ These first two steps can be done all in one if there are no extra actions (like crates release) are needed
to be done in between.
In case of a crates release: when it is done, the changes done by the Parity-Publish needs to be revereted and
merged back to the stable branch via a PR as the direct pushes are restricted. When this is done,
the new RC tag can be created using the flow from above.
From the forked Pezkuwi-sdk repository in the [paritytech-release org](https://github.com/paritytech-release/polkadot-sdk/actions):
1. Sync the forks before continuing with the release using
[Sync the forked repo with the upstream](https://github.com/paritytech-release/polkadot-sdk/actions/workflows/fork-sync-action.yml)
2. To build binaries or testnet runtimes and to create a release draft in the github
trigger [Release - Combined Builds Flow](/.github/workflows/release-22_combined-rc-runtime-builds-release-draft.yml)
3. Repeat steps 1 and 2 to prepare a new rc if there are any changes were pushed to release branch
4. When the release is finalized and ready to go, publish crates using `parity-publish` tool and push changes
to the release branch
5. Trigger [Release - Promote RC to final candidate on S3](/.github/workflows/release-31_promote-rc-to-final.yml)
to have it as a final rc on the S3
6. Publish deb package for the `pezkuwi` binary using
[Release - Publish Pezkuwi deb package](/.github/workflows/release-40_publish-deb-package.yml)
7. Adjust the release draft and publish release on GitHub.
8. Publish docker images using [Release - Publish Docker Image](/.github/workflows/release-50_publish-docker.yml)
From the main Pezkuwi-sdk repository in the paritytech org:
1. Synchronize templates using [Synchronize templates](/.github/workflows/misc-sync-templates.yml)
2. Update the [Release Registry](https://github.com/pezkuwichain/release-registry/)
follwoing the [instructions](https://github.com/pezkuwichain/release-registry?tab=readme-ov-file#maintenance)
in the repo with the actual release dates.
## Patch release for the latest stable version
Cadence: every month. Responsible: Developer
Backporting rules can be found in the [BACKPORT.md](/docs/BACKPORT.md)
The release itself is similar to [the new stable release](#steps-to-execute-a-new-stable-release) process without
the branching-off step, as the branch already exists and depending on the patch
(whether it is for the current `stable` release or one of the previous ones) the binary build can be skipped
and only crates and GitHub publishing is done.
Builds are pinned-toolchain reproducible (deterministic given `rust-toolchain.toml` +
`Cargo.lock` + flags). Fully hermetic srtool builds are pending a Rust 1.96 srtool image
(latest published is 1.93); the release pipeline will adopt srtool once available.