Closes#4248
Impose additional constraint on configuration consistency:
`validation_upgrade_delay` should not be less than or equal to 1.
See the original issue for more details.
This commit refactors the consistency checks. Instead of each individual
setter performs its checks locally, we delegate those checks to the
already existing function `check_consistency`. This removes duplication
and simplifies the logic.
A motivating example of this one is the next PR in the stack that will
introduce a check for a field, which validity depends on the validity of
other two fields. Without this refactoring we will have to place a check
not only to the field in question, but also to the other two fields so
that if they are changed they do not violate consistency criteria. It's
easy to imagine how this can go unwieldy with the number of checks.
This also adds a test that verifies that the default chain spec host
configuration is consistent.
Refactor the configuration module's initializer_on_new_session in such a
way that it returns the configuration. This would make it inline with
other special initialization routines like `shared`'s or `paras`.
This will be useful in a following PR that will check consistency of the
configuration before setting it.
* parachains: Fix configuration module
Closes#4529Closes#4533
I figured that trying to avoid updates does not really worth it to keep.
This is because we seem to not update the configuration often and when
we do we approach this carefully. Thus possibility of a redundant update
is really negligable. At the same time, if such a redundant update does
happen then the effects of that are really small: just some wasted
storage interactions.
On the other hand, making it work was a little bit annoying. With the
proper fix for the pending updates this would be even more annoying
since now we would have to add combinatorically more cases to test this.
So I figured that I will just scrap that and simplify the code.
* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::configuration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_configuration.rs
* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::configuration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_configuration.rs
* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::configuration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_configuration.rs
* review fixes
Co-authored-by: Parity Bot <admin@parity.io>
* pvf-precheck: Integrate PVF pre-checking into paras module
Closes#4009
This is the most of the runtime-side change needed for #3211.
Here is how it works.
The PVF pre-checking can be triggered either by an upgrade or by
onboarding (i.e. calling `schedule_para_initialize`). The PVF
pre-checking process is identified by the PVF code hash that is being
voted on. If there is already PVF pre-checking process running, then no
new PVF pre-checking process will be started. Instead, we just subscribe
to the existing one.
If there is no PVF pre-checking process running but the PVF code hash
was already saved in the storage, that necessarily means (I invite the
reviewers to double-check this invariant) that the PVF already passed
pre-checking. This is equivalent to instant approving of the PVF.
The pre-checking process can be concluded either by obtaining a
supermajority or if it expires.
Each validator checks the list of PVFs available for voting. The vote is
binary, i.e. accept or reject a given PVF. As soon as the supermajority
of votes are collected for one of the sides of the vote, the voting is
concluded in that direction and the effects of the voting are enacted.
Only validators from the active set can participate in the vote. The set
of active validators can change each session. That's why we reset the
votes each session. A voting that observed a certain number of sessions
will be rejected.
The effects of the PVF accepting depend on the operations requested it:
1. All onboardings subscribed to the approved PVF pre-checking process will
get scheduled and after passing 2 session boundaries they will be onboarded.
2. All upgrades subscribed to the approved PVF pre-checking process will
get scheduled very similarly to the existing process. Upgrades with
pre-checking are really the same process that is just delayed by the
time required for pre-checking voting. In case of instant approval the
mechanism is exactly the same. This is important from parachains
compatibility standpoint since following the delayed upgrade requires
the parachain to implement
https://github.com/paritytech/cumulus/pull/517.
In case, PVF pre-checking process was concluded with rejection, then all
the requesting operations get cancelled. For onboarding it means it gets
without movement: the lifecycle of such parachain is terminated on the
`Onboarding` state and after rejection the lifecycle is none. That in
turn means that the caller can attempt registering the parachain once
more. For upgrading it means that the upgrade process is aborted: that
flashes go-ahead signal with `Abort` flag.
Rejection leads to removing the allegedly bad validation code from the
chain storage. Among other things, this implies that the operation can
be re-requested. That allows for retrying an operation in case there was
some bug. At the same time it does not look as a DoS vector due to the
caching performed by the nodes.
PVF pre-checking can be enabled and disabled. Initially, according to
the changes in #4420, this mechanism is disabled. Triggering the PVF
pre-checking when it is disabled just means that we insta approve the
requesting operation. This should lead to the behavior being unchanged.
Follow-ups:
- expose runtime APIs
* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras.rs
* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_paras.rs
* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras.rs
* cargo run --quiet --release --features runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/rococo/src/weights/runtime_parachains_paras.rs
* Review fixes
Co-authored-by: Parity Bot <admin@parity.io>
This PR is a part of #3211.
This PR adds three new fields into the `HostConfiguration` structure.
The fields are going to be used in PRs down the stack.
This change requires migration, so this PR performs runtime storage
migration for configuration module from version 1 to version 2.
This PR closes#4010 and subsumes #4177.
* Introduce new config: ump_max_individual_weight
* Implement overweight msg stashing
* Test
* Add migration module.
Also introduces a test for migration
* Integrate ExecuteOverweightOrigin to runtimes
* Fix more stuff
* Add `yeet` into dictionary
* Use suggested `Error` variant names
* typo
* Use 20ms as the maximum individual message weight
* Update the test value
* rustfmt
* Clean up
* Remove deprecated field from host config
* Remove missed _hrmp_open_request_ttl
* Apply typo fix suggestion
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
* Rename `migration::migrate_to_latest`
* Restore `_hrmp_open_request_ttl` in `v0::HostConfiguration`
* Apply suggestion for a rustdoc
* Apply the suggestion
* Test v0 config with the raw production data fetched from Kusama
* Update runtime/parachains/src/ump.rs
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
* Expose migration functions
* Fix spellcheck
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
* Do not expire HRMP open channel requests
* Fix the build and update the docs
* Implement canceling requests and do not remove them automatically
* Fix a borked merge
* Fix fmt
* Please spellchecker
* Apply suggestions from code review
Co-authored-by: Amar Singh <asinghchrony@protonmail.com>
* Use `mutate_exists` for maintaining request counts
* Apply `rustfmt`
* Move newly introduced entrypoint to end to preserve ordering
Co-authored-by: Amar Singh <asinghchrony@protonmail.com>
* Change pallet naming in test-runtime
This is required to make the `well_know_keys::ACTIVE_CONFIG` match the
`ActiveConfig` key.
* Use correct name
* 🤦
* gotta migrate them all
* migrate rococo construct_runtime
* trigger ci
* fix warnings
* get mocks to work
* add pallet to test runtime
* comments
* calm down mr tabrizi lol
* CI: add spellcheck
* revert me
* CI: explicit command for spellchecker
* spellcheck: edit misspells
* CI: run spellcheck on diff
* spellcheck: edits
* spellcheck: edit misspells
* spellcheck: add rules
* spellcheck: mv configs
* spellcheck: more edits
* spellcheck: chore
* spellcheck: one more thing
* spellcheck: and another one
* spellcheck: seems like it doesn't get to an end
* spellcheck: new words after rebase
* spellcheck: new words appearing out of nowhere
* chore
* review edits
* more review edits
* more edits
* wonky behavior
* wonky behavior 2
* wonky behavior 3
* change git behavior
* spellcheck: another bunch of new edits
* spellcheck: new words are koming out of nowhere
* CI: finding the master
* CI: fetching master implicitly
* CI: undebug
* new errors
* a bunch of new edits
* and some more
* Update node/core/approval-voting/src/approval_db/v1/mod.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* Update xcm/xcm-executor/src/assets.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* Apply suggestions from code review
Co-authored-by: Andronik Ordian <write@reusable.software>
* Suggestions from the code review
* CI: scan only changed files
Co-authored-by: Andronik Ordian <write@reusable.software>
* guide: max_validators
* guide: new approach
* restrict validators based on configurable max
* add some tests
* fix wasm build (Vec)
* clean up warnings
* add logging
* set validator indices in tests as well
* fix common tests
* Update runtime/parachains/src/util.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Andronik Ordian <write@reusable.software>
* initial implementation of lifecycles and upgrades
* clean up a bit
* fix doc comment
* more rigid lifecycle checks
* include paras which are transitioning, and lifecycle query
* format guide
* update api
* update guide
* explicit outgoing state, fix genesis
* handle outgoing with transitioning paras
* do not include transitioning paras in identifier
* Update roadmap/implementers-guide/src/runtime/paras.md
* Update roadmap/implementers-guide/src/runtime/paras.md
* Update roadmap/implementers-guide/src/runtime/paras.md
* Apply suggestions from code review
* Use matches macro
* Correct terms
* Apply suggestions from code review
* actions queue
* Revert "actions queue"
This reverts commit b2e9011ec8937d6c73e99292416c9692aeb30f73.
* collapse onboarding state
* starting actions queue
* consolidate actions queue
* schedule para initialize result
* more actions queue for upgrade/downgrade
* clean up with fully implemented actions queue
* fix tests
* fix scheduler tests
* fix hrmp tests
* fix test
* doc fixes
* fix hrmp test w/ valid para
* Update paras.md
* fix paras registrar
* Update propose_parachain.rs
* fix merge
* Introduce "shared" module
* fix rococo build
* fix up and use shared
* guide updates
* add shared config to common tests
* add shared to test-runtime
* remove println
* fix note
Co-authored-by: Gavin Wood <gavin@parity.io>
* Add well_known_keys
* Reorder HrmpChannel and HostConfiguration members
* abridged versions and well known keys tests
* Add some comments
* Add a note on generation of the prefixes
and other magic values
* Recommend accessing the well known values through abridged structs
* Adds consistency checks for the `HostConfiguration`
Besides that it fixes the chain specs to make the consistency checks happy.
* Update runtime/parachains/src/configuration.rs
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
* Review feedback and test fixes etc
* Update
* More
* I'm an idiot
* Fix tests...
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
* guide: fix formatting for SessionInfo module
* primitives: SessionInfo type
* punt on approval keys
* ah, revert the type alias
* session info runtime module skeleton
* update the guide
* runtime/configuration: sync with the guide
* runtime/configuration: setters for newly added fields
* runtime/configuration: set codec indexes
* runtime/configuration: update test
* primitives: fix SessionInfo definition
* runtime/session_info: initial impl
* runtime/session_info: use initializer for session handling (wip)
* runtime/session_info: mock authority discovery trait
* guide: update the initializer's order
* runtime/session_info: tests skeleton
* runtime/session_info: store n_delay_tranches in Configuration
* runtime/session_info: punt on approval keys
* runtime/session_info: add some basic tests
* Update primitives/src/v1.rs
* small fixes
* remove codec index annotation on structs
* fix off-by-one error
* validator_discovery: accept a session index
* runtime: replace validator_discovery api with session_info
* Update runtime/parachains/src/session_info.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* runtime/session_info: add a comment about missing entries
* runtime/session_info: define the keys
* util: expose connect_to_past_session_validators
* util: allow session_info requests for jobs
* runtime-api: add mock test for session_info
* collator-protocol: add session_index to test state
* util: fix error message for runtime error
* fix compilation
* fix tests after merge with master
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* HRMP: Update the impl guide
* HRMP: Incorporate the channel notifications into the guide
* HRMP: Renaming in the impl guide
* HRMP: Constrain the maximum number of HRMP messages per candidate
This commit addresses the HRMP part of https://github.com/paritytech/polkadot/issues/1869
* XCM: Introduce HRMP related message types
* HRMP: Data structures and plumbing
* HRMP: Configuration
* HRMP: Data layout
* HRMP: Acceptance & Enactment
* HRMP: Test base logic
* Update adder collator
* HRMP: Runtime API for accessing inbound messages
Also, removing some redundant fully-qualified names.
* HRMP: Add diagnostic logging in acceptance criteria
* HRMP: Additional tests
* Self-review fixes
* save test refactorings for the next time
* Missed a return statement.
* a formatting blip
* Add missing logic for appending HRMP digests
* Remove the channel contents vectors which became empty
* Tighten HRMP channel digests invariants.
* Apply suggestions from code review
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Remove a note about sorting for channel id
* Add missing rustdocs to the configuration
* Clarify and update the invariant for HrmpChannelDigests
* Make the onboarding invariant less sloppy
Namely, introduce `Paras::is_valid_para` (in fact, it already is present
in the implementation) and hook up the invariant to that.
Note that this says "within a session" because I don't want to make it
super strict on the session boundary. The logic on the session boundary
should be extremely careful.
* Make `CandidateCheckContext` use T::BlockNumber for hrmp_watermark
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Parachain improvements
- Set the parachains configuration in Rococo genesis
- Don't stop the overseer when a subsystem job is stopped
- Several small code changes
* Remove unused functionality
* Return error from the runtime instead of printing it
* Apply suggestions from code review
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Update primitives/src/v1.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Update primitives/src/v1.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Fix test
* Revert "Update primitives/src/v1.rs"
This reverts commit 11fce2785acd1de481ca57815b8e18400f09fd52.
* Revert "Update primitives/src/v1.rs"
This reverts commit d6439fed4f954360c89fb1e12b73954902c76a41.
* Revert "Return error from the runtime instead of printing it"
This reverts commit cb4b5c0830ac516a6d54b2c24197e9354f2b98cb.
* Revert "Fix test"
This reverts commit 0c5fa1b5566d4cd3c55a55d485e707165ce7a59e.
* Update runtime/parachains/src/runtime_api_impl/v1.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* UMP: Update the impl guide
* UMP: Incorporate XCM related changes into the guide
* UMP: Data structures and configuration
* UMP: Initial plumbing
* UMP: Data layout
* UMP: Acceptance criteria & enactment
* UMP: Fix dispatcher bug and add the test for it
* UMP: Constrain the maximum size of an UMP message
This commit addresses the UMP part of https://github.com/paritytech/polkadot/issues/1869
* Fix failing test due to misconfiguration
* Make the type of RelayDispatchQueueSize be more apparent in the guide
* Revert renaming `max_upward_queue_capacity` to `max_upward_queue_count`
* convert spaces to tabs
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* Update runtime/parachains/src/router/ump.rs
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* DMP: data structures and plumbing
* DMP: Implement DMP logic in the router module
DMP: Integrate DMP parts into the inclusion module
* DMP: Introduce the max size limit for the size of a downward message
* DMP: Runtime API for accessing inbound messages
* OCD
Small clean ups
* DMP: fix the naming of the error
* DMP: add caution about a non-existent recipient
* update primitives
* correct parent_head field
* make hrmp field pub
* refactor validation data: runtime
* refactor validation data: messages
* add arguments to full_validation_data runtime API
* port runtime API
* mostly port over candidate validation
* remove some parameters from ValidationParams
* guide: update candidate validation
* update candidate outputs
* update ValidationOutputs in primitives
* port over candidate validation
* add a new test for no-transient behavior
* update util runtime API wrappers
* candidate backing
* fix missing imports
* change some fields of validation data around
* runtime API impl
* update candidate validation
* fix backing tests
* grumbles from review
* fix av-store tests
* fix some more crates
* fix provisioner tests
* fix availability distribution tests
* port collation-generation to new validation data
* fix overseer tests
* Update roadmap/implementers-guide/src/node/utility/candidate-validation.md
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* rename GlobalValidationSchedule to GlobalValidationData
* guide: update candidate descriptor to contain validation data hash
* guide: add note in inclusion module about checking validation data hash
* primitives: update CandidateDescriptor to contain new hash
* fix payload computation
* add helpers for computing validation data to runtime modules
* guide: note routines
* inclusion: check validation data hash and fix local_validation_data bug
* add a case to candidate_checks and improve that test substantially
* bump versions
* address review comments
* add a test for including code upgrade
* bump kusama version
* bump westend & polkadot versions
* runtime: rename parachain_rotation_frequency to group_rotation_frequency
* scheduler: handle timeouts for no group rotation
* scheduler: apply fixes from code review
* scheduler: remove my comments
* create a v1 primitives module
* Improve guide on availability types
* punctuate
* new parachains runtime uses new primitives
* tests of new runtime now use new primitives
* add ErasureChunk to guide
* export erasure chunk from v1 primitives
* subsystem crate uses v1 primitives
* node-primitives uses new v1 primitives
* port overseer to new primitives
* new-proposer uses v1 primitives (no ParachainHost anymore)
* fix no-std compilation for primitives
* service-new uses v1 primitives
* network-bridge uses new primitives
* statement distribution uses v1 primitives
* PoV distribution uses v1 primitives; add PoV::hash fn
* move parachain to v0
* remove inclusion_inherent module and place into v1
* remove everything from primitives crate root
* remove some unused old types from v0 primitives
* point everything else at primitives::v0
* squanch some warns up
* add RuntimeDebug import to no-std as well
* port over statement-table and validation
* fix final errors in validation and node-primitives
* add dummy Ord impl to committed candidate receipt
* guide: update CandidateValidationMessage
* add primitive for validationoutputs
* expand CandidateValidationMessage further
* bikeshed
* add some impls to omitted-validation-data and available-data
* expand CandidateValidationMessage
* make erasure-coding generic over v1/v0
* update usages of erasure-coding
* implement commitments.hash()
* use Arc<Pov> for CandidateValidation
* improve new erasure-coding method names
* fix up candidate backing
* update docs a bit
* fix most tests and add short-circuiting to make_pov_available
* fix remainder of candidate backing tests
* squanching warns
* squanch it up
* some fallout
* overseer fallout
* free from polkadot-test-service hell
* remove IsCallable make use of in-origin filter
* update lock
* bump version for runtimes as spec as changed
* trigger CI
* Revert "trigger CI"
This reverts commit 5ac58fd42ac50dfb2fcd41ca866c7f6a605c5112.
* scheduler module skeleton
* update scheduler skeleton to match latest version of guide
* better session change notification
* add mock randomness and fix test compilation
* shuffle validators into groups
* finish implementing session change logic for scheduler
* tweak core assignment type to track retries of parathread
* reframe queued parathread core as offset
* implement initialzation and finalization routines
* implement parathread claim queuing
* implement core_para
* implement the group_validators routine and fix errors
* add a reason for freeing cores
* implement `schedule` function
* add some docs to the scheduled function
* implement `occupied` helper
* implement availability predicate
* fix some warnings
* integrate scheduler into initializer
* integrate scheduler into mock module
* avoid conflict with Substrate's scheduler storage
* add parathreads index to paras module
* implement parathreads map in paras module
* add is_parathread to paras
* test adding parathread claim
* test that you cannot add claims when no parathread cores exist
* check session change parathread queue pruning
* test validator shuffling
* add allow_unused to scheduler items
* add test for scheduling
* add some more tests for scheduling logic
* test core rotation
* check parathread claim pruning after retries
* add bound notes
* Apply suggestions from code review
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* more suggestions from review
* test availability predicate, add box to please compiler
* add changes to guide
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* file structure and initializer skeleton
* ensure session changes happen before initialization
* add a couple tests for initializer flow
* integrate with session handling
* configuration update logic
* configuration methods
* move test mock to its own module
* integrate configuration into initializer
* add note about initialization order
* integrate configuration module into mock
* add some tests for config module
* paras module storage
* implement paras session change operation
* amend past code pruning to fully cover acceptance period
* update guide again
* do pruning of historical validation code
* add weight to initialization
* integrate into mock & leave notes for next session
* clean up un-ended sentence
* alter test to account for double index in past code meta
* port over code-at logic test
* clarify checking for conflicting code upgrades
* add genesis for paras, include in mock, ensure incoming paras are processed
* note on return value of `validation_code_at`
* implement paras routines from implementor's guide
* bring over some existing tests and begin porting
* port over code upgrade tests
* test parachain registration
* test code_at with intermediate block
* fix warnings
* clean up docs and extract to separate struct
* adjust implementor's guide to include replacementtimes
* kill stray println
* rename expected_at to applied_after
* rewrite ParaPastCodeMeta to avoid reversal
* clarify and test interface of validation_code_at
* make FutureCode optional
* rename do_old_code_pruning
* add comment on Option<()> to answer FAQ
* address some more grumbles
Sergei Shulepov