Cumulus test-parachain node and test runtime were still using relay
chain consensus and 12s blocktimes. With async backing around the corner
on the major chains we should switch our tests too.
Also needed to nicely test the changes coming to collators in #3168.
### Changes Overview
- Followed the [migration
guide](https://wiki.polkadot.network/docs/maintain-guides-async-backing)
for async backing for the cumulus-test-runtime
- Adjusted the cumulus-test-service to use the correct import-queue,
lookahead collator etc.
- The block validation function now uses the Aura Ext Executor so that
the seal of the block is validated
- Previous point requires that we seal block before calling into
`validate_block`, I introduced a helper function for that
- Test client adjusted to provide a slot to the relay chain proof and
the aura pre-digest
**Update:** Pushed additional changes based on the review comments.
**This pull request fixes various spelling mistakes in this
repository.**
Most of the changes are contained in the first **3** commits:
- `Fix spelling mistakes in comments and docs`
- `Fix spelling mistakes in test names`
- `Fix spelling mistakes in error messages, panic messages, logs and
tracing`
Other source code spelling mistakes are separated into individual
commits for easier reviewing:
- `Fix the spelling of 'authority'`
- `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'`
- `Fix the spelling of 'prev_enqueud_messages'`
- `Fix the spelling of 'endpoint'`
- `Fix the spelling of 'children'`
- `Fix the spelling of 'PenpalSiblingSovereignAccount'`
- `Fix the spelling of 'PenpalSudoAccount'`
- `Fix the spelling of 'insufficient'`
- `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'`
- `Fix the spelling of 'subtracted'`
- `Fix the spelling of 'CandidatePendingAvailability'`
- `Fix the spelling of 'exclusive'`
- `Fix the spelling of 'until'`
- `Fix the spelling of 'discriminator'`
- `Fix the spelling of 'nonexistent'`
- `Fix the spelling of 'subsystem'`
- `Fix the spelling of 'indices'`
- `Fix the spelling of 'committed'`
- `Fix the spelling of 'topology'`
- `Fix the spelling of 'response'`
- `Fix the spelling of 'beneficiary'`
- `Fix the spelling of 'formatted'`
- `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'`
- `Fix the spelling of 'succeeded'`
- `Fix the spelling of 'reopened'`
- `Fix the spelling of 'proposer'`
- `Fix the spelling of 'InstantiationNonce'`
- `Fix the spelling of 'depositor'`
- `Fix the spelling of 'expiration'`
- `Fix the spelling of 'phantom'`
- `Fix the spelling of 'AggregatedKeyValue'`
- `Fix the spelling of 'randomness'`
- `Fix the spelling of 'defendant'`
- `Fix the spelling of 'AquaticMammal'`
- `Fix the spelling of 'transactions'`
- `Fix the spelling of 'PassingTracingSubscriber'`
- `Fix the spelling of 'TxSignaturePayload'`
- `Fix the spelling of 'versioning'`
- `Fix the spelling of 'descendant'`
- `Fix the spelling of 'overridden'`
- `Fix the spelling of 'network'`
Let me know if this structure is adequate.
**Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`,
`Merkelization`, `Merkleization`, is somewhat inconsistent but I left it
as it is.
~~**Note:** In some places the term `Receival` is used to refer to
message reception, IMO `Reception` is the correct word here, but I left
it as it is.~~
~~**Note:** In some places the term `Overlayed` is used instead of the
more acceptable version `Overlaid` but I also left it as it is.~~
~~**Note:** In some places the term `Applyable` is used instead of the
correct version `Applicable` but I also left it as it is.~~
**Note:** Some usage of British vs American english e.g. `judgement` vs
`judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc.
are both present in different places, but I suppose that's
understandable given the number of contributors.
~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it
triggers errors in CI when I make changes to it, so I left it as it
is.~~
This introduces a check to ensure that the parachain code matches the
validation code stored in the relay chain state. If not, it will print a
warning. This should be mainly useful for parachain builders to make
sure they have setup everything correctly.
This PR adds the `delayed_best_block` flag to `ParachainBlockImport`. If
not set, the `params.fork_choice` is not updated (to
`ForkChoiceStrategy::Custom`) during the block import.
When `delayed_best_block` is set to `false` all parachain blocks on the
[longest
fork](https://github.com/paritytech/polkadot-sdk/blob/552be4800d9e4b480f79a300fc531783e04be099/substrate/client/service/src/client/client.rs#L708-L709)
will be notified as the best block, allowing transaction pool to be
updated with every imported block.
Otherwise imported blocks will not be notified as best blocks
(`fork_choice=ForkChoiceStrategy::Custom(false)`), and transaction pool
would be updated only with best block received from relay-chain.
Improvement for: #1202
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
When warp syncing a node we import a header of the parachain around the
tip of the chain. This header is currently not imported as finalized
block (should be fixed at some point as well), the parent headers are
not yet present (still being synced) and thus, we run into a panic. Even
if there is a case where a leaf could not be found in the database, this
probably means that the db is broken and it will fail somewhere elese.
* Update substrate & polkadot
* min changes to make async backing compile
* (async backing) parachain-system: track limitations for unincluded blocks (#2438)
* unincluded segment draft
* read para head from storage proof
* read_para_head -> read_included_para_head
* Provide pub interface
* add errors
* fix unincluded segment update
* BlockTracker -> Ancestor
* add a dmp limit
* Read para head depending on the storage switch
* doc comments
* storage items docs
* add a sanity check on block initialize
* Check watermark
* append to the segment on block finalize
* Move segment update into set_validation_data
* Resolve para head todo
* option watermark
* fix comment
* Drop dmq check
* fix weight
* doc-comments on inherent invariant
* Remove TODO
* add todo
* primitives tests
* pallet tests
* doc comments
* refactor unincluded segment length into a ConsensusHook (#2501)
* refactor unincluded segment length into a ConsensusHook
* add docs
* refactor bandwidth_out calculation
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
* test for limits from impl
* fmt
* make tests compile
* update comment
* uncomment test
* fix collator test by adding parent to state proof
* patch HRMP watermark rules for unincluded segment
* get consensus-common tests to pass, using unincluded segment
* fix unincluded segment tests
* get all tests passing
* fmt
* rustdoc CI
* aura-ext: limit the number of authored blocks per slot (#2551)
* aura_ext consensus hook
* reverse dependency
* include weight into hook
* fix tests
* remove stray println
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
* fix test warning
* fix doc link
---------
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
Co-authored-by: Chris Sosnin <chris125_@live.com>
* parachain-system: ignore go ahead signal once upgrade is processed (#2594)
* handle goahead signal for unincluded segment
* doc comment
* add test
* parachain-system: drop processed messages from inherent data (#2590)
* implement `drop_processed_messages`
* drop messages based on relay parent number
* adjust tests
* drop changes to mqc
* fix comment
* drop test
* drop more dead code
* clippy
* aura-ext: check slot in consensus hook and remove all `CheckInherents` logic (#2658)
* aura-ext: check slot in consensus hook
* convert relay chain slot
* Make relay chain slot duration generic
* use fixed velocity hook for pallets with aura
* purge timestamp inherent
* fix warning
* adjust runtime tests
* fix slots in tests
* Make `xcm-emulator` test pass for new consensus hook (#2722)
* add pallets on_initialize
* tests pass
* add AuraExt on_init
* ".git/.scripts/commands/fmt/fmt.sh"
---------
Co-authored-by: command-bot <>
---------
Co-authored-by: Ignacio Palacios <ignacio.palacios.santos@gmail.com>
* update polkadot git refs
* CollationGenerationConfig closure is now optional (#2772)
* CollationGenerationConfig closure is now optional
* fix test
* propagate network-protocol-staging feature (#2899)
* Feature Flagging Consensus Hook Type Parameter (#2911)
* First pass
* fmt
* Added as default feature in tomls
* Changed to direct dependency feature
* Dealing with clippy error
* Update pallets/parachain-system/src/lib.rs
Co-authored-by: asynchronous rob <rphmeier@gmail.com>
---------
Co-authored-by: asynchronous rob <rphmeier@gmail.com>
* fmt
* bump deps and remove warning
* parachain-system: update RelevantMessagingState according to the unincluded segment (#2948)
* mostly address 2471 with a bug introduced
* adjust relevant messaging state after computing total
* fmt
* max -> min
* fix test implementation of xcmp source
* add test
* fix test message sending logic
* fix + test
* add more to unincluded segment test
* fmt
---------
Co-authored-by: Chris Sosnin <chris125_@live.com>
* Integrate new Aura / Parachain Consensus Logic in Parachain-Template / Polkadot-Parachain (#2864)
* add a comment
* refactor client/service utilities
* deprecate start_collator
* update parachain-template
* update test-service in the same way
* update polkadot-parachain crate
* fmt
* wire up new SubmitCollation message
* some runtime utilities for implementing unincluded segment runtime APIs
* allow parachains to configure their level of sybil-resistance when starting the network
* make aura-ext compile
* update to specify sybil resistance levels
* fmt
* specify relay chain slot duration in milliseconds
* update Aura to explicitly produce Send futures
also, make relay_chain_slot_duration a Duration
* add authoring duration to basic collator and document params
* integrate new basic collator into parachain-template
* remove assert_send used for testing
* basic-aura: only author when parent included
* update polkadot-parachain-bin
* fmt
* some fixes
* fixes
* add a RelayNumberMonotonicallyIncreases
* add a utility function for initializing subsystems
* some logging for timestamp adjustment
* fmt
* some fixes for lookahead collator
* add a log
* update `find_potential_parents` to account for sessions
* bound the loop
* restore & deprecate old start_collator and start_full_node functions.
* remove unnecessary await calls
* fix warning
* clippy
* more clippy
* remove unneeded logic
* ci
* update comment
Co-authored-by: Marcin S. <marcin@bytedude.com>
* (async backing) restore `CheckInherents` for backwards-compatibility (#2977)
* bring back timestamp
* Restore CheckInherents
* revert to empty CheckInherents
* make CheckInherents optional
* attempt
* properly end system blocks
* add some more comments
* ignore failing system parachain tests
* update refs after main feature branch merge
* comment out the offending tests because CI runs ignored tests
* fix warnings
* fmt
* revert to polkadot master
* cargo update -p polkadot-primitives -p sp-io
---------
Co-authored-by: asynchronous rob <rphmeier@gmail.com>
Co-authored-by: Ignacio Palacios <ignacio.palacios.santos@gmail.com>
Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com>
Co-authored-by: Marcin S. <marcin@bytedude.com>
Co-authored-by: eskimor <eskimor@users.noreply.github.com>
Co-authored-by: Andronik <write@reusable.software>
* rough draft of potential parent search
* get things compiling
* fmt
* add new function to all RelayChainInterface implementations
* fix compilation
* set slot and timestamp based on relay parent, prepare for find-parent
* skeleton of new aura logic
* fmt
* introduce a collator module in the Aura crate
* extract different implementations into own modules
* make interface more convenient
* docs and todos for lookahead
* refactor basic collator to use new collator utility
* some more refactoring
* finish most of the control flow for new aura
* introduce backend as parameter
* fix compilation
* fix a couple more TODOs
* add an `announce_block` function to collator service
* announce with barrier
* rename block announcement validator to be more specific
* fmt
* clean up unused import errors
* update references to BlockAnnounceValidator
* rename unstable_reimpl
* add AuraUnincludedSegmentApi
* finish rename
* integrate AuraUnincludedSegmentApi
* add a new block announcement validator for backwards compatibility
* add some naive equivocation defenses
* rustfmt
* clean up remaining TODO [now]s
* fmt
* try to fix inprocess-interface
* actually fix compilation
* ignored -> rejected rephrase
* fix test compilation
* fmt
* clippy
* implement a proposer utility for consensus
* tidy up deps of new proposer crate
* implement a collator-service crate
* rewrite cumulus-collator to use new service struct
* implement a module for relay-chain-driven collators
* adapt start_collator to use the new relay_chain_driven module
* move collator-service to a public submodule
* create an interface trait for the proposer
* begin aura reimplementation
* address review comments
* update substrrate git ref
* update polkadot-primitives refs
* rough draft of aura collation using standalone fns
* add a ServiceInterface
* port aura reimpl to use new service trait
* add an import queue utility crate
* remove import queue crate in favor of module in common
* implement new verification queue for aura
* implement remaining behaviors
* split 'collate' into smaller functions that could be pub
* add telemetry
* fix doc job?
* Specify async-trait patch version
Co-authored-by: Bastian Köcher <git@kchr.de>
* remove 'fn@' in doc string.
Co-authored-by: Bastian Köcher <git@kchr.de>
* update variable names to be more readable
* refactor proposer errors to anyhow/thiserror
* remove manual span instrumentation
Co-authored-by: Bastian Köcher <git@kchr.de>
* make slot_claim private
* fix unused import
* fmt
* fmt
* make clippy happy
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
* BlockId removal: refactor of runtime API
It changes the first argument of all generated runtime API calls from: `BlockId<Block>` to: `Block::Hash`
* fmt
* cargo update -p polkadot-primitives
* Revert "cargo update -p polkadot-primitives"
This reverts commit 58506ecd31e1e5e42b78c2db3982acbc4d8edc70.
* update lockfile for {"substrate", "polkadot"}
---------
Co-authored-by: parity-processbot <>
* Use primitives reexported from `polkadot_primitives` crate root
* restart CI
* Fixes after merge
* update lockfile for {"polkadot", "substrate"}
Co-authored-by: parity-processbot <>
* Remove the `RelaychainClient` trait
It was just some historical trait that isn't really required anymore. Besides that this pr re-exports
types that are being used by the relay chain interface to make its usage easier.
* Fix warning
* BlockId removal: refactor: BlockBackend::block|block_status
It changes the arguments of:
- `BlockBackend::block`
- `BlockBackend::block_status`
method from: `BlockId<Block>` to: `Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* update lockfile for {"polkadot", "substrate"}
Co-authored-by: parity-processbot <>
Prevents the StateDbError::TooManySiblingBlocks error from being triggered by eagerly removing
stale blocks from the backend on block import and before the error condition is met.
Introduces a just in time block recovery mechanism for blocks that were wrongly removed
via an explicit pov-recovery method
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* BlockId removal: &Hash to Hash
It changes &Block::Hash argument to Block::Hash.
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* update lockfile for {"polkadot", "substrate"}
Co-authored-by: parity-processbot <>
* BlockId removal: refactor: Finalizer
It changes the arguments of methods of `Finalizer` trait from: block:
`BlockId<Block>` to: hash: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* update lockfile for {"polkadot", "substrate"}
* update Cargo.lock
* Cargo.lock update again
Co-authored-by: parity-processbot <>
* Initial network interface preparations
* Implement get_storage_by_key
* Implement `validators` and `session_index_for_child`
* Implement persisted_validation_data and candidate_pending_availability
* Fix method name for persisted_validation_data and add encoded params
* Implement `retrieve_dmq_contents` and `retrieve_all_inbound_hrmp_channel_contents`
* Implement `prove_read`
* Introduce separate RPC client, expose JsonRpSee errors
* Simplify closure in call_remote_runtime_function
* Implement import stream, upgrade JsonRpSee
* Implement finality stream
* Remove unused method from interface
* Implement `is_major_syncing`
* Implement `wait_on_block`
* Fix tests
* Unify error handling `ApiError`
* Replace WaitError with RelayChainError
* Wrap BlockChainError in RelayChainError
* Unify error handling in relay chain intefaces
* Fix return type of proof method
* Improve error handling of new methods
* Improve error handling and move logging outside of interface
* Clean up
* Remove unwanted changes, clean up
* Remove unused import
* Add format for StatemachineError and remove nused From trait
* Use 'thiserror' crate to simplify error handling
* Expose error for overseer, further simplify error handling
* Reintroduce network interface
* Implement cli option
* Adjust call_state method to use hashes
* Disable PoV recovery when RPC is used
* Add integration test for network full node
* Use Hash instead of BlockId to ensure compatibility with RPC interface
* Fix cargo check warnings
* Implement retries
* Remove `expect` statements from code
* Update jsonrpsee to 0.8.0 and make collator keys optional
* Make cli arguments conflicting
* Remove unused `block_status` method
* Add clippy fixes
* Cargo fmt
* Validate relay chain rpc url
* Clean up dependencies and add one more integration test
* Clean up
* Clean up dependencies of relay-chain-network
* Use hash instead of blockid for rpc methods
* Fix tests
* Update client/cli/src/lib.rs
Co-authored-by: Koute <koute@users.noreply.github.com>
* Improve error message of cli validation
* Add rpc client constructor
* Do not use debug formatting for errors
* Improve logging for remote runtime methods
* Only retry on transport problems
* Use PHash by value, rename test
* Improve tracing, return error on relay-chain-interface build
* Fix naming, use generics instead of deserializing manually
* Rename RelayChainLocal and RelayChainNetwork
* lock
* Format
* Use impl trait for encodable runtime payload
* Only instantiate full node in tests when we need it
* Upgrade scale-codec to 3.0.0
* Improve expect log
Co-authored-by: Koute <koute@users.noreply.github.com>
It doesn't make sense to log them as a warning, especially when it could
happen that a parachain id could be reused over the lifetime of a relay
chain.
* Start with a failing integration test & some refactorings
* More work
* Make it "work"
* Add `NullConsensus` for the test
* More refactorings
* Move stuff over to its own crate
* Refactorings
* Integrate it into `service` and make the test working
* Docs and some exit condition
* Use the real import queue
* Fix tests
* Update client/pov-recovery/src/active_candidate_recovery.rs
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* Fetch slot duration from the relay chain
* Docs
* Fixes
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* Add a command to purge the relay chain only
* WIP
* Update rococo-parachains/src/cli.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Move cli stuff to its own crate
* Copyright dates
* Test not working for some reason...
* WIP
* Revert "WIP"
This reverts commit f97cd63742c7df822e4a6e52a29db5e0f56b7bfa.
* Fix test to use provided relay chain
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Add hint about which database could not be purged
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Move consensus to consensus-common crate
* Move the parachain consensus out of the collator
* Add first relay chain consensus stuff
* Remove some warnings
* Fix more stuff
* Fix collator test
* Change `ParachainConsensus` to take a mutable self
* Make everything compile
* Feedback
Sebastian Kunert