This commit introduces a new concept called `NotificationService` which
allows Polkadot protocols to communicate with the underlying
notification protocol implementation directly, without routing events
through `NetworkWorker`. This implies that each protocol has its own
service which it uses to communicate with remote peers and that each
`NotificationService` is unique with respect to the underlying
notification protocol, meaning `NotificationService` for the transaction
protocol can only be used to send and receive transaction-related
notifications.
The `NotificationService` concept introduces two additional benefits:
* allow protocols to start using custom handshakes
* allow protocols to accept/reject inbound peers
Previously the validation of inbound connections was solely the
responsibility of `ProtocolController`. This caused issues with light
peers and `SyncingEngine` as `ProtocolController` would accept more
peers than `SyncingEngine` could accept which caused peers to have
differing views of their own states. `SyncingEngine` would reject excess
peers but these rejections were not properly communicated to those peers
causing them to assume that they were accepted.
With `NotificationService`, the local handshake is not sent to remote
peer if peer is rejected which allows it to detect that it was rejected.
This commit also deprecates the use of `NetworkEventStream` for all
notification-related events and going forward only DHT events are
provided through `NetworkEventStream`. If protocols wish to follow each
other's events, they must introduce additional abtractions, as is done
for GRANDPA and transactions protocols by following the syncing protocol
through `SyncEventStream`.
Fixes https://github.com/paritytech/polkadot-sdk/issues/512
Fixes https://github.com/paritytech/polkadot-sdk/issues/514
Fixes https://github.com/paritytech/polkadot-sdk/issues/515
Fixes https://github.com/paritytech/polkadot-sdk/issues/554
Fixes https://github.com/paritytech/polkadot-sdk/issues/556
---
These changes are transferred from
https://github.com/paritytech/substrate/pull/14197 but there are no
functional changes compared to that PR
---------
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
This PR contains some fixes and cleanups for parachain nodes:
1. When using async backing, node no longer complains about being unable
to reach the prospective-parachain subsystem.
2. Parachain warp sync now informs users that the finalized para block
has been retrieved.
```
2023-11-08 13:24:42 [Parachain] 🎉 Received finalized parachain header #5747719 (0xa0aa…674b) from the relay chain.
```
3. When a user supplied an invalid `--relay-chain-rpc-url`, we were
crashing with a very verbose message. Removed the `expect` and improved
the error message.
```
2023-11-08 13:57:56 [Parachain] No valid RPC url found. Stopping RPC worker.
2023-11-08 13:57:56 [Parachain] Essential task `relay-chain-rpc-worker` failed. Shutting down service.
Error: Service(Application(WorkerCommunicationError("RPC worker channel closed. This can hint and connectivity issues with the supplied RPC endpoints. Message: oneshot canceled")))
```
- Async-backing related primitives are stable `primitives::v6`
- Async-backing API is now part of `api_version(7)`
- It's enabled on Rococo and Westend runtimes
---------
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* Update substrate & polkadot
* min changes to make async backing compile
* (async backing) parachain-system: track limitations for unincluded blocks (#2438)
* unincluded segment draft
* read para head from storage proof
* read_para_head -> read_included_para_head
* Provide pub interface
* add errors
* fix unincluded segment update
* BlockTracker -> Ancestor
* add a dmp limit
* Read para head depending on the storage switch
* doc comments
* storage items docs
* add a sanity check on block initialize
* Check watermark
* append to the segment on block finalize
* Move segment update into set_validation_data
* Resolve para head todo
* option watermark
* fix comment
* Drop dmq check
* fix weight
* doc-comments on inherent invariant
* Remove TODO
* add todo
* primitives tests
* pallet tests
* doc comments
* refactor unincluded segment length into a ConsensusHook (#2501)
* refactor unincluded segment length into a ConsensusHook
* add docs
* refactor bandwidth_out calculation
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
* test for limits from impl
* fmt
* make tests compile
* update comment
* uncomment test
* fix collator test by adding parent to state proof
* patch HRMP watermark rules for unincluded segment
* get consensus-common tests to pass, using unincluded segment
* fix unincluded segment tests
* get all tests passing
* fmt
* rustdoc CI
* aura-ext: limit the number of authored blocks per slot (#2551)
* aura_ext consensus hook
* reverse dependency
* include weight into hook
* fix tests
* remove stray println
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
* fix test warning
* fix doc link
---------
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
Co-authored-by: Chris Sosnin <chris125_@live.com>
* parachain-system: ignore go ahead signal once upgrade is processed (#2594)
* handle goahead signal for unincluded segment
* doc comment
* add test
* parachain-system: drop processed messages from inherent data (#2590)
* implement `drop_processed_messages`
* drop messages based on relay parent number
* adjust tests
* drop changes to mqc
* fix comment
* drop test
* drop more dead code
* clippy
* aura-ext: check slot in consensus hook and remove all `CheckInherents` logic (#2658)
* aura-ext: check slot in consensus hook
* convert relay chain slot
* Make relay chain slot duration generic
* use fixed velocity hook for pallets with aura
* purge timestamp inherent
* fix warning
* adjust runtime tests
* fix slots in tests
* Make `xcm-emulator` test pass for new consensus hook (#2722)
* add pallets on_initialize
* tests pass
* add AuraExt on_init
* ".git/.scripts/commands/fmt/fmt.sh"
---------
Co-authored-by: command-bot <>
---------
Co-authored-by: Ignacio Palacios <ignacio.palacios.santos@gmail.com>
* update polkadot git refs
* CollationGenerationConfig closure is now optional (#2772)
* CollationGenerationConfig closure is now optional
* fix test
* propagate network-protocol-staging feature (#2899)
* Feature Flagging Consensus Hook Type Parameter (#2911)
* First pass
* fmt
* Added as default feature in tomls
* Changed to direct dependency feature
* Dealing with clippy error
* Update pallets/parachain-system/src/lib.rs
Co-authored-by: asynchronous rob <rphmeier@gmail.com>
---------
Co-authored-by: asynchronous rob <rphmeier@gmail.com>
* fmt
* bump deps and remove warning
* parachain-system: update RelevantMessagingState according to the unincluded segment (#2948)
* mostly address 2471 with a bug introduced
* adjust relevant messaging state after computing total
* fmt
* max -> min
* fix test implementation of xcmp source
* add test
* fix test message sending logic
* fix + test
* add more to unincluded segment test
* fmt
---------
Co-authored-by: Chris Sosnin <chris125_@live.com>
* Integrate new Aura / Parachain Consensus Logic in Parachain-Template / Polkadot-Parachain (#2864)
* add a comment
* refactor client/service utilities
* deprecate start_collator
* update parachain-template
* update test-service in the same way
* update polkadot-parachain crate
* fmt
* wire up new SubmitCollation message
* some runtime utilities for implementing unincluded segment runtime APIs
* allow parachains to configure their level of sybil-resistance when starting the network
* make aura-ext compile
* update to specify sybil resistance levels
* fmt
* specify relay chain slot duration in milliseconds
* update Aura to explicitly produce Send futures
also, make relay_chain_slot_duration a Duration
* add authoring duration to basic collator and document params
* integrate new basic collator into parachain-template
* remove assert_send used for testing
* basic-aura: only author when parent included
* update polkadot-parachain-bin
* fmt
* some fixes
* fixes
* add a RelayNumberMonotonicallyIncreases
* add a utility function for initializing subsystems
* some logging for timestamp adjustment
* fmt
* some fixes for lookahead collator
* add a log
* update `find_potential_parents` to account for sessions
* bound the loop
* restore & deprecate old start_collator and start_full_node functions.
* remove unnecessary await calls
* fix warning
* clippy
* more clippy
* remove unneeded logic
* ci
* update comment
Co-authored-by: Marcin S. <marcin@bytedude.com>
* (async backing) restore `CheckInherents` for backwards-compatibility (#2977)
* bring back timestamp
* Restore CheckInherents
* revert to empty CheckInherents
* make CheckInherents optional
* attempt
* properly end system blocks
* add some more comments
* ignore failing system parachain tests
* update refs after main feature branch merge
* comment out the offending tests because CI runs ignored tests
* fix warnings
* fmt
* revert to polkadot master
* cargo update -p polkadot-primitives -p sp-io
---------
Co-authored-by: asynchronous rob <rphmeier@gmail.com>
Co-authored-by: Ignacio Palacios <ignacio.palacios.santos@gmail.com>
Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com>
Co-authored-by: Marcin S. <marcin@bytedude.com>
Co-authored-by: eskimor <eskimor@users.noreply.github.com>
Co-authored-by: Andronik <write@reusable.software>
* Use primitives reexported from `polkadot_primitives` crate root
* restart CI
* Fixes after merge
* update lockfile for {"polkadot", "substrate"}
Co-authored-by: parity-processbot <>
* Allow specification of multiple urls for relay chain rpc nodes
* Add pooled RPC client basics
* Add list of clients to pooled client
* Improve
* Forward requests to dispatcher
* Switch clients on error
* Implement rotation logic
* Improve subscription handling
* Error handling cleanup
* Remove retry from rpc-client
* Improve naming
* Improve documentation
* Improve `ClientManager` abstraction
* Adjust zombienet test
* Add more comments
* fmt
* Apply reviewers comments
* Extract reconnection to extra method
* Add comment to reconnection method
* Clean up some dependencies
* Fix build
* fmt
* Provide alias for cli argument
* Apply review comments
* Rename P* to Relay*
* Improve zombienet test
* fmt
* Fix zombienet sleep
* Simplify zombienet test
* Reduce log clutter and fix starting position
* Do not distribute duplicated imported and finalized blocks
* fmt
* Apply code review suggestions
* Move building of relay chain interface to `cumulus-client-service`
* Refactoring to not push back into channel
* FMT
* Add minimal overseer gen with dummy subsystems
* Fix dependencies
* no-compile: only client transaction pool missing
* Remove unused imports
* Continue to hack towards PoC
* Continue
* Make mini node compile
* Compiling version with blockchainevents trait
* Continue
* Check in lockfile
* Block with tokio
* update patches
* Update polkadot patches
* Use polkadot-primitives v2
* Fix build problems
* First working version
* Adjust cargo.lock
* Add integration test
* Make integration test work
* Allow startinc collator without relay-chain args
* Make OverseerRuntimeClient async
* Create separate integration test
* Remove unused ChainSelection code
* Remove unused parameters on new-mini
* Connect collator node in test to relay chain nodes
* Make BlockChainRPCClient obsolete
* Clean up
* Clean up
* Reimplement blockchain-rpc-events
* Revert "Allow startinc collator without relay-chain args"
This reverts commit f22c70e16521f375fe125df5616d48ceea926b1a.
* Add `strict_record_validation` to AuthorityDiscovery
* Move network to cumulus
* Remove BlockchainRPCEvents
* Remove `BlockIdTo` and `BlockchainEvents`
* Make AuthorityDiscovery async
* Use hash in OverseerRuntime
* Adjust naming of runtime client trait
* Implement more rpc-client methods
* Improve error handling for `ApiError`
* Extract authority-discovery creationand cleanup
* RPC -> Rpc
* Extract bitswap
* Adjust to changes on master
* Implement `hash` method
* Introduce DummyChainSync, remove ProofProvider and BlockBackend
* Remove `HeaderMetadata` from blockchain-rpc-client
* Make ChainSync work
* Implement NetworkHeaderBackend
* Cleanup
* Adjustments after master merge
* Remove ImportQueue from network parameters
* Remove cargo patches
* Eliminate warnings
* Revert to HeaderBackend
* Add zombienet test
* Implement `status()` method
* Add more comments, improve readability
* Remove patches from Cargo.toml
* Remove integration test in favor of zombienet
* Remove unused dependencies, rename minimal node crate
* Adjust to latest master changes
* fmt
* Execute zombienet test on gitlab ci
* Reuse network metrics
* Chainsync metrics
* fmt
* Feed RPC node as boot node to the relay chain minimal node
* fmt
* Add bootnodes to zombienet collators
* Allow specification of relay chain args
* Apply review suggestions
* Remove unnecessary casts
* Enable PoV recovery for rpc full nodes
* Revert unwanted changes
* Make overseerHandle non-optional
* Add availability-store subsystem
* Add AuxStore and ChainApiSubsystem
* Add availability distribution subsystem
* Improve pov-recovery logging and add RPC nodes to tests
* fmt
* Make availability config const
* lock
* Enable debug logs for pov-recovery in zombienet
* Add log filters to test binary
* Allow wss
* Address review comments
* Apply reviewer comments
* Adjust to master changes
* Apply reviewer suggestions
* Bump polkadot
* Add builder method for minimal node
* Bump substrate and polkadot
* Clean up overseer building
* Add bootnode to two in pov_recovery test
* Fix missing quote in pov recovery zombienet test
* Improve zombienet pov test
* More debug logs for pov-recovery
* Remove reserved nodes like on original test
* Revert zombienet test to master
Aaro Altonen