The `BlockBuilderProvider` was a trait that was defined in
`sc-block-builder`. The trait was implemented for `Client`. This
basically meant that you needed to import `sc-block-builder` any way to
have access to the block builder. So, this trait was not providing any
real value. This pull request is removing the said trait. Instead of the
trait it introduces a builder for creating a `BlockBuilder`. The builder
currently has the quite fabulous name `BlockBuilderBuilder` (I'm open to
any better name 😅). The rest of the pull request is about
replacing the old trait with the new builder.
# Downstream code changes
If you used `new_block` or `new_block_at` before you now need to switch
it over to the new `BlockBuilderBuilder` pattern:
```rust
// `new` requires a type that implements `CallApiAt`.
let mut block_builder = BlockBuilderBuilder::new(client)
// Then you need to specify the hash of the parent block the block will be build on top of
.on_parent_block(at)
// The block builder also needs the block number of the parent block.
// Here it is fetched from the given `client` using the `HeaderBackend`
// However, there also exists `with_parent_block_number` for directly passing the number
.fetch_parent_block_number(client)
.unwrap()
// Enable proof recording if required. This call is optional.
.enable_proof_recording()
// Pass the digests. This call is optional.
.with_inherent_digests(digests)
.build()
.expect("Creates new block builder");
```
---------
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: command-bot <>
Removal of verbatim duplication of BABE's `Epoch` struct in the client.
I think is better to have one single definition and wrap the primitive
`Epoch` in a newtype (required because we need to implement the `Epoch`
trait).
* Start
* More work!
* Moar
* More changes
* More fixes
* More worrk
* More fixes
* More fixes to make it compile
* Adds `NoOffchainStorage`
* Pass the extensions
* Small basti making small progress
* Fix merge errors and remove `ExecutionContext`
* Move registration of `ReadRuntimeVersionExt` to `ExecutionExtension`
Instead of registering `ReadRuntimeVersionExt` in `sp-state-machine` it is moved to
`ExecutionExtension` which provides the default extensions.
* Fix compilation
* Register the global extensions inside runtime api instance
* Fixes
* Fix `generate_initial_session_keys` by passing the keystore extension
* Fix the grandpa tests
* Fix more tests
* Fix more tests
* Don't set any heap pages if there isn't an override
* Fix small fallout
* FMT
* Fix tests
* More tests
* Offchain worker custom extensions
* More fixes
* Make offchain tx pool creation reusable
Introduces an `OffchainTransactionPoolFactory` for creating offchain transactions pools that can be
registered in the runtime externalities context. This factory will be required for a later pr to
make the creation of offchain transaction pools easier.
* Fixes
* Fixes
* Set offchain transaction pool in BABE before using it in the runtime
* Add the `offchain_tx_pool` to Grandpa as well
* Fix the nodes
* Print some error when using the old warnings
* Fix merge issues
* Fix compilation
* Rename `babe_link`
* Rename to `offchain_tx_pool_factory`
* Cleanup
* FMT
* Fix benchmark name
* Fix `try-runtime`
* Remove `--execution` CLI args
* Make clippy happy
* Forward bls functions
* Fix docs
* Update UI tests
* Update client/api/src/execution_extensions.rs
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Koute <koute@users.noreply.github.com>
* Update client/cli/src/params/import_params.rs
Co-authored-by: Koute <koute@users.noreply.github.com>
* Update client/api/src/execution_extensions.rs
Co-authored-by: Koute <koute@users.noreply.github.com>
* Pass the offchain storage to the MMR RPC
* Update client/api/src/execution_extensions.rs
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Review comments
* Fixes
---------
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* substrate-test-runtime migrated to pure-frame based
* test block builder: helpers added
* simple renaming
* basic_authorship test adjusted
* block_building storage_proof test adjusted
* babe: tests: should_panic expected added
* babe: tests adjusted
ConsensusLog::NextEpochData is now added by pallet_babe as
pallet_babe::SameAuthoritiesForever trigger is used in runtime config.
* beefy: tests adjusted
test-substrate-runtime is now using frame::executive to finalize the
block. during finalization the digests stored during block execution are
checked against header digests:
https://github.com/paritytech/substrate/blob/91bb2d29ca905599098a5b35eaf24867c4fbd60a/frame/executive/src/lib.rs#L585-L591
It makes impossible to directly manipulate header's digets, w/o
depositing logs into system pallet storage `Digest<T: Config>`.
Instead of this dedicated extrinsic allowing to store logs items
(MmrRoot / AuthoritiesChange) is used.
* grandpa: tests adjusted
test-substrate-runtime is now using frame::executive to finalize the
block. during finalization the digest logs stored during block execution are
checked against header digest logs:
https://github.com/paritytech/substrate/blob/91bb2d29ca905599098a5b35eaf24867c4fbd60a/frame/executive/src/lib.rs#L585-L591
It makes impossible to directly manipulate header's digets, w/o
depositing logs into system pallet storage `Digest<T: Config>`.
Instead of this dedicated extrinsic allowing to store logs items
(ScheduledChange / ForcedChange and DigestItem::Other) is used.
* network:bitswap: test adjusted
The size of unchecked extrinsic was increased. The pattern used in test will
be placed at the end of scale-encoded buffer.
* runtime apis versions adjusted
* storage keys used in runtime adjusted
* wasm vs native tests removed
* rpc tests: adjusted
Transfer transaction processing was slightly improved, test was
adjusted.
* tests: sizes adjusted
Runtime extrinsic size was increased. Size of data read during block
execution was also increased due to usage of new pallets in runtime.
Sizes were adjusted in tests.
* cargo.lock update
cargo update -p substrate-test-runtime -p substrate-test-runtime-client
* warnings fixed
* builders cleanup: includes / std
* extrinsic validation cleanup
* txpool: benches performance fixed
* fmt
* spelling
* Apply suggestions from code review
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Apply code review suggestions
* Apply code review suggestions
* get rid of 1063 const
* renaming: UncheckedExtrinsic -> Extrinsic
* test-utils-runtime: further step to pure-frame
* basic-authorship: tests OK
* CheckSubstrateCall added + tests fixes
* test::Transfer call removed
* priority / propagate / no sudo+root-testing
* fixing warnings + format
* cleanup: build2/nonce + format
* final tests fixes
all tests are passing
* logs/comments removal
* should_not_accept_old_signatures test removed
* make txpool benches work again
* Cargo.lock reset
* format
* sudo hack removed
* txpool benches fix+cleanup
* .gitignore reverted
* rebase fixing + unsigned cleanup
* Cargo.toml/Cargo.lock cleanup
* force-debug feature removed
* mmr tests fixed
* make cargo-clippy happy
* network sync test uses unsigned extrinsic
* cleanup
* ".git/.scripts/commands/fmt/fmt.sh"
* push_storage_change signed call remove
* GenesisConfig cleanup
* fix
* fix
* GenesisConfig simplified
* storage_keys_works: reworked
* storage_keys_works: expected keys in vec
* storage keys list moved to substrate-test-runtime
* substrate-test: some sanity tests + GenesisConfigBuilder rework
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Apply suggestions from code review
* Review suggestions
* fix
* fix
* beefy: generate_blocks_and_sync block_num sync with actaul value
* Apply suggestions from code review
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Update test-utils/runtime/src/genesismap.rs
Co-authored-by: Davide Galassi <davxy@datawok.net>
* cargo update -p sc-rpc -p sc-transaction-pool
* Review suggestions
* fix
* doc added
* slot_duration adjusted for Babe::slot_duration
* small doc fixes
* array_bytes::hex used instead of hex
* tiny -> medium name fix
* Apply suggestions from code review
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* TransferData::try_from_unchecked_extrinsic -> try_from
* Update Cargo.lock
---------
Co-authored-by: parity-processbot <>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Allow extra signing data
* Fix tests after renaming
* Rename VrfSecret/VrfVerifier to VrfSecret/VrfPublic
* Further encrapsulation of 'transcript' type to the sr25519 implementation
* Keystore sr25519 pre-output
* Leave additional custom input field hidden in the associated VrfInput type
* Fix test
* More ergonomic output_bytes
* Trigger pipeline
* Define a separated type for vrf signature data
* Fix docs
* Fix doc
* Remove annotation
* Directly use dleq_proove and dleq_verify in sr25519
* Trigger CI
* Remove cruft before merge
* First iteration to encapsulate schnorrkel and merlin usage
* Remove schnorkel direct dependency from BABE pallet
* Remove schnorrkel direct dependency from BABE client
* Trivial renaming for VrfTranscript data and value
* Better errors
* Expose a function to get a schnorrkel friendly transcript
* Keep the vrf signature stuff together (preventing some clones around)
* Fix tests
* Remove vrf agnostic transcript and define it as an associated type for VrfSigner and VrfVerifier
* Fix babe pallet mock
* Inner types are required to be public for polkadot
* Update client/consensus/babe/src/verification.rs
Co-authored-by: Koute <koute@users.noreply.github.com>
* Nit
* Remove Deref implementations
* make_bytes as a method
* Trigger CI
---------
Co-authored-by: Koute <koute@users.noreply.github.com>
* Move import queue out of `sc-network`
Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.
This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.
* Move stuff to SyncingEngine
* Move `ChainSync` instanation to `SyncingEngine`
Some of the tests have to be rewritten
* Move peer hashmap to `SyncingEngine`
* Let `SyncingEngine` to implement `ChainSyncInterface`
* Introduce `SyncStatusProvider`
* Move `sync_peer_(connected|disconnected)` to `SyncingEngine`
* Implement `SyncEventStream`
Remove `SyncConnected`/`SyncDisconnected` events from
`NetworkEvenStream` and provide those events through
`ChainSyncInterface` instead.
Modify BEEFY/GRANDPA/transactions protocol and `NetworkGossip` to take
`SyncEventStream` object which they listen to for incoming sync peer
events.
* Introduce `ChainSyncInterface`
This interface provides a set of miscellaneous functions that other
subsystems can use to query, for example, the syncing status.
* Move event stream polling to `SyncingEngine`
Subscribe to `NetworkStreamEvent` and poll the incoming notifications
and substream events from `SyncingEngine`.
The code needs refactoring.
* Make `SyncingEngine` into an asynchronous runner
This commits removes the last hard dependency of syncing from
`sc-network` meaning the protocol now lives completely outside of
`sc-network`, ignoring the hardcoded peerset entry which will be
addressed in the future.
Code needs a lot of refactoring.
* Fix warnings
* Code refactoring
* Use `SyncingService` for BEEFY
* Use `SyncingService` for GRANDPA
* Remove call delegation from `NetworkService`
* Remove `ChainSyncService`
* Remove `ChainSync` service tests
They were written for the sole purpose of verifying that `NetworWorker`
continues to function while the calls are being dispatched to
`ChainSync`.
* Refactor code
* Refactor code
* Update client/finality-grandpa/src/communication/tests.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Fix warnings
* Apply review comments
* Fix docs
* Fix test
* cargo-fmt
* Update client/network/sync/src/engine.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Update client/network/sync/src/engine.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Add missing docs
* Refactor code
---------
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
* `BlockId` removal: `BlockBuilderProvider::new_block_at`
It changes the arguments of `BlockBuilderProvider::new_block_at` from:
`BlockId<Block>` to: `Block::Hash`
* fmt
* fix
* more fixes
* BlockId removal: refactor of runtime API
It changes the arguments of:
- `ApiExt` methods: `has_api`, `has_api_with`, `api_version`
- `CallApiAt` method: `runtime_version_at`
from: `BlockId<Block>` to: `Block::Hash`
It also changes the first argument of all generated runtime API calls from: `BlockId<Block>` to: `Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* BlockId removal: refactor of runtime API - tests
- tests adjusted to new runtime API,
- some tests migrated from block number to block hash
* benchmarking-cli: BlockId(0) migrated to info().genesis_hash
`runtime_api.call()` now requires the block hash instead of BlockId::Number.
To access the genesis hash widely used in benchmarking engine the Client
was constrained to satisfy `sp_blockchain::HeaderBackend<Block>` trait
which provides `info().genesis_hash`.
* trivial: api.call(BlockId) -> api.call(Hash)
- Migrated all `runtime_api.calls` to use Hash
- Noteworthy (?):
-- `validate_transaction_blocking` in transaction pool,
* CallApiAtParams::at changed to Block::Hash
* missed doc updated
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* ".git/.scripts/commands/fmt/fmt.sh"
* BlockId removal: Benchmark::consumed_weight
Little refactor around `Benchmark::consumed_weight`: `BlockId` removed.
* at_hash renamed
* wrong merge fixed
* beefy worker: merged with master
* beefy: tests: missing block problem fixed
* Apply review suggestion
* fix
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>
* Detect and correct epoch-index for skipped epochs
* Code refactory
* Epoch index should be also be fixed for secondary claims with VRF
* Fix typo
* Make clippy happy
* Fix typo
* Trigger pipeline
The `authoring_blocks` test of BABE was calculating the slot based on the timestamp it sometimes
failed in CI. The problem is that we combine all the notifications and authoring futures in one big
future. This one big future may first polls one authoring future to build a block. Then it polls all
notification futures again to import the block. Then some other authoring future is polled and
builds on the imported block using the same slot and making the import fail. The solution is that we
just artificially increase the slot to make the test work.
* babe: allow skipping epochs in pallet
* babe: detect and skip epochs on client
* babe: cleaner epoch util functions
* babe: add test for runtime handling of skipped epochs
* babe: simpler implementation of client handling of skipped epochs
* babe: test client-side handling of skipped epochs
* babe: add comments on client-side skipped epochs
* babe: remove emptyline
* babe: make it resilient to forks
* babe: typo
* babe: overflow-safe math
* babe: add test for skipping epochs across different forks
* Fix tests
* FMT
Co-authored-by: Bastian Köcher <info@kchr.de>
* Replace deprecated libp2p feature specs with correct ones
* Bump tokio to 1.21.2
* Replace async-std libp2p primitives with tokio ones
* minor: rustfmt
* Fix TestNet to run initialization in the tokio context
* Convert telemetry test from async-std to tokio
* Convert notifications tests from async-std to tokio
* Convert chain sync tests from async-std to tokio
* Ditch async-std completely
* Make executor mandatory
* Bump tokio to 1.22.0
* minor: rustfmt
* Explicitly use tokio runtime in tests
* Move more tests to explicit tokio runtime
* Explicitly set multithreaded runtime in tokio test
* minor: rustfmt
* minor: fix comment
* Replace async-std with tokio in MMR tests
* Remove all not required nodes from the epoch-changes tree
Some outdated nodes were left there because of the predicate
* Test to excercise the fix
* Add a fork on genesis to the test
* Fix typo in comments
* BlockId removal: refactor: Finalizer
It changes the arguments of methods of `Finalizer` trait from:
block: `BlockId<Block>` to: hash: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* minor corrections
* failing test corrected
* minor rework
* Remove timestamp from SlotInfo
* Expose as millis instead of secs
* Nits
* Fix test after field removal
* Yet another test fix
* On the fly timestamp computation
* Removed slot timestamp from logs
* Removed reference to timestamp from slots subsystem
* Slot based algorithm tests do not require timstamp inherent anymore
* Remove junk files
* Further tests cleanup
* Trigger pipeline
* Apply code suggestions
* Trigger pipeline
Co-authored-by: André Silva <andrerfosilva@gmail.com>
* Remove CanAuthorWith trait
CanAuthotWith trait removed. Also all dependencies, parameters, type
paramers were removed. This is related to removal of native runtime.
* Remove commented code
* Fix code formatting
* trigger CI job
* trigger CI job
* trigger CI job
* trigger CI job
* trigger CI job
* trigger CI job
* trigger CI job
* Fetch babe config data from runtime state
* Some renaming
* More renaming
* Final nits
* Fix tests and benches
* Rename to in BabeConfiguration
* Remove duplicate babe parameter description
Already specified over the 'PRIMARY_PROBABILITY' constant value
* trigger pipeline
* trigger pipeline
* Remove direct dependency of `sc-network` on `sc-network-light`
* Move `WarpSyncProvider` trait and surrounding data structures into `sc-network-common`
* Move `WarpSyncProvider` trait and surrounding data structures into `sc-network-common`
* Create `sync` module in `sc-network-common`, create `ChainSync` trait there (not used yet), move a bunch of associated data structures from `sc-network-sync`
* Switch from concrete implementation to `ChainSync` trait from `sc-network-common`
* Introduce `OpaqueStateRequest`/`OpaqueStateResponse` to remove generics from `StateSync` trait
* Introduce `OpaqueBlockRequest`/`OpaqueBlockResponse`, make `scheme` module of `sc-network-sync` private
* Surface `sc-network-sync` into `sc-service` and make `sc-network` not depend on it anymore
* Remove now unnecessary dependency from `sc-network`
* Replace crate links with just text since dependencies are gone now
* Remove `warp_sync` re-export from `sc-network-common`
* Update copyright in network-related files
* Address review comments about documentation
* Apply review suggestion
* Rename `extra_requests` module to `metrics`
Co-authored-by: Bastian Köcher <info@kchr.de>
* Fix Babe revert when a leaf is the last finalized block
Without this fix the last finalized block weight data is wrongly removed
on revert scenario where the last finalized block is a leaf.
* Remove redundant check
* Added test to exercise the fix
* Rename test
* Give variables better names
With the latest optimizations of the `FinalityNotification` generation, the aux data pruning started
to print a warning. The problem here was that we printed a warning and stopped the adding of blocks
to prune when we hit the `heigh_limit`. This is now wrong, as we could for example have two 512 long
forks and then we start finalizing one of them. The second fork head would be part of the stale
heads at some point (in the current implementation when we finalize second fork head number + 1),
but then we would actually need to go back into the past than `heigh_limit` (which was actually
last_finalized - 1). We now go back until we reach the canonical chain.
Also fixed some wrong comment that was added by be about the content of the `finalized` blocks in
the `FinalityNotification`.
* Finality notification: Optimize calculation of stale heads
While looking into some problem on Versi where a collator seemed to be stuck. I found out that it
was not stuck but there was a huge gap between last finalized and best block. This lead to a lot
leaves and it was basically trapped inside some loop of reading block headers from the db to find
the stale heads. While looking into this I found out that `leaves` already supports the feature to
give us the stale heads relative easily. However, the semantics change a little bit. Instead of
returning all stale heads of blocks that are not reachable anymore after finalizing a block, we
currently only return heads with a number lower than the finalized block. This should be no problem,
because these other leaves that are stale will be returned later when a block gets finalized which
number is bigger than the block number of these leaves.
While doing that, I also changed `tree_route` of the `FinalityNotification` to include the
`old_finalized`. Based on the comment I assumed that this was already part of it. However, if
wanted, I can revert this change.
* FMT
* Update client/service/src/client/client.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Do not include the last finalized block
* Rename function
* FMT
* Fix tests
* Update figure
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* First rough draft for BABE revert
* Proper babe revert test
* Cleanup
* Test trivial cleanup
* Fix to make clippy happy
* Check polkadot companion
* Check cumulus companion
* Remove babe's blocks weight on revert
* Handle "empty" blockchain edge case
* Run companions
* Simplify the filter predicate
* Saturating sub is not required
* Run pipeline
* Run pipeline again...
* Clean obsolete BABE weight data
* Take out test assertion from check closure
* Optimize metadata access using `HeaderMetadata` trait
* Apply suggestions from code review
* Introduce finalize and import pre-commit synchronous actions
* Do not hold locks between internal methods calls
* Remove unused generic bound
* Apply suggestions from code review
* Register BABE's pre-commit actions on `block_import` instead of `start_babe`
* PreCommit actions should be `Fn` instead of `FnMut`
* More robust safenet in case of malformed finality notifications
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* SlotDuration: Always fetch the slot duration from the runtime
The slot duration should always be fetched from the runtime instead of being cached in the db. The
slot duration is only fetched on startup of the node, so the performance isn't that important. This
is especially helpful for the case when the slot duration of a chain should be changed through a
runtime upgrade (there be dragons, so take care).
* Fix docs
* Remove logging
* Fix warning
* Split native executor stuff from wasm executor stuff
* Remove `native_runtime_version` in places
* Fix warning
* Fix test warning
* Remove redundant NativeRuntimeInfo trait
* Add a warning for use_native
* Run cargo fmt
* Revert "Add a warning for use_native"
This reverts commit 9494f765a06037e991dd60524f2ed1b14649bfd6.
* Make choosing an executor (native/wasm) an explicit part of service construction
* Add Cargo.lock
* Rename Executor to ExecutorDispatch
* Update bin/node/executor/src/lib.rs
Co-authored-by: Squirrel <gilescope@gmail.com>
* Fix tests
* Fix minor node-executor error
* Fix node cli command thing
Co-authored-by: Squirrel <gilescope@gmail.com>
* Run cargo fmt on the whole code base
* Second run
* Add CI check
* Fix compilation
* More unnecessary braces
* Handle weights
* Use --all
* Use correct attributes...
* Fix UI tests
* AHHHHHHHHH
* 🤦
* Docs
* Fix compilation
* 🤷
* Please stop
* 🤦 x 2
* More
* make rustfmt.toml consistent with polkadot
Co-authored-by: André Silva <andrerfosilva@gmail.com>
Alexander Samusev