* substrate-test-runtime migrated to pure-frame based
* test block builder: helpers added
* simple renaming
* basic_authorship test adjusted
* block_building storage_proof test adjusted
* babe: tests: should_panic expected added
* babe: tests adjusted
ConsensusLog::NextEpochData is now added by pallet_babe as
pallet_babe::SameAuthoritiesForever trigger is used in runtime config.
* beefy: tests adjusted
test-substrate-runtime is now using frame::executive to finalize the
block. during finalization the digests stored during block execution are
checked against header digests:
https://github.com/paritytech/substrate/blob/91bb2d29ca905599098a5b35eaf24867c4fbd60a/frame/executive/src/lib.rs#L585-L591
It makes impossible to directly manipulate header's digets, w/o
depositing logs into system pallet storage `Digest<T: Config>`.
Instead of this dedicated extrinsic allowing to store logs items
(MmrRoot / AuthoritiesChange) is used.
* grandpa: tests adjusted
test-substrate-runtime is now using frame::executive to finalize the
block. during finalization the digest logs stored during block execution are
checked against header digest logs:
https://github.com/paritytech/substrate/blob/91bb2d29ca905599098a5b35eaf24867c4fbd60a/frame/executive/src/lib.rs#L585-L591
It makes impossible to directly manipulate header's digets, w/o
depositing logs into system pallet storage `Digest<T: Config>`.
Instead of this dedicated extrinsic allowing to store logs items
(ScheduledChange / ForcedChange and DigestItem::Other) is used.
* network:bitswap: test adjusted
The size of unchecked extrinsic was increased. The pattern used in test will
be placed at the end of scale-encoded buffer.
* runtime apis versions adjusted
* storage keys used in runtime adjusted
* wasm vs native tests removed
* rpc tests: adjusted
Transfer transaction processing was slightly improved, test was
adjusted.
* tests: sizes adjusted
Runtime extrinsic size was increased. Size of data read during block
execution was also increased due to usage of new pallets in runtime.
Sizes were adjusted in tests.
* cargo.lock update
cargo update -p substrate-test-runtime -p substrate-test-runtime-client
* warnings fixed
* builders cleanup: includes / std
* extrinsic validation cleanup
* txpool: benches performance fixed
* fmt
* spelling
* Apply suggestions from code review
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Apply code review suggestions
* Apply code review suggestions
* get rid of 1063 const
* renaming: UncheckedExtrinsic -> Extrinsic
* test-utils-runtime: further step to pure-frame
* basic-authorship: tests OK
* CheckSubstrateCall added + tests fixes
* test::Transfer call removed
* priority / propagate / no sudo+root-testing
* fixing warnings + format
* cleanup: build2/nonce + format
* final tests fixes
all tests are passing
* logs/comments removal
* should_not_accept_old_signatures test removed
* make txpool benches work again
* Cargo.lock reset
* format
* sudo hack removed
* txpool benches fix+cleanup
* .gitignore reverted
* rebase fixing + unsigned cleanup
* Cargo.toml/Cargo.lock cleanup
* force-debug feature removed
* mmr tests fixed
* make cargo-clippy happy
* network sync test uses unsigned extrinsic
* cleanup
* ".git/.scripts/commands/fmt/fmt.sh"
* push_storage_change signed call remove
* GenesisConfig cleanup
* fix
* fix
* GenesisConfig simplified
* storage_keys_works: reworked
* storage_keys_works: expected keys in vec
* storage keys list moved to substrate-test-runtime
* substrate-test: some sanity tests + GenesisConfigBuilder rework
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Apply suggestions from code review
* Review suggestions
* fix
* fix
* beefy: generate_blocks_and_sync block_num sync with actaul value
* Apply suggestions from code review
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Update test-utils/runtime/src/genesismap.rs
Co-authored-by: Davide Galassi <davxy@datawok.net>
* cargo update -p sc-rpc -p sc-transaction-pool
* Review suggestions
* fix
* doc added
* slot_duration adjusted for Babe::slot_duration
* small doc fixes
* array_bytes::hex used instead of hex
* tiny -> medium name fix
* Apply suggestions from code review
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* TransferData::try_from_unchecked_extrinsic -> try_from
* Update Cargo.lock
---------
Co-authored-by: parity-processbot <>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* globally upgrade quote to latest 1.0.x (1.0.26)
* globally upgrade syn to final 1.0.x version (1.0.109)
* globally upgrade proc-macro2 to 1.0.56
* upgrade to syn v2.0.13 and fix everything except NestedMeta
* fix parse nested metadata code in decl_runtime_apis.rs
* Port more stuff to syn 2.0
* Make the rest compile
* Ignore error
* update to syn 2.0.14
---------
Co-authored-by: Bastian Köcher <info@kchr.de>
This removes the deprecated batch verification. This was actually never really activated.
Nevertheless, we need to keep the host functions around to support old runtimes which may import
these host functions. However, we do not give access to these functions anymore. This means that any new
runtime can not call them anymore. The host function implementations we keep will not do batch verification and will
instead fall back to the always existing option of directly verifying the passed signature.
`finish_batch_verification` will return the combined result of all the batch verify calls.
This removes the `TaskExecutorExt` which only existed to support the batch verification. So, any
code that used this extension can just remove the registration of them. It also removes
`SignatureBatching` that was used by `frame-executive` to control the batch verification.
However, there wasn't any `Verify` implementation that called the batch verification functions.
* Move service tests to `client/network/tests`
These tests depend on `sc-network` and `sc-network-sync` so they should
live outside the crate.
* Move some configs from `sc-network-common` to `sc-network`
* Move `NetworkService` traits to `sc-network`
* Move request-responses to `sc-network`
* Remove more stuff
* Remove rest of configs from `sc-network-common` to `sc-network`
* Remove more stuff
* Fix warnings
* Update client/network/src/request_responses.rs
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
* Fix cargo doc
---------
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
* Move import queue out of `sc-network`
Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.
This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.
* Move stuff to SyncingEngine
* Move `ChainSync` instanation to `SyncingEngine`
Some of the tests have to be rewritten
* Move peer hashmap to `SyncingEngine`
* Let `SyncingEngine` to implement `ChainSyncInterface`
* Introduce `SyncStatusProvider`
* Move `sync_peer_(connected|disconnected)` to `SyncingEngine`
* Implement `SyncEventStream`
Remove `SyncConnected`/`SyncDisconnected` events from
`NetworkEvenStream` and provide those events through
`ChainSyncInterface` instead.
Modify BEEFY/GRANDPA/transactions protocol and `NetworkGossip` to take
`SyncEventStream` object which they listen to for incoming sync peer
events.
* Introduce `ChainSyncInterface`
This interface provides a set of miscellaneous functions that other
subsystems can use to query, for example, the syncing status.
* Move event stream polling to `SyncingEngine`
Subscribe to `NetworkStreamEvent` and poll the incoming notifications
and substream events from `SyncingEngine`.
The code needs refactoring.
* Make `SyncingEngine` into an asynchronous runner
This commits removes the last hard dependency of syncing from
`sc-network` meaning the protocol now lives completely outside of
`sc-network`, ignoring the hardcoded peerset entry which will be
addressed in the future.
Code needs a lot of refactoring.
* Fix warnings
* Code refactoring
* Use `SyncingService` for BEEFY
* Use `SyncingService` for GRANDPA
* Remove call delegation from `NetworkService`
* Remove `ChainSyncService`
* Remove `ChainSync` service tests
They were written for the sole purpose of verifying that `NetworWorker`
continues to function while the calls are being dispatched to
`ChainSync`.
* Refactor code
* Refactor code
* Update client/finality-grandpa/src/communication/tests.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Fix warnings
* Apply review comments
* Fix docs
* Fix test
* cargo-fmt
* Update client/network/sync/src/engine.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Update client/network/sync/src/engine.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Add missing docs
* Refactor code
---------
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Rework storage iterators
* Make sure storage iteration is also accounted for when benchmarking
* Use `trie-db` from crates.io
* Appease clippy
* Bump `trie-bench` to 0.35.0
* Fix tests' compilation
* Update comment to clarify how `IterArgs::start_at` works
* Add extra tests
* Fix iterators on `Client` so that they behave as before
* Add extra `unwrap`s in tests
* More clippy fixes
* Come on clippy, give me a break already
* Rename `allow_missing` to `stop_on_incomplete_database`
* Add `#[inline]` to `with_recorder_and_cache`
* Use `with_recorder_and_cache` in `with_trie_db`; add doc comment
* Simplify code: use `with_trie_db` in `next_storage_key_from_root`
* Remove `expect`s in the benchmarking CLI
* Add extra doc comments
* Move `RawIter` before `TrieBackendEssence` (no code changes; just cut-paste)
* Remove a TODO in tests
* Update comment for `StorageIterator::was_complete`
* Update `trie-db` to 0.25.1
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
* `BlockId` removal: `BlockBuilderProvider::new_block_at`
It changes the arguments of `BlockBuilderProvider::new_block_at` from:
`BlockId<Block>` to: `Block::Hash`
* fmt
* fix
* more fixes
* BlockId removal: refactor of runtime API
It changes the arguments of:
- `ApiExt` methods: `has_api`, `has_api_with`, `api_version`
- `CallApiAt` method: `runtime_version_at`
from: `BlockId<Block>` to: `Block::Hash`
It also changes the first argument of all generated runtime API calls from: `BlockId<Block>` to: `Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* BlockId removal: refactor of runtime API - tests
- tests adjusted to new runtime API,
- some tests migrated from block number to block hash
* benchmarking-cli: BlockId(0) migrated to info().genesis_hash
`runtime_api.call()` now requires the block hash instead of BlockId::Number.
To access the genesis hash widely used in benchmarking engine the Client
was constrained to satisfy `sp_blockchain::HeaderBackend<Block>` trait
which provides `info().genesis_hash`.
* trivial: api.call(BlockId) -> api.call(Hash)
- Migrated all `runtime_api.calls` to use Hash
- Noteworthy (?):
-- `validate_transaction_blocking` in transaction pool,
* CallApiAtParams::at changed to Block::Hash
* missed doc updated
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* ".git/.scripts/commands/fmt/fmt.sh"
* BlockId removal: Benchmark::consumed_weight
Little refactor around `Benchmark::consumed_weight`: `BlockId` removed.
* at_hash renamed
* wrong merge fixed
* beefy worker: merged with master
* beefy: tests: missing block problem fixed
* Apply review suggestion
* fix
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>
* Finalization target should be chosed as some ancestor of SelectChain::best_chain
* More test assertions
* Improve docs
* Removed stale docs
* Rename 'target' to 'base' in lookup method
* Fix typo
* Apply suggestions from code review
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Rename 'target_hash' to 'base_hash' in 'SelectChain::finality_target()'
* Apply suggestions from code review
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Docs improvement
* Doc fix
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Apply more code suggestions
---------
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Anton <anton.kalyaev@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Rename `*-private-ipv4` to `*-private-ip` CLI args
Renames the `*-private-ipv4` to `*-private-ip` in the CLI interface. The old names are staying as
alias, thus it will not break for anyone. Besides that it also fixes the naming in the rest of the code.
* FMT
* Set genesis block data using the built genesis block
* Make resolve_state_version_from_wasm a separate function and some small refactorings
Useful for the commit following.
* Introduce trait BuildGenesisBlock
Substrate users can use this trait to implement their custom genesis block when constructing the
client.
* Make call_executor test compile
* cargo +nightly fmt --all
* Fix test
* Remove unnecessary clone
* FMT
* Apply review suggestions
* Revert changes to new_full_client() and new_full_parts() signature
* Remove needless `Block` type in `resolve_state_version_from_wasm`
* Replace deprecated libp2p feature specs with correct ones
* Bump tokio to 1.21.2
* Replace async-std libp2p primitives with tokio ones
* minor: rustfmt
* Fix TestNet to run initialization in the tokio context
* Convert telemetry test from async-std to tokio
* Convert notifications tests from async-std to tokio
* Convert chain sync tests from async-std to tokio
* Ditch async-std completely
* Make executor mandatory
* Bump tokio to 1.22.0
* minor: rustfmt
* Explicitly use tokio runtime in tests
* Move more tests to explicit tokio runtime
* Explicitly set multithreaded runtime in tokio test
* minor: rustfmt
* minor: fix comment
* Replace async-std with tokio in MMR tests
* ed25519_verify: Support using dalek for historical blocks
The switch from `ed25519-dalek` to `ed25519-zebra` was actually a breaking change. `ed25519-zebra`
is more permissive. To support historical blocks when syncing a chain this pull request introduces
an externalities extension `UseDalekExt`. This extension is just used as a signaling mechanism to
`ed25519_verify` to use `ed25519-dalek` when it is present. Together with `ExtensionBeforeBlock` it
can be used to setup a node in way to sync historical blocks that require `ed25519-dalek`, because
they included a transaction that verified differently as when using `ed25519-zebra`.
This feature can be enabled in the following way. In the chain service file, directly after the
client is created, the following code should be added:
```
use sc_client_api::ExecutorProvider;
client.execution_extensions().set_extensions_factory(
sc_client_api::execution_extensions::ExtensionBeforeBlock::<Block, sp_io::UseDalekExt>::new(BLOCK_NUMBER_UNTIL_DALEK_SHOULD_BE_USED)
);
```
* Fix doc
* More fixes
* Update client/api/src/execution_extensions.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Fix merge and warning
* Fix docs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* BlockId removal: refactor: Backend::justifications
It changes the arguments of `Backend::justifications` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* trigger CI job
* trigger CI job
* bug fix
* match -> if
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
It changes the arguments of `Backend::body` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
Co-authored-by: parity-processbot <>
* BlockId removal: refactor: Finalizer
It changes the arguments of methods of `Finalizer` trait from:
block: `BlockId<Block>` to: hash: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* minor corrections
* failing test corrected
* minor rework
* 1.Add pruning param "canonical" in sc-cli.
2.Make PruningMode's default value to ArchiveCanonical.
* Update tests in sc-state-db.
* Update tests in sc-state-db.
* 1.Add a new value `AllWithNonFinalized` in `enum BlocksPruning` which Corresponds to `blocks_pruning 0` in CLI .
2.Change value `All` to `AllFinalized` in `enum BlocksPruning` and make it to keep full finalized block history.
* Make some corresponding adjustments based on the content in the conversation.
* Update client/db/src/lib.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* Apply suggestions from code review.
* 1.Change `blocks_pruning` to be like `state_pruning` .
* Fmt and add some doc.
* Update client/cli/src/params/pruning_params.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update client/cli/src/params/pruning_params.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update doc.
* Change `new_test_with_tx_storage` to take `BlocksPruning`.
* Fmt
Co-authored-by: Bastian Köcher <git@kchr.de>
* Use `array-bytes` for All Array/Bytes/Hex Operations
Signed-off-by: Xavier Lau <xavier@inv.cafe>
* Reorder
* Self Review
* Format
* Fix Tests
* Bump `array-bytes`
* Optimize large test res
Signed-off-by: Xavier Lau <xavier@inv.cafe>
Co-authored-by: parity-processbot <>
* trie state cache
* Also cache missing access on read.
* fix comp
* bis
* fix
* use has_lru
* remove local storage cache on size 0.
* No cache.
* local cache only
* trie cache and local cache
* storage cache (with local)
* trie cache no local cache
* Add state access benchmark
* Remove warnings etc
* Add trie cache benchmark
* No extra "clone" required
* Change benchmark to use multiple blocks
* Use patches
* Integrate shitty implementation
* More stuff
* Revert "Merge branch 'master' into trie_state_cache"
This reverts commit 947cd8e6d43fced10e21b76d5b92ffa57b57c318, reversing
changes made to 29ff036463.
* Improve benchmark
* Adapt to latest changes
* Adapt to changes in trie
* Add a test that uses iterator
* Start fixing it
* Remove obsolete file
* Make it compile
* Start rewriting the trie node cache
* More work on the cache
* More docs and code etc
* Make data cache an optional
* Tests
* Remove debug stuff
* Recorder
* Some docs and a simple test for the recorder
* Compile fixes
* Make it compile
* More fixes
* More fixes
* Fix fix fix
* Make sure cache and recorder work together for basic stuff
* Test that data caching and recording works
* Test `TrieDBMut` with caching
* Try something
* Fixes, fixes, fixes
* Forward the recorder
* Make it compile
* Use recorder in more places
* Switch to new `with_optional_recorder` fn
* Refactor and cleanups
* Move `ProvingBackend` tests
* Simplify
* Move over all functionality to the essence
* Fix compilation
* Implement estimate encoded size for StorageProof
* Start using the `cache` everywhere
* Use the cache everywhere
* Fix compilation
* Fix tests
* Adds `TrieBackendBuilder` and enhances the tests
* Ensure that recorder drain checks that values are found as expected
* Switch over to `TrieBackendBuilder`
* Start fixing the problem with child tries and recording
* Fix recording of child tries
* Make it compile
* Overwrite `storage_hash` in `TrieBackend`
* Add `storage_cache` to the benchmarks
* Fix `no_std` build
* Speed up cache lookup
* Extend the state access benchmark to also hash a runtime
* Fix build
* Fix compilation
* Rewrite value cache
* Add lru cache
* Ensure that the cache lru works
* Value cache should not be optional
* Add support for keeping the shared node cache in its bounds
* Make the cache configurable
* Check that the cache respects the bounds
* Adds a new test
* Fixes
* Docs and some renamings
* More docs
* Start using the new recorder
* Fix more code
* Take `self` argument
* Remove warnings
* Fix benchmark
* Fix accounting
* Rip off the state cache
* Start fixing fallout after removing the state cache
* Make it compile after trie changes
* Fix test
* Add some logging
* Some docs
* Some fixups and clean ups
* Fix benchmark
* Remove unneeded file
* Use git for patching
* Make CI happy
* Update primitives/trie/Cargo.toml
Co-authored-by: Koute <koute@users.noreply.github.com>
* Update primitives/state-machine/src/trie_backend.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Introduce new `AsTrieBackend` trait
* Make the LocalTrieCache not clonable
* Make it work in no_std and add docs
* Remove duplicate dependency
* Switch to ahash for better performance
* Speedup value cache merge
* Output errors on underflow
* Ensure the internal LRU map doesn't grow too much
* Use const fn to calculate the value cache element size
* Remove cache configuration
* Fix
* Clear the cache in between for more testing
* Try to come up with a failing test case
* Make the test fail
* Fix the child trie recording
* Make everything compile after the changes to trie
* Adapt to latest trie-db changes
* Fix on stable
* Update primitives/trie/src/cache.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Fix wrong merge
* Docs
* Fix warnings
* Cargo.lock
* Bump pin-project
* Fix warnings
* Switch to released crate version
* More fixes
* Make clippy and rustdocs happy
* More clippy
* Print error when using deprecated `--state-cache-size`
* 🤦
* Fixes
* Fix storage_hash linkings
* Update client/rpc/src/dev/mod.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Review feedback
* encode bound
* Rework the shared value cache
Instead of using a `u64` to represent the key we now use an `Arc<[u8]>`. This arc is also stored in
some extra `HashSet`. We store the key are in an extra `HashSet` to de-duplicate the keys accross
different storage roots. When the latest key usage is dropped in the lru, we also remove the key
from the `HashSet`.
* Improve of the cache by merging the old and new solution
* FMT
* Please stop coming back all the time :crying:
* Update primitives/trie/src/cache/shared_cache.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Fixes
* Make clippy happy
* Ensure we don't deadlock
* Only use one lock to simplify the code
* Do not depend on `Hasher`
* Fix tests
* FMT
* Clippy 🤦
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
Qinxuan Chen