* Fix benchmarks and adds CI to test them
Instead of waiting for benchmarks failing in Polkadot CI, we also can just test them in Substrate :P
* Do not overflow
* trie state cache
* Also cache missing access on read.
* fix comp
* bis
* fix
* use has_lru
* remove local storage cache on size 0.
* No cache.
* local cache only
* trie cache and local cache
* storage cache (with local)
* trie cache no local cache
* Add state access benchmark
* Remove warnings etc
* Add trie cache benchmark
* No extra "clone" required
* Change benchmark to use multiple blocks
* Use patches
* Integrate shitty implementation
* More stuff
* Revert "Merge branch 'master' into trie_state_cache"
This reverts commit 947cd8e6d43fced10e21b76d5b92ffa57b57c318, reversing
changes made to 29ff036463.
* Improve benchmark
* Adapt to latest changes
* Adapt to changes in trie
* Add a test that uses iterator
* Start fixing it
* Remove obsolete file
* Make it compile
* Start rewriting the trie node cache
* More work on the cache
* More docs and code etc
* Make data cache an optional
* Tests
* Remove debug stuff
* Recorder
* Some docs and a simple test for the recorder
* Compile fixes
* Make it compile
* More fixes
* More fixes
* Fix fix fix
* Make sure cache and recorder work together for basic stuff
* Test that data caching and recording works
* Test `TrieDBMut` with caching
* Try something
* Fixes, fixes, fixes
* Forward the recorder
* Make it compile
* Use recorder in more places
* Switch to new `with_optional_recorder` fn
* Refactor and cleanups
* Move `ProvingBackend` tests
* Simplify
* Move over all functionality to the essence
* Fix compilation
* Implement estimate encoded size for StorageProof
* Start using the `cache` everywhere
* Use the cache everywhere
* Fix compilation
* Fix tests
* Adds `TrieBackendBuilder` and enhances the tests
* Ensure that recorder drain checks that values are found as expected
* Switch over to `TrieBackendBuilder`
* Start fixing the problem with child tries and recording
* Fix recording of child tries
* Make it compile
* Overwrite `storage_hash` in `TrieBackend`
* Add `storage_cache` to the benchmarks
* Fix `no_std` build
* Speed up cache lookup
* Extend the state access benchmark to also hash a runtime
* Fix build
* Fix compilation
* Rewrite value cache
* Add lru cache
* Ensure that the cache lru works
* Value cache should not be optional
* Add support for keeping the shared node cache in its bounds
* Make the cache configurable
* Check that the cache respects the bounds
* Adds a new test
* Fixes
* Docs and some renamings
* More docs
* Start using the new recorder
* Fix more code
* Take `self` argument
* Remove warnings
* Fix benchmark
* Fix accounting
* Rip off the state cache
* Start fixing fallout after removing the state cache
* Make it compile after trie changes
* Fix test
* Add some logging
* Some docs
* Some fixups and clean ups
* Fix benchmark
* Remove unneeded file
* Use git for patching
* Make CI happy
* Update primitives/trie/Cargo.toml
Co-authored-by: Koute <koute@users.noreply.github.com>
* Update primitives/state-machine/src/trie_backend.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Introduce new `AsTrieBackend` trait
* Make the LocalTrieCache not clonable
* Make it work in no_std and add docs
* Remove duplicate dependency
* Switch to ahash for better performance
* Speedup value cache merge
* Output errors on underflow
* Ensure the internal LRU map doesn't grow too much
* Use const fn to calculate the value cache element size
* Remove cache configuration
* Fix
* Clear the cache in between for more testing
* Try to come up with a failing test case
* Make the test fail
* Fix the child trie recording
* Make everything compile after the changes to trie
* Adapt to latest trie-db changes
* Fix on stable
* Update primitives/trie/src/cache.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Fix wrong merge
* Docs
* Fix warnings
* Cargo.lock
* Bump pin-project
* Fix warnings
* Switch to released crate version
* More fixes
* Make clippy and rustdocs happy
* More clippy
* Print error when using deprecated `--state-cache-size`
* 🤦
* Fixes
* Fix storage_hash linkings
* Update client/rpc/src/dev/mod.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Review feedback
* encode bound
* Rework the shared value cache
Instead of using a `u64` to represent the key we now use an `Arc<[u8]>`. This arc is also stored in
some extra `HashSet`. We store the key are in an extra `HashSet` to de-duplicate the keys accross
different storage roots. When the latest key usage is dropped in the lru, we also remove the key
from the `HashSet`.
* Improve of the cache by merging the old and new solution
* FMT
* Please stop coming back all the time :crying:
* Update primitives/trie/src/cache/shared_cache.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Fixes
* Make clippy happy
* Ensure we don't deadlock
* Only use one lock to simplify the code
* Do not depend on `Hasher`
* Fix tests
* FMT
* Clippy 🤦
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* transactional: Wrap `pallet::calls` directly in storage layers
Before this pr we only wrapped `pallet::calls` into storage layers when executing the calls with
`dispatch`. This pr is solving that by wrapping each call function inside a storage layer.
* Teach `BasicExternalities` transactions support
* Fix crates
* FMT
* Fix benchmarking tests
* Use correct span
* Support old decl macros
* Fix test
* Apply suggestions from code review
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update frame/state-trie-migration/src/lib.rs
* Update frame/state-trie-migration/src/lib.rs
* Update frame/state-trie-migration/src/lib.rs
* Feedback
* Apply suggestions from code review
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Before `highestBlock` was an optional that was omitted when it was `None`. We recently changed the
way the `highestBlock` is determined, this resulted in having this value in 99.99% of the time being
`None` when the node is syncing blocks at the tip. Now we always return a block for `highestBlock`.
If sync doesn't return us any best seen block, we return our own local best block as `highestBlock`.
This should mainly reflect the same behavior to before we changed the way the best seen block is determined.
* Timestamp: `set_timestamp` sets `DidUpdate`
There exists the `set_timestamp` in the Timestamp pallet for setting the current timestamp. The
problem is that it doesn't set `DidUpdate`. This results in `on_finalize` panicking. There is no
real reason why the function doesn't also set `DidUpdate`.
* Update frame/timestamp/src/lib.rs
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix Babe tests
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Revert non-best block
This makes `revert` also revert non-best blocks.
* Update client/db/src/lib.rs
* Do not count leaves against the maximum number to revert
* Add some explanation
* Fix bug
* Apply suggestions from code review
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Recently we added the wasm binaries to the `rerun-if-changed` list. The problem with that is that
they have a later mtime than the `invoked.timestamp` file and this file's mtime is used to determine
if the `build.rs` script needs to be re-run. The solution to this is that we copy the mtime of this
`invoked.timestamp` file and add it to the wasm binaries. Then cargo/rustc doesn't constantly wants
to rerun the `build.rs` script.
* Sync: Improve major sync detection
When we still have a full import queue, we should still report that we are in major sync mode,
otherwise validators may will already start producing blocks.
* Use median
* Review comments
If `--validator` is passed we also listen on this address and as `--dev` is a shortcut for multiple
CLI args, including `--validator`, we should make it consistent.
* trie: Optimize `keys` function
Instead of iterating the entire state and collecting all keys that match the given prefix, we can
directly use the optimized prefix iterator.
* Add a test
* generate_storage_alias: Rewrite as proc macro attribute
This rewrites the `generate_storage_alias!` declarative macro as proc-macro attribute. While doing
this the name is changed to `storage_alias`. The prefix can now also be the name of a pallet. This
makes storage aliases work in migrations for all kind of chains and not just for the ones that use
predefined prefixes.
* Fix compilation and FMT
* Moare fixes
* 🤦
* ......
* Rework the syntax and support instancing
* FMT
* Prefix variants with `Storage`
* Make it compile
* Fix where clause on rust stable
When passing reserved nodes only with a peer id it was failing with the `DuplicateBootnode` error.
Besides that there are some clean ups. We for example added the bootnodes twice to the `known_addresses`.
* pallet-asset: Fix transfer of a large amount of an asset
Before this pr transferring a large amount of an asset would check that transferring the asset would
not overflow the supply of the asset. However, it doesn't make sense to check for asset supply
overflow when we just transfer from one account to another account and don't increase the supply in
any way. It also required to extend the `can_deposit` method of `fungible` and `fungibles` with a
`mint` parameter. If this parameter is set to `true`, it means we want to mint the amount of an
asset before transferring it into an account. For `can_withdraw` we don't need to add an extra
parameter, because withdrawing should never be able to underflow the supply. If that would happen,
it would mean that somewhere the supply wasn't increased while increasing the balance of an account.
* Update frame/assets/src/functions.rs
* Update frame/assets/src/functions.rs
* Update frame/assets/src/functions.rs
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
* FMT
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
This tries to fix the CI if there is no polkadot companion. Currently we don't update Polkadot
master in Cumulus, which means we may use some old commit that isn't compiling with the latest
Substrate master anymore. This can happen if there was a pr that had a companion in Polkadot, but no
companion was required for Cumulus. Then Cumulus will still point to some old Polkadot commit that
isn't compiling anymore with the latest Substrate commit. So, we need to tell the script to use the
latest master of Polkadot. If there is a companion for Polkadot, it would simply override the extra
dependency patch later on.
With the latest optimizations of the `FinalityNotification` generation, the aux data pruning started
to print a warning. The problem here was that we printed a warning and stopped the adding of blocks
to prune when we hit the `heigh_limit`. This is now wrong, as we could for example have two 512 long
forks and then we start finalizing one of them. The second fork head would be part of the stale
heads at some point (in the current implementation when we finalize second fork head number + 1),
but then we would actually need to go back into the past than `heigh_limit` (which was actually
last_finalized - 1). We now go back until we reach the canonical chain.
Also fixed some wrong comment that was added by be about the content of the `finalized` blocks in
the `FinalityNotification`.
* Finality notification: Optimize calculation of stale heads
While looking into some problem on Versi where a collator seemed to be stuck. I found out that it
was not stuck but there was a huge gap between last finalized and best block. This lead to a lot
leaves and it was basically trapped inside some loop of reading block headers from the db to find
the stale heads. While looking into this I found out that `leaves` already supports the feature to
give us the stale heads relative easily. However, the semantics change a little bit. Instead of
returning all stale heads of blocks that are not reachable anymore after finalizing a block, we
currently only return heads with a number lower than the finalized block. This should be no problem,
because these other leaves that are stale will be returned later when a block gets finalized which
number is bigger than the block number of these leaves.
While doing that, I also changed `tree_route` of the `FinalityNotification` to include the
`old_finalized`. Based on the comment I assumed that this was already part of it. However, if
wanted, I can revert this change.
* FMT
* Update client/service/src/client/client.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Do not include the last finalized block
* Rename function
* FMT
* Fix tests
* Update figure
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Prepare for rust stable 1.59
Besides preparing the UI tests this also adds a new script update-rust-stable.sh script for
simplifying the update of a rust stable version. This script will run all UI tests for the new
rust stable version and updating the expected output.
* Ensure we run the UI tests in CI
* use staging ci image
* More test updates
* Unignore test (#11097)
* empty commit for pipeline rerun
* empty commit for pipeline rerun
* Try to make clippy happy
* More clippy fixes
* FMT
* ci image production
Co-authored-by: alvicsam <alvicsam@gmail.com>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
These docs didn't mention how the removal works internally. This is important for the user to know
that calling such a method multiple times in the same block leads always to the same result.
This pr changes all default execution strategies to `Wasm`. This is basically a deprecation of the
native runtime. While the native runtime isn't removed and can still be used, it will not be used
anymore by default.
This will also improve the usage for people who want to run random commits, which most often forget
to run with `--execution wasm`. Otherwise they often run into storage root mismatches because of
using the native runtime.
* transactional: Fix some nitpicks
This fixes some nitpicks related to the transactional storage stuff from me. As everything was
merged too fast, here are some nitpicks from me. First, the entire functionality is moved into its
own file to have a clear separation. Secondly I changed the `set_transactional_level` to not take
`Layer` by reference. Besides that I have added some docs etc.
* Add some comment
* Move tests
* 🤦
* import-blocks: Do not read `stdin` to memory
This fixes a bug with `import-blocks` reading the entire `stdin` before starting to import the
blocks. However, for huge files that uses quite a lot of memory. We can just read from `stdin` step
by step as we do it with a file. This ensures that we don't read the entire input at once into memory.
* FMT
* Fix warning
* pallet-macro: Ensure that building with `missing_docs` works
Before this pr it was failing when compiling with `missing_docs`, because the macro was generating
functions etc without the appropriate docs. In the case of this pr it is mainly about hiding these
functions in the docs as they are internal api anyway.
* Fix UI test
Ensure that the sp api macros don't use functions without providing the full path to the function.
This hygiene ensures that we don't actually try to call a method of an imported trait for example.
* sp-core: `full_crypto` doesn't imply `std`
This pr changes the feature set of `secp256k1` to not use `global-context` when only the
`full_crypto` is enabled. It will be slower when the `std` feature is not enabled as the context
always needs to be recreated, but that is fine.
* Update client/cli/src/arg_enums.rs
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Because Rust currently doesn't support specialization, it prevents users from implementing
`SlotWorker` for their own types. This pr solves this by removing the generic implementation of
`SlotWorker` for `SimpleSlotWorker` and providing some wrapper type for that.
* sc-cli: Fix bugs after switching to clap3
Before switching to clap3 we support cli options like `--reserved-nodes A B` and after you needed to
pass `--reserved-nodes` cli option multiple times `--reserved-nodes A --reserved-nodes B`. This is
fixed by setting `multiple_occurrences(true)` option. This also done for all the other `Vec` cli
options in `sc-cli`. Besides that `--sync` wasn't supporting case insensitive parsing of the value.
This is now also supported. For both regressions a test is added. Besides that the pr removes all
the `rename_all = PascalCase` attributes, because they are not needed. All other `ArgEnum`s were
checked and all are already using `ignore_case(true)`.
* Bring back `PascalCase`, because otherwise it falls back to `kebab-case`...
* state-machine: Move all functionality from trie backend to the essence
This is required for some future changes of me and it also makes more sense to have all the
functionality inside the essence. Besides that it changes the child root cache to directly
cache the hash.
* Update primitives/state-machine/src/trie_backend_essence.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* FMT
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* wasm-builder: Support latest nightly
With latest nightly, aka rust version 1.60+ namespaced features are added. This changes the handling
of optional dependencies. We currently have features that enable optional dependencies when `std` is
enabled. This was before no problem, but now the wasm-builder detects them as enabled. To support
the transition period until 1.60 is released as stable, this pr adds an heuristic to not enable these
optional crates in the wasm build when they are enabled in the `std` feature. This heuristic fails
when someones enables these optional dependencies from the outside as well as via the `std` feature,
however we hope that no one is doing this at the moment. When namespaced features are enabled, these
dependencies needs to be enabled using `dep:dependency-name` to solve this properly.
https://doc.rust-lang.org/cargo/reference/unstable.html#namespaced-features
* Remove accidentally added features
* Remove `u32_trait`
This trait only existed because there wasn't any const generic support at time of creation. However,
we now have support for it :)
* FMT
* pallet-scheduler: Fix migrations V2 to V3
V2 already supported origins, so we need to move them over instead of setting it to `Root`. Besides
that it also removes the custom `Releases` enum and moves it over to `StorageVersion`.
* Fixes
* Fixes
* 🤦
This is required for projects like Cumulus that have dependencies on `westend-runtime`, but this
runtime is only added when the `runtime-benchmarks` feature is enabled. By having all features
"enabled" in `cargo metadata` we ensure that all crates can be found.
* Remove some last `AccountId32::default()` leftovers
As we removed `Default` of account id, we have overseen some last bits. This pr removes these last
bits.
* Fix
When building a wasm binary from a different repo inside a local workspace, we did not used the
correct `Cargo.toml` to find the correct patches and features. The solution to this is to just walk
up from the target directory until we find the workspace we are currently compiling. If this
heuristic isn't working, we print a warning and let the user set an env variable
`WASM_BUILD_WORKSPACE_HINT` to tell the `wasm-builder` where the actual workspace is.
* runtime-interface: Implement `register_only` functions
The runtime interface supports versioning of functions. Currently, if you add a new function it will
be used by the runtime automatically. This results in requiring all nodes of a network to upgrade
before the runtime is upgraded, otherwise they will fail to instantiate the new runtime because of
missing host functions. This pr introduces `register_only` functions. This can be used when a new
runtime interface function should be introduced, but the actual usage can be deferred. This means
that nodes will have the host function for this, but the runtime will still use the old version of
the function when being compiled for wasm. However, when a runtime is enacted that uses the new host
function, the "old nodes" will already have the host function and will continue to work.
* Update primitives/runtime-interface/src/lib.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Update primitives/runtime-interface/proc-macro/src/utils.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* FMT
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
`runtime_print!` is printed by default using `debug`, aka not being visible. With `log::error!` it
will be printed directly to the user. Production networks like Polkadot disable logging, but for
them we run special nodes that have logging enabled.
The keystore would print "Invalid password" when a key was stored using an incorrect public key.
This pr improves the error message to communicate better to the user on what is wrong.
* Remove transaction-pool `test-helpers` feature
`test-helpers` feature is a bad idea in general, because once the feature is enabled somewhere in
the workspace, it is enabled anywhere. While removing the feature, the tests were also rewritten to
get rid off other "only test" related code.
Contributes towards: https://github.com/paritytech/substrate/issues/9727
* Apply suggestions from code review
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Fix benches
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Improve SS58 related errors
This improves the SS58 error, especially when it comes to parsing public keys with unknown SS58
address formats.
* Make CI happy
* More fixes
* More
* 🤦
* fml...
* chain-spec: Make chain spec writing deterministic
This switches the chain spec to use `BTreeMap` internally instead of `HashMap`. This ensures that
reading and writing the same chain spec always results in the same file.
* fmt
* SlotDuration: Always fetch the slot duration from the runtime
The slot duration should always be fetched from the runtime instead of being cached in the db. The
slot duration is only fetched on startup of the node, so the performance isn't that important. This
is especially helpful for the case when the slot duration of a chain should be changed through a
runtime upgrade (there be dragons, so take care).
* Fix docs
* Remove logging
* Fix warning
* rpc-author: Downgrade the logging level of some logs
Currently these messages are printed with `warn` levels, but they don't bring the node operator
anything. The problem with these log lines is that they are triggered by external users when they
send invalid transactions. This pr "resolves" this by downgrading the log severity to debug, aka not
shown by default.
* FMT
* Remove cargo unleash CI job
The CI job is currently breaking, because we released some crates. We also don't need the job
currently because releases are not done. So, the job currently just annoys people and eats CPU time.
This brings us to the conclusion that this job is currently not required and can be readded at a
later point when we are sure that we need it.
* Hopefully fix UI test
* Another try
* Remove `reserved_keyword` ui
* Introduce `SecretUri`
* `inspect-key`: Adds support for `expect-public`
`expect-public` can be used to check that a given secret uri corresponds to the given public key.
This is mainly useful when the secret uri is protected by a password and a new derived account
should be generated. With `--expect-public` the user can pass the public key/account-id of the
"base" secret uri aka the one without any derivation to ensure the correct password was inserted.
* Fixes
* 🤦
* Apply suggestions from code review
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Review feedback
* FMT
* Bump the versions
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
This adds support to the wasm-override feature to compare the spec_name. If the spec_name doesn't
match, a warning will be printed and the override will be ignored.
There is no reason for this function, tokio already blocks automatically until all tasks are ended.
Another reason to remove this feature is `mpsc_background_tasks` unbounded channel. Recently this
channel was reporting too many unprocessed elements. We assume that this was a result of a lot of
very shot lived tasks that somehow flooded this channel.
* Offchain worker: Enable http2 and improve logging
Apparently some webpages now return http2 by default and that silently breaks the offchain http
extension. The solution to this is to enable the `http2` feature of hyper. Besides that, this pr
improves the logging to make it easier to debug such errors.
* FMT
* Adds http2 test
* authority-discovery: Support multiple authority ids per peer id
An peer id can be mapped to multiple authority ids, because an authority id is a session key that
could be changed every session. Before this pr the internal authority discovery cache assumed that
each authority id can only be mapped to one peer id. However, this isn't true since we changed the
default implementation of the authority discovery to combine the current and next session authorities.
* Review feedback
* Update client/authority-discovery/src/worker/addr_cache.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* Early return on no peer ids
* Update client/authority-discovery/src/worker/addr_cache.rs
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Update types in comment
* FMT
* Add warning
* Update client/authority-discovery/src/worker/addr_cache.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* Feedback
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Cli: Introduce `--detailed-log-output` flag
If this CLI flag is given, detailed log output will be enabled. This includes the log level, log
target ad the thread name. Before this was only enabled when a log level higher than `info` should
be logged.
* Update client/tracing/src/logging/mod.rs
Co-authored-by: David <dvdplm@gmail.com>
Co-authored-by: David <dvdplm@gmail.com>
Bastian Köcher