* Fix benchmarks and adds CI to test them
Instead of waiting for benchmarks failing in Polkadot CI, we also can just test them in Substrate :P
* Do not overflow
* trie state cache
* Also cache missing access on read.
* fix comp
* bis
* fix
* use has_lru
* remove local storage cache on size 0.
* No cache.
* local cache only
* trie cache and local cache
* storage cache (with local)
* trie cache no local cache
* Add state access benchmark
* Remove warnings etc
* Add trie cache benchmark
* No extra "clone" required
* Change benchmark to use multiple blocks
* Use patches
* Integrate shitty implementation
* More stuff
* Revert "Merge branch 'master' into trie_state_cache"
This reverts commit 947cd8e6d43fced10e21b76d5b92ffa57b57c318, reversing
changes made to 29ff036463.
* Improve benchmark
* Adapt to latest changes
* Adapt to changes in trie
* Add a test that uses iterator
* Start fixing it
* Remove obsolete file
* Make it compile
* Start rewriting the trie node cache
* More work on the cache
* More docs and code etc
* Make data cache an optional
* Tests
* Remove debug stuff
* Recorder
* Some docs and a simple test for the recorder
* Compile fixes
* Make it compile
* More fixes
* More fixes
* Fix fix fix
* Make sure cache and recorder work together for basic stuff
* Test that data caching and recording works
* Test `TrieDBMut` with caching
* Try something
* Fixes, fixes, fixes
* Forward the recorder
* Make it compile
* Use recorder in more places
* Switch to new `with_optional_recorder` fn
* Refactor and cleanups
* Move `ProvingBackend` tests
* Simplify
* Move over all functionality to the essence
* Fix compilation
* Implement estimate encoded size for StorageProof
* Start using the `cache` everywhere
* Use the cache everywhere
* Fix compilation
* Fix tests
* Adds `TrieBackendBuilder` and enhances the tests
* Ensure that recorder drain checks that values are found as expected
* Switch over to `TrieBackendBuilder`
* Start fixing the problem with child tries and recording
* Fix recording of child tries
* Make it compile
* Overwrite `storage_hash` in `TrieBackend`
* Add `storage_cache` to the benchmarks
* Fix `no_std` build
* Speed up cache lookup
* Extend the state access benchmark to also hash a runtime
* Fix build
* Fix compilation
* Rewrite value cache
* Add lru cache
* Ensure that the cache lru works
* Value cache should not be optional
* Add support for keeping the shared node cache in its bounds
* Make the cache configurable
* Check that the cache respects the bounds
* Adds a new test
* Fixes
* Docs and some renamings
* More docs
* Start using the new recorder
* Fix more code
* Take `self` argument
* Remove warnings
* Fix benchmark
* Fix accounting
* Rip off the state cache
* Start fixing fallout after removing the state cache
* Make it compile after trie changes
* Fix test
* Add some logging
* Some docs
* Some fixups and clean ups
* Fix benchmark
* Remove unneeded file
* Use git for patching
* Make CI happy
* Update primitives/trie/Cargo.toml
Co-authored-by: Koute <koute@users.noreply.github.com>
* Update primitives/state-machine/src/trie_backend.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Introduce new `AsTrieBackend` trait
* Make the LocalTrieCache not clonable
* Make it work in no_std and add docs
* Remove duplicate dependency
* Switch to ahash for better performance
* Speedup value cache merge
* Output errors on underflow
* Ensure the internal LRU map doesn't grow too much
* Use const fn to calculate the value cache element size
* Remove cache configuration
* Fix
* Clear the cache in between for more testing
* Try to come up with a failing test case
* Make the test fail
* Fix the child trie recording
* Make everything compile after the changes to trie
* Adapt to latest trie-db changes
* Fix on stable
* Update primitives/trie/src/cache.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Fix wrong merge
* Docs
* Fix warnings
* Cargo.lock
* Bump pin-project
* Fix warnings
* Switch to released crate version
* More fixes
* Make clippy and rustdocs happy
* More clippy
* Print error when using deprecated `--state-cache-size`
* 🤦
* Fixes
* Fix storage_hash linkings
* Update client/rpc/src/dev/mod.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Review feedback
* encode bound
* Rework the shared value cache
Instead of using a `u64` to represent the key we now use an `Arc<[u8]>`. This arc is also stored in
some extra `HashSet`. We store the key are in an extra `HashSet` to de-duplicate the keys accross
different storage roots. When the latest key usage is dropped in the lru, we also remove the key
from the `HashSet`.
* Improve of the cache by merging the old and new solution
* FMT
* Please stop coming back all the time :crying:
* Update primitives/trie/src/cache/shared_cache.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Fixes
* Make clippy happy
* Ensure we don't deadlock
* Only use one lock to simplify the code
* Do not depend on `Hasher`
* Fix tests
* FMT
* Clippy 🤦
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* transactional: Wrap `pallet::calls` directly in storage layers
Before this pr we only wrapped `pallet::calls` into storage layers when executing the calls with
`dispatch`. This pr is solving that by wrapping each call function inside a storage layer.
* Teach `BasicExternalities` transactions support
* Fix crates
* FMT
* Fix benchmarking tests
* Use correct span
* Support old decl macros
* Fix test
* Apply suggestions from code review
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update frame/state-trie-migration/src/lib.rs
* Update frame/state-trie-migration/src/lib.rs
* Update frame/state-trie-migration/src/lib.rs
* Feedback
* Apply suggestions from code review
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Before `highestBlock` was an optional that was omitted when it was `None`. We recently changed the
way the `highestBlock` is determined, this resulted in having this value in 99.99% of the time being
`None` when the node is syncing blocks at the tip. Now we always return a block for `highestBlock`.
If sync doesn't return us any best seen block, we return our own local best block as `highestBlock`.
This should mainly reflect the same behavior to before we changed the way the best seen block is determined.
* Timestamp: `set_timestamp` sets `DidUpdate`
There exists the `set_timestamp` in the Timestamp pallet for setting the current timestamp. The
problem is that it doesn't set `DidUpdate`. This results in `on_finalize` panicking. There is no
real reason why the function doesn't also set `DidUpdate`.
* Update frame/timestamp/src/lib.rs
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix Babe tests
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Revert non-best block
This makes `revert` also revert non-best blocks.
* Update client/db/src/lib.rs
* Do not count leaves against the maximum number to revert
* Add some explanation
* Fix bug
* Apply suggestions from code review
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Recently we added the wasm binaries to the `rerun-if-changed` list. The problem with that is that
they have a later mtime than the `invoked.timestamp` file and this file's mtime is used to determine
if the `build.rs` script needs to be re-run. The solution to this is that we copy the mtime of this
`invoked.timestamp` file and add it to the wasm binaries. Then cargo/rustc doesn't constantly wants
to rerun the `build.rs` script.
* Ensure relay chain block number strictly increases
This is a safeguard for when async backing is enabled on the relay chain and multiple parachain
blocks per relay chain block are legal. We will need to later change this to support then multiple
parachain blocks per relay chain block.
* Make the check configurable
* Fix compilation
* Update pallets/parachain-system/src/lib.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Update pallets/parachain-system/src/lib.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Fix test
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Sync: Improve major sync detection
When we still have a full import queue, we should still report that we are in major sync mode,
otherwise validators may will already start producing blocks.
* Use median
* Review comments
If `--validator` is passed we also listen on this address and as `--dev` is a shortcut for multiple
CLI args, including `--validator`, we should make it consistent.
* trie: Optimize `keys` function
Instead of iterating the entire state and collecting all keys that match the given prefix, we can
directly use the optimized prefix iterator.
* Add a test
* generate_storage_alias: Rewrite as proc macro attribute
This rewrites the `generate_storage_alias!` declarative macro as proc-macro attribute. While doing
this the name is changed to `storage_alias`. The prefix can now also be the name of a pallet. This
makes storage aliases work in migrations for all kind of chains and not just for the ones that use
predefined prefixes.
* Fix compilation and FMT
* Moare fixes
* 🤦
* ......
* Rework the syntax and support instancing
* FMT
* Prefix variants with `Storage`
* Make it compile
* Fix where clause on rust stable
When passing reserved nodes only with a peer id it was failing with the `DuplicateBootnode` error.
Besides that there are some clean ups. We for example added the bootnodes twice to the `known_addresses`.
* pallet-asset: Fix transfer of a large amount of an asset
Before this pr transferring a large amount of an asset would check that transferring the asset would
not overflow the supply of the asset. However, it doesn't make sense to check for asset supply
overflow when we just transfer from one account to another account and don't increase the supply in
any way. It also required to extend the `can_deposit` method of `fungible` and `fungibles` with a
`mint` parameter. If this parameter is set to `true`, it means we want to mint the amount of an
asset before transferring it into an account. For `can_withdraw` we don't need to add an extra
parameter, because withdrawing should never be able to underflow the supply. If that would happen,
it would mean that somewhere the supply wasn't increased while increasing the balance of an account.
* Update frame/assets/src/functions.rs
* Update frame/assets/src/functions.rs
* Update frame/assets/src/functions.rs
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
* FMT
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
This tries to fix the CI if there is no polkadot companion. Currently we don't update Polkadot
master in Cumulus, which means we may use some old commit that isn't compiling with the latest
Substrate master anymore. This can happen if there was a pr that had a companion in Polkadot, but no
companion was required for Cumulus. Then Cumulus will still point to some old Polkadot commit that
isn't compiling anymore with the latest Substrate commit. So, we need to tell the script to use the
latest master of Polkadot. If there is a companion for Polkadot, it would simply override the extra
dependency patch later on.
With the latest optimizations of the `FinalityNotification` generation, the aux data pruning started
to print a warning. The problem here was that we printed a warning and stopped the adding of blocks
to prune when we hit the `heigh_limit`. This is now wrong, as we could for example have two 512 long
forks and then we start finalizing one of them. The second fork head would be part of the stale
heads at some point (in the current implementation when we finalize second fork head number + 1),
but then we would actually need to go back into the past than `heigh_limit` (which was actually
last_finalized - 1). We now go back until we reach the canonical chain.
Also fixed some wrong comment that was added by be about the content of the `finalized` blocks in
the `FinalityNotification`.
* Finality notification: Optimize calculation of stale heads
While looking into some problem on Versi where a collator seemed to be stuck. I found out that it
was not stuck but there was a huge gap between last finalized and best block. This lead to a lot
leaves and it was basically trapped inside some loop of reading block headers from the db to find
the stale heads. While looking into this I found out that `leaves` already supports the feature to
give us the stale heads relative easily. However, the semantics change a little bit. Instead of
returning all stale heads of blocks that are not reachable anymore after finalizing a block, we
currently only return heads with a number lower than the finalized block. This should be no problem,
because these other leaves that are stale will be returned later when a block gets finalized which
number is bigger than the block number of these leaves.
While doing that, I also changed `tree_route` of the `FinalityNotification` to include the
`old_finalized`. Based on the comment I assumed that this was already part of it. However, if
wanted, I can revert this change.
* FMT
* Update client/service/src/client/client.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Do not include the last finalized block
* Rename function
* FMT
* Fix tests
* Update figure
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Prepare for rust stable 1.59
Besides preparing the UI tests this also adds a new script update-rust-stable.sh script for
simplifying the update of a rust stable version. This script will run all UI tests for the new
rust stable version and updating the expected output.
* Ensure we run the UI tests in CI
* use staging ci image
* More test updates
* Unignore test (#11097)
* empty commit for pipeline rerun
* empty commit for pipeline rerun
* Try to make clippy happy
* More clippy fixes
* FMT
* ci image production
Co-authored-by: alvicsam <alvicsam@gmail.com>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
These docs didn't mention how the removal works internally. This is important for the user to know
that calling such a method multiple times in the same block leads always to the same result.
This pr changes all default execution strategies to `Wasm`. This is basically a deprecation of the
native runtime. While the native runtime isn't removed and can still be used, it will not be used
anymore by default.
This will also improve the usage for people who want to run random commits, which most often forget
to run with `--execution wasm`. Otherwise they often run into storage root mismatches because of
using the native runtime.
* transactional: Fix some nitpicks
This fixes some nitpicks related to the transactional storage stuff from me. As everything was
merged too fast, here are some nitpicks from me. First, the entire functionality is moved into its
own file to have a clear separation. Secondly I changed the `set_transactional_level` to not take
`Layer` by reference. Besides that I have added some docs etc.
* Add some comment
* Move tests
* 🤦
* import-blocks: Do not read `stdin` to memory
This fixes a bug with `import-blocks` reading the entire `stdin` before starting to import the
blocks. However, for huge files that uses quite a lot of memory. We can just read from `stdin` step
by step as we do it with a file. This ensures that we don't read the entire input at once into memory.
* FMT
* Fix warning
* pallet-macro: Ensure that building with `missing_docs` works
Before this pr it was failing when compiling with `missing_docs`, because the macro was generating
functions etc without the appropriate docs. In the case of this pr it is mainly about hiding these
functions in the docs as they are internal api anyway.
* Fix UI test
Ensure that the sp api macros don't use functions without providing the full path to the function.
This hygiene ensures that we don't actually try to call a method of an imported trait for example.
* sp-core: `full_crypto` doesn't imply `std`
This pr changes the feature set of `secp256k1` to not use `global-context` when only the
`full_crypto` is enabled. It will be slower when the `std` feature is not enabled as the context
always needs to be recreated, but that is fine.
* Update client/cli/src/arg_enums.rs
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Davide Galassi <davxy@datawok.net>
* polkadot-collator: Switch to wasm only
This switches the polkadot-collator to run everything in wasm only mode. While we should not that
yet with the relay chain, because it can happen that we run out of memory (very unlikely). On the
relay chain that would be bad, because we only have at max 2 sessions to bring everything back, for
Parachains that isn't such a problem as they would only stall and we could roll out a release that
fixes it. Besides that, Parachain validation on the relay chain happens in Wasm already all the time
and there is the memory usage even higher then on block import.
* cargo fmt
* remove unused var
Co-authored-by: Squirrel <gilescope@gmail.com>
Because Rust currently doesn't support specialization, it prevents users from implementing
`SlotWorker` for their own types. This pr solves this by removing the generic implementation of
`SlotWorker` for `SimpleSlotWorker` and providing some wrapper type for that.
* sc-cli: Fix bugs after switching to clap3
Before switching to clap3 we support cli options like `--reserved-nodes A B` and after you needed to
pass `--reserved-nodes` cli option multiple times `--reserved-nodes A --reserved-nodes B`. This is
fixed by setting `multiple_occurrences(true)` option. This also done for all the other `Vec` cli
options in `sc-cli`. Besides that `--sync` wasn't supporting case insensitive parsing of the value.
This is now also supported. For both regressions a test is added. Besides that the pr removes all
the `rename_all = PascalCase` attributes, because they are not needed. All other `ArgEnum`s were
checked and all are already using `ignore_case(true)`.
* Bring back `PascalCase`, because otherwise it falls back to `kebab-case`...
This is a crate that shouldn't be used outside of the Substrate Node. Here the correct crate is the
`polkadot-core-primitives` to get the `Balance` type.
* state-machine: Move all functionality from trie backend to the essence
This is required for some future changes of me and it also makes more sense to have all the
functionality inside the essence. Besides that it changes the child root cache to directly
cache the hash.
* Update primitives/state-machine/src/trie_backend_essence.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* FMT
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Before this fix the relay chain node name was always a random generated, because we did not had
implemented all the required methods of the `CliConfiguration` trait.
* wasm-builder: Support latest nightly
With latest nightly, aka rust version 1.60+ namespaced features are added. This changes the handling
of optional dependencies. We currently have features that enable optional dependencies when `std` is
enabled. This was before no problem, but now the wasm-builder detects them as enabled. To support
the transition period until 1.60 is released as stable, this pr adds an heuristic to not enable these
optional crates in the wasm build when they are enabled in the `std` feature. This heuristic fails
when someones enables these optional dependencies from the outside as well as via the `std` feature,
however we hope that no one is doing this at the moment. When namespaced features are enabled, these
dependencies needs to be enabled using `dep:dependency-name` to solve this properly.
https://doc.rust-lang.org/cargo/reference/unstable.html#namespaced-features
* Remove accidentally added features
* Remove `u32_trait`
This trait only existed because there wasn't any const generic support at time of creation. However,
we now have support for it :)
* FMT
* Return compressed PoV from collation
This enables us to also print the size of the compressed PoV.
* FMT
* Update Polkadot & Substrate
* Fix compilation
* Fixes
* FMT
Bastian Köcher