# Description
- What does this PR do?
1. Upgrades `trie-db`'s version to the latest release. This release
includes, among others, an implementation of `DoubleEndedIterator` for
the `TrieDB` struct, allowing to iterate both backwards and forwards
within the leaves of a trie.
2. Upgrades `trie-bench` to `0.39.0` for compatibility.
3. Upgrades `criterion` to `0.5.1` for compatibility.
- Why are these changes needed?
Besides keeping up with the upgrade of `trie-db`, this specifically adds
the functionality of iterating back on the leafs of a trie, with
`sp-trie`. In a project we're currently working on, this comes very
handy to verify a Merkle proof that is the response to a challenge. The
challenge is a random hash that (most likely) will not be an existing
leaf in the trie. So the challenged user, has to provide a Merkle proof
of the previous and next existing leafs in the trie, that surround the
random challenged hash.
Without having DoubleEnded iterators, we're forced to iterate until we
find the first existing leaf, like so:
```rust
// ************* VERIFIER (RUNTIME) *************
// Verify proof. This generates a partial trie based on the proof and
// checks that the root hash matches the `expected_root`.
let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();
// Print all leaf node keys and values.
println!("\nPrinting leaf nodes of partial tree...");
for key in trie.key_iter().unwrap() {
if key.is_ok() {
println!("Leaf node key: {:?}", key.clone().unwrap());
let val = trie.get(&key.unwrap());
if val.is_ok() {
println!("Leaf node value: {:?}", val.unwrap());
} else {
println!("Leaf node value: None");
}
}
}
println!("RECONSTRUCTED TRIE {:#?}", trie);
// Create an iterator over the leaf nodes.
let mut iter = trie.iter().unwrap();
// First element with a value should be the previous existing leaf to the challenged hash.
let mut prev_key = None;
for element in &mut iter {
if element.is_ok() {
let (key, _) = element.unwrap();
prev_key = Some(key);
break;
}
}
assert!(prev_key.is_some());
// Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
assert!(prev_key.unwrap() <= challenge_hash.to_vec());
// The next element should exist (meaning there is no other existing leaf between the
// previous and next leaf) and it should be greater than the challenged hash.
let next_key = iter.next().unwrap().unwrap().0;
assert!(next_key >= challenge_hash.to_vec());
```
With DoubleEnded iterators, we can avoid that, like this:
```rust
// ************* VERIFIER (RUNTIME) *************
// Verify proof. This generates a partial trie based on the proof and
// checks that the root hash matches the `expected_root`.
let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();
// Print all leaf node keys and values.
println!("\nPrinting leaf nodes of partial tree...");
for key in trie.key_iter().unwrap() {
if key.is_ok() {
println!("Leaf node key: {:?}", key.clone().unwrap());
let val = trie.get(&key.unwrap());
if val.is_ok() {
println!("Leaf node value: {:?}", val.unwrap());
} else {
println!("Leaf node value: None");
}
}
}
// println!("RECONSTRUCTED TRIE {:#?}", trie);
println!("\nChallenged key: {:?}", challenge_hash);
// Create an iterator over the leaf nodes.
let mut double_ended_iter = trie.into_double_ended_iter().unwrap();
// First element with a value should be the previous existing leaf to the challenged hash.
double_ended_iter.seek(&challenge_hash.to_vec()).unwrap();
let next_key = double_ended_iter.next_back().unwrap().unwrap().0;
let prev_key = double_ended_iter.next_back().unwrap().unwrap().0;
// Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
println!("Prev key: {:?}", prev_key);
assert!(prev_key <= challenge_hash.to_vec());
println!("Next key: {:?}", next_key);
assert!(next_key >= challenge_hash.to_vec());
```
- How were these changes implemented and what do they affect?
All that is needed for this functionality to be exposed is changing the
version number of `trie-db` in all the `Cargo.toml`s applicable, and
re-exporting some additional structs from `trie-db` in `sp-trie`.
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Working towards migrating the `parity-bridges-common` repo inside
`polkadot-sdk`. This PR upgrades some dependencies in order to align
them with the versions used in `parity-bridges-common`
Related to
https://github.com/paritytech/parity-bridges-common/issues/2538
**Update:** Pushed additional changes based on the review comments.
**This pull request fixes various spelling mistakes in this
repository.**
Most of the changes are contained in the first **3** commits:
- `Fix spelling mistakes in comments and docs`
- `Fix spelling mistakes in test names`
- `Fix spelling mistakes in error messages, panic messages, logs and
tracing`
Other source code spelling mistakes are separated into individual
commits for easier reviewing:
- `Fix the spelling of 'authority'`
- `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'`
- `Fix the spelling of 'prev_enqueud_messages'`
- `Fix the spelling of 'endpoint'`
- `Fix the spelling of 'children'`
- `Fix the spelling of 'PenpalSiblingSovereignAccount'`
- `Fix the spelling of 'PenpalSudoAccount'`
- `Fix the spelling of 'insufficient'`
- `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'`
- `Fix the spelling of 'subtracted'`
- `Fix the spelling of 'CandidatePendingAvailability'`
- `Fix the spelling of 'exclusive'`
- `Fix the spelling of 'until'`
- `Fix the spelling of 'discriminator'`
- `Fix the spelling of 'nonexistent'`
- `Fix the spelling of 'subsystem'`
- `Fix the spelling of 'indices'`
- `Fix the spelling of 'committed'`
- `Fix the spelling of 'topology'`
- `Fix the spelling of 'response'`
- `Fix the spelling of 'beneficiary'`
- `Fix the spelling of 'formatted'`
- `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'`
- `Fix the spelling of 'succeeded'`
- `Fix the spelling of 'reopened'`
- `Fix the spelling of 'proposer'`
- `Fix the spelling of 'InstantiationNonce'`
- `Fix the spelling of 'depositor'`
- `Fix the spelling of 'expiration'`
- `Fix the spelling of 'phantom'`
- `Fix the spelling of 'AggregatedKeyValue'`
- `Fix the spelling of 'randomness'`
- `Fix the spelling of 'defendant'`
- `Fix the spelling of 'AquaticMammal'`
- `Fix the spelling of 'transactions'`
- `Fix the spelling of 'PassingTracingSubscriber'`
- `Fix the spelling of 'TxSignaturePayload'`
- `Fix the spelling of 'versioning'`
- `Fix the spelling of 'descendant'`
- `Fix the spelling of 'overridden'`
- `Fix the spelling of 'network'`
Let me know if this structure is adequate.
**Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`,
`Merkelization`, `Merkleization`, is somewhat inconsistent but I left it
as it is.
~~**Note:** In some places the term `Receival` is used to refer to
message reception, IMO `Reception` is the correct word here, but I left
it as it is.~~
~~**Note:** In some places the term `Overlayed` is used instead of the
more acceptable version `Overlaid` but I also left it as it is.~~
~~**Note:** In some places the term `Applyable` is used instead of the
correct version `Applicable` but I also left it as it is.~~
**Note:** Some usage of British vs American english e.g. `judgement` vs
`judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc.
are both present in different places, but I suppose that's
understandable given the number of contributors.
~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it
triggers errors in CI when I make changes to it, so I left it as it
is.~~
This PR removes sp-std crate from substrate/primitives sub-directories.
For now crates that have `pub use` of sp-std or export macros that would
necessitate users of the macros to `extern crate alloc` have been
excluded from this PR.
There should be no breaking changes in this PR.
---------
Co-authored-by: Koute <koute@users.noreply.github.com>
Lifting some more dependencies to the workspace. Just using the
most-often updated ones for now.
It can be reproduced locally.
```sh
# First you can check if there would be semver incompatible bumps (looks good in this case):
$ zepter transpose dependency lift-to-workspace --ignore-errors syn quote thiserror "regex:^serde.*"
# Then apply the changes:
$ zepter transpose dependency lift-to-workspace --version-resolver=highest syn quote thiserror "regex:^serde.*" --fix
# And format the changes:
$ taplo format --config .config/taplo.toml
```
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
First in a series of PRs that reduces our use of sp-std with a view to
deprecating it.
This is just looking at /substrate and moving some of the references
from `sp-std` to `core`.
These particular changes should be uncontroversial.
Where macros are used `::core` should be used to remove any ambiguity.
part of https://github.com/paritytech/polkadot-sdk/issues/2101
We currently use a bit of a hack in `.cargo/config` to make sure that
clippy isn't too annoying by specifying the list of lints.
There is now a stable way to define lints for a workspace. The only down
side is that every crate seems to have to opt into this so there's a
*few* files modified in this PR.
Dependencies:
- [x] PR that upgrades CI to use rust 1.74 is merged.
---------
Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
Using taplo, fixes all our broken and inconsistent toml formatting and
adds CI to keep them tidy.
If people want we can customise the format rules as described here
https://taplo.tamasfe.dev/configuration/formatter-options.html
@ggwpez, I suggest zepter is used only for checking features are
propagated, and leave formatting for taplo to avoid duplicate work and
conflicts.
TODO
- [x] Use `exclude = [...]` syntax in taplo file to ignore zombienet
tests instead of deleting the dir
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
# Description
- What does this PR do?
This PR adds function to get recorded keys from proof recorder instance
- Why are these changes needed?
This change is required to get the keys accessed by the trie backend
during the runtime execution. The keys are already tracked by proof
recorder, just aren't exposed publicly.
- How were these changes implemented and what do they affect?
The changes are implemented by adding a public function in proof
recorder that simply clones the `recorded_keys` field. It is pure
addition of function and AFAIK does not affect anything.
# Checklist
- [x] My PR includes a detailed description as outlined in the
"Description" section above
- [ ] My PR follows the [labeling requirements](CONTRIBUTING.md#Process)
of this project (at minimum one label for `T`
required)
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [ ] I have added tests that prove my fix is effective or that my
feature works (if applicable)
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
This PR provides the infrastructure for the pov-reclaim mechanism
discussed in #209. The goal is to provide the current proof size to the
runtime so it can be used to reclaim storage weight.
## New Host Function
- A new host function is provided
[here](https://github.com/skunert/polkadot-sdk/blob/5b317fda3be205f4136f10d4490387ccd4f9765d/cumulus/primitives/pov-reclaim/src/lib.rs#L23).
It returns the size of the current proof size to the runtime. If
recording is not enabled, it returns 0.
## Implementation Overview
- Implement option to enable proof recording during import in the
client. This is currently enabled for `polkadot-parachain`,
`parachain-template` and the cumulus test node.
- Make the proof recorder ready for no-std. It was previously only
enabled for std environments, but we need to record the proof size in
`validate_block` too.
- Provide a recorder implementation that only the records the size of
incoming nodes and does not store the nodes itself.
- Fix benchmarks that were broken by async backing changes
- Provide new externalities extension that is registered by default if
proof recording is enabled.
- I think we should discuss the naming, pov-reclaim was more intuitive
to me, but we could also go with clawback like in the issue.
## Impact of proof recording during import
With proof recording: 6.3058 Kelem/s
Without proof recording: 6.3427 Kelem/s
The measured impact on the importing performance is quite low on my
machine using the block import benchmark. With proof recording I am
seeing a performance hit of 0.585%.
---------
Co-authored-by: command-bot <>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
The `xcm` crate was renamed to `staging-xcm` to be able to publish it to
crates.io as someone as squatted `xcm`. The problem with this rename is
that the `TypeInfo` includes the crate name which ultimately lands in
the metadata. The metadata is consumed by downstream users like
`polkadot-js` or people building on top of `polkadot-js`. These people
are using the entire `path` to find the type in the type registry. Thus,
their code would break as the type path would now be [`staging_xcm`,
`VersionedXcm`] instead of [`xcm`, `VersionedXcm`]. This pull request
fixes this by renaming the path segment `staging_xcm` to `xcm`.
This requires: https://github.com/paritytech/scale-info/pull/197
---------
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
This PR updates:
- trie-db from 0.27.1 to 0.28.0
- trie-bench from 0.37.0 to 0.38.0 (deb-dependency)
While at it, also adapts the recorder to take into account the newly
added `TrieAccess::InlineValue`.
Needed by:
- https://github.com/paritytech/polkadot-sdk/pull/1153
@paritytech/subxt-team
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Fix std, runtime-benchmarks and try-runtime features
zepter lint propagate-feature --feature try-runtime --left-side-feature-missing=ignore --workspace --fix --feature-enables-dep="try-runtime:frame-try-runtime"
zepter lint propagate-feature --feature runtime-benchmarks --left-side-feature-missing=ignore --workspace --fix --feature-enables-dep="runtime-benchmarks:frame-benchmarking"
zepter lint propagate-feature --feature std --left-side-feature-missing=ignore --workspace --fix
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add propagate feature CI check
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Test CI by adding an error
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Use --locked
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add help msg
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Revert "Test CI by adding an error"
This reverts commit cf4ff6cc0632269b0a109e547686e5e3314b02de.
* Test CI by adding an error
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* No newline in help msg
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Revert "Test CI by adding an error"
This reverts commit 5daa06ada8e01f5bebafb9d1c76804dd79bc1006.
* Test CI by adding an error
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Revert "Test CI by adding an error"
This reverts commit ca15de5729507a564f140a10ec2e87b19516ec4c.
* Fix msg
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Revert back to master
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Re-do with Zepter v0.7.4
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update Zepter to 0.7.4
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Disable rococo try-runtime check
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* More review fixes
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
First, we increase the size to 10MiB for the local cache. Second, we give the node cache a bigger
max size than the value cache, see the changed comment on why.
In general this should ensure that we are able to cache the `:code` and not directly throw it out of
the cache because it is too big (which currently happens when the size of the runtime > 2MiB). In
the future this should be improved to ensure that certain values are not removed from the cache at
all, like `:code`.
* TrieRecorder: Start adding support for transactions
* Adds `transactions` functions and some test
* More tests
* Docs
* Ensure that we rollback failed transactions in the storage proof
* FMT
* Update primitives/trie/src/recorder.rs
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
* Review comments
* Update primitives/trie/src/recorder.rs
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* ".git/.scripts/commands/fmt/fmt.sh"
* For the holy clippy!
* Update primitives/trie/src/recorder.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
---------
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: command-bot <>
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Speed up storage iteration from within the runtime
* Move the cached iterator into an `Option`
* Use `RefCell` in no_std
* Simplify the code slightly
* Use `Option::replace`
* Update doc comment for `next_storage_key_slow`
* Rework storage iterators
* Make sure storage iteration is also accounted for when benchmarking
* Use `trie-db` from crates.io
* Appease clippy
* Bump `trie-bench` to 0.35.0
* Fix tests' compilation
* Update comment to clarify how `IterArgs::start_at` works
* Add extra tests
* Fix iterators on `Client` so that they behave as before
* Add extra `unwrap`s in tests
* More clippy fixes
* Come on clippy, give me a break already
* Rename `allow_missing` to `stop_on_incomplete_database`
* Add `#[inline]` to `with_recorder_and_cache`
* Use `with_recorder_and_cache` in `with_trie_db`; add doc comment
* Simplify code: use `with_trie_db` in `next_storage_key_from_root`
* Remove `expect`s in the benchmarking CLI
* Add extra doc comments
* Move `RawIter` before `TrieBackendEssence` (no code changes; just cut-paste)
* Remove a TODO in tests
* Update comment for `StorageIterator::was_complete`
* Update `trie-db` to 0.25.1
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
With `StateVersion::V1` values over a certain size are not inlined and being put into the backend
with their own hash. When accessing a value in the trie with a recorder, we check if the value is maybe already
recorded and thus, we can check the cache. To check if a value is already recorded, we use the key
of the value to differentiate them. The problem is when there are multiple tries, like multiple
child tries that all have different values under the same key. Before this pull request we didn't
have differentiated for which trie we already had recorded a (key, value) pair. This is now done by also taking
the storage root into account in the recorder to differentiate the different (key, value) pair in
the tries.
* Guard some invalid node for proof decoding.
* only forbid bitmap with no children.
* change format
* scale error.
* small test
Co-authored-by: parity-processbot <>
* Use `array-bytes` for All Array/Bytes/Hex Operations
Signed-off-by: Xavier Lau <xavier@inv.cafe>
* Reorder
* Self Review
* Format
* Fix Tests
* Bump `array-bytes`
* Optimize large test res
Signed-off-by: Xavier Lau <xavier@inv.cafe>
Co-authored-by: parity-processbot <>
Facundo Farall