Using taplo, fixes all our broken and inconsistent toml formatting and
adds CI to keep them tidy.
If people want we can customise the format rules as described here
https://taplo.tamasfe.dev/configuration/formatter-options.html
@ggwpez, I suggest zepter is used only for checking features are
propagated, and leave formatting for taplo to avoid duplicate work and
conflicts.
TODO
- [x] Use `exclude = [...]` syntax in taplo file to ignore zombienet
tests instead of deleting the dir
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Adds a config file that allows to run `zepter` without any arguments in
the workspace to address all issues.
A secondary workflow for the CI is provided as `zepter run check`. Both
the formatting and linting are now in one check for efficiancy.
The latest version also detects some more things that `featalign` was
already showing.
Error message [in the
CI](https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/3916205)
now looks like this:
```pre
...
crate 'test-parachains' (/Users/vados/Documents/work/polkadot-sdk/polkadot/parachain/test-parachains/Cargo.toml)
feature 'std'
must propagate to:
parity-scale-codec
Found 55 issues (run with --fix to fix).
Error: Command 'lint propagate-feature' failed with exit code 1
Polkadot-SDK uses the Zepter CLI to detect abnormalities in the feature configuration.
It looks like one more more checks failed; please check the console output. You can try to automatically address them by running `zepter`.
Otherwise please ask directly in the Merge Request, GitHub Discussions or on Matrix Chat, thank you.
For more information, see:
- https://github.com/paritytech/polkadot-sdk/issues/1831
- https://github.com/ggwpez/zepter
```
TODO:
- [x] Check that CI fails correctly
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
The `xcm` crate was renamed to `staging-xcm` to be able to publish it to
crates.io as someone as squatted `xcm`. The problem with this rename is
that the `TypeInfo` includes the crate name which ultimately lands in
the metadata. The metadata is consumed by downstream users like
`polkadot-js` or people building on top of `polkadot-js`. These people
are using the entire `path` to find the type in the type registry. Thus,
their code would break as the type path would now be [`staging_xcm`,
`VersionedXcm`] instead of [`xcm`, `VersionedXcm`]. This pull request
fixes this by renaming the path segment `staging_xcm` to `xcm`.
This requires: https://github.com/paritytech/scale-info/pull/197
---------
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
This PR updates:
- trie-db from 0.27.1 to 0.28.0
- trie-bench from 0.37.0 to 0.38.0 (deb-dependency)
While at it, also adapts the recorder to take into account the newly
added `TrieAccess::InlineValue`.
Needed by:
- https://github.com/paritytech/polkadot-sdk/pull/1153
@paritytech/subxt-team
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Fix std, runtime-benchmarks and try-runtime features
zepter lint propagate-feature --feature try-runtime --left-side-feature-missing=ignore --workspace --fix --feature-enables-dep="try-runtime:frame-try-runtime"
zepter lint propagate-feature --feature runtime-benchmarks --left-side-feature-missing=ignore --workspace --fix --feature-enables-dep="runtime-benchmarks:frame-benchmarking"
zepter lint propagate-feature --feature std --left-side-feature-missing=ignore --workspace --fix
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add propagate feature CI check
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Test CI by adding an error
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Use --locked
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add help msg
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Revert "Test CI by adding an error"
This reverts commit cf4ff6cc0632269b0a109e547686e5e3314b02de.
* Test CI by adding an error
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* No newline in help msg
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Revert "Test CI by adding an error"
This reverts commit 5daa06ada8e01f5bebafb9d1c76804dd79bc1006.
* Test CI by adding an error
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Revert "Test CI by adding an error"
This reverts commit ca15de5729507a564f140a10ec2e87b19516ec4c.
* Fix msg
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Revert back to master
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Re-do with Zepter v0.7.4
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update Zepter to 0.7.4
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Disable rococo try-runtime check
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* More review fixes
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
* substrate-test-runtime migrated to pure-frame based
* test block builder: helpers added
* simple renaming
* basic_authorship test adjusted
* block_building storage_proof test adjusted
* babe: tests: should_panic expected added
* babe: tests adjusted
ConsensusLog::NextEpochData is now added by pallet_babe as
pallet_babe::SameAuthoritiesForever trigger is used in runtime config.
* beefy: tests adjusted
test-substrate-runtime is now using frame::executive to finalize the
block. during finalization the digests stored during block execution are
checked against header digests:
https://github.com/paritytech/substrate/blob/91bb2d29ca905599098a5b35eaf24867c4fbd60a/frame/executive/src/lib.rs#L585-L591
It makes impossible to directly manipulate header's digets, w/o
depositing logs into system pallet storage `Digest<T: Config>`.
Instead of this dedicated extrinsic allowing to store logs items
(MmrRoot / AuthoritiesChange) is used.
* grandpa: tests adjusted
test-substrate-runtime is now using frame::executive to finalize the
block. during finalization the digest logs stored during block execution are
checked against header digest logs:
https://github.com/paritytech/substrate/blob/91bb2d29ca905599098a5b35eaf24867c4fbd60a/frame/executive/src/lib.rs#L585-L591
It makes impossible to directly manipulate header's digets, w/o
depositing logs into system pallet storage `Digest<T: Config>`.
Instead of this dedicated extrinsic allowing to store logs items
(ScheduledChange / ForcedChange and DigestItem::Other) is used.
* network:bitswap: test adjusted
The size of unchecked extrinsic was increased. The pattern used in test will
be placed at the end of scale-encoded buffer.
* runtime apis versions adjusted
* storage keys used in runtime adjusted
* wasm vs native tests removed
* rpc tests: adjusted
Transfer transaction processing was slightly improved, test was
adjusted.
* tests: sizes adjusted
Runtime extrinsic size was increased. Size of data read during block
execution was also increased due to usage of new pallets in runtime.
Sizes were adjusted in tests.
* cargo.lock update
cargo update -p substrate-test-runtime -p substrate-test-runtime-client
* warnings fixed
* builders cleanup: includes / std
* extrinsic validation cleanup
* txpool: benches performance fixed
* fmt
* spelling
* Apply suggestions from code review
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Apply code review suggestions
* Apply code review suggestions
* get rid of 1063 const
* renaming: UncheckedExtrinsic -> Extrinsic
* test-utils-runtime: further step to pure-frame
* basic-authorship: tests OK
* CheckSubstrateCall added + tests fixes
* test::Transfer call removed
* priority / propagate / no sudo+root-testing
* fixing warnings + format
* cleanup: build2/nonce + format
* final tests fixes
all tests are passing
* logs/comments removal
* should_not_accept_old_signatures test removed
* make txpool benches work again
* Cargo.lock reset
* format
* sudo hack removed
* txpool benches fix+cleanup
* .gitignore reverted
* rebase fixing + unsigned cleanup
* Cargo.toml/Cargo.lock cleanup
* force-debug feature removed
* mmr tests fixed
* make cargo-clippy happy
* network sync test uses unsigned extrinsic
* cleanup
* ".git/.scripts/commands/fmt/fmt.sh"
* push_storage_change signed call remove
* GenesisConfig cleanup
* fix
* fix
* GenesisConfig simplified
* storage_keys_works: reworked
* storage_keys_works: expected keys in vec
* storage keys list moved to substrate-test-runtime
* substrate-test: some sanity tests + GenesisConfigBuilder rework
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Apply suggestions from code review
* Review suggestions
* fix
* fix
* beefy: generate_blocks_and_sync block_num sync with actaul value
* Apply suggestions from code review
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Update test-utils/runtime/src/genesismap.rs
Co-authored-by: Davide Galassi <davxy@datawok.net>
* cargo update -p sc-rpc -p sc-transaction-pool
* Review suggestions
* fix
* doc added
* slot_duration adjusted for Babe::slot_duration
* small doc fixes
* array_bytes::hex used instead of hex
* tiny -> medium name fix
* Apply suggestions from code review
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* TransferData::try_from_unchecked_extrinsic -> try_from
* Update Cargo.lock
---------
Co-authored-by: parity-processbot <>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* Speed up storage iteration from within the runtime
* Move the cached iterator into an `Option`
* Use `RefCell` in no_std
* Simplify the code slightly
* Use `Option::replace`
* Update doc comment for `next_storage_key_slow`
* Rework storage iterators
* Make sure storage iteration is also accounted for when benchmarking
* Use `trie-db` from crates.io
* Appease clippy
* Bump `trie-bench` to 0.35.0
* Fix tests' compilation
* Update comment to clarify how `IterArgs::start_at` works
* Add extra tests
* Fix iterators on `Client` so that they behave as before
* Add extra `unwrap`s in tests
* More clippy fixes
* Come on clippy, give me a break already
* Rename `allow_missing` to `stop_on_incomplete_database`
* Add `#[inline]` to `with_recorder_and_cache`
* Use `with_recorder_and_cache` in `with_trie_db`; add doc comment
* Simplify code: use `with_trie_db` in `next_storage_key_from_root`
* Remove `expect`s in the benchmarking CLI
* Add extra doc comments
* Move `RawIter` before `TrieBackendEssence` (no code changes; just cut-paste)
* Remove a TODO in tests
* Update comment for `StorageIterator::was_complete`
* Update `trie-db` to 0.25.1
* move BeefyMmrApi to pallet-beefy-mmr
* fix test_utils use pallet-beefy-mmr BeefyMmrApi
* Move beefy-merkle-tree to utils and Rename to Merkle-tree
* fix fmt and test
* Update merkle-tree to binary-merkle-tree and Remove Keccak256 mod from merkle-tree
* change merkle-tree name to binary-merkle-tree
* mirr fix
* trie state cache
* Also cache missing access on read.
* fix comp
* bis
* fix
* use has_lru
* remove local storage cache on size 0.
* No cache.
* local cache only
* trie cache and local cache
* storage cache (with local)
* trie cache no local cache
* Add state access benchmark
* Remove warnings etc
* Add trie cache benchmark
* No extra "clone" required
* Change benchmark to use multiple blocks
* Use patches
* Integrate shitty implementation
* More stuff
* Revert "Merge branch 'master' into trie_state_cache"
This reverts commit 947cd8e6d43fced10e21b76d5b92ffa57b57c318, reversing
changes made to 29ff036463.
* Improve benchmark
* Adapt to latest changes
* Adapt to changes in trie
* Add a test that uses iterator
* Start fixing it
* Remove obsolete file
* Make it compile
* Start rewriting the trie node cache
* More work on the cache
* More docs and code etc
* Make data cache an optional
* Tests
* Remove debug stuff
* Recorder
* Some docs and a simple test for the recorder
* Compile fixes
* Make it compile
* More fixes
* More fixes
* Fix fix fix
* Make sure cache and recorder work together for basic stuff
* Test that data caching and recording works
* Test `TrieDBMut` with caching
* Try something
* Fixes, fixes, fixes
* Forward the recorder
* Make it compile
* Use recorder in more places
* Switch to new `with_optional_recorder` fn
* Refactor and cleanups
* Move `ProvingBackend` tests
* Simplify
* Move over all functionality to the essence
* Fix compilation
* Implement estimate encoded size for StorageProof
* Start using the `cache` everywhere
* Use the cache everywhere
* Fix compilation
* Fix tests
* Adds `TrieBackendBuilder` and enhances the tests
* Ensure that recorder drain checks that values are found as expected
* Switch over to `TrieBackendBuilder`
* Start fixing the problem with child tries and recording
* Fix recording of child tries
* Make it compile
* Overwrite `storage_hash` in `TrieBackend`
* Add `storage_cache` to the benchmarks
* Fix `no_std` build
* Speed up cache lookup
* Extend the state access benchmark to also hash a runtime
* Fix build
* Fix compilation
* Rewrite value cache
* Add lru cache
* Ensure that the cache lru works
* Value cache should not be optional
* Add support for keeping the shared node cache in its bounds
* Make the cache configurable
* Check that the cache respects the bounds
* Adds a new test
* Fixes
* Docs and some renamings
* More docs
* Start using the new recorder
* Fix more code
* Take `self` argument
* Remove warnings
* Fix benchmark
* Fix accounting
* Rip off the state cache
* Start fixing fallout after removing the state cache
* Make it compile after trie changes
* Fix test
* Add some logging
* Some docs
* Some fixups and clean ups
* Fix benchmark
* Remove unneeded file
* Use git for patching
* Make CI happy
* Update primitives/trie/Cargo.toml
Co-authored-by: Koute <koute@users.noreply.github.com>
* Update primitives/state-machine/src/trie_backend.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Introduce new `AsTrieBackend` trait
* Make the LocalTrieCache not clonable
* Make it work in no_std and add docs
* Remove duplicate dependency
* Switch to ahash for better performance
* Speedup value cache merge
* Output errors on underflow
* Ensure the internal LRU map doesn't grow too much
* Use const fn to calculate the value cache element size
* Remove cache configuration
* Fix
* Clear the cache in between for more testing
* Try to come up with a failing test case
* Make the test fail
* Fix the child trie recording
* Make everything compile after the changes to trie
* Adapt to latest trie-db changes
* Fix on stable
* Update primitives/trie/src/cache.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Fix wrong merge
* Docs
* Fix warnings
* Cargo.lock
* Bump pin-project
* Fix warnings
* Switch to released crate version
* More fixes
* Make clippy and rustdocs happy
* More clippy
* Print error when using deprecated `--state-cache-size`
* 🤦
* Fixes
* Fix storage_hash linkings
* Update client/rpc/src/dev/mod.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Review feedback
* encode bound
* Rework the shared value cache
Instead of using a `u64` to represent the key we now use an `Arc<[u8]>`. This arc is also stored in
some extra `HashSet`. We store the key are in an extra `HashSet` to de-duplicate the keys accross
different storage roots. When the latest key usage is dropped in the lru, we also remove the key
from the `HashSet`.
* Improve of the cache by merging the old and new solution
* FMT
* Please stop coming back all the time :crying:
* Update primitives/trie/src/cache/shared_cache.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Fixes
* Make clippy happy
* Ensure we don't deadlock
* Only use one lock to simplify the code
* Do not depend on `Hasher`
* Fix tests
* FMT
* Clippy 🤦
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* pallet-beefy: add Config::OnNewValidatorSet type
Add a hook to pallet-beefy for doing specific work when
BEEFY validator set changes.
For example, this can be used by pallet-beefy-mmr to cache
a lightweight MMR root over validators and make it available
to light clients.
* pallet-beefy-mmr: implement OnNewValidatorSet
Implement pallet-beefy::OnNewValidatorSet to be notified of BEEFY
validator set changes. Use the notifications to compute and cache
a light weight 'BEEFY authority set' which is an MMR root over
BEEFY validator set plus some extra info.
Previously, pallet-beefy-mmr was interogating pallet-beefy about
validator set id on every block to find out when it needs to recompute
the authority set.
By using the event-driven approach in this commit, we also save one
extra state interogation per block.
* pallet-beefy-mmr: add new authority_set() API
Expose current and next BEEFY authority sets through runtime API.
These can be directly used by light clients to avoid having them
compute them themselves based on BEEFY validator sets.
Signed-off-by: acatangiu <adrian@parity.io>
* rename BeefyMmr exposed runtime api
Simplified BEEFY worker logic based on the invariant that GRANDPA
will always finalize 1st block of each new session, meaning BEEFY
worker is guaranteed to receive finality notification for the
BEEFY mandatory blocks.
Under these conditions the current design is as follows:
- session changes are detected based on BEEFY Digest present in
BEEFY mandatory blocks,
- on each new session new `Rounds` of voting is created, with old
rounds being dropped (for gossip rounds, last 3 are still alive
so votes are still being gossiped),
- after processing finality for a block, the worker votes if
a new voting target has become available as a result of said
block finality processing,
- incoming votes as well as self-created votes are processed
and signed commitments are created for completed BEEFY voting
rounds,
- the worker votes if a new voting target becomes available
once a round successfully completes.
On worker startup, the current validator set is retrieved from
the BEEFY pallet. If it is the genesis validator set, worker
starts voting right away considering Block #1 as session start.
Otherwise (not genesis), the worker will vote starting with
mandatory block of the next session.
Later on when we add the BEEFY initial-sync (catch-up) logic,
the worker will sync all past mandatory blocks Signed Commitments
and will be able to start voting right away.
BEEFY mandatory block is the block with header containing the BEEFY
`AuthoritiesChange` Digest, this block is guaranteed to be finalized
by GRANDPA.
This session-boundary block is signed by the ending-session's
validator set. Next blocks will be signed by the new session's
validator set. This behavior is consistent with what GRANDPA does
as well.
Also drop the limit N on active gossip rounds. In an adversarial
network, a bad actor could create and gossip N invalid votes with
round numbers larger than the current correct round number. This
would lead to votes for correct rounds to no longer be gossiped.
Add unit-tests for all components, including full voter consensus
tests.
Signed-off-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: David Salami <Wizdave97>
* Upgraded dependencies
* Adapting code to scale v3
* Empty commit to trigger CI
* Triggering CI
* Fixing UI test
* Remove superfluous dev-dep added by #9228
* Cryout for CI
Liam Aharon