* transactional: Wrap `pallet::calls` directly in storage layers
Before this pr we only wrapped `pallet::calls` into storage layers when executing the calls with
`dispatch`. This pr is solving that by wrapping each call function inside a storage layer.
* Teach `BasicExternalities` transactions support
* Fix crates
* FMT
* Fix benchmarking tests
* Use correct span
* Support old decl macros
* Fix test
* Apply suggestions from code review
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update frame/state-trie-migration/src/lib.rs
* Update frame/state-trie-migration/src/lib.rs
* Update frame/state-trie-migration/src/lib.rs
* Feedback
* Apply suggestions from code review
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Extract `NetworkKVProvider` trait in `sc-authority-discovery` and remove unnecessary dependency
* Extract `NetworkSyncForkRequest` trait in `sc-finality-grandpa`
* Relax requirements on `SyncOracle` trait, remove extra native methods from `NetworkService` that are already provided by trait impls
* Move `NetworkSigner` trait from `sc-authority-discovery` into `sc-network-common` and de-duplicate methods on `NetworkService`
* Move `NetworkKVProvider` trait from `sc-authority-discovery` into `sc-network-common` and de-duplicate methods on `NetworkService`
* Minimize `sc-authority-discovery` dependency on `sc-network`
* Move `NetworkSyncForkRequest` trait from `sc-finality-grandpa` to `sc-network-common` and de-duplicate methods in `NetworkService`
* Extract `NetworkStatusProvider` trait and de-duplicate methods on `NetworkService`
* Extract `NetworkPeers` trait and de-duplicate methods on `NetworkService`
* Extract `NetworkEventStream` trait and de-duplicate methods on `NetworkService`
* Move more methods from `NetworkService` into `NetworkPeers` trait
* Move `NetworkStateInfo` trait into `sc-network-common`
* Extract `NetworkNotification` trait and de-duplicate methods on `NetworkService`
* Extract `NetworkRequest` trait and de-duplicate methods on `NetworkService`
* Remove `NetworkService::local_peer_id()`, it is already provided by `NetworkStateInfo` impl
* Extract `NetworkTransaction` trait and de-duplicate methods on `NetworkService`
* Extract `NetworkBlock` trait and de-duplicate methods on `NetworkService`
* Remove dependencies on `NetworkService` from most of the methods of `sc-service`
* Address simple review comments
* client/beefy: don't accept vote for older rounds
* client/beefy: clean up and reorg the worker struct
* client/beefy: first step towards Full BEEFY
The first step from Lean->Full BEEFY is to have the worker
enforce uninterrupted line of BEEFY finalized mandatory blocks.
There is one mandatory block per session (the first block in the
session). As such, votes processing and votes generation now
enforces that all mandatory blocks are finalized in strict
monotonically increasing sequence and no block 'N' will be worked
on if there is any GRANDPA finalized but BEEFY non-final mandatory
block 'M', where 'M < N'.
Implementation details:
- Introduced 'VoterOracle' to separate the voting decisions logic,
and track new/pending sessions.
- New sessions get queued up with the worker operating either:
1. up-to-date - all mandatory blocks leading up to current GRANDPA
finalized: queue has ONE element, the 'current session' where
`mandatory_done == true`,
2. lagging behind GRANDPA: queue has [1, N] elements, where all
`mandatory_done == false`.
In this state, everytime a session gets its mandatory block
BEEFY finalized, the session is popped off the queue,
eventually getting to operating mode `1. up-to-date`.
- Votes get triaged and those that fall withing the `VoterOracle`
allowed window get processed, the others get dropped if stale,
or buffered for later processing (when they reach the window).
- Worker general code was also updated to fall in one of two roles:
1. react to external events and change internal 'state',
2. generate events/votes based on internal 'state'.
Signed-off-by: acatangiu <adrian@parity.io>
* client/beefy: sketch idea for block import and sync
Signed-off-by: acatangiu <adrian@parity.io>
* client/beefy: add BEEFY block import
* client/beefy: process justifications from block import
* client/beefy: add TODOs for sync protocol
* client/beefy: add more docs and comments
* client/beefy-rpc: fix RPC error
* client/beefy: verify justification validity on block import
* client/beefy: more tests
* client/beefy: small fixes
- first handle and note the self vote before gossiping it,
- don't shortcircuit on err when processing pending votes.
* client/beefy: remove invalid justifications at block import
* todo: beefy block import tests
* RFC: ideas for multiple justifications per block
* Revert "RFC: ideas for multiple justifications per block"
This reverts commit 8256fb07d3124db69daf252720b3c0208202624d.
* client/beefy: append justif to backend on block import
* client/beefy: groundwork for block import test
* client/beefy: groundwork2 for block import test
* client/beefy: groundwork3 for block import test
* client/beefy: add block import test
* client/beefy: add required trait bounds to block import builder
* remove client from beefy block import, backend gets the job done
Signed-off-by: acatangiu <adrian@parity.io>
* Expose allocation stats in `FreeingBumpHeapAllocator`
* Return allocation stats when calling into the runtime
* Bump `parity-scale-codec` to 3.1.3 (fork)
* Prevent double allocation of the payload when calling `sp_io::storage::get`
* Fix tests
* Remove unnecessary `mut`
* Enable the `bytes` feature for `parity-scale-codec` in `sp-runtime-interface`
* Update client/allocator/src/freeing_bump.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Bump `parity-scale-codec` to 3.1.3
* Fix some of the UI tests
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update UI test output for rust 1.62.1
* switch ci to staging image to check that everything works
* fix artifacts node-bench-regression-guard
* Imeplement `scale_info::TypeInfo` manually to silence aggressive rust warning
* Fix more clippy lints
* Make clippy happy by relying on auto-deref were possible
* Add tracking issue to the comments
* pin ci image
Co-authored-by: alvicsam <alvicsam@gmail.com>
* pallet-mmr: fix some typos
* pallet-mmr: make the MMR resilient to chain forks
* pallet-mmr: get hash for block that added node
* beefy-mmr: add debug logging
* add explanatory comment
* account for block offset of pallet activation
* add support for finding all nodes added by leaf
* minor improvements
* add helper to return all nodes added to mmr with a leaf append
* simplify leaf_node_index_to_leaf_index
summing the (shifted) differences in peak positions adds up to the (shifted) final position, so
don't need to fold over positions.
* dead fish: this also doesn't work
The idea was to keep a rolling window of `(parent_hash, pos)` leaf
entries in the offchain db, with the window matching the one
that provides `block_num -> block_hash` mappings in `frame_system`.
Once a leaf exits the window it would be "canonicalized" by switching
its offchain db key from `(parent_hash, pos)` to simple `pos`.
This doesn't work however because there's no way to get leaf contents
from offchain db while in runtime context.. so no way to get+clear+set
leaf to change its key in offchain db.
Ideas:
1. move the "canonicalization" logic to offchain worker
2. enhance IndexingApi with "offchain::move(old_key, new_key)"
This is weird, but correct, deterministic and safe AFAICT, so
it could be exposed to runtime.
* simplify rightmost_leaf_node_index_from_pos
* minor fix
* move leaf canonicalization to offchain worker
* move storage related code to storage.rs
* on offchain reads use canonic key for old leaves
* fix offchain worker write using canon key
* fix pallet-mmr tests
* add documentation and fix logging
* add offchain mmr canonicalization test
* test canon + generate + verify
* fix pallet-beefy-mmr tests
* implement review suggestions
* improve test
* pallet-mmr: add offchain pruning of forks
* pallet-mmr: improve offchain pruning
Instead of keeping pruning map as single blob in offchain db,
keep individual parent-hash lists with block-num identifier as part
of the offchain key.
Signed-off-by: acatangiu <adrian@parity.io>
* pallet-mmr: improve MMRStore<OffchainStorage>::get()
Do the math and retrieve node using correct (canon or non-canon)
offchain db key, instead of blindly looking in both canon and non-canon
offchain db locations for each node.
Still fallback on looking at both if for any reason it's not where
expected.
Signed-off-by: acatangiu <adrian@parity.io>
* pallet-mmr: storage: improve logs
* fix tests: correctly persist overlay
runtime indexing API works on overlay, whereas offchain context
bypasses overlay, so for loops > canon-window, canon would fail.
* pallet-mmr: fix numeric typo in test
* add comment around LeafData requirements
Signed-off-by: acatangiu <adrian@parity.io>
Co-authored-by: Robert Hambrock <roberthambrock@gmail.com>
* pallet-beefy: add Config::OnNewValidatorSet type
Add a hook to pallet-beefy for doing specific work when
BEEFY validator set changes.
For example, this can be used by pallet-beefy-mmr to cache
a lightweight MMR root over validators and make it available
to light clients.
* pallet-beefy-mmr: implement OnNewValidatorSet
Implement pallet-beefy::OnNewValidatorSet to be notified of BEEFY
validator set changes. Use the notifications to compute and cache
a light weight 'BEEFY authority set' which is an MMR root over
BEEFY validator set plus some extra info.
Previously, pallet-beefy-mmr was interogating pallet-beefy about
validator set id on every block to find out when it needs to recompute
the authority set.
By using the event-driven approach in this commit, we also save one
extra state interogation per block.
* pallet-beefy-mmr: add new authority_set() API
Expose current and next BEEFY authority sets through runtime API.
These can be directly used by light clients to avoid having them
compute them themselves based on BEEFY validator sets.
Signed-off-by: acatangiu <adrian@parity.io>
* rename BeefyMmr exposed runtime api
* Initial implementation of mms
* Some more attempts at `mms`
* Functioning `MMS` algorithm implementation.
Adding some tests too
* More tests and typos fixed.
* Adding fuzzer for `mms`
(but could not test it on Mac M1)
* Missing imports
* Fixing rustdoc
* More accurate implementation of `mms`
* Removing the fuzzer `mms` implementation
* Implementing `NposSolver` for `MMS`
had to add the `Clone` trait, maybe I could see if I can get rid of it.
* Fixing rust docs by adding () to resolve ambiguity
* Amending `unwrap` to `expect`
removing unneeded `Clone` trait
* Removing redundant `mms3.rs`
* Implementing `BalancingConfig` and rustdoc changes
* Implementing `weight` for `MMS`
* Implementing `weight` for `MMS`
* Fixing post merge
* Initial implementation of mms
* Some more attempts at `mms`
* Functioning `MMS` algorithm implementation.
Adding some tests too
* More tests and typos fixed.
* Adding fuzzer for `mms`
(but could not test it on Mac M1)
* Missing imports
* Fixing rustdoc
* More accurate implementation of `mms`
* Removing the fuzzer `mms` implementation
* Implementing `NposSolver` for `MMS`
had to add the `Clone` trait, maybe I could see if I can get rid of it.
* Amending `unwrap` to `expect`
removing unneeded `Clone` trait
* Fixing rust docs by adding () to resolve ambiguity
* Removing redundant `mms3.rs`
* Implementing `BalancingConfig` and rustdoc changes
* Implementing `weight` for `MMS`
* Implementing `weight` for `MMS`
* Fixing post merge
* Removing left over from rebase
* Fixing tests
* Removing unneeded import
* Removing unneeded functions
* Removing useless imports
Co-authored-by: kianenigma <kian@parity.io>
* grandpa: fix creation of justification ancestry
we would reject commits that have precommits targeting blocks lower than the
commit target. when there is an equivocation (or if it the commit is not
minimal) it is possible to have such precommits and we should assume that they
are the round base.
* grandpa: bump to 0.16.0
* grandpa: add test for justification with equivocation
* grandpa: fix failing test
* make pool roles optional
* undo lock file changes?
* add migration
* add the ability for pools to chill themselves
* boilerplate of tests
* somewhat stable, but I think I found another bug as well
* Fix it all
* Add more more sophisticated test + capture one more bug.
* Update frame/staking/src/lib.rs
* reduce the diff a little bit
* add some test for the slashing bug
* cleanup
* fix lock file?
* Fix
* fmt
* Update frame/nomination-pools/src/lib.rs
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update frame/nomination-pools/src/lib.rs
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update frame/nomination-pools/src/lib.rs
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update frame/nomination-pools/src/mock.rs
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix build
* fix some fishy tests..
* add one last integrity check for MinCreateBond
* remove bad assertion -- needs to be dealt with later
* nits
* fix tests and add benchmarks for chill
* remove stuff
* fix benchmarks
* cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_nomination_pools --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/nomination-pools/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* remove defensive
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Parity Bot <admin@parity.io>
* Implement MaxEncodedLen on pallet-beefy
* Return Result in intialize_authorities
* Update docs
* Log error when authorities list gets truncated
* Update frame/beefy/src/lib.rs
Co-authored-by: Adrian Catangiu <adrian@parity.io>
* cargo fmt
Co-authored-by: Adrian Catangiu <adrian@parity.io>
* trie: Optimize `keys` function
Instead of iterating the entire state and collecting all keys that match the given prefix, we can
directly use the optimized prefix iterator.
* Add a test
* fix a few more things with nomination pools
* add bench
* use weight fn
* cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_nomination_pools --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/nomination-pools/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* allow real root to also set roles
* Update frame/nomination-pools/src/lib.rs
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update frame/nomination-pools/src/lib.rs
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* move out of the closure
* fix a few more things
Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
benluelo