Fix situation where BEEFY initial validator set could not be determined.
If state is unavailable at BEEFY genesis block to get initial validator
set, get the info from header digests. For this, we need to walk back
the chain starting from BEEFY genesis looking for the BEEFY digest
announcing the active validator set for that respective session.
This commit fixes a silly bug where walking back the chain was stopped
when reaching BEEFY genesis block, which is incorrect when BEEFY genesis
is not session boundary block. When BEEFY genesis is set to some random
block within a session, we need to walk back to the start of the session
to see the validator set announcement.
Added regression test for this fix.
Fixes https://github.com/paritytech/polkadot-sdk/issues/1885
Signed-off-by: Adrian Catangiu <adrian@parity.io>
- Remove cached messages used for deduplication in `GossipValidator`
since they're already deduplicated in upper layer `NetworkGossip`.
- Add cache for "justified rounds" to quickly discard any further (even
if potentially different) justifications at the gossip level, once a
valid one (for a respective round) is submitted to the worker.
- Add short-circuit in worker `finalize()` method to not attempt to
finalize same block multiple times (for example when we get
justifications for same block from multiple components like
block-import, gossip or on-demand).
- Change a test which had A LOT of latency in syncing blocks for some
weird reason and would only run after ~150seconds. It now runs
instantly.
Fixes https://github.com/paritytech/polkadot-sdk/issues/1728
* Merged BEEFY primitives with generic signature and keyset commitment support from old pull to current code
* - Add bls-experimental feature to application-crypto and beefy primitives
- Fix remaining crypto -> ecdsa_crypto
- code build but not tests
* Make beefy primitive tests compile
* move bls related beefy primitives code and test behind bls-experimental flag
* Make BEEFY clients complies with BEEFY API depending on AuthorityId
* - Rename `BeefyAuthoritySet.root` → `BeefyAuthoritySet.keyset_commitment`.
- Remove apk proof keyset_commitment from `BeefyAuthoritySet`.
- Fix failing signed commitment and signature to witness test.
- Make client compatible with BeefyAPI generic on AuthorityId.
- `crypto` → `ecdsa_crypto` in BEEFY client and frame.
* Commit Cargo lock remove ark-serialize from BEEFY primitives
* Use Codec instead of Encode + Decode in primitives/consensus/beefy/src/lib.rs
Co-authored-by: Davide Galassi <davxy@datawok.net>
* - Make `BeefyApi` generic over Signature type.
- Make new `BeeyApi` functinos also generic over AuthorityId and Signature
* Unmake BeefyAPI generic over Signature. Recover Signature type from AuthId.
* - dont use hex or hex-literal use array-bytes instead in beefy primitives and bls crypto.
- CamelCase ECDSA and BLS everywhere.
* Move the definition of BEEFY key type from `primitives/beefy` to `crypto.rs` according to new convention.
* - Add bls377_generate_new to `sp-io` and `application_crypto::bls`.
- Add `bls-experimental` to `sp-io`
Does not compile because PassByCodec can not derive PassBy using customly implemented PassByIner.
* Implement PassBy for `bls::Public` manually
* fix Beefy `KEY_TYPE` in `frame/beefy` tests to come from `sp-core::key_types` enum
* specify both generic for `hex2array_unchecked` in `sp-core/bls.rs`
* Rename `crypto`→`ecdsa_crypto` in `primitives/consensus/beefy/src/test_utils.rs` docs
* remove commented-out code in `primitives/consensus/beefy/src/commitment.rs`
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Fix inconsistency in panic message in `primitives/io/src/lib.rs`
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Remove redundant feature activation in `primitives/io/Cargo.toml`
Co-authored-by: Davide Galassi <davxy@datawok.net>
* - make `w3f-bls` a dev-dependancy only for beefy primitives.
- clean up comments.
Co-authored-by: Davide Galassi <davxy@datawok.net>
* export BEEFY KEY_TYPE from primitives/consensus/beefy
make `frame/consensus/beefy` in dependent of sp_crypto_app
use consistent naming in the beefy primitive tests.
* - implement `BeefyAuthorityId` for `bls_crypto::AuthorityId`.
- implement `bls_verify_works` test for BEEFY `bls_crypto`.
* Remove BEEFY `ecdsa_n_bls_crypto` for now for later re-introduction
* Make commitment and witness BEEFY tests not use Keystore.
* put `bls_beefy_verify_works` test under `bls-experimental` flag.
* bump up Runtime `BeefyAPI` to version 3 due to introducing generic AuthorityId.
* reuse code and encapsulate w3f-bls backend in sp-core as most as possible
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Make comments in primities BEEFY `commitment.rs` and `witness.rs``tests convention conforming
* Use master dep versions
* Trivial change. Mostly to trigger CI
* Apply suggestions from code review
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Fix Cargo.toml
* Trigger CI with cumulus companion
* Trigger CI after polkadot companion change
---------
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Test was using too small a timeout of 250ms to verify equivocation is reported.
If the test machine is loaded with many async tasks running, the voters work
and subsequent gossiping does not fit within the 250ms window.
Do multiple tries to verify equivocation reporting with exponential timeouts,
most times it will be within 250ms, but the test will retry several times up
to 5seconds total before giving up.
Signed-off-by: Adrian Catangiu <adrian@parity.io>
BEEFY consensus can be restarted by resetting "genesisBlock" in
pallet-beefy, but we don't want to also reset authority set IDs so
that they are uniquely identified across the entire chain history
regardless of how many times BEEFY consensus has been reset/restarted.
This is why the client now also accepts initial authority_set_id != 0.
BEEFY client now detects pallet-beefy reset/reinit and errors-out and
asks for a restart.
BEEFY client persisted state should be discarded on client restarts
following pallet-beefy reset/reinit.
End result is BEEFY client/voter can now completely reinitialize using
"new" on-chain info following pallet-beefy reset/reinit, discarding old state.
Fixes#14203Fixes#14204
Signed-off-by: acatangiu <adrian@parity.io>
* substrate-test-runtime migrated to pure-frame based
* test block builder: helpers added
* simple renaming
* basic_authorship test adjusted
* block_building storage_proof test adjusted
* babe: tests: should_panic expected added
* babe: tests adjusted
ConsensusLog::NextEpochData is now added by pallet_babe as
pallet_babe::SameAuthoritiesForever trigger is used in runtime config.
* beefy: tests adjusted
test-substrate-runtime is now using frame::executive to finalize the
block. during finalization the digests stored during block execution are
checked against header digests:
https://github.com/paritytech/substrate/blob/91bb2d29ca905599098a5b35eaf24867c4fbd60a/frame/executive/src/lib.rs#L585-L591
It makes impossible to directly manipulate header's digets, w/o
depositing logs into system pallet storage `Digest<T: Config>`.
Instead of this dedicated extrinsic allowing to store logs items
(MmrRoot / AuthoritiesChange) is used.
* grandpa: tests adjusted
test-substrate-runtime is now using frame::executive to finalize the
block. during finalization the digest logs stored during block execution are
checked against header digest logs:
https://github.com/paritytech/substrate/blob/91bb2d29ca905599098a5b35eaf24867c4fbd60a/frame/executive/src/lib.rs#L585-L591
It makes impossible to directly manipulate header's digets, w/o
depositing logs into system pallet storage `Digest<T: Config>`.
Instead of this dedicated extrinsic allowing to store logs items
(ScheduledChange / ForcedChange and DigestItem::Other) is used.
* network:bitswap: test adjusted
The size of unchecked extrinsic was increased. The pattern used in test will
be placed at the end of scale-encoded buffer.
* runtime apis versions adjusted
* storage keys used in runtime adjusted
* wasm vs native tests removed
* rpc tests: adjusted
Transfer transaction processing was slightly improved, test was
adjusted.
* tests: sizes adjusted
Runtime extrinsic size was increased. Size of data read during block
execution was also increased due to usage of new pallets in runtime.
Sizes were adjusted in tests.
* cargo.lock update
cargo update -p substrate-test-runtime -p substrate-test-runtime-client
* warnings fixed
* builders cleanup: includes / std
* extrinsic validation cleanup
* txpool: benches performance fixed
* fmt
* spelling
* Apply suggestions from code review
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Apply code review suggestions
* Apply code review suggestions
* get rid of 1063 const
* renaming: UncheckedExtrinsic -> Extrinsic
* test-utils-runtime: further step to pure-frame
* basic-authorship: tests OK
* CheckSubstrateCall added + tests fixes
* test::Transfer call removed
* priority / propagate / no sudo+root-testing
* fixing warnings + format
* cleanup: build2/nonce + format
* final tests fixes
all tests are passing
* logs/comments removal
* should_not_accept_old_signatures test removed
* make txpool benches work again
* Cargo.lock reset
* format
* sudo hack removed
* txpool benches fix+cleanup
* .gitignore reverted
* rebase fixing + unsigned cleanup
* Cargo.toml/Cargo.lock cleanup
* force-debug feature removed
* mmr tests fixed
* make cargo-clippy happy
* network sync test uses unsigned extrinsic
* cleanup
* ".git/.scripts/commands/fmt/fmt.sh"
* push_storage_change signed call remove
* GenesisConfig cleanup
* fix
* fix
* GenesisConfig simplified
* storage_keys_works: reworked
* storage_keys_works: expected keys in vec
* storage keys list moved to substrate-test-runtime
* substrate-test: some sanity tests + GenesisConfigBuilder rework
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Apply suggestions from code review
* Review suggestions
* fix
* fix
* beefy: generate_blocks_and_sync block_num sync with actaul value
* Apply suggestions from code review
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Update test-utils/runtime/src/genesismap.rs
Co-authored-by: Davide Galassi <davxy@datawok.net>
* cargo update -p sc-rpc -p sc-transaction-pool
* Review suggestions
* fix
* doc added
* slot_duration adjusted for Babe::slot_duration
* small doc fixes
* array_bytes::hex used instead of hex
* tiny -> medium name fix
* Apply suggestions from code review
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* TransferData::try_from_unchecked_extrinsic -> try_from
* Update Cargo.lock
---------
Co-authored-by: parity-processbot <>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
* sc-consensus-beefy: improve beefy gossip validator
Old gossip validator was pretty dumb, being very permissive with
incoming votes - only condition it had was to be newer than best
finalized.
New filter conditions:
- voter rounds are initialized (discarding votes until voter is
actually active),
- only votes for current active set id are accepted,
- only votes for rounds in the current voting session are accepted,
- only votes for GRANDPA finalized blocks are accepted,
- when BEEFY voter reaches mandatory round, only votes for said
mandatory round are accepted.
New validator uses the VoterOracle to easily implement above conditions
and only allow through votes that are immediately useful to the voter.
After every GRANDPA or BEEFY finality, the gossip validator filter is
updated.
* sc-consensus-beefy: remove votes enqueueing
Since gossip validator will simply disallow votes for future rounds,
and only allow votes that the voter can immediately process, there
is no need for the voter to enqueue votes.
It will see these "future" votes later in rebroadcasts, when voter
will also be able to process them. Only at that point does gossip
accept and consume them.
* sc-consensus-beefy: refactor persistent state
Move best-beefy and best-grandpa into VoterOracle instead
of passing them around as params.
VoterOracle ultimately needs to know best-beefy and/or best-grandpa
for most of its functions.
* sc-consensus-beefy: further restrict gossip validator
Assuming mandatory done in current session:
Instead of allowing votes for any round in the current session, only
accept votes for rounds equal or better than best BEEFY finalized.
* sc-consensus-beefy: add a couple of comments
* sc-consensus-beefy: fix tests involving multiple tasks
Finalize blocks one a time in tests where we want gossip to happen
in a certain round. Otherwise, some tasks may be left behind in
terms of gossip round numbers because once "scheduled" a task will
greedily process as much as possible.
This change should be in line with the real-world scenario where
voters run "in parallel" across nodes, the only points of
synchronization being the finality notifications.
* sc-consensus-beefy: address review comments
---------
Signed-off-by: acatangiu <adrian@parity.io>
`futures_util::pending!()` macro only yields to the event loop once,
resulting in many calls to the `OnDemandJustificationsEngine::next()`
made by the tokio reactor even though the on-demand-engine is idle.
Replace it with the correct `futures::future::pending::<()>()` function
which forever yields to the event loop, which is what we want when
the engine is idle.
Signed-off-by: Adrian Catangiu <adrian@parity.io>
* Move import queue out of `sc-network`
Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.
This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.
* Move stuff to SyncingEngine
* Move `ChainSync` instanation to `SyncingEngine`
Some of the tests have to be rewritten
* Move peer hashmap to `SyncingEngine`
* Let `SyncingEngine` to implement `ChainSyncInterface`
* Introduce `SyncStatusProvider`
* Move `sync_peer_(connected|disconnected)` to `SyncingEngine`
* Implement `SyncEventStream`
Remove `SyncConnected`/`SyncDisconnected` events from
`NetworkEvenStream` and provide those events through
`ChainSyncInterface` instead.
Modify BEEFY/GRANDPA/transactions protocol and `NetworkGossip` to take
`SyncEventStream` object which they listen to for incoming sync peer
events.
* Introduce `ChainSyncInterface`
This interface provides a set of miscellaneous functions that other
subsystems can use to query, for example, the syncing status.
* Move event stream polling to `SyncingEngine`
Subscribe to `NetworkStreamEvent` and poll the incoming notifications
and substream events from `SyncingEngine`.
The code needs refactoring.
* Make `SyncingEngine` into an asynchronous runner
This commits removes the last hard dependency of syncing from
`sc-network` meaning the protocol now lives completely outside of
`sc-network`, ignoring the hardcoded peerset entry which will be
addressed in the future.
Code needs a lot of refactoring.
* Fix warnings
* Code refactoring
* Use `SyncingService` for BEEFY
* Use `SyncingService` for GRANDPA
* Remove call delegation from `NetworkService`
* Remove `ChainSyncService`
* Remove `ChainSync` service tests
They were written for the sole purpose of verifying that `NetworWorker`
continues to function while the calls are being dispatched to
`ChainSync`.
* Refactor code
* Refactor code
* Update client/finality-grandpa/src/communication/tests.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Fix warnings
* Apply review comments
* Fix docs
* Fix test
* cargo-fmt
* Update client/network/sync/src/engine.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Update client/network/sync/src/engine.rs
Co-authored-by: Anton <anton.kalyaev@gmail.com>
* Add missing docs
* Refactor code
---------
Co-authored-by: Anton <anton.kalyaev@gmail.com>
Adrian Catangiu