Integrate BEEFY (#9833)

* Initial project setup and skeleton (#4)

* initial project setup for beefy gadget client

* update editorconfig

* update gitignore

* add initial skeleton for beefy gadget worker

* add skeleton for gossip processing

* add app crypto

* move around some code

* add basic flow for voting

* add logic for picking blocks to sign

* add rustfmt config

* add example node with beefy gadget

* use u32::next_power_of_two

* make maximum periodicity configurable

* add copyright header

* rename max_periodicity to min_interval

* CI stuff (#5)

* CI stuff.

* Fix workspace.

* cargo fmt --all

* Add license for beefy-gadget

* One toolchain to rule them all.

* Clippy.

* Fix clippy.

* Clippy in the runtime.

* Fix clippy grumbles.

* cargo fmt --all

* Primitives & Light Client examples (#8)

* Primitives.

* Docs.

* Document primitives.

* Simple tests.

* Light client examples.

* Fix stuff.

* cargo fmt --all

* Add a bunch of tests for imports.

* Add more examples.

* cargo fmt --all

* Fix clippy.

* cargo fmt --all

* Apply suggestions from code review

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Add GRANDPA / FG clarifications.

* Fix min number of signatures.

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update to substrate master (#22)

* update to substrate master

* update dependencies

* fix clippy issues

Co-authored-by: Tomasz Drwięga <tomasz@parity.io>

* Add beefy pallet (#25)

* move beefy application crypto to primitives

* make primitives compile under no_std

* add beefy pallet that maintains authority set

* add beefy pallet to node example runtime

* tabify node-example cargo.toml files

* use double quotes in Cargo.toml files

* add missing hex-literal dependency

* add runtime api to fetch BEEFY authorities

* fix clippy warnings

* rename beefy-pallet to pallet-beefy

* sort dependencies in node-example/runtime/Cargo.toml

* Signed commitments rpc pubsub (#26)

* move beefy application crypto to primitives

* make primitives compile under no_std

* add beefy pallet that maintains authority set

* add beefy pallet to node example runtime

* tabify node-example cargo.toml files

* use double quotes in Cargo.toml files

* add missing hex-literal dependency

* add runtime api to fetch BEEFY authorities

* fix clippy warnings

* gadget: use commitment and signedcommitment

* gadget: send notifications for signed commitments

* gadget: add rpc pubsub for signed commitments

* node-example: enable beefy rpc

* gadget: fix clippy warnings

* rename beefy-pallet to pallet-beefy

* sort dependencies in node-example/runtime/Cargo.toml

* gadget: add documentation on SignedCommitment rpc wrapper type

* gadget: add todos about dummy beefy commitments

* gadget: remove redundant closure

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Integrate MMR and deposit root into the digest. (#24)

* Add basic MMR.

* Deposit digest item.

* cargo fmt --all

* Merge with primitives.

* cargo fmt --all

* Fix extra spaces.

* cargo fmt --all

* Switch branch.

* remove stray whitespace

* update to latest td-mmr commit

* fix clippy error

Co-authored-by: André Silva <andrerfosilva@gmail.com>

* use new mmr root as commitment payload (#27)

* use new mmr root as commitment payload

* fix mmr root codec index

* warn on MMR root digest not found

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* add type alias for MMR root hash

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Bump serde_json from 1.0.59 to 1.0.60 (#28)

* Update to latest substrate. (#32)

* Update to latest substrate.

* Fix tests.

* cargo fmt --all

* Switch to master.

* Bump serde from 1.0.117 to 1.0.118 (#29)

* Bump serde from 1.0.117 to 1.0.118

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.117 to 1.0.118.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.117...v1.0.118)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Bump arc-swap.

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomasz@parity.io>

* Remove transition flag (#35)

* Get rid of is_set_transition_flag

* Fix tests.

* cargo fmt --all

* Bump futures from 0.3.9 to 0.3.12 (#50)

* Bump log from 0.4.11 to 0.4.13 (#52)

* Bump Substrate and Deps (#57)

* Update README (#58)

* Update README

* Apply suggestions from code review

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* address review comments

* missed a typo

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Add validator set to the pallet. (#65)

* Bump Substrate and Deps (#71)

* Bump Substrate and Deps

* pin serde and syn

* bump Substrate again for '__Nonexhaustive' fix

* add cargo deny ignore

* Beefy pallet test (#74)

* setup mock

* test session change

* silence beefy

* clippy still

* no change - no log

* clippy again

* Apply suggestions from code review

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* code review changes, added additional test

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Beefy node cleanup (#75)

* bump serde

* bump substrate, scale-codec 2.0.0

* we need a proper beefy node

* rename primitives as well

* Sort members.

Co-authored-by: Tomasz Drwięga <tomasz@parity.io>

* Migrate beefy-pallet to FRAMEv2 (#76)

* migrate beefy-pallet to FRAMEv2

* Code review

Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>

Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>

* Run BEEFY worker as non-validator (#77)

* run BEEFY worker as non-validator

* don't check for roloe.is_authority

* change enum type name

* Bump Substrate and Deps (#79)

* Add BEEFY gadget as extra peer set (#80)

* Add BEEFY gadget as extra peer set

* use BEEFY protocol

* Add ValidatorSetId to BEEFY digest (#85)

* add ValidatorSetId to BEEFY digest

* apply review changes

* Bump Substrate and Deps (#91)

* Bump Substrate and Deps

* Bump Substrate again in order to include a hot-fix

* redo again

* use CryptoStore issue

* cargo fmt

* Bump serde_json from 1.0.63 to 1.0.64 (#93)

* Track BEEFY validator set (#94)

* Track BEEFY validator set

* Add validator_set_id to BeefyWorker

* Make validattor_set_id optional

* Ad 92 (#97)

* sign_commitment()

* Error handling todo

* Add error type (#99)

* Add error type

* Address review

* Extract worker and round logic (#104)

* Bump serde from 1.0.123 to 1.0.124 (#106)

* Rework BeefyAPI (#110)

* Initialize BeefyWorker with current validator set (#111)

* Update toolchain (#115)

* Use nightly toolchain

* dongradde to latest clippy stable

* GH workflow trail and error

* next try

* use stable for clippy

* update wasm builder

* yet another try

* fun with CI

* no env var

* and one more

* allow from_over_into bco contruct_runtime

* back to start

* well ...

* full circle

* old version was still used

* Bump Substrate and Deps (#117)

* Bump Substrate and Deps

* cargo fmt should enforce uniform imports

* merge some imports

* Delayed BEEFY worker initialization (#121)

* lifecycle state

* add Client convenience trait

* rework trait identifiers

* WIP

* rework BeefyWorker::new() signature

* Delayed BEEFY gadget initialization

* address review

* Bump substrate. (#123)

* Bump substrate.

* Fix tests.

* Lower log-level for a missing validator set (#124)

* lower log-level for a missing validator set

* move best_finalized_block initialization

* Setup Prometheus metrics (#125)

* setup Prometheus metrics

* expose validator set id

* cargo fmt

* Update beefy-gadget/src/lib.rs

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* add vote messages gossiped metric

* track authorities change, before checking for MMR root digest

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Make Client convenience trait public (#126)

* Bump serde from 1.0.124 to 1.0.125 (#131)

* Reset rounds on new validator set. (#133)

* Re-set rounds on new validator set.

* Fix docs.

* Bump Substrate and Deps (#134)

* beefy: authority set changes fixes (#139)

* node: fix grandpa peers set config

* gadget: update best finalized number only when finalized with beefy

* gadget: process authorities changes regardless of vote status

* gadget: remove superfluous signature type (#140)

* node: fix grandpa peers set config

* gadget: update best finalized number only when finalized with beefy

* gadget: process authorities changes regardless of vote status

* gadget: remove superfluous signature type

Co-authored-by: Tomasz Drwięga <tomasz@parity.io>

* gadget: reduce gossip spam (#141)

* node: fix grandpa peers set config

* gadget: update best finalized number only when finalized with beefy

* gadget: process authorities changes regardless of vote status

* gadget: remove superfluous signature type

* gadget: only gossip last 5 rounds

* gadget: note round to gossip validator before gossiping message

* gadget: fix clippy warnings

* gadget: update docs

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: adoerr <0xad@gmx.net>

* gadget: verify SignedCommitment message signature (#142)

* gadget: verify SignedCommitment message signature

* gadget: log messages with bad sigs

* gadget: move todo comment

* Bump futures from 0.3.13 to 0.3.14 (#145)

* Milestone 1 (#144)

* use best_finalized, prevent race

* make best_finalized_block an Option, should_vote_on bails on None

* Bump futures from 0.3.13 to 0.3.14

* Revert futures bump

* Revert "Revert futures bump"

This reverts commit a1b5e7e9bac526f2897ebfdfee7f02dd29a13ac5.

* Revert "Bump futures from 0.3.13 to 0.3.14"

This reverts commit a4e508b118ad2c4b52909d24143c284073961458.

* debug msg if the bail voting

* validator_set()

* local_id()

* get rid of worker state

* Apply review suggestions

* fix should_vote_on()

* Extract BeefyGossipValidator (#147)

* Extract BeefyGossipValidator

* Apply review suggestions

* Add block_delta parameter to start_beefy_gadget (#151)

* Add block_delta parameter

* rename to min_block_delta

* Add additional metrics (#152)

* Add additional metrics

* add skipped session metric

* add some comment for temp metric

* don't log under info for every concluded round (#156)

* don't log error on missing validator keys (#157)

* don't log error on missing validator keys

* remove unused import

* Fix validator set change handling (#158)

* reduce some logs from debug to trace

* fix validator set changes handling

* rename validator module to gossip

* run rustfmt

* Fix should_vote_on() (#160)

* Fix should_vote_on()

* by the textbook

* fix the algorithm

* Apply review suggestions

* don't use NumberFor in vote_target

Co-authored-by: André Silva <andrerfosilva@gmail.com>

* Make KeyStore optional (#173)

* Use builder pattern for NonDefaultSetConfig (#178)

Co-authored-by: adoerr <0xad@gmx.net>

* Append SignedCommitment to block justifications (#177)

* Append SignedCommitment

* add BeefyParams

* add WorkerParams

* use warn

* versioned variant for SignedCommitment

* Bump serde from 1.0.125 to 1.0.126 (#184)

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.125 to 1.0.126.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.125...v1.0.126)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump strum from 0.20.0 to 0.21.0 (#195)

* Bump strum from 0.20.0 to 0.21.0

Bumps [strum](https://github.com/Peternator7/strum) from 0.20.0 to 0.21.0.
- [Release notes](https://github.com/Peternator7/strum/releases)
- [Changelog](https://github.com/Peternator7/strum/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Peternator7/strum/commits)

---
updated-dependencies:
- dependency-name: strum
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* use dervie feature for strum; clippy and deny housekeeping

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: adoerr <0xad@gmx.net>

* Make concluded round an info log (#200)

* Remove external crypto trait bounds (#207)

* BeefyKeystore newtype

* WIP

* remove mod ecdsa

* WIP

* fix tests

* some polishing

* Rename AuthorityId to BeefyId to avoid type conflict in UI (#211)

* Add trace points; Reduce MAX_LIVE_GOSSIP_ROUNDS (#210)

* Add trace points; Reduce MAX_LIVE_GOSSIP_ROUNDS

* log local authority id

* Additional initial authority id's (#217)

* Scratch concluded rounds

* adjust testnet doc

* fix authority key typo

* We don't want no scratches

* address review comments

* Fix note_round() (#219)

* rename BeefyGossipValidator

* Fix note_round()

* use const for assert

* put message trace points back in

* test case note_same_round_twice()

* address review comments

* remove redundant check

* Use LocalKeystore for tests (#224)

* private_keys()

* Use LocalKeystore for tests

* Use keystore helper

* Address review

* some reformatting

* Cache known votes in gossip (#227)

* Implement known messages cache.

* Add tests.

* Appease clippy.

* More clippy

Co-authored-by: adoerr <0xad@gmx.net>

* Some key store sanity checks (#232)

* verify vote message

* verify_validator_set()

* rework logging

* some rework

* Tone down warnings.

* Add signature verification.

* Tone down more.

* Fix clippy

Co-authored-by: Tomasz Drwięga <tomasz@parity.io>

* Use Binary Merkle Tree instead of a trie (#225)

* Binary tree merkle root.

* Add proofs and verification.

* Clean up debug.

* Use BEEFY addresses instead of pubkeys.

* Use new merkle tree.

* Optimize allocations.

* Add test for larger trees.

* Add tests for larger cases.

* Appease clippy

* Appease clippy2.

* Fix proof generation & verification.

* Add more test data.

* Fix CLI.

* Update README

* Bump version.

* Update docs.

* Rename beefy-merkle-root to beefy-merkle-tree

Co-authored-by: adoerr <0xad@gmx.net>

* Bump Substrate and Deps (#235)

* BEEFY+MMR pallet (#236)

* Add MMR leaf format to primitives.

* Fix tests

* Initial work on the BEEFY-MMR pallet.

* Add tests to MMR pallet.

* Use eth addresses.

* Use binary merkle tree.

* Bump libsecp256k1

* Fix compilation.

* Bump deps.

* Appease cargo deny.

* Re-format.

* Module-level docs.

* no-std fix.

* update README

Co-authored-by: adoerr <0xad@gmx.net>

* Fix noting rounds for non-authorities (#238)

* Bump env_logger from 0.8.4 to 0.9.0 (#242)

Bumps [env_logger](https://github.com/env-logger-rs/env_logger) from 0.8.4 to 0.9.0.
- [Release notes](https://github.com/env-logger-rs/env_logger/releases)
- [Changelog](https://github.com/env-logger-rs/env_logger/blob/main/CHANGELOG.md)
- [Commits](https://github.com/env-logger-rs/env_logger/compare/v0.8.4...v0.9.0)

---
updated-dependencies:
- dependency-name: env_logger
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* gadget: add global timeout for rebroadcasting messages (#243)

* gadget: add global timeout for rebroadcasting messages

* update rustfmt.toml

* make message_allowed() a debug trace

Co-authored-by: adoerr <0xad@gmx.net>

* Bump Substrate and Deps (#245)

* Bump Substrate and Deps

* Bump Substrate again

* Bump futures from 0.3.15 to 0.3.16 (#247)

Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.15 to 0.3.16.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.15...0.3.16)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump libsecp256k1 from 0.5.0 to 0.6.0 (#249)

* Bump libsecp256k1 from 0.5.0 to 0.6.0

Bumps [libsecp256k1](https://github.com/paritytech/libsecp256k1) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/paritytech/libsecp256k1/releases)
- [Changelog](https://github.com/paritytech/libsecp256k1/blob/master/CHANGELOG.md)
- [Commits](https://github.com/paritytech/libsecp256k1/commits)

---
updated-dependencies:
- dependency-name: libsecp256k1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* use correct crate name

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: adoerr <0xad@gmx.net>

* Derive `scale_info::TypeInfo` for types used in polkadot (#218)

* Add scale-info TypeInfo derives

* Update scale-info

* Add crates.io patches

* Use substrate aj-metadata-vnext branch

* Revert master branch substrate deps

* Add scale-info to beefy-pallet

* scale-info v0.9.0

* Remove github dependencies and patches

* More TypeInfo derives

* Update scale-info to 0.10.0

* Add missing scale-info dependency

* Add missing TypeInfo derive

* Hide TypeInfo under a feature.

Co-authored-by: Tomasz Drwięga <tomasz@parity.io>

* Bump serde from 1.0.126 to 1.0.127 (#260)

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.126 to 1.0.127.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.126...v1.0.127)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump Substrate and Deps (#262)

* Update jsonrpc (#265)

* Update jsonrpc

* Update Substrate

* bump Substrate and Deps (#268)

* Bump serde from 1.0.127 to 1.0.128 (#272)

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.127 to 1.0.128.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.127...v1.0.128)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix spelling (#271)

* Bump serde from 1.0.128 to 1.0.130 (#276)

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.128 to 1.0.130.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.128...v1.0.130)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump scale-info from 0.10.0 to 0.12.0 (#275)

Bumps [scale-info](https://github.com/paritytech/scale-info) from 0.10.0 to 0.12.0.
- [Release notes](https://github.com/paritytech/scale-info/releases)
- [Changelog](https://github.com/paritytech/scale-info/blob/master/CHANGELOG.md)
- [Commits](https://github.com/paritytech/scale-info/commits)

---
updated-dependencies:
- dependency-name: scale-info
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: adoerr <0xad@gmx.net>

* Update to scale-info 1.0 (#278)

* bump substrate (#282)

* bump Substrate and Deps

* cargo fmt

Co-authored-by: Wenfeng Wang <kalot.wang@gmail.com>

* Update worker.rs (#287)

* Bump anyhow from 1.0.43 to 1.0.44 (#290)

* Bump anyhow from 1.0.43 to 1.0.44

Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.43 to 1.0.44.
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](https://github.com/dtolnay/anyhow/compare/1.0.43...1.0.44)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* derive Default

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: adoerr <0xad@gmx.net>

* Remove optional `scale-info` feature (#292)

* Make scale-info dependency non-optional

* Remove feature gated TypeInfo derives

* Import TypeInfo

* Update substrate

* Fix up runtime

* prune .git suffix (#294)

* remove unused deps (#295)

* remove unused deps

* update lock file

* Bump libsecp256k1 from 0.6.0 to 0.7.0 (#296)

* Bump libsecp256k1 from 0.6.0 to 0.7.0

Bumps [libsecp256k1](https://github.com/paritytech/libsecp256k1) from 0.6.0 to 0.7.0.
- [Release notes](https://github.com/paritytech/libsecp256k1/releases)
- [Changelog](https://github.com/paritytech/libsecp256k1/blob/master/CHANGELOG.md)
- [Commits](https://github.com/paritytech/libsecp256k1/commits)

---
updated-dependencies:
- dependency-name: libsecp256k1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update sec advisories

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: adoerr <0xad@gmx.net>

* clean compile

* use path dependencies

* beefy-gadget license header

* pallet-beefy license header

* pallet-beefy-mmr license header

* beefy-primitves license header

* carg fmt

* more formatting

* shorten line

* downgrade parity-scale-codec to 2.2.0

* use path dependency for Prometheus endpoint

* remove clippy annotations

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomasz@parity.io>
Co-authored-by: André Silva <andrerfosilva@gmail.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Jones <ascjones@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: drewstone <drewstone329@gmail.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Wenfeng Wang <kalot.wang@gmail.com>
Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>
Co-authored-by: Squirrel <gilescope@gmail.com>
This commit is contained in:
Andreas Doerr
2021-09-23 21:02:30 +02:00
committed by GitHub
parent 6845666c5c
commit 283c8daa81
31 changed files with 4992 additions and 12 deletions
+236
View File
@@ -0,0 +1,236 @@
// This file is part of Substrate.
// Copyright (C) 2021 Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#![cfg_attr(not(feature = "std"), no_std)]
#![warn(missing_docs)]
//! A BEEFY+MMR pallet combo.
//!
//! While both BEEFY and Merkle Mountain Range (MMR) can be used separately,
//! these tools were designed to work together in unison.
//!
//! The pallet provides a standardized MMR Leaf format that is can be used
//! to bridge BEEFY+MMR-based networks (both standalone and polkadot-like).
//!
//! The MMR leaf contains:
//! 1. Block number and parent block hash.
//! 2. Merkle Tree Root Hash of next BEEFY validator set.
//! 3. Merkle Tree Root Hash of current parachain heads state.
//!
//! and thanks to versioning can be easily updated in the future.
use sp_runtime::traits::{Convert, Hash};
use sp_std::prelude::*;
use beefy_primitives::mmr::{BeefyNextAuthoritySet, MmrLeaf, MmrLeafVersion};
use pallet_mmr::primitives::LeafDataProvider;
use codec::Encode;
use frame_support::traits::Get;
pub use pallet::*;
#[cfg(test)]
mod mock;
#[cfg(test)]
mod tests;
/// A BEEFY consensus digest item with MMR root hash.
pub struct DepositBeefyDigest<T>(sp_std::marker::PhantomData<T>);
impl<T> pallet_mmr::primitives::OnNewRoot<beefy_primitives::MmrRootHash> for DepositBeefyDigest<T>
where
T: pallet_mmr::Config<Hash = beefy_primitives::MmrRootHash>,
T: pallet_beefy::Config,
{
fn on_new_root(root: &<T as pallet_mmr::Config>::Hash) {
let digest = sp_runtime::generic::DigestItem::Consensus(
beefy_primitives::BEEFY_ENGINE_ID,
codec::Encode::encode(&beefy_primitives::ConsensusLog::<
<T as pallet_beefy::Config>::BeefyId,
>::MmrRoot(*root)),
);
<frame_system::Pallet<T>>::deposit_log(digest);
}
}
/// Convert BEEFY secp256k1 public keys into Ethereum addresses
pub struct BeefyEcdsaToEthereum;
impl Convert<beefy_primitives::crypto::AuthorityId, Vec<u8>> for BeefyEcdsaToEthereum {
fn convert(a: beefy_primitives::crypto::AuthorityId) -> Vec<u8> {
use sp_core::crypto::Public;
let compressed_key = a.as_slice();
libsecp256k1::PublicKey::parse_slice(
compressed_key,
Some(libsecp256k1::PublicKeyFormat::Compressed),
)
// uncompress the key
.map(|pub_key| pub_key.serialize().to_vec())
// now convert to ETH address
.map(|uncompressed| sp_io::hashing::keccak_256(&uncompressed[1..])[12..].to_vec())
.map_err(|_| {
log::error!(target: "runtime::beefy", "Invalid BEEFY PublicKey format!");
})
.unwrap_or_default()
}
}
type MerkleRootOf<T> = <T as pallet_mmr::Config>::Hash;
type ParaId = u32;
type ParaHead = Vec<u8>;
/// A type that is able to return current list of parachain heads that end up in the MMR leaf.
pub trait ParachainHeadsProvider {
/// Return a list of tuples containing a `ParaId` and Parachain Header data (ParaHead).
///
/// The returned data does not have to be sorted.
fn parachain_heads() -> Vec<(ParaId, ParaHead)>;
}
/// A default implementation for runtimes without parachains.
impl ParachainHeadsProvider for () {
fn parachain_heads() -> Vec<(ParaId, ParaHead)> {
Default::default()
}
}
#[frame_support::pallet]
pub mod pallet {
#![allow(missing_docs)]
use super::*;
use frame_support::pallet_prelude::*;
/// BEEFY-MMR pallet.
#[pallet::pallet]
#[pallet::generate_store(pub(super) trait Store)]
pub struct Pallet<T>(_);
/// The module's configuration trait.
#[pallet::config]
#[pallet::disable_frame_system_supertrait_check]
pub trait Config: pallet_mmr::Config + pallet_beefy::Config {
/// Current leaf version.
///
/// Specifies the version number added to every leaf that get's appended to the MMR.
/// Read more in [`MmrLeafVersion`] docs about versioning leaves.
type LeafVersion: Get<MmrLeafVersion>;
/// Convert BEEFY AuthorityId to a form that would end up in the Merkle Tree.
///
/// For instance for ECDSA (secp256k1) we want to store uncompressed public keys (65 bytes)
/// and later to Ethereum Addresses (160 bits) to simplify using them on Ethereum chain,
/// but the rest of the Substrate codebase is storing them compressed (33 bytes) for
/// efficiency reasons.
type BeefyAuthorityToMerkleLeaf: Convert<<Self as pallet_beefy::Config>::BeefyId, Vec<u8>>;
/// Retrieve a list of current parachain heads.
///
/// The trait is implemented for `paras` module, but since not all chains might have
/// parachains, and we want to keep the MMR leaf structure uniform, it's possible to use
/// `()` as well to simply put dummy data to the leaf.
type ParachainHeads: ParachainHeadsProvider;
}
/// Details of next BEEFY authority set.
///
/// This storage entry is used as cache for calls to [`update_beefy_next_authority_set`].
#[pallet::storage]
#[pallet::getter(fn beefy_next_authorities)]
pub type BeefyNextAuthorities<T: Config> =
StorageValue<_, BeefyNextAuthoritySet<MerkleRootOf<T>>, ValueQuery>;
}
impl<T: Config> LeafDataProvider for Pallet<T>
where
MerkleRootOf<T>: From<beefy_merkle_tree::Hash> + Into<beefy_merkle_tree::Hash>,
{
type LeafData = MmrLeaf<
<T as frame_system::Config>::BlockNumber,
<T as frame_system::Config>::Hash,
MerkleRootOf<T>,
>;
fn leaf_data() -> Self::LeafData {
MmrLeaf {
version: T::LeafVersion::get(),
parent_number_and_hash: frame_system::Pallet::<T>::leaf_data(),
parachain_heads: Pallet::<T>::parachain_heads_merkle_root(),
beefy_next_authority_set: Pallet::<T>::update_beefy_next_authority_set(),
}
}
}
impl<T: Config> beefy_merkle_tree::Hasher for Pallet<T>
where
MerkleRootOf<T>: Into<beefy_merkle_tree::Hash>,
{
fn hash(data: &[u8]) -> beefy_merkle_tree::Hash {
<T as pallet_mmr::Config>::Hashing::hash(data).into()
}
}
impl<T: Config> Pallet<T>
where
MerkleRootOf<T>: From<beefy_merkle_tree::Hash> + Into<beefy_merkle_tree::Hash>,
{
/// Returns latest root hash of a merkle tree constructed from all active parachain headers.
///
/// The leafs are sorted by `ParaId` to allow more efficient lookups and non-existence proofs.
///
/// NOTE this does not include parathreads - only parachains are part of the merkle tree.
///
/// NOTE This is an initial and inefficient implementation, which re-constructs
/// the merkle tree every block. Instead we should update the merkle root in
/// [Self::on_initialize] call of this pallet and update the merkle tree efficiently (use
/// on-chain storage to persist inner nodes).
fn parachain_heads_merkle_root() -> MerkleRootOf<T> {
let mut para_heads = T::ParachainHeads::parachain_heads();
para_heads.sort();
let para_heads = para_heads.into_iter().map(|pair| pair.encode());
beefy_merkle_tree::merkle_root::<Self, _, _>(para_heads).into()
}
/// Returns details of the next BEEFY authority set.
///
/// Details contain authority set id, authority set length and a merkle root,
/// constructed from uncompressed secp256k1 public keys converted to Ethereum addresses
/// of the next BEEFY authority set.
///
/// This function will use a storage-cached entry in case the set didn't change, or compute and
/// cache new one in case it did.
fn update_beefy_next_authority_set() -> BeefyNextAuthoritySet<MerkleRootOf<T>> {
let id = pallet_beefy::Pallet::<T>::validator_set_id() + 1;
let current_next = Self::beefy_next_authorities();
// avoid computing the merkle tree if validator set id didn't change.
if id == current_next.id {
return current_next
}
let beefy_addresses = pallet_beefy::Pallet::<T>::next_authorities()
.into_iter()
.map(T::BeefyAuthorityToMerkleLeaf::convert)
.collect::<Vec<_>>();
let len = beefy_addresses.len() as u32;
let root = beefy_merkle_tree::merkle_root::<Self, _, _>(beefy_addresses).into();
let next_set = BeefyNextAuthoritySet { id, len, root };
// cache the result
BeefyNextAuthorities::<T>::put(&next_set);
next_set
}
}
+205
View File
@@ -0,0 +1,205 @@
// This file is part of Substrate.
// Copyright (C) 2021 Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::vec;
use beefy_primitives::mmr::MmrLeafVersion;
use frame_support::{
construct_runtime, parameter_types, sp_io::TestExternalities, BasicExternalities,
};
use sp_core::{Hasher, H256};
use sp_runtime::{
app_crypto::ecdsa::Public,
impl_opaque_keys,
testing::Header,
traits::{BlakeTwo256, ConvertInto, IdentityLookup, Keccak256, OpaqueKeys},
Perbill,
};
use crate as pallet_beefy_mmr;
pub use beefy_primitives::{crypto::AuthorityId as BeefyId, ConsensusLog, BEEFY_ENGINE_ID};
impl_opaque_keys! {
pub struct MockSessionKeys {
pub dummy: pallet_beefy::Pallet<Test>,
}
}
type UncheckedExtrinsic = frame_system::mocking::MockUncheckedExtrinsic<Test>;
type Block = frame_system::mocking::MockBlock<Test>;
construct_runtime!(
pub enum Test where
Block = Block,
NodeBlock = Block,
UncheckedExtrinsic = UncheckedExtrinsic,
{
System: frame_system::{Pallet, Call, Config, Storage, Event<T>},
Session: pallet_session::{Pallet, Call, Storage, Event, Config<T>},
Mmr: pallet_mmr::{Pallet, Storage},
Beefy: pallet_beefy::{Pallet, Config<T>, Storage},
BeefyMmr: pallet_beefy_mmr::{Pallet, Storage},
}
);
parameter_types! {
pub const BlockHashCount: u64 = 250;
pub const SS58Prefix: u8 = 42;
}
impl frame_system::Config for Test {
type BaseCallFilter = frame_support::traits::Everything;
type BlockWeights = ();
type BlockLength = ();
type DbWeight = ();
type Origin = Origin;
type Index = u64;
type BlockNumber = u64;
type Hash = H256;
type Call = Call;
type Hashing = BlakeTwo256;
type AccountId = u64;
type Lookup = IdentityLookup<Self::AccountId>;
type Header = Header;
type Event = Event;
type BlockHashCount = BlockHashCount;
type Version = ();
type PalletInfo = PalletInfo;
type AccountData = ();
type OnNewAccount = ();
type OnKilledAccount = ();
type SystemWeightInfo = ();
type SS58Prefix = SS58Prefix;
type OnSetCode = ();
}
parameter_types! {
pub const Period: u64 = 1;
pub const Offset: u64 = 0;
pub const DisabledValidatorsThreshold: Perbill = Perbill::from_percent(33);
}
impl pallet_session::Config for Test {
type Event = Event;
type ValidatorId = u64;
type ValidatorIdOf = ConvertInto;
type ShouldEndSession = pallet_session::PeriodicSessions<Period, Offset>;
type NextSessionRotation = pallet_session::PeriodicSessions<Period, Offset>;
type SessionManager = MockSessionManager;
type SessionHandler = <MockSessionKeys as OpaqueKeys>::KeyTypeIdProviders;
type Keys = MockSessionKeys;
type DisabledValidatorsThreshold = DisabledValidatorsThreshold;
type WeightInfo = ();
}
pub type MmrLeaf = beefy_primitives::mmr::MmrLeaf<
<Test as frame_system::Config>::BlockNumber,
<Test as frame_system::Config>::Hash,
<Test as pallet_mmr::Config>::Hash,
>;
impl pallet_mmr::Config for Test {
const INDEXING_PREFIX: &'static [u8] = b"mmr";
type Hashing = Keccak256;
type Hash = <Keccak256 as Hasher>::Out;
type LeafData = BeefyMmr;
type OnNewRoot = pallet_beefy_mmr::DepositBeefyDigest<Test>;
type WeightInfo = ();
}
impl pallet_beefy::Config for Test {
type BeefyId = BeefyId;
}
parameter_types! {
pub LeafVersion: MmrLeafVersion = MmrLeafVersion::new(1, 5);
}
impl pallet_beefy_mmr::Config for Test {
type LeafVersion = LeafVersion;
type BeefyAuthorityToMerkleLeaf = pallet_beefy_mmr::BeefyEcdsaToEthereum;
type ParachainHeads = DummyParaHeads;
}
pub struct DummyParaHeads;
impl pallet_beefy_mmr::ParachainHeadsProvider for DummyParaHeads {
fn parachain_heads() -> Vec<(pallet_beefy_mmr::ParaId, pallet_beefy_mmr::ParaHead)> {
vec![(15, vec![1, 2, 3]), (5, vec![4, 5, 6])]
}
}
pub struct MockSessionManager;
impl pallet_session::SessionManager<u64> for MockSessionManager {
fn end_session(_: sp_staking::SessionIndex) {}
fn start_session(_: sp_staking::SessionIndex) {}
fn new_session(idx: sp_staking::SessionIndex) -> Option<Vec<u64>> {
if idx == 0 || idx == 1 {
Some(vec![1, 2])
} else if idx == 2 {
Some(vec![3, 4])
} else {
None
}
}
}
// Note, that we can't use `UintAuthorityId` here. Reason is that the implementation
// of `to_public_key()` assumes, that a public key is 32 bytes long. This is true for
// ed25519 and sr25519 but *not* for ecdsa. An ecdsa public key is 33 bytes.
pub fn mock_beefy_id(id: u8) -> BeefyId {
let buf: [u8; 33] = [id; 33];
let pk = Public::from_raw(buf);
BeefyId::from(pk)
}
pub fn mock_authorities(vec: Vec<u8>) -> Vec<(u64, BeefyId)> {
vec.into_iter().map(|id| ((id as u64), mock_beefy_id(id))).collect()
}
pub fn new_test_ext(ids: Vec<u8>) -> TestExternalities {
new_test_ext_raw_authorities(mock_authorities(ids))
}
pub fn new_test_ext_raw_authorities(authorities: Vec<(u64, BeefyId)>) -> TestExternalities {
let mut t = frame_system::GenesisConfig::default().build_storage::<Test>().unwrap();
let session_keys: Vec<_> = authorities
.iter()
.enumerate()
.map(|(_, id)| (id.0 as u64, id.0 as u64, MockSessionKeys { dummy: id.1.clone() }))
.collect();
BasicExternalities::execute_with_storage(&mut t, || {
for (ref id, ..) in &session_keys {
frame_system::Pallet::<Test>::inc_providers(id);
}
});
pallet_session::GenesisConfig::<Test> { keys: session_keys }
.assimilate_storage(&mut t)
.unwrap();
t.into()
}
+148
View File
@@ -0,0 +1,148 @@
// This file is part of Substrate.
// Copyright (C) 2021 Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::vec;
use beefy_primitives::{
mmr::{BeefyNextAuthoritySet, MmrLeafVersion},
ValidatorSet,
};
use codec::{Decode, Encode};
use hex_literal::hex;
use sp_core::H256;
use sp_io::TestExternalities;
use sp_runtime::{traits::Keccak256, DigestItem};
use frame_support::traits::OnInitialize;
use crate::mock::*;
fn init_block(block: u64) {
System::set_block_number(block);
Session::on_initialize(block);
Mmr::on_initialize(block);
Beefy::on_initialize(block);
BeefyMmr::on_initialize(block);
}
pub fn beefy_log(log: ConsensusLog<BeefyId>) -> DigestItem<H256> {
DigestItem::Consensus(BEEFY_ENGINE_ID, log.encode())
}
fn offchain_key(pos: usize) -> Vec<u8> {
(<Test as pallet_mmr::Config>::INDEXING_PREFIX, pos as u64).encode()
}
fn read_mmr_leaf(ext: &mut TestExternalities, index: usize) -> MmrLeaf {
type Node = pallet_mmr_primitives::DataOrHash<Keccak256, MmrLeaf>;
ext.persist_offchain_overlay();
let offchain_db = ext.offchain_db();
offchain_db
.get(&offchain_key(index))
.map(|d| Node::decode(&mut &*d).unwrap())
.map(|n| match n {
Node::Data(d) => d,
_ => panic!("Unexpected MMR node."),
})
.unwrap()
}
#[test]
fn should_contain_mmr_digest() {
let mut ext = new_test_ext(vec![1, 2, 3, 4]);
ext.execute_with(|| {
init_block(1);
assert_eq!(
System::digest().logs,
vec![beefy_log(ConsensusLog::MmrRoot(
hex!("f3e3afbfa69e89cd1e99f8d3570155962f3346d1d8758dc079be49ef70387758").into()
))]
);
// unique every time
init_block(2);
assert_eq!(
System::digest().logs,
vec![
beefy_log(ConsensusLog::MmrRoot(
hex!("f3e3afbfa69e89cd1e99f8d3570155962f3346d1d8758dc079be49ef70387758").into()
)),
beefy_log(ConsensusLog::AuthoritiesChange(ValidatorSet {
validators: vec![mock_beefy_id(3), mock_beefy_id(4),],
id: 1,
})),
beefy_log(ConsensusLog::MmrRoot(
hex!("7d4ae4524bae75d52b63f08eab173b0c263eb95ae2c55c3a1d871241bd0cc559").into()
)),
]
);
});
}
#[test]
fn should_contain_valid_leaf_data() {
let mut ext = new_test_ext(vec![1, 2, 3, 4]);
ext.execute_with(|| {
init_block(1);
});
let mmr_leaf = read_mmr_leaf(&mut ext, 0);
assert_eq!(
mmr_leaf,
MmrLeaf {
version: MmrLeafVersion::new(1, 5),
parent_number_and_hash: (0_u64, H256::repeat_byte(0x45)),
beefy_next_authority_set: BeefyNextAuthoritySet {
id: 1,
len: 2,
root: hex!("01b1a742589773fc054c8f5021a456316ffcec0370b25678b0696e116d1ef9ae")
.into(),
},
parachain_heads: hex!(
"ed893c8f8cc87195a5d4d2805b011506322036bcace79642aa3e94ab431e442e"
)
.into(),
}
);
// build second block on top
ext.execute_with(|| {
init_block(2);
});
let mmr_leaf = read_mmr_leaf(&mut ext, 1);
assert_eq!(
mmr_leaf,
MmrLeaf {
version: MmrLeafVersion::new(1, 5),
parent_number_and_hash: (1_u64, H256::repeat_byte(0x45)),
beefy_next_authority_set: BeefyNextAuthoritySet {
id: 2,
len: 2,
root: hex!("9c6b2c1b0d0b25a008e6c882cc7b415f309965c72ad2b944ac0931048ca31cd5")
.into(),
},
parachain_heads: hex!(
"ed893c8f8cc87195a5d4d2805b011506322036bcace79642aa3e94ab431e442e"
)
.into(),
}
);
}