Commit Graph

18 Commits

Author SHA1 Message Date
Robert Habermeier ccfabaa0c6 Node-side subsystems for Disputes (#2566)
* dispute subsystem files

* rename

* fix linkcheck

* flesh out section README

* coordinator schema

* DisputeCoordinatorMessage

* stub & coordinator protocol

* dispute coordinator

* add some more message fields

* move links to bottom

* dispute participation

* Cleen It Up !

* runtime: store candidate receipts in dispute state

yeah, this is a little heavier. why are you reading this?

* Revert "runtime: store candidate receipts in dispute state"

This reverts commit 51c10bfd4d866e287e6bd88f317ed57ed987eaee.

* add dispute availability statement type and prepare for availability

* add 'spam slots' to disputes runtmie

* return Spam Slots info from runtime

* rework `ImportStatement` to `ImportStatements`

* some more methods for dispute coordinator

* candidates-included runtime API

* algo for providing disputes to runtime.

* handle signing with coordinator

* dispute coordinator chain ops

* remove dead file

* remove keystore from dispute participation

* adjust ApprovedAncestor to return the necssary data

* discuss how approved ancestor and determine undisputed chain are used together

* add TODO

* initiate disputes from approval voting

* route statements from candidate backing and approval voting

* fix guide build
2021-04-01 03:00:46 +02:00
Robert Habermeier 349879df6b f+1 validators always approve (#2699)
* f+1 always approves

* guide

* grumbles

* grumbles

* fix test

* fix tests

* Update roadmap/implementers-guide/src/node/approval/approval-voting.md

Co-authored-by: Sergei Shulepov <sergei@parity.io>

Co-authored-by: Sergei Shulepov <sergei@parity.io>
2021-03-25 13:47:00 +01:00
Robert Habermeier 3300b53306 Approval Checking Improvements Omnibus (#2480)
* add tracing to approval voting

* notify if session info is not working

* add dispute period to chain specs

* propagate genesis session to parachains runtime

* use `on_genesis_session`

* protect against zero cores in computation

* tweak voting rule to be based off of best and add logs

* genesis configuration should use VRF slots only

* swallow more keystore errors

* add some docs

* make validation-worker args non-optional and update clap

* better tracing for bitfield signing and provisioner

* pass amount of bits in bitfields to inclusion instead of recomputing

* debug -> warn for some logs

* better tracing for availability recovery

* a little av-store tracing

* bridge: forward availability recovery messages

* add missing try_from impl

* some more tracing

* improve approval distribution tracing

* guide: hold onto pending approval messages until NewBlocks

* Hold onto pending approval messages until NewBlocks

* guide: adjust comment

* process all actions for one wakeup at a time

* vec

* fix network bridge test

* replace randomness-collective-flip with Babe

* remove PairNotFound
2021-02-23 14:12:28 -06:00
Robert Habermeier b7aac51341 A fast-path for requesting AvailableData from backing validators (#2453)
* guide changes for a fast-path requesting from backing validators

* add backing group to availability recovery message

* add new phase to interaction

* typos

* add full data messages

* handle new network messages

* dispatch full data requests

* cleanup

* check chunk index

* test for invalid recovery

* tests

* Typos.

* fix some grumbles

* be more explicit about error handling and control flow

* fast-path param

* use with_chunks_only in Service

Co-authored-by: Robert Klotzner <robert.klotzner@gmx.at>
2021-02-17 13:51:50 -06:00
Robert Habermeier 4f21cc7e5c Integrate Approval Voting into Overseer / Service / GRANDPA (#2412)
* integrate approval voting into overseer

* expose public API and make keystore arc

* integrate overseer in service

* guide: `ApprovedAncestor` returns block number

* return block number along with hash from ApprovedAncestor

* introduce a voting rule for reporting on approval checking

* integrate the delay voting rule

* Rococo configuration

* fix compilation and add slack

* fix web-wasm build

* tweak parameterization

* migrate voting rules to asycn

* remove hack comment
2021-02-15 22:37:13 -06:00
Robert Habermeier e48c687504 Implement Approval Voting Subsystem (#2112)
* skeleton

* skeleton aux-schema module

* start approval types

* start aux schema with aux store

* doc

* finish basic types

* start approval types

* doc

* finish basic types

* write out schema types

* add debug and codec impls to approval types

* add debug and codec impls to approval types

also add some key computation

* add debug and codec impls to approval types

* getters for block and candidate entries

* grumbles

* remove unused AssignmentId

* load_decode utility

* implement DB clearing

* function for adding new block entry to aux store

* start `canonicalize` implementation

* more skeleton

* finish implementing canonicalize

* tag TODO

* implement a test AuxStore

* add allow(unused)

* basic loading and deleting test

* block_entry test function

* add a test for `add_block_entry`

* ensure range is exclusive at end

* test clear()

* test that add_block sets children

* add a test for canonicalize

* extract Pre-digest from header

* utilities for extracting RelayVRFStory from the header-chain

* add approval voting message types

* approval distribution message type

* subsystem skeleton

* state struct

* add futures-timer

* prepare service for babe slot duration

* more skeleton

* better integrate AuxStore

* RelayVRF -> RelayVRFStory

* canonicalize

* implement some tick functionality

* guide: tweaks

* check_approval

* more tweaks and helpers

* guide: add core index to candidate event

* primitives: add core index to candidate event

* runtime: add core index to candidate events

* head handling (session window)

* implement `determine_new_blocks`

* add TODO

* change error type on functions

* compute RelayVRFModulo assignments

* compute RelayVRFDelay assignments

* fix delay tranche calc

* assignment checking

* pluralize

* some dummy code for fetching assignments

* guide: add babe epoch runtime API

* implement a current_epoch() runtime API

* compute assignments

* candidate events get backing group

* import blocks and assignments into DB

* push block approval meta

* add message types, no overseer integration yet

* notify approval distribution of new blocks

* refactor import into separate functions

* impl tranches_to_approve

* guide: improve function signatures

* guide: remove Tick from ApprovalEntry

* trigger and broadcast assignment

* most of approval launching

* remove byteorder crate

* load blocks back to finality, except on startup

* check unchecked assignments

* add claimed core to approval voting message

* fix checks

* assign only to backing group

* remove import_checked_assignment from guide

* newline

* import assignments

* abstract out a bit

* check and import approvals

* check full approvals from assignment import too

* comment

* create a Transaction utility

* must_use

* use transaction in `check_full_approvals`

* wire up wakeups

* add Ord to CandidateHash

* wakeup refactoring

* return candidate info from add_block_entry

* schedule wakeups

* background task: do candidate validation

* forward candidate validation requests

* issue approval votes when requested

* clean up a couple TODOs

* fix up session caching

* clean up last unimplemented!() items

* fix remaining warnings

* remove TODO

* implement handle_approved_ancestor

* update Cargo.lock

* fix runtime API tests

* guide: cleanup assignment checking

* use claimed candidate index instead of core

* extract time to a trait

* tests module

* write a mock clock for testing

* allow swapping out the clock

* make abstract over assignment criteria

* add some skeleton tests and simplify params

* fix backing group check

* do backing group check inside check_assignment_cert

* write some empty test functions to implement

* add a test for non-backing

* test that produced checks pass

* some empty test ideas

* runtime/inclusion: remove outdated TODO

* fix compilation

* av-store: fix tests

* dummy cert

* criteria tests

* move `TestStore` to main tests file

* fix unused warning

* test harness beginnings

* resolve slots renaming fallout

* more compilation fixes

* wip: extract pure data into a separate module

* wip: extract pure data into a separate module

* move types completely to v1

* add persisted_entries

* add conversion trait impls

* clean up some warnings

* extract import logic to own module

* schedule wakeups

* experiment with Actions

* uncomment approval-checking

* separate module for approval checking utilities

* port more code to use actions

* get approval pipeline using actions

* all logic is uncommented

* main loop processes actions

* all loop logic uncommented

* separate function for handling actions

* remove last unimplemented item

* clean up warnings

* State gives read-only access to underlying DB

* tests for approval checking

* tests for approval criteria

* skeleton test module for import

* list of import tests to do

* some test glue code

* test reject bad assignment

* test slot too far in future

* test reject assignment with unknown candidate

* remove loads_blocks tests

* determine_new_blocks back to finalized & harness

* more coverage for determining new blocks

* make `imported_block_info` have less reliance on State

* candidate_info tests

* tests for session caching

* remove println

* extricate DB and main TestStores

* rewrite approval checking logic to counteract early delays

* move state out of function

* update approval-checking tests

* tweak wakeups & scheduling logic

* rename check_full_approvals

* test that assignment import updates candidate

* some approval import tests

* some tests for check_and_apply_approval

* add 'full' qualifier to avoid confusion

* extract should-trigger logic to separate function

* some tests for all triggering

* tests for when we trigger assignments

* test wakeups

* add block utilities for testing

* some more tests for approval updates

* approved_ancestor tests

* new action type for launch approval

* process-wakeup tests

* clean up some warnings

* fix in_future test

* approval checking tests

* tighten up too-far-in-future

* special-case genesis when caching sessions

* fix bitfield len

Co-authored-by: Andronik Ordian <write@reusable.software>
2021-02-11 10:21:47 -06:00
Robert Habermeier 28cab8e190 guide: more robust approval counting procedure (#2378)
* guide: more robust approval counting procedure

* elaborate

* redefine non-empty

* improve writing

* remove redundant
2021-02-07 16:21:55 -06:00
Robert Habermeier 956bc300c1 Approval voting guide changes (#2347)
* grab guide updates from approval-voting branch

* focus on control flow and behavior, not functions

* add approval voting preamble info

* tabs -> spaces
2021-01-29 17:06:16 +00:00
Bastian Köcher 7f019517ca Companion for #7997 (#2345)
* Companion for #7997

https://github.com/paritytech/substrate/pull/7997

* rename slot_number to slot

* rename SlotNumber type in overseer docs

* "Update Substrate"

Co-authored-by: André Silva <andrerfosilva@gmail.com>
Co-authored-by: parity-processbot <>
2021-01-28 20:04:43 +00:00
Andronik Ordian 3f1e1a6ff7 impl approval distribution (#2160)
* initial impl approval distribution

* initial tests and fixes

* batching seems difficult: different peers have different needs

* bridge: fix test after merge

* some guide updates

* only send assignments to peers who know about the block

* fix a test, add approvals test

* simplify

* do not send assignment to peers for finalized blocks

* guide: protocol input and output

* one more test

* more comments, logs, initial metrics

* fix a typo

* one more thing: early return when reimporting a thing locally
2021-01-25 18:14:32 -05:00
Robert Habermeier b009cbe801 Approval Voting Database (#2162)
* skeleton

* skeleton aux-schema module

* start approval types

* start aux schema with aux store

* doc

* finish basic types

* start approval types

* doc

* finish basic types

* write out schema types

* add debug and codec impls to approval types

* add debug and codec impls to approval types

also add some key computation

* add debug and codec impls to approval types

* getters for block and candidate entries

* grumbles

* remove unused AssignmentId

* load_decode utility

* implement DB clearing

* function for adding new block entry to aux store

* start `canonicalize` implementation

* more skeleton

* finish implementing canonicalize

* tag TODO

* implement a test AuxStore

* add allow(unused)

* basic loading and deleting test

* block_entry test function

* add a test for `add_block_entry`

* ensure range is exclusive at end

* test clear()

* test that add_block sets children

* add a test for canonicalize

* Update node/core/approval-voting/src/aux_schema/mod.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Update node/core/approval-voting/src/aux_schema/tests.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Update node/core/approval-voting/src/aux_schema/mod.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
2021-01-18 11:07:04 -05:00
Robert Habermeier 15c253117d Add assignment keys to session keys, no separate approvals key (#2092)
* guide: merge backing and approval keys

* bump substrate master & update primitives

* use new SessionInfo struct in session_info

* session keys upgrade for Polkadot

* kusama & westend runtimes

* bump westend, kusama, and polkadot versions

* add session key to rococo & test-runtime

* update prepare-test-net to latest subkey

* update chain specs to support new para_assignment session key

* get cargo.lock from master

* formatting

* update kill_storage based on substrate master

* fix test-service

* assgn -> asgn

* use session info module for assignment session key
2020-12-10 20:30:27 -06:00
Robert Habermeier 9071120de1 guide: Availability Recovery (#2011)
* expand description

* basics of availability recovery

* finish availability recovery

* fill out missing text

* Update roadmap/implementers-guide/src/node/availability/availability-recovery.md

Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>

* fix signal handling

Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
2020-12-01 14:58:31 -05:00
Robert Habermeier 0c84214814 change approval voting counting procedure (#1972)
* change approval voting counting procedure

* language

* Update roadmap/implementers-guide/src/node/approval/approval-voting.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* improve language

* time-shifting

* tweak time-shifting

* expand

* typo

* tweaks to ensure we always get woken up

* move timing check into `tranches_to_approve`

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
2020-11-28 18:35:46 +00:00
Robert Habermeier 51f6cb1979 Approval Distribution Subsystem (#1951)
* skeleton flow control

* tweaks & rename to approvals distribution

* Update roadmap/implementers-guide/src/node/approval/approval-distribution.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Update roadmap/implementers-guide/src/node/approval/approval-distribution.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* add a `NewBlocks` message and dispatch

* new data format for approval distribution

* guide: update view to include finalized block number

* approvals: document view updating

* pruning when peers disconnect

* add remaining message types

* fix link

* network message type

* handle incoming assignments

* import_and_circulate_approval

* handle new blocks

* address review comments

* address review comments and use nifty VRFProof

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
2020-11-24 23:40:07 -05:00
Robert Habermeier 5603e0f166 Session management for approval voting (#1973)
* elaborate on runtime API

* clarify what to do if the runtime API calls fail

* Update roadmap/implementers-guide/src/node/approval/approval-voting.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Update roadmap/implementers-guide/src/node/approval/approval-voting.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-11-24 16:44:02 -05:00
Robert Habermeier 1f4121c444 Runtime API for historical validation code (#1893)
* fix: ensure candidate validation gets code based on occupied core assumption

* guide: runtime API for historical validation code

* add historical runtime API

* integrate into runtime API subsystem

* remove blocked TODO

* fix service build: enable notifications protocol only under real overseer

* Update node/subsystem/src/messages.rs

Co-authored-by: Sergei Shulepov <sergei@parity.io>

* fix compilation

Co-authored-by: Robert Habermeier <robert@Roberts-MacBook-Pro.local>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
2020-11-02 12:19:53 -06:00
Robert Habermeier 43be64f2f7 Implementer's guide: Approval Voting Subsystem (#1691)
* add storage for approvals module

* basics of approval logic

* fix links

* Session info module

* create an approvals_inherent module

* integrate approvals module with inclusion

* Remove approvals runtime

* tweak mentions of on-chain logic

* add note on finality-grandpa voting rule

* elaborate on node-side components

* stub for availability recovery

* add another note on voting rule

* Beginnings of approval subsystems

* flesh out approval voting now

* logic for checking assignment certs

* initial scheduler logic

* scheduler logic

* adjst tranche taking logic

* approval voting import

* approval work (voting side)

* amend some TODOs

* mark some TODOs

* describe `ApprovedAncestor`

* reference protocol-approval.md

* clarity on bitfield

* remove approvals_inherent

* tweak session_info module according to review

* formatting & nits

Co-authored-by: Robert Habermeier <robert@Roberts-MacBook-Pro.local>
2020-10-30 22:07:52 -05:00