pezkuwichain/pezkuwi-subxt
1
0
Fork 0
mirror of https://github.com/pezkuwichain/pezkuwi-subxt.git synced 2026-06-25 01:21:06 +00:00
Code Issues Packages Projects Releases Wiki Activity
14,458 Commits 132 Branches 288 Tags
7cbe0c76ef8fd2aabf9f07de0156941ce3ed44b0
Commit Graph

154 Commits

Author SHA1 Message Date
Peter Goodspeed-Niklaus a0239785b9 Message and Data Type Flow Visualizations (#2286) 
* start graphing message flow between subsystems

This commit includes messages originating from the first three subsystems
(alphabetically).

* use standard arrowhead names

* add messages from bitfield distribution && use circo layout

* finish adding all subsystem message traffic to graph

* reduce line spam

* start work on graphing v1 types

* finish graphing candidate-related structs

* show hash relations, format

* show provenance of ValidityAttestation

* add the rest of the v1 polkadot primitives

* add type diagram from polkadot::parachain::primitives

* start work digramming the path to backing

* diagram message flow from collation generation -> candidate selection

* document flow through CandidateBacking, CandidateValidation

* graph data flow through StatementDistribution

* trace data flow through PoVDistribution to Backing

* finish documenting node side subsystem data flow
 2021-01-26 11:34:45 +01:00
Andronik Ordian 3f1e1a6ff7 impl approval distribution (#2160) 
* initial impl approval distribution

* initial tests and fixes

* batching seems difficult: different peers have different needs

* bridge: fix test after merge

* some guide updates

* only send assignments to peers who know about the block

* fix a test, add approvals test

* simplify

* do not send assignment to peers for finalized blocks

* guide: protocol input and output

* one more test

* more comments, logs, initial metrics

* fix a typo

* one more thing: early return when reimporting a thing locally
 2021-01-25 18:14:32 -05:00
Sergei Shulepov 226af6a877 Remove TransientValidationData (#2272) 
* collation-generation: use persisted validation data

* node: remote FullValidationData API

* runtime: remove FullValidationData API

* backing tests: use persisted validation data

* FullCandidateReceipt: use persisted validation data

This is not a big change since this type is not used anywhere

* Remove ValidationData and TransientValidationData

Also update the guide
 2021-01-18 18:57:09 -05:00
Robert Habermeier b009cbe801 Approval Voting Database (#2162) 
* skeleton

* skeleton aux-schema module

* start approval types

* start aux schema with aux store

* doc

* finish basic types

* start approval types

* doc

* finish basic types

* write out schema types

* add debug and codec impls to approval types

* add debug and codec impls to approval types

also add some key computation

* add debug and codec impls to approval types

* getters for block and candidate entries

* grumbles

* remove unused AssignmentId

* load_decode utility

* implement DB clearing

* function for adding new block entry to aux store

* start `canonicalize` implementation

* more skeleton

* finish implementing canonicalize

* tag TODO

* implement a test AuxStore

* add allow(unused)

* basic loading and deleting test

* block_entry test function

* add a test for `add_block_entry`

* ensure range is exclusive at end

* test clear()

* test that add_block sets children

* add a test for canonicalize

* Update node/core/approval-voting/src/aux_schema/mod.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Update node/core/approval-voting/src/aux_schema/tests.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Update node/core/approval-voting/src/aux_schema/mod.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
 2021-01-18 11:07:04 -05:00
Robert Klotzner 32d4670b0d Some refactoring in network-bridge in the course of dealing with #2177 (#2263) 
* More doc fixes.

* Minor refactorings in the process of #2177

By having everything peer set related depend directly on the enum the
code becomes more clear and it is also straight forward to add more
peersets/protocols as the compiler will complain if you forget to
implement parts of it.

* Add peer set infos on startup properly

For feature real_overseer.

+ Fixes from review. Thanks @coriolinus and @ordian!

* More structure in network-bridge

Some changes, which would have helped me in groking the code faster.

Entry points/public types more to the top. Factored out implementation
in their own files, to clear up the top-level view.

* Get rid of local ProtocolName type definition.

Does not add much at this level.

* Fix tests + import cleanup.

* Make spaces tabs.

* Clarify what correct parameters to send_message are

* Be more less vague in docs of send_message.

* Apply suggestions from code review

Extend copyright on new files to 2021 as well.

Co-authored-by: Andronik Ordian <write@reusable.software>

Co-authored-by: Andronik Ordian <write@reusable.software>
 2021-01-14 10:29:02 +00:00
Robert Habermeier aedf089691 alternate availability store schema (#2237) 
* alternate availability store schema

* improvements

* tweaks

* new DB schema and skeleton

* expand skeleton and tweaks

* handle backing and inclusion

* let finality be handled later

* handle finalized blocks

* implement query methods

* implement chunk storing

* StoreAvailableData

* fix an off-by-one

* implement pruning

* reinstate subsystem trait impl

* reinstate metrics

* fix warnings

* remove chunks_cache

* oops

* actually store the available data

* mockable pruning interval

* fix tests

* spacing

* fix code grumbles

* guide improvements

* make time mockable

* implement a mocked clock for testing

* return DB errors

* Update node/core/av-store/Cargo.toml

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update roadmap/implementers-guide/src/node/utility/availability-store.md

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update roadmap/implementers-guide/src/node/utility/availability-store.md

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* review grumbles & clarity

* fix review grumbles

* Add docs

Co-authored-by: Andronik Ordian <write@reusable.software>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
 2021-01-11 17:46:09 +00:00
Robert Habermeier 9dee08af2f Batch messages to network bridge and introduce a timeout to SubsystemContext::send_message (#2197) 
* guide: batch network messages

* bridge: batch

* av-dist: batch outgoing messages

* time-out message sends in subsystem context

* Update node/subsystem/src/messages.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Revert "time-out message sends in subsystem context"

This reverts commit d49be62557ec37c8a350b93718acad723df704ef.

* Update node/network/availability-distribution/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
 2021-01-07 16:01:03 -05:00
Robert Habermeier 01d271caeb Fix statement distribution: forward statements to other peers. (#2146) 
* add candidate hash statement circulation span

* add relay-parent to hash-span

* Some typos and misspellings in docs I found, during my studies. (#2144)

* Fix stale link to overseer docs

* Some typos and mispellings in docs/comments

I found during studying how Polkadot works.

* Rococo V1 (#2141)

* Update to latest master and use 30 minutes sessions

* add bootnodes to chainspec

* Update Substrate

* Update chain-spec

* Update Cargo.lock

* GENESIS

* Change session length to one hour

* Bump spec_version to not fuck anything up ;)

Co-authored-by: Erin Grasmick <erin@parity.io>

* avoid creating duplicate unbacked spans when we see extra statements (#2145)

* improve jaeger spans for statement distribution

* tweak and add failing test for repropagation

* make a change that gets the test passing

* guide: clarify

* remove semicolon

Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Erin Grasmick <erin@parity.io>
 2020-12-20 18:52:07 -05:00
Robert Klotzner 9ebb9015d3 Some typos and misspellings in docs I found, during my studies. (#2144) 
* Fix stale link to overseer docs

* Some typos and mispellings in docs/comments

I found during studying how Polkadot works.
 2020-12-18 18:31:43 -05:00
Robert Habermeier 38276b08a4 Add candidate info to OccupiedCore (#2134) 
* guide: add candidate information to OccupiedCore

* add descriptor and hash to occupied core type

* guide: add candidate hash to inclusion

* runtime: return candidate info in core state

* bitfield signing: stop querying runtime as much

* minimize going to runtime in availability distribution

* fix availability distribution tests

* guide: remove para ID from Occupied core

* get all crates compiling
 2020-12-18 11:20:37 +03:00
Bastian Köcher d0c97539e4 Fix bug and further optimizations in availability distribution (#2104) 
* Fix bug and further optimizations in availability distribution

- There was a bug that resulted in only getting one candidate per block
as the candidates were put into the hashmap with the relay block hash as
key. The solution for this is to use the candidate hash and the relay
block hash as key.
- We stored received/sent messages with the candidate hash and chunk
index as key. The candidate hash wasn't required in this case, as the
messages are already stored per candidate.

* Update node/core/bitfield-signing/src/lib.rs

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* Remove the reverse map

* major refactor of receipts & query_live

* finish refactoring

remove ancestory mapping,

improve relay-parent cleanup & receipts-cache cleanup,
add descriptor to `PerCandidate`

* rename and rewrite query_pending_availability

* add a bunch of consistency tests

* Add some last changes

* xy

* fz

* Make it compile again

* Fix one test

* Fix logging

* Remove some buggy code

* Make tests work again

* Move stuff around

* Remove dbg

* Remove state from test_harness

* More refactor and new test

* New test and fixes

* Move metric

* Remove "duplicated code"

* Fix tests

* New test

* Change break to continue

* Update node/core/av-store/src/lib.rs

* Update node/core/av-store/src/lib.rs

* Update node/core/bitfield-signing/src/lib.rs

Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>

* update guide to match live_candidates changes

* add comment

* fix bitfield signing

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
 2020-12-17 19:09:17 +00:00
Andronik Ordian 3f5156e866 refactor View to include finalized_number (#2128) 
* refactor View to include finalized_number

* guide: update the NetworkBridge on BlockFinalized

* av-store: fix the tests

* actually fix tests

* grumbles

* ignore macro doctest

* use Hash::repeat_bytes more consistently

* broadcast empty leaves updates as well

* fix issuing view updates on empty leaves updates
 2020-12-17 12:50:58 -05:00
Andronik Ordian 1aa2554000 guide: minor fixes approval distribution (#2129) 
* guide: add missing ApprovalDistributionMessage variant

* guide: deduplicate sensence
 2020-12-17 15:02:04 +00:00
Robert Habermeier 15c253117d Add assignment keys to session keys, no separate approvals key (#2092) 
* guide: merge backing and approval keys

* bump substrate master & update primitives

* use new SessionInfo struct in session_info

* session keys upgrade for Polkadot

* kusama & westend runtimes

* bump westend, kusama, and polkadot versions

* add session key to rococo & test-runtime

* update prepare-test-net to latest subkey

* update chain specs to support new para_assignment session key

* get cargo.lock from master

* formatting

* update kill_storage based on substrate master

* fix test-service

* assgn -> asgn

* use session info module for assignment session key
 2020-12-10 20:30:27 -06:00
Robert Habermeier 48746afd6a add a brief description of goals of the GRANDPA voting rule (#2053) 2020-12-07 17:23:44 -06:00
Peter Goodspeed-Niklaus e7e9605f87 do not store backed candidates in the provisioner (#1909) 
* guide: non-semantic changes

* guide: update per the issue description

* GetBackedCandidates operates on multiple hashes now

* GetBackedCandidates still needs a relay parent

* implement changes specified in guide

* distinguish between various occasions for canceled oneshots

* add tracing info to getbackedcandidates

* REVERT ME: add tracing messages for GetBackedCandidates

Note that these messages are only sometimes actually passed on to the
candidate backing subsystem, with the consequence that it is
unexpectedly frequent that the provisioner fails to create its
provisionable data.

* REVERT ME: more tracing logging

* REVERT ME: log when CandidateBackingJob receives any message at all

* REVERT ME: log when send_msg sends a message to a job

* fix candidate-backing tests

* streamline GetBackedCandidates

This uses table.attested_candidate instead of table.get_candidate, because
it's not obvious how to get a BackedCandidate from just a
CommittedCandidateReceipt.

* REVERT ME: more logging tracing job lifespans

* promote warning about job premature demise

* don't terminate CandiateBackingJob::run_loop in event of failure to process message

* Revert "REVERT ME: more logging tracing job lifespans"

This reverts commit 7365f2fb3dec988d95cfcd317eba75587fe7fd16.

* Revert "REVERT ME: log when send_msg sends a message to a job"

This reverts commit 58e46aad038e6517d6d56390c8be65b046a21884.

* Revert "REVERT ME: log when CandidateBackingJob receives any message at all"

This reverts commit 0d6f38413c7c66b5e9e81dabc587906fa9f82656.

* Revert "REVERT ME: more tracing logging"

This reverts commit 675fd2628e84d1596965280e7314155ef21b28e6.

* Revert "REVERT ME: add tracing messages for GetBackedCandidates"

This reverts commit e09e156493430b33b6c8ab4b5cedb3f2f91afd51.

* formatting

* add logging message to CandidateBackingJob::run_loop start

* REVERT ME: add tracing to candidate-backing job creation

* run candidatebacking loop even if no assignment

* use unique error variants for each canceled oneshot

* Revert "REVERT ME: add tracing to candidate-backing job creation"

This reverts commit 8ce5f4f0bd7186dade134b118751480f72ea1fd6.

* try_runtime_api more to reduce silent exits

* add sanity check that returned backed candidates preserve ordering

* remove redundant err attribute
 2020-12-04 11:24:59 +01:00
Robert Habermeier 9071120de1 guide: Availability Recovery (#2011) 
* expand description

* basics of availability recovery

* finish availability recovery

* fill out missing text

* Update roadmap/implementers-guide/src/node/availability/availability-recovery.md

Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>

* fix signal handling

Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
 2020-12-01 14:58:31 -05:00
Robert Habermeier 0c84214814 change approval voting counting procedure (#1972) 
* change approval voting counting procedure

* language

* Update roadmap/implementers-guide/src/node/approval/approval-voting.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* improve language

* time-shifting

* tweak time-shifting

* expand

* typo

* tweaks to ensure we always get woken up

* move timing check into `tranches_to_approve`

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
 2020-11-28 18:35:46 +00:00
Robert Habermeier 51f6cb1979 Approval Distribution Subsystem (#1951) 
* skeleton flow control

* tweaks & rename to approvals distribution

* Update roadmap/implementers-guide/src/node/approval/approval-distribution.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Update roadmap/implementers-guide/src/node/approval/approval-distribution.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* add a `NewBlocks` message and dispatch

* new data format for approval distribution

* guide: update view to include finalized block number

* approvals: document view updating

* pruning when peers disconnect

* add remaining message types

* fix link

* network message type

* handle incoming assignments

* import_and_circulate_approval

* handle new blocks

* address review comments

* address review comments and use nifty VRFProof

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
 2020-11-24 23:40:07 -05:00
Robert Habermeier 5603e0f166 Session management for approval voting (#1973) 
* elaborate on runtime API

* clarify what to do if the runtime API calls fail

* Update roadmap/implementers-guide/src/node/approval/approval-voting.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Update roadmap/implementers-guide/src/node/approval/approval-voting.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
 2020-11-24 16:44:02 -05:00
Sergei Shulepov 2026228b77 Cleanups (#1933) 
* Introduce CollatorFn type alias

* Make test-runtime imports consistent with rococo-runtime

* Update node/primitives/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* fix warnings

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
 2020-11-09 14:21:05 +00:00
Bastian Köcher 8a2911b85d Distribute a PoV after seconding it (#1924) 
We need to distribute the PoV after we have seconded it. Other nodes
that will receive our `Secondded` statement and want to validate the
candidate another time will request this PoV from us.
 2020-11-06 08:38:44 +00:00
Bastian Köcher f2b286d62a Do not validate a candidate in candidate selection (#1912) 
* Do not validate a candidate in candidate selection

The candidate selection subsystem should not validate a candidate, as
this is done by the backing subsystem on a `Second` request. Otherwise
we validate one candidate twice.

* Update candidate-selection.md
 2020-11-04 22:06:43 -05:00
Robert Habermeier 1f4121c444 Runtime API for historical validation code (#1893) 
* fix: ensure candidate validation gets code based on occupied core assumption

* guide: runtime API for historical validation code

* add historical runtime API

* integrate into runtime API subsystem

* remove blocked TODO

* fix service build: enable notifications protocol only under real overseer

* Update node/subsystem/src/messages.rs

Co-authored-by: Sergei Shulepov <sergei@parity.io>

* fix compilation

Co-authored-by: Robert Habermeier <robert@Roberts-MacBook-Pro.local>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
 2020-11-02 12:19:53 -06:00
Robert Habermeier 43be64f2f7 Implementer's guide: Approval Voting Subsystem (#1691) 
* add storage for approvals module

* basics of approval logic

* fix links

* Session info module

* create an approvals_inherent module

* integrate approvals module with inclusion

* Remove approvals runtime

* tweak mentions of on-chain logic

* add note on finality-grandpa voting rule

* elaborate on node-side components

* stub for availability recovery

* add another note on voting rule

* Beginnings of approval subsystems

* flesh out approval voting now

* logic for checking assignment certs

* initial scheduler logic

* scheduler logic

* adjst tranche taking logic

* approval voting import

* approval work (voting side)

* amend some TODOs

* mark some TODOs

* describe `ApprovedAncestor`

* reference protocol-approval.md

* clarity on bitfield

* remove approvals_inherent

* tweak session_info module according to review

* formatting & nits

Co-authored-by: Robert Habermeier <robert@Roberts-MacBook-Pro.local>
 2020-10-30 22:07:52 -05:00
Sergei Shulepov e0336626ae let go fees (#1867) 2020-10-28 13:11:06 +01:00
Sergei Shulepov abb282dfd0 Runtime API for checking validation outputs (#1842) 
* annoying whitespaces

* update guide

Add `CheckValidationOutputs` runtime api and also change the
candidate-validation stuff

* promote ValidationOutputs to global primitives

i.e. move it from node specific primitives to global v1 primitives. This
will be needed when we share it later in the runtime inclusion module

* refactor acceptance checks in the inclusion module

factor out the common code to share it during the block inclusion and
for the forthcoming CheckValidationOutputs runtime api.

Also note that the acceptance criteria was updated to incorporate checks
that exist now in candidate-validation

* plumb the runtime api outside

* extract validation_data from ValidationOutputs

* use runtime-api to check validation outputs

apart from that refactor, update docs and tidy a bit

* Update the maxium code size

This is to fix a test that performs an upgrade.
 2020-10-24 01:48:36 -05:00
Bastian Köcher 2b798f8571 Use Pin<Box<>> for collation future (#1792) 2020-10-07 11:29:00 +00:00
Bastian Köcher 325c24580e Pass relay parent as argument when collating (#1789) 
This pr changes the collation function to also pass the current relay
parent the parachain block should be build on.
 2020-10-06 18:37:19 +00:00
Andronik Ordian d00bdfef08 chain-api subsystem: implement BlockHeader messsage (#1778) 
* chain-api subsystem: implement BlockHeader messsage

* update the guide
 2020-10-06 16:46:54 +02:00
Andronik Ordian ca89e3edbe NetworkBridge: validator (authorities) discovery api (#1699) 
* stupid, but it compiles

* redo

* cleanup

* add ValidatorDiscovery to msgs

* sketch network bridge code

* ConnectToAuthorities instead of validators

* more stuff

* cleanup

* more stuff

* complete ConnectToAuthoritiesState

* Update node/network/bridge/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Collator protocol subsystem (#1659)

* WIP

* The initial implementation of the collator side.

* Improve comments

* Multiple collation requests

* Add more tests and comments to validator side

* Add comments, remove dead code

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Fix build after suggested changes

* Also connect to the next validator group

* Remove a Future impl and move TimeoutExt to util

* Minor nits

* Fix build

* Change FetchCollations back to FetchCollation

* Try this

* Final fixes

* Fix build

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* handle multiple in-flight connection requests

* handle cancelled requests

* Update node/core/runtime-api/src/lib.rs

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* redo it again

* more stuff

* redo it again

* update comments

* workaround Future is not Send

* fix trailing spaces

* clarify comments

* bridge: fix compilation in tests

* update more comments

* small fixes

* port collator protocol to new validator discovery api

* collator tests compile

* collator tests pass

* do not revoke a request when the stream receiver is closed

* make revoking opt-in

* fix is_fulfilled

* handle request revokation in collator

* tests

* wait for validator connections asyncronously

* fix compilation

* relabel my todos

* apply Fedor's patch

* resolve reconnection TODO

* resolve revoking TODO

* resolve channel capacity TODO

* resolve peer cloning TODO

* resolve peer disconnected TODO

* resolve PeerSet TODO

* wip tests

* more tests

* resolve Arc TODO

* rename pending to non_revoked

* one more test

* extract utility function into util crate

* fix compilation in tests

* Apply suggestions from code review

Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>

* revert pin_project removal

* fix while let loop

* Revert "revert pin_project removal"

This reverts commit ae7f529d8de982ef66c3007dd1ff74c6ddce80d2.

* fix compilation

* Update node/subsystem/src/messages.rs

* docs on pub items

* guide updates

* remove a TODO

* small guide update

* fix a typo

* link to the issue

* validator discovery: on_request docs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
 2020-10-06 13:34:57 +02:00
Bastian Köcher a4662104db Make collation an optional return (#1787) 
This pr changes the collator interface function to return an optional
collation instead of a collation. This is required as the parachain
itself can fail to generate a valid collation for various reason. Now if
the collation fails it will return `None`.

Besides that the pr adds some `RuntimeDebug` derive for `ValidationData`
and removes some whitespaces.
 2020-10-06 11:57:10 +02:00
Sergei Shulepov 796de5f3e3 collation-generation: guide and tidying (#1753) 
* Guide: Change for {Global,Local}ValidationData to ValidationData

Seems like that was missed during the transition from notions of global and local validation data to persistent and transient

* collation-generation: a small drive-by fix of line length

* collation-generation: extract an error module

this allows us to avoid piling Error and Result types from different contexts, specifically std vs. module-local

* collation-generation: extract LOG_TARGET from log statements
 2020-09-25 16:33:10 +00:00
Peter Goodspeed-Niklaus d1b1c17285 implement candidate selection subsystem (#1645) 
* choose the straightforward candidate selection algorithm for now

* add draft implementation of candidate selection

* fix typo in summary

* more properly report misbehaving collators

* describe how CandidateSelection subsystem becomes aware of candidates

* revise candidate selection / collator protocol interaction pattern

* implement rest of candidate selection per the guide

* review: resolve nits

* start writing test suite, harness

* implement first test

* add second test

* implement third test

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
 2020-09-08 09:48:48 +00:00
Robert Habermeier d246a2889e Extract Assignments and Approval guide text to an Approval informational section (#1638) 
* move validity module to disputes module

* prepare a section on approvals

* extract approval/assignments text to an overview section

* Apply suggestions from code review

Spelling

Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
 2020-08-31 11:57:51 +03:00
Jeff Burdges 46b895e869 Initial guide text for approvals and especially approvals assignments (#1518) 
* Initial guide modifications for approvals

* Split approval assignments keys and approval votes keys

This should avoid the political problems with validator operators
wnting everything to be a remote signer.

* Update roadmap/implementers-guide/src/runtime/validity.md

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* Update roadmap/implementers-guide/src/node/validity/assignments.md

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* Update roadmap/implementers-guide/src/node/validity/assignments.md

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* Update roadmap/implementers-guide/src/node/validity/assignments.md

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* Update roadmap/implementers-guide/src/node/validity/assignments.md

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* Update roadmap/implementers-guide/src/node/validity/assignments.md

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* Update roadmap/implementers-guide/src/node/validity/assignments.md

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* Update roadmap/implementers-guide/src/node/validity/assignments.md

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* Update roadmap/implementers-guide/src/node/validity/approvals.md

Co-authored-by: hndnklnc <handankilinc1@gmail.com>

* Update roadmap/implementers-guide/src/node/validity/approvals.md

Co-authored-by: hndnklnc <handankilinc1@gmail.com>

* Update roadmap/implementers-guide/src/runtime/validity.md

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* Not sure where these go..

* Update roadmap/implementers-guide/src/node/validity/approvals.md

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* Local clock leniency

* Explain HSM blabla

* double word

* Clarify okasing things on-chain

* Assignments integration with relative time

* minor tweaks

* Clarify structure and work savings

* Use included more

* Cleanup and paramaters

See https://github.com/w3f/research-internal/issues/515

* More on GRANDPA

* Explain relay equivocation relevance

* Address postponement and on-chain verification somewhat

* small bit more explination

* just use ed25519 here: could -> should

* spelling

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: hndnklnc <handankilinc1@gmail.com>
 2020-08-18 15:14:55 +02:00
Robert Habermeier 262574fc49 Implement validation data refactor (#1585) 
* update primitives

* correct parent_head field

* make hrmp field pub

* refactor validation data: runtime

* refactor validation data: messages

* add arguments to full_validation_data runtime API

* port runtime API

* mostly port over candidate validation

* remove some parameters from ValidationParams

* guide: update candidate validation

* update candidate outputs

* update ValidationOutputs in primitives

* port over candidate validation

* add a new test for no-transient behavior

* update util runtime API wrappers

* candidate backing

* fix missing imports

* change some fields of validation data around

* runtime API impl

* update candidate validation

* fix backing tests

* grumbles from review

* fix av-store tests

* fix some more crates

* fix provisioner tests

* fix availability distribution tests

* port collation-generation to new validation data

* fix overseer tests

* Update roadmap/implementers-guide/src/node/utility/candidate-validation.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
 2020-08-18 14:41:40 +02:00
Peter Goodspeed-Niklaus 54bec3bfc0 implement collation generation subsystem (#1557) 
* start sketching out a collation generation subsystem

* invent a basic strategy for double initialization

* clean up warnings

* impl util requests from runtime assuming a context instead of a FromJob sender

* implement collation generation algorithm from guide

* update AllMessages in tests

* fix trivial review comments

* remove another redundant declaration from merge

* filter availability cores by para_id

* handle new activations each in their own async task

* update guide according to the actual current implementation

* add initialization to guide

* add general-purpose subsystem_test_harness helper

* write first handle_new_activations test

* add test that handle_new_activations filters local_validation_data requests

* add (failing) test of collation distribution message sending

* rustfmt

* broken: work on fixing sender test

Unfortunately, for reasons that are not yet clear, despite the public key
and checked data being identical, the signer is not producing an identical
signature. This commit produces this output (among more):

signing with  Public(c4733ab0bbe3ba4c096685d1737a7f498cdbdd167a767d04a21dc7df12b8c858 (5GWHUNm5...))
checking with Public(c4733ab0bbe3ba4c096685d1737a7f498cdbdd167a767d04a21dc7df12b8c858 (5GWHUNm5...))
signed payload:  [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 10, 0, 0, 0, c7, e5, c0, 64, 7a, db, fe, 44, 81, e5, 51, 11, 79, 9f, a5, 63, 93, 94, 3c, c4, 36, c6, 30, 36, c2, c5, 44, a2, 1b, db, b7, 82, 3, 17, a, 2e, 75, 97, b7, b7, e3, d8, 4c, 5, 39, 1d, 13, 9a, 62, b1, 57, e7, 87, 86, d8, c0, 82, f2, 9d, cf, 4c, 11, 13, 14]
checked payload: [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 10, 0, 0, 0, c7, e5, c0, 64, 7a, db, fe, 44, 81, e5, 51, 11, 79, 9f, a5, 63, 93, 94, 3c, c4, 36, c6, 30, 36, c2, c5, 44, a2, 1b, db, b7, 82, 3, 17, a, 2e, 75, 97, b7, b7, e3, d8, 4c, 5, 39, 1d, 13, 9a, 62, b1, 57, e7, 87, 86, d8, c0, 82, f2, 9d, cf, 4c, 11, 13, 14]

* fix broken test

* collation function returns commitments hash

It doesn't look like we use the actual commitments data anywhere, and
it's not obvious if there are any fields of `CandidateCommitments`
not available to the collator, so this commit just assigns them the
entire responsibility of generating the hash.

* add missing overseer impls

* calculating erasure coding is polkadot's responsibility, not cumulus

* concurrentize per-relay_parent requests
 2020-08-17 14:27:37 +02:00
Robert Habermeier 2a96c19370 guide: validation data refactoring (#1576) 
* guide: validation data refactoring

* address grumbles from review

* Update roadmap/implementers-guide/src/types/candidate.md

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* last comments from review

Co-authored-by: Sergei Shulepov <sergei@parity.io>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
 2020-08-13 14:22:13 +00:00
Bernhard Schuster 4bdfd02f93 impl availability distribution 
Closes #1237
 2020-08-10 15:02:30 +02:00
Peter Goodspeed-Niklaus 21cec309a4 implement provisioner (#1473) 
* sketch out provisioner basics

* handle provisionable data

* stub out select_inherent_data

* split runtime APIs into sub-chapters to improve linkability

* explain SignedAvailabilityBitfield semantics

* add internal link to further documentation

* some more work figuring out how the provisioner can do its thing

* fix broken link

* don't import enum variants where it's one layer deep

* make request_availability_cores a free fn in util

* document more precisely what should happen on block production

* finish first-draft implementation of provisioner

* start working on the full and proper backed candidate selection rule

* Pass number of block under construction via RequestInherentData

* Revert "Pass number of block under construction via RequestInherentData"

This reverts commit 850fe62cc0dfb04252580c21a985962000e693c8.

That initially looked like the better approach--it spent the time
budget for fetching the block number in the proposer, instead of
the provisioner, and that felt more appropriate--but it turns out
not to be obvious how to get the block number of the block under
construction from within the proposer. The Chain API may be less
ideal, but it should be easier to implement.

* wip: get the block under production from the Chain API

* add ChainApiMessage to AllMessages

* don't break the run loop if a provisionable data channel closes

* clone only those backed candidates which are coherent

* propagate chain_api subsystem through various locations

* add delegated_subsystem! macro to ease delegating subsystems

Unfortunately, it doesn't work right:

```
error[E0446]: private type `CandidateBackingJob` in public interface
   --> node/core/backing/src/lib.rs:775:1
    |
86  | struct CandidateBackingJob {
    | - `CandidateBackingJob` declared as private
...
775 | delegated_subsystem!(CandidateBackingJob as CandidateBackingSubsystem);
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ can't leak private type
```

I'm not sure precisely what's going wrong, here; I suspect the problem is
the use of `$job as JobTrait>::RunArgs` and `::ToJob`; the failure would be
that it's not reifying the types to verify that the actual types are public,
but instead referring to them via `CandidateBackingJob`, which is in fact private;
that privacy is the point.

Going to see if I can generic my way out of this, but we may be headed for a
quick revert here.

* fix delegated_subsystem

The invocation is a bit more verbose than I'd prefer, but it's also
more explicit about what types need to be public. I'll take it as a win.

* add provisioning subsystem; reduce public interface of provisioner

* deny missing docs in provisioner

* refactor core selection per code review suggestion

This is twice as much code when measured by line, but IMO it is
in fact somewhat clearer to read, so overall a win.

Also adds an improved rule for selecting availability bitfields,
which (unlike the previous implementation) guarantees that the
appropriate postconditions hold there.

* fix bad merge double-declaration

* update guide with (hopefully) complete provisioner candidate selection procedure

* clarify candidate selection algorithm

* Revert "clarify candidate selection algorithm"

This reverts commit c68a02ac9cf42b3a4a28eb197d38633a40d0e3e6.

* clarify candidate selection algorithm

* update provisioner to implement candidate selection per the guide

* add test that no more than one bitfield is selected per validator

* add test that each selected bitfield corresponds to an occupied core

* add test that more set bits win conflicts

* add macro for specializing runtime requests; specailize all runtime requests

* add tests harness for select_candidates tests

* add first real select_candidates test, fix test_harness

* add mock overseer and test that success is possible

* add test that the candidate selection algorithm picks the right ones

* make candidate selection test somewhat more stringent
 2020-08-06 18:12:15 +02:00
Robert Habermeier f554868cd4 Network Bridge Refactoring (#1535) 
* rename protocol ID to network capability

* guide: `ProtocolId` -> `NetworkCapability`

* guide: remove `RegisterEventProducer`

* capabilities and expand on underlying network assumptions

* guide: create network.md types file

* guide: network bridge is aware of network messages

* revert changes in code

* Update roadmap/implementers-guide/src/SUMMARY.md

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* remove references to NetworkCapability

* Update roadmap/implementers-guide/src/types/network.md

Co-authored-by: Sergei Shulepov <sergei@parity.io>

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
 2020-08-05 09:04:05 +00:00
Andronik Ordian 596ce826e8 Chain API subsystem (#1498) 
* chain-api subsystem skeleton

* chain-api subsystem: draft impl

* chain-api subsystem: mock testclient

* chain-api subsystem: impl HeaderBacked for TestClient

* chain-api subsystem: impl basic tests

* chain-api subsystem: tiny guide

* chain-api subsystem: rename ChainApiRequestMessage to ChainApiMessage

* chain-api subsystem: add the page to the ToC

* chain-api subsystem: proper error type

* chain-api subsystem: impl ancestors request

* chain-api subsystem: tests for ancestors request

* guide: fix ancestor return type

* runtime-api subsystem: remove unused dep

* fix fmt

* fix outdated comment

* chain-api subsystem: s/format/to_string

* lower-case subsystem names

* chain-api subsystem: resolve Finalized todo

* chain-api subsystem: remove TODO

* extract request errors into a module

* remove caching TODO

* fix imports
 2020-07-31 16:02:16 +00:00
Robert Habermeier 490f80d8c7 guide: collator networking & subsystems (#1452) 
* Do a small write-up on collation-generation

* preamble to collator protocol

* notes on protocol

* collation-generation: point to collator protocol

* fix missing bracket

* expand on collator protocol wire protocol

* add a couple more sentences

* expand on requests some more

* go higher level

* network bridge: note peerset

* note peer-set = validation for protocols

* add `ConnectToValidators` message

* use ConnectToValidators in collator protocol

* typo

* remove references to sentry nodes
 2020-07-31 11:07:31 -04:00
Robert Habermeier 5253a05555 Candidate Validation Subsystem (#1432) 
* skeleton for candidate-validation

* add to workspace

* implement candidate validation logic

* guide: note occupied-core assumption for candidate validation

* adjust message doc

* wire together `run` asynchronously

* add a Subsystem implementation

* clean up a couple warnings

* fix compilation errors due to merge

* improve candidate-validation.md

* remove old reference to subsystem-test helpers crate

* update Cargo.lock

* add a couple new Runtime API methods

* add a candidate validation message

* fetch validation data from the chain state

* some tests for assumption checking

* make spawn_validate_exhaustive mockable

* more tests on the error handling side

* fix all other grumbles except for wasm validation API change

* wrap a SpawnNamed in candidate-validation

* warn

* amend guide

* squanch warning

* remove duplicate after merge
 2020-07-30 21:50:11 +00:00
Fedor Sakharov 824c6245e2 Availability store subsystem guide (#1424) 
* Improve AVStore and Scheduler docs

* Update roadmap/implementers-guide/src/node/utility/availability-store.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Bug in linking to README.md

* Update against new runtime apis

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
 2020-07-27 18:12:31 +03:00
Peter Goodspeed-Niklaus ba4bfa4dd0 implement bitfield signing subsystem (#1364) 
* update guide to reduce confusion and TODOs

* work from previous bitfield signing effort

There were large merge issues with the old bitfield signing PR, so
we're just copying all the work from that onto this and restarting.

Much of the existing work will be discarded because we now have better
tools available, but that's fine.

* start rewriting bitfield signing in terms of the util module

* implement construct_availability_bitvec

It's not an ideal implementation--we can make it much more concurrent--
but at least it compiles.

* implement the unimplemented portions of bitfield signing

* get core availability concurrently, not sequentially

* use sp-std instead of std for a parachain item

* resolve type inference failure caused by multiple From impls

* handle bitfield signing subsystem & Allmessages variant in overseer

* fix more multi-From inference issues

* more concisely handle overflow

Co-authored-by: Andronik Ordian <write@reusable.software>

* Revert "resolve type inference failure caused by multiple From impls"

This reverts commit 7fc77805de5e5074a1b01037f8d4e3919e03e0e1.

* Revert "fix more multi-From inference issues"

This reverts commit f14ffe589e20d664d8a900ed62f68b6fb844a514.

* impl From<i32> for ParaId

* handle another instance of AllSubsystems

* improve consistency when returning existing options

Co-authored-by: Andronik Ordian <write@reusable.software>
 2020-07-23 16:05:48 +02:00
Bernhard Schuster a1c704d446 implement bitfield distribution subsystem (#1368) 
* feat bitfield distribution

* feat bitfield distribution part 2

* pair programming with rustc & cargo

* lets go

* move bitfield-distribution to the node/network folder

* shape shifting

* lunchtime

* ignore the two fn recursion for now

* step by step

* triplesteps

* bandaid commit

* unordered futures magic

* chore

* reword markdown

* clarify

* lacks abortable processing impl details

* slimify

* fix: warnings and avoid ctx.clone() improve comments

* review comments

* fix details

* make sure outgoing messages are tracked

* fix name

* fix subsystem

* partial test impl

* relax context bounds

* test

* X

* X

* initial test

* fix relay_message not tracked when origin is self

* fix/guide: grammar

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* work around missing Eq+PartialEq

* fix: add missing message to provisioner

* unify per_job to job_data

* fix/review: part one

* fix/review: more grumbles

* fix/review: track incoming messages per peer

* fix/review: extract fn, avoid nested matches

* fix/review: more tests, simplify test

* fix/review: extend tests to cover more cases

* chore/rename: Tracker -> ProtocolState

* chore check and comment rewording

* feat test: invalid peer message

* remove ignored test cases and unused macros

* fix master merge fallout + warnings

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
 2020-07-23 15:46:22 +03:00
Robert Habermeier 5be366d6dc Replace StartWork and StopWork with an ActiveLeavesUpdate signal (#1431) 2020-07-22 12:12:49 -04:00
Robert Habermeier f070202d4c add assumptions made by the Node-side (#1408) 
* add assumptions made by the Node-side

* Update roadmap/implementers-guide/src/node/README.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Update roadmap/implementers-guide/src/node/README.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Update roadmap/implementers-guide/src/node/README.md

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* more refinements

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
 2020-07-15 20:02:18 -04:00