Commit Graph
100 Commits
Author SHA1 Message Date
Robert Habermeier 151d73af5b Implement PoV Distribution Subsystem (#1344)
* introduce candidatedescriptor type

* add PoVDistribution message type

* loosen bound on PoV Distribution to account for equivocations

* re-export some types from the messages module

* begin PoV Distribution subsystem

* remove redundant index from PoV distribution

* define state machine for pov distribution

* handle overseer signals

* set up control flow

* remove `ValidatorStatement` section

* implement PoV fetching

* implement distribution logic

* add missing `

* implement some network bridge event handlers

* stub for message processing, handle our view change

* control flow for handling messages

* handle `awaiting` message

* handle any incoming PoVs and redistribute

* actually provide a subsystem implementation

* remove set-builder notation

* begin testing PoV distribution

* test that we send awaiting messages only to peers with same view

* ensure we distribute awaited PoVs to peers on view changes

* test that peers can complete fetch and are rewarded

* test some reporting logic

* ensure peer is reported for flooding

* test punishing peers diverging from awaited protocol

* test that we eagerly complete peers' awaited PoVs based on what we receive

* test that we prune the awaited set after receiving

* expand pov-distribution in guide to match a change I made

* remove unneeded import
2020-07-08 18:15:39 -04:00
Robert Habermeier 42bd096413 I am dumb and can't spell (#1366)
* rename implementor's guide to implementer's guide

* fix typos in more places
2020-07-07 10:10:36 -04:00
Robert Habermeier ac8e1ca206 Implement the Statement Distribution Subsystem (#1326)
* set up data types and control flow for statement distribution

* add some set-like methods to View

* implement sending to peers

* start fixing equivocation handling

* Add a section to the statement distribution subsystem on equivocations and flood protection

* fix typo and amend wording

* implement flood protection

* have peer knowledge tracker follow when peer first learns about a candidate

* send dependents after circulating

* add another TODO

* trigger send in one more place

* refactors from review

* send new statements to candidate backing

* instantiate active head data with runtime API values

* track our view changes and peer view changes

* apply a benefit to peers who send us statements we want

* remove unneeded TODO

* add some comments and improve Hash implementation

* start tests and fix `note_statement`

* test active_head seconding logic

* test that the per-peer tracking logic works

* test per-peer knowledge tracker

* test that peer view updates lead to messages being sent

* test statement circulation

* address review comments

* have view set methods return references
2020-07-06 11:16:17 -04:00
Robert Habermeier 2a3e607d14 Subsystem::start takes self by-value (#1325)
* Subsystem::start takes self by-value

* fix doc-test compilation
2020-06-30 15:16:37 -04:00
Robert HabermeierandPeter Goodspeed-Niklaus d16e7485d4 Implement Network Bridge (#1280)
* network bridge skeleton

* move some primitives around and add debug impls

* protocol registration glue & abstract network interface

* add send_msgs to subsystemctx

* select logic

* transform different events into actions and handle

* implement remaining network bridge state machine

* start test skeleton

* make network methods asynchronous

* extract subsystem out to subsystem crate

* port over overseer to subsystem context trait

* fix minimal example

* fix overseer doc test

* update network-bridge crate

* write a subsystem test-helpers crate

* write a network test helper for network-bridge

* set up (broken) view test

* Revamp network to be more async-friendly and not require Sync

* fix spacing

* fix test compilation

* insert side-channel for actions

* Add some more message types to AllMessages

* introduce a test harness

* add some tests

* ensure service compiles and passes tests

* fix typo

* fix service-new compilation

* Subsystem test helpers send messages synchronously

* remove smelly action inspector

* remove superfluous let binding

* fix warnings

* Update node/network/bridge/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* fix compilation

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
2020-06-30 11:21:40 -04:00
Robert Habermeier 424d61818e Expand PoV Distribution section and improve candidate types (#1294)
* rejig candidate types a bit

* adjust doc

* describe basic PoV distribution

* replace some straggling html links

* address grumbles from review and fix links
2020-06-22 15:01:38 -04:00
Robert Habermeier 879892d3f9 Inclusion Module (#1242)
* add availability bitfield types to primitives

* begin inclusion module

* use GitHub issue link for limitation

* fix some compiler errors

* integrate validators into initializer

* add generic signing context

* make signing-context more generic

* fix issues with inclusion module

* add TODO

* guide: add validators and session index to inclusion

* guide: add session index to change notification

* implement session change logic

* add BackedCandidate type

* guide: refine inclusion pipeline

* guide: rename group_on to group_validators

* guide: add check about collator for parathread

* guide: add last_code_upgrade to paras and use in inclusion

* implement Paras::last_code_upgrade

* implement most checks in process_candidates

* make candidate receipt structs more generic

* make BackedCandidate struct more generic

* use hash param, not block number

* check that candidate is in context of the parent block

* include inclusion module in initializer

* implement enact-candidate

* check that only occupied cores have bits set

* finish implementing bitfield processing

* restructure consistency checks on candidates

* make some more primitives generic

* signature checking logic for backed candidates

* finish implementing process_candidates

* implement collect_pending

* add some trait implementations to primitives

* implement InclusionInherent and squash warnings

* test bitfield signing checks

* rename parachain head to para_head

* fix note_new_head bug in paras

* test bitfield enactment in inclusion

* helpers for candidate checks

* add test for most candidate checks

* add test for backing setting storage

* test session change logic

* remove extraneous type parameter

* remove some allow(unused)s

* extract threshold computation to const fn

* remove some more allow(unused)s

* improve doc

* add debug assertion

* fix primitive test compilation

* tag unanimous variant as unused
2020-06-18 19:38:07 -04:00
c226c4403d Add some message types from subsystem definitions (#1265)
* introduce polkadot-node-primitives

* guide: change statement distribution message types

* guide: remove variant from `CandidateSelectionMessage`

* add a few more message types

* add TODOs

* Almost all messages

* NewBackedCandidate notification

* Formatting

* Use AttestedCandidate as BackedCandidate

* Update node/primitives/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Fix the tests

* Bring in types from #1242

* Adds network bridge messages

* More message types from doc

* use fn pointer type

* Fixes from the review

* Add missing Runtime subsystem message

* rename to CandidateValidationMessage and fix tests

Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
2020-06-17 19:49:17 -04:00
Robert HabermeierandPeter Goodspeed-Niklaus 87ae6e42f5 Various Guide Improvements (#1270)
* types chapter & candidate types

* Factor out some message types

* backing and availability types.

* spacing

* finish thought

* info on signed payloads

* update links

* explicit wrapper types

* add remaining message types

* add runtime API subsystem

* adjust language

* tweak candidate validation message

* find homes for some misfit types

* embed abridged within full candidate receipt

* Update roadmap/implementors-guide/src/SUMMARY.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Update roadmap/implementors-guide/src/SUMMARY.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* adjust text on candidate selection message to match

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
2020-06-17 15:09:43 -04:00
04c8603042 Scheduler Module (#1162)
* scheduler module skeleton

* update scheduler skeleton to match latest version of guide

* better session change notification

* add mock randomness and fix test compilation

* shuffle validators into groups

* finish implementing session change logic for scheduler

* tweak core assignment type to track retries of parathread

* reframe queued parathread core as offset

* implement initialzation and finalization routines

* implement parathread claim queuing

* implement core_para

* implement the group_validators routine and fix errors

* add a reason for freeing cores

* implement `schedule` function

* add some docs to the scheduled function

* implement `occupied` helper

* implement availability predicate

* fix some warnings

* integrate scheduler into initializer

* integrate scheduler into mock module

* avoid conflict with Substrate's scheduler storage

* add parathreads index to paras module

* implement parathreads map in paras module

* add is_parathread to paras

* test adding parathread claim

* test that you cannot add claims when no parathread cores exist

* check session change parathread queue pruning

* test validator shuffling

* add allow_unused to scheduler items

* add test for scheduling

* add some more tests for scheduling logic

* test core rotation

* check parathread claim pruning after retries

* add bound notes

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* more suggestions from review

* test availability predicate, add box to please compiler

* add changes to guide

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
2020-06-11 15:15:30 -04:00
Robert HabermeierandMax Inden b925981971 Utility subsystem for actually connecting to network (#1205)
* define subsystems skeleton

* Create a blank section for each subsystem in the initial batch

* write out basics for most subsystems

* define network bridge subsystem

* integrate network bridge into network systems

* Apply suggestions from code review

some nits

Co-authored-by: Max Inden <mail@max-inden.de>

* clean up wording

Co-authored-by: Max Inden <mail@max-inden.de>
2020-06-10 13:31:29 -04:00
Robert HabermeierandPeter Goodspeed-Niklaus 88161da402 Write out stubs for most backing and availability subsystems (#1199)
* define subsystems skeleton

* Create a blank section for each subsystem in the initial batch

* write out basics for most subsystems

* Update roadmap/implementors-guide/guide.md

clean up language

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* ToC indentation

* clean up wording

* renaming block authorship (provisioning) to Provisioner

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
2020-06-10 11:24:37 -04:00
Robert Habermeier 2ff55ca4b6 Add exception to overseer communication and write up local and remote disputes (#1176)
* clarify exception to communication rule for responses

* add note about outside world

* write up my understanding of local and remote disputes

* clean up secondary checker language

* amend to match Jeff's notes
2020-06-05 13:55:16 -04:00
Robert Habermeier 9d5eae6ea3 establish new node folder for overseer, messages, and subsystems (#1200)
* establish new `node` folder for overseer, messages, and subsystems

* extract message types from overseer crate

* remove doc links
2020-06-05 10:48:35 -04:00
Robert Habermeier 593fa2b0c1 note pruning of orphan blocks in the overseer description (#1189)
* note pruning of orphan blocks in the overseer description

* remove note about finalization events being maybe not needed
2020-06-02 16:19:53 -04:00
Robert Habermeier bd2304ec98 New parachain runtime skeleton (#1158)
* file structure and initializer skeleton

* ensure session changes happen before initialization

* add a couple tests for initializer flow

* integrate with session handling

* configuration update logic

* configuration methods

* move test mock to its own module

* integrate configuration into initializer

* add note about initialization order

* integrate configuration module into mock

* add some tests for config module

* paras module storage

* implement paras session change operation

* amend past code pruning to fully cover acceptance period

* update guide again

* do pruning of historical validation code

* add weight to initialization

* integrate into mock & leave notes for next session

* clean up un-ended sentence

* alter test to account for double index in past code meta

* port over code-at logic test

* clarify checking for conflicting code upgrades

* add genesis for paras, include in mock, ensure incoming paras are processed

* note on return value of `validation_code_at`

* implement paras routines from implementor's guide

* bring over some existing tests and begin porting

* port over code upgrade tests

* test parachain registration

* test code_at with intermediate block

* fix warnings

* clean up docs and extract to separate struct

* adjust implementor's guide to include replacementtimes

* kill stray println

* rename expected_at to applied_after

* rewrite ParaPastCodeMeta to avoid reversal

* clarify and test interface of validation_code_at

* make FutureCode optional

* rename do_old_code_pruning

* add comment on Option<()> to answer FAQ

* address some more grumbles
2020-06-02 12:34:07 -04:00
Robert Habermeier 03576b3707 Remove old runtime requirements section, reorganize, and improve ToC (#1180)
* remove old runtime requirements section

* reorganize guide and improve ToC
2020-06-01 17:52:42 -04:00
Robert Habermeier a3e4893a7c Improve the implementor's guide section on the Scheduler module (#1168)
* remove redundant resize instruction from guide

* amend past code pruning to fully cover acceptance period

* update guide again

* note on return value of `validation_code_at`

* rename execution cores to availability cores

* clarify terminology

* add ReplacementTimes to the guide

* fully decouple inclusion and scheduler

* clarify parathread scheduling substantially

* clarify validator group selection
2020-05-29 15:52:59 +02:00
Robert Habermeier 1a3557a597 clarify some things about subsystems (#1157) 2020-05-28 14:15:07 +03:00
Robert Habermeier cbb412f515 include implementor's guide in this repo (#1137)
* include implementor's guide in this repo

* most updated guide
2020-05-25 19:11:36 -04:00
Robert Habermeier a5034dbe98 Use strong types in runtime for parachain heads and validation code (#964)
* use stronger types for HeadData and ValidationCode in runtime

* fix weird debug compile error

* fix runtime build

* update invocations invalidation.rs

* fix tests
2020-04-13 10:24:25 +02:00
Robert Habermeier 10cec3b591 Upgradeable validation functions (#918)
* upgrade primitives to allow changing validation function

* set up storage schema for old parachains code

* fix compilation errors

* fix test compilation

* add some tests for past code meta

* most of the runtime logic for code upgrades

* implement old-code pruning

* add a couple tests

* clean up remaining TODOs

* add a whole bunch of tests for runtime functionality

* remove unused function

* fix runtime compilation

* extract some primitives to parachain crate

* add validation-code upgrades to validation params and result

* extend validation params with code upgrade fields

* provide maximums to validation params

* port test-parachains

* add a code-upgrader test-parachain and tests

* fix collator tests

* move test-parachains to own folder to work around compilation errors

* fix test compilation

* update the Cargo.lock

* fix parachains tests

* remove dbg! invocation

* use new pool in code-upgrader

* bump lockfile

* link TODO to issue
2020-04-06 10:43:19 -04:00
Robert Habermeier 15a83079ba refactor out validation hosts to pool struct (#972)
* refactor out validation hosts to pool struct

* make web-wasm compatible

* typo

* remove now-unused static hosts
2020-04-04 15:15:54 -04:00
Robert Habermeier 0252e89a8c Introduce a skill-tree for remaining tasks in phase-1 (#961)
* add dummy parachains.toml

* flesh out parachains.toml

* finish phase-1 rendering

* render to svg instead

* put graphviz svg through sanitizer so github can render

* return to PNG
2020-04-02 11:22:33 -04:00
Robert Habermeier cb5defc91d PoV-block gossip (#930)
* add pov-block gossip message to network

* tests for pov-block gossip

* integrate pov-block gossip into main protocol

* message validation fetches pov blocks

* remove stray dbg! invocation

* test that pov-block is fetched from relay-parent topic
2020-03-31 23:05:42 +02:00
Robert Habermeier 21bd2f19d3 use strong key in parachains module KeyOwnerProofSystem (#951) 2020-03-31 11:47:49 -04:00
Robert Habermeier 260b2fa336 Tests for new network code (#897)
* move protocol.rs into subfolder

* add trait for mocking network behavior

* add a mock version of network ops

* remove some redundant parameters from service messages

* ensure fetching erasure chunks automatically cancels

* introduce dummy ProvideRuntimeApi

* abstract over gossip somewhat

* add mock gossip handler

* skeleton test

* remove dependence of shared table on router

* remove worker dependence on its own sender

* test shutdown

* add tests

* test that gossip streams are cleaned up correctly

* refactor worker out into its own struct and reduce bound on executor

* remove reliance of tests on global thread pool
2020-03-16 11:17:08 +01:00
Robert Habermeier 7931380825 Remove legacy network code (#860)
* expunge legacy code from polkadot-network

* mostly rip out old legacy protocol from service

* ensure validation work is spawned by incoming messages

* decouple availabliity store from network logic; clean up data flow

* av_store: test helpers and use futures-abort

* update polkadot-validation to pass n_validators when submitting chunks

* fallible erasure-chunk fetching

* implement `ErasureNetworking` for new network prot

* API for registering availability store in network

* fully integrate new network service into service

* fix validation tests

* scaffolding for porting collator over to new network

* track connected validators' peer IDs and distribute collators' collations

* helper in network for fetching all checked statements

* fix adder-collator

* actually register notifications protocol

* Update service/src/lib.rs

* merge with master
2020-03-05 10:11:21 -08:00
Robert Habermeier b7d30aa379 A more comprehensive model for PoV-Blocks and Candidate receipts (#843)
* encode the candidate statement as only the hash

* refactor CandidateReceipt and CollationInfo

* introduce an abridged candidate receipt type

* erasure coding stores candidate receipt

* store omitted data instead and introduce AvailableData type

* refactor availability-store schema

* tweak schema and APIs a bit more

* get availability-store tests passing

* accept AbridgedCandidateReceipt in `set_heads`

* change statement type in primitives to be hash-only

* fix parachains runtime tests

* fix bad merge

* rewrite validation pipeline

* remove evaluation module

* use abridged candidate hash as canonical

* statement table uses abridged candidate receipts

* kill availability_store::Data struct

* port shared table to new validation pipelines

* extract full validation pipeline to helper

* remove old validation pipeline from collation module

* polkadot-validation compiles

* polkadot-validation tests compile

* make local collation available in validation service

* port legacy network code

* polkadot-network fully ported

* network: ensure fresh statement is propagated

* remove pov_block_hash from LocalValidationData

* remove candidate_hash field from AttestedCandidate and update runtime

* port runtimes to new ParachainHost definition

* port over polkadot-collator

* fix test compilation

* better fix

* remove unrelated validation work dispatch fix

* address grumbles

* fix equality check
2020-02-25 15:16:58 -08:00
Robert Habermeier 295151338d Introduce a maximum code size and head data size (#835)
* add a maximum code size and head data size

* get existing tests passing

* add tests for slots logic

* test registrar behavior

* introduce maximums and bump versions

* address review grumbles

* work around publicizing derive

* remove unneeded and wrong doc
2020-02-11 19:21:56 +01:00
Robert Habermeier 9b23f3f1f0 rewrite network code to use notifications_protocol APIs from Substrate (#788)
* extract all network code to legacy submodule

* update references to legacy proto

* skeleton of futures-based protocol

* refactor skeleton to use background task

* rename communication_for to build_table_router

* implement internal message types for validation network

* basic ParachainNetwork and TableRouter implementations

* add some module docs

* remove exit-future from validation

* hack: adapt legacy protocol to lack of exit-future

* generalize RegisteredMessageValidator somewhat

* instantiate and teardown table routers

* clean up RouterInner drop logic

* implement most of the statement import loop

* implement statement loop in async/await

* remove unneeded TODO

* most of the collation skeleton

* send session keys and validator roles

* also send role after status

* use config in startup

* point TODO to issue

* fix test compilation
2020-02-10 15:20:45 +01:00
Robert Habermeier 27bf34522d enable migrations in polkadot and kusama runtimes (#824)
* enable migrations in polkadot and kusama runtimes

* bump runtime version
2020-02-10 10:31:50 +01:00
Robert Habermeier 1c2aff5b4d Include parent head in CandidateReceipt (#826)
* runtime: candidate receipt must pass parent head

* construct parachain candidates using correct parent_head

* validate that the parent header is correct in candidate receipt

* fix test fallout

* bump runtime versions
2020-02-10 10:14:12 +01:00
Robert Habermeier 3e17fcfb3d Validation service refactoring (#773)
* add some more docs about statement import

* instantiate environment async

* move attestation service into subfolder

* refactor validation service architecture somewhat

* remove dependence on validation service in proposer

* fix a bunch of warnings

* improve docs

* introduce a builder for the validation service

* extract block production to its own file

* integrate new API into service

* address review grumbles
2020-01-17 10:28:19 +01:00
Robert HabermeierandAshley a5d9645bf4 Instantiate environment with asynchronous API (#768)
* point to in-progress Substrate branch

* instantiate environment async

* Fix futures

* Bump runtime

* Fix collation tests

* point to polkadot-master again

* point to polkadot-master again

* update deps

Co-authored-by: Ashley <ashley.ruglys@gmail.com>
2020-01-16 16:09:06 +01:00
Robert HabermeierandBastian Köcher 16043df46f rough draft of parachains roadmap (#740)
* rough draft of parachains roadmap

* flesh out first 2 phases

* Fix typo

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Fix typos and add clarifications

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* remove extra space

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* add XCMP work items

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-01-14 19:14:20 +01:00
Robert Habermeier ab2333da89 Add versioning to ParachainHost runtime API (#652)
* add version to ParachainHost API

* better error message
2019-12-04 02:04:03 +01:00
55c4c830fe ICMP message-routing gossip (#304)
* core logic for ICMP gossip

* refactor gossip to make more extension friendly

* move files aroun

* extract attestation-gossip logic to its own module

* message validation and broadcast logic

* fix upstream crates' compilation

* add a test

* another test for overlapping

* Some grammar and phrasing tweaks

Co-Authored-By: Luke Schoen <ltfschoen@users.noreply.github.com>

* add since parameter to ingress runtime API

* broadcast out known unrouted message queues

* fix compilation of service and collator

* remove useless index_mapping

* some tests for icmp propagation

* fix decoding bug and test icmp queue validation

* simplify engine-id definition

Co-Authored-By: Sergei Pepyakin <sergei@parity.io>

* address some grumbles

* some cleanup of old circulation code

* give network a handle to extrinsic store on startup

* an honest collator ensures data available as well

* address some grumbles

* add docs; rename the attestation session to "leaf work"

* module docs

* move gossip back to gossip.rs

* clean up and document attestation-gossip a bit

* some more docs on the availability store

* store all outgoing message queues in the availability store

* filter `Extrinsic` out of validation crate

* expunge Extrinsic from network

* expunge Extrinsic from erasure-coding

* expunge Extrinsic from collator

* expunge from adder-collator

* rename ExtrinsicStore to AvailabilityStore everywhere

* annotate and clean up message-routing tests
2019-08-29 11:49:59 +02:00
Robert Habermeier 7d78d9ace6 pay out slashes to the treasury (#391) 2019-08-20 18:33:14 +02:00
Robert Habermeier 742730c865 error variant for failed deadline compute in validation worker (#383)
* error variant for failed deadline compute

* fix compilation
2019-08-18 17:12:46 +02:00
Robert Habermeier 4d19de4bd9 Track received attestations from recent blocks (#337)
* record attestations in storage ringbuffer

* remove some reliance on Aura

* fix up test configuration

* extract attestations stuff out to its own module

* add dummy inherent

* use double_map

* fix a couple more compilation errors
2019-08-06 23:08:38 +02:00
Robert Habermeier 8c4d882407 sanity check on validity-votes length (#354) 2019-08-06 15:37:38 +02:00
Robert Habermeier 7a5b9bddf5 Charge fees for parachain execution (#293)
* burn parachain funds depending on candidate fees

* charge fees when executing parachain

* fix test compilation

* branch grumble addressed

* test that Balance >= usize
2019-06-25 21:09:54 +02:00
Robert HabermeierandGavin Wood 58ab4f6b9f Track and accumulate ingress roots in runtime (#287)
* track unrouted ingress in runtime

* test ingress routing

* fix compilation

* add space

Co-Authored-By: Gavin Wood <github@gavwood.com>
2019-06-17 14:38:03 +02:00
Robert HabermeierandBastian Köcher bc59254f41 allow asynchronous collation (#290)
* allow asynchronous collation

* remove unnecessary leading colons

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* remove unneeded bound

* Fixes compilation
2019-06-17 08:35:01 +02:00
Robert Habermeier 164943b961 strip out all ICMP network code and begin gossip refactor for attestations (#256)
* strip out all ICMP code and begin gossip refactor

* validate incoming statements

* message_allowed logic

* compiles

* do reporting and neighbor packet validation

* tests compile

* propagate gossip messages

* test message_allowed

* some more tests

* address grumbles
2019-05-17 14:30:10 -04:00
Robert Habermeier 36dd42523d update substrate reference (#249)
* update substrate reference

* bump wasm
2019-05-08 12:10:26 +02:00
Robert Habermeier a65be1b2df update substrate reference (#244)
* port polkadot_runtime and polkadot_validation

* update storages build (#245)

* all tests pass

* rebuild wasm
2019-05-06 18:25:55 +02:00
Robert Habermeier 1437c8e224 Introduce a Proof-of-Validation block type and use that in place of BlockData (#227)
* validators expect collators to give them parachain messages

* mostly port network to use pov_block

* network tests pass

* verify ingress when fetching pov block

* fix runtime compilation

* all tests build

* fix some grumbles

* Update validation/src/collation.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* Update primitives/src/parachain.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>

* Update network/src/lib.rs

Co-Authored-By: rphmeier <rphmeier@gmail.com>
2019-04-24 13:24:03 +02:00
Robert Habermeier c1ec4bd821 use authorities when calculating duty roster (#184) 2019-03-22 20:36:10 +01:00
Robert Habermeier 454ddf8921 Collators get incoming parachain messages (#149)
* refactor out a consensus data fetcher from table router

* move statement checking logic into router

* refuse to start authority if collator

* support building the table router asynchronously

* instantiate_consensus does not overwrite old

* update key in new consensus if there was none before

* collator collects ingress from network

* test produced egress roots

* fix adder-collator compilation

* address first grumbles

* integrate new gossip with collator network launch

* address review
2019-03-22 00:48:36 +01:00
Robert Habermeier 7e6183f74c Update to new gossip system. (#172)
* Integrates new gossip system into Polkadot (#166)

* new gossip validation in network

* integrate new gossip into service

* Fix build

* Fix claims module

* fix warning

* update to latest master again

* update runtime
2019-03-06 20:47:09 +01:00
Robert Habermeier 4ef53912e6 define Id type in polkadot-parachain and depend on that in primitives. (#144)
* define Id type in polkadot-parachain and depend on that in primitives.

* fix tests
2019-02-20 12:33:52 +01:00
Robert Habermeier 84d8629de6 some erasure-coding tweaks (#143) 2019-02-19 17:18:29 -03:00
Robert Habermeier f8af277006 Interchain message-passing (#117)
* compute ingress and routing in polkadot runtime

* extract parent candidates from block when beginning consensus

* fetch incoming messages when validating

* fix consensus tests

* parachain wasm execution uses messages

* update parachain tests to check if messages are executed

* abstract out network service to make room for network tests

* skeleton for incoming data fetch

* collate ingress from consensus-gossip

* keep track of validated candidates in the shared-table

* add some shared_table tests for new behavior

* broadcast egress messages on gossip

* test compute_ingress

* move network tests to module folder

* dummy network for consensus-network tests

* make consensus network generic over executor

* test egress broadcast and ingress fetch

* fix test compilation

* address some grumbles

* address grumbles and fix parachain shuffle

* remove broadcast parameter from consensus network trait
2019-02-19 13:59:29 -03:00
Robert Habermeier da409f6c9e update to latest substrate (#141) 2019-02-18 17:20:35 -03:00
Robert Habermeier a05d422a55 Initial erasure-coding of availability data (#56)
* erasure-coding block data

* adjust error handling

* merkleize chunks and yield branches for each

* construction and proving of merkle branches

* port over to new GF(2^16) impl

* some tests for wrapped_shard

* handle extra byte from GF(2^16) better

* point to github dependency

* add issue link

* point to master for reed-solomon-erasure

* add missing license header
2019-01-24 12:31:22 -03:00
Robert Habermeier fe6351ca65 Parachain execution yields messages to send (#96)
* read head-data directly out of WASM memory

* implement ext_post_message for parachain WASM

* further refactoring of the parachain module

* add externalities error type

* accumulate posted messages when validating parachain candidate

* define Extrinsic type in primitives

* availability-store: store extrinsic data

* compute extrinsic and check against candidate

* add some egress queue tests

* grumbles & substrate update

* ensure everything builds
2019-01-22 12:32:32 +01:00
Robert Habermeier 152bb30889 limit number of transactions when building blocks (#91) 2019-01-18 16:31:39 +01:00
Robert Habermeier 32cef97681 Remove availability statement (#79)
* remove availability vote type from statement-table

* expunge availability statement from consensus module

* expunge availability from duty roster

* rename StatementProducer to ParachainWork

* fix runtime tests and remove availability statement variant

* update wasm
2019-01-18 11:45:05 +01:00
Robert Habermeier 7ac1fe9c01 default to alexander testnet (#83)
* default to alexander testnet

* use empty pattern-match
2019-01-17 09:27:46 -03:00
Robert Habermeier 75bce2efc6 update to latest substrate (#84) 2019-01-15 18:40:33 -03:00
Robert Habermeier b2db7857a7 ensure all spawned futures are exit-guarded (#59) 2018-12-29 18:05:45 +01:00
Robert Habermeier ccbae389c2 Update to latest substrate (#57)
* update to latest substrate

* update WASM and runtime

* Rename Id to ParaId in decl_{module,storage} (exported metadata type) (#58)

* Rename Id to ParaId in decla_module (exported type)

* AccountParaId -> AccountId
2018-12-20 13:43:48 +01:00
Robert Habermeier 014215181b wait less time for parachain candidates before releasing a block (#55) 2018-12-14 16:37:15 +01:00
Robert Habermeier 19095168ce Authorship works again (#50)
* provide through inherent-data when authoring

* remove unneeded codec round-trip in proposer

* refactor polkadot-consensus service architecture

* integrate block authorship into polkadot service

* remove unused extern substrate-network crate in service

* write wrapper for unifying errors in consensus proposer

* extend wrapper further

* switch temporarily to macro-changing branch

* runtime compiles

* implement `inherent_extrinsics` for runtime

* block authorship works

* add GRANDPA to polkadot runtime

* get everything compiling

* use substrate master branch again

* remove some unneeded params

* update WASM

* parse only extrinsics when pruning availability store

* update recent deps

* runtime almost compiles

* need to expose trait type in build : I had to put phantomdata manually.

* finish updating authorship to latest GRANDPA and Aura

* fix tests

* update wasm
2018-12-11 17:55:04 +01:00
Robert Habermeier e008813f8b Allow many attestation instances to live at once in network (#46)
* multiple consensus sessions in network

* tests compile, add a test for RecentSessionKeys

* track recently received session keys from validators

* add a test for desired key-sending behavior
2018-11-28 15:14:06 +01:00
Robert Habermeier 7b46856159 Blocks carry full attestations for candidates (#42)
* statement table yields fully-attested candidates

* attestation types in polkadot-primitives

* propose block with fully-attested candidates in consensus

* some signature-checking logic in the runtime

* fix runtime compilation

* ensure attestations are full and without duplicate when checking

* fix consensus-service compilation

* add some tests

* use bitvec from crates.io now that it's published

* sign statements based on primitive statement's encoding

* remove some serialize bounds

* Fix error message with duplicate availability attestations

Co-Authored-By: rphmeier <rphmeier@gmail.com>
2018-11-26 17:30:04 +01:00
Robert Habermeier 624c2ca0dc Merge remote-tracking branch 'substrate-disk/polkadot-testnet-fixes-fp' 2018-08-23 13:57:43 +02:00
Robert Habermeier 430e3920e5 Rollup of various testnet-related fixes.
fix a deadlock when spawning agreement as non-authority

fix test compilation for BFT

more accurate consensus superseding logic

mild revision to `can_build_on` logic

block evaluation without redundant initialisation

refactor BFT delay: update rhododendron and poll after delaying. (#589)

dropping BFT future before poll doesn't lead to service deadlock
2018-08-22 18:14:58 +02:00
Robert Habermeier f1cc82ae53 Continue from next round when re-building on same block (#582)
* keep rounds consistent when encountering bad block

* fix interval logic

* Fixed indentation
2018-08-17 22:13:01 +02:00
Robert Habermeier 321441c1f7 fix interval logic 2018-08-17 13:56:13 +02:00
Robert Habermeier 898c7d3138 parity-substrate -> substrate 2018-08-15 14:34:49 +02:00
Robert Habermeier 56759c781c replacing references to staging branch 2018-08-15 14:17:28 +02:00
Robert Habermeier 3e1e14d2c1 remove WASM publish step 2018-08-15 13:57:46 +02:00
Robert Habermeier 649bb2fdaa build new WASM 2018-08-15 13:42:34 +02:00
Robert Habermeier 670fc459b2 copy scripts 2018-08-15 13:40:49 +02:00
Robert Habermeier f85fcd4fc2 get building 2018-08-15 13:40:46 +02:00
Robert Habermeier 6e0903613b point to upstream split-substrate branch 2018-08-15 13:16:54 +02:00
Robert Habermeier cfdda87d44 point things to substrate git repo 2018-08-14 18:53:12 +02:00
Robert Habermeier e24537e653 move over some files 2018-08-14 18:49:38 +02:00
Robert Habermeier 74260a0847 Vote out offline authorities (#524)
* notify when an authority appears to have missed their block

* Runtime API

* offline tracker

* Move to consensus

* generating reports of offline indices

* stubbed-out evaluation logic

* Slashing data pathwat

* usize -> u32

* Slash bad validators.

* update to rhododendron 0.3

* fix compilation of polkadot-consensus

* Support offline noting in checked_block

* include offline reports in block authorship voting

* do not vote validators offline after some time

* add test for offline-tracker

* fix test build

* bump spec version

* update wasm

* Only allow validators that are possible to slash

* Fix grumble

* More idiomatic

* New Wasm.

* update rhododendron

* improve logging and reduce round time exponent

* format offline validators in ss58
2018-08-11 11:29:30 +02:00
Robert Habermeier 22c9ac76aa format offline validators in ss58 2018-08-10 18:40:39 +02:00
Robert Habermeier 539650c063 Slashing for voted-offline validators. (#541)
[Backport] Vote out offline authorities
2018-08-10 16:12:17 +02:00
Robert Habermeier 5547b2797b fix block body fetch for availability pruning (#539) 2018-08-10 16:02:07 +02:00
Robert Habermeier b2451ca6cc don't wait for all futures to resolve before shutdown 2018-08-10 13:57:26 +02:00
Robert Habermeier f2c19dce6e don't wait for all futures to resolve before shutdown (#536) 2018-08-10 13:56:54 +02:00
Robert Habermeier fc16b193de Fix some formatting grumbles (#501) 2018-08-06 14:42:39 +03:00
Robert Habermeier 7143e85f39 Availability/Extrinsic store (#465) 2018-08-06 11:55:55 +02:00
Robert Habermeier a7c7bd49d9 force BFT delay in consensus service, not in proposer logic (#477)
* move forced delay to consensus service

* fiddle with logging
2018-08-02 17:01:27 +02:00
Robert Habermeier f4cd995558 Collator for the "adder" (formerly basic-add) parachain and various small fixes (#438)
* update basic_add wasm

* wasm feature and collator feature

* move test parachains around a little

* fix wasm build for basic_add

* move basic_add to adder, introduce README

* minimal basic_add collator

* ensure collator messages are sent in the right order

* more logging

* route consensus statements to all peers

* minor bugfixes for parachains

* genesis builder accounts for parachain heads

* fix parachains tests

* targets for txpool

* tweak runtime + collator

* fix version in adder-collator

* consistency for overflowing

* adjust comment

* fix stable test run

* remove dummy registration test

* final grumbles
2018-08-01 17:04:04 +02:00
Robert Habermeier 5a09802e57 Collator-side of collator protocol (#351)
* skeleton of collators object

* awaiting and handling collations. rename `collators` to CollationPool

* add some tests

* add tests

* implement Collators trait for ConsensusNetwork

* plug collators into main polkadot-network

* ignore collator role message

* add a couple more tests

* garbage collection for collations

* extract session-key tracking from consensus

* add local_collations.rs

* finish polish of local_collations

* integrate local_collations into network layer

* introduce API for adding local collations

* mostly finish collator implementation pending service fix

* Specialized network()

* push collations to the network

* grumbles

* substrate-service has custom configuration

* initialize network in collator mode as necessary
2018-07-18 15:04:26 +02:00
Robert Habermeier 90ea33c9fe hash random seed when using to choose round proposer (#336) 2018-07-16 18:17:52 +02:00
Robert Habermeier 88c40c8fb4 Validator side of the collation protocol. (#295)
* skeleton of collators object

* awaiting and handling collations. rename `collators` to CollationPool

* add some tests

* add tests

* implement Collators trait for ConsensusNetwork

* plug collators into main polkadot-network

* ignore collator role message

* add a couple more tests

* garbage collection for collations

* ensure disconnected backup collator is removed from pool

* address other grumbles
2018-07-12 10:47:10 +02:00
Robert Habermeier 88dc3ad8da PoC-1 consensus extracted to "rhododendron" (#284) 2018-07-09 11:26:04 +02:00
Robert Habermeier 24f7b548dc Collator node workflow (#280)
* arbitrary application logic in CLI

* collation work

* split up exit and work futures in application

* collation node workflow

* typo

* indentation fix

* doc grumbles

* rename Application to Worker

* refactor Worker::exit to exit_only
2018-07-06 17:13:31 +03:00
Robert Habermeier 6bfcbd6d59 Minimal parachains part 2: Parachain statement and data routing (#173)
* dynamic inclusion threshold calculator

* collators interface

* collation helpers

* initial proposal-creation future

* create proposer when asked to propose

* remove local_availability duty

* statement table tracks includable parachain count

* beginnings of timing future

* finish proposal logic

* remove stray println

* extract shared table to separate module

* change ordering

* includability tracking

* fix doc

* initial changes to parachains module

* initialise dummy block before API calls

* give polkadot control over round proposer based on random seed

* propose only after enough candidates

* flesh out parachains module a bit more

* set_heads

* actually introduce set_heads to runtime

* update block_builder to accept parachains

* split block validity errors from real errors in evaluation

* update WASM runtimes

* polkadot-api methods for parachains additions

* delay evaluation until candidates are ready

* comments

* fix dynamic inclusion with zero initial

* test for includability tracker

* wasm validation of parachain candidates

* move primitives to primitives crate

* remove runtime-std dependency from codec

* adjust doc

* polkadot-parachain-primitives

* kill legacy polkadot-validator crate

* basic-add test chain

* test for basic_add parachain

* move to test-chains dir

* use wasm-build

* new wasm directory layout

* reorganize a bit more

* Fix for rh-minimal-parachain (#141)

* Remove extern "C"

We already encountered such behavior (bug?) in pwasm-std, I believe.

* Fix `panic_fmt` signature by adding `_col`

Wrong `panic_fmt` signature can inhibit some optimizations in LTO mode.

* Add linker flags and use wasm-gc in build script

Pass --import-memory to LLD to emit wasm binary with imported memory.

Also use wasm-gc instead of wasm-build.

* Fix effective_max.

I'm not sure why it was the way it was actually.

* Recompile wasm.

* Fix indent

* more basic_add tests

* validate parachain WASM

* produce statements on receiving statements

* tests for reactive statement production

* fix build

* add OOM lang item to runtime-io

* use dynamic_inclusion when evaluating as well

* fix update_includable_count

* remove dead code

* grumbles

* actually defer round_proposer logic

* update wasm

* address a few more grumbles

* schedule collation work as soon as BFT is started

* impl future in collator

* fix comment

* governance proposals for adding and removing parachains

* bump protocol version

* tear out polkadot-specific pieces of substrate-network

* extract out polkadot-specific stuff from substrate-network

* begin polkadot network subsystem

* grumbles

* update WASM checkins

* parse status from polkadot peer

* allow invoke of network specialization

* begin statement router implementation

* remove dependency on tokio-timer

* fix sanity check and have proposer factory create communication streams

* pull out statement routing from consensus library

* fix comments

* adjust typedefs

* extract consensus_gossip out of main network protocol handler

* port substrate-bft to new tokio

* port polkadot-consensus to new tokio

* fix typo

* start message processing task

* initial consensus network implementation

* remove known tracking from statement-table crate

* extract router into separate module

* defer statements until later

* double signature is invalid

* propagating statements

* grumbles

* request block data

* fix compilation

* embed new consensus network into service

* port demo CLI to tokio

* all test crates compile

* some tests for fetching block data

* whitespace

* adjusting some tokio stuff

* update exit-future

* remove overly noisy warning

* clean up collation work a bit

* address review grumbles

* fix lock order in protocol handler

* rebuild wasm artifacts

* tag AuthorityId::from_slice for std only

* address formatting grumbles

* rename event_loop to executor

* some more docs for polkadot-network crate
2018-07-06 14:17:03 +02:00
Robert Habermeier b8115b257f fix typo (#285) 2018-07-06 13:50:40 +02:00
Robert Habermeier c473c0c734 Minimal parachain framework part 1 (#113)
* dynamic inclusion threshold calculator

* collators interface

* collation helpers

* initial proposal-creation future

* create proposer when asked to propose

* remove local_availability duty

* statement table tracks includable parachain count

* beginnings of timing future

* finish proposal logic

* remove stray println

* extract shared table to separate module

* change ordering

* includability tracking

* fix doc

* initial changes to parachains module

* initialise dummy block before API calls

* give polkadot control over round proposer based on random seed

* propose only after enough candidates

* flesh out parachains module a bit more

* set_heads

* actually introduce set_heads to runtime

* update block_builder to accept parachains

* split block validity errors from real errors in evaluation

* update WASM runtimes

* polkadot-api methods for parachains additions

* delay evaluation until candidates are ready

* comments

* fix dynamic inclusion with zero initial

* test for includability tracker

* wasm validation of parachain candidates

* move primitives to primitives crate

* remove runtime-std dependency from codec

* adjust doc

* polkadot-parachain-primitives

* kill legacy polkadot-validator crate

* basic-add test chain

* test for basic_add parachain

* move to test-chains dir

* use wasm-build

* new wasm directory layout

* reorganize a bit more

* Fix for rh-minimal-parachain (#141)

* Remove extern "C"

We already encountered such behavior (bug?) in pwasm-std, I believe.

* Fix `panic_fmt` signature by adding `_col`

Wrong `panic_fmt` signature can inhibit some optimizations in LTO mode.

* Add linker flags and use wasm-gc in build script

Pass --import-memory to LLD to emit wasm binary with imported memory.

Also use wasm-gc instead of wasm-build.

* Fix effective_max.

I'm not sure why it was the way it was actually.

* Recompile wasm.

* Fix indent

* more basic_add tests

* validate parachain WASM

* produce statements on receiving statements

* tests for reactive statement production

* fix build

* add OOM lang item to runtime-io

* use dynamic_inclusion when evaluating as well

* fix update_includable_count

* remove dead code

* grumbles

* actually defer round_proposer logic

* update wasm

* address a few more grumbles

* grumbles

* update WASM checkins

* remove dependency on tokio-timer
2018-05-25 16:16:01 +02:00
Robert Habermeier ef8d2b4d4b install wasm-gc using nightly (#167) 2018-05-17 14:26:11 +02:00