Commit Graph
57 Commits
Author SHA1 Message Date
Robert Habermeier 624c2ca0dc Merge remote-tracking branch 'substrate-disk/polkadot-testnet-fixes-fp' 2018-08-23 13:57:43 +02:00
Robert Habermeier 430e3920e5 Rollup of various testnet-related fixes.
fix a deadlock when spawning agreement as non-authority

fix test compilation for BFT

more accurate consensus superseding logic

mild revision to `can_build_on` logic

block evaluation without redundant initialisation

refactor BFT delay: update rhododendron and poll after delaying. (#589)

dropping BFT future before poll doesn't lead to service deadlock
2018-08-22 18:14:58 +02:00
Robert Habermeier f1cc82ae53 Continue from next round when re-building on same block (#582)
* keep rounds consistent when encountering bad block

* fix interval logic

* Fixed indentation
2018-08-17 22:13:01 +02:00
Robert Habermeier 321441c1f7 fix interval logic 2018-08-17 13:56:13 +02:00
Robert Habermeier 898c7d3138 parity-substrate -> substrate 2018-08-15 14:34:49 +02:00
Robert Habermeier 56759c781c replacing references to staging branch 2018-08-15 14:17:28 +02:00
Robert Habermeier 3e1e14d2c1 remove WASM publish step 2018-08-15 13:57:46 +02:00
Robert Habermeier 649bb2fdaa build new WASM 2018-08-15 13:42:34 +02:00
Robert Habermeier 670fc459b2 copy scripts 2018-08-15 13:40:49 +02:00
Robert Habermeier f85fcd4fc2 get building 2018-08-15 13:40:46 +02:00
Robert Habermeier 6e0903613b point to upstream split-substrate branch 2018-08-15 13:16:54 +02:00
Robert Habermeier cfdda87d44 point things to substrate git repo 2018-08-14 18:53:12 +02:00
Robert Habermeier e24537e653 move over some files 2018-08-14 18:49:38 +02:00
Robert Habermeier 74260a0847 Vote out offline authorities (#524)
* notify when an authority appears to have missed their block

* Runtime API

* offline tracker

* Move to consensus

* generating reports of offline indices

* stubbed-out evaluation logic

* Slashing data pathwat

* usize -> u32

* Slash bad validators.

* update to rhododendron 0.3

* fix compilation of polkadot-consensus

* Support offline noting in checked_block

* include offline reports in block authorship voting

* do not vote validators offline after some time

* add test for offline-tracker

* fix test build

* bump spec version

* update wasm

* Only allow validators that are possible to slash

* Fix grumble

* More idiomatic

* New Wasm.

* update rhododendron

* improve logging and reduce round time exponent

* format offline validators in ss58
2018-08-11 11:29:30 +02:00
Robert Habermeier 22c9ac76aa format offline validators in ss58 2018-08-10 18:40:39 +02:00
Robert Habermeier 539650c063 Slashing for voted-offline validators. (#541)
[Backport] Vote out offline authorities
2018-08-10 16:12:17 +02:00
Robert Habermeier 5547b2797b fix block body fetch for availability pruning (#539) 2018-08-10 16:02:07 +02:00
Robert Habermeier b2451ca6cc don't wait for all futures to resolve before shutdown 2018-08-10 13:57:26 +02:00
Robert Habermeier f2c19dce6e don't wait for all futures to resolve before shutdown (#536) 2018-08-10 13:56:54 +02:00
Robert Habermeier fc16b193de Fix some formatting grumbles (#501) 2018-08-06 14:42:39 +03:00
Robert Habermeier 7143e85f39 Availability/Extrinsic store (#465) 2018-08-06 11:55:55 +02:00
Robert Habermeier a7c7bd49d9 force BFT delay in consensus service, not in proposer logic (#477)
* move forced delay to consensus service

* fiddle with logging
2018-08-02 17:01:27 +02:00
Robert Habermeier f4cd995558 Collator for the "adder" (formerly basic-add) parachain and various small fixes (#438)
* update basic_add wasm

* wasm feature and collator feature

* move test parachains around a little

* fix wasm build for basic_add

* move basic_add to adder, introduce README

* minimal basic_add collator

* ensure collator messages are sent in the right order

* more logging

* route consensus statements to all peers

* minor bugfixes for parachains

* genesis builder accounts for parachain heads

* fix parachains tests

* targets for txpool

* tweak runtime + collator

* fix version in adder-collator

* consistency for overflowing

* adjust comment

* fix stable test run

* remove dummy registration test

* final grumbles
2018-08-01 17:04:04 +02:00
Robert Habermeier 5a09802e57 Collator-side of collator protocol (#351)
* skeleton of collators object

* awaiting and handling collations. rename `collators` to CollationPool

* add some tests

* add tests

* implement Collators trait for ConsensusNetwork

* plug collators into main polkadot-network

* ignore collator role message

* add a couple more tests

* garbage collection for collations

* extract session-key tracking from consensus

* add local_collations.rs

* finish polish of local_collations

* integrate local_collations into network layer

* introduce API for adding local collations

* mostly finish collator implementation pending service fix

* Specialized network()

* push collations to the network

* grumbles

* substrate-service has custom configuration

* initialize network in collator mode as necessary
2018-07-18 15:04:26 +02:00
Robert Habermeier 90ea33c9fe hash random seed when using to choose round proposer (#336) 2018-07-16 18:17:52 +02:00
Robert Habermeier 88c40c8fb4 Validator side of the collation protocol. (#295)
* skeleton of collators object

* awaiting and handling collations. rename `collators` to CollationPool

* add some tests

* add tests

* implement Collators trait for ConsensusNetwork

* plug collators into main polkadot-network

* ignore collator role message

* add a couple more tests

* garbage collection for collations

* ensure disconnected backup collator is removed from pool

* address other grumbles
2018-07-12 10:47:10 +02:00
Robert Habermeier 88dc3ad8da PoC-1 consensus extracted to "rhododendron" (#284) 2018-07-09 11:26:04 +02:00
Robert Habermeier 24f7b548dc Collator node workflow (#280)
* arbitrary application logic in CLI

* collation work

* split up exit and work futures in application

* collation node workflow

* typo

* indentation fix

* doc grumbles

* rename Application to Worker

* refactor Worker::exit to exit_only
2018-07-06 17:13:31 +03:00
Robert Habermeier 6bfcbd6d59 Minimal parachains part 2: Parachain statement and data routing (#173)
* dynamic inclusion threshold calculator

* collators interface

* collation helpers

* initial proposal-creation future

* create proposer when asked to propose

* remove local_availability duty

* statement table tracks includable parachain count

* beginnings of timing future

* finish proposal logic

* remove stray println

* extract shared table to separate module

* change ordering

* includability tracking

* fix doc

* initial changes to parachains module

* initialise dummy block before API calls

* give polkadot control over round proposer based on random seed

* propose only after enough candidates

* flesh out parachains module a bit more

* set_heads

* actually introduce set_heads to runtime

* update block_builder to accept parachains

* split block validity errors from real errors in evaluation

* update WASM runtimes

* polkadot-api methods for parachains additions

* delay evaluation until candidates are ready

* comments

* fix dynamic inclusion with zero initial

* test for includability tracker

* wasm validation of parachain candidates

* move primitives to primitives crate

* remove runtime-std dependency from codec

* adjust doc

* polkadot-parachain-primitives

* kill legacy polkadot-validator crate

* basic-add test chain

* test for basic_add parachain

* move to test-chains dir

* use wasm-build

* new wasm directory layout

* reorganize a bit more

* Fix for rh-minimal-parachain (#141)

* Remove extern "C"

We already encountered such behavior (bug?) in pwasm-std, I believe.

* Fix `panic_fmt` signature by adding `_col`

Wrong `panic_fmt` signature can inhibit some optimizations in LTO mode.

* Add linker flags and use wasm-gc in build script

Pass --import-memory to LLD to emit wasm binary with imported memory.

Also use wasm-gc instead of wasm-build.

* Fix effective_max.

I'm not sure why it was the way it was actually.

* Recompile wasm.

* Fix indent

* more basic_add tests

* validate parachain WASM

* produce statements on receiving statements

* tests for reactive statement production

* fix build

* add OOM lang item to runtime-io

* use dynamic_inclusion when evaluating as well

* fix update_includable_count

* remove dead code

* grumbles

* actually defer round_proposer logic

* update wasm

* address a few more grumbles

* schedule collation work as soon as BFT is started

* impl future in collator

* fix comment

* governance proposals for adding and removing parachains

* bump protocol version

* tear out polkadot-specific pieces of substrate-network

* extract out polkadot-specific stuff from substrate-network

* begin polkadot network subsystem

* grumbles

* update WASM checkins

* parse status from polkadot peer

* allow invoke of network specialization

* begin statement router implementation

* remove dependency on tokio-timer

* fix sanity check and have proposer factory create communication streams

* pull out statement routing from consensus library

* fix comments

* adjust typedefs

* extract consensus_gossip out of main network protocol handler

* port substrate-bft to new tokio

* port polkadot-consensus to new tokio

* fix typo

* start message processing task

* initial consensus network implementation

* remove known tracking from statement-table crate

* extract router into separate module

* defer statements until later

* double signature is invalid

* propagating statements

* grumbles

* request block data

* fix compilation

* embed new consensus network into service

* port demo CLI to tokio

* all test crates compile

* some tests for fetching block data

* whitespace

* adjusting some tokio stuff

* update exit-future

* remove overly noisy warning

* clean up collation work a bit

* address review grumbles

* fix lock order in protocol handler

* rebuild wasm artifacts

* tag AuthorityId::from_slice for std only

* address formatting grumbles

* rename event_loop to executor

* some more docs for polkadot-network crate
2018-07-06 14:17:03 +02:00
Robert Habermeier b8115b257f fix typo (#285) 2018-07-06 13:50:40 +02:00
Robert Habermeier c473c0c734 Minimal parachain framework part 1 (#113)
* dynamic inclusion threshold calculator

* collators interface

* collation helpers

* initial proposal-creation future

* create proposer when asked to propose

* remove local_availability duty

* statement table tracks includable parachain count

* beginnings of timing future

* finish proposal logic

* remove stray println

* extract shared table to separate module

* change ordering

* includability tracking

* fix doc

* initial changes to parachains module

* initialise dummy block before API calls

* give polkadot control over round proposer based on random seed

* propose only after enough candidates

* flesh out parachains module a bit more

* set_heads

* actually introduce set_heads to runtime

* update block_builder to accept parachains

* split block validity errors from real errors in evaluation

* update WASM runtimes

* polkadot-api methods for parachains additions

* delay evaluation until candidates are ready

* comments

* fix dynamic inclusion with zero initial

* test for includability tracker

* wasm validation of parachain candidates

* move primitives to primitives crate

* remove runtime-std dependency from codec

* adjust doc

* polkadot-parachain-primitives

* kill legacy polkadot-validator crate

* basic-add test chain

* test for basic_add parachain

* move to test-chains dir

* use wasm-build

* new wasm directory layout

* reorganize a bit more

* Fix for rh-minimal-parachain (#141)

* Remove extern "C"

We already encountered such behavior (bug?) in pwasm-std, I believe.

* Fix `panic_fmt` signature by adding `_col`

Wrong `panic_fmt` signature can inhibit some optimizations in LTO mode.

* Add linker flags and use wasm-gc in build script

Pass --import-memory to LLD to emit wasm binary with imported memory.

Also use wasm-gc instead of wasm-build.

* Fix effective_max.

I'm not sure why it was the way it was actually.

* Recompile wasm.

* Fix indent

* more basic_add tests

* validate parachain WASM

* produce statements on receiving statements

* tests for reactive statement production

* fix build

* add OOM lang item to runtime-io

* use dynamic_inclusion when evaluating as well

* fix update_includable_count

* remove dead code

* grumbles

* actually defer round_proposer logic

* update wasm

* address a few more grumbles

* grumbles

* update WASM checkins

* remove dependency on tokio-timer
2018-05-25 16:16:01 +02:00
Robert Habermeier ef8d2b4d4b install wasm-gc using nightly (#167) 2018-05-17 14:26:11 +02:00
Robert Habermeier 748238eb40 Archive Mode and groundwork for state-preserving handles (#166)
* less eager deletion of DB values

* archive mode
2018-05-17 13:47:53 +02:00
Robert Habermeier 0fa049b0c7 Store trie nodes in DB (#157)
* move responsibility of storage_root calculation to state backend

* have `storage_root` produce a memoizable transaction

* store trie nodes in kvdb

* fix up test fallout

* remove stray newline

* Fix comment

* test for setting and checking state data

* fiddle with dependencies

* all parity deps on same commit hash

* fix network protocol registration
2018-05-16 16:56:24 +02:00
Robert Habermeier 7c51525ad3 Hook RPC extrinsic import into propagation (#158)
* call `on_new_transactions` when we import

* fix trace

* pass correct bytes to network

* clean up

* cull before repropagating; repropagate on timer

* add a little tracing
2018-05-15 10:27:18 +02:00
Robert Habermeier 1283545b4b Fix sync: ensure block hash gets read from the DB correctly (#152)
* update bootnodes

* fix block_hash loading

* test block hash gets inserted correctly

* specify revision to pull kvdb-memorydb from

* update parity repo dep
2018-05-11 15:46:54 +02:00
Robert Habermeier 6a9292b71c update bootnodes (#151)
(insubstantial)
2018-05-09 18:42:49 +02:00
Robert Habermeier 54f5debf5d Ensure VoteAdvance messages propagate (#148)
* avoid clobbering accumulator state with late propose

* log state when polling

* trace on input messages
2018-05-09 12:25:17 +02:00
Robert Habermeier ab946b2b9f Ensure all known BFT messages are imported when starting consensus (#147)
* a little more BFT tracing

* import cached BFT messages into the produced stream
2018-05-08 17:01:26 +02:00
Robert Habermeier f9d5a6a09f update wasm 2018-05-08 17:00:32 +02:00
Robert Habermeier 39d691821d More testnet fixes (#145)
* initialise dummy block before API calls

* test for random-seed

* revert polkadot-runtime changes
2018-05-08 14:55:21 +02:00
Robert Habermeier 353d9438c8 don't return async::notready for messages without scheduling wakeup 2018-05-06 13:25:37 +02:00
Robert Habermeier 7f2c798a06 handle exit and avoid threads hanging (#137)
* barrier on starting network

* handle exit better

* give consensus service its own internal exit signal

* update comment

* remove stop_notifications and fix build
2018-04-18 13:59:46 +02:00
Robert Habermeier 39f78aa7c4 prune finalized transactions from the pool (#127) 2018-04-15 12:53:53 +02:00
Robert Habermeier 2bafd5c0be Revert "prune finalized transactions from the pool"
This reverts commit 7ab024ceb65c31edcf6414f98a31dd7a29688c96.
2018-04-14 15:35:54 +02:00
Robert Habermeier f39bbd1c7f prune finalized transactions from the pool 2018-04-14 15:34:08 +02:00
Robert Habermeier 002cacd26c delay before voting on proposals (#123) 2018-04-12 16:35:48 +02:00
Robert Habermeier 1c5d6d59a7 kill post_finalise (#121)
* kill post_finalise

* update wasm runtimes
2018-04-12 13:17:43 +02:00
Robert Habermeier e505fe0186 KeyStore implementation + key derivation (#97)
* improve ed25519 bindings

* probably broken child derivation

* basic keystore

* keystore integration in CLI

* constant-time mac comparison

* fix spaces
2018-03-16 17:39:49 +01:00
Robert Habermeier fdc991f973 Slash Authorities for irrefutable misbehavior (#84)
* double-commit and prepare misbehavior

* get misbehavior on completion

* collect misbehavior on drop, not only on success

* kill unused transaction_index field

* add primitive misbehavior report type

* add misbehavior report transaction

* store prior session

* fix set_items

* basic checks for misbehavior reports

* crate for substrate bft misbehavior checking

* integrate misbehavior check crate

* fix comment

* new wasm binaries

* fix hash in test

* import misbehavior transactions into queue

* fix test build

* sign on digest and full proposal when proposing

* detect proposal misbehavior

* fix fallout

* restore balance/bondage types
2018-03-13 16:39:27 +01:00
Robert Habermeier b772188145 Integrate transaction pool to the proposal logic (#80)
* reshuffle consensus libraries

* polkadot-useful type definitions for statement table

* begin BftService

* primary selection logic

* bft service implementation without I/O

* extract out `BlockImport` trait

* allow bft primitives to compile on wasm

* Block builder (substrate)

* take polkadot-consensus down to the core.

* test for preemption

* fix test build

* Fix wasm build

* Bulid on any block

* Test for block builder.

* Block import tests for client.

* Tidy ups

* clean up block builder instantiation

* justification verification logic

* JustifiedHeader and import

* Propert block generation for tests

* network and tablerouter trait

* use statement import to drive creation of further statements

* Fixed rpc tests

* custom error type for consensus

* create proposer

* asynchronous proposal evaluation

* inherent transactions in polkadot runtime

* fix tests to match real polkadot block constraints

* implicitly generate inherent functions

* add inherent transaction functionality to block body

* block builder logic for polkadot

* some tests for the polkadot API

* avoid redundancy in native code compatibility check

* helper for extracting nonce

* transaction pool implementation

* transaction pool

* integrate transaction pool with proposer

* indentation

* kill storage keys module

* accept new transactions to replace old
2018-03-02 09:38:58 +01:00
Robert Habermeier 1e6cad908e Proposal creation and evaluation to plug into BFT (#77)
* reshuffle consensus libraries

* polkadot-useful type definitions for statement table

* begin BftService

* primary selection logic

* bft service implementation without I/O

* extract out `BlockImport` trait

* allow bft primitives to compile on wasm

* Block builder (substrate)

* take polkadot-consensus down to the core.

* test for preemption

* fix test build

* Fix wasm build

* Bulid on any block

* Test for block builder.

* Block import tests for client.

* Tidy ups

* clean up block builder instantiation

* justification verification logic

* JustifiedHeader and import

* Propert block generation for tests

* network and tablerouter trait

* use statement import to drive creation of further statements

* Fixed rpc tests

* custom error type for consensus

* create proposer

* asynchronous proposal evaluation

* inherent transactions in polkadot runtime

* fix tests to match real polkadot block constraints

* implicitly generate inherent functions

* add inherent transaction functionality to block body

* block builder logic for polkadot

* some tests for the polkadot API
2018-02-25 10:58:17 +01:00
Robert Habermeier 0a9a144378 Verify header justification during import (#76)
* reshuffle consensus libraries

* polkadot-useful type definitions for statement table

* begin BftService

* primary selection logic

* bft service implementation without I/O

* extract out `BlockImport` trait

* allow bft primitives to compile on wasm

* Block builder (substrate)

* take polkadot-consensus down to the core.

* test for preemption

* fix test build

* Fix wasm build

* Bulid on any block

* Test for block builder.

* Block import tests for client.

* Tidy ups

* clean up block builder instantiation

* justification verification logic

* JustifiedHeader and import

* Propert block generation for tests

* Fixed rpc tests
2018-02-16 17:28:42 +01:00
Robert Habermeier 6fbe366f23 Split BFT into substrate-bft and runtime-specific proposer logic (#72)
* reshuffle consensus libraries

* polkadot-useful type definitions for statement table

* begin BftService

* primary selection logic

* bft service implementation without I/O

* extract out `BlockImport` trait

* allow bft primitives to compile on wasm

* take polkadot-consensus down to the core.

* test for preemption

* fix test build
2018-02-15 14:20:18 +01:00
Robert Habermeier 56fd052a26 fix runtime_test 2018-02-08 23:01:26 +01:00
Robert Habermeier 33fff3965f update wasm binaries 2018-02-08 21:07:13 +01:00
Robert Habermeier 7c62519781 Re-introduce zero copy codec and add minimal polkadot client API which uses linked native runtime (#65)
* client-api type and move duty roster types to primitives

* tuple implementation for slicable

* mild cleanup of deserialization code

* stubs which handle encoding and decoding themselves

* fancier impl_stubs macro

* zero-copy slicable API

* minimal polkadot-client API

* fix WASM API generation

* move native environment stuff to substrate executor

* fix warnings and grumbles
2018-02-08 19:20:34 +01:00