* add a magic number to backing statements encoded
* fix fallout in statement table
* fix some fallout in backing
* add magic to approval votes
* remove last references to Candidate variant
* update size-hint
We only checked all validity votes, ignoring invalid votes. So, the
condidition could not hold. Besides fixing the panic, I removed some old
cruft that isn't required anymore.
* use proper descriptive generic type names
* cleanup
* Table stores a list of detected misbehavior per authority
* add Table::drain_misbehaviors_for
* WIP: unify misbehavior types; report multiple misbehaviors per validator
Code checks, but tests don't yet pass.
* update drain_misbehaviors: return authority id as well as specific misbehavior
* enable unchecked construction of Signed structs in tests
* remove test-features feature & unnecessary generic
* fix backing tests
This took a while to figure out, because where we'd previously been
passing around `SignedFullStatement`s, we now needed to construct
those on the fly within the test, to take advantage of the signature-
checking in the constructor. That, in turn, necessitated changing the
iterable type of `drain_misbehaviors` to return the validator index,
and passing that validator index along within the misbehavior report.
Once that was sorted, however, it became relatively straightforward:
just needed to add appropriate methods to deconstruct the misbehavior
reports, and then we could construct the signed statements directly.
* fix bad merge
* Make `CandidateHash` a real type
This pr adds a new type `CandidateHash` that is used instead of the
opaque `Hash` type. This helps to ensure on the type system level that
we are passing the correct types.
This pr also fixes wrong usage of `relay_parent` as `candidate_hash`
when communicating with the av storage.
* Update core-primitives/src/lib.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Wrap the lines
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Moare fixes for parachains
- Sending data to a job should always contain a relay parent. Done this
for the provisioner
- Fixed the `select_availability_bitfields` function. It was assuming we
have one core per validator, while we only have one core per parachain.
- Drive by async "rewrite" in proposer
* Make tests compile
* Update primitives/src/v1.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* create a v1 primitives module
* Improve guide on availability types
* punctuate
* new parachains runtime uses new primitives
* tests of new runtime now use new primitives
* add ErasureChunk to guide
* export erasure chunk from v1 primitives
* subsystem crate uses v1 primitives
* node-primitives uses new v1 primitives
* port overseer to new primitives
* new-proposer uses v1 primitives (no ParachainHost anymore)
* fix no-std compilation for primitives
* service-new uses v1 primitives
* network-bridge uses new primitives
* statement distribution uses v1 primitives
* PoV distribution uses v1 primitives; add PoV::hash fn
* move parachain to v0
* remove inclusion_inherent module and place into v1
* remove everything from primitives crate root
* remove some unused old types from v0 primitives
* point everything else at primitives::v0
* squanch some warns up
* add RuntimeDebug import to no-std as well
* port over statement-table and validation
* fix final errors in validation and node-primitives
* add dummy Ord impl to committed candidate receipt
* guide: update CandidateValidationMessage
* add primitive for validationoutputs
* expand CandidateValidationMessage further
* bikeshed
* add some impls to omitted-validation-data and available-data
* expand CandidateValidationMessage
* make erasure-coding generic over v1/v0
* update usages of erasure-coding
* implement commitments.hash()
* use Arc<Pov> for CandidateValidation
* improve new erasure-coding method names
* fix up candidate backing
* update docs a bit
* fix most tests and add short-circuiting to make_pov_available
* fix remainder of candidate backing tests
* squanching warns
* squanch it up
* some fallout
* overseer fallout
* free from polkadot-test-service hell
* Updates guide for CandidateBacking
* Move assignment types to primitives
* Initial implementation.
* More functionality
* use assert_matches
* Changes to report misbehaviors
* Some fixes after a review
* Remove a blank line
* Update guide and some types
* Adds run_job function
* Some comments and refactorings
* Fix review
* Remove warnings
* Use summary in kicking off validation
* Parallelize requests
* Validation provides local and global validation params
* Test issued validity tracking
* Nits from review
* add signed wrapper, typedef SignedStatement
* typedef SignedAvailabilityBitfield
* implement Signed wrapper
This is strictly an addition as of this commit; nothing is yet
changed in existing behavior.
* inline getters, remove review comment
* move EncodeAs, Signed from node::primitives to primitives::parachain
* Refactor SignedAvailabilityBitfield to use Signed
* don't double-encode real payload
This isn't an ideal solution, because it depends on the
implementation details of how SCALE encodes tuples, but OTOH
that behavior seems unlikely to change anytime soon.
* fix build errors
* cause the runtime to build properly with the new changes
Not sure why cargo check didn't catch this earlier; oh well.
* fix runtime tests and separate SignedStatement from SignedFullStatement
* better explain why CompactStatement exists
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* encode the candidate statement as only the hash
* refactor CandidateReceipt and CollationInfo
* introduce an abridged candidate receipt type
* erasure coding stores candidate receipt
* store omitted data instead and introduce AvailableData type
* refactor availability-store schema
* tweak schema and APIs a bit more
* get availability-store tests passing
* accept AbridgedCandidateReceipt in `set_heads`
* change statement type in primitives to be hash-only
* fix parachains runtime tests
* fix bad merge
* rewrite validation pipeline
* remove evaluation module
* use abridged candidate hash as canonical
* statement table uses abridged candidate receipts
* kill availability_store::Data struct
* port shared table to new validation pipelines
* extract full validation pipeline to helper
* remove old validation pipeline from collation module
* polkadot-validation compiles
* polkadot-validation tests compile
* make local collation available in validation service
* port legacy network code
* polkadot-network fully ported
* network: ensure fresh statement is propagated
* remove pov_block_hash from LocalValidationData
* remove candidate_hash field from AttestedCandidate and update runtime
* port runtimes to new ParachainHost definition
* port over polkadot-collator
* fix test compilation
* better fix
* remove unrelated validation work dispatch fix
* address grumbles
* fix equality check
* add some more docs about statement import
* instantiate environment async
* move attestation service into subfolder
* refactor validation service architecture somewhat
* remove dependence on validation service in proposer
* fix a bunch of warnings
* improve docs
* introduce a builder for the validation service
* extract block production to its own file
* integrate new API into service
* address review grumbles
* Integrate srml/im-online
* Fix all build errors with old aura.
* Fix most of the build errors.
* Builds and tests seem to pass (I will not trust this commit yet)
* Apply suggestions from code review
Co-Authored-By: Robert Habermeier <rphmeier@gmail.com>
* Kill some warnings.
* fix panics on 0 validators
* Fix dev chain.
* Fix author stuff
* fix im online integration.
* Some tweaks
* Introduce app-crypto
* Initial build work
* codec update / tweaks
* patch polkadot-erasure-coding input
* More fixes for new crypto
* More fixes
* Update parachains module
* evamp parachain crypto
* More crypto work.
* Chain spec and service.
* ChainSpec stuff
* Last bits for a clean build
* Tweak coment
* adapt polkadot-validation to the new keystore
* polkadot-network compiles, but tests don't
* Integrate the new parachain validation stuff
* delete message_routing file
* make polkadot-network tests compile and pass
* runtime tests compile and pass
* update substrate ref
* service compiles
* all tests pass
* Add TODO, change branch back to polkadot-master
* Lock file
* TODOs done
* Issue number
* Remove old tODO
* Remove commented code
* strip out all ICMP code and begin gossip refactor
* validate incoming statements
* message_allowed logic
* compiles
* do reporting and neighbor packet validation
* tests compile
* propagate gossip messages
* test message_allowed
* some more tests
* address grumbles
* Don't pass validators' public keys with attestations
* Update statement-table's Misbehaviour typedef
* Update network/router
* Expand MessageValidationData
* Try to fix tests
* Extend ApiContext
* Remove 'index_mapping' from the SessionParams
* Construct index_mapping from authorities
* Move index_mapping to TableContext
* Add test for index_mapping order
* Update to master
This introduces a new type `CollatorId`, currently just `SessionKey`
but which would forseeably change to its own thing. It seems to work
like this (despite there being a lot of the new-incompatible
`AccountId` replaced). No idea if it does anything sensible, though.
* Cleanups
* Fix tests
* Remove commented code
* Specify commit hash
* Remove commented code
* Correct version
* Update runtime/Cargo.toml
Co-Authored-By: gavofyork <github@gavwood.com>
* PairT instead of _Pair
* Update lock file
* Remove rev causing upset
* statement table yields fully-attested candidates
* attestation types in polkadot-primitives
* propose block with fully-attested candidates in consensus
* some signature-checking logic in the runtime
* fix runtime compilation
* ensure attestations are full and without duplicate when checking
* fix consensus-service compilation
* add some tests
* use bitvec from crates.io now that it's published
* sign statements based on primitive statement's encoding
* remove some serialize bounds
* Fix error message with duplicate availability attestations
Co-Authored-By: rphmeier <rphmeier@gmail.com>
* dynamic inclusion threshold calculator
* collators interface
* collation helpers
* initial proposal-creation future
* create proposer when asked to propose
* remove local_availability duty
* statement table tracks includable parachain count
* beginnings of timing future
* finish proposal logic
* remove stray println
* extract shared table to separate module
* change ordering
* includability tracking
* fix doc
* initial changes to parachains module
* initialise dummy block before API calls
* give polkadot control over round proposer based on random seed
* propose only after enough candidates
* flesh out parachains module a bit more
* set_heads
* actually introduce set_heads to runtime
* update block_builder to accept parachains
* split block validity errors from real errors in evaluation
* update WASM runtimes
* polkadot-api methods for parachains additions
* delay evaluation until candidates are ready
* comments
* fix dynamic inclusion with zero initial
* test for includability tracker
* wasm validation of parachain candidates
* move primitives to primitives crate
* remove runtime-std dependency from codec
* adjust doc
* polkadot-parachain-primitives
* kill legacy polkadot-validator crate
* basic-add test chain
* test for basic_add parachain
* move to test-chains dir
* use wasm-build
* new wasm directory layout
* reorganize a bit more
* Fix for rh-minimal-parachain (#141)
* Remove extern "C"
We already encountered such behavior (bug?) in pwasm-std, I believe.
* Fix `panic_fmt` signature by adding `_col`
Wrong `panic_fmt` signature can inhibit some optimizations in LTO mode.
* Add linker flags and use wasm-gc in build script
Pass --import-memory to LLD to emit wasm binary with imported memory.
Also use wasm-gc instead of wasm-build.
* Fix effective_max.
I'm not sure why it was the way it was actually.
* Recompile wasm.
* Fix indent
* more basic_add tests
* validate parachain WASM
* produce statements on receiving statements
* tests for reactive statement production
* fix build
* add OOM lang item to runtime-io
* use dynamic_inclusion when evaluating as well
* fix update_includable_count
* remove dead code
* grumbles
* actually defer round_proposer logic
* update wasm
* address a few more grumbles
* schedule collation work as soon as BFT is started
* impl future in collator
* fix comment
* governance proposals for adding and removing parachains
* bump protocol version
* tear out polkadot-specific pieces of substrate-network
* extract out polkadot-specific stuff from substrate-network
* begin polkadot network subsystem
* grumbles
* update WASM checkins
* parse status from polkadot peer
* allow invoke of network specialization
* begin statement router implementation
* remove dependency on tokio-timer
* fix sanity check and have proposer factory create communication streams
* pull out statement routing from consensus library
* fix comments
* adjust typedefs
* extract consensus_gossip out of main network protocol handler
* port substrate-bft to new tokio
* port polkadot-consensus to new tokio
* fix typo
* start message processing task
* initial consensus network implementation
* remove known tracking from statement-table crate
* extract router into separate module
* defer statements until later
* double signature is invalid
* propagating statements
* grumbles
* request block data
* fix compilation
* embed new consensus network into service
* port demo CLI to tokio
* all test crates compile
* some tests for fetching block data
* whitespace
* adjusting some tokio stuff
* update exit-future
* remove overly noisy warning
* clean up collation work a bit
* address review grumbles
* fix lock order in protocol handler
* rebuild wasm artifacts
* tag AuthorityId::from_slice for std only
* address formatting grumbles
* rename event_loop to executor
* some more docs for polkadot-network crate
* Some initial work on RPC and client
* Rephrase as params
* More work on traitifying substrate.
* Traitify in_mem.rs
* traitify client.rs
* Make new primitives (mainly traits) build again.
* Many (superficial) build fixes throughout.
* Fix remaining build issues up to bft interface.
* Make bft primitives be generic.
* Switch out MisBehaviorReport for generic version.
* Merge Hashing into Header.
* Update runtime for new generics (with Hashing).
* Update demo runtime.
* Make runtime compile.
* Build fixes for runtime
* Remove old modules.
* port substrate-bft to use generic substrate types
* port client
* port substrate-test-runtime
* mostly port test-runtime to get compiling for std
* Ensure `AccountId` has a `Default`.
* Fix type deps.
* finish porting
* initialize test_runtime from genesis correctly
* remove commented code
* maybe unsigned signatures
* runtimes compile
* port over most of network
* serialization for generic types
* fix comment
* remove some unnecessary trait bounds
* network compiles
* tests compile for sync
* fix deserialization
* temporarily remove deserialize derives
* workarounds for serde issues for deriving deserialization
* get demo-runtime compiling on std
* port extrinsic-pool
* primitives reshuffling
* get network compiling again
* remove debugging file
* runtime tests now passing
* port client-db
* start to port over substrate-rpc
* mostly port over PolkadotApi
* test_runtime follows normal conventions
* substrate runtime tests pass
* deal with inherent extrinsics correctly in polkadot-api
* port transaction-pool
* port polkadot-consensus
* port substrate-rpc
* everything compiles
* tests compile
* fix grumbles
* test-runtime uses its own transfer type
* switch to master branch of jsonrpc
* fix network tests and some warnings
* all tests pass locally
* [ci-skip] add another comment about issue
* remove some curlies
* dynamic inclusion threshold calculator
* collators interface
* collation helpers
* initial proposal-creation future
* create proposer when asked to propose
* remove local_availability duty
* statement table tracks includable parachain count
* beginnings of timing future
* finish proposal logic
* remove stray println
* extract shared table to separate module
* change ordering
* includability tracking
* fix doc
* initial changes to parachains module
* initialise dummy block before API calls
* give polkadot control over round proposer based on random seed
* propose only after enough candidates
* flesh out parachains module a bit more
* set_heads
* actually introduce set_heads to runtime
* update block_builder to accept parachains
* split block validity errors from real errors in evaluation
* update WASM runtimes
* polkadot-api methods for parachains additions
* delay evaluation until candidates are ready
* comments
* fix dynamic inclusion with zero initial
* test for includability tracker
* wasm validation of parachain candidates
* move primitives to primitives crate
* remove runtime-std dependency from codec
* adjust doc
* polkadot-parachain-primitives
* kill legacy polkadot-validator crate
* basic-add test chain
* test for basic_add parachain
* move to test-chains dir
* use wasm-build
* new wasm directory layout
* reorganize a bit more
* Fix for rh-minimal-parachain (#141)
* Remove extern "C"
We already encountered such behavior (bug?) in pwasm-std, I believe.
* Fix `panic_fmt` signature by adding `_col`
Wrong `panic_fmt` signature can inhibit some optimizations in LTO mode.
* Add linker flags and use wasm-gc in build script
Pass --import-memory to LLD to emit wasm binary with imported memory.
Also use wasm-gc instead of wasm-build.
* Fix effective_max.
I'm not sure why it was the way it was actually.
* Recompile wasm.
* Fix indent
* more basic_add tests
* validate parachain WASM
* produce statements on receiving statements
* tests for reactive statement production
* fix build
* add OOM lang item to runtime-io
* use dynamic_inclusion when evaluating as well
* fix update_includable_count
* remove dead code
* grumbles
* actually defer round_proposer logic
* update wasm
* address a few more grumbles
* grumbles
* update WASM checkins
* remove dependency on tokio-timer
* reshuffle consensus libraries
* polkadot-useful type definitions for statement table
* begin BftService
* primary selection logic
* bft service implementation without I/O
* extract out `BlockImport` trait
* allow bft primitives to compile on wasm
* take polkadot-consensus down to the core.
* test for preemption
* fix test build
Robert Habermeier