* send our view to new peers immediately
* guide: update
* Fix tests by expecting the view updates
* Add test that we send our view on connection
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* initial implementation of lifecycles and upgrades
* clean up a bit
* fix doc comment
* more rigid lifecycle checks
* include paras which are transitioning, and lifecycle query
* format guide
* update api
* update guide
* explicit outgoing state, fix genesis
* handle outgoing with transitioning paras
* do not include transitioning paras in identifier
* Update roadmap/implementers-guide/src/runtime/paras.md
* Update roadmap/implementers-guide/src/runtime/paras.md
* Update roadmap/implementers-guide/src/runtime/paras.md
* Apply suggestions from code review
* Use matches macro
* Correct terms
* Apply suggestions from code review
* actions queue
* Revert "actions queue"
This reverts commit b2e9011ec8937d6c73e99292416c9692aeb30f73.
* collapse onboarding state
Co-authored-by: Gavin Wood <gavin@parity.io>
* PVD: `block_number`->`relay_parent_number`
* ValidationParams: `relay_chain_height`->`relay_parent_number`
* Expose DMQ MQC hash as a well-known-key
This way the relay storage merkle proofs will be able to obtain the DMQ
MQC hash and we will be able to remove the it from the
PersistedValidationData struct.
* PersistedValidationData: Remove HRMP MQC heads
* PersistedValidationData: Remove `dmq_mqc_head`
* Expose the HRMP ingress channel index as a well-known-key
This way a parachain (PVF and collator) can find all the parachains that
have an outbound channel to the given one. That allows in turn to find
all the inbound channels for the given para.
Having access to that allows the parachain to get the same information
as the hrmp_mqc_heads now provide.
* Rename `relay_storage_root` to `relay_parent_storage_root`
* grab guide updates from approval-voting branch
* focus on control flow and behavior, not functions
* add approval voting preamble info
* tabs -> spaces
* use proper descriptive generic type names
* cleanup
* Table stores a list of detected misbehavior per authority
* add Table::drain_misbehaviors_for
* WIP: unify misbehavior types; report multiple misbehaviors per validator
Code checks, but tests don't yet pass.
* update drain_misbehaviors: return authority id as well as specific misbehavior
* enable unchecked construction of Signed structs in tests
* remove test-features feature & unnecessary generic
* fix backing tests
This took a while to figure out, because where we'd previously been
passing around `SignedFullStatement`s, we now needed to construct
those on the fly within the test, to take advantage of the signature-
checking in the constructor. That, in turn, necessitated changing the
iterable type of `drain_misbehaviors` to return the validator index,
and passing that validator index along within the misbehavior report.
Once that was sorted, however, it became relatively straightforward:
just needed to add appropriate methods to deconstruct the misbehavior
reports, and then we could construct the signed statements directly.
* fix bad merge
* start graphing message flow between subsystems
This commit includes messages originating from the first three subsystems
(alphabetically).
* use standard arrowhead names
* add messages from bitfield distribution && use circo layout
* finish adding all subsystem message traffic to graph
* reduce line spam
* start work on graphing v1 types
* finish graphing candidate-related structs
* show hash relations, format
* show provenance of ValidityAttestation
* add the rest of the v1 polkadot primitives
* add type diagram from polkadot::parachain::primitives
* start work digramming the path to backing
* diagram message flow from collation generation -> candidate selection
* document flow through CandidateBacking, CandidateValidation
* graph data flow through StatementDistribution
* trace data flow through PoVDistribution to Backing
* finish documenting node side subsystem data flow
* initial impl approval distribution
* initial tests and fixes
* batching seems difficult: different peers have different needs
* bridge: fix test after merge
* some guide updates
* only send assignments to peers who know about the block
* fix a test, add approvals test
* simplify
* do not send assignment to peers for finalized blocks
* guide: protocol input and output
* one more test
* more comments, logs, initial metrics
* fix a typo
* one more thing: early return when reimporting a thing locally
* Compress the PoV block before sending it over the network
This pr changes the way we send PoV blocks over the network. We now
compress the PoV block before it is send over the network. This should
reduce the size significant for PoVs which contain the runtime WASM for
example.
* Preallocate 1KB
* Try something..
* Switch to zstd and some renamings
* Make compression/decompression fail in browsers
* Use some sane maximum value
* Update roadmap/implementers-guide/src/types/network.md
Co-authored-by: Andronik Ordian <write@reusable.software>
* Fix and add test
* add
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* glossary: define DMQ
We are using DMQ quite often including in rustdoc, however, it may be
hard to know what is that. Adding it to the glossary.
* Clarify that DMP and HRMP are also module names
* Define PDK
* Define preimage
* collation-generation: use persisted validation data
* node: remote FullValidationData API
* runtime: remove FullValidationData API
* backing tests: use persisted validation data
* FullCandidateReceipt: use persisted validation data
This is not a big change since this type is not used anywhere
* Remove ValidationData and TransientValidationData
Also update the guide
* skeleton
* skeleton aux-schema module
* start approval types
* start aux schema with aux store
* doc
* finish basic types
* start approval types
* doc
* finish basic types
* write out schema types
* add debug and codec impls to approval types
* add debug and codec impls to approval types
also add some key computation
* add debug and codec impls to approval types
* getters for block and candidate entries
* grumbles
* remove unused AssignmentId
* load_decode utility
* implement DB clearing
* function for adding new block entry to aux store
* start `canonicalize` implementation
* more skeleton
* finish implementing canonicalize
* tag TODO
* implement a test AuxStore
* add allow(unused)
* basic loading and deleting test
* block_entry test function
* add a test for `add_block_entry`
* ensure range is exclusive at end
* test clear()
* test that add_block sets children
* add a test for canonicalize
* Update node/core/approval-voting/src/aux_schema/mod.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Update node/core/approval-voting/src/aux_schema/tests.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Update node/core/approval-voting/src/aux_schema/mod.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* More doc fixes.
* Minor refactorings in the process of #2177
By having everything peer set related depend directly on the enum the
code becomes more clear and it is also straight forward to add more
peersets/protocols as the compiler will complain if you forget to
implement parts of it.
* Add peer set infos on startup properly
For feature real_overseer.
+ Fixes from review. Thanks @coriolinus and @ordian!
* More structure in network-bridge
Some changes, which would have helped me in groking the code faster.
Entry points/public types more to the top. Factored out implementation
in their own files, to clear up the top-level view.
* Get rid of local ProtocolName type definition.
Does not add much at this level.
* Fix tests + import cleanup.
* Make spaces tabs.
* Clarify what correct parameters to send_message are
* Be more less vague in docs of send_message.
* Apply suggestions from code review
Extend copyright on new files to 2021 as well.
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Andronik Ordian <write@reusable.software>
* scheduler: handle re-scheduling around finalization correctly
* also make sure parathreads get cleaned
* run scheduling in finalization
* Remove stray println!
* Update the schedule call site in inclusion inherent
* Clarify subtlety around SessionStartBlock
* Remove double semi-colon
* reschedule prior to `availability_cores` and in on-initialize
* improve docs
* fix line
* more doc reformat
* remove unneeded call
* avoid unnecessary scheduling on initialize
* split `clear` and `schedule
* Update runtime/parachains/src/scheduler.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* Store all chunks and in a single transaction
* Adds chunks LRU to store
* Add pruning records metrics
* Use honest cache instead of LRU
* Remove unnecessary optional cache
* Fix review nits that are fixable
* Cont.: Implement the state root obtaining during inclusion
During inclusion now we obtain the storage root by passing it through
the inclusion_inherent.
* Fix tests
* Bump rococo spec version
* Reorder the parent header into the end
of the inclusion inherent.
When the parent header is in the beginning, it shifts the other two
fields, so that a previous version won't be able to decode that. If
we put the parent header in the end, the other two fields will stay
at their positions, thus make it possible to decode with the previous
version.
That allows us to perform upgrade of rococo runtime without needing of
simultanuous upgrade of nodes and runtime, or restart of the network.
* Squash a stray tab
* Don't initialize block when calling runtime APIs
* Adapt check_validation_outputs
We split the code path for the inclusion and for the commitments checking.
* Slap #[skip_initialize_block] on safe runtime APIs
That is, those that should not be affected by this attribute
* Make `Scheduled` not ephemeral
So that it is persisted in the storage and ready to be inspected
by the runtime APIs. This is in contrast to what was before, where we
would remove the storage entry and then rely on the scheduling performed
by `on_initialize` again.
* Add a big fat comment
* Typos
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* Move session change to the end of the current block
Previously, it was the beginning of the next block. This allows us to
put #[skip_initialize_block]
* Update tests
* Fix a test in paras registrar
Also refactor it a bit so the next time there are more chances this kind
of issue is diagnosed quicker.
* Add for_runtime_api to inclusion's check_validation_outputs
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* add candidate hash statement circulation span
* add relay-parent to hash-span
* Some typos and misspellings in docs I found, during my studies. (#2144)
* Fix stale link to overseer docs
* Some typos and mispellings in docs/comments
I found during studying how Polkadot works.
* Rococo V1 (#2141)
* Update to latest master and use 30 minutes sessions
* add bootnodes to chainspec
* Update Substrate
* Update chain-spec
* Update Cargo.lock
* GENESIS
* Change session length to one hour
* Bump spec_version to not fuck anything up ;)
Co-authored-by: Erin Grasmick <erin@parity.io>
* avoid creating duplicate unbacked spans when we see extra statements (#2145)
* improve jaeger spans for statement distribution
* tweak and add failing test for repropagation
* make a change that gets the test passing
* guide: clarify
* remove semicolon
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Erin Grasmick <erin@parity.io>
* guide: add candidate information to OccupiedCore
* add descriptor and hash to occupied core type
* guide: add candidate hash to inclusion
* runtime: return candidate info in core state
* bitfield signing: stop querying runtime as much
* minimize going to runtime in availability distribution
* fix availability distribution tests
* guide: remove para ID from Occupied core
* get all crates compiling
* Fix bug and further optimizations in availability distribution
- There was a bug that resulted in only getting one candidate per block
as the candidates were put into the hashmap with the relay block hash as
key. The solution for this is to use the candidate hash and the relay
block hash as key.
- We stored received/sent messages with the candidate hash and chunk
index as key. The candidate hash wasn't required in this case, as the
messages are already stored per candidate.
* Update node/core/bitfield-signing/src/lib.rs
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* Remove the reverse map
* major refactor of receipts & query_live
* finish refactoring
remove ancestory mapping,
improve relay-parent cleanup & receipts-cache cleanup,
add descriptor to `PerCandidate`
* rename and rewrite query_pending_availability
* add a bunch of consistency tests
* Add some last changes
* xy
* fz
* Make it compile again
* Fix one test
* Fix logging
* Remove some buggy code
* Make tests work again
* Move stuff around
* Remove dbg
* Remove state from test_harness
* More refactor and new test
* New test and fixes
* Move metric
* Remove "duplicated code"
* Fix tests
* New test
* Change break to continue
* Update node/core/av-store/src/lib.rs
* Update node/core/av-store/src/lib.rs
* Update node/core/bitfield-signing/src/lib.rs
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
* update guide to match live_candidates changes
* add comment
* fix bitfield signing
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
* refactor View to include finalized_number
* guide: update the NetworkBridge on BlockFinalized
* av-store: fix the tests
* actually fix tests
* grumbles
* ignore macro doctest
* use Hash::repeat_bytes more consistently
* broadcast empty leaves updates as well
* fix issuing view updates on empty leaves updates
* plumbing for rewarding backers
* give validators reward points for participating
* fix tests
* add bitfield rewarding
* add mocks for backing rewards
* add testing for backing & availability rewards
* implement RewardValidators on top of staking
* add to test-runtime and rococo
* add to test-runtime & rococo
* point to source on rewards values
* fix common tests
* do not reward availability anymore
* Adds consistency checks for the `HostConfiguration`
Besides that it fixes the chain specs to make the consistency checks happy.
* Update runtime/parachains/src/configuration.rs
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
* Review feedback and test fixes etc
* Update
* More
* I'm an idiot
* Fix tests...
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
* guide: non-semantic changes
* guide: update per the issue description
* GetBackedCandidates operates on multiple hashes now
* GetBackedCandidates still needs a relay parent
* implement changes specified in guide
* distinguish between various occasions for canceled oneshots
* add tracing info to getbackedcandidates
* REVERT ME: add tracing messages for GetBackedCandidates
Note that these messages are only sometimes actually passed on to the
candidate backing subsystem, with the consequence that it is
unexpectedly frequent that the provisioner fails to create its
provisionable data.
* REVERT ME: more tracing logging
* REVERT ME: log when CandidateBackingJob receives any message at all
* REVERT ME: log when send_msg sends a message to a job
* fix candidate-backing tests
* streamline GetBackedCandidates
This uses table.attested_candidate instead of table.get_candidate, because
it's not obvious how to get a BackedCandidate from just a
CommittedCandidateReceipt.
* REVERT ME: more logging tracing job lifespans
* promote warning about job premature demise
* don't terminate CandiateBackingJob::run_loop in event of failure to process message
* Revert "REVERT ME: more logging tracing job lifespans"
This reverts commit 7365f2fb3dec988d95cfcd317eba75587fe7fd16.
* Revert "REVERT ME: log when send_msg sends a message to a job"
This reverts commit 58e46aad038e6517d6d56390c8be65b046a21884.
* Revert "REVERT ME: log when CandidateBackingJob receives any message at all"
This reverts commit 0d6f38413c7c66b5e9e81dabc587906fa9f82656.
* Revert "REVERT ME: more tracing logging"
This reverts commit 675fd2628e84d1596965280e7314155ef21b28e6.
* Revert "REVERT ME: add tracing messages for GetBackedCandidates"
This reverts commit e09e156493430b33b6c8ab4b5cedb3f2f91afd51.
* formatting
* add logging message to CandidateBackingJob::run_loop start
* REVERT ME: add tracing to candidate-backing job creation
* run candidatebacking loop even if no assignment
* use unique error variants for each canceled oneshot
* Revert "REVERT ME: add tracing to candidate-backing job creation"
This reverts commit 8ce5f4f0bd7186dade134b118751480f72ea1fd6.
* try_runtime_api more to reduce silent exits
* add sanity check that returned backed candidates preserve ordering
* remove redundant err attribute
* guide: fix formatting for SessionInfo module
* primitives: SessionInfo type
* punt on approval keys
* ah, revert the type alias
* session info runtime module skeleton
* update the guide
* runtime/configuration: sync with the guide
* runtime/configuration: setters for newly added fields
* runtime/configuration: set codec indexes
* runtime/configuration: update test
* primitives: fix SessionInfo definition
* runtime/session_info: initial impl
* runtime/session_info: use initializer for session handling (wip)
* runtime/session_info: mock authority discovery trait
* guide: update the initializer's order
* runtime/session_info: tests skeleton
* runtime/session_info: store n_delay_tranches in Configuration
* runtime/session_info: punt on approval keys
* runtime/session_info: add some basic tests
* Update primitives/src/v1.rs
* small fixes
* remove codec index annotation on structs
* fix off-by-one error
* validator_discovery: accept a session index
* runtime: replace validator_discovery api with session_info
* Update runtime/parachains/src/session_info.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* runtime/session_info: add a comment about missing entries
* runtime/session_info: define the keys
* util: expose connect_to_past_session_validators
* util: allow session_info requests for jobs
* runtime-api: add mock test for session_info
* collator-protocol: add session_index to test state
* util: fix error message for runtime error
* fix compilation
* fix tests after merge with master
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* skeleton flow control
* tweaks & rename to approvals distribution
* Update roadmap/implementers-guide/src/node/approval/approval-distribution.md
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Update roadmap/implementers-guide/src/node/approval/approval-distribution.md
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* add a `NewBlocks` message and dispatch
* new data format for approval distribution
* guide: update view to include finalized block number
* approvals: document view updating
* pruning when peers disconnect
* add remaining message types
* fix link
* network message type
* handle incoming assignments
* import_and_circulate_approval
* handle new blocks
* address review comments
* address review comments and use nifty VRFProof
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* elaborate on runtime API
* clarify what to do if the runtime API calls fail
* Update roadmap/implementers-guide/src/node/approval/approval-voting.md
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Update roadmap/implementers-guide/src/node/approval/approval-voting.md
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* use snake_case for log targets
* remove unused continue
* validator_discovery: when disconnecting, use all addresses
* validator_discovery: simplify request revokation
* fix a typo
* Guide: Split router module in guide.
Now we have: DMP, UMP and Router module.
* Add a glossary entry for what used to be called Router
* Extract DMP
* Extract UMP
* Extract HRMP
* Switch over to new modules
* Router: goodbye sweet prince
* Link to messaging overview for details.
* Update missed rococo and test runtimes.
* Commit destroyed by rebase changes
* Don't deprecate Router but rather make it a meta-project
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* Fix typos suggestion
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* Fix repetition in the impl guide
* Clarify that processed_downward_messages has the u32 type
* Remove the router subdir.
* Deabbreviate DMP,UMP,HRMP
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* HRMP: Update the impl guide
* HRMP: Incorporate the channel notifications into the guide
* HRMP: Renaming in the impl guide
* HRMP: Constrain the maximum number of HRMP messages per candidate
This commit addresses the HRMP part of https://github.com/paritytech/polkadot/issues/1869
* XCM: Introduce HRMP related message types
* HRMP: Data structures and plumbing
* HRMP: Configuration
* HRMP: Data layout
* HRMP: Acceptance & Enactment
* HRMP: Test base logic
* Update adder collator
* HRMP: Runtime API for accessing inbound messages
Also, removing some redundant fully-qualified names.
* HRMP: Add diagnostic logging in acceptance criteria
* HRMP: Additional tests
* Self-review fixes
* save test refactorings for the next time
* Missed a return statement.
* a formatting blip
* Add missing logic for appending HRMP digests
* Remove the channel contents vectors which became empty
* Tighten HRMP channel digests invariants.
* Apply suggestions from code review
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Remove a note about sorting for channel id
* Add missing rustdocs to the configuration
* Clarify and update the invariant for HrmpChannelDigests
* Make the onboarding invariant less sloppy
Namely, introduce `Paras::is_valid_para` (in fact, it already is present
in the implementation) and hook up the invariant to that.
Note that this says "within a session" because I don't want to make it
super strict on the session boundary. The logic on the session boundary
should be extremely careful.
* Make `CandidateCheckContext` use T::BlockNumber for hrmp_watermark
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
We need to distribute the PoV after we have seconded it. Other nodes
that will receive our `Secondded` statement and want to validate the
candidate another time will request this PoV from us.
* Make `CandidateHash` a real type
This pr adds a new type `CandidateHash` that is used instead of the
opaque `Hash` type. This helps to ensure on the type system level that
we are passing the correct types.
This pr also fixes wrong usage of `relay_parent` as `candidate_hash`
when communicating with the av storage.
* Update core-primitives/src/lib.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Wrap the lines
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Do not validate a candidate in candidate selection
The candidate selection subsystem should not validate a candidate, as
this is done by the backing subsystem on a `Second` request. Otherwise
we validate one candidate twice.
* Update candidate-selection.md
Robert Habermeier