* guide: fix formatting for SessionInfo module
* primitives: SessionInfo type
* punt on approval keys
* ah, revert the type alias
* session info runtime module skeleton
* update the guide
* runtime/configuration: sync with the guide
* runtime/configuration: setters for newly added fields
* runtime/configuration: set codec indexes
* runtime/configuration: update test
* primitives: fix SessionInfo definition
* runtime/session_info: initial impl
* runtime/session_info: use initializer for session handling (wip)
* runtime/session_info: mock authority discovery trait
* guide: update the initializer's order
* runtime/session_info: tests skeleton
* runtime/session_info: store n_delay_tranches in Configuration
* runtime/session_info: punt on approval keys
* runtime/session_info: add some basic tests
* Update primitives/src/v1.rs
* small fixes
* remove codec index annotation on structs
* fix off-by-one error
* validator_discovery: accept a session index
* runtime: replace validator_discovery api with session_info
* Update runtime/parachains/src/session_info.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* runtime/session_info: add a comment about missing entries
* runtime/session_info: define the keys
* util: expose connect_to_past_session_validators
* util: allow session_info requests for jobs
* runtime-api: add mock test for session_info
* collator-protocol: add session_index to test state
* util: fix error message for runtime error
* fix compilation
* fix tests after merge with master
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* Guide: Split router module in guide.
Now we have: DMP, UMP and Router module.
* Add a glossary entry for what used to be called Router
* Extract DMP
* Extract UMP
* Extract HRMP
* Switch over to new modules
* Router: goodbye sweet prince
* Link to messaging overview for details.
* Update missed rococo and test runtimes.
* Commit destroyed by rebase changes
* Don't deprecate Router but rather make it a meta-project
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* Fix typos suggestion
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* Fix repetition in the impl guide
* Clarify that processed_downward_messages has the u32 type
* Remove the router subdir.
* Deabbreviate DMP,UMP,HRMP
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* HRMP: Update the impl guide
* HRMP: Incorporate the channel notifications into the guide
* HRMP: Renaming in the impl guide
* HRMP: Constrain the maximum number of HRMP messages per candidate
This commit addresses the HRMP part of https://github.com/paritytech/polkadot/issues/1869
* XCM: Introduce HRMP related message types
* HRMP: Data structures and plumbing
* HRMP: Configuration
* HRMP: Data layout
* HRMP: Acceptance & Enactment
* HRMP: Test base logic
* Update adder collator
* HRMP: Runtime API for accessing inbound messages
Also, removing some redundant fully-qualified names.
* HRMP: Add diagnostic logging in acceptance criteria
* HRMP: Additional tests
* Self-review fixes
* save test refactorings for the next time
* Missed a return statement.
* a formatting blip
* Add missing logic for appending HRMP digests
* Remove the channel contents vectors which became empty
* Tighten HRMP channel digests invariants.
* Apply suggestions from code review
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Remove a note about sorting for channel id
* Add missing rustdocs to the configuration
* Clarify and update the invariant for HrmpChannelDigests
* Make the onboarding invariant less sloppy
Namely, introduce `Paras::is_valid_para` (in fact, it already is present
in the implementation) and hook up the invariant to that.
Note that this says "within a session" because I don't want to make it
super strict on the session boundary. The logic on the session boundary
should be extremely careful.
* Make `CandidateCheckContext` use T::BlockNumber for hrmp_watermark
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
If a core is freed, we need to make sure it is inserted as free into
availability cores, because we can not be sure that the core is occupied
directly again.
* Parachain improvements
- Set the parachains configuration in Rococo genesis
- Don't stop the overseer when a subsystem job is stopped
- Several small code changes
* Remove unused functionality
* Return error from the runtime instead of printing it
* Apply suggestions from code review
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Update primitives/src/v1.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Update primitives/src/v1.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Fix test
* Revert "Update primitives/src/v1.rs"
This reverts commit 11fce2785acd1de481ca57815b8e18400f09fd52.
* Revert "Update primitives/src/v1.rs"
This reverts commit d6439fed4f954360c89fb1e12b73954902c76a41.
* Revert "Return error from the runtime instead of printing it"
This reverts commit cb4b5c0830ac516a6d54b2c24197e9354f2b98cb.
* Revert "Fix test"
This reverts commit 0c5fa1b5566d4cd3c55a55d485e707165ce7a59e.
* Update runtime/parachains/src/runtime_api_impl/v1.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* add native logging to check_upward_messages
This doesn't affect the WASM builds, but ensures that native versions
(such as are used for testing) emit diagnostics anytime the
check_upward_messages function exits, which should reduce the pain
of debugging when something goes wrong.
* Apply suggestions from code review
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* verb tense
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* fix: ensure candidate validation gets code based on occupied core assumption
* guide: runtime API for historical validation code
* add historical runtime API
* integrate into runtime API subsystem
* remove blocked TODO
* fix service build: enable notifications protocol only under real overseer
* Update node/subsystem/src/messages.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* fix compilation
Co-authored-by: Robert Habermeier <robert@Roberts-MacBook-Pro.local>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* UMP: Update the impl guide
* UMP: Incorporate XCM related changes into the guide
* UMP: Data structures and configuration
* UMP: Initial plumbing
* UMP: Data layout
* UMP: Acceptance criteria & enactment
* UMP: Fix dispatcher bug and add the test for it
* UMP: Constrain the maximum size of an UMP message
This commit addresses the UMP part of https://github.com/paritytech/polkadot/issues/1869
* Fix failing test due to misconfiguration
* Make the type of RelayDispatchQueueSize be more apparent in the guide
* Revert renaming `max_upward_queue_capacity` to `max_upward_queue_count`
* convert spaces to tabs
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* Update runtime/parachains/src/router/ump.rs
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* DMP: data structures and plumbing
* DMP: Implement DMP logic in the router module
DMP: Integrate DMP parts into the inclusion module
* DMP: Introduce the max size limit for the size of a downward message
* DMP: Runtime API for accessing inbound messages
* OCD
Small clean ups
* DMP: fix the naming of the error
* DMP: add caution about a non-existent recipient
* annoying whitespaces
* update guide
Add `CheckValidationOutputs` runtime api and also change the
candidate-validation stuff
* promote ValidationOutputs to global primitives
i.e. move it from node specific primitives to global v1 primitives. This
will be needed when we share it later in the runtime inclusion module
* refactor acceptance checks in the inclusion module
factor out the common code to share it during the block inclusion and
for the forthcoming CheckValidationOutputs runtime api.
Also note that the acceptance criteria was updated to incorporate checks
that exist now in candidate-validation
* plumb the runtime api outside
* extract validation_data from ValidationOutputs
* use runtime-api to check validation outputs
apart from that refactor, update docs and tidy a bit
* Update the maxium code size
This is to fix a test that performs an upgrade.
* Make register/deregister parachain dispatchables
This makes `register_parachain` and `deregister_parachain` of
`Registrar` dispatchables.
Besides that it brings back the functionality of the test node to
register a parachain.
* Fix tests
* PR review comments
* Remove old service, 3rd try
i.e.
Revert "Revert "Remove Old Service, 2nd try (#1732)" (#1758)"
This reverts commit 9a0f08bfe1.
Closes#1757.
We now have some evidence that the polkadot validator was producing
blocks after all; the reason the blocks_constructed metric was 0 was
that as a new metric it hadn't yet been incorporated into that
branch's codebase. See
https://github.com/paritytech/polkadot/issues/1757#issuecomment-700977602
As this PR is based on a newer `master` branch than the previous one,
that should hopefully no longer be an issue.
* paras trait now has an Origin type
* initial work running a two node local net
* use the right incantations so the nodes produce blocks together
* improve internal documentation
Co-authored-by: Bastian Köcher <git@kchr.de>
* stupid, but it compiles
* redo
* cleanup
* add ValidatorDiscovery to msgs
* sketch network bridge code
* ConnectToAuthorities instead of validators
* more stuff
* cleanup
* more stuff
* complete ConnectToAuthoritiesState
* Update node/network/bridge/src/lib.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Collator protocol subsystem (#1659)
* WIP
* The initial implementation of the collator side.
* Improve comments
* Multiple collation requests
* Add more tests and comments to validator side
* Add comments, remove dead code
* Apply suggestions from code review
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Fix build after suggested changes
* Also connect to the next validator group
* Remove a Future impl and move TimeoutExt to util
* Minor nits
* Fix build
* Change FetchCollations back to FetchCollation
* Try this
* Final fixes
* Fix build
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* handle multiple in-flight connection requests
* handle cancelled requests
* Update node/core/runtime-api/src/lib.rs
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* redo it again
* more stuff
* redo it again
* update comments
* workaround Future is not Send
* fix trailing spaces
* clarify comments
* bridge: fix compilation in tests
* update more comments
* small fixes
* port collator protocol to new validator discovery api
* collator tests compile
* collator tests pass
* do not revoke a request when the stream receiver is closed
* make revoking opt-in
* fix is_fulfilled
* handle request revokation in collator
* tests
* wait for validator connections asyncronously
* fix compilation
* relabel my todos
* apply Fedor's patch
* resolve reconnection TODO
* resolve revoking TODO
* resolve channel capacity TODO
* resolve peer cloning TODO
* resolve peer disconnected TODO
* resolve PeerSet TODO
* wip tests
* more tests
* resolve Arc TODO
* rename pending to non_revoked
* one more test
* extract utility function into util crate
* fix compilation in tests
* Apply suggestions from code review
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
* revert pin_project removal
* fix while let loop
* Revert "revert pin_project removal"
This reverts commit ae7f529d8de982ef66c3007dd1ff74c6ddce80d2.
* fix compilation
* Update node/subsystem/src/messages.rs
* docs on pub items
* guide updates
* remove a TODO
* small guide update
* fix a typo
* link to the issue
* validator discovery: on_request docs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* update primitives
* correct parent_head field
* make hrmp field pub
* refactor validation data: runtime
* refactor validation data: messages
* add arguments to full_validation_data runtime API
* port runtime API
* mostly port over candidate validation
* remove some parameters from ValidationParams
* guide: update candidate validation
* update candidate outputs
* update ValidationOutputs in primitives
* port over candidate validation
* add a new test for no-transient behavior
* update util runtime API wrappers
* candidate backing
* fix missing imports
* change some fields of validation data around
* runtime API impl
* update candidate validation
* fix backing tests
* grumbles from review
* fix av-store tests
* fix some more crates
* fix provisioner tests
* fix availability distribution tests
* port collation-generation to new validation data
* fix overseer tests
* Update roadmap/implementers-guide/src/node/utility/candidate-validation.md
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* rename GlobalValidationSchedule to GlobalValidationData
* guide: update candidate descriptor to contain validation data hash
* guide: add note in inclusion module about checking validation data hash
* primitives: update CandidateDescriptor to contain new hash
* fix payload computation
* add helpers for computing validation data to runtime modules
* guide: note routines
* inclusion: check validation data hash and fix local_validation_data bug
* add a case to candidate_checks and improve that test substantially
* bump versions
* address review comments
* add a test for including code upgrade
* bump kusama version
* bump westend & polkadot versions
* runtime: rename parachain_rotation_frequency to group_rotation_frequency
* scheduler: handle timeouts for no group rotation
* scheduler: apply fixes from code review
* scheduler: remove my comments
* create a README on Runtime APIs
* add ParaId type
* write up runtime APIs
* more preamble
* rename
* rejig runtime APIs
* add occupied_since to `BlockNumber`
* skeleton crate for runtime API subsystem
* improve group_for_core
* improve docs on availability cores runtime API
* guide: freed -> free
* add primitives for runtime APIs
* create a v1 ParachainHost API trait
* guide: make validation code return `Option`al.
* skeleton runtime API helpers
* make parachain-host runtime-generic
* skeleton for most runtime API implementation functions
* guide: add runtime API helper methods
* implement new helpers of the inclusion module
* guide: remove retries check, as it is unneeded
* implement helpers for scheduler module for Runtime APIs
* clean up `validator_groups` implementation
* implement next_rotation_at and last_rotation_at
* guide: more helpers on GroupRotationInfo
* almost finish implementing runtime APIs
* add explicit block parameter to runtime API fns
* guide: generalize number parameter
* guide: add group_responsible to occupied-core
* update primitives due to guide changes
* finishing touches on runtime API implementation; squash warnings
* break out runtime API impl to separate file
* add tests for next_up logic
* test group rotation info
* point to filed TODO
* remove unused TODO [now]
* indentation
* guide: para -> para_id
* rename para field to para_id for core meta
* remove reference to outdated AvailabilityCores type
* add an event in `inclusion` for candidates being included or timing out
* guide: candidate events
* guide: adjust language
* Candidate events type from guide and adjust inclusion event
* implement `candidate_events` runtime API
* fix runtime test compilation
* max -> min
* fix typos
* guide: add `RuntimeAPIRequest::CandidateEvents`
* inclusion: split PendingAvailability storage into descriptor and commitments
* inclusion: fix tests
* implementers-guide: update CandidatePendingAvailability type
* inclusion: simplify process_candidates a bit
* implementers-guide: more updates to the inclusion module
* inclusion: fix copy-paste errors in tests
* inclusion: revert some of the changes
* inclusion: lazy commitments loading and a test
* guide: revert enact_candidate changes
* inclusion: test process_bitfield for no commitments
* Grammar
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* create a v1 primitives module
* Improve guide on availability types
* punctuate
* new parachains runtime uses new primitives
* tests of new runtime now use new primitives
* add ErasureChunk to guide
* export erasure chunk from v1 primitives
* subsystem crate uses v1 primitives
* node-primitives uses new v1 primitives
* port overseer to new primitives
* new-proposer uses v1 primitives (no ParachainHost anymore)
* fix no-std compilation for primitives
* service-new uses v1 primitives
* network-bridge uses new primitives
* statement distribution uses v1 primitives
* PoV distribution uses v1 primitives; add PoV::hash fn
* move parachain to v0
* remove inclusion_inherent module and place into v1
* remove everything from primitives crate root
* remove some unused old types from v0 primitives
* point everything else at primitives::v0
* squanch some warns up
* add RuntimeDebug import to no-std as well
* port over statement-table and validation
* fix final errors in validation and node-primitives
* add dummy Ord impl to committed candidate receipt
* guide: update CandidateValidationMessage
* add primitive for validationoutputs
* expand CandidateValidationMessage further
* bikeshed
* add some impls to omitted-validation-data and available-data
* expand CandidateValidationMessage
* make erasure-coding generic over v1/v0
* update usages of erasure-coding
* implement commitments.hash()
* use Arc<Pov> for CandidateValidation
* improve new erasure-coding method names
* fix up candidate backing
* update docs a bit
* fix most tests and add short-circuiting to make_pov_available
* fix remainder of candidate backing tests
* squanching warns
* squanch it up
* some fallout
* overseer fallout
* free from polkadot-test-service hell
* Updates guide for CandidateBacking
* Move assignment types to primitives
* Initial implementation.
* More functionality
* use assert_matches
* Changes to report misbehaviors
* Some fixes after a review
* Remove a blank line
* Update guide and some types
* Adds run_job function
* Some comments and refactorings
* Fix review
* Remove warnings
* Use summary in kicking off validation
* Parallelize requests
* Validation provides local and global validation params
* Test issued validity tracking
* Nits from review
* note that the initializer is responsible for buffering session changes
* amend initializer definition to include session change buffering
* support buffered changes before `on_initialize`
* implement and test session buffering
* Update roadmap/implementors-guide/src/runtime/README.md
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* expand on how this affects misbehavior reports
* fix typo
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* impl ProvideInherent for InclusionInherent
* reduce import churn; correct expect message
* move inclusion inherent identifier into primitives
It's not clear precisely why this is desired, but it's a pattern
I've seen in several places, so I'm going this to be on the
safe side. Worst case, we can revert this commit pretty easily.
* bump kusama spec_version to placate CI
* add license header
* empty commit; maybe github will notice the one with changes
* add sanity check to only include valid inherents
* add signed wrapper, typedef SignedStatement
* typedef SignedAvailabilityBitfield
* implement Signed wrapper
This is strictly an addition as of this commit; nothing is yet
changed in existing behavior.
* inline getters, remove review comment
* move EncodeAs, Signed from node::primitives to primitives::parachain
* Refactor SignedAvailabilityBitfield to use Signed
* don't double-encode real payload
This isn't an ideal solution, because it depends on the
implementation details of how SCALE encodes tuples, but OTOH
that behavior seems unlikely to change anytime soon.
* fix build errors
* cause the runtime to build properly with the new changes
Not sure why cargo check didn't catch this earlier; oh well.
* fix runtime tests and separate SignedStatement from SignedFullStatement
* better explain why CompactStatement exists
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* add availability bitfield types to primitives
* begin inclusion module
* use GitHub issue link for limitation
* fix some compiler errors
* integrate validators into initializer
* add generic signing context
* make signing-context more generic
* fix issues with inclusion module
* add TODO
* guide: add validators and session index to inclusion
* guide: add session index to change notification
* implement session change logic
* add BackedCandidate type
* guide: refine inclusion pipeline
* guide: rename group_on to group_validators
* guide: add check about collator for parathread
* guide: add last_code_upgrade to paras and use in inclusion
* implement Paras::last_code_upgrade
* implement most checks in process_candidates
* make candidate receipt structs more generic
* make BackedCandidate struct more generic
* use hash param, not block number
* check that candidate is in context of the parent block
* include inclusion module in initializer
* implement enact-candidate
* check that only occupied cores have bits set
* finish implementing bitfield processing
* restructure consistency checks on candidates
* make some more primitives generic
* signature checking logic for backed candidates
* finish implementing process_candidates
* implement collect_pending
* add some trait implementations to primitives
* implement InclusionInherent and squash warnings
* test bitfield signing checks
* rename parachain head to para_head
* fix note_new_head bug in paras
* test bitfield enactment in inclusion
* helpers for candidate checks
* add test for most candidate checks
* add test for backing setting storage
* test session change logic
* remove extraneous type parameter
* remove some allow(unused)s
* extract threshold computation to const fn
* remove some more allow(unused)s
* improve doc
* add debug assertion
* fix primitive test compilation
* tag unanimous variant as unused
Andronik Ordian