* Update shared-memory to new version & refactor
This two are combined in a single commit because the new version of
shared-memory doesn't provide the used functionality anymore.
Therefore in order to update the version of this crate we implement the
functionality that we need by ourselves, providing a cleaner API along
the way.
* Significantly decrease the required memory for a workspace
For some reason it was allocating an entire GiB of memory. I suspect
this has something to do with the current memory size limit of a PVF
execution environment (the prior name suggests that). However, we don't
need so much memory anywhere near that amount.
In fact, we could reduce the allocated size even more, but that maybe
for the next time.
* Unlink shmem just after opening
That will make sure that we don't leak the shmem accidentally.
* Do not compile workspace mod for androind and wasm
* Address some review comments
* Fix the test runner
* Fix missed +1 for the attached flag
* Use .expect rather than .unwrap
* Add a rustdoc for the workspace module
* fixup! Use .expect rather than .unwrap
* Add some doc comments to pub members
* Warn on error removing shm_unlink
* Change the alignment implementation
* Fix the comment nit
* PVD: `block_number`->`relay_parent_number`
* ValidationParams: `relay_chain_height`->`relay_parent_number`
* Expose DMQ MQC hash as a well-known-key
This way the relay storage merkle proofs will be able to obtain the DMQ
MQC hash and we will be able to remove the it from the
PersistedValidationData struct.
* PersistedValidationData: Remove HRMP MQC heads
* PersistedValidationData: Remove `dmq_mqc_head`
* Expose the HRMP ingress channel index as a well-known-key
This way a parachain (PVF and collator) can find all the parachains that
have an outbound channel to the given one. That allows in turn to find
all the inbound channels for the given para.
Having access to that allows the parachain to get the same information
as the hrmp_mqc_heads now provide.
* Rename `relay_storage_root` to `relay_parent_storage_root`
* Improve logging to make debugging parachains easier
This pr should make debugging parachains easier, by printing more
information about the validation process.
* 🤦
* moare
* Convert to debug
* Drive by fixes
The visibility modifiers are remnants of the previous structure where
HRMP wasn't a standalone module, by rather a submodule of the router
module.
* Add Currency assoc type to Config
This would allow us to reserve balance for deposits. This commit also
integrates the HRMP module in rococo, test-runtime and mocks to use the
balances pallet.
* Fix a bug that doesn't increment the age
In case the request is not confirmed, the age would be incremented but
not persisted.
* Fix cleaning the indexes
Before that change, the cleaning of the channel indexes was wrong, because it
naively removed entire rows that was pertaining to the para we delete.
This approach is flawed because it doesn't account for the rows that are
pertaining to other paras that contain the outgoing one.
This clearly violates the invariant imposed on the indexes, that all
the index rows must contain alive paras, but apart from that it also
lead to the situation where ingress index would contain the a different
set of channels that an egress have.
* Reserve currency for opening the channels
Note the ugly `unique_saturated_into` calls. The reason for them is the
currency trait accepts and defines the `Balance` associated type and the
deposit values are coming from the `HostConfiguration` where they are
defined using the `Balance`.
I figured that parameterising `HostConfiguration` would be annoying. On
the other hand, I don't expect these `unique_saturated_into` calls to
give us problems since it seems to be a reasonable assumption that this
module will be instantiated within a runtime where the Currency provided
will have a Balance that matches the one used in the configuration.
* Tests: Adapt `run_to_block` so that it submits a proper config
* Tests: exercise the deposit logic
* Rename ExecutionMode to IsolationStrategy
Execution mode is too generic name and can imply a lot of different
aspects of execution. The notion of isolation better describes the
meant aspect.
And while I am at it, I also renamed mode -> strategy cause it seems a
bit more appropriate, although that is way more subjective.
* Fix compilation in wasm_executor tests.
* Add a comment to IsolationStrategy
* Update comments on IsolationStrategy
* Update node/core/candidate-validation/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Accomodate the point on interruption
* Update parachain/src/wasm_executor/mod.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* Naming nits
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
* Adds integration test based on adder collator
This adds an integration test for parachains that uses the adder
collator. The test will start two relay chain nodes and one collator and
waits until 4 blocks are build and enacted by the parachain.
* Make sure the integration test is run in CI
* Fix wasm compilation
* Update parachain/test-parachains/adder/collator/src/lib.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* Update cli/src/command.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* HRMP: Update the impl guide
* HRMP: Incorporate the channel notifications into the guide
* HRMP: Renaming in the impl guide
* HRMP: Constrain the maximum number of HRMP messages per candidate
This commit addresses the HRMP part of https://github.com/paritytech/polkadot/issues/1869
* XCM: Introduce HRMP related message types
* HRMP: Data structures and plumbing
* HRMP: Configuration
* HRMP: Data layout
* HRMP: Acceptance & Enactment
* HRMP: Test base logic
* Update adder collator
* HRMP: Runtime API for accessing inbound messages
Also, removing some redundant fully-qualified names.
* HRMP: Add diagnostic logging in acceptance criteria
* HRMP: Additional tests
* Self-review fixes
* save test refactorings for the next time
* Missed a return statement.
* a formatting blip
* Add missing logic for appending HRMP digests
* Remove the channel contents vectors which became empty
* Tighten HRMP channel digests invariants.
* Apply suggestions from code review
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Remove a note about sorting for channel id
* Add missing rustdocs to the configuration
* Clarify and update the invariant for HrmpChannelDigests
* Make the onboarding invariant less sloppy
Namely, introduce `Paras::is_valid_para` (in fact, it already is present
in the implementation) and hook up the invariant to that.
Note that this says "within a session" because I don't want to make it
super strict on the session boundary. The logic on the session boundary
should be extremely careful.
* Make `CandidateCheckContext` use T::BlockNumber for hrmp_watermark
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* UMP: Update the impl guide
* UMP: Incorporate XCM related changes into the guide
* UMP: Data structures and configuration
* UMP: Initial plumbing
* UMP: Data layout
* UMP: Acceptance criteria & enactment
* UMP: Fix dispatcher bug and add the test for it
* UMP: Constrain the maximum size of an UMP message
This commit addresses the UMP part of https://github.com/paritytech/polkadot/issues/1869
* Fix failing test due to misconfiguration
* Make the type of RelayDispatchQueueSize be more apparent in the guide
* Revert renaming `max_upward_queue_capacity` to `max_upward_queue_count`
* convert spaces to tabs
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* Update runtime/parachains/src/router/ump.rs
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* start working on building the real overseer
Unfortunately, this fails to compile right now due to an upstream
failure to compile which is probably brought on by a recent upgrade
to rustc v1.47.
* fill in AllSubsystems internal constructors
* replace fn make_metrics with Metrics::attempt_to_register
* update to account for #1740
* remove Metrics::register, rename Metrics::attempt_to_register
* add 'static bounds to real_overseer type params
* pass authority_discovery and network_service to real_overseer
It's not straightforwardly obvious that this is the best way to handle
the case when there is no authority discovery service, but it seems
to be the best option available at the moment.
* select a proper database configuration for the availability store db
* use subdirectory for av-store database path
* apply Basti's patch which avoids needing to parameterize everything on Block
* simplify path extraction
* get all tests to compile
* Fix Prometheus double-registry error
for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:
```rust
Some(registry) => Self::try_register(registry).map_err(|err| {
eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
err
}),
```
That pointed out where the registration was failing, which led to
this fix. The test still doesn't pass, but it now fails in a new
and different way!
* authorities must have authority discovery, but not necessarily overseer handlers
* fix broken SpawnedSubsystem impls
detailed logging determined that using the `Box::new` style of
future generation, the `self.run` method was never being called,
leading to dropped receivers / closed senders for those subsystems,
causing the overseer to shut down immediately.
This is not the final fix needed to get things working properly,
but it's a good start.
* use prometheus properly
Prometheus lets us register simple counters, which aren't very
interesting. It also allows us to register CounterVecs, which are.
With a CounterVec, you can provide a set of labels, which can
later be used to filter the counts.
We were using them wrong, though. This pattern was repeated in a
variety of places in the code:
```rust
// panics with an cardinality mismatch
let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
my_counter.with_label_values(&["succeeded"]).inc()
```
The problem is that the labels provided in the constructor are not
the set of legal values which can be annotated, but a set of individual
label names which can have individual, arbitrary values.
This commit fixes that.
* get av-store subsystem to actually run properly and not die on first signal
* typo fix: incomming -> incoming
* don't disable authority discovery in test nodes
* Fix rococo-v1 missing session keys
* Update node/core/av-store/Cargo.toml
* try dummying out av-store on non-full-nodes
* overseer and subsystems are required only for full nodes
* Reduce the amount of warnings on browser target
* Fix two more warnings
* InclusionInherent should actually have an Inherent module on rococo
* Ancestry: don't return genesis' parent hash
* Update Cargo.lock
* fix broken test
* update test script: specify chainspec as script argument
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update node/service/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* node/service/src/lib: Return error via ? operator
* post-merge blues
* add is_collator flag
* prevent occasional av-store test panic
* simplify fix; expand application
* run authority_discovery in Role::Discover when collating
* distinguish between proposer closed channel errors
* add IsCollator enum, remove is_collator CLI flag
* improve formatting
* remove nop loop
* Fix some stuff
* Adds test parachain adder collator
* Add sudo to Rococo, change session length to 30 seconds and some renaming
* Update to the latest changes on master
* Some fixes
* Fix compilation
* Update parachain/test-parachains/adder/collator/src/lib.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* Review comments
* Downgrade transaction version
* Fixes
* MOARE
* Register notification protocols
* utils: remove unused error
* av-store: more resilient to some errors
* address review nits
* address more review nits
Co-authored-by: Peter Goodspeed-Niklaus <peter.r.goodspeedniklaus@gmail.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
Co-authored-by: Robert Habermeier <robert@Roberts-MBP.lan1>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Sergey Shulepov <s.pepyakin@gmail.com>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* DMP: data structures and plumbing
* DMP: Implement DMP logic in the router module
DMP: Integrate DMP parts into the inclusion module
* DMP: Introduce the max size limit for the size of a downward message
* DMP: Runtime API for accessing inbound messages
* OCD
Small clean ups
* DMP: fix the naming of the error
* DMP: add caution about a non-existent recipient
* update primitives
* correct parent_head field
* make hrmp field pub
* refactor validation data: runtime
* refactor validation data: messages
* add arguments to full_validation_data runtime API
* port runtime API
* mostly port over candidate validation
* remove some parameters from ValidationParams
* guide: update candidate validation
* update candidate outputs
* update ValidationOutputs in primitives
* port over candidate validation
* add a new test for no-transient behavior
* update util runtime API wrappers
* candidate backing
* fix missing imports
* change some fields of validation data around
* runtime API impl
* update candidate validation
* fix backing tests
* grumbles from review
* fix av-store tests
* fix some more crates
* fix provisioner tests
* fix availability distribution tests
* port collation-generation to new validation data
* fix overseer tests
* Update roadmap/implementers-guide/src/node/utility/candidate-validation.md
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Make parachain validation wasm executor functional
- Increase the size of the validation result in the shared memory. The
validation result holds the new runtime when a runtime upgrade is
scheduled. So, we need to give it enough memory to send the data between
the validator and the wasm execution host.
- Add the `CallInWasmExt`. This is required when doing a runtime upgrade
to check that we upgrade to something meaningful.
* Update parachain/src/wasm_executor/mod.rs
* Update parachain/src/wasm_executor/mod.rs
Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
* update guide to reduce confusion and TODOs
* work from previous bitfield signing effort
There were large merge issues with the old bitfield signing PR, so
we're just copying all the work from that onto this and restarting.
Much of the existing work will be discarded because we now have better
tools available, but that's fine.
* start rewriting bitfield signing in terms of the util module
* implement construct_availability_bitvec
It's not an ideal implementation--we can make it much more concurrent--
but at least it compiles.
* implement the unimplemented portions of bitfield signing
* get core availability concurrently, not sequentially
* use sp-std instead of std for a parachain item
* resolve type inference failure caused by multiple From impls
* handle bitfield signing subsystem & Allmessages variant in overseer
* fix more multi-From inference issues
* more concisely handle overflow
Co-authored-by: Andronik Ordian <write@reusable.software>
* Revert "resolve type inference failure caused by multiple From impls"
This reverts commit 7fc77805de5e5074a1b01037f8d4e3919e03e0e1.
* Revert "fix more multi-From inference issues"
This reverts commit f14ffe589e20d664d8a900ed62f68b6fb844a514.
* impl From<i32> for ParaId
* handle another instance of AllSubsystems
* improve consistency when returning existing options
Co-authored-by: Andronik Ordian <write@reusable.software>
* Downward messages, the front-end.
* Move types around to make them accessible from Parachains
* Fix compilation
* Fix branch
* Make it compile for Cumulus
* Update the branch names
* Add default generic parameter
* Implement `Partialeq`
* Move upward messages into the `ValidationResult`
* Support disabling of the runtime api
* Update branch
* Adds support for handling downward messages
* Implement sending XCMP messages as up/downward messages
* service: update to latest ServiceBuilder changes
* Make it compile
* Initial commit
Forked at: ef2aa428d7
Parent branch: origin/master
* Update substrate branch to cecton-update-polkadot-substrate
* Update substrate & polkadot to cumulus-branch
* Reset branch
* Update primitives/src/parachain.rs
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* Update runtime/common/src/parachains.rs
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* Update runtime/common/src/parachains.rs
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* Minor fixes
* Fix wasm build
Co-authored-by: Gav Wood <gavin@parity.io>
Co-authored-by: André Silva <andre.beat@gmail.com>
Co-authored-by: Cecile Tonglet <cecile.tonglet@cecton.com>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* add new apis to externalities
* Update mod.rs
* update for new host function
* Rename unused
* cargo update -p sp-io
* Add whitelists
* Apply changes to polkadot runtime as well
Co-authored-by: kianenigma <kian.peymani@gmail.com>
* upgrade primitives to allow changing validation function
* set up storage schema for old parachains code
* fix compilation errors
* fix test compilation
* add some tests for past code meta
* most of the runtime logic for code upgrades
* implement old-code pruning
* add a couple tests
* clean up remaining TODOs
* add a whole bunch of tests for runtime functionality
* remove unused function
* fix runtime compilation
* extract some primitives to parachain crate
* add validation-code upgrades to validation params and result
* extend validation params with code upgrade fields
* provide maximums to validation params
* port test-parachains
* add a code-upgrader test-parachain and tests
* fix collator tests
* move test-parachains to own folder to work around compilation errors
* fix test compilation
* update the Cargo.lock
* fix parachains tests
* remove dbg! invocation
* use new pool in code-upgrader
* bump lockfile
* link TODO to issue
* Flag to force kusama runtime
* Chainspecs for kusama
* Polkadot config for westend
Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>
* network/src/legacy/gossip: Wrap GossipEngine in Arc Mutex & lock it on use
`GossipEngine` in itself has no need to be Send and Sync, given that it
does not rely on separately spawned background tasks anymore.
`RegisteredMessageValidator` needs to be `Send` and `Sync` due to the
inherited trait bounds from implementing `GossipService`. In addition
`RegisteredMessageValidator` derives `Clone`. Thereby `GossipEngine`
needs to be wrapped in an `Arc` and `Mutex` to keep the status quo.
* Needed fixes.
* Fixes
* Fixed build
* Fixed build w benchmarking CLI
* Fixed building tests
* Added --dev shortcut
Co-authored-by: arkpar <arkady.paronyan@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Max Inden <mail@max-inden.de>
Andronik Ordian