* Implement PVF validation host
* WIP: Diener
* Increase the alloted compilation time
* Add more comments
* Minor clean up
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Fix pruning artifact removal
* Fix formatting and newlines
* Fix the thread pool
* Update node/core/pvf/src/executor_intf.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Remove redundant test declaration
* Don't convert the path into an intermediate string
* Try to workaround the test failure
* Use the puppet_worker trick again
* Fix a blip
* Move `ensure_wasmtime_version` under the tests mod
* Add a macro for puppet_workers
* fix build for not real-overseer
* Rename the puppet worker for adder collator
* play it safe with the name of adder puppet worker
* Typo: triggered
* Add more comments
* Do not kill exec worker on every error
* Plumb Duration for timeouts
* typo: critical
* Add proofs
* Clean unused imports
* Revert "WIP: Diener"
This reverts commit b9f54e513366c7a6dfdd117ac19fbdc46b900b4d.
* Sync version of wasmtime
* Update cargo.lock
* Update Substrate
* Merge fixes still
* Update wasmtime version in test
* bastifmt
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Squash spaces
* Trailing new line for testing.rs
* Remove controversial code
* comment about biasing
* Fix suggestion
* Add comments
* make it more clear why unwrap_err
* tmpfile retry
* proper proofs for claim_idle
* Remove mutex from ValidationHost
* Add some more logging
* Extract exec timeout into a constant
* Add some clarifying logging
* Use blake2_256
* Clean up the merge
Specifically the leftovers after removing real-overseer
* Update parachain/test-parachains/adder/collator/Cargo.toml
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
* use compressed blob in candidate-validation
* add some tests for compressed code blobs
* remove CompressedPoV and apply compression in collation-generation
* decompress BlockData before executing
* don't produce oversized collations
* add test for PoV decompression failure
* fix tests and clean up
* fix test
* address review and fix CI
* take this )
* remove real-overseer
* overseer: only activate leaves which support parachains
* integrate HeadSupportsParachains into service
* remove unneeded line
* Remove stuff out of the runtime that does not belong there.
There might be more, but it is a start.
* White space fixes.
* Fix tests.
* Leave whitespace in ui tests alone.
* Add back zstd for no reason.
* Fix browser wasm (hopefully)
As futures most of the time are catching panics and we don't check this,
it could happen that we have some statement that isn't correct but the
test succeeds successfully.
* Update shared-memory to new version & refactor
This two are combined in a single commit because the new version of
shared-memory doesn't provide the used functionality anymore.
Therefore in order to update the version of this crate we implement the
functionality that we need by ourselves, providing a cleaner API along
the way.
* Significantly decrease the required memory for a workspace
For some reason it was allocating an entire GiB of memory. I suspect
this has something to do with the current memory size limit of a PVF
execution environment (the prior name suggests that). However, we don't
need so much memory anywhere near that amount.
In fact, we could reduce the allocated size even more, but that maybe
for the next time.
* Unlink shmem just after opening
That will make sure that we don't leak the shmem accidentally.
* Do not compile workspace mod for androind and wasm
* Address some review comments
* Fix the test runner
* Fix missed +1 for the attached flag
* Use .expect rather than .unwrap
* Add a rustdoc for the workspace module
* fixup! Use .expect rather than .unwrap
* Add some doc comments to pub members
* Warn on error removing shm_unlink
* Change the alignment implementation
* Fix the comment nit
* Notify collators about seconded collation
This pr adds functionality to inform a collator that its collation was
seconded by a parachain validator. Before this signed statement was only
gossiped over the validation substream. Now, we explicitly send the
seconded statement to the collator after it was validated successfully.
Besides that it changes the `CollatorFn` to return an optional result
sender that is informed when the build collation was seconded by a
parachain validator.
* Add test
* Make sure we only send `Seconded` statements
* Make sure we only receive valid statements
* Review feedback
* PVD: `block_number`->`relay_parent_number`
* ValidationParams: `relay_chain_height`->`relay_parent_number`
* Expose DMQ MQC hash as a well-known-key
This way the relay storage merkle proofs will be able to obtain the DMQ
MQC hash and we will be able to remove the it from the
PersistedValidationData struct.
* PersistedValidationData: Remove HRMP MQC heads
* PersistedValidationData: Remove `dmq_mqc_head`
* Expose the HRMP ingress channel index as a well-known-key
This way a parachain (PVF and collator) can find all the parachains that
have an outbound channel to the given one. That allows in turn to find
all the inbound channels for the given para.
Having access to that allows the parachain to get the same information
as the hrmp_mqc_heads now provide.
* Rename `relay_storage_root` to `relay_parent_storage_root`
* collation-generation: use persisted validation data
* node: remote FullValidationData API
* runtime: remove FullValidationData API
* backing tests: use persisted validation data
* FullCandidateReceipt: use persisted validation data
This is not a big change since this type is not used anywhere
* Remove ValidationData and TransientValidationData
Also update the guide
* remove low information density error doc comments
* another round of error dancing
* fix compilation
* remove stale `None` argument
* adjust test, minor slip in command
* only add AvailabilityError for full node features
* another None where none shuld be
* include new parameter in test `Configuration`
* update calls to `init_logger`
* "Update Substrate"
* cargo update -p sp-io
Co-authored-by: Matt <mattrutherford@users.noreply.github.com>
Co-authored-by: parity-processbot <>
* Make it possible for the adder collator to calculate any state
This is very useful for when wanting to have multiple running or when
wanting to restart the collator.
* Update parachain/test-parachains/adder/collator/src/lib.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* *: Update authority discovery and remove WorkerConfig
With https://github.com/paritytech/substrate/pull/7545 the authority
discovery module queries and publishes addresses on an exponentially
increasing interval. Doing so should make custom configurations
obsolete, as operations are retried in a timely fashion in the first
minutes.
* */Cargo.{lock,toml}: Point to mxinden substrate auth-disc-timing
* Revert "*/Cargo.{lock,toml}: Point to mxinden substrate auth-disc-timing"
This reverts commit 0785943a1e377454f088814ef20f4432de09da7a.
* "Update Substrate"
* Revert ""Update Substrate""
This reverts commit 377b221e1853b2c383f0c416d686535b545796cb.
* Cargo.lock: Manual Substrate update
* node/test/service/src/lib: Remove unused import
* parachain/test-parachains/adder: Remove unused import
Co-authored-by: parity-processbot <>
* reexport prometheus-super for ease of use of other subsystems
* add some prometheus timers for collation generation subsystem
* add timing metrics to av-store
* add metrics to candidate backing
* add timing metric to bitfield signing
* add timing metrics to candidate selection
* add timing metrics to candidate-validation
* add timing metrics to chain-api
* add timing metrics to provisioner
* add timing metrics to runtime-api
* add timing metrics to availability-distribution
* add timing metrics to bitfield-distribution
* add timing metrics to collator protocol: collator side
* add timing metrics to collator protocol: validator side
* fix candidate validation test failures
* add timing metrics to pov distribution
* add timing metrics to statement-distribution
* use substrate_prometheus_endpoint prometheus reexport instead of prometheus_super
* don't include JOB_DELAY in bitfield-signing metrics
* give adder-collator ability to easily export its genesis-state and validation code
* wip: adder-collator pushbutton script
* don't attempt to register the adder-collator automatically
Instead, get these values with
```sh
target/release/adder-collator export-genesis-state
target/release/adder-collator export-genesis-wasm
```
And then register the parachain on https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9944#/explorer
To collect prometheus data, after running the script, create `prometheus.yml` per the instructions
at https://www.notion.so/paritytechnologies/Setting-up-Prometheus-locally-835cb3a9df7541a781c381006252b5ff
and then run:
```sh
docker run -v `pwd`/prometheus.yml:/etc/prometheus/prometheus.yml:z --network host prom/prometheus
```
Demonstrates that data makes it across to prometheus, though it is likely to be useful in the future
to tweak the buckets.
* Update parachain/test-parachains/adder/collator/src/cli.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* use the grandpa-pause parameter
* skip metrics in tracing instrumentation
* remove unnecessary grandpa_pause cli param
Co-authored-by: Andronik Ordian <write@reusable.software>
* Rename ExecutionMode to IsolationStrategy
Execution mode is too generic name and can imply a lot of different
aspects of execution. The notion of isolation better describes the
meant aspect.
And while I am at it, I also renamed mode -> strategy cause it seems a
bit more appropriate, although that is way more subjective.
* Fix compilation in wasm_executor tests.
* Add a comment to IsolationStrategy
* Update comments on IsolationStrategy
* Update node/core/candidate-validation/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Accomodate the point on interruption
* Update parachain/src/wasm_executor/mod.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* Naming nits
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
* Adds integration test based on adder collator
This adds an integration test for parachains that uses the adder
collator. The test will start two relay chain nodes and one collator and
waits until 4 blocks are build and enacted by the parachain.
* Make sure the integration test is run in CI
* Fix wasm compilation
* Update parachain/test-parachains/adder/collator/src/lib.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* Update cli/src/command.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
Martin Pugh