* Implement runtime apis for fake runtime
These runtime api implementations are only used to make the compiler
think that we have implemented all required runtime apis. They will not
be called as we switch the executor to `WasmExecutor`. In the near
future we will not require these fake implementations anymore after
Substrate has shifted away from this compile time requirement.
This brings us the advantage that the `polkadot-service` doesn't need to
depend on the runtimes for getting the `RuntimeApi` type.
It also removes around 1min of build time on my machine ;)
* Fix warning
* FMT
* ".git/.scripts/commands/fmt/fmt.sh"
* Use more descriptive id
* Fix warnings
* Adapt path
* Fix 🙈
---------
Co-authored-by: command-bot <>
* Warn if participated in the losing side of a dispute
* Update naming
* Additionally filter by candidate hash
* Debug zombienet tests
* Update 0002-parachains-disputes.zndsl
* Debug zombienet
* Update node/core/dispute-coordinator/src/initialized.rs
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* Add checking to zombienet tests
---------
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* docs: update references to NetworkConfiguration::extra_sets
Since paritytech/substrate#14080, this struct field no longer exists,
now the `add_notification_protocol()` function of
`sc_network::config::FullNetworkConfiguration` is used.
Also neuter the doc links for now; rustdoc can't resolve them
(presumably because sc_network::config isn't in scope, though weirdly
enough even spelling the link out as
``[`FullNetworkConfiguration`](struct@sc_network::config::FullNetworkConfiguration)``
doesn't work?). Normally this wouldn't be an issue and rustdoc would
just not generate links, but rust 1.70 has a bug that completely crashes
rustdoc in this case.
There is a race condition in `NetworkBridgeRx` between registering
incoming peers and handling overseer signals. While the peer connection
was sent to `NetworkBridgeRx` first, sometimes the peer would be added to
`validation/collation_peers` with enough delay that the bridge handled
an overseer signal that was meant do send notifications to the registered
peer and as the peer was not present in the bridge, the notification was
never sent and the test would hang.
When peers are registered to `NetworkBridgeRx` using `connect_peer()`,
wait until they show up in `shared.validation/collation_peers` before
doing anything else.
Co-authored-by: parity-processbot <>
* Move vstaging to production (and thus past session slashing).
WIP: test-runtime still needs to be fixed.
* Fix test-runtime.
---------
Co-authored-by: eskimor <eskimor@no-such-url.com>
* av-store: Move prune on a separate thread
There are situations where pruning of the data could take more than a few
seconds and that might make the whole subsystem unreponsive. To avoid this just
move the prune process on a separate thread.
See: https://github.com/paritytech/polkadot/issues/7237, for more details.
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
* av-store: Add log that prunning started
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
* av-store: modify log severity
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
---------
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
* metrics: Increase the resolution of histogram metrics
These metrics are using the default histogram buckets:
```
pub const DEFAULT_BUCKETS: &[f64; 11] = &[
0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0,
];
```
Which give us a resolution of 5ms, that's good, but there are some subsystems
where we process hundreds or even a few thousands of messages per second like
approval-voting or approval-distribution, so it makes sense to increse the
resoution of the bucket to better understand if the procesisng is in the range
of useconds.
The new bucket ranges will be:
```
[0.0001, 0.0004, 0.0016, 0.0064, 0.0256, 0.1024, 0.4096, 1.6384, 6.5536]
```
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
* Use buckets with higher resolution
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
---------
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
* approval-distribution: Add approvals/assignments spans on all paths
The approval and assignment logic gets called from multiple paths, so make sure
we create a tracing span on all paths to make debugging easier and be able and
correlate with the spans from approval-voting.
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
* Tag each label with a difference tracing name
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
* Address review feedback
Use the source to determine the tag name
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
---------
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
* runtime/vstaging: unapplied_slashes runtime API
* runtime/vstaging: key_ownership_proof runtime API
* runtime/ParachainHost: submit_report_dispute_lost
* fix key_ownership_proof API
* runtime: submit_report_dispute_lost runtime API
* nits
* Update node/subsystem-types/src/messages.rs
Co-authored-by: Marcin S. <marcin@bytedude.com>
* revert unrelated fmt changes
* dispute-coordinator: past session dispute slashing
* encapsule runtime api call for submitting report
* prettify: extract into a function
* do not exit on runtime api error
* fix tests
* try initial zombienet test
* try something
* fix a typo
* try cumulus-based collator
* fix clippy
* build polkadot-debug images with fast-runtime enabled
* wip
* runtime/inclusion: fix availability_threshold
* fix wip
* fix wip II
* revert native provider
* propagate tx submission error
* DEBUG: sync=trace
* print key ownership proof len
* panic repro
* log validator index in panic message
* post merge fixes
* replace debug assertion with a log
* fix compilation
* Let's log the dispatch info in validate block.
* fix double encoding
* Revert "Let's log the dispatch info in validate block."
This reverts commit a70fbc51b464d7f4355dbada5e16cd83cf71eab4.
* Revert "Let's log the dispatch info in validate block."
This reverts commit a70fbc51b464d7f4355dbada5e16cd83cf71eab4.
* fix compilation
* update to latest zombienet and fix test
* lower finality lag to 11
* bump zombienet again
* add a workaround, but still does not work
* Update .gitlab-ci.yml
bump zombienet.
* add a comment and search logs on all nodes
---------
Co-authored-by: Marcin S. <marcin@bytedude.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Javier Viola <javier@parity.io>
* Replace `RollingSessionWindow` with `RuntimeInfo` - initial commit
* Fix tests in import
* Fix the rest of the tests
* Remove dead code
* Fix todos
* Simplify session caching
* Comments for `SessionInfoProvider`
* Separate `SessionInfoProvider` from `State`
* `cache_session_info_for_head` becomes freestanding function
* Remove unneeded `mut` usage
* fn session_info -> fn get_session_info() to avoid name clashes. The function also tries to initialize `SessionInfoProvider`
* Fix SessionInfo retrieval
* Code cleanup
* Don't wrap `SessionInfoProvider` in an `Option`
* Remove `earliest_session()`
* Remove pre-caching -> wip
* Fix some tests and code cleanup
* Fix all tests
* Fixes in tests
* Fix comments, variable names and small style changes
* Fix a warning
* impl From<SessionWindowSize> for NonZeroUsize
* Fix logging for `get_session_info` - remove redundant logs and decrease log level to DEBUG
* Code review feedback
* Storage migration removing `COL_SESSION_WINDOW_DATA` from parachains db
* Remove `col_session_data` usages
* Storage migration clearing columns w/o removing them
* Remove session data column usages from `approval-voting` and `dispute-coordinator` tests
* Add some test cases from `RollingSessionWindow` to `dispute-coordinator` tests
* Fix formatting in initialized.rs
* Fix a corner case in `SessionInfo` caching for `dispute-coordinator`
* Remove `RollingSessionWindow` ;(
* Revert "Fix formatting in initialized.rs"
This reverts commit 0f94664ec9f3a7e3737a30291195990e1e7065fc.
* v2 to v3 migration drops `COL_DISPUTE_COORDINATOR_DATA` instead of clearing it
* Fix `NUM_COLUMNS` in `approval-voting`
* Use `columns::v3::NUM_COLUMNS` when opening db
* Update node/service/src/parachains_db/upgrade.rs
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* Don't write in `COL_DISPUTE_COORDINATOR_DATA` for `test_rocksdb_migrate_2_to_3`
* Fix `NUM+COLUMNS` in approval_voting
* Fix formatting
* Fix columns usage
* Clarification comments about the different db versions
---------
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
We consider BEEFY mature enough to run by default on all nodes
for test networks (Rococo/Wococo/Versi).
Right now, most nodes are not running it since it's opt-in using
--beefy flag. Switch to an opt-out model for test networks.
Replace --beefy flag from CLI with --no-beefy and have BEEFY
client start by default on test networks.
Signed-off-by: acatangiu <adrian@parity.io>
* PVF: Refactor workers into separate crates, remove host dependency
* Fix compile error
* Remove some leftover code
* Fix compile errors
* Update Cargo.lock
* Remove worker main.rs files
I accidentally copied these from the other PR. This PR isn't intended to
introduce standalone workers yet.
* Address review comments
* cargo fmt
* Update a couple of comments
* Update log targets
* impl guide: Update Collator Generation
* Address review comments
* Fix compile errors
I don't remember why I did this. Maybe it only made sense with the async backing
changes.
* Remove leftover glossary
* metrics: tests: Fix flaky runtime_can_publish_metrics
When an re-org happens wait_for_blocks(2) would actually exit after the second
import of blocks 1, so the conditions for the metric to exist won't be met
hence the occasional test failure.
More details in:
https://github.com/paritytech/polkadot/issues/7267
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
* metrics: tests: Cleanup un-needed box pin
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
---------
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
* Make `issue_explicit_statement_with_index` regular function
* Make `issue_backing_statement_with_index` regular function
* Issue `RevertBlocks` as soon as a dispute has `byzantine threshold + 1` invalid votes.
* Remove a comment
* Fix `has_fresh_byzantine_threshold_against()`
* Extend `informs_chain_selection_when_dispute_concluded_against` test
* PVF: Remove `rayon` and some uses of `tokio`
1. We were using `rayon` to spawn a superfluous thread to do execution, so it was removed.
2. We were using `rayon` to set a threadpool-specific thread stack size, and AFAIK we couldn't do that with `tokio` (it's possible [per-runtime](https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.html#method.thread_stack_size) but not per-thread). Since we want to remove `tokio` from the workers [anyway](https://github.com/paritytech/polkadot/issues/7117), I changed it to spawn threads with the `std::thread` API instead of `tokio`.[^1]
[^1]: NOTE: This PR does not totally remove the `tokio` dependency just yet.
3. Since `std::thread` API is not async, we could no longer `select!` on the threads as futures, so the `select!` was changed to a naive loop.
4. The order of thread selection was flipped to make (3) sound (see note in code).
I left some TODO's related to panics which I'm going to address soon as part of https://github.com/paritytech/polkadot/issues/7045.
* PVF: Vote invalid on panics in execution thread (after a retry)
Also make sure we kill the worker process on panic errors and internal errors to
potentially clear any error states independent of the candidate.
* Address a couple of TODOs
Addresses a couple of follow-up TODOs from
https://github.com/paritytech/polkadot/pull/7153.
* Add some documentation to implementer's guide
* Fix compile error
* Fix compile errors
* Fix compile error
* Update roadmap/implementers-guide/src/node/utility/candidate-validation.md
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* Address comments + couple other changes (see message)
- Measure the CPU time in the prepare thread, so the observed time is not
affected by any delays in joining on the thread.
- Measure the full CPU time in the execute thread.
* Implement proper thread synchronization
Use condvars i.e. `Arc::new((Mutex::new(true), Condvar::new()))` as per the std
docs.
Considered also using a condvar to signal the CPU thread to end, in place of an
mpsc channel. This was not done because `Condvar::wait_timeout_while` is
documented as being imprecise, and `mpsc::Receiver::recv_timeout` is not
documented as such. Also, we would need a separate condvar, to avoid this case:
the worker thread finishes its job, notifies the condvar, the CPU thread returns
first, and we join on it and not the worker thread. So it was simpler to leave
this part as is.
* Catch panics in threads so we always notify condvar
* Use `WaitOutcome` enum instead of bool condition variable
* Fix retry timeouts to depend on exec timeout kind
* Address review comments
* Make the API for condvars in workers nicer
* Add a doc
* Use condvar for memory stats thread
* Small refactor
* Enumerate internal validation errors in an enum
* Fix comment
* Add a log
* Fix test
* Update variant naming
* Address a missed TODO
---------
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* Replace `RollingSessionWindow` with `RuntimeInfo` - initial commit
* Fix tests in import
* Fix the rest of the tests
* Remove dead code
* Fix todos
* Simplify session caching
* Comments for `SessionInfoProvider`
* Separate `SessionInfoProvider` from `State`
* `cache_session_info_for_head` becomes freestanding function
* Remove unneeded `mut` usage
* fn session_info -> fn get_session_info() to avoid name clashes. The function also tries to initialize `SessionInfoProvider`
* Fix SessionInfo retrieval
* Code cleanup
* Don't wrap `SessionInfoProvider` in an `Option`
* Remove `earliest_session()`
* Remove pre-caching -> wip
* Fix some tests and code cleanup
* Fix all tests
* Fixes in tests
* Fix comments, variable names and small style changes
* Fix a warning
* impl From<SessionWindowSize> for NonZeroUsize
* Fix logging for `get_session_info` - remove redundant logs and decrease log level to DEBUG
* Code review feedback
There is a deny(clippy::dbg_macro) in the crate root, so newer
Clippy fails here since tests use dbg.
But dbg in tests are fine IMHO.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Bastian Köcher