Commit Graph

83 Commits

Author SHA1 Message Date
Davide Galassi 260d073658 Companion for #13683 (#6944)
* Companion for #13683

* Wraps trait is not required

* update lockfile for {"substrate"}

---------

Co-authored-by: parity-processbot <>
2023-03-24 14:43:21 +00:00
Davide Galassi 46c36e5a4f [Companion #13615] Keystore overhaul (#6892)
* Remove not required async calls

* Fixed missing renaming

* make_keystore can be sync

* More fixes

* Trivial nitpicks

* Cherry pick test fix from master

* Fixes after master merge

* update lockfile for {"substrate"}

---------

Co-authored-by: parity-processbot <>
2023-03-17 12:09:15 +00:00
s0me0ne-unkn0wn 27c9124411 Release parachain host API v4 (#6885) 2023-03-15 17:06:18 +00:00
s0me0ne-unkn0wn dd0a556665 Executor Environment parameterization (#6161)
* Re-apply changes without Diener, rebase to the lastest master

* Cache pruning

* Bit-pack InstantiationStrategy

* Move ExecutorParams version inside the structure itself

* Rework runtime API and executor parameters storage

* Pass executor parameters through backing subsystem

* Update Cargo.lock

* Introduce `ExecutorParams` to approval voting subsys

* Introduce `ExecutorParams` to dispute coordinator

* `cargo fmt`

* Simplify requests from backing subsys

* Fix tests

* Replace manual config cloning with `.clone()`

* Move constants to module

* Parametrize executor performing PVF pre-check

* Fix Malus

* Fix test runtime

* Introduce session executor params as a constant defined by session info
pallet

* Use Parity SCALE codec instead of hand-crafted binary encoding

* Get rid of constants; Add docs

* Get rid of constants

* Minor typo

* Fix Malus after rebase

* `cargo fmt`

* Use transparent SCALE encoding instead of explicit

* Clean up

* Get rid of relay parent to session index mapping

* Join environment type and version in a single enum element

* Use default execution parameters if running an old runtime

* `unwrap()` -> `expect()`

* Correct API version

* Constants are back in town

* Use constants for execution environment types

* Artifact separation, first try

* Get rid of explicit version

* PVF execution queue worker separation

* Worker handshake

* Global renaming

* Minor fixes resolving discussions

* Two-stage requesting of executor params to make use of runtime API cache

* Proper error handling in pvf-checker

* Executor params storage bootstrapping

* Propagate migration to v3 network runtimes

* Fix storage versioning

* Ensure `ExecutorParams` serialization determinism; Add comments

* Rename constants to make things a bit more deterministic
Get rid of stale code

* Tidy up a structure of active PVFs

* Minor formatting

* Fix comment

* Add try-runtime hooks

* Add storage version write on upgrade

Co-authored-by: Andronik <write@reusable.software>

* Add pre- and post-upgrade assertions

* Require to specify environment type; Remove redundant `impl`s

* Add `ExecutorParamHash` creation from `H256`

* Fix candidate validation subsys tests

* Return splittable error from executor params request fn

* Revert "Return splittable error from executor params request fn"

This reverts commit a0b274177d8bb2f6e13c066741892ecd2e72a456.

* Decompose approval voting metrics

* Use more relevant errors

* Minor formatting fix

* Assert a valid environment type instead of checking

* Fix `try-runtime` hooks

* After-merge fixes

* Add migration logs

* Remove dead code

* Fix tests

* Fix tests

* Back to the strongly typed implementation

* Promote strong types to executor interface

* Remove stale comment

* Move executor params to `SessionInfo`: primitives and runtime

* Move executor params to `SessionInfo`: node

* Try to bump primitives and API version

* Get rid of `MallocSizeOf`

* Bump target API version to v4

* Make use of session index already in place

* Back to v3

* Fix all the tests

* Add migrations to all the runtimes

* Make use of existing `SessionInfo` in approval voting subsys

* Rename `TARGET` -> `LOG_TARGET`

* Bump all the primitives to v3

* Fix Rococo ParachainHost API version

* Use `RollingSessionWindow` to acquire `ExecutorParams` in disputes

* Fix nits from discussions; add comments

* Re-evaluate queue logic

* Rework job assignment in execution queue

* Add documentation

* Use `RuntimeInfo` to obtain `SessionInfo` (with blackjack and caching)

* Couple `Pvf` with `ExecutorParams` wherever possible

* Put members of `PvfWithExecutorParams` under `Arc` for cheap cloning

* Fix comment

* Fix CI tests

* Fix clippy warnings

* Address nits from discussions

* Add a placeholder for raw data

* Fix non exhaustive match

* Remove redundant reexports and fix imports

* Keep only necessary semantic features, as discussed

* Rework `RuntimeInfo` to support mock implementation for tests

* Remove unneeded bound

* `cargo fmt`

* Revert "Remove unneeded bound"

This reverts commit 932463f26b00ce290e1e61848eb9328632ef8a61.

* Fix PVF host tests

* Fix PVF checker tests

* Fix overseer declarations

* Simplify tests

* `MAX_KEEP_WAITING` timeout based on `BACKGING_EXECUTION_TIMEOUT`

* Add a unit test for varying executor parameters

* Minor fixes from discussions

* Add prechecking max. memory parameter (see paritytech/srlabs_findings#110)

* Fix and improve a test

* Remove `ExecutionEnvironment` and `RawData`

* New primitives versioning in parachain host API

* `disputes()` implementation for Kusama and Polkadot

* Move `ExecutorParams` from `vstaging` to stable primitives

* Move disputes from `vstaging` to stable implementation

* Fix `try-runtime`

* Fixes after merge

* Move `ExecutorParams` to the bottom of `SessionInfo`

* Revert "Move executor params to `SessionInfo`: primitives and runtime"

This reverts commit dfcfb85fefd1c5be6c8a8f72dc09fd1809cfa9ce.

* Always use fresh activated live hash in pvf precheck
(re-apply 34b09a4c20de17e7926ed942cd0d657d18f743fa)

* Fixing tests (broken commit)

* Fix candidate validation tests

* Fix PVF host test

* Minor fixes

* Address discussions

* Restore migration

* Fix `use` to only include what is needed instead of `*`

* Add comment to never touch `DEFAULT_CONFIG`

* Update migration to set default `ExecutorParams` for `dispute_period`
sessions back

* Use `earliest_stored_session` instead of calculations

* Nit

* Add logs

* Treat any runtime error as `NotSupported` again

* Always return default executor params if not available

* Revert "Always return default executor params if not available"

This reverts commit b58ac4482ef444c67a9852d5776550d08e312f30.

* Add paritytech/substrate#9997 workaround

* `cargo fmt`

* Remove migration (again!)

* Bump executor params to API v4 (backport from #6698)

---------

Co-authored-by: Andronik <write@reusable.software>
2023-02-15 11:26:09 +00:00
s0me0ne-unkn0wn 1cb1d03c08 Re-export current primitives in crate root (#6487)
* Re-export current primitives in crate root

* Add missing exports

* restart CI
2023-01-11 11:28:12 +00:00
eskimor cc650fe53d Fix cycle dispute-coordinator <-> dispute-distribution (#6489)
* First iteration of message sender.

* dyn Fn variant (no cloning)

* Full implementation + Clone, without allocs on `Send`

* Further clarifications/cleanup.

* MessageSender -> NestingSender

* Doc update/clarification.

* dispute-coordinator: Send disputes on startup.

+ Some fixes, cleanup.

* Fix whitespace.

* Dispute distribution fixes, cleanup.

* Cargo.lock

* Fix spaces.

* More format fixes.

What is cargo fmt doing actually?

* More fmt fixes.

* Fix nesting sender.

* Fixes.

* Whitespace

* Enable logging.

* Guide update.

* Fmt fixes, typos.

* Remove unused function.

* Simplifications, doc fixes.

* Update roadmap/implementers-guide/src/node/disputes/dispute-coordinator.md

Co-authored-by: Marcin S. <marcin@bytedude.com>

* Fmt + doc example fix.

Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: Marcin S. <marcin@bytedude.com>
2023-01-10 12:04:05 +01:00
Chris Sosnin c9316799ea Remove Job trait (#5600) 2022-05-26 14:26:51 -05:00
Bernhard Schuster fd51ecbe70 prepare orchestra crate publishing (#5579) 2022-05-24 11:06:47 +02:00
Bernhard Schuster 450ca2baca overseer becomes orchestra (#5542)
* rename overseer-gen to orchestra

Also drop `gum` and use `tracing`.

* make orchestra compile as standalone

* introduce Spawner trait to split from sp_core

Finalizes the independence of orchestra from polkadot-overseer

* slip of the pen

* other fixins

* remove unused import

* Update node/overseer/orchestra/proc-macro/src/impl_builder.rs

Co-authored-by: Vsevolod Stakhov <vsevolod.stakhov@parity.io>

* Update node/overseer/orchestra/proc-macro/src/impl_builder.rs

Co-authored-by: Vsevolod Stakhov <vsevolod.stakhov@parity.io>

* orchestra everywhere

* leaky data

* Bump scale-info from 2.1.1 to 2.1.2 (#5552)

Bumps [scale-info](https://github.com/paritytech/scale-info) from 2.1.1 to 2.1.2.
- [Release notes](https://github.com/paritytech/scale-info/releases)
- [Changelog](https://github.com/paritytech/scale-info/blob/master/CHANGELOG.md)
- [Commits](https://github.com/paritytech/scale-info/compare/v2.1.1...v2.1.2)

---
updated-dependencies:
- dependency-name: scale-info
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add missing markdown code block delimiter (#5555)

* bitfield-signing: remove util::jobs usage  (#5523)

* Switch to pooling copy-on-write instantiation strategy for WASM (companion for Substrate#11232) (#5337)

* Switch to pooling copy-on-write instantiation strategy for WASM

* Fix compilation of `polkadot-test-service`

* Update comments

* Move `max_memory_size` to `Semantics`

* Rename `WasmInstantiationStrategy` to `WasmtimeInstantiationStrategy`

* Update a safety comment

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>

* Fix build

Co-authored-by: Vsevolod Stakhov <vsevolod.stakhov@parity.io>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Malte Kliemann <mail@maltekliemann.com>
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
2022-05-19 13:42:02 +01:00
Bernhard Schuster 511891dcce refactor+feat: allow subsystems to send only declared messages, generate graphviz (#5314)
Closes #3774
Closes #3826
2022-05-12 17:39:05 +02:00
Robert Klotzner f820db494f dispute-coordinator: Cleanup + Bug fixes (#5323)
* Make import confirmation oneshot optional.

* Cleanup for further improvements.

* Queue adoptions.

* Fix fieldname

* Use correct relay parent

* Fix scraper tests.

* Small optimization.

* Fix all tests.

* Fix other tests.

* fmt

* spelling

* Fix warning.
2022-04-19 14:51:02 +02:00
Qinxuan Chen 74078d8eb9 Comanion for substrate#11136 (#5218)
* Comanion for substrate#11136

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* revert changes in bridge

Signed-off-by: koushiro <koushiro.cqx@gmail.com>
2022-04-04 11:13:34 +02:00
Vsevolod Stakhov af94fc9534 Try to fix out of view statements (#5177)
This issue happens when some peer sends a good but already known Seconded statement and the statement-distribution code does not update the statements_received field in the peer_knowledge structure. Subsequently, a Valid statement causes out-of-view message that is incorrectly emitted and causes reputation lose.

This PR also introduces a concept of passing the specific pseudo-random generator to subsystems to make it easier to write deterministic tests. This functionality is not really necessary for the specific issue and unit test but it can be useful for other tests and subsystems.
2022-03-24 20:18:43 +00:00
Bernhard Schuster d631f1dea8 observability: tracing gum, automatically cross ref traceID (#5079)
* add some gum

* bump expander

* gum

* fix all remaining issues

* last fixup

* Update node/gum/proc-macro/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* change

* netowrk

* fixins

* chore

* allow optional fmt str + args, prep for expr as kv field

* tracing -> gum rename fallout

* restrict further

* allow multiple levels of field accesses

* another round of docs and a slip of the pen

* update ADR

* fixup lock fiel

* use target: instead of target=

* minors

* fix

* chore

* Update node/gum/README.md

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
2022-03-15 11:05:16 +00:00
Robert Habermeier 49f7e5cce4 Finish migration to v2 primitives (#5037)
* remove v0 primitives from polkadot-primitives

* first pass: remove v0

* fix fallout in erasure-coding

* remove v1 primitives, consolidate to v2

* the great import update

* update runtime_api_impl_v1 to v2 as well

* guide: add `Version` request for runtime API

* add version query to runtime API

* reintroduce OldV1SessionInfo in a limited way
2022-03-09 14:01:13 -06:00
cheme d5ddb1a809 paritydb support for parachains db. (#4838)
* parity db subsystem without cache and no splitted column

* fmt

* fix path (auto from parity-db fail)

* lru cache for db column with cache

* Revert "lru cache for db column with cache"

This reverts commit ae177bc5e107a075eff6a21f651218ada6599b74.

* Write_lock mutex

* theoric code for bridges

* revert changes

* Revert bridge changes

* fix spec_version

* update parity db

* test purge-db

* Use specific ordered collection with paritydb.

* Revert "Use specific ordered collection with paritydb."

This reverts commit 8b66d0a4ae914cba1af0f44050d45dd6d9327c6b.

* fix chain selection tests.

* remove patch

* fix auto.

* Remove useless exists directory method

* purge chain without parity-db removal

* spellcheck

* renamings and filtering.

* fix assertion

* format

* update parity-db and fmt

* Auto keep using rocksdb when it exists.

* Revert "Auto keep using rocksdb when it exists."

This reverts commit cea49b32ae590bdce31fed5c45f3c028ae0c7564.

* Update kvdb version.
2022-03-03 11:49:38 +00:00
Bastian Köcher d9d53c17a1 Switch to edition 2021 (#4976)
* Switch to edition 2021

* Reset bridges

* Fix compilation

* Fix test-runtime

* Fix malus
2022-02-24 10:20:58 +00:00
Sergei Shulepov 91aff5d341 runtime-api: add validation_code_hash API (#4629)
This is the first step to close
https://github.com/paritytech/polkadot/issues/4524
2021-12-28 20:16:03 +00:00
Andronik Ordian b342ae11d3 session-info: add new fields + migration (#4545)
* session_info: v2 + migration

* use primitives::v2

* use polkadot_primitives::v2

* impl primitives::v2

* fix approval-voting tests

* fix other tests

* hook storage migration up

* backwards compat (1)

* backwards compat (2)

* fmt

* fix tests

* FMT

* do not reexport v1 in v2

* fmt

* set storage version to 1

Co-authored-by: Javier Viola <javier@parity.io>
2021-12-27 08:01:30 +00:00
Gavin Wood ca72ad636c Companion #10403: Remove Default for AccountId (#4500)
* Some work

* Fixes

* Tests builds

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Formatting

* Formatting

* Fix

* Fixes

* Fixes

* Fixes

* Fixes

* Update Cargo.lock

* Bump

* Fixes
2021-12-14 08:17:26 +01:00
Robert Klotzner bd5721fbf5 Revert loop prevention (#4472)
* Provisioner: Only include and sign bitfields on fresh leaves.
2021-12-13 12:20:49 +01:00
Sergei Shulepov 9f778ccfa4 pvf-precheck: Add sign in subsystem-util (#4407)
Right now, most of operations that sign stuff in polkadot protocol are
handled by a very convenient tool - `Signed`. However `Signed` assumes
that whatever is signed is anchored to some `parent_hash` which works
for most cases, but does not work for others.

One instance of such a case is pre-checking (#3211). There validators
submit signed votes on-chain. A vote is valid for the entire session. If
we were to use `Signed` we would have to root a vote in some block of
that session and during vote verification check that this block is
indeed within the session. This is especially annoying since we agreed
to use unsigned extrinsics to submit votes and we need to make the
unsigned extrinsic validation as slim as possible.

(FWIW, the definition of a pre-checking vote can be seen in the next
diff in the stack)

That's the reason why we opted-out from using `Signed` for pre-checking
and decided to go with the manual signing approach. Almost every piece
of machinery is in place except for signing which is presented in this
PR.
2021-11-30 11:31:42 +01:00
Javier Viola ea5dbd0475 introduce malus + zombienet based integration tests (#4131)
* test/malus: craft the first maliciously disputing actor

* initial draft

* Add Dockerfile and instructions how to use it to build malus image locally

* Forgot one flag for the build cmd

* we are not docker specific, we are happy to use any containerruntime

* shuffle things around

* add initial tera based integration test

* chores

* fixins

* simple setup to start

* other samples (WIP)

* add Docker version with cargo-chef

* update substarte and small change of orders in commands in the container file

* metrics one

* fmt

* minor

* fixin

* fix metric names

* -d

* add open gauge

* fmt

* spellcheck

* fix test

* adjust to changed error messages

* refactor, more malus impls

* more malus changes

* foo

* minor cleanup

* suggest garbage candidate

* chore

* fix suggest garabge malus

* malus: back garbage candidate

* cargo lock

* re-introduce metrics

* chore: cargo fmt

* undoe 1.54.0 output, CI uses 1.53.0 rustc

* update location of js types

* Fix trybuild

* add tag to image name also; this will be replaced in the prod version

* Tests fixed

* add some fix me

* add dockerfile for ci

* Add docker file for malus for ci

* use variables in .toml file

* add chnages for malus test

* some fixes

* some more fixes

* Update .gitlab-ci.yml

* add local build for polkadot and malus

* some fixes

* enable disputes feature in CI

* ok, ok

* rename: MsgFilter -> MessageInterceptor

* remove TODO that would not have worked

* intercept

* refactor

* fix README and containers

* fix

* chore: cargo fmt

* avoid some more malus-$VARIANT references

* fix argument order

* chore: add about

* Update sanity check in relay chain selection

* fix order, add dispute-unavailable-block malus

* fixup: avoid underflow issue

* it's all u32

* fix conditional use

* Revert "it's all u32"

This reverts commit 6b3ae25bfd0bbf0b51d90d743642a75a4a815d6e.

* Revert "fixup: avoid underflow issue"

This reverts commit 336cbe2938e9720f870d37d8feeab7ca69200f47.

* Revert "Update sanity check in relay chain selection"

This reverts commit 970647f35e1116136e12fd91cd9f2fb7e18ad28d.

* update the malus bin

* Update node/malus/integrationtests/0003-dispute-unavailable-block.feature

Co-authored-by: Andronik Ordian <write@reusable.software>

* add some FIXME reminders

* update path to index.js

* Update .gitlab-ci.yml

* Update node/malus/integrationtests/0001-dispute-valid-block.toml

* try 1: make malus test run

* chore: cargo fmt

* temporary fix

* use subcommand syntax from latest gurke

* cargo +nightly fmt

* add collator to a a test

* docs: add env vars to README

* update ci to run dispute-valid-block test

* needs the polkadot image

* Fix path for nodejs container

* post merge fix

* download proper dir  containg configs for malus test

* update the malus ci job

* rm a whitespace

* temp build for malus

* use correct build command for temp malus

* remove subcommands for now

* set max validators per core in the default HostConfig

* tabs

* update beefy

* fixup

* fixup II

* make one variant compile

* make other variants compile

* revert changes to chain_spec

* fmt

* build malus image from polkadot-test-malus again

* revert unrelated changes

* try fixing build-malus job

* Revert "remove subcommands for now"

This reverts commit 5d8292bc49252124937affec4b7c28181a5deb7e.

* try fixing build-malus job II

* MVP working dispute-ancestor

* renames

* fix PVF execution on malus

* fix test

* fix typo

* fmt

* checkmate

* try something

* make it actually work

* some tweaks to 01 feature test

* fmt

* sleep a bit more

* complete wococoization

* some tweaks to 01 feature test

* typo fix

* use correct metric names

* fix

* ffs

* .

* try some rearrangement

* Attempt to wait till initial node bootstrap in test

* Fix test syntax

* Run malus tests with v2 script

* Proper symlink created

* simnet v14

* add zombienet tests

* add zombie net test - draft

* add more tests to dispute suite

* add within to fix tests

* replace test directory and start test migration

* migrate all the tests

* add timeout to tests

* reduce debug

* make easy to test in dev

* set appropriated debug

* use image from ci

* fix config for test

* set images from ci

* fix config

* add COLIMAGE env

* tweek tests

* reduce debug

* typo

* wip, migrate old test to zombie-net

* adjunt test config for zombie-net

* run mauls 0001 test only

* clean and setup smoke-test in zombie-net

* add extra time to assertinons

* clean code to merge and improve README

* add info to access logs

* improved readme

* merge master and resolve conflicts

* Update zombienet_tests/README.md

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* clean and consolidate zombienet name

* change runner in gitlab

* add comment explain why we use wococo

* change tag for runner

* remove unused tests

* remove dup Dockerfile and update description

* fmt

* fix compilation post-merge

* fmt

* cut me Some slack

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: radupopa2010 <radupopa2010@yahoo.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Anton Gavrilov <AntonE.Gavrilov@gmail.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com>
2021-11-20 14:03:28 +00:00
Robert Klotzner 25974f2076 Dispute spam protection (#4134)
* Mostly notes.

* Better error messages.

* Introduce Fatal/NonFatal + drop back channel participation

- Fatal/NonFatal - in order to make it easier to use utility functions.
- We drop the back channel in dispute participation as it won't be
needed any more.

* Better error messages.

* Utility function for receiving `CandidateEvent`s.

* Ordering module typechecks.

* cargo fmt

* Prepare spam slots module.

* Implement SpamSlots mechanism.

* Implement queues.

* cargo fmt

* Participation.

* Participation taking shape.

* Finish participation.

* cargo fmt

* Cleanup.

* WIP: Cleanup + Integration.

* Make `RollingSessionWindow` initialized by default.

* Make approval voting typecheck.

* Get rid of lazy_static & fix approval voting tests

* Move `SessionWindowSize` to node primitives.

* Implement dispute coordinator initialization.

* cargo fmt

* Make queues return error instead of boolean.

* Initialized: WIP

* Introduce chain api for getting finalized block.

* Fix ordering to only prune candidates on finalized events.

* Pruning of old sessions in spam slots.

* New import logic.

* Make everything typecheck.

* Fix warnings.

* Get rid of obsolete dispute-participation.

* Fixes.

* Add back accidentelly deleted Cargo.lock

* Deliver disputes in an ordered fashion.

* Add module docs for errors

* Use type synonym.

* hidden docs.

* Fix overseer tests.

* Ordering provider taking `CandidateReceipt`.

... To be kicked on one next commit.

* Fix ordering to use relay_parent

as included block is not unique per candidate.

* Add comment in ordering.rs.

* Take care of duplicate entries in queues.

* Better spam slots.

* Review remarks + docs.

* Fix db tests.

* Participation tests.

* Also scrape votes on first leaf for good measure.

* Make tests typecheck.

* Spelling.

* Only participate in actual disputes, not on every import.

* Don't account backing votes to spam slots.

* Fix more tests.

* Don't participate if we don't have keys.

* Fix tests, typos and warnings.

* Fix merge error.

* Spelling fixes.

* Add missing docs.

* Queue tests.

* More tests.

* Add metrics + don't short circuit import.

* Basic test for ordering provider.

* Import fix.

* Remove dead link.

* One more dead link.

Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com>
2021-11-19 17:08:21 +00:00
sandreim b0f89bbfbc Per subsystem CPU usage tracking (#4239)
* SubsystemContext: add subsystem name str

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* Overseer builder proc macro changes

* initilize SubsystemContext name field.
* Add subsystem name in TaskKind::launch_task()

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* Update ToOverseer enum

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* Assign subsystem names to orphan tasks

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* cargo fmt

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* SubsystemContext: add subsystem name str

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* Overseer builder proc macro changes

* initilize SubsystemContext name field.
* Add subsystem name in TaskKind::launch_task()

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* Update ToOverseer enum

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* Assign subsystem names to orphan tasks

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* cargo fmt

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Rebase changes for new spawn() group param

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Add subsystem constat in JobTrait

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Add subsystem string

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix spawn() calls

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* cargo fmt

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix more tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Address PR review feedback #1

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Address PR review round 2

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fixes
- remove JobTrait::Subsystem
- fix tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* update Cargo.lock

Co-authored-by: Andronik Ordian <write@reusable.software>
2021-11-11 18:53:37 +00:00
Chris Sosnin ad33b8749b Introduce new Runtime API endpoint for fetching the validation data (#3728)
* Introduce new Runtime API endpoint

`persisted_validation_data_with_code_hash` that will be used
by the candidate validation subsystem in order to decrease amount
of runtime API requests.

* Node-side part of new runtime API request

* Define code hash getter via macro

* Rename new endpoint to `assumed_validation_data`

* Docs for runtime API impl of new endpoint

* AssumedValidationData specialized request function

* fmt
2021-10-22 12:49:26 +00:00
Andronik Ordian 8aba3ea8ca some spelling fixes (#4088)
* quotes around Job::ToJob
* add new words to dictionary
2021-10-18 11:59:46 +02:00
Robert Klotzner 55154a8d37 Remove request multiplexer (#3624)
* WIP: Get rid of request multiplexer.

* WIP

* Receiver for handling of incoming requests.

* Get rid of useless `Fault` abstraction.

The things the type system let us do are not worth getting abstracted in
its own type. Instead error handling is going to be merely a pattern.

* Make most things compile again.

* Port availability distribution away from request multiplexer.

* Formatting.

* Port dispute distribution over.

* Fixup statement distribution.

* Handle request directly in collator protocol.

+ Only allow fatal errors at top level.

* Use direct request channel for availability recovery.

* Finally get rid of request multiplexer

Fixes #2842 and paves the way for more back pressure possibilities.

* Fix overseer and statement distribution tests.

* Fix collator protocol and network bridge tests.

* Fix tests in availability recovery.

* Fix availability distribution tests.

* Fix dispute distribution tests.

* Add missing dependency

* Typos.

* Review remarks.

* More remarks.
2021-08-12 13:11:36 +02:00
Shawn Tabrizi ff5d56fb76 cargo +nightly fmt (#3540)
* cargo +nightly fmt

* add cargo-fmt check to ci

* update ci

* fmt

* fmt

* skip macro

* ignore bridges
2021-08-02 10:47:33 +00:00
Denis Pisarev fc253e6e4d WIP: CI: add spellcheck (#3421)
* CI: add spellcheck

* revert me

* CI: explicit command for spellchecker

* spellcheck: edit misspells

* CI: run spellcheck on diff

* spellcheck: edits

* spellcheck: edit misspells

* spellcheck: add rules

* spellcheck: mv configs

* spellcheck: more edits

* spellcheck: chore

* spellcheck: one more thing

* spellcheck: and another one

* spellcheck: seems like it doesn't get to an end

* spellcheck: new words after rebase

* spellcheck: new words appearing out of nowhere

* chore

* review edits

* more review edits

* more edits

* wonky behavior

* wonky behavior 2

* wonky behavior 3

* change git behavior

* spellcheck: another bunch of new edits

* spellcheck: new words are koming out of nowhere

* CI: finding the master

* CI: fetching master implicitly

* CI: undebug

* new errors

* a bunch of new edits

* and some more

* Update node/core/approval-voting/src/approval_db/v1/mod.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Update xcm/xcm-executor/src/assets.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Apply suggestions from code review

Co-authored-by: Andronik Ordian <write@reusable.software>

* Suggestions from the code review

* CI: scan only changed files

Co-authored-by: Andronik Ordian <write@reusable.software>
2021-07-14 19:22:58 +02:00
Bernhard Schuster 3c9104daff refactor overseer into proc-macro based pattern (#2962) 2021-07-08 21:09:26 +02:00
Andronik Ordian ffc6f7c731 make ctx.spawn blocking (#3337)
* make spawn sync

* improve error type
2021-06-21 20:43:40 -05:00
Bastian Köcher 3d2931ace5 Remove the streamunordered crate (#3339)
The functionality is now provided by the `futures` crate.
2021-06-21 19:48:33 +00:00
Andronik Ordian ad9c02886d improved gossip topology (#3270)
* gossip-support: gossip topology

* some fixes

* handle view update for newly added gossip peers

* fix neighbors calculation

* fix test

* resolve TODOs

* typo

* guide updates

* spaces in the guide

* sneaky spaces

* hash randomness

* address some review nits

* use unbounded in bridge for subsystem msg
2021-06-18 14:30:35 -05:00
Robert Habermeier 462ca043e5 extract determine_new_blocks into a separate utility (#3261)
* extract determine_new_blocks into a separate utility

* rework docs
2021-06-16 14:10:50 -05:00
Robert Habermeier 5bc2b2779d Dispute Coordinator Subsystem (#3150)
* skeleton for dispute-coordinator

* add coordinator and participation message types

* begin dispute-coordinator DB

* functions for loading

* implement strongly-typed DB transaction

* add some tests for DB transaction

* core logic for pruning

* guide: update candidate-votes key for coordinator

* update candidate-votes key

* use big-endian encoding for session, and implement upper bound generator

* finish implementing pruning

* add a test for note_current_session

* define state of the subsystem itself

* barebones subsystem definition

* control flow

* more control flow

* implement session-updating logic

* trace

* control flow for message handling

* Update node/core/dispute-coordinator/src/lib.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update node/subsystem/src/messages.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* some more control flow

* guide: remove overlay

* more control flow

* implement some DB getters

* make progress on importing statements

* add SignedDisputeStatement struct

* move ApprovalVote to shared primitives

* add a signing-payload API to explicit dispute statements

* add signing-payload to CompactStatement

* add relay-parent hash to seconded/valid dispute variatns

* correct import

* type-safe wrapper around dispute statements

* use checked dispute statement in message type

* extract rolling session window cache to subsystem-util

* extract session window tests

* approval-voting: use rolling session info cache

* reduce dispute window to match runtime in practice

* add byzantine_threshold and supermajority_threshold utilities to primitives

* integrate rolling session window

* Add PartialOrd to CandidateHash

* add Ord to CandidateHash

* implement active dispute update

* add dispute messages to AllMessages

* add dispute stubs to overseer

* inform dispute participation to participate

* implement issue_local_statement

* implement `determine_undisputed_chain`

* fix warnings

* test harness for dispute coordinator tests

* add more helpers to test harness

* add some more helpers

* some tests for dispute coordinator

* ignore wrong validator indices

* test finality voting rule constraint

* add more tests

* add variants to network bridge

* fix test compilation

* remove most dispute coordinator functionality

as of #3222 we can do most of the work within the approval voting subsystem

* Revert "remove most dispute coordinator functionality"

This reverts commit 9cd615e8eb6ca0b382cbaff525d813e753d6004e.

* Use thiserror

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* Update node/core/dispute-coordinator/src/lib.rs

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* extract tests to separate module

* address nit

* adjust run_iteration API

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
2021-06-13 06:35:18 -05:00
Andronik Ordian cbeb7d0afd tabify tests (#3220)
* tabify tests

* move mod tests; up
2021-06-12 15:39:18 +02:00
Bernhard Schuster f698bf8548 chore: move tests into separate files (#3206)
Moves tests into separate files
in order to limit the loc per file.
2021-06-11 12:50:19 +00:00
Lldenaurois 2abaca3a8c Remove candidate selection (#3148)
* Create validator_side module

* Subsume Candidate Selection

* Add test to ensure candidate backing logic is correct

* Ensure secondings are adequately cleaned up and address test flakyness

* Address Feedback
2021-06-08 14:07:19 -04:00
Robert Habermeier 963993d288 Reversion Safety tools for overseer and subsystems (#3104)
* guide: reversion safety

* guide: manage reversion safety in subsystems

* add leaf status to ActivatedLeaf

* add an LRU-cache to overseer for staleness detection

* update ActivatedLeaf usages in tests to contain status field

* add variant where missed accidentally

* add some helpers to LeafStatus

* address grumbles
2021-05-31 20:54:05 +02:00
Andronik Ordian 60fbca3c2a validator_discovery: simplification (#3009)
* validator_discovery: simplification

* compilation fixes

* compilation fixes II

* compilation fixes III

* compilation fixes IV
2021-05-13 11:31:15 +02:00
Robert Klotzner 0dbdfef95e More secure Signed implementation (#2963)
* Remove signature verification in backing.

`SignedFullStatement` now signals that the signature has already been
checked.

* Remove unused check_payload function.

* Introduced unchecked signed variants.

* Fix inclusion to use unchecked variant.

* More unchecked variants.

* Use unchecked variants in protocols.

* Start fixing statement-distribution.

* Fixup statement distribution.

* Fix inclusion.

* Fix warning.

* Fix backing properly.

* Fix bitfield distribution.

* Make crypto store optional for `RuntimeInfo`.

* Factor out utility functions.

* get_group_rotation_info

* WIP: Collator cleanup + check signatures.

* Convenience signature checking functions.

* Check signature on collator-side.

* Fix warnings.

* Fix collator side tests.

* Get rid of warnings.

* Better Signed/UncheckedSigned implementation.

Also get rid of Encode/Decode for Signed! *party*

* Get rid of dead code.

* Move Signed in its own module.

* into_checked -> try_into_checked

* Fix merge.
2021-05-03 21:41:14 +02:00
Robert Klotzner c86a774b9d Send statements to own backing group first (#2927)
* Factor out runtime module into utils.

* First fatal error design.

* Better error handling infra.

* Error handling cleanup.

* Send to peers of our group first.

* Finish backing group prioritization.

* Little cleanup.

* More cleanup.

* Forgot to checkin error.rs.

* Notes.

* Runtime -> RuntimeInfo

* qed in debug assert.

* PolkaErr -> Fault.
2021-04-27 21:47:32 +02:00
Robert Klotzner dacde443f7 Infrastructure improvements (#2897)
* Factor out runtime module into utils.

* Add maybe_authority information to `PeerConnected` event.

We already gather this information in authority discovery, so we might
as well share it with others.

This opens up an easy path to trigger validators differently from normal
nodes, e.g. for prioritization. This change has become more important
now, that we just connect to all validators and therefore just have a
long peer list without any information about those nodes.

* Test fix.
2021-04-16 21:42:20 +02:00
Andronik Ordian e62939ad7a distribution: handle sqrt peer view updates (#2871)
* distribution: handle sqrt peer view updates

* someone please put rustc into my brain

* guide updates
2021-04-10 00:45:25 +02:00
Robert Habermeier 11b8e4c821 Collation protocol: stricter validators (#2810)
* guide: declare one para as a collator

* add ParaId to Declare messages and clean up

* fix build

* fix the testerinos

* begin adding keystore to collator-protocol

* remove request_x_ctx

* add core_for_group

* add bump_rotation

* add some more helpers to subsystem-util

* change signing_key API to take ref

* determine current and next para assignments

* disconnect collators who are not on current or next para

* add collator peer count metric

* notes for later

* some fixes

* add data & keystore to test state

* add a test utility for answering runtime API requests

* fix existing collator tests

* add new tests

* remove sc_keystore

* update cargo lock

Co-authored-by: Andronik Ordian <write@reusable.software>
2021-04-03 21:48:58 +02:00
Andronik Ordian 98082c5326 gossip: move authorities request to runtime api subsystem (#2798) 2021-04-01 23:51:01 +02:00
Andronik Ordian 9ac35d9f2b gossip: choose a random subset on send instead of limiting connections (#2776)
* gossip: choose random subset on send

* naming bikeshed
2021-03-30 20:59:53 +02:00
Robert Habermeier 8ebbe19d10 Split NetworkBridge and break cycles with Unbounded (#2736)
* overseer: pass messages directly between subsystems

* test that message is held on to

* Update node/overseer/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* give every subsystem an unbounded sender too

* remove metered_channel::name

1. we don't provide good names
2. these names are never used anywhere

* unused mut

* remove unnecessary &mut

* subsystem unbounded_send

* remove unused MaybeTimer

We have channel size metrics that serve the same purpose better now and the implementation of message timing was pretty ugly.

* remove comment

* split up senders and receivers

* update metrics

* fix tests

* fix test subsystem context

* use SubsystemSender in jobs system now

* refactor of awful jobs code

* expose public `run` on JobSubsystem

* update candidate backing to new jobs & use unbounded

* bitfield signing

* candidate-selection

* provisioner

* approval voting: send unbounded for assignment/approvals

* async not needed

* begin bridge split

* split up network tasks into background worker

* port over network bridge

* Update node/network/bridge/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* rename ValidationWorkerNotifications

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
2021-03-29 01:18:53 +02:00
Robert Habermeier 064df81ee4 Add block number to activated leaves and associated fixes (#2718)
* add number to `ActivatedLeavesUpdate`

* update subsystem util and overseer

* use new ActivatedLeaf everywhere

* sort view

* sorted and limited view in network bridge

* use live block hash only if it's newer

* grumples
2021-03-26 13:06:40 +01:00