pezkuwichain/pezkuwi-subxt
1
0
Fork 0
mirror of https://github.com/pezkuwichain/pezkuwi-subxt.git synced 2026-04-27 01:07:57 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,885 Commits 132 Branches 288 Tags
f4ec9b034127224acfcd6a1a8300a2c0af58a5cd
Commit Graph

48 Commits

Author SHA1 Message Date
Sergei Shulepov 87c753e35c collation-gen: don't hash validation code (#4628) 
* runtime-api: add validation_code_hash API

This is the first step to close
https://github.com/paritytech/polkadot/issues/4524

* collation-gen: don't hash validation code

Closes https://github.com/paritytech/polkadot/issues/4524
 2021-12-28 20:16:28 +00:00
Pierre Krieger 77d70ac0d3 Companion PR for removing Prometheus metrics prefix (#3623) 
* Companion PR for removing Prometheus metrics prefix

* Was missing some metrics

* Fix missing renames

* Fix test

* Fixes

* Update test

* Update Substrate

* Second time

* remove prefix from intergration test for zombienet

* update zombienet image

* Update Substrate

Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Javier Viola <pepoviola@gmail.com>
 2021-12-10 15:24:54 +01:00
Bernhard Schuster 0f1a9fb1eb remove Default from CandidateDescriptor (#4484) 
* remove Default from CandidateHash

* Apply suggestions from code review

Co-authored-by: Andronik Ordian <write@reusable.software>

* chore: fmt

* remove backed candidate default

* Partial migration away from CandidateReceipt::default

* Remove more CandidateReceipt defaults

* fmt

* Mostly remove CommittedCandidateReceipt default usage

* Remove CommittedCandidateReceipt

* Remove more Defaults from polakdot primitives v1 + fmt

* Remove more Default from polkadot primites v1

* WIP trying to get overseer example + tests to compile

* feat: add primitives test helpers

* reduce deps of helper

* update primitive helpers

* make candidate validation compile

* fixup cargo lock

* make av-store compile

* fixup disputes coordinator tests

* test: fixup backing

* test: fixup approval voting

* fixup bitfield signing

* test: fixup runtime-api

* test: fixup availability dist

* foxi[ pverseer test]

* remove some Defaults, remove bounds from `dummy`

All `fn dummy` in primitives need to be removed anyways.
This aids in the transition.

* it's a test helper, so always use std

* test: fixup parachains runtime tests

Excluding benches.

* fix keyring

* fix paras runtime properly, no more default

* Remove fn dummy() usage from approval voting

* Move TestCandidateBuilder out of av store to test helpers

* Make candidate validation tests pass

* Make most dispute coirdinator tests pass

* Make provisioner tests work

* Make availability recovery tests work with test helpers

* Update polkadot-collator-protocol tests

* Update statement distribution tests

* Update polkadot overseer examples and tests

* Derive default for validation code so we don't break unrelated things

* Make para runtime test pass (no bench)

* Some more work

* chore: cargo fmt

* cargo fix

* avoid some Default::default

* fixup dispute coordinator test

* remove unused crate deps

* remove Default::default wherever possible, replace by dummy_* for the most part

* chore: cargo fmt

* Remove some warnings

* Remove CommittedCandidateReceipt dummy

* Remove CandidateReceipt dummy

* Remove CandidateDescriptor dummy

* Remove commented out code

* Fix para runtime tests

* chore: nightly

* Some updates to the builder

* Dynamically adjust mock head data size

* Make dispute cooridinator tests work

* Fix test candidate_backing_reorders_votes work

* +nightly-2021-10-29 fmt

* Spelling and remove a default use in builder

* Various clean up

* More small updates

* fmt

* More small updates

* Doc comments for test helpers

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras_inherent.rs

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras_inherent.rs

* Update lib.rs

* review comments

* fix warnings

* fix test by using correct candidate receipt relay parent

Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: emostov <32168567+emostov@users.noreply.github.com>
Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Gavin Wood <gavin@parity.io>
 2021-12-10 12:12:07 +00:00
Bernhard Schuster 4adb8466a3 dev-comment spelling mistakes (#4434) 2021-12-06 15:20:29 +01:00
sandreim b0f89bbfbc Per subsystem CPU usage tracking (#4239) 
* SubsystemContext: add subsystem name str

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* Overseer builder proc macro changes

* initilize SubsystemContext name field.
* Add subsystem name in TaskKind::launch_task()

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* Update ToOverseer enum

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* Assign subsystem names to orphan tasks

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* cargo fmt

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* SubsystemContext: add subsystem name str

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* Overseer builder proc macro changes

* initilize SubsystemContext name field.
* Add subsystem name in TaskKind::launch_task()

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* Update ToOverseer enum

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* Assign subsystem names to orphan tasks

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* cargo fmt

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Rebase changes for new spawn() group param

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Add subsystem constat in JobTrait

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Add subsystem string

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix spawn() calls

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* cargo fmt

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix more tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Address PR review feedback #1

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Address PR review round 2

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fixes
- remove JobTrait::Subsystem
- fix tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* update Cargo.lock

Co-authored-by: Andronik Ordian <write@reusable.software>
 2021-11-11 18:53:37 +00:00
Andronik Ordian 8aba3ea8ca some spelling fixes (#4088) 
* quotes around Job::ToJob
* add new words to dictionary
 2021-10-18 11:59:46 +02:00
Shawn Tabrizi ff5d56fb76 cargo +nightly fmt (#3540) 
* cargo +nightly fmt

* add cargo-fmt check to ci

* update ci

* fmt

* fmt

* skip macro

* ignore bridges
 2021-08-02 10:47:33 +00:00
Denis Pisarev fc253e6e4d WIP: CI: add spellcheck (#3421) 
* CI: add spellcheck

* revert me

* CI: explicit command for spellchecker

* spellcheck: edit misspells

* CI: run spellcheck on diff

* spellcheck: edits

* spellcheck: edit misspells

* spellcheck: add rules

* spellcheck: mv configs

* spellcheck: more edits

* spellcheck: chore

* spellcheck: one more thing

* spellcheck: and another one

* spellcheck: seems like it doesn't get to an end

* spellcheck: new words after rebase

* spellcheck: new words appearing out of nowhere

* chore

* review edits

* more review edits

* more edits

* wonky behavior

* wonky behavior 2

* wonky behavior 3

* change git behavior

* spellcheck: another bunch of new edits

* spellcheck: new words are koming out of nowhere

* CI: finding the master

* CI: fetching master implicitly

* CI: undebug

* new errors

* a bunch of new edits

* and some more

* Update node/core/approval-voting/src/approval_db/v1/mod.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Update xcm/xcm-executor/src/assets.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Apply suggestions from code review

Co-authored-by: Andronik Ordian <write@reusable.software>

* Suggestions from the code review

* CI: scan only changed files

Co-authored-by: Andronik Ordian <write@reusable.software>
 2021-07-14 19:22:58 +02:00
Bernhard Schuster 3c9104daff refactor overseer into proc-macro based pattern (#2962) 2021-07-08 21:09:26 +02:00
Andronik Ordian ffc6f7c731 make ctx.spawn blocking (#3337) 
* make spawn sync

* improve error type
 2021-06-21 20:43:40 -05:00
Andronik Ordian cbeb7d0afd tabify tests (#3220) 
* tabify tests

* move mod tests; up
 2021-06-12 15:39:18 +02:00
Bernhard Schuster f698bf8548 chore: move tests into separate files (#3206) 
Moves tests into separate files
in order to limit the loc per file.
 2021-06-11 12:50:19 +00:00
Andronik Ordian 29b531f4ec remove tracing::intrument annotations (#3197) 
* remove tracing::intrument annotations

* remove unused param and leftover

* more leftovers
 2021-06-09 10:35:18 +00:00
Robert Habermeier 896ec8dbc3 Code, PoV compression and remove CompressedPoV struct (#2852) 
* use compressed blob in candidate-validation

* add some tests for compressed code blobs

* remove CompressedPoV and apply compression in collation-generation

* decompress BlockData before executing

* don't produce oversized collations

* add test for PoV decompression failure

* fix tests and clean up

* fix test

* address review and fix CI

* take this )
 2021-04-08 22:09:36 +02:00
Robert Habermeier 11b8e4c821 Collation protocol: stricter validators (#2810) 
* guide: declare one para as a collator

* add ParaId to Declare messages and clean up

* fix build

* fix the testerinos

* begin adding keystore to collator-protocol

* remove request_x_ctx

* add core_for_group

* add bump_rotation

* add some more helpers to subsystem-util

* change signing_key API to take ref

* determine current and next para assignments

* disconnect collators who are not on current or next para

* add collator peer count metric

* notes for later

* some fixes

* add data & keystore to test state

* add a test utility for answering runtime API requests

* fix existing collator tests

* add new tests

* remove sc_keystore

* update cargo lock

Co-authored-by: Andronik Ordian <write@reusable.software>
 2021-04-03 21:48:58 +02:00
Guillaume Thiolliere beca01f118 Ease parachain candidate code fetching (#2593) 
* code stored in para + modify CandidateDescriptor.

* WIP: digest + some more impl

* validation_code_hash in payload + check in inclusion

* check in client + refator

* tests

* fix encoding indices

* remove old todos

* fix test

* fix test

* add test

* fetch validation code inside collation-generation from the relay-chain

* HashMismatch -> PoVHashMismatch + miscompilation

* refactor, store hash when needed

* storage rename: more specific but slightly too verbose

* do not hash on candidate validation, fetch hash instead

* better test

* fix test

* guide updates

* don't panic in runtime

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
 2021-04-01 23:55:39 +02:00
Robert Habermeier 0794f69306 Add dispute types and change InclusionInherent to ParasInherent (#2791) 
* dispute types

* add Debug to dispute primitives in std and InherentData

* use ParachainsInherentData on node-side

* change inclusion_inherent to paras_inherent

* RuntimeDebug

* add type parameter to PersistedValidationData users

* fix test client

* spaces

* fix collation-generation test

* fix provisioner tests

* remove references to inclusion inherent
 2021-04-01 18:23:27 +02:00
Robert Klotzner 0a9fe852df Move non runtime related stuff into node/primitives (#2743) 
* Remove stuff out of the runtime that does not belong there.

There might be more, but it is a start.

* White space fixes.

* Fix tests.

* Leave whitespace in ui tests alone.

* Add back zstd for no reason.

* Fix browser wasm (hopefully)
 2021-03-29 02:15:44 +02:00
Robert Habermeier 064df81ee4 Add block number to activated leaves and associated fixes (#2718) 
* add number to `ActivatedLeavesUpdate`

* update subsystem util and overseer

* use new ActivatedLeaf everywhere

* sort view

* sorted and limited view in network bridge

* use live block hash only if it's newer

* grumples
 2021-03-26 13:06:40 +01:00
Bastian Köcher 15ae5dd410 Improve the logging (#2645) 2021-03-18 23:28:43 +00:00
Andronik Ordian baa691deb1 prefix parachain log targets with parachain:: (#2600) 
* prefix parachain log targets with parachain::

* even more consistent
 2021-03-10 17:07:56 +01:00
Bastian Köcher 4975521d48 Notify collators about seconded collation (#2430) 
* Notify collators about seconded collation

This pr adds functionality to inform a collator that its collation was
seconded by a parachain validator. Before this signed statement was only
gossiped over the validation substream. Now, we explicitly send the
seconded statement to the collator after it was validated successfully.

Besides that it changes the `CollatorFn` to return an optional result
sender that is informed when the build collation was seconded by a
parachain validator.

* Add test

* Make sure we only send `Seconded` statements

* Make sure we only receive valid statements

* Review feedback
 2021-02-14 17:36:04 +01:00
Sergei Shulepov 82a20778fd Diagnostics quality of life improvements (#2375) 
* Implement `Debug` manually for CandidateHash

This will make candidate hashes printed consistently without the
`CandidateHash(` and `)` decorations.

* Do not print CompressedPov's guts

It can be overwhelming. Better just use the size.

* Log when candidate is generated

* Print para_id and candidate_hash upon receiving a collation
 2021-02-04 10:02:20 +01:00
Bastian Köcher 9563652bf2 Put parachain head hash into CandidateDescriptor (#2310) 
* Put parachain head hash into `CandidateDescriptor`

* Update guide

* Add some checks
 2021-01-25 13:56:17 +01:00
Sergei Shulepov 226af6a877 Remove TransientValidationData (#2272) 
* collation-generation: use persisted validation data

* node: remote FullValidationData API

* runtime: remove FullValidationData API

* backing tests: use persisted validation data

* FullCandidateReceipt: use persisted validation data

This is not a big change since this type is not used anywhere

* Remove ValidationData and TransientValidationData

Also update the guide
 2021-01-18 18:57:09 -05:00
Bastian Köcher 5be092894e Add one Jaeger span per relay parent (#2196) 
* Add one Jaeger span per relay parent

This adds one Jaeger span per relay parent, instead of always creating
new spans per relay parent. This should improve the UI view, because
subsystems are now grouped below one common span.

* Fix doc tests

* Replace `PerLeaveSpan` to `PerLeafSpan`

* More renaming

* Moare

* Update node/subsystem/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Skip the spans

* Increase `spec_version`

Co-authored-by: Andronik Ordian <write@reusable.software>
 2021-01-05 15:09:25 +01:00
Andronik Ordian 3f5156e866 refactor View to include finalized_number (#2128) 
* refactor View to include finalized_number

* guide: update the NetworkBridge on BlockFinalized

* av-store: fix the tests

* actually fix tests

* grumbles

* ignore macro doctest

* use Hash::repeat_bytes more consistently

* broadcast empty leaves updates as well

* fix issuing view updates on empty leaves updates
 2020-12-17 12:50:58 -05:00
Sergei Shulepov 34e5812171 Add logging to collation-generation (#2121) 
Right now if the collation is not happening one will have to sprinkle
log statements and then recompile the code. It's doubly annoying if that
happens when working with Cumulus: that means one has to resort to
.cargo/config's `paths` or `diener`, which both are not ideal.

This just adds some verbose logging to save the investigators some time
when looking why the collations are not happening
 2020-12-16 16:34:26 +00:00
Robert Habermeier 709f990fb2 be more careful about fusing in select! (#2052) 2020-12-02 14:52:51 +01:00
Robert Habermeier 0a79d663e4 Move erasure root out of candidate commitments and into descriptor (#2010) 
* guide: move erasure-root to candidate descriptor

* primitives: move erasure root to descriptor

* guide: unify candidate commitments and validation outputs

* primitives: unify validation outputs and candidate commitments

* parachains-runtime: fix fallout

* runtimes: fix fallout

* collation generation: fix fallout

* fix stray reference in primitives

* fix fallout in node-primitives

* fix remaining fallout in collation generation

* fix fallout in candidate validation

* fix fallout in runtime API subsystem

* fix fallout in subsystem messages

* fix fallout in candidate backing

* fix fallout in availability distribution

* don't clone

* clone

Co-authored-by: Sergei Shulepov <sergei@parity.io>
 2020-11-27 16:39:42 +00:00
Andronik Ordian 69b103b1d5 overseer: send_msg should not return an error (#1995) 
* send_message should not return an error

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* s/send_logging_error/send_and_log_error

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
 2020-11-23 12:42:14 +01:00
Peter Goodspeed-Niklaus 0a5bc82529 Add Prometheus timers to the subsystems (#1923) 
* reexport prometheus-super for ease of use of other subsystems

* add some prometheus timers for collation generation subsystem

* add timing metrics to av-store

* add metrics to candidate backing

* add timing metric to bitfield signing

* add timing metrics to candidate selection

* add timing metrics to candidate-validation

* add timing metrics to chain-api

* add timing metrics to provisioner

* add timing metrics to runtime-api

* add timing metrics to availability-distribution

* add timing metrics to bitfield-distribution

* add timing metrics to collator protocol: collator side

* add timing metrics to collator protocol: validator side

* fix candidate validation test failures

* add timing metrics to pov distribution

* add timing metrics to statement-distribution

* use substrate_prometheus_endpoint prometheus reexport instead of prometheus_super

* don't include JOB_DELAY in bitfield-signing metrics

* give adder-collator ability to easily export its genesis-state and validation code

* wip: adder-collator pushbutton script

* don't attempt to register the adder-collator automatically

Instead, get these values with

```sh
target/release/adder-collator export-genesis-state
target/release/adder-collator export-genesis-wasm
```

And then register the parachain on https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9944#/explorer

To collect prometheus data, after running the script, create `prometheus.yml` per the instructions
at https://www.notion.so/paritytechnologies/Setting-up-Prometheus-locally-835cb3a9df7541a781c381006252b5ff
and then run:

```sh
docker run -v `pwd`/prometheus.yml:/etc/prometheus/prometheus.yml:z --network host prom/prometheus
```

Demonstrates that data makes it across to prometheus, though it is likely to be useful in the future
to tweak the buckets.

* Update parachain/test-parachains/adder/collator/src/cli.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* use the grandpa-pause parameter

* skip metrics in tracing instrumentation

* remove unnecessary grandpa_pause cli param

Co-authored-by: Andronik Ordian <write@reusable.software>
 2020-11-20 15:04:51 +01:00
Peter Goodspeed-Niklaus e49989971d Add tracing support to node (#1940) 
* drop in tracing to replace log

* add structured logging to trace messages

* add structured logging to debug messages

* add structured logging to info messages

* add structured logging to warn messages

* add structured logging to error messages

* normalize spacing and Display vs Debug

* add instrumentation to the various 'fn run'

* use explicit tracing module throughout

* fix availability distribution test

* don't double-print errors

* remove further redundancy from logs

* fix test errors

* fix more test errors

* remove unused kv_log_macro

* fix unused variable

* add tracing spans to collation generation

* add tracing spans to av-store

* add tracing spans to backing

* add tracing spans to bitfield-signing

* add tracing spans to candidate-selection

* add tracing spans to candidate-validation

* add tracing spans to chain-api

* add tracing spans to provisioner

* add tracing spans to runtime-api

* add tracing spans to availability-distribution

* add tracing spans to bitfield-distribution

* add tracing spans to network-bridge

* add tracing spans to collator-protocol

* add tracing spans to pov-distribution

* add tracing spans to statement-distribution

* add tracing spans to overseer

* cleanup
 2020-11-20 12:02:04 +01:00
Sergei Shulepov c96f8cfcca Implement HRMP (#1900) 
* HRMP: Update the impl guide

* HRMP: Incorporate the channel notifications into the guide

* HRMP: Renaming in the impl guide

* HRMP: Constrain the maximum number of HRMP messages per candidate

This commit addresses the HRMP part of https://github.com/paritytech/polkadot/issues/1869

* XCM: Introduce HRMP related message types

* HRMP: Data structures and plumbing

* HRMP: Configuration

* HRMP: Data layout

* HRMP: Acceptance & Enactment

* HRMP: Test base logic

* Update adder collator

* HRMP: Runtime API for accessing inbound messages

Also, removing some redundant fully-qualified names.

* HRMP: Add diagnostic logging in acceptance criteria

* HRMP: Additional tests

* Self-review fixes

* save test refactorings for the next time

* Missed a return statement.

* a formatting blip

* Add missing logic for appending HRMP digests

* Remove the channel contents vectors which became empty

* Tighten HRMP channel digests invariants.

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Remove a note about sorting for channel id

* Add missing rustdocs to the configuration

* Clarify and update the invariant for HrmpChannelDigests

* Make the onboarding invariant less sloppy

Namely, introduce `Paras::is_valid_para` (in fact, it already is present
in the implementation) and hook up the invariant to that.

Note that this says "within a session" because I don't want to make it
super strict on the session boundary. The logic on the session boundary
should be extremely careful.

* Make `CandidateCheckContext` use T::BlockNumber for hrmp_watermark

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
 2020-11-06 15:35:36 +00:00
Bastian Köcher 8a2911b85d Distribute a PoV after seconding it (#1924) 
We need to distribute the PoV after we have seconded it. Other nodes
that will receive our `Secondded` statement and want to validate the
candidate another time will request this PoV from us.
 2020-11-06 08:38:44 +00:00
Andronik Ordian 2cde7732da more resilient subsystems (#1908) 
* backing: extract log target

* bitfield-signing: extract log target

* utils: fix a typo

* provisioner: extract log target

* candidate selection: remove unused error variant

* bitfield-distribution: change the return type of run

* pov-distribution: extract log target

* collator-protocol: simplify runtime request

* collation-generation: do not exit early on error

* collation-generation: do not exit on double init

* collator-protocol: do not exit on errors and rename LOG_TARGET

* collator-protocol: a workaround for ununused imports warning

* Update node/network/bitfield-distribution/src/lib.rs

* collation-generation: elevate warn! to error!

* collator-protocol: fix imports

* post merge fix

* fix compilation
 2020-11-05 14:22:41 +01:00
Fedor Sakharov 935fcd1666 Change SpawnedSubsystem type to log subsystem errors (#1878) 
* Change SpawnedSubsystem type to log subsystem errors

* Remove clone
 2020-10-28 20:57:06 +01:00
Sergei Shulepov e0336626ae let go fees (#1867) 2020-10-28 13:11:06 +01:00
Sergei Shulepov 9903bca259 Downward Message Processing implementation (#1859) 
* DMP: data structures and plumbing

* DMP: Implement DMP logic in the router module

DMP: Integrate DMP parts into the inclusion module

* DMP: Introduce the max size limit for the size of a downward message

* DMP: Runtime API for accessing inbound messages

* OCD

Small clean ups

* DMP: fix the naming of the error

* DMP: add caution about a non-existent recipient
 2020-10-28 10:41:42 +00:00
Bernhard Schuster f345123748 introduce errors with info (#1834) 2020-10-27 08:10:03 +01:00
Bastian Köcher 2b798f8571 Use Pin<Box<>> for collation future (#1792) 2020-10-07 11:29:00 +00:00
Bastian Köcher 325c24580e Pass relay parent as argument when collating (#1789) 
This pr changes the collation function to also pass the current relay
parent the parachain block should be build on.
 2020-10-06 18:37:19 +00:00
Bastian Köcher a4662104db Make collation an optional return (#1787) 
This pr changes the collator interface function to return an optional
collation instead of a collation. This is required as the parachain
itself can fail to generate a valid collation for various reason. Now if
the collation fails it will return `None`.

Besides that the pr adds some `RuntimeDebug` derive for `ValidationData`
and removes some whitespaces.
 2020-10-06 11:57:10 +02:00
Andronik Ordian de05bec4d6 move Metrics to utils (#1765) 2020-09-29 11:42:20 +00:00
Sergei Shulepov 796de5f3e3 collation-generation: guide and tidying (#1753) 
* Guide: Change for {Global,Local}ValidationData to ValidationData

Seems like that was missed during the transition from notions of global and local validation data to persistent and transient

* collation-generation: a small drive-by fix of line length

* collation-generation: extract an error module

this allows us to avoid piling Error and Result types from different contexts, specifically std vs. module-local

* collation-generation: extract LOG_TARGET from log statements
 2020-09-25 16:33:10 +00:00
Robert Habermeier 262574fc49 Implement validation data refactor (#1585) 
* update primitives

* correct parent_head field

* make hrmp field pub

* refactor validation data: runtime

* refactor validation data: messages

* add arguments to full_validation_data runtime API

* port runtime API

* mostly port over candidate validation

* remove some parameters from ValidationParams

* guide: update candidate validation

* update candidate outputs

* update ValidationOutputs in primitives

* port over candidate validation

* add a new test for no-transient behavior

* update util runtime API wrappers

* candidate backing

* fix missing imports

* change some fields of validation data around

* runtime API impl

* update candidate validation

* fix backing tests

* grumbles from review

* fix av-store tests

* fix some more crates

* fix provisioner tests

* fix availability distribution tests

* port collation-generation to new validation data

* fix overseer tests

* Update roadmap/implementers-guide/src/node/utility/candidate-validation.md

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
 2020-08-18 14:41:40 +02:00
Andronik Ordian e7ead40255 initial prometheus metrics (#1536) 
* service-new: cosmetic changes

* overseer: draft of prometheus metrics

* metrics: update active_leaves metrics

* metrics: extract into functions

* metrics: resolve XXX

* metrics: it's ugly, but it works

* Bump Substrate

* metrics: move a bunch of code around

* Bumb substrate again

* metrics: fix a warning

* fix a warning in runtime

* metrics: statements signed

* metrics: statements impl RegisterMetrics

* metrics: refactor Metrics trait

* metrics: add Metrics assoc type to JobTrait

* metrics: move Metrics trait to util

* metrics: fix overseer

* metrics: fix backing

* metrics: fix candidate validation

* metrics: derive Default

* metrics: docs

* metrics: add stubs for other subsystems

* metrics: add more stubs and fix compilation

* metrics: fix doctest

* metrics: move to subsystem

* metrics: fix candidate validation

* metrics: bitfield signing

* metrics: av store

* metrics: chain API

* metrics: runtime API

* metrics: stub for avad

* metrics: candidates seconded

* metrics: ok I gave up

* metrics: provisioner

* metrics: remove a clone by requiring Metrics: Sync

* metrics: YAGNI

* metrics: remove another TODO

* metrics: for later

* metrics: add parachain_ prefix

* metrics: s/signed_statement/signed_statements

* utils: add a comment for job metrics

* metrics: address review comments

* metrics: oops

* metrics: make sure to save files before commit 😅

* use _total suffix for requests metrics

Co-authored-by: Max Inden <mail@max-inden.de>

* metrics: add tests for overseer

* update Cargo.lock

* overseer: add a test for CollationGeneration

* collation-generation: impl metrics

* collation-generation: use kebab-case for name

* collation-generation: add a constructor

Co-authored-by: Gav Wood <gavin@parity.io>
Co-authored-by: Ashley Ruglys <ashley.ruglys@gmail.com>
Co-authored-by: Max Inden <mail@max-inden.de>
 2020-08-18 11:18:54 +02:00
Peter Goodspeed-Niklaus 54bec3bfc0 implement collation generation subsystem (#1557) 
* start sketching out a collation generation subsystem

* invent a basic strategy for double initialization

* clean up warnings

* impl util requests from runtime assuming a context instead of a FromJob sender

* implement collation generation algorithm from guide

* update AllMessages in tests

* fix trivial review comments

* remove another redundant declaration from merge

* filter availability cores by para_id

* handle new activations each in their own async task

* update guide according to the actual current implementation

* add initialization to guide

* add general-purpose subsystem_test_harness helper

* write first handle_new_activations test

* add test that handle_new_activations filters local_validation_data requests

* add (failing) test of collation distribution message sending

* rustfmt

* broken: work on fixing sender test

Unfortunately, for reasons that are not yet clear, despite the public key
and checked data being identical, the signer is not producing an identical
signature. This commit produces this output (among more):

signing with  Public(c4733ab0bbe3ba4c096685d1737a7f498cdbdd167a767d04a21dc7df12b8c858 (5GWHUNm5...))
checking with Public(c4733ab0bbe3ba4c096685d1737a7f498cdbdd167a767d04a21dc7df12b8c858 (5GWHUNm5...))
signed payload:  [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 10, 0, 0, 0, c7, e5, c0, 64, 7a, db, fe, 44, 81, e5, 51, 11, 79, 9f, a5, 63, 93, 94, 3c, c4, 36, c6, 30, 36, c2, c5, 44, a2, 1b, db, b7, 82, 3, 17, a, 2e, 75, 97, b7, b7, e3, d8, 4c, 5, 39, 1d, 13, 9a, 62, b1, 57, e7, 87, 86, d8, c0, 82, f2, 9d, cf, 4c, 11, 13, 14]
checked payload: [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 10, 0, 0, 0, c7, e5, c0, 64, 7a, db, fe, 44, 81, e5, 51, 11, 79, 9f, a5, 63, 93, 94, 3c, c4, 36, c6, 30, 36, c2, c5, 44, a2, 1b, db, b7, 82, 3, 17, a, 2e, 75, 97, b7, b7, e3, d8, 4c, 5, 39, 1d, 13, 9a, 62, b1, 57, e7, 87, 86, d8, c0, 82, f2, 9d, cf, 4c, 11, 13, 14]

* fix broken test

* collation function returns commitments hash

It doesn't look like we use the actual commitments data anywhere, and
it's not obvious if there are any fields of `CandidateCommitments`
not available to the collator, so this commit just assigns them the
entire responsibility of generating the hash.

* add missing overseer impls

* calculating erasure coding is polkadot's responsibility, not cumulus

* concurrentize per-relay_parent requests
 2020-08-17 14:27:37 +02:00