Commit Graph
100 Commits
Author SHA1 Message Date
eskimorandeskimor 5b3f659ce3 Remove redundant book keeping in dispute coordinator. (#1432)
* Remove redundant book keeping in dispute coordinator.

* Fix docs.

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
2023-09-11 12:25:40 +03:00
eskimorandeskimor 1e2a2f0c69 Fix nothing scheduled on session boundary (#1403)
* Fix scheduled state at session boundaries.

* Cleanup + better docs.

* More cleanup and fixes.

* Remove 12s hack.

* Add dep.

* Make clippy happy

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
2023-09-06 18:19:21 +02:00
eskimorandeskimor 23a2b7b5b3 Fix wrong ref counting (#1358)
* Fix wrong ref counting

in case of multiple inserts at same height.

* More warn

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
2023-09-01 21:56:09 +03:00
eaf057c5ed Parathreads Feature Branch (#6969)
* First baby steps

* Split scheduler into several modules

* Towards a more modular approach for scheduling

* move free_cores; IntoInterator -> BTreeMap

* Move clear()

* Move more functions out of scheduler

* Change weight composition

* More abstraction

* Further refactor

* clippy

* fmt

* fix test-runtime

* Add parathreads pallet to construct_runtime!

* Make all runtimes use (Parachains, Parathreads) scheduling

* Delete commented out code

* Remove parathreads scheduler from westend, rococo, and kusama

* fix rococo, westend, and kusama config

* Revert "fix rococo, westend, and kusama config"

This reverts commit 59e4de380d5c7d17eaaba5e2c2b81405de3465e3.

* Revert "Remove parathreads scheduler from westend, rococo, and kusama"

This reverts commit 4c44255296083ac5670560790ed77104917890a4.

* Remove CoreIndex from free_cores

* Remove unnecessary struct for parathreads

* parathreads provider take 1

* Comment out parathread tests

* Pop into lookahead

* fmt

* Fill lookahead with two entries for parachains

* fmt

* Current stage

* Towards ab parathreads

* no AB use

* Make tests typecheck

* quick hack to set scheduling lookahead to 1

* Fix scheduler tests

* fix paras_inherent tests

* misc

* Update more of a test

* cfg(test)

* some cleanup

* Undo paras_inherent changes

* Adjust paras inherent tests

* Undo changes to v2 primitives

* Undo v2 mod changes to tests

* minor

* Remove parathreads assigner and pallet

* minor

* minor

* more cleanup

* fmt

* minor

* minor

* minor

* Remove on_new_session from assignment provider

* Make adder collator integration test pass

* disable failing unit tests

* minor

* minor

* re-enable one unit test

* minor

* handle retries, add concluded para to pop interface

* comment out unused code

* Remove core_para from interface

* Remove first claimqueue element on clear if None instead removing all Nones

* Move claimqueue get out of loop

* Use VecDeque instead of Ved in ClaimQueue

* Make occupied() AB ready(?)

* handle freed disputed in clear_and_fill_claimqueue

* clear_and_fill_claimqueue returns scheduled Vec

* Rename and minor refactor

* return position of assignment taken from claimqueue

* minor

* Fix session boundary parachains number change + extended test

* Fix runtimes

* Fix polkadot runtime

* Remove polkadot pallet from benchmarks

* fix test runtime

* Add storage migration

* Minor refactor

* Minor

* migratin typechecks

* Add migration to runtimes

* Towards modular scheduling II (#6568)

* Add post migration check

* pebkac

* Disable migrations but mine

* Revert "Disable migrations but mine"

This reverts commit 4fa5c5a370c199944a7e0926f50b08626bfbad4c.

* Move scheduler migration

* Revert "Move scheduler migration"

This reverts commit a16b1659a907950bae048a9f7010f2aa76e02b6d.

* Fix migration

* cleanup

* Don't lose retries value anymore

* comment out test function

* Remove retries value from Assignment again

* minor

* Make collator for parathreads optional

* data type refactor

* update scheduler tests

* Change test function cfg

* comment out test function

* Try cfg(test) only

* fix cfg flags

* Add get_max_retries function to provider interface (#7047)

* Fix merge commit

* pebkac

* fix merge

* update cargo.lock

* fix merge

* fix merge

* Use btreemap instead of vec, fix scheduler calls.

* Use imported `ScheduledCore`

* Remove unused import in inclusion tests

* Use keys() instead of mapping over a BTreeMap

* Fix migrations for parachains scheduler

* Use BlockNumberFor<T> everywhere in scheduler

* Add on demand assignment provider pallet (#7110)

* Address some PR comments

* minor

* more cleanup

* find_map and timeout availability fixes

* Change default scheduling_lookahead to 1

* Add on demand assignment provider pallet

* Move test-runtime to new assignment provider

* Run cargo format on scheduler tests

* minor

* Mutate cores in single loop

* timeout predicate simplification

* claimqueue desired size fix

* Replace expect by ok_or

* More improvements

* Fix push back order and next_up_on_timeout

* minor

* session change docs

* Add pre_new_session call to hand pre session updates

* Remove sc_network dependency and PeerId from unnecessary data structures

* Remove unnecessary peer_ids

* Add OnDemandOrdering proxy (#7156)

* Add OnDemandBidding proxy

* Fix names

* OnDemandAssigner for rococo only

* Check PeerId in collator protocol before fetching collation

* On occupied, remove non occupied cores from the claimqueue front and refill

* Add missing docs

* Comment out unused field

* fix ScheduledCore in tests

* Fix the fix

* pebkac

* fmt

* Fix occupied dropping

* Remove double import

* ScheduledCore fixes

* Readd sc-network dep

* pebkac

* OpaquePeerId -> PeerId in can_collate interface

* Cargo.lock update for interface change

* Remove checks not needed anymore?

* Drop occupied core on session change if it would time out after the new session

* Add on demand assignment provider pallet

* Move test-runtime to new assignment provider

* Run cargo format on scheduler tests

* Add OnDemandOrdering proxy (#7156)

* Add OnDemandBidding proxy

* Fix names

* OnDemandAssigner for rococo only

* Remove unneeded config values

* Update comments

* Use and_then for queue position

* Return the max size of the spot queue on error

* Add comments to add_parathread_entry

* Add module comments

* Add log for when can_collate fails

* Change assigner queue type to `Assignment`

* Update assignment provider tests

* More logs

* Remove unused keyring import

* disable can_collate

* comment out can_collate

* Can collate first checks set if empty

* Move can_collate call to collation advertisement

* Fix backing test

* map to loop

* Remove obsolete check

* Move invalid collation test from backing to collator-protocol

* fix unused imports

* fix test

* fix Debug derivation

* Increase time limit on zombienet predicates

* Increase zombienet timeout

* Minor

* Address some PR comments

* Address PR comments

* Comment out failing assert due to on-demand assigner missing

* remove collator_restrictions info from backing

* Move can_collate to ActiveParas

* minor

* minor

* Update weight information for on demand config

* Add ttl to parasentry

* Fix tests missing parasentry ttl

* Adjust scheduler tests to use ttl default values

* Use match instead of if let for ttl drop

* Use RuntimeDebug trait for `ParasEntry` fields

* Add comments to on demand assignment pallet

* Fix spot traffic calculation

* Revert runtimedebug changes to primitives

* Remove runtimedebug derivation from `ParasEntry`

* Mention affinity in pallet level docs

* Use RuntimeDebug trait for ParasEntry child types

* Remove collator restrictions

* Fix primitive versioning and other merge issues

* Fix tests post merge

* Fix node side tests

* Edit parascheduler migration for clarity

* Move parascheduler migration up to next release

* Remove vestiges from merge

* Fix tests

* Refactor ttl handling

* Remove unused things from scheduler tests

* Move on demand assigner to own directory

* Update documentation

* Remove unused sc-network dependency in primitives

Was used for collator restrictions

* Remove unused import

* Reenable scheduler test

* Remove unused storage value

* Enable timeout predicate test and fix fn

Turns out that the issue with the compiler is fixed and we can now
use impl Trait in the manner used here.

* Remove unused imports

* Add benchmarking entry for perbill in config

* Correct typo

* Address review comments

* Log out errors when calculating spot traffic.

* Change parascheduler's log target name

* Update scheduler_common documentation

* Use mutate for affinity fns, add tests

* Add another on demand affinity test

* Unify parathreads and parachains in HostConfig (take 2) (#7452)

* Unify parathreads and parachains in HostConfig

* Fixed missed occurences

* Remove commented out lines

* `HostConfiguration v7`

* Fix version check

* Add `MigrateToV7` to `Unreleased`

* fmt

* fmt

* Fix compilation errors after the rebase

* Update runtime/parachains/src/scheduler/tests.rs

Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com>

* Update runtime/parachains/src/scheduler/tests.rs

Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com>

* fmt

* Fix migration test

* Fix tests

* Remove unneeded assert from tests

* parathread_cores -> on_demand_cores; parathread_retries -> on_demand_retries

* Fix a compilation error in tests

* Remove unused `use`

* update colander image version

---------

Co-authored-by: alexgparity <alex.gremm@parity.io>
Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com>
Co-authored-by: Javier Viola <javier@parity.io>

* Fix branch after merge with master

* Refactor out duplicate checks into a helper fn

* Fix tests post merge

* Rename add_parathread_assignment, add test

* Update docs

* Remove unused on_finalize function

* Add weight info to on demand pallet

* Update runtime/parachains/src/configuration.rs

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>

* Update runtime/parachains/src/scheduler_common/mod.rs

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>

* Update runtime/parachains/src/assigner_on_demand/mod.rs

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>

* Add benchmarking to on demand pallet

* Make place_order test check for success

* Add on demand benchmarks

* Add local test weights to rococo runtime

* Modify TTL drop behaviour to not skip claims

Previous behaviour would jump a new claim from the assignment provider
ahead in the claimqueue, assuming lookahead is larger than 1.

* Refactor ttl test to test claimqueue order

* Disable place_order ext. when no on_demand cores

* Use default genesis config for benchmark tests

* Refactor config builder param

* Move lifecycle test from scheduler to on demand

* Remove unneeded lifecycle test

Paras module via the parachain assignment provider doesn't provide
new assignments if a parachain loses it's lease. The on demand
assignment provider doesn't provide an assignment that is not a
parathread.

* Re enable validator shuffle test

* More realistic weights for place_order

* Remove redundant import

* Fix backwards compatibility (hopefully)

* ".git/.scripts/commands/bench/bench.sh" --subcommand=runtime --runtime=rococo --target_dir=polkadot --pallet=runtime_parachains::assigner_on_demand

* Fix tests.

* Fix off-by-one.

* Re enable claimqueue fills test

* Re enable schedule_rotates_groups test

* Fix fill_claimqueue_fills test

* Re enable next_up_on_timeout test, move fn

* Do not pop from assignment provider when retrying

* Fix tests missing collator in scheduledcore

* Add comment about timeout predicate.

* Rename parasentry retries to availability timeouts

* Re enable schedule_schedules... test

* Refactor prune retried test to new scheduler

* Have all scheduler tests use genesis_cfg fn

* Update docs

* Update copyright notices on new files

* Rename is_parachain_core to is_bulk_core

* Remove erroneous TODO

* Simplify import

* ".git/.scripts/commands/bench/bench.sh" --subcommand=runtime --runtime=rococo --target_dir=polkadot --pallet=runtime_parachains::configuration

* Revert AdvertiseCollation order shuffle

* Refactor place_order into keepalive and allowdeath

* Revert rename of hrmp max inbound channels

parachain encompasses both on demand and slot auction / bulk.

* Restore availability_timeout_predicate function

* Clean up leftover comments

* Update runtime/parachains/src/scheduler/tests.rs

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>

* ".git/.scripts/commands/bench/bench.sh" --subcommand=runtime --runtime=westend --target_dir=polkadot --pallet=runtime_parachains::configuration

---------

Co-authored-by: alexgparity <alex.gremm@parity.io>
Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Co-authored-by: Javier Viola <javier@parity.io>
Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: command-bot <>

* On Demand - update weights and small nits (#7605)

* Remove collator restriction test in inclusion

On demand parachains won't have collator restrictions implemented in
this way but will instead use a preferred collator registered to a
`ParaId` in `paras_registrar`.

* Remove redundant config guard for test fns

* Update weights

* Update WeightInfo for on_demand assigner

* Unify assignment provider parameters into one call (#7606)

* Combine assignmentprovider params into one fn call

* Move scheduler_common to a module under scheduler

* Fix ttl handling in benchmark builder

* Run cargo format

* Remove obsolete test.

* Small improvement.

* Use same migration pattern as config module

* Remove old TODO

* Change log target name for assigner on demand

* Fix migration

* Fix clippy warnings

* Add HostConfiguration storage migration to V8

* Add `MigrateToV8` to unreleased migrations for all runtimes

* Fix storage version check for config v8

* Set `StorageVersion` to 8 in `MigrateToV8`

* Remove dups.

* Update primitives/src/v5/mod.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

---------

Co-authored-by: alexgparity <alex.gremm@parity.io>
Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com>
Co-authored-by: antonva <anton.asgeirsson@parity.io>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com>
Co-authored-by: Javier Viola <javier@parity.io>
Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
2023-08-17 12:52:23 +00:00
367098bae5 Take into account size as well in weight limiting. (#7369)
* Take into account size as well in weight limiting.

* Fix logging.

* More logs.

* Remove randomized selection in provisioner

No longer supported by runtime.

* Fix and simplify weight calculation.

Random filtering of remote disputes got dropped.

* Make existing tests pass.

* Tests for size limiting.

* Fix provisioner.

* Remove rand dependency.

* Better default block length for tests.

* ".git/.scripts/commands/bench/bench.sh" runtime kusama runtime_parachains::paras_inherent

* ".git/.scripts/commands/bench/bench.sh" runtime polkadot runtime_parachains::paras_inherent

* ".git/.scripts/commands/bench/bench.sh" runtime westend runtime_parachains::paras_inherent

* Update runtime/parachains/src/paras_inherent/mod.rs

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>

* Update runtime/parachains/src/paras_inherent/mod.rs

Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>

* Add back missing line.

* Fix test.

* fmt fix.

* Add missing test annotation

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: command-bot <>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
2023-08-01 12:43:54 +00:00
eskimorandeskimor a25b871a16 Clarification on the use of vstaging (#7359)
* Clarifications on vstaging.

* Typos.

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
2023-06-21 12:23:03 +02:00
eskimoreskimorcommand-bot
596a9ccd02 Unify code paths of create_inherent and enter (#7137)
* Remove redundant enter call.

* Remove optionality in dispute signature checking

* Make enter_inner and create_inherent the same.

* Remove redundant metric.

* Unification: enter and create_inherent.

* Remove `enter_inner` function.

* Remove dead code.

* Remove redundant import.

* Remove dead code in disputes.

* ".git/.scripts/commands/bench/bench.sh" runtime polkadot runtime_parachains::paras_inherent

* ".git/.scripts/commands/bench/bench.sh" runtime kusama runtime_parachains::paras_inherent

* ".git/.scripts/commands/bench/bench.sh" runtime westend runtime_parachains::paras_inherent

* Merge fix.

* Fix tests.

* Remove obsolete comment.

* ".git/.scripts/commands/bench/bench.sh" runtime polkadot runtime_parachains::paras_inherent

* ".git/.scripts/commands/bench/bench.sh" runtime westend runtime_parachains::paras_inherent

* ".git/.scripts/commands/bench/bench.sh" runtime kusama runtime_parachains::paras_inherent

* Review remarks, fixes.

* Guide updates.

* Fmt.

* ".git/.scripts/commands/bench/bench.sh" runtime polkadot runtime_parachains::paras_inherent

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: command-bot <>
2023-06-14 10:57:06 +02:00
eskimoreskimorparity-processbot
f27d4b5bfe Companion for https://github.com/paritytech/polkadot/pull/7341 (#2726)
* Companion to enabling past session slashing on production.

* update lockfile for {"substrate", "polkadot"}

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: parity-processbot <>
2023-06-12 11:58:37 +00:00
eskimorandeskimor 4527f24735 Move vstaging to production (#7341)
* Move vstaging to production (and thus past session slashing).

WIP: test-runtime still needs to be fixed.

* Fix test-runtime.

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
2023-06-12 11:15:37 +00:00
eskimorandeskimor 798a9bac70 Version independent import. (#2708)
Co-authored-by: eskimor <eskimor@no-such-url.com>
2023-06-09 10:57:40 +02:00
eskimorandeskimor 33dd2584df Fix stalling dispute coordinator. (#7125)
* Fix stalling dispute coordinator.

* Initialization.

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
2023-04-25 16:42:45 +02:00
eskimorandeskimor 9fe528d5c7 Avoid redundant clone. (#6989)
Co-authored-by: eskimor <eskimor@no-such-url.com>
2023-03-31 14:53:05 +00:00
eskimorandeskimor 7d9a0b7cc3 Freeze note on SessionInfo. (#6818)
* Add note about frozen `SessionInfo`.

* Better comment.

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
2023-03-03 21:50:42 +01:00
eskimorandeskimor 4931eb0a97 Get rid of unnecessary cloning and work. (#6808)
Co-authored-by: eskimor <eskimor@no-such-url.com>
2023-03-02 07:53:32 +01:00
eskimor bec4168baa Fix some unjustified disputes (#6103)
* Fix indentation + add warning

on participation errors.

* Don't vote invalid on internal errors.

* Don't dispute on code compression error.

* Remove CodeDecompressionError

* Candidate not invalid if PVF preparation fails.

Instead: Report error.

* Fix malus

* Add clarifying comment.

* cargo fmt

* Fix indentation.
2023-01-22 20:21:37 +01:00
cc650fe53d Fix cycle dispute-coordinator <-> dispute-distribution (#6489)
* First iteration of message sender.

* dyn Fn variant (no cloning)

* Full implementation + Clone, without allocs on `Send`

* Further clarifications/cleanup.

* MessageSender -> NestingSender

* Doc update/clarification.

* dispute-coordinator: Send disputes on startup.

+ Some fixes, cleanup.

* Fix whitespace.

* Dispute distribution fixes, cleanup.

* Cargo.lock

* Fix spaces.

* More format fixes.

What is cargo fmt doing actually?

* More fmt fixes.

* Fix nesting sender.

* Fixes.

* Whitespace

* Enable logging.

* Guide update.

* Fmt fixes, typos.

* Remove unused function.

* Simplifications, doc fixes.

* Update roadmap/implementers-guide/src/node/disputes/dispute-coordinator.md

Co-authored-by: Marcin S. <marcin@bytedude.com>

* Fmt + doc example fix.

Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: Marcin S. <marcin@bytedude.com>
2023-01-10 12:04:05 +01:00
b0c7a1571d Update disputes subsytems section. (#6329)
* Update disputes subsytems section.

* Update roadmap/implementers-guide/src/node/disputes/README.md

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

* Update roadmap/implementers-guide/src/node/disputes/README.md

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

* Update roadmap/implementers-guide/src/node/disputes/README.md

Co-authored-by: ordian <write@reusable.software>

* Update roadmap/implementers-guide/src/node/disputes/README.md

Co-authored-by: ordian <write@reusable.software>

* Review feedback.

Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
Co-authored-by: ordian <write@reusable.software>
2022-12-27 12:27:55 +01:00
eskimorandeskimor 5c787f1512 Make sure errors get logged (#6451)
instead of silently dropped.

Co-authored-by: eskimor <eskimor@no-such-url.com>
2022-12-16 21:37:57 +01:00
eskimorandeskimor 3c1a28276d Fix wrong rate limit + add a few logs. (#6440)
* Add trace log

* More tracing.

* Fix redundant rate limit.

* Add trace log.

* Improve logging.

Co-authored-by: eskimor <eskimor@no-such-url.com>
2022-12-15 09:47:06 +00:00
c748250748 Make sure to preserve backing votes (#6382)
* Guide updates

* Consider more dead forks.

* Ensure backing votes don't get overridden.

* Fix spelling.

* Fix comments.

* Update node/primitives/src/lib.rs

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>

Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
2022-12-07 17:14:13 +00:00
7ba0847349 Rate limit improvements (#6315)
* We actually don't need to rate limit redundant requests.

Those redundant requests should not actually happen, but still.

* Add some logging.

* Also log message when the receiving side hit the rate limit.

* Update node/network/dispute-distribution/src/sender/mod.rs

Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>

Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
2022-11-23 16:43:30 +01:00
eskimorandeskimor c606deb32d Revert special casing of Kusama for grandpa rounds. (#6217)
Co-authored-by: eskimor <eskimor@no-such-url.com>
2022-11-15 10:04:24 +01:00
eskimorandeskimor 97dddbab93 Only report concluded if there is an actual dispute. (#6270)
* Only report concluded if there is an actual dispute.

Hence no "non"-disputes will be added to disputes anymore.

* Fix redundant check.

* Test for no onesided disputes.

Co-authored-by: eskimor <eskimor@no-such-url.com>
2022-11-14 12:41:45 +01:00
eskimorandeskimor bae4f0d3c2 Make ValidateUnsigned available on all chains for paras. (#6214)
Co-authored-by: eskimor <eskimor@no-such-url.com>
2022-10-30 17:15:42 +01:00
Robert Klotzner e0e836671f Add unknown words (#6105) 2022-10-04 18:47:52 +02:00
Robert KlotznerandTsvetomir Dimitrov 938bc96a2c Batch vote import in dispute-distribution (#5894)
* Start work on batching in dispute-distribution.

* Guide work.

* More guide changes. Still very much WIP.

* Finish guide changes.

* Clarification

* Adjust argument about slashing.

* WIP: Add constants to receiver.

* Maintain order of disputes.

* dispute-distribuion sender Rate limit.

* Cleanup

* WIP: dispute-distribution receiver.

- [ ] Rate limiting
- [ ] Batching

* WIP: Batching.

* fmt

* Update `PeerQueues` to maintain more invariants.

* WIP: Batching.

* Small cleanup

* Batching logic.

* Some integration work.

* Finish.

Missing: Tests

* Typo.

* Docs.

* Report missing metric.

* Doc pass.

* Tests for waiting_queue.

* Speed up some crypto by 10x.

* Fix redundant import.

* Add some tracing.

* Better sender rate limit

* Some tests.

* Tests

* Add logging to rate limiter

* Update roadmap/implementers-guide/src/node/disputes/dispute-distribution.md

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>

* Update roadmap/implementers-guide/src/node/disputes/dispute-distribution.md

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>

* Update node/network/dispute-distribution/src/receiver/mod.rs

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>

* Review feedback.

* Also log peer in log messages.

* Fix indentation.

* waker -> timer

* Guide improvement.

* Remove obsolete comment.

* waker -> timer

* Fix spell complaints.

* Fix Cargo.lock

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
2022-10-04 18:02:05 +02:00
Robert Klotzner 92dc27584a Use active_leaves instead of known_leaves (#6068)
* Use active_leaves for synchronization.

Otherwise on reverts the syncrhonization of create inherent would not
work.

* Add some trace logs.
2022-10-04 10:11:02 +02:00
Robert Klotzner 260f70e39e Demote warning (#6080)
This can happen under heavy load - no value in warning here.

Scenario this happens:

- New slot
- We get asked about what fork to build on
- We start building (create inherent gets called)
- We learn about a better fork (but Babe does not care as it already has a fork)
- 6 seconds passed - New slot
- Babe asks about what for to build on - we answer with the best block we learned about 6 seconds ago (slightly after we got asked the last time)
- We start building (on that old block)
- Milliseconds later we learn about a new block (the one from the slot we are actually in now)
- We kick the old leaf -> create inherent fails

Full discussion: https://github.com/paritytech/substrate/issues/12375
2022-09-29 15:19:19 +02:00
Robert Klotzner 548b4c6c71 Validate chunks from disk in availability-recovery (#6078)
* Don't use corrupted chunks from disk.

Otherwise we would be going to dispute the candidate and get slashed.

* Add tests
2022-09-29 14:16:12 +02:00
Robert Klotzner 5045aeb2f9 Some more information on disputes for the guide (#6006)
* Add some notes about treatment of already finalized blocks.

* More info in the guide as discussed with Jakob.

* Remove references to private repo.
2022-09-23 14:41:06 +02:00
Robert Klotzner 7e4bea2578 Double approval checking timeout. (#5951) 2022-09-05 16:45:25 +00:00
Robert KlotznerandVsevolod Stakhov 254582df7f Don't store available data on disputes (#5950)
* Don't store available data on disputes

If there are lots of disputes, this leads to blowing up disk space on
validators. Rob luckily remembered that we do store the full
availability in participation.

The argument in the code does not make too much sense with the current
implementation, as no validator will ever request anything else from us,
than the one piece we are meant to posess.

* Fix warnings.

* Fix compile warnings

* Remove redundant field.

Co-authored-by: Vsevolod Stakhov <vsevolod.stakhov@parity.io>
2022-09-01 13:42:01 +02:00
Robert Klotzner 979483fd80 Fix wrong logic. (#5931) 2022-08-25 16:32:31 +02:00
Robert KlotznerandAndrei Sandu c83eaa2475 Send back empty votes + log in approval-voting in case candidate entry is missing. (#5925)
* Send back empty votes + log.

* Update node/core/approval-voting/src/lib.rs

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
2022-08-24 15:44:13 +00:00
Robert Klotzner 16a356524b Break cycle collator-procol - backing (#5883) 2022-08-16 15:31:12 +00:00
Robert Klotzner 05a7966505 Don't request votes if there is nothing to request. (#5888)
* Don't request votes if there is nothing to request.

* Add trace log
2022-08-16 14:49:25 +00:00
Robert Klotznerandasynchronous rob ca6297c853 Reduce dispute coordinator load (#5785)
* Don't import backing statements directly

into the dispute coordinator. This also gets rid of a redundant
signature check. Both should have some impact on backing performance.
In general this PR should make us scale better in the number of parachains.

Reasoning (aka why this is fine):

For the signature check: As mentioned, it is a redundant check. The
signature has already been checked at this point. This is even made
obvious by the used types. The smart constructor is not perfect as
discussed [here](https://github.com/paritytech/polkadot/issues/3455),
but is still a reasonable security.

For not importing to the dispute-coordinator: This should be good as the
dispute coordinator does scrape backing votes from chain. This suffices
in practice as a super majority of validators must have seen a backing
fork in order for a candidate to get included and only included
candidates pose a threat to our system. The import from chain is
preferable over direct import of backing votes for two reasons:

1. The import is batched, greatly improving import performance. All
   backing votes for a candidate are imported with a single import.
   And indeed we were able to see in metrics that importing votes
   from chain is fast.
2. We do less work in general as not every candidate for which
   statements are gossiped might actually make it on a chain. The
   dispute coordinator as with the current implementation would still
   import and keep those votes around for six sessions.

While redundancy is good for reliability in the event of bugs, this also
comes at a non negligible cost. The dispute-coordinator right now is the
subsystem with the highest load, despite the fact that it should not be
doing much during mormal operation and it is only getting worse
with more parachains as the load is a direct function of the number of statements.

We'll see on Versi how much of a performance improvement this PR

* Get rid of dead code.

* Dont send approval vote

* Make it pass CI

* Bring back tests for fixing them later.

* Explicit signature check.

* Resurrect approval-voting tests (not fixed yet)

* Send out approval votes in dispute-distribution.

Use BTreeMap for ordered dispute votes.

* Bring back an important warning.

* Fix approval voting tests.

* Don't send out dispute message on import + test

+ Some cleanup.

* Guide changes.

Note that the introduced complexity is actually redundant.

* WIP: guide changes.

* Finish guide changes about dispute-coordinator

conceputally. Requires more proof read still.

Also removed obsolete implementation details, where the code is better
suited as the source of truth.

* Finish guide changes for now.

* Remove own approval vote import logic.

* Implement logic for retrieving approval-votes

into approval-voting and approval-distribution subsystems.

* Update roadmap/implementers-guide/src/node/disputes/dispute-coordinator.md

Co-authored-by: asynchronous rob <rphmeier@gmail.com>

* Review feedback.

In particular: Add note about disputes of non included candidates.

* Incorporate Review Remarks

* Get rid of superfluous space.

* Tidy up import logic a bit.

Logical vote import is now separated, making the code more readable and
maintainable.

Also: Accept import if there is at least one invalid signer that has not
exceeded its spam slots, instead of requiring all of them to not exceed
their limits. This is more correct and a preparation for vote batching.

* We don't need/have empty imports.

* Fix tests and bugs.

* Remove error prone redundancy.

* Import approval votes on dispute initiated/concluded.

* Add test for approval vote import.

* Make guide checker happy (hopefully)

* Another sanity check + better logs.

* Reasoning about boundedness.

* Use `CandidateIndex` as opposed to `CoreIndex`.

* Remove redundant import.

* Review remarks.

* Add metric for calls to request signatures

* More review remarks.

* Add metric on imported approval votes.

* Include candidate hash in logs.

* More trace log

* Break cycle.

* Add some tracing.

* Cleanup allowed messages.

* fmt

* Tracing + timeout for get inherent data.

* Better error.

* Break cycle in all places.

* Clarified comment some more.

* Typo.

* Break cycle approval-distribution - approval-voting.

Co-authored-by: asynchronous rob <rphmeier@gmail.com>
2022-08-16 08:16:30 +02:00
Robert Klotzner 1dc991374b Get rid of unnecessary clone. (#5685) 2022-06-16 19:15:20 +00:00
Robert KlotznerandAndronik c3c10ce4c4 Fix a storage leak in parachains db (#5594)
* Fix cleanup of old votes.

* Cleanup.

* Get rid of redundant import

* Tests + logging

* Fix db key name.

* Add some reasoning to batch size.

* Add dispute data to indexed columns

* Fix fmt

* Add helper function.

* Fix typos.

* Update node/core/dispute-coordinator/src/db/v1.rs

Co-authored-by: Andronik <write@reusable.software>

* Update node/core/dispute-coordinator/src/db/v1.rs

Co-authored-by: Andronik <write@reusable.software>

* Add metric for how long cleanup takes.

Co-authored-by: Andronik <write@reusable.software>
2022-06-13 15:38:42 +02:00
Robert Klotzner 72723c8a9f Request backed candidates unbounded. (#5610)
We are awaiting on the oneshot anyways, so we have back pressure. By
using the unbounded channel make log messages like the following less
likely (due to higher priority):

	2022-05-30 13:46:38
2022-05-30 11:46:38.565  WARN tokio-runtime-worker parachain::provisioner: failed to assemble or send inherent data err=CanceledBackedCandidates(Canceled)
2022-05-30 22:04:12 +02:00
Robert Klotzner 7a602c47ab Double grandpa gossip duration for Kusama and test networks (#5448)
* Double grandpa gossip duration.

* Make resend period slightly larger.

So it won't get triggered by additional grandpa delay.

* Bump other values as well.

* Don't change gossip duration on Polkadot.

(and Westend as it is meant to be a testbed for Polkadot)
2022-05-10 12:48:18 +00:00
Robert Klotzner 35a87e0fa3 Include log target for parachains-inherent. (#5470) 2022-05-06 13:20:56 +00:00
Robert Klotzner 1213a3a0b9 Make versi-staging match previous configs (#5416)
Also get rid of pointless versi chainspec.
2022-05-04 22:20:57 +02:00
Robert KlotznerandAndronik 8dbc4d8a6e Reduce log verbosity (#5440)
* Reduce log verbosity

* Update node/network/availability-recovery/src/lib.rs

Co-authored-by: Andronik <write@reusable.software>

Co-authored-by: Andronik <write@reusable.software>
2022-05-03 14:04:07 +02:00
Robert Klotzner 2bbae6e6ef Increase timeout, as it seems to be hit on CI (#5427)
occasionally.

If this keeps being a problem, we might also increase the channel size:

https://github.com/paritytech/polkadot/blob/49f7e5cce45c44b6fcbad3c612b92657eb7a52df/node/subsystem-test-helpers/src/lib.rs#L239
2022-05-02 12:50:35 +02:00
Robert KlotznerandBastian Köcher ac23b3f96c Quick'n dirty Versi chainspec config. (#5412)
* Quick'n dirty Versi chainspec config.

* Right order

* Fix wrong variable usage

* Fix weird typo.

* Update node/service/src/chain_spec.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2022-04-28 17:10:14 +02:00
Robert Klotzner f820db494f dispute-coordinator: Cleanup + Bug fixes (#5323)
* Make import confirmation oneshot optional.

* Cleanup for further improvements.

* Queue adoptions.

* Fix fieldname

* Use correct relay parent

* Fix scraper tests.

* Small optimization.

* Fix all tests.

* Fix other tests.

* fmt

* spelling

* Fix warning.
2022-04-19 14:51:02 +02:00
Robert Klotzner 286fedf04a Make debug builds more useful & testing faster. (#4807)
* Make debug builds more useful & testing likely faster.

Stolen from substrate.

* Add some more packages from

https://github.com/paritytech/polkadot/pull/3344/files

* Fix compilation.
2022-04-16 13:42:33 +02:00
Robert Klotzner 243c855f03 Session is actually ancient not unknown. (#5286) 2022-04-08 13:04:59 +02:00
Robert Klotzner 4ca9691dcf Better metrics for availability-recovery (#5249)
* Better metrics.

- Fix time metric
- Add counters

* Typo

* Better docs.
2022-04-04 14:31:08 +00:00
Robert Klotzner 3688e46d1b Don't wait for dispute coordinator (#5157)
* Don't wait for dispute coordinator

in backing and approval-voting - we are single threaded there, so this
is blocking everything.

* Add missing import.

* Don't warn on dropped receiver.
2022-03-18 15:15:50 +00:00
Robert Klotzner 569599853d Logs for large messages + metrics on size. (#5143)
* Logs for large messages + metrics on size.

* Fix typo
2022-03-17 19:57:35 +00:00
Robert Klotzner 582f8628ea Add more info to logs. (#5123) 2022-03-15 19:39:43 +00:00
Robert Klotzner acaf0d84fe Improve parachain liveness by reducing required number of backing votes (#5016)
* Let node side take advantage reduced requirements

in the runtime and hopefully improve liveness of parachains by means of
that.

* Fix tests.
2022-03-08 20:33:55 +00:00
93b7263019 Better spam slots handling (#4845)
* Only bypass spam slots on actualy approval/backing vote import.

* Fix spam slot treatment.

* More tests.

* Make sure backing statements import works.

* Aaaaaand the actual proper fix.

* Better docs + spelling fixes.

* Fix.

* Typo

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* Fix doc.

* Update node/core/dispute-coordinator/src/real/initialized.rs

* minor rewording, line wrap

* fix test case

* Fix obsolete comment.

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
2022-02-17 15:18:52 +01:00
0302271053 Fix incomplete sorting. (#4795)
* Fix incomplete sorting.

* fmt.

* Better test.

* Update runtime/parachains/src/disputes.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* chore fmt

* simplify the sorting for two items

* add test for assure duplicates are detected

* fixup tests

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
2022-01-27 18:16:31 +00:00
Robert Klotzner 8ec793e67f Bump substrate. (#4747) 2022-01-19 14:03:22 +01:00
Robert Klotzner fc594e5254 Report the actual missing session. (#4722) 2022-01-15 10:41:24 +00:00
Robert Klotzner f2bdd99532 Add some docs to prevent a time loop. (#4702)
* Add some docs to prevent a time loop.

* Review remarks.
2022-01-13 08:15:13 +00:00
Robert Klotzner 9514caee88 Small mostly doc improvements (#4661)
* Better docs on non initialized state.

* Document better what is happening.

* More precise errors.

* cargo fmt
2022-01-10 12:28:20 +01:00
Robert Klotzner 2f1643b430 Better error for spurious cache misses. (#4669)
Happened on Kusama for some not yet known reason.
2022-01-07 10:23:20 +00:00
Robert Klotzner 223d929f86 Warn on chain selection error. (#10587)
At least for Polkadot, this should not result in spam and a warning will
very likely indicate some serious issue.
2022-01-05 16:02:32 +00:00
Robert Klotzner 83c8afcbb9 Don't change rep on timeout in collator protocol. (#4642)
* Don't change rep on timeout in collator protocol.

* Fix tests.

* Fixes.
2022-01-04 11:40:20 +00:00
Robert Klotzner 0349595894 Better logs and metrics on PoV fetching. (#4593) 2021-12-24 08:11:29 +00:00
Robert Klotzner 846828f61c First step in implementing #4386 (#4437)
* First step in implementing https://github.com/paritytech/polkadot/issues/4386

This PR:

- Reduces MAX_UNSHARED_UPLOAD_TIME to 150ms
- Increases timeout on collation fetching to 1200ms
- Reduces limit on needed backing votes in the runtime

This PR does not yet reduce the number of needed backing votes on the
node as this can only be meaningfully enacted once the changed limit in
the runtime is live.

* Fix tests.

* Guide updates.

* Review remarks.

* Bump minimum required backing votes to 2 in runtime.

* Make sure node side code won't make runtime vomit.

* cargo +nightly fmt
2021-12-23 16:41:34 +00:00
f88d980252 Versi chainspec based on Rococo. (#4485)
* Versi chainspec based on Rococo.

* Fix.

* Fix compilation without features.

* Spelling

* Another typo.

* Support versi in native version

* 🤦

* Regenerated genesis for versi.

* Update cli/src/command.rs

Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-12-16 10:47:06 +00:00
Robert Klotzner bd5721fbf5 Revert loop prevention (#4472)
* Provisioner: Only include and sign bitfields on fresh leaves.
2021-12-13 12:20:49 +01:00
Robert Klotzner 34339c6805 Don't cache unavailable results. (#4509) 2021-12-10 23:52:20 +01:00
Robert Klotzner 0cafdd4d5e Some fixes the compiler pointed out to me (#4482)
* Fix bug in availability-distribution.

Old fetches would not get cancelled as intended.

* Fix a few more warnings.
2021-12-07 17:48:20 +01:00
Robert Klotzner ce27250cb5 Double time we give spawned tasks. (#4415)
* Double time we give spawned tasks.

* Increase time even more, for better robustness.
2021-11-30 16:44:58 +01:00
Robert KlotznerandAndronik Ordian 72a373411a Remove dead dep. (#765)
* Remove dead dep.

* cargo update -p polkadot-primitives

* Revert "cargo update -p polkadot-primitives"

This reverts commit 4e48e1dabe44fc1fb99c0d074281998e82584ba8.

* cargo update -p polkadot-primitives

Co-authored-by: Andronik Ordian <write@reusable.software>
2021-11-19 17:41:11 +00:00
Robert KlotznerandLldenaurois 25974f2076 Dispute spam protection (#4134)
* Mostly notes.

* Better error messages.

* Introduce Fatal/NonFatal + drop back channel participation

- Fatal/NonFatal - in order to make it easier to use utility functions.
- We drop the back channel in dispute participation as it won't be
needed any more.

* Better error messages.

* Utility function for receiving `CandidateEvent`s.

* Ordering module typechecks.

* cargo fmt

* Prepare spam slots module.

* Implement SpamSlots mechanism.

* Implement queues.

* cargo fmt

* Participation.

* Participation taking shape.

* Finish participation.

* cargo fmt

* Cleanup.

* WIP: Cleanup + Integration.

* Make `RollingSessionWindow` initialized by default.

* Make approval voting typecheck.

* Get rid of lazy_static & fix approval voting tests

* Move `SessionWindowSize` to node primitives.

* Implement dispute coordinator initialization.

* cargo fmt

* Make queues return error instead of boolean.

* Initialized: WIP

* Introduce chain api for getting finalized block.

* Fix ordering to only prune candidates on finalized events.

* Pruning of old sessions in spam slots.

* New import logic.

* Make everything typecheck.

* Fix warnings.

* Get rid of obsolete dispute-participation.

* Fixes.

* Add back accidentelly deleted Cargo.lock

* Deliver disputes in an ordered fashion.

* Add module docs for errors

* Use type synonym.

* hidden docs.

* Fix overseer tests.

* Ordering provider taking `CandidateReceipt`.

... To be kicked on one next commit.

* Fix ordering to use relay_parent

as included block is not unique per candidate.

* Add comment in ordering.rs.

* Take care of duplicate entries in queues.

* Better spam slots.

* Review remarks + docs.

* Fix db tests.

* Participation tests.

* Also scrape votes on first leaf for good measure.

* Make tests typecheck.

* Spelling.

* Only participate in actual disputes, not on every import.

* Don't account backing votes to spam slots.

* Fix more tests.

* Don't participate if we don't have keys.

* Fix tests, typos and warnings.

* Fix merge error.

* Spelling fixes.

* Add missing docs.

* Queue tests.

* More tests.

* Add metrics + don't short circuit import.

* Basic test for ordering provider.

* Import fix.

* Remove dead link.

* One more dead link.

Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com>
2021-11-19 17:08:21 +00:00
Robert Klotzner 6950ea66be Increase maximum chunk size to adjust for small networks. (#4220)
* Increase maximum chunk size to adjust for small networks.

* Issue warning on invalid merkle proofs.

* warn -> debug on invalid merkle proof.
2021-11-08 18:00:40 -06:00
Robert KlotznerandAndronik Ordian 6fde727c93 Fix misleading logs in collator protocol. (#4201)
* Fix misleading logs in collator protocol.

* Make logs debug again.

Co-authored-by: Andronik Ordian <write@reusable.software>
2021-11-04 12:34:17 +01:00
Robert Klotzner 5f5358279f Fix wrong warning. (#4163) 2021-10-27 21:36:41 +00:00
8e5b4e3826 Slightly improved documentation. (#9976)
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Giles Cope <gilescope@gmail.com>
2021-10-19 18:52:50 +01:00
Robert Klotzner 44824ae472 Expose set_reserved_peers (#9960) 2021-10-07 14:03:47 +00:00
Robert Klotzner 57a99c960f Silence some alerts due to overly verbose warnings. (#3946)
* statement-distribution: Only warn on relevant stuff.

* Silence warnings in availability-distribution.

* Demote more warnings.

* More consistency.

* info -> debug
2021-10-06 17:00:51 +00:00
Robert Klotzner aa311c3734 Silence alert about large-statement-fetcher (#9882)
* Silence alert about large-statement-fetcher

That task is not continuous and behaving exactly as desired.

* Get rid of rule completely

- Pierre thinks it does not add any value
- Basti rightly complains that Polkadot specific exceptions make little
  sense in the Substrate repo.
2021-09-30 16:42:08 +02:00
Robert Klotzner 7c3b3c4a59 Log info about low connectivity and unreachable validators (#3916)
* Attempt to add log stats to gossip-support.

* WIP: Keep track of connected validators.

* Clarify metric.

* WIP: Make gossip support report connectivity.

* WIP: Fixing tests.

* Fix network bridge + integrate in overseer.

* Consistent naming.

* Fix logic error

* cargo fmt

* Pretty logs.

* cargo fmt

* Use `Delay` to trigger periodic checks.

* fmt

* Fix warning for authority set size of 1.

* More correct ratio report if there are no resolved validators.

* Prettier rendering of empty set.

* Fix typo.

* Another typo.

* Don't check on every leaf update.

* Make compatible with older rustc.

* Fix tests.

* Demote warning.
2021-09-27 16:30:02 +02:00
Robert Klotzner 63a520b056 Dispute distribution improvements (#3853)
* Better warning policy on dispute-distribution.

Network errors happen, only warn on a minimum number of failed attempts.

* Add metric to see how long it takes to get a dispute vote out.

* Let's be alert.
2021-09-17 12:10:50 +02:00
Robert Klotzner dde0918717 Raised nits on #3813 (#3844) 2021-09-16 16:05:41 +02:00
Robert KlotznerandAndronik Ordian 973183cfb1 Don't try to connect to ourselves. (#3855)
* Don't try to connecto to ourselves.

* Use swap_remove instead of remove O(1)

Co-authored-by: Andronik Ordian <write@reusable.software>

Co-authored-by: Andronik Ordian <write@reusable.software>
2021-09-16 16:05:07 +02:00
Robert Klotzner de637b966a Better error messages. (#3835)
* Better error messages.

Right now, logged messages are hardly useful.

Together with https://github.com/paritytech/substrate/pull/9768, this
should help with debugging.

* cargo fmt
2021-09-14 14:18:32 +02:00
Robert Klotzner 135d9ae931 Put all authorities of a session into SessionInfo. (#3813)
* Make SessionInfo include all authorities

of the current session.

* Add missing import.

* Only take subset for assignment keys.

* Fix typo.

* Make spellcheck happy.

* Really.

* Fix test.

* More clear documentation.

* Update comments in `SessionInfo`.
2021-09-14 13:58:37 +02:00
Robert Klotzner 2c36adf898 More descriptive errors. (#9768) 2021-09-13 12:02:00 +00:00
Robert Klotzner a14b667723 Fix flaky availability-recovery test (#3812)
* Increase timeout in tests.

Fixes #3798

* Fix timeout.
2021-09-08 12:56:45 +00:00
Robert Klotzner ffcde1e5e7 Fixes/improvements for disputes (#3753)
* More debugging output.

* Fix chain selection in case of disputes.

* Fix flaky test.
2021-09-01 14:25:56 -05:00
e56efb82d9 Further improved availability recovery (#3711)
* WiP.

* Things compile.

* cargo fmt

* Passing tests + fix warnings.

* Metrics for availability recovery.

* Basic test.

* Fix typos and actually check for overflow.

* cargo fmt

* Register metrics.

* More tests.

* Fix warning.

* cargo +nightly fmt

* Fix metrics

* Get rid of unsafe.

* tabify

* spellcheck

Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Bastian Köcher <info@kchr.de>
2021-08-27 18:59:23 +02:00
Robert Klotzner 85897b5d36 Don't err on deactivated leaf during valiation. (#3708)
* Don't err on deactivated leaf during valiation.

* Fix error level + formatting of unrelated code.
2021-08-25 09:25:45 +02:00
Robert Klotzner 489a8e6da1 Fill up requests slots via launch_parallel_requests (#3681)
in case waiting for the next response takes too long.
2021-08-24 15:05:25 +02:00
Robert Klotzner d6abe70c06 Better logs. (#3650) 2021-08-19 20:07:59 +02:00
Robert Klotzner 1dafa08934 Provide dummy dispute coordinator by default. (#3661)
* Provide dummy dispute coordinator by default.

* cargo fmt
2021-08-19 17:03:33 +02:00
Robert Klotzner 55154a8d37 Remove request multiplexer (#3624)
* WIP: Get rid of request multiplexer.

* WIP

* Receiver for handling of incoming requests.

* Get rid of useless `Fault` abstraction.

The things the type system let us do are not worth getting abstracted in
its own type. Instead error handling is going to be merely a pattern.

* Make most things compile again.

* Port availability distribution away from request multiplexer.

* Formatting.

* Port dispute distribution over.

* Fixup statement distribution.

* Handle request directly in collator protocol.

+ Only allow fatal errors at top level.

* Use direct request channel for availability recovery.

* Finally get rid of request multiplexer

Fixes #2842 and paves the way for more back pressure possibilities.

* Fix overseer and statement distribution tests.

* Fix collator protocol and network bridge tests.

* Fix tests in availability recovery.

* Fix availability distribution tests.

* Fix dispute distribution tests.

* Add missing dependency

* Typos.

* Review remarks.

* More remarks.
2021-08-12 13:11:36 +02:00
Robert Klotzner d86bb658a0 Follow up for #3282 (#3571)
Incorporating post merge review remarks.
2021-08-06 18:18:13 +02:00
Robert KlotznerandAndronik Ordian e0b3c5cba0 Better size limits. (#3572)
* Better size limits.

* Remove reference to private repo.

* Update node/network/protocol/src/request_response/mod.rs

Fix statement response size.

Co-authored-by: Andronik Ordian <write@reusable.software>

Co-authored-by: Andronik Ordian <write@reusable.software>
2021-08-04 17:00:56 +02:00
Robert Klotzner 1718ed1dcc Fix flaky test (#3569)
Fixes #3557
2021-08-04 14:15:44 +02:00
Robert KlotznerandAndronik Ordian 6994682a86 DoS protection on the collator protocol (#3446)
* Move on to next validator after timeout.

* Better naming.

* Wrong implementation of validator fetch timeouts.

* Validator side: Move on to next collator

if download takes too long.

* Drop multiple requests from same validator.

* Add test that next response is sent after timeout.

* Multiple requests by same validator should get dropped.

* Test that another collator is tried

after exclusive download time.

* Add dep.

* Cleanup.

* Merge fix.

* Review remarks.

* Fixes.

* Add log targets to trace logs

Co-authored-by: Andronik Ordian <write@reusable.software>
2021-07-12 13:01:17 +02:00
Robert Klotzner b5257b2407 Dispute distribution implementation (#3282)
* Dispute protocol.

* Dispute distribution protocol.

* Get network requests routed.

* WIP: Basic dispute sender logic.

* Basic validator determination logic.

* WIP: Getting things to typecheck.

* Slightly larger timeout.

* More typechecking stuff.

* Cleanup.

* Finished most of the sending logic.

* Handle active leaves updates

- Cleanup dead disputes
- Update sends for new sessions
- Retry on errors

* Pass sessions in already.

* Startup dispute sending.

* Provide incoming decoding facilities

and use them in statement-distribution.

* Relaxed runtime util requirements.

We only need a `SubsystemSender` not a full `SubsystemContext`.

* Better usability of incoming requests.

Make it possible to consume stuff without clones.

* Add basic receiver functionality.

* Cleanup + fixes for sender.

* One more sender fix.

* Start receiver.

* Make sure to send responses back.

* WIP: Exposed authority discovery

* Make tests pass.

* Fully featured receiver.

* Decrease cost of `NotAValidator`.

* Make `RuntimeInfo` LRU cache size configurable.

* Cache more sessions.

* Fix collator protocol.

* Disable metrics for now.

* Make dispute-distribution a proper subsystem.

* Fix naming.

* Code style fixes.

* Factored out 4x copied mock function.

* WIP: Tests.

* Whitespace cleanup.

* Accessor functions.

* More testing.

* More Debug instances.

* Fix busy loop.

* Working tests.

* More tests.

* Cleanup.

* Fix build.

* Basic receiving test.

* Non validator message gets dropped.

* More receiving tests.

* Test nested and subsequent imports.

* Fix spaces.

* Better formatted imports.

* Import cleanup.

* Metrics.

* Message -> MuxedMessage

* Message -> MuxedMessage

* More review remarks.

* Add missing metrics.rs.

* Fix flaky test.

* Dispute coordinator - deliver confirmations.

* Send out `DisputeMessage` on issue local statement.

* Unwire dispute distribution.

* Review remarks.

* Review remarks.

* Better docs.
2021-07-09 04:29:53 +02:00
Robert KlotznerandSergei Shulepov 20993b32b1 Guide updates for disputes. (#3401)
* Guide updates for disputes.

* Working availability recovery flood protection.

* More fixes.

* Formatting.

* Fix.

* Update roadmap/implementers-guide/src/node/disputes/dispute-participation.md

Co-authored-by: Sergei Shulepov <sergei@parity.io>

* Review remarks.

Co-authored-by: Sergei Shulepov <sergei@parity.io>
2021-07-08 23:19:08 +02:00