Commit Graph

21 Commits

Author SHA1 Message Date
s0me0ne-unkn0wn 1cb1d03c08 Re-export current primitives in crate root (#6487)
* Re-export current primitives in crate root

* Add missing exports

* restart CI
2023-01-11 11:28:12 +00:00
Andrei Sandu eb00a21ef7 dispute-coordinator: fix earliest session checks for pruning and import (#6358)
* RollingSession: add fn contains

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* handle_import_statements fix ancient dispute check

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* work with earliest session instead of latest

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* update comment

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
2022-11-28 17:43:13 +01:00
Andrei Sandu c261353380 Make rolling session more resilient in case of long finality stalls (#6106)
* Impl dynamic window size. Keep sessions for unfinalized chain

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* feedback

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Stretch also in contructor plus  tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* review feedback

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix approval-voting tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* grunting: dispute coordinator tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* add session window column

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* integrate approval vote and fix tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix rolling session tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Small refactor

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* WIP, tests failing

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix approval voting tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix dispute-coordinator tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* remove uneeded param

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fmt

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix loose ends

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* allow failure and tests for it

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix comment

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* comment fix

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* style fix

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* new col doesn't need to be ordered

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fmt and spellcheck

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* db persist tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Add v2 config and cols

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* DB upgrade WIP

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix comments

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* add todo

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* update to parity-db to "0.4.2"

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* migration complete

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* One session window size

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix merge damage

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix build errors

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fmt

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* comment fix

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix build

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* make error more explicit

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* add comment

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* refactor conflict merge

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* rename col_data

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* add doc comment

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix build

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* migration: move all cols to v2

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
2022-11-08 18:16:31 +01:00
Boluwatife Bakre 8eb1f4617f Use a more typesafe approach for managing indexed data (#6150)
* Fix for issue #2403

* Nightly fmt

* Quick documentation fixes

* Default Implementation

* iter() function integrated

* Implemented iter functionalities

* Fmt

* small change

* updates node-network

* updates in dispute-coordinator

* Updates

* benchmarking fix

* minor fix

* test fixes in runtime api

* Update primitives/src/v2/mod.rs

Co-authored-by: Andronik <write@reusable.software>

* Update primitives/src/v2/mod.rs

Co-authored-by: Andronik <write@reusable.software>

* Update primitives/src/v2/mod.rs

Co-authored-by: Andronik <write@reusable.software>

* Update primitives/src/v2/mod.rs

Co-authored-by: Andronik <write@reusable.software>

* Update primitives/src/v2/mod.rs

Co-authored-by: Andronik <write@reusable.software>

* Removal of [index], shorting of FromIterator, Renaming of GroupValidators to ValidatorGroups

* Removal of ops import

* documentation fixes for spell check

* implementation of generic type

* Refactoring

* Test and documentation fixes

* minor test fix

* minor test fix

* minor test fix

* Update node/network/statement-distribution/src/lib.rs

Co-authored-by: Andronik <write@reusable.software>

* Update primitives/src/v2/mod.rs

Co-authored-by: Andronik <write@reusable.software>

* Update primitives/src/v2/mod.rs

Co-authored-by: Andronik <write@reusable.software>

* removed IterMut

* Update node/core/dispute-coordinator/src/import.rs

Co-authored-by: Andronik <write@reusable.software>

* Update node/core/dispute-coordinator/src/initialized.rs

Co-authored-by: Andronik <write@reusable.software>

* Update primitives/src/v2/mod.rs

Co-authored-by: Andronik <write@reusable.software>

* fmt

* IterMut

* documentation update

Co-authored-by: Andronik <write@reusable.software>

* minor adjustments and new TypeIndex trait

* spelling fix

* TypeIndex fix

Co-authored-by: Andronik <write@reusable.software>
2022-10-22 08:39:11 +00:00
Andrei Sandu 7114a8cfca Keep sessions in window for the full unfinalized chain (#6054)
* Impl dynamic window size. Keep sessions for unfinalized chain

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* feedback

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Stretch also in contructor plus  tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* review feedback

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix approval-voting tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* grunting: dispute coordinator tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
2022-10-04 13:36:42 +03:00
Bernhard Schuster 3240cb5e4d split NetworkBridge into two subsystems (#5616)
* foo

* rolling session window

* fixup

* remove use statemetn

* fmt

* split NetworkBridge into two subsystems

Pending cleanup

* split

* chore: reexport OrchestraError as OverseerError

* chore: silence warnings

* fixup tests

* chore: add default timenout of 30s to subsystem test helper ctx handle

* single item channel

* fixins

* fmt

* cleanup

* remove dead code

* remove sync bounds again

* wire up shared state

* deal with some FIXMEs

* use distinct tags

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

* use tag

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

* address naming

tx and rx are common in networking and also have an implicit meaning regarding networking
compared to incoming and outgoing which are already used with subsystems themselvesq

* remove unused sync oracle

* remove unneeded state

* fix tests

* chore: fmt

* do not try to register twice

* leak Metrics type

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
Co-authored-by: Andronik <write@reusable.software>
2022-07-12 16:22:36 +00:00
Bernhard Schuster 511891dcce refactor+feat: allow subsystems to send only declared messages, generate graphviz (#5314)
Closes #3774
Closes #3826
2022-05-12 17:39:05 +02:00
Robert Habermeier 49f7e5cce4 Finish migration to v2 primitives (#5037)
* remove v0 primitives from polkadot-primitives

* first pass: remove v0

* fix fallout in erasure-coding

* remove v1 primitives, consolidate to v2

* the great import update

* update runtime_api_impl_v1 to v2 as well

* guide: add `Version` request for runtime API

* add version query to runtime API

* reintroduce OldV1SessionInfo in a limited way
2022-03-09 14:01:13 -06:00
sandreim 806cb541b7 Refactor and fix usage of get_session_index() and get_session_info_by_index() (#4735)
* Rename/refactor around get_session_index

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* choose proper head for fetching session

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* revert rename

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix comments

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* renaming and more comments

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* review feedback

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Run Fetch task in correct session

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Log warning when ancestors unavailable

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fixes

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
2022-01-26 15:17:46 +00:00
sandreim 2661930b77 relay chain selection and dispute-coordinator fixes and improvements (#4752)
* Dont error in finality_target_with_longest_chain

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Add error flag

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Add error flag in dispute-coordinator

Make sure to send errors to subsystems requesting data depending on missing session info

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Scrape ancestors

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fmt

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix naming

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* review feedback

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fmt

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* 💬 fixes

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* consume

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* typo

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* review fixes

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Bump scraped blocks LRU capacity

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* 🧯 🔥

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* remove prints

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Increase scraped blocks cache size

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* more review fixes

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* another fix

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix target_ancestor

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Scrape up to max finalized block

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* undo comment change

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Limit ancestry lookup to last finalized block or
max finality lag

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* debug damage

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
2022-01-26 14:06:27 +00:00
Robert Klotzner fc594e5254 Report the actual missing session. (#4722) 2022-01-15 10:41:24 +00:00
Andronik Ordian 9efb223a97 session_info: add dispute_period and random_seed (#4547) 2021-12-27 09:03:50 +00:00
Andronik Ordian b342ae11d3 session-info: add new fields + migration (#4545)
* session_info: v2 + migration

* use primitives::v2

* use polkadot_primitives::v2

* impl primitives::v2

* fix approval-voting tests

* fix other tests

* hook storage migration up

* backwards compat (1)

* backwards compat (2)

* fmt

* fix tests

* FMT

* do not reexport v1 in v2

* fmt

* set storage version to 1

Co-authored-by: Javier Viola <javier@parity.io>
2021-12-27 08:01:30 +00:00
Robert Klotzner 25974f2076 Dispute spam protection (#4134)
* Mostly notes.

* Better error messages.

* Introduce Fatal/NonFatal + drop back channel participation

- Fatal/NonFatal - in order to make it easier to use utility functions.
- We drop the back channel in dispute participation as it won't be
needed any more.

* Better error messages.

* Utility function for receiving `CandidateEvent`s.

* Ordering module typechecks.

* cargo fmt

* Prepare spam slots module.

* Implement SpamSlots mechanism.

* Implement queues.

* cargo fmt

* Participation.

* Participation taking shape.

* Finish participation.

* cargo fmt

* Cleanup.

* WIP: Cleanup + Integration.

* Make `RollingSessionWindow` initialized by default.

* Make approval voting typecheck.

* Get rid of lazy_static & fix approval voting tests

* Move `SessionWindowSize` to node primitives.

* Implement dispute coordinator initialization.

* cargo fmt

* Make queues return error instead of boolean.

* Initialized: WIP

* Introduce chain api for getting finalized block.

* Fix ordering to only prune candidates on finalized events.

* Pruning of old sessions in spam slots.

* New import logic.

* Make everything typecheck.

* Fix warnings.

* Get rid of obsolete dispute-participation.

* Fixes.

* Add back accidentelly deleted Cargo.lock

* Deliver disputes in an ordered fashion.

* Add module docs for errors

* Use type synonym.

* hidden docs.

* Fix overseer tests.

* Ordering provider taking `CandidateReceipt`.

... To be kicked on one next commit.

* Fix ordering to use relay_parent

as included block is not unique per candidate.

* Add comment in ordering.rs.

* Take care of duplicate entries in queues.

* Better spam slots.

* Review remarks + docs.

* Fix db tests.

* Participation tests.

* Also scrape votes on first leaf for good measure.

* Make tests typecheck.

* Spelling.

* Only participate in actual disputes, not on every import.

* Don't account backing votes to spam slots.

* Fix more tests.

* Don't participate if we don't have keys.

* Fix tests, typos and warnings.

* Fix merge error.

* Spelling fixes.

* Add missing docs.

* Queue tests.

* More tests.

* Add metrics + don't short circuit import.

* Basic test for ordering provider.

* Import fix.

* Remove dead link.

* One more dead link.

Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com>
2021-11-19 17:08:21 +00:00
Andronik Ordian f66440dc19 revert +1 change to be on the safer side (#3972) 2021-09-29 19:24:12 +00:00
Andronik Ordian cb89dfc039 approval-voting: populate session cache in advance (#3954)
* try populating session cache in advance

* remove unused arg

* fmt

* fix compilation

* fix tests

* Revert "fix tests"

This reverts commit e8222b1108e09a39727a38e3b4e4c3061642a213.

* fix tests

* bump dispute window const by 1

* fix tests
2021-09-29 09:53:44 +00:00
Sergei Shulepov 68c03f66f3 Mass replace ,); pattern (#3580)
This is an artifact left by rustfmt which is not dare to remove the
comma being conservative.
2021-08-05 19:53:17 +02:00
Shawn Tabrizi ff5d56fb76 cargo +nightly fmt (#3540)
* cargo +nightly fmt

* add cargo-fmt check to ci

* update ci

* fmt

* fmt

* skip macro

* ignore bridges
2021-08-02 10:47:33 +00:00
Bernhard Schuster 3c9104daff refactor overseer into proc-macro based pattern (#2962) 2021-07-08 21:09:26 +02:00
Robert Habermeier 4797fb7dfb use fresh_start in rolling window error (#3229) 2021-06-13 14:26:36 +00:00
Robert Habermeier 5bc2b2779d Dispute Coordinator Subsystem (#3150)
* skeleton for dispute-coordinator

* add coordinator and participation message types

* begin dispute-coordinator DB

* functions for loading

* implement strongly-typed DB transaction

* add some tests for DB transaction

* core logic for pruning

* guide: update candidate-votes key for coordinator

* update candidate-votes key

* use big-endian encoding for session, and implement upper bound generator

* finish implementing pruning

* add a test for note_current_session

* define state of the subsystem itself

* barebones subsystem definition

* control flow

* more control flow

* implement session-updating logic

* trace

* control flow for message handling

* Update node/core/dispute-coordinator/src/lib.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update node/subsystem/src/messages.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* some more control flow

* guide: remove overlay

* more control flow

* implement some DB getters

* make progress on importing statements

* add SignedDisputeStatement struct

* move ApprovalVote to shared primitives

* add a signing-payload API to explicit dispute statements

* add signing-payload to CompactStatement

* add relay-parent hash to seconded/valid dispute variatns

* correct import

* type-safe wrapper around dispute statements

* use checked dispute statement in message type

* extract rolling session window cache to subsystem-util

* extract session window tests

* approval-voting: use rolling session info cache

* reduce dispute window to match runtime in practice

* add byzantine_threshold and supermajority_threshold utilities to primitives

* integrate rolling session window

* Add PartialOrd to CandidateHash

* add Ord to CandidateHash

* implement active dispute update

* add dispute messages to AllMessages

* add dispute stubs to overseer

* inform dispute participation to participate

* implement issue_local_statement

* implement `determine_undisputed_chain`

* fix warnings

* test harness for dispute coordinator tests

* add more helpers to test harness

* add some more helpers

* some tests for dispute coordinator

* ignore wrong validator indices

* test finality voting rule constraint

* add more tests

* add variants to network bridge

* fix test compilation

* remove most dispute coordinator functionality

as of #3222 we can do most of the work within the approval voting subsystem

* Revert "remove most dispute coordinator functionality"

This reverts commit 9cd615e8eb6ca0b382cbaff525d813e753d6004e.

* Use thiserror

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* Update node/core/dispute-coordinator/src/lib.rs

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* extract tests to separate module

* address nit

* adjust run_iteration API

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
2021-06-13 06:35:18 -05:00