mirror of
https://github.com/pezkuwichain/pezkuwi-subxt.git
synced 2026-11-02 08:51:08 +00:00
f0d443a055ee7a6f6e0b0d4cfea2f98fe98d0864
70
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
+22 Peter Goodspeed-NiklausAndronik Ordianparity-processbotShawn Tabriziradupopa2010CI systemdependabot[bot]André SilvathiolliereAndré SilvaKeith YeungBernhard SchusterKian PaimaniRobert HabermeierRobert KlotznerAlexander TheißenLldenauroisQinxuan ChenSeun LanlegeDenis PisarevAndreas DoerrChevdorTomasz DrwięgaDenis TsaiPierre BessonkianenigmaBastian KöcherXiliang ChenHernando Castanoferrell-codeShaun WZeke Mostov |
9a6f43ab8f |
Companion for Store voters in unsorted bags to get good stake iteration properties (#3413)
* add voter bags generation script and generated values for all runtimes * VoterBagThresholds for test-runtime * semicolon * turns out the header is called file_header.txt * perform benchmarks to get real numbers * add note about when to re-run this script * Suggestions for #3413 (Store voters in unsorted bags companion) (#3488) Co-authored-by: Andronik Ordian <write@reusable.software> Co-authored-by: parity-processbot <> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: radupopa2010 <radupopa2010@yahoo.com> Co-authored-by: CI system <> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> Co-authored-by: thiolliere <gui.thiolliere@gmail.com> Co-authored-by: André Silva <andrerfosilva@gmail.com> Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> Co-authored-by: Bernhard Schuster <bernhard@ahoi.io> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com> Co-authored-by: Alexander Theißen <alex.theissen@me.com> Co-authored-by: Lldenaurois <ljdenaurois@gmail.com> Co-authored-by: Qinxuan Chen <koushiro.cqx@gmail.com> Co-authored-by: Seun Lanlege <seun@parity.io> Co-authored-by: Denis Pisarev <denis.pisarev@parity.io> Co-authored-by: Andreas Doerr <adoerr@users.noreply.github.com> Co-authored-by: Chevdor <chevdor@users.noreply.github.com> Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com> Co-authored-by: Denis Tsai <denistsai@aetheras.io> Co-authored-by: Pierre Besson <pierre.besson@parity.io> Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Xiliang Chen <xlchen1291@gmail.com> Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com> Co-authored-by: ferrell-code <automatedcharlesferrell@gmail.com> Co-authored-by: Shaun W <spxwang@gmail.com> * Try resolve Cargo.lock conflicts * Add temp VoterSnapshotPerBlock value to compile * Improve voter_bags test with nominator/validator breakdown * Remove miner.log * Update README * Module doc comments * Build working * remote-ext-tests very WIP * Remove system module query * formatting and log target * clean up * Save * Fix issues with polkadot runtime after mergin * Save * Some updates to runtimes to make sure things compile * save * udpate weights * Use UseNominatorsAndUpdateBagsList * make remote-ext tests a bin * Adjust remote-ext test to use pub; update some comments * Use list_bags_get * Try fix cargo.lock * Gate kusama feat code * Carg.lock .. again * point bags-list deps to master * Update voter-bags CLI to take total issuance and min balance as args * Switch polkadot runtime to UseNominatorMap * Some tweaks to the remote-test * Delete some stuff for polkadot; comments * Various fixes * Add comment for VoterSnapshotPerBlock * Remove bags-list import from polakdot cargo * use benchmarking feature for pallet-bags-list * Add custom migration for westend and kusama for bags-list * :facepalm * Fix kusama runtime * Small clean up * Be more alphabetical ordered * Use 22_500 for VoterSnapshotPerBlock * point back to master * remove unused type * SortedListProvider for staking runtime * Remove polkadot for remote-ext bags test * Revert "Remove polkadot for remote-ext bags test" This reverts commit 3608c466002063761cd669880a00b2e05c1ff7d1. * Remove polkadot for remote-ext bags test * update Substrate Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com> Co-authored-by: Andronik Ordian <write@reusable.software> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: radupopa2010 <radupopa2010@yahoo.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> Co-authored-by: thiolliere <gui.thiolliere@gmail.com> Co-authored-by: André Silva <andrerfosilva@gmail.com> Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> Co-authored-by: Bernhard Schuster <bernhard@ahoi.io> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com> Co-authored-by: Alexander Theißen <alex.theissen@me.com> Co-authored-by: Lldenaurois <ljdenaurois@gmail.com> Co-authored-by: Qinxuan Chen <koushiro.cqx@gmail.com> Co-authored-by: Seun Lanlege <seun@parity.io> Co-authored-by: Denis Pisarev <denis.pisarev@parity.io> Co-authored-by: Andreas Doerr <adoerr@users.noreply.github.com> Co-authored-by: Chevdor <chevdor@users.noreply.github.com> Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com> Co-authored-by: Denis Tsai <denistsai@aetheras.io> Co-authored-by: Pierre Besson <pierre.besson@parity.io> Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Xiliang Chen <xlchen1291@gmail.com> Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com> Co-authored-by: ferrell-code <automatedcharlesferrell@gmail.com> Co-authored-by: Shaun W <spxwang@gmail.com> Co-authored-by: parity-processbot <> |
||
|
Peter Goodspeed-Niklaus
|
9fc86cb55f |
make submit_unsigned into DispatchClass::Operational (#9309)
Closes #8511. |
||
|
Peter Goodspeed-Niklausandparity-processbot
|
95736d1bf1 |
Companion for Decouple Staking and Election - Part 3: Signed Phase (#2793)
* Companion for Decouple Staking and Election - Part 3: Signed Phase https://github.com/paritytech/substrate/pull/7910 * remove some config types * allow up to 5 signed submissions on polkadot and kusama * signed phase is equal induration to unsigned phase * use chain defaults for base and per-byte deposits; >= 16 SignedMaxSubmissions * use a small but non-trivial solution reward * reduce signed deposit per byte fee * reduce signed reward, adjust polkadot expected soln size * copy submit benchmark from substrate * demo calculating an appropriate fee for the signed reward Unfortunately, this doesn't work: it needs to be a constant function, and AFAIK there's no way to make a trait method constant. * SignedRewardBase is 1.5x the fee to submit a signed solution * all chains use deposit byte of base per 50k * update Substrate * cargo update -p pallet-election-provider-multi-phase Co-authored-by: parity-processbot <> |
||
|
Peter Goodspeed-Niklaus
|
ee192467e2 | move set_emergency_election_result before submit (#9215) | ||
|
|
46ccf17e7c |
fix some failing ui tests (#9157)
* fix some failing ui tests * Update frame/support/test/tests/pallet_ui/call_argument_invalid_bound_2.stderr * Update frame/support/test/tests/pallet_ui/call_argument_invalid_bound_2.stderr * fix ui test * fix ui test * TRYBUILD=overwrite cargo test --workspace -- ui Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: thiolliere <gui.thiolliere@gmail.com> |
||
|
Peter Goodspeed-Niklaus
|
0975a92818 |
fix ordering of staking weight arguments (#9063)
Closes #9054. |
||
|
Peter Goodspeed-NiklausandKian Paimani
|
655ebc95fb |
Add BoundedBTreeSet (#8750)
* Add `BoundedBTreeSet` Part of https://github.com/paritytech/substrate/issues/8719 * fix copy-pasta errors Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> |
||
|
Peter Goodspeed-Niklaus
|
b689790171 |
Relax BoundedVec trait restrictions (#8749)
* requiring users to maintain an unchecked invariant is unsafe * relax trait restrictions on BoundedVec<T, S> A normal `Vec<T>` can do many things without any particular trait bounds on `T`. This commit relaxes the bounds on `BoundedVec<T, S>` to give it similar capabilities. |
||
|
Peter Goodspeed-Niklaus
|
17a1997d18 |
#[derive(MaxEncodedLen)] (#8737)
* impl #[derive(MaxEncodedLen)] for structs * impl #[derive(MaxEncodedLen)] for enums, unions * break long comments onto multiple lines * add doc for public item * add examples to macro documentation * move MaxEncodedLen macro docs, un-ignore doc-tests |
||
|
Peter Goodspeed-Niklaus
|
db69eb04bb |
Add BoundedBTreeMap to frame_support::storage (#8745)
* Add `BoundedBTreeMap` to `frame_support::storage` Part of https://github.com/paritytech/substrate/issues/8719. * max_encoded_len will never encode length > bound * requiring users to maintain an unchecked invariant is unsafe * only impl debug when std * add some marker traits * add tests |
||
|
Peter Goodspeed-Niklaus
|
a935274ef9 | BoundedVec MaxEncodedLen microoptimization (#8746) | ||
|
Peter Goodspeed-NiklausandBastian Köcher
|
98744eb3b2 |
implement BoundedEncodedLen (#8720)
* implement BoundedEncodedLen * update header * update imports * use impl_for_tuples instead of a custom macro * remove redundant where clause Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * impl for Compact<T> * impl BoundedEncodedLen for BoundedVec (#8727) * impl BoundedEncodedLen for bool * explicitly implement BoundedEncodedLen for each Compact form Turns out that u16 doesn't play nicely with the pattern; those values take two extra bytes, where all other cases take one. :( * rename BoundedEncodedLen -> MaxEncodedLen * add tests of compact encoded length Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> |
||
|
Peter Goodspeed-Niklausandparity-processbot
|
774d612eef |
Companion for Multi-phase elections solution resubmission (#2648)
* Companion for Multi-phase elections solution resubmission https://github.com/paritytech/substrate/pull/8290 * update Substrate Co-authored-by: parity-processbot <> |
||
|
|
de5d0b2312 |
Multi-phase elections solution resubmission (#8290)
* not climate * explain the intent of the bool in the unsigned phase * remove glob imports from unsigned.rs * add OffchainRepeat parameter to ElectionProviderMultiPhase * migrate core logic from #7976 This is a much smaller diff than that PR contained, but I think it contains all the essentials. * improve formatting * fix test build failures * cause test to pass * Apply suggestions from code review Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * collapse imports * threshold acquired directly within try_acquire_offchain_lock * add test of resubmission after interval * add test that ocw can regenerate a failed cache when resubmitting * ensure that OCW solutions are of the correct round This should help prevent stale cached solutions from persisting past the election for which they are intended. * add test of pre-dispatch round check * use `RawSolution.round` instead of redundantly externally * unpack imports Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * rename `OFFCHAIN_HEAD_DB` -> `OFFCHAIN_LOCK` * rename `mine_call` -> `mine_checked_call` * eliminate extraneous comma * check cached call is current before submitting * remove unused consts introduced by bad merge. Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * resubmit when our solution beats queued solution * clear call cache if solution fails to submit * use local storage; clear on ElectionFinalized * Revert "use local storage; clear on ElectionFinalized" This reverts commit 4b46a9388532d0c09b337dc7c7edf76044a6cee8. * BROKEN: try to filter local events in OCW * use local storage; simplify score fetching * fix event filter * mutate storage instead of setting it * StorageValueRef::local isn't actually implemented yet * add logging for some events of interest in OCW miner * rename kill_solution -> kill_ocw_solution to avoid ambiguity * defensive err instead of unreachable given unreachable code * doc punctuation Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * distinguish miner errors between "out of date" and "call invalid" * downgrade info logs -> debug * ensure encoded call decodes as a call * fix semantics of validation of pre-dispatch failure for wrong round * move score check within `and_then` * add test that offchain workers clear their cache after election * ensure that bad ocw submissions are not retained for resubmission * simplify fn ocw_solution_exists * add feasibility check when restoring cached solution should address https://github.com/paritytech/substrate/pull/8290/files#r617533358 restructures how the checks are sequenced, which simplifies legibility. * simplify checks again Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> |
||
|
Peter Goodspeed-Niklaus
|
c786fb21a0 |
Refactor election solution trimming for efficiency (#8614)
* Refactor election solution trimming for efficiency
The previous version always trimmed the `CompactOf<T>` instance,
which was intrinsically inefficient: that's a packed data structure,
which is naturally expensive to edit. It's much easier to edit
the unpacked data structures: the `voters` and `assignments` lists.
* rework length-trim tests to work with the new interface
Test suite now compiles. Tests still don't pass because the macro
generating the compact structure still generates `unimplemented!()`
for the actual `compact_length_of` implementation.
* simplify
* add a fuzzer which can validate `Compact::encoded_size_for`
The `Compact` solution type is generated distinctly for each runtime,
and has both three type parameters and a built-in limit to the number
of candidates that each voter can vote for. Finally, they have an
optional `#[compact]` attribute which changes the encoding behavior.
The assignment truncation algorithm we're using depends on the ability
to efficiently and accurately determine how much space a `Compact`
solution will take once encoded.
Together, these two facts imply that simple unit tests are not
sufficient to validate the behavior of `Compact::encoded_size_for`.
This commit adds such a fuzzer. It is designed such that it is possible
to add a new fuzzer to the family by simply adjusting the
`generate_solution_type` macro invocation as desired, and making a
few minor documentation edits.
Of course, the fuzzer still fails for now: the generated implementation
for `encoded_size_for` is still `unimplemented!()`. However, once
the macro is updated appropriately, this fuzzer family should allow
us to gain confidence in the correctness of the generated code.
* Revert "add a fuzzer which can validate `Compact::encoded_size_for`"
This reverts commit 916038790887e64217c6a46e9a6d281386762bfb.
The design of `Compact::encoded_size_for` is flawed. When `#[compact]`
mode is enabled, every integer in the dataset is encoded using run-
length encoding. This means that it is impossible to compute the final
length faster than actually encoding the data structure, because the
encoded length of every field varies with the actual value stored.
Given that we won't be adding that method to the trait, we won't be
needing a fuzzer to validate its performance.
* revert changes to `trait CompactSolution`
If `CompactSolution::encoded_size_for` can't be implemented in the
way that we wanted, there's no point in adding it.
* WIP: restructure trim_assignments_length by actually encoding
This is not as efficient as what we'd hoped for, but it should still
be better than what it's replacing. Overall efficiency of
`fn trim_assignments_length` is now `O(edges * lg assignments.len())`.
* fix compiler errors
* don't sort voters, just assignments
Sorting the `voters` list causes lots of problems; an invariant that
we need to maintain is that an index into the voters list has a stable
meaning.
Luckily, it turns out that there is no need for the assignments list
to correspond to the voters list. That isn't an invariant, though previously
I'd thought that it was.
This simplifies things; we can just leave the voters list alone,
and sort the assignments list the way that is convenient.
* WIP: add `IndexAssignment` type to speed up repeatedly creating `Compact`
Next up: `impl<'a, T> From<&'a [IndexAssignmentOf<T>]> for Compact`,
in the proc-macro which makes `Compact`. Should be a pretty straightforward
adaptation of `from_assignment`.
* Add IndexAssignment and conversion method to CompactSolution
This involves a bit of duplication of types from
`election-provider-multi-phase`; we'll clean those up shortly.
I'm not entirely happy that we had to add a `from_index_assignments`
method to `CompactSolution`, but we couldn't define
`trait CompactSolution: TryFrom<&'a [Self::IndexAssignment]` because
that made trait lookup recursive, and I didn't want to propagate
`CompactSolutionOf<T> + TryFrom<&[IndexAssignmentOf<T>]>` everywhere
that compact solutions are specified.
* use `CompactSolution::from_index_assignment` and clean up dead code
* get rid of `from_index_assignments` in favor of `TryFrom`
* cause `pallet-election-provider-multi-phase` tests to compile successfully
Mostly that's just updating the various test functions to keep track of
refactorings elsewhere, though in a few places we needed to refactor some
test-only helpers as well.
* fix infinite binary search loop
Turns out that moving `low` and `high` into an averager function is a
bad idea, because the averager gets copies of those values, which
of course are never updated. Can't use mutable references, because
we want to read them elsewhere in the code. Just compute the average
directly; life is better that way.
* fix a test failure
* fix the rest of test failures
* remove unguarded subtraction
* fix npos-elections tests compilation
* ensure we use sp_std::vec::Vec in assignments
* add IndexAssignmentOf to sp_npos_elections
* move miner types to `unsigned`
* use stable sort
* rewrap some long comments
* use existing cache instead of building a dedicated stake map
* generalize the TryFrom bound on CompactSolution
* undo adding sp-core dependency
* consume assignments to produce index_assignments
* Add a test of Assignment -> IndexAssignment -> Compact
* fix `IndexAssignmentOf` doc
* move compact test from sp-npos-elections-compact to sp-npos-elections
This means that we can put the mocking parts of that into a proper
mock package, put the test into a test package among other tests.
Having the mocking parts in a mock package enables us to create a
benchmark (which is treated as a separate crate) import them.
* rename assignments -> sorted_assignments
* sort after reducing to avoid potential re-sort issues
* add runtime benchmark, fix critical binary search error
"Why don't you add a benchmark?", he said. "It'll be good practice,
and can help demonstrate that this isn't blowing up the runtime."
He was absolutely right.
The biggest discovery is that adding a parametric benchmark means that
you get a bunch of new test cases, for free. This is excellent, because
those test cases uncovered a binary search bug. Fixing that simplified
that part of the code nicely.
The other nice thing you get from a parametric benchmark is data about
what each parameter does. In this case, `f` is the size factor: what
percent of the votes (by size) should be removed. 0 means that we should
keep everything, 95 means that we should trim down to 5% of original size
or less.
```
Median Slopes Analysis
========
-- Extrinsic Time --
Model:
Time ~= 3846
+ v 0.015
+ t 0
+ a 0.192
+ d 0
+ f 0
µs
Min Squares Analysis
========
-- Extrinsic Time --
Data points distribution:
v t a d f mean µs sigma µs %
<snip>
6000 1600 3000 800 0 4385 75.87 1.7%
6000 1600 3000 800 9 4089 46.28 1.1%
6000 1600 3000 800 18 3793 36.45 0.9%
6000 1600 3000 800 27 3365 41.13 1.2%
6000 1600 3000 800 36 3096 7.498 0.2%
6000 1600 3000 800 45 2774 17.96 0.6%
6000 1600 3000 800 54 2057 37.94 1.8%
6000 1600 3000 800 63 1885 2.515 0.1%
6000 1600 3000 800 72 1591 3.203 0.2%
6000 1600 3000 800 81 1219 25.72 2.1%
6000 1600 3000 800 90 859 5.295 0.6%
6000 1600 3000 800 95 684.6 2.969 0.4%
Quality and confidence:
param error
v 0.008
t 0.029
a 0.008
d 0.044
f 0.185
Model:
Time ~= 3957
+ v 0.009
+ t 0
+ a 0.185
+ d 0
+ f 0
µs
```
What's nice about this is the clear negative correlation between
amount removed and total time. The more we remove, the less total
time things take.
|
||
|
|
f822381c4c |
Companion for Trim compact solution for length during preparation (#2649)
* Companion for Trim compact solution for length during preparation https://github.com/paritytech/substrate/pull/8317 * eliminate potential for overflow in OffchainSolutionLengthLimit * Apply suggestions from code review Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * update substrate: cargo update -p sp-io Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> |
||
|
|
33425ce21f |
Trim compact solution for length during preparation (#8317)
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> |
||
|
Peter Goodspeed-Niklausandparity-processbot
|
27b6d83974 |
use named generic types in generate_solution_type (#2707)
* use named generic types in generate_solution_type * "Update Substrate" Co-authored-by: parity-processbot <> |
||
|
Peter Goodspeed-Niklaus
|
c2dd5e21a4 |
make types within generate_solution_type macro explicit (#8447)
* make types within `generate_solution_type` macro explicit Closes #8444. Just changes the parsing logic for that macro; does not change any emitted code. The associated types associated with the macro now require explicit, keyword-style declaration. **Old**: ```rust sp_npos_elections::generate_solution_type!( #[compact] pub struct TestCompact::<VoterIndex, TargetIndex, PerU16>(16) ); ``` **New**: ```rust sp_npos_elections::generate_solution_type!( #[compact] pub struct TestCompact::<VoterIndex = VoterIndex, CandidateIndex = TargetIndex, Accuracy = PerU16>(16) ); ``` * un-ignore doc-tests * use new form in bin/node/runtime/ * rename CandidateIndex -> TargetIndex * add tests demonstrating some potential compile failures |
||
|
Peter Goodspeed-NiklausandKian Paimani
|
05f24931a9 |
Fast CompactAssignment search (#8385)
* use more efficient search through candidates in offchain-election * mark linear accessors as test-only in election-provider-multi-phase This prevents production code which uses them from compiling. Also write an efficient helper for getting the target index. * doc grammar * use faster target_index_fn in benchmarks * unbox helper functions * remove unnecessary import * write lifetime after primary trait Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> |
||
|
Peter Goodspeed-Niklaus
|
23b32e7543 |
Improve complexity of CompactAssignments::unique_targets (#8314)
* Improve complexity of CompactAssignments::unique_targets Original implementation was O(n**2). Current impl is O(n log n). Avoided the original proposed mitigation because it does not retain the de-duplicating property present in the original implementation. This implementation does a little more work, but retains that property. * Explicitly choose sp_std Vec and BTreeSet Ensures that the macro still works if someone uses it in a context in which sp_std is not imported or is renamed. * explicitly use sp_std vectors throughout compact macro |
||
|
Peter Goodspeed-NiklausandKian Paimani
|
49be0579db |
NPoS Challenge Mode (#8236)
* Add PJR challenge functions - Updates the PJR check to return a counterexample if one exists - Adds functions to cheaply check counterexamples This is in support of off-chain PJR challenges: if a miner discovers that an accepted election solution does not satisfy PJR, it will be eligible for substantial rewards. This helps ensure that validator elections have an absolute quality floor, so even if someone manages to censor well-behaved solutions to give themselves unfair representation, we can catch them in the act and penalize them. * counterexample -> counter_example * reorganize: high -> low abstraction * reorganize challenges high -> low abstraction * add note justifying linear search * Simplify max_pre_score validation Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * add minor test of pjr challenge validation Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> |
||
|
|
781f908760 |
Implement PJR checker (#8160)
* Apply. * get rid of glob import * use meaningful generic type name * pjr_check operates on `Supports` struct used elsewhere * improve algorithmic complexity of `prepare_pjr_input` * fix rustdoc warnings * improve module docs * typo * simplify debug assertion * add test finding the phase-change threshold value for a constructed scenario * add more threshold scenarios to disambiguate plausible interpretations * add link to npos paper reference * docs: staked_assignment -> supports Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * add utility method for generating npos inputs * add a fuzzer which asserts that all unbalanced seq_phragmen are PJR Note that this currently fails. I hope that this can be rectified by calculating the threshold instead of choosing some arbitrary number. * assert in all cases, not just debug * leverage a native solution to choose candidates * use existing helper methods * add pjr-check and incorporate into the fuzzer We should probably have one of the W3F people look at this to ensure we're not misconstruing any definitions, but this seems like a fairly straightforward implementation. * fix compilation errors * Enable manually setting iteration parameters in single run. This gives us the ability to reproducably extract cases where honggfuzz has discovered a panic. For example: $ cargo run --release --bin phragmen_pjr -- --candidates 569 --voters 100 Tue 23 Feb 2021 11:23:39 AM CET Compiling bitflags v1.2.1 Compiling unicode-width v0.1.8 Compiling unicode-segmentation v1.7.1 Compiling ansi_term v0.11.0 Compiling strsim v0.8.0 Compiling vec_map v0.8.2 Compiling proc-macro-error-attr v1.0.4 Compiling proc-macro-error v1.0.4 Compiling textwrap v0.11.0 Compiling atty v0.2.14 Compiling heck v0.3.2 Compiling clap v2.33.3 Compiling structopt-derive v0.4.14 Compiling structopt v0.3.21 Compiling sp-npos-elections-fuzzer v2.0.0-alpha.5 (/home/coriolinus/Documents/Projects/paritytech/substrate/primitives/npos-elections/fuzzer) Finished release [optimized] target(s) in 6.15s Running `/home/coriolinus/Documents/Projects/paritytech/substrate/target/release/phragmen_pjr -c 569 -v 100` thread 'main' panicked at 'unbalanced sequential phragmen must satisfy PJR', primitives/npos-elections/fuzzer/src/phragmen_pjr.rs:133:5 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace This is still not adequate proof that seq_phragmen is broken; it could very well be that our PJR checker is doing the wrong thing, or we've somehow missed a parameter of interest. Still, it's concerning. * update comment verbiage for accuracy * it is valid in PJR for an elected candidate to have 0 support * Fix phragmen_pjr fuzzer It turns out that the fundamental problem causing previous implementations of the fuzzer to fail wasn't in `seq_phragmen` _or_ in `pjr_check`: it was in the rounding errors introduced in the various conversions between the internal data representation and the external one. Fixing the fuzzer is then simply an issue of using the internal representation and staying in that representation. However, that leaves the issue that `seq_phragmen` occasionally produces an output which is technically not PJR due to rounding errors. In the future we will need to add some kind of "close-enough" threshold. However, that is explicitly out of scope of this PR. * restart ci; it appears to be stalled * use necessary import for no-std * use a more realistic distribution of voters and candidates This isn't ideal; more realistic numbers would be about twice these. However, either case generation or voting has nonlinear execution time, and doubling these values brings iteration time from ~20s to ~180s. Fuzzing 6x as fast should make up for fuzzing cases half the size. * identify specifically which PJR check may fail * move candidate collection comment into correct place * standard_threshold: use a calculation method which cannot overflow * Apply suggestions from code review (update comments) Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * clarify the effectiveness bounds for t-pjr check * how to spell "committee" * reorganize: high -> low abstraction * ensure standard threshold calc cannot panic Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: kianenigma <kian.peymani@gmail.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> |
||
|
Peter Goodspeed-NiklausandGuillaume Thiolliere
|
44d5aba80d |
Create a macro which automates creation of benchmark test suites. (#8104)
* Create a macro which automates creation of benchmark test suites. * bump impl_version * allow unused on test_bench_by_name * use proper doctest ignore attribute * Explicitly hand the Module to the test suite Much better practice than depending on it showing up implicitly in the namespace. * explicitly import what we need into `mod tests` * bench_module is `ident` not `tt` Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * allow end users to specify arguments for new_test_ext This turned out to be surprisingly easy. On reflection, it turns out that of course the compiler can't eagerly evaluate the function call, but needs to paste it in everywhere desired. * enable explicitly specifying the path to the benchmarks invocation also enable optional trailing commas * Revert "bump impl_version" This reverts commit 0209e4de33fd43873f8cfc6875815d0fd6151e63. * list failing benchmark tests and the errors which caused the failure * harden benchmark tests against internal panics * suppress warning about ignored profiles unfortunately, setting the profile here doesn't do anything; we'd need to set it in every leaf package anyway. However, as this was just making the default explicit anyway, I think it's safe enough to remove entirely. * impl_benchmark_test_suite for assets * impl_benchmark_test_suite for balances * impl_benchmark_test_suite for bounties * impl_benchmark_test_suite for Collective * impl_benchmark_test_suite for Contracts * impl_benchmark_test_suite for Democracy * don't impl_benchmark_test_suite for Elections-Phragmen * impl_benchmark_test_suite for Identity Note that Identity tests currently fail. They failed in an identical way before this change, so as far as I'm concerned, the status quo is good enough for now. * impl_benchmark_test_suite for ImOnline * impl_benchmark_test_suite for indices For this crate also, the test suite fails identically with and without this change, so we can say that this change is not the cause of the tests' failure to compile. * impl_benchmark_test_suite for lottery * impl_benchmark_test_suite for merkle-mountain-range * impl_benchmark_test_suite for Multisig These tests fail identically with and without the change, so the change seems unlikely to be the origin of the failures. * impl_benchmark_test_suite for offences * impl_benchmark_test_suite for Proxy Fails identically with and without this change. * impl_benchmark_test_suite for scheduler * impl_benchmark_test_suite for session It turns out to be important to be able to exclude items marked `#[extra]` sometimes. Who knew? * impl_benchmark_test_suite for staking * impl_benchmark_test_suite for system * impl_benchmark_test_suite for timestamp * impl_benchmark_test_suite for tips * impl_benchmark_test_suite for treasury * impl_benchmark_test_suite for utility Note that benchmark tests fail identically before and after this change. * impl_benchmark_test_suite for vesting * fix wrong module name in impl_benchmark_test_suite in Offences * address line length nits * enable optional keyword argument: exec_name Took a _lot_ of macro-wrangling to get the functionality that I want, but now you have the option to pass in ```rust impl_benchmark_test_suite!( Elections, crate::tests::ExtBuilder::default().desired_members(13).desired_runners_up(7), crate::tests::Test, exec_name = build_and_execute, ); ``` and have it expand out properly. A selected fragment of the expansion: ```rust fn test_benchmarks() { crate::tests::ExtBuilder::default() .desired_members(13) .desired_runners_up(7) .build_and_execute(|| { ``` * get rid of dead code Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> |
||
|
Peter Goodspeed-Niklaus
|
25cfb884af |
misbehavior: report multiple offenses per validator as necessary (#2222)
* use proper descriptive generic type names * cleanup * Table stores a list of detected misbehavior per authority * add Table::drain_misbehaviors_for * WIP: unify misbehavior types; report multiple misbehaviors per validator Code checks, but tests don't yet pass. * update drain_misbehaviors: return authority id as well as specific misbehavior * enable unchecked construction of Signed structs in tests * remove test-features feature & unnecessary generic * fix backing tests This took a while to figure out, because where we'd previously been passing around `SignedFullStatement`s, we now needed to construct those on the fly within the test, to take advantage of the signature- checking in the constructor. That, in turn, necessitated changing the iterable type of `drain_misbehaviors` to return the validator index, and passing that validator index along within the misbehavior report. Once that was sorted, however, it became relatively straightforward: just needed to add appropriate methods to deconstruct the misbehavior reports, and then we could construct the signed statements directly. * fix bad merge |
||
|
Peter Goodspeed-Niklaus
|
a0239785b9 |
Message and Data Type Flow Visualizations (#2286)
* start graphing message flow between subsystems This commit includes messages originating from the first three subsystems (alphabetically). * use standard arrowhead names * add messages from bitfield distribution && use circo layout * finish adding all subsystem message traffic to graph * reduce line spam * start work on graphing v1 types * finish graphing candidate-related structs * show hash relations, format * show provenance of ValidityAttestation * add the rest of the v1 polkadot primitives * add type diagram from polkadot::parachain::primitives * start work digramming the path to backing * diagram message flow from collation generation -> candidate selection * document flow through CandidateBacking, CandidateValidation * graph data flow through StatementDistribution * trace data flow through PoVDistribution to Backing * finish documenting node side subsystem data flow |
||
|
Peter Goodspeed-Niklaus
|
089ba85c32 |
Add warning when dropping inclusion inherent (#2261)
* Add warning when dropping inclusion inherent * improve warning when dropping inclusion inherent - use the actual runtime `warn!` macro - do not speculate about why the error occurred - show the actual error returned * BROKEN: add note if the current block matches the session start block This doesn't compile for a variety of reasons; this commit demonstrates what the issues are. I don't think this is an essential feature, so I'm stopping work on it here instead of pushing through it. * Revert "BROKEN: add note if the current block matches the session start block" This reverts commit eeb79ab7708ec94224fa60c9e1a46e12c18d52fd. * ensure a runtime logger exists before attempting to log a warning to it |
||
|
Peter Goodspeed-Niklaus
|
cd58c02cd3 |
Add metrics timing message passing from OverseerSubsystemContext to Overseer::route_message (#2201)
* add timing setup to OverseerSubsystemContext * figure out how to initialize the rng * attach a timer to a portion of the messages traveling to the Overseer This timer only exists / logs a fraction of the time (configurable by `MESSAGE_TIMER_METRIC_CAPTURE_RATE`). When it exists, it tracks the span between the `OverSubsystemContext` receiving the message and its receipt in `Overseer::run`. * propagate message timing to the start of route_message This should be more accurate; it ensures that the timer runs at least as long as that function. As `route_message` is async, it may not actually run for some time after it is called (or ever). * fix failing test * rand_chacha apparently implicitly has getrandom feature * change rng initialization The previous impl using `from_entropy` depends on the `getrandom` crate, which uses the system entropy source, and which does not work on `wasm32-unknown-unknown` because it wants to fall back to a JS implementation which we can't assume exists. This impl depends only on `rand::thread_rng`, which has no documentation stating that it's similarly limited. * remove randomness in favor of a simpler 1 of N procedure This deserves a bit of explanation, as the motivating issue explicitly requested randomness. In short, it's hard to get randomness to compile for `wasm32-unknown-unknown` because that is explicitly intended to be as deterministic as practical. Additionally, even though it would never be used for consensus purposes, it still felt offputting to intentionally introduce randomness into a node's operations. Except, it wasn't really random, either: it was a deterministic PRNG varying only in its state, and getting the state to work right for that target would have required initializing from a constant. Given that it was a deterministic sequence anyway, it seemed much simpler and more explicit to simply select one of each N messages instead of attempting any kind of realistic randomness. * reinstate randomness for better statistical properties This partially reverts commit 0ab8594c328b3f9ce1f696fe405556d4000630e9. `oorandom` is much lighter than the previous `rand`-based implementation, which makes this easier to work with. This implementation gives each subsystem and each child RNG a distinct increment, which should ensure they produce distinct streams of values. |
||
|
Peter Goodspeed-NiklausandGuillaume Thiolliere
|
fcc0fca161 |
Reduce the inclusion inherent's actual weight if the block is already heavy (#2060)
* don't modify inherent data on heavy block * write up current thinking on block weight detection * extract inherent inclusion check into its own function * put heavy block check into runtime * the `inclusion` inherent call is Operational, not Mandatory This resolves a lot of the trickiness about this issue, because we no longer need to override or supplant any existing proposer logic; the existing logic should exhibit these behaviors: - the `inclusion` inherent is prioritized over standard transactions - but if it's too heavy, i.e. in case of runtime upgrade, it'll be dropped in favor of that. It is my belief that allowing the proposer to just not include this data won't have any adverse effects: it's equivalent to replacing them with empty versions of themselves, which the `ProvideInherent` impl already does. * Revert "the `inclusion` inherent call is Operational, not Mandatory" This reverts commit e58858d109b18b84e7af3ac47981c6900b2d9a3e. * Revert "write up current thinking on block weight detection" This reverts commit fd587b80c46761b2a2b62448193348237863f99f. * Revert "don't modify inherent data on heavy block" This reverts commit 38299d3c23e9efb5a354d8cfa658e62a5c8c7ddf. * add backed candidate block weight assumption to configuration * Limit backed candidates according to a candidate weight heuristic. This approach replaces making the inclusion inherent non-mandatory. It's still not ideal in that we have to configure a heuristic for how much each backed candidate 'weighs', instead of directly measuring it somehow. This approach also never truncates the signed bitfields. The rationale for that depends on some assumptions: - processing the signed bitfields is cheap compared to the backed candidates - it is beneficial to the progress of the relay chain to update the signed bitfields even if not all backed candidates are updated * simplify limit_backed_candidates and weight assumption * don't trust the provisioner to fairly distribute candidates * use saturating subtraction * empty commit to restart ci * use new mechanism for getting max block weight * apply weight refunds to the inclusion inherent This makes some assumptions about fundamental weights, which are encapsulated as constants. From there, it lets Substrate know what the actual computed weight of the inherent is. * use a correct fixed weight for the inclusion inherent Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * use dynamic inclusion weight so we reduce calculated weight when excluding candidates * don't double-count this intrinsic's weight in the block weight * add unit tests of fn limit_backed_candidates * add tests that the inclusion inherent's weight correctly updates Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> |
||
|
Peter Goodspeed-Niklaus
|
e7e9605f87 |
do not store backed candidates in the provisioner (#1909)
* guide: non-semantic changes * guide: update per the issue description * GetBackedCandidates operates on multiple hashes now * GetBackedCandidates still needs a relay parent * implement changes specified in guide * distinguish between various occasions for canceled oneshots * add tracing info to getbackedcandidates * REVERT ME: add tracing messages for GetBackedCandidates Note that these messages are only sometimes actually passed on to the candidate backing subsystem, with the consequence that it is unexpectedly frequent that the provisioner fails to create its provisionable data. * REVERT ME: more tracing logging * REVERT ME: log when CandidateBackingJob receives any message at all * REVERT ME: log when send_msg sends a message to a job * fix candidate-backing tests * streamline GetBackedCandidates This uses table.attested_candidate instead of table.get_candidate, because it's not obvious how to get a BackedCandidate from just a CommittedCandidateReceipt. * REVERT ME: more logging tracing job lifespans * promote warning about job premature demise * don't terminate CandiateBackingJob::run_loop in event of failure to process message * Revert "REVERT ME: more logging tracing job lifespans" This reverts commit 7365f2fb3dec988d95cfcd317eba75587fe7fd16. * Revert "REVERT ME: log when send_msg sends a message to a job" This reverts commit 58e46aad038e6517d6d56390c8be65b046a21884. * Revert "REVERT ME: log when CandidateBackingJob receives any message at all" This reverts commit 0d6f38413c7c66b5e9e81dabc587906fa9f82656. * Revert "REVERT ME: more tracing logging" This reverts commit 675fd2628e84d1596965280e7314155ef21b28e6. * Revert "REVERT ME: add tracing messages for GetBackedCandidates" This reverts commit e09e156493430b33b6c8ab4b5cedb3f2f91afd51. * formatting * add logging message to CandidateBackingJob::run_loop start * REVERT ME: add tracing to candidate-backing job creation * run candidatebacking loop even if no assignment * use unique error variants for each canceled oneshot * Revert "REVERT ME: add tracing to candidate-backing job creation" This reverts commit 8ce5f4f0bd7186dade134b118751480f72ea1fd6. * try_runtime_api more to reduce silent exits * add sanity check that returned backed candidates preserve ordering * remove redundant err attribute |
||
|
Peter Goodspeed-NiklausandAndronik Ordian
|
0a5bc82529 |
Add Prometheus timers to the subsystems (#1923)
* reexport prometheus-super for ease of use of other subsystems * add some prometheus timers for collation generation subsystem * add timing metrics to av-store * add metrics to candidate backing * add timing metric to bitfield signing * add timing metrics to candidate selection * add timing metrics to candidate-validation * add timing metrics to chain-api * add timing metrics to provisioner * add timing metrics to runtime-api * add timing metrics to availability-distribution * add timing metrics to bitfield-distribution * add timing metrics to collator protocol: collator side * add timing metrics to collator protocol: validator side * fix candidate validation test failures * add timing metrics to pov distribution * add timing metrics to statement-distribution * use substrate_prometheus_endpoint prometheus reexport instead of prometheus_super * don't include JOB_DELAY in bitfield-signing metrics * give adder-collator ability to easily export its genesis-state and validation code * wip: adder-collator pushbutton script * don't attempt to register the adder-collator automatically Instead, get these values with ```sh target/release/adder-collator export-genesis-state target/release/adder-collator export-genesis-wasm ``` And then register the parachain on https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9944#/explorer To collect prometheus data, after running the script, create `prometheus.yml` per the instructions at https://www.notion.so/paritytechnologies/Setting-up-Prometheus-locally-835cb3a9df7541a781c381006252b5ff and then run: ```sh docker run -v `pwd`/prometheus.yml:/etc/prometheus/prometheus.yml:z --network host prom/prometheus ``` Demonstrates that data makes it across to prometheus, though it is likely to be useful in the future to tweak the buckets. * Update parachain/test-parachains/adder/collator/src/cli.rs Co-authored-by: Andronik Ordian <write@reusable.software> * use the grandpa-pause parameter * skip metrics in tracing instrumentation * remove unnecessary grandpa_pause cli param Co-authored-by: Andronik Ordian <write@reusable.software> |
||
|
Peter Goodspeed-Niklaus
|
e49989971d |
Add tracing support to node (#1940)
* drop in tracing to replace log * add structured logging to trace messages * add structured logging to debug messages * add structured logging to info messages * add structured logging to warn messages * add structured logging to error messages * normalize spacing and Display vs Debug * add instrumentation to the various 'fn run' * use explicit tracing module throughout * fix availability distribution test * don't double-print errors * remove further redundancy from logs * fix test errors * fix more test errors * remove unused kv_log_macro * fix unused variable * add tracing spans to collation generation * add tracing spans to av-store * add tracing spans to backing * add tracing spans to bitfield-signing * add tracing spans to candidate-selection * add tracing spans to candidate-validation * add tracing spans to chain-api * add tracing spans to provisioner * add tracing spans to runtime-api * add tracing spans to availability-distribution * add tracing spans to bitfield-distribution * add tracing spans to network-bridge * add tracing spans to collator-protocol * add tracing spans to pov-distribution * add tracing spans to statement-distribution * add tracing spans to overseer * cleanup |
||
|
Peter Goodspeed-Niklaus
|
d13a335df6 |
add max_pov_size to runtime config and PersistedValidationData (#1984)
* add max_pov_size to runtime config and PersistedValidationData Closes #1572. * set default genesis max_pov_size * apply suggestions from code review * add default max_pov_size to polkadot_testnet_genesis |
||
|
Peter Goodspeed-Niklaus
|
75da335d66 |
Restrict collating_using_adder_collator test on feature real-overseer (#1947)
This test can only pass given that feature, and runs forever otherwise. This change ensure that a careless `cargo test --all` doesn't run forever. |
||
|
Peter Goodspeed-NiklausandSergei Shulepov
|
45c9aefd27 |
add native logging to check_upward_messages (#1906)
* add native logging to check_upward_messages This doesn't affect the WASM builds, but ensures that native versions (such as are used for testing) emit diagnostics anytime the check_upward_messages function exits, which should reduce the pain of debugging when something goes wrong. * Apply suggestions from code review Co-authored-by: Sergei Shulepov <sergei@parity.io> * verb tense Co-authored-by: Sergei Shulepov <sergei@parity.io> |
||
|
Peter Goodspeed-Niklaus
|
62daee2286 | remove the --disable-authority-discovery flag & related (#1870) | ||
|
|
1a25c41277 |
start working on building the real overseer (#1795)
* start working on building the real overseer Unfortunately, this fails to compile right now due to an upstream failure to compile which is probably brought on by a recent upgrade to rustc v1.47. * fill in AllSubsystems internal constructors * replace fn make_metrics with Metrics::attempt_to_register * update to account for #1740 * remove Metrics::register, rename Metrics::attempt_to_register * add 'static bounds to real_overseer type params * pass authority_discovery and network_service to real_overseer It's not straightforwardly obvious that this is the best way to handle the case when there is no authority discovery service, but it seems to be the best option available at the moment. * select a proper database configuration for the availability store db * use subdirectory for av-store database path * apply Basti's patch which avoids needing to parameterize everything on Block * simplify path extraction * get all tests to compile * Fix Prometheus double-registry error for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476: ```rust Some(registry) => Self::try_register(registry).map_err(|err| { eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err); err }), ``` That pointed out where the registration was failing, which led to this fix. The test still doesn't pass, but it now fails in a new and different way! * authorities must have authority discovery, but not necessarily overseer handlers * fix broken SpawnedSubsystem impls detailed logging determined that using the `Box::new` style of future generation, the `self.run` method was never being called, leading to dropped receivers / closed senders for those subsystems, causing the overseer to shut down immediately. This is not the final fix needed to get things working properly, but it's a good start. * use prometheus properly Prometheus lets us register simple counters, which aren't very interesting. It also allows us to register CounterVecs, which are. With a CounterVec, you can provide a set of labels, which can later be used to filter the counts. We were using them wrong, though. This pattern was repeated in a variety of places in the code: ```rust // panics with an cardinality mismatch let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?; my_counter.with_label_values(&["succeeded"]).inc() ``` The problem is that the labels provided in the constructor are not the set of legal values which can be annotated, but a set of individual label names which can have individual, arbitrary values. This commit fixes that. * get av-store subsystem to actually run properly and not die on first signal * typo fix: incomming -> incoming * don't disable authority discovery in test nodes * Fix rococo-v1 missing session keys * Update node/core/av-store/Cargo.toml * try dummying out av-store on non-full-nodes * overseer and subsystems are required only for full nodes * Reduce the amount of warnings on browser target * Fix two more warnings * InclusionInherent should actually have an Inherent module on rococo * Ancestry: don't return genesis' parent hash * Update Cargo.lock * fix broken test * update test script: specify chainspec as script argument * Apply suggestions from code review Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Update node/service/src/lib.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * node/service/src/lib: Return error via ? operator * post-merge blues * add is_collator flag * prevent occasional av-store test panic * simplify fix; expand application * run authority_discovery in Role::Discover when collating * distinguish between proposer closed channel errors * add IsCollator enum, remove is_collator CLI flag * improve formatting * remove nop loop * Fix some stuff Co-authored-by: Andronik Ordian <write@reusable.software> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com> Co-authored-by: Robert Habermeier <robert@Roberts-MBP.lan1> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Max Inden <mail@max-inden.de> |
||
|
Peter Goodspeed-NiklausandBastian Köcher
|
c2941a2789 |
Remove old service, 3rd try (#1776)
* Remove old service, 3rd try
i.e.
Revert "Revert "Remove Old Service, 2nd try (#1732)" (#1758)"
This reverts commit
|
||
|
Peter Goodspeed-Niklaus
|
f31c904140 |
add two node local net script (#1781)
* initial work running a two node local net * use the right incantations so the nodes produce blocks together * improve internal documentation |
||
|
Peter Goodspeed-Niklaus
|
9a0f08bfe1 |
Revert "Remove Old Service, 2nd try (#1732)" (#1758)
This reverts commit
|
||
|
|
96f2615582 |
Remove Old Service, 2nd try (#1732)
* Restore "Remove service, migrate all to service-new (#1630)"
i.e.
Revert "Revert "Remove service, migrate all to service-new (#1630)" (#1731)"
This reverts commit
|
||
|
Peter Goodspeed-Niklaus
|
b4457f555b |
Revert "Remove service, migrate all to service-new (#1630)" (#1731)
This reverts commit
|
||
|
|
d531ba561c |
Remove service, migrate all to service-new (#1630)
* import rococo into chain-spec * make a few stabs at moving forward * wip: rococo readme * remove /service crate - Move the chain-spec files to node-service - update sufficient cargo files that polkadot-service-new builds - not everything else builds yet * wip: chase down some build errors in polkadot-cli There's a lot more to go, but some progress has happened. * make more progress getting polkadot-cli to build * don't ignore polkadot.json within the res directory * don't recreate pathbufs * Prepare Polkadot to be used by Cumulus This begins to make Polkadot usable from Cumulus. * Remove old test * migrate new_chain_ops fix from /service * partially remove node/test-service * Reset some changes * Revert "partially remove node/test-service" This reverts commit 7b8f9ba5bfc286a309df89853ae11facf3277ffb. * WIP: replace v0 ParachainHost impl with v1 for test runtime This is necessary because one of the current errors when building the test service boils down to: the trait bound `polkadot_test_runtime::RuntimeApiImpl<...>`: `polkadot_primitives::v1::ParachainHost<...>` is not satisfied This is WIP because it appears to be causing some std leakage into the wasm environment, or something; the compiler is currently complaining about duplicate definitions of `panic_handler` and `oom`. Presumably I have to identify all std types (Vec etc) and replace them with sp_std equivalents. * fix test runtime build it wasn't std leakage, after all * bump westend spec version * use service-new as service within cli * to revert: demo that forwarding the test runtime to the real impl blows up * Revert "to revert: demo that forwarding the test runtime to the real impl blows up" This reverts commit 68d2f385f378721c7433e3e39133434610cd2a51. * Revert "Revert "to revert: demo that forwarding the test runtime to the real impl blows up"" This reverts commit 04cb1cbf8873b4429cb9c9fdccb7f4bb137dc720. Might have just forgotten to disable default features * More reverts * MOARE * plug in the runtime as the generic instantiation This feels closer to a solution, but it still has problems: in particular, it's assumed that Runtime implements all appropriate Trait traits, which this one apparently does not. * implement necessary traits to get the test runtime compiling This is almost certainly not correct in some way; it really looks like I need to mess with the construct_runtime! macro somehow, to inject the inclusion trait's event type as a Event variant. Still, better lock down this changeset while it all compiles. * add inclusion::Event as variant into Event enum * implement unimplemented bits in kusama * implement unimplemented bits in polkadot runtime * implement unimplemented bits in westend runtime * migrate client upgrades from master * update test service with new node changes * package metadata--that wasn't intended to be removed * add parachains v1 modules to each runtime It's not clear what precisely this does, but it's probably the right thing to do. * enable cli to opt out of full node features * adjust rococo chainspec per example https://github.com/paritytech/polkadot/blob/26f1fa47f7836ab4bee5d4aad127ebce748320dd/service/src/chain_spec.rs#L362 * try to fix Cargo.lock Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Andronik Ordian <write@reusable.software> |
||
|
Peter Goodspeed-NiklausandBernhard Schuster
|
d1b1c17285 |
implement candidate selection subsystem (#1645)
* choose the straightforward candidate selection algorithm for now * add draft implementation of candidate selection * fix typo in summary * more properly report misbehaving collators * describe how CandidateSelection subsystem becomes aware of candidates * revise candidate selection / collator protocol interaction pattern * implement rest of candidate selection per the guide * review: resolve nits * start writing test suite, harness * implement first test * add second test * implement third test Co-authored-by: Bernhard Schuster <bernhard@ahoi.io> |
||
|
Peter Goodspeed-Niklaus
|
54bec3bfc0 |
implement collation generation subsystem (#1557)
* start sketching out a collation generation subsystem * invent a basic strategy for double initialization * clean up warnings * impl util requests from runtime assuming a context instead of a FromJob sender * implement collation generation algorithm from guide * update AllMessages in tests * fix trivial review comments * remove another redundant declaration from merge * filter availability cores by para_id * handle new activations each in their own async task * update guide according to the actual current implementation * add initialization to guide * add general-purpose subsystem_test_harness helper * write first handle_new_activations test * add test that handle_new_activations filters local_validation_data requests * add (failing) test of collation distribution message sending * rustfmt * broken: work on fixing sender test Unfortunately, for reasons that are not yet clear, despite the public key and checked data being identical, the signer is not producing an identical signature. This commit produces this output (among more): signing with Public(c4733ab0bbe3ba4c096685d1737a7f498cdbdd167a767d04a21dc7df12b8c858 (5GWHUNm5...)) checking with Public(c4733ab0bbe3ba4c096685d1737a7f498cdbdd167a767d04a21dc7df12b8c858 (5GWHUNm5...)) signed payload: [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 10, 0, 0, 0, c7, e5, c0, 64, 7a, db, fe, 44, 81, e5, 51, 11, 79, 9f, a5, 63, 93, 94, 3c, c4, 36, c6, 30, 36, c2, c5, 44, a2, 1b, db, b7, 82, 3, 17, a, 2e, 75, 97, b7, b7, e3, d8, 4c, 5, 39, 1d, 13, 9a, 62, b1, 57, e7, 87, 86, d8, c0, 82, f2, 9d, cf, 4c, 11, 13, 14] checked payload: [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 10, 0, 0, 0, c7, e5, c0, 64, 7a, db, fe, 44, 81, e5, 51, 11, 79, 9f, a5, 63, 93, 94, 3c, c4, 36, c6, 30, 36, c2, c5, 44, a2, 1b, db, b7, 82, 3, 17, a, 2e, 75, 97, b7, b7, e3, d8, 4c, 5, 39, 1d, 13, 9a, 62, b1, 57, e7, 87, 86, d8, c0, 82, f2, 9d, cf, 4c, 11, 13, 14] * fix broken test * collation function returns commitments hash It doesn't look like we use the actual commitments data anywhere, and it's not obvious if there are any fields of `CandidateCommitments` not available to the collator, so this commit just assigns them the entire responsibility of generating the hash. * add missing overseer impls * calculating erasure coding is polkadot's responsibility, not cumulus * concurrentize per-relay_parent requests |
||
|
Peter Goodspeed-Niklaus
|
9fda6cb416 |
break out subsystem-util and subsystem-test-helpers into individual crates (#1553)
* break out subsystem-util and subsystem-test-helpers into individual crates * cause all packages to check successfully |
||
|
Peter Goodspeed-Niklaus
|
21cec309a4 |
implement provisioner (#1473)
* sketch out provisioner basics
* handle provisionable data
* stub out select_inherent_data
* split runtime APIs into sub-chapters to improve linkability
* explain SignedAvailabilityBitfield semantics
* add internal link to further documentation
* some more work figuring out how the provisioner can do its thing
* fix broken link
* don't import enum variants where it's one layer deep
* make request_availability_cores a free fn in util
* document more precisely what should happen on block production
* finish first-draft implementation of provisioner
* start working on the full and proper backed candidate selection rule
* Pass number of block under construction via RequestInherentData
* Revert "Pass number of block under construction via RequestInherentData"
This reverts commit 850fe62cc0dfb04252580c21a985962000e693c8.
That initially looked like the better approach--it spent the time
budget for fetching the block number in the proposer, instead of
the provisioner, and that felt more appropriate--but it turns out
not to be obvious how to get the block number of the block under
construction from within the proposer. The Chain API may be less
ideal, but it should be easier to implement.
* wip: get the block under production from the Chain API
* add ChainApiMessage to AllMessages
* don't break the run loop if a provisionable data channel closes
* clone only those backed candidates which are coherent
* propagate chain_api subsystem through various locations
* add delegated_subsystem! macro to ease delegating subsystems
Unfortunately, it doesn't work right:
```
error[E0446]: private type `CandidateBackingJob` in public interface
--> node/core/backing/src/lib.rs:775:1
|
86 | struct CandidateBackingJob {
| - `CandidateBackingJob` declared as private
...
775 | delegated_subsystem!(CandidateBackingJob as CandidateBackingSubsystem);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ can't leak private type
```
I'm not sure precisely what's going wrong, here; I suspect the problem is
the use of `$job as JobTrait>::RunArgs` and `::ToJob`; the failure would be
that it's not reifying the types to verify that the actual types are public,
but instead referring to them via `CandidateBackingJob`, which is in fact private;
that privacy is the point.
Going to see if I can generic my way out of this, but we may be headed for a
quick revert here.
* fix delegated_subsystem
The invocation is a bit more verbose than I'd prefer, but it's also
more explicit about what types need to be public. I'll take it as a win.
* add provisioning subsystem; reduce public interface of provisioner
* deny missing docs in provisioner
* refactor core selection per code review suggestion
This is twice as much code when measured by line, but IMO it is
in fact somewhat clearer to read, so overall a win.
Also adds an improved rule for selecting availability bitfields,
which (unlike the previous implementation) guarantees that the
appropriate postconditions hold there.
* fix bad merge double-declaration
* update guide with (hopefully) complete provisioner candidate selection procedure
* clarify candidate selection algorithm
* Revert "clarify candidate selection algorithm"
This reverts commit c68a02ac9cf42b3a4a28eb197d38633a40d0e3e6.
* clarify candidate selection algorithm
* update provisioner to implement candidate selection per the guide
* add test that no more than one bitfield is selected per validator
* add test that each selected bitfield corresponds to an occupied core
* add test that more set bits win conflicts
* add macro for specializing runtime requests; specailize all runtime requests
* add tests harness for select_candidates tests
* add first real select_candidates test, fix test_harness
* add mock overseer and test that success is possible
* add test that the candidate selection algorithm picks the right ones
* make candidate selection test somewhat more stringent
|
||
|
Peter Goodspeed-Niklaus
|
4b2cb04e56 |
Fix bitfield signing (#1466)
* Apply suggestions from #1364 code review - use CoreState, not CoreOccupied - query for availability chunks, not the whole PoV - create a stub `fn availability_cores` * link to issue documenting unimplemented * implement get_availability_cores by adding a new runtime api request * back out an unrelated change properly part of #1404 * av-store: handle QueryChunkAvailability * simplify QueryDataAvailability * remove extraneous whitespace * compact primitive imports |
||
|
Peter Goodspeed-Niklaus
|
bcfd6c2d3a |
add simple name derivation heuristic for JobManager Subsystem impl (#1490)
Subsystems are encouraged to either typedef themselves as appropriate `JobManager` instances for their job type, or wrap a `JobManager` instance and delegate the `Subsystem` impl. In both cases, we want to use a sensible, non-repeated subsystem name for appropriate logging and debugging. This PR adds a heuristic: if the job name ends in the literal "Job", then that gets stripped. Otherwise, the job name is used. This improves on the previous situation, in which subsystems typedef'd to or wrapping `JobManager` all got the same constant (!) name. |
||
|
Peter Goodspeed-Niklaus
|
106bd929ce |
add ActiveLeavesUpdate, remove StartWork, StopWork (#1458)
* add ActiveLeavesUpdate, remove StartWork, StopWork * replace StartWork, StopWork in subsystem crate tests * mechanically update OverseerSignal in other modules * convert overseer to take advantage of new multi-hash update abilities Note: this does not yet convert the tests; some of the tests now freeze: test tests::overseer_start_stop_works ... test tests::overseer_start_stop_works has been running for over 60 seconds test tests::overseer_finalize_works ... test tests::overseer_finalize_works has been running for over 60 seconds * fix broken overseer tests * manually impl PartialEq for ActiveLeavesUpdate, rm trait Equivalent This cleans up the code a bit and makes it easier in the future to do the right thing when comparing ALUs. * use target in all network bridge logging * reduce spamming of and |
||
|
Peter Goodspeed-Niklaus
|
1ed17cd467 | assign authorship to parity (#1459) | ||
|
Peter Goodspeed-NiklausandAndronik Ordian
|
ba4bfa4dd0 |
implement bitfield signing subsystem (#1364)
* update guide to reduce confusion and TODOs * work from previous bitfield signing effort There were large merge issues with the old bitfield signing PR, so we're just copying all the work from that onto this and restarting. Much of the existing work will be discarded because we now have better tools available, but that's fine. * start rewriting bitfield signing in terms of the util module * implement construct_availability_bitvec It's not an ideal implementation--we can make it much more concurrent-- but at least it compiles. * implement the unimplemented portions of bitfield signing * get core availability concurrently, not sequentially * use sp-std instead of std for a parachain item * resolve type inference failure caused by multiple From impls * handle bitfield signing subsystem & Allmessages variant in overseer * fix more multi-From inference issues * more concisely handle overflow Co-authored-by: Andronik Ordian <write@reusable.software> * Revert "resolve type inference failure caused by multiple From impls" This reverts commit 7fc77805de5e5074a1b01037f8d4e3919e03e0e1. * Revert "fix more multi-From inference issues" This reverts commit f14ffe589e20d664d8a900ed62f68b6fb844a514. * impl From<i32> for ParaId * handle another instance of AllSubsystems * improve consistency when returning existing options Co-authored-by: Andronik Ordian <write@reusable.software> |
||
|
Peter Goodspeed-Niklaus
|
5cfcc8446c |
Add test suite and minor refinements to the utility subsystem (#1403)
* get conclude signal working properly; don't allocate a vector * wip: add test suite / example / explanation for using utility subsystem Unfortunately, the test fails right now for reasons which seem very odd. Just have to keep poking at it. * explicitly import everything * fix subsystem-util test The root problem here was two-fold: - there was a circular dependency from subsystem -> test-helpers/subsystem -> subsystem - cfg(test) doesn't propagate between crates The solution: move the subsystem test helpers into a sub-module within subsystem. Publicly export them from the previous location so no other code breaks. Doing this has an additional benefit: it ensures that no production code can ever accidentally use the subsystem helpers, as they are compile- gated on cfg(test). * fully commit to moving test helpers into a subsystem module * add some more tests * get rid of log tests in favor of real error forwarding It's not obvious whether we'll ever really want to chase down these errors outside a testing context, but having the capability won't hurt. * fix issue which caused test to hang on osx * only require that job errors are PartialEq when testing also fix polkadot-node-core-backing tests * get rid of any notion of partialeq * rethink testing Combine tests of starting and stopping job: leaving a test executor with a job running was pretty clearly the cause of the sometimes-hang. Also, add a timeout so tests _can't_ hang anymore; they just fail after a while. * rename fwd_errors -> forward_errors * warn on error propagation failure * fix unused import leftover from merge * derive eq for subsystemerror |
||
|
Peter Goodspeed-Niklaus
|
46e835f4e6 |
mod subsystem-util (#1376)
* Add subsystem-util crate.
Start by moving the JobCanceler here.
* copy utility functions for requesting runtime data; generalize
* convert subsystem-util from crate to module in subsystem
The point of making a sub-crate is to ensure that only the necessary
parts of a program get compiled; if a dependent package needed only
subsystem-util, and not subsystem, then subsystem wouldn't need to
be compiled.
However, that will never happen: subsystem-util depends on
subsystem::messages, so subsystem will always be compiled.
Therefore, it makes more sense to add it as a module in the existing
crate than as a new and distinct crate.
* make runtime request sender type generic
* candidate backing subsystem uses util for api requests
* add struct Validator representing the local validator
This struct can be constructed when the local node is a validator;
the constructor fails otherwise. It stores a bit of local data, and
provides some utility methods.
* add alternate constructor for better efficiency
* refactor candidate backing to use utility methods
* fix test breakage caused by reordering tests
* restore test which accidentally got deleted during merge
* start extracting jobs management into helper traits + structs
* use util::{JobHandle, Jobs} in CandidateBackingSubsystem
* implement generic job-manager subsystem impl
This means that the work of implementing a subsystem boils down
to implementing the job, and then writing an appropriate
type definition, i.e.
pub type CandidateBackingSubsystem<Spawner, Context> =
util::JobManager<Spawner, Context, CandidateBackingJob>;
* add hash-extraction helper to messages
* fix errors caused by improper rebase
* doc improvement
* simplify conversion from overseer communication to job message
* document fn hash for all messages
* rename fn hash() -> fn relay_parent
* gracefully shut down running futures on Conclude
* ensure we're validating with the proper validator index
* rename: handle_unhashed_msg -> handle_orphan_msg
* impl Stream for Jobs<Spawner, Job>
This turns out to be relatively complicated and requires some
unsafe code, so we'll want either detailed review, or to choose
to revert this commit.
* add missing documentation for public items
* use pin-project to eliminate unsafe code from this codebase
* rename SenderMessage -> FromJob
* reenvision the subsystem requests as an extension trait
This works within `util.rs`, but fails in `core/backing/src/lib.rs`,
because we don't actually create the struct soon enough. Continuing
down this path would imply substantial rewriting.
* Revert "reenvision the subsystem requests as an extension trait"
This reverts commit a5639e36017a72656b478caddcaa30e2d4e6112a.
The fact is, the new API is more complicated to no real benefit.
* apply suggested futuresunordered join_all impl
* CandidateValidationMessage variants have no top-level relay parents
* rename handle_orphan_msg -> handle_unanchored_msg
* make most node-core-backing types private
Now the only public types exposed in that module are
CandidateBackingSubsystem and ToJob. While ideally we could reduce
the public interface to only the former type, that doesn't work
because ToJob appears in the public interface of CandidateBackingSubsystem.
This also involves changing the definition of CandidateBackingSubsystem;
it is no longer a typedef, but a struct wrapping the job manager.
|
||
|
Peter Goodspeed-Niklaus
|
3725bb6228 |
add more detail to provisioner block production section (#1370)
Per https://github.com/paritytech/polkadot/commit/f210456#r40392855 |
||
|
Peter Goodspeed-Niklaus
|
f3c2db1a28 |
convert SignedAvailabilityBitfields from newtype to typedef (#1342)
Closes #1339. |
||
|
|
f2104562d8 |
implement custom proposer (#1320)
* network bridge skeleton
* move some primitives around and add debug impls
* protocol registration glue & abstract network interface
* add send_msgs to subsystemctx
* select logic
* transform different events into actions and handle
* implement remaining network bridge state machine
* start test skeleton
* make network methods asynchronous
* extract subsystem out to subsystem crate
* port over overseer to subsystem context trait
* fix minimal example
* fix overseer doc test
* update network-bridge crate
* write a subsystem test-helpers crate
* write a network test helper for network-bridge
* set up (broken) view test
* Revamp network to be more async-friendly and not require Sync
* fix spacing
* fix test compilation
* insert side-channel for actions
* Add some more message types to AllMessages
* introduce a test harness
* impl ProvideInherent for InclusionInherent
* reduce import churn; correct expect message
* move inclusion inherent identifier into primitives
It's not clear precisely why this is desired, but it's a pattern
I've seen in several places, so I'm going this to be on the
safe side. Worst case, we can revert this commit pretty easily.
* bump kusama spec_version to placate CI
* copy sc_basic_authorship::{ProposerFactory, Proposer}
We have from the problem description:
> This Proposer will require an OverseerHandle to make requests via.
That's next on the plate.
* use polkadot custom proposer instead of basic-authorship one
* add some tests
* ensure service compiles and passes tests
* fix typo
* fix service-new compilation
* Subsystem test helpers send messages synchronously
* remove smelly action inspector
* remove superfluous let binding
* fix warnings
* add license header
* empty commit; maybe github will notice the one with changes
* Update node/network/bridge/src/lib.rs
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* add sanity check to only include valid inherents
* stub: encapsulate block production mechanics instead of copying them
The goal is to end up with something like what's in
validation::block_production::*, which encapsulates
basic block production mechanics. This is a better idea than
just straight-up copying those mechanics.
* partial implementation of propose fn
Doesn't actually compile yet; need to bring in some other
commits to ensure ProvisionerMessage is a thing, and also
figure out how to get the block hash given the current
context.
* fix compilation
* clear a few more compile errors
* finish fn propose
* broken: add timeout to proposal
* add timeout to proposal
* guide: provisioner is responsible for selecting parachain candidates
* implement ProvisionerMessage::RequestInherentData & update fn propose
* impl CreateProposer::init; clean up
* impl std::error::Error for Error
* document error-handling rationale
* cause polkadot-service-new to compile correctly
* Move potentially-blocking call from fn init -> fn propose
This means that we can wrap the delayed call into the same
timeout check used elsewhere.
* document struct Proposer
* extract provisioner data fetch
This satisfies two requirements:
- only applies the timeout to actually fetching the provisioner data,
not to constructing the block after
- simplifies the problem of injecting default data if we could not
get the real provisioner data in time.
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Gavin Wood <gavin@parity.io>
|
||
|
Peter Goodspeed-Niklaus
|
90de55918a |
impl ProvideInherent for InclusionInherent (#1318)
* impl ProvideInherent for InclusionInherent * reduce import churn; correct expect message * move inclusion inherent identifier into primitives It's not clear precisely why this is desired, but it's a pattern I've seen in several places, so I'm going this to be on the safe side. Worst case, we can revert this commit pretty easily. * bump kusama spec_version to placate CI * add license header * empty commit; maybe github will notice the one with changes * add sanity check to only include valid inherents |
||
|
Peter Goodspeed-NiklausandRobert Habermeier
|
14ce04c9cd |
signed wrapper (#1283)
* add signed wrapper, typedef SignedStatement * typedef SignedAvailabilityBitfield * implement Signed wrapper This is strictly an addition as of this commit; nothing is yet changed in existing behavior. * inline getters, remove review comment * move EncodeAs, Signed from node::primitives to primitives::parachain * Refactor SignedAvailabilityBitfield to use Signed * don't double-encode real payload This isn't an ideal solution, because it depends on the implementation details of how SCALE encodes tuples, but OTOH that behavior seems unlikely to change anytime soon. * fix build errors * cause the runtime to build properly with the new changes Not sure why cargo check didn't catch this earlier; oh well. * fix runtime tests and separate SignedStatement from SignedFullStatement * better explain why CompactStatement exists Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Robert Habermeier <rphmeier@gmail.com> |
||
|
Peter Goodspeed-Niklaus
|
04097c881d |
Update guide candidate validation module (#1264)
* first pass updating candidate validation module in the guide * expand candidate validation functionality section * add candidate fetch; validation requires PoV to be provided * remove candidate fetch |
||
|
Peter Goodspeed-Niklaus
|
0dffa4e4d8 |
small tweaks to candidate selection (#1253)
* small tweaks to candidate selection * clarify the nature of inputs |
||
|
Peter Goodspeed-Niklaus
|
c1b329491f |
update provisioner subsystem (#1257)
* update provisioner subsystem Closes #1143 * update with answers to the questions posed by previous todos * add misbehavior reports, disputes to provisioner messages * expand on the protocol * updates per code review |
||
|
Peter Goodspeed-Niklaus
|
053bfc2d0c |
Convert guide from single markdown file to mdbook (#1247)
* move old implementers' guide, add skeleton of new * Split the old implementers' guide into the new one's sections This is mostly a straightforward copying operation, moving the appropriate sections from the old guide to the new. However, there are certain differences between the old text and the new: - removed horizontal rules between the sections - promoted headers appropriately within each section - deleted certain sections which were in the old guide's ToC but which were not actually present in the old guide. - added Peer Set Manager to the new ToC * remove description headers It is redundant and unnecessary. Descriptions fall directly under the top-level header for any given section. * add stub description of the backing module * add stub description for the availability module * add stub description for collators * add stub description for validity * add stub description for utility * highlight TODO and REVIEW comments * add guide readme describing how to use mdbook * fix markdownlint lints * re-title parachains overview * internal linking for types * module and subsystem internal links * .gitignore should have a trailing newline * node does not have modules, just subsystems |
||
|
|
67e7fba60d |
expand validity subsystems descriptions independent of subsystems overview (#1185)
* expand subsystems descriptions independent of subsystems overview The subsystems overview in https://github.com/paritytech/polkadot/pull/1161 is a very useful reference for what subsystems need to talk to each other, when, and why; it helps us design the actual messages. However, administratively, it belongs in a different PR. This commit brings in all the changes made there so far as a base for an independent PR. * Reorder subsystem descriptions, add some messages Update ordering of subsystem descriptions to rough order of use, mirroring the order in the overview document. Added some message types. Added OverseerSignal variants to several types, such that each subsystem only needs to listen for a single type. * add some more message types, Statement Distribution overview * add more detail on Statement Distribution, Misbehavior Arbitration * intentionally punt MA details for a future PR * reduce duplication from overseer signal handling * reword for clarity * clarify: other modules and subsystems also get to talk to the network * finish current work on candidate selection * update candidate backing subsystem description according to current thought * update mechanism for candidate backing to report collator misbehavior to candidate selection * sketch out the peer receipt state machine * Fix typo in roadmap/implementors-guide/guide.md Co-authored-by: Robert Habermeier <rphmeier@gmail.com> * Don't specify 'peer validators', as messages from non-validator peers are ignored regardless * clarify instancing of peer receipt state machine * add section on peer knowledge tracking * fix typo in roadmap/implementors-guide/guide.md Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Max Inden <mail@max-inden.de> |
||
|
Peter Goodspeed-Niklaus
|
3d0bda8a42 |
Add an overview of Validity subsystems (#1161)
* migrate subsystem expansion PR from w3f/research As the file at this new location included changes not present in https://github.com/w3f/research/pull/87, this is effectively a rebase, applied manually. I believe that I have successfully retained all of, and only, the intended changes. * add section on collators * note why the overseer is the terminal message point for the validation subsystem * add detail about how the validitiy system gets PoV blocks * rename to Validity Subsystems Overview * get rid of changes to the implementor's guide I think it makes most sense to track and review the subsystems overview in a different PR than the one which adds content to the implementor's guide. * punt misbehavior arbitration details to the future * empty commit to rerun CI |
||
|
Peter Goodspeed-NiklausandCecile Tonglet
|
a29ac85f1d |
fix docker script (#93)
* fix test collator compilation failure Closes #88. * copy method instead of using feature Using a feature gate like "test-features" is problematic because it is leaky: depending on situational considitons such as the current working directory when compilation is attempted, the feature may or may not be applied, which makes success inconsistent. It's simpler in this case to copy a dozen lines of code than to work out all the issues with test features. * docker script shoudld work on cumulus-branch (default) * Add storage_append to runtime Co-authored-by: Cecile Tonglet <cecile.tonglet@cecton.com> |
||
|
Peter Goodspeed-Niklaus
|
00da3f46e7 |
fix test collator compilation failure (#89)
* fix test collator compilation failure Closes #88. * copy method instead of using feature Using a feature gate like "test-features" is problematic because it is leaky: depending on situational considitons such as the current working directory when compilation is attempted, the feature may or may not be applied, which makes success inconsistent. It's simpler in this case to copy a dozen lines of code than to work out all the issues with test features. |
||
|
|
0db3088d2f |
runtime upgrade (#70)
* Initial commit * Update to 3e65111 * Add cfg_attr ... no_std * Fix version * WIP: add really simple validate_block insert validity check * WIP: create a parachain upgrade pallet This pallet will eventually make life much easier for people attempting to upgrade parachains on their validator nodes, but for the moment, key sections remain unimplemented while dependency details are worked out. * Implement basic admin-auth pallet functionality. This compiles, which means it's probably mostly correct. However, it's pretty far from being finished. Work yet to come: - Integrate with the democracy pallet somehow to eliminate the requirement for the root user to initiate this process. - Figure out what to do in the event that the parachain blocks and relay chain blocks get out of sync / delayed. - Add testing... somehow. (What's reasonable to test?) Open questions: - Is the block number parameter in `on_initialize` the parachain block number, or the relay chain block number? If, as I suspect, it's the parachain block number, how do we deal with the fact that the real upgrade should happen on a very specific parachain block number? - In line 68, is it reasonable to use `if n >= apply_block`, or should that line require strict equality? - Is it reasonable to store/retrieve `CurrentBlockNumber` on every block, or is there a more streamlined way to pass that data between functions? Maybe it can be inserted into `struct Module` somehow? - Can we somehow parametrize ValidationUpgradeDelayBlocks by T in order to eliminate the `.into()` call? * use a better storage name * Add checks ensuring runtime versions increase Largely cribbed from https://github.com/paritytech/substrate/blob/a439a7aa5a9a3df2a42d9b25ea04288d3a0866e8/frame/system/src/lib.rs#L467-L494 * fix tests * WIP: add tests from frame/system set_code Currently doesn't build: line 230 is the problem. Removing or commenting that line results in the new tests failing due to a missing block number. Adding it, in an attempt to fix the problem, fails to compile with this error: ``` Compiling parachain-upgrade-pallet v2.0.0 (/home/coriolinus/Documents/Projects/coriolinus/parachain-upgrade-pallet) error[E0599]: no function or associated item named `set_block_number` found for struct `Module<tests::Test>` in the current scope --> src/lib.rs:230:21 | 47 | / decl_module! { 48 | | pub struct Module<T: Trait> for enum Call where origin: T::Origin { 49 | | // Initializing events 50 | | // this is needed only if you are using events in your pallet ... | 100 | | } 101 | | } | |_- function or associated item `set_block_number` not found for this ... 230 | System::set_block_number(123); | ^^^^^^^^^^^^^^^^ | | | function or associated item not found in `Module<tests::Test>` | help: there is a method with a similar name: `current_block_number` | = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info) error: aborting due to previous error For more information about this error, try `rustc --explain E0599`. error: could not compile `parachain-upgrade-pallet`. ``` That error is very weird, because the function does in fact exist: https://github.com/paritytech/substrate/blob/a439a7aa5a9a3df2a42d9b25ea04288d3a0866e8/frame/system/src/lib.rs#L897 * cause tests to pass Turns out that in fact there was some setup required in order to get everything testing properly, but now we have a set of passing unit tests which test some of the more common error cases. * Add overlapping upgrades test This currently fails, and I don't yet know why. TODO! * Fix some logic errors - In particular, only remove the pending validation function from storage when it's time to apply it. - Don't store our own copy of the current block number. * WIP: delegate most code upgrade permissions checks They're defined in System::can_set_code, so may as well use them. Unfortunately, the tests all fail for this right now, and I don't yet understand why. Pushing to get immutable line number references. * fix tests after delegating runtime checks to can_set_code * WIP: events test Right now, the events struct doesn't seem to contain enough information to validate the particular events that we should have fired. Almost certainly, this is a usage error on my part. * fully initialize and finalize in event test This doesn't change the results, though. * fix events test This was complicated to figure out. For the record, testing events requires: - a private module which publicly exports the crate's event type - impl_outer_event macro to construct an enum wrapping the event types from the listed modules - system::Trait and local Trait both declare `type Event = TestEvent;` - (optional) group events with `System::<Test>::initialize(...)` and `System::<Test>::finalize()` calls. It's not entirely clear why both events show up during the initialization phase; my suspicion is that this is an artifact of not mocking a particular extrinsic, such that they end up in initialization by default. * cleanup and move crate to subdirectory this prepares us to merge this pallet into the cumulus repo * provisionally incorporate polkadot changes to hook everything together This feels like the logical next step, and compiles, at least. Still, there are some big TODOs remaining: - merge the polkadot PR upstream and reset the polkadot branch in `runtime/Cargo.toml` - in `runtime/src/validate_block/implementation.rs:116`, we should almost certainly return `Some(something)` sometime. When, precisely, and how to keep track of the appropriate data are all still open questions. * WIP: further updates to work with the polkadot implementation Hopefully we can upstream `ValidationFunctionParams` into the polkadot trait defs so we can just copy the struct out of `ValidationParams`, but no huge loss if not. This should be more or less everything required at this level. Next up: fix up `pallet-parachain-upgrade` so it reads from `VALIDATION_FUNCTION_PARAMS` to determine upgrade legality and upgrade block, and writes to `NEW_VALIDATION_CODE` when appropriate. * update pallet-parachain-upgrade appropriately to handle new expectations Implements the pallet side of the new flow. Basic tests appear to work. Next up: - make the "real blob" test work - add a bunch of additional tests of all the corners * remove test which set a real WASM blob This test didn't directly test any of the code in this pallet; it existed because we were just copying tests out of the substrate implementation. Now that we have real code of our own to test, (and because it's not compatible with the `BlockTests` abstraction,) we don't need that test anymore. Also added a `Drop` impl to `BlockTests` ensuring they get run at least once. * add test that storage gets manipulated as expected * get validate_block tests compiling again * Check validation function size against polkadot parameters Generate a user-handlable error message if it's too big, so that nothing panics later. * demonstrate that block tests run * don't actually store any magic values in parachain storage We're allowed to use it as a transport layer between validate_block and the parachain upgrade pallet, but actually editing it or, in particular, attempting to persist data there which didn't originate in the extrinsic, breaks things. This means that we can't keep the :code insertion check, because the validate_block layer doesn't know when it is legal to actually upgrade the parachain. However, the rest of the features survive, and all tests currently pass, so I'm counting it as a win. Next up: look into adding an inherent which publishes the ValidationFunctionParams struct to arbitrary pallets. * Add reference to polkadot_client to Collator This enables us to get the validation function parameters at runtime, which unblocks creating an inherent containing them. * remove unused imports * Remove VFPX; build VFP from existing data structures I almost don't want to know how long both global_validation and local_validation have existed in the produce_candidate function signature; they were precisely what I needed, without needing to add anything to the Collator struct at all. Oh well, at least I noticed it before putting the PR up for review. Next up: create a proper inherent definition for the ValidationFunctionParams. * WIP: add cumulus-validation-function-params crate Modeled on the substrate timestamp crate. It's not currently obvious to me why it is desirable to publish an entire crate for what amounts to a single const definition; going to ask about that. * refactor: get rid of validation-function-params crate Everything about the VFPs has been moved into a module of runtime * WIP: get VFP from inherent, when possible Doesn't compile for weird trait errors; probable next steps: just copy over the relevant code directly. * ensure VFPs are available during block production and validation * cleanup in preparation for review request * Copy cumulus-primitives crate from bkchr-message-broker That branch is visible as #80; this message copies the crate as of d4b2045573796955de4e5bf8f74b6c48b44c3bee. This isn't even a cherry-pick, because the commit which introduced the primitives crate also did some work which from the perspective of this PR is irrelevant. With any luck, by coping the crate directly, there won't be too many merge conflicts when the second of these open PRs is merged. * move mod validation_function_params to cumulus_primitives There is some very weird behavior going on with cargo check: every individual crate checks out fine, as verified with this invocation: for crate in $(fd Cargo.toml | xargs dirname); do if [ "$crate" == . ] || [[ "$crate" == *test* ]]; then continue; fi; name=$(toml2json "$crate/Cargo.toml" | jq -r '.package.name') if ! cargo check -p "$name" >/dev/null 2>/dev/null; then echo "failed to build $name" fi done However, `cargo check .` no longer works; it is suddenly demanding clang in order to build an indirect dependency. I'm not going to keep messing around with this anymore; it's more profitable for the moment to knock out the rest of the requested changes. Still, this behavior is weird enough that I really don't have any idea why it might be happening. * convert indentation to tabs * rename parachain upgrade pallet per PR review * use compact form for dependencies * remove pallet readme Move pertinent documentation into pallet's rustdoc. * Add weight data in compliance with updated substrate requirements The substrate API changed, so now we _have_ to invent some kind of weight data for the dispatchables. This commit does so, with the caveat that the numbers are pulled straight out of nowhere. Benchmarking remains a TODO item. * use anonymous fatal error type for brevity * Create, use a Call for setting VFPs Modeled on Timestamp; makes the ProvideInherent impl work much better. * fix pallet tests * Apply suggestions from code review Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com> * fix formatting * add license header * refactor primitive inherents / keys into appropriate modules * impl From<(GlobalValidationSchedule, LocalValidationData)> for ValidationFunctionParams * extract inherent data vfp injection into a function * collapse parachain dependency into compact form * always store vfps under same storage key * fix docs * use minimum weight for VFP inherent * rename module methods for clarity * fix tests: set_code -> schedule_upgrade * Apply pending validation function at inherent creation, not init Initialization happens before inherent creation, which means that at the time of `on_initialize`, the VFPs for the current block have not yet been loaded. This is a problem, because it means that updates would happen one block late, every time. Moving that logic into inherent creation means that we always have current information to work with. * typo: default_features -> default-features * do not panic in create_inherent * revert f741cf0f2bc; don't change behavior, but use correct spelling * move block initialization logic from inherent creation into the inherent * re-disable default features It is very difficult to come up with a coherent theory of what's going on with these default features. Builds were all broken as of 3eb1618. Renaming them in f741cf0 seemed to fix that behavior. Then they broke again locally, prompting aaee1c0. This commit restores the status quo as of f741cf0; with any luck, the build will succeed both locally and in CI. * regenerate Cargo.lock This updates several packages, but by inspection, they are all published crates from crates.io; by semver, this should not cause any behavioral changes. This also updates the lockfile format to the new format. The point of this commit is to deal with the fact that `sc-client` no longer exists. * fix checks given new dependencies Appropriate weight declarations have changed; this follows them, still using timestamp examples. Note that these weights are almost certainly wrong. * fix tests given new dependencies * add another check preventing block VFPs from contaminating validity checks * Add OnValidationFunctionParams trait so other modules can callback There isn't yet an obvious use case for other modules to get the validation function params from this one, but we may as well support it on the off chance. * Add get_validation_function_params This getter allows other modules to simply get the validation function parameters, if they have been updated for this block. Otherwise, it returns None, and they can try again later. * upgrade substrate: panic on div by 0 * Apply whitespace suggestions from code review These suggestions should make no semantic difference. Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Apply semantic from code review These changes affect the semantics of the code; I'll follow up by ensuring that everything still works. Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * add documentation to ValidationFunction type * removing panicing private fn validation_function_params() * expect validation function params to be in inherent data * move OnValidationFunctionParams to primitives * resolve weird formatting * move mod validation_function_params into its own file * add license to new file Co-authored-by: Ricardo Rius <ricardo@parity.io> Co-authored-by: Ricardo Rius <9488369+riusricardo@users.noreply.github.com> Co-authored-by: Joshy Orndorff <admin@joshyorndorff.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> |
||
|
Peter Goodspeed-Niklaus
|
a9b9ca5fa8 |
Factor out can_set_code (#5317)
* Factor out can_set_code
The motivation for this feature is parachain runtime upgrades, which
happen in two different moments: the initial call, when the checks
should be performed, and at the actual upgrade block. It's much
better and more maintainable to just call `can_set_code` in the
function than to manually copy those checks.
I'm not entirely thrilled with the interface, which requires
cloning the entire code vector in `set_code`. However, it looks
like something in `decl_module!` does not play nicely with
references in its function arguments. If `code` has a lifetime,
it _must_ be named, which gives us this signature.
```rust
pub fn can_set_code<'a>(origin, code: &'a [u8]) {
```
Unfortunately, attempting to compile with that signature generates
a very large and baffling collection of build errors, so I decided
to abandon that path for now.
* make can_set_code non-dispatchable per PR revew
Not only can we now borrow the `code` slice, but we also no longer
need to worry about people sending a `can_set_code` transaction because
of some misunderstanding.
* move into existing impl block
|
||
|
Peter Goodspeed-NiklausandBastian Köcher
|
5678c8a188 |
Automatic Example Collator (#67)
* add polkadot build script * Add scripting to bring up a simple alice-bob example net Demonstrated to produce blocks, but as of right now there's still trouble getting it to respond to external queries on its ports. * enable external rpc access to the nodes Also shrink the build context by excluding some extraneous data. * Ensure external RPC access works Also set default branch appropriately, and have the stop command clean itself up more thoroughly. * Add multi-stage dockerfile for building the cumulus-test-parachain-collator - Exclude the docker/ directory from build context because we're never going to build recursively, and this prevents spurious cache misses - build the parachain collator in three stages. The build stage is discarded; the collator stage has a wrapper script to simplify generating the right bootnodes flags, and the default stage has just the binary in a small runtime. - build_collator.sh collects appropriate build flags for the dockerfile - inject_bootnodes.sh discovers the testnet node IDs and inserts them into the arguments list for cumulus-test-parachain-collator * Add services which generate genesis state, run the collator - Ignore the scripts directory to reduce spurious cache misses. - Move inject_bootnodes.sh from the scripts directory into the root: It can't stay in the scripts directory, because that's ignored; I didn't want to invent _another_ top-level subdirectory for it. That decision could certainly be appealed, though. - Move docker-compose.yml, add dc.sh, modify *_collator.sh: by taking docker-compose.yml out of the root directory, we can further reduce cache misses. However, docker-compose normally has a strong expectation that docker-compose.yml exist in the project root; it takes a moderately complicated invocation to override that expectation. That override is encoded in dc.sh; the updates to the other scripts are just to use the override. The expectation as of now is that scripts/run_collator.sh runs both chain nodes and the collator, generates the genesis state into a volume with a transient container, and runs the collator as specified in the repo README. Upcoming work: Steps 5 and 6 from the readme. * Launch the collator node The biggest change here is adding the testing_net network to the collator node's networks list. This lets it successfully connect to the alice and bob nodes, which in turn lets it get their node IDs, which was the blocker for a long time. Remove httpie in favor of curl: makes for a smaller docker image, and has fewer weird failure modes within docker. Unfortunately this doesn't yet actually connect to the relay chain nodes; that's the next area to figure out. * enable external websocket access to indexer nodes * Reorganize for improved caching, again - Manually enumerate the set of source directories to copy when building. This bloats the cache a bit, but means that rebuilds on script changes don't bust that cache, which saves a _lot_ of time. - Un-.dockerignore the scripts directory; it's small and will no longer trigger cache misses. - Move inject_bootnodes.sh back into scripts directory for better organization. - inject_bootnodes.sh: use rpc port for rpc call and p2p port for generating the bootnode string. I'm not 100% sure this is correct, but upwards of 80% at least. - docker-compose.yml: reorganize the launch commands such that alice and bob still present the same external port mapping to the world, but within the docker-compose network, they both use the same (standard) p2p, rpc, and websocket ports. This makes life easier for inject_bootnodes.sh The collator node still doesn't actually connect, but I think this commit still represents real progress in that direction. * Get the collator talking to the indexer nodes In the end, it was four characters: -- and two = signs in the launch arguments. They turn out to be critical characters for correct operation, though! Next up: automating step 5. * Add runtime stage to collect runtime wasm blob into volume We can't just copy the blob in the builder stage because the volumes aren't available at that point. Rewrite build_collator.sh into build_docker.sh and update for generality. * WIP: add registrar service and partial work to actually register the collator This is likely to be discarded; the Python library in use is 3rd party and not well documented, while the official polkadot-js repo has a CLI tool: https://github.com/polkadot-js/tools/tree/master/packages/api-cli * Add a parachain registrar which should properly register the parachain Doesn't work at the moment because it depends on two api-cli features which I added today, which have not yet made it out into a published release. Next up: figure out how to add the `api-cli` at its `master` branch, then run tests to ensure the collator is producing blocks. Then, automate the block production tests. * BROKEN attempt to demo registrar communication with the blockchain This is a really weird bug. After running `scripts/run_collector.sh`, which brings everything up, it's perfectly possible to get into a state very much like what the registrar is in, and communicate with the blockchain without issue: ```sh $ docker run --rm --net cumulus_testing_net para-reg:latest polkadot-js-api --ws ws://172.28.1.1:9944 query.sudo.key Thu 20 Feb 2020 12:19:20 PM CET { "key": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY" } ``` However, the registrar itself, doing the same thing from within `register_para.sh`, is failing to find the right place in the network: ``` /runtime/cumulus_test_parachain_runtime.compact.wasm found after 0 seconds /genesis/genesis-state found after 0 seconds 2020-02-20 10:43:22 API-WS: disconnected from ws://172.28.1.1:9944 code: '1006' reason: 'connection failed' _Event { type: 'error', isTrusted: false, _yaeti: true, target: W3CWebSocket { _listeners: {}, addEventListener: [Function: _addEventListener], removeEventListener: [Function: _removeEventListener], dispatchEvent: [Function: _dispatchEvent], _url: 'ws://172.28.1.1:9944', _readyState: 3, _protocol: undefined, _extensions: '', _bufferedAmount: 0, _binaryType: 'arraybuffer', _connection: undefined, _client: WebSocketClient { _events: [Object: null prototype] {}, _eventsCount: 0, _maxListeners: undefined, config: [Object], _req: null, protocols: [], origin: undefined, url: [Url], secure: false, base64nonce: 'aJ6J3pYDz8l5owVWHGbzHg==', [Symbol(kCapture)]: false }, onclose: [Function (anonymous)], onerror: [Function (anonymous)], onmessage: [Function (anonymous)], onopen: [Function (anonymous)] }, cancelable: true, stopImmediatePropagation: [Function (anonymous)] } ``` They should be connected to the same network, running the same image, doing the same call. The only difference is the file existence checks, which really shouldn't be affecting the network state at all. Pushing this commit to ask for outside opinions on it, because this is very weird and I clearly don't understand some part of what's happening. * Fix broken parachain registrar The problem was that the registrar container was coming up too fast, so the Alice node wasn't yet ready to receive connections. Using a well-known wait script fixes the issue. Next up: verify that the collator is in fact building blocks. * fixes which cause the collator to correctly produce new parachain blocks It didn't take much! The biggest issue was that the genesis state was previously being double-encoded. * add documentation for running the parachain automatically * Add health check to collator * minor scripting improvements * Apply suggestions from code review Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com> * Docker: copy the whole workspace in one go Pro: future-proofing against the time we add or remove a directory Con: changing any file in the workspace busts Rust's build cache, which takes a long time. Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> |