mirror of
https://github.com/pezkuwichain/pezkuwi-subxt.git
synced 2026-06-19 04:11:09 +00:00
c42023269afaba15b41e7dc353fad78ad74c83d0
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dcompoze
|
002d9260f9 |
Fix spelling mistakes across the whole repository (#3808)
**Update:** Pushed additional changes based on the review comments. **This pull request fixes various spelling mistakes in this repository.** Most of the changes are contained in the first **3** commits: - `Fix spelling mistakes in comments and docs` - `Fix spelling mistakes in test names` - `Fix spelling mistakes in error messages, panic messages, logs and tracing` Other source code spelling mistakes are separated into individual commits for easier reviewing: - `Fix the spelling of 'authority'` - `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'` - `Fix the spelling of 'prev_enqueud_messages'` - `Fix the spelling of 'endpoint'` - `Fix the spelling of 'children'` - `Fix the spelling of 'PenpalSiblingSovereignAccount'` - `Fix the spelling of 'PenpalSudoAccount'` - `Fix the spelling of 'insufficient'` - `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'` - `Fix the spelling of 'subtracted'` - `Fix the spelling of 'CandidatePendingAvailability'` - `Fix the spelling of 'exclusive'` - `Fix the spelling of 'until'` - `Fix the spelling of 'discriminator'` - `Fix the spelling of 'nonexistent'` - `Fix the spelling of 'subsystem'` - `Fix the spelling of 'indices'` - `Fix the spelling of 'committed'` - `Fix the spelling of 'topology'` - `Fix the spelling of 'response'` - `Fix the spelling of 'beneficiary'` - `Fix the spelling of 'formatted'` - `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'` - `Fix the spelling of 'succeeded'` - `Fix the spelling of 'reopened'` - `Fix the spelling of 'proposer'` - `Fix the spelling of 'InstantiationNonce'` - `Fix the spelling of 'depositor'` - `Fix the spelling of 'expiration'` - `Fix the spelling of 'phantom'` - `Fix the spelling of 'AggregatedKeyValue'` - `Fix the spelling of 'randomness'` - `Fix the spelling of 'defendant'` - `Fix the spelling of 'AquaticMammal'` - `Fix the spelling of 'transactions'` - `Fix the spelling of 'PassingTracingSubscriber'` - `Fix the spelling of 'TxSignaturePayload'` - `Fix the spelling of 'versioning'` - `Fix the spelling of 'descendant'` - `Fix the spelling of 'overridden'` - `Fix the spelling of 'network'` Let me know if this structure is adequate. **Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`, `Merkelization`, `Merkleization`, is somewhat inconsistent but I left it as it is. ~~**Note:** In some places the term `Receival` is used to refer to message reception, IMO `Reception` is the correct word here, but I left it as it is.~~ ~~**Note:** In some places the term `Overlayed` is used instead of the more acceptable version `Overlaid` but I also left it as it is.~~ ~~**Note:** In some places the term `Applyable` is used instead of the correct version `Applicable` but I also left it as it is.~~ **Note:** Some usage of British vs American english e.g. `judgement` vs `judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc. are both present in different places, but I suppose that's understandable given the number of contributors. ~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it triggers errors in CI when I make changes to it, so I left it as it is.~~ |
||
|
Alexandru Gheorghe
|
f9f886886b |
Introduce approval-voting/distribution benchmark (#2621)
## Summary Built on top of the tooling and ideas introduced in https://github.com/paritytech/polkadot-sdk/pull/2528, this PR introduces a synthetic benchmark for measuring and assessing the performance characteristics of the approval-voting and approval-distribution subsystems. Currently this allows, us to simulate the behaviours of these systems based on the following dimensions: ``` TestConfiguration: # Test 1 - objective: !ApprovalsTest last_considered_tranche: 89 min_coalesce: 1 max_coalesce: 6 enable_assignments_v2: true send_till_tranche: 60 stop_when_approved: false coalesce_tranche_diff: 12 workdir_prefix: "/tmp" num_no_shows_per_candidate: 0 approval_distribution_expected_tof: 6.0 approval_distribution_cpu_ms: 3.0 approval_voting_cpu_ms: 4.30 n_validators: 500 n_cores: 100 n_included_candidates: 100 min_pov_size: 1120 max_pov_size: 5120 peer_bandwidth: 524288000000 bandwidth: 524288000000 latency: min_latency: secs: 0 nanos: 1000000 max_latency: secs: 0 nanos: 100000000 error: 0 num_blocks: 10 ``` ## The approach 1. We build a real overseer with the real implementations for approval-voting and approval-distribution subsystems. 2. For a given network size, for each validator we pre-computed all potential assignments and approvals it would send, because this a computation heavy operation this will be cached on a file on disk and be re-used if the generation parameters don't change. 3. The messages will be sent accordingly to the configured parameters and those are split into 3 main benchmarking scenarios. ## Benchmarking scenarios ### Best case scenario *approvals_throughput_best_case.yaml* It send to the approval-distribution only the minimum required tranche to gathered the needed_approvals, so that a candidate is approved. ### Behaviour in the presence of no-shows *approvals_no_shows.yaml* It sends the tranche needed to approve a candidate when we have a maximum of *num_no_shows_per_candidate* tranches with no-shows for each candidate. ### Maximum throughput *approvals_throughput.yaml* It sends all the tranches for each block and measures the used CPU and necessary network bandwidth. by the approval-voting and approval-distribution subsystem. ## How to run it ``` cargo run -p polkadot-subsystem-bench --release -- test-sequence --path polkadot/node/subsystem-bench/examples/approvals_throughput.yaml ``` ## Evaluating performance ### Use the real subsystems metrics If you follow the steps in https://github.com/paritytech/polkadot-sdk/tree/master/polkadot/node/subsystem-bench#install-grafana for installing locally prometheus and grafana, all real metrics for the `approval-distribution`, `approval-voting` and overseer are available. E.g: <img width="2149" alt="Screenshot 2023-12-05 at 11 07 46" src="https://github.com/paritytech/polkadot-sdk/assets/49718502/cb8ae2dd-178b-4922-bfa4-dc37e572ed38"> <img width="2551" alt="Screenshot 2023-12-05 at 11 09 42" src="https://github.com/paritytech/polkadot-sdk/assets/49718502/8b4542ba-88b9-46f9-9b70-cc345366081b"> <img width="2154" alt="Screenshot 2023-12-05 at 11 10 15" src="https://github.com/paritytech/polkadot-sdk/assets/49718502/b8874d8d-632e-443a-9840-14ad8e90c54f"> <img width="2535" alt="Screenshot 2023-12-05 at 11 10 52" src="https://github.com/paritytech/polkadot-sdk/assets/49718502/779a439f-fd18-4985-bb80-85d5afad78e2"> ### Profile with pyroscope 1. Setup pyroscope following the steps in https://github.com/paritytech/polkadot-sdk/tree/master/polkadot/node/subsystem-bench#install-pyroscope, then run any of the benchmark scenario with `--profile` as the arguments. 2. Open the pyroscope dashboard in grafana, e.g: <img width="2544" alt="Screenshot 2024-01-09 at 17 09 58" src="https://github.com/paritytech/polkadot-sdk/assets/49718502/58f50c99-a910-4d20-951a-8b16639303d9"> ### Useful logs 1. Network bandwidth requirements: ``` Payload bytes received from peers: 503993 KiB total, 50399 KiB/block Payload bytes sent to peers: 629971 KiB total, 62997 KiB/block ``` 2. Cpu usage by the approval-distribution/approval-voting subsystems. ``` approval-distribution CPU usage 84.061s approval-distribution CPU usage per block 8.406s approval-voting CPU usage 96.532s approval-voting CPU usage per block 9.653s ``` 3. Time passed until a given block is approved ``` Chain selection approved after 3500 ms hash=0x0101010101010101010101010101010101010101010101010101010101010101 Chain selection approved after 4500 ms hash=0x0202020202020202020202020202020202020202020202020202020202020202 ``` ### Using benchmark to quantify improvements from https://github.com/paritytech/polkadot-sdk/pull/1178 + https://github.com/paritytech/polkadot-sdk/pull/1191 Using a versi-node we compare the scenarios where all new optimisations are disabled with a scenarios where tranche0 assignments are sent in a single message and a conservative simulation where the coalescing of approvals gives us just 50% reduction in the number of messages we send. Overall, what we see is a speedup of around 30-40% in the time it takes to process the necessary messages and a 30-40% reduction in the necessary bandwidth. #### Best case scenario comparison(minimum required tranches sent). Unoptimised ``` Number of blocks: 10 Payload bytes received from peers: 53289 KiB total, 5328 KiB/block Payload bytes sent to peers: 52489 KiB total, 5248 KiB/block approval-distribution CPU usage 6.732s approval-distribution CPU usage per block 0.673s approval-voting CPU usage 9.523s approval-voting CPU usage per block 0.952s ``` vs Optimisation enabled ``` Number of blocks: 10 Payload bytes received from peers: 32141 KiB total, 3214 KiB/block Payload bytes sent to peers: 37314 KiB total, 3731 KiB/block approval-distribution CPU usage 4.658s approval-distribution CPU usage per block 0.466s approval-voting CPU usage 6.236s approval-voting CPU usage per block 0.624s ``` #### Worst case all tranches sent, very unlikely happens when sharding breaks. Unoptimised ``` Number of blocks: 10 Payload bytes received from peers: 746393 KiB total, 74639 KiB/block Payload bytes sent to peers: 729151 KiB total, 72915 KiB/block approval-distribution CPU usage 118.681s approval-distribution CPU usage per block 11.868s approval-voting CPU usage 124.118s approval-voting CPU usage per block 12.412s ``` vs optimised ``` Number of blocks: 10 Payload bytes received from peers: 503993 KiB total, 50399 KiB/block Payload bytes sent to peers: 629971 KiB total, 62997 KiB/block approval-distribution CPU usage 84.061s approval-distribution CPU usage per block 8.406s approval-voting CPU usage 96.532s approval-voting CPU usage per block 9.653s ``` ## TODOs [x] Polish implementation. [x] Use what we have so far to evaluate https://github.com/paritytech/polkadot-sdk/pull/1191 before merging. [x] List of features and additional dimensions we want to use for benchmarking. [x] Run benchmark on hardware similar with versi and kusama nodes. [ ] Add benchmark to be run in CI for catching regression in performance. [ ] Rebase on latest changes for network emulation. --------- Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Co-authored-by: Andrei Sandu <andrei-mihail@parity.io> Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> |
||
|
Alexandru Gheorghe
|
a84dd0dba5 |
Approve multiple candidates with a single signature (#1191)
Initial implementation for the plan discussed here: https://github.com/paritytech/polkadot-sdk/issues/701 Built on top of https://github.com/paritytech/polkadot-sdk/pull/1178 v0: https://github.com/paritytech/polkadot/pull/7554, ## Overall idea When approval-voting checks a candidate and is ready to advertise the approval, defer it in a per-relay chain block until we either have MAX_APPROVAL_COALESCE_COUNT candidates to sign or a candidate has stayed MAX_APPROVALS_COALESCE_TICKS in the queue, in both cases we sign what candidates we have available. This should allow us to reduce the number of approvals messages we have to create/send/verify. The parameters are configurable, so we should find some values that balance: - Security of the network: Delaying broadcasting of an approval shouldn't but the finality at risk and to make sure that never happens we won't delay sending a vote if we are past 2/3 from the no-show time. - Scalability of the network: MAX_APPROVAL_COALESCE_COUNT = 1 & MAX_APPROVALS_COALESCE_TICKS =0, is what we have now and we know from the measurements we did on versi, it bottlenecks approval-distribution/approval-voting when increase significantly the number of validators and parachains - Block storage: In case of disputes we have to import this votes on chain and that increase the necessary storage with MAX_APPROVAL_COALESCE_COUNT * CandidateHash per vote. Given that disputes are not the normal way of the network functioning and we will limit MAX_APPROVAL_COALESCE_COUNT in the single digits numbers, this should be good enough. Alternatively, we could try to create a better way to store this on-chain through indirection, if that's needed. ## Other fixes: - Fixed the fact that we were sending random assignments to non-validators, that was wrong because those won't do anything with it and they won't gossip it either because they do not have a grid topology set, so we would waste the random assignments. - Added metrics to be able to debug potential no-shows and mis-processing of approvals/assignments. ## TODO: - [x] Get feedback, that this is moving in the right direction. @ordian @sandreim @eskimor @burdges, let me know what you think. - [x] More and more testing. - [x] Test in versi. - [x] Make MAX_APPROVAL_COALESCE_COUNT & MAX_APPROVAL_COALESCE_WAIT_MILLIS a parachain host configuration. - [x] Make sure the backwards compatibility works correctly - [x] Make sure this direction is compatible with other streams of work: https://github.com/paritytech/polkadot-sdk/issues/635 & https://github.com/paritytech/polkadot-sdk/issues/742 - [x] Final versi burn-in before merging --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> |
||
|
Andrei Sandu
|
0570b6fa9e |
approval-voting improvement: include all tranche0 assignments in one certificate (#1178)
**_PR migrated from https://github.com/paritytech/polkadot/pull/6782_** This PR will upgrade the network protocol to version 3 -> VStaging which will later be renamed to V3. This version introduces a new kind of assignment certificate that will be used for tranche0 assignments. Instead of issuing/importing one tranche0 assignment per candidate, there will be just one certificate per relay chain block per validator. However, we will not be sending out the new assignment certificates, yet. So everything should work exactly as before. Once the majority of the validators have been upgraded to the new protocol version we will enable the new certificates (starting at a specific relay chain block) with a new client update. There are still a few things that need to be done: - [x] Use bitfield instead of Vec<CandidateIndex>: https://github.com/paritytech/polkadot/pull/6802 - [x] Fix existing approval-distribution and approval-voting tests - [x] Fix bitfield-distribution and statement-distribution tests - [x] Fix network bridge tests - [x] Implement todos in the code - [x] Add tests to cover new code - [x] Update metrics - [x] Remove the approval distribution aggression levels: TBD PR - [x] Parachains DB migration - [x] Test network protocol upgrade on Versi - [x] Versi Load test - [x] Add Zombienet test - [x] Documentation updates - [x] Fix for sending DistributeAssignment for each candidate claimed by a v2 assignment (warning: Importing locally an already known assignment) - [x] Fix AcceptedDuplicate - [x] Fix DB migration so that we can still keep old data. - [x] Final Versi burn in --------- Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Co-authored-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> |
||
|
s0me0ne-unkn0wn
|
64660ee8d2 |
Remove years from copyright notes (#7034)
* Happy New Year! * Remove year entierly Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove years from copyright notice in the entire repo --------- Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> |
||
|
Shawn Tabrizi
|
ff5d56fb76 |
cargo +nightly fmt (#3540)
* cargo +nightly fmt * add cargo-fmt check to ci * update ci * fmt * fmt * skip macro * ignore bridges |
||
|
Denis Pisarev
|
fc253e6e4d |
WIP: CI: add spellcheck (#3421)
* CI: add spellcheck * revert me * CI: explicit command for spellchecker * spellcheck: edit misspells * CI: run spellcheck on diff * spellcheck: edits * spellcheck: edit misspells * spellcheck: add rules * spellcheck: mv configs * spellcheck: more edits * spellcheck: chore * spellcheck: one more thing * spellcheck: and another one * spellcheck: seems like it doesn't get to an end * spellcheck: new words after rebase * spellcheck: new words appearing out of nowhere * chore * review edits * more review edits * more edits * wonky behavior * wonky behavior 2 * wonky behavior 3 * change git behavior * spellcheck: another bunch of new edits * spellcheck: new words are koming out of nowhere * CI: finding the master * CI: fetching master implicitly * CI: undebug * new errors * a bunch of new edits * and some more * Update node/core/approval-voting/src/approval_db/v1/mod.rs Co-authored-by: Andronik Ordian <write@reusable.software> * Update xcm/xcm-executor/src/assets.rs Co-authored-by: Andronik Ordian <write@reusable.software> * Apply suggestions from code review Co-authored-by: Andronik Ordian <write@reusable.software> * Suggestions from the code review * CI: scan only changed files Co-authored-by: Andronik Ordian <write@reusable.software> |
||
|
Robert Habermeier
|
e48c687504 |
Implement Approval Voting Subsystem (#2112)
* skeleton * skeleton aux-schema module * start approval types * start aux schema with aux store * doc * finish basic types * start approval types * doc * finish basic types * write out schema types * add debug and codec impls to approval types * add debug and codec impls to approval types also add some key computation * add debug and codec impls to approval types * getters for block and candidate entries * grumbles * remove unused AssignmentId * load_decode utility * implement DB clearing * function for adding new block entry to aux store * start `canonicalize` implementation * more skeleton * finish implementing canonicalize * tag TODO * implement a test AuxStore * add allow(unused) * basic loading and deleting test * block_entry test function * add a test for `add_block_entry` * ensure range is exclusive at end * test clear() * test that add_block sets children * add a test for canonicalize * extract Pre-digest from header * utilities for extracting RelayVRFStory from the header-chain * add approval voting message types * approval distribution message type * subsystem skeleton * state struct * add futures-timer * prepare service for babe slot duration * more skeleton * better integrate AuxStore * RelayVRF -> RelayVRFStory * canonicalize * implement some tick functionality * guide: tweaks * check_approval * more tweaks and helpers * guide: add core index to candidate event * primitives: add core index to candidate event * runtime: add core index to candidate events * head handling (session window) * implement `determine_new_blocks` * add TODO * change error type on functions * compute RelayVRFModulo assignments * compute RelayVRFDelay assignments * fix delay tranche calc * assignment checking * pluralize * some dummy code for fetching assignments * guide: add babe epoch runtime API * implement a current_epoch() runtime API * compute assignments * candidate events get backing group * import blocks and assignments into DB * push block approval meta * add message types, no overseer integration yet * notify approval distribution of new blocks * refactor import into separate functions * impl tranches_to_approve * guide: improve function signatures * guide: remove Tick from ApprovalEntry * trigger and broadcast assignment * most of approval launching * remove byteorder crate * load blocks back to finality, except on startup * check unchecked assignments * add claimed core to approval voting message * fix checks * assign only to backing group * remove import_checked_assignment from guide * newline * import assignments * abstract out a bit * check and import approvals * check full approvals from assignment import too * comment * create a Transaction utility * must_use * use transaction in `check_full_approvals` * wire up wakeups * add Ord to CandidateHash * wakeup refactoring * return candidate info from add_block_entry * schedule wakeups * background task: do candidate validation * forward candidate validation requests * issue approval votes when requested * clean up a couple TODOs * fix up session caching * clean up last unimplemented!() items * fix remaining warnings * remove TODO * implement handle_approved_ancestor * update Cargo.lock * fix runtime API tests * guide: cleanup assignment checking * use claimed candidate index instead of core * extract time to a trait * tests module * write a mock clock for testing * allow swapping out the clock * make abstract over assignment criteria * add some skeleton tests and simplify params * fix backing group check * do backing group check inside check_assignment_cert * write some empty test functions to implement * add a test for non-backing * test that produced checks pass * some empty test ideas * runtime/inclusion: remove outdated TODO * fix compilation * av-store: fix tests * dummy cert * criteria tests * move `TestStore` to main tests file * fix unused warning * test harness beginnings * resolve slots renaming fallout * more compilation fixes * wip: extract pure data into a separate module * wip: extract pure data into a separate module * move types completely to v1 * add persisted_entries * add conversion trait impls * clean up some warnings * extract import logic to own module * schedule wakeups * experiment with Actions * uncomment approval-checking * separate module for approval checking utilities * port more code to use actions * get approval pipeline using actions * all logic is uncommented * main loop processes actions * all loop logic uncommented * separate function for handling actions * remove last unimplemented item * clean up warnings * State gives read-only access to underlying DB * tests for approval checking * tests for approval criteria * skeleton test module for import * list of import tests to do * some test glue code * test reject bad assignment * test slot too far in future * test reject assignment with unknown candidate * remove loads_blocks tests * determine_new_blocks back to finalized & harness * more coverage for determining new blocks * make `imported_block_info` have less reliance on State * candidate_info tests * tests for session caching * remove println * extricate DB and main TestStores * rewrite approval checking logic to counteract early delays * move state out of function * update approval-checking tests * tweak wakeups & scheduling logic * rename check_full_approvals * test that assignment import updates candidate * some approval import tests * some tests for check_and_apply_approval * add 'full' qualifier to avoid confusion * extract should-trigger logic to separate function * some tests for all triggering * tests for when we trigger assignments * test wakeups * add block utilities for testing * some more tests for approval updates * approved_ancestor tests * new action type for launch approval * process-wakeup tests * clean up some warnings * fix in_future test * approval checking tests * tighten up too-far-in-future * special-case genesis when caching sessions * fix bitfield len Co-authored-by: Andronik Ordian <write@reusable.software> |