mirror of
https://github.com/pezkuwichain/pezkuwi-subxt.git
synced 2026-04-26 04:07:57 +00:00
2aa006e094e248110af14a742d4e2f56b7931959
1825 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 André Silva
|
30c32e3d84 |
move substrate-bip39 into polkadot-sdk (#3579)
Moves [substrate-bip39](https://github.com/paritytech/substrate-bip39) into substrate. All git history is preserved. Dependencies have been updated to use the same version as the rest of the repo. Fixes https://github.com/paritytech/polkadot-sdk/issues/1934. --------- Co-authored-by: Maciej Hirsz <maciej.hirsz@gmail.com> Co-authored-by: Maciej Hirsz <1096222+maciejhirsz@users.noreply.github.com> Co-authored-by: Gav Wood <i@gavwood.com> Co-authored-by: Stanislav Tkach <stanislav.tkach@gmail.com> Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com> Co-authored-by: Demi M. Obenour <demi@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: NikVolf <nikvolf@gmail.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Benjamin Kampmann <ben@gnunicorn.org> Co-authored-by: Maciej Hirsz <hello@maciej.codes> Co-authored-by: cheme <emericchevalier.pro@gmail.com> Co-authored-by: adoerr <0xad@gmx.net> Co-authored-by: Jun Jiang <jasl9187@hotmail.com> Co-authored-by: Dan Shields <35669742+NukeManDan@users.noreply.github.com> Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com> |
||
|
Gavin Wood
|
fd5f9292f5 |
FRAME: Create TransactionExtension as a replacement for SignedExtension (#2280)
Closes #2160 First part of [Extrinsic Horizon](https://github.com/paritytech/polkadot-sdk/issues/2415) Introduces a new trait `TransactionExtension` to replace `SignedExtension`. Introduce the idea of transactions which obey the runtime's extensions and have according Extension data (né Extra data) yet do not have hard-coded signatures. Deprecate the terminology of "Unsigned" when used for transactions/extrinsics owing to there now being "proper" unsigned transactions which obey the extension framework and "old-style" unsigned which do not. Instead we have __*General*__ for the former and __*Bare*__ for the latter. (Ultimately, the latter will be phased out as a type of transaction, and Bare will only be used for Inherents.) Types of extrinsic are now therefore: - Bare (no hardcoded signature, no Extra data; used to be known as "Unsigned") - Bare transactions (deprecated): Gossiped, validated with `ValidateUnsigned` (deprecated) and the `_bare_compat` bits of `TransactionExtension` (deprecated). - Inherents: Not gossiped, validated with `ProvideInherent`. - Extended (Extra data): Gossiped, validated via `TransactionExtension`. - Signed transactions (with a hardcoded signature). - General transactions (without a hardcoded signature). `TransactionExtension` differs from `SignedExtension` because: - A signature on the underlying transaction may validly not be present. - It may alter the origin during validation. - `pre_dispatch` is renamed to `prepare` and need not contain the checks present in `validate`. - `validate` and `prepare` is passed an `Origin` rather than a `AccountId`. - `validate` may pass arbitrary information into `prepare` via a new user-specifiable type `Val`. - `AdditionalSigned`/`additional_signed` is renamed to `Implicit`/`implicit`. It is encoded *for the entire transaction* and passed in to each extension as a new argument to `validate`. This facilitates the ability of extensions to acts as underlying crypto. There is a new `DispatchTransaction` trait which contains only default function impls and is impl'ed for any `TransactionExtension` impler. It provides several utility functions which reduce some of the tedium from using `TransactionExtension` (indeed, none of its regular functions should now need to be called directly). Three transaction version discriminator ("versions") are now permissible: - 0b000000100: Bare (used to be called "Unsigned"): contains Signature or Extra (extension data). After bare transactions are no longer supported, this will strictly identify an Inherents only. - 0b100000100: Old-school "Signed" Transaction: contains Signature and Extra (extension data). - 0b010000100: New-school "General" Transaction: contains Extra (extension data), but no Signature. For the New-school General Transaction, it becomes trivial for authors to publish extensions to the mechanism for authorizing an Origin, e.g. through new kinds of key-signing schemes, ZK proofs, pallet state, mutations over pre-authenticated origins or any combination of the above. ## Code Migration ### NOW: Getting it to build Wrap your `SignedExtension`s in `AsTransactionExtension`. This should be accompanied by renaming your aggregate type in line with the new terminology. E.g. Before: ```rust /// The SignedExtension to the basic transaction logic. pub type SignedExtra = ( /* snip */ MySpecialSignedExtension, ); /// Unchecked extrinsic type as expected by this runtime. pub type UncheckedExtrinsic = generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, SignedExtra>; ``` After: ```rust /// The extension to the basic transaction logic. pub type TxExtension = ( /* snip */ AsTransactionExtension<MySpecialSignedExtension>, ); /// Unchecked extrinsic type as expected by this runtime. pub type UncheckedExtrinsic = generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, TxExtension>; ``` You'll also need to alter any transaction building logic to add a `.into()` to make the conversion happen. E.g. Before: ```rust fn construct_extrinsic( /* snip */ ) -> UncheckedExtrinsic { let extra: SignedExtra = ( /* snip */ MySpecialSignedExtension::new(/* snip */), ); let payload = SignedPayload::new(call.clone(), extra.clone()).unwrap(); let signature = payload.using_encoded(|e| sender.sign(e)); UncheckedExtrinsic::new_signed( /* snip */ Signature::Sr25519(signature), extra, ) } ``` After: ```rust fn construct_extrinsic( /* snip */ ) -> UncheckedExtrinsic { let tx_ext: TxExtension = ( /* snip */ MySpecialSignedExtension::new(/* snip */).into(), ); let payload = SignedPayload::new(call.clone(), tx_ext.clone()).unwrap(); let signature = payload.using_encoded(|e| sender.sign(e)); UncheckedExtrinsic::new_signed( /* snip */ Signature::Sr25519(signature), tx_ext, ) } ``` ### SOON: Migrating to `TransactionExtension` Most `SignedExtension`s can be trivially converted to become a `TransactionExtension`. There are a few things to know. - Instead of a single trait like `SignedExtension`, you should now implement two traits individually: `TransactionExtensionBase` and `TransactionExtension`. - Weights are now a thing and must be provided via the new function `fn weight`. #### `TransactionExtensionBase` This trait takes care of anything which is not dependent on types specific to your runtime, most notably `Call`. - `AdditionalSigned`/`additional_signed` is renamed to `Implicit`/`implicit`. - Weight must be returned by implementing the `weight` function. If your extension is associated with a pallet, you'll probably want to do this via the pallet's existing benchmarking infrastructure. #### `TransactionExtension` Generally: - `pre_dispatch` is now `prepare` and you *should not reexecute the `validate` functionality in there*! - You don't get an account ID any more; you get an origin instead. If you need to presume an account ID, then you can use the trait function `AsSystemOriginSigner::as_system_origin_signer`. - You get an additional ticket, similar to `Pre`, called `Val`. This defines data which is passed from `validate` into `prepare`. This is important since you should not be duplicating logic from `validate` to `prepare`, you need a way of passing your working from the former into the latter. This is it. - This trait takes two type parameters: `Call` and `Context`. `Call` is the runtime call type which used to be an associated type; you can just move it to become a type parameter for your trait impl. `Context` is not currently used and you can safely implement over it as an unbounded type. - There's no `AccountId` associated type any more. Just remove it. Regarding `validate`: - You get three new parameters in `validate`; all can be ignored when migrating from `SignedExtension`. - `validate` returns a tuple on success; the second item in the tuple is the new ticket type `Self::Val` which gets passed in to `prepare`. If you use any information extracted during `validate` (off-chain and on-chain, non-mutating) in `prepare` (on-chain, mutating) then you can pass it through with this. For the tuple's last item, just return the `origin` argument. Regarding `prepare`: - This is renamed from `pre_dispatch`, but there is one change: - FUNCTIONALITY TO VALIDATE THE TRANSACTION NEED NOT BE DUPLICATED FROM `validate`!! - (This is different to `SignedExtension` which was required to run the same checks in `pre_dispatch` as in `validate`.) Regarding `post_dispatch`: - Since there are no unsigned transactions handled by `TransactionExtension`, `Pre` is always defined, so the first parameter is `Self::Pre` rather than `Option<Self::Pre>`. If you make use of `SignedExtension::validate_unsigned` or `SignedExtension::pre_dispatch_unsigned`, then: - Just use the regular versions of these functions instead. - Have your logic execute in the case that the `origin` is `None`. - Ensure your transaction creation logic creates a General Transaction rather than a Bare Transaction; this means having to include all `TransactionExtension`s' data. - `ValidateUnsigned` can still be used (for now) if you need to be able to construct transactions which contain none of the extension data, however these will be phased out in stage 2 of the Transactions Horizon, so you should consider moving to an extension-centric design. ## TODO - [x] Introduce `CheckSignature` impl of `TransactionExtension` to ensure it's possible to have crypto be done wholly in a `TransactionExtension`. - [x] Deprecate `SignedExtension` and move all uses in codebase to `TransactionExtension`. - [x] `ChargeTransactionPayment` - [x] `DummyExtension` - [x] `ChargeAssetTxPayment` (asset-tx-payment) - [x] `ChargeAssetTxPayment` (asset-conversion-tx-payment) - [x] `CheckWeight` - [x] `CheckTxVersion` - [x] `CheckSpecVersion` - [x] `CheckNonce` - [x] `CheckNonZeroSender` - [x] `CheckMortality` - [x] `CheckGenesis` - [x] `CheckOnlySudoAccount` - [x] `WatchDummy` - [x] `PrevalidateAttests` - [x] `GenericSignedExtension` - [x] `SignedExtension` (chain-polkadot-bulletin) - [x] `RefundSignedExtensionAdapter` - [x] Implement `fn weight` across the board. - [ ] Go through all pre-existing extensions which assume an account signer and explicitly handle the possibility of another kind of origin. - [x] `CheckNonce` should probably succeed in the case of a non-account origin. - [x] `CheckNonZeroSender` should succeed in the case of a non-account origin. - [x] `ChargeTransactionPayment` and family should fail in the case of a non-account origin. - [ ] - [x] Fix any broken tests. --------- Signed-off-by: georgepisaltu <george.pisaltu@parity.io> Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> Co-authored-by: Nikhil Gupta <17176722+gupnik@users.noreply.github.com> Co-authored-by: georgepisaltu <52418509+georgepisaltu@users.noreply.github.com> Co-authored-by: Chevdor <chevdor@users.noreply.github.com> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Maciej <maciej.zyszkiewicz@parity.io> Co-authored-by: Javier Viola <javier@parity.io> Co-authored-by: Marcin S. <marcin@realemail.net> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io> Co-authored-by: Javier Bullrich <javier@bullrich.dev> Co-authored-by: Koute <koute@users.noreply.github.com> Co-authored-by: Adrian Catangiu <adrian@parity.io> Co-authored-by: Vladimir Istyufeev <vladimir@parity.io> Co-authored-by: Ross Bulat <ross@parity.io> Co-authored-by: Gonçalo Pestana <g6pestana@gmail.com> Co-authored-by: Liam Aharon <liam.aharon@hotmail.com> Co-authored-by: Svyatoslav Nikolsky <svyatonik@gmail.com> Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com> Co-authored-by: ordian <write@reusable.software> Co-authored-by: Sebastian Kunert <skunert49@gmail.com> Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com> Co-authored-by: Dmitry Markin <dmitry@markin.tech> Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com> Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com> Co-authored-by: Julian Eager <eagr@tutanota.com> Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com> Co-authored-by: Davide Galassi <davxy@datawok.net> Co-authored-by: Dónal Murray <donal.murray@parity.io> Co-authored-by: yjh <yjh465402634@gmail.com> Co-authored-by: Tom Mi <tommi@niemi.lol> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Will | Paradox | ParaNodes.io <79228812+paradox-tt@users.noreply.github.com> Co-authored-by: Bastian Köcher <info@kchr.de> Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com> Co-authored-by: Joshy Orndorff <git-user-email.h0ly5@simplelogin.com> Co-authored-by: PG Herveou <pgherveou@gmail.com> Co-authored-by: Alexander Theißen <alex.theissen@me.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Juan Girini <juangirini@gmail.com> Co-authored-by: bader y <ibnbassem@gmail.com> Co-authored-by: James Wilson <james@jsdw.me> Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> Co-authored-by: asynchronous rob <rphmeier@gmail.com> Co-authored-by: Parth <desaiparth08@gmail.com> Co-authored-by: Andrew Jones <ascjones@gmail.com> Co-authored-by: Jonathan Udd <jonathan@dwellir.com> Co-authored-by: Serban Iorga <serban@parity.io> Co-authored-by: Egor_P <egor@parity.io> Co-authored-by: Branislav Kontur <bkontur@gmail.com> Co-authored-by: Evgeny Snitko <evgeny@parity.io> Co-authored-by: Just van Stam <vstam1@users.noreply.github.com> Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com> Co-authored-by: gupnik <nikhilgupta.iitk@gmail.com> Co-authored-by: dzmitry-lahoda <dzmitry@lahoda.pro> Co-authored-by: zhiqiangxu <652732310@qq.com> Co-authored-by: Nazar Mokrynskyi <nazar@mokrynskyi.com> Co-authored-by: Anwesh <anweshknayak@gmail.com> Co-authored-by: cheme <emericchevalier.pro@gmail.com> Co-authored-by: Sam Johnson <sam@durosoft.com> Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: Jegor Sidorenko <5252494+jsidorenko@users.noreply.github.com> Co-authored-by: Muharem <ismailov.m.h@gmail.com> Co-authored-by: joepetrowski <joe@parity.io> Co-authored-by: Alexandru Gheorghe <49718502+alexggh@users.noreply.github.com> Co-authored-by: Gabriel Facco de Arruda <arrudagates@gmail.com> Co-authored-by: Squirrel <gilescope@gmail.com> Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> Co-authored-by: georgepisaltu <george.pisaltu@parity.io> Co-authored-by: command-bot <> |
||
|
Alin Dima
|
62b78a1615 |
provisioner: allow multiple cores assigned to the same para (#3233)
https://github.com/paritytech/polkadot-sdk/issues/3130 builds on top of https://github.com/paritytech/polkadot-sdk/pull/3160 Processes the availability cores and builds a record of how many candidates it should request from prospective-parachains and their predecessors. Tries to supply as many candidates as the runtime can back. Note that the runtime changes to back multiple candidates per para are not yet done, but this paves the way for it. The following backing/inclusion policy is assumed: 1. the runtime will never back candidates of the same para which don't form a chain with the already backed candidates. Even if the others are still pending availability. We're optimistic that they won't time out and we don't want to back parachain forks (as the complexity would be huge). 2. if a candidate is timed out of the core before being included, all of its successors occupying a core will be evicted. 3. only the candidates which are made available and form a chain starting from the on-chain para head may be included/enacted and cleared from the cores. In other words, if para head is at A and the cores are occupied by B->C->D, and B and D are made available, only B will be included and its core cleared. C and D will remain on the cores awaiting for C to be made available or timed out. As point (2) above already says, if C is timed out, D will also be dropped. 4. The runtime will deduplicate candidates which form a cycle. For example if the provisioner supplies candidates A->B->A, the runtime will only back A (as the state output will be the same) Note that if a candidate is timed out, we don't guarantee that in the next relay chain block the block author will be able to fill all of the timed out cores of the para. That increases complexity by a lot. Instead, the provisioner will supply N candidates where N is the number of candidates timed out, but doesn't include their successors which will be also deleted by the runtime. This'll be backfilled in the next relay chain block. Adjacent changes: - Also fixes: https://github.com/paritytech/polkadot-sdk/issues/3141 - For non prospective-parachains, don't supply multiple candidates per para (we can't have elastic scaling without prospective parachains enabled). paras_inherent should already sanitise this input but it's more efficient this way. Note: all of these changes are backwards-compatible with the non-elastic-scaling scenario (one core per para). |
||
|
Andrei Eres
|
f0e589d72e |
subsystem-bench: add regression tests for availability read and write (#3311)
### What's been done - `subsystem-bench` has been split into two parts: a cli benchmark runner and a library. - The cli runner is quite simple. It just allows us to run `.yaml` based test sequences. Now it should only be used to run benchmarks during development. - The library is used in the cli runner and in regression tests. Some code is changed to make the library independent of the runner. - Added first regression tests for availability read and write that replicate existing test sequences. ### How we run regression tests - Regression tests are simply rust integration tests without the harnesses. - They should only be compiled under the `subsystem-benchmarks` feature to prevent them from running with other tests. - This doesn't work when running tests with `nextest` in CI, so additional filters have been added to the `nextest` runs. - Each benchmark run takes a different time in the beginning, so we "warm up" the tests until their CPU usage differs by only 1%. - After the warm-up, we run the benchmarks a few more times and compare the average with the exception using a precision. ### What is still wrong? - I haven't managed to set up approval voting tests. The spread of their results is too large and can't be narrowed down in a reasonable amount of time in the warm-up phase. - The tests start an unconfigurable prometheus endpoint inside, which causes errors because they use the same 9999 port. I disable it with a flag, but I think it's better to extract the endpoint launching outside the test, as we already do with `valgrind` and `pyroscope`. But we still use `prometheus` inside the tests. ### Future work * https://github.com/paritytech/polkadot-sdk/issues/3528 * https://github.com/paritytech/polkadot-sdk/issues/3529 * https://github.com/paritytech/polkadot-sdk/issues/3530 * https://github.com/paritytech/polkadot-sdk/issues/3531 --------- Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com> |
||
|
Egor_P
|
59b2661444 |
[Backport] Node version and spec_version bumps and ordering of the prdoc files from 1.8.0 (#3508)
This PR backports Node version and `spec_version` bumps to `1.8.0` from the latest release and orders prdoc files related to it. |
||
|
Andrei Sandu
|
7f5d308d29 |
Enable elastic scaling node feature in local testnets genesis (#3509)
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> |
||
|
Alexandru Gheorghe
|
761937ecdc |
Fix accidental no-shows on node restart (#3277)
If approval was in progress we didn't actually restart it, so we end up in a situation where we distribute our assignment, but we don't distribute any approval. --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> |
||
|
Tsvetomir Dimitrov
|
a035dc9be7 |
Remove AssignmentProviderConfig and use parameters from HostConfiguration instead (#3181)
This PR removes `AssignmentProviderConfig` and uses the corresponding ondemand parameters from `HostConfiguration` instead. Additionally `scheduling_lookahead` and all coretime/ondemand related parameters are extracted in a separate struct - `SchedulerParams`. The most relevant commit from the PR is [this one](https://github.com/paritytech/polkadot-sdk/pull/3181/commits/830bc0f5e858944474171bbe33382ad96040b535). Fixes https://github.com/paritytech/polkadot-sdk/issues/2268 --------- Co-authored-by: command-bot <> |
||
|
Oliver Tale-Yazdi
|
eefd5fe449 |
Multi-Block-Migrations, poll hook and new System callbacks (#1781)
This MR is the merge of https://github.com/paritytech/substrate/pull/14414 and https://github.com/paritytech/substrate/pull/14275. It implements [RFC#13](https://github.com/polkadot-fellows/RFCs/pull/13), closes https://github.com/paritytech/polkadot-sdk/issues/198. ----- This Merge request introduces three major topicals: 1. Multi-Block-Migrations 1. New pallet `poll` hook for periodic service work 1. Replacement hooks for `on_initialize` and `on_finalize` in cases where `poll` cannot be used and some more general changes to FRAME. The changes for each topical span over multiple crates. They are listed in topical order below. # 1.) Multi-Block-Migrations Multi-Block-Migrations are facilitated by creating `pallet_migrations` and configuring `System::Config::MultiBlockMigrator` to point to it. Executive picks this up and triggers one step of the migrations pallet per block. The chain is in lockdown mode for as long as an MBM is ongoing. Executive does this by polling `MultiBlockMigrator::ongoing` and not allowing any transaction in a block, if true. A MBM is defined through trait `SteppedMigration`. A condensed version looks like this: ```rust /// A migration that can proceed in multiple steps. pub trait SteppedMigration { type Cursor: FullCodec + MaxEncodedLen; type Identifier: FullCodec + MaxEncodedLen; fn id() -> Self::Identifier; fn max_steps() -> Option<u32>; fn step( cursor: Option<Self::Cursor>, meter: &mut WeightMeter, ) -> Result<Option<Self::Cursor>, SteppedMigrationError>; } ``` `pallet_migrations` can be configured with an aggregated tuple of these migrations. It then starts to migrate them one-by-one on the next runtime upgrade. Two things are important here: - 1. Doing another runtime upgrade while MBMs are ongoing is not a good idea and can lead to messed up state. - 2. **Pallet Migrations MUST BE CONFIGURED IN `System::Config`, otherwise it is not used.** The pallet supports an `UpgradeStatusHandler` that can be used to notify external logic of upgrade start/finish (for example to pause XCM dispatch). Error recovery is very limited in the case that a migration errors or times out (exceeds its `max_steps`). Currently the runtime dev can decide in `FailedMigrationHandler::failed` how to handle this. One follow-up would be to pair this with the `SafeMode` pallet and enact safe mode when an upgrade fails, to allow governance to rescue the chain. This is currently not possible, since governance is not `Mandatory`. ## Runtime API - `Core`: `initialize_block` now returns `ExtrinsicInclusionMode` to inform the Block Author whether they can push transactions. ### Integration Add it to your runtime implementation of `Core` and `BlockBuilder`: ```patch diff --git a/runtime/src/lib.rs b/runtime/src/lib.rs @@ impl_runtime_apis! { impl sp_block_builder::Core<Block> for Runtime { - fn initialize_block(header: &<Block as BlockT>::Header) { + fn initialize_block(header: &<Block as BlockT>::Header) -> RuntimeExecutiveMode { Executive::initialize_block(header) } ... } ``` # 2.) `poll` hook A new pallet hook is introduced: `poll`. `Poll` is intended to replace mostly all usage of `on_initialize`. The reason for this is that any code that can be called from `on_initialize` cannot be migrated through an MBM. Currently there is no way to statically check this; the implication is to use `on_initialize` as rarely as possible. Failing to do so can result in broken storage invariants. The implementation of the poll hook depends on the `Runtime API` changes that are explained above. # 3.) Hard-Deadline callbacks Three new callbacks are introduced and configured on `System::Config`: `PreInherents`, `PostInherents` and `PostTransactions`. These hooks are meant as replacement for `on_initialize` and `on_finalize` in cases where the code that runs cannot be moved to `poll`. The reason for this is to make the usage of HD-code (hard deadline) more explicit - again to prevent broken invariants by MBMs. # 4.) FRAME (general changes) ## `frame_system` pallet A new memorize storage item `InherentsApplied` is added. It is used by executive to track whether inherents have already been applied. Executive and can then execute the MBMs directly between inherents and transactions. The `Config` gets five new items: - `SingleBlockMigrations` this is the new way of configuring migrations that run in a single block. Previously they were defined as last generic argument of `Executive`. This shift is brings all central configuration about migrations closer into view of the developer (migrations that are configured in `Executive` will still work for now but is deprecated). - `MultiBlockMigrator` this can be configured to an engine that drives MBMs. One example would be the `pallet_migrations`. Note that this is only the engine; the exact MBMs are injected into the engine. - `PreInherents` a callback that executes after `on_initialize` but before inherents. - `PostInherents` a callback that executes after all inherents ran (including MBMs and `poll`). - `PostTransactions` in symmetry to `PreInherents`, this one is called before `on_finalize` but after all transactions. A sane default is to set all of these to `()`. Example diff suitable for any chain: ```patch @@ impl frame_system::Config for Test { type MaxConsumers = ConstU32<16>; + type SingleBlockMigrations = (); + type MultiBlockMigrator = (); + type PreInherents = (); + type PostInherents = (); + type PostTransactions = (); } ``` An overview of how the block execution now looks like is here. The same graph is also in the rust doc. <details><summary>Block Execution Flow</summary> <p>  </p> </details> ## Inherent Order Moved to https://github.com/paritytech/polkadot-sdk/pull/2154 --------------- ## TODO - [ ] Check that `try-runtime` still works - [ ] Ensure backwards compatibility with old Runtime APIs - [x] Consume weight correctly - [x] Cleanup --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Liam Aharon <liam.aharon@hotmail.com> Co-authored-by: Juan Girini <juangirini@gmail.com> Co-authored-by: command-bot <> Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com> Co-authored-by: Gavin Wood <gavin@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de> |
||
|
maksimryndin
|
426136671a |
PVF: re-preparing artifact on failed runtime construction (#3187)
resolve https://github.com/paritytech/polkadot-sdk/issues/3139 - [x] use a distinguishable error for `execute_artifact` - [x] remove artifact in case of a `RuntimeConstruction` error during the execution - [x] augment the `validate_candidate_with_retry` of `ValidationBackend` with the case of retriable `RuntimeConstruction` error during the execution - [x] update the book (https://paritytech.github.io/polkadot-sdk/book/node/utility/pvf-host-and-workers.html#retrying-execution-requests) - [x] add a test - [x] run zombienet tests --------- Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com> |
||
|
Liam Aharon
|
12ce4f7d04 |
Runtime Upgrade ref docs and Single Block Migration example pallet (#1554)
Closes https://github.com/paritytech/polkadot-sdk-docs/issues/55 - Changes 'current storage version' terminology to less ambiguous 'in-code storage version' (suggestion by @ggwpez) - Adds a new example pallet `pallet-example-single-block-migrations` - Adds a new reference doc to replace https://docs.substrate.io/maintain/runtime-upgrades/ (temporarily living in the pallet while we wait for developer hub PR to merge) - Adds documentation for the `storage_alias` macro - Improves `trait Hooks` docs - Improves `trait GetStorageVersion` docs - Update the suggested patterns for using `VersionedMigration`, so that version unchecked migrations are never exported - Prevents accidental usage of version unchecked migrations in runtimes https://github.com/paritytech/substrate/pull/14421#discussion_r1255467895 - Unversioned migration code is kept inside `mod version_unchecked`, versioned code is kept in `pub mod versioned` - It is necessary to use modules to limit visibility because the inner migration must be `pub`. See https://github.com/rust-lang/rust/issues/30905 and https://internals.rust-lang.org/t/lang-team-minutes-private-in-public-rules/4504/40 for more. ### todo - [x] move to reference docs to proper place within sdk-docs (now that https://github.com/paritytech/polkadot-sdk/pull/2102 is merged) - [x] prdoc --------- Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Juan <juangirini@gmail.com> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: command-bot <> Co-authored-by: gupnik <nikhilgupta.iitk@gmail.com> |
||
|
maksimryndin
|
7ec0b8741b |
Collator overseer builder unification (#3335)
resolve https://github.com/paritytech/polkadot-sdk/issues/3116 a follow-up on https://github.com/paritytech/polkadot-sdk/pull/3061#pullrequestreview-1847530265: - [x] reuse collator overseer builder for polkadot-node and collator - [x] run zombienet test (0001-parachains-smoke-test.toml) - [x] make wasm build errors more user-friendly for an easier problem detection when using different toolchains in Rust --------- Co-authored-by: ordian <write@reusable.software> Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com> |
||
|
Petr Mensik
|
0cc9b9003c |
Add Polkadotters bootnoders per IBP application (#3423)
Hey everyone, this PR will replace existing Polkadotters bootnodes for Polkadot, Kusama and Westend and add Paseo bootnode to the relay chain suite. At the same time, it will add new bootnodes for all the system parachains, including People on Westend. This PR is a part of our membership in the IBP, meaning that all the bootnodes are hosted on our hardware housed in the data center in Christchurch, New Zealand. All the bootnodes were tested with an empty chain spec file with the following command yielding 1 peer. The test commands used are as follows: ``` ./polkadot --base-path /tmp/node --reserved-only --chain paseo --reserved-nodes "/dns/paseo.bootnodes.polkadotters.com/tcp/30540/wss/p2p/12D3KooWPbbFy4TefEGTRF5eTYhq8LEzc4VAHdNUVCbY4nAnhqPP" ./polkadot --base-path /tmp/node --reserved-only --chain westend --reserved-nodes "/dns/westend.bootnodes.polkadotters.com/tcp/30310/wss/p2p/12D3KooWHPHb64jXMtSRJDrYFATWeLnvChL8NtWVttY67DCH1eC5" ./polkadot --base-path /tmp/node --reserved-only --chain kusama --reserved-nodes "/dns/kusama.bootnodes.polkadotters.com/tcp/30313/wss/p2p/12D3KooWHB5rTeNkQdXNJ9ynvGz8Lpnmsctt7Tvp7mrYv6bcwbPG" ./polkadot --base-path /tmp/node --no-hardware-benchmarks --reserved-only --chain polkadot --reserved-nodes "/dns/polkadot.bootnodes.polkadotters.com/tcp/30316/wss/p2p/12D3KooWPAVUgBaBk6n8SztLrMk8ESByncbAfRKUdxY1nygb9zG3" ./polkadot-parachain --base-path /tmp/node --reserved-only --chain asset-hub-kusama --reserved-nodes "/dns/asset-hub-kusama.bootnodes.polkadotters.com/tcp/30513/wss/p2p/12D3KooWDpk7wVH7RgjErEvbvAZ2kY5VeaAwRJP5ojmn1e8b8UbU" ./polkadot-parachain --base-path /tmp/node --reserved-only --chain asset-hub-polkadot --reserved-nodes "/dns/asset-hub-polkadot.bootnodes.polkadotters.com/tcp/30510/wss/p2p/12D3KooWKbfY9a9oywxMJKiALmt7yhrdQkjXMtvxhhDDN23vG93R" ./polkadot-parachain --base-path /tmp/node --reserved-only --chain asset-hub-westend --reserved-nodes "/dns/asset-hub-westend.bootnodes.polkadotters.com/tcp/30516/wss/p2p/12D3KooWNFYysCqmojxqjjaTfD2VkWBNngfyUKWjcR4WFixfHNTk" ./polkadot-parachain --base-path /tmp/node --reserved-only --chain bridge-hub-kusama --reserved-nodes "/dns/bridge-hub-kusama.bootnodes.polkadotters.com/tcp/30522/wss/p2p/12D3KooWH3pucezRRS5esoYyzZsUkKWcPSByQxEvmM819QL1HPLV" ./polkadot-parachain --base-path /tmp/node --reserved-only --chain bridge-hub-kusama --reserved-nodes "/dns/bridge-hub-kusama.bootnodes.polkadotters.com/tcp/30522/wss/p2p/12D3KooWH3pucezRRS5esoYyzZsUkKWcPSByQxEvmM819QL1HPLV" ./polkadot-parachain --base-path /tmp/node --reserved-only --chain bridge-hub-westend --reserved-nodes "/dns/bridge-hub-westend.bootnodes.polkadotters.com/tcp/30525/wss/p2p/12D3KooWPkwgJofp4GeeRwNgXqkp2aFwdLkCWv3qodpBJLwK43Jj" ./polkadot-parachain --base-path /tmp/node --reserved-only --chain collectives-polkadot --reserved-nodes "/dns/collectives-polkadot.bootnodes.polkadotters.com/tcp/30528/wss/p2p/12D3KooWNohUjvJtGKUa8Vhy8C1ZBB5N8JATB6e7rdLVCioeb3ff" ./polkadot-parachain --base-path /tmp/node --reserved-only --chain collectives-westend --reserved-nodes "/dns/collectives-westend.bootnodes.polkadotters.com/tcp/30531/wss/p2p/12D3KooWAFkXNSBfyPduZVgfS7pj5NuVpbU8Ee5gHeF8wvos7Yqn" ./polkadot-parachain --base-path /tmp/node --reserved-only --chain people-westend --reserved-nodes "/dns/identity-westend.bootnodes.polkadotters.com/tcp/30534/wss/p2p/12D3KooWKr9San6KTM7REJ95cBaDoiciGcWnW8TTftEJgxGF5Ehb" ``` Best regards, Petr, Polkadotters |
||
|
Alexandru Gheorghe
|
b9c792a4b8 |
Add more debug logs to understand if statement-distribution misbehaving (#3419)
Add more debug logs to understand if statement-distribution is in a bad state, should be useful for debugging https://github.com/paritytech/polkadot-sdk/issues/3314 on production networks. Additionally, increase the number of parallel requests should make, since I notice that requests take around 100ms on kusama, and the 5 parallel request was picked mostly random, no reason why we can do more than that. --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Co-authored-by: ordian <write@reusable.software> |
||
|
Andrei Sandu
|
2431001ec0 |
Runtime: allow backing multiple candidates of same parachain on different cores (#3231)
Fixes https://github.com/paritytech/polkadot-sdk/issues/3144 Builds on top of https://github.com/paritytech/polkadot-sdk/pull/3229 ### Summary Some preparations for Runtime to support elastic scaling, guarded by config node features bit `FeatureIndex::ElasticScalingMVP`. This PR introduces a per-candidate `CoreIndex` but does it in a hacky way to avoid changing `CandidateCommitments`, `CandidateReceipts` primitives and networking protocols. #### Including `CoreIndex` in `BackedCandidate` If the `ElasticScalingMVP` feature bit is enabled then `BackedCandidate::validator_indices` is extended by 8 bits. The value stored in these bits represents the assumed core index for the candidate. It is temporary solution which works by creating a mapping from `BackedCandidate` to `CoreIndex` by assuming the `CoreIndex` can be discovered by checking in which validator group the validator that signed the statement is. TODO: - [x] fix tests - [x] add new tests - [x] Bump runtime API for Kusama, so we have that node features thing! -> https://github.com/polkadot-fellows/runtimes/pull/194 --------- Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> Signed-off-by: alindima <alin@parity.io> Co-authored-by: alindima <alin@parity.io> |
||
|
Ignacio Palacios
|
11b5354fd3 |
Fix DepositReserveAsset fees payment (#3340)
The `fee` should be calculated with the reanchored asset, otherwise it could lead to a failure where the set aside fee ends up not being enough. @acatangiu --------- Co-authored-by: Adrian Catangiu <adrian@parity.io> |
||
|
Bastian Köcher
|
9bf1a5e238 |
Check that the validation code matches the parachain code (#3433)
This introduces a check to ensure that the parachain code matches the validation code stored in the relay chain state. If not, it will print a warning. This should be mainly useful for parachain builders to make sure they have setup everything correctly. |
||
|
Andrei Sandu
|
60e537b95f |
Elastic scaling: use an assumed CoreIndex in candidate-backing (#3229)
First step in implementing https://github.com/paritytech/polkadot-sdk/issues/3144 ### Summary of changes - switch statement `Table` candidate mapping from `ParaId` to `CoreIndex` - introduce experimental `InjectCoreIndex` node feature. - determine and assume a `CoreIndex` for a candidate based on statement validator index. If the signature is valid it means validator controls the validator that index and we can easily map it to a validator group/core. - introduce a temporary provisioner fix until we fully enable elastic scaling in the subystem. The fix ensures we don't fetch the same backable candidate when calling `get_backable_candidate` for each core. TODO: - [x] fix backing tests - [x] fix statement table tests - [x] add new test --------- Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> Signed-off-by: alindima <alin@parity.io> Co-authored-by: alindima <alin@parity.io> |
||
|
Niklas Adolfsson
|
fee810a5ea |
rpc server: make possible to disable/enable batch requests (#3364)
The rationale behind this, is that it may be useful for some users actually disable RPC batch requests or limit them by length instead of the total size bytes of the batch. This PR adds two new CLI options: ``` --rpc-disable-batch-requests - disable batch requests on the server --rpc-max-batch-request-len <LEN> - limit batches to LEN on the server. ``` |
||
|
Oliver Tale-Yazdi
|
e89d0fca35 |
Lift dependencies to the workspace (Part 2/x) (#3366)
Lifting some more dependencies to the workspace. Just using the most-often updated ones for now. It can be reproduced locally. ```sh # First you can check if there would be semver incompatible bumps (looks good in this case): $ zepter transpose dependency lift-to-workspace --ignore-errors syn quote thiserror "regex:^serde.*" # Then apply the changes: $ zepter transpose dependency lift-to-workspace --version-resolver=highest syn quote thiserror "regex:^serde.*" --fix # And format the changes: $ taplo format --config .config/taplo.toml ``` --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> |
||
|
Bastian Köcher
|
ef6ac94f52 |
Downgrade log message to trace (#3405)
This spams logs in `Debug` with no useful information. |
||
|
Alexandru Gheorghe
|
197c6cf9e0 |
gossip-support: add unittests for update authorities (#3258)
~The previous fix was actually incomplete because we update the authorties only on the situation where we decided to reconnect because we had a low connectivity issue. Now the problem is that update_authority_ids use the list of connected peers, so on restart that does contain anything, so calling immediately after issue_connection_request won't detect all authorities, so we need to also check every block as the comment said, but that did not match the code.~ Actually the fix was correct the flow is follow if more than 1/3 of the authorities can not be resolved we set last_failure and call `ConnectToResolvedValidators`. We will call UpdateAuthorities for all the authorities already connected and for which we already know the address and for the ones that will connect later on `PeerConnected` will have the AuthorityId field set, because it is already known, so approval-distribution will update its cache topology. --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com> |
||
|
dependabot[bot]
|
3168ed4d6a |
Bump the known_good_semver group with 1 update (#3379)
Bumps the known_good_semver group with 1 update: [clap](https://github.com/clap-rs/clap). Updates `clap` from 4.5.0 to 4.5.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/releases">clap's releases</a>.</em></p> <blockquote> <h2>v4.5.1</h2> <h2>[4.5.1] - 2024-02-16</h2> <h3>Fixes</h3> <ul> <li><em>(error)</em> Include suggestion to add <code>--</code> even if there is a "did you mean" so long as <code>last</code> or <code>trailing_var_arg</code> is used</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/blob/master/CHANGELOG.md">clap's changelog</a>.</em></p> <blockquote> <h2>[4.5.1] - 2024-02-16</h2> <h3>Fixes</h3> <ul> <li><em>(error)</em> Include suggestion to add <code>--</code> even if there is a "did you mean" so long as <code>last</code> or <code>trailing_var_arg</code> is used</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/clap-rs/clap/commit/0c01b5558de0a1a513a2af429981099e550d1a78"><code>0c01b55</code></a> chore: Release</li> <li><a href="https://github.com/clap-rs/clap/commit/08e0b5bde4b45567be98d12e7d394762fa8da6a4"><code>08e0b5b</code></a> docs: Update changelog</li> <li><a href="https://github.com/clap-rs/clap/commit/f2c4e6ec740cce29cea3fd7232e4d2716f089fc2"><code>f2c4e6e</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/5359">#5359</a> from poliorcetics/ab/push-szymvyzpmnqx</li> <li><a href="https://github.com/clap-rs/clap/commit/e782775229081697e47d340a155d6b087eeb46f8"><code>e782775</code></a> fix(complete): Handle newlines in command/arg descriptions</li> <li><a href="https://github.com/clap-rs/clap/commit/fba7c8597bf3028400d6f6a22d83cedb7a2bd5a5"><code>fba7c85</code></a> test(complete): Show newline issue</li> <li><a href="https://github.com/clap-rs/clap/commit/8a7a13a5618cfdc4ff328624a5266e7b4d88649a"><code>8a7a13a</code></a> chore: Release</li> <li><a href="https://github.com/clap-rs/clap/commit/7b3a3e1e5eee57fc37110343b980cbe8170d20ad"><code>7b3a3e1</code></a> docs: Update changelog</li> <li><a href="https://github.com/clap-rs/clap/commit/7b624ca74336f6f14f01007b9039990d53acda0f"><code>7b624ca</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/5356">#5356</a> from epage/escape</li> <li><a href="https://github.com/clap-rs/clap/commit/446328a8d3cdaac28884bf8fdfcc85f35c8b5134"><code>446328a</code></a> fix(error): Include -- in more cases</li> <li><a href="https://github.com/clap-rs/clap/commit/7de6df878238ca8e3d9723bb9650f7fe9470d8bd"><code>7de6df8</code></a> test(error): Show existing last behavior</li> <li>Additional commits viewable in <a href="https://github.com/clap-rs/clap/compare/clap_complete-v4.5.0...clap_complete-v4.5.1">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Niklas Adolfsson
|
de73dd9ac5 |
rpc server: add rate limiting middleware (#3301)
Add RPC server rate limiting which can be utilized by the CLI `--rpc-rate-limit <calls/per minute>` Resolves first part of https://github.com/paritytech/polkadot-sdk/issues/3028 //cc @PierreBesson @kogeler you might be interested in this one --------- Co-authored-by: James Wilson <james@jsdw.me> Co-authored-by: Xiliang Chen <xlchen1291@gmail.com> |
||
|
ordian
|
612587b7b6 |
do not block finality for "disabled" disputes (#3358)
- [x] test with zombienet-sdk - [x] prdoc Relevant Issues: https://github.com/paritytech/polkadot-sdk/issues/3314 (connected to the cause) https://github.com/paritytech/polkadot-sdk/issues/3345 (solves) --------- Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> |
||
|
dependabot[bot]
|
4b484e6a84 |
Bump the known_good_semver group with 6 updates (#3347)
Bumps the known_good_semver group with 6 updates: | Package | From | To | | --- | --- | --- | | [serde](https://github.com/serde-rs/serde) | `1.0.195` | `1.0.196` | | [serde_json](https://github.com/serde-rs/json) | `1.0.111` | `1.0.113` | | [clap](https://github.com/clap-rs/clap) | `4.4.18` | `4.5.0` | | [syn](https://github.com/dtolnay/syn) | `2.0.48` | `2.0.49` | | [serde_yaml](https://github.com/dtolnay/serde-yaml) | `0.9.30` | `0.9.31` | | [serde_derive](https://github.com/serde-rs/serde) | `1.0.195` | `1.0.196` | Updates `serde` from 1.0.195 to 1.0.196 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/serde-rs/serde/releases">serde's releases</a>.</em></p> <blockquote> <h2>v1.0.196</h2> <ul> <li>Improve formatting of "invalid type" error messages involving floats (<a href="https://redirect.github.com/serde-rs/serde/issues/2682">#2682</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/serde-rs/serde/commit/ede9762a583c3cc3b87c10a53551828fad339525"><code>ede9762</code></a> Release 1.0.196</li> <li><a href="https://github.com/serde-rs/serde/commit/d438c2d67bf30e3edab31c2272c4829c12cf4cb5"><code>d438c2d</code></a> Merge pull request <a href="https://redirect.github.com/serde-rs/serde/issues/2682">#2682</a> from dtolnay/decimalpoint</li> <li><a href="https://github.com/serde-rs/serde/commit/bef110b92a171ac568a47339f5bd97938a8c9da2"><code>bef110b</code></a> Format Unexpected::Float with decimal point</li> <li><a href="https://github.com/serde-rs/serde/commit/b971ef11d1b53da7673e0c8199e87509c003c1a8"><code>b971ef1</code></a> Merge pull request <a href="https://redirect.github.com/serde-rs/serde/issues/2681">#2681</a> from dtolnay/workspacedeps</li> <li><a href="https://github.com/serde-rs/serde/commit/29d9f693996d199748136d5561a971ed68626724"><code>29d9f69</code></a> Fix workspace.dependencies default-features future compat warning</li> <li><a href="https://github.com/serde-rs/serde/commit/aecb4083bde754155752f5d7d442b64eb7dc636f"><code>aecb408</code></a> Sort workspace dependencies</li> <li><a href="https://github.com/serde-rs/serde/commit/1c675ab3a38e46df4e62465800970f8b20a2055d"><code>1c675ab</code></a> Merge pull request <a href="https://redirect.github.com/serde-rs/serde/issues/2678">#2678</a> from rodoufu/workspaceDependencies</li> <li><a href="https://github.com/serde-rs/serde/commit/dd619630a337139424725697ccd9a9f7596a2d3a"><code>dd61963</code></a> Adding workspace dependencies</li> <li><a href="https://github.com/serde-rs/serde/commit/111803ab0768d010c606f2fc0d0add12750d5eef"><code>111803a</code></a> Merge pull request <a href="https://redirect.github.com/serde-rs/serde/issues/2673">#2673</a> from Sky9x/msrv-badge</li> <li><a href="https://github.com/serde-rs/serde/commit/0024f74f34fbbdc44a7b22457faebe36c5cbe7f8"><code>0024f74</code></a> Use shields.io's MSRV badges</li> <li>See full diff in <a href="https://github.com/serde-rs/serde/compare/v1.0.195...v1.0.196">compare view</a></li> </ul> </details> <br /> Updates `serde_json` from 1.0.111 to 1.0.113 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/serde-rs/json/releases">serde_json's releases</a>.</em></p> <blockquote> <h2>v1.0.113</h2> <ul> <li>Add <code>swap_remove</code> and <code>shift_remove</code> methods on Map (<a href="https://redirect.github.com/serde-rs/json/issues/1109">#1109</a>)</li> </ul> <h2>v1.0.112</h2> <ul> <li>Improve formatting of "invalid type" error messages involving floats (<a href="https://redirect.github.com/serde-rs/json/issues/1107">#1107</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/serde-rs/json/commit/09d865b34b9701be52764dc9bf571b1a16e9d3dc"><code>09d865b</code></a> Release 1.0.113</li> <li><a href="https://github.com/serde-rs/json/commit/5aeab4eaf69d7959f013f8081865c264d6c00551"><code>5aeab4e</code></a> Merge pull request <a href="https://redirect.github.com/serde-rs/json/issues/1109">#1109</a> from serde-rs/remove</li> <li><a href="https://github.com/serde-rs/json/commit/ca3c2ca3696cab79b8b279be7569ee1647250f1e"><code>ca3c2ca</code></a> Add swap_remove and shift_remove methods on Map</li> <li><a href="https://github.com/serde-rs/json/commit/7fece969e3b480ec620419d65c2aeb08776bebcb"><code>7fece96</code></a> Release 1.0.112</li> <li><a href="https://github.com/serde-rs/json/commit/6a6d2bbd9e8b8bd72573b863f12a4ec991f33232"><code>6a6d2bb</code></a> Merge pull request <a href="https://redirect.github.com/serde-rs/json/issues/1107">#1107</a> from serde-rs/unexpectedfloat</li> <li><a href="https://github.com/serde-rs/json/commit/83d7bad54ba5db3a44198d6df0ff2e81621683fa"><code>83d7bad</code></a> Format f64 in error messages using ryu</li> <li><a href="https://github.com/serde-rs/json/commit/107c2d1c42817f0d71f07a4d5b0ea2f29dbce8b8"><code>107c2d1</code></a> Merge pull request <a href="https://redirect.github.com/serde-rs/json/issues/1106">#1106</a> from serde-rs/invalidvalue</li> <li><a href="https://github.com/serde-rs/json/commit/62ca3e4c01c2e62cd5c2a32e9104f386e5ce7808"><code>62ca3e4</code></a> Handle Unexpected::Unit in Error::invalid_value</li> <li><a href="https://github.com/serde-rs/json/commit/296fafb8f32e8442ef8e4d5725c15ffca726b288"><code>296fafb</code></a> Factor out JSON-specific Display impl for serde::de::Unexpected</li> <li><a href="https://github.com/serde-rs/json/commit/e56cc696bd7c112e5dd4ccfa23d094c3a1c1c1ff"><code>e56cc69</code></a> Merge pull request <a href="https://redirect.github.com/serde-rs/json/issues/1105">#1105</a> from keienWang/master</li> <li>Additional commits viewable in <a href="https://github.com/serde-rs/json/compare/v1.0.111...v1.0.113">compare view</a></li> </ul> </details> <br /> Updates `clap` from 4.4.18 to 4.5.0 <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/blob/master/CHANGELOG.md">clap's changelog</a>.</em></p> <blockquote> <h2>[4.5.0] - 2024-02-08</h2> <h3>Compatibility</h3> <ul> <li>Update MSRV to 1.74</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/clap-rs/clap/commit/a751c5fe65cd33cb09e85ff3039b4fd0182cdb6e"><code>a751c5f</code></a> chore: Release</li> <li><a href="https://github.com/clap-rs/clap/commit/9ec6c942b81c370a8c14652e42295933244555ac"><code>9ec6c94</code></a> chore: Release</li> <li><a href="https://github.com/clap-rs/clap/commit/0735119775c2d27fef6b3c232cb9ef2fcbbd963f"><code>0735119</code></a> docs: Update changelog</li> <li><a href="https://github.com/clap-rs/clap/commit/c4d3959506f5068f86ffb7ab34b622bd2da40dd8"><code>c4d3959</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/5344">#5344</a> from epage/encode</li> <li><a href="https://github.com/clap-rs/clap/commit/f750e577789e1dd34c6950d8c8fe16d1bfd1f49c"><code>f750e57</code></a> fix(lex): Use new-ish OsStr API</li> <li><a href="https://github.com/clap-rs/clap/commit/1d9a554cdfddffcb92d197706b9720e2760cb443"><code>1d9a554</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/5343">#5343</a> from epage/msrv</li> <li><a href="https://github.com/clap-rs/clap/commit/4b45d361b1b96eb1f37b7d1db2684203ca828a94"><code>4b45d36</code></a> chore: Update MSRV to 1.74</li> <li><a href="https://github.com/clap-rs/clap/commit/55b1f945157d4d0e480dea6c492e04d0b541088a"><code>55b1f94</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/5342">#5342</a> from epage/divan</li> <li><a href="https://github.com/clap-rs/clap/commit/ea77b98da3f321069ae136a241a2d266c2b28bd7"><code>ea77b98</code></a> test(complete): Make it order independent</li> <li><a href="https://github.com/clap-rs/clap/commit/b0fea2bac60819bdb042e10c032817fded67f815"><code>b0fea2b</code></a> test(bench): Switch to divan</li> <li>Additional commits viewable in <a href="https://github.com/clap-rs/clap/compare/v4.4.18...clap_complete-v4.5.0">compare view</a></li> </ul> </details> <br /> Updates `syn` from 2.0.48 to 2.0.49 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/dtolnay/syn/releases">syn's releases</a>.</em></p> <blockquote> <h2>2.0.49</h2> <ul> <li>Improve error location when parsing from an empty string literal using <code>LitStr::parse</code> (<a href="https://redirect.github.com/dtolnay/syn/issues/1590">#1590</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/dtolnay/syn/commit/e64c0636042bcd8cf56f9e6c3e828a49cb434649"><code>e64c063</code></a> Release 2.0.49</li> <li><a href="https://github.com/dtolnay/syn/commit/981359c5f46f25f3db42d1b341cc607f634688e7"><code>981359c</code></a> Merge pull request <a href="https://redirect.github.com/dtolnay/syn/issues/1590">#1590</a> from dtolnay/streof</li> <li><a href="https://github.com/dtolnay/syn/commit/51298d40a447b9e5b3cd8c898c2ea9d24b31cd6d"><code>51298d4</code></a> Improve error location at eof in LitStr::parse</li> <li><a href="https://github.com/dtolnay/syn/commit/270c63384a9d891f65880a03a5a92f0bf4605bbf"><code>270c633</code></a> Update test suite to nightly-2024-02-13</li> <li><a href="https://github.com/dtolnay/syn/commit/dc9cf16b9b0601d2d393d059c46a8f8e0220cc21"><code>dc9cf16</code></a> Remove FilterAttrs trait when unused</li> <li><a href="https://github.com/dtolnay/syn/commit/7dcfac79eda716fa806d68cc0aa811f3be717dc0"><code>7dcfac7</code></a> Ignore dead_code warning in test</li> <li><a href="https://github.com/dtolnay/syn/commit/98318441089f9a9bb596fc010ed14c3b593d4bda"><code>9831844</code></a> Update signature of Emitter::emit_diagnostic in nightly-2024-02-07</li> <li><a href="https://github.com/dtolnay/syn/commit/9e8033f63da93dfb6a3e698449c6edff5e56727d"><code>9e8033f</code></a> Update test suite to nightly-2024-02-07</li> <li><a href="https://github.com/dtolnay/syn/commit/cb3348cd94be50995054796facdfa055b6e0e9e8"><code>cb3348c</code></a> Update test suite to nightly-2024-01-23</li> <li><a href="https://github.com/dtolnay/syn/commit/15b9dbcd67bc56f3da7d33bbede46f7380fc7164"><code>15b9dbc</code></a> Update test suite to nightly-2024-01-19</li> <li>Additional commits viewable in <a href="https://github.com/dtolnay/syn/compare/2.0.48...2.0.49">compare view</a></li> </ul> </details> <br /> Updates `serde_yaml` from 0.9.30 to 0.9.31 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/dtolnay/serde-yaml/releases">serde_yaml's releases</a>.</em></p> <blockquote> <h2>0.9.31</h2> <ul> <li>Add <code>swap_remove</code> and <code>shift_remove</code> methods on Mapping (<a href="https://redirect.github.com/dtolnay/serde-yaml/issues/408">#408</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/dtolnay/serde-yaml/commit/2a77483b23897115bdd69af32ae34d593268d555"><code>2a77483</code></a> Release 0.9.31</li> <li><a href="https://github.com/dtolnay/serde-yaml/commit/d8d1a839cf42ef4627c66bebd743f5fecd24e3ed"><code>d8d1a83</code></a> Merge pull request <a href="https://redirect.github.com/dtolnay/serde-yaml/issues/408">#408</a> from dtolnay/remove</li> <li><a href="https://github.com/dtolnay/serde-yaml/commit/f8a99a496836ccfa6c547af9136986d13011be98"><code>f8a99a4</code></a> Add swap_remove and shift_remove methods on Mapping</li> <li><a href="https://github.com/dtolnay/serde-yaml/commit/8b26413e3307e39329dc68a96b065058aec38f9a"><code>8b26413</code></a> Work around dead_code warning in tests</li> <li>See full diff in <a href="https://github.com/dtolnay/serde-yaml/compare/0.9.30...0.9.31">compare view</a></li> </ul> </details> <br /> Updates `serde_derive` from 1.0.195 to 1.0.196 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/serde-rs/serde/releases">serde_derive's releases</a>.</em></p> <blockquote> <h2>v1.0.196</h2> <ul> <li>Improve formatting of "invalid type" error messages involving floats (<a href="https://redirect.github.com/serde-rs/serde/issues/2682">#2682</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/serde-rs/serde/commit/ede9762a583c3cc3b87c10a53551828fad339525"><code>ede9762</code></a> Release 1.0.196</li> <li><a href="https://github.com/serde-rs/serde/commit/d438c2d67bf30e3edab31c2272c4829c12cf4cb5"><code>d438c2d</code></a> Merge pull request <a href="https://redirect.github.com/serde-rs/serde/issues/2682">#2682</a> from dtolnay/decimalpoint</li> <li><a href="https://github.com/serde-rs/serde/commit/bef110b92a171ac568a47339f5bd97938a8c9da2"><code>bef110b</code></a> Format Unexpected::Float with decimal point</li> <li><a href="https://github.com/serde-rs/serde/commit/b971ef11d1b53da7673e0c8199e87509c003c1a8"><code>b971ef1</code></a> Merge pull request <a href="https://redirect.github.com/serde-rs/serde/issues/2681">#2681</a> from dtolnay/workspacedeps</li> <li><a href="https://github.com/serde-rs/serde/commit/29d9f693996d199748136d5561a971ed68626724"><code>29d9f69</code></a> Fix workspace.dependencies default-features future compat warning</li> <li><a href="https://github.com/serde-rs/serde/commit/aecb4083bde754155752f5d7d442b64eb7dc636f"><code>aecb408</code></a> Sort workspace dependencies</li> <li><a href="https://github.com/serde-rs/serde/commit/1c675ab3a38e46df4e62465800970f8b20a2055d"><code>1c675ab</code></a> Merge pull request <a href="https://redirect.github.com/serde-rs/serde/issues/2678">#2678</a> from rodoufu/workspaceDependencies</li> <li><a href="https://github.com/serde-rs/serde/commit/dd619630a337139424725697ccd9a9f7596a2d3a"><code>dd61963</code></a> Adding workspace dependencies</li> <li><a href="https://github.com/serde-rs/serde/commit/111803ab0768d010c606f2fc0d0add12750d5eef"><code>111803a</code></a> Merge pull request <a href="https://redirect.github.com/serde-rs/serde/issues/2673">#2673</a> from Sky9x/msrv-badge</li> <li><a href="https://github.com/serde-rs/serde/commit/0024f74f34fbbdc44a7b22457faebe36c5cbe7f8"><code>0024f74</code></a> Use shields.io's MSRV badges</li> <li>See full diff in <a href="https://github.com/serde-rs/serde/compare/v1.0.195...v1.0.196">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Oliver Tale-Yazdi
|
e80c24733f |
Lift dependencies to the workspace (Part 1) (#2070)
Changes (partial https://github.com/paritytech/polkadot-sdk/issues/994): - Set log to `0.4.20` everywhere - Lift `log` to the workspace Starting with a simpler one after seeing https://github.com/paritytech/polkadot-sdk/pull/2065 from @jsdw. This sets the `default-features` to `false` in the root and then overwrites that in each create to its original value. This is necessary since otherwise the `default` features are additive and its impossible to disable them in the crate again once they are enabled in the workspace. I am using a tool to do this, so its mostly a test to see that it works as expected. --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> |
||
|
Alexandru Gheorghe
|
8362a6810d |
statement-distribution: Fix CostMinor("Unexpected Statement") (#3223)
On grid distribution messages have two paths of reaching a node, so there is the possiblity of a race when two peers send each other the same statement around the same time. Statement local_knowledge will tell us that the peer should have not send the statement because we sent it to it. Fix it by also keeping track only of the statement we received from a given peer and penalize it only if it sends it to us more than once. Fixes: https://github.com/paritytech/polkadot-sdk/issues/2346 Additionally, also use different Cost labels for different paths to make it easier to debug things. --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> |
||
|
Andrei Eres
|
cbd684673b | subsystem-bench: polish imports (#3262) | ||
|
maksimryndin
|
4883e14482 |
refactor pvf security module (#3047)
resolve https://github.com/paritytech/polkadot-sdk/issues/2321 - [x] refactor `security` module into a conditionally compiled - [x] rename `amd64` into x86-64 for consistency with conditional compilation guards and remove reference to a particular vendor - [x] run unit tests and zombienet --------- Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com> |
||
|
Egor_P
|
b2c81b5800 |
[Backport] Version bumps from 1.7.0 release (#3254)
This PR backports version bumps from `1.7.0` release branch and moves related prdoc files to the appropriate folder. |
||
|
Andrei Eres
|
07f8592974 |
subsystem-bench: run cli benchmarks only using config files (#3239)
This PR removes the configuration of subsystem benchmarks via CLI arguments. After this, we only keep configurations only in yaml files. It removes unnecessary code duplication |
||
|
Andrei Eres
|
9e6298e7f7 |
subsystem-bench: Prepare CI output (#3158)
1. Benchmark results are collected in a single struct. 2. The output of the results is prettified. 3. The result struct used to save the output as a yaml and store it in artifacts in a CI job. ``` $ cargo run -p polkadot-subsystem-bench --release -- test-sequence --path polkadot/node/subsystem-bench/examples/availability_read.yaml | tee output.txt $ cat output.txt polkadot/node/subsystem-bench/examples/availability_read.yaml #1 Network usage, KiB total per block Received from peers 510796.000 170265.333 Sent to peers 221.000 73.667 CPU usage, s total per block availability-recovery 38.671 12.890 Test environment 0.255 0.085 polkadot/node/subsystem-bench/examples/availability_read.yaml #2 Network usage, KiB total per block Received from peers 413633.000 137877.667 Sent to peers 353.000 117.667 CPU usage, s total per block availability-recovery 52.630 17.543 Test environment 0.271 0.090 polkadot/node/subsystem-bench/examples/availability_read.yaml #3 Network usage, KiB total per block Received from peers 424379.000 141459.667 Sent to peers 703.000 234.333 CPU usage, s total per block availability-recovery 51.128 17.043 Test environment 0.502 0.167 ``` ``` $ cargo run -p polkadot-subsystem-bench --release -- --ci test-sequence --path polkadot/node/subsystem-bench/examples/availability_read.yaml | tee output.txt $ cat output.txt - benchmark_name: 'polkadot/node/subsystem-bench/examples/availability_read.yaml #1' network: - resource: Received from peers total: 509011.0 per_block: 169670.33333333334 - resource: Sent to peers total: 220.0 per_block: 73.33333333333333 cpu: - resource: availability-recovery total: 31.845848445 per_block: 10.615282815 - resource: Test environment total: 0.23582828799999941 per_block: 0.07860942933333313 - benchmark_name: 'polkadot/node/subsystem-bench/examples/availability_read.yaml #2' network: - resource: Received from peers total: 411738.0 per_block: 137246.0 - resource: Sent to peers total: 351.0 per_block: 117.0 cpu: - resource: availability-recovery total: 18.93596025099999 per_block: 6.31198675033333 - resource: Test environment total: 0.2541994199999979 per_block: 0.0847331399999993 - benchmark_name: 'polkadot/node/subsystem-bench/examples/availability_read.yaml #3' network: - resource: Received from peers total: 424548.0 per_block: 141516.0 - resource: Sent to peers total: 703.0 per_block: 234.33333333333334 cpu: - resource: availability-recovery total: 16.54178526900001 per_block: 5.513928423000003 - resource: Test environment total: 0.43960946299999537 per_block: 0.14653648766666513 ``` --------- Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> |
||
|
Alin Dima
|
7df1ae3b81 |
prospective-parachains: respond with multiple backable candidates (#3160)
Fixes https://github.com/paritytech/polkadot-sdk/issues/3129 |
||
|
Alexandru Gheorghe
|
f9f886886b |
Introduce approval-voting/distribution benchmark (#2621)
## Summary Built on top of the tooling and ideas introduced in https://github.com/paritytech/polkadot-sdk/pull/2528, this PR introduces a synthetic benchmark for measuring and assessing the performance characteristics of the approval-voting and approval-distribution subsystems. Currently this allows, us to simulate the behaviours of these systems based on the following dimensions: ``` TestConfiguration: # Test 1 - objective: !ApprovalsTest last_considered_tranche: 89 min_coalesce: 1 max_coalesce: 6 enable_assignments_v2: true send_till_tranche: 60 stop_when_approved: false coalesce_tranche_diff: 12 workdir_prefix: "/tmp" num_no_shows_per_candidate: 0 approval_distribution_expected_tof: 6.0 approval_distribution_cpu_ms: 3.0 approval_voting_cpu_ms: 4.30 n_validators: 500 n_cores: 100 n_included_candidates: 100 min_pov_size: 1120 max_pov_size: 5120 peer_bandwidth: 524288000000 bandwidth: 524288000000 latency: min_latency: secs: 0 nanos: 1000000 max_latency: secs: 0 nanos: 100000000 error: 0 num_blocks: 10 ``` ## The approach 1. We build a real overseer with the real implementations for approval-voting and approval-distribution subsystems. 2. For a given network size, for each validator we pre-computed all potential assignments and approvals it would send, because this a computation heavy operation this will be cached on a file on disk and be re-used if the generation parameters don't change. 3. The messages will be sent accordingly to the configured parameters and those are split into 3 main benchmarking scenarios. ## Benchmarking scenarios ### Best case scenario *approvals_throughput_best_case.yaml* It send to the approval-distribution only the minimum required tranche to gathered the needed_approvals, so that a candidate is approved. ### Behaviour in the presence of no-shows *approvals_no_shows.yaml* It sends the tranche needed to approve a candidate when we have a maximum of *num_no_shows_per_candidate* tranches with no-shows for each candidate. ### Maximum throughput *approvals_throughput.yaml* It sends all the tranches for each block and measures the used CPU and necessary network bandwidth. by the approval-voting and approval-distribution subsystem. ## How to run it ``` cargo run -p polkadot-subsystem-bench --release -- test-sequence --path polkadot/node/subsystem-bench/examples/approvals_throughput.yaml ``` ## Evaluating performance ### Use the real subsystems metrics If you follow the steps in https://github.com/paritytech/polkadot-sdk/tree/master/polkadot/node/subsystem-bench#install-grafana for installing locally prometheus and grafana, all real metrics for the `approval-distribution`, `approval-voting` and overseer are available. E.g: <img width="2149" alt="Screenshot 2023-12-05 at 11 07 46" src="https://github.com/paritytech/polkadot-sdk/assets/49718502/cb8ae2dd-178b-4922-bfa4-dc37e572ed38"> <img width="2551" alt="Screenshot 2023-12-05 at 11 09 42" src="https://github.com/paritytech/polkadot-sdk/assets/49718502/8b4542ba-88b9-46f9-9b70-cc345366081b"> <img width="2154" alt="Screenshot 2023-12-05 at 11 10 15" src="https://github.com/paritytech/polkadot-sdk/assets/49718502/b8874d8d-632e-443a-9840-14ad8e90c54f"> <img width="2535" alt="Screenshot 2023-12-05 at 11 10 52" src="https://github.com/paritytech/polkadot-sdk/assets/49718502/779a439f-fd18-4985-bb80-85d5afad78e2"> ### Profile with pyroscope 1. Setup pyroscope following the steps in https://github.com/paritytech/polkadot-sdk/tree/master/polkadot/node/subsystem-bench#install-pyroscope, then run any of the benchmark scenario with `--profile` as the arguments. 2. Open the pyroscope dashboard in grafana, e.g: <img width="2544" alt="Screenshot 2024-01-09 at 17 09 58" src="https://github.com/paritytech/polkadot-sdk/assets/49718502/58f50c99-a910-4d20-951a-8b16639303d9"> ### Useful logs 1. Network bandwidth requirements: ``` Payload bytes received from peers: 503993 KiB total, 50399 KiB/block Payload bytes sent to peers: 629971 KiB total, 62997 KiB/block ``` 2. Cpu usage by the approval-distribution/approval-voting subsystems. ``` approval-distribution CPU usage 84.061s approval-distribution CPU usage per block 8.406s approval-voting CPU usage 96.532s approval-voting CPU usage per block 9.653s ``` 3. Time passed until a given block is approved ``` Chain selection approved after 3500 ms hash=0x0101010101010101010101010101010101010101010101010101010101010101 Chain selection approved after 4500 ms hash=0x0202020202020202020202020202020202020202020202020202020202020202 ``` ### Using benchmark to quantify improvements from https://github.com/paritytech/polkadot-sdk/pull/1178 + https://github.com/paritytech/polkadot-sdk/pull/1191 Using a versi-node we compare the scenarios where all new optimisations are disabled with a scenarios where tranche0 assignments are sent in a single message and a conservative simulation where the coalescing of approvals gives us just 50% reduction in the number of messages we send. Overall, what we see is a speedup of around 30-40% in the time it takes to process the necessary messages and a 30-40% reduction in the necessary bandwidth. #### Best case scenario comparison(minimum required tranches sent). Unoptimised ``` Number of blocks: 10 Payload bytes received from peers: 53289 KiB total, 5328 KiB/block Payload bytes sent to peers: 52489 KiB total, 5248 KiB/block approval-distribution CPU usage 6.732s approval-distribution CPU usage per block 0.673s approval-voting CPU usage 9.523s approval-voting CPU usage per block 0.952s ``` vs Optimisation enabled ``` Number of blocks: 10 Payload bytes received from peers: 32141 KiB total, 3214 KiB/block Payload bytes sent to peers: 37314 KiB total, 3731 KiB/block approval-distribution CPU usage 4.658s approval-distribution CPU usage per block 0.466s approval-voting CPU usage 6.236s approval-voting CPU usage per block 0.624s ``` #### Worst case all tranches sent, very unlikely happens when sharding breaks. Unoptimised ``` Number of blocks: 10 Payload bytes received from peers: 746393 KiB total, 74639 KiB/block Payload bytes sent to peers: 729151 KiB total, 72915 KiB/block approval-distribution CPU usage 118.681s approval-distribution CPU usage per block 11.868s approval-voting CPU usage 124.118s approval-voting CPU usage per block 12.412s ``` vs optimised ``` Number of blocks: 10 Payload bytes received from peers: 503993 KiB total, 50399 KiB/block Payload bytes sent to peers: 629971 KiB total, 62997 KiB/block approval-distribution CPU usage 84.061s approval-distribution CPU usage per block 8.406s approval-voting CPU usage 96.532s approval-voting CPU usage per block 9.653s ``` ## TODOs [x] Polish implementation. [x] Use what we have so far to evaluate https://github.com/paritytech/polkadot-sdk/pull/1191 before merging. [x] List of features and additional dimensions we want to use for benchmarking. [x] Run benchmark on hardware similar with versi and kusama nodes. [ ] Add benchmark to be run in CI for catching regression in performance. [ ] Rebase on latest changes for network emulation. --------- Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Co-authored-by: Andrei Sandu <andrei-mihail@parity.io> Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> |
||
|
Bastian Köcher
|
e0674cb3ad |
Enable async backing by default for Rococo/Westend (#3162)
This change is mainly for people running the local variants. They can directly start with async backing. --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Co-authored-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> |
||
|
Alexandru Gheorghe
|
261f4b713b |
Fail early at obviously invalid approvals (#3152)
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> |
||
|
Alexander Samusev
|
5b7f24fca1 |
[ci] Update rust in ci image (1.75 and 2024-01-22) (#3016)
cc https://github.com/paritytech/ci_cd/issues/926 --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: command-bot <> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Liam Aharon <liam.aharon@hotmail.com> Co-authored-by: Dónal Murray <donal.murray@parity.io> Co-authored-by: Vladimir Istyufeev <vladimir@parity.io> |
||
|
Andrei Sandu
|
4220503d28 |
Chore: link code TODO for removing redundant message information (#3126)
fixes #675 --------- Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> |
||
|
Alexandru Gheorghe
|
ff2e7dbd35 |
Move candidate-validation on blocking tasks (#3122)
Candidate validation has a lot of operations that are cpu bound on its main loop things, like: ``` validation_code.hash() sp_maybe_compressed_blob::decompress( &validation_code.0, VALIDATION_CODE_BOMB_LIMIT, ) sp_maybe_compressed_blob::decompress(&pov.block_data.0, POV_BOMB_LIMIT) let code_hash = sp_crypto_hashing::blake2_256(&code).into(); ``` When you add all that you for large POV and CODE it is going to take in the order of 10s of ms and because these are cpu bound operation it is going to hog the executor thread and negatively affect other subsystems around it, so it is better to just move the subsystem on the blocking pool to make sure such unexpected behaviour is avoided. Note! In practice this subsystem does not have a high number of work to be done, so probably the impact of it is really low, but better safe than sorry. Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> |
||
|
Alexandru Gheorghe
|
a664908304 |
Fix recv_signal in orchestra (#3105)
Pulls: https://github.com/paritytech/orchestra/pull/71 --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> |
||
|
Alejandro Martinez Andres
|
008e0fe45e |
Add paseo network --chain option and chain spec (#2668)
After some conversations around the topic of launching a community staging network ( [forum post 1](https://forum.polkadot.network/t/a-new-test-network-for-polkadot/4325), [forum post 2](https://forum.polkadot.network/t/the-new-polkadot-community-testnet/4956) ). And Erin having introduced the community testnet in the [last episode of AAG](https://www.youtube.com/watch?v=uzJEGVG78_E&t=1612s). I would like to include the `paseo` options as one of the chains available for launching using the polkadot node. Note that the chain spec implementation is not yet included at this stage. UPDATE: Merged master with rebase enabled and introduced quite some noise in this PR. Relevant changes are in: - `polkadot/node/service/chain-specs/paseo.json` - `polkadot/node/service/src/chain_spec.rs` - `polkadot/cli/src/command.rs` --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Branislav Kontur <bkontur@gmail.com> Co-authored-by: Liam Aharon <liam.aharon@hotmail.com> Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Jegor Sidorenko <5252494+jsidorenko@users.noreply.github.com> Co-authored-by: Alexander Theißen <alex.theissen@me.com> Co-authored-by: Sergej Sakac <73715684+Szegoo@users.noreply.github.com> Co-authored-by: Dónal Murray <donal.murray@parity.io> Co-authored-by: Adel Arja <adelarja@gmail.com> Co-authored-by: Gonçalo Pestana <g6pestana@gmail.com> Co-authored-by: Ross Bulat <ross@parity.io> Co-authored-by: Francisco Gamundi <52399794+fgamundi@users.noreply.github.com> Co-authored-by: Adrian Catangiu <adrian@parity.io> |
||
|
s0me0ne-unkn0wn
|
3e8139e7de |
Do not run unneeded subsystems on collator and its alongside node (#3061)
Currently, collators and their alongside nodes spin up a full-scale overseer running a bunch of subsystems that are not needed if the node is not a validator. That was considered to be harmless; however, we've got problems with unused subsystems getting stalled for a reason not currently known, resulting in the overseer exiting and bringing down the whole node. This PR aims to only run needed subsystems on such nodes, replacing the rest with `DummySubsystem`. It also enables collator-optimized availability recovery subsystem implementation. Partially solves #1730. |
||
|
Alexandru Gheorghe
|
987edd8864 |
statement-distribution: fix parachains stalling on async_backing enablement (#3063)
Topology is coming only at the beginning of each session, so we might lose it if prospective parachains was not enabled at the begining of the session, so cache it for later use. Fixes: https://github.com/paritytech/polkadot-sdk/issues/3058 --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> |
||
|
Liam Aharon
|
3717ec3802 |
Sync Cargo.toml and crates.io versions (#3034)
Related https://github.com/paritytech/polkadot-sdk/issues/3032 --- Using https://github.com/liamaharon/cargo-workspace-version-tools/ `cargo run -- sync --path ../polkadot-sdk` --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> |
||
|
Andrei Sandu
|
7f28ac6204 |
Remove duplicate subsystem-bench from Cargo.toml (#3073)
🤦🏼 --------- Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> |
||
|
Andrei Sandu
|
47e46d178b |
Add subsystem benchmarks for availability-distribution and biftield-distribution (availability write) (#2970)
Introduce a new test objective : `DataAvailabilityWrite`. The new benchmark measures the network and cpu usage of `availability-distribution`, `biftield-distribution` and `availability-store` subsystems from the perspective of a validator node during the process when candidates are made available. Additionally I refactored the networking emulation to support bandwidth acounting and limits of incoming and outgoing requests. Screenshot of succesful run <img width="1293" alt="Screenshot 2024-01-17 at 19 17 44" src="https://github.com/paritytech/polkadot-sdk/assets/54316454/fde11280-e25b-4dc3-9dc9-d4b9752f9b7a"> --------- Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> |
||
|
Alexandru Gheorghe
|
a6952c7469 |
approval-distribution: Update topology if authorities are discovered later (#2981)
Fixes: https://github.com/paritytech/polkadot-sdk/issues/2138. Especially on restart AuthorithyDiscovery cache is not populated so we create an invalid topology and messages won't be routed correctly for the entire session. This PR proposes to try to fix this by updating the topology as soon as we now the Authority/PeerId mapping, that should impact the situation dramatically. [This issue was hit yesterday](https://grafana.teleport.parity.io/goto/o9q2625Sg?orgId=1), on Westend and resulted in stalling the finality. # TODO - [x] Unit tests - [x] Test impact on versi --------- Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> |
||
|
asynchronous rob
|
5b8abbc884 |
statement-dist: activate leaves before unconditional deactivate (#3048)
https://github.com/paritytech/polkadot-sdk/pull/3042/files#r1465115145 --------- Co-authored-by: command-bot <> |
||
|
Bastian Köcher
|
a989ddfab9 |
statement-distribution: Ensure we deactivate leaves before activating (#3042)
As activation can fail, we ensure that we don't miss deactivation of leaves. |