If an XCM execution fails or ends with leftover assets, these will be
trapped.
In order to claim them, a custom XCM has to be executed, with the
`ClaimAsset` instruction.
However, arbitrary XCM execution is not allowed everywhere yet and XCM
itself is still not easy enough to use for users out there with trapped
assets.
This new extrinsic in `pallet-xcm` will allow these users to easily
claim their assets, without concerning themselves with writing arbitrary
XCMs.
Part of fixing https://github.com/paritytech/polkadot-sdk/issues/3495
---------
Co-authored-by: command-bot <>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
If approval was in progress we didn't actually restart it, so we end up
in a situation where we distribute our assignment, but we don't
distribute any approval.
---------
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
This MR is the merge of
https://github.com/paritytech/substrate/pull/14414 and
https://github.com/paritytech/substrate/pull/14275. It implements
[RFC#13](https://github.com/polkadot-fellows/RFCs/pull/13), closes
https://github.com/paritytech/polkadot-sdk/issues/198.
-----
This Merge request introduces three major topicals:
1. Multi-Block-Migrations
1. New pallet `poll` hook for periodic service work
1. Replacement hooks for `on_initialize` and `on_finalize` in cases
where `poll` cannot be used
and some more general changes to FRAME.
The changes for each topical span over multiple crates. They are listed
in topical order below.
# 1.) Multi-Block-Migrations
Multi-Block-Migrations are facilitated by creating `pallet_migrations`
and configuring `System::Config::MultiBlockMigrator` to point to it.
Executive picks this up and triggers one step of the migrations pallet
per block.
The chain is in lockdown mode for as long as an MBM is ongoing.
Executive does this by polling `MultiBlockMigrator::ongoing` and not
allowing any transaction in a block, if true.
A MBM is defined through trait `SteppedMigration`. A condensed version
looks like this:
```rust
/// A migration that can proceed in multiple steps.
pub trait SteppedMigration {
type Cursor: FullCodec + MaxEncodedLen;
type Identifier: FullCodec + MaxEncodedLen;
fn id() -> Self::Identifier;
fn max_steps() -> Option<u32>;
fn step(
cursor: Option<Self::Cursor>,
meter: &mut WeightMeter,
) -> Result<Option<Self::Cursor>, SteppedMigrationError>;
}
```
`pallet_migrations` can be configured with an aggregated tuple of these
migrations. It then starts to migrate them one-by-one on the next
runtime upgrade.
Two things are important here:
- 1. Doing another runtime upgrade while MBMs are ongoing is not a good
idea and can lead to messed up state.
- 2. **Pallet Migrations MUST BE CONFIGURED IN `System::Config`,
otherwise it is not used.**
The pallet supports an `UpgradeStatusHandler` that can be used to notify
external logic of upgrade start/finish (for example to pause XCM
dispatch).
Error recovery is very limited in the case that a migration errors or
times out (exceeds its `max_steps`). Currently the runtime dev can
decide in `FailedMigrationHandler::failed` how to handle this. One
follow-up would be to pair this with the `SafeMode` pallet and enact
safe mode when an upgrade fails, to allow governance to rescue the
chain. This is currently not possible, since governance is not
`Mandatory`.
## Runtime API
- `Core`: `initialize_block` now returns `ExtrinsicInclusionMode` to
inform the Block Author whether they can push transactions.
### Integration
Add it to your runtime implementation of `Core` and `BlockBuilder`:
```patch
diff --git a/runtime/src/lib.rs b/runtime/src/lib.rs
@@ impl_runtime_apis! {
impl sp_block_builder::Core<Block> for Runtime {
- fn initialize_block(header: &<Block as BlockT>::Header) {
+ fn initialize_block(header: &<Block as BlockT>::Header) -> RuntimeExecutiveMode {
Executive::initialize_block(header)
}
...
}
```
# 2.) `poll` hook
A new pallet hook is introduced: `poll`. `Poll` is intended to replace
mostly all usage of `on_initialize`.
The reason for this is that any code that can be called from
`on_initialize` cannot be migrated through an MBM. Currently there is no
way to statically check this; the implication is to use `on_initialize`
as rarely as possible.
Failing to do so can result in broken storage invariants.
The implementation of the poll hook depends on the `Runtime API` changes
that are explained above.
# 3.) Hard-Deadline callbacks
Three new callbacks are introduced and configured on `System::Config`:
`PreInherents`, `PostInherents` and `PostTransactions`.
These hooks are meant as replacement for `on_initialize` and
`on_finalize` in cases where the code that runs cannot be moved to
`poll`.
The reason for this is to make the usage of HD-code (hard deadline) more
explicit - again to prevent broken invariants by MBMs.
# 4.) FRAME (general changes)
## `frame_system` pallet
A new memorize storage item `InherentsApplied` is added. It is used by
executive to track whether inherents have already been applied.
Executive and can then execute the MBMs directly between inherents and
transactions.
The `Config` gets five new items:
- `SingleBlockMigrations` this is the new way of configuring migrations
that run in a single block. Previously they were defined as last generic
argument of `Executive`. This shift is brings all central configuration
about migrations closer into view of the developer (migrations that are
configured in `Executive` will still work for now but is deprecated).
- `MultiBlockMigrator` this can be configured to an engine that drives
MBMs. One example would be the `pallet_migrations`. Note that this is
only the engine; the exact MBMs are injected into the engine.
- `PreInherents` a callback that executes after `on_initialize` but
before inherents.
- `PostInherents` a callback that executes after all inherents ran
(including MBMs and `poll`).
- `PostTransactions` in symmetry to `PreInherents`, this one is called
before `on_finalize` but after all transactions.
A sane default is to set all of these to `()`. Example diff suitable for
any chain:
```patch
@@ impl frame_system::Config for Test {
type MaxConsumers = ConstU32<16>;
+ type SingleBlockMigrations = ();
+ type MultiBlockMigrator = ();
+ type PreInherents = ();
+ type PostInherents = ();
+ type PostTransactions = ();
}
```
An overview of how the block execution now looks like is here. The same
graph is also in the rust doc.
<details><summary>Block Execution Flow</summary>
<p>
</p>
</details>
## Inherent Order
Moved to https://github.com/paritytech/polkadot-sdk/pull/2154
---------------
## TODO
- [ ] Check that `try-runtime` still works
- [ ] Ensure backwards compatibility with old Runtime APIs
- [x] Consume weight correctly
- [x] Cleanup
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
Co-authored-by: Juan Girini <juangirini@gmail.com>
Co-authored-by: command-bot <>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: Gavin Wood <gavin@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Hey everyone,
this PR will replace existing Polkadotters bootnodes for Polkadot,
Kusama and Westend and add Paseo bootnode to the relay chain suite. At
the same time, it will add new bootnodes for all the system parachains,
including People on Westend. This PR is a part of our membership in the
IBP, meaning that all the bootnodes are hosted on our hardware housed in
the data center in Christchurch, New Zealand.
All the bootnodes were tested with an empty chain spec file with the
following command yielding 1 peer.
The test commands used are as follows:
```
./polkadot --base-path /tmp/node --reserved-only --chain paseo --reserved-nodes "/dns/paseo.bootnodes.polkadotters.com/tcp/30540/wss/p2p/12D3KooWPbbFy4TefEGTRF5eTYhq8LEzc4VAHdNUVCbY4nAnhqPP"
./polkadot --base-path /tmp/node --reserved-only --chain westend --reserved-nodes "/dns/westend.bootnodes.polkadotters.com/tcp/30310/wss/p2p/12D3KooWHPHb64jXMtSRJDrYFATWeLnvChL8NtWVttY67DCH1eC5"
./polkadot --base-path /tmp/node --reserved-only --chain kusama --reserved-nodes "/dns/kusama.bootnodes.polkadotters.com/tcp/30313/wss/p2p/12D3KooWHB5rTeNkQdXNJ9ynvGz8Lpnmsctt7Tvp7mrYv6bcwbPG"
./polkadot --base-path /tmp/node --no-hardware-benchmarks --reserved-only --chain polkadot --reserved-nodes "/dns/polkadot.bootnodes.polkadotters.com/tcp/30316/wss/p2p/12D3KooWPAVUgBaBk6n8SztLrMk8ESByncbAfRKUdxY1nygb9zG3"
./polkadot-parachain --base-path /tmp/node --reserved-only --chain asset-hub-kusama --reserved-nodes "/dns/asset-hub-kusama.bootnodes.polkadotters.com/tcp/30513/wss/p2p/12D3KooWDpk7wVH7RgjErEvbvAZ2kY5VeaAwRJP5ojmn1e8b8UbU"
./polkadot-parachain --base-path /tmp/node --reserved-only --chain asset-hub-polkadot --reserved-nodes "/dns/asset-hub-polkadot.bootnodes.polkadotters.com/tcp/30510/wss/p2p/12D3KooWKbfY9a9oywxMJKiALmt7yhrdQkjXMtvxhhDDN23vG93R"
./polkadot-parachain --base-path /tmp/node --reserved-only --chain asset-hub-westend --reserved-nodes "/dns/asset-hub-westend.bootnodes.polkadotters.com/tcp/30516/wss/p2p/12D3KooWNFYysCqmojxqjjaTfD2VkWBNngfyUKWjcR4WFixfHNTk"
./polkadot-parachain --base-path /tmp/node --reserved-only --chain bridge-hub-kusama --reserved-nodes "/dns/bridge-hub-kusama.bootnodes.polkadotters.com/tcp/30522/wss/p2p/12D3KooWH3pucezRRS5esoYyzZsUkKWcPSByQxEvmM819QL1HPLV"
./polkadot-parachain --base-path /tmp/node --reserved-only --chain bridge-hub-kusama --reserved-nodes "/dns/bridge-hub-kusama.bootnodes.polkadotters.com/tcp/30522/wss/p2p/12D3KooWH3pucezRRS5esoYyzZsUkKWcPSByQxEvmM819QL1HPLV"
./polkadot-parachain --base-path /tmp/node --reserved-only --chain bridge-hub-westend --reserved-nodes "/dns/bridge-hub-westend.bootnodes.polkadotters.com/tcp/30525/wss/p2p/12D3KooWPkwgJofp4GeeRwNgXqkp2aFwdLkCWv3qodpBJLwK43Jj"
./polkadot-parachain --base-path /tmp/node --reserved-only --chain collectives-polkadot --reserved-nodes "/dns/collectives-polkadot.bootnodes.polkadotters.com/tcp/30528/wss/p2p/12D3KooWNohUjvJtGKUa8Vhy8C1ZBB5N8JATB6e7rdLVCioeb3ff"
./polkadot-parachain --base-path /tmp/node --reserved-only --chain collectives-westend --reserved-nodes "/dns/collectives-westend.bootnodes.polkadotters.com/tcp/30531/wss/p2p/12D3KooWAFkXNSBfyPduZVgfS7pj5NuVpbU8Ee5gHeF8wvos7Yqn"
./polkadot-parachain --base-path /tmp/node --reserved-only --chain people-westend --reserved-nodes "/dns/identity-westend.bootnodes.polkadotters.com/tcp/30534/wss/p2p/12D3KooWKr9San6KTM7REJ95cBaDoiciGcWnW8TTftEJgxGF5Ehb"
```
Best regards,
Petr, Polkadotters
with the deprecation of Rococo, Encointer needs a new staging
environment. Paseo will be Polkadot-focused and westend Kusama-focused,
so we propose to use Westend
## Problem
During the bumping of the `polkadot-fellows` repository to
`polkadot-sdk@1.6.0`, I encountered a situation where the benchmarks
`teleport_assets` and `reserve_transfer_assets` in AssetHubKusama
started to fail. This issue arose due to a decreased ED balance for
AssetHubs introduced
[here](https://github.com/polkadot-fellows/runtimes/pull/158/files#diff-80668ff8e793b64f36a9a3ec512df5cbca4ad448c157a5d81abda1b15f35f1daR213),
and also because of a [missing CI
pipeline](https://github.com/polkadot-fellows/runtimes/issues/197) to
check the benchmarks, which went unnoticed.
These benchmarks expect the `caller` to have enough:
1. balance to transfer (BTT)
2. balance for paying delivery (BFPD).
So the initial balance was calculated as `ED * 100`, which seems
reasonable:
```
const ED_MULTIPLIER: u32 = 100;
let balance = existential_deposit.saturating_mul(ED_MULTIPLIER.into());`
```
The problem arises when the price for delivery is 100 times higher than
the existential deposit. In other words, when `ED * 100` does not cover
`BTT` + `BFPD`.
I check AHR/AHW/AHK/AHP and this problem has only AssetHubKusama
```
ED: 3333333
calculated price to parent delivery: 1031666634 (from xcm logs from the benchmark)
---
3333333 * 100 - BTT(3333333) - BFPD(1031666634) = −701666667
```
which results in the error;
```
2024-02-23 09:19:42 Unable to charge fee with error Module(ModuleError { index: 31, error: [17, 0, 0, 0], message: Some("FeesNotMet") })
Error: Input("Benchmark pallet_xcm::reserve_transfer_assets failed: FeesNotMet")
```
## Solution
The benchmarks `teleport_assets` and `reserve_transfer_assets` were
fixed by removing `ED * 100` and replacing it with `DeliveryHelper`
logic, which calculates the (almost real) price for delivery and sets it
along with the existential deposit as the initial balance for the
account used in the benchmark.
## TODO
- [ ] patch for 1.6 -
https://github.com/paritytech/polkadot-sdk/pull/3466
- [ ] patch for 1.7 -
https://github.com/paritytech/polkadot-sdk/pull/3465
- [ ] patch for 1.8 - TODO: PR
---------
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Add more debug logs to understand if statement-distribution is in a bad
state, should be useful for debugging
https://github.com/paritytech/polkadot-sdk/issues/3314 on production
networks.
Additionally, increase the number of parallel requests should make,
since I notice that requests take around 100ms on kusama, and the 5
parallel request was picked mostly random, no reason why we can do more
than that.
---------
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: ordian <write@reusable.software>
Fixes https://github.com/paritytech/polkadot-sdk/issues/3144
Builds on top of https://github.com/paritytech/polkadot-sdk/pull/3229
### Summary
Some preparations for Runtime to support elastic scaling, guarded by
config node features bit `FeatureIndex::ElasticScalingMVP`. This PR
introduces a per-candidate `CoreIndex` but does it in a hacky way to
avoid changing `CandidateCommitments`, `CandidateReceipts` primitives
and networking protocols.
#### Including `CoreIndex` in `BackedCandidate`
If the `ElasticScalingMVP` feature bit is enabled then
`BackedCandidate::validator_indices` is extended by 8 bits.
The value stored in these bits represents the assumed core index for the
candidate.
It is temporary solution which works by creating a mapping from
`BackedCandidate` to `CoreIndex` by assuming the `CoreIndex` can be
discovered by checking in which validator group the validator that
signed the statement is.
TODO:
- [x] fix tests
- [x] add new tests
- [x] Bump runtime API for Kusama, so we have that node features thing!
-> https://github.com/polkadot-fellows/runtimes/pull/194
---------
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: alindima <alin@parity.io>
Co-authored-by: alindima <alin@parity.io>
The `fee` should be calculated with the reanchored asset, otherwise it
could lead to a failure where the set aside fee ends up not being
enough.
@acatangiu
---------
Co-authored-by: Adrian Catangiu <adrian@parity.io>
This introduces a check to ensure that the parachain code matches the
validation code stored in the relay chain state. If not, it will print a
warning. This should be mainly useful for parachain builders to make
sure they have setup everything correctly.
First step in implementing
https://github.com/paritytech/polkadot-sdk/issues/3144
### Summary of changes
- switch statement `Table` candidate mapping from `ParaId` to
`CoreIndex`
- introduce experimental `InjectCoreIndex` node feature.
- determine and assume a `CoreIndex` for a candidate based on statement
validator index. If the signature is valid it means validator controls
the validator that index and we can easily map it to a validator
group/core.
- introduce a temporary provisioner fix until we fully enable elastic
scaling in the subystem. The fix ensures we don't fetch the same
backable candidate when calling `get_backable_candidate` for each core.
TODO:
- [x] fix backing tests
- [x] fix statement table tests
- [x] add new test
---------
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: alindima <alin@parity.io>
Co-authored-by: alindima <alin@parity.io>
The rationale behind this, is that it may be useful for some users
actually disable RPC batch requests or limit them by length instead of
the total size bytes of the batch.
This PR adds two new CLI options:
```
--rpc-disable-batch-requests - disable batch requests on the server
--rpc-max-batch-request-len <LEN> - limit batches to LEN on the server.
```
Lifting some more dependencies to the workspace. Just using the
most-often updated ones for now.
It can be reproduced locally.
```sh
# First you can check if there would be semver incompatible bumps (looks good in this case):
$ zepter transpose dependency lift-to-workspace --ignore-errors syn quote thiserror "regex:^serde.*"
# Then apply the changes:
$ zepter transpose dependency lift-to-workspace --version-resolver=highest syn quote thiserror "regex:^serde.*" --fix
# And format the changes:
$ taplo format --config .config/taplo.toml
```
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
## Xcm changes:
- Fix `pallet_xcm::execute`, move the logic into The `ExecuteController`
so it can be shared with anything that implement that trait.
- Make `ExecuteController::execute` retursn `DispatchErrorWithPostInfo`
instead of `DispatchError`, so that we don't charge the full
`max_weight` provided if the execution is incomplete (useful for
force_batch or contracts calls)
- Fix docstring for `pallet_xcm::execute`, to reflect the changes from
#2405
- Update the signature for `ExecuteController::execute`, we don't need
to return the `Outcome` anymore since we only care about
`Outcome::Complete`
## Contracts changes:
- Update host fn `xcm_exexute`, we don't need to write the `Outcome` to
the sandbox memory anymore. This was also not charged as well before so
it if fixes this too.
- One of the issue was that the dry_run of a contract that call
`xcm_execute` would exhaust the `gas_limit`.
This is because `XcmExecuteController::execute` takes a `max_weight`
argument, and since we don't want the user to specify it manually we
were passing everything left by pre-charghing
`ctx.ext.gas_meter().gas_left()`
- To fix it I added a `fn influence_lowest_limit` on the `Token` trait
and make it return false for `RuntimeCost::XcmExecute`.
- Got rid of the `RuntimeToken` indirection, we can just use
`RuntimeCost` directly.
---------
Co-authored-by: command-bot <>
Adds the coretime and on demand pallets to enable Coretime on Westend.
In order for the migration to run successfully, we need the
Broker/Coretime parachain to be live.
TODO:
- [ ] Broker parachain is live
https://github.com/paritytech/polkadot-sdk/pull/3272
---------
Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <info@kchr.de>
~The previous fix was actually incomplete because we update the
authorties only on the situation where we decided to reconnect because
we had a low connectivity issue. Now the problem is that
update_authority_ids use the list of connected peers, so on restart that
does contain anything, so calling immediately after
issue_connection_request won't detect all authorities, so we need to
also check every block as the comment said, but that did not match the
code.~
Actually the fix was correct the flow is follow if more than 1/3 of the
authorities can not be resolved we set last_failure and call
`ConnectToResolvedValidators`.
We will call UpdateAuthorities for all the authorities already connected
and for which we already know the address and for the ones that will
connect later on `PeerConnected` will have the AuthorityId field set,
because it is already known, so approval-distribution will update its
cache topology.
---------
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
This is a follow-up for `im-online` pallet removal that is cleaning up
its off-chain storage. Must be merged no earlier than #2265 is enacted.
Related: #1964
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Add RPC server rate limiting which can be utilized by the CLI
`--rpc-rate-limit <calls/per minute>`
Resolves first part of
https://github.com/paritytech/polkadot-sdk/issues/3028
//cc @PierreBesson @kogeler you might be interested in this one
---------
Co-authored-by: James Wilson <james@jsdw.me>
Co-authored-by: Xiliang Chen <xlchen1291@gmail.com>
Fixes#3014
This PR adds retry mechanics to `pallet-scheduler`, as described in the
issue above.
Users can now set a retry configuration for a task so that, in case its
scheduled run fails, it will be retried after a number of blocks, for a
specified number of times or until it succeeds.
If a retried task runs successfully before running out of retries, its
remaining retry counter will be reset to the initial value. If a retried
task runs out of retries, it will be removed from the schedule.
Tasks which need to be scheduled for a retry are still subject to weight
metering and agenda space, same as a regular task. Periodic tasks will
have their periodic schedule put on hold while the task is retrying.
---------
Signed-off-by: georgepisaltu <george.pisaltu@parity.io>
Co-authored-by: command-bot <>
This PR implements an (optional) cap of the era inflation that is
allocated to staking rewards. The remaining is minted directly into the
[`RewardRemainder`](https://github.com/paritytech/polkadot-sdk/blob/fb0fd3e62445eb2dee2b2456a0c8574d1ecdcc73/substrate/frame/staking/src/pallet/mod.rs#L160)
account, which is the treasury pot account in Polkadot and Kusama.
The staking pallet now has a percent storage item, `MaxStakersRewards`,
which defines the max percentage of the era inflation that should be
allocated to staking rewards. The remaining era inflation (i.e.
`remaining = max_era_payout - staking_payout.min(staking_payout *
MaxStakersRewards))` is minted directly into the treasury.
The `MaxStakersRewards` can be set by a privileged origin through the
`set_staking_configs` extrinsic.
**To finish**
- [x] run benchmarks for westend-runtime
Replaces https://github.com/paritytech/polkadot-sdk/pull/1483
Closes https://github.com/paritytech/polkadot-sdk/issues/403
---------
Co-authored-by: command-bot <>
Close#2992
Breaking changes:
- rpc server grafana metric `substrate_rpc_requests_started` is removed
(not possible to implement anymore)
- rpc server grafana metric `substrate_rpc_requests_finished` is removed
(not possible to implement anymore)
- rpc server ws ping/pong not ACK:ed within 30 seconds more than three
times then the connection will be closed
Added
- rpc server grafana metric `substrate_rpc_sessions_time` is added to
get the duration for each websocket session
Currently, anyone can registrar a code that exceeds the code size limit
when performing the upgrade from the registrar. This PR fixes that and
adds a new test to cover this.
cc @bkchr @eskimor
I found out during the cleanup of this deprecation message in the
`polkadot-fellows` repository that we deprecated `CurrencyAdapter`
without making the recommended changes.
## TODO
- [ ] fix `polkadot-fellows` bump to 1.6.0
https://github.com/polkadot-fellows/runtimes/pull/159
---------
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Changes (partial https://github.com/paritytech/polkadot-sdk/issues/994):
- Set log to `0.4.20` everywhere
- Lift `log` to the workspace
Starting with a simpler one after seeing
https://github.com/paritytech/polkadot-sdk/pull/2065 from @jsdw.
This sets the `default-features` to `false` in the root and then
overwrites that in each create to its original value. This is necessary
since otherwise the `default` features are additive and its impossible
to disable them in the crate again once they are enabled in the
workspace.
I am using a tool to do this, so its mostly a test to see that it works
as expected.
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
On grid distribution messages have two paths of reaching a node, so
there is the possiblity of a race when two peers send each other the
same statement around the same time. Statement local_knowledge will tell
us that the peer should have not send the statement because we sent it
to it.
Fix it by also keeping track only of the statement we received from a
given peer and penalize it only if it sends it to us more than once.
Fixes: https://github.com/paritytech/polkadot-sdk/issues/2346
Additionally, also use different Cost labels for different paths to make
it easier to debug things.
---------
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Add [forklift
caching](https://gitlab.parity.io/parity/infrastructure/ci_cd/forklift/forklift)
to remainig jobs
by .sh and .py scripts:
- cargo-check-each-crate x6 (`.gitlab/check-each-crate.py`)
- build-linux-stable (`polkadot/scripts/build-only-wasm.sh`)
by before_script:
- build-linux-substrate
- build-subkey-linux (with `.build-subkey` job)
- cargo-check-benches x2
**To disable feature set FORKLIFT_BYPASS variable to true in [project
settings in
gitlab](https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/settings/ci_cd)**
(forklift now handles FORKLIFT_BYPASS by itself)
The `TotalLockedValue` storage value in nomination pools pallet may get
out of sync if the staking pallet does implicit withdrawal of unlocking
chunks belonging to a bonded pool stash. This fix is based on a new
method in the `OnStakingUpdate` traits, `on_withdraw`, which allows the
nomination pools pallet to adjust the `TotalLockedValue` every time
there is an implicit or explicit withdrawal from a bonded pool's stash.
This PR also adds a migration that checks and updates the on-chain TVL
if it got out of sync due to the bug this PR fixes.
**Changes to `trait OnStakingUpdate`**
In order for staking to notify the nomination pools pallet that chunks
where withdrew, we add a new method, `on_withdraw` to the
`OnStakingUpdate` trait. The nomination pools pallet filters the
withdraws that are related to bonded pool accounts and updates the
`TotalValueLocked` accordingly.
**Others**
- Adds try-state checks to the EPM/staking e2e tests
- Adds tests for auto withdrawing in the context of nomination pools
**To-do**
- [x] check if we need a migration to fix the current `TotalValueLocked`
(run try-runtime)
- [x] migrations to fix the current on-chain TVL value
✅ **Kusama**:
```
TotalValueLocked: 99.4559 kKSM
TotalValueLocked (calculated) 99.4559 kKSM
```
⚠️ **Westend**:
```
TotalValueLocked: 18.4060 kWND
TotalValueLocked (calculated) 18.4050 kWND
```
**Polkadot**: TVL not released yet.
Closes https://github.com/paritytech/polkadot-sdk/issues/3055
---------
Co-authored-by: command-bot <>
Co-authored-by: Ross Bulat <ross@parity.io>
Co-authored-by: Dónal Murray <donal.murray@parity.io>
Preparation for https://github.com/paritytech/polkadot-sdk/issues/2664
Changes:
- Only require `Hash` instead of `Block` for the benchmarking
- Refactor DB types to do the same
## Integration
This breaking change can easily be integrated into your node via:
```patch
- cmd.run::<Block, ()>(config)
+ cmd.run::<HashingFor<Block>, ()>(config)
```
Status: waiting for CI checks
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
This PR removes the configuration of subsystem benchmarks via CLI
arguments. After this, we only keep configurations only in yaml files.
It removes unnecessary code duplication
1. Benchmark results are collected in a single struct.
2. The output of the results is prettified.
3. The result struct used to save the output as a yaml and store it in
artifacts in a CI job.
```
$ cargo run -p polkadot-subsystem-bench --release -- test-sequence --path polkadot/node/subsystem-bench/examples/availability_read.yaml | tee output.txt
$ cat output.txt
polkadot/node/subsystem-bench/examples/availability_read.yaml #1
Network usage, KiB total per block
Received from peers 510796.000 170265.333
Sent to peers 221.000 73.667
CPU usage, s total per block
availability-recovery 38.671 12.890
Test environment 0.255 0.085
polkadot/node/subsystem-bench/examples/availability_read.yaml #2
Network usage, KiB total per block
Received from peers 413633.000 137877.667
Sent to peers 353.000 117.667
CPU usage, s total per block
availability-recovery 52.630 17.543
Test environment 0.271 0.090
polkadot/node/subsystem-bench/examples/availability_read.yaml #3
Network usage, KiB total per block
Received from peers 424379.000 141459.667
Sent to peers 703.000 234.333
CPU usage, s total per block
availability-recovery 51.128 17.043
Test environment 0.502 0.167
```
```
$ cargo run -p polkadot-subsystem-bench --release -- --ci test-sequence --path polkadot/node/subsystem-bench/examples/availability_read.yaml | tee output.txt
$ cat output.txt
- benchmark_name: 'polkadot/node/subsystem-bench/examples/availability_read.yaml #1'
network:
- resource: Received from peers
total: 509011.0
per_block: 169670.33333333334
- resource: Sent to peers
total: 220.0
per_block: 73.33333333333333
cpu:
- resource: availability-recovery
total: 31.845848445
per_block: 10.615282815
- resource: Test environment
total: 0.23582828799999941
per_block: 0.07860942933333313
- benchmark_name: 'polkadot/node/subsystem-bench/examples/availability_read.yaml #2'
network:
- resource: Received from peers
total: 411738.0
per_block: 137246.0
- resource: Sent to peers
total: 351.0
per_block: 117.0
cpu:
- resource: availability-recovery
total: 18.93596025099999
per_block: 6.31198675033333
- resource: Test environment
total: 0.2541994199999979
per_block: 0.0847331399999993
- benchmark_name: 'polkadot/node/subsystem-bench/examples/availability_read.yaml #3'
network:
- resource: Received from peers
total: 424548.0
per_block: 141516.0
- resource: Sent to peers
total: 703.0
per_block: 234.33333333333334
cpu:
- resource: availability-recovery
total: 16.54178526900001
per_block: 5.513928423000003
- resource: Test environment
total: 0.43960946299999537
per_block: 0.14653648766666513
```
---------
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
Egor_P