This fixes an issue introduced in
https://github.com/paritytech/substrate/pull/14101, in which I removed
the `Call` enum's documentation and replaced it with a link to the
`Pallet` struct, but this also removed any docs related to call from the
metadata.
I tried to add a regression test for this, but it seems to me that this
is not possible, given that using `type-info` we only assert in type-ids
for `Call`, `Event` and `Error`. I removed some doc comments from a test
setup in `frame-support-test` to demonstrate the issue there. @jsdw do
you have any comments on this?
I also fixed a small issue in the custom html/css of `polkadot-sdk-doc`
crate, making sure it does not affect the rust-doc page of all other
crates.
- [x] Investigate a regression test
- [x] prdoc
Fixing:
```
Verification failed for block 0x07bbf1e04121d70a4bdb21cc055132b53ac2390fa95c4d05497fc91b1e8bf7f5 received from (12D3KooWJzLd8skcAgA24EcJey7aJAhYctfUxWGjSP5Usk9wbpPZ): "Header 0x07bbf1e04121d70a4bdb21cc055132b53ac2390fa95c4d05497fc91b1e8bf7f5 rejected: too far in the future"
```
---------
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: Dmitry Sinyavin <dmitry.sinyavin@parity.io>
Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
This MR is the merge of
https://github.com/paritytech/substrate/pull/14414 and
https://github.com/paritytech/substrate/pull/14275. It implements
[RFC#13](https://github.com/polkadot-fellows/RFCs/pull/13), closes
https://github.com/paritytech/polkadot-sdk/issues/198.
-----
This Merge request introduces three major topicals:
1. Multi-Block-Migrations
1. New pallet `poll` hook for periodic service work
1. Replacement hooks for `on_initialize` and `on_finalize` in cases
where `poll` cannot be used
and some more general changes to FRAME.
The changes for each topical span over multiple crates. They are listed
in topical order below.
# 1.) Multi-Block-Migrations
Multi-Block-Migrations are facilitated by creating `pallet_migrations`
and configuring `System::Config::MultiBlockMigrator` to point to it.
Executive picks this up and triggers one step of the migrations pallet
per block.
The chain is in lockdown mode for as long as an MBM is ongoing.
Executive does this by polling `MultiBlockMigrator::ongoing` and not
allowing any transaction in a block, if true.
A MBM is defined through trait `SteppedMigration`. A condensed version
looks like this:
```rust
/// A migration that can proceed in multiple steps.
pub trait SteppedMigration {
type Cursor: FullCodec + MaxEncodedLen;
type Identifier: FullCodec + MaxEncodedLen;
fn id() -> Self::Identifier;
fn max_steps() -> Option<u32>;
fn step(
cursor: Option<Self::Cursor>,
meter: &mut WeightMeter,
) -> Result<Option<Self::Cursor>, SteppedMigrationError>;
}
```
`pallet_migrations` can be configured with an aggregated tuple of these
migrations. It then starts to migrate them one-by-one on the next
runtime upgrade.
Two things are important here:
- 1. Doing another runtime upgrade while MBMs are ongoing is not a good
idea and can lead to messed up state.
- 2. **Pallet Migrations MUST BE CONFIGURED IN `System::Config`,
otherwise it is not used.**
The pallet supports an `UpgradeStatusHandler` that can be used to notify
external logic of upgrade start/finish (for example to pause XCM
dispatch).
Error recovery is very limited in the case that a migration errors or
times out (exceeds its `max_steps`). Currently the runtime dev can
decide in `FailedMigrationHandler::failed` how to handle this. One
follow-up would be to pair this with the `SafeMode` pallet and enact
safe mode when an upgrade fails, to allow governance to rescue the
chain. This is currently not possible, since governance is not
`Mandatory`.
## Runtime API
- `Core`: `initialize_block` now returns `ExtrinsicInclusionMode` to
inform the Block Author whether they can push transactions.
### Integration
Add it to your runtime implementation of `Core` and `BlockBuilder`:
```patch
diff --git a/runtime/src/lib.rs b/runtime/src/lib.rs
@@ impl_runtime_apis! {
impl sp_block_builder::Core<Block> for Runtime {
- fn initialize_block(header: &<Block as BlockT>::Header) {
+ fn initialize_block(header: &<Block as BlockT>::Header) -> RuntimeExecutiveMode {
Executive::initialize_block(header)
}
...
}
```
# 2.) `poll` hook
A new pallet hook is introduced: `poll`. `Poll` is intended to replace
mostly all usage of `on_initialize`.
The reason for this is that any code that can be called from
`on_initialize` cannot be migrated through an MBM. Currently there is no
way to statically check this; the implication is to use `on_initialize`
as rarely as possible.
Failing to do so can result in broken storage invariants.
The implementation of the poll hook depends on the `Runtime API` changes
that are explained above.
# 3.) Hard-Deadline callbacks
Three new callbacks are introduced and configured on `System::Config`:
`PreInherents`, `PostInherents` and `PostTransactions`.
These hooks are meant as replacement for `on_initialize` and
`on_finalize` in cases where the code that runs cannot be moved to
`poll`.
The reason for this is to make the usage of HD-code (hard deadline) more
explicit - again to prevent broken invariants by MBMs.
# 4.) FRAME (general changes)
## `frame_system` pallet
A new memorize storage item `InherentsApplied` is added. It is used by
executive to track whether inherents have already been applied.
Executive and can then execute the MBMs directly between inherents and
transactions.
The `Config` gets five new items:
- `SingleBlockMigrations` this is the new way of configuring migrations
that run in a single block. Previously they were defined as last generic
argument of `Executive`. This shift is brings all central configuration
about migrations closer into view of the developer (migrations that are
configured in `Executive` will still work for now but is deprecated).
- `MultiBlockMigrator` this can be configured to an engine that drives
MBMs. One example would be the `pallet_migrations`. Note that this is
only the engine; the exact MBMs are injected into the engine.
- `PreInherents` a callback that executes after `on_initialize` but
before inherents.
- `PostInherents` a callback that executes after all inherents ran
(including MBMs and `poll`).
- `PostTransactions` in symmetry to `PreInherents`, this one is called
before `on_finalize` but after all transactions.
A sane default is to set all of these to `()`. Example diff suitable for
any chain:
```patch
@@ impl frame_system::Config for Test {
type MaxConsumers = ConstU32<16>;
+ type SingleBlockMigrations = ();
+ type MultiBlockMigrator = ();
+ type PreInherents = ();
+ type PostInherents = ();
+ type PostTransactions = ();
}
```
An overview of how the block execution now looks like is here. The same
graph is also in the rust doc.
<details><summary>Block Execution Flow</summary>
<p>
</p>
</details>
## Inherent Order
Moved to https://github.com/paritytech/polkadot-sdk/pull/2154
---------------
## TODO
- [ ] Check that `try-runtime` still works
- [ ] Ensure backwards compatibility with old Runtime APIs
- [x] Consume weight correctly
- [x] Cleanup
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
Co-authored-by: Juan Girini <juangirini@gmail.com>
Co-authored-by: command-bot <>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: Gavin Wood <gavin@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
This PR adds tests for the `transaction_broadcast` method.
The testing needs to coordinate the following components:
- The `TestApi` marks transactions as invalid and implements
`ChainApi::validate_transaction`
- this is what dictates if a transaction is valid or not and is called
from within the `BasicPool`
- The `BasicPool` which maintains the transactions and implements
`submit_and_watch` needed by the tx broadcast to submit the transaction
- The status of the transaction pool is exposed by mocking the BasicPool
- The `ChainHeadMockClient` which mocks the
`BlockchainEvents::import_notification_stream` needed by the tx
broadcast to know to which blocks the transaction is submitted
The following changes have been added to the substrate testing to
accommodate this:
- `TestApi` gets ` remove_invalid`, counterpart to `add_invalid` to
ensure an invalid transaction can become valid again; as well as a
priority setter for extrinsics
- `BasicPool` test constructor is extended with options for the
`PoolRotator`
- this mechanism is needed because transactions are banned for 30mins
(default) after they are declared invalid
- testing bypasses this by providing a `Duration::ZERO`
### Testing Scenarios
- Capture the status of the transaction as it is normally broadcasted
- `transaction_stop` is valid while the transaction is in progress
- A future transaction is handled when the dependencies are completed
- Try to resubmit the transaction at a later block (currently invalid)
- An invalid transaction status is propagated; the transaction is marked
as temporarily banned; then the ban expires and transaction is
resubmitted
This builds on top of:
https://github.com/paritytech/polkadot-sdk/pull/3079
Part of: https://github.com/paritytech/polkadot-sdk/issues/3084
cc @paritytech/subxt-team
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: James Wilson <james@jsdw.me>
Closes https://github.com/paritytech/polkadot-sdk/issues/2560
Allows marking storage items with `#[disable_try_decode_storage]`, and
uses it with `System::Events`.
Question: what's the recommended way to write a test for this? I
couldn't find a test for similar existing macro `#[whitelist_storage]`.
While investigating some pruning issues I found some room for
improvement in the notification pin handling.
**Problem:** It was not possible to define an upper limit on
notification pins. The block pinning cache has a limit, but only handles
bodies and justifications.
After this PR, bookkeeping for notifications is managed in the pinning
worker. A limit can be defined in the worker. If that limit is crossed,
blocks that were pinned for that notification are unpinned, which now
affects the state as well as bodies and justifications. The pinned
blocks cache still has a limit, but should never be hit.
closes#19
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Instead of only generating the error, we now generate the actual code
and the error. This generates in total less errors and helps the user to
identify the actual problem and not being confronted with tons of
errors.
Changes the runtime hash algorithm used in
`resolve_state_version_from_wasm` from `DefaultHasher` to a
caller-provided one (usually `HashingFor<Block>`), to match the one used
elsewhere.
This fixes an issue where the runtime wasm is compiled 3 times when
starting the `tanssi-node` with `--dev`. With this fix, the runtime wasm
is only compiled 2 times. The other redundant compilation is caused by
the `GenesisConfigBuilderRuntimeCaller` struct, which ignores the
runtime cache.
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
# Runtime side for PoV Reclaim
## Implementation Overview
- Hostfunction to fetch the storage proof size has been added to the
PVF. It uses the size tracking recorder that was introduced in my
previous PR.
- Mechanisms to use the reclaim HostFunction have been introduced.
- 1. A SignedExtension that checks the node-reported proof size before
and after application of an extrinsic. Then it reclaims the difference.
- 2. A manual helper to make reclaiming easier when manual interaction
is required, for example in `on_idle` or other hooks.
- In order to utilize the manual reclaiming, I modified `WeightMeter` to
support the reduction of consumed weight, at least for storage proof
size.
## How to use
To enable the general functionality for a parachain:
1. Add the SignedExtension to your parachain runtime.
2. Provide the HostFunction to the node
3. Enable proof recording during block import
## TODO
- [x] PRDoc
---------
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
## Summary
* use benchamarked weights instead of hardcoded ones for
`pallet-membership`
* rename benchmark to match extrinsic name
* remove unnecessary dependency from `clear_prime`
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Add a `ApiVersion` constant to the pallet-contracts Config to
communicate with developers the current state of the host functions
exposed by the pallet
As part of BEEFY worker/voter initialization the task waits for certain
chain and backend conditions to be fulfilled:
- BEEFY consensus enabled on-chain & GRANDPA best finalized higher than
on-chain BEEFY genesis block,
- backend has synced headers for BEEFY mandatory blocks between best
BEEFY and best GRANDPA.
During this waiting time, any messages gossiped on the BEEFY topic for
current chain get enqueued in the gossip engine, leading to RAM bloating
and output warning/error messages when the wait time is non-negligible
(like during a clean sync).
This PR adds logic to pump the gossip engine while waiting for other
things to make sure gossiped messages get consumed (practically
discarded until worker is fully initialized).
Also raises the warning threshold for enqueued messages from 10k to
100k. This is in line with the other gossip protocols on the node.
Fixes https://github.com/paritytech/polkadot-sdk/issues/3390
---------
Signed-off-by: Adrian Catangiu <adrian@parity.io>
This PR fixes a subtle bug in `wasm-builder` first introduced in
https://github.com/paritytech/polkadot-sdk/pull/1851 (sorry, my bad! I
should have caught this during review) where the status code of the
`cargo` subprocess is not properly checked, which results in builds
silently succeeding when they shouldn't (that is: if we successfully
build a runtime blob, and then modify the code so that it won't compile,
and recompile it again, then the build will succeed and silently use the
*old* blob).
cc @athei This is the bug you were seeing.
[edit]Also fixes a similar PolkaVM-specific bug where I accidentally
used the wrong comparison operator.[/edit]
In Tanssi, we need a way to stop the collator code and then start it
again. This is to support rotating the same collator between different
runtimes. Currently, this works very well, except for the proposer
metrics, because they only get registered the first time they are
started. Afterwards, we see this warning log:
> Failed to register proposer prometheus metrics: Duplicate metrics
collector registration attempted
~~So this PR adds a method to set metrics, to allow us to register
metrics manually before creating the `ProposerFactory`, and then clone
the same metrics every time we need to start the collator.~~ Implemented
Clone instead
This PR backports the changes from the rpc-v2 spec:
https://github.com/paritytech/json-rpc-interface-spec/pull/134
The `Broadcasted` event has been removed:
- it is hard to enforce a `Dropped { broadcasted: bool }` event in cases
of a load-balancer being placed in front of an RPC server
- when the server exists, it is impossible to guarantee this field if
the server did not previously send a `Broadcasted` event
- the number of peers reported by this event does not guarantee that
peers are unique
- the same peer can disconnect and reconnect, increasing this metric
number
- the number of peers that receive this transaction offers no guarantee
about the transaction being included in the chain at a later time
cc @paritytech/subxt-team
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: James Wilson <james@jsdw.me>
closes#2844
- adds `list-pallets` option which prints all unique available pallets
for benchmarking
```bash
./target/release/node benchmark pallet --list=pallets
```
- adds `all` option which runs benchmarks for all available pallets and
extrinsics (equivalent to `--pallet * --extrinsic *`)
```bash
./target/release/node benchmark pallet --all
```
- use the `list=pallets` syntax in `run_all_benchmarks.sh` script
cc ggwpez
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
This PR is fixing a bug in the sync mechanism between wasmi and
pallet-contracts. This bug leads to essentially double charging all the
gas that was used during the execution of the host function. When the
`call` host function is used for recursion this will lead to a quadratic
amount of gas consumption with regard to the nesting depth.We also took
the chance to refactor the code in question and improve the rust docs.
The bug was caused by not updating `GasMeter::executor_consumed`
(previously `engine_consumed`) when leaving the host function. This lead
to the value being stale (too low) when entering another host function.
---------
Co-authored-by: PG Herveou <pgherveou@gmail.com>
The rationale behind this, is that it may be useful for some users
actually disable RPC batch requests or limit them by length instead of
the total size bytes of the batch.
This PR adds two new CLI options:
```
--rpc-disable-batch-requests - disable batch requests on the server
--rpc-max-batch-request-len <LEN> - limit batches to LEN on the server.
```
Lifting some more dependencies to the workspace. Just using the
most-often updated ones for now.
It can be reproduced locally.
```sh
# First you can check if there would be semver incompatible bumps (looks good in this case):
$ zepter transpose dependency lift-to-workspace --ignore-errors syn quote thiserror "regex:^serde.*"
# Then apply the changes:
$ zepter transpose dependency lift-to-workspace --version-resolver=highest syn quote thiserror "regex:^serde.*" --fix
# And format the changes:
$ taplo format --config .config/taplo.toml
```
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
## Xcm changes:
- Fix `pallet_xcm::execute`, move the logic into The `ExecuteController`
so it can be shared with anything that implement that trait.
- Make `ExecuteController::execute` retursn `DispatchErrorWithPostInfo`
instead of `DispatchError`, so that we don't charge the full
`max_weight` provided if the execution is incomplete (useful for
force_batch or contracts calls)
- Fix docstring for `pallet_xcm::execute`, to reflect the changes from
#2405
- Update the signature for `ExecuteController::execute`, we don't need
to return the `Outcome` anymore since we only care about
`Outcome::Complete`
## Contracts changes:
- Update host fn `xcm_exexute`, we don't need to write the `Outcome` to
the sandbox memory anymore. This was also not charged as well before so
it if fixes this too.
- One of the issue was that the dry_run of a contract that call
`xcm_execute` would exhaust the `gas_limit`.
This is because `XcmExecuteController::execute` takes a `max_weight`
argument, and since we don't want the user to specify it manually we
were passing everything left by pre-charghing
`ctx.ext.gas_meter().gas_left()`
- To fix it I added a `fn influence_lowest_limit` on the `Token` trait
and make it return false for `RuntimeCost::XcmExecute`.
- Got rid of the `RuntimeToken` indirection, we can just use
`RuntimeCost` directly.
---------
Co-authored-by: command-bot <>
# Description
This PR removes redundant type definition from test definition config
implementations like
```
#[derive_impl(frame_system::config_preludes::TestDefaultConfig as frame_system::DefaultConfig)]
impl frame_system::Config for Test {
type A = A;
...
}
```
This changes avoid redundancies in the code as the macro `derive_impl`
defines the relevant types. To implement the changes, it was a simple
fact of running tests and making sure that the tests would still run
while the definition would be removed.
Closes#3237
As a note, here is a brief account of things done from the Issue's
description statement
```
alliance migrate alliance, fast-unstake and bags list to use derive-impl #1636
asset-conversion DONE
asset-rate DONE
assets DONE
atomic-swap DONE
aura DONE
authority-discovery DONE
authorship migrate babe and authorship to use derive-impl #1790
babe migrate babe and authorship to use derive-impl #1790
bags-list migrate alliance, fast-unstake and bags list to use derive-impl #1636
balances DONE
beefy NOTHING TO DO --- also noted this error without failing tests Feb 13 13:49:08.941 ERROR runtime::timestamp: `pallet_timestamp::UnixTime::now` is called at genesis, invalid value returned: 0
beefy-mmr NOTHING TO DO
bounties DONE
child-bounties DONE
collective DONE
contracts DONE
conviction-voting DONE
core-fellowship NOTHING TO DO
democracy DONE
election-provider-multi-phase NOTHING TO DO
elections-phragmen DONE
executive NOTHING TO DO
fast-unstake migrate alliance, fast-unstake and bags list to use derive-impl #1636
glutton DONE
grandpa DONE
identity DONE
im-online NOTHING TO DO
indices Refactor indices pallet #1789
insecure-randomness-collective-flip DONE
lottery DONE
membership DONE
merkle-mountain-range NOTHING TO DO
message-queue DONE
multisig add frame_system::DefaultConfig to individual pallet DefaultConfigs substrate#14453
nft-fractionalization DONE
nfts DONE
nicks Refactor pallet-state-trie-migration to fungible::* traits #1801 NOT IN REPO
nis DONE
node-authorization DONE
nomination-pools NOTHING TO DO -- ONLY impl for Runtime
offences DELETED EVERYTHING -- IS THAT CORRECT??
preimage DONE
proxy add frame_system::DefaultConfig to individual pallet DefaultConfigs substrate#14453
ranked-collective NOTHING TO DO
recovery DONE
referenda DONE
remark DONE
root-offences DONE
root-testing NOTHING TO DO
salary NOTHING TO DO
scheduler DONE
scored-pool DONE
session DONE -- substrate/frame/session/benchmarking/src/mock.rs untouched
society NOTHING TO DO
staking DONE
staking-bags-benchmarks NOT IN REPO
state-trie-migration NOTHING TO DO
statement DONE
sudo DONE
system DONE
timestamp DONE
tips DONE
transaction-payment NOTHING TO DO
transaction-storage NOTHING TO DO
treasury DONE
try-runtime NOTHING TO DO -- no specific mention of 'for Test'
uniques DONE
utility DONE
vesting DONE
whitelist DONE
```
---------
Co-authored-by: command-bot <>
Co-authored-by: gupnik <nikhilgupta.iitk@gmail.com>
This is a follow-up for `im-online` pallet removal that is cleaning up
its off-chain storage. Must be merged no earlier than #2265 is enacted.
Related: #1964
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Add RPC server rate limiting which can be utilized by the CLI
`--rpc-rate-limit <calls/per minute>`
Resolves first part of
https://github.com/paritytech/polkadot-sdk/issues/3028
//cc @PierreBesson @kogeler you might be interested in this one
---------
Co-authored-by: James Wilson <james@jsdw.me>
Co-authored-by: Xiliang Chen <xlchen1291@gmail.com>
This brings functionality to Westend's Coretime Chain runtime, where
previously it was not much more than a shell.
It is assumed that the Coretime pallet will have the same index in the
Westend runtime as it does in Rococo for the runtime calls.
TODO:
- [x] Generate chainspec
- [x] Regenerate weights
- [x] Check hardcoded RuntimeCall weights against relay weights for
transacts
Aura key generation: https://github.com/paritytech/devops/issues/2725
---------
Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com>
Fixes#3014
This PR adds retry mechanics to `pallet-scheduler`, as described in the
issue above.
Users can now set a retry configuration for a task so that, in case its
scheduled run fails, it will be retried after a number of blocks, for a
specified number of times or until it succeeds.
If a retried task runs successfully before running out of retries, its
remaining retry counter will be reset to the initial value. If a retried
task runs out of retries, it will be removed from the schedule.
Tasks which need to be scheduled for a retry are still subject to weight
metering and agenda space, same as a regular task. Periodic tasks will
have their periodic schedule put on hold while the task is retrying.
---------
Signed-off-by: georgepisaltu <george.pisaltu@parity.io>
Co-authored-by: command-bot <>
This PR implements an (optional) cap of the era inflation that is
allocated to staking rewards. The remaining is minted directly into the
[`RewardRemainder`](https://github.com/paritytech/polkadot-sdk/blob/fb0fd3e62445eb2dee2b2456a0c8574d1ecdcc73/substrate/frame/staking/src/pallet/mod.rs#L160)
account, which is the treasury pot account in Polkadot and Kusama.
The staking pallet now has a percent storage item, `MaxStakersRewards`,
which defines the max percentage of the era inflation that should be
allocated to staking rewards. The remaining era inflation (i.e.
`remaining = max_era_payout - staking_payout.min(staking_payout *
MaxStakersRewards))` is minted directly into the treasury.
The `MaxStakersRewards` can be set by a privileged origin through the
`set_staking_configs` extrinsic.
**To finish**
- [x] run benchmarks for westend-runtime
Replaces https://github.com/paritytech/polkadot-sdk/pull/1483
Closes https://github.com/paritytech/polkadot-sdk/issues/403
---------
Co-authored-by: command-bot <>
Dmitry Markin