There has been a case that a referenda failed because the length given
to `submit` was incorrect. The pallet can actually check the length if
the pre-image already exists to ensure that these kind of issues are not
happening again.
This PR adds a new extrinsic `Call::restore_ledger ` gated by
`StakingAdmin` origin that restores a corrupted staking ledger. This
extrinsic will be used to recover ledgers that were affected by the
issue discussed in
https://github.com/paritytech/polkadot-sdk/issues/3245.
The extrinsic will re-write the storage items associated with a stash
account provided as input parameter. The data used to reset the ledger
can be either i) fetched on-chain or ii) partially/totally set by the
input parameters of the call.
In order to use on-chain data to restore the staking locks, we need a
way to read the current lock in the balances pallet. This PR adds a
`InspectLockableCurrency` trait and implements it in the pallet
balances. An alternative would be to tightly couple staking with the
pallet balances but that's inelegant (an example of how it would look
like in [this
branch](https://github.com/paritytech/polkadot-sdk/tree/gpestana/ledger-badstate-clean_tightly)).
More details on the type of corruptions and corresponding fixes
https://hackmd.io/DLb5jEYWSmmvqXC9ae4yRg?view#/
We verified that the `Call::restore_ledger` does fix all current
corrupted ledgers in Polkadot and Kusama. You can verify it here
https://hackmd.io/v-XNrEoGRpe7APR-EZGhOA.
**Changes introduced**
- Adds `Call::restore_ledger ` extrinsic to recover a corrupted ledger;
- Adds trait `frame_support::traits::currency::InspectLockableCurrency`
to allow external pallets to read current locks given an account and
lock ID;
- Implements the `InspectLockableCurrency` in the pallet-balances.
- Adds staking locks try-runtime checks
(https://github.com/paritytech/polkadot-sdk/issues/3751)
**Todo**
- [x] benchmark `Call::restore_ledger`
- [x] throughout testing of all ledger recovering cases
- [x] consider adding the staking locks try-runtime checks to this PR
(https://github.com/paritytech/polkadot-sdk/issues/3751)
- [x] simulate restoring all ledgers
(https://hackmd.io/Dsa2tvhISNSs7zcqriTaxQ?view) in Polkadot and Kusama
using chopsticks -- https://hackmd.io/v-XNrEoGRpe7APR-EZGhOA
Related to https://github.com/paritytech/polkadot-sdk/issues/3245
Closes https://github.com/paritytech/polkadot-sdk/issues/3751
---------
Co-authored-by: command-bot <>
The authority-discovery mechanism has implemented a few exponential
timers for:
- publishing the authority records
- goes from 2 seconds (when freshly booted) to 1 hour if the node is
long-running
- set to 1 hour after successfully publishing the authority record
- discovering other authority records
- goes from 2 seconds (when freshly booted) to 10 minutes if the node is
long-running
This PR resets the exponential publishing and discovery interval to
defaults ensuring that long-running nodes:
- will retry publishing the authority records as aggressively as freshly
booted nodes
- Currently, if a long-running node fails to publish the DHT record when
the keys change (ie DhtEvent::ValuePutFailed), it will only retry after
1 hour
- will rediscover other authorities faster (since there is a chance that
other authority keys changed)
The subp2p-explorer has difficulties discovering the authorities when
the authority set changes in the first few hours. This might be entirely
due to the recursive nature of the DHT and the needed time to propagate
the records. However, there is a small chance that the authority
publishing failed and is only retried in 1h.
Let me know if this makes sense 🙏
cc @paritytech/networking
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
In preparation for the merkleized metadata, we need to ensure that
constants are actually constant. If we want to test the unsigned phase
we could for example just disable signed voter. Or we add some extra
mechanism to the pallet to disable the signed phase from time to time.
---------
Co-authored-by: Ankan <10196091+Ank4n@users.noreply.github.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
**Update:** Pushed additional changes based on the review comments.
**This pull request fixes various spelling mistakes in this
repository.**
Most of the changes are contained in the first **3** commits:
- `Fix spelling mistakes in comments and docs`
- `Fix spelling mistakes in test names`
- `Fix spelling mistakes in error messages, panic messages, logs and
tracing`
Other source code spelling mistakes are separated into individual
commits for easier reviewing:
- `Fix the spelling of 'authority'`
- `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'`
- `Fix the spelling of 'prev_enqueud_messages'`
- `Fix the spelling of 'endpoint'`
- `Fix the spelling of 'children'`
- `Fix the spelling of 'PenpalSiblingSovereignAccount'`
- `Fix the spelling of 'PenpalSudoAccount'`
- `Fix the spelling of 'insufficient'`
- `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'`
- `Fix the spelling of 'subtracted'`
- `Fix the spelling of 'CandidatePendingAvailability'`
- `Fix the spelling of 'exclusive'`
- `Fix the spelling of 'until'`
- `Fix the spelling of 'discriminator'`
- `Fix the spelling of 'nonexistent'`
- `Fix the spelling of 'subsystem'`
- `Fix the spelling of 'indices'`
- `Fix the spelling of 'committed'`
- `Fix the spelling of 'topology'`
- `Fix the spelling of 'response'`
- `Fix the spelling of 'beneficiary'`
- `Fix the spelling of 'formatted'`
- `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'`
- `Fix the spelling of 'succeeded'`
- `Fix the spelling of 'reopened'`
- `Fix the spelling of 'proposer'`
- `Fix the spelling of 'InstantiationNonce'`
- `Fix the spelling of 'depositor'`
- `Fix the spelling of 'expiration'`
- `Fix the spelling of 'phantom'`
- `Fix the spelling of 'AggregatedKeyValue'`
- `Fix the spelling of 'randomness'`
- `Fix the spelling of 'defendant'`
- `Fix the spelling of 'AquaticMammal'`
- `Fix the spelling of 'transactions'`
- `Fix the spelling of 'PassingTracingSubscriber'`
- `Fix the spelling of 'TxSignaturePayload'`
- `Fix the spelling of 'versioning'`
- `Fix the spelling of 'descendant'`
- `Fix the spelling of 'overridden'`
- `Fix the spelling of 'network'`
Let me know if this structure is adequate.
**Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`,
`Merkelization`, `Merkleization`, is somewhat inconsistent but I left it
as it is.
~~**Note:** In some places the term `Receival` is used to refer to
message reception, IMO `Reception` is the correct word here, but I left
it as it is.~~
~~**Note:** In some places the term `Overlayed` is used instead of the
more acceptable version `Overlaid` but I also left it as it is.~~
~~**Note:** In some places the term `Applyable` is used instead of the
correct version `Applicable` but I also left it as it is.~~
**Note:** Some usage of British vs American english e.g. `judgement` vs
`judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc.
are both present in different places, but I suppose that's
understandable given the number of contributors.
~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it
triggers errors in CI when I make changes to it, so I left it as it
is.~~
_This PR is being continued from
https://github.com/paritytech/polkadot-sdk/pull/2206, which was closed
when the developer_hub was merged._
closes https://github.com/paritytech/polkadot-sdk-docs/issues/44
---
# Description
This PR adds a reference document to the `developer-hub` crate (see
https://github.com/paritytech/polkadot-sdk/pull/2102). This specific
reference document covers defensive programming practices common within
the context of developing a runtime with Substrate.
In particular, this covers the following areas:
- Default behavior of how Rust deals with numbers in general
- How to deal with floating point numbers in runtime / fixed point
arithmetic
- How to deal with Integer overflows
- General "safe math" / defensive programming practices for common
pallet development scenarios
- Defensive traits that exist within Substrate, i.e.,
`defensive_saturating_add `, `defensive_unwrap_or`
- More general defensive programming examples (keep it concise)
- Link to relevant examples where these practices are actually in
production / being used
- Unwrapping (or rather lack thereof) 101
todo
--
- [x] Apply feedback from previous PR
- [x] This may warrant a PR to append some of these docs to
`sp_arithmetic`
---------
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Gonçalo Pestana <g6pestana@gmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: Radha <86818441+DrW3RK@users.noreply.github.com>
Introduces `CryptoBytes` type defined as:
```rust
pub struct CryptoBytes<const N: usize, Tag = ()>(pub [u8; N], PhantomData<fn() -> Tag>);
```
The type implements a bunch of methods and traits which are typically
expected from a byte array newtype
(NOTE: some of the methods and trait implementations IMO are a bit
redundant, but I decided to maintain them all to not change too much
stuff in this PR)
It also introduces two (generic) typical consumers of `CryptoBytes`:
`PublicBytes` and `SignatureBytes`.
```rust
pub struct PublicTag;
pub PublicBytes<const N: usize, CryptoTag> = CryptoBytes<N, (PublicTag, CryptoTag)>;
pub struct SignatureTag;
pub SignatureBytes<const N: usize, CryptoTag> = CryptoBytes<N, (SignatureTag, CryptoTag)>;
```
Both of them use a tag to differentiate the two types at a higher level.
Downstream specializations will further specialize using a dedicated
crypto tag. For example in ECDSA:
```rust
pub struct EcdsaTag;
pub type Public = PublicBytes<PUBLIC_KEY_SERIALIZED_SIZE, EcdsaTag>;
pub type Signature = PublicBytes<PUBLIC_KEY_SERIALIZED_SIZE, EcdsaTag>;
```
Overall we have a cleaner and most importantly **consistent** code for
all the types involved
All these details are opaque to the end user which can use `Public` and
`Signature` for the cryptos as before
This PR enables the `transaction_unstable_broadcast ` and
`transaction_unstable_stop` RPC API.
Since the API is unstable, we don't need to expose this in the release
notes.
After merging this, we could validate the API in subxt and stabilize it.
Spec PR that stabilizes the API:
https://github.com/paritytech/json-rpc-interface-spec/pull/139
cc @paritytech/subxt-team
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Part of #3326
cc @kianenigma @ggwpez @liamaharon
polkadot address: 12poSUQPtcF1HUPQGY3zZu2P8emuW9YnsPduA4XG3oCEfJVp
---------
Signed-off-by: Matteo Muraca <mmuraca247@gmail.com>
Changes:
- Port the `pov_mode` attribute from the V1 syntax to V2
- Update `pallet-whitelist` and `frame-benchmarking-pallet-pov`
Follow up: also allow this attribute on top-level benchmark modules.
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
Currently the xcm-executor returns an `Unimplemented` error if it
receives any HRMP-related instruction.
What I propose here, which is what we are currently doing in our forked
executor at polimec, is to introduce a trait implemented by the executor
which will handle those instructions.
This way, if parachains want to keep the default behavior, they just use
`()` and it will return unimplemented, but they can also implement their
own logic to establish HRMP channels with other chains in an automated
fashion, without requiring to go through governance.
Our implementation is mentioned in the [polkadot HRMP
docs](https://arc.net/l/quote/hduiivbu), and it was suggested to us to
submit a PR to add these changes to polkadot-sdk.
---------
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: command-bot <>
There is a problem in the way we update `authorithy-discovery` next keys
and because of that nodes that enter the active set would be noticed at
the start of the session they become active, instead of the start of the
previous session as it was intended. This is problematic because:
1. The node itself advertises its addresses on the DHT only when it
notices it should become active on around ~10m loop, so in this case it
would notice after it becomes active.
2. The other nodes won't be able to detect the new nodes addresses at
the beginning of the session, so it won't added them to the reserved
set.
With 1 + 2, we end-up in a situation where the the new node won't be
able to properly connect to its peers because it won't be in its peers
reserved set. Now, the nodes accept by default`MIN_GOSSIP_PEERS: usize =
25` connections to nodes that are not in the reserved set, but given
Kusama size(> 1000 nodes) you could easily have more than`25` new nodes
entering the active set or simply the nodes don't have slots anymore
because, they already have connections to peers not in the active set.
In the end what the node would notice is 0 backing rewards because it
wasn't directly connected to the peers in its backing group.
## Root-cause
The flow is like this:
1. At BAD_SESSION - 1, in `rotate_session` new nodes are added to
QueuedKeys
https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/session/src/lib.rs#L609
```
<QueuedKeys<T>>::put(queued_amalgamated.clone());
<QueuedChanged<T>>::put(next_changed);
```
2. AuthorityDiscovery::on_new_session is called with `changed` being the
value of `<QueuedChanged<T>>:` at BAD_SESSION - **2** because it was
saved before being updated
https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/session/src/lib.rs#L613
3. At BAD_SESSION - 1, `AuthorityDiscovery::on_new_session` doesn't
updated its next_keys because `changed` was false.
4. For the entire durations of `BAD_SESSION - 1` everyone calling
runtime api `authorities`(should return past, present and future
authorities) won't discover the nodes that should become active .
5. At the beginning of BAD_SESSION, all nodes discover the new nodes are
authorities, but it is already too late because reserved_nodes are
updated only at the beginning of the session by the `gossip-support`.
See above why this bad.
## Fix
Update next keys with the queued_validators at every session, not matter
the value of `changed` this is the same way babe pallet correctly does
it.
https://github.com/paritytech/polkadot-sdk/blob/02e1a7f476d7d7c67153e975ab9a1bdc02ffea12/substrate/frame/babe/src/lib.rs#L655
## Notes
- The issue doesn't reproduce with proof-authorities changes like
`versi` because `changed` would always be true and `AuthorityDiscovery`
correctly updates its next_keys every time.
- Confirmed at session `37651` on kusama that this is exactly what it
happens by looking at blocks with polkadot.js.
## TODO
- [ ] Move versi on proof of stake and properly test before and after
fix to confirm there is no other issue.
---------
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Removed the `pallet::getter` macro call from storage type definitions
and added the corresponding implementations directly.
fixes#3330
polkadot address: 14JzTPPUd8x8phKi8qLxHgNTnTMg6DUukCLXoWprejkaHXPz
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
This PR removes sp-std crate from substrate/primitives sub-directories.
For now crates that have `pub use` of sp-std or export macros that would
necessitate users of the macros to `extern crate alloc` have been
excluded from this PR.
There should be no breaking changes in this PR.
---------
Co-authored-by: Koute <koute@users.noreply.github.com>
Currently, the staking logic does not prevent a controller from becoming
a stash of *another* ledger (introduced by [removing this
check](https://github.com/paritytech/polkadot-sdk/pull/1484/files#diff-3aa6ceab5aa4e0ab2ed73a7245e0f5b42e0832d8ca5b1ed85d7b2a52fb196524L850)).
Given that the remaining of the code expects that never happens, bonding
a ledger with a stash that is a controller of another ledger may lead to
data inconsistencies and data losses in bonded ledgers. For more
detailed explanation of this issue:
https://hackmd.io/@gpestana/HJoBm2tqo/%2FTPdi28H7Qc2mNUqLSMn15w
In a nutshell, when fetching a ledger with a given controller, we may be
end up getting the wrong ledger which can lead to unexpected ledger
states.
This PR also ensures that `set_controller` does not lead to data
inconsistencies in the staking ledger and bonded storage in the case
when a controller of a stash is a stash of *another* ledger. and
improves the staking `try-runtime` checks to catch potential issues with
the storage preemptively.
In summary, there are two important cases here:
1. **"Sane" double bonded ledger**
When a controller of a ledger is a stash of *another* ledger. In this
case, we have:
```
> Bonded(stash, controller)
(A, B) // stash A with controller B
(B, C) // B is also a stash of another ledger
(C, D)
> Ledger(controller)
Ledger(B) = L_a (stash = A)
Ledger(C) = L_b (stash = B)
Ledger(D) = L_c (stash = C)
```
In this case, the ledgers can be mutated and all operations are OK.
However, we should not allow `set_controller` to be called if it means
it results in a "corrupt" double bonded ledger (see below).
3. **"Corrupt" double bonded ledger**
```
> Bonded(stash, controller)
(A, B) // stash A with controller B
(B, B)
(C, D)
```
In this case, B is a stash and controller AND is corrupted, since B is
responsible for 2 ledgers which is not correct and will lead to
inconsistent states. Thus, in this case, in this PR we are preventing
these ledgers from mutating (i.e. operations like bonding extra etc)
until the ledger is brought back to a consistent state.
---
**Changes**:
- Checks if stash is already a controller when calling `Call::bond`
(fixes the regression introduced by [removing this
check](https://github.com/paritytech/polkadot-sdk/pull/1484/files#diff-3aa6ceab5aa4e0ab2ed73a7245e0f5b42e0832d8ca5b1ed85d7b2a52fb196524L850));
- Ensures that all fetching ledgers from storage are done through the
`StakingLedger` API;
- Ensures that -- when fetching a ledger from storage using the
`StakingLedger` API --, a `Error::BadState` is returned if the ledger
bonding is in a bad state. This prevents bad ledgers from mutating (e.g.
`bond_extra`, `set_controller`, etc) its state and avoid further data
inconsistencies.
- Prevents stashes which are controllers or another ledger from calling
`set_controller`, since that may lead to a bad state.
- Adds further try-state runtime checks that check if there are ledgers
in a bad state based on their bonded metadata.
Related to https://github.com/paritytech/polkadot-sdk/issues/3245
---------
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: kianenigma <kian@parity.io>
This PR adds a debug log for displaying all the public addresses that
will later be advertised in the DHT record of the authority. The
Authority DHT record will contain the address ++ `/p2p/peerID` (if not
already present).
This log enables us to check if different nodes will advertise in the
DHT record of the authority the same IP address, however with different
peer IDs.
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Moved from https://github.com/paritytech/substrate/pull/14788
----
Fixes https://github.com/paritytech/polkadot-sdk/issues/232
This PR introduces outer-macro approach for `construct_runtime` as
discussed in the linked issue. It looks like the following:
```rust
#[frame_support::runtime]
mod runtime {
#[runtime::runtime]
#[runtime::derive(
RuntimeCall,
RuntimeEvent,
RuntimeError,
RuntimeOrigin,
RuntimeFreezeReason,
RuntimeHoldReason,
RuntimeSlashReason,
RuntimeLockId,
RuntimeTask,
)]
pub struct Runtime;
#[runtime::pallet_index(0)]
pub type System = frame_system;
#[runtime::pallet_index(1)]
pub type Timestamp = pallet_timestamp;
#[runtime::pallet_index(2)]
pub type Aura = pallet_aura;
#[runtime::pallet_index(3)]
pub type Grandpa = pallet_grandpa;
#[runtime::pallet_index(4)]
pub type Balances = pallet_balances;
#[runtime::pallet_index(5)]
pub type TransactionPayment = pallet_transaction_payment;
#[runtime::pallet_index(6)]
pub type Sudo = pallet_sudo;
// Include the custom logic from the pallet-template in the runtime.
#[runtime::pallet_index(7)]
pub type TemplateModule = pallet_template;
}
```
## Features
- `#[runtime::runtime]` attached to a struct defines the main runtime
- `#[runtime::derive]` attached to this struct defines the types
generated by runtime
- `#[runtime::pallet_index]` must be attached to a pallet to define its
index
- `#[runtime::disable_call]` can be optionally attached to a pallet to
disable its calls
- `#[runtime::disable_unsigned]` can be optionally attached to a pallet
to disable unsigned calls
- A pallet instance can be defined as `TemplateModule:
pallet_template<Instance>`
- An optional attribute can be defined as
`#[frame_support::runtime(legacy_ordering)]` to ensure that the order of
hooks is same as the order of pallets (and not based on the
pallet_index). This is to support legacy runtimes and should be avoided
for new ones.
## Todo
- [x] Update the latest syntax in kitchensink and tests
- [x] Update UI tests
- [x] Docs
## Extension
- Abstract away the Executive similar to
https://github.com/paritytech/substrate/pull/14742
- Optionally avoid the need to specify all runtime types (TBD)
---------
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: Nikhil Gupta <>
This PR adds a new PolkaVM-based executor to Substrate.
- The executor can now be used to actually run a PolkaVM-based runtime,
and successfully produces blocks.
- The executor is always compiled-in, but is disabled by default.
- The `SUBSTRATE_ENABLE_POLKAVM` environment variable must be set to `1`
to enable the executor, in which case the node will accept both WASM and
PolkaVM program blobs (otherwise it'll default to WASM-only). This is
deliberately undocumented and not explicitly exposed anywhere (e.g. in
the command line arguments, or in the API) to disincentivize anyone from
enabling it in production. If/when we'll move this into production usage
I'll remove the environment variable and do it "properly".
- I did not use our legacy runtime allocator for the PolkaVM executor,
so currently every allocation inside of the runtime will leak guest
memory until that particular instance is destroyed. The idea here is
that I will work on the https://github.com/polkadot-fellows/RFCs/pull/4
which will remove the need for the legacy allocator under WASM, and that
will also allow us to use a proper non-leaking allocator under PolkaVM.
- I also did some minor cleanups of the WASM executor and deleted some
dead code.
No prdocs included since this is not intended to be an end-user feature,
but an unofficial experiment, and shouldn't affect any current
production user. Once this is production-ready a full Polkadot
Fellowship RFC will be necessary anyway.
As I've been dancing around this pallet for quite some time anyway, I
decided to remove getters at once. There were only a few leftovers in
tests.
Related: #3326
CC @muraca
This fixes the behaviour of `Linear` which is the default implementation
of the `AdaptPrice` trait in the broker pallet. Previously if cores were
offered but not sold in only one sale, the price would be set to zero
and due to the logic being purely multiplicative, the price would stay
at 0 indefinitely.
This could be further paired with a configurable minimum in the broker
pallet itself, which will be a future PR.
This affects the Rococo and Westend Coretime chains, but Kusama has a
different implementation so this isn't required for the Kusama launch. I
actually thought I opened this a while ago.
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Rustfmt will add a trailing comma for longer expression, this change
will make sure that the Range parameters can still be parsed.
---------
Co-authored-by: command-bot <>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Bastian Köcher