Commit Graph

8941 Commits

Author SHA1 Message Date
gui ca7c01c8d8 Improve doc for pallet macro and config macro (#4146)
Improve doc:

* the pallet macro is actually referring to 2 places, for the module and
for the struct placeholder but doesn't really clarify it (I should have
named the latter just `pallet_struct` or something but it is a bit late)

* The doc of `with_default` is a bit confusing too IMO.

CC @kianenigma

---------

Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
2024-04-17 15:22:46 +00:00
Muharem 4e10d3b0a6 Asset Conversion: Pool Account ID derivation with additional Pallet ID seed (#3250)
Introduce `PalletId` as an additional seed parameter for pool's account
id derivation.

The PR also introduces the `pallet_asset_conversion_ops` pallet with a
call to migrate a given pool to thew new account. Additionally
`fungibles::lifetime::ResetTeam` and `fungible::lifetime::Refund`
traits, to facilitate the migration of pools.

---------

Co-authored-by: command-bot <>
2024-04-17 10:39:23 +00:00
Sergej Sakac e6f3106d89 XCM coretime region transfers (#3455)
This PR introduces changes enabling the transfer of coretime regions via
XCM.

TL;DR: There are two primary issues that are resolved in this PR:

1. The `mint` and `burn` functions were not implemented for coretime
regions. These operations are essential for moving assets to and from
the XCM holding register.
2. The transfer of non-fungible assets through XCM was previously
disallowed. This was due to incorrectly benchmarking non-fungible asset
transfers via XCM, which led to assigning it a weight of `Weight::Max`,
effectively preventing its execution.

### `mint_into` and `burn` implementation

This PR addresses the issue with cross-chain transferring regions back
to the Coretime chain. Remote reserve transfers are performed by
withdrawing and depositing the asset to and from the holding registry.
This requires the asset to support burning and minting functionality.

This PR adds burning and minting; however, they work a bit differently
than usual so that the associated region record is not lost when
burning. Instead of removing all the data, burning will set the owner of
the region to `None`, and when minting it back, it will set it to an
actual value. So, when cross-chain transferring, withdrawing into the
registry will remove the region from its original owner, and when
depositing it from the registry, it will set its owner to another
account

This was originally implemented in this PR: #3455, however we decided to
move all of it to this single PR
(https://github.com/paritytech/polkadot-sdk/pull/3455#discussion_r1547324892)

### Fixes made in this PR

- Update the `XcmReserveTransferFilter` on coretime chain since it is
meant as a reserve chain for coretime regions.
- Update the XCM benchmark to use `AssetTransactor` instead of assuming
`pallet-balances` for fungible transfers.
- Update the XCM benchmark to properly measure weight consumption for
nonfungible reserve asset transfers. ATM reserve transfers via the
extrinsic do not work since the weight for it is set to `Weight::max()`.

Closes: https://github.com/paritytech/polkadot-sdk/issues/865

---------

Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: Dónal Murray <donalm@seadanda.dev>
2024-04-17 09:25:33 +00:00
Alexandru Vasile 8fd839df9d grandpa: Send neighbor packet to lightclients with every finalized head (#4135)
This PR sends the GrandpaNeighbor packet to lightclients similarly to
the full-nodes.

Previously, the lightclient would receive a GrandpaNeigbor packet only
when the note set changed.

Related to: https://github.com/paritytech/polkadot-sdk/issues/4120

Next steps:
-  [ ] check with lightclient

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
2024-04-17 08:29:08 +00:00
PG Herveou e81322bc3e Contracts verify benchmark block (#4130)
Add verify statement to ensure that benchmarks call do not revert

Also updated
[benchmarks](https://weights.tasty.limo/compare?unit=time&ignore_errors=true&threshold=10&method=asymptotic&repo=polkadot-sdk&old=master&new=pg/verify-benchmarks&path_pattern=substrate%2Fframe%2Fcontracts%2Fsrc%2Fweights.rs)

---------

Co-authored-by: command-bot <>
2024-04-16 23:38:35 +00:00
Muharem 6f3d890ed3 FRAME: Unity Balance Conversion for Different IDs of Native Asset (#3659)
Introduce types to define 1:1 balance conversion for different relative
asset ids/locations of native asset.

Examples:
native asset on Asset Hub presented as `VersionedLocatableAsset` type in
the context of Relay Chain is
```
{
  `location`: (0, Parachain(1000)),
  `asset_id`: (1, Here),
}
```
and it's balance should be converted 1:1 by implementations of
`ConversionToAssetBalance` trait.

---------

Co-authored-by: Branislav Kontur <bkontur@gmail.com>
2024-04-16 16:11:14 +00:00
Oliver Tale-Yazdi 8891b70fe2 [pallet-broker] Use saturating math in input validation (#4151)
Changes:
- Saturate in the input validation of he drop history function or
pallet-broker.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2024-04-16 10:08:26 +00:00
gui a8f4f4f00f pallet assets: Fix errors (#4118)
`LiveAsset` is an error to be returned when an asset is not supposed to
be live.
And `AssetNotLive` is an error to be returned when an asset is supposed
to be live, I don't think frozen qualifies as live.
2024-04-15 18:45:44 +00:00
Dónal Murray 0c9ad5306c [pallet-broker] add tests for renewing leases (#4099)
The first test proves that parachains who were migrated over on a legacy
lease can renew without downtime.

The exception is if their lease expires in period 0 - aka within
`region_length` timeslices after `start_sales` is called. The second
test is designed such that it passes if the issue exists and should be
fixed.
This will require an intervention on Kusama to add these renewals to
storage as it is too tight to schedule a runtime upgrade before the
start_sales call. All leases will still have at least two full regions
of coretime.
2024-04-15 16:28:33 +00:00
Alexandru Vasile d1f9fe0a99 logging(fix): Use the proper log target for logging (#4124)
This PR ensures the proper logging target (ie `libp2p_tcp` or `beefy`)
is displayed.

The issue has been introduced in:
https://github.com/paritytech/polkadot-sdk/pull/4059, which removes the
normalized metadata of logs.

From
[documentation](https://docs.rs/tracing-log/latest/tracing_log/trait.NormalizeEvent.html#tymethod.normalized_metadata):

> In tracing-log, an Event produced by a log (through
[AsTrace](https://docs.rs/tracing-log/latest/tracing_log/trait.AsTrace.html))
has an hard coded “log” target

>
[normalized_metadata](https://docs.rs/tracing-log/latest/tracing_log/trait.NormalizeEvent.html#tymethod.normalized_metadata):
If this Event comes from a log, this method provides a new normalized
Metadata which has all available attributes from the original log,
including file, line, module_path and target

This has low implications if a version was deployed containing the
mentioned pull request, as we'll lose the ability to distinguish between
log targets.

### Before this PR

```
2024-04-15 12:45:40.327  INFO main log: Parity Polkadot
2024-04-15 12:45:40.328  INFO main log: ✌️  version 1.10.0-d1b0ef76a8b
2024-04-15 12:45:40.328  INFO main log: ❤️  by Parity Technologies <admin@parity.io>, 2017-2024
2024-04-15 12:45:40.328  INFO main log: 📋 Chain specification: Development
2024-04-15 12:45:40.328  INFO main log: 🏷  Node name: yellow-eyes-2963
2024-04-15 12:45:40.328  INFO main log: 👤 Role: AUTHORITY
2024-04-15 12:45:40.328  INFO main log: 💾 Database: RocksDb at /tmp/substrated39i9J/chains/rococo_dev/db/full
2024-04-15 12:45:44.508  WARN main log: Took active validators from set with wrong size
...

2024-04-15 12:45:45.805  INFO                 main log: 👶 Starting BABE Authorship worker
2024-04-15 12:45:45.806  INFO tokio-runtime-worker log: 🥩 BEEFY gadget waiting for BEEFY pallet to become available...
2024-04-15 12:45:45.806 DEBUG tokio-runtime-worker log: New listen address: /ip6/::1/tcp/30333
2024-04-15 12:45:45.806 DEBUG tokio-runtime-worker log: New listen address: /ip4/127.0.0.1/tcp/30333
```

### After this PR

```
2024-04-15 12:59:45.623  INFO main sc_cli:🏃 Parity Polkadot
2024-04-15 12:59:45.623  INFO main sc_cli:🏃 ✌️  version 1.10.0-d1b0ef76a8b
2024-04-15 12:59:45.623  INFO main sc_cli:🏃 ❤️  by Parity Technologies <admin@parity.io>, 2017-2024
2024-04-15 12:59:45.623  INFO main sc_cli:🏃 📋 Chain specification: Development
2024-04-15 12:59:45.623  INFO main sc_cli:🏃 🏷  Node name: helpless-lizards-0550
2024-04-15 12:59:45.623  INFO main sc_cli:🏃 👤 Role: AUTHORITY
...
2024-04-15 12:59:50.204  INFO tokio-runtime-worker beefy: 🥩 BEEFY gadget waiting for BEEFY pallet to become available...
2024-04-15 12:59:50.204 DEBUG tokio-runtime-worker libp2p_tcp: New listen address: /ip6/::1/tcp/30333
2024-04-15 12:59:50.204 DEBUG tokio-runtime-worker libp2p_tcp: New listen address: /ip4/127.0.0.1/tcp/30333
```

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
2024-04-15 14:33:55 +00:00
Bastian Köcher 6f73b746d3 pallet_broker: Support renewing leases expired in a previous period (#4089)
Part of: https://github.com/paritytech/polkadot-sdk/issues/4107
2024-04-15 13:06:22 +00:00
Bastian Köcher d1b0ef76a8 sp-api: Use macro to detect if frame-metadata is enabled (#4117)
While `sp-api-proc-macro` isn't used directly and thus, it should have
the same features enabled as `sp-api`. However, I have seen issues
around `frame-metadata` not being enabled for `sp-api`, but for
`sp-api-proc-macro`. This can be prevented by using the
`frame_metadata_enabled` macro from `sp-api` that ensures we have the
same feature set between both crates.
2024-04-15 08:11:51 +00:00
Alexandru Gheorghe 2bc4ed1153 Prevent accidental change of network-key for active authorities (#3852)
As discovered during investigation of
https://github.com/paritytech/polkadot-sdk/issues/3314 and
https://github.com/paritytech/polkadot-sdk/issues/3673 there are active
validators which accidentally might change their network key during
restart, that's not a safe operation when you are in the active set
because of distributed nature of DHT, so the old records would still
exist in the network until they expire 36h, so unless they have a good
reason validators should avoid changing their key when they restart
their nodes.

There is an effort in parallel to improve this situation
https://github.com/paritytech/polkadot-sdk/pull/3786, but those changes
are way more intrusive and will need more rigorous testing, additionally
they will reduce the time to less than 36h, but the propagation won't be
instant anyway, so not changing your network during restart should be
the safest way to run your node, unless you have a really good reason to
change it.

## Proposal
1. Do not auto-generate the network if the network file does not exist
in the provided path. Nodes where the key file does not exist will get
the following error:
```
Error: 
   0: Starting an authorithy without network key in /home/alexggh/.local/share/polkadot/chains/ksmcc3/network/secret_ed25519.
      
       This is not a safe operation because the old identity still lives in the dht for 36 hours.
      
       Because of it your node might suffer from not being properly connected to other nodes for validation purposes.
      
       If it is the first time running your node you could use one of the following methods.
      
       1. Pass --unsafe-force-node-key-generation and make sure you remove it for subsequent node restarts
      
       2. Separetly generate the key with: polkadot key generate-node-key --file <YOUR_PATH_TO_NODE_KEY>
```

2. Add an explicit parameters for nodes that do want to change their
network despite the warnings or if they run the node for the first time.
`--unsafe-force-node-key-generation`

3. For `polkadot key generate-node-key` add two new mutually exclusive
parameters `base_path` and `default_base_path` to help with the key
generation in the same path the polkadot main command would expect it.
 
4. Modify the installation scripts to auto-generate a key in default
path if one was not present already there, this should help with making
the executable work out of the box after an instalation.

## Notes

Nodes that do not have already the key persisted will fail to start
after this change, however I do consider that better than the current
situation where they start but they silently hide that they might not be
properly connected to their peers.

## TODO
- [x] Make sure only nodes that are authorities on producation chains
will be affected by this restrictions.
- [x] Proper PRDOC, to make sure node operators are aware this is
coming.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
2024-04-15 06:23:35 +00:00
Oliver Tale-Yazdi 30c58fa22a Deploy pallet-parameters to rococo and fix dynamic_params name expand (#4006)
Changes:
- Add pallet-parameters to Rococo to configure the NIS and preimage
pallet.
- Fix names of expanded dynamic params. Apparently, `to_class_case`
removes suffix `s`, and `Nis` becomes `Ni` 😑. Now using
`to_pascal_case`.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Alessandro Siniscalchi <asiniscalchi@gmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: command-bot <>
2024-04-13 11:20:42 +00:00
Bastian Köcher 7c698502d1 sc_network_test: Announce only the highest block (#4111)
Closes: https://github.com/paritytech/polkadot-sdk/issues/4100
2024-04-13 08:40:10 +00:00
Vedhavyas Singareddi 5b513cc0e9 define block hash provider and default impl using frame_system (#4080)
This PR introduces `BlockHashProvider` into `pallet_mmr::Config`
This type is used to get `block_hash` for a given `block_number` rather
than directly using `frame_system::Pallet::block_hash`

The `DefaultBlockHashProvider` uses `frame_system::Pallet::block_hash`
to get the `block_hash`

Closes: #4062
2024-04-12 21:57:05 +00:00
Squirrel b28ba4ae96 Remove redundent logging code (#4059)
1. The `CustomFmtContext::ContextWithFormatFields` enum arm isn't
actually used and thus we don't need the enum anymore.

2. We don't do anything with most of the normalized metadata that's
created by calling `event.normalized_metadata();` - the `target` we can
get from `event.metadata.target()` and level we can get from
`event.metadata.level()` - let's just call them direct to simplify
things. (`event.metadata()` is just a field call under the hood)

Changelog: No functional changes, might run a tad faster with lots of
logging turned on.

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2024-04-12 21:18:48 +00:00
eskimor a64009ae00 Improve docs of broker pallet (#3980)
Small adjustments which should make understanding what is going on much
easier for future readers.

Initialization is a bit messy, the very least we should do is adding
documentation to make it harder to use wrongly.

I was thinking about calling `request_core_count` right from
`start_sales`, but as explained in the docs, this is not necessarily
what you want.

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Dónal Murray <donal.murray@parity.io>
2024-04-12 10:23:49 +00:00
PG Herveou 033484c3ba Use defaultConfig for pallet_contracts (#1817) 2024-04-12 09:35:17 +00:00
Liam Aharon 39b1f50f1c Remove deprecated TryRuntime subcommand (#4017)
Completes the removal of `try-runtime-cli` logic from `polkadot-sdk`.
2024-04-11 20:01:16 +00:00
Oliver Tale-Yazdi 832570545b Fix link check (#4074)
Closes #4041

Changes:
- Increase cache size and reduce retries.
- Ignore Substrate SE links :(
- Fix broken link.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2024-04-11 10:55:42 +00:00
PG Herveou 643aa2be2a Contracts: Remove ED from base deposit (#3536)
- Update internal logic so that the storage_base_deposit does not
include ED
- add v16 migration to update ContractInfo struct with this change

Before:
<img width="820" alt="Screenshot 2024-03-21 at 11 23 29"
src="https://github.com/paritytech/polkadot-sdk/assets/521091/a0a8df0d-e743-42c5-9e16-cf2ec1aa949c">

After:
![Screenshot 2024-03-21 at 11 23
42](https://github.com/paritytech/polkadot-sdk/assets/521091/593235b0-b866-4915-b653-2071d793228b)

---------

Co-authored-by: Cyrill Leutwiler <cyrill@parity.io>
Co-authored-by: command-bot <>
2024-04-10 20:32:53 +00:00
PG Herveou 1da8d12dd1 Contracts: Reuse module when validating (#3789)
Reuse wasmi Module when validating.
Prepare the code for 0.32 and the addition of Module::new_unchecked
2024-04-10 18:56:51 +00:00
Alexandru Vasile cd010925e1 net/strategy: Log bad peerId from on_validated_block_announce (#4051)
This tiny PR extends the `on_validated_block_announce` log with the bad
PeerID.
Used to identify if the peerID is malicious by correlating with other
logs (ie peer-set).

While at it, have removed the `\n` from a multiline log, which did not
play well with
[sub-triage-logs](https://github.com/lexnv/sub-triage-logs/tree/master).

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
2024-04-10 15:29:36 +00:00
PG Herveou 0d71753e0e Contracts: Only exec parsed code in benchmarks (#3915)
[Weights
compare](https://weights.tasty.limo/compare?unit=weight&ignore_errors=true&threshold=10&method=asymptotic&repo=polkadot-sdk&old=master&new=pg%2Fbench_tweaks&path_pattern=substrate%2Fframe%2F**%2Fsrc%2Fweights.rs%2Cpolkadot%2Fruntime%2F*%2Fsrc%2Fweights%2F**%2F*.rs%2Cpolkadot%2Fbridges%2Fmodules%2F*%2Fsrc%2Fweights.rs%2Ccumulus%2F**%2Fweights%2F*.rs%2Ccumulus%2F**%2Fweights%2Fxcm%2F*.rs%2Ccumulus%2F**%2Fsrc%2Fweights.rs)

Note: Raw weights change does not mean much here, as this PR reduce the
scope of what is benchmarked, they are therefore decreased by a good
margin. One should instead print the Schedule using

cargo test --features runtime-benchmarks bench_print_schedule --
--nocapture
or following the instructions from the
[README](https://github.com/paritytech/polkadot-sdk/tree/pg/bench_tweaks/substrate/frame/contracts#schedule)
for looking at the Schedule of a specific runtime

---------

Co-authored-by: command-bot <>
2024-04-10 13:05:34 +00:00
Serban Iorga fb84c4618a Fix chain_spec_as_json_fails_with_invalid_config 2024-04-10 10:28:37 +02:00
PG Herveou d38f6e6728 Update benchmarking macros (#3934)
Current benchmarking macro returns a closure with the captured
benchmarked code.
This can cause issues when the benchmarked code has complex lifetime
requirements.

This PR updates the existing macro by injecting the recording parameter
and invoking the start / stop method around the benchmarked block
instead of returning a closure

One other added benefit is that you can write this kind of code now as
well:

```rust
let v;
#[block]
{ v = func.call(); }
dbg!(v); // or assert something on v
```


[Weights compare
link](https://weights.tasty.limo/compare?unit=weight&ignore_errors=true&threshold=10&method=asymptotic&repo=polkadot-sdk&old=pg/fix-weights&new=pg/bench_update&path_pattern=substrate/frame/**/src/weights.rs,polkadot/runtime/*/src/weights/**/*.rs,polkadot/bridges/modules/*/src/weights.rs,cumulus/**/weights/*.rs,cumulus/**/weights/xcm/*.rs,cumulus/**/src/weights.rs)

---------

Co-authored-by: command-bot <>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
2024-04-10 06:44:46 +00:00
PG Herveou 2d927b0772 Contracts: Fix legacy uapi (#3994)
Fix some broken legacy definitions of pallet_contracts_uapi storage host
functions
2024-04-10 05:05:21 +00:00
Sebastian Kunert df818d2974 Move cumulus zombienet tests to aura & async backing (#3568)
Cumulus test-parachain node and test runtime were still using relay
chain consensus and 12s blocktimes. With async backing around the corner
on the major chains we should switch our tests too.

Also needed to nicely test the changes coming to collators in #3168.

### Changes Overview
- Followed the [migration
guide](https://wiki.polkadot.network/docs/maintain-guides-async-backing)
for async backing for the cumulus-test-runtime
- Adjusted the cumulus-test-service to use the correct import-queue,
lookahead collator etc.
- The block validation function now uses the Aura Ext Executor so that
the seal of the block is validated
- Previous point requires that we seal block before calling into
`validate_block`, I introduced a helper function for that
- Test client adjusted to provide a slot to the relay chain proof and
the aura pre-digest
2024-04-09 16:53:30 +00:00
Alexandru Vasile 598e95577d rpc-v2/transaction: Generate Invalid events and add tests (#3784)
This PR ensures that the transaction API generates an `Invalid` events
for transaction bytes that fail to decode.

The spec mentioned the `Invalid` event at the jsonrpc error section,
however this spec PR makes things clearer:
- https://github.com/paritytech/json-rpc-interface-spec/pull/146

While at it have discovered an inconsistency with the generated events.
The drop event from the transaction pool was incorrectly mapped to the
`invalid` event.

Added tests for the API stabilize the API soon:
- https://github.com/paritytech/json-rpc-interface-spec/pull/144


Closes: https://github.com/paritytech/polkadot-sdk/issues/3083


cc @paritytech/subxt-team

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
2024-04-09 13:57:44 +00:00
Dmitry Markin a26d25d5c7 Detect closed notification substreams instead of evicting all peers (#3983)
This PR brings the fix
https://github.com/paritytech/substrate/pull/13396 to polkadot-sdk.

In the past, due to insufficient inbound slot count on polkadot &
kusama, this fix led to low peer count. The situation has improved since
then after changing the default ratio between `--in-peers` &
`--out-peers`.

Nevertheless, it's expected that the reported total peer count with this
fix is going to be lower than without it. This should be seen as the
correct number of working connections reported, as opposed to also
reporting already closed connections, and not as lower count of working
connections with peers.

This PR also removes the peer eviction mechanism, as closed substream
detection is a more granular way of detecting peers that stopped syncing
with us.

The burn-in has been already performed as part of testing these changes
in https://github.com/paritytech/polkadot-sdk/pull/3426.

---------

Co-authored-by: Aaro Altonen <a.altonen@hotmail.com>
2024-04-09 12:40:52 +00:00
Alexandru Vasile b1c9209a6c peer_store: Increase peer ban time until escapes banned threshold (#4031)
This is a tiny PR to increase the time a peer remains banned.

A peer is banned when the reputation drops below a threshold.
With every second, the peer reputation is exponentially decayed towards
zero.

For the previous setup:
- decaying to zero from (i32::MAX or i32::MIN) would take 948 seconds
(15mins 48seconds)
- from i32::MIN to escaping the banned threshold would take 10 seconds
This means we are decaying reputation a bit too aggressive and
misbehaving peers can misbehave again in 10 seconds.
Another side effect of this is that we have encountered multiple
warnings caused by a few misbehaving peers.

In the new setup:
- decaying to zero from (i32::MAX or i32::MIN) would take 3544 seconds
(59 minutes)
- from i32::MIN to escaping the banned threshold would take ~69 seconds

This is a followup of:
- https://github.com/paritytech/polkadot-sdk/pull/4000.

### Testing Done
- Created a misbehaving client with
[subp2p-explorer](https://github.com/lexnv/subp2p-explorer), the client
is banned for approx 69seconds until it is allowed to connect again.

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
2024-04-09 11:24:49 +00:00
Facundo Farall 4e73c0fcd3 Upgrade trie-db from 0.28.0 to 0.29.0 (#3982)
# Description
- What does this PR do?
1. Upgrades `trie-db`'s version to the latest release. This release
includes, among others, an implementation of `DoubleEndedIterator` for
the `TrieDB` struct, allowing to iterate both backwards and forwards
within the leaves of a trie.
2. Upgrades `trie-bench` to `0.39.0` for compatibility.
3. Upgrades `criterion` to `0.5.1` for compatibility.
- Why are these changes needed?
Besides keeping up with the upgrade of `trie-db`, this specifically adds
the functionality of iterating back on the leafs of a trie, with
`sp-trie`. In a project we're currently working on, this comes very
handy to verify a Merkle proof that is the response to a challenge. The
challenge is a random hash that (most likely) will not be an existing
leaf in the trie. So the challenged user, has to provide a Merkle proof
of the previous and next existing leafs in the trie, that surround the
random challenged hash.

Without having DoubleEnded iterators, we're forced to iterate until we
find the first existing leaf, like so:
```rust
        // ************* VERIFIER (RUNTIME) *************
        // Verify proof. This generates a partial trie based on the proof and
        // checks that the root hash matches the `expected_root`.
        let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
        let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();

        // Print all leaf node keys and values.
        println!("\nPrinting leaf nodes of partial tree...");
        for key in trie.key_iter().unwrap() {
            if key.is_ok() {
                println!("Leaf node key: {:?}", key.clone().unwrap());

                let val = trie.get(&key.unwrap());

                if val.is_ok() {
                    println!("Leaf node value: {:?}", val.unwrap());
                } else {
                    println!("Leaf node value: None");
                }
            }
        }

        println!("RECONSTRUCTED TRIE {:#?}", trie);

        // Create an iterator over the leaf nodes.
        let mut iter = trie.iter().unwrap();

        // First element with a value should be the previous existing leaf to the challenged hash.
        let mut prev_key = None;
        for element in &mut iter {
            if element.is_ok() {
                let (key, _) = element.unwrap();
                prev_key = Some(key);
                break;
            }
        }
        assert!(prev_key.is_some());

        // Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
        assert!(prev_key.unwrap() <= challenge_hash.to_vec());

        // The next element should exist (meaning there is no other existing leaf between the
        // previous and next leaf) and it should be greater than the challenged hash.
        let next_key = iter.next().unwrap().unwrap().0;
        assert!(next_key >= challenge_hash.to_vec());
```

With DoubleEnded iterators, we can avoid that, like this:
```rust
        // ************* VERIFIER (RUNTIME) *************
        // Verify proof. This generates a partial trie based on the proof and
        // checks that the root hash matches the `expected_root`.
        let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
        let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();

        // Print all leaf node keys and values.
        println!("\nPrinting leaf nodes of partial tree...");
        for key in trie.key_iter().unwrap() {
            if key.is_ok() {
                println!("Leaf node key: {:?}", key.clone().unwrap());

                let val = trie.get(&key.unwrap());

                if val.is_ok() {
                    println!("Leaf node value: {:?}", val.unwrap());
                } else {
                    println!("Leaf node value: None");
                }
            }
        }

        // println!("RECONSTRUCTED TRIE {:#?}", trie);
        println!("\nChallenged key: {:?}", challenge_hash);

        // Create an iterator over the leaf nodes.
        let mut double_ended_iter = trie.into_double_ended_iter().unwrap();

        // First element with a value should be the previous existing leaf to the challenged hash.
        double_ended_iter.seek(&challenge_hash.to_vec()).unwrap();
        let next_key = double_ended_iter.next_back().unwrap().unwrap().0;
        let prev_key = double_ended_iter.next_back().unwrap().unwrap().0;

        // Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
        println!("Prev key: {:?}", prev_key);
        assert!(prev_key <= challenge_hash.to_vec());

        println!("Next key: {:?}", next_key);
        assert!(next_key >= challenge_hash.to_vec());
```
- How were these changes implemented and what do they affect?
All that is needed for this functionality to be exposed is changing the
version number of `trie-db` in all the `Cargo.toml`s applicable, and
re-exporting some additional structs from `trie-db` in `sp-trie`.

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2024-04-09 10:47:33 +00:00
PG Herveou b6231c79ca Contracts: Refactor API to use WeightMeter (#2943)
Update the Contracts API to use `WeightMeter`, as it simplifies the code
and makes it easier to reason about, rather than taking a mutable weight
or returning a tuple with the weight consumed

---------

Co-authored-by: Alexander Theißen <alex.theissen@me.com>
2024-04-09 10:22:54 +00:00
Ankan 10ed76437f Nomination pool configurations can be managed by custom origin (#3959)
closes https://github.com/paritytech/polkadot-sdk/issues/3894

Allows Nomination Pool configuration to be set by a custom origin
instead of root.

In runtimes, we would set this to be `StakingAdmin`, same as for
pallet-staking.

---------

Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
2024-04-09 10:14:19 +00:00
gupnik 0e27b881a4 Fixes validation for SkipCheckIfFeeless extension (#3993)
During validation, `SkipCheckIfFeeless` should check if the call is
`feeless` and delegate to the wrapped extension if not.
2024-04-09 08:35:46 +00:00
Léa Narzis d733c77ee2 Adapt RemoteExternalities and its related types to be used with generic hash parameters (#3953)
Closes  https://github.com/paritytech/polkadot-sdk/issues/3737

---------

Co-authored-by: command-bot <>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
2024-04-08 21:56:41 +00:00
Aaro Altonen 80616f6d03 Integrate litep2p into Polkadot SDK (#2944)
[litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P
networking library. It supports all of the features of `rust-libp2p`
that are currently being utilized by Polkadot SDK.

Compared to `rust-libp2p`, `litep2p` has a quite different architecture
which is why the new `litep2p` network backend is only able to use a
little of the existing code in `sc-network`. The design has been mainly
influenced by how we'd wish to structure our networking-related code in
Polkadot SDK: independent higher-levels protocols directly communicating
with the network over links that support bidirectional backpressure. A
good example would be `NotificationHandle`/`RequestResponseHandle`
abstractions which allow, e.g., `SyncingEngine` to directly communicate
with peers to announce/request blocks.

I've tried running `polkadot --network-backend litep2p` with a few
different peer configurations and there is a noticeable reduction in
networking CPU usage. For high load (`--out-peers 200`), networking CPU
usage goes down from ~110% to ~30% (80 pp) and for normal load
(`--out-peers 40`), the usage goes down from ~55% to ~18% (37 pp).

These should not be taken as final numbers because:

a) there are still some low-hanging optimization fruits, such as
enabling [receive window
auto-tuning](https://github.com/libp2p/rust-yamux/pull/176), integrating
`Peerset` more closely with `litep2p` or improving memory usage of the
WebSocket transport
b) fixing bugs/instabilities that incorrectly cause `litep2p` to do less
work will increase the networking CPU usage
c) verification in a more diverse set of tests/conditions is needed

Nevertheless, these numbers should give an early estimate for CPU usage
of the new networking backend.

This PR consists of three separate changes:
* introduce a generic `PeerId` (wrapper around `Multihash`) so that we
don't have use `NetworkService::PeerId` in every part of the code that
uses a `PeerId`
* introduce `NetworkBackend` trait, implement it for the libp2p network
stack and make Polkadot SDK generic over `NetworkBackend`
  * implement `NetworkBackend` for litep2p

The new library should be considered experimental which is why
`rust-libp2p` will remain as the default option for the time being. This
PR currently depends on the master branch of `litep2p` but I'll cut a
new release for the library once all review comments have been
addresses.

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Alexandru Vasile <alexandru.vasile@parity.io>
2024-04-08 16:44:13 +00:00
Oliver Tale-Yazdi 9543d31474 [FRAME] Runtime Omni Bencher (#3512)
This MR contains two major changes and some maintenance cleanup.  

## 1. Free Standing Pallet Benchmark Runner

Closes https://github.com/paritytech/polkadot-sdk/issues/3045, depends
on your runtime exposing the `GenesisBuilderApi` (like
https://github.com/paritytech/polkadot-sdk/pull/1492).

Introduces a new binary crate: `frame-omni-bencher`.  
It allows to directly benchmark a WASM blob - without needing a node or
chain spec.

This makes it much easier to generate pallet weights and should allow us
to remove bloaty code from the node.
It should work for all FRAME runtimes that dont use 3rd party host calls
or non `BlakeTwo256` block hashing (basically all polkadot parachains
should work).

It is 100% backwards compatible with the old CLI args, when the `v1`
compatibility command is used. This is done to allow for forwards
compatible addition of new commands.

### Example (full example in the Rust docs)

Installing the CLI:
```sh
cargo install --locked --path substrate/utils/frame/omni-bencher
frame-omni-bencher --help
```

Building the Westend runtime:
```sh
cargo build -p westend-runtime --release --features runtime-benchmarks
```

Benchmarking the runtime:
```sh
frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all
```

## 2. Building the Benchmark Genesis State in the Runtime

Closes https://github.com/paritytech/polkadot-sdk/issues/2664

This adds `--runtime` and `--genesis-builder=none|runtime|spec`
arguments to the `benchmark pallet` command to make it possible to
generate the genesis storage by the runtime. This can be used with both
the node and the freestanding benchmark runners. It utilizes the new
`GenesisBuilder` RA and depends on having
https://github.com/paritytech/polkadot-sdk/pull/3412 deployed.

## 3. Simpler args for `PalletCmd::run`

You can do three things here to integrate the changes into your node:
- nothing: old code keeps working as before but emits a deprecated
warning
- delete: remove the pallet benchmarking code from your node and use the
omni-bencher instead
- patch: apply the patch below and keep using as currently. This emits a
deprecated warning at runtime, since it uses the old way to generate a
genesis state, but is the smallest change.

```patch
runner.sync_run(|config| cmd
-    .run::<HashingFor<Block>, ReclaimHostFunctions>(config)
+    .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec))
)
```

## 4. Maintenance Change
- `pallet-nis` get a `BenchmarkSetup` config item to prepare its
counterparty asset.
- Add percent progress print when running benchmarks.
- Dont immediately exit on benchmark error but try to run as many as
possible and print errors last.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
2024-04-08 16:03:56 +00:00
Sebastian Kunert fdb1dba2e1 Add best block indicator to informant message + print parent block on import message (#4021)
Sometimes you need to debug some issues just by the logs and reconstruct
what happened.
In these scenarios it would be nice to know if a block was imported as
best block, and what it parent was.
So here I propose to change the output of the informant to this:

```
2024-04-05 20:38:22.004  INFO ⋮substrate: [Parachain]  Imported #18 (0xe7b3…4555 -> 0xbd6f…ced7)    
2024-04-05 20:38:24.005  INFO ⋮substrate: [Parachain]  Imported #19 (0xbd6f…ced7 -> 0x4dd0…d81f)    
2024-04-05 20:38:24.011  INFO ⋮substrate: [jobless-children-5352] 🌟 Imported #42 (0xed2e…27fc -> 0x718f…f30e)    
2024-04-05 20:38:26.005  INFO ⋮substrate: [Parachain]  Imported #20 (0x4dd0…d81f -> 0x6e85…e2b8)    
2024-04-05 20:38:28.004  INFO ⋮substrate: [Parachain] 🌟 Imported #21 (0x6e85…e2b8 -> 0xad53…2a97)    
2024-04-05 20:38:30.004  INFO ⋮substrate: [Parachain] 🌟 Imported #22 (0xad53…2a97 -> 0xa874…890f)    
```

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2024-04-08 13:30:32 +00:00
Bastian Köcher c1063a530e sc-beefy-consensus: Remove unneeded stream. (#4015)
The stream was just used to communicate from the validator the peer
reports back to the gossip engine. Internally the gossip engine just
forwards these reports to the networking engine. So, we can just do this
directly.

The reporting stream was also pumped [in the worker behind the
engine](https://github.com/paritytech/polkadot-sdk/blob/9d6261892814fa27c97881c0321c008d7340b54b/substrate/client/consensus/beefy/src/worker.rs#L939).
This means if there was a lot of data incoming over the engine, the
reporting stream was almost never processed and thus, it could have
started to grow and we have seen issues around this.

Partly Closes: https://github.com/paritytech/polkadot-sdk/issues/3945
2024-04-08 08:28:42 +00:00
HongKuang bd4471b4fc Fix some typos (#4018)
Signed-off-by: hongkuang <liurenhong@outlook.com>
2024-04-08 04:21:11 +00:00
Squirrel 9940038543 Major bump of tracing-subscriber version (#3891)
I don't think there are any more releases to the 0.2.x versions, so best
we're on the 0.3.x release.

No change on the benchmarks, fast local time is still just as fast as
before:

new version bench:
```
fast_local_time         time:   [30.551 ns 30.595 ns 30.668 ns]
```

old version bench:
```
fast_local_time         time:   [30.598 ns 30.646 ns 30.723 ns]
```

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2024-04-06 13:54:09 +00:00
Liam Aharon 74d6309c0c Improve frame umbrella crate doc experience (#4007)
1. Add `#[doc(no_inline)]` to frame umbrella crate re-exports that
eventually resolve to `frame_support_procedural` so docs don't look like
the screenshot below and instead link to the proper `frame-support`
docs.
<img width="1512" alt="Screenshot 2024-04-05 at 20 05 01"
src="https://github.com/paritytech/polkadot-sdk/assets/16665596/a41daa4c-ebca-44a4-9fea-f9f336314e13">


2. Remove `"Rust-Analyzer Users: "` prefix from
`frame_support_procedural` doc comments, since these doc comments are
visible in the web documentation and possible to stumble upon especially
when navigating from the frame umbrella crate.

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2024-04-06 10:02:37 +00:00
Liam Aharon 9d62618928 pallet-uniques: decrement total_deposit when clearing collection metadata (#3976)
Decrements `total_deposit` when collection metadata is cleared in
`pallet-nfts` and `pallet-uniques`.
2024-04-06 05:10:46 +00:00
Sergej Sakac 1c85bfe901 Broker: sale price runtime api (#3485)
Defines a runtime api for `pallet-broker` for getting the current price
of a core if there is an ongoing sale.

Closes: #3413

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2024-04-05 23:29:35 +00:00
divdeploy a0eed0a6f8 chore: fix some comments (#4004)
Signed-off-by: divdeploy <chenguangxue@outlook.com>
2024-04-05 18:35:57 +00:00
Dónal Murray ba0f8de0c7 [pallet-broker] Fix claim revenue behaviour for zero timeslices (#3997)
This PR adds a check that `max_timeslices > 0` and errors if not. It
also adds a test for this behaviour and cleans up some misleading docs.
2024-04-05 13:18:48 +00:00
Michal Kucharczyk f910a15c1c GenesisConfig presets for runtime (#2714)
The runtime now can provide a number of predefined presets of
`RuntimeGenesisConfig` struct. This presets are intended to be used in
different deployments, e.g.: `local`, `staging`, etc, and should be
included into the corresponding chain-specs.

Having `GenesisConfig` presets in runtime allows to fully decouple node
from runtime types (the problem is described in #1984).

**Summary of changes:**
- The `GenesisBuilder` API was adjusted to enable this functionality
(and provide better naming - #150):
   ```rust
    fn preset_names() -> Vec<PresetId>;
fn get_preset(id: Option<PresetId>) -> Option<serde_json::Value>;
//`None` means default
    fn build_state(value: serde_json::Value);
    pub struct PresetId(Vec<u8>);
   ```

- **Breaking change**: Old `create_default_config` method was removed,
`build_config` was renamed to `build_state`. As a consequence a node
won't be able to interact with genesis config for older runtimes. The
cleanup was made for sake of API simplicity. Also IMO maintaining
compatibility with old API is not so crucial.
- Reference implementation was provided for `substrate-test-runtime` and
`rococo` runtimes. For rococo new
[`genesis_configs_presets`](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b97c5ff0ec4a1989da5ffd8b9f3f588e3/polkadot/runtime/rococo/src/genesis_config_presets.rs#L530)
module was added and is used in `GenesisBuilder`
[_presets-related_](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b97c5ff0ec4a1989da5ffd8b9f3f588e3/polkadot/runtime/rococo/src/lib.rs#L2462-L2485)
methods.

- The `chain-spec-builder` util was also improved and allows to
([_doc_](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b97c5ff0ec4a1989da5ffd8b9f3f588e3/substrate/bin/utils/chain-spec-builder/src/lib.rs#L19)):
   - list presets provided by given runtime (`list-presets`),
- display preset or default config provided by the runtime
(`display-preset`),
   - build chain-spec using named preset (`create ... named-preset`),


- The `ChainSpecBuilder` is extended with
[`with_genesis_config_preset_name`](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b97c5ff0ec4a1989da5ffd8b9f3f588e3/substrate/client/chain-spec/src/chain_spec.rs#L447)
method which allows to build chain-spec using named preset provided by
the runtime. Sample usage on the node side
[here](https://github.com/paritytech/polkadot-sdk/blob/2caffaae803e08a3d5b46c860e8016da023ff4ce/polkadot/node/service/src/chain_spec.rs#L404).

Implementation of #1984.
fixes: #150
part of: #25

---------

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2024-04-04 18:30:54 +00:00
Shawn Tabrizi 9d052b7e09 Make BlockNumberProvider::set_block_number available in tests / std (#3974)
This function is currently only exposed to runtime-benchmarks, where it
should be available in all tests.

Also made it available in `std` because this is how `set_block_number`
works in FRAME System, however, not sure if it is needed in the latest
std/no_std paradigm.

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2024-04-04 16:20:09 +00:00