Commit Graph

248 Commits

Author SHA1 Message Date
Aaro Altonen 80616f6d03 Integrate litep2p into Polkadot SDK (#2944)
[litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P
networking library. It supports all of the features of `rust-libp2p`
that are currently being utilized by Polkadot SDK.

Compared to `rust-libp2p`, `litep2p` has a quite different architecture
which is why the new `litep2p` network backend is only able to use a
little of the existing code in `sc-network`. The design has been mainly
influenced by how we'd wish to structure our networking-related code in
Polkadot SDK: independent higher-levels protocols directly communicating
with the network over links that support bidirectional backpressure. A
good example would be `NotificationHandle`/`RequestResponseHandle`
abstractions which allow, e.g., `SyncingEngine` to directly communicate
with peers to announce/request blocks.

I've tried running `polkadot --network-backend litep2p` with a few
different peer configurations and there is a noticeable reduction in
networking CPU usage. For high load (`--out-peers 200`), networking CPU
usage goes down from ~110% to ~30% (80 pp) and for normal load
(`--out-peers 40`), the usage goes down from ~55% to ~18% (37 pp).

These should not be taken as final numbers because:

a) there are still some low-hanging optimization fruits, such as
enabling [receive window
auto-tuning](https://github.com/libp2p/rust-yamux/pull/176), integrating
`Peerset` more closely with `litep2p` or improving memory usage of the
WebSocket transport
b) fixing bugs/instabilities that incorrectly cause `litep2p` to do less
work will increase the networking CPU usage
c) verification in a more diverse set of tests/conditions is needed

Nevertheless, these numbers should give an early estimate for CPU usage
of the new networking backend.

This PR consists of three separate changes:
* introduce a generic `PeerId` (wrapper around `Multihash`) so that we
don't have use `NetworkService::PeerId` in every part of the code that
uses a `PeerId`
* introduce `NetworkBackend` trait, implement it for the libp2p network
stack and make Polkadot SDK generic over `NetworkBackend`
  * implement `NetworkBackend` for litep2p

The new library should be considered experimental which is why
`rust-libp2p` will remain as the default option for the time being. This
PR currently depends on the master branch of `litep2p` but I'll cut a
new release for the library once all review comments have been
addresses.

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Alexandru Vasile <alexandru.vasile@parity.io>
2024-04-08 16:44:13 +00:00
Serban Iorga 8e95a3e1aa Align dependencies with parity-bridges-common (#3937)
Working towards migrating the `parity-bridges-common` repo inside
`polkadot-sdk`. This PR upgrades some dependencies in order to align
them with the versions used in `parity-bridges-common`

Related to
https://github.com/paritytech/parity-bridges-common/issues/2538
2024-04-02 13:41:01 +00:00
Davide Galassi 1e9fd23776 Implement crypto byte array newtypes in term of a shared type (#3684)
Introduces `CryptoBytes` type defined as:

```rust
pub struct CryptoBytes<const N: usize, Tag = ()>(pub [u8; N], PhantomData<fn() -> Tag>);
```

The type implements a bunch of methods and traits which are typically
expected from a byte array newtype
(NOTE: some of the methods and trait implementations IMO are a bit
redundant, but I decided to maintain them all to not change too much
stuff in this PR)

It also introduces two (generic) typical consumers of `CryptoBytes`:
`PublicBytes` and `SignatureBytes`.

```rust
pub struct PublicTag;
pub PublicBytes<const N: usize, CryptoTag> = CryptoBytes<N, (PublicTag, CryptoTag)>;

pub struct SignatureTag;
pub SignatureBytes<const N: usize, CryptoTag> = CryptoBytes<N, (SignatureTag, CryptoTag)>;
```

Both of them use a tag to differentiate the two types at a higher level.
Downstream specializations will further specialize using a dedicated
crypto tag. For example in ECDSA:


```rust
pub struct EcdsaTag;

pub type Public = PublicBytes<PUBLIC_KEY_SERIALIZED_SIZE, EcdsaTag>;
pub type Signature = PublicBytes<PUBLIC_KEY_SERIALIZED_SIZE, EcdsaTag>;
```

Overall we have a cleaner and most importantly **consistent** code for
all the types involved

All these details are opaque to the end user which can use `Public` and
`Signature` for the cryptos as before
2024-03-19 15:47:42 +00:00
Niklas Adolfsson fee810a5ea rpc server: make possible to disable/enable batch requests (#3364)
The rationale behind this, is that it may be useful for some users
actually disable RPC batch requests or limit them by length instead of
the total size bytes of the batch.

This PR adds two new CLI options:

```
--rpc-disable-batch-requests - disable batch requests on the server
--rpc-max-batch-request-len <LEN> - limit batches to LEN on the server.
```
2024-02-20 16:16:21 +00:00
Niklas Adolfsson de73dd9ac5 rpc server: add rate limiting middleware (#3301)
Add RPC server rate limiting which can be utilized by the CLI
`--rpc-rate-limit <calls/per minute>`

Resolves first part of
https://github.com/paritytech/polkadot-sdk/issues/3028

//cc @PierreBesson @kogeler  you might be interested in this one

---------

Co-authored-by: James Wilson <james@jsdw.me>
Co-authored-by: Xiliang Chen <xlchen1291@gmail.com>
2024-02-17 10:18:15 +00:00
Oliver Tale-Yazdi e80c24733f Lift dependencies to the workspace (Part 1) (#2070)
Changes (partial https://github.com/paritytech/polkadot-sdk/issues/994):
- Set log to `0.4.20` everywhere
- Lift `log` to the workspace

Starting with a simpler one after seeing
https://github.com/paritytech/polkadot-sdk/pull/2065 from @jsdw.
This sets the `default-features` to `false` in the root and then
overwrites that in each create to its original value. This is necessary
since otherwise the `default` features are additive and its impossible
to disable them in the crate again once they are enabled in the
workspace.

I am using a tool to do this, so its mostly a test to see that it works
as expected.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2024-02-12 11:19:20 +00:00
Niklas Adolfsson e16ef0861f rpc: backpressured RPC server (bump jsonrpsee 0.20) (#1313)
This is a rather big change in jsonrpsee, the major things in this bump
are:
- Server backpressure (the subscription impls are modified to deal with
that)
- Allow custom error types / return types (remove jsonrpsee::core::Error
and jsonrpee::core::CallError)
- Bug fixes (graceful shutdown in particular not used by substrate
anyway)
   - Less dependencies for the clients in particular
   - Return type requires Clone in method call responses
   - Moved to tokio channels
   - Async subscription API (not used in this PR)

Major changes in this PR:
- The subscriptions are now bounded and if subscription can't keep up
with the server it is dropped
- CLI: add parameter to configure the jsonrpc server bounded message
buffer (default is 64)
- Add our own subscription helper to deal with the unbounded streams in
substrate

The most important things in this PR to review is the added helpers
functions in `substrate/client/rpc/src/utils.rs` and the rest is pretty
much chore.

Regarding the "bounded buffer limit" it may cause the server to handle
the JSON-RPC calls
slower than before.

The message size limit is bounded by "--rpc-response-size" thus "by
default 10MB * 64 = 640MB"
but the subscription message size is not covered by this limit and could
be capped as well.

Hopefully the last release prior to 1.0, sorry in advance for a big PR

Previous attempt: https://github.com/paritytech/substrate/pull/13992

Resolves https://github.com/paritytech/polkadot-sdk/issues/748, resolves
https://github.com/paritytech/polkadot-sdk/issues/627
2024-01-23 08:55:13 +00:00
Squirrel be8e626806 Set clippy lints in workspace (requires rust 1.74) (#2390)
We currently use a bit of a hack in `.cargo/config` to make sure that
clippy isn't too annoying by specifying the list of lints.

There is now a stable way to define lints for a workspace. The only down
side is that every crate seems to have to opt into this so there's a
*few* files modified in this PR.

Dependencies:

- [x] PR that upgrades CI to use rust 1.74 is merged.

---------

Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
2023-12-13 15:11:07 +01:00
Liam Aharon 4a293bc5a2 Enforce consistent and correct toml formatting (#2518)
Using taplo, fixes all our broken and inconsistent toml formatting and
adds CI to keep them tidy.

If people want we can customise the format rules as described here
https://taplo.tamasfe.dev/configuration/formatter-options.html

@ggwpez, I suggest zepter is used only for checking features are
propagated, and leave formatting for taplo to avoid duplicate work and
conflicts.

TODO
- [x] Use `exclude = [...]` syntax in taplo file to ignore zombienet
tests instead of deleting the dir

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
2023-12-01 07:38:02 +00:00
Nazar Mokrynskyi 079b14f624 Do not panic if the fdlimit call to increase the file descriptor limit fails (#2155)
# Description

Sometimes changing file descriptor limits is not allowed, but there is
no need to crash the node if/when this happens. Since `fdlimit`'s author
decided to use panics instead of returning `Result`, we need to catch
it.

# Checklist

- [x] My PR includes a detailed description as outlined in the
"Description" section above
- [ ] My PR follows the [labeling requirements](CONTRIBUTING.md#Process)
of this project (at minimum one label for `T`
  required)
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [ ] I have added tests that prove my fix is effective or that my
feature works (if applicable)

---------

Co-authored-by: Koute <koute@users.noreply.github.com>
2023-11-17 17:01:08 +01:00
Bastian Köcher ca5f10567a sc-block-builder: Remove BlockBuilderProvider (#2099)
The `BlockBuilderProvider` was a trait that was defined in
`sc-block-builder`. The trait was implemented for `Client`. This
basically meant that you needed to import `sc-block-builder` any way to
have access to the block builder. So, this trait was not providing any
real value. This pull request is removing the said trait. Instead of the
trait it introduces a builder for creating a `BlockBuilder`. The builder
currently has the quite fabulous name `BlockBuilderBuilder` (I'm open to
any better name 😅). The rest of the pull request is about
replacing the old trait with the new builder.

# Downstream code changes

If you used `new_block` or `new_block_at` before you now need to switch
it over to the new `BlockBuilderBuilder` pattern:

```rust
// `new` requires a type that implements `CallApiAt`. 
let mut block_builder = BlockBuilderBuilder::new(client)
                // Then you need to specify the hash of the parent block the block will be build on top of
		.on_parent_block(at)
                // The block builder also needs the block number of the parent block. 
                // Here it is fetched from the given `client` using the `HeaderBackend`
                // However, there also exists `with_parent_block_number` for directly passing the number
		.fetch_parent_block_number(client)
		.unwrap()
                // Enable proof recording if required. This call is optional.
		.enable_proof_recording()
                // Pass the digests. This call is optional.
                .with_inherent_digests(digests)
		.build()
		.expect("Creates new block builder");
```

---------

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: command-bot <>
2023-11-03 19:06:31 +01:00
David Emett 93d9c8c24e Make CheckNonce refuse transactions signed by accounts with no providers (#1578)
See #1453.

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-10-10 09:14:56 +02:00
Michal Kucharczyk ab3a3bc278 BlockId removal: tx-pool refactor (#1678)
It changes following APIs:
- trait `ChainApi`
-- `validate_transaction`

- trait `TransactionPool` 
--`submit_at`
--`submit_one`
--`submit_and_watch`

and some implementation details, in particular:
- impl `Pool` 
--`submit_at`
--`resubmit_at`
--`submit_one`
--`submit_and_watch`
--`prune_known`
--`prune`
--`prune_tags`
--`resolve_block_number`
--`verify`
--`verify_one`

- revalidation queue

All tests are also adjusted.

---------

Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <git@kchr.de>
2023-09-27 11:58:39 +02:00
Oliver Tale-Yazdi dcda0e50f5 Fix build profiles (#1229)
* Fix build profiles

Closes https://github.com/paritytech/polkadot-sdk/issues/1155

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Manually set version to 1.0.0

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use workspace repo

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* 'Authors and Edition from workspace

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2023-08-29 13:39:41 +02:00
alvicsam f441a5fc93 Diener workspacify
Signed-off-by: alvicsam <alvicsam@gmail.com>
2023-08-25 11:05:17 +02:00
Adrian Catangiu 687b4b4b27 Revert "sc-cli: add no-beefy flag to cli config (#14754)" (#14766)
This reverts commit 64dda775f4.
2023-08-15 15:01:17 +00:00
Adrian Catangiu 64dda775f4 sc-cli: add no-beefy flag to cli config (#14754)
Signed-off-by: Adrian Catangiu <adrian@parity.io>
2023-08-11 14:35:29 +00:00
Bastian Köcher 5eb816d7a6 Removal of execution strategies (#14387)
* Start

* More work!

* Moar

* More changes

* More fixes

* More worrk

* More fixes

* More fixes to make it compile

* Adds `NoOffchainStorage`

* Pass the extensions

* Small basti making small progress

* Fix merge errors and remove `ExecutionContext`

* Move registration of `ReadRuntimeVersionExt` to `ExecutionExtension`

Instead of registering `ReadRuntimeVersionExt` in `sp-state-machine` it is moved to
`ExecutionExtension` which provides the default extensions.

* Fix compilation

* Register the global extensions inside runtime api instance

* Fixes

* Fix `generate_initial_session_keys` by passing the keystore extension

* Fix the grandpa tests

* Fix more tests

* Fix more tests

* Don't set any heap pages if there isn't an override

* Fix small fallout

* FMT

* Fix tests

* More tests

* Offchain worker custom extensions

* More fixes

* Make offchain tx pool creation reusable

Introduces an `OffchainTransactionPoolFactory` for creating offchain transactions pools that can be
registered in the runtime externalities context. This factory will be required for a later pr to
make the creation of offchain transaction pools easier.

* Fixes

* Fixes

* Set offchain transaction pool in BABE before using it in the runtime

* Add the `offchain_tx_pool` to Grandpa as well

* Fix the nodes

* Print some error when using the old warnings

* Fix merge issues

* Fix compilation

* Rename `babe_link`

* Rename to `offchain_tx_pool_factory`

* Cleanup

* FMT

* Fix benchmark name

* Fix `try-runtime`

* Remove `--execution` CLI args

* Make clippy happy

* Forward bls functions

* Fix docs

* Update UI tests

* Update client/api/src/execution_extensions.rs

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/cli/src/params/import_params.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/api/src/execution_extensions.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Pass the offchain storage to the MMR RPC

* Update client/api/src/execution_extensions.rs

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

* Review comments

* Fixes

---------

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
2023-07-11 14:21:38 +00:00
Qinxuan Chen e8e2048b88 chore: update some deps to prune duplicated deps (#14515)
* chore: update some deps to prune duplicated deps

- array-bytes: 4.1.0 => 6.1.0
- cid: 0.8.6 => 0.9.0

* fix
2023-07-06 16:46:44 +02:00
Sebastian Kunert ede49c7ae6 Remove unused dependencies (#14464) 2023-06-27 13:33:42 +00:00
Koute 9d1741eb3c Bump parity-scale-codec to 3.6.1 (#14428) 2023-06-21 11:37:11 +00:00
Davide Galassi b14238cb7d Finalized should be before best (#14308)
* Finalized block should not be after best block

* Remove unwrap

* Apply code review suggestion

Co-authored-by: Koute <koute@users.noreply.github.com>

* Add test

---------

Co-authored-by: Koute <koute@users.noreply.github.com>
2023-06-07 11:17:45 +02:00
Alexandru Vasile 3da9449067 Bump sp-crates from latest crates.io version + release (#14265)
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
2023-05-31 12:11:01 +00:00
Alexandru Vasile 98a0550ea2 Release: Bump SP crates to release on crates.io (#14237)
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
2023-05-29 10:40:59 +00:00
Davide Galassi 0b0bef5cb1 Postpone public key creation in the test GenesisStorageBuilder (#14142)
* Postpone public key creation from account id in the test genesis storage builder

* Store raw sr25519 in substrate test pallet

* Nitpick
2023-05-15 13:56:52 +02:00
Niklas Adolfsson bd96f00146 fix cli: make port Option<u16> again (#14088) 2023-05-07 19:11:03 +00:00
Michal Kucharczyk 6a295e7c28 substrate-test-runtime migrated to "pure" frame runtime (#13737)
* substrate-test-runtime migrated to pure-frame based

* test block builder: helpers added

* simple renaming

* basic_authorship test adjusted

* block_building storage_proof test adjusted

* babe: tests: should_panic expected added

* babe: tests adjusted

ConsensusLog::NextEpochData is now added by pallet_babe as
pallet_babe::SameAuthoritiesForever trigger is used in runtime config.

* beefy: tests adjusted

test-substrate-runtime is now using frame::executive to finalize the
block. during finalization the digests stored during block execution are
checked against header digests:
https://github.com/paritytech/substrate/blob/91bb2d29ca905599098a5b35eaf24867c4fbd60a/frame/executive/src/lib.rs#L585-L591

It makes impossible to directly manipulate header's digets, w/o
depositing logs into system pallet storage `Digest<T: Config>`.

Instead of this dedicated extrinsic allowing to store logs items
(MmrRoot / AuthoritiesChange) is used.

* grandpa: tests adjusted

test-substrate-runtime is now using frame::executive to finalize the
block. during finalization the digest logs stored during block execution are
checked against header digest logs:
https://github.com/paritytech/substrate/blob/91bb2d29ca905599098a5b35eaf24867c4fbd60a/frame/executive/src/lib.rs#L585-L591

It makes impossible to directly manipulate header's digets, w/o
depositing logs into system pallet storage `Digest<T: Config>`.

Instead of this dedicated extrinsic allowing to store logs items
(ScheduledChange / ForcedChange and DigestItem::Other) is used.

* network:bitswap: test adjusted

The size of unchecked extrinsic was increased. The pattern used in test will
be placed at the end of scale-encoded buffer.

* runtime apis versions adjusted

* storage keys used in runtime adjusted

* wasm vs native tests removed

* rpc tests: adjusted

Transfer transaction processing was slightly improved, test was
adjusted.

* tests: sizes adjusted

Runtime extrinsic size was increased. Size of data read during block
execution was also increased due to usage of new pallets in runtime.

Sizes were adjusted in tests.

* cargo.lock update

cargo update -p substrate-test-runtime -p substrate-test-runtime-client

* warnings fixed

* builders cleanup: includes / std

* extrinsic validation cleanup

* txpool: benches performance fixed

* fmt

* spelling

* Apply suggestions from code review

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Apply code review suggestions

* Apply code review suggestions

* get rid of 1063 const

* renaming: UncheckedExtrinsic -> Extrinsic

* test-utils-runtime: further step to pure-frame

* basic-authorship: tests OK

* CheckSubstrateCall added + tests fixes

* test::Transfer call removed

* priority / propagate / no sudo+root-testing

* fixing warnings + format

* cleanup: build2/nonce + format

* final tests fixes

all tests are passing

* logs/comments removal

* should_not_accept_old_signatures test removed

* make txpool benches work again

* Cargo.lock reset

* format

* sudo hack removed

* txpool benches fix+cleanup

* .gitignore reverted

* rebase fixing + unsigned cleanup

* Cargo.toml/Cargo.lock cleanup

* force-debug feature removed

* mmr tests fixed

* make cargo-clippy happy

* network sync test uses unsigned extrinsic

* cleanup

* ".git/.scripts/commands/fmt/fmt.sh"

* push_storage_change signed call remove

* GenesisConfig cleanup

* fix

* fix

* GenesisConfig simplified

* storage_keys_works: reworked

* storage_keys_works: expected keys in vec

* storage keys list moved to substrate-test-runtime

* substrate-test: some sanity tests + GenesisConfigBuilder rework

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Apply suggestions from code review

* Review suggestions

* fix

* fix

* beefy: generate_blocks_and_sync block_num sync with actaul value

* Apply suggestions from code review

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Update test-utils/runtime/src/genesismap.rs

Co-authored-by: Davide Galassi <davxy@datawok.net>

* cargo update -p sc-rpc -p sc-transaction-pool

* Review suggestions

* fix

* doc added

* slot_duration adjusted for Babe::slot_duration

* small doc fixes

* array_bytes::hex used instead of hex

* tiny -> medium name fix

* Apply suggestions from code review

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

* TransferData::try_from_unchecked_extrinsic -> try_from

* Update Cargo.lock

---------

Co-authored-by: parity-processbot <>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
2023-05-04 16:20:22 +00:00
Arkadiy Paronyan bfafbf7bac Statement store (#13701)
* WIP Statement store

* Sync with networking changes in master

* WIP statement pallet

* Statement validation

* pallet tests

* Validation queue

* Store maintenance

* Basic statement refactoring + tests + docs

* Store metrics

* Store tests

* Store maintenance test

* cargo fmt

* Build fix

* OCW Api

* Offchain worker

* Enable host functions

* fmt

* Minor tweaks

* Fixed a warning

* Removed tracing

* Manual expiration

* Reworked constraint management

* Updated pallet constraint calculation

* Added small test

* Added remove function to the APIs

* Copy-paste spec into readme

* Comments

* Made the store optional

* Removed network protocol controller

* fmt

* Clippy fixes

* fmt

* fmt

* More clippy fixes

* More clippy fixes

* More clippy fixes

* Update client/statement-store/README.md

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Removed sstore from node-template

* Sort out data path

* Added offline check

* Removed dispatch_statement

* Renamed into_generic

* Fixed commit placement

* Use HashSet for tracking peers/statements

* fmt

* Use ExtendedHostFunctions

* Fixed benches

* Tweaks

* Apply suggestions from code review

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Fixed priority mixup

* Rename

* newtypes for priorities

* Added MAX_TOPICS

* Fixed key filtering logic

* Remove empty entrie

* Removed prefix from signing

* More documentation

* fmt

* Moved store setup from sc-service to node

* Handle maintenance task in sc-statement-store

* Use statement iterator

* Renamed runtime API mod

* fmt

* Remove dump_encoded

* fmt

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Fixed build after applying review suggestions

* License exceptions

* fmt

* Store options

* Moved pallet consts to config trait

* Removed global priority

* Validate fields when decoding

* Limit validation channel size

* Made a comment into module doc

* Removed submit_encoded

---------

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
2023-05-04 10:24:32 +00:00
Niklas Adolfsson 5a1074712a rpc server: break legacy CLI options and remove "backward compatible HTTP server" (#13384)
* jsonrpsee v0.16

* breaking: remove old CLI configs

* remove patch.crates-io

* fix bad merge

* fix clippy

* fix bad merge

* fix grumbles

* Update client/service/src/lib.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* revert block_in_place

* add issue link in todo

* Update client/cli/src/config.rs

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* grumbles: add ipv6 loopback address

* Revert "grumbles: add ipv6 loopback address"

This reverts commit 3a0b1ece6c4e36055d666896c29d1da55ffa1c4f.

* remove nits

* bump zombienet version

* adress grumbles: provide structopt default_val_t

* remove duplicate from structopt

* bump zombienet v1.3.47

* bump zombienet version

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Javier Viola <javier@parity.io>
2023-05-03 15:18:25 +00:00
Sam Johnson b83bf4784e Globally upgrade to syn 2.x and latest quote and proc_macro2 1x versions (#13846)
* globally upgrade quote to latest 1.0.x (1.0.26)

* globally upgrade syn to final 1.0.x version (1.0.109)

* globally upgrade proc-macro2 to 1.0.56

* upgrade to syn v2.0.13 and fix everything except NestedMeta

* fix parse nested metadata code in decl_runtime_apis.rs

* Port more stuff to syn 2.0

* Make the rest compile

* Ignore error

* update to syn 2.0.14

---------

Co-authored-by: Bastian Köcher <info@kchr.de>
2023-04-12 18:42:22 +00:00
Mira Ressel 17765733f0 simplify some pattern matches to appease 1.68 clippy (#13833) 2023-04-11 10:57:14 +00:00
yjh d5e460b3bf refactor(sc-executor): use wasm executor builder instead of old apis (#13740)
* refactor: use builder api for all executors

* improve a lot

* remove unused args

* cleanup deps

* fix inconsistency about heap alloc

* add `heap_pages` back to try-runtime

* fix

* chore: reduce duplicated code for sc-service-test

* cleanup code

* fmt

* improve test executor

* improve

* use #[deprecated]

* set runtime_cache_size: 4

* fix and improve

* refactor builder

* fix

* fix bench

* fix tests

* fix warnings

* fix warnings

* fix

* fix

* update by suggestions

* update name
2023-04-09 23:48:40 +00:00
Bastian Köcher 846ec8cd01 Remove deprecated batch verification (#13799)
This removes the deprecated batch verification. This was actually never really activated.
Nevertheless, we need to keep the host functions around to support old runtimes which may import
these host functions. However, we do not give access to these functions anymore. This means that any new
runtime can not call them anymore. The host function implementations we keep will not do batch verification and will
instead fall back to the always existing option of directly verifying the passed signature.
`finish_batch_verification` will return the combined result of all the batch verify calls.

This removes the `TaskExecutorExt` which only existed to support the batch verification. So, any
code that used this extension can just remove the registration of them. It also removes
`SignatureBatching` that was used by `frame-executive` to control the batch verification.
However, there wasn't any `Verify` implementation that called the batch verification functions.
2023-04-04 10:02:47 +00:00
Davide Galassi 480396fe06 Keystore overhaul (iter 2) (#13634)
* Remove bloat about remote keystore

* Update docs and remove unused 'KeystoreRef' trait

* Use wherever possible, MemoryKeystore for testing

* Remove unrequired fully qualified method syntax for Keystore
2023-03-20 18:21:26 +00:00
Oliver Tale-Yazdi 58a7f57594 Ignore flaky test (#13631)
* Ignore flaky test

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Re-enable running_the_node_works_and_can_be_interrupted

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Re-enable notifications_back_pressure

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Unused import

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2023-03-17 16:06:40 +00:00
Aaro Altonen 9ced14e2de Move code from sc-network-common back to sc-network (#13592)
* Move service tests to `client/network/tests`

These tests depend on `sc-network` and `sc-network-sync` so they should
live outside the crate.

* Move some configs from `sc-network-common` to `sc-network`

* Move `NetworkService` traits to `sc-network`

* Move request-responses to `sc-network`

* Remove more stuff

* Remove rest of configs from `sc-network-common` to `sc-network`

* Remove more stuff

* Fix warnings

* Update client/network/src/request_responses.rs

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* Fix cargo doc

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
2023-03-14 12:06:40 +00:00
André Silva 13b0f24abd consensus: remove caching functionality from block import pipeline (#13551)
* consensus: remove caching functionality from block import pipeline

* client: update docs on Verifier::verify

* node: fix block production benchmark
2023-03-07 11:19:19 +00:00
Dmitry Markin 3118026576 Make unbounded channels size warning exact (part 2) (#13504) 2023-03-07 10:57:26 +03:00
Aaro Altonen 1a7f5be07f Extract syncing protocol from sc-network (#12828)
* Move import queue out of `sc-network`

Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.

This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.

* Move stuff to SyncingEngine

* Move `ChainSync` instanation to `SyncingEngine`

Some of the tests have to be rewritten

* Move peer hashmap to `SyncingEngine`

* Let `SyncingEngine` to implement `ChainSyncInterface`

* Introduce `SyncStatusProvider`

* Move `sync_peer_(connected|disconnected)` to `SyncingEngine`

* Implement `SyncEventStream`

Remove `SyncConnected`/`SyncDisconnected` events from
`NetworkEvenStream` and provide those events through
`ChainSyncInterface` instead.

Modify BEEFY/GRANDPA/transactions protocol and `NetworkGossip` to take
`SyncEventStream` object which they listen to for incoming sync peer
events.

* Introduce `ChainSyncInterface`

This interface provides a set of miscellaneous functions that other
subsystems can use to query, for example, the syncing status.

* Move event stream polling to `SyncingEngine`

Subscribe to `NetworkStreamEvent` and poll the incoming notifications
and substream events from `SyncingEngine`.

The code needs refactoring.

* Make `SyncingEngine` into an asynchronous runner

This commits removes the last hard dependency of syncing from
`sc-network` meaning the protocol now lives completely outside of
`sc-network`, ignoring the hardcoded peerset entry which will be
addressed in the future.

Code needs a lot of refactoring.

* Fix warnings

* Code refactoring

* Use `SyncingService` for BEEFY

* Use `SyncingService` for GRANDPA

* Remove call delegation from `NetworkService`

* Remove `ChainSyncService`

* Remove `ChainSync` service tests

They were written for the sole purpose of verifying that `NetworWorker`
continues to function while the calls are being dispatched to
`ChainSync`.

* Refactor code

* Refactor code

* Update client/finality-grandpa/src/communication/tests.rs

Co-authored-by: Anton <anton.kalyaev@gmail.com>

* Fix warnings

* Apply review comments

* Fix docs

* Fix test

* cargo-fmt

* Update client/network/sync/src/engine.rs

Co-authored-by: Anton <anton.kalyaev@gmail.com>

* Update client/network/sync/src/engine.rs

Co-authored-by: Anton <anton.kalyaev@gmail.com>

* Add missing docs

* Refactor code

---------

Co-authored-by: Anton <anton.kalyaev@gmail.com>
2023-03-06 16:33:38 +00:00
Bastian Köcher 941288c6d0 wasm-executor: Support growing the memory (#12520)
* As always, start with something :P

* Add support for max_heap_pages

* Add support for wasmtime

* Make it compile

* Fix compilation

* Copy wrongly merged code

* Fix compilation

* Some fixes

* Fix

* Get stuff working

* More work

* More fixes

* ...

* More

* FIXEs

* Switch wasmi to use `RuntimeBlob` like wasmtime

* Removed unused stuff

* Cleanup

* More cleanups

* Introduce `CallContext`

* Fixes

* More fixes

* Add builder for creating the `WasmExecutor`

* Adds some docs

* FMT

* First round of feedback.

* Review feedback round 2

* More fixes

* Fix try-runtime

* Update client/executor/wasmtime/src/instance_wrapper.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/executor/common/src/wasm_runtime.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/executor/common/src/runtime_blob/runtime_blob.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/executor/common/src/wasm_runtime.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/allocator/src/freeing_bump.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/allocator/src/freeing_bump.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Feedback round 3

* FMT

* Review comments

---------

Co-authored-by: Koute <koute@users.noreply.github.com>
2023-02-24 11:43:01 +00:00
Koute f8e3bdad3d Rework storage iterators (#13284)
* Rework storage iterators

* Make sure storage iteration is also accounted for when benchmarking

* Use `trie-db` from crates.io

* Appease clippy

* Bump `trie-bench` to 0.35.0

* Fix tests' compilation

* Update comment to clarify how `IterArgs::start_at` works

* Add extra tests

* Fix iterators on `Client` so that they behave as before

* Add extra `unwrap`s in tests

* More clippy fixes

* Come on clippy, give me a break already

* Rename `allow_missing` to `stop_on_incomplete_database`

* Add `#[inline]` to `with_recorder_and_cache`

* Use `with_recorder_and_cache` in `with_trie_db`; add doc comment

* Simplify code: use `with_trie_db` in `next_storage_key_from_root`

* Remove `expect`s in the benchmarking CLI

* Add extra doc comments

* Move `RawIter` before `TrieBackendEssence` (no code changes; just cut-paste)

* Remove a TODO in tests

* Update comment for `StorageIterator::was_complete`

* Update `trie-db` to 0.25.1
2023-02-22 07:49:25 +00:00
Vivek Pandya bc53b9a03a Remove years from copyright notes. (#13415)
* Change copyright year to 2023 from 2022

* Fix incorrect update of copyright year

* Remove years from copy right header

* Fix remaining files

* Fix typo in a header and remove update-copyright.sh
2023-02-21 18:46:41 +00:00
Michal Kucharczyk 5ef88dd398 BlockId removal: BlockBuilderProvider::new_block_at (#13401)
* `BlockId` removal: `BlockBuilderProvider::new_block_at`

It changes the arguments of `BlockBuilderProvider::new_block_at` from:
`BlockId<Block>` to: `Block::Hash`

* fmt

* fix

* more fixes
2023-02-21 18:36:00 +00:00
Michal Kucharczyk 7a10154188 BlockId removal: runtime-api refactor (#13255)
* BlockId removal: refactor of runtime API

It changes the arguments of:
- `ApiExt` methods:  `has_api`, `has_api_with`, `api_version`
- `CallApiAt` method: `runtime_version_at`
from: `BlockId<Block>` to: `Block::Hash`

It also changes the first argument of all generated runtime API calls from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* BlockId removal: refactor of runtime API - tests

- tests adjusted to new runtime API,
- some tests migrated from block number to block hash

* benchmarking-cli: BlockId(0) migrated to info().genesis_hash

`runtime_api.call()` now requires the block hash instead of BlockId::Number.
To access the genesis hash widely used in benchmarking engine the Client
was constrained to satisfy `sp_blockchain::HeaderBackend<Block>` trait
which provides `info().genesis_hash`.

* trivial: api.call(BlockId) -> api.call(Hash)

- Migrated all `runtime_api.calls` to use Hash
- Noteworthy (?):
-- `validate_transaction_blocking` in transaction pool,

* CallApiAtParams::at changed to Block::Hash

* missed doc updated

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* ".git/.scripts/commands/fmt/fmt.sh"

* BlockId removal: Benchmark::consumed_weight

Little refactor around `Benchmark::consumed_weight`: `BlockId` removed.

* at_hash renamed

* wrong merge fixed

* beefy worker: merged with master

* beefy: tests: missing block problem fixed

* Apply review suggestion

* fix

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>
2023-02-20 22:47:21 +00:00
Davide Galassi d48fc58729 Fix longest chain finalization target lookup (#13289)
* Finalization target should be chosed as some ancestor of SelectChain::best_chain

* More test assertions

* Improve docs

* Removed stale docs

* Rename 'target' to 'base' in lookup method

* Fix typo

* Apply suggestions from code review

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Rename 'target_hash' to 'base_hash' in 'SelectChain::finality_target()'

* Apply suggestions from code review

Co-authored-by: Anton <anton.kalyaev@gmail.com>

* Docs improvement

* Doc fix

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Apply more code suggestions

---------

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Anton <anton.kalyaev@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
2023-02-11 18:35:04 +01:00
Bastian Köcher d20a9bdc21 Rename *-private-ipv4 to *-private-ip CLI args (#13208)
* Rename `*-private-ipv4` to `*-private-ip` CLI args

Renames the `*-private-ipv4` to `*-private-ip` in the CLI interface. The old names are staying as
alias, thus it will not break for anyone. Besides that it also fixes the naming in the rest of the code.

* FMT
2023-01-23 11:35:04 +00:00
Gavin Wood 2d7fa18e73 Make DispatchError impl MEL (#13169)
* Make DispatchError impl MEL

* Upgrade SCALE codec to support `codec(skip)` for MEL

Co-authored-by: Bastian Köcher <info@kchr.de>
2023-01-19 16:47:47 +00:00
Michal Kucharczyk 9c69fc1b32 BlockId removal: refactor: BlockBackend::block|block_status (#13014)
* BlockId removal: refactor: BlockBackend::block|block_status

It changes the arguments of:
-  `BlockBackend::block`
-  `BlockBackend::block_status`

method from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* non-obvious reworks

* doc fix

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: parity-processbot <>
2023-01-02 09:42:05 +00:00
Michal Kucharczyk 548955a73f BlockId removal: refactor: HeaderBackend::header (#12874)
* BlockId removal: refactor: HeaderBackend::header

It changes the arguments of:
- `HeaderBackend::header`,
- `Client::header`,
- `PeersClient::header`
- `ChainApi::block_header`

methods from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* non-trivial usages of haeder(block_id) refactored

This may required introduction of dedicated function:
header_for_block_num

* fmt

* fix

* doc fixed

* ".git/.scripts/fmt.sh"

* BlockId removal: refactor: HeaderBackend::expect_header

It changes the arguments of `HeaderBackend::expect_header` method from: `BlockId<Block>` to: `Block::Hash`

* ".git/.scripts/fmt.sh"

* readme updated

* ".git/.scripts/fmt.sh"

* fix

Co-authored-by: parity-processbot <>
2022-12-20 09:43:31 +00:00
Liu-Cheng Xu b92aa3dbdc Support custom genesis block (#12291)
* Set genesis block data using the built genesis block

* Make resolve_state_version_from_wasm a separate function and some small refactorings

Useful for the commit following.

* Introduce trait BuildGenesisBlock

Substrate users can use this trait to implement their custom genesis block when constructing the
client.

* Make call_executor test compile

* cargo +nightly fmt --all

* Fix test

* Remove unnecessary clone

* FMT

* Apply review suggestions

* Revert changes to new_full_client() and new_full_parts() signature

* Remove needless `Block` type in `resolve_state_version_from_wasm`
2022-12-19 15:26:31 +01:00