Commit Graph

31 Commits

Author SHA1 Message Date
Aaro Altonen 80616f6d03 Integrate litep2p into Polkadot SDK (#2944)
[litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P
networking library. It supports all of the features of `rust-libp2p`
that are currently being utilized by Polkadot SDK.

Compared to `rust-libp2p`, `litep2p` has a quite different architecture
which is why the new `litep2p` network backend is only able to use a
little of the existing code in `sc-network`. The design has been mainly
influenced by how we'd wish to structure our networking-related code in
Polkadot SDK: independent higher-levels protocols directly communicating
with the network over links that support bidirectional backpressure. A
good example would be `NotificationHandle`/`RequestResponseHandle`
abstractions which allow, e.g., `SyncingEngine` to directly communicate
with peers to announce/request blocks.

I've tried running `polkadot --network-backend litep2p` with a few
different peer configurations and there is a noticeable reduction in
networking CPU usage. For high load (`--out-peers 200`), networking CPU
usage goes down from ~110% to ~30% (80 pp) and for normal load
(`--out-peers 40`), the usage goes down from ~55% to ~18% (37 pp).

These should not be taken as final numbers because:

a) there are still some low-hanging optimization fruits, such as
enabling [receive window
auto-tuning](https://github.com/libp2p/rust-yamux/pull/176), integrating
`Peerset` more closely with `litep2p` or improving memory usage of the
WebSocket transport
b) fixing bugs/instabilities that incorrectly cause `litep2p` to do less
work will increase the networking CPU usage
c) verification in a more diverse set of tests/conditions is needed

Nevertheless, these numbers should give an early estimate for CPU usage
of the new networking backend.

This PR consists of three separate changes:
* introduce a generic `PeerId` (wrapper around `Multihash`) so that we
don't have use `NetworkService::PeerId` in every part of the code that
uses a `PeerId`
* introduce `NetworkBackend` trait, implement it for the libp2p network
stack and make Polkadot SDK generic over `NetworkBackend`
  * implement `NetworkBackend` for litep2p

The new library should be considered experimental which is why
`rust-libp2p` will remain as the default option for the time being. This
PR currently depends on the master branch of `litep2p` but I'll cut a
new release for the library once all review comments have been
addresses.

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Alexandru Vasile <alexandru.vasile@parity.io>
2024-04-08 16:44:13 +00:00
Dcompoze 002d9260f9 Fix spelling mistakes across the whole repository (#3808)
**Update:** Pushed additional changes based on the review comments.

**This pull request fixes various spelling mistakes in this
repository.**

Most of the changes are contained in the first **3** commits:

- `Fix spelling mistakes in comments and docs`

- `Fix spelling mistakes in test names`

- `Fix spelling mistakes in error messages, panic messages, logs and
tracing`

Other source code spelling mistakes are separated into individual
commits for easier reviewing:

- `Fix the spelling of 'authority'`

- `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'`

- `Fix the spelling of 'prev_enqueud_messages'`

- `Fix the spelling of 'endpoint'`

- `Fix the spelling of 'children'`

- `Fix the spelling of 'PenpalSiblingSovereignAccount'`

- `Fix the spelling of 'PenpalSudoAccount'`

- `Fix the spelling of 'insufficient'`

- `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'`

- `Fix the spelling of 'subtracted'`

- `Fix the spelling of 'CandidatePendingAvailability'`

- `Fix the spelling of 'exclusive'`

- `Fix the spelling of 'until'`

- `Fix the spelling of 'discriminator'`

- `Fix the spelling of 'nonexistent'`

- `Fix the spelling of 'subsystem'`

- `Fix the spelling of 'indices'`

- `Fix the spelling of 'committed'`

- `Fix the spelling of 'topology'`

- `Fix the spelling of 'response'`

- `Fix the spelling of 'beneficiary'`

- `Fix the spelling of 'formatted'`

- `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'`

- `Fix the spelling of 'succeeded'`

- `Fix the spelling of 'reopened'`

- `Fix the spelling of 'proposer'`

- `Fix the spelling of 'InstantiationNonce'`

- `Fix the spelling of 'depositor'`

- `Fix the spelling of 'expiration'`

- `Fix the spelling of 'phantom'`

- `Fix the spelling of 'AggregatedKeyValue'`

- `Fix the spelling of 'randomness'`

- `Fix the spelling of 'defendant'`

- `Fix the spelling of 'AquaticMammal'`

- `Fix the spelling of 'transactions'`

- `Fix the spelling of 'PassingTracingSubscriber'`

- `Fix the spelling of 'TxSignaturePayload'`

- `Fix the spelling of 'versioning'`

- `Fix the spelling of 'descendant'`

- `Fix the spelling of 'overridden'`

- `Fix the spelling of 'network'`

Let me know if this structure is adequate.

**Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`,
`Merkelization`, `Merkleization`, is somewhat inconsistent but I left it
as it is.

~~**Note:** In some places the term `Receival` is used to refer to
message reception, IMO `Reception` is the correct word here, but I left
it as it is.~~

~~**Note:** In some places the term `Overlayed` is used instead of the
more acceptable version `Overlaid` but I also left it as it is.~~

~~**Note:** In some places the term `Applyable` is used instead of the
correct version `Applicable` but I also left it as it is.~~

**Note:** Some usage of British vs American english e.g. `judgement` vs
`judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc.
are both present in different places, but I suppose that's
understandable given the number of contributors.

~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it
triggers errors in CI when I make changes to it, so I left it as it
is.~~
2024-03-26 13:57:57 +00:00
Davide Galassi 4c10fd2a41 Move cryptographic hashing procedures to crypto folder. (#2306)
Step towards https://github.com/paritytech/polkadot-sdk/issues/1975

As reported
https://github.com/paritytech/polkadot-sdk/issues/1975#issuecomment-1774534225
I'd like to encapsulate crypto related stuff in a dedicated folder.

Currently all cryptographic primitive wrappers are all sparsed in
`substrate/core` which contains "misc core" stuff.

To simplify the process, as the first step with this PR I propose to
move the cryptographic hashing there.

The `substrate/crypto` folder was already created to contains `ec-utils`
crate.

Notes:
- rename `sp-core-hashing` to `sp-crypto-hashing`
- rename `sp-core-hashing-proc-macro` to `sp-crypto-hashing-proc-macro`
- As the crates name is changed I took the freedom to restart fresh from
version 0.1.0 for both crates

---------

Co-authored-by: Robert Hambrock <roberthambrock@gmail.com>
2024-01-22 23:36:14 +00:00
Dmitry Markin 5208bed7d2 Extract warp sync strategy from ChainSync (#2467)
Extract `WarpSync` (and `StateSync` as part of warp sync) from
`ChainSync` as independent syncing strategy called by `SyncingEngine`.
Introduce `SyncingStrategy` enum as a proxy between `SyncingEngine` and
specific syncing strategies.

## Limitations
Gap sync is kept in `ChainSync` for now because it shares the same set
of peers as block syncing implementation in `ChainSync`. Extraction of a
common context responsible for peer management in syncing strategies
able to run in parallel is planned for a follow-up PR.

## Further improvements
A possibility of conversion of `SyncingStartegy` into a trait should be
evaluated. The main stopper for this is that different strategies need
to communicate different actions to `SyncingEngine` and respond to
different events / provide different APIs (e.g., requesting
justifications is only possible via `ChainSync` and not through
`WarpSync`; `SendWarpProofRequest` action is only relevant to
`WarpSync`, etc.)

---------

Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
2024-01-12 15:17:15 +00:00
Nazar Mokrynskyi 9f3c67b4ff Support querying peer reputation (#2392)
# Description

Trivial change that resolves
https://github.com/paritytech/polkadot-sdk/issues/2185.

Since there was a mix of `who` and `peer_id` argument names nearby I
changed them all to `peer_id`.

# Checklist

- [x] My PR includes a detailed description as outlined in the
"Description" section above
- [x] My PR follows the [labeling requirements](CONTRIBUTING.md#Process)
of this project (at minimum one label for `T`
  required)
- [x] I have made corresponding changes to the documentation (if
applicable)
- [ ] I have added tests that prove my fix is effective or that my
feature works (if applicable)

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-12-07 13:07:12 +02:00
Aaro Altonen e71c484d5b Rework the event system of sc-network (#1370)
This commit introduces a new concept called `NotificationService` which
allows Polkadot protocols to communicate with the underlying
notification protocol implementation directly, without routing events
through `NetworkWorker`. This implies that each protocol has its own
service which it uses to communicate with remote peers and that each
`NotificationService` is unique with respect to the underlying
notification protocol, meaning `NotificationService` for the transaction
protocol can only be used to send and receive transaction-related
notifications.

The `NotificationService` concept introduces two additional benefits:
  * allow protocols to start using custom handshakes
  * allow protocols to accept/reject inbound peers

Previously the validation of inbound connections was solely the
responsibility of `ProtocolController`. This caused issues with light
peers and `SyncingEngine` as `ProtocolController` would accept more
peers than `SyncingEngine` could accept which caused peers to have
differing views of their own states. `SyncingEngine` would reject excess
peers but these rejections were not properly communicated to those peers
causing them to assume that they were accepted.

With `NotificationService`, the local handshake is not sent to remote
peer if peer is rejected which allows it to detect that it was rejected.

This commit also deprecates the use of `NetworkEventStream` for all
notification-related events and going forward only DHT events are
provided through `NetworkEventStream`. If protocols wish to follow each
other's events, they must introduce additional abtractions, as is done
for GRANDPA and transactions protocols by following the syncing protocol
through `SyncEventStream`.

Fixes https://github.com/paritytech/polkadot-sdk/issues/512
Fixes https://github.com/paritytech/polkadot-sdk/issues/514
Fixes https://github.com/paritytech/polkadot-sdk/issues/515
Fixes https://github.com/paritytech/polkadot-sdk/issues/554
Fixes https://github.com/paritytech/polkadot-sdk/issues/556

---
These changes are transferred from
https://github.com/paritytech/substrate/pull/14197 but there are no
functional changes compared to that PR

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
2023-11-28 20:18:52 +02:00
Bastian Köcher ebcf0a0f1c pallet-grandpa: Remove GRANDPA_AUTHORITIES_KEY (#2181)
Remove the `GRANDPA_AUTHORITIES_KEY` key and its usage. Apparently this
was used in the early days to communicate the grandpa authorities to the
node. However, we have now a runtime api that does this for us. So, this
pull request is moving from the custom managed storage item to a FRAME
managed storage item.

This pr also includes a migration for doing the switch on a running
chain.

---------

Co-authored-by: Davide Galassi <davxy@datawok.net>
2023-11-13 15:32:02 +02:00
Bastian Köcher ca5f10567a sc-block-builder: Remove BlockBuilderProvider (#2099)
The `BlockBuilderProvider` was a trait that was defined in
`sc-block-builder`. The trait was implemented for `Client`. This
basically meant that you needed to import `sc-block-builder` any way to
have access to the block builder. So, this trait was not providing any
real value. This pull request is removing the said trait. Instead of the
trait it introduces a builder for creating a `BlockBuilder`. The builder
currently has the quite fabulous name `BlockBuilderBuilder` (I'm open to
any better name 😅). The rest of the pull request is about
replacing the old trait with the new builder.

# Downstream code changes

If you used `new_block` or `new_block_at` before you now need to switch
it over to the new `BlockBuilderBuilder` pattern:

```rust
// `new` requires a type that implements `CallApiAt`. 
let mut block_builder = BlockBuilderBuilder::new(client)
                // Then you need to specify the hash of the parent block the block will be build on top of
		.on_parent_block(at)
                // The block builder also needs the block number of the parent block. 
                // Here it is fetched from the given `client` using the `HeaderBackend`
                // However, there also exists `with_parent_block_number` for directly passing the number
		.fetch_parent_block_number(client)
		.unwrap()
                // Enable proof recording if required. This call is optional.
		.enable_proof_recording()
                // Pass the digests. This call is optional.
                .with_inherent_digests(digests)
		.build()
		.expect("Creates new block builder");
```

---------

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: command-bot <>
2023-11-03 19:06:31 +01:00
Dmitry Markin 1cd6acdff3 Move syncing code from sc-network-common to sc-network-sync (#1912)
This PR moves syncing-related code from `sc-network-common` to
`sc-network-sync`.

Unfortunately, some parts are tightly integrated with networking, so
they were left in `sc-network-common` for now:

1. `SyncMode` in `common/src/sync.rs` (used in `NetworkConfiguration`).
2. `BlockAnnouncesHandshake`, `BlockRequest`, `BlockResponse`, etc. in
`common/src/sync/message.rs` (used in `src/protocol.rs` and
`src/protocol/message.rs`).

More substantial refactoring is needed to decouple syncing and
networking completely, including getting rid of the hardcoded sync
protocol.

## Release notes

Move syncing-related code from `sc-network-common` to `sc-network-sync`.
Delete `ChainSync` trait as it's never used (the only implementation is
accessed directly from `SyncingEngine` and exposes a lot of public
methods that are not part of the trait). Some new trait(s) for syncing
will likely be introduced as part of Sync 2.0 refactoring to represent
syncing strategies.
2023-11-01 15:10:33 +02:00
Nazar Mokrynskyi ee6eeb7466 Allow justifications on non-finalized blocks (#1211)
One assertion was unnecessary because of the check right above it,
second assertion resolves
https://github.com/paritytech/polkadot-sdk/issues/1159

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-09-12 16:47:43 +02:00
Przemek Rzad bfb241d7f3 Add missing licenses and tune the scanning workflow (#1288)
* Add missing Cumulus licenses

* Typo

* Add missing Substrate licenses

* Single job checking the sub-repos in steps

* Remove dates

* Remove dates

* Add missing (C)

* Update FRAME UI tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update more UI tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2023-08-30 15:45:49 +03:00
Gavin Wood 46bd466e48 Frame: Agile Coretime Broker pallet (RFC-1) (#14568)
* Add Broker pallet

* Flesh out CorePart

* Repotting and fleshing out

* more drafting

* process timeslice

* Test Fungibles completed

* Auctions

* Price morphing

* First tests

* Tidying up config/status

* Docs

* Timeslice todying

* More Timeslice tidying

* Tests]

* Repotting.

* Tests

* Tests

* System InstaPool cores and payout

* Better Relay Test framework

* Tests and instapool fixes

* Support NFT interface

* Proper renewals

* Better events, results

* Test transfer

* Renewal test

* Repot some impls and make dispatchables.

* Better weight

* Test migration

* Document events

* Introduce durations

* Core count

* Allow reassignment

* Better naming

* Error docs

* Docs

* Formatting

* Advance notice period is in RC blocks, not timeslices

* Docs

* Formatting

* Docs

* Missing file

* Added some events

* Events for all dispatchables

* Remove benchmark

* Fix

* Adds benchmark for configure and some basic setup

* Adds benchmark for reserve and unreserve

* Adds a couple of more benchmarks

* Docs

* Event

* Fix

* Adds benchmark for purchase

* Dedup

* Add some weight breakdowns

* Repotting

* Adds more benchmarks

* Renaming and one more event

* Sale event

* Better price API and docs

* Avoid possibility of clobbering renewal record

* Avoid possibility of clobbering renewal record

* Fixes a few benchmarks

* Another test

* More tests

* Drop history test

* Rename and CORE_MASK_BITS constant

* Update frame/broker/src/dispatchable_impls.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update frame/broker/src/dispatchable_impls.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update frame/broker/src/dispatchable_impls.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update frame/broker/src/utility_impls.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update frame/broker/src/dispatchable_impls.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update frame/broker/src/mock.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Addresses few review comments

* Addresses few review comments

* Addresses few review comments

* Merge

* Merge

* ".git/.scripts/commands/fmt/fmt.sh"

* Integrates broker in kitchensink

* Minor update

* Fixes typo

* Moves balance back to u64

* Fixes kitchensink build

* Fixes worst case for assign

* Adds benchmark for process_core_count

* Adds a couple of more benchmarks

* Adds an assert for partition

* Uses max_timeslices as input in claim_revenue benchmark

* Adds benchmark for drop_renewal

* Adds benchmark for process_core_schedule

* Adds benchmark for process_pool

* Adds assertion for transfer

* Fixes benchmark for broker in kitchensink

* Adds todo for process_revenue benchmark

* Minor update

* Fix for pool revenue history

* remove TODOs

* Fix tests

* Document CoretimeInterface

* rename part to mask

* Fixes

* Grumble

* ".git/.scripts/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_broker

* Adds benchmark for drop_history and fixes worst case for claim_revenue

* Adds drop_history in WeightInfo

* ".git/.scripts/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_broker

* Minor fix for Quick Benchmark CI

* Fixes

* Docs

* Headers

* Expose a couple of APIs for benchmarking (#14688)

* Expose a couple of APIs for benchmarking

* Adds doc

* Minor fix in CoretimeInterface impl for kitchensik

* Minor

* Cap renewal price

* Adds a few tests

* Adds more tests

* Minor updates

* Adds a test for an edge case

* Fixes feature propagation

* Fixes feature propagation

* Adds doc fix

* Syntax nits

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Reuse Bit assign functions

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Bitwise tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* adapt_price: Edge case for sold == target

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add sanity checking to ConfigRecord

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add deny(missing_docs) where possible

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* partition: forbid pivot_offset == 0

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Sort features

zepter format features

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Import Zero from new location

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Clippy: remove redundant clone

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* try to fix build

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix CI

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Nikhil Gupta <17176722+gupnik@users.noreply.github.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
2023-08-24 20:37:20 +00:00
Bastian Köcher ecf8035da6 Set StateBackend::Transaction to PrefixedMemoryDB (#14612)
* Yep

* Try to get it working everywhere

* Make `from_raw_storage` start with an empty db

* More fixes!

* Make everything compile

* Fix `child_storage_root`

* Fix after merge

* Cleanups

* Update primitives/state-machine/src/overlayed_changes/mod.rs

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Review comments

* Fix issues

* Silence warning

* FMT

* Clippy

---------

Co-authored-by: Davide Galassi <davxy@datawok.net>
2023-08-17 10:49:38 +00:00
Dmitry Markin 8dc3bd729b Get rid of Peerset compatibility layer (#14337)
* Move bootnodes from individual `SetConfig`s to `PeersetConfig`

* Move `SetId` & `SetConfig` from `peerset` to `protocol_controller`

* Remove unused `DropReason`

* Move `Message` & `IncomingIndex` from `peerset` to `protocol_controller`

* Restore running fuzz test

* Get rid of `Peerset` in `fuzz` test

* Spawn runners instead of manual polling in `fuzz` test

* Migrate `Protocol` from `Peerset` to `PeerStore` & `ProtocolController`

* Migrate `NetworkService` from `Peerset` to `PeerStore` & `ProtocolController`

* Migrate `Notifications` from `Peerset` to `ProtocolController`s

* Migrate `Notifications` tests from `Peerset` to `ProtocolController`

* Fix compilation of `NetworkService` & `Protocol`

* Fix borrowing issues in `Notifications`

* Migrate `RequestResponse`from `Peerset` to `PeerStore`

* rustfmt

* Migrate request-response tests from `Peerset` to `PeerStore`

* Migrate `reconnect_after_disconnect` test to `PeerStore` & `ProtocolController`

* Fix `Notifications` tests

* Remove `Peerset` completely

* Fix bug with counting sync peers in `Protocol`

* Eliminate indirect calls to `PeerStore` via `Protocol`

* Eliminate indirect calls to `ProtocolController` via `Protocol`

* Handle `Err` outcome from `remove_peers_from_reserved_set`

* Add note about disconnecting sync peers in `Protocol`

* minor: remove unneeded `clone()`

* minor: extra comma removed

* minor: use `Stream` API of `from_protocol_controllers` channel

* minor: remove TODO

* minor: replace `.map().flatten()` with `.flat_map()`

* minor: update `ProtocolController` docs

* rustfmt

* Apply suggestions from code review

Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>

* Extract `MockPeerStore` to `mock.rs`

* Move `PeerStore` initialization to `build_network`

* minor: remove unused import

* minor: clarify error message

* Convert `syncs_header_only_forks` test into single-threaded

---------

Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
2023-08-02 13:01:35 +00:00
Bastian Köcher b16976053e Do some cleanups (#14608)
* Do some cleanups

Found them while looking over the code.

* More

* Fix
2023-07-21 23:54:01 +02:00
André Silva c761d4c39e grandpa: avoid importing unnecessary justifications (#14423)
* grandpa: avoid importing unnecessary justifications

* grandpa: make justification_import_period configurable

* grandpa: keep the first justification

* grandpa: add test for justification import period

* grandpa: fix test
2023-07-17 17:38:34 +00:00
Bastian Köcher 5eb816d7a6 Removal of execution strategies (#14387)
* Start

* More work!

* Moar

* More changes

* More fixes

* More worrk

* More fixes

* More fixes to make it compile

* Adds `NoOffchainStorage`

* Pass the extensions

* Small basti making small progress

* Fix merge errors and remove `ExecutionContext`

* Move registration of `ReadRuntimeVersionExt` to `ExecutionExtension`

Instead of registering `ReadRuntimeVersionExt` in `sp-state-machine` it is moved to
`ExecutionExtension` which provides the default extensions.

* Fix compilation

* Register the global extensions inside runtime api instance

* Fixes

* Fix `generate_initial_session_keys` by passing the keystore extension

* Fix the grandpa tests

* Fix more tests

* Fix more tests

* Don't set any heap pages if there isn't an override

* Fix small fallout

* FMT

* Fix tests

* More tests

* Offchain worker custom extensions

* More fixes

* Make offchain tx pool creation reusable

Introduces an `OffchainTransactionPoolFactory` for creating offchain transactions pools that can be
registered in the runtime externalities context. This factory will be required for a later pr to
make the creation of offchain transaction pools easier.

* Fixes

* Fixes

* Set offchain transaction pool in BABE before using it in the runtime

* Add the `offchain_tx_pool` to Grandpa as well

* Fix the nodes

* Print some error when using the old warnings

* Fix merge issues

* Fix compilation

* Rename `babe_link`

* Rename to `offchain_tx_pool_factory`

* Cleanup

* FMT

* Fix benchmark name

* Fix `try-runtime`

* Remove `--execution` CLI args

* Make clippy happy

* Forward bls functions

* Fix docs

* Update UI tests

* Update client/api/src/execution_extensions.rs

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/cli/src/params/import_params.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/api/src/execution_extensions.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Pass the offchain storage to the MMR RPC

* Update client/api/src/execution_extensions.rs

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

* Review comments

* Fixes

---------

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
2023-07-11 14:21:38 +00:00
Qinxuan Chen e8e2048b88 chore: update some deps to prune duplicated deps (#14515)
* chore: update some deps to prune duplicated deps

- array-bytes: 4.1.0 => 6.1.0
- cid: 0.8.6 => 0.9.0

* fix
2023-07-06 16:46:44 +02:00
yjh ddb46a8aa0 refactor: define trait HashOutput for some Hash associate type (#14220)
* define trait `HashOutput`

* improve

* improve

* Update primitives/runtime/src/traits.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* remove `Block::Hash: Ord`

* fmt

* add `MaybeFromStr`

* cleanup

* fix

* remove useless `HashOutput`

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-06-08 02:02:13 +00:00
Dmitry Markin 01107e9ca5 Split Peerset into PeerStore & ProtocolControllers (#13611) 2023-05-23 14:49:02 +03:00
Aaro Altonen f36749b99e Prepare sc-network for ProtocolController/NotificationService (#14080)
* Prepare `sc-network` for `ProtocolController`/`NotificationService`

The upcoming notification protocol refactoring requires that protocols
are able to communicate with `sc-network` over unique and direct links.
This means that `sc-network` side of the link has to be created before
`sc-network` is initialized and that it is allowed to consume the object
as the receiver half of the link may not implement `Clone`.

Remove request-response and notification protocols from `NetworkConfiguration`
and create a new object that contains the configurations of these protocols
and which is consumable by `sc-network`. This is needed needed because, e.g.,
the receiver half of `NotificationService` is not clonable so `sc-network`
must consume it when it's initializing the protocols in `Notifications`.

Similar principe applies to `PeerStore`/`ProtocolController`: as per current
design, protocols are created before the network so `Protocol` cannot be
the one creating the `PeerStore` object. `FullNetworkConfiguration` will be
used to store the objects that `sc-network` will use to communicate with
protocols and it will also allow protocols to allocate handles so they
can directly communicate with `sc-network`.

* Fixes

* Update client/service/src/builder.rs

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* Updates

* Doc updates + cargo-fmt

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
2023-05-11 10:27:21 +00:00
Michal Kucharczyk 6a295e7c28 substrate-test-runtime migrated to "pure" frame runtime (#13737)
* substrate-test-runtime migrated to pure-frame based

* test block builder: helpers added

* simple renaming

* basic_authorship test adjusted

* block_building storage_proof test adjusted

* babe: tests: should_panic expected added

* babe: tests adjusted

ConsensusLog::NextEpochData is now added by pallet_babe as
pallet_babe::SameAuthoritiesForever trigger is used in runtime config.

* beefy: tests adjusted

test-substrate-runtime is now using frame::executive to finalize the
block. during finalization the digests stored during block execution are
checked against header digests:
https://github.com/paritytech/substrate/blob/91bb2d29ca905599098a5b35eaf24867c4fbd60a/frame/executive/src/lib.rs#L585-L591

It makes impossible to directly manipulate header's digets, w/o
depositing logs into system pallet storage `Digest<T: Config>`.

Instead of this dedicated extrinsic allowing to store logs items
(MmrRoot / AuthoritiesChange) is used.

* grandpa: tests adjusted

test-substrate-runtime is now using frame::executive to finalize the
block. during finalization the digest logs stored during block execution are
checked against header digest logs:
https://github.com/paritytech/substrate/blob/91bb2d29ca905599098a5b35eaf24867c4fbd60a/frame/executive/src/lib.rs#L585-L591

It makes impossible to directly manipulate header's digets, w/o
depositing logs into system pallet storage `Digest<T: Config>`.

Instead of this dedicated extrinsic allowing to store logs items
(ScheduledChange / ForcedChange and DigestItem::Other) is used.

* network:bitswap: test adjusted

The size of unchecked extrinsic was increased. The pattern used in test will
be placed at the end of scale-encoded buffer.

* runtime apis versions adjusted

* storage keys used in runtime adjusted

* wasm vs native tests removed

* rpc tests: adjusted

Transfer transaction processing was slightly improved, test was
adjusted.

* tests: sizes adjusted

Runtime extrinsic size was increased. Size of data read during block
execution was also increased due to usage of new pallets in runtime.

Sizes were adjusted in tests.

* cargo.lock update

cargo update -p substrate-test-runtime -p substrate-test-runtime-client

* warnings fixed

* builders cleanup: includes / std

* extrinsic validation cleanup

* txpool: benches performance fixed

* fmt

* spelling

* Apply suggestions from code review

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Apply code review suggestions

* Apply code review suggestions

* get rid of 1063 const

* renaming: UncheckedExtrinsic -> Extrinsic

* test-utils-runtime: further step to pure-frame

* basic-authorship: tests OK

* CheckSubstrateCall added + tests fixes

* test::Transfer call removed

* priority / propagate / no sudo+root-testing

* fixing warnings + format

* cleanup: build2/nonce + format

* final tests fixes

all tests are passing

* logs/comments removal

* should_not_accept_old_signatures test removed

* make txpool benches work again

* Cargo.lock reset

* format

* sudo hack removed

* txpool benches fix+cleanup

* .gitignore reverted

* rebase fixing + unsigned cleanup

* Cargo.toml/Cargo.lock cleanup

* force-debug feature removed

* mmr tests fixed

* make cargo-clippy happy

* network sync test uses unsigned extrinsic

* cleanup

* ".git/.scripts/commands/fmt/fmt.sh"

* push_storage_change signed call remove

* GenesisConfig cleanup

* fix

* fix

* GenesisConfig simplified

* storage_keys_works: reworked

* storage_keys_works: expected keys in vec

* storage keys list moved to substrate-test-runtime

* substrate-test: some sanity tests + GenesisConfigBuilder rework

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Apply suggestions from code review

* Review suggestions

* fix

* fix

* beefy: generate_blocks_and_sync block_num sync with actaul value

* Apply suggestions from code review

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Update test-utils/runtime/src/genesismap.rs

Co-authored-by: Davide Galassi <davxy@datawok.net>

* cargo update -p sc-rpc -p sc-transaction-pool

* Review suggestions

* fix

* doc added

* slot_duration adjusted for Babe::slot_duration

* small doc fixes

* array_bytes::hex used instead of hex

* tiny -> medium name fix

* Apply suggestions from code review

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

* TransferData::try_from_unchecked_extrinsic -> try_from

* Update Cargo.lock

---------

Co-authored-by: parity-processbot <>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
2023-05-04 16:20:22 +00:00
Mira Ressel 17765733f0 simplify some pattern matches to appease 1.68 clippy (#13833) 2023-04-11 10:57:14 +00:00
Davide Galassi 40e1704e1c Keystore overhaul (final) (#13683)
* Introduce keystore specialized sign methods

* Get rid of 'AppKey::UntypedGeneric' associated type.

Untyped generics are accessible using associated types 'Generic' associated type.
I.e. <T as AppKey>::Public::Generic

* Get rid of 'CryptoTypePublicPair'

* Trivial fix

* Small refactory of local keystore implementations

* Remove 'crypto_id' method from 'Public'

* Trivial rename of 'AppKey' to 'AppCrypto'

* Remove unused import

* Improve docs

* Better signature related errors for authority-discovery

* Apply review suggestion

* Apply review suggestions

Co-authored-by: Koute <koute@users.noreply.github.com>

* Authority discoverty signing error revisited

* Signing error revisited for babe and aura as well

* Further cleanup

---------

Co-authored-by: Koute <koute@users.noreply.github.com>
2023-03-24 13:46:02 +00:00
Davide Galassi 480396fe06 Keystore overhaul (iter 2) (#13634)
* Remove bloat about remote keystore

* Update docs and remove unused 'KeystoreRef' trait

* Use wherever possible, MemoryKeystore for testing

* Remove unrequired fully qualified method syntax for Keystore
2023-03-20 18:21:26 +00:00
Davide Galassi f110941b7f Keystore overhaul (#13615)
* Remove 'supported_keys' 'sign_with_any' and 'sign_with_all' from keystore trait

* Remove the aync keystore

* Renaming:
- SyncCryptoStore -> Keystore
- SyncCryptoStorePtr -> KeystorePtr
- KeyStore -> MemoryKeystore

* Fix authority discovery worker and tests

* Rename 'insert_unknown' to 'insert'

* Remove leftover
2023-03-17 11:24:14 +00:00
Aaro Altonen 9ced14e2de Move code from sc-network-common back to sc-network (#13592)
* Move service tests to `client/network/tests`

These tests depend on `sc-network` and `sc-network-sync` so they should
live outside the crate.

* Move some configs from `sc-network-common` to `sc-network`

* Move `NetworkService` traits to `sc-network`

* Move request-responses to `sc-network`

* Remove more stuff

* Remove rest of configs from `sc-network-common` to `sc-network`

* Remove more stuff

* Fix warnings

* Update client/network/src/request_responses.rs

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* Fix cargo doc

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
2023-03-14 12:06:40 +00:00
André Silva da7190a113 grandpa: send neighbor packets to light clients on set transition (#13559)
* grandpa: send neighbor packets to light clients on set transition

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* remove match statement

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-03-09 14:05:04 +00:00
André Silva 13b0f24abd consensus: remove caching functionality from block import pipeline (#13551)
* consensus: remove caching functionality from block import pipeline

* client: update docs on Verifier::verify

* node: fix block production benchmark
2023-03-07 11:19:19 +00:00
Aaro Altonen 1a7f5be07f Extract syncing protocol from sc-network (#12828)
* Move import queue out of `sc-network`

Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.

This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.

* Move stuff to SyncingEngine

* Move `ChainSync` instanation to `SyncingEngine`

Some of the tests have to be rewritten

* Move peer hashmap to `SyncingEngine`

* Let `SyncingEngine` to implement `ChainSyncInterface`

* Introduce `SyncStatusProvider`

* Move `sync_peer_(connected|disconnected)` to `SyncingEngine`

* Implement `SyncEventStream`

Remove `SyncConnected`/`SyncDisconnected` events from
`NetworkEvenStream` and provide those events through
`ChainSyncInterface` instead.

Modify BEEFY/GRANDPA/transactions protocol and `NetworkGossip` to take
`SyncEventStream` object which they listen to for incoming sync peer
events.

* Introduce `ChainSyncInterface`

This interface provides a set of miscellaneous functions that other
subsystems can use to query, for example, the syncing status.

* Move event stream polling to `SyncingEngine`

Subscribe to `NetworkStreamEvent` and poll the incoming notifications
and substream events from `SyncingEngine`.

The code needs refactoring.

* Make `SyncingEngine` into an asynchronous runner

This commits removes the last hard dependency of syncing from
`sc-network` meaning the protocol now lives completely outside of
`sc-network`, ignoring the hardcoded peerset entry which will be
addressed in the future.

Code needs a lot of refactoring.

* Fix warnings

* Code refactoring

* Use `SyncingService` for BEEFY

* Use `SyncingService` for GRANDPA

* Remove call delegation from `NetworkService`

* Remove `ChainSyncService`

* Remove `ChainSync` service tests

They were written for the sole purpose of verifying that `NetworWorker`
continues to function while the calls are being dispatched to
`ChainSync`.

* Refactor code

* Refactor code

* Update client/finality-grandpa/src/communication/tests.rs

Co-authored-by: Anton <anton.kalyaev@gmail.com>

* Fix warnings

* Apply review comments

* Fix docs

* Fix test

* cargo-fmt

* Update client/network/sync/src/engine.rs

Co-authored-by: Anton <anton.kalyaev@gmail.com>

* Update client/network/sync/src/engine.rs

Co-authored-by: Anton <anton.kalyaev@gmail.com>

* Add missing docs

* Refactor code

---------

Co-authored-by: Anton <anton.kalyaev@gmail.com>
2023-03-06 16:33:38 +00:00
Davide Galassi 40c36c0c8a Move grandpa crates to consensus folder (#13458)
* Move grandpa under consensus dir
* Rename grandpa folder
* Finish grandpa renaming
* Minor tweaks
* Cargo fmt
* Adjust path to chain spec
2023-02-27 17:15:08 +01:00