* Run RustFmt as part of the CI
* Format repo
* Run RustFmt before the default Travis build step
Apparently if you override `script` you also need to make
sure to `build` and `test` the code yourself.
* Format repo
* Update dependencies
Upgrades Substrate based dependencies from v2.0.0 -> v2.0.0-alpha.1
and uses the `jsonrpsee`'s new feature flags. The actual code hasn't
been updated though, so this won't compile.
* Use `RawClient`s from `jsonrpsee`
* Update to use jsonrpsee's new API
* Hook up Ethereum Bridge Runtime, Relay, and Node Runtime
* Bump `parity-crypto` from v0.4 to v0.6
Fixes error when trying to compile tests. This was caused by
`parity-crypto` v0.4's use of `parity-secp256k1` over `secp256k1'.
Using the Parity fork meant multiple version of the same underlying
C library were being pulled in. `parity-crypto` v0.6 moved away from
this, only relying on `secp256k1` thus fixing the issue.
* Copy node-template over from Substrate repo
Got the template at rev=6e6d06c33911
* Use dependencies from crates.io + stop renaming on import
* Remove template pallet
* Stop using crates.io dependencies
Instead they're going to be pinned at v2.0.0-alpha.2
at commit `2afecf81ee19b8a6edb364b419190ea47c4a4a31`
until something stable comes along.
* Remove LICENSE
* Change references of `node-template` to `bridge-node`
* Remove README
* Fix some missed node-template references
* Add WASM toolchain to CI
* Be more specific about nightly version to use
* Maybe don't tie to a specific nightly
* Use composite accounts
* Update to use lazy reaping
* Only use Development chain config
* Initial commit. CLI which parses RPC urls.
* Establish ws connections and make simple RPC requests.
* Complete bridge setup.
* Process subscription events.
* Ctrl-C handler.
* Write a bare-bones README and copy in design doc.
* Modularize code a little bit.
* Communicate with each chain in a separate task.
* Parse headers from RPC subscription notifications.
* Send (fake) extrinsics across bridge channels.
And now it's deadlocked.
* Fix deadlock.
* Clarify in README that this is not-in-progress.
* Move everything into a single folder
* Move Substrate relay into appropriate folder
* Get the Substrate Relay node compiling
* Update Cargo.lock
* Use new composite accounts from Substrate
* Remove specification document
It has been moved to the Wiki on the Github repo.
* Update author + remove comments
* Use latest master for jsonrpsee
Required renaming some stuff (e.g Client -> RawClient)
Co-authored-by: Jim Posen <jim.posen@gmail.com>
commit 265365920836bb1d286c9b48b1902a2de278fdd9
Author: Hernando Castano <castano.ha@gmail.com>
Date: Wed Jan 29 19:51:15 2020 -0500
Move hc-jp-bridge repo to different folder
commit 8271991e95320baba70bd1cb9c4234d0ffd5b638
Merge: 57d0811 304cbc5
Author: Hernando Castano <castano.ha@gmail.com>
Date: Wed Jan 29 19:36:41 2020 -0500
Merge branch 'hc-jp-bridge-module' of hc-jp-bridge-module
commit 304cbc5f02d003ffa5404c1c01e461e5b8539888
Author: Hernando Castano <HCastano@users.noreply.github.com>
Date: Wed Jan 29 00:38:27 2020 -0500
Update bridge pallet to work with the (almost) lastest master (#4672)
* Update decl_error usage
* WIP: Update error handling to use DispatchResult
* Get module compiling with new error handling
* Make tests compile again
Main change was updating the usage of InMemoryBackend
* Move `sp-state-machine` into dev-dependencies
* Bump dependencies to v2.0.0
* Remove some stray comments
* Appy code review suggestion
commit 510cd6d96372688517496efa61773ea2839f8474
Author: Hernando Castano <HCastano@users.noreply.github.com>
Date: Tue Dec 17 12:52:51 2019 -0500
Move Bridge Pallet into FRAME (#4373)
* Move `bridge` crate into `frame` folder
* Make `bridge` pallet compile after `the-big-reorg`
commit ab54e838ef75e6a3f68fd0944bf22598c10c552f
Author: Hernando Castano <castano.ha@gmail.com>
Date: Mon Nov 11 21:56:40 2019 +0100
Use new StorageProof type from #3834
commit 8fc8911fd1b4acc2274c6863fb3dba91b30c90af
Author: Hernando Castano <HCastano@users.noreply.github.com>
Date: Tue Nov 5 00:50:34 2019 +0100
Verify Ancestry between Headers (#3963)
* Create module for checking ancestry proofs
* Use Vec of Headers instead of a HashMap
* Move the ancestry verification into the lib.rs file
* Change the proof format to exclude `child` and `ancestor` headers
* Add a testing function for building header chains
* Rename AncestorNotFound error to InvalidAncestryProof
* Use ancestor hash instead of header when verifying ancestry
* Clean up some stuff missed in the merge
commit dbe85738b68358b790cf927b34a804b965a88f96
or: Hernando Castano <HCastano@users.noreply.github.com>
Date: Fri Nov 1 15:41:58 2019 +0100
Check given Grandpa validator set against set found in storage (#3915)
* Make StorageProofChecker happy
* Update some tests
* Check given validator set against set found in storage
* Use Finality Grandpa's Authority Id and Weight
* Add better error handling
* Use error type from decl_error! macro
commit 31b09216603d3e9c21144ce8c0b6bf59307a4f97
or: Hernando Castano <HCastano@users.noreply.github.com>
Date: Wed Oct 23 14:55:37 2019 +0200
Make tests work after the changes introduced in #3793 (#3874)
* Make tests work after the changes introduced in #3793
* Remove unneccessary import
commit bce6d804aa86504599ff912387295c58f846cbf3
Author: Jim Posen <jim.posen@gmail.com>
Date: Thu Oct 10 12:18:58 2019 +0200
Logic for checking Substrate proofs from within runtime module. (#3783)
commit a7013e94b6c772c1d45a7cacbb445f73f6554fca
Author: Hernando Castano <castano.ha@gmail.com>
Date: Fri Oct 4 15:21:00 2019 +0300
Allow tracking of multiple bridges
commit 3cf648242d631e32bd553a67df54bf5a48912839
Author: Hernando Castano <castano.ha@gmail.com>
Date: Tue Oct 1 14:55:04 2019 +0200
Add BridgeId => Bridge mapping
commit 001c74c45072213e01857d0a2454379b447c5a76
Author: Hernando Castano <castano.ha@gmail.com>
Date: Tue Oct 1 11:10:19 2019 +0200
Get the mock runtime for tests set up
commit 38443a1e8b424ed2f148eb95121d009f730e3b5a
Author: Hernando Castano <castano.ha@gmail.com>
Date: Fri Sep 27 14:52:53 2019 +0200
Clean up some warnings
commit bdc3b01401e89c7111f8bf71f84c50750d25089f
Author: Hernando Castano <castano.ha@gmail.com>
Date: Thu Sep 26 16:41:01 2019 +0200
Add more skeleton code
commit 26995efbf4bac2842eb2822322f7ad3c3e88feb8
Author: Hernando Castano <castano.ha@gmail.com>
Date: Wed Sep 25 15:16:57 2019 +0200
Create `bridge` module skeleton
Cumulus test-parachain node and test runtime were still using relay
chain consensus and 12s blocktimes. With async backing around the corner
on the major chains we should switch our tests too.
Also needed to nicely test the changes coming to collators in #3168.
### Changes Overview
- Followed the [migration
guide](https://wiki.polkadot.network/docs/maintain-guides-async-backing)
for async backing for the cumulus-test-runtime
- Adjusted the cumulus-test-service to use the correct import-queue,
lookahead collator etc.
- The block validation function now uses the Aura Ext Executor so that
the seal of the block is validated
- Previous point requires that we seal block before calling into
`validate_block`, I introduced a helper function for that
- Test client adjusted to provide a slot to the relay chain proof and
the aura pre-digest
Added check if forklift already exists in ci image as forklift binary is
now bundled with the ci-unified.
This is a temporary check for the transition period
This PR brings the fix
https://github.com/paritytech/substrate/pull/13396 to polkadot-sdk.
In the past, due to insufficient inbound slot count on polkadot &
kusama, this fix led to low peer count. The situation has improved since
then after changing the default ratio between `--in-peers` &
`--out-peers`.
Nevertheless, it's expected that the reported total peer count with this
fix is going to be lower than without it. This should be seen as the
correct number of working connections reported, as opposed to also
reporting already closed connections, and not as lower count of working
connections with peers.
This PR also removes the peer eviction mechanism, as closed substream
detection is a more granular way of detecting peers that stopped syncing
with us.
The burn-in has been already performed as part of testing these changes
in https://github.com/paritytech/polkadot-sdk/pull/3426.
---------
Co-authored-by: Aaro Altonen <a.altonen@hotmail.com>
This is a tiny PR to increase the time a peer remains banned.
A peer is banned when the reputation drops below a threshold.
With every second, the peer reputation is exponentially decayed towards
zero.
For the previous setup:
- decaying to zero from (i32::MAX or i32::MIN) would take 948 seconds
(15mins 48seconds)
- from i32::MIN to escaping the banned threshold would take 10 seconds
This means we are decaying reputation a bit too aggressive and
misbehaving peers can misbehave again in 10 seconds.
Another side effect of this is that we have encountered multiple
warnings caused by a few misbehaving peers.
In the new setup:
- decaying to zero from (i32::MAX or i32::MIN) would take 3544 seconds
(59 minutes)
- from i32::MIN to escaping the banned threshold would take ~69 seconds
This is a followup of:
- https://github.com/paritytech/polkadot-sdk/pull/4000.
### Testing Done
- Created a misbehaving client with
[subp2p-explorer](https://github.com/lexnv/subp2p-explorer), the client
is banned for approx 69seconds until it is allowed to connect again.
cc @paritytech/networking
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
# Description
- What does this PR do?
1. Upgrades `trie-db`'s version to the latest release. This release
includes, among others, an implementation of `DoubleEndedIterator` for
the `TrieDB` struct, allowing to iterate both backwards and forwards
within the leaves of a trie.
2. Upgrades `trie-bench` to `0.39.0` for compatibility.
3. Upgrades `criterion` to `0.5.1` for compatibility.
- Why are these changes needed?
Besides keeping up with the upgrade of `trie-db`, this specifically adds
the functionality of iterating back on the leafs of a trie, with
`sp-trie`. In a project we're currently working on, this comes very
handy to verify a Merkle proof that is the response to a challenge. The
challenge is a random hash that (most likely) will not be an existing
leaf in the trie. So the challenged user, has to provide a Merkle proof
of the previous and next existing leafs in the trie, that surround the
random challenged hash.
Without having DoubleEnded iterators, we're forced to iterate until we
find the first existing leaf, like so:
```rust
// ************* VERIFIER (RUNTIME) *************
// Verify proof. This generates a partial trie based on the proof and
// checks that the root hash matches the `expected_root`.
let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();
// Print all leaf node keys and values.
println!("\nPrinting leaf nodes of partial tree...");
for key in trie.key_iter().unwrap() {
if key.is_ok() {
println!("Leaf node key: {:?}", key.clone().unwrap());
let val = trie.get(&key.unwrap());
if val.is_ok() {
println!("Leaf node value: {:?}", val.unwrap());
} else {
println!("Leaf node value: None");
}
}
}
println!("RECONSTRUCTED TRIE {:#?}", trie);
// Create an iterator over the leaf nodes.
let mut iter = trie.iter().unwrap();
// First element with a value should be the previous existing leaf to the challenged hash.
let mut prev_key = None;
for element in &mut iter {
if element.is_ok() {
let (key, _) = element.unwrap();
prev_key = Some(key);
break;
}
}
assert!(prev_key.is_some());
// Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
assert!(prev_key.unwrap() <= challenge_hash.to_vec());
// The next element should exist (meaning there is no other existing leaf between the
// previous and next leaf) and it should be greater than the challenged hash.
let next_key = iter.next().unwrap().unwrap().0;
assert!(next_key >= challenge_hash.to_vec());
```
With DoubleEnded iterators, we can avoid that, like this:
```rust
// ************* VERIFIER (RUNTIME) *************
// Verify proof. This generates a partial trie based on the proof and
// checks that the root hash matches the `expected_root`.
let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();
// Print all leaf node keys and values.
println!("\nPrinting leaf nodes of partial tree...");
for key in trie.key_iter().unwrap() {
if key.is_ok() {
println!("Leaf node key: {:?}", key.clone().unwrap());
let val = trie.get(&key.unwrap());
if val.is_ok() {
println!("Leaf node value: {:?}", val.unwrap());
} else {
println!("Leaf node value: None");
}
}
}
// println!("RECONSTRUCTED TRIE {:#?}", trie);
println!("\nChallenged key: {:?}", challenge_hash);
// Create an iterator over the leaf nodes.
let mut double_ended_iter = trie.into_double_ended_iter().unwrap();
// First element with a value should be the previous existing leaf to the challenged hash.
double_ended_iter.seek(&challenge_hash.to_vec()).unwrap();
let next_key = double_ended_iter.next_back().unwrap().unwrap().0;
let prev_key = double_ended_iter.next_back().unwrap().unwrap().0;
// Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
println!("Prev key: {:?}", prev_key);
assert!(prev_key <= challenge_hash.to_vec());
println!("Next key: {:?}", next_key);
assert!(next_key >= challenge_hash.to_vec());
```
- How were these changes implemented and what do they affect?
All that is needed for this functionality to be exposed is changing the
version number of `trie-db` in all the `Cargo.toml`s applicable, and
re-exporting some additional structs from `trie-db` in `sp-trie`.
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Update the Contracts API to use `WeightMeter`, as it simplifies the code
and makes it easier to reason about, rather than taking a mutable weight
or returning a tuple with the weight consumed
---------
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
[litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P
networking library. It supports all of the features of `rust-libp2p`
that are currently being utilized by Polkadot SDK.
Compared to `rust-libp2p`, `litep2p` has a quite different architecture
which is why the new `litep2p` network backend is only able to use a
little of the existing code in `sc-network`. The design has been mainly
influenced by how we'd wish to structure our networking-related code in
Polkadot SDK: independent higher-levels protocols directly communicating
with the network over links that support bidirectional backpressure. A
good example would be `NotificationHandle`/`RequestResponseHandle`
abstractions which allow, e.g., `SyncingEngine` to directly communicate
with peers to announce/request blocks.
I've tried running `polkadot --network-backend litep2p` with a few
different peer configurations and there is a noticeable reduction in
networking CPU usage. For high load (`--out-peers 200`), networking CPU
usage goes down from ~110% to ~30% (80 pp) and for normal load
(`--out-peers 40`), the usage goes down from ~55% to ~18% (37 pp).
These should not be taken as final numbers because:
a) there are still some low-hanging optimization fruits, such as
enabling [receive window
auto-tuning](https://github.com/libp2p/rust-yamux/pull/176), integrating
`Peerset` more closely with `litep2p` or improving memory usage of the
WebSocket transport
b) fixing bugs/instabilities that incorrectly cause `litep2p` to do less
work will increase the networking CPU usage
c) verification in a more diverse set of tests/conditions is needed
Nevertheless, these numbers should give an early estimate for CPU usage
of the new networking backend.
This PR consists of three separate changes:
* introduce a generic `PeerId` (wrapper around `Multihash`) so that we
don't have use `NetworkService::PeerId` in every part of the code that
uses a `PeerId`
* introduce `NetworkBackend` trait, implement it for the libp2p network
stack and make Polkadot SDK generic over `NetworkBackend`
* implement `NetworkBackend` for litep2p
The new library should be considered experimental which is why
`rust-libp2p` will remain as the default option for the time being. This
PR currently depends on the master branch of `litep2p` but I'll cut a
new release for the library once all review comments have been
addresses.
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Alexandru Vasile <alexandru.vasile@parity.io>
This MR contains two major changes and some maintenance cleanup.
## 1. Free Standing Pallet Benchmark Runner
Closes https://github.com/paritytech/polkadot-sdk/issues/3045, depends
on your runtime exposing the `GenesisBuilderApi` (like
https://github.com/paritytech/polkadot-sdk/pull/1492).
Introduces a new binary crate: `frame-omni-bencher`.
It allows to directly benchmark a WASM blob - without needing a node or
chain spec.
This makes it much easier to generate pallet weights and should allow us
to remove bloaty code from the node.
It should work for all FRAME runtimes that dont use 3rd party host calls
or non `BlakeTwo256` block hashing (basically all polkadot parachains
should work).
It is 100% backwards compatible with the old CLI args, when the `v1`
compatibility command is used. This is done to allow for forwards
compatible addition of new commands.
### Example (full example in the Rust docs)
Installing the CLI:
```sh
cargo install --locked --path substrate/utils/frame/omni-bencher
frame-omni-bencher --help
```
Building the Westend runtime:
```sh
cargo build -p westend-runtime --release --features runtime-benchmarks
```
Benchmarking the runtime:
```sh
frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all
```
## 2. Building the Benchmark Genesis State in the Runtime
Closes https://github.com/paritytech/polkadot-sdk/issues/2664
This adds `--runtime` and `--genesis-builder=none|runtime|spec`
arguments to the `benchmark pallet` command to make it possible to
generate the genesis storage by the runtime. This can be used with both
the node and the freestanding benchmark runners. It utilizes the new
`GenesisBuilder` RA and depends on having
https://github.com/paritytech/polkadot-sdk/pull/3412 deployed.
## 3. Simpler args for `PalletCmd::run`
You can do three things here to integrate the changes into your node:
- nothing: old code keeps working as before but emits a deprecated
warning
- delete: remove the pallet benchmarking code from your node and use the
omni-bencher instead
- patch: apply the patch below and keep using as currently. This emits a
deprecated warning at runtime, since it uses the old way to generate a
genesis state, but is the smallest change.
```patch
runner.sync_run(|config| cmd
- .run::<HashingFor<Block>, ReclaimHostFunctions>(config)
+ .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec))
)
```
## 4. Maintenance Change
- `pallet-nis` get a `BenchmarkSetup` config item to prepare its
counterparty asset.
- Add percent progress print when running benchmarks.
- Dont immediately exit on benchmark error but try to run as many as
possible and print errors last.
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
This PR introduces the github flow which will create a release draft
automatically when the rc tag is pushed. The flow contains the following
steps:
- Gets the info about rust version used to build the node
- Builds the runtimes using `srtool`
- Extracts the info about each runtime
- Aggregates the changelog from the prdocs
- Creates the release draft containing all the info related to the
release (changelog, runtimes, rust versions)
- Attaches the runtimes to the draft
- Posts the message to the RelEng internal channel to inform that the
build is done.
Related to the #3295
---------
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Sometimes you need to debug some issues just by the logs and reconstruct
what happened.
In these scenarios it would be nice to know if a block was imported as
best block, and what it parent was.
So here I propose to change the output of the informant to this:
```
2024-04-05 20:38:22.004 INFO ⋮substrate: [Parachain] ✨ Imported #18 (0xe7b3…4555 -> 0xbd6f…ced7)
2024-04-05 20:38:24.005 INFO ⋮substrate: [Parachain] ✨ Imported #19 (0xbd6f…ced7 -> 0x4dd0…d81f)
2024-04-05 20:38:24.011 INFO ⋮substrate: [jobless-children-5352] 🌟 Imported #42 (0xed2e…27fc -> 0x718f…f30e)
2024-04-05 20:38:26.005 INFO ⋮substrate: [Parachain] ✨ Imported #20 (0x4dd0…d81f -> 0x6e85…e2b8)
2024-04-05 20:38:28.004 INFO ⋮substrate: [Parachain] 🌟 Imported #21 (0x6e85…e2b8 -> 0xad53…2a97)
2024-04-05 20:38:30.004 INFO ⋮substrate: [Parachain] 🌟 Imported #22 (0xad53…2a97 -> 0xa874…890f)
```
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Improves `adder-collator` to also compute the parachain velocity. The
velocity is defined as number of parachain blocks progressing per relay
chain block.
In this test we're asserting that the elastic parachain always
progresses by 3 blocks per RCB, while the non-elastic parachain
progresses normally - 1 block per RCB.
---------
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: ordian <write@reusable.software>
With Coretime enabled we can no longer assume there is a static 1:1
mapping between core index and para id. This mapping should be obtained
from the scheduler/claimqueue on block by block basis.
This PR modifies `para_id()` (from `CoreState`) to return the scheduled
`ParaId` for occupied cores and removes its usages in the code.
Closes https://github.com/paritytech/polkadot-sdk/issues/3948
---------
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
I don't think there are any more releases to the 0.2.x versions, so best
we're on the 0.3.x release.
No change on the benchmarks, fast local time is still just as fast as
before:
new version bench:
```
fast_local_time time: [30.551 ns 30.595 ns 30.668 ns]
```
old version bench:
```
fast_local_time time: [30.598 ns 30.646 ns 30.723 ns]
```
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
1. Add `#[doc(no_inline)]` to frame umbrella crate re-exports that
eventually resolve to `frame_support_procedural` so docs don't look like
the screenshot below and instead link to the proper `frame-support`
docs.
<img width="1512" alt="Screenshot 2024-04-05 at 20 05 01"
src="https://github.com/paritytech/polkadot-sdk/assets/16665596/a41daa4c-ebca-44a4-9fea-f9f336314e13">
2. Remove `"Rust-Analyzer Users: "` prefix from
`frame_support_procedural` doc comments, since these doc comments are
visible in the web documentation and possible to stumble upon especially
when navigating from the frame umbrella crate.
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
dependabot-preview[bot]