Commit Graph

273 Commits

Author SHA1 Message Date
Andrew Plaza 4040eb0813 make ClientConfig public (#7544) 2020-11-16 22:50:44 +01:00
Andrew Plaza 32be264526 make LocalCallExecutor public (#7528) 2020-11-16 12:40:35 +01:00
Wei Tang 2bd9486b0f Export app-crypto specific keystore functions (#7489)
* Export app-crypto specific keystore functions

* Also add back the insert function

* Switch KeystoreContainer to an enum

* Only export the bare minimal for LocalKeystore and fix service compile

* fix: should return Arc

* Add docs stating that functions only available in local keystore

* Remove insert and generate functions

* fix: generate function should be available in test

* Add keypair function to trait

* Revert "Add keypair function to trait"

This reverts commit ad921b09ca73d3c09298e3a51b562ef8e0067781.

* Add note for local_keystore function in service
2020-11-11 18:19:27 +01:00
Seun Lanlege 6328f8740c Allow BabeConsensusDataProvider fork existing chain (#7078)
* parent affbc38afd
author Seun Lanlege <seunlanlege@gmail.com> 1599568164 +0100
committer Seun Lanlege <seunlanlege@gmail.com> 1604321289 +0100
gpgsig -----BEGIN PGP SIGNATURE-----

 iQGzBAABCgAdFiEECvQ02MnjnssnSbjr3HzzEhjN254FAl+gAAkACgkQ3HzzEhjN
 254soAv+KO5JA0HXSe0R0XS5TnwA3IxYsW+UvdF5dXFeC3jFdGTMvor818uoBePD
 dxzYEsUK6gjsNcM9+hpFhoy5JnUrUPInd2BZ7pmZiDuXmYJrHi0s7K5qL0EYDoe0
 m1egPNNyRR125ozJ24M+09c3OQsi3bvTx1TJaV9Aov8hK4So8UmlJTHWpkLw97ku
 HuTre2IPSFbV4GwJE40V+KNuDVHxaKL7zrInYScqbr6/hOTqBCvFn4ib3CjpF5HG
 zDAA5S2PrcbL9NQOothVcVB/TZr3IkhglCFqEjVyCX80IL0JkNZkw8jAh0B8uqXx
 Ug/c1/Mssa8F1jLZMmW45Cway60txqVbcWntPJAymGJbrRErOO/++oUrV0u1C65u
 LW7gXAaIJWQTX9KnX0SEyejNod7ubZktBz7n5WfkJAPIzdw5wtJalhLa673YTgQ9
 zyTPKiWjJj2myCq1AYrJvlK8hSsIBqbBFcUf1zX4SzZWKS+5mtp51o4gfVzcCRPd
 z/6/iPbB
 =g5tx
 -----END PGP SIGNATURE-----

BabeConsensusDataProvider works with existing chains

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-11-04 14:16:08 +01:00
Shawn Tabrizi 21d1cd0c89 Fix typo in toml (#7431) 2020-10-27 10:52:56 +00:00
Andrew Plaza 74910c4806 WASM Local-blob override (#7317)
* Provide WASM overwrite functionality in LocalCallExecutor

- add a new module `wasm_overwrite.rs` in client
  - scrapes given folder for runtimes
- add two new CLI Options `wasm-overwrite` and `wasm_overwrite_path`

* formatting

* Make comment clearer

remove sc-runtime-test from dev-dependencies

* comments

* Update client/service/src/client/wasm_overwrite.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/wasm_overwrite.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Fix spaces, remove call into backend for 'heap_pages' in 'try_replace'

* Error if path is not a directory, Comments,

Doc Comment for WasmOverwrite

* make WasmOverwrite Option<>

* Change to one CLI argument for overwrites

- move getting runtime version into LocalCallExecutor

* change unwrap() to expect()

* comment

* Remove `check_overwrites`

* Encapsulate checking for overwrites in LocalCallExecutor

* move duplicate code into function

* Update client/cli/src/params/import_params.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* comma

* Update client/service/src/client/wasm_overwrite.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/wasm_overwrite.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* cache hash in WasmBlob

* Update client/service/src/client/wasm_overwrite.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/client.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* move getting overwrite into its own function

* fix error when directory is not a directory

* Error on duplicate WASM runtimes

* better comment, grammar

* docs

* Revert StateBackend back to _

* Update client/service/src/client/wasm_overwrite.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/wasm_overwrite.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/call_executor.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Add two tests, fix doc comments

Add a test for the runtime_version method of WasmOverwrite
Add a test for check_overwrite method of LocalCallExecutor

* remove redundant `Return` from expect msg

* Update client/cli/src/params/import_params.rs

Co-authored-by: David <dvdplm@gmail.com>

* Update client/service/src/client/call_executor.rs

Co-authored-by: David <dvdplm@gmail.com>

* Update client/service/src/client/wasm_overwrite.rs

Co-authored-by: David <dvdplm@gmail.com>

* Update client/service/src/config.rs

Co-authored-by: David <dvdplm@gmail.com>

* Update client/service/src/client/wasm_overwrite.rs

Co-authored-by: David <dvdplm@gmail.com>

* Add Module Documentation, match on '.wasm' extension

* Add test for scraping WASM blob

* fix expect

* remove creating another block in LocalCallExecutor test

* remove unused import

* add tests for duplicates and scraping wasm

* make tests a bit nicer

* add test for ignoring non-.wasm files

* check error message in test

* Update client/service/src/client/wasm_overwrite.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* remove println

* Update client/service/src/client/wasm_overwrite.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* make tests prettier

* Update client/service/src/client/wasm_overwrite.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* comment for seemingly random client

* locally-built -> custom

* remove unused import

* fix comment

* rename all references to overwrite with override

* fix cli flag in module documentation

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: David <dvdplm@gmail.com>
2020-10-26 14:28:33 +01:00
Bastian Köcher 5808f467c5 Make Substrate compile with latest nightly (#7381)
* Try to get it compiling

* One more

* Make stable happy

* Make stable even more happy ;)

* Update shell.nix
2020-10-23 21:08:04 +02:00
Cecile Tonglet 8cebbd142d Add node name to the log lines (#7328)
* Initial commit

Forked at: 601e2fa139
Parent branch: origin/master

* WIP

Forked at: 601e2fa139
Parent branch: origin/master

* WIP

Forked at: 601e2fa139
Parent branch: origin/master

* WIP

Forked at: 601e2fa139
Parent branch: origin/master

* WIP

Forked at: 601e2fa139
Parent branch: origin/master

* WIP

Forked at: 601e2fa139
Parent branch: origin/master

* CLEANUP

Forked at: 601e2fa139
Parent branch: origin/master

* Add notes to original source code

* CLEANUP

Forked at: 601e2fa139
Parent branch: origin/master

* CLEANUP

Forked at: 601e2fa139
Parent branch: origin/master

* WIP

Forked at: 601e2fa139
Parent branch: origin/master

* WIP

Forked at: 601e2fa139
Parent branch: origin/master

* WIP

Forked at: 601e2fa139
Parent branch: origin/master

* CLEANUP

Forked at: 601e2fa139
Parent branch: origin/master

* WIP

Forked at: 601e2fa139
Parent branch: origin/master

* Some doc

* Test with trybuild

* Revert "Test with trybuild" (issue with trybuild atm)

This reverts commit 9055ec2206808ba3ddce6e3d87eb358907fa5e42.

https://github.com/dtolnay/trybuild/issues/53

* Apply suggestions

* Rename derive to proc-macro

* Remove "prefix" feature from informant

* Blocking task should use SpawnHandle::spawn_blocking

* Improve doc as suggested

* Fixes

Forked at: 601e2fa139
Parent branch: origin/master

* Apply suggestion

* Update client/cli/proc-macro/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* More suggestions

* CLEANUP

Forked at: 601e2fa139
Parent branch: origin/master

* Improve error message

* CLEANUP

Forked at: 601e2fa139
Parent branch: origin/master

* Fix async issue

* CLEANUP

Forked at: 601e2fa139
Parent branch: origin/master

* CLEANUP

Forked at: 601e2fa139
Parent branch: origin/master

* Add test

* fix doc test

* Update client/cli/src/logging.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/basic-authorship/src/basic_authorship.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/basic-authorship/src/basic_authorship.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Apply suggestions

* Suggestions

* Clarify doc

* WIP

Forked at: 601e2fa139
Parent branch: origin/master

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-10-21 15:13:07 +00:00
Max Inden 918a0c8077 *: Bump async-std to v1.6.5 (#7306)
* *: Bump async-std to v1.6.5

Prevent users from using v1.6.4 which faces issues receiving incoming
TCP connections. See https://github.com/async-rs/async-std/issues/888
for details.

* client/network/src/gossip: Use channel instead of condvar

`async_std::sync::Condvar::wait_timeout` uses
`gloo_timers::callback::Timeout` when compiled for
`wasm32-unknown-unknown`. This timeout implementation does not fulfill
the requirement of being `Send`.

Instead of using a `Condvar` use a `futures::channel::mpsc` to signal
progress from the `QueuedSender` to the background `Future`.

* client/network/Cargo.toml: Remove async-std unstable feature

* client/network/src/gossip: Forward all queued messages

* client/network/gossip: Have QueuedSender methods take &mut self

* client/network/gossip: Move queue_size_limit into QueuedSender

The `queue_size_limit` field is only accessed by `QueuedSender`, thus
there is no need to share it between the background future and the
`QueuedSender`.

* client/network/gossip: Rename background task to future

To be a bit picky the background task is not a task in the sense of an
asynchonous task, but rather a background future in the sense of
`futures::future::Future`.
2020-10-20 09:23:27 +00:00
Max Inden bd2dbc055c client/network: Remove option to disable yamux flow control (#7358)
With the `OnRead` flow control option yamux "send[s] window updates only
when data is read on the receiving end" and not as soon as "a Stream's
receive window drops to 0".

Yamux flow control has proven itself. This commit removes the feature
flag. Yamux flow control is now always enabled.
2020-10-19 21:43:32 +02:00
Bastian Köcher 74e8691aa8 Make queryStorage and storagePairs unsafe RPC functions (#7342)
The RPC calls can be rather expensive and can easily bring a RPC node in
some problems ;)
2020-10-18 19:58:52 +02:00
Ashley d1ef8507d7 Add missing fields to the light sync state (#7225)
* Initial draft

* Add an iterator that helps us get most items

* Revert changes to grandpa

* Change fields to just be the grandpa authority set and babe epoch changes

* Only use the fields we need from the shared authority set

* Switch to RPC call

* Revert "Only use the fields we need from the shared authority set"

This reverts commit 6ede87b0c5fe53f251d7cb45951006a7dc8f9b83.

* Add babe_finalized_block_weight from `ashley-improve-sync-state-WIP-loading`

* Fix rpc test

* Move sync state rpc stuff into sc-sync-state-rpc

* Remove as_json_value and remove unwraps from sc-sync-state-rpc

* Add clone_inner to SharedAuthoritySet
2020-10-15 10:10:43 +00:00
Pierre Krieger bcde7b4f3f Add an system_syncState RPC method (#7315)
* Add system_syncState RPC method

* Update client/rpc-api/src/system/helpers.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-10-13 17:53:49 +02:00
Rakan Alhneiti 3aa4bfacfc Async keystore + Authority-Discovery async/await (#7000)
* Asyncify sign_with

* Asyncify generate/get keys

* Complete BareCryptoStore asyncification

* Cleanup

* Rebase

* Add Proxy

* Inject keystore proxy into extensions

* Implement some methods

* Await on send

* Cleanup

* Send result over the oneshot channel sender

* Process one future at a time

* Fix cargo stuff

* Asyncify sr25519_vrf_sign

* Cherry-pick and fix changes

* Introduce SyncCryptoStore

* SQUASH ME WITH THE first commit

* Implement into SyncCryptoStore

* Implement BareCryptoStore for KeystoreProxyAdapter

* authority-discovery

* AURA

* BABE

* finality-grandpa

* offchain-workers

* benchmarking-cli

* sp_io

* test-utils

* application-crypto

* Extensions and RPC

* Client Service

* bin

* Update cargo.lock

* Implement BareCryptoStore on proxy directly

* Simplify proxy setup

* Fix authority-discover

* Pass async keystore to authority-discovery

* Fix tests

* Use async keystore in authority-discovery

* Rename BareCryptoStore to CryptoStore

* WIP

* Remote mutable borrow in CryptoStore trait

* Implement Keystore with backends

* Remove Proxy implementation

* Fix service builder and keystore user-crates

* Fix tests

* Rework authority-discovery after refactoring

* futures::select!

* Fix multiple mut borrows in authority-discovery

* Merge fixes

* Require sync

* Restore Cargo.lock

* PR feedback - round 1

* Remove Keystore and use LocalKeystore directly

Also renamed KeystoreParams to KeystoreContainer

* Join

* Remove sync requirement

* Fix keystore tests

* Fix tests

* client/authority-discovery: Remove event stream dynamic dispatching

With authority-discovery moving from a poll based future to an `async`
future Rust has difficulties propagating the `Sync` trade through the
generated state machine.

Instead of using dynamic dispatching, use a trait parameter to specify
the DHT event stream.

* Make it compile

* Fix submit_transaction

* Fix block_on issue

* Use await in async context

* Fix manual seal keystore

* Fix authoring_blocks test

* fix aura authoring_blocks

* Try to fix tests for auth-discovery

* client/authority-discovery: Fix lookup_throttling test

* client/authority-discovery: Fix triggers_dht_get_query test

* Fix epoch_authorship_works

* client/authority-discovery: Remove timing assumption in unit test

* client/authority-discovery: Revert changes to termination test

* PR feedback

* Remove deadcode and mark test code

* Fix test_sync

* Use the correct keyring type

* Return when from_service stream is closed

* Convert SyncCryptoStore to a trait

* Fix line width

* Fix line width - take 2

* Remove unused import

* Fix keystore instantiation

* PR feedback

* Remove KeystoreContainer

* Revert "Remove KeystoreContainer"

This reverts commit ea4a37c7d74f9772b93d974e05e4498af6192730.

* Take a ref of keystore

* Move keystore to dev-dependencies

* Address some PR feedback

* Missed one

* Pass keystore reference - take 2

* client/finality-grandpa: Use `Arc<dyn CryptoStore>` instead of SyncXXX

Instead of using `SyncCryptoStorePtr` within `client/finality-grandpa`,
which is a type alias for `Arc<dyn SyncCryptoStore>`, use `Arc<dyn
CryptoStore>`. Benefits are:

1. No additional mental overhead of a `SyncCryptoStorePtr`.

2. Ability for new code to use the asynchronous methods of `CryptoStore`
instead of the synchronous `SyncCryptoStore` methods within
`client/finality-granpa` without the need for larger refactorings.

Note: This commit uses `Arc<dyn CryptoStore>` instead of
`CryptoStorePtr`, as I find the type signature more descriptive. This is
subjective and in no way required.

* Remove SyncCryptoStorePtr

* Remove KeystoreContainer & SyncCryptoStorePtr

* PR feedback

* *: Use CryptoStorePtr whereever possible

* *: Define SyncCryptoStore as a pure extension trait of CryptoStore

* Follow up to SyncCryptoStore extension trait

* Adjust docs for SyncCryptoStore as Ben suggested

* Cleanup unnecessary requirements

* sp-keystore

* Use async_std::task::block_on in keystore

* Fix block_on std requirement

* Update primitives/keystore/src/lib.rs

Co-authored-by: Max Inden <mail@max-inden.de>

* Fix wasm build

* Remove unused var

* Fix wasm compilation - take 2

* Revert async-std in keystore

* Fix indent

* Fix version and copyright

* Cleanup feature = "std"

* Auth Discovery: Ignore if from_service is cloed

* Max's suggestion

* Revert async-std usage for block_on

* Address PR feedback

* Fix example offchain worker build

* Address PR feedback

* Update Cargo.lock

* Move unused methods to test helper functions

* Restore accidentally deleted cargo.lock files

* Fix unused imports

Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2020-10-08 20:56:35 +00:00
Max Inden a772eb27a2 client/cli: Update to fdlimit 0.2.1 (#7249)
fdlimit 0.2.1 contains a small patch allowing compilation on platforms
where libc represents rlimits as u32.
2020-10-02 08:51:21 +00:00
Benjamin Kampmann ecdc94420e Releasing 2.0 – two dot 😮 (#7182)
Tagging the release of substrate 2.0


[ci: skip-checks]
2020-09-22 19:47:38 +02:00
Steve Degosserie 22632efc5f Update READMEs, fix links (#7176)
* Re-generate READMEs to fix doc links; set readme field in package manifests

* Re-generate READMEs to fix doc links; set readme field in package manifests

* Re-generate READMEs to fix doc links; set readme field in package manifests

* Re-generate READMEs to fix doc links; set readme field in package manifests

* Revert stuff that shouldn't have been committed

* Revert stuff that shouldn't have been committed

* Fix parent relative link generation

* Manually remove this incorrect link for now.
2020-09-22 13:38:23 +00:00
Maciej Hirsz ae52587f41 Bump jsonrpc-* dependencies to v15 (#7157)
* Bump jsonrpc-* dependencies to v15

* Remove unused imports
2020-09-21 08:46:10 +00:00
Anton Gavrilov 2c5a82f8f1 Prometheus metrics for RPC calls (#7088)
* WS and HTTP middlewares added

* Prometheus endpoint added

* Counters renamed

* Proper style for inc

* Metrics initialization re-written

* Rework to handler middleware

* Introduce transport prefix for metrics

* String shortened

* Commented code removed, new line inserted

* One more string shortened

* Wasm build fixed

* Wasm build fixed once again

* Rework to shared metrics

* Added collectors label

* Tilde removed from cargo

* Switch to owned metrics in parameters
2020-09-18 15:15:19 +02:00
Benjamin Kampmann a9c73113a8 Tracing for wasm with bridging to native (#6916)
* implement events handling, implement parent_id for spans & events

* add events to sp_io::storage

* update test

* add tests

* adjust limit

* let tracing crate handle parent_ids

* re-enable current-id tracking

* add test for threads with CurrentSpan

* fix log level

* remove redundant check for non wasm traces

* remove duplicate definition in test

* Adding conditional events API

* prefer explicit parent_id over current,

enhance test

* limit changes to client::tracing event implementation

* remove From impl due to fallback required on parent_id

* make tracing codecable

* replace with global tracing

* new tracing interface

* impl TracingSubscriber in client

* implement access to global TracingSubscriber from primitives

* span for wasm

* increment towards Wasm Tracing Subscriber implementation

* increment, remove sp-tracing from runtime-interface

* increment, it compiles

* attained original functionality with new mechanism

* implement remaining TracingSubscriber functions

* remove spans from decl_module

* add handling for encoded values

* Revert "replace with global tracing"

This reverts commit 8824a60deea54d9b437407a21c8ceaf6a1902ee5.

* Wasm Side Tracing

* tracing on wasm

* enable tracing wasm on node-runtime

* export all the macros in std

* tracing subscriber on wasm-side only

* pass spans and events over and record them

* reactivate previous code and  cleanup

* further cleaning up

* extend the span macros, activate through executive

* tracking the actual extrinsic, too

* style

* fixing tests

* spaces -> tabs

* attempting to reactivate params

* activate our tests in CI

* some passing

* tests passing

* with core lazy

* global tracer for wasm side with pass over

* fixing metadata referencing

* remove const_fn feature requirement

* reenable dispatch traces

* reset client tracing

* further cleaning up

* fixing runtime-test

* move tracing-build setup into runtime-test

* Merge DebugWriter from tracing and frame-support, move to sp-std

* remove dangling fixme

* Docs for tracing primitives

* cleaning up a bit more

* Wasm interface docs

* optimise docs.rs setup

* adding tracing flags to uncomment

* remove brace

* fixing imports

* fixing broken syntax

* add required modules

* nicer formatting

* better target management

* adding low level storage tracing events into frame

* add custom Debug impl for WasmMetadata

* cloning profiler

* adding info about cloning profiler

* using in-scope for within calls

* proper time tracing, cleaning up println

* allow to disable tracing on runtime_interface-macro

* disable tracing for wasm-tracing-interface

* simplify wasm-tracing-api

* update client to new interface

* fixing docs and tests for sp-tracing

* update integration tests

* re-activating enter_span

* dropping FIXME, it's documented

* fix formatting

* fix formatting

* fix imports

* more debug info

* inform wasm about it being disabled by returning 1

* only one tracer, but enabled multi-all support

* make trait pub again for tests

* Apply suggestions from code review

Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>

* fixing wasm doc tests for proper usage

* remove unnecessary import

* fixing formatting

* minor style fixes

* downgrading wabt

* update error message for UI

* Fix interface test

* next attempt to fix macros

* geee

* revert tracing on hashed for future PR

* remove local macros, use originals

* we are able to convert to static items

* implement more WasmValue types

* adding support to convert str, debug and encoded values

* more minor fixes

* revert unsafe 'static making

* fix indentation

* remove commented lines

* bump all them tracing versions

* cleaning up docs and info

* document new flag

* the new layered system handles span cloning better

* Apply suggestions from code review

Co-authored-by: David <dvdplm@gmail.com>

Co-authored-by: Matt Rutherford <mattrutherford@users.noreply.github.com>
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
Co-authored-by: David <dvdplm@gmail.com>
2020-09-18 06:10:59 +00:00
Wei Tang 9fdd4672b0 Use tracing-based subscriber logging (#6825)
* init_logger: switch from log-based to tracing-based and add compatibility layer

* Move tracing profiling subscriber related config realization

* sp-tracing: change profiling to be a layer instead of a subscriber

* Enable profiling layer in cli

* Change all test env_logger init to sp_tracing::try_init_simple

* Remove all local env_logger dependency

* Add missing tracing-subscriber dependency

* frame-sudo: fix tests

* frame-support: fix tests

* Fix frame/pallet and executor tests

* Fix the remaining tests

* Use subscriber's try_init as recommended by @davidbarsky

* Be explict that the tracing-log feature is needed

* Set subscriber writer to stderr

* Shorter line width

* Update cargo lock tracing version

* Fix sc_tracing crate compile

* Fix sc_authority_discovery crate test

* unremove default-features

* Leave enabled to default true

* Warn if global default cannot be set

* Fix unused import

* Remove unused PROXY_TARGET

* Change all reference from rc5 to rc6

* Change all reference of rc2 to rc6

* Fix styling

* Fix typo

* make logger init error'ing

* re-fixing the test issue

Co-authored-by: Benjamin Kampmann <ben@parity.io>
2020-09-17 11:04:43 +02:00
Bastian Köcher 0a6f7b08d7 Send import notification always for re-orgs (#7118)
* Send import notification always for re-orgs

This pr changes the behavior of sending import notifications. Before we
only send notifications when importing blocks on the tip of the chain or
on similar conditions. However we did not send a notification when we
for example being in a state where we import multiple blocks to catch
up. If we re-org in this process, systems like the transaction pool
would not be notified about this re-org. This means, that we would also
not resubmit transactions of these retracted blocks. This pr fixes this,
by always sending a notification on a re-org.

See
https://github.com/substrate-developer-hub/substrate-node-template/issues/82
for some context about the bug.

* Update client/service/src/client/client.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2020-09-16 16:43:43 +02:00
Pierre Krieger eadf5b09f9 Allow remotes to not open a legacy substream (#7075)
* Allow remotes to not open a legacy substream

* Misc fixes

* Special case first protocol as the one bearing the handshake
2020-09-14 15:08:23 +00:00
Jon Häggblad 2da6adfdf1 grandpa-rpc don't share subscription manager, only executor (#7039)
* service builder: fix todo about jsonrpc Option workaround

* grandpa-rpc: only share executor instead of sub manager

* grandpa-rpc: fix compilation

* grandpa-rpc: rename to subscription_executor

* node/cli: remove another unused jsonrpc dependency

* grandpa: apply style fixes from code review

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2020-09-14 11:02:37 +00:00
Ashley 614c6a743f Add a build-sync-spec subcommand and remove the CHT roots from the light sync state. (#6999)
* Move subcommands from sc-cli to nodes

* Add --build-sync-spec subcommand

* Remove CHTs from snapshots

* Keep ProvideChtRoots
2020-09-11 14:50:12 +02:00
kaichao 91c0213413 Set reserved nodes with offchain worker. (#6996)
* add offchain worker api to set reserved nodes.

* new offchain api to get node public key.

* node public key from converter

* refactor set reserved nodes ocw api.

* new ndoe authorization pallet

* remove unnecessary clone and more.

* more

* tests for node authorization pallet

* remove dependency

* fix build

* more tests.

* refactor

* Update primitives/core/src/offchain/testing.rs

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update frame/node-authorization/src/lib.rs

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update frame/node-authorization/src/lib.rs

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update frame/node-authorization/src/lib.rs

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* format code

* expose NetworkService

* remove NetworkStateInfo in offchain

* replace NodePublicKey with PeerId.

* set max length of peer id.

* clear more

* use BTreeSet for set of peers.

* decode opaque peer id.

* extract NetworkProvider for client offchain.

* use OpaquePeerId in node authorization pallet.

* fix test

* better documentation

* fix test

* doc

* more fix

* Update primitives/core/src/offchain/mod.rs

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* Update client/offchain/src/api.rs

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* derive serialize and deserialize

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-09-10 15:26:09 +00:00
Roman Borschel ec47877288 Refactor & detach network metrics. (#6986)
* Refactor sc-network/service metrics.

  1. Aggregate sc-network metrics into a submodule, introducing
  two more sourced metrics to avoid duplicate atomics.

  2. Decouple periodic sc-service network metrics from other
  metrics, so that they can be updated independently.

* Update client/service/src/metrics.rs

* Update client/service/src/metrics.rs
2020-09-06 19:59:05 +02:00
Max Inden 875daa31bf client/cli/src/config: Warn on low file descriptor limit (#6956)
* client/cli/src/config: Warn on low file descriptor limit

Substrate sets the soft file descriptor limit to the hard limit at
startup. In the case of the latter being low already (< 10_000) a
Substrate node under high demand might run into issues e.g. when opening
up new TCP connections or persisting data to the database.

With this commit a warn message is printed to stderr.

* client/cli/Cargo.toml: Update to fdlimit 0.2.0
2020-08-31 12:37:06 +00:00
Pierre Krieger 1bd6082cf7 Enforce that ProtocolId is a string (#6953)
* Enforce that ProtocolId is a string

* Fix test
2020-08-26 12:27:30 +00:00
Ashley c8ca2ce183 Dynamically generate CHT roots on a full client (#6944)
* Generate CHT roots on a full client

* add changes_trie_root function

* Add a test

* Line widths

* Fix sc-service-test

* Clarify comments

* Revert comments
2020-08-24 15:11:21 +02:00
Ashley f009f6c227 Add a LightSyncState field to the chain spec (#6894)
* Reset code, almost ready for PR

* Improved build_hardcoded_spec

* Fix line widths

* Fix tests

* Fix sc-service-test

* Suggestions from code review

* Rename to LightSyncState

* It's not syncing :^(

* It syncs!

* Remove rpc call

* Convert spaces to tabs

* Moved sc-service things to export_sync_state.rs

* Fix tests

* Wait for syncing with network_status_sinks

* Remove sc-network from node-template

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Various changes, split the flag up into 2 pieces to make testing easier.

* Update client/cli/src/commands/build_spec_cmd.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Revert a lot of changes

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-08-24 08:34:16 +00:00
Qinxuan Chen 8ae1695526 Remove duplicated dependencies (#6930)
Signed-off-by: koushiro <koushiro.cqx@gmail.com>
2020-08-21 16:25:58 +02:00
Benjamin Kampmann 2416aa7999 prepping for releasing rc6 (#6922)
* Bump version

* update test-utils crates to be ready for publishing

* adding changelog

* Adding automaticly generated READMEs

* fixing versions

* another version mishap
2020-08-20 17:04:42 +02:00
Seun Lanlege 6c1737fe77 RpcHandlers Refactorings (#6846)
* allow access to the underlying Pubsub instance from RpcHandlers

* bump Cargo.lock

* no more Arc<RpcHandlers>

* bump Cargo.lock

* Debug,.

* Arc<RpcHandlers>

* RpcHandler

* RpcHandlers::io_handler

* remove chain spec from cli

* address pr comments

* remove stray newline

Co-authored-by: Ashley <ashley.ruglys@gmail.com>

Co-authored-by: Tomasz Drwięga <tomasz@parity.io>
Co-authored-by: Ashley <ashley.ruglys@gmail.com>
2020-08-15 09:08:31 +00:00
Roman Borschel 327e02942c Update to libp2p-0.23. (#6870)
* Update to libp2p-0.23.

Thereby incorporate bandwidth measurement along the
lines previously done by libp2p itself.

* Tweak dependencies for wasm32 compilation.

For wasm32 we need to enable unstable features to
make `task::Builder::local` available.

* Simplify dependencies.

* Simplify.

Leave the calculation of bytes sent/received per second
to the outer layers of the code, subject to their own
individual update intervals.

* Cleanup

* Re-add lost dev dependency.

* Avoid division by zero.

* Remove redundant metric.

* Enable sending of noise legacy handshakes.

* Add comment about monotonic gauge.

* CI
2020-08-14 08:41:47 +00:00
mattrutherford 9442c19b48 Implement tracing::Event handling & parent_id for spans and events (#6672)
* implement events handling, implement parent_id for spans & events

* add events to sp_io::storage

* update test

* add tests

* adjust limit

* let tracing crate handle parent_ids

* re-enable current-id tracking

* add test for threads with CurrentSpan

* fix log level

* remove redundant check for non wasm traces

* remove duplicate definition in test

* Adding conditional events API

* prefer explicit parent_id over current,

enhance test

* limit changes to client::tracing event implementation

* remove From impl due to fallback required on parent_id

* implement SPAN_LIMIT

change event log output

* change version of tracing-core

* update dependancies

* revert limit

* remove duplicate dependency

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Matt Rutherford <mattrutherford@users.noreply.github.com>
Co-authored-by: Benjamin Kampmann <ben@parity.io>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-08-12 11:53:21 +00:00
Hernando Castano 433b7214f5 Add Subscription RPC for Grandpa Finality (#5732)
* Rough skeleton for what I think the RPC should look like

* Create channel for sending justifications

Sends finalized header and justification from Grandpa to the
client. This lays the groundwork for hooking into the RPC module.

* WIP: Add subscribers for justifications to Grandpa

Adds the Sender end of a channel into Grandpa, through which notifications
about block finality events can be sent.

* WIP: Add a struct for managing subscriptions

Slightly different approach from the last commit, but same
basic idea. Still a rough sketch, very much doesn't compile yet.

* Make naming more clear and lock data in Arc

* Rough idea of what RPC would look like

* Remove code from previous approach

* Missed some things

* Update client/rpc-api/src/chain/mod.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update client/rpc-api/src/chain/mod.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Split justification subscription into sender and receiver halves

* Replace RwLock with a Mutex

* Add sample usage from the Service's point of view

* Remove code that referred to "chain_" RPC

* Use the Justification sender/receivers from Grandpa LinkHalf

* Add some PubSub boilerplate

* Add guiding comments

* TMP: comment out to fix compilation

* Return MetaIoHandler from PubSubHandler in create_full

* Uncomment pubsub methods in rpc handler (fails to build)

* node/rpc: make Metadata concrete in create_full to fix compilation

* node: pass in SubscriptionManger to grandpa rpc handler

* grandpa-rpc: use SubscriptionManger to add subscriber

* grandpa-rpc: attempt at setting up the justification stream (fails to build)

* grandpa-rpc: fix compilation of connecting stream to sink

* grandpa-rpc: implement unsubscribe

* grandpa-rpc: update older tests

* grandpa-rpc: add full prefix to avoid confusing rust-analyzer

* grandpa-rpc: add test for pubsub not available

* grandpa-rpc: tidy up leftover code

* grandpa-rpc: add test for sub and unsub of justifications

* grandpa-rpc: minor stylistic changes

* grandpa-rpc: split unit test

* grandpa-rpc: minor stylistic changes in test

* grandpa-rpc: skip returning future when cancelling

* grandpa-rpc: reuse testing executor from sc-rpc

* grandpa-rpc: don't need to use PubSubHandler in tests

* node-rpc: use MetaIoHandler rather than PubSubHandler

* grandpa: log if getting header failed

* grandpa: move justification channel creation into factory function

* grandpa: make the justification sender optional

* grandpa: fix compilation warnings

* grandpa: move justification notification types to new file

* grandpa-rpc: move JustificationNotification to grandpa-rpc

* grandpa-rpc: move JustificationNotification to its own file

* grandpa: rename justification channel pairs

* grandpa: rename notifier types

* grandpa: pass justification as GrandpaJustification to the rpc module

* Move Metadata to sc-rpc-api

* grandpa-rpc: remove unsed error code

* grandpa: fix bug for checking if channel is closed before sendind

* grandpa-rpc: unit test for sending justifications

* grandpa-rpc: update comments for the pubsub test

* grandpa-rpc: update pubsub tests with more steps

* grandpa-rpc: fix pubsub test

* grandpa-rpc: minor indendation

* grandpa-rpc: decode instead of encode in test

* grandpa: fix review comments

* grandpa: remove unused serde dependency

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Jon Häggblad <jon.haggblad@gmail.com>
Co-authored-by: Tomasz Drwięga <tomasz@parity.io>
2020-08-10 10:31:36 +00:00
Cecile Tonglet 837c18ddfb Renamed add_children to add_child since it adds only one child (#6838) 2020-08-07 10:56:32 +02:00
Cecile Tonglet 4e03c87953 Allow task manager to have children (#6771)
* Initial commit

Forked at: 7df97abab4
Parent branch: origin/master

* WIP

Forked at: 7df97abab4
Parent branch: origin/master

* WIP

Forked at: 7df97abab4
Parent branch: origin/master

* WIP

Forked at: 7df97abab4
Parent branch: origin/master

* WIP

Forked at: 7df97abab4
Parent branch: origin/master

* WIP

Forked at: 7df97abab4
Parent branch: origin/master

* WIP

Forked at: 7df97abab4
Parent branch: origin/master

* changelog

* Remove Box

* Make future nicer

* Revert "Make future nicer"

This reverts commit 49fb8fb6f245c3ca2c384468df14b34f34616736.

* Simplify

* Additional check

* Simplify more

Co-authored-by: Bastian Köcher <git@kchr.de>
2020-08-06 21:20:46 +02:00
Pierre Krieger dbc66479f3 Delay network startup to after complete initialization (#6833)
* Delay network startup to after complete initialization

* Update client/service/src/builder.rs

Co-authored-by: Ashley <ashley.ruglys@gmail.com>

Co-authored-by: Ashley <ashley.ruglys@gmail.com>
2020-08-06 12:46:34 +00:00
André Silva 4a94fe860d service: remove collection of system/process metrics (#6822) 2020-08-05 15:58:07 +00:00
Ashley 07facb13c2 Convert spaces to tabs (#6799) 2020-08-03 15:48:32 +02:00
Wei Tang 3c5cbb00aa Allow blacklisting blocks from being finalized again after block revert (#6301)
* Allow blacklisting blocks from being finalized again after block revert

* Use BlockRules for storing unfinalized and add have_state_at in revert

* Move finalization_check in finalize_block upward

* Directly mark finalization blacklist as badblocks

* Remove obselete comment
2020-07-31 14:32:13 +02:00
Bastian Köcher e7d8040af8 Update parity-scale-codec to prepare for breaking rustc release (#6746)
This updates parity-scale-codec{-derive} to prepare for a rustc release
that would otherwise break the derive implementation:
https://github.com/rust-lang/rust/pull/73084
2020-07-28 20:26:01 +00:00
Ashley 9220b646d2 Various small improvements to service construction. (#6738)
* Remove service components and add build_network, build_offchain_workers etc

* Improve transaction pool api

* Remove commented out line

* Add PartialComponents

* Add BuildNetworkParams, documentation

* Remove unused imports in tests

* Apply suggestions from code review

Co-authored-by: Nikolay Volf <nikvolf@gmail.com>

* Remove unused imports in node-bench

Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
2020-07-28 15:21:33 +02:00
Bastian Köcher 9310f15ac2 Name all the tasks! (#6726)
* Remove any implementation of `Spawn` or `Executor` from our task executors

* Fix compilation

* Rename `SpawnBlockingExecutor`

* Update primitives/core/src/traits.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Fix tests

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2020-07-26 12:56:17 +00:00
Ashley ffa2177d46 Remove the service builder (#6557)
* :)

* Slight tidy

* Remove ServiceBuilderCommand

* Remove whitespace

* Keep task manager alive for check_block/import_blocks

* Pass task_manager to run_until_exit

* WIP

* WIP

* Get rid of the macros

* Simplify a few chain components creation APIs related to the service

* Fix basic-authorship doc tests

* Remove DefaultQueue

* Update client/service/src/builder.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Move ExecutionExtensions comment around

* Remove unused BlakeTwo256

* Add sc-prelude

* Rename sc-prelude to sc-service-prelude

* Rename to sc-service-types

* Improve service types

* Fix line widths

* Remove sc-service-types and move type definitions to crates

* Update bin/node-template/node/src/service.rs

Co-authored-by: Seun Lanlege <seunlanlege@gmail.com>

* Add TLightClientWithHash

* Rework types

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Seun Lanlege <seunlanlege@gmail.com>
2020-07-22 19:10:27 +00:00
Benjamin Kampmann 7c161ec177 Update to latest sysinfo prevents leaking fd-handlers (#6708)
* Ensure we close old file handles hold by sysinfo

* Dropping is needed unfortunately

* enable process refreshing, ignore result from refresh_process

* jumping to proposed patch

* switch to latest sysinfo
2020-07-22 15:48:12 +00:00
Cecile Tonglet de3181076a Graceful shutdown for the task manager (#6654)
* Initial commit

Forked at: 74655edebc
Parent branch: origin/master

* Move task_manager.rs to mod.rs

* Graceful shutdown for the task manager

* Await all background task JoinHandle at the same time

* Add tests

* Make future() wait also for exit signal + fix essential task failed

Probably related to https://github.com/paritytech/cumulus/issues/111

* add comments for non-obvious code

* Use clean_shutdown() in sc-cli

* Adapt code and upgrade tokio in sc-cli

* cleanup spacing in doc

* Add license

* I guess actually running the clean shutdown would be a good idea

* fix tests

* Update client/cli/src/runner.rs

Co-authored-by: Benjamin Kampmann <ben@gnunicorn.org>

* Improve error logging

* disable other tests (can't reproduce on my machine)

* Revert "disable other tests (can't reproduce on my machine)"

This reverts commit c133c590f33c253123ba0555ce719a71ededd60d.

* It is possible that the tasks are ended first

* Revert "It is possible that the tasks are ended first"

This reverts commit 502aba4a49fb5d892e704c412b8a81768a3f2c71.

* Use single threaded scheduler for more predictability

* enable_time

* Revert "enable_time"

This reverts commit 4e152140764a4bddeedff06a4e36ec701909e8c7.

* Revert "Use single threaded scheduler for more predictability"

This reverts commit ee5e13c5f13ff71e012dcda13579dffeb15f8ffc.

* Revert "Revert "It is possible that the tasks are ended first""

This reverts commit 1b91a8ca3eebbdc18be199c8ca188e88669ae649.

* This cannot be verified either with a threaded pool

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Benjamin Kampmann <ben@parity.io>
Co-authored-by: Benjamin Kampmann <ben@gnunicorn.org>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-07-22 07:15:22 +00:00
pscott 046fda914a Improve overall performance (#6699)
* Improve overall performance

* Clean up code

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Remove needless ::

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Remove needless ::

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2020-07-21 14:46:49 +02:00