Commit Graph

833 Commits

Author SHA1 Message Date
André Silva 7fa96261ba grandpa: change some logging from trace to debug (#6872)
* grandpa: change some logging from trace to debug

* grandpa: cleanup unused import
2020-08-11 15:05:59 +00:00
Nikolay Volf c347300e3e Move to upstream wasmtime, refactor globals snapshot (#6759)
* refactor globals snapshot

* ignore test

* update pwasm-utils ref

* line width

* add doc comment for internal struct

* add explanation for iteration

* Demote rustdoc to a comment

* use 0.14

Co-authored-by: Sergei Shulepov <sergei@parity.io>
2020-08-11 15:05:31 +00:00
Arkadiy Paronyan 58ebf50839 Block packet size limit (#6398)
* Block packet size limit

* Update client/network/src/protocol.rs

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* Add block response limit

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-08-11 16:12:30 +02:00
Kian Paimani b0342f0b12 state_getSize RPC for storage maps (#6847)
* Fancy compact encode/decode impl for compact solution

* Make it optional

* Remove extra file

* Update primitives/npos-elections/compact/src/lib.rs

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

* Final fixes.

* getSize rpc should work for maps as well

* Fix future types

* Remove minimum_validator_count stale const

* Update client/rpc/src/state/mod.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* "Optimize" `storage_size`

* Remove unused import

* Update doc

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
2020-08-11 12:09:52 +00:00
André Silva 253f906a4d grandpa: log errors and grandpa unexpected conclusion (#6867) 2020-08-10 21:52:39 +00:00
Hernando Castano 433b7214f5 Add Subscription RPC for Grandpa Finality (#5732)
* Rough skeleton for what I think the RPC should look like

* Create channel for sending justifications

Sends finalized header and justification from Grandpa to the
client. This lays the groundwork for hooking into the RPC module.

* WIP: Add subscribers for justifications to Grandpa

Adds the Sender end of a channel into Grandpa, through which notifications
about block finality events can be sent.

* WIP: Add a struct for managing subscriptions

Slightly different approach from the last commit, but same
basic idea. Still a rough sketch, very much doesn't compile yet.

* Make naming more clear and lock data in Arc

* Rough idea of what RPC would look like

* Remove code from previous approach

* Missed some things

* Update client/rpc-api/src/chain/mod.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update client/rpc-api/src/chain/mod.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Split justification subscription into sender and receiver halves

* Replace RwLock with a Mutex

* Add sample usage from the Service's point of view

* Remove code that referred to "chain_" RPC

* Use the Justification sender/receivers from Grandpa LinkHalf

* Add some PubSub boilerplate

* Add guiding comments

* TMP: comment out to fix compilation

* Return MetaIoHandler from PubSubHandler in create_full

* Uncomment pubsub methods in rpc handler (fails to build)

* node/rpc: make Metadata concrete in create_full to fix compilation

* node: pass in SubscriptionManger to grandpa rpc handler

* grandpa-rpc: use SubscriptionManger to add subscriber

* grandpa-rpc: attempt at setting up the justification stream (fails to build)

* grandpa-rpc: fix compilation of connecting stream to sink

* grandpa-rpc: implement unsubscribe

* grandpa-rpc: update older tests

* grandpa-rpc: add full prefix to avoid confusing rust-analyzer

* grandpa-rpc: add test for pubsub not available

* grandpa-rpc: tidy up leftover code

* grandpa-rpc: add test for sub and unsub of justifications

* grandpa-rpc: minor stylistic changes

* grandpa-rpc: split unit test

* grandpa-rpc: minor stylistic changes in test

* grandpa-rpc: skip returning future when cancelling

* grandpa-rpc: reuse testing executor from sc-rpc

* grandpa-rpc: don't need to use PubSubHandler in tests

* node-rpc: use MetaIoHandler rather than PubSubHandler

* grandpa: log if getting header failed

* grandpa: move justification channel creation into factory function

* grandpa: make the justification sender optional

* grandpa: fix compilation warnings

* grandpa: move justification notification types to new file

* grandpa-rpc: move JustificationNotification to grandpa-rpc

* grandpa-rpc: move JustificationNotification to its own file

* grandpa: rename justification channel pairs

* grandpa: rename notifier types

* grandpa: pass justification as GrandpaJustification to the rpc module

* Move Metadata to sc-rpc-api

* grandpa-rpc: remove unsed error code

* grandpa: fix bug for checking if channel is closed before sendind

* grandpa-rpc: unit test for sending justifications

* grandpa-rpc: update comments for the pubsub test

* grandpa-rpc: update pubsub tests with more steps

* grandpa-rpc: fix pubsub test

* grandpa-rpc: minor indendation

* grandpa-rpc: decode instead of encode in test

* grandpa: fix review comments

* grandpa: remove unused serde dependency

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Jon Häggblad <jon.haggblad@gmail.com>
Co-authored-by: Tomasz Drwięga <tomasz@parity.io>
2020-08-10 10:31:36 +00:00
Bastian Köcher cde60b871e Revalidate transactions only on latest best block (#6824)
* Revalidate transactions only on latest best block

We should revalidate transactions only on the latest best block and not
on any arbitrary block. The revalidation before failed when there were
multiple blocks on the height given to the revalidation function, but no
block was imported as best block.

* Update test-utils/runtime/transaction-pool/src/lib.rs

Co-authored-by: Jaco Greeff <jacogr@gmail.com>

* Fix tests

* Only process best blocks in the transaction pool

Co-authored-by: Jaco Greeff <jacogr@gmail.com>
2020-08-07 13:58:51 +02:00
Cecile Tonglet 837c18ddfb Renamed add_children to add_child since it adds only one child (#6838) 2020-08-07 10:56:32 +02:00
André Silva cef250cbc6 grandpa: fix enacting forced changes with no delay (#6828)
* grandpa: fix enacting forced changes with no delay

* grandpa: fix formatting
2020-08-06 19:43:36 +00:00
Cecile Tonglet 4e03c87953 Allow task manager to have children (#6771)
* Initial commit

Forked at: 7df97abab4
Parent branch: origin/master

* WIP

Forked at: 7df97abab4
Parent branch: origin/master

* WIP

Forked at: 7df97abab4
Parent branch: origin/master

* WIP

Forked at: 7df97abab4
Parent branch: origin/master

* WIP

Forked at: 7df97abab4
Parent branch: origin/master

* WIP

Forked at: 7df97abab4
Parent branch: origin/master

* WIP

Forked at: 7df97abab4
Parent branch: origin/master

* changelog

* Remove Box

* Make future nicer

* Revert "Make future nicer"

This reverts commit 49fb8fb6f245c3ca2c384468df14b34f34616736.

* Simplify

* Additional check

* Simplify more

Co-authored-by: Bastian Köcher <git@kchr.de>
2020-08-06 21:20:46 +02:00
Pierre Krieger dbc66479f3 Delay network startup to after complete initialization (#6833)
* Delay network startup to after complete initialization

* Update client/service/src/builder.rs

Co-authored-by: Ashley <ashley.ruglys@gmail.com>

Co-authored-by: Ashley <ashley.ruglys@gmail.com>
2020-08-06 12:46:34 +00:00
André Silva 886d79e0cb grandpa: never overwrite current rounds voter state (#6823)
* grandpa: never overwrite current rounds voter state

* grandpa: add test for voter state overwrite
2020-08-06 11:30:29 +02:00
Pierre Krieger ee35dc9415 Fix legacy substream fallback not working (#6826)
* Fix legacy substream fallback not working

* Make it nicer
2020-08-05 16:20:09 +00:00
André Silva 4a94fe860d service: remove collection of system/process metrics (#6822) 2020-08-05 15:58:07 +00:00
Pierre Krieger 840781cddb Fix warning being printed by authority-discovery (#6820) 2020-08-05 09:16:19 +00:00
Pierre Krieger e449330e27 Add details to legacy requests (#6747) 2020-08-03 14:55:32 +00:00
Ashley 07facb13c2 Convert spaces to tabs (#6799) 2020-08-03 15:48:32 +02:00
Max Inden 0553dabe32 client/network: Fix wrong metric help text (#6794)
The `sub_libp2p_kademlia_query_duration` metric only has the dimension
`type` not `protocol`.
2020-08-03 09:08:06 +00:00
Max Inden 1365eef4c1 client/network: Expose DHT query duration to Prometheus (#6784)
Expose duration of DHT put and get request as a Prometheus histogram.
2020-08-03 08:30:06 +00:00
Pierre Krieger a1786a92ec Don't close inbound notifications substreams immediately (#6781)
* Don't close inbound notifications substreams immediately

* Fix not closing in return to node A closing
2020-07-31 14:42:53 +00:00
Wei Tang 3c5cbb00aa Allow blacklisting blocks from being finalized again after block revert (#6301)
* Allow blacklisting blocks from being finalized again after block revert

* Use BlockRules for storing unfinalized and add have_state_at in revert

* Move finalization_check in finalize_block upward

* Directly mark finalization blacklist as badblocks

* Remove obselete comment
2020-07-31 14:32:13 +02:00
Cecile Tonglet 3d42e2c7eb Fix graceful shutdown skipped if future ends with error (#6769)
* Initial commit

Forked at: 5060324b91
Parent branch: origin/master

* Fix graceful shutdown skipped if future ends with error

* apply suggestion
2020-07-30 15:59:07 +00:00
Wei Tang b6dedd9016 BABE slot and epoch event notifications (#6563)
* BabeWorker -> BabeSlotWorker

* SlotWorker::notify_slot: similar to claim_slot, but called no matter authoring

* Wrap the future with a new struct BabeWorker

* Add type definition slot_notification_sinks

* Function slot_notification_streams for the receiver side

* Get a handle of slot_notification_sinks in BabeSlotWorker

* Implement notify_slot

* Switch to use bounded mpsc

* Do not drop the sink when channel is full

Only skip sending the message and emit a warning, because it is recoverable.

* Fix future type bounds

* Add must_use and sink type alias
2020-07-30 17:07:27 +02:00
Bastian Köcher 9ac30f7116 Rename task name to stick to the default naming scheme (#6768) 2020-07-30 14:19:34 +00:00
Garrett MacDonald 814911f414 Add " Successfully mined block" log message (#6764) 2020-07-30 13:23:25 +00:00
Pierre Krieger 5060324b91 Ignore flaky test (#6767) 2020-07-30 14:58:30 +02:00
Ashley 7df97abab4 Add a DefaultQueue type alias to remove the need to use sp_api::TransactionFor (#6761)
* Add DefaultQueue

* Add DefaultImportQueue to the top level of sp-consensus
2020-07-30 09:02:12 +00:00
Max Inden dc33e64643 client/network: Add peers to DHT only if protocols match (#6549)
* client/network/src/discovery: Adjust to Kademlia  API changes

* client/network: Add peers to DHT only if protocols match

With https://github.com/libp2p/rust-libp2p/pull/1628 rust-libp2p allows
manually controlling which peers are inserted into the routing table.
Instead of adding each peer to the routing table automatically, insert
them only if they support the local nodes protocol id (e.g. `dot`)
retrieved via the `identify` behaviour.

For now this works around
https://github.com/libp2p/rust-libp2p/issues/1611. In the future one
might add more requirements. For example one might try to exclude
light-clients.

* Cargo.toml: Remove crates.io patch for libp2p

* client/network/src/behaviour: Adjust to PeerInfo name change

* client/network/src/discovery: Rework Kademlia event matching

* client/network/discovery: Add trace on adding peer to DHT

* client/network/discovery: Retrieve protocol name from kad behaviour

* client/network/discovery: Fix formatting

* client/network: Change DiscoveryBehaviour::add_self_reported signature

* client/network: Document manual insertion strategy

* client/network/discovery: Remove TODO for ignoring DHT address

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-07-29 14:44:15 +02:00
Gavin Wood 6bfbb7c6f1 Cleanup our sort usage (#6754) 2020-07-29 14:00:51 +02:00
Pierre Krieger 1ab7719314 Add a back-pressure-friendly alternative to NetworkService::write_notifications 🎉 (#6692)
* Add NetworkService::send_notifications

* Doc

* Doc

* API adjustment

* Address concerns

* Make it compile

* Start implementation

* Progress in the implementation

* Change implementation strategy again

* More work before weekend

* Finish changes

* Minor doc fix

* Revert some minor changes

* Apply suggestions from code review

* GroupError -> NotifsHandlerError

* Apply suggestions from code review

Co-authored-by: Roman Borschel <romanb@users.noreply.github.com>

* state_transition_waker -> close_waker

* Apply suggestions from code review

Co-authored-by: Roman Borschel <romanb@users.noreply.github.com>

* Finish renames in service.rs

* More renames

* More review suggestsions applied

* More review addressing

* Final change

* 512 -> 2048

Co-authored-by: Roman Borschel <romanb@users.noreply.github.com>
2020-07-29 11:23:19 +00:00
Max Inden e674d64a72 client/network: Adjust wording (#6755)
Rename `NetworkWorker::from_worker` to `NetworkWorker::from_service` as
it is a channel from the `NetworkService` to the `NetworkWorker`.
2020-07-29 11:18:40 +00:00
Bastian Köcher e7d8040af8 Update parity-scale-codec to prepare for breaking rustc release (#6746)
This updates parity-scale-codec{-derive} to prepare for a rustc release
that would otherwise break the derive implementation:
https://github.com/rust-lang/rust/pull/73084
2020-07-28 20:26:01 +00:00
Ashley 9220b646d2 Various small improvements to service construction. (#6738)
* Remove service components and add build_network, build_offchain_workers etc

* Improve transaction pool api

* Remove commented out line

* Add PartialComponents

* Add BuildNetworkParams, documentation

* Remove unused imports in tests

* Apply suggestions from code review

Co-authored-by: Nikolay Volf <nikvolf@gmail.com>

* Remove unused imports in node-bench

Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
2020-07-28 15:21:33 +02:00
Pierre Krieger 5af85552af Remove Unpin requirement for Slots (#6711) 2020-07-27 14:31:35 +00:00
Bastian Köcher 9310f15ac2 Name all the tasks! (#6726)
* Remove any implementation of `Spawn` or `Executor` from our task executors

* Fix compilation

* Rename `SpawnBlockingExecutor`

* Update primitives/core/src/traits.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Fix tests

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2020-07-26 12:56:17 +00:00
André Silva 2ec131142b grandpa: allow noting that the set has stalled (#6725)
* grandpa: remove unused methods to convert digest

* grandpa: add root extrinsic for scheduling forced change

* grandpa: add benchmark for schedule_forced_change

* grandpa: don't take authority weight in schedule_forced_change

* grandpa: add const for default forced change delay

* grandpa: adjust weights after benchmark on ref hardware

* grandpa: fix cleanup of forced changes on standard change application

* grandpa: replace schedule_forced_change with note_stalled

* grandpa: always trigger a session change when the set is stalled

* grandpa: fix bug on set id mutation after failed scheduled change

* grandpa: take delay as parameter in note_stalled

* grandpa: fix tests

* grandpa: fix cleanup of forced changes

* grandpa: add test for forced changes cleanup

* grandpa: add test for session rotation set id

* grandpa: add test for scheduling of forced changes on new session
2020-07-24 21:02:12 +00:00
Benjamin Kampmann d9df1716da Switching from git back to released versions for wasmtime, fix cargo-unleash (#6722)
* Switching from git back to released versions for wasmtime

* filter out cratelift_codegen messages-a

Co-authored-by: NikVolf <nikvolf@gmail.com>
2020-07-24 14:45:20 +00:00
Wei Tang ac8de557d6 Better handling of stable-only build (#6569)
* Better handling of stable-only build

* Fix node template build

* Fix wasm builder node-template version mismatch

* Fix load_spec error

* Add , in parameter

* Add descrptive panic messages in tests

* Add descriptive tests in node/executor benches

* Fix missing compact_code_unwrap

* Add missing wasm_binary_unwrap function for executor integration test

* Only define import_sp_io in no_std

* Small Cargo.toml styling fix

* Bump wasm-builder to 2.0.0

* Fix all `with_wasm_builder_from_crates` version in Substrate

* Use `with_wasm_builder_from_crates` for node-template

Co-authored-by: Gavin Wood <gavin@parity.io>
2020-07-22 21:40:11 +02:00
Ashley ffa2177d46 Remove the service builder (#6557)
* :)

* Slight tidy

* Remove ServiceBuilderCommand

* Remove whitespace

* Keep task manager alive for check_block/import_blocks

* Pass task_manager to run_until_exit

* WIP

* WIP

* Get rid of the macros

* Simplify a few chain components creation APIs related to the service

* Fix basic-authorship doc tests

* Remove DefaultQueue

* Update client/service/src/builder.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Move ExecutionExtensions comment around

* Remove unused BlakeTwo256

* Add sc-prelude

* Rename sc-prelude to sc-service-prelude

* Rename to sc-service-types

* Improve service types

* Fix line widths

* Remove sc-service-types and move type definitions to crates

* Update bin/node-template/node/src/service.rs

Co-authored-by: Seun Lanlege <seunlanlege@gmail.com>

* Add TLightClientWithHash

* Rework types

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Seun Lanlege <seunlanlege@gmail.com>
2020-07-22 19:10:27 +00:00
Nikolay Volf 85a9af64d8 Update wasmtime to (almost) lastest master (#6662)
* update wasmtime to (almost) lastest master

* bump lock

* bump once more
2020-07-22 21:09:31 +02:00
Benjamin Kampmann 7c161ec177 Update to latest sysinfo prevents leaking fd-handlers (#6708)
* Ensure we close old file handles hold by sysinfo

* Dropping is needed unfortunately

* enable process refreshing, ignore result from refresh_process

* jumping to proposed patch

* switch to latest sysinfo
2020-07-22 15:48:12 +00:00
mattrutherford c4dd079119 Tracing values (#6679)
* Switch to serde_json::Value for Visitor values

* add tests

* switch Visitor to use Map

* refactor change names

* \n

* update integration test

* use discrete maps for each type of recorded value

* update integration test

* add docs

* Update client/tracing/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* add docs

Co-authored-by: Matt Rutherford <mattrutherford@users.noreply.github.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2020-07-22 11:29:33 +00:00
Cecile Tonglet de3181076a Graceful shutdown for the task manager (#6654)
* Initial commit

Forked at: 74655edebc
Parent branch: origin/master

* Move task_manager.rs to mod.rs

* Graceful shutdown for the task manager

* Await all background task JoinHandle at the same time

* Add tests

* Make future() wait also for exit signal + fix essential task failed

Probably related to https://github.com/paritytech/cumulus/issues/111

* add comments for non-obvious code

* Use clean_shutdown() in sc-cli

* Adapt code and upgrade tokio in sc-cli

* cleanup spacing in doc

* Add license

* I guess actually running the clean shutdown would be a good idea

* fix tests

* Update client/cli/src/runner.rs

Co-authored-by: Benjamin Kampmann <ben@gnunicorn.org>

* Improve error logging

* disable other tests (can't reproduce on my machine)

* Revert "disable other tests (can't reproduce on my machine)"

This reverts commit c133c590f33c253123ba0555ce719a71ededd60d.

* It is possible that the tasks are ended first

* Revert "It is possible that the tasks are ended first"

This reverts commit 502aba4a49fb5d892e704c412b8a81768a3f2c71.

* Use single threaded scheduler for more predictability

* enable_time

* Revert "enable_time"

This reverts commit 4e152140764a4bddeedff06a4e36ec701909e8c7.

* Revert "Use single threaded scheduler for more predictability"

This reverts commit ee5e13c5f13ff71e012dcda13579dffeb15f8ffc.

* Revert "Revert "It is possible that the tasks are ended first""

This reverts commit 1b91a8ca3eebbdc18be199c8ca188e88669ae649.

* This cannot be verified either with a threaded pool

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Benjamin Kampmann <ben@parity.io>
Co-authored-by: Benjamin Kampmann <ben@gnunicorn.org>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-07-22 07:15:22 +00:00
pscott 046fda914a Improve overall performance (#6699)
* Improve overall performance

* Clean up code

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Remove needless ::

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Remove needless ::

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2020-07-21 14:46:49 +02:00
Pierre Krieger ab82eb1c98 Add sync_legacy_requests_received metric (#6698) 2020-07-21 11:59:49 +00:00
Benjamin Kampmann 629a801b6a Fixup cargo unleash (#6689)
* Bumping to rc5

* Revert "Bumping to rc5"

This reverts commit 5a63a8230e5a12db1b4ce4ac62e4967c52291da7.

* Bumping for rc5, correctly this time
2020-07-21 10:26:47 +00:00
Roman Borschel 025c501766 Update libp2p to 0.22. (#6690)
* Update libp2p to 0.22.

Primarily to begin the libp2p-noise upgrade process.

See https://github.com/libp2p/rust-libp2p/issues/1631.

* CI
2020-07-21 07:44:41 +00:00
Amar Singh 0d2a794d48 done (#6673) 2020-07-19 16:12:04 +02:00
André Silva ae38a806ed grandpa: report equivocations with unsigned extrinsics (#6656)
* grandpa: use unsigned extrinsics for equivocation reporting

* grandpa: allow signed reports as well

* grandpa: change runtime api for submitting unsigned extrinsics

* grandpa: fix tests

* grandpa: add test for unsigned validation

* grandpa: add benchmark for equivocation proof checking

* offences: fix grandpa benchmark

* grandpa: add proper weight for equivocation reporting extrinsic

* grandpa: fix weight unit
2020-07-17 12:32:20 +02:00
Wei Tang 8ae1aa4c28 Switch to bounded mpsc for txpool import notification stream (#6640)
* Switch to bounded mpsc for txpool import notification stream

* Update client/transaction-pool/graph/src/validated_pool.rs

Co-authored-by: Nikolay Volf <nikvolf@gmail.com>

Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
2020-07-17 12:31:47 +02:00