Commit Graph

398 Commits

Author SHA1 Message Date
Arkadiy Paronyan d146cc3ae3 Don't format when telemetry is disabled (#5273) 2020-03-17 12:22:05 +01:00
Max Inden e91d4be998 client/network/src/protocol: Refactor Prometheus metric logic (#5266)
* client/network/src/protocol: Start Prometheus metric help with capital

* client/network/src/protocol: Differentiate metric status as label

Prometheus query language is powerful through its multi-dimensional data
model. Metric names are hirarchical whereas labels enable data to become
multi-dimensional.

Exposing the justification of finality-proof status as a label allows
for more powerful queries.

* client/network/src/protocol: Remove 'Total' from non counter metric help

The word 'total' is reserved for accumulating counters. Counters have to
be monotonically increasing. `obsolete_requests` can decrease, thereby
it is defined as a `Gauge` and not a `Counter`.

For more details on metric naming see
https://prometheus.io/docs/practices/naming/
2020-03-17 11:47:55 +01:00
Wei Tang 846a9ce8c6 epoch-changes: separate epoch header with epoch data (#4881)
* fork-tree: prune returns all pruned node data

* epoch-changes: split EpochHeader vs epoch data

* EpochChanges::viable_epoch and add missing comments

* Incoperate the new epoch_changes interface for BABE

* Fix BABE tests

* Fix fork-tree pruning issue

* Fix tests

* Fix pruning algorithm

* fork-tree: implement map function for mapping one value type to another

* Add migration script for new epoch changes scheme

* Update utils/fork-tree/src/lib.rs

Co-Authored-By: André Silva <andre.beat@gmail.com>

* Update client/consensus/slots/src/lib.rs

Co-Authored-By: André Silva <andre.beat@gmail.com>

* Remove authorities_len.is_none check, which is duplicate of unwrap_or(false)

* Update client/consensus/epochs/src/lib.rs

Co-Authored-By: André Silva <andre.beat@gmail.com>

* Update client/consensus/epochs/src/lib.rs

Co-Authored-By: André Silva <andre.beat@gmail.com>

* No trailing ; for return statement

* Use VERSION_KEY for migration

* Fix issues that removed nodes are not properly added into removed list

* Add comments indicating end_slot is non-inclusive

* fork-tree: use &mut F for map type declaration

* Add tests for v0 epoch_changes migration

* Fix babe RPC tests

Co-authored-by: André Silva <andre.beat@gmail.com>
2020-03-16 17:06:13 -04:00
Nikolay Volf 372745705d create parallel tasks extension (#5249) 2020-03-16 16:30:39 +01:00
Stanislav Tkach 5e0b6f2f29 Update parity-common dependencies (#5254) 2020-03-16 09:36:04 +01:00
Bastian Köcher 26fa6e3c76 Move runtime upgrade to frame-executive (#5197)
* Move runtime upgrade to `frame-executive`

Instead of storing the runtime upgraded in the space of `frame-system`,
this moves it to `frame-executive`. We also start storing the
`spec_version` and `impl_version` the last runtime upgrade was applied.

This scheme has multiple advantages:
- We don't need to make sure that runtime upgrade was set.
- This will work in the future when we want to test a runtime upgrade,
because the `on_runtime_upgrade` will be executed automatically when the
runtime changes.

* Move `LAST_RUNTIME_UPGRADE` key to `frame-executive`

* Add some more documentation

* Make sure `spec_version` always increases between runtime upgrades

* Fix test

* Upgrade `parity-multiaddr` to fix compilation

* Also check if the `spec_name` changed

* Remove `RuntimeUpgraded` storage entry

* Don't run on genesis
2020-03-15 20:28:57 +01:00
Shawn Tabrizi b57fd17998 Fix genesis bug in bench db (#5253) 2020-03-15 20:19:01 +01:00
Max Inden 79fc16e4d7 client/finality-grandpa: Add Prometheus metrics to GossipValidator (#5237)
* client/finality-grandpa: Add Prometheus metrics to GossipValidator

Instrument finality grandpa `GossipValidator` exposing count of messages
validated by message type and message action.

```
\# HELP substrate_finality_grandpa_communication_gossip_validator_messages Number of messages validated by the finality grandpa gossip validator.
\# TYPE substrate_finality_grandpa_communication_gossip_validator_messages counter
substrate_finality_grandpa_communication_gossip_validator_messages{action="discard",message="neighbor"} 39
substrate_finality_grandpa_communication_gossip_validator_messages{action="keep",message="vote"} 28
```

* client/finality-grandpa: Add None as Prometheus registry in tests

* client/finality-granpda/src/communication: Refactor metric registration
2020-03-14 12:51:44 +01:00
Kian Paimani 7e2cd0edee Remove unused imports and extern crate (#5245) 2020-03-14 12:48:47 +01:00
Seun Lanlege 3193b0d4aa removes use of sc_client::Client from node-transaction-factory (#5158)
* removes use of sc_client::Client from node-transaction-factory

* move sc-block-builder to [dev-dependencies] in substrate-test-runtime
2020-03-12 19:19:38 +01:00
Seun Lanlege 9898fc7e81 removes use of sc_client::Client from sc_network (#5147)
* removes use of sc_client::Client from sc_network

* rename BlockProvider to BlockBackend

* fix broken test
2020-03-12 14:02:04 +01:00
Stanislav Tkach 6e1b4cd560 Update sysinfo version to 0.11 (#5216) 2020-03-12 14:00:07 +01:00
Arkadiy Paronyan dc0bf210fb ChainSpec trait (#5185)
* ChainSpec trait

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Added docs

* Fixed build

* Fixed build

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-03-12 00:00:10 +01:00
Max Inden d2345e8d5c client/authority-discovery: Instrument code with Prometheus (#5195)
* client/authority-discovery: Instrument code with Prometheus

Introduce Prometheus metrics into the authority discovery module
enabling one to observe:

- authority_discovery_times_published_total

- authority_discovery_amount_external_addresses_last_published

- authority_discovery_times_requested_total

- authority_discovery_dht_event_received

* client/authority-discovery/src/lib.rs: Reword metric help texts

Co-Authored-By: Ashley <ashley.ruglys@gmail.com>

* client/authority-discovery/src/lib.rs: Reword metric help text

Co-Authored-By: Ashley <ashley.ruglys@gmail.com>

* client/authority-discovery/tests: Fix struct instantiation + basic test

Co-authored-by: Ashley <ashley.ruglys@gmail.com>
2020-03-11 23:59:33 +01:00
Stanislav Tkach 93ee3104e7 Use CLI to configure max instances cache (#5177)
* Use CLI to configure max instances cache

* Fix tests

* Move default value into CLI

* Use SmallVec

* Apply review comments

* Get rid of `SmallVec`

Co-authored-by: Bastian Köcher <git@kchr.de>
2020-03-11 09:43:37 +01:00
Pierre Krieger f21680209f Minor fixes for Android compilation (#5209)
* Minor fixes for Android

* Update parity-multiaddr

* Bump in service Cargo.toml too
2020-03-10 20:54:58 +01:00
Bastian Köcher 1cfcf5cbfe Don't include :code by default in storage proofs (#5179)
* Don't include `:code` by default in storage proofs (#5060)

* Adds test to verify that the runtime currently is always contained in
the proof

* Start passing the runtime wasm code from the outside

* Fix compilation

* More build fixes

* Make the test work as expected now :)

* Last fixes

* Fixes benchmarks

* Review feedback

* Apply suggestions from code review

Co-Authored-By: Sergei Pepyakin <sergei@parity.io>

* Review feedback

* Fix compilation

Co-authored-by: Sergei Pepyakin <s.pepyakin@gmail.com>

* Fix compilation and change the way `RuntimeCode` works

* Fix tests

* Switch to `Cow`

Co-authored-by: Benjamin Kampmann <ben@gnunicorn.org>
Co-authored-by: Sergei Pepyakin <s.pepyakin@gmail.com>
2020-03-10 11:13:20 +01:00
Max Inden 0cc3b96076 client/service/src/builder.rs: Add build_info metric (#5192)
* client/service/src/builder.rs: Add build_info metric

Add static Prometheus metric exposing the chain name, the version and
the commit.

* client/service/src/builder.rs: Move node_role to static metrics

The Prometheus metrics `node_role` is static and thus there is no need
to keep a reference of it within `ServiceMetrics`. This follows the
example of the `build_info` metric.

* client/service/src/builder.rs: Adjust indentation
2020-03-10 11:12:32 +01:00
Igor Matuszewski 3d73dd1c3b Make sc-offchain tests more resilient on macOS (#5191)
* offchain: Simplify bits of http code

* offchain: Sort dev-dependencies

* deps: Bump parity-multiaddr to 0.7.3

Fixes build failure when using:
rustc 1.43.0-nightly (96bb8b31c 2020-03-05).

* offchain: Raise FD limit for HTTP tests

* offchain: Reword the comment on increasing the test fd limit
2020-03-10 10:45:10 +01:00
André Silva cfd6824e57 grandpa: improve handling of global gossip messages (#5133)
* grandpa: only gossip commits to peers on the same set

* grandpa: track commits uniquely by round and set

* grandpa: fix communication test

* grandpa: add tests for commit gossip handling

* grandpa: add missing docs
2020-03-09 16:46:03 -07:00
Max Inden e8f3c6a686 client/network-gossip: Merge GossipEngine and GossipEngineInner (#5042)
* client/network-gossip: Merge GossipEngine and GossipEngineInner

Given that GossipEngine and GossipEngineInner are not shared between
threads anyone (public interface + background tasks), neither depends on
being Send or Sync. Thus one can merge the two as done in this patch.
One only needs to wrap an `Arc<Mutex<>>` around the whole structure when
the owner (e.g. finality-grandpa) needs to share the gossip engine
between threads.

* client/finality-grandpa: Wrap GossipEngine in Arc Mutex & lock it on use

GossipEngine in itself has no need to be Send and Sync, given that it
does not rely on separately spawned background tasks anymore. Given that
finality-grandpa shares the `NetworkBridge` potentially between threads
its components need to be clonable, thus this patch wraps `GossipEngine`
in an `Arc<Mutex<>>`.
2020-03-09 16:31:29 +01:00
Pierre Krieger 9c6d53f436 Fix NotificationStreamClosed reported when it shouldn't (#5160)
* Fix NotificationStreamClosed reported when it shouldn't

* Fix test

* Add test

* Update client/network/src/protocol.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Oops, fix test

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-03-06 21:24:51 +01:00
Nikolay Volf 4aac39c667 Fix possible corner case in revalidation (#5172)
* remove all from members

* add test
2020-03-06 21:24:39 +01:00
Ashley 7177d2fb2b Fix browser node (#5166) 2020-03-06 18:34:53 +01:00
André Silva 9eaa972c21 client: fix notification sinks leak during initial sync (#5161)
* client: fix notification sinks leak during initial sync

* client: add test for notification sink cleanup

* Make it compile

* Further cleanup

* client: fix test for notification sinks cleanup

Co-authored-by: Bastian Köcher <git@kchr.de>
2020-03-06 18:32:50 +01:00
Nikolay Volf 5e1667c406 Add state/database caches to prometheus (#5165)
* add state/database caches to prometheus

* also state-db memory

* use suggestions from review
2020-03-06 18:32:33 +01:00
André Silva 3a792a55cd client: don't log on 0 gossip messages (#5173) 2020-03-06 18:31:35 +01:00
Bastian Köcher c244b1d036 Revert "Build block without checking signatures (#4916)" (#5159)
* Revert "Build block without checking signatures (#4916)"

This reverts commit e50f610907.

* Some further clean ups
2020-03-06 15:27:59 +01:00
Gavin Wood bbaf96e047 Revert quote fixed version and upgrade failure (#5152)
* Revert "use fixed quote (#5135)"

This reverts commit 8bf855b09d.

* Upgrade failure version

* Update frame/staking/reward-curve/Cargo.toml

Co-Authored-By: André Silva <andre.beat@gmail.com>

* Ahh I'm dumb

Co-authored-by: Gavin Wood <github@gavwood.com>
Co-authored-by: André Silva <andre.beat@gmail.com>
2020-03-05 22:14:55 +01:00
Gavin Wood d957220435 Add lots of networking metrics for Prometheus (#5126)
* Add some metrics

* Address concerns
2020-03-05 21:38:41 +01:00
Gavin Wood a8441ff968 Remove Backend::destroy_state (#5068)
* Remove `Backend::destroy_state`

This removes the `destroy_state` function of `Backend` and instead moves
the functionality into the `Drop` implementation of the state. This
makes it much easier to work with the state, as the user no longer needs
to call` destroy_state` manually. However, it requires that we switch
from `RwLock` to `ReentrantMutex` as while importing a block we maybe
need to lock again in `drop`.

* Bring back the `RwLock` and some other clean ups

* Fix compilation
2020-03-05 17:01:52 +01:00
Gavin Wood 26fc97f635 Remove substrate-ui.parity.io from CORS whitelist (#5142)
The only up-to-date deployment of PolkadotJS apps is currently on https://polkadot.js.org/apps/.
https://substrate-ui.parity.io is only useful as deliberately kept outdated version of Apps, to be used for projects still on Substrate 1.0
It cannot be used with the up-to-date chains running the Substrate 2.0 codebase, and probably won't be maintained much longer.
2020-03-05 17:01:36 +01:00
Gavin Wood 868d8b8252 Add more prometheus metrics to network::Protocol. (#5145) 2020-03-05 17:01:24 +01:00
Gavin Wood f5dc69b404 Don't remove invalid transactions when skipping. (#5121)
* Don't remove invalid transactions when skipping.

* Use a special kind of extrinsic instead of arbitrary limit.

* Fix txpool tests.

* Attempt to create more blocks.

* Bump lock

Co-authored-by: Gavin Wood <github@gavwood.com>
Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
2020-03-05 17:00:31 +01:00
Benjamin Kampmann 99ae5342eb removes use of sc_client::Client from sc-rpc (#5063)
* removes use of sc_client::Client from sc-rpc

* remove Client impl from sc-finality-benches

* remove client impl from sc-finality-grandpa

* read_proof accepts iterator

* remove generic Executor param from ExecutorProvider

* fix long ass line

* code style changes

* merge with master

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
2020-03-05 16:41:10 +01:00
Benjamin Kampmann d3208aa7bc Better wasm instance cache (#5109)
* Wasm instance cache

* Reduce slot locking

* Fixed test

* Dispose of instance in case of error

* Fixed benches

* Style, comments, some renames

* Replaced upgradable lock with mutex

* Bump dependencies

* Re-export CallInWasm

* Update client/executor/src/wasm_runtime.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/executor/src/native_executor.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/executor/src/native_executor.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/executor/src/wasm_runtime.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/executor/wasmtime/src/runtime.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/executor/src/wasm_runtime.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/executor/src/wasm_runtime.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/executor/src/wasm_runtime.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Indents

* Whitespace

* Formatting

* Added issue link

Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com>
Co-authored-by: Gavin Wood <github@gavwood.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-03-05 14:02:04 +01:00
Gavin Wood 906e08ee2a Pass Prometheus Registry into Client (#5120)
* Add a few metrics to Client

* Improve PrometheusConfig

* Fix client docs
2020-03-05 13:40:33 +01:00
Gavin Wood ab58a08a53 remove on_exit from grandpa (#5098) 2020-03-05 13:38:16 +01:00
Benjamin Kampmann 0f2606ccaf Use 128mb for db cache default (#5134) 2020-03-05 11:25:28 +01:00
Arkadiy Paronyan 9a1b3b53f2 Revert "Don't include :code by default in storage proofs (#5060)" (#5136)
This reverts commit 6ee39261c8.
2020-03-05 10:55:05 +01:00
Gavin Wood 8bf855b09d use fixed quote (#5135) 2020-03-05 10:29:38 +01:00
Gavin Wood 5bda7c5072 Task manager for background/async tasks in service (#5092)
* Reorganize tasks into task manager

* move to separate file and improve api

* address api issues

* fix spawning inside closures

* decouple executor

* tasks_setup -> tasks_builder

* remove drops

* add deprecatiion comment

* add pub(super)

* fix identation
2020-03-05 09:29:12 +01:00
Benjamin Kampmann 5a33228ea9 Reduce usage of Blake2Hasher (#5132)
This reduces the usage of `Blake2Hasher` in the code base and replaces
it with `BlakeTwo256`. The most important change is the removal of the
custom extern function for `Blake2Hasher`. The runtime `Hash` trait is
now also simplified and directly requires that the implementing type
implements `Hashable`.
2020-03-05 08:51:03 +01:00
Benjamin Kampmann 6ee39261c8 Don't include :code by default in storage proofs (#5060)
* Adds test to verify that the runtime currently is always contained in
the proof

* Start passing the runtime wasm code from the outside

* Fix compilation

* More build fixes

* Make the test work as expected now :)

* Last fixes

* Fixes benchmarks

* Review feedback

* Apply suggestions from code review

Co-Authored-By: Sergei Pepyakin <sergei@parity.io>

* Review feedback

* Fix compilation

Co-authored-by: Sergei Pepyakin <s.pepyakin@gmail.com>
2020-03-04 20:26:16 +01:00
Benjamin Kampmann 67837c6233 Forget peerset manager nodes after 5mn (#5108)
* Forget peerstore nodes after 5mn

* Bump to one hour

* Also bump on rediscover
2020-03-04 20:24:17 +01:00
Benjamin Kampmann 6cf944cf21 Drop db-cache default from 1gig to 32mb (#5128) 2020-03-04 17:22:02 +01:00
Shawn Tabrizi 3d6329a22d A Pallet for Benchmarking Common Runtime Operations (#4902)
* Benchmark pallet

* Add a bunch more benchmarks

* do nothing test

* new benchmarks

* Clean up extra tests

* Encode and Decode Vec<T::AccountId>

* Starting to migrate benchmarks to macro

* Use macro

* Remove call and storage

* Update Cargo.toml

* Add storage recalc benchmark

* Add support for custom functions in benchmark! macro

* Reset DB for storage recalc

* Feedback from review

* Add more comments

* Remove benchmark pallet from node

* Fix cargo files

* Fix comments

* Change `crate` to `super`

* missed one

* Use results of benchmark encode/decode

* Pass generic to extra functions

* reset macro to master

* Update lib.rs

* Update to use standard syntax
2020-03-04 17:21:42 +01:00
Jakub Pánik e366b5bd2c Added subscribe_all_heads RPC function (#4979)
* Added subscribe_all_heads RPC function

* Update client/rpc/src/chain/tests.rs

Fixed style ( spacing )

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
2020-03-04 14:19:38 +01:00
Bastian Köcher a1b53280b4 Update to SCALE 1.2.0 (#5113)
This updates `parity-scale-codec` to `1.2.0`, which includes multiple
performance improvements and a fix that bounds the capacity of a vector
at decoding.
2020-03-03 14:40:02 +01:00
Ashley 883ddfc897 Add more metrics to prometheus (#5034)
* Add a few things

* Add finality_grandpa_round

* fix fg tests

* Nitpicks

* Nitpicks

* Fix name of prometheus crate
2020-03-03 11:36:58 +01:00