Commit Graph

294 Commits

Author SHA1 Message Date
Bastian Köcher 9277a53052 Move tracing cli args to ImportParams (#4850) 2020-02-07 13:21:46 +01:00
Nikolay Volf 41967186e4 Subsystems memory tracking: 1. Transaction pool (#4822)
* update sp-runtime

* total update

* usage informant

* update to crates.io version

* update Cargo.lock

* update dummy update

* fix todo

* cleanup

* avoid custom impl

* Update client/transaction-pool/graph/src/future.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* remove another custom impl

* remove another custom impl

* add kb in report

* update Cargo.lock

* review suggestions

* --amend

* --amend

* bump parity-util-mem to 0.5.0

* bumps

* update macro and versions

* add to grafana

* naming

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
2020-02-07 11:53:11 +01:00
Marcio Diaz 41644c2ae8 Enable trace timings logs for transaction factory (#4845)
* Enable trace timings logs for transaction factory.
2020-02-07 08:12:19 +01:00
Cecile Tonglet be075893b5 CLI improvements & fixes (#4812)
These are a few changes I missed during the refactoring.

1. Initialization issue and boilerplate

    Most importantly: part of the `Configuration` initialization was done in `sc_cli::init`. This means the user can not benefit from this initialization boilerplate if they have multiple `Configuration` since `sc_cli::init` can only be called once.

2. Boilerplate for `VersionInfo` and `Configuration`

    I'm also answering to the critic of @bkchr on the initialization using version: https://github.com/paritytech/substrate/pull/4692/files/bea809d4c14a2ede953227ac885e3b3f9771c548#r372047238 This will allow initializing a `Configuration` and provide the version by default.

3. Loading the `chain_spec` explicitly

    In the past it was done automatically but in some cases we want to delay this. I moved the code to `Configuration.load_spec()` so it can be called later on. `chain_spec` can also be written directly to the `Configuration` without using this `load_spec` helper.

4. [deleted]

5. Fixing issue that prevents the user to override the port

    In the refactoring I introduced a bug by mistake that could potentially prevent the CLI user to override the ports if defaults where provided for these ports (only on cumulus).

6. Change task_executor from Box to Arc

    This is useful for cumulus where we have 2 nodes with 2 separate Configuration that need to spawn tasks to the same runtime.

7. Renamed TasksExecutorRequired to TaskExecutor

    For consistency.

This is related to https://github.com/paritytech/cumulus/issues/24

This is the continuation (and hopefully the end of) #4692
2020-02-06 15:46:49 +01:00
Marcio Diaz f891342f20 Fix broken factory by adding keystore back. (#4840) 2020-02-06 14:48:19 +01:00
Arkadiy Paronyan 3c79e6d03c Fixed a few warnings (#4841) 2020-02-06 14:06:53 +01:00
Nikolay Volf 63ad759c69 Increase transaction pool default limits (#4792)
* increase limits, improve logging

* format in kB
2020-02-01 00:27:53 -08:00
thiolliere a36a170c15 remove ambiguous into_iter (#4794) 2020-01-31 18:49:02 +01:00
Cecile Tonglet 605f643eed CLI API refactoring and improvement (#4692)
It changes the way we extended the CLI functionalities of substrate to allow more flexibility. (If this was not clear, here is another version: it changes the `sc_cli` API to allow more flexibility).

This touches a few important things:
 - the startup of the async task with tokei:
    This was in node and node-template and I moved it to substrate. The idea is to have 1 time the code that handles unix signals (SIGTERM and SIGINT) properly. It is however possible to make this more generic to wait for a future instead and provide only a helper for the basic handling of SIGTERM and SIGINT.
 - increased the version of structopt and tokei
 - no more use of structopt internal's API
 - less use of generics

Related to #4643 and https://github.com/paritytech/cumulus/pull/42: the implementation of "into_configuration" and "get_config" are similar but with better flexibility so it is now possible in cumulus to have the command-line arguments only of the run command for polkadot if we want

Related to https://github.com/paritytech/cumulus/issues/24 and https://github.com/paritytech/cumulus/issues/34 : it will now be possible to make a configuration struct for polkadot with some overrides of the default parameters much more easily.
2020-01-30 11:40:08 +01:00
Bastian Köcher 34c1c4b954 Don't run dev light client as validator (#4757)
A light client can not be a validator, so we should not set `authority
== true` in dev mode.
2020-01-29 11:52:54 +03:00
Bastian Köcher 85aa632278 Make debug builds more usable (#4683)
* Make debug builds more usable

This pr makes debug builds more usable in terms of `cargo run -- --dev`.

1. `--dev` activates `--execution native`, iff `--execution` is not
given or no sub `--execution-*` is given.
2. It was probably a mistake to compile WASM in debug for a debug build.
So, we now build the WASM binary always as `release` (if not requested
differently by the user). So, we trade compilation time for a better
debug experience.

* Make sure we only overwrite default values

* Make it work

* Apply suggestion
2020-01-21 12:33:28 +01:00
Cecile Tonglet 60feeb7ad6 Getting configuration from commands (#4643)
* Expose a method that allows converting RunCmd to Configuration

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP
2020-01-16 13:57:19 +01:00
Nikolay Volf 058e98ef34 use default in-memory for commands that don't use keystore (#4634) 2020-01-16 10:08:05 +01:00
Ashley 3219be2508 Update the service to std futures (#4447)
* Switch service to futures03

* Fix tests

* Fix service test and cli

* Re-add Executor trait to SpawnTaskHandle

* Fix node-service

* Update babe

* Fix browser node

* Update aura

* Revert back to tokio-executor to fix runtime panic

* Add todo item

* Fix service tests again

* Timeout test futures

* Fix tests

* nits

* Fix service test

* Remove zstd patch

* Re-add futures01 to aura and babe tests as a dev-dep

* Change failing test to tee

* Fix node

* Upgrade tokio

* fix society

* Start switching grandpa to stable futures

* Revert "Start switching grandpa to stable futures"

This reverts commit 9c1976346237637effc07c13f7d0403daf5e71cf.

* Fix utils

* Revert substrate service test

* Revert gitlab

Co-authored-by: thiolliere <gui.thiolliere@gmail.com>
2020-01-14 15:43:45 +01:00
Bastian Köcher e8dd1205ee Add test for cli keystore path generation (#4571)
* Add test for cli keystore path generation

* Fix test
2020-01-11 18:47:26 +01:00
Bastian Köcher 189bd2349c Keystore fix default path (#4570) 2020-01-08 15:19:14 +01:00
Nikolay Volf df4058b556 i/o stats for backend databases (#4525) 2020-01-07 21:53:03 +03:00
Ashley bb44f8fc24 Add a browser-utils crate (#4394)
* Squash

* Fix keystore on wasm

* Update utils/browser/Cargo.toml

Co-Authored-By: Benjamin Kampmann <ben@gnunicorn.org>

* export console functions

* Use an Option<PathBuf> in keystore instead of cfg flags

* Add a KeystoreConfig

* Update libp2p

* Bump kvdb-web version

* Fix cli

* Upgrade versions

* Update wasm-bindgen stuff

Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com>
2020-01-07 16:30:04 +01:00
Arkadiy Paronyan bc9676737f Expose load_spec (#4556) 2020-01-07 15:08:27 +01:00
Shawn Tabrizi 7efa62725e Update copyright year (#4532) 2020-01-05 14:05:44 +01:00
Nikolay Volf 6d06a19f41 Refactor to use only chain info (#4516) 2020-01-02 12:46:07 +01:00
Bastian Köcher 56355879be Fix cli for structopt 0.3.7 and pin to that version (#4509)
* Fix cli for structopt 0.3.7 and pin to that version

This is just some hotfix to make everything compile. In the future it
will require another pr to not depend on internals of StructOpt, but
that will probably also require some additions to StructOpt itself. To
not break the code again with another StructOpt, this also pins the
StructOpt version.

* Fix benches

* Fix for fix
2019-12-28 22:52:18 +01:00
Arkadiy Paronyan 9950ea98fc Shared params in CLI API (#4466)
* Common shared parames getter

* Expose more types from `service-builder`

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2019-12-20 16:05:01 +01:00
Stanislav Tkach d46eaf79aa Warn about using --rpc-external and --ws-external options (#4448)
* Warn about using --rpc-external and --ws-external options

* Apply review comments

* Remove links placeholders

* Add links to wiki
2019-12-19 17:57:42 +01:00
Niklas Adolfsson 7c6ad9dd1c [client cli] generic blocknumber (#4376)
* rewrite me

* [cli]: make `BlockNumber` generic

* cleanup
2019-12-17 11:45:20 +01:00
Tomasz Drwięga 8778ca7dc8 [big refactor] Remove crate aliasing. (#4395)
* Rename: Phase 1.

* Unify codec.

* Fixing: Phase 2

* Fixing: Phase 3.

* Fixing: Phase 4.

* Fixing: Phase 5.

* Fixing: Phase 6.

* Fixing: Phase 7.

* Fixing: Phase 8. Tests

* Fixing: Phase 9. Tests!!!

* Fixing: Phase 10. Moar tests!

* Finally done!

* More fixes.

* Rename primitives:: to sp_core::

* Apply renames in finality-grandpa.

* Fix benches.

* Fix benches 2.

* Revert node-template.

* Fix frame-system in our modules.
2019-12-16 20:36:49 +08:00
Ashley a58e8a6e45 Update exit-future and make sc-cli compile on wasm (#4289)
* updated exit-future (github repo)

* Switch to broadcast crate

* Migrate client/cli

* Switch exit-future to modernize branch

* Small changes

* Switch to cargo version and fix fg tests

* Revert "Small changes"

This reverts commit a488106805d220cb4aee9e46a71481424c6d87d5.
2019-12-04 18:35:33 +01:00
Max Inden 2de6f82b8a client: Introduce --sentry-nodes flag (#4285)
* client/authority-discovery: Add smoke tests for intervall_at

* client/authority-discovery: Fix interval_at to fire on start

* .maintain/sentry-node: Update docker compose file

* client: Introduce --sentry-nodes flag

Enable operators to specify the public addresses of sentry nodes infront
of a validator node so that the validator node can announce the sentry
node addresses instead of its own public addresses on the DHT via the
authority discovery module.

* client/authority-discovery: Break lines at 100 characters

Limit line length to 100 instead of 120 characters.
2019-12-03 18:03:08 +01:00
Benjamin Kampmann 927e13c13a The crate rename (#4223)
* Adding script for rename, could be applicable for nodes on top of it, too

* add stderr and gitlab ci features

* apply script

* fix now minor details in expected stderr

* Update the Cargo.lock

* fix name: sc-transaction -> sc-tracing

* fix rename in script, too
2019-12-02 11:23:53 +01:00
Marcio Diaz c791615387 Fix db path on normal run. (#4254) 2019-11-30 19:37:03 +01:00
Bastian Köcher a924230ccb Fix filling the db path in the CLI (#4253) 2019-11-30 12:25:37 +01:00
Arkadiy Paronyan 0b52f194f5 Fixed node binary (#4244)
* Fixed node binary

* Remove extra params
2019-11-29 09:09:12 +01:00
Arkadiy Paronyan 504e2f8bd5 Check block CLI command (#4240)
* Check block operation

* Update client/cli/src/lib.rs

* Update client/cli/src/params.rs
2019-11-28 11:24:28 +00:00
Qinxuan Chen 4a21f9bbfd Prune some duplicate dependencies (#4219)
* Prune some duplicate dependencies

* Remove ed25519-dalek 0.9.1
* Remove hex 0.3.2
* Remove parity-wasm 0.40.3
* Remove pwasm-utils 0.11.0
* Remove wasmi-validation 0.2.0
* Remove quickcheck 0.8.5
* Remove tempdir (Replace tempdir with tempfile)

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* Remove useless tempdir_with_prefix

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* Remove derive_more 0.15

Signed-off-by: koushiro <koushiro.cqx@gmail.com>
2019-11-28 09:55:30 +01:00
Benjamin Kampmann bd652793db Remove all (non-dev) client references from frame, activate dependency enforcer (#4184)
* Move transaction pool to primitives

* move backend, errors into primitives

* remove unused client depencies

* Move rpc-api into primitives

* Move peerset back to client

* Move rpc/api back to client, move palette/support/rpc into utils

* move support-rpc into subfolder

* move system-rpc into utils

* move transaction-pool  and -graph back into client

* fix broken imports

* Clean up test primitives

* Make support test utils independent of frame

* remove unnecessary node dependencies from service

* Reactivate dependency script:
 - only enforce the now achieved status quo will remain
 - allow for primitives to depend on /client for now without failing
 - more discriptive error message so people understand, what it wants
 - minor fix to differentiative between ../client and /client (which may be a subfolder)
 - don't allow this to fail anylonger.

* fix doc comment

* 'Should not' rather than 'must not'.

* Revert unwanted dependency changes

* fix faulty import

* fixup derive_more version

* fix wrong import path
2019-11-26 21:38:28 +01:00
Qinxuan Chen c7d9c58ce8 Update some dependencies: (#4207)
- derive_more 0.15 -> 0.99
- strum 0.15 -> 0.16
- slog_derive 0.1 -> 0.2

Signed-off-by: koushiro <koushiro.cqx@gmail.com>
2019-11-26 18:39:58 +01:00
Marcio Diaz e1a911cac4 Add receiver for Grafana (#4211) 2019-11-26 18:17:34 +01:00
Bastian Köcher 9ead395bff Give state_col 90% of memory budget and fix other col calculation (#4208)
* Give `state_col` 90% of memory budget and fix other col calculation

* Set default db cache size to 1024
2019-11-26 13:25:43 +01:00
Tomasz Drwięga 86b6ac5571 Offchain execution extensions (#4145)
* Pass Extensions instead of individual objects.

* Move TransactionPool to a separate ExternalitiesExtension.

* Fix compilation.?

* Clean up.

* Refactor testing utilities.

* Add docs, fix tests.

* Fix doctest.

* Fix formatting and add some logs.

* Add some docs.

* Remove unused files.
2019-11-22 17:10:23 +01:00
Ashley 145efab68f Grafana integration (#3913)
* Very WIP

* record_metrics macro works

* Integrate into service

* Licenses and documentation

* Remove unused Debugs, make respond function clearer

* Conform to line widths, fix service test

* Switch to storing the timestamps as millis instead

* Update core/grafana-data-source/src/lib.rs

Co-Authored-By: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Transform timestamps to i64 in serialization

* Fix license date

* Binary sort to find selection range for metrics

* Obey maxDataPoints

* Run a cleaning future

* Newlines at EOF

* Update core/service/Cargo.toml

Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>

* Update core/grafana-data-source/src/lib.rs

Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>

* Fix indentation

* Improve select_points

* Made test more accurate

* Inprogress

* Use the same futures version as hyper for now

* Error handling

* Remove dependence on hyper's tokio feature

* Added target_os flag

* Update Cargo.toml

Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>

* Simplify example

* Remove compat wildcard

* Updated lock file

* Fix indentation 😉
2019-11-22 15:49:49 +01:00
mattrutherford 9cc16e539e Add instrumentation, with performance profiling capability (#4132)
* Implement instrumentation for performance profiling

* Add profiling to runtime functions declared in `decl_module` inc.
`on_initialize` and `on_finalize`

* Remove changes

* add docs, tidy up

* fix versions

* fix copyright date

* switch to hashmap and instant

* update example

* update example

* implement receiver for logger and make default

* fix comment

* use `if_std!` macro

* remove whitespace

* fix whitespace

* fix nits
2019-11-22 15:21:36 +01:00
Ashley 1735683cc9 Upgrade from futures-preview to futures 0.3.1, and remove futures 0.1 where currently possible (#4083)
* Migrate node and node-template

* Migrate srml

* Simple changes

* Add async-std for interval

* Fix test-runtime warning

* Small changes

* move futures01 in core/rpc to dev-deps

* Change wasm CI builds

* Switch to async-std 1.0.1

* Remove async-std dep of network

* Add modified lockfile

* Fix node cli browser build

* Remove authority-discovery async-std dep

* Add Send + Sync to interval dyn stream
2019-11-22 13:06:23 +01:00
Pierre Krieger f9a4cff50a Remove the exit parameter from importing and exporting blocks (#3954)
* export_blocks now generates a Future

* Handle the export blocks exit in substrate-cli instead

* Turn import_blocks entirely into a Future

* Move the exit parameter to substrate-cli for imports
2019-11-20 10:03:43 +01:00
Benjamin Kampmann 60e5011c72 Reorganising the repository - external renames and moves (#4074)
* Adding first rough ouline of the repository structure

* Remove old CI stuff

* add title

* formatting fixes

* move node-exits job's script to scripts dir

* Move docs into subdir

* move to bin

* move maintainence scripts, configs and helpers into its own dir

* add .local to ignore

* move core->client

* start up 'test' area

* move test client

* move test runtime

* make test move compile

* Add dependencies rule enforcement.

* Fix indexing.

* Update docs to reflect latest changes

* Moving /srml->/paint

* update docs

* move client/sr-* -> primitives/

* clean old readme

* remove old broken code in rhd

* update lock

* Step 1.

* starting to untangle client

* Fix after merge.

* start splitting out client interfaces

* move children and blockchain interfaces

* Move trie and state-machine to primitives.

* Fix WASM builds.

* fixing broken imports

* more interface moves

* move backend and light to interfaces

* move CallExecutor

* move cli off client

* moving around more interfaces

* re-add consensus crates into the mix

* fix subkey path

* relieve client from executor

* starting to pull out client from grandpa

* move is_decendent_of out of client

* grandpa still depends on client directly

* lemme tests pass

* rename srml->paint

* Make it compile.

* rename interfaces->client-api

* Move keyring to primitives.

* fixup libp2p dep

* fix broken use

* allow dependency enforcement to fail

* move fork-tree

* Moving wasm-builder

* make env

* move build-script-utils

* fixup broken crate depdencies and names

* fix imports for authority discovery

* fix typo

* update cargo.lock

* fixing imports

* Fix paths and add missing crates

* re-add missing crates
2019-11-14 21:51:17 +01:00