Commit Graph

27 Commits

Author SHA1 Message Date
André Silva a26ec52450 make SelectChain async (#9128)
* make SelectChain async

* make JustificationImport async
2021-06-20 11:01:09 +00:00
Bastian Köcher 2675741a09 Rework inherent data client side (#8526)
* Lol

* Yeah

* Moare

* adaasda

* Convert AURA to new pallet macro

* AURA: Switch to `CurrentSlot` instead of `LastTimestamp`

This switches AURA to use `CurrentSlot` instead of `LastTimestamp`.

* Add missing file

* Update frame/aura/src/migrations.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Remove the runtime side provide inherent code

* Use correct weight

* Add TODO

* Remove the Inherent from AURA

* 🤦

* Remove unused stuff

* Update primitives authorship

* Fix babe inherent data provider

* Fix consensus-uncles

* Fix BABE

* Do some further changes to authorship primitives... :D

* More work

* Make it compile the happy path

* Make it async!

* Take hash

* More stuff

* Hacks

* Revert "Hacks"

This reverts commit cfffad88668cfdebf632a59c4fbfada001ef8251.

* Fix

* Make `execute_block` return the final block header

* Move Aura digest stuff

* Make it possible to disable equivocation checking

* Fix fix fix

* Some refactorings

* Comment

* Fixes fixes fixes

* More cleanups

* Some love

* Better love

* Make slot duration being exposed as `Duration` to the outside

* Some slot info love

* Add `build_aura_worker` utility function

* Copy copy copy

* Some stuff

* Start fixing pow

* Fix pow

* Remove some bounds

* More work

* Make grandpa work

* Make slots use `async_trait`

* Introduce `SharedData`

* Add test and fix bugs

* Switch to `SharedData`

* Make grandpa tests working

* More Babe work

* Make grandpa work

* Introduce `SharedData`

* Add test and fix bugs

* Switch to `SharedData`

* Make grandpa tests working

* More Babe work

* Make it async

* Fix fix

* Use `async_trait` in sc-consensus-slots

This makes the code a little bit easier to read and also expresses that
there can always only be one call at a time to `on_slot`.

* Make grandpa tests compile

* More Babe tests work

* Fix network test

* Start fixing service test

* Finish service-test

* Fix sc-consensus-aura

* Fix fix fix

* More fixes

* Make everything compile *yeah*

* Make manual-seal compile

* More fixes

* Start fixing Aura

* Fix Aura tests

* Fix Babe tests

* Make everything compile

* Move code around and switch to async_trait

* Fix Babe

* Docs docs docs

* Move to FRAME

* Fix fix fix

* Make everything compile

* Last cleanups

* Fix integration test

* Change slot usage of the timestamp

* We really need to switch to `impl-trait-for-tuples`

* Update primitives/inherents/src/lib.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update primitives/inherents/src/lib.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update primitives/inherents/src/lib.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Some extra logging

* Remove dbg!

* Update primitives/consensus/common/src/import_queue/basic_queue.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2021-05-03 14:39:25 +00:00
Bastian Köcher 217c4be226 Make BlockImport and Verifier async (#8472)
* Make grandpa work

* Introduce `SharedData`

* Add test and fix bugs

* Switch to `SharedData`

* Make grandpa tests working

* More Babe work

* Make it async

* Fix fix

* Use `async_trait` in sc-consensus-slots

This makes the code a little bit easier to read and also expresses that
there can always only be one call at a time to `on_slot`.

* Make grandpa tests compile

* More Babe tests work

* Fix network test

* Start fixing service test

* Finish service-test

* Fix sc-consensus-aura

* Fix fix fix

* More fixes

* Make everything compile *yeah*

* Fix build when we have Rust 1.51

* Update client/consensus/common/src/shared_data.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update client/consensus/common/src/shared_data.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update client/consensus/common/src/shared_data.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update client/consensus/common/src/shared_data.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update client/consensus/common/src/shared_data.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update client/consensus/babe/src/tests.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update client/consensus/babe/src/tests.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Fix warning

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2021-03-30 09:19:49 +00:00
Jon Häggblad 0d6884b919 Storing multiple Justifications per block (#7640)
* primitives/runtime: initial changes on supporting multiple Justifications

* primitives/runtime: make Justifications strongly typed

* Encode/decode Justifications

* primitives/runtime: add Justification type

* backend: apply_finality and finalize_block takes a single Justification

* manual-seal: create engine id and let rpc take encoded justification

* backend: skeleton functions for appending justifications

* backend: initial implementation append_justification

Initial implementation of append_justification on the Backend trait, and also remove unused skeleton
functions for append_justificaton on Finaziler trait.
k

* backend: guard against duplicate consensus engine id

* client/db: add check for block finality

* client/api: add append_justification to in_mem db

* client/light: add no-op append_justification

* network: fix decode call for Justification

* network: only send a single Justification in BlockData

* network: minor comment update

* protocol: update field names to distinguish single justification

* client: further field renames to plural

* client: update function names to plural justifications

* client/db: upgrade existing database for new format

* network: remove dependency on grandpa crate

* db: fix check for finalized block

* grandpa: check for multiple grandpa justifications hwne importing

* backend: update Finalizer trait to take multiple Justifications

* db: remove debugging statements in migration code

* manual-seal: update note about engine id

* db: fix check for finalized block

* client: update variable name to reflect it is now plural

* grandpa: fix incorrect empty Justications in test

* primitives: make Justifications opaque to avoid being empty

* network: fix detecting empty Justification

* runtime: doc strings for Justifications functions

* runtime: add into_justifications

* primitives: check for duplicates in when adding to Justifications

* network/test: use real grandpa engine id in test

* client: fix reviewer comments

* primitives: rename Justifications::push to append

* backend: revert changes to Finalizer trait

* backend: revert mark_finalized

* backend: revert changes to finalize_block

* backend: revert finalized_blocks

* db: add a quick early return for performance

* client: minor reviewer comments

* service/test: use local ConsensusEngineId

* network: add link to issue for sending multiple Justifications

* Apply suggestions from code review

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* Apply suggestions from code review

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* network: tweaks to review suggestions

* network: revert change to BlockData for backwards compatibility

* Apply suggestion from code review

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* Apply suggestions from code review

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* primitives: update doc comment for Justifications

* client/db/upgrade: avoid grandpa crate dependency

* consensus: revert to single Justification for import_justification

* primitives: improve justifications docs

* style cleanups

* use and_then

* client: rename JUSTIFICATIONS db column

* network: revert to using FRNK in network-test

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: André Silva <andrerfosilva@gmail.com>
2021-03-17 21:18:16 +00:00
Bastian Köcher 821e018d75 Ensure we spawn the block import worker as an essential task (#8155)
* Ensure we spawn the block import worker as an essential task

This pr ensures that we spawn the block import worker as an essential
task. This is quite important as we need to bring down the node when the
block import is done. Besides that it adds some debug output to the
block import worker.

* Don't be stupid :D
2021-02-19 16:31:03 +00:00
Bastian Köcher e21a61eac8 Rewrite the async code in BasicQueue (#7988)
* Rewrite the async code in `BasicQueue`

This is some smaller change to rewrite the async code in `BasicQueue`. I
require this for some other pr I'm working on ;)

* Update primitives/consensus/common/src/import_queue/basic_queue.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update primitives/consensus/common/src/import_queue/basic_queue.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update primitives/consensus/common/src/import_queue/basic_queue.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Hmm :D

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2021-01-26 16:38:57 +01:00
Bastian Köcher e3e651f72c Happy new year (#7814)
* Happy new year

Updates the copyright years and fixes wrong license headers.

* Fix the template

* Split HEADER into HEADER-APACHE & HEADER-GPL
2021-01-04 09:03:13 +00:00
André Silva 1871a95088 grandpa: remove light-client specific block import pipeline (#7546)
* grandpa: remove light-client specific block import

* consensus, network: remove finality proofs
2020-11-23 14:28:55 +00:00
André Silva 463ba54d00 consensus: prioritize finality work over block import in queue (#7307)
* consensus: prioritize finality work over block import in queue

* consensus: add test for import queue task priority
2020-10-16 16:22:50 +02:00
Pierre Krieger 1b350a050c Add more metrics to the import queue (#7207) 2020-09-24 21:11:29 +02:00
Pierre Krieger c0ebb00fd3 Fuse the import queue receiver (#6876)
* Fix the import queue receiver

* Add logging
2020-08-12 08:58:16 +00:00
pscott 046fda914a Improve overall performance (#6699)
* Improve overall performance

* Clean up code

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Remove needless ::

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Remove needless ::

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2020-07-21 14:46:49 +02:00
Ashley 761dbd7dcc Split the service initialisation up into seperate functions (#6332)
* Seperate out the complexity in ServiceBuilder::build_common into seperate functions

* Fix line widths

* Move some functions to their respective crates
2020-06-16 15:50:21 +02:00
Nikolay Volf 5759cee2e9 Update import metrics and add verification time (#6170)
* refactor import reporting and add time

* Update primitives/consensus/common/src/metrics.rs

Co-authored-by: Max Inden <mail@max-inden.de>

* remove (crate)

* fix longer lines

* swap names to avoid api breaking

Co-authored-by: Max Inden <mail@max-inden.de>
2020-06-01 11:08:59 +02:00
Benjamin Kampmann efc4849f1a Meter block import results via prometheus (#6025) 2020-05-15 17:25:51 +02:00
Benjamin Kampmann be8c96adec Relax substrate licensing scheme (#5947) 2020-05-15 13:21:56 +02:00
Bastian Köcher 9c5536e01a Expose that BasicQueue expects blocking spawn (#5860)
* Expose that `BasicQueue` expects blocking spawn

Up to now `BasicQueue` expected a closure that to spawn a `Future`.
This was expected to be a closure that spawns a blocking future.
However, this wasn't documented anywhere. This pr introduces a new trait
`SpawnBlocking` that exposes this requirement to the outside.

* Feedback
2020-05-04 19:40:29 +02:00
pscott 0f401e4699 Move spawning tasks from thread pools to Service's TaskManager for block importing (#5647)
Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>
2020-04-29 18:46:39 +02:00
Benjamin Kampmann 247822bb33 Additional Metrics collected and exposed via prometheus (#5414)
This PR refactors the metrics measuring and Prometheus exposing entity in sc-service into its own submodule and extends the parameters it exposes by:

- system load average (over one, five and 15min)
- the TCP connection state of the process (lsof), refs #5304
- number of tokio threads
- number of known forks
- counter for items in each unbounded queue (with internal unbounded channels)
- number of file descriptors opened by this process (*nix only at this point)
- number of system threads (*nix only at this point)

refs #4679

Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Ashley <ashley.ruglys@gmail.com>
2020-04-04 15:13:35 +02:00
Shawn Tabrizi c7a14db792 Fix various spelling errors (#4940)
* meaningfull -> meaningful

* initialise -> initialize

* tokans -> tokens

* incentivise -> incentivize

* lenght -> length

* incentivisation -> incentivization

* doesnt't -> doesn't

* overwriten -> overwritten

* lifecycle -> life cycle

* lifecycle -> life cycle

* usefull -> useful

* noone -> no one

* spaming -> spamming

* defered -> deferred

* hieght -> height

* sumation -> summation

* ingore -> ignore

* registed -> registered

* Auxialary -> Auxiliary

* loggin -> logging

* independance -> independence

* trailling -> trailing

* responsability -> responsibility

* trunkated -> truncated

* Weither -> Whether

* informations -> information

* Runtume -> Runtime

* choosen -> chosen

* delcataion -> declaration

* Unchekced -> Unchecked

* defintion -> definition

* scrach -> scratch

* imput -> input

* transfered -> transferred

* endownment -> endowment

* Determinator -> Determiner

* relevent -> relevant

* emited -> emitted

* acocunt -> account

* proprotional -> proportional

* instantiaion -> instantiation

* commited -> committed

* tombstonedead -> tombstone

* uwnrap -> unwrap

* acount -> account

* specialised -> specialized

* existant -> existent

* requried -> required

* Anull -> Annul

* AUTHORITES -> AUTHORITIES

* underyling -> underlying

* recognisable -> recognizable

* Capitalise -> Capitalize

* reportfor -> report for

* hearbeat -> heartbeat

* onlineness -> being online

* creater -> creator

* Bytearray -> Byte array

* Despoit -> Deposit

* substratced -> subtracted

* Curent -> Current

* imbalanes -> imbalances

* countfown -> countdown

* inexisting -> inexistent

* additionaly -> additionally

* substracted -> subtracted

* auxilary -> auxiliary

* parital -> partial

* in't -> isn't

* compatability -> compatibility

* infomation -> information

* etected -> detected

* extrinsiscs -> extrinsics

* reprensentation -> representation

* coonfiguration -> configuration

* primtives -> primitives

* miscelanious -> miscellaneous

* VERISON -> VERSION

* endcoded -> encoded

* Genrates -> Generates

* miliseconds -> milliseconds

* occured -> occurred

* trully -> truely

* truely -> truly

* conjuction -> conjunction

* encouters -> encounters

* customised -> customized

* deterministicly -> deterministically

* finalisation -> finalization

* pluggable -> plugable

* wakeup -> wake-up

* interemdiate -> intermediate

* intepreting -> interpreting

* finalzied -> finalized

* throgh -> through

* extinsic -> extrinsic

* convient -> convenient

* allocater -> allocator

* propagateable -> propagatable

* succesfuly -> successfully

* finalising -> finalizing

* publically -> publicly

* phrasee -> phrase

* substration -> substractions

* substractions -> subtractions

* neccessarily -> necessarily

* Inlucde -> Include

* unefficient -> inefficient

* thay -> they

* funtion -> function

* datastructures -> data structures

* infromation -> information

* propagatable -> propagable

* ecountered -> encountered

* recognise -> recognize

* intergration -> integration

* lastet -> latest

* datatypes -> data types

* datatype -> data type

* Strongarming -> Strong Arming

* avaible -> available

* Commiting -> Committing

* Retreiving -> Retrieving

* shoud -> should

* canonicaliziation -> canonicalization

* comitted -> committed

* clonable -> cloneable

* Uknown -> Unknown

* reponse -> response

* arbitary -> arbitrary

* Capapbilities -> Capabilities

* responsbile -> responsible

* initialisation -> initialization

* cames -> came

* intemediate -> intermediate

* reqeust -> request

* intance -> instance

* explcitly -> explicitly

* neighor -> neighbor

* reolving -> resolving

* untill -> until

* Validte -> Validate

* deserailize -> deserialize

* literaly -> literally

* preceeding -> preceding

* abpve -> above

* chcecked -> checked

* numbet -> number

* Unknow -> Unknown

* halfs -> halves

* gossup -> gossip

* givent -> given

* immediatelly -> immediately

* slicable -> sliceable

* conensus -> consensus

* Mimicks -> Mimics

* acccept -> accept

* serialise -> serialize

* exstrinsics -> extrinsics

* panicks -> panics

* maintaince -> maintenance

* repeatidely -> repeatedly

* anecstor -> ancestor

* becasue -> because

* processer -> processor

* Prunning -> Pruning

* insterested -> interested

* unuseful -> not useful

* yeided -> yielded

* descendfing -> descending

* corresponts -> corresponds

* survivew -> survive

* keps -> keeps

* ligh -> light

* prerequisities -> prerequisites

* positiion -> position

* depedency -> dependency

* extrinisic -> extrinsic

* atomicaly -> atomically

* staticly -> statically

* resul -> result

* timestamb -> timestamp

* Utilites -> Utilities

* ammount -> amount

* pocess -> process

* exteral -> external

* Update client/finality-grandpa/src/tests.rs

* Update primitives/io/src/lib.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update primitives/blockchain/src/lib.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update frame/support/src/weights.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update bin/node/cli/tests/common.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update client/api/src/execution_extensions.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update client/cli/src/params.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update client/executor/common/src/sandbox.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update client/api/src/execution_extensions.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update client/finality-grandpa/src/communication/mod.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update client/state-db/src/pruning.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update frame/contracts/src/tests.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update client/api/src/execution_extensions.rs

* bump impl

* timestamb -> timestamp

Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
2020-02-17 15:07:24 +01:00
Arkadiy Paronyan c8fa6518bf Fix import queue thread pool shutdown (#4929)
* Fix import queue thread pool shutdown

* Make sure runtime is disposed before telemetry

* Close channel istead of sending a message

* Fixed test
2020-02-17 10:49:40 +01:00
Pierre Krieger b452867eb7 Add diagnostics to tasks (#4752) 2020-01-29 11:46:39 +01:00
Bastian Köcher fd6b29dd2c Remove requirement on Hash = H256, make Proposer return StorageChanges and Proof (#3860)
* Extend `Proposer` to optionally generate a proof of the proposal

* Something

* Refactor sr-api to not depend on client anymore

* Fix benches

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Apply suggestions from code review

* Introduce new `into_storage_changes` function

* Switch to runtime api for `execute_block` and don't require `H256`
anywhere in the code

* Put the `StorageChanges` into the `Proposal`

* Move the runtime api error to its own trait

* Adds `StorageTransactionCache` to the runtime api

This requires that we add `type NodeBlock = ` to the
`impl_runtime_apis!` macro to work around some bugs in rustc :(

* Remove `type NodeBlock` and switch to a "better" hack

* Start using the transaction cache from the runtime api

* Make it compile

* Move `InMemory` to its own file

* Make all tests work again

* Return block, storage_changes and proof from Blockbuilder::bake()

* Make sure that we use/set `storage_changes` when possible

* Add test

* Fix deadlock

* Remove accidentally added folders

* Introduce `RecordProof` as argument type to be more explicit

* Update client/src/client.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update primitives/state-machine/src/ext.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Integrates review feedback

* Remove `unsafe` usage

* Update client/block-builder/src/lib.rs

Co-Authored-By: Benjamin Kampmann <ben@gnunicorn.org>

* Update client/src/call_executor.rs

* Bump versions

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com>
2020-01-10 10:48:32 +01:00
Shawn Tabrizi 7efa62725e Update copyright year (#4532) 2020-01-05 14:05:44 +01:00
Benjamin Kampmann 605c0e655e Fix a bunch of low work dependency cycles (#4354)
* take test-client off sp-consensus

* use test primitives rather than test client in authority discovery tests

* move runtime-interface tests

* don't forget to remove the dev-dependency

* remove more unneeded dev deps

* add changes_trie_config to test prrimitives

* Separate network crates from its integration tests

* Fix up consensus crates for networking test changes

* remove unnecessary dependencies

* remove unused addition

* remove unnecessary dev-dependencies

* fixing finality grandpa tests

* removing unnecessary executor dependencies
2019-12-11 10:27:34 +01:00
Benjamin Kampmann 927e13c13a The crate rename (#4223)
* Adding script for rename, could be applicable for nodes on top of it, too

* add stderr and gitlab ci features

* apply script

* fix now minor details in expected stderr

* Update the Cargo.lock

* fix name: sc-transaction -> sc-tracing

* fix rename in script, too
2019-12-02 11:23:53 +01:00
Benjamin Kampmann 60e5011c72 Reorganising the repository - external renames and moves (#4074)
* Adding first rough ouline of the repository structure

* Remove old CI stuff

* add title

* formatting fixes

* move node-exits job's script to scripts dir

* Move docs into subdir

* move to bin

* move maintainence scripts, configs and helpers into its own dir

* add .local to ignore

* move core->client

* start up 'test' area

* move test client

* move test runtime

* make test move compile

* Add dependencies rule enforcement.

* Fix indexing.

* Update docs to reflect latest changes

* Moving /srml->/paint

* update docs

* move client/sr-* -> primitives/

* clean old readme

* remove old broken code in rhd

* update lock

* Step 1.

* starting to untangle client

* Fix after merge.

* start splitting out client interfaces

* move children and blockchain interfaces

* Move trie and state-machine to primitives.

* Fix WASM builds.

* fixing broken imports

* more interface moves

* move backend and light to interfaces

* move CallExecutor

* move cli off client

* moving around more interfaces

* re-add consensus crates into the mix

* fix subkey path

* relieve client from executor

* starting to pull out client from grandpa

* move is_decendent_of out of client

* grandpa still depends on client directly

* lemme tests pass

* rename srml->paint

* Make it compile.

* rename interfaces->client-api

* Move keyring to primitives.

* fixup libp2p dep

* fix broken use

* allow dependency enforcement to fail

* move fork-tree

* Moving wasm-builder

* make env

* move build-script-utils

* fixup broken crate depdencies and names

* fix imports for authority discovery

* fix typo

* update cargo.lock

* fixing imports

* Fix paths and add missing crates

* re-add missing crates
2019-11-14 21:51:17 +01:00