Commit Graph

28 Commits

Author SHA1 Message Date
Wei Tang 8ae1aa4c28 Switch to bounded mpsc for txpool import notification stream (#6640)
* Switch to bounded mpsc for txpool import notification stream

* Update client/transaction-pool/graph/src/validated_pool.rs

Co-authored-by: Nikolay Volf <nikvolf@gmail.com>

Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
2020-07-17 12:31:47 +02:00
Bastian Köcher 94cddee160 Improve transaction submission (#6599)
* Improve transaction submission

Before this pr the transaction pool validated each transaction, even if
the transaction was already known to the pool. This pr changes the
behavior to first check if we are already aware of a transaction and
thus, to only validate them if we don't know them yet. However, there is
still the possibility that a given transaction is validated multiple
times. This can happen if the transaction is added the first time, but
is not yet validated and added to the validated pool.

Besides that, this pr fixes the wrong metrics of gossiped transactions
in the network. It also moves some metrics to the transaction pool api,
to better track when a transaction actually is scheduled for validation.

* Make sure we don't submit the same transaction twice from the network concurrently

* Remove added listener call

* Feedback

* Ignore banned on resubmit
2020-07-08 17:42:42 +02:00
Bastian Köcher d2846e2b9a Make transaction pool prune transactions only of canonical blocks (#6123)
* Make tx pool aware of retracted fork blocks

* Make it compile

* Update client/transaction-pool/src/lib.rs

Co-authored-by: Nikolay Volf <nikvolf@gmail.com>

* Fix doc test

* Simplify the implementation

* Send tree route as arc to prevent heavy clones

* Switch to use `ExtrinsicHash` to make it more clear

* Fix benchmark

Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
2020-06-05 23:12:00 +02:00
Pierre Krieger 717fa95bc7 Restore the empty line after the license (#6088) 2020-05-20 13:08:27 +00:00
Nikolay Volf d5411969ac fix whitespace (#6062) 2020-05-18 13:51:28 +02:00
Benjamin Kampmann be8c96adec Relax substrate licensing scheme (#5947) 2020-05-15 13:21:56 +02:00
Benjamin Kampmann 247822bb33 Additional Metrics collected and exposed via prometheus (#5414)
This PR refactors the metrics measuring and Prometheus exposing entity in sc-service into its own submodule and extends the parameters it exposes by:

- system load average (over one, five and 15min)
- the TCP connection state of the process (lsof), refs #5304
- number of tokio threads
- number of known forks
- counter for items in each unbounded queue (with internal unbounded channels)
- number of file descriptors opened by this process (*nix only at this point)
- number of system threads (*nix only at this point)

refs #4679

Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Ashley <ashley.ruglys@gmail.com>
2020-04-04 15:13:35 +02:00
Tomasz Drwięga 04ccb179e9 Pass transaction source to validate_transaction (#5366)
* WiP

* Support source in the runtime API.

* Finish implementation in txpool.

* Fix warning.

* Fix tests.

* Apply suggestions from code review

Co-Authored-By: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-Authored-By: Nikolay Volf <nikvolf@gmail.com>

* Extra changes.

* Fix test and benches.

* fix test

* Fix test & benches again.

* Fix tests.

* Update bumpalo

* Fix doc test.

* Fix doctest.

* Fix doctest.

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
2020-03-25 14:09:23 +01:00
Gavin Wood f5dc69b404 Don't remove invalid transactions when skipping. (#5121)
* Don't remove invalid transactions when skipping.

* Use a special kind of extrinsic instead of arbitrary limit.

* Fix txpool tests.

* Attempt to create more blocks.

* Bump lock

Co-authored-by: Gavin Wood <github@gavwood.com>
Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
2020-03-05 17:00:31 +01:00
Shawn Tabrizi c7a14db792 Fix various spelling errors (#4940)
* meaningfull -> meaningful

* initialise -> initialize

* tokans -> tokens

* incentivise -> incentivize

* lenght -> length

* incentivisation -> incentivization

* doesnt't -> doesn't

* overwriten -> overwritten

* lifecycle -> life cycle

* lifecycle -> life cycle

* usefull -> useful

* noone -> no one

* spaming -> spamming

* defered -> deferred

* hieght -> height

* sumation -> summation

* ingore -> ignore

* registed -> registered

* Auxialary -> Auxiliary

* loggin -> logging

* independance -> independence

* trailling -> trailing

* responsability -> responsibility

* trunkated -> truncated

* Weither -> Whether

* informations -> information

* Runtume -> Runtime

* choosen -> chosen

* delcataion -> declaration

* Unchekced -> Unchecked

* defintion -> definition

* scrach -> scratch

* imput -> input

* transfered -> transferred

* endownment -> endowment

* Determinator -> Determiner

* relevent -> relevant

* emited -> emitted

* acocunt -> account

* proprotional -> proportional

* instantiaion -> instantiation

* commited -> committed

* tombstonedead -> tombstone

* uwnrap -> unwrap

* acount -> account

* specialised -> specialized

* existant -> existent

* requried -> required

* Anull -> Annul

* AUTHORITES -> AUTHORITIES

* underyling -> underlying

* recognisable -> recognizable

* Capitalise -> Capitalize

* reportfor -> report for

* hearbeat -> heartbeat

* onlineness -> being online

* creater -> creator

* Bytearray -> Byte array

* Despoit -> Deposit

* substratced -> subtracted

* Curent -> Current

* imbalanes -> imbalances

* countfown -> countdown

* inexisting -> inexistent

* additionaly -> additionally

* substracted -> subtracted

* auxilary -> auxiliary

* parital -> partial

* in't -> isn't

* compatability -> compatibility

* infomation -> information

* etected -> detected

* extrinsiscs -> extrinsics

* reprensentation -> representation

* coonfiguration -> configuration

* primtives -> primitives

* miscelanious -> miscellaneous

* VERISON -> VERSION

* endcoded -> encoded

* Genrates -> Generates

* miliseconds -> milliseconds

* occured -> occurred

* trully -> truely

* truely -> truly

* conjuction -> conjunction

* encouters -> encounters

* customised -> customized

* deterministicly -> deterministically

* finalisation -> finalization

* pluggable -> plugable

* wakeup -> wake-up

* interemdiate -> intermediate

* intepreting -> interpreting

* finalzied -> finalized

* throgh -> through

* extinsic -> extrinsic

* convient -> convenient

* allocater -> allocator

* propagateable -> propagatable

* succesfuly -> successfully

* finalising -> finalizing

* publically -> publicly

* phrasee -> phrase

* substration -> substractions

* substractions -> subtractions

* neccessarily -> necessarily

* Inlucde -> Include

* unefficient -> inefficient

* thay -> they

* funtion -> function

* datastructures -> data structures

* infromation -> information

* propagatable -> propagable

* ecountered -> encountered

* recognise -> recognize

* intergration -> integration

* lastet -> latest

* datatypes -> data types

* datatype -> data type

* Strongarming -> Strong Arming

* avaible -> available

* Commiting -> Committing

* Retreiving -> Retrieving

* shoud -> should

* canonicaliziation -> canonicalization

* comitted -> committed

* clonable -> cloneable

* Uknown -> Unknown

* reponse -> response

* arbitary -> arbitrary

* Capapbilities -> Capabilities

* responsbile -> responsible

* initialisation -> initialization

* cames -> came

* intemediate -> intermediate

* reqeust -> request

* intance -> instance

* explcitly -> explicitly

* neighor -> neighbor

* reolving -> resolving

* untill -> until

* Validte -> Validate

* deserailize -> deserialize

* literaly -> literally

* preceeding -> preceding

* abpve -> above

* chcecked -> checked

* numbet -> number

* Unknow -> Unknown

* halfs -> halves

* gossup -> gossip

* givent -> given

* immediatelly -> immediately

* slicable -> sliceable

* conensus -> consensus

* Mimicks -> Mimics

* acccept -> accept

* serialise -> serialize

* exstrinsics -> extrinsics

* panicks -> panics

* maintaince -> maintenance

* repeatidely -> repeatedly

* anecstor -> ancestor

* becasue -> because

* processer -> processor

* Prunning -> Pruning

* insterested -> interested

* unuseful -> not useful

* yeided -> yielded

* descendfing -> descending

* corresponts -> corresponds

* survivew -> survive

* keps -> keeps

* ligh -> light

* prerequisities -> prerequisites

* positiion -> position

* depedency -> dependency

* extrinisic -> extrinsic

* atomicaly -> atomically

* staticly -> statically

* resul -> result

* timestamb -> timestamp

* Utilites -> Utilities

* ammount -> amount

* pocess -> process

* exteral -> external

* Update client/finality-grandpa/src/tests.rs

* Update primitives/io/src/lib.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update primitives/blockchain/src/lib.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update frame/support/src/weights.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update bin/node/cli/tests/common.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update client/api/src/execution_extensions.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update client/cli/src/params.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update client/executor/common/src/sandbox.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update client/api/src/execution_extensions.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update client/finality-grandpa/src/communication/mod.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update client/state-db/src/pruning.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update frame/contracts/src/tests.rs

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* Update client/api/src/execution_extensions.rs

* bump impl

* timestamb -> timestamp

Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
2020-02-17 15:07:24 +01:00
Nikolay Volf 86ab0cb4d9 Revalidation queue for transaction pool (#4781)
* Revalidation queeue.

* add docs and license

* move test

* refactor worker to async/await

* address review

* fix warnings

* update Cargo.lock

* move background task to service

* use tomusdrw loop

* naming

* return From::from

* add doc comment

* add more doc comments

* fix merge bug

* add doc comment for test function

* Update client/transaction-pool/src/testing/pool.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* more review fixes

* refactor to allow service keep background tasks from isntantiated subsystems

* use const delay

* fix fallout

* remove fallout

* remove already moved test

* fix doc test

* add valid_at helper

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
2020-02-17 16:48:24 +03:00
Seun Lanlege d3a3e288b6 Adds fork-awareness and finalization notifications to transaction pool watchers. (#4740)
* adds finalization support to sc-transaction-pool using MaintainedTransactionPool for finalization events

* adds TransactionStatus::Retracted, notify watchers of retracted blocks, finalized now finalizes, transactions for current finalized -> last finalized block

* adds last_finalized to ChainApi, use generic BlockT for ChainEvent

* fix tests

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* tests

* fix tests, docs, lazily dedupe pruned hashes

* fix tests, Cargo.lock

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* remove tree_route, last_finalized from ChainApi, add block hash to Finalization and Retracted events

* prune finality watchers

* fix tests

* remove HeaderBackend bound from FullChainApi

* code style nits, terminate stream in finality_timeout

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
2020-02-14 14:44:58 +01:00
Ashley ead6815ae4 Fix timer panics in the wasm light client (#4561)
* Make WASM browser thing compile

* Fix

* updated exit-future (github repo)

* Switch to broadcast crate

* Migrate client/cli

* Switch exit-future to modernize branch

* Small changes

* Switch to cargo version and fix fg tests

* fix basic-authorship

* Fix crash on grafana macro

* Fix grafana macro

* Switch node python version

* Disable record_metrics_slice in grafana macro on wasm

* Update client/grafana-data-source/src/lib.rs

* Revert "Update client/grafana-data-source/src/lib.rs"

This reverts commit 888009a8e0b7051bd4bfbbfdb0448bcf2e2aae93.

* Add wasm support for state machine

* Switch to my own libp2p version

* Revert "Switch to my own libp2p version"

This reverts commit ce613871b59264b3165b45c37943e6560240daa7.

* Revert "Add wasm support for state machine"

This reverts commit de7eaa0694d9534fc3b164621737968e9a6a7c5f.

* Add sc-browser

* Squash

* remove sc-browser

* Fix keystore on wasm

* stubs for removed functions to make env compatible with old runtimes

* Add test (that doesn't work)

* Fix build scripts

* Revert basic-authorship due to no panics

* Revert cli/informant

* Revert consensus

* revert offchain

* Update utils/browser/Cargo.toml

Co-Authored-By: Benjamin Kampmann <ben@gnunicorn.org>

* export console functions

* Add new chainspec

* Fix ws in chain spec

* revert chainspec

* Fix chainspec

* Use an Option<PathBuf> in keystore instead of cfg flags

* Remove crud

* Only use wasm-timer for instant and systemtime

* Remove telemetry changes

* Assuming this is ok

* Add a KeystoreConfig

* Add stubs back in

* Update libp2p

* Revert "Add stubs back in"

This reverts commit 4690cf1882aa0f99f7f00a58c4080c8aa9b77c36.

* Remove commented js again

* Bump kvdb-web version

* Fix cli

* Switch branch on futures-timer

* Fix tests

* Remove sc-client test build in check-web-wasm because there isn't a good way to build futures-timer with wasm-bindgen support in the build

* Remove more things ^^

* Switch branch on futures-timer back

* Put DB io stats behind a cfg flag

* Fix things

* Don't timeout transports on wasm

* Update branch of futures-timer and fix bad merge

* Spawn informant

* Fix network test

* Fix delay resets

* Changes

* Fix tests

* use wasm_timer for transaction pool

* Fixes

* Switch futures-timer to crates

* Only diagnose futures on native

* Fix sc-network-test tests

* Select log level in js

* Fix syncing ;^)

* Allow disabling colours in the informant

* Use OutputFormat enum for informant

* MallocSizeOf impl on transaction pool broke stuff because wasm_timer::Instant doesnt impl it so just revert the transaction pool to master

* Update futures-diagnose

* Revert "MallocSizeOf impl on transaction pool broke stuff because wasm_timer::Instant doesnt impl it so just revert the transaction pool to master"

This reverts commit baa4ffc94fd968b6660a2c17ba8113e06af15548.

* Pass whole chain spec in start_client

* Get Instant::now to work in transaction pool again

* Informant dep reordering

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Svyatoslav Nikolsky <svyatonik@gmail.com>
Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com>
Co-authored-by: Demi Obenour <48690212+DemiMarie-parity@users.noreply.github.com>
2020-02-10 12:23:55 +01:00
Nikolay Volf 41967186e4 Subsystems memory tracking: 1. Transaction pool (#4822)
* update sp-runtime

* total update

* usage informant

* update to crates.io version

* update Cargo.lock

* update dummy update

* fix todo

* cleanup

* avoid custom impl

* Update client/transaction-pool/graph/src/future.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* remove another custom impl

* remove another custom impl

* add kb in report

* update Cargo.lock

* review suggestions

* --amend

* --amend

* bump parity-util-mem to 0.5.0

* bumps

* update macro and versions

* add to grafana

* naming

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
2020-02-07 11:53:11 +01:00
Nikolay Volf 6148c7ce15 do join_all (#4832) 2020-02-06 13:07:52 +01:00
Nikolay Volf ed3da9f903 Fix quadratic iterations over transaction pool (#4736)
* transaction pool changes

* service & network changes

* address review

* reduce future pool
2020-01-27 18:26:42 +01:00
Nikolay Volf 1614ce0996 Async/await in transaction-graph (#4645)
* async/await in tx graph

* review notes

* remove unused typedef
2020-01-24 05:22:39 -08:00
Nikolay Volf 14e95f3398 Refactor tx-pool maintenance and other high-level api (#4629)
* Reduction.

* Reformation.

* add locked timer stuff

* fix issues and introduce full pool

* arrange together

* fix benches

* fix new_light

* Add revalidation test case

* review fixes

* review fixes

* use just ready future

* address review
2020-01-24 04:21:24 -08:00
Shawn Tabrizi 7efa62725e Update copyright year (#4532) 2020-01-05 14:05:44 +01:00
Tomasz Drwięga 216f77d91f Document TransactionStatus and fix termination conditions. (#4446)
* Document TransactionStatus and fix termination conditions.

* Update client/rpc-api/src/author/mod.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>
2019-12-18 21:28:25 +01:00
Tomasz Drwięga 8778ca7dc8 [big refactor] Remove crate aliasing. (#4395)
* Rename: Phase 1.

* Unify codec.

* Fixing: Phase 2

* Fixing: Phase 3.

* Fixing: Phase 4.

* Fixing: Phase 5.

* Fixing: Phase 6.

* Fixing: Phase 7.

* Fixing: Phase 8. Tests

* Fixing: Phase 9. Tests!!!

* Fixing: Phase 10. Moar tests!

* Finally done!

* More fixes.

* Rename primitives:: to sp_core::

* Apply renames in finality-grandpa.

* Fix benches.

* Fix benches 2.

* Revert node-template.

* Fix frame-system in our modules.
2019-12-16 20:36:49 +08:00
Tomasz Drwięga 1628ba3388 Revalidate some transactions on every block import. (#4292)
* Revalidate some transactions on every block import.

* Fix endless loop in revalidate_ready.

* Clean up logging a bit.

* More clean ups.

* Print status after resubmitting.

* Remove env_logger.

* Remove redundant log.
2019-12-04 19:16:00 +01:00
Benjamin Kampmann 927e13c13a The crate rename (#4223)
* Adding script for rename, could be applicable for nodes on top of it, too

* add stderr and gitlab ci features

* apply script

* fix now minor details in expected stderr

* Update the Cargo.lock

* fix name: sc-transaction -> sc-tracing

* fix rename in script, too
2019-12-02 11:23:53 +01:00
Tomasz Drwięga a401acc690 Add some logs, fix reorg import. (#4250) 2019-12-01 10:14:01 +01:00
Svyatoslav Nikolsky a782021ee8 Recover transaction pool on light client (#3833)
* recover tx pool on light client

* revert local tests fix

* removed import renamings

* futures03::Future -> std::future::Future

* Update core/transaction-pool/graph/src/error.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* replace remove_from_ready with remove_invalid

* avoid excess hashing

* debug -> warn

* TransactionPool + BasicTransactionPool

* pause future tx reject when resubmitting

* bump impl_version to make CI happy

* and revert back local test fixes

* alter doc to restart CI

* Transaction::clone() -> Transaction::duplicate()

* transactions -> updated_tranasctions

* remove explicit consensus-common ref

* ::std:: -> std::

* manual set/unset flag -> calling clusore with given flag value

* removed comments

* removed force argument

* BestIterator -> Box<Iterator>

* separate crate for TxPool + Maintainer trait

* long line fix

* pos-merge fix

* fix benches compilation

* Rename txpoolapi to txpool_api

* Clean up.

* Finalize merge.

* post-merge fix

* Move transaction pool api to primitives directly.

* Consistent naming for txpool-runtime-api

* Warn about missing docs.

* Move  abstraction for offchain calls to tx-pool-api.

* Merge RPC instantiation.

* Update cargo.lock

* Post merge fixes.

* Avoid depending on client.

* Fix build
2019-11-28 00:00:54 +00:00
Tomasz Drwięga 86b6ac5571 Offchain execution extensions (#4145)
* Pass Extensions instead of individual objects.

* Move TransactionPool to a separate ExternalitiesExtension.

* Fix compilation.?

* Clean up.

* Refactor testing utilities.

* Add docs, fix tests.

* Fix doctest.

* Fix formatting and add some logs.

* Add some docs.

* Remove unused files.
2019-11-22 17:10:23 +01:00
Seun LanLege 04bd926923 Better docs for TxPool::import_notification_stream (#4157)
* better docs for import_notification_stream

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
2019-11-22 13:45:05 +01:00
Benjamin Kampmann 60e5011c72 Reorganising the repository - external renames and moves (#4074)
* Adding first rough ouline of the repository structure

* Remove old CI stuff

* add title

* formatting fixes

* move node-exits job's script to scripts dir

* Move docs into subdir

* move to bin

* move maintainence scripts, configs and helpers into its own dir

* add .local to ignore

* move core->client

* start up 'test' area

* move test client

* move test runtime

* make test move compile

* Add dependencies rule enforcement.

* Fix indexing.

* Update docs to reflect latest changes

* Moving /srml->/paint

* update docs

* move client/sr-* -> primitives/

* clean old readme

* remove old broken code in rhd

* update lock

* Step 1.

* starting to untangle client

* Fix after merge.

* start splitting out client interfaces

* move children and blockchain interfaces

* Move trie and state-machine to primitives.

* Fix WASM builds.

* fixing broken imports

* more interface moves

* move backend and light to interfaces

* move CallExecutor

* move cli off client

* moving around more interfaces

* re-add consensus crates into the mix

* fix subkey path

* relieve client from executor

* starting to pull out client from grandpa

* move is_decendent_of out of client

* grandpa still depends on client directly

* lemme tests pass

* rename srml->paint

* Make it compile.

* rename interfaces->client-api

* Move keyring to primitives.

* fixup libp2p dep

* fix broken use

* allow dependency enforcement to fail

* move fork-tree

* Moving wasm-builder

* make env

* move build-script-utils

* fixup broken crate depdencies and names

* fix imports for authority discovery

* fix typo

* update cargo.lock

* fixing imports

* Fix paths and add missing crates

* re-add missing crates
2019-11-14 21:51:17 +01:00