* Store the database in a role specific subdirectory
This is a cleaned up version of #8658 fixing #6880
polkadot companion: paritytech/polkadot#2923
* Disable prometheus in tests
* Also change p2p port
* Fix migration logic
* Use different identification file for rocks and parity db
Add tests for paritydb migration
Before we fetched the runtime code from the `TrieBackend` and this lead
to not using the storage cache. Thus, we recalculated the storage hash
for the runtime code on every call into the runtime and this killed the
performance on parachains block authoring. The solution is to fetch the
runtime code from the storage cache, to make sure we use the cached
storage cache.
* implement "auto" database backend in client/db, in progress, #9201
* move fn supports_ref_counting from DatabaseSource enum to Database trait to make it work correctly for all types of dbs
* update kvdb_rocksdb to 0.13 and use it's new config feature to properly auto start existing database
* tests for auto database reopening
* introduce OpenDbError to cleanup opening database error handling and handle case when database is not enabled at the compile time
* cargo fmt strings again
* cargo fmt strings again
* rename DataSettingsSrc to fix test compilation
* fix the call to the new kvdb-rocksdb interdace in tests to fix compilation
* simplify OpenDbError and make it compile even when paritydb and rocksdb are disabled
* cargo fmt
* fix compilation without flag with-parity-db
* fix unused var compilation warning
* support different paths for rocksdb and paritydb in DatabaseSouce::Auto
* support "auto" database option in substrate cli
* enable Lz4 compression for some of the parity-db colums as per review suggestion
* applied review suggestions
* Run cargo fmt on the whole code base
* Second run
* Add CI check
* Fix compilation
* More unnecessary braces
* Handle weights
* Use --all
* Use correct attributes...
* Fix UI tests
* AHHHHHHHHH
* 🤦
* Docs
* Fix compilation
* 🤷
* Please stop
* 🤦 x 2
* More
* make rustfmt.toml consistent with polkadot
Co-authored-by: André Silva <andrerfosilva@gmail.com>
* Removes unnecessary blank impl for Backend
This commit removes a from my perspective unneccessary implementation
for &T which implement Backend.
The current implementation exists (again from my perspective) solely
to satisfy a methods &mut self parameters (i.e. allows to satisfy
this for an & reference via using &mut &Backend).
As all implementors use a RefCell with borrow_mut() where actually
calling the mentioned &mut self method and then forwad to the
{} implementation of either TrieBackend or ProvingBackend, the
current &mut self seems to be not needed.
* Fixed tests client
* Fixes `storage_hash` caching issue and enables better caching for Cumulus
There was a caching issue with `storage_hash` that resulted in not
reverting cached storage hashes when required. In Cumulus this resulted
in nodes failing to import new blocks after a runtime upgrade, because
they were using the old runtime version.
Besides that, this pr optimizes for the Cumulus use case. In particular
that we always import blocks first as non-best blocks and enact them
later. In current version of the caching that would mean we would always
throw away the complete cache of the latest imported block. Now, we
always update the cache for the first block of a new block height. This
enables us to use the cache if this block will enacted as best block
later. If there is a fork and that is enacted as best, we revert all the
changes to the cache.
* Apply suggestions from code review
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Indentation
* Update client/db/src/storage_cache.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* primitives/runtime: initial changes on supporting multiple Justifications
* primitives/runtime: make Justifications strongly typed
* Encode/decode Justifications
* primitives/runtime: add Justification type
* backend: apply_finality and finalize_block takes a single Justification
* manual-seal: create engine id and let rpc take encoded justification
* backend: skeleton functions for appending justifications
* backend: initial implementation append_justification
Initial implementation of append_justification on the Backend trait, and also remove unused skeleton
functions for append_justificaton on Finaziler trait.
k
* backend: guard against duplicate consensus engine id
* client/db: add check for block finality
* client/api: add append_justification to in_mem db
* client/light: add no-op append_justification
* network: fix decode call for Justification
* network: only send a single Justification in BlockData
* network: minor comment update
* protocol: update field names to distinguish single justification
* client: further field renames to plural
* client: update function names to plural justifications
* client/db: upgrade existing database for new format
* network: remove dependency on grandpa crate
* db: fix check for finalized block
* grandpa: check for multiple grandpa justifications hwne importing
* backend: update Finalizer trait to take multiple Justifications
* db: remove debugging statements in migration code
* manual-seal: update note about engine id
* db: fix check for finalized block
* client: update variable name to reflect it is now plural
* grandpa: fix incorrect empty Justications in test
* primitives: make Justifications opaque to avoid being empty
* network: fix detecting empty Justification
* runtime: doc strings for Justifications functions
* runtime: add into_justifications
* primitives: check for duplicates in when adding to Justifications
* network/test: use real grandpa engine id in test
* client: fix reviewer comments
* primitives: rename Justifications::push to append
* backend: revert changes to Finalizer trait
* backend: revert mark_finalized
* backend: revert changes to finalize_block
* backend: revert finalized_blocks
* db: add a quick early return for performance
* client: minor reviewer comments
* service/test: use local ConsensusEngineId
* network: add link to issue for sending multiple Justifications
* Apply suggestions from code review
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Apply suggestions from code review
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* network: tweaks to review suggestions
* network: revert change to BlockData for backwards compatibility
* Apply suggestion from code review
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Apply suggestions from code review
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* primitives: update doc comment for Justifications
* client/db/upgrade: avoid grandpa crate dependency
* consensus: revert to single Justification for import_justification
* primitives: improve justifications docs
* style cleanups
* use and_then
* client: rename JUSTIFICATIONS db column
* network: revert to using FRNK in network-test
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: André Silva <andrerfosilva@gmail.com>
* Moving offchain change set to state machine overlay change set,
preparing use of change set internally.
* Make change set generic over key and value, and use it for offchain
indexing.
* test ui change
* remaining delta
* generating with standard method
* Remove 'drain_committed' function, and documentation.
* Default constructor for enabling offchain indexing.
* Remove offchain change specific iterators.
* remove pub accessor
* keep previous hierarchy, just expose iterator instead.
* Update primitives/state-machine/src/overlayed_changes/mod.rs
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* fix line break
* missing renamings
* fix import
* fix new state-machine tests.
* Don't expose InnerValue type.
* Add test similar to set_storage.
* Remove conditional offchain storage (hard to instantiate correctly).
* fix
* offchain as children cannot fail if top doesn't
Co-authored-by: Addie Wagenknecht <addie@nortd.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Make offchain indexing work
This fixes some bugs with offchain indexing to make it actually working ;)
* Fix tests
* Fix browser build
* Update client/db/src/offchain.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Remove seperation between prefix and key
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Allow Backend::for_keys_in_child_storage to be aborted by the closure
* Ext::kill_child_storage now takes an upper limit for backend deletion
* Add Storage::storage_kill_limited() runtime interface
* review: Use a new version of kill_storage instead of a new interface
* review: Simplify boolean expression
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* review: Rename for_keys_in_child_storage
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* chore/error: remove from str conversion and add deprecation notifications
* fixup changes
* fix test looking for gone ::Msg variant
* another test fix
* one is duplicate, the other is not, so duplicates reported are n-1
* darn spaces
Co-authored-by: Andronik Ordian <write@reusable.software>
* remove pointless doc comments of error variants without any value
* low hanging fruits (for a tall person)
* moar error type variants
* avoid the storage modules for now
They are in need of a refactor, and the pain is rather large
removing all String error and DefaultError occurences.
* chore remove pointless error generic
* fix test for mocks, add a bunch of non_exhaustive
* max line width
* test fixes due to error changes
* fin
* error outputs... again
* undo stderr adjustments
* Update client/consensus/slots/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* remove closure clutter
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* more error types
* introduce ApiError
* extract Mock error
* ApiError refactor
* even more error types
* the last for now
* chore unused deps
* another extraction
* reduce should panic, due to extended error messages
* error test happiness
* shift error lines by one
* doc tests
* white space
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Into -> From
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* remove pointless codec
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* avoid pointless self import
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@parity.io>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* init_logger: switch from log-based to tracing-based and add compatibility layer
* Move tracing profiling subscriber related config realization
* sp-tracing: change profiling to be a layer instead of a subscriber
* Enable profiling layer in cli
* Change all test env_logger init to sp_tracing::try_init_simple
* Remove all local env_logger dependency
* Add missing tracing-subscriber dependency
* frame-sudo: fix tests
* frame-support: fix tests
* Fix frame/pallet and executor tests
* Fix the remaining tests
* Use subscriber's try_init as recommended by @davidbarsky
* Be explict that the tracing-log feature is needed
* Set subscriber writer to stderr
* Shorter line width
* Update cargo lock tracing version
* Fix sc_tracing crate compile
* Fix sc_authority_discovery crate test
* unremove default-features
* Leave enabled to default true
* Warn if global default cannot be set
* Fix unused import
* Remove unused PROXY_TARGET
* Change all reference from rc5 to rc6
* Change all reference of rc2 to rc6
* Fix styling
* Fix typo
* make logger init error'ing
* re-fixing the test issue
Co-authored-by: Benjamin Kampmann <ben@parity.io>
* change (ci): docs job optimized; runs every commit; fails on warnings
* change (ci): rename jobs; temporary allow failing
* change (ci): better warnings filtering
* fix (ci): hotfix Docker release
* test (ci): run docs job with flags
* test (ci): pwd fails
* change (ci): pass just //doc dir as an artifact; debug
* change (ci): return to the previous structure; undebug
* change (ci): typo
* rebase on upstream 2
* fix the jobname
* Fix some warnings (#7079)
* Partial fix for transaction priority (#7034)
* Partial fix for priority stuff.
* Small fix
* Fix tests.
* Update frame/transaction-payment/src/lib.rs
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Better doc
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* What happens if we remove wat? (#7056)
* What happens if we remove wat?
* Update Cargo.lock
* Make SlashingSpans Public (#6961)
* Make SlashingSpans Public
Offchain Applications will often need to inspect this type because it is directly used in staking election, thus worthy of being `pub`. Rest of the slashing api can remain private, only this and the `fn last_non_zero_slash()` of `SlashingSpans` are of interest.
* Update frame/staking/src/lib.rs
* client/authority-discovery/src/service: Improve docs (#7059)
* Decrease poll interval (#7063)
* Remove unused code (#7027)
Signed-off-by: Jimmy Chu <jimmychu0807@gmail.com>
* Disambiguate `BlockNumber` type in `decl_module` (#7061)
* Disambiguate `BlockNumber` type in `decl_module`
* fix `frame-support-tests`
* fix ui tests
* fix trait order
* Implement `FromStr` for `Ss58AddressFormat` (#7068)
* Implement `FromStr` for `Ss58AddressFormat`
* Update primitives/core/src/crypto.rs
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
* Set reserved nodes with offchain worker. (#6996)
* add offchain worker api to set reserved nodes.
* new offchain api to get node public key.
* node public key from converter
* refactor set reserved nodes ocw api.
* new ndoe authorization pallet
* remove unnecessary clone and more.
* more
* tests for node authorization pallet
* remove dependency
* fix build
* more tests.
* refactor
* Update primitives/core/src/offchain/testing.rs
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Update frame/node-authorization/src/lib.rs
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Update frame/node-authorization/src/lib.rs
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Update frame/node-authorization/src/lib.rs
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* format code
* expose NetworkService
* remove NetworkStateInfo in offchain
* replace NodePublicKey with PeerId.
* set max length of peer id.
* clear more
* use BTreeSet for set of peers.
* decode opaque peer id.
* extract NetworkProvider for client offchain.
* use OpaquePeerId in node authorization pallet.
* fix test
* better documentation
* fix test
* doc
* more fix
* Update primitives/core/src/offchain/mod.rs
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Update client/offchain/src/api.rs
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* derive serialize and deserialize
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Fix some warnings
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: s3krit <pugh@s3kr.it>
Co-authored-by: Jimmy Chu <jimmy@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: kaichao <kaichaosuna@gmail.com>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Fix more doc errors
* More doc fixes
* Remove subdb to make `rustdoc` happy
* Make the line length check happy
* Fix compilation error
* Another try
* Allow unused
Co-authored-by: Dan Forbes <dan@danforbes.dev>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: s3krit <pugh@s3kr.it>
Co-authored-by: Jimmy Chu <jimmy@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: kaichao <kaichaosuna@gmail.com>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Generate CHT roots on a full client
* add changes_trie_root function
* Add a test
* Line widths
* Fix sc-service-test
* Clarify comments
* Revert comments
* Allow blacklisting blocks from being finalized again after block revert
* Use BlockRules for storing unfinalized and add have_state_at in revert
* Move finalization_check in finalize_block upward
* Directly mark finalization blacklist as badblocks
* Remove obselete comment
* make all features explicit
* Change to -feature suffix to with- prefix
* Add newline at the end of the Cargo.toml file
* Remove rhd feature
* Remove some features from Cargo.toml
* Remove test-helpers feature in tx pool
* Return db_open_error("with-"..
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Rename subdb feature to with-subdb
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Remove 'strict' feature and cfg_attr
* Check for with-subdb feature instead of subdb
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Don't clone values when calculating storage root
Instead of cloning all the keys and values of the overlay when
calculating the storage root, we pass all the values by reference. This
should probably bring some performance improvements when calculating the
storage root.
* no cow version (#6113)
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Drop client from sc-network and sc-client-db, move LongestChain to sc-client-api
* move leaves, cht, in_mem to sc-client-api, drop client from sc-finality-grandpa
* drop sc-service from sc-rpc
* drop sc-service from sc-consensus-aura
* drop sc-client from manual-seal and babe
* drop sc-client from utils/frame/rpc/system and utils/frame/benchmarking-cli
* drop sc-client from bin/node and bin/node-template
* drop sc-client
* fix tests
* remove check -p sc-client from gitlab.yml
* fix warnings
* fixes ui test
* fix light client tests
* adds associated Client type to AbstractService
* adds UsageProvider to Client
* fixed ui test, again
* tried and failed to get node-cli to compile for wasm
* thanks to tomaka for helping me get node-cli to compile for wasmm
* ui test pls pas 🙏🏾
* all tests passing 🪄
* no_run documentation code
* rm -f documentation code
* ClientProvider
* fix mega trait
* move LongestChain to sc-consensus, use adds minimal bounds to AbstractService::Client
* adds license to sc-consensus
Co-authored-by: Benjamin Kampmann <ben@parity.io>
Falco Hirschenberger