Original PR https://github.com/paritytech/substrate/pull/14746
---
## Fixing stall
### Introduction
I experienced an apparent stall downloading state from
`https://rococo-try-runtime-node.parity-chains.parity.io:443` which was
having networking difficulties only responding to my JSONRPC requests
with 50-200KB/s of bandwidth.
This PR fixes the issue causing the stall, and generally improves
performance remote-ext when it downloads state by greatly reducing the
chances of a timeout occuring.
### Description
Introduces a new `REQUEST_DURATION_TARGET` constant and modifies
`get_storage_data_dynamic_batch_size` to
- Increase or decrease the batch size of the next request depending on
whether the elapsed time of the last request was gt or lt the target
- Reset the batch size to 1 if the request times out
This fixes an issue on slow connections that can otherwise cause
multiple timeouts and a stalled download when:
1. The batch size increases rapidly as remote-ext downloads keys with
small associated storage values
2. remote-ext tries to process a large series of subsequent keys all
with extremely large associated storage values (Rococo has a series of
keys 1-5MB large)
3. The huge storage values download for 5 minutes until the request
times out
4. The partially downloaded keys are thrown out and remote-ext tries
again with a smaller batch size, but the batch size is still far too
large and takes 5 minutes to be reduced again
5. The download will be essentially stalled for many hours while the
above step cycles
After this PR, the request size will
- Not grow as large to begin with, as it is regulated downwards as the
request duration exceeds the target
- Drop immediately to 1 if the request times out. A timeout indicates
the keys next in line to download have extremely large storage values
compared to previously downloaded keys, and we need to reset the batch
size to figure out what our new ideal batch size is. By not resetting
down to 1, we risk the next request timing out again.
## Reducing memory
As suggested by @bkchr, I adjusted `get_storage_data_dynamic_batch_size`
from being recursive to a loop which allows removing a bunch of clones
that were chewing through a lot of memory. I noticed actually it was
using up to 50GB swap previously when downloading Polkadot keys on a
slow connection, because it needed to recurse and clone a lot.
After this change it uses only ~1.5GB memory.
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
* fix online/offline confusion
* unified cache file
* multi-threaded babyyy
* checkpoint for niklas
* compiles
* all tests pass with --test-threads 1
* child-tree scrape is also multi-threaded now.
* better thread splitting
* some suggestions (#12532)
* some suggestions
* tokio multithread
* move unused dependencies
* snapshot command
* fix rem
* a bit of cleanup
* support optional checks
* fix
* OCW command migrated to wasm-only, as an example
* state-version management fully in remote-ext
* almost everything move to wasm executor, some CLI flags reduced
* follow-chain works as well
* Master.into()
* everything builds now
* concurrent insertion and download for remote builds
* minor fix
* fix a bug
* checkpoint
* some updates
* fmt
* review comments
* fmt
* fix
* fmt
* update
* fmt
* rename
* fix the damn UI tests
* fmt
* remoe the thread abstraction for the time being
* cleanup
* fix CI
* fmt
* fix
* fix a few more things
* tweak log levels
* better error handling
* address grumbles: use futures::mpsc
* review comments
* fmt
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update utils/frame/try-runtime/cli/src/lib.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* better api version stuff
* some doc update
* a whole lot of docs
* fmt
* fix all docs
* fmt
* rpc rebase: Try-runtime Revamp and Facelift (#12921)
* Introduce sensible weight constants (#12868)
* Introduce sensible weight constants
* cargo fmt
* Remove unused import
* Add missing import
* ".git/.scripts/bench-bot.sh" pallet dev pallet_lottery
Co-authored-by: command-bot <>
* Checkout to the branch HEAD explicitly in `build-linux-substrate` (#12876)
* cli: Improve pruning documentation (#12819)
* cli: Improve pruning documentation
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Keep `finalized` notation and remove `canonical` one
* cli: Fix cargo doc
* cli: `PruningModeClap` IR enum
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Convert PruningModeClap into pruning modes
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Use `PruningModeClap`
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Rename to `DatabasePruningMode`
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Implement `FromStr` instead of `clap::ValueEnum`
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Update client/cli/src/params/pruning_params.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* Fix clippy
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Add option documentation back
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Apply suggestions from code review
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Revert "Move LockableCurrency trait to fungibles::Lockable and deprecate LockableCurrency (#12798)" (#12882)
This reverts commit 9a014d1ecd.
* Don't indefinitely block on shutting down Tokio (#12885)
* Don't indefinitely on shutting down Tokio
Now we wait in maximum 60 seconds before we shutdown the node. Tasks are may be leaked and leading
to some data corruption.
* Drink less :thinking_face:
* General Message Queue Pallet (#12485)
* The message queue
* Make fully generic
* Refactor
* Docs
* Refactor
* Use iter not slice
* Per-origin queues
* Multi-queue processing
* Introduce MaxReady
* Remove MaxReady in favour of ready ring
* Cleanups
* ReadyRing and tests
* Stale page reaping
* from_components -> from_parts
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Move WeightCounter to sp_weights
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add MockedWeightInfo
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Deploy to kitchensink
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Use WeightCounter
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Small fixes and logging
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add service_page
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Typo
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Move service_page below service_queue
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add service_message
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Use correct weight function
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Overweight execution
* Refactor
* Missing file
* Fix WeightCounter usage in scheduler
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix peek_index
Take into account that decoding from a mutable slice modifies it.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add tests and bench service_page_item
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add debug_info
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add no-progress check to service_queues
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add more benches
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Bound from_message and try_append_message
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add PageReaped event
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Rename BookStateOf and BookStateFor
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update tests and remove logging
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Remove redundant per-message origins; add footprint() and sweep_queue()
* Move testing stuff to mock.rs
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add integration test
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix no-progress check
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix debug_info
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fixup merge and tests
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix footprint tracking
* Introduce
* Formatting
* OverweightEnqueued event, auto-servicing config item
* Update tests and benchmarks
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Clippy
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add tests
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Provide change handler
* Add missing BookStateFor::insert and call QueueChangeHandler
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Docs
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update benchmarks and weights
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* More tests...
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Use weight metering functions
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* weightInfo::process_message_payload is gone
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add defensive_saturating_accrue
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Rename WeightCounter to WeightMeter
Ctr+Shift+H should do the trick.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Test on_initialize
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add module docs
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Remove origin from MaxMessageLen
The message origin is not encoded into the heap and does
therefore not influence the max message length anymore.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add BoundedVec::as_slice
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Test Page::{from_message, try_append_message}
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fixup docs
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Docs
* Do nothing in sweep_queue if the queue does not exist
... otherwise it inserts default values into the storage.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Test ring (un)knitting
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Upgrade stress-test
Change the test to not assume that all queued messages will be
processed in the next block but split it over multiple.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* More tests...
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Beauty fixes
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* clippy
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Rename BoundedVec::as_slice to as_bounded_slice
Conflicts with deref().as_slice() otherwise.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix imports
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Remove ReadyRing struct
Was used for testing only. Instead use 'fn assert_ring' which also
check the service head and backlinks.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Beauty fixes
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix stale page watermark
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Cleanup
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix test feature and clippy
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* QueueChanged handler is called correctly
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update benches
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Abstract testing functions
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* More tests
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Cleanup
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Clippy
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* fmt
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Simplify tests
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Make stuff compile
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Extend overweight execution benchmark
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Remove TODOs
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Test service queue with faulty MessageProcessor
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* fmt
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update pallet ui tests to 1.65
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* More docs
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Review doc fixes
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add weight_limit to extrinsic weight of execute_overweight
* Correctly return unused weight
* Return actual weight consumed in do_execute_overweight
* Review fixes
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Set version 7.0.0-dev
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Make it compile
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Switch message_size to u64
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Switch message_count to u64
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix benchmarks
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Make CI green
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Docs
* Update tests
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* ".git/.scripts/bench-bot.sh" pallet dev pallet_message_queue
* Dont mention README.md in the Cargo.toml
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Remove reference to readme
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: parity-processbot <>
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
* zombienet timings adjusted (#12890)
* zombinet tests: add some timeout to allow net spin-up
Sometimes tests are failing at first try, as the pods were not up yet.
Adding timeout should allow the network to spin up properly.
* initial timeout increased to 30s
* Move import queue out of `sc-network` (#12764)
* Move import queue out of `sc-network`
Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.
This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.
* Fix tests
* Apply review comments
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update client/network/sync/src/lib.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Trace response payload in default `jsonrpsee` middleware (#12886)
* Trace result in default `jsonrpsee` middleware
* `rpc_metrics::extra`
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Ensure that we inform all tasks to stop before starting the 60 seconds shutdown (#12897)
* Ensure that we inform all tasks to stop before starting the 60 seconds shutdown
The change of waiting in maximum 60 seconds for the node to shutdown actually introduced a bug. We
were actually waiting always 60 seconds as we didn't informed our tasks to shutdown. The solution to
this problem is to drop the task manager as this will then inform all tasks to end. It also adds
tests to ensure that the behaviors work as expected. (This should already have been done in the
first pr! :()
* ".git/.scripts/fmt.sh" 1
Co-authored-by: command-bot <>
* Safe desired targets call (#12826)
* checked call for desired targets
* fix compile
* fmt
* fix tests
* cleaner with and_then
* Fix typo (#12900)
* ValidateUnsigned: Improve docs. (#12870)
* ValidateUnsigned: Improve docs.
* Review comments
* rpc server with HTTP/WS on the same socket (#12663)
* jsonrpsee v0.16
add backwards compatibility
run old http server on http only
* cargo fmt
* update jsonrpsee 0.16.1
* less verbose cors log
* fix nit in log: WS -> HTTP
* revert needless changes in Cargo.lock
* remove unused features in tower
* fix nits; add client-core feature
* jsonrpsee v0.16.2
* `pallet-message-queue`: Fix license (#12895)
* Fix license
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add mock doc
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Use explicit call indices (#12891)
* frame-system: explicit call index
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Use explicit call indices
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* pallet-template: explicit call index
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* DNM: Temporarily require call_index
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Revert "DNM: Temporarily require call_index"
This reverts commit c4934e312e12af72ca05a8029d7da753a9c99346.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Pin canonincalized block (#12902)
* Remove implicit approval chilling upon slash. (#12420)
* don't read slashing spans when taking election snapshot
* update cargo.toml
* bring back remote test
* fix merge stuff
* fix npos-voters function sig
* remove as much redundant diff as you can
* Update frame/staking/src/pallet/mod.rs
Co-authored-by: Andronik <write@reusable.software>
* fix
* Update frame/staking/src/pallet/impls.rs
* update lock
* fix all tests
* review comments
* fmt
* fix offence bench
* clippy
* ".git/.scripts/bench-bot.sh" pallet dev pallet_staking
Co-authored-by: Andronik <write@reusable.software>
Co-authored-by: Ankan <ankan.anurag@gmail.com>
Co-authored-by: command-bot <>
* bounties calls docs fix (#12909)
Co-authored-by: parity-processbot <>
* pallet-contracts migration pre-upgrade fix for v8 (#12905)
* Only run pre-v8 migration check for versions older than 8
* Logix fix
* use custom environment for publishing crates (#12912)
* [contracts] Add debug buffer limit + enforcement (#12845)
* Add debug buffer limit + enforcement
Add debug buffer limit + enforcement
* use BoundedVec for the debug buffer
* revert schedule (debug buf len limit not needed anymore)
* return DispatchError
* addressed review comments
* frame/remote-externalities: Fix clippy
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* frame/rpc: Add previous export
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Fixup some wrong dependencies (#12899)
* Fixup some wrong dependencies
Dev dependencies should not appear in the feature list. If features are required, they should be
directly enabled for the `dev-dependency`.
* More fixups
* Fix fix
* Remove deprecated feature
* Make all work properly and nice!!
* FMT
* Fix formatting
* add numerator and denominator to Rational128 Debug impl and increase precision of float representation (#12914)
* Fix state-db pinning (#12927)
* Pin all canonicalized blocks
* Added a test
* Docs
* [ci] add job switcher (#12922)
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Co-authored-by: Vlad <vladimir@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com>
Co-authored-by: Gavin Wood <gavin@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
Co-authored-by: tgmichel <telmo@purestake.com>
Co-authored-by: Ankan <10196091+Ank4n@users.noreply.github.com>
Co-authored-by: Luke Schoen <ltfschoen@users.noreply.github.com>
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Andronik <write@reusable.software>
Co-authored-by: Ankan <ankan.anurag@gmail.com>
Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com>
Co-authored-by: Dino Pačandi <3002868+Dinonard@users.noreply.github.com>
Co-authored-by: João Paulo Silva de Souza <77391175+joao-paulo-parity@users.noreply.github.com>
Co-authored-by: Sasha Gryaznov <hi@agryaznov.com>
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
* Revert "rpc rebase: Try-runtime Revamp and Facelift (#12921)"
This reverts commit 4ce770a9cb8daf1401529bda7d974b8c703f6b3e.
* Lexnv/kiz revamp try runtime stuff (#12932)
* Introduce sensible weight constants (#12868)
* Introduce sensible weight constants
* cargo fmt
* Remove unused import
* Add missing import
* ".git/.scripts/bench-bot.sh" pallet dev pallet_lottery
Co-authored-by: command-bot <>
* Checkout to the branch HEAD explicitly in `build-linux-substrate` (#12876)
* cli: Improve pruning documentation (#12819)
* cli: Improve pruning documentation
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Keep `finalized` notation and remove `canonical` one
* cli: Fix cargo doc
* cli: `PruningModeClap` IR enum
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Convert PruningModeClap into pruning modes
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Use `PruningModeClap`
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Rename to `DatabasePruningMode`
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Implement `FromStr` instead of `clap::ValueEnum`
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Update client/cli/src/params/pruning_params.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* Fix clippy
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* cli: Add option documentation back
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Apply suggestions from code review
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Revert "Move LockableCurrency trait to fungibles::Lockable and deprecate LockableCurrency (#12798)" (#12882)
This reverts commit 9a014d1ecd.
* Don't indefinitely block on shutting down Tokio (#12885)
* Don't indefinitely on shutting down Tokio
Now we wait in maximum 60 seconds before we shutdown the node. Tasks are may be leaked and leading
to some data corruption.
* Drink less :thinking_face:
* General Message Queue Pallet (#12485)
* The message queue
* Make fully generic
* Refactor
* Docs
* Refactor
* Use iter not slice
* Per-origin queues
* Multi-queue processing
* Introduce MaxReady
* Remove MaxReady in favour of ready ring
* Cleanups
* ReadyRing and tests
* Stale page reaping
* from_components -> from_parts
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Move WeightCounter to sp_weights
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add MockedWeightInfo
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Deploy to kitchensink
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Use WeightCounter
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Small fixes and logging
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add service_page
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Typo
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Move service_page below service_queue
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add service_message
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Use correct weight function
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Overweight execution
* Refactor
* Missing file
* Fix WeightCounter usage in scheduler
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix peek_index
Take into account that decoding from a mutable slice modifies it.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add tests and bench service_page_item
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add debug_info
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add no-progress check to service_queues
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add more benches
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Bound from_message and try_append_message
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add PageReaped event
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Rename BookStateOf and BookStateFor
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update tests and remove logging
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Remove redundant per-message origins; add footprint() and sweep_queue()
* Move testing stuff to mock.rs
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add integration test
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix no-progress check
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix debug_info
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fixup merge and tests
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix footprint tracking
* Introduce
* Formatting
* OverweightEnqueued event, auto-servicing config item
* Update tests and benchmarks
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Clippy
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add tests
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Provide change handler
* Add missing BookStateFor::insert and call QueueChangeHandler
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Docs
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update benchmarks and weights
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* More tests...
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Use weight metering functions
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* weightInfo::process_message_payload is gone
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add defensive_saturating_accrue
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Rename WeightCounter to WeightMeter
Ctr+Shift+H should do the trick.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Test on_initialize
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add module docs
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Remove origin from MaxMessageLen
The message origin is not encoded into the heap and does
therefore not influence the max message length anymore.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add BoundedVec::as_slice
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Test Page::{from_message, try_append_message}
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fixup docs
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Docs
* Do nothing in sweep_queue if the queue does not exist
... otherwise it inserts default values into the storage.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Test ring (un)knitting
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Upgrade stress-test
Change the test to not assume that all queued messages will be
processed in the next block but split it over multiple.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* More tests...
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Beauty fixes
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* clippy
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Rename BoundedVec::as_slice to as_bounded_slice
Conflicts with deref().as_slice() otherwise.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix imports
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Remove ReadyRing struct
Was used for testing only. Instead use 'fn assert_ring' which also
check the service head and backlinks.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Beauty fixes
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix stale page watermark
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Cleanup
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix test feature and clippy
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* QueueChanged handler is called correctly
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update benches
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Abstract testing functions
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* More tests
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Cleanup
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Clippy
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* fmt
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Simplify tests
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Make stuff compile
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Extend overweight execution benchmark
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Remove TODOs
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Test service queue with faulty MessageProcessor
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* fmt
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update pallet ui tests to 1.65
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* More docs
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Review doc fixes
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add weight_limit to extrinsic weight of execute_overweight
* Correctly return unused weight
* Return actual weight consumed in do_execute_overweight
* Review fixes
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Set version 7.0.0-dev
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Make it compile
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Switch message_size to u64
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Switch message_count to u64
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix benchmarks
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Make CI green
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Docs
* Update tests
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* ".git/.scripts/bench-bot.sh" pallet dev pallet_message_queue
* Dont mention README.md in the Cargo.toml
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Remove reference to readme
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: parity-processbot <>
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
* zombienet timings adjusted (#12890)
* zombinet tests: add some timeout to allow net spin-up
Sometimes tests are failing at first try, as the pods were not up yet.
Adding timeout should allow the network to spin up properly.
* initial timeout increased to 30s
* Move import queue out of `sc-network` (#12764)
* Move import queue out of `sc-network`
Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.
This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.
* Fix tests
* Apply review comments
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update client/network/sync/src/lib.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Trace response payload in default `jsonrpsee` middleware (#12886)
* Trace result in default `jsonrpsee` middleware
* `rpc_metrics::extra`
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Ensure that we inform all tasks to stop before starting the 60 seconds shutdown (#12897)
* Ensure that we inform all tasks to stop before starting the 60 seconds shutdown
The change of waiting in maximum 60 seconds for the node to shutdown actually introduced a bug. We
were actually waiting always 60 seconds as we didn't informed our tasks to shutdown. The solution to
this problem is to drop the task manager as this will then inform all tasks to end. It also adds
tests to ensure that the behaviors work as expected. (This should already have been done in the
first pr! :()
* ".git/.scripts/fmt.sh" 1
Co-authored-by: command-bot <>
* Safe desired targets call (#12826)
* checked call for desired targets
* fix compile
* fmt
* fix tests
* cleaner with and_then
* Fix typo (#12900)
* ValidateUnsigned: Improve docs. (#12870)
* ValidateUnsigned: Improve docs.
* Review comments
* rpc server with HTTP/WS on the same socket (#12663)
* jsonrpsee v0.16
add backwards compatibility
run old http server on http only
* cargo fmt
* update jsonrpsee 0.16.1
* less verbose cors log
* fix nit in log: WS -> HTTP
* revert needless changes in Cargo.lock
* remove unused features in tower
* fix nits; add client-core feature
* jsonrpsee v0.16.2
* `pallet-message-queue`: Fix license (#12895)
* Fix license
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add mock doc
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Use explicit call indices (#12891)
* frame-system: explicit call index
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Use explicit call indices
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* pallet-template: explicit call index
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* DNM: Temporarily require call_index
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Revert "DNM: Temporarily require call_index"
This reverts commit c4934e312e12af72ca05a8029d7da753a9c99346.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Pin canonincalized block (#12902)
* Remove implicit approval chilling upon slash. (#12420)
* don't read slashing spans when taking election snapshot
* update cargo.toml
* bring back remote test
* fix merge stuff
* fix npos-voters function sig
* remove as much redundant diff as you can
* Update frame/staking/src/pallet/mod.rs
Co-authored-by: Andronik <write@reusable.software>
* fix
* Update frame/staking/src/pallet/impls.rs
* update lock
* fix all tests
* review comments
* fmt
* fix offence bench
* clippy
* ".git/.scripts/bench-bot.sh" pallet dev pallet_staking
Co-authored-by: Andronik <write@reusable.software>
Co-authored-by: Ankan <ankan.anurag@gmail.com>
Co-authored-by: command-bot <>
* bounties calls docs fix (#12909)
Co-authored-by: parity-processbot <>
* pallet-contracts migration pre-upgrade fix for v8 (#12905)
* Only run pre-v8 migration check for versions older than 8
* Logix fix
* use custom environment for publishing crates (#12912)
* [contracts] Add debug buffer limit + enforcement (#12845)
* Add debug buffer limit + enforcement
Add debug buffer limit + enforcement
* use BoundedVec for the debug buffer
* revert schedule (debug buf len limit not needed anymore)
* return DispatchError
* addressed review comments
* frame/remote-externalities: Fix clippy
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* frame/rpc: Add previous export
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* Fixup some wrong dependencies (#12899)
* Fixup some wrong dependencies
Dev dependencies should not appear in the feature list. If features are required, they should be
directly enabled for the `dev-dependency`.
* More fixups
* Fix fix
* Remove deprecated feature
* Make all work properly and nice!!
* FMT
* Fix formatting
* add numerator and denominator to Rational128 Debug impl and increase precision of float representation (#12914)
* Fix state-db pinning (#12927)
* Pin all canonicalized blocks
* Added a test
* Docs
* [ci] add job switcher (#12922)
* Use LOG_TARGET in consensus related crates (#12875)
* Use shared LOG_TARGET in consensus related crates
* Rename target from "afg" to "grandpa"
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Co-authored-by: Vlad <vladimir@parity.io>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com>
Co-authored-by: Gavin Wood <gavin@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
Co-authored-by: tgmichel <telmo@purestake.com>
Co-authored-by: Ankan <10196091+Ank4n@users.noreply.github.com>
Co-authored-by: Luke Schoen <ltfschoen@users.noreply.github.com>
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
Co-authored-by: Andronik <write@reusable.software>
Co-authored-by: Ankan <ankan.anurag@gmail.com>
Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com>
Co-authored-by: Dino Pačandi <3002868+Dinonard@users.noreply.github.com>
Co-authored-by: João Paulo Silva de Souza <77391175+joao-paulo-parity@users.noreply.github.com>
Co-authored-by: Sasha Gryaznov <hi@agryaznov.com>
Co-authored-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
Co-authored-by: Davide Galassi <davxy@datawok.net>
* Revert "Lexnv/kiz revamp try runtime stuff (#12932)"
This reverts commit 378cfb26d984bcde467781f07ef8ddb6998212cb.
* fmt
* update
* fix publish
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Co-authored-by: Vlad <vladimir@parity.io>
Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com>
Co-authored-by: Gavin Wood <gavin@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
Co-authored-by: tgmichel <telmo@purestake.com>
Co-authored-by: Ankan <10196091+Ank4n@users.noreply.github.com>
Co-authored-by: Luke Schoen <ltfschoen@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
Co-authored-by: Andronik <write@reusable.software>
Co-authored-by: Ankan <ankan.anurag@gmail.com>
Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com>
Co-authored-by: Dino Pačandi <3002868+Dinonard@users.noreply.github.com>
Co-authored-by: João Paulo Silva de Souza <77391175+joao-paulo-parity@users.noreply.github.com>
Co-authored-by: Sasha Gryaznov <hi@agryaznov.com>
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
Co-authored-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Davide Galassi <davxy@datawok.net>
* jsonrpsee v0.16
add backwards compatibility
run old http server on http only
* cargo fmt
* update jsonrpsee 0.16.1
* less verbose cors log
* fix nit in log: WS -> HTTP
* revert needless changes in Cargo.lock
* remove unused features in tower
* fix nits; add client-core feature
* jsonrpsee v0.16.2
* add missing version to dependencies
* Huh
* add features more
* more fixing
* last touches
* it all finally works
* remove some feature gates
* remove unused
* fix old macro
* make it work again
* fmt
* remove unused import
* ".git/.scripts/fmt.sh" 1
* Cleanup more
* fix and rename everything
* a few clippy fixes
* Add try-runtime feature
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* small fixes
* fmt
* Update bin/node-template/runtime/src/lib.rs
* fix build
* Update utils/frame/try-runtime/cli/src/lib.rs
Co-authored-by: David <dvdplm@gmail.com>
* Update utils/frame/try-runtime/cli/src/commands/execute_block.rs
Co-authored-by: David <dvdplm@gmail.com>
* address all review comments
* fix typos
* revert spec change
* last touches
* update docs
* fmt
* remove some debug_assertions
* fmt
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: David <dvdplm@gmail.com>
* Update UI test output for rust 1.62.1
* switch ci to staging image to check that everything works
* fix artifacts node-bench-regression-guard
* Imeplement `scale_info::TypeInfo` manually to silence aggressive rust warning
* Fix more clippy lints
* Make clippy happy by relying on auto-deref were possible
* Add tracking issue to the comments
* pin ci image
Co-authored-by: alvicsam <alvicsam@gmail.com>
* The 2022-02-10 nightly can't build some deps
There's a sig that returns `impl IntoIterator<Item = (&'static str, OsString)>` and an Option is being retured but it's incorrectly not allowing a `?` in the method.
* removing duplicate test
* [ci] add job cargo-check-nixos
* add dummy variables nix check
* fix check-dependent jobs
* fix check-dependent-project template
Co-authored-by: alvicsam <alvicsam@gmail.com>
* Prepare for rust stable 1.59
Besides preparing the UI tests this also adds a new script update-rust-stable.sh script for
simplifying the update of a rust stable version. This script will run all UI tests for the new
rust stable version and updating the expected output.
* Ensure we run the UI tests in CI
* use staging ci image
* More test updates
* Unignore test (#11097)
* empty commit for pipeline rerun
* empty commit for pipeline rerun
* Try to make clippy happy
* More clippy fixes
* FMT
* ci image production
Co-authored-by: alvicsam <alvicsam@gmail.com>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
* starting
* Updated from other branch.
* setting flag
* flag in storage struct
* fix flagging to access and insert.
* added todo to fix
* also missing serialize meta to storage proof
* extract meta.
* Isolate old trie layout.
* failing test that requires storing in meta when old hash scheme is used.
* old hash compatibility
* Db migrate.
* runing tests with both states when interesting.
* fix chain spec test with serde default.
* export state (missing trie function).
* Pending using new branch, lacking genericity on layout resolution.
* extract and set global meta
* Update to branch 4
* fix iterator with root flag (no longer insert node).
* fix trie root hashing of root
* complete basic backend.
* Remove old_hash meta from proof that do not use inner_hashing.
* fix trie test for empty (force layout on empty deltas).
* Root update fix.
* debug on meta
* Use trie key iteration that do not include value in proofs.
* switch default test ext to use inner hash.
* small integration test, and fix tx cache mgmt in ext.
test failing
* Proof scenario at state-machine level.
* trace for db upgrade
* try different param
* act more like iter_from.
* Bigger batches.
* Update trie dependency.
* drafting codec changes and refact
* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.
* alt hashing only for branch with value.
* fix trie tests
* Hash of value include the encoded size.
* removing fields(broken)
* fix trie_stream to also include value length in inner hash.
* triedbmut only using alt type if inner hashing.
* trie_stream to also only use alt hashing type when actually alt hashing.
* Refactor meta state, logic should work with change of trie treshold.
* Remove NoMeta variant.
* Remove state_hashed trigger specific functions.
* pending switching to using threshold, new storage root api does not
make much sense.
* refactoring to use state from backend (not possible payload changes).
* Applying from previous state
* Remove default from storage, genesis need a special build.
* rem empty space
* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.
* fix compilation
* Right logic to avoid switch on reencode when default layout.
* Clean up some todos
* remove trie meta from root upstream
* update upstream and fix benches.
* split some long lines.
* UPdate trie crate to work with new design.
* Finish update to refactored upstream.
* update to latest triedb changes.
* Clean up.
* fix executor test.
* rust fmt from master.
* rust format.
* rustfmt
* fix
* start host function driven versioning
* update state-machine part
* still need access to state version from runtime
* state hash in mem: wrong
* direction likely correct, but passing call to code exec for genesis
init seem awkward.
* state version serialize in runtime, wrong approach, just initialize it
with no threshold for core api < 4 seems more proper.
* stateversion from runtime version (core api >= 4).
* update trie, fix tests
* unused import
* clean some TODOs
* Require RuntimeVersionOf for executor
* use RuntimeVersionOf to resolve genesis state version.
* update runtime version test
* fix state-machine tests
* TODO
* Use runtime version from storage wasm with fast sync.
* rustfmt
* fmt
* fix test
* revert useless changes.
* clean some unused changes
* fmt
* removing useless trait function.
* remove remaining reference to state_hash
* fix some imports
* Follow chain state version management.
* trie update, fix and constant threshold for trie layouts.
* update deps
* Update to latest trie pr changes.
* fix benches
* Verify proof requires right layout.
* update trie_root
* Update trie deps to latest
* Update to latest trie versioning
* Removing patch
* update lock
* extrinsic for sc-service-test using layout v0.
* Adding RuntimeVersionOf to CallExecutor works.
* fmt
* error when resolving version and no wasm in storage.
* use existing utils to instantiate runtime code.
* migration pallet
* Patch to delay runtime switch.
* Revert "Patch to delay runtime switch."
This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da.
* fix test
* fix child migration calls.
* useless closure
* remove remaining state_hash variables.
* Fix and add more tests
* Remove outdated comment
* useless inner hash
* fmt
* remote tests
* finally ksm works
* batches are broken
* clean the benchmarks
* Apply suggestions from code review
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
* Apply suggestions from code review
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
* Update frame/state-trie-migration/src/lib.rs
Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>
* Update frame/state-trie-migration/src/lib.rs
* brand new version
* fix build
* Update frame/state-trie-migration/src/lib.rs
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
* Update frame/state-trie-migration/src/lib.rs
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
* Update primitives/storage/src/lib.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Update frame/state-trie-migration/src/lib.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Update frame/state-trie-migration/src/lib.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* fmt and opt-in feature to apply state change.
* feature gate core version, use new test feature for node and test node
* Use a 'State' api version instead of Core one.
* fix merge of test function
* use blake macro.
* Fix state api (require declaring the api in runtime).
* Opt out feature, fix macro for io to select a given version
instead of latest.
* run test nodes on new state.
* fix
* new test structure
* new testing stuff from emeric
* Add commit_all, still not working
* Fix all tests
* add comment
* we have PoV tracking baby
* document stuff, but proof size is still wrong
* FUCK YEAH
* a big batch of review comments
* add more tests
* tweak test
* update config
* some remote-ext stuff
* delete some of the old stuff
* sync more files with master to minimize the diff
* Fix all tests
* make signed migration a bit more relaxed
* add witness check to signed submissions
* allow custom migration to also go above limit
* Fix these pesky tests
* ==== removal of the unsigned stuff ====
* Make all tests work again
* separate the tests from the logic so it can be reused easier
* fix overall build
* Update frame/state-trie-migration/src/lib.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Update frame/state-trie-migration/src/lib.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Slightly better termination
* some final tweaks
* Fix tests
* Restrict access to signed migrations
* address most of the review comments
* fix defensive
* New simplified code
* Fix weights
* fmt
* Update frame/state-trie-migration/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* make the tests correctly fail
* Fix build
* Fix build
* try and fix the benchmarks
* fix build
* Fix cargo file
* Fix runtime deposit
* make rustdoc happy
* cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_state_trie_migration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/state-trie-migration/src/weights.rs --template=./.maintain/frame-weight-template.hbs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Parity Bot <admin@parity.io>
* deadlock, need to ask someone to help now
* Finally it seems to be working.. at least for a few blocks
* self-review
* major mega revamp
* some small fixes
* another mega refactor
* add license
* Apply suggestions from code review
* hack around signature verification
* Some fixes
* Update utils/frame/try-runtime/cli/src/lib.rs
Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>
* Update utils/frame/try-runtime/cli/src/lib.rs
Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>
* Update utils/frame/try-runtime/cli/src/lib.rs
Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>
* final tweaks, hopefully.
* a little self-review
* Add the ext root check
Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>
* Run cargo fmt on the whole code base
* Second run
* Add CI check
* Fix compilation
* More unnecessary braces
* Handle weights
* Use --all
* Use correct attributes...
* Fix UI tests
* AHHHHHHHHH
* 🤦
* Docs
* Fix compilation
* 🤷
* Please stop
* 🤦 x 2
* More
* make rustfmt.toml consistent with polkadot
Co-authored-by: André Silva <andrerfosilva@gmail.com>
* make remote ext use batch ws-client
* Add debug log for key length
* better assertions
* new sanity_checl
* try and make it work with batch
* update test
* remove exctra uri
* add missing at
* remove unused rpc stuff
* improve
Co-authored-by: emostov <32168567+emostov@users.noreply.github.com>
* make builder generic to allow using different hash types
* expose "cache", "block_number" and "modules" as cli options for live state
* Change Builder to be generic over Block instead of Hash
add rpc method to get hash from block number
allow passing of block numbers and hashes
* fix live tests
* fix formatting in utils/frame/remote-externalities/src/lib.rs
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
* change cli to only accept block hashes
break up lines that were too long
use starts_with instead of match s.get
use unwrap_or_default instead of unwrap_or(Vec::new())
* improve error message
* fix indentation
* replace Block with sp_runtime::testing::Block
* Move cache test out of remote-test feature tests
Add cache file (contains only "Proxy" module) for local test
* simplify match expression to and_then
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
* Combine the two cfg attributes into one
Co-authored-by: David <dvdplm@gmail.com>
* Restrict visibility of test_prelude use statements to crate level
* Fix usage of and_then
* Rename cache to snapshot
* Remove fully qualified path for Debug
* Refine naming. snapshot to state_snapshot
* Remove unnecessary comment
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: David <dvdplm@gmail.com>
* Use 'Pallet' struct in construct_runtime.
* Fix genesis and metadata macro.
* Fix 'Pallet' type alias.
* Replace 'Module' with 'Pallet' for all construct_runtime use cases.
* Replace more deprecated 'Module' struct.
* Bring back AllModules and AllPalletsWithSystem type, but deprecate them.
* Replace deprecated 'Module' struct from merge master.
* Minor fix.
* Fix UI tests.
* Revert UI override in derive_no_bound.
* Fix more deprecated 'Module' use from master branch.
* Fix more deprecated 'Module' use from master branch.
* A clean new attempt
* Checkpoint to move remote.
* A lot of dependency wiring to make it feature gated.
* bad macro, bad macro.
* refactor(remote ext): use jsonrpsee
* refactor(remote ext): use jsonrpsee
* Undo the DB mess.
* fix(remote ext): use max limit `u32::MAX`
* resolve TODOs
* jsonrpsee switch to `hyper` as backend
* Update utils/frame/try-runtime/remote-externalities/src/lib.rs
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
* update jsonrpsee
* remove boiler-plate
* suppress warnings to CI happy
* Unbreak his build
* Use option
* fix nit; make it work again
* fix err message.
* Update utils/frame/remote-externalities/Cargo.toml
* Fix uri stuff
* remove needless clone
Co-authored-by: kianenigma <kian.peymani@gmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: kianenigma <kian@parity.io>
Julian Eager