6c8946ae21
fix tracker repeat writes ( #9653 )
2021-08-31 12:41:44 +00:00
30ce8bbc8c
Fetch runtime code from storage cache when using proofing backend ( #9611 )
...
Before we fetched the runtime code from the `TrieBackend` and this lead
to not using the storage cache. Thus, we recalculated the storage hash
for the runtime code on every call into the runtime and this killed the
performance on parachains block authoring. The solution is to fetch the
runtime code from the storage cache, to make sure we use the cached
storage cache.
2021-08-24 09:54:53 +00:00
0489c1768c
FRAME Weights with Storage Metadata ( #9471 )
...
* weights with metadata
* fix
* fix contract test
* skip metadata tag
* special handling for `frame_system`
* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=frame_system --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/system/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* add skip metadata to contracts
* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* fix contract test
* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_democracy --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/democracy/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_bounties --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/bounties/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* expose component information
* fix test generation
* refactor list benchmarks
* move component selection out of runtime
* add benchmark verification
* missing feature
* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_bounties --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/bounties/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_democracy --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/democracy/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* add internal repeats
* update weights with internal repeats
* fix warning
* return error with pov
* try without tracking
* Revert "return error with pov"
This reverts commit 44c36cbbd3c6818f36f377e3e291f1df156e40f7.
* Revert "try without tracking"
This reverts commit f401c44aebff2232389d8d307b20924891e5d77d.
* Revert "Revert "try without tracking""
This reverts commit 4b4e05929802ad3e8154e107359447634e5fb21b.
* state without tracking
* fix build
* temp test
* split db and timing benchmarks
* extend db results?
* default repeat is internal
* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_democracy --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/democracy/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* fix warning
* bump linked hash map
* use linked hash map for storage tracker
* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_democracy --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/democracy/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* remove conflicting short command
* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_democracy --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/democracy/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* missed one linked hashmap
* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_bounties --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/bounties/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_bounties --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/bounties/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/election-provider-multi-phase/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* new weights with latest changes
* Update frame/benchmarking/src/utils.rs
Co-authored-by: Parity Benchmarking Bot <admin@parity.io >
2021-08-07 16:05:12 +00:00
7b56ab15b4
Run cargo fmt on the whole code base ( #9394 )
...
* Run cargo fmt on the whole code base
* Second run
* Add CI check
* Fix compilation
* More unnecessary braces
* Handle weights
* Use --all
* Use correct attributes...
* Fix UI tests
* AHHHHHHHHH
* 🤦
* Docs
* Fix compilation
* 🤷
* Please stop
* 🤦 x 2
* More
* make rustfmt.toml consistent with polkadot
Co-authored-by: André Silva <andrerfosilva@gmail.com >
2021-07-21 14:32:32 +00:00
b42b8fc5fb
Include StorageInfo in Benchmarking Pipeline ( #9090 )
...
* extend storageinfo
* extend_storage_info
* use vec
* add storage info to pipeline
* get read and written keys
* undo storageinfo move
* refactor keytracker
* return read / write count
* playing with key matching
* add basic `StorageInfo` constructor
* add whitelisted to returned info
* fix some test stuff
* pipe comments into benchmark data
* add_storage_comments
* add comments to template
* track only storage prefix
* Update frame/benchmarking/src/lib.rs
* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* fix test
* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* remove test logs
* add temp benchmark script
* Apply suggestions from code review
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com >
* remove keytracker and use trackedstoragekey
* add comment for unknown keys
* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_timestamp --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/timestamp/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* remove duplicate comments with unknown keys
* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_timestamp --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/timestamp/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* refactor bench tracker, and fix results
* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* fix child tries in new tracker
* extra newline
* fix unused warning
* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_timestamp --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/timestamp/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* fix master merge
* storage info usage refactor
* remove now unused
* fix refactor
* use a vec for prefix
* fix tests
* also update writer to use vec
* disable read and written keys for now
* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=frame_system --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/system/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* Update frame/system/src/weights.rs
* fix test
* Delete weights.rs
* reset weights
Co-authored-by: Parity Bot <admin@parity.io >
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com >
2021-07-07 22:06:06 +00:00
77a4b980ae
Fast sync ( #8884 )
...
* State sync
* Importing state fixes
* Bugfixes
* Sync with proof
* Status reporting
* Unsafe sync mode
* Sync test
* Cleanup
* Apply suggestions from code review
Co-authored-by: cheme <emericchevalier.pro@gmail.com >
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com >
* set_genesis_storage
* Extract keys from range proof
* Detect iter completion
* Download and import bodies with fast sync
* Replaced meta updates tuple with a struct
* Fixed reverting finalized state
* Reverted timeout
* Typo
* Doc
* Doc
* Fixed light client test
* Fixed error handling
* Tweaks
* More UpdateMeta changes
* Rename convert_transaction
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com >
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com >
* Code review suggestions
* Fixed count handling
Co-authored-by: cheme <emericchevalier.pro@gmail.com >
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com >
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com >
2021-06-22 09:32:43 +00:00
693b39f43b
Frame remove_all with size limit. ( #9106 )
...
* remove prefixed content with limit.
* test match
* factor comment and factor ext limit removal.
* fix benchmark
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com >
2021-06-15 13:23:58 +00:00
f01c396677
Compact proof utilities in sp_trie. ( #8574 )
...
* validation extension in sp_io
* need paths
* arc impl
* missing host function in executor
* io to pkdot
* decode function.
* encode primitive.
* trailing tab
* multiple patch
* fix child trie logic
* restore master versionning
* bench compact proof size
* trie-db 22.3 is needed
* line width
* split line
* fixes for bench (additional root may not be needed as original issue was
with empty proof).
* revert compact from block size calculation.
* New error type for compression.
* Adding test (incomplete (failing)).
Also lacking real proof checking (no good primitives in sp-trie crate).
* There is currently no proof recording utility in sp_trie, removing
test.
* small test of child root in proof without a child proof.
* remove empty test.
* remove non compact proof size
* Missing revert.
* proof method to encode decode.
2021-06-07 09:06:38 +00:00
ad76ac798e
Removes unnecessary blank impl for Backend ( #8897 )
...
* Removes unnecessary blank impl for Backend
This commit removes a from my perspective unneccessary implementation
for &T which implement Backend.
The current implementation exists (again from my perspective) solely
to satisfy a methods &mut self parameters (i.e. allows to satisfy
this for an & reference via using &mut &Backend).
As all implementors use a RefCell with borrow_mut() where actually
calling the mentioned &mut self method and then forwad to the
{} implementation of either TrieBackend or ProvingBackend, the
current &mut self seems to be not needed.
* Fixed tests client
2021-05-25 19:50:12 +00:00
14b5acab86
Introduce a "dynamic" block size limit for proposing ( #8588 )
...
* Introduce a "dynamic" block size limit for proposing
This adds support for using a dynamic block size limit per call to
`propose`. This is required for Cumulus/Parachains to always use stay in
the limits of the maximum allowed PoV size.
As described in the docs, the block limit is only checked in the process
of pushing transactions. As we normally do some other operations in
`on_finalize`, it can happen that the block size still grows when there
is some proof being collected (as we do for parachains). This means,
that the given block limit needs to be rather conservative on the actual
value and should not be the upper limit.
* Update client/basic-authorship/src/basic_authorship.rs
Co-authored-by: Andronik Ordian <write@reusable.software >
* More future proof encoded size updating
* Use `ProofRecorderInner`
* Update client/basic-authorship/src/basic_authorship.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com >
* Update client/basic-authorship/src/basic_authorship.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com >
* Update client/basic-authorship/src/basic_authorship.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com >
* Update client/consensus/slots/src/lib.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com >
* Update client/consensus/slots/src/slots.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com >
* Update client/basic-authorship/src/basic_authorship.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com >
* Update client/basic-authorship/src/basic_authorship.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com >
* Update client/basic-authorship/src/basic_authorship.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com >
Co-authored-by: Andronik Ordian <write@reusable.software >
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com >
2021-04-14 17:56:22 +00:00
c04b44b0a3
Add PoV Tracking to Benchmarking Pipeline ( #8559 )
...
* Added a function to estimate proof size for benchmarking
* integrate proof_size into benchmarking pipeline
* Update client/db/src/bench.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com >
* Update client/db/src/bench.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com >
* fix tests
* one more test
* Update bench.rs
* Update utils/frame/benchmarking-cli/src/writer.rs
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io >
* Update utils/frame/benchmarking-cli/src/command.rs
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com >
Co-authored-by: arkpar <arkady.paronyan@gmail.com >
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com >
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io >
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com >
2021-04-08 12:54:46 +02:00
e3e651f72c
Happy new year ( #7814 )
...
* Happy new year
Updates the copyright years and fixes wrong license headers.
* Fix the template
* Split HEADER into HEADER-APACHE & HEADER-GPL
2021-01-04 09:03:13 +00:00
9ce24fe1f4
Allow capping the amount of work performed when deleting a child trie ( #7671 )
...
* Allow Backend::for_keys_in_child_storage to be aborted by the closure
* Ext::kill_child_storage now takes an upper limit for backend deletion
* Add Storage::storage_kill_limited() runtime interface
* review: Use a new version of kill_storage instead of a new interface
* review: Simplify boolean expression
Co-authored-by: cheme <emericchevalier.pro@gmail.com >
* review: Rename for_keys_in_child_storage
Co-authored-by: cheme <emericchevalier.pro@gmail.com >
2020-12-09 01:17:28 +00:00
5b7af66384
fix bench db wipe ( #6965 )
2020-08-26 18:36:37 +00:00
9b42720ac5
Fix benchmark read/write key tracker for keys in child storages. ( #6905 )
...
* WIP: read child trie and write child trie
* add test
* refactor a bit + improve log
* better naming
* trigger CI
* Revert "trigger CI"
This reverts commit d0aadaeb6a12fc6c39f01b3c1b5725d19f085865.
2020-08-24 13:40:16 +00:00
481ad884d6
Dynamic Benchmarking DB Whitelist ( #6815 )
...
* Add `get_whitelist` api
* add whitelisted caller
* Whitelist caller
* remove caller 0
* initial piping of origin (not actual value yet)
* remove attempt to pass origin around
* Add whitelist for `DidUpdate` storage on `pallet_timestamp`
* fix traits
* only add to whitelist if !contains
* PassBy not implemented error
* Whitelist read/writes explicitly per key
* update docs
* reduce trait constraint
* copy pasta
* Apply suggestions from code review
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com >
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io >
* rename functions @apopiak
* missed some renaming
* enable doc tests
* Update docs
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com >
Co-authored-by: Alexander Popiak <alexander.popiak@parity.io >
2020-08-19 16:15:50 +00:00
935ee6f545
Add DB Read/Write Tracking to Benchmarking Pipeline ( #6386 )
...
* initial mockup
* add and wipe
* track writes
* start to add to pipeline
* return all reads/writes
* Log reads and writes from bench db
* causes panic
* Allow multiple commits
* commit before ending benchmark
* doesn't work???
* fix
* Update lib.rs
* switch to struct for `BenchmarkResults`
* add to output
* fix test
* line width
* @kianenigma review
* Add Whitelist to DB Tracking in Benchmarks Pipeline (#6405 )
* hardcoded whitelist
* Add whitelist to pipeline
* Remove whitelist pipeline from CLI, add to runtime
* clean-up unused db initialized whitelist
* Add regression analysis to DB Tracking (#6475 )
* Add selector
* add tests
* debug formatter for easy formula
* Update client/db/src/bench.rs
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com >
Co-authored-by: arkpar <arkady.paronyan@gmail.com >
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com >
2020-06-24 21:03:55 +02:00
82a832bc3a
Don't clone values when calculating storage root ( #6108 )
...
* Don't clone values when calculating storage root
Instead of cloning all the keys and values of the overlay when
calculating the storage root, we pass all the values by reference. This
should probably bring some performance improvements when calculating the
storage root.
* no cow version (#6113 )
Co-authored-by: cheme <emericchevalier.pro@gmail.com >
2020-05-23 11:28:34 +02:00
717fa95bc7
Restore the empty line after the license ( #6088 )
2020-05-20 13:08:27 +00:00
d5411969ac
fix whitespace ( #6062 )
2020-05-18 13:51:28 +02:00
be8c96adec
Relax substrate licensing scheme ( #5947 )
2020-05-15 13:21:56 +02:00
4ffcf98d8d
Child trie api changes BREAKING ( #4857 )
...
Co-Authored-By: thiolliere <gui.thiolliere@gmail.com >
2020-04-20 15:21:22 +02:00
92b39365e3
Benchmarks now use in-memory db & cache ( #5586 )
...
* in-mem state for benchmarks
* Use caching state
* Update Cargo.lock
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com >
2020-04-09 11:27:39 +02:00
58578af074
State metrics possible changes ( #5168 )
...
* Registering state from overlay.
* fix
* fix2
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com >
2020-04-01 19:46:40 +02:00
433824511e
Reset benchmarks by removing post-genesis changes ( #5435 )
...
* Reset by removing pos-genesis changes
* CLI option for DB cache size
* Update Cargo.lock
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com >
2020-04-01 14:08:23 +02:00
b57fd17998
Fix genesis bug in bench db ( #5253 )
2020-03-15 20:19:01 +01:00
5a33228ea9
Reduce usage of Blake2Hasher ( #5132 )
...
This reduces the usage of `Blake2Hasher` in the code base and replaces
it with `BlakeTwo256`. The most important change is the removal of the
custom extern function for `Blake2Hasher`. The runtime `Hash` trait is
now also simplified and directly requires that the implementing type
implements `Hashable`.
2020-03-05 08:51:03 +01:00
e5a7fcc8ea
Create Benchmarking Setup for Identity Pallet #4695 ( #4818 )
...
* Starting
* closer
* Compiles!
* comments
* Create seperate mock
* Remove changes to test env
* Fix step calculation
* Add host function
* Add runtime api
* compiles
* Update to use offchain timestamp
* Gives a result
* added some CLI wip
* make generic
* Update instance
* Remove CLI stuff
* Remove last cli stuff
* undo more changes
* Update benchmarks
* Update Cargo.lock
* remove test
* Move loop out of runtime
* Benchmarking externalities
* Benchmarking state
* Implemented commit
* Make CLI work, move loop back into runtime
* Wipe resets to genesis
* Speedup benchmarks
* Use enum to select extrinsic within pallet
* CLI controls which module and extrinsic to call
* Select a pallet with cli
* Add steps and repeats to cli
* Output as CSV format
* Introduce benchmark pallet
* Append bench
* Use Results
* fix merge
* Clear Identity benchmark
* Bench request judgment and cancel request
* Add final benchmarks
* Fix CSV output
* Start cleaning up for PR
* Bump numbers in `wasmtime` integration tests.
* More docs
* Add rockdb feature to bench
* Fix formatting issues
* Add test feature to bench
* Add test feature to bench
* Add rocksdb feature flag
* Update bench.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com >
Co-authored-by: Gavin Wood <github@gavwood.com >
2020-02-10 10:23:08 +01:00
Guillaume Thiolliere