Before we fetched the runtime code from the `TrieBackend` and this lead
to not using the storage cache. Thus, we recalculated the storage hash
for the runtime code on every call into the runtime and this killed the
performance on parachains block authoring. The solution is to fetch the
runtime code from the storage cache, to make sure we use the cached
storage cache.
* Improve `state` related logs to use a more uniform format
The logging before wasn't that uniform and not that great to read/parse.
Now we are using a uniform format for all the logs. Besides these
changes, there are some minor changes around the code that calls the
state machine.
* Make CI happy
* Use HexDisplay for `ext_id`
* Run cargo fmt on the whole code base
* Second run
* Add CI check
* Fix compilation
* More unnecessary braces
* Handle weights
* Use --all
* Use correct attributes...
* Fix UI tests
* AHHHHHHHHH
* 🤦
* Docs
* Fix compilation
* 🤷
* Please stop
* 🤦 x 2
* More
* make rustfmt.toml consistent with polkadot
Co-authored-by: André Silva <andrerfosilva@gmail.com>
* validation extension in sp_io
* need paths
* arc impl
* missing host function in executor
* io to pkdot
* decode function.
* encode primitive.
* trailing tab
* multiple patch
* fix child trie logic
* restore master versionning
* bench compact proof size
* trie-db 22.3 is needed
* line width
* split line
* fixes for bench (additional root may not be needed as original issue was
with empty proof).
* revert compact from block size calculation.
* New error type for compression.
* Adding test (incomplete (failing)).
Also lacking real proof checking (no good primitives in sp-trie crate).
* There is currently no proof recording utility in sp_trie, removing
test.
* small test of child root in proof without a child proof.
* remove empty test.
* remove non compact proof size
* Missing revert.
* proof method to encode decode.
* emit a custom section from impl_runtime_apis!
This change emits a custom section from the impl_runtime_apis! proc macro.
Each implemented API will result to emitting a link section `runtime_apis`.
During linking all sections with this name will be concatenated and
placed into the final wasm binary under the same name.
* Introduce `runtime_version` proc macro
This macro takes an existing `RuntimeVersion` const declaration, parses
it and emits the version information in form of a linking section.
Ultimately such a linking section will result into a custom wasm
section.
* Parse custom wasm section for runtime version
* Apply suggestions from code review
Co-authored-by: David <dvdplm@gmail.com>
* Fix sc-executor integration tests
* Nits
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Refactor apis section deserialization
* Fix version decoding
* Reuse uncompressed value for CallInWasm
* Log on decompression error
* Simplify if
* Reexport proc-macro from sp_version
* Merge ReadRuntimeVersionExt
* Export `read_embedded_version`
* Fix test
* Simplify searching for custom section
Co-authored-by: David <dvdplm@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Add filter reload handle
* add RPC, move logging module from cli to tracing
* remove dup fn
* working example
* Update client/rpc-api/src/system/mod.rs
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Prefer "set" to "reload"
* Re-enable the commented out features of the logger
* Remove duplicate code
* cleanup
* unneeded lvar
* Bump to latest patch release
* Add new CLI option to disable log filter reloading,
Move profiling CLI options to SharedParams
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Applied suggestions from reviews
* Fix calls to init_logger()
* Handle errors when parsing logging directives
* Deny `system_setLogFilter` RPC by default
* One more time
* Don't ignore parse errors for log directives set via CLI or RPC
* Improve docs
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update client/cli/src/config.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* fix merge errors
* include default directives with system_setLogFilter RPC,
implement system_rawSetLogFilter RPC to exclude defaults
* docs etc...
* update test
* refactor: rename fn
* Add a test for system_set_log_filter – NOTE: the code should likely change to return an error when bad directives are passed
* Update client/cli/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Address review grumbles
* Add doc note on panicking behaviour
* print all invalid directives before panic
* change RPCs to: addLogFilter and resetLogFilter
* make CLI log directives default
* add comments
* restore previous behaviour to panic when hard-coded directives are invalid
* change/refactor directive parsing
* fix line width
* add test for log filter reloading
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* finish up suggestions from code review
* improve test
* change expect message
* change fn name
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* add docs, remove unused fn
* propagate Err on invalid log directive
* Update tracing-subscriber version
* Improve docs for `disable_log_reloading` CLI param
* WIP implementation: RPC and trace capturing
* WIP
* fix incorrect number of digest items
* return errors
* add From impl for Values, rename structs
* fixes
* implement option to choose targets for traces
* rename fn
* fix EnvFilter and add root span
* fix root span
* add docs, remove unnecessary traits
* fix regression on parent_id introduced in a9c73113a8
* fix line width
* remove unused
* include block hash, parent hash & targets in response
* move types from sp-tracing into sp-rpc
move block and parent hash into root of BlockTrace
* switch from log::trace to tracing::trace in state-machine
* use unsigned integer type to represent Ext::id in traces
* ensure id is unique by implementing Subscriber
tracing_subscriber::FmtSubscriber does not guarantee
unique ids
* indentation
* fix typo
* update types
* add sp_io::storage events
* Change response format
- update types
- record distinct timestamps
- sort spans by first entered
* convert to HexDisplay, refactor
* Sort out fallout from merge
* Update client/rpc-api/src/state/mod.rs
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Exit early unless the node runs with --rpc-methods=Unsafe
* Better error handling
* Use wasm-timer
* revert trace alteration in `state-machine` and remove events in `sp_io::storage`
Resolve in follow-up PR
* Review feedback: less collects
* Without Arcs
* Fix span exit
* typo
* cleanup
* Add a few debug messages to tracing module
* Structure traces state-machine/ext; Dispatchable extrinsics spans not working
* Correctly encode Option storage values
* Remove test field for Put and Get
* Try out some changes to dispatch macro
* Add various log messages in dispatch
* Add span dispatch span to new proc macro
* Remove debug messages in dispatch
* Trivial clean up
* Structure remaining state-machine traces (ChangesRoot*)
* Removed unnesecary tracing targets
* Remove log
* New cargo.lock post merge
* Add logging for wasm_overrides
* remove temp logs
* remove temp logs
* remove unused dep
* remove temp logs
* add logging to wasm_overrides
* add logging to state_tracing
* add logging for spans to substrate (includes timings)
* Skip serializing some event fields; Remove most storage traces
* Bring back all ext.rs traces
* Do not skip bool values in events
* Skip serializing span values
* Serialize span values; remove some trace events in ext
* Remove more trace events
* Delete commented out traces
* Remove all unused traces
* Add event filtering
* Fix typo
* wip - change response types to be more efficient
missing import
type
* Serialize struct fields as camelCase
* Add back in event filtering
* Remove name field from event
* Sort spans by time entered
* Sort spans in ASCending order
* Add storage keys target param to rpc
* Limit payload size; improve hash fields; include storage keys
- cleanup event_key_filter
- better block hash representation
- limit payload size
- cleanup based on andrews comments
* Error when serialized payload is to big
* Import MAX_PAYLOAD from rpc-servers
* Clean up ext.rs
* Misc. cleaning and comments
* Strict ordering span Id; no span sort; adjust for rpc base payload
* Add RPC docs to rpc-api/src/state/mod
* Make params bullet points
* Update primitives/rpc/src/tracing.rs
* Put all tracing logic within trace
* Remove attr.record in new_span
* Add back value record in new_span
* restore result collection in ext
* Revert "Add back value record in new_span"
This reverts commit baf1a735f23e5eef1bf6264adfabb788910fa661.
* 🤦
* more 🤦
* Update docs; Try fix line width issues
* Improve docs
* Improve docs
* Remove default key filters + add key recs to docs
* Try restore old traces
* Add back doc comment
* Clean up newlines in ext.rs
* More new line remova;
l
* Use FxHashMap
* Try use EnvFilter directives for event filtering
* Remove directive, filter events by fields
* Use trace metadata correctly
* Try EnvFilter directive with all default targets
* Revert "Try EnvFilter directive with all default targets"
This reverts commit 4cc6ebc721d207c3b846444174f89d45038525ac.
* Clean up clippy warning
* Incorporate Niklas feedback
* Update trace/log macro calls to have better syntx
* Use Ordering::Relaxed
* Improve patch and filter doc comment
* Clean up `BlockSubscriber::new`
* Try optimize `BlockSubscriber::enabled`
* Apply suggestions from code review
Co-authored-by: David <dvdplm@gmail.com>
* Apply suggestions from code review
Co-authored-by: David <dvdplm@gmail.com>
* Use contains_key
* use heuristic for payload size
* Add error tupe for client::tracing::block
* Minor tweaks
* Make a note about `--features with-tracing`
* Add CURL example to RPC docs
* Link to substrate-archibe wasm
* Trivial doc clean up based on David feedback
* Explicit result type name
* Respect line length
* Use the error
* Don't print timings when spans close
* Fix failing sc-rpc-api
* Update sp-tracing inner-line doc
* Update client/tracing/src/block/mod.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update client/service/src/client/call_executor.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update client/service/src/client/call_executor.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update client/tracing/src/block/mod.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update client/tracing/src/block/mod.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Address some review grumbles
* Update primitives/state-machine/src/ext.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Use result_encoded structure fields in ext.rs
* Use value key for ext put
* Add notes about tracing key names matter
Co-authored-by: Matt <mattrutherford@users.noreply.github.com>
Co-authored-by: David <dvdplm@gmail.com>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: emostov <32168567+emostov@users.noreply.github.com>
* Initial piping of returning amount of keys killed
* One more test for `None` limit
* forgot to update
* fix return value
* use version 3
* Update to return `KillOutcome`
* Update name to KillChildStorageResult
* Moving offchain change set to state machine overlay change set,
preparing use of change set internally.
* Make change set generic over key and value, and use it for offchain
indexing.
* test ui change
* remaining delta
* generating with standard method
* Remove 'drain_committed' function, and documentation.
* Default constructor for enabling offchain indexing.
* Remove offchain change specific iterators.
* remove pub accessor
* keep previous hierarchy, just expose iterator instead.
* Update primitives/state-machine/src/overlayed_changes/mod.rs
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* fix line break
* missing renamings
* fix import
* fix new state-machine tests.
* Don't expose InnerValue type.
* Add test similar to set_storage.
* Remove conditional offchain storage (hard to instantiate correctly).
* fix
* offchain as children cannot fail if top doesn't
Co-authored-by: Addie Wagenknecht <addie@nortd.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Make offchain indexing work
This fixes some bugs with offchain indexing to make it actually working ;)
* Fix tests
* Fix browser build
* Update client/db/src/offchain.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Remove seperation between prefix and key
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Allow Backend::for_keys_in_child_storage to be aborted by the closure
* Ext::kill_child_storage now takes an upper limit for backend deletion
* Add Storage::storage_kill_limited() runtime interface
* review: Use a new version of kill_storage instead of a new interface
* review: Simplify boolean expression
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* review: Rename for_keys_in_child_storage
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Remove runtime registered extensions after execution
This prevents a bug when an extension was registered in native, but the
native execution aborted without removing the extension again. Now, when
executing the wasm code the extension is still registered and it fails
of being registered. So, the wasm execution fails as well. This can
happen for example for the `VerificationExt`.
* Make it better
* checkpoint before removing CT from change trie
* before trie backend without tx
* undo
* Started no transaction, but would need using a different root
calculation method, out of the scope of this pr, will roll back.
* Remove NoTransaction.
* partially address review.
dummy stats implementation for no_std.
* Remove ChangeTrieOverlay.
* modified function
* Remove witness_ext
* need noops changes root
* update from cumulus branch
* line break
* remove warning
* line break
* From review: renamings and stats active in no std (except time).
* include cache, exclude change trie cache with individual temporary bad looking
no_std check
* little test
* fuse imports and filter_map prepare_extrinsics_input_inner fold.
* put back ExtInner into Ext, awkward double proto for new function.
* Apply suggestions from code review
* Update primitives/state-machine/Cargo.toml
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Add transactional storage functionality to OverlayChanges
A collection already has a natural None state. No need to
wrap it with an option.
* Add storage transactions runtime interface
* Add frame support for transactions
* Fix committed typo
* Rename 'changes' variable to 'overlay'
* Fix renaming change
* Fixed strange line break
* Rename clear to clear_where
* Add comment regarding delete value on mutation
* Add comment which changes are covered by a transaction
* Do force the arg to with_transaction return a Result
* Use rust doc comments on every documentable place
* Fix wording of insert_diry doc
* Improve doc on start_transaction
* Rename value to overlayed in close_transaction
* Inline negation
* Improve wording of close_transaction comments
* Get rid of an expect by using get_or_insert_with
* Remove trailing whitespace
* Rename should to expected in tests
* Rolling back a transaction must mark the overlay as dirty
* Protect client initiated storage tx from being droped by runtime
* Review nits
* Return Err when entering or exiting runtime fails
* Documentation fixup
* Remove close type
* Move enter/exit runtime to excute_aux in the state-machine
* Rename Discard -> Rollback
* Move child changeset creation to constructor
* Move child spawning into the closure
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Fixup for code suggestion
* Unify re-exports
* Rename overlay_changes to mod.rs and move into subdir
* Change proof wording
* Adapt a new test from master to storage-tx
* Suggestions from the latest round of review
* Fix warning message
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Hide internal structure of OverlayChanges
* Fix tests for OverlayChanges refactor
* Do not clone pending changes
Discarding prospective changes should be equivalent as a state machine
is not to be called with peding changes.
This will be replaced by a storage transaction that is rolled back before
executing the call the second time removing this constraint.
* Doc fixes
* Remove overlong line
* Revert "Do not clone pending changes"
This reverts commit 4799491f4ac16f8517287a0fcf4a3f84ad56f46e.
* Deduplicate chield tries returned from child_infos()
* Remove redundant type annotation
* Avoid changing the storage root in tests
* Preserve extrinsic indices in trie build test
* Swap order of comitted and prospective in fn child_infos
This is only for consistency and does not impact the result.
* Rename set_pending to replace_pending for clearity
* Avoid changing overlay committed layer.
* basic test
* Add some tx in the test.
* only update from backend value on missing entry in both layer.
deleted entry is replace by empty vec.
* test and review changes
* additional test and review change
* remove test on changing existing value, it does not always panic
depending on existing content
* Update primitives/state-machine/src/overlayed_changes.rs
* Update primitives/state-machine/src/overlayed_changes.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Don't include `:code` by default in storage proofs (#5060)
* Adds test to verify that the runtime currently is always contained in
the proof
* Start passing the runtime wasm code from the outside
* Fix compilation
* More build fixes
* Make the test work as expected now :)
* Last fixes
* Fixes benchmarks
* Review feedback
* Apply suggestions from code review
Co-Authored-By: Sergei Pepyakin <sergei@parity.io>
* Review feedback
* Fix compilation
Co-authored-by: Sergei Pepyakin <s.pepyakin@gmail.com>
* Fix compilation and change the way `RuntimeCode` works
* Fix tests
* Switch to `Cow`
Co-authored-by: Benjamin Kampmann <ben@gnunicorn.org>
Co-authored-by: Sergei Pepyakin <s.pepyakin@gmail.com>
This reduces the usage of `Blake2Hasher` in the code base and replaces
it with `BlakeTwo256`. The most important change is the removal of the
custom extern function for `Blake2Hasher`. The runtime `Hash` trait is
now also simplified and directly requires that the implementing type
implements `Hashable`.
* Adds test to verify that the runtime currently is always contained in
the proof
* Start passing the runtime wasm code from the outside
* Fix compilation
* More build fixes
* Make the test work as expected now :)
* Last fixes
* Fixes benchmarks
* Review feedback
* Apply suggestions from code review
Co-Authored-By: Sergei Pepyakin <sergei@parity.io>
* Review feedback
* Fix compilation
Co-authored-by: Sergei Pepyakin <s.pepyakin@gmail.com>
* Implement runtime version checks in `set_code`
Check that the new runtime code given to `set_code` fullfills some
requirements:
- `spec_name` matches
- `spec_version` does not decreases
- `impl_version` does not decreases
- Either `spec_version` and `impl_version` increase
* Make tests almost work
* Some fixes after master merge
* Fix tests
* Add missed file
* Make depedency check happy?
* Remove leftover `sc-executor`
* AHHHHH
* Reset debug stuff
* Remove some 'static
* More 'static
* Some docs
* Update `Cargo.lock`
* Extend `Proposer` to optionally generate a proof of the proposal
* Something
* Refactor sr-api to not depend on client anymore
* Fix benches
* Apply suggestions from code review
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Apply suggestions from code review
* Introduce new `into_storage_changes` function
* Switch to runtime api for `execute_block` and don't require `H256`
anywhere in the code
* Put the `StorageChanges` into the `Proposal`
* Move the runtime api error to its own trait
* Adds `StorageTransactionCache` to the runtime api
This requires that we add `type NodeBlock = ` to the
`impl_runtime_apis!` macro to work around some bugs in rustc :(
* Remove `type NodeBlock` and switch to a "better" hack
* Start using the transaction cache from the runtime api
* Make it compile
* Move `InMemory` to its own file
* Make all tests work again
* Return block, storage_changes and proof from Blockbuilder::bake()
* Make sure that we use/set `storage_changes` when possible
* Add test
* Fix deadlock
* Remove accidentally added folders
* Introduce `RecordProof` as argument type to be more explicit
* Update client/src/client.rs
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Update primitives/state-machine/src/ext.rs
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Integrates review feedback
* Remove `unsafe` usage
* Update client/block-builder/src/lib.rs
Co-Authored-By: Benjamin Kampmann <ben@gnunicorn.org>
* Update client/src/call_executor.rs
* Bump versions
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com>
Kian Paimani