Commit Graph

64 Commits

Author SHA1 Message Date
Bastian Köcher f3662b4bba Happy new year 2022 (#10573) 2022-01-03 09:22:14 +01:00
cheme 4c651637f2 Inner hashing of value in state trie (runtime versioning). (#9732)
* starting

* Updated from other branch.

* setting flag

* flag in storage struct

* fix flagging to access and insert.

* added todo to fix

* also missing serialize meta to storage proof

* extract meta.

* Isolate old trie layout.

* failing test that requires storing in meta when old hash scheme is used.

* old hash compatibility

* Db migrate.

* runing tests with both states when interesting.

* fix chain spec test with serde default.

* export state (missing trie function).

* Pending using new branch, lacking genericity on layout resolution.

* extract and set global meta

* Update to branch 4

* fix iterator with root flag (no longer insert node).

* fix trie root hashing of root

* complete basic backend.

* Remove old_hash meta from proof that do not use inner_hashing.

* fix trie test for empty (force layout on empty deltas).

* Root update fix.

* debug on meta

* Use trie key iteration that do not include value in proofs.

* switch default test ext to use inner hash.

* small integration test, and fix tx cache mgmt in ext.
test  failing

* Proof scenario at state-machine level.

* trace for db upgrade

* try different param

* act more like iter_from.

* Bigger batches.

* Update trie dependency.

* drafting codec changes and refact

* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.

* alt hashing only for branch with value.

* fix trie tests

* Hash of value include the encoded size.

* removing fields(broken)

* fix trie_stream to also include value length in inner hash.

* triedbmut only using alt type if inner hashing.

* trie_stream to also only use alt hashing type when actually alt hashing.

* Refactor meta state, logic should work with change of trie treshold.

* Remove NoMeta variant.

* Remove state_hashed trigger specific functions.

* pending switching to using threshold, new storage root api does not
make much sense.

* refactoring to use state from backend (not possible payload changes).

* Applying from previous state

* Remove default from storage, genesis need a special build.

* rem empty space

* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.

* fix compilation

* Right logic to avoid switch on reencode when default layout.

* Clean up some todos

* remove trie meta from root upstream

* update upstream and fix benches.

* split some long lines.

* UPdate trie crate to work with new design.

* Finish update to refactored upstream.

* update to latest triedb changes.

* Clean up.

* fix executor test.

* rust fmt from master.

* rust format.

* rustfmt

* fix

* start host function driven versioning

* update state-machine part

* still need access to state version from runtime

* state hash in mem: wrong

* direction likely correct, but passing call to code exec for genesis
init seem awkward.

* state version serialize in runtime, wrong approach, just initialize it
with no threshold for core api < 4 seems more proper.

* stateversion from runtime version (core api >= 4).

* update trie, fix tests

* unused import

* clean some TODOs

* Require RuntimeVersionOf for executor

* use RuntimeVersionOf to resolve genesis state version.

* update runtime version test

* fix state-machine tests

* TODO

* Use runtime version from storage wasm with fast sync.

* rustfmt

* fmt

* fix test

* revert useless changes.

* clean some unused changes

* fmt

* removing useless trait function.

* remove remaining reference to state_hash

* fix some imports

* Follow chain state version management.

* trie update, fix and constant threshold for trie layouts.

* update deps

* Update to latest trie pr changes.

* fix benches

* Verify proof requires right layout.

* update trie_root

* Update trie deps to  latest

* Update to latest trie versioning

* Removing patch

* update lock

* extrinsic for sc-service-test using layout v0.

* Adding RuntimeVersionOf to CallExecutor works.

* fmt

* error when resolving version and no wasm in storage.

* use existing utils to instantiate runtime code.

* Patch to delay runtime switch.

* Revert "Patch to delay runtime switch."

This reverts commit 67e55fee468f1a0cda853f5362b22e0d775786da.

* useless closure

* remove remaining state_hash variables.

* Remove outdated comment

* useless inner hash

* fmt

* fmt and opt-in feature to apply state change.

* feature gate core version, use new test feature for node and test node

* Use a 'State' api version instead of Core one.

* fix merge of test function

* use blake macro.

* Fix state api (require declaring the api in runtime).

* Opt out feature, fix macro for io to select a given version
instead of latest.

* run test nodes on new state.

* fix

* Apply review change (docs and error).

* fmt

* use explicit runtime_interface in doc test

* fix ui test

* fix doc test

* fmt

* use default for path and specname when resolving version.

* small review related changes.

* doc value size requirement.

* rename old_state feature

* Remove macro changes

* feature rename

* state version as host function parameter

* remove flag for client api

* fix tests

* switch storage chain proof to V1

* host functions, pass by state version enum

* use WrappedRuntimeCode

* start

* state_version in runtime version

* rust fmt

* Update storage proof of max size.

* fix runtime version rpc test

* right intent of convert from compat

* fix doc test

* fix doc test

* split proof

* decode without replay, and remove some reexports.

* Decode with compatibility by default.

* switch state_version to u8. And remove RuntimeVersionBasis.

* test

* use api when reading embedded version

* fix decode with apis

* extract core version instead

* test fix

* unused import

* review changes.

Co-authored-by: kianenigma <kian@parity.io>
2021-12-24 08:54:07 +00:00
sandreim 9b73a8a6fc Add logger configuration hook (#10440)
* Initial poc

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Make config available to logger hook

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fmt

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fmt

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Add metric prefix option in sc_cli::RunCmd

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Remove print

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Review fixes

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fmt

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix docs

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
2021-12-16 11:18:24 +00:00
Koute 7711f5266e Statically register host WASM functions (#10394)
* Statically register host WASM functions

* Fix `substrate-test-client` compilation

* Move `ExtendedHostFunctions` to `sp-wasm-interface`

* Fix `sp-runtime-interface` tests' compilation

* Fix `sc-executor-wasmtime` tests' compilation

* Use `runtime_interface` macro in `test-runner`

* Fix `sc-executor` tests' compilation

* Reformatting/`rustfmt`

* Add an extra comment regarding the `H` generic arg in `create_runtime`

* Even more `rustfmt`

* Depend on `wasmtime` without default features in `sp-wasm-interface`

* Bump version of `sp-wasm-interface` to 4.0.1

* Bump `sp-wasm-interface` in `Cargo.lock` too

* Bump all of the `sp-wasm-interface` requirements to 4.0.1

Maybe this will appease cargo-unleash?

* Revert "Bump all of the `sp-wasm-interface` requirements to 4.0.1"

This reverts commit 0f7ccf8e0f371542861121b145ab87af6541ac30.

* Make `cargo-unleash` happy (maybe)

* Use `cargo-unleash` to bump the crates' versions

* Align to review comments
2021-12-14 08:26:40 +00:00
Éloïs 3acd335a8d Make wasm runtime cache size configurable (#10177)
* Make wasm runtime cache size configurable

* apply review comments

* remove VersionedRuntimeValue

* fix compilation

* VersionedRuntime: replace clone by Arc

* fmt

* fix warnings

* fix tests compilation

* fmt
2021-12-09 16:10:16 +01:00
Koute 15a0bfb0f6 Clear WASM linear memory on other OSes besides Linux too (#10291) 2021-11-18 11:16:38 +00:00
Arkadiy Paronyan 4cbbf0cf43 Kill the light client, CHTs and change tries. (#10080)
* Remove light client, change tries and CHTs

* Update tests

* fmt

* Restore changes_root

* Fixed benches

* Cargo fmt

* fmt

* fmt
2021-11-12 14:15:01 +01:00
Koute a7e3d819f8 Refactor sp-sandbox; make sure both sandbox executors are always tested (#10173)
* sp-sandbox: convert executors into normal `mod`s instead of using `include!`

* sp-sandbox: run `cargo fmt` on `host_executor.rs`

* sp-sandbox: abstract away the executors behind traits

* sp_sandbox: always compile both executors when possible

* sc-executor: make sure all sandbox tests run on both sandbox executors

* sc-executor: fix brainfart: actually call into the sandbox through the trait

* sc-runtime-test: fix cargo fmt

* sc-runtime-test: deduplicate executor-specific sandbox test entrypoints

* sc-executor: test each sandbox executor in a separate test

* cargo fmt (Github's conflict resolving thingy broke indentation)
2021-11-08 12:52:11 +00:00
Bastian Köcher 2deed49706 Update wasmtime to 0.29.0 (#9552)
* Start

* Move to ctx

* Make it compile for now

* More work

* Get rid off state-holder

* Use less Refcells

* 🤦

* Don't use RefCell

* Use names for parameters

* Fixes after merge

* Fixes after merge

* Review feedback

* FMT
2021-09-29 12:30:46 +00:00
Squirrel aafe64315a Remove extra commas made redundent after rustfmt (#9404)
* Remove extra commas made redundent after rustfmt
2021-07-22 11:06:17 +01:00
Bastian Köcher 7b56ab15b4 Run cargo fmt on the whole code base (#9394)
* Run cargo fmt on the whole code base

* Second run

* Add CI check

* Fix compilation

* More unnecessary braces

* Handle weights

* Use --all

* Use correct attributes...

* Fix UI tests

* AHHHHHHHHH

* 🤦

* Docs

* Fix compilation

* 🤷

* Please stop

* 🤦 x 2

* More

* make rustfmt.toml consistent with polkadot

Co-authored-by: André Silva <andrerfosilva@gmail.com>
2021-07-21 14:32:32 +00:00
Sergei Shulepov 5f0257f3b3 Test restoring zeroed data (#9011)
* Test restoring zeroed data

* Change to u64
2021-06-15 10:58:09 +00:00
Sergei Shulepov 3df32a5411 Decommit instance memory after a runtime call on Linux (#8998)
* Decommit instance memory after a runtime call on Linux

* Update documentation for the test

* Remove unfinished comment

* Use saturating_sub.

Also update the doc comment.

* Precise RSS tracking in the test

Instead of tracking RSS for the whole process we just look at the particular mapping that is associated with the linear memory of the runtime instance

* Remove unused import

* Fix unused imports

* Fix the unused imports error for good

* Rollback an accidental change to benches

* Fix the test

* Remove now unneeded code
2021-06-14 21:07:06 +00:00
Sergei Shulepov 0849bcce0e Embed runtime version as a custom section (#8688)
* emit a custom section from impl_runtime_apis!

This change emits a custom section from the impl_runtime_apis! proc macro.

Each implemented API will result to emitting a link section `runtime_apis`.
During linking all sections with this name will be concatenated and
placed into the final wasm binary under the same name.

* Introduce `runtime_version` proc macro

This macro takes an existing `RuntimeVersion` const declaration, parses
it and emits the version information in form of a linking section.
Ultimately such a linking section will result into a custom wasm
section.

* Parse custom wasm section for runtime version

* Apply suggestions from code review

Co-authored-by: David <dvdplm@gmail.com>

* Fix sc-executor integration tests

* Nits

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Refactor apis section deserialization

* Fix version decoding

* Reuse uncompressed value for CallInWasm

* Log on decompression error

* Simplify if

* Reexport proc-macro from sp_version

* Merge ReadRuntimeVersionExt

* Export `read_embedded_version`

* Fix test

* Simplify searching for custom section

Co-authored-by: David <dvdplm@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-05-12 13:39:08 +00:00
Tomasz Drwięga f3d4355a20 Custom RPC for Merkle Mountain Range pallet (#8137)
* Add MMR custom RPC.

* Change RuntimeApi to avoid hardcoding leaf type.

* Properly implement the new RuntimeAPI and wire up RPC.

* Extract Offchain DB as separate execution extension.

* Enable offchain DB access for offchain calls.

* Fix offchain_election tests.

* Skip block initialisation for proof generation.

* Fix integration test setup.

* Fix offchain tests. Not sure how I missed them earlier 🤷.

* Fix long line.

* One more test missing.

* Update mock for multi-phase.

* Address review grumbbles.

* Address review grumbles.

* Fix line width of a comment
2021-03-10 15:28:56 +00:00
Sergei Shulepov eb7226bb4a WasmExecutor takes a cache directory (#8057)
That is useful for executors like wasmtime which produces compiled code
and can actually benefit from caching under some circumstances
2021-02-09 16:48:29 +00:00
cheme 878f7ccf6e Allow transaction for offchain indexing (#7290)
* Moving offchain change set to state machine overlay change set,
preparing use of change set internally.

* Make change set generic over key and value, and use it for offchain
indexing.

* test ui change

* remaining delta

* generating with standard method

* Remove 'drain_committed' function, and documentation.

* Default constructor for enabling offchain indexing.

* Remove offchain change specific iterators.

* remove pub accessor

* keep previous hierarchy, just expose iterator instead.

* Update primitives/state-machine/src/overlayed_changes/mod.rs

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* fix line break

* missing renamings

* fix import

* fix new state-machine tests.

* Don't expose InnerValue type.

* Add test similar to set_storage.

* Remove conditional offchain storage (hard to instantiate correctly).

* fix

* offchain as children cannot fail if top doesn't

Co-authored-by: Addie Wagenknecht <addie@nortd.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
2021-01-22 12:27:43 +00:00
Bastian Köcher cd0ad4805d Make offchain indexing work (#7940)
* Make offchain indexing work

This fixes some bugs with offchain indexing to make it actually working ;)

* Fix tests

* Fix browser build

* Update client/db/src/offchain.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Remove seperation between prefix and key

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
2021-01-21 13:12:42 +01:00
Bastian Köcher 8cdb5c01b6 Fix incorrect use of syn::exports (#7838)
* Fix incorrect use of syn::exports

Instead of using `syn::exports` we should import the trait from the
quote crate directly.

* Use own macro for test cases to fix compilation with latest syn

* Fix test
2021-01-06 22:22:23 +01:00
Bastian Köcher e3e651f72c Happy new year (#7814)
* Happy new year

Updates the copyright years and fixes wrong license headers.

* Fix the template

* Split HEADER into HEADER-APACHE & HEADER-GPL
2021-01-04 09:03:13 +00:00
Tomasz Drwięga 02f66e8823 Merkle Mountain Range pallet (#7312)
* Add MMR pallet.

* WiP

* Working on testing.

* WiP - test

* Tests passing.

* Add proof generation.

* Generate and verify proofs.

* Allow verification of older proofs.

* Move stuff to a module.

* Split MMR stuff to it's own module.

* Add docs.

* Make parent hash optional.

* LeafData failed approach.

* Finally implement Compact stuff.

* Compact encoding WiP

* Implement remaining pieces.

* Fix tests

* Add docs to compact.

* Implement for tuples.

* Fix documentation.

* Fix warnings and address review suggestion.

* Update frame/merkle-mountain-range/src/primitives.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Address review grumbles.

* Removing missing crate.

* Fix test.

* Add some docs and test.

* Add multiple instances.

* Cargo.toml sync.

* Fix no_std compilation.

* More no_std stuff.

* Rename MMR struct.

* Addressing other grumbles.

* Fix test.

* Remove format for no_std compat.

* Add test for MMR pallet.

* Fix std feature.

* Update versions.

* Add to node/runtime.

* Add hook to insert digest.

* Make primitives public.

* Update lib.rs

tech spec/typos etc

* Use WeightInfo and benchmarks.

* Fix test.

* Fix benchmarks.

* Trait -> Config.

* Fix typo.

* Fix tests.

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Addie Wagenknecht <addie@nortd.com>
2020-12-09 16:35:13 +01:00
Bernhard Schuster 4ccf6672b1 resolve unresolved error nits of #7617 (#7631)
* handle executor should_panic test better

* Revert "reduce should panic, due to extended error messages"

This reverts commit c0805940184a62cd9302603ad911c3591e70a60c.

* remove excessive constraints

* remove duplicate documentation messages for error variants

* reduce T: constraints to the abs minimum

* whoops

* fewer bounds again

Co-authored-by: Bernhard Schuster <bernhard@parity.io>
2020-11-30 16:53:33 +01:00
Andrew Plaza f6dc35284f Change TRACING_SET to static (#7607)
* change TRACING_SET to static

* Update primitives/io/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* modify test with nested spans

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-11-28 09:40:24 +01:00
Bernhard Schuster 8c7d217091 chore/error: remove from str conversion and add deprecation notificat… (#7472)
* chore/error: remove from str conversion and add deprecation notifications

* fixup changes

* fix test looking for gone ::Msg variant

* another test fix

* one is duplicate, the other is not, so duplicates reported are n-1

* darn spaces

Co-authored-by: Andronik Ordian <write@reusable.software>

* remove pointless doc comments of error variants without any value

* low hanging fruits (for a tall person)

* moar error type variants

* avoid the storage modules for now

They are in need of a refactor, and the pain is rather large
removing all String error and DefaultError occurences.

* chore remove pointless error generic

* fix test for mocks, add a bunch of non_exhaustive

* max line width

* test fixes due to error changes

* fin

* error outputs... again

* undo stderr adjustments

* Update client/consensus/slots/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* remove closure clutter

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* more error types

* introduce ApiError

* extract Mock error

* ApiError refactor

* even more error types

* the last for now

* chore unused deps

* another extraction

* reduce should panic, due to extended error messages

* error test happiness

* shift error lines by one

* doc tests

* white space

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Into -> From

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* remove pointless codec

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* avoid pointless self import

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Bernhard Schuster <bernhard@parity.io>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-11-27 18:37:53 +00:00
Nikolay Volf a062bc2f1d Runtime worker threads (#7089)
* std variant

* principal work

* format and naming

* format and naming continued

* working nested fork

* add comment

* naming and tabs

* line width

* fix wording

* address review

* refactor dynamic dispatch

* update wasmtime

* some care

* move ext

* more refactor

* doc effort

* simplify

* doc effort

* tests and docs

* address review

* naming

* explain some args

* add example

* unwinding for native and tests

* rename stray

* fix refs

* fix tests

* fix warnings

* stray naming

* fixes and comments

* Update primitives/io/src/tasks.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* make examples "compile"

* dyn_dispatch -> spawn_call

* fix impl

* address review

* Update primitives/io/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update primitives/io/src/tasks.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update primitives/io/src/async_externalities.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update primitives/io/src/tasks.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Update frame/example-parallel/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* fix compilation

* Update client/executor/common/src/wasm_runtime.rs

Co-authored-by: Sergei Shulepov <sergei@parity.io>

* address review

* Update client/executor/wasmtime/src/instance_wrapper.rs

Co-authored-by: Sergei Shulepov <sergei@parity.io>

* Update client/executor/src/native_executor.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update primitives/io/src/tasks.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/executor/src/native_executor.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update primitives/io/src/tasks.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/executor/wasmtime/src/instance_wrapper.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* address some issues

* address more issues

* wasm_only interface

* define sp_tasks

* avoid anyhow

* fix example

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-10-20 12:41:51 +00:00
Benjamin Kampmann a9c73113a8 Tracing for wasm with bridging to native (#6916)
* implement events handling, implement parent_id for spans & events

* add events to sp_io::storage

* update test

* add tests

* adjust limit

* let tracing crate handle parent_ids

* re-enable current-id tracking

* add test for threads with CurrentSpan

* fix log level

* remove redundant check for non wasm traces

* remove duplicate definition in test

* Adding conditional events API

* prefer explicit parent_id over current,

enhance test

* limit changes to client::tracing event implementation

* remove From impl due to fallback required on parent_id

* make tracing codecable

* replace with global tracing

* new tracing interface

* impl TracingSubscriber in client

* implement access to global TracingSubscriber from primitives

* span for wasm

* increment towards Wasm Tracing Subscriber implementation

* increment, remove sp-tracing from runtime-interface

* increment, it compiles

* attained original functionality with new mechanism

* implement remaining TracingSubscriber functions

* remove spans from decl_module

* add handling for encoded values

* Revert "replace with global tracing"

This reverts commit 8824a60deea54d9b437407a21c8ceaf6a1902ee5.

* Wasm Side Tracing

* tracing on wasm

* enable tracing wasm on node-runtime

* export all the macros in std

* tracing subscriber on wasm-side only

* pass spans and events over and record them

* reactivate previous code and  cleanup

* further cleaning up

* extend the span macros, activate through executive

* tracking the actual extrinsic, too

* style

* fixing tests

* spaces -> tabs

* attempting to reactivate params

* activate our tests in CI

* some passing

* tests passing

* with core lazy

* global tracer for wasm side with pass over

* fixing metadata referencing

* remove const_fn feature requirement

* reenable dispatch traces

* reset client tracing

* further cleaning up

* fixing runtime-test

* move tracing-build setup into runtime-test

* Merge DebugWriter from tracing and frame-support, move to sp-std

* remove dangling fixme

* Docs for tracing primitives

* cleaning up a bit more

* Wasm interface docs

* optimise docs.rs setup

* adding tracing flags to uncomment

* remove brace

* fixing imports

* fixing broken syntax

* add required modules

* nicer formatting

* better target management

* adding low level storage tracing events into frame

* add custom Debug impl for WasmMetadata

* cloning profiler

* adding info about cloning profiler

* using in-scope for within calls

* proper time tracing, cleaning up println

* allow to disable tracing on runtime_interface-macro

* disable tracing for wasm-tracing-interface

* simplify wasm-tracing-api

* update client to new interface

* fixing docs and tests for sp-tracing

* update integration tests

* re-activating enter_span

* dropping FIXME, it's documented

* fix formatting

* fix formatting

* fix imports

* more debug info

* inform wasm about it being disabled by returning 1

* only one tracer, but enabled multi-all support

* make trait pub again for tests

* Apply suggestions from code review

Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>

* fixing wasm doc tests for proper usage

* remove unnecessary import

* fixing formatting

* minor style fixes

* downgrading wabt

* update error message for UI

* Fix interface test

* next attempt to fix macros

* geee

* revert tracing on hashed for future PR

* remove local macros, use originals

* we are able to convert to static items

* implement more WasmValue types

* adding support to convert str, debug and encoded values

* more minor fixes

* revert unsafe 'static making

* fix indentation

* remove commented lines

* bump all them tracing versions

* cleaning up docs and info

* document new flag

* the new layered system handles span cloning better

* Apply suggestions from code review

Co-authored-by: David <dvdplm@gmail.com>

Co-authored-by: Matt Rutherford <mattrutherford@users.noreply.github.com>
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
Co-authored-by: David <dvdplm@gmail.com>
2020-09-18 06:10:59 +00:00
Wei Tang 9fdd4672b0 Use tracing-based subscriber logging (#6825)
* init_logger: switch from log-based to tracing-based and add compatibility layer

* Move tracing profiling subscriber related config realization

* sp-tracing: change profiling to be a layer instead of a subscriber

* Enable profiling layer in cli

* Change all test env_logger init to sp_tracing::try_init_simple

* Remove all local env_logger dependency

* Add missing tracing-subscriber dependency

* frame-sudo: fix tests

* frame-support: fix tests

* Fix frame/pallet and executor tests

* Fix the remaining tests

* Use subscriber's try_init as recommended by @davidbarsky

* Be explict that the tracing-log feature is needed

* Set subscriber writer to stderr

* Shorter line width

* Update cargo lock tracing version

* Fix sc_tracing crate compile

* Fix sc_authority_discovery crate test

* unremove default-features

* Leave enabled to default true

* Warn if global default cannot be set

* Fix unused import

* Remove unused PROXY_TARGET

* Change all reference from rc5 to rc6

* Change all reference of rc2 to rc6

* Fix styling

* Fix typo

* make logger init error'ing

* re-fixing the test issue

Co-authored-by: Benjamin Kampmann <ben@parity.io>
2020-09-17 11:04:43 +02:00
Sergei Shulepov a137160924 Change wabt to wat (#7050) 2020-09-08 22:49:54 +02:00
mattrutherford 9442c19b48 Implement tracing::Event handling & parent_id for spans and events (#6672)
* implement events handling, implement parent_id for spans & events

* add events to sp_io::storage

* update test

* add tests

* adjust limit

* let tracing crate handle parent_ids

* re-enable current-id tracking

* add test for threads with CurrentSpan

* fix log level

* remove redundant check for non wasm traces

* remove duplicate definition in test

* Adding conditional events API

* prefer explicit parent_id over current,

enhance test

* limit changes to client::tracing event implementation

* remove From impl due to fallback required on parent_id

* implement SPAN_LIMIT

change event log output

* change version of tracing-core

* update dependancies

* revert limit

* remove duplicate dependency

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Matt Rutherford <mattrutherford@users.noreply.github.com>
Co-authored-by: Benjamin Kampmann <ben@parity.io>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-08-12 11:53:21 +00:00
Wei Tang ac8de557d6 Better handling of stable-only build (#6569)
* Better handling of stable-only build

* Fix node template build

* Fix wasm builder node-template version mismatch

* Fix load_spec error

* Add , in parameter

* Add descrptive panic messages in tests

* Add descriptive tests in node/executor benches

* Fix missing compact_code_unwrap

* Add missing wasm_binary_unwrap function for executor integration test

* Only define import_sp_io in no_std

* Small Cargo.toml styling fix

* Bump wasm-builder to 2.0.0

* Fix all `with_wasm_builder_from_crates` version in Substrate

* Use `with_wasm_builder_from_crates` for node-template

Co-authored-by: Gavin Wood <gavin@parity.io>
2020-07-22 21:40:11 +02:00
mattrutherford c4dd079119 Tracing values (#6679)
* Switch to serde_json::Value for Visitor values

* add tests

* switch Visitor to use Map

* refactor change names

* \n

* update integration test

* use discrete maps for each type of recorded value

* update integration test

* add docs

* Update client/tracing/src/lib.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* add docs

Co-authored-by: Matt Rutherford <mattrutherford@users.noreply.github.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2020-07-22 11:29:33 +00:00
David d73de3bed7 Fix mocking multiple http calls in the same function call (#6510)
* Fix mocking multiple http calls in the same function call

Fixes an issue where a function call would perform more than one http request and wait for each to complete before proceeding. The `RequestId` comes from the length of the `requests` collection in the `OffchainState` and if a request is completed before the next one starts it will be removed and the "next expected" will be off by one. This PR tries to fix that by using a request counter that tracks how many requests have been performed so that we can `remove()` items from the `expected_requests` at the right index.

I suspect that this is a sub-optimal soluton and perhaps requests and their mocks should live side by side in the same collection, e.g. in a tuple of `(PendingRequest, Option<ExpectedRequest>)`.

* Update primitives/core/src/offchain/testing.rs

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* Update primitives/core/src/offchain/testing.rs

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* Panic on overflow

* Update primitives/core/src/offchain/testing.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Use a Deque and push/pop expected requests

* fix test

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-07-01 10:22:47 +02:00
mattrutherford 74a989f353 Runtime interface to add support for tracing from wasm (#6381)
* Add span recording to tracing implementation

* Add tracing proxy

* switch to rustc_hash::FxHashMap

* Replace lazy_static and hashmap with thread_local and vec.

* fix marking valid span as invalid while removing invalid spans

* refactor, add wasm_tracing module in `support`

* update registered spans

* tidy up

* typos

* refactor

* update flag name to signal lost trace - `is_valid_trace`

* update flag name to signal lost trace - `is_valid_trace`

* update docs

* update docs

* Use tracing Field recording to store the actual `name` and `target`
from wasm traces.

* fix debug log in subscriber + small refactor

* add tests

* handle misuse in case trying to exit span not held

* Implement filter for wasm traces, simplify field recording for primitive types

* remove superfluous warning

* update docs

* Update primitives/tracing/src/proxy.rs

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* update docs, apply suggestions

* move Proxy from thread_local to `Extension`, rename macro

* fix test

* unify native & wasm span macro calls

* implement wasm tracing control facility in primitives and frame

* add cli flag `--wasm-tracing`

* fix

* switch to `Option<u4>` (possible performance degradation), switch
to static mut bool

* performance improvement using u64 vs Option<u64>

* performance improvement moving concat to client

* update docs

* Update client/cli/src/params/import_params.rs

Co-authored-by: Cecile Tonglet <cecile@parity.io>

* performance improvement

* Revert "performance improvement"

This reverts commit 55ff8817a86302cd93bb6197eb4ca5bc7f4fb524.

* small refactor

* formatting

* bump impl_version

* Update client/cli/src/config.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* update docs

* small fixes, remove pub static

* nit

* add integration tests and refactor Subscriber

* tests

* revert formatting

* try fix test that works locally but not in CI

* try fix test that works locally but not in CI

* debug test that works locally but not in CI

* fix test that works locally but not in CI

* remove pub visibility from bool in runtime

* make TracingSpanGuard #[cfg(not(feature = "std"))], update docs, comments

* make TracingProxy drop implementation conditional on !empty state

* add docs for TraceHandler

* remove blank line

* update expect message

* update tests

* rename cli option to tracing_enable_wasm

* rename cli option to tracing_enable_wasm

* fix

* ensure wasm-tracing features are wasm only

* bump impl_version

* bump impl_version

* add `"pallet-scheduler/std"` to `[features]` `std` in node/runtime

* refactor service to remove sp_tracing dependency

* refactor: line width, trait bounds

* improve LogTraceHandler output

* fix test

* improve tracing log output

* Apply suggestions from code review

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* swap wasm indication from trace name to a separate value

* Update client/tracing/src/lib.rs

* add docs

* remove runtime features

remove wasm_tracing option from CLI

remove wasm_tracing flag from ProfilingSubscriber

Co-authored-by: Matt Rutherford <mattrutherford@users.noreply.github.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Cecile Tonglet <cecile@parity.io>
2020-06-18 09:44:03 +02:00
Pierre Krieger 717fa95bc7 Restore the empty line after the license (#6088) 2020-05-20 13:08:27 +00:00
Arkadiy Paronyan d1bf24caef Fix parallel code execution in wasmtime (#6055)
* Bump wasmtime version

* Proper test
2020-05-18 18:42:25 +02:00
Nikolay Volf d5411969ac fix whitespace (#6062) 2020-05-18 13:51:28 +02:00
Benjamin Kampmann be8c96adec Relax substrate licensing scheme (#5947) 2020-05-15 13:21:56 +02:00
Arkadiy Paronyan e0b17dd7a9 Bump wasmtime again (#5950)
* Update wasmtime (#5822)

* update wasmtime

* update tests

* Update client/executor/wasmtime/src/host.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* wip

* use master-candidate

* update with patches

* update versions

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Bump version

* Added parallel execution test

Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-05-08 22:22:54 +02:00
Bastian Köcher eff1680aed Revert "Update wasmtime (#5822)" (#5935)
This reverts commit e5163380e7.

We found some bugs that we first need to fix.
2020-05-07 11:47:38 +02:00
Nikolay Volf e5163380e7 Update wasmtime (#5822)
* update wasmtime

* update tests

* Update client/executor/wasmtime/src/host.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* wip

* use master-candidate

* update with patches

* update versions

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-05-04 15:24:05 +02:00
Bernhard Schuster 72ee7d5797 feat/ocw/bookkeeping (#5200)
Co-Authored-By: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
2020-04-24 16:46:19 +02:00
Bastian Köcher 46677555ac Allow missing functions when checking the new runtime's version (#5741) 2020-04-23 14:54:52 +02:00
cheme 4ffcf98d8d Child trie api changes BREAKING (#4857)
Co-Authored-By: thiolliere <gui.thiolliere@gmail.com>
2020-04-20 15:21:22 +02:00
Stanislav Tkach 93ee3104e7 Use CLI to configure max instances cache (#5177)
* Use CLI to configure max instances cache

* Fix tests

* Move default value into CLI

* Use SmallVec

* Apply review comments

* Get rid of `SmallVec`

Co-authored-by: Bastian Köcher <git@kchr.de>
2020-03-11 09:43:37 +01:00
Bastian Köcher 1cfcf5cbfe Don't include :code by default in storage proofs (#5179)
* Don't include `:code` by default in storage proofs (#5060)

* Adds test to verify that the runtime currently is always contained in
the proof

* Start passing the runtime wasm code from the outside

* Fix compilation

* More build fixes

* Make the test work as expected now :)

* Last fixes

* Fixes benchmarks

* Review feedback

* Apply suggestions from code review

Co-Authored-By: Sergei Pepyakin <sergei@parity.io>

* Review feedback

* Fix compilation

Co-authored-by: Sergei Pepyakin <s.pepyakin@gmail.com>

* Fix compilation and change the way `RuntimeCode` works

* Fix tests

* Switch to `Cow`

Co-authored-by: Benjamin Kampmann <ben@gnunicorn.org>
Co-authored-by: Sergei Pepyakin <s.pepyakin@gmail.com>
2020-03-10 11:13:20 +01:00
Benjamin Kampmann d3208aa7bc Better wasm instance cache (#5109)
* Wasm instance cache

* Reduce slot locking

* Fixed test

* Dispose of instance in case of error

* Fixed benches

* Style, comments, some renames

* Replaced upgradable lock with mutex

* Bump dependencies

* Re-export CallInWasm

* Update client/executor/src/wasm_runtime.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/executor/src/native_executor.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/executor/src/native_executor.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/executor/src/wasm_runtime.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/executor/wasmtime/src/runtime.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/executor/src/wasm_runtime.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/executor/src/wasm_runtime.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/executor/src/wasm_runtime.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Indents

* Whitespace

* Formatting

* Added issue link

Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com>
Co-authored-by: Gavin Wood <github@gavwood.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-03-05 14:02:04 +01:00
Benjamin Kampmann 5a33228ea9 Reduce usage of Blake2Hasher (#5132)
This reduces the usage of `Blake2Hasher` in the code base and replaces
it with `BlakeTwo256`. The most important change is the removal of the
custom extern function for `Blake2Hasher`. The runtime `Hash` trait is
now also simplified and directly requires that the implementing type
implements `Hashable`.
2020-03-05 08:51:03 +01:00
Bastian Köcher d3fa8c91af Adds a test to ensure that we clear the heap between calls into runtime (#4903)
* Adds a test to ensure that we clear the heap between calls into runtime

The tests shows that we currently not clearing the heap in wasmtime.
For now we don't run the test for wasmtime.

* Fix compilation
2020-02-14 01:42:27 +01:00
Sergei Pepyakin 49af986ad4 executor: Migrate wasmtime backend to a high-level API (#4686)
* Migrate wasmtime backend to wasmtime-api

* Port to a newer version of wasmtime

* Update to the latest changes.

* Rejig the sandbox module a bit

* Materialze

* Fixes.

* executor wasm_runtime fix

* Refactor everything

* More refactoring

* Even more refactorings

* More cleaning.

* Update to the latest wasmtime

* Reformat

* Renames

* Refactoring and comments.

* Docs

* Rename FunctionExecutor to host.

* Imrpove docs.

* fmt

* Remove panic

* Assert the number of arguments are equal between wasmtime and hostfunc.

* Comment a possible panic if there is no corresponding value variant.

* Check signature of the entrypoint.

* Use git version of wasmtime

* Refine and doc the sandbox code.

* Comment RefCells.

* Update wasmtime to the latest-ish master.

This may solve a problem with segfaults.

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Use full SHA1 hash of wasmtime commit.

* Add a panic message.

* Add some documentation

* Update wasmtime version to include SIGSEGV fix

* Update to crates.io version of wasmtime

* Make it work.

* Move the creation of memory into `InstanceWrapper::new`

* Make `InstanceWrapper` !Send & !Sync

* Avoid using `take_mut`

* Update client/executor/wasmtime/Cargo.toml

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Limit maximum size of memory.

* Rename `init_state` to `with_initialized_state`

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2020-02-13 14:54:19 +01:00
Bastian Köcher 4254fbf56d Don't expose Benchmarking host functions by default (#4875)
* Don't expose `Benchmarking` host functions by default

* Fix tests

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2020-02-10 13:13:04 +01:00