The PVF host is designed to avoid spawning tasks to minimize knowledge
of outer code. Using `async_std::task::spawn` (or Tokio's counterpart)
deemed unacceptable, `SpawnNamed` undesirable. Instead there is only one
task returned that is spawned by the candidate-validation subsystem.
The tasks from the sub-components are polled by that root task.
However, the way the tasks are bundled was incorrect. There was a giant
select that was polling those tasks. Particularly, that implies that as soon as
one of the arms of that select goes into await those sub-tasks stop
getting polled. This is a recipe for a deadlock which indeed happened
here.
Specifically, the deadlock happened during sending messages to the
execute queue by calling
[`send_execute`](https://github.com/paritytech/polkadot/blob/a68d9be35656dcd96e378fd9dd3d613af754d48a/node/core/pvf/src/host.rs#L601).
When the channel to the queue reaches the capacity, the control flow is
suspended until the queue handles those messages. Since this code is
essentially reached from [one of the select
arms](https://github.com/paritytech/polkadot/blob/a68d9be35656dcd96e378fd9dd3d613af754d48a/node/core/pvf/src/host.rs#L371),
the queue won't be given the control and thus no further progress can be
made.
This problem is solved by bundling the tasks one level higher instead,
by `selecting` over those long-running tasks.
We also stop treating returning from those long-running tasks as error
conditions, since that can happen during legit shutdown.
* paras: `include_pvf_check_statement` rt bench
Resolves#4933
This PR adds a benchmark for the `include_pvf_check_statement`
dispatchable. This is a necessary step to make it work without
modifications. That enables us to proceed with testing on Versi.
This introduces 5 new benchmarks. Those measure performance of the
`include_pvf_check_statement` under 2 different conditions:
1. regular vote submission. That's the common case.
2. submission of the last vote. That happens only once and leads to a
heavy finalization stage.
There are 2 different types of finalization (one for onboarding, one for
upgrading) and there are two outcomes: accepted and rejected. Those 4
are similar but I decided to cover them all and assign the maximum of
all 4. This is to avoid a situation when one of those paths becomes more
heavier than others and opens up an attack venue.
The regular vote submission weight is drastically different from the
submission last vote weight. That's why in case during runtime
finalization was not executed the weight consumed value will be lowered
down to the regular vote submission.
The finalization weight is proportional to the number of "causes", i.e.
the events that caused the PVF pre-checking vote in the first place, and
here we assume that the maximum number of causes is 100.
Theoretically, there is nothing that prevents an adversary to
register/upgrade to more than 100 parachains. In that case, the consumed
weight will be lower than the actual time consumed by the finalization
process. That can enable a DoS vector.
However, practically, it is not very possible. Right now it is very
expensive to call `schedule_para_initialize` because it requires a very
large lock up of funds. Moreover, finalizing a vote with 100 causes
leads to around 31ms time spent. Finalizing more will require more time.
However, finalizing with 200 causes will cause ≈62ms delay. This is not
that bad since even though we had a full block and the adversary tried
to finalize 200 causes it won't be able to even exceed the operational
extrinsic boundary of 250ms and even if so it won't make big difference.
That said, this should be addressed later on, esp. when we enable
parathreads, which will make creating causes easier. One of potential
solutions will be shifting the logic of finalization into
`on_initialize`/`on_finalize`. Another is to create a maximum number of
causes and then reject upgrades or onboardings if that was reached.
* cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras.rs
* cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras.rs
* cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_paras.rs
* cargo run --quiet --profile=production --features runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/rococo/src/weights/runtime_parachains_paras.rs
* Fix import error
Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Robert Klotzner <robert.klotzner@gmx.at>
Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com>
This commit adds a new primitive called `ParachainsCache` to manipulate
the `Parachains` storage entry in a more convenient way.
Then, on top of that, this commit changes the logic of
`initialize_para_now` so that it is identical to what is used for
initialization of onboarding.
* Introduce an IndexedEvent
* Plumb topics through the Ext interface.
* Add topics to ext_deposit_event
* Charging for events.
* Check the number of topics.
* Check for duplicate topics.
* Bump API version.
* Move derive(*Eq) under test.
* Use sorting for finding duplicates.
* Sketch of indexed events.
* Get EventIndex by holding another variable.
* Add some docs.
* Use DoubleMap to store reverse topic index
* Implement StorageDoubleMap::append
* Use append for EventTopics.
* Refactor.
* Avoid `mutate`
* Docs.
* Add topics to EventRecord
* Update tests.
* Rebuild.
* Bump version.
* Event topics test.
* Mix in BlockNumber to distinguish updates
* Fix srml-system test.
* Post merge fixes.
* Comments/TODO.
* Intro `ChildStorageKey` for checked child keys
* Get rid of Into in Externalities trait
* Use Cow in ChildStorageKey
* Fix tests for state-machine.
* Clean
* child_storage_root always return a value
* Don't return Option from Ext::child_storage_root
* Return 42 in child_storage_root
* Return CHILD_STORAGE_KEY_PREFIX from trie id gen
* Bump spec and impl version.
* Require `:default:` in `is_child_trie_key_valid`
* Add `default:` prefix.
* Introduce `into_owned` for `ChildStorageKey`.
* Add documentation.
* Fix state-machine tests
* Remove outdated TODO
I check out with Emeric and he is ok with that
* child_storage_root is infailable
* Nit
* Move assert after check.
* Apply suggestions from @DemiMarie-parity
Co-Authored-By: pepyakin <s.pepyakin@gmail.com>
* Formatting nit in core/executor/src/wasm_executor.rs
Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>
* Formatting nits from @thiolliere
* Declare storage in contracts module
This adds storage declarations of the contract module to the runtime metadata and this ultimately gives ability to inspect the storage of contract module (except contracts storage) for polkadot ui
* Bump the runtime version to 17.
* Rebuild the binaries.
* Rebuild
* Rebuild [2]
* WIP
* Use system::Origin::Signed as an origin
* Add a vm test for ext_dispatch_call
* Take fee for dispatching a Call
# Conflicts:
# node/runtime/wasm/target/wasm32-unknown-unknown/release/node_runtime.compact.wasm
* Clean & Rebuild
# Conflicts:
# node/runtime/wasm/target/wasm32-unknown-unknown/release/node_runtime.compact.wasm
* Dispatch call test.
* Rebuild the runtime.
* Fix the length of the buffer.
* Rebuild binaries.
* Introduce data and salt into ContractAddressFor
* Accept salt arg in ext_create.
* Integrate contracts into the demo runtime
* Make libcontract compile to wasm
* Remove salt parameter.
This now is concern of userspace.
* Rebuild binaries.
* Rename ext_transfer → ext_call
There are several tests where gas was changed. This is due to shrinking
the name and consequently a size of contracts.
* Fix macros
Specifically, make fixes that allows to use a function without arguments (apart from ctx).
For example:
```rust
ext_return_42(ctx) -> u32 => {
Ok(42)
}
```
Also, add impl ConvertibleToWasm for u64.
* Implement passing input data buffer from top-level
This allows the transaction initiator to pass some input data that can be accessed from the executed contract.
However, this only works for the top-level contract. The contract still cannot pass input data to the callee contract. The reason for that is that API doesn't support this presently.
This will be addressed in the following commits.
* Input data passing thru `ext_call`
`ext_call` now takes a buffer (or more specifically, pointer to the start of and size of a buffer).
This allows contracts to pass input data while calling into other contracts.
* Input data passing thru `ext_create`
Same as previous commit but for `ext_create`.
* Trap on decoding error instead of panicking
* Don't cascade failure in ext_call
Instead, if callee doesn't complete successfuly, return a non-zero status code.
This will prevent cascading traps up to the top-level. Due to this some tests were altered so that they now expect successful transfer instead of a failure.
* Don't cascade failure in ext_create
Same as previous commit but for ext_create.
* Specify how much gas to allot for a call
* Specify how much gas to allot for a create
* Update tests.rs
* Convert T in vm::Ext to a associated type
* WIP
* Fix BTreeMap
* Extract prepare module from vm.
* Move tests as well.
* Fix doc comment.
* macro for env defintion
* Fix prepare tests.
* Clean up
* Renamings
* Refactor scan_imports
* Improve docs
* Docs.
* Add some tests for sandbox module
* Clean up
* Use Error::Instantiate instead of ::Deserialize
* Add test for imports
* Add wrong sig import
* Clean up
* Rebuild binaries.
* Use "another_module" instead of obscure "vne"
Since "vne" looks like an actual typo
* Don't use --export-table anymore
Because it was turned on by default in the recent nightlies.
See https://github.com/rust-lang/rust/pull/53237
* use_extern_macros stabilization
With recent nightlies rustc produces a warning
```
the feature `use_extern_macros` has been stable since 1.30.0 and no longer requires an attribute to enable
```
* decl_module and extract runtime mod
* Invert dependency staking←→contract
* Remove CodeOf
* Remove StorageOf and move double_map
* Comment staking test
* Clean
* Add gas_price and gas_limit
* Commit.
* Renames
* Params
* WIP
* Rename transfer to call
* WIP
* Rebuild binaries.
* WIP
* Backport ctro changes
* Call wiring
* Commit overlay.
* Rename merge → commit, into_state → ..._change_set
* WIP
* Contract creation routines
* Set code of the created account.
* Fix the ID of `create` Call
* Fix most of the warning.
* Add the simplest test in the contract crate
* Transfers work!
* Add contract_create test.
* Clean
* Add top-level create test
* Clean a bit.
* Pass gas_limit and data via create.
* Introduce OnAccountKill callback in staking
* Hook up OnAccountKill
* Comments
* Pay for gas.
* Refund unused gas in call
* Tests for zero call and zero endownment.
* Add todo about rewriting docs
* Pay for gas in create transactions
* Fix refunds
* Clean unrelevant comments
* fixup! Fix refunds
* fixup! Clean unrelevant comments
* Move DetermineContractAddress to contract
Also restore account removal test
* fixup! Clean unrelevant comments
* Inline effect_transfer, remove effect_create
Remove account_db!
* Use own new_test_ext.
* Don't account for liability
* Add some docs
* Move contract_fee into contract module
* Take GasMeter in vm::execute
* Use GasMeter throughout contract module for meter
* gas module refactoring
* Clean
* Add base call fee
* note about gas price should be taken from storage
* Add base fee for create
* Rename send → call
* Clean
* Take fee expressed in dots in gas
* Add Checked{Add,Sub,Mul,Div} to SimpleArithmetic
* Make Gas generic
* Store {call,create}_base_fee in storage
* Clean
* Rename buy_gas
* Store gas_price in the storage
* Remove unneeded comment.
* Bail out if contract already has code.
* Todos
* Refund even if top-level contract fails.
* Fix error msg
* Fix caller issue
* Extract tests module
* Add max_depth var in storage
* Remove left over gas_left
* Refactor exec
* Add test oog test.
* set_free_balance_creating
* Docs and comments.
* Update storage roots because of ContractFee move
* Rebuild binaries.
* Simplify vm code.
* Wrapping.
* Refactor a bit.
* Typo
* UpdateBalanceOutcome enum
* Style grumbles.
* Rebuild binaries.
* Always consume the given amount of gas.
* [skip ci] endownment → endowment
* Rename `AccountId` generic in on_account_kill
* Fix Cargo.lock
* Refine docs for gas meter.
* [skip ci] Add comments for gas module
* Directly assign to `return_data` at declaration
* Use slices instead of vecs to pass the input data
* Add todo about passing return data without copy
* Use checked_add instead of add with overflow
* Use return_data directly.
* Rebuild binaries.
* Rebuild binaries.
* decl_module and extract runtime mod
* Invert dependency staking←→contract
* Remove CodeOf
* Remove StorageOf and move double_map
* Comment staking test
* Clean
* Add gas_price and gas_limit
* Commit.
* Renames
* Params
* WIP
* Rename transfer to call
* WIP
* Rebuild binaries.
* WIP
* Backport ctro changes
* Call wiring
* Commit overlay.
* Rename merge → commit, into_state → ..._change_set
* WIP
* Contract creation routines
* Set code of the created account.
* Fix the ID of `create` Call
* Fix most of the warning.
* Add the simplest test in the contract crate
* Transfers work!
* Add contract_create test.
* Clean
* Add top-level create test
* Clean a bit.
* Pass gas_limit and data via create.
* Introduce OnAccountKill callback in staking
* Hook up OnAccountKill
* Comments
* Pay for gas.
* Refund unused gas in call
* Tests for zero call and zero endownment.
* Add todo about rewriting docs
* Pay for gas in create transactions
* Fix refunds
* Clean unrelevant comments
* fixup! Fix refunds
* fixup! Clean unrelevant comments
* Move DetermineContractAddress to contract
Also restore account removal test
* fixup! Clean unrelevant comments
* Inline effect_transfer, remove effect_create
Remove account_db!
* Use own new_test_ext.
* Don't account for liability
* Add some docs
* Move contract_fee into contract module
* Take GasMeter in vm::execute
* Use GasMeter throughout contract module for meter
* gas module refactoring
* Clean
* Add base call fee
* note about gas price should be taken from storage
* Add base fee for create
* Rename send → call
* Clean
* Take fee expressed in dots in gas
* Add Checked{Add,Sub,Mul,Div} to SimpleArithmetic
* Make Gas generic
* Store {call,create}_base_fee in storage
* Clean
* Rename buy_gas
* Store gas_price in the storage
* Remove unneeded comment.
* Bail out if contract already has code.
* Todos
* Refund even if top-level contract fails.
* Fix error msg
* Fix caller issue
* Extract tests module
* Add max_depth var in storage
* Remove left over gas_left
* Refactor exec
* Add test oog test.
* set_free_balance_creating
* Docs and comments.
* Update storage roots because of ContractFee move
* Rebuild binaries.
* Simplify vm code.
* Wrapping.
* Refactor a bit.
* Typo
* UpdateBalanceOutcome enum
* Style grumbles.
* Rebuild binaries.
* Always consume the given amount of gas.
* [skip ci] endownment → endowment
* Rename `AccountId` generic in on_account_kill
* Fix Cargo.lock
* Refine docs for gas meter.
* [skip ci] Add comments for gas module
* Directly assign to `return_data` at declaration
* Use slices instead of vecs to pass the input data
* Add todo about passing return data without copy
* Use checked_add instead of add with overflow
* Use return_data directly.
* Rebuild binaries.
* Rebuild binaries.
* ok_or → ok_or_else
* Correct the sides of the given and expected args
* Indent } properly
* Convert identation from tabs to spaces.
* Convert identation from tabs to spaces. [2]
* WIP
* Iteration over all keys with the specified prefix
* Add clear_prefix in runtime-io
* Introduce a custom storage impl: Double Map
* Remove prefix
* Impl for_keys_with_prefix for light client
* Fix wasm_executor
* Test storage removal leads to removal of stroage
* Check for ok result in storage tests.
* Add docs.
* Remove commented code under decl_storage!
* Add clear_prefix test in runtime-io
* Add test for wasm_executor
* Prefix walking test.
* Rebuild binaries.
* WIP
* Iteration over all keys with the specified prefix
* Add clear_prefix in runtime-io
* Introduce a custom storage impl: Double Map
* Remove prefix
* Impl for_keys_with_prefix for light client
* Fix wasm_executor
* Test storage removal leads to removal of stroage
* Check for ok result in storage tests.
* Add docs.
* Remove commented code under decl_storage!
* Add clear_prefix test in runtime-io
* Add test for wasm_executor
* Prefix walking test.
* Rebuild binaries.
