**Update:** Pushed additional changes based on the review comments.
**This pull request fixes various spelling mistakes in this
repository.**
Most of the changes are contained in the first **3** commits:
- `Fix spelling mistakes in comments and docs`
- `Fix spelling mistakes in test names`
- `Fix spelling mistakes in error messages, panic messages, logs and
tracing`
Other source code spelling mistakes are separated into individual
commits for easier reviewing:
- `Fix the spelling of 'authority'`
- `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'`
- `Fix the spelling of 'prev_enqueud_messages'`
- `Fix the spelling of 'endpoint'`
- `Fix the spelling of 'children'`
- `Fix the spelling of 'PenpalSiblingSovereignAccount'`
- `Fix the spelling of 'PenpalSudoAccount'`
- `Fix the spelling of 'insufficient'`
- `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'`
- `Fix the spelling of 'subtracted'`
- `Fix the spelling of 'CandidatePendingAvailability'`
- `Fix the spelling of 'exclusive'`
- `Fix the spelling of 'until'`
- `Fix the spelling of 'discriminator'`
- `Fix the spelling of 'nonexistent'`
- `Fix the spelling of 'subsystem'`
- `Fix the spelling of 'indices'`
- `Fix the spelling of 'committed'`
- `Fix the spelling of 'topology'`
- `Fix the spelling of 'response'`
- `Fix the spelling of 'beneficiary'`
- `Fix the spelling of 'formatted'`
- `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'`
- `Fix the spelling of 'succeeded'`
- `Fix the spelling of 'reopened'`
- `Fix the spelling of 'proposer'`
- `Fix the spelling of 'InstantiationNonce'`
- `Fix the spelling of 'depositor'`
- `Fix the spelling of 'expiration'`
- `Fix the spelling of 'phantom'`
- `Fix the spelling of 'AggregatedKeyValue'`
- `Fix the spelling of 'randomness'`
- `Fix the spelling of 'defendant'`
- `Fix the spelling of 'AquaticMammal'`
- `Fix the spelling of 'transactions'`
- `Fix the spelling of 'PassingTracingSubscriber'`
- `Fix the spelling of 'TxSignaturePayload'`
- `Fix the spelling of 'versioning'`
- `Fix the spelling of 'descendant'`
- `Fix the spelling of 'overridden'`
- `Fix the spelling of 'network'`
Let me know if this structure is adequate.
**Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`,
`Merkelization`, `Merkleization`, is somewhat inconsistent but I left it
as it is.
~~**Note:** In some places the term `Receival` is used to refer to
message reception, IMO `Reception` is the correct word here, but I left
it as it is.~~
~~**Note:** In some places the term `Overlayed` is used instead of the
more acceptable version `Overlaid` but I also left it as it is.~~
~~**Note:** In some places the term `Applyable` is used instead of the
correct version `Applicable` but I also left it as it is.~~
**Note:** Some usage of British vs American english e.g. `judgement` vs
`judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc.
are both present in different places, but I suppose that's
understandable given the number of contributors.
~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it
triggers errors in CI when I make changes to it, so I left it as it
is.~~
Closes#2160
First part of [Extrinsic
Horizon](https://github.com/paritytech/polkadot-sdk/issues/2415)
Introduces a new trait `TransactionExtension` to replace
`SignedExtension`. Introduce the idea of transactions which obey the
runtime's extensions and have according Extension data (né Extra data)
yet do not have hard-coded signatures.
Deprecate the terminology of "Unsigned" when used for
transactions/extrinsics owing to there now being "proper" unsigned
transactions which obey the extension framework and "old-style" unsigned
which do not. Instead we have __*General*__ for the former and
__*Bare*__ for the latter. (Ultimately, the latter will be phased out as
a type of transaction, and Bare will only be used for Inherents.)
Types of extrinsic are now therefore:
- Bare (no hardcoded signature, no Extra data; used to be known as
"Unsigned")
- Bare transactions (deprecated): Gossiped, validated with
`ValidateUnsigned` (deprecated) and the `_bare_compat` bits of
`TransactionExtension` (deprecated).
- Inherents: Not gossiped, validated with `ProvideInherent`.
- Extended (Extra data): Gossiped, validated via `TransactionExtension`.
- Signed transactions (with a hardcoded signature).
- General transactions (without a hardcoded signature).
`TransactionExtension` differs from `SignedExtension` because:
- A signature on the underlying transaction may validly not be present.
- It may alter the origin during validation.
- `pre_dispatch` is renamed to `prepare` and need not contain the checks
present in `validate`.
- `validate` and `prepare` is passed an `Origin` rather than a
`AccountId`.
- `validate` may pass arbitrary information into `prepare` via a new
user-specifiable type `Val`.
- `AdditionalSigned`/`additional_signed` is renamed to
`Implicit`/`implicit`. It is encoded *for the entire transaction* and
passed in to each extension as a new argument to `validate`. This
facilitates the ability of extensions to acts as underlying crypto.
There is a new `DispatchTransaction` trait which contains only default
function impls and is impl'ed for any `TransactionExtension` impler. It
provides several utility functions which reduce some of the tedium from
using `TransactionExtension` (indeed, none of its regular functions
should now need to be called directly).
Three transaction version discriminator ("versions") are now
permissible:
- 0b000000100: Bare (used to be called "Unsigned"): contains Signature
or Extra (extension data). After bare transactions are no longer
supported, this will strictly identify an Inherents only.
- 0b100000100: Old-school "Signed" Transaction: contains Signature and
Extra (extension data).
- 0b010000100: New-school "General" Transaction: contains Extra
(extension data), but no Signature.
For the New-school General Transaction, it becomes trivial for authors
to publish extensions to the mechanism for authorizing an Origin, e.g.
through new kinds of key-signing schemes, ZK proofs, pallet state,
mutations over pre-authenticated origins or any combination of the
above.
## Code Migration
### NOW: Getting it to build
Wrap your `SignedExtension`s in `AsTransactionExtension`. This should be
accompanied by renaming your aggregate type in line with the new
terminology. E.g. Before:
```rust
/// The SignedExtension to the basic transaction logic.
pub type SignedExtra = (
/* snip */
MySpecialSignedExtension,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, SignedExtra>;
```
After:
```rust
/// The extension to the basic transaction logic.
pub type TxExtension = (
/* snip */
AsTransactionExtension<MySpecialSignedExtension>,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, TxExtension>;
```
You'll also need to alter any transaction building logic to add a
`.into()` to make the conversion happen. E.g. Before:
```rust
fn construct_extrinsic(
/* snip */
) -> UncheckedExtrinsic {
let extra: SignedExtra = (
/* snip */
MySpecialSignedExtension::new(/* snip */),
);
let payload = SignedPayload::new(call.clone(), extra.clone()).unwrap();
let signature = payload.using_encoded(|e| sender.sign(e));
UncheckedExtrinsic::new_signed(
/* snip */
Signature::Sr25519(signature),
extra,
)
}
```
After:
```rust
fn construct_extrinsic(
/* snip */
) -> UncheckedExtrinsic {
let tx_ext: TxExtension = (
/* snip */
MySpecialSignedExtension::new(/* snip */).into(),
);
let payload = SignedPayload::new(call.clone(), tx_ext.clone()).unwrap();
let signature = payload.using_encoded(|e| sender.sign(e));
UncheckedExtrinsic::new_signed(
/* snip */
Signature::Sr25519(signature),
tx_ext,
)
}
```
### SOON: Migrating to `TransactionExtension`
Most `SignedExtension`s can be trivially converted to become a
`TransactionExtension`. There are a few things to know.
- Instead of a single trait like `SignedExtension`, you should now
implement two traits individually: `TransactionExtensionBase` and
`TransactionExtension`.
- Weights are now a thing and must be provided via the new function `fn
weight`.
#### `TransactionExtensionBase`
This trait takes care of anything which is not dependent on types
specific to your runtime, most notably `Call`.
- `AdditionalSigned`/`additional_signed` is renamed to
`Implicit`/`implicit`.
- Weight must be returned by implementing the `weight` function. If your
extension is associated with a pallet, you'll probably want to do this
via the pallet's existing benchmarking infrastructure.
#### `TransactionExtension`
Generally:
- `pre_dispatch` is now `prepare` and you *should not reexecute the
`validate` functionality in there*!
- You don't get an account ID any more; you get an origin instead. If
you need to presume an account ID, then you can use the trait function
`AsSystemOriginSigner::as_system_origin_signer`.
- You get an additional ticket, similar to `Pre`, called `Val`. This
defines data which is passed from `validate` into `prepare`. This is
important since you should not be duplicating logic from `validate` to
`prepare`, you need a way of passing your working from the former into
the latter. This is it.
- This trait takes two type parameters: `Call` and `Context`. `Call` is
the runtime call type which used to be an associated type; you can just
move it to become a type parameter for your trait impl. `Context` is not
currently used and you can safely implement over it as an unbounded
type.
- There's no `AccountId` associated type any more. Just remove it.
Regarding `validate`:
- You get three new parameters in `validate`; all can be ignored when
migrating from `SignedExtension`.
- `validate` returns a tuple on success; the second item in the tuple is
the new ticket type `Self::Val` which gets passed in to `prepare`. If
you use any information extracted during `validate` (off-chain and
on-chain, non-mutating) in `prepare` (on-chain, mutating) then you can
pass it through with this. For the tuple's last item, just return the
`origin` argument.
Regarding `prepare`:
- This is renamed from `pre_dispatch`, but there is one change:
- FUNCTIONALITY TO VALIDATE THE TRANSACTION NEED NOT BE DUPLICATED FROM
`validate`!!
- (This is different to `SignedExtension` which was required to run the
same checks in `pre_dispatch` as in `validate`.)
Regarding `post_dispatch`:
- Since there are no unsigned transactions handled by
`TransactionExtension`, `Pre` is always defined, so the first parameter
is `Self::Pre` rather than `Option<Self::Pre>`.
If you make use of `SignedExtension::validate_unsigned` or
`SignedExtension::pre_dispatch_unsigned`, then:
- Just use the regular versions of these functions instead.
- Have your logic execute in the case that the `origin` is `None`.
- Ensure your transaction creation logic creates a General Transaction
rather than a Bare Transaction; this means having to include all
`TransactionExtension`s' data.
- `ValidateUnsigned` can still be used (for now) if you need to be able
to construct transactions which contain none of the extension data,
however these will be phased out in stage 2 of the Transactions Horizon,
so you should consider moving to an extension-centric design.
## TODO
- [x] Introduce `CheckSignature` impl of `TransactionExtension` to
ensure it's possible to have crypto be done wholly in a
`TransactionExtension`.
- [x] Deprecate `SignedExtension` and move all uses in codebase to
`TransactionExtension`.
- [x] `ChargeTransactionPayment`
- [x] `DummyExtension`
- [x] `ChargeAssetTxPayment` (asset-tx-payment)
- [x] `ChargeAssetTxPayment` (asset-conversion-tx-payment)
- [x] `CheckWeight`
- [x] `CheckTxVersion`
- [x] `CheckSpecVersion`
- [x] `CheckNonce`
- [x] `CheckNonZeroSender`
- [x] `CheckMortality`
- [x] `CheckGenesis`
- [x] `CheckOnlySudoAccount`
- [x] `WatchDummy`
- [x] `PrevalidateAttests`
- [x] `GenericSignedExtension`
- [x] `SignedExtension` (chain-polkadot-bulletin)
- [x] `RefundSignedExtensionAdapter`
- [x] Implement `fn weight` across the board.
- [ ] Go through all pre-existing extensions which assume an account
signer and explicitly handle the possibility of another kind of origin.
- [x] `CheckNonce` should probably succeed in the case of a non-account
origin.
- [x] `CheckNonZeroSender` should succeed in the case of a non-account
origin.
- [x] `ChargeTransactionPayment` and family should fail in the case of a
non-account origin.
- [ ]
- [x] Fix any broken tests.
---------
Signed-off-by: georgepisaltu <george.pisaltu@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Nikhil Gupta <17176722+gupnik@users.noreply.github.com>
Co-authored-by: georgepisaltu <52418509+georgepisaltu@users.noreply.github.com>
Co-authored-by: Chevdor <chevdor@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Maciej <maciej.zyszkiewicz@parity.io>
Co-authored-by: Javier Viola <javier@parity.io>
Co-authored-by: Marcin S. <marcin@realemail.net>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Co-authored-by: Javier Bullrich <javier@bullrich.dev>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Vladimir Istyufeev <vladimir@parity.io>
Co-authored-by: Ross Bulat <ross@parity.io>
Co-authored-by: Gonçalo Pestana <g6pestana@gmail.com>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
Co-authored-by: Svyatoslav Nikolsky <svyatonik@gmail.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com>
Co-authored-by: ordian <write@reusable.software>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
Co-authored-by: Julian Eager <eagr@tutanota.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Dónal Murray <donal.murray@parity.io>
Co-authored-by: yjh <yjh465402634@gmail.com>
Co-authored-by: Tom Mi <tommi@niemi.lol>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Will | Paradox | ParaNodes.io <79228812+paradox-tt@users.noreply.github.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>
Co-authored-by: Joshy Orndorff <git-user-email.h0ly5@simplelogin.com>
Co-authored-by: PG Herveou <pgherveou@gmail.com>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Juan Girini <juangirini@gmail.com>
Co-authored-by: bader y <ibnbassem@gmail.com>
Co-authored-by: James Wilson <james@jsdw.me>
Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
Co-authored-by: asynchronous rob <rphmeier@gmail.com>
Co-authored-by: Parth <desaiparth08@gmail.com>
Co-authored-by: Andrew Jones <ascjones@gmail.com>
Co-authored-by: Jonathan Udd <jonathan@dwellir.com>
Co-authored-by: Serban Iorga <serban@parity.io>
Co-authored-by: Egor_P <egor@parity.io>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Evgeny Snitko <evgeny@parity.io>
Co-authored-by: Just van Stam <vstam1@users.noreply.github.com>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: gupnik <nikhilgupta.iitk@gmail.com>
Co-authored-by: dzmitry-lahoda <dzmitry@lahoda.pro>
Co-authored-by: zhiqiangxu <652732310@qq.com>
Co-authored-by: Nazar Mokrynskyi <nazar@mokrynskyi.com>
Co-authored-by: Anwesh <anweshknayak@gmail.com>
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Sam Johnson <sam@durosoft.com>
Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Jegor Sidorenko <5252494+jsidorenko@users.noreply.github.com>
Co-authored-by: Muharem <ismailov.m.h@gmail.com>
Co-authored-by: joepetrowski <joe@parity.io>
Co-authored-by: Alexandru Gheorghe <49718502+alexggh@users.noreply.github.com>
Co-authored-by: Gabriel Facco de Arruda <arrudagates@gmail.com>
Co-authored-by: Squirrel <gilescope@gmail.com>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
Co-authored-by: georgepisaltu <george.pisaltu@parity.io>
Co-authored-by: command-bot <>
While investigating some pruning issues I found some room for
improvement in the notification pin handling.
**Problem:** It was not possible to define an upper limit on
notification pins. The block pinning cache has a limit, but only handles
bodies and justifications.
After this PR, bookkeeping for notifications is managed in the pinning
worker. A limit can be defined in the worker. If that limit is crossed,
blocks that were pinned for that notification are unpinned, which now
affects the state as well as bodies and justifications. The pinned
blocks cache still has a limit, but should never be hit.
closes#19
---------
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Preparation for https://github.com/paritytech/polkadot-sdk/issues/2664
Changes:
- Only require `Hash` instead of `Block` for the benchmarking
- Refactor DB types to do the same
## Integration
This breaking change can easily be integrated into your node via:
```patch
- cmd.run::<Block, ()>(config)
+ cmd.run::<HashingFor<Block>, ()>(config)
```
Status: waiting for CI checks
---------
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
The
[archive](https://github.com/paritytech/json-rpc-interface-spec/blob/main/src/api/archive.md)
API is unstable and subject to change.
This PR enables the `archive` class of the RPC-V2 spec to substrate
based chains.
The `archive` API is enabled for archive nodes:
- the state of the blocks is in archive mode
- the block's bodies are in archive mode
While at it, this PR extends the `BlocksPrunning` enum with an
`is_archive` helper to check if the pruning mode keeps the block's
bodies for long enough.
Defaults used for the `archive` API:
- a maximum of 5 responses are provided for descendants queries (this is
similar to chainHead)
- a maximum of 8 item queries are accepted at a time
Before stabilizing the API we should look into these defaults and adjust
after collecting some data.
---------
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
* globally upgrade quote to latest 1.0.x (1.0.26)
* globally upgrade syn to final 1.0.x version (1.0.109)
* globally upgrade proc-macro2 to 1.0.56
* upgrade to syn v2.0.13 and fix everything except NestedMeta
* fix parse nested metadata code in decl_runtime_apis.rs
* Port more stuff to syn 2.0
* Make the rest compile
* Ignore error
* update to syn 2.0.14
---------
Co-authored-by: Bastian Köcher <info@kchr.de>
* Rework storage iterators
* Make sure storage iteration is also accounted for when benchmarking
* Use `trie-db` from crates.io
* Appease clippy
* Bump `trie-bench` to 0.35.0
* Fix tests' compilation
* Update comment to clarify how `IterArgs::start_at` works
* Add extra tests
* Fix iterators on `Client` so that they behave as before
* Add extra `unwrap`s in tests
* More clippy fixes
* Come on clippy, give me a break already
* Rename `allow_missing` to `stop_on_incomplete_database`
* Add `#[inline]` to `with_recorder_and_cache`
* Use `with_recorder_and_cache` in `with_trie_db`; add doc comment
* Simplify code: use `with_trie_db` in `next_storage_key_from_root`
* Remove `expect`s in the benchmarking CLI
* Add extra doc comments
* Move `RawIter` before `TrieBackendEssence` (no code changes; just cut-paste)
* Remove a TODO in tests
* Update comment for `StorageIterator::was_complete`
* Update `trie-db` to 0.25.1
* Change copyright year to 2023 from 2022
* Fix incorrect update of copyright year
* Remove years from copy right header
* Fix remaining files
* Fix typo in a header and remove update-copyright.sh
* sc-client-db: Fix `PruningMode::ArchiveCanonical`
When running a node with `--state-pruning archive-canonical` it was directly failing on genesis.
There was an issue in the state-db `pin` implementation. It was not checking the state of a block
correctly when running with archive canonical (and also not for every other block after they are canonicalized).
* FMT
* initial impl
* add template test
* linear fit proof size
* always record proof when tracking storage
* calculate worst case pov
* remove duplicate worst case
* cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_assets --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/assets/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* more comment output
* add cli for worst case map size
* update name
* clap does not support underscores
* rename
* expose worst case map values
* improve some comments
* cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_assets --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/assets/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* update template
* cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_assets --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/assets/src/weights.rs --template=./.maintain/frame-weight-template.hbs
* fix fmt
* more fmt
* more fmt
* Dont panic when there is no proof
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix test features
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Whitelist :extrinsic_index
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Use whitelist when recording proof
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add logs
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add PoV testing pallet
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Deploy PoV testing pallet
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Storage benches reside in the PoV pallet
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Linear regress PoV per component
Splits the PoV calculation into "measured" and "estimated".
The measured part is reported by the Proof recorder and linear
regressed over all components at once.
The estimated part is calculated as worst-case by using the max
PoV size per storage access and calculating one linear regress per
component. This gives each component a (possibly) independent PoV.
For now the measured size will always be lower than the PoV on
Polkadot since it is measured on an empty snapshot. The measured
part is therefor only used as diagnostic for debugging.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Put PoV into the weight templates
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* fmt
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Extra alanysis choise for PoV
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add+Fix tests
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Make benches faster
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Cleanup
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Use same template comments
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* ".git/.scripts/bench-bot.sh" pallet dev pallet_balances
* ".git/.scripts/bench-bot.sh" pallet dev pallet_democracy
* Update referenda mock BlockWeights
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Take measured value size into account
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* clippy
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* ".git/.scripts/bench-bot.sh" pallet dev pallet_scheduler
* WIP
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* proof_size: None
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* WIP
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* WIP
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* WIP
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* ugly, but works
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* WIP
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* WIP
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* WIP
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* wup
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* WIP
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* WIP
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* WIP
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* WIP
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add pov_mode attribute to the benchmarks! macro
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Use pov_mode attribute in PoV benchmarking
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update tests
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Scheduler, Whitelist: Add pov_mode attr
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update PoV weights
* Add CLI arg: default-pov-mode
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix tests
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* fmt
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* fix
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Revert "Update PoV weights"
This reverts commit 2f3ac2387396470b118122a6ff8fa4ee12216f4b.
* Revert "WIP"
This reverts commit c34b538cd2bc45da4544e887180184e30957904a.
* Revert first approach
This reverts commit range 8ddaa2fffe5930f225a30bee314d0b7c94c344dd^..4c84f8748e5395852a9e0e25b0404953fee1a59e
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Clippy
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add extra benchmarks
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_alliance
* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_whitelist
* ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_scheduler
* fmt
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Clippy
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Clippy 🤦
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add reference benchmarks
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix doc comments
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Undo logging
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Add 'Ignored' pov_mode
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Allow multiple attributes per benchmark
Turns out that the current benchmarking syntax does not support
multiple attributes per bench 🤦. Changing it to support that
since otherwise the `pov_mode` would conflict with the others.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Validate pov_mode syntax
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Ignore PoV for all contract benchmarks
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Test
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* test
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Bump macro recursion limit
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* fmt
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update contract weights
They dont have a PoV component anymore.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* fix test ffs
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* pov_mode is unsupported in V2 syntax
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix pallet ui tests
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* update pallet ui
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Fix pallet ui tests
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Update weights
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
Co-authored-by: Your Name <you@example.com>
* Worker
* Reorganize and unpin onnotification drop
* Pin in state-db, pass block number
* Pin blocks in blockchain db
* Switch to reference counted LRU
* Disable pinning when we keep all blocks
* Fix pinning hint for state-db
* Remove pinning from backend layer
* Improve readability
* Add justifications to test
* Fix justification behaviour
* Remove debug prints
* Convert channels to tracing_unbounded
* Add comments to the test
* Documentation and Cleanup
* Move task start to client
* Simplify cache
* Improve test, remove unwanted log
* Add tracing logs, remove expect for block number
* Cleanup
* Add conversion method for unpin handle to Finalitynotification
* Revert unwanted changes
* Improve naming
* Make clippy happy
* Fix docs
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
* Use `NumberFor` instead of u64 in API
* Hand over weak reference to unpin worker task
* Unwanted
* &Hash -> Hash
* Remove number from interface, rename `_unpin_handle`, LOG_TARGET
* Move RwLock one layer up
* Apply code style suggestions
* Improve comments
* Replace lru crate by schnellru
* Only insert values for pinned items + better docs
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <git@kchr.de>
* Improve comments, log target and test
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
We don't canonicalize on archive nodes and thus `best_canonical` always returned `None`. So, the
moment such a node tried to force canonicalize, it was trapped in some endless loop.
This pr solves this by renaming `best_canonical` to `last_canonicalized` and also making the return
value more clear by introducing a custom enum `LastCanonicalized`.
It changes the arguments of `HeaderBackend::status` method from: `BlockId<Block>` to: `Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
Co-authored-by: parity-processbot <>
* Fix missing block number issue on forced canonicalization
There is this issue about missing block numbers on forced canonicalization. I looked over the code
now 10000 times and there are possible ways this can be triggered, but I don't really know how this
is triggered. So, this pr is going to solve the symptom and not the cause. The block number to hash
mapping is set when we import a new best block. Forced canonicalization will now stop at the best
block and it will canonicalize the other blocks later when the best block moved. As the error
reports indicated that this issue mainly happened on major sync, there should not be any forks, so
not doing the canonicalization directly shouldn't be that harmful. All known implementations should
import all blocks as best block on major sync anyway (I mean somewhere there is the bug, but I
didn't yet found it).
I will also do some changes to Cumulus around some potential culprit for this issue.
Closes: https://github.com/paritytech/substrate/issues/12613
* Add some docs
* Fix fix
* Review comments
* Review comments
* Prune discarded blocks immediately
* state-db refactoring part 1
* Some renames
* Get rid of pending state
* Revert "Prune discarded blocks immediately"
This reverts commit b60d98c569e8af18d99087da93f0911d4f24006e.
* Cleanup
* Make clippy happy
* Minor changes
* BlockId removal: refactor: Backend::block_indexed_body
It changes the arguments of `Backend::block_indexed_body` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* trigger CI job
* BlockId removal: refactor: Backend::justifications
It changes the arguments of `Backend::justifications` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* trigger CI job
* trigger CI job
* bug fix
* match -> if
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
It changes the arguments of `Backend::body` method from: `BlockId<Block>` to: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
Co-authored-by: parity-processbot <>
* BlockId removal: refactor: Backend::append_justification
It changes the arguments of `Backend::append_justification`
from: block: `BlockId<Block>` to: hash: `&Block::Hash`
This PR is part of `BlockId::Number` refactoring analysis (paritytech/substrate#11292)
* Error message improved
Co-authored-by: Adrian Catangiu <adrian@parity.io>
* single error message in beefy::finalize
* println removed
Co-authored-by: Adrian Catangiu <adrian@parity.io>
It changes the arguments of `Backend::begin_state_operation`
from: block: `BlockId<Block>` to: hash: `&Block::Hash`
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* BlockId removal: refactor: BlockImportOperation+Bknd::finalize_block
It changes the arguments of methods of `BlockImportOperation` trait
from: block: `BlockId<Block>` to: hash: `&Block::Hash`
`Backend::finalize_block` was also changed.
This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
* Review suggestion applied
thx to @davxy
* trigger CI job
* 1.Add pruning param "canonical" in sc-cli.
2.Make PruningMode's default value to ArchiveCanonical.
* Update tests in sc-state-db.
* Update tests in sc-state-db.
* 1.Add a new value `AllWithNonFinalized` in `enum BlocksPruning` which Corresponds to `blocks_pruning 0` in CLI .
2.Change value `All` to `AllFinalized` in `enum BlocksPruning` and make it to keep full finalized block history.
* Make some corresponding adjustments based on the content in the conversation.
* Update client/db/src/lib.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* Apply suggestions from code review.
* 1.Change `blocks_pruning` to be like `state_pruning` .
* Fmt and add some doc.
* Update client/cli/src/params/pruning_params.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update client/cli/src/params/pruning_params.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* Update doc.
* Change `new_test_with_tx_storage` to take `BlocksPruning`.
* Fmt
Co-authored-by: Bastian Köcher <git@kchr.de>
* trie state cache
* Also cache missing access on read.
* fix comp
* bis
* fix
* use has_lru
* remove local storage cache on size 0.
* No cache.
* local cache only
* trie cache and local cache
* storage cache (with local)
* trie cache no local cache
* Add state access benchmark
* Remove warnings etc
* Add trie cache benchmark
* No extra "clone" required
* Change benchmark to use multiple blocks
* Use patches
* Integrate shitty implementation
* More stuff
* Revert "Merge branch 'master' into trie_state_cache"
This reverts commit 947cd8e6d43fced10e21b76d5b92ffa57b57c318, reversing
changes made to 29ff036463.
* Improve benchmark
* Adapt to latest changes
* Adapt to changes in trie
* Add a test that uses iterator
* Start fixing it
* Remove obsolete file
* Make it compile
* Start rewriting the trie node cache
* More work on the cache
* More docs and code etc
* Make data cache an optional
* Tests
* Remove debug stuff
* Recorder
* Some docs and a simple test for the recorder
* Compile fixes
* Make it compile
* More fixes
* More fixes
* Fix fix fix
* Make sure cache and recorder work together for basic stuff
* Test that data caching and recording works
* Test `TrieDBMut` with caching
* Try something
* Fixes, fixes, fixes
* Forward the recorder
* Make it compile
* Use recorder in more places
* Switch to new `with_optional_recorder` fn
* Refactor and cleanups
* Move `ProvingBackend` tests
* Simplify
* Move over all functionality to the essence
* Fix compilation
* Implement estimate encoded size for StorageProof
* Start using the `cache` everywhere
* Use the cache everywhere
* Fix compilation
* Fix tests
* Adds `TrieBackendBuilder` and enhances the tests
* Ensure that recorder drain checks that values are found as expected
* Switch over to `TrieBackendBuilder`
* Start fixing the problem with child tries and recording
* Fix recording of child tries
* Make it compile
* Overwrite `storage_hash` in `TrieBackend`
* Add `storage_cache` to the benchmarks
* Fix `no_std` build
* Speed up cache lookup
* Extend the state access benchmark to also hash a runtime
* Fix build
* Fix compilation
* Rewrite value cache
* Add lru cache
* Ensure that the cache lru works
* Value cache should not be optional
* Add support for keeping the shared node cache in its bounds
* Make the cache configurable
* Check that the cache respects the bounds
* Adds a new test
* Fixes
* Docs and some renamings
* More docs
* Start using the new recorder
* Fix more code
* Take `self` argument
* Remove warnings
* Fix benchmark
* Fix accounting
* Rip off the state cache
* Start fixing fallout after removing the state cache
* Make it compile after trie changes
* Fix test
* Add some logging
* Some docs
* Some fixups and clean ups
* Fix benchmark
* Remove unneeded file
* Use git for patching
* Make CI happy
* Update primitives/trie/Cargo.toml
Co-authored-by: Koute <koute@users.noreply.github.com>
* Update primitives/state-machine/src/trie_backend.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Introduce new `AsTrieBackend` trait
* Make the LocalTrieCache not clonable
* Make it work in no_std and add docs
* Remove duplicate dependency
* Switch to ahash for better performance
* Speedup value cache merge
* Output errors on underflow
* Ensure the internal LRU map doesn't grow too much
* Use const fn to calculate the value cache element size
* Remove cache configuration
* Fix
* Clear the cache in between for more testing
* Try to come up with a failing test case
* Make the test fail
* Fix the child trie recording
* Make everything compile after the changes to trie
* Adapt to latest trie-db changes
* Fix on stable
* Update primitives/trie/src/cache.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Fix wrong merge
* Docs
* Fix warnings
* Cargo.lock
* Bump pin-project
* Fix warnings
* Switch to released crate version
* More fixes
* Make clippy and rustdocs happy
* More clippy
* Print error when using deprecated `--state-cache-size`
* 🤦
* Fixes
* Fix storage_hash linkings
* Update client/rpc/src/dev/mod.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Review feedback
* encode bound
* Rework the shared value cache
Instead of using a `u64` to represent the key we now use an `Arc<[u8]>`. This arc is also stored in
some extra `HashSet`. We store the key are in an extra `HashSet` to de-duplicate the keys accross
different storage roots. When the latest key usage is dropped in the lru, we also remove the key
from the `HashSet`.
* Improve of the cache by merging the old and new solution
* FMT
* Please stop coming back all the time :crying:
* Update primitives/trie/src/cache/shared_cache.rs
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Fixes
* Make clippy happy
* Ensure we don't deadlock
* Only use one lock to simplify the code
* Do not depend on `Hasher`
* Fix tests
* FMT
* Clippy 🤦
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>
* Fix leaf block removal in the backend
The fix introduced the new 'removal' method for the backend leaves set
and the improvement of the undo features.
* Update docs
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Fix docs typo
* On block block removal the new children list should be persisted.
* Align leaves set removal tests to the new interface
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Prevent duplicated leaves in the backend
* Comments...
* Use highest known heaf as a shortcut for not existing header detection
* Apply code review suggestion
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Revert non-best block
This makes `revert` also revert non-best blocks.
* Update client/db/src/lib.rs
* Do not count leaves against the maximum number to revert
* Add some explanation
* Fix bug
* Apply suggestions from code review
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Dcompoze