Commit Graph

86 Commits

Author SHA1 Message Date
georgepisaltu bbd51ce867 Revert "FRAME: Create TransactionExtension as a replacement for SignedExtension (#2280)" (#3665)
This PR reverts #2280 which introduced `TransactionExtension` to replace
`SignedExtension`.

As a result of the discussion
[here](https://github.com/paritytech/polkadot-sdk/pull/3623#issuecomment-1986789700),
the changes will be reverted for now with plans to reintroduce the
concept in the future.

---------

Signed-off-by: georgepisaltu <george.pisaltu@parity.io>
2024-03-13 14:10:59 +00:00
Gavin Wood fd5f9292f5 FRAME: Create TransactionExtension as a replacement for SignedExtension (#2280)
Closes #2160

First part of [Extrinsic
Horizon](https://github.com/paritytech/polkadot-sdk/issues/2415)

Introduces a new trait `TransactionExtension` to replace
`SignedExtension`. Introduce the idea of transactions which obey the
runtime's extensions and have according Extension data (né Extra data)
yet do not have hard-coded signatures.

Deprecate the terminology of "Unsigned" when used for
transactions/extrinsics owing to there now being "proper" unsigned
transactions which obey the extension framework and "old-style" unsigned
which do not. Instead we have __*General*__ for the former and
__*Bare*__ for the latter. (Ultimately, the latter will be phased out as
a type of transaction, and Bare will only be used for Inherents.)

Types of extrinsic are now therefore:
- Bare (no hardcoded signature, no Extra data; used to be known as
"Unsigned")
- Bare transactions (deprecated): Gossiped, validated with
`ValidateUnsigned` (deprecated) and the `_bare_compat` bits of
`TransactionExtension` (deprecated).
  - Inherents: Not gossiped, validated with `ProvideInherent`.
- Extended (Extra data): Gossiped, validated via `TransactionExtension`.
  - Signed transactions (with a hardcoded signature).
  - General transactions (without a hardcoded signature).

`TransactionExtension` differs from `SignedExtension` because:
- A signature on the underlying transaction may validly not be present.
- It may alter the origin during validation.
- `pre_dispatch` is renamed to `prepare` and need not contain the checks
present in `validate`.
- `validate` and `prepare` is passed an `Origin` rather than a
`AccountId`.
- `validate` may pass arbitrary information into `prepare` via a new
user-specifiable type `Val`.
- `AdditionalSigned`/`additional_signed` is renamed to
`Implicit`/`implicit`. It is encoded *for the entire transaction* and
passed in to each extension as a new argument to `validate`. This
facilitates the ability of extensions to acts as underlying crypto.

There is a new `DispatchTransaction` trait which contains only default
function impls and is impl'ed for any `TransactionExtension` impler. It
provides several utility functions which reduce some of the tedium from
using `TransactionExtension` (indeed, none of its regular functions
should now need to be called directly).

Three transaction version discriminator ("versions") are now
permissible:
- 0b000000100: Bare (used to be called "Unsigned"): contains Signature
or Extra (extension data). After bare transactions are no longer
supported, this will strictly identify an Inherents only.
- 0b100000100: Old-school "Signed" Transaction: contains Signature and
Extra (extension data).
- 0b010000100: New-school "General" Transaction: contains Extra
(extension data), but no Signature.

For the New-school General Transaction, it becomes trivial for authors
to publish extensions to the mechanism for authorizing an Origin, e.g.
through new kinds of key-signing schemes, ZK proofs, pallet state,
mutations over pre-authenticated origins or any combination of the
above.

## Code Migration

### NOW: Getting it to build

Wrap your `SignedExtension`s in `AsTransactionExtension`. This should be
accompanied by renaming your aggregate type in line with the new
terminology. E.g. Before:

```rust
/// The SignedExtension to the basic transaction logic.
pub type SignedExtra = (
	/* snip */
	MySpecialSignedExtension,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, SignedExtra>;
```

After:

```rust
/// The extension to the basic transaction logic.
pub type TxExtension = (
	/* snip */
	AsTransactionExtension<MySpecialSignedExtension>,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, TxExtension>;
```

You'll also need to alter any transaction building logic to add a
`.into()` to make the conversion happen. E.g. Before:

```rust
fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let extra: SignedExtra = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */),
	);
	let payload = SignedPayload::new(call.clone(), extra.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		extra,
	)
}
```

After:

```rust
fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let tx_ext: TxExtension = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */).into(),
	);
	let payload = SignedPayload::new(call.clone(), tx_ext.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		tx_ext,
	)
}
```

### SOON: Migrating to `TransactionExtension`

Most `SignedExtension`s can be trivially converted to become a
`TransactionExtension`. There are a few things to know.

- Instead of a single trait like `SignedExtension`, you should now
implement two traits individually: `TransactionExtensionBase` and
`TransactionExtension`.
- Weights are now a thing and must be provided via the new function `fn
weight`.

#### `TransactionExtensionBase`

This trait takes care of anything which is not dependent on types
specific to your runtime, most notably `Call`.

- `AdditionalSigned`/`additional_signed` is renamed to
`Implicit`/`implicit`.
- Weight must be returned by implementing the `weight` function. If your
extension is associated with a pallet, you'll probably want to do this
via the pallet's existing benchmarking infrastructure.

#### `TransactionExtension`

Generally:
- `pre_dispatch` is now `prepare` and you *should not reexecute the
`validate` functionality in there*!
- You don't get an account ID any more; you get an origin instead. If
you need to presume an account ID, then you can use the trait function
`AsSystemOriginSigner::as_system_origin_signer`.
- You get an additional ticket, similar to `Pre`, called `Val`. This
defines data which is passed from `validate` into `prepare`. This is
important since you should not be duplicating logic from `validate` to
`prepare`, you need a way of passing your working from the former into
the latter. This is it.
- This trait takes two type parameters: `Call` and `Context`. `Call` is
the runtime call type which used to be an associated type; you can just
move it to become a type parameter for your trait impl. `Context` is not
currently used and you can safely implement over it as an unbounded
type.
- There's no `AccountId` associated type any more. Just remove it.

Regarding `validate`:
- You get three new parameters in `validate`; all can be ignored when
migrating from `SignedExtension`.
- `validate` returns a tuple on success; the second item in the tuple is
the new ticket type `Self::Val` which gets passed in to `prepare`. If
you use any information extracted during `validate` (off-chain and
on-chain, non-mutating) in `prepare` (on-chain, mutating) then you can
pass it through with this. For the tuple's last item, just return the
`origin` argument.

Regarding `prepare`:
- This is renamed from `pre_dispatch`, but there is one change:
- FUNCTIONALITY TO VALIDATE THE TRANSACTION NEED NOT BE DUPLICATED FROM
`validate`!!
- (This is different to `SignedExtension` which was required to run the
same checks in `pre_dispatch` as in `validate`.)

Regarding `post_dispatch`:
- Since there are no unsigned transactions handled by
`TransactionExtension`, `Pre` is always defined, so the first parameter
is `Self::Pre` rather than `Option<Self::Pre>`.

If you make use of `SignedExtension::validate_unsigned` or
`SignedExtension::pre_dispatch_unsigned`, then:
- Just use the regular versions of these functions instead.
- Have your logic execute in the case that the `origin` is `None`.
- Ensure your transaction creation logic creates a General Transaction
rather than a Bare Transaction; this means having to include all
`TransactionExtension`s' data.
- `ValidateUnsigned` can still be used (for now) if you need to be able
to construct transactions which contain none of the extension data,
however these will be phased out in stage 2 of the Transactions Horizon,
so you should consider moving to an extension-centric design.

## TODO

- [x] Introduce `CheckSignature` impl of `TransactionExtension` to
ensure it's possible to have crypto be done wholly in a
`TransactionExtension`.
- [x] Deprecate `SignedExtension` and move all uses in codebase to
`TransactionExtension`.
  - [x] `ChargeTransactionPayment`
  - [x] `DummyExtension`
  - [x] `ChargeAssetTxPayment` (asset-tx-payment)
  - [x] `ChargeAssetTxPayment` (asset-conversion-tx-payment)
  - [x] `CheckWeight`
  - [x] `CheckTxVersion`
  - [x] `CheckSpecVersion`
  - [x] `CheckNonce`
  - [x] `CheckNonZeroSender`
  - [x] `CheckMortality`
  - [x] `CheckGenesis`
  - [x] `CheckOnlySudoAccount`
  - [x] `WatchDummy`
  - [x] `PrevalidateAttests`
  - [x] `GenericSignedExtension`
  - [x] `SignedExtension` (chain-polkadot-bulletin)
  - [x] `RefundSignedExtensionAdapter`
- [x] Implement `fn weight` across the board.
- [ ] Go through all pre-existing extensions which assume an account
signer and explicitly handle the possibility of another kind of origin.
- [x] `CheckNonce` should probably succeed in the case of a non-account
origin.
- [x] `CheckNonZeroSender` should succeed in the case of a non-account
origin.
- [x] `ChargeTransactionPayment` and family should fail in the case of a
non-account origin.
  - [ ] 
- [x] Fix any broken tests.

---------

Signed-off-by: georgepisaltu <george.pisaltu@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Nikhil Gupta <17176722+gupnik@users.noreply.github.com>
Co-authored-by: georgepisaltu <52418509+georgepisaltu@users.noreply.github.com>
Co-authored-by: Chevdor <chevdor@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Maciej <maciej.zyszkiewicz@parity.io>
Co-authored-by: Javier Viola <javier@parity.io>
Co-authored-by: Marcin S. <marcin@realemail.net>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Co-authored-by: Javier Bullrich <javier@bullrich.dev>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Vladimir Istyufeev <vladimir@parity.io>
Co-authored-by: Ross Bulat <ross@parity.io>
Co-authored-by: Gonçalo Pestana <g6pestana@gmail.com>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
Co-authored-by: Svyatoslav Nikolsky <svyatonik@gmail.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com>
Co-authored-by: ordian <write@reusable.software>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
Co-authored-by: Julian Eager <eagr@tutanota.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Dónal Murray <donal.murray@parity.io>
Co-authored-by: yjh <yjh465402634@gmail.com>
Co-authored-by: Tom Mi <tommi@niemi.lol>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Will | Paradox | ParaNodes.io <79228812+paradox-tt@users.noreply.github.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>
Co-authored-by: Joshy Orndorff <git-user-email.h0ly5@simplelogin.com>
Co-authored-by: PG Herveou <pgherveou@gmail.com>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Juan Girini <juangirini@gmail.com>
Co-authored-by: bader y <ibnbassem@gmail.com>
Co-authored-by: James Wilson <james@jsdw.me>
Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
Co-authored-by: asynchronous rob <rphmeier@gmail.com>
Co-authored-by: Parth <desaiparth08@gmail.com>
Co-authored-by: Andrew Jones <ascjones@gmail.com>
Co-authored-by: Jonathan Udd <jonathan@dwellir.com>
Co-authored-by: Serban Iorga <serban@parity.io>
Co-authored-by: Egor_P <egor@parity.io>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Evgeny Snitko <evgeny@parity.io>
Co-authored-by: Just van Stam <vstam1@users.noreply.github.com>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: gupnik <nikhilgupta.iitk@gmail.com>
Co-authored-by: dzmitry-lahoda <dzmitry@lahoda.pro>
Co-authored-by: zhiqiangxu <652732310@qq.com>
Co-authored-by: Nazar Mokrynskyi <nazar@mokrynskyi.com>
Co-authored-by: Anwesh <anweshknayak@gmail.com>
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Sam Johnson <sam@durosoft.com>
Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Jegor Sidorenko <5252494+jsidorenko@users.noreply.github.com>
Co-authored-by: Muharem <ismailov.m.h@gmail.com>
Co-authored-by: joepetrowski <joe@parity.io>
Co-authored-by: Alexandru Gheorghe <49718502+alexggh@users.noreply.github.com>
Co-authored-by: Gabriel Facco de Arruda <arrudagates@gmail.com>
Co-authored-by: Squirrel <gilescope@gmail.com>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
Co-authored-by: georgepisaltu <george.pisaltu@parity.io>
Co-authored-by: command-bot <>
2024-03-04 19:12:43 +00:00
Alexandru Vasile f1b2189e83 rpc-v2/tx/tests: Add transaction broadcast tests and check propagated tx status (#3193)
This PR adds tests for the `transaction_broadcast` method.


The testing needs to coordinate the following components:
- The `TestApi` marks transactions as invalid and implements
`ChainApi::validate_transaction`
- this is what dictates if a transaction is valid or not and is called
from within the `BasicPool`
- The `BasicPool` which maintains the transactions and implements
`submit_and_watch` needed by the tx broadcast to submit the transaction
- The status of the transaction pool is exposed by mocking the BasicPool
- The `ChainHeadMockClient` which mocks the
`BlockchainEvents::import_notification_stream` needed by the tx
broadcast to know to which blocks the transaction is submitted

The following changes have been added to the substrate testing to
accommodate this:
- `TestApi` gets ` remove_invalid`, counterpart to `add_invalid` to
ensure an invalid transaction can become valid again; as well as a
priority setter for extrinsics
- `BasicPool` test constructor is extended with options for the
`PoolRotator`
- this mechanism is needed because transactions are banned for 30mins
(default) after they are declared invalid
  - testing bypasses this by providing a `Duration::ZERO`

### Testing Scenarios

- Capture the status of the transaction as it is normally broadcasted
- `transaction_stop` is valid while the transaction is in progress
- A future transaction is handled when the dependencies are completed
- Try to resubmit the transaction at a later block (currently invalid)
- An invalid transaction status is propagated; the transaction is marked
as temporarily banned; then the ban expires and transaction is
resubmitted
  
This builds on top of:
https://github.com/paritytech/polkadot-sdk/pull/3079
Part of: https://github.com/paritytech/polkadot-sdk/issues/3084

cc @paritytech/subxt-team

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: James Wilson <james@jsdw.me>
2024-02-28 09:43:58 +00:00
Michal Kucharczyk ab3a3bc278 BlockId removal: tx-pool refactor (#1678)
It changes following APIs:
- trait `ChainApi`
-- `validate_transaction`

- trait `TransactionPool` 
--`submit_at`
--`submit_one`
--`submit_and_watch`

and some implementation details, in particular:
- impl `Pool` 
--`submit_at`
--`resubmit_at`
--`submit_one`
--`submit_and_watch`
--`prune_known`
--`prune`
--`prune_tags`
--`resolve_block_number`
--`verify`
--`verify_one`

- revalidation queue

All tests are also adjusted.

---------

Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <git@kchr.de>
2023-09-27 11:58:39 +02:00
yjh 4053bdac03 feat: add futures api to TransactionPool (#1348)
* feat: add futures api to `TransactionPool`

* fix clippy
2023-09-03 21:52:20 +02:00
Bastian Köcher 5eb816d7a6 Removal of execution strategies (#14387)
* Start

* More work!

* Moar

* More changes

* More fixes

* More worrk

* More fixes

* More fixes to make it compile

* Adds `NoOffchainStorage`

* Pass the extensions

* Small basti making small progress

* Fix merge errors and remove `ExecutionContext`

* Move registration of `ReadRuntimeVersionExt` to `ExecutionExtension`

Instead of registering `ReadRuntimeVersionExt` in `sp-state-machine` it is moved to
`ExecutionExtension` which provides the default extensions.

* Fix compilation

* Register the global extensions inside runtime api instance

* Fixes

* Fix `generate_initial_session_keys` by passing the keystore extension

* Fix the grandpa tests

* Fix more tests

* Fix more tests

* Don't set any heap pages if there isn't an override

* Fix small fallout

* FMT

* Fix tests

* More tests

* Offchain worker custom extensions

* More fixes

* Make offchain tx pool creation reusable

Introduces an `OffchainTransactionPoolFactory` for creating offchain transactions pools that can be
registered in the runtime externalities context. This factory will be required for a later pr to
make the creation of offchain transaction pools easier.

* Fixes

* Fixes

* Set offchain transaction pool in BABE before using it in the runtime

* Add the `offchain_tx_pool` to Grandpa as well

* Fix the nodes

* Print some error when using the old warnings

* Fix merge issues

* Fix compilation

* Rename `babe_link`

* Rename to `offchain_tx_pool_factory`

* Cleanup

* FMT

* Fix benchmark name

* Fix `try-runtime`

* Remove `--execution` CLI args

* Make clippy happy

* Forward bls functions

* Fix docs

* Update UI tests

* Update client/api/src/execution_extensions.rs

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/cli/src/params/import_params.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update client/api/src/execution_extensions.rs

Co-authored-by: Koute <koute@users.noreply.github.com>

* Pass the offchain storage to the MMR RPC

* Update client/api/src/execution_extensions.rs

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

* Review comments

* Fixes

---------

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
2023-07-11 14:21:38 +00:00
Bastian Köcher 00e5ecdcf7 Make offchain tx pool creation reusable (#14230)
* Make offchain tx pool creation reusable

Introduces an `OffchainTransactionPoolFactory` for creating offchain transactions pools that can be
registered in the runtime externalities context. This factory will be required for a later pr to
make the creation of offchain transaction pools easier.

* Update client/transaction-pool/api/src/lib.rs

Co-authored-by: Anton <anton.kalyaev@gmail.com>

---------

Co-authored-by: Anton <anton.kalyaev@gmail.com>
2023-05-31 01:12:53 +00:00
Vivek Pandya bc53b9a03a Remove years from copyright notes. (#13415)
* Change copyright year to 2023 from 2022

* Fix incorrect update of copyright year

* Remove years from copy right header

* Fix remaining files

* Fix typo in a header and remove update-copyright.sh
2023-02-21 18:46:41 +00:00
Michal Kucharczyk 57b1de3f47 txpool: LOG_TARGET const added (#13180)
* txpool: LOG_TARGET const added

part of: #12873

* LOG_TARGET added to tests mod

* txpool::api for api

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* ".git/.scripts/commands/fmt/fmt.sh"

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>
2023-01-19 22:21:41 +00:00
Michal Kucharczyk 5134dabb4a txpool: don't maintain the pool during major sync (#13004)
* txpool: don't maintain the pool during major sync

Fix shall prevent from wasting the CPU during the major sync. No actions
are actually required in transaction pool during the major sync.

Fixes: #12903

* passing sync_oracle to maintain method

* fixed: builder, txpool tests

* do not maintain tx-pool if node gone out of sync

* EnactmentAction: all logic moved to EnactmentState

Tests to be done.

* maintain guard logic moved directly to MaintainedTransactionPool

* minor fixes

* EnactmentAction: all logic moved to EnactmentState (again)

* SyncOracle fixes here and there

* Update client/transaction-pool/src/enactment_state.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/transaction-pool/src/enactment_state.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* sync_oracle removed

* spelling + fmt + doc

* Review suggestions applied

* log::info -> debug

* Update client/transaction-pool/src/enactment_state.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* ".git/.scripts/commands/fmt/fmt.sh"

Co-authored-by: parity-processbot <>
Co-authored-by: Bastian Köcher <git@kchr.de>
2023-01-16 23:15:22 +01:00
Michal Kucharczyk 548955a73f BlockId removal: refactor: HeaderBackend::header (#12874)
* BlockId removal: refactor: HeaderBackend::header

It changes the arguments of:
- `HeaderBackend::header`,
- `Client::header`,
- `PeersClient::header`
- `ChainApi::block_header`

methods from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* non-trivial usages of haeder(block_id) refactored

This may required introduction of dedicated function:
header_for_block_num

* fmt

* fix

* doc fixed

* ".git/.scripts/fmt.sh"

* BlockId removal: refactor: HeaderBackend::expect_header

It changes the arguments of `HeaderBackend::expect_header` method from: `BlockId<Block>` to: `Block::Hash`

* ".git/.scripts/fmt.sh"

* readme updated

* ".git/.scripts/fmt.sh"

* fix

Co-authored-by: parity-processbot <>
2022-12-20 09:43:31 +00:00
Marcin S aa21e56744 Remove mem_info and references to parity-util-mem (#12795)
* Remove mem_info and some references to parity-util-mem

* [Draft] Finish removing references to `parity-util-mem`

* Upgrade dependencies

* Update scripts/ci/deny.toml

Co-authored-by: ordian <write@reusable.software>

* Fix Cargo.lock (remove unwanted dependency changes)

* Removed unused argument

* Run cargo fmt (didn't have pre-commit set up)

* Fix some CI errors

* Fix another CI error

* Remove unused dependency

Co-authored-by: ordian <write@reusable.software>
2022-12-06 14:55:10 +00:00
Michal Kucharczyk 1ed70004e7 BlockId removal: &Hash to Hash (#12626)
It changes &Block::Hash argument to Block::Hash.

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
2022-11-07 21:42:16 +00:00
Michal Kucharczyk cf5c78dbe8 txpool: enactment state forced update (#12632)
* txpool: enactment state forced update

When `tree_route` computation fails, we still need to update the
`enactment_state` to be aligned with last known finalized/best block.

We do not execute enactment phase of maintain procedure, but we do
update the state.

* error -> debug

* test added
2022-11-07 21:12:26 +01:00
Michal Kucharczyk 76bcbd09a5 BlockId removal: refactor: Backend::body (#12587)
It changes the arguments of `Backend::body` method from: `BlockId<Block>` to: `&Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

Co-authored-by: parity-processbot <>
2022-11-01 16:24:09 +00:00
Michal Kucharczyk 62bca87f3a Finalized block event triggers tx maintanance (#12305)
* finalized block event triggers tx maintanance

* tx-pool: enactment helper introduced

* tx-pool: ChainApi: added tree_route method

* enactment logic implemented + tests

Signed-off-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

* Some additional tests

* minor improvements

* trigger CI job

* fix compilation errors

ChainApi::tree_route return type changed to Result<Option<..>>, as some
implementations (tests) are not required to provide this tree route.

* formatting

* trait removed

* implementation slightly simplified

(thanks to @koute)

* get rid of Arc<> in EnactmentState return value

* minor improvement

* Apply suggestions from code review

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Apply suggestions from code review

* comment updated + formatting

* Apply suggestions from code review

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Davide Galassi <davxy@datawok.net>

* formatting

* finalization notification bug fix

+ new test case
+ log::warn message when finalized block is being retracted by new event

* added error message on tree_route failure

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* use provided tree_route in Finalized event

* Option removed from ChainApi::tree_route

* doc added, test and logs improved

* handle_enactment aligned with original implementation

* use async-await

* Apply suggestions from code review

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* formatting + warn->debug

* compilation error fix

* enactment_state initializers added

* enactment_state: Option removed

* manual-seal: compilation & tests fix

* manual-seal: tests fixed

* tests cleanup

* another compilation error fixed

* TreeRoute::new added

* get rid of pub hack

* one more test added

* formatting

* TreeRoute::new doc added + formatting

* Apply suggestions from code review

Co-authored-by: Davide Galassi <davxy@datawok.net>

* (bool,Option) simplified to Option

* log message improved

* yet another review suggestions applied

* get rid of hash in handle_enactment

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update client/transaction-pool/src/lib.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* minor corrections

* EnactmentState moved to new file

* File header corrected

* error formatting aligned with codebase

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* remove commented code

* small nits

Signed-off-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: André Silva <andrerfosilva@gmail.com>
2022-10-11 20:20:13 +00:00
Liu-Cheng Xu 6b1e871137 Expose ValidatedTransaction from transaction pool (#11588)
This is required to make a tx pool wrapper for some custom transaction validation, specificially
required to make a wrapper around the `LocalTransactionPool` implementation (https://github.com/subspace/subspace/blob/f54881a9b5/crates/subspace-service/src/pool.rs#L232-L269).

Related: https://github.com/paritytech/substrate/discussions/11520
2022-06-03 14:33:59 +02:00
Falco Hirschenberger b581604aa7 Apply some clippy lints (#11154)
* Apply some clippy hints

* Revert clippy ci changes

* Update client/cli/src/commands/generate.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/inspect_key.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/db/src/bench.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/db/src/bench.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/block_rules.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/block_rules.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/network/src/transactions.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/network/src/protocol.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Revert due to missing `or_default` function.

* Fix compilation and simplify code

* Undo change that corrupts benchmark.

* fix clippy

* Update client/service/test/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/state-db/src/noncanonical.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/state-db/src/noncanonical.rs

remove leftovers!

* Update client/tracing/src/logging/directives.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/fork-tree/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* added needed ref

* Update frame/referenda/src/benchmarking.rs

* Simplify byte-vec creation

* let's just not overlap the ranges

* Correction

* cargo fmt

* Update utils/frame/benchmarking-cli/src/shared/stats.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/frame/benchmarking-cli/src/pallet/command.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/frame/benchmarking-cli/src/pallet/command.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Giles Cope <gilescope@gmail.com>
2022-04-30 21:28:27 +00:00
Qinxuan Chen 41c84b0b41 Remove useless imports (#11136)
Because `TryInto`/`TryFrom` are in prelude by default from edition 2021

Signed-off-by: koushiro <koushiro.cqx@gmail.com>
2022-04-03 21:44:29 +02:00
Koute 9a31b2c341 Add a new host function for reporting fatal errors; make WASM backtraces readable when printing out errors (#10741)
* Add a new host function for reporting fatal errors

* Fix one of the wasmtime executor tests

* Have `#[runtime_interface(wasm_only)]` actually mean WASM-only, and not no_std-only

* Print out errors through `Display` instead of `Debug`

* Switch one more trait to require `Error` for its error instead of only `Debug`

* Align to review comments
2022-02-09 09:12:55 +00:00
Davide Galassi 7f3bb8d0da Add stale branches heads to finality notifications (#10639)
* Add stale branches heads to finality notifications

Warning. Previous implementation was sending a notification for
each block between the previous (explicitly) finalized block and
the new finalized one (with an hardcoded limit of 256).

Now finality notification is sent only for the new finalized head and it
contains the hash of the new finalized head, new finalized head header,
a list of all the implicitly finalized blocks and a list of stale
branches heads (i.e. the branches heads that are not part of the
canonical chain anymore).

* Add implicitly finalized blocks list to `ChainEvent::Finalized` message

The list contains all the blocks between the previously finalized block
up to the parent of the currently finalized one, sorted by block number.

`Finalized` messages handler, part of the `MaintainedTransactionPool`
implementation for `BasicPool`, still propagate full set of finalized
blocks to the txpool by iterating over implicitly finalized blocks list.

* Rust fmt

* Greedy evaluation of `stale_heads` during finalization

* Fix outdated assumption in a comment

* Removed a test optimization that is no more relevant

The loop was there to prevent sending to
`peer.network.on_block_finalized` the full list of finalized blocks.

Now only the finalized heads are received.

* Last finalized block lookup not required anymore

* Tests for block finality notifications payloads

* Document a bit tricky condition to avoid duplicate finalization notifications

* More idiomatic way to skip an iterator entry

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Cargo fmt iteration

* Typo fix

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Fix potential failure when a finalized orphan block is imported

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2022-01-27 18:21:58 +01:00
Qinxuan Chen e956c2e1c7 use thiserror instead of derive_more for error handling (#10696)
* use thiserror instead of derive_more for error handling

Signed-off-by: koushiro <koushiro.cqx@gmail.com>

* Update utils/prometheus/src/lib.rs

* Update utils/prometheus/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2022-01-25 19:48:46 +00:00
Bastian Köcher 424ddf33ee Remove transaction-pool test-helpers feature (#10571)
* Remove transaction-pool `test-helpers` feature

`test-helpers` feature is a bad idea in general, because once the feature is enabled somewhere in
the workspace, it is enabled anywhere. While removing the feature, the tests were also rewritten to
get rid off other "only test" related code.

Contributes towards: https://github.com/paritytech/substrate/issues/9727

* Apply suggestions from code review

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Fix benches

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2022-01-03 15:08:42 +00:00
Bastian Köcher f3662b4bba Happy new year 2022 (#10573) 2022-01-03 09:22:14 +01:00
Arkadiy Paronyan 4cbbf0cf43 Kill the light client, CHTs and change tries. (#10080)
* Remove light client, change tries and CHTs

* Update tests

* fmt

* Restore changes_root

* Fixed benches

* Cargo fmt

* fmt

* fmt
2021-11-12 14:15:01 +01:00
sandreim fdb3c64243 Add group name in task metrics (#10196)
* SpawnNamed: add new trait methods

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* Implement new methods

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* cargo fmt

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* SpawnNamed: add new trait methods

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* Implement new methods

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* cargo fmt

Signed-off-by: Andrei Sandu <sandu.andrei@gmail.com>

* New approach - spaw() group param

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Update traits: SpawnNamed and SpawnNamed

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Update TaskManager tests

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Update test TaskExecutor

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Fix typo

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* grunt work: fix spawn() calls

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* cargo fmt

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* remove old code

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* cargo fmt - the right version

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Implement review feedback

- use Option group name in SpawnNamed methods
- switch to kebab case
- implement default group name
- add group name to some tasks

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
2021-11-11 17:15:09 +00:00
Bastian Köcher 451413f47e Transaction pool: Adds benchmark and improves performance (#9958)
* Yep

* Make it compile

* Make the benchmark work

* Some stuff

* Optimize transaction pool `BestIterator`

* Some docs

* Fix more warnings

* Fix compilation

* FMT
2021-10-08 10:02:25 +02:00
Tomasz Drwięga 085935dd0a Quickly skip invalid transactions during block authorship. (#9789)
* Support skipping invalid transactions in the iterator.

* Expose concrete iterator.

* cargo +nightly fmt --all

* More consistent placement.

* Update Cargo.lock

* Pass transaction to 'report_invalid'
2021-10-01 14:25:13 +00:00
Bastian Köcher 129c9ed09e Remove the last bits of unknown_os in the code base (#9718)
* Remove the last bits of unknown_os in the code base

* Fmt
2021-09-09 11:17:16 +02:00
tgmichel f7a421f488 graph::ChainApi and graph::Pool reexport (#9726)
* `graph::ChainApi` and `graph::Pool` reexport

* Redundant import cleanup
2021-09-08 15:57:34 +02:00
David 2de7e51c2a Remove substrate-in-the-browser (#9541)
* Comment out browser stuff

* Remove browser stuff

* Remove more wasm transport code

* Remove ExtTransport and rework how telemetry initialises.

* Change (most) wasm-timer using code to use std::time

* Rename CI-job

* Aura does not compile for wasm

* Remove testing in the browser on CI

* Update README

* Leave `StreamSink` be

* fmt
2021-08-17 18:06:23 +00:00
Bastian Köcher c44aba89e6 Upgrade jsonrpc to 0.18.0 (#9547)
* Upgrade jsonrpc to 0.18.0

I think this says all :P

* 🤦

* Fmt etc

* Fix tests

* Fix tests again...

* Better impl

* Revert "Tell dependabot to ignore jsonrpc-* updates (#9518)"

This reverts commit 6e0cd5587d.
2021-08-13 06:46:07 +00:00
Bastian Köcher 7b56ab15b4 Run cargo fmt on the whole code base (#9394)
* Run cargo fmt on the whole code base

* Second run

* Add CI check

* Fix compilation

* More unnecessary braces

* Handle weights

* Use --all

* Use correct attributes...

* Fix UI tests

* AHHHHHHHHH

* 🤦

* Docs

* Fix compilation

* 🤷

* Please stop

* 🤦 x 2

* More

* make rustfmt.toml consistent with polkadot

Co-authored-by: André Silva <andrerfosilva@gmail.com>
2021-07-21 14:32:32 +00:00
Seun Lanlege 2ae9d36758 sc-transcation-pool refactor (#9228)
* Use TransactionPool trait

* sc-transaction-pool-primitives

* sc-transaction-pool-api

* TP

* bye sc_transaction_graph

* fix line widths

* fix import errors

* fix import errors

* fix import errors 🤦🏾‍♂️

* fix import errors 🤦🏾‍♂️🤦🏾‍♂️🤦🏾‍♂️

* remove sp-keyring
2021-07-08 13:33:34 +00:00
Ashley 361d02f92d Remove txpool as an export of sc_transaction_pool, exporting the used components instead. (#9217)
* Remove `txpool` as an export of `sc_transaction_pool`, exporting the used components instead.

* Fix tests
2021-06-29 08:23:39 +00:00
Bastian Köcher 68833498c6 Transaction pool: Remove futures-diagnose and thread pool (#9074)
* Transaction pool: Remove futures-diagnose and thread pool

This pr removes `futures-diagnose` as this isn't used anymore. Besides
that the pr also removes the thread pool that was used to validate the
transactions in the background. Instead of this thread pool we now spawn
two separate long running tasks that we use to validate the
transactions. All tasks of the transaction pool are now also spawned as
essential tasks. This means, if any of these tasks is stopping, the node
will stop as well.

* Update client/transaction-pool/src/api.rs
2021-06-11 17:24:30 +00:00
Bastian Köcher 258c1a86f6 Transaction pool: Ensure that we prune transactions properly (#8963)
* Transaction pool: Ensure that we prune transactions properly

There was a bug in the transaction pool that we didn't pruned
transactions properly because we called `prune_known`, instead of `prune`.

This bug was introduced by:
https://github.com/paritytech/substrate/pull/4629

This is required to have stale extrinsics being removed properly, so
that they don't fill up the tx pool.

* Fix compilation

* Fix benches

* ...
2021-06-03 16:04:29 +02:00
Bastian Köcher 2ab5f9aeca Transactionpool: Make ready_at return earlier (#8995)
`ready_at` returns when we have processed the requested block. However,
on startup we already have processed the best block and there
are no transactions in the pool on startup anyway. So, we can set `updated_at`
to the best block on startup.

Besides that `ready_at` now returns early when there are no ready nor
any future transactions in the pool.
2021-06-02 18:13:47 +00:00
Ashley c408515501 Revert "Use SpawnTaskHandles for spawning tasks in the tx pool (#8958)" (#8983)
This reverts commit 84e402389b.
2021-06-01 15:14:33 +00:00
Ashley 84e402389b Use SpawnTaskHandles for spawning tasks in the tx pool (#8958)
* Remove futures-diagnose

* Use `SpawnTaskHandle`s for spawning tasks in the tx pool

* Box the spawner

* Fix tests

* Use the testing task executor
2021-06-01 14:28:03 +00:00
Squirrel c0dfe2c382 Spellling corrections (no code changes) (#8971)
* Spelling corrections

* As this might break let's do as a separate PR
2021-06-01 12:51:38 +02:00
Liu-Cheng Xu 7e59d172b8 Add trivial improvements to transaction pool (#8572)
* Add trival improvements to transaction pool

* .

* Add trival improvements to transaction pool

* Update client/transaction-pool/graph/src/future.rs

* Update client/transaction-pool/graph/src/base_pool.rs

* Fix transaction_debug test

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
2021-04-09 11:37:40 +00:00
Bastian Köcher 33f9becf41 Simplify runtime api error handling (#8114)
* Ahh

* Work work work

* Fix all the compilation errors

* Fix test

* More fixes...
2021-02-15 11:55:40 +00:00
André Silva 54def5f3d3 transaction-pool: drop unpropagable txs if local node cant author blocks (#8048)
* transaction-pool: drop unpropagable txs if local node cant author blocks

* fix test compilation

* transaction-pool: remove unnecessary static bound on CanAuthor

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* rpc-api: add translation for PoolError::Unactionable

* transaction-pool: add test for rejecting unactionable transactions

* basic-authorship: fix doc test

* transaction-pool: fix benchmark compilation

* transaction-pool: rename CanAuthor to IsValidator

* transaction-pool: nit in error message

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
2021-02-04 20:18:44 +01:00
Bastian Köcher e3e651f72c Happy new year (#7814)
* Happy new year

Updates the copyright years and fixes wrong license headers.

* Fix the template

* Split HEADER into HEADER-APACHE & HEADER-GPL
2021-01-04 09:03:13 +00:00
Bastian Köcher 5808f467c5 Make Substrate compile with latest nightly (#7381)
* Try to get it compiling

* One more

* Make stable happy

* Make stable even more happy ;)

* Update shell.nix
2020-10-23 21:08:04 +02:00
Bastian Köcher cde60b871e Revalidate transactions only on latest best block (#6824)
* Revalidate transactions only on latest best block

We should revalidate transactions only on the latest best block and not
on any arbitrary block. The revalidation before failed when there were
multiple blocks on the height given to the revalidation function, but no
block was imported as best block.

* Update test-utils/runtime/transaction-pool/src/lib.rs

Co-authored-by: Jaco Greeff <jacogr@gmail.com>

* Fix tests

* Only process best blocks in the transaction pool

Co-authored-by: Jaco Greeff <jacogr@gmail.com>
2020-08-07 13:58:51 +02:00
Ashley 9220b646d2 Various small improvements to service construction. (#6738)
* Remove service components and add build_network, build_offchain_workers etc

* Improve transaction pool api

* Remove commented out line

* Add PartialComponents

* Add BuildNetworkParams, documentation

* Remove unused imports in tests

* Apply suggestions from code review

Co-authored-by: Nikolay Volf <nikvolf@gmail.com>

* Remove unused imports in node-bench

Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
2020-07-28 15:21:33 +02:00
Ashley ffa2177d46 Remove the service builder (#6557)
* :)

* Slight tidy

* Remove ServiceBuilderCommand

* Remove whitespace

* Keep task manager alive for check_block/import_blocks

* Pass task_manager to run_until_exit

* WIP

* WIP

* Get rid of the macros

* Simplify a few chain components creation APIs related to the service

* Fix basic-authorship doc tests

* Remove DefaultQueue

* Update client/service/src/builder.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Move ExecutionExtensions comment around

* Remove unused BlakeTwo256

* Add sc-prelude

* Rename sc-prelude to sc-service-prelude

* Rename to sc-service-types

* Improve service types

* Fix line widths

* Remove sc-service-types and move type definitions to crates

* Update bin/node-template/node/src/service.rs

Co-authored-by: Seun Lanlege <seunlanlege@gmail.com>

* Add TLightClientWithHash

* Rework types

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Seun Lanlege <seunlanlege@gmail.com>
2020-07-22 19:10:27 +00:00
Ashley 234e7d0c3d Simplify a few chain components creation APIs related to the service (#6611)
* Simplify a few chain components creation APIs related to the service

* Fix basic-authorship doc tests

* Remove DefaultQueue

* Update client/service/src/builder.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Move ExecutionExtensions comment around

* Remove unused BlakeTwo256

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
2020-07-09 15:43:04 +02:00