mirror of
https://github.com/pezkuwichain/pezkuwi-subxt.git
synced 2026-06-28 20:17:25 +00:00
fd5f9292f5
Closes #2160 First part of [Extrinsic Horizon](https://github.com/paritytech/polkadot-sdk/issues/2415) Introduces a new trait `TransactionExtension` to replace `SignedExtension`. Introduce the idea of transactions which obey the runtime's extensions and have according Extension data (né Extra data) yet do not have hard-coded signatures. Deprecate the terminology of "Unsigned" when used for transactions/extrinsics owing to there now being "proper" unsigned transactions which obey the extension framework and "old-style" unsigned which do not. Instead we have __*General*__ for the former and __*Bare*__ for the latter. (Ultimately, the latter will be phased out as a type of transaction, and Bare will only be used for Inherents.) Types of extrinsic are now therefore: - Bare (no hardcoded signature, no Extra data; used to be known as "Unsigned") - Bare transactions (deprecated): Gossiped, validated with `ValidateUnsigned` (deprecated) and the `_bare_compat` bits of `TransactionExtension` (deprecated). - Inherents: Not gossiped, validated with `ProvideInherent`. - Extended (Extra data): Gossiped, validated via `TransactionExtension`. - Signed transactions (with a hardcoded signature). - General transactions (without a hardcoded signature). `TransactionExtension` differs from `SignedExtension` because: - A signature on the underlying transaction may validly not be present. - It may alter the origin during validation. - `pre_dispatch` is renamed to `prepare` and need not contain the checks present in `validate`. - `validate` and `prepare` is passed an `Origin` rather than a `AccountId`. - `validate` may pass arbitrary information into `prepare` via a new user-specifiable type `Val`. - `AdditionalSigned`/`additional_signed` is renamed to `Implicit`/`implicit`. It is encoded *for the entire transaction* and passed in to each extension as a new argument to `validate`. This facilitates the ability of extensions to acts as underlying crypto. There is a new `DispatchTransaction` trait which contains only default function impls and is impl'ed for any `TransactionExtension` impler. It provides several utility functions which reduce some of the tedium from using `TransactionExtension` (indeed, none of its regular functions should now need to be called directly). Three transaction version discriminator ("versions") are now permissible: - 0b000000100: Bare (used to be called "Unsigned"): contains Signature or Extra (extension data). After bare transactions are no longer supported, this will strictly identify an Inherents only. - 0b100000100: Old-school "Signed" Transaction: contains Signature and Extra (extension data). - 0b010000100: New-school "General" Transaction: contains Extra (extension data), but no Signature. For the New-school General Transaction, it becomes trivial for authors to publish extensions to the mechanism for authorizing an Origin, e.g. through new kinds of key-signing schemes, ZK proofs, pallet state, mutations over pre-authenticated origins or any combination of the above. ## Code Migration ### NOW: Getting it to build Wrap your `SignedExtension`s in `AsTransactionExtension`. This should be accompanied by renaming your aggregate type in line with the new terminology. E.g. Before: ```rust /// The SignedExtension to the basic transaction logic. pub type SignedExtra = ( /* snip */ MySpecialSignedExtension, ); /// Unchecked extrinsic type as expected by this runtime. pub type UncheckedExtrinsic = generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, SignedExtra>; ``` After: ```rust /// The extension to the basic transaction logic. pub type TxExtension = ( /* snip */ AsTransactionExtension<MySpecialSignedExtension>, ); /// Unchecked extrinsic type as expected by this runtime. pub type UncheckedExtrinsic = generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, TxExtension>; ``` You'll also need to alter any transaction building logic to add a `.into()` to make the conversion happen. E.g. Before: ```rust fn construct_extrinsic( /* snip */ ) -> UncheckedExtrinsic { let extra: SignedExtra = ( /* snip */ MySpecialSignedExtension::new(/* snip */), ); let payload = SignedPayload::new(call.clone(), extra.clone()).unwrap(); let signature = payload.using_encoded(|e| sender.sign(e)); UncheckedExtrinsic::new_signed( /* snip */ Signature::Sr25519(signature), extra, ) } ``` After: ```rust fn construct_extrinsic( /* snip */ ) -> UncheckedExtrinsic { let tx_ext: TxExtension = ( /* snip */ MySpecialSignedExtension::new(/* snip */).into(), ); let payload = SignedPayload::new(call.clone(), tx_ext.clone()).unwrap(); let signature = payload.using_encoded(|e| sender.sign(e)); UncheckedExtrinsic::new_signed( /* snip */ Signature::Sr25519(signature), tx_ext, ) } ``` ### SOON: Migrating to `TransactionExtension` Most `SignedExtension`s can be trivially converted to become a `TransactionExtension`. There are a few things to know. - Instead of a single trait like `SignedExtension`, you should now implement two traits individually: `TransactionExtensionBase` and `TransactionExtension`. - Weights are now a thing and must be provided via the new function `fn weight`. #### `TransactionExtensionBase` This trait takes care of anything which is not dependent on types specific to your runtime, most notably `Call`. - `AdditionalSigned`/`additional_signed` is renamed to `Implicit`/`implicit`. - Weight must be returned by implementing the `weight` function. If your extension is associated with a pallet, you'll probably want to do this via the pallet's existing benchmarking infrastructure. #### `TransactionExtension` Generally: - `pre_dispatch` is now `prepare` and you *should not reexecute the `validate` functionality in there*! - You don't get an account ID any more; you get an origin instead. If you need to presume an account ID, then you can use the trait function `AsSystemOriginSigner::as_system_origin_signer`. - You get an additional ticket, similar to `Pre`, called `Val`. This defines data which is passed from `validate` into `prepare`. This is important since you should not be duplicating logic from `validate` to `prepare`, you need a way of passing your working from the former into the latter. This is it. - This trait takes two type parameters: `Call` and `Context`. `Call` is the runtime call type which used to be an associated type; you can just move it to become a type parameter for your trait impl. `Context` is not currently used and you can safely implement over it as an unbounded type. - There's no `AccountId` associated type any more. Just remove it. Regarding `validate`: - You get three new parameters in `validate`; all can be ignored when migrating from `SignedExtension`. - `validate` returns a tuple on success; the second item in the tuple is the new ticket type `Self::Val` which gets passed in to `prepare`. If you use any information extracted during `validate` (off-chain and on-chain, non-mutating) in `prepare` (on-chain, mutating) then you can pass it through with this. For the tuple's last item, just return the `origin` argument. Regarding `prepare`: - This is renamed from `pre_dispatch`, but there is one change: - FUNCTIONALITY TO VALIDATE THE TRANSACTION NEED NOT BE DUPLICATED FROM `validate`!! - (This is different to `SignedExtension` which was required to run the same checks in `pre_dispatch` as in `validate`.) Regarding `post_dispatch`: - Since there are no unsigned transactions handled by `TransactionExtension`, `Pre` is always defined, so the first parameter is `Self::Pre` rather than `Option<Self::Pre>`. If you make use of `SignedExtension::validate_unsigned` or `SignedExtension::pre_dispatch_unsigned`, then: - Just use the regular versions of these functions instead. - Have your logic execute in the case that the `origin` is `None`. - Ensure your transaction creation logic creates a General Transaction rather than a Bare Transaction; this means having to include all `TransactionExtension`s' data. - `ValidateUnsigned` can still be used (for now) if you need to be able to construct transactions which contain none of the extension data, however these will be phased out in stage 2 of the Transactions Horizon, so you should consider moving to an extension-centric design. ## TODO - [x] Introduce `CheckSignature` impl of `TransactionExtension` to ensure it's possible to have crypto be done wholly in a `TransactionExtension`. - [x] Deprecate `SignedExtension` and move all uses in codebase to `TransactionExtension`. - [x] `ChargeTransactionPayment` - [x] `DummyExtension` - [x] `ChargeAssetTxPayment` (asset-tx-payment) - [x] `ChargeAssetTxPayment` (asset-conversion-tx-payment) - [x] `CheckWeight` - [x] `CheckTxVersion` - [x] `CheckSpecVersion` - [x] `CheckNonce` - [x] `CheckNonZeroSender` - [x] `CheckMortality` - [x] `CheckGenesis` - [x] `CheckOnlySudoAccount` - [x] `WatchDummy` - [x] `PrevalidateAttests` - [x] `GenericSignedExtension` - [x] `SignedExtension` (chain-polkadot-bulletin) - [x] `RefundSignedExtensionAdapter` - [x] Implement `fn weight` across the board. - [ ] Go through all pre-existing extensions which assume an account signer and explicitly handle the possibility of another kind of origin. - [x] `CheckNonce` should probably succeed in the case of a non-account origin. - [x] `CheckNonZeroSender` should succeed in the case of a non-account origin. - [x] `ChargeTransactionPayment` and family should fail in the case of a non-account origin. - [ ] - [x] Fix any broken tests. --------- Signed-off-by: georgepisaltu <george.pisaltu@parity.io> Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io> Signed-off-by: Andrei Sandu <andrei-mihail@parity.io> Co-authored-by: Nikhil Gupta <17176722+gupnik@users.noreply.github.com> Co-authored-by: georgepisaltu <52418509+georgepisaltu@users.noreply.github.com> Co-authored-by: Chevdor <chevdor@users.noreply.github.com> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Maciej <maciej.zyszkiewicz@parity.io> Co-authored-by: Javier Viola <javier@parity.io> Co-authored-by: Marcin S. <marcin@realemail.net> Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io> Co-authored-by: Javier Bullrich <javier@bullrich.dev> Co-authored-by: Koute <koute@users.noreply.github.com> Co-authored-by: Adrian Catangiu <adrian@parity.io> Co-authored-by: Vladimir Istyufeev <vladimir@parity.io> Co-authored-by: Ross Bulat <ross@parity.io> Co-authored-by: Gonçalo Pestana <g6pestana@gmail.com> Co-authored-by: Liam Aharon <liam.aharon@hotmail.com> Co-authored-by: Svyatoslav Nikolsky <svyatonik@gmail.com> Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com> Co-authored-by: ordian <write@reusable.software> Co-authored-by: Sebastian Kunert <skunert49@gmail.com> Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com> Co-authored-by: Dmitry Markin <dmitry@markin.tech> Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com> Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com> Co-authored-by: Julian Eager <eagr@tutanota.com> Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com> Co-authored-by: Davide Galassi <davxy@datawok.net> Co-authored-by: Dónal Murray <donal.murray@parity.io> Co-authored-by: yjh <yjh465402634@gmail.com> Co-authored-by: Tom Mi <tommi@niemi.lol> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Will | Paradox | ParaNodes.io <79228812+paradox-tt@users.noreply.github.com> Co-authored-by: Bastian Köcher <info@kchr.de> Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com> Co-authored-by: Joshy Orndorff <git-user-email.h0ly5@simplelogin.com> Co-authored-by: PG Herveou <pgherveou@gmail.com> Co-authored-by: Alexander Theißen <alex.theissen@me.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Juan Girini <juangirini@gmail.com> Co-authored-by: bader y <ibnbassem@gmail.com> Co-authored-by: James Wilson <james@jsdw.me> Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> Co-authored-by: asynchronous rob <rphmeier@gmail.com> Co-authored-by: Parth <desaiparth08@gmail.com> Co-authored-by: Andrew Jones <ascjones@gmail.com> Co-authored-by: Jonathan Udd <jonathan@dwellir.com> Co-authored-by: Serban Iorga <serban@parity.io> Co-authored-by: Egor_P <egor@parity.io> Co-authored-by: Branislav Kontur <bkontur@gmail.com> Co-authored-by: Evgeny Snitko <evgeny@parity.io> Co-authored-by: Just van Stam <vstam1@users.noreply.github.com> Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com> Co-authored-by: gupnik <nikhilgupta.iitk@gmail.com> Co-authored-by: dzmitry-lahoda <dzmitry@lahoda.pro> Co-authored-by: zhiqiangxu <652732310@qq.com> Co-authored-by: Nazar Mokrynskyi <nazar@mokrynskyi.com> Co-authored-by: Anwesh <anweshknayak@gmail.com> Co-authored-by: cheme <emericchevalier.pro@gmail.com> Co-authored-by: Sam Johnson <sam@durosoft.com> Co-authored-by: kianenigma <kian@parity.io> Co-authored-by: Jegor Sidorenko <5252494+jsidorenko@users.noreply.github.com> Co-authored-by: Muharem <ismailov.m.h@gmail.com> Co-authored-by: joepetrowski <joe@parity.io> Co-authored-by: Alexandru Gheorghe <49718502+alexggh@users.noreply.github.com> Co-authored-by: Gabriel Facco de Arruda <arrudagates@gmail.com> Co-authored-by: Squirrel <gilescope@gmail.com> Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> Co-authored-by: georgepisaltu <george.pisaltu@parity.io> Co-authored-by: command-bot <>
833 lines
24 KiB
Rust
833 lines
24 KiB
Rust
// This file is part of Substrate.
|
|
|
|
// Copyright (C) Parity Technologies (UK) Ltd.
|
|
// SPDX-License-Identifier: GPL-3.0-or-later WITH Classpath-exception-2.0
|
|
|
|
// This program is free software: you can redistribute it and/or modify
|
|
// it under the terms of the GNU General Public License as published by
|
|
// the Free Software Foundation, either version 3 of the License, or
|
|
// (at your option) any later version.
|
|
|
|
// This program is distributed in the hope that it will be useful,
|
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
// GNU General Public License for more details.
|
|
|
|
// You should have received a copy of the GNU General Public License
|
|
// along with this program. If not, see <https://www.gnu.org/licenses/>.
|
|
|
|
//! Db-based backend utility structures and functions, used by both
|
|
//! full and light storages.
|
|
|
|
use std::{fmt, fs, io, path::Path, sync::Arc};
|
|
|
|
use log::{debug, info};
|
|
|
|
use crate::{Database, DatabaseSource, DbHash};
|
|
use codec::Decode;
|
|
use sp_database::Transaction;
|
|
use sp_runtime::{
|
|
generic::BlockId,
|
|
traits::{Block as BlockT, Header as HeaderT, UniqueSaturatedFrom, UniqueSaturatedInto, Zero},
|
|
};
|
|
use sp_trie::DBValue;
|
|
|
|
/// Number of columns in the db. Must be the same for both full && light dbs.
|
|
/// Otherwise RocksDb will fail to open database && check its type.
|
|
pub const NUM_COLUMNS: u32 = 13;
|
|
/// Meta column. The set of keys in the column is shared by full && light storages.
|
|
pub const COLUMN_META: u32 = 0;
|
|
|
|
/// Keys of entries in COLUMN_META.
|
|
pub mod meta_keys {
|
|
/// Type of storage (full or light).
|
|
pub const TYPE: &[u8; 4] = b"type";
|
|
/// Best block key.
|
|
pub const BEST_BLOCK: &[u8; 4] = b"best";
|
|
/// Last finalized block key.
|
|
pub const FINALIZED_BLOCK: &[u8; 5] = b"final";
|
|
/// Last finalized state key.
|
|
pub const FINALIZED_STATE: &[u8; 6] = b"fstate";
|
|
/// Block gap.
|
|
pub const BLOCK_GAP: &[u8; 3] = b"gap";
|
|
/// Genesis block hash.
|
|
pub const GENESIS_HASH: &[u8; 3] = b"gen";
|
|
/// Leaves prefix list key.
|
|
pub const LEAF_PREFIX: &[u8; 4] = b"leaf";
|
|
/// Children prefix list key.
|
|
pub const CHILDREN_PREFIX: &[u8; 8] = b"children";
|
|
}
|
|
|
|
/// Database metadata.
|
|
#[derive(Debug)]
|
|
pub struct Meta<N, H> {
|
|
/// Hash of the best known block.
|
|
pub best_hash: H,
|
|
/// Number of the best known block.
|
|
pub best_number: N,
|
|
/// Hash of the best finalized block.
|
|
pub finalized_hash: H,
|
|
/// Number of the best finalized block.
|
|
pub finalized_number: N,
|
|
/// Hash of the genesis block.
|
|
pub genesis_hash: H,
|
|
/// Finalized state, if any
|
|
pub finalized_state: Option<(H, N)>,
|
|
/// Block gap, start and end inclusive, if any.
|
|
pub block_gap: Option<(N, N)>,
|
|
}
|
|
|
|
/// A block lookup key: used for canonical lookup from block number to hash
|
|
pub type NumberIndexKey = [u8; 4];
|
|
|
|
/// Database type.
|
|
#[derive(Clone, Copy, Debug, PartialEq)]
|
|
pub enum DatabaseType {
|
|
/// Full node database.
|
|
Full,
|
|
}
|
|
|
|
/// Convert block number into short lookup key (LE representation) for
|
|
/// blocks that are in the canonical chain.
|
|
///
|
|
/// In the current database schema, this kind of key is only used for
|
|
/// lookups into an index, NOT for storing header data or others.
|
|
pub fn number_index_key<N: TryInto<u32>>(n: N) -> sp_blockchain::Result<NumberIndexKey> {
|
|
let n = n.try_into().map_err(|_| {
|
|
sp_blockchain::Error::Backend("Block number cannot be converted to u32".into())
|
|
})?;
|
|
|
|
Ok([(n >> 24) as u8, ((n >> 16) & 0xff) as u8, ((n >> 8) & 0xff) as u8, (n & 0xff) as u8])
|
|
}
|
|
|
|
/// Convert number and hash into long lookup key for blocks that are
|
|
/// not in the canonical chain.
|
|
pub fn number_and_hash_to_lookup_key<N, H>(number: N, hash: H) -> sp_blockchain::Result<Vec<u8>>
|
|
where
|
|
N: TryInto<u32>,
|
|
H: AsRef<[u8]>,
|
|
{
|
|
let mut lookup_key = number_index_key(number)?.to_vec();
|
|
lookup_key.extend_from_slice(hash.as_ref());
|
|
Ok(lookup_key)
|
|
}
|
|
|
|
/// Delete number to hash mapping in DB transaction.
|
|
pub fn remove_number_to_key_mapping<N: TryInto<u32>>(
|
|
transaction: &mut Transaction<DbHash>,
|
|
key_lookup_col: u32,
|
|
number: N,
|
|
) -> sp_blockchain::Result<()> {
|
|
transaction.remove(key_lookup_col, number_index_key(number)?.as_ref());
|
|
Ok(())
|
|
}
|
|
|
|
/// Place a number mapping into the database. This maps number to current perceived
|
|
/// block hash at that position.
|
|
pub fn insert_number_to_key_mapping<N: TryInto<u32> + Clone, H: AsRef<[u8]>>(
|
|
transaction: &mut Transaction<DbHash>,
|
|
key_lookup_col: u32,
|
|
number: N,
|
|
hash: H,
|
|
) -> sp_blockchain::Result<()> {
|
|
transaction.set_from_vec(
|
|
key_lookup_col,
|
|
number_index_key(number.clone())?.as_ref(),
|
|
number_and_hash_to_lookup_key(number, hash)?,
|
|
);
|
|
Ok(())
|
|
}
|
|
|
|
/// Insert a hash to key mapping in the database.
|
|
pub fn insert_hash_to_key_mapping<N: TryInto<u32>, H: AsRef<[u8]> + Clone>(
|
|
transaction: &mut Transaction<DbHash>,
|
|
key_lookup_col: u32,
|
|
number: N,
|
|
hash: H,
|
|
) -> sp_blockchain::Result<()> {
|
|
transaction.set_from_vec(
|
|
key_lookup_col,
|
|
hash.as_ref(),
|
|
number_and_hash_to_lookup_key(number, hash.clone())?,
|
|
);
|
|
Ok(())
|
|
}
|
|
|
|
/// Convert block id to block lookup key.
|
|
/// block lookup key is the DB-key header, block and justification are stored under.
|
|
/// looks up lookup key by hash from DB as necessary.
|
|
pub fn block_id_to_lookup_key<Block>(
|
|
db: &dyn Database<DbHash>,
|
|
key_lookup_col: u32,
|
|
id: BlockId<Block>,
|
|
) -> Result<Option<Vec<u8>>, sp_blockchain::Error>
|
|
where
|
|
Block: BlockT,
|
|
::sp_runtime::traits::NumberFor<Block>: UniqueSaturatedFrom<u64> + UniqueSaturatedInto<u64>,
|
|
{
|
|
Ok(match id {
|
|
BlockId::Number(n) => db.get(key_lookup_col, number_index_key(n)?.as_ref()),
|
|
BlockId::Hash(h) => db.get(key_lookup_col, h.as_ref()),
|
|
})
|
|
}
|
|
|
|
/// Opens the configured database.
|
|
pub fn open_database<Block: BlockT>(
|
|
db_source: &DatabaseSource,
|
|
db_type: DatabaseType,
|
|
create: bool,
|
|
) -> OpenDbResult {
|
|
// Maybe migrate (copy) the database to a type specific subdirectory to make it
|
|
// possible that light and full databases coexist
|
|
// NOTE: This function can be removed in a few releases
|
|
maybe_migrate_to_type_subdir::<Block>(db_source, db_type)?;
|
|
|
|
open_database_at::<Block>(db_source, db_type, create)
|
|
}
|
|
|
|
fn open_database_at<Block: BlockT>(
|
|
db_source: &DatabaseSource,
|
|
db_type: DatabaseType,
|
|
create: bool,
|
|
) -> OpenDbResult {
|
|
let db: Arc<dyn Database<DbHash>> = match &db_source {
|
|
DatabaseSource::ParityDb { path } => open_parity_db::<Block>(path, db_type, create)?,
|
|
#[cfg(feature = "rocksdb")]
|
|
DatabaseSource::RocksDb { path, cache_size } =>
|
|
open_kvdb_rocksdb::<Block>(path, db_type, create, *cache_size)?,
|
|
DatabaseSource::Custom { db, require_create_flag } => {
|
|
if *require_create_flag && !create {
|
|
return Err(OpenDbError::DoesNotExist)
|
|
}
|
|
db.clone()
|
|
},
|
|
DatabaseSource::Auto { paritydb_path, rocksdb_path, cache_size } => {
|
|
// check if rocksdb exists first, if not, open paritydb
|
|
match open_kvdb_rocksdb::<Block>(rocksdb_path, db_type, false, *cache_size) {
|
|
Ok(db) => db,
|
|
Err(OpenDbError::NotEnabled(_)) | Err(OpenDbError::DoesNotExist) =>
|
|
open_parity_db::<Block>(paritydb_path, db_type, create)?,
|
|
Err(as_is) => return Err(as_is),
|
|
}
|
|
},
|
|
};
|
|
|
|
check_database_type(&*db, db_type)?;
|
|
Ok(db)
|
|
}
|
|
|
|
#[derive(Debug)]
|
|
pub enum OpenDbError {
|
|
// constructed only when rocksdb and paritydb are disabled
|
|
#[allow(dead_code)]
|
|
NotEnabled(&'static str),
|
|
DoesNotExist,
|
|
Internal(String),
|
|
DatabaseError(sp_database::error::DatabaseError),
|
|
UnexpectedDbType {
|
|
expected: DatabaseType,
|
|
found: Vec<u8>,
|
|
},
|
|
}
|
|
|
|
type OpenDbResult = Result<Arc<dyn Database<DbHash>>, OpenDbError>;
|
|
|
|
impl fmt::Display for OpenDbError {
|
|
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
|
|
match self {
|
|
OpenDbError::Internal(e) => write!(f, "{}", e),
|
|
OpenDbError::DoesNotExist => write!(f, "Database does not exist at given location"),
|
|
OpenDbError::NotEnabled(feat) => {
|
|
write!(f, "`{}` feature not enabled, database can not be opened", feat)
|
|
},
|
|
OpenDbError::DatabaseError(db_error) => {
|
|
write!(f, "Database Error: {}", db_error)
|
|
},
|
|
OpenDbError::UnexpectedDbType { expected, found } => {
|
|
write!(
|
|
f,
|
|
"Unexpected DB-Type. Expected: {:?}, Found: {:?}",
|
|
expected.as_str().as_bytes(),
|
|
found
|
|
)
|
|
},
|
|
}
|
|
}
|
|
}
|
|
|
|
impl From<OpenDbError> for sp_blockchain::Error {
|
|
fn from(err: OpenDbError) -> Self {
|
|
sp_blockchain::Error::Backend(err.to_string())
|
|
}
|
|
}
|
|
|
|
impl From<parity_db::Error> for OpenDbError {
|
|
fn from(err: parity_db::Error) -> Self {
|
|
if matches!(err, parity_db::Error::DatabaseNotFound) {
|
|
OpenDbError::DoesNotExist
|
|
} else {
|
|
OpenDbError::Internal(err.to_string())
|
|
}
|
|
}
|
|
}
|
|
|
|
impl From<io::Error> for OpenDbError {
|
|
fn from(err: io::Error) -> Self {
|
|
if err.to_string().contains("create_if_missing is false") {
|
|
OpenDbError::DoesNotExist
|
|
} else {
|
|
OpenDbError::Internal(err.to_string())
|
|
}
|
|
}
|
|
}
|
|
|
|
fn open_parity_db<Block: BlockT>(path: &Path, db_type: DatabaseType, create: bool) -> OpenDbResult {
|
|
match crate::parity_db::open(path, db_type, create, false) {
|
|
Ok(db) => Ok(db),
|
|
Err(parity_db::Error::InvalidConfiguration(_)) => {
|
|
log::warn!("Invalid parity db configuration, attempting database metadata update.");
|
|
// Try to update the database with the new config
|
|
Ok(crate::parity_db::open(path, db_type, create, true)?)
|
|
},
|
|
Err(e) => Err(e.into()),
|
|
}
|
|
}
|
|
|
|
#[cfg(any(feature = "rocksdb", test))]
|
|
fn open_kvdb_rocksdb<Block: BlockT>(
|
|
path: &Path,
|
|
db_type: DatabaseType,
|
|
create: bool,
|
|
cache_size: usize,
|
|
) -> OpenDbResult {
|
|
// first upgrade database to required version
|
|
match crate::upgrade::upgrade_db::<Block>(path, db_type) {
|
|
// in case of missing version file, assume that database simply does not exist at given
|
|
// location
|
|
Ok(_) | Err(crate::upgrade::UpgradeError::MissingDatabaseVersionFile) => (),
|
|
Err(err) => return Err(io::Error::new(io::ErrorKind::Other, err.to_string()).into()),
|
|
}
|
|
|
|
// and now open database assuming that it has the latest version
|
|
let mut db_config = kvdb_rocksdb::DatabaseConfig::with_columns(NUM_COLUMNS);
|
|
db_config.create_if_missing = create;
|
|
|
|
let mut memory_budget = std::collections::HashMap::new();
|
|
match db_type {
|
|
DatabaseType::Full => {
|
|
let state_col_budget = (cache_size as f64 * 0.9) as usize;
|
|
let other_col_budget = (cache_size - state_col_budget) / (NUM_COLUMNS as usize - 1);
|
|
|
|
for i in 0..NUM_COLUMNS {
|
|
if i == crate::columns::STATE {
|
|
memory_budget.insert(i, state_col_budget);
|
|
} else {
|
|
memory_budget.insert(i, other_col_budget);
|
|
}
|
|
}
|
|
log::trace!(
|
|
target: "db",
|
|
"Open RocksDB database at {:?}, state column budget: {} MiB, others({}) column cache: {} MiB",
|
|
path,
|
|
state_col_budget,
|
|
NUM_COLUMNS,
|
|
other_col_budget,
|
|
);
|
|
},
|
|
}
|
|
db_config.memory_budget = memory_budget;
|
|
|
|
let db = kvdb_rocksdb::Database::open(&db_config, path)?;
|
|
// write database version only after the database is succesfully opened
|
|
crate::upgrade::update_version(path)?;
|
|
Ok(sp_database::as_database(db))
|
|
}
|
|
|
|
#[cfg(not(any(feature = "rocksdb", test)))]
|
|
fn open_kvdb_rocksdb<Block: BlockT>(
|
|
_path: &Path,
|
|
_db_type: DatabaseType,
|
|
_create: bool,
|
|
_cache_size: usize,
|
|
) -> OpenDbResult {
|
|
Err(OpenDbError::NotEnabled("with-kvdb-rocksdb"))
|
|
}
|
|
|
|
/// Check database type.
|
|
pub fn check_database_type(
|
|
db: &dyn Database<DbHash>,
|
|
db_type: DatabaseType,
|
|
) -> Result<(), OpenDbError> {
|
|
match db.get(COLUMN_META, meta_keys::TYPE) {
|
|
Some(stored_type) =>
|
|
if db_type.as_str().as_bytes() != &*stored_type {
|
|
return Err(OpenDbError::UnexpectedDbType {
|
|
expected: db_type,
|
|
found: stored_type.to_owned(),
|
|
})
|
|
},
|
|
None => {
|
|
let mut transaction = Transaction::new();
|
|
transaction.set(COLUMN_META, meta_keys::TYPE, db_type.as_str().as_bytes());
|
|
db.commit(transaction).map_err(OpenDbError::DatabaseError)?;
|
|
},
|
|
}
|
|
|
|
Ok(())
|
|
}
|
|
|
|
fn maybe_migrate_to_type_subdir<Block: BlockT>(
|
|
source: &DatabaseSource,
|
|
db_type: DatabaseType,
|
|
) -> Result<(), OpenDbError> {
|
|
if let Some(p) = source.path() {
|
|
let mut basedir = p.to_path_buf();
|
|
basedir.pop();
|
|
|
|
// Do we have to migrate to a database-type-based subdirectory layout:
|
|
// See if there's a file identifying a rocksdb or paritydb folder in the parent dir and
|
|
// the target path ends in a role specific directory
|
|
if (basedir.join("db_version").exists() || basedir.join("metadata").exists()) &&
|
|
(p.ends_with(DatabaseType::Full.as_str()))
|
|
{
|
|
// Try to open the database to check if the current `DatabaseType` matches the type of
|
|
// database stored in the target directory and close the database on success.
|
|
let mut old_source = source.clone();
|
|
old_source.set_path(&basedir);
|
|
open_database_at::<Block>(&old_source, db_type, false)?;
|
|
|
|
info!(
|
|
"Migrating database to a database-type-based subdirectory: '{:?}' -> '{:?}'",
|
|
basedir,
|
|
basedir.join(db_type.as_str())
|
|
);
|
|
|
|
let mut tmp_dir = basedir.clone();
|
|
tmp_dir.pop();
|
|
tmp_dir.push("tmp");
|
|
|
|
fs::rename(&basedir, &tmp_dir)?;
|
|
fs::create_dir_all(&p)?;
|
|
fs::rename(tmp_dir, &p)?;
|
|
}
|
|
}
|
|
|
|
Ok(())
|
|
}
|
|
|
|
/// Read database column entry for the given block.
|
|
pub fn read_db<Block>(
|
|
db: &dyn Database<DbHash>,
|
|
col_index: u32,
|
|
col: u32,
|
|
id: BlockId<Block>,
|
|
) -> sp_blockchain::Result<Option<DBValue>>
|
|
where
|
|
Block: BlockT,
|
|
{
|
|
block_id_to_lookup_key(db, col_index, id).map(|key| match key {
|
|
Some(key) => db.get(col, key.as_ref()),
|
|
None => None,
|
|
})
|
|
}
|
|
|
|
/// Remove database column entry for the given block.
|
|
pub fn remove_from_db<Block>(
|
|
transaction: &mut Transaction<DbHash>,
|
|
db: &dyn Database<DbHash>,
|
|
col_index: u32,
|
|
col: u32,
|
|
id: BlockId<Block>,
|
|
) -> sp_blockchain::Result<()>
|
|
where
|
|
Block: BlockT,
|
|
{
|
|
block_id_to_lookup_key(db, col_index, id).map(|key| {
|
|
if let Some(key) = key {
|
|
transaction.remove(col, key.as_ref());
|
|
}
|
|
})
|
|
}
|
|
|
|
/// Read a header from the database.
|
|
pub fn read_header<Block: BlockT>(
|
|
db: &dyn Database<DbHash>,
|
|
col_index: u32,
|
|
col: u32,
|
|
id: BlockId<Block>,
|
|
) -> sp_blockchain::Result<Option<Block::Header>> {
|
|
match read_db(db, col_index, col, id)? {
|
|
Some(header) => match Block::Header::decode(&mut &header[..]) {
|
|
Ok(header) => Ok(Some(header)),
|
|
Err(_) => Err(sp_blockchain::Error::Backend("Error decoding header".into())),
|
|
},
|
|
None => Ok(None),
|
|
}
|
|
}
|
|
|
|
/// Read meta from the database.
|
|
pub fn read_meta<Block>(
|
|
db: &dyn Database<DbHash>,
|
|
col_header: u32,
|
|
) -> Result<Meta<<<Block as BlockT>::Header as HeaderT>::Number, Block::Hash>, sp_blockchain::Error>
|
|
where
|
|
Block: BlockT,
|
|
{
|
|
let genesis_hash: Block::Hash = match read_genesis_hash(db)? {
|
|
Some(genesis_hash) => genesis_hash,
|
|
None =>
|
|
return Ok(Meta {
|
|
best_hash: Default::default(),
|
|
best_number: Zero::zero(),
|
|
finalized_hash: Default::default(),
|
|
finalized_number: Zero::zero(),
|
|
genesis_hash: Default::default(),
|
|
finalized_state: None,
|
|
block_gap: None,
|
|
}),
|
|
};
|
|
|
|
let load_meta_block = |desc, key| -> Result<_, sp_blockchain::Error> {
|
|
if let Some(Some(header)) = db
|
|
.get(COLUMN_META, key)
|
|
.and_then(|id| db.get(col_header, &id).map(|b| Block::Header::decode(&mut &b[..]).ok()))
|
|
{
|
|
let hash = header.hash();
|
|
debug!(
|
|
target: "db",
|
|
"Opened blockchain db, fetched {} = {:?} ({})",
|
|
desc,
|
|
hash,
|
|
header.number(),
|
|
);
|
|
Ok((hash, *header.number()))
|
|
} else {
|
|
Ok((Default::default(), Zero::zero()))
|
|
}
|
|
};
|
|
|
|
let (best_hash, best_number) = load_meta_block("best", meta_keys::BEST_BLOCK)?;
|
|
let (finalized_hash, finalized_number) = load_meta_block("final", meta_keys::FINALIZED_BLOCK)?;
|
|
let (finalized_state_hash, finalized_state_number) =
|
|
load_meta_block("final_state", meta_keys::FINALIZED_STATE)?;
|
|
let finalized_state = if finalized_state_hash != Default::default() {
|
|
Some((finalized_state_hash, finalized_state_number))
|
|
} else {
|
|
None
|
|
};
|
|
let block_gap = db
|
|
.get(COLUMN_META, meta_keys::BLOCK_GAP)
|
|
.and_then(|d| Decode::decode(&mut d.as_slice()).ok());
|
|
debug!(target: "db", "block_gap={:?}", block_gap);
|
|
|
|
Ok(Meta {
|
|
best_hash,
|
|
best_number,
|
|
finalized_hash,
|
|
finalized_number,
|
|
genesis_hash,
|
|
finalized_state,
|
|
block_gap,
|
|
})
|
|
}
|
|
|
|
/// Read genesis hash from database.
|
|
pub fn read_genesis_hash<Hash: Decode>(
|
|
db: &dyn Database<DbHash>,
|
|
) -> sp_blockchain::Result<Option<Hash>> {
|
|
match db.get(COLUMN_META, meta_keys::GENESIS_HASH) {
|
|
Some(h) => match Decode::decode(&mut &h[..]) {
|
|
Ok(h) => Ok(Some(h)),
|
|
Err(err) =>
|
|
Err(sp_blockchain::Error::Backend(format!("Error decoding genesis hash: {}", err))),
|
|
},
|
|
None => Ok(None),
|
|
}
|
|
}
|
|
|
|
impl DatabaseType {
|
|
/// Returns str representation of the type.
|
|
pub fn as_str(&self) -> &'static str {
|
|
match *self {
|
|
DatabaseType::Full => "full",
|
|
}
|
|
}
|
|
}
|
|
|
|
pub(crate) struct JoinInput<'a, 'b>(&'a [u8], &'b [u8]);
|
|
|
|
pub(crate) fn join_input<'a, 'b>(i1: &'a [u8], i2: &'b [u8]) -> JoinInput<'a, 'b> {
|
|
JoinInput(i1, i2)
|
|
}
|
|
|
|
impl<'a, 'b> codec::Input for JoinInput<'a, 'b> {
|
|
fn remaining_len(&mut self) -> Result<Option<usize>, codec::Error> {
|
|
Ok(Some(self.0.len() + self.1.len()))
|
|
}
|
|
|
|
fn read(&mut self, into: &mut [u8]) -> Result<(), codec::Error> {
|
|
let mut read = 0;
|
|
if self.0.len() > 0 {
|
|
read = std::cmp::min(self.0.len(), into.len());
|
|
self.0.read(&mut into[..read])?;
|
|
}
|
|
if read < into.len() {
|
|
self.1.read(&mut into[read..])?;
|
|
}
|
|
Ok(())
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
use codec::Input;
|
|
use sp_runtime::{
|
|
generic::UncheckedExtrinsic,
|
|
testing::{Block as RawBlock, MockCallU64},
|
|
};
|
|
|
|
pub type UncheckedXt = UncheckedExtrinsic<u64, MockCallU64, (), ()>;
|
|
type Block = RawBlock<UncheckedXt>;
|
|
|
|
#[cfg(feature = "rocksdb")]
|
|
#[test]
|
|
fn database_type_subdir_migration() {
|
|
use std::path::PathBuf;
|
|
type Block = RawBlock<UncheckedXt>;
|
|
|
|
fn check_dir_for_db_type(
|
|
db_type: DatabaseType,
|
|
mut source: DatabaseSource,
|
|
db_check_file: &str,
|
|
) {
|
|
let base_path = tempfile::TempDir::new().unwrap();
|
|
let old_db_path = base_path.path().join("chains/dev/db");
|
|
|
|
source.set_path(&old_db_path);
|
|
|
|
{
|
|
let db_res = open_database::<Block>(&source, db_type, true);
|
|
assert!(db_res.is_ok(), "New database should be created.");
|
|
assert!(old_db_path.join(db_check_file).exists());
|
|
assert!(!old_db_path.join(db_type.as_str()).join("db_version").exists());
|
|
}
|
|
|
|
source.set_path(&old_db_path.join(db_type.as_str()));
|
|
|
|
let db_res = open_database::<Block>(&source, db_type, true);
|
|
assert!(db_res.is_ok(), "Reopening the db with the same role should work");
|
|
// check if the database dir had been migrated
|
|
assert!(!old_db_path.join(db_check_file).exists());
|
|
assert!(old_db_path.join(db_type.as_str()).join(db_check_file).exists());
|
|
}
|
|
|
|
check_dir_for_db_type(
|
|
DatabaseType::Full,
|
|
DatabaseSource::RocksDb { path: PathBuf::new(), cache_size: 128 },
|
|
"db_version",
|
|
);
|
|
|
|
check_dir_for_db_type(
|
|
DatabaseType::Full,
|
|
DatabaseSource::ParityDb { path: PathBuf::new() },
|
|
"metadata",
|
|
);
|
|
|
|
// check failure on reopening with wrong role
|
|
{
|
|
let base_path = tempfile::TempDir::new().unwrap();
|
|
let old_db_path = base_path.path().join("chains/dev/db");
|
|
|
|
let source = DatabaseSource::RocksDb { path: old_db_path.clone(), cache_size: 128 };
|
|
{
|
|
let db_res = open_database::<Block>(&source, DatabaseType::Full, true);
|
|
assert!(db_res.is_ok(), "New database should be created.");
|
|
|
|
// check if the database dir had been migrated
|
|
assert!(old_db_path.join("db_version").exists());
|
|
assert!(!old_db_path.join("light/db_version").exists());
|
|
assert!(!old_db_path.join("full/db_version").exists());
|
|
}
|
|
// assert nothing was changed
|
|
assert!(old_db_path.join("db_version").exists());
|
|
assert!(!old_db_path.join("full/db_version").exists());
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn number_index_key_doesnt_panic() {
|
|
let id = BlockId::<Block>::Number(72340207214430721);
|
|
match id {
|
|
BlockId::Number(n) => number_index_key(n).expect_err("number should overflow u32"),
|
|
_ => unreachable!(),
|
|
};
|
|
}
|
|
|
|
#[test]
|
|
fn database_type_as_str_works() {
|
|
assert_eq!(DatabaseType::Full.as_str(), "full");
|
|
}
|
|
|
|
#[test]
|
|
fn join_input_works() {
|
|
let buf1 = [1, 2, 3, 4];
|
|
let buf2 = [5, 6, 7, 8];
|
|
let mut test = [0, 0, 0];
|
|
let mut joined = join_input(buf1.as_ref(), buf2.as_ref());
|
|
assert_eq!(joined.remaining_len().unwrap(), Some(8));
|
|
|
|
joined.read(&mut test).unwrap();
|
|
assert_eq!(test, [1, 2, 3]);
|
|
assert_eq!(joined.remaining_len().unwrap(), Some(5));
|
|
|
|
joined.read(&mut test).unwrap();
|
|
assert_eq!(test, [4, 5, 6]);
|
|
assert_eq!(joined.remaining_len().unwrap(), Some(2));
|
|
|
|
joined.read(&mut test[0..2]).unwrap();
|
|
assert_eq!(test, [7, 8, 6]);
|
|
assert_eq!(joined.remaining_len().unwrap(), Some(0));
|
|
}
|
|
|
|
#[cfg(feature = "rocksdb")]
|
|
#[test]
|
|
fn test_open_database_auto_new() {
|
|
let db_dir = tempfile::TempDir::new().unwrap();
|
|
let db_path = db_dir.path().to_owned();
|
|
let paritydb_path = db_path.join("paritydb");
|
|
let rocksdb_path = db_path.join("rocksdb_path");
|
|
let source = DatabaseSource::Auto {
|
|
paritydb_path: paritydb_path.clone(),
|
|
rocksdb_path: rocksdb_path.clone(),
|
|
cache_size: 128,
|
|
};
|
|
|
|
// it should create new auto (paritydb) database
|
|
{
|
|
let db_res = open_database::<Block>(&source, DatabaseType::Full, true);
|
|
assert!(db_res.is_ok(), "New database should be created.");
|
|
}
|
|
|
|
// it should reopen existing auto (pairtydb) database
|
|
{
|
|
let db_res = open_database::<Block>(&source, DatabaseType::Full, true);
|
|
assert!(db_res.is_ok(), "Existing parity database should be reopened");
|
|
}
|
|
|
|
// it should fail to open existing auto (pairtydb) database
|
|
{
|
|
let db_res = open_database::<Block>(
|
|
&DatabaseSource::RocksDb { path: rocksdb_path, cache_size: 128 },
|
|
DatabaseType::Full,
|
|
true,
|
|
);
|
|
assert!(db_res.is_ok(), "New database should be opened.");
|
|
}
|
|
|
|
// it should reopen existing auto (pairtydb) database
|
|
{
|
|
let db_res = open_database::<Block>(
|
|
&DatabaseSource::ParityDb { path: paritydb_path },
|
|
DatabaseType::Full,
|
|
true,
|
|
);
|
|
assert!(db_res.is_ok(), "Existing parity database should be reopened");
|
|
}
|
|
}
|
|
|
|
#[cfg(feature = "rocksdb")]
|
|
#[test]
|
|
fn test_open_database_rocksdb_new() {
|
|
let db_dir = tempfile::TempDir::new().unwrap();
|
|
let db_path = db_dir.path().to_owned();
|
|
let paritydb_path = db_path.join("paritydb");
|
|
let rocksdb_path = db_path.join("rocksdb_path");
|
|
|
|
let source = DatabaseSource::RocksDb { path: rocksdb_path.clone(), cache_size: 128 };
|
|
|
|
// it should create new rocksdb database
|
|
{
|
|
let db_res = open_database::<Block>(&source, DatabaseType::Full, true);
|
|
assert!(db_res.is_ok(), "New rocksdb database should be created");
|
|
}
|
|
|
|
// it should reopen existing auto (rocksdb) database
|
|
{
|
|
let db_res = open_database::<Block>(
|
|
&DatabaseSource::Auto {
|
|
paritydb_path: paritydb_path.clone(),
|
|
rocksdb_path: rocksdb_path.clone(),
|
|
cache_size: 128,
|
|
},
|
|
DatabaseType::Full,
|
|
true,
|
|
);
|
|
assert!(db_res.is_ok(), "Existing rocksdb database should be reopened");
|
|
}
|
|
|
|
// it should fail to open existing auto (rocksdb) database
|
|
{
|
|
let db_res = open_database::<Block>(
|
|
&DatabaseSource::ParityDb { path: paritydb_path },
|
|
DatabaseType::Full,
|
|
true,
|
|
);
|
|
assert!(db_res.is_ok(), "New paritydb database should be created");
|
|
}
|
|
|
|
// it should reopen existing auto (pairtydb) database
|
|
{
|
|
let db_res = open_database::<Block>(
|
|
&DatabaseSource::RocksDb { path: rocksdb_path, cache_size: 128 },
|
|
DatabaseType::Full,
|
|
true,
|
|
);
|
|
assert!(db_res.is_ok(), "Existing rocksdb database should be reopened");
|
|
}
|
|
}
|
|
|
|
#[cfg(feature = "rocksdb")]
|
|
#[test]
|
|
fn test_open_database_paritydb_new() {
|
|
let db_dir = tempfile::TempDir::new().unwrap();
|
|
let db_path = db_dir.path().to_owned();
|
|
let paritydb_path = db_path.join("paritydb");
|
|
let rocksdb_path = db_path.join("rocksdb_path");
|
|
|
|
let source = DatabaseSource::ParityDb { path: paritydb_path.clone() };
|
|
|
|
// it should create new paritydb database
|
|
{
|
|
let db_res = open_database::<Block>(&source, DatabaseType::Full, true);
|
|
assert!(db_res.is_ok(), "New database should be created.");
|
|
}
|
|
|
|
// it should reopen existing pairtydb database
|
|
{
|
|
let db_res = open_database::<Block>(&source, DatabaseType::Full, true);
|
|
assert!(db_res.is_ok(), "Existing parity database should be reopened");
|
|
}
|
|
|
|
// it should fail to open existing pairtydb database
|
|
{
|
|
let db_res = open_database::<Block>(
|
|
&DatabaseSource::RocksDb { path: rocksdb_path.clone(), cache_size: 128 },
|
|
DatabaseType::Full,
|
|
true,
|
|
);
|
|
assert!(db_res.is_ok(), "New rocksdb database should be created");
|
|
}
|
|
|
|
// it should reopen existing auto (pairtydb) database
|
|
{
|
|
let db_res = open_database::<Block>(
|
|
&DatabaseSource::Auto { paritydb_path, rocksdb_path, cache_size: 128 },
|
|
DatabaseType::Full,
|
|
true,
|
|
);
|
|
assert!(db_res.is_ok(), "Existing parity database should be reopened");
|
|
}
|
|
}
|
|
}
|