pezkuwichain/pezkuwi-subxt
1
0
Fork 0
mirror of https://github.com/pezkuwichain/pezkuwi-subxt.git synced 2026-06-13 19:51:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

Refactor away from opaque hashes (#5226)

Browse Source 
* System.BlockHash

* Fix hash

* Introduce K/V iteration in all _concat maps

Also move across:
- System.Account (blake2_128_concat)
- Balances.Locks (twox_64_concat)
- ElectionsPhragmen.VotesOf (twox_64_concat)
- ElectionsPhragmen.StakeOf (twox_64_concat)
- Identity.IdentityOf (twox_64_concat)
- Identity.SubsOf (twox_64_concat)
- Society.Payouts (twox_64_concat)
- Session.NextKeys (twox_64_concat)
- Identity.SuperOf (blake2_128_concat)
- Session.KeyOwner (blake2_128_concat)
- Society.SuspendedCandidates (twox_64_concat)
- Society.SuspendedMembers (twox_64_concat)
- Society.Vouching (twox_64_concat)
- Society.Strikes (twox_64_concat)
- System.EventTopics
- Balances.Account

* Build fixes

* Ensure migration happens in correct order

* Staking.*

* Vesting.* Offences.*

* Democracy.*

* Babe.* Collective.*

* Grandpa.*

* Assets.* Benchmark.* Contracts.* Elections.* Asset.* Nicks.*

Also introduce real account list

* ImOnline.*

* Treasury.*

* Recovery.*

* Final bits.

* Docs

* Fix one test

* Fix test

* All passing except the UI tests

* Remove linked_map part 1

* Remove linked_map

* Some iterator utils for double maps.

* Remove old migrations

* Introduce tombstone for LinkedMap type

* Migration for genesis hash

* Fix build

* Fix hash

* Rename Map is_linked -> unused, keeping backwards compat (#5256)

* Update frame/balances/src/lib.rs

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* Update frame/elections/src/lib.rs

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* Remove old migration code.

* Update frame/system/src/lib.rs

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* Update bin/node/runtime/src/lib.rs

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* Fix hash

* fix session migration

* Fix watning

Co-authored-by: Jaco Greeff <jacogr@gmail.com>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
This commit is contained in:
Gavin Wood
2020-03-16 23:19:53 +01:00
committed by GitHub
parent 846a9ce8c6
commit af9083f53b
94 changed files with 1111 additions and 2020 deletions
Download Patch File Download Diff File Expand all files Collapse all files
substrate/frame/support/Cargo.toml
+1 -1
View File
@@ -14,7 +14,7 @@ serde = { version = "1.0.101", optional = true, features = ["derive"] }
codec = { package = "parity-scale-codec", version = "1.2.0", default-features = false, features = ["derive"] }
frame-metadata = { version = "11.0.0-alpha.2", default-features = false, path = "../metadata" }
sp-std = { version = "2.0.0-alpha.2", default-features = false, path = "../../primitives/std" }
sp-io ={ path = "../../primitives/io", default-features = false , version = "2.0.0-alpha.2"}
sp-io = { path = "../../primitives/io", default-features = false , version = "2.0.0-alpha.2"}
sp-runtime = { version = "2.0.0-alpha.2", default-features = false, path = "../../primitives/runtime" }
sp-core = { version = "2.0.0-alpha.2", default-features = false, path = "../../primitives/core" }
sp-arithmetic = { version = "2.0.0-alpha.2", default-features = false, path = "../../primitives/arithmetic" }
substrate/frame/support/procedural/src/lib.rs
+31 -45
View File
@@ -33,8 +33,8 @@ use proc_macro::TokenStream;
/// decl_storage! {
/// trait Store for Module<T: Trait> as Example {
/// Foo get(fn foo) config(): u32=12;
/// Bar: map hasher(blake2_256) u32 => u32;
/// pub Zed build(|config| vec![(0, 0)]): linked_map hasher(blake2_256) u32 => u32;
/// Bar: map hasher(identity) u32 => u32;
/// pub Zed build(|config| vec![(0, 0)]): map hasher(identity) u32 => u32;
/// }
/// }
/// ```
@@ -70,10 +70,28 @@ use proc_macro::TokenStream;
/// And [`StoragePrefixedMap`](../frame_support/storage/trait.StoragePrefixedMap.html).
///
/// `$hash` representing a choice of hashing algorithms available in the
/// [`Hashable`](../frame_support/trait.Hashable.html) trait.
/// [`Hashable`](../frame_support/trait.Hashable.html) trait. You will generally want to use one
/// of three hashers:
/// * `blake2_128_concat`: The default, safe choice. Use if you are unsure or don't care. It is
/// secure against user-tainted keys, fairly fast and memory-efficient and supports
/// iteration over its keys and values. This must be used if the keys of your map can be
/// selected *en masse* by untrusted users.
/// * `twox_64_concat`: This is an insecure hasher and can only be used safely if you know that
/// the preimages cannot be chosen at will by untrusted users. It is memory-efficient, extremely
/// performant and supports iteration over its keys and values. You can safely use this is the
/// key is:
/// - A (slowly) incrementing index.
/// - Known to be the result of a cryptographic hash (though `identity` is a better choice here).
/// - Known to be the public key of a cryptographic key pair in existence.
/// * `identity`: This is not a hasher at all, and just uses the key material directly. Since it
/// does no hashing or appending, it's the fastest possible hasher, however, it's also the least
/// secure. It can be used only if you know that the key will be cryptographically/securely
/// randomly distributed over the binary encoding space. In most cases this will not be true.
/// One case where it is true, however, if where the key is itself the result of a cryptographic
/// hash of some existent data.
///
/// `blake2_256` and `blake2_128_concat` are strong hasher. One should use another hasher
/// with care, see generator documentation.
/// Other hashers will tend to be "opaque" and not support iteration over the keys in the
/// map. It is not recommended to use these.
///
/// The generator is implemented with:
/// * `module_prefix`: $module_prefix
@@ -85,36 +103,6 @@ use proc_macro::TokenStream;
/// twox128(module_prefix) ++ twox128(storage_prefix) ++ hasher(encode(key))
/// ```
///
/// * Linked map: `Foo: linked_map hasher($hash) type => type`: Implements the
/// [`StorageLinkedMap`](../frame_support/storage/trait.StorageLinkedMap.html) trait using the
/// [`StorageLinkedMap generator`](../frame_support/storage/generator/trait.StorageLinkedMap.html).
/// And [`StoragePrefixedMap`](../frame_support/storage/trait.StoragePrefixedMap.html).
///
/// `$hash` representing a choice of hashing algorithms available in the
/// [`Hashable`](../frame_support/trait.Hashable.html) trait.
///
/// `blake2_256` and `blake2_128_concat` are strong hasher. One should use another hasher
/// with care, see generator documentation.
///
/// All key formatting logic can be accessed in a type-agnostic format via the
/// `KeyFormat` trait, which
/// is implemented for the storage linked map type as well.
///
/// The generator key format is implemented with:
/// * `module_prefix`: $module_prefix
/// * `storage_prefix`: storage_name
/// * `head_prefix`: `"HeadOf" ++ storage_name`
/// * `Hasher`: $hash
///
/// Thus the keys are stored at:
/// ```nocompile
/// Twox128(module_prefix) ++ Twox128(storage_prefix) ++ Hasher(encode(key))
/// ```
/// and head is stored at:
/// ```nocompile
/// Twox128(module_prefix) ++ Twox128(head_prefix)
/// ```
///
/// * Double map: `Foo: double_map hasher($hash1) u32, hasher($hash2) u32 => u32`: Implements the
/// [`StorageDoubleMap`](../frame_support/storage/trait.StorageDoubleMap.html) trait using the
/// [`StorageDoubleMap generator`](../frame_support/storage/generator/trait.StorageDoubleMap.html).
@@ -124,14 +112,6 @@ use proc_macro::TokenStream;
/// [`Hashable`](../frame_support/trait.Hashable.html) trait. They must be chosen with care, see
/// generator documentation.
///
/// If the first key is untrusted, a cryptographic `hasher` such as `blake2_256` or
/// `blake2_128_concat` must be used.
/// Otherwise, other values of all storage items can be compromised.
///
/// If the second key is untrusted, a cryptographic `hasher` such as `blake2_256` or
/// `blake2_128_concat` must be used.
/// Otherwise, other items in storage with the same first key can be compromised.
///
/// The generator is implemented with:
/// * `module_prefix`: $module_prefix
/// * `storage_prefix`: storage_name
@@ -145,10 +125,16 @@ use proc_macro::TokenStream;
///
/// Supported hashers (ordered from least to best security):
///
/// * `twox_64_concat` - TwoX with 64bit + key concatenated.
/// * `identity` - Just the unrefined key material. Use only when it is known to be a secure hash
/// already. The most efficient and iterable over keys.
/// * `twox_64_concat` - TwoX with 64bit + key concatenated. Use only when an untrusted source
/// cannot select and insert key values. Very efficient and iterable over keys.
/// * `blake2_128_concat` - Blake2 with 128bit + key concatenated. Slower but safe to use in all
/// circumstances. Iterable over keys.
///
/// Deprecated hashers, which do not support iteration over keys include:
/// * `twox_128` - TwoX with 128bit.
/// * `twox_256` - TwoX with with 256bit.
/// * `blake2_128_concat` - Blake2 with 128bit + key concatenated.
/// * `blake2_128` - Blake2 with 128bit.
/// * `blake2_256` - Blake2 with 256bit.
///
substrate/frame/support/procedural/src/storage/genesis_config/builder_def.rs
+1 -1
View File
@@ -88,7 +88,7 @@ impl BuilderDef {
}}
},
StorageLineTypeDef::Simple(_) => unreachable!(),
StorageLineTypeDef::Map(map) | StorageLineTypeDef::LinkedMap(map) => {
StorageLineTypeDef::Map(map) => {
let key = &map.key;
quote!{{
#data
substrate/frame/support/procedural/src/storage/genesis_config/genesis_config_def.rs
+1 -1
View File
@@ -93,7 +93,7 @@ impl GenesisConfigDef {
let typ = match &line.storage_type {
StorageLineTypeDef::Simple(_) => (*value_type).clone(),
StorageLineTypeDef::Map(map) | StorageLineTypeDef::LinkedMap(map) => {
StorageLineTypeDef::Map(map) => {
let key = &map.key;
parse_quote!( Vec<(#key, #value_type)> )
},
substrate/frame/support/procedural/src/storage/getters.rs
+1 -1
View File
@@ -40,7 +40,7 @@ pub fn impl_getters(scrate: &TokenStream, def: &DeclStorageDefExt) -> TokenStrea
}
}
},
StorageLineTypeDef::Map(map) | StorageLineTypeDef::LinkedMap(map) => {
StorageLineTypeDef::Map(map) => {
let key = &map.key;
let value = &map.value;
quote!{
substrate/frame/support/procedural/src/storage/metadata.rs
+1 -14
View File
@@ -41,20 +41,7 @@ fn storage_line_metadata_type(scrate: &TokenStream, line: &StorageLineDefExt) ->
hasher: #scrate::metadata::#hasher,
key: #scrate::metadata::DecodeDifferent::Encode(#key),
value: #scrate::metadata::DecodeDifferent::Encode(#value_type),
is_linked: false,
}
}
},
StorageLineTypeDef::LinkedMap(map) => {
let hasher = map.hasher.into_metadata();
let key = &map.key;
let key = clean_type_string(&quote!(#key).to_string());
quote!{
#scrate::metadata::StorageEntryType::Map {
hasher: #scrate::metadata::#hasher,
key: #scrate::metadata::DecodeDifferent::Encode(#key),
value: #scrate::metadata::DecodeDifferent::Encode(#value_type),
is_linked: true,
unused: false,
}
}
},
substrate/frame/support/procedural/src/storage/mod.rs
+3 -11
View File
@@ -235,10 +235,6 @@ impl StorageLineDefExt {
ext::type_contains_ident(&map.key, &def.module_runtime_generic)
|| ext::type_contains_ident(&map.value, &def.module_runtime_generic)
}
StorageLineTypeDef::LinkedMap(map) => {
ext::type_contains_ident(&map.key, &def.module_runtime_generic)
|| ext::type_contains_ident(&map.value, &def.module_runtime_generic)
}
StorageLineTypeDef::DoubleMap(map) => {
ext::type_contains_ident(&map.key1, &def.module_runtime_generic)
|| ext::type_contains_ident(&map.key2, &def.module_runtime_generic)
@@ -249,7 +245,6 @@ impl StorageLineDefExt {
let query_type = match &storage_def.storage_type {
StorageLineTypeDef::Simple(value) => value.clone(),
StorageLineTypeDef::Map(map) => map.value.clone(),
StorageLineTypeDef::LinkedMap(map) => map.value.clone(),
StorageLineTypeDef::DoubleMap(map) => map.value.clone(),
};
let is_option = ext::extract_type_option(&query_type).is_some();
@@ -291,10 +286,6 @@ impl StorageLineDefExt {
let key = &map.key;
quote!( StorageMap<#key, #value_type> )
},
StorageLineTypeDef::LinkedMap(map) => {
let key = &map.key;
quote!( StorageLinkedMap<#key, #value_type> )
},
StorageLineTypeDef::DoubleMap(map) => {
let key1 = &map.key1;
let key2 = &map.key2;
@@ -336,7 +327,6 @@ impl StorageLineDefExt {
pub enum StorageLineTypeDef {
Map(MapDef),
LinkedMap(MapDef),
DoubleMap(DoubleMapDef),
Simple(syn::Type),
}
@@ -372,6 +362,7 @@ pub enum HasherKind {
Twox256,
Twox128,
Twox64Concat,
Identity,
}
impl HasherKind {
@@ -383,6 +374,7 @@ impl HasherKind {
HasherKind::Twox256 => quote!( Twox256 ),
HasherKind::Twox128 => quote!( Twox128 ),
HasherKind::Twox64Concat => quote!( Twox64Concat ),
HasherKind::Identity => quote!( Identity ),
}
}
@@ -394,6 +386,7 @@ impl HasherKind {
HasherKind::Twox256 => quote!( StorageHasher::Twox256 ),
HasherKind::Twox128 => quote!( StorageHasher::Twox128 ),
HasherKind::Twox64Concat => quote!( StorageHasher::Twox64Concat ),
HasherKind::Identity => quote!( StorageHasher::Identity ),
}
}
}
@@ -420,7 +413,6 @@ pub fn decl_storage_impl(input: proc_macro::TokenStream) -> proc_macro::TokenStr
use #scrate::{
StorageValue as _,
StorageMap as _,
StorageLinkedMap as _,
StorageDoubleMap as _,
StoragePrefixedMap as _,
};
substrate/frame/support/procedural/src/storage/parse.rs
+27 -33
View File
@@ -27,15 +27,18 @@ mod keyword {
syn::custom_keyword!(build);
syn::custom_keyword!(get);
syn::custom_keyword!(map);
syn::custom_keyword!(linked_map);
syn::custom_keyword!(double_map);
syn::custom_keyword!(blake2_256);
syn::custom_keyword!(blake2_128);
syn::custom_keyword!(opaque_blake2_256);
syn::custom_keyword!(opaque_blake2_128);
syn::custom_keyword!(blake2_128_concat);
syn::custom_keyword!(twox_256);
syn::custom_keyword!(twox_128);
syn::custom_keyword!(opaque_twox_256);
syn::custom_keyword!(opaque_twox_128);
syn::custom_keyword!(twox_64_concat);
syn::custom_keyword!(identity);
syn::custom_keyword!(hasher);
syn::custom_keyword!(tainted);
syn::custom_keyword!(natural);
syn::custom_keyword!(prehashed);
}
/// Specific `Opt` to implement structure with optional parsing
@@ -194,7 +197,6 @@ impl_parse_for_opt!(DeclStorageBuild => keyword::build);
#[derive(ToTokens, Debug)]
enum DeclStorageType {
Map(DeclStorageMap),
LinkedMap(DeclStorageLinkedMap),
DoubleMap(DeclStorageDoubleMap),
Simple(syn::Type),
}
@@ -203,8 +205,6 @@ impl syn::parse::Parse for DeclStorageType {
fn parse(input: syn::parse::ParseStream) -> syn::parse::Result<Self> {
if input.peek(keyword::map) {
Ok(Self::Map(input.parse()?))
} else if input.peek(keyword::linked_map) {
Ok(Self::LinkedMap(input.parse()?))
} else if input.peek(keyword::double_map) {
Ok(Self::DoubleMap(input.parse()?))
} else {
@@ -222,15 +222,6 @@ struct DeclStorageMap {
pub value: syn::Type,
}
#[derive(Parse, ToTokens, Debug)]
struct DeclStorageLinkedMap {
pub map_keyword: keyword::linked_map,
pub hasher: Opt<SetHasher>,
pub key: syn::Type,
pub ass_keyword: Token![=>],
pub value: syn::Type,
}
#[derive(Parse, ToTokens, Debug)]
struct DeclStorageDoubleMap {
pub map_keyword: keyword::double_map,
@@ -245,29 +236,38 @@ struct DeclStorageDoubleMap {
#[derive(ToTokens, Debug)]
enum Hasher {
Blake2_256(keyword::blake2_256),
Blake2_128(keyword::blake2_128),
Blake2_256(keyword::opaque_blake2_256),
Blake2_128(keyword::opaque_blake2_128),
Blake2_128Concat(keyword::blake2_128_concat),
Twox256(keyword::twox_256),
Twox128(keyword::twox_128),
Twox256(keyword::opaque_twox_256),
Twox128(keyword::opaque_twox_128),
Twox64Concat(keyword::twox_64_concat),
Identity(keyword::identity),
}
impl syn::parse::Parse for Hasher {
fn parse(input: syn::parse::ParseStream) -> syn::parse::Result<Self> {
let lookahead = input.lookahead1();
if lookahead.peek(keyword::blake2_256) {
if lookahead.peek(keyword::opaque_blake2_256) {
Ok(Self::Blake2_256(input.parse()?))
} else if lookahead.peek(keyword::blake2_128) {
} else if lookahead.peek(keyword::opaque_blake2_128) {
Ok(Self::Blake2_128(input.parse()?))
} else if lookahead.peek(keyword::blake2_128_concat) {
Ok(Self::Blake2_128Concat(input.parse()?))
} else if lookahead.peek(keyword::twox_256) {
} else if lookahead.peek(keyword::opaque_twox_256) {
Ok(Self::Twox256(input.parse()?))
} else if lookahead.peek(keyword::twox_128) {
} else if lookahead.peek(keyword::opaque_twox_128) {
Ok(Self::Twox128(input.parse()?))
} else if lookahead.peek(keyword::twox_64_concat) {
Ok(Self::Twox64Concat(input.parse()?))
} else if lookahead.peek(keyword::identity) {
Ok(Self::Identity(input.parse()?))
} else if lookahead.peek(keyword::tainted) {
Ok(Self::Blake2_128Concat(input.parse()?))
} else if lookahead.peek(keyword::natural) {
Ok(Self::Twox64Concat(input.parse()?))
} else if lookahead.peek(keyword::prehashed) {
Ok(Self::Identity(input.parse()?))
} else {
Err(lookahead.error())
}
@@ -313,6 +313,7 @@ impl From<Hasher> for super::HasherKind {
Hasher::Twox256(_) => super::HasherKind::Twox256,
Hasher::Twox128(_) => super::HasherKind::Twox128,
Hasher::Twox64Concat(_) => super::HasherKind::Twox64Concat,
Hasher::Identity(_) => super::HasherKind::Identity,
}
}
}
@@ -464,7 +465,7 @@ fn parse_storage_line_defs(
let span = line.storage_type.span();
let no_hasher_error = || syn::Error::new(
span,
"Default hasher has been removed, use explicit hasher(blake2_256) instead."
"Default hasher has been removed, use explicit hasher(blake2_128_concat) instead."
);
let storage_type = match line.storage_type {
@@ -475,13 +476,6 @@ fn parse_storage_line_defs(
value: map.value,
}
),
DeclStorageType::LinkedMap(map) => super::StorageLineTypeDef::LinkedMap(
super::MapDef {
hasher: map.hasher.inner.ok_or_else(no_hasher_error)?.into(),
key: map.key,
value: map.value,
}
),
DeclStorageType::DoubleMap(map) => super::StorageLineTypeDef::DoubleMap(
super::DoubleMapDef {
hasher1: map.hasher1.inner.ok_or_else(no_hasher_error)?.into(),
substrate/frame/support/procedural/src/storage/storage_struct.rs
-41
View File
@@ -158,47 +158,6 @@ pub fn decl_and_impl(scrate: &TokenStream, def: &DeclStorageDefExt) -> TokenStre
}
)
},
StorageLineTypeDef::LinkedMap(map) => {
let hasher = map.hasher.to_storage_hasher_struct();
let head_prefix_str = syn::LitStr::new(
&format!("HeadOf{}", line.name.to_string()),
line.name.span(),
);
quote!(
impl<#impl_trait> #scrate::#storage_generator_trait for #storage_struct
#optional_storage_where_clause
{
type Query = #query_type;
type KeyFormat = Self;
fn from_optional_value_to_query(v: Option<#value_type>) -> Self::Query {
#from_optional_value_to_query
}
fn from_query_to_optional_value(v: Self::Query) -> Option<#value_type> {
#from_query_to_optional_value
}
}
impl<#impl_trait> #scrate::storage::generator::LinkedMapKeyFormat for #storage_struct {
type Hasher = #scrate::#hasher;
fn module_prefix() -> &'static [u8] {
#instance_or_inherent::PREFIX.as_bytes()
}
fn storage_prefix() -> &'static [u8] {
#storage_name_str.as_bytes()
}
fn head_prefix() -> &'static [u8] {
#head_prefix_str.as_bytes()
}
}
)
},
StorageLineTypeDef::DoubleMap(map) => {
let hasher1 = map.hasher1.to_storage_hasher_struct();
let hasher2 = map.hasher2.to_storage_hasher_struct();
substrate/frame/support/src/hash.rs
+31
View File
@@ -28,6 +28,7 @@ pub trait Hashable: Sized {
fn twox_128(&self) -> [u8; 16];
fn twox_256(&self) -> [u8; 32];
fn twox_64_concat(&self) -> Vec<u8>;
fn identity(&self) -> Vec<u8>;
}
impl<T: Codec> Hashable for T {
@@ -49,6 +50,7 @@ impl<T: Codec> Hashable for T {
fn twox_64_concat(&self) -> Vec<u8> {
self.using_encoded(Twox64Concat::hash)
}
fn identity(&self) -> Vec<u8> { self.encode() }
}
/// Hasher to use to hash keys to insert to storage.
@@ -57,6 +59,25 @@ pub trait StorageHasher: 'static {
fn hash(x: &[u8]) -> Self::Output;
}
/// Hasher to use to hash keys to insert to storage.
pub trait ReversibleStorageHasher: StorageHasher {
fn reverse(x: &[u8]) -> &[u8];
}
/// Store the key directly.
pub struct Identity;
impl StorageHasher for Identity {
type Output = Vec<u8>;
fn hash(x: &[u8]) -> Vec<u8> {
x.to_vec()
}
}
impl ReversibleStorageHasher for Identity {
fn reverse(x: &[u8]) -> &[u8] {
x
}
}
/// Hash storage keys with `concat(twox64(key), key)`
pub struct Twox64Concat;
impl StorageHasher for Twox64Concat {
@@ -69,6 +90,11 @@ impl StorageHasher for Twox64Concat {
.collect::<Vec<_>>()
}
}
impl ReversibleStorageHasher for Twox64Concat {
fn reverse(x: &[u8]) -> &[u8] {
&x[8..]
}
}
/// Hash storage keys with `concat(blake2_128(key), key)`
pub struct Blake2_128Concat;
@@ -82,6 +108,11 @@ impl StorageHasher for Blake2_128Concat {
.collect::<Vec<_>>()
}
}
impl ReversibleStorageHasher for Blake2_128Concat {
fn reverse(x: &[u8]) -> &[u8] {
&x[16..]
}
}
/// Hash storage keys with blake2 128
pub struct Blake2_128;
substrate/frame/support/src/lib.rs
+53 -53
View File
@@ -67,11 +67,12 @@ pub mod traits;
pub mod weights;
pub use self::hash::{
Twox256, Twox128, Blake2_256, Blake2_128, Twox64Concat, Blake2_128Concat, Hashable,
Twox256, Twox128, Blake2_256, Blake2_128, Identity, Twox64Concat, Blake2_128Concat, Hashable,
StorageHasher
};
pub use self::storage::{
StorageValue, StorageMap, StorageLinkedMap, StorageDoubleMap, StoragePrefixedMap
StorageValue, StorageMap, StorageDoubleMap, StoragePrefixedMap, IterableStorageMap,
IterableStorageDoubleMap,
};
pub use self::dispatch::{Parameter, Callable, IsSubType};
pub use sp_runtime::{self, ConsensusEngineId, print, traits::Printable};
@@ -253,24 +254,24 @@ mod tests {
decl_storage! {
trait Store for Module<T: Trait> as Test {
pub Data get(fn data) build(|_| vec![(15u32, 42u64)]):
linked_map hasher(twox_64_concat) u32 => u64;
pub OptionLinkedMap: linked_map hasher(blake2_256) u32 => Option<u32>;
map hasher(twox_64_concat) u32 => u64;
pub OptionLinkedMap: map hasher(blake2_128_concat) u32 => Option<u32>;
pub GenericData get(fn generic_data):
linked_map hasher(twox_128) T::BlockNumber => T::BlockNumber;
map hasher(identity) T::BlockNumber => T::BlockNumber;
pub GenericData2 get(fn generic_data2):
linked_map hasher(blake2_256) T::BlockNumber => Option<T::BlockNumber>;
map hasher(blake2_128_concat) T::BlockNumber => Option<T::BlockNumber>;
pub GetterNoFnKeyword get(no_fn): Option<u32>;
pub DataDM config(test_config) build(|_| vec![(15u32, 16u32, 42u64)]):
double_map hasher(twox_64_concat) u32, hasher(blake2_256) u32 => u64;
double_map hasher(twox_64_concat) u32, hasher(blake2_128_concat) u32 => u64;
pub GenericDataDM:
double_map hasher(blake2_256) T::BlockNumber, hasher(twox_128) T::BlockNumber
double_map hasher(blake2_128_concat) T::BlockNumber, hasher(identity) T::BlockNumber
=> T::BlockNumber;
pub GenericData2DM:
double_map hasher(blake2_256) T::BlockNumber, hasher(twox_256) T::BlockNumber
double_map hasher(blake2_128_concat) T::BlockNumber, hasher(twox_64_concat) T::BlockNumber
=> Option<T::BlockNumber>;
pub AppendableDM:
double_map hasher(blake2_256) u32, hasher(blake2_256) T::BlockNumber => Vec<u32>;
double_map hasher(blake2_128_concat) u32, hasher(blake2_128_concat) T::BlockNumber => Vec<u32>;
}
}
@@ -286,8 +287,16 @@ mod tests {
type Map = Data;
trait Sorted { fn sorted(self) -> Self; }
impl<T: Ord> Sorted for Vec<T> {
fn sorted(mut self) -> Self {
self.sort();
self
}
}
#[test]
fn linked_map_issue_3318() {
fn map_issue_3318() {
new_test_ext().execute_with(|| {
OptionLinkedMap::insert(1, 1);
assert_eq!(OptionLinkedMap::get(1), Some(1));
@@ -297,31 +306,31 @@ mod tests {
}
#[test]
fn linked_map_swap_works() {
fn map_swap_works() {
new_test_ext().execute_with(|| {
OptionLinkedMap::insert(0, 0);
OptionLinkedMap::insert(1, 1);
OptionLinkedMap::insert(2, 2);
OptionLinkedMap::insert(3, 3);
let collect = || OptionLinkedMap::enumerate().collect::<Vec<_>>();
assert_eq!(collect(), vec![(3, 3), (2, 2), (1, 1), (0, 0)]);
let collect = || OptionLinkedMap::iter().collect::<Vec<_>>().sorted();
assert_eq!(collect(), vec![(0, 0), (1, 1), (2, 2), (3, 3)]);
// Two existing
OptionLinkedMap::swap(1, 2);
assert_eq!(collect(), vec![(3, 3), (2, 1), (1, 2), (0, 0)]);
assert_eq!(collect(), vec![(0, 0), (1, 2), (2, 1), (3, 3)]);
// Back to normal
OptionLinkedMap::swap(2, 1);
assert_eq!(collect(), vec![(3, 3), (2, 2), (1, 1), (0, 0)]);
assert_eq!(collect(), vec![(0, 0), (1, 1), (2, 2), (3, 3)]);
// Left existing
OptionLinkedMap::swap(2, 5);
assert_eq!(collect(), vec![(5, 2), (3, 3), (1, 1), (0, 0)]);
assert_eq!(collect(), vec![(0, 0), (1, 1), (3, 3), (5, 2)]);
// Right existing
OptionLinkedMap::swap(5, 2);
assert_eq!(collect(), vec![(2, 2), (3, 3), (1, 1), (0, 0)]);
assert_eq!(collect(), vec![(0, 0), (1, 1), (2, 2), (3, 3)]);
});
}
@@ -356,7 +365,7 @@ mod tests {
}
#[test]
fn linked_map_basic_insert_remove_should_work() {
fn map_basic_insert_remove_should_work() {
new_test_ext().execute_with(|| {
// initialized during genesis
assert_eq!(Map::get(&15u32), 42u64);
@@ -382,54 +391,45 @@ mod tests {
}
#[test]
fn linked_map_enumeration_and_head_should_work() {
fn map_iteration_should_work() {
new_test_ext().execute_with(|| {
assert_eq!(Map::head(), Some(15));
assert_eq!(Map::enumerate().collect::<Vec<_>>(), vec![(15, 42)]);
assert_eq!(Map::iter().collect::<Vec<_>>().sorted(), vec![(15, 42)]);
// insert / remove
let key = 17u32;
Map::insert(key, 4u64);
assert_eq!(Map::head(), Some(key));
assert_eq!(Map::enumerate().collect::<Vec<_>>(), vec![(key, 4), (15, 42)]);
assert_eq!(Map::iter().collect::<Vec<_>>().sorted(), vec![(15, 42), (key, 4)]);
assert_eq!(Map::take(&15), 42u64);
assert_eq!(Map::take(&key), 4u64);
assert_eq!(Map::head(), None);
assert_eq!(Map::enumerate().collect::<Vec<_>>(), vec![]);
assert_eq!(Map::iter().collect::<Vec<_>>().sorted(), vec![]);
// Add couple of more elements
Map::insert(key, 42u64);
assert_eq!(Map::head(), Some(key));
assert_eq!(Map::enumerate().collect::<Vec<_>>(), vec![(key, 42)]);
assert_eq!(Map::iter().collect::<Vec<_>>().sorted(), vec![(key, 42)]);
Map::insert(key + 1, 43u64);
assert_eq!(Map::head(), Some(key + 1));
assert_eq!(Map::enumerate().collect::<Vec<_>>(), vec![(key + 1, 43), (key, 42)]);
assert_eq!(Map::iter().collect::<Vec<_>>().sorted(), vec![(key, 42), (key + 1, 43)]);
// mutate
let key = key + 2;
Map::mutate(&key, |val| {
*val = 15;
});
assert_eq!(Map::enumerate().collect::<Vec<_>>(), vec![(key, 15), (key - 1, 43), (key - 2, 42)]);
assert_eq!(Map::head(), Some(key));
assert_eq!(Map::iter().collect::<Vec<_>>().sorted(), vec![(key - 2, 42), (key - 1, 43), (key, 15)]);
Map::mutate(&key, |val| {
*val = 17;
});
assert_eq!(Map::enumerate().collect::<Vec<_>>(), vec![(key, 17), (key - 1, 43), (key - 2, 42)]);
assert_eq!(Map::iter().collect::<Vec<_>>().sorted(), vec![(key - 2, 42), (key - 1, 43), (key, 17)]);
// remove first
Map::remove(&key);
assert_eq!(Map::head(), Some(key - 1));
assert_eq!(Map::enumerate().collect::<Vec<_>>(), vec![(key - 1, 43), (key - 2, 42)]);
assert_eq!(Map::iter().collect::<Vec<_>>().sorted(), vec![(key - 2, 42), (key - 1, 43)]);
// remove last from the list
Map::remove(&(key - 2));
assert_eq!(Map::head(), Some(key - 1));
assert_eq!(Map::enumerate().collect::<Vec<_>>(), vec![(key - 1, 43)]);
assert_eq!(Map::iter().collect::<Vec<_>>().sorted(), vec![(key - 1, 43)]);
// remove the last element
Map::remove(&(key - 1));
assert_eq!(Map::head(), None);
assert_eq!(Map::enumerate().collect::<Vec<_>>(), vec![]);
assert_eq!(Map::iter().collect::<Vec<_>>().sorted(), vec![]);
});
}
@@ -498,7 +498,7 @@ mod tests {
hasher: StorageHasher::Twox64Concat,
key: DecodeDifferent::Encode("u32"),
value: DecodeDifferent::Encode("u64"),
is_linked: true,
unused: false,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructData(PhantomData::<Test>))
@@ -509,10 +509,10 @@ mod tests {
name: DecodeDifferent::Encode("OptionLinkedMap"),
modifier: StorageEntryModifier::Optional,
ty: StorageEntryType::Map {
hasher: StorageHasher::Blake2_256,
hasher: StorageHasher::Blake2_128Concat,
key: DecodeDifferent::Encode("u32"),
value: DecodeDifferent::Encode("u32"),
is_linked: true,
unused: false,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructOptionLinkedMap(PhantomData::<Test>))
@@ -523,10 +523,10 @@ mod tests {
name: DecodeDifferent::Encode("GenericData"),
modifier: StorageEntryModifier::Default,
ty: StorageEntryType::Map{
hasher: StorageHasher::Twox128,
hasher: StorageHasher::Identity,
key: DecodeDifferent::Encode("T::BlockNumber"),
value: DecodeDifferent::Encode("T::BlockNumber"),
is_linked: true
unused: false
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructGenericData(PhantomData::<Test>))
@@ -537,10 +537,10 @@ mod tests {
name: DecodeDifferent::Encode("GenericData2"),
modifier: StorageEntryModifier::Optional,
ty: StorageEntryType::Map{
hasher: StorageHasher::Blake2_256,
hasher: StorageHasher::Blake2_128Concat,
key: DecodeDifferent::Encode("T::BlockNumber"),
value: DecodeDifferent::Encode("T::BlockNumber"),
is_linked: true
unused: false
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructGenericData2(PhantomData::<Test>))
@@ -564,7 +564,7 @@ mod tests {
key1: DecodeDifferent::Encode("u32"),
key2: DecodeDifferent::Encode("u32"),
value: DecodeDifferent::Encode("u64"),
key2_hasher: StorageHasher::Blake2_256,
key2_hasher: StorageHasher::Blake2_128Concat,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructDataDM(PhantomData::<Test>))
@@ -575,11 +575,11 @@ mod tests {
name: DecodeDifferent::Encode("GenericDataDM"),
modifier: StorageEntryModifier::Default,
ty: StorageEntryType::DoubleMap{
hasher: StorageHasher::Blake2_256,
hasher: StorageHasher::Blake2_128Concat,
key1: DecodeDifferent::Encode("T::BlockNumber"),
key2: DecodeDifferent::Encode("T::BlockNumber"),
value: DecodeDifferent::Encode("T::BlockNumber"),
key2_hasher: StorageHasher::Twox128,
key2_hasher: StorageHasher::Identity,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructGenericDataDM(PhantomData::<Test>))
@@ -590,11 +590,11 @@ mod tests {
name: DecodeDifferent::Encode("GenericData2DM"),
modifier: StorageEntryModifier::Optional,
ty: StorageEntryType::DoubleMap{
hasher: StorageHasher::Blake2_256,
hasher: StorageHasher::Blake2_128Concat,
key1: DecodeDifferent::Encode("T::BlockNumber"),
key2: DecodeDifferent::Encode("T::BlockNumber"),
value: DecodeDifferent::Encode("T::BlockNumber"),
key2_hasher: StorageHasher::Twox256,
key2_hasher: StorageHasher::Twox64Concat,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructGenericData2DM(PhantomData::<Test>))
@@ -605,11 +605,11 @@ mod tests {
name: DecodeDifferent::Encode("AppendableDM"),
modifier: StorageEntryModifier::Default,
ty: StorageEntryType::DoubleMap{
hasher: StorageHasher::Blake2_256,
hasher: StorageHasher::Blake2_128Concat,
key1: DecodeDifferent::Encode("u32"),
key2: DecodeDifferent::Encode("T::BlockNumber"),
value: DecodeDifferent::Encode("Vec<u32>"),
key2_hasher: StorageHasher::Blake2_256,
key2_hasher: StorageHasher::Blake2_128Concat,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructGenericData2DM(PhantomData::<Test>))
substrate/frame/support/src/storage/generator/double_map.rs
+189 -36
View File
@@ -16,8 +16,9 @@
use sp_std::prelude::*;
use sp_std::borrow::Borrow;
use codec::{Ref, FullCodec, FullEncode, Encode, EncodeLike, EncodeAppend};
use crate::{storage::{self, unhashed}, hash::{StorageHasher, Twox128}, traits::Len};
use codec::{Ref, FullCodec, FullEncode, Decode, Encode, EncodeLike, EncodeAppend};
use crate::{storage::{self, unhashed}, traits::Len};
use crate::hash::{StorageHasher, Twox128, ReversibleStorageHasher};
/// Generator for `StorageDoubleMap` used by `decl_storage`.
///
@@ -55,6 +56,22 @@ pub trait StorageDoubleMap<K1: FullEncode, K2: FullEncode, V: FullCodec> {
/// Storage prefix. Used for generating final key.
fn storage_prefix() -> &'static [u8];
/// The full prefix; just the hash of `module_prefix` concatenated to the hash of
/// `storage_prefix`.
fn prefix_hash() -> Vec<u8> {
let module_prefix_hashed = Twox128::hash(Self::module_prefix());
let storage_prefix_hashed = Twox128::hash(Self::storage_prefix());
let mut result = Vec::with_capacity(
module_prefix_hashed.len() + storage_prefix_hashed.len()
);
result.extend_from_slice(&module_prefix_hashed[..]);
result.extend_from_slice(&storage_prefix_hashed[..]);
result
}
/// Convert an optional value retrieved from storage to the type queried.
fn from_optional_value_to_query(v: Option<V>) -> Self::Query;
@@ -62,8 +79,7 @@ pub trait StorageDoubleMap<K1: FullEncode, K2: FullEncode, V: FullCodec> {
fn from_query_to_optional_value(v: Self::Query) -> Option<V>;
/// Generate the first part of the key used in top storage.
fn storage_double_map_final_key1<KArg1>(k1: KArg1) -> Vec<u8>
where
fn storage_double_map_final_key1<KArg1>(k1: KArg1) -> Vec<u8> where
KArg1: EncodeLike<K1>,
{
let module_prefix_hashed = Twox128::hash(Self::module_prefix());
@@ -82,19 +98,32 @@ pub trait StorageDoubleMap<K1: FullEncode, K2: FullEncode, V: FullCodec> {
}
/// Generate the full key used in top storage.
fn storage_double_map_final_key<KArg1, KArg2>(k1: KArg1, k2: KArg2) -> Vec<u8>
where
fn storage_double_map_final_key<KArg1, KArg2>(k1: KArg1, k2: KArg2) -> Vec<u8> where
KArg1: EncodeLike<K1>,
KArg2: EncodeLike<K2>,
{
let mut final_key = Self::storage_double_map_final_key1(k1);
final_key.extend_from_slice(k2.using_encoded(Self::Hasher2::hash).as_ref());
let module_prefix_hashed = Twox128::hash(Self::module_prefix());
let storage_prefix_hashed = Twox128::hash(Self::storage_prefix());
let key1_hashed = k1.borrow().using_encoded(Self::Hasher1::hash);
let key2_hashed = k2.borrow().using_encoded(Self::Hasher2::hash);
let mut final_key = Vec::with_capacity(
module_prefix_hashed.len()
+ storage_prefix_hashed.len()
+ key1_hashed.as_ref().len()
+ key2_hashed.as_ref().len()
);
final_key.extend_from_slice(&module_prefix_hashed[..]);
final_key.extend_from_slice(&storage_prefix_hashed[..]);
final_key.extend_from_slice(key1_hashed.as_ref());
final_key.extend_from_slice(key2_hashed.as_ref());
final_key
}
}
impl<K1, K2, V, G> storage::StorageDoubleMap<K1, K2, V> for G
where
impl<K1, K2, V, G> storage::StorageDoubleMap<K1, K2, V> for G where
K1: FullEncode,
K2: FullEncode,
V: FullCodec,
@@ -102,32 +131,28 @@ where
{
type Query = G::Query;
fn hashed_key_for<KArg1, KArg2>(k1: KArg1, k2: KArg2) -> Vec<u8>
where
fn hashed_key_for<KArg1, KArg2>(k1: KArg1, k2: KArg2) -> Vec<u8> where
KArg1: EncodeLike<K1>,
KArg2: EncodeLike<K2>,
{
Self::storage_double_map_final_key(k1, k2)
}
fn contains_key<KArg1, KArg2>(k1: KArg1, k2: KArg2) -> bool
where
fn contains_key<KArg1, KArg2>(k1: KArg1, k2: KArg2) -> bool where
KArg1: EncodeLike<K1>,
KArg2: EncodeLike<K2>,
{
unhashed::exists(&Self::storage_double_map_final_key(k1, k2))
}
fn get<KArg1, KArg2>(k1: KArg1, k2: KArg2) -> Self::Query
where
fn get<KArg1, KArg2>(k1: KArg1, k2: KArg2) -> Self::Query where
KArg1: EncodeLike<K1>,
KArg2: EncodeLike<K2>,
{
G::from_optional_value_to_query(unhashed::get(&Self::storage_double_map_final_key(k1, k2)))
}
fn take<KArg1, KArg2>(k1: KArg1, k2: KArg2) -> Self::Query
where
fn take<KArg1, KArg2>(k1: KArg1, k2: KArg2) -> Self::Query where
KArg1: EncodeLike<K1>,
KArg2: EncodeLike<K2>,
{
@@ -137,8 +162,12 @@ where
G::from_optional_value_to_query(value)
}
fn swap<XKArg1, XKArg2, YKArg1, YKArg2>(x_k1: XKArg1, x_k2: XKArg2, y_k1: YKArg1, y_k2: YKArg2)
where
fn swap<XKArg1, XKArg2, YKArg1, YKArg2>(
x_k1: XKArg1,
x_k2: XKArg2,
y_k1: YKArg1,
y_k2: YKArg2
) where
XKArg1: EncodeLike<K1>,
XKArg2: EncodeLike<K2>,
YKArg1: EncodeLike<K1>,
@@ -160,8 +189,7 @@ where
}
}
fn insert<KArg1, KArg2, VArg>(k1: KArg1, k2: KArg2, val: VArg)
where
fn insert<KArg1, KArg2, VArg>(k1: KArg1, k2: KArg2, val: VArg) where
KArg1: EncodeLike<K1>,
KArg2: EncodeLike<K2>,
VArg: EncodeLike<V>,
@@ -169,8 +197,7 @@ where
unhashed::put(&Self::storage_double_map_final_key(k1, k2), &val.borrow())
}
fn remove<KArg1, KArg2>(k1: KArg1, k2: KArg2)
where
fn remove<KArg1, KArg2>(k1: KArg1, k2: KArg2) where
KArg1: EncodeLike<K1>,
KArg2: EncodeLike<K2>,
{
@@ -181,8 +208,8 @@ where
unhashed::kill_prefix(Self::storage_double_map_final_key1(k1).as_ref())
}
fn iter_prefix<KArg1>(k1: KArg1) -> storage::PrefixIterator<V>
where KArg1: ?Sized + EncodeLike<K1>
fn iter_prefix<KArg1>(k1: KArg1) -> storage::PrefixIterator<V> where
KArg1: ?Sized + EncodeLike<K1>
{
let prefix = Self::storage_double_map_final_key1(k1);
storage::PrefixIterator::<V> {
@@ -192,8 +219,7 @@ where
}
}
fn mutate<KArg1, KArg2, R, F>(k1: KArg1, k2: KArg2, f: F) -> R
where
fn mutate<KArg1, KArg2, R, F>(k1: KArg1, k2: KArg2, f: F) -> R where
KArg1: EncodeLike<K1>,
KArg2: EncodeLike<K2>,
F: FnOnce(&mut Self::Query) -> R,
@@ -213,8 +239,7 @@ where
k1: KArg1,
k2: KArg2,
items: Items,
) -> Result<(), &'static str>
where
) -> Result<(), &'static str> where
KArg1: EncodeLike<K1>,
KArg2: EncodeLike<K2>,
Item: Encode,
@@ -246,8 +271,7 @@ where
k1: KArg1,
k2: KArg2,
items: Items,
)
where
) where
KArg1: EncodeLike<K1>,
KArg2: EncodeLike<K2>,
Item: Encode,
@@ -260,10 +284,10 @@ where
.unwrap_or_else(|_| Self::insert(k1, k2, items));
}
fn decode_len<KArg1, KArg2>(key1: KArg1, key2: KArg2) -> Result<usize, &'static str>
where KArg1: EncodeLike<K1>,
KArg2: EncodeLike<K2>,
V: codec::DecodeLength + Len,
fn decode_len<KArg1, KArg2>(key1: KArg1, key2: KArg2) -> Result<usize, &'static str> where
KArg1: EncodeLike<K1>,
KArg2: EncodeLike<K2>,
V: codec::DecodeLength + Len,
{
let final_key = Self::storage_double_map_final_key(key1, key2);
if let Some(v) = unhashed::get_raw(&final_key) {
@@ -276,6 +300,135 @@ where
Ok(len)
}
}
fn migrate_keys<
OldHasher1: StorageHasher,
OldHasher2: StorageHasher,
KeyArg1: EncodeLike<K1>,
KeyArg2: EncodeLike<K2>,
>(key1: KeyArg1, key2: KeyArg2) -> Option<V> {
let old_key = {
let module_prefix_hashed = Twox128::hash(Self::module_prefix());
let storage_prefix_hashed = Twox128::hash(Self::storage_prefix());
let key1_hashed = key1.borrow().using_encoded(OldHasher1::hash);
let key2_hashed = key2.borrow().using_encoded(OldHasher2::hash);
let mut final_key = Vec::with_capacity(
module_prefix_hashed.len()
+ storage_prefix_hashed.len()
+ key1_hashed.as_ref().len()
+ key2_hashed.as_ref().len()
);
final_key.extend_from_slice(&module_prefix_hashed[..]);
final_key.extend_from_slice(&storage_prefix_hashed[..]);
final_key.extend_from_slice(key1_hashed.as_ref());
final_key.extend_from_slice(key2_hashed.as_ref());
final_key
};
unhashed::take(old_key.as_ref()).map(|value| {
unhashed::put(Self::storage_double_map_final_key(key1, key2).as_ref(), &value);
value
})
}
}
/// Utility to iterate through items in a storage map.
pub struct MapIterator<K, V, Hasher> {
prefix: Vec<u8>,
previous_key: Vec<u8>,
drain: bool,
_phantom: ::sp_std::marker::PhantomData<(K, V, Hasher)>,
}
impl<
K: Decode + Sized,
V: Decode + Sized,
Hasher: ReversibleStorageHasher
> Iterator for MapIterator<K, V, Hasher> {
type Item = (K, V);
fn next(&mut self) -> Option<(K, V)> {
loop {
let maybe_next = sp_io::storage::next_key(&self.previous_key)
.filter(|n| n.starts_with(&self.prefix));
break match maybe_next {
Some(next) => {
self.previous_key = next;
match unhashed::get::<V>(&self.previous_key) {
Some(value) => {
if self.drain {
unhashed::kill(&self.previous_key)
}
let mut key_material = Hasher::reverse(&self.previous_key[self.prefix.len()..]);
match K::decode(&mut key_material) {
Ok(key) => Some((key, value)),
Err(_) => continue,
}
}
None => continue,
}
}
None => None,
}
}
}
}
impl<
K1: FullCodec,
K2: FullCodec,
V: FullCodec,
G: StorageDoubleMap<K1, K2, V>,
> storage::IterableStorageDoubleMap<K1, K2, V> for G where
G::Hasher1: ReversibleStorageHasher,
G::Hasher2: ReversibleStorageHasher
{
type Iterator = MapIterator<K2, V, G::Hasher2>;
/// Enumerate all elements in the map.
fn iter(k1: impl EncodeLike<K1>) -> Self::Iterator {
let prefix = G::storage_double_map_final_key1(k1);
Self::Iterator {
prefix: prefix.clone(),
previous_key: prefix,
drain: false,
_phantom: Default::default(),
}
}
/// Enumerate all elements in the map.
fn drain(k1: impl EncodeLike<K1>) -> Self::Iterator {
let prefix = G::storage_double_map_final_key1(k1);
Self::Iterator {
prefix: prefix.clone(),
previous_key: prefix,
drain: true,
_phantom: Default::default(),
}
}
fn translate<O: Decode, F: Fn(O) -> Option<V>>(f: F) {
let prefix = G::prefix_hash();
let mut previous_key = prefix.clone();
loop {
match sp_io::storage::next_key(&previous_key).filter(|n| n.starts_with(&prefix)) {
Some(next) => {
previous_key = next;
let maybe_value = unhashed::get::<O>(&previous_key);
match maybe_value {
Some(value) => match f(value) {
Some(new) => unhashed::put::<V>(&previous_key, &new),
None => unhashed::kill(&previous_key),
},
None => continue,
}
}
None => return,
}
}
}
}
#[cfg(test)]
substrate/frame/support/src/storage/generator/linked_map.rs
-499
View File
@@ -1,499 +0,0 @@
// Copyright 2019-2020 Parity Technologies (UK) Ltd.
// This file is part of Substrate.
// Substrate is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Substrate is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Substrate. If not, see <http://www.gnu.org/licenses/>.
use codec::{FullCodec, Encode, Decode, EncodeLike, Ref};
use crate::{storage::{self, unhashed}, hash::{StorageHasher, Twox128}, traits::Len};
use sp_std::{prelude::*, marker::PhantomData};
/// Generator for `StorageLinkedMap` used by `decl_storage`.
///
/// By default final key generation rely on `KeyFormat`.
pub trait StorageLinkedMap<K: FullCodec, V: FullCodec> {
/// The type that get/take returns.
type Query;
/// The family of key formats used for this map.
type KeyFormat: KeyFormat;
/// Convert an optional value retrieved from storage to the type queried.
fn from_optional_value_to_query(v: Option<V>) -> Self::Query;
/// Convert a query to an optional value into storage.
fn from_query_to_optional_value(v: Self::Query) -> Option<V>;
/// Generate the full key used in top storage.
fn storage_linked_map_final_key<KeyArg>(key: KeyArg) -> Vec<u8>
where
KeyArg: EncodeLike<K>,
{
<Self::KeyFormat as KeyFormat>::storage_linked_map_final_key::<KeyArg>(&key)
}
/// Generate the hashed key for head
fn storage_linked_map_final_head_key() -> Vec<u8> {
<Self::KeyFormat as KeyFormat>::storage_linked_map_final_head_key()
}
}
/// A type-abstracted key format used for a family of linked-map types.
///
/// # Default mapping of keys to a storage path
///
/// The key for the head of the map is stored at one fixed path:
/// ```nocompile
/// Twox128(module_prefix) ++ Twox128(head_prefix)
/// ```
///
/// For each key, the value stored under that key is appended with a
/// [`Linkage`](struct.Linkage.html) (which hold previous and next key) at the path:
/// ```nocompile
/// Twox128(module_prefix) ++ Twox128(storage_prefix) ++ Hasher(encode(key))
/// ```
///
/// Enumeration is done by getting the head of the linked map and then iterating getting the
/// value and linkage stored at the key until the found linkage has no next key.
///
/// # Warning
///
/// If the keys are not trusted (e.g. can be set by a user), a cryptographic `hasher` such as
/// `blake2_256` must be used. Otherwise, other values in storage can be compromised.
pub trait KeyFormat {
/// Hasher. Used for generating final key and final head key.
type Hasher: StorageHasher;
/// Module prefix. Used for generating final key.
fn module_prefix() -> &'static [u8];
/// Storage prefix. Used for generating final key.
fn storage_prefix() -> &'static [u8];
/// Storage prefix. Used for generating final head key.
fn head_prefix() -> &'static [u8];
/// Generate the full key used in top storage.
fn storage_linked_map_final_key<K>(key: &K) -> Vec<u8>
where
K: Encode,
{
let module_prefix_hashed = Twox128::hash(Self::module_prefix());
let storage_prefix_hashed = Twox128::hash(Self::storage_prefix());
let key_hashed = key.using_encoded(Self::Hasher::hash);
let mut final_key = Vec::with_capacity(
module_prefix_hashed.len() + storage_prefix_hashed.len() + key_hashed.as_ref().len()
);
final_key.extend_from_slice(&module_prefix_hashed[..]);
final_key.extend_from_slice(&storage_prefix_hashed[..]);
final_key.extend_from_slice(key_hashed.as_ref());
final_key
}
/// Generate the full key used in top storage to store the head of the linked map.
fn storage_linked_map_final_head_key() -> Vec<u8> {
[
Twox128::hash(Self::module_prefix()),
Twox128::hash(Self::head_prefix()),
].concat()
}
}
/// Linkage data of an element (it's successor and predecessor)
#[derive(Encode, Decode)]
pub struct Linkage<Key> {
/// Previous element key in storage (None for the first element)
pub previous: Option<Key>,
/// Next element key in storage (None for the last element)
pub next: Option<Key>,
}
impl<Key> Default for Linkage<Key> {
fn default() -> Self {
Self {
previous: None,
next: None,
}
}
}
// Encode like a linkage.
#[derive(Encode)]
struct EncodeLikeLinkage<PKey: EncodeLike<Key>, NKey: EncodeLike<Key>, Key: Encode> {
// Previous element key in storage (None for the first element)
previous: Option<PKey>,
// Next element key in storage (None for the last element)
next: Option<NKey>,
// The key of the linkage this type encode to
phantom: core::marker::PhantomData<Key>,
}
/// A key-value pair iterator for enumerable map.
pub struct Enumerator<K, V, F> {
next: Option<K>,
_phantom: PhantomData<(V, F)>,
}
impl<K, V, F> Enumerator<K, V, F> {
/// Create an explicit enumerator for testing.
#[cfg(test)]
pub fn from_head(head: K) -> Self {
Enumerator {
next: Some(head),
_phantom: Default::default(),
}
}
}
impl<K, V, F> Iterator for Enumerator<K, V, F>
where
K: FullCodec,
V: FullCodec,
F: KeyFormat,
{
type Item = (K, V);
fn next(&mut self) -> Option<Self::Item> {
let next = self.next.take()?;
let (val, linkage): (V, Linkage<K>) = {
let next_full_key = F::storage_linked_map_final_key(&next);
match read_with_linkage::<K, V>(next_full_key.as_ref()) {
Some(value) => value,
None => {
// TODO #3700: error should be handleable.
runtime_print!(
"ERROR: Corrupted state: linked map {:?}{:?}: \
next value doesn't exist at {:?}",
F::module_prefix(), F::storage_prefix(), next_full_key,
);
return None
}
}
};
self.next = linkage.next;
Some((next, val))
}
}
/// Update linkage when this element is removed.
///
/// Takes care of updating previous and next elements points
/// as well as updates head if the element is first or last.
fn remove_linkage<K, V, F>(linkage: Linkage<K>)
where
K: FullCodec,
V: FullCodec,
F: KeyFormat,
{
let next_key = linkage.next.as_ref().map(|k| F::storage_linked_map_final_key(k));
let prev_key = linkage.previous.as_ref().map(|k| F::storage_linked_map_final_key(k));
if let Some(prev_key) = prev_key {
// Retrieve previous element and update `next`
if let Some(mut res) = read_with_linkage::<K, V>(prev_key.as_ref()) {
res.1.next = linkage.next;
unhashed::put(prev_key.as_ref(), &res);
} else {
// TODO #3700: error should be handleable.
runtime_print!(
"ERROR: Corrupted state: linked map {:?}{:?}: \
previous value doesn't exist at {:?}",
F::module_prefix(), F::storage_prefix(), prev_key,
);
}
} else {
// we were first so let's update the head
write_head::<&K, K, F>(linkage.next.as_ref());
}
if let Some(next_key) = next_key {
// Update previous of next element
if let Some(mut res) = read_with_linkage::<K, V>(next_key.as_ref()) {
res.1.previous = linkage.previous;
unhashed::put(next_key.as_ref(), &res);
} else {
// TODO #3700: error should be handleable.
runtime_print!(
"ERROR: Corrupted state: linked map {:?}{:?}: \
next value doesn't exist at {:?}",
F::module_prefix(), F::storage_prefix(), next_key,
);
}
}
}
/// Read the contained data and its linkage.
pub(super) fn read_with_linkage<K, V>(key: &[u8]) -> Option<(V, Linkage<K>)>
where
K: Decode,
V: Decode,
{
unhashed::get(key)
}
/// Generate linkage for newly inserted element.
///
/// Takes care of updating head and previous head's pointer.
pub(super) fn new_head_linkage<KeyArg, K, V, F>(key: KeyArg) -> Linkage<K>
where
KeyArg: EncodeLike<K>,
K: FullCodec,
V: FullCodec,
F: KeyFormat,
{
if let Some(head) = read_head::<K, F>() {
// update previous head predecessor
{
let head_key = F::storage_linked_map_final_key(&head);
if let Some((data, linkage)) = read_with_linkage::<K, V>(head_key.as_ref()) {
let new_linkage = EncodeLikeLinkage::<_, _, K> {
previous: Some(Ref::from(&key)),
next: linkage.next.as_ref(),
phantom: Default::default(),
};
unhashed::put(head_key.as_ref(), &(data, new_linkage));
} else {
// TODO #3700: error should be handleable.
runtime_print!(
"ERROR: Corrupted state: linked map {:?}{:?}: \
head value doesn't exist at {:?}",
F::module_prefix(), F::storage_prefix(), head_key,
);
// Thus we consider we are first - update the head and produce empty linkage
write_head::<_, _, F>(Some(key));
return Linkage::default();
}
}
// update to current head
write_head::<_, _, F>(Some(key));
// return linkage with pointer to previous head
let mut linkage = Linkage::default();
linkage.next = Some(head);
linkage
} else {
// we are first - update the head and produce empty linkage
write_head::<_, _, F>(Some(key));
Linkage::default()
}
}
/// Read current head pointer.
pub(crate) fn read_head<K, F>() -> Option<K>
where
K: Decode,
F: KeyFormat,
{
unhashed::get(F::storage_linked_map_final_head_key().as_ref())
}
/// Overwrite current head pointer.
///
/// If `None` is given head is removed from storage.
pub(super) fn write_head<KeyArg, K, F>(head: Option<KeyArg>)
where
KeyArg: EncodeLike<K>,
K: FullCodec,
F: KeyFormat,
{
match head.as_ref() {
Some(head) => unhashed::put(F::storage_linked_map_final_head_key().as_ref(), head),
None => unhashed::kill(F::storage_linked_map_final_head_key().as_ref()),
}
}
impl<K, V, G> storage::StorageLinkedMap<K, V> for G
where
K: FullCodec,
V: FullCodec,
G: StorageLinkedMap<K, V>,
{
type Query = G::Query;
type Enumerator = Enumerator<K, V, G::KeyFormat>;
fn contains_key<KeyArg: EncodeLike<K>>(key: KeyArg) -> bool {
unhashed::exists(Self::storage_linked_map_final_key(key).as_ref())
}
fn get<KeyArg: EncodeLike<K>>(key: KeyArg) -> Self::Query {
let val = unhashed::get(Self::storage_linked_map_final_key(key).as_ref());
G::from_optional_value_to_query(val)
}
fn swap<KeyArg1: EncodeLike<K>, KeyArg2: EncodeLike<K>>(key1: KeyArg1, key2: KeyArg2) {
let final_key1 = Self::storage_linked_map_final_key(Ref::from(&key1));
let final_key2 = Self::storage_linked_map_final_key(Ref::from(&key2));
let full_value_1 = read_with_linkage::<K, V>(final_key1.as_ref());
let full_value_2 = read_with_linkage::<K, V>(final_key2.as_ref());
match (full_value_1, full_value_2) {
// Just keep linkage in order and only swap values.
(Some((value1, linkage1)), Some((value2, linkage2))) => {
unhashed::put(final_key1.as_ref(), &(value2, linkage1));
unhashed::put(final_key2.as_ref(), &(value1, linkage2));
}
// Remove key and insert the new one.
(Some((value, _linkage)), None) => {
Self::remove(key1);
let linkage = new_head_linkage::<_, _, V, G::KeyFormat>(key2);
unhashed::put(final_key2.as_ref(), &(value, linkage));
}
// Remove key and insert the new one.
(None, Some((value, _linkage))) => {
Self::remove(key2);
let linkage = new_head_linkage::<_, _, V, G::KeyFormat>(key1);
unhashed::put(final_key1.as_ref(), &(value, linkage));
}
// No-op.
(None, None) => (),
}
}
fn insert<KeyArg: EncodeLike<K>, ValArg: EncodeLike<V>>(key: KeyArg, val: ValArg) {
let final_key = Self::storage_linked_map_final_key(Ref::from(&key));
let linkage = match read_with_linkage::<_, V>(final_key.as_ref()) {
// overwrite but reuse existing linkage
Some((_data, linkage)) => linkage,
// create new linkage
None => new_head_linkage::<_, _, V, G::KeyFormat>(key),
};
unhashed::put(final_key.as_ref(), &(val, linkage))
}
fn remove<KeyArg: EncodeLike<K>>(key: KeyArg) {
G::take(key);
}
fn mutate<KeyArg: EncodeLike<K>, R, F: FnOnce(&mut Self::Query) -> R>(key: KeyArg, f: F) -> R {
let final_key = Self::storage_linked_map_final_key(Ref::from(&key));
let (mut val, _linkage) = read_with_linkage::<K, V>(final_key.as_ref())
.map(|(data, linkage)| (G::from_optional_value_to_query(Some(data)), Some(linkage)))
.unwrap_or_else(|| (G::from_optional_value_to_query(None), None));
let ret = f(&mut val);
match G::from_query_to_optional_value(val) {
Some(ref val) => G::insert(key, val),
None => G::remove(key),
}
ret
}
fn take<KeyArg: EncodeLike<K>>(key: KeyArg) -> Self::Query {
let final_key = Self::storage_linked_map_final_key(key);
let full_value: Option<(V, Linkage<K>)> = unhashed::take(final_key.as_ref());
let value = full_value.map(|(data, linkage)| {
remove_linkage::<K, V, G::KeyFormat>(linkage);
data
});
G::from_optional_value_to_query(value)
}
fn enumerate() -> Self::Enumerator {
Enumerator::<_, _, G::KeyFormat> {
next: read_head::<_, G::KeyFormat>(),
_phantom: Default::default(),
}
}
fn head() -> Option<K> {
read_head::<_, G::KeyFormat>()
}
fn decode_len<KeyArg: EncodeLike<K>>(key: KeyArg) -> Result<usize, &'static str>
where V: codec::DecodeLength + Len
{
let key = Self::storage_linked_map_final_key(key);
if let Some(v) = unhashed::get_raw(key.as_ref()) {
<V as codec::DecodeLength>::len(&v).map_err(|e| e.what())
} else {
let len = G::from_query_to_optional_value(G::from_optional_value_to_query(None))
.map(|v| v.len())
.unwrap_or(0);
Ok(len)
}
}
/// The translation happens in-place, new keys are inserted at the same time as old keys are
/// removed, thus new keys must not collide with still remaining old keys.
fn translate<K2, V2, TK, TV>(translate_key: TK, translate_val: TV) -> Result<(), Option<K2>>
where K2: FullCodec + Clone, V2: Decode, TK: Fn(K2) -> K, TV: Fn(V2) -> V
{
let head_key = read_head::<K2, G::KeyFormat>().ok_or(None)?;
let mut last_key = None;
let mut current_key = head_key.clone();
write_head::<&K, K, G::KeyFormat>(Some(&translate_key(head_key)));
let translate_linkage = |old: Linkage<K2>| -> Linkage<K> {
Linkage {
previous: old.previous.map(&translate_key),
next: old.next.map(&translate_key),
}
};
loop {
let old_raw_key = G::KeyFormat::storage_linked_map_final_key(&current_key);
let x = unhashed::take(old_raw_key.as_ref());
let (val, linkage): (V2, Linkage<K2>) = match x {
Some(v) => v,
None => {
// we failed to read value and linkage. Update the last key's linkage
// to end the map early, since it's impossible to iterate further.
if let Some(last_key) = last_key {
let last_raw_key = G::storage_linked_map_final_key(&last_key);
if let Some((val, mut linkage))
= read_with_linkage::<K, V>(last_raw_key.as_ref())
{
// defensive: should always happen, since it was just written
// in the last iteration of the loop.
linkage.next = None;
unhashed::put(last_raw_key.as_ref(), &(&val, &linkage));
}
}
return Err(Some(current_key));
}
};
let next = linkage.next.clone();
let val = translate_val(val);
let linkage = translate_linkage(linkage);
// and write in the value and linkage under the new key.
let new_key = translate_key(current_key.clone());
let new_raw_key = G::storage_linked_map_final_key(&new_key);
unhashed::put(new_raw_key.as_ref(), &(&val, &linkage));
match next {
None => break,
Some(next) => {
last_key = Some(new_key);
current_key = next
},
}
}
Ok(())
}
}
substrate/frame/support/src/storage/generator/map.rs
+143 -4
View File
@@ -17,8 +17,9 @@
#[cfg(not(feature = "std"))]
use sp_std::prelude::*;
use sp_std::borrow::Borrow;
use codec::{FullCodec, FullEncode, Encode, EncodeLike, Ref, EncodeAppend};
use crate::{storage::{self, unhashed}, hash::{StorageHasher, Twox128}, traits::Len};
use codec::{FullCodec, FullEncode, Decode, Encode, EncodeLike, Ref, EncodeAppend};
use crate::{storage::{self, unhashed}, traits::Len};
use crate::hash::{StorageHasher, Twox128, ReversibleStorageHasher};
/// Generator for `StorageMap` used by `decl_storage`.
///
@@ -44,6 +45,22 @@ pub trait StorageMap<K: FullEncode, V: FullCodec> {
/// Storage prefix. Used for generating final key.
fn storage_prefix() -> &'static [u8];
/// The full prefix; just the hash of `module_prefix` concatenated to the hash of
/// `storage_prefix`.
fn prefix_hash() -> Vec<u8> {
let module_prefix_hashed = Twox128::hash(Self::module_prefix());
let storage_prefix_hashed = Twox128::hash(Self::storage_prefix());
let mut result = Vec::with_capacity(
module_prefix_hashed.len() + storage_prefix_hashed.len()
);
result.extend_from_slice(&module_prefix_hashed[..]);
result.extend_from_slice(&storage_prefix_hashed[..]);
result
}
/// Convert an optional value retrieved from storage to the type queried.
fn from_optional_value_to_query(v: Option<V>) -> Self::Query;
@@ -51,8 +68,7 @@ pub trait StorageMap<K: FullEncode, V: FullCodec> {
fn from_query_to_optional_value(v: Self::Query) -> Option<V>;
/// Generate the full key used in top storage.
fn storage_map_final_key<KeyArg>(key: KeyArg) -> Vec<u8>
where
fn storage_map_final_key<KeyArg>(key: KeyArg) -> Vec<u8> where
KeyArg: EncodeLike<K>,
{
let module_prefix_hashed = Twox128::hash(Self::module_prefix());
@@ -71,6 +87,107 @@ pub trait StorageMap<K: FullEncode, V: FullCodec> {
}
}
/// Utility to iterate through items in a storage map.
pub struct StorageMapIterator<K, V, Hasher> {
prefix: Vec<u8>,
previous_key: Vec<u8>,
drain: bool,
_phantom: ::sp_std::marker::PhantomData<(K, V, Hasher)>,
}
impl<
K: Decode + Sized,
V: Decode + Sized,
Hasher: ReversibleStorageHasher
> Iterator for StorageMapIterator<K, V, Hasher> {
type Item = (K, V);
fn next(&mut self) -> Option<(K, V)> {
loop {
let maybe_next = sp_io::storage::next_key(&self.previous_key)
.filter(|n| n.starts_with(&self.prefix));
break match maybe_next {
Some(next) => {
self.previous_key = next;
match unhashed::get::<V>(&self.previous_key) {
Some(value) => {
if self.drain {
unhashed::kill(&self.previous_key)
}
let mut key_material = Hasher::reverse(&self.previous_key[self.prefix.len()..]);
match K::decode(&mut key_material) {
Ok(key) => Some((key, value)),
Err(_) => continue,
}
}
None => continue,
}
}
None => None,
}
}
}
}
impl<
K: FullCodec,
V: FullCodec,
G: StorageMap<K, V>,
> storage::IterableStorageMap<K, V> for G where
G::Hasher: ReversibleStorageHasher
{
type Iterator = StorageMapIterator<K, V, G::Hasher>;
/// Enumerate all elements in the map.
fn iter() -> Self::Iterator {
let prefix = G::prefix_hash();
Self::Iterator {
prefix: prefix.clone(),
previous_key: prefix,
drain: false,
_phantom: Default::default(),
}
}
/// Enumerate all elements in the map.
fn drain() -> Self::Iterator {
let prefix = G::prefix_hash();
Self::Iterator {
prefix: prefix.clone(),
previous_key: prefix,
drain: true,
_phantom: Default::default(),
}
}
fn translate<O: Decode, F: Fn(K, O) -> Option<V>>(f: F) {
let prefix = G::prefix_hash();
let mut previous_key = prefix.clone();
loop {
match sp_io::storage::next_key(&previous_key).filter(|n| n.starts_with(&prefix)) {
Some(next) => {
previous_key = next;
let maybe_value = unhashed::get::<O>(&previous_key);
match maybe_value {
Some(value) => {
let mut key_material = G::Hasher::reverse(&previous_key[prefix.len()..]);
match K::decode(&mut key_material) {
Ok(key) => match f(key, value) {
Some(new) => unhashed::put::<V>(&previous_key, &new),
None => unhashed::kill(&previous_key),
},
Err(_) => continue,
}
}
None => continue,
}
}
None => return,
}
}
}
}
impl<K: FullEncode, V: FullCodec, G: StorageMap<K, V>> storage::StorageMap<K, V> for G {
type Query = G::Query;
@@ -228,4 +345,26 @@ impl<K: FullEncode, V: FullCodec, G: StorageMap<K, V>> storage::StorageMap<K, V>
Ok(len)
}
}
fn migrate_key<OldHasher: StorageHasher, KeyArg: EncodeLike<K>>(key: KeyArg) -> Option<V> {
let old_key = {
let module_prefix_hashed = Twox128::hash(Self::module_prefix());
let storage_prefix_hashed = Twox128::hash(Self::storage_prefix());
let key_hashed = key.borrow().using_encoded(OldHasher::hash);
let mut final_key = Vec::with_capacity(
module_prefix_hashed.len() + storage_prefix_hashed.len() + key_hashed.as_ref().len()
);
final_key.extend_from_slice(&module_prefix_hashed[..]);
final_key.extend_from_slice(&storage_prefix_hashed[..]);
final_key.extend_from_slice(key_hashed.as_ref());
final_key
};
unhashed::take(old_key.as_ref()).map(|value| {
unhashed::put(Self::storage_map_final_key(key).as_ref(), &value);
value
})
}
}
substrate/frame/support/src/storage/generator/mod.rs
+10 -35
View File
@@ -23,23 +23,20 @@
//!
//! This is internal api and is subject to change.
mod linked_map;
mod map;
mod double_map;
mod value;
pub use linked_map::{StorageLinkedMap, Enumerator, Linkage, KeyFormat as LinkedMapKeyFormat};
pub use map::StorageMap;
pub use double_map::StorageDoubleMap;
pub use value::StorageValue;
#[cfg(test)]
#[allow(dead_code)]
mod tests {
use sp_io::TestExternalities;
use codec::{Encode, Decode};
use crate::storage::{unhashed, generator::{StorageValue, StorageLinkedMap}};
use codec::Encode;
use crate::storage::{unhashed, generator::StorageValue, IterableStorageMap};
struct Runtime {}
pub trait Trait {
@@ -56,16 +53,10 @@ mod tests {
pub struct Module<T: Trait> for enum Call where origin: T::Origin {}
}
#[derive(Encode, Decode, Clone, Debug, Eq, PartialEq)]
struct NumberNumber {
a: u32,
b: u32,
}
crate::decl_storage! {
trait Store for Module<T: Trait> as Runtime {
Value get(fn value) config(): (u64, u64);
NumberMap: linked_map hasher(blake2_256) NumberNumber => u64;
NumberMap: map hasher(identity) u32 => u64;
}
}
@@ -89,41 +80,25 @@ mod tests {
}
#[test]
fn linked_map_translate_works() {
use super::linked_map::{self, Enumerator, KeyFormat};
type Format = <NumberMap as StorageLinkedMap<NumberNumber, u64>>::KeyFormat;
fn map_translate_works() {
let t = GenesisConfig::default().build_storage().unwrap();
TestExternalities::new(t).execute_with(|| {
// start with a map of u32 -> u32.
for i in 0u32..100u32 {
let final_key = <Format as KeyFormat>::storage_linked_map_final_key(&i);
let linkage = linked_map::new_head_linkage::<_, u32, u32, Format>(&i);
unhashed::put(final_key.as_ref(), &(&i, linkage));
unhashed::put(&NumberMap::hashed_key_for(&i), &(i as u64));
}
let head = linked_map::read_head::<u32, Format>().unwrap();
assert_eq!(
Enumerator::<u32, u32, Format>::from_head(head).collect::<Vec<_>>(),
(0..100).rev().map(|x| (x, x)).collect::<Vec<_>>(),
NumberMap::iter().collect::<Vec<_>>(),
(0..100).map(|x| (x as u32, x as u64)).collect::<Vec<_>>(),
);
// do translation.
NumberMap::translate(
|k: u32| NumberNumber { a: k, b: k },
|v: u32| (v as u64) << 32 | v as u64,
).unwrap();
NumberMap::translate(|k: u32, v: u64| if k % 2 == 0 { Some((k as u64) << 32 | v) } else { None });
assert!(linked_map::read_head::<NumberNumber, Format>().is_some());
assert_eq!(
NumberMap::enumerate().collect::<Vec<_>>(),
(0..100u32).rev().map(|x| (
NumberNumber { a: x, b: x },
(x as u64) << 32 | x as u64,
)).collect::<Vec<_>>(),
NumberMap::iter().collect::<Vec<_>>(),
(0..50u32).map(|x| x * 2).map(|x| (x, (x as u64) << 32 | x as u64)).collect::<Vec<_>>(),
);
})
}
substrate/frame/support/src/storage/mod.rs
+63 -70
View File
@@ -202,78 +202,60 @@ pub trait StorageMap<K: FullEncode, V: FullCodec> {
/// function for this purpose.
fn decode_len<KeyArg: EncodeLike<K>>(key: KeyArg) -> Result<usize, &'static str>
where V: codec::DecodeLength + Len;
/// Migrate an item with the given `key` from a defunct `OldHasher` to the current hasher.
///
/// If the key doesn't exist, then it's a no-op. If it does, then it returns its value.
fn migrate_key<OldHasher: StorageHasher, KeyArg: EncodeLike<K>>(key: KeyArg) -> Option<V>;
/// Migrate an item with the given `key` from a `blake2_256` hasher to the current hasher.
///
/// If the key doesn't exist, then it's a no-op. If it does, then it returns its value.
fn migrate_key_from_blake<KeyArg: EncodeLike<K>>(key: KeyArg) -> Option<V> {
Self::migrate_key::<crate::hash::Blake2_256, KeyArg>(key)
}
}
/// A strongly-typed linked map in storage.
///
/// Similar to `StorageMap` but allows to enumerate other elements and doesn't implement append.
///
/// Details on implementation can be found at
/// [`generator::StorageLinkedMap`]
pub trait StorageLinkedMap<K: FullCodec, V: FullCodec> {
/// The type that get/take return.
type Query;
/// A strongly-typed map in storage whose keys and values can be iterated over.
pub trait IterableStorageMap<K: FullEncode, V: FullCodec>: StorageMap<K, V> {
/// The type that iterates over all `(key, value)`.
type Enumerator: Iterator<Item = (K, V)>;
type Iterator: Iterator<Item = (K, V)>;
/// Does the value (explicitly) exist in storage?
fn contains_key<KeyArg: EncodeLike<K>>(key: KeyArg) -> bool;
/// Enumerate all elements in the map in no particular order. If you alter the map while doing
/// this, you'll get undefined results.
fn iter() -> Self::Iterator;
/// Load the value associated with the given key from the map.
fn get<KeyArg: EncodeLike<K>>(key: KeyArg) -> Self::Query;
/// Remove all elements from the map and iterate through them in no particular order. If you
/// add elements to the map while doing this, you'll get undefined results.
fn drain() -> Self::Iterator;
/// Swap the values of two keys.
fn swap<KeyArg1: EncodeLike<K>, KeyArg2: EncodeLike<K>>(key1: KeyArg1, key2: KeyArg2);
/// Translate the values of all elements by a function `f`, in the map in no particular order.
/// By returning `None` from `f` for an element, you'll remove it from the map.
fn translate<O: Decode, F: Fn(K, O) -> Option<V>>(f: F);
}
/// Store a value to be associated with the given key from the map.
fn insert<KeyArg: EncodeLike<K>, ValArg: EncodeLike<V>>(key: KeyArg, val: ValArg);
/// A strongly-typed double map in storage whose secondary keys and values can be iterated over.
pub trait IterableStorageDoubleMap<
K1: FullCodec,
K2: FullCodec,
V: FullCodec
>: StorageDoubleMap<K1, K2, V> {
/// The type that iterates over all `(key, value)`.
type Iterator: Iterator<Item = (K2, V)>;
/// Remove the value under a key.
fn remove<KeyArg: EncodeLike<K>>(key: KeyArg);
/// Enumerate all elements in the map with first key `k1` in no particular order. If you add or
/// remove values whose first key is `k1` to the map while doing this, you'll get undefined
/// results.
fn iter(k1: impl EncodeLike<K1>) -> Self::Iterator;
/// Mutate the value under a key.
fn mutate<KeyArg: EncodeLike<K>, R, F: FnOnce(&mut Self::Query) -> R>(key: KeyArg, f: F) -> R;
/// Remove all elements from the map with first key `k1` and iterate through them in no
/// particular order. If you add elements with first key `k1` to the map while doing this,
/// you'll get undefined results.
fn drain(k1: impl EncodeLike<K1>) -> Self::Iterator;
/// Take the value under a key.
fn take<KeyArg: EncodeLike<K>>(key: KeyArg) -> Self::Query;
/// Return current head element.
fn head() -> Option<K>;
/// Enumerate all elements in the map.
fn enumerate() -> Self::Enumerator;
/// Read the length of the value in a fast way, without decoding the entire value.
///
/// `T` is required to implement `Codec::DecodeLength`.
///
/// Note that `0` is returned as the default value if no encoded value exists at the given key.
/// Therefore, this function cannot be used as a sign of _existence_. use the `::contains_key()`
/// function for this purpose.
fn decode_len<KeyArg: EncodeLike<K>>(key: KeyArg) -> Result<usize, &'static str>
where V: codec::DecodeLength + Len;
/// Translate the keys and values from some previous `(K2, V2)` to the current type.
///
/// `TK` translates keys from the old type, and `TV` translates values.
///
/// Returns `Err` if the map could not be interpreted as the old type, and Ok if it could.
/// The `Err` contains the first key which could not be migrated, or `None` if the
/// head of the list could not be read.
///
/// # Warning
///
/// This function must be used with care, before being updated the storage still contains the
/// old type, thus other calls (such as `get`) will fail at decoding it.
///
/// # Usage
///
/// This would typically be called inside the module implementation of on_runtime_upgrade, while
/// ensuring **no usage of this storage are made before the call to `on_runtime_upgrade`**. (More
/// precisely prior initialized modules doesn't make use of this storage).
fn translate<K2, V2, TK, TV>(translate_key: TK, translate_val: TV) -> Result<(), Option<K2>>
where K2: FullCodec + Clone, V2: Decode, TK: Fn(K2) -> K, TV: Fn(V2) -> V;
/// Translate the values of all elements by a function `f`, in the map in no particular order.
/// By returning `None` from `f` for an element, you'll remove it from the map.
fn translate<O: Decode, F: Fn(O) -> Option<V>>(f: F);
}
/// An implementation of a map with a two keys.
@@ -377,6 +359,17 @@ pub trait StorageDoubleMap<K1: FullEncode, K2: FullEncode, V: FullCodec> {
KArg1: EncodeLike<K1>,
KArg2: EncodeLike<K2>,
V: codec::DecodeLength + Len;
/// Migrate an item with the given `key1` and `key2` from defunct `OldHasher1` and
/// `OldHasher2` to the current hashers.
///
/// If the key doesn't exist, then it's a no-op. If it does, then it returns its value.
fn migrate_keys<
OldHasher1: StorageHasher,
OldHasher2: StorageHasher,
KeyArg1: EncodeLike<K1>,
KeyArg2: EncodeLike<K2>,
>(key1: KeyArg1, key2: KeyArg2) -> Option<V>;
}
/// Iterator for prefixed map.
@@ -440,7 +433,7 @@ pub trait StoragePrefixedMap<Value: FullCodec> {
}
/// Iter over all value of the storage.
fn iter() -> PrefixIterator<Value> {
fn iter_values() -> PrefixIterator<Value> {
let prefix = Self::final_prefix();
PrefixIterator {
prefix: prefix.to_vec(),
@@ -535,26 +528,26 @@ mod test {
assert_eq!(MyStorage::final_prefix().to_vec(), k);
// test iteration
assert_eq!(MyStorage::iter().collect::<Vec<_>>(), vec![]);
assert_eq!(MyStorage::iter_values().collect::<Vec<_>>(), vec![]);
unhashed::put(&[&k[..], &vec![1][..]].concat(), &1u64);
unhashed::put(&[&k[..], &vec![1, 1][..]].concat(), &2u64);
unhashed::put(&[&k[..], &vec![8][..]].concat(), &3u64);
unhashed::put(&[&k[..], &vec![10][..]].concat(), &4u64);
assert_eq!(MyStorage::iter().collect::<Vec<_>>(), vec![1, 2, 3, 4]);
assert_eq!(MyStorage::iter_values().collect::<Vec<_>>(), vec![1, 2, 3, 4]);
// test removal
MyStorage::remove_all();
assert_eq!(MyStorage::iter().collect::<Vec<_>>(), vec![]);
assert_eq!(MyStorage::iter_values().collect::<Vec<_>>(), vec![]);
// test migration
unhashed::put(&[&k[..], &vec![1][..]].concat(), &1u32);
unhashed::put(&[&k[..], &vec![8][..]].concat(), &2u32);
assert_eq!(MyStorage::iter().collect::<Vec<_>>(), vec![]);
assert_eq!(MyStorage::iter_values().collect::<Vec<_>>(), vec![]);
MyStorage::translate_values(|v: u32| v as u64).unwrap();
assert_eq!(MyStorage::iter().collect::<Vec<_>>(), vec![1, 2]);
assert_eq!(MyStorage::iter_values().collect::<Vec<_>>(), vec![1, 2]);
MyStorage::remove_all();
// test migration 2
@@ -564,9 +557,9 @@ mod test {
unhashed::put(&[&k[..], &vec![10][..]].concat(), &4u32);
// (contains some value that successfully decoded to u64)
assert_eq!(MyStorage::iter().collect::<Vec<_>>(), vec![1, 2, 3]);
assert_eq!(MyStorage::iter_values().collect::<Vec<_>>(), vec![1, 2, 3]);
assert_eq!(MyStorage::translate_values(|v: u128| v as u64), Err(2));
assert_eq!(MyStorage::iter().collect::<Vec<_>>(), vec![1, 3]);
assert_eq!(MyStorage::iter_values().collect::<Vec<_>>(), vec![1, 3]);
MyStorage::remove_all();
// test that other values are not modified.
substrate/frame/support/src/traits.rs
+7 -1
View File
@@ -26,10 +26,16 @@ use sp_runtime::{
ConsensusEngineId, DispatchResult, DispatchError,
traits::{MaybeSerializeDeserialize, AtLeast32Bit, Saturating, TrailingZeroInput},
};
use crate::dispatch::Parameter;
use crate::storage::StorageMap;
/// Migrate a given account.
#[impl_trait_for_tuples::impl_for_tuples(30)]
pub trait MigrateAccount<A> {
/// Migrate the `account`.
fn migrate_account(account: &A);
}
/// An abstraction of a value stored within storage, but possibly as part of a larger composite
/// item.
pub trait StoredMap<K, T> {
substrate/frame/support/test/tests/decl_storage.rs
+36 -116
View File
@@ -37,10 +37,10 @@ mod tests {
// non-getters: pub / $default
/// Hello, this is doc!
U32 : Option<u32>;
pub PUBU32 : Option<u32>;
U32MYDEF : Option<u32>;
pub PUBU32MYDEF : Option<u32>;
U32: Option<u32>;
pub PUBU32: Option<u32>;
U32MYDEF: Option<u32>;
pub PUBU32MYDEF: Option<u32>;
// getters: pub / $default
// we need at least one type which uses T, otherwise GenesisConfig will complain.
@@ -59,31 +59,23 @@ mod tests {
GetOptU32WithBuilderNone get(fn opt_u32_with_builder_none) build(|_| None): Option<u32>;
// map non-getters: pub / $default
MAPU32 : map hasher(blake2_256) u32 => Option<String>;
pub PUBMAPU32 : map hasher(blake2_256) u32 => Option<String>;
MAPU32MYDEF : map hasher(blake2_256) u32 => Option<String>;
pub PUBMAPU32MYDEF : map hasher(blake2_256) u32 => Option<String>;
MAPU32: map hasher(blake2_128_concat) u32 => Option<String>;
pub PUBMAPU32: map hasher(blake2_128_concat) u32 => Option<String>;
MAPU32MYDEF: map hasher(blake2_128_concat) u32 => Option<String>;
pub PUBMAPU32MYDEF: map hasher(blake2_128_concat) u32 => Option<String>;
// map getters: pub / $default
GETMAPU32 get(fn map_u32_getter): map hasher(blake2_256) u32 => String;
pub PUBGETMAPU32 get(fn pub_map_u32_getter): map hasher(blake2_256) u32 => String;
GETMAPU32 get(fn map_u32_getter): map hasher(blake2_128_concat) u32 => String;
pub PUBGETMAPU32 get(fn pub_map_u32_getter): map hasher(blake2_128_concat) u32 => String;
GETMAPU32MYDEF get(fn map_u32_getter_mydef):
map hasher(blake2_256) u32 => String = "map".into();
map hasher(blake2_128_concat) u32 => String = "map".into();
pub PUBGETMAPU32MYDEF get(fn pub_map_u32_getter_mydef):
map hasher(blake2_256) u32 => String = "pubmap".into();
// linked map
LINKEDMAPU32 : linked_map hasher(blake2_256) u32 => Option<String>;
pub PUBLINKEDMAPU32MYDEF : linked_map hasher(blake2_256) u32 => Option<String>;
GETLINKEDMAPU32 get(fn linked_map_u32_getter):
linked_map hasher(blake2_256) u32 => String;
pub PUBGETLINKEDMAPU32MYDEF get(fn pub_linked_map_u32_getter_mydef):
linked_map hasher(blake2_256) u32 => String = "pubmap".into();
map hasher(blake2_128_concat) u32 => String = "pubmap".into();
COMPLEXTYPE1: ::std::vec::Vec<<T as Trait>::Origin>;
COMPLEXTYPE2: (Vec<Vec<(u16,Box<( )>)>>, u32);
COMPLEXTYPE3: [u32;25];
COMPLEXTYPE2: (Vec<Vec<(u16, Box<()>)>>, u32);
COMPLEXTYPE3: [u32; 25];
}
add_extra_genesis {
build(|_| {});
@@ -249,10 +241,10 @@ mod tests {
name: DecodeDifferent::Encode("MAPU32"),
modifier: StorageEntryModifier::Optional,
ty: StorageEntryType::Map {
hasher: StorageHasher::Blake2_256,
hasher: StorageHasher::Blake2_128Concat,
key: DecodeDifferent::Encode("u32"),
value: DecodeDifferent::Encode("String"),
is_linked: false,
unused: false,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructMAPU32(PhantomData::<TraitImpl>))
@@ -263,10 +255,10 @@ mod tests {
name: DecodeDifferent::Encode("PUBMAPU32"),
modifier: StorageEntryModifier::Optional,
ty: StorageEntryType::Map {
hasher: StorageHasher::Blake2_256,
hasher: StorageHasher::Blake2_128Concat,
key: DecodeDifferent::Encode("u32"),
value: DecodeDifferent::Encode("String"),
is_linked: false,
unused: false,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructPUBMAPU32(PhantomData::<TraitImpl>))
@@ -277,10 +269,10 @@ mod tests {
name: DecodeDifferent::Encode("MAPU32MYDEF"),
modifier: StorageEntryModifier::Optional,
ty: StorageEntryType::Map {
hasher: StorageHasher::Blake2_256,
hasher: StorageHasher::Blake2_128Concat,
key: DecodeDifferent::Encode("u32"),
value: DecodeDifferent::Encode("String"),
is_linked: false,
unused: false,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructMAPU32MYDEF(PhantomData::<TraitImpl>))
@@ -291,10 +283,10 @@ mod tests {
name: DecodeDifferent::Encode("PUBMAPU32MYDEF"),
modifier: StorageEntryModifier::Optional,
ty: StorageEntryType::Map {
hasher: StorageHasher::Blake2_256,
hasher: StorageHasher::Blake2_128Concat,
key: DecodeDifferent::Encode("u32"),
value: DecodeDifferent::Encode("String"),
is_linked: false,
unused: false,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructPUBMAPU32MYDEF(PhantomData::<TraitImpl>))
@@ -305,10 +297,10 @@ mod tests {
name: DecodeDifferent::Encode("GETMAPU32"),
modifier: StorageEntryModifier::Default,
ty: StorageEntryType::Map {
hasher: StorageHasher::Blake2_256,
hasher: StorageHasher::Blake2_128Concat,
key: DecodeDifferent::Encode("u32"),
value: DecodeDifferent::Encode("String"),
is_linked: false,
unused: false,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructGETMAPU32(PhantomData::<TraitImpl>))
@@ -319,10 +311,10 @@ mod tests {
name: DecodeDifferent::Encode("PUBGETMAPU32"),
modifier: StorageEntryModifier::Default,
ty: StorageEntryType::Map {
hasher: StorageHasher::Blake2_256,
hasher: StorageHasher::Blake2_128Concat,
key: DecodeDifferent::Encode("u32"),
value: DecodeDifferent::Encode("String"),
is_linked: false,
unused: false,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructPUBGETMAPU32(PhantomData::<TraitImpl>))
@@ -333,10 +325,10 @@ mod tests {
name: DecodeDifferent::Encode("GETMAPU32MYDEF"),
modifier: StorageEntryModifier::Default,
ty: StorageEntryType::Map {
hasher: StorageHasher::Blake2_256,
hasher: StorageHasher::Blake2_128Concat,
key: DecodeDifferent::Encode("u32"),
value: DecodeDifferent::Encode("String"),
is_linked: false,
unused: false,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructGETMAPU32MYDEF(PhantomData::<TraitImpl>))
@@ -347,72 +339,16 @@ mod tests {
name: DecodeDifferent::Encode("PUBGETMAPU32MYDEF"),
modifier: StorageEntryModifier::Default,
ty: StorageEntryType::Map {
hasher: StorageHasher::Blake2_256,
hasher: StorageHasher::Blake2_128Concat,
key: DecodeDifferent::Encode("u32"),
value: DecodeDifferent::Encode("String"),
is_linked: false,
unused: false,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructPUBGETMAPU32MYDEF(PhantomData::<TraitImpl>))
),
documentation: DecodeDifferent::Encode(&[]),
},
StorageEntryMetadata {
name: DecodeDifferent::Encode("LINKEDMAPU32"),
modifier: StorageEntryModifier::Optional,
ty: StorageEntryType::Map {
hasher: StorageHasher::Blake2_256,
key: DecodeDifferent::Encode("u32"),
value: DecodeDifferent::Encode("String"),
is_linked: true,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructLINKEDMAPU32(PhantomData::<TraitImpl>))
),
documentation: DecodeDifferent::Encode(&[]),
},
StorageEntryMetadata {
name: DecodeDifferent::Encode("PUBLINKEDMAPU32MYDEF"),
modifier: StorageEntryModifier::Optional,
ty: StorageEntryType::Map {
hasher: StorageHasher::Blake2_256,
key: DecodeDifferent::Encode("u32"),
value: DecodeDifferent::Encode("String"),
is_linked: true,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructPUBLINKEDMAPU32MYDEF(PhantomData::<TraitImpl>))
),
documentation: DecodeDifferent::Encode(&[]),
},
StorageEntryMetadata {
name: DecodeDifferent::Encode("GETLINKEDMAPU32"),
modifier: StorageEntryModifier::Default,
ty: StorageEntryType::Map {
hasher: StorageHasher::Blake2_256,
key: DecodeDifferent::Encode("u32"),
value: DecodeDifferent::Encode("String"),
is_linked: true,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructGETLINKEDMAPU32(PhantomData::<TraitImpl>))
),
documentation: DecodeDifferent::Encode(&[]),
},
StorageEntryMetadata {
name: DecodeDifferent::Encode("PUBGETLINKEDMAPU32MYDEF"),
modifier: StorageEntryModifier::Default,
ty: StorageEntryType::Map {
hasher: StorageHasher::Blake2_256,
key: DecodeDifferent::Encode("u32"),
value: DecodeDifferent::Encode("String"),
is_linked: true,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(&__GetByteStructPUBGETLINKEDMAPU32MYDEF(PhantomData::<TraitImpl>))
),
documentation: DecodeDifferent::Encode(&[]),
},
StorageEntryMetadata {
name: DecodeDifferent::Encode("COMPLEXTYPE1"),
modifier: StorageEntryModifier::Default,
@@ -562,17 +498,13 @@ mod test_append_and_len {
JustVecWithDefault: Vec<u32> = vec![6, 9];
OptionVec: Option<Vec<u32>>;
MapVec: map hasher(blake2_256) u32 => Vec<u32>;
MapVecWithDefault: map hasher(blake2_256) u32 => Vec<u32> = vec![6, 9];
OptionMapVec: map hasher(blake2_256) u32 => Option<Vec<u32>>;
MapVec: map hasher(blake2_128_concat) u32 => Vec<u32>;
MapVecWithDefault: map hasher(blake2_128_concat) u32 => Vec<u32> = vec![6, 9];
OptionMapVec: map hasher(blake2_128_concat) u32 => Option<Vec<u32>>;
DoubleMapVec: double_map hasher(blake2_256) u32, hasher(blake2_256) u32 => Vec<u32>;
DoubleMapVecWithDefault: double_map hasher(blake2_256) u32, hasher(blake2_256) u32 => Vec<u32> = vec![6, 9];
OptionDoubleMapVec: double_map hasher(blake2_256) u32, hasher(blake2_256) u32 => Option<Vec<u32>>;
LinkedMapVec: linked_map hasher(blake2_256) u32 => Vec<u32>;
LinkedMapVecWithDefault: linked_map hasher(blake2_256) u32 => Vec<u32> = vec![6, 9];
OptionLinkedMapVec: linked_map hasher(blake2_256) u32 => Option<Vec<u32>>;
DoubleMapVec: double_map hasher(blake2_128_concat) u32, hasher(blake2_128_concat) u32 => Vec<u32>;
DoubleMapVecWithDefault: double_map hasher(blake2_128_concat) u32, hasher(blake2_128_concat) u32 => Vec<u32> = vec![6, 9];
OptionDoubleMapVec: double_map hasher(blake2_128_concat) u32, hasher(blake2_128_concat) u32 => Option<Vec<u32>>;
}
}
@@ -644,13 +576,11 @@ mod test_append_and_len {
JustVec::put(&vec![1, 2, 3, 4]);
OptionVec::put(&vec![1, 2, 3, 4, 5]);
MapVec::insert(1, &vec![1, 2, 3, 4, 5, 6]);
LinkedMapVec::insert(2, &vec![1, 2, 3]);
DoubleMapVec::insert(0, 1, &vec![1, 2]);
assert_eq!(JustVec::decode_len().unwrap(), 4);
assert_eq!(OptionVec::decode_len().unwrap(), 5);
assert_eq!(MapVec::decode_len(1).unwrap(), 6);
assert_eq!(LinkedMapVec::decode_len(2).unwrap(), 3);
assert_eq!(DoubleMapVec::decode_len(0, 1).unwrap(), 2);
});
}
@@ -678,16 +608,6 @@ mod test_append_and_len {
assert_eq!(OptionMapVec::get(0), None);
assert_eq!(OptionMapVec::decode_len(0), Ok(0));
// linked map
assert_eq!(LinkedMapVec::get(0), vec![]);
assert_eq!(LinkedMapVec::decode_len(0), Ok(0));
assert_eq!(LinkedMapVecWithDefault::get(0), vec![6, 9]);
assert_eq!(LinkedMapVecWithDefault::decode_len(0), Ok(2));
assert_eq!(OptionLinkedMapVec::get(0), None);
assert_eq!(OptionLinkedMapVec::decode_len(0), Ok(0));
// Double map
assert_eq!(DoubleMapVec::get(0, 0), vec![]);
assert_eq!(DoubleMapVec::decode_len(0, 1), Ok(0));
substrate/frame/support/test/tests/final_keys.rs
+42 -78
View File
@@ -16,8 +16,8 @@
use frame_support::storage::unhashed;
use codec::Encode;
use frame_support::{StorageDoubleMap, StorageLinkedMap, StorageMap, StorageValue, StoragePrefixedMap};
use sp_io::{TestExternalities, hashing::{twox_128, blake2_128, blake2_256}};
use frame_support::{StorageDoubleMap, StorageMap, StorageValue, StoragePrefixedMap};
use sp_io::{TestExternalities, hashing::{twox_64, twox_128, blake2_128}};
mod no_instance {
use codec::{Encode, Decode, EncodeLike};
@@ -35,18 +35,15 @@ mod no_instance {
trait Store for Module<T: Trait> as FinalKeysNone {
pub Value config(value): u32;
pub Map: map hasher(blake2_256) u32 => u32;
pub Map2: map hasher(twox_128) u32 => u32;
pub Map: map hasher(blake2_128_concat) u32 => u32;
pub Map2: map hasher(twox_64_concat) u32 => u32;
pub LinkedMap: linked_map hasher(blake2_256) u32 => u32;
pub LinkedMap2: linked_map hasher(twox_128) u32 => u32;
pub DoubleMap: double_map hasher(blake2_256) u32, hasher(blake2_256) u32 => u32;
pub DoubleMap2: double_map hasher(twox_128) u32, hasher(blake2_128) u32 => u32;
pub DoubleMap: double_map hasher(blake2_128_concat) u32, hasher(blake2_128_concat) u32 => u32;
pub DoubleMap2: double_map hasher(twox_64_concat) u32, hasher(twox_64_concat) u32 => u32;
pub TestGenericValue get(fn test_generic_value) config(): Option<T::BlockNumber>;
pub TestGenericDoubleMap get(fn foo2) config(test_generic_double_map):
double_map hasher(blake2_256) u32, hasher(blake2_256) T::BlockNumber => Option<u32>;
double_map hasher(blake2_128_concat) u32, hasher(blake2_128_concat) T::BlockNumber => Option<u32>;
}
}
}
@@ -65,18 +62,15 @@ mod instance {
{
pub Value config(value): u32;
pub Map: map hasher(blake2_256) u32 => u32;
pub Map2: map hasher(twox_128) u32 => u32;
pub Map: map hasher(blake2_128_concat) u32 => u32;
pub Map2: map hasher(twox_64_concat) u32 => u32;
pub LinkedMap: linked_map hasher(blake2_256) u32 => u32;
pub LinkedMap2: linked_map hasher(twox_128) u32 => u32;
pub DoubleMap: double_map hasher(blake2_256) u32, hasher(blake2_256) u32 => u32;
pub DoubleMap2: double_map hasher(twox_128) u32, hasher(blake2_128) u32 => u32;
pub DoubleMap: double_map hasher(blake2_128_concat) u32, hasher(blake2_128_concat) u32 => u32;
pub DoubleMap2: double_map hasher(twox_64_concat) u32, hasher(twox_64_concat) u32 => u32;
pub TestGenericValue get(fn test_generic_value) config(): Option<T::BlockNumber>;
pub TestGenericDoubleMap get(fn foo2) config(test_generic_double_map):
double_map hasher(blake2_256) u32, hasher(blake2_256) T::BlockNumber => Option<u32>;
double_map hasher(blake2_128_concat) u32, hasher(blake2_128_concat) T::BlockNumber => Option<u32>;
}
add_extra_genesis {
// See `decl_storage` limitation.
@@ -85,6 +79,18 @@ mod instance {
}
}
fn twox_64_concat(d: &[u8]) -> Vec<u8> {
let mut v = twox_64(d).to_vec();
v.extend_from_slice(d);
v
}
fn blake2_128_concat(d: &[u8]) -> Vec<u8> {
let mut v = blake2_128(d).to_vec();
v.extend_from_slice(d);
v
}
#[test]
fn final_keys_no_instance() {
TestExternalities::default().execute_with(|| {
@@ -94,41 +100,27 @@ fn final_keys_no_instance() {
no_instance::Map::insert(1, 2);
let mut k = [twox_128(b"FinalKeysNone"), twox_128(b"Map")].concat();
k.extend(1u32.using_encoded(blake2_256).to_vec());
k.extend(1u32.using_encoded(blake2_128_concat));
assert_eq!(unhashed::get::<u32>(&k), Some(2u32));
assert_eq!(&k[..32], &<no_instance::Map>::final_prefix());
no_instance::Map2::insert(1, 2);
let mut k = [twox_128(b"FinalKeysNone"), twox_128(b"Map2")].concat();
k.extend(1u32.using_encoded(twox_128).to_vec());
k.extend(1u32.using_encoded(twox_64_concat));
assert_eq!(unhashed::get::<u32>(&k), Some(2u32));
assert_eq!(&k[..32], &<no_instance::Map2>::final_prefix());
let head = [twox_128(b"FinalKeysNone"), twox_128(b"HeadOfLinkedMap")].concat();
assert_eq!(unhashed::get::<u32>(&head), None);
no_instance::LinkedMap::insert(1, 2);
let mut k = [twox_128(b"FinalKeysNone"), twox_128(b"LinkedMap")].concat();
k.extend(1u32.using_encoded(blake2_256).to_vec());
assert_eq!(unhashed::get::<u32>(&k), Some(2u32));
assert_eq!(unhashed::get::<u32>(&head), Some(1u32));
no_instance::LinkedMap2::insert(1, 2);
let mut k = [twox_128(b"FinalKeysNone"), twox_128(b"LinkedMap2")].concat();
k.extend(1u32.using_encoded(twox_128).to_vec());
assert_eq!(unhashed::get::<u32>(&k), Some(2u32));
no_instance::DoubleMap::insert(&1, &2, &3);
let mut k = [twox_128(b"FinalKeysNone"), twox_128(b"DoubleMap")].concat();
k.extend(1u32.using_encoded(blake2_256).to_vec());
k.extend(2u32.using_encoded(blake2_256).to_vec());
k.extend(1u32.using_encoded(blake2_128_concat));
k.extend(2u32.using_encoded(blake2_128_concat));
assert_eq!(unhashed::get::<u32>(&k), Some(3u32));
assert_eq!(&k[..32], &<no_instance::DoubleMap>::final_prefix());
no_instance::DoubleMap2::insert(&1, &2, &3);
let mut k = [twox_128(b"FinalKeysNone"), twox_128(b"DoubleMap2")].concat();
k.extend(1u32.using_encoded(twox_128).to_vec());
k.extend(2u32.using_encoded(blake2_128).to_vec());
k.extend(1u32.using_encoded(twox_64_concat));
k.extend(2u32.using_encoded(twox_64_concat));
assert_eq!(unhashed::get::<u32>(&k), Some(3u32));
assert_eq!(&k[..32], &<no_instance::DoubleMap2>::final_prefix());
});
@@ -143,41 +135,27 @@ fn final_keys_default_instance() {
<instance::Map<instance::DefaultInstance>>::insert(1, 2);
let mut k = [twox_128(b"FinalKeysSome"), twox_128(b"Map")].concat();
k.extend(1u32.using_encoded(blake2_256).to_vec());
k.extend(1u32.using_encoded(blake2_128_concat));
assert_eq!(unhashed::get::<u32>(&k), Some(2u32));
assert_eq!(&k[..32], &<instance::Map<instance::DefaultInstance>>::final_prefix());
<instance::Map2<instance::DefaultInstance>>::insert(1, 2);
let mut k = [twox_128(b"FinalKeysSome"), twox_128(b"Map2")].concat();
k.extend(1u32.using_encoded(twox_128).to_vec());
k.extend(1u32.using_encoded(twox_64_concat));
assert_eq!(unhashed::get::<u32>(&k), Some(2u32));
assert_eq!(&k[..32], &<instance::Map2<instance::DefaultInstance>>::final_prefix());
let head = [twox_128(b"FinalKeysSome"), twox_128(b"HeadOfLinkedMap")].concat();
assert_eq!(unhashed::get::<u32>(&head), None);
<instance::LinkedMap<instance::DefaultInstance>>::insert(1, 2);
let mut k = [twox_128(b"FinalKeysSome"), twox_128(b"LinkedMap")].concat();
k.extend(1u32.using_encoded(blake2_256).to_vec());
assert_eq!(unhashed::get::<u32>(&k), Some(2u32));
assert_eq!(unhashed::get::<u32>(&head), Some(1u32));
<instance::LinkedMap2<instance::DefaultInstance>>::insert(1, 2);
let mut k = [twox_128(b"FinalKeysSome"), twox_128(b"LinkedMap2")].concat();
k.extend(1u32.using_encoded(twox_128).to_vec());
assert_eq!(unhashed::get::<u32>(&k), Some(2u32));
<instance::DoubleMap<instance::DefaultInstance>>::insert(&1, &2, &3);
let mut k = [twox_128(b"FinalKeysSome"), twox_128(b"DoubleMap")].concat();
k.extend(1u32.using_encoded(blake2_256).to_vec());
k.extend(2u32.using_encoded(blake2_256).to_vec());
k.extend(1u32.using_encoded(blake2_128_concat));
k.extend(2u32.using_encoded(blake2_128_concat));
assert_eq!(unhashed::get::<u32>(&k), Some(3u32));
assert_eq!(&k[..32], &<instance::DoubleMap<instance::DefaultInstance>>::final_prefix());
<instance::DoubleMap2<instance::DefaultInstance>>::insert(&1, &2, &3);
let mut k = [twox_128(b"FinalKeysSome"), twox_128(b"DoubleMap2")].concat();
k.extend(1u32.using_encoded(twox_128).to_vec());
k.extend(2u32.using_encoded(blake2_128).to_vec());
k.extend(1u32.using_encoded(twox_64_concat));
k.extend(2u32.using_encoded(twox_64_concat));
assert_eq!(unhashed::get::<u32>(&k), Some(3u32));
assert_eq!(&k[..32], &<instance::DoubleMap2<instance::DefaultInstance>>::final_prefix());
});
@@ -192,41 +170,27 @@ fn final_keys_instance_2() {
<instance::Map<instance::Instance2>>::insert(1, 2);
let mut k = [twox_128(b"Instance2FinalKeysSome"), twox_128(b"Map")].concat();
k.extend(1u32.using_encoded(blake2_256).to_vec());
k.extend(1u32.using_encoded(blake2_128_concat));
assert_eq!(unhashed::get::<u32>(&k), Some(2u32));
assert_eq!(&k[..32], &<instance::Map<instance::Instance2>>::final_prefix());
<instance::Map2<instance::Instance2>>::insert(1, 2);
let mut k = [twox_128(b"Instance2FinalKeysSome"), twox_128(b"Map2")].concat();
k.extend(1u32.using_encoded(twox_128).to_vec());
k.extend(1u32.using_encoded(twox_64_concat));
assert_eq!(unhashed::get::<u32>(&k), Some(2u32));
assert_eq!(&k[..32], &<instance::Map2<instance::Instance2>>::final_prefix());
let head = [twox_128(b"Instance2FinalKeysSome"), twox_128(b"HeadOfLinkedMap")].concat();
assert_eq!(unhashed::get::<u32>(&head), None);
<instance::LinkedMap<instance::Instance2>>::insert(1, 2);
let mut k = [twox_128(b"Instance2FinalKeysSome"), twox_128(b"LinkedMap")].concat();
k.extend(1u32.using_encoded(blake2_256).to_vec());
assert_eq!(unhashed::get::<u32>(&k), Some(2u32));
assert_eq!(unhashed::get::<u32>(&head), Some(1u32));
<instance::LinkedMap2<instance::Instance2>>::insert(1, 2);
let mut k = [twox_128(b"Instance2FinalKeysSome"), twox_128(b"LinkedMap2")].concat();
k.extend(1u32.using_encoded(twox_128).to_vec());
assert_eq!(unhashed::get::<u32>(&k), Some(2u32));
<instance::DoubleMap<instance::Instance2>>::insert(&1, &2, &3);
let mut k = [twox_128(b"Instance2FinalKeysSome"), twox_128(b"DoubleMap")].concat();
k.extend(1u32.using_encoded(blake2_256).to_vec());
k.extend(2u32.using_encoded(blake2_256).to_vec());
k.extend(1u32.using_encoded(blake2_128_concat));
k.extend(2u32.using_encoded(blake2_128_concat));
assert_eq!(unhashed::get::<u32>(&k), Some(3u32));
assert_eq!(&k[..32], &<instance::DoubleMap<instance::Instance2>>::final_prefix());
<instance::DoubleMap2<instance::Instance2>>::insert(&1, &2, &3);
let mut k = [twox_128(b"Instance2FinalKeysSome"), twox_128(b"DoubleMap2")].concat();
k.extend(1u32.using_encoded(twox_128).to_vec());
k.extend(2u32.using_encoded(blake2_128).to_vec());
k.extend(1u32.using_encoded(twox_64_concat));
k.extend(2u32.using_encoded(twox_64_concat));
assert_eq!(unhashed::get::<u32>(&k), Some(3u32));
assert_eq!(&k[..32], &<instance::DoubleMap2<instance::Instance2>>::final_prefix());
});
substrate/frame/support/test/tests/genesisconfig.rs
+1 -1
View File
@@ -25,7 +25,7 @@ frame_support::decl_module! {
frame_support::decl_storage! {
trait Store for Module<T: Trait> as Test {
pub AppendableDM config(t): double_map hasher(blake2_256) u32, hasher(blake2_256) T::BlockNumber => Vec<u32>;
pub AppendableDM config(t): double_map hasher(identity) u32, hasher(identity) T::BlockNumber => Vec<u32>;
}
}
substrate/frame/support/test/tests/instance.rs
+10 -57
View File
@@ -23,7 +23,7 @@ use frame_support::{
DecodeDifferent, StorageMetadata, StorageEntryModifier, StorageEntryType, DefaultByteGetter,
StorageEntryMetadata, StorageHasher,
},
StorageValue, StorageMap, StorageLinkedMap, StorageDoubleMap,
StorageValue, StorageMap, StorageDoubleMap,
};
use sp_inherents::{ProvideInherent, InherentData, InherentIdentifier, MakeFatalError};
use sp_core::{H256, sr25519};
@@ -67,8 +67,7 @@ mod module1 {
T::BlockNumber: From<u32> + std::fmt::Display
{
pub Value config(value): T::GenericType;
pub Map: map hasher(blake2_256) u32 => u64;
pub LinkedMap: linked_map hasher(blake2_256) u32 => u64;
pub Map: map hasher(identity) u32 => u64;
}
add_extra_genesis {
@@ -136,9 +135,8 @@ mod module2 {
frame_support::decl_storage! {
trait Store for Module<T: Trait<I>, I: Instance=DefaultInstance> as Module2 {
pub Value config(value): T::Amount;
pub Map config(map): map hasher(blake2_256) u64 => u64;
pub LinkedMap config(linked_map): linked_map hasher(blake2_256) u64 => Vec<u8>;
pub DoubleMap config(double_map): double_map hasher(blake2_256) u64, hasher(blake2_256) u64 => u64;
pub Map config(map): map hasher(identity) u64 => u64;
pub DoubleMap config(double_map): double_map hasher(identity) u64, hasher(identity) u64 => u64;
}
}
@@ -285,13 +283,11 @@ fn new_test_ext() -> sp_io::TestExternalities {
module2: Some(module2::GenesisConfig {
value: 4,
map: vec![(0, 0)],
linked_map: vec![(0, vec![0])],
double_map: vec![(0, 0, 0)],
}),
module2_Instance1: Some(module2::GenesisConfig {
value: 4,
map: vec![(0, 0)],
linked_map: vec![(0, vec![0])],
double_map: vec![(0, 0, 0)],
}),
module2_Instance2: None,
@@ -314,17 +310,13 @@ fn storage_instance_independence() {
module2::Map::<module2::Instance1>::insert(0, 0);
module2::Map::<module2::Instance2>::insert(0, 0);
module2::Map::<module2::Instance3>::insert(0, 0);
module2::LinkedMap::<module2::DefaultInstance>::insert::<_, Vec<u8>>(0, vec![]);
module2::LinkedMap::<module2::Instance1>::insert::<_, Vec<u8>>(0, vec![]);
module2::LinkedMap::<module2::Instance2>::insert::<_, Vec<u8>>(0, vec![]);
module2::LinkedMap::<module2::Instance3>::insert::<_, Vec<u8>>(0, vec![]);
module2::DoubleMap::<module2::DefaultInstance>::insert(&0, &0, &0);
module2::DoubleMap::<module2::Instance1>::insert(&0, &0, &0);
module2::DoubleMap::<module2::Instance2>::insert(&0, &0, &0);
module2::DoubleMap::<module2::Instance3>::insert(&0, &0, &0);
});
// 16 storage values + 4 linked_map head.
assert_eq!(storage.top.len(), 16 + 4);
// 12 storage values.
assert_eq!(storage.top.len(), 12);
}
#[test]
@@ -332,7 +324,6 @@ fn storage_with_instance_basic_operation() {
new_test_ext().execute_with(|| {
type Value = module2::Value<Runtime, module2::Instance1>;
type Map = module2::Map<module2::Instance1>;
type LinkedMap = module2::LinkedMap<module2::Instance1>;
type DoubleMap = module2::DoubleMap<module2::Instance1>;
assert_eq!(Value::exists(), true);
@@ -360,26 +351,6 @@ fn storage_with_instance_basic_operation() {
assert_eq!(Map::contains_key(key), false);
assert_eq!(Map::get(key), 0);
assert_eq!(LinkedMap::contains_key(0), true);
assert_eq!(LinkedMap::contains_key(key), false);
LinkedMap::insert(key, vec![1]);
assert_eq!(LinkedMap::enumerate().count(), 2);
assert_eq!(LinkedMap::get(key), vec![1]);
assert_eq!(LinkedMap::take(key), vec![1]);
assert_eq!(LinkedMap::enumerate().count(), 1);
assert_eq!(LinkedMap::get(key), vec![]);
LinkedMap::mutate(key, |a| *a=vec![2]);
assert_eq!(LinkedMap::enumerate().count(), 2);
assert_eq!(LinkedMap::get(key), vec![2]);
LinkedMap::remove(key);
assert_eq!(LinkedMap::enumerate().count(), 1);
assert_eq!(LinkedMap::contains_key(key), false);
assert_eq!(LinkedMap::get(key), vec![]);
assert_eq!(LinkedMap::contains_key(key), false);
assert_eq!(LinkedMap::enumerate().count(), 1);
LinkedMap::insert(key, &vec![1]);
assert_eq!(LinkedMap::enumerate().count(), 2);
let key1 = 1;
let key2 = 1;
assert_eq!(DoubleMap::contains_key(&0, &0), true);
@@ -416,10 +387,10 @@ const EXPECTED_METADATA: StorageMetadata = StorageMetadata {
name: DecodeDifferent::Encode("Map"),
modifier: StorageEntryModifier::Default,
ty: StorageEntryType::Map {
hasher: StorageHasher::Blake2_256,
hasher: StorageHasher::Identity,
key: DecodeDifferent::Encode("u64"),
value: DecodeDifferent::Encode("u64"),
is_linked: false,
unused: false,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(
@@ -430,30 +401,12 @@ const EXPECTED_METADATA: StorageMetadata = StorageMetadata {
),
documentation: DecodeDifferent::Encode(&[]),
},
StorageEntryMetadata {
name: DecodeDifferent::Encode("LinkedMap"),
modifier: StorageEntryModifier::Default,
ty: StorageEntryType::Map {
hasher: StorageHasher::Blake2_256,
key: DecodeDifferent::Encode("u64"),
value: DecodeDifferent::Encode("Vec<u8>"),
is_linked: true,
},
default: DecodeDifferent::Encode(
DefaultByteGetter(
&module2::__GetByteStructLinkedMap(
std::marker::PhantomData::<(Runtime, module2::Instance2)>
)
)
),
documentation: DecodeDifferent::Encode(&[]),
},
StorageEntryMetadata {
name: DecodeDifferent::Encode("DoubleMap"),
modifier: StorageEntryModifier::Default,
ty: StorageEntryType::DoubleMap {
hasher: StorageHasher::Blake2_256,
key2_hasher: StorageHasher::Blake2_256,
hasher: StorageHasher::Identity,
key2_hasher: StorageHasher::Identity,
key1: DecodeDifferent::Encode("u64"),
key2: DecodeDifferent::Encode("u64"),
value: DecodeDifferent::Encode("u64"),
substrate/frame/support/test/tests/issue2219.rs
+3 -3
View File
@@ -109,7 +109,7 @@ mod module {
} else {
vec![]
}
}): map hasher(blake2_256) Role => Option<RoleParameters<T>>;
}): map hasher(blake2_128_concat) Role => Option<RoleParameters<T>>;
/// the roles members can enter into
pub AvailableRoles get(fn available_roles) build(|config: &GenesisConfig| {
@@ -125,11 +125,11 @@ mod module {
/// actor accounts associated with a role
pub AccountIdsByRole get(fn account_ids_by_role):
map hasher(blake2_256) Role => Vec<T::AccountId>;
map hasher(blake2_128_concat) Role => Vec<T::AccountId>;
/// tokens locked until given block number
pub Bondage get(fn bondage):
map hasher(blake2_256) T::AccountId => T::BlockNumber;
map hasher(blake2_128_concat) T::AccountId => T::BlockNumber;
/// First step before enter a role is registering intent with a new account/key.
/// This is done by sending a role_entry_request() from the new account.