Refactor away from opaque hashes (#5226)

* System.BlockHash * Fix hash * Introduce K/V iteration in all _concat maps Also move across: - System.Account (blake2_128_concat) - Balances.Locks (twox_64_concat) - ElectionsPhragmen.VotesOf (twox_64_concat) - ElectionsPhragmen.StakeOf (twox_64_concat) - Identity.IdentityOf (twox_64_concat) - Identity.SubsOf (twox_64_concat) - Society.Payouts (twox_64_concat) - Session.NextKeys (twox_64_concat) - Identity.SuperOf (blake2_128_concat) - Session.KeyOwner (blake2_128_concat) - Society.SuspendedCandidates (twox_64_concat) - Society.SuspendedMembers (twox_64_concat) - Society.Vouching (twox_64_concat) - Society.Strikes (twox_64_concat) - System.EventTopics - Balances.Account * Build fixes * Ensure migration happens in correct order * Staking.* * Vesting.* Offences.* * Democracy.* * Babe.* Collective.* * Grandpa.* * Assets.* Benchmark.* Contracts.* Elections.* Asset.* Nicks.* Also introduce real account list * ImOnline.* * Treasury.* * Recovery.* * Final bits. * Docs * Fix one test * Fix test * All passing except the UI tests * Remove linked_map part 1 * Remove linked_map * Some iterator utils for double maps. * Remove old migrations * Introduce tombstone for LinkedMap type * Migration for genesis hash * Fix build * Fix hash * Rename Map is_linked -> unused, keeping backwards compat (#5256) * Update frame/balances/src/lib.rs Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com> * Update frame/elections/src/lib.rs Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com> * Remove old migration code. * Update frame/system/src/lib.rs Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com> * Update bin/node/runtime/src/lib.rs Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com> * Fix hash * fix session migration * Fix watning Co-authored-by: Jaco Greeff <jacogr@gmail.com> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
2026-06-12 17:01:09 +00:00 · 2020-03-16 23:19:53 +01:00
parent 846a9ce8c6
commit af9083f53b
94 changed files with 1111 additions and 2020 deletions
@@ -33,8 +33,8 @@ use proc_macro::TokenStream;
 /// decl_storage! {
 /// 	trait Store for Module<T: Trait> as Example {
 /// 		Foo get(fn foo) config(): u32=12;
-/// 		Bar: map hasher(blake2_256) u32 => u32;
-/// 		pub Zed build(|config| vec![(0, 0)]): linked_map hasher(blake2_256) u32 => u32;
+/// 		Bar: map hasher(identity) u32 => u32;
+/// 		pub Zed build(|config| vec![(0, 0)]): map hasher(identity) u32 => u32;
 /// 	}
 /// }
 /// ```
@@ -70,10 +70,28 @@ use proc_macro::TokenStream;
 ///   And [`StoragePrefixedMap`](../frame_support/storage/trait.StoragePrefixedMap.html).
 ///
 ///   `$hash` representing a choice of hashing algorithms available in the
-///   [`Hashable`](../frame_support/trait.Hashable.html) trait.
+///   [`Hashable`](../frame_support/trait.Hashable.html) trait. You will generally want to use one
+///   of three hashers:
+///   * `blake2_128_concat`: The default, safe choice. Use if you are unsure or don't care. It is
+///     secure against user-tainted keys, fairly fast and memory-efficient and supports
+///     iteration over its keys and values. This must be used if the keys of your map can be
+///     selected *en masse* by untrusted users.
+///   * `twox_64_concat`: This is an insecure hasher and can only be used safely if you know that
+///     the preimages cannot be chosen at will by untrusted users. It is memory-efficient, extremely
+///     performant and supports iteration over its keys and values. You can safely use this is the
+///     key is:
+///     - A (slowly) incrementing index.
+///     - Known to be the result of a cryptographic hash (though `identity` is a better choice here).
+///     - Known to be the public key of a cryptographic key pair in existence.
+///   * `identity`: This is not a hasher at all, and just uses the key material directly. Since it
+///     does no hashing or appending, it's the fastest possible hasher, however, it's also the least
+///     secure. It can be used only if you know that the key will be cryptographically/securely
+///     randomly distributed over the binary encoding space. In most cases this will not be true.
+///     One case where it is true, however, if where the key is itself the result of a cryptographic
+///     hash of some existent data.
 ///
-///   `blake2_256` and `blake2_128_concat` are strong hasher. One should use another hasher
-///   with care, see generator documentation.
+///   Other hashers will tend to be "opaque" and not support iteration over the keys in the
+///   map. It is not recommended to use these.
 ///
 ///   The generator is implemented with:
 ///   * `module_prefix`: $module_prefix
@@ -85,36 +103,6 @@ use proc_macro::TokenStream;
 ///   twox128(module_prefix) ++ twox128(storage_prefix) ++ hasher(encode(key))
 ///   ```
 ///
-/// * Linked map: `Foo: linked_map hasher($hash) type => type`: Implements the
-///   [`StorageLinkedMap`](../frame_support/storage/trait.StorageLinkedMap.html) trait using the
-///   [`StorageLinkedMap generator`](../frame_support/storage/generator/trait.StorageLinkedMap.html).
-///   And [`StoragePrefixedMap`](../frame_support/storage/trait.StoragePrefixedMap.html).
-///
-///   `$hash` representing a choice of hashing algorithms available in the
-///   [`Hashable`](../frame_support/trait.Hashable.html) trait.
-///
-///   `blake2_256` and `blake2_128_concat` are strong hasher. One should use another hasher
-///   with care, see generator documentation.
-///
-///   All key formatting logic can be accessed in a type-agnostic format via the
-///   `KeyFormat` trait, which
-///   is implemented for the storage linked map type as well.
-///
-///   The generator key format is implemented with:
-///   * `module_prefix`: $module_prefix
-///   * `storage_prefix`: storage_name
-///   * `head_prefix`: `"HeadOf" ++ storage_name`
-///   * `Hasher`: $hash
-///
-///   Thus the keys are stored at:
-///   ```nocompile
-///   Twox128(module_prefix) ++ Twox128(storage_prefix) ++ Hasher(encode(key))
-///   ```
-///   and head is stored at:
-///   ```nocompile
-///   Twox128(module_prefix) ++ Twox128(head_prefix)
-///   ```
-///
 /// * Double map: `Foo: double_map hasher($hash1) u32, hasher($hash2) u32 => u32`: Implements the
 ///   [`StorageDoubleMap`](../frame_support/storage/trait.StorageDoubleMap.html) trait using the
 ///   [`StorageDoubleMap generator`](../frame_support/storage/generator/trait.StorageDoubleMap.html).
@@ -124,14 +112,6 @@ use proc_macro::TokenStream;
 ///   [`Hashable`](../frame_support/trait.Hashable.html) trait. They must be chosen with care, see
 ///   generator documentation.
 ///
-///   If the first key is untrusted, a cryptographic `hasher` such as `blake2_256` or
-///   `blake2_128_concat`  must be used.
-///   Otherwise, other values of all storage items can be compromised.
-///
-///   If the second key is untrusted, a cryptographic `hasher` such as `blake2_256` or
-///   `blake2_128_concat` must be used.
-///   Otherwise, other items in storage with the same first key can be compromised.
-///
 ///   The generator is implemented with:
 ///   * `module_prefix`: $module_prefix
 ///   * `storage_prefix`: storage_name
@@ -145,10 +125,16 @@ use proc_macro::TokenStream;
 ///
 /// Supported hashers (ordered from least to best security):
 ///
-/// * `twox_64_concat` - TwoX with 64bit + key concatenated.
+/// * `identity` - Just the unrefined key material. Use only when it is known to be a secure hash
+///   already. The most efficient and iterable over keys.
+/// * `twox_64_concat` - TwoX with 64bit + key concatenated. Use only when an untrusted source
+///   cannot select and insert key values. Very efficient and iterable over keys.
+/// * `blake2_128_concat` - Blake2 with 128bit + key concatenated. Slower but safe to use in all
+///   circumstances. Iterable over keys.
+///
+/// Deprecated hashers, which do not support iteration over keys include:
 /// * `twox_128` - TwoX with 128bit.
 /// * `twox_256` - TwoX with with 256bit.
-/// * `blake2_128_concat` - Blake2 with 128bit + key concatenated.
 /// * `blake2_128` - Blake2 with 128bit.
 /// * `blake2_256` - Blake2 with 256bit.
 ///
@@ -88,7 +88,7 @@ impl BuilderDef {
 						}}
 					},
 					StorageLineTypeDef::Simple(_) => unreachable!(),
-					StorageLineTypeDef::Map(map) | StorageLineTypeDef::LinkedMap(map) => {
+					StorageLineTypeDef::Map(map) => {
 						let key = &map.key;
 						quote!{{
 							#data
@@ -93,7 +93,7 @@ impl GenesisConfigDef {

 			let typ = match &line.storage_type {
 				StorageLineTypeDef::Simple(_) => (*value_type).clone(),
-				StorageLineTypeDef::Map(map) | StorageLineTypeDef::LinkedMap(map) => {
+				StorageLineTypeDef::Map(map) => {
 					let key = &map.key;
 					parse_quote!( Vec<(#key, #value_type)> )
 				},
@@ -40,7 +40,7 @@ pub fn impl_getters(scrate: &TokenStream, def: &DeclStorageDefExt) -> TokenStrea
 					}
 				}
 			},
-			StorageLineTypeDef::Map(map) | StorageLineTypeDef::LinkedMap(map) => {
+			StorageLineTypeDef::Map(map) => {
 				let key = &map.key;
 				let value = &map.value;
 				quote!{
@@ -41,20 +41,7 @@ fn storage_line_metadata_type(scrate: &TokenStream, line: &StorageLineDefExt) ->
 					hasher: #scrate::metadata::#hasher,
 					key: #scrate::metadata::DecodeDifferent::Encode(#key),
 					value: #scrate::metadata::DecodeDifferent::Encode(#value_type),
-					is_linked: false,
-				}
-			}
-		},
-		StorageLineTypeDef::LinkedMap(map) => {
-			let hasher = map.hasher.into_metadata();
-			let key = &map.key;
-			let key = clean_type_string(&quote!(#key).to_string());
-			quote!{
-				#scrate::metadata::StorageEntryType::Map {
-					hasher: #scrate::metadata::#hasher,
-					key: #scrate::metadata::DecodeDifferent::Encode(#key),
-					value: #scrate::metadata::DecodeDifferent::Encode(#value_type),
-					is_linked: true,
+					unused: false,
 				}
 			}
 		},
@@ -235,10 +235,6 @@ impl StorageLineDefExt {
 				ext::type_contains_ident(&map.key, &def.module_runtime_generic)
 					|| ext::type_contains_ident(&map.value, &def.module_runtime_generic)
 			}
-			StorageLineTypeDef::LinkedMap(map) => {
-				ext::type_contains_ident(&map.key, &def.module_runtime_generic)
-					|| ext::type_contains_ident(&map.value, &def.module_runtime_generic)
-			}
 			StorageLineTypeDef::DoubleMap(map) => {
 				ext::type_contains_ident(&map.key1, &def.module_runtime_generic)
 					|| ext::type_contains_ident(&map.key2, &def.module_runtime_generic)
@@ -249,7 +245,6 @@ impl StorageLineDefExt {
 		let query_type = match &storage_def.storage_type {
 			StorageLineTypeDef::Simple(value) => value.clone(),
 			StorageLineTypeDef::Map(map) => map.value.clone(),
-			StorageLineTypeDef::LinkedMap(map) => map.value.clone(),
 			StorageLineTypeDef::DoubleMap(map) => map.value.clone(),
 		};
 		let is_option = ext::extract_type_option(&query_type).is_some();
@@ -291,10 +286,6 @@ impl StorageLineDefExt {
 				let key = &map.key;
 				quote!( StorageMap<#key, #value_type> )
 			},
-			StorageLineTypeDef::LinkedMap(map) => {
-				let key = &map.key;
-				quote!( StorageLinkedMap<#key, #value_type> )
-			},
 			StorageLineTypeDef::DoubleMap(map) => {
 				let key1 = &map.key1;
 				let key2 = &map.key2;
@@ -336,7 +327,6 @@ impl StorageLineDefExt {

 pub enum StorageLineTypeDef {
 	Map(MapDef),
-	LinkedMap(MapDef),
 	DoubleMap(DoubleMapDef),
 	Simple(syn::Type),
 }
@@ -372,6 +362,7 @@ pub enum HasherKind {
 	Twox256,
 	Twox128,
 	Twox64Concat,
+	Identity,
 }

 impl HasherKind {
@@ -383,6 +374,7 @@ impl HasherKind {
 			HasherKind::Twox256 => quote!( Twox256 ),
 			HasherKind::Twox128 => quote!( Twox128 ),
 			HasherKind::Twox64Concat => quote!( Twox64Concat ),
+			HasherKind::Identity => quote!( Identity ),
 		}
 	}

@@ -394,6 +386,7 @@ impl HasherKind {
 			HasherKind::Twox256 => quote!( StorageHasher::Twox256 ),
 			HasherKind::Twox128 => quote!( StorageHasher::Twox128 ),
 			HasherKind::Twox64Concat => quote!( StorageHasher::Twox64Concat ),
+			HasherKind::Identity => quote!( StorageHasher::Identity ),
 		}
 	}
 }
@@ -420,7 +413,6 @@ pub fn decl_storage_impl(input: proc_macro::TokenStream) -> proc_macro::TokenStr
 		use #scrate::{
 			StorageValue as _,
 			StorageMap as _,
-			StorageLinkedMap as _,
 			StorageDoubleMap as _,
 			StoragePrefixedMap as _,
 		};
@@ -27,15 +27,18 @@ mod keyword {
 	syn::custom_keyword!(build);
 	syn::custom_keyword!(get);
 	syn::custom_keyword!(map);
-	syn::custom_keyword!(linked_map);
 	syn::custom_keyword!(double_map);
-	syn::custom_keyword!(blake2_256);
-	syn::custom_keyword!(blake2_128);
+	syn::custom_keyword!(opaque_blake2_256);
+	syn::custom_keyword!(opaque_blake2_128);
 	syn::custom_keyword!(blake2_128_concat);
-	syn::custom_keyword!(twox_256);
-	syn::custom_keyword!(twox_128);
+	syn::custom_keyword!(opaque_twox_256);
+	syn::custom_keyword!(opaque_twox_128);
 	syn::custom_keyword!(twox_64_concat);
+	syn::custom_keyword!(identity);
 	syn::custom_keyword!(hasher);
+	syn::custom_keyword!(tainted);
+	syn::custom_keyword!(natural);
+	syn::custom_keyword!(prehashed);
 }

 /// Specific `Opt` to implement structure with optional parsing
@@ -194,7 +197,6 @@ impl_parse_for_opt!(DeclStorageBuild => keyword::build);
 #[derive(ToTokens, Debug)]
 enum DeclStorageType {
 	Map(DeclStorageMap),
-	LinkedMap(DeclStorageLinkedMap),
 	DoubleMap(DeclStorageDoubleMap),
 	Simple(syn::Type),
 }
@@ -203,8 +205,6 @@ impl syn::parse::Parse for DeclStorageType {
 	fn parse(input: syn::parse::ParseStream) -> syn::parse::Result<Self> {
 		if input.peek(keyword::map) {
 			Ok(Self::Map(input.parse()?))
-		} else if input.peek(keyword::linked_map) {
-			Ok(Self::LinkedMap(input.parse()?))
 		} else if input.peek(keyword::double_map) {
 			Ok(Self::DoubleMap(input.parse()?))
 		} else {
@@ -222,15 +222,6 @@ struct DeclStorageMap {
 	pub value: syn::Type,
 }

-#[derive(Parse, ToTokens, Debug)]
-struct DeclStorageLinkedMap {
-	pub map_keyword: keyword::linked_map,
-	pub hasher: Opt<SetHasher>,
-	pub key: syn::Type,
-	pub ass_keyword: Token![=>],
-	pub value: syn::Type,
-}
-
 #[derive(Parse, ToTokens, Debug)]
 struct DeclStorageDoubleMap {
 	pub map_keyword: keyword::double_map,
@@ -245,29 +236,38 @@ struct DeclStorageDoubleMap {

 #[derive(ToTokens, Debug)]
 enum Hasher {
-	Blake2_256(keyword::blake2_256),
-	Blake2_128(keyword::blake2_128),
+	Blake2_256(keyword::opaque_blake2_256),
+	Blake2_128(keyword::opaque_blake2_128),
 	Blake2_128Concat(keyword::blake2_128_concat),
-	Twox256(keyword::twox_256),
-	Twox128(keyword::twox_128),
+	Twox256(keyword::opaque_twox_256),
+	Twox128(keyword::opaque_twox_128),
 	Twox64Concat(keyword::twox_64_concat),
+	Identity(keyword::identity),
 }

 impl syn::parse::Parse for Hasher {
 	fn parse(input: syn::parse::ParseStream) -> syn::parse::Result<Self> {
 		let lookahead = input.lookahead1();
-		if lookahead.peek(keyword::blake2_256) {
+		if lookahead.peek(keyword::opaque_blake2_256) {
 			Ok(Self::Blake2_256(input.parse()?))
-		} else if lookahead.peek(keyword::blake2_128) {
+		} else if lookahead.peek(keyword::opaque_blake2_128) {
 			Ok(Self::Blake2_128(input.parse()?))
 		} else if lookahead.peek(keyword::blake2_128_concat) {
 			Ok(Self::Blake2_128Concat(input.parse()?))
-		} else if lookahead.peek(keyword::twox_256) {
+		} else if lookahead.peek(keyword::opaque_twox_256) {
 			Ok(Self::Twox256(input.parse()?))
-		} else if lookahead.peek(keyword::twox_128) {
+		} else if lookahead.peek(keyword::opaque_twox_128) {
 			Ok(Self::Twox128(input.parse()?))
 		} else if lookahead.peek(keyword::twox_64_concat) {
 			Ok(Self::Twox64Concat(input.parse()?))
+		} else if lookahead.peek(keyword::identity) {
+			Ok(Self::Identity(input.parse()?))
+		} else if lookahead.peek(keyword::tainted) {
+			Ok(Self::Blake2_128Concat(input.parse()?))
+		} else if lookahead.peek(keyword::natural) {
+			Ok(Self::Twox64Concat(input.parse()?))
+		} else if lookahead.peek(keyword::prehashed) {
+			Ok(Self::Identity(input.parse()?))
 		} else {
 			Err(lookahead.error())
 		}
@@ -313,6 +313,7 @@ impl From<Hasher> for super::HasherKind {
 			Hasher::Twox256(_) => super::HasherKind::Twox256,
 			Hasher::Twox128(_) => super::HasherKind::Twox128,
 			Hasher::Twox64Concat(_) => super::HasherKind::Twox64Concat,
+			Hasher::Identity(_) => super::HasherKind::Identity,
 		}
 	}
 }
@@ -464,7 +465,7 @@ fn parse_storage_line_defs(
 		let span = line.storage_type.span();
 		let no_hasher_error = || syn::Error::new(
 			span,
-			"Default hasher has been removed, use explicit hasher(blake2_256) instead."
+			"Default hasher has been removed, use explicit hasher(blake2_128_concat) instead."
 		);

 		let storage_type = match line.storage_type {
@@ -475,13 +476,6 @@ fn parse_storage_line_defs(
 					value: map.value,
 				}
 			),
-			DeclStorageType::LinkedMap(map) => super::StorageLineTypeDef::LinkedMap(
-				super::MapDef {
-					hasher: map.hasher.inner.ok_or_else(no_hasher_error)?.into(),
-					key: map.key,
-					value: map.value,
-				}
-			),
 			DeclStorageType::DoubleMap(map) => super::StorageLineTypeDef::DoubleMap(
 				super::DoubleMapDef {
 					hasher1: map.hasher1.inner.ok_or_else(no_hasher_error)?.into(),
@@ -158,47 +158,6 @@ pub fn decl_and_impl(scrate: &TokenStream, def: &DeclStorageDefExt) -> TokenStre
 					}
 				)
 			},
-			StorageLineTypeDef::LinkedMap(map) => {
-				let hasher = map.hasher.to_storage_hasher_struct();
-
-				let head_prefix_str = syn::LitStr::new(
-					&format!("HeadOf{}", line.name.to_string()),
-					line.name.span(),
-				);
-
-				quote!(
-					impl<#impl_trait> #scrate::#storage_generator_trait for #storage_struct
-					#optional_storage_where_clause
-					{
-						type Query = #query_type;
-						type KeyFormat = Self;
-
-						fn from_optional_value_to_query(v: Option<#value_type>) -> Self::Query {
-							#from_optional_value_to_query
-						}
-
-						fn from_query_to_optional_value(v: Self::Query) -> Option<#value_type> {
-							#from_query_to_optional_value
-						}
-					}
-
-					impl<#impl_trait> #scrate::storage::generator::LinkedMapKeyFormat for #storage_struct {
-						type Hasher = #scrate::#hasher;
-
-						fn module_prefix() -> &'static [u8] {
-							#instance_or_inherent::PREFIX.as_bytes()
-						}
-
-						fn storage_prefix() -> &'static [u8] {
-							#storage_name_str.as_bytes()
-						}
-
-						fn head_prefix() -> &'static [u8] {
-							#head_prefix_str.as_bytes()
-						}
-					}
-				)
-			},
 			StorageLineTypeDef::DoubleMap(map) => {
 				let hasher1 = map.hasher1.to_storage_hasher_struct();
 				let hasher2 = map.hasher2.to_storage_hasher_struct();