pallet-mmr: handle forks without collisions in offchain storage (#11594)

* pallet-mmr: fix some typos

* pallet-mmr: make the MMR resilient to chain forks

* pallet-mmr: get hash for block that added node

* beefy-mmr: add debug logging

* add explanatory comment

* account for block offset of pallet activation

* add support for finding all nodes added by leaf

* minor improvements

* add helper to return all nodes added to mmr with a leaf append

* simplify leaf_node_index_to_leaf_index

summing the (shifted) differences in peak positions adds up to the (shifted) final position, so
don't need to fold over positions.

* dead fish: this also doesn't work

The idea was to keep a rolling window of `(parent_hash, pos)` leaf
entries in the offchain db, with the window matching the one
that provides `block_num -> block_hash` mappings in `frame_system`.

Once a leaf exits the window it would be "canonicalized" by switching
its offchain db key from `(parent_hash, pos)` to simple `pos`.

This doesn't work however because there's no way to get leaf contents
from offchain db while in runtime context.. so no way to get+clear+set
leaf to change its key in offchain db.

Ideas:
1. move the "canonicalization" logic to offchain worker
2. enhance IndexingApi with "offchain::move(old_key, new_key)"
   This is weird, but correct, deterministic and safe AFAICT, so
   it could be exposed to runtime.

* simplify rightmost_leaf_node_index_from_pos

* minor fix

* move leaf canonicalization to offchain worker

* move storage related code to storage.rs

* on offchain reads use canonic key for old leaves

* fix offchain worker write using canon key

* fix pallet-mmr tests

* add documentation and fix logging

* add offchain mmr canonicalization test

* test canon + generate + verify

* fix pallet-beefy-mmr tests

* implement review suggestions

* improve test

* pallet-mmr: add offchain pruning of forks

* pallet-mmr: improve offchain pruning

Instead of keeping pruning map as single blob in offchain db,
keep individual parent-hash lists with block-num identifier as part
of the offchain key.

Signed-off-by: acatangiu <adrian@parity.io>

* pallet-mmr: improve MMRStore<OffchainStorage>::get()

Do the math and retrieve node using correct (canon or non-canon)
offchain db key, instead of blindly looking in both canon and non-canon
offchain db locations for each node.

Still fallback on looking at both if for any reason it's not where
expected.

Signed-off-by: acatangiu <adrian@parity.io>

* pallet-mmr: storage: improve logs

* fix tests: correctly persist overlay

runtime indexing API works on overlay, whereas offchain context
bypasses overlay, so for loops > canon-window, canon would fail.

* pallet-mmr: fix numeric typo in test

* add comment around LeafData requirements

Signed-off-by: acatangiu <adrian@parity.io>

Co-authored-by: Robert Hambrock <roberthambrock@gmail.com>

substrate/primitives/merkle-mountain-range/src/lib.rs

@@ -81,7 +81,7 @@ pub struct Proof<Hash> {
 
 /// A full leaf content stored in the offchain-db.
 pub trait FullLeaf: Clone + PartialEq + fmt::Debug {
-	/// Encode the leaf either in it's full or compact form.
+	/// Encode the leaf either in its full or compact form.
 	///
 	/// NOTE the encoding returned here MUST be `Decode`able into `FullLeaf`.
 	fn using_encoded<R, F: FnOnce(&[u8]) -> R>(&self, f: F, compact: bool) -> R;
@@ -167,18 +167,18 @@ impl EncodableOpaqueLeaf {
 	}
 }
 
-/// An element representing either full data or it's hash.
+/// An element representing either full data or its hash.
 ///
 /// See [Compact] to see how it may be used in practice to reduce the size
 /// of proofs in case multiple [LeafDataProvider]s are composed together.
 /// This is also used internally by the MMR to differentiate leaf nodes (data)
 /// and inner nodes (hashes).
 ///
-/// [DataOrHash::hash] method calculates the hash of this element in it's compact form,
+/// [DataOrHash::hash] method calculates the hash of this element in its compact form,
 /// so should be used instead of hashing the encoded form (which will always be non-compact).
 #[derive(RuntimeDebug, Clone, PartialEq)]
 pub enum DataOrHash<H: traits::Hash, L> {
-	/// Arbitrary data in it's full form.
+	/// Arbitrary data in its full form.
 	Data(L),
 	/// A hash of some data.
 	Hash(H::Output),
@@ -339,7 +339,7 @@ where
 	A: FullLeaf,
 	B: FullLeaf,
 {
-	/// Retrieve a hash of this item in it's compact form.
+	/// Retrieve a hash of this item in its compact form.
 	pub fn hash(&self) -> H::Output {
 		self.using_encoded(<H as traits::Hash>::hash, true)
 	}
@@ -447,7 +447,7 @@ sp_api::decl_runtime_apis! {
 		/// Note this function does not require any on-chain storage - the
 		/// proof is verified against given MMR root hash.
 		///
-		/// The leaf data is expected to be encoded in it's compact form.
+		/// The leaf data is expected to be encoded in its compact form.
 		fn verify_proof_stateless(root: Hash, leaf: EncodableOpaqueLeaf, proof: Proof<Hash>)
 			-> Result<(), Error>;