pallet-mmr: handle forks without collisions in offchain storage (#11594)

* pallet-mmr: fix some typos

* pallet-mmr: make the MMR resilient to chain forks

* pallet-mmr: get hash for block that added node

* beefy-mmr: add debug logging

* add explanatory comment

* account for block offset of pallet activation

* add support for finding all nodes added by leaf

* minor improvements

* add helper to return all nodes added to mmr with a leaf append

* simplify leaf_node_index_to_leaf_index

summing the (shifted) differences in peak positions adds up to the (shifted) final position, so
don't need to fold over positions.

* dead fish: this also doesn't work

The idea was to keep a rolling window of `(parent_hash, pos)` leaf
entries in the offchain db, with the window matching the one
that provides `block_num -> block_hash` mappings in `frame_system`.

Once a leaf exits the window it would be "canonicalized" by switching
its offchain db key from `(parent_hash, pos)` to simple `pos`.

This doesn't work however because there's no way to get leaf contents
from offchain db while in runtime context.. so no way to get+clear+set
leaf to change its key in offchain db.

Ideas:
1. move the "canonicalization" logic to offchain worker
2. enhance IndexingApi with "offchain::move(old_key, new_key)"
   This is weird, but correct, deterministic and safe AFAICT, so
   it could be exposed to runtime.

* simplify rightmost_leaf_node_index_from_pos

* minor fix

* move leaf canonicalization to offchain worker

* move storage related code to storage.rs

* on offchain reads use canonic key for old leaves

* fix offchain worker write using canon key

* fix pallet-mmr tests

* add documentation and fix logging

* add offchain mmr canonicalization test

* test canon + generate + verify

* fix pallet-beefy-mmr tests

* implement review suggestions

* improve test

* pallet-mmr: add offchain pruning of forks

* pallet-mmr: improve offchain pruning

Instead of keeping pruning map as single blob in offchain db,
keep individual parent-hash lists with block-num identifier as part
of the offchain key.

Signed-off-by: acatangiu <adrian@parity.io>

* pallet-mmr: improve MMRStore<OffchainStorage>::get()

Do the math and retrieve node using correct (canon or non-canon)
offchain db key, instead of blindly looking in both canon and non-canon
offchain db locations for each node.

Still fallback on looking at both if for any reason it's not where
expected.

Signed-off-by: acatangiu <adrian@parity.io>

* pallet-mmr: storage: improve logs

* fix tests: correctly persist overlay

runtime indexing API works on overlay, whereas offchain context
bypasses overlay, so for loops > canon-window, canon would fail.

* pallet-mmr: fix numeric typo in test

* add comment around LeafData requirements

Signed-off-by: acatangiu <adrian@parity.io>

Co-authored-by: Robert Hambrock <roberthambrock@gmail.com>

substrate/frame/beefy-mmr/src/tests.rs

@@ -44,16 +44,12 @@ pub fn beefy_log(log: ConsensusLog<BeefyId>) -> DigestItem {
 	DigestItem::Consensus(BEEFY_ENGINE_ID, log.encode())
 }
 
-fn offchain_key(pos: usize) -> Vec<u8> {
-	(<Test as pallet_mmr::Config>::INDEXING_PREFIX, pos as u64).encode()
-}
-
-fn read_mmr_leaf(ext: &mut TestExternalities, index: usize) -> MmrLeaf {
+fn read_mmr_leaf(ext: &mut TestExternalities, key: Vec<u8>) -> MmrLeaf {
 	type Node = pallet_mmr::primitives::DataOrHash<Keccak256, MmrLeaf>;
 	ext.persist_offchain_overlay();
 	let offchain_db = ext.offchain_db();
 	offchain_db
-		.get(&offchain_key(index))
+		.get(&key)
 		.map(|d| Node::decode(&mut &*d).unwrap())
 		.map(|n| match n {
 			Node::Data(d) => d,
@@ -105,12 +101,17 @@ fn should_contain_mmr_digest() {
 
 #[test]
 fn should_contain_valid_leaf_data() {
+	fn node_offchain_key(parent_hash: H256, pos: usize) -> Vec<u8> {
+		(<Test as pallet_mmr::Config>::INDEXING_PREFIX, parent_hash, pos as u64).encode()
+	}
+
 	let mut ext = new_test_ext(vec![1, 2, 3, 4]);
-	ext.execute_with(|| {
+	let parent_hash = ext.execute_with(|| {
 		init_block(1);
+		<frame_system::Pallet<Test>>::parent_hash()
 	});
 
-	let mmr_leaf = read_mmr_leaf(&mut ext, 0);
+	let mmr_leaf = read_mmr_leaf(&mut ext, node_offchain_key(parent_hash, 0));
 	assert_eq!(
 		mmr_leaf,
 		MmrLeaf {
@@ -128,11 +129,12 @@ fn should_contain_valid_leaf_data() {
 	);
 
 	// build second block on top
-	ext.execute_with(|| {
+	let parent_hash = ext.execute_with(|| {
 		init_block(2);
+		<frame_system::Pallet<Test>>::parent_hash()
 	});
 
-	let mmr_leaf = read_mmr_leaf(&mut ext, 1);
+	let mmr_leaf = read_mmr_leaf(&mut ext, node_offchain_key(parent_hash, 1));
 	assert_eq!(
 		mmr_leaf,
 		MmrLeaf {