substrate-test-runtime migrated to "pure" frame runtime (#13737)

* substrate-test-runtime migrated to pure-frame based

* test block builder: helpers added

* simple renaming

* basic_authorship test adjusted

* block_building storage_proof test adjusted

* babe: tests: should_panic expected added

* babe: tests adjusted

ConsensusLog::NextEpochData is now added by pallet_babe as
pallet_babe::SameAuthoritiesForever trigger is used in runtime config.

* beefy: tests adjusted

test-substrate-runtime is now using frame::executive to finalize the
block. during finalization the digests stored during block execution are
checked against header digests:
https://github.com/paritytech/substrate/blob/91bb2d29ca905599098a5b35eaf24867c4fbd60a/frame/executive/src/lib.rs#L585-L591

It makes impossible to directly manipulate header's digets, w/o
depositing logs into system pallet storage `Digest<T: Config>`.

Instead of this dedicated extrinsic allowing to store logs items
(MmrRoot / AuthoritiesChange) is used.

* grandpa: tests adjusted

test-substrate-runtime is now using frame::executive to finalize the
block. during finalization the digest logs stored during block execution are
checked against header digest logs:
https://github.com/paritytech/substrate/blob/91bb2d29ca905599098a5b35eaf24867c4fbd60a/frame/executive/src/lib.rs#L585-L591

It makes impossible to directly manipulate header's digets, w/o
depositing logs into system pallet storage `Digest<T: Config>`.

Instead of this dedicated extrinsic allowing to store logs items
(ScheduledChange / ForcedChange and DigestItem::Other) is used.

* network:bitswap: test adjusted

The size of unchecked extrinsic was increased. The pattern used in test will
be placed at the end of scale-encoded buffer.

* runtime apis versions adjusted

* storage keys used in runtime adjusted

* wasm vs native tests removed

* rpc tests: adjusted

Transfer transaction processing was slightly improved, test was
adjusted.

* tests: sizes adjusted

Runtime extrinsic size was increased. Size of data read during block
execution was also increased due to usage of new pallets in runtime.

Sizes were adjusted in tests.

* cargo.lock update

cargo update -p substrate-test-runtime -p substrate-test-runtime-client

* warnings fixed

* builders cleanup: includes / std

* extrinsic validation cleanup

* txpool: benches performance fixed

* fmt

* spelling

* Apply suggestions from code review

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Apply code review suggestions

* Apply code review suggestions

* get rid of 1063 const

* renaming: UncheckedExtrinsic -> Extrinsic

* test-utils-runtime: further step to pure-frame

* basic-authorship: tests OK

* CheckSubstrateCall added + tests fixes

* test::Transfer call removed

* priority / propagate / no sudo+root-testing

* fixing warnings + format

* cleanup: build2/nonce + format

* final tests fixes

all tests are passing

* logs/comments removal

* should_not_accept_old_signatures test removed

* make txpool benches work again

* Cargo.lock reset

* format

* sudo hack removed

* txpool benches fix+cleanup

* .gitignore reverted

* rebase fixing + unsigned cleanup

* Cargo.toml/Cargo.lock cleanup

* force-debug feature removed

* mmr tests fixed

* make cargo-clippy happy

* network sync test uses unsigned extrinsic

* cleanup

* ".git/.scripts/commands/fmt/fmt.sh"

* push_storage_change signed call remove

* GenesisConfig cleanup

* fix

* fix

* GenesisConfig simplified

* storage_keys_works: reworked

* storage_keys_works: expected keys in vec

* storage keys list moved to substrate-test-runtime

* substrate-test: some sanity tests + GenesisConfigBuilder rework

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <git@kchr.de>

* Apply suggestions from code review

* Review suggestions

* fix

* fix

* beefy: generate_blocks_and_sync block_num sync with actaul value

* Apply suggestions from code review

Co-authored-by: Davide Galassi <davxy@datawok.net>

* Update test-utils/runtime/src/genesismap.rs

Co-authored-by: Davide Galassi <davxy@datawok.net>

* cargo update -p sc-rpc -p sc-transaction-pool

* Review suggestions

* fix

* doc added

* slot_duration adjusted for Babe::slot_duration

* small doc fixes

* array_bytes::hex used instead of hex

* tiny -> medium name fix

* Apply suggestions from code review

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

* TransferData::try_from_unchecked_extrinsic -> try_from

* Update Cargo.lock

---------

Co-authored-by: parity-processbot <>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

substrate/client/rpc/src/author/tests.rs

@@ -37,10 +37,11 @@ use sp_core::{
 	H256,
 };
 use sp_keystore::{testing::MemoryKeystore, Keystore};
+use sp_runtime::Perbill;
 use std::sync::Arc;
 use substrate_test_runtime_client::{
 	self,
-	runtime::{Block, Extrinsic, SessionKeys, Transfer},
+	runtime::{Block, Extrinsic, ExtrinsicBuilder, SessionKeys, Transfer},
 	AccountKeyring, Backend, Client, DefaultTestClientBuilderExt, TestClientBuilderExt,
 };
 
@@ -51,7 +52,7 @@ fn uxt(sender: AccountKeyring, nonce: u64) -> Extrinsic {
 		from: sender.into(),
 		to: AccountKeyring::Bob.into(),
 	};
-	tx.into_signed_tx()
+	ExtrinsicBuilder::new_transfer(tx).build()
 }
 
 type FullTransactionPool = BasicPool<FullChainApi<Client<Backend>, Block>, Block>;
@@ -111,7 +112,13 @@ async fn author_submit_transaction_should_not_cause_error() {
 #[tokio::test]
 async fn author_should_watch_extrinsic() {
 	let api = TestSetup::into_rpc();
-	let xt = to_hex(&uxt(AccountKeyring::Alice, 0).encode(), true);
+	let xt = to_hex(
+		&ExtrinsicBuilder::new_call_with_priority(0)
+			.signer(AccountKeyring::Alice.into())
+			.build()
+			.encode(),
+		true,
+	);
 
 	let mut sub = api.subscribe("author_submitAndWatchExtrinsic", [xt]).await.unwrap();
 	let (tx, sub_id) = timeout_secs(10, sub.next::<TransactionStatus<H256, Block>>())
@@ -125,15 +132,11 @@ async fn author_should_watch_extrinsic() {
 
 	// Replace the extrinsic and observe the subscription is notified.
 	let (xt_replacement, xt_hash) = {
-		let tx = Transfer {
-			amount: 5,
-			nonce: 0,
-			from: AccountKeyring::Alice.into(),
-			to: AccountKeyring::Bob.into(),
-		};
-		let tx = tx.into_signed_tx().encode();
+		let tx = ExtrinsicBuilder::new_call_with_priority(1)
+			.signer(AccountKeyring::Alice.into())
+			.build()
+			.encode();
 		let hash = blake2_256(&tx);
-
 		(to_hex(&tx, true), hash)
 	};
 
@@ -152,10 +155,10 @@ async fn author_should_watch_extrinsic() {
 async fn author_should_return_watch_validation_error() {
 	const METHOD: &'static str = "author_submitAndWatchExtrinsic";
 
+	let invalid_xt = ExtrinsicBuilder::new_fill_block(Perbill::from_percent(100)).build();
+
 	let api = TestSetup::into_rpc();
-	let failed_sub = api
-		.subscribe(METHOD, [to_hex(&uxt(AccountKeyring::Alice, 179).encode(), true)])
-		.await;
+	let failed_sub = api.subscribe(METHOD, [to_hex(&invalid_xt.encode(), true)]).await;
 
 	assert_matches!(
 		failed_sub,

substrate/client/rpc/src/dev/tests.rs

@@ -28,6 +28,22 @@ async fn block_stats_work() {
 	let api = <Dev<Block, _>>::new(client.clone(), DenyUnsafe::No).into_rpc();
 
 	let block = client.new_block(Default::default()).unwrap().build().unwrap().block;
+
+	let (expected_witness_len, expected_witness_compact_len, expected_block_len) = {
+		let genesis_hash = client.chain_info().genesis_hash;
+		let mut runtime_api = client.runtime_api();
+		runtime_api.record_proof();
+		runtime_api.execute_block(genesis_hash, block.clone()).unwrap();
+		let witness = runtime_api.extract_proof().unwrap();
+		let pre_root = *client.header(genesis_hash).unwrap().unwrap().state_root();
+
+		(
+			witness.clone().encoded_size() as u64,
+			witness.into_compact_proof::<HasherOf<Block>>(pre_root).unwrap().encoded_size() as u64,
+			block.encoded_size() as u64,
+		)
+	};
+
 	client.import(BlockOrigin::Own, block).await.unwrap();
 
 	// Can't gather stats for a block without a parent.
@@ -43,9 +59,9 @@ async fn block_stats_work() {
 			.await
 			.unwrap(),
 		Some(BlockStats {
-			witness_len: 630,
-			witness_compact_len: 534,
-			block_len: 99,
+			witness_len: expected_witness_len,
+			witness_compact_len: expected_witness_compact_len,
+			block_len: expected_block_len,
 			num_extrinsics: 0,
 		}),
 	);

substrate/client/rpc/src/state/tests.rs

@@ -29,9 +29,11 @@ use sc_block_builder::BlockBuilderProvider;
 use sc_rpc_api::DenyUnsafe;
 use sp_consensus::BlockOrigin;
 use sp_core::{hash::H256, storage::ChildInfo};
-use sp_io::hashing::blake2_256;
 use std::sync::Arc;
-use substrate_test_runtime_client::{prelude::*, runtime};
+use substrate_test_runtime_client::{
+	prelude::*,
+	runtime::{ExtrinsicBuilder, Transfer},
+};
 
 const STORAGE_KEY: &[u8] = b"child";
 
@@ -218,7 +220,7 @@ async fn should_notify_about_storage_changes() {
 		// Cause a change:
 		let mut builder = client.new_block(Default::default()).unwrap();
 		builder
-			.push_transfer(runtime::Transfer {
+			.push_transfer(Transfer {
 				from: AccountKeyring::Alice.into(),
 				to: AccountKeyring::Ferdie.into(),
 				amount: 42,
@@ -244,18 +246,26 @@ async fn should_send_initial_storage_changes_and_notifications() {
 		let mut client = Arc::new(substrate_test_runtime_client::new());
 		let (api, _child) = new_full(client.clone(), test_executor(), DenyUnsafe::No);
 
-		let alice_balance_key =
-			blake2_256(&runtime::system::balance_of_key(AccountKeyring::Alice.into()));
+		let alice_balance_key = [
+			sp_core::hashing::twox_128(b"System"),
+			sp_core::hashing::twox_128(b"Account"),
+			sp_core::hashing::blake2_128(&AccountKeyring::Alice.public()),
+		]
+		.concat()
+		.iter()
+		.chain(AccountKeyring::Alice.public().0.iter())
+		.cloned()
+		.collect::<Vec<u8>>();
 
 		let api_rpc = api.into_rpc();
 		let sub = api_rpc
-			.subscribe("state_subscribeStorage", [[StorageKey(alice_balance_key.to_vec())]])
+			.subscribe("state_subscribeStorage", [[StorageKey(alice_balance_key)]])
 			.await
 			.unwrap();
 
 		let mut builder = client.new_block(Default::default()).unwrap();
 		builder
-			.push_transfer(runtime::Transfer {
+			.push_transfer(Transfer {
 				from: AccountKeyring::Alice.into(),
 				to: AccountKeyring::Ferdie.into(),
 				amount: 42,
@@ -280,22 +290,42 @@ async fn should_query_storage() {
 	async fn run_tests(mut client: Arc<TestClient>) {
 		let (api, _child) = new_full(client.clone(), test_executor(), DenyUnsafe::No);
 
-		let mut add_block = |nonce| {
+		let mut add_block = |index| {
 			let mut builder = client.new_block(Default::default()).unwrap();
 			// fake change: None -> None -> None
-			builder.push_storage_change(vec![1], None).unwrap();
+			builder
+				.push(ExtrinsicBuilder::new_storage_change(vec![1], None).build())
+				.unwrap();
 			// fake change: None -> Some(value) -> Some(value)
-			builder.push_storage_change(vec![2], Some(vec![2])).unwrap();
+			builder
+				.push(ExtrinsicBuilder::new_storage_change(vec![2], Some(vec![2])).build())
+				.unwrap();
 			// actual change: None -> Some(value) -> None
 			builder
-				.push_storage_change(vec![3], if nonce == 0 { Some(vec![3]) } else { None })
+				.push(
+					ExtrinsicBuilder::new_storage_change(
+						vec![3],
+						if index == 0 { Some(vec![3]) } else { None },
+					)
+					.build(),
+				)
 				.unwrap();
 			// actual change: None -> Some(value)
 			builder
-				.push_storage_change(vec![4], if nonce == 0 { None } else { Some(vec![4]) })
+				.push(
+					ExtrinsicBuilder::new_storage_change(
+						vec![4],
+						if index == 0 { None } else { Some(vec![4]) },
+					)
+					.build(),
+				)
 				.unwrap();
 			// actual change: Some(value1) -> Some(value2)
-			builder.push_storage_change(vec![5], Some(vec![nonce as u8])).unwrap();
+			builder
+				.push(
+					ExtrinsicBuilder::new_storage_change(vec![5], Some(vec![index as u8])).build(),
+				)
+				.unwrap();
 			let block = builder.build().unwrap().block;
 			let hash = block.header.hash();
 			executor::block_on(client.import(BlockOrigin::Own, block)).unwrap();
@@ -482,12 +512,13 @@ async fn should_return_runtime_version() {
 	let client = Arc::new(substrate_test_runtime_client::new());
 	let (api, _child) = new_full(client.clone(), test_executor(), DenyUnsafe::No);
 
+	// it is basically json-encoded substrate_test_runtime_client::runtime::VERSION
 	let result = "{\"specName\":\"test\",\"implName\":\"parity-test\",\"authoringVersion\":1,\
 		\"specVersion\":2,\"implVersion\":2,\"apis\":[[\"0xdf6acb689907609b\",4],\
 		[\"0x37e397fc7c91f5e4\",2],[\"0xd2bc9897eed08f15\",3],[\"0x40fe3ad401f8959a\",6],\
-		[\"0xc6e9a76309f39b09\",1],[\"0xdd718d5cc53262d4\",1],[\"0xcbca25e39f142387\",2],\
-		[\"0xf78b278be53f454c\",2],[\"0xab3c0572291feb8b\",1],[\"0xbc9d89904f5b923f\",1]],\
-		\"transactionVersion\":1,\"stateVersion\":1}";
+		[\"0xbc9d89904f5b923f\",1],[\"0xc6e9a76309f39b09\",2],[\"0xdd718d5cc53262d4\",1],\
+		[\"0xcbca25e39f142387\",2],[\"0xf78b278be53f454c\",2],[\"0xab3c0572291feb8b\",1],\
+		[\"0xed99c5acb25eedf5\",3]],\"transactionVersion\":1,\"stateVersion\":1}";
 
 	let runtime_version = api.runtime_version(None.into()).unwrap();
 	let serialized = serde_json::to_string(&runtime_version).unwrap();