RPC to trace execution of specified block (#7780)

* Add filter reload handle

* add RPC, move logging module from cli to tracing

* remove dup fn

* working example

* Update client/rpc-api/src/system/mod.rs

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* Prefer "set" to "reload"

* Re-enable the commented out features of the logger

* Remove duplicate code

* cleanup

* unneeded lvar

* Bump to latest patch release

* Add new CLI option to disable log filter reloading,

Move profiling CLI options to SharedParams

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Applied suggestions from reviews

* Fix calls to init_logger()

* Handle errors when parsing logging directives

* Deny `system_setLogFilter` RPC by default

* One more time

* Don't ignore parse errors for log directives set via CLI or RPC

* Improve docs

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/config.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* fix merge errors

* include default directives with system_setLogFilter RPC,

implement system_rawSetLogFilter RPC to exclude defaults

* docs etc...

* update test

* refactor: rename fn

* Add a test for system_set_log_filter – NOTE: the code should likely change to return an error when bad directives are passed

* Update client/cli/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Address review grumbles

* Add doc note on panicking behaviour

* print all invalid directives before panic

* change RPCs to: addLogFilter and resetLogFilter

* make CLI log directives default

* add comments

* restore previous behaviour to panic when hard-coded directives are invalid

* change/refactor directive parsing

* fix line width

* add test for log filter reloading

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* finish up suggestions from code review

* improve test

* change expect message

* change fn name

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* add docs, remove unused fn

* propagate Err on invalid log directive

* Update tracing-subscriber version

* Improve docs for `disable_log_reloading` CLI param

* WIP implementation: RPC and trace capturing

* WIP

* fix incorrect number of digest items

* return errors

* add From impl for Values, rename structs

* fixes

* implement option to choose targets for traces

* rename fn

* fix EnvFilter and add root span

* fix root span

* add docs, remove unnecessary traits

* fix regression on parent_id introduced in a9c73113a8

* fix line width

* remove unused

* include block hash, parent hash & targets in response

* move types from sp-tracing into sp-rpc

move block and parent hash into root of BlockTrace

* switch from log::trace to tracing::trace in state-machine

* use unsigned integer type to represent Ext::id in traces

* ensure id is unique by implementing Subscriber

tracing_subscriber::FmtSubscriber does not guarantee
unique ids

* indentation

* fix typo

* update types

* add sp_io::storage events

* Change response format

- update types
- record distinct timestamps
- sort spans by first entered

* convert to HexDisplay, refactor

* Sort out fallout from merge

* Update client/rpc-api/src/state/mod.rs

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Exit early unless the node runs with --rpc-methods=Unsafe

* Better error handling

* Use wasm-timer

* revert trace alteration in `state-machine` and remove events in `sp_io::storage`

Resolve in follow-up PR

* Review feedback: less collects

* Without Arcs

* Fix span exit

* typo

* cleanup

* Add a few debug messages to tracing module

* Structure traces state-machine/ext; Dispatchable extrinsics spans not working

* Correctly encode Option storage values

* Remove test field for Put and Get

* Try out some changes to dispatch macro

* Add various log messages in dispatch

* Add span dispatch span to new proc macro

* Remove debug messages in dispatch

* Trivial clean up

* Structure remaining state-machine traces (ChangesRoot*)

* Removed unnesecary tracing targets

* Remove log

* New cargo.lock post merge

* Add logging for wasm_overrides

* remove temp logs

* remove temp logs

* remove unused dep

* remove temp logs

* add logging to wasm_overrides

* add logging to state_tracing

* add logging for spans to substrate (includes timings)

* Skip serializing some event fields; Remove most storage traces

* Bring back all ext.rs traces

* Do not skip bool values in events

* Skip serializing span values

* Serialize span values; remove some trace events in ext

* Remove more trace events

* Delete commented out traces

* Remove all unused traces

* Add event filtering

* Fix typo

* wip - change response types to be more efficient

missing import

type

* Serialize struct fields as camelCase

* Add back in event filtering

* Remove name field from event

* Sort spans by time entered

* Sort spans in ASCending order

* Add storage keys target param to rpc

* Limit payload size; improve hash fields; include storage keys

- cleanup event_key_filter
- better block hash representation
- limit payload size
- cleanup based on andrews comments

* Error when serialized payload is to big

* Import MAX_PAYLOAD from rpc-servers

* Clean up ext.rs

* Misc. cleaning and comments

* Strict ordering span Id; no span sort; adjust for rpc base payload

* Add RPC docs to rpc-api/src/state/mod

* Make params bullet points

* Update primitives/rpc/src/tracing.rs

* Put all tracing logic within trace

* Remove attr.record in new_span

* Add back value record in new_span

* restore result collection in ext

* Revert "Add back value record in new_span"

This reverts commit baf1a735f23e5eef1bf6264adfabb788910fa661.

* 🤦

* more 🤦

* Update docs; Try fix line width issues

* Improve docs

* Improve docs

* Remove default key filters + add key recs to docs

* Try restore old traces

* Add back doc comment

* Clean up newlines in ext.rs

* More new line remova;
l

* Use FxHashMap

* Try use EnvFilter directives for event filtering

* Remove directive, filter events by fields

* Use trace metadata correctly

* Try EnvFilter directive with all default targets

* Revert "Try EnvFilter directive with all default targets"

This reverts commit 4cc6ebc721d207c3b846444174f89d45038525ac.

* Clean up clippy warning

* Incorporate Niklas feedback

* Update trace/log macro calls to have better syntx

* Use Ordering::Relaxed

* Improve patch and filter doc comment

* Clean up `BlockSubscriber::new`

* Try optimize `BlockSubscriber::enabled`

* Apply suggestions from code review

Co-authored-by: David <dvdplm@gmail.com>

* Apply suggestions from code review

Co-authored-by: David <dvdplm@gmail.com>

* Use contains_key

* use heuristic for payload size

* Add error tupe for client::tracing::block

* Minor tweaks

* Make a note about `--features with-tracing`

* Add CURL example to RPC docs

* Link to substrate-archibe wasm

* Trivial doc clean up based on David feedback

* Explicit result type name

* Respect line length

* Use the error

* Don't print timings when spans close

* Fix failing sc-rpc-api

* Update  sp-tracing inner-line doc

* Update client/tracing/src/block/mod.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/call_executor.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/call_executor.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/tracing/src/block/mod.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/tracing/src/block/mod.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Address some review grumbles

* Update primitives/state-machine/src/ext.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Use result_encoded structure fields in ext.rs

* Use value key for ext put

* Add notes about tracing key names matter

Co-authored-by: Matt <mattrutherford@users.noreply.github.com>
Co-authored-by: David <dvdplm@gmail.com>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: emostov <32168567+emostov@users.noreply.github.com>

substrate/client/rpc/src/state/mod.rs

@@ -40,7 +40,9 @@ use self::error::{Error, FutureResult};
 
 pub use sc_rpc_api::state::*;
 pub use sc_rpc_api::child_state::*;
-use sc_client_api::{ExecutorProvider, StorageProvider, BlockchainEvents, Backend, ProofProvider};
+use sc_client_api::{
+	ExecutorProvider, StorageProvider, BlockchainEvents, Backend, BlockBackend, ProofProvider
+};
 use sp_blockchain::{HeaderMetadata, HeaderBackend};
 
 const STORAGE_KEYS_PAGED_MAX_COUNT: u32 = 1000;
@@ -165,6 +167,14 @@ pub trait StateBackend<Block: BlockT, Client>: Send + Sync + 'static
 		_meta: Option<crate::Metadata>,
 		id: SubscriptionId,
 	) -> RpcResult<bool>;
+
+	/// Trace storage changes for block
+	fn trace_block(
+		&self,
+		block: Block::Hash,
+		targets: Option<String>,
+		storage_keys: Option<String>,
+	) -> FutureResult<sp_rpc::tracing::TraceBlockResponse>;
 }
 
 /// Create new state API that works on full node.
@@ -176,9 +186,10 @@ pub fn new_full<BE, Block: BlockT, Client>(
 	where
 		Block: BlockT + 'static,
 		BE: Backend<Block> + 'static,
-		Client: ExecutorProvider<Block> + StorageProvider<Block, BE> + ProofProvider<Block> + HeaderBackend<Block>
+		Client: ExecutorProvider<Block> + StorageProvider<Block, BE> + ProofProvider<Block>
 			+ HeaderMetadata<Block, Error = sp_blockchain::Error> + BlockchainEvents<Block>
-			+ CallApiAt<Block> + ProvideRuntimeApi<Block> + Send + Sync + 'static,
+			+ CallApiAt<Block> + HeaderBackend<Block>
+			+ BlockBackend<Block> + ProvideRuntimeApi<Block> + Send + Sync + 'static,
 		Client::Api: Metadata<Block>,
 {
 	let child_backend = Box::new(
@@ -346,6 +357,23 @@ impl<Block, Client> StateApi<Block::Hash> for State<Block, Client>
 	) -> RpcResult<bool> {
 		self.backend.unsubscribe_runtime_version(meta, id)
 	}
+
+	/// Re-execute the given block with the tracing targets given in `targets`
+	/// and capture all state changes.
+	///
+	/// Note: requires the node to run with `--rpc-methods=Unsafe`.
+	/// Note: requires runtimes compiled with wasm tracing support, `--features with-tracing`.
+	fn trace_block(
+		&self, block: Block::Hash,
+		targets: Option<String>,
+		storage_keys: Option<String>
+	) -> FutureResult<sp_rpc::tracing::TraceBlockResponse> {
+		if let Err(err) = self.deny_unsafe.check_if_safe() {
+			return Box::new(result(Err(err.into())))
+		}
+
+		self.backend.trace_block(block, targets, storage_keys)
+	}
 }
 
 /// Child state backend API.

substrate/client/rpc/src/state/state_full.rs

@@ -27,9 +27,10 @@ use jsonrpc_pubsub::{typed::Subscriber, SubscriptionId, manager::SubscriptionMan
 use rpc::{Result as RpcResult, futures::{stream, Future, Sink, Stream, future::result}};
 
 use sc_rpc_api::state::ReadProof;
-use sc_client_api::backend::Backend;
-use sp_blockchain::{Result as ClientResult, Error as ClientError, HeaderMetadata, CachedHeaderMetadata, HeaderBackend};
-use sc_client_api::BlockchainEvents;
+use sp_blockchain::{
+	Result as ClientResult, Error as ClientError, HeaderMetadata, CachedHeaderMetadata,
+	HeaderBackend
+};
 use sp_core::{
 	Bytes, storage::{well_known_keys, StorageKey, StorageData, StorageChangeSet,
 	ChildInfo, ChildType, PrefixedStorageKey},
@@ -43,7 +44,10 @@ use sp_api::{Metadata, ProvideRuntimeApi, CallApiAt};
 
 use super::{StateBackend, ChildStateBackend, error::{FutureResult, Error, Result}, client_err};
 use std::marker::PhantomData;
-use sc_client_api::{CallExecutor, StorageProvider, ExecutorProvider, ProofProvider};
+use sc_client_api::{
+	Backend, BlockBackend, BlockchainEvents, CallExecutor, StorageProvider, ExecutorProvider,
+	ProofProvider
+};
 
 /// Ranges to query in state_queryStorage.
 struct QueryStorageRange<Block: BlockT> {
@@ -69,7 +73,7 @@ pub struct FullState<BE, Block: BlockT, Client> {
 impl<BE, Block: BlockT, Client> FullState<BE, Block, Client>
 	where
 		BE: Backend<Block>,
-		Client: StorageProvider<Block, BE> + HeaderBackend<Block>
+		Client: StorageProvider<Block, BE> + HeaderBackend<Block> + BlockBackend<Block>
 			+ HeaderMetadata<Block, Error = sp_blockchain::Error>,
 		Block: BlockT + 'static,
 {
@@ -221,9 +225,11 @@ impl<BE, Block: BlockT, Client> FullState<BE, Block, Client>
 impl<BE, Block, Client> StateBackend<Block, Client> for FullState<BE, Block, Client> where
 	Block: BlockT + 'static,
 	BE: Backend<Block> + 'static,
-	Client: ExecutorProvider<Block> + StorageProvider<Block, BE> + ProofProvider<Block> + HeaderBackend<Block>
+	Client: ExecutorProvider<Block> + StorageProvider<Block, BE>
+		+ ProofProvider<Block> + HeaderBackend<Block>
 		+ HeaderMetadata<Block, Error = sp_blockchain::Error> + BlockchainEvents<Block>
 		+ CallApiAt<Block> + ProvideRuntimeApi<Block>
+		+ BlockBackend<Block>
 		+ Send + Sync + 'static,
 	Client::Api: Metadata<Block>,
 {
@@ -527,12 +533,26 @@ impl<BE, Block, Client> StateBackend<Block, Client> for FullState<BE, Block, Cli
 	) -> RpcResult<bool> {
 		Ok(self.subscriptions.cancel(id))
 	}
+
+	fn trace_block(
+		&self,
+		block: Block::Hash,
+		targets: Option<String>,
+		storage_keys: Option<String>,
+	) -> FutureResult<sp_rpc::tracing::TraceBlockResponse> {
+		Box::new(result(
+			sc_tracing::block::BlockExecutor::new(self.client.clone(), block, targets, storage_keys)
+				.trace_block()
+				.map_err(|e| invalid_block::<Block>(block, None, e.to_string()))
+		))
+	}
 }
 
 impl<BE, Block, Client> ChildStateBackend<Block, Client> for FullState<BE, Block, Client> where
 	Block: BlockT + 'static,
 	BE: Backend<Block> + 'static,
-	Client: ExecutorProvider<Block> + StorageProvider<Block, BE> + HeaderBackend<Block>
+	Client: ExecutorProvider<Block> + StorageProvider<Block, BE>
+		+ HeaderBackend<Block> + BlockBackend<Block>
 		+ HeaderMetadata<Block, Error = sp_blockchain::Error> + BlockchainEvents<Block>
 		+ CallApiAt<Block> + ProvideRuntimeApi<Block>
 		+ Send + Sync + 'static,

substrate/client/rpc/src/state/state_light.rs

@@ -474,6 +474,15 @@ impl<Block, F, Client> StateBackend<Block, Client> for LightState<Block, F, Clie
 	) -> RpcResult<bool> {
 		Ok(self.subscriptions.cancel(id))
 	}
+
+	fn trace_block(
+		&self,
+		_block: Block::Hash,
+		_targets: Option<String>,
+		_storage_keys: Option<String>,
+	) -> FutureResult<sp_rpc::tracing::TraceBlockResponse> {
+		Box::new(result(Err(client_err(ClientError::NotAvailableOnLightClient))))
+	}
 }
 
 impl<Block, F, Client> ChildStateBackend<Block, Client> for LightState<Block, F, Client>