RPC to trace execution of specified block (#7780)

* Add filter reload handle

* add RPC, move logging module from cli to tracing

* remove dup fn

* working example

* Update client/rpc-api/src/system/mod.rs

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* Prefer "set" to "reload"

* Re-enable the commented out features of the logger

* Remove duplicate code

* cleanup

* unneeded lvar

* Bump to latest patch release

* Add new CLI option to disable log filter reloading,

Move profiling CLI options to SharedParams

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Applied suggestions from reviews

* Fix calls to init_logger()

* Handle errors when parsing logging directives

* Deny `system_setLogFilter` RPC by default

* One more time

* Don't ignore parse errors for log directives set via CLI or RPC

* Improve docs

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/config.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* fix merge errors

* include default directives with system_setLogFilter RPC,

implement system_rawSetLogFilter RPC to exclude defaults

* docs etc...

* update test

* refactor: rename fn

* Add a test for system_set_log_filter – NOTE: the code should likely change to return an error when bad directives are passed

* Update client/cli/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Address review grumbles

* Add doc note on panicking behaviour

* print all invalid directives before panic

* change RPCs to: addLogFilter and resetLogFilter

* make CLI log directives default

* add comments

* restore previous behaviour to panic when hard-coded directives are invalid

* change/refactor directive parsing

* fix line width

* add test for log filter reloading

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* finish up suggestions from code review

* improve test

* change expect message

* change fn name

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* add docs, remove unused fn

* propagate Err on invalid log directive

* Update tracing-subscriber version

* Improve docs for `disable_log_reloading` CLI param

* WIP implementation: RPC and trace capturing

* WIP

* fix incorrect number of digest items

* return errors

* add From impl for Values, rename structs

* fixes

* implement option to choose targets for traces

* rename fn

* fix EnvFilter and add root span

* fix root span

* add docs, remove unnecessary traits

* fix regression on parent_id introduced in a9c73113a8

* fix line width

* remove unused

* include block hash, parent hash & targets in response

* move types from sp-tracing into sp-rpc

move block and parent hash into root of BlockTrace

* switch from log::trace to tracing::trace in state-machine

* use unsigned integer type to represent Ext::id in traces

* ensure id is unique by implementing Subscriber

tracing_subscriber::FmtSubscriber does not guarantee
unique ids

* indentation

* fix typo

* update types

* add sp_io::storage events

* Change response format

- update types
- record distinct timestamps
- sort spans by first entered

* convert to HexDisplay, refactor

* Sort out fallout from merge

* Update client/rpc-api/src/state/mod.rs

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Exit early unless the node runs with --rpc-methods=Unsafe

* Better error handling

* Use wasm-timer

* revert trace alteration in `state-machine` and remove events in `sp_io::storage`

Resolve in follow-up PR

* Review feedback: less collects

* Without Arcs

* Fix span exit

* typo

* cleanup

* Add a few debug messages to tracing module

* Structure traces state-machine/ext; Dispatchable extrinsics spans not working

* Correctly encode Option storage values

* Remove test field for Put and Get

* Try out some changes to dispatch macro

* Add various log messages in dispatch

* Add span dispatch span to new proc macro

* Remove debug messages in dispatch

* Trivial clean up

* Structure remaining state-machine traces (ChangesRoot*)

* Removed unnesecary tracing targets

* Remove log

* New cargo.lock post merge

* Add logging for wasm_overrides

* remove temp logs

* remove temp logs

* remove unused dep

* remove temp logs

* add logging to wasm_overrides

* add logging to state_tracing

* add logging for spans to substrate (includes timings)

* Skip serializing some event fields; Remove most storage traces

* Bring back all ext.rs traces

* Do not skip bool values in events

* Skip serializing span values

* Serialize span values; remove some trace events in ext

* Remove more trace events

* Delete commented out traces

* Remove all unused traces

* Add event filtering

* Fix typo

* wip - change response types to be more efficient

missing import

type

* Serialize struct fields as camelCase

* Add back in event filtering

* Remove name field from event

* Sort spans by time entered

* Sort spans in ASCending order

* Add storage keys target param to rpc

* Limit payload size; improve hash fields; include storage keys

- cleanup event_key_filter
- better block hash representation
- limit payload size
- cleanup based on andrews comments

* Error when serialized payload is to big

* Import MAX_PAYLOAD from rpc-servers

* Clean up ext.rs

* Misc. cleaning and comments

* Strict ordering span Id; no span sort; adjust for rpc base payload

* Add RPC docs to rpc-api/src/state/mod

* Make params bullet points

* Update primitives/rpc/src/tracing.rs

* Put all tracing logic within trace

* Remove attr.record in new_span

* Add back value record in new_span

* restore result collection in ext

* Revert "Add back value record in new_span"

This reverts commit baf1a735f23e5eef1bf6264adfabb788910fa661.

* 🤦

* more 🤦

* Update docs; Try fix line width issues

* Improve docs

* Improve docs

* Remove default key filters + add key recs to docs

* Try restore old traces

* Add back doc comment

* Clean up newlines in ext.rs

* More new line remova;
l

* Use FxHashMap

* Try use EnvFilter directives for event filtering

* Remove directive, filter events by fields

* Use trace metadata correctly

* Try EnvFilter directive with all default targets

* Revert "Try EnvFilter directive with all default targets"

This reverts commit 4cc6ebc721d207c3b846444174f89d45038525ac.

* Clean up clippy warning

* Incorporate Niklas feedback

* Update trace/log macro calls to have better syntx

* Use Ordering::Relaxed

* Improve patch and filter doc comment

* Clean up `BlockSubscriber::new`

* Try optimize `BlockSubscriber::enabled`

* Apply suggestions from code review

Co-authored-by: David <dvdplm@gmail.com>

* Apply suggestions from code review

Co-authored-by: David <dvdplm@gmail.com>

* Use contains_key

* use heuristic for payload size

* Add error tupe for client::tracing::block

* Minor tweaks

* Make a note about `--features with-tracing`

* Add CURL example to RPC docs

* Link to substrate-archibe wasm

* Trivial doc clean up based on David feedback

* Explicit result type name

* Respect line length

* Use the error

* Don't print timings when spans close

* Fix failing sc-rpc-api

* Update  sp-tracing inner-line doc

* Update client/tracing/src/block/mod.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/call_executor.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/client/call_executor.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/tracing/src/block/mod.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/tracing/src/block/mod.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Address some review grumbles

* Update primitives/state-machine/src/ext.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Use result_encoded structure fields in ext.rs

* Use value key for ext put

* Add notes about tracing key names matter

Co-authored-by: Matt <mattrutherford@users.noreply.github.com>
Co-authored-by: David <dvdplm@gmail.com>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: emostov <32168567+emostov@users.noreply.github.com>

substrate/client/tracing/src/block/mod.rs

@@ -0,0 +1,338 @@
+// Copyright 2021 Parity Technologies (UK) Ltd.
+// This file is part of Substrate.
+
+// Substrate is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Substrate is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Substrate.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Utilities for tracing block execution
+
+use std::{collections::HashMap, sync::{Arc, atomic::{AtomicU64, Ordering}}, time::Instant};
+
+use parking_lot::Mutex;
+use tracing::{Dispatch, dispatcher, Subscriber, Level, span::{Attributes, Record, Id}};
+use tracing_subscriber::CurrentSpan;
+
+use sc_client_api::BlockBackend;
+use sc_rpc_server::MAX_PAYLOAD;
+use sp_api::{Core, Metadata, ProvideRuntimeApi, Encode};
+use sp_blockchain::HeaderBackend;
+use sp_runtime::{
+	generic::BlockId,
+	traits::{Block as BlockT, Header},
+};
+use sp_rpc::tracing::{BlockTrace, Span, TraceError, TraceBlockResponse};
+use sp_tracing::{WASM_NAME_KEY, WASM_TARGET_KEY, WASM_TRACE_IDENTIFIER};
+use sp_core::hexdisplay::HexDisplay;
+use crate::{SpanDatum, TraceEvent, Values};
+
+// Heuristic for average event size in bytes.
+const AVG_EVENT: usize = 600 * 8;
+// Heuristic for average span size in bytes.
+const AVG_SPAN: usize = 100 * 8;
+// Estimate of the max base RPC payload size when the Id is bound as a u64. If strings
+// are used for the RPC Id this may need to be adjusted. Note: The base payload
+// does not include the RPC result.
+//
+// The estimate is based on the JSONRPC response message which has the following format:
+// `{"jsonrpc":"2.0","result":[],"id":18446744073709551615}`.
+//
+// We care about the total size of the payload because jsonrpc-server will simply ignore
+// messages larger than `sc_rpc_server::MAX_PAYLOAD` and the caller will not get any
+// response.
+const BASE_PAYLOAD: usize = 100;
+// Default to only pallet, frame support and state related traces
+const DEFAULT_TARGETS: &str = "pallet,frame,state";
+const TRACE_TARGET: &str = "block_trace";
+// The name of a field required for all events.
+const REQUIRED_EVENT_FIELD: &str  = "method";
+
+/// Tracing Block Result type alias
+pub type TraceBlockResult<T> = Result<T, Error>;
+
+/// Tracing Block error
+#[derive(Debug, thiserror::Error)]
+#[allow(missing_docs)]
+#[non_exhaustive]
+pub enum Error {
+	#[error("Invalid block Id: {0}")]
+	InvalidBlockId(#[from] sp_blockchain::Error),
+	#[error("Missing block component: {0}")]
+	MissingBlockComponent(String),
+	#[error("Dispatch error: {0}")]
+	Dispatch(String)
+}
+
+struct BlockSubscriber {
+	targets: Vec<(String, Level)>,
+	next_id: AtomicU64,
+	current_span: CurrentSpan,
+	spans: Mutex<HashMap<Id, SpanDatum>>,
+	events: Mutex<Vec<TraceEvent>>,
+}
+
+impl BlockSubscriber {
+	fn new(targets: &str) -> Self {
+		let next_id = AtomicU64::new(1);
+		let mut targets: Vec<_> = targets
+			.split(',')
+			.map(crate::parse_target)
+			.collect();
+		// Ensure that WASM traces are always enabled
+		// Filtering happens when decoding the actual target / level
+		targets.push((WASM_TRACE_IDENTIFIER.to_owned(), Level::TRACE));
+		BlockSubscriber {
+			targets,
+			next_id,
+			current_span: CurrentSpan::default(),
+			spans: Mutex::new(HashMap::new()),
+			events: Mutex::new(Vec::new()),
+		}
+	}
+}
+
+impl Subscriber for BlockSubscriber {
+	fn enabled(&self, metadata: &tracing::Metadata<'_>) -> bool {
+		if !metadata.is_span() && !metadata.fields().field(REQUIRED_EVENT_FIELD).is_some() {
+			return false;
+		}
+		for (target, level) in &self.targets {
+			if metadata.level() <= level && metadata.target().starts_with(target) {
+				return true;
+			}
+		}
+		false
+	}
+
+	fn new_span(&self, attrs: &Attributes<'_>) -> Id {
+		let id = Id::from_u64(self.next_id.fetch_add(1, Ordering::Relaxed));
+		let mut values = Values::default();
+		attrs.record(&mut values);
+		let parent_id = attrs.parent().cloned()
+			.or_else(|| self.current_span.id());
+		let span = SpanDatum {
+			id: id.clone(),
+			parent_id,
+			name: attrs.metadata().name().to_owned(),
+			target: attrs.metadata().target().to_owned(),
+			level: *attrs.metadata().level(),
+			line: attrs.metadata().line().unwrap_or(0),
+			start_time: Instant::now(),
+			values,
+			overall_time: Default::default()
+		};
+
+		self.spans.lock().insert(id.clone(), span);
+		id
+	}
+
+	fn record(&self, span: &Id, values: &Record<'_>) {
+		let mut span_data = self.spans.lock();
+		if let Some(s) = span_data.get_mut(span) {
+			values.record(&mut s.values);
+		}
+	}
+
+	fn record_follows_from(&self, _span: &Id, _follows: &Id) {
+		// Not currently used
+		unimplemented!("record_follows_from is not implemented");
+	}
+
+	fn event(&self, event: &tracing::Event<'_>) {
+		let mut values = crate::Values::default();
+		event.record(&mut values);
+		let parent_id = event.parent().cloned()
+			.or_else(|| self.current_span.id());
+		let trace_event = TraceEvent {
+			name: event.metadata().name().to_owned(),
+			target: event.metadata().target().to_owned(),
+			level: *event.metadata().level(),
+			values,
+			parent_id,
+		};
+		self.events.lock().push(trace_event);
+	}
+
+	fn enter(&self, id: &Id) {
+		self.current_span.enter(id.clone());
+	}
+
+	fn exit(&self, span: &Id) {
+		if self.spans.lock().contains_key(span) {
+			self.current_span.exit();
+		}
+	}
+}
+
+/// Holds a reference to the client in order to execute the given block.
+/// Records spans & events for the supplied targets (eg. "pallet,frame,state") and
+/// only records events with the specified hex encoded storage key prefixes.
+/// Note: if `targets` or `storage_keys` is an empty string then nothing is
+/// filtered out.
+pub struct BlockExecutor<Block: BlockT, Client> {
+	client: Arc<Client>,
+	block: Block::Hash,
+	targets: Option<String>,
+	storage_keys: Option<String>,
+}
+
+impl<Block, Client> BlockExecutor<Block, Client>
+	where
+		Block: BlockT + 'static,
+		Client: HeaderBackend<Block> + BlockBackend<Block> + ProvideRuntimeApi<Block>
+		+ Send + Sync + 'static,
+		Client::Api: Metadata<Block>,
+{
+	/// Create a new `BlockExecutor`
+	pub fn new(
+		client: Arc<Client>,
+		block: Block::Hash,
+		targets: Option<String>,
+		storage_keys: Option<String>,
+	) -> Self {
+		Self { client, block, targets, storage_keys }
+	}
+
+	/// Execute block, record all spans and events belonging to `Self::targets`
+	/// and filter out events which do not have keys starting with one of the
+	/// prefixes in `Self::storage_keys`.
+	pub fn trace_block(&self) -> TraceBlockResult<TraceBlockResponse> {
+		tracing::debug!(target: "state_tracing", "Tracing block: {}", self.block);
+		// Prepare the block
+		let id = BlockId::Hash(self.block);
+		let mut header = self.client.header(id)
+			.map_err(|e| Error::InvalidBlockId(e))?
+			.ok_or_else(|| Error::MissingBlockComponent("Header not found".to_string()))?;
+		let extrinsics = self.client.block_body(&id)
+			.map_err(|e| Error::InvalidBlockId(e))?
+			.ok_or_else(|| Error::MissingBlockComponent("Extrinsics not found".to_string()))?;
+		tracing::debug!(target: "state_tracing", "Found {} extrinsics", extrinsics.len());
+		let parent_hash = *header.parent_hash();
+		let parent_id = BlockId::Hash(parent_hash);
+		// Remove all `Seal`s as they are added by the consensus engines after building the block.
+		// On import they are normally removed by the consensus engine.
+		header.digest_mut().logs.retain(|d| d.as_seal().is_none());
+		let block = Block::new(header, extrinsics);
+
+		let targets = if let Some(t) = &self.targets { t } else { DEFAULT_TARGETS };
+		let block_subscriber = BlockSubscriber::new(targets);
+		let dispatch = Dispatch::new(block_subscriber);
+
+		{
+			let dispatcher_span = tracing::debug_span!(
+				target: "state_tracing",
+				"execute_block",
+				extrinsics_len = block.extrinsics().len(),
+			);
+			let _guard = dispatcher_span.enter();
+			if let Err(e) = dispatcher::with_default(&dispatch, || {
+				let span = tracing::info_span!(
+					target: TRACE_TARGET,
+					"trace_block",
+				);
+				let _enter = span.enter();
+				self.client.runtime_api().execute_block(&parent_id, block)
+			}) {
+				return Err(Error::Dispatch(format!("Failed to collect traces and execute block: {:?}", e).to_string()));
+			}
+		}
+
+		let block_subscriber = dispatch.downcast_ref::<BlockSubscriber>()
+			.ok_or(Error::Dispatch(
+				"Cannot downcast Dispatch to BlockSubscriber after tracing block".to_string()
+			))?;
+		let spans: Vec<_> = block_subscriber.spans
+			.lock()
+			.drain()
+			// Patch wasm identifiers
+			.filter_map(|(_, s)| patch_and_filter(SpanDatum::from(s), targets))
+			.collect();
+		let events: Vec<_> = block_subscriber.events
+			.lock()
+			.drain(..)
+			.filter(|e| self.storage_keys
+				.as_ref()
+				.map(|keys| event_key_filter(e, keys))
+				.unwrap_or(false)
+			)
+			.map(|s| s.into())
+			.collect();
+		tracing::debug!(target: "state_tracing", "Captured {} spans and {} events", spans.len(), events.len());
+
+		let approx_payload_size = BASE_PAYLOAD + events.len() * AVG_EVENT + spans.len() * AVG_SPAN;
+		let response = if approx_payload_size > MAX_PAYLOAD {
+				TraceBlockResponse::TraceError(TraceError {
+					error:
+						"Payload likely exceeds max payload size of RPC server.".to_string()
+				})
+		} else {
+			TraceBlockResponse::BlockTrace(BlockTrace {
+				block_hash: block_id_as_string(id),
+				parent_hash: block_id_as_string(parent_id),
+				tracing_targets: targets.to_string(),
+				storage_keys: self.storage_keys.clone().unwrap_or_default(),
+				spans,
+				events,
+			})
+		};
+
+		Ok(response)
+	}
+}
+
+fn event_key_filter(event: &TraceEvent, storage_keys: &str) -> bool {
+	event.values.string_values.get("key")
+		.and_then(|key| Some(check_target(storage_keys, key, &event.level)))
+		.unwrap_or(false)
+}
+
+/// Filter out spans that do not match our targets and if the span is from WASM update its `name`
+/// and `target` fields to the WASM values for those fields.
+//
+// The `tracing` crate requires trace metadata to be static. This does not work for wasm code in
+// substrate, as it is regularly updated with new code from on-chain events. The workaround for this
+// is for substrate's WASM tracing wrappers to put the `name` and `target` data in the `values` map
+// (normally they would be in the static metadata assembled at compile time). Here, if a special
+// WASM `name` or `target` key is found in the `values` we remove it and put the key value pair in
+// the span's metadata, making it consistent with spans that come from native code.
+fn patch_and_filter(mut span: SpanDatum, targets: &str) -> Option<Span> {
+	if span.name == WASM_TRACE_IDENTIFIER {
+		span.values.bool_values.insert("wasm".to_owned(), true);
+		if let Some(n) = span.values.string_values.remove(WASM_NAME_KEY) {
+			span.name = n;
+		}
+		if let Some(t) = span.values.string_values.remove(WASM_TARGET_KEY) {
+			span.target = t;
+		}
+		if !check_target(targets, &span.target, &span.level) {
+			return None;
+		}
+	}
+	Some(span.into())
+}
+
+/// Check if a `target` matches any `targets` by prefix
+fn check_target(targets: &str, target: &str, level: &Level) -> bool {
+	for (t, l) in targets.split(',').map(crate::parse_target) {
+		if target.starts_with(t.as_str()) && level <= &l {
+			return true;
+		}
+	}
+	false
+}
+
+fn block_id_as_string<T: BlockT>(block_id: BlockId<T>) -> String {
+	 match block_id {
+		BlockId::Hash(h) => HexDisplay::from(&h.encode()).to_string(),
+		BlockId::Number(n) =>  HexDisplay::from(&n.encode()).to_string()
+	}
+}

substrate/client/tracing/src/lib.rs

@@ -29,6 +29,7 @@
 #![warn(missing_docs)]
 
 pub mod logging;
+pub mod block;
 
 use rustc_hash::FxHashMap;
 use std::fmt;
@@ -86,7 +87,7 @@ pub trait TraceHandler: Send + Sync {
 #[derive(Debug)]
 pub struct TraceEvent {
 	/// Name of the event.
-	pub name: &'static str,
+	pub name: String,
 	/// Target of the event.
 	pub target: String,
 	/// Level of the event.
@@ -123,13 +124,13 @@ pub struct SpanDatum {
 /// Holds associated values for a tracing span
 #[derive(Default, Clone, Debug)]
 pub struct Values {
-	/// HashMap of `bool` values
+	/// FxHashMap of `bool` values
 	pub bool_values: FxHashMap<String, bool>,
-	/// HashMap of `i64` values
+	/// FxHashMap of `i64` values
 	pub i64_values: FxHashMap<String, i64>,
-	/// HashMap of `u64` values
+	/// FxHashMap of `u64` values
 	pub u64_values: FxHashMap<String, u64>,
-	/// HashMap of `String` values
+	/// FxHashMap of `String` values
 	pub string_values: FxHashMap<String, String>,
 }
 
@@ -265,7 +266,7 @@ impl<S: Subscriber> Layer<S> for ProfilingLayer {
 			parent_id: attrs.parent().cloned().or_else(|| self.current_span.id()),
 			name: attrs.metadata().name().to_owned(),
 			target: attrs.metadata().target().to_owned(),
-			level: attrs.metadata().level().clone(),
+			level: *attrs.metadata().level(),
 			line: attrs.metadata().line().unwrap_or(0),
 			start_time: Instant::now(),
 			overall_time: ZERO_DURATION,
@@ -285,9 +286,9 @@ impl<S: Subscriber> Layer<S> for ProfilingLayer {
 		let mut values = Values::default();
 		event.record(&mut values);
 		let trace_event = TraceEvent {
-			name: event.metadata().name(),
+			name: event.metadata().name().to_owned(),
 			target: event.metadata().target().to_owned(),
-			level: event.metadata().level().clone(),
+			level: *event.metadata().level(),
 			values,
 			parent_id: event.parent().cloned().or_else(|| self.current_span.id()),
 		};
@@ -304,7 +305,6 @@ impl<S: Subscriber> Layer<S> for ProfilingLayer {
 	}
 
 	fn on_exit(&self, span: &Id, _ctx: Context<S>) {
-		self.current_span.exit();
 		let end_time = Instant::now();
 		let span_datum = {
 			let mut span_data = self.span_data.lock();
@@ -312,6 +312,8 @@ impl<S: Subscriber> Layer<S> for ProfilingLayer {
 		};
 
 		if let Some(mut span_datum) = span_datum {
+			// If `span_datum` is `None` we don't exit (we'd be exiting the parent span)
+			self.current_span.exit();
 			span_datum.overall_time += end_time - span_datum.start_time;
 			if span_datum.name == WASM_TRACE_IDENTIFIER {
 				span_datum.values.bool_values.insert("wasm".to_owned(), true);
@@ -330,9 +332,7 @@ impl<S: Subscriber> Layer<S> for ProfilingLayer {
 		};
 	}
 
-	fn on_close(&self, span: Id, ctx: Context<S>) {
-		self.on_exit(&span, ctx)
-	}
+	fn on_close(&self, _span: Id, _ctx: Context<S>) {}
 }
 
 /// TraceHandler for sending span data to the logger
@@ -385,6 +385,32 @@ impl TraceHandler for LogTraceHandler {
 	}
 }
 
+impl From<TraceEvent> for sp_rpc::tracing::Event {
+	fn from(trace_event: TraceEvent) -> Self {
+		let data = sp_rpc::tracing::Data {
+			string_values: trace_event.values.string_values
+		};
+		sp_rpc::tracing::Event {
+			target: trace_event.target,
+			data,
+			parent_id: trace_event.parent_id.map(|id| id.into_u64())
+		}
+	}
+}
+
+impl From<SpanDatum> for sp_rpc::tracing::Span {
+	fn from(span_datum: SpanDatum) -> Self {
+		let wasm = span_datum.values.bool_values.get("wasm").is_some();
+		sp_rpc::tracing::Span {
+			id: span_datum.id.into_u64(),
+			parent_id: span_datum.parent_id.map(|id| id.into_u64()),
+			name: span_datum.name,
+			target: span_datum.target,
+			wasm,
+		}
+	}
+}
+
 #[cfg(test)]
 mod tests {
 	use super::*;
@@ -555,7 +581,7 @@ mod tests {
 						break;
 					}
 				}
-				// gard2 and span2 dropped / exited
+				// guard2 and span2 dropped / exited
 			});
 
 			// wait for Event to be dispatched and stored

substrate/client/tracing/src/logging/mod.rs

@@ -177,6 +177,9 @@ where
 	};
 	let builder = FmtSubscriber::builder().with_env_filter(env_filter);
 
+	#[cfg(not(target_os = "unknown"))]
+	let builder = builder.with_span_events(format::FmtSpan::NONE);
+
 	#[cfg(not(target_os = "unknown"))]
 	let builder = builder.with_writer(std::io::stderr as _);