rpc: Implement chainHead RPC API (#12544)

* rpc/chain_head: Add event structure for serialization

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Add tests for events

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Add API trait for `chainHead`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Add RPC errors

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Manage subscription ID tracking for pinned blocks

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Add tests for subscription management

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Constructor for the API

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Placeholders for API implementation

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Accept RPC subscription sink

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Generate the runtime API event

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Implement the `follow` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Implement the `body` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Implement the `header` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Implement the `storage` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Implement the `call` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Implement the `unpin` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update `Cargo.lock`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Implement `getGenesis` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Fix clippy

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Parse params from hex string

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Constuct API with genesis hash

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Add the finalized block to reported tree route

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Export the API and events for better ergonomics

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Add test module with helper functions

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Test block events from the `follow` pubsub

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Test `genesisHash` getter

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Test `header` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Test `body` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Test calling into the runtime API

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Test runtime for the `follow` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Add runtime code changes for `follow` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Remove space from rustdoc

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Use the `child_key` for storage queries

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Test `storage` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Test child trie query for `storage` method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Event serialization typo

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Remove subscription aliases

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Add `NetworkConfig` parameter

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Named parameters as camelCase if present

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Implement From<ApiError> for RuntimeEvents

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Handle pruning of the best block in finalization window

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Generate initial block events

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Verify that initial in-memory blocks are reported

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Verify the finalized event with forks and pruned blocks

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Fix clippy

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Separate logic for generating initial events

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Handle stopping a subscription ID

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Submit events until the "Stop" event is triggered

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Separate logic for handling new and finalized blocks

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Extend subscription logic with subId handle

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Adjust to the new subscription mngmt API

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Refuse RuntimeAPI calls without the runtime flag

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Verify RuntimeAPI calls without runtime flag

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Add best block per subscription

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Check storage keys for prefixes

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Check storage queries with invalid prefixes

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Allow maximum number of pinned blocks

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Test the maximum number of pinned blocks

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Adjust to origin/master and apply clippy

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* client/service: Enable the `chainHead` API

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Stop subscription on client disconnect and add debug logs

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Fix sending `Stop` on subscription exit

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Check best block is descendent of latest finalized

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* chain_head/tests: Report events before pruning the best block

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Nonrecursive initial block generation

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Generate initial events on subscription executor

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Reduce dev-dependencies for tokio

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Apply suggestions from code review

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

* rpc/chain_head: Accept empty parameters

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Use debug of `HexDisplay` for full format

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Enable subscription ID

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Use jsonrpsee 16.2 camelCase feature for paramaters

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Use `NonZeroUsize` for `NetworkConfig` param

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc/chain_head: Rename `runtime_updates` to `has_runtime_updates`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

substrate/Cargo.lock

@@ -2503,7 +2503,7 @@ dependencies = [
  "indexmap",
  "slab",
  "tokio",
- "tokio-util",
+ "tokio-util 0.7.4",
  "tracing",
 ]
 
@@ -2928,7 +2928,7 @@ dependencies = [
  "thiserror",
  "tokio",
  "tokio-rustls",
- "tokio-util",
+ "tokio-util 0.7.4",
  "tracing",
  "webpki-roots",
 ]
@@ -2991,7 +2991,7 @@ dependencies = [
  "soketto",
  "tokio",
  "tokio-stream",
- "tokio-util",
+ "tokio-util 0.7.4",
  "tower",
  "tracing",
 ]
@@ -4279,7 +4279,7 @@ dependencies = [
  "substrate-rpc-client",
  "tempfile",
  "tokio",
- "tokio-util",
+ "tokio-util 0.7.4",
  "try-runtime-cli",
  "wait-timeout",
 ]
@@ -7774,7 +7774,7 @@ dependencies = [
  "tempfile",
  "thiserror",
  "tokio",
- "tokio-util",
+ "tokio-util 0.7.4",
  "unsigned-varint",
  "zeroize",
 ]
@@ -8085,20 +8085,33 @@ dependencies = [
 name = "sc-rpc-spec-v2"
 version = "0.10.0-dev"
 dependencies = [
+ "array-bytes",
+ "assert_matches",
  "futures",
+ "futures-util",
  "hex",
  "jsonrpsee",
+ "log",
  "parity-scale-codec",
+ "parking_lot 0.12.1",
+ "sc-block-builder",
  "sc-chain-spec",
+ "sc-client-api",
  "sc-transaction-pool-api",
  "serde",
  "serde_json",
  "sp-api",
  "sp-blockchain",
+ "sp-consensus",
  "sp-core",
+ "sp-maybe-compressed-blob",
  "sp-runtime",
+ "sp-version",
+ "substrate-test-runtime",
+ "substrate-test-runtime-client",
  "thiserror",
  "tokio",
+ "tokio-stream",
 ]
 
 [[package]]
@@ -10264,6 +10277,7 @@ dependencies = [
  "futures-core",
  "pin-project-lite 0.2.9",
  "tokio",
+ "tokio-util 0.6.10",
 ]
 
 [[package]]
@@ -10279,6 +10293,20 @@ dependencies = [
  "tokio-stream",
 ]
 
+[[package]]
+name = "tokio-util"
+version = "0.6.10"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "36943ee01a6d67977dd3f84a5a1d2efeb4ada3a1ae771cadfaa535d9d9fc6507"
+dependencies = [
+ "bytes",
+ "futures-core",
+ "futures-sink",
+ "log",
+ "pin-project-lite 0.2.9",
+ "tokio",
+]
+
 [[package]]
 name = "tokio-util"
 version = "0.7.4"