Generic request/response infrastructure for Polkadot (#2352)

* Move NetworkBridgeEvent to subsystem::messages.

It is not protocol related at all, it is in fact only part of the
subsystem communication as it gets wrapped into messages of each
subsystem.

* Request/response infrastructure is taking shape.

WIP: Does not compile.

* Multiplexer variant not supported by Rusts type system.

* request_response::request type checks.

* Cleanup.

* Minor fixes for request_response.

* Implement request sending + move multiplexer.

Request multiplexer is moved to bridge, as there the implementation is
more straight forward as we can specialize on `AllMessages` for the
multiplexing target.

Sending of requests is mostly complete, apart from a few `From`
instances. Receiving is also almost done, initializtion needs to be
fixed and the multiplexer needs to be invoked.

* Remove obsolete multiplexer.

* Initialize bridge with multiplexer.

* Finish generic request sending/receiving.

Subsystems are now able to receive and send requests and responses via
the overseer.

* Doc update.

* Fixes.

* Link issue for not yet implemented code.

* Fixes suggested by @ordian - thanks!

- start encoding at 0
- don't crash on zero protocols
- don't panic on not yet implemented request handling

* Update node/network/protocol/src/request_response/v1.rs

Use index 0 instead of 1.

Co-authored-by: Andronik Ordian <write@reusable.software>

* Update node/network/protocol/src/request_response.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Fix existing tests.

* Better avoidance of division by zoro errors.

* Doc fixes.

* send_request -> start_request.

* Fix missing renamings.

* Update substrate.

* Pass TryConnect instead of true.

* Actually import `IfDisconnected`.

* Fix wrong import.

* Update node/network/bridge/src/lib.rs

typo

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* Update node/network/bridge/src/multiplexer.rs

Remove redundant import.

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* Stop doing tracing from within `From` instance.

Thanks for the catch @tomaka!

* Get rid of redundant import.

* Formatting cleanup.

* Fix tests.

* Add link to issue.

* Clarify comments some more.

* Fix tests.

* Formatting fix.

* tabs

* Fix link

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* Use map_err.

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* Improvements inspired by suggestions by @drahnr.

- Channel size is now determined by function.
- Explicitely scope NetworkService::start_request.

Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

polkadot/node/subsystem/src/messages.rs

@@ -25,7 +25,8 @@
 use futures::channel::{mpsc, oneshot};
 use thiserror::Error;
 use polkadot_node_network_protocol::{
-	v1 as protocol_v1, NetworkBridgeEvent, ReputationChange, PeerId,
+	v1 as protocol_v1, ReputationChange, PeerId,
+	request_response::{Requests, request::IncomingRequest, v1 as req_res_v1},
 };
 use polkadot_node_primitives::{
 	CollationGenerationConfig, SignedFullStatement, ValidationResult,
@@ -44,6 +45,11 @@ use polkadot_primitives::v1::{
 use polkadot_statement_table::v1::Misbehavior;
 use std::{sync::Arc, collections::btree_map::BTreeMap};
 
+
+/// Network events as transmitted to other subsystems, wrapped in their message types.
+pub mod network_bridge_event;
+pub use network_bridge_event::NetworkBridgeEvent;
+
 /// Subsystem messages where each message is always bound to a relay parent.
 pub trait BoundToRelayParent {
 	/// Returns the relay parent this message is bound to.
@@ -212,6 +218,9 @@ pub enum NetworkBridgeMessage {
 	/// Send a batch of collation messages.
 	SendCollationMessages(Vec<(Vec<PeerId>, protocol_v1::CollationProtocol)>),
 
+	/// Send requests via substrate request/response.
+	SendRequests(Vec<Requests>),
+
 	/// Connect to peers who represent the given `validator_ids`.
 	///
 	/// Also ask the network to stay connected to these peers at least
@@ -237,6 +246,7 @@ impl NetworkBridgeMessage {
 			Self::SendValidationMessages(_) => None,
 			Self::SendCollationMessages(_) => None,
 			Self::ConnectToValidators { .. } => None,
+			Self::SendRequests { .. } => None,
 		}
 	}
 }
@@ -246,6 +256,8 @@ impl NetworkBridgeMessage {
 pub enum AvailabilityDistributionMessage {
 	/// Event from the network bridge.
 	NetworkBridgeUpdateV1(NetworkBridgeEvent<protocol_v1::AvailabilityDistributionMessage>),
+	/// Incoming request for an availability chunk.
+	AvailabilityFetchingRequest(IncomingRequest<req_res_v1::AvailabilityFetchingRequest>)
 }
 
 /// Availability Recovery Message.
@@ -266,6 +278,7 @@ impl AvailabilityDistributionMessage {
 	pub fn relay_parent(&self) -> Option<Hash> {
 		match self {
 			Self::NetworkBridgeUpdateV1(_) => None,
+			Self::AvailabilityFetchingRequest(_) => None,
 		}
 	}
 }
@@ -693,3 +706,9 @@ pub enum AllMessages {
 	/// Message for the Approval Distribution subsystem.
 	ApprovalDistribution(ApprovalDistributionMessage),
 }
+
+impl From<IncomingRequest<req_res_v1::AvailabilityFetchingRequest>> for AllMessages {
+	fn from(req: IncomingRequest<req_res_v1::AvailabilityFetchingRequest>) -> Self {
+		From::<AvailabilityDistributionMessage>::from(From::from(req))
+	}
+}

polkadot/node/subsystem/src/messages/network_bridge_event.rs

@@ -0,0 +1,73 @@
+// Copyright 2017-2021 Parity Technologies (UK) Ltd.
+// This file is part of Polkadot.
+
+// Polkadot is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Polkadot is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Polkadot.  If not, see <http://www.gnu.org/licenses/>.
+
+use std::convert::TryFrom;
+
+pub use sc_network::{ReputationChange, PeerId};
+use polkadot_node_network_protocol::{WrongVariant, ObservedRole, OurView, View};
+
+/// Events from network.
+#[derive(Debug, Clone, PartialEq)]
+pub enum NetworkBridgeEvent<M> {
+	/// A peer has connected.
+	PeerConnected(PeerId, ObservedRole),
+
+	/// A peer has disconnected.
+	PeerDisconnected(PeerId),
+
+	/// Peer has sent a message.
+	PeerMessage(PeerId, M),
+
+	/// Peer's `View` has changed.
+	PeerViewChange(PeerId, View),
+
+	/// Our view has changed.
+	OurViewChange(OurView),
+}
+
+impl<M> NetworkBridgeEvent<M> {
+	/// Focus an overarching network-bridge event into some more specific variant.
+	///
+	/// This tries to transform M in `PeerMessage` to a message type specific to a subsystem.
+	/// It is used to dispatch events coming from a peer set to the various subsystems that are
+	/// handled within that peer set. More concretly a `ValidationProtocol` will be transformed
+	/// for example into a `BitfieldDistributionMessage` in case of the `BitfieldDistribution`
+	/// constructor.
+	///
+	/// Therefore a NetworkBridgeEvent<ValidationProtocol> will become for example a
+	/// NetworkBridgeEvent<BitfieldDistributionMessage>, with the more specific message type
+	/// `BitfieldDistributionMessage`.
+	///
+	/// This acts as a call to `clone`, except in the case where the event is a message event,
+	/// in which case the clone can be expensive and it only clones if the message type can
+	/// be focused.
+	pub fn focus<'a, T>(&'a self) -> Result<NetworkBridgeEvent<T>, WrongVariant>
+		where T: 'a + Clone, &'a T: TryFrom<&'a M, Error = WrongVariant>
+	{
+		Ok(match *self {
+			NetworkBridgeEvent::PeerConnected(ref peer, ref role)
+				=> NetworkBridgeEvent::PeerConnected(peer.clone(), role.clone()),
+			NetworkBridgeEvent::PeerDisconnected(ref peer)
+				=> NetworkBridgeEvent::PeerDisconnected(peer.clone()),
+			NetworkBridgeEvent::PeerMessage(ref peer, ref msg)
+				=> NetworkBridgeEvent::PeerMessage(peer.clone(), <&'a T>::try_from(msg)?.clone()),
+			NetworkBridgeEvent::PeerViewChange(ref peer, ref view)
+				=> NetworkBridgeEvent::PeerViewChange(peer.clone(), view.clone()),
+			NetworkBridgeEvent::OurViewChange(ref view)
+				=> NetworkBridgeEvent::OurViewChange(view.clone()),
+		})
+	}
+}