improved gossip topology (#3270)

* gossip-support: gossip topology * some fixes * handle view update for newly added gossip peers * fix neighbors calculation * fix test * resolve TODOs * typo * guide updates * spaces in the guide * sneaky spaces * hash randomness * address some review nits * use unbounded in bridge for subsystem msg
2026-06-23 15:31:08 +00:00 · 2021-06-18 21:30:35 +02:00
parent ae5b355754
commit ad9c02886d
21 changed files with 720 additions and 287 deletions
@@ -1,15 +1,19 @@
 # Gossip Support

 The Gossip Support Subsystem is responsible for keeping track of session changes
-and issuing a connection request to all validators in the next, current and a few past sessions
-if we are a validator in these sessions.
-The request will add all validators to a reserved PeerSet, meaning we will not reject a connection request
-from any validator in that set.
+and issuing a connection request to all validators in the next, current and
+a few past sessions if we are a validator in these sessions.
+The request will add all validators to a reserved PeerSet, meaning we will not
+reject a connection request from any validator in that set.

-Gossiping subsystems will be notified when a new peer connects or disconnects by network bridge.
-It is their responsibility to limit the amount of outgoing gossip messages.
-At the moment we enforce a cap of `max(sqrt(peers.len()), 25)` message recipients at a time in each gossiping subsystem.
+In addition to that, it creates a gossip overlay topology per session which
+limits the amount of messages sent and received to be an order of sqrt of the
+validators. Our neighbors in this graph will be forwarded to the network bridge
+with the `NetworkBridgeMessage::NewGossipTopology` message.

-We also flip a coin with the same probability when handling peer view updates in the distribution subsystems.
-Over time the probability of not handling a peer view update converges to zero, so it shouldn't be cause much trouble.
-This should be considered as a temporary measure until we implement a more robust solution for gossiping.
+See https://github.com/paritytech/polkadot/issues/3239 for more details.
+
+The gossip topology is used by parachain distribution subsystems,
+such as Bitfield Distrubution, (small) Statement Distributuion and
+Approval Distibution to limit the amount of peers we send messages to
+and handle view updates.
@@ -21,11 +21,10 @@ Input: [`NetworkBridgeMessage`][NBM]


 Output:
-	- [`AvailabilityDistributionMessage`][AvD]`::NetworkBridgeUpdateV1`
+	- [`ApprovalDistributionMessage`][AppD]`::NetworkBridgeUpdateV1`
 	- [`BitfieldDistributionMessage`][BitD]`::NetworkBridgeUpdateV1`
-	- [`PoVDistributionMessage`][PoVD]`::NetworkBridgeUpdateV1`
-	- [`StatementDistributionMessage`][StmtD]`::NetworkBridgeUpdateV1`
 	- [`CollatorProtocolMessage`][CollP]`::NetworkBridgeUpdateV1`
+	- [`StatementDistributionMessage`][StmtD]`::NetworkBridgeUpdateV1`

 ## Functionality

@@ -59,7 +58,7 @@ Each network event is associated with a particular peer-set.

 The `activated` and `deactivated` lists determine the evolution of our local view over time. A `ProtocolMessage::ViewUpdate` is issued to each connected peer on each peer-set, and a `NetworkBridgeEvent::OurViewChange` is issued to each event handler for each protocol.

-We only send view updates if the node has indicated that it has finished major blockchain synchronization. 
+We only send view updates if the node has indicated that it has finished major blockchain synchronization.

 If we are connected to the same peer on both peer-sets, we will send the peer two view updates as a result.

@@ -107,25 +106,28 @@ Map the message onto the corresponding [Event Handler](#event-handlers) based on
 - Send all `(ValidatorId, PeerId)` pairs on the response channel.
 - Feed all Peer IDs to peer set manager the underlying network provides.

+### NewGossipTopology
+
+- Map all `AuthorityDiscoveryId`s to `PeerId`s and issue a corresponding `NetworkBridgeUpdateV1`
+  to all validation subsystems.
+
 ## Event Handlers

 Network bridge event handlers are the intended recipients of particular network protocol messages. These are each a variant of a message to be sent via the overseer.

 ### Validation V1

-* `StatementDistributionV1Message -> StatementDistributionMessage::NetworkBridgeUpdateV1`
-* `PoVDistributionV1Message -> PoVDistributionMessage::NetworkBridgeUpdateV1`
-* `AvailabilityDistributionV1Message -> AvailabilityDistributionMessage::NetworkBridgeUpdateV1`
+* `ApprovalDistributionV1Message -> ApprovalDistributionMessage::NetworkBridgeUpdateV1`
 * `BitfieldDistributionV1Message -> BitfieldDistributionMessage::NetworkBridgeUpdateV1`
+* `StatementDistributionV1Message -> StatementDistributionMessage::NetworkBridgeUpdateV1`

 ### Collation V1

 * `CollatorProtocolV1Message -> CollatorProtocolMessage::NetworkBridgeUpdateV1`

 [NBM]: ../../types/overseer-protocol.md#network-bridge-message
-[AvD]: ../../types/overseer-protocol.md#availability-distribution-message
+[AppD]: ../../types/overseer-protocol.md#approval-distribution-message
 [BitD]: ../../types/overseer-protocol.md#bitfield-distribution-message
-[PoVD]: ../../types/overseer-protocol.md#pov-distribution-message
 [StmtD]: ../../types/overseer-protocol.md#statement-distribution-message
 [CollP]: ../../types/overseer-protocol.md#collator-protocol-message

@@ -147,6 +147,14 @@ enum NetworkBridgeEvent<M> {
 	PeerConnected(PeerId, ObservedRole),
 	/// A peer with given ID is now disconnected.
 	PeerDisconnected(PeerId),
+	/// Our neighbors in the new gossip topology.
+	/// We're not necessarily connected to all of them.
+	///
+	/// This message is issued only on the validation peer set.
+	///
+	/// Note, that the distribution subsystems need to handle the last
+	/// view update of the newly added gossip peers manually.
+	NewGossipTopology(HashSet<PeerId>),
 	/// We received a message from the given peer.
 	PeerMessage(PeerId, M),
 	/// The given peer has updated its description of its view.
@@ -45,7 +45,35 @@ struct ActiveLeavesUpdate {
 }
 ```

-## Approval Voting
+## All Messages
+
+A message type tying together all message types that are used across Subsystems.
+
+```rust
+enum AllMessages {
+    CandidateValidation(CandidateValidationMessage),
+    CandidateBacking(CandidateBackingMessage),
+    ChainApi(ChainApiMessage),
+    CollatorProtocol(CollatorProtocolMessage),
+    StatementDistribution(StatementDistributionMessage),
+    AvailabilityDistribution(AvailabilityDistributionMessage),
+    AvailabilityRecovery(AvailabilityRecoveryMessage),
+    BitfieldDistribution(BitfieldDistributionMessage),
+    BitfieldSigning(BitfieldSigningMessage),
+    Provisioner(ProvisionerMessage),
+    RuntimeApi(RuntimeApiMessage),
+    AvailabilityStore(AvailabilityStoreMessage),
+    NetworkBridge(NetworkBridgeMessage),
+    CollationGeneration(CollationGenerationMessage),
+    ApprovalVoting(ApprovalVotingMessage),
+    ApprovalDistribution(ApprovalDistributionMessage),
+    GossipSupport(GossipSupportMessage),
+    DisputeCoordinator(DisputeCoordinatorMessage),
+    DisputeParticipation(DisputeParticipationMessage),
+}
+```
+
+## Approval Voting Message

 Messages received by the approval voting subsystem.

@@ -127,9 +155,9 @@ enum ApprovalVotingMessage {
 }
 ```

-## Approval Distribution
+## Approval Distribution Message

-Messages received by the approval Distribution subsystem.
+Messages received by the approval distribution subsystem.

 ```rust
 /// Metadata about a block which is now live in the approval protocol.
@@ -166,10 +194,6 @@ enum ApprovalDistributionMessage {
 }
 ```

-## All Messages
-
-> TODO (now)
-
 ## Availability Distribution Message

 Messages received by the availability distribution subsystem.
@@ -334,7 +358,7 @@ enum ChainSelectionMessage {
    /// Request the best leaf containing the given block in its ancestry. Return `None` if
    /// there is no such leaf.
    BestLeafContaining(Hash, ResponseChannel<Option<Hash>>),
-    
+
 }
 ```

@@ -495,6 +519,13 @@ enum NetworkBridgeMessage {
        /// authority discovery has failed to resolve.
        failed: oneshot::Sender<usize>,
    },
+    /// Inform the distribution subsystems about the new
+    /// gossip network topology formed.
+    NewGossipTopology {
+        /// Ids of our neighbors in the new gossip topology.
+        /// We're not necessarily connected to all of them, but we should.
+        our_neighbors: HashSet<AuthorityDiscoveryId>,
+    }
 }
 ```