mirror of
https://github.com/pezkuwichain/pezkuwi-subxt.git
synced 2026-05-30 22:11:02 +00:00
Request based PoV distribution (#2640)
* Indentation fix. * Prepare request-response for PoV fetching. * Drop old PoV distribution. * WIP: Fetch PoV directly from backing. * Backing compiles. * Runtime access and connection management for PoV distribution. * Get rid of seemingly dead code. * Implement PoV fetching. Backing does not yet use it. * Don't send `ConnectToValidators` for empty list. * Even better - no need to check over and over again. * PoV fetching implemented. + Typechecks + Should work Missing: - Guide - Tests - Do fallback fetching in case fetching from seconding validator fails. * Check PoV hash upon reception. * Implement retry of PoV fetching in backing. * Avoid pointless validation spawning. * Add jaeger span to pov requesting. * Add back tracing. * Review remarks. * Whitespace. * Whitespace again. * Cleanup + fix tests. * Log to log target in overseer. * Fix more tests. * Don't fail if group cannot be found. * Simple test for PoV fetcher. * Handle missing group membership better. * Add test for retry functionality. * Fix flaky test. * Spaces again. * Guide updates. * Spaces.
This commit is contained in:
Robert Klotzner
+52
-18
@@ -1,49 +1,79 @@
|
||||
# Availability Distribution
|
||||
|
||||
Distribute availability erasure-coded chunks to validators.
|
||||
This subsystem is responsible for distribution availability data to peers.
|
||||
Availability data are chunks, `PoV`s and `AvailableData` (which is `PoV` +
|
||||
`PersistedValidationData`). It does so via request response protocols.
|
||||
|
||||
After a candidate is backed, the availability of the PoV block must be confirmed
|
||||
by 2/3+ of all validators. Backing nodes will serve chunks for a PoV block from
|
||||
their [Availability Store](../utility/availability-store.md), all other
|
||||
validators request their chunks from backing nodes and store those received chunks in
|
||||
their local availability store.
|
||||
In particular this subsystem is responsible for:
|
||||
|
||||
- Respond to network requests requesting availability data by querying the
|
||||
[Availability Store](../utility/availability-store.md).
|
||||
- Request chunks from backing validators to put them in the local `Availability
|
||||
Store` whenever we find an occupied core on the chain,
|
||||
this is to ensure availability by at least 2/3+ of all validators, this
|
||||
happens after a candidate is backed.
|
||||
- Fetch `PoV` from validators, when requested via `FetchPoV` message from
|
||||
backing (pov_requester module).
|
||||
-
|
||||
The backing subsystem is responsible of making available data available in the
|
||||
local `Availability Store` upon validation. This subsystem will serve any
|
||||
network requests by querying that store.
|
||||
|
||||
## Protocol
|
||||
|
||||
This subsystem has no associated peer set right now, but instead relies on
|
||||
a request/response protocol, defined by `Protocol::ChunkFetching`.
|
||||
This subsystem does not handle any peer set messages, but the `pov_requester`
|
||||
does connecto to validators of the same backing group on the validation peer
|
||||
set, to ensure fast propagation of statements between those validators and for
|
||||
ensuring already established connections for requesting `PoV`s. Other than that
|
||||
this subsystem drives request/response protocols.
|
||||
|
||||
Input:
|
||||
|
||||
- OverseerSignal::ActiveLeaves(`[ActiveLeavesUpdate]`)
|
||||
- AvailabilityDistributionMessage{msg: ChunkFetchingRequest}
|
||||
- AvailabilityDistributionMessage{msg: PoVFetchingRequest}
|
||||
- AvailabilityDistributionMessage{msg: FetchPoV}
|
||||
|
||||
Output:
|
||||
|
||||
- NetworkBridgeMessage::SendRequests(`[Requests]`, IfDisconnected::TryConnect)
|
||||
- AvailabilityStore::QueryChunk(candidate_hash, index, response_channel)
|
||||
- AvailabilityStore::StoreChunk(candidate_hash, chunk)
|
||||
- AvailabilityStore::QueryAvailableData(candidate_hash, response_channel)
|
||||
- RuntimeApiRequest::SessionIndexForChild
|
||||
- RuntimeApiRequest::SessionInfo
|
||||
- RuntimeApiRequest::AvailabilityCores
|
||||
|
||||
## Functionality
|
||||
|
||||
### Requesting
|
||||
### PoV Requester
|
||||
|
||||
This subsystems monitors currently occupied cores for all active leaves. For
|
||||
each occupied core it will spawn a task fetching the erasure chunk which has the
|
||||
`ValidatorIndex` of the node. For this an `ChunkFetchingRequest` is
|
||||
issued, via substrate's generic request/response protocol.
|
||||
The PoV requester in the `pov_requester` module takes care of staying connected
|
||||
to validators of the current backing group of this very validator on the `Validation`
|
||||
peer set and it will handle `FetchPoV` requests by issuing network requests to
|
||||
those validators. It will check the hash of the received `PoV`, but will not do any
|
||||
further validation. That needs to be done by the original `FetchPoV` sender
|
||||
(backing subsystem).
|
||||
|
||||
### Chunk Requester
|
||||
|
||||
After a candidate is backed, the availability of the PoV block must be confirmed
|
||||
by 2/3+ of all validators. The chunk requester is responsible of making that
|
||||
availability a reality.
|
||||
|
||||
It does that by querying checking occupied cores for all active leaves. For each
|
||||
occupied core it will spawn a task fetching the erasure chunk which has the
|
||||
`ValidatorIndex` of the node. For this an `ChunkFetchingRequest` is issued, via
|
||||
substrate's generic request/response protocol.
|
||||
|
||||
The spawned task will start trying to fetch the chunk from validators in
|
||||
responsible group of the occupied core, in a random order. For ensuring that we
|
||||
use already open TCP connections wherever possible, the subsystem maintains a
|
||||
use already open TCP connections wherever possible, the requester maintains a
|
||||
cache and preserves that random order for the entire session.
|
||||
|
||||
Note however that, because not all validators in a group have to be actual
|
||||
backers, not all of them are required to have the needed chunk. This in turn
|
||||
could lead to low throughput, as we have to wait for a fetches to fail,
|
||||
could lead to low throughput, as we have to wait for fetches to fail,
|
||||
before reaching a validator finally having our chunk. We do rank back validators
|
||||
not delivering our chunk, but as backers could vary from block to block on a
|
||||
perfectly legitimate basis, this is still not ideal. See issues [2509](https://github.com/paritytech/polkadot/issues/2509) and [2512](https://github.com/paritytech/polkadot/issues/2512)
|
||||
@@ -59,6 +89,10 @@ as we would like as many validators as possible to have their chunk. See this
|
||||
|
||||
### Serving
|
||||
|
||||
On the other side the subsystem will listen for incoming
|
||||
`ChunkFetchingRequest`s from the network bridge and will respond to
|
||||
queries, by looking the requested chunk up in the availability store.
|
||||
On the other side the subsystem will listen for incoming `ChunkFetchingRequest`s
|
||||
and `PoVFetchingRequest`s from the network bridge and will respond to queries,
|
||||
by looking the requested chunks and `PoV`s up in the availability store, this
|
||||
happens in the `responder` module.
|
||||
|
||||
We rely on the backing subsystem to make available data available locally in the
|
||||
`Availability Store` after it has validated it.
|
||||
|
||||
@@ -18,7 +18,7 @@ Output:
|
||||
- [`RuntimeApiMessage`][RAM]
|
||||
- [`CandidateSelectionMessage`][CSM]
|
||||
- [`ProvisionerMessage`][PM]
|
||||
- [`PoVDistributionMessage`][PDM]
|
||||
- [`AvailabilityDistributionMessage`][ADM]
|
||||
- [`StatementDistributionMessage`][SDM]
|
||||
|
||||
## Functionality
|
||||
@@ -39,8 +39,11 @@ The subsystem should maintain a set of handles to Candidate Backing Jobs that ar
|
||||
### On Receiving `CandidateBackingMessage`
|
||||
|
||||
* If the message is a [`CandidateBackingMessage`][CBM]`::GetBackedCandidates`, get all backable candidates from the statement table and send them back.
|
||||
* If the message is a [`CandidateBackingMessage`][CBM]`::Second`, sign and dispatch a `Seconded` statement only if we have not seconded any other candidate and have not signed a `Valid` statement for the requested candidate. Signing both a `Seconded` and `Valid` message is a double-voting misbehavior with a heavy penalty, and this could occur if another validator has seconded the same candidate and we've received their message before the internal seconding request. After successfully dispatching the `Seconded` statement we have to distribute the PoV.
|
||||
* If the message is a [`CandidateBackingMessage`][CBM]`::Statement`, count the statement to the quorum. If the statement in the message is `Seconded` and it contains a candidate that belongs to our assignment, request the corresponding `PoV` from the `PoVDistribution` and launch validation. Issue our own `Valid` or `Invalid` statement as a result.
|
||||
* If the message is a [`CandidateBackingMessage`][CBM]`::Second`, sign and dispatch a `Seconded` statement only if we have not seconded any other candidate and have not signed a `Valid` statement for the requested candidate. Signing both a `Seconded` and `Valid` message is a double-voting misbehavior with a heavy penalty, and this could occur if another validator has seconded the same candidate and we've received their message before the internal seconding request.
|
||||
* If the message is a [`CandidateBackingMessage`][CBM]`::Statement`, count the statement to the quorum. If the statement in the message is `Seconded` and it contains a candidate that belongs to our assignment, request the corresponding `PoV` from the backing node via `AvailabilityDistribution` and launch validation. Issue our own `Valid` or `Invalid` statement as a result.
|
||||
|
||||
If the seconding node did not provide us with the `PoV` we will retry fetching from other backing validators.
|
||||
|
||||
|
||||
> big TODO: "contextual execution"
|
||||
>
|
||||
@@ -112,11 +115,8 @@ fn spawn_validation_work(candidate, parachain head, validation function) {
|
||||
### Fetch Pov Block
|
||||
|
||||
Create a `(sender, receiver)` pair.
|
||||
Dispatch a [`PoVDistributionMessage`][PDM]`::FetchPoV(relay_parent, candidate_hash, sender)` and listen on the receiver for a response.
|
||||
Dispatch a [`AvailabilityDistributionMessage`][PDM]`::FetchPoV{ validator_index, pov_hash, candidate_hash, tx, } and listen on the passed receiver for a response. Availability distribution will send the request to the validator specified by `validator_index`, which might not be serving it for whatever reasons, therefore we need to retry with other backing validators in that case.
|
||||
|
||||
### Distribute Pov Block
|
||||
|
||||
Dispatch a [`PoVDistributionMessage`][PDM]`::DistributePoV(relay_parent, candidate_descriptor, pov)`.
|
||||
|
||||
### Validate PoV Block
|
||||
|
||||
@@ -135,7 +135,7 @@ Dispatch a [`StatementDistributionMessage`][PDM]`::Share(relay_parent, SignedFul
|
||||
[CVM]: ../../types/overseer-protocol.md#validation-request-type
|
||||
[PM]: ../../types/overseer-protocol.md#provisioner-message
|
||||
[CBM]: ../../types/overseer-protocol.md#candidate-backing-message
|
||||
[PDM]: ../../types/overseer-protocol.md#pov-distribution-message
|
||||
[ADM]: ../../types/overseer-protocol.md#availability-distribution-message
|
||||
[SDM]: ../../types/overseer-protocol.md#statement-distribution-message
|
||||
|
||||
[CS]: candidate-selection.md
|
||||
|
||||
@@ -1,110 +0,0 @@
|
||||
# PoV Distribution
|
||||
|
||||
This subsystem is responsible for distributing PoV blocks. For now, unified with [Statement Distribution subsystem](statement-distribution.md).
|
||||
|
||||
## Protocol
|
||||
|
||||
`PeerSet`: `Validation`
|
||||
|
||||
Input: [`PoVDistributionMessage`](../../types/overseer-protocol.md#pov-distribution-message)
|
||||
|
||||
|
||||
Output:
|
||||
|
||||
- NetworkBridge::SendMessage(`[PeerId]`, message)
|
||||
- NetworkBridge::ReportPeer(PeerId, cost_or_benefit)
|
||||
|
||||
|
||||
## Functionality
|
||||
|
||||
This network protocol is responsible for distributing [`PoV`s](../../types/availability.md#proof-of-validity) by gossip. Since PoVs are heavy in practice, gossip is far from the most efficient way to distribute them. In the future, this should be replaced by a better network protocol that finds validators who have validated the block and connects to them directly. This protocol is described.
|
||||
|
||||
This protocol is described in terms of "us" and our peers, with the understanding that this is the procedure that any honest node will run. It has the following goals:
|
||||
- We never have to buffer an unbounded amount of data
|
||||
- PoVs will flow transitively across a network of honest nodes, stemming from the validators that originally seconded candidates requiring those PoVs.
|
||||
|
||||
As we are gossiping, we need to track which PoVs our peers are waiting for to avoid sending them data that they are not expecting. It is not reasonable to expect our peers to buffer unexpected PoVs, just as we will not buffer unexpected PoVs. So notifying our peers about what is being awaited is key. However it is important that the notifications system is also bounded.
|
||||
|
||||
For this, in order to avoid reaching into the internals of the [Statement Distribution](statement-distribution.md) Subsystem, we can rely on an expected property of candidate backing: that each validator can second up to 2 candidates per chain head. This will typically be only one, because they are only supposed to issue one, but they can equivocate if they are willing to be slashed. So we can set a cap on the number of PoVs each peer is allowed to notify us that they are waiting for at a given relay-parent. This cap will be twice the number of validators at that relay-parent. In practice, this is a very lax upper bound that can be reduced much further if desired.
|
||||
|
||||
The view update mechanism of the [Network Bridge](../utility/network-bridge.md) ensures that peers are only allowed to consider a certain set of relay-parents as live. So this bounding mechanism caps the amount of data we need to store per peer at any time at `sum({ 2 * n_validators_at_head(head) * sizeof(hash) for head in view_heads })`. Additionally, peers should only be allowed to notify us of PoV hashes they are waiting for in the context of relay-parents in our own local view, which means that `n_validators_at_head` is implied to be `0` for relay-parents not in our own local view.
|
||||
|
||||
View updates from peers and our own view updates are received from the network bridge. These will lag somewhat behind the `ActiveLeavesUpdate` messages received from the overseer, which will influence the actual data we store. The `OurViewUpdate`s from the [`NetworkBridgeEvent`](../../types/overseer-protocol.md#network-bridge-update) must be considered canonical in terms of our peers' perception of us.
|
||||
|
||||
Lastly, the system needs to be bootstrapped with our own perception of which PoVs we are cognizant of but awaiting data for. This is done by receipt of the [`PoVDistributionMessage`](../../types/overseer-protocol.md#pov-distribution-message)::FetchPoV variant. Proper operation of this subsystem depends on the descriptors passed faithfully representing candidates which have been seconded by other validators.
|
||||
|
||||
## Formal Description
|
||||
|
||||
This protocol can be implemented as a state machine with the following state:
|
||||
|
||||
```rust
|
||||
struct State {
|
||||
relay_parent_state: Map<Hash, BlockBasedState>,
|
||||
peer_state: Map<PeerId, PeerState>,
|
||||
our_view: View,
|
||||
}
|
||||
|
||||
struct BlockBasedState {
|
||||
known: Map<Hash, PoV>, // should be a shared PoV in practice. these things are heavy.
|
||||
fetching: Map<Hash, [ResponseChannel<PoV>]>,
|
||||
n_validators: usize,
|
||||
}
|
||||
|
||||
struct PeerState {
|
||||
awaited: Map<Hash, Set<Hash>>,
|
||||
}
|
||||
```
|
||||
|
||||
We also use the [`PoVDistributionV1Message`](../../types/network.md#pov-distribution) as our `NetworkMessage`, which are sent and received by the [Network Bridge](../utility/network-bridge.md)
|
||||
|
||||
Here is the logic of the state machine:
|
||||
|
||||
*Overseer Signals*
|
||||
- On `ActiveLeavesUpdate(relay_parent)`:
|
||||
- For each relay-parent in the `activated` list:
|
||||
- Get the number of validators at that relay parent by querying the [Runtime API](../utility/runtime-api.md) for the validators and then counting them.
|
||||
- Create a blank entry in `relay_parent_state` under `relay_parent` with correct `n_validators` set.
|
||||
- For each relay-parent in the `deactivated` list:
|
||||
- Remove the entry for `relay_parent` from `relay_parent_state`.
|
||||
- On `Conclude`: conclude.
|
||||
|
||||
*PoV Distribution Messages*
|
||||
- On `FetchPoV(relay_parent, descriptor, response_channel)`
|
||||
- If there is no entry in `relay_parent_state` under `relay_parent`, ignore.
|
||||
- If there is a PoV under `descriptor.pov_hash` in the `known` map, send that PoV on the channel and return.
|
||||
- Otherwise, place the `response_channel` in the `fetching` map under `descriptor.pov_hash`.
|
||||
- If the `pov_hash` had no previous entry in `fetching` and there are `2 * n_validators` or fewer entries in the `fetching` set, send `NetworkMessage::Awaiting(relay_parent, vec![pov_hash])` to all peers.
|
||||
- On `DistributePoV(relay_parent, descriptor, PoV)`
|
||||
- If there is no entry in `relay_parent_state` under `relay_parent`, ignore.
|
||||
- Complete and remove any channels under `descriptor.pov_hash` in the `fetching` map.
|
||||
- Send `NetworkMessage::SendPoV(relay_parent, descriptor.pov_hash, PoV)` to all peers who have the `descriptor.pov_hash` in the set under `relay_parent` in the `peer.awaited` map and remove the entry from `peer.awaited`.
|
||||
- Note the PoV under `descriptor.pov_hash` in `known`.
|
||||
|
||||
*Network Bridge Updates*
|
||||
- On `PeerConnected(peer_id, observed_role)`
|
||||
- Make a fresh entry in the `peer_state` map for the `peer_id`.
|
||||
- On `PeerDisconnected(peer_id)`
|
||||
- Remove the entry for `peer_id` from the `peer_state` map.
|
||||
- On `PeerMessage(peer_id, bytes)`
|
||||
- If the bytes do not decode to a `NetworkMessage` or the `peer_id` has no entry in the `peer_state` map, report and ignore.
|
||||
- If this is `NetworkMessage::Awaiting(relay_parent, pov_hashes)`:
|
||||
- If there is no entry under `peer_state.awaited` for the `relay_parent`, report and ignore.
|
||||
- If `relay_parent` is not contained within `our_view`, report and ignore.
|
||||
- Otherwise, if the peer's `awaited` map combined with the `pov_hashes` would have more than ` 2 * relay_parent_state[relay_parent].n_validators` entries, report and ignore. Note that we are leaning on the property of the network bridge that it sets our view based on `activated` heads in `ActiveLeavesUpdate` signals.
|
||||
- For each new `pov_hash` in `pov_hashes`, if there is a `pov` under `pov_hash` in the `known` map, send the peer a `NetworkMessage::SendPoV(relay_parent, pov_hash, pov)`.
|
||||
- Otherwise, add the `pov_hash` to the `awaited` map
|
||||
- If this is `NetworkMessage::SendPoV(relay_parent, pov_hash, pov)`:
|
||||
- If there is no entry under `relay_parent` in `relay_parent_state` or no entry under `pov_hash` in our `fetching` map for that `relay_parent`, report and ignore.
|
||||
- If the blake2-256 hash of the pov doesn't equal `pov_hash`, report and ignore.
|
||||
- Complete and remove any listeners in the `fetching` map under `pov_hash`. However, leave an empty set of listeners in the `fetching` map to denote that this was something we once awaited. This will allow us to recognize peers who have sent us something we were expecting, but just a little late.
|
||||
- Add to `known` map.
|
||||
- Remove the `pov_hash` from the `peer.awaited` map, if any.
|
||||
- Send `NetworkMessage::SendPoV(relay_parent, descriptor.pov_hash, PoV)` to all peers who have the `descriptor.pov_hash` in the set under `relay_parent` in the `peer.awaited` map and remove the entry from `peer.awaited`.
|
||||
- On `PeerViewChange(peer_id, view)`
|
||||
- If Peer is unknown, ignore.
|
||||
- Ensure there is an entry under `relay_parent` for each `relay_parent` in `view` within the `peer.awaited` map, creating blank `awaited` lists as necessary.
|
||||
- Remove all entries under `peer.awaited` that are not within `view`.
|
||||
- For all hashes in `view` but were not within the old, send the peer all the keys in our `fetching` map under the block-based state for that hash - i.e. notify the peer of everything we are awaiting at that hash.
|
||||
- On `OurViewChange(view)`
|
||||
- Update `our_view` to `view`
|
||||
|
||||
@@ -136,13 +136,27 @@ This is a network protocol that receives messages of type [`AvailabilityDistribu
|
||||
|
||||
```rust
|
||||
enum AvailabilityDistributionMessage {
|
||||
/// Distribute an availability chunk to other validators.
|
||||
DistributeChunk(Hash, ErasureChunk),
|
||||
/// Fetch an erasure chunk from network by candidate hash and chunk index.
|
||||
FetchChunk(Hash, u32),
|
||||
/// Event from the network.
|
||||
/// An update on network state from the network bridge.
|
||||
NetworkBridgeUpdateV1(NetworkBridgeEvent<AvailabilityDistributionV1Message>),
|
||||
/// Incoming network request for an availability chunk.
|
||||
ChunkFetchingRequest(IncomingRequest<req_res_v1::ChunkFetchingRequest>),
|
||||
/// Incoming network request for a seconded PoV.
|
||||
PoVFetchingRequest(IncomingRequest<req_res_v1::PoVFetchingRequest>),
|
||||
/// Instruct availability distribution to fetch a remote PoV.
|
||||
///
|
||||
/// NOTE: The result of this fetch is not yet locally validated and could be bogus.
|
||||
FetchPoV {
|
||||
/// The relay parent giving the necessary context.
|
||||
relay_parent: Hash,
|
||||
/// Validator to fetch the PoV from.
|
||||
from_validator: ValidatorIndex,
|
||||
/// Candidate hash to fetch the PoV for.
|
||||
candidate_hash: CandidateHash,
|
||||
/// Expected hash of the PoV, a PoV not matching this hash will be rejected.
|
||||
pov_hash: Hash,
|
||||
/// Sender for getting back the result of this fetch.
|
||||
///
|
||||
/// The sender will be canceled if the fetching failed for some reason.
|
||||
tx: oneshot::Sender<PoV>,
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
Reference in New Issue
Block a user