mirror of
https://github.com/pezkuwichain/pezkuwi-subxt.git
synced 2026-06-13 17:31:05 +00:00
Markdown linter (#1309)
* Add markdown linting - add linter default rules - adapt rules to current code - fix the code for linting to pass - add CI check fix #1243 * Fix markdown for Substrate * Fix tooling install * Fix workflow * Add documentation * Remove trailing spaces * Update .github/.markdownlint.yaml Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix mangled markdown/lists * Fix captalization issues on known words
This commit is contained in:
Chevdor
@@ -9,13 +9,20 @@ The two data types:
|
||||
|
||||
For each of these data we have pruning rules that determine how long we need to keep that data available.
|
||||
|
||||
PoV hypothetically only need to be kept around until the block where the data was made fully available is finalized. However, disputes can revert finality, so we need to be a bit more conservative and we add a delay. We should keep the PoV until a block that finalized availability of it has been finalized for 1 day + 1 hour.
|
||||
PoV hypothetically only need to be kept around until the block where the data was made fully available is finalized.
|
||||
However, disputes can revert finality, so we need to be a bit more conservative and we add a delay. We should keep the
|
||||
PoV until a block that finalized availability of it has been finalized for 1 day + 1 hour.
|
||||
|
||||
Availability chunks need to be kept available until the dispute period for the corresponding candidate has ended. We can accomplish this by using the same criterion as the above. This gives us a pruning condition of the block finalizing availability of the chunk being final for 1 day + 1 hour.
|
||||
Availability chunks need to be kept available until the dispute period for the corresponding candidate has ended. We can
|
||||
accomplish this by using the same criterion as the above. This gives us a pruning condition of the block finalizing
|
||||
availability of the chunk being final for 1 day + 1 hour.
|
||||
|
||||
There is also the case where a validator commits to make a PoV available, but the corresponding candidate is never backed. In this case, we keep the PoV available for 1 hour.
|
||||
There is also the case where a validator commits to make a PoV available, but the corresponding candidate is never
|
||||
backed. In this case, we keep the PoV available for 1 hour.
|
||||
|
||||
There may be multiple competing blocks all ending the availability phase for a particular candidate. Until finality, it will be unclear which of those is actually the canonical chain, so the pruning records for PoVs and Availability chunks should keep track of all such blocks.
|
||||
There may be multiple competing blocks all ending the availability phase for a particular candidate. Until finality, it
|
||||
will be unclear which of those is actually the canonical chain, so the pruning records for PoVs and Availability chunks
|
||||
should keep track of all such blocks.
|
||||
|
||||
## Lifetime of the block data and chunks in storage
|
||||
|
||||
@@ -44,7 +51,8 @@ We use an underlying Key-Value database where we assume we have the following op
|
||||
|
||||
- `write(key, value)`
|
||||
- `read(key) -> Option<value>`
|
||||
- `iter_with_prefix(prefix) -> Iterator<(key, value)>` - gives all keys and values in lexicographical order where the key starts with `prefix`.
|
||||
- `iter_with_prefix(prefix) -> Iterator<(key, value)>` - gives all keys and values in lexicographical order where the
|
||||
key starts with `prefix`.
|
||||
|
||||
We use this database to encode the following schema:
|
||||
|
||||
@@ -57,7 +65,8 @@ We use this database to encode the following schema:
|
||||
("prune_by_time", Timestamp, CandidateHash) -> Option<()>
|
||||
```
|
||||
|
||||
Timestamps are the wall-clock seconds since Unix epoch. Timestamps and block numbers are both encoded as big-endian so lexicographic order is ascending.
|
||||
Timestamps are the wall-clock seconds since Unix epoch. Timestamps and block numbers are both encoded as big-endian so
|
||||
lexicographic order is ascending.
|
||||
|
||||
The meta information that we track per-candidate is defined as the `CandidateMeta` struct
|
||||
|
||||
@@ -80,9 +89,12 @@ enum State {
|
||||
}
|
||||
```
|
||||
|
||||
We maintain the invariant that if a candidate has a meta entry, its available data exists on disk if `data_available` is true. All chunks mentioned in the meta entry are available.
|
||||
We maintain the invariant that if a candidate has a meta entry, its available data exists on disk if `data_available` is
|
||||
true. All chunks mentioned in the meta entry are available.
|
||||
|
||||
Additionally, there is exactly one `prune_by_time` entry which holds the candidate hash unless the state is `Unfinalized`. There may be zero, one, or many "unfinalized" keys with the given candidate, and this will correspond to the `state` of the meta entry.
|
||||
Additionally, there is exactly one `prune_by_time` entry which holds the candidate hash unless the state is
|
||||
`Unfinalized`. There may be zero, one, or many "unfinalized" keys with the given candidate, and this will correspond to
|
||||
the `state` of the meta entry.
|
||||
|
||||
## Protocol
|
||||
|
||||
@@ -96,9 +108,15 @@ Output:
|
||||
|
||||
For each head in the `activated` list:
|
||||
|
||||
- Load all ancestors of the head back to the finalized block so we don't miss anything if import notifications are missed. If a `StoreChunk` message is received for a candidate which has no entry, then we will prematurely lose the data.
|
||||
- Note any new candidates backed in the head. Update the `CandidateMeta` for each. If the `CandidateMeta` does not exist, create it as `Unavailable` with the current timestamp. Register a `"prune_by_time"` entry based on the current timestamp + 1 hour.
|
||||
- Note any new candidate included in the head. Update the `CandidateMeta` for each, performing a transition from `Unavailable` to `Unfinalized` if necessary. That includes removing the `"prune_by_time"` entry. Add the head hash and number to the state, if unfinalized. Add an `"unfinalized"` entry for the block and candidate.
|
||||
- Load all ancestors of the head back to the finalized block so we don't miss anything if import notifications are
|
||||
missed. If a `StoreChunk` message is received for a candidate which has no entry, then we will prematurely lose the
|
||||
data.
|
||||
- Note any new candidates backed in the head. Update the `CandidateMeta` for each. If the `CandidateMeta` does not
|
||||
exist, create it as `Unavailable` with the current timestamp. Register a `"prune_by_time"` entry based on the current
|
||||
timestamp + 1 hour.
|
||||
- Note any new candidate included in the head. Update the `CandidateMeta` for each, performing a transition from
|
||||
`Unavailable` to `Unfinalized` if necessary. That includes removing the `"prune_by_time"` entry. Add the head hash and
|
||||
number to the state, if unfinalized. Add an `"unfinalized"` entry for the block and candidate.
|
||||
- The `CandidateEvent` runtime API can be used for this purpose.
|
||||
|
||||
On `OverseerSignal::BlockFinalized(finalized)` events:
|
||||
@@ -106,17 +124,22 @@ On `OverseerSignal::BlockFinalized(finalized)` events:
|
||||
- for each key in `iter_with_prefix("unfinalized")`
|
||||
- Stop if the key is beyond `("unfinalized, finalized)`
|
||||
- For each block number f that we encounter, load the finalized hash for that block.
|
||||
- The state of each `CandidateMeta` we encounter here must be `Unfinalized`, since we loaded the candidate from an `"unfinalized"` key.
|
||||
- The state of each `CandidateMeta` we encounter here must be `Unfinalized`, since we loaded the candidate from an
|
||||
`"unfinalized"` key.
|
||||
- For each candidate that we encounter under `f` and the finalized block hash,
|
||||
- Update the `CandidateMeta` to have `State::Finalized`. Remove all `"unfinalized"` entries from the old `Unfinalized` state.
|
||||
- Update the `CandidateMeta` to have `State::Finalized`. Remove all `"unfinalized"` entries from the old
|
||||
`Unfinalized` state.
|
||||
- Register a `"prune_by_time"` entry for the candidate based on the current time + 1 day + 1 hour.
|
||||
- For each candidate that we encounter under `f` which is not under the finalized block hash,
|
||||
- Remove all entries under `f` in the `Unfinalized` state.
|
||||
- If the `CandidateMeta` has state `Unfinalized` with an empty list of blocks, downgrade to `Unavailable` and re-schedule pruning under the timestamp + 1 hour. We do not prune here as the candidate still may be included in a descendant of the finalized chain.
|
||||
- If the `CandidateMeta` has state `Unfinalized` with an empty list of blocks, downgrade to `Unavailable` and
|
||||
re-schedule pruning under the timestamp + 1 hour. We do not prune here as the candidate still may be included in
|
||||
a descendant of the finalized chain.
|
||||
- Remove all `"unfinalized"` keys under `f`.
|
||||
- Update `last_finalized` = finalized.
|
||||
|
||||
This is roughly `O(n * m)` where n is the number of blocks finalized since the last update, and `m` is the number of parachains.
|
||||
This is roughly `O(n * m)` where n is the number of blocks finalized since the last update, and `m` is the number of
|
||||
parachains.
|
||||
|
||||
On `QueryAvailableData` message:
|
||||
|
||||
@@ -139,7 +162,8 @@ On `QueryChunk` message:
|
||||
On `QueryAllChunks` message:
|
||||
|
||||
- Query `("meta", candidate_hash)`. If `None`, send an empty response and return.
|
||||
- For all `1` bits in the `chunks_stored`, query `("chunk", candidate_hash, index)`. Ignore but warn on errors, and return a vector of all loaded chunks.
|
||||
- For all `1` bits in the `chunks_stored`, query `("chunk", candidate_hash, index)`. Ignore but warn on errors, and
|
||||
return a vector of all loaded chunks.
|
||||
|
||||
On `QueryChunkAvailability` message:
|
||||
|
||||
@@ -149,14 +173,17 @@ On `QueryChunkAvailability` message:
|
||||
|
||||
On `StoreChunk` message:
|
||||
|
||||
- If there is a `CandidateMeta` under the candidate hash, set the bit of the erasure-chunk in the `chunks_stored` bitfield to `1`. If it was not `1` already, write the chunk under `("chunk", candidate_hash, chunk_index)`.
|
||||
- If there is a `CandidateMeta` under the candidate hash, set the bit of the erasure-chunk in the `chunks_stored`
|
||||
bitfield to `1`. If it was not `1` already, write the chunk under `("chunk", candidate_hash, chunk_index)`.
|
||||
|
||||
This is `O(n)` in the size of the chunk.
|
||||
|
||||
On `StoreAvailableData` message:
|
||||
|
||||
- Compute the erasure root of the available data and compare it with `expected_erasure_root`. Return `StoreAvailableDataError::InvalidErasureRoot` on mismatch.
|
||||
- If there is no `CandidateMeta` under the candidate hash, create it with `State::Unavailable(now)`. Load the `CandidateMeta` otherwise.
|
||||
- Compute the erasure root of the available data and compare it with `expected_erasure_root`. Return
|
||||
`StoreAvailableDataError::InvalidErasureRoot` on mismatch.
|
||||
- If there is no `CandidateMeta` under the candidate hash, create it with `State::Unavailable(now)`. Load the
|
||||
`CandidateMeta` otherwise.
|
||||
- Store `data` under `("available", candidate_hash)` and set `data_available` to true.
|
||||
- Store each chunk under `("chunk", candidate_hash, index)` and set every bit in `chunks_stored` to `1`.
|
||||
|
||||
@@ -172,12 +199,13 @@ Every 5 minutes, run a pruning routine:
|
||||
- For each erasure chunk bit set, remove `("chunk", candidate_hash, bit_index)`.
|
||||
- If `data_available`, remove `("available", candidate_hash)`
|
||||
|
||||
This is O(n * m) in the amount of candidates and average size of the data stored. This is probably the most expensive operation but does not need
|
||||
to be run very often.
|
||||
This is O(n * m) in the amount of candidates and average size of the data stored. This is probably the most expensive
|
||||
operation but does not need to be run very often.
|
||||
|
||||
## Basic scenarios to test
|
||||
|
||||
Basically we need to test the correctness of data flow through state FSMs described earlier. These tests obviously assume that some mocking of time is happening.
|
||||
Basically we need to test the correctness of data flow through state FSMs described earlier. These tests obviously
|
||||
assume that some mocking of time is happening.
|
||||
|
||||
- Stored data that is never included pruned in necessary timeout
|
||||
- A block (and/or a chunk) is added to the store.
|
||||
|
||||
@@ -2,7 +2,8 @@
|
||||
|
||||
This subsystem is responsible for handling candidate validation requests. It is a simple request/response server.
|
||||
|
||||
A variety of subsystems want to know if a parachain block candidate is valid. None of them care about the detailed mechanics of how a candidate gets validated, just the results. This subsystem handles those details.
|
||||
A variety of subsystems want to know if a parachain block candidate is valid. None of them care about the detailed
|
||||
mechanics of how a candidate gets validated, just the results. This subsystem handles those details.
|
||||
|
||||
## Protocol
|
||||
|
||||
@@ -12,35 +13,53 @@ Output: Validation result via the provided response side-channel.
|
||||
|
||||
## Functionality
|
||||
|
||||
This subsystem groups the requests it handles in two categories: *candidate validation* and *PVF pre-checking*.
|
||||
This subsystem groups the requests it handles in two categories: *candidate validation* and *PVF pre-checking*.
|
||||
|
||||
The first category can be further subdivided in two request types: one which draws out validation data from the state, and another which accepts all validation data exhaustively. Validation returns three possible outcomes on the response channel: the candidate is valid, the candidate is invalid, or an internal error occurred.
|
||||
The first category can be further subdivided in two request types: one which draws out validation data from the state,
|
||||
and another which accepts all validation data exhaustively. Validation returns three possible outcomes on the response
|
||||
channel: the candidate is valid, the candidate is invalid, or an internal error occurred.
|
||||
|
||||
Parachain candidates are validated against their validation function: A piece of Wasm code that describes the state-transition of the parachain. Validation function execution is not metered. This means that an execution which is an infinite loop or simply takes too long must be forcibly exited by some other means. For this reason, we recommend dispatching candidate validation to be done on subprocesses which can be killed if they time-out.
|
||||
Parachain candidates are validated against their validation function: A piece of Wasm code that describes the
|
||||
state-transition of the parachain. Validation function execution is not metered. This means that an execution which is
|
||||
an infinite loop or simply takes too long must be forcibly exited by some other means. For this reason, we recommend
|
||||
dispatching candidate validation to be done on subprocesses which can be killed if they time-out.
|
||||
|
||||
Upon receiving a validation request, the first thing the candidate validation subsystem should do is make sure it has all the necessary parameters to the validation function. These are:
|
||||
Upon receiving a validation request, the first thing the candidate validation subsystem should do is make sure it has
|
||||
all the necessary parameters to the validation function. These are:
|
||||
* The Validation Function itself.
|
||||
* The [`CandidateDescriptor`](../../types/candidate.md#candidatedescriptor).
|
||||
* The [`ValidationData`](../../types/candidate.md#validationdata).
|
||||
* The [`PoV`](../../types/availability.md#proofofvalidity).
|
||||
|
||||
The second category is for PVF pre-checking. This is primarly used by the [PVF pre-checker](pvf-prechecker.md) subsystem.
|
||||
The second category is for PVF pre-checking. This is primarly used by the [PVF pre-checker](pvf-prechecker.md)
|
||||
subsystem.
|
||||
|
||||
### Determining Parameters
|
||||
|
||||
For a [`CandidateValidationMessage`][CVM]`::ValidateFromExhaustive`, these parameters are exhaustively provided.
|
||||
|
||||
For a [`CandidateValidationMessage`][CVM]`::ValidateFromChainState`, some more work needs to be done. Due to the uncertainty of Availability Cores (implemented in the [`Scheduler`](../../runtime/scheduler.md) module of the runtime), a candidate at a particular relay-parent and for a particular para may have two different valid validation-data to be executed under depending on what is assumed to happen if the para is occupying a core at the onset of the new block. This is encoded as an `OccupiedCoreAssumption` in the runtime API.
|
||||
For a [`CandidateValidationMessage`][CVM]`::ValidateFromChainState`, some more work needs to be done. Due to the
|
||||
uncertainty of Availability Cores (implemented in the [`Scheduler`](../../runtime/scheduler.md) module of the runtime),
|
||||
a candidate at a particular relay-parent and for a particular para may have two different valid validation-data to be
|
||||
executed under depending on what is assumed to happen if the para is occupying a core at the onset of the new block.
|
||||
This is encoded as an `OccupiedCoreAssumption` in the runtime API.
|
||||
|
||||
The way that we can determine which assumption the candidate is meant to be executed under is simply to do an exhaustive check of both possibilities based on the state of the relay-parent. First we fetch the validation data under the assumption that the block occupying becomes available. If the `validation_data_hash` of the `CandidateDescriptor` matches this validation data, we use that. Otherwise, if the `validation_data_hash` matches the validation data fetched under the `TimedOut` assumption, we use that. Otherwise, we return a `ValidationResult::Invalid` response and conclude.
|
||||
The way that we can determine which assumption the candidate is meant to be executed under is simply to do an exhaustive
|
||||
check of both possibilities based on the state of the relay-parent. First we fetch the validation data under the
|
||||
assumption that the block occupying becomes available. If the `validation_data_hash` of the `CandidateDescriptor`
|
||||
matches this validation data, we use that. Otherwise, if the `validation_data_hash` matches the validation data fetched
|
||||
under the `TimedOut` assumption, we use that. Otherwise, we return a `ValidationResult::Invalid` response and conclude.
|
||||
|
||||
Then, we can fetch the validation code from the runtime based on which type of candidate this is. This gives us all the parameters. The descriptor and PoV come from the request itself, and the other parameters have been derived from the state.
|
||||
Then, we can fetch the validation code from the runtime based on which type of candidate this is. This gives us all the
|
||||
parameters. The descriptor and PoV come from the request itself, and the other parameters have been derived from the
|
||||
state.
|
||||
|
||||
> TODO: This would be a great place for caching to avoid making lots of runtime requests. That would need a job, though.
|
||||
|
||||
### Execution of the Parachain Wasm
|
||||
|
||||
Once we have all parameters, we can spin up a background task to perform the validation in a way that doesn't hold up the entire event loop. Before invoking the validation function itself, this should first do some basic checks:
|
||||
Once we have all parameters, we can spin up a background task to perform the validation in a way that doesn't hold up
|
||||
the entire event loop. Before invoking the validation function itself, this should first do some basic checks:
|
||||
* The collator signature is valid
|
||||
* The PoV provided matches the `pov_hash` field of the descriptor
|
||||
|
||||
@@ -48,6 +67,8 @@ For more details please see [PVF Host and Workers](pvf-host-and-workers.md).
|
||||
|
||||
### Checking Validation Outputs
|
||||
|
||||
If we can assume the presence of the relay-chain state (that is, during processing [`CandidateValidationMessage`][CVM]`::ValidateFromChainState`) we can run all the checks that the relay-chain would run at the inclusion time thus confirming that the candidate will be accepted.
|
||||
If we can assume the presence of the relay-chain state (that is, during processing
|
||||
[`CandidateValidationMessage`][CVM]`::ValidateFromChainState`) we can run all the checks that the relay-chain would run
|
||||
at the inclusion time thus confirming that the candidate will be accepted.
|
||||
|
||||
[CVM]: ../../types/overseer-protocol.md#validationrequesttype
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# Chain API
|
||||
|
||||
The Chain API subsystem is responsible for providing a single point of access to chain state data via a set of pre-determined queries.
|
||||
The Chain API subsystem is responsible for providing a single point of access to chain state data via a set of
|
||||
pre-determined queries.
|
||||
|
||||
## Protocol
|
||||
|
||||
@@ -10,7 +11,8 @@ Output: None
|
||||
|
||||
## Functionality
|
||||
|
||||
On receipt of `ChainApiMessage`, answer the request and provide the response to the side-channel embedded within the request.
|
||||
On receipt of `ChainApiMessage`, answer the request and provide the response to the side-channel embedded within the
|
||||
request.
|
||||
|
||||
Currently, the following requests are supported:
|
||||
* Block hash to number
|
||||
|
||||
@@ -1,8 +1,12 @@
|
||||
# Chain Selection Subsystem
|
||||
|
||||
This subsystem implements the necessary metadata for the implementation of the [chain selection](../../protocol-chain-selection.md) portion of the protocol.
|
||||
This subsystem implements the necessary metadata for the implementation of the [chain
|
||||
selection](../../protocol-chain-selection.md) portion of the protocol.
|
||||
|
||||
The subsystem wraps a database component which maintains a view of the unfinalized chain and records the properties of each block: whether the block is **viable**, whether it is **stagnant**, and whether it is **reverted**. It should also maintain an updated set of active leaves in accordance with this view, which should be cheap to query. Leaves are ordered descending first by weight and then by block number.
|
||||
The subsystem wraps a database component which maintains a view of the unfinalized chain and records the properties of
|
||||
each block: whether the block is **viable**, whether it is **stagnant**, and whether it is **reverted**. It should also
|
||||
maintain an updated set of active leaves in accordance with this view, which should be cheap to query. Leaves are
|
||||
ordered descending first by weight and then by block number.
|
||||
|
||||
This subsystem needs to update its information on the unfinalized chain:
|
||||
* On every leaf-activated signal
|
||||
@@ -11,32 +15,47 @@ This subsystem needs to update its information on the unfinalized chain:
|
||||
* On every `ChainSelectionMessage::RevertBlocks`
|
||||
* Periodically, to detect stagnation.
|
||||
|
||||
Simple implementations of these updates do `O(n_unfinalized_blocks)` disk operations. If the amount of unfinalized blocks is relatively small, the updates should not take very much time. However, in cases where there are hundreds or thousands of unfinalized blocks the naive implementations of these update algorithms would have to be replaced with more sophisticated versions.
|
||||
Simple implementations of these updates do `O(n_unfinalized_blocks)` disk operations. If the amount of unfinalized
|
||||
blocks is relatively small, the updates should not take very much time. However, in cases where there are hundreds or
|
||||
thousands of unfinalized blocks the naive implementations of these update algorithms would have to be replaced with more
|
||||
sophisticated versions.
|
||||
|
||||
### `OverseerSignal::ActiveLeavesUpdate`
|
||||
## `OverseerSignal::ActiveLeavesUpdate`
|
||||
|
||||
Determine all new blocks implicitly referenced by any new active leaves and add them to the view. Update the set of viable leaves accordingly. The weights of imported blocks can be determined by the [`ChainApiMessage::BlockWeight`](../../types/overseer-protocol.md#chain-api-message).
|
||||
Determine all new blocks implicitly referenced by any new active leaves and add them to the view. Update the set of
|
||||
viable leaves accordingly. The weights of imported blocks can be determined by the
|
||||
[`ChainApiMessage::BlockWeight`](../../types/overseer-protocol.md#chain-api-message).
|
||||
|
||||
### `OverseerSignal::BlockFinalized`
|
||||
## `OverseerSignal::BlockFinalized`
|
||||
|
||||
Delete data for all orphaned chains and update all metadata descending from the new finalized block accordingly, along with the set of viable leaves. Note that finalizing a **reverted** or **stagnant** block means that the descendants of those blocks may lose that status because the definitions of those properties don't include the finalized chain. Update the set of viable leaves accordingly.
|
||||
Delete data for all orphaned chains and update all metadata descending from the new finalized block accordingly, along
|
||||
with the set of viable leaves. Note that finalizing a **reverted** or **stagnant** block means that the descendants of
|
||||
those blocks may lose that status because the definitions of those properties don't include the finalized chain. Update
|
||||
the set of viable leaves accordingly.
|
||||
|
||||
### `ChainSelectionMessage::Approved`
|
||||
## `ChainSelectionMessage::Approved`
|
||||
|
||||
Update the approval status of the referenced block. If the block was stagnant and thus non-viable and is now viable, then the metadata of all of its descendants needs to be updated as well, as they may no longer be stagnant either. Update the set of viable leaves accordingly.
|
||||
Update the approval status of the referenced block. If the block was stagnant and thus non-viable and is now viable,
|
||||
then the metadata of all of its descendants needs to be updated as well, as they may no longer be stagnant either.
|
||||
Update the set of viable leaves accordingly.
|
||||
|
||||
### `ChainSelectionMessage::Leaves`
|
||||
## `ChainSelectionMessage::Leaves`
|
||||
|
||||
Gets all leaves of the chain, i.e. block hashes that are suitable to build upon and have no suitable children. Supplies the leaves in descending order by score.
|
||||
Gets all leaves of the chain, i.e. block hashes that are suitable to build upon and have no suitable children. Supplies
|
||||
the leaves in descending order by score.
|
||||
|
||||
### `ChainSelectionMessage::BestLeafContaining`
|
||||
## `ChainSelectionMessage::BestLeafContaining`
|
||||
|
||||
If the required block is unknown or not viable, then return `None`. Iterate over all leaves in order of descending weight, returning the first leaf containing the required block in its chain, and `None` otherwise.
|
||||
If the required block is unknown or not viable, then return `None`. Iterate over all leaves in order of descending
|
||||
weight, returning the first leaf containing the required block in its chain, and `None` otherwise.
|
||||
|
||||
### `ChainSelectionMessage::RevertBlocks`
|
||||
This message indicates that a dispute has concluded against a parachain block candidate. The message passes along a vector containing the block number and block hash of each block where the disputed candidate was included. The passed blocks will be marked as reverted, and their descendants will be marked as non-viable.
|
||||
## `ChainSelectionMessage::RevertBlocks`
|
||||
This message indicates that a dispute has concluded against a parachain block candidate. The message passes along a
|
||||
vector containing the block number and block hash of each block where the disputed candidate was included. The passed
|
||||
blocks will be marked as reverted, and their descendants will be marked as non-viable.
|
||||
|
||||
|
||||
### Periodically
|
||||
## Periodically
|
||||
|
||||
Detect stagnant blocks and apply the stagnant definition to all descendants. Update the set of viable leaves accordingly.
|
||||
Detect stagnant blocks and apply the stagnant definition to all descendants. Update the set of viable leaves
|
||||
accordingly.
|
||||
|
||||
@@ -1,30 +1,43 @@
|
||||
# Network Bridge
|
||||
|
||||
One of the main features of the overseer/subsystem duality is to avoid shared ownership of resources and to communicate via message-passing. However, implementing each networking subsystem as its own network protocol brings a fair share of challenges.
|
||||
One of the main features of the overseer/subsystem duality is to avoid shared ownership of resources and to communicate
|
||||
via message-passing. However, implementing each networking subsystem as its own network protocol brings a fair share of
|
||||
challenges.
|
||||
|
||||
The most notable challenge is coordinating and eliminating race conditions of peer connection and disconnection events. If we have many network protocols that peers are supposed to be connected on, it is difficult to enforce that a peer is indeed connected on all of them or the order in which those protocols receive notifications that peers have connected. This becomes especially difficult when attempting to share peer state across protocols. All of the Parachain-Host's gossip protocols eliminate DoS with a data-dependency on current chain heads. However, it is inefficient and confusing to implement the logic for tracking our current chain heads as well as our peers' on each of those subsystems. Having one subsystem for tracking this shared state and distributing it to the others is an improvement in architecture and efficiency.
|
||||
The most notable challenge is coordinating and eliminating race conditions of peer connection and disconnection events.
|
||||
If we have many network protocols that peers are supposed to be connected on, it is difficult to enforce that a peer is
|
||||
indeed connected on all of them or the order in which those protocols receive notifications that peers have connected.
|
||||
This becomes especially difficult when attempting to share peer state across protocols. All of the Parachain-Host's
|
||||
gossip protocols eliminate DoS with a data-dependency on current chain heads. However, it is inefficient and confusing
|
||||
to implement the logic for tracking our current chain heads as well as our peers' on each of those subsystems. Having
|
||||
one subsystem for tracking this shared state and distributing it to the others is an improvement in architecture and
|
||||
efficiency.
|
||||
|
||||
One other piece of shared state to track is peer reputation. When peers are found to have provided value or cost, we adjust their reputation accordingly.
|
||||
One other piece of shared state to track is peer reputation. When peers are found to have provided value or cost, we
|
||||
adjust their reputation accordingly.
|
||||
|
||||
So in short, this Subsystem acts as a bridge between an actual network component and a subsystem's protocol. The implementation of the underlying network component is beyond the scope of this module. We make certain assumptions about the network component:
|
||||
* The network allows registering of protocols and multiple versions of each protocol.
|
||||
* The network handles version negotiation of protocols with peers and only connects the peer on the highest version of the protocol.
|
||||
* Each protocol has its own peer-set, although there may be some overlap.
|
||||
* The network provides peer-set management utilities for discovering the peer-IDs of validators and a means of dialing peers with given IDs.
|
||||
So in short, this Subsystem acts as a bridge between an actual network component and a subsystem's protocol. The
|
||||
implementation of the underlying network component is beyond the scope of this module. We make certain assumptions about
|
||||
the network component:
|
||||
- The network allows registering of protocols and multiple versions of each protocol.
|
||||
- The network handles version negotiation of protocols with peers and only connects the peer on the highest version of
|
||||
the protocol.
|
||||
- Each protocol has its own peer-set, although there may be some overlap.
|
||||
- The network provides peer-set management utilities for discovering the peer-IDs of validators and a means of dialing
|
||||
peers with given IDs.
|
||||
|
||||
|
||||
The network bridge makes use of the peer-set feature, but is not generic over peer-set. Instead, it exposes two peer-sets that event producers can attach to: `Validation` and `Collation`. More information can be found on the documentation of the [`NetworkBridgeMessage`][NBM].
|
||||
The network bridge makes use of the peer-set feature, but is not generic over peer-set. Instead, it exposes two
|
||||
peer-sets that event producers can attach to: `Validation` and `Collation`. More information can be found on the
|
||||
documentation of the [`NetworkBridgeMessage`][NBM].
|
||||
|
||||
## Protocol
|
||||
|
||||
Input: [`NetworkBridgeMessage`][NBM]
|
||||
|
||||
|
||||
Output:
|
||||
- [`ApprovalDistributionMessage`][AppD]`::NetworkBridgeUpdate`
|
||||
- [`BitfieldDistributionMessage`][BitD]`::NetworkBridgeUpdate`
|
||||
- [`CollatorProtocolMessage`][CollP]`::NetworkBridgeUpdate`
|
||||
- [`StatementDistributionMessage`][StmtD]`::NetworkBridgeUpdate`
|
||||
Output: - [`ApprovalDistributionMessage`][AppD]`::NetworkBridgeUpdate` -
|
||||
[`BitfieldDistributionMessage`][BitD]`::NetworkBridgeUpdate` -
|
||||
[`CollatorProtocolMessage`][CollP]`::NetworkBridgeUpdate` -
|
||||
[`StatementDistributionMessage`][StmtD]`::NetworkBridgeUpdate`
|
||||
|
||||
## Functionality
|
||||
|
||||
@@ -37,7 +50,8 @@ enum WireMessage<M> {
|
||||
}
|
||||
```
|
||||
|
||||
and instantiates this type twice, once using the [`ValidationProtocolV1`][VP1] message type, and once with the [`CollationProtocolV1`][CP1] message type.
|
||||
and instantiates this type twice, once using the [`ValidationProtocolV1`][VP1] message type, and once with the
|
||||
[`CollationProtocolV1`][CP1] message type.
|
||||
|
||||
```rust
|
||||
type ValidationV1Message = WireMessage<ValidationProtocolV1>;
|
||||
@@ -46,17 +60,21 @@ type CollationV1Message = WireMessage<CollationProtocolV1>;
|
||||
|
||||
### Startup
|
||||
|
||||
On startup, we register two protocols with the underlying network utility. One for validation and one for collation. We register only version 1 of each of these protocols.
|
||||
On startup, we register two protocols with the underlying network utility. One for validation and one for collation. We
|
||||
register only version 1 of each of these protocols.
|
||||
|
||||
### Main Loop
|
||||
|
||||
The bulk of the work done by this subsystem is in responding to network events, signals from the overseer, and messages from other subsystems.
|
||||
The bulk of the work done by this subsystem is in responding to network events, signals from the overseer, and messages
|
||||
from other subsystems.
|
||||
|
||||
Each network event is associated with a particular peer-set.
|
||||
|
||||
### Overseer Signal: `ActiveLeavesUpdate`
|
||||
|
||||
The `activated` and `deactivated` lists determine the evolution of our local view over time. A `ProtocolMessage::ViewUpdate` is issued to each connected peer on each peer-set, and a `NetworkBridgeEvent::OurViewChange` is issued to each event handler for each protocol.
|
||||
The `activated` and `deactivated` lists determine the evolution of our local view over time. A
|
||||
`ProtocolMessage::ViewUpdate` is issued to each connected peer on each peer-set, and a
|
||||
`NetworkBridgeEvent::OurViewChange` is issued to each event handler for each protocol.
|
||||
|
||||
We only send view updates if the node has indicated that it has finished major blockchain synchronization.
|
||||
|
||||
@@ -64,24 +82,31 @@ If we are connected to the same peer on both peer-sets, we will send the peer tw
|
||||
|
||||
### Overseer Signal: `BlockFinalized`
|
||||
|
||||
We update our view's `finalized_number` to the provided one and delay `ProtocolMessage::ViewUpdate` and `NetworkBridgeEvent::OurViewChange` till the next `ActiveLeavesUpdate`.
|
||||
We update our view's `finalized_number` to the provided one and delay `ProtocolMessage::ViewUpdate` and
|
||||
`NetworkBridgeEvent::OurViewChange` till the next `ActiveLeavesUpdate`.
|
||||
|
||||
### Network Event: `PeerConnected`
|
||||
|
||||
Issue a `NetworkBridgeEvent::PeerConnected` for each [Event Handler](#event-handlers) of the peer-set and negotiated protocol version of the peer. Also issue a `NetworkBridgeEvent::PeerViewChange` and send the peer our current view, but only if the node has indicated that it has finished major blockchain synchronization. Otherwise, we only send the peer an empty view.
|
||||
Issue a `NetworkBridgeEvent::PeerConnected` for each [Event Handler](#event-handlers) of the peer-set and negotiated
|
||||
protocol version of the peer. Also issue a `NetworkBridgeEvent::PeerViewChange` and send the peer our current view, but
|
||||
only if the node has indicated that it has finished major blockchain synchronization. Otherwise, we only send the peer
|
||||
an empty view.
|
||||
|
||||
### Network Event: `PeerDisconnected`
|
||||
|
||||
Issue a `NetworkBridgeEvent::PeerDisconnected` for each [Event Handler](#event-handlers) of the peer-set and negotiated protocol version of the peer.
|
||||
Issue a `NetworkBridgeEvent::PeerDisconnected` for each [Event Handler](#event-handlers) of the peer-set and negotiated
|
||||
protocol version of the peer.
|
||||
|
||||
### Network Event: `ProtocolMessage`
|
||||
|
||||
Map the message onto the corresponding [Event Handler](#event-handlers) based on the peer-set this message was received on and dispatch via overseer.
|
||||
Map the message onto the corresponding [Event Handler](#event-handlers) based on the peer-set this message was received
|
||||
on and dispatch via overseer.
|
||||
|
||||
### Network Event: `ViewUpdate`
|
||||
|
||||
- Check that the new view is valid and note it as the most recent view update of the peer on this peer-set.
|
||||
- Map a `NetworkBridgeEvent::PeerViewChange` onto the corresponding [Event Handler](#event-handlers) based on the peer-set this message was received on and dispatch via overseer.
|
||||
- Map a `NetworkBridgeEvent::PeerViewChange` onto the corresponding [Event Handler](#event-handlers) based on the
|
||||
peer-set this message was received on and dispatch via overseer.
|
||||
|
||||
### `ReportPeer`
|
||||
|
||||
@@ -108,22 +133,23 @@ Map the message onto the corresponding [Event Handler](#event-handlers) based on
|
||||
|
||||
### `NewGossipTopology`
|
||||
|
||||
- Map all `AuthorityDiscoveryId`s to `PeerId`s and issue a corresponding `NetworkBridgeUpdate`
|
||||
to all validation subsystems.
|
||||
- Map all `AuthorityDiscoveryId`s to `PeerId`s and issue a corresponding `NetworkBridgeUpdate` to all validation
|
||||
subsystems.
|
||||
|
||||
## Event Handlers
|
||||
|
||||
Network bridge event handlers are the intended recipients of particular network protocol messages. These are each a variant of a message to be sent via the overseer.
|
||||
Network bridge event handlers are the intended recipients of particular network protocol messages. These are each a
|
||||
variant of a message to be sent via the overseer.
|
||||
|
||||
### Validation V1
|
||||
|
||||
* `ApprovalDistributionV1Message -> ApprovalDistributionMessage::NetworkBridgeUpdate`
|
||||
* `BitfieldDistributionV1Message -> BitfieldDistributionMessage::NetworkBridgeUpdate`
|
||||
* `StatementDistributionV1Message -> StatementDistributionMessage::NetworkBridgeUpdate`
|
||||
- `ApprovalDistributionV1Message -> ApprovalDistributionMessage::NetworkBridgeUpdate`
|
||||
- `BitfieldDistributionV1Message -> BitfieldDistributionMessage::NetworkBridgeUpdate`
|
||||
- `StatementDistributionV1Message -> StatementDistributionMessage::NetworkBridgeUpdate`
|
||||
|
||||
### Collation V1
|
||||
|
||||
* `CollatorProtocolV1Message -> CollatorProtocolMessage::NetworkBridgeUpdate`
|
||||
- `CollatorProtocolV1Message -> CollatorProtocolMessage::NetworkBridgeUpdate`
|
||||
|
||||
[NBM]: ../../types/overseer-protocol.md#network-bridge-message
|
||||
[AppD]: ../../types/overseer-protocol.md#approval-distribution-message
|
||||
|
||||
@@ -1,28 +1,43 @@
|
||||
# Provisioner
|
||||
|
||||
Relay chain block authorship authority is governed by BABE and is beyond the scope of the Overseer and the rest of the subsystems. That said, ultimately the block author needs to select a set of backable parachain candidates and other consensus data, and assemble a block from them. This subsystem is responsible for providing the necessary data to all potential block authors.
|
||||
Relay chain block authorship authority is governed by BABE and is beyond the scope of the Overseer and the rest of the
|
||||
subsystems. That said, ultimately the block author needs to select a set of backable parachain candidates and other
|
||||
consensus data, and assemble a block from them. This subsystem is responsible for providing the necessary data to all
|
||||
potential block authors.
|
||||
|
||||
## Provisionable Data
|
||||
|
||||
There are several distinct types of provisionable data, but they share this property in common: all should eventually be included in a relay chain block.
|
||||
There are several distinct types of provisionable data, but they share this property in common: all should eventually be
|
||||
included in a relay chain block.
|
||||
|
||||
### Backed Candidates
|
||||
|
||||
The block author can choose 0 or 1 backed parachain candidates per parachain; the only constraint is that each backable candidate has the appropriate relay parent. However, the choice of a backed candidate must be the block author's. The provisioner subsystem is how those block authors make this choice in practice.
|
||||
The block author can choose 0 or 1 backed parachain candidates per parachain; the only constraint is that each backable
|
||||
candidate has the appropriate relay parent. However, the choice of a backed candidate must be the block author's. The
|
||||
provisioner subsystem is how those block authors make this choice in practice.
|
||||
|
||||
### Signed Bitfields
|
||||
|
||||
[Signed bitfields](../../types/availability.md#signed-availability-bitfield) are attestations from a particular validator about which candidates it believes are available. Those will only be provided on fresh leaves.
|
||||
[Signed bitfields](../../types/availability.md#signed-availability-bitfield) are attestations from a particular
|
||||
validator about which candidates it believes are available. Those will only be provided on fresh leaves.
|
||||
|
||||
### Misbehavior Reports
|
||||
|
||||
Misbehavior reports are self-contained proofs of misbehavior by a validator or group of validators. For example, it is very easy to verify a double-voting misbehavior report: the report contains two votes signed by the same key, advocating different outcomes. Concretely, misbehavior reports become inherents which cause dots to be slashed.
|
||||
Misbehavior reports are self-contained proofs of misbehavior by a validator or group of validators. For example, it is
|
||||
very easy to verify a double-voting misbehavior report: the report contains two votes signed by the same key, advocating
|
||||
different outcomes. Concretely, misbehavior reports become inherents which cause dots to be slashed.
|
||||
|
||||
Note that there is no mechanism in place which forces a block author to include a misbehavior report which it doesn't like, for example if it would be slashed by such a report. The chain's defense against this is to have a relatively long slash period, such that it's likely to encounter an honest author before the slash period expires.
|
||||
Note that there is no mechanism in place which forces a block author to include a misbehavior report which it doesn't
|
||||
like, for example if it would be slashed by such a report. The chain's defense against this is to have a relatively long
|
||||
slash period, such that it's likely to encounter an honest author before the slash period expires.
|
||||
|
||||
### Dispute Inherent
|
||||
|
||||
The dispute inherent is similar to a misbehavior report in that it is an attestation of misbehavior on the part of a validator or group of validators. Unlike a misbehavior report, it is not self-contained: resolution requires coordinated action by several validators. The canonical example of a dispute inherent involves an approval checker discovering that a set of validators has improperly approved an invalid parachain block: resolving this requires the entire validator set to re-validate the block, so that the minority can be slashed.
|
||||
The dispute inherent is similar to a misbehavior report in that it is an attestation of misbehavior on the part of a
|
||||
validator or group of validators. Unlike a misbehavior report, it is not self-contained: resolution requires coordinated
|
||||
action by several validators. The canonical example of a dispute inherent involves an approval checker discovering that
|
||||
a set of validators has improperly approved an invalid parachain block: resolving this requires the entire validator set
|
||||
to re-validate the block, so that the minority can be slashed.
|
||||
|
||||
Dispute resolution is complex and is explained in substantially more detail [here](../../runtime/disputes.md).
|
||||
|
||||
@@ -34,58 +49,85 @@ The subsystem should maintain a set of handles to Block Authorship Provisioning
|
||||
|
||||
- `ActiveLeavesUpdate`:
|
||||
- For each `activated` head:
|
||||
- spawn a Block Authorship Provisioning iteration with the given relay parent, storing a bidirectional channel with that iteration.
|
||||
- spawn a Block Authorship Provisioning iteration with the given relay parent, storing a bidirectional channel with
|
||||
that iteration.
|
||||
- For each `deactivated` head:
|
||||
- terminate the Block Authorship Provisioning iteration for the given relay parent, if any.
|
||||
- `Conclude`: Forward `Conclude` to all iterations, waiting a small amount of time for them to join, and then hard-exiting.
|
||||
- `Conclude`: Forward `Conclude` to all iterations, waiting a small amount of time for them to join, and then
|
||||
hard-exiting.
|
||||
|
||||
### On `ProvisionerMessage`
|
||||
|
||||
Forward the message to the appropriate Block Authorship Provisioning iteration, or discard if no appropriate iteration is currently active.
|
||||
Forward the message to the appropriate Block Authorship Provisioning iteration, or discard if no appropriate iteration
|
||||
is currently active.
|
||||
|
||||
### Per Provisioning Iteration
|
||||
|
||||
Input: [`ProvisionerMessage`](../../types/overseer-protocol.md#provisioner-message). Backed candidates come from the [Candidate Backing subsystem](../backing/candidate-backing.md), signed bitfields come from the [Bitfield Distribution subsystem](../availability/bitfield-distribution.md), and disputes come from the [Disputes Subsystem](../disputes/dispute-coordinator.md). Misbehavior reports are currently sent from the [Candidate Backing subsystem](../backing/candidate-backing.md) and contain the following misbehaviors:
|
||||
Input: [`ProvisionerMessage`](../../types/overseer-protocol.md#provisioner-message). Backed candidates come from the
|
||||
[Candidate Backing subsystem](../backing/candidate-backing.md), signed bitfields come from the [Bitfield Distribution
|
||||
subsystem](../availability/bitfield-distribution.md), and disputes come from the [Disputes
|
||||
Subsystem](../disputes/dispute-coordinator.md). Misbehavior reports are currently sent from the [Candidate Backing
|
||||
subsystem](../backing/candidate-backing.md) and contain the following misbehaviors:
|
||||
|
||||
1. `Misbehavior::ValidityDoubleVote`
|
||||
2. `Misbehavior::MultipleCandidates`
|
||||
3. `Misbehavior::UnauthorizedStatement`
|
||||
4. `Misbehavior::DoubleSign`
|
||||
|
||||
But we choose not to punish these forms of misbehavior for the time being. Risks from misbehavior are sufficiently mitigated at the protocol level via reputation changes. Punitive actions here may become desirable enough to dedicate time to in the future.
|
||||
But we choose not to punish these forms of misbehavior for the time being. Risks from misbehavior are sufficiently
|
||||
mitigated at the protocol level via reputation changes. Punitive actions here may become desirable enough to dedicate
|
||||
time to in the future.
|
||||
|
||||
At initialization, this subsystem has no outputs.
|
||||
|
||||
Block authors request the inherent data they should use for constructing the inherent in the block which contains parachain execution information.
|
||||
Block authors request the inherent data they should use for constructing the inherent in the block which contains
|
||||
parachain execution information.
|
||||
|
||||
## Block Production
|
||||
|
||||
When a validator is selected by BABE to author a block, it becomes a block producer. The provisioner is the subsystem best suited to choosing which specific backed candidates and availability bitfields should be assembled into the block. To engage this functionality, a `ProvisionerMessage::RequestInherentData` is sent; the response is a [`ParaInherentData`](../../types/runtime.md#parainherentdata). Each relay chain block backs at most one backable parachain block candidate per parachain. Additionally no further block candidate can be backed until the previous one either gets declared available or expired. If bitfields indicate that candidate A, predecessor of B, should be declared available, then B can be backed in the same relay block. Appropriate bitfields, as outlined in the section on [bitfield selection](#bitfield-selection), and any dispute statements should be attached as well.
|
||||
When a validator is selected by BABE to author a block, it becomes a block producer. The provisioner is the subsystem
|
||||
best suited to choosing which specific backed candidates and availability bitfields should be assembled into the block.
|
||||
To engage this functionality, a `ProvisionerMessage::RequestInherentData` is sent; the response is a
|
||||
[`ParaInherentData`](../../types/runtime.md#parainherentdata). Each relay chain block backs at most one backable
|
||||
parachain block candidate per parachain. Additionally no further block candidate can be backed until the previous one
|
||||
either gets declared available or expired. If bitfields indicate that candidate A, predecessor of B, should be declared
|
||||
available, then B can be backed in the same relay block. Appropriate bitfields, as outlined in the section on [bitfield
|
||||
selection](#bitfield-selection), and any dispute statements should be attached as well.
|
||||
|
||||
### Bitfield Selection
|
||||
|
||||
Our goal with respect to bitfields is simple: maximize availability. However, it's not quite as simple as always including all bitfields; there are constraints which still need to be met:
|
||||
Our goal with respect to bitfields is simple: maximize availability. However, it's not quite as simple as always
|
||||
including all bitfields; there are constraints which still need to be met:
|
||||
|
||||
- not more than one bitfield per validator
|
||||
- each 1 bit must correspond to an occupied core
|
||||
|
||||
Beyond that, a semi-arbitrary selection policy is fine. In order to meet the goal of maximizing availability, a heuristic of picking the bitfield with the greatest number of 1 bits set in the event of conflict is useful.
|
||||
Beyond that, a semi-arbitrary selection policy is fine. In order to meet the goal of maximizing availability, a
|
||||
heuristic of picking the bitfield with the greatest number of 1 bits set in the event of conflict is useful.
|
||||
|
||||
### Dispute Statement Selection
|
||||
|
||||
This is the point at which the block author provides further votes to active disputes or initiates new disputes in the runtime state.
|
||||
This is the point at which the block author provides further votes to active disputes or initiates new disputes in the
|
||||
runtime state.
|
||||
|
||||
The block-authoring logic of the runtime has an extra step between handling the inherent-data and producing the actual inherent call, which we assume performs the work of filtering out disputes which are not relevant to the on-chain state. Backing votes are always kept in the dispute statement set. This ensures we punish the maximum number of misbehaving backers.
|
||||
The block-authoring logic of the runtime has an extra step between handling the inherent-data and producing the actual
|
||||
inherent call, which we assume performs the work of filtering out disputes which are not relevant to the on-chain state.
|
||||
Backing votes are always kept in the dispute statement set. This ensures we punish the maximum number of misbehaving
|
||||
backers.
|
||||
|
||||
To select disputes:
|
||||
|
||||
- Issue a `DisputeCoordinatorMessage::RecentDisputes` message and wait for the response. This is a set of all disputes in recent sessions which we are aware of.
|
||||
- Issue a `DisputeCoordinatorMessage::RecentDisputes` message and wait for the response. This is a set of all disputes
|
||||
in recent sessions which we are aware of.
|
||||
|
||||
### Determining Bitfield Availability
|
||||
|
||||
An occupied core has a `CoreAvailability` bitfield. We also have a list of `SignedAvailabilityBitfield`s. We need to determine from these whether or not a core at a particular index has become available.
|
||||
An occupied core has a `CoreAvailability` bitfield. We also have a list of `SignedAvailabilityBitfield`s. We need to
|
||||
determine from these whether or not a core at a particular index has become available.
|
||||
|
||||
The key insight required is that `CoreAvailability` is transverse to the `SignedAvailabilityBitfield`s: if we conceptualize the list of bitfields as many rows, each bit of which is its own column, then `CoreAvailability` for a given core index is the vertical slice of bits in the set at that index.
|
||||
The key insight required is that `CoreAvailability` is transverse to the `SignedAvailabilityBitfield`s: if we
|
||||
conceptualize the list of bitfields as many rows, each bit of which is its own column, then `CoreAvailability` for a
|
||||
given core index is the vertical slice of bits in the set at that index.
|
||||
|
||||
To compute bitfield availability, then:
|
||||
|
||||
@@ -97,16 +139,22 @@ To compute bitfield availability, then:
|
||||
|
||||
### Candidate Selection: Prospective Parachains Mode
|
||||
|
||||
The state of the provisioner `PerRelayParent` tracks an important setting, `ProspectiveParachainsMode`. This setting determines which backable candidate selection method the provisioner uses.
|
||||
The state of the provisioner `PerRelayParent` tracks an important setting, `ProspectiveParachainsMode`. This setting
|
||||
determines which backable candidate selection method the provisioner uses.
|
||||
|
||||
`ProspectiveParachainsMode::Disabled` - The provisioner uses its own internal legacy candidate selection.
|
||||
`ProspectiveParachainsMode::Enabled` - The provisioner requests that [prospective parachains](../backing/prospective-parachains.md) provide selected candidates.
|
||||
`ProspectiveParachainsMode::Disabled` - The provisioner uses its own internal legacy candidate selection.
|
||||
`ProspectiveParachainsMode::Enabled` - The provisioner requests that [prospective
|
||||
parachains](../backing/prospective-parachains.md) provide selected candidates.
|
||||
|
||||
Candidates selected with `ProspectiveParachainsMode::Enabled` are able to benefit from the increased block production time asynchronous backing allows. For this reason all Polkadot protocol networks will eventually use prospective parachains candidate selection. Then legacy candidate selection will be removed as obsolete.
|
||||
Candidates selected with `ProspectiveParachainsMode::Enabled` are able to benefit from the increased block production
|
||||
time asynchronous backing allows. For this reason all Polkadot protocol networks will eventually use prospective
|
||||
parachains candidate selection. Then legacy candidate selection will be removed as obsolete.
|
||||
|
||||
### Prospective Parachains Candidate Selection
|
||||
|
||||
The goal of candidate selection is to determine which cores are free, and then to the degree possible, pick a candidate appropriate to each free core. In prospective parachains candidate selection the provisioner handles the former process while [prospective parachains](../backing/prospective-parachains.md) handles the latter.
|
||||
The goal of candidate selection is to determine which cores are free, and then to the degree possible, pick a candidate
|
||||
appropriate to each free core. In prospective parachains candidate selection the provisioner handles the former process
|
||||
while [prospective parachains](../backing/prospective-parachains.md) handles the latter.
|
||||
|
||||
To select backable candidates:
|
||||
|
||||
@@ -116,32 +164,50 @@ To select backable candidates:
|
||||
- The core is unscheduled and doesn’t need to be provisioned with a candidate
|
||||
- On `CoreState::Scheduled`
|
||||
- The core is unoccupied and scheduled to accept a backed block for a particular `para_id`.
|
||||
- The provisioner requests a backable candidate from [prospective parachains](../backing/prospective-parachains.md) with the desired relay parent, the core’s scheduled `para_id`, and an empty required path.
|
||||
- The provisioner requests a backable candidate from [prospective parachains](../backing/prospective-parachains.md)
|
||||
with the desired relay parent, the core’s scheduled `para_id`, and an empty required path.
|
||||
- On `CoreState::Occupied`
|
||||
- The availability core is occupied by a parachain block candidate pending availability. A further candidate need not be provided by the provisioner unless the core will be vacated this block. This is the case when either bitfields indicate the current core occupant has been made available or a timeout is reached.
|
||||
- The availability core is occupied by a parachain block candidate pending availability. A further candidate need
|
||||
not be provided by the provisioner unless the core will be vacated this block. This is the case when either
|
||||
bitfields indicate the current core occupant has been made available or a timeout is reached.
|
||||
- If `bitfields_indicate_availability`
|
||||
- If `Some(scheduled_core) = occupied_core.next_up_on_available`, the core will be vacated and in need of a provisioned candidate. The provisioner requests a backable candidate from [prospective parachains](../backing/prospective-parachains.md) with the core’s scheduled `para_id` and a required path with one entry. This entry corresponds to the parablock candidate previously occupying this core, which was made available and can be built upon even though it hasn’t been seen as included in a relay chain block yet. See the Required Path section below for more detail.
|
||||
- If `occupied_core.next_up_on_available` is `None`, then the core being vacated is unscheduled and doesn’t need to be provisioned with a candidate.
|
||||
- If `Some(scheduled_core) = occupied_core.next_up_on_available`, the core will be vacated and in need of a
|
||||
provisioned candidate. The provisioner requests a backable candidate from [prospective
|
||||
parachains](../backing/prospective-parachains.md) with the core’s scheduled `para_id` and a required path with
|
||||
one entry. This entry corresponds to the parablock candidate previously occupying this core, which was made
|
||||
available and can be built upon even though it hasn’t been seen as included in a relay chain block yet. See the
|
||||
Required Path section below for more detail.
|
||||
- If `occupied_core.next_up_on_available` is `None`, then the core being vacated is unscheduled and doesn’t need
|
||||
to be provisioned with a candidate.
|
||||
- Else-if `occupied_core.time_out_at == block_number`
|
||||
- If `Some(scheduled_core) = occupied_core.next_up_on_timeout`, the core will be vacated and in need of a provisioned candidate. A candidate is requested in exactly the same way as with `CoreState::Scheduled`.
|
||||
- Else the core being vacated is unscheduled and doesn’t need to be provisioned with a candidate
|
||||
The end result of this process is a vector of `CandidateHash`s, sorted in order of their core index.
|
||||
- If `Some(scheduled_core) = occupied_core.next_up_on_timeout`, the core will be vacated and in need of a
|
||||
provisioned candidate. A candidate is requested in exactly the same way as with `CoreState::Scheduled`.
|
||||
- Else the core being vacated is unscheduled and doesn’t need to be provisioned with a candidate The end result of
|
||||
this process is a vector of `CandidateHash`s, sorted in order of their core index.
|
||||
|
||||
#### Required Path
|
||||
|
||||
Required path is a parameter for `ProspectiveParachainsMessage::GetBackableCandidate`, which the provisioner sends in candidate selection.
|
||||
Required path is a parameter for `ProspectiveParachainsMessage::GetBackableCandidate`, which the provisioner sends in
|
||||
candidate selection.
|
||||
|
||||
An empty required path indicates that the requested candidate should be a direct child of the most recently included parablock for the given `para_id` as of the given relay parent.
|
||||
An empty required path indicates that the requested candidate should be a direct child of the most recently included
|
||||
parablock for the given `para_id` as of the given relay parent.
|
||||
|
||||
In contrast, a required path with one or more entries prompts [prospective parachains](../backing/prospective-parachains.md) to step forward through its fragment tree for the given `para_id` and relay parent until the desired parablock is reached. We then select a direct child of that parablock to pass to the provisioner.
|
||||
In contrast, a required path with one or more entries prompts [prospective
|
||||
parachains](../backing/prospective-parachains.md) to step forward through its fragment tree for the given `para_id` and
|
||||
relay parent until the desired parablock is reached. We then select a direct child of that parablock to pass to the
|
||||
provisioner.
|
||||
|
||||
The parablocks making up a required path do not need to have been previously seen as included in relay chain blocks. Thus the ability to provision backable candidates based on a required path effectively decouples backing from inclusion.
|
||||
The parablocks making up a required path do not need to have been previously seen as included in relay chain blocks.
|
||||
Thus the ability to provision backable candidates based on a required path effectively decouples backing from inclusion.
|
||||
|
||||
### Legacy Candidate Selection
|
||||
### Legacy Candidate Selection
|
||||
|
||||
Legacy candidate selection takes place in the provisioner. Thus the provisioner needs to keep an up to date record of all [backed_candidates](../../types/backing.md#backed-candidate) `PerRelayParent` to pick from.
|
||||
Legacy candidate selection takes place in the provisioner. Thus the provisioner needs to keep an up to date record of
|
||||
all [backed_candidates](../../types/backing.md#backed-candidate) `PerRelayParent` to pick from.
|
||||
|
||||
The goal of candidate selection is to determine which cores are free, and then to the degree possible, pick a candidate appropriate to each free core.
|
||||
The goal of candidate selection is to determine which cores are free, and then to the degree possible, pick a candidate
|
||||
appropriate to each free core.
|
||||
|
||||
To determine availability:
|
||||
|
||||
@@ -149,38 +215,54 @@ To determine availability:
|
||||
- For each core state:
|
||||
- On `CoreState::Scheduled`, then we can make an `OccupiedCoreAssumption::Free`.
|
||||
- On `CoreState::Occupied`, then we may be able to make an assumption:
|
||||
- If the bitfields indicate availability and there is a scheduled `next_up_on_available`, then we can make an `OccupiedCoreAssumption::Included`.
|
||||
- If the bitfields do not indicate availability, and there is a scheduled `next_up_on_time_out`, and `occupied_core.time_out_at == block_number_under_production`, then we can make an `OccupiedCoreAssumption::TimedOut`.
|
||||
- If the bitfields indicate availability and there is a scheduled `next_up_on_available`, then we can make an
|
||||
`OccupiedCoreAssumption::Included`.
|
||||
- If the bitfields do not indicate availability, and there is a scheduled `next_up_on_time_out`, and
|
||||
`occupied_core.time_out_at == block_number_under_production`, then we can make an
|
||||
`OccupiedCoreAssumption::TimedOut`.
|
||||
- If we did not make an `OccupiedCoreAssumption`, then continue on to the next core.
|
||||
- Now compute the core's `validation_data_hash`: get the `PersistedValidationData` from the runtime, given the known `ParaId` and `OccupiedCoreAssumption`;
|
||||
- Now compute the core's `validation_data_hash`: get the `PersistedValidationData` from the runtime, given the known
|
||||
`ParaId` and `OccupiedCoreAssumption`;
|
||||
- Find an appropriate candidate for the core.
|
||||
- There are two constraints: `backed_candidate.candidate.descriptor.para_id == scheduled_core.para_id && candidate.candidate.descriptor.validation_data_hash == computed_validation_data_hash`.
|
||||
- In the event that more than one candidate meets the constraints, selection between the candidates is arbitrary. However, not more than one candidate can be selected per core.
|
||||
- There are two constraints: `backed_candidate.candidate.descriptor.para_id == scheduled_core.para_id &&
|
||||
candidate.candidate.descriptor.validation_data_hash == computed_validation_data_hash`.
|
||||
- In the event that more than one candidate meets the constraints, selection between the candidates is arbitrary.
|
||||
However, not more than one candidate can be selected per core.
|
||||
|
||||
The end result of this process is a vector of `CandidateHash`s, sorted in order of their core index.
|
||||
|
||||
### Retrieving Full `BackedCandidate`s for Selected Hashes
|
||||
|
||||
Legacy candidate selection and prospective parachains candidate selection both leave us with a vector of `CandidateHash`s. These are passed to the backing subsystem with `CandidateBackingMessage::GetBackedCandidates`.
|
||||
Legacy candidate selection and prospective parachains candidate selection both leave us with a vector of
|
||||
`CandidateHash`s. These are passed to the backing subsystem with `CandidateBackingMessage::GetBackedCandidates`.
|
||||
|
||||
The response is a vector of `BackedCandidate`s, sorted in order of their core index and ready to be provisioned to block authoring. The candidate selection and retrieval process should select at maximum one candidate which upgrades the runtime validation code.
|
||||
The response is a vector of `BackedCandidate`s, sorted in order of their core index and ready to be provisioned to block
|
||||
authoring. The candidate selection and retrieval process should select at maximum one candidate which upgrades the
|
||||
runtime validation code.
|
||||
|
||||
## Glossary
|
||||
|
||||
- **Relay-parent:**
|
||||
- A particular relay-chain block which serves as an anchor and reference point for processes and data which depend on relay-chain state.
|
||||
- **Active Leaf:**
|
||||
- A relay chain block which is the head of an active fork of the relay chain.
|
||||
- **Relay-parent:**
|
||||
- A particular relay-chain block which serves as an anchor and reference point for processes and data which depend on
|
||||
relay-chain state.
|
||||
- **Active Leaf:**
|
||||
- A relay chain block which is the head of an active fork of the relay chain.
|
||||
- Block authorship provisioning jobs are spawned per active leaf and concluded for any leaves which become inactive.
|
||||
- **Candidate Selection:**
|
||||
- **Candidate Selection:**
|
||||
- The process by which the provisioner selects backable parachain block candidates to pass to block authoring.
|
||||
- Two versions, prospective parachains candidate selection and legacy candidate selection. See their respective protocol sections for details.
|
||||
- **Availability Core:**
|
||||
- Often referred to simply as "cores", availability cores are an abstraction used for resource management. For the provisioner, availability cores are most relevant in that core states determine which `para_id`s to provision backable candidates for.
|
||||
- For more on availability cores see [Scheduler Module: Availability Cores](../../runtime/scheduler.md#availability-cores)
|
||||
- Two versions, prospective parachains candidate selection and legacy candidate selection. See their respective
|
||||
protocol sections for details.
|
||||
- **Availability Core:**
|
||||
- Often referred to simply as "cores", availability cores are an abstraction used for resource management. For the
|
||||
provisioner, availability cores are most relevant in that core states determine which `para_id`s to provision
|
||||
backable candidates for.
|
||||
- For more on availability cores see [Scheduler Module: Availability
|
||||
Cores](../../runtime/scheduler.md#availability-cores)
|
||||
- **Availability Bitfield:**
|
||||
- Often referred to simply as a "bitfield", an availability bitfield represents the view of parablock candidate availability from a particular validator's perspective. Each bit in the bitfield corresponds to a single [availability core](../../runtime-api/availability-cores.md).
|
||||
- Often referred to simply as a "bitfield", an availability bitfield represents the view of parablock candidate
|
||||
availability from a particular validator's perspective. Each bit in the bitfield corresponds to a single
|
||||
[availability core](../../runtime-api/availability-cores.md).
|
||||
- For more on availability bitfields see [availability](../../types/availability.md)
|
||||
- **Backable vs. Backed:**
|
||||
- Note that we sometimes use "backed" to refer to candidates that are "backable", but not yet backed on chain.
|
||||
- Backable means that a quorum of the candidate's assigned backing group have provided signed affirming statements.
|
||||
- Backable means that a quorum of the candidate's assigned backing group have provided signed affirming statements.
|
||||
|
||||
@@ -1,45 +1,70 @@
|
||||
# PVF Pre-checker
|
||||
|
||||
The PVF pre-checker is a subsystem that is responsible for watching the relay chain for new PVFs that require pre-checking. Head over to [overview] for the PVF pre-checking process overview.
|
||||
The PVF pre-checker is a subsystem that is responsible for watching the relay chain for new PVFs that require
|
||||
pre-checking. Head over to [overview] for the PVF pre-checking process overview.
|
||||
|
||||
## Protocol
|
||||
|
||||
There is no dedicated input mechanism for PVF pre-checker. Instead, PVF pre-checker looks on the `ActiveLeavesUpdate` event stream for work.
|
||||
There is no dedicated input mechanism for PVF pre-checker. Instead, PVF pre-checker looks on the `ActiveLeavesUpdate`
|
||||
event stream for work.
|
||||
|
||||
This subsytem does not produce any output messages either. The subsystem will, however, send messages to the [Runtime API] subsystem to query for the pending PVFs and to submit votes. In addition to that, it will also communicate with [Candidate Validation] Subsystem to request PVF pre-check.
|
||||
This subsytem does not produce any output messages either. The subsystem will, however, send messages to the [Runtime
|
||||
API] subsystem to query for the pending PVFs and to submit votes. In addition to that, it will also communicate with
|
||||
[Candidate Validation] Subsystem to request PVF pre-check.
|
||||
|
||||
## Functionality
|
||||
|
||||
If the node is running in a collator mode, this subsystem will be disabled. The PVF pre-checker subsystem keeps track of the PVFs that are relevant for the subsystem.
|
||||
If the node is running in a collator mode, this subsystem will be disabled. The PVF pre-checker subsystem keeps track of
|
||||
the PVFs that are relevant for the subsystem.
|
||||
|
||||
To be relevant for the subsystem, a PVF must be returned by the [`pvfs_require_precheck` runtime API][PVF pre-checking runtime API] in any of the active leaves. If the PVF is not present in any of the active leaves, it ceases to be relevant.
|
||||
To be relevant for the subsystem, a PVF must be returned by the [`pvfs_require_precheck` runtime API][PVF pre-checking
|
||||
runtime API] in any of the active leaves. If the PVF is not present in any of the active leaves, it ceases to be
|
||||
relevant.
|
||||
|
||||
When a PVF just becomes relevant, the subsystem will send a message to the [Candidate Validation] subsystem asking for the pre-check.
|
||||
When a PVF just becomes relevant, the subsystem will send a message to the [Candidate Validation] subsystem asking for
|
||||
the pre-check.
|
||||
|
||||
Upon receving a message from the candidate-validation subsystem, the pre-checker will note down that the PVF has its judgement and will also sign and submit a [`PvfCheckStatement`][PvfCheckStatement] via the [`submit_pvf_check_statement` runtime API][PVF pre-checking runtime API]. In case, a judgement was received for a PVF that is no longer in view it is ignored.
|
||||
Upon receving a message from the candidate-validation subsystem, the pre-checker will note down that the PVF has its
|
||||
judgement and will also sign and submit a [`PvfCheckStatement`][PvfCheckStatement] via the [`submit_pvf_check_statement`
|
||||
runtime API][PVF pre-checking runtime API]. In case, a judgement was received for a PVF that is no longer in view it is
|
||||
ignored.
|
||||
|
||||
Since a vote only is valid during [one session][overview], the subsystem will have to resign and submit the statements for the new session. The new session is assumed to be started if at least one of the leaves has a greater session index that was previously observed in any of the leaves.
|
||||
Since a vote only is valid during [one session][overview], the subsystem will have to resign and submit the statements
|
||||
for the new session. The new session is assumed to be started if at least one of the leaves has a greater session index
|
||||
that was previously observed in any of the leaves.
|
||||
|
||||
The subsystem tracks all the statements that it submitted within a session. If for some reason a PVF became irrelevant and then becomes relevant again, the subsystem will not submit a new statement for that PVF within the same session.
|
||||
The subsystem tracks all the statements that it submitted within a session. If for some reason a PVF became irrelevant
|
||||
and then becomes relevant again, the subsystem will not submit a new statement for that PVF within the same session.
|
||||
|
||||
If the node is not in the active validator set, it will still perform all the checks. However, it will only submit the check statements when the node is in the active validator set.
|
||||
If the node is not in the active validator set, it will still perform all the checks. However, it will only submit the
|
||||
check statements when the node is in the active validator set.
|
||||
|
||||
### Rejecting failed PVFs
|
||||
|
||||
It is possible that the candidate validation was not able to check the PVF, e.g. if it timed out. In that case, the PVF pre-checker will vote against it. This is considered safe, as there is no slashing for being on the wrong side of a pre-check vote.
|
||||
It is possible that the candidate validation was not able to check the PVF, e.g. if it timed out. In that case, the PVF
|
||||
pre-checker will vote against it. This is considered safe, as there is no slashing for being on the wrong side of a
|
||||
pre-check vote.
|
||||
|
||||
Rejecting instead of abstaining is better in several ways:
|
||||
|
||||
1. Conclusion is reached faster - we have actual votes, instead of relying on a timeout.
|
||||
1. Being strict in pre-checking makes it safer to be more lenient in preparation errors afterwards. Hence we have more leeway in avoiding raising dubious disputes, without making things less secure.
|
||||
1. Being strict in pre-checking makes it safer to be more lenient in preparation errors afterwards. Hence we have more
|
||||
leeway in avoiding raising dubious disputes, without making things less secure.
|
||||
|
||||
Also, if we only abstain, an attacker can specially craft a PVF wasm blob so that it will fail on e.g. 50% of the validators. In that case a supermajority will never be reached and the vote will repeat multiple times, most likely with the same result (since all votes are cleared on a session change). This is avoided by rejecting failed PVFs, and by only requiring 1/3 of validators to reject a PVF to reach a decision.
|
||||
Also, if we only abstain, an attacker can specially craft a PVF wasm blob so that it will fail on e.g. 50% of the
|
||||
validators. In that case a supermajority will never be reached and the vote will repeat multiple times, most likely with
|
||||
the same result (since all votes are cleared on a session change). This is avoided by rejecting failed PVFs, and by only
|
||||
requiring 1/3 of validators to reject a PVF to reach a decision.
|
||||
|
||||
### Note on Disputes
|
||||
|
||||
Having a pre-checking phase allows us to make certain assumptions later when preparing the PVF for execution. If a runtime passed pre-checking, then we know that the runtime should be valid, and therefore any issue during preparation for execution can be assumed to be a local problem on the current node.
|
||||
Having a pre-checking phase allows us to make certain assumptions later when preparing the PVF for execution. If a
|
||||
runtime passed pre-checking, then we know that the runtime should be valid, and therefore any issue during preparation
|
||||
for execution can be assumed to be a local problem on the current node.
|
||||
|
||||
For this reason, even deterministic preparation errors should not trigger disputes. And since we do not dispute as a result of the pre-checking phase, as stated above, it should be impossible for preparation in general to result in disputes.
|
||||
For this reason, even deterministic preparation errors should not trigger disputes. And since we do not dispute as a
|
||||
result of the pre-checking phase, as stated above, it should be impossible for preparation in general to result in
|
||||
disputes.
|
||||
|
||||
[overview]: ../../pvf-prechecking.md
|
||||
[Runtime API]: runtime-api.md
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# Runtime API
|
||||
|
||||
The Runtime API subsystem is responsible for providing a single point of access to runtime state data via a set of pre-determined queries. This prevents shared ownership of a blockchain client resource by providing
|
||||
The Runtime API subsystem is responsible for providing a single point of access to runtime state data via a set of
|
||||
pre-determined queries. This prevents shared ownership of a blockchain client resource by providing
|
||||
|
||||
## Protocol
|
||||
|
||||
@@ -10,8 +11,11 @@ Output: None
|
||||
|
||||
## Functionality
|
||||
|
||||
On receipt of `RuntimeApiMessage::Request(relay_parent, request)`, answer the request using the post-state of the `relay_parent` provided and provide the response to the side-channel embedded within the request.
|
||||
On receipt of `RuntimeApiMessage::Request(relay_parent, request)`, answer the request using the post-state of the
|
||||
`relay_parent` provided and provide the response to the side-channel embedded within the request.
|
||||
|
||||
## Jobs
|
||||
|
||||
> TODO Don't limit requests based on parent hash, but limit caching. No caching should be done for any requests on `relay_parent`s that are not active based on `ActiveLeavesUpdate` messages. Maybe with some leeway for things that have just been stopped.
|
||||
> TODO Don't limit requests based on parent hash, but limit caching. No caching should be done for any requests on
|
||||
> `relay_parent`s that are not active based on `ActiveLeavesUpdate` messages. Maybe with some leeway for things that
|
||||
> have just been stopped.
|
||||
|
||||
Reference in New Issue
Block a user