mirror of
https://github.com/pezkuwichain/pezkuwi-subxt.git
synced 2026-04-30 09:37:55 +00:00
WIP: CI: add spellcheck (#3421)
* CI: add spellcheck * revert me * CI: explicit command for spellchecker * spellcheck: edit misspells * CI: run spellcheck on diff * spellcheck: edits * spellcheck: edit misspells * spellcheck: add rules * spellcheck: mv configs * spellcheck: more edits * spellcheck: chore * spellcheck: one more thing * spellcheck: and another one * spellcheck: seems like it doesn't get to an end * spellcheck: new words after rebase * spellcheck: new words appearing out of nowhere * chore * review edits * more review edits * more edits * wonky behavior * wonky behavior 2 * wonky behavior 3 * change git behavior * spellcheck: another bunch of new edits * spellcheck: new words are koming out of nowhere * CI: finding the master * CI: fetching master implicitly * CI: undebug * new errors * a bunch of new edits * and some more * Update node/core/approval-voting/src/approval_db/v1/mod.rs Co-authored-by: Andronik Ordian <write@reusable.software> * Update xcm/xcm-executor/src/assets.rs Co-authored-by: Andronik Ordian <write@reusable.software> * Apply suggestions from code review Co-authored-by: Andronik Ordian <write@reusable.software> * Suggestions from the code review * CI: scan only changed files Co-authored-by: Andronik Ordian <write@reusable.software>
This commit is contained in:
Denis Pisarev
@@ -26,6 +26,6 @@ The Node-side code comes with a set of assumptions that we build upon. These ass
|
||||
We assume the following constraints regarding provided basic functionality:
|
||||
* The underlying **consensus** algorithm, whether it is BABE or SASSAFRAS is implemented.
|
||||
* There is a **chain synchronization** protocol which will search for and download the longest available chains at all times.
|
||||
* The **state** of all blocks at the head of the chain is available. There may be **state pruning** such that state of the last `k` blocks behind the last finalized block are available, as well as the state of all their descendents. This assumption implies that the state of all active leaves and their last `k` ancestors are all available. The underlying implementation is expected to support `k` of a few hundred blocks, but we reduce this to a very conservative `k=5` for our purposes.
|
||||
* The **state** of all blocks at the head of the chain is available. There may be **state pruning** such that state of the last `k` blocks behind the last finalized block are available, as well as the state of all their descendants. This assumption implies that the state of all active leaves and their last `k` ancestors are all available. The underlying implementation is expected to support `k` of a few hundred blocks, but we reduce this to a very conservative `k=5` for our purposes.
|
||||
* There is an underlying **networking** framework which provides **peer discovery** services which will provide us with peers and will not create "loopback" connections to our own node. The number of peers we will have is assumed to be bounded at 1000.
|
||||
* There is a **transaction pool** and a **transaction propagation** mechanism which maintains a set of current transactions and distributes to connected peers. Current transactions are those which are not outdated relative to some "best" fork of the chain, which is part of the active heads, and have not been included in the best fork.
|
||||
|
||||
@@ -4,9 +4,9 @@ Reading the [section on the approval protocol](../../protocol-approval.md) will
|
||||
|
||||
Approval votes are split into two parts: Assignments and Approvals. Validators first broadcast their assignment to indicate intent to check a candidate. Upon successfully checking, they broadcast an approval vote. If a validator doesn't broadcast their approval vote shortly after issuing an assignment, this is an indication that they are being prevented from recovering or validating the block data and that more validators should self-select to check the candidate. This is known as a "no-show".
|
||||
|
||||
The core of this subsystem is a Tick-based timer loop, where Ticks are 500ms. We also reason about time in terms of DelayTranches, which measure the number of ticks elapsed since a block was produced. We track metadata for all un-finalized but included candidates. We compute our local assignments to check each candidate, as well as which DelayTranche those assignments may be minimally triggered at. As the same candidate may appear in more than one block, we must produce our potential assignments for each (Block, Candidate) pair. The timing loop is based on waiting for assignments to become no-shows or waiting to broadcast and begin our own assignment to check.
|
||||
The core of this subsystem is a Tick-based timer loop, where Ticks are 500ms. We also reason about time in terms of `DelayTranche`s, which measure the number of ticks elapsed since a block was produced. We track metadata for all un-finalized but included candidates. We compute our local assignments to check each candidate, as well as which `DelayTranche` those assignments may be minimally triggered at. As the same candidate may appear in more than one block, we must produce our potential assignments for each (Block, Candidate) pair. The timing loop is based on waiting for assignments to become no-shows or waiting to broadcast and begin our own assignment to check.
|
||||
|
||||
Another main component of this subsystem is the logic for determining when a (Block, Candidate) pair has been approved and when to broadcast and trigger our own assignment. Once a (Block, Candidate) pair has been approved, we mark a corresponding bit in the BlockEntry that indicates the candidate has been approved under the block. When we trigger our own assignment, we broadcast it via Approval Distribution, begin fetching the data from Availability Recovery, and then pass it through to the Candidate Validation. Once these steps are successful, we issue our approval vote. If any of these steps fail, we don't issue any vote and will "no-show" from the perspective of other validators. In the future we will initiate disputes as well.
|
||||
Another main component of this subsystem is the logic for determining when a (Block, Candidate) pair has been approved and when to broadcast and trigger our own assignment. Once a (Block, Candidate) pair has been approved, we mark a corresponding bit in the `BlockEntry` that indicates the candidate has been approved under the block. When we trigger our own assignment, we broadcast it via Approval Distribution, begin fetching the data from Availability Recovery, and then pass it through to the Candidate Validation. Once these steps are successful, we issue our approval vote. If any of these steps fail, we don't issue any vote and will "no-show" from the perspective of other validators. In the future we will initiate disputes as well.
|
||||
|
||||
Where this all fits into Polkadot is via block finality. Our goal is to not finalize any block containing a candidate that is not approved. We provide a hook for a custom GRANDPA voting rule - GRANDPA makes requests of the form (target, minimum) consisting of a target block (i.e. longest chain) that it would like to finalize, and a minimum block which, due to the rules of GRANDPA, must be voted on. The minimum is typically the last finalized block, but may be beyond it, in the case of having a last-round-estimate beyond the last finalized. Thus, our goal is to inform GRANDPA of some block between target and minimum which we believe can be finalized safely. We do this by iterating backwards from the target to the minimum and finding the longest continuous chain from minimum where all candidates included by those blocks have been approved.
|
||||
|
||||
@@ -164,23 +164,23 @@ Main loop:
|
||||
|
||||
#### `OverseerSignal::BlockFinalized`
|
||||
|
||||
On receiving an `OverseerSignal::BlockFinalized(h)`, we fetch the block number `b` of that block from the ChainApi subsystem. We update our `StoredBlockRange` to begin at `b+1`. Additionally, we remove all block entries and candidates referenced by them up to and including `b`. Lastly, we prune out all descendents of `h` transitively: when we remove a `BlockEntry` with number `b` that is not equal to `h`, we recursively delete all the `BlockEntry`s referenced as children. We remove the `block_assignments` entry for the block hash and if `block_assignments` is now empty, remove the `CandidateEntry`. We also update each of the `BlockNumber -> Vec<Hash>` keys in the database to reflect the blocks at that height, clearing if empty.
|
||||
On receiving an `OverseerSignal::BlockFinalized(h)`, we fetch the block number `b` of that block from the `ChainApi` subsystem. We update our `StoredBlockRange` to begin at `b+1`. Additionally, we remove all block entries and candidates referenced by them up to and including `b`. Lastly, we prune out all descendants of `h` transitively: when we remove a `BlockEntry` with number `b` that is not equal to `h`, we recursively delete all the `BlockEntry`s referenced as children. We remove the `block_assignments` entry for the block hash and if `block_assignments` is now empty, remove the `CandidateEntry`. We also update each of the `BlockNumber -> Vec<Hash>` keys in the database to reflect the blocks at that height, clearing if empty.
|
||||
|
||||
|
||||
#### `OverseerSignal::ActiveLeavesUpdate`
|
||||
|
||||
On receiving an `OverseerSignal::ActiveLeavesUpdate(update)`:
|
||||
* We determine the set of new blocks that were not in our previous view. This is done by querying the ancestry of all new items in the view and contrasting against the stored `BlockNumber`s. Typically, there will be only one new block. We fetch the headers and information on these blocks from the ChainApi subsystem. Stale leaves in the update can be ignored.
|
||||
* We determine the set of new blocks that were not in our previous view. This is done by querying the ancestry of all new items in the view and contrasting against the stored `BlockNumber`s. Typically, there will be only one new block. We fetch the headers and information on these blocks from the `ChainApi` subsystem. Stale leaves in the update can be ignored.
|
||||
* We update the `StoredBlockRange` and the `BlockNumber` maps.
|
||||
* We use the RuntimeApiSubsystem to determine information about these blocks. It is generally safe to assume that runtime state is available for recent, unfinalized blocks. In the case that it isn't, it means that we are catching up to the head of the chain and needn't worry about assignments to those blocks anyway, as the security assumption of the protocol tolerates nodes being temporarily offline or out-of-date.
|
||||
* We use the `RuntimeApiSubsystem` to determine information about these blocks. It is generally safe to assume that runtime state is available for recent, unfinalized blocks. In the case that it isn't, it means that we are catching up to the head of the chain and needn't worry about assignments to those blocks anyway, as the security assumption of the protocol tolerates nodes being temporarily offline or out-of-date.
|
||||
* We fetch the set of candidates included by each block by dispatching a `RuntimeApiRequest::CandidateEvents` and checking the `CandidateIncluded` events.
|
||||
* We fetch the session of the block by dispatching a `session_index_for_child` request with the parent-hash of the block.
|
||||
* If the `session index - APPROVAL_SESSIONS > state.earliest_session`, then bump `state.earliest_sessions` to that amount and prune earlier sessions.
|
||||
* If the session isn't in our `state.session_info`, load the session info for it and for all sessions since the earliest-session, including the earliest-session, if that is missing. And it can be, just after pruning, if we've done a big jump forward, as is the case when we've just finished chain synchronization.
|
||||
* If any of the runtime API calls fail, we just warn and skip the block.
|
||||
* We use the RuntimeApiSubsystem to determine the set of candidates included in these blocks and use BABE logic to determine the slot number and VRF of the blocks.
|
||||
* We use the `RuntimeApiSubsystem` to determine the set of candidates included in these blocks and use BABE logic to determine the slot number and VRF of the blocks.
|
||||
* We also note how late we appear to have received the block. We create a `BlockEntry` for each block and a `CandidateEntry` for each candidate obtained from `CandidateIncluded` events after making a `RuntimeApiRequest::CandidateEvents` request.
|
||||
* For each candidate, if the amount of needed approvals is more than the validators remaining after the backing group of the candidate is subtracted, then the candidate is insta-approved as approval would be impossible otherwise. If all candidates in the block are insta-approved, or there are no candidates in the block, then the block is insta-approved. If the block is insta-approved, a [`ChainSelectionMessage::Approvedl][CSM] should be sent for the block.
|
||||
* For each candidate, if the amount of needed approvals is more than the validators remaining after the backing group of the candidate is subtracted, then the candidate is insta-approved as approval would be impossible otherwise. If all candidates in the block are insta-approved, or there are no candidates in the block, then the block is insta-approved. If the block is insta-approved, a [`ChainSelectionMessage::Approved`][CSM] should be sent for the block.
|
||||
* Ensure that the `CandidateEntry` contains a `block_assignments` entry for the block, with the correct backing group set.
|
||||
* If a validator in this session, compute and assign `our_assignment` for the `block_assignments`
|
||||
* Only if not a member of the backing group.
|
||||
@@ -262,25 +262,27 @@ On receiving an `ApprovedAncestor(Hash, BlockNumber, response_channel)`:
|
||||
* [Schedule a new wakeup](#schedule-wakeup) of the candidate.
|
||||
|
||||
#### Schedule Wakeup
|
||||
|
||||
* Requires `(approval_entry, candidate_entry)` which effectively denotes a `(Block Hash, Candidate Hash)` pair - the candidate, along with the block it appears in.
|
||||
* Also requires `RequiredTranches`
|
||||
* If the `approval_entry` is approved, this doesn't need to be woken up again.
|
||||
* If `RequiredTranches::All` - no wakeup. We assume other incoming votes will trigger wakeup and potentially re-schedule.
|
||||
* If `RequiredTranches::Pending { considered, next_no_show, uncovered, maximum_broadcast, clock_drift }` - schedule at the lesser of the next no-show tick, or the tick, offset positively by `clock_drift` of the next non-empty tranche we are aware of after `considered`, including any tranche containing our own unbroadcast assignment. This can lead to no wakeup in the case that we have already broadcast our assignment and there are no pending no-shows; that is, we have approval votes for every assignment we've received that is not already a no-show. In this case, we will be re-triggered by other validators broadcasting their assignments.
|
||||
* If `RequiredTranches::Exact { next_no_show, .. } - set a wakeup for the next no-show tick.
|
||||
* If `RequiredTranches::Exact { next_no_show, .. }` - set a wakeup for the next no-show tick.
|
||||
|
||||
#### Launch Approval Work
|
||||
* Requires `(SessionIndex, SessionInfo, CandidateReceipt, ValidatorIndex, backing_group, block_hash, candidate_index)`
|
||||
* Extract the public key of the `ValidatorIndex` from the `SessionInfo` for the session.
|
||||
* Issue an `AvailabilityRecoveryMessage::RecoverAvailableData(candidate, session_index, Some(backing_group), response_sender)`
|
||||
* Load the historical validation code of the parachain by dispatching a `RuntimeApiRequest::ValidationCodeByHash(`descriptor.validation_code_hash`)` against the state of `block_hash`.
|
||||
* Spawn a background task with a clone of `background_tx`
|
||||
* Wait for the available data
|
||||
* Issue a `CandidateValidationMessage::ValidateFromExhaustive` message
|
||||
* Wait for the result of validation
|
||||
* Check that the result of validation, if valid, matches the commitments in the receipt.
|
||||
* If valid, issue a message on `background_tx` detailing the request.
|
||||
* If any of the data, the candidate, or the commitments are invalid, issue on `background_tx` a [`DisputeCoordinatorMessage::IssueLocalStatement`](../../types/overseer-protocol.md#dispute-coordinator-message) with `valid = false` to initiate a dispute.
|
||||
|
||||
* Requires `(SessionIndex, SessionInfo, CandidateReceipt, ValidatorIndex, backing_group, block_hash, candidate_index)`
|
||||
* Extract the public key of the `ValidatorIndex` from the `SessionInfo` for the session.
|
||||
* Issue an `AvailabilityRecoveryMessage::RecoverAvailableData(candidate, session_index, Some(backing_group), response_sender)`
|
||||
* Load the historical validation code of the parachain by dispatching a `RuntimeApiRequest::ValidationCodeByHash(descriptor.validation_code_hash)` against the state of `block_hash`.
|
||||
* Spawn a background task with a clone of `background_tx`
|
||||
* Wait for the available data
|
||||
* Issue a `CandidateValidationMessage::ValidateFromExhaustive` message
|
||||
* Wait for the result of validation
|
||||
* Check that the result of validation, if valid, matches the commitments in the receipt.
|
||||
* If valid, issue a message on `background_tx` detailing the request.
|
||||
* If any of the data, the candidate, or the commitments are invalid, issue on `background_tx` a [`DisputeCoordinatorMessage::IssueLocalStatement`](../../types/overseer-protocol.md#dispute-coordinator-message) with `valid = false` to initiate a dispute.
|
||||
|
||||
#### Issue Approval Vote
|
||||
* Fetch the block entry and candidate entry. Ignore if `None` - we've probably just lost a race with finality.
|
||||
|
||||
@@ -81,13 +81,14 @@ dispute participation subsystem.
|
||||
### On `OverseerSignal::ActiveLeavesUpdate`
|
||||
|
||||
For each leaf in the leaves update:
|
||||
* Fetch the session index for the child of the block with a [`RuntimeApiMessage::SessionIndexForChild`][RuntimeApiMessage].
|
||||
* If the session index is higher than `state.highest_session`:
|
||||
* update `state.highest_session`
|
||||
* remove everything with session index less than `state.highest_session - DISPUTE_WINDOW` from the `"recent-disputes"` in the DB.
|
||||
* Use `iter_with_prefix` to remove everything from `"earliest-session"` up to `state.highest_session - DISPUTE_WINDOW` from the DB under `"candidate-votes"`.
|
||||
* Update `"earliest-session"` to be equal to `state.highest_session - DISPUTE_WINDOW`.
|
||||
* For each new block, explicitly or implicitly, under the new leaf, scan for a dispute digest which indicates a rollback. If a rollback is detected, use the ChainApi subsystem to blacklist the chain.
|
||||
|
||||
* Fetch the session index for the child of the block with a [`RuntimeApiMessage::SessionIndexForChild`][RuntimeApiMessage].
|
||||
* If the session index is higher than `state.highest_session`:
|
||||
* update `state.highest_session`
|
||||
* remove everything with session index less than `state.highest_session - DISPUTE_WINDOW` from the `"recent-disputes"` in the DB.
|
||||
* Use `iter_with_prefix` to remove everything from `"earliest-session"` up to `state.highest_session - DISPUTE_WINDOW` from the DB under `"candidate-votes"`.
|
||||
* Update `"earliest-session"` to be equal to `state.highest_session - DISPUTE_WINDOW`.
|
||||
* For each new block, explicitly or implicitly, under the new leaf, scan for a dispute digest which indicates a rollback. If a rollback is detected, use the `ChainApi` subsystem to blacklist the chain.
|
||||
|
||||
### On `OverseerSignal::Conclude`
|
||||
|
||||
@@ -144,7 +145,7 @@ Do nothing.
|
||||
### On `DisputeCoordinatorMessage::QueryCandidateVotes`
|
||||
|
||||
* Load `"candidate-votes"` for every `(SessionIndex, CandidateHash)` in the query and return data within each `CandidateVote`.
|
||||
If a particular `candidate-vote` is missing, that particular request is ommitted from the response.
|
||||
If a particular `candidate-vote` is missing, that particular request is omitted from the response.
|
||||
|
||||
### On `DisputeCoordinatorMessage::IssueLocalStatement`
|
||||
|
||||
|
||||
@@ -21,33 +21,34 @@ There may be multiple competing blocks all ending the availability phase for a p
|
||||
|
||||
```dot process
|
||||
digraph {
|
||||
label = "Block data FSM\n\n\n";
|
||||
labelloc = "t";
|
||||
rankdir="LR";
|
||||
label = "Block data FSM\n\n\n";
|
||||
labelloc = "t";
|
||||
rankdir="LR";
|
||||
|
||||
st [label = "Stored"; shape = circle]
|
||||
inc [label = "Included"; shape = circle]
|
||||
fin [label = "Finalized"; shape = circle]
|
||||
prn [label = "Pruned"; shape = circle]
|
||||
st [label = "Stored"; shape = circle]
|
||||
inc [label = "Included"; shape = circle]
|
||||
fin [label = "Finalized"; shape = circle]
|
||||
prn [label = "Pruned"; shape = circle]
|
||||
|
||||
st -> inc [label = "Block\nincluded"]
|
||||
st -> prn [label = "Stored block\ntimed out"]
|
||||
inc -> fin [label = "Block\nfinalized"]
|
||||
inc -> st [label = "Competing blocks\nfinalized"]
|
||||
fin -> prn [label = "Block keep time\n(1 day + 1 hour) elapsed"]
|
||||
st -> inc [label = "Block\nincluded"]
|
||||
st -> prn [label = "Stored block\ntimed out"]
|
||||
inc -> fin [label = "Block\nfinalized"]
|
||||
inc -> st [label = "Competing blocks\nfinalized"]
|
||||
fin -> prn [label = "Block keep time\n(1 day + 1 hour) elapsed"]
|
||||
}
|
||||
```
|
||||
|
||||
## Database Schema
|
||||
|
||||
We use an underlying Key-Value database where we assume we have the following operations available:
|
||||
* `write(key, value)`
|
||||
* `read(key) -> Option<value>`
|
||||
* `iter_with_prefix(prefix) -> Iterator<(key, value)>` - gives all keys and values in lexicographical order where the key starts with `prefix`.
|
||||
|
||||
- `write(key, value)`
|
||||
- `read(key) -> Option<value>`
|
||||
- `iter_with_prefix(prefix) -> Iterator<(key, value)>` - gives all keys and values in lexicographical order where the key starts with `prefix`.
|
||||
|
||||
We use this database to encode the following schema:
|
||||
|
||||
```
|
||||
```rust
|
||||
("available", CandidateHash) -> Option<AvailableData>
|
||||
("chunk", CandidateHash, u32) -> Option<ErasureChunk>
|
||||
("meta", CandidateHash) -> Option<CandidateMeta>
|
||||
@@ -56,7 +57,7 @@ We use this database to encode the following schema:
|
||||
("prune_by_time", Timestamp, CandidateHash) -> Option<()>
|
||||
```
|
||||
|
||||
Timestamps are the wall-clock seconds since unix epoch. Timestamps and block numbers are both encoded as big-endian so lexicographic order is ascending.
|
||||
Timestamps are the wall-clock seconds since Unix epoch. Timestamps and block numbers are both encoded as big-endian so lexicographic order is ascending.
|
||||
|
||||
The meta information that we track per-candidate is defined as the `CandidateMeta` struct
|
||||
|
||||
@@ -88,84 +89,87 @@ Additionally, there is exactly one `prune_by_time` entry which holds the candida
|
||||
Input: [`AvailabilityStoreMessage`][ASM]
|
||||
|
||||
Output:
|
||||
- [`RuntimeApiMessage`][RAM]
|
||||
|
||||
- [`RuntimeApiMessage`][RAM]
|
||||
|
||||
## Functionality
|
||||
|
||||
For each head in the `activated` list:
|
||||
- Load all ancestors of the head back to the finalized block so we don't miss anything if import notifications are missed. If a `StoreChunk` message is received for a candidate which has no entry, then we will prematurely lose the data.
|
||||
- Note any new candidates backed in the head. Update the `CandidateMeta` for each. If the `CandidateMeta` does not exist, create it as `Unavailable` with the current timestamp. Register a `"prune_by_time"` entry based on the current timestamp + 1 hour.
|
||||
- Note any new candidate included in the head. Update the `CandidateMeta` for each, performing a transition from `Unavailable` to `Unfinalized` if necessary. That includes removing the `"prune_by_time"` entry. Add the head hash and number to the state, if unfinalized. Add an `"unfinalized"` entry for the block and candidate.
|
||||
- The `CandidateEvent` runtime API can be used for this purpose.
|
||||
|
||||
- Load all ancestors of the head back to the finalized block so we don't miss anything if import notifications are missed. If a `StoreChunk` message is received for a candidate which has no entry, then we will prematurely lose the data.
|
||||
- Note any new candidates backed in the head. Update the `CandidateMeta` for each. If the `CandidateMeta` does not exist, create it as `Unavailable` with the current timestamp. Register a `"prune_by_time"` entry based on the current timestamp + 1 hour.
|
||||
- Note any new candidate included in the head. Update the `CandidateMeta` for each, performing a transition from `Unavailable` to `Unfinalized` if necessary. That includes removing the `"prune_by_time"` entry. Add the head hash and number to the state, if unfinalized. Add an `"unfinalized"` entry for the block and candidate.
|
||||
- The `CandidateEvent` runtime API can be used for this purpose.
|
||||
|
||||
On `OverseerSignal::BlockFinalized(finalized)` events:
|
||||
- for each key in `iter_with_prefix("unfinalized")`
|
||||
- Stop if the key is beyond `("unfinalized, finalized)`
|
||||
- For each block number f that we encounter, load the finalized hash for that block.
|
||||
- The state of each `CandidateMeta` we encounter here must be `Unfinalized`, since we loaded the candidate from an `"unfinalized"` key.
|
||||
- For each candidate that we encounter under `f` and the finalized block hash,
|
||||
- Update the `CandidateMeta` to have `State::Finalized`. Remove all `"unfinalized"` entries from the old `Unfinalized` state.
|
||||
- Register a `"prune_by_time"` entry for the candidate based on the current time + 1 day + 1 hour.
|
||||
- For each candidate that we encounter under `f` which is not under the finalized block hash,
|
||||
- Remove all entries under `f` in the `Unfinalized` state.
|
||||
- If the `CandidateMeta` has state `Unfinalized` with an empty list of blocks, downgrade to `Unavailable` and re-schedule pruning under the timestamp + 1 hour. We do not prune here as the candidate still may be included in a descendent of the finalized chain.
|
||||
- Remove all `"unfinalized"` keys under `f`.
|
||||
- Update last_finalized = finalized.
|
||||
|
||||
- for each key in `iter_with_prefix("unfinalized")`
|
||||
- Stop if the key is beyond `("unfinalized, finalized)`
|
||||
- For each block number f that we encounter, load the finalized hash for that block.
|
||||
- The state of each `CandidateMeta` we encounter here must be `Unfinalized`, since we loaded the candidate from an `"unfinalized"` key.
|
||||
- For each candidate that we encounter under `f` and the finalized block hash,
|
||||
- Update the `CandidateMeta` to have `State::Finalized`. Remove all `"unfinalized"` entries from the old `Unfinalized` state.
|
||||
- Register a `"prune_by_time"` entry for the candidate based on the current time + 1 day + 1 hour.
|
||||
- For each candidate that we encounter under `f` which is not under the finalized block hash,
|
||||
- Remove all entries under `f` in the `Unfinalized` state.
|
||||
- If the `CandidateMeta` has state `Unfinalized` with an empty list of blocks, downgrade to `Unavailable` and re-schedule pruning under the timestamp + 1 hour. We do not prune here as the candidate still may be included in a descendant of the finalized chain.
|
||||
- Remove all `"unfinalized"` keys under `f`.
|
||||
- Update `last_finalized` = finalized.
|
||||
|
||||
This is roughly `O(n * m)` where n is the number of blocks finalized since the last update, and `m` is the number of parachains.
|
||||
|
||||
On `QueryAvailableData` message:
|
||||
|
||||
- Query `("available", candidate_hash)`
|
||||
- Query `("available", candidate_hash)`
|
||||
|
||||
This is `O(n)` in the size of the data, which may be large.
|
||||
|
||||
On `QueryDataAvailability` message:
|
||||
|
||||
- Query whether `("meta", candidate_hash)` exists and `data_available == true`.
|
||||
- Query whether `("meta", candidate_hash)` exists and `data_available == true`.
|
||||
|
||||
This is `O(n)` in the size of the metadata which is small.
|
||||
|
||||
On `QueryChunk` message:
|
||||
|
||||
- Query `("chunk", candidate_hash, index)`
|
||||
- Query `("chunk", candidate_hash, index)`
|
||||
|
||||
This is `O(n)` in the size of the data, which may be large.
|
||||
|
||||
On `QueryAllChunks` message:
|
||||
- Query `("meta", candidate_hash)`. If `None`, send an empty response and return.
|
||||
- For all `1` bits in the `chunks_stored`, query `("chunk", candidate_hash, index)`. Ignore but warn on errors, and return a vector of all loaded chunks.
|
||||
|
||||
On `QueryChunkAvailability message:
|
||||
- Query `("meta", candidate_hash)`. If `None`, send an empty response and return.
|
||||
- For all `1` bits in the `chunks_stored`, query `("chunk", candidate_hash, index)`. Ignore but warn on errors, and return a vector of all loaded chunks.
|
||||
|
||||
- Query whether `("meta", candidate_hash)` exists and the bit at `index` is set.
|
||||
On `QueryChunkAvailability` message:
|
||||
|
||||
- Query whether `("meta", candidate_hash)` exists and the bit at `index` is set.
|
||||
|
||||
This is `O(n)` in the size of the metadata which is small.
|
||||
|
||||
On `StoreChunk` message:
|
||||
|
||||
- If there is a `CandidateMeta` under the candidate hash, set the bit of the erasure-chunk in the `chunks_stored` bitfield to `1`. If it was not `1` already, write the chunk under `("chunk", candidate_hash, chunk_index)`.
|
||||
- If there is a `CandidateMeta` under the candidate hash, set the bit of the erasure-chunk in the `chunks_stored` bitfield to `1`. If it was not `1` already, write the chunk under `("chunk", candidate_hash, chunk_index)`.
|
||||
|
||||
This is `O(n)` in the size of the chunk.
|
||||
|
||||
On `StoreAvailableData` message:
|
||||
|
||||
- If there is no `CandidateMeta` under the candidate hash, create it with `State::Unavailable(now)`. Load the `CandidateMeta` otherwise.
|
||||
- Store `data` under `("available", candidate_hash)` and set `data_available` to true.
|
||||
- Store each chunk under `("chunk", candidate_hash, index)` and set every bit in `chunks_stored` to `1`.
|
||||
- If there is no `CandidateMeta` under the candidate hash, create it with `State::Unavailable(now)`. Load the `CandidateMeta` otherwise.
|
||||
- Store `data` under `("available", candidate_hash)` and set `data_available` to true.
|
||||
- Store each chunk under `("chunk", candidate_hash, index)` and set every bit in `chunks_stored` to `1`.
|
||||
|
||||
This is `O(n)` in the size of the data as the aggregate size of the chunks is proportional to the data.
|
||||
|
||||
Every 5 minutes, run a pruning routine:
|
||||
|
||||
- for each key in `iter_with_prefix("prune_by_time")`:
|
||||
- If the key is beyond ("prune_by_time", now), return.
|
||||
- Remove the key.
|
||||
- Extract `candidate_hash` from the key.
|
||||
- Load and remove the `("meta", candidate_hash)`
|
||||
- For each erasure chunk bit set, remove `("chunk", candidate_hash, bit_index)`.
|
||||
- If `data_available`, remove `("available", candidate_hash)
|
||||
- for each key in `iter_with_prefix("prune_by_time")`:
|
||||
- If the key is beyond `("prune_by_time", now)`, return.
|
||||
- Remove the key.
|
||||
- Extract `candidate_hash` from the key.
|
||||
- Load and remove the `("meta", candidate_hash)`
|
||||
- For each erasure chunk bit set, remove `("chunk", candidate_hash, bit_index)`.
|
||||
- If `data_available`, remove `("available", candidate_hash)`
|
||||
|
||||
This is O(n * m) in the amount of candidates and average size of the data stored. This is probably the most expensive operation but does not need
|
||||
to be run very often.
|
||||
@@ -193,7 +197,7 @@ Basically we need to test the correctness of data flow through state FSMs descri
|
||||
- Wait until the data should have been pruned.
|
||||
- The data is no longer available.
|
||||
|
||||
- Forkfulness of the relay chain is taken into account
|
||||
- Fork-awareness of the relay chain is taken into account
|
||||
- Block `B1` is added to the store.
|
||||
- Block `B2` is added to the store.
|
||||
- Notify the subsystem that both `B1` and `B2` were included in different leafs of relay chain.
|
||||
|
||||
Reference in New Issue
Block a user