Replace StartWork and StopWork with an ActiveLeavesUpdate signal (#1431)

polkadot/roadmap/implementers-guide/src/node/availability/bitfield-signing.md

@@ -11,7 +11,7 @@ Output:
 
 ## Functionality
 
-Upon onset of a new relay-chain head with `StartWork`, launch bitfield signing job for the head. Stop the job on `StopWork`.
+Upon receipt of an `ActiveLeavesUpdate`, launch bitfield signing job for each `activated` head. Stop the job for each `deactivated` head.
 
 ## Bitfield Signing Job
 

polkadot/roadmap/implementers-guide/src/node/backing/candidate-backing.md

@@ -31,8 +31,10 @@ The subsystem should maintain a set of handles to Candidate Backing Jobs that ar
 
 ### On Overseer Signal
 
-* If the signal is an [`OverseerSignal`][OverseerSignal]`::StartWork(relay_parent)`, spawn a Candidate Backing Job with the given relay parent, storing a bidirectional channel with the Candidate Backing Job in the set of handles.
+* If the signal is an [`OverseerSignal`][OverseerSignal]`::ActiveLeavesUpdate`:
-* If the signal is an [`OverseerSignal`][OverseerSignal]`::StopWork(relay_parent)`, cease the Candidate Backing Job under that relay parent, if any.
+  * spawn a Candidate Backing Job for each `activated` head, storing a bidirectional channel with the Candidate Backing Job in the set of handles.
+  * cease the Candidate Backing Job for each `deactivated` head, if any.
+* If the signal is an [`OverseerSignal`][OverseerSignal]`::Conclude`: Forward conclude messages to all jobs, wait a small amount of time for them to join, and then exit.
 
 ### On Receiving `CandidateBackingMessage`
 

polkadot/roadmap/implementers-guide/src/node/backing/pov-distribution.md

@@ -30,7 +30,7 @@ For this, in order to avoid reaching into the internals of the [Statement Distri
 
 The view update mechanism of the [Network Bridge](../utility/network-bridge.md) ensures that peers are only allowed to consider a certain set of relay-parents as live. So this bounding mechanism caps the amount of data we need to store per peer at any time at `sum({ 2 * n_validators_at_head(head) * sizeof(hash) for head in view_heads })`. Additionally, peers should only be allowed to notify us of PoV hashes they are waiting for in the context of relay-parents in our own local view, which means that `n_validators_at_head` is implied to be `0` for relay-parents not in our own local view.
 
-View updates from peers and our own view updates are received from the network bridge. These will lag somewhat behind the `StartWork` and `StopWork` messages received from the overseer, which will influence the actual data we store. The `OurViewUpdate`s from the [`NetworkBridgeEvent`](../../types/overseer-protocol.md#network-bridge-update) must be considered canonical in terms of our peers' perception of us.
+View updates from peers and our own view updates are received from the network bridge. These will lag somewhat behind the `ActiveLeavesUpdate` messages received from the overseer, which will influence the actual data we store. The `OurViewUpdate`s from the [`NetworkBridgeEvent`](../../types/overseer-protocol.md#network-bridge-update) must be considered canonical in terms of our peers' perception of us.
 
 Lastly, the system needs to be bootstrapped with our own perception of which PoVs we are cognizant of but awaiting data for. This is done by receipt of the [`PoVDistributionMessage`](../../types/overseer-protocol.md#pov-distribution-message)::FetchPoV variant. Proper operation of this subsystem depends on the descriptors passed faithfully representing candidates which have been seconded by other validators.
 
@@ -72,12 +72,13 @@ enum NetworkMessage {
 Here is the logic of the state machine:
 
 *Overseer Signals*
-- On `StartWork(relay_parent)`:
+- On `ActiveLeavesUpdate(relay_parent)`:
+	- For each relay-parent in the `activated` list:
 		- Get the number of validators at that relay parent by querying the [Runtime API](../utility/runtime-api.md) for the validators and then counting them.
 		- Create a blank entry in `relay_parent_state` under `relay_parent` with correct `n_validators` set.
-- On `StopWork(relay_parent)`:
+	- For each relay-parent in the `deactivated` list:
 		- Remove the entry for `relay_parent` from `relay_parent_state`.
-- On `Concluded`: conclude.
+- On `Conclude`: conclude.
 
 *PoV Distribution Messages*
 - On `FetchPoV(relay_parent, descriptor, response_channel)`
@@ -101,7 +102,7 @@ Here is the logic of the state machine:
 	- If this is `NetworkMessage::Awaiting(relay_parent, pov_hashes)`:
 		- If there is no entry under `peer_state.awaited` for the `relay_parent`, report and ignore.
 		- If `relay_parent` is not contained within `our_view`, report and ignore.
-		- Otherwise, if the peer's `awaited` map combined with the `pov_hashes` would have more than ` 2 * relay_parent_state[relay_parent].n_validators` entries, report and ignore. Note that we are leaning on the property of the network bridge that it sets our view based on `StartWork` messages.
+		- Otherwise, if the peer's `awaited` map combined with the `pov_hashes` would have more than ` 2 * relay_parent_state[relay_parent].n_validators` entries, report and ignore. Note that we are leaning on the property of the network bridge that it sets our view based on `activated` heads in `ActiveLeavesUpdate` signals.
 		- For each new `pov_hash` in `pov_hashes`, if there is a `pov` under `pov_hash` in the `known` map, send the peer a `NetworkMessage::SendPoV(relay_parent, pov_hash, pov)`.
 		- Otherwise, add the `pov_hash` to the `awaited` map
 	- If this is `NetworkMessage::SendPoV(relay_parent, pov_hash, pov)`:

polkadot/roadmap/implementers-guide/src/node/overseer.md

@@ -24,7 +24,7 @@ The hierarchy of subsystems:
 
 ```
 
-The overseer determines work to do based on block import events and block finalization events. It does this by keeping track of the set of relay-parents for which work is currently being done. This is known as the "active leaves" set. It determines an initial set of active leaves on startup based on the data on-disk, and uses events about blockchain import to update the active leaves. Updates lead to [`OverseerSignal`](../types/overseer-protocol.md#overseer-signal)`::StartWork` and [`OverseerSignal`](../types/overseer-protocol.md#overseer-signal)`::StopWork` being sent according to new relay-parents, as well as relay-parents to stop considering. Block import events inform the overseer of leaves that no longer need to be built on, now that they have children, and inform us to begin building on those children. Block finalization events inform us when we can stop focusing on blocks that appear to have been orphaned.
+The overseer determines work to do based on block import events and block finalization events. It does this by keeping track of the set of relay-parents for which work is currently being done. This is known as the "active leaves" set. It determines an initial set of active leaves on startup based on the data on-disk, and uses events about blockchain import to update the active leaves. Updates lead to [`OverseerSignal`](../types/overseer-protocol.md#overseer-signal)`::ActiveLeavesUpdate` being sent according to new relay-parents, as well as relay-parents to stop considering. Block import events inform the overseer of leaves that no longer need to be built on, now that they have children, and inform us to begin building on those children. Block finalization events inform us when we can stop focusing on blocks that appear to have been orphaned.
 
 The overseer's logic can be described with these functions:
 
@@ -32,15 +32,14 @@ The overseer's logic can be described with these functions:
 
 * Start all subsystems
 * Determine all blocks of the blockchain that should be built on. This should typically be the head of the best fork of the chain we are aware of. Sometimes add recent forks as well.
-* For each of these blocks, send an `OverseerSignal::StartWork` to all subsystems.
+* Send an `OverseerSignal::ActiveLeavesUpdate` to all subsystems with `activated` containing each of these blocks.
 * Begin listening for block import and finality events
 
 ## On Block Import Event
 
 * Apply the block import event to the active leaves. A new block should lead to its addition to the active leaves set and its parent being deactivated.
-* For any deactivated leaves send an `OverseerSignal::StopWork` message to all subsystems.
+* Send an `OverseerSignal::ActiveLeavesUpdate` message to all subsystems containing all activated and deactivated leaves.
-* For any activated leaves send an `OverseerSignal::StartWork` message to all subsystems.
+* Ensure all `ActiveLeavesUpdate` messages are flushed before resuming activity as a message router.
-* Ensure all `StartWork` messages are flushed before resuming activity as a message router.
 
 > TODO: in the future, we may want to avoid building on too many sibling blocks at once. the notion of a "preferred head" among many competing sibling blocks would imply changes in our "active leaves" update rules here
 
@@ -48,7 +47,7 @@ The overseer's logic can be described with these functions:
 
 * Note the height `h` of the newly finalized block `B`.
 * Prune all leaves from the active leaves which have height `<= h` and are not `B`.
-* Issue `OverseerSignal::StopWork` for all deactivated leaves.
+* Issue `OverseerSignal::ActiveLeavesUpdate` containing all deactivated leaves.
 
 ## On Subsystem Failure
 
@@ -78,15 +77,19 @@ When a subsystem wants to communicate with another subsystem, or, more typically
 
 First, the subsystem that spawned a job is responsible for handling the first step of the communication. The overseer is not aware of the hierarchy of tasks within any given subsystem and is only responsible for subsystem-to-subsystem communication. So the sending subsystem must pass on the message via the overseer to the receiving subsystem, in such a way that the receiving subsystem can further address the communication to one of its internal tasks, if necessary.
 
-This communication prevents a certain class of race conditions. When the Overseer determines that it is time for subsystems to begin working on top of a particular relay-parent, it will dispatch a `StartWork` message to all subsystems to do so, and those messages will be handled asynchronously by those subsystems. Some subsystems will receive those messsages before others, and it is important that a message sent by subsystem A after receiving `StartWork` message will arrive at subsystem B after its `StartWork` message. If subsystem A maintaned an independent channel with subsystem B to communicate, it would be possible for subsystem B to handle the side message before the `StartWork` message, but it wouldn't have any logical course of action to take with the side message - leading to it being discarded or improperly handled. Well-architectured state machines should have a single source of inputs, so that is what we do here.
+This communication prevents a certain class of race conditions. When the Overseer determines that it is time for subsystems to begin working on top of a particular relay-parent, it will dispatch a `ActiveLeavesUpdate` message to all subsystems to do so, and those messages will be handled asynchronously by those subsystems. Some subsystems will receive those messsages before others, and it is important that a message sent by subsystem A after receiving `ActiveLeavesUpdate` message will arrive at subsystem B after its `ActiveLeavesUpdate` message. If subsystem A maintaned an independent channel with subsystem B to communicate, it would be possible for subsystem B to handle the side message before the `ActiveLeavesUpdate` message, but it wouldn't have any logical course of action to take with the side message - leading to it being discarded or improperly handled. Well-architectured state machines should have a single source of inputs, so that is what we do here.
 
-One exception is reasonable to make for responses to requests. A request should be made via the overseer in order to ensure that it arrives after any relevant `StartWork` message. A subsystem issuing a request as a result of a `StartWork` message can safely receive the response via a side-channel for two reasons:
+One exception is reasonable to make for responses to requests. A request should be made via the overseer in order to ensure that it arrives after any relevant `ActiveLeavesUpdate` message. A subsystem issuing a request as a result of a `ActiveLeavesUpdate` message can safely receive the response via a side-channel for two reasons:
 
 1. It's impossible for a request to be answered before it arrives, it is provable that any response to a request obeys the same ordering constraint.
-1. The request was sent as a result of handling a `StartWork` message. Then there is no possible future in which the `StartWork` message has not been handled upon the receipt of the response.
+1. The request was sent as a result of handling a `ActiveLeavesUpdate` message. Then there is no possible future in which the `ActiveLeavesUpdate` message has not been handled upon the receipt of the response.
 
 So as a single exception to the rule that all communication must happen via the overseer we allow the receipt of responses to requests via a side-channel, which may be established for that purpose. This simplifies any cases where the outside world desires to make a request to a subsystem, as the outside world can then establish a side-channel to receive the response on.
 
 It's important to note that the overseer is not aware of the internals of subsystems, and this extends to the jobs that they spawn. The overseer isn't aware of the existence or definition of those jobs, and is only aware of the outer subsystems with which it interacts. This gives subsystem implementations leeway to define internal jobs as they see fit, and to wrap a more complex hierarchy of state machines than having a single layer of jobs for relay-parent-based work. Likewise, subsystems aren't required to spawn jobs. Certain types of subsystems, such as those for shared storage or networking resources, won't perform block-based work but would still benefit from being on the Overseer's message bus. These subsystems can just ignore the overseer's signals for block-based work.
 
 Furthermore, the protocols by which subsystems communicate with each other should be well-defined irrespective of the implementation of the subsystem. In other words, their interface should be distinct from their implementation. This will prevent subsystems from accessing aspects of each other that are beyond the scope of the communication boundary.
+
+## On shutdown
+
+Send an `OverseerSignal::Conclude` message to each subsystem and wait some time for them to conclude before hard-exiting.

polkadot/roadmap/implementers-guide/src/node/subsystems-and-jobs.md

@@ -9,3 +9,5 @@ Most work that happens on the Node-side is related to building on top of a speci
 Since this goal of determining when to start and conclude work relative to a specific relay-parent is common to most, if not all subsystems, it is logically the job of the Overseer to distribute those signals as opposed to each subsystem duplicating that effort, potentially being out of synchronization with each other. Subsystem A should be able to expect that subsystem B is working on the same relay-parents as it is. One of the Overseer's tasks is to provide this heartbeat, or synchronized rhythm, to the system.
 
 The work that subsystems spawn to be done on a specific relay-parent is known as a job. Subsystems should set up and tear down jobs according to the signals received from the overseer. Subsystems may share or cache state between jobs.
+
+Subsystems must be robust to spurious exits. The outputs of the set of subsystems as a whole comprises of signed messages and data committed to disk. Care must be taken to avoid issuing messages that are not substantiated. Since subsystems need to be safe under spurious exits, it is the expected behavior that an `OverseerSignal::Conclude` can just lead to breaking the loop and exiting directly as opposed to waiting for everything to shut down gracefully.

polkadot/roadmap/implementers-guide/src/node/utility/availability-store.md

@@ -29,10 +29,11 @@ Input: [`AvailabilityStoreMessage`](../../types/overseer-protocol.md#availabilit
 
 ## Functionality
 
-On `StartWork`:
+On `ActiveLeavesUpdate`:
 
-- Note any new candidates backed in the block. Update pruning records for any stored `PoVBlock`s.
+For each head in the `activated` list:
-- Note any newly-included candidates backed in the block. Update pruning records for any stored availability chunks.
+	- Note any new candidates backed in the block. Update pruning records for any stored `PoVBlock`s.
+	- Note any newly-included candidates backed in the block. Update pruning records for any stored availability chunks.
 
 On block finality events:
 

polkadot/roadmap/implementers-guide/src/node/utility/network-bridge.md

@@ -21,7 +21,7 @@ There are two types of network messages this sends and receives:
 - ProtocolMessage(ProtocolId, Bytes)
 - ViewUpdate(View)
 
-`StartWork` and `StopWork` determine the computation of our local view. A `ViewUpdate` is issued to each connected peer, and a `NetworkBridgeUpdate::OurViewChange` is issued for each registered event producer.
+`ActiveLeavesUpdate`'s `activated` and `deactivated` lists determine the evolution of our local view over time. A `ViewUpdate` is issued to each connected peer after each update, and a `NetworkBridgeUpdate::OurViewChange` is issued for each registered event producer.
 
 On `RegisterEventProducer`:
 

polkadot/roadmap/implementers-guide/src/node/utility/provisioner.md

@@ -50,8 +50,12 @@ The subsystem should maintain a set of handles to Block Authorship Provisioning
 
 ### On Overseer Signal
 
-- `StartWork`: spawn a Block Authorship Provisioning Job with the given relay parent, storing a bidirectional channel with that job.
+- `ActiveLeavesUpdate`:
-- `StopWork`: terminate the Block Authorship Provisioning Job for the given relay parent, if any.
+	- For each `activated` head:
+		- spawn a Block Authorship Provisioning Job with the given relay parent, storing a bidirectional channel with that job.
+	- For each `deactivated` head:
+		- terminate the Block Authorship Provisioning Job for the given relay parent, if any.
+- `Conclude`: Forward `Conclude` to all jobs, waiting a small amount of time for them to join, and then hard-exiting.
 
 ### On `ProvisionerMessage`
 

polkadot/roadmap/implementers-guide/src/node/utility/runtime-api.md

@@ -16,4 +16,4 @@ On receipt of `RuntimeApiMessage::Request(relay_parent, request)`, answer the re
 
 ## Jobs
 
-> TODO Don't limit requests based on parent hash, but limit caching. No caching should be done for any requests on relay_parents that are not live based on `StartWork` or `StopWork` messages. Maybe with some leeway for things that have just been stopped.
+> TODO Don't limit requests based on parent hash, but limit caching. No caching should be done for any requests on relay_parents that are not active based on `ActiveLeavesUpdate` messages. Maybe with some leeway for things that have just been stopped.

polkadot/roadmap/implementers-guide/src/types/overseer-protocol.md

@@ -8,10 +8,10 @@ Signals from the overseer to a subsystem to request change in execution that has
 
 ```rust
 enum OverseerSignal {
-  /// Signal to start work localized to the relay-parent hash.
+  /// Signal about a change in active leaves.
-  StartWork(Hash),
+  ActiveLeavesUpdate(ActiveLeavesUpdate),
-  /// Signal to stop (or phase down) work localized to the relay-parent hash.
+  /// Conclude all operation.
-  StopWork(Hash),
+  Conclude,
 }
 ```
 
@@ -22,6 +22,17 @@ All subsystems have their own message types; all of them need to be able to list
 
 Either way, there will be some top-level type encapsulating messages from the overseer to each subsystem.
 
+## Active Leaves Update
+
+Indicates a change in active leaves. Activated leaves should have jobs, whereas deactivated leaves should lead to winding-down of work based on those leaves.
+
+```rust
+struct ActiveLeavesUpdate {
+	activated: [Hash], // in practice, these should probably be a SmallVec
+	deactivated: [Hash],
+}
+```
+
 ## All Messages
 
 > TODO (now)