Scheduler Module (#1162)

* scheduler module skeleton

* update scheduler skeleton to match latest version of guide

* better session change notification

* add mock randomness and fix test compilation

* shuffle validators into groups

* finish implementing session change logic for scheduler

* tweak core assignment type to track retries of parathread

* reframe queued parathread core as offset

* implement initialzation and finalization routines

* implement parathread claim queuing

* implement core_para

* implement the group_validators routine and fix errors

* add a reason for freeing cores

* implement `schedule` function

* add some docs to the scheduled function

* implement `occupied` helper

* implement availability predicate

* fix some warnings

* integrate scheduler into initializer

* integrate scheduler into mock module

* avoid conflict with Substrate's scheduler storage

* add parathreads index to paras module

* implement parathreads map in paras module

* add is_parathread to paras

* test adding parathread claim

* test that you cannot add claims when no parathread cores exist

* check session change parathread queue pruning

* test validator shuffling

* add allow_unused to scheduler items

* add test for scheduling

* add some more tests for scheduling logic

* test core rotation

* check parathread claim pruning after retries

* add bound notes

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* more suggestions from review

* test availability predicate, add box to please compiler

* add changes to guide

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

polkadot/roadmap/implementors-guide/src/runtime/scheduler.md

@@ -90,14 +90,15 @@ struct ParathreadEntry {
 
 // A queued parathread entry, pre-assigned to a core.
 struct QueuedParathread {
- claim: ParathreadEntry,
- core: CoreIndex,
+  claim: ParathreadEntry,
+  /// offset within the set of para-threads ranged `0..config.parathread_cores`.
+  core_offset: u32,
 }
 
 struct ParathreadQueue {
- queue: Vec<QueuedParathread>,
- // this value is between 0 and config.parathread_cores
- next_core: CoreIndex,
+  queue: Vec<QueuedParathread>,
+  /// offset within the set of para-threads ranged `0..config.parathread_cores`.
+  next_core_offset: u32,
 }
 
 enum CoreOccupied {
@@ -105,12 +106,22 @@ enum CoreOccupied {
   Parachain,
 }
 
+enum AssignmentKind {
+  Parachain,
+  Parathread(CollatorId, u32),
+}
+
 struct CoreAssignment {
   core: CoreIndex,
   para_id: ParaId,
-  collator: Option<CollatorId>,
+  kind: AssignmentKind,
   group_idx: GroupIndex,
 }
+// reasons a core might be freed.
+enum FreedReason {
+  Concluded,
+  TimedOut,
+}
 ```
 
 Storage layout:
@@ -150,6 +161,7 @@ Actions:
    - First, we obtain "shuffled validators" `SV` by shuffling the validators using the `SessionChangeNotification`'s random seed.
    - The groups are selected by partitioning `SV`. The first V % N groups will have (V / N) + 1 members, while the remaining groups will have (V / N) members each.
 1. Prune the parathread queue to remove all retries beyond `configuration.parathread_retries`.
+   - Also prune all parathread claims corresponding to de-registered parathreads.
    - all pruned claims should have their entry removed from the parathread index.
    - assign all non-pruned claims to new cores if the number of parathread cores has changed between the `new_config` and `old_config` of the `SessionChangeNotification`.
    - Assign claims in equal balance across all cores if rebalancing, and set the `next_core` of the `ParathreadQueue` by incrementing the relative index of the last assigned core and taking it modulo the number of parathread cores.
@@ -172,15 +184,16 @@ Actions:
   - The core used for the parathread claim is the `next_core` field of the `ParathreadQueue` and adding `Paras::parachains().len()` to it.
   - `next_core` is then updated by adding 1 and taking it modulo `config.parathread_cores`.
   - The claim is then added to the claim index.
-- `schedule(Vec<CoreIndex>)`: schedule new core assignments, with a parameter indicating previously-occupied cores which are to be considered returned.
+- `schedule(Vec<(CoreIndex, FreedReason)>)`: schedule new core assignments, with a parameter indicating previously-occupied cores which are to be considered returned and why they are being returned.
   - All freed parachain cores should be assigned to their respective parachain
-  - All freed parathread cores should have the claim removed from the claim index.
+  - All freed parathread cores whose reason for freeing was `FreedReason::Concluded` should have the claim removed from the claim index.
+  - All freed parathread cores whose reason for freeing was `FreedReason::TimedOut` should have the claim added to the parathread queue again without retries incremented
   - All freed parathread cores should take the next parathread entry from the queue.
   - The i'th validator group will be assigned to the `(i+k)%n`'th core at any point in time, where `k` is the number of rotations that have occurred in the session, and `n` is the total number of cores. This makes upcoming rotations within the same session predictable.
 - `scheduled() -> Vec<CoreAssignment>`: Get currently scheduled core assignments.
 - `occupied(Vec<CoreIndex>)`. Note that the given cores have become occupied.
-  - Fails if any given cores were not scheduled.
-  - Fails if the given cores are not sorted ascending by core index
+  - Behavior undefined if any given cores were not scheduled.
+  - Behavior undefined if the given cores are not sorted ascending by core index
   - This clears them from `Scheduled` and marks each corresponding `core` in the `AvailabilityCores` as occupied.
   - Since both the availability cores and the newly-occupied cores lists are sorted ascending, this method can be implemented efficiently.
 - `core_para(CoreIndex) -> ParaId`: return the currently-scheduled or occupied ParaId for the given core.