Elastic scaling: runtime dependency tracking and enactment (#3479)

Changes needed to implement the runtime part of elastic scaling:
https://github.com/paritytech/polkadot-sdk/issues/3131,
https://github.com/paritytech/polkadot-sdk/issues/3132,
https://github.com/paritytech/polkadot-sdk/issues/3202

Also fixes https://github.com/paritytech/polkadot-sdk/issues/3675

TODOs:

- [x] storage migration
- [x] optimise process_candidates from O(N^2)
- [x] drop backable candidates which form cycles
- [x] fix unit tests
- [x] add more unit tests
- [x] check the runtime APIs which use the pending availability storage.
We need to expose all of them, see
https://github.com/paritytech/polkadot-sdk/issues/3576
- [x] optimise the candidate selection. we're currently picking randomly
until we satisfy the weight limit. we need to be smart about not
breaking candidate chains while being fair to all paras -
https://github.com/paritytech/polkadot-sdk/pull/3573

Relies on the changes made in
https://github.com/paritytech/polkadot-sdk/pull/3233 in terms of the
inclusion policy and the candidate ordering

---------

Signed-off-by: alindima <alin@parity.io>
Co-authored-by: command-bot <>
Co-authored-by: eskimor <eskimor@users.noreply.github.com>

polkadot/roadmap/implementers-guide/src/runtime/inclusion.md

@@ -147,15 +147,16 @@ All failed checks should lead to an unrecoverable error making the block invalid
       // return a vector of cleaned-up core IDs.
     }
   ```
-* `force_enact(ParaId)`: Forcibly enact the candidate with the given ID as though it had been deemed available by
-  bitfields. Is a no-op if there is no candidate pending availability for this para-id. This should generally not be
-  used but it is useful during execution of Runtime APIs, where the changes to the state are expected to be discarded
-  directly after.
+* `force_enact(ParaId)`: Forcibly enact the pending candidates of the given paraid as though they had been deemed
+  available by bitfields. Is a no-op if there is no candidate pending availability for this para-id.
+  If there are multiple candidates pending availability for this para-id, it will enact all of
+  them. This should generally not be used but it is useful during execution of Runtime APIs,
+  where the changes to the state are expected to be discarded directly after.
 * `candidate_pending_availability(ParaId) -> Option<CommittedCandidateReceipt>`: returns the `CommittedCandidateReceipt`
   pending availability for the para provided, if any.
 * `pending_availability(ParaId) -> Option<CandidatePendingAvailability>`: returns the metadata around the candidate
   pending availability for the para, if any.
-* `collect_disputed(disputed: Vec<CandidateHash>) -> Vec<CoreIndex>`: Sweeps through all paras pending availability. If
+* `free_disputed(disputed: Vec<CandidateHash>) -> Vec<CoreIndex>`: Sweeps through all paras pending availability. If
   the candidate hash is one of the disputed candidates, then clean up the corresponding storage for that candidate and
   the commitments. Return a vector of cleaned-up core IDs.
 

polkadot/roadmap/implementers-guide/src/runtime/parainherent.md

@@ -17,7 +17,7 @@ There are a couple of important notes to the operations in this inherent as they
    this fork.
 1. When disputes are initiated, we remove the block from pending availability. This allows us to roll back chains to the
    block before blocks are included as opposed to backing. It's important to do this before processing bitfields.
-1. `Inclusion::collect_disputed` is kind of expensive so it's important to gate this on whether there are actually any
+1. `Inclusion::free_disputed` is kind of expensive so it's important to gate this on whether there are actually any
    new disputes. Which should be never.
 1. And we don't accept parablocks that have open disputes or disputes that have concluded against the candidate. It's
    important to import dispute statements before backing, but this is already the case as disputes are imported before

polkadot/roadmap/implementers-guide/src/runtime/scheduler.md

@@ -285,7 +285,6 @@ No finalization routine runs for this module.
   - This clears them from `Scheduled` and marks each corresponding `core` in the `AvailabilityCores` as occupied.
   - Since both the availability cores and the newly-occupied cores lists are sorted ascending, this method can be
     implemented efficiently.
-- `core_para(CoreIndex) -> ParaId`: return the currently-scheduled or occupied ParaId for the given core.
 - `group_validators(GroupIndex) -> Option<Vec<ValidatorIndex>>`: return all validators in a given group, if the group
   index is valid for this session.
 - `availability_timeout_predicate() -> Option<impl Fn(CoreIndex, BlockNumber) -> bool>`: returns an optional predicate