Also fixes: https://github.com/paritytech/polkadot-sdk/issues/1417
- [x] CoreIndex -> AssignmentProvider mapping will be able to change any
time.
- [x] Implement
- [x] Provide Migrations
- [x] Add and fix tests
- [x] Implement bulk assigner logic
- [x] bulk assigner tests
- [x] Port over current assigner to use bulk designer (+ share on-demand
with bulk): top-level assigner has core ranges: legacy, bulk
- [x] Adjust migrations to reflect new assigner structure
- [x] Move migration code to Assignment code directly and make it
recursive (make it possible to skip releases) -> follow up ticket.
- [x] Test migrations
- [x] Add migration PR to runtimes repo -> follow up ticket.
- [x] Wire up with actual UMP messages
- [x] Write PR docs
---------
Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com>
Co-authored-by: BradleyOlson64 <lotrftw9@gmail.com>
Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com>
Co-authored-by: antonva <anton.asgeirsson@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Marcin S. <marcin@realemail.net>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: command-bot <>
* First baby steps
* Split scheduler into several modules
* Towards a more modular approach for scheduling
* move free_cores; IntoInterator -> BTreeMap
* Move clear()
* Move more functions out of scheduler
* Change weight composition
* More abstraction
* Further refactor
* clippy
* fmt
* fix test-runtime
* Add parathreads pallet to construct_runtime!
* Make all runtimes use (Parachains, Parathreads) scheduling
* Delete commented out code
* Remove parathreads scheduler from westend, rococo, and kusama
* fix rococo, westend, and kusama config
* Revert "fix rococo, westend, and kusama config"
This reverts commit 59e4de380d5c7d17eaaba5e2c2b81405de3465e3.
* Revert "Remove parathreads scheduler from westend, rococo, and kusama"
This reverts commit 4c44255296083ac5670560790ed77104917890a4.
* Remove CoreIndex from free_cores
* Remove unnecessary struct for parathreads
* parathreads provider take 1
* Comment out parathread tests
* Pop into lookahead
* fmt
* Fill lookahead with two entries for parachains
* fmt
* Current stage
* Towards ab parathreads
* no AB use
* Make tests typecheck
* quick hack to set scheduling lookahead to 1
* Fix scheduler tests
* fix paras_inherent tests
* misc
* Update more of a test
* cfg(test)
* some cleanup
* Undo paras_inherent changes
* Adjust paras inherent tests
* Undo changes to v2 primitives
* Undo v2 mod changes to tests
* minor
* Remove parathreads assigner and pallet
* minor
* minor
* more cleanup
* fmt
* minor
* minor
* minor
* Remove on_new_session from assignment provider
* Make adder collator integration test pass
* disable failing unit tests
* minor
* minor
* re-enable one unit test
* minor
* handle retries, add concluded para to pop interface
* comment out unused code
* Remove core_para from interface
* Remove first claimqueue element on clear if None instead removing all Nones
* Move claimqueue get out of loop
* Use VecDeque instead of Ved in ClaimQueue
* Make occupied() AB ready(?)
* handle freed disputed in clear_and_fill_claimqueue
* clear_and_fill_claimqueue returns scheduled Vec
* Rename and minor refactor
* return position of assignment taken from claimqueue
* minor
* Fix session boundary parachains number change + extended test
* Fix runtimes
* Fix polkadot runtime
* Remove polkadot pallet from benchmarks
* fix test runtime
* Add storage migration
* Minor refactor
* Minor
* migratin typechecks
* Add migration to runtimes
* Towards modular scheduling II (#6568)
* Add post migration check
* pebkac
* Disable migrations but mine
* Revert "Disable migrations but mine"
This reverts commit 4fa5c5a370c199944a7e0926f50b08626bfbad4c.
* Move scheduler migration
* Revert "Move scheduler migration"
This reverts commit a16b1659a907950bae048a9f7010f2aa76e02b6d.
* Fix migration
* cleanup
* Don't lose retries value anymore
* comment out test function
* Remove retries value from Assignment again
* minor
* Make collator for parathreads optional
* data type refactor
* update scheduler tests
* Change test function cfg
* comment out test function
* Try cfg(test) only
* fix cfg flags
* Add get_max_retries function to provider interface (#7047)
* Fix merge commit
* pebkac
* fix merge
* update cargo.lock
* fix merge
* fix merge
* Use btreemap instead of vec, fix scheduler calls.
* Use imported `ScheduledCore`
* Remove unused import in inclusion tests
* Use keys() instead of mapping over a BTreeMap
* Fix migrations for parachains scheduler
* Use BlockNumberFor<T> everywhere in scheduler
* Add on demand assignment provider pallet (#7110)
* Address some PR comments
* minor
* more cleanup
* find_map and timeout availability fixes
* Change default scheduling_lookahead to 1
* Add on demand assignment provider pallet
* Move test-runtime to new assignment provider
* Run cargo format on scheduler tests
* minor
* Mutate cores in single loop
* timeout predicate simplification
* claimqueue desired size fix
* Replace expect by ok_or
* More improvements
* Fix push back order and next_up_on_timeout
* minor
* session change docs
* Add pre_new_session call to hand pre session updates
* Remove sc_network dependency and PeerId from unnecessary data structures
* Remove unnecessary peer_ids
* Add OnDemandOrdering proxy (#7156)
* Add OnDemandBidding proxy
* Fix names
* OnDemandAssigner for rococo only
* Check PeerId in collator protocol before fetching collation
* On occupied, remove non occupied cores from the claimqueue front and refill
* Add missing docs
* Comment out unused field
* fix ScheduledCore in tests
* Fix the fix
* pebkac
* fmt
* Fix occupied dropping
* Remove double import
* ScheduledCore fixes
* Readd sc-network dep
* pebkac
* OpaquePeerId -> PeerId in can_collate interface
* Cargo.lock update for interface change
* Remove checks not needed anymore?
* Drop occupied core on session change if it would time out after the new session
* Add on demand assignment provider pallet
* Move test-runtime to new assignment provider
* Run cargo format on scheduler tests
* Add OnDemandOrdering proxy (#7156)
* Add OnDemandBidding proxy
* Fix names
* OnDemandAssigner for rococo only
* Remove unneeded config values
* Update comments
* Use and_then for queue position
* Return the max size of the spot queue on error
* Add comments to add_parathread_entry
* Add module comments
* Add log for when can_collate fails
* Change assigner queue type to `Assignment`
* Update assignment provider tests
* More logs
* Remove unused keyring import
* disable can_collate
* comment out can_collate
* Can collate first checks set if empty
* Move can_collate call to collation advertisement
* Fix backing test
* map to loop
* Remove obsolete check
* Move invalid collation test from backing to collator-protocol
* fix unused imports
* fix test
* fix Debug derivation
* Increase time limit on zombienet predicates
* Increase zombienet timeout
* Minor
* Address some PR comments
* Address PR comments
* Comment out failing assert due to on-demand assigner missing
* remove collator_restrictions info from backing
* Move can_collate to ActiveParas
* minor
* minor
* Update weight information for on demand config
* Add ttl to parasentry
* Fix tests missing parasentry ttl
* Adjust scheduler tests to use ttl default values
* Use match instead of if let for ttl drop
* Use RuntimeDebug trait for `ParasEntry` fields
* Add comments to on demand assignment pallet
* Fix spot traffic calculation
* Revert runtimedebug changes to primitives
* Remove runtimedebug derivation from `ParasEntry`
* Mention affinity in pallet level docs
* Use RuntimeDebug trait for ParasEntry child types
* Remove collator restrictions
* Fix primitive versioning and other merge issues
* Fix tests post merge
* Fix node side tests
* Edit parascheduler migration for clarity
* Move parascheduler migration up to next release
* Remove vestiges from merge
* Fix tests
* Refactor ttl handling
* Remove unused things from scheduler tests
* Move on demand assigner to own directory
* Update documentation
* Remove unused sc-network dependency in primitives
Was used for collator restrictions
* Remove unused import
* Reenable scheduler test
* Remove unused storage value
* Enable timeout predicate test and fix fn
Turns out that the issue with the compiler is fixed and we can now
use impl Trait in the manner used here.
* Remove unused imports
* Add benchmarking entry for perbill in config
* Correct typo
* Address review comments
* Log out errors when calculating spot traffic.
* Change parascheduler's log target name
* Update scheduler_common documentation
* Use mutate for affinity fns, add tests
* Add another on demand affinity test
* Unify parathreads and parachains in HostConfig (take 2) (#7452)
* Unify parathreads and parachains in HostConfig
* Fixed missed occurences
* Remove commented out lines
* `HostConfiguration v7`
* Fix version check
* Add `MigrateToV7` to `Unreleased`
* fmt
* fmt
* Fix compilation errors after the rebase
* Update runtime/parachains/src/scheduler/tests.rs
Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com>
* Update runtime/parachains/src/scheduler/tests.rs
Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com>
* fmt
* Fix migration test
* Fix tests
* Remove unneeded assert from tests
* parathread_cores -> on_demand_cores; parathread_retries -> on_demand_retries
* Fix a compilation error in tests
* Remove unused `use`
* update colander image version
---------
Co-authored-by: alexgparity <alex.gremm@parity.io>
Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com>
Co-authored-by: Javier Viola <javier@parity.io>
* Fix branch after merge with master
* Refactor out duplicate checks into a helper fn
* Fix tests post merge
* Rename add_parathread_assignment, add test
* Update docs
* Remove unused on_finalize function
* Add weight info to on demand pallet
* Update runtime/parachains/src/configuration.rs
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
* Update runtime/parachains/src/scheduler_common/mod.rs
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
* Update runtime/parachains/src/assigner_on_demand/mod.rs
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
* Add benchmarking to on demand pallet
* Make place_order test check for success
* Add on demand benchmarks
* Add local test weights to rococo runtime
* Modify TTL drop behaviour to not skip claims
Previous behaviour would jump a new claim from the assignment provider
ahead in the claimqueue, assuming lookahead is larger than 1.
* Refactor ttl test to test claimqueue order
* Disable place_order ext. when no on_demand cores
* Use default genesis config for benchmark tests
* Refactor config builder param
* Move lifecycle test from scheduler to on demand
* Remove unneeded lifecycle test
Paras module via the parachain assignment provider doesn't provide
new assignments if a parachain loses it's lease. The on demand
assignment provider doesn't provide an assignment that is not a
parathread.
* Re enable validator shuffle test
* More realistic weights for place_order
* Remove redundant import
* Fix backwards compatibility (hopefully)
* ".git/.scripts/commands/bench/bench.sh" --subcommand=runtime --runtime=rococo --target_dir=polkadot --pallet=runtime_parachains::assigner_on_demand
* Fix tests.
* Fix off-by-one.
* Re enable claimqueue fills test
* Re enable schedule_rotates_groups test
* Fix fill_claimqueue_fills test
* Re enable next_up_on_timeout test, move fn
* Do not pop from assignment provider when retrying
* Fix tests missing collator in scheduledcore
* Add comment about timeout predicate.
* Rename parasentry retries to availability timeouts
* Re enable schedule_schedules... test
* Refactor prune retried test to new scheduler
* Have all scheduler tests use genesis_cfg fn
* Update docs
* Update copyright notices on new files
* Rename is_parachain_core to is_bulk_core
* Remove erroneous TODO
* Simplify import
* ".git/.scripts/commands/bench/bench.sh" --subcommand=runtime --runtime=rococo --target_dir=polkadot --pallet=runtime_parachains::configuration
* Revert AdvertiseCollation order shuffle
* Refactor place_order into keepalive and allowdeath
* Revert rename of hrmp max inbound channels
parachain encompasses both on demand and slot auction / bulk.
* Restore availability_timeout_predicate function
* Clean up leftover comments
* Update runtime/parachains/src/scheduler/tests.rs
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
* ".git/.scripts/commands/bench/bench.sh" --subcommand=runtime --runtime=westend --target_dir=polkadot --pallet=runtime_parachains::configuration
---------
Co-authored-by: alexgparity <alex.gremm@parity.io>
Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Co-authored-by: Javier Viola <javier@parity.io>
Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: command-bot <>
* On Demand - update weights and small nits (#7605)
* Remove collator restriction test in inclusion
On demand parachains won't have collator restrictions implemented in
this way but will instead use a preferred collator registered to a
`ParaId` in `paras_registrar`.
* Remove redundant config guard for test fns
* Update weights
* Update WeightInfo for on_demand assigner
* Unify assignment provider parameters into one call (#7606)
* Combine assignmentprovider params into one fn call
* Move scheduler_common to a module under scheduler
* Fix ttl handling in benchmark builder
* Run cargo format
* Remove obsolete test.
* Small improvement.
* Use same migration pattern as config module
* Remove old TODO
* Change log target name for assigner on demand
* Fix migration
* Fix clippy warnings
* Add HostConfiguration storage migration to V8
* Add `MigrateToV8` to unreleased migrations for all runtimes
* Fix storage version check for config v8
* Set `StorageVersion` to 8 in `MigrateToV8`
* Remove dups.
* Update primitives/src/v5/mod.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
---------
Co-authored-by: alexgparity <alex.gremm@parity.io>
Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com>
Co-authored-by: antonva <anton.asgeirsson@parity.io>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com>
Co-authored-by: Javier Viola <javier@parity.io>
Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
* Happy New Year!
* Remove year entierly
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Remove years from copyright notice in the entire repo
---------
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* remove v0 primitives from polkadot-primitives
* first pass: remove v0
* fix fallout in erasure-coding
* remove v1 primitives, consolidate to v2
* the great import update
* update runtime_api_impl_v1 to v2 as well
* guide: add `Version` request for runtime API
* add version query to runtime API
* reintroduce OldV1SessionInfo in a limited way
Tsvetomir Dimitrov