Parathreads Feature Branch (#6969)

* First baby steps

* Split scheduler into several modules

* Towards a more modular approach for scheduling

* move free_cores; IntoInterator -> BTreeMap

* Move clear()

* Move more functions out of scheduler

* Change weight composition

* More abstraction

* Further refactor

* clippy

* fmt

* fix test-runtime

* Add parathreads pallet to construct_runtime!

* Make all runtimes use (Parachains, Parathreads) scheduling

* Delete commented out code

* Remove parathreads scheduler from westend, rococo, and kusama

* fix rococo, westend, and kusama config

* Revert "fix rococo, westend, and kusama config"

This reverts commit 59e4de380d5c7d17eaaba5e2c2b81405de3465e3.

* Revert "Remove parathreads scheduler from westend, rococo, and kusama"

This reverts commit 4c44255296083ac5670560790ed77104917890a4.

* Remove CoreIndex from free_cores

* Remove unnecessary struct for parathreads

* parathreads provider take 1

* Comment out parathread tests

* Pop into lookahead

* fmt

* Fill lookahead with two entries for parachains

* fmt

* Current stage

* Towards ab parathreads

* no AB use

* Make tests typecheck

* quick hack to set scheduling lookahead to 1

* Fix scheduler tests

* fix paras_inherent tests

* misc

* Update more of a test

* cfg(test)

* some cleanup

* Undo paras_inherent changes

* Adjust paras inherent tests

* Undo changes to v2 primitives

* Undo v2 mod changes to tests

* minor

* Remove parathreads assigner and pallet

* minor

* minor

* more cleanup

* fmt

* minor

* minor

* minor

* Remove on_new_session from assignment provider

* Make adder collator integration test pass

* disable failing unit tests

* minor

* minor

* re-enable one unit test

* minor

* handle retries, add concluded para to pop interface

* comment out unused code

* Remove core_para from interface

* Remove first claimqueue element on clear if None instead removing all Nones

* Move claimqueue get out of loop

* Use VecDeque instead of Ved in ClaimQueue

* Make occupied() AB ready(?)

* handle freed disputed in clear_and_fill_claimqueue

* clear_and_fill_claimqueue returns scheduled Vec

* Rename and minor refactor

* return position of assignment taken from claimqueue

* minor

* Fix session boundary parachains number change + extended test

* Fix runtimes

* Fix polkadot runtime

* Remove polkadot pallet from benchmarks

* fix test runtime

* Add storage migration

* Minor refactor

* Minor

* migratin typechecks

* Add migration to runtimes

* Towards modular scheduling II (#6568)

* Add post migration check

* pebkac

* Disable migrations but mine

* Revert "Disable migrations but mine"

This reverts commit 4fa5c5a370c199944a7e0926f50b08626bfbad4c.

* Move scheduler migration

* Revert "Move scheduler migration"

This reverts commit a16b1659a907950bae048a9f7010f2aa76e02b6d.

* Fix migration

* cleanup

* Don't lose retries value anymore

* comment out test function

* Remove retries value from Assignment again

* minor

* Make collator for parathreads optional

* data type refactor

* update scheduler tests

* Change test function cfg

* comment out test function

* Try cfg(test) only

* fix cfg flags

* Add get_max_retries function to provider interface (#7047)

* Fix merge commit

* pebkac

* fix merge

* update cargo.lock

* fix merge

* fix merge

* Use btreemap instead of vec, fix scheduler calls.

* Use imported `ScheduledCore`

* Remove unused import in inclusion tests

* Use keys() instead of mapping over a BTreeMap

* Fix migrations for parachains scheduler

* Use BlockNumberFor<T> everywhere in scheduler

* Add on demand assignment provider pallet (#7110)

* Address some PR comments

* minor

* more cleanup

* find_map and timeout availability fixes

* Change default scheduling_lookahead to 1

* Add on demand assignment provider pallet

* Move test-runtime to new assignment provider

* Run cargo format on scheduler tests

* minor

* Mutate cores in single loop

* timeout predicate simplification

* claimqueue desired size fix

* Replace expect by ok_or

* More improvements

* Fix push back order and next_up_on_timeout

* minor

* session change docs

* Add pre_new_session call to hand pre session updates

* Remove sc_network dependency and PeerId from unnecessary data structures

* Remove unnecessary peer_ids

* Add OnDemandOrdering proxy (#7156)

* Add OnDemandBidding proxy

* Fix names

* OnDemandAssigner for rococo only

* Check PeerId in collator protocol before fetching collation

* On occupied, remove non occupied cores from the claimqueue front and refill

* Add missing docs

* Comment out unused field

* fix ScheduledCore in tests

* Fix the fix

* pebkac

* fmt

* Fix occupied dropping

* Remove double import

* ScheduledCore fixes

* Readd sc-network dep

* pebkac

* OpaquePeerId -> PeerId in can_collate interface

* Cargo.lock update for interface change

* Remove checks not needed anymore?

* Drop occupied core on session change if it would time out after the new session

* Add on demand assignment provider pallet

* Move test-runtime to new assignment provider

* Run cargo format on scheduler tests

* Add OnDemandOrdering proxy (#7156)

* Add OnDemandBidding proxy

* Fix names

* OnDemandAssigner for rococo only

* Remove unneeded config values

* Update comments

* Use and_then for queue position

* Return the max size of the spot queue on error

* Add comments to add_parathread_entry

* Add module comments

* Add log for when can_collate fails

* Change assigner queue type to `Assignment`

* Update assignment provider tests

* More logs

* Remove unused keyring import

* disable can_collate

* comment out can_collate

* Can collate first checks set if empty

* Move can_collate call to collation advertisement

* Fix backing test

* map to loop

* Remove obsolete check

* Move invalid collation test from backing to collator-protocol

* fix unused imports

* fix test

* fix Debug derivation

* Increase time limit on zombienet predicates

* Increase zombienet timeout

* Minor

* Address some PR comments

* Address PR comments

* Comment out failing assert due to on-demand assigner missing

* remove collator_restrictions info from backing

* Move can_collate to ActiveParas

* minor

* minor

* Update weight information for on demand config

* Add ttl to parasentry

* Fix tests missing parasentry ttl

* Adjust scheduler tests to use ttl default values

* Use match instead of if let for ttl drop

* Use RuntimeDebug trait for `ParasEntry` fields

* Add comments to on demand assignment pallet

* Fix spot traffic calculation

* Revert runtimedebug changes to primitives

* Remove runtimedebug derivation from `ParasEntry`

* Mention affinity in pallet level docs

* Use RuntimeDebug trait for ParasEntry child types

* Remove collator restrictions

* Fix primitive versioning and other merge issues

* Fix tests post merge

* Fix node side tests

* Edit parascheduler migration for clarity

* Move parascheduler migration up to next release

* Remove vestiges from merge

* Fix tests

* Refactor ttl handling

* Remove unused things from scheduler tests

* Move on demand assigner to own directory

* Update documentation

* Remove unused sc-network dependency in primitives

Was used for collator restrictions

* Remove unused import

* Reenable scheduler test

* Remove unused storage value

* Enable timeout predicate test and fix fn

Turns out that the issue with the compiler is fixed and we can now
use impl Trait in the manner used here.

* Remove unused imports

* Add benchmarking entry for perbill in config

* Correct typo

* Address review comments

* Log out errors when calculating spot traffic.

* Change parascheduler's log target name

* Update scheduler_common documentation

* Use mutate for affinity fns, add tests

* Add another on demand affinity test

* Unify parathreads and parachains in HostConfig (take 2) (#7452)

* Unify parathreads and parachains in HostConfig

* Fixed missed occurences

* Remove commented out lines

* `HostConfiguration v7`

* Fix version check

* Add `MigrateToV7` to `Unreleased`

* fmt

* fmt

* Fix compilation errors after the rebase

* Update runtime/parachains/src/scheduler/tests.rs

Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com>

* Update runtime/parachains/src/scheduler/tests.rs

Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com>

* fmt

* Fix migration test

* Fix tests

* Remove unneeded assert from tests

* parathread_cores -> on_demand_cores; parathread_retries -> on_demand_retries

* Fix a compilation error in tests

* Remove unused `use`

* update colander image version

---------

Co-authored-by: alexgparity <alex.gremm@parity.io>
Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com>
Co-authored-by: Javier Viola <javier@parity.io>

* Fix branch after merge with master

* Refactor out duplicate checks into a helper fn

* Fix tests post merge

* Rename add_parathread_assignment, add test

* Update docs

* Remove unused on_finalize function

* Add weight info to on demand pallet

* Update runtime/parachains/src/configuration.rs

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>

* Update runtime/parachains/src/scheduler_common/mod.rs

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>

* Update runtime/parachains/src/assigner_on_demand/mod.rs

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>

* Add benchmarking to on demand pallet

* Make place_order test check for success

* Add on demand benchmarks

* Add local test weights to rococo runtime

* Modify TTL drop behaviour to not skip claims

Previous behaviour would jump a new claim from the assignment provider
ahead in the claimqueue, assuming lookahead is larger than 1.

* Refactor ttl test to test claimqueue order

* Disable place_order ext. when no on_demand cores

* Use default genesis config for benchmark tests

* Refactor config builder param

* Move lifecycle test from scheduler to on demand

* Remove unneeded lifecycle test

Paras module via the parachain assignment provider doesn't provide
new assignments if a parachain loses it's lease. The on demand
assignment provider doesn't provide an assignment that is not a
parathread.

* Re enable validator shuffle test

* More realistic weights for place_order

* Remove redundant import

* Fix backwards compatibility (hopefully)

* ".git/.scripts/commands/bench/bench.sh" --subcommand=runtime --runtime=rococo --target_dir=polkadot --pallet=runtime_parachains::assigner_on_demand

* Fix tests.

* Fix off-by-one.

* Re enable claimqueue fills test

* Re enable schedule_rotates_groups test

* Fix fill_claimqueue_fills test

* Re enable next_up_on_timeout test, move fn

* Do not pop from assignment provider when retrying

* Fix tests missing collator in scheduledcore

* Add comment about timeout predicate.

* Rename parasentry retries to availability timeouts

* Re enable schedule_schedules... test

* Refactor prune retried test to new scheduler

* Have all scheduler tests use genesis_cfg fn

* Update docs

* Update copyright notices on new files

* Rename is_parachain_core to is_bulk_core

* Remove erroneous TODO

* Simplify import

* ".git/.scripts/commands/bench/bench.sh" --subcommand=runtime --runtime=rococo --target_dir=polkadot --pallet=runtime_parachains::configuration

* Revert AdvertiseCollation order shuffle

* Refactor place_order into keepalive and allowdeath

* Revert rename of hrmp max inbound channels

parachain encompasses both on demand and slot auction / bulk.

* Restore availability_timeout_predicate function

* Clean up leftover comments

* Update runtime/parachains/src/scheduler/tests.rs

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>

* ".git/.scripts/commands/bench/bench.sh" --subcommand=runtime --runtime=westend --target_dir=polkadot --pallet=runtime_parachains::configuration

---------

Co-authored-by: alexgparity <alex.gremm@parity.io>
Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Co-authored-by: Javier Viola <javier@parity.io>
Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: command-bot <>

* On Demand - update weights and small nits (#7605)

* Remove collator restriction test in inclusion

On demand parachains won't have collator restrictions implemented in
this way but will instead use a preferred collator registered to a
`ParaId` in `paras_registrar`.

* Remove redundant config guard for test fns

* Update weights

* Update WeightInfo for on_demand assigner

* Unify assignment provider parameters into one call (#7606)

* Combine assignmentprovider params into one fn call

* Move scheduler_common to a module under scheduler

* Fix ttl handling in benchmark builder

* Run cargo format

* Remove obsolete test.

* Small improvement.

* Use same migration pattern as config module

* Remove old TODO

* Change log target name for assigner on demand

* Fix migration

* Fix clippy warnings

* Add HostConfiguration storage migration to V8

* Add `MigrateToV8` to unreleased migrations for all runtimes

* Fix storage version check for config v8

* Set `StorageVersion` to 8 in `MigrateToV8`

* Remove dups.

* Update primitives/src/v5/mod.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

---------

Co-authored-by: alexgparity <alex.gremm@parity.io>
Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com>
Co-authored-by: antonva <anton.asgeirsson@parity.io>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com>
Co-authored-by: Javier Viola <javier@parity.io>
Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

polkadot/runtime/parachains/src/configuration/benchmarking.rs

@@ -47,6 +47,8 @@ benchmarks! {
 		ExecutorParam::PvfExecTimeout(PvfExecTimeoutKind::Approval, 12_000),
 	][..]))
 
+	set_config_with_perbill {}: set_on_demand_fee_variability(RawOrigin::Root, Perbill::from_percent(100))
+
 	impl_benchmark_test_suite!(
 		Pallet,
 		crate::mock::new_test_ext(Default::default()),

polkadot/runtime/parachains/src/configuration/migration.rs

@@ -18,3 +18,4 @@
 
 pub mod v6;
 pub mod v7;
+pub mod v8;

polkadot/runtime/parachains/src/configuration/migration/v7.rs

@@ -23,13 +23,106 @@ use frame_support::{
 	weights::Weight,
 };
 use frame_system::pallet_prelude::BlockNumberFor;
-use primitives::SessionIndex;
+use primitives::{vstaging::AsyncBackingParams, Balance, ExecutorParams, SessionIndex};
 use sp_std::vec::Vec;
 
 use frame_support::traits::OnRuntimeUpgrade;
 
 use super::v6::V6HostConfiguration;
-type V7HostConfiguration<BlockNumber> = configuration::HostConfiguration<BlockNumber>;
+
+#[derive(parity_scale_codec::Encode, parity_scale_codec::Decode, Debug, Clone)]
+pub struct V7HostConfiguration<BlockNumber> {
+	pub max_code_size: u32,
+	pub max_head_data_size: u32,
+	pub max_upward_queue_count: u32,
+	pub max_upward_queue_size: u32,
+	pub max_upward_message_size: u32,
+	pub max_upward_message_num_per_candidate: u32,
+	pub hrmp_max_message_num_per_candidate: u32,
+	pub validation_upgrade_cooldown: BlockNumber,
+	pub validation_upgrade_delay: BlockNumber,
+	pub async_backing_params: AsyncBackingParams,
+	pub max_pov_size: u32,
+	pub max_downward_message_size: u32,
+	pub hrmp_max_parachain_outbound_channels: u32,
+	pub hrmp_max_parathread_outbound_channels: u32,
+	pub hrmp_sender_deposit: Balance,
+	pub hrmp_recipient_deposit: Balance,
+	pub hrmp_channel_max_capacity: u32,
+	pub hrmp_channel_max_total_size: u32,
+	pub hrmp_max_parachain_inbound_channels: u32,
+	pub hrmp_max_parathread_inbound_channels: u32,
+	pub hrmp_channel_max_message_size: u32,
+	pub executor_params: ExecutorParams,
+	pub code_retention_period: BlockNumber,
+	pub parathread_cores: u32,
+	pub parathread_retries: u32,
+	pub group_rotation_frequency: BlockNumber,
+	pub chain_availability_period: BlockNumber,
+	pub thread_availability_period: BlockNumber,
+	pub scheduling_lookahead: u32,
+	pub max_validators_per_core: Option<u32>,
+	pub max_validators: Option<u32>,
+	pub dispute_period: SessionIndex,
+	pub dispute_post_conclusion_acceptance_period: BlockNumber,
+	pub no_show_slots: u32,
+	pub n_delay_tranches: u32,
+	pub zeroth_delay_tranche_width: u32,
+	pub needed_approvals: u32,
+	pub relay_vrf_modulo_samples: u32,
+	pub pvf_voting_ttl: SessionIndex,
+	pub minimum_validation_upgrade_delay: BlockNumber,
+}
+
+impl<BlockNumber: Default + From<u32>> Default for V7HostConfiguration<BlockNumber> {
+	fn default() -> Self {
+		Self {
+			async_backing_params: AsyncBackingParams {
+				max_candidate_depth: 0,
+				allowed_ancestry_len: 0,
+			},
+			group_rotation_frequency: 1u32.into(),
+			chain_availability_period: 1u32.into(),
+			thread_availability_period: 1u32.into(),
+			no_show_slots: 1u32.into(),
+			validation_upgrade_cooldown: Default::default(),
+			validation_upgrade_delay: 2u32.into(),
+			code_retention_period: Default::default(),
+			max_code_size: Default::default(),
+			max_pov_size: Default::default(),
+			max_head_data_size: Default::default(),
+			parathread_cores: Default::default(),
+			parathread_retries: Default::default(),
+			scheduling_lookahead: Default::default(),
+			max_validators_per_core: Default::default(),
+			max_validators: None,
+			dispute_period: 6,
+			dispute_post_conclusion_acceptance_period: 100.into(),
+			n_delay_tranches: Default::default(),
+			zeroth_delay_tranche_width: Default::default(),
+			needed_approvals: Default::default(),
+			relay_vrf_modulo_samples: Default::default(),
+			max_upward_queue_count: Default::default(),
+			max_upward_queue_size: Default::default(),
+			max_downward_message_size: Default::default(),
+			max_upward_message_size: Default::default(),
+			max_upward_message_num_per_candidate: Default::default(),
+			hrmp_sender_deposit: Default::default(),
+			hrmp_recipient_deposit: Default::default(),
+			hrmp_channel_max_capacity: Default::default(),
+			hrmp_channel_max_total_size: Default::default(),
+			hrmp_max_parachain_inbound_channels: Default::default(),
+			hrmp_max_parathread_inbound_channels: Default::default(),
+			hrmp_channel_max_message_size: Default::default(),
+			hrmp_max_parachain_outbound_channels: Default::default(),
+			hrmp_max_parathread_outbound_channels: Default::default(),
+			hrmp_max_message_num_per_candidate: Default::default(),
+			pvf_voting_ttl: 2u32.into(),
+			minimum_validation_upgrade_delay: 2.into(),
+			executor_params: Default::default(),
+		}
+	}
+}
 
 mod v6 {
 	use super::*;

polkadot/runtime/parachains/src/configuration/migration/v8.rs

@@ -0,0 +1,319 @@
+// Copyright (C) Parity Technologies (UK) Ltd.
+// This file is part of Polkadot.
+
+// Polkadot is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Polkadot is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Polkadot.  If not, see <http://www.gnu.org/licenses/>.
+
+//! A module that is responsible for migration of storage.
+
+use crate::configuration::{self, Config, Pallet};
+use frame_support::{
+	pallet_prelude::*,
+	traits::{Defensive, StorageVersion},
+	weights::Weight,
+};
+use frame_system::pallet_prelude::BlockNumberFor;
+use primitives::SessionIndex;
+use sp_runtime::Perbill;
+use sp_std::vec::Vec;
+
+use frame_support::traits::OnRuntimeUpgrade;
+
+use super::v7::V7HostConfiguration;
+type V8HostConfiguration<BlockNumber> = configuration::HostConfiguration<BlockNumber>;
+
+mod v7 {
+	use super::*;
+
+	#[frame_support::storage_alias]
+	pub(crate) type ActiveConfig<T: Config> =
+		StorageValue<Pallet<T>, V7HostConfiguration<BlockNumberFor<T>>, OptionQuery>;
+
+	#[frame_support::storage_alias]
+	pub(crate) type PendingConfigs<T: Config> = StorageValue<
+		Pallet<T>,
+		Vec<(SessionIndex, V7HostConfiguration<BlockNumberFor<T>>)>,
+		OptionQuery,
+	>;
+}
+
+mod v8 {
+	use super::*;
+
+	#[frame_support::storage_alias]
+	pub(crate) type ActiveConfig<T: Config> =
+		StorageValue<Pallet<T>, V8HostConfiguration<BlockNumberFor<T>>, OptionQuery>;
+
+	#[frame_support::storage_alias]
+	pub(crate) type PendingConfigs<T: Config> = StorageValue<
+		Pallet<T>,
+		Vec<(SessionIndex, V8HostConfiguration<BlockNumberFor<T>>)>,
+		OptionQuery,
+	>;
+}
+
+pub struct MigrateToV8<T>(sp_std::marker::PhantomData<T>);
+impl<T: Config> OnRuntimeUpgrade for MigrateToV8<T> {
+	#[cfg(feature = "try-runtime")]
+	fn pre_upgrade() -> Result<Vec<u8>, sp_runtime::TryRuntimeError> {
+		log::trace!(target: crate::configuration::LOG_TARGET, "Running pre_upgrade() for HostConfiguration MigrateToV8");
+		Ok(Vec::new())
+	}
+
+	fn on_runtime_upgrade() -> Weight {
+		log::info!(target: configuration::LOG_TARGET, "HostConfiguration MigrateToV8 started");
+		if StorageVersion::get::<Pallet<T>>() == 7 {
+			let weight_consumed = migrate_to_v8::<T>();
+
+			log::info!(target: configuration::LOG_TARGET, "HostConfiguration MigrateToV8 executed successfully");
+			StorageVersion::new(8).put::<Pallet<T>>();
+
+			weight_consumed
+		} else {
+			log::warn!(target: configuration::LOG_TARGET, "HostConfiguration MigrateToV8 should be removed.");
+			T::DbWeight::get().reads(1)
+		}
+	}
+
+	#[cfg(feature = "try-runtime")]
+	fn post_upgrade(_state: Vec<u8>) -> Result<(), sp_runtime::TryRuntimeError> {
+		log::trace!(target: crate::configuration::LOG_TARGET, "Running post_upgrade() for HostConfiguration MigrateToV8");
+		ensure!(
+			StorageVersion::get::<Pallet<T>>() >= 8,
+			"Storage version should be >= 8 after the migration"
+		);
+
+		Ok(())
+	}
+}
+
+fn migrate_to_v8<T: Config>() -> Weight {
+	// Unusual formatting is justified:
+	// - make it easier to verify that fields assign what they supposed to assign.
+	// - this code is transient and will be removed after all migrations are done.
+	// - this code is important enough to optimize for legibility sacrificing consistency.
+	#[rustfmt::skip]
+	let translate =
+		|pre: V7HostConfiguration<BlockNumberFor<T>>| ->
+		V8HostConfiguration<BlockNumberFor<T>>
+	{
+		V8HostConfiguration {
+max_code_size                            : pre.max_code_size,
+max_head_data_size                       : pre.max_head_data_size,
+max_upward_queue_count                   : pre.max_upward_queue_count,
+max_upward_queue_size                    : pre.max_upward_queue_size,
+max_upward_message_size                  : pre.max_upward_message_size,
+max_upward_message_num_per_candidate     : pre.max_upward_message_num_per_candidate,
+hrmp_max_message_num_per_candidate       : pre.hrmp_max_message_num_per_candidate,
+validation_upgrade_cooldown              : pre.validation_upgrade_cooldown,
+validation_upgrade_delay                 : pre.validation_upgrade_delay,
+max_pov_size                             : pre.max_pov_size,
+max_downward_message_size                : pre.max_downward_message_size,
+hrmp_sender_deposit                      : pre.hrmp_sender_deposit,
+hrmp_recipient_deposit                   : pre.hrmp_recipient_deposit,
+hrmp_channel_max_capacity                : pre.hrmp_channel_max_capacity,
+hrmp_channel_max_total_size              : pre.hrmp_channel_max_total_size,
+hrmp_max_parachain_inbound_channels      : pre.hrmp_max_parachain_inbound_channels,
+hrmp_max_parachain_outbound_channels     : pre.hrmp_max_parachain_outbound_channels,
+hrmp_channel_max_message_size            : pre.hrmp_channel_max_message_size,
+code_retention_period                    : pre.code_retention_period,
+on_demand_cores                          : pre.parathread_cores,
+on_demand_retries                        : pre.parathread_retries,
+group_rotation_frequency                 : pre.group_rotation_frequency,
+paras_availability_period                : pre.chain_availability_period,
+scheduling_lookahead                     : pre.scheduling_lookahead,
+max_validators_per_core                  : pre.max_validators_per_core,
+max_validators                           : pre.max_validators,
+dispute_period                           : pre.dispute_period,
+dispute_post_conclusion_acceptance_period: pre.dispute_post_conclusion_acceptance_period,
+no_show_slots                            : pre.no_show_slots,
+n_delay_tranches                         : pre.n_delay_tranches,
+zeroth_delay_tranche_width               : pre.zeroth_delay_tranche_width,
+needed_approvals                         : pre.needed_approvals,
+relay_vrf_modulo_samples                 : pre.relay_vrf_modulo_samples,
+pvf_voting_ttl                           : pre.pvf_voting_ttl,
+minimum_validation_upgrade_delay         : pre.minimum_validation_upgrade_delay,
+async_backing_params                     : pre.async_backing_params,
+executor_params                          : pre.executor_params,
+on_demand_queue_max_size                 : 10_000u32,
+on_demand_base_fee                       : 10_000_000u128,
+on_demand_fee_variability                : Perbill::from_percent(3),
+on_demand_target_queue_utilization       : Perbill::from_percent(25),
+on_demand_ttl                            : 5u32.into(),
+		}
+	};
+
+	let v7 = v7::ActiveConfig::<T>::get()
+		.defensive_proof("Could not decode old config")
+		.unwrap_or_default();
+	let v8 = translate(v7);
+	v8::ActiveConfig::<T>::set(Some(v8));
+
+	// Allowed to be empty.
+	let pending_v7 = v7::PendingConfigs::<T>::get().unwrap_or_default();
+	let mut pending_v8 = Vec::new();
+
+	for (session, v7) in pending_v7.into_iter() {
+		let v8 = translate(v7);
+		pending_v8.push((session, v8));
+	}
+	v8::PendingConfigs::<T>::set(Some(pending_v8.clone()));
+
+	let num_configs = (pending_v8.len() + 1) as u64;
+	T::DbWeight::get().reads_writes(num_configs, num_configs)
+}
+
+#[cfg(test)]
+mod tests {
+	use super::*;
+	use crate::mock::{new_test_ext, Test};
+
+	#[test]
+	fn v8_deserialized_from_actual_data() {
+		// Example how to get new `raw_config`:
+		// We'll obtain the raw_config at a specified a block
+		// Steps:
+		// 1. Go to Polkadot.js -> Developer -> Chain state -> Storage: https://polkadot.js.org/apps/#/chainstate
+		// 2. Set these parameters:
+		//   2.1. selected state query: configuration; activeConfig():
+		//        PolkadotRuntimeParachainsConfigurationHostConfiguration
+		//   2.2. blockhash to query at:
+		//        0xf89d3ab5312c5f70d396dc59612f0aa65806c798346f9db4b35278baed2e0e53 (the hash of
+		//        the block)
+		//   2.3. Note the value of encoded storage key ->
+		//        0x06de3d8a54d27e44a9d5ce189618f22db4b49d95320d9021994c850f25b8e385 for the
+		// referenced        block.
+		//   2.4. You'll also need the decoded values to update the test.
+		// 3. Go to Polkadot.js -> Developer -> Chain state -> Raw storage
+		//   3.1 Enter the encoded storage key and you get the raw config.
+
+		// This exceeds the maximal line width length, but that's fine, since this is not code and
+		// doesn't need to be read and also leaving it as one line allows to easily copy it.
+		let raw_config =
+	hex_literal::hex!["
+	0000300000800000080000000000100000c8000005000000050000000200000002000000000000000000000000005000000010000400000000000000000000000000000000000000000000000000000000000000000000000800000000200000040000000000100000b004000000000000000000001027000080b2e60e80c3c9018096980000000000000000000000000005000000140000000400000001000000010100000000060000006400000002000000190000000000000002000000020000000200000005000000"
+	];
+
+		let v8 =
+			V8HostConfiguration::<primitives::BlockNumber>::decode(&mut &raw_config[..]).unwrap();
+
+		// We check only a sample of the values here. If we missed any fields or messed up data
+		// types that would skew all the fields coming after.
+		assert_eq!(v8.max_code_size, 3_145_728);
+		assert_eq!(v8.validation_upgrade_cooldown, 2);
+		assert_eq!(v8.max_pov_size, 5_242_880);
+		assert_eq!(v8.hrmp_channel_max_message_size, 1_048_576);
+		assert_eq!(v8.n_delay_tranches, 25);
+		assert_eq!(v8.minimum_validation_upgrade_delay, 5);
+		assert_eq!(v8.group_rotation_frequency, 20);
+		assert_eq!(v8.on_demand_cores, 0);
+		assert_eq!(v8.on_demand_base_fee, 10_000_000);
+	}
+
+	#[test]
+	fn test_migrate_to_v8() {
+		// Host configuration has lots of fields. However, in this migration we only remove one
+		// field. The most important part to check are a couple of the last fields. We also pick
+		// extra fields to check arbitrarily, e.g. depending on their position (i.e. the middle) and
+		// also their type.
+		//
+		// We specify only the picked fields and the rest should be provided by the `Default`
+		// implementation. That implementation is copied over between the two types and should work
+		// fine.
+		let v7 = V7HostConfiguration::<primitives::BlockNumber> {
+			needed_approvals: 69,
+			thread_availability_period: 55,
+			hrmp_recipient_deposit: 1337,
+			max_pov_size: 1111,
+			chain_availability_period: 33,
+			minimum_validation_upgrade_delay: 20,
+			..Default::default()
+		};
+
+		let mut pending_configs = Vec::new();
+		pending_configs.push((100, v7.clone()));
+		pending_configs.push((300, v7.clone()));
+
+		new_test_ext(Default::default()).execute_with(|| {
+			// Implant the v6 version in the state.
+			v7::ActiveConfig::<Test>::set(Some(v7));
+			v7::PendingConfigs::<Test>::set(Some(pending_configs));
+
+			migrate_to_v8::<Test>();
+
+			let v8 = v8::ActiveConfig::<Test>::get().unwrap();
+			let mut configs_to_check = v8::PendingConfigs::<Test>::get().unwrap();
+			configs_to_check.push((0, v8.clone()));
+
+			for (_, v7) in configs_to_check {
+				#[rustfmt::skip]
+				{
+					assert_eq!(v7.max_code_size                            , v8.max_code_size);
+					assert_eq!(v7.max_head_data_size                       , v8.max_head_data_size);
+					assert_eq!(v7.max_upward_queue_count                   , v8.max_upward_queue_count);
+					assert_eq!(v7.max_upward_queue_size                    , v8.max_upward_queue_size);
+					assert_eq!(v7.max_upward_message_size                  , v8.max_upward_message_size);
+					assert_eq!(v7.max_upward_message_num_per_candidate     , v8.max_upward_message_num_per_candidate);
+					assert_eq!(v7.hrmp_max_message_num_per_candidate       , v8.hrmp_max_message_num_per_candidate);
+					assert_eq!(v7.validation_upgrade_cooldown              , v8.validation_upgrade_cooldown);
+					assert_eq!(v7.validation_upgrade_delay                 , v8.validation_upgrade_delay);
+					assert_eq!(v7.max_pov_size                             , v8.max_pov_size);
+					assert_eq!(v7.max_downward_message_size                , v8.max_downward_message_size);
+					assert_eq!(v7.hrmp_max_parachain_outbound_channels     , v8.hrmp_max_parachain_outbound_channels);
+					assert_eq!(v7.hrmp_sender_deposit                      , v8.hrmp_sender_deposit);
+					assert_eq!(v7.hrmp_recipient_deposit                   , v8.hrmp_recipient_deposit);
+					assert_eq!(v7.hrmp_channel_max_capacity                , v8.hrmp_channel_max_capacity);
+					assert_eq!(v7.hrmp_channel_max_total_size              , v8.hrmp_channel_max_total_size);
+					assert_eq!(v7.hrmp_max_parachain_inbound_channels      , v8.hrmp_max_parachain_inbound_channels);
+					assert_eq!(v7.hrmp_channel_max_message_size            , v8.hrmp_channel_max_message_size);
+					assert_eq!(v7.code_retention_period                    , v8.code_retention_period);
+					assert_eq!(v7.on_demand_cores                          , v8.on_demand_cores);
+					assert_eq!(v7.on_demand_retries                        , v8.on_demand_retries);
+					assert_eq!(v7.group_rotation_frequency                 , v8.group_rotation_frequency);
+					assert_eq!(v7.paras_availability_period                , v8.paras_availability_period);
+					assert_eq!(v7.scheduling_lookahead                     , v8.scheduling_lookahead);
+					assert_eq!(v7.max_validators_per_core                  , v8.max_validators_per_core);
+					assert_eq!(v7.max_validators                           , v8.max_validators);
+					assert_eq!(v7.dispute_period                           , v8.dispute_period);
+					assert_eq!(v7.no_show_slots                            , v8.no_show_slots);
+					assert_eq!(v7.n_delay_tranches                         , v8.n_delay_tranches);
+					assert_eq!(v7.zeroth_delay_tranche_width               , v8.zeroth_delay_tranche_width);
+					assert_eq!(v7.needed_approvals                         , v8.needed_approvals);
+					assert_eq!(v7.relay_vrf_modulo_samples                 , v8.relay_vrf_modulo_samples);
+					assert_eq!(v7.pvf_voting_ttl                           , v8.pvf_voting_ttl);
+					assert_eq!(v7.minimum_validation_upgrade_delay         , v8.minimum_validation_upgrade_delay);
+					assert_eq!(v7.async_backing_params.allowed_ancestry_len, v8.async_backing_params.allowed_ancestry_len);
+					assert_eq!(v7.async_backing_params.max_candidate_depth , v8.async_backing_params.max_candidate_depth);
+					assert_eq!(v7.executor_params						   , v8.executor_params);
+				}; // ; makes this a statement. `rustfmt::skip` cannot be put on an expression.
+			}
+		});
+	}
+
+	// Test that migration doesn't panic in case there're no pending configurations upgrades in
+	// pallet's storage.
+	#[test]
+	fn test_migrate_to_v8_no_pending() {
+		let v7 = V7HostConfiguration::<primitives::BlockNumber>::default();
+
+		new_test_ext(Default::default()).execute_with(|| {
+			// Implant the v6 version in the state.
+			v7::ActiveConfig::<Test>::set(Some(v7));
+			// Ensure there're no pending configs.
+			v7::PendingConfigs::<Test>::set(None);
+
+			// Shouldn't fail.
+			migrate_to_v8::<Test>();
+		});
+	}
+}

polkadot/runtime/parachains/src/configuration/tests.rs

@@ -216,11 +216,7 @@ fn invariants() {
 		);
 
 		assert_err!(
-			Configuration::set_chain_availability_period(RuntimeOrigin::root(), 0),
-			Error::<Test>::InvalidNewValue
-		);
-		assert_err!(
-			Configuration::set_thread_availability_period(RuntimeOrigin::root(), 0),
+			Configuration::set_paras_availability_period(RuntimeOrigin::root(), 0),
 			Error::<Test>::InvalidNewValue
 		);
 		assert_err!(
@@ -229,17 +225,12 @@ fn invariants() {
 		);
 
 		ActiveConfig::<Test>::put(HostConfiguration {
-			chain_availability_period: 10,
-			thread_availability_period: 8,
+			paras_availability_period: 10,
 			minimum_validation_upgrade_delay: 11,
 			..Default::default()
 		});
 		assert_err!(
-			Configuration::set_chain_availability_period(RuntimeOrigin::root(), 12),
-			Error::<Test>::InvalidNewValue
-		);
-		assert_err!(
-			Configuration::set_thread_availability_period(RuntimeOrigin::root(), 12),
+			Configuration::set_paras_availability_period(RuntimeOrigin::root(), 12),
 			Error::<Test>::InvalidNewValue
 		);
 		assert_err!(
@@ -291,11 +282,10 @@ fn setting_pending_config_members() {
 			max_code_size: 100_000,
 			max_pov_size: 1024,
 			max_head_data_size: 1_000,
-			parathread_cores: 2,
-			parathread_retries: 5,
+			on_demand_cores: 2,
+			on_demand_retries: 5,
 			group_rotation_frequency: 20,
-			chain_availability_period: 10,
-			thread_availability_period: 8,
+			paras_availability_period: 10,
 			scheduling_lookahead: 3,
 			max_validators_per_core: None,
 			max_validators: None,
@@ -316,14 +306,17 @@ fn setting_pending_config_members() {
 			hrmp_channel_max_capacity: 3921,
 			hrmp_channel_max_total_size: 7687,
 			hrmp_max_parachain_inbound_channels: 37,
-			hrmp_max_parathread_inbound_channels: 19,
 			hrmp_channel_max_message_size: 8192,
 			hrmp_max_parachain_outbound_channels: 10,
-			hrmp_max_parathread_outbound_channels: 20,
 			hrmp_max_message_num_per_candidate: 20,
 			pvf_voting_ttl: 3,
 			minimum_validation_upgrade_delay: 20,
 			executor_params: Default::default(),
+			on_demand_queue_max_size: 10_000u32,
+			on_demand_base_fee: 10_000_000u128,
+			on_demand_fee_variability: Perbill::from_percent(3),
+			on_demand_target_queue_utilization: Perbill::from_percent(25),
+			on_demand_ttl: 5u32,
 		};
 
 		Configuration::set_validation_upgrade_cooldown(
@@ -345,9 +338,9 @@ fn setting_pending_config_members() {
 		Configuration::set_max_pov_size(RuntimeOrigin::root(), new_config.max_pov_size).unwrap();
 		Configuration::set_max_head_data_size(RuntimeOrigin::root(), new_config.max_head_data_size)
 			.unwrap();
-		Configuration::set_parathread_cores(RuntimeOrigin::root(), new_config.parathread_cores)
+		Configuration::set_on_demand_cores(RuntimeOrigin::root(), new_config.on_demand_cores)
 			.unwrap();
-		Configuration::set_parathread_retries(RuntimeOrigin::root(), new_config.parathread_retries)
+		Configuration::set_on_demand_retries(RuntimeOrigin::root(), new_config.on_demand_retries)
 			.unwrap();
 		Configuration::set_group_rotation_frequency(
 			RuntimeOrigin::root(),
@@ -361,14 +354,9 @@ fn setting_pending_config_members() {
 			new_config.minimum_validation_upgrade_delay,
 		)
 		.unwrap();
-		Configuration::set_chain_availability_period(
+		Configuration::set_paras_availability_period(
 			RuntimeOrigin::root(),
-			new_config.chain_availability_period,
-		)
-		.unwrap();
-		Configuration::set_thread_availability_period(
-			RuntimeOrigin::root(),
-			new_config.thread_availability_period,
+			new_config.paras_availability_period,
 		)
 		.unwrap();
 		Configuration::set_scheduling_lookahead(
@@ -462,11 +450,6 @@ fn setting_pending_config_members() {
 			new_config.hrmp_max_parachain_inbound_channels,
 		)
 		.unwrap();
-		Configuration::set_hrmp_max_parathread_inbound_channels(
-			RuntimeOrigin::root(),
-			new_config.hrmp_max_parathread_inbound_channels,
-		)
-		.unwrap();
 		Configuration::set_hrmp_channel_max_message_size(
 			RuntimeOrigin::root(),
 			new_config.hrmp_channel_max_message_size,
@@ -477,11 +460,6 @@ fn setting_pending_config_members() {
 			new_config.hrmp_max_parachain_outbound_channels,
 		)
 		.unwrap();
-		Configuration::set_hrmp_max_parathread_outbound_channels(
-			RuntimeOrigin::root(),
-			new_config.hrmp_max_parathread_outbound_channels,
-		)
-		.unwrap();
 		Configuration::set_hrmp_max_message_num_per_candidate(
 			RuntimeOrigin::root(),
 			new_config.hrmp_max_message_num_per_candidate,