New sessions, kill consensus module (#2802)

* Draft of new sessions

* Reintroduce tuple impls

* Move staking module to new session API

* More work on staking and grandpa.

* Use iterator to avoid cloning and tuple macro

* Make runtime build again

* Polish the OpaqueKeys devex

* Move consensus logic into system & aura.

* Fix up system module

* Get build mostly going. Stuck at service.rs

* Building again

* Update srml/staking/src/lib.rs

Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>

* Refactoring out Consensus module, AuthorityIdOf, &c.

* Refactored out DigestItem::AuthoritiesChanged. Building.

* Remove tentative code

* Remove invalid comment

* Make Seal opaque and introduce nice methods for handling opaque items.

* Start to use proper digest for Aura authorities tracking.

* Fix up grandpa, remove system::Raw/Log

* Refactor Grandpa to use new logging infrastructure.

Also make authorityid/sessionkey static. Switch over to storing
authorities in a straight Vec.

* Building again

* Tidy up some AuthorityIds

* Expunge most of the rest of the AuthorityKey confusion.

Also, de-generify Babe and re-generify Aura.

* Remove cruft

* Untangle last of the `AuthorityId`s.

* Sort out finality_tracker

* Refactor median getting

* Apply suggestions from code review

Co-Authored-By: Robert Habermeier <rphmeier@gmail.com>

* Session tests works

* Update core/sr-primitives/src/generic/digest.rs

Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>

* Session tests works

* Fix for staking from @dvc94ch

* log an error

* fix test runtime build

* Some test fixes

* Staking mock update to new session api.

* Fix build.

* Move OpaqueKeys to primitives.

* Use on_initialize instead of check_rotate_session.

* Update tests to new staking api.

* fixup mock

* Fix bond_extra_and_withdraw_unbonded_works.

* Fix bond_with_little_staked_value_bounded_by_slot_stake.

* Fix bond_with_no_staked_value.

* Fix change_controller_works.

* Fix less_than_needed_candidates_works.

* Fix multi_era_reward_should_work.

* Fix nominating_and_rewards_should_work.

* Fix nominators_also_get_slashed.

* Fix phragmen_large_scale_test.

* Fix phragmen_poc_works.

* Fix phragmen_score_should_be_accurate_on_large_stakes.

* Fix phragmen_should_not_overflow.

* Fix reward_destination_works.

* Fix rewards_should_work.

* Fix sessions_and_eras_should_work.

* Fix slot_stake_is_least_staked_validator.

* Fix too_many_unbond_calls_should_not_work.

* Fix wrong_vote_is_null.

* Fix runtime.

* Fix wasm runtime build.

* Update Cargo.lock

* Fix warnings.

* Fix grandpa tests.

* Fix test-runtime build.

* Fix template node build.

* Fix stuff.

* Update Cargo.lock to fix CI

* Re-add missing AuRa logs

Runtimes are required to know about every digest they receive ― they
panic otherwise.  This re-adds support for AuRa pre-runtime digests.

* Update core/consensus/babe/src/digest.rs

Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>

* Kill log trait and all that jazz.

* Refactor staking tests.

* Fix ci runtime wasm check.

* Line length 120.

* Make tests build again

* Remove trailing commas in function declarations

The `extern_functions!` macro doesn’t like them, perhaps due to a bug in
rustc.

* Fix type error

* Fix compilation errors

* Fix a test

* Another couple of fixes

* Fix another test

* More test fixes

* Another test fix

* Bump runtime.

* Wrap long line

* Fix build, remove redundant code.

* Issue to track TODO

* Leave the benchmark code alone.

* Fix missing `std::time::{Instant, Duration}`

* Indentation

* Aura ConsensusLog as enum

substrate/srml/support/src/dispatch.rs

@@ -750,6 +750,7 @@ macro_rules! decl_module {
 		$vis:vis fn $name:ident ( root $(, $param:ident : $param_ty:ty )* ) { $( $impl:tt )* }
 	) => {
 		$(#[doc = $doc_attr])*
+		#[allow(unreachable_code)]
 		$vis fn $name($( $param: $param_ty ),* ) -> $crate::dispatch::Result {
 			{ $( $impl )* }
 			Ok(())

substrate/srml/support/src/lib.rs

@@ -270,7 +270,8 @@ mod tests {
 			pub GenericData get(generic_data): linked_map hasher(twox_128) T::BlockNumber => T::BlockNumber;
 			pub GenericData2 get(generic_data2): linked_map T::BlockNumber => Option<T::BlockNumber>;
 
-			pub DataDM config(test_config) build(|_| vec![(15u32, 16u32, 42u64)]): double_map hasher(twox_64_concat) u32, blake2_256(u32) => u64;
+			pub DataDM config(test_config) build(|_| vec![(15u32, 16u32, 42u64)]):
+				double_map hasher(twox_64_concat) u32, blake2_256(u32) => u64;
 			pub GenericDataDM: double_map T::BlockNumber, twox_128(T::BlockNumber) => T::BlockNumber;
 			pub GenericData2DM: double_map T::BlockNumber, twox_256(T::BlockNumber) => Option<T::BlockNumber>;
 			pub AppendableDM: double_map u32, blake2_256(T::BlockNumber) => Vec<u32>;
@@ -440,7 +441,9 @@ mod tests {
 				modifier: StorageFunctionModifier::Default,
 				ty: StorageFunctionType::Map{
 					hasher: StorageHasher::Twox128,
-					key: DecodeDifferent::Encode("T::BlockNumber"), value: DecodeDifferent::Encode("T::BlockNumber"), is_linked: true
+					key: DecodeDifferent::Encode("T::BlockNumber"),
+					value: DecodeDifferent::Encode("T::BlockNumber"),
+					is_linked: true
 				},
 				default: DecodeDifferent::Encode(
 					DefaultByteGetter(&__GetByteStructGenericData(PhantomData::<Test>))
@@ -452,7 +455,9 @@ mod tests {
 				modifier: StorageFunctionModifier::Optional,
 				ty: StorageFunctionType::Map{
 					hasher: StorageHasher::Blake2_256,
-					key: DecodeDifferent::Encode("T::BlockNumber"), value: DecodeDifferent::Encode("T::BlockNumber"), is_linked: true
+					key: DecodeDifferent::Encode("T::BlockNumber"),
+					value: DecodeDifferent::Encode("T::BlockNumber"),
+					is_linked: true
 				},
 				default: DecodeDifferent::Encode(
 					DefaultByteGetter(&__GetByteStructGenericData2(PhantomData::<Test>))

substrate/srml/support/src/runtime.rs

@@ -30,13 +30,13 @@
 ///
 /// ```nocompile
 /// construct_runtime!(
-///     pub enum Runtime with Log(interalIdent: DigestItem<SessionKey>) where
+///     pub enum Runtime where
 ///         Block = Block,
 ///         NodeBlock = runtime::Block,
 ///         UncheckedExtrinsic = UncheckedExtrinsic
 ///     {
 ///         System: system,
-///         Test: test::{default, Log(Test)},
+///         Test: test::{default},
 ///         Test2: test_with_long_module::{Module},
 ///
 ///         // Module with instances
@@ -50,8 +50,8 @@
 /// The identifier `System` is the name of the module and the lower case identifier `system` is the
 /// name of the Rust module/crate for this Substrate module.
 ///
-/// The module `Test: test::{default, Log(Test)}` will expand to
-/// `Test: test::{Module, Call, Storage, Event<T>, Config<T>, Log(Test)}`.
+/// The module `Test: test::{default}` will expand to
+/// `Test: test::{Module, Call, Storage, Event<T>, Config<T>}`.
 ///
 /// The module `Test2: test_with_long_module::{Module}` will expand to
 /// `Test2: test_with_long_module::{Module}`.
@@ -64,7 +64,6 @@
 /// - `Event` or `Event<T>` (if the event is generic) or `Event<T, I>` (if also over instance)
 /// - `Origin` or `Origin<T>` (if the origin is generic) or `Origin<T, I>` (if also over instance)
 /// - `Config` or `Config<T>` (if the config is generic) or `Config<T, I>` (if also over instance)
-/// - `Log( $(IDENT),* )`
 /// - `Inherent $( (CALL) )*` - If the module provides/can check inherents. The optional parameter
 ///                             is for modules that use a `Call` from a different module as
 ///                             inherent.
@@ -82,7 +81,7 @@ macro_rules! construct_runtime {
 	// form)
 
 	(
-		pub enum $runtime:ident with Log ($log_internal:ident: DigestItem<$( $log_genarg:ty ),+>)
+		pub enum $runtime:ident
 			where
 				Block = $block:ident,
 				NodeBlock = $node_block:ty,
@@ -97,7 +96,6 @@ macro_rules! construct_runtime {
 				$block;
 				$node_block;
 				$uncheckedextrinsic;
-				$log_internal < $( $log_genarg ),* >;
 			};
 			{};
 			$( $rest )*
@@ -204,7 +202,6 @@ macro_rules! construct_runtime {
 			$block:ident;
 			$node_block:ty;
 			$uncheckedextrinsic:ident;
-			$log_internal:ident <$( $log_genarg:ty ),+>;
 		};
 		{
 			$(
@@ -264,14 +261,6 @@ macro_rules! construct_runtime {
 				$name: $module:: $( < $module_instance >:: )? { $( $modules )* }
 			)*
 		);
-		$crate::__decl_outer_log!(
-			$runtime;
-			$log_internal < $( $log_genarg ),* >;
-			{};
-			$(
-				$name: $module:: $( < $module_instance >:: )? { $( $modules $( ( $( $modules_args )* ) )* )* }
-			)*
-		);
 		$crate::__decl_outer_config!(
 			$runtime;
 			{};
@@ -539,26 +528,6 @@ macro_rules! __decl_all_modules {
 #[macro_export]
 #[doc(hidden)]
 macro_rules! __decl_outer_dispatch {
-	(
-		$runtime:ident;
-		$( $parsed_modules:ident :: $parsed_name:ident ),*;
-		System: $module:ident::{
-			$ignore:ident $( <$ignor:ident> )* $(, $modules:ident $( <$modules_generic:ident> )* )*
-		}
-		$(, $rest_name:ident : $rest_module:ident::{
-			$( $rest_modules:ident $( <$rest_modules_generic:ident> )* ),*
-		})*;
-	) => {
-		$crate::__decl_outer_dispatch!(
-			$runtime;
-			$( $parsed_modules :: $parsed_name ),*;
-			$(
-				$rest_name: $rest_module::{
-					$( $rest_modules $( <$rest_modules_generic> )* ),*
-				}
-			),*;
-		);
-	};
 	(
 		$runtime:ident;
 		$( $parsed_modules:ident :: $parsed_name:ident ),*;
@@ -698,75 +667,6 @@ macro_rules! __decl_runtime_metadata {
 				$( $parsed_modules::Module $( < $module_instance > )? with $( $withs )* , )*
 		);
 	}
-
-}
-/// A private macro that generates Log enum for the runtime. See impl_outer_log macro.
-#[macro_export]
-#[doc(hidden)]
-macro_rules! __decl_outer_log {
-	(
-		$runtime:ident;
-		$log_internal:ident <$( $log_genarg:ty ),+>;
-		{ $( $parsed:tt )* };
-		$name:ident: $module:ident:: $(<$module_instance:ident>::)? {
-			Log ( $( $args:ident )* ) $( $modules:ident $( ( $( $modules_args:ident )* ) )* )*
-		}
-		$( $rest:tt )*
-	) => {
-		$crate::__decl_outer_log!(
-			$runtime;
-			$log_internal < $( $log_genarg ),* >;
-			{ $( $parsed )* $module $(<$module_instance>)? ( $( $args )* )};
-			$( $rest )*
-		);
-	};
-	(
-		$runtime:ident;
-		$log_internal:ident <$( $log_genarg:ty ),+>;
-		{ $( $parsed:tt )* };
-		$name:ident: $module:ident:: $(<$module_instance:ident>::)? {
-			$ignore:ident $( ( $( $args_ignore:ident )* ) )*
-			$( $modules:ident $( ( $( $modules_args:ident )* ) )* )*
-		}
-		$( $rest:tt )*
-	) => {
-		$crate::__decl_outer_log!(
-			$runtime;
-			$log_internal < $( $log_genarg ),* >;
-			{ $( $parsed )* };
-			$name: $module:: $(<$module_instance>::)? { $( $modules $( ( $( $modules_args )* ) )* )* }
-			$( $rest )*
-		);
-	};
-	(
-		$runtime:ident;
-		$log_internal:ident <$( $log_genarg:ty ),+>;
-		{ $( $parsed:tt )* };
-		$name:ident: $module:ident:: $(<$module_instance:ident>::)? {}
-		$( $rest:tt )*
-	) => {
-		$crate::__decl_outer_log!(
-			$runtime;
-			$log_internal < $( $log_genarg ),* >;
-			{ $( $parsed )* };
-			$( $rest )*
-		);
-	};
-	(
-		$runtime:ident;
-		$log_internal:ident <$( $log_genarg:ty ),+>;
-		{ $(
-			$parsed_modules:ident $(< $parsed_instance:ident >)? ( $( $parsed_args:ident )* )
-		)* };
-	) => {
-		$crate::paste::item! {
-			$crate::runtime_primitives::impl_outer_log!(
-				pub enum Log($log_internal: DigestItem<$( $log_genarg ),*>) for $runtime {
-					$( [< $parsed_modules $(_ $parsed_instance)? >] $(< $parsed_modules::$parsed_instance >)? ( $( $parsed_args ),* ) ),*
-				}
-			);
-		}
-	};
 }
 
 /// A private macro that generates GenesisConfig for the runtime. See impl_outer_config macro.

substrate/srml/support/src/storage/hashed/mod.rs

@@ -24,73 +24,136 @@ use runtime_io::{self, twox_128};
 use crate::codec::{Codec, Encode, Decode, KeyedVec};
 
 /// Return the value of the item in storage under `key`, or `None` if there is no explicit entry.
-pub fn get<T: Decode + Sized, HashFn: Fn(&[u8]) -> R, R: AsRef<[u8]>>(hash: &HashFn, key: &[u8]) -> Option<T> {
+pub fn get<T, HashFn, R>(hash: &HashFn, key: &[u8]) -> Option<T>
+where
+	T: Decode + Sized,
+	HashFn: Fn(&[u8]) -> R,
+	R: AsRef<[u8]>,
+{
 	unhashed::get(&hash(key).as_ref())
 }
 
 /// Return the value of the item in storage under `key`, or the type's default if there is no
 /// explicit entry.
-pub fn get_or_default<T: Decode + Sized + Default, HashFn: Fn(&[u8]) -> R, R: AsRef<[u8]>>(hash: &HashFn, key: &[u8]) -> T {
+pub fn get_or_default<T, HashFn, R>(hash: &HashFn, key: &[u8]) -> T
+where
+	T: Decode + Sized + Default,
+	HashFn: Fn(&[u8]) -> R,
+	R: AsRef<[u8]>,
+{
 	unhashed::get_or_default(&hash(key).as_ref())
 }
 
 /// Return the value of the item in storage under `key`, or `default_value` if there is no
 /// explicit entry.
-pub fn get_or<T: Decode + Sized, HashFn: Fn(&[u8]) -> R, R: AsRef<[u8]>>(hash: &HashFn, key: &[u8], default_value: T) -> T {
+pub fn get_or<T, HashFn, R>(hash: &HashFn, key: &[u8], default_value: T) -> T
+where
+	T: Decode + Sized,
+	HashFn: Fn(&[u8]) -> R,
+	R: AsRef<[u8]>,
+{
 	unhashed::get_or(&hash(key).as_ref(), default_value)
 }
 
 /// Return the value of the item in storage under `key`, or `default_value()` if there is no
 /// explicit entry.
-pub fn get_or_else<T: Decode + Sized, F: FnOnce() -> T, HashFn: Fn(&[u8]) -> R, R: AsRef<[u8]>>(hash: &HashFn, key: &[u8], default_value: F) -> T {
+pub fn get_or_else<T, F, HashFn, R>(hash: &HashFn, key: &[u8], default_value: F) -> T
+where
+	T: Decode + Sized,
+	F: FnOnce() -> T,
+	HashFn: Fn(&[u8]) -> R,
+	R: AsRef<[u8]>,
+{
 	unhashed::get_or_else(&hash(key).as_ref(), default_value)
 }
 
 /// Put `value` in storage under `key`.
-pub fn put<T: Encode, HashFn: Fn(&[u8]) -> R, R: AsRef<[u8]>>(hash: &HashFn, key: &[u8], value: &T) {
+pub fn put<T, HashFn, R>(hash: &HashFn, key: &[u8], value: &T)
+where
+	T: Encode,
+	HashFn: Fn(&[u8]) -> R,
+	R: AsRef<[u8]>,
+{
 	unhashed::put(&hash(key).as_ref(), value)
 }
 
 /// Remove `key` from storage, returning its value if it had an explicit entry or `None` otherwise.
-pub fn take<T: Decode + Sized, HashFn: Fn(&[u8]) -> R, R: AsRef<[u8]>>(hash: &HashFn, key: &[u8]) -> Option<T> {
+pub fn take<T, HashFn, R>(hash: &HashFn, key: &[u8]) -> Option<T>
+where
+	T: Decode + Sized,
+	HashFn: Fn(&[u8]) -> R,
+	R: AsRef<[u8]>,
+{
 	unhashed::take(&hash(key).as_ref())
 }
 
 /// Remove `key` from storage, returning its value, or, if there was no explicit entry in storage,
 /// the default for its type.
-pub fn take_or_default<T: Decode + Sized + Default, HashFn: Fn(&[u8]) -> R, R: AsRef<[u8]>>(hash: &HashFn, key: &[u8]) -> T {
+pub fn take_or_default<T, HashFn, R>(hash: &HashFn, key: &[u8]) -> T
+where
+	T: Decode + Sized + Default,
+	HashFn: Fn(&[u8]) -> R,
+	R: AsRef<[u8]>,
+{
 	unhashed::take_or_default(&hash(key).as_ref())
 }
 
 /// Return the value of the item in storage under `key`, or `default_value` if there is no
 /// explicit entry. Ensure there is no explicit entry on return.
-pub fn take_or<T: Decode + Sized, HashFn: Fn(&[u8]) -> R, R: AsRef<[u8]>>(hash: &HashFn, key: &[u8], default_value: T) -> T {
+pub fn take_or<T, HashFn, R>(hash: &HashFn, key: &[u8], default_value: T) -> T
+where
+	T: Decode + Sized,
+	HashFn: Fn(&[u8]) -> R,
+	R: AsRef<[u8]>,
+{
 	unhashed::take_or(&hash(key).as_ref(), default_value)
 }
 
 /// Return the value of the item in storage under `key`, or `default_value()` if there is no
 /// explicit entry. Ensure there is no explicit entry on return.
-pub fn take_or_else<T: Decode + Sized, F: FnOnce() -> T, HashFn: Fn(&[u8]) -> R, R: AsRef<[u8]>>(hash: &HashFn, key: &[u8], default_value: F) -> T {
+pub fn take_or_else<T, F, HashFn, R>(hash: &HashFn, key: &[u8], default_value: F) -> T
+where
+	T: Decode + Sized,
+	F: FnOnce() -> T,
+	HashFn: Fn(&[u8]) -> R,
+	R: AsRef<[u8]>,
+{
 	unhashed::take_or_else(&hash(key).as_ref(), default_value)
 }
 
 /// Check to see if `key` has an explicit entry in storage.
-pub fn exists<HashFn: Fn(&[u8]) -> R, R: AsRef<[u8]>>(hash: &HashFn, key: &[u8]) -> bool {
+pub fn exists<HashFn, R>(hash: &HashFn, key: &[u8]) -> bool
+where
+	HashFn: Fn(&[u8]) -> R,
+	R: AsRef<[u8]>,
+{
 	unhashed::exists(&hash(key).as_ref())
 }
 
 /// Ensure `key` has no explicit entry in storage.
-pub fn kill<HashFn: Fn(&[u8]) -> R, R: AsRef<[u8]>>(hash: &HashFn, key: &[u8]) {
+pub fn kill<HashFn, R>(hash: &HashFn, key: &[u8])
+where
+	HashFn: Fn(&[u8]) -> R,
+	R: AsRef<[u8]>,
+{
 	unhashed::kill(&hash(key).as_ref())
 }
 
 /// Get a Vec of bytes from storage.
-pub fn get_raw<HashFn: Fn(&[u8]) -> R, R: AsRef<[u8]>>(hash: &HashFn, key: &[u8]) -> Option<Vec<u8>> {
+pub fn get_raw<HashFn, R>(hash: &HashFn, key: &[u8]) -> Option<Vec<u8>>
+where
+	HashFn: Fn(&[u8]) -> R,
+	R: AsRef<[u8]>,
+{
 	unhashed::get_raw(&hash(key).as_ref())
 }
 
 /// Put a raw byte slice into storage.
-pub fn put_raw<HashFn: Fn(&[u8]) -> R, R: AsRef<[u8]>>(hash: &HashFn, key: &[u8], value: &[u8]) {
+pub fn put_raw<HashFn, R>(hash: &HashFn, key: &[u8], value: &[u8])
+where
+	HashFn: Fn(&[u8]) -> R,
+	R: AsRef<[u8]>,
+{
 	unhashed::put_raw(&hash(key).as_ref(), value)
 }
 

substrate/srml/support/src/storage/unhashed/generator.rs

@@ -27,11 +27,15 @@ pub trait UnhashedStorage {
 
 	/// Load the bytes of a key from storage. Can panic if the type is incorrect. Will panic if
 	/// it's not there.
-	fn require<T: codec::Decode>(&self, key: &[u8]) -> T { self.get(key).expect("Required values must be in storage") }
+	fn require<T: codec::Decode>(&self, key: &[u8]) -> T {
+		self.get(key).expect("Required values must be in storage")
+	}
 
 	/// Load the bytes of a key from storage. Can panic if the type is incorrect. The type's
 	/// default is returned if it's not there.
-	fn get_or_default<T: codec::Decode + Default>(&self, key: &[u8]) -> T { self.get(key).unwrap_or_default() }
+	fn get_or_default<T: codec::Decode + Default>(&self, key: &[u8]) -> T {
+		self.get(key).unwrap_or_default()
+	}
 
 	/// Put a value in under a key.
 	fn put<T: codec::Encode>(&mut self, key: &[u8], val: &T);
@@ -50,10 +54,14 @@ pub trait UnhashedStorage {
 	}
 
 	/// Take a value from storage, deleting it after reading.
-	fn take_or_panic<T: codec::Decode>(&mut self, key: &[u8]) -> T { self.take(key).expect("Required values must be in storage") }
+	fn take_or_panic<T: codec::Decode>(&mut self, key: &[u8]) -> T {
+		self.take(key).expect("Required values must be in storage")
+	}
 
 	/// Take a value from storage, deleting it after reading.
-	fn take_or_default<T: codec::Decode + Default>(&mut self, key: &[u8]) -> T { self.take(key).unwrap_or_default() }
+	fn take_or_default<T: codec::Decode + Default>(&mut self, key: &[u8]) -> T {
+		self.take(key).unwrap_or_default()
+	}
 
 	/// Get a Vec of bytes from storage.
 	fn get_raw(&self, key: &[u8]) -> Option<Vec<u8>>;