CLI: refactoring: remove Options from sc_service::Configuration's fields (#5271)

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* Rename IntoConfiguration to CliConfiguration

* Renamed into_configuration to create_configuration

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* Move keystore params to its own module

* Use in-memory keystore even for build-spec

* Enforce proper value for node name

* dev_key_seed

* Telemetry endpoints

* rustfmt

* Converted all RunCmd

* rustfmt

* Added export-blocks

* Missed something

* Removed config_path in NetworkConfiguration (not used)

* Fixed warnings

* public_addresses is used but never set, keeping it

* Merge Configuration.node and NetworkConfiguration.node_name

...because they are the same thing

* Added: import-blocks

* Adding a proc_macro to help impl SubstrateCli

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* Re-export spec_factory from sc_cli

* Re-added all the commands

* Refactored node_key_params

* Fixed previous refucktoring

* Clean-up and removed full_version()

* Renamed get_is_dev to not confuse with Configuration field

* Fixed sc-cli-derive example

* Fixing tests

* Fixing tests and removing some (will re-add later)

* Fixing more tests

* Removes the need of type parameter

* Converting bin/node and simplifying API

* Converting more

* Converting last command

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* Fixing tests and added default for WasmExecutionMethod

* Fixing stuff

* Fixed something I broke oops

* Update Cargo.lock

* Moving things around

* Convert everything to Result

* Added new macros to simplify the impl of CliConfiguration

* Added a macro to generate CliConfiguration automatically for subcommands

* Revert... too many macros (this one is not really useful)

This reverts commit 9c516dd38b40fbc420b02c1f8e61d5b2b1a4e434.

* Renamed is_dev to get_is_dev

Good enough for now

* Fixed name roles (this is plural, not singular)

* Clean-up

* Re-export NodeKeyConfig and TelemetryEndpoints from sc_service

* Improve styling/formatting

* Added copyrights

* Added doc and fixed warnings

* Added myself to code owners

* Yes it is needed according to the history

* Revert formatting

* Fixing conflict

* Updated build.rs

* Cargo.lock

* Clean-up

* Update client/cli-derive/Cargo.toml

Co-Authored-By: Seun Lanlege <seunlanlege@gmail.com>

* Fail if using proc_macro and build.rs is not set properly

* Dropped all get_ in front of methods

* Clean-up

* Fixing proc macro missing env var

* Get the configuration inside the Runtime (needed for polkadot)

* Clean-up

* Get is_dev from argument like the others

* Get chain ID instead of chain spec from shared params

* &self is passed to spec_factory/load_spec

* Wrong text

* Fix example

* Officialize macro and made a cool doc

* Renamed spec_factory to load_spec (substrate_cli_configuration)

* Removed not so useful ChainSpec

* Renamed SubstrateCLI to SubstrateCli

* Added changelog for impl_version being full now

* Renamed Runtime to Runner

* Update changelog to show example

* Removed option on database cache size

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* Fix on removal of option

* typo

* Clean-up imports

* Added info in Cargo.toml

* typo

* remarks

* Moved function for build.rs to substrate-build-script-utils

* Fixed example & test of cli-derive

* Moved function for build.rs to substrate-build-script-utils

* Renamed substrate_cli_configuration to substrate_cli oops

It implements SubstrateCli not CliConfiguration!

* Added documentation and wrapper macro

* Removed option on database cache size

* Removed option on database cache size

* Clean-up

* Reduce risk of errors due to typos

* Removed option on database cache size

* Added NOTE as suggested

* Added doc as suggested

* Fixed test

* typo

* renamed runtime to runner

* Fixed weird argument

* More commas

* Moved client/cli-derive to client/cli/derive

* Added 7 tests for the macros

* Improve error message

* Upgrade assert_cmd

* Fixing missing stuff

* Fixed unused import

* Improve SubstrateCli doc

* Applied suggestions

* Fix and clean-up imports

* Started replacing macros WIP

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* Started removing substrate_cli

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* WIP

Forked at: d6aa8e954c
Parent branch: origin/master

* fixed bug introduced while refactoring

* Renamed NetworkConfigurationParams to NetworkParams for consistency sake

* Fixed test

* Update client/cli/src/commands/runcmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/runcmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/check_block_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update bin/node/cli/src/command.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update bin/node/cli/src/command.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Revert "Update client/cli/src/commands/export_blocks_cmd.rs"

This reverts commit 5906776953392c02beac6bc0bf50f8cbe1a12a01.

* Revert "Update client/cli/src/commands/check_block_cmd.rs"

This reverts commit f705f42b7f3d732be001141afee210fe46a1ef47.

* Revert "Update client/cli/src/commands/export_blocks_cmd.rs"

This reverts commit 8d57c0550164449e6eb2d3bacb04c750c714fcea.

* Revert "Update client/cli/src/commands/runcmd.rs"

This reverts commit 93e74cf5d2e1c0dc49cdff8608d59fc40fc59338.

* Revert "Update client/cli/src/commands/runcmd.rs"

This reverts commit 11d527ba345c0d79f0d3b5b071933d95474d0614.

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/import_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Changed ::sc_cli to $crate in the macro

* fixed tests

* fixed conflicts

* Fixing test

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/params/pruning_params.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Remove comment as suggested

* Apply suggestion

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/frame/benchmarking-cli/src/command.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/params/pruning_params.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/params/node_key_params.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/params/network_params.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/lib.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/config.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Added doc

* Fixed error introduced after applying suggestion

* Revert "Update client/cli/src/params/pruning_params.rs"

This reverts commit 0574d06a4f1efd86e94c1214420a12e7a4be0099.

* Print error

* Apply suggestions from code review

* Remove useless Results

* Fixed CI failing on polkadot approval

Co-authored-by: Seun Lanlege <seunlanlege@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
This commit is contained in:
Cecile Tonglet
2020-04-07 11:38:07 +02:00
committed by GitHub
parent fb9bbf306d
commit 3da069e359
66 changed files with 2916 additions and 2156 deletions
+462
View File
@@ -0,0 +1,462 @@
// Copyright 2020 Parity Technologies (UK) Ltd.
// This file is part of Substrate.
// Substrate is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Substrate is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Substrate. If not, see <http://www.gnu.org/licenses/>.
//! Configuration trait for a CLI based on substrate
use crate::error::Result;
use crate::{
init_logger, ImportParams, KeystoreParams, NetworkParams, NodeKeyParams,
PruningParams, SharedParams, SubstrateCli,
};
use app_dirs::{AppDataType, AppInfo};
use names::{Generator, Name};
use sc_service::config::{
Configuration, DatabaseConfig, ExecutionStrategies, ExtTransport, KeystoreConfig,
NetworkConfiguration, NodeKeyConfig, PrometheusConfig, PruningMode, Role, TelemetryEndpoints,
TransactionPoolOptions, WasmExecutionMethod,
};
use sc_service::{ChainSpec, TracingReceiver};
use std::future::Future;
use std::net::SocketAddr;
use std::path::PathBuf;
use std::pin::Pin;
use std::sync::Arc;
/// The maximum number of characters for a node name.
pub(crate) const NODE_NAME_MAX_LENGTH: usize = 32;
/// default sub directory to store network config
pub(crate) const DEFAULT_NETWORK_CONFIG_PATH: &'static str = "network";
/// A trait that allows converting an object to a Configuration
pub trait CliConfiguration: Sized {
/// Get the SharedParams for this object
fn shared_params(&self) -> &SharedParams;
/// Get the ImportParams for this object
fn import_params(&self) -> Option<&ImportParams> {
None
}
/// Get the PruningParams for this object
fn pruning_params(&self) -> Option<&PruningParams> {
self.import_params().map(|x| &x.pruning_params)
}
/// Get the KeystoreParams for this object
fn keystore_params(&self) -> Option<&KeystoreParams> {
None
}
/// Get the NetworkParams for this object
fn network_params(&self) -> Option<&NetworkParams> {
None
}
/// Get the NodeKeyParams for this object
fn node_key_params(&self) -> Option<&NodeKeyParams> {
self.network_params()
.map(|x| &x.node_key_params)
}
/// Get the base path of the configuration (if any)
///
/// By default this is retrieved from `SharedParams`.
fn base_path(&self) -> Result<Option<PathBuf>> {
Ok(self.shared_params().base_path())
}
/// Returns `true` if the node is for development or not
///
/// By default this is retrieved from `SharedParams`.
fn is_dev(&self) -> Result<bool> {
Ok(self.shared_params().is_dev())
}
/// Gets the role
///
/// By default this is `Role::Full`.
fn role(&self, _is_dev: bool) -> Result<Role> {
Ok(Role::Full)
}
/// Get the transaction pool options
///
/// By default this is `TransactionPoolOptions::default()`.
fn transaction_pool(&self) -> Result<TransactionPoolOptions> {
Ok(Default::default())
}
/// Get the network configuration
///
/// By default this is retrieved from `NetworkParams` if it is available otherwise it creates
/// a default `NetworkConfiguration` based on `node_name`, `client_id`, `node_key` and
/// `net_config_dir`.
fn network_config(
&self,
chain_spec: &Box<dyn ChainSpec>,
is_dev: bool,
net_config_dir: &PathBuf,
client_id: &str,
node_name: &str,
node_key: NodeKeyConfig,
) -> Result<NetworkConfiguration> {
Ok(if let Some(network_params) = self.network_params() {
network_params.network_config(
chain_spec,
is_dev,
net_config_dir,
client_id,
node_name,
node_key,
)
} else {
NetworkConfiguration::new(
node_name,
client_id,
node_key,
net_config_dir,
)
})
}
/// Get the keystore configuration.
///
/// Bu default this is retrieved from `KeystoreParams` if it is available. Otherwise it uses
/// `KeystoreConfig::InMemory`.
fn keystore_config(&self, base_path: &PathBuf) -> Result<KeystoreConfig> {
self.keystore_params()
.map(|x| x.keystore_config(base_path))
.unwrap_or(Ok(KeystoreConfig::InMemory))
}
/// Get the database cache size.
///
/// By default this is retrieved from `ImportParams` if it is available. Otherwise its `None`.
fn database_cache_size(&self) -> Result<Option<usize>> {
Ok(self.import_params()
.map(|x| x.database_cache_size())
.unwrap_or(Default::default()))
}
/// Get the database configuration.
///
/// By default this is retrieved from `SharedParams`
fn database_config(&self, base_path: &PathBuf, cache_size: usize) -> Result<DatabaseConfig> {
Ok(self.shared_params().database_config(base_path, cache_size))
}
/// Get the state cache size.
///
/// By default this is retrieved from `ImportParams` if it is available. Otherwise its `0`.
fn state_cache_size(&self) -> Result<usize> {
Ok(self.import_params()
.map(|x| x.state_cache_size())
.unwrap_or(Default::default()))
}
/// Get the state cache child ratio (if any).
///
/// By default this is `None`.
fn state_cache_child_ratio(&self) -> Result<Option<usize>> {
Ok(Default::default())
}
/// Get the pruning mode.
///
/// By default this is retrieved from `PruningMode` if it is available. Otherwise its
/// `PruningMode::default()`.
fn pruning(&self, is_dev: bool, role: &Role) -> Result<PruningMode> {
self.pruning_params()
.map(|x| x.pruning(is_dev, role))
.unwrap_or(Ok(Default::default()))
}
/// Get the chain ID (string).
///
/// By default this is retrieved from `SharedParams`.
fn chain_id(&self, is_dev: bool) -> Result<String> {
Ok(self.shared_params().chain_id(is_dev))
}
/// Get the name of the node.
///
/// By default a random name is generated.
fn node_name(&self) -> Result<String> {
Ok(generate_node_name())
}
/// Get the WASM execution method.
///
/// By default this is retrieved from `ImportParams` if it is available. Otherwise its
/// `WasmExecutionMethod::default()`.
fn wasm_method(&self) -> Result<WasmExecutionMethod> {
Ok(self.import_params()
.map(|x| x.wasm_method())
.unwrap_or(Default::default()))
}
/// Get the execution strategies.
///
/// By default this is retrieved from `ImportParams` if it is available. Otherwise its
/// `ExecutionStrategies::default()`.
fn execution_strategies(&self, is_dev: bool) -> Result<ExecutionStrategies> {
Ok(self.import_params()
.map(|x| x.execution_strategies(is_dev))
.unwrap_or(Default::default()))
}
/// Get the RPC HTTP address (`None` if disabled).
///
/// By default this is `None`.
fn rpc_http(&self) -> Result<Option<SocketAddr>> {
Ok(Default::default())
}
/// Get the RPC websocket address (`None` if disabled).
///
/// By default this is `None`.
fn rpc_ws(&self) -> Result<Option<SocketAddr>> {
Ok(Default::default())
}
/// Get the RPC websockets maximum connections (`None` if unlimited).
///
/// By default this is `None`.
fn rpc_ws_max_connections(&self) -> Result<Option<usize>> {
Ok(Default::default())
}
/// Get the RPC cors (`None` if disabled)
///
/// By default this is `None`.
fn rpc_cors(&self, _is_dev: bool) -> Result<Option<Vec<String>>> {
Ok(Some(Vec::new()))
}
/// Get the prometheus configuration (`None` if disabled)
///
/// By default this is `None`.
fn prometheus_config(&self) -> Result<Option<PrometheusConfig>> {
Ok(Default::default())
}
/// Get the telemetry endpoints (if any)
///
/// By default this is retrieved from the chain spec loaded by `load_spec`.
fn telemetry_endpoints(
&self,
chain_spec: &Box<dyn ChainSpec>,
) -> Result<Option<TelemetryEndpoints>> {
Ok(chain_spec.telemetry_endpoints().clone())
}
/// Get the telemetry external transport
///
/// By default this is `None`.
fn telemetry_external_transport(&self) -> Result<Option<ExtTransport>> {
Ok(Default::default())
}
/// Get the default value for heap pages
///
/// By default this is `None`.
fn default_heap_pages(&self) -> Result<Option<u64>> {
Ok(Default::default())
}
/// Returns `Ok(true)` if offchain worker should be used
///
/// By default this is `false`.
fn offchain_worker(&self, _role: &Role) -> Result<bool> {
Ok(Default::default())
}
/// Returns `Ok(true)` if authoring should be forced
///
/// By default this is `false`.
fn force_authoring(&self) -> Result<bool> {
Ok(Default::default())
}
/// Returns `Ok(true)` if grandpa should be disabled
///
/// By default this is `false`.
fn disable_grandpa(&self) -> Result<bool> {
Ok(Default::default())
}
/// Get the development key seed from the current object
///
/// By default this is `None`.
fn dev_key_seed(&self, _is_dev: bool) -> Result<Option<String>> {
Ok(Default::default())
}
/// Get the tracing targets from the current object (if any)
///
/// By default this is retrieved from `ImportParams` if it is available. Otherwise its
/// `None`.
fn tracing_targets(&self) -> Result<Option<String>> {
Ok(self.import_params()
.map(|x| x.tracing_targets())
.unwrap_or(Default::default()))
}
/// Get the TracingReceiver value from the current object
///
/// By default this is retrieved from `ImportParams` if it is available. Otherwise its
/// `TracingReceiver::default()`.
fn tracing_receiver(&self) -> Result<TracingReceiver> {
Ok(self.import_params()
.map(|x| x.tracing_receiver())
.unwrap_or(Default::default()))
}
/// Get the node key from the current object
///
/// By default this is retrieved from `NodeKeyParams` if it is available. Otherwise its
/// `NodeKeyConfig::default()`.
fn node_key(&self, net_config_dir: &PathBuf) -> Result<NodeKeyConfig> {
self.node_key_params()
.map(|x| x.node_key(net_config_dir))
.unwrap_or(Ok(Default::default()))
}
/// Get maximum runtime instances
///
/// By default this is `None`.
fn max_runtime_instances(&self) -> Result<Option<usize>> {
Ok(Default::default())
}
/// Activate or not the automatic announcing of blocks after import
///
/// By default this is `false`.
fn announce_block(&self) -> Result<bool> {
Ok(true)
}
/// Create a Configuration object from the current object
fn create_configuration<C: SubstrateCli>(
&self,
cli: &C,
task_executor: Arc<dyn Fn(Pin<Box<dyn Future<Output = ()> + Send>>) + Send + Sync>,
) -> Result<Configuration> {
let is_dev = self.is_dev()?;
let chain_id = self.chain_id(is_dev)?;
let chain_spec = cli.load_spec(chain_id.as_str())?;
let config_dir = self
.base_path()?
.unwrap_or_else(|| {
app_dirs::get_app_root(
AppDataType::UserData,
&AppInfo {
name: C::executable_name(),
author: C::author(),
},
)
.expect("app directories exist on all supported platforms; qed")
})
.join("chains")
.join(chain_spec.id());
let net_config_dir = config_dir.join(DEFAULT_NETWORK_CONFIG_PATH);
let client_id = C::client_id();
let database_cache_size = self.database_cache_size()?.unwrap_or(128);
let node_key = self.node_key(&net_config_dir)?;
let role = self.role(is_dev)?;
let max_runtime_instances = self.max_runtime_instances()?.unwrap_or(8);
Ok(Configuration {
impl_name: C::impl_name(),
impl_version: C::impl_version(),
task_executor,
transaction_pool: self.transaction_pool()?,
network: self.network_config(
&chain_spec,
is_dev,
&net_config_dir,
client_id.as_str(),
self.node_name()?.as_str(),
node_key,
)?,
keystore: self.keystore_config(&config_dir)?,
database: self.database_config(&config_dir, database_cache_size)?,
state_cache_size: self.state_cache_size()?,
state_cache_child_ratio: self.state_cache_child_ratio()?,
pruning: self.pruning(is_dev, &role)?,
wasm_method: self.wasm_method()?,
execution_strategies: self.execution_strategies(is_dev)?,
rpc_http: self.rpc_http()?,
rpc_ws: self.rpc_ws()?,
rpc_ws_max_connections: self.rpc_ws_max_connections()?,
rpc_cors: self.rpc_cors(is_dev)?,
prometheus_config: self.prometheus_config()?,
telemetry_endpoints: self.telemetry_endpoints(&chain_spec)?,
telemetry_external_transport: self.telemetry_external_transport()?,
default_heap_pages: self.default_heap_pages()?,
offchain_worker: self.offchain_worker(&role)?,
force_authoring: self.force_authoring()?,
disable_grandpa: self.disable_grandpa()?,
dev_key_seed: self.dev_key_seed(is_dev)?,
tracing_targets: self.tracing_targets()?,
tracing_receiver: self.tracing_receiver()?,
chain_spec,
max_runtime_instances,
announce_block: self.announce_block()?,
role,
})
}
/// Get the filters for the logging.
///
/// By default this is retrieved from `SharedParams`.
fn log_filters(&self) -> Result<Option<String>> {
Ok(self.shared_params().log_filters())
}
/// Initialize substrate. This must be done only once.
///
/// This method:
///
/// 1. Set the panic handler
/// 2. Raise the FD limit
/// 3. Initialize the logger
fn init<C: SubstrateCli>(&self) -> Result<()> {
let logger_pattern = self.log_filters()?.unwrap_or_default();
sp_panic_handler::set(C::support_url(), C::impl_version());
fdlimit::raise_fd_limit();
init_logger(logger_pattern.as_str());
Ok(())
}
}
/// Generate a valid random name for the node
pub fn generate_node_name() -> String {
loop {
let node_name = Generator::with_naming(Name::Numbered)
.next()
.expect("RNG is available on all supported platforms; qed");
let count = node_name.chars().count();
if count < NODE_NAME_MAX_LENGTH {
return node_name;
}
};
}