feat: Rebrand Polkadot/Substrate references to PezkuwiChain
This commit systematically rebrands various references from Parity Technologies' Polkadot/Substrate ecosystem to PezkuwiChain within the kurdistan-sdk. Key changes include: - Updated external repository URLs (zombienet-sdk, parity-db, parity-scale-codec, wasm-instrument) to point to pezkuwichain forks. - Modified internal documentation and code comments to reflect PezkuwiChain naming and structure. - Replaced direct references to with or specific paths within the for XCM, Pezkuwi, and other modules. - Cleaned up deprecated issue and PR references in various and files, particularly in and modules. - Adjusted image and logo URLs in documentation to point to PezkuwiChain assets. - Removed or rephrased comments related to external Polkadot/Substrate PRs and issues. This is a significant step towards fully customizing the SDK for the PezkuwiChain ecosystem.
This commit is contained in:
@@ -0,0 +1,83 @@
|
||||
# The `benchmark machine` command
|
||||
|
||||
Different Bizinikiwi chains can have different hardware requirements.
|
||||
It is therefore important to be able to quickly gauge if a piece of hardware fits a chains' requirements.
|
||||
The `benchmark machine` command archives this by measuring key metrics and making them comparable.
|
||||
|
||||
Invoking the command looks like this:
|
||||
```sh
|
||||
cargo run --profile=production -- benchmark machine --dev
|
||||
```
|
||||
|
||||
## Output
|
||||
|
||||
The output on reference hardware:
|
||||
|
||||
```pre
|
||||
+----------+----------------+---------------+--------------+-------------------+
|
||||
| Category | Function | Score | Minimum | Result |
|
||||
+----------+----------------+---------------+--------------+-------------------+
|
||||
| CPU | BLAKE2-256 | 1023.00 MiB/s | 1.00 GiB/s | ✅ Pass ( 99.4 %) |
|
||||
+----------+----------------+---------------+--------------+-------------------+
|
||||
| CPU | SR25519-Verify | 665.13 KiB/s | 666.00 KiB/s | ✅ Pass ( 99.9 %) |
|
||||
+----------+----------------+---------------+--------------+-------------------+
|
||||
| Memory | Copy | 14.39 GiB/s | 14.32 GiB/s | ✅ Pass (100.4 %) |
|
||||
+----------+----------------+---------------+--------------+-------------------+
|
||||
| Disk | Seq Write | 457.00 MiB/s | 450.00 MiB/s | ✅ Pass (101.6 %) |
|
||||
+----------+----------------+---------------+--------------+-------------------+
|
||||
| Disk | Rnd Write | 190.00 MiB/s | 200.00 MiB/s | ✅ Pass ( 95.0 %) |
|
||||
+----------+----------------+---------------+--------------+-------------------+
|
||||
```
|
||||
|
||||
The *score* is the average result of each benchmark. It always adheres to "higher is better".
|
||||
|
||||
The *category* indicate which part of the hardware was benchmarked:
|
||||
- **CPU** Processor intensive task
|
||||
- **Memory** RAM intensive task
|
||||
- **Disk** Hard drive intensive task
|
||||
|
||||
The *function* is the concrete benchmark that was run:
|
||||
- **BLAKE2-256** The throughput of the [Blake2-256] cryptographic hashing function with 32 KiB input. The [blake2_256
|
||||
function] is used in many places in Bizinikiwi. The throughput of a hash function strongly depends on the input size,
|
||||
therefore we settled to use a fixed input size for comparable results.
|
||||
- **SR25519 Verify** Sr25519 is an optimized version of the [Curve25519] signature scheme. Signature verification is
|
||||
used by Bizinikiwi when verifying extrinsics and blocks.
|
||||
- **Copy** The throughput of copying memory from one place in the RAM to another.
|
||||
- **Seq Write** The throughput of writing data to the storage location sequentially. It is important that the same disk
|
||||
is used that will later-on be used to store the chain data.
|
||||
- **Rnd Write** The throughput of writing data to the storage location in a random order. This is normally much slower
|
||||
than the sequential write.
|
||||
|
||||
The *score* needs to reach the *minimum* in order to pass the benchmark. This can be reduced with the `--tolerance`
|
||||
flag.
|
||||
|
||||
The *result* indicated if a specific benchmark was passed by the machine or not. The percent number is the relative
|
||||
score reached to the *minimum* that is needed. The `--tolerance` flag is taken into account for this decision. For
|
||||
example a benchmark that passes even with 95% since the *tolerance* was set to 10% would look like this: `✅ Pass ( 95.0
|
||||
%)`.
|
||||
|
||||
## Interpretation
|
||||
|
||||
Ideally all results show a `Pass` and the program exits with code 0. Currently some of the benchmarks can fail even on
|
||||
reference hardware; they are still being improved to make them more deterministic.
|
||||
Make sure to run nothing else on the machine when benchmarking it.
|
||||
You can re-run them multiple times to get more reliable results.
|
||||
|
||||
## Arguments
|
||||
|
||||
- `--tolerance` A percent number to reduce the *minimum* requirement. This should be used to ignore outliers of the
|
||||
benchmarks. The default value is 10%.
|
||||
- `--verify-duration` How long the verification benchmark should run.
|
||||
- `--disk-duration` How long the *read* and *write* benchmarks should run each.
|
||||
- `--allow-fail` Always exit the program with code 0.
|
||||
- `--chain` / `--dev` Specify the chain config to use. This will be used to compare the results with the requirements of
|
||||
the chain (WIP).
|
||||
- [`--base-path`]
|
||||
|
||||
License: Apache-2.0
|
||||
|
||||
<!-- LINKS -->
|
||||
[Blake2-256]: https://www.blake2.net/
|
||||
[blake2_256 function]: https://docs.rs/pezsp-crypto-hashing/latest/sp_crypto_hashing/fn.blake2_256.html
|
||||
[Curve25519]: https://en.wikipedia.org/wiki/Curve25519
|
||||
[`--base-path`]: ../shared/README.md#arguments
|
||||
@@ -0,0 +1,85 @@
|
||||
// This file is part of Bizinikiwi.
|
||||
|
||||
// Copyright (C) Parity Technologies (UK) Ltd.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Contains types to define hardware requirements.
|
||||
|
||||
use pezsc_sysinfo::Requirements;
|
||||
use std::sync::LazyLock;
|
||||
|
||||
/// The hardware requirements as measured on reference hardware.
|
||||
///
|
||||
/// These values are provided by Parity, however it is possible
|
||||
/// to use your own requirements if you are running a custom chain.
|
||||
pub static BIZINIKIWI_REFERENCE_HARDWARE: LazyLock<Requirements> = LazyLock::new(|| {
|
||||
let raw = include_bytes!("reference_hardware.json").as_slice();
|
||||
serde_json::from_slice(raw).expect("Hardcoded data is known good; qed")
|
||||
});
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use pezsc_sysinfo::{Metric, Requirement, Requirements, Throughput};
|
||||
|
||||
/// `BIZINIKIWI_REFERENCE_HARDWARE` can be decoded.
|
||||
#[test]
|
||||
fn json_static_data() {
|
||||
let raw = serde_json::to_string(&*BIZINIKIWI_REFERENCE_HARDWARE).unwrap();
|
||||
let decoded: Requirements = serde_json::from_str(&raw).unwrap();
|
||||
|
||||
assert_eq!(decoded, BIZINIKIWI_REFERENCE_HARDWARE.clone());
|
||||
}
|
||||
|
||||
/// The hard-coded values are correct.
|
||||
#[test]
|
||||
fn json_static_data_is_correct() {
|
||||
assert_eq!(
|
||||
*BIZINIKIWI_REFERENCE_HARDWARE,
|
||||
Requirements(vec![
|
||||
Requirement {
|
||||
metric: Metric::Blake2256,
|
||||
minimum: Throughput::from_mibs(1000.00),
|
||||
validator_only: false
|
||||
},
|
||||
Requirement {
|
||||
metric: Metric::Blake2256Parallel { num_cores: 8 },
|
||||
minimum: Throughput::from_mibs(1000.00),
|
||||
validator_only: true,
|
||||
},
|
||||
Requirement {
|
||||
metric: Metric::Sr25519Verify,
|
||||
minimum: Throughput::from_kibs(637.619999744),
|
||||
validator_only: false
|
||||
},
|
||||
Requirement {
|
||||
metric: Metric::MemCopy,
|
||||
minimum: Throughput::from_gibs(11.4925205078125003),
|
||||
validator_only: false,
|
||||
},
|
||||
Requirement {
|
||||
metric: Metric::DiskSeqWrite,
|
||||
minimum: Throughput::from_mibs(950.0),
|
||||
validator_only: false,
|
||||
},
|
||||
Requirement {
|
||||
metric: Metric::DiskRndWrite,
|
||||
minimum: Throughput::from_mibs(420.0),
|
||||
validator_only: false
|
||||
},
|
||||
])
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,242 @@
|
||||
// This file is part of Bizinikiwi.
|
||||
|
||||
// Copyright (C) Parity Technologies (UK) Ltd.
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Contains the [`MachineCmd`] as entry point for the node
|
||||
//! and the core benchmarking logic.
|
||||
|
||||
pub mod hardware;
|
||||
|
||||
use std::{boxed::Box, fs, path::Path};
|
||||
|
||||
use clap::Parser;
|
||||
use comfy_table::{Row, Table};
|
||||
use log::{error, info, warn};
|
||||
|
||||
use pezsc_cli::{CliConfiguration, Result, SharedParams};
|
||||
use pezsc_service::Configuration;
|
||||
use pezsc_sysinfo::{
|
||||
benchmark_cpu, benchmark_cpu_parallelism, benchmark_disk_random_writes,
|
||||
benchmark_disk_sequential_writes, benchmark_memory, benchmark_sr25519_verify, ExecutionLimit,
|
||||
Metric, Requirement, Requirements, Throughput,
|
||||
};
|
||||
|
||||
use crate::shared::check_build_profile;
|
||||
pub use hardware::BIZINIKIWI_REFERENCE_HARDWARE;
|
||||
|
||||
/// Command to benchmark the hardware.
|
||||
///
|
||||
/// Runs multiple benchmarks and prints their output to console.
|
||||
/// Can be used to gauge if the hardware is fast enough to keep up with a chain's requirements.
|
||||
/// This command must be integrated by the client since the client can set compiler flags
|
||||
/// which influence the results.
|
||||
///
|
||||
/// You can use the `--base-path` flag to set a location for the disk benchmarks.
|
||||
#[derive(Debug, Parser)]
|
||||
pub struct MachineCmd {
|
||||
#[allow(missing_docs)]
|
||||
#[clap(flatten)]
|
||||
pub shared_params: SharedParams,
|
||||
|
||||
/// Do not return an error if any check fails.
|
||||
///
|
||||
/// Should only be used for debugging.
|
||||
#[arg(long)]
|
||||
pub allow_fail: bool,
|
||||
|
||||
/// Set a fault tolerance for passing a requirement.
|
||||
///
|
||||
/// 10% means that the test would pass even when only 90% score was archived.
|
||||
/// Can be used to mitigate outliers of the benchmarks.
|
||||
#[arg(long, default_value_t = 10.0, value_name = "PERCENT")]
|
||||
pub tolerance: f64,
|
||||
|
||||
/// Time limit for the verification benchmark.
|
||||
#[arg(long, default_value_t = 5.0, value_name = "SECONDS")]
|
||||
pub verify_duration: f32,
|
||||
|
||||
/// Time limit for the hash function benchmark.
|
||||
#[arg(long, default_value_t = 5.0, value_name = "SECONDS")]
|
||||
pub hash_duration: f32,
|
||||
|
||||
/// Time limit for the memory benchmark.
|
||||
#[arg(long, default_value_t = 5.0, value_name = "SECONDS")]
|
||||
pub memory_duration: f32,
|
||||
|
||||
/// Time limit for each disk benchmark.
|
||||
#[arg(long, default_value_t = 5.0, value_name = "SECONDS")]
|
||||
pub disk_duration: f32,
|
||||
}
|
||||
|
||||
/// Helper for the result of a concrete benchmark.
|
||||
struct BenchResult {
|
||||
/// Did the hardware pass the benchmark?
|
||||
passed: bool,
|
||||
|
||||
/// The absolute score that was archived.
|
||||
score: Throughput,
|
||||
|
||||
/// The score relative to the minimal required score.
|
||||
///
|
||||
/// Is in range [0, 1].
|
||||
rel_score: f64,
|
||||
}
|
||||
|
||||
/// Errors that can be returned by the this command.
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
#[allow(missing_docs)]
|
||||
pub enum Error {
|
||||
#[error("One of the benchmarks had a score that was lower than its requirement")]
|
||||
UnmetRequirement,
|
||||
|
||||
#[error("The build profile is unfit for benchmarking: {0}")]
|
||||
BadBuildProfile(String),
|
||||
|
||||
#[error("Benchmark results are off by at least factor 100")]
|
||||
BadResults,
|
||||
}
|
||||
|
||||
impl MachineCmd {
|
||||
/// Execute the benchmark and print the results.
|
||||
pub fn run(&self, cfg: &Configuration, requirements: Requirements) -> Result<()> {
|
||||
self.validate_args()?;
|
||||
// Ensure that the dir exists since the node is not started to take care of it.
|
||||
let dir = cfg.database.path().ok_or("No DB directory provided")?;
|
||||
fs::create_dir_all(dir)?;
|
||||
|
||||
info!("Running machine benchmarks...");
|
||||
let mut results = Vec::new();
|
||||
for requirement in &requirements.0 {
|
||||
let result = self.run_benchmark(requirement, &dir)?;
|
||||
results.push(result);
|
||||
}
|
||||
self.print_summary(requirements, results)
|
||||
}
|
||||
|
||||
/// Benchmarks a specific metric of the hardware and judges the resulting score.
|
||||
fn run_benchmark(&self, requirement: &Requirement, dir: &Path) -> Result<BenchResult> {
|
||||
// Dispatch the concrete function from `sc-sysinfo`.
|
||||
|
||||
let score = self.measure(&requirement.metric, dir)?;
|
||||
let rel_score = score.as_bytes() / requirement.minimum.as_bytes();
|
||||
|
||||
// Sanity check if the result is off by factor >100x.
|
||||
if rel_score >= 100.0 || rel_score <= 0.01 {
|
||||
self.check_failed(Error::BadResults)?;
|
||||
}
|
||||
let passed = rel_score >= (1.0 - (self.tolerance / 100.0));
|
||||
Ok(BenchResult { passed, score, rel_score })
|
||||
}
|
||||
|
||||
/// Measures a metric of the hardware.
|
||||
fn measure(&self, metric: &Metric, dir: &Path) -> Result<Throughput> {
|
||||
let verify_limit = ExecutionLimit::from_secs_f32(self.verify_duration);
|
||||
let disk_limit = ExecutionLimit::from_secs_f32(self.disk_duration);
|
||||
let hash_limit = ExecutionLimit::from_secs_f32(self.hash_duration);
|
||||
let memory_limit = ExecutionLimit::from_secs_f32(self.memory_duration);
|
||||
|
||||
let score = match metric {
|
||||
Metric::Blake2256 => benchmark_cpu(hash_limit),
|
||||
Metric::Blake2256Parallel { num_cores } =>
|
||||
benchmark_cpu_parallelism(hash_limit, *num_cores),
|
||||
Metric::Sr25519Verify => benchmark_sr25519_verify(verify_limit),
|
||||
Metric::MemCopy => benchmark_memory(memory_limit),
|
||||
Metric::DiskSeqWrite => benchmark_disk_sequential_writes(disk_limit, dir)?,
|
||||
Metric::DiskRndWrite => benchmark_disk_random_writes(disk_limit, dir)?,
|
||||
};
|
||||
Ok(score)
|
||||
}
|
||||
|
||||
/// Prints a human-readable summary.
|
||||
fn print_summary(&self, requirements: Requirements, results: Vec<BenchResult>) -> Result<()> {
|
||||
// Use a table for nicer console output.
|
||||
let mut table = Table::new();
|
||||
table.set_header(["Category", "Function", "Score", "Minimum", "Result"]);
|
||||
// Count how many passed and how many failed.
|
||||
let (mut passed, mut failed) = (0, 0);
|
||||
for (requirement, result) in requirements.0.iter().zip(results.iter()) {
|
||||
if result.passed {
|
||||
passed += 1
|
||||
} else {
|
||||
failed += 1
|
||||
}
|
||||
|
||||
table.add_row(result.to_row(requirement));
|
||||
}
|
||||
// Print the table and a summary.
|
||||
info!(
|
||||
"\n{}\nFrom {} benchmarks in total, {} passed and {} failed ({:.0?}% fault tolerance).",
|
||||
table,
|
||||
passed + failed,
|
||||
passed,
|
||||
failed,
|
||||
self.tolerance
|
||||
);
|
||||
// Print the final result.
|
||||
if failed != 0 {
|
||||
info!("The hardware fails to meet the requirements");
|
||||
self.check_failed(Error::UnmetRequirement)?;
|
||||
} else {
|
||||
info!("The hardware meets the requirements ");
|
||||
}
|
||||
// Check that the results were not created by a bad build profile.
|
||||
if let Err(err) = check_build_profile() {
|
||||
self.check_failed(Error::BadBuildProfile(err))?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Returns `Ok` if [`self.allow_fail`] is set and otherwise the error argument.
|
||||
fn check_failed(&self, e: Error) -> Result<()> {
|
||||
if !self.allow_fail {
|
||||
error!("Failing since --allow-fail is not set");
|
||||
Err(pezsc_cli::Error::Application(Box::new(e)))
|
||||
} else {
|
||||
warn!("Ignoring error since --allow-fail is set: {:?}", e);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// Validates the CLI arguments.
|
||||
fn validate_args(&self) -> Result<()> {
|
||||
if self.tolerance > 100.0 || self.tolerance < 0.0 {
|
||||
return Err("The --tolerance argument is out of range".into());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl BenchResult {
|
||||
/// Format [`Self`] as row that can be printed in a table.
|
||||
fn to_row(&self, req: &Requirement) -> Row {
|
||||
let passed = if self.passed { "✅ Pass" } else { "❌ Fail" };
|
||||
vec![
|
||||
req.metric.category().into(),
|
||||
req.metric.name().into(),
|
||||
format!("{}", self.score),
|
||||
format!("{}", req.minimum),
|
||||
format!("{} ({: >5.1?} %)", passed, self.rel_score * 100.0),
|
||||
]
|
||||
.into()
|
||||
}
|
||||
}
|
||||
|
||||
// Boilerplate
|
||||
impl CliConfiguration for MachineCmd {
|
||||
fn shared_params(&self) -> &SharedParams {
|
||||
&self.shared_params
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
[
|
||||
{
|
||||
"metric": "Blake2256",
|
||||
"minimum": 1000.00
|
||||
},
|
||||
{
|
||||
"metric": {"Blake2256Parallel":{"num_cores":8}},
|
||||
"minimum": 1000.00,
|
||||
"validator_only": true
|
||||
},
|
||||
{
|
||||
"metric": "Sr25519Verify",
|
||||
"minimum": 0.622675781
|
||||
},
|
||||
{
|
||||
"metric": "MemCopy",
|
||||
"minimum": 11768.341
|
||||
},
|
||||
{
|
||||
"metric": "DiskSeqWrite",
|
||||
"minimum": 950.0
|
||||
},
|
||||
{
|
||||
"metric": "DiskRndWrite",
|
||||
"minimum": 420.0
|
||||
}
|
||||
]
|
||||
Reference in New Issue
Block a user