pezkuwichain/pezkuwi-sdk
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

feat: Vendor pezkuwi-subxt and pezkuwi-zombienet-sdk into monorepo

Browse Source 
- Add pezkuwi-subxt crates to vendor/pezkuwi-subxt
- Add pezkuwi-zombienet-sdk crates to vendor/pezkuwi-zombienet-sdk
- Convert git dependencies to path dependencies
- Add vendor crates to workspace members
- Remove test/example crates from vendor (not needed for SDK)
- Fix feature propagation issues detected by zepter
- Fix workspace inheritance for internal dependencies
- All 606 crates now in workspace
- All 6919 internal dependency links verified correct
- No git dependencies remaining
This commit is contained in:
Kurdistan Tech Ministry
2025-12-22 23:31:24 +03:00
parent 4c8f281051
commit 70ddb6516f
386 changed files with 76759 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
vendor/pezkuwi-zombienet-sdk/.config/lingua.dic
Vendored
+50
View File
@@ -0,0 +1,50 @@
90
=
CLI
Deserialization
Deserialized
IFF
IPv4
JSON
NetworkNode
Ok
P2P
PjsResult
PoS
RPC
RUN_IN_CI
SDK
WASM
arg
args
chain_spec_command
cmd
declaratively
deserialize
deserialized
dir
env
fs
invulnerables
ip
js
k8s
msg
multiaddress
natively
ns
p2p
parachaing
pjs_rs
polkadot
polkadot_
rococo_local_testnet
rpc
serde_json
tgz
tmp
u128
u64
validator
ws
vendor/pezkuwi-zombienet-sdk/.config/spellcheck.toml
Vendored
+13
View File
@@ -0,0 +1,13 @@
[hunspell]
lang = "en_US"
search_dirs = ["."]
extra_dictionaries = ["lingua.dic"]
skip_os_lookups = true
use_builtin = true
[hunspell.quirks]
# `Type`'s
# 5x
transform_regex = ["^'([^\\s])'$", "^[0-9]+(?:\\.[0-9]*)?x$", "^'s$", "^\\+$", "[><+-]"]
allow_concatenation = true
allow_dashes = true
vendor/pezkuwi-zombienet-sdk/.dockerignore
Vendored
+5
View File
@@ -0,0 +1,5 @@
target
Dockerfile
.dockerignore
.git
.gitignore
vendor/pezkuwi-zombienet-sdk/.github/CODEOWNERS
Vendored
+2
View File
@@ -0,0 +1,2 @@
@pepoviola
@l0r1s
vendor/pezkuwi-zombienet-sdk/.github/ISSUE_TEMPLATE/bug_report.yaml
Vendored
+124
View File
@@ -0,0 +1,124 @@
name: Bug Report
description: File a bug report
labels: ["triage-needed"]
body:
- type: markdown
attributes:
value: |
Thanks for taking the time to fill out this bug report!
**NOTE** A number of issues reported against Zombienet are often found to already be fixed in more current versions of the project.
Before reporting an issue, please verify the version you are running with `zombienet version` and compare it to the latest release.
If they differ, please update your version of Zombienet to the latest possible and retry your command before creating an issue.
- type: textarea
id: description
attributes:
label: Issue Description
description: Please explain your issue
value: "Describe your issue"
validations:
required: true
- type: textarea
id: reproducer
attributes:
label: Steps to reproduce the issue
description: Please explain the steps to reproduce the issue, including configuration files needed.
value: "Steps to reproduce the issue\n1.\n2.\n3.\n"
validations:
required: true
- type: textarea
id: received_results
attributes:
label: Describe the results you received
description: Please explain the results you are noticing, including stacktrace and error logs.
value: "Describe the results you received"
validations:
required: true
- type: textarea
id: expected_results
attributes:
label: Describe the results you expected
description: Please explain the results you are expecting
value: "Describe the results you expected"
validations:
required: true
- type: input
id: zombienet_version
attributes:
label: Zombienet version
description: Which zombienet version are you using ?
validations:
required: true
- type: dropdown
id: provider
attributes:
label: Provider
description: Which provider are you using ?
options:
- Native
- Kubernetes
- Podman
validations:
required: true
- type: textarea
id: provider_version
attributes:
label: Provider version
description: Which provider version / binaries versions are you using ?
value: |
## For binaries
polkadot 0.9.40-a2b62fb872b
polkadot-parachain 0.9.380-fe24f39507f
## For Kubernetes/Podman
podman version 4.4.1
OR
kubectl version v0.26.3
cluster version 1.25.2
render: yaml
validations:
required: true
- type: dropdown
id: upstream_latest
attributes:
label: Upstream Latest Release
description: Have you tried running the [latest upstream release](https://github.com/paritytech/zombienet/releases/latest)
options:
- 'Yes'
- 'No'
validations:
required: true
- type: textarea
id: additional_environment
attributes:
label: Additional environment details
description: Please describe any additional environment details like (Cloud, Local, OS, Provider versions...)
value: "Additional environment details"
- type: textarea
id: additional_info
attributes:
label: Additional information
description: Please explain the additional information you deem important
value: "Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting"
validations:
required: false
- type: textarea
id: screenshots
attributes:
label: Screenshots
description: Provide us with screenshots if needed to have a better understanding of the issue
validations:
required: false
vendor/pezkuwi-zombienet-sdk/.github/ISSUE_TEMPLATE/feature_report.yaml
Vendored
+43
View File
@@ -0,0 +1,43 @@
name: Feature request
description: File a feature request
labels: ["triage-needed"]
body:
- type: markdown
attributes:
value: |
Thanks for taking the time to fill out this feature report!
Please make sure to describe your feature and the problem it would solve.
- type: textarea
id: description
attributes:
label: Is your feature request related to a problem? Please describe.
description: A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
value: "Describe the feature"
validations:
required: true
- type: textarea
id: solution
attributes:
label: Describe the solution you'd like
description: A clear and concise description of what you want to happen..
validations:
required: true
- type: textarea
id: alt_solution
attributes:
label: Describe alternatives you've considered
description: A clear and concise description of any alternative solutions or features you've considered.
validations:
required: false
- type: textarea
id: additional_context
attributes:
label: Additional context
description: Add any other context or screenshots about the feature request here.
validations:
required: false
vendor/pezkuwi-zombienet-sdk/.github/workflows/ci.yml
Vendored
+97
View File
@@ -0,0 +1,97 @@
name: Cargo Build & Test
on:
push:
branches: [main]
pull_request:
branches: [main]
env:
CARGO_TERM_COLOR: always
RUSTFLAGS: "-Dwarnings"
jobs:
build:
name: Zombienet SDK - latest
runs-on: ubuntu-latest
strategy:
matrix:
toolchain:
- stable
# TODO 24-02-08: Disable nightly due to tkaitchuck/aHash#200.
#- nightly
steps:
# https://github.com/jlumbroso/free-disk-space
- name: Free Disk Space (Ubuntu)
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- name: install_deps
run: sudo apt-get update && sudo apt-get install protobuf-compiler
- name: Init nigthly install for fmt
run: rustup update nightly && rustup default nightly && rustup component add rustfmt
- name: Check format
run: cargo +nightly fmt --check --all
- name: Init install
run: rustup update ${{ matrix.toolchain }} && rustup default ${{ matrix.toolchain }} && rustup component add clippy
- name: Fetch cache
uses: Swatinem/rust-cache@a95ba195448af2da9b00fb742d14ffaaf3c21f43 # v2.7.0
with:
shared-key: "zombie-cache"
- name: Clippy
# disable needless_lifetimes until we align the version with polakdot-sdk
run: cargo clippy --all-targets --all-features -- -A clippy::needless_lifetimes
- name: Build
run: cargo build
- name: Tests
run: cargo test --workspace -- --skip ci_k8s
# TODO: fix and re-enable
# coverage:
# name: Zombienet SDK - coverage
# needs: build
# runs-on: ubuntu-20.04
# if: github.event_name == 'pull_request'
# permissions:
# issues: write
# pull-requests: write
# steps:
# - uses: actions/checkout@v3
# # https://github.com/jlumbroso/free-disk-space
# - name: Free Disk Space (Ubuntu)
# uses: jlumbroso/free-disk-space@main
# with:
# tool-cache: false
# - name: Fetch cache
# uses: Swatinem/rust-cache@a95ba195448af2da9b00fb742d14ffaaf3c21f43 # v2.7.0
# with:
# shared-key: "zombie-cache"
# - name: Install latest nextest release
# uses: taiki-e/install-action@nextest
# - name: Install cargo-llvm-cov
# uses: taiki-e/install-action@cargo-llvm-cov
# - name: Collect coverage data
# run: cargo llvm-cov nextest --workspace --exclude zombienet-sdk --test-threads 1 --lcov --output-path lcov.info
# - name: Report code coverage
# uses: Nef10/lcov-reporter-action@v0.4.0
# with:
# lcov-file: lcov.info
# pr-number: ${{ github.event.pull_request.number }}
vendor/pezkuwi-zombienet-sdk/.github/workflows/ci_integration.yml
Vendored
+206
View File
@@ -0,0 +1,206 @@
name: Integration test
on:
push:
branches: [main]
pull_request:
branches: [main]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
env:
RUN_IN_CONTAINER: 1
FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR: 1
GHA_CLUSTER_SERVER_ADDR: "https://kubernetes.default:443"
CARGO_TERM_COLOR: always
RUSTFLAGS: "-Dwarnings"
BASE_IMAGE: docker.io/paritytech/ci-unified:bullseye-1.88.0-2025-06-27-v202506301118
RUN_IN_CI: "1"
RUST_LOG: "zombienet_orchestrator=debug,zombienet_provider=debug"
CARGO_TARGET_DIR: /tmp/target
jobs:
build-tests:
runs-on: ubuntu-latest
timeout-minutes: 60
container:
image: docker.io/paritytech/ci-unified:bullseye-1.88.0-2025-06-27-v202506301118
steps:
- name: Checkout
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: Swatinem/rust-cache@98c8021b550208e191a6a3145459bfc9fb29c4c0 # v2.8.0
with:
cache-on-failure: true
- name: Build tests
run: |
cargo build --tests --keep-going --locked
mkdir -p artifacts
cd artifacts
find /tmp/target/debug/deps/ -maxdepth 1 -name "smoke-*" ! -name "*.d" -exec mv {} $(pwd)/smoke \;
find /tmp/target/debug/deps/ -maxdepth 1 -name "smoke_native-*" ! -name "*.d" -exec mv {} $(pwd)/smoke_native \;
cd ..
tar cvfz artifacts.tar.gz artifacts
- name: Upload artifacts
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: zombienet-tests-${{ github.sha }}
path: artifacts.tar.gz
k8s-integration-test-smoke:
runs-on: ubuntu-latest
needs: build-tests
timeout-minutes: 60
container:
image: docker.io/paritytech/ci-unified:bullseye-1.88.0-2025-06-27-v202506301118
steps:
- name: Download artifacts
uses: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0 # v5.0.0
with:
name: zombienet-tests-${{ github.sha }}
path: /tmp
- name: script
timeout-minutes: 45
run: |
export ZOMBIE_K8S_CI_NAMESPACE=$(cat /data/namespace)
export ZOMBIE_PROVIDER="k8s"
cd /tmp
ls -la
tar xvfz artifacts.tar.gz
./artifacts/smoke --nocapture
- name: dump logs
if: always()
run: |
export ZOMBIE_K8S_CI_NAMESPACE=$(cat /data/namespace)
mkdir -p /tmp/zombie-1/logs
# Install kubectl if not available
if ! command -v kubectl &> /dev/null; then
echo "Installing kubectl..."
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
mv kubectl /usr/local/bin/
fi
echo "Listing pods in namespace $ZOMBIE_K8S_CI_NAMESPACE..."
kubectl get pods -n "$ZOMBIE_K8S_CI_NAMESPACE" -o wide || true
for pod in $(kubectl get pods -n "$ZOMBIE_K8S_CI_NAMESPACE" -o jsonpath='{.items[*].metadata.name}' 2>/dev/null || true); do
echo "Dumping logs for pod: $pod"
kubectl logs -n "$ZOMBIE_K8S_CI_NAMESPACE" "$pod" --all-containers=true > "/tmp/zombie-1/logs/${pod}.log" 2>&1 || true
done
find /tmp/zombie* -name "*.log" -type f ! -regex '.*/[0-9]+\.log' -exec cp {} /tmp/zombie-1/logs/ \; 2>/dev/null || true
echo "Collected logs:"
ls -la /tmp/zombie-1/logs/ || true
- name: upload logs
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: zombienet-logs-${{ github.job }}-${{ github.sha }}
path: |
/tmp/zombie-1/logs/*
docker-integration-test-smoke:
runs-on: ubuntu-latest
needs: build-tests
timeout-minutes: 60
steps:
- name: Download artifacts
uses: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0 # v5.0.0
with:
name: zombienet-tests-${{ github.sha }}
path: /tmp
- name: Install dependencies
run: |
sudo apt-get -y update
sudo apt-get -y install wget
# Manually download and install the OpenSSL 1.1 library
wget http://security.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2_amd64.deb
sudo dpkg -i libssl1.1_1.1.1f-1ubuntu2_amd64.deb
- name: script
timeout-minutes: 45
run: |
export ZOMBIE_PROVIDER="docker"
cd /tmp
ls -la
tar xvfz artifacts.tar.gz
./artifacts/smoke --nocapture
- name: dump logs
if: always()
run: |
mkdir -p /tmp/zombie-1/logs
for container in $(docker ps -a --filter "name=zombie" --format "{{.Names}}" 2>/dev/null || true); do
echo "Dumping logs for container: $container"
docker logs "$container" > "/tmp/zombie-1/logs/${container}.log" 2>&1 || true
done
find /tmp/zombie* -name "*.log" -type f ! -regex '.*/[0-9]+\.log' -exec cp {} /tmp/zombie-1/logs/ \; 2>/dev/null || true
ls -la /tmp/zombie-1/logs/ || true
- name: upload logs
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: zombienet-logs-${{ github.job }}-${{ github.sha }}
path: |
/tmp/zombie-1/logs/*
native-integration-test-smoke:
runs-on: ubuntu-latest
needs: build-tests
timeout-minutes: 60
container:
image: docker.io/paritytech/ci-unified:bullseye-1.88.0-2025-06-27-v202506301118
steps:
- name: Download artifacts
uses: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0 # v5.0.0
with:
name: zombienet-tests-${{ github.sha }}
path: /tmp
- name: Download bins
shell: bash
run: |
for bin in polkadot polkadot-execute-worker polkadot-prepare-worker polkadot-omni-node polkadot-parachain; do
echo "downloading $bin";
curl -L -o /tmp/$bin https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2503-1/$bin;
chmod 755 /tmp/$bin;
done
ls -ltr /tmp
export PATH=/tmp:$PATH
echo $PATH
- name: script
run: |
export PATH=/tmp:$PATH
echo $PATH
# mv artifacts.tar.gz /tmp
cd /tmp
ls -la
tar xvfz artifacts.tar.gz
export ZOMBIE_PROVIDER="native"
./artifacts/smoke_native --nocapture
# cargo test --test smoke-native -- --nocapture
- name: collect logs
if: always()
run: |
mkdir -p /tmp/zombie-1/logs
find /tmp/zombie* -name "*.log" -type f ! -regex '.*/[0-9]+\.log' -exec cp {} /tmp/zombie-1/logs/ \; 2>/dev/null || true
ls -la /tmp/zombie-1/logs/ || true
- name: upload logs
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: zombienet-logs-${{ github.job }}-${{ github.sha }}
path: |
/tmp/zombie-1/logs/*
vendor/pezkuwi-zombienet-sdk/.github/workflows/documentation.yml
Vendored
+62
View File
@@ -0,0 +1,62 @@
name: Cargo Create Docs
on:
push:
branches: [main]
pull_request:
branches: [main]
env:
CARGO_TERM_COLOR: always
RUSTFLAGS: "-Dwarnings"
jobs:
build-rust-doc:
name: Zombienet SDK - Rust Docs
runs-on: ubuntu-latest
strategy:
matrix:
toolchain:
# TODO 24-02-08: Disable nightly due to tkaitchuck/aHash#200.
#- nightly
- stable
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- name: Init nigthly install for fmt
run: rustup update nightly && rustup default nightly && rustup component add rustfmt
- name: Check format
run: cargo +nightly fmt --check --all
- name: Init install
run: rustup update ${{ matrix.toolchain }} && rustup default ${{ matrix.toolchain }} && rustup component add clippy
- name: install_deps
run: sudo apt-get update && sudo apt-get install protobuf-compiler
- name: Create docs
run: |
cargo doc --no-deps
echo "<meta http-equiv=\"refresh\" content=\"0; url=zombienet_sdk\">" > target/doc/index.html
- name: Move docs
run: |
mkdir -p ./doc
mv ./target/doc/* ./doc
git config user.email "github-action@users.noreply.github.com"
git config user.name "GitHub Action"
git config user.password "${{ secrets.GH_PAGES_TOKEN }}"
git checkout --orphan gh-pages
mkdir to_delete
shopt -s extglob
mv !(to_delete) ./to_delete
mv ./to_delete/doc/* .
rm -rf ./to_delete
git add --all
git commit -m "Documentation"
shell: bash # Necessary for `shopt` to work
- run: git push -f origin gh-pages:gh-pages
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
vendor/pezkuwi-zombienet-sdk/.github/workflows/fileserver.yml
Vendored
+52
View File
@@ -0,0 +1,52 @@
name: File server build & image publish
run-name: Deploy file server ${{ github.ref }}
on:
push:
branches:
- main
paths:
- "Cargo.toml"
- "crates/file-server/**"
workflow_dispatch: {}
env:
PROJECT_ID: "parity-zombienet"
GCR_REGISTRY: "europe-west3-docker.pkg.dev"
GCR_REPOSITORY: "zombienet-public-images"
jobs:
build_and_push:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- name: Setup gcloud CLI
uses: google-github-actions/setup-gcloud@aa5489c8933f4cc7a4f7d45035b3b1440c9c10db # v3.0.1
with:
service_account_key: ${{ secrets.GCP_SA_KEY }}
project_id: ${{ env.PROJECT_ID }}
export_default_credentials: true
- name: Login to GCP
uses: google-github-actions/auth@7c6bc770dae815cd3e89ee6cdf493a5fab2cc093 # v3.0.0
with:
credentials_json: ${{ secrets.GCP_SA_KEY }}
- name: Artifact registry authentication
run: |
gcloud auth configure-docker ${{ env.GCR_REGISTRY }}
- name: Build, tag, and push image to GCP Artifact registry
id: build-image
env:
IMAGE: "${{ env.GCR_REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.GCR_REPOSITORY }}/zombienet-file-server"
run: |
docker build -t $IMAGE:${{ github.sha }} -f ./crates/file-server/Dockerfile .
docker tag $IMAGE:${{ github.sha }} $IMAGE:latest
docker push --all-tags $IMAGE
echo "image=$IMAGE:${{ github.sha }}" >> $GITHUB_OUTPUT
echo "image=$IMAGE:latest" >> $GITHUB_OUTPUT
vendor/pezkuwi-zombienet-sdk/.github/workflows/publish.yml
Vendored
+26
View File
@@ -0,0 +1,26 @@
name: Publish to crates.io
on:
release:
types: [published]
jobs:
check-publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- name: Rust Cache
uses: Swatinem/rust-cache@a95ba195448af2da9b00fb742d14ffaaf3c21f43 # v2.7.0
with:
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: install parity-publish
run: cargo install parity-publish@0.10.6 --locked -q
- name: parity-publish check
run: parity-publish --color always check --allow-unpublished
# TODO: remove dry-run once we confirm everything works as expected
- name: parity-publish dry-run
run: parity-publish --color always apply --dry-run
vendor/pezkuwi-zombienet-sdk/.github/workflows/release.yml
Vendored
+67
View File
@@ -0,0 +1,67 @@
name: Release bin for zombie-cli
on:
push:
tags:
- 'v*.*.*'
jobs:
build:
strategy:
matrix:
include:
- os: ubuntu-latest
target: x86_64-unknown-linux-gnu
- os: macos-latest
target: aarch64-apple-darwin
runs-on: ${{ matrix.os }}
steps:
- name: Checkout repository
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # 5.0.0
- name: Install Rust
uses: dtolnay/rust-toolchain@stable
with:
targets: ${{ matrix.target }}
- name: Build release binary
run: cargo build --release --bin zombie-cli --target ${{ matrix.target }}
- name: Package binary
run: |
mkdir -p dist
cp target/${{ matrix.target }}/release/zombie-cli dist/zombie-cli-${{ matrix.target }}
- name: Upload artifact
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: zombie-cli-${{ matrix.os }}-${{ matrix.target }}-${{ github.ref_name }}
path: dist/*
release:
runs-on: ubuntu-latest
needs: build
steps:
- name: Download artifacts
uses: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0 # v5.0.0
with:
path: artifacts
- name: Generate checksums
run: |
cd artifacts
sha256sum * > checksums.txt
- name: Create Release
uses: softprops/action-gh-release@6cbd405e2c4e67a21c47fa9e383d020e4e28b836 # v2.3.3
with:
files: |
artifacts/**/*
artifacts/checksums.txt
generate_release_notes: true
draft: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
vendor/pezkuwi-zombienet-sdk/.github/workflows/spellcheck.yml
Vendored
+31
View File
@@ -0,0 +1,31 @@
name: Spellcheck
on:
push:
branches:
- main
pull_request:
branches:
- main
jobs:
spellcheck:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- name: Install Rust
uses: dtolnay/rust-toolchain@e97e2d8cc328f1b50210efc529dca0028893a2d9 # v1.0.0
with:
toolchain: stable
- name: Install cargo-spellcheck
run: |
sudo apt-get install libclang-dev
export LIBCLANG_PATH=/usr/lib/llvm-18/lib/
cargo install cargo-spellcheck
- name: Run cargo-spellcheck
run: cargo spellcheck
vendor/pezkuwi-zombienet-sdk/.gitignore
+26
View File
@@ -0,0 +1,26 @@
# Generated by Cargo
# will have compiled files and executables
debug/
target/
# These are backup files generated by rustfmt
**/*.rs.bk
# MSVC Windows builds of rustc generate these, which store debugging information
*.pdb
node_modules
dist
log.md
.env
bins
.DS_Store
**/target/
*.swp
.vscode
# nix
result
# docs
docs
vendor/pezkuwi-zombienet-sdk/Cargo.toml
Vendored
+5
View File
@@ -0,0 +1,5 @@
# This is a virtual manifest for the vendored pezkuwi-zombienet-sdk crates
# Individual crates are managed by the main pezkuwi-sdk workspace
[workspace]
# Empty workspace - crates are part of parent workspace
vendor/pezkuwi-zombienet-sdk/LICENSE
Vendored
+674
View File
@@ -0,0 +1,674 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
<program> Copyright (C) <year> <name of author>
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<https://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<https://www.gnu.org/licenses/why-not-lgpl.html>.
vendor/pezkuwi-zombienet-sdk/README.md
Vendored
+84
View File
@@ -0,0 +1,84 @@
# 🚧⚠️ [WIP] ZombieNet SDK ⚠️🚧
[Rust Docs](https://paritytech.github.io/zombienet-sdk)
# The Vision
This issue will track the progress of the new ZombieNet SDK.
We want to create a new SDK for `ZombieNet` that allow users to build more complex use cases and interact with the network in a more flexible and programatic way.
The SDK will provide a set of `building blocks` that users can combine in order to spawn and interact (test/query/etc) with the network providing a *fluent* api to craft different topologies and assertions to the running network. The new `SDK` will support the same range of `providers` and configurations that can be created in the current version (v1).
We also want to continue supporting the `CLI` interface *but* should be updated to use the `SDK` under the hood.
# The Plan
We plan to divide the work phases to. ensure we cover all the requirement and inside each phase in small tasks, covering one of the building blocks and the interaction between them.
## Prototype building blocks
Prototype each building block with a clear interface and how to interact with it
- [Building block Network #2](https://github.com/paritytech/zombienet-sdk/issues/2)
- [Building block Node #3](https://github.com/paritytech/zombienet-sdk/issues/3)
- [Building block NodeGroup #4](https://github.com/paritytech/zombienet-sdk/issues/4)
- [Building block Parachain #5](https://github.com/paritytech/zombienet-sdk/issues/5)
- [Building block Collator #6](https://github.com/paritytech/zombienet-sdk/issues/6)
- [Building block CollatorGroup #7](https://github.com/paritytech/zombienet-sdk/issues/7)
- [Building block Assertion #8](https://github.com/paritytech/zombienet-sdk/issues/8)
## Integrate, test interactions and document
We want to integrate the interactions for all building blocks and document the way that they work together.
- [Spawning Integration #9](https://github.com/paritytech/zombienet-sdk/issues/9)
- [Assertion Integration #10](https://github.com/paritytech/zombienet-sdk/issues/10)
- [Documentation #11](https://github.com/paritytech/zombienet-sdk/issues/11)
## Refactor `CLI` and ensure backwards compatibility
Refactor the `CLI` module to use the new `SDK` under the hood.
- [Refactor CLI #12](https://github.com/paritytech/zombienet-sdk/issues/12)
- [Ensure that spawning from toml works #13](https://github.com/paritytech/zombienet-sdk/issues/13)
- [Ensure that test-runner from DSL works #14](https://github.com/paritytech/zombienet-sdk/issues/14)
## ROADMAP
## Infra
- Chaos testing, add examples and explore possibilities in `native` and `podman` provider
- Add `docker` provider
- Add `nomad` provider
- Create [helm chart](https://helm.sh/docs/topics/charts/) to allow other use zombienet in k8s
- Auth system to not use k8s users
- Create GitHub Action and publish in NPM marketplace (Completed)
- Rename `@paritytech/zombienet` npm package to `zombienet`. Keep all zombienet modules under `@zombienet/*` org (Completed)
## Internal teams
- Add more teams (wip)
## Registry
- Create decorators registry and allow override by paras (wip)
- Explore how to get info from paras.
## Functional tasks
- Add subxt integration, allow to compile/run on the fly
- Move parser to pest (wip)
- Detach phases and use JSON to communicate instead of `paths`
- Add relative values assertions (for metrics/scripts)
- Allow to define nodes that are not started in the launching phase and can be started by the test-runner
- Allow to define `race` assertions
- Rust integration -> Create multiples libs (crates)
- Explore backchannel use case
- Add support to run test agains a running network (wip)
- Add more CLI subcommands
- Add js/subxt snippets ready to use in assertions (e.g transfers)
- Add XCM support in built-in assertions
- Add `ink! smart contract` support
- Add support to start from a live network (fork-off) [check subalfred]
- Create "default configuration" - (if `zombieconfig.json` exists in same dir with zombienet then the config applied in it will override the default configuration of zombienet. E.G if user wants to have as default `native` instead of `k8s` he can add to
## UI
- Create UI to create `.zndls` and `network` files.
- Improve VSCode extension (grammar/snippets/syntax highlighting/file validations) ([repo](https://github.com/paritytech/zombienet-vscode-extension))
- Create UI app (desktop) to run zombienet without the need of terminal.
vendor/pezkuwi-zombienet-sdk/adrs/001-node-rust-foreign-function-interface/001-node-to-rust-foreign-function-interface.md
Vendored
+112
View File
@@ -0,0 +1,112 @@
# Mechanism to call Rust code from Javascript/Typescript
### Status: proposed | rejected | **accepted** | deprecated
### Deciders: [@pepoviola](https://github.com/pepoviola) [@wirednkod](https://github.com/wirednkod) [@l0r1s](https://github.com/l0r1s)
### Creation date: 18/05/2023
### Update date: -
---
## Context and Problem Statement
The `zombienet-sdk` will be developed in Rust. Our objective is make it easily integrable into existing Typescript/Javascript project. To achieve this goal, we need to find a way to call the Rust code from a Javascript/Typescript program.
Many mechanisms exists for this purpose like Wasm or N(ode)-API but some may or may not fit our use case, for example, executing async code.
---
## Decision drivers
- We can use the standard library (for filesystem or networking in providers).
- We can execute asynchronous code: our goal is not to make the program fully sequential as many operations (e.g: bootstrapping the relaychain nodes) can be done concurrently.
- Easy to package and deploy
---
## Considered Options
- #### WASM
- [wasm-pack](https://github.com/neon-bindings/neon)
- #### Native node modules (Node-API / V8 / libuv)
- [napi-rs](https://github.com/napi-rs/napi-rs)
---
## Prototyping
To demonstrate and learn which options fit the best for our use case, we will create a small test program which will have the following functionalities:
- Has a function taking an arbitratry object and a callback as parameters in the Typescript code, calling the callback with the function result on Rust side.
- Has a function taking an arbitrary object as parameter and a returning a promise in Typescript, signaling an asynchronous operation on Rust side.
- Make an HTTP request asynchronously in the Rust code, using a dependency using the standard library.
The prototype assume versions of `rustc` and `cargo` to be `1.69.0`, use of `stable` channel and `Linux` on `amd64` architecture.
- ### [Boilerplate app to execute prototype](boilerplate-app-prototype.md)
- ### [Wasm-pack prototype](wasm-prototype.md)
- ### [Napi-rs prototype](napi-prototype.md)
---
## Pros and cons of each options
- ### Napi-rs
- Pros 👍
- Support many types correctly including typed callback, typed array, class and all JS primitives types (Null, Undefined, Numbers, String, BigInt, ...)
- Support top level async function because it detects if it needs to be run inside an async runtime (tokio by default)
- Standard library can be used without limitations, including threading, networking, etc...
- Extremely well documented with examples
- Provide full Github action pipeline template to compile on all architecture easily
- Support complex use cases
- Used by many big names (Prisma, Parcel, Tailwind, Next.js, Bitwarden)
- Cons 👎
- Node-API is not simple for complex use case
- Bound to NodeJS, if we want to expose the same logic to others languages (Go, C++, Python, ...) we need to wrap the Rust code inside a dynamic library and adapt to others languages primitives by creating a small adapter over the library
- Not universally compiled
- ### Wasm-pack
- Pros 👍
- Rich ecosystem and developing fast
- Used in many places across web, backend (Docker supports WASM)
- Easy to use and distribute
- Universally compiled and used across languages (if they support WASM execution)
- Good for simple use case where you do pure function (taking input, returning output, without side effects like writing to filesystem or making networking calls)
- Cons 👎
- Limited in the use of the standard library, can't access networking/filesystem primitives without having to use WASI which is inconsistent across languages/runtimes
- Only support 32 bits
- No support for concurrent programming (async/threads), even if we can returns Promise from WASM exposed functions but could see the light in few months (maybe?)
- wasm-bindgen types are too generic, for example, we return a JsValue but we would like to be more specific for the type
## Decision outcome
- ### **Napi-rs** for crates dependant on async, filesystem or networking: *support*, *orchestrator*, *test-runner*, *providers* from [schema](https://github.com/paritytech/zombienet-sdk/issues/22)
- ### **Wasm-pack** for the rest of the crates: *configuration* from [schema](https://github.com/paritytech/zombienet-sdk/issues/22)
vendor/pezkuwi-zombienet-sdk/adrs/001-node-rust-foreign-function-interface/boilerplate-app-prototype.md
Vendored
+32
View File
@@ -0,0 +1,32 @@
## [Back](001-node-to-rust-foreign-function-interface.md)
## Boilerplate app to execute prototypes
1. Create the new node app :
```bash
$ mkdir -p ffi-prototype/app && cd ffi-prototype/app && npm init -y
```
2. Install required packages :
```bash
[ffi-prototype/app]$ npm i -D @tsconfig/recommended ts-node typescript
```
3. Add a new script :
```json
{
"scripts": {
"build+exec": "tsc && node ./index.js"
}
}
```
4. Add tsconfig.json
```json
{
"extends": "@tsconfig/recommended/tsconfig.json"
}
```
vendor/pezkuwi-zombienet-sdk/adrs/001-node-rust-foreign-function-interface/napi-prototype.md
Vendored
+142
View File
@@ -0,0 +1,142 @@
## [Back](001-node-to-rust-foreign-function-interface.md)
## Napi-rs prototype
___
1. Install the napi CLI
```bash
[ffi-prototype]$ npm install -g @napi-rs/cli
```
2. Create a new napi project
```bash
[ffi-prototype]$ napi new napi-prototype
```
3. Install cargo dependencies
```bash
[ffi-prototype/napi-prototype]$ cargo add tokio --features full
[ffi-prototype/napi-prototype]$ cargo add reqwest --features blocking
[ffi-prototype/napi-prototype]$ cargo add napi --no-default-features --features napi4,async
```
4. Copy the following code to `napi-prototype/src/lib.rs`
```rust
#![deny(clippy::all)]
use std::thread;
use napi::{
bindgen_prelude::*,
threadsafe_function::{
ErrorStrategy, ThreadSafeCallContext, ThreadsafeFunction, ThreadsafeFunctionCallMode,
},
};
use reqwest;
#[macro_use]
extern crate napi_derive;
// native async with tokio is supported without annotating a main function
#[napi]
pub async fn fetch_promise() -> Result<String> {
let body = reqwest::get("https://paritytech.github.io/zombienet/")
.await
.map_err(|_| napi::Error::from_reason("Error while fetching page"))?
.text()
.await
.map_err(|_| napi::Error::from_reason("Error while extracting body"))?;
Ok(body)
}
#[napi]
pub fn fetch_callback(callback: JsFunction) -> Result<()> {
// createa thread safe callback from the JsFunction
let thread_safe_callback: ThreadsafeFunction<String, ErrorStrategy::CalleeHandled> = callback
.create_threadsafe_function(0, |ctx: ThreadSafeCallContext<String>| {
ctx.env.create_string(&ctx.value).map(|s| vec![s])
})?;
// spawn a thread to execute our logic
thread::spawn(move || {
let response = reqwest::blocking::get("https://paritytech.github.io/zombienet/");
if response.is_err() {
let response = response
.map(|_| "".into())
.map_err(|_| napi::Error::from_reason("Error while fetching page"));
// error are returned by calling the callback with an empty response and the error mapped
return thread_safe_callback.call(response, ThreadsafeFunctionCallMode::Blocking);
}
let body = response.unwrap().text();
if body.is_err() {
let body = body
.map(|_| "".into())
.map_err(|_| napi::Error::from_reason("Error while extracting body"));
return thread_safe_callback.call(body, ThreadsafeFunctionCallMode::Blocking);
}
// result is returned as a string
thread_safe_callback.call(Ok(body.unwrap()), ThreadsafeFunctionCallMode::Blocking)
});
Ok(())
}
```
5. Build the project :
```bash
[ffi-prototype/napi-prototype]$ npm run build
```
6. Copy artifacts :
```bash
[ffi-prorotype/napi-prototype]$ mv napi-prototype.linux-x64-gnu.node index.d.ts index.js npm/linux-x64-gnu
```
7. Install package in ```ffi-prototype/app``` :
```bash
[ffi-prototype/app]$ npm i ../napi-prototype/npm/linux-x64-gnu/
```
8. Copy the following code to the ```ffi-prototype/app/index.ts``` file :
```ts
import { fetchCallback, fetchPromise } from "napi-prototype-linux-x64-gnu";
(async () => {
fetchCallback((_err: any, result: string) => {
console.log(`HTTP request through FFI with callback: ${result.length}`);
});
console.log(
`HTTP request through FFI with promise ${(await fetchPromise()).length}`
);
})();
```
9. Build and execute the app :
```bash
[ffi-prototype/app]$ npm run build+exec
```
Expected output:
```tty
> app@1.0.0 build+exec
> tsc && node ./index.js
HTTP request through FFI with promise 12057
HTTP request through FFI with callback: 12057
```
That's it !
vendor/pezkuwi-zombienet-sdk/adrs/001-node-rust-foreign-function-interface/wasm-prototype.md
Vendored
+153
View File
@@ -0,0 +1,153 @@
## [Back](001-node-to-rust-foreign-function-interface.md)
## Wasm-pack prototype
___
1. Install the wasm-pack CLI
```bash
curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
```
2. Create a new wasm-pack project
```bash
[ffi-prototype]$ wasm-pack new wasm-prototype
```
3. Install cargo dependencies
```bash
[ffi-prototype/wasm-prototype]$ cargo add tokio --features full
[ffi-prototype/wasm-prototype]$ cargo add reqwest --features blocking
[ffi-prototype/wasm-prototype]$ cargo add wasm-bindgen-futures
cargo add js-sys
```
4. Copy the following code to `wasm-prototype/src/lib.rs`
```rust
mod utils;
use wasm_bindgen::prelude::*;
// When the `wee_alloc` feature is enabled, use `wee_alloc` as the global
// allocator.
#[cfg(feature = "wee_alloc")]
#[global_allocator]
static ALLOC: wee_alloc::WeeAlloc = wee_alloc::WeeAlloc::INIT;
#[wasm_bindgen]
pub async fn fetch_promise() -> Result<String, JsError> {
let body = reqwest::get("https://paritytech.github.io/zombienet/")
.await
.map_err(|_| JsError::new("Error while fetching page"))?
.text()
.await
.map_err(|_| JsError::new("Error while extracting body"))?;
Ok(body)
}
#[wasm_bindgen]
pub fn fetch_callback(callback: &js_sys::Function) -> Result<JsValue, JsValue> {
let this = JsValue::null();
let response = reqwest::blocking::get("https://paritytech.github.io/zombienet/");
if response.is_err() {
return callback.call2(
&this,
&JsError::new("Error while fetching page").into(),
&JsValue::null(),
);
}
let body = response.unwrap().text();
if body.is_err() {
return callback.call2(
&this,
&JsError::new("Error while extracting body").into(),
&JsValue::null(),
);
}
Ok(body.unwrap().into())
}
```
5. Build the project :
```bash
[ffi-prototype/wasm-prototype]$ wasm-pack build -t nodejs
```
Error are shown, this is expected because WASM doesn't support networking primitives,
as you can see, we removed the thread call from the fetch_callback function because ```JsValue```
is using *const u8 under the hood and it's not ```Send``` so can't be passed safely across thread:
```bash
[INFO]: 🎯 Checking for the Wasm target...
[INFO]: 🌀 Compiling to Wasm...
Compiling mio v0.8.6
Compiling parking_lot v0.12.1
Compiling serde_json v1.0.96
Compiling url v2.3.1
error[E0432]: unresolved import `crate::sys::IoSourceState`
--> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/io_source.rs:12:5
|
12 | use crate::sys::IoSourceState;
| ^^^^^^^^^^^^^^^^^^^^^^^^^ no `IoSourceState` in `sys`
error[E0432]: unresolved import `crate::sys::tcp`
--> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/net/tcp/listener.rs:15:17
|
15 | use crate::sys::tcp::{bind, listen, new_for_addr};
| ^^^ could not find `tcp` in `sys`
error[E0432]: unresolved import `crate::sys::tcp`
--> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/net/tcp/stream.rs:13:17
|
13 | use crate::sys::tcp::{connect, new_for_addr};
| ^^^ could not find `tcp` in `sys`
error[E0433]: failed to resolve: could not find `Selector` in `sys`
--> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/poll.rs:301:18
|
301 | sys::Selector::new().map(|selector| Poll {
| ^^^^^^^^ could not find `Selector` in `sys`
error[E0433]: failed to resolve: could not find `event` in `sys`
--> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/event/event.rs:24:14
|
24 | sys::event::token(&self.inner)
| ^^^^^ could not find `event` in `sys`
error[E0433]: failed to resolve: could not find `event` in `sys`
--> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/event/event.rs:38:14
|
38 | sys::event::is_readable(&self.inner)
| ^^^^^ could not find `event` in `sys`
error[E0433]: failed to resolve: could not find `event` in `sys`
--> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/event/event.rs:43:14
|
43 | sys::event::is_writable(&self.inner)
| ^^^^^ could not find `event` in `sys`
error[E0433]: failed to resolve: could not find `event` in `sys`
--> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/event/event.rs:68:14
|
68 | sys::event::is_error(&self.inner)
| ^^^^^ could not find `event` in `sys`
error[E0433]: failed to resolve: could not find `event` in `sys`
--> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/event/event.rs:99:14
|
99 | sys::event::is_read_closed(&self.inner)
| ^^^^^ could not find `event` in `sys`
error[E0433]: failed to resolve: could not find `event` in `sys`
--> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.6/src/event/event.rs:129:14
|
129 | sys::event::is_write_closed(&self.inner)
| ^^^^^ could not find `event` in `sys`
```
vendor/pezkuwi-zombienet-sdk/crates/configuration/.gitignore
+2
View File
@@ -0,0 +1,2 @@
/target
/Cargo.lock
vendor/pezkuwi-zombienet-sdk/crates/configuration/Cargo.toml
Vendored
+29
View File
@@ -0,0 +1,29 @@
[package]
name = "zombienet-configuration"
version.workspace = true
authors.workspace = true
edition.workspace = true
rust-version.workspace = true
publish = true
license.workspace = true
repository.workspace = true
description = "Zombienet sdk config builder, allow to build a network configuration"
keywords = ["zombienet", "configuration", "sdk"]
[dependencies]
regex = { workspace = true }
lazy_static = { workspace = true }
multiaddr = { workspace = true }
url = { workspace = true, features = ["serde"] }
thiserror = { workspace = true }
anyhow = { workspace = true }
serde = { workspace = true, features = ["derive"] }
toml = { workspace = true }
serde_json = { workspace = true }
reqwest = { workspace = true }
tokio = { workspace = true, features = ["fs"] }
tracing = { workspace = true }
# zombienet deps
support = { workspace = true }
vendor/pezkuwi-zombienet-sdk/crates/configuration/src/global_settings.rs
Vendored
+383
View File
@@ -0,0 +1,383 @@
use std::{
error::Error,
fmt::Display,
net::IpAddr,
path::{Path, PathBuf},
str::FromStr,
};
use multiaddr::Multiaddr;
use serde::{Deserialize, Serialize};
use crate::{
shared::{
errors::{ConfigError, FieldError},
helpers::{merge_errors, merge_errors_vecs},
types::Duration,
},
utils::{default_as_true, default_node_spawn_timeout, default_timeout},
};
/// Global settings applied to an entire network.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct GlobalSettings {
/// Global bootnodes to use (we will then add more)
#[serde(skip_serializing_if = "std::vec::Vec::is_empty", default)]
bootnodes_addresses: Vec<Multiaddr>,
// TODO: parse both case in zombienet node version to avoid renamed ?
/// Global spawn timeout
#[serde(rename = "timeout", default = "default_timeout")]
network_spawn_timeout: Duration,
// TODO: not used yet
/// Node spawn timeout
#[serde(default = "default_node_spawn_timeout")]
node_spawn_timeout: Duration,
// TODO: not used yet
/// Local ip to use for construct the direct links
local_ip: Option<IpAddr>,
/// Directory to use as base dir
/// Used to reuse the same files (database) from a previous run,
/// also note that we will override the content of some of those files.
base_dir: Option<PathBuf>,
/// Number of concurrent spawning process to launch, None means try to spawn all at the same time.
spawn_concurrency: Option<usize>,
/// If enabled, will launch a task to monitor nodes' liveness and tear down the network if there are any.
#[serde(default = "default_as_true")]
tear_down_on_failure: bool,
}
impl GlobalSettings {
/// External bootnode address.
pub fn bootnodes_addresses(&self) -> Vec<&Multiaddr> {
self.bootnodes_addresses.iter().collect()
}
/// Global spawn timeout in seconds.
pub fn network_spawn_timeout(&self) -> Duration {
self.network_spawn_timeout
}
/// Individual node spawn timeout in seconds.
pub fn node_spawn_timeout(&self) -> Duration {
self.node_spawn_timeout
}
/// Local IP used to expose local services (including RPC, metrics and monitoring).
pub fn local_ip(&self) -> Option<&IpAddr> {
self.local_ip.as_ref()
}
/// Base directory to use (instead a random tmp one)
/// All the artifacts will be created in this directory.
pub fn base_dir(&self) -> Option<&Path> {
self.base_dir.as_deref()
}
/// Number of concurrent spawning process to launch
pub fn spawn_concurrency(&self) -> Option<usize> {
self.spawn_concurrency
}
/// A flag to tear down the network if there are any unresponsive nodes detected.
pub fn tear_down_on_failure(&self) -> bool {
self.tear_down_on_failure
}
}
impl Default for GlobalSettings {
fn default() -> Self {
Self {
bootnodes_addresses: Default::default(),
network_spawn_timeout: default_timeout(),
node_spawn_timeout: default_node_spawn_timeout(),
local_ip: Default::default(),
base_dir: Default::default(),
spawn_concurrency: Default::default(),
tear_down_on_failure: true,
}
}
}
/// A global settings builder, used to build [`GlobalSettings`] declaratively with fields validation.
#[derive(Default)]
pub struct GlobalSettingsBuilder {
config: GlobalSettings,
errors: Vec<anyhow::Error>,
}
impl GlobalSettingsBuilder {
pub fn new() -> Self {
Self::default()
}
// Transition to the next state of the builder.
fn transition(config: GlobalSettings, errors: Vec<anyhow::Error>) -> Self {
Self { config, errors }
}
/// Set the external bootnode address.
///
/// Note: Bootnode address replacements are NOT supported here.
/// Only arguments (`args`) support dynamic replacements. Bootnode addresses must be a valid address.
pub fn with_raw_bootnodes_addresses<T>(self, bootnodes_addresses: Vec<T>) -> Self
where
T: TryInto<Multiaddr> + Display + Copy,
T::Error: Error + Send + Sync + 'static,
{
let mut addrs = vec![];
let mut errors = vec![];
for (index, addr) in bootnodes_addresses.into_iter().enumerate() {
match addr.try_into() {
Ok(addr) => addrs.push(addr),
Err(error) => errors.push(
FieldError::BootnodesAddress(index, addr.to_string(), error.into()).into(),
),
}
}
Self::transition(
GlobalSettings {
bootnodes_addresses: addrs,
..self.config
},
merge_errors_vecs(self.errors, errors),
)
}
/// Set global spawn timeout in seconds.
pub fn with_network_spawn_timeout(self, timeout: Duration) -> Self {
Self::transition(
GlobalSettings {
network_spawn_timeout: timeout,
..self.config
},
self.errors,
)
}
/// Set individual node spawn timeout in seconds.
pub fn with_node_spawn_timeout(self, timeout: Duration) -> Self {
Self::transition(
GlobalSettings {
node_spawn_timeout: timeout,
..self.config
},
self.errors,
)
}
/// Set local IP used to expose local services (including RPC, metrics and monitoring).
pub fn with_local_ip(self, local_ip: &str) -> Self {
match IpAddr::from_str(local_ip) {
Ok(local_ip) => Self::transition(
GlobalSettings {
local_ip: Some(local_ip),
..self.config
},
self.errors,
),
Err(error) => Self::transition(
self.config,
merge_errors(self.errors, FieldError::LocalIp(error.into()).into()),
),
}
}
/// Set the directory to use as base (instead of a random tmp one).
pub fn with_base_dir(self, base_dir: impl Into<PathBuf>) -> Self {
Self::transition(
GlobalSettings {
base_dir: Some(base_dir.into()),
..self.config
},
self.errors,
)
}
/// Set the spawn concurrency
pub fn with_spawn_concurrency(self, spawn_concurrency: usize) -> Self {
Self::transition(
GlobalSettings {
spawn_concurrency: Some(spawn_concurrency),
..self.config
},
self.errors,
)
}
/// Set the `tear_down_on_failure` flag
pub fn with_tear_down_on_failure(self, tear_down_on_failure: bool) -> Self {
Self::transition(
GlobalSettings {
tear_down_on_failure,
..self.config
},
self.errors,
)
}
/// Seals the builder and returns a [`GlobalSettings`] if there are no validation errors, else returns errors.
pub fn build(self) -> Result<GlobalSettings, Vec<anyhow::Error>> {
if !self.errors.is_empty() {
return Err(self
.errors
.into_iter()
.map(|error| ConfigError::GlobalSettings(error).into())
.collect::<Vec<_>>());
}
Ok(self.config)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn global_settings_config_builder_should_succeeds_and_returns_a_global_settings_config() {
let global_settings_config = GlobalSettingsBuilder::new()
.with_raw_bootnodes_addresses(vec![
"/ip4/10.41.122.55/tcp/45421",
"/ip4/51.144.222.10/tcp/2333",
])
.with_network_spawn_timeout(600)
.with_node_spawn_timeout(120)
.with_local_ip("10.0.0.1")
.with_base_dir("/home/nonroot/mynetwork")
.with_spawn_concurrency(5)
.with_tear_down_on_failure(true)
.build()
.unwrap();
let bootnodes_addresses: Vec<Multiaddr> = vec![
"/ip4/10.41.122.55/tcp/45421".try_into().unwrap(),
"/ip4/51.144.222.10/tcp/2333".try_into().unwrap(),
];
assert_eq!(
global_settings_config.bootnodes_addresses(),
bootnodes_addresses.iter().collect::<Vec<_>>()
);
assert_eq!(global_settings_config.network_spawn_timeout(), 600);
assert_eq!(global_settings_config.node_spawn_timeout(), 120);
assert_eq!(
global_settings_config
.local_ip()
.unwrap()
.to_string()
.as_str(),
"10.0.0.1"
);
assert_eq!(
global_settings_config.base_dir().unwrap(),
Path::new("/home/nonroot/mynetwork")
);
assert_eq!(global_settings_config.spawn_concurrency().unwrap(), 5);
assert!(global_settings_config.tear_down_on_failure());
}
#[test]
fn global_settings_config_builder_should_succeeds_when_node_spawn_timeout_is_missing() {
let global_settings_config = GlobalSettingsBuilder::new()
.with_raw_bootnodes_addresses(vec![
"/ip4/10.41.122.55/tcp/45421",
"/ip4/51.144.222.10/tcp/2333",
])
.with_network_spawn_timeout(600)
.with_local_ip("10.0.0.1")
.build()
.unwrap();
let bootnodes_addresses: Vec<Multiaddr> = vec![
"/ip4/10.41.122.55/tcp/45421".try_into().unwrap(),
"/ip4/51.144.222.10/tcp/2333".try_into().unwrap(),
];
assert_eq!(
global_settings_config.bootnodes_addresses(),
bootnodes_addresses.iter().collect::<Vec<_>>()
);
assert_eq!(global_settings_config.network_spawn_timeout(), 600);
assert_eq!(global_settings_config.node_spawn_timeout(), 600);
assert_eq!(
global_settings_config
.local_ip()
.unwrap()
.to_string()
.as_str(),
"10.0.0.1"
);
}
#[test]
fn global_settings_builder_should_fails_and_returns_an_error_if_one_bootnode_address_is_invalid(
) {
let errors = GlobalSettingsBuilder::new()
.with_raw_bootnodes_addresses(vec!["/ip4//tcp/45421"])
.build()
.unwrap_err();
assert_eq!(errors.len(), 1);
assert_eq!(
errors.first().unwrap().to_string(),
"global_settings.bootnodes_addresses[0]: '/ip4//tcp/45421' failed to parse: invalid IPv4 address syntax"
);
}
#[test]
fn global_settings_builder_should_fails_and_returns_multiple_errors_if_multiple_bootnodes_addresses_are_invalid(
) {
let errors = GlobalSettingsBuilder::new()
.with_raw_bootnodes_addresses(vec!["/ip4//tcp/45421", "//10.42.153.10/tcp/43111"])
.build()
.unwrap_err();
assert_eq!(errors.len(), 2);
assert_eq!(
errors.first().unwrap().to_string(),
"global_settings.bootnodes_addresses[0]: '/ip4//tcp/45421' failed to parse: invalid IPv4 address syntax"
);
assert_eq!(
errors.get(1).unwrap().to_string(),
"global_settings.bootnodes_addresses[1]: '//10.42.153.10/tcp/43111' unknown protocol string: "
);
}
#[test]
fn global_settings_builder_should_fails_and_returns_an_error_if_local_ip_is_invalid() {
let errors = GlobalSettingsBuilder::new()
.with_local_ip("invalid")
.build()
.unwrap_err();
assert_eq!(errors.len(), 1);
assert_eq!(
errors.first().unwrap().to_string(),
"global_settings.local_ip: invalid IP address syntax"
);
}
#[test]
fn global_settings_builder_should_fails_and_returns_multiple_errors_if_multiple_fields_are_invalid(
) {
let errors = GlobalSettingsBuilder::new()
.with_raw_bootnodes_addresses(vec!["/ip4//tcp/45421", "//10.42.153.10/tcp/43111"])
.with_local_ip("invalid")
.build()
.unwrap_err();
assert_eq!(errors.len(), 3);
assert_eq!(
errors.first().unwrap().to_string(),
"global_settings.bootnodes_addresses[0]: '/ip4//tcp/45421' failed to parse: invalid IPv4 address syntax"
);
assert_eq!(
errors.get(1).unwrap().to_string(),
"global_settings.bootnodes_addresses[1]: '//10.42.153.10/tcp/43111' unknown protocol string: "
);
assert_eq!(
errors.get(2).unwrap().to_string(),
"global_settings.local_ip: invalid IP address syntax"
);
}
}
vendor/pezkuwi-zombienet-sdk/crates/configuration/src/hrmp_channel.rs
Vendored
+137
View File
@@ -0,0 +1,137 @@
use std::marker::PhantomData;
use serde::{Deserialize, Serialize};
use crate::shared::{macros::states, types::ParaId};
/// HRMP channel configuration, with fine-grained configuration options.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct HrmpChannelConfig {
sender: ParaId,
recipient: ParaId,
max_capacity: u32,
max_message_size: u32,
}
impl HrmpChannelConfig {
/// The sending parachain ID.
pub fn sender(&self) -> ParaId {
self.sender
}
/// The receiving parachain ID.
pub fn recipient(&self) -> ParaId {
self.recipient
}
/// The maximum capacity of messages in the channel.
pub fn max_capacity(&self) -> u32 {
self.max_capacity
}
/// The maximum size of a message in the channel.
pub fn max_message_size(&self) -> u32 {
self.max_message_size
}
}
states! {
Initial,
WithSender,
WithRecipient
}
/// HRMP channel configuration builder, used to build an [`HrmpChannelConfig`] declaratively with fields validation.
pub struct HrmpChannelConfigBuilder<State> {
config: HrmpChannelConfig,
_state: PhantomData<State>,
}
impl Default for HrmpChannelConfigBuilder<Initial> {
fn default() -> Self {
Self {
config: HrmpChannelConfig {
sender: 0,
recipient: 0,
max_capacity: 8,
max_message_size: 512,
},
_state: PhantomData,
}
}
}
impl<A> HrmpChannelConfigBuilder<A> {
fn transition<B>(&self, config: HrmpChannelConfig) -> HrmpChannelConfigBuilder<B> {
HrmpChannelConfigBuilder {
config,
_state: PhantomData,
}
}
}
impl HrmpChannelConfigBuilder<Initial> {
pub fn new() -> Self {
Self::default()
}
/// Set the sending parachain ID.
pub fn with_sender(self, sender: ParaId) -> HrmpChannelConfigBuilder<WithSender> {
self.transition(HrmpChannelConfig {
sender,
..self.config
})
}
}
impl HrmpChannelConfigBuilder<WithSender> {
/// Set the receiving parachain ID.
pub fn with_recipient(self, recipient: ParaId) -> HrmpChannelConfigBuilder<WithRecipient> {
self.transition(HrmpChannelConfig {
recipient,
..self.config
})
}
}
impl HrmpChannelConfigBuilder<WithRecipient> {
/// Set the max capacity of messages in the channel.
pub fn with_max_capacity(self, max_capacity: u32) -> Self {
self.transition(HrmpChannelConfig {
max_capacity,
..self.config
})
}
/// Set the maximum size of a message in the channel.
pub fn with_max_message_size(self, max_message_size: u32) -> Self {
self.transition(HrmpChannelConfig {
max_message_size,
..self.config
})
}
pub fn build(self) -> HrmpChannelConfig {
self.config
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn hrmp_channel_config_builder_should_build_a_new_hrmp_channel_config_correctly() {
let hrmp_channel_config = HrmpChannelConfigBuilder::new()
.with_sender(1000)
.with_recipient(2000)
.with_max_capacity(50)
.with_max_message_size(100)
.build();
assert_eq!(hrmp_channel_config.sender(), 1000);
assert_eq!(hrmp_channel_config.recipient(), 2000);
assert_eq!(hrmp_channel_config.max_capacity(), 50);
assert_eq!(hrmp_channel_config.max_message_size(), 100);
}
}
vendor/pezkuwi-zombienet-sdk/crates/configuration/src/lib.rs
Vendored
+102
View File
@@ -0,0 +1,102 @@
//! This crate is used to create type safe configuration for Zombienet SDK using nested builders.
//!
//!
//! The main entry point of this crate is the [`NetworkConfigBuilder`] which is used to build a full network configuration
//! but all inner builders are also exposed to allow more granular control over the configuration.
//!
//! **Note**: Not all options can be checked at compile time and some will be checked at runtime when spawning a
//! network (e.g.: supported args for a specific node version).
//!
//! # Example
//! ```
//! use zombienet_configuration::NetworkConfigBuilder;
//!
//! let simple_configuration = NetworkConfigBuilder::new()
//! .with_relaychain(|relaychain| {
//! relaychain
//! .with_chain("polkadot")
//! .with_random_nominators_count(10)
//! .with_default_resources(|resources| {
//! resources
//! .with_limit_cpu("1000m")
//! .with_request_memory("1Gi")
//! .with_request_cpu(100_000)
//! })
//! .with_node(|node| {
//! node.with_name("node")
//! .with_command("command")
//! .validator(true)
//! })
//! })
//! .with_parachain(|parachain| {
//! parachain
//! .with_id(1000)
//! .with_chain("myparachain1")
//! .with_initial_balance(100_000)
//! .with_default_image("myimage:version")
//! .with_collator(|collator| {
//! collator
//! .with_name("collator1")
//! .with_command("command1")
//! .validator(true)
//! })
//! })
//! .with_parachain(|parachain| {
//! parachain
//! .with_id(2000)
//! .with_chain("myparachain2")
//! .with_initial_balance(50_0000)
//! .with_collator(|collator| {
//! collator
//! .with_name("collator2")
//! .with_command("command2")
//! .validator(true)
//! })
//! })
//! .with_hrmp_channel(|hrmp_channel1| {
//! hrmp_channel1
//! .with_sender(1)
//! .with_recipient(2)
//! .with_max_capacity(200)
//! .with_max_message_size(500)
//! })
//! .with_hrmp_channel(|hrmp_channel2| {
//! hrmp_channel2
//! .with_sender(2)
//! .with_recipient(1)
//! .with_max_capacity(100)
//! .with_max_message_size(250)
//! })
//! .with_global_settings(|global_settings| {
//! global_settings
//! .with_network_spawn_timeout(1200)
//! .with_node_spawn_timeout(240)
//! })
//! .build();
//!
//! assert!(simple_configuration.is_ok())
//! ```
#![allow(clippy::expect_fun_call)]
mod global_settings;
mod hrmp_channel;
mod network;
mod relaychain;
pub mod shared;
mod teyrchain;
mod utils;
pub use global_settings::{GlobalSettings, GlobalSettingsBuilder};
pub use hrmp_channel::{HrmpChannelConfig, HrmpChannelConfigBuilder};
pub use network::{NetworkConfig, NetworkConfigBuilder, WithRelaychain};
pub use relaychain::{RelaychainConfig, RelaychainConfigBuilder};
// re-export shared
pub use shared::{node::NodeConfig, types};
pub use teyrchain::{
states as para_states, RegistrationStrategy, TeyrchainConfig, TeyrchainConfigBuilder,
};
// Backward compatibility aliases for external crates that use Polkadot SDK terminology
// These allow zombienet-orchestrator and other external crates to work with our renamed types
pub type ParachainConfig = TeyrchainConfig;
pub type ParachainConfigBuilder<S, C> = TeyrchainConfigBuilder<S, C>;
vendor/pezkuwi-zombienet-sdk/crates/configuration/src/network.rs
Vendored
+1923
View File
File diff suppressed because it is too large Load Diff
vendor/pezkuwi-zombienet-sdk/crates/configuration/src/relaychain.rs
Vendored
+1096
View File
File diff suppressed because it is too large Load Diff
vendor/pezkuwi-zombienet-sdk/crates/configuration/src/shared.rs
Vendored
+6
View File
@@ -0,0 +1,6 @@
pub mod errors;
pub mod helpers;
pub mod macros;
pub mod node;
pub mod resources;
pub mod types;
vendor/pezkuwi-zombienet-sdk/crates/configuration/src/shared/errors.rs
Vendored
+116
View File
@@ -0,0 +1,116 @@
use super::types::{ParaId, Port};
/// An error at the configuration level.
#[derive(thiserror::Error, Debug)]
pub enum ConfigError {
#[error("relaychain.{0}")]
Relaychain(anyhow::Error),
#[error("teyrchain[{0}].{1}")]
Teyrchain(ParaId, anyhow::Error),
#[error("global_settings.{0}")]
GlobalSettings(anyhow::Error),
#[error("nodes['{0}'].{1}")]
Node(String, anyhow::Error),
#[error("collators['{0}'].{1}")]
Collator(String, anyhow::Error),
}
/// An error at the field level.
#[derive(thiserror::Error, Debug)]
pub enum FieldError {
#[error("name: {0}")]
Name(anyhow::Error),
#[error("chain: {0}")]
Chain(anyhow::Error),
#[error("image: {0}")]
Image(anyhow::Error),
#[error("default_image: {0}")]
DefaultImage(anyhow::Error),
#[error("command: {0}")]
Command(anyhow::Error),
#[error("default_command: {0}")]
DefaultCommand(anyhow::Error),
#[error("bootnodes_addresses[{0}]: '{1}' {2}")]
BootnodesAddress(usize, String, anyhow::Error),
#[error("genesis_wasm_generator: {0}")]
GenesisWasmGenerator(anyhow::Error),
#[error("genesis_state_generator: {0}")]
GenesisStateGenerator(anyhow::Error),
#[error("local_ip: {0}")]
LocalIp(anyhow::Error),
#[error("default_resources.{0}")]
DefaultResources(anyhow::Error),
#[error("resources.{0}")]
Resources(anyhow::Error),
#[error("request_memory: {0}")]
RequestMemory(anyhow::Error),
#[error("request_cpu: {0}")]
RequestCpu(anyhow::Error),
#[error("limit_memory: {0}")]
LimitMemory(anyhow::Error),
#[error("limit_cpu: {0}")]
LimitCpu(anyhow::Error),
#[error("ws_port: {0}")]
WsPort(anyhow::Error),
#[error("rpc_port: {0}")]
RpcPort(anyhow::Error),
#[error("prometheus_port: {0}")]
PrometheusPort(anyhow::Error),
#[error("p2p_port: {0}")]
P2pPort(anyhow::Error),
#[error("session_key: {0}")]
SessionKey(anyhow::Error),
#[error("registration_strategy: {0}")]
RegistrationStrategy(anyhow::Error),
}
/// A conversion error for shared types across fields.
#[derive(thiserror::Error, Debug, Clone)]
pub enum ConversionError {
#[error("'{0}' shouldn't contains whitespace")]
ContainsWhitespaces(String),
#[error("'{}' doesn't match regex '{}'", .value, .regex)]
DoesntMatchRegex { value: String, regex: String },
#[error("can't be empty")]
CantBeEmpty,
#[error("deserialize error")]
DeserializeError(String),
}
/// A validation error for shared types across fields.
#[derive(thiserror::Error, Debug, Clone)]
pub enum ValidationError {
#[error("'{0}' is already used across config")]
PortAlreadyUsed(Port),
#[error("can't be empty")]
CantBeEmpty(),
}
vendor/pezkuwi-zombienet-sdk/crates/configuration/src/shared/helpers.rs
Vendored
+118
View File
@@ -0,0 +1,118 @@
use std::{cell::RefCell, collections::HashSet, rc::Rc};
use support::constants::{BORROWABLE, THIS_IS_A_BUG};
use tracing::warn;
use super::{
errors::ValidationError,
types::{ParaId, Port, ValidationContext},
};
pub fn merge_errors(errors: Vec<anyhow::Error>, new_error: anyhow::Error) -> Vec<anyhow::Error> {
let mut errors = errors;
errors.push(new_error);
errors
}
pub fn merge_errors_vecs(
errors: Vec<anyhow::Error>,
new_errors: Vec<anyhow::Error>,
) -> Vec<anyhow::Error> {
let mut errors = errors;
for new_error in new_errors.into_iter() {
errors.push(new_error);
}
errors
}
/// Generates a unique name from a base name and the names already present in a
/// [`ValidationContext`].
///
/// Uses [`generate_unique_node_name_from_names()`] internally to ensure uniqueness.
/// Logs a warning if the generated name differs from the original due to duplicates.
pub fn generate_unique_node_name(
node_name: impl Into<String>,
validation_context: Rc<RefCell<ValidationContext>>,
) -> String {
let mut context = validation_context
.try_borrow_mut()
.expect(&format!("{BORROWABLE}, {THIS_IS_A_BUG}"));
generate_unique_node_name_from_names(node_name, &mut context.used_nodes_names)
}
/// Returns `node_name` if it is not already in `names`.
///
/// Otherwise, appends an incrementing `-{counter}` suffix until a unique name is found,
/// then returns it. Logs a warning when a duplicate is detected.
pub fn generate_unique_node_name_from_names(
node_name: impl Into<String>,
names: &mut HashSet<String>,
) -> String {
let node_name = node_name.into();
if names.insert(node_name.clone()) {
return node_name;
}
let mut counter = 1;
let mut candidate = node_name.clone();
while names.contains(&candidate) {
candidate = format!("{node_name}-{counter}");
counter += 1;
}
warn!(
original = %node_name,
adjusted = %candidate,
"Duplicate node name detected."
);
names.insert(candidate.clone());
candidate
}
pub fn ensure_value_is_not_empty(value: &str) -> Result<(), anyhow::Error> {
if value.is_empty() {
Err(ValidationError::CantBeEmpty().into())
} else {
Ok(())
}
}
pub fn ensure_port_unique(
port: Port,
validation_context: Rc<RefCell<ValidationContext>>,
) -> Result<(), anyhow::Error> {
let mut context = validation_context
.try_borrow_mut()
.expect(&format!("{BORROWABLE}, {THIS_IS_A_BUG}"));
if !context.used_ports.contains(&port) {
context.used_ports.push(port);
return Ok(());
}
Err(ValidationError::PortAlreadyUsed(port).into())
}
pub fn generate_unique_para_id(
para_id: ParaId,
validation_context: Rc<RefCell<ValidationContext>>,
) -> String {
let mut context = validation_context
.try_borrow_mut()
.expect(&format!("{BORROWABLE}, {THIS_IS_A_BUG}"));
if let Some(suffix) = context.used_para_ids.get_mut(&para_id) {
*suffix += 1;
format!("{para_id}-{suffix}")
} else {
// insert 0, since will be used next time.
context.used_para_ids.insert(para_id, 0);
para_id.to_string()
}
}
vendor/pezkuwi-zombienet-sdk/crates/configuration/src/shared/macros.rs
Vendored
+11
View File
@@ -0,0 +1,11 @@
// Helper to define states of a type.
// We use an enum with no variants because it can't be constructed by definition.
macro_rules! states {
($($ident:ident),*) => {
$(
pub enum $ident {}
)*
};
}
pub(crate) use states;
vendor/pezkuwi-zombienet-sdk/crates/configuration/src/shared/node.rs
Vendored
+1355
View File
File diff suppressed because it is too large Load Diff
vendor/pezkuwi-zombienet-sdk/crates/configuration/src/shared/resources.rs
Vendored
+489
View File
@@ -0,0 +1,489 @@
use std::error::Error;
use lazy_static::lazy_static;
use regex::Regex;
use serde::{
de::{self},
ser::SerializeStruct,
Deserialize, Serialize,
};
use support::constants::{SHOULD_COMPILE, THIS_IS_A_BUG};
use super::{
errors::{ConversionError, FieldError},
helpers::merge_errors,
};
/// A resource quantity used to define limits (k8s/podman only).
/// It can be constructed from a `&str` or u64, if it fails, it returns a [`ConversionError`].
/// Possible optional prefixes are: m, K, M, G, T, P, E, Ki, Mi, Gi, Ti, Pi, Ei
///
/// # Examples
///
/// ```
/// use zombienet_configuration::shared::resources::ResourceQuantity;
///
/// let quantity1: ResourceQuantity = "100000".try_into().unwrap();
/// let quantity2: ResourceQuantity = "1000m".try_into().unwrap();
/// let quantity3: ResourceQuantity = "1Gi".try_into().unwrap();
/// let quantity4: ResourceQuantity = 10_000.into();
///
/// assert_eq!(quantity1.as_str(), "100000");
/// assert_eq!(quantity2.as_str(), "1000m");
/// assert_eq!(quantity3.as_str(), "1Gi");
/// assert_eq!(quantity4.as_str(), "10000");
/// ```
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct ResourceQuantity(String);
impl ResourceQuantity {
pub fn as_str(&self) -> &str {
&self.0
}
}
impl TryFrom<&str> for ResourceQuantity {
type Error = ConversionError;
fn try_from(value: &str) -> Result<Self, Self::Error> {
lazy_static! {
static ref RE: Regex = Regex::new(r"^\d+(.\d+)?(m|K|M|G|T|P|E|Ki|Mi|Gi|Ti|Pi|Ei)?$")
.expect(&format!("{SHOULD_COMPILE}, {THIS_IS_A_BUG}"));
}
if !RE.is_match(value) {
return Err(ConversionError::DoesntMatchRegex {
value: value.to_string(),
regex: r"^\d+(.\d+)?(m|K|M|G|T|P|E|Ki|Mi|Gi|Ti|Pi|Ei)?$".to_string(),
});
}
Ok(Self(value.to_string()))
}
}
impl From<u64> for ResourceQuantity {
fn from(value: u64) -> Self {
Self(value.to_string())
}
}
/// Resources limits used in the context of podman/k8s.
#[derive(Debug, Default, Clone, PartialEq)]
pub struct Resources {
request_memory: Option<ResourceQuantity>,
request_cpu: Option<ResourceQuantity>,
limit_memory: Option<ResourceQuantity>,
limit_cpu: Option<ResourceQuantity>,
}
#[derive(Serialize, Deserialize)]
struct ResourcesField {
memory: Option<ResourceQuantity>,
cpu: Option<ResourceQuantity>,
}
impl Serialize for Resources {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
let mut state = serializer.serialize_struct("Resources", 2)?;
if self.request_memory.is_some() || self.request_memory.is_some() {
state.serialize_field(
"requests",
&ResourcesField {
memory: self.request_memory.clone(),
cpu: self.request_cpu.clone(),
},
)?;
} else {
state.skip_field("requests")?;
}
if self.limit_memory.is_some() || self.limit_memory.is_some() {
state.serialize_field(
"limits",
&ResourcesField {
memory: self.limit_memory.clone(),
cpu: self.limit_cpu.clone(),
},
)?;
} else {
state.skip_field("limits")?;
}
state.end()
}
}
struct ResourcesVisitor;
impl<'de> de::Visitor<'de> for ResourcesVisitor {
type Value = Resources;
fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
formatter.write_str("a resources object")
}
fn visit_map<A>(self, mut map: A) -> Result<Self::Value, A::Error>
where
A: de::MapAccess<'de>,
{
let mut resources: Resources = Resources::default();
while let Some((key, value)) = map.next_entry::<String, ResourcesField>()? {
match key.as_str() {
"requests" => {
resources.request_memory = value.memory;
resources.request_cpu = value.cpu;
},
"limits" => {
resources.limit_memory = value.memory;
resources.limit_cpu = value.cpu;
},
_ => {
return Err(de::Error::unknown_field(
&key,
&["requests", "limits", "cpu", "memory"],
))
},
}
}
Ok(resources)
}
}
impl<'de> Deserialize<'de> for Resources {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: serde::Deserializer<'de>,
{
deserializer.deserialize_any(ResourcesVisitor)
}
}
impl Resources {
/// Memory limit applied to requests.
pub fn request_memory(&self) -> Option<&ResourceQuantity> {
self.request_memory.as_ref()
}
/// CPU limit applied to requests.
pub fn request_cpu(&self) -> Option<&ResourceQuantity> {
self.request_cpu.as_ref()
}
/// Overall memory limit applied.
pub fn limit_memory(&self) -> Option<&ResourceQuantity> {
self.limit_memory.as_ref()
}
/// Overall CPU limit applied.
pub fn limit_cpu(&self) -> Option<&ResourceQuantity> {
self.limit_cpu.as_ref()
}
}
/// A resources builder, used to build a [`Resources`] declaratively with fields validation.
#[derive(Debug, Default)]
pub struct ResourcesBuilder {
config: Resources,
errors: Vec<anyhow::Error>,
}
impl ResourcesBuilder {
pub fn new() -> ResourcesBuilder {
Self::default()
}
fn transition(config: Resources, errors: Vec<anyhow::Error>) -> Self {
Self { config, errors }
}
/// Set the requested memory for a pod. This is the minimum memory allocated for a pod.
pub fn with_request_memory<T>(self, quantity: T) -> Self
where
T: TryInto<ResourceQuantity>,
T::Error: Error + Send + Sync + 'static,
{
match quantity.try_into() {
Ok(quantity) => Self::transition(
Resources {
request_memory: Some(quantity),
..self.config
},
self.errors,
),
Err(error) => Self::transition(
self.config,
merge_errors(self.errors, FieldError::RequestMemory(error.into()).into()),
),
}
}
/// Set the requested CPU limit for a pod. This is the minimum CPU allocated for a pod.
pub fn with_request_cpu<T>(self, quantity: T) -> Self
where
T: TryInto<ResourceQuantity>,
T::Error: Error + Send + Sync + 'static,
{
match quantity.try_into() {
Ok(quantity) => Self::transition(
Resources {
request_cpu: Some(quantity),
..self.config
},
self.errors,
),
Err(error) => Self::transition(
self.config,
merge_errors(self.errors, FieldError::RequestCpu(error.into()).into()),
),
}
}
/// Set the overall memory limit for a pod. This is the maximum memory threshold for a pod.
pub fn with_limit_memory<T>(self, quantity: T) -> Self
where
T: TryInto<ResourceQuantity>,
T::Error: Error + Send + Sync + 'static,
{
match quantity.try_into() {
Ok(quantity) => Self::transition(
Resources {
limit_memory: Some(quantity),
..self.config
},
self.errors,
),
Err(error) => Self::transition(
self.config,
merge_errors(self.errors, FieldError::LimitMemory(error.into()).into()),
),
}
}
/// Set the overall CPU limit for a pod. This is the maximum CPU threshold for a pod.
pub fn with_limit_cpu<T>(self, quantity: T) -> Self
where
T: TryInto<ResourceQuantity>,
T::Error: Error + Send + Sync + 'static,
{
match quantity.try_into() {
Ok(quantity) => Self::transition(
Resources {
limit_cpu: Some(quantity),
..self.config
},
self.errors,
),
Err(error) => Self::transition(
self.config,
merge_errors(self.errors, FieldError::LimitCpu(error.into()).into()),
),
}
}
/// Seals the builder and returns a [`Resources`] if there are no validation errors, else returns errors.
pub fn build(self) -> Result<Resources, Vec<anyhow::Error>> {
if !self.errors.is_empty() {
return Err(self.errors);
}
Ok(self.config)
}
}
#[cfg(test)]
#[allow(non_snake_case)]
mod tests {
use super::*;
use crate::NetworkConfig;
macro_rules! impl_resources_quantity_unit_test {
($val:literal) => {{
let resources = ResourcesBuilder::new()
.with_request_memory($val)
.build()
.unwrap();
assert_eq!(resources.request_memory().unwrap().as_str(), $val);
assert_eq!(resources.request_cpu(), None);
assert_eq!(resources.limit_cpu(), None);
assert_eq!(resources.limit_memory(), None);
}};
}
#[test]
fn converting_a_string_a_resource_quantity_without_unit_should_succeeds() {
impl_resources_quantity_unit_test!("1000");
}
#[test]
fn converting_a_str_with_m_unit_into_a_resource_quantity_should_succeeds() {
impl_resources_quantity_unit_test!("100m");
}
#[test]
fn converting_a_str_with_K_unit_into_a_resource_quantity_should_succeeds() {
impl_resources_quantity_unit_test!("50K");
}
#[test]
fn converting_a_str_with_M_unit_into_a_resource_quantity_should_succeeds() {
impl_resources_quantity_unit_test!("100M");
}
#[test]
fn converting_a_str_with_G_unit_into_a_resource_quantity_should_succeeds() {
impl_resources_quantity_unit_test!("1G");
}
#[test]
fn converting_a_str_with_T_unit_into_a_resource_quantity_should_succeeds() {
impl_resources_quantity_unit_test!("0.01T");
}
#[test]
fn converting_a_str_with_P_unit_into_a_resource_quantity_should_succeeds() {
impl_resources_quantity_unit_test!("0.00001P");
}
#[test]
fn converting_a_str_with_E_unit_into_a_resource_quantity_should_succeeds() {
impl_resources_quantity_unit_test!("0.000000001E");
}
#[test]
fn converting_a_str_with_Ki_unit_into_a_resource_quantity_should_succeeds() {
impl_resources_quantity_unit_test!("50Ki");
}
#[test]
fn converting_a_str_with_Mi_unit_into_a_resource_quantity_should_succeeds() {
impl_resources_quantity_unit_test!("100Mi");
}
#[test]
fn converting_a_str_with_Gi_unit_into_a_resource_quantity_should_succeeds() {
impl_resources_quantity_unit_test!("1Gi");
}
#[test]
fn converting_a_str_with_Ti_unit_into_a_resource_quantity_should_succeeds() {
impl_resources_quantity_unit_test!("0.01Ti");
}
#[test]
fn converting_a_str_with_Pi_unit_into_a_resource_quantity_should_succeeds() {
impl_resources_quantity_unit_test!("0.00001Pi");
}
#[test]
fn converting_a_str_with_Ei_unit_into_a_resource_quantity_should_succeeds() {
impl_resources_quantity_unit_test!("0.000000001Ei");
}
#[test]
fn resources_config_builder_should_succeeds_and_returns_a_resources_config() {
let resources = ResourcesBuilder::new()
.with_request_memory("200M")
.with_request_cpu("1G")
.with_limit_cpu("500M")
.with_limit_memory("2G")
.build()
.unwrap();
assert_eq!(resources.request_memory().unwrap().as_str(), "200M");
assert_eq!(resources.request_cpu().unwrap().as_str(), "1G");
assert_eq!(resources.limit_cpu().unwrap().as_str(), "500M");
assert_eq!(resources.limit_memory().unwrap().as_str(), "2G");
}
#[test]
fn resources_config_toml_import_should_succeeds_and_returns_a_resources_config() {
let load_from_toml =
NetworkConfig::load_from_toml("./testing/snapshots/0001-big-network.toml").unwrap();
let resources = load_from_toml.relaychain().default_resources().unwrap();
assert_eq!(resources.request_memory().unwrap().as_str(), "500M");
assert_eq!(resources.request_cpu().unwrap().as_str(), "100000");
assert_eq!(resources.limit_cpu().unwrap().as_str(), "10Gi");
assert_eq!(resources.limit_memory().unwrap().as_str(), "4000M");
}
#[test]
fn resources_config_builder_should_fails_and_returns_an_error_if_couldnt_parse_request_memory()
{
let resources_builder = ResourcesBuilder::new().with_request_memory("invalid");
let errors = resources_builder.build().err().unwrap();
assert_eq!(errors.len(), 1);
assert_eq!(
errors.first().unwrap().to_string(),
r"request_memory: 'invalid' doesn't match regex '^\d+(.\d+)?(m|K|M|G|T|P|E|Ki|Mi|Gi|Ti|Pi|Ei)?$'"
);
}
#[test]
fn resources_config_builder_should_fails_and_returns_an_error_if_couldnt_parse_request_cpu() {
let resources_builder = ResourcesBuilder::new().with_request_cpu("invalid");
let errors = resources_builder.build().err().unwrap();
assert_eq!(errors.len(), 1);
assert_eq!(
errors.first().unwrap().to_string(),
r"request_cpu: 'invalid' doesn't match regex '^\d+(.\d+)?(m|K|M|G|T|P|E|Ki|Mi|Gi|Ti|Pi|Ei)?$'"
);
}
#[test]
fn resources_config_builder_should_fails_and_returns_an_error_if_couldnt_parse_limit_memory() {
let resources_builder = ResourcesBuilder::new().with_limit_memory("invalid");
let errors = resources_builder.build().err().unwrap();
assert_eq!(errors.len(), 1);
assert_eq!(
errors.first().unwrap().to_string(),
r"limit_memory: 'invalid' doesn't match regex '^\d+(.\d+)?(m|K|M|G|T|P|E|Ki|Mi|Gi|Ti|Pi|Ei)?$'"
);
}
#[test]
fn resources_config_builder_should_fails_and_returns_an_error_if_couldnt_parse_limit_cpu() {
let resources_builder = ResourcesBuilder::new().with_limit_cpu("invalid");
let errors = resources_builder.build().err().unwrap();
assert_eq!(errors.len(), 1);
assert_eq!(
errors.first().unwrap().to_string(),
r"limit_cpu: 'invalid' doesn't match regex '^\d+(.\d+)?(m|K|M|G|T|P|E|Ki|Mi|Gi|Ti|Pi|Ei)?$'"
);
}
#[test]
fn resources_config_builder_should_fails_and_returns_multiple_error_if_couldnt_parse_multiple_fields(
) {
let resources_builder = ResourcesBuilder::new()
.with_limit_cpu("invalid")
.with_request_memory("invalid");
let errors = resources_builder.build().err().unwrap();
assert_eq!(errors.len(), 2);
assert_eq!(
errors.first().unwrap().to_string(),
r"limit_cpu: 'invalid' doesn't match regex '^\d+(.\d+)?(m|K|M|G|T|P|E|Ki|Mi|Gi|Ti|Pi|Ei)?$'"
);
assert_eq!(
errors.get(1).unwrap().to_string(),
r"request_memory: 'invalid' doesn't match regex '^\d+(.\d+)?(m|K|M|G|T|P|E|Ki|Mi|Gi|Ti|Pi|Ei)?$'"
);
}
}
vendor/pezkuwi-zombienet-sdk/crates/configuration/src/shared/types.rs
Vendored
+930
View File
@@ -0,0 +1,930 @@
use std::{
collections::{HashMap, HashSet},
error::Error,
fmt::{self, Display},
path::PathBuf,
str::FromStr,
};
use anyhow::anyhow;
use lazy_static::lazy_static;
use regex::Regex;
use serde::{
de::{self, IntoDeserializer},
Deserialize, Deserializer, Serialize,
};
use support::constants::{INFAILABLE, SHOULD_COMPILE, THIS_IS_A_BUG};
use tokio::fs;
use url::Url;
use super::{errors::ConversionError, resources::Resources};
/// An alias for a duration in seconds.
pub type Duration = u32;
/// An alias for a port.
pub type Port = u16;
/// An alias for a parachain ID.
pub type ParaId = u32;
/// Custom type wrapping u128 to add custom Serialization/Deserialization logic because it's not supported
/// issue tracking the problem: <https://github.com/toml-rs/toml/issues/540>
#[derive(Default, Debug, Clone, PartialEq)]
pub struct U128(pub(crate) u128);
impl From<u128> for U128 {
fn from(value: u128) -> Self {
Self(value)
}
}
impl TryFrom<&str> for U128 {
type Error = Box<dyn Error>;
fn try_from(value: &str) -> Result<Self, Self::Error> {
Ok(Self(value.to_string().parse::<u128>()?))
}
}
impl Serialize for U128 {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
// here we add a prefix to the string to be able to replace the wrapped
// value with "" to a value without "" in the TOML string
serializer.serialize_str(&format!("U128%{}", self.0))
}
}
struct U128Visitor;
impl de::Visitor<'_> for U128Visitor {
type Value = U128;
fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
formatter.write_str("an integer between 0 and 2^128 − 1.")
}
fn visit_str<E>(self, v: &str) -> Result<Self::Value, E>
where
E: de::Error,
{
v.try_into().map_err(de::Error::custom)
}
}
impl<'de> Deserialize<'de> for U128 {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: Deserializer<'de>,
{
deserializer.deserialize_str(U128Visitor)
}
}
/// A chain name.
/// It can be constructed for an `&str`, if it fails, it will returns a [`ConversionError`].
///
/// # Examples:
/// ```
/// use zombienet_configuration::shared::types::Chain;
///
/// let polkadot: Chain = "polkadot".try_into().unwrap();
/// let kusama: Chain = "kusama".try_into().unwrap();
/// let myparachain: Chain = "myparachain".try_into().unwrap();
///
/// assert_eq!(polkadot.as_str(), "polkadot");
/// assert_eq!(kusama.as_str(), "kusama");
/// assert_eq!(myparachain.as_str(), "myparachain");
/// ```
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct Chain(String);
impl TryFrom<&str> for Chain {
type Error = ConversionError;
fn try_from(value: &str) -> Result<Self, Self::Error> {
if value.contains(char::is_whitespace) {
return Err(ConversionError::ContainsWhitespaces(value.to_string()));
}
if value.is_empty() {
return Err(ConversionError::CantBeEmpty);
}
Ok(Self(value.to_string()))
}
}
impl Chain {
pub fn as_str(&self) -> &str {
&self.0
}
}
/// A container image.
/// It can be constructed from an `&str` including a combination of name, version, IPv4 or/and hostname, if it fails, it will returns a [`ConversionError`].
///
/// # Examples:
/// ```
/// use zombienet_configuration::shared::types::Image;
///
/// let image1: Image = "name".try_into().unwrap();
/// let image2: Image = "name:version".try_into().unwrap();
/// let image3: Image = "myrepo.com/name:version".try_into().unwrap();
/// let image4: Image = "10.15.43.155/name:version".try_into().unwrap();
///
/// assert_eq!(image1.as_str(), "name");
/// assert_eq!(image2.as_str(), "name:version");
/// assert_eq!(image3.as_str(), "myrepo.com/name:version");
/// assert_eq!(image4.as_str(), "10.15.43.155/name:version");
/// ```
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct Image(String);
impl TryFrom<&str> for Image {
type Error = ConversionError;
fn try_from(value: &str) -> Result<Self, Self::Error> {
static IP_PART: &str = "((([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]))";
static HOSTNAME_PART: &str = "((([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9]).)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9-]*[A-Za-z0-9]))";
static TAG_NAME_PART: &str = "([a-z0-9](-*[a-z0-9])*)";
static TAG_VERSION_PART: &str = "([a-z0-9_]([-._a-z0-9])*)";
lazy_static! {
static ref RE: Regex = Regex::new(&format!(
"^({IP_PART}|{HOSTNAME_PART}/)?{TAG_NAME_PART}(:{TAG_VERSION_PART})?$",
))
.expect(&format!("{SHOULD_COMPILE}, {THIS_IS_A_BUG}"));
};
if !RE.is_match(value) {
return Err(ConversionError::DoesntMatchRegex {
value: value.to_string(),
regex: "^([ip]|[hostname]/)?[tag_name]:[tag_version]?$".to_string(),
});
}
Ok(Self(value.to_string()))
}
}
impl Image {
pub fn as_str(&self) -> &str {
&self.0
}
}
/// A command that will be executed natively (native provider) or in a container (podman/k8s).
/// It can be constructed from an `&str`, if it fails, it will returns a [`ConversionError`].
///
/// # Examples:
/// ```
/// use zombienet_configuration::shared::types::Command;
///
/// let command1: Command = "mycommand".try_into().unwrap();
/// let command2: Command = "myothercommand".try_into().unwrap();
///
/// assert_eq!(command1.as_str(), "mycommand");
/// assert_eq!(command2.as_str(), "myothercommand");
/// ```
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct Command(String);
impl TryFrom<&str> for Command {
type Error = ConversionError;
fn try_from(value: &str) -> Result<Self, Self::Error> {
if value.contains(char::is_whitespace) {
return Err(ConversionError::ContainsWhitespaces(value.to_string()));
}
Ok(Self(value.to_string()))
}
}
impl Default for Command {
fn default() -> Self {
Self(String::from("polkadot"))
}
}
impl Command {
pub fn as_str(&self) -> &str {
&self.0
}
}
/// A command with optional custom arguments, the command will be executed natively (native provider) or in a container (podman/k8s).
/// It can be constructed from an `&str`, if it fails, it will returns a [`ConversionError`].
///
/// # Examples:
/// ```
/// use zombienet_configuration::shared::types::CommandWithCustomArgs;
///
/// let command1: CommandWithCustomArgs = "mycommand --demo=2 --other-flag".try_into().unwrap();
/// let command2: CommandWithCustomArgs = "my_other_cmd_without_args".try_into().unwrap();
///
/// assert_eq!(command1.cmd().as_str(), "mycommand");
/// assert_eq!(command2.cmd().as_str(), "my_other_cmd_without_args");
/// ```
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct CommandWithCustomArgs(Command, Vec<Arg>);
impl TryFrom<&str> for CommandWithCustomArgs {
type Error = ConversionError;
fn try_from(value: &str) -> Result<Self, Self::Error> {
if value.is_empty() {
return Err(ConversionError::CantBeEmpty);
}
let mut parts = value.split_whitespace().collect::<Vec<&str>>();
let cmd = parts.remove(0).try_into().unwrap();
let args = parts
.iter()
.map(|x| {
Arg::deserialize(x.into_deserializer()).map_err(|_: serde_json::Error| {
ConversionError::DeserializeError(String::from(*x))
})
})
.collect::<Result<Vec<Arg>, _>>()?;
Ok(Self(cmd, args))
}
}
impl Default for CommandWithCustomArgs {
fn default() -> Self {
Self("polkadot".try_into().unwrap(), vec![])
}
}
impl CommandWithCustomArgs {
pub fn cmd(&self) -> &Command {
&self.0
}
pub fn args(&self) -> &Vec<Arg> {
&self.1
}
}
/// A location for a locally or remotely stored asset.
/// It can be constructed from an [`url::Url`], a [`std::path::PathBuf`] or an `&str`.
///
/// # Examples:
/// ```
/// use url::Url;
/// use std::{path::PathBuf, str::FromStr};
/// use zombienet_configuration::shared::types::AssetLocation;
///
/// let url_location: AssetLocation = Url::from_str("https://mycloudstorage.com/path/to/my/file.tgz").unwrap().into();
/// let url_location2: AssetLocation = "https://mycloudstorage.com/path/to/my/file.tgz".into();
/// let path_location: AssetLocation = PathBuf::from_str("/tmp/path/to/my/file").unwrap().into();
/// let path_location2: AssetLocation = "/tmp/path/to/my/file".into();
///
/// assert!(matches!(url_location, AssetLocation::Url(value) if value.as_str() == "https://mycloudstorage.com/path/to/my/file.tgz"));
/// assert!(matches!(url_location2, AssetLocation::Url(value) if value.as_str() == "https://mycloudstorage.com/path/to/my/file.tgz"));
/// assert!(matches!(path_location, AssetLocation::FilePath(value) if value.to_str().unwrap() == "/tmp/path/to/my/file"));
/// assert!(matches!(path_location2, AssetLocation::FilePath(value) if value.to_str().unwrap() == "/tmp/path/to/my/file"));
/// ```
#[derive(Debug, Clone, PartialEq)]
pub enum AssetLocation {
Url(Url),
FilePath(PathBuf),
}
impl From<Url> for AssetLocation {
fn from(value: Url) -> Self {
Self::Url(value)
}
}
impl From<PathBuf> for AssetLocation {
fn from(value: PathBuf) -> Self {
Self::FilePath(value)
}
}
impl From<&str> for AssetLocation {
fn from(value: &str) -> Self {
if let Ok(parsed_url) = Url::parse(value) {
return Self::Url(parsed_url);
}
Self::FilePath(PathBuf::from_str(value).expect(&format!("{INFAILABLE}, {THIS_IS_A_BUG}")))
}
}
impl Display for AssetLocation {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
AssetLocation::Url(value) => write!(f, "{}", value.as_str()),
AssetLocation::FilePath(value) => write!(f, "{}", value.display()),
}
}
}
impl AssetLocation {
/// Get the current asset (from file or url) and return the content
pub async fn get_asset(&self) -> Result<Vec<u8>, anyhow::Error> {
let contents = match self {
AssetLocation::Url(location) => {
let res = reqwest::get(location.as_ref()).await.map_err(|err| {
anyhow!("Error dowinloding asset from url {location} - {err}")
})?;
res.bytes().await.unwrap().into()
},
AssetLocation::FilePath(filepath) => {
tokio::fs::read(filepath).await.map_err(|err| {
anyhow!(
"Error reading asset from path {} - {}",
filepath.to_string_lossy(),
err
)
})?
},
};
Ok(contents)
}
/// Write asset (from file or url) to the destination path.
pub async fn dump_asset(&self, dst_path: impl Into<PathBuf>) -> Result<(), anyhow::Error> {
let contents = self.get_asset().await?;
fs::write(dst_path.into(), contents).await?;
Ok(())
}
}
impl Serialize for AssetLocation {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
serializer.serialize_str(&self.to_string())
}
}
struct AssetLocationVisitor;
impl de::Visitor<'_> for AssetLocationVisitor {
type Value = AssetLocation;
fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
formatter.write_str("a string")
}
fn visit_str<E>(self, v: &str) -> Result<Self::Value, E>
where
E: de::Error,
{
Ok(AssetLocation::from(v))
}
}
impl<'de> Deserialize<'de> for AssetLocation {
fn deserialize<D>(deserializer: D) -> Result<AssetLocation, D::Error>
where
D: Deserializer<'de>,
{
deserializer.deserialize_any(AssetLocationVisitor)
}
}
/// A CLI argument passed to an executed command, can be an option with an assigned value or a simple flag to enable/disable a feature.
/// A flag arg can be constructed from a `&str` and a option arg can be constructed from a `(&str, &str)`.
///
/// # Examples:
/// ```
/// use zombienet_configuration::shared::types::Arg;
///
/// let flag_arg: Arg = "myflag".into();
/// let option_arg: Arg = ("name", "value").into();
///
/// assert!(matches!(flag_arg, Arg::Flag(value) if value == "myflag"));
/// assert!(matches!(option_arg, Arg::Option(name, value) if name == "name" && value == "value"));
/// ```
#[derive(Debug, Clone, PartialEq)]
pub enum Arg {
Flag(String),
Option(String, String),
Array(String, Vec<String>),
}
impl From<&str> for Arg {
fn from(flag: &str) -> Self {
Self::Flag(flag.to_owned())
}
}
impl From<(&str, &str)> for Arg {
fn from((option, value): (&str, &str)) -> Self {
Self::Option(option.to_owned(), value.to_owned())
}
}
impl<T> From<(&str, &[T])> for Arg
where
T: AsRef<str> + Clone,
{
fn from((option, values): (&str, &[T])) -> Self {
Self::Array(
option.to_owned(),
values.iter().map(|v| v.as_ref().to_string()).collect(),
)
}
}
impl<T> From<(&str, Vec<T>)> for Arg
where
T: AsRef<str>,
{
fn from((option, values): (&str, Vec<T>)) -> Self {
Self::Array(
option.to_owned(),
values.into_iter().map(|v| v.as_ref().to_string()).collect(),
)
}
}
impl Serialize for Arg {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
match self {
Arg::Flag(value) => serializer.serialize_str(value),
Arg::Option(option, value) => serializer.serialize_str(&format!("{option}={value}")),
Arg::Array(option, values) => {
serializer.serialize_str(&format!("{}=[{}]", option, values.join(",")))
},
}
}
}
struct ArgVisitor;
impl de::Visitor<'_> for ArgVisitor {
type Value = Arg;
fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
formatter.write_str("a string")
}
fn visit_str<E>(self, v: &str) -> Result<Self::Value, E>
where
E: de::Error,
{
// covers the "-lruntime=debug,parachain=trace" case
// TODO: Make this more generic by adding the scenario in the regex below
if v.starts_with("-l") || v.starts_with("-log") {
return Ok(Arg::Flag(v.to_string()));
}
// Handle argument removal syntax: -:--flag-name
if v.starts_with("-:") {
return Ok(Arg::Flag(v.to_string()));
}
let re = Regex::new("^(?<name_prefix>(?<prefix>-{1,2})?(?<name>[a-zA-Z]+(-[a-zA-Z]+)*))((?<separator>=| )(?<value>\\[[^\\]]*\\]|[^ ]+))?$").unwrap();
let captures = re.captures(v);
if let Some(captures) = captures {
if let Some(value) = captures.name("value") {
let name_prefix = captures
.name("name_prefix")
.expect("BUG: name_prefix capture group missing")
.as_str()
.to_string();
let val = value.as_str();
if val.starts_with('[') && val.ends_with(']') {
// Remove brackets and split by comma
let inner = &val[1..val.len() - 1];
let items: Vec<String> = inner
.split(',')
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.collect();
return Ok(Arg::Array(name_prefix, items));
} else {
return Ok(Arg::Option(name_prefix, val.to_string()));
}
}
if let Some(name_prefix) = captures.name("name_prefix") {
return Ok(Arg::Flag(name_prefix.as_str().to_string()));
}
}
Err(de::Error::custom(
"the provided argument is invalid and doesn't match Arg::Option, Arg::Flag or Arg::Array",
))
}
}
impl<'de> Deserialize<'de> for Arg {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: Deserializer<'de>,
{
deserializer.deserialize_any(ArgVisitor)
}
}
#[derive(Debug, Default, Clone)]
pub struct ValidationContext {
pub used_ports: Vec<Port>,
pub used_nodes_names: HashSet<String>,
// Store para_id already used
pub used_para_ids: HashMap<ParaId, u8>,
}
#[derive(Default, Debug, Clone, PartialEq, Deserialize)]
pub struct ChainDefaultContext {
pub(crate) default_command: Option<Command>,
pub(crate) default_image: Option<Image>,
pub(crate) default_resources: Option<Resources>,
pub(crate) default_db_snapshot: Option<AssetLocation>,
#[serde(default)]
pub(crate) default_args: Vec<Arg>,
}
/// Represent a runtime (.wasm) asset location and an
/// optional preset to use for chain-spec generation.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct ChainSpecRuntime {
pub location: AssetLocation,
pub preset: Option<String>,
}
impl ChainSpecRuntime {
pub fn new(location: AssetLocation) -> Self {
ChainSpecRuntime {
location,
preset: None,
}
}
pub fn with_preset(location: AssetLocation, preset: impl Into<String>) -> Self {
ChainSpecRuntime {
location,
preset: Some(preset.into()),
}
}
}
/// Represents a set of JSON overrides for a configuration.
///
/// The overrides can be provided as an inline JSON object or loaded from a
/// separate file via a path or URL.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(untagged)]
pub enum JsonOverrides {
/// A path or URL pointing to a JSON file containing the overrides.
Location(AssetLocation),
/// An inline JSON value representing the overrides.
Json(serde_json::Value),
}
impl From<AssetLocation> for JsonOverrides {
fn from(value: AssetLocation) -> Self {
Self::Location(value)
}
}
impl From<serde_json::Value> for JsonOverrides {
fn from(value: serde_json::Value) -> Self {
Self::Json(value)
}
}
impl From<&str> for JsonOverrides {
fn from(value: &str) -> Self {
Self::Location(AssetLocation::from(value))
}
}
impl Display for JsonOverrides {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
JsonOverrides::Location(location) => write!(f, "{location}"),
JsonOverrides::Json(json) => write!(f, "{json}"),
}
}
}
impl JsonOverrides {
pub async fn get(&self) -> Result<serde_json::Value, anyhow::Error> {
let contents = match self {
Self::Location(location) => serde_json::from_slice(&location.get_asset().await?)
.map_err(|err| anyhow!("Error converting asset to json {location} - {err}")),
Self::Json(json) => Ok(json.clone()),
};
contents
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_arg_flag_roundtrip() {
let arg = Arg::from("verbose");
let serialized = serde_json::to_string(&arg).unwrap();
let deserialized: Arg = serde_json::from_str(&serialized).unwrap();
assert_eq!(arg, deserialized);
}
#[test]
fn test_arg_option_roundtrip() {
let arg = Arg::from(("mode", "fast"));
let serialized = serde_json::to_string(&arg).unwrap();
let deserialized: Arg = serde_json::from_str(&serialized).unwrap();
assert_eq!(arg, deserialized);
}
#[test]
fn test_arg_array_roundtrip() {
let arg = Arg::from(("items", ["a", "b", "c"].as_slice()));
let serialized = serde_json::to_string(&arg).unwrap();
println!("serialized = {serialized}");
let deserialized: Arg = serde_json::from_str(&serialized).unwrap();
assert_eq!(arg, deserialized);
}
#[test]
fn test_arg_option_valid_input() {
let expected = Arg::from(("--foo", "bar"));
// name and value delimited with =
let valid = "\"--foo=bar\"";
let result: Result<Arg, _> = serde_json::from_str(valid);
assert_eq!(result.unwrap(), expected);
// name and value delimited with space
let valid = "\"--foo bar\"";
let result: Result<Arg, _> = serde_json::from_str(valid);
assert_eq!(result.unwrap(), expected);
// value contains =
let expected = Arg::from(("--foo", "bar=baz"));
let valid = "\"--foo=bar=baz\"";
let result: Result<Arg, _> = serde_json::from_str(valid);
assert_eq!(result.unwrap(), expected);
}
#[test]
fn test_arg_array_valid_input() {
let expected = Arg::from(("--foo", vec!["bar", "baz"]));
// name and values delimited with =
let valid = "\"--foo=[bar,baz]\"";
let result: Result<Arg, _> = serde_json::from_str(valid);
assert_eq!(result.unwrap(), expected);
// name and values delimited with space
let valid = "\"--foo [bar,baz]\"";
let result: Result<Arg, _> = serde_json::from_str(valid);
assert_eq!(result.unwrap(), expected);
// values delimited with commas and space
let valid = "\"--foo [bar , baz]\"";
let result: Result<Arg, _> = serde_json::from_str(valid);
assert_eq!(result.unwrap(), expected);
// empty values array
let expected = Arg::from(("--foo", Vec::<&str>::new()));
let valid = "\"--foo []\"";
let result: Result<Arg, _> = serde_json::from_str(valid);
assert_eq!(result.unwrap(), expected);
}
#[test]
fn test_arg_invalid_input() {
// missing = or space
let invalid = "\"--foo[bar]\"";
let result: Result<Arg, _> = serde_json::from_str(invalid);
assert!(result.is_err());
// value contains space
let invalid = "\"--foo=bar baz\"";
let result: Result<Arg, _> = serde_json::from_str(invalid);
println!("result = {result:?}");
assert!(result.is_err());
}
#[test]
fn converting_a_str_without_whitespaces_into_a_chain_should_succeeds() {
let got: Result<Chain, ConversionError> = "mychain".try_into();
assert_eq!(got.unwrap().as_str(), "mychain");
}
#[test]
fn converting_a_str_containing_tag_name_into_an_image_should_succeeds() {
let got: Result<Image, ConversionError> = "myimage".try_into();
assert_eq!(got.unwrap().as_str(), "myimage");
}
#[test]
fn converting_a_str_containing_tag_name_and_tag_version_into_an_image_should_succeeds() {
let got: Result<Image, ConversionError> = "myimage:version".try_into();
assert_eq!(got.unwrap().as_str(), "myimage:version");
}
#[test]
fn converting_a_str_containing_hostname_and_tag_name_into_an_image_should_succeeds() {
let got: Result<Image, ConversionError> = "myrepository.com/myimage".try_into();
assert_eq!(got.unwrap().as_str(), "myrepository.com/myimage");
}
#[test]
fn converting_a_str_containing_hostname_tag_name_and_tag_version_into_an_image_should_succeeds()
{
let got: Result<Image, ConversionError> = "myrepository.com/myimage:version".try_into();
assert_eq!(got.unwrap().as_str(), "myrepository.com/myimage:version");
}
#[test]
fn converting_a_str_containing_ip_and_tag_name_into_an_image_should_succeeds() {
let got: Result<Image, ConversionError> = "myrepository.com/myimage".try_into();
assert_eq!(got.unwrap().as_str(), "myrepository.com/myimage");
}
#[test]
fn converting_a_str_containing_ip_tag_name_and_tag_version_into_an_image_should_succeeds() {
let got: Result<Image, ConversionError> = "127.0.0.1/myimage:version".try_into();
assert_eq!(got.unwrap().as_str(), "127.0.0.1/myimage:version");
}
#[test]
fn converting_a_str_without_whitespaces_into_a_command_should_succeeds() {
let got: Result<Command, ConversionError> = "mycommand".try_into();
assert_eq!(got.unwrap().as_str(), "mycommand");
}
#[test]
fn converting_an_url_into_an_asset_location_should_succeeds() {
let url = Url::from_str("https://mycloudstorage.com/path/to/my/file.tgz").unwrap();
let got: AssetLocation = url.clone().into();
assert!(matches!(got, AssetLocation::Url(value) if value == url));
}
#[test]
fn converting_a_pathbuf_into_an_asset_location_should_succeeds() {
let pathbuf = PathBuf::from_str("/tmp/path/to/my/file").unwrap();
let got: AssetLocation = pathbuf.clone().into();
assert!(matches!(got, AssetLocation::FilePath(value) if value == pathbuf));
}
#[test]
fn converting_a_str_into_an_url_asset_location_should_succeeds() {
let url = "https://mycloudstorage.com/path/to/my/file.tgz";
let got: AssetLocation = url.into();
assert!(matches!(got, AssetLocation::Url(value) if value == Url::from_str(url).unwrap()));
}
#[test]
fn converting_a_str_into_an_filepath_asset_location_should_succeeds() {
let filepath = "/tmp/path/to/my/file";
let got: AssetLocation = filepath.into();
assert!(matches!(
got,
AssetLocation::FilePath(value) if value == PathBuf::from_str(filepath).unwrap()
));
}
#[test]
fn converting_a_str_into_an_flag_arg_should_succeeds() {
let got: Arg = "myflag".into();
assert!(matches!(got, Arg::Flag(flag) if flag == "myflag"));
}
#[test]
fn converting_a_str_tuple_into_an_option_arg_should_succeeds() {
let got: Arg = ("name", "value").into();
assert!(matches!(got, Arg::Option(name, value) if name == "name" && value == "value"));
}
#[test]
fn converting_a_str_with_whitespaces_into_a_chain_should_fails() {
let got: Result<Chain, ConversionError> = "my chain".try_into();
assert!(matches!(
got.clone().unwrap_err(),
ConversionError::ContainsWhitespaces(_)
));
assert_eq!(
got.unwrap_err().to_string(),
"'my chain' shouldn't contains whitespace"
);
}
#[test]
fn converting_an_empty_str_into_a_chain_should_fails() {
let got: Result<Chain, ConversionError> = "".try_into();
assert!(matches!(
got.clone().unwrap_err(),
ConversionError::CantBeEmpty
));
assert_eq!(got.unwrap_err().to_string(), "can't be empty");
}
#[test]
fn converting_a_str_containing_only_ip_into_an_image_should_fails() {
let got: Result<Image, ConversionError> = "127.0.0.1".try_into();
assert!(matches!(
got.clone().unwrap_err(),
ConversionError::DoesntMatchRegex { value: _, regex: _ }
));
assert_eq!(
got.unwrap_err().to_string(),
"'127.0.0.1' doesn't match regex '^([ip]|[hostname]/)?[tag_name]:[tag_version]?$'"
);
}
#[test]
fn converting_a_str_containing_only_ip_and_tag_version_into_an_image_should_fails() {
let got: Result<Image, ConversionError> = "127.0.0.1:version".try_into();
assert!(matches!(
got.clone().unwrap_err(),
ConversionError::DoesntMatchRegex { value: _, regex: _ }
));
assert_eq!(got.unwrap_err().to_string(), "'127.0.0.1:version' doesn't match regex '^([ip]|[hostname]/)?[tag_name]:[tag_version]?$'");
}
#[test]
fn converting_a_str_containing_only_hostname_into_an_image_should_fails() {
let got: Result<Image, ConversionError> = "myrepository.com".try_into();
assert!(matches!(
got.clone().unwrap_err(),
ConversionError::DoesntMatchRegex { value: _, regex: _ }
));
assert_eq!(got.unwrap_err().to_string(), "'myrepository.com' doesn't match regex '^([ip]|[hostname]/)?[tag_name]:[tag_version]?$'");
}
#[test]
fn converting_a_str_containing_only_hostname_and_tag_version_into_an_image_should_fails() {
let got: Result<Image, ConversionError> = "myrepository.com:version".try_into();
assert!(matches!(
got.clone().unwrap_err(),
ConversionError::DoesntMatchRegex { value: _, regex: _ }
));
assert_eq!(got.unwrap_err().to_string(), "'myrepository.com:version' doesn't match regex '^([ip]|[hostname]/)?[tag_name]:[tag_version]?$'");
}
#[test]
fn converting_a_str_with_whitespaces_into_a_command_should_fails() {
let got: Result<Command, ConversionError> = "my command".try_into();
assert!(matches!(
got.clone().unwrap_err(),
ConversionError::ContainsWhitespaces(_)
));
assert_eq!(
got.unwrap_err().to_string(),
"'my command' shouldn't contains whitespace"
);
}
#[test]
fn test_convert_to_json_overrides() {
let url: AssetLocation = "https://example.com/overrides.json".into();
assert!(matches!(
url.into(),
JsonOverrides::Location(AssetLocation::Url(_))
));
let path: AssetLocation = "/path/to/overrides.json".into();
assert!(matches!(
path.into(),
JsonOverrides::Location(AssetLocation::FilePath(_))
));
let inline = serde_json::json!({ "para_id": 2000});
assert!(matches!(
inline.into(),
JsonOverrides::Json(serde_json::Value::Object(_))
));
}
}
vendor/pezkuwi-zombienet-sdk/crates/configuration/src/teyrchain.rs
Vendored
+1873
View File
File diff suppressed because it is too large Load Diff
vendor/pezkuwi-zombienet-sdk/crates/configuration/src/utils.rs
Vendored
+65
View File
@@ -0,0 +1,65 @@
use std::env;
use support::constants::ZOMBIE_NODE_SPAWN_TIMEOUT_SECONDS;
use crate::types::{Chain, Command, Duration};
pub(crate) fn is_true(value: &bool) -> bool {
*value
}
pub(crate) fn is_false(value: &bool) -> bool {
!(*value)
}
pub(crate) fn default_as_true() -> bool {
true
}
pub(crate) fn default_as_false() -> bool {
false
}
pub(crate) fn default_initial_balance() -> crate::types::U128 {
2_000_000_000_000.into()
}
/// Default timeout for spawning a node (10mins)
pub(crate) fn default_node_spawn_timeout() -> Duration {
env::var(ZOMBIE_NODE_SPAWN_TIMEOUT_SECONDS)
.ok()
.and_then(|s| s.parse::<u32>().ok())
.unwrap_or(600)
}
/// Default timeout for spawning the whole network (1hr)
pub(crate) fn default_timeout() -> Duration {
3600
}
pub(crate) fn default_command_polkadot() -> Option<Command> {
TryInto::<Command>::try_into("polkadot").ok()
}
pub(crate) fn default_relaychain_chain() -> Chain {
TryInto::<Chain>::try_into("rococo-local").expect("'rococo-local' should be a valid chain")
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn default_node_spawn_timeout_works_before_and_after_env_is_set() {
// The default should be 600 seconds if the env var is not set
assert_eq!(default_node_spawn_timeout(), 600);
// If env var is set to a valid number, it should return that number
env::set_var(ZOMBIE_NODE_SPAWN_TIMEOUT_SECONDS, "123");
assert_eq!(default_node_spawn_timeout(), 123);
// If env var is set to a NOT valid number, it should return 600
env::set_var(ZOMBIE_NODE_SPAWN_TIMEOUT_SECONDS, "NOT_A_NUMBER");
assert_eq!(default_node_spawn_timeout(), 600);
}
}
vendor/pezkuwi-zombienet-sdk/crates/configuration/testing/snapshots/0000-small-network.toml
Vendored
+25
View File
@@ -0,0 +1,25 @@
[settings]
timeout = 3600
node_spawn_timeout = 600
tear_down_on_failure = true
[relaychain]
chain = "rococo-local"
default_command = "polkadot"
default_image = "docker.io/parity/polkadot:latest"
default_args = ["-lparachain=debug"]
[[relaychain.nodes]]
name = "alice"
validator = true
invulnerable = true
bootnode = false
balance = 2000000000000
[[relaychain.nodes]]
name = "bob"
args = ["--database=paritydb-experimental"]
validator = true
invulnerable = false
bootnode = true
balance = 2000000000000
vendor/pezkuwi-zombienet-sdk/crates/configuration/testing/snapshots/0001-big-network.toml
Vendored
+105
View File
@@ -0,0 +1,105 @@
[settings]
timeout = 3600
node_spawn_timeout = 600
tear_down_on_failure = true
[relaychain]
chain = "polkadot"
default_command = "polkadot"
default_image = "docker.io/parity/polkadot:latest"
[relaychain.default_resources.requests]
memory = "500M"
cpu = "100000"
[relaychain.default_resources.limits]
memory = "4000M"
cpu = "10Gi"
[[relaychain.nodes]]
name = "alice"
validator = true
invulnerable = true
bootnode = true
balance = 1000000000
[[relaychain.nodes]]
name = "bob"
validator = true
invulnerable = true
bootnode = true
balance = 2000000000000
[[teyrchains]]
id = 1000
chain = "myparachain"
register_para = true
onboard_as_teyrchain = false
balance = 2000000000000
default_db_snapshot = "https://storage.com/path/to/db_snapshot.tgz"
chain_spec_path = "/path/to/my/chain/spec.json"
cumulus_based = true
evm_based = false
[[teyrchains.collators]]
name = "john"
validator = true
invulnerable = true
bootnode = true
balance = 5000000000
[[teyrchains.collators]]
name = "charles"
validator = false
invulnerable = true
bootnode = true
balance = 0
[[teyrchains.collators]]
name = "frank"
validator = true
invulnerable = false
bootnode = true
balance = 1000000000
[[teyrchains]]
id = 2000
chain = "myotherparachain"
add_to_genesis = true
balance = 2000000000000
chain_spec_path = "/path/to/my/other/chain/spec.json"
cumulus_based = true
evm_based = false
[[teyrchains.collators]]
name = "mike"
validator = true
invulnerable = true
bootnode = true
balance = 5000000000
[[teyrchains.collators]]
name = "georges"
validator = false
invulnerable = true
bootnode = true
balance = 0
[[teyrchains.collators]]
name = "victor"
validator = true
invulnerable = false
bootnode = true
balance = 1000000000
[[hrmp_channels]]
sender = 1000
recipient = 2000
max_capacity = 150
max_message_size = 5000
[[hrmp_channels]]
sender = 2000
recipient = 1000
max_capacity = 200
max_message_size = 8000
vendor/pezkuwi-zombienet-sdk/crates/configuration/testing/snapshots/0002-overridden-defaults.toml
Vendored
+76
View File
@@ -0,0 +1,76 @@
[settings]
timeout = 3600
node_spawn_timeout = 600
tear_down_on_failure = true
[relaychain]
chain = "polkadot"
default_command = "polkadot"
default_image = "docker.io/parity/polkadot:latest"
default_db_snapshot = "https://storage.com/path/to/db_snapshot.tgz"
default_args = [
"-name=value",
"--flag",
]
[relaychain.default_resources.requests]
memory = "500M"
cpu = "100000"
[relaychain.default_resources.limits]
memory = "4000M"
cpu = "10Gi"
[[relaychain.nodes]]
name = "alice"
validator = true
invulnerable = true
bootnode = true
balance = 1000000000
[[relaychain.nodes]]
name = "bob"
image = "mycustomimage:latest"
command = "my-custom-command"
args = ["-myothername=value"]
validator = true
invulnerable = true
bootnode = true
balance = 2000000000000
db_snapshot = "https://storage.com/path/to/other/db_snapshot.tgz"
[relaychain.nodes.resources.requests]
memory = "250Mi"
cpu = "1000"
[relaychain.nodes.resources.limits]
memory = "2Gi"
cpu = "5Gi"
[[teyrchains]]
id = 1000
chain = "myparachain"
add_to_genesis = true
balance = 2000000000000
default_command = "my-default-command"
default_image = "mydefaultimage:latest"
default_db_snapshot = "https://storage.com/path/to/other_snapshot.tgz"
chain_spec_path = "/path/to/my/chain/spec.json"
cumulus_based = true
evm_based = false
[[teyrchains.collators]]
name = "john"
image = "anotherimage:latest"
command = "my-non-default-command"
validator = true
invulnerable = true
bootnode = true
balance = 5000000000
[[teyrchains.collators]]
name = "charles"
validator = false
invulnerable = true
bootnode = true
balance = 0
vendor/pezkuwi-zombienet-sdk/crates/configuration/testing/snapshots/0003-small-network_w_teyrchain.toml
Vendored
+57
View File
@@ -0,0 +1,57 @@
[settings]
timeout = 3600
node_spawn_timeout = 600
[relaychain]
chain = "rococo-local"
default_command = "polkadot"
default_image = "docker.io/parity/polkadot:latest"
default_args = ["-lparachain=debug"]
[[relaychain.nodes]]
name = "alice"
validator = true
invulnerable = true
bootnode = false
balance = 2000000000000
[[relaychain.nodes]]
name = "bob"
args = ["--database=paritydb-experimental"]
validator = true
invulnerable = false
bootnode = true
balance = 2000000000000
[[teyrchains]]
id = 1000
chain = "myparachain"
onboard_as_teyrchain = false
balance = 2000000000000
default_db_snapshot = "https://storage.com/path/to/db_snapshot.tgz"
chain_spec_path = "/path/to/my/chain/spec.json"
cumulus_based = true
[teyrchains.collator]
name = "john"
validator = true
invulnerable = true
bootnode = true
balance = 5000000000
[[teyrchains]]
id = 1000
chain = "myparachain"
onboard_as_teyrchain = false
balance = 2000000000000
default_db_snapshot = "https://storage.com/path/to/db_snapshot.tgz"
chain_spec_path = "/path/to/my/chain/spec.json"
cumulus_based = true
evm_based = true
[[teyrchains.collators]]
name = "john"
validator = true
invulnerable = true
bootnode = true
balance = 5000000000
vendor/pezkuwi-zombienet-sdk/crates/configuration/testing/snapshots/0004-small-network-without-settings.toml
Vendored
+9
View File
@@ -0,0 +1,9 @@
[relaychain]
chain = "rococo-local"
default_command = "polkadot"
[[relaychain.nodes]]
name = "alice"
[[relaychain.nodes]]
name = "bob"
vendor/pezkuwi-zombienet-sdk/crates/configuration/testing/snapshots/0005-small-networl-with-wasm-override.toml
Vendored
+17
View File
@@ -0,0 +1,17 @@
[relaychain]
chain = "rococo-local"
default_command = "polkadot"
wasm_override = "/some/path/runtime.wasm"
[[relaychain.nodes]]
name = "alice"
[[relaychain.nodes]]
name = "bob"
[[teyrchains]]
id = 1000
wasm_override = "https://some.com/runtime.wasm"
[teyrchains.collator]
name = "john"
vendor/pezkuwi-zombienet-sdk/crates/configuration/testing/snapshots/0006-without-rc-chain-name.toml
Vendored
+27
View File
@@ -0,0 +1,27 @@
[relaychain]
default_command = "polkadot"
chain_spec_path = "./rc.json"
[[relaychain.nodes]]
name = "alice"
validator = true
rpc_port = 9944
[[relaychain.nodes]]
name = "bob"
validator = true
rpc_port = 9945
args = [
"-lruntime::system=debug,runtime::session=trace,runtime::staking::ah-client=trace,runtime::ah-client=debug",
]
[[teyrchains]]
id = 1100
chain_spec_path = "./parachain.json"
[teyrchains.collator]
name = "charlie"
rpc_port = 9946
args = [
"-lruntime::system=debug,runtime::multiblock-election=trace,runtime::staking=debug,runtime::staking::rc-client=trace,runtime::rc-client=debug",
]
vendor/pezkuwi-zombienet-sdk/crates/configuration/testing/snapshots/0007-small-network_w_teyrchain_w_duplicate_node_names.toml
Vendored
+40
View File
@@ -0,0 +1,40 @@
[settings]
timeout = 3600
node_spawn_timeout = 600
[relaychain]
chain = "rococo-local"
default_command = "polkadot"
default_image = "docker.io/parity/polkadot:latest"
default_args = ["-lparachain=debug"]
[[relaychain.nodes]]
name = "alice"
validator = true
invulnerable = true
bootnode = false
balance = 2000000000000
[[relaychain.nodes]]
name = "bob"
args = ["--database=paritydb-experimental"]
validator = true
invulnerable = false
bootnode = true
balance = 2000000000000
[[teyrchains]]
id = 1000
chain = "myparachain"
onboard_as_teyrchain = false
balance = 2000000000000
default_db_snapshot = "https://storage.com/path/to/db_snapshot.tgz"
chain_spec_path = "/path/to/my/chain/spec.json"
cumulus_based = true
[teyrchains.collator]
name = "alice"
validator = true
invulnerable = true
bootnode = true
balance = 5000000000
vendor/pezkuwi-zombienet-sdk/crates/configuration/testing/snapshots/0008-small-network-with-raw-spec-override.toml
Vendored
+17
View File
@@ -0,0 +1,17 @@
[relaychain]
chain = "rococo-local"
default_command = "polkadot"
raw_spec_override = "/some/path/raw_spec_override.json"
[[relaychain.nodes]]
name = "alice"
[[relaychain.nodes]]
name = "bob"
[[teyrchains]]
id = 1000
raw_spec_override = "https://some.com/raw_spec_override.json"
[teyrchains.collator]
name = "john"
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/.gitignore
+2
View File
@@ -0,0 +1,2 @@
/target
/Cargo.lock
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/Cargo.toml
Vendored
+54
View File
@@ -0,0 +1,54 @@
[package]
name = "zombienet-orchestrator"
version.workspace = true
authors.workspace = true
edition.workspace = true
rust-version.workspace = true
publish = true
license.workspace = true
repository.workspace = true
description = "Zombienet Orchestrator, drive network spwan through providers"
keywords = ["zombienet", "orchestrator", "sdk"]
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
tokio = { workspace = true, features = ["time"] }
thiserror = { workspace = true }
multiaddr = { workspace = true }
serde_json = { workspace = true, features = ["arbitrary_precision"] }
futures = { workspace = true }
anyhow = { workspace = true }
rand = { workspace = true }
sha2 = { workspace = true, default-features = false }
hex = { workspace = true }
sp-core = { workspace = true }
libp2p = { workspace = true }
pezkuwi-subxt = { workspace = true }
pezkuwi-subxt-signer = { workspace = true }
reqwest = { workspace = true }
tracing = { workspace = true }
uuid = { workspace = true }
regex = { workspace = true }
glob-match = { workspace = true }
async-trait = { workspace = true }
serde = { workspace = true, features = ["derive"] }
libsecp256k1 = { workspace = true }
fancy-regex = { workspace = true }
# staging-chain-spec-builder = { workspace = true }
# parity-scale-codec = { version = "3.7.5", features = ["derive"] }
# sc-chain-spec = {workspace = true, default-features = false}
sc-chain-spec = { workspace = true }
erased-serde = { workspace = true }
# Zombienet deps
configuration = { workspace = true }
support = { workspace = true }
provider = { workspace = true }
prom-metrics-parser = { workspace = true }
[dev-dependencies]
toml = { workspace = true }
async-trait = { workspace = true }
lazy_static = { workspace = true }
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/errors.rs
Vendored
+31
View File
@@ -0,0 +1,31 @@
//! Zombienet Orchestrator error definitions.
use provider::ProviderError;
use support::fs::FileSystemError;
use crate::generators;
#[derive(Debug, thiserror::Error)]
pub enum OrchestratorError {
// TODO: improve invalid config reporting
#[error("Invalid network configuration: {0}")]
InvalidConfig(String),
#[error("Invalid network config to use provider {0}: {1}")]
InvalidConfigForProvider(String, String),
#[error("Invalid configuration for node: {0}, field: {1}")]
InvalidNodeConfig(String, String),
#[error("Invariant not fulfilled {0}")]
InvariantError(&'static str),
#[error("Global network spawn timeout: {0} secs")]
GlobalTimeOut(u32),
#[error("Generator error: {0}")]
GeneratorError(#[from] generators::errors::GeneratorError),
#[error("Provider error")]
ProviderError(#[from] ProviderError),
#[error("FileSystem error")]
FileSystemError(#[from] FileSystemError),
#[error("Serialization error")]
SerializationError(#[from] serde_json::Error),
#[error(transparent)]
SpawnerError(#[from] anyhow::Error),
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/generators.rs
Vendored
+22
View File
@@ -0,0 +1,22 @@
pub mod chain_spec;
pub mod errors;
pub mod key;
pub mod para_artifact;
mod arg_filter;
mod bootnode_addr;
mod command;
mod identity;
mod keystore;
mod keystore_key_types;
mod port;
pub use bootnode_addr::generate as generate_node_bootnode_addr;
pub use command::{
generate_for_cumulus_node as generate_node_command_cumulus,
generate_for_node as generate_node_command, GenCmdOptions,
};
pub use identity::generate as generate_node_identity;
pub use key::generate as generate_node_keys;
pub use keystore::generate as generate_node_keystore;
pub use port::generate as generate_node_port;
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/generators/arg_filter.rs
Vendored
+138
View File
@@ -0,0 +1,138 @@
use configuration::types::Arg;
/// Parse args to extract those marked for removal (with `-:` prefix).
/// Returns a set of arg names/flags that should be removed from the final command.
///
/// # Examples
/// - `-:--insecure-validator-i-know-what-i-do` -> removes `--insecure-validator-i-know-what-i-do`
/// - `-:insecure-validator` -> removes `--insecure-validator` (normalized)
/// - `-:--prometheus-port` -> removes `--prometheus-port`
pub fn parse_removal_args(args: &[Arg]) -> Vec<String> {
args.iter()
.filter_map(|arg| match arg {
Arg::Flag(flag) if flag.starts_with("-:") => {
let mut flag_to_exclude = flag[2..].to_string();
// Normalize flag format - ensure it starts with --
if !flag_to_exclude.starts_with("--") {
flag_to_exclude = format!("--{flag_to_exclude}");
}
Some(flag_to_exclude)
},
_ => None,
})
.collect()
}
/// Apply arg removals to a vector of string arguments.
/// This filters out any args that match the removal list.
///
/// # Arguments
/// * `args` - The command arguments to filter
/// * `removals` - List of arg names/flags to remove
///
/// # Returns
/// Filtered vector with specified args removed
pub fn apply_arg_removals(args: Vec<String>, removals: &[String]) -> Vec<String> {
if removals.is_empty() {
return args;
}
let mut res = Vec::new();
let mut skip_next = false;
for (i, arg) in args.iter().enumerate() {
if skip_next {
skip_next = false;
continue;
}
let should_remove = removals
.iter()
.any(|removal| arg == removal || arg.starts_with(&format!("{removal}=")));
if should_remove {
// Only skip next if this looks like an option (starts with --) and next arg doesn't start with --
if !arg.contains("=") && i + 1 < args.len() {
let next_arg = &args[i + 1];
if !next_arg.starts_with("-") {
skip_next = true;
}
}
continue;
}
res.push(arg.clone());
}
res
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_parse_removal_args() {
let args = vec![
Arg::Flag("-:--insecure-validator-i-know-what-i-do".to_string()),
Arg::Flag("--validator".to_string()),
Arg::Flag("-:--no-telemetry".to_string()),
];
let removals = parse_removal_args(&args);
assert_eq!(removals.len(), 2);
assert!(removals.contains(&"--insecure-validator-i-know-what-i-do".to_string()));
assert!(removals.contains(&"--no-telemetry".to_string()));
}
#[test]
fn test_apply_arg_removals_flag() {
let args = vec![
"--validator".to_string(),
"--insecure-validator-i-know-what-i-do".to_string(),
"--no-telemetry".to_string(),
];
let removals = vec!["--insecure-validator-i-know-what-i-do".to_string()];
let res = apply_arg_removals(args, &removals);
assert_eq!(res.len(), 2);
assert!(res.contains(&"--validator".to_string()));
assert!(res.contains(&"--no-telemetry".to_string()));
assert!(!res.contains(&"--insecure-validator-i-know-what-i-do".to_string()));
}
#[test]
fn test_apply_arg_removals_option_with_equals() {
let args = vec!["--name=alice".to_string(), "--port=30333".to_string()];
let removals = vec!["--port".to_string()];
let res = apply_arg_removals(args, &removals);
assert_eq!(res.len(), 1);
assert_eq!(res[0], "--name=alice");
}
#[test]
fn test_apply_arg_removals_option_with_space() {
let args = vec![
"--name".to_string(),
"alice".to_string(),
"--port".to_string(),
"30333".to_string(),
];
let removals = vec!["--port".to_string()];
let res = apply_arg_removals(args, &removals);
assert_eq!(res.len(), 2);
assert_eq!(res[0], "--name");
assert_eq!(res[1], "alice");
}
#[test]
fn test_apply_arg_removals_empty() {
let args = vec!["--validator".to_string()];
let removals = vec![];
let res = apply_arg_removals(args, &removals);
assert_eq!(res, vec!["--validator".to_string()]);
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/generators/bootnode_addr.rs
Vendored
+111
View File
@@ -0,0 +1,111 @@
use std::{fmt::Display, net::IpAddr};
use super::errors::GeneratorError;
pub fn generate<T: AsRef<str> + Display>(
peer_id: &str,
ip: &IpAddr,
port: u16,
args: &[T],
p2p_cert: &Option<String>,
) -> Result<String, GeneratorError> {
let addr = if let Some(index) = args.iter().position(|arg| arg.as_ref().eq("--listen-addr")) {
let listen_value = args
.as_ref()
.get(index + 1)
.ok_or(GeneratorError::BootnodeAddrGeneration(
"can not generate bootnode address from args".into(),
))?
.to_string();
let ip_str = ip.to_string();
let port_str = port.to_string();
let mut parts = listen_value.split('/').collect::<Vec<&str>>();
parts[2] = &ip_str;
parts[4] = port_str.as_str();
parts.join("/")
} else {
format!("/ip4/{ip}/tcp/{port}/ws")
};
let mut addr_with_peer = format!("{addr}/p2p/{peer_id}");
if let Some(p2p_cert) = p2p_cert {
addr_with_peer.push_str("/certhash/");
addr_with_peer.push_str(p2p_cert)
}
Ok(addr_with_peer)
}
#[cfg(test)]
mod tests {
use provider::constants::LOCALHOST;
use super::*;
#[test]
fn generate_for_alice_without_args() {
let peer_id = "12D3KooWQCkBm1BYtkHpocxCwMgR8yjitEeHGx8spzcDLGt2gkBm"; // from alice as seed
let args: Vec<&str> = vec![];
let bootnode_addr = generate(peer_id, &LOCALHOST, 5678, &args, &None).unwrap();
assert_eq!(
&bootnode_addr,
"/ip4/127.0.0.1/tcp/5678/ws/p2p/12D3KooWQCkBm1BYtkHpocxCwMgR8yjitEeHGx8spzcDLGt2gkBm"
);
}
#[test]
fn generate_for_alice_with_listen_addr() {
// Should override the ip/port
let peer_id = "12D3KooWQCkBm1BYtkHpocxCwMgR8yjitEeHGx8spzcDLGt2gkBm"; // from alice as seed
let args: Vec<String> = [
"--some",
"other",
"--listen-addr",
"/ip4/192.168.100.1/tcp/30333/ws",
]
.iter()
.map(|x| x.to_string())
.collect();
let bootnode_addr =
generate(peer_id, &LOCALHOST, 5678, args.iter().as_ref(), &None).unwrap();
assert_eq!(
&bootnode_addr,
"/ip4/127.0.0.1/tcp/5678/ws/p2p/12D3KooWQCkBm1BYtkHpocxCwMgR8yjitEeHGx8spzcDLGt2gkBm"
);
}
#[test]
fn generate_for_alice_with_listen_addr_without_value_must_fail() {
// Should override the ip/port
let peer_id = "12D3KooWQCkBm1BYtkHpocxCwMgR8yjitEeHGx8spzcDLGt2gkBm"; // from alice as seed
let args: Vec<String> = ["--some", "other", "--listen-addr"]
.iter()
.map(|x| x.to_string())
.collect();
let bootnode_addr = generate(peer_id, &LOCALHOST, 5678, args.iter().as_ref(), &None);
assert!(bootnode_addr.is_err());
assert!(matches!(
bootnode_addr,
Err(GeneratorError::BootnodeAddrGeneration(_))
));
}
#[test]
fn generate_for_alice_withcert() {
let peer_id = "12D3KooWQCkBm1BYtkHpocxCwMgR8yjitEeHGx8spzcDLGt2gkBm"; // from alice as seed
let args: Vec<&str> = vec![];
let bootnode_addr = generate(
peer_id,
&LOCALHOST,
5678,
&args,
&Some(String::from("data")),
)
.unwrap();
assert_eq!(
&bootnode_addr,
"/ip4/127.0.0.1/tcp/5678/ws/p2p/12D3KooWQCkBm1BYtkHpocxCwMgR8yjitEeHGx8spzcDLGt2gkBm/certhash/data"
);
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/generators/chain_spec.rs
Vendored
+2073
View File
File diff suppressed because it is too large Load Diff
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/generators/command.rs
Vendored
+634
View File
@@ -0,0 +1,634 @@
use configuration::types::Arg;
use support::constants::THIS_IS_A_BUG;
use super::arg_filter::{apply_arg_removals, parse_removal_args};
use crate::{network_spec::node::NodeSpec, shared::constants::*};
pub struct GenCmdOptions<'a> {
pub relay_chain_name: &'a str,
pub cfg_path: &'a str,
pub data_path: &'a str,
pub relay_data_path: &'a str,
pub use_wrapper: bool,
pub bootnode_addr: Vec<String>,
pub use_default_ports_in_cmd: bool,
pub is_native: bool,
}
impl Default for GenCmdOptions<'_> {
fn default() -> Self {
Self {
relay_chain_name: "rococo-local",
cfg_path: "/cfg",
data_path: "/data",
relay_data_path: "/relay-data",
use_wrapper: true,
bootnode_addr: vec![],
use_default_ports_in_cmd: false,
is_native: true,
}
}
}
const FLAGS_ADDED_BY_US: [&str; 3] = ["--no-telemetry", "--collator", "--"];
const OPS_ADDED_BY_US: [&str; 6] = [
"--chain",
"--name",
"--rpc-cors",
"--rpc-methods",
"--parachain-id",
"--node-key",
];
// TODO: can we abstract this and use only one fn (or at least split and reuse in small fns)
pub fn generate_for_cumulus_node(
node: &NodeSpec,
options: GenCmdOptions,
para_id: u32,
) -> (String, Vec<String>) {
let NodeSpec {
key,
args,
is_validator,
bootnodes_addresses,
..
} = node;
let mut tmp_args: Vec<String> = vec!["--node-key".into(), key.clone()];
if !args.contains(&Arg::Flag("--prometheus-external".into())) {
tmp_args.push("--prometheus-external".into())
}
if *is_validator && !args.contains(&Arg::Flag("--validator".into())) {
tmp_args.push("--collator".into())
}
if !bootnodes_addresses.is_empty() {
tmp_args.push("--bootnodes".into());
let bootnodes = bootnodes_addresses
.iter()
.map(|m| m.to_string())
.collect::<Vec<String>>()
.join(" ");
tmp_args.push(bootnodes)
}
// ports
let (prometheus_port, rpc_port, p2p_port) =
resolve_ports(node, options.use_default_ports_in_cmd);
tmp_args.push("--prometheus-port".into());
tmp_args.push(prometheus_port.to_string());
tmp_args.push("--rpc-port".into());
tmp_args.push(rpc_port.to_string());
tmp_args.push("--listen-addr".into());
tmp_args.push(format!("/ip4/0.0.0.0/tcp/{p2p_port}/ws"));
let mut collator_args: &[Arg] = &[];
let mut full_node_args: &[Arg] = &[];
if !args.is_empty() {
if let Some(index) = args.iter().position(|arg| match arg {
Arg::Flag(flag) => flag.eq("--"),
Arg::Option(..) => false,
Arg::Array(..) => false,
}) {
(collator_args, full_node_args) = args.split_at(index);
} else {
// Assume args are those specified for collator only
collator_args = args;
}
}
// set our base path
tmp_args.push("--base-path".into());
tmp_args.push(options.data_path.into());
let node_specific_bootnodes: Vec<String> = node
.bootnodes_addresses
.iter()
.map(|b| b.to_string())
.collect();
let full_bootnodes = [node_specific_bootnodes, options.bootnode_addr].concat();
if !full_bootnodes.is_empty() {
tmp_args.push("--bootnodes".into());
tmp_args.push(full_bootnodes.join(" "));
}
let mut full_node_p2p_needs_to_be_injected = true;
let mut full_node_prometheus_needs_to_be_injected = true;
let mut full_node_args_filtered = full_node_args
.iter()
.filter_map(|arg| match arg {
Arg::Flag(flag) => {
if flag.starts_with("-:") || FLAGS_ADDED_BY_US.contains(&flag.as_str()) {
None
} else {
Some(vec![flag.to_owned()])
}
},
Arg::Option(k, v) => {
if OPS_ADDED_BY_US.contains(&k.as_str()) {
None
} else if k.eq(&"port") {
if v.eq(&"30333") {
full_node_p2p_needs_to_be_injected = true;
None
} else {
// non default
full_node_p2p_needs_to_be_injected = false;
Some(vec![k.to_owned(), v.to_owned()])
}
} else if k.eq(&"--prometheus-port") {
if v.eq(&"9616") {
full_node_prometheus_needs_to_be_injected = true;
None
} else {
// non default
full_node_prometheus_needs_to_be_injected = false;
Some(vec![k.to_owned(), v.to_owned()])
}
} else {
Some(vec![k.to_owned(), v.to_owned()])
}
},
Arg::Array(k, v) => {
let mut args = vec![k.to_owned()];
args.extend(v.to_owned());
Some(args)
},
})
.flatten()
.collect::<Vec<String>>();
let full_p2p_port = node
.full_node_p2p_port
.as_ref()
.expect(&format!(
"full node p2p_port should be specifed: {THIS_IS_A_BUG}"
))
.0;
let full_prometheus_port = node
.full_node_prometheus_port
.as_ref()
.expect(&format!(
"full node prometheus_port should be specifed: {THIS_IS_A_BUG}"
))
.0;
// full_node: change p2p port if is the default
if full_node_p2p_needs_to_be_injected {
full_node_args_filtered.push("--port".into());
full_node_args_filtered.push(full_p2p_port.to_string());
}
// full_node: change prometheus port if is the default
if full_node_prometheus_needs_to_be_injected {
full_node_args_filtered.push("--prometheus-port".into());
full_node_args_filtered.push(full_prometheus_port.to_string());
}
let mut args_filtered = collator_args
.iter()
.filter_map(|arg| match arg {
Arg::Flag(flag) => {
if flag.starts_with("-:") || FLAGS_ADDED_BY_US.contains(&flag.as_str()) {
None
} else {
Some(vec![flag.to_owned()])
}
},
Arg::Option(k, v) => {
if OPS_ADDED_BY_US.contains(&k.as_str()) {
None
} else {
Some(vec![k.to_owned(), v.to_owned()])
}
},
Arg::Array(k, v) => {
let mut args = vec![k.to_owned()];
args.extend(v.to_owned());
Some(args)
},
})
.flatten()
.collect::<Vec<String>>();
tmp_args.append(&mut args_filtered);
let parachain_spec_path = format!("{}/{}.json", options.cfg_path, para_id);
let mut final_args = vec![
node.command.as_str().to_string(),
"--chain".into(),
parachain_spec_path,
"--name".into(),
node.name.clone(),
"--rpc-cors".into(),
"all".into(),
"--rpc-methods".into(),
"unsafe".into(),
];
// The `--unsafe-rpc-external` option spawns an additional RPC server on a random port,
// which can conflict with reserved ports, causing an "Address already in use" error
// when using the `native` provider. Since this option isn't needed for `native`,
// it should be omitted in that case.
if !options.is_native {
final_args.push("--unsafe-rpc-external".into());
}
final_args.append(&mut tmp_args);
let relaychain_spec_path = format!("{}/{}.json", options.cfg_path, options.relay_chain_name);
let mut full_node_injected: Vec<String> = vec![
"--".into(),
"--base-path".into(),
options.relay_data_path.into(),
"--chain".into(),
relaychain_spec_path,
"--execution".into(),
"wasm".into(),
];
final_args.append(&mut full_node_injected);
final_args.append(&mut full_node_args_filtered);
let removals = parse_removal_args(args);
final_args = apply_arg_removals(final_args, &removals);
if options.use_wrapper {
("/cfg/zombie-wrapper.sh".to_string(), final_args)
} else {
(final_args.remove(0), final_args)
}
}
pub fn generate_for_node(
node: &NodeSpec,
options: GenCmdOptions,
para_id: Option<u32>,
) -> (String, Vec<String>) {
let NodeSpec {
key,
args,
is_validator,
bootnodes_addresses,
..
} = node;
let mut tmp_args: Vec<String> = vec![
"--node-key".into(),
key.clone(),
// TODO:(team) we should allow to set the telemetry url from config
"--no-telemetry".into(),
];
if !args.contains(&Arg::Flag("--prometheus-external".into())) {
tmp_args.push("--prometheus-external".into())
}
if let Some(para_id) = para_id {
tmp_args.push("--parachain-id".into());
tmp_args.push(para_id.to_string());
}
if *is_validator && !args.contains(&Arg::Flag("--validator".into())) {
tmp_args.push("--validator".into());
if node.supports_arg("--insecure-validator-i-know-what-i-do") {
tmp_args.push("--insecure-validator-i-know-what-i-do".into());
}
}
if !bootnodes_addresses.is_empty() {
tmp_args.push("--bootnodes".into());
let bootnodes = bootnodes_addresses
.iter()
.map(|m| m.to_string())
.collect::<Vec<String>>()
.join(" ");
tmp_args.push(bootnodes)
}
// ports
let (prometheus_port, rpc_port, p2p_port) =
resolve_ports(node, options.use_default_ports_in_cmd);
// Prometheus
tmp_args.push("--prometheus-port".into());
tmp_args.push(prometheus_port.to_string());
// RPC
// TODO (team): do we want to support old --ws-port?
tmp_args.push("--rpc-port".into());
tmp_args.push(rpc_port.to_string());
let listen_value = if let Some(listen_val) = args.iter().find_map(|arg| match arg {
Arg::Flag(_) => None,
Arg::Option(k, v) => {
if k.eq("--listen-addr") {
Some(v)
} else {
None
}
},
Arg::Array(..) => None,
}) {
let mut parts = listen_val.split('/').collect::<Vec<&str>>();
// TODO: move this to error
let port_part = parts
.get_mut(4)
.expect(&format!("should have at least 5 parts {THIS_IS_A_BUG}"));
let port_to_use = p2p_port.to_string();
*port_part = port_to_use.as_str();
parts.join("/")
} else {
format!("/ip4/0.0.0.0/tcp/{p2p_port}/ws")
};
tmp_args.push("--listen-addr".into());
tmp_args.push(listen_value);
// set our base path
tmp_args.push("--base-path".into());
tmp_args.push(options.data_path.into());
let node_specific_bootnodes: Vec<String> = node
.bootnodes_addresses
.iter()
.map(|b| b.to_string())
.collect();
let full_bootnodes = [node_specific_bootnodes, options.bootnode_addr].concat();
if !full_bootnodes.is_empty() {
tmp_args.push("--bootnodes".into());
tmp_args.push(full_bootnodes.join(" "));
}
// add the rest of the args
let mut args_filtered = args
.iter()
.filter_map(|arg| match arg {
Arg::Flag(flag) => {
if flag.starts_with("-:") || FLAGS_ADDED_BY_US.contains(&flag.as_str()) {
None
} else {
Some(vec![flag.to_owned()])
}
},
Arg::Option(k, v) => {
if OPS_ADDED_BY_US.contains(&k.as_str()) {
None
} else {
Some(vec![k.to_owned(), v.to_owned()])
}
},
Arg::Array(k, v) => {
let mut args = vec![k.to_owned()];
args.extend(v.to_owned());
Some(args)
},
})
.flatten()
.collect::<Vec<String>>();
tmp_args.append(&mut args_filtered);
let chain_spec_path = format!("{}/{}.json", options.cfg_path, options.relay_chain_name);
let mut final_args = vec![
node.command.as_str().to_string(),
"--chain".into(),
chain_spec_path,
"--name".into(),
node.name.clone(),
"--rpc-cors".into(),
"all".into(),
"--rpc-methods".into(),
"unsafe".into(),
];
// The `--unsafe-rpc-external` option spawns an additional RPC server on a random port,
// which can conflict with reserved ports, causing an "Address already in use" error
// when using the `native` provider. Since this option isn't needed for `native`,
// it should be omitted in that case.
if !options.is_native {
final_args.push("--unsafe-rpc-external".into());
}
final_args.append(&mut tmp_args);
if let Some(ref subcommand) = node.subcommand {
final_args.insert(1, subcommand.as_str().to_string());
}
let removals = parse_removal_args(args);
final_args = apply_arg_removals(final_args, &removals);
if options.use_wrapper {
("/cfg/zombie-wrapper.sh".to_string(), final_args)
} else {
(final_args.remove(0), final_args)
}
}
/// Returns (prometheus, rpc, p2p) ports to use in the command
fn resolve_ports(node: &NodeSpec, use_default_ports_in_cmd: bool) -> (u16, u16, u16) {
if use_default_ports_in_cmd {
(PROMETHEUS_PORT, RPC_PORT, P2P_PORT)
} else {
(node.prometheus_port.0, node.rpc_port.0, node.p2p_port.0)
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::{generators, shared::types::NodeAccounts};
fn get_node_spec(full_node_present: bool) -> NodeSpec {
let mut name = String::from("luca");
let initial_balance = 1_000_000_000_000_u128;
let seed = format!("//{}{name}", name.remove(0).to_uppercase());
let accounts = NodeAccounts {
accounts: generators::generate_node_keys(&seed).unwrap(),
seed,
};
let (full_node_p2p_port, full_node_prometheus_port) = if full_node_present {
(
Some(generators::generate_node_port(None).unwrap()),
Some(generators::generate_node_port(None).unwrap()),
)
} else {
(None, None)
};
NodeSpec {
name,
accounts,
initial_balance,
full_node_p2p_port,
full_node_prometheus_port,
..Default::default()
}
}
#[test]
fn generate_for_native_cumulus_node_works() {
let node = get_node_spec(true);
let opts = GenCmdOptions {
use_wrapper: false,
is_native: true,
..GenCmdOptions::default()
};
let (program, args) = generate_for_cumulus_node(&node, opts, 1000);
assert_eq!(program.as_str(), "polkadot");
let divider_flag = args.iter().position(|x| x == "--").unwrap();
// ensure full node ports
let i = args[divider_flag..]
.iter()
.position(|x| {
x == node
.full_node_p2p_port
.as_ref()
.unwrap()
.0
.to_string()
.as_str()
})
.unwrap();
assert_eq!(&args[divider_flag + i - 1], "--port");
let i = args[divider_flag..]
.iter()
.position(|x| {
x == node
.full_node_prometheus_port
.as_ref()
.unwrap()
.0
.to_string()
.as_str()
})
.unwrap();
assert_eq!(&args[divider_flag + i - 1], "--prometheus-port");
assert!(!args.iter().any(|arg| arg == "--unsafe-rpc-external"));
}
#[test]
fn generate_for_native_cumulus_node_rpc_external_is_not_removed_if_is_set_by_user() {
let mut node = get_node_spec(true);
node.args.push("--unsafe-rpc-external".into());
let opts = GenCmdOptions {
use_wrapper: false,
is_native: true,
..GenCmdOptions::default()
};
let (_, args) = generate_for_cumulus_node(&node, opts, 1000);
assert!(args.iter().any(|arg| arg == "--unsafe-rpc-external"));
}
#[test]
fn generate_for_non_native_cumulus_node_works() {
let node = get_node_spec(true);
let opts = GenCmdOptions {
use_wrapper: false,
is_native: false,
..GenCmdOptions::default()
};
let (program, args) = generate_for_cumulus_node(&node, opts, 1000);
assert_eq!(program.as_str(), "polkadot");
let divider_flag = args.iter().position(|x| x == "--").unwrap();
// ensure full node ports
let i = args[divider_flag..]
.iter()
.position(|x| {
x == node
.full_node_p2p_port
.as_ref()
.unwrap()
.0
.to_string()
.as_str()
})
.unwrap();
assert_eq!(&args[divider_flag + i - 1], "--port");
let i = args[divider_flag..]
.iter()
.position(|x| {
x == node
.full_node_prometheus_port
.as_ref()
.unwrap()
.0
.to_string()
.as_str()
})
.unwrap();
assert_eq!(&args[divider_flag + i - 1], "--prometheus-port");
// we expect to find this arg in collator node part
assert!(&args[0..divider_flag]
.iter()
.any(|arg| arg == "--unsafe-rpc-external"));
}
#[test]
fn generate_for_native_node_rpc_external_works() {
let node = get_node_spec(false);
let opts = GenCmdOptions {
use_wrapper: false,
is_native: true,
..GenCmdOptions::default()
};
let (program, args) = generate_for_node(&node, opts, Some(1000));
assert_eq!(program.as_str(), "polkadot");
assert!(!args.iter().any(|arg| arg == "--unsafe-rpc-external"));
}
#[test]
fn generate_for_non_native_node_rpc_external_works() {
let node = get_node_spec(false);
let opts = GenCmdOptions {
use_wrapper: false,
is_native: false,
..GenCmdOptions::default()
};
let (program, args) = generate_for_node(&node, opts, Some(1000));
assert_eq!(program.as_str(), "polkadot");
assert!(args.iter().any(|arg| arg == "--unsafe-rpc-external"));
}
#[test]
fn test_arg_removal_removes_insecure_validator_flag() {
let mut node = get_node_spec(false);
node.args
.push(Arg::Flag("-:--insecure-validator-i-know-what-i-do".into()));
node.is_validator = true;
node.available_args_output = Some("--insecure-validator-i-know-what-i-do".to_string());
let opts = GenCmdOptions {
use_wrapper: false,
is_native: true,
..GenCmdOptions::default()
};
let (program, args) = generate_for_node(&node, opts, Some(1000));
assert_eq!(program.as_str(), "polkadot");
assert!(args.iter().any(|arg| arg == "--validator"));
assert!(!args
.iter()
.any(|arg| arg == "--insecure-validator-i-know-what-i-do"));
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/generators/errors.rs
Vendored
+24
View File
@@ -0,0 +1,24 @@
use provider::ProviderError;
use support::fs::FileSystemError;
#[derive(Debug, thiserror::Error)]
pub enum GeneratorError {
#[error("Generating key {0} with input {1}")]
KeyGeneration(String, String),
#[error("Generating port {0}, err {1}")]
PortGeneration(u16, String),
#[error("Chain-spec build error: {0}")]
ChainSpecGeneration(String),
#[error("Provider error: {0}")]
ProviderError(#[from] ProviderError),
#[error("FileSystem error")]
FileSystemError(#[from] FileSystemError),
#[error("Generating identity, err {0}")]
IdentityGeneration(String),
#[error("Generating bootnode address, err {0}")]
BootnodeAddrGeneration(String),
#[error("Error overriding wasm on raw chain-spec, err {0}")]
OverridingWasm(String),
#[error("Error overriding raw chain-spec, err {0}")]
OverridingRawSpec(String),
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/generators/identity.rs
Vendored
+41
View File
@@ -0,0 +1,41 @@
use hex::FromHex;
use libp2p::identity::{ed25519, Keypair};
use sha2::digest::Digest;
use super::errors::GeneratorError;
// Generate p2p identity for node
// return `node-key` and `peerId`
pub fn generate(node_name: &str) -> Result<(String, String), GeneratorError> {
let key = hex::encode(sha2::Sha256::digest(node_name));
let bytes = <[u8; 32]>::from_hex(key.clone()).map_err(|_| {
GeneratorError::IdentityGeneration("can not transform hex to [u8;32]".into())
})?;
let sk = ed25519::SecretKey::try_from_bytes(bytes)
.map_err(|_| GeneratorError::IdentityGeneration("can not create sk from bytes".into()))?;
let local_identity: Keypair = ed25519::Keypair::from(sk).into();
let local_public = local_identity.public();
let local_peer_id = local_public.to_peer_id();
Ok((key, local_peer_id.to_base58()))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn generate_for_alice() {
let s = "alice";
let (key, peer_id) = generate(s).unwrap();
assert_eq!(
&key,
"2bd806c97f0e00af1a1fc3328fa763a9269723c8db8fac4f93af71db186d6e90"
);
assert_eq!(
&peer_id,
"12D3KooWQCkBm1BYtkHpocxCwMgR8yjitEeHGx8spzcDLGt2gkBm"
);
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/generators/key.rs
Vendored
+151
View File
@@ -0,0 +1,151 @@
use sp_core::{crypto::SecretStringError, ecdsa, ed25519, keccak_256, sr25519, Pair, H160, H256};
use super::errors::GeneratorError;
use crate::shared::types::{Accounts, NodeAccount};
const KEYS: [&str; 5] = ["sr", "sr_stash", "ed", "ec", "eth"];
pub fn generate_pair<T: Pair>(seed: &str) -> Result<T::Pair, SecretStringError> {
let pair = T::Pair::from_string(seed, None)?;
Ok(pair)
}
pub fn generate(seed: &str) -> Result<Accounts, GeneratorError> {
let mut accounts: Accounts = Default::default();
for k in KEYS {
let (address, public_key) = match k {
"sr" => {
let pair = generate_pair::<sr25519::Pair>(seed)
.map_err(|_| GeneratorError::KeyGeneration(k.into(), seed.into()))?;
(pair.public().to_string(), hex::encode(pair.public()))
},
"sr_stash" => {
let pair = generate_pair::<sr25519::Pair>(&format!("{seed}//stash"))
.map_err(|_| GeneratorError::KeyGeneration(k.into(), seed.into()))?;
(pair.public().to_string(), hex::encode(pair.public()))
},
"ed" => {
let pair = generate_pair::<ed25519::Pair>(seed)
.map_err(|_| GeneratorError::KeyGeneration(k.into(), seed.into()))?;
(pair.public().to_string(), hex::encode(pair.public()))
},
"ec" => {
let pair = generate_pair::<ecdsa::Pair>(seed)
.map_err(|_| GeneratorError::KeyGeneration(k.into(), seed.into()))?;
(pair.public().to_string(), hex::encode(pair.public()))
},
"eth" => {
let pair = generate_pair::<ecdsa::Pair>(seed)
.map_err(|_| GeneratorError::KeyGeneration(k.into(), seed.into()))?;
let decompressed = libsecp256k1::PublicKey::parse_compressed(&pair.public().0)
.map_err(|_| GeneratorError::KeyGeneration(k.into(), seed.into()))?
.serialize();
let mut m = [0u8; 64];
m.copy_from_slice(&decompressed[1..65]);
let account = H160::from(H256::from(keccak_256(&m)));
(hex::encode(account), hex::encode(account))
},
_ => unreachable!(),
};
accounts.insert(k.into(), NodeAccount::new(address, public_key));
}
Ok(accounts)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn generate_for_alice() {
use sp_core::crypto::Ss58Codec;
let s = "Alice";
let seed = format!("//{s}");
let pair = generate_pair::<sr25519::Pair>(&seed).unwrap();
assert_eq!(
"5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
pair.public().to_ss58check()
);
let pair = generate_pair::<ecdsa::Pair>(&seed).unwrap();
assert_eq!(
"0x020a1091341fe5664bfa1782d5e04779689068c916b04cb365ec3153755684d9a1",
format!("0x{}", hex::encode(pair.public()))
);
let pair = generate_pair::<ed25519::Pair>(&seed).unwrap();
assert_eq!(
"5FA9nQDVg267DEd8m1ZypXLBnvN7SFxYwV7ndqSYGiN9TTpu",
pair.public().to_ss58check()
);
}
#[test]
fn generate_for_zombie() {
use sp_core::crypto::Ss58Codec;
let s = "Zombie";
let seed = format!("//{s}");
let pair = generate_pair::<sr25519::Pair>(&seed).unwrap();
assert_eq!(
"5FTcLfwFc7ctvqp3RhbEig6UuHLHcHVRujuUm8r21wy4dAR8",
pair.public().to_ss58check()
);
}
#[test]
fn generate_pair_invalid_should_fail() {
let s = "Alice";
let seed = s.to_string();
let pair = generate_pair::<sr25519::Pair>(&seed);
assert!(pair.is_err());
}
#[test]
fn generate_invalid_should_fail() {
let s = "Alice";
let seed = s.to_string();
let pair = generate(&seed);
assert!(pair.is_err());
assert!(matches!(pair, Err(GeneratorError::KeyGeneration(_, _))));
}
#[test]
fn generate_work() {
let s = "Alice";
let seed = format!("//{s}");
let pair = generate(&seed).unwrap();
let sr = pair.get("sr").unwrap();
let sr_stash = pair.get("sr_stash").unwrap();
let ed = pair.get("ed").unwrap();
let ec = pair.get("ec").unwrap();
let eth = pair.get("eth").unwrap();
assert_eq!(
sr.address,
"5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY"
);
assert_eq!(
sr_stash.address,
"5GNJqTPyNqANBkUVMN1LPPrxXnFouWXoe2wNSmmEoLctxiZY"
);
assert_eq!(
ed.address,
"5FA9nQDVg267DEd8m1ZypXLBnvN7SFxYwV7ndqSYGiN9TTpu"
);
assert_eq!(
format!("0x{}", ec.public_key),
"0x020a1091341fe5664bfa1782d5e04779689068c916b04cb365ec3153755684d9a1"
);
assert_eq!(
format!("0x{}", eth.public_key),
"0xe04cc55ebee1cbce552f250e85c57b70b2e2625b"
)
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/generators/keystore.rs
Vendored
+290
View File
@@ -0,0 +1,290 @@
use std::{
path::{Path, PathBuf},
vec,
};
use hex::encode;
use support::{constants::THIS_IS_A_BUG, fs::FileSystem};
use super::errors::GeneratorError;
use crate::{
generators::keystore_key_types::{parse_keystore_key_types, KeystoreKeyType},
shared::types::NodeAccounts,
ScopedFilesystem,
};
/// Generates keystore files for a node.
///
/// # Arguments
/// * `acc` - The node accounts containing the seed and public keys
/// * `node_files_path` - The path where keystore files will be created
/// * `scoped_fs` - The scoped filesystem for file operations
/// * `asset_hub_polkadot` - Whether this is for asset-hub-polkadot (affects aura key scheme)
/// * `keystore_key_types` - Optional list of key type specifications
///
/// If `keystore_key_types` is empty, all default key types will be generated.
/// Otherwise, only the specified key types will be generated.
pub async fn generate<'a, T>(
acc: &NodeAccounts,
node_files_path: impl AsRef<Path>,
scoped_fs: &ScopedFilesystem<'a, T>,
asset_hub_polkadot: bool,
keystore_key_types: Vec<&str>,
) -> Result<Vec<PathBuf>, GeneratorError>
where
T: FileSystem,
{
// Create local keystore
scoped_fs.create_dir_all(node_files_path.as_ref()).await?;
let mut filenames = vec![];
// Parse the key type specifications
let key_types = parse_keystore_key_types(&keystore_key_types, asset_hub_polkadot);
let futures: Vec<_> = key_types
.iter()
.map(|key_type| {
let filename = generate_keystore_filename(key_type, acc);
let file_path = PathBuf::from(format!(
"{}/{}",
node_files_path.as_ref().to_string_lossy(),
filename
));
let content = format!("\"{}\"", acc.seed);
(filename, scoped_fs.write(file_path, content))
})
.collect();
for (filename, future) in futures {
future.await?;
filenames.push(PathBuf::from(filename));
}
Ok(filenames)
}
/// Generates the keystore filename for a given key type.
///
/// The filename format is: `{hex_encoded_key_type}{public_key}`
fn generate_keystore_filename(key_type: &KeystoreKeyType, acc: &NodeAccounts) -> String {
let account_key = key_type.scheme.account_key();
let pk = acc
.accounts
.get(account_key)
.expect(&format!(
"Key '{}' should be set for node {THIS_IS_A_BUG}",
account_key
))
.public_key
.as_str();
format!("{}{}", encode(&key_type.key_type), pk)
}
#[cfg(test)]
mod tests {
use std::{collections::HashMap, ffi::OsString, str::FromStr};
use support::fs::in_memory::{InMemoryFile, InMemoryFileSystem};
use super::*;
use crate::shared::types::{NodeAccount, NodeAccounts};
fn create_test_accounts() -> NodeAccounts {
let mut accounts = HashMap::new();
accounts.insert(
"sr".to_string(),
NodeAccount::new("sr_address", "sr_public_key"),
);
accounts.insert(
"ed".to_string(),
NodeAccount::new("ed_address", "ed_public_key"),
);
accounts.insert(
"ec".to_string(),
NodeAccount::new("ec_address", "ec_public_key"),
);
NodeAccounts {
seed: "//Alice".to_string(),
accounts,
}
}
fn create_test_fs() -> InMemoryFileSystem {
InMemoryFileSystem::new(HashMap::from([(
OsString::from_str("/").unwrap(),
InMemoryFile::dir(),
)]))
}
#[tokio::test]
async fn generate_creates_default_keystore_files_when_no_key_types_specified() {
let accounts = create_test_accounts();
let fs = create_test_fs();
let base_dir = "/tmp/test";
let scoped_fs = ScopedFilesystem { fs: &fs, base_dir };
let key_types: Vec<&str> = vec![];
let res = generate(&accounts, "node1", &scoped_fs, false, key_types).await;
assert!(res.is_ok());
let filenames = res.unwrap();
assert!(filenames.len() > 10);
let filename_strs: Vec<String> = filenames
.iter()
.map(|p| p.to_string_lossy().to_string())
.collect();
// Check that aura key is generated (hex of "aura" is 61757261)
assert!(filename_strs.iter().any(|f| f.starts_with("61757261")));
// Check that babe key is generated (hex of "babe" is 62616265)
assert!(filename_strs.iter().any(|f| f.starts_with("62616265")));
// Check that gran key is generated (hex of "gran" is 6772616e)
assert!(filename_strs.iter().any(|f| f.starts_with("6772616e")));
}
#[tokio::test]
async fn generate_creates_only_specified_keystore_files() {
let accounts = create_test_accounts();
let fs = create_test_fs();
let base_dir = "/tmp/test";
let scoped_fs = ScopedFilesystem { fs: &fs, base_dir };
let key_types = vec!["audi", "gran"];
let res = generate(&accounts, "node1", &scoped_fs, false, key_types).await;
assert!(res.is_ok());
let filenames = res.unwrap();
assert_eq!(filenames.len(), 2);
let filename_strs: Vec<String> = filenames
.iter()
.map(|p| p.to_string_lossy().to_string())
.collect();
// audi uses sr scheme by default
assert!(filename_strs
.iter()
.any(|f| f.starts_with("61756469") && f.contains("sr_public_key")));
// gran uses ed scheme by default
assert!(filename_strs
.iter()
.any(|f| f.starts_with("6772616e") && f.contains("ed_public_key")));
}
#[tokio::test]
async fn generate_produces_correct_keystore_files() {
struct TestCase {
name: &'static str,
key_types: Vec<&'static str>,
asset_hub_polkadot: bool,
expected_prefix: &'static str,
expected_public_key: &'static str,
}
let test_cases = vec![
TestCase {
name: "explicit scheme override (gran_sr)",
key_types: vec!["gran_sr"],
asset_hub_polkadot: false,
expected_prefix: "6772616e", // "gran" in hex
expected_public_key: "sr_public_key",
},
TestCase {
name: "aura with asset_hub_polkadot uses ed",
key_types: vec!["aura"],
asset_hub_polkadot: true,
expected_prefix: "61757261", // "aura" in hex
expected_public_key: "ed_public_key",
},
TestCase {
name: "aura without asset_hub_polkadot uses sr",
key_types: vec!["aura"],
asset_hub_polkadot: false,
expected_prefix: "61757261", // "aura" in hex
expected_public_key: "sr_public_key",
},
TestCase {
name: "custom key type with explicit ec scheme",
key_types: vec!["cust_ec"],
asset_hub_polkadot: false,
expected_prefix: "63757374", // "cust" in hex
expected_public_key: "ec_public_key",
},
];
for tc in test_cases {
let accounts = create_test_accounts();
let fs = create_test_fs();
let scoped_fs = ScopedFilesystem {
fs: &fs,
base_dir: "/tmp/test",
};
let key_types: Vec<&str> = tc.key_types.clone();
let res = generate(
&accounts,
"node1",
&scoped_fs,
tc.asset_hub_polkadot,
key_types,
)
.await;
assert!(
res.is_ok(),
"[{}] Expected Ok but got: {:?}",
tc.name,
res.err()
);
let filenames = res.unwrap();
assert_eq!(filenames.len(), 1, "[{}] Expected 1 file", tc.name);
let filename = filenames[0].to_string_lossy().to_string();
assert!(
filename.starts_with(tc.expected_prefix),
"[{}] Expected prefix '{}', got '{}'",
tc.name,
tc.expected_prefix,
filename
);
assert!(
filename.contains(tc.expected_public_key),
"[{}] Expected public key '{}' in '{}'",
tc.name,
tc.expected_public_key,
filename
);
}
}
#[tokio::test]
async fn generate_ignores_invalid_key_specs_and_uses_defaults() {
let accounts = create_test_accounts();
let fs = create_test_fs();
let scoped_fs = ScopedFilesystem {
fs: &fs,
base_dir: "/tmp/test",
};
let key_types = vec![
"invalid", // Too long
"xxx", // Too short
"audi_xx", // Invalid sceme
];
let res = generate(&accounts, "node1", &scoped_fs, false, key_types).await;
assert!(res.is_ok());
let filenames = res.unwrap();
// Should fall back to defaults since all specs are invalid
assert!(filenames.len() > 10);
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/generators/keystore_key_types.rs
Vendored
+282
View File
@@ -0,0 +1,282 @@
use std::{collections::HashMap, fmt::Formatter};
use serde::{Deserialize, Serialize};
/// Supported cryptographic schemes for keystore keys.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]
pub enum KeyScheme {
/// Sr25519 scheme
Sr,
/// Ed25519 scheme
Ed,
/// ECDSA scheme
Ec,
}
impl KeyScheme {
/// Returns the account key suffix used in `NodeAccounts` for this scheme.
pub fn account_key(&self) -> &'static str {
match self {
KeyScheme::Sr => "sr",
KeyScheme::Ed => "ed",
KeyScheme::Ec => "ec",
}
}
}
impl std::fmt::Display for KeyScheme {
fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
match self {
KeyScheme::Sr => write!(f, "sr"),
KeyScheme::Ed => write!(f, "ed"),
KeyScheme::Ec => write!(f, "ec"),
}
}
}
impl TryFrom<&str> for KeyScheme {
type Error = String;
fn try_from(value: &str) -> Result<Self, Self::Error> {
match value.to_lowercase().as_str() {
"sr" => Ok(KeyScheme::Sr),
"ed" => Ok(KeyScheme::Ed),
"ec" => Ok(KeyScheme::Ec),
_ => Err(format!("Unsupported key scheme: {}", value)),
}
}
}
/// A parsed keystore key type.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct KeystoreKeyType {
/// The 4-character key type identifier (e.g., "aura", "babe", "gran").
pub key_type: String,
/// The cryptographic scheme to use for this key type.
pub scheme: KeyScheme,
}
impl KeystoreKeyType {
pub fn new(key_type: impl Into<String>, scheme: KeyScheme) -> Self {
Self {
key_type: key_type.into(),
scheme,
}
}
}
/// Returns the default predefined key schemes for known key types.
/// Special handling for `aura` when `is_asset_hub_polkadot` is true.
fn get_predefined_schemes(is_asset_hub_polkadot: bool) -> HashMap<&'static str, KeyScheme> {
let mut schemes = HashMap::new();
// aura has special handling for asset-hub-polkadot
if is_asset_hub_polkadot {
schemes.insert("aura", KeyScheme::Ed);
} else {
schemes.insert("aura", KeyScheme::Sr);
}
schemes.insert("babe", KeyScheme::Sr);
schemes.insert("imon", KeyScheme::Sr);
schemes.insert("gran", KeyScheme::Ed);
schemes.insert("audi", KeyScheme::Sr);
schemes.insert("asgn", KeyScheme::Sr);
schemes.insert("para", KeyScheme::Sr);
schemes.insert("beef", KeyScheme::Ec);
schemes.insert("nmbs", KeyScheme::Sr); // Nimbus
schemes.insert("rand", KeyScheme::Sr); // Randomness (Moonbeam)
schemes.insert("rate", KeyScheme::Ed); // Equilibrium rate module
schemes.insert("acco", KeyScheme::Sr);
schemes.insert("bcsv", KeyScheme::Sr); // BlockchainSrvc (StorageHub)
schemes.insert("ftsv", KeyScheme::Ed); // FileTransferSrvc (StorageHub)
schemes.insert("mixn", KeyScheme::Sr); // Mixnet
schemes
}
/// Parses a single keystore key type specification string.
///
/// Supports two formats:
/// - Short: `audi` - creates key type with predefined default scheme (defaults to `sr` if not predefined)
/// - Long: `audi_sr` - creates key type with explicit scheme
///
/// Returns `None` if the spec is invalid or doesn't match the expected format.
fn parse_key_spec(spec: &str, predefined: &HashMap<&str, KeyScheme>) -> Option<KeystoreKeyType> {
let spec = spec.trim();
// Try parsing as long form first: key_type_scheme (e.g., "audi_sr")
if let Some((key_type, scheme_str)) = spec.split_once('_') {
if key_type.len() != 4 {
return None;
}
let scheme = KeyScheme::try_from(scheme_str).ok()?;
return Some(KeystoreKeyType::new(key_type, scheme));
}
// Try parsing as short form: key_type only (e.g., "audi")
if spec.len() == 4 {
// Look up predefined scheme; default to Sr if not found
let scheme = predefined.get(spec).copied().unwrap_or(KeyScheme::Sr);
return Some(KeystoreKeyType::new(spec, scheme));
}
None
}
/// Parses a list of keystore key type specifications.
///
/// Each spec can be in short form (`audi`) or long form (`audi_sr`).
/// Invalid specs are silently ignored.
///
/// If the resulting list is empty, returns the default keystore key types.
pub fn parse_keystore_key_types<T: AsRef<str>>(
specs: &[T],
is_asset_hub_polkadot: bool,
) -> Vec<KeystoreKeyType> {
let predefined_schemes = get_predefined_schemes(is_asset_hub_polkadot);
let parsed: Vec<KeystoreKeyType> = specs
.iter()
.filter_map(|spec| parse_key_spec(spec.as_ref(), &predefined_schemes))
.collect();
if parsed.is_empty() {
get_default_keystore_key_types(is_asset_hub_polkadot)
} else {
parsed
}
}
/// Returns the default keystore key types when none are specified.
pub fn get_default_keystore_key_types(is_asset_hub_polkadot: bool) -> Vec<KeystoreKeyType> {
let predefined_schemes = get_predefined_schemes(is_asset_hub_polkadot);
let default_keys = [
"aura", "babe", "imon", "gran", "audi", "asgn", "para", "beef", "nmbs", "rand", "rate",
"mixn", "bcsv", "ftsv",
];
default_keys
.iter()
.filter_map(|key_type| {
predefined_schemes
.get(*key_type)
.map(|scheme| KeystoreKeyType::new(*key_type, *scheme))
})
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parse_keystore_key_types_ignores_invalid_specs() {
let specs = vec![
"audi".to_string(),
"invalid".to_string(), // Too long - ignored
"xxx".to_string(), // Too short - ignored
"xxxx".to_string(), // Unknown key - defaults to sr
"audi_xx".to_string(), // Invalid scheme - ignored
"gran".to_string(),
];
let result = parse_keystore_key_types(&specs, false);
assert_eq!(result.len(), 3);
assert_eq!(result[1], KeystoreKeyType::new("xxxx", KeyScheme::Sr)); // Unknown defaults to sr
assert_eq!(result[2], KeystoreKeyType::new("gran", KeyScheme::Ed));
}
#[test]
fn parse_keystore_key_types_returns_specified_keys() {
let specs = vec!["audi".to_string(), "gran".to_string()];
let res = parse_keystore_key_types(&specs, false);
assert_eq!(res.len(), 2);
assert_eq!(res[0], KeystoreKeyType::new("audi", KeyScheme::Sr));
assert_eq!(res[1], KeystoreKeyType::new("gran", KeyScheme::Ed));
}
#[test]
fn parse_keystore_key_types_mixed_short_and_long_forms() {
let specs = vec![
"audi".to_string(),
"gran_sr".to_string(), // Override gran's default ed to sr
"gran".to_string(),
"beef".to_string(),
];
let res = parse_keystore_key_types(&specs, false);
assert_eq!(res.len(), 4);
assert_eq!(res[0], KeystoreKeyType::new("audi", KeyScheme::Sr));
assert_eq!(res[1], KeystoreKeyType::new("gran", KeyScheme::Sr)); // Overridden
assert_eq!(res[2], KeystoreKeyType::new("gran", KeyScheme::Ed));
assert_eq!(res[3], KeystoreKeyType::new("beef", KeyScheme::Ec));
}
#[test]
fn parse_keystore_key_types_returns_defaults_when_empty() {
let specs: Vec<String> = vec![];
let res = parse_keystore_key_types(&specs, false);
// Should return all default keys
assert!(!res.is_empty());
assert!(res.iter().any(|k| k.key_type == "aura"));
assert!(res.iter().any(|k| k.key_type == "babe"));
assert!(res.iter().any(|k| k.key_type == "gran"));
}
#[test]
fn parse_keystore_key_types_allows_custom_key_with_explicit_scheme() {
let specs = vec![
"cust_sr".to_string(), // Custom key with explicit scheme
"audi".to_string(),
];
let result = parse_keystore_key_types(&specs, false);
assert_eq!(result.len(), 2);
assert_eq!(result[0], KeystoreKeyType::new("cust", KeyScheme::Sr));
assert_eq!(result[1], KeystoreKeyType::new("audi", KeyScheme::Sr));
}
#[test]
fn full_workflow_asset_hub_polkadot() {
// For asset-hub-polkadot, aura should default to ed
let specs = vec!["aura".to_string(), "babe".to_string()];
let res = parse_keystore_key_types(&specs, true);
assert_eq!(res.len(), 2);
assert_eq!(res[0].key_type, "aura");
assert_eq!(res[0].scheme, KeyScheme::Ed); // ed for asset-hub-polkadot
assert_eq!(res[1].key_type, "babe");
assert_eq!(res[1].scheme, KeyScheme::Sr);
}
#[test]
fn full_workflow_custom_key_types() {
let specs = vec![
"aura".to_string(), // Use default scheme
"gran_sr".to_string(), // Override gran to use sr instead of ed
"cust_ec".to_string(), // Custom key type with ecdsa
];
let res = parse_keystore_key_types(&specs, false);
assert_eq!(res.len(), 3);
// aura uses default sr
assert_eq!(res[0].key_type, "aura");
assert_eq!(res[0].scheme, KeyScheme::Sr);
// gran overridden to sr
assert_eq!(res[1].key_type, "gran");
assert_eq!(res[1].scheme, KeyScheme::Sr);
// custom key with ec
assert_eq!(res[2].key_type, "cust");
assert_eq!(res[2].scheme, KeyScheme::Ec);
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/generators/para_artifact.rs
Vendored
+165
View File
@@ -0,0 +1,165 @@
use std::path::{Path, PathBuf};
use configuration::types::CommandWithCustomArgs;
use provider::{
constants::NODE_CONFIG_DIR,
types::{GenerateFileCommand, GenerateFilesOptions, TransferedFile},
DynNamespace,
};
use serde::{Deserialize, Serialize};
use support::fs::FileSystem;
use uuid::Uuid;
use super::errors::GeneratorError;
use crate::ScopedFilesystem;
#[derive(Debug, Clone, Serialize, Deserialize)]
pub(crate) enum ParaArtifactType {
Wasm,
State,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub(crate) enum ParaArtifactBuildOption {
Path(String),
Command(String),
CommandWithCustomArgs(CommandWithCustomArgs),
}
/// Parachain artifact (could be either the genesis state or genesis wasm)
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ParaArtifact {
artifact_type: ParaArtifactType,
build_option: ParaArtifactBuildOption,
artifact_path: Option<PathBuf>,
// image to use for building the para artifact
image: Option<String>,
}
impl ParaArtifact {
pub(crate) fn new(
artifact_type: ParaArtifactType,
build_option: ParaArtifactBuildOption,
) -> Self {
Self {
artifact_type,
build_option,
artifact_path: None,
image: None,
}
}
pub(crate) fn image(mut self, image: Option<String>) -> Self {
self.image = image;
self
}
pub(crate) fn artifact_path(&self) -> Option<&PathBuf> {
self.artifact_path.as_ref()
}
pub(crate) async fn build<'a, T>(
&mut self,
chain_spec_path: Option<impl AsRef<Path>>,
artifact_path: impl AsRef<Path>,
ns: &DynNamespace,
scoped_fs: &ScopedFilesystem<'a, T>,
maybe_output_path: Option<PathBuf>,
) -> Result<(), GeneratorError>
where
T: FileSystem,
{
let (cmd, custom_args) = match &self.build_option {
ParaArtifactBuildOption::Path(path) => {
let t = TransferedFile::new(PathBuf::from(path), artifact_path.as_ref().into());
scoped_fs.copy_files(vec![&t]).await?;
self.artifact_path = Some(artifact_path.as_ref().into());
return Ok(()); // work done!
},
ParaArtifactBuildOption::Command(cmd) => (cmd, &vec![]),
ParaArtifactBuildOption::CommandWithCustomArgs(cmd_with_custom_args) => {
(
&cmd_with_custom_args.cmd().as_str().to_string(),
cmd_with_custom_args.args(),
)
// (cmd.cmd_as_str().to_string(), cmd.1)
},
};
let generate_subcmd = match self.artifact_type {
ParaArtifactType::Wasm => "export-genesis-wasm",
ParaArtifactType::State => "export-genesis-state",
};
// TODO: replace uuid with para_id-random
let temp_name = format!("temp-{}-{}", generate_subcmd, Uuid::new_v4());
let mut args: Vec<String> = vec![generate_subcmd.into()];
let files_to_inject = if let Some(chain_spec_path) = chain_spec_path {
// TODO: we should get the full path from the scoped filesystem
let chain_spec_path_local = format!(
"{}/{}",
ns.base_dir().to_string_lossy(),
chain_spec_path.as_ref().to_string_lossy()
);
// Remote path to be injected
let chain_spec_path_in_pod = format!(
"{}/{}",
NODE_CONFIG_DIR,
chain_spec_path.as_ref().to_string_lossy()
);
// Path in the context of the node, this can be different in the context of the providers (e.g native)
let chain_spec_path_in_args = if ns.capabilities().prefix_with_full_path {
// In native
format!(
"{}/{}{}",
ns.base_dir().to_string_lossy(),
&temp_name,
&chain_spec_path_in_pod
)
} else {
chain_spec_path_in_pod.clone()
};
args.push("--chain".into());
args.push(chain_spec_path_in_args);
for custom_arg in custom_args {
match custom_arg {
configuration::types::Arg::Flag(flag) => {
args.push(flag.into());
},
configuration::types::Arg::Option(flag, flag_value) => {
args.push(flag.into());
args.push(flag_value.into());
},
configuration::types::Arg::Array(flag, values) => {
args.push(flag.into());
values.iter().for_each(|v| args.push(v.into()));
},
}
}
vec![TransferedFile::new(
chain_spec_path_local,
chain_spec_path_in_pod,
)]
} else {
vec![]
};
let artifact_path_ref = artifact_path.as_ref();
let generate_command = GenerateFileCommand::new(cmd.as_str(), artifact_path_ref).args(args);
let options = GenerateFilesOptions::with_files(
vec![generate_command],
self.image.clone(),
&files_to_inject,
maybe_output_path,
)
.temp_name(temp_name);
ns.generate_files(options).await?;
self.artifact_path = Some(artifact_path_ref.into());
Ok(())
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/generators/port.rs
Vendored
+48
View File
@@ -0,0 +1,48 @@
use std::net::TcpListener;
use configuration::shared::types::Port;
use support::constants::THIS_IS_A_BUG;
use super::errors::GeneratorError;
use crate::shared::types::ParkedPort;
// TODO: (team), we want to continue support ws_port? No
enum PortTypes {
Rpc,
P2P,
Prometheus,
}
pub fn generate(port: Option<Port>) -> Result<ParkedPort, GeneratorError> {
let port = port.unwrap_or(0);
let listener = TcpListener::bind(format!("0.0.0.0:{port}"))
.map_err(|_e| GeneratorError::PortGeneration(port, "Can't bind".into()))?;
let port = listener
.local_addr()
.expect(&format!(
"We should always get the local_addr from the listener {THIS_IS_A_BUG}"
))
.port();
Ok(ParkedPort::new(port, listener))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn generate_random() {
let port = generate(None).unwrap();
let listener = port.1.write().unwrap();
assert!(listener.is_some());
}
#[test]
fn generate_fixed_port() {
let port = generate(Some(33056)).unwrap();
let listener = port.1.write().unwrap();
assert!(listener.is_some());
assert_eq!(port.0, 33056);
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/lib.rs
Vendored
+1352
View File
File diff suppressed because it is too large Load Diff
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/network.rs
Vendored
+842
View File
@@ -0,0 +1,842 @@
pub mod chain_upgrade;
pub mod node;
pub mod relaychain;
pub mod teyrchain;
use std::{cell::RefCell, collections::HashMap, path::PathBuf, rc::Rc, sync::Arc, time::Duration};
use configuration::{
para_states::{Initial, Running},
shared::{helpers::generate_unique_node_name_from_names, node::EnvVar},
types::{Arg, Command, Image, Port, ValidationContext},
ParachainConfig, ParachainConfigBuilder, RegistrationStrategy,
};
use provider::{types::TransferedFile, DynNamespace, ProviderError};
use serde::Serialize;
use support::fs::FileSystem;
use tokio::sync::RwLock;
use tracing::{error, warn};
use self::{node::NetworkNode, relaychain::Relaychain, teyrchain::Parachain};
use crate::{
generators::chain_spec::ChainSpec,
network_spec::{self, NetworkSpec},
shared::{
constants::{NODE_MONITORING_FAILURE_THRESHOLD_SECONDS, NODE_MONITORING_INTERVAL_SECONDS},
macros,
types::{ChainDefaultContext, RegisterParachainOptions},
},
spawner::{self, SpawnNodeCtx},
ScopedFilesystem, ZombieRole,
};
#[derive(Serialize)]
pub struct Network<T: FileSystem> {
#[serde(skip)]
ns: DynNamespace,
#[serde(skip)]
filesystem: T,
relay: Relaychain,
initial_spec: NetworkSpec,
parachains: HashMap<u32, Vec<Parachain>>,
#[serde(skip)]
nodes_by_name: HashMap<String, NetworkNode>,
#[serde(skip)]
nodes_to_watch: Arc<RwLock<Vec<NetworkNode>>>,
}
impl<T: FileSystem> std::fmt::Debug for Network<T> {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Network")
.field("ns", &"ns_skipped")
.field("relay", &self.relay)
.field("initial_spec", &self.initial_spec)
.field("parachains", &self.parachains)
.field("nodes_by_name", &self.nodes_by_name)
.finish()
}
}
macros::create_add_options!(AddNodeOptions {
chain_spec: Option<PathBuf>,
override_eth_key: Option<String>
});
macros::create_add_options!(AddCollatorOptions {
chain_spec: Option<PathBuf>,
chain_spec_relay: Option<PathBuf>,
override_eth_key: Option<String>
});
impl<T: FileSystem> Network<T> {
pub(crate) fn new_with_relay(
relay: Relaychain,
ns: DynNamespace,
fs: T,
initial_spec: NetworkSpec,
) -> Self {
Self {
ns,
filesystem: fs,
relay,
initial_spec,
parachains: Default::default(),
nodes_by_name: Default::default(),
nodes_to_watch: Default::default(),
}
}
// Pubic API
pub fn ns_name(&self) -> String {
self.ns.name().to_string()
}
pub fn base_dir(&self) -> Option<&str> {
self.ns.base_dir().to_str()
}
pub fn relaychain(&self) -> &Relaychain {
&self.relay
}
// Teardown the network
pub async fn destroy(self) -> Result<(), ProviderError> {
self.ns.destroy().await
}
/// Add a node to the relaychain
// The new node is added to the running network instance.
/// # Example:
/// ```rust
/// # use provider::NativeProvider;
/// # use support::{fs::local::LocalFileSystem};
/// # use zombienet_orchestrator::{errors, AddNodeOptions, Orchestrator};
/// # use configuration::NetworkConfig;
/// # async fn example() -> Result<(), errors::OrchestratorError> {
/// # let provider = NativeProvider::new(LocalFileSystem {});
/// # let orchestrator = Orchestrator::new(LocalFileSystem {}, provider);
/// # let config = NetworkConfig::load_from_toml("config.toml")?;
/// let mut network = orchestrator.spawn(config).await?;
///
/// // Create the options to add the new node
/// let opts = AddNodeOptions {
/// rpc_port: Some(9444),
/// is_validator: true,
/// ..Default::default()
/// };
///
/// network.add_node("new-node", opts).await?;
/// # Ok(())
/// # }
/// ```
pub async fn add_node(
&mut self,
name: impl Into<String>,
options: AddNodeOptions,
) -> Result<(), anyhow::Error> {
let name = generate_unique_node_name_from_names(
name,
&mut self.nodes_by_name.keys().cloned().collect(),
);
let relaychain = self.relaychain();
let chain_spec_path = if let Some(chain_spec_custom_path) = &options.chain_spec {
chain_spec_custom_path.clone()
} else {
PathBuf::from(format!(
"{}/{}.json",
self.ns.base_dir().to_string_lossy(),
relaychain.chain
))
};
let chain_context = ChainDefaultContext {
default_command: self.initial_spec.relaychain.default_command.as_ref(),
default_image: self.initial_spec.relaychain.default_image.as_ref(),
default_resources: self.initial_spec.relaychain.default_resources.as_ref(),
default_db_snapshot: self.initial_spec.relaychain.default_db_snapshot.as_ref(),
default_args: self.initial_spec.relaychain.default_args.iter().collect(),
};
let mut node_spec = network_spec::node::NodeSpec::from_ad_hoc(
&name,
options.into(),
&chain_context,
false,
false,
)?;
node_spec.available_args_output = Some(
self.initial_spec
.node_available_args_output(&node_spec, self.ns.clone())
.await?,
);
let base_dir = self.ns.base_dir().to_string_lossy();
let scoped_fs = ScopedFilesystem::new(&self.filesystem, &base_dir);
let ctx = SpawnNodeCtx {
chain_id: &relaychain.chain_id,
parachain_id: None,
chain: &relaychain.chain,
role: ZombieRole::Node,
ns: &self.ns,
scoped_fs: &scoped_fs,
parachain: None,
bootnodes_addr: &vec![],
wait_ready: true,
nodes_by_name: serde_json::to_value(&self.nodes_by_name)?,
global_settings: &self.initial_spec.global_settings,
};
let global_files_to_inject = vec![TransferedFile::new(
chain_spec_path,
PathBuf::from(format!("/cfg/{}.json", relaychain.chain)),
)];
let node = spawner::spawn_node(&node_spec, global_files_to_inject, &ctx).await?;
// TODO: register the new node as validator in the relaychain
// STEPS:
// - check balance of `stash` derivation for validator account
// - call rotate_keys on the new validator
// - call setKeys on the new validator
// if node_spec.is_validator {
// let running_node = self.relay.nodes.first().unwrap();
// // tx_helper::validator_actions::register(vec![&node], &running_node.ws_uri, None).await?;
// }
// Let's make sure node is up before adding
node.wait_until_is_up(self.initial_spec.global_settings.network_spawn_timeout())
.await?;
// Add node to relaychain data
self.add_running_node(node.clone(), None).await;
Ok(())
}
/// Add a new collator to a parachain
///
/// NOTE: if more parachains with given id available (rare corner case)
/// then it adds collator to the first parachain
///
/// # Example:
/// ```rust
/// # use provider::NativeProvider;
/// # use support::{fs::local::LocalFileSystem};
/// # use zombienet_orchestrator::{errors, AddCollatorOptions, Orchestrator};
/// # use configuration::NetworkConfig;
/// # async fn example() -> Result<(), anyhow::Error> {
/// # let provider = NativeProvider::new(LocalFileSystem {});
/// # let orchestrator = Orchestrator::new(LocalFileSystem {}, provider);
/// # let config = NetworkConfig::load_from_toml("config.toml")?;
/// let mut network = orchestrator.spawn(config).await?;
///
/// let col_opts = AddCollatorOptions {
/// command: Some("polkadot-parachain".try_into()?),
/// ..Default::default()
/// };
///
/// network.add_collator("new-col-1", col_opts, 100).await?;
/// # Ok(())
/// # }
/// ```
pub async fn add_collator(
&mut self,
name: impl Into<String>,
options: AddCollatorOptions,
para_id: u32,
) -> Result<(), anyhow::Error> {
let name = generate_unique_node_name_from_names(
name,
&mut self.nodes_by_name.keys().cloned().collect(),
);
let spec = self
.initial_spec
.parachains
.iter()
.find(|para| para.id == para_id)
.ok_or(anyhow::anyhow!(format!("parachain: {para_id} not found!")))?;
let role = if spec.is_cumulus_based {
ZombieRole::CumulusCollator
} else {
ZombieRole::Collator
};
let chain_context = ChainDefaultContext {
default_command: spec.default_command.as_ref(),
default_image: spec.default_image.as_ref(),
default_resources: spec.default_resources.as_ref(),
default_db_snapshot: spec.default_db_snapshot.as_ref(),
default_args: spec.default_args.iter().collect(),
};
let parachain = self
.parachains
.get_mut(&para_id)
.ok_or(anyhow::anyhow!(format!("parachain: {para_id} not found!")))?
.get_mut(0)
.ok_or(anyhow::anyhow!(format!("parachain: {para_id} not found!")))?;
let base_dir = self.ns.base_dir().to_string_lossy();
let scoped_fs = ScopedFilesystem::new(&self.filesystem, &base_dir);
// TODO: we want to still supporting spawn a dedicated bootnode??
let ctx = SpawnNodeCtx {
chain_id: &self.relay.chain_id,
parachain_id: parachain.chain_id.as_deref(),
chain: &self.relay.chain,
role,
ns: &self.ns,
scoped_fs: &scoped_fs,
parachain: Some(spec),
bootnodes_addr: &vec![],
wait_ready: true,
nodes_by_name: serde_json::to_value(&self.nodes_by_name)?,
global_settings: &self.initial_spec.global_settings,
};
let relaychain_spec_path = if let Some(chain_spec_custom_path) = &options.chain_spec_relay {
chain_spec_custom_path.clone()
} else {
PathBuf::from(format!(
"{}/{}.json",
self.ns.base_dir().to_string_lossy(),
self.relay.chain
))
};
let mut global_files_to_inject = vec![TransferedFile::new(
relaychain_spec_path,
PathBuf::from(format!("/cfg/{}.json", self.relay.chain)),
)];
let para_chain_spec_local_path = if let Some(para_chain_spec_custom) = &options.chain_spec {
Some(para_chain_spec_custom.clone())
} else if let Some(para_spec_path) = &parachain.chain_spec_path {
Some(PathBuf::from(format!(
"{}/{}",
self.ns.base_dir().to_string_lossy(),
para_spec_path.to_string_lossy()
)))
} else {
None
};
if let Some(para_spec_path) = para_chain_spec_local_path {
global_files_to_inject.push(TransferedFile::new(
para_spec_path,
PathBuf::from(format!("/cfg/{para_id}.json")),
));
}
let mut node_spec = network_spec::node::NodeSpec::from_ad_hoc(
name,
options.into(),
&chain_context,
true,
spec.is_evm_based,
)?;
node_spec.available_args_output = Some(
self.initial_spec
.node_available_args_output(&node_spec, self.ns.clone())
.await?,
);
let node = spawner::spawn_node(&node_spec, global_files_to_inject, &ctx).await?;
// Let's make sure node is up before adding
node.wait_until_is_up(self.initial_spec.global_settings.network_spawn_timeout())
.await?;
parachain.collators.push(node.clone());
self.add_running_node(node, None).await;
Ok(())
}
/// Get a parachain config builder from a running network
///
/// This allow you to build a new parachain config to be deployed into
/// the running network.
pub fn para_config_builder(&self) -> ParachainConfigBuilder<Initial, Running> {
let used_ports = self
.nodes_iter()
.map(|node| node.spec())
.flat_map(|spec| {
[
spec.ws_port.0,
spec.rpc_port.0,
spec.prometheus_port.0,
spec.p2p_port.0,
]
})
.collect();
let used_nodes_names = self.nodes_by_name.keys().cloned().collect();
// need to inverse logic of generate_unique_para_id
let used_para_ids = self
.parachains
.iter()
.map(|(id, paras)| (*id, paras.len().saturating_sub(1) as u8))
.collect();
let context = ValidationContext {
used_ports,
used_nodes_names,
used_para_ids,
};
let context = Rc::new(RefCell::new(context));
ParachainConfigBuilder::new_with_running(context)
}
/// Add a new parachain to the running network
///
/// # Arguments
/// * `para_config` - Parachain configuration to deploy
/// * `custom_relaychain_spec` - Optional path to a custom relaychain spec to use
/// * `custom_parchain_fs_prefix` - Optional prefix to use when artifacts are created
///
///
/// # Example:
/// ```rust
/// # use anyhow::anyhow;
/// # use provider::NativeProvider;
/// # use support::{fs::local::LocalFileSystem};
/// # use zombienet_orchestrator::{errors, AddCollatorOptions, Orchestrator};
/// # use configuration::NetworkConfig;
/// # async fn example() -> Result<(), anyhow::Error> {
/// # let provider = NativeProvider::new(LocalFileSystem {});
/// # let orchestrator = Orchestrator::new(LocalFileSystem {}, provider);
/// # let config = NetworkConfig::load_from_toml("config.toml")?;
/// let mut network = orchestrator.spawn(config).await?;
/// let para_config = network
/// .para_config_builder()
/// .with_id(100)
/// .with_default_command("polkadot-parachain")
/// .with_collator(|c| c.with_name("col-100-1"))
/// .build()
/// .map_err(|_e| anyhow!("Building config"))?;
///
/// network.add_parachain(&para_config, None, None).await?;
///
/// # Ok(())
/// # }
/// ```
pub async fn add_parachain(
&mut self,
para_config: &ParachainConfig,
custom_relaychain_spec: Option<PathBuf>,
custom_parchain_fs_prefix: Option<String>,
) -> Result<(), anyhow::Error> {
let base_dir = self.ns.base_dir().to_string_lossy().to_string();
let scoped_fs = ScopedFilesystem::new(&self.filesystem, &base_dir);
let mut global_files_to_inject = vec![];
// get relaychain id
let relay_chain_id = if let Some(custom_path) = custom_relaychain_spec {
// use this file as relaychain spec
global_files_to_inject.push(TransferedFile::new(
custom_path.clone(),
PathBuf::from(format!("/cfg/{}.json", self.relaychain().chain)),
));
let content = std::fs::read_to_string(custom_path)?;
ChainSpec::chain_id_from_spec(&content)?
} else {
global_files_to_inject.push(TransferedFile::new(
PathBuf::from(format!(
"{}/{}",
scoped_fs.base_dir,
self.relaychain().chain_spec_path.to_string_lossy()
)),
PathBuf::from(format!("/cfg/{}.json", self.relaychain().chain)),
));
self.relay.chain_id.clone()
};
let mut para_spec = network_spec::teyrchain::TeyrchainSpec::from_config(
para_config,
relay_chain_id.as_str().try_into()?,
)?;
let chain_spec_raw_path = para_spec
.build_chain_spec(&relay_chain_id, &self.ns, &scoped_fs)
.await?;
// Para artifacts
let para_path_prefix = if let Some(custom_prefix) = custom_parchain_fs_prefix {
custom_prefix
} else {
para_spec.id.to_string()
};
scoped_fs.create_dir(&para_path_prefix).await?;
// create wasm/state
para_spec
.genesis_state
.build(
chain_spec_raw_path.as_ref(),
format!("{}/genesis-state", &para_path_prefix),
&self.ns,
&scoped_fs,
None,
)
.await?;
para_spec
.genesis_wasm
.build(
chain_spec_raw_path.as_ref(),
format!("{}/para_spec-wasm", &para_path_prefix),
&self.ns,
&scoped_fs,
None,
)
.await?;
let parachain =
Parachain::from_spec(&para_spec, &global_files_to_inject, &scoped_fs).await?;
let parachain_id = parachain.chain_id.clone();
// Create `ctx` for spawn the nodes
let ctx_para = SpawnNodeCtx {
parachain: Some(&para_spec),
parachain_id: parachain_id.as_deref(),
role: if para_spec.is_cumulus_based {
ZombieRole::CumulusCollator
} else {
ZombieRole::Collator
},
bootnodes_addr: &para_config
.bootnodes_addresses()
.iter()
.map(|&a| a.to_string())
.collect(),
chain_id: &self.relaychain().chain_id,
chain: &self.relaychain().chain,
ns: &self.ns,
scoped_fs: &scoped_fs,
wait_ready: false,
nodes_by_name: serde_json::to_value(&self.nodes_by_name)?,
global_settings: &self.initial_spec.global_settings,
};
// Register the parachain to the running network
let first_node_url = self
.relaychain()
.nodes
.first()
.ok_or(anyhow::anyhow!(
"At least one node of the relaychain should be running"
))?
.ws_uri();
if para_config.registration_strategy() == Some(&RegistrationStrategy::UsingExtrinsic) {
let register_para_options = RegisterParachainOptions {
id: parachain.para_id,
// This needs to resolve correctly
wasm_path: para_spec
.genesis_wasm
.artifact_path()
.ok_or(anyhow::anyhow!(
"artifact path for wasm must be set at this point",
))?
.to_path_buf(),
state_path: para_spec
.genesis_state
.artifact_path()
.ok_or(anyhow::anyhow!(
"artifact path for state must be set at this point",
))?
.to_path_buf(),
node_ws_url: first_node_url.to_string(),
onboard_as_para: para_spec.onboard_as_parachain,
seed: None, // TODO: Seed is passed by?
finalization: false,
};
Parachain::register(register_para_options, &scoped_fs).await?;
}
// Spawn the nodes
let spawning_tasks = para_spec
.collators
.iter()
.map(|node| spawner::spawn_node(node, parachain.files_to_inject.clone(), &ctx_para));
let running_nodes = futures::future::try_join_all(spawning_tasks).await?;
// Let's make sure nodes are up before adding them
let waiting_tasks = running_nodes.iter().map(|node| {
node.wait_until_is_up(self.initial_spec.global_settings.network_spawn_timeout())
});
let _ = futures::future::try_join_all(waiting_tasks).await?;
let running_para_id = parachain.para_id;
self.add_para(parachain);
for node in running_nodes {
self.add_running_node(node, Some(running_para_id)).await;
}
Ok(())
}
/// Register a parachain, which has already been added to the network (with manual registration
/// strategy)
///
/// # Arguments
/// * `para_id` - Parachain Id
///
///
/// # Example:
/// ```rust
/// # use anyhow::anyhow;
/// # use provider::NativeProvider;
/// # use support::{fs::local::LocalFileSystem};
/// # use zombienet_orchestrator::Orchestrator;
/// # use configuration::{NetworkConfig, NetworkConfigBuilder, RegistrationStrategy};
/// # async fn example() -> Result<(), anyhow::Error> {
/// # let provider = NativeProvider::new(LocalFileSystem {});
/// # let orchestrator = Orchestrator::new(LocalFileSystem {}, provider);
/// # let config = NetworkConfigBuilder::new()
/// # .with_relaychain(|r| {
/// # r.with_chain("rococo-local")
/// # .with_default_command("polkadot")
/// # .with_node(|node| node.with_name("alice"))
/// # })
/// # .with_parachain(|p| {
/// # p.with_id(100)
/// # .with_registration_strategy(RegistrationStrategy::Manual)
/// # .with_default_command("test-parachain")
/// # .with_collator(|n| n.with_name("dave").validator(false))
/// # })
/// # .build()
/// # .map_err(|_e| anyhow!("Building config"))?;
/// let mut network = orchestrator.spawn(config).await?;
///
/// network.register_parachain(100).await?;
///
/// # Ok(())
/// # }
/// ```
pub async fn register_parachain(&mut self, para_id: u32) -> Result<(), anyhow::Error> {
let para = self
.initial_spec
.parachains
.iter()
.find(|p| p.id == para_id)
.ok_or(anyhow::anyhow!(
"no parachain with id = {para_id} available",
))?;
let para_genesis_config = para.get_genesis_config()?;
let first_node_url = self
.relaychain()
.nodes
.first()
.ok_or(anyhow::anyhow!(
"At least one node of the relaychain should be running"
))?
.ws_uri();
let register_para_options: RegisterParachainOptions = RegisterParachainOptions {
id: para_id,
// This needs to resolve correctly
wasm_path: para_genesis_config.wasm_path.clone(),
state_path: para_genesis_config.state_path.clone(),
node_ws_url: first_node_url.to_string(),
onboard_as_para: para_genesis_config.as_parachain,
seed: None, // TODO: Seed is passed by?
finalization: false,
};
let base_dir = self.ns.base_dir().to_string_lossy().to_string();
let scoped_fs = ScopedFilesystem::new(&self.filesystem, &base_dir);
Parachain::register(register_para_options, &scoped_fs).await?;
Ok(())
}
// deregister and stop the collator?
// remove_parachain()
pub fn get_node(&self, name: impl Into<String>) -> Result<&NetworkNode, anyhow::Error> {
let name = name.into();
if let Some(node) = self.nodes_iter().find(|&n| n.name == name) {
return Ok(node);
}
let list = self
.nodes_iter()
.map(|n| &n.name)
.cloned()
.collect::<Vec<_>>()
.join(", ");
Err(anyhow::anyhow!(
"can't find node with name: {name:?}, should be one of {list}"
))
}
pub fn get_node_mut(
&mut self,
name: impl Into<String>,
) -> Result<&mut NetworkNode, anyhow::Error> {
let name = name.into();
self.nodes_iter_mut()
.find(|n| n.name == name)
.ok_or(anyhow::anyhow!("can't find node with name: {name:?}"))
}
pub fn nodes(&self) -> Vec<&NetworkNode> {
self.nodes_by_name.values().collect::<Vec<&NetworkNode>>()
}
pub async fn detach(&self) {
self.ns.detach().await
}
// Internal API
pub(crate) async fn add_running_node(&mut self, node: NetworkNode, para_id: Option<u32>) {
if let Some(para_id) = para_id {
if let Some(para) = self.parachains.get_mut(&para_id).and_then(|p| p.get_mut(0)) {
para.collators.push(node.clone());
} else {
// is the first node of the para, let create the entry
unreachable!()
}
} else {
self.relay.nodes.push(node.clone());
}
// TODO: we should hold a ref to the node in the vec in the future.
node.set_is_running(true);
let node_name = node.name.clone();
self.nodes_by_name.insert(node_name, node.clone());
self.nodes_to_watch.write().await.push(node);
}
pub(crate) fn add_para(&mut self, para: Parachain) {
self.parachains.entry(para.para_id).or_default().push(para);
}
pub fn name(&self) -> &str {
self.ns.name()
}
/// Get a first parachain from the list of the parachains with specified id.
/// NOTE!
/// Usually the list will contain only one parachain.
/// Multiple parachains with the same id is a corner case.
/// If this is the case then one can get such parachain with
/// `parachain_by_unique_id()` method
///
/// # Arguments
/// * `para_id` - Parachain Id
pub fn parachain(&self, para_id: u32) -> Option<&Parachain> {
self.parachains.get(&para_id)?.first()
}
/// Get a parachain by its unique id.
///
/// This is particularly useful if there are multiple parachains
/// with the same id (this is a rare corner case).
///
/// # Arguments
/// * `unique_id` - unique id of the parachain
pub fn parachain_by_unique_id(&self, unique_id: impl AsRef<str>) -> Option<&Parachain> {
self.parachains
.values()
.flat_map(|p| p.iter())
.find(|p| p.unique_id == unique_id.as_ref())
}
pub fn parachains(&self) -> Vec<&Parachain> {
self.parachains.values().flatten().collect()
}
pub(crate) fn nodes_iter(&self) -> impl Iterator<Item = &NetworkNode> {
self.relay.nodes.iter().chain(
self.parachains
.values()
.flat_map(|p| p.iter())
.flat_map(|p| &p.collators),
)
}
pub(crate) fn nodes_iter_mut(&mut self) -> impl Iterator<Item = &mut NetworkNode> {
self.relay.nodes.iter_mut().chain(
self.parachains
.values_mut()
.flat_map(|p| p.iter_mut())
.flat_map(|p| &mut p.collators),
)
}
/// Waits given number of seconds until all nodes in the network report that they are
/// up and running.
///
/// # Arguments
/// * `timeout_secs` - The number of seconds to wait.
///
/// # Returns
/// * `Ok()` if the node is up before timeout occured.
/// * `Err(e)` if timeout or other error occurred while waiting.
pub async fn wait_until_is_up(&self, timeout_secs: u64) -> Result<(), anyhow::Error> {
let handles = self
.nodes_iter()
.map(|node| node.wait_until_is_up(timeout_secs));
futures::future::try_join_all(handles).await?;
Ok(())
}
pub(crate) fn spawn_watching_task(&self) {
let nodes_to_watch = Arc::clone(&self.nodes_to_watch);
let ns = Arc::clone(&self.ns);
tokio::spawn(async move {
loop {
tokio::time::sleep(Duration::from_secs(NODE_MONITORING_INTERVAL_SECONDS)).await;
let all_running = {
let guard = nodes_to_watch.read().await;
let nodes = guard.iter().filter(|n| n.is_running()).collect::<Vec<_>>();
let all_running =
futures::future::try_join_all(nodes.iter().map(|n| {
n.wait_until_is_up(NODE_MONITORING_FAILURE_THRESHOLD_SECONDS)
}))
.await;
// Re-check `is_running` to make sure we don't kill the network unnecessarily
if nodes.iter().any(|n| !n.is_running()) {
continue;
} else {
all_running
}
};
if let Err(e) = all_running {
warn!("\n\t🧟 One of the nodes crashed: {e}. tearing the network down...");
if let Err(e) = ns.destroy().await {
error!("an error occurred during network teardown: {}", e);
}
std::process::exit(1);
}
}
});
}
pub(crate) fn set_parachains(&mut self, parachains: HashMap<u32, Vec<Parachain>>) {
self.parachains = parachains;
}
pub(crate) fn insert_node(&mut self, node: NetworkNode) {
self.nodes_by_name.insert(node.name.clone(), node);
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/network/chain_upgrade.rs
Vendored
+41
View File
@@ -0,0 +1,41 @@
use std::str::FromStr;
use anyhow::anyhow;
use async_trait::async_trait;
use pezkuwi_subxt_signer::{sr25519::Keypair, SecretUri};
use super::node::NetworkNode;
use crate::{shared::types::RuntimeUpgradeOptions, tx_helper};
#[async_trait]
pub trait ChainUpgrade {
/// Perform a runtime upgrade (with sudo)
///
/// This call 'System.set_code_without_checks' wrapped in
/// 'Sudo.sudo_unchecked_weight'
async fn runtime_upgrade(&self, options: RuntimeUpgradeOptions) -> Result<(), anyhow::Error>;
/// Perform a runtime upgrade (with sudo), inner call with the node pass as arg.
///
/// This call 'System.set_code_without_checks' wrapped in
/// 'Sudo.sudo_unchecked_weight'
async fn perform_runtime_upgrade(
&self,
node: &NetworkNode,
options: RuntimeUpgradeOptions,
) -> Result<(), anyhow::Error> {
let sudo = if let Some(possible_seed) = options.seed {
Keypair::from_secret_key(possible_seed)
.map_err(|_| anyhow!("seed should return a Keypair"))?
} else {
let uri = SecretUri::from_str("//Alice")?;
Keypair::from_uri(&uri).map_err(|_| anyhow!("'//Alice' should return a Keypair"))?
};
let wasm_data = options.wasm.get_asset().await?;
tx_helper::runtime_upgrade::upgrade(node, &wasm_data, &sudo).await?;
Ok(())
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/network/node.rs
Vendored
+1176
View File
File diff suppressed because it is too large Load Diff
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/network/relaychain.rs
Vendored
+75
View File
@@ -0,0 +1,75 @@
use std::path::PathBuf;
use anyhow::anyhow;
use async_trait::async_trait;
use serde::{Deserialize, Serialize};
use super::node::NetworkNode;
use crate::{
network::chain_upgrade::ChainUpgrade, shared::types::RuntimeUpgradeOptions,
utils::default_as_empty_vec,
};
#[derive(Debug, Serialize, Deserialize)]
pub struct Relaychain {
pub(crate) chain: String,
pub(crate) chain_id: String,
pub(crate) chain_spec_path: PathBuf,
#[serde(default, deserialize_with = "default_as_empty_vec")]
pub(crate) nodes: Vec<NetworkNode>,
}
#[derive(Debug, Deserialize)]
pub(crate) struct RawRelaychain {
#[serde(flatten)]
pub(crate) inner: Relaychain,
pub(crate) nodes: serde_json::Value,
}
#[async_trait]
impl ChainUpgrade for Relaychain {
async fn runtime_upgrade(&self, options: RuntimeUpgradeOptions) -> Result<(), anyhow::Error> {
// check if the node is valid first
let node = if let Some(node_name) = &options.node_name {
if let Some(node) = self
.nodes()
.into_iter()
.find(|node| node.name() == node_name)
{
node
} else {
return Err(anyhow!("Node: {node_name} is not part of the set of nodes"));
}
} else {
// take the first node
if let Some(node) = self.nodes().first() {
node
} else {
return Err(anyhow!("chain doesn't have any node!"));
}
};
self.perform_runtime_upgrade(node, options).await
}
}
impl Relaychain {
pub(crate) fn new(chain: String, chain_id: String, chain_spec_path: PathBuf) -> Self {
Self {
chain,
chain_id,
chain_spec_path,
nodes: Default::default(),
}
}
// Public API
pub fn nodes(&self) -> Vec<&NetworkNode> {
self.nodes.iter().collect()
}
/// Get chain name
pub fn chain(&self) -> &str {
&self.chain
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/network/teyrchain.rs
Vendored
+330
View File
@@ -0,0 +1,330 @@
use std::{
path::{Path, PathBuf},
str::FromStr,
};
use anyhow::anyhow;
use async_trait::async_trait;
use pezkuwi_subxt::{dynamic::Value, tx::TxStatus, BizinikiwConfig, OnlineClient};
use pezkuwi_subxt_signer::{sr25519::Keypair, SecretUri};
use provider::types::TransferedFile;
use serde::{Deserialize, Serialize};
use support::{constants::THIS_IS_A_BUG, fs::FileSystem, net::wait_ws_ready};
use tracing::info;
use super::{chain_upgrade::ChainUpgrade, node::NetworkNode};
use crate::{
network_spec::teyrchain::TeyrchainSpec,
shared::types::{RegisterParachainOptions, RuntimeUpgradeOptions},
tx_helper::client::get_client_from_url,
utils::default_as_empty_vec,
ScopedFilesystem,
};
#[derive(Debug, Serialize, Deserialize)]
pub struct Parachain {
pub(crate) chain: Option<String>,
pub(crate) para_id: u32,
// unique_id is internally used to allow multiple parachains with the same id
// See `ParachainConfig` for more details
pub(crate) unique_id: String,
pub(crate) chain_id: Option<String>,
pub(crate) chain_spec_path: Option<PathBuf>,
#[serde(default, deserialize_with = "default_as_empty_vec")]
pub(crate) collators: Vec<NetworkNode>,
pub(crate) files_to_inject: Vec<TransferedFile>,
pub(crate) bootnodes_addresses: Vec<multiaddr::Multiaddr>,
}
#[derive(Debug, Deserialize)]
pub(crate) struct RawParachain {
#[serde(flatten)]
pub(crate) inner: Parachain,
pub(crate) collators: serde_json::Value,
}
#[async_trait]
impl ChainUpgrade for Parachain {
async fn runtime_upgrade(&self, options: RuntimeUpgradeOptions) -> Result<(), anyhow::Error> {
// check if the node is valid first
let node = if let Some(node_name) = &options.node_name {
if let Some(node) = self
.collators()
.into_iter()
.find(|node| node.name() == node_name)
{
node
} else {
return Err(anyhow!("Node: {node_name} is not part of the set of nodes"));
}
} else {
// take the first node
if let Some(node) = self.collators().first() {
node
} else {
return Err(anyhow!("chain doesn't have any node!"));
}
};
self.perform_runtime_upgrade(node, options).await
}
}
impl Parachain {
pub(crate) fn new(para_id: u32, unique_id: impl Into<String>) -> Self {
Self {
chain: None,
para_id,
unique_id: unique_id.into(),
chain_id: None,
chain_spec_path: None,
collators: Default::default(),
files_to_inject: Default::default(),
bootnodes_addresses: vec![],
}
}
pub(crate) fn with_chain_spec(
para_id: u32,
unique_id: impl Into<String>,
chain_id: impl Into<String>,
chain_spec_path: impl AsRef<Path>,
) -> Self {
Self {
para_id,
unique_id: unique_id.into(),
chain: None,
chain_id: Some(chain_id.into()),
chain_spec_path: Some(chain_spec_path.as_ref().into()),
collators: Default::default(),
files_to_inject: Default::default(),
bootnodes_addresses: vec![],
}
}
pub(crate) async fn from_spec(
para: &TeyrchainSpec,
files_to_inject: &[TransferedFile],
scoped_fs: &ScopedFilesystem<'_, impl FileSystem>,
) -> Result<Self, anyhow::Error> {
let mut para_files_to_inject = files_to_inject.to_owned();
// parachain id is used for the keystore
let mut parachain = if let Some(chain_spec) = para.chain_spec.as_ref() {
let id = chain_spec.read_chain_id(scoped_fs).await?;
// add the spec to global files to inject
let spec_name = chain_spec.chain_spec_name();
let base = PathBuf::from_str(scoped_fs.base_dir)?;
para_files_to_inject.push(TransferedFile::new(
base.join(format!("{spec_name}.json")),
PathBuf::from(format!("/cfg/{}.json", para.id)),
));
let raw_path = chain_spec
.raw_path()
.ok_or(anyhow::anyhow!("chain-spec path should be set by now.",))?;
let mut running_para =
Parachain::with_chain_spec(para.id, &para.unique_id, id, raw_path);
if let Some(chain_name) = chain_spec.chain_name() {
running_para.chain = Some(chain_name.to_string());
}
running_para
} else {
Parachain::new(para.id, &para.unique_id)
};
parachain.bootnodes_addresses = para.bootnodes_addresses().into_iter().cloned().collect();
parachain.files_to_inject = para_files_to_inject;
Ok(parachain)
}
pub async fn register(
options: RegisterParachainOptions,
scoped_fs: &ScopedFilesystem<'_, impl FileSystem>,
) -> Result<(), anyhow::Error> {
info!("Registering parachain: {:?}", options);
// get the seed
let sudo: Keypair;
if let Some(possible_seed) = options.seed {
sudo = Keypair::from_secret_key(possible_seed)
.expect(&format!("seed should return a Keypair {THIS_IS_A_BUG}"));
} else {
let uri = SecretUri::from_str("//Alice")?;
sudo = Keypair::from_uri(&uri)?;
}
let genesis_state = scoped_fs
.read_to_string(options.state_path)
.await
.expect(&format!(
"State Path should be ok by this point {THIS_IS_A_BUG}"
));
let wasm_data = scoped_fs
.read_to_string(options.wasm_path)
.await
.expect(&format!(
"Wasm Path should be ok by this point {THIS_IS_A_BUG}"
));
wait_ws_ready(options.node_ws_url.as_str())
.await
.map_err(|_| {
anyhow::anyhow!(
"Error waiting for ws to be ready, at {}",
options.node_ws_url.as_str()
)
})?;
let api: OnlineClient<BizinikiwConfig> = get_client_from_url(&options.node_ws_url).await?;
let schedule_para = pezkuwi_subxt::dynamic::tx(
"ParasSudoWrapper",
"sudo_schedule_para_initialize",
vec![
Value::primitive(options.id.into()),
Value::named_composite([
(
"genesis_head",
Value::from_bytes(hex::decode(&genesis_state[2..])?),
),
(
"validation_code",
Value::from_bytes(hex::decode(&wasm_data[2..])?),
),
("para_kind", Value::bool(options.onboard_as_para)),
]),
],
);
let sudo_call =
pezkuwi_subxt::dynamic::tx("Sudo", "sudo", vec![schedule_para.into_value()]);
// TODO: uncomment below and fix the sign and submit (and follow afterwards until
// finalized block) to register the parachain
let mut tx = api
.tx()
.sign_and_submit_then_watch_default(&sudo_call, &sudo)
.await?;
// Below we use the low level API to replicate the `wait_for_in_block` behaviour
// which was removed in subxt 0.33.0. See https://github.com/paritytech/subxt/pull/1237.
while let Some(status) = tx.next().await {
match status? {
TxStatus::InBestBlock(tx_in_block) | TxStatus::InFinalizedBlock(tx_in_block) => {
let _result = tx_in_block.wait_for_success().await?;
info!("In block: {:#?}", tx_in_block.block_hash());
},
TxStatus::Error { message }
| TxStatus::Invalid { message }
| TxStatus::Dropped { message } => {
return Err(anyhow::format_err!("Error submitting tx: {message}"));
},
_ => continue,
}
}
Ok(())
}
pub fn para_id(&self) -> u32 {
self.para_id
}
pub fn unique_id(&self) -> &str {
self.unique_id.as_str()
}
pub fn chain_id(&self) -> Option<&str> {
self.chain_id.as_deref()
}
pub fn collators(&self) -> Vec<&NetworkNode> {
self.collators.iter().collect()
}
pub fn bootnodes_addresses(&self) -> Vec<&multiaddr::Multiaddr> {
self.bootnodes_addresses.iter().collect()
}
}
#[cfg(test)]
mod tests {
use std::collections::HashMap;
use super::*;
#[test]
fn create_with_is_works() {
let para = Parachain::new(100, "100");
// only para_id and unique_id should be set
assert_eq!(para.para_id, 100);
assert_eq!(para.unique_id, "100");
assert_eq!(para.chain_id, None);
assert_eq!(para.chain, None);
assert_eq!(para.chain_spec_path, None);
}
#[test]
fn create_with_chain_spec_works() {
let para = Parachain::with_chain_spec(100, "100", "rococo-local", "/tmp/rococo-local.json");
assert_eq!(para.para_id, 100);
assert_eq!(para.unique_id, "100");
assert_eq!(para.chain_id, Some("rococo-local".to_string()));
assert_eq!(para.chain, None);
assert_eq!(
para.chain_spec_path,
Some(PathBuf::from("/tmp/rococo-local.json"))
);
}
#[tokio::test]
async fn create_with_para_spec_works() {
use configuration::ParachainConfigBuilder;
use crate::network_spec::teyrchain::TeyrchainSpec;
let bootnode_addresses = vec!["/ip4/10.41.122.55/tcp/45421"];
let para_config = ParachainConfigBuilder::new(Default::default())
.with_id(100)
.cumulus_based(false)
.with_default_command("adder-collator")
.with_raw_bootnodes_addresses(bootnode_addresses.clone())
.with_collator(|c| c.with_name("col"))
.build()
.unwrap();
let para_spec =
TeyrchainSpec::from_config(&para_config, "rococo-local".try_into().unwrap()).unwrap();
let fs = support::fs::in_memory::InMemoryFileSystem::new(HashMap::default());
let scoped_fs = ScopedFilesystem {
fs: &fs,
base_dir: "/tmp/some",
};
let files = vec![TransferedFile::new(
PathBuf::from("/tmp/some"),
PathBuf::from("/tmp/some"),
)];
let para = Parachain::from_spec(&para_spec, &files, &scoped_fs)
.await
.unwrap();
println!("{para:#?}");
assert_eq!(para.para_id, 100);
assert_eq!(para.unique_id, "100");
assert_eq!(para.chain_id, None);
assert_eq!(para.chain, None);
// one file should be added.
assert_eq!(para.files_to_inject.len(), 1);
assert_eq!(
para.bootnodes_addresses()
.iter()
.map(|addr| addr.to_string())
.collect::<Vec<_>>(),
bootnode_addresses
);
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/network_helper.rs
Vendored
+2
View File
@@ -0,0 +1,2 @@
pub mod metrics;
pub mod verifier;
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/network_helper/metrics.rs
Vendored
+62
View File
@@ -0,0 +1,62 @@
use std::collections::HashMap;
use async_trait::async_trait;
use reqwest::Url;
#[async_trait]
pub trait MetricsHelper {
async fn metric(&self, metric_name: &str) -> Result<f64, anyhow::Error>;
async fn metric_with_url(
metric: impl AsRef<str> + Send,
endpoint: impl Into<Url> + Send,
) -> Result<f64, anyhow::Error>;
}
pub struct Metrics {
endpoint: Url,
}
impl Metrics {
fn new(endpoint: impl Into<Url>) -> Self {
Self {
endpoint: endpoint.into(),
}
}
async fn fetch_metrics(
endpoint: impl AsRef<str>,
) -> Result<HashMap<String, f64>, anyhow::Error> {
let response = reqwest::get(endpoint.as_ref()).await?;
Ok(prom_metrics_parser::parse(&response.text().await?)?)
}
fn get_metric(
metrics_map: HashMap<String, f64>,
metric_name: &str,
) -> Result<f64, anyhow::Error> {
let treat_not_found_as_zero = true;
if let Some(val) = metrics_map.get(metric_name) {
Ok(*val)
} else if treat_not_found_as_zero {
Ok(0_f64)
} else {
Err(anyhow::anyhow!("MetricNotFound: {metric_name}"))
}
}
}
#[async_trait]
impl MetricsHelper for Metrics {
async fn metric(&self, metric_name: &str) -> Result<f64, anyhow::Error> {
let metrics_map = Metrics::fetch_metrics(self.endpoint.as_str()).await?;
Metrics::get_metric(metrics_map, metric_name)
}
async fn metric_with_url(
metric_name: impl AsRef<str> + Send,
endpoint: impl Into<Url> + Send,
) -> Result<f64, anyhow::Error> {
let metrics_map = Metrics::fetch_metrics(endpoint.into()).await?;
Metrics::get_metric(metrics_map, metric_name.as_ref())
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/network_helper/verifier.rs
Vendored
+34
View File
@@ -0,0 +1,34 @@
use std::time::Duration;
use tokio::time::timeout;
use tracing::trace;
use crate::network::node::NetworkNode;
pub(crate) async fn verify_nodes(nodes: &[&NetworkNode]) -> Result<(), anyhow::Error> {
timeout(Duration::from_secs(90), check_nodes(nodes))
.await
.map_err(|_| anyhow::anyhow!("one or more nodes are not ready!"))
}
// TODO: we should inject in someway the logic to make the request
// in order to allow us to `mock` and easily test this.
// maybe moved to the provider with a NodeStatus, and some helpers like wait_running, wait_ready, etc... ? to be discussed
async fn check_nodes(nodes: &[&NetworkNode]) {
loop {
let tasks: Vec<_> = nodes
.iter()
.map(|node| {
trace!("🔎 checking node: {} ", node.name);
reqwest::get(node.prometheus_uri.clone())
})
.collect();
let all_ready = futures::future::try_join_all(tasks).await;
if all_ready.is_ok() {
return;
}
tokio::time::sleep(Duration::from_millis(1000)).await;
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/network_spec.rs
Vendored
+330
View File
@@ -0,0 +1,330 @@
use std::{
collections::{hash_map::Entry, HashMap},
sync::Arc,
};
use configuration::{GlobalSettings, HrmpChannelConfig, NetworkConfig};
use futures::future::try_join_all;
use provider::{DynNamespace, ProviderError, ProviderNamespace};
use serde::{Deserialize, Serialize};
use support::{constants::THIS_IS_A_BUG, fs::FileSystem};
use tracing::{debug, trace};
use crate::{errors::OrchestratorError, ScopedFilesystem};
pub mod node;
pub mod relaychain;
pub mod teyrchain;
use self::{node::NodeSpec, relaychain::RelaychainSpec, teyrchain::TeyrchainSpec};
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct NetworkSpec {
/// Relaychain configuration.
pub(crate) relaychain: RelaychainSpec,
/// Parachains configurations.
pub(crate) parachains: Vec<TeyrchainSpec>,
/// HRMP channels configurations.
pub(crate) hrmp_channels: Vec<HrmpChannelConfig>,
/// Global settings
pub(crate) global_settings: GlobalSettings,
}
impl NetworkSpec {
pub async fn from_config(
network_config: &NetworkConfig,
) -> Result<NetworkSpec, OrchestratorError> {
let mut errs = vec![];
let relaychain = RelaychainSpec::from_config(network_config.relaychain())?;
let mut parachains = vec![];
// TODO: move to `fold` or map+fold
for para_config in network_config.parachains() {
match TeyrchainSpec::from_config(para_config, relaychain.chain.clone()) {
Ok(para) => parachains.push(para),
Err(err) => errs.push(err),
}
}
if errs.is_empty() {
Ok(NetworkSpec {
relaychain,
parachains,
hrmp_channels: network_config
.hrmp_channels()
.into_iter()
.cloned()
.collect(),
global_settings: network_config.global_settings().clone(),
})
} else {
let errs_str = errs
.into_iter()
.map(|e| e.to_string())
.collect::<Vec<String>>()
.join("\n");
Err(OrchestratorError::InvalidConfig(errs_str))
}
}
pub async fn populate_nodes_available_args(
&mut self,
ns: Arc<dyn ProviderNamespace + Send + Sync>,
) -> Result<(), OrchestratorError> {
let network_nodes = self.collect_network_nodes();
let mut image_command_to_nodes_mapping =
Self::create_image_command_to_nodes_mapping(network_nodes);
let available_args_outputs =
Self::retrieve_all_nodes_available_args_output(ns, &image_command_to_nodes_mapping)
.await?;
Self::update_nodes_available_args_output(
&mut image_command_to_nodes_mapping,
available_args_outputs,
);
Ok(())
}
//
pub async fn node_available_args_output(
&self,
node_spec: &NodeSpec,
ns: Arc<dyn ProviderNamespace + Send + Sync>,
) -> Result<String, ProviderError> {
// try to find a node that use the same combination of image/cmd
let cmp_fn = |ad_hoc: &&NodeSpec| -> bool {
ad_hoc.image == node_spec.image && ad_hoc.command == node_spec.command
};
// check if we already had computed the args output for this cmd/[image]
let node = self.relaychain.nodes.iter().find(cmp_fn);
let node = if let Some(node) = node {
Some(node)
} else {
let node = self
.parachains
.iter()
.find_map(|para| para.collators.iter().find(cmp_fn));
node
};
let output = if let Some(node) = node {
node.available_args_output.clone().expect(&format!(
"args_output should be set for running nodes {THIS_IS_A_BUG}"
))
} else {
// we need to compute the args output
let image = node_spec
.image
.as_ref()
.map(|image| image.as_str().to_string());
let command = node_spec.command.as_str().to_string();
ns.get_node_available_args((command, image)).await?
};
Ok(output)
}
pub fn relaychain(&self) -> &RelaychainSpec {
&self.relaychain
}
pub fn relaychain_mut(&mut self) -> &mut RelaychainSpec {
&mut self.relaychain
}
pub fn parachains_iter(&self) -> impl Iterator<Item = &TeyrchainSpec> {
self.parachains.iter()
}
pub fn parachains_iter_mut(&mut self) -> impl Iterator<Item = &mut TeyrchainSpec> {
self.parachains.iter_mut()
}
pub fn set_global_settings(&mut self, global_settings: GlobalSettings) {
self.global_settings = global_settings;
}
pub async fn build_parachain_artifacts<'a, T: FileSystem>(
&mut self,
ns: DynNamespace,
scoped_fs: &ScopedFilesystem<'a, T>,
relaychain_id: &str,
base_dir_exists: bool,
) -> Result<(), anyhow::Error> {
for para in self.parachains.iter_mut() {
let chain_spec_raw_path = para.build_chain_spec(relaychain_id, &ns, scoped_fs).await?;
trace!("creating dirs for {}", &para.unique_id);
if base_dir_exists {
scoped_fs.create_dir_all(&para.unique_id).await?;
} else {
scoped_fs.create_dir(&para.unique_id).await?;
};
trace!("created dirs for {}", &para.unique_id);
// create wasm/state
para.genesis_state
.build(
chain_spec_raw_path.clone(),
format!("{}/genesis-state", para.unique_id),
&ns,
scoped_fs,
None,
)
.await?;
debug!("parachain genesis state built!");
para.genesis_wasm
.build(
chain_spec_raw_path,
format!("{}/genesis-wasm", para.unique_id),
&ns,
scoped_fs,
None,
)
.await?;
debug!("parachain genesis wasm built!");
}
Ok(())
}
// collect mutable references to all nodes from relaychain and parachains
fn collect_network_nodes(&mut self) -> Vec<&mut NodeSpec> {
vec![
self.relaychain.nodes.iter_mut().collect::<Vec<_>>(),
self.parachains
.iter_mut()
.flat_map(|para| para.collators.iter_mut())
.collect(),
]
.into_iter()
.flatten()
.collect::<Vec<_>>()
}
// initialize the mapping of all possible node image/commands to corresponding nodes
fn create_image_command_to_nodes_mapping(
network_nodes: Vec<&mut NodeSpec>,
) -> HashMap<(Option<String>, String), Vec<&mut NodeSpec>> {
network_nodes.into_iter().fold(
HashMap::new(),
|mut acc: HashMap<(Option<String>, String), Vec<&mut node::NodeSpec>>, node| {
// build mapping key using image and command if image is present or command only
let key = node
.image
.as_ref()
.map(|image| {
(
Some(image.as_str().to_string()),
node.command.as_str().to_string(),
)
})
.unwrap_or_else(|| (None, node.command.as_str().to_string()));
// append the node to the vector of nodes for this image/command tuple
if let Entry::Vacant(entry) = acc.entry(key.clone()) {
entry.insert(vec![node]);
} else {
acc.get_mut(&key).unwrap().push(node);
}
acc
},
)
}
async fn retrieve_all_nodes_available_args_output(
ns: Arc<dyn ProviderNamespace + Send + Sync>,
image_command_to_nodes_mapping: &HashMap<(Option<String>, String), Vec<&mut NodeSpec>>,
) -> Result<Vec<(Option<String>, String, String)>, OrchestratorError> {
try_join_all(
image_command_to_nodes_mapping
.keys()
.map(|(image, command)| {
let ns = ns.clone();
let image = image.clone();
let command = command.clone();
async move {
// get node available args output from image/command
let available_args = ns
.get_node_available_args((command.clone(), image.clone()))
.await?;
debug!(
"retrieved available args for image: {:?}, command: {}",
image, command
);
// map the result to include image and command
Ok::<_, OrchestratorError>((image, command, available_args))
}
})
.collect::<Vec<_>>(),
)
.await
}
fn update_nodes_available_args_output(
image_command_to_nodes_mapping: &mut HashMap<(Option<String>, String), Vec<&mut NodeSpec>>,
available_args_outputs: Vec<(Option<String>, String, String)>,
) {
for (image, command, available_args_output) in available_args_outputs {
let nodes = image_command_to_nodes_mapping
.get_mut(&(image, command))
.expect(&format!(
"node image/command key should exist {THIS_IS_A_BUG}"
));
for node in nodes {
node.available_args_output = Some(available_args_output.clone());
}
}
}
}
#[cfg(test)]
mod tests {
#[tokio::test]
async fn small_network_config_get_spec() {
use configuration::NetworkConfigBuilder;
use super::*;
let config = NetworkConfigBuilder::new()
.with_relaychain(|r| {
r.with_chain("rococo-local")
.with_default_command("polkadot")
.with_validator(|node| node.with_name("alice"))
.with_fullnode(|node| node.with_name("bob").with_command("polkadot1"))
})
.with_parachain(|p| {
p.with_id(100)
.with_default_command("adder-collator")
.with_collator(|c| c.with_name("collator1"))
})
.build()
.unwrap();
let network_spec = NetworkSpec::from_config(&config).await.unwrap();
let alice = network_spec.relaychain.nodes.first().unwrap();
let bob = network_spec.relaychain.nodes.get(1).unwrap();
assert_eq!(alice.command.as_str(), "polkadot");
assert_eq!(bob.command.as_str(), "polkadot1");
assert!(alice.is_validator);
assert!(!bob.is_validator);
// paras
assert_eq!(network_spec.parachains.len(), 1);
let para_100 = network_spec.parachains.first().unwrap();
assert_eq!(para_100.id, 100);
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/network_spec/node.rs
Vendored
+356
View File
@@ -0,0 +1,356 @@
use std::path::PathBuf;
use configuration::shared::{
node::{EnvVar, NodeConfig},
resources::Resources,
types::{Arg, AssetLocation, Command, Image},
};
use multiaddr::Multiaddr;
use provider::types::Port;
use serde::{Deserialize, Serialize};
use support::constants::THIS_IS_A_BUG;
use crate::{
errors::OrchestratorError,
generators,
network::AddNodeOptions,
shared::{
macros,
types::{ChainDefaultContext, NodeAccount, NodeAccounts, ParkedPort},
},
AddCollatorOptions,
};
macros::create_add_options!(AddNodeSpecOpts {
override_eth_key: Option<String>
});
macro_rules! impl_from_for_add_node_opts {
($struct:ident) => {
impl From<$struct> for AddNodeSpecOpts {
fn from(value: $struct) -> Self {
Self {
image: value.image,
command: value.command,
subcommand: value.subcommand,
args: value.args,
env: value.env,
is_validator: value.is_validator,
rpc_port: value.rpc_port,
prometheus_port: value.prometheus_port,
p2p_port: value.p2p_port,
override_eth_key: value.override_eth_key,
}
}
}
};
}
impl_from_for_add_node_opts!(AddNodeOptions);
impl_from_for_add_node_opts!(AddCollatorOptions);
/// A node configuration, with fine-grained configuration options.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct NodeSpec {
// Node name (should be unique or an index will be appended).
pub(crate) name: String,
/// Node key, used for compute the p2p identity.
pub(crate) key: String,
// libp2p local identity
pub(crate) peer_id: String,
/// Accounts to be injected in the keystore.
pub(crate) accounts: NodeAccounts,
/// Image to run (only podman/k8s). Override the default.
pub(crate) image: Option<Image>,
/// Command to run the node. Override the default.
pub(crate) command: Command,
/// Optional subcommand for the node.
pub(crate) subcommand: Option<Command>,
/// Arguments to use for node. Appended to default.
pub(crate) args: Vec<Arg>,
// The help command output containing the available arguments.
pub(crate) available_args_output: Option<String>,
/// Wether the node is a validator.
pub(crate) is_validator: bool,
/// Whether the node keys must be added to invulnerables.
pub(crate) is_invulnerable: bool,
/// Whether the node is a bootnode.
pub(crate) is_bootnode: bool,
/// Node initial balance present in genesis.
pub(crate) initial_balance: u128,
/// Environment variables to set (inside pod for podman/k8s, inside shell for native).
pub(crate) env: Vec<EnvVar>,
/// List of node's bootnodes addresses to use. Appended to default.
pub(crate) bootnodes_addresses: Vec<Multiaddr>,
/// Default resources. Override the default.
pub(crate) resources: Option<Resources>,
/// Websocket port to use.
pub(crate) ws_port: ParkedPort,
/// RPC port to use.
pub(crate) rpc_port: ParkedPort,
/// Prometheus port to use.
pub(crate) prometheus_port: ParkedPort,
/// P2P port to use.
pub(crate) p2p_port: ParkedPort,
/// libp2p cert hash to use with `webrtc` transport.
pub(crate) p2p_cert_hash: Option<String>,
/// Database snapshot. Override the default.
pub(crate) db_snapshot: Option<AssetLocation>,
/// P2P port to use by full node if this is the case
pub(crate) full_node_p2p_port: Option<ParkedPort>,
/// Prometheus port to use by full node if this is the case
pub(crate) full_node_prometheus_port: Option<ParkedPort>,
/// Optionally specify a log path for the node
pub(crate) node_log_path: Option<PathBuf>,
/// Optionally specify a keystore path for the node
pub(crate) keystore_path: Option<PathBuf>,
/// Keystore key types to generate.
/// Supports short form (e.g., "audi") using predefined schemas,
/// or long form (e.g., "audi_sr") with explicit schema (sr, ed, ec).
pub(crate) keystore_key_types: Vec<String>,
}
impl NodeSpec {
pub fn from_config(
node_config: &NodeConfig,
chain_context: &ChainDefaultContext,
full_node_present: bool,
evm_based: bool,
) -> Result<Self, OrchestratorError> {
// Check first if the image is set at node level, then try with the default
let image = node_config.image().or(chain_context.default_image).cloned();
// Check first if the command is set at node level, then try with the default
let command = if let Some(cmd) = node_config.command() {
cmd.clone()
} else if let Some(cmd) = chain_context.default_command {
cmd.clone()
} else {
return Err(OrchestratorError::InvalidNodeConfig(
node_config.name().into(),
"command".to_string(),
));
};
let subcommand = node_config.subcommand().cloned();
// If `args` is set at `node` level use them
// otherwise use the default_args (can be empty).
let args: Vec<Arg> = if node_config.args().is_empty() {
chain_context
.default_args
.iter()
.map(|x| x.to_owned().clone())
.collect()
} else {
node_config.args().into_iter().cloned().collect()
};
let (key, peer_id) = generators::generate_node_identity(node_config.name())?;
let mut name = node_config.name().to_string();
let seed = format!("//{}{name}", name.remove(0).to_uppercase());
let accounts = generators::generate_node_keys(&seed)?;
let mut accounts = NodeAccounts { seed, accounts };
if evm_based {
if let Some(session_key) = node_config.override_eth_key() {
accounts
.accounts
.insert("eth".into(), NodeAccount::new(session_key, session_key));
}
}
let db_snapshot = match (node_config.db_snapshot(), chain_context.default_db_snapshot) {
(Some(db_snapshot), _) => Some(db_snapshot),
(None, Some(db_snapshot)) => Some(db_snapshot),
_ => None,
};
let (full_node_p2p_port, full_node_prometheus_port) = if full_node_present {
(
Some(generators::generate_node_port(None)?),
Some(generators::generate_node_port(None)?),
)
} else {
(None, None)
};
Ok(Self {
name: node_config.name().to_string(),
key,
peer_id,
image,
command,
subcommand,
args,
available_args_output: None,
is_validator: node_config.is_validator(),
is_invulnerable: node_config.is_invulnerable(),
is_bootnode: node_config.is_bootnode(),
initial_balance: node_config.initial_balance(),
env: node_config.env().into_iter().cloned().collect(),
bootnodes_addresses: node_config
.bootnodes_addresses()
.into_iter()
.cloned()
.collect(),
resources: node_config.resources().cloned(),
p2p_cert_hash: node_config.p2p_cert_hash().map(str::to_string),
db_snapshot: db_snapshot.cloned(),
accounts,
ws_port: generators::generate_node_port(node_config.ws_port())?,
rpc_port: generators::generate_node_port(node_config.rpc_port())?,
prometheus_port: generators::generate_node_port(node_config.prometheus_port())?,
p2p_port: generators::generate_node_port(node_config.p2p_port())?,
full_node_p2p_port,
full_node_prometheus_port,
node_log_path: node_config.node_log_path().cloned(),
keystore_path: node_config.keystore_path().cloned(),
keystore_key_types: node_config
.keystore_key_types()
.into_iter()
.map(str::to_string)
.collect(),
})
}
pub fn from_ad_hoc(
name: impl Into<String>,
options: AddNodeSpecOpts,
chain_context: &ChainDefaultContext,
full_node_present: bool,
evm_based: bool,
) -> Result<Self, OrchestratorError> {
// Check first if the image is set at node level, then try with the default
let image = if let Some(img) = options.image {
Some(img.clone())
} else {
chain_context.default_image.cloned()
};
let name = name.into();
// Check first if the command is set at node level, then try with the default
let command = if let Some(cmd) = options.command {
cmd.clone()
} else if let Some(cmd) = chain_context.default_command {
cmd.clone()
} else {
return Err(OrchestratorError::InvalidNodeConfig(
name,
"command".to_string(),
));
};
let subcommand = options.subcommand.clone();
// If `args` is set at `node` level use them
// otherwise use the default_args (can be empty).
let args: Vec<Arg> = if options.args.is_empty() {
chain_context
.default_args
.iter()
.map(|x| x.to_owned().clone())
.collect()
} else {
options.args
};
let (key, peer_id) = generators::generate_node_identity(&name)?;
let mut name_capitalized = name.clone();
let seed = format!(
"//{}{name_capitalized}",
name_capitalized.remove(0).to_uppercase()
);
let accounts = generators::generate_node_keys(&seed)?;
let mut accounts = NodeAccounts { seed, accounts };
if evm_based {
if let Some(session_key) = options.override_eth_key.as_ref() {
accounts
.accounts
.insert("eth".into(), NodeAccount::new(session_key, session_key));
}
}
let (full_node_p2p_port, full_node_prometheus_port) = if full_node_present {
(
Some(generators::generate_node_port(None)?),
Some(generators::generate_node_port(None)?),
)
} else {
(None, None)
};
//
Ok(Self {
name,
key,
peer_id,
image,
command,
subcommand,
args,
available_args_output: None,
is_validator: options.is_validator,
is_invulnerable: false,
is_bootnode: false,
initial_balance: 0,
env: options.env,
bootnodes_addresses: vec![],
resources: None,
p2p_cert_hash: None,
db_snapshot: None,
accounts,
// should be deprecated now!
ws_port: generators::generate_node_port(None)?,
rpc_port: generators::generate_node_port(options.rpc_port)?,
prometheus_port: generators::generate_node_port(options.prometheus_port)?,
p2p_port: generators::generate_node_port(options.p2p_port)?,
full_node_p2p_port,
full_node_prometheus_port,
node_log_path: None,
keystore_path: None,
keystore_key_types: vec![],
})
}
pub(crate) fn supports_arg(&self, arg: impl AsRef<str>) -> bool {
self.available_args_output
.as_ref()
.expect(&format!(
"available args should be present at this point {THIS_IS_A_BUG}"
))
.contains(arg.as_ref())
}
pub fn command(&self) -> &str {
self.command.as_str()
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/network_spec/relaychain.rs
Vendored
+181
View File
@@ -0,0 +1,181 @@
use std::collections::{HashMap, HashSet};
use configuration::{
shared::{
helpers::generate_unique_node_name_from_names,
resources::Resources,
types::{Arg, AssetLocation, Chain, Command, Image},
},
types::JsonOverrides,
NodeConfig, RelaychainConfig,
};
use serde::{Deserialize, Serialize};
use support::replacer::apply_replacements;
use super::node::NodeSpec;
use crate::{
errors::OrchestratorError,
generators::chain_spec::{ChainSpec, Context},
shared::{constants::DEFAULT_CHAIN_SPEC_TPL_COMMAND, types::ChainDefaultContext},
};
/// A relaychain configuration spec
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RelaychainSpec {
/// Chain to use (e.g. rococo-local).
pub(crate) chain: Chain,
/// Default command to run the node. Can be overridden on each node.
pub(crate) default_command: Option<Command>,
/// Default image to use (only podman/k8s). Can be overridden on each node.
pub(crate) default_image: Option<Image>,
/// Default resources. Can be overridden on each node.
pub(crate) default_resources: Option<Resources>,
/// Default database snapshot. Can be overridden on each node.
pub(crate) default_db_snapshot: Option<AssetLocation>,
/// Default arguments to use in nodes. Can be overridden on each node.
pub(crate) default_args: Vec<Arg>,
// chain_spec_path: Option<AssetLocation>,
pub(crate) chain_spec: ChainSpec,
/// Set the count of nominators to generator (used with PoS networks).
pub(crate) random_nominators_count: u32,
/// Set the max nominators value (used with PoS networks).
pub(crate) max_nominations: u8,
/// Genesis overrides as JSON value.
pub(crate) runtime_genesis_patch: Option<serde_json::Value>,
/// Wasm override path/url to use.
pub(crate) wasm_override: Option<AssetLocation>,
/// Nodes to run.
pub(crate) nodes: Vec<NodeSpec>,
/// Raw chain-spec override path, url or inline json to use.
pub(crate) raw_spec_override: Option<JsonOverrides>,
}
impl RelaychainSpec {
pub fn from_config(config: &RelaychainConfig) -> Result<RelaychainSpec, OrchestratorError> {
// Relaychain main command to use, in order:
// set as `default_command` or
// use the command of the first node.
// If non of those is set, return an error.
let main_cmd = config
.default_command()
.or(config.nodes().first().and_then(|node| node.command()))
.ok_or(OrchestratorError::InvalidConfig(
"Relaychain, either default_command or first node with a command needs to be set."
.to_string(),
))?;
// TODO: internally we use image as String
let main_image = config
.default_image()
.or(config.nodes().first().and_then(|node| node.image()))
.map(|image| image.as_str().to_string());
let replacements = HashMap::from([
("disableBootnodes", "--disable-default-bootnode"),
("mainCommand", main_cmd.as_str()),
]);
let tmpl = if let Some(tmpl) = config.chain_spec_command() {
apply_replacements(tmpl, &replacements)
} else {
apply_replacements(DEFAULT_CHAIN_SPEC_TPL_COMMAND, &replacements)
};
let chain_spec = ChainSpec::new(config.chain().as_str(), Context::Relay)
.set_chain_name(config.chain().as_str())
.command(
tmpl.as_str(),
config.chain_spec_command_is_local(),
config.chain_spec_command_output_path(),
)
.image(main_image.clone());
// Add asset location if present
let chain_spec = if let Some(chain_spec_path) = config.chain_spec_path() {
chain_spec.asset_location(chain_spec_path.clone())
} else {
chain_spec
};
// add chain-spec runtime if present
let chain_spec = if let Some(chain_spec_runtime) = config.chain_spec_runtime() {
chain_spec.runtime(chain_spec_runtime.clone())
} else {
chain_spec
};
// build the `node_specs`
let chain_context = ChainDefaultContext {
default_command: config.default_command(),
default_image: config.default_image(),
default_resources: config.default_resources(),
default_db_snapshot: config.default_db_snapshot(),
default_args: config.default_args(),
};
let mut nodes: Vec<NodeConfig> = config.nodes().into_iter().cloned().collect();
nodes.extend(
config
.group_node_configs()
.into_iter()
.flat_map(|node_group| node_group.expand_group_configs()),
);
let mut names = HashSet::new();
let (nodes, mut errs) = nodes
.iter()
.map(|node_config| NodeSpec::from_config(node_config, &chain_context, false, false))
.fold((vec![], vec![]), |(mut nodes, mut errs), result| {
match result {
Ok(mut node) => {
let unique_name =
generate_unique_node_name_from_names(node.name, &mut names);
node.name = unique_name;
nodes.push(node);
},
Err(err) => errs.push(err),
}
(nodes, errs)
});
if !errs.is_empty() {
// TODO: merge errs, maybe return something like Result<Sometype, Vec<OrchestratorError>>
return Err(errs.swap_remove(0));
}
Ok(RelaychainSpec {
chain: config.chain().clone(),
default_command: config.default_command().cloned(),
default_image: config.default_image().cloned(),
default_resources: config.default_resources().cloned(),
default_db_snapshot: config.default_db_snapshot().cloned(),
wasm_override: config.wasm_override().cloned(),
default_args: config.default_args().into_iter().cloned().collect(),
chain_spec,
random_nominators_count: config.random_nominators_count().unwrap_or(0),
max_nominations: config.max_nominations().unwrap_or(24),
runtime_genesis_patch: config.runtime_genesis_patch().cloned(),
nodes,
raw_spec_override: config.raw_spec_override().cloned(),
})
}
pub fn chain_spec(&self) -> &ChainSpec {
&self.chain_spec
}
pub fn chain_spec_mut(&mut self) -> &mut ChainSpec {
&mut self.chain_spec
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/network_spec/teyrchain.rs
Vendored
+386
View File
@@ -0,0 +1,386 @@
use std::{
collections::{HashMap, HashSet},
path::PathBuf,
};
use configuration::{
shared::{helpers::generate_unique_node_name_from_names, resources::Resources},
types::{Arg, AssetLocation, Chain, Command, Image, JsonOverrides},
NodeConfig, ParachainConfig, RegistrationStrategy,
};
use provider::DynNamespace;
use serde::{Deserialize, Serialize};
use support::{fs::FileSystem, replacer::apply_replacements};
use tracing::debug;
use super::node::NodeSpec;
use crate::{
errors::OrchestratorError,
generators::{
chain_spec::{ChainSpec, Context, ParaGenesisConfig},
para_artifact::*,
},
shared::{constants::DEFAULT_CHAIN_SPEC_TPL_COMMAND, types::ChainDefaultContext},
ScopedFilesystem,
};
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TeyrchainSpec {
// `name` of the parachain (used in some corner cases)
// name: Option<Chain>,
/// Parachain id
pub(crate) id: u32,
/// Unique id of the parachain, in the patter of <para_id>-<n>
/// where the suffix is only present if more than one parachain is set with the same id
pub(crate) unique_id: String,
/// Default command to run the node. Can be overridden on each node.
pub(crate) default_command: Option<Command>,
/// Default image to use (only podman/k8s). Can be overridden on each node.
pub(crate) default_image: Option<Image>,
/// Default resources. Can be overridden on each node.
pub(crate) default_resources: Option<Resources>,
/// Default database snapshot. Can be overridden on each node.
pub(crate) default_db_snapshot: Option<AssetLocation>,
/// Default arguments to use in nodes. Can be overridden on each node.
pub(crate) default_args: Vec<Arg>,
/// Chain-spec, only needed by cumulus based paras
pub(crate) chain_spec: Option<ChainSpec>,
/// Do not automatically assign a bootnode role if no nodes are marked as bootnodes.
pub(crate) no_default_bootnodes: bool,
/// Registration strategy to use
pub(crate) registration_strategy: RegistrationStrategy,
/// Onboard as parachain or parathread
pub(crate) onboard_as_parachain: bool,
/// Is the parachain cumulus-based
pub(crate) is_cumulus_based: bool,
/// Is the parachain evm-based
pub(crate) is_evm_based: bool,
/// Initial balance
pub(crate) initial_balance: u128,
/// Genesis state (head) to register the parachain
pub(crate) genesis_state: ParaArtifact,
/// Genesis WASM to register the parachain
pub(crate) genesis_wasm: ParaArtifact,
/// Genesis overrides as JSON value.
pub(crate) genesis_overrides: Option<serde_json::Value>,
/// Wasm override path/url to use.
pub(crate) wasm_override: Option<AssetLocation>,
/// Collators to spawn
pub(crate) collators: Vec<NodeSpec>,
/// Raw chain-spec override path, url or inline json to use.
pub(crate) raw_spec_override: Option<JsonOverrides>,
/// Bootnodes addresses to use for the parachain nodes
pub(crate) bootnodes_addresses: Vec<multiaddr::Multiaddr>,
}
impl TeyrchainSpec {
pub fn from_config(
config: &ParachainConfig,
relay_chain: Chain,
) -> Result<TeyrchainSpec, OrchestratorError> {
let main_cmd = if let Some(cmd) = config.default_command() {
cmd
} else if let Some(first_node) = config.collators().first() {
let Some(cmd) = first_node.command() else {
return Err(OrchestratorError::InvalidConfig(format!("Parachain {}, either default_command or command in the first node needs to be set.", config.id())));
};
cmd
} else {
return Err(OrchestratorError::InvalidConfig(format!(
"Parachain {}, without nodes and default_command isn't set.",
config.id()
)));
};
// TODO: internally we use image as String
let main_image = config
.default_image()
.or(config.collators().first().and_then(|node| node.image()))
.map(|image| image.as_str().to_string());
let chain_spec = if config.is_cumulus_based() {
// we need a chain-spec
let chain_name = if let Some(chain_name) = config.chain() {
chain_name.as_str()
} else {
""
};
let chain_spec_builder = if chain_name.is_empty() {
// if the chain don't have name use the unique_id for the name of the file
ChainSpec::new(
config.unique_id().to_string(),
Context::Para {
relay_chain,
para_id: config.id(),
},
)
} else {
let chain_spec_file_name = if config.unique_id().contains('-') {
&format!("{}-{}", chain_name, config.unique_id())
} else {
chain_name
};
ChainSpec::new(
chain_spec_file_name,
Context::Para {
relay_chain,
para_id: config.id(),
},
)
};
let chain_spec_builder = chain_spec_builder.set_chain_name(chain_name);
let replacements = HashMap::from([
("disableBootnodes", "--disable-default-bootnode"),
("mainCommand", main_cmd.as_str()),
]);
let tmpl = if let Some(tmpl) = config.chain_spec_command() {
apply_replacements(tmpl, &replacements)
} else {
apply_replacements(DEFAULT_CHAIN_SPEC_TPL_COMMAND, &replacements)
};
let chain_spec = chain_spec_builder
.command(
tmpl.as_str(),
config.chain_spec_command_is_local(),
config.chain_spec_command_output_path(),
)
.image(main_image.clone());
let chain_spec = if let Some(chain_spec_path) = config.chain_spec_path() {
chain_spec.asset_location(chain_spec_path.clone())
} else {
chain_spec
};
// add chain-spec runtime if present
let chain_spec = if let Some(chain_spec_runtime) = config.chain_spec_runtime() {
chain_spec.runtime(chain_spec_runtime.clone())
} else {
chain_spec
};
Some(chain_spec)
} else {
None
};
// build the `node_specs`
let chain_context = ChainDefaultContext {
default_command: config.default_command(),
default_image: config.default_image(),
default_resources: config.default_resources(),
default_db_snapshot: config.default_db_snapshot(),
default_args: config.default_args(),
};
// We want to track the errors for all the nodes and report them ones
let mut errs: Vec<OrchestratorError> = Default::default();
let mut collators: Vec<NodeSpec> = Default::default();
let mut nodes: Vec<NodeConfig> = config.collators().into_iter().cloned().collect();
nodes.extend(
config
.group_collators_configs()
.into_iter()
.flat_map(|node_group| node_group.expand_group_configs()),
);
let mut names = HashSet::new();
for node_config in nodes {
match NodeSpec::from_config(&node_config, &chain_context, true, config.is_evm_based()) {
Ok(mut node) => {
let unique_name = generate_unique_node_name_from_names(node.name, &mut names);
node.name = unique_name;
collators.push(node)
},
Err(err) => errs.push(err),
}
}
let genesis_state = if let Some(path) = config.genesis_state_path() {
ParaArtifact::new(
ParaArtifactType::State,
ParaArtifactBuildOption::Path(path.to_string()),
)
} else {
let cmd = if let Some(cmd) = config.genesis_state_generator() {
cmd.cmd()
} else {
main_cmd
};
ParaArtifact::new(
ParaArtifactType::State,
ParaArtifactBuildOption::Command(cmd.as_str().into()),
)
.image(main_image.clone())
};
let genesis_wasm = if let Some(path) = config.genesis_wasm_path() {
ParaArtifact::new(
ParaArtifactType::Wasm,
ParaArtifactBuildOption::Path(path.to_string()),
)
} else {
let cmd = if let Some(cmd) = config.genesis_wasm_generator() {
cmd.as_str()
} else {
main_cmd.as_str()
};
ParaArtifact::new(
ParaArtifactType::Wasm,
ParaArtifactBuildOption::Command(cmd.into()),
)
.image(main_image.clone())
};
let para_spec = TeyrchainSpec {
id: config.id(),
// ensure unique id is set at this point, if not just set to the para_id
unique_id: if config.unique_id().is_empty() {
config.id().to_string()
} else {
config.unique_id().to_string()
},
default_command: config.default_command().cloned(),
default_image: config.default_image().cloned(),
default_resources: config.default_resources().cloned(),
default_db_snapshot: config.default_db_snapshot().cloned(),
wasm_override: config.wasm_override().cloned(),
default_args: config.default_args().into_iter().cloned().collect(),
chain_spec,
no_default_bootnodes: config.no_default_bootnodes(),
registration_strategy: config
.registration_strategy()
.unwrap_or(&RegistrationStrategy::InGenesis)
.clone(),
onboard_as_parachain: config.onboard_as_parachain(),
is_cumulus_based: config.is_cumulus_based(),
is_evm_based: config.is_evm_based(),
initial_balance: config.initial_balance(),
genesis_state,
genesis_wasm,
genesis_overrides: config.genesis_overrides().cloned(),
collators,
raw_spec_override: config.raw_spec_override().cloned(),
bootnodes_addresses: config.bootnodes_addresses().into_iter().cloned().collect(),
};
Ok(para_spec)
}
pub fn registration_strategy(&self) -> &RegistrationStrategy {
&self.registration_strategy
}
pub fn get_genesis_config(&self) -> Result<ParaGenesisConfig<&PathBuf>, OrchestratorError> {
let genesis_config = ParaGenesisConfig {
state_path: self.genesis_state.artifact_path().ok_or(
OrchestratorError::InvariantError(
"artifact path for state must be set at this point",
),
)?,
wasm_path: self.genesis_wasm.artifact_path().ok_or(
OrchestratorError::InvariantError(
"artifact path for wasm must be set at this point",
),
)?,
id: self.id,
as_parachain: self.onboard_as_parachain,
};
Ok(genesis_config)
}
pub fn id(&self) -> u32 {
self.id
}
pub fn chain_spec(&self) -> Option<&ChainSpec> {
self.chain_spec.as_ref()
}
pub fn chain_spec_mut(&mut self) -> Option<&mut ChainSpec> {
self.chain_spec.as_mut()
}
/// Build parachain chain-spec
///
/// This function customize the chain-spec (if is possible) and build the raw version
/// of the chain-spec.
pub(crate) async fn build_chain_spec<'a, T>(
&mut self,
relay_chain_id: &str,
ns: &DynNamespace,
scoped_fs: &ScopedFilesystem<'a, T>,
) -> Result<Option<PathBuf>, anyhow::Error>
where
T: FileSystem,
{
let cloned = self.clone();
let chain_spec_raw_path = if let Some(chain_spec) = self.chain_spec.as_mut() {
debug!("parachain chain-spec building!");
chain_spec.build(ns, scoped_fs).await?;
debug!("parachain chain-spec built!");
chain_spec
.customize_para(&cloned, relay_chain_id, scoped_fs)
.await?;
debug!("parachain chain-spec customized!");
chain_spec
.build_raw(ns, scoped_fs, Some(relay_chain_id.try_into()?))
.await?;
debug!("parachain chain-spec raw built!");
// override wasm if needed
if let Some(ref wasm_override) = self.wasm_override {
chain_spec.override_code(scoped_fs, wasm_override).await?;
}
// override raw spec if needed
if let Some(ref raw_spec_override) = self.raw_spec_override {
chain_spec
.override_raw_spec(scoped_fs, raw_spec_override)
.await?;
}
let chain_spec_raw_path =
chain_spec
.raw_path()
.ok_or(OrchestratorError::InvariantError(
"chain-spec raw path should be set now",
))?;
Some(chain_spec_raw_path.to_path_buf())
} else {
None
};
Ok(chain_spec_raw_path)
}
/// Get the bootnodes addresses for the parachain spec
pub(crate) fn bootnodes_addresses(&self) -> Vec<&multiaddr::Multiaddr> {
self.bootnodes_addresses.iter().collect()
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/shared.rs
Vendored
+3
View File
@@ -0,0 +1,3 @@
pub mod constants;
pub mod macros;
pub mod types;
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/shared/constants.rs
Vendored
+17
View File
@@ -0,0 +1,17 @@
/// Prometheus exporter default port
pub const PROMETHEUS_PORT: u16 = 9615;
/// Prometheus exporter default port in collator full-node
pub const FULL_NODE_PROMETHEUS_PORT: u16 = 9616;
/// JSON-RPC server (ws)
pub const RPC_PORT: u16 = 9944;
// JSON-RPC server (http, used by old versions)
pub const RPC_HTTP_PORT: u16 = 9933;
// P2P default port
pub const P2P_PORT: u16 = 30333;
// default command template to build chain-spec
pub const DEFAULT_CHAIN_SPEC_TPL_COMMAND: &str =
"{{mainCommand}} build-spec --chain {{chainName}} {{disableBootnodes}}";
// interval to determine how often to run node liveness checks
pub const NODE_MONITORING_INTERVAL_SECONDS: u64 = 15;
// how long to wait before a node is considered unresponsive
pub const NODE_MONITORING_FAILURE_THRESHOLD_SECONDS: u64 = 5;
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/shared/macros.rs
Vendored
+32
View File
@@ -0,0 +1,32 @@
macro_rules! create_add_options {
($struct:ident {$( $field:ident:$type:ty ),*}) =>{
#[derive(Default, Debug, Clone)]
pub struct $struct {
/// Image to run the node
pub image: Option<Image>,
/// Command to run the node
pub command: Option<Command>,
/// Subcommand for the node
pub subcommand: Option<Command>,
/// Arguments to pass to the node
pub args: Vec<Arg>,
/// Env vars to set
pub env: Vec<EnvVar>,
/// Make the node a validator
///
/// This implies `--validator` or `--collator`
pub is_validator: bool,
/// RPC port to use, if None a random one will be set
pub rpc_port: Option<Port>,
/// Prometheus port to use, if None a random one will be set
pub prometheus_port: Option<Port>,
/// P2P port to use, if None a random one will be set
pub p2p_port: Option<Port>,
$(
pub $field: $type,
)*
}
};
}
pub(crate) use create_add_options;
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/shared/types.rs
Vendored
+99
View File
@@ -0,0 +1,99 @@
use std::{
collections::HashMap,
net::TcpListener,
path::PathBuf,
sync::{Arc, RwLock},
};
use configuration::shared::{
resources::Resources,
types::{Arg, AssetLocation, Command, Image, Port},
};
use serde::{Deserialize, Serialize};
pub type Accounts = HashMap<String, NodeAccount>;
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct NodeAccount {
pub address: String,
pub public_key: String,
}
impl NodeAccount {
pub fn new(addr: impl Into<String>, pk: impl Into<String>) -> Self {
Self {
address: addr.into(),
public_key: pk.into(),
}
}
}
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub struct NodeAccounts {
pub seed: String,
pub accounts: Accounts,
}
#[derive(Clone, Default, Debug, Serialize, Deserialize)]
pub struct ParkedPort(
pub(crate) Port,
#[serde(skip)] pub(crate) Arc<RwLock<Option<TcpListener>>>,
);
impl ParkedPort {
pub(crate) fn new(port: u16, listener: TcpListener) -> ParkedPort {
let listener = Arc::new(RwLock::new(Some(listener)));
ParkedPort(port, listener)
}
pub(crate) fn drop_listener(&self) {
// drop the listener will allow the running node to start listenen connections
let mut l = self.1.write().unwrap();
*l = None;
}
}
#[derive(Debug, Clone, Default)]
pub struct ChainDefaultContext<'a> {
pub default_command: Option<&'a Command>,
pub default_image: Option<&'a Image>,
pub default_resources: Option<&'a Resources>,
pub default_db_snapshot: Option<&'a AssetLocation>,
pub default_args: Vec<&'a Arg>,
}
#[derive(Debug, Clone)]
pub struct RegisterParachainOptions {
pub id: u32,
pub wasm_path: PathBuf,
pub state_path: PathBuf,
pub node_ws_url: String,
pub onboard_as_para: bool,
pub seed: Option<[u8; 32]>,
pub finalization: bool,
}
pub struct RuntimeUpgradeOptions {
/// Location of the wasm file (could be either a local file or an url)
pub wasm: AssetLocation,
/// Name of the node to use as rpc endpoint
pub node_name: Option<String>,
/// Seed to use to sign and submit (default to //Alice)
pub seed: Option<[u8; 32]>,
}
impl RuntimeUpgradeOptions {
pub fn new(wasm: AssetLocation) -> Self {
Self {
wasm,
node_name: None,
seed: None,
}
}
}
#[derive(Debug, Clone)]
pub struct ParachainGenesisArgs {
pub genesis_head: String,
pub validation_code: String,
pub parachain: bool,
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/spawner.rs
Vendored
+305
View File
@@ -0,0 +1,305 @@
use std::{collections::HashMap, path::PathBuf};
use anyhow::Context;
use configuration::GlobalSettings;
use provider::{
constants::{LOCALHOST, NODE_CONFIG_DIR, NODE_DATA_DIR, NODE_RELAY_DATA_DIR, P2P_PORT},
shared::helpers::running_in_ci,
types::{SpawnNodeOptions, TransferedFile},
DynNamespace,
};
use support::{
constants::THIS_IS_A_BUG, fs::FileSystem, replacer::apply_running_network_replacements,
};
use tracing::info;
use crate::{
generators,
network::node::NetworkNode,
network_spec::{node::NodeSpec, teyrchain::TeyrchainSpec},
shared::constants::{FULL_NODE_PROMETHEUS_PORT, PROMETHEUS_PORT, RPC_PORT},
ScopedFilesystem, ZombieRole,
};
#[derive(Clone)]
pub struct SpawnNodeCtx<'a, T: FileSystem> {
/// Relaychain id, from the chain-spec (e.g rococo_local_testnet)
pub(crate) chain_id: &'a str,
// Parachain id, from the chain-spec (e.g local_testnet)
pub(crate) parachain_id: Option<&'a str>,
/// Relaychain chain name (e.g rococo-local)
pub(crate) chain: &'a str,
/// Role of the node in the network
pub(crate) role: ZombieRole,
/// Ref to the namespace
pub(crate) ns: &'a DynNamespace,
/// Ref to an scoped filesystem (encapsulate fs actions inside the ns directory)
pub(crate) scoped_fs: &'a ScopedFilesystem<'a, T>,
/// Ref to a parachain (used to spawn collators)
pub(crate) parachain: Option<&'a TeyrchainSpec>,
/// The string representation of the bootnode address to pass to nodes
pub(crate) bootnodes_addr: &'a Vec<String>,
/// Flag to wait node is ready or not
/// Ready state means we can query Prometheus internal server
pub(crate) wait_ready: bool,
/// A json representation of the running nodes with their names as 'key'
pub(crate) nodes_by_name: serde_json::Value,
/// A ref to the global settings
pub(crate) global_settings: &'a GlobalSettings,
}
pub async fn spawn_node<'a, T>(
node: &NodeSpec,
mut files_to_inject: Vec<TransferedFile>,
ctx: &SpawnNodeCtx<'a, T>,
) -> Result<NetworkNode, anyhow::Error>
where
T: FileSystem,
{
let mut created_paths = vec![];
// Create and inject the keystore IFF
// - The node is validator in the relaychain
// - The node is collator (encoded as validator) and the parachain is cumulus_based
// (parachain_id) should be set then.
if node.is_validator && (ctx.parachain.is_none() || ctx.parachain_id.is_some()) {
// Generate keystore for node
let node_files_path = if let Some(para) = ctx.parachain {
para.id.to_string()
} else {
node.name.clone()
};
let asset_hub_polkadot = ctx
.parachain_id
.map(|id| id.starts_with("asset-hub-polkadot"))
.unwrap_or_default();
let keystore_key_types = node.keystore_key_types.iter().map(String::as_str).collect();
let key_filenames = generators::generate_node_keystore(
&node.accounts,
&node_files_path,
ctx.scoped_fs,
asset_hub_polkadot,
keystore_key_types,
)
.await
.unwrap();
// Paths returned are relative to the base dir, we need to convert into
// fullpaths to inject them in the nodes.
let remote_keystore_chain_id = if let Some(id) = ctx.parachain_id {
id
} else {
ctx.chain_id
};
let keystore_path = node.keystore_path.clone().unwrap_or(PathBuf::from(format!(
"/data/chains/{remote_keystore_chain_id}/keystore",
)));
for key_filename in key_filenames {
let f = TransferedFile::new(
PathBuf::from(format!(
"{}/{}/{}",
ctx.ns.base_dir().to_string_lossy(),
node_files_path,
key_filename.to_string_lossy()
)),
keystore_path.join(key_filename),
);
files_to_inject.push(f);
}
created_paths.push(keystore_path);
}
let base_dir = format!("{}/{}", ctx.ns.base_dir().to_string_lossy(), &node.name);
let (cfg_path, data_path, relay_data_path) = if !ctx.ns.capabilities().prefix_with_full_path {
(
NODE_CONFIG_DIR.into(),
NODE_DATA_DIR.into(),
NODE_RELAY_DATA_DIR.into(),
)
} else {
let cfg_path = format!("{}{NODE_CONFIG_DIR}", &base_dir);
let data_path = format!("{}{NODE_DATA_DIR}", &base_dir);
let relay_data_path = format!("{}{NODE_RELAY_DATA_DIR}", &base_dir);
(cfg_path, data_path, relay_data_path)
};
let gen_opts = generators::GenCmdOptions {
relay_chain_name: ctx.chain,
cfg_path: &cfg_path, // TODO: get from provider/ns
data_path: &data_path, // TODO: get from provider
relay_data_path: &relay_data_path, // TODO: get from provider
use_wrapper: false, // TODO: get from provider
bootnode_addr: ctx.bootnodes_addr.clone(),
use_default_ports_in_cmd: ctx.ns.capabilities().use_default_ports_in_cmd,
// IFF the provider require an image (e.g k8s) we know this is not native
is_native: !ctx.ns.capabilities().requires_image,
};
let mut collator_full_node_prom_port: Option<u16> = None;
let mut collator_full_node_prom_port_external: Option<u16> = None;
let (program, args) = match ctx.role {
// Collator should be `non-cumulus` one (e.g adder/undying)
ZombieRole::Node | ZombieRole::Collator => {
let maybe_para_id = ctx.parachain.map(|para| para.id);
generators::generate_node_command(node, gen_opts, maybe_para_id)
},
ZombieRole::CumulusCollator => {
let para = ctx.parachain.expect(&format!(
"parachain must be part of the context {THIS_IS_A_BUG}"
));
collator_full_node_prom_port = node.full_node_prometheus_port.as_ref().map(|p| p.0);
generators::generate_node_command_cumulus(node, gen_opts, para.id)
},
_ => unreachable!(), /* TODO: do we need those?
* ZombieRole::Bootnode => todo!(),
* ZombieRole::Companion => todo!(), */
};
// apply running networ replacements
let args: Vec<String> = args
.iter()
.map(|arg| apply_running_network_replacements(arg, &ctx.nodes_by_name))
.collect();
info!(
"🚀 {}, spawning.... with command: {} {}",
node.name,
program,
args.join(" ")
);
let ports = if ctx.ns.capabilities().use_default_ports_in_cmd {
// should use default ports to as internal
[
(P2P_PORT, node.p2p_port.0),
(RPC_PORT, node.rpc_port.0),
(PROMETHEUS_PORT, node.prometheus_port.0),
]
} else {
[
(P2P_PORT, P2P_PORT),
(RPC_PORT, RPC_PORT),
(PROMETHEUS_PORT, PROMETHEUS_PORT),
]
};
let spawn_ops = SpawnNodeOptions::new(node.name.clone(), program)
.args(args)
.env(
node.env
.iter()
.map(|var| (var.name.clone(), var.value.clone())),
)
.injected_files(files_to_inject)
.created_paths(created_paths)
.db_snapshot(node.db_snapshot.clone())
.port_mapping(HashMap::from(ports))
.node_log_path(node.node_log_path.clone());
let spawn_ops = if let Some(image) = node.image.as_ref() {
spawn_ops.image(image.as_str())
} else {
spawn_ops
};
// Drops the port parking listeners before spawn
node.ws_port.drop_listener();
node.p2p_port.drop_listener();
node.rpc_port.drop_listener();
node.prometheus_port.drop_listener();
if let Some(port) = &node.full_node_p2p_port {
port.drop_listener();
}
if let Some(port) = &node.full_node_prometheus_port {
port.drop_listener();
}
let running_node = ctx.ns.spawn_node(&spawn_ops).await.with_context(|| {
format!(
"Failed to spawn node: {} with opts: {:#?}",
node.name, spawn_ops
)
})?;
let mut ip_to_use = if let Some(local_ip) = ctx.global_settings.local_ip() {
*local_ip
} else {
LOCALHOST
};
let (rpc_port_external, prometheus_port_external, p2p_external);
if running_in_ci() && ctx.ns.provider_name() == "k8s" {
// running kubernets in ci require to use ip and default port
(rpc_port_external, prometheus_port_external, p2p_external) =
(RPC_PORT, PROMETHEUS_PORT, P2P_PORT);
collator_full_node_prom_port_external = Some(FULL_NODE_PROMETHEUS_PORT);
ip_to_use = running_node.ip().await?;
} else {
// Create port-forward iff we are not in CI or provider doesn't use the default ports (native)
let ports = futures::future::try_join_all(vec![
running_node.create_port_forward(node.rpc_port.0, RPC_PORT),
running_node.create_port_forward(node.prometheus_port.0, PROMETHEUS_PORT),
])
.await?;
(rpc_port_external, prometheus_port_external, p2p_external) = (
ports[0].unwrap_or(node.rpc_port.0),
ports[1].unwrap_or(node.prometheus_port.0),
// p2p don't need port-fwd
node.p2p_port.0,
);
if let Some(full_node_prom_port) = collator_full_node_prom_port {
let port_fwd = running_node
.create_port_forward(full_node_prom_port, FULL_NODE_PROMETHEUS_PORT)
.await?;
collator_full_node_prom_port_external = Some(port_fwd.unwrap_or(full_node_prom_port));
}
}
let multiaddr = generators::generate_node_bootnode_addr(
&node.peer_id,
&running_node.ip().await?,
p2p_external,
running_node.args().as_ref(),
&node.p2p_cert_hash,
)?;
let ws_uri = format!("ws://{ip_to_use}:{rpc_port_external}");
let prometheus_uri = format!("http://{ip_to_use}:{prometheus_port_external}/metrics");
info!("🚀 {}, should be running now", node.name);
info!(
"💻 {}: direct link (pjs) https://polkadot.js.org/apps/?rpc={ws_uri}#/explorer",
node.name
);
info!(
"💻 {}: direct link (papi) https://dev.papi.how/explorer#networkId=custom&endpoint={ws_uri}",
node.name
);
info!("📊 {}: metrics link {prometheus_uri}", node.name);
if let Some(full_node_prom_port) = collator_full_node_prom_port_external {
info!(
"📊 {}: collator full-node metrics link http://{}:{}/metrics",
node.name, ip_to_use, full_node_prom_port
);
}
info!("📓 logs cmd: {}", running_node.log_cmd());
Ok(NetworkNode::new(
node.name.clone(),
ws_uri,
prometheus_uri,
multiaddr,
node.clone(),
running_node,
))
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/tx_helper.rs
Vendored
+2
View File
@@ -0,0 +1,2 @@
pub mod client;
pub mod runtime_upgrade;
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/tx_helper/client.rs
Vendored
+43
View File
@@ -0,0 +1,43 @@
use pezkuwi_subxt::{backend::rpc::RpcClient, OnlineClient};
#[async_trait::async_trait]
pub trait ClientFromUrl: Sized {
async fn from_secure_url(url: &str) -> Result<Self, pezkuwi_subxt::Error>;
async fn from_insecure_url(url: &str) -> Result<Self, pezkuwi_subxt::Error>;
}
#[async_trait::async_trait]
impl<Config: pezkuwi_subxt::Config + Send + Sync> ClientFromUrl for OnlineClient<Config> {
async fn from_secure_url(url: &str) -> Result<Self, pezkuwi_subxt::Error> {
Self::from_url(url).await.map_err(Into::into)
}
async fn from_insecure_url(url: &str) -> Result<Self, pezkuwi_subxt::Error> {
Self::from_insecure_url(url).await.map_err(Into::into)
}
}
#[async_trait::async_trait]
impl ClientFromUrl for RpcClient {
async fn from_secure_url(url: &str) -> Result<Self, pezkuwi_subxt::Error> {
Self::from_url(url)
.await
.map_err(pezkuwi_subxt::Error::from)
}
async fn from_insecure_url(url: &str) -> Result<Self, pezkuwi_subxt::Error> {
Self::from_insecure_url(url)
.await
.map_err(pezkuwi_subxt::Error::from)
}
}
pub async fn get_client_from_url<T: ClientFromUrl + Send>(
url: &str,
) -> Result<T, pezkuwi_subxt::Error> {
if pezkuwi_subxt::utils::url_is_secure(url)? {
T::from_secure_url(url).await
} else {
T::from_insecure_url(url).await
}
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/tx_helper/runtime_upgrade.rs
Vendored
+69
View File
@@ -0,0 +1,69 @@
use pezkuwi_subxt::{dynamic::Value, tx::TxStatus, BizinikiwConfig, OnlineClient};
use pezkuwi_subxt_signer::sr25519::Keypair;
use tracing::{debug, info};
use crate::network::node::NetworkNode;
pub async fn upgrade(
node: &NetworkNode,
wasm_data: &[u8],
sudo: &Keypair,
) -> Result<(), anyhow::Error> {
debug!(
"Upgrading runtime, using node: {} with endpoting {}",
node.name, node.ws_uri
);
let api: OnlineClient<BizinikiwConfig> = node.wait_client().await?;
let upgrade = pezkuwi_subxt::dynamic::tx(
"System",
"set_code_without_checks",
vec![Value::from_bytes(wasm_data)],
);
let sudo_call = pezkuwi_subxt::dynamic::tx(
"Sudo",
"sudo_unchecked_weight",
vec![
upgrade.into_value(),
Value::named_composite([
("ref_time", Value::primitive(1.into())),
("proof_size", Value::primitive(1.into())),
]),
],
);
let mut tx = api
.tx()
.sign_and_submit_then_watch_default(&sudo_call, sudo)
.await?;
// Below we use the low level API to replicate the `wait_for_in_block` behaviour
// which was removed in subxt 0.33.0. See https://github.com/paritytech/subxt/pull/1237.
while let Some(status) = tx.next().await {
let status = status?;
match &status {
TxStatus::InBestBlock(tx_in_block) | TxStatus::InFinalizedBlock(tx_in_block) => {
let _result = tx_in_block.wait_for_success().await?;
let block_status = if status.as_finalized().is_some() {
"Finalized"
} else {
"Best"
};
info!(
"[{}] In block: {:#?}",
block_status,
tx_in_block.block_hash()
);
},
TxStatus::Error { message }
| TxStatus::Invalid { message }
| TxStatus::Dropped { message } => {
return Err(anyhow::format_err!("Error submitting tx: {message}"));
},
_ => continue,
}
}
Ok(())
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/src/utils.rs
Vendored
+8
View File
@@ -0,0 +1,8 @@
use serde::Deserializer;
pub fn default_as_empty_vec<'de, D, T>(_deserializer: D) -> Result<Vec<T>, D::Error>
where
D: Deserializer<'de>,
{
Ok(Vec::new())
}
vendor/pezkuwi-zombienet-sdk/crates/orchestrator/testing/rococo-local-plain.json
Vendored
+211
View File
@@ -0,0 +1,211 @@
{
"name": "Rococo Local Testnet",
"id": "rococo_local_testnet",
"chainType": "Local",
"bootNodes": [
"/ip4/127.0.0.1/tcp/30333/p2p/12D3KooWJcDp2Cdok4uSHz5zpjWzfduNCzis9GsMfpej1jwdaYij"
],
"telemetryEndpoints": null,
"protocolId": "dot",
"properties": null,
"forkBlocks": null,
"badBlocks": null,
"lightSyncState": null,
"codeSubstitutes": {},
"genesis": {
"runtime": {
"system": {
"code": "0x52"
},
"babe": {
"authorities": [],
"epochConfig": {
"c": [
1,
4
],
"allowed_slots": "PrimaryAndSecondaryVRFSlots"
}
},
"indices": {
"indices": []
},
"balances": {
"balances": [
[
"5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
1000000000000000000
],
[
"5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty",
1000000000000000000
],
[
"5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y",
1000000000000000000
],
[
"5DAAnrj7VHTznn2AWBemMuyBwZWs6FNFjdyVXUeYum3PTXFy",
1000000000000000000
],
[
"5HGjWAeFDfFCWPsjFQdVV2Msvz2XtMktvgocEZcCj68kUMaw",
1000000000000000000
],
[
"5CiPPseXPECbkjWCa6MnjNokrgYjMqmKndv2rSnekmSK2DjL",
1000000000000000000
],
[
"5GNJqTPyNqANBkUVMN1LPPrxXnFouWXoe2wNSmmEoLctxiZY",
1000000000000000000
],
[
"5HpG9w8EBLe5XCrbczpwq5TSXvedjrBGCwqxK1iQ7qUsSWFc",
1000000000000000000
],
[
"5Ck5SLSHYac6WFt5UZRSsdJjwmpSZq85fd5TRNAdZQVzEAPT",
1000000000000000000
],
[
"5HKPmK9GYtE1PSLsS1qiYU9xQ9Si1NcEhdeCq9sw5bqu4ns8",
1000000000000000000
],
[
"5FCfAonRZgTFrTd9HREEyeJjDpT397KMzizE6T3DvebLFE7n",
1000000000000000000
],
[
"5CRmqmsiNFExV6VbdmPJViVxrWmkaXXvBrSX8oqBT8R9vmWk",
1000000000000000000
]
]
},
"beefy": {
"authorities": [],
"genesisBlock": 1
},
"session": {
"keys": [
[
"5GNJqTPyNqANBkUVMN1LPPrxXnFouWXoe2wNSmmEoLctxiZY",
"5GNJqTPyNqANBkUVMN1LPPrxXnFouWXoe2wNSmmEoLctxiZY",
{
"grandpa": "5FA9nQDVg267DEd8m1ZypXLBnvN7SFxYwV7ndqSYGiN9TTpu",
"babe": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
"im_online": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
"para_validator": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
"para_assignment": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
"authority_discovery": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
"beefy": "KW39r9CJjAVzmkf9zQ4YDb2hqfAVGdRqn53eRqyruqpxAP5YL"
}
],
[
"5HpG9w8EBLe5XCrbczpwq5TSXvedjrBGCwqxK1iQ7qUsSWFc",
"5HpG9w8EBLe5XCrbczpwq5TSXvedjrBGCwqxK1iQ7qUsSWFc",
{
"grandpa": "5GoNkf6WdbxCFnPdAnYYQyCjAKPJgLNxXwPjwTh6DGg6gN3E",
"babe": "5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty",
"im_online": "5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty",
"para_validator": "5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty",
"para_assignment": "5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty",
"authority_discovery": "5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty",
"beefy": "KWByAN7WfZABWS5AoWqxriRmF5f2jnDqy3rB5pfHLGkY93ibN"
}
]
]
},
"grandpa": {
"authorities": []
},
"imOnline": {
"keys": []
},
"authorityDiscovery": {
"keys": []
},
"treasury": {},
"claims": {
"claims": [],
"vesting": []
},
"vesting": {
"vesting": []
},
"nisCounterpartBalances": {
"balances": []
},
"configuration": {
"config": {
"max_code_size": 3145728,
"max_head_data_size": 32768,
"max_upward_queue_count": 8,
"max_upward_queue_size": 1048576,
"max_upward_message_size": 51200,
"max_upward_message_num_per_candidate": 5,
"hrmp_max_message_num_per_candidate": 5,
"validation_upgrade_cooldown": 2,
"validation_upgrade_delay": 2,
"async_backing_params": {
"max_candidate_depth": 0,
"allowed_ancestry_len": 0
},
"max_pov_size": 5242880,
"max_downward_message_size": 1048576,
"hrmp_max_parachain_outbound_channels": 4,
"hrmp_sender_deposit": 0,
"hrmp_recipient_deposit": 0,
"hrmp_channel_max_capacity": 8,
"hrmp_channel_max_total_size": 8192,
"hrmp_max_parachain_inbound_channels": 4,
"hrmp_channel_max_message_size": 1048576,
"executor_params": [],
"code_retention_period": 1200,
"on_demand_cores": 0,
"on_demand_retries": 0,
"on_demand_queue_max_size": 10000,
"on_demand_target_queue_utilization": 250000000,
"on_demand_fee_variability": 30000000,
"on_demand_base_fee": 10000000,
"on_demand_ttl": 5,
"group_rotation_frequency": 20,
"paras_availability_period": 4,
"scheduling_lookahead": 1,
"max_validators_per_core": 1,
"max_validators": null,
"dispute_period": 6,
"dispute_post_conclusion_acceptance_period": 100,
"no_show_slots": 2,
"n_delay_tranches": 25,
"zeroth_delay_tranche_width": 0,
"needed_approvals": 2,
"relay_vrf_modulo_samples": 2,
"pvf_voting_ttl": 2,
"minimum_validation_upgrade_delay": 5,
"minimum_backing_votes": 2
}
},
"paras": {
"paras": []
},
"hrmp": {
"preopenHrmpChannels": []
},
"registrar": {
"nextFreeParaId": 2000
},
"xcmPallet": {
"safeXcmVersion": 3
},
"assignedSlots": {
"maxTemporarySlots": 0,
"maxPermanentSlots": 0,
"config": null
},
"sudo": {
"key": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY"
}
}
}
}
vendor/pezkuwi-zombienet-sdk/crates/prom-metrics-parser/Cargo.toml
Vendored
+18
View File
@@ -0,0 +1,18 @@
[package]
name = "zombienet-prom-metrics-parser"
version.workspace = true
authors.workspace = true
edition.workspace = true
rust-version.workspace = true
publish = true
license.workspace = true
repository.workspace = true
description = "Prometheus metric parser, parse metrics provided by internal prometheus server"
keywords = ["zombienet", "prometheus"]
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
pest = { workspace = true }
pest_derive = { workspace = true }
thiserror = { workspace = true }
vendor/pezkuwi-zombienet-sdk/crates/prom-metrics-parser/src/grammar.pest
Vendored
+47
View File
@@ -0,0 +1,47 @@
// Grammar taken from https://github.com/mitghi/promerge/ with
// some small modifications.
alpha = _{'a'..'z' | 'A'..'Z'}
alphanum = _{'a'..'z' | 'A'..'Z' | '0'..'9'}
number = @{
"-"?
~ ("0" | ASCII_NONZERO_DIGIT ~ ASCII_DIGIT*)
~ ("." ~ ASCII_DIGIT*)?
~ (^"e" ~ ("+" | "-")? ~ ASCII_DIGIT+)?
}
string = ${"\"" ~ inner ~ "\""}
inner = @{char*}
char = {
!("\"" | "\\") ~ ANY
| "\\" ~ ("\"" | "\\" | "/" | "b" | "f" | "n" | "r" | "t")
| "\\" ~ ("u" ~ ASCII_HEX_DIGIT{4})
}
whitespace_or_newline = _{(" "| "\n")*}
hash = _{"#"}
posInf = {"+Inf"}
negInf = {"-Inf"}
NaN = {"NaN"}
lbrace = _{"{"}
rbrace = _{"}"}
typelit = _{"TYPE"}
helplit = _{"HELP"}
comma = _{","}
countertype = {"counter"}
gaugetype = {"gauge"}
histogramtype = {"histogram"}
summarytype = {"summary"}
untyped = {"untyped"}
ident = {alphanum+}
key = @{ident ~ ("_" ~ ident)*}
label = {key ~ "=" ~ string}
labels = {label ~ (comma ~ label)*}
helpkey = {key}
helpval = {inner}
typekey = {key}
typeval = {countertype | gaugetype | histogramtype | summarytype | untyped}
commentval = @{((ASCII_DIGIT| ASCII_NONZERO_DIGIT | ASCII_BIN_DIGIT | ASCII_OCT_DIGIT | ASCII_HEX_DIGIT | ASCII_ALPHA_LOWER | ASCII_ALPHA_UPPER | ASCII_ALPHA | ASCII_ALPHANUMERIC | !"\n" ~ ANY ))*}
helpexpr = {hash ~ whitespace_or_newline ~ helplit ~ whitespace_or_newline ~ helpkey ~ whitespace_or_newline ~ commentval}
typexpr = {hash ~ whitespace_or_newline ~ typelit ~ whitespace_or_newline ~ typekey ~ whitespace_or_newline ~ typeval }
genericomment = {hash ~ whitespace_or_newline ~ commentval}
promstmt = {key ~ (lbrace ~ (labels)* ~ rbrace){0,1} ~ whitespace_or_newline ~ ((posInf | negInf | NaN | number) ~ whitespace_or_newline ){1,2}}
block = {((helpexpr | typexpr | genericomment)~ NEWLINE?)+ ~ (promstmt ~ NEWLINE?)+}
statement = {SOI ~ block+ ~ EOI}
vendor/pezkuwi-zombienet-sdk/crates/prom-metrics-parser/src/lib.rs
Vendored
+178
View File
@@ -0,0 +1,178 @@
use std::{collections::HashMap, num::ParseFloatError};
use pest::Parser;
use pest_derive::Parser;
/// An error at parsing level.
#[derive(thiserror::Error, Debug)]
pub enum ParserError {
#[error("error parsing input")]
ParseError(Box<pest::error::Error<Rule>>),
#[error("root node should be valid: {0}")]
ParseRootNodeError(String),
#[error("can't cast metric value as f64: {0}")]
CastValueError(#[from] ParseFloatError),
}
// This include forces recompiling this source file if the grammar file changes.
// Uncomment it when doing changes to the .pest file
const _GRAMMAR: &str = include_str!("grammar.pest");
#[derive(Parser)]
#[grammar = "grammar.pest"]
pub struct MetricsParser;
pub type MetricMap = HashMap<String, f64>;
pub fn parse(input: &str) -> Result<MetricMap, ParserError> {
let mut metric_map: MetricMap = Default::default();
let mut pairs = MetricsParser::parse(Rule::statement, input)
.map_err(|e| ParserError::ParseError(Box::new(e)))?;
let root = pairs
.next()
.ok_or(ParserError::ParseRootNodeError(pairs.as_str().to_string()))?;
for token in root.into_inner() {
if token.as_rule() == Rule::block {
let inner = token.into_inner();
for value in inner {
match value.as_rule() {
Rule::genericomment | Rule::typexpr | Rule::helpexpr => {
// don't need to collect comments/types/helpers blocks.
continue;
},
Rule::promstmt => {
let mut key: &str = "";
let mut labels: Vec<(&str, &str)> = Vec::new();
let mut val: f64 = 0_f64;
for v in value.clone().into_inner() {
match &v.as_rule() {
Rule::key => {
key = v.as_span().as_str();
},
Rule::NaN | Rule::posInf | Rule::negInf => {
// noop (not used in substrate metrics)
},
Rule::number => {
val = v.as_span().as_str().parse::<f64>()?;
},
Rule::labels => {
// SAFETY: use unwrap should be safe since we are just
// walking the parser struct and if are matching a label
// should have a key/vals
for p in v.into_inner() {
let mut inner = p.into_inner();
let key = inner.next().unwrap().as_span().as_str();
let value = inner
.next()
.unwrap()
.into_inner()
.next()
.unwrap()
.as_span()
.as_str();
labels.push((key, value));
}
},
_ => {
todo!("not implemented");
},
}
}
// we should store to make it compatible with zombienet v1:
// key_without_prefix
// key_without_prefix_and_without_chain
// key_with_prefix_with_chain
// key_with_prefix_and_without_chain
let key_with_out_prefix =
key.split('_').collect::<Vec<&str>>()[1..].join("_");
let (labels_without_chain, labels_with_chain) =
labels.iter().fold((vec![], vec![]), |mut acc, item| {
if item.0.eq("chain") {
acc.1.push(format!("{}=\"{}\"", item.0, item.1));
} else {
acc.0.push(format!("{}=\"{}\"", item.0, item.1));
acc.1.push(format!("{}=\"{}\"", item.0, item.1));
}
acc
});
let labels_with_chain_str = if labels_with_chain.is_empty() {
String::from("")
} else {
format!("{{{}}}", labels_with_chain.join(","))
};
let labels_without_chain_str = if labels_without_chain.is_empty() {
String::from("")
} else {
format!("{{{}}}", labels_without_chain.join(","))
};
metric_map.insert(format!("{key}{labels_without_chain_str}"), val);
metric_map.insert(
format!("{key_with_out_prefix}{labels_without_chain_str}"),
val,
);
metric_map.insert(format!("{key}{labels_with_chain_str}"), val);
metric_map
.insert(format!("{key_with_out_prefix}{labels_with_chain_str}"), val);
},
_ => {},
}
}
}
}
Ok(metric_map)
}
#[cfg(test)]
mod tests {
use std::fs;
use super::*;
#[test]
fn parse_metrics_works() {
let metrics_raw = fs::read_to_string("./testing/metrics.txt").unwrap();
let metrics = parse(&metrics_raw).unwrap();
// full key
assert_eq!(
metrics
.get("polkadot_node_is_active_validator{chain=\"rococo_local_testnet\"}")
.unwrap(),
&1_f64
);
// with prefix and no chain
assert_eq!(
metrics.get("polkadot_node_is_active_validator").unwrap(),
&1_f64
);
// no prefix with chain
assert_eq!(
metrics
.get("node_is_active_validator{chain=\"rococo_local_testnet\"}")
.unwrap(),
&1_f64
);
// no prefix without chain
assert_eq!(metrics.get("node_is_active_validator").unwrap(), &1_f64);
}
#[test]
fn parse_invalid_metrics_str_should_fail() {
let metrics_raw = r"
# HELP polkadot_node_is_active_validator Tracks if the validator is in the active set. Updates at session boundary.
# TYPE polkadot_node_is_active_validator gauge
polkadot_node_is_active_validator{chain=} 1
";
let metrics = parse(metrics_raw);
assert!(metrics.is_err());
assert!(matches!(metrics, Err(ParserError::ParseError(_))));
}
}
vendor/pezkuwi-zombienet-sdk/crates/prom-metrics-parser/testing/metrics.txt
Vendored
+3879
View File
File diff suppressed because it is too large Load Diff
vendor/pezkuwi-zombienet-sdk/crates/provider/.gitignore
+2
View File
@@ -0,0 +1,2 @@
/target
/Cargo.lock
vendor/pezkuwi-zombienet-sdk/crates/provider/Cargo.toml
Vendored
+47
View File
@@ -0,0 +1,47 @@
[package]
name = "zombienet-provider"
version.workspace = true
authors.workspace = true
edition.workspace = true
rust-version.workspace = true
publish = true
license.workspace = true
repository.workspace = true
description = "Zombienet provider, implement the logic to run the nodes in the native provider"
keywords = ["zombienet", "provider", "native"]
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
async-trait = { workspace = true }
futures = { workspace = true }
serde = { workspace = true, features = ["derive", "rc"] }
serde_json = { workspace = true }
serde_yaml = { workspace = true }
tokio = { workspace = true, features = [
"process",
"macros",
"fs",
"time",
"rt",
] }
tokio-util = { workspace = true, features = ["compat"] }
thiserror = { workspace = true }
anyhow = { workspace = true }
uuid = { workspace = true, features = ["v4"] }
nix = { workspace = true, features = ["signal"] }
kube = { workspace = true, features = ["ws", "runtime"] }
k8s-openapi = { workspace = true, features = ["v1_27"] }
tar = { workspace = true }
sha2 = { workspace = true }
hex = { workspace = true }
tracing = { workspace = true }
reqwest = { workspace = true }
regex = { workspace = true }
url = { workspace = true }
flate2 = { workspace = true }
erased-serde = { workspace = true }
# Zomebienet deps
support = { workspace = true }
configuration = { workspace = true }
vendor/pezkuwi-zombienet-sdk/crates/provider/src/docker.rs
Vendored
+6
View File
@@ -0,0 +1,6 @@
mod client;
mod namespace;
mod node;
mod provider;
pub use provider::DockerProvider;
vendor/pezkuwi-zombienet-sdk/crates/provider/src/docker/client.rs
Vendored
+596
View File
@@ -0,0 +1,596 @@
use std::{collections::HashMap, path::Path, process::Stdio};
use anyhow::anyhow;
use futures::future::try_join_all;
use serde::{Deserialize, Deserializer};
use tokio::process::Command;
use tracing::{info, trace};
use crate::types::{ExecutionResult, Port};
#[derive(thiserror::Error, Debug)]
#[error(transparent)]
pub struct Error(#[from] anyhow::Error);
pub type Result<T> = core::result::Result<T, Error>;
#[derive(Clone)]
pub struct DockerClient {
using_podman: bool,
}
#[derive(Debug)]
pub struct ContainerRunOptions {
image: String,
command: Vec<String>,
env: Option<Vec<(String, String)>>,
volume_mounts: Option<HashMap<String, String>>,
name: Option<String>,
entrypoint: Option<String>,
port_mapping: HashMap<Port, Port>,
rm: bool,
detach: bool,
}
enum Container {
Docker(DockerContainer),
Podman(PodmanContainer),
}
// TODO: we may don't need this
#[allow(dead_code)]
#[derive(Deserialize, Debug)]
struct DockerContainer {
#[serde(alias = "Names", deserialize_with = "deserialize_list")]
names: Vec<String>,
#[serde(alias = "Ports", deserialize_with = "deserialize_list")]
ports: Vec<String>,
#[serde(alias = "State")]
state: String,
}
// TODO: we may don't need this
#[allow(dead_code)]
#[derive(Deserialize, Debug)]
struct PodmanPort {
host_ip: String,
container_port: u16,
host_port: u16,
range: u16,
protocol: String,
}
// TODO: we may don't need this
#[allow(dead_code)]
#[derive(Deserialize, Debug)]
struct PodmanContainer {
#[serde(alias = "Id")]
id: String,
#[serde(alias = "Image")]
image: String,
#[serde(alias = "Mounts")]
mounts: Vec<String>,
#[serde(alias = "Names")]
names: Vec<String>,
#[serde(alias = "Ports", deserialize_with = "deserialize_null_as_default")]
ports: Vec<PodmanPort>,
#[serde(alias = "State")]
state: String,
}
fn deserialize_list<'de, D>(deserializer: D) -> std::result::Result<Vec<String>, D::Error>
where
D: Deserializer<'de>,
{
let str_sequence = String::deserialize(deserializer)?;
Ok(str_sequence
.split(',')
.filter(|item| !item.is_empty())
.map(|item| item.to_owned())
.collect())
}
fn deserialize_null_as_default<'de, D, T>(deserializer: D) -> std::result::Result<T, D::Error>
where
T: Default + Deserialize<'de>,
D: Deserializer<'de>,
{
let opt = Option::deserialize(deserializer)?;
Ok(opt.unwrap_or_default())
}
impl ContainerRunOptions {
pub fn new<S>(image: &str, command: Vec<S>) -> Self
where
S: Into<String> + std::fmt::Debug + Send + Clone,
{
ContainerRunOptions {
image: image.to_string(),
command: command
.clone()
.into_iter()
.map(|s| s.into())
.collect::<Vec<_>>(),
env: None,
volume_mounts: None,
name: None,
entrypoint: None,
port_mapping: HashMap::default(),
rm: false,
detach: true, // add -d flag by default
}
}
pub fn env<S>(mut self, env: Vec<(S, S)>) -> Self
where
S: Into<String> + std::fmt::Debug + Send + Clone,
{
self.env = Some(
env.into_iter()
.map(|(name, value)| (name.into(), value.into()))
.collect(),
);
self
}
pub fn volume_mounts<S>(mut self, volume_mounts: HashMap<S, S>) -> Self
where
S: Into<String> + std::fmt::Debug + Send + Clone,
{
self.volume_mounts = Some(
volume_mounts
.into_iter()
.map(|(source, target)| (source.into(), target.into()))
.collect(),
);
self
}
pub fn name<S>(mut self, name: S) -> Self
where
S: Into<String> + std::fmt::Debug + Send + Clone,
{
self.name = Some(name.into());
self
}
pub fn entrypoint<S>(mut self, entrypoint: S) -> Self
where
S: Into<String> + std::fmt::Debug + Send + Clone,
{
self.entrypoint = Some(entrypoint.into());
self
}
pub fn port_mapping(mut self, port_mapping: &HashMap<Port, Port>) -> Self {
self.port_mapping.clone_from(port_mapping);
self
}
pub fn rm(mut self) -> Self {
self.rm = true;
self
}
pub fn detach(mut self, choice: bool) -> Self {
self.detach = choice;
self
}
}
impl DockerClient {
pub async fn new() -> Result<Self> {
let using_podman = Self::is_using_podman().await?;
Ok(DockerClient { using_podman })
}
pub fn client_binary(&self) -> String {
String::from(if self.using_podman {
"podman"
} else {
"docker"
})
}
async fn is_using_podman() -> Result<bool> {
if let Ok(output) = tokio::process::Command::new("docker")
.arg("version")
.output()
.await
{
// detect whether we're actually running podman with docker emulation
return Ok(String::from_utf8_lossy(&output.stdout)
.to_lowercase()
.contains("podman"));
}
tokio::process::Command::new("podman")
.arg("--version")
.output()
.await
.map_err(|err| anyhow!("Failed to detect container engine: {err}"))?;
Ok(true)
}
}
impl DockerClient {
fn client_command(&self) -> tokio::process::Command {
tokio::process::Command::new(self.client_binary())
}
pub async fn create_volume(&self, name: &str) -> Result<()> {
let result = self
.client_command()
.args(["volume", "create", name])
.output()
.await
.map_err(|err| anyhow!("Failed to create volume '{name}': {err}"))?;
if !result.status.success() {
return Err(anyhow!(
"Failed to create volume '{name}': {}",
String::from_utf8_lossy(&result.stderr)
)
.into());
}
Ok(())
}
pub async fn container_run(&self, options: ContainerRunOptions) -> Result<String> {
let mut cmd = self.client_command();
cmd.args(["run", "--platform", "linux/amd64"]);
if options.detach {
cmd.arg("-d");
}
Self::apply_cmd_options(&mut cmd, &options);
trace!("cmd: {:?}", cmd);
let result = cmd.output().await.map_err(|err| {
anyhow!(
"Failed to run container with image '{image}' and command '{command}': {err}",
image = options.image,
command = options.command.join(" "),
)
})?;
if !result.status.success() {
return Err(anyhow!(
"Failed to run container with image '{image}' and command '{command}': {err}",
image = options.image,
command = options.command.join(" "),
err = String::from_utf8_lossy(&result.stderr)
)
.into());
}
Ok(String::from_utf8_lossy(&result.stdout).to_string())
}
pub async fn container_create(&self, options: ContainerRunOptions) -> Result<String> {
let mut cmd = self.client_command();
cmd.args(["container", "create"]);
Self::apply_cmd_options(&mut cmd, &options);
trace!("cmd: {:?}", cmd);
let result = cmd.output().await.map_err(|err| {
anyhow!(
"Failed to run container with image '{image}' and command '{command}': {err}",
image = options.image,
command = options.command.join(" "),
)
})?;
if !result.status.success() {
return Err(anyhow!(
"Failed to run container with image '{image}' and command '{command}': {err}",
image = options.image,
command = options.command.join(" "),
err = String::from_utf8_lossy(&result.stderr)
)
.into());
}
Ok(String::from_utf8_lossy(&result.stdout).to_string())
}
pub async fn container_exec<S>(
&self,
name: &str,
command: Vec<S>,
env: Option<Vec<(S, S)>>,
as_user: Option<S>,
) -> Result<ExecutionResult>
where
S: Into<String> + std::fmt::Debug + Send + Clone,
{
let mut cmd = self.client_command();
cmd.arg("exec");
if let Some(env) = env {
for env_var in env {
cmd.args(["-e", &format!("{}={}", env_var.0.into(), env_var.1.into())]);
}
}
if let Some(user) = as_user {
cmd.args(["-u", user.into().as_ref()]);
}
cmd.arg(name);
cmd.args(
command
.clone()
.into_iter()
.map(|s| <S as Into<String>>::into(s)),
);
trace!("cmd is : {:?}", cmd);
let result = cmd.output().await.map_err(|err| {
anyhow!(
"Failed to exec '{}' on '{}': {err}",
command
.into_iter()
.map(|s| <S as Into<String>>::into(s))
.collect::<Vec<_>>()
.join(" "),
name,
)
})?;
if !result.status.success() {
return Ok(Err((
result.status,
String::from_utf8_lossy(&result.stderr).to_string(),
)));
}
Ok(Ok(String::from_utf8_lossy(&result.stdout).to_string()))
}
pub async fn container_cp(
&self,
name: &str,
local_path: &Path,
remote_path: &Path,
) -> Result<()> {
let result = self
.client_command()
.args([
"cp",
local_path.to_string_lossy().as_ref(),
&format!("{name}:{}", remote_path.to_string_lossy().as_ref()),
])
.output()
.await
.map_err(|err| {
anyhow!(
"Failed copy file '{file}' to container '{name}': {err}",
file = local_path.to_string_lossy(),
)
})?;
if !result.status.success() {
return Err(anyhow!(
"Failed to copy file '{file}' to container '{name}': {err}",
file = local_path.to_string_lossy(),
err = String::from_utf8_lossy(&result.stderr)
)
.into());
}
Ok(())
}
pub async fn container_rm(&self, name: &str) -> Result<()> {
let result = self
.client_command()
.args(["rm", "--force", "--volumes", name])
.output()
.await
.map_err(|err| anyhow!("Failed do remove container '{name}: {err}"))?;
if !result.status.success() {
return Err(anyhow!(
"Failed to remove container '{name}': {err}",
err = String::from_utf8_lossy(&result.stderr)
)
.into());
}
Ok(())
}
pub async fn namespaced_containers_rm(&self, namespace: &str) -> Result<()> {
let container_names: Vec<String> = self
.get_containers()
.await?
.into_iter()
.filter_map(|container| match container {
Container::Docker(container) => {
if let Some(name) = container.names.first() {
if name.starts_with(namespace) {
return Some(name.to_string());
}
}
None
},
Container::Podman(container) => {
if let Some(name) = container.names.first() {
if name.starts_with(namespace) {
return Some(name.to_string());
}
}
None
},
})
.collect();
info!("{:?}", container_names);
let futures = container_names
.iter()
.map(|name| self.container_rm(name))
.collect::<Vec<_>>();
try_join_all(futures).await?;
Ok(())
}
pub async fn container_ip(&self, container_name: &str) -> Result<String> {
let ip = if self.using_podman {
"127.0.0.1".into()
} else {
let mut cmd = tokio::process::Command::new("docker");
cmd.args(vec![
"inspect",
"-f",
"{{ .NetworkSettings.IPAddress }}",
container_name,
]);
trace!("CMD: {cmd:?}");
let res = cmd
.output()
.await
.map_err(|err| anyhow!("Failed to get docker container ip, output: {err}"))?;
String::from_utf8(res.stdout)
.map_err(|err| anyhow!("Failed to get docker container ip, output: {err}"))?
.trim()
.into()
};
trace!("IP: {ip}");
Ok(ip)
}
async fn get_containers(&self) -> Result<Vec<Container>> {
let containers = if self.using_podman {
self.get_podman_containers()
.await?
.into_iter()
.map(Container::Podman)
.collect()
} else {
self.get_docker_containers()
.await?
.into_iter()
.map(Container::Docker)
.collect()
};
Ok(containers)
}
async fn get_podman_containers(&self) -> Result<Vec<PodmanContainer>> {
let res = tokio::process::Command::new("podman")
.args(vec!["ps", "--all", "--no-trunc", "--format", "json"])
.output()
.await
.map_err(|err| anyhow!("Failed to get podman containers output: {err}"))?;
let stdout = String::from_utf8_lossy(&res.stdout);
let containers = serde_json::from_str(&stdout)
.map_err(|err| anyhow!("Failed to parse podman containers output: {err}"))?;
Ok(containers)
}
async fn get_docker_containers(&self) -> Result<Vec<DockerContainer>> {
let res = tokio::process::Command::new("docker")
.args(vec!["ps", "--all", "--no-trunc", "--format", "json"])
.output()
.await
.unwrap();
let stdout = String::from_utf8_lossy(&res.stdout);
let mut containers = vec![];
for line in stdout.lines() {
containers.push(
serde_json::from_str::<DockerContainer>(line)
.map_err(|err| anyhow!("Failed to parse docker container output: {err}"))?,
);
}
Ok(containers)
}
pub(crate) async fn container_logs(&self, container_name: &str) -> Result<String> {
let output = Command::new("sh")
.arg("-c")
.arg(format!("docker logs -t '{container_name}' 2>&1"))
.stdout(Stdio::piped())
.output()
.await
.map_err(|err| {
anyhow!(
"Failed to spawn docker logs command for container '{container_name}': {err}"
)
})?;
let logs = String::from_utf8_lossy(&output.stdout).to_string();
if !output.status.success() {
// stderr was redirected to stdout, so logs should contain the error message if any
return Err(anyhow!(
"Failed to get logs for container '{name}': {logs}",
name = container_name,
logs = &logs
)
.into());
}
Ok(logs)
}
fn apply_cmd_options(cmd: &mut Command, options: &ContainerRunOptions) {
if options.rm {
cmd.arg("--rm");
}
if let Some(entrypoint) = options.entrypoint.as_ref() {
cmd.args(["--entrypoint", entrypoint]);
}
if let Some(volume_mounts) = options.volume_mounts.as_ref() {
for (source, target) in volume_mounts {
cmd.args(["-v", &format!("{source}:{target}")]);
}
}
if let Some(env) = options.env.as_ref() {
for env_var in env {
cmd.args(["-e", &format!("{}={}", env_var.0, env_var.1)]);
}
}
// add published ports
for (container_port, host_port) in options.port_mapping.iter() {
cmd.args(["-p", &format!("{host_port}:{container_port}")]);
}
if let Some(name) = options.name.as_ref() {
cmd.args(["--name", name]);
}
cmd.arg(&options.image);
for arg in &options.command {
cmd.arg(arg);
}
}
}
vendor/pezkuwi-zombienet-sdk/crates/provider/src/docker/namespace.rs
Vendored
+494
View File
@@ -0,0 +1,494 @@
use std::{
collections::HashMap,
path::{Path, PathBuf},
sync::{Arc, Weak},
thread,
};
use async_trait::async_trait;
use support::{constants::THIS_IS_A_BUG, fs::FileSystem};
use tokio::sync::{Mutex, RwLock};
use tracing::{debug, trace, warn};
use uuid::Uuid;
use super::{
client::{ContainerRunOptions, DockerClient},
node::DockerNode,
DockerProvider,
};
use crate::{
constants::NAMESPACE_PREFIX,
docker::{
node::{DeserializableDockerNodeOptions, DockerNodeOptions},
provider,
},
shared::helpers::extract_execution_result,
types::{
GenerateFileCommand, GenerateFilesOptions, ProviderCapabilities, RunCommandOptions,
SpawnNodeOptions,
},
DynNode, ProviderError, ProviderNamespace, ProviderNode,
};
pub struct DockerNamespace<FS>
where
FS: FileSystem + Send + Sync + Clone,
{
weak: Weak<DockerNamespace<FS>>,
#[allow(dead_code)]
provider: Weak<DockerProvider<FS>>,
name: String,
base_dir: PathBuf,
capabilities: ProviderCapabilities,
docker_client: DockerClient,
filesystem: FS,
delete_on_drop: Arc<Mutex<bool>>,
pub(super) nodes: RwLock<HashMap<String, Arc<DockerNode<FS>>>>,
}
impl<FS> DockerNamespace<FS>
where
FS: FileSystem + Send + Sync + Clone + 'static,
{
pub(super) async fn new(
provider: &Weak<DockerProvider<FS>>,
tmp_dir: &PathBuf,
capabilities: &ProviderCapabilities,
docker_client: &DockerClient,
filesystem: &FS,
custom_base_dir: Option<&Path>,
) -> Result<Arc<Self>, ProviderError> {
let name = format!("{}{}", NAMESPACE_PREFIX, Uuid::new_v4());
let base_dir = if let Some(custom_base_dir) = custom_base_dir {
if !filesystem.exists(custom_base_dir).await {
filesystem.create_dir(custom_base_dir).await?;
} else {
warn!(
"⚠️ Using and existing directory {} as base dir",
custom_base_dir.to_string_lossy()
);
}
PathBuf::from(custom_base_dir)
} else {
let base_dir = PathBuf::from_iter([tmp_dir, &PathBuf::from(&name)]);
filesystem.create_dir(&base_dir).await?;
base_dir
};
let namespace = Arc::new_cyclic(|weak| DockerNamespace {
weak: weak.clone(),
provider: provider.clone(),
name,
base_dir,
capabilities: capabilities.clone(),
filesystem: filesystem.clone(),
docker_client: docker_client.clone(),
nodes: RwLock::new(HashMap::new()),
delete_on_drop: Arc::new(Mutex::new(true)),
});
namespace.initialize().await?;
Ok(namespace)
}
pub(super) async fn attach_to_live(
provider: &Weak<DockerProvider<FS>>,
capabilities: &ProviderCapabilities,
docker_client: &DockerClient,
filesystem: &FS,
custom_base_dir: &Path,
name: &str,
) -> Result<Arc<Self>, ProviderError> {
let base_dir = custom_base_dir.to_path_buf();
let namespace = Arc::new_cyclic(|weak| DockerNamespace {
weak: weak.clone(),
provider: provider.clone(),
name: name.to_owned(),
base_dir,
capabilities: capabilities.clone(),
filesystem: filesystem.clone(),
docker_client: docker_client.clone(),
nodes: RwLock::new(HashMap::new()),
delete_on_drop: Arc::new(Mutex::new(false)),
});
Ok(namespace)
}
async fn initialize(&self) -> Result<(), ProviderError> {
// let ns_scripts_shared = PathBuf::from_iter([&self.base_dir, &PathBuf::from("shared-scripts")]);
// self.filesystem.create_dir(&ns_scripts_shared).await?;
self.initialize_zombie_scripts_volume().await?;
self.initialize_helper_binaries_volume().await?;
Ok(())
}
async fn initialize_zombie_scripts_volume(&self) -> Result<(), ProviderError> {
let local_zombie_wrapper_path =
PathBuf::from_iter([&self.base_dir, &PathBuf::from("zombie-wrapper.sh")]);
self.filesystem
.write(
&local_zombie_wrapper_path,
include_str!("../shared/scripts/zombie-wrapper.sh"),
)
.await?;
let local_helper_binaries_downloader_path = PathBuf::from_iter([
&self.base_dir,
&PathBuf::from("helper-binaries-downloader.sh"),
]);
self.filesystem
.write(
&local_helper_binaries_downloader_path,
include_str!("../shared/scripts/helper-binaries-downloader.sh"),
)
.await?;
let zombie_wrapper_volume_name = format!("{}-zombie-wrapper", self.name);
let zombie_wrapper_container_name = format!("{}-scripts", self.name);
self.docker_client
.create_volume(&zombie_wrapper_volume_name)
.await
.map_err(|err| ProviderError::CreateNamespaceFailed(self.name.clone(), err.into()))?;
self.docker_client
.container_create(
ContainerRunOptions::new("alpine:latest", vec!["tail", "-f", "/dev/null"])
.volume_mounts(HashMap::from([(
zombie_wrapper_volume_name.as_str(),
"/scripts",
)]))
.name(&zombie_wrapper_container_name)
.rm(),
)
.await
.map_err(|err| ProviderError::CreateNamespaceFailed(self.name.clone(), err.into()))?;
// copy the scripts
self.docker_client
.container_cp(
&zombie_wrapper_container_name,
&local_zombie_wrapper_path,
&PathBuf::from("/scripts/zombie-wrapper.sh"),
)
.await
.map_err(|err| ProviderError::CreateNamespaceFailed(self.name.clone(), err.into()))?;
self.docker_client
.container_cp(
&zombie_wrapper_container_name,
&local_helper_binaries_downloader_path,
&PathBuf::from("/scripts/helper-binaries-downloader.sh"),
)
.await
.map_err(|err| ProviderError::CreateNamespaceFailed(self.name.clone(), err.into()))?;
// set permissions for rwx on whole volume recursively
self.docker_client
.container_run(
ContainerRunOptions::new("alpine:latest", vec!["chmod", "-R", "777", "/scripts"])
.volume_mounts(HashMap::from([(
zombie_wrapper_volume_name.as_ref(),
"/scripts",
)]))
.rm(),
)
.await
.map_err(|err| ProviderError::CreateNamespaceFailed(self.name.clone(), err.into()))?;
Ok(())
}
async fn initialize_helper_binaries_volume(&self) -> Result<(), ProviderError> {
let helper_binaries_volume_name = format!("{}-helper-binaries", self.name);
let zombie_wrapper_volume_name = format!("{}-zombie-wrapper", self.name);
self.docker_client
.create_volume(&helper_binaries_volume_name)
.await
.map_err(|err| ProviderError::CreateNamespaceFailed(self.name.clone(), err.into()))?;
// download binaries to volume
self.docker_client
.container_run(
ContainerRunOptions::new(
"alpine:latest",
vec!["ash", "/scripts/helper-binaries-downloader.sh"],
)
.volume_mounts(HashMap::from([
(
helper_binaries_volume_name.as_str(),
"/helpers",
),
(
zombie_wrapper_volume_name.as_ref(),
"/scripts",
)
]))
// wait until complete
.detach(false)
.rm(),
)
.await
.map_err(|err| ProviderError::CreateNamespaceFailed(self.name.clone(), err.into()))?;
// set permissions for rwx on whole volume recursively
self.docker_client
.container_run(
ContainerRunOptions::new("alpine:latest", vec!["chmod", "-R", "777", "/helpers"])
.volume_mounts(HashMap::from([(
helper_binaries_volume_name.as_ref(),
"/helpers",
)]))
.rm(),
)
.await
.map_err(|err| ProviderError::CreateNamespaceFailed(self.name.clone(), err.into()))?;
Ok(())
}
pub async fn set_delete_on_drop(&self, delete_on_drop: bool) {
*self.delete_on_drop.lock().await = delete_on_drop;
}
pub async fn delete_on_drop(&self) -> bool {
if let Ok(delete_on_drop) = self.delete_on_drop.try_lock() {
*delete_on_drop
} else {
// if we can't lock just remove the ns
true
}
}
}
#[async_trait]
impl<FS> ProviderNamespace for DockerNamespace<FS>
where
FS: FileSystem + Send + Sync + Clone + 'static,
{
fn name(&self) -> &str {
&self.name
}
fn base_dir(&self) -> &PathBuf {
&self.base_dir
}
fn capabilities(&self) -> &ProviderCapabilities {
&self.capabilities
}
fn provider_name(&self) -> &str {
provider::PROVIDER_NAME
}
async fn detach(&self) {
self.set_delete_on_drop(false).await;
}
async fn is_detached(&self) -> bool {
self.delete_on_drop().await
}
async fn nodes(&self) -> HashMap<String, DynNode> {
self.nodes
.read()
.await
.iter()
.map(|(name, node)| (name.clone(), node.clone() as DynNode))
.collect()
}
async fn get_node_available_args(
&self,
(command, image): (String, Option<String>),
) -> Result<String, ProviderError> {
let node_image = image.expect(&format!("image should be present when getting node available args with docker provider {THIS_IS_A_BUG}"));
let temp_node = self
.spawn_node(
&SpawnNodeOptions::new(format!("temp-{}", Uuid::new_v4()), "cat".to_string())
.image(node_image.clone()),
)
.await?;
let available_args_output = temp_node
.run_command(RunCommandOptions::new(command.clone()).args(vec!["--help"]))
.await?
.map_err(|(_exit, status)| {
ProviderError::NodeAvailableArgsError(node_image, command, status)
})?;
temp_node.destroy().await?;
Ok(available_args_output)
}
async fn spawn_node(&self, options: &SpawnNodeOptions) -> Result<DynNode, ProviderError> {
debug!("spawn option {:?}", options);
let node = DockerNode::new(DockerNodeOptions {
namespace: &self.weak,
namespace_base_dir: &self.base_dir,
name: &options.name,
image: options.image.as_ref(),
program: &options.program,
args: &options.args,
env: &options.env,
startup_files: &options.injected_files,
db_snapshot: options.db_snapshot.as_ref(),
docker_client: &self.docker_client,
container_name: format!("{}-{}", self.name, options.name),
filesystem: &self.filesystem,
port_mapping: options.port_mapping.as_ref().unwrap_or(&HashMap::default()),
})
.await?;
self.nodes
.write()
.await
.insert(node.name().to_string(), node.clone());
Ok(node)
}
async fn spawn_node_from_json(
&self,
json_value: &serde_json::Value,
) -> Result<DynNode, ProviderError> {
let deserializable: DeserializableDockerNodeOptions =
serde_json::from_value(json_value.clone())?;
let options = DockerNodeOptions::from_deserializable(
&deserializable,
&self.weak,
&self.base_dir,
&self.docker_client,
&self.filesystem,
);
let node = DockerNode::attach_to_live(options).await?;
self.nodes
.write()
.await
.insert(node.name().to_string(), node.clone());
Ok(node)
}
async fn generate_files(&self, options: GenerateFilesOptions) -> Result<(), ProviderError> {
debug!("generate files options {options:#?}");
let node_name = options
.temp_name
.unwrap_or_else(|| format!("temp-{}", Uuid::new_v4()));
let node_image = options.image.expect(&format!(
"image should be present when generating files with docker provider {THIS_IS_A_BUG}"
));
// run dummy command in a new container
let temp_node = self
.spawn_node(
&SpawnNodeOptions::new(node_name, "cat".to_string())
.injected_files(options.injected_files)
.image(node_image),
)
.await?;
for GenerateFileCommand {
program,
args,
env,
local_output_path,
} in options.commands
{
let local_output_full_path = format!(
"{}{}{}",
self.base_dir.to_string_lossy(),
if local_output_path.starts_with("/") {
""
} else {
"/"
},
local_output_path.to_string_lossy()
);
let contents = extract_execution_result(
&temp_node,
RunCommandOptions { program, args, env },
options.expected_path.as_ref(),
)
.await?;
self.filesystem
.write(local_output_full_path, contents)
.await
.map_err(|err| ProviderError::FileGenerationFailed(err.into()))?;
}
temp_node.destroy().await
}
async fn static_setup(&self) -> Result<(), ProviderError> {
todo!()
}
async fn destroy(&self) -> Result<(), ProviderError> {
let _ = self
.docker_client
.namespaced_containers_rm(&self.name)
.await
.map_err(|err| ProviderError::DeleteNamespaceFailed(self.name.clone(), err.into()))?;
if let Some(provider) = self.provider.upgrade() {
provider.namespaces.write().await.remove(&self.name);
}
Ok(())
}
}
impl<FS> Drop for DockerNamespace<FS>
where
FS: FileSystem + Send + Sync + Clone,
{
fn drop(&mut self) {
let ns_name = self.name.clone();
if let Ok(delete_on_drop) = self.delete_on_drop.try_lock() {
if *delete_on_drop {
let client = self.docker_client.clone();
let provider = self.provider.upgrade();
let handler = thread::spawn(move || {
let rt = tokio::runtime::Runtime::new().unwrap();
rt.block_on(async move {
trace!("🧟 deleting ns {ns_name} from cluster");
let _ = client.namespaced_containers_rm(&ns_name).await;
trace!("✅ deleted");
});
});
if handler.join().is_ok() {
if let Some(provider) = provider {
if let Ok(mut p) = provider.namespaces.try_write() {
p.remove(&self.name);
} else {
warn!(
"⚠️ Can not acquire write lock to the provider, ns {} not removed",
self.name
);
}
}
}
} else {
trace!("⚠️ leaking ns {ns_name} in cluster");
}
};
}
}
vendor/pezkuwi-zombienet-sdk/crates/provider/src/docker/node.rs
Vendored
+659
View File
@@ -0,0 +1,659 @@
use std::{
collections::HashMap,
net::IpAddr,
path::{Component, Path, PathBuf},
sync::{Arc, Weak},
time::Duration,
};
use anyhow::anyhow;
use async_trait::async_trait;
use configuration::types::AssetLocation;
use futures::future::try_join_all;
use serde::{Deserialize, Serialize};
use support::{constants::THIS_IS_A_BUG, fs::FileSystem};
use tokio::{time::sleep, try_join};
use tracing::debug;
use super::{
client::{ContainerRunOptions, DockerClient},
namespace::DockerNamespace,
};
use crate::{
constants::{NODE_CONFIG_DIR, NODE_DATA_DIR, NODE_RELAY_DATA_DIR, NODE_SCRIPTS_DIR},
docker,
types::{ExecutionResult, Port, RunCommandOptions, RunScriptOptions, TransferedFile},
ProviderError, ProviderNamespace, ProviderNode,
};
pub(super) struct DockerNodeOptions<'a, FS>
where
FS: FileSystem + Send + Sync + Clone + 'static,
{
pub(super) namespace: &'a Weak<DockerNamespace<FS>>,
pub(super) namespace_base_dir: &'a PathBuf,
pub(super) name: &'a str,
pub(super) image: Option<&'a String>,
pub(super) program: &'a str,
pub(super) args: &'a [String],
pub(super) env: &'a [(String, String)],
pub(super) startup_files: &'a [TransferedFile],
pub(super) db_snapshot: Option<&'a AssetLocation>,
pub(super) docker_client: &'a DockerClient,
pub(super) container_name: String,
pub(super) filesystem: &'a FS,
pub(super) port_mapping: &'a HashMap<Port, Port>,
}
impl<'a, FS> DockerNodeOptions<'a, FS>
where
FS: FileSystem + Send + Sync + Clone + 'static,
{
pub fn from_deserializable(
deserializable: &'a DeserializableDockerNodeOptions,
namespace: &'a Weak<DockerNamespace<FS>>,
namespace_base_dir: &'a PathBuf,
docker_client: &'a DockerClient,
filesystem: &'a FS,
) -> Self {
DockerNodeOptions {
namespace,
namespace_base_dir,
name: &deserializable.name,
image: deserializable.image.as_ref(),
program: &deserializable.program,
args: &deserializable.args,
env: &deserializable.env,
startup_files: &[],
db_snapshot: None,
docker_client,
container_name: deserializable.container_name.clone(),
filesystem,
port_mapping: &deserializable.port_mapping,
}
}
}
#[derive(Deserialize)]
pub(super) struct DeserializableDockerNodeOptions {
pub(super) name: String,
pub(super) image: Option<String>,
pub(super) program: String,
pub(super) args: Vec<String>,
pub(super) env: Vec<(String, String)>,
pub(super) container_name: String,
pub(super) port_mapping: HashMap<Port, Port>,
}
#[derive(Serialize)]
pub struct DockerNode<FS>
where
FS: FileSystem + Send + Sync + Clone,
{
#[serde(skip)]
namespace: Weak<DockerNamespace<FS>>,
name: String,
image: String,
program: String,
args: Vec<String>,
env: Vec<(String, String)>,
base_dir: PathBuf,
config_dir: PathBuf,
data_dir: PathBuf,
relay_data_dir: PathBuf,
scripts_dir: PathBuf,
log_path: PathBuf,
#[serde(skip)]
docker_client: DockerClient,
container_name: String,
port_mapping: HashMap<Port, Port>,
#[allow(dead_code)]
#[serde(skip)]
filesystem: FS,
provider_tag: String,
}
impl<FS> DockerNode<FS>
where
FS: FileSystem + Send + Sync + Clone + 'static,
{
pub(super) async fn new(
options: DockerNodeOptions<'_, FS>,
) -> Result<Arc<Self>, ProviderError> {
let image = options.image.ok_or_else(|| {
ProviderError::MissingNodeInfo(options.name.to_string(), "missing image".to_string())
})?;
let filesystem = options.filesystem.clone();
let base_dir =
PathBuf::from_iter([options.namespace_base_dir, &PathBuf::from(options.name)]);
filesystem.create_dir_all(&base_dir).await?;
let base_dir_raw = base_dir.to_string_lossy();
let config_dir = PathBuf::from(format!("{base_dir_raw}{NODE_CONFIG_DIR}"));
let data_dir = PathBuf::from(format!("{base_dir_raw}{NODE_DATA_DIR}"));
let relay_data_dir = PathBuf::from(format!("{base_dir_raw}{NODE_RELAY_DATA_DIR}"));
let scripts_dir = PathBuf::from(format!("{base_dir_raw}{NODE_SCRIPTS_DIR}"));
let log_path = base_dir.join("node.log");
try_join!(
filesystem.create_dir_all(&config_dir),
filesystem.create_dir_all(&data_dir),
filesystem.create_dir_all(&relay_data_dir),
filesystem.create_dir_all(&scripts_dir),
)?;
let node = Arc::new(DockerNode {
namespace: options.namespace.clone(),
name: options.name.to_string(),
image: image.to_string(),
program: options.program.to_string(),
args: options.args.to_vec(),
env: options.env.to_vec(),
base_dir,
config_dir,
data_dir,
relay_data_dir,
scripts_dir,
log_path,
filesystem: filesystem.clone(),
docker_client: options.docker_client.clone(),
container_name: options.container_name,
port_mapping: options.port_mapping.clone(),
provider_tag: docker::provider::PROVIDER_NAME.to_string(),
});
node.initialize_docker().await?;
if let Some(db_snap) = options.db_snapshot {
node.initialize_db_snapshot(db_snap).await?;
}
node.initialize_startup_files(options.startup_files).await?;
node.start().await?;
Ok(node)
}
pub(super) async fn attach_to_live(
options: DockerNodeOptions<'_, FS>,
) -> Result<Arc<Self>, ProviderError> {
let image = options.image.ok_or_else(|| {
ProviderError::MissingNodeInfo(options.name.to_string(), "missing image".to_string())
})?;
let filesystem = options.filesystem.clone();
let base_dir =
PathBuf::from_iter([options.namespace_base_dir, &PathBuf::from(options.name)]);
filesystem.create_dir_all(&base_dir).await?;
let base_dir_raw = base_dir.to_string_lossy();
let config_dir = PathBuf::from(format!("{base_dir_raw}{NODE_CONFIG_DIR}"));
let data_dir = PathBuf::from(format!("{base_dir_raw}{NODE_DATA_DIR}"));
let relay_data_dir = PathBuf::from(format!("{base_dir_raw}{NODE_RELAY_DATA_DIR}"));
let scripts_dir = PathBuf::from(format!("{base_dir_raw}{NODE_SCRIPTS_DIR}"));
let log_path = base_dir.join("node.log");
let node = Arc::new(DockerNode {
namespace: options.namespace.clone(),
name: options.name.to_string(),
image: image.to_string(),
program: options.program.to_string(),
args: options.args.to_vec(),
env: options.env.to_vec(),
base_dir,
config_dir,
data_dir,
relay_data_dir,
scripts_dir,
log_path,
filesystem: filesystem.clone(),
docker_client: options.docker_client.clone(),
container_name: options.container_name,
port_mapping: options.port_mapping.clone(),
provider_tag: docker::provider::PROVIDER_NAME.to_string(),
});
Ok(node)
}
async fn initialize_docker(&self) -> Result<(), ProviderError> {
let command = [vec![self.program.to_string()], self.args.to_vec()].concat();
self.docker_client
.container_run(
ContainerRunOptions::new(&self.image, command)
.name(&self.container_name)
.env(self.env.clone())
.volume_mounts(HashMap::from([
(
format!("{}-zombie-wrapper", self.namespace_name(),),
"/scripts".to_string(),
),
(
format!("{}-helper-binaries", self.namespace_name()),
"/helpers".to_string(),
),
(
self.config_dir.to_string_lossy().into_owned(),
"/cfg".to_string(),
),
(
self.data_dir.to_string_lossy().into_owned(),
"/data".to_string(),
),
(
self.relay_data_dir.to_string_lossy().into_owned(),
"/relay-data".to_string(),
),
]))
.entrypoint("/scripts/zombie-wrapper.sh")
.port_mapping(&self.port_mapping),
)
.await
.map_err(|err| ProviderError::NodeSpawningFailed(self.name.clone(), err.into()))?;
// change dirs permission
let _ = self
.docker_client
.container_exec(
&self.container_name,
["chmod", "777", "/cfg", "/data", "/relay-data"].into(),
None,
Some("root"),
)
.await
.map_err(|err| ProviderError::NodeSpawningFailed(self.name.clone(), err.into()))?;
Ok(())
}
async fn initialize_db_snapshot(
&self,
_db_snapshot: &AssetLocation,
) -> Result<(), ProviderError> {
todo!()
// trace!("snap: {db_snapshot}");
// let url_of_snap = match db_snapshot {
// AssetLocation::Url(location) => location.clone(),
// AssetLocation::FilePath(filepath) => self.upload_to_fileserver(filepath).await?,
// };
// // we need to get the snapshot from a public access
// // and extract to /data
// let opts = RunCommandOptions::new("mkdir").args([
// "-p",
// "/data/",
// "&&",
// "mkdir",
// "-p",
// "/relay-data/",
// "&&",
// // Use our version of curl
// "/cfg/curl",
// url_of_snap.as_ref(),
// "--output",
// "/data/db.tgz",
// "&&",
// "cd",
// "/",
// "&&",
// "tar",
// "--skip-old-files",
// "-xzvf",
// "/data/db.tgz",
// ]);
// trace!("cmd opts: {:#?}", opts);
// let _ = self.run_command(opts).await?;
// Ok(())
}
async fn initialize_startup_files(
&self,
startup_files: &[TransferedFile],
) -> Result<(), ProviderError> {
try_join_all(
startup_files
.iter()
.map(|file| self.send_file(&file.local_path, &file.remote_path, &file.mode)),
)
.await?;
Ok(())
}
pub(super) async fn start(&self) -> Result<(), ProviderError> {
self.docker_client
.container_exec(
&self.container_name,
vec!["sh", "-c", "echo start > /tmp/zombiepipe"],
None,
None,
)
.await
.map_err(|err| {
ProviderError::NodeSpawningFailed(
format!("failed to start pod {} after spawning", self.name),
err.into(),
)
})?
.map_err(|err| {
ProviderError::NodeSpawningFailed(
format!("failed to start pod {} after spawning", self.name,),
anyhow!("command failed in container: status {}: {}", err.0, err.1),
)
})?;
Ok(())
}
fn get_remote_parent_dir(&self, remote_file_path: &Path) -> Option<PathBuf> {
if let Some(remote_parent_dir) = remote_file_path.parent() {
if matches!(
remote_parent_dir.components().rev().peekable().peek(),
Some(Component::Normal(_))
) {
return Some(remote_parent_dir.to_path_buf());
}
}
None
}
async fn create_remote_dir(&self, remote_dir: &Path) -> Result<(), ProviderError> {
let _ = self
.docker_client
.container_exec(
&self.container_name,
vec!["mkdir", "-p", &remote_dir.to_string_lossy()],
None,
None,
)
.await
.map_err(|err| {
ProviderError::NodeSpawningFailed(
format!(
"failed to create dir {} for container {}",
remote_dir.to_string_lossy(),
&self.name
),
err.into(),
)
})?;
Ok(())
}
fn namespace_name(&self) -> String {
self.namespace
.upgrade()
.map(|namespace| namespace.name().to_string())
.unwrap_or_else(|| panic!("namespace shouldn't be dropped, {THIS_IS_A_BUG}"))
}
}
#[async_trait]
impl<FS> ProviderNode for DockerNode<FS>
where
FS: FileSystem + Send + Sync + Clone + 'static,
{
fn name(&self) -> &str {
&self.name
}
fn args(&self) -> Vec<&str> {
self.args.iter().map(|arg| arg.as_str()).collect()
}
fn base_dir(&self) -> &PathBuf {
&self.base_dir
}
fn config_dir(&self) -> &PathBuf {
&self.config_dir
}
fn data_dir(&self) -> &PathBuf {
&self.data_dir
}
fn relay_data_dir(&self) -> &PathBuf {
&self.relay_data_dir
}
fn scripts_dir(&self) -> &PathBuf {
&self.scripts_dir
}
fn log_path(&self) -> &PathBuf {
&self.log_path
}
fn log_cmd(&self) -> String {
format!(
"{} logs -f {}",
self.docker_client.client_binary(),
self.container_name
)
}
fn path_in_node(&self, file: &Path) -> PathBuf {
// here is just a noop op since we will receive the path
// for the file inside the pod
PathBuf::from(file)
}
async fn logs(&self) -> Result<String, ProviderError> {
self.docker_client
.container_logs(&self.container_name)
.await
.map_err(|err| ProviderError::GetLogsFailed(self.name.to_string(), err.into()))
}
async fn dump_logs(&self, local_dest: PathBuf) -> Result<(), ProviderError> {
let logs = self.logs().await?;
self.filesystem
.write(local_dest, logs)
.await
.map_err(|err| ProviderError::DumpLogsFailed(self.name.to_string(), err.into()))?;
Ok(())
}
async fn run_command(
&self,
options: RunCommandOptions,
) -> Result<ExecutionResult, ProviderError> {
debug!(
"running command for {} with options {:?}",
self.name, options
);
let command = [vec![options.program], options.args].concat();
self.docker_client
.container_exec(
&self.container_name,
vec!["sh", "-c", &command.join(" ")],
Some(
options
.env
.iter()
.map(|(k, v)| (k.as_ref(), v.as_ref()))
.collect(),
),
None,
)
.await
.map_err(|err| {
ProviderError::RunCommandError(
format!("sh -c {}", &command.join(" ")),
format!("in pod {}", self.name),
err.into(),
)
})
}
async fn run_script(
&self,
_options: RunScriptOptions,
) -> Result<ExecutionResult, ProviderError> {
todo!()
}
async fn send_file(
&self,
local_file_path: &Path,
remote_file_path: &Path,
mode: &str,
) -> Result<(), ProviderError> {
if let Some(remote_parent_dir) = self.get_remote_parent_dir(remote_file_path) {
self.create_remote_dir(&remote_parent_dir).await?;
}
debug!(
"starting sending file for {}: {} to {} with mode {}",
self.name,
local_file_path.to_string_lossy(),
remote_file_path.to_string_lossy(),
mode
);
let _ = self
.docker_client
.container_cp(&self.container_name, local_file_path, remote_file_path)
.await
.map_err(|err| {
ProviderError::SendFile(
local_file_path.to_string_lossy().to_string(),
self.name.clone(),
err.into(),
)
});
let _ = self
.docker_client
.container_exec(
&self.container_name,
vec!["chmod", mode, &remote_file_path.to_string_lossy()],
None,
None,
)
.await
.map_err(|err| {
ProviderError::SendFile(
self.name.clone(),
local_file_path.to_string_lossy().to_string(),
err.into(),
)
})?;
Ok(())
}
async fn receive_file(
&self,
_remote_src: &Path,
_local_dest: &Path,
) -> Result<(), ProviderError> {
Ok(())
}
async fn ip(&self) -> Result<IpAddr, ProviderError> {
let ip = self
.docker_client
.container_ip(&self.container_name)
.await
.map_err(|err| {
ProviderError::InvalidConfig(format!("Error getting container ip, err: {err}"))
})?;
Ok(ip.parse::<IpAddr>().map_err(|err| {
ProviderError::InvalidConfig(format!(
"Can not parse the container ip: {ip}, err: {err}"
))
})?)
}
async fn pause(&self) -> Result<(), ProviderError> {
self.docker_client
.container_exec(
&self.container_name,
vec!["sh", "-c", "echo pause > /tmp/zombiepipe"],
None,
None,
)
.await
.map_err(|err| ProviderError::PauseNodeFailed(self.name.to_string(), err.into()))?
.map_err(|err| {
ProviderError::PauseNodeFailed(
self.name.to_string(),
anyhow!("error when pausing node: status {}: {}", err.0, err.1),
)
})?;
Ok(())
}
async fn resume(&self) -> Result<(), ProviderError> {
self.docker_client
.container_exec(
&self.container_name,
vec!["sh", "-c", "echo resume > /tmp/zombiepipe"],
None,
None,
)
.await
.map_err(|err| ProviderError::PauseNodeFailed(self.name.to_string(), err.into()))?
.map_err(|err| {
ProviderError::PauseNodeFailed(
self.name.to_string(),
anyhow!("error when pausing node: status {}: {}", err.0, err.1),
)
})?;
Ok(())
}
async fn restart(&self, after: Option<Duration>) -> Result<(), ProviderError> {
if let Some(duration) = after {
sleep(duration).await;
}
self.docker_client
.container_exec(
&self.container_name,
vec!["sh", "-c", "echo restart > /tmp/zombiepipe"],
None,
None,
)
.await
.map_err(|err| ProviderError::PauseNodeFailed(self.name.to_string(), err.into()))?
.map_err(|err| {
ProviderError::PauseNodeFailed(
self.name.to_string(),
anyhow!("error when pausing node: status {}: {}", err.0, err.1),
)
})?;
Ok(())
}
async fn destroy(&self) -> Result<(), ProviderError> {
self.docker_client
.container_rm(&self.container_name)
.await
.map_err(|err| ProviderError::KillNodeFailed(self.name.to_string(), err.into()))?;
if let Some(namespace) = self.namespace.upgrade() {
namespace.nodes.write().await.remove(&self.name);
}
Ok(())
}
}
vendor/pezkuwi-zombienet-sdk/crates/provider/src/docker/provider.rs
Vendored
+161
View File
@@ -0,0 +1,161 @@
use std::{
collections::HashMap,
path::{Path, PathBuf},
sync::{Arc, Weak},
};
use async_trait::async_trait;
use support::fs::FileSystem;
use tokio::sync::RwLock;
use super::{client::DockerClient, namespace::DockerNamespace};
use crate::{
shared::helpers::extract_namespace_info, types::ProviderCapabilities, DynNamespace, Provider,
ProviderError, ProviderNamespace,
};
pub const PROVIDER_NAME: &str = "docker";
pub struct DockerProvider<FS>
where
FS: FileSystem + Send + Sync + Clone,
{
weak: Weak<DockerProvider<FS>>,
capabilities: ProviderCapabilities,
tmp_dir: PathBuf,
docker_client: DockerClient,
filesystem: FS,
pub(super) namespaces: RwLock<HashMap<String, Arc<DockerNamespace<FS>>>>,
}
impl<FS> DockerProvider<FS>
where
FS: FileSystem + Send + Sync + Clone + 'static,
{
pub async fn new(filesystem: FS) -> Arc<Self> {
let docker_client = DockerClient::new().await.unwrap();
let provider = Arc::new_cyclic(|weak| DockerProvider {
weak: weak.clone(),
capabilities: ProviderCapabilities {
requires_image: true,
has_resources: false,
prefix_with_full_path: false,
use_default_ports_in_cmd: true,
},
tmp_dir: std::env::temp_dir(),
docker_client,
filesystem,
namespaces: RwLock::new(HashMap::new()),
});
let cloned_provider = provider.clone();
tokio::spawn(async move {
tokio::signal::ctrl_c().await.unwrap();
for (_, ns) in cloned_provider.namespaces().await {
if ns.is_detached().await {
// best effort
let _ = ns.destroy().await;
}
}
// exit the process (130, SIGINT)
std::process::exit(130)
});
provider
}
pub fn tmp_dir(mut self, tmp_dir: impl Into<PathBuf>) -> Self {
self.tmp_dir = tmp_dir.into();
self
}
}
#[async_trait]
impl<FS> Provider for DockerProvider<FS>
where
FS: FileSystem + Send + Sync + Clone + 'static,
{
fn name(&self) -> &str {
PROVIDER_NAME
}
fn capabilities(&self) -> &ProviderCapabilities {
&self.capabilities
}
async fn namespaces(&self) -> HashMap<String, DynNamespace> {
self.namespaces
.read()
.await
.iter()
.map(|(name, namespace)| (name.clone(), namespace.clone() as DynNamespace))
.collect()
}
async fn create_namespace(&self) -> Result<DynNamespace, ProviderError> {
let namespace = DockerNamespace::new(
&self.weak,
&self.tmp_dir,
&self.capabilities,
&self.docker_client,
&self.filesystem,
None,
)
.await?;
self.namespaces
.write()
.await
.insert(namespace.name().to_string(), namespace.clone());
Ok(namespace)
}
async fn create_namespace_with_base_dir(
&self,
base_dir: &Path,
) -> Result<DynNamespace, ProviderError> {
let namespace = DockerNamespace::new(
&self.weak,
&self.tmp_dir,
&self.capabilities,
&self.docker_client,
&self.filesystem,
Some(base_dir),
)
.await?;
self.namespaces
.write()
.await
.insert(namespace.name().to_string(), namespace.clone());
Ok(namespace)
}
async fn create_namespace_from_json(
&self,
json_value: &serde_json::Value,
) -> Result<DynNamespace, ProviderError> {
let (base_dir, name) = extract_namespace_info(json_value)?;
let namespace = DockerNamespace::attach_to_live(
&self.weak,
&self.capabilities,
&self.docker_client,
&self.filesystem,
&base_dir,
&name,
)
.await?;
self.namespaces
.write()
.await
.insert(namespace.name().to_string(), namespace.clone());
Ok(namespace)
}
}
vendor/pezkuwi-zombienet-sdk/crates/provider/src/kubernetes.rs
Vendored
+7
View File
@@ -0,0 +1,7 @@
mod client;
mod namespace;
mod node;
mod pod_spec_builder;
mod provider;
pub use provider::KubernetesProvider;
vendor/pezkuwi-zombienet-sdk/crates/provider/src/kubernetes/client.rs
Vendored
+602
View File
@@ -0,0 +1,602 @@
use std::{
collections::BTreeMap, fmt::Debug, os::unix::process::ExitStatusExt, process::ExitStatus,
time::Duration,
};
use anyhow::anyhow;
use futures::{StreamExt, TryStreamExt};
use k8s_openapi::api::core::v1::{
ConfigMap, Namespace, Pod, PodSpec, PodStatus, Service, ServiceSpec,
};
use kube::{
api::{AttachParams, DeleteParams, ListParams, LogParams, PostParams, WatchParams},
core::{DynamicObject, GroupVersionKind, ObjectMeta, TypeMeta, WatchEvent},
discovery::ApiResource,
runtime::{conditions, wait::await_condition},
Api, Resource,
};
use serde::de::DeserializeOwned;
use support::constants::THIS_IS_A_BUG;
use tokio::{
io::{AsyncRead, ErrorKind},
net::TcpListener,
task::JoinHandle,
};
use tokio_util::compat::FuturesAsyncReadCompatExt;
use tracing::{debug, trace};
use crate::{constants::LOCALHOST, types::ExecutionResult};
#[derive(thiserror::Error, Debug)]
#[error(transparent)]
pub struct Error(#[from] anyhow::Error);
pub type Result<T> = core::result::Result<T, Error>;
#[derive(Clone)]
pub struct KubernetesClient {
inner: kube::Client,
}
impl KubernetesClient {
pub(super) async fn new() -> Result<Self> {
Ok(Self {
// TODO: make it more flexible with path to kube config
inner: kube::Client::try_default()
.await
.map_err(|err| Error::from(anyhow!("error initializing kubers client: {err}")))?,
})
}
#[allow(dead_code)]
pub(super) async fn get_namespace(&self, name: &str) -> Result<Option<Namespace>> {
Api::<Namespace>::all(self.inner.clone())
.get_opt(name.as_ref())
.await
.map_err(|err| Error::from(anyhow!("error while getting namespace {name}: {err}")))
}
#[allow(dead_code)]
pub(super) async fn get_namespaces(&self) -> Result<Vec<Namespace>> {
Ok(Api::<Namespace>::all(self.inner.clone())
.list(&ListParams::default())
.await
.map_err(|err| Error::from(anyhow!("error while getting all namespaces: {err}")))?
.into_iter()
.filter(|ns| matches!(&ns.meta().name, Some(name) if name.starts_with("zombienet")))
.collect())
}
pub(super) async fn create_namespace(
&self,
name: &str,
labels: BTreeMap<String, String>,
) -> Result<Namespace> {
let namespaces = Api::<Namespace>::all(self.inner.clone());
let namespace = Namespace {
metadata: ObjectMeta {
name: Some(name.to_string()),
labels: Some(labels),
..Default::default()
},
..Default::default()
};
namespaces
.create(&PostParams::default(), &namespace)
.await
.map_err(|err| Error::from(anyhow!("error while created namespace {name}: {err}")))?;
self.wait_created(namespaces, name).await?;
Ok(namespace)
}
pub(super) async fn delete_namespace(&self, name: &str) -> Result<()> {
let namespaces = Api::<Namespace>::all(self.inner.clone());
namespaces
.delete(name, &DeleteParams::default())
.await
.map_err(|err| Error::from(anyhow!("error while deleting namespace {name}: {err}")))?;
Ok(())
}
pub(super) async fn create_config_map_from_file(
&self,
namespace: &str,
name: &str,
file_name: &str,
file_contents: &str,
labels: BTreeMap<String, String>,
) -> Result<ConfigMap> {
let config_maps = Api::<ConfigMap>::namespaced(self.inner.clone(), namespace);
let config_map = ConfigMap {
metadata: ObjectMeta {
name: Some(name.to_string()),
namespace: Some(namespace.to_string()),
labels: Some(labels),
..Default::default()
},
data: Some(BTreeMap::from([(
file_name.to_string(),
file_contents.to_string(),
)])),
..Default::default()
};
config_maps
.create(&PostParams::default(), &config_map)
.await
.map_err(|err| {
Error::from(anyhow!(
"error while creating config map {name} for {file_name}: {err}"
))
})?;
self.wait_created(config_maps, name).await?;
Ok(config_map)
}
pub(super) async fn create_pod(
&self,
namespace: &str,
name: &str,
spec: PodSpec,
labels: BTreeMap<String, String>,
) -> Result<Pod> {
let pods = Api::<Pod>::namespaced(self.inner.clone(), namespace);
let pod = Pod {
metadata: ObjectMeta {
name: Some(name.to_string()),
namespace: Some(namespace.to_string()),
labels: Some(labels),
..Default::default()
},
spec: Some(spec),
..Default::default()
};
pods.create(&PostParams::default(), &pod)
.await
.map_err(|err| Error::from(anyhow!("error while creating pod {name}: {err}")))?;
trace!("Pod {name} checking for ready state!");
let wait_ready = await_condition(pods, name, helpers::is_pod_ready());
// TODO: we should use the `node_spawn_timeout` from global settings here.
let _ = tokio::time::timeout(Duration::from_secs(600), wait_ready)
.await
.map_err(|err| {
Error::from(anyhow!("error while awaiting pod {name} running: {err}"))
})?;
debug!("Pod {name} is Ready!");
Ok(pod)
}
pub(super) async fn pod_logs(&self, namespace: &str, name: &str) -> Result<String> {
Api::<Pod>::namespaced(self.inner.clone(), namespace)
.logs(
name,
&LogParams {
pretty: true,
timestamps: true,
..Default::default()
},
)
.await
.map_err(|err| Error::from(anyhow!("error while getting logs for pod {name}: {err}")))
}
pub(super) async fn pod_status(&self, namespace: &str, name: &str) -> Result<PodStatus> {
let pod = Api::<Pod>::namespaced(self.inner.clone(), namespace)
.get(name)
.await
.map_err(|err| Error::from(anyhow!("error while getting pod {name}: {err}")))?;
let status = pod.status.ok_or(Error::from(anyhow!(
"error while getting status for pod {name}"
)))?;
Ok(status)
}
#[allow(dead_code)]
pub(super) async fn create_pod_logs_stream(
&self,
namespace: &str,
name: &str,
) -> Result<Box<dyn AsyncRead + Send + Unpin>> {
Ok(Box::new(
Api::<Pod>::namespaced(self.inner.clone(), namespace)
.log_stream(
name,
&LogParams {
follow: true,
pretty: true,
timestamps: true,
..Default::default()
},
)
.await
.map_err(|err| {
Error::from(anyhow!(
"error while getting a log stream for {name}: {err}"
))
})?
.compat(),
))
}
pub(super) async fn pod_exec<S>(
&self,
namespace: &str,
name: &str,
command: Vec<S>,
) -> Result<ExecutionResult>
where
S: Into<String> + std::fmt::Debug + Send,
{
trace!("running command: {command:?} on pod {name} for ns {namespace}");
let mut process = Api::<Pod>::namespaced(self.inner.clone(), namespace)
.exec(
name,
command,
&AttachParams::default().stdout(true).stderr(true),
)
.await
.map_err(|err| Error::from(anyhow!("error while exec in the pod {name}: {err}")))?;
let stdout_stream = process.stdout().expect(&format!(
"stdout shouldn't be None when true passed to exec {THIS_IS_A_BUG}"
));
let stdout = tokio_util::io::ReaderStream::new(stdout_stream)
.filter_map(|r| async { r.ok().and_then(|v| String::from_utf8(v.to_vec()).ok()) })
.collect::<Vec<_>>()
.await
.join("");
let stderr_stream = process.stderr().expect(&format!(
"stderr shouldn't be None when true passed to exec {THIS_IS_A_BUG}"
));
let stderr = tokio_util::io::ReaderStream::new(stderr_stream)
.filter_map(|r| async { r.ok().and_then(|v| String::from_utf8(v.to_vec()).ok()) })
.collect::<Vec<_>>()
.await
.join("");
let status = process
.take_status()
.expect(&format!(
"first call to status shouldn't fail {THIS_IS_A_BUG}"
))
.await;
// await process to finish
process.join().await.map_err(|err| {
Error::from(anyhow!(
"error while joining process during exec for {name}: {err}"
))
})?;
match status {
// command succeeded with stdout
Some(status) if status.status.as_ref().is_some_and(|s| s == "Success") => {
Ok(Ok(stdout))
},
// command failed
Some(status) if status.status.as_ref().is_some_and(|s| s == "Failure") => {
match status.reason {
// due to exit code
Some(reason) if reason == "NonZeroExitCode" => {
let exit_status = status
.details
.and_then(|details| {
details.causes.and_then(|causes| {
causes.first().and_then(|cause| {
cause.message.as_deref().and_then(|message| {
message.parse::<i32>().ok().map(ExitStatus::from_raw)
})
})
})
})
.expect(
&format!("command with non-zero exit code should have exit code present {THIS_IS_A_BUG}")
);
Ok(Err((exit_status, stderr)))
},
// due to other unknown reason
Some(ref reason) => Err(Error::from(anyhow!(
format!("unhandled reason while exec for {name}: {reason}, stderr: {stderr}, status: {status:?}")
))),
None => {
panic!("command had status but no reason was present, this is a bug")
},
}
},
Some(_) => {
unreachable!("command had status but it didn't matches either Success or Failure, this is a bug from the kube.rs library itself");
},
None => {
panic!("command has no status following its execution, this is a bug");
},
}
}
pub(super) async fn delete_pod(&self, namespace: &str, name: &str) -> Result<()> {
let pods = Api::<Pod>::namespaced(self.inner.clone(), namespace);
pods.delete(name, &DeleteParams::default())
.await
.map_err(|err| Error::from(anyhow!("error when deleting pod {name}: {err}")))?;
await_condition(pods, name, conditions::is_deleted(name))
.await
.map_err(|err| {
Error::from(anyhow!(
"error when waiting for pod {name} to be deleted: {err}"
))
})?;
Ok(())
}
pub(super) async fn create_service(
&self,
namespace: &str,
name: &str,
spec: ServiceSpec,
labels: BTreeMap<String, String>,
) -> Result<Service> {
let services = Api::<Service>::namespaced(self.inner.clone(), namespace);
let service = Service {
metadata: ObjectMeta {
name: Some(name.to_string()),
namespace: Some(namespace.to_string()),
labels: Some(labels),
..Default::default()
},
spec: Some(spec),
..Default::default()
};
services
.create(&PostParams::default(), &service)
.await
.map_err(|err| Error::from(anyhow!("error while creating service {name}: {err}")))?;
Ok(service)
}
pub(super) async fn create_pod_port_forward(
&self,
namespace: &str,
name: &str,
local_port: u16,
remote_port: u16,
) -> Result<(u16, JoinHandle<()>)> {
let pods = Api::<Pod>::namespaced(self.inner.clone(), namespace);
let bind = TcpListener::bind((LOCALHOST, local_port))
.await
.map_err(|err| {
Error::from(anyhow!(
"error binding port {local_port} for pod {name}: {err}"
))
})?;
let local_port = bind.local_addr().map_err(|err| Error(err.into()))?.port();
let name = name.to_string();
const MAX_FAILURES: usize = 5;
let monitor_handle = tokio::spawn(async move {
let mut consecutive_failures = 0;
loop {
let (mut client_conn, _) = match bind.accept().await {
Ok(conn) => {
consecutive_failures = 0;
conn
},
Err(e) => {
if consecutive_failures < MAX_FAILURES {
trace!("Port-forward accept error: {e:?}, retrying in 1s");
tokio::time::sleep(Duration::from_secs(1)).await;
consecutive_failures += 1;
continue;
} else {
trace!("Port-forward accept failed too many times, giving up");
break;
}
},
};
let peer = match client_conn.peer_addr() {
Ok(addr) => addr,
Err(e) => {
trace!("Failed to get peer address: {e:?}");
break;
},
};
trace!("new connection on local_port: {local_port}, peer: {peer}");
let (name, pods) = (name.clone(), pods.clone());
tokio::spawn(async move {
loop {
// Try to establish port-forward
let mut forwarder = match pods.portforward(&name, &[remote_port]).await {
Ok(f) => {
consecutive_failures = 0;
f
},
Err(e) => {
consecutive_failures += 1;
if consecutive_failures < MAX_FAILURES {
trace!("portforward failed to establish ({}/{}): {e:?}, retrying in 1s",
consecutive_failures, MAX_FAILURES);
tokio::time::sleep(Duration::from_secs(1)).await;
continue;
} else {
trace!("portforward failed to establish after {} attempts: {e:?}, closing connection",
consecutive_failures);
break;
}
},
};
let mut upstream_conn = match forwarder.take_stream(remote_port) {
Some(s) => s,
None => {
trace!("Failed to take stream for remote_port: {remote_port}, retrying in 1s");
tokio::time::sleep(Duration::from_secs(1)).await;
continue;
},
};
match tokio::io::copy_bidirectional(&mut client_conn, &mut upstream_conn)
.await
{
Ok((_n1, _n2)) => {
// EOF reached, close connection
trace!("copy_bidirectional finished (EOF), closing connection");
drop(upstream_conn);
let _ = forwarder.join().await;
break;
},
Err(e) => {
let kind = e.kind();
match kind {
ErrorKind::ConnectionReset
| ErrorKind::ConnectionAborted
| ErrorKind::ConnectionRefused
| ErrorKind::TimedOut => {
consecutive_failures += 1;
if consecutive_failures < MAX_FAILURES {
trace!("Network error ({kind:?}): {e:?}, retrying port-forward for this connection");
tokio::time::sleep(Duration::from_secs(1)).await;
continue;
} else {
trace!("portforward failed to establish after {} attempts: {e:?}, closing connection",
consecutive_failures);
break;
}
},
_ => {
trace!("Non-network error ({kind:?}): {e:?}, closing connection");
break;
},
}
},
}
}
});
trace!("finished forwarder process for local port: {local_port}, peer: {peer}");
}
});
Ok((local_port, monitor_handle))
}
/// Create resources from yamls in `static-configs` directory
pub(super) async fn create_static_resource(
&self,
namespace: &str,
raw_manifest: &str,
) -> Result<()> {
let tm: TypeMeta = serde_yaml::from_str(raw_manifest).map_err(|err| {
Error::from(anyhow!(
"error while extracting TypeMeta from manifest: {raw_manifest}: {err}"
))
})?;
let gvk = GroupVersionKind::try_from(&tm).map_err(|err| {
Error::from(anyhow!(
"error while extracting GroupVersionKind from manifest: {raw_manifest}: {err}"
))
})?;
let ar = ApiResource::from_gvk(&gvk);
let api: Api<DynamicObject> = Api::namespaced_with(self.inner.clone(), namespace, &ar);
api.create(
&PostParams::default(),
&serde_yaml::from_str(raw_manifest).unwrap(),
)
.await
.map_err(|err| {
Error::from(anyhow!(
"error while creating static-config {raw_manifest}: {err}"
))
})?;
Ok(())
}
async fn wait_created<K>(&self, api: Api<K>, name: &str) -> Result<()>
where
K: Clone + DeserializeOwned + Debug,
{
let params = &WatchParams::default().fields(&format!("metadata.name={name}"));
let mut stream = api
.watch(params, "0")
.await
.map_err(|err| {
Error::from(anyhow!(
"error while awaiting first response when resource {name} is created: {err}"
))
})?
.boxed();
while let Some(status) = stream.try_next().await.map_err(|err| {
Error::from(anyhow!(
"error while awaiting next change after resource {name} is created: {err}"
))
})? {
match status {
WatchEvent::Added(_) => break,
WatchEvent::Error(err) => Err(Error::from(anyhow!(
"error while awaiting resource {name} is created: {err}"
)))?,
WatchEvent::Bookmark(_) => {
// bookmark events are periodically sent as keep-alive/checkpoint, we should continue waiting
}
any_other_event => panic!("Unexpected event happened while creating '{name}' {THIS_IS_A_BUG}. Event: {any_other_event:?}"),
}
}
Ok(())
}
}
mod helpers {
use k8s_openapi::api::core::v1::Pod;
use kube::runtime::wait::Condition;
use tracing::trace;
/// An await condition for `Pod` that returns `true` once it is ready
/// based on [`kube::runtime::wait::conditions::is_pod_running`]
pub fn is_pod_ready() -> impl Condition<Pod> {
|obj: Option<&Pod>| {
if let Some(pod) = &obj {
if let Some(status) = &pod.status {
if let Some(conditions) = &status.conditions {
let ready = conditions
.iter()
.any(|cond| cond.status == "True" && cond.type_ == "Ready");
if ready {
trace!("{:#?}", status);
return ready;
}
}
}
}
false
}
}
}
vendor/pezkuwi-zombienet-sdk/crates/provider/src/kubernetes/namespace.rs
Vendored
+600
View File
@@ -0,0 +1,600 @@
use std::{
collections::{BTreeMap, HashMap},
env,
path::{Path, PathBuf},
sync::{Arc, Weak},
};
use async_trait::async_trait;
use k8s_openapi::{
api::core::v1::{
Container, ContainerPort, HTTPGetAction, PodSpec, Probe, ServicePort, ServiceSpec,
},
apimachinery::pkg::util::intstr::IntOrString,
};
use support::{constants::THIS_IS_A_BUG, fs::FileSystem, replacer::apply_replacements};
use tokio::sync::{Mutex, RwLock};
use tracing::{debug, trace, warn};
use uuid::Uuid;
use super::{client::KubernetesClient, node::KubernetesNode};
use crate::{
constants::NAMESPACE_PREFIX,
kubernetes::{
node::{DeserializableKubernetesNodeOptions, KubernetesNodeOptions},
provider,
},
shared::helpers::{extract_execution_result, running_in_ci},
types::{
GenerateFileCommand, GenerateFilesOptions, ProviderCapabilities, RunCommandOptions,
SpawnNodeOptions,
},
DynNode, KubernetesProvider, ProviderError, ProviderNamespace, ProviderNode,
};
const FILE_SERVER_IMAGE: &str = "europe-west3-docker.pkg.dev/parity-zombienet/zombienet-public-images/zombienet-file-server:latest";
// env var used by our internal CI to pass the namespace created and ready to use
const ZOMBIE_K8S_CI_NAMESPACE: &str = "ZOMBIE_K8S_CI_NAMESPACE";
pub(super) struct KubernetesNamespace<FS>
where
FS: FileSystem + Send + Sync + Clone,
{
weak: Weak<KubernetesNamespace<FS>>,
provider: Weak<KubernetesProvider<FS>>,
name: String,
base_dir: PathBuf,
capabilities: ProviderCapabilities,
k8s_client: KubernetesClient,
filesystem: FS,
file_server_fw_task: RwLock<Option<tokio::task::JoinHandle<()>>>,
delete_on_drop: Arc<Mutex<bool>>,
pub(super) file_server_port: RwLock<Option<u16>>,
pub(super) nodes: RwLock<HashMap<String, Arc<KubernetesNode<FS>>>>,
}
impl<FS> KubernetesNamespace<FS>
where
FS: FileSystem + Send + Sync + Clone + 'static,
{
pub(super) async fn new(
provider: &Weak<KubernetesProvider<FS>>,
tmp_dir: &PathBuf,
capabilities: &ProviderCapabilities,
k8s_client: &KubernetesClient,
filesystem: &FS,
custom_base_dir: Option<&Path>,
) -> Result<Arc<Self>, ProviderError> {
// If the namespace is already provided
let name = if let Ok(name) = env::var(ZOMBIE_K8S_CI_NAMESPACE) {
name
} else {
format!("{}{}", NAMESPACE_PREFIX, Uuid::new_v4())
};
let base_dir = if let Some(custom_base_dir) = custom_base_dir {
if !filesystem.exists(custom_base_dir).await {
filesystem.create_dir(custom_base_dir).await?;
} else {
warn!(
"⚠️ Using and existing directory {} as base dir",
custom_base_dir.to_string_lossy()
);
}
PathBuf::from(custom_base_dir)
} else {
let base_dir = PathBuf::from_iter([tmp_dir, &PathBuf::from(&name)]);
filesystem.create_dir(&base_dir).await?;
base_dir
};
let namespace = Arc::new_cyclic(|weak| KubernetesNamespace {
weak: weak.clone(),
provider: provider.clone(),
name,
base_dir,
capabilities: capabilities.clone(),
filesystem: filesystem.clone(),
k8s_client: k8s_client.clone(),
file_server_port: RwLock::new(None),
file_server_fw_task: RwLock::new(None),
nodes: RwLock::new(HashMap::new()),
delete_on_drop: Arc::new(Mutex::new(true)),
});
namespace.initialize().await?;
Ok(namespace)
}
pub(super) async fn attach_to_live(
provider: &Weak<KubernetesProvider<FS>>,
capabilities: &ProviderCapabilities,
k8s_client: &KubernetesClient,
filesystem: &FS,
custom_base_dir: &Path,
name: &str,
) -> Result<Arc<Self>, ProviderError> {
let base_dir = custom_base_dir.to_path_buf();
let namespace = Arc::new_cyclic(|weak| KubernetesNamespace {
weak: weak.clone(),
provider: provider.clone(),
name: name.to_owned(),
base_dir,
capabilities: capabilities.clone(),
filesystem: filesystem.clone(),
k8s_client: k8s_client.clone(),
file_server_port: RwLock::new(None),
file_server_fw_task: RwLock::new(None),
nodes: RwLock::new(HashMap::new()),
delete_on_drop: Arc::new(Mutex::new(false)),
});
namespace.setup_file_server_port_fwd("fileserver").await?;
Ok(namespace)
}
async fn initialize(&self) -> Result<(), ProviderError> {
// Initialize the namespace IFF
// we are not in CI or we don't have the env `ZOMBIE_NAMESPACE` set
if env::var(ZOMBIE_K8S_CI_NAMESPACE).is_err() || !running_in_ci() {
self.initialize_k8s().await?;
}
// Ensure namespace isolation and minimal resources IFF we are running in CI
if running_in_ci() {
self.initialize_static_resources().await?
}
self.initialize_file_server().await?;
self.setup_script_config_map(
"zombie-wrapper",
include_str!("../shared/scripts/zombie-wrapper.sh"),
"zombie_wrapper_config_map_manifest.yaml",
// TODO: add correct labels
BTreeMap::new(),
)
.await?;
self.setup_script_config_map(
"helper-binaries-downloader",
include_str!("../shared/scripts/helper-binaries-downloader.sh"),
"helper_binaries_downloader_config_map_manifest.yaml",
// TODO: add correct labels
BTreeMap::new(),
)
.await?;
Ok(())
}
async fn initialize_k8s(&self) -> Result<(), ProviderError> {
// TODO (javier): check with Hamid if we are using this labels in any scheduling logic.
let labels = BTreeMap::from([
(
"jobId".to_string(),
env::var("CI_JOB_ID").unwrap_or("".to_string()),
),
(
"projectName".to_string(),
env::var("CI_PROJECT_NAME").unwrap_or("".to_string()),
),
(
"projectId".to_string(),
env::var("CI_PROJECT_ID").unwrap_or("".to_string()),
),
]);
let manifest = self
.k8s_client
.create_namespace(&self.name, labels)
.await
.map_err(|err| {
ProviderError::CreateNamespaceFailed(self.name.to_string(), err.into())
})?;
let serialized_manifest = serde_yaml::to_string(&manifest).map_err(|err| {
ProviderError::CreateNamespaceFailed(self.name.to_string(), err.into())
})?;
let dest_path =
PathBuf::from_iter([&self.base_dir, &PathBuf::from("namespace_manifest.yaml")]);
self.filesystem
.write(dest_path, serialized_manifest)
.await?;
Ok(())
}
async fn initialize_static_resources(&self) -> Result<(), ProviderError> {
let np_manifest = apply_replacements(
include_str!("./static-configs/namespace-network-policy.yaml"),
&HashMap::from([("namespace", self.name())]),
);
// Apply NetworkPolicy manifest
self.k8s_client
.create_static_resource(&self.name, &np_manifest)
.await
.map_err(|err| {
ProviderError::CreateNamespaceFailed(self.name.to_string(), err.into())
})?;
// Apply LimitRange manifest
self.k8s_client
.create_static_resource(
&self.name,
include_str!("./static-configs/baseline-resources.yaml"),
)
.await
.map_err(|err| {
ProviderError::CreateNamespaceFailed(self.name.to_string(), err.into())
})?;
Ok(())
}
async fn initialize_file_server(&self) -> Result<(), ProviderError> {
let name = "fileserver".to_string();
let labels = BTreeMap::from([
("app.kubernetes.io/name".to_string(), name.clone()),
(
"x-infra-instance".to_string(),
env::var("X_INFRA_INSTANCE").unwrap_or("ondemand".to_string()),
),
]);
let pod_spec = PodSpec {
hostname: Some(name.clone()),
containers: vec![Container {
name: name.clone(),
image: Some(FILE_SERVER_IMAGE.to_string()),
image_pull_policy: Some("Always".to_string()),
ports: Some(vec![ContainerPort {
container_port: 80,
..Default::default()
}]),
startup_probe: Some(Probe {
http_get: Some(HTTPGetAction {
path: Some("/".to_string()),
port: IntOrString::Int(80),
..Default::default()
}),
initial_delay_seconds: Some(1),
period_seconds: Some(2),
failure_threshold: Some(3),
..Default::default()
}),
..Default::default()
}],
restart_policy: Some("OnFailure".into()),
..Default::default()
};
let pod_manifest = self
.k8s_client
.create_pod(&self.name, &name, pod_spec, labels.clone())
.await
.map_err(|err| ProviderError::FileServerSetupError(err.into()))?;
// TODO: remove duplication across methods
let pod_serialized_manifest = serde_yaml::to_string(&pod_manifest)
.map_err(|err| ProviderError::FileServerSetupError(err.into()))?;
let pod_dest_path = PathBuf::from_iter([
&self.base_dir,
&PathBuf::from("file_server_pod_manifest.yaml"),
]);
self.filesystem
.write(pod_dest_path, pod_serialized_manifest)
.await?;
let service_spec = ServiceSpec {
selector: Some(labels.clone()),
ports: Some(vec![ServicePort {
port: 80,
..Default::default()
}]),
..Default::default()
};
let service_manifest = self
.k8s_client
.create_service(&self.name, &name, service_spec, labels)
.await
.map_err(|err| ProviderError::FileServerSetupError(err.into()))?;
let serialized_service_manifest = serde_yaml::to_string(&service_manifest)
.map_err(|err| ProviderError::FileServerSetupError(err.into()))?;
let service_dest_path = PathBuf::from_iter([
&self.base_dir,
&PathBuf::from("file_server_service_manifest.yaml"),
]);
self.filesystem
.write(service_dest_path, serialized_service_manifest)
.await?;
self.setup_file_server_port_fwd(&name).await?;
Ok(())
}
async fn setup_file_server_port_fwd(&self, name: &str) -> Result<(), ProviderError> {
let (port, task) = self
.k8s_client
.create_pod_port_forward(&self.name, name, 0, 80)
.await
.map_err(|err| ProviderError::FileServerSetupError(err.into()))?;
*self.file_server_port.write().await = Some(port);
*self.file_server_fw_task.write().await = Some(task);
Ok(())
}
async fn setup_script_config_map(
&self,
name: &str,
script_contents: &str,
local_manifest_name: &str,
labels: BTreeMap<String, String>,
) -> Result<(), ProviderError> {
let manifest = self
.k8s_client
.create_config_map_from_file(
&self.name,
name,
&format!("{name}.sh"),
script_contents,
labels,
)
.await
.map_err(|err| {
ProviderError::CreateNamespaceFailed(self.name.to_string(), err.into())
})?;
let serializer_manifest = serde_yaml::to_string(&manifest).map_err(|err| {
ProviderError::CreateNamespaceFailed(self.name.to_string(), err.into())
})?;
let dest_path = PathBuf::from_iter([&self.base_dir, &PathBuf::from(local_manifest_name)]);
self.filesystem
.write(dest_path, serializer_manifest)
.await?;
Ok(())
}
pub async fn set_delete_on_drop(&self, delete_on_drop: bool) {
*self.delete_on_drop.lock().await = delete_on_drop;
}
pub async fn delete_on_drop(&self) -> bool {
if let Ok(delete_on_drop) = self.delete_on_drop.try_lock() {
*delete_on_drop
} else {
// if we can't lock just remove the ns
true
}
}
}
impl<FS> Drop for KubernetesNamespace<FS>
where
FS: FileSystem + Send + Sync + Clone,
{
fn drop(&mut self) {
let ns_name = self.name.clone();
if let Ok(delete_on_drop) = self.delete_on_drop.try_lock() {
if *delete_on_drop {
let client = self.k8s_client.clone();
let provider = self.provider.upgrade();
futures::executor::block_on(async move {
trace!("🧟 deleting ns {ns_name} from cluster");
let _ = client.delete_namespace(&ns_name).await;
if let Some(provider) = provider {
provider.namespaces.write().await.remove(&ns_name);
}
trace!("✅ deleted");
});
} else {
trace!("⚠️ leaking ns {ns_name} in cluster");
}
};
}
}
#[async_trait]
impl<FS> ProviderNamespace for KubernetesNamespace<FS>
where
FS: FileSystem + Send + Sync + Clone + 'static,
{
fn name(&self) -> &str {
&self.name
}
fn base_dir(&self) -> &PathBuf {
&self.base_dir
}
fn capabilities(&self) -> &ProviderCapabilities {
&self.capabilities
}
fn provider_name(&self) -> &str {
provider::PROVIDER_NAME
}
async fn detach(&self) {
self.set_delete_on_drop(false).await;
}
async fn is_detached(&self) -> bool {
self.delete_on_drop().await
}
async fn nodes(&self) -> HashMap<String, DynNode> {
self.nodes
.read()
.await
.iter()
.map(|(name, node)| (name.clone(), node.clone() as DynNode))
.collect()
}
async fn get_node_available_args(
&self,
(command, image): (String, Option<String>),
) -> Result<String, ProviderError> {
let node_image = image.expect(&format!("image should be present when getting node available args with kubernetes provider {THIS_IS_A_BUG}"));
// run dummy command in new pod
let temp_node = self
.spawn_node(
&SpawnNodeOptions::new(format!("temp-{}", Uuid::new_v4()), "cat".to_string())
.image(node_image.clone()),
)
.await?;
let available_args_output = temp_node
.run_command(RunCommandOptions::new(command.clone()).args(vec!["--help"]))
.await?
.map_err(|(_exit, status)| {
ProviderError::NodeAvailableArgsError(node_image, command, status)
})?;
temp_node.destroy().await?;
Ok(available_args_output)
}
async fn spawn_node(&self, options: &SpawnNodeOptions) -> Result<DynNode, ProviderError> {
trace!("spawn node options {options:?}");
let node = KubernetesNode::new(KubernetesNodeOptions {
namespace: &self.weak,
namespace_base_dir: &self.base_dir,
name: &options.name,
image: options.image.as_ref(),
program: &options.program,
args: &options.args,
env: &options.env,
startup_files: &options.injected_files,
resources: options.resources.as_ref(),
db_snapshot: options.db_snapshot.as_ref(),
k8s_client: &self.k8s_client,
filesystem: &self.filesystem,
})
.await?;
self.nodes
.write()
.await
.insert(node.name().to_string(), node.clone());
Ok(node)
}
async fn spawn_node_from_json(
&self,
json_value: &serde_json::Value,
) -> Result<DynNode, ProviderError> {
let deserializable: DeserializableKubernetesNodeOptions =
serde_json::from_value(json_value.clone())?;
let options = KubernetesNodeOptions::from_deserializable(
&deserializable,
&self.weak,
&self.base_dir,
&self.k8s_client,
&self.filesystem,
);
let node = KubernetesNode::attach_to_live(options).await?;
self.nodes
.write()
.await
.insert(node.name().to_string(), node.clone());
Ok(node)
}
async fn generate_files(&self, options: GenerateFilesOptions) -> Result<(), ProviderError> {
debug!("generate files options {options:#?}");
let node_name = options
.temp_name
.unwrap_or_else(|| format!("temp-{}", Uuid::new_v4()));
let node_image = options
.image
.expect(&format!("image should be present when generating files with kubernetes provider {THIS_IS_A_BUG}"));
// run dummy command in new pod
let temp_node = self
.spawn_node(
&SpawnNodeOptions::new(node_name, "cat".to_string())
.injected_files(options.injected_files)
.image(node_image),
)
.await?;
for GenerateFileCommand {
program,
args,
env,
local_output_path,
} in options.commands
{
let local_output_full_path = format!(
"{}{}{}",
self.base_dir.to_string_lossy(),
if local_output_path.starts_with("/") {
""
} else {
"/"
},
local_output_path.to_string_lossy()
);
let contents = extract_execution_result(
&temp_node,
RunCommandOptions { program, args, env },
options.expected_path.as_ref(),
)
.await?;
self.filesystem
.write(local_output_full_path, contents)
.await
.map_err(|err| ProviderError::FileGenerationFailed(err.into()))?;
}
temp_node.destroy().await
}
async fn static_setup(&self) -> Result<(), ProviderError> {
todo!()
}
async fn destroy(&self) -> Result<(), ProviderError> {
let _ = self
.k8s_client
.delete_namespace(&self.name)
.await
.map_err(|err| ProviderError::DeleteNamespaceFailed(self.name.clone(), err.into()))?;
if let Some(provider) = self.provider.upgrade() {
provider.namespaces.write().await.remove(&self.name);
}
Ok(())
}
}
vendor/pezkuwi-zombienet-sdk/crates/provider/src/kubernetes/node.rs
Vendored
+886
View File
@@ -0,0 +1,886 @@
use std::{
collections::{BTreeMap, HashMap},
env,
net::IpAddr,
path::{Component, Path, PathBuf},
sync::{Arc, Weak},
time::Duration,
};
use anyhow::anyhow;
use async_trait::async_trait;
use configuration::{shared::resources::Resources, types::AssetLocation};
use futures::future::try_join_all;
use k8s_openapi::api::core::v1::{ServicePort, ServiceSpec};
use serde::{Deserialize, Serialize};
use sha2::Digest;
use support::{constants::THIS_IS_A_BUG, fs::FileSystem};
use tokio::{sync::RwLock, task::JoinHandle, time::sleep, try_join};
use tracing::{debug, trace, warn};
use url::Url;
use super::{
client::KubernetesClient, namespace::KubernetesNamespace, pod_spec_builder::PodSpecBuilder,
};
use crate::{
constants::{
NODE_CONFIG_DIR, NODE_DATA_DIR, NODE_RELAY_DATA_DIR, NODE_SCRIPTS_DIR, P2P_PORT,
PROMETHEUS_PORT, RPC_HTTP_PORT, RPC_WS_PORT,
},
kubernetes,
types::{ExecutionResult, RunCommandOptions, RunScriptOptions, TransferedFile},
ProviderError, ProviderNamespace, ProviderNode,
};
pub(super) struct KubernetesNodeOptions<'a, FS>
where
FS: FileSystem + Send + Sync + Clone + 'static,
{
pub(super) namespace: &'a Weak<KubernetesNamespace<FS>>,
pub(super) namespace_base_dir: &'a PathBuf,
pub(super) name: &'a str,
pub(super) image: Option<&'a String>,
pub(super) program: &'a str,
pub(super) args: &'a [String],
pub(super) env: &'a [(String, String)],
pub(super) startup_files: &'a [TransferedFile],
pub(super) resources: Option<&'a Resources>,
pub(super) db_snapshot: Option<&'a AssetLocation>,
pub(super) k8s_client: &'a KubernetesClient,
pub(super) filesystem: &'a FS,
}
impl<'a, FS> KubernetesNodeOptions<'a, FS>
where
FS: FileSystem + Send + Sync + Clone + 'static,
{
pub(super) fn from_deserializable(
deserializable: &'a DeserializableKubernetesNodeOptions,
namespace: &'a Weak<KubernetesNamespace<FS>>,
namespace_base_dir: &'a PathBuf,
k8s_client: &'a KubernetesClient,
filesystem: &'a FS,
) -> KubernetesNodeOptions<'a, FS> {
KubernetesNodeOptions {
namespace,
namespace_base_dir,
name: &deserializable.name,
image: deserializable.image.as_ref(),
program: &deserializable.program,
args: &deserializable.args,
env: &deserializable.env,
startup_files: &[],
resources: deserializable.resources.as_ref(),
db_snapshot: None,
k8s_client,
filesystem,
}
}
}
#[derive(Deserialize)]
pub(super) struct DeserializableKubernetesNodeOptions {
pub(super) name: String,
pub(super) image: Option<String>,
pub(super) program: String,
pub(super) args: Vec<String>,
pub(super) env: Vec<(String, String)>,
pub(super) resources: Option<Resources>,
}
type FwdInfo = (u16, JoinHandle<()>);
#[derive(Serialize)]
pub(super) struct KubernetesNode<FS>
where
FS: FileSystem + Send + Sync + Clone,
{
#[serde(skip)]
namespace: Weak<KubernetesNamespace<FS>>,
name: String,
image: String,
program: String,
args: Vec<String>,
env: Vec<(String, String)>,
resources: Option<Resources>,
base_dir: PathBuf,
config_dir: PathBuf,
data_dir: PathBuf,
relay_data_dir: PathBuf,
scripts_dir: PathBuf,
log_path: PathBuf,
#[serde(skip)]
k8s_client: KubernetesClient,
#[serde(skip)]
http_client: reqwest::Client,
#[serde(skip)]
filesystem: FS,
#[serde(skip)]
port_fwds: RwLock<HashMap<u16, FwdInfo>>,
provider_tag: String,
}
impl<FS> KubernetesNode<FS>
where
FS: FileSystem + Send + Sync + Clone + 'static,
{
pub(super) async fn new(
options: KubernetesNodeOptions<'_, FS>,
) -> Result<Arc<Self>, ProviderError> {
let image = options.image.ok_or_else(|| {
ProviderError::MissingNodeInfo(options.name.to_string(), "missing image".to_string())
})?;
let filesystem = options.filesystem.clone();
let base_dir =
PathBuf::from_iter([options.namespace_base_dir, &PathBuf::from(options.name)]);
filesystem.create_dir_all(&base_dir).await?;
let base_dir_raw = base_dir.to_string_lossy();
let config_dir = PathBuf::from(format!("{base_dir_raw}{NODE_CONFIG_DIR}"));
let data_dir = PathBuf::from(format!("{base_dir_raw}{NODE_DATA_DIR}"));
let relay_data_dir = PathBuf::from(format!("{base_dir_raw}{NODE_RELAY_DATA_DIR}"));
let scripts_dir = PathBuf::from(format!("{base_dir_raw}{NODE_SCRIPTS_DIR}"));
let log_path = base_dir.join("node.log");
try_join!(
filesystem.create_dir(&config_dir),
filesystem.create_dir(&data_dir),
filesystem.create_dir(&relay_data_dir),
filesystem.create_dir(&scripts_dir),
)?;
let node = Arc::new(KubernetesNode {
namespace: options.namespace.clone(),
name: options.name.to_string(),
image: image.to_string(),
program: options.program.to_string(),
args: options.args.to_vec(),
env: options.env.to_vec(),
resources: options.resources.cloned(),
base_dir,
config_dir,
data_dir,
relay_data_dir,
scripts_dir,
log_path,
filesystem: filesystem.clone(),
k8s_client: options.k8s_client.clone(),
http_client: reqwest::Client::new(),
port_fwds: Default::default(),
provider_tag: kubernetes::provider::PROVIDER_NAME.to_string(),
});
node.initialize_k8s().await?;
if let Some(db_snap) = options.db_snapshot {
node.initialize_db_snapshot(db_snap).await?;
}
node.initialize_startup_files(options.startup_files).await?;
node.start().await?;
Ok(node)
}
pub(super) async fn attach_to_live(
options: KubernetesNodeOptions<'_, FS>,
) -> Result<Arc<Self>, ProviderError> {
let image = options.image.ok_or_else(|| {
ProviderError::MissingNodeInfo(options.name.to_string(), "missing image".to_string())
})?;
let filesystem = options.filesystem.clone();
let base_dir =
PathBuf::from_iter([options.namespace_base_dir, &PathBuf::from(options.name)]);
filesystem.create_dir_all(&base_dir).await?;
let base_dir_raw = base_dir.to_string_lossy();
let config_dir = PathBuf::from(format!("{base_dir_raw}{NODE_CONFIG_DIR}"));
let data_dir = PathBuf::from(format!("{base_dir_raw}{NODE_DATA_DIR}"));
let relay_data_dir = PathBuf::from(format!("{base_dir_raw}{NODE_RELAY_DATA_DIR}"));
let scripts_dir = PathBuf::from(format!("{base_dir_raw}{NODE_SCRIPTS_DIR}"));
let log_path = base_dir.join("node.log");
let node = Arc::new(KubernetesNode {
namespace: options.namespace.clone(),
name: options.name.to_string(),
image: image.to_string(),
program: options.program.to_string(),
args: options.args.to_vec(),
env: options.env.to_vec(),
resources: options.resources.cloned(),
base_dir,
config_dir,
data_dir,
relay_data_dir,
scripts_dir,
log_path,
filesystem: filesystem.clone(),
k8s_client: options.k8s_client.clone(),
http_client: reqwest::Client::new(),
port_fwds: Default::default(),
provider_tag: kubernetes::provider::PROVIDER_NAME.to_string(),
});
Ok(node)
}
async fn initialize_k8s(&self) -> Result<(), ProviderError> {
let labels = BTreeMap::from([
(
"app.kubernetes.io/name".to_string(),
self.name().to_string(),
),
(
"x-infra-instance".to_string(),
env::var("X_INFRA_INSTANCE").unwrap_or("ondemand".to_string()),
),
]);
// Create pod
let pod_spec = PodSpecBuilder::build(
&self.name,
&self.image,
self.resources.as_ref(),
&self.program,
&self.args,
&self.env,
);
let manifest = self
.k8s_client
.create_pod(&self.namespace_name(), &self.name, pod_spec, labels.clone())
.await
.map_err(|err| ProviderError::NodeSpawningFailed(self.name.clone(), err.into()))?;
let serialized_manifest = serde_yaml::to_string(&manifest)
.map_err(|err| ProviderError::NodeSpawningFailed(self.name.to_string(), err.into()))?;
let dest_path = PathBuf::from_iter([
&self.base_dir,
&PathBuf::from(format!("{}_manifest.yaml", &self.name)),
]);
self.filesystem
.write(dest_path, serialized_manifest)
.await
.map_err(|err| ProviderError::NodeSpawningFailed(self.name.to_string(), err.into()))?;
// Create service for pod
let service_spec = ServiceSpec {
selector: Some(labels.clone()),
ports: Some(vec![
ServicePort {
port: P2P_PORT.into(),
name: Some("p2p".into()),
..Default::default()
},
ServicePort {
port: RPC_WS_PORT.into(),
name: Some("rpc".into()),
..Default::default()
},
ServicePort {
port: RPC_HTTP_PORT.into(),
name: Some("rpc-http".into()),
..Default::default()
},
ServicePort {
port: PROMETHEUS_PORT.into(),
name: Some("prom".into()),
..Default::default()
},
]),
..Default::default()
};
let service_manifest = self
.k8s_client
.create_service(&self.namespace_name(), &self.name, service_spec, labels)
.await
.map_err(|err| ProviderError::FileServerSetupError(err.into()))?;
let serialized_service_manifest = serde_yaml::to_string(&service_manifest)
.map_err(|err| ProviderError::FileServerSetupError(err.into()))?;
let service_dest_path = PathBuf::from_iter([
&self.base_dir,
&PathBuf::from(format!("{}_svc_manifest.yaml", &self.name)),
]);
self.filesystem
.write(service_dest_path, serialized_service_manifest)
.await?;
Ok(())
}
async fn initialize_db_snapshot(
&self,
db_snapshot: &AssetLocation,
) -> Result<(), ProviderError> {
trace!("snap: {db_snapshot}");
let url_of_snap = match db_snapshot {
AssetLocation::Url(location) => location.clone(),
AssetLocation::FilePath(filepath) => {
let (url, _) = self.upload_to_fileserver(filepath).await?;
url
},
};
// we need to get the snapshot from a public access
// and extract to /data
let opts = RunCommandOptions::new("mkdir").args([
"-p",
"/data/",
"&&",
"mkdir",
"-p",
"/relay-data/",
"&&",
// Use our version of curl
"/cfg/curl",
url_of_snap.as_ref(),
"--output",
"/data/db.tgz",
"&&",
"cd",
"/",
"&&",
"tar",
"--skip-old-files",
"-xzvf",
"/data/db.tgz",
]);
trace!("cmd opts: {:#?}", opts);
let _ = self.run_command(opts).await?;
Ok(())
}
async fn initialize_startup_files(
&self,
startup_files: &[TransferedFile],
) -> Result<(), ProviderError> {
try_join_all(
startup_files
.iter()
.map(|file| self.send_file(&file.local_path, &file.remote_path, &file.mode)),
)
.await?;
Ok(())
}
pub(super) async fn start(&self) -> Result<(), ProviderError> {
self.k8s_client
.pod_exec(
&self.namespace_name(),
&self.name,
vec!["sh", "-c", "echo start > /tmp/zombiepipe"],
)
.await
.map_err(|err| {
ProviderError::NodeSpawningFailed(
format!("failed to start pod {} after spawning", self.name),
err.into(),
)
})?
.map_err(|err| {
ProviderError::NodeSpawningFailed(
format!("failed to start pod {} after spawning", self.name,),
anyhow!("command failed in container: status {}: {}", err.0, err.1),
)
})?;
Ok(())
}
fn get_remote_parent_dir(&self, remote_file_path: &Path) -> Option<PathBuf> {
if let Some(remote_parent_dir) = remote_file_path.parent() {
if matches!(
remote_parent_dir.components().rev().peekable().peek(),
Some(Component::Normal(_))
) {
return Some(remote_parent_dir.to_path_buf());
}
}
None
}
async fn create_remote_dir(&self, remote_dir: &Path) -> Result<(), ProviderError> {
let _ = self
.k8s_client
.pod_exec(
&self.namespace_name(),
&self.name,
vec!["mkdir", "-p", &remote_dir.to_string_lossy()],
)
.await
.map_err(|err| {
ProviderError::NodeSpawningFailed(
format!(
"failed to create dir {} for pod {}",
remote_dir.to_string_lossy(),
&self.name
),
err.into(),
)
})?;
Ok(())
}
fn namespace_name(&self) -> String {
self.namespace
.upgrade()
.map(|namespace| namespace.name().to_string())
.unwrap_or_else(|| panic!("namespace shouldn't be dropped, {THIS_IS_A_BUG}"))
}
async fn upload_to_fileserver(&self, location: &Path) -> Result<(Url, String), ProviderError> {
let file_name = if let Some(name) = location.file_name() {
name.to_string_lossy()
} else {
"unnamed".into()
};
let data = self.filesystem.read(location).await?;
let content_hashed = hex::encode(sha2::Sha256::digest(&data));
let req = self
.http_client
.head(format!(
"http://{}/{content_hashed}__{file_name}",
self.file_server_local_host().await?
))
.build()
.map_err(|err| {
ProviderError::UploadFile(location.to_string_lossy().to_string(), err.into())
})?;
let url = req.url().clone();
let res = self.http_client.execute(req).await.map_err(|err| {
ProviderError::UploadFile(location.to_string_lossy().to_string(), err.into())
})?;
if res.status() != reqwest::StatusCode::OK {
// we need to upload the file
self.http_client
.post(url.as_ref())
.body(data)
.send()
.await
.map_err(|err| {
ProviderError::UploadFile(location.to_string_lossy().to_string(), err.into())
})?;
}
Ok((url, content_hashed))
}
async fn file_server_local_host(&self) -> Result<String, ProviderError> {
if let Some(namespace) = self.namespace.upgrade() {
if let Some(port) = *namespace.file_server_port.read().await {
return Ok(format!("localhost:{port}"));
}
}
Err(ProviderError::FileServerSetupError(anyhow!(
"file server port not bound locally"
)))
}
async fn download_file(
&self,
url: &str,
remote_file_path: &Path,
hash: Option<&str>,
) -> Result<(), ProviderError> {
let r = self
.k8s_client
.pod_exec(
&self.namespace_name(),
&self.name,
vec![
"/cfg/curl",
url,
"--output",
&remote_file_path.to_string_lossy(),
],
)
.await
.map_err(|err| {
ProviderError::DownloadFile(
remote_file_path.to_string_lossy().to_string(),
anyhow!(format!("node: {}, err: {}", self.name(), err)),
)
})?;
trace!("download url {} result: {:?}", url, r);
if r.is_err() {
return Err(ProviderError::DownloadFile(
remote_file_path.to_string_lossy().to_string(),
anyhow!(format!("node: {}, err downloading file", self.name())),
));
}
if let Some(hash) = hash {
// check if the hash of the file is correct
let res = self
.k8s_client
.pod_exec(
&self.namespace_name(),
&self.name,
vec![
"/cfg/coreutils",
"sha256sum",
&remote_file_path.to_string_lossy(),
],
)
.await
.map_err(|err| {
ProviderError::DownloadFile(
remote_file_path.to_string_lossy().to_string(),
anyhow!(format!("node: {}, err: {}", self.name(), err)),
)
})?;
if let Ok(output) = res {
if !output.contains(hash) {
return Err(ProviderError::DownloadFile(
remote_file_path.to_string_lossy().to_string(),
anyhow!(format!("node: {}, invalid sha256sum hash: {hash} for file, output was {output}", self.name())),
));
}
} else {
return Err(ProviderError::DownloadFile(
remote_file_path.to_string_lossy().to_string(),
anyhow!(format!(
"node: {}, err calculating sha256sum for file {:?}",
self.name(),
res
)),
));
}
}
Ok(())
}
}
#[async_trait]
impl<FS> ProviderNode for KubernetesNode<FS>
where
FS: FileSystem + Send + Sync + Clone + 'static,
{
fn name(&self) -> &str {
&self.name
}
fn args(&self) -> Vec<&str> {
self.args.iter().map(|arg| arg.as_str()).collect()
}
fn base_dir(&self) -> &PathBuf {
&self.base_dir
}
fn config_dir(&self) -> &PathBuf {
&self.config_dir
}
fn data_dir(&self) -> &PathBuf {
&self.data_dir
}
fn relay_data_dir(&self) -> &PathBuf {
&self.relay_data_dir
}
fn scripts_dir(&self) -> &PathBuf {
&self.scripts_dir
}
fn log_path(&self) -> &PathBuf {
&self.log_path
}
fn log_cmd(&self) -> String {
format!("kubectl -n {} logs {}", self.namespace_name(), self.name)
}
fn path_in_node(&self, file: &Path) -> PathBuf {
// here is just a noop op since we will receive the path
// for the file inside the pod
PathBuf::from(file)
}
// TODO: handle log rotation as we do in v1
async fn logs(&self) -> Result<String, ProviderError> {
self.k8s_client
.pod_logs(&self.namespace_name(), &self.name)
.await
.map_err(|err| ProviderError::GetLogsFailed(self.name.to_string(), err.into()))
}
async fn dump_logs(&self, local_dest: PathBuf) -> Result<(), ProviderError> {
let logs = self.logs().await?;
self.filesystem
.write(local_dest, logs)
.await
.map_err(|err| ProviderError::DumpLogsFailed(self.name.to_string(), err.into()))?;
Ok(())
}
async fn create_port_forward(
&self,
local_port: u16,
remote_port: u16,
) -> Result<Option<u16>, ProviderError> {
// If the fwd exist just return the local port
if let Some(fwd_info) = self.port_fwds.read().await.get(&remote_port) {
return Ok(Some(fwd_info.0));
};
let (port, task) = self
.k8s_client
.create_pod_port_forward(&self.namespace_name(), &self.name, local_port, remote_port)
.await
.map_err(|err| ProviderError::PortForwardError(local_port, remote_port, err.into()))?;
self.port_fwds
.write()
.await
.insert(remote_port, (port, task));
Ok(Some(port))
}
async fn run_command(
&self,
options: RunCommandOptions,
) -> Result<ExecutionResult, ProviderError> {
let mut command = vec![];
for (name, value) in options.env {
command.push(format!("export {name}={value};"));
}
command.push(options.program);
for arg in options.args {
command.push(arg);
}
self.k8s_client
.pod_exec(
&self.namespace_name(),
&self.name,
vec!["sh", "-c", &command.join(" ")],
)
.await
.map_err(|err| {
ProviderError::RunCommandError(
format!("sh -c {}", &command.join(" ")),
format!("in pod {}", self.name),
err.into(),
)
})
}
async fn run_script(
&self,
options: RunScriptOptions,
) -> Result<ExecutionResult, ProviderError> {
let file_name = options
.local_script_path
.file_name()
.expect(&format!(
"file name should be present at this point {THIS_IS_A_BUG}"
))
.to_string_lossy();
self.run_command(RunCommandOptions {
program: format!("/tmp/{file_name}"),
args: options.args,
env: options.env,
})
.await
.map_err(|err| ProviderError::RunScriptError(self.name.to_string(), err.into()))
}
async fn send_file(
&self,
local_file_path: &Path,
remote_file_path: &Path,
mode: &str,
) -> Result<(), ProviderError> {
if let Some(remote_parent_dir) = self.get_remote_parent_dir(remote_file_path) {
self.create_remote_dir(&remote_parent_dir).await?;
}
debug!(
"Uploading file: {} IFF not present in the fileserver",
local_file_path.to_string_lossy()
);
// we need to override the url to use inside the pod
let (mut url, hash) = self.upload_to_fileserver(local_file_path).await?;
let _ = url.set_host(Some("fileserver"));
let _ = url.set_port(Some(80));
// Sometimes downloading the file fails (the file is corrupted)
// Add at most 5 retries
let mut last_err = None;
for i in 0..5 {
if i > 0 {
warn!("retrying number {i} download file {:?}", remote_file_path);
tokio::time::sleep(Duration::from_secs(i)).await;
}
let res = self
.download_file(url.as_ref(), remote_file_path, Some(&hash))
.await;
last_err = res.err();
if last_err.is_none() {
// ready to continue
break;
}
}
if let Some(last_err) = last_err {
return Err(last_err);
}
let _ = self
.k8s_client
.pod_exec(
&self.namespace_name(),
&self.name,
vec!["chmod", mode, &remote_file_path.to_string_lossy()],
)
.await
.map_err(|err| {
ProviderError::SendFile(
self.name.clone(),
local_file_path.to_string_lossy().to_string(),
err.into(),
)
})?;
Ok(())
}
async fn receive_file(
&self,
_remote_src: &Path,
_local_dest: &Path,
) -> Result<(), ProviderError> {
Ok(())
}
async fn ip(&self) -> Result<IpAddr, ProviderError> {
let status = self
.k8s_client
.pod_status(&self.namespace_name(), &self.name)
.await
.map_err(|_| ProviderError::MissingNode(self.name.clone()))?;
if let Some(ip) = status.pod_ip {
// Pod ip should be parseable
Ok(ip.parse::<IpAddr>().map_err(|err| {
ProviderError::InvalidConfig(format!("Can not parse the pod ip: {ip}, err: {err}"))
})?)
} else {
Err(ProviderError::InvalidConfig(format!(
"Can not find ip of pod: {}",
self.name()
)))
}
}
async fn pause(&self) -> Result<(), ProviderError> {
self.k8s_client
.pod_exec(
&self.namespace_name(),
&self.name,
vec!["sh", "-c", "echo pause > /tmp/zombiepipe"],
)
.await
.map_err(|err| ProviderError::PauseNodeFailed(self.name.to_string(), err.into()))?
.map_err(|err| {
ProviderError::PauseNodeFailed(
self.name.to_string(),
anyhow!("error when pausing node: status {}: {}", err.0, err.1),
)
})?;
Ok(())
}
async fn resume(&self) -> Result<(), ProviderError> {
self.k8s_client
.pod_exec(
&self.namespace_name(),
&self.name,
vec!["sh", "-c", "echo resume > /tmp/zombiepipe"],
)
.await
.map_err(|err| ProviderError::ResumeNodeFailed(self.name.to_string(), err.into()))?
.map_err(|err| {
ProviderError::ResumeNodeFailed(
self.name.to_string(),
anyhow!("error when pausing node: status {}: {}", err.0, err.1),
)
})?;
Ok(())
}
async fn restart(&self, after: Option<Duration>) -> Result<(), ProviderError> {
if let Some(duration) = after {
sleep(duration).await;
}
self.k8s_client
.pod_exec(
&self.namespace_name(),
&self.name,
vec!["sh", "-c", "echo restart > /tmp/zombiepipe"],
)
.await
.map_err(|err| ProviderError::RestartNodeFailed(self.name.to_string(), err.into()))?
.map_err(|err| {
ProviderError::RestartNodeFailed(
self.name.to_string(),
anyhow!("error when restarting node: status {}: {}", err.0, err.1),
)
})?;
Ok(())
}
async fn destroy(&self) -> Result<(), ProviderError> {
self.k8s_client
.delete_pod(&self.namespace_name(), &self.name)
.await
.map_err(|err| ProviderError::KillNodeFailed(self.name.to_string(), err.into()))?;
if let Some(namespace) = self.namespace.upgrade() {
namespace.nodes.write().await.remove(&self.name);
}
Ok(())
}
}
vendor/pezkuwi-zombienet-sdk/crates/provider/src/kubernetes/pod_spec_builder.rs
Vendored
+188
View File
@@ -0,0 +1,188 @@
use std::collections::BTreeMap;
use configuration::shared::resources::{ResourceQuantity, Resources};
use k8s_openapi::{
api::core::v1::{
ConfigMapVolumeSource, Container, EnvVar, PodSpec, ResourceRequirements, Volume,
VolumeMount,
},
apimachinery::pkg::api::resource::Quantity,
};
pub(super) struct PodSpecBuilder;
impl PodSpecBuilder {
pub(super) fn build(
name: &str,
image: &str,
resources: Option<&Resources>,
program: &str,
args: &[String],
env: &[(String, String)],
) -> PodSpec {
PodSpec {
hostname: Some(name.to_string()),
init_containers: Some(vec![Self::build_helper_binaries_setup_container()]),
containers: vec![Self::build_main_container(
name, image, resources, program, args, env,
)],
volumes: Some(Self::build_volumes()),
..Default::default()
}
}
fn build_main_container(
name: &str,
image: &str,
resources: Option<&Resources>,
program: &str,
args: &[String],
env: &[(String, String)],
) -> Container {
Container {
name: name.to_string(),
image: Some(image.to_string()),
image_pull_policy: Some("Always".to_string()),
command: Some(
[
vec!["/zombie-wrapper.sh".to_string(), program.to_string()],
args.to_vec(),
]
.concat(),
),
env: Some(
env.iter()
.map(|(name, value)| EnvVar {
name: name.clone(),
value: Some(value.clone()),
value_from: None,
})
.collect(),
),
volume_mounts: Some(Self::build_volume_mounts(vec![VolumeMount {
name: "zombie-wrapper-volume".to_string(),
mount_path: "/zombie-wrapper.sh".to_string(),
sub_path: Some("zombie-wrapper.sh".to_string()),
..Default::default()
}])),
resources: Self::build_resources_requirements(resources),
..Default::default()
}
}
fn build_helper_binaries_setup_container() -> Container {
Container {
name: "helper-binaries-setup".to_string(),
image: Some("europe-west3-docker.pkg.dev/parity-zombienet/zombienet-public-images/alpine:latest".to_string()),
image_pull_policy: Some("IfNotPresent".to_string()),
volume_mounts: Some(Self::build_volume_mounts(vec![VolumeMount {
name: "helper-binaries-downloader-volume".to_string(),
mount_path: "/helper-binaries-downloader.sh".to_string(),
sub_path: Some("helper-binaries-downloader.sh".to_string()),
..Default::default()
}])),
command: Some(vec![
"ash".to_string(),
"/helper-binaries-downloader.sh".to_string(),
]),
..Default::default()
}
}
fn build_volumes() -> Vec<Volume> {
vec![
Volume {
name: "cfg".to_string(),
..Default::default()
},
Volume {
name: "data".to_string(),
..Default::default()
},
Volume {
name: "relay-data".to_string(),
..Default::default()
},
Volume {
name: "zombie-wrapper-volume".to_string(),
config_map: Some(ConfigMapVolumeSource {
name: Some("zombie-wrapper".to_string()),
default_mode: Some(0o755),
..Default::default()
}),
..Default::default()
},
Volume {
name: "helper-binaries-downloader-volume".to_string(),
config_map: Some(ConfigMapVolumeSource {
name: Some("helper-binaries-downloader".to_string()),
default_mode: Some(0o755),
..Default::default()
}),
..Default::default()
},
]
}
fn build_volume_mounts(non_default_mounts: Vec<VolumeMount>) -> Vec<VolumeMount> {
[
vec![
VolumeMount {
name: "cfg".to_string(),
mount_path: "/cfg".to_string(),
read_only: Some(false),
..Default::default()
},
VolumeMount {
name: "data".to_string(),
mount_path: "/data".to_string(),
read_only: Some(false),
..Default::default()
},
VolumeMount {
name: "relay-data".to_string(),
mount_path: "/relay-data".to_string(),
read_only: Some(false),
..Default::default()
},
],
non_default_mounts,
]
.concat()
}
fn build_resources_requirements(resources: Option<&Resources>) -> Option<ResourceRequirements> {
resources.map(|resources| ResourceRequirements {
limits: Self::build_resources_requirements_quantities(
resources.limit_cpu(),
resources.limit_memory(),
),
requests: Self::build_resources_requirements_quantities(
resources.request_cpu(),
resources.request_memory(),
),
..Default::default()
})
}
fn build_resources_requirements_quantities(
cpu: Option<&ResourceQuantity>,
memory: Option<&ResourceQuantity>,
) -> Option<BTreeMap<String, Quantity>> {
let mut quantities = BTreeMap::new();
if let Some(cpu) = cpu {
quantities.insert("cpu".to_string(), Quantity(cpu.as_str().to_string()));
}
if let Some(memory) = memory {
quantities.insert("memory".to_string(), Quantity(memory.as_str().to_string()));
}
if !quantities.is_empty() {
Some(quantities)
} else {
None
}
}
}

Some files were not shown because too many files have changed in this diff Show More