* Don't import backing statements directly into the dispute coordinator. This also gets rid of a redundant signature check. Both should have some impact on backing performance. In general this PR should make us scale better in the number of parachains. Reasoning (aka why this is fine): For the signature check: As mentioned, it is a redundant check. The signature has already been checked at this point. This is even made obvious by the used types. The smart constructor is not perfect as discussed [here](https://github.com/paritytech/polkadot/issues/3455), but is still a reasonable security. For not importing to the dispute-coordinator: This should be good as the dispute coordinator does scrape backing votes from chain. This suffices in practice as a super majority of validators must have seen a backing fork in order for a candidate to get included and only included candidates pose a threat to our system. The import from chain is preferable over direct import of backing votes for two reasons: 1. The import is batched, greatly improving import performance. All backing votes for a candidate are imported with a single import. And indeed we were able to see in metrics that importing votes from chain is fast. 2. We do less work in general as not every candidate for which statements are gossiped might actually make it on a chain. The dispute coordinator as with the current implementation would still import and keep those votes around for six sessions. While redundancy is good for reliability in the event of bugs, this also comes at a non negligible cost. The dispute-coordinator right now is the subsystem with the highest load, despite the fact that it should not be doing much during mormal operation and it is only getting worse with more parachains as the load is a direct function of the number of statements. We'll see on Versi how much of a performance improvement this PR * Get rid of dead code. * Dont send approval vote * Make it pass CI * Bring back tests for fixing them later. * Explicit signature check. * Resurrect approval-voting tests (not fixed yet) * Send out approval votes in dispute-distribution. Use BTreeMap for ordered dispute votes. * Bring back an important warning. * Fix approval voting tests. * Don't send out dispute message on import + test + Some cleanup. * Guide changes. Note that the introduced complexity is actually redundant. * WIP: guide changes. * Finish guide changes about dispute-coordinator conceputally. Requires more proof read still. Also removed obsolete implementation details, where the code is better suited as the source of truth. * Finish guide changes for now. * Remove own approval vote import logic. * Implement logic for retrieving approval-votes into approval-voting and approval-distribution subsystems. * Update roadmap/implementers-guide/src/node/disputes/dispute-coordinator.md Co-authored-by: asynchronous rob <rphmeier@gmail.com> * Review feedback. In particular: Add note about disputes of non included candidates. * Incorporate Review Remarks * Get rid of superfluous space. * Tidy up import logic a bit. Logical vote import is now separated, making the code more readable and maintainable. Also: Accept import if there is at least one invalid signer that has not exceeded its spam slots, instead of requiring all of them to not exceed their limits. This is more correct and a preparation for vote batching. * We don't need/have empty imports. * Fix tests and bugs. * Remove error prone redundancy. * Import approval votes on dispute initiated/concluded. * Add test for approval vote import. * Make guide checker happy (hopefully) * Another sanity check + better logs. * Reasoning about boundedness. * Use `CandidateIndex` as opposed to `CoreIndex`. * Remove redundant import. * Review remarks. * Add metric for calls to request signatures * More review remarks. * Add metric on imported approval votes. * Include candidate hash in logs. * More trace log * Break cycle. * Add some tracing. * Cleanup allowed messages. * fmt * Tracing + timeout for get inherent data. * Better error. * Break cycle in all places. * Clarified comment some more. * Typo. * Break cycle approval-distribution - approval-voting. Co-authored-by: asynchronous rob <rphmeier@gmail.com>
Polkadot
Implementation of a https://polkadot.network node in Rust based on the Substrate framework.
NOTE: In 2018, we split our implementation of "Polkadot" from its development framework "Substrate". See the Substrate repo for git history prior to 2018.
This repo contains runtimes for the Polkadot, Kusama, and Westend networks. The README provides
information about installing the polkadot binary and developing on the codebase. For more
specific guides, like how to be a validator, see the
Polkadot Wiki.
Installation
If you just wish to run a Polkadot node without compiling it yourself, you may either run the latest binary from our releases page, or install Polkadot from one of our package repositories.
Installation from the Debian or rpm repositories will create a systemd
service that can be used to run a Polkadot node. This is disabled by default,
and can be started by running systemctl start polkadot on demand (use
systemctl enable polkadot to make it auto-start after reboot). By default, it
will run as the polkadot user. Command-line flags passed to the binary can
be customized by editing /etc/default/polkadot. This file will not be
overwritten on updating polkadot. You may also just run the node directly from
the command-line.
Debian-based (Debian, Ubuntu)
Currently supports Debian 10 (Buster) and Ubuntu 20.04 (Focal), and
derivatives. Run the following commands as the root user.
# Import the security@parity.io GPG key
gpg --recv-keys --keyserver hkps://keys.mailvelope.com 9D4B2B6EB8F97156D19669A9FF0812D491B96798
gpg --export 9D4B2B6EB8F97156D19669A9FF0812D491B96798 > /usr/share/keyrings/parity.gpg
# Add the Parity repository and update the package index
echo 'deb [signed-by=/usr/share/keyrings/parity.gpg] https://releases.parity.io/deb release main' > /etc/apt/sources.list.d/parity.list
apt update
# Install the `parity-keyring` package - This will ensure the GPG key
# used by APT remains up-to-date
apt install parity-keyring
# Install polkadot
apt install polkadot
RPM-based (Fedora, CentOS)
Currently supports Fedora 32 and CentOS 8, and derivatives.
# Install dnf-plugins-core (This might already be installed)
dnf install dnf-plugins-core
# Add the repository and enable it
dnf config-manager --add-repo https://releases.parity.io/rpm/polkadot.repo
dnf config-manager --set-enabled polkadot
# Install polkadot (You may have to confirm the import of the GPG key, which
# should have the following fingerprint: 9D4B2B6EB8F97156D19669A9FF0812D491B96798)
dnf install polkadot
Building
Install via Cargo
Make sure you have the support software installed from the Build from Source section below this section.
If you want to install Polkadot in your PATH, you can do so with with:
cargo install --git https://github.com/paritytech/polkadot --tag <version> polkadot --locked
Build from Source
If you'd like to build from source, first install Rust. You may need to add Cargo's bin directory to your PATH environment variable. Restarting your computer will do this for you automatically.
curl https://sh.rustup.rs -sSf | sh
If you already have Rust installed, make sure you're using the latest version by running:
rustup update
Once done, finish installing the support software:
sudo apt install build-essential git clang libclang-dev pkg-config libssl-dev
Build the client by cloning this repository and running the following commands from the root directory of the repo:
git checkout <latest tagged release>
./scripts/init.sh
cargo build --release
Note that compilation is a memory intensive process. We recommend having 4 GiB of physical RAM or swap available (keep in mind that if a build hits swap it tends to be very slow).
Build from Source with Docker
You can also build from source using Parity CI docker image:
git checkout <latest tagged release>
docker run --rm -it -w /shellhere/polkadot \
-v $(pwd):/shellhere/polkadot \
paritytech/ci-linux:production cargo build --release
sudo chown -R $(id -u):$(id -g) target/
If you want to reproduce other steps of CI process you can use the following guide.
Networks
This repo supports runtimes for Polkadot, Kusama, and Westend.
Connect to Polkadot Mainnet
Connect to the global Polkadot Mainnet network by running:
./target/release/polkadot --chain=polkadot
You can see your node on telemetry (set a custom name with --name "my custom name").
Connect to the "Kusama" Canary Network
Connect to the global Kusama canary network by running:
./target/release/polkadot --chain=kusama
You can see your node on telemetry (set a custom name with --name "my custom name").
Connect to the Westend Testnet
Connect to the global Westend testnet by running:
./target/release/polkadot --chain=westend
You can see your node on telemetry (set a custom name with --name "my custom name").
Obtaining DOTs
If you want to do anything on Polkadot, Kusama, or Westend, then you'll need to get an account and some DOT, KSM, or WND tokens, respectively. See the claims instructions for Polkadot if you have DOTs to claim. For Westend's WND tokens, see the faucet instructions on the Wiki.
Hacking on Polkadot
If you'd actually like to hack on Polkadot, you can grab the source code and build it. Ensure you have Rust and the support software installed. This script will install or update Rust and install the required dependencies (this may take up to 30 minutes on Mac machines):
curl https://getsubstrate.io -sSf | bash -s -- --fast
Then, grab the Polkadot source code:
git clone https://github.com/paritytech/polkadot.git
cd polkadot
Then build the code. You will need to build in release mode (--release) to start a network. Only
use debug mode for development (faster compile times for development and testing).
./scripts/init.sh # Install WebAssembly. Update Rust
cargo build # Builds all native code
You can run the tests if you like:
cargo test --all --release
You can start a development chain with:
cargo run -- --dev
Detailed logs may be shown by running the node with the following environment variables set:
RUST_LOG=debug RUST_BACKTRACE=1 cargo run -- --dev
Development
You can run a simple single-node development "network" on your machine by running:
polkadot --dev
You can muck around by heading to https://polkadot.js.org/apps and choose "Local Node" from the Settings menu.
Local Two-node Testnet
If you want to see the multi-node consensus algorithm in action locally, then you can create a local testnet. You'll need two terminals open. In one, run:
polkadot --chain=polkadot-local --alice -d /tmp/alice
And in the other, run:
polkadot --chain=polkadot-local --bob -d /tmp/bob --port 30334 --bootnodes '/ip4/127.0.0.1/tcp/30333/p2p/ALICE_BOOTNODE_ID_HERE'
Ensure you replace ALICE_BOOTNODE_ID_HERE with the node ID from the output of the first terminal.
Monitoring
Once you set this up you can take a look at the Polkadot Grafana dashboards that we currently maintain.
Using Docker
Shell Completion
Contributing
Contributing Guidelines
Contributor Code of Conduct
License
Polkadot is GPL 3.0 licensed.