From 44bb4c32a65747d08713c9d4776d4663b301a0db Mon Sep 17 00:00:00 2001 From: David Palm Date: Thu, 25 Mar 2021 14:23:03 +0100 Subject: [PATCH 1/7] Bump version to 0.3 Fix markdown Add CHANGELOG --- README.md | 8 ++++---- ROADMAP.md | 12 ++++++++---- backend/Cargo.toml | 2 +- 3 files changed, 13 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 9e8daee..eafd012 100644 --- a/README.md +++ b/README.md @@ -63,14 +63,14 @@ For the sake of brevity below, I will name the containers `backend` and `fronten Let's start the backend first. We will be using the published [chevdor](https://hub.docker.com/u/chevdor) images here, feel free to replace with your own image. -``` +```sh docker run --rm -i --name backend -p 8000:8000 \ chevdor/substrate-telemetry-backend -l 0.0.0.0:8000 ``` Let's now start the frontend: -``` +```sh docker run --rm -i --name frontend --link backend -p 80:80 \ -e SUBSTRATE_TELEMETRY_URL=ws://localhost:8000/feed \ chevdor/substrate-telemetry-frontend @@ -84,7 +84,7 @@ At that point, you can already open your browser at [http://localhost](http://lo Let's bring some data in with a node: -``` +```sh docker run --rm -i --name substrate --link backend -p 9944:9944 \ chevdor/substrate substrate --dev --telemetry-url 'ws://backend:8000/submit 0' ``` @@ -111,6 +111,6 @@ Now navigate to [http://localhost:3000](http://localhost:3000/) in your browser The building process is standard. You just need to notice that the Dockerfile is in ./packages/frontend/ and tell docker about it. The context must remain the repository's root though. -``` +```sh DOCKER_USER=chevdor ./scripts/build-docker-frontend.sh ``` diff --git a/ROADMAP.md b/ROADMAP.md index 9500027..7cec2aa 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,13 +1,17 @@ -# Frontend +# Roadmap + +## Frontend - Provide a pie chart of node implementations/versions above the list, next to Last Block timer, to have an overview of what the dominant impl/version is, and whether there is a significant amount of nodes running older versions. - + Latest partial implementation of this can be found in the `mh-piechart-new` branch. + + Latest partial implementation of this can be found in the `mh-piechart-new` branch. - Provide a block propagation graph: X axis being a number of blocks a node is behind, while Y axis is the number of nodes in that bucket. The size of buckets should grow: [0, 1, 2, 5, 10, 20, 50, 100, 200, 500, 1000] - + Partial implementation of this can be found in `mh-version-piecharts` branch, although that branch is pre FE refactoring, so starting a new one and splitting it from pie charts would be advised. + + Partial implementation of this can be found in `mh-version-piecharts` branch, although that branch is pre FE refactoring, so starting a new one and splitting it from pie charts would be advised. - Network search by name when viewing the complete list (#276). -# Backend +## Backend - To keep up with increasing traffic, we should split out a new service from the current backend that replaces the `/submit` endpoint. This new service should take ownership of JSON deserialization of incoming messages from the nodes, discarding messages that Telemetry does not need, resolving chain multiplexing (this will likely need some two-way communication with the main backend when a new node connects), and then forwarding those messages using a lightweight protocol (Cap'n Proto or Protocol Buffers) to the main telemetry backend. Unlike the backend, which needs to have a single instance to keep track of all state changes, this new service should be stateless and therefore we should be able to spawn multiple instances of it behind a load balancer. This would solve the two bottlenecks we're currently having: the number of concurrent connections going to the backend, and the CPU use that comes from IO switching and JSON deserialization. diff --git a/backend/Cargo.toml b/backend/Cargo.toml index f1d0adf..b5e7082 100644 --- a/backend/Cargo.toml +++ b/backend/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "telemetry" -version = "0.2.1" +version = "0.3" authors = ["Parity Technologies Ltd. "] edition = "2018" license = "GPL-3.0" From 2f84c9530e41a324b1756a538dc896a4538f02c3 Mon Sep 17 00:00:00 2001 From: David Palm Date: Thu, 25 Mar 2021 14:23:37 +0100 Subject: [PATCH 2/7] Add CHANGELOG --- CHANGELOG.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..fc8496a --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,14 @@ +# Changelog + +All notable changes to this crate are documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), +and this crate adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [0.3] - 2021-03-25 + +### Added + +- Cap third party networks to at most 500 connected nodes. Polkadot, Kusama, Westend and Rococo are not subject to this limit. [#314](https://github.com/paritytech/substrate-telemetry/pull/314) +- Fix clippy warnings [#314](https://github.com/paritytech/substrate-telemetry/pull/314) +- Add `--log` switch to configure log levels [#314](https://github.com/paritytech/substrate-telemetry/pull/314) From 9403d31cbe248976280ed385dfdf643b3852b06e Mon Sep 17 00:00:00 2001 From: David Palm Date: Thu, 25 Mar 2021 14:27:13 +0100 Subject: [PATCH 3/7] Proper version number --- backend/Cargo.lock | 2 +- backend/Cargo.toml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/backend/Cargo.lock b/backend/Cargo.lock index 8bac687..8ad86b1 100644 --- a/backend/Cargo.lock +++ b/backend/Cargo.lock @@ -2375,7 +2375,7 @@ dependencies = [ [[package]] name = "telemetry" -version = "0.2.1" +version = "0.3.0" dependencies = [ "actix", "actix-http", diff --git a/backend/Cargo.toml b/backend/Cargo.toml index b5e7082..1cdcace 100644 --- a/backend/Cargo.toml +++ b/backend/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "telemetry" -version = "0.3" +version = "0.3.0" authors = ["Parity Technologies Ltd. "] edition = "2018" license = "GPL-3.0" From 857c61251836ebc526abfe9e40010b14cb1e2ff3 Mon Sep 17 00:00:00 2001 From: David Palm Date: Fri, 26 Mar 2021 07:59:39 +0100 Subject: [PATCH 4/7] Print current version at startup --- backend/src/main.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/backend/src/main.rs b/backend/src/main.rs index c5168c9..1cef0fc 100644 --- a/backend/src/main.rs +++ b/backend/src/main.rs @@ -165,7 +165,7 @@ async fn health(aggregator: web::Data>) -> Result std::io::Result<()> { - let opts: Opts = Opts::parse(); + let opts = Opts::parse(); let log_level = &opts.log_level; SimpleLogger::new().with_level(log_level.into()).init().expect("Must be able to start a logger"); @@ -173,7 +173,7 @@ async fn main() -> std::io::Result<()> { let aggregator = Aggregator::new(denylist).start(); let factory = LocatorFactory::new(); let locator = SyncArbiter::start(4, move || factory.create()); - + log::info!("Starting telemetry version: {}", env!("CARGO_PKG_VERSION")); HttpServer::new(move || { App::new() .wrap(middleware::NormalizePath::default()) From aefac014b9c9e3d5ae3ea1192c6d51f236d7243a Mon Sep 17 00:00:00 2001 From: David Palm Date: Fri, 26 Mar 2021 10:58:59 +0100 Subject: [PATCH 5/7] Update changelog --- CHANGELOG.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fc8496a..c083b0e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,5 +10,13 @@ and this crate adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.h ### Added - Cap third party networks to at most 500 connected nodes. Polkadot, Kusama, Westend and Rococo are not subject to this limit. [#314](https://github.com/paritytech/substrate-telemetry/pull/314) -- Fix clippy warnings [#314](https://github.com/paritytech/substrate-telemetry/pull/314) - Add `--log` switch to configure log levels [#314](https://github.com/paritytech/substrate-telemetry/pull/314) +- Add `--denylist network1 network2` switch to deny nodes on the given networks to connect [#315](https://github.com/paritytech/substrate-telemetry/pull/314) + +### Fixed + +- Fix clippy warnings [#314](https://github.com/paritytech/substrate-telemetry/pull/314) + +### Changed + +- Update to actix v4 beta [#316](https://github.com/paritytech/substrate-telemetry/pull/317) From 3cf6837b75a5e8ec98a71d5d800c5f146196a298 Mon Sep 17 00:00:00 2001 From: David Palm Date: Fri, 26 Mar 2021 10:59:18 +0100 Subject: [PATCH 6/7] No default value for denylist --- backend/src/main.rs | 1 - 1 file changed, 1 deletion(-) diff --git a/backend/src/main.rs b/backend/src/main.rs index 1cef0fc..262e03b 100644 --- a/backend/src/main.rs +++ b/backend/src/main.rs @@ -40,7 +40,6 @@ struct Opts { #[clap( required = false, long = "denylist", - default_value = "Earth", about = "Space delimited list of chains that are not allowed to connect to telemetry. Case sensitive." )] denylist: Vec, From f327e528e09c56297998af5d8b52fe6f42143800 Mon Sep 17 00:00:00 2001 From: David Palm Date: Mon, 29 Mar 2021 22:21:33 +0200 Subject: [PATCH 7/7] update CHANGELOG --- CHANGELOG.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c083b0e..0db0ce4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,9 +9,8 @@ and this crate adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.h ### Added -- Cap third party networks to at most 500 connected nodes. Polkadot, Kusama, Westend and Rococo are not subject to this limit. [#314](https://github.com/paritytech/substrate-telemetry/pull/314) -- Add `--log` switch to configure log levels [#314](https://github.com/paritytech/substrate-telemetry/pull/314) - Add `--denylist network1 network2` switch to deny nodes on the given networks to connect [#315](https://github.com/paritytech/substrate-telemetry/pull/314) +- Add `--log` switch to configure log levels [#314](https://github.com/paritytech/substrate-telemetry/pull/314) ### Fixed @@ -19,4 +18,8 @@ and this crate adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.h ### Changed +- Docker image use alpine (for now) [#326](https://github.com/paritytech/substrate-telemetry/pull/326) +- Mute denied nodes [#322](https://github.com/paritytech/substrate-telemetry/pull/322) +- Build actix-web without compression support [#319](https://github.com/paritytech/substrate-telemetry/pull/319) - Update to actix v4 beta [#316](https://github.com/paritytech/substrate-telemetry/pull/317) +- Cap third party networks to at most 500 connected nodes. Polkadot, Kusama, Westend and Rococo are not subject to this limit. [#314](https://github.com/paritytech/substrate-telemetry/pull/314)