From 750aaa06dcce41c8edf92a82cedc6070cef4b73e Mon Sep 17 00:00:00 2001 From: James Wilson Date: Mon, 5 Jul 2021 10:26:32 +0100 Subject: [PATCH] Tweak docker(compose) files (can't test them yet) and add a little backend documentation --- backend/.dockerignore | 2 +- backend/README.md | 18 +++++++++++++++ backend/docs/architecture.svg | 3 +++ backend/shard.Dockerfile | 24 ++++++++++++++++++++ backend/{Dockerfile => telemetry.Dockerfile} | 2 +- docker-compose.yml | 17 ++++++++++++-- 6 files changed, 62 insertions(+), 4 deletions(-) create mode 100644 backend/README.md create mode 100644 backend/docs/architecture.svg create mode 100644 backend/shard.Dockerfile rename backend/{Dockerfile => telemetry.Dockerfile} (89%) diff --git a/backend/.dockerignore b/backend/.dockerignore index 8f6db69..af65baa 100644 --- a/backend/.dockerignore +++ b/backend/.dockerignore @@ -1,3 +1,3 @@ target -Dockerfile +*.Dockerfile .git diff --git a/backend/README.md b/backend/README.md new file mode 100644 index 0000000..bbf46dd --- /dev/null +++ b/backend/README.md @@ -0,0 +1,18 @@ +# Backend Crates + +This folder contains the rust crates and documentation specific to the telemetry backend. A description of the folders: + +- [telemetry](./telemetry): The Telemetry Core. This aggregates data received from shards and allows UI feeds to connect and receive this information. +- [shard](./shard): A Shard. It's expected that multiple of these will run. Nodes will connect to Shard instances and send JSON telemetry to them, and Shard instances will each connect to the Telemetry Core and relay on relevant data to it. +- [common](./common): common code shared between the telemetry shard and core +- [docs](./docs): Material supporting the documentation lives here + +# Architecture + +As we move to a sharded version of this telemetry server, this set of architecture diagrams may be useful in helping to understand the current setup (middle diagram), previous setup (first diagram) and possible future setup if we need to scale further (last diagram): + +![Architecture Diagram](./docs/architecture.svg) + +# Deployment + +A `shard.Dockerfile` and `telemetry.Dockerfile` exist to build the Shard and Telemetry Core binaries into docker containers. A `socker-compose.yaml` in the root of the repository can serve as an example of these services, along with the UI, running together. \ No newline at end of file diff --git a/backend/docs/architecture.svg b/backend/docs/architecture.svg new file mode 100644 index 0000000..f82280e --- /dev/null +++ b/backend/docs/architecture.svg @@ -0,0 +1,3 @@ + + +
Telemetry Core
Telemetry Core
Feed 1
Feed 1
Feed 2
Feed 2
Feed 3
Feed 3
Feed N
Feed N
Node
Node
Node
Node
Node
Node
Node
Node

Nodes connect to the telemetry address and their connection is routed to the telemetry core process.

The core process stores chain-specific state, and routes messages to the relevant chain to update that state and send updates to feeds subscribed to that chain.

Feeds subscribe to chains and are routed to the relevant chain actor to receive updates from it.


Nodes connect to the telemetry address and...
JSON
JSON
JSON
(compact format)
JSON...
The current architecture
The current architecture
Shard 1
Shard 1
Shard 2
Shard 2
Shard 3
Shard 3
Shard N
Shard N
Telemetry Core
Telemetry Core
Feed 1
Feed 1
Feed 2
Feed 2
Feed 3
Feed 3
Feed N
Feed N
Node
Node
Node
Node
Node
Node
Node
Node
Load balancer
Load balancer
k8s
k8s

Nodes connect to the telemetry address and their connection is routed to one of the available shards. On connection, they send "system connected" messages for each set of node information they wish to send out and then start sending updates about that node.
Nodes connect to the telemetry address and...

One way or another, we'd like to distribute incoming connections across shards. Nodes will try to reconnect in a few seconds if they are disconnected.
One way or another, we'd like to distribut...

Shards receive JSON telemetry from nodes. Their main task is to deserialize it, and send along to the Telemetry Core only data that it cares about, ignoring the rest. It re-serializes to a more compact format (bincode currently) to try and minimise bandwidth to the core.
Shards receive JSON telemetry from nodes....

The core process keeps track of the state of connected nodes as information comes in, and broadcasts relevant updates out to feeds. If tells shards to "mute" messages from nodes when there are too many nodes on a chain, or a chain on our deny list.

If this core process dies and restarts, shards will force nodes to reconnect after reconnecting to the core, so that they send out their "system connected" message again.

If a shard connection is lost, the core removes all knowledge of nodes connected via that shard, and broadcasts relevant updates to feeds.
The core process keeps track of the state...

Feeds are browsers that have connected to the telemetry core. Feeds get sent basic information about the chains that exist, and then tell the core what chain they want to subscribe to. The core keeps track of this and sends chain-specific messages only to interested feeds.
Feeds are browsers that have connected to...
bincode
bincode
JSON
JSON
JSON
(compact format)
JSON...
The new architecture ("sharded")
The new architecture ("sharded")
Shard 1
Shard 1
Shard 2
Shard 2
Shard 3
Shard 3
Shard N
Shard N
Telemetry Core
Telemetry Core
Feed 1
Feed 1
Feed 2
Feed 2
Feed 3
Feed 3
Feed N
Feed N
Node
Node
Node
Node
Node
Node
Node
Node
Load balancer
Load balancer
k8s
k8s

Shards connect to multiple telemetry core processes and clone messages to each of them. Potentially, we could "bucket" things based on genesis hashes, but an easy first step is to send all messages to every core.

If any of the connected "core" processes go down, we force nodes to reconnect when it comes back up, to get new uptodate "system connected" messages.
Shards connect to multiple telemetry core...

Each telemetry core process would have a complete state of the world, and work in the same way that it does in the current sharded approach.
Each telemetry core process would have a c...

Because we have multiple "core" processes with a complete state of the world, we can load balance connected feeds across them.
Because we have multiple "core" processes...
bincode
bincode
JSON
JSON
JSON
(compact format)
JSON...
One possible future from this
One possible future from this
Telemetry Core
Telemetry Core
Telemetry Core
Telemetry Core
Load balancer
Load balancer

With shards, we remove the bottleneck on incoming node traffic. We could also "shard" the telemetry core process to help reduce the load from connected feeds if we find it is necessary (either by completely separate processes as this diagram shows, or multiple "loops" running inside a single telemetry core process to internally split feed connections across.)
With shards, we remove the bottleneck on incoming node traffic. We could also "shard" the tele...

With a single telemetry service, we have a bunch of data coming in to a single process, which pushed bandwidth limits and puts more pressure on the single process to handle deserialization and such. We can add shards to spread the load of handling the incoming message deserialization and bandwidth, and they can each strip away all unneeded info before passing it on to the core process.
With a single telemetry service, we have a bunch of data coming in to a single process, which...
JSON
(compact format)
JSON...Viewer does not support full SVG 1.1 \ No newline at end of file diff --git a/backend/shard.Dockerfile b/backend/shard.Dockerfile new file mode 100644 index 0000000..edcc5bc --- /dev/null +++ b/backend/shard.Dockerfile @@ -0,0 +1,24 @@ +FROM paritytech/ci-linux:production as builder + +ARG PROFILE=release +WORKDIR /app + +COPY . . + +RUN cargo build --${PROFILE} --bins + +# MAIN IMAGE FOR PEOPLE TO PULL --- small one# +FROM debian:buster-slim +LABEL maintainer="Parity Technologies" +LABEL description="Polkadot Telemetry backend shard, static build" + +ARG PROFILE=release +WORKDIR /usr/local/bin + +COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ +COPY --from=builder /app/target/$PROFILE/shard /usr/local/bin +RUN apt-get -y update && apt-get -y install openssl && apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/ + +EXPOSE 8000 + +ENTRYPOINT ["shard"] diff --git a/backend/Dockerfile b/backend/telemetry.Dockerfile similarity index 89% rename from backend/Dockerfile rename to backend/telemetry.Dockerfile index a19f453..4e26bf9 100644 --- a/backend/Dockerfile +++ b/backend/telemetry.Dockerfile @@ -10,7 +10,7 @@ RUN cargo build --${PROFILE} --bins # MAIN IMAGE FOR PEOPLE TO PULL --- small one# FROM debian:buster-slim LABEL maintainer="Parity Technologies" -LABEL description="Polkadot Telemetry backend, static build" +LABEL description="Polkadot Telemetry backend core, static build" ARG PROFILE=release WORKDIR /usr/local/bin diff --git a/docker-compose.yml b/docker-compose.yml index 67d807a..0e37e8c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -11,9 +11,22 @@ services: - ./packages:/app/packages ports: - 3000:80 - telemetry-backend: + expose: + - 3000 + telemetry-backend-shard: build: - dockerfile: Dockerfile + dockerfile: shard.Dockerfile + context: ./backend/ + environment: + - PORT=8001 + command: ['--core', 'http://telemetry-backend-core:8000'] + ports: + - 8001:8001 + expose: + - 8001 + telemetry-backend-core: + build: + dockerfile: telemetry.Dockerfile context: ./backend/ environment: - PORT=8000