Commit Graph

76 Commits

Author SHA1 Message Date
Alexandru Vasile 4378fd2192 Improve reported statistics (#553)
* frontend: Update package lock

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* frontend/stats: Format linux kernel version

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* backend: Add [64; 128) RAM bucket

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* backend: Parse kernel string to include only version numbers

The linux kernel version string is parsed to include only
the kernel version, major version and minor version.
Ignoring the patch number and kernel specific info leads to
a better aggregation of data.

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* backend: Fix typo

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* core: Add CPU vendor to reported chain stats

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* frontend: Propagate CPU vendor to UI

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* backend: Parse kernel version by `+`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* backend: Add CPU vendors and ignore ascii case

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Revert "frontend/stats: Format linux kernel version"

This reverts commit 411b9a4ceef9c664816404eaee1fb64f61fe85b3.

* backend: Fix plus sign test

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* backend: Trim kernel versions

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* backend: Modify cpu_vendor

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
2023-11-03 15:13:38 +02:00
Alexandru Vasile 35b583d5f0 artifacts: Update GeoLite database (#551)
* Update GeoLite city database

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* core: Update DB download link and add copyright link

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* core/tests: Adjust location for random IP address

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
2023-10-17 17:41:44 +03:00
Alexandru Vasile b25f3fbf22 Add per Node Hardware Information to Telemetry Feed (#519)
* code: Add `sysinfo` and `hwbench` to the `AddedNode` message

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* code: Update `AddedNode` on `hwbench` updates

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* core: Rename cli flag `expose_node_ips` to `expose_node_details`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* core: Add hardware info only if the cli flag is enabled

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* tests: Adjust testing to the new `AddedNode` message

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* core: Move serialization of fields to the details array

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
2023-01-27 17:33:48 +02:00
James Wilson e7d15d03b3 Tweak logs and attempt to avoid races around removing nodes (#504)
* Tweak logs and attempt to avoid races around removing nodes

* wrapping_add in assign_id
2022-10-10 13:12:07 +01:00
Alexandru Vasile d525056190 core: Fix noisy logs caused by closing the /feed browser (#496)
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
2022-08-29 16:20:35 +03:00
Alexandru Vasile 09b44ad00f Expose node's IP address via feed (#491)
* Fix typos

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* core: Extend `feed::LocatedNode` message with optional IP address

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* core: Expose IpAddr from locator task

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* core: Expose CLI flag to handle IP

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* tests: Consider Option<String> for IP address

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* core: Add node's IP directly to the Node's details

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Adjust testing and serialize node's ip address

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* frontend: Propagate IP address for deserialization purposes

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* core: Clarify the CLI flag documentation

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
2022-08-17 15:35:36 +03:00
Ivan 5bff41d0c1 Add geoip db (#477)
* Use geoip for ipv4 city lookup

Signed-off-by: i1i1 <vanyarybin1@live.ru>

* Add support for ipv6

Signed-off-by: i1i1 <vanyarybin1@live.ru>

* Wrap locator into a blocking task

* Dummy change to try and trigger CI pipeline.

Co-authored-by: James Wilson <james@jsdw.me>
2022-07-19 22:06:07 +01:00
Koute 45878f9876 Add per-chain aggregate software/hardware telemetry (#464)
* Add per-chain aggregate software/hardware telemetry

* Fix tests' compilation

* Add extra comments for the `Counter` struct

* Replace the boolean argument with an enum

* Rename `replace_hwbench` to `update_hwbench`

* Move `Counter` into a separate file

* Move `ChainStatsCollator` to `chain_stats.rs`

* Fix incorrect key on the unknown table

* Improve types for the stats component; get rid of `any`
2022-04-27 18:44:34 +09:00
James Wilson 17432d712f Remove broken "Consensus" tab from UI and related code (#434)
* Remove consensus-tab related stuff, and unused messages, from the telemetry backend

* also remove AfgAuthoritySet feed message, and handle same from node

* Blat everything consensus related that I can find in the UI

* cargo fmt

* README: had -> has

Co-authored-by: David <dvdplm@gmail.com>

Co-authored-by: David <dvdplm@gmail.com>
2021-11-25 12:34:36 +00:00
James Wilson 7ff9a316d1 First/Third party by genesis hash, not label. Make limit configurable and default to 1k (#424)
* First/Third party by genesis hash, not label. Make limit configurable

* Fix a test that relies on not being a node limit

* remove a now-invalid comment

* Cargo fmt

* Fix another naff comment

* Update backend/telemetry_core/src/state/chain.rs

Comment tweak

Co-authored-by: David <dvdplm@gmail.com>

Co-authored-by: David <dvdplm@gmail.com>
2021-10-13 13:08:53 +01:00
James Wilson 38c5dff0b7 Saturating sub timestamp from now to avoid possible undeflowr (#420) 2021-10-01 16:55:35 +01:00
James Wilson 7ac88a7e84 Expose total messages sent to aggregator, too (#416)
* Expose total messages sent to aggregator so we can make better graphs with total dropped msgs

* cargo fmt

* use write to hopefully avoid some allocating

* add 'core' namespace to telemetry metrics for better future clarity

* cargo fmt
2021-09-30 12:21:37 +01:00
James Wilson ed6e292d25 Clean up shard channel when shard disconnects (#411) 2021-09-24 10:48:54 +01:00
James Wilson b4b128f9fe Tidy up stale connections. (#406)
* If messageId changes and network ID doesn't, remove 'old' message_id

* Boot nodes/connection when no recent messages received for it

* Separate task needed for soketto recv to avoid cancel-safety issues with new interval

* Wee tidy up

* cargo fmt

* Add some logging around node adding/removing

* Another log info msg

* a bit of tidy up

* bump stale node timeout to 60s
2021-09-21 15:49:42 +01:00
James Wilson a3ffaf3c44 Flume fix part 2: avoid using flume in a couple of cases, and revert attempted fix#1 (#396)
* A second attempt to avoid the flume memory leak, since the first didn't actually work

* No luck with second flume fix, so revert to futures::mpsc in a few places)

* cargo fmt

* Add a comment to cover use of into_stream
2021-09-03 15:55:25 +01:00
James Wilson 2932075783 Avoid using flume::Receiver::into_stream() to avoid memory leaks until the issue is resolved upstream (#394)
* Tweak rolling_total test to also confirm capacity doesn't go nuts

* Use Jemalloc

* Avoid flume's into_stream and use a workaround for now

* cargo fmt

* Improve comments now that there's an issue to point to
2021-09-03 08:40:43 +01:00
Maciej Hirsz a4069e4b3d Subscribe to chains by genesis hash (#395)
* Handle subscription by hash in the frontend

* Forward-ported backend changes

* Fix unit tests

* Remove unused `chains_by_label`

* fmt

* Updated but failing E2E tests

* subscribe by genesis hash in tests

* fmt

* Copy `BlockHash` instead of returning a ref

* Pin chains by genesisHash

Co-authored-by: James Wilson <james@jsdw.me>
2021-09-02 17:54:19 +02:00
James Wilson 87866b2d42 Improve logging and error reporting around IP and location info (#386)
* Beef up error reporting of IP and location info

* Tidy up error reporting after some manual testing of it

* Don't cache erroneous locations; try again when asked again

* cargo fmt
2021-08-27 16:16:26 +01:00
James Wilson 7a3e30cb01 Don't remove all feeds subscribed to a chain when one disconnects (#383)
* Only remove the feed that disconnected to not break the rest...

* use multimap struct to avoid sync issues between feed and chain

* add a remove test, too

* cargo fmt

* fix name of test

* move multimap to common so we can doctest it and add 'unique' to name

* cargo fmt

* Return old key if value moved to make uniqueness more obvious
2021-08-27 08:05:44 +01:00
James Wilson 46b0641dfd Clarify wording 2021-08-13 11:45:03 +01:00
James Wilson 77460ffc27 cargo fmt 2021-08-13 11:35:24 +01:00
James Wilson b842c7fc8b expose dropped message counts and fix some typos/wording 2021-08-13 11:33:53 +01:00
James Wilson 05a3ba3fef Fix/expand a few comments 2021-08-12 16:20:05 +01:00
James Wilson 230987036a cargo fmt 2021-08-12 16:01:35 +01:00
James Wilson 9017f328f0 Add comment explaining prometheus metrics endpoint body 2021-08-12 16:01:33 +01:00
James Wilson 92da674d4d Expose metrics in a format that prometheus understands 2021-08-12 16:01:32 +01:00
James Wilson 6db7f484ef Fix compile err with diagnostic msg 2021-08-12 16:01:29 +01:00
James Wilson ab2303ce5c more diagnostic logging 2021-08-12 16:01:27 +01:00
James Wilson 3319709f7b Add periodic interval to core loop and print debug info 2021-08-12 16:01:25 +01:00
James Wilson bd7a21ec39 Flumify everything 2021-08-12 16:01:17 +01:00
James Wilson 11b0b3a3c7 remove final use of futures::mpsc and replace with flume 2021-08-12 16:01:15 +01:00
James Wilson 703a9ddc4e use flume throughout telemetry_core 2021-08-12 16:01:14 +01:00
James Wilson 8268cf2afe print feed 1 msg len 2021-08-12 16:01:12 +01:00
James Wilson 98c9ccd278 fmt, clean warnings, tidy aggregator opts and add queue length limit 2021-08-12 16:01:11 +01:00
James Wilson 968dd2b957 Try to force new thread for msg counter to ensure it has time to print 2021-08-12 16:01:09 +01:00
James Wilson b97aec99a8 monitoring queue len 2021-08-12 16:01:07 +01:00
James Wilson 87c0ee7d0d monitor aggregator length (dont discard msgs yet) 2021-08-12 16:00:56 +01:00
James Wilson 770dd04b57 invert logic to make name make sense and fix comment typo 2021-08-12 12:47:29 +01:00
James Wilson 80d6ad916e Address David's comments 2021-08-11 17:23:22 +01:00
James Wilson f26b39ac63 Address feedback from Niklas 2021-08-11 16:59:11 +01:00
James Wilson 9f76fabaed give tokio threads a more convenient name for monitoring purposes 2021-08-09 11:36:46 +01:00
James Wilson b22efc804a Fix comment typo 2021-08-09 10:56:01 +01:00
James Wilson 626fe95d89 1 aggregator loop by default for now 2021-08-09 10:13:10 +01:00
James Wilson c469ef8dfe make AggregatorSet close to zero cost when only 1 aggregator asked for 2021-08-09 10:09:17 +01:00
James Wilson c99cbee1e9 Allow multiple aggregator loops in the core to try and spread feed subscription/send cost out 2021-08-07 17:08:58 +01:00
James Wilson 9c001bdcfd Add a comment on core worker threads 2021-08-06 17:50:50 +01:00
James Wilson 8017d8b00f Cargo fmt 2021-08-06 17:49:19 +01:00
James Wilson 78ad7115e5 set sensible worker thread limits again but allow full control to caller 2021-08-06 17:49:04 +01:00
James Wilson 88c3db3562 cargo fmt 2021-08-06 17:44:26 +01:00
James Wilson 74cf55174e Use rayon to speed up subscribe message serializing 2021-08-06 17:42:03 +01:00