pezkuwichain/pezkuwi-telemetry
1
Fork 0
mirror of https://github.com/pezkuwichain/pezkuwi-telemetry.git synced 2026-10-20 18:56:21 +00:00
Code Releases Activity
681 Commits 13 Branches 0 Tags
5bff41d0c12e5e51cdba3d74d9ca0f81630f2cf2
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
James Wilson a93a6cadfc Add manual deploy step with no restrictions. (#481) 2022-06-16 14:26:57 +01:00
James Wilson b9f93242cd fix readme image names and remove network bit in docker-compose (#447) 
* fix readme image names and remove network bit in docker-compose

* undo naff tweak
 2022-01-18 11:12:33 +00:00
James WilsonandTarik Gul c00cab33c9 Allow chains to be sorted and filtered (#440) 
* Allow soak tests to generate lots of chains for testing

* Style tweaks, and redo 'all chains' modal

* make highlighted text readable on selected chain

* cargo fmt

* Update frontend/src/index.css

Fix a typo

Co-authored-by: Tarik Gul <47201679+TarikGul@users.noreply.github.com>

* A couple more wee telemetry style tweaks

* ..but make the tab animation faster

* Be more defensive checking for event target

* Comment out animation for now

Co-authored-by: Tarik Gul <47201679+TarikGul@users.noreply.github.com>
 2022-01-10 14:57:18 +00:00
James Wilson b4fd955c78 catch icon generation errors and log to console (#441) 2022-01-07 20:37:07 +00:00
James WilsonandDavid 17432d712f Remove broken "Consensus" tab from UI and related code (#434) 
* Remove consensus-tab related stuff, and unused messages, from the telemetry backend

* also remove AfgAuthoritySet feed message, and handle same from node

* Blat everything consensus related that I can find in the UI

* cargo fmt

* README: had -> has

Co-authored-by: David <dvdplm@gmail.com>

Co-authored-by: David <dvdplm@gmail.com>
 2021-11-25 12:34:36 +00:00
James Wilson 6e2faa43be We settled on using tags to deploy to live, so reflect that in the docs (#426) 
* We settled on using tags to deploy to live, so reflect that in the docs

* not semantic versioning, and how to generate short hash
 2021-10-14 09:24:06 +01:00
James WilsonandDavid 7ff9a316d1 First/Third party by genesis hash, not label. Make limit configurable and default to 1k (#424) 
* First/Third party by genesis hash, not label. Make limit configurable

* Fix a test that relies on not being a node limit

* remove a now-invalid comment

* Cargo fmt

* Fix another naff comment

* Update backend/telemetry_core/src/state/chain.rs

Comment tweak

Co-authored-by: David <dvdplm@gmail.com>

Co-authored-by: David <dvdplm@gmail.com>
 2021-10-13 13:08:53 +01:00
James WilsonandMaciej Hirsz 82f0b234c4 Timeout based on last msg received and not ping timeout (#418) 
* Timeout based on last msg received and not ping timeout

* Use object method shorthand

Co-authored-by: Maciej Hirsz <1096222+maciejhirsz@users.noreply.github.com>

* Types.Milliseconds instead of number

Co-authored-by: Maciej Hirsz <1096222+maciejhirsz@users.noreply.github.com>
 2021-10-01 17:27:19 +01:00
James Wilson 38c5dff0b7 Saturating sub timestamp from now to avoid possible undeflowr (#420) 2021-10-01 16:55:35 +01:00
James Wilson 18a3d7a42f Add internal deployment steps to the README (#414) 
* Add internal deployment steps to the README

* wee tweak in deployment steps

* use FORCE_DEPLOY as primary deploy approach

* slight wording tweak
 2021-10-01 12:09:38 +01:00
James Wilson 7ac88a7e84 Expose total messages sent to aggregator, too (#416) 
* Expose total messages sent to aggregator so we can make better graphs with total dropped msgs

* cargo fmt

* use write to hopefully avoid some allocating

* add 'core' namespace to telemetry metrics for better future clarity

* cargo fmt
 2021-09-30 12:21:37 +01:00
James Wilson ed6e292d25 Clean up shard channel when shard disconnects (#411) 2021-09-24 10:48:54 +01:00
James Wilson b4b128f9fe Tidy up stale connections. (#406) 
* If messageId changes and network ID doesn't, remove 'old' message_id

* Boot nodes/connection when no recent messages received for it

* Separate task needed for soketto recv to avoid cancel-safety issues with new interval

* Wee tidy up

* cargo fmt

* Add some logging around node adding/removing

* Another log info msg

* a bit of tidy up

* bump stale node timeout to 60s
 2021-09-21 15:49:42 +01:00
James WilsonandDavid Palm 0b0cec0512 Make soak tests work again now that we subscribe by genesis hash (#399) 
* Make soak tests work again with genesis hash subscribing

* cargo fmt

* derive hex from hash

* actually compile code

* fmt

Co-authored-by: David Palm <dvdplm@gmail.com>
 2021-09-10 11:28:34 +01:00
James Wilson fe19a75414 Un-brittle-ify backend E2E tests and have them run by default again in CI (#397) 
* test running tests

* Add delay so that core knows about node before feed subscribes to limit chance of race

* move delaye to the right place

* Don't do expensive docker step untill we will be pushing the image

* docker test skipped as hoped, so push to 'true'

* just remove docker steps in github CI entirely since they aren't needed by anything (gitlab CI does this stuff now)

* run CI on pull requests too to catch PRs from forks
 2021-09-06 11:30:17 +01:00
James Wilson a3ffaf3c44 Flume fix part 2: avoid using flume in a couple of cases, and revert attempted fix#1 (#396) 
* A second attempt to avoid the flume memory leak, since the first didn't actually work

* No luck with second flume fix, so revert to futures::mpsc in a few places)

* cargo fmt

* Add a comment to cover use of into_stream
 2021-09-03 15:55:25 +01:00
James Wilson 2932075783 Avoid using flume::Receiver::into_stream() to avoid memory leaks until the issue is resolved upstream (#394) 
* Tweak rolling_total test to also confirm capacity doesn't go nuts

* Use Jemalloc

* Avoid flume's into_stream and use a workaround for now

* cargo fmt

* Improve comments now that there's an issue to point to
 2021-09-03 08:40:43 +01:00
James Wilson ec5db0fbbf Bump tokio to 1.10 and add a test to confirm memory usage of rolling_total (#392) 2021-08-31 20:18:46 +02:00
James Wilson 87866b2d42 Improve logging and error reporting around IP and location info (#386) 
* Beef up error reporting of IP and location info

* Tidy up error reporting after some manual testing of it

* Don't cache erroneous locations; try again when asked again

* cargo fmt
 2021-08-27 16:16:26 +01:00
James Wilson 7a3e30cb01 Don't remove all feeds subscribed to a chain when one disconnects (#383) 
* Only remove the feed that disconnected to not break the rest...

* use multimap struct to avoid sync issues between feed and chain

* add a remove test, too

* cargo fmt

* fix name of test

* move multimap to common so we can doctest it and add 'unique' to name

* cargo fmt

* Return old key if value moved to make uniqueness more obvious
 2021-08-27 08:05:44 +01:00
James Wilson 1e05000e8d Merge pull request #372 from arshamparity/fix-ci 
fixing the ci
 2021-08-16 16:01:29 +01:00
James Wilson 01748185f0 Merge pull request #371 from arshamparity/helm-servicemonitor 
Helm servicemonitor
 2021-08-16 15:46:17 +01:00
James Wilson c0bceb7fc0 Merge pull request #368 from paritytech/jsdw-simpler-docker 
Simplify Helm/Docker usage
 2021-08-16 14:26:32 +01:00
James Wilson 701a431ac1 Remove parity specific bit in nginx file (not needed anyway), name default network, tweak a dockerfile comment 2021-08-16 14:08:28 +01:00
James Wilson 65eac5c3e8 Add env to frontend image of docker-compose (since it doesnt proxy internally now) 2021-08-14 12:15:49 +01:00
James Wilson 53f7769146 Make frontend nginx conf suitable for k8s or local deployment (no need for 'old' deployment bits) 2021-08-14 11:57:52 +01:00
James Wilson 8793d0fd5b pass the feed URL via env var rather than config map so that standalone Docker usage is simplified 2021-08-14 11:32:48 +01:00
James Wilson 502fd2e91a Merge pull request #367 from paritytech/jsdw-sharding-gatekeeper 
Flume, Metrics and Message control when overloaded
 2021-08-13 14:35:43 +01:00
James Wilson 46b0641dfd Clarify wording 2021-08-13 11:45:03 +01:00
James Wilson 77460ffc27 cargo fmt 2021-08-13 11:35:24 +01:00
James Wilson b842c7fc8b expose dropped message counts and fix some typos/wording 2021-08-13 11:33:53 +01:00
James Wilson 811babca27 Merge branch 'master' into jsdw-sharding-gatekeeper 2021-08-13 11:16:47 +01:00
James Wilson 425ac73422 Merge pull request #365 from paritytech/jsdw-minor-fixes 
Jsdw minor fixes
 2021-08-13 10:51:17 +01:00
James Wilson 18627a9f02 No e2e feature flag; just ignore and pattern match on 'e2e' to run 2021-08-12 16:58:35 +01:00
James Wilson 05a3ba3fef Fix/expand a few comments 2021-08-12 16:20:05 +01:00
James Wilson 230987036a cargo fmt 2021-08-12 16:01:35 +01:00
James Wilson 9017f328f0 Add comment explaining prometheus metrics endpoint body 2021-08-12 16:01:33 +01:00
James Wilson 92da674d4d Expose metrics in a format that prometheus understands 2021-08-12 16:01:32 +01:00
James Wilson 4f7b2c8ec5 Confirm that densemap len wont panic if lots of retired items 2021-08-12 16:01:30 +01:00
James Wilson 6db7f484ef Fix compile err with diagnostic msg 2021-08-12 16:01:29 +01:00
James Wilson ab2303ce5c more diagnostic logging 2021-08-12 16:01:27 +01:00
James Wilson 3319709f7b Add periodic interval to core loop and print debug info 2021-08-12 16:01:25 +01:00
James Wilson f72f8c1fd5 test runner: fix soak test for multiple ids per ndoe 2021-08-12 16:01:24 +01:00
James Wilson 20463ce159 test runner: enable tokio features 2021-08-12 16:01:19 +01:00
James Wilson e3fcd4e8c2 Clean up soak test runner and add more config options 2021-08-12 16:01:18 +01:00
James Wilson bd7a21ec39 Flumify everything 2021-08-12 16:01:17 +01:00
James Wilson 11b0b3a3c7 remove final use of futures::mpsc and replace with flume 2021-08-12 16:01:15 +01:00
James Wilson 703a9ddc4e use flume throughout telemetry_core 2021-08-12 16:01:14 +01:00
James Wilson 8268cf2afe print feed 1 msg len 2021-08-12 16:01:12 +01:00
James Wilson 98c9ccd278 fmt, clean warnings, tidy aggregator opts and add queue length limit 2021-08-12 16:01:11 +01:00
James Wilson 968dd2b957 Try to force new thread for msg counter to ensure it has time to print 2021-08-12 16:01:09 +01:00
James Wilson b97aec99a8 monitoring queue len 2021-08-12 16:01:07 +01:00
James Wilson 87c0ee7d0d monitor aggregator length (dont discard msgs yet) 2021-08-12 16:00:56 +01:00
James Wilson 35cfb16ed9 Copy and adapt better CI from Soketto instead. Hopefully cache bits will improve build speed 2021-08-12 14:57:10 +01:00
James Wilson d4b5c2b0c8 split e2e tests out and run them separately, not blocking the build or marking it as failed 2021-08-12 13:38:29 +01:00
James Wilson 7aa4ad49b3 bump the msg timeout a little higher 2021-08-12 13:10:17 +01:00
James WilsonandNiklas Adolfsson 7563909609 Fix typo 
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
 2021-08-12 12:59:39 +01:00
James Wilson 28b798debe Fix gitlab CI to run on master and auto build/deploy to staging on merge to master 2021-08-12 12:57:03 +01:00
James Wilson bb25a8c3e2 enforce rustfmt and valid doc links in CI 2021-08-12 12:53:14 +01:00
James Wilson 770dd04b57 invert logic to make name make sense and fix comment typo 2021-08-12 12:47:29 +01:00
James Wilson f887510beb remove lint warning on cargo test 2021-08-12 12:42:44 +01:00
James Wilson 4480bbe72a Allow errors as well as closes for now to remove some brittleness 2021-08-12 12:40:53 +01:00
James Wilson 705d57a3b6 Merge pull request #349 from paritytech/jsdw-sharding 
Sharded Telemetry Server
 2021-08-12 10:40:36 +01:00
James Wilson 9f2db87811 Add a note about the verbosity level of --telemetry-url 2021-08-12 10:20:26 +01:00
James Wilson 80d6ad916e Address David's comments 2021-08-11 17:23:22 +01:00
James Wilson f26b39ac63 Address feedback from Niklas 2021-08-11 16:59:11 +01:00
James Wilson f089ad1758 Tweak CI with fix from Arsham 2021-08-11 12:42:53 +01:00
James Wilson 9f76fabaed give tokio threads a more convenient name for monitoring purposes 2021-08-09 11:36:46 +01:00
James Wilson b22efc804a Fix comment typo 2021-08-09 10:56:01 +01:00
James Wilson 626fe95d89 1 aggregator loop by default for now 2021-08-09 10:13:10 +01:00
James Wilson c469ef8dfe make AggregatorSet close to zero cost when only 1 aggregator asked for 2021-08-09 10:09:17 +01:00
James Wilson c99cbee1e9 Allow multiple aggregator loops in the core to try and spread feed subscription/send cost out 2021-08-07 17:08:58 +01:00
James Wilson 9c001bdcfd Add a comment on core worker threads 2021-08-06 17:50:50 +01:00
James Wilson 8017d8b00f Cargo fmt 2021-08-06 17:49:19 +01:00
James Wilson 78ad7115e5 set sensible worker thread limits again but allow full control to caller 2021-08-06 17:49:04 +01:00
James Wilson 88c3db3562 cargo fmt 2021-08-06 17:44:26 +01:00
James Wilson 74cf55174e Use rayon to speed up subscribe message serializing 2021-08-06 17:42:03 +01:00
James Wilson 0788270756 Add 'subscribe' benchmark 2021-08-06 14:22:01 +01:00
James Wilson 2dae6c90ce Keep file handle limit comment in sync 2021-08-05 12:22:22 +01:00
James Wilson dbabd453e0 Remove feed message cap; it wasn't pulling its weight 2021-08-05 12:21:16 +01:00
James Wilson bc75ebb068 default core/shard worker threads to 8/4 respectively 2021-08-05 12:18:44 +01:00
James Wilson 4da299bf76 Tweak logging around connections opening and messages coming in 2021-08-05 12:11:46 +01:00
James Wilson acfe3d45d1 num-cpus to worker-threads; more accurate name 2021-08-05 11:31:47 +01:00
James Wilson 5c5c7e592a allow configuration of cpus in soak tests 2021-08-05 11:16:10 +01:00
James Wilson 759d28b1a2 add --num-cpus option 2021-08-05 11:11:23 +01:00
James Wilson 6221cbfd17 Dev tweaks for testing 2021-08-05 11:11:05 +01:00
James Wilson 7d00d72baa Small tweak to make it more obvious when feeds are disconnected 2021-08-04 10:52:53 +01:00
James Wilson db5284a3db Add a comment explaining realistic_soak_test and --release on all 2021-08-03 15:42:42 +01:00
James Wilson 5d81128e74 create more realistic soak test so that we can see a more representative UI in action 2021-08-03 15:40:20 +01:00
James Wilson f5df1ef68b Merge pull request #356 from arshamparity/helm 
Helm
 2021-08-02 14:18:33 +01:00
James Wilson 4b00ac79bb Point sahrd to correct port for core 2021-08-02 14:16:20 +01:00
James Wilson c2892286a6 Add a comment 2021-08-02 09:54:11 +01:00
James Wilson b79759a412 Tidy up close handling and allow either side to close 2021-08-02 09:50:31 +01:00
James Wilson 963ef576f3 add on_close to abstract and simplify ws_client socket closing 2021-08-02 09:30:29 +01:00
James Wilson 13afb96f93 Be more fussy with Soketto channel wrapper; Soketto might do otherwise, but we only explicitly close the connection when both ends of the channel handed out are closed (or we encounter a connection error) 2021-07-30 21:26:52 +01:00
James Wilson 18918e6d3e undo a previous change; close the connection if either side of the channel handed back is dropped/fails 2021-07-30 19:43:26 +01:00
James Wilson a5b04cb4f0 improve socket channel close handling, and test the node banning (roughly) 2021-07-30 18:58:10 +01:00
James Wilson 3f523178be Fix broken test (node being banned) and allow test config of new max-bytes option on shard 2021-07-30 16:55:06 +01:00
James Wilson a368553085 Add note and reduce max data per second default as we expect much less traffic than existing avg 2021-07-30 15:39:30 +01:00
James Wilson 0b1a6f45fa readme tweaks 2021-07-30 12:36:23 +01:00