fe19a75414
Un-brittle-ify backend E2E tests and have them run by default again in CI ( #397 )
...
* test running tests
* Add delay so that core knows about node before feed subscribes to limit chance of race
* move delaye to the right place
* Don't do expensive docker step untill we will be pushing the image
* docker test skipped as hoped, so push to 'true'
* just remove docker steps in github CI entirely since they aren't needed by anything (gitlab CI does this stuff now)
* run CI on pull requests too to catch PRs from forks
2021-09-06 11:30:17 +01:00
a3ffaf3c44
Flume fix part 2: avoid using flume in a couple of cases, and revert attempted fix#1 ( #396 )
...
* A second attempt to avoid the flume memory leak, since the first didn't actually work
* No luck with second flume fix, so revert to futures::mpsc in a few places)
* cargo fmt
* Add a comment to cover use of into_stream
2021-09-03 15:55:25 +01:00
2932075783
Avoid using flume::Receiver::into_stream() to avoid memory leaks until the issue is resolved upstream ( #394 )
...
* Tweak rolling_total test to also confirm capacity doesn't go nuts
* Use Jemalloc
* Avoid flume's into_stream and use a workaround for now
* cargo fmt
* Improve comments now that there's an issue to point to
2021-09-03 08:40:43 +01:00
a4069e4b3d
Subscribe to chains by genesis hash ( #395 )
...
* Handle subscription by hash in the frontend
* Forward-ported backend changes
* Fix unit tests
* Remove unused `chains_by_label`
* fmt
* Updated but failing E2E tests
* subscribe by genesis hash in tests
* fmt
* Copy `BlockHash` instead of returning a ref
* Pin chains by genesisHash
Co-authored-by: James Wilson <james@jsdw.me >
2021-09-02 17:54:19 +02:00
ec5db0fbbf
Bump tokio to 1.10 and add a test to confirm memory usage of rolling_total ( #392 )
2021-08-31 20:18:46 +02:00
87866b2d42
Improve logging and error reporting around IP and location info ( #386 )
...
* Beef up error reporting of IP and location info
* Tidy up error reporting after some manual testing of it
* Don't cache erroneous locations; try again when asked again
* cargo fmt
2021-08-27 16:16:26 +01:00
7a3e30cb01
Don't remove all feeds subscribed to a chain when one disconnects ( #383 )
...
* Only remove the feed that disconnected to not break the rest...
* use multimap struct to avoid sync issues between feed and chain
* add a remove test, too
* cargo fmt
* fix name of test
* move multimap to common so we can doctest it and add 'unique' to name
* cargo fmt
* Return old key if value moved to make uniqueness more obvious
2021-08-27 08:05:44 +01:00
19db1a48ef
Hardening of the Backend docker image ( #379 )
...
* Add script to build the backend
* harden the backend docker image
* fix docker-compose
* fix doc
2021-08-26 14:32:11 +02:00
46b0641dfd
Clarify wording
2021-08-13 11:45:03 +01:00
77460ffc27
cargo fmt
2021-08-13 11:35:24 +01:00
b842c7fc8b
expose dropped message counts and fix some typos/wording
2021-08-13 11:33:53 +01:00
811babca27
Merge branch 'master' into jsdw-sharding-gatekeeper
2021-08-13 11:16:47 +01:00
18627a9f02
No e2e feature flag; just ignore and pattern match on 'e2e' to run
2021-08-12 16:58:35 +01:00
05a3ba3fef
Fix/expand a few comments
2021-08-12 16:20:05 +01:00
230987036a
cargo fmt
2021-08-12 16:01:35 +01:00
9017f328f0
Add comment explaining prometheus metrics endpoint body
2021-08-12 16:01:33 +01:00
92da674d4d
Expose metrics in a format that prometheus understands
2021-08-12 16:01:32 +01:00
4f7b2c8ec5
Confirm that densemap len wont panic if lots of retired items
2021-08-12 16:01:30 +01:00
6db7f484ef
Fix compile err with diagnostic msg
2021-08-12 16:01:29 +01:00
ab2303ce5c
more diagnostic logging
2021-08-12 16:01:27 +01:00
3319709f7b
Add periodic interval to core loop and print debug info
2021-08-12 16:01:25 +01:00
f72f8c1fd5
test runner: fix soak test for multiple ids per ndoe
2021-08-12 16:01:24 +01:00
20463ce159
test runner: enable tokio features
2021-08-12 16:01:19 +01:00
e3fcd4e8c2
Clean up soak test runner and add more config options
2021-08-12 16:01:18 +01:00
bd7a21ec39
Flumify everything
2021-08-12 16:01:17 +01:00
11b0b3a3c7
remove final use of futures::mpsc and replace with flume
2021-08-12 16:01:15 +01:00
703a9ddc4e
use flume throughout telemetry_core
2021-08-12 16:01:14 +01:00
8268cf2afe
print feed 1 msg len
2021-08-12 16:01:12 +01:00
98c9ccd278
fmt, clean warnings, tidy aggregator opts and add queue length limit
2021-08-12 16:01:11 +01:00
968dd2b957
Try to force new thread for msg counter to ensure it has time to print
2021-08-12 16:01:09 +01:00
b97aec99a8
monitoring queue len
2021-08-12 16:01:07 +01:00
87c0ee7d0d
monitor aggregator length (dont discard msgs yet)
2021-08-12 16:00:56 +01:00
d4b5c2b0c8
split e2e tests out and run them separately, not blocking the build or marking it as failed
2021-08-12 13:38:29 +01:00
7aa4ad49b3
bump the msg timeout a little higher
2021-08-12 13:10:17 +01:00
7563909609
Fix typo
...
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com >
2021-08-12 12:59:39 +01:00
770dd04b57
invert logic to make name make sense and fix comment typo
2021-08-12 12:47:29 +01:00
f887510beb
remove lint warning on cargo test
2021-08-12 12:42:44 +01:00
4480bbe72a
Allow errors as well as closes for now to remove some brittleness
2021-08-12 12:40:53 +01:00
80d6ad916e
Address David's comments
2021-08-11 17:23:22 +01:00
f26b39ac63
Address feedback from Niklas
2021-08-11 16:59:11 +01:00
9f76fabaed
give tokio threads a more convenient name for monitoring purposes
2021-08-09 11:36:46 +01:00
b22efc804a
Fix comment typo
2021-08-09 10:56:01 +01:00
626fe95d89
1 aggregator loop by default for now
2021-08-09 10:13:10 +01:00
c469ef8dfe
make AggregatorSet close to zero cost when only 1 aggregator asked for
2021-08-09 10:09:17 +01:00
c99cbee1e9
Allow multiple aggregator loops in the core to try and spread feed subscription/send cost out
2021-08-07 17:08:58 +01:00
9c001bdcfd
Add a comment on core worker threads
2021-08-06 17:50:50 +01:00
8017d8b00f
Cargo fmt
2021-08-06 17:49:19 +01:00
78ad7115e5
set sensible worker thread limits again but allow full control to caller
2021-08-06 17:49:04 +01:00
88c3db3562
cargo fmt
2021-08-06 17:44:26 +01:00
74cf55174e
Use rayon to speed up subscribe message serializing
2021-08-06 17:42:03 +01:00
James Wilson