* Compress the PoV block before sending it over the network
This pr changes the way we send PoV blocks over the network. We now
compress the PoV block before it is send over the network. This should
reduce the size significant for PoVs which contain the runtime WASM for
example.
* Preallocate 1KB
* Try something..
* Switch to zstd and some renamings
* Make compression/decompression fail in browsers
* Use some sane maximum value
* Update roadmap/implementers-guide/src/types/network.md
Co-authored-by: Andronik Ordian <write@reusable.software>
* Fix and add test
* add
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* reexport prometheus-super for ease of use of other subsystems
* add some prometheus timers for collation generation subsystem
* add timing metrics to av-store
* add metrics to candidate backing
* add timing metric to bitfield signing
* add timing metrics to candidate selection
* add timing metrics to candidate-validation
* add timing metrics to chain-api
* add timing metrics to provisioner
* add timing metrics to runtime-api
* add timing metrics to availability-distribution
* add timing metrics to bitfield-distribution
* add timing metrics to collator protocol: collator side
* add timing metrics to collator protocol: validator side
* fix candidate validation test failures
* add timing metrics to pov distribution
* add timing metrics to statement-distribution
* use substrate_prometheus_endpoint prometheus reexport instead of prometheus_super
* don't include JOB_DELAY in bitfield-signing metrics
* give adder-collator ability to easily export its genesis-state and validation code
* wip: adder-collator pushbutton script
* don't attempt to register the adder-collator automatically
Instead, get these values with
```sh
target/release/adder-collator export-genesis-state
target/release/adder-collator export-genesis-wasm
```
And then register the parachain on https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9944#/explorer
To collect prometheus data, after running the script, create `prometheus.yml` per the instructions
at https://www.notion.so/paritytechnologies/Setting-up-Prometheus-locally-835cb3a9df7541a781c381006252b5ff
and then run:
```sh
docker run -v `pwd`/prometheus.yml:/etc/prometheus/prometheus.yml:z --network host prom/prometheus
```
Demonstrates that data makes it across to prometheus, though it is likely to be useful in the future
to tweak the buckets.
* Update parachain/test-parachains/adder/collator/src/cli.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* use the grandpa-pause parameter
* skip metrics in tracing instrumentation
* remove unnecessary grandpa_pause cli param
Co-authored-by: Andronik Ordian <write@reusable.software>
* WIP: add initial check_extrinsics_ordering.sh script
* iterate through runtimes, add gitlab job
* move job to publish
* temp force build-linux-release to run
* update check_extrinsics_ordering.sh
* maybe we have to fetch release
* use node docker image
* revert before opening pr: force bad extrinsic ordering
* revert commits to prepare for PR
* move job to build stage, use bin from test-linux-release
* remove FIXME
* fix PR nags
* Revert "Restrict collating_using_adder_collator test on feature real-overseer (#1947)"
This reverts commit 75da335d66.
* Revert "Revert "Restrict collating_using_adder_collator test on feature real-overseer (#1947)""
This reverts commit 4e92cdfb1046b0e3da7627297050ae71ae3dd6e3.
* Test something
* Adds integration test based on adder collator
This adds an integration test for parachains that uses the adder
collator. The test will start two relay chain nodes and one collator and
waits until 4 blocks are build and enacted by the parachain.
* Make sure the integration test is run in CI
* Fix wasm compilation
* Update parachain/test-parachains/adder/collator/src/lib.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* Update cli/src/command.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* start working on building the real overseer
Unfortunately, this fails to compile right now due to an upstream
failure to compile which is probably brought on by a recent upgrade
to rustc v1.47.
* fill in AllSubsystems internal constructors
* replace fn make_metrics with Metrics::attempt_to_register
* update to account for #1740
* remove Metrics::register, rename Metrics::attempt_to_register
* add 'static bounds to real_overseer type params
* pass authority_discovery and network_service to real_overseer
It's not straightforwardly obvious that this is the best way to handle
the case when there is no authority discovery service, but it seems
to be the best option available at the moment.
* select a proper database configuration for the availability store db
* use subdirectory for av-store database path
* apply Basti's patch which avoids needing to parameterize everything on Block
* simplify path extraction
* get all tests to compile
* Fix Prometheus double-registry error
for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:
```rust
Some(registry) => Self::try_register(registry).map_err(|err| {
eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
err
}),
```
That pointed out where the registration was failing, which led to
this fix. The test still doesn't pass, but it now fails in a new
and different way!
* authorities must have authority discovery, but not necessarily overseer handlers
* fix broken SpawnedSubsystem impls
detailed logging determined that using the `Box::new` style of
future generation, the `self.run` method was never being called,
leading to dropped receivers / closed senders for those subsystems,
causing the overseer to shut down immediately.
This is not the final fix needed to get things working properly,
but it's a good start.
* use prometheus properly
Prometheus lets us register simple counters, which aren't very
interesting. It also allows us to register CounterVecs, which are.
With a CounterVec, you can provide a set of labels, which can
later be used to filter the counts.
We were using them wrong, though. This pattern was repeated in a
variety of places in the code:
```rust
// panics with an cardinality mismatch
let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
my_counter.with_label_values(&["succeeded"]).inc()
```
The problem is that the labels provided in the constructor are not
the set of legal values which can be annotated, but a set of individual
label names which can have individual, arbitrary values.
This commit fixes that.
* get av-store subsystem to actually run properly and not die on first signal
* typo fix: incomming -> incoming
* don't disable authority discovery in test nodes
* Fix rococo-v1 missing session keys
* Update node/core/av-store/Cargo.toml
* try dummying out av-store on non-full-nodes
* overseer and subsystems are required only for full nodes
* Reduce the amount of warnings on browser target
* Fix two more warnings
* InclusionInherent should actually have an Inherent module on rococo
* Ancestry: don't return genesis' parent hash
* Update Cargo.lock
* fix broken test
* update test script: specify chainspec as script argument
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update node/service/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* node/service/src/lib: Return error via ? operator
* post-merge blues
* add is_collator flag
* prevent occasional av-store test panic
* simplify fix; expand application
* run authority_discovery in Role::Discover when collating
* distinguish between proposer closed channel errors
* add IsCollator enum, remove is_collator CLI flag
* improve formatting
* remove nop loop
* Fix some stuff
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
Co-authored-by: Robert Habermeier <robert@Roberts-MBP.lan1>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Max Inden <mail@max-inden.de>
* initial build-wasms commit
* fix wasm builds
* add caching and artifact upload
* test for git dir
* Revert "test for git dir"
This reverts commit 295e4655484691aed9162f479a87f09c1b1c9317.
* fix git version incompability
* attempt to fix caching
* Revert "fix git version incompability"
This reverts commit 1d22ffd26b7eff635c5abcc9bd04cb3c88629a1d.
* add kusama job
* use a build matrix for building runtimes
* combine build-wasms and publish draft release
* attempt to name runtimes according to version
* Revert "attempt to name runtimes according to version"
This reverts commit 82f7b4dc0aea995f2599293bba9f11b879f459e5.
Failed experiment
* name runtimes according to version #2
* fix asset path and name
* build wasms first, include prop hash
* Remove Sudo
NOTE: To ensure minimal index changes to pre-existing pallet deployments,
this is done with a "swap_remove" style; the previous last pallet
(Purchase), which is hitherto unused, has been shifted into the old index
of Sudo.
* Remove CC1 designation.
* Fixes
* Bump
* Fixes
* Fixes
* Fixes
* Fixes
* Fixes
* Fixes
* Fixes
* Fixes
* Fixes
* Fixes
* Initial check_runtime.sh redux
* Skip further checks if no change to runtime
* remove polkadot-master reference
* Fix issues found during review :)
* replace static common dir with common_dirs array
* Refactor publish_draft_release.sh
* Switch to using arrays for storing labelled changes
* Combine changes for Polkadot and Substrate
* Change sanitised_git_logs in `lib.sh` to drop `*`s at the start
* Only look for priorities of medium or above (presence of one or more
C1 label is enforeced by check-labels anyway, saves us some API calls)
* Ensure priorities >C1-low aren't labelled B0-silent
* Enforces presence of C-labels to designate the importance of a release
* Iterates over them on a release, takes the highest present and includes it in the release notes
* Lists the change(s) that caused the release to be that priority
* Add westend spec version and check labels for polkadot
... And remove reference to `polkadot-master`
* Add changelog section for runtimenoteworthy labels
And change markdown nesting header style
* Move publish_draft_release to test stage
Also remove the alert_pending_release job since it's no longer needed
* delete alert_pending_release.sh
* Add splitting substrate changes into subsections
* CI: Split common functions into lib.sh and make...
... alert_pending_release.sh check the substrate changes for various
labels. If labelled, will be included in the changelog, otherwise will
silently be left out.
* replace non-standard /bin/bash with /usr/bin/env bash
* CI: make lib.sh /bin/sh-compatible
* lib.sh: fix newline chars sometimes breaking has_label()
* check_runtime.sh: tag change with 'B2-breaksapi'...
... if any substrate changes are also labelled 'B2-breaksapi'
* Remove debug exits and uncomment labelling
* exit early if we label with breaksapi
* fix publish_draft_release.sh
* CI: Split common functions into lib.sh and make...
... alert_pending_release.sh check the substrate changes for various
labels. If labelled, will be included in the changelog, otherwise will
silently be left out.
* replace non-standard /bin/bash with /usr/bin/env bash
* CI: make lib.sh /bin/sh-compatible
Bastian Köcher