* [WIP] PVF: Split out worker binaries
* Address compilation problems and re-design a bit
* Reorganize once more, fix tests
* Reformat with new nightly to make `cargo fmt` test happy
* Address `clippy` warnings
* Add temporary trace to debug zombienet tests
* Fix zombienet node upgrade test
* Fix malus and its CI
* Fix building worker binaries with malus
* More fixes for malus
* Remove unneeded cli subcommands
* Support placing auxiliary binaries to `/usr/libexec`
* Fix spelling
* Spelling
Co-authored-by: Marcin S. <marcin@realemail.net>
* Implement review comments (mostly nits)
* Fix worker node version flag
* Rework getting the worker paths
* Address a couple of review comments
* Minor restructuring
* Fix CI error
* Add tests for worker binaries detection
* Improve tests; try to fix CI
* Move workers module into separate file
* Try to fix failing test and workers not printing latest version
- Tests were not finding the worker binaries
- Workers were not being rebuilt when the version changed
- Made some errors easier to read
* Make a bunch of fixes
* Rebuild nodes on version change
* Fix more issues
* Fix tests
* Pass node version from node into dependencies to avoid recompiles
- [X] get version in CLI
- [X] pass it in to service
- [X] pass version along to PVF
- [X] remove rerun from service
- [X] add rerun to CLI
- [X] don’t rerun pvf/worker’s (these should be built by nodes which have rerun enabled)
* Some more improvements for smoother tests
- [X] Fix tests
- [X] Make puppet workers pass None for version and remove rerun
- [X] Make test collators self-contained
* Add back rerun to PVF workers
* Move worker binaries into files in cli crate
As a final optimization I've separated out each worker binary from its own crate
into the CLI crate. Before, the worker bin shared a crate with the worker lib,
so when the binaries got recompiled so did the libs and everything transitively
depending on the libs. This commit fixes this regression that was causing
recompiles after every commit.
* Fix bug (was passing worker version for node version)
* Move workers out of cli into root src/bin/ dir
- [X] Pass in node version from top-level (polkadot)
- [X] Add build.rs with rerun-git-head to root dir
* Add some sanity checks for workers to dockerfiles
* Update malus
+ [X] Make it self-contained
+ [X] Undo multiple binary changes
* Try to fix clippy errors
* Address `cargo run` issue
- [X] Add default-run for polkadot
- [X] Add note about installation to error
* Update readme (installation instructions)
* Allow disabling external workers for local/testing setups
+ [X] cli flag to enable single-binary mode
+ [X] Add message to error
* Revert unnecessary Cargo.lock changes
* Remove unnecessary build scripts from collators
* Add back missing malus commands (should fix failing ZN job)
* Some minor fixes
* Update Cargo.lock
* Fix some build errors
* Undo self-contained binaries; cli flag to disable version check
+ [X] Remove --dont-run-external-workers
+ [X] Add --disable-worker-version-check
+ [X] Remove PVF subcommands
+ [X] Redo malus changes
* Try to fix failing job and add some docs for local tests
---------
Co-authored-by: Dmitry Sinyavin <dmitry.sinyavin@parity.io>
Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com>
Co-authored-by: parity-processbot <>
* Happy New Year!
* Remove year entierly
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Remove years from copyright notice in the entire repo
---------
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Introduce jemalloc-stats feature flag
* remove unneeded space
* Update node/overseer/src/lib.rs
Co-authored-by: Marcin S. <marcin@bytedude.com>
* Update Cargo.toml
Co-authored-by: Marcin S. <marcin@bytedude.com>
* revert making tikv-jemallocator depend on jemalloc-stats
* conditionally import memory_stats instead of using dead_code
* fix test via expllicit import
* Add jemalloc-stats feature to crates, propagate it from root
* Apply `jemalloc-stats` feature to prepare mem stats; small refactor
* effect changes recommended on PR
* Update node/overseer/src/metrics.rs
Co-authored-by: Marcin S. <marcin@bytedude.com>
* fix compile error on in pipeline for linux. missing import
* Update node/overseer/src/lib.rs
Co-authored-by: Bastian Köcher <git@kchr.de>
* revert to defining collect_memory_stats inline
---------
Co-authored-by: Marcin S. <marcin@bytedude.com>
Co-authored-by: Marcin S <marcin@realemail.net>
Co-authored-by: Bastian Köcher <git@kchr.de>
* polkadot-service: Make native runtime configurable
This pull requests adds support for configuring the native runtimes used
by polkadot-service. While this whole pr doesn't change that much for
polkadot, besides not having the light-node enabled for the default
polkadot binary. However, downstream projects (parachains) will have a
much better compile time. In cumulus for example the `cargo test --all
--release` is about 4m faster to compile.
* Fixes
* Fix
* Enable rococo-native
* Fix light client
* 🤦
* Fixes
* Allow both polkadot and kusama runtimes
* Allow both polkadot and kusama runtimes
* Make `collator` build
* Removed kusama runtime
* Introduced common runtime
* Updated for latest substrate
* Updated CI targets
* Updated CI version check
* Removed unused dependency
* Pulled latests substrate
* Pulled latest substrate
* Fixed version
* Apply suggestions from code review
Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>
* NEW_HEADS_IDENTIFIER moved to primitives
* Updated CI check script
* Fixed script
* Set epoch duration for polkadot
* ci: check_runtime for both runtimes
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: gabriel klawitter <gabreal@users.noreply.github.com>
* Make availability-store compile for WASM
* Use --manifest-path instead
* Make validation work on wasm!
* Switch to Spawn trait
* Migrate validation to std futures
* Migrate network to std futures
* Final changes to validation
* Tidy up network
* Tidy up validation
* Switch branch
* Migrate service
* Get polkadot to compile via wasm!
* Add browser-demo
* Add initial browser file
* Add browser-demo
* Tidy
* Temp switch back to substrate/master
* tidy
* Fix wasm build
* Re-add release flag
* Switch to polkadot-master
* Revert cli tokio version to avoid libp2p panic
* Update tokio version
* Fix availability store tests
* Fix validation tests
* Remove futures01 from availability-store
* Fix network tests
* Small changes
* Fix collator
* Fix typo
* Revert removal of tokio_executor that causes tokio version mismatch panic
* Fix adder test parachain
* Revert "Revert removal of tokio_executor that causes tokio version mismatch panic"
This reverts commit cfeb50c01d8df5e209483406a711e64761b44ae9.
* Update availability-store/src/worker.rs
Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>
* Update network/src/lib.rs
Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>
* Update network/src/lib.rs
Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>
* Box pin changes
* Asyncify network functions
* Clean up browser validation worker error
* Fix av store test
* Nits
* Fix validation test
* Switch favicon
* Fix validation test again
* Revert "Asyncify network functions"
This reverts commit f20ae6548dc482cb1e75bc80641cfe55c6131a53.
* Add async blocks back in
* refactor out a consensus data fetcher from table router
* move statement checking logic into router
* refuse to start authority if collator
* support building the table router asynchronously
* instantiate_consensus does not overwrite old
* update key in new consensus if there was none before
* collator collects ingress from network
* test produced egress roots
* fix adder-collator compilation
* address first grumbles
* integrate new gossip with collator network launch
* address review
* add issue link
* add issue link
* add issue link
* Update README.adoc
* add parens
* add issue link
* add issue link
* add issue link
* add issue link
* add issue link
* add issue link
* add issue link
* Fix up wasm runtime build
* Fixes for runtime
* Fix.
* More fixes
* Runtime builds on native.
* Native and wasm both build without warnings.
* Fix runtime tests.
* Merge #20
* Final fix for native runtime.
* Compile polkadot wo consensus
* Reverted changes to polkadot-consensus
* reintroduce minimal subset of consensus
* reintroduce checked_block to runtime for std
* polkadot_consensus compiles without most of the code
* remove checked_block again and do more checks in parachains for runtime
* uncomment proposer
* remove offline tracker
* extract out parachain-attestation logic from proposal directly
* reintroduce transaction_pool
* write some custom aura verification logic for the block verifier
* use transaction pool in more generic way
* service compiles again
* polkadot-network and tests pass
* remove unused session_key function from router
* everything but CLI compiles due to service hell
* Fixes compilation of `polkadot_cli`
* everything compiles
* update adder wasm
* Moved chain ops to substrate-service
* moved RPC to susbstrate-service WIP
* Moved RPC to substrate-service
* Version
* Removed redundadnt messages
* substrate CLI
* Removed obsolete files
* Sorted out startup messages
* Pass executable name to CLI
* Substrate service
* Splitting polkadot service
* Specialised components
* Specialised components
* Docs and style
* Docs and style
* Final touches
* Added db key assertion
* arbitrary application logic in CLI
* collation work
* split up exit and work futures in application
* collation node workflow
* typo
* indentation fix
* doc grumbles
* rename Application to Worker
* refactor Worker::exit to exit_only
* CLI options and keystore integration
* Replace multiqueue with future::mpsc
* BFT gossip
* Revert to app_dirs
* generate_from_seed commented
* Refactor event loop
* Start consensus by timer
* Message buffering
* Minor fixes
* Work around duty-roster issue.
* some more minor fixes
* fix compilation
* more consistent formatting
* make bft input stream never conclude
* Minor fixes
* add timestamp module to executive
* more cleanups and logging
* Fixed message propagation
s0me0ne-unkn0wn