* start working on building the real overseer
Unfortunately, this fails to compile right now due to an upstream
failure to compile which is probably brought on by a recent upgrade
to rustc v1.47.
* fill in AllSubsystems internal constructors
* replace fn make_metrics with Metrics::attempt_to_register
* update to account for #1740
* remove Metrics::register, rename Metrics::attempt_to_register
* add 'static bounds to real_overseer type params
* pass authority_discovery and network_service to real_overseer
It's not straightforwardly obvious that this is the best way to handle
the case when there is no authority discovery service, but it seems
to be the best option available at the moment.
* select a proper database configuration for the availability store db
* use subdirectory for av-store database path
* apply Basti's patch which avoids needing to parameterize everything on Block
* simplify path extraction
* get all tests to compile
* Fix Prometheus double-registry error
for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:
```rust
Some(registry) => Self::try_register(registry).map_err(|err| {
eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
err
}),
```
That pointed out where the registration was failing, which led to
this fix. The test still doesn't pass, but it now fails in a new
and different way!
* authorities must have authority discovery, but not necessarily overseer handlers
* fix broken SpawnedSubsystem impls
detailed logging determined that using the `Box::new` style of
future generation, the `self.run` method was never being called,
leading to dropped receivers / closed senders for those subsystems,
causing the overseer to shut down immediately.
This is not the final fix needed to get things working properly,
but it's a good start.
* use prometheus properly
Prometheus lets us register simple counters, which aren't very
interesting. It also allows us to register CounterVecs, which are.
With a CounterVec, you can provide a set of labels, which can
later be used to filter the counts.
We were using them wrong, though. This pattern was repeated in a
variety of places in the code:
```rust
// panics with an cardinality mismatch
let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
my_counter.with_label_values(&["succeeded"]).inc()
```
The problem is that the labels provided in the constructor are not
the set of legal values which can be annotated, but a set of individual
label names which can have individual, arbitrary values.
This commit fixes that.
* get av-store subsystem to actually run properly and not die on first signal
* typo fix: incomming -> incoming
* don't disable authority discovery in test nodes
* Fix rococo-v1 missing session keys
* Update node/core/av-store/Cargo.toml
* try dummying out av-store on non-full-nodes
* overseer and subsystems are required only for full nodes
* Reduce the amount of warnings on browser target
* Fix two more warnings
* InclusionInherent should actually have an Inherent module on rococo
* Ancestry: don't return genesis' parent hash
* Update Cargo.lock
* fix broken test
* update test script: specify chainspec as script argument
* Apply suggestions from code review
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* Update node/service/src/lib.rs
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
* node/service/src/lib: Return error via ? operator
* post-merge blues
* add is_collator flag
* prevent occasional av-store test panic
* simplify fix; expand application
* run authority_discovery in Role::Discover when collating
* distinguish between proposer closed channel errors
* add IsCollator enum, remove is_collator CLI flag
* improve formatting
* remove nop loop
* Fix some stuff
* Adds test parachain adder collator
* Add sudo to Rococo, change session length to 30 seconds and some renaming
* Update to the latest changes on master
* Some fixes
* Fix compilation
* Update parachain/test-parachains/adder/collator/src/lib.rs
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* Review comments
* Downgrade transaction version
* Fixes
* MOARE
* Register notification protocols
* utils: remove unused error
* av-store: more resilient to some errors
* address review nits
* address more review nits
Co-authored-by: Peter Goodspeed-Niklaus <peter.r.goodspeedniklaus@gmail.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
Co-authored-by: Robert Habermeier <robert@Roberts-MBP.lan1>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Sergey Shulepov <s.pepyakin@gmail.com>
Co-authored-by: Sergei Shulepov <sergei@parity.io>
* DMP: data structures and plumbing
* DMP: Implement DMP logic in the router module
DMP: Integrate DMP parts into the inclusion module
* DMP: Introduce the max size limit for the size of a downward message
* DMP: Runtime API for accessing inbound messages
* OCD
Small clean ups
* DMP: fix the naming of the error
* DMP: add caution about a non-existent recipient
* update primitives
* correct parent_head field
* make hrmp field pub
* refactor validation data: runtime
* refactor validation data: messages
* add arguments to full_validation_data runtime API
* port runtime API
* mostly port over candidate validation
* remove some parameters from ValidationParams
* guide: update candidate validation
* update candidate outputs
* update ValidationOutputs in primitives
* port over candidate validation
* add a new test for no-transient behavior
* update util runtime API wrappers
* candidate backing
* fix missing imports
* change some fields of validation data around
* runtime API impl
* update candidate validation
* fix backing tests
* grumbles from review
* fix av-store tests
* fix some more crates
* fix provisioner tests
* fix availability distribution tests
* port collation-generation to new validation data
* fix overseer tests
* Update roadmap/implementers-guide/src/node/utility/candidate-validation.md
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
* Make parachain validation wasm executor functional
- Increase the size of the validation result in the shared memory. The
validation result holds the new runtime when a runtime upgrade is
scheduled. So, we need to give it enough memory to send the data between
the validator and the wasm execution host.
- Add the `CallInWasmExt`. This is required when doing a runtime upgrade
to check that we upgrade to something meaningful.
* Update parachain/src/wasm_executor/mod.rs
* Update parachain/src/wasm_executor/mod.rs
Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
Co-authored-by: Nikolay Volf <nikvolf@gmail.com>
* update guide to reduce confusion and TODOs
* work from previous bitfield signing effort
There were large merge issues with the old bitfield signing PR, so
we're just copying all the work from that onto this and restarting.
Much of the existing work will be discarded because we now have better
tools available, but that's fine.
* start rewriting bitfield signing in terms of the util module
* implement construct_availability_bitvec
It's not an ideal implementation--we can make it much more concurrent--
but at least it compiles.
* implement the unimplemented portions of bitfield signing
* get core availability concurrently, not sequentially
* use sp-std instead of std for a parachain item
* resolve type inference failure caused by multiple From impls
* handle bitfield signing subsystem & Allmessages variant in overseer
* fix more multi-From inference issues
* more concisely handle overflow
Co-authored-by: Andronik Ordian <write@reusable.software>
* Revert "resolve type inference failure caused by multiple From impls"
This reverts commit 7fc77805de5e5074a1b01037f8d4e3919e03e0e1.
* Revert "fix more multi-From inference issues"
This reverts commit f14ffe589e20d664d8a900ed62f68b6fb844a514.
* impl From<i32> for ParaId
* handle another instance of AllSubsystems
* improve consistency when returning existing options
Co-authored-by: Andronik Ordian <write@reusable.software>
* Downward messages, the front-end.
* Move types around to make them accessible from Parachains
* Fix compilation
* Fix branch
* Make it compile for Cumulus
* Update the branch names
* Add default generic parameter
* Implement `Partialeq`
* Move upward messages into the `ValidationResult`
* Support disabling of the runtime api
* Update branch
* Adds support for handling downward messages
* Implement sending XCMP messages as up/downward messages
* service: update to latest ServiceBuilder changes
* Make it compile
* Initial commit
Forked at: ef2aa428d7
Parent branch: origin/master
* Update substrate branch to cecton-update-polkadot-substrate
* Update substrate & polkadot to cumulus-branch
* Reset branch
* Update primitives/src/parachain.rs
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* Update runtime/common/src/parachains.rs
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* Update runtime/common/src/parachains.rs
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* Minor fixes
* Fix wasm build
Co-authored-by: Gav Wood <gavin@parity.io>
Co-authored-by: André Silva <andre.beat@gmail.com>
Co-authored-by: Cecile Tonglet <cecile.tonglet@cecton.com>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* add new apis to externalities
* Update mod.rs
* update for new host function
* Rename unused
* cargo update -p sp-io
* Add whitelists
* Apply changes to polkadot runtime as well
Co-authored-by: kianenigma <kian.peymani@gmail.com>
* upgrade primitives to allow changing validation function
* set up storage schema for old parachains code
* fix compilation errors
* fix test compilation
* add some tests for past code meta
* most of the runtime logic for code upgrades
* implement old-code pruning
* add a couple tests
* clean up remaining TODOs
* add a whole bunch of tests for runtime functionality
* remove unused function
* fix runtime compilation
* extract some primitives to parachain crate
* add validation-code upgrades to validation params and result
* extend validation params with code upgrade fields
* provide maximums to validation params
* port test-parachains
* add a code-upgrader test-parachain and tests
* fix collator tests
* move test-parachains to own folder to work around compilation errors
* fix test compilation
* update the Cargo.lock
* fix parachains tests
* remove dbg! invocation
* use new pool in code-upgrader
* bump lockfile
* link TODO to issue
* Flag to force kusama runtime
* Chainspecs for kusama
* Polkadot config for westend
Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>
* network/src/legacy/gossip: Wrap GossipEngine in Arc Mutex & lock it on use
`GossipEngine` in itself has no need to be Send and Sync, given that it
does not rely on separately spawned background tasks anymore.
`RegisteredMessageValidator` needs to be `Send` and `Sync` due to the
inherited trait bounds from implementing `GossipService`. In addition
`RegisteredMessageValidator` derives `Clone`. Thereby `GossipEngine`
needs to be wrapped in an `Arc` and `Mutex` to keep the status quo.
* Needed fixes.
* Fixes
* Fixed build
* Fixed build w benchmarking CLI
* Fixed building tests
* Added --dev shortcut
Co-authored-by: arkpar <arkady.paronyan@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Max Inden <mail@max-inden.de>
* expunge legacy code from polkadot-network
* mostly rip out old legacy protocol from service
* ensure validation work is spawned by incoming messages
* decouple availabliity store from network logic; clean up data flow
* av_store: test helpers and use futures-abort
* update polkadot-validation to pass n_validators when submitting chunks
* fallible erasure-chunk fetching
* implement `ErasureNetworking` for new network prot
* API for registering availability store in network
* fully integrate new network service into service
* fix validation tests
* scaffolding for porting collator over to new network
* track connected validators' peer IDs and distribute collators' collations
* helper in network for fetching all checked statements
* fix adder-collator
* actually register notifications protocol
* Update service/src/lib.rs
* Make needed changes to service
* Merge two companion PRs.
- #880
- #881
* Some effort towards compilation
* Fix
* remove `NetworkSpecialization` references from network
* fix compilation errors in service and collator
* ensure protocol name is valid
* Fixes
* Fix
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Ashley <ashley.ruglys@gmail.com>
* encode the candidate statement as only the hash
* refactor CandidateReceipt and CollationInfo
* introduce an abridged candidate receipt type
* erasure coding stores candidate receipt
* store omitted data instead and introduce AvailableData type
* refactor availability-store schema
* tweak schema and APIs a bit more
* get availability-store tests passing
* accept AbridgedCandidateReceipt in `set_heads`
* change statement type in primitives to be hash-only
* fix parachains runtime tests
* fix bad merge
* rewrite validation pipeline
* remove evaluation module
* use abridged candidate hash as canonical
* statement table uses abridged candidate receipts
* kill availability_store::Data struct
* port shared table to new validation pipelines
* extract full validation pipeline to helper
* remove old validation pipeline from collation module
* polkadot-validation compiles
* polkadot-validation tests compile
* make local collation available in validation service
* port legacy network code
* polkadot-network fully ported
* network: ensure fresh statement is propagated
* remove pov_block_hash from LocalValidationData
* remove candidate_hash field from AttestedCandidate and update runtime
* port runtimes to new ParachainHost definition
* port over polkadot-collator
* fix test compilation
* better fix
* remove unrelated validation work dispatch fix
* address grumbles
* fix equality check
* WIP
* WIp
* Mostly get tests to compile
* Fix adder collator
* Remove more stuff
* Revert some changes to av store
* Fix av store tests
* Nitpicks
* Restore some things
* Small changes
* Remvoe unused error variants
* Make use of `runtime_interface` for parachain externalities
This also changes the encoding of the `ValidationResult` return value to
match the default encoding used in Substrate.
* Fix compilation for web
* Update `Cargo.lock`
* Include feedback
* Move proc macro
* Update parachain/src/lib.rs
Co-Authored-By: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
* rename crates as appropriate
* Rename to use master - easy to revert, but shows it's working
* Bump runtime version
* Bump version
* Bump Substrate again
* Rest of parathread draft implementation, parachain permissioning.
* Update Substrate
* Update Substrate again
* Integrate weight/fee stuff.
* Council
* Build fixes
* More fixes
* Minor additions
* fix some small errors
* Revert "fix some small errors"
This reverts commit 4fb52c82adfdaf3af98edfe36b280133bcd4f9d3.
* Merge fix.
* do_swap -> on_swap
* Update depdendency to polkadot-master
* Fix more merge problems
* Some patching of errors
* Fix storage closure
* Actually fix storage. It builds!
* Tests run... but not successfully.
* Add `run_to_block` to get parachains active to start
* More `run_to_block`
* Fix build
* Queue up changes to threads
* Move registration test
* Fix regsiter/deregister test
* Retry queue.
* Minor refactor
* Refactor to avoid heavy storage items
* Make tests pass
* remove para on deregister, add events
* Remove println
* Fix register/deregister parathread test
* fix merge
* Parathread can be activated test
* Test auction
* Add `Debtors` storage item
I considered putting the debtor information in `ParaInfo`, but it did not make sense to me since this information only applies to parathreads, not `paras` in general.
* remove comment code
* Some new tests
* Fixes for removing threads when scheduled. Tests.
* Test progression of threads.
* Test that reschedule queuing works properly.
* Make test slightly more interesting
* whitespace
* Swap works properly.
* Update locks
* Build
* Rename can_swap
* Add test for funds to be correctly returned after a swap
Swap does not seem to have logic which correctly swaps the debtor account to the new parathread.
* Make tests consistant
* Add check that `PendingSwap` is cleaned up
* Update runtime/src/parachains.rs
Co-Authored-By: Robert Habermeier <rphmeier@gmail.com>
* Update runtime/src/registrar.rs
Co-Authored-By: Robert Habermeier <rphmeier@gmail.com>
* Some fixes/suggestions from review
* Docs
* Apply suggestions from code review
Co-Authored-By: Robert Habermeier <rphmeier@gmail.com>
Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>
* Update network/src/gossip.rs
Co-Authored-By: Robert Habermeier <rphmeier@gmail.com>
* Rename OnSwap
* Add missing `]`
* Rejig ordering semantics, making everything a bit slower but correct.
* Some Fixes to Parathread Compile (#470)
* Some Fixes
* Fix queue_upward_messages
* Change back to const
* Build fixes
* Fix tests
Bastian Köcher