* Happy New Year!
* Remove year entierly
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Remove years from copyright notice in the entire repo
---------
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
* Re-apply changes without Diener, rebase to the lastest master
* Cache pruning
* Bit-pack InstantiationStrategy
* Move ExecutorParams version inside the structure itself
* Rework runtime API and executor parameters storage
* Pass executor parameters through backing subsystem
* Update Cargo.lock
* Introduce `ExecutorParams` to approval voting subsys
* Introduce `ExecutorParams` to dispute coordinator
* `cargo fmt`
* Simplify requests from backing subsys
* Fix tests
* Replace manual config cloning with `.clone()`
* Move constants to module
* Parametrize executor performing PVF pre-check
* Fix Malus
* Fix test runtime
* Introduce session executor params as a constant defined by session info
pallet
* Use Parity SCALE codec instead of hand-crafted binary encoding
* Get rid of constants; Add docs
* Get rid of constants
* Minor typo
* Fix Malus after rebase
* `cargo fmt`
* Use transparent SCALE encoding instead of explicit
* Clean up
* Get rid of relay parent to session index mapping
* Join environment type and version in a single enum element
* Use default execution parameters if running an old runtime
* `unwrap()` -> `expect()`
* Correct API version
* Constants are back in town
* Use constants for execution environment types
* Artifact separation, first try
* Get rid of explicit version
* PVF execution queue worker separation
* Worker handshake
* Global renaming
* Minor fixes resolving discussions
* Two-stage requesting of executor params to make use of runtime API cache
* Proper error handling in pvf-checker
* Executor params storage bootstrapping
* Propagate migration to v3 network runtimes
* Fix storage versioning
* Ensure `ExecutorParams` serialization determinism; Add comments
* Rename constants to make things a bit more deterministic
Get rid of stale code
* Tidy up a structure of active PVFs
* Minor formatting
* Fix comment
* Add try-runtime hooks
* Add storage version write on upgrade
Co-authored-by: Andronik <write@reusable.software>
* Add pre- and post-upgrade assertions
* Require to specify environment type; Remove redundant `impl`s
* Add `ExecutorParamHash` creation from `H256`
* Fix candidate validation subsys tests
* Return splittable error from executor params request fn
* Revert "Return splittable error from executor params request fn"
This reverts commit a0b274177d8bb2f6e13c066741892ecd2e72a456.
* Decompose approval voting metrics
* Use more relevant errors
* Minor formatting fix
* Assert a valid environment type instead of checking
* Fix `try-runtime` hooks
* After-merge fixes
* Add migration logs
* Remove dead code
* Fix tests
* Fix tests
* Back to the strongly typed implementation
* Promote strong types to executor interface
* Remove stale comment
* Move executor params to `SessionInfo`: primitives and runtime
* Move executor params to `SessionInfo`: node
* Try to bump primitives and API version
* Get rid of `MallocSizeOf`
* Bump target API version to v4
* Make use of session index already in place
* Back to v3
* Fix all the tests
* Add migrations to all the runtimes
* Make use of existing `SessionInfo` in approval voting subsys
* Rename `TARGET` -> `LOG_TARGET`
* Bump all the primitives to v3
* Fix Rococo ParachainHost API version
* Use `RollingSessionWindow` to acquire `ExecutorParams` in disputes
* Fix nits from discussions; add comments
* Re-evaluate queue logic
* Rework job assignment in execution queue
* Add documentation
* Use `RuntimeInfo` to obtain `SessionInfo` (with blackjack and caching)
* Couple `Pvf` with `ExecutorParams` wherever possible
* Put members of `PvfWithExecutorParams` under `Arc` for cheap cloning
* Fix comment
* Fix CI tests
* Fix clippy warnings
* Address nits from discussions
* Add a placeholder for raw data
* Fix non exhaustive match
* Remove redundant reexports and fix imports
* Keep only necessary semantic features, as discussed
* Rework `RuntimeInfo` to support mock implementation for tests
* Remove unneeded bound
* `cargo fmt`
* Revert "Remove unneeded bound"
This reverts commit 932463f26b00ce290e1e61848eb9328632ef8a61.
* Fix PVF host tests
* Fix PVF checker tests
* Fix overseer declarations
* Simplify tests
* `MAX_KEEP_WAITING` timeout based on `BACKGING_EXECUTION_TIMEOUT`
* Add a unit test for varying executor parameters
* Minor fixes from discussions
* Add prechecking max. memory parameter (see paritytech/srlabs_findings#110)
* Fix and improve a test
* Remove `ExecutionEnvironment` and `RawData`
* New primitives versioning in parachain host API
* `disputes()` implementation for Kusama and Polkadot
* Move `ExecutorParams` from `vstaging` to stable primitives
* Move disputes from `vstaging` to stable implementation
* Fix `try-runtime`
* Fixes after merge
* Move `ExecutorParams` to the bottom of `SessionInfo`
* Revert "Move executor params to `SessionInfo`: primitives and runtime"
This reverts commit dfcfb85fefd1c5be6c8a8f72dc09fd1809cfa9ce.
* Always use fresh activated live hash in pvf precheck
(re-apply 34b09a4c20de17e7926ed942cd0d657d18f743fa)
* Fixing tests (broken commit)
* Fix candidate validation tests
* Fix PVF host test
* Minor fixes
* Address discussions
* Restore migration
* Fix `use` to only include what is needed instead of `*`
* Add comment to never touch `DEFAULT_CONFIG`
* Update migration to set default `ExecutorParams` for `dispute_period`
sessions back
* Use `earliest_stored_session` instead of calculations
* Nit
* Add logs
* Treat any runtime error as `NotSupported` again
* Always return default executor params if not available
* Revert "Always return default executor params if not available"
This reverts commit b58ac4482ef444c67a9852d5776550d08e312f30.
* Add paritytech/substrate#9997 workaround
* `cargo fmt`
* Remove migration (again!)
* Bump executor params to API v4 (backport from #6698)
---------
Co-authored-by: Andronik <write@reusable.software>
* foo
* rolling session window
* fixup
* remove use statemetn
* fmt
* split NetworkBridge into two subsystems
Pending cleanup
* split
* chore: reexport OrchestraError as OverseerError
* chore: silence warnings
* fixup tests
* chore: add default timenout of 30s to subsystem test helper ctx handle
* single item channel
* fixins
* fmt
* cleanup
* remove dead code
* remove sync bounds again
* wire up shared state
* deal with some FIXMEs
* use distinct tags
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* use tag
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* address naming
tx and rx are common in networking and also have an implicit meaning regarding networking
compared to incoming and outgoing which are already used with subsystems themselvesq
* remove unused sync oracle
* remove unneeded state
* fix tests
* chore: fmt
* do not try to register twice
* leak Metrics type
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
Co-authored-by: Andronik <write@reusable.software>
This issue happens when some peer sends a good but already known Seconded statement and the statement-distribution code does not update the statements_received field in the peer_knowledge structure. Subsequently, a Valid statement causes out-of-view message that is incorrectly emitted and causes reputation lose.
This PR also introduces a concept of passing the specific pseudo-random generator to subsystems to make it easier to write deterministic tests. This functionality is not really necessary for the specific issue and unit test but it can be useful for other tests and subsystems.
* split metrics from bitfield signing
* cleanup all logging
* add a unit test for subset generation
* chore: add one more test to assert need is properly represented
* u8 as usize
* chore: overseer fixin
* fix test
* Update node/network/bitfield-distribution/src/metrics.rs
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* Update node/network/bitfield-distribution/src/metrics.rs
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* fallout from suggested rename
* consistency
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
* remove v0 primitives from polkadot-primitives
* first pass: remove v0
* fix fallout in erasure-coding
* remove v1 primitives, consolidate to v2
* the great import update
* update runtime_api_impl_v1 to v2 as well
* guide: add `Version` request for runtime API
* add version query to runtime API
* reintroduce OldV1SessionInfo in a limited way
* parity db subsystem without cache and no splitted column
* fmt
* fix path (auto from parity-db fail)
* lru cache for db column with cache
* Revert "lru cache for db column with cache"
This reverts commit ae177bc5e107a075eff6a21f651218ada6599b74.
* Write_lock mutex
* theoric code for bridges
* revert changes
* Revert bridge changes
* fix spec_version
* update parity db
* test purge-db
* Use specific ordered collection with paritydb.
* Revert "Use specific ordered collection with paritydb."
This reverts commit 8b66d0a4ae914cba1af0f44050d45dd6d9327c6b.
* fix chain selection tests.
* remove patch
* fix auto.
* Remove useless exists directory method
* purge chain without parity-db removal
* spellcheck
* renamings and filtering.
* fix assertion
* format
* update parity-db and fmt
* Auto keep using rocksdb when it exists.
* Revert "Auto keep using rocksdb when it exists."
This reverts commit cea49b32ae590bdce31fed5c45f3c028ae0c7564.
* Update kvdb version.
* seed commit for fatality based errors
* fatality
* first draft of fatality
* cleanup
* differnt approach
* simplify
* first working version for enums, with documentation
* add split
* fix simple split test case
* extend README.md
* update fatality impl
* make tests passed
* apply fatality to first subsystem
* fatality fixes
* use fatality in a subsystem
* fix subsystemg
* fixup proc macro
* fix/test: log::*! do not execute when log handler is missing
* fix spelling
* rename Runtime2 to something sane
* allow nested split with `forward` annotations
* add free license
* enable and fixup all tests
* use external fatality
Makes this more reviewable.
* bump fatality dep
Avoid duplicate expander compilations.
* migrate availability distribution
* more fatality usage
* chore: bump fatality to 0.0.6
* fixup remaining subsystems
* chore: fmt
* make cargo spellcheck happy
* remove single instance of `#[fatal(false)]`
* last quality sweep
* fixup
* remove Default from CandidateHash
* Apply suggestions from code review
Co-authored-by: Andronik Ordian <write@reusable.software>
* chore: fmt
* remove backed candidate default
* Partial migration away from CandidateReceipt::default
* Remove more CandidateReceipt defaults
* fmt
* Mostly remove CommittedCandidateReceipt default usage
* Remove CommittedCandidateReceipt
* Remove more Defaults from polakdot primitives v1 + fmt
* Remove more Default from polkadot primites v1
* WIP trying to get overseer example + tests to compile
* feat: add primitives test helpers
* reduce deps of helper
* update primitive helpers
* make candidate validation compile
* fixup cargo lock
* make av-store compile
* fixup disputes coordinator tests
* test: fixup backing
* test: fixup approval voting
* fixup bitfield signing
* test: fixup runtime-api
* test: fixup availability dist
* foxi[ pverseer test]
* remove some Defaults, remove bounds from `dummy`
All `fn dummy` in primitives need to be removed anyways.
This aids in the transition.
* it's a test helper, so always use std
* test: fixup parachains runtime tests
Excluding benches.
* fix keyring
* fix paras runtime properly, no more default
* Remove fn dummy() usage from approval voting
* Move TestCandidateBuilder out of av store to test helpers
* Make candidate validation tests pass
* Make most dispute coirdinator tests pass
* Make provisioner tests work
* Make availability recovery tests work with test helpers
* Update polkadot-collator-protocol tests
* Update statement distribution tests
* Update polkadot overseer examples and tests
* Derive default for validation code so we don't break unrelated things
* Make para runtime test pass (no bench)
* Some more work
* chore: cargo fmt
* cargo fix
* avoid some Default::default
* fixup dispute coordinator test
* remove unused crate deps
* remove Default::default wherever possible, replace by dummy_* for the most part
* chore: cargo fmt
* Remove some warnings
* Remove CommittedCandidateReceipt dummy
* Remove CandidateReceipt dummy
* Remove CandidateDescriptor dummy
* Remove commented out code
* Fix para runtime tests
* chore: nightly
* Some updates to the builder
* Dynamically adjust mock head data size
* Make dispute cooridinator tests work
* Fix test candidate_backing_reorders_votes work
* +nightly-2021-10-29 fmt
* Spelling and remove a default use in builder
* Various clean up
* More small updates
* fmt
* More small updates
* Doc comments for test helpers
* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras_inherent.rs
* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras_inherent.rs
* Update lib.rs
* review comments
* fix warnings
* fix test by using correct candidate receipt relay parent
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: emostov <32168567+emostov@users.noreply.github.com>
Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Gavin Wood <gavin@parity.io>
Right now, most of operations that sign stuff in polkadot protocol are
handled by a very convenient tool - `Signed`. However `Signed` assumes
that whatever is signed is anchored to some `parent_hash` which works
for most cases, but does not work for others.
One instance of such a case is pre-checking (#3211). There validators
submit signed votes on-chain. A vote is valid for the entire session. If
we were to use `Signed` we would have to root a vote in some block of
that session and during vote verification check that this block is
indeed within the session. This is especially annoying since we agreed
to use unsigned extrinsics to submit votes and we need to make the
unsigned extrinsic validation as slim as possible.
(FWIW, the definition of a pre-checking vote can be seen in the next
diff in the stack)
That's the reason why we opted-out from using `Signed` for pre-checking
and decided to go with the manual signing approach. Almost every piece
of machinery is in place except for signing which is presented in this
PR.
* test/malus: craft the first maliciously disputing actor
* initial draft
* Add Dockerfile and instructions how to use it to build malus image locally
* Forgot one flag for the build cmd
* we are not docker specific, we are happy to use any containerruntime
* shuffle things around
* add initial tera based integration test
* chores
* fixins
* simple setup to start
* other samples (WIP)
* add Docker version with cargo-chef
* update substarte and small change of orders in commands in the container file
* metrics one
* fmt
* minor
* fixin
* fix metric names
* -d
* add open gauge
* fmt
* spellcheck
* fix test
* adjust to changed error messages
* refactor, more malus impls
* more malus changes
* foo
* minor cleanup
* suggest garbage candidate
* chore
* fix suggest garabge malus
* malus: back garbage candidate
* cargo lock
* re-introduce metrics
* chore: cargo fmt
* undoe 1.54.0 output, CI uses 1.53.0 rustc
* update location of js types
* Fix trybuild
* add tag to image name also; this will be replaced in the prod version
* Tests fixed
* add some fix me
* add dockerfile for ci
* Add docker file for malus for ci
* use variables in .toml file
* add chnages for malus test
* some fixes
* some more fixes
* Update .gitlab-ci.yml
* add local build for polkadot and malus
* some fixes
* enable disputes feature in CI
* ok, ok
* rename: MsgFilter -> MessageInterceptor
* remove TODO that would not have worked
* intercept
* refactor
* fix README and containers
* fix
* chore: cargo fmt
* avoid some more malus-$VARIANT references
* fix argument order
* chore: add about
* Update sanity check in relay chain selection
* fix order, add dispute-unavailable-block malus
* fixup: avoid underflow issue
* it's all u32
* fix conditional use
* Revert "it's all u32"
This reverts commit 6b3ae25bfd0bbf0b51d90d743642a75a4a815d6e.
* Revert "fixup: avoid underflow issue"
This reverts commit 336cbe2938e9720f870d37d8feeab7ca69200f47.
* Revert "Update sanity check in relay chain selection"
This reverts commit 970647f35e1116136e12fd91cd9f2fb7e18ad28d.
* update the malus bin
* Update node/malus/integrationtests/0003-dispute-unavailable-block.feature
Co-authored-by: Andronik Ordian <write@reusable.software>
* add some FIXME reminders
* update path to index.js
* Update .gitlab-ci.yml
* Update node/malus/integrationtests/0001-dispute-valid-block.toml
* try 1: make malus test run
* chore: cargo fmt
* temporary fix
* use subcommand syntax from latest gurke
* cargo +nightly fmt
* add collator to a a test
* docs: add env vars to README
* update ci to run dispute-valid-block test
* needs the polkadot image
* Fix path for nodejs container
* post merge fix
* download proper dir containg configs for malus test
* update the malus ci job
* rm a whitespace
* temp build for malus
* use correct build command for temp malus
* remove subcommands for now
* set max validators per core in the default HostConfig
* tabs
* update beefy
* fixup
* fixup II
* make one variant compile
* make other variants compile
* revert changes to chain_spec
* fmt
* build malus image from polkadot-test-malus again
* revert unrelated changes
* try fixing build-malus job
* Revert "remove subcommands for now"
This reverts commit 5d8292bc49252124937affec4b7c28181a5deb7e.
* try fixing build-malus job II
* MVP working dispute-ancestor
* renames
* fix PVF execution on malus
* fix test
* fix typo
* fmt
* checkmate
* try something
* make it actually work
* some tweaks to 01 feature test
* fmt
* sleep a bit more
* complete wococoization
* some tweaks to 01 feature test
* typo fix
* use correct metric names
* fix
* ffs
* .
* try some rearrangement
* Attempt to wait till initial node bootstrap in test
* Fix test syntax
* Run malus tests with v2 script
* Proper symlink created
* simnet v14
* add zombienet tests
* add zombie net test - draft
* add more tests to dispute suite
* add within to fix tests
* replace test directory and start test migration
* migrate all the tests
* add timeout to tests
* reduce debug
* make easy to test in dev
* set appropriated debug
* use image from ci
* fix config for test
* set images from ci
* fix config
* add COLIMAGE env
* tweek tests
* reduce debug
* typo
* wip, migrate old test to zombie-net
* adjunt test config for zombie-net
* run mauls 0001 test only
* clean and setup smoke-test in zombie-net
* add extra time to assertinons
* clean code to merge and improve README
* add info to access logs
* improved readme
* merge master and resolve conflicts
* Update zombienet_tests/README.md
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
* clean and consolidate zombienet name
* change runner in gitlab
* add comment explain why we use wococo
* change tag for runner
* remove unused tests
* remove dup Dockerfile and update description
* fmt
* fix compilation post-merge
* fmt
* cut me Some slack
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: radupopa2010 <radupopa2010@yahoo.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Anton Gavrilov <AntonE.Gavrilov@gmail.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com>
* Mostly notes.
* Better error messages.
* Introduce Fatal/NonFatal + drop back channel participation
- Fatal/NonFatal - in order to make it easier to use utility functions.
- We drop the back channel in dispute participation as it won't be
needed any more.
* Better error messages.
* Utility function for receiving `CandidateEvent`s.
* Ordering module typechecks.
* cargo fmt
* Prepare spam slots module.
* Implement SpamSlots mechanism.
* Implement queues.
* cargo fmt
* Participation.
* Participation taking shape.
* Finish participation.
* cargo fmt
* Cleanup.
* WIP: Cleanup + Integration.
* Make `RollingSessionWindow` initialized by default.
* Make approval voting typecheck.
* Get rid of lazy_static & fix approval voting tests
* Move `SessionWindowSize` to node primitives.
* Implement dispute coordinator initialization.
* cargo fmt
* Make queues return error instead of boolean.
* Initialized: WIP
* Introduce chain api for getting finalized block.
* Fix ordering to only prune candidates on finalized events.
* Pruning of old sessions in spam slots.
* New import logic.
* Make everything typecheck.
* Fix warnings.
* Get rid of obsolete dispute-participation.
* Fixes.
* Add back accidentelly deleted Cargo.lock
* Deliver disputes in an ordered fashion.
* Add module docs for errors
* Use type synonym.
* hidden docs.
* Fix overseer tests.
* Ordering provider taking `CandidateReceipt`.
... To be kicked on one next commit.
* Fix ordering to use relay_parent
as included block is not unique per candidate.
* Add comment in ordering.rs.
* Take care of duplicate entries in queues.
* Better spam slots.
* Review remarks + docs.
* Fix db tests.
* Participation tests.
* Also scrape votes on first leaf for good measure.
* Make tests typecheck.
* Spelling.
* Only participate in actual disputes, not on every import.
* Don't account backing votes to spam slots.
* Fix more tests.
* Don't participate if we don't have keys.
* Fix tests, typos and warnings.
* Fix merge error.
* Spelling fixes.
* Add missing docs.
* Queue tests.
* More tests.
* Add metrics + don't short circuit import.
* Basic test for ordering provider.
* Import fix.
* Remove dead link.
* One more dead link.
Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com>
* Introduce new Runtime API endpoint
`persisted_validation_data_with_code_hash` that will be used
by the candidate validation subsystem in order to decrease amount
of runtime API requests.
* Node-side part of new runtime API request
* Define code hash getter via macro
* Rename new endpoint to `assumed_validation_data`
* Docs for runtime API impl of new endpoint
* AssumedValidationData specialized request function
* fmt
* Make SessionInfo include all authorities
of the current session.
* Add missing import.
* Only take subset for assignment keys.
* Fix typo.
* Make spellcheck happy.
* Really.
* Fix test.
* More clear documentation.
* Update comments in `SessionInfo`.
Andrei Sandu