Commit Graph

732 Commits

Author SHA1 Message Date
Alexander Samusev cd63276d31 [ci] Update rust to 1.74 (#2545)
cc https://github.com/paritytech/ci_cd/issues/900

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Dónal Murray <donal.murray@parity.io>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
2023-12-07 09:57:43 +01:00
André Silva 1f023deab8 sp-core: Rename VrfOutput to VrfPreOutput (#2534)
This will make more sense after
https://github.com/paritytech/polkadot-sdk/pull/2524 since the
schnorrkel type for VRF outputs is also renamed in the latest version.
Can be reviewed independently though.

Can be merged after https://github.com/paritytech/polkadot-sdk/pull/1577
so that there is less pain for @davxy.

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-12-06 02:19:56 +02:00
Marcin S c046a9d5ed PVF: Add Secure Validator Mode (#2486)
Co-authored-by: Javier Viola <javier@parity.io>
2023-12-05 14:32:56 +02:00
Liam Aharon 4a293bc5a2 Enforce consistent and correct toml formatting (#2518)
Using taplo, fixes all our broken and inconsistent toml formatting and
adds CI to keep them tidy.

If people want we can customise the format rules as described here
https://taplo.tamasfe.dev/configuration/formatter-options.html

@ggwpez, I suggest zepter is used only for checking features are
propagated, and leave formatting for taplo to avoid duplicate work and
conflicts.

TODO
- [x] Use `exclude = [...]` syntax in taplo file to ignore zombienet
tests instead of deleting the dir

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
2023-12-01 07:38:02 +00:00
Marcin S dbd8d20b25 PVF: Add test instructions (#2058) 2023-11-28 14:23:25 +01:00
Marcin S cfa19c37e6 PVF: remove audit log access (#2461) 2023-11-25 18:03:58 +02:00
Julian Eager e3242d2c1e Adapt test worker to profile flag (#2450)
closes #2194 

cc @mrcnski

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-11-24 12:32:43 +02:00
Michal Kucharczyk 49874882cf polkadot-node-subsystems: ChainApiBackend added + polkadot-debug image version fixed (#2411)
The out-dated version (bad tag) of [polkadot
image](https://github.com/paritytech/polkadot-sdk/blob/ede4a362622dfa69eb167eaa876246b1289f4b41/.gitlab/pipeline/zombienet/cumulus.yml#L31)
([docker
info](https://hub.docker.com/layers/paritypr/polkadot-debug/master/images/sha256:adb1658052cf671b50c90d5cece5c7a131efa1a95978249bd5cb85a5ad654f7a?context=explore))
was used. This PR fixes this.

See also:
https://github.com/paritytech/polkadot-sdk/pull/2411#issuecomment-1822632724

Also adds an abstraction that allows asynchronous backends to be passed to `ChainApiSubsystem`
---------

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
2023-11-22 19:24:11 +01:00
Marcin S 408af9b32d PVF: Fix unshare "no such file or directory" error (#2426) 2023-11-22 15:45:52 +01:00
Marcin S 552be4800d PVF worker: switch on seccomp networking restrictions (#2221) 2023-11-21 12:52:46 +01:00
Julian Eager 06190498a0 Refactor ValidationError (#2406) 2023-11-21 12:10:59 +01:00
jserrat ede4a36262 remove retry from backers on failed candidate validation (#2182)
Hey guys, as discussed I've changed the name to a more general one
`PvfExecKind`, is this good or too general?
Creating this as a draft, I still have to fix the tests.

Closes #1585

Kusama address: FkB6QEo8VnV3oifugNj5NeVG3Mvq1zFbrUu4P5YwRoe5mQN

---------

Co-authored-by: command-bot <>
Co-authored-by: Marcin S <marcin@realemail.net>
2023-11-20 13:00:19 +02:00
Julian Eager b5858936e1 Preserve artifact cache unless stale (#1918)
Co-authored-by: Marcin S <marcin@realemail.net>
2023-11-19 19:04:22 +01:00
Alin Dima fc12f435e3 add NodeFeatures field to HostConfiguration and runtime API (#2177)
Adds a `NodeFeatures` bitfield value to the runtime `HostConfiguration`,
with the purpose of coordinating the enabling of node-side features,
such as: https://github.com/paritytech/polkadot-sdk/issues/628 and
https://github.com/paritytech/polkadot-sdk/issues/598.
These are features that require all validators enable them at the same
time, assuming all/most nodes have upgraded their node versions.

This PR doesn't add any feature yet. These are coming in future PRs.

Also adds a runtime API for querying the state of the client features
and an extrinsic for setting/unsetting a feature by its index in the bitfield.

Note: originally part of:
https://github.com/paritytech/polkadot-sdk/pull/1644, but posted as
standalone to be reused by other PRs until the initial PR is merged
2023-11-14 20:48:32 +02:00
jserrat 54f84285bf change prepare worker to use fork instead of threads (#1685)
Co-authored-by: Marcin S <marcin@realemail.net>
2023-11-14 18:50:18 +01:00
Marcin S 7cfc233cdc PVF: fix detection of unshare-and-change-root security capability (#2304) 2023-11-14 15:03:19 +01:00
Marcin S 5f4ce80266 PVF host: Make unavailable security features print a warning (#2244)
Co-authored-by: Bastian Köcher <git@kchr.de>
2023-11-13 11:21:16 +01:00
Lulu 03ee44d9e1 Add license to tracking-allocator and add staging-prefix (#2261)
The staging- rename commit was missing from the last PR for some reason.
2023-11-09 22:27:08 +00:00
Oliver Tale-Yazdi 48ea86f0e8 Add descriptions to all published crates (#2029)
Missing descriptions (47):  

- [x] `cumulus/client/collator/Cargo.toml`
- [x] `cumulus/client/relay-chain-inprocess-interface/Cargo.toml`
- [x] `cumulus/client/cli/Cargo.toml`
- [x] `cumulus/client/service/Cargo.toml`
- [x] `cumulus/client/relay-chain-rpc-interface/Cargo.toml`
- [x] `cumulus/client/relay-chain-interface/Cargo.toml`
- [x] `cumulus/client/relay-chain-minimal-node/Cargo.toml`
- [x] `cumulus/parachains/pallets/parachain-info/Cargo.toml`
- [x] `cumulus/parachains/pallets/ping/Cargo.toml`
- [x] `cumulus/primitives/utility/Cargo.toml`
- [x] `cumulus/primitives/aura/Cargo.toml`
- [x] `cumulus/primitives/core/Cargo.toml`
- [x] `cumulus/primitives/parachain-inherent/Cargo.toml`
- [x] `cumulus/test/relay-sproof-builder/Cargo.toml`
- [x] `cumulus/pallets/xcmp-queue/Cargo.toml`
- [x] `cumulus/pallets/dmp-queue/Cargo.toml`
- [x] `cumulus/pallets/xcm/Cargo.toml`
- [x] `polkadot/erasure-coding/Cargo.toml`
- [x] `polkadot/statement-table/Cargo.toml`
- [x] `polkadot/primitives/Cargo.toml`
- [x] `polkadot/rpc/Cargo.toml`
- [x] `polkadot/node/service/Cargo.toml`
- [x] `polkadot/node/core/parachains-inherent/Cargo.toml`
- [x] `polkadot/node/core/approval-voting/Cargo.toml`
- [x] `polkadot/node/core/dispute-coordinator/Cargo.toml`
- [x] `polkadot/node/core/av-store/Cargo.toml`
- [x] `polkadot/node/core/chain-api/Cargo.toml`
- [x] `polkadot/node/core/prospective-parachains/Cargo.toml`
- [x] `polkadot/node/core/backing/Cargo.toml`
- [x] `polkadot/node/core/provisioner/Cargo.toml`
- [x] `polkadot/node/core/runtime-api/Cargo.toml`
- [x] `polkadot/node/core/bitfield-signing/Cargo.toml`
- [x] `polkadot/node/network/dispute-distribution/Cargo.toml`
- [x] `polkadot/node/network/bridge/Cargo.toml`
- [x] `polkadot/node/network/collator-protocol/Cargo.toml`
- [x] `polkadot/node/network/approval-distribution/Cargo.toml`
- [x] `polkadot/node/network/availability-distribution/Cargo.toml`
- [x] `polkadot/node/network/bitfield-distribution/Cargo.toml`
- [x] `polkadot/node/network/gossip-support/Cargo.toml`
- [x] `polkadot/node/network/availability-recovery/Cargo.toml`
- [x] `polkadot/node/collation-generation/Cargo.toml`
- [x] `polkadot/node/overseer/Cargo.toml`
- [x] `polkadot/runtime/parachains/Cargo.toml`
- [x] `polkadot/runtime/common/slot_range_helper/Cargo.toml`
- [x] `polkadot/runtime/metrics/Cargo.toml`
- [x] `polkadot/xcm/pallet-xcm-benchmarks/Cargo.toml`
- [x] `polkadot/utils/generate-bags/Cargo.toml`
- [x]  `substrate/bin/minimal/runtime/Cargo.toml`

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: alindima <alin@parity.io>
Co-authored-by: ordian <noreply@reusable.software>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Co-authored-by: Marcin S <marcin@realemail.net>
Co-authored-by: alindima <alin@parity.io>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
2023-11-09 17:37:28 +02:00
s0me0ne-unkn0wn 50390950d8 Refactor candidate validation messages (#2219) 2023-11-08 15:21:58 +01:00
Andrei Sandu 0570b6fa9e approval-voting improvement: include all tranche0 assignments in one certificate (#1178)
**_PR migrated from https://github.com/paritytech/polkadot/pull/6782_** 

This PR will upgrade the network protocol to version 3 -> VStaging which
will later be renamed to V3. This version introduces a new kind of
assignment certificate that will be used for tranche0 assignments.
Instead of issuing/importing one tranche0 assignment per candidate,
there will be just one certificate per relay chain block per validator.
However, we will not be sending out the new assignment certificates,
yet. So everything should work exactly as before. Once the majority of
the validators have been upgraded to the new protocol version we will
enable the new certificates (starting at a specific relay chain block)
with a new client update.

There are still a few things that need to be done:

- [x] Use bitfield instead of Vec<CandidateIndex>:
https://github.com/paritytech/polkadot/pull/6802
  - [x] Fix existing approval-distribution and approval-voting tests
  - [x] Fix bitfield-distribution and statement-distribution tests
  - [x] Fix network bridge tests
  - [x] Implement todos in the code
  - [x] Add tests to cover new code
  - [x] Update metrics
  - [x] Remove the approval distribution aggression levels: TBD PR
  - [x] Parachains DB migration 
  - [x] Test network protocol upgrade on Versi
  - [x] Versi Load test
  - [x] Add Zombienet test
  - [x] Documentation updates
- [x] Fix for sending DistributeAssignment for each candidate claimed by
a v2 assignment (warning: Importing locally an already known assignment)
 - [x]  Fix AcceptedDuplicate
 - [x] Fix DB migration so that we can still keep old data.
 - [x] Final Versi burn in

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
2023-11-06 15:21:32 +02:00
s0me0ne-unkn0wn cd2d5d2579 Tracking/limiting memory allocator (#1192) 2023-11-03 17:48:41 +02:00
Julian Eager 4f05f9a686 Build workers for testing on demand (#2018) 2023-11-01 21:20:53 +01:00
jserrat 2726d5af65 remove gum dependency on jaeger (#2106)
Co-authored-by: Marcin S <marcin@realemail.net>
2023-11-01 15:58:46 +01:00
Marcin S 9faea380dc PVF worker: Add seccomp restrictions (restrict networking) (#2009) 2023-10-31 11:08:08 +01:00
Alexandru Gheorghe 3069b0af39 make polkadot die graciously (#2056)
While investigating some db migrations that make the node startup fail,
I noticed that the node wasn't exiting and that the log file were
growing exponentially, until my whole system was freezing and that makes
it really hard to actually find why it was failing in the first place.

E.g:
```
 ls -lh /tmp/zombie-01a04c2a2c0265d85f6440cf01c0f44a_-51319-uyggzuD4wEpV/bob.log
 32,6G oct 27 11:16 /tmp/zombie-01a04c2a2c0265d85f6440cf01c0f44a_-51319-uyggzuD4wEpV/bob.log
```

This was happening because the following errors were being printed
continously without the subsystem main loop exiting:

From dispute-coordinator:
```
WARN tokio-runtime-worker parachain::dispute-coordinator: error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
```

From availability recovery:
```
Erasure task channel closed. Node shutting down ?
```

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
2023-10-27 13:50:30 +02:00
Marcin S e39253c022 PVF: Add worker check during tests and benches (#1771) 2023-10-24 16:22:15 +02:00
ordian 5ca909cc09 polkadot: eradicate LeafStatus (#1565)
Fixes #768.
2023-10-23 16:22:37 +02:00
asynchronous rob a46183c706 Vstaging statement distribution omnibus (#1436)
in-progress PR adding new tests and solving bugs

---------

Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com>
Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: eskimor <eskimor@users.noreply.github.com>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
2023-10-21 11:01:14 +02:00
Julian Eager 9e1447042b Include polkadot version in artifact path (#1828)
closes #695

Could potentially be helpful to preserving caches when applicable, as
discussed in #685

kusama address: FvpsvV1GQAAbwqX6oyRjemgdKV11QU5bXsMg9xsonD1FLGK
2023-10-15 10:39:03 +02:00
Julian Eager 9f7656df15 Discard Executor (#1855)
closes #622 

Pros:
* simpler interface, just functions:
`create_runtime_from_artifact_bytes()` and `execute_artifact()`

Cons:
* extra overhead of constructing executor semantics each time

I could make it a combination of
* `create_runtime_config(params)` (such that we could clone the
constructed semantics)
* `create_runtime(blob, config)`
* `execute_artifact(blob, config, params)`

Not sure if it's worth it though.

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2023-10-14 23:06:00 +02:00
Julian Eager 681e7bbfb2 Check executor params coherence (#1774)
Co-authored-by: Marcin S <marcin@realemail.net>
2023-10-13 15:52:04 +02:00
Anton Vilhelm Ásgeirsson d2fc1d7c91 Fix links to implementers' guide (#1865)
# Description

In a couple of cases, there were links pointing to the w3f github pages
domain. In other instances, there were links pointing to the old
polkadot repo's github pages. Both of these are now pointing to the
relevant links in
https://paritytech.github.io/polkadot-sdk/book/index.html.

These changes were made specifically because the w3f github pages
returns a 404, and while fixing the links, the old polkadot repo links
were touched up as well even if they do redirect properly.

This shouldn't affect anything as these are documentation link changes
only.
2023-10-12 21:29:10 +02:00
Tsvetomir Dimitrov 7aace06b3d Disabled validators runtime API (#1257)
Exposes disabled validators list via a runtime API.

---------

Co-authored-by: ordian <noreply@reusable.software>
Co-authored-by: ordian <write@reusable.software>
2023-10-12 15:01:07 +02:00
Mira Ressel 447e753323 ci: bump ci image to rust 1.73.0 (#1830)
Co-authored-by: command-bot <>
2023-10-11 19:49:59 +02:00
Marcin S 132ba0c89f PVF worker: bump landlock, update ABI docs (#1850) 2023-10-11 16:13:07 +02:00
Marcin S 51c0c24213 PVF: Add back socket path parameter, use tmp socket path (#1780) 2023-10-05 18:37:54 +02:00
Marcin S c1eb342b14 PVF: more filesystem sandboxing (#1373) 2023-09-28 18:24:29 +02:00
ordian 769bdd3ff3 runtime-api: cleanup after v7 stabilization (#1729)
Follow-up to #1543.
2023-09-28 01:07:32 +02:00
Chris Sosnin 7cbe0c76ef Migrate polkadot-primitives to v6 (#1543)
- Async-backing related primitives are stable `primitives::v6`
- Async-backing API is now part of `api_version(7)`
- It's enabled on Rococo and Westend runtimes

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
2023-09-27 13:32:02 +03:00
yjh e38998801e Executor: Remove LegacyInstanceReuse strategy (#1486)
It seems the old strategy have been depracted more than one year. 
So maybe it's time to clean up old strategy for wasm executor.


---
polkadot address: 15ouFh2SHpGbHtDPsJ6cXQfes9Cx1gEFnJJsJVqPGzBSTudr

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Koute <koute@users.noreply.github.com>
2023-09-18 14:53:06 +09:00
Tsvetomir Dimitrov d81a2e2fa9 Fix code duplication in tests (#1575) 2023-09-14 22:49:37 +03:00
Maciej 756347ab24 Inclusion pruning tweaks (#1550)
In follow-up to https://github.com/paritytech/polkadot-sdk/pull/1518

Adding extra tests for inclusion pruning. Primarily focusing on various
cases surrounding candidates included in different forks (with different
relay parents).

All cases fall into a few buckets based on 3 degrees of freedom - number
of candidates, number of blocks (height), number of forks + extra case
for pruning multiple heights at once.

Added small tweak to the original pruning function to disregard stale
candidate duplicates which should keep the same behaviour.

---------

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
2023-09-14 20:40:19 +01:00
Maciej f7c95c5fc1 Inclusions struct unit tests (#1518)
In follow-up to https://github.com/paritytech/polkadot-sdk/pull/1432

Some additional unit tests for the inclusion struct used in the scraper.
2023-09-13 11:46:33 +01:00
Bastian Köcher e005aef59b Make the node version independent of the crate version (#1495) 2023-09-12 14:12:10 +02:00
Tsvetomir Dimitrov d08170278b Additional logging for dispute-coordinator (#1494) 2023-09-12 09:49:48 +03:00
s0me0ne-unkn0wn 2c8021f998 Retire puppet workers (#1449)
Closes #583

After the separation of PVF worker binaries, dedicated puppet workers
are not needed for tests anymore. The production workers can be used
instead, avoiding some code duplication and decreasing complexity.

The changes also make it possible to further refactor the code to
isolate workers completely.
2023-09-11 19:14:07 +02:00
eskimor 5b3f659ce3 Remove redundant book keeping in dispute coordinator. (#1432)
* Remove redundant book keeping in dispute coordinator.

* Fix docs.

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
2023-09-11 12:25:40 +03:00
dependabot[bot] bce7c2465c Bump hex-literal from 0.3.4 to 0.4.1 (#1438)
Bumps [hex-literal](https://github.com/RustCrypto/utils) from 0.3.4 to 0.4.1.
- [Commits](https://github.com/RustCrypto/utils/compare/hex-literal-v0.3.4...hex-literal-v0.4.1)

---
updated-dependencies:
- dependency-name: hex-literal
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
2023-09-08 12:50:56 +03:00
ordian 15503883e2 polkadot: pin one block per session (#1220)
* polkadot: propagate UnpinHandle to ActiveLeafUpdate

Also extract the leaf creation for tests
into a common function.

* dispute-coordinator: try pinned blocks for slashin

* apparently 1.72 is smarter than 1.70

* address nits

* rename fresh_leaf to new_leaf
2023-09-07 13:24:40 +03:00