Commit Graph

786 Commits

Author SHA1 Message Date
Tsvetomir Dimitrov 59f868d1e9 Deprecate para_id() from CoreState in polkadot primitives (#3979)
With Coretime enabled we can no longer assume there is a static 1:1
mapping between core index and para id. This mapping should be obtained
from the scheduler/claimqueue on block by block basis.

This PR modifies `para_id()` (from `CoreState`) to return the scheduled
`ParaId` for occupied cores and removes its usages in the code.

Closes https://github.com/paritytech/polkadot-sdk/issues/3948

---------

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
2024-04-08 05:58:12 +00:00
HongKuang bd4471b4fc Fix some typos (#4018)
Signed-off-by: hongkuang <liurenhong@outlook.com>
2024-04-08 04:21:11 +00:00
divdeploy a0eed0a6f8 chore: fix some comments (#4004)
Signed-off-by: divdeploy <chenguangxue@outlook.com>
2024-04-05 18:35:57 +00:00
Serban Iorga 8e95a3e1aa Align dependencies with parity-bridges-common (#3937)
Working towards migrating the `parity-bridges-common` repo inside
`polkadot-sdk`. This PR upgrades some dependencies in order to align
them with the versions used in `parity-bridges-common`

Related to
https://github.com/paritytech/parity-bridges-common/issues/2538
2024-04-02 13:41:01 +00:00
Alexandru Gheorghe d6f68bb906 primitives: Move out of staging released APIs (#3925)
Runtime release 1.2 includes bumping of the ParachainHost APIs up to
v10, so let's move all the released APIs out of vstaging folder, this PR
does not include any logic changes only renaming of the modules and some
moving around.

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
2024-04-01 13:03:26 +00:00
Alin Dima 6a0859ebf7 bugfix: request fragment tree membership for all candidates (#3874) 2024-03-28 13:14:17 +00:00
ordian 3fc5b82653 fix regression in approval-voting introduced in #3747 (#3831)
Fixes #3826.

The docs on the `candidates` field of `BlockEntry` were incorrectly
stating that they are sorted by core index. The (incorrect) optimization
was introduced in #3747 based on this assumption. The actual ordering is
based on `CandidateIncluded` events ordering in the runtime. We revert
this optimization here.

- [x] verify the underlying issue
- [x] add a regression test

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
2024-03-26 22:59:47 +00:00
Dcompoze 002d9260f9 Fix spelling mistakes across the whole repository (#3808)
**Update:** Pushed additional changes based on the review comments.

**This pull request fixes various spelling mistakes in this
repository.**

Most of the changes are contained in the first **3** commits:

- `Fix spelling mistakes in comments and docs`

- `Fix spelling mistakes in test names`

- `Fix spelling mistakes in error messages, panic messages, logs and
tracing`

Other source code spelling mistakes are separated into individual
commits for easier reviewing:

- `Fix the spelling of 'authority'`

- `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'`

- `Fix the spelling of 'prev_enqueud_messages'`

- `Fix the spelling of 'endpoint'`

- `Fix the spelling of 'children'`

- `Fix the spelling of 'PenpalSiblingSovereignAccount'`

- `Fix the spelling of 'PenpalSudoAccount'`

- `Fix the spelling of 'insufficient'`

- `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'`

- `Fix the spelling of 'subtracted'`

- `Fix the spelling of 'CandidatePendingAvailability'`

- `Fix the spelling of 'exclusive'`

- `Fix the spelling of 'until'`

- `Fix the spelling of 'discriminator'`

- `Fix the spelling of 'nonexistent'`

- `Fix the spelling of 'subsystem'`

- `Fix the spelling of 'indices'`

- `Fix the spelling of 'committed'`

- `Fix the spelling of 'topology'`

- `Fix the spelling of 'response'`

- `Fix the spelling of 'beneficiary'`

- `Fix the spelling of 'formatted'`

- `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'`

- `Fix the spelling of 'succeeded'`

- `Fix the spelling of 'reopened'`

- `Fix the spelling of 'proposer'`

- `Fix the spelling of 'InstantiationNonce'`

- `Fix the spelling of 'depositor'`

- `Fix the spelling of 'expiration'`

- `Fix the spelling of 'phantom'`

- `Fix the spelling of 'AggregatedKeyValue'`

- `Fix the spelling of 'randomness'`

- `Fix the spelling of 'defendant'`

- `Fix the spelling of 'AquaticMammal'`

- `Fix the spelling of 'transactions'`

- `Fix the spelling of 'PassingTracingSubscriber'`

- `Fix the spelling of 'TxSignaturePayload'`

- `Fix the spelling of 'versioning'`

- `Fix the spelling of 'descendant'`

- `Fix the spelling of 'overridden'`

- `Fix the spelling of 'network'`

Let me know if this structure is adequate.

**Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`,
`Merkelization`, `Merkleization`, is somewhat inconsistent but I left it
as it is.

~~**Note:** In some places the term `Receival` is used to refer to
message reception, IMO `Reception` is the correct word here, but I left
it as it is.~~

~~**Note:** In some places the term `Overlayed` is used instead of the
more acceptable version `Overlaid` but I also left it as it is.~~

~~**Note:** In some places the term `Applyable` is used instead of the
correct version `Applicable` but I also left it as it is.~~

**Note:** Some usage of British vs American english e.g. `judgement` vs
`judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc.
are both present in different places, but I suppose that's
understandable given the number of contributors.

~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it
triggers errors in CI when I make changes to it, so I left it as it
is.~~
2024-03-26 13:57:57 +00:00
Alin Dima c6f7ccf51a elastic scaling: preserve candidate ordering in provisioner (#3778)
https://github.com/paritytech/polkadot-sdk/issues/3742
2024-03-25 13:38:13 +00:00
Serban Iorga 0711729d25 [Bridges] Move chain definitions to separate folder (#3822)
Related to
https://github.com/paritytech/parity-bridges-common/issues/2538

This PR doesn't contain any functional changes. 

The PR moves specific bridged chain definitions from
`bridges/primitives` to `bridges/chains` folder in order to facilitate
the migration of the `parity-bridges-repo` into `polkadot-sdk` as
discussed in https://hackmd.io/LprWjZ0bQXKpFeveYHIRXw?view

Apart from this it also includes some cosmetic changes to some
`Cargo.toml` files as a result of running `diener workspacify`.
2024-03-25 12:11:30 +00:00
ordian 64a707a482 approval-voting: remove some inefficiences on startup (#3747)
Small refactoring to reduce the algorithmic complexity of the initial
message distribution in approval voting after a sync from O(n_candidates
^ 2) to O(n_candidates).
2024-03-21 11:22:16 +00:00
Tsvetomir Dimitrov e58e854a32 Expose ClaimQueue via a runtime api and use it in collation-generation (#3580)
The PR adds two things:
1. Runtime API exposing the whole claim queue
2. Consumes the API in `collation-generation` to fetch the next
scheduled `ParaEntry` for an occupied core.

Related to https://github.com/paritytech/polkadot-sdk/issues/1797
2024-03-20 06:55:58 +00:00
ordian 02e1a7f476 collator protocol changes for elastic scaling (validator side) (#3302)
Fixes #3128.

This introduces a new variant for the collation response from the
collator that includes the parent head data. For now, collators won't
send this new variant. We'll need to change the collator side of the
collator protocol to detect all the cores assigned to a para and send
the parent head data in the case when it's more than 1 core.

- [x] validate approach
- [x] check head data hash
2024-03-15 19:43:28 +00:00
Alexandru Gheorghe 1ead59773e Add api-name in cannot query the runtime API version warning (#3653)
Sometimes we see nodes printing this warning:
```
cannot query the runtime API version: Api called for an unknown Block: State already discarded for
```

The log is harmless, but let's print the api we got this for, so that we
can track its call site and truly confirm it is harmless or fix it.

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
2024-03-12 08:33:54 +00:00
Koute b0f34e4b29 Add a PolkaVM-based executor (#3458)
This PR adds a new PolkaVM-based executor to Substrate.

- The executor can now be used to actually run a PolkaVM-based runtime,
and successfully produces blocks.
- The executor is always compiled-in, but is disabled by default.
- The `SUBSTRATE_ENABLE_POLKAVM` environment variable must be set to `1`
to enable the executor, in which case the node will accept both WASM and
PolkaVM program blobs (otherwise it'll default to WASM-only). This is
deliberately undocumented and not explicitly exposed anywhere (e.g. in
the command line arguments, or in the API) to disincentivize anyone from
enabling it in production. If/when we'll move this into production usage
I'll remove the environment variable and do it "properly".
- I did not use our legacy runtime allocator for the PolkaVM executor,
so currently every allocation inside of the runtime will leak guest
memory until that particular instance is destroyed. The idea here is
that I will work on the https://github.com/polkadot-fellows/RFCs/pull/4
which will remove the need for the legacy allocator under WASM, and that
will also allow us to use a proper non-leaking allocator under PolkaVM.
- I also did some minor cleanups of the WASM executor and deleted some
dead code.

No prdocs included since this is not intended to be an end-user feature,
but an unofficial experiment, and shouldn't affect any current
production user. Once this is production-ready a full Polkadot
Fellowship RFC will be necessary anyway.
2024-03-12 05:23:06 +00:00
cuinix ea458d0b95 fix some typos (#3587)
Signed-off-by: cuinix <915115094@qq.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
2024-03-08 21:28:04 +00:00
André Silva 30c32e3d84 move substrate-bip39 into polkadot-sdk (#3579)
Moves [substrate-bip39](https://github.com/paritytech/substrate-bip39)
into substrate. All git history is preserved. Dependencies have been
updated to use the same version as the rest of the repo.

Fixes https://github.com/paritytech/polkadot-sdk/issues/1934.

---------

Co-authored-by: Maciej Hirsz <maciej.hirsz@gmail.com>
Co-authored-by: Maciej Hirsz <1096222+maciejhirsz@users.noreply.github.com>
Co-authored-by: Gav Wood <i@gavwood.com>
Co-authored-by: Stanislav Tkach <stanislav.tkach@gmail.com>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Demi M. Obenour <demi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: NikVolf <nikvolf@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Benjamin Kampmann <ben@gnunicorn.org>
Co-authored-by: Maciej Hirsz <hello@maciej.codes>
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: adoerr <0xad@gmx.net>
Co-authored-by: Jun Jiang <jasl9187@hotmail.com>
Co-authored-by: Dan Shields <35669742+NukeManDan@users.noreply.github.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
2024-03-07 10:45:56 +00:00
Alin Dima 62b78a1615 provisioner: allow multiple cores assigned to the same para (#3233)
https://github.com/paritytech/polkadot-sdk/issues/3130

builds on top of https://github.com/paritytech/polkadot-sdk/pull/3160

Processes the availability cores and builds a record of how many
candidates it should request from prospective-parachains and their
predecessors.
Tries to supply as many candidates as the runtime can back. Note that
the runtime changes to back multiple candidates per para are not yet
done, but this paves the way for it.

The following backing/inclusion policy is assumed:
1. the runtime will never back candidates of the same para which don't
form a chain with the already backed candidates. Even if the others are
still pending availability. We're optimistic that they won't time out
and we don't want to back parachain forks (as the complexity would be
huge).
2. if a candidate is timed out of the core before being included, all of
its successors occupying a core will be evicted.
3. only the candidates which are made available and form a chain
starting from the on-chain para head may be included/enacted and cleared
from the cores. In other words, if para head is at A and the cores are
occupied by B->C->D, and B and D are made available, only B will be
included and its core cleared. C and D will remain on the cores awaiting
for C to be made available or timed out. As point (2) above already
says, if C is timed out, D will also be dropped.
4. The runtime will deduplicate candidates which form a cycle. For
example if the provisioner supplies candidates A->B->A, the runtime will
only back A (as the state output will be the same)

Note that if a candidate is timed out, we don't guarantee that in the
next relay chain block the block author will be able to fill all of the
timed out cores of the para. That increases complexity by a lot.
Instead, the provisioner will supply N candidates where N is the number
of candidates timed out, but doesn't include their successors which will
be also deleted by the runtime. This'll be backfilled in the next relay
chain block.

Adjacent changes:
- Also fixes: https://github.com/paritytech/polkadot-sdk/issues/3141
- For non prospective-parachains, don't supply multiple candidates per
para (we can't have elastic scaling without prospective parachains
enabled). paras_inherent should already sanitise this input but it's
more efficient this way.

Note: all of these changes are backwards-compatible with the
non-elastic-scaling scenario (one core per para).
2024-03-01 18:25:24 +00:00
Alexandru Gheorghe 761937ecdc Fix accidental no-shows on node restart (#3277)
If approval was in progress we didn't actually restart it, so we end up
in a situation where we distribute our assignment, but we don't
distribute any approval.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
2024-02-29 08:21:00 +00:00
maksimryndin 426136671a PVF: re-preparing artifact on failed runtime construction (#3187)
resolve https://github.com/paritytech/polkadot-sdk/issues/3139

- [x] use a distinguishable error for `execute_artifact`
- [x] remove artifact in case of a `RuntimeConstruction` error during
the execution
- [x] augment the `validate_candidate_with_retry` of `ValidationBackend`
with the case of retriable `RuntimeConstruction` error during the
execution
- [x] update the book
(https://paritytech.github.io/polkadot-sdk/book/node/utility/pvf-host-and-workers.html#retrying-execution-requests)
- [x] add a test
- [x] run zombienet tests

---------

Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com>
2024-02-28 16:29:27 +00:00
Andrei Sandu 2431001ec0 Runtime: allow backing multiple candidates of same parachain on different cores (#3231)
Fixes https://github.com/paritytech/polkadot-sdk/issues/3144

Builds on top of https://github.com/paritytech/polkadot-sdk/pull/3229

### Summary
Some preparations for Runtime to support elastic scaling, guarded by
config node features bit `FeatureIndex::ElasticScalingMVP`. This PR
introduces a per-candidate `CoreIndex` but does it in a hacky way to
avoid changing `CandidateCommitments`, `CandidateReceipts` primitives
and networking protocols.

#### Including `CoreIndex` in `BackedCandidate`
If the `ElasticScalingMVP` feature bit is enabled then
`BackedCandidate::validator_indices` is extended by 8 bits.
The value stored in these bits represents the assumed core index for the
candidate.

It is temporary solution which works by creating a mapping from
`BackedCandidate` to `CoreIndex` by assuming the `CoreIndex` can be
discovered by checking in which validator group the validator that
signed the statement is.

TODO:
- [x] fix tests
- [x] add new tests
- [x] Bump runtime API for Kusama, so we have that node features thing!
-> https://github.com/polkadot-fellows/runtimes/pull/194

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: alindima <alin@parity.io>
Co-authored-by: alindima <alin@parity.io>
2024-02-23 16:35:48 +00:00
Bastian Köcher 9bf1a5e238 Check that the validation code matches the parachain code (#3433)
This introduces a check to ensure that the parachain code matches the
validation code stored in the relay chain state. If not, it will print a
warning. This should be mainly useful for parachain builders to make
sure they have setup everything correctly.
2024-02-22 13:35:00 +00:00
Andrei Sandu 60e537b95f Elastic scaling: use an assumed CoreIndex in candidate-backing (#3229)
First step in implementing
https://github.com/paritytech/polkadot-sdk/issues/3144

### Summary of changes
- switch statement `Table` candidate mapping from `ParaId` to
`CoreIndex`
- introduce experimental `InjectCoreIndex`  node feature.
- determine and assume a `CoreIndex` for a candidate based on statement
validator index. If the signature is valid it means validator controls
the validator that index and we can easily map it to a validator
group/core.
- introduce a temporary provisioner fix until we fully enable elastic
scaling in the subystem. The fix ensures we don't fetch the same
backable candidate when calling `get_backable_candidate` for each core.

TODO:
- [x] fix backing tests
- [x] fix statement table tests
- [x] add new test

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: alindima <alin@parity.io>
Co-authored-by: alindima <alin@parity.io>
2024-02-22 08:22:31 +00:00
Oliver Tale-Yazdi e89d0fca35 Lift dependencies to the workspace (Part 2/x) (#3366)
Lifting some more dependencies to the workspace. Just using the
most-often updated ones for now.
It can be reproduced locally.

```sh
# First you can check if there would be semver incompatible bumps (looks good in this case):
$ zepter transpose dependency lift-to-workspace --ignore-errors syn quote thiserror "regex:^serde.*"

# Then apply the changes:
$ zepter transpose dependency lift-to-workspace --version-resolver=highest syn quote thiserror "regex:^serde.*" --fix

# And format the changes:
$ taplo format --config .config/taplo.toml
```

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2024-02-20 14:28:05 +00:00
Bastian Köcher ef6ac94f52 Downgrade log message to trace (#3405)
This spams logs in `Debug` with no useful information.
2024-02-20 14:28:05 +00:00
ordian 612587b7b6 do not block finality for "disabled" disputes (#3358)
- [x] test with zombienet-sdk
- [x] prdoc

Relevant Issues:
https://github.com/paritytech/polkadot-sdk/issues/3314 (connected to the
cause)
https://github.com/paritytech/polkadot-sdk/issues/3345 (solves)

---------

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
2024-02-17 02:00:22 +00:00
Oliver Tale-Yazdi e80c24733f Lift dependencies to the workspace (Part 1) (#2070)
Changes (partial https://github.com/paritytech/polkadot-sdk/issues/994):
- Set log to `0.4.20` everywhere
- Lift `log` to the workspace

Starting with a simpler one after seeing
https://github.com/paritytech/polkadot-sdk/pull/2065 from @jsdw.
This sets the `default-features` to `false` in the root and then
overwrites that in each create to its original value. This is necessary
since otherwise the `default` features are additive and its impossible
to disable them in the crate again once they are enabled in the
workspace.

I am using a tool to do this, so its mostly a test to see that it works
as expected.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2024-02-12 11:19:20 +00:00
maksimryndin 4883e14482 refactor pvf security module (#3047)
resolve https://github.com/paritytech/polkadot-sdk/issues/2321

- [x] refactor `security` module into a conditionally compiled
- [x] rename `amd64` into x86-64 for consistency with conditional
compilation guards and remove reference to a particular vendor
- [x] run unit tests and zombienet

---------

Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com>
2024-02-11 09:59:10 +00:00
Alin Dima 7df1ae3b81 prospective-parachains: respond with multiple backable candidates (#3160)
Fixes https://github.com/paritytech/polkadot-sdk/issues/3129
2024-02-06 08:21:28 +00:00
Alexandru Gheorghe f9f886886b Introduce approval-voting/distribution benchmark (#2621)
## Summary
Built on top of the tooling and ideas introduced in
https://github.com/paritytech/polkadot-sdk/pull/2528, this PR introduces
a synthetic benchmark for measuring and assessing the performance
characteristics of the approval-voting and approval-distribution
subsystems.

Currently this allows, us to simulate the behaviours of these systems
based on the following dimensions:
```
TestConfiguration:
# Test 1
- objective: !ApprovalsTest
    last_considered_tranche: 89
    min_coalesce: 1
    max_coalesce: 6
    enable_assignments_v2: true
    send_till_tranche: 60
    stop_when_approved: false
    coalesce_tranche_diff: 12
    workdir_prefix: "/tmp"
    num_no_shows_per_candidate: 0
    approval_distribution_expected_tof: 6.0
    approval_distribution_cpu_ms: 3.0
    approval_voting_cpu_ms: 4.30
  n_validators: 500
  n_cores: 100
  n_included_candidates: 100
  min_pov_size: 1120
  max_pov_size: 5120
  peer_bandwidth: 524288000000
  bandwidth: 524288000000
  latency:
    min_latency:
      secs: 0
      nanos: 1000000
    max_latency:
      secs: 0
      nanos: 100000000
  error: 0
  num_blocks: 10
```

## The approach
1. We build a real overseer with the real implementations for
approval-voting and approval-distribution subsystems.
2. For a given network size, for each validator we pre-computed all
potential assignments and approvals it would send, because this a
computation heavy operation this will be cached on a file on disk and be
re-used if the generation parameters don't change.
3. The messages will be sent accordingly to the configured parameters
and those are split into 3 main benchmarking scenarios.

## Benchmarking scenarios

### Best case scenario *approvals_throughput_best_case.yaml*
It send to the approval-distribution only the minimum required tranche
to gathered the needed_approvals, so that a candidate is approved.

### Behaviour in the presence of no-shows *approvals_no_shows.yaml*
It sends the tranche needed to approve a candidate when we have a
maximum of *num_no_shows_per_candidate* tranches with no-shows for each
candidate.

### Maximum throughput *approvals_throughput.yaml*
It sends all the tranches for each block and measures the used CPU and
necessary network bandwidth. by the approval-voting and
approval-distribution subsystem.

## How to run it
```
cargo run -p polkadot-subsystem-bench --release -- test-sequence --path polkadot/node/subsystem-bench/examples/approvals_throughput.yaml
```

## Evaluating performance
### Use the real subsystems metrics
If you follow the steps in
https://github.com/paritytech/polkadot-sdk/tree/master/polkadot/node/subsystem-bench#install-grafana
for installing locally prometheus and grafana, all real metrics for the
`approval-distribution`, `approval-voting` and overseer are available.
E.g:
<img width="2149" alt="Screenshot 2023-12-05 at 11 07 46"
src="https://github.com/paritytech/polkadot-sdk/assets/49718502/cb8ae2dd-178b-4922-bfa4-dc37e572ed38">

<img width="2551" alt="Screenshot 2023-12-05 at 11 09 42"
src="https://github.com/paritytech/polkadot-sdk/assets/49718502/8b4542ba-88b9-46f9-9b70-cc345366081b">

<img width="2154" alt="Screenshot 2023-12-05 at 11 10 15"
src="https://github.com/paritytech/polkadot-sdk/assets/49718502/b8874d8d-632e-443a-9840-14ad8e90c54f">

<img width="2535" alt="Screenshot 2023-12-05 at 11 10 52"
src="https://github.com/paritytech/polkadot-sdk/assets/49718502/779a439f-fd18-4985-bb80-85d5afad78e2">

### Profile with pyroscope
1. Setup pyroscope following the steps in
https://github.com/paritytech/polkadot-sdk/tree/master/polkadot/node/subsystem-bench#install-pyroscope,
then run any of the benchmark scenario with `--profile` as the
arguments.
2. Open the pyroscope dashboard in grafana, e.g:
<img width="2544" alt="Screenshot 2024-01-09 at 17 09 58"
src="https://github.com/paritytech/polkadot-sdk/assets/49718502/58f50c99-a910-4d20-951a-8b16639303d9">



### Useful  logs
1. Network bandwidth requirements:
```
Payload bytes received from peers: 503993 KiB total, 50399 KiB/block
Payload bytes sent to peers: 629971 KiB total, 62997 KiB/block
```

2. Cpu usage by the approval-distribution/approval-voting subsystems.
```
approval-distribution CPU usage 84.061s
approval-distribution CPU usage per block 8.406s
approval-voting CPU usage 96.532s
approval-voting CPU usage per block 9.653s
```

3. Time passed until a given block is approved
```
 Chain selection approved  after 3500 ms hash=0x0101010101010101010101010101010101010101010101010101010101010101
Chain selection approved  after 4500 ms hash=0x0202020202020202020202020202020202020202020202020202020202020202
```

### Using benchmark to quantify improvements from
https://github.com/paritytech/polkadot-sdk/pull/1178 +
https://github.com/paritytech/polkadot-sdk/pull/1191

Using a versi-node we compare the scenarios where all new optimisations
are disabled with a scenarios where tranche0 assignments are sent in a
single message and a conservative simulation where the coalescing of
approvals gives us just 50% reduction in the number of messages we send.

Overall, what we see is a speedup of around 30-40% in the time it takes
to process the necessary messages and a 30-40% reduction in the
necessary bandwidth.

#### Best case scenario comparison(minimum required tranches sent).
Unoptimised
```
    Number of blocks: 10
    Payload bytes received from peers: 53289 KiB total, 5328 KiB/block
    Payload bytes sent to peers: 52489 KiB total, 5248 KiB/block
    approval-distribution CPU usage 6.732s
    approval-distribution CPU usage per block 0.673s
    approval-voting CPU usage 9.523s
    approval-voting CPU usage per block 0.952s
```

vs Optimisation enabled
```
   Number of blocks: 10
   Payload bytes received from peers: 32141 KiB total, 3214 KiB/block
   Payload bytes sent to peers: 37314 KiB total, 3731 KiB/block
   approval-distribution CPU usage 4.658s
   approval-distribution CPU usage per block 0.466s
   approval-voting CPU usage 6.236s
   approval-voting CPU usage per block 0.624s
```

#### Worst case all tranches sent, very unlikely happens when sharding
breaks.

Unoptimised
```
   Number of blocks: 10
   Payload bytes received from peers: 746393 KiB total, 74639 KiB/block
   Payload bytes sent to peers: 729151 KiB total, 72915 KiB/block
   approval-distribution CPU usage 118.681s
   approval-distribution CPU usage per block 11.868s
   approval-voting CPU usage 124.118s
   approval-voting CPU usage per block 12.412s
```

vs optimised
```
    Number of blocks: 10
    Payload bytes received from peers: 503993 KiB total, 50399 KiB/block
    Payload bytes sent to peers: 629971 KiB total, 62997 KiB/block
    approval-distribution CPU usage 84.061s
    approval-distribution CPU usage per block 8.406s
    approval-voting CPU usage 96.532s
    approval-voting CPU usage per block 9.653s
```


## TODOs
[x] Polish implementation.
[x] Use what we have so far to evaluate
https://github.com/paritytech/polkadot-sdk/pull/1191 before merging.
[x] List of features and additional dimensions we want to use for
benchmarking.
[x] Run benchmark on hardware similar with versi and kusama nodes.
[ ] Add benchmark to be run in CI for catching regression in
performance.
[ ] Rebase on latest changes for network emulation.

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
2024-02-05 06:46:22 +00:00
Bastian Köcher e0674cb3ad Enable async backing by default for Rococo/Westend (#3162)
This change is mainly for people running the local variants. They can
directly start with async backing.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
2024-02-02 19:51:56 +00:00
s0me0ne-unkn0wn 3e8139e7de Do not run unneeded subsystems on collator and its alongside node (#3061)
Currently, collators and their alongside nodes spin up a full-scale
overseer running a bunch of subsystems that are not needed if the node
is not a validator. That was considered to be harmless; however, we've
got problems with unused subsystems getting stalled for a reason not
currently known, resulting in the overseer exiting and bringing down the
whole node.

This PR aims to only run needed subsystems on such nodes, replacing the
rest with `DummySubsystem`.

It also enables collator-optimized availability recovery subsystem
implementation.

Partially solves #1730.
2024-01-29 09:53:51 +00:00
Liam Aharon 3717ec3802 Sync Cargo.toml and crates.io versions (#3034)
Related https://github.com/paritytech/polkadot-sdk/issues/3032

---

Using https://github.com/liamaharon/cargo-workspace-version-tools/ 

`cargo run -- sync --path ../polkadot-sdk`

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
2024-01-26 18:14:03 +00:00
Davide Galassi 4c10fd2a41 Move cryptographic hashing procedures to crypto folder. (#2306)
Step towards https://github.com/paritytech/polkadot-sdk/issues/1975

As reported
https://github.com/paritytech/polkadot-sdk/issues/1975#issuecomment-1774534225
I'd like to encapsulate crypto related stuff in a dedicated folder.

Currently all cryptographic primitive wrappers are all sparsed in
`substrate/core` which contains "misc core" stuff.

To simplify the process, as the first step with this PR I propose to
move the cryptographic hashing there.

The `substrate/crypto` folder was already created to contains `ec-utils`
crate.

Notes:
- rename `sp-core-hashing` to `sp-crypto-hashing`
- rename `sp-core-hashing-proc-macro` to `sp-crypto-hashing-proc-macro`
- As the crates name is changed I took the freedom to restart fresh from
version 0.1.0 for both crates

---------

Co-authored-by: Robert Hambrock <roberthambrock@gmail.com>
2024-01-22 23:36:14 +00:00
Alexandru Gheorghe 95ff9b2c54 approval-voting: Move verbose log from debug to tracing (#3019)
... it was like that before it was accidentally changed in
https://github.com/paritytech/polkadot-sdk/commit/a84dd0dba58d51503b8942360aa4fb30a5a96af5

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
2024-01-22 21:38:34 +00:00
s0me0ne-unkn0wn d37a45650e Make candidate validation bounded again (#2125)
This PR aims to channel the backpressure of the PVF host's preparation
and execution queues to the candidate validation subsystem consumers.

Related: #708
2024-01-21 13:56:44 +00:00
jserrat 21ef949b6e Use clone instead of fork on pvf (#2477)
@mrcnski Done the change on the prepare worker, once the prepare worker
part is good I'll do the same for the execute worker.

This is based on
https://github.com/koute/polkavm/blob/11beebd06276ce9b84f335350138479e714f6caf/crates/polkavm/src/sandbox/linux.rs#L711.

## TODO

- [x] Add a check for this capability at startup
- [x] Add prdoc mentioning the new Secure Validator Mode (optional)
requirement.

## Related

Closes #2162

---------

Co-authored-by: Marcin S <marcin@realemail.net>
2024-01-21 11:15:36 +00:00
dependabot[bot] 66b2fa2e59 Bump libc from 0.2.149 to 0.2.152 (#2994)
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.149 to 0.2.152.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/rust-lang/libc/releases">libc's
releases</a>.</em></p>
<blockquote>
<h2>0.2.152</h2>
<h2>What's Changed</h2>
<ul>
<li>openbsd: syscall() has been removed in upcoming OpenBSD 7.5 by <a
href="https://github.com/semarie"><code>@​semarie</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3479">rust-lang/libc#3479</a></li>
<li>adding tcp_info to openbsd by <a
href="https://github.com/devnexen"><code>@​devnexen</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3481">rust-lang/libc#3481</a></li>
<li>iadding yser_fpxregs_struct data to linux/musl i686. by <a
href="https://github.com/devnexen"><code>@​devnexen</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3477">rust-lang/libc#3477</a></li>
<li>strftime* api for *BSD by <a
href="https://github.com/devnexen"><code>@​devnexen</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3483">rust-lang/libc#3483</a></li>
<li>strftime_l for Linux glibc/musl by <a
href="https://github.com/devnexen"><code>@​devnexen</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3484">rust-lang/libc#3484</a></li>
<li>adding iocb data for io_submit syscall for linux/glibc. by <a
href="https://github.com/devnexen"><code>@​devnexen</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3486">rust-lang/libc#3486</a></li>
<li>Re-enable <code>i686-pc-windows-gnu</code> CI by <a
href="https://github.com/JohnTitor"><code>@​JohnTitor</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3489">rust-lang/libc#3489</a></li>
<li>Stop testing on FreeBSD 12 by <a
href="https://github.com/asomers"><code>@​asomers</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3475">rust-lang/libc#3475</a></li>
<li>Add <code>ifreq</code>, <code>ifconf</code> and related constants to
Android by <a
href="https://github.com/arctic-alpaca"><code>@​arctic-alpaca</code></a>
in <a
href="https://redirect.github.com/rust-lang/libc/pull/3487">rust-lang/libc#3487</a></li>
<li>Add waitid function for OpenBSD by <a
href="https://github.com/lcheylus"><code>@​lcheylus</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3490">rust-lang/libc#3490</a></li>
<li>adding SOMAXCONN to redox by <a
href="https://github.com/devnexen"><code>@​devnexen</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3491">rust-lang/libc#3491</a></li>
<li>Fix typos in comments by <a
href="https://github.com/asomers"><code>@​asomers</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3492">rust-lang/libc#3492</a></li>
<li>fix typos in libc by <a
href="https://github.com/Takashiidobe"><code>@​Takashiidobe</code></a>
in <a
href="https://redirect.github.com/rust-lang/libc/pull/3496">rust-lang/libc#3496</a></li>
<li>apple adding tcp_connection_info struct by <a
href="https://github.com/devnexen"><code>@​devnexen</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3482">rust-lang/libc#3482</a></li>
<li>Improve the version parser of Emscripten by <a
href="https://github.com/kleisauke"><code>@​kleisauke</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3498">rust-lang/libc#3498</a></li>
<li>Add constants from <!-- raw HTML omitted --> by <a
href="https://github.com/GuillaumeGomez"><code>@​GuillaumeGomez</code></a>
in <a
href="https://redirect.github.com/rust-lang/libc/pull/3505">rust-lang/libc#3505</a></li>
<li>Define <code>TFD_TIMER_*</code> constants on FreeBSD. by <a
href="https://github.com/sunfishcode"><code>@​sunfishcode</code></a> in
<a
href="https://redirect.github.com/rust-lang/libc/pull/3506">rust-lang/libc#3506</a></li>
<li>Add support for posix_spawn on OpenBSD by <a
href="https://github.com/nuudlman"><code>@​nuudlman</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3502">rust-lang/libc#3502</a></li>
<li>clock_nanosleep for dragonflybsd, moving constants freebsd only too.
by <a href="https://github.com/devnexen"><code>@​devnexen</code></a> in
<a
href="https://redirect.github.com/rust-lang/libc/pull/3509">rust-lang/libc#3509</a></li>
<li>solarish add fcntl's O_DIRECT constant. by <a
href="https://github.com/devnexen"><code>@​devnexen</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3503">rust-lang/libc#3503</a></li>
<li>Unpin cc dependency version by <a
href="https://github.com/JohnTitor"><code>@​JohnTitor</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3511">rust-lang/libc#3511</a></li>
<li>redox: add openpty, login_tty, TIOCSCTTY, and organize functions by
<a href="https://github.com/jackpot51"><code>@​jackpot51</code></a> in
<a
href="https://redirect.github.com/rust-lang/libc/pull/3512">rust-lang/libc#3512</a></li>
<li>Move all seccomp consts and structs into top-level mod by <a
href="https://github.com/boustrophedon"><code>@​boustrophedon</code></a>
in <a
href="https://redirect.github.com/rust-lang/libc/pull/3343">rust-lang/libc#3343</a></li>
<li>freebsd 15 support proposal. by <a
href="https://github.com/devnexen"><code>@​devnexen</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3508">rust-lang/libc#3508</a></li>
<li>Prepare workflow for merge queue by <a
href="https://github.com/JohnTitor"><code>@​JohnTitor</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3514">rust-lang/libc#3514</a></li>
<li>Allow dead_code on <code>clockid_t</code> by <a
href="https://github.com/JohnTitor"><code>@​JohnTitor</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3516">rust-lang/libc#3516</a></li>
<li>Add more items from <code>include/linux/sched.h</code> header by <a
href="https://github.com/GuillaumeGomez"><code>@​GuillaumeGomez</code></a>
in <a
href="https://redirect.github.com/rust-lang/libc/pull/3515">rust-lang/libc#3515</a></li>
<li>fix typo by <a
href="https://github.com/shuoer86"><code>@​shuoer86</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3517">rust-lang/libc#3517</a></li>
<li>Add sigsuspend to more targets: bsd, haiku, and solarish by <a
href="https://github.com/asomers"><code>@​asomers</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3518">rust-lang/libc#3518</a></li>
<li>Tweak libc-0.2 CI by <a
href="https://github.com/JohnTitor"><code>@​JohnTitor</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3519">rust-lang/libc#3519</a></li>
<li>fuchsia adding pthread_set/getname_np by <a
href="https://github.com/devnexen"><code>@​devnexen</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3520">rust-lang/libc#3520</a></li>
<li>Prepare docs for libc v0.3 by <a
href="https://github.com/JohnTitor"><code>@​JohnTitor</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3510">rust-lang/libc#3510</a></li>
<li>Merge main into libc-0.2 by <a
href="https://github.com/JohnTitor"><code>@​JohnTitor</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3521">rust-lang/libc#3521</a></li>
<li>Prepare release for v0.2.152 by <a
href="https://github.com/JohnTitor"><code>@​JohnTitor</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3522">rust-lang/libc#3522</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/Takashiidobe"><code>@​Takashiidobe</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/libc/pull/3496">rust-lang/libc#3496</a></li>
<li><a href="https://github.com/nuudlman"><code>@​nuudlman</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/libc/pull/3502">rust-lang/libc#3502</a></li>
<li><a
href="https://github.com/boustrophedon"><code>@​boustrophedon</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/libc/pull/3343">rust-lang/libc#3343</a></li>
<li><a href="https://github.com/shuoer86"><code>@​shuoer86</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/libc/pull/3517">rust-lang/libc#3517</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/rust-lang/libc/compare/0.2.151...0.2.152">https://github.com/rust-lang/libc/compare/0.2.151...0.2.152</a></p>
<h2>0.2.151</h2>
<h2>What's Changed</h2>
<ul>
<li>Add new constants to be used in (linux) sysctl by <a
href="https://github.com/GuillaumeGomez"><code>@​GuillaumeGomez</code></a>
in <a
href="https://redirect.github.com/rust-lang/libc/pull/3423">rust-lang/libc#3423</a></li>
<li>openbsd ifreq implementation refinement ifru_data member using
proper… by <a
href="https://github.com/devnexen"><code>@​devnexen</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3390">rust-lang/libc#3390</a></li>
<li>adding reboot to netbsd/openbsd. by <a
href="https://github.com/devnexen"><code>@​devnexen</code></a> in <a
href="https://redirect.github.com/rust-lang/libc/pull/3426">rust-lang/libc#3426</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/rust-lang/libc/commit/3d175191e6c6ad2b97de090d86d28dbfa4d271a3"><code>3d17519</code></a>
Merge pull request <a
href="https://redirect.github.com/rust-lang/libc/issues/3522">#3522</a>
from JohnTitor/libc-0.2.152</li>
<li><a
href="https://github.com/rust-lang/libc/commit/0f5d2c1d419543819f072169eab72a85abd2e9d2"><code>0f5d2c1</code></a>
Prepare release for v0.2.152</li>
<li><a
href="https://github.com/rust-lang/libc/commit/ee500ca0541809005b4ec4c7157add1e170dc545"><code>ee500ca</code></a>
Merge pull request <a
href="https://redirect.github.com/rust-lang/libc/issues/3521">#3521</a>
from rust-lang/main</li>
<li><a
href="https://github.com/rust-lang/libc/commit/72093f38fbc3c3fec485b0aba6f1ef81ad59ca1e"><code>72093f3</code></a>
Auto merge of <a
href="https://redirect.github.com/rust-lang/libc/issues/3510">#3510</a>
- JohnTitor:prepare-libc-0.3, r=JohnTitor</li>
<li><a
href="https://github.com/rust-lang/libc/commit/e5612b92d30668eea24495a2d6d84ed1f601f437"><code>e5612b9</code></a>
Auto merge of <a
href="https://redirect.github.com/rust-lang/libc/issues/3520">#3520</a>
- devnexen:fuchsia_upd, r=JohnTitor</li>
<li><a
href="https://github.com/rust-lang/libc/commit/44ba265df55df13b37a3e1e2145053b68196074d"><code>44ba265</code></a>
fuchsia adding pthread_set/getname_np</li>
<li><a
href="https://github.com/rust-lang/libc/commit/2f93bfb7678e18a9fc5373dec49384bd23f601c3"><code>2f93bfb</code></a>
Auto merge of <a
href="https://redirect.github.com/rust-lang/libc/issues/3519">#3519</a>
- JohnTitor:tweak-libc-0.2-ci, r=JohnTitor</li>
<li><a
href="https://github.com/rust-lang/libc/commit/11f7c7b89d9bfe8c53dcca7571c4965fa3375aab"><code>11f7c7b</code></a>
Auto merge of <a
href="https://redirect.github.com/rust-lang/libc/issues/3518">#3518</a>
- asomers:more-sigsuspend, r=JohnTitor</li>
<li><a
href="https://github.com/rust-lang/libc/commit/0b9596b22c59cd97409c017cc4dc0620e15bd87c"><code>0b9596b</code></a>
Tweak libc-0.2 CI</li>
<li><a
href="https://github.com/rust-lang/libc/commit/5594447694db909c1ad8fa191a8ac0df734cdd23"><code>5594447</code></a>
Auto merge of <a
href="https://redirect.github.com/rust-lang/libc/issues/3517">#3517</a>
- shuoer86:main, r=JohnTitor</li>
<li>Additional commits viewable in <a
href="https://github.com/rust-lang/libc/compare/0.2.149...0.2.152">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=libc&package-manager=cargo&previous-version=0.2.149&new-version=0.2.152)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-19 15:04:02 +00:00
maksimryndin 47a3faa41e Pvf thiserror (#2958)
resolve #2157 

- [x] fix broken doc links
- [x] fix codec macro typo
https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/node/core/pvf/common/src/error.rs#L81
(see the comment below)
- [x] refactor `ValidationError`, `PrepareError` and related error types
to use `thiserror` crate

## `codec` issue

`codec` macro was mistakenly applied two times to `Kernel` error (so it
was encoded with 10 instead of 11 and the same as `JobDied`). The PR
changes it to 11 because

- it was an initial goal of the code author
- Kernel is less frequent than JobDied so in case of existing error
encoding it is more probable to have 10 as JobDied than Kernel

See https://github.com/paritytech/parity-scale-codec/issues/555

----
polkadot address: 13zCyRG2a1W2ih5SioL8byqmQ6mc8vkgFwQgVzJSdRUUmp46

---------

Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com>
2024-01-19 14:10:09 +00:00
Bastian Köcher 38205bf139 Downgrade logging to debug (#2980)
There is no need to spawn operators with this debug log.

Closes: https://github.com/paritytech/polkadot-sdk/issues/2974
2024-01-18 10:49:13 +00:00
Marcin S 6a80c10a6f PVF: Remove artifact persistence across restarts (#2895)
Considering the complexity of
https://github.com/paritytech/polkadot-sdk/pull/2871 and the discussion
therein, as well as the further complexity introduced by the hardening
in https://github.com/paritytech/polkadot-sdk/pull/2742, as well as the
eventual replacement of wasmtime by PolkaVM, it seems best to remove
this persistence as it is creating more problems than it solves.

## Related

Closes https://github.com/paritytech/polkadot-sdk/issues/2863
2024-01-10 16:50:55 +01:00
dependabot[bot] ccd55552a8 Bump parking_lot from 0.11.2 to 0.12.1 (#2901)
Bumps [parking_lot](https://github.com/Amanieu/parking_lot) from 0.11.2
to 0.12.1.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md">parking_lot's
changelog</a>.</em></p>
<blockquote>
<h2>parking_lot 0.12.1 (2022-05-31)</h2>
<ul>
<li>Fixed incorrect memory ordering in <code>RwLock</code>. (<a
href="https://redirect.github.com/Amanieu/parking_lot/issues/344">#344</a>)</li>
<li>Added <code>Condvar::wait_while</code> convenience methods (<a
href="https://redirect.github.com/Amanieu/parking_lot/issues/343">#343</a>)</li>
</ul>
<h2>parking_lot_core 0.9.3 (2022-04-30)</h2>
<ul>
<li>Bump windows-sys dependency to 0.36. (<a
href="https://redirect.github.com/Amanieu/parking_lot/issues/339">#339</a>)</li>
</ul>
<h2>parking_lot_core 0.9.2, lock_api 0.4.7 (2022-03-25)</h2>
<ul>
<li>Enable const new() on lock types on stable. (<a
href="https://redirect.github.com/Amanieu/parking_lot/issues/325">#325</a>)</li>
<li>Added <code>MutexGuard::leak</code> function. (<a
href="https://redirect.github.com/Amanieu/parking_lot/issues/333">#333</a>)</li>
<li>Bump windows-sys dependency to 0.34. (<a
href="https://redirect.github.com/Amanieu/parking_lot/issues/331">#331</a>)</li>
<li>Bump petgraph dependency to 0.6. (<a
href="https://redirect.github.com/Amanieu/parking_lot/issues/326">#326</a>)</li>
<li>Don't use pthread attributes on the espidf platform. (<a
href="https://redirect.github.com/Amanieu/parking_lot/issues/319">#319</a>)</li>
</ul>
<h2>parking_lot_core 0.9.1 (2022-02-06)</h2>
<ul>
<li>Bump windows-sys dependency to 0.32. (<a
href="https://redirect.github.com/Amanieu/parking_lot/issues/316">#316</a>)</li>
</ul>
<h2>parking_lot 0.12.0, parking_lot_core 0.9.0, lock_api 0.4.6
(2022-01-28)</h2>
<ul>
<li>The MSRV is bumped to 1.49.0.</li>
<li>Disabled eventual fairness on wasm32-unknown-unknown. (<a
href="https://redirect.github.com/Amanieu/parking_lot/issues/302">#302</a>)</li>
<li>Added a rwlock method to report if lock is held exclusively. (<a
href="https://redirect.github.com/Amanieu/parking_lot/issues/303">#303</a>)</li>
<li>Use new <code>asm!</code> macro. (<a
href="https://redirect.github.com/Amanieu/parking_lot/issues/304">#304</a>)</li>
<li>Use windows-rs instead of winapi for faster builds. (<a
href="https://redirect.github.com/Amanieu/parking_lot/issues/311">#311</a>)</li>
<li>Moved hardware lock elision support to a separate Cargo feature. (<a
href="https://redirect.github.com/Amanieu/parking_lot/issues/313">#313</a>)</li>
<li>Removed used of deprecated <code>spin_loop_hint</code>. (<a
href="https://redirect.github.com/Amanieu/parking_lot/issues/314">#314</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/Amanieu/parking_lot/commit/336a9b31ff385728d00eb7ef173e4d054584b787"><code>336a9b3</code></a>
Release parking_lot 0.12.1</li>
<li><a
href="https://github.com/Amanieu/parking_lot/commit/b69a0547ce6c92b164528c65daafa48e8f1c08bb"><code>b69a054</code></a>
Merge pull request <a
href="https://redirect.github.com/Amanieu/parking_lot/issues/343">#343</a>
from bryanhitc/master</li>
<li><a
href="https://github.com/Amanieu/parking_lot/commit/3fe7233ee05fbae9bc94a831d4e66df0257880a2"><code>3fe7233</code></a>
Merge pull request <a
href="https://redirect.github.com/Amanieu/parking_lot/issues/344">#344</a>
from Amanieu/fix_rwlock_ordering</li>
<li><a
href="https://github.com/Amanieu/parking_lot/commit/ef12b00daf0dbc6dd025098ec2cd0517bb9f737c"><code>ef12b00</code></a>
small test update</li>
<li><a
href="https://github.com/Amanieu/parking_lot/commit/fdb063cd4e33d9c44cc08aea0c341ffec7e7b282"><code>fdb063c</code></a>
wait_while can't timeout fix</li>
<li><a
href="https://github.com/Amanieu/parking_lot/commit/26e19dced4ae01000abc0dfca251f56695b1b8aa"><code>26e19dc</code></a>
Remove WaitWhileResult</li>
<li><a
href="https://github.com/Amanieu/parking_lot/commit/d26c284fe8413bdeeed3ef4f70e23b5919a9ffeb"><code>d26c284</code></a>
Fix incorrect memory ordering in RwLock</li>
<li><a
href="https://github.com/Amanieu/parking_lot/commit/045828381a5facff71b1aad3c01af15801959458"><code>0458283</code></a>
Use saturating_add for WaitWhileResult</li>
<li><a
href="https://github.com/Amanieu/parking_lot/commit/686db4755913947219f83a04b21885032685d246"><code>686db47</code></a>
Add Condvar::wait_while convenience methods</li>
<li><a
href="https://github.com/Amanieu/parking_lot/commit/6f6e021ced6f67fef8002442aa97c4a08ed5c3e4"><code>6f6e021</code></a>
Merge pull request <a
href="https://redirect.github.com/Amanieu/parking_lot/issues/342">#342</a>
from MarijnS95/patch-1</li>
<li>Additional commits viewable in <a
href="https://github.com/Amanieu/parking_lot/compare/0.11.2...0.12.1">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=parking_lot&package-manager=cargo&previous-version=0.11.2&new-version=0.12.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-10 11:07:29 +00:00
dependabot[bot] ce4e5496de Bump lycheeverse/lychee-action (#2875)
Bumps
[lycheeverse/lychee-action](https://github.com/lycheeverse/lychee-action)
from 2ac9f030ccdea0033e2510a23a67da2a2da98492 to
fdea7032675810093199f485fe075f057cc37b3e.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/lycheeverse/lychee-action/commit/fdea7032675810093199f485fe075f057cc37b3e"><code>fdea703</code></a>
Update secure git hash for 1.9.0</li>
<li><a
href="https://github.com/lycheeverse/lychee-action/commit/22134d37a1fff6c2974df9c92a7c7e1e86a08f9c"><code>22134d3</code></a>
Bump version to 1.9.0</li>
<li><a
href="https://github.com/lycheeverse/lychee-action/commit/ebd95ebde89b46eaf3567432efb0b52bad2187d7"><code>ebd95eb</code></a>
Update to latest lychee 0.14.0</li>
<li><a
href="https://github.com/lycheeverse/lychee-action/commit/37d79d3b6eda9990480a57eed21c847792dec37a"><code>37d79d3</code></a>
Add integration tests for absolute output path and <code>--dump</code>
(<a
href="https://redirect.github.com/lycheeverse/lychee-action/issues/218">#218</a>)</li>
<li><a
href="https://github.com/lycheeverse/lychee-action/commit/57219e4f1d23b32484889db3dbd73f6b814c9bfa"><code>57219e4</code></a>
Add integration test for custom output paths (<a
href="https://redirect.github.com/lycheeverse/lychee-action/issues/1309">#1309</a>)
(<a
href="https://redirect.github.com/lycheeverse/lychee-action/issues/217">#217</a>)</li>
<li><a
href="https://github.com/lycheeverse/lychee-action/commit/a4282891d5805c0d9df0c0d524e2e95a3b0d1ac5"><code>a428289</code></a>
fix: cleanup if statement (<a
href="https://redirect.github.com/lycheeverse/lychee-action/issues/215">#215</a>)</li>
<li><a
href="https://github.com/lycheeverse/lychee-action/commit/ef8c8f32c278e9eed290c6e6c96748f16b6cd335"><code>ef8c8f3</code></a>
Add documentation on using the <code>--base</code> param when testing
local files (<a
href="https://redirect.github.com/lycheeverse/lychee-action/issues/211">#211</a>)</li>
<li>See full diff in <a
href="https://github.com/lycheeverse/lychee-action/compare/2ac9f030ccdea0033e2510a23a67da2a2da98492...fdea7032675810093199f485fe075f057cc37b3e">compare
view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: PG Herveou <pgherveou@gmail.com>
2024-01-10 00:12:35 +00:00
ordian 0ff3f4d3af dispute-coordinator: disabling in participation (#2637)
Closes #2225.

- [x] tests
- [x] fix todos
- [x] fix duplicates
- [x] make the check part of `potential_spam` 
- [x] fix a bug with votes insertion
- [x] guide changes
- [x] docs

---------

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
2024-01-09 08:44:19 +02:00
ordian a02b53475b backport to master: Handling of disabled validators in backing subsystem (#1259) (#2764)
#1259 was merged into a feature branch, but we've decided to merge
node-side changes for disabling straight into master.
This is a dependency of #1841 and #2637.

---------

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
2024-01-08 21:58:30 +02:00
Niklas Adolfsson a97a6f2095 rpc: add rpc v2 chainSpec to polkadot (#2859)
The [chainSpec RPC API from the v2
spec](https://paritytech.github.io/json-rpc-interface-spec/api/chainSpec.html)
was only added to substrate-node and should be added to polkadot as well

/cc @lexnv
2024-01-08 13:35:12 +01:00
Marcin S 8bf5a1c0b3 PVF: ensure job processes are cleaned up, add tests (#2643)
Fixes a potential memory leak.

`PR_SET_PDEATHSIG` is used to terminate children when the parent dies.
Note that this is subject to a race. There seems to be a raceless
alternative [here](https://stackoverflow.com/a/42498370/6085242), but
the concern is small enough that a bit more complexity doesn't seem
worth it. Left a bit more info in the code comment.
2023-12-29 16:27:18 +01:00
eskimor 69434d9a32 Coretime Feature branch (relay chain) (#1694)
Also fixes: https://github.com/paritytech/polkadot-sdk/issues/1417

- [x] CoreIndex -> AssignmentProvider mapping will be able to change any
time.
- [x] Implement
- [x] Provide Migrations
- [x] Add and fix tests
- [x] Implement bulk assigner logic
- [x] bulk assigner tests
- [x] Port over current assigner to use bulk designer (+ share on-demand
with bulk): top-level assigner has core ranges: legacy, bulk
- [x] Adjust migrations to reflect new assigner structure
- [x] Move migration code to Assignment code directly and make it
recursive (make it possible to skip releases) -> follow up ticket.
- [x] Test migrations
- [x] Add migration PR to runtimes repo -> follow up ticket.
- [x] Wire up with actual UMP messages
- [x] Write PR docs

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com>
Co-authored-by: BradleyOlson64 <lotrftw9@gmail.com>
Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com>
Co-authored-by: antonva <anton.asgeirsson@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Marcin S. <marcin@realemail.net>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: command-bot <>
2023-12-21 19:06:58 +01:00
André Silva 421af26b1f Update schnorrkel to 0.11.4 (#2524) 2023-12-19 23:24:02 +00:00
dependabot[bot] a250652b3b Bump async-trait from 0.1.73 to 0.1.74 (#2730)
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.73
to 0.1.74.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/dtolnay/async-trait/releases">async-trait's
releases</a>.</em></p>
<blockquote>
<h2>0.1.74</h2>
<ul>
<li>Documentation improvements</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/dtolnay/async-trait/commit/265979b07a9af573e1edd3b2a9b179533cfa7a6c"><code>265979b</code></a>
Release 0.1.74</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/5e677097d2e67f7a5c5e3023e2f3b99b36a9e132"><code>5e67709</code></a>
Fix doc test when async fn in trait is natively supported</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/ef144aed28b636eb65759505b2323afc4c753fbe"><code>ef144ae</code></a>
Update ui test suite to nightly-2023-10-15</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/9398a28d6fc977ccf8c286bd85b4b87a883f92ac"><code>9398a28</code></a>
Test docs.rs documentation build in CI</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/8737173dafa371e5e9e491d736513be1baf697f4"><code>8737173</code></a>
Update ui test suite to nightly-2023-09-24</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/5ba643c001a55f70c4a44690e040cdfab873ba56"><code>5ba643c</code></a>
Test dyn Trait containing async fn</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/247c8e7b0b3ff69c9518ebf93e69fe74d47f17b6"><code>247c8e7</code></a>
Add ui test testing the recommendation to use async-trait</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/799db66a84834c403860df4a8c0227d8fb7f9d9d"><code>799db66</code></a>
Update ui test suite to nightly-2023-09-23</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/0e60248011f751d8ccf58219d0a79aacfe9619f1"><code>0e60248</code></a>
Update actions/checkout@v3 -&gt; v4</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/7fcbc83993d5ef483d048c271a7f6c4ac8c98388"><code>7fcbc83</code></a>
Update ui test suite to nightly-2023-08-29</li>
<li>See full diff in <a
href="https://github.com/dtolnay/async-trait/compare/0.1.73...0.1.74">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=async-trait&package-manager=cargo&previous-version=0.1.73&new-version=0.1.74)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-12-18 09:35:33 +01:00