diff --git a/404.html b/404.html index a60c357..80f0ede 100644 --- a/404.html +++ b/404.html @@ -91,7 +91,7 @@ diff --git a/approved/0001-agile-coretime.html b/approved/0001-agile-coretime.html index 0435c3a..c3a03ab 100644 --- a/approved/0001-agile-coretime.html +++ b/approved/0001-agile-coretime.html @@ -90,7 +90,7 @@ diff --git a/approved/0005-coretime-interface.html b/approved/0005-coretime-interface.html index c002f83..596b563 100644 --- a/approved/0005-coretime-interface.html +++ b/approved/0005-coretime-interface.html @@ -90,7 +90,7 @@ diff --git a/approved/0007-system-collator-selection.html b/approved/0007-system-collator-selection.html index ba5cc67..4ced1f3 100644 --- a/approved/0007-system-collator-selection.html +++ b/approved/0007-system-collator-selection.html @@ -90,7 +90,7 @@ diff --git a/approved/0008-parachain-bootnodes-dht.html b/approved/0008-parachain-bootnodes-dht.html index bce336e..fe53b8d 100644 --- a/approved/0008-parachain-bootnodes-dht.html +++ b/approved/0008-parachain-bootnodes-dht.html @@ -90,7 +90,7 @@ diff --git a/approved/0012-process-for-adding-new-collectives.html b/approved/0012-process-for-adding-new-collectives.html index 8b07149..789a005 100644 --- a/approved/0012-process-for-adding-new-collectives.html +++ b/approved/0012-process-for-adding-new-collectives.html @@ -90,7 +90,7 @@ diff --git a/approved/0014-improve-locking-mechanism-for-parachains.html b/approved/0014-improve-locking-mechanism-for-parachains.html index fe0bb44..8fc601a 100644 --- a/approved/0014-improve-locking-mechanism-for-parachains.html +++ b/approved/0014-improve-locking-mechanism-for-parachains.html @@ -90,7 +90,7 @@ diff --git a/approved/0022-adopt-encointer-runtime.html b/approved/0022-adopt-encointer-runtime.html index 63d7687..b7d18f6 100644 --- a/approved/0022-adopt-encointer-runtime.html +++ b/approved/0022-adopt-encointer-runtime.html @@ -90,7 +90,7 @@ diff --git a/approved/0032-minimal-relay.html b/approved/0032-minimal-relay.html index 7d8f456..23e5721 100644 --- a/approved/0032-minimal-relay.html +++ b/approved/0032-minimal-relay.html @@ -90,7 +90,7 @@ diff --git a/approved/0050-fellowship-salaries.html b/approved/0050-fellowship-salaries.html index 893f112..9d88118 100644 --- a/approved/0050-fellowship-salaries.html +++ b/approved/0050-fellowship-salaries.html @@ -90,7 +90,7 @@ diff --git a/approved/0056-one-transaction-per-notification.html b/approved/0056-one-transaction-per-notification.html index f25bdaa..d703e38 100644 --- a/approved/0056-one-transaction-per-notification.html +++ b/approved/0056-one-transaction-per-notification.html @@ -90,7 +90,7 @@ diff --git a/index.html b/index.html index 14f0dae..6ae2c87 100644 --- a/index.html +++ b/index.html @@ -90,7 +90,7 @@ diff --git a/introduction.html b/introduction.html index 14f0dae..6ae2c87 100644 --- a/introduction.html +++ b/introduction.html @@ -90,7 +90,7 @@ diff --git a/print.html b/print.html index 7ddff74..c206799 100644 --- a/print.html +++ b/print.html @@ -91,7 +91,7 @@ @@ -3357,69 +3357,6 @@ is the correct way of introducing this change.

I do not have any specific questions about this change at the moment.

IMO, this change is pretty self-contained and there won't be any future work necessary.

-

(source)

-

Table of Contents

- -

RFC-0043: Introduce storage_proof_size Host Function for Improved Parachain Block Utilization

-
- - - -
Start Date30 October 2023
DescriptionHost function to provide the storage proof size to runtimes.
AuthorsSebastian Kunert
-
-

Summary

-

This RFC proposes a new host function for parachains, storage_proof_size. It shall provide the size of the currently recorded storage proof to the runtime. Runtime authors can use the proof size to improve block utilization by retroactively reclaiming unused storage weight.

-

Motivation

-

The number of extrinsics that are included in a parachain block is limited by two constraints: execution time and proof size. FRAME weights cover both concepts, and block-builders use them to decide how many extrinsics to include in a block. However, these weights are calculated ahead of time by benchmarking on a machine with reference hardware. The execution-time properties of the state-trie and its storage items are unknown at benchmarking time. Therefore, we make some assumptions about the state-trie:

- -

These pessimistic assumptions lead to an overestimation of storage weight, negatively impacting block utilization on parachains.

-

In addition, the current model does not account for multiple accesses to the same storage items. While these repetitive accesses will not increase storage-proof size, the runtime-side weight monitoring will account for them multiple times. Since the proof size is completely opaque to the runtime, we can not implement retroactive storage weight correction.

-

A solution must provide a way for the runtime to track the exact storage-proof size consumed on a per-extrinsic basis.

-

Stakeholders

- -

Explanation

-

This RFC proposes a new host function that exposes the storage-proof size to the runtime. As a result, runtimes can implement storage weight reclaiming mechanisms that improve block utilization.

-

This RFC proposes the following host function signature:

-
#![allow(unused)]
-fn main() {
-fn ext_storage_proof_size_version_1() -> u64;
-}
-

The host function MUST return an unsigned 64-bit integer value representing the current proof size. In block-execution and block-import contexts, this function MUST return the current size of the proof. To achieve this, parachain node implementors need to enable proof recording for block imports. In other contexts, this function MUST return 18446744073709551615 (u64::MAX), which represents disabled proof recording.

-

Performance, Ergonomics, and Compatibility

-

Performance

-

Parachain nodes need to enable proof recording during block import to correctly implement the proposed host function. Benchmarking conducted with balance transfers has shown a performance reduction of around 0.6% when proof recording is enabled.

-

Ergonomics

-

The host function proposed in this RFC allows parachain runtime developers to keep track of the proof size. Typical usage patterns would be to keep track of the overall proof size or the difference between subsequent calls to the host function.

-

Compatibility

-

Parachain teams will need to include this host function to upgrade.

-

Prior Art and References

-

(source)

Table of Contents

-

Stakeholders

+

Stakeholders

All chain teams are stakeholders, as implementing this feature would require timely effort on their side and would impact compatibility with older tools.

This feature is essential for all offline signer tools; many regular signing tools might make use of it. In general, this RFC greatly improves security of any network implementing it, as many governing keys are used with offline signers.

Implementing this RFC would remove requirement to maintain metadata portals manually, as task of metadata verification would be effectively moved to consensus mechanism of the chain.

-

Explanation

+

Explanation

Detailed description of metadata shortening and digest process is provided in metadata-shortener crate (see cargo doc --open and examples). Below are presented algorithms of the process.

Definitions

Metadata structure

@@ -3844,14 +3781,14 @@ modularized_registry.sort(|a, b| {

Testing, Security, and Privacy

The metadata shortening protocol should be extensively tested on all available examples of metadata before releasing changes to either metadata or shortener. Careful code review should be performed on shortener implementation code to ensure security. The main metadata tree would inevitably be constructed on runtime build which would also ensure correctness.

To be able to recall shortener protocol in case of vulnerability issues, a version byte is included.

-

Performance, Ergonomics, and Compatibility

-

Performance

+

Performance, Ergonomics, and Compatibility

+

Performance

This is negligibly short pessimization during build time on the chain side. Cold wallets performance would improve mostly as metadata validity mechanism that was taking most of effort in cold wallet support would become trivial.

-

Ergonomics

+

Ergonomics

The proposal was optimized for cold storage wallets usage with minimal impact on all other parts of the ecosystem

-

Compatibility

+

Compatibility

Proposal in this form is not compatible with older tools that do not implement proper MetadataV14 self-descriptive features; those would have to be upgraded to include a new signed extensions field.

-

Prior Art and References

+

Prior Art and References

This project was developed upon a Polkadot Treasury grant; relevant development links are located in metadata-offline-project repository.

Unresolved Questions

    @@ -3901,12 +3838,12 @@ modularized_registry.sort(|a, b| { AuthorsAlin Dima -

    Summary

    +

    Summary

    Propose a way of permuting the availability chunk indices assigned to validators for a given core and relay chain block, in the context of recovering available data from systematic chunks, with the purpose of fairly distributing network bandwidth usage.

    -

    Motivation

    +

    Motivation

    Currently, the ValidatorIndex is always identical to the ChunkIndex. Since the validator array is only shuffled once per session, naively using the ValidatorIndex as the ChunkIndex would pose an unreasonable stress on the first N/3 validators during an entire session, when favouring availability recovery from systematic chunks.

    @@ -3914,9 +3851,9 @@ validators during an entire session, when favouring availability recovery from s systematic availability chunks to different validators, based on the relay chain block and core. The main purpose is to ensure fair distribution of network bandwidth usage for availability recovery in general and in particular for systematic chunk holders.

    -

    Stakeholders

    +

    Stakeholders

    Relay chain node core developers.

    -

    Explanation

    +

    Explanation

    Systematic erasure codes

    An erasure coding algorithm is considered systematic if it preserves the original unencoded data as part of the resulting code. @@ -4115,20 +4052,20 @@ mitigate this problem and will likely be needed in the future for CoreJam.

    Testing, Security, and Privacy

    Extensive testing will be conducted - both automated and manual. This proposal doesn't affect security or privacy.

    -

    Performance, Ergonomics, and Compatibility

    -

    Performance

    +

    Performance, Ergonomics, and Compatibility

    +

    Performance

    This is a necessary data availability optimisation, as reed-solomon erasure coding has proven to be a top consumer of CPU time in polkadot as we scale up the parachain block size and number of availability cores.

    With this optimisation, preliminary performance results show that CPU time used for reed-solomon coding/decoding can be halved and total POV recovery time decrease by 80% for large POVs. See more here.

    -

    Ergonomics

    +

    Ergonomics

    Not applicable.

    -

    Compatibility

    +

    Compatibility

    This is a breaking change. See upgrade path section above. All validators need to have upgraded their node versions before the feature will be enabled via a runtime upgrade governance call.

    -

    Prior Art and References

    +

    Prior Art and References

    See comments on the tracking issue and the in-progress PR

    Unresolved Questions

    @@ -4215,20 +4152,20 @@ occupying. However, considering it's part of an unimported fork, the validator c AuthorsPierre Krieger -

    Summary

    +

    Summary

    This RFC proposes to make the mechanism of RFC #8 more generic by introducing the concept of "capabilities".

    Implementations can implement certain "capabilities", such as serving old block headers or being a parachain bootnode.

    The discovery mechanism of RFC #8 is extended to be able to discover nodes of specific capabilities.

    -

    Motivation

    +

    Motivation

    The Polkadot peer-to-peer network is made of nodes. Not all these nodes are equal. Some nodes store only the headers of recently blocks, some nodes store all the block headers and bodies since the genesis, some nodes store the storage of all blocks since the genesis, and so on.

    It is currently not possible to know ahead of time (without connecting to it and asking) which nodes have which data available, and it is not easily possible to build a list of nodes that have a specific piece of data available.

    If you want to download for example the header of block 500, you have to connect to a randomly-chosen node, ask it for block 500, and if it says that it doesn't have the block, disconnect and try another randomly-chosen node. In certain situations such as downloading the storage of old blocks, nodes that have the information are relatively rare, and finding through trial and error a node that has the data can take a long time.

    This RFC attempts to solve this problem by giving the possibility to build a list of nodes that are capable of serving specific data.

    -

    Stakeholders

    +

    Stakeholders

    Low-level client developers. People interested in accessing the archive of the chain.

    -

    Explanation

    +

    Explanation

    Reading RFC #8 first might help with comprehension, as this RFC is very similar.

    Please keep in mind while reading that everything below applies for both relay chains and parachains, except mentioned otherwise.

    Capabilities

    @@ -4272,17 +4209,17 @@ If blocks pruning is enabled and the chain is a relay chain, then Substrate unfo Furthermore, when a large number of providers are registered, only the providers closest to the key are kept, up to a certain implementation-defined limit.

    For this reason, an attacker can abuse this mechanism by randomly generating libp2p PeerIds until they find the 20 entries closest to the key representing the target capability. They are then in control of the list of nodes with that capability. While doing this can in no way be actually harmful, it could lead to eclipse attacks.

    Because the key changes periodically and isn't predictable, and assuming that the Polkadot DHT is sufficiently large, it is not realistic for an attack like this to be maintained in the long term.

    -

    Performance, Ergonomics, and Compatibility

    -

    Performance

    +

    Performance, Ergonomics, and Compatibility

    +

    Performance

    The DHT mechanism generally has a low overhead, especially given that publishing providers is done only every 24 hours.

    Doing a Kademlia iterative query then sending a provider record shouldn't take more than around 50 kiB in total of bandwidth for the parachain bootnode.

    Assuming 1000 nodes with a specific capability, the 20 Polkadot full nodes corresponding to that capability will each receive a sudden spike of a few megabytes of networking traffic when the key rotates. Again, this is relatively negligible. If this becomes a problem, one can add a random delay before a node registers itself to be the provider of the key corresponding to BabeApi_next_epoch.

    Maybe the biggest uncertainty is the traffic that the 20 Polkadot full nodes will receive from light clients that desire knowing the nodes with a capability. If this every becomes a problem, this value of 20 is an arbitrary constant that can be increased for more redundancy.

    -

    Ergonomics

    +

    Ergonomics

    Irrelevant.

    -

    Compatibility

    +

    Compatibility

    Irrelevant.

    -

    Prior Art and References

    +

    Prior Art and References

    Unknown.

    Unresolved Questions

    While it fundamentally doesn't change much to this RFC, using BabeApi_currentEpoch and BabeApi_nextEpoch might be inappropriate. I'm not familiar enough with good practices within the runtime to have an opinion here. Should it be an entirely new pallet?

    @@ -4326,19 +4263,19 @@ We could even add to the peer-to-peer network nodes that are only capable of ser AuthorsJiahao Ye -

    Summary

    +

    Summary

    Currently, substrate runtime use an simple allocator defined by host side. Every runtime MUST import these allocator functions for normal execution. This situation make runtime code not versatile enough.

    So this RFC proposes to define a new spec for allocator part to make substrate runtime more generic.

    -

    Motivation

    +

    Motivation

    Since this RFC define a new way for allocator, we now regard the old one as legacy allocator. As we all know, since the allocator implementation details are defined by the substrate client, parachain/parathread cannot customize memory allocator algorithm, so the new specification allows the runtime to customize memory allocation, and then export the allocator function according to the specification for the client side to use. Another benefit is that some new host functions can be designed without allocating memory on the client, which may have potential performance improvements. Also it will help provide a unified and clean specification if substrate runtime support multi-targets(e.g. RISC-V). There is also a potential benefit. Many programming languages that support compilation to wasm may not be friendly to supporting external allocator. This is beneficial for other programming languages ​​to enter the substrate runtime ecosystem. The last and most important benefit is that for offchain context execution, the runtime can fully support pure wasm. What this means here is that all imported host functions could not actually be called (as stub functions), then the various verification logic of the runtime can be converted into pure wasm, which provides the possibility for the substrate runtime to run block verification in other environments (such as in browsers and other non-substrate environments).

    -

    Stakeholders

    +

    Stakeholders

    No attempt was made at convincing stakeholders.

    -

    Explanation

    +

    Explanation

    Runtime side spec

    This section contains a list of functions should be exported by substrate runtime.

    We define the spec as version 1, so the following dummy function v1 MUST be exported to hint @@ -4384,18 +4321,18 @@ The allocator inside of the runtime maybe slow down(or speed up) the runtime, st

    Testing, Security, and Privacy

    Keep the legacy allocator runtime test cases, and add new feature to compile test cases for v1 allocator spec. And then update the test asserts.

    Update template runtime to enable v1 spec. Once the dev network runs well, it seems that the spec is implmented correctly.

    -

    Performance, Ergonomics, and Compatibility

    -

    Performance

    +

    Performance, Ergonomics, and Compatibility

    +

    Performance

    As the above says, not obvious impact about performance. And polkadot-sdk could offer the best practice allocator for all chains. Third party also could customized by theirself. So the performance could be improved over time.

    -

    Ergonomics

    +

    Ergonomics

    Only for runtime developer, Just need to import a new crate and enable a new feature. Maybe it's convienient for other wasm-target language to implment.

    -

    Compatibility

    +

    Compatibility

    It's 100% compatible. Only Some runtime configs and executor configs need to be depreacted.

    For support new runtime spec, we MUST upgrade the client binary to support new spec of client part firstly.

    We SHALL add an optional primtive crate to enable the version 1 spec and disable the legacy allocator by cargo feature. For the first year, we SHALL disable the v1 by default, and enable it by default start in the next year.

    -

    Prior Art and References

    +

    Prior Art and References

-

Stakeholders

+

Stakeholders

The primary stakeholders of this RFC are:

-

Explanation

+

Explanation

Overview

The dynamic pricing model sets the new price based on supply and demand in the previous period. The model is a function of the number of Regions sold, piecewise-defined by two power functions.