pezkuwichain/pezkuwi-fellows
1
0
Fork 0
mirror of https://github.com/pezkuwichain/pezkuwi-fellows.git synced 2026-04-25 10:27:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
9a0b5b5d1555ff8fcb1cd0ba19481bb2bfe0ccfa
pezkuwi-fellows/print.html
T
bkchr 9a0b5b5d15 deploy: bcca93886c
2024-02-23 00:48:32 +00:00

7257 lines
627 KiB
HTML
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE HTML>
<html lang="en" class="polkadot" dir="ltr">
<head>
<!-- Book generated using mdBook -->
<meta charset="UTF-8">
<title>Polkadot Fellowship RFCs</title>
<meta name="robots" content="noindex">
<!-- Custom HTML head -->
<meta name="description" content="An online book of RFCs approved or proposed within the Polkadot Fellowship.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#ffffff">
<link rel="icon" href="favicon.svg">
<link rel="shortcut icon" href="favicon.png">
<link rel="stylesheet" href="css/variables.css">
<link rel="stylesheet" href="css/general.css">
<link rel="stylesheet" href="css/chrome.css">
<link rel="stylesheet" href="css/print.css" media="print">
<!-- Fonts -->
<link rel="stylesheet" href="FontAwesome/css/font-awesome.css">
<link rel="stylesheet" href="fonts/fonts.css">
<!-- Highlight.js Stylesheets -->
<link rel="stylesheet" href="highlight.css">
<link rel="stylesheet" href="tomorrow-night.css">
<link rel="stylesheet" href="ayu-highlight.css">
<!-- Custom theme stylesheets -->
<link rel="stylesheet" href="theme/polkadot.css">
</head>
<body class="sidebar-visible no-js">
<div id="body-container">
<!-- Provide site root to javascript -->
<script>
var path_to_root = "";
var default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "polkadot" : "polkadot";
</script>
<!-- Work around some values being stored in localStorage wrapped in quotes -->
<script>
try {
var theme = localStorage.getItem('mdbook-theme');
var sidebar = localStorage.getItem('mdbook-sidebar');
if (theme.startsWith('"') && theme.endsWith('"')) {
localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1));
}
if (sidebar.startsWith('"') && sidebar.endsWith('"')) {
localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1));
}
} catch (e) { }
</script>
<!-- Set the theme before any content is loaded, prevents flash -->
<script>
var theme;
try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { }
if (theme === null || theme === undefined) { theme = default_theme; }
var html = document.querySelector('html');
html.classList.remove('polkadot')
html.classList.add(theme);
var body = document.querySelector('body');
body.classList.remove('no-js')
body.classList.add('js');
</script>
<input type="checkbox" id="sidebar-toggle-anchor" class="hidden">
<!-- Hide / unhide sidebar before it is displayed -->
<script>
var body = document.querySelector('body');
var sidebar = null;
var sidebar_toggle = document.getElementById("sidebar-toggle-anchor");
if (document.body.clientWidth >= 1080) {
try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { }
sidebar = sidebar || 'visible';
} else {
sidebar = 'hidden';
}
sidebar_toggle.checked = sidebar === 'visible';
body.classList.remove('sidebar-visible');
body.classList.add("sidebar-" + sidebar);
</script>
<nav id="sidebar" class="sidebar" aria-label="Table of contents">
<div class="sidebar-scrollbox">
<ol class="chapter"><li class="chapter-item expanded affix "><a href="introduction.html">Introduction</a></li><li class="spacer"></li><li class="chapter-item expanded affix "><li class="part-title">Newly Proposed</li><li class="chapter-item expanded "><a href="new/0076-increase-max-length-of-identity-raw-data-values.html">RFC-0076: Increase maximum length of identity raw data values from 32 bytes</a></li><li class="chapter-item expanded "><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html">RFC-0077: Increase maximum length of identity PGP fingerprint values from 20 bytes</a></li><li class="chapter-item expanded "><a href="new/0078-merkelized-metadata.html">RFC-0078: Merkelized Metadata</a></li><li class="spacer"></li><li class="chapter-item expanded affix "><li class="part-title">Proposed</li><li class="chapter-item expanded "><a href="proposed/0013-prepare-blockbuilder-and-core-runtime-apis-for-mbms.html">RFC-0013: Prepare Core runtime API for MBMs</a></li><li class="chapter-item expanded "><a href="proposed/0045-nft-deposits-asset-hub.html">RFC-0045: Lowering NFT Deposits on Asset Hub</a></li><li class="chapter-item expanded "><a href="proposed/0046-metadata-for-offline-signers.html">RFC-0046: Metadata for Offline Signers</a></li><li class="chapter-item expanded "><a href="proposed/0061-allocator-inside-of-runtime.html">RFC-0061: Support allocator inside of runtime</a></li><li class="chapter-item expanded "><a href="proposed/0066-add-smartcontracts-to-assethub.html">RFC-0066: Add EVM+ink! Contracts Pallets to Asset Hub for Polkadot</a></li><li class="chapter-item expanded "><a href="proposed/0070-x-track-kusamanetwork.html">RFC-0070: X Track for @kusamanetwork</a></li><li class="chapter-item expanded "><a href="proposed/0073-referedum-deposit-track.html">RFC-0073: Decision Deposit Referendum Track</a></li><li class="chapter-item expanded "><a href="proposed/0074-stateful-multisig-pallet.html">RFC-0074: Stateful Multisig Pallet</a></li><li class="spacer"></li><li class="chapter-item expanded affix "><li class="part-title">Approved</li><li class="chapter-item expanded "><a href="approved/0001-agile-coretime.html">RFC-1: Agile Coretime</a></li><li class="chapter-item expanded "><a href="approved/0005-coretime-interface.html">RFC-5: Coretime Interface</a></li><li class="chapter-item expanded "><a href="approved/0007-system-collator-selection.html">RFC-0007: System Collator Selection</a></li><li class="chapter-item expanded "><a href="approved/0008-parachain-bootnodes-dht.html">RFC-0008: Store parachain bootnodes in relay chain DHT</a></li><li class="chapter-item expanded "><a href="approved/0010-burn-coretime-revenue.html">RFC-0010: Burn Coretime Revenue</a></li><li class="chapter-item expanded "><a href="approved/0012-process-for-adding-new-collectives.html">RFC-0012: Process for Adding New System Collectives</a></li><li class="chapter-item expanded "><a href="approved/0014-improve-locking-mechanism-for-parachains.html">RFC-0014: Improve locking mechanism for parachains</a></li><li class="chapter-item expanded "><a href="approved/0022-adopt-encointer-runtime.html">RFC-0022: Adopt Encointer Runtime</a></li><li class="chapter-item expanded "><a href="approved/0032-minimal-relay.html">RFC-0032: Minimal Relay</a></li><li class="chapter-item expanded "><a href="approved/0042-extrinsics-state-version.html">RFC-0042: Add System version that replaces StateVersion on RuntimeVersion</a></li><li class="chapter-item expanded "><a href="approved/0043-storage-proof-size-hostfunction.html">RFC-0043: Introduce storage_proof_size Host Function for Improved Parachain Block Utilization</a></li><li class="chapter-item expanded "><a href="approved/0047-assignment-of-availability-chunks.html">RFC-0047: Assignment of availability chunks to validators</a></li><li class="chapter-item expanded "><a href="approved/0050-fellowship-salaries.html">RFC-0050: Fellowship Salaries</a></li><li class="chapter-item expanded "><a href="approved/0056-one-transaction-per-notification.html">RFC-0056: Enforce only one transaction per notification</a></li><li class="chapter-item expanded "><a href="approved/0059-nodes-capabilities-discovery.html">RFC-0059: Add a discovery mechanism for nodes based on their capabilities</a></li><li class="spacer"></li><li class="chapter-item expanded affix "><li class="part-title">Stale</li><li class="chapter-item expanded "><a href="stale/0004-remove-unnecessary-allocator-usage.html">RFC-0004: Remove the host-side runtime memory allocator</a></li><li class="chapter-item expanded "><a href="stale/0006-dynamic-pricing-for-bulk-coretime-sales.html">RFC-0006: Dynamic Pricing for Bulk Coretime Sales</a></li><li class="chapter-item expanded "><a href="stale/0009-improved-net-light-client-requests.html">RFC-0009: Improved light client requests networking protocol</a></li><li class="chapter-item expanded "><a href="stale/0015-market-design-revisit.html">RFC-0015: Market Design Revisit</a></li><li class="chapter-item expanded "><a href="stale/0020-treasurer-track-confirmation-period-duration-modification.html">RFC-0020: Treasurer Track Confirmation Period Duration Modification</a></li><li class="chapter-item expanded "><a href="stale/0026-sassafras-consensus.html">RFC-0026: Sassafras Consensus Protocol</a></li><li class="chapter-item expanded "><a href="stale/0034-xcm-absolute-location-account-derivation.html">RFC-34: XCM Absolute Location Account Derivation</a></li><li class="chapter-item expanded "><a href="stale/0035-conviction-voting-delegation-modifications.html"> RFC-0035: Conviction Voting Delegation Modifications</a></li><li class="chapter-item expanded "><a href="stale/0044-rent-based-registration.html">RFC-0044: Rent based registration model</a></li><li class="chapter-item expanded "><a href="stale/0048-session-keys-runtime-api.html">RFC-0048: Generate ownership proof for SessionKeys</a></li><li class="chapter-item expanded "><a href="stale/0054-remove-heap-pages.html">RFC-0054: Remove the concept of "heap pages" from the client</a></li></ol>
</div>
<div id="sidebar-resize-handle" class="sidebar-resize-handle"></div>
</nav>
<!-- Track and set sidebar scroll position -->
<script>
var sidebarScrollbox = document.querySelector('#sidebar .sidebar-scrollbox');
sidebarScrollbox.addEventListener('click', function(e) {
if (e.target.tagName === 'A') {
sessionStorage.setItem('sidebar-scroll', sidebarScrollbox.scrollTop);
}
}, { passive: true });
var sidebarScrollTop = sessionStorage.getItem('sidebar-scroll');
sessionStorage.removeItem('sidebar-scroll');
if (sidebarScrollTop) {
// preserve sidebar scroll position when navigating via links within sidebar
sidebarScrollbox.scrollTop = sidebarScrollTop;
} else {
// scroll sidebar to current active section when navigating via "next/previous chapter" buttons
var activeSection = document.querySelector('#sidebar .active');
if (activeSection) {
activeSection.scrollIntoView({ block: 'center' });
}
}
</script>
<div id="page-wrapper" class="page-wrapper">
<div class="page">
<div id="menu-bar-hover-placeholder"></div>
<div id="menu-bar" class="menu-bar sticky">
<div class="left-buttons">
<label id="sidebar-toggle" class="icon-button" for="sidebar-toggle-anchor" title="Toggle Table of Contents" aria-label="Toggle Table of Contents" aria-controls="sidebar">
<i class="fa fa-bars"></i>
</label>
<button id="theme-toggle" class="icon-button" type="button" title="Change theme" aria-label="Change theme" aria-haspopup="true" aria-expanded="false" aria-controls="theme-list">
<i class="fa fa-paint-brush"></i>
</button>
<ul id="theme-list" class="theme-popup" aria-label="Themes" role="menu">
<li role="none"><button role="menuitem" class="theme" id="polkadot">Polkadot</button></li>
<li role="none"><button role="menuitem" class="theme" id="light">Light</button></li>
<li role="none"><button role="menuitem" class="theme" id="rust">Rust</button></li>
<li role="none"><button role="menuitem" class="theme" id="coal">Coal</button></li>
<li role="none"><button role="menuitem" class="theme" id="navy">Navy</button></li>
<li role="none"><button role="menuitem" class="theme" id="ayu">Ayu</button></li>
</ul>
<button id="search-toggle" class="icon-button" type="button" title="Search. (Shortkey: s)" aria-label="Toggle Searchbar" aria-expanded="false" aria-keyshortcuts="S" aria-controls="searchbar">
<i class="fa fa-search"></i>
</button>
</div>
<h1 class="menu-title">Polkadot Fellowship RFCs</h1>
<div class="right-buttons">
<a href="print.html" title="Print this book" aria-label="Print this book">
<i id="print-button" class="fa fa-print"></i>
</a>
</div>
</div>
<div id="search-wrapper" class="hidden">
<form id="searchbar-outer" class="searchbar-outer">
<input type="search" id="searchbar" name="searchbar" placeholder="Search this book ..." aria-controls="searchresults-outer" aria-describedby="searchresults-header">
</form>
<div id="searchresults-outer" class="searchresults-outer hidden">
<div id="searchresults-header" class="searchresults-header"></div>
<ul id="searchresults">
</ul>
</div>
</div>
<!-- Apply ARIA attributes after the sidebar and the sidebar toggle button are added to the DOM -->
<script>
document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible');
document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible');
Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) {
link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1);
});
</script>
<div id="content" class="content">
<main>
<p><img width="30%" src="images/Polkadot_Logo_Horizontal_Pink_Black.svg" alt="Polkadot logo" /></p>
<h1 id="introduction"><a class="header" href="#introduction">Introduction</a></h1>
<p>This book contains the Polkadot Fellowship Requests for Comments (RFCs)
detailing proposed changes to the technical implementation of the Polkadot network.</p>
<p><img width="2%" src="images/github-mark.svg" alt="GitHub logo" />&nbsp;<a href="https://github.com/polkadot-fellows/RFCs/">polkadot-fellows/RFCs</a></p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/76">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#rfc-0076-increase-maximum-length-of-identity-raw-data-values-from-32-bytes">RFC-0076: Increase maximum length of identity raw data values from 32 bytes</a>
<ul>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#summary">Summary</a></li>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#motivation">Motivation</a>
<ul>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#background">Background</a></li>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#problem">Problem</a></li>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#solution-requirements">Solution Requirements</a></li>
</ul>
</li>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#stakeholders">Stakeholders</a></li>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#explanation">Explanation</a></li>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#drawbacks">Drawbacks</a>
<ul>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#performance">Performance</a></li>
</ul>
</li>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#performance-1">Performance</a></li>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#ergonomics">Ergonomics</a></li>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#prior-art-and-references">Prior Art and References</a>
<ul>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#prior-art">Prior Art</a></li>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#references">References</a></li>
</ul>
</li>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="new/0076-increase-max-length-of-identity-raw-data-values.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0076-increase-maximum-length-of-identity-raw-data-values-from-32-bytes"><a class="header" href="#rfc-0076-increase-maximum-length-of-identity-raw-data-values-from-32-bytes">RFC-0076: Increase maximum length of identity raw data values from 32 bytes</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>20 Feb 2024</td></tr>
<tr><td><strong>Description</strong></td><td>Increase the maximum length of identity raw data values from 32 bytes</td></tr>
<tr><td><strong>Authors</strong></td><td>Luke Schoen</td></tr>
</tbody></table>
</div>
<h2 id="summary"><a class="header" href="#summary">Summary</a></h2>
<p>This proposes to increase the maximum length of identity raw data values from a 32 bytes/chars limit to either a 64 bytes/chars limit or a 128 bytes/chars limit.</p>
<h2 id="motivation"><a class="header" href="#motivation">Motivation</a></h2>
<h3 id="background"><a class="header" href="#background">Background</a></h3>
<p>At the moment if you upload a file and pin it to IPFS storage you get an IPFS Content Identifier (CID), which is the unique ID of the file in the IPFS Network. That CID may be used to retrieve the file again via a standard IPFS interface and gateway from anywhere.</p>
<p>CIDs are reasonably short regardless of the size the underlying content and are based on the cryptographic hash of the content.</p>
<p>IPFS providers may default to using and recommending using CIDv1 [0] when using SHA256 generates a CID 46 bytes long, rather than CIDv0 [0] that generates a CID 59 bytes long, however CIDv0 may be be the preferred choice for minting and storing NFTs on-chain since it is cheaper.</p>
<p>Further, the URI protocol + CID for CIDv0 (ipfs://Qm...) will be 7+46=53 chars while for CIDv1 (ipfs://baf...) it will be 7+59=66 chars [1], so neither CIDv0 nor CIDv1 have a CID that supports a maximum length of 32 bytes, which exceeds the maximum 32 bytes/chars limit for bytes/string fields of their Polkadot on-chain identity.</p>
<h3 id="problem"><a class="header" href="#problem">Problem</a></h3>
<p>If you want to set a Polkadot on-chain identity, users may provide raw data values of their email address &quot;email&quot; field, which may be longer than 32 bytes/chars (e.g. abcdefghijklmnopqrstuvwxyz@me.com or longer), and either just a CID or the URI protocol + CID associated with a legal document, as the value of the &quot;legal&quot; field, and the URI protocol + CID associated with the profile image to associated with their on-chain identity, as the value of the &quot;image&quot; field, however each field can only store a maximum length of 32 bytes/chars of information [3]. They may also want to set a custom value in the &quot;additional&quot; field, which currently only stores a maximum length of 32 bytes, since it currently has a <a href="https://github.com/paritytech/polkadot-sdk/blob/master/substrate/frame/identity/src/legacy.rs#L82C43-L82C53"><code>FieldLimit</code></a>.</p>
<p>Possible disadvantages of the current 32 bytes/chars limitation:</p>
<ul>
<li>Discourages users from using on-chain Web3 storage providers instead of Web2 storage providers to link to their on-chain identity. For example, in my Decentralized Voices application [4], since it is not possible to proactively provide a link in my on-chain identity to the IPFS CID associated with Conflict of Interest document that has been signed for integration with dApps or immediate verification by interested parties, it would instead be necessary to reactively share that IPFS CID upon each individual request from interested parties. </li>
<li>Encourages users to use Web2 storage providers and URL shorteners that may result in a plethora of on-chain profiles that have dead links.</li>
<li>Encourages dApps to use Web2 storage providers for their users, for example Polkassembly requesting users to upload a profile image that is stored in a Web2 storage provider rather than first defaulting to use the &quot;image&quot; field from their on-chain identity, since it may be the case that the &quot;image&quot; field of most on-chain identities is not widely used due to the maximum length of 32 bytes/chars restriction.</li>
<li>Discourages users from setting an on-chain identity by creating an extrinsic using Polkadot.js with <code>identity</code> &gt; <code>setIdentity(info)</code>, since if they try to provide their email address or website domain name that is longer than 32 characters, or try to use a IPFS CID, they will encounter an error.</li>
<li>Discourages users from using on-chain Web3 registrars to judge on-chain identity fields, where the shortest value they are able to generate is not less than or equal to the maximum length of 32 bytes.</li>
</ul>
<h3 id="solution-requirements"><a class="header" href="#solution-requirements">Solution Requirements</a></h3>
<p>The maximum length of identity raw data values should be increased from the current 32 bytes/chars limit at least a 59 bytes/chars limit (or a 66 bytes/chars limit) to support IPFS CIDs that are either CIDv0 or CIDv1.</p>
<p>They maximum length of &quot;additional&quot; field values should be increased by the same amount.</p>
<h2 id="stakeholders"><a class="header" href="#stakeholders">Stakeholders</a></h2>
<ul>
<li>Any Polkadot account holder wishing to use a Polkadot on-chain identity for their:
<ul>
<li>Email addresses that are longer than 32 characters</li>
<li>Website domain names that are longer than 32 characters</li>
<li>Files that are stored on the IPFS Network since associated CIDs are longer than 32 characters</li>
</ul>
</li>
</ul>
<h2 id="explanation"><a class="header" href="#explanation">Explanation</a></h2>
<p>If a user tries to setting an on-chain identity by creating an extrinsic using Polkadot.js with <code>identity</code> &gt; <code>setIdentity(info)</code>, then if they try to provide an email address or a website domain name that is longer than 32 characters, or try to use the IPFS CID (which may be associated with a file such as a document or image), then they will encounter this error:</p>
<pre><code>createType(Call):: Call: failed decoding identity.setIdentity:: Struct: failed on args: {...}:: Data.Raw values are limited to a maximum length of 32 bytes
</code></pre>
<p>Increasing maximum length of identity raw data values from the current 32 bytes/chars limit to at least a 59 bytes/chars limit (or a 66 bytes/chars limit) would overcome these errors and support IPFS CIDs that are either CIDv0 or CIDv1, satisfying the solution requirements.</p>
<p>Increasing the maximum length of &quot;additional&quot; field values would also overcome these errors and should be increased from the current 32 bytes/chars limit<a href="https://github.com/paritytech/polkadot-sdk/blob/master/substrate/frame/identity/src/legacy.rs#L82C43-L82C53"><code>FieldLimit</code></a> by the same amount.</p>
<h2 id="drawbacks"><a class="header" href="#drawbacks">Drawbacks</a></h2>
<h3 id="performance"><a class="header" href="#performance">Performance</a></h3>
<p>If Polkadot on-chain identities are able to store raw data values greater than the current maximum length of 32 bytes, then each identity may want to use the maximum (or more) amount of &quot;additional&quot; custom fields or more, which would impact storage and performance on the network.</p>
<h2 id="testing-security-and-privacy"><a class="header" href="#testing-security-and-privacy">Testing, Security, and Privacy</a></h2>
<p>Implementations would need to be tested for adherance by checking that IPFS CIDs that are either CIDv0 or CIDv1 are supported.</p>
<p>No effect on security or privacy has been identified than already exists.</p>
<p>No implementation pitfalls have been identified.</p>
<h2 id="performance-ergonomics-and-compatibility"><a class="header" href="#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-1"><a class="header" href="#performance-1">Performance</a></h3>
<p>It would be an optimization, since the associated exposed interfaces to developers and end-users could start being used.</p>
<p>To minimize additional overhead the proposal suggests at least a 59 bytes/chars limit (or a 66 bytes/chars limit) since that would at least provide support for IPFS CIDs that are either CIDv0 or CIDv1, satisfying the solution requirements.</p>
<h3 id="ergonomics"><a class="header" href="#ergonomics">Ergonomics</a></h3>
<p>It alters exposed interfaces to developers and end-users since they will now be able to provide IPFS CIDs that are either CIDv0 or CIDv1 as the values of Polkadot on-chain identity raw data input fields. Optionally 66 bytes/chars limit could be established to optimise for the usage pattern end-users, since that may be more intuitive to them, as it would allow users to provide a link (e.g. URI protocol + CID) rather than just the CID.</p>
<h3 id="compatibility"><a class="header" href="#compatibility">Compatibility</a></h3>
<p>Updates to Polkadot.js Apps, API and its documentation and those referring to it may be required.</p>
<h2 id="prior-art-and-references"><a class="header" href="#prior-art-and-references">Prior Art and References</a></h2>
<h3 id="prior-art"><a class="header" href="#prior-art">Prior Art</a></h3>
<p>No prior articles.</p>
<h3 id="references"><a class="header" href="#references">References</a></h3>
<ul>
<li><a href="https://docs.ipfs.tech/concepts/content-addressing/#cid-versions">1</a></li>
<li><a href="https://cardano.stackexchange.com/questions/9144/why-nfts-on-cardano-use-ipfs-cidv0-instead-of-recommended-cidv1">2</a></li>
<li><a href="https://support.polkadot.network/support/solutions/articles/65000181981-how-to-set-and-clear-an-identity">3</a></li>
<li><a href="https://forum.polkadot.network/t/decentralized-voices-program-luke-schoen/6111/7?u=ltfschoen">4</a></li>
</ul>
<h2 id="unresolved-questions"><a class="header" href="#unresolved-questions">Unresolved Questions</a></h2>
<p>Why can't we increase the maximum length of Polkadot on-chain identity raw data values and the &quot;additional&quot; field limit from 32 bytes/chars to an even longer maximum length than proposed of 128 bytes/chars?</p>
<h2 id="future-directions-and-related-material"><a class="header" href="#future-directions-and-related-material">Future Directions and Related Material</a></h2>
<p>Relates to RFC entitled &quot;Increase maximum length of identity PGP fingerprint values from 20 bytes&quot;.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/77">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html#rfc-0077-increase-maximum-length-of-identity-pgp-fingerprint-values-from-20-bytes">RFC-0077: Increase maximum length of identity PGP fingerprint values from 20 bytes</a>
<ul>
<li><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html#summary">Summary</a></li>
<li><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html#motivation">Motivation</a>
<ul>
<li><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html#background">Background</a></li>
<li><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html#problem">Problem</a></li>
<li><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html#solution-requirements">Solution Requirements</a></li>
</ul>
</li>
<li><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html#stakeholders">Stakeholders</a></li>
<li><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html#explanation">Explanation</a></li>
<li><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html#drawbacks">Drawbacks</a></li>
<li><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html#performance">Performance</a></li>
<li><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html#ergonomics">Ergonomics</a></li>
<li><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="new/0077-increase-max-length-of-identity-pgp-fingerprint-value.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0077-increase-maximum-length-of-identity-pgp-fingerprint-values-from-20-bytes"><a class="header" href="#rfc-0077-increase-maximum-length-of-identity-pgp-fingerprint-values-from-20-bytes">RFC-0077: Increase maximum length of identity PGP fingerprint values from 20 bytes</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>20 Feb 2024</td></tr>
<tr><td><strong>Description</strong></td><td>Increase the maximum length of identity PGP fingerprint values from 20 bytes</td></tr>
<tr><td><strong>Authors</strong></td><td>Luke Schoen</td></tr>
</tbody></table>
</div>
<h2 id="summary-1"><a class="header" href="#summary-1">Summary</a></h2>
<p>This proposes to increase the maximum length of PGP Fingerprint values from a 20 bytes/chars limit to a 40 bytes/chars limit.</p>
<h2 id="motivation-1"><a class="header" href="#motivation-1">Motivation</a></h2>
<h3 id="background-1"><a class="header" href="#background-1">Background</a></h3>
<p>Pretty Good Privacy (PGP) Fingerprints are shorter versions of their corresponding Public Key that may be printed on a business card.</p>
<p>They may be used by someone to validate the correct corresponding Public Key.</p>
<p>It should be possible to add PGP Fingerprints to Polkadot on-chain identities.</p>
<p>GNU Privacy Guard (GPG) is compliant with PGP and the two acronyms are used interchangeably.</p>
<h3 id="problem-1"><a class="header" href="#problem-1">Problem</a></h3>
<p>If you want to set a Polkadot on-chain identity, users may provide a PGP Fingerprint value in the &quot;pgpFingerprint&quot; field, which may be longer than 20 bytes/chars (e.g. PGP Fingerprints are 40 bytes/chars long), however that field can only store a maximum length of 20 bytes/chars of information.</p>
<p>Possible disadvantages of the current 20 bytes/chars limitation:</p>
<ul>
<li>Discourages users from using the &quot;pgpFingerprint&quot; field.</li>
<li>Discourages users from using Polkadot on-chain identities for Web2 and Web3 dApp software releases where the latest &quot;pgpFingerprint&quot; field could be used to verify the correct PGP Fingerprint that has been used to sign the software releases so users that download the software know that it was from a trusted source.</li>
<li>Encourages dApps to link to Web2 sources to allow their users verify the correct fingerprint associated with software releases, rather than to use the Web3 Polkadot on-chain identity &quot;pgpFingerprint&quot; field of the releaser of the software, since it may be the case that the &quot;pgpFingerprint&quot; field of most on-chain identities is not widely used due to the maximum length of 20 bytes/chars restriction.</li>
<li>Discourages users from setting an on-chain identity by creating an extrinsic using Polkadot.js with <code>identity</code> &gt; <code>setIdentity(info)</code>, since if they try to provide their 40 character long PGP Fingerprint or GPG Fingerprint, which is longer than the maximum length of 20 bytes/chars, they will encounter an error.</li>
<li>Discourages users from using on-chain Web3 registrars to judge on-chain identity fields, where the shortest value they are able to generate for a &quot;pgpFingerprint&quot; is not less than or equal to the maximum length of 20 bytes.</li>
</ul>
<h3 id="solution-requirements-1"><a class="header" href="#solution-requirements-1">Solution Requirements</a></h3>
<p>The maximum length of identity PGP Fingerprint values should be increased from the current 20 bytes/chars limit at least a 40 bytes/chars limit to support PGP Fingerprints and GPG Fingerprints.</p>
<h2 id="stakeholders-1"><a class="header" href="#stakeholders-1">Stakeholders</a></h2>
<ul>
<li>Any Polkadot account holder wishing to use a Polkadot on-chain identity for their:
<ul>
<li>PGP Fingerprints that are longer than 32 characters</li>
<li>GPG Fingerprints that are longer than 32 characters</li>
</ul>
</li>
</ul>
<h2 id="explanation-1"><a class="header" href="#explanation-1">Explanation</a></h2>
<p>If a user tries to setting an on-chain identity by creating an extrinsic using Polkadot.js with <code>identity</code> &gt; <code>setIdentity(info)</code>, then if they try to provide their 40 character long PGP Fingerprint or GPG Fingerprint, which is longer than the maximum length of 20 bytes/chars <code>[u8;20]</code>, then they will encounter this error:</p>
<pre><code>createType(Call):: Call: failed decoding identity.setIdentity:: Struct: failed on args: {...}:: Struct: failed on pgpFingerprint: Option&lt;[u8;20]&gt;:: Expected input with 20 bytes (160 bits), found 40 bytes
</code></pre>
<p>Increasing maximum length of identity PGP Fingerprint values from the current 20 bytes/chars limit to at least a 40 bytes/chars limit would overcome these errors and support PGP Fingerprints and GPG Fingerprints, satisfying the solution requirements.</p>
<h2 id="drawbacks-1"><a class="header" href="#drawbacks-1">Drawbacks</a></h2>
<p>No drawbacks have been identified.</p>
<h2 id="testing-security-and-privacy-1"><a class="header" href="#testing-security-and-privacy-1">Testing, Security, and Privacy</a></h2>
<p>Implementations would be tested for adherance by checking that 40 bytes/chars PGP Fingerprints are supported.</p>
<p>No effect on security or privacy has been identified than already exists.</p>
<p>No implementation pitfalls have been identified.</p>
<h2 id="performance-ergonomics-and-compatibility-1"><a class="header" href="#performance-ergonomics-and-compatibility-1">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-2"><a class="header" href="#performance-2">Performance</a></h3>
<p>It would be an optimization, since the associated exposed interfaces to developers and end-users could start being used.</p>
<p>To minimize additional overhead the proposal suggests a 40 bytes/chars limit since that would at least provide support for PGP Fingerprints, satisfying the solution requirements.</p>
<h3 id="ergonomics-1"><a class="header" href="#ergonomics-1">Ergonomics</a></h3>
<p>No potential ergonomic optimizations have been identified. </p>
<h3 id="compatibility-1"><a class="header" href="#compatibility-1">Compatibility</a></h3>
<p>Updates to Polkadot.js Apps, API and its documentation and those referring to it may be required.</p>
<h2 id="prior-art-and-references-1"><a class="header" href="#prior-art-and-references-1">Prior Art and References</a></h2>
<p>No prior articles or references.</p>
<h2 id="unresolved-questions-1"><a class="header" href="#unresolved-questions-1">Unresolved Questions</a></h2>
<p>No further questions at this stage.</p>
<h2 id="future-directions-and-related-material-1"><a class="header" href="#future-directions-and-related-material-1">Future Directions and Related Material</a></h2>
<p>Relates to RFC entitled &quot;Increase maximum length of identity raw data values from 32 bytes&quot;.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/78">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="new/0078-merkelized-metadata.html#rfc-0078-merkelized-metadata">RFC-0078: Merkelized Metadata</a>
<ul>
<li><a href="new/0078-merkelized-metadata.html#summary">Summary</a></li>
<li><a href="new/0078-merkelized-metadata.html#motivation">Motivation</a></li>
<li><a href="new/0078-merkelized-metadata.html#requirements">Requirements</a>
<ul>
<li><a href="new/0078-merkelized-metadata.html#reduce-metadata-size">Reduce metadata size</a></li>
</ul>
</li>
<li><a href="new/0078-merkelized-metadata.html#stakeholders">Stakeholders</a></li>
<li><a href="new/0078-merkelized-metadata.html#explanation">Explanation</a>
<ul>
<li><a href="new/0078-merkelized-metadata.html#metadata-digest">Metadata digest</a></li>
<li><a href="new/0078-merkelized-metadata.html#extrinsic-metadata">Extrinsic metadata</a></li>
<li><a href="new/0078-merkelized-metadata.html#type-information">Type Information</a></li>
<li><a href="new/0078-merkelized-metadata.html#prune-unrelated-types">Prune unrelated Types</a></li>
<li><a href="new/0078-merkelized-metadata.html#generating-typeref">Generating <code>TypeRef</code></a></li>
<li><a href="new/0078-merkelized-metadata.html#building-the-merkle-tree-root">Building the Merkle Tree Root</a></li>
<li><a href="new/0078-merkelized-metadata.html#inclusion-in-an-extrinsic">Inclusion in an Extrinsic</a></li>
</ul>
</li>
<li><a href="new/0078-merkelized-metadata.html#drawbacks">Drawbacks</a></li>
<li><a href="new/0078-merkelized-metadata.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="new/0078-merkelized-metadata.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="new/0078-merkelized-metadata.html#performance">Performance</a></li>
<li><a href="new/0078-merkelized-metadata.html#ergonomics--compatibility">Ergonomics &amp; Compatibility</a></li>
</ul>
</li>
<li><a href="new/0078-merkelized-metadata.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="new/0078-merkelized-metadata.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="new/0078-merkelized-metadata.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0078-merkelized-metadata"><a class="header" href="#rfc-0078-merkelized-metadata">RFC-0078: Merkelized Metadata</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>22 February 2024</td></tr>
<tr><td><strong>Description</strong></td><td>Include merkelized metadata hash in extrinsic signature for trust-less metadata verification.</td></tr>
<tr><td><strong>Authors</strong></td><td>Zondax AG, Parity Technologies</td></tr>
</tbody></table>
</div>
<h2 id="summary-2"><a class="header" href="#summary-2">Summary</a></h2>
<p>To interact with chains in the Polkadot ecosystem it is required to know how transactions are encoded and how to read state. For doing this, Polkadot-SDK, the framework used by most of the chains in the Polkadot ecosystem, exposes metadata about the runtime to the outside. UIs, wallets, and others can use this metadata to interact with these chains. This makes the metadata a crucial piece of the transaction encoding as users are relying on the interacting software to encode the transactions in the correct format.</p>
<p>It gets even more important when the user signs the transaction in an offline wallet, as the device by its nature cannot get access to the metadata without relying on the online wallet to provide it. This makes it so that the offline wallet either needs to <em>trust</em> an online party, deeming the security assumptions of the offline devices, mute. </p>
<p>This RFC proposes a way for offline wallets to leverage metadata, within the constraints of these. The design idea is that the metadata is chunked and these chunks are put into a merkle tree. The root hash of this merkle tree represents the metadata. The offline wallets can use the root hash to decode transactions by getting proofs for the individual chunks of the metadata. This root hash is also included in the signed data of the transaction (but not sent as part of the transaction). The runtime is then including its known metadata root hash when verifying the transaction. If the metadata root hash known by the runtime differs from the one that the offline wallet used, it very likely means that the online wallet provided some fake data and the verification of the transaction fails.</p>
<p>The user is depending on the offline wallet showing the correct decoded transaction before signing and with the merkelized metadata they can be sure that it was the correct transaction or the runtime will reject the transaction.</p>
<h2 id="motivation-2"><a class="header" href="#motivation-2">Motivation</a></h2>
<p>Polkadot's innovative design (both relay chain and parachains) present the ability to developers to upgrade their network as frequently as they need. These systems manage to have integrations working after the upgrades with the help of FRAME Metadata. This Metadata, which is in the order of half a MiB for most Polkadot-SDK chains, completely describes chain interfaces and properties. Securing this metadata is key for users to be able to interact with the Polkadot-SDK chain in the expected way.</p>
<p>On the other hand, offline wallets provide a secure way for Blockchain users to hold their own keys (some do a better job than others). These devices seldomly get upgraded, usually account for one particular network and hold very small internal memories. Currently in the Polkadot ecosystem there is no secure way of having these offline devices know the latest Metadata of the Polkadot-SDK chain they are interacting with. This results in a plethora of similar yet slightly different offline wallets for all different Polkadot-SDK chains, as well as the impediment of keeping these regularly updated, leveraging Polkadot-SDKs unique forkless upgrade ability.</p>
<p>The two main reasons why this is not possible today are:</p>
<ol>
<li><strong>Metadata is too large for offline devices</strong>. Currently Polkadot-SDK metadata is on average 500 KiB, which is more than what the mostly adopted offline devices can hold.</li>
<li><strong>Metadata is not authenticated</strong>. Even if there was enough space on offline devices to hold the metadata, the user would be trusting the entity providing this metadata to the hardware wallet. In the Polkadot ecosystem, this is how currently Polkadot Vault works.</li>
</ol>
<p><strong>This RFC proposes a solution to make FRAME Metadata compatible with offline signers in a secure way.</strong> This does not only ensure that offline devices can always keep up to date with every Polkadot-SDK chain, but at the same time every offline wallet will be compatible with all Polkadot-SDK chains at the same time, as it leverages FRAME metadata instead of relying on a custom per-chain implementation.</p>
<h2 id="requirements"><a class="header" href="#requirements">Requirements</a></h2>
<ol>
<li>Metadata's integrity MUST be preserved. If any compromise were to happen, extrinsics sent with compromised metadata SHOULD fail.</li>
<li>Metadata information that could be used in signable extrinsic decoding MAY be included in digest, yet its inclusion MUST be indicated in signed extensions.</li>
<li>Digest MUST be deterministic with respect to metadata.</li>
<li>Digest MUST be cryptographically strong against pre-image, both first (finding an input that results in given digest) and second (finding an input that results in same digest as some other input given).</li>
<li>Extra-metadata information necessary for extrinsic decoding and constant within runtime version MUST be included in digest.</li>
<li>It SHOULD be possible to quickly withdraw offline signing mechanism without access to cold signing devices.</li>
<li>Digest format SHOULD be versioned.</li>
<li>Work necessary for proving metadata authenticity MAY be omitted at discretion of signer device design (to support automation tools).</li>
</ol>
<h3 id="reduce-metadata-size"><a class="header" href="#reduce-metadata-size">Reduce metadata size</a></h3>
<p>Metadata should be stripped from parts that are not necessary to parse a signable extrinsic, then it should be separated into a finite set of self-descriptive chunks. Thus, a subset of chunks necessary for signable extrinsic decoding and rendering could be sent, possibly in small portions (ultimately, one at a time), to cold devices together with the proof.</p>
<ol>
<li>Single chunk with proof payload size SHOULD fit within few kB;</li>
<li>Chunks handling mechanism SHOULD support chunks being sent in any order without memory utilization overhead;</li>
<li>Unused enum variants MUST be stripped (this has great impact on transmitted metadata size; examples: era enum, enum with all calls for call batching).</li>
</ol>
<h2 id="stakeholders-2"><a class="header" href="#stakeholders-2">Stakeholders</a></h2>
<ul>
<li>Runtime implementors</li>
<li>UI/wallet implementors</li>
<li>Offline wallet implementors</li>
</ul>
<p>The idea for this RFC was brought up by runtime implementors and was extensively discussed with offline wallet implementors. It was designed in such a way that it can work easily with the existing offline wallet solutions in the Polkadot ecosystem.</p>
<h2 id="explanation-2"><a class="header" href="#explanation-2">Explanation</a></h2>
<p>The FRAME metadata provides a wide range of information about a FRAME based runtime. It contains information about the pallets, the calls per pallet, the storage entries per pallet, runtime APIs, and type information about most of the types that are used in the runtime. For decoding extrinsics on an offline wallet, what is mainly required is type information. Most of the other information in the FRAME metadata is actually not required for decoding extrinsics and thus it can be removed. Therefore, the following is a proposal on a custom representation of the metadata and how this custom metadata is chunked, ensuring that only the needed chunks required for decoding a particular extrinsic are sent to the offline wallet. The necessary information to transform the FRAME metadata type information into the type information presented in this RFC will be provided. However, not every single detail on how to convert from FRAME metadata into the RFC type information is provided.</p>
<p>First, the <code>MetadataDigest</code> is introduced. After that, <code>ExtrinsicMetadata</code> is covered and finally the actual format of the type information. Then pruning of unrelated type information is covered and how to generate the <code>TypeRef</code>s. In the latest step, merkle tree calculation is explained.</p>
<h3 id="metadata-digest"><a class="header" href="#metadata-digest">Metadata digest</a></h3>
<p>The metadata digest is the compact representation of the metadata. The hash of this digest is the <em>metadata hash</em>. Below the type declaration of the <code>Hash</code> type and the <code>MetadatDigest</code> itself can be found:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>type Hash = [u8; 32];
enum MetadataDigest {
#[index = 1]
V1 {
type_information_tree_root: Hash,
extrinsic_metadata_hash: Hash,
spec_version: u32,
spec_name: String,
base58_prefix: u16,
decimals: u8,
token_symbol: String,
},
}
<span class="boring">}</span></code></pre></pre>
<p>The <code>Hash</code> is 32 bytes long and <code>blake3</code> is used for calculating it. The hash of the <code>MetadataDigest</code> is calculated by <code>blake3(SCALE(MetadataDigest))</code>. Therefore, <code>MetadataDigest</code> is at first <code>SCALE</code> encoded, and then those bytes are hashed.</p>
<p>The <code>MetadataDigest</code> itself is represented as an <code>enum</code>. This is done to make it future proof, because a <code>SCALE</code> encoded <code>enum</code> is prefixed by the <code>index</code> of the variant. This <code>index</code> represents the version of the digest. As seen above, there is no <code>index</code> zero and it starts directly with one. Version one of the digest contains the following elements:</p>
<ul>
<li><code>type_information_tree_root</code>: The root of the <a href="new/0078-merkelized-metadata.html#merkelizing-type-information">Merkelized type information</a> tree.</li>
<li><code>extrinsic_metadata_hash</code>: The hash of the <a href="new/0078-merkelized-metadata.html#extrinsic-metadata">Extrinsic metadata</a>.</li>
<li><code>spec_version</code>: The <code>spec_version</code> of the runtime as found in the <code>RuntimeVersion</code> when generating the metadata. While this information can also be found in the metadata, it is hidden in a big blob of data. To avoid transferring this big blob of data, we directly add this information here.</li>
<li><code>spec_name</code>: Similar to <code>spec_version</code>, but being the <code>spec_name</code> found in the <code>RuntimeVersion</code>.</li>
<li><code>base58_prefix</code>: The <code>base58</code> prefix used for addresses.</li>
<li><code>decimals</code>: The number of decimals for the token.</li>
<li><code>token_symbol</code>: The symbol of the token.</li>
</ul>
<h3 id="extrinsic-metadata"><a class="header" href="#extrinsic-metadata">Extrinsic metadata</a></h3>
<p>For decoding an extrinsic, more information on what types are being used is required. The actual format of the extrinsic is the format as described in the <a href="https://spec.polkadot.network/id-extrinsics">Polkadot specification</a>. The metadata for an extrinsic is as follows:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>struct ExtrinsicMetadata {
version: u8,
address_ty: TypeRef,
call_ty: TypeRef,
signature_ty: TypeRef,
signed_extensions: Vec&lt;SignedExtensionMetadata&gt;,
}
struct SignedExtensionMetadata {
identifier: String,
included_in_extrinsic: TypeRef,
included_in_signed_data: TypeRef,
}
<span class="boring">}</span></code></pre></pre>
<p>To begin with, <code>TypeRef</code>. This is a unique identifier for a type as found in the type information. Using this <code>TypeRef</code>, it is possible to look up the type in the type information tree. More details on this process can be found in the section <a href="new/0078-merkelized-metadata.html#merkelizing-type-information">Merkelizing type information</a>.</p>
<p>The actual <code>ExtrinsicMetadata</code> contains the following information:</p>
<ul>
<li><code>version</code>: The version of the extrinsic format. As of writing this, the latest version is <code>4</code>.</li>
<li><code>address_ty</code>: The address type used by the chain.</li>
<li><code>call_ty</code>: The <code>call</code> type used by the chain. The <code>call</code> in FRAME based runtimes represents the type of transaction being executed on chain. It references the actual function to execute and the parameters of this function.</li>
<li><code>signature_ty</code>: The signature type used by the chain.</li>
<li><code>signed_extensions</code>: FRAME based runtimes can extend the base extrinsic with extra information. This extra information that is put into an extrinsic is called &quot;signed extensions&quot;. These extensions offer the runtime developer the possibility to include data directly into the extrinsic, like <code>nonce</code>, <code>tip</code>, amongst others. This means that the this data is sent alongside the extrinsic to the runtime. The other possibility these extensions offer is to include extra information only in the signed data that is signed by the sender. This means that this data needs to be known by both sides, the signing side and the verification side. An example for this kind of data is the <em>genesis hash</em> that ensures that extrinsics are unique per chain. Another example is the <em>metadata hash</em> itself that will also be included in the signed data. The offline wallets need to know which signed extensions are present in the chain and this is communicated to them using this field.</li>
</ul>
<p>The <code>SignedExtensionMetadata</code> provides information about a signed extension:</p>
<ul>
<li><code>identifier</code>: The <code>identifier</code> of the signed extension. An <code>identifier</code> is required to be unique in the Polkadot ecosystem as otherwise extrinsics are maybe built incorrectly.</li>
<li><code>included_in_extrinsic</code>: The type that will be included in the extrinsic by this signed extension.</li>
<li><code>included_in_signed_data</code>: The type that will be included in the signed data by this signed extension.</li>
</ul>
<h3 id="type-information"><a class="header" href="#type-information">Type Information</a></h3>
<p>As SCALE is not self descriptive like JSON, a decoder always needs to know the format of the type to decode it properly. This is where the type information comes into play. The format of the extrinsic is fixed as described above and <code>ExtrinsicMetadata</code> provides information on which type information is required for which part of the extrinsic. So, offline wallets only need access to the actual type information. It is a requirement that the type information can be chunked into logical pieces to reduce the amount of data that is sent to the offline wallets for decoding the extrinsics. So, the type information is structured in the following way:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>struct Type {
path: Vec&lt;String&gt;,
type_def: TypeDef,
}
enum TypeDef {
Composite(Vec&lt;Field&gt;),
Enumeration(EnumerationVariant),
Sequence(TypeRef),
Array(Array),
Tuple(Vec&lt;TypeRef&gt;),
BitSequence(BitSequence),
}
struct Field {
name: Option&lt;String&gt;,
ty: TypeRef,
type_name: Option&lt;String&gt;,
}
struct Array {
len: u32,
type_param: TypeRef,
}
struct BitSequence {
num_bytes: u8,
least_significant_bit_first: bool,
}
struct EnumerationVariant {
name: String,
fields: Vec&lt;Field&gt;,
index: u8,
}
enum TypeRef {
Bool,
Char,
Str,
U8,
U16,
U32,
U64,
U128,
U256,
I8,
I16,
I32,
I64,
I128,
I256,
CompactU8,
CompactU16,
CompactU32,
CompactU64,
CompactU128,
CompactVoid,
PerId(Compact&lt;u32&gt;),
}
<span class="boring">}</span></code></pre></pre>
<p>The <code>Type</code> declares the structure of a type. The <code>type</code> has the following fields:</p>
<ul>
<li><code>path</code>: A <code>path</code> declares the position of a type locally to the place where it is defined. The <code>path</code> is not globally unique, this means that there can be multiple types with the same <code>path</code>.</li>
<li><code>type_def</code>: The high-level type definition, e.g. the type is a composition of fields where each field has a type, the type is a composition of different types as <code>tuple</code> etc.</li>
</ul>
<p>Every <code>Type</code> is composed of multiple different types. Each of these &quot;sub types&quot; can reference either a full <code>Type</code> again or reference one of the primitive types. This is where <code>TypeRef</code> comes into play as the type referencing information. To reference a <code>Type</code> in the type information a unique identifier is used. As primitive types can be represented using a single byte, they are not put as separate types into the type information. Instead the primitive types are directly part of <code>TypeRef</code> to not require the overhead of referencing them in an extra <code>Type</code>. </p>
<p>The <code>TypeDef</code> variants have the following meaning:</p>
<ul>
<li><code>Composite</code>: A <code>struct</code> like type that is composed of multiple different fields. Each <code>Field</code> can have its own type.</li>
<li><code>Enumeration</code>: Stores a <code>EnumerationVariant</code>. A <code>EnumerationVariant</code> is a <code>struct</code> that is described by a <code>name</code>, an <code>index</code> and a <code>vector</code> of <code>Field</code>s, each of which can have it's own type. <code>Enumeration</code>s in most cases have more variants that only one and in this case the <code>Enumeration</code> appears multiple times, each time with a different variant, in the type information. As <code>Enumeration</code>s can get quite big and for decoding a type most of the time only one variant is required, this optimization is done to reduce the size of the proof.</li>
<li><code>Sequence</code>: A <code>vector</code> like type wrapping the given type.</li>
<li><code>BitSequence</code>: A <code>vector</code> storing bits. <code>num_bytes</code> represents the size in bytes of the internal storage. If <code>least_significant_bit_first</code> is <code>true</code> the least significant bit is first, otherwise the most significant bit is first.</li>
<li><code>Array</code>: A fixed-length array of a specific type.</li>
<li><code>Tuple</code>: A composition of multiple types.</li>
</ul>
<p>Using the type information together with the <a href="https://spec.polkadot.network/id-cryptography-encoding#sect-scale-codec">SCALE specification</a> provides enough information on how to decode types.</p>
<h3 id="prune-unrelated-types"><a class="header" href="#prune-unrelated-types">Prune unrelated Types</a></h3>
<p>The FRAME metadata contains not only the type information for decoding extrinsics, but it also contains type information about storage types. The scope of the RFC is only about decoding transactions on offline wallets. Thus, a lot of type information can be pruned. To know which type information are required to decode all possible extrinsics, <code>ExtrinsicMetadata</code> has been defined. The extrinsic metadata contains all the types that define the layout of an extrinsic. Therefore, all the types that are accessible from the types declared in the extrinsic metadata can be collected. This results in a list of types that are required to decode each possible extrinsic.</p>
<h3 id="generating-typeref"><a class="header" href="#generating-typeref">Generating <code>TypeRef</code></a></h3>
<p>Each <code>TypeRef</code> basically references one of the following types:</p>
<ul>
<li>One of the primitive types. All primitive types can be represented by 1 byte and thus, they are directly part of the <code>TypeRef</code> itself to remove an extra level of indirection.</li>
<li>A <code>Type</code> using its unique identifier.</li>
</ul>
<p>In FRAME metadata a primitive type is represented like any other type. So, the first step is to remove all the primitive only types from the list of types that were generated in the previous section. The resulting list of types is sorted using the <code>id</code> provided by FRAME metadata. In the last step the <code>TypeRef</code>s are created. Each reference to a primitive type is replaced by one of the corresponding <code>TypeRef</code> primitive type variants and every other reference is replaced by the type's unique identifier. The unique identifier of a type is the index of the type in our sorted list. For <code>Enumeration</code>s all variants have the same unique identifier, while they are represented as multiple type information. All variants need to have the same unique identifier as the reference doesn't know which variant will appear in the actual encoded data.</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>let pruned_types = get_pruned_types();
for ty in pruned_types {
if ty.is_primitive_type() {
pruned_types.remove(ty);
}
}
pruned_types.sort(|(left, right)|
if left.frame_metadata_id() == right.frame_metadata_id() {
left.variant_index() &lt; right.variant_index()
} else {
left.frame_metadata_id() &lt; right.frame_metadata_id()
}
);
fn generate_type_ref(ty, ty_list) -&gt; TypeRef {
if ty.is_primitive_type() {
TypeRef::primtive_from_ty(ty)
}
TypeRef::from_id(
// Determine the id by using the position of the type in the
// list of unique frame metadata ids.
ty_list.position_by_frame_metadata_id(ty.frame_metadata_id())
)
}
fn replace_all_sub_types_with_type_refs(ty, ty_list) -&gt; Type {
for sub_ty in ty.sub_types() {
replace_all_sub_types_with_type_refs(sub_ty, ty_list);
sub_ty = generate_type_ref(sub_ty, ty_list)
}
ty
}
let final_ty_list = Vec::new();
for ty in pruned_types {
final_ty_list.push(replace_all_sub_types_with_type_refs(ty, ty_list))
}
<span class="boring">}</span></code></pre></pre>
<h3 id="building-the-merkle-tree-root"><a class="header" href="#building-the-merkle-tree-root">Building the Merkle Tree Root</a></h3>
<p>A normal binary merkle tree with <code>blake3</code> as the hashing function is proposed. For building the merkle tree root, the initial data has to be hashed as a first step. This initial data is referred to as the <em>leaves</em> of the merkle tree. The leaves need to be sorted to make the tree root deterministic. The type information is sorted using their unique identifiers and for the <code>Enumeration</code>, variants are sort using their <code>index</code>. After sorting and hashing all leaves, two leaves have to be combined to one hash. The combination of these of two hashes is referred to as a <em>node</em>.</p>
<p>The initial list of <code>nodes</code> is the list of <code>leaves</code>.</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>let nodes = leaves;
while nodes.len() &gt; 1 {
let left = nodes.pop_front();
let right = nodes.pop_front();
nodes.push_back(blake3::hash(SCALE::encode((left, right))));
}
let merkle_tree_root = if nodes.is_empty() { [0u8; 32] } else { nodes.back() };
<span class="boring">}</span></code></pre></pre>
<p>The <code>merkle_tree_root</code> in the end is the last node left in the list of nodes. If there are no nodes in the list left, it means that the initial data set was empty. In this case, all zeros hash are used to represent the empty tree. </p>
<h3 id="inclusion-in-an-extrinsic"><a class="header" href="#inclusion-in-an-extrinsic">Inclusion in an Extrinsic</a></h3>
<p>To ensure that the offline wallet used the correct metadata to show the extrinsic to the user the metadata hash needs to be included in the extrinsic. The metadata hash is generated by hashing the SCALE encoded <code>MetadataDigest</code>:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>blake3::hash(SCALE::encode(MetadataDigest::V1 { .. }))
<span class="boring">}</span></code></pre></pre>
<p>For the runtime the metadata hash is generated at compile time. Wallets will have to generate the hash using the FRAME metadata. </p>
<p>The signing side should control whether it wants to add the metadata hash or if it wants to omit it. To accomplish this it is required to add one extra byte to the extrinsic itself. If this byte is <code>0</code> the metadata hash is not required and if the byte is <code>1</code> the metadata hash is added using <code>V1</code> of the <code>MetadataDigest</code>. This leaves room for future versions of the <code>MetadataDigest</code> format. When the metadata hash should be included, it is only added to the data that is signed. This brings the advantage of not requiring to include 32 bytes into the extrinsic itself, because the runtime knows the metadata hash as well and can add it to the signed data as well if required. This is similar to the genesis hash, while this isn't added conditionally to the signed data.</p>
<h2 id="drawbacks-2"><a class="header" href="#drawbacks-2">Drawbacks</a></h2>
<p>The chunking may not be the optimal case for every kind of offline wallet. </p>
<h2 id="testing-security-and-privacy-2"><a class="header" href="#testing-security-and-privacy-2">Testing, Security, and Privacy</a></h2>
<p>All implementations are required to strictly follow the RFC to generate the metadata hash. This includes which hash function to use and how to construct the metadata types tree. So, all implementations are following the same security criteria. As the chains will calculate the metadata hash at compile time, the build process needs to be trusted. However, this is already a solved problem in the Polkadot ecosystem by using reproducible builds. So, anyone can rebuild a chain runtime to ensure that a proposal is actually containing the changes as advertised.</p>
<p>Implementations can also be tested easily against each other by taking some metadata and ensuring that they all come to the same metadata hash.</p>
<p>Privacy of users should also not be impacted. This assumes that wallets will generate the metadata hash locally and don't leak any information to third party services about which chunks a user will send to their offline wallet. Besides that, there is no leak of private information as getting the raw metadata from the chain is an operation that is done by almost everyone.</p>
<h2 id="performance-ergonomics-and-compatibility-2"><a class="header" href="#performance-ergonomics-and-compatibility-2">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-3"><a class="header" href="#performance-3">Performance</a></h3>
<p>There should be no measurable impact on performance to Polkadot or any other chain using this feature. The metadata root hash is calculated at compile time and at runtime it is optionally used when checking the signature of a transaction. This means that at runtime no performance heavy operations are done. </p>
<h3 id="ergonomics--compatibility"><a class="header" href="#ergonomics--compatibility">Ergonomics &amp; Compatibility</a></h3>
<p>The proposal alters the way a transaction is built, signed, and verified. So, this imposes some required changes to any kind of developer who wants to construct transactions for Polkadot or any chain using this feature. As the developer can pass <code>0</code> for disabling the verification of the metadata root hash, it can be easily ignored.</p>
<h2 id="prior-art-and-references-2"><a class="header" href="#prior-art-and-references-2">Prior Art and References</a></h2>
<p><a href="https://github.com/polkadot-fellows/RFCs/pull/46">RFC 46</a> produce by the Alzymologist team is a previous work reference that goes in this direction as well.</p>
<p>On other ecosystems, there are other solutions to the problem of trusted signing. Cosmos for example has a standardized way of transforming a transaction into some textual representation and this textual representation is included in the signed data. Basically achieving the same as what the RFC proposes, but it requires that for every transaction applied in a block, every node in the network always has to generate this textual representation to ensure the transaction signature is valid.</p>
<h2 id="unresolved-questions-2"><a class="header" href="#unresolved-questions-2">Unresolved Questions</a></h2>
<p>None.</p>
<h2 id="future-directions-and-related-material-2"><a class="header" href="#future-directions-and-related-material-2">Future Directions and Related Material</a></h2>
<ul>
<li>Does it work with all kind of offline wallets?</li>
<li>Generic types currently appear multiple times in the metadata with each instantiation. It could be may be useful to have generic type only once in the metadata and declare the generic parameters at their instantiation. </li>
<li>The metadata doesn't contain any kind of semantic information. This means that the offline wallet for example doesn't know what is a balance etc. The current solution for this problem is to match on the type name, but this isn't a sustainable solution.</li>
<li><code>ExtrinsicMetadata</code> only provides one <code>token</code> and <code>decimal</code>. However, chains support a lot of chains support multiple tokens for paying fees etc. Probably more a question of having semantic information as mentioned above.</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/13">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="proposed/0013-prepare-blockbuilder-and-core-runtime-apis-for-mbms.html#rfc-0013-prepare-core-runtime-api-for-mbms">RFC-0013: Prepare <code>Core</code> runtime API for MBMs</a>
<ul>
<li><a href="proposed/0013-prepare-blockbuilder-and-core-runtime-apis-for-mbms.html#summary">Summary</a></li>
<li><a href="proposed/0013-prepare-blockbuilder-and-core-runtime-apis-for-mbms.html#motivation">Motivation</a></li>
<li><a href="proposed/0013-prepare-blockbuilder-and-core-runtime-apis-for-mbms.html#stakeholders">Stakeholders</a></li>
<li><a href="proposed/0013-prepare-blockbuilder-and-core-runtime-apis-for-mbms.html#explanation">Explanation</a>
<ul>
<li><a href="proposed/0013-prepare-blockbuilder-and-core-runtime-apis-for-mbms.html#coreinitialize_block"><code>Core::initialize_block</code></a></li>
</ul>
</li>
<li><a href="proposed/0013-prepare-blockbuilder-and-core-runtime-apis-for-mbms.html#drawbacks">Drawbacks</a></li>
<li><a href="proposed/0013-prepare-blockbuilder-and-core-runtime-apis-for-mbms.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="proposed/0013-prepare-blockbuilder-and-core-runtime-apis-for-mbms.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="proposed/0013-prepare-blockbuilder-and-core-runtime-apis-for-mbms.html#performance">Performance</a></li>
<li><a href="proposed/0013-prepare-blockbuilder-and-core-runtime-apis-for-mbms.html#ergonomics">Ergonomics</a></li>
<li><a href="proposed/0013-prepare-blockbuilder-and-core-runtime-apis-for-mbms.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="proposed/0013-prepare-blockbuilder-and-core-runtime-apis-for-mbms.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="proposed/0013-prepare-blockbuilder-and-core-runtime-apis-for-mbms.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="proposed/0013-prepare-blockbuilder-and-core-runtime-apis-for-mbms.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0013-prepare-core-runtime-api-for-mbms"><a class="header" href="#rfc-0013-prepare-core-runtime-api-for-mbms">RFC-0013: Prepare <code>Core</code> runtime API for MBMs</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>July 24, 2023</td></tr>
<tr><td><strong>Description</strong></td><td>Prepare the <code>Core</code> Runtime API for Multi-Block-Migrations</td></tr>
<tr><td><strong>Authors</strong></td><td>Oliver Tale-Yazdi</td></tr>
</tbody></table>
</div>
<h2 id="summary-3"><a class="header" href="#summary-3">Summary</a></h2>
<p>Introduces breaking changes to the <code>Core</code> runtime API by letting <code>Core::initialize_block</code> return an enum. The versions of <code>Core</code> is bumped from 4 to 5.</p>
<h2 id="motivation-3"><a class="header" href="#motivation-3">Motivation</a></h2>
<p>The main feature that motivates this RFC are Multi-Block-Migrations (MBM); these make it possible to split a migration over multiple blocks.<br />
Further it would be nice to not hinder the possibility of implementing a new hook <code>poll</code>, that runs at the beginning of the block when there are no MBMs and has access to <code>AllPalletsWithSystem</code>. This hook can then be used to replace the use of <code>on_initialize</code> and <code>on_finalize</code> for non-deadline critical logic.<br />
In a similar fashion, it should not hinder the future addition of a <code>System::PostInherents</code> callback that always runs after all inherents were applied.</p>
<h2 id="stakeholders-3"><a class="header" href="#stakeholders-3">Stakeholders</a></h2>
<ul>
<li>Substrate Maintainers: They have to implement this, including tests, audit and
maintenance burden.</li>
<li>Polkadot Runtime developers: They will have to adapt the runtime files to this breaking change.</li>
<li>Polkadot Parachain Teams: They have to adapt to the breaking changes but then eventually have
multi-block migrations available.</li>
</ul>
<h2 id="explanation-3"><a class="header" href="#explanation-3">Explanation</a></h2>
<h3 id="coreinitialize_block"><a class="header" href="#coreinitialize_block"><code>Core::initialize_block</code></a></h3>
<p>This runtime API function is changed from returning <code>()</code> to <code>ExtrinsicInclusionMode</code>:</p>
<pre><code class="language-patch">fn initialize_block(header: &amp;&lt;Block as BlockT&gt;::Header)
+ -&gt; ExtrinsicInclusionMode;
</code></pre>
<p>With <code>ExtrinsicInclusionMode</code> is defined as:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>enum ExtrinsicInclusionMode {
/// All extrinsics are allowed in this block.
AllExtrinsics,
/// Only inherents are allowed in this block.
OnlyInherents,
}
<span class="boring">}</span></code></pre></pre>
<p>A block author MUST respect the <code>ExtrinsicInclusionMode</code> that is returned by <code>initialize_block</code>. The runtime MUST reject blocks that have non-inherent extrinsics in them while <code>OnlyInherents</code> was returned.</p>
<p>Coming back to the motivations and how they can be implemented with this runtime API change:</p>
<p><strong>1. Multi-Block-Migrations</strong>: The runtime is being put into lock-down mode for the duration of the migration process by returning <code>OnlyInherents</code> from <code>initialize_block</code>. This ensures that no user provided transaction can interfere with the migration process. It is absolutely necessary to ensure this, otherwise a transaction could call into un-migrated storage and violate storage invariants.</p>
<p><strong>2. <code>poll</code></strong> is possible by using <code>apply_extrinsic</code> as entry-point and not hindered by this approach. It would not be possible to use a pallet inherent like <code>System::last_inherent</code> to achieve this for two reasons: First is that pallets do not have access to <code>AllPalletsWithSystem</code> which is required to invoke the <code>poll</code> hook on all pallets. Second is that the runtime does currently not enforce an order of inherents. </p>
<p><strong>3. <code>System::PostInherents</code></strong> can be done in the same manner as <code>poll</code>.</p>
<h2 id="drawbacks-3"><a class="header" href="#drawbacks-3">Drawbacks</a></h2>
<p>The previous drawback of cementing the order of inherents has been addressed and removed by redesigning the approach. No further drawbacks have been identified thus far.</p>
<h2 id="testing-security-and-privacy-3"><a class="header" href="#testing-security-and-privacy-3">Testing, Security, and Privacy</a></h2>
<p>The new logic of <code>initialize_block</code> can be tested by checking that the block-builder will skip transactions when <code>OnlyInherents</code> is returned.</p>
<p>Security: n/a</p>
<p>Privacy: n/a</p>
<h2 id="performance-ergonomics-and-compatibility-3"><a class="header" href="#performance-ergonomics-and-compatibility-3">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-4"><a class="header" href="#performance-4">Performance</a></h3>
<p>The performance overhead is minimal in the sense that no clutter was added after fulfilling the
requirements. The only performance difference is that <code>initialize_block</code> also returns an enum that needs to be passed through the WASM boundary. This should be negligible.</p>
<h3 id="ergonomics-2"><a class="header" href="#ergonomics-2">Ergonomics</a></h3>
<p>The new interface allows for more extensible runtime logic. In the future, this will be utilized for
multi-block-migrations which should be a huge ergonomic advantage for parachain developers.</p>
<h3 id="compatibility-2"><a class="header" href="#compatibility-2">Compatibility</a></h3>
<p>The advice here is OPTIONAL and outside of the RFC. To not degrade
user experience, it is recommended to ensure that an updated node can still import historic blocks.</p>
<h2 id="prior-art-and-references-3"><a class="header" href="#prior-art-and-references-3">Prior Art and References</a></h2>
<p>The RFC is currently being implemented in <a href="https://github.com/paritytech/polkadot-sdk/pull/1781">polkadot-sdk#1781</a> (formerly <a href="https://github.com/paritytech/substrate/pull/14275">substrate#14275</a>). Related issues and merge
requests:</p>
<ul>
<li><a href="https://github.com/paritytech/substrate/pull/14275">Simple multi block migrations</a></li>
<li><a href="https://github.com/paritytech/substrate/issues/9210">Execute a hook after inherent but before
transactions</a></li>
<li><a href="https://github.com/paritytech/substrate/issues/5757">There is no module hook after inherents and before
transactions</a></li>
</ul>
<h2 id="unresolved-questions-3"><a class="header" href="#unresolved-questions-3">Unresolved Questions</a></h2>
<p><del>Please suggest a better name for <code>BlockExecutiveMode</code>. We already tried: <code>RuntimeExecutiveMode</code>,
<code>ExtrinsicInclusionMode</code>. The names of the modes <code>Normal</code> and <code>Minimal</code> were also called
<code>AllExtrinsics</code> and <code>OnlyInherents</code>, so if you have naming preferences; please post them.</del><br />
=&gt; renamed to <code>ExtrinsicInclusionMode</code></p>
<p><del>Is <code>post_inherents</code> more consistent instead of <code>last_inherent</code>? Then we should change it.</del><br />
<del>=&gt; renamed to <code>last_inherent</code></del></p>
<h2 id="future-directions-and-related-material-3"><a class="header" href="#future-directions-and-related-material-3">Future Directions and Related Material</a></h2>
<p>The long-term future here is to move the block building logic into the runtime. Currently there is a tight dance between the block author and the runtime; the author has to call into different runtime functions in quick succession and exact order. Any misstep causes the block to be invalid.<br />
This can be unified and simplified by moving both parts into the runtime.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/45">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#rfc-0045-lowering-nft-deposits-on-asset-hub">RFC-0045: Lowering NFT Deposits on Asset Hub</a>
<ul>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#summary">Summary</a></li>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#motivation">Motivation</a>
<ul>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#requirements">Requirements</a></li>
</ul>
</li>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#stakeholders">Stakeholders</a></li>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#explanation">Explanation</a>
<ul>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#enhanced-approach-to-further-lower-barriers-for-entry">Enhanced Approach to Further Lower Barriers for Entry</a></li>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#short--and-long-term-plans">Short- and Long-Term Plans</a></li>
</ul>
</li>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#drawbacks">Drawbacks</a></li>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#testing-security-and-privacy">Testing, Security, and Privacy</a>
<ul>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#security-concerns">Security concerns</a></li>
</ul>
</li>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#performance">Performance</a></li>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#ergonomics">Ergonomics</a></li>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#addendum">Addendum</a>
<ul>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#enhanced-weak-governance-origin-model">Enhanced Weak Governance Origin Model</a></li>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#function-based-pricing-model">Function-Based Pricing Model</a></li>
<li><a href="proposed/0045-nft-deposits-asset-hub.html#linking-deposit-to-usdx-value">Linking Deposit to USD(x) Value</a></li>
</ul>
</li>
</ul>
</li>
</ul>
<h1 id="rfc-0045-lowering-nft-deposits-on-asset-hub"><a class="header" href="#rfc-0045-lowering-nft-deposits-on-asset-hub">RFC-0045: Lowering NFT Deposits on Asset Hub</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>2 November 2023</td></tr>
<tr><td><strong>Description</strong></td><td>A proposal to reduce the minimum deposit required for collection creation on the Polkadot and Kusama Asset Hubs.</td></tr>
<tr><td><strong>Authors</strong></td><td><a href="https://github.com/poppyseedDev">Aurora Poppyseed</a>, <a href="https://github.com/justLuuuu">Just_Luuuu</a>, <a href="https://github.com/vikiival">Viki Val</a>, <a href="https://github.com/joepetrowski">Joe Petrowski</a></td></tr>
</tbody></table>
</div>
<h2 id="summary-4"><a class="header" href="#summary-4">Summary</a></h2>
<p>This RFC proposes changing the current deposit requirements on the Polkadot and Kusama Asset Hub for
creating an NFT collection, minting an individual NFT, and lowering its corresponding metadata and
attribute deposits. The objective is to lower the barrier to entry for NFT creators, fostering a
more inclusive and vibrant ecosystem while maintaining network integrity and preventing spam.</p>
<h2 id="motivation-4"><a class="header" href="#motivation-4">Motivation</a></h2>
<p>The current deposit of 10 DOT for collection creation (along with 0.01 DOT for item deposit and 0.2
DOT for metadata and attribute deposits) on the Polkadot Asset Hub and 0.1 KSM on Kusama Asset Hub
presents a significant financial barrier for many NFT creators. By lowering the deposit
requirements, we aim to encourage more NFT creators to participate in the Polkadot NFT ecosystem,
thereby enriching the diversity and vibrancy of the community and its offerings.</p>
<p>The initial introduction of a 10 DOT deposit was an arbitrary starting point that does not consider
the actual storage footprint of an NFT collection. This proposal aims to adjust the deposit first to
a value based on the <code>deposit</code> function, which calculates a deposit based on the number of keys
introduced to storage and the size of corresponding values stored.</p>
<p>Further, it suggests a direction for a future of calculating deposits variably based on adoption
and/or market conditions. There is a discussion on tradeoffs of setting deposits too high or too
low.</p>
<h3 id="requirements-1"><a class="header" href="#requirements-1">Requirements</a></h3>
<ul>
<li>Deposits SHOULD be derived from <code>deposit</code> function, adjusted by correspoding pricing mechansim.</li>
</ul>
<h2 id="stakeholders-4"><a class="header" href="#stakeholders-4">Stakeholders</a></h2>
<ul>
<li><strong>NFT Creators</strong>: Primary beneficiaries of the proposed change, particularly those who found the
current deposit requirements prohibitive.</li>
<li><strong>NFT Platforms</strong>: As the facilitator of artists' relations, NFT marketplaces have a vested
interest in onboarding new users and making their platforms more accessible.</li>
<li><strong>dApp Developers</strong>: Making the blockspace more accessible will encourage developers to create and
build unique dApps in the Polkadot ecosystem.</li>
<li><strong>Polkadot Community</strong>: Stands to benefit from an influx of artists, creators, and diverse NFT
collections, enhancing the overall ecosystem.</li>
</ul>
<p>Previous discussions have been held within the <a href="https://forum.polkadot.network/t/polkadot-assethub-high-nft-collection-deposit/4262">Polkadot
Forum</a>, with
artists expressing their concerns about the deposit amounts.</p>
<h2 id="explanation-4"><a class="header" href="#explanation-4">Explanation</a></h2>
<p>This RFC proposes a revision of the deposit constants in the configuration of the NFTs pallet on the
Polkadot Asset Hub. The new deposit amounts would be determined by a standard deposit formula.</p>
<p>As of v1.1.1, the Collection Deposit is 10 DOT and the Item Deposit is 0.01 DOT (see
<a href="https://github.com/polkadot-fellows/runtimes/blob/v1.1.1/system-parachains/asset-hubs/asset-hub-polkadot/src/lib.rs#L687">here</a>).</p>
<p>Based on the storage footprint of these items, this RFC proposes changing them to:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>pub const NftsCollectionDeposit: Balance = system_para_deposit(1, 130);
pub const NftsItemDeposit: Balance = system_para_deposit(1, 164);
<span class="boring">}</span></code></pre></pre>
<p>This results in the following deposits (calculted using <a href="https://github.com/vikiival/rfc-pricing">this
repository</a>):</p>
<p><strong>Polkadot</strong></p>
<div class="table-wrapper"><table><thead><tr><th><strong>Name</strong></th><th style="text-align: center"><strong>Current Rate (DOT)</strong></th><th style="text-align: center"><strong>Calculated with Function (DOT)</strong></th></tr></thead><tbody>
<tr><td><code>collectionDeposit</code></td><td style="text-align: center">10</td><td style="text-align: center">0.20064</td></tr>
<tr><td><code>itemDeposit</code></td><td style="text-align: center">0.01</td><td style="text-align: center">0.20081</td></tr>
<tr><td><code>metadataDepositBase</code></td><td style="text-align: center">0.20129</td><td style="text-align: center">0.20076</td></tr>
<tr><td><code>attributeDepositBase</code></td><td style="text-align: center">0.2</td><td style="text-align: center">0.2</td></tr>
</tbody></table>
</div>
<p>Similarly, the prices for Kusama were calculated as:</p>
<p><strong>Kusama:</strong></p>
<div class="table-wrapper"><table><thead><tr><th><strong>Name</strong></th><th style="text-align: center"><strong>Current Rate (KSM)</strong></th><th style="text-align: center"><strong>Calculated with Function (KSM)</strong></th></tr></thead><tbody>
<tr><td><code>collectionDeposit</code></td><td style="text-align: center">0.1</td><td style="text-align: center">0.006688</td></tr>
<tr><td><code>itemDeposit</code></td><td style="text-align: center">0.001</td><td style="text-align: center">0.000167</td></tr>
<tr><td><code>metadataDepositBase</code></td><td style="text-align: center">0.006709666617</td><td style="text-align: center">0.0006709666617</td></tr>
<tr><td><code>attributeDepositBase</code></td><td style="text-align: center">0.00666666666</td><td style="text-align: center">0.000666666666</td></tr>
</tbody></table>
</div>
<h3 id="enhanced-approach-to-further-lower-barriers-for-entry"><a class="header" href="#enhanced-approach-to-further-lower-barriers-for-entry">Enhanced Approach to Further Lower Barriers for Entry</a></h3>
<p>This RFC proposes further lowering these deposits below the rate normally charged for such a storage
footprint. This is based on the economic argument that sub-rate deposits are a subsididy for growth
and adoption of a specific technology. If the NFT functionality on Polkadot gains adoption, it makes
it more attractive for future entrants, who would be willing to pay the non-subsidized rate because
of the existing community.</p>
<p><strong>Proposed Rate Adjustments</strong></p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>parameter_types! {
pub const NftsCollectionDeposit: Balance = system_para_deposit(1, 130);
pub const NftsItemDeposit: Balance = system_para_deposit(1, 164) / 40;
pub const NftsMetadataDepositBase: Balance = system_para_deposit(1, 129) / 10;
pub const NftsAttributeDepositBase: Balance = system_para_deposit(1, 0) / 10;
pub const NftsDepositPerByte: Balance = system_para_deposit(0, 1);
}
<span class="boring">}</span></code></pre></pre>
<p>This adjustment would result in the following DOT and KSM deposit values:</p>
<div class="table-wrapper"><table><thead><tr><th><strong>Name</strong></th><th style="text-align: center"><strong>Proposed Rate Polkadot</strong></th><th style="text-align: center"><strong>Proposed Rate Kusama</strong></th></tr></thead><tbody>
<tr><td><code>collectionDeposit</code></td><td style="text-align: center">0.20064 DOT</td><td style="text-align: center">0.006688 KSM</td></tr>
<tr><td><code>itemDeposit</code></td><td style="text-align: center">0.005 DOT</td><td style="text-align: center">0.000167 KSM</td></tr>
<tr><td><code>metadataDepositBase</code></td><td style="text-align: center">0.002 DOT</td><td style="text-align: center">0.0006709666617 KSM</td></tr>
<tr><td><code>attributeDepositBase</code></td><td style="text-align: center">0.002 DOT</td><td style="text-align: center">0.000666666666 KSM</td></tr>
</tbody></table>
</div>
<h3 id="short--and-long-term-plans"><a class="header" href="#short--and-long-term-plans">Short- and Long-Term Plans</a></h3>
<p>The plan presented above is recommended as an immediate step to make Polkadot a more attractive
place to launch NFTs, although one would note that a forty fold reduction in the Item Deposit is
just as arbitrary as the value it was replacing. As explained earlier, this is meant as a subsidy to
gain more momentum for NFTs on Polkadot.</p>
<p>In the long term, an implementation should account for what should happen to the deposit rates
assuming that the subsidy is successful and attracts a lot of deployments. Many options are
discussed in the <a href="proposed/0045-nft-deposits-asset-hub.html#addendum">Addendum</a>.</p>
<p>The deposit should be calculated as a function of the number of existing collections with maximum
DOT and stablecoin values limiting the amount. With asset rates available via the Asset Conversion
pallet, the system could take the lower value required. A sigmoid curve would make sense for this
application to avoid sudden rate changes, as in:</p>
<p>$$ minDeposit + \frac{\mathrm{min(DotDeposit, StableDeposit) - minDeposit} }{\mathrm{1 + e^{a - b * x}} }$$</p>
<p>where the constant <code>a</code> moves the inflection to lower or higher <code>x</code> values, the constant <code>b</code> adjusts
the rate of the deposit increase, and the independent variable <code>x</code> is the number of collections or
items, depending on application.</p>
<h2 id="drawbacks-4"><a class="header" href="#drawbacks-4">Drawbacks</a></h2>
<p>Modifying deposit requirements necessitates a balanced assessment of the potential drawbacks.
Highlighted below are cogent points extracted from the discourse on the <a href="https://forum.polkadot.network/t/polkadot-assethub-high-nft-collection-deposit/4262">Polkadot Forum
conversation</a>,
which provide critical perspectives on the implications of such changes.</p>
<p>Adjusting NFT deposit requirements on Polkadot and Kusama Asset Hubs involves key challenges:</p>
<ol>
<li>
<p><strong>State Growth and Technical Concerns</strong>: Lowering deposit requirements can lead to increased
blockchain state size, potentially causing state bloat. This growth needs to be managed to
prevent strain on the network's resources and maintain operational efficiency. As stated earlier,
the deposit levels proposed here are intentionally low with the thesis that future participants
would pay the standard rate.</p>
</li>
<li>
<p><strong>Network Security and Market Response</strong>: Adapting to the cryptocurrency market's volatility is
crucial. The mechanism for setting deposit amounts must be responsive yet stable, avoiding undue
complexity for users.</p>
</li>
<li>
<p><strong>Economic Impact on Previous Stakeholders</strong>: The change could have varied economic effects on
previous (before the change) creators, platform operators, and investors. Balancing these
interests is essential to ensure the adjustment benefits the ecosystem without negatively
impacting its value dynamics. However in the particular case of Polkadot and Kusama Asset Hub
this does not pose a concern since there are very few collections currently and thus previous
stakeholders wouldn't be much affected. As of date 9th January 2024 there are 42 collections on
Polkadot Asset Hub and 191 on Kusama Asset Hub with a relatively low volume.</p>
</li>
</ol>
<h2 id="testing-security-and-privacy-4"><a class="header" href="#testing-security-and-privacy-4">Testing, Security, and Privacy</a></h2>
<h3 id="security-concerns"><a class="header" href="#security-concerns">Security concerns</a></h3>
<p>As noted above, state bloat is a security concern. In the case of abuse, governance could adapt by
increasing deposit rates and/or using <code>forceDestroy</code> on collections agreed to be spam.</p>
<h2 id="performance-ergonomics-and-compatibility-4"><a class="header" href="#performance-ergonomics-and-compatibility-4">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-5"><a class="header" href="#performance-5">Performance</a></h3>
<p>The primary performance consideration stems from the potential for state bloat due to increased
activity from lower deposit requirements. It's vital to monitor and manage this to avoid any
negative impact on the chain's performance. Strategies for mitigating state bloat, including
efficient data management and periodic reviews of storage requirements, will be essential.</p>
<h3 id="ergonomics-3"><a class="header" href="#ergonomics-3">Ergonomics</a></h3>
<p>The proposed change aims to enhance the user experience for artists, traders, and utilizers of
Kusama and Polkadot Asset Hubs, making Polkadot and Kusama more accessible and user-friendly.</p>
<h3 id="compatibility-3"><a class="header" href="#compatibility-3">Compatibility</a></h3>
<p>The change does not impact compatibility as a <code>redeposit</code> function is already implemented.</p>
<h2 id="unresolved-questions-4"><a class="header" href="#unresolved-questions-4">Unresolved Questions</a></h2>
<p>If this RFC is accepted, there should not be any unresolved questions regarding how to adapt the
implementation of deposits for NFT collections.</p>
<h2 id="addendum"><a class="header" href="#addendum">Addendum</a></h2>
<p>Several innovative proposals have been considered to enhance the network's adaptability and manage
deposit requirements more effectively. The RFC recommends a mixture of the function-based model and
the stablecoin model, but some tradeoffs of each are maintained here for those interested.</p>
<h3 id="enhanced-weak-governance-origin-model"><a class="header" href="#enhanced-weak-governance-origin-model">Enhanced Weak Governance Origin Model</a></h3>
<p>The concept of a weak governance origin, controlled by a consortium like a system collective, has
been proposed. This model would allow for dynamic adjustments of NFT deposit requirements in
response to market conditions, adhering to storage deposit norms.</p>
<ul>
<li><strong>Responsiveness</strong>: To address concerns about delayed responses, the model could incorporate
automated triggers based on predefined market indicators, ensuring timely adjustments.</li>
<li><strong>Stability vs. Flexibility</strong>: Balancing stability with the need for flexibility is challenging.
To mitigate the issue of frequent changes in DOT-based deposits, a mechanism for gradual and
predictable adjustments could be introduced.</li>
<li><strong>Scalability</strong>: The model's scalability is a concern, given the numerous deposits across the
system. A more centralized approach to deposit management might be needed to avoid constant,
decentralized adjustments.</li>
</ul>
<h3 id="function-based-pricing-model"><a class="header" href="#function-based-pricing-model">Function-Based Pricing Model</a></h3>
<p>Another proposal is to use a mathematical function to regulate deposit prices, initially allowing
low prices to encourage participation, followed by a gradual increase to prevent network bloat.</p>
<ul>
<li><strong>Choice of Function</strong>: A logarithmic or sigmoid function is favored over an exponential one, as
these functions increase prices at a rate that encourages participation while preventing
prohibitive costs.</li>
<li><strong>Adjustment of Constants</strong>: To finely tune the pricing rise, one of the function's constants
could correlate with the total number of NFTs on Asset Hub. This would align the deposit
requirements with the actual usage and growth of the network.</li>
</ul>
<h3 id="linking-deposit-to-usdx-value"><a class="header" href="#linking-deposit-to-usdx-value">Linking Deposit to USD(x) Value</a></h3>
<p>This approach suggests pegging the deposit value to a stable currency like the USD, introducing
predictability and stability for network users.</p>
<ul>
<li><strong>Market Dynamics</strong>: One perspective is that fluctuations in native currency value naturally
balance user participation and pricing, deterring network spam while encouraging higher-value
collections. Conversely, there's an argument for allowing broader participation if the DOT/KSM
value increases.</li>
<li><strong>Complexity and Risks</strong>: Implementing a USD-based pricing system could add complexity and
potential risks. The implementation needs to be carefully designed to avoid unintended
consequences, such as excessive reliance on external financial systems or currencies.</li>
</ul>
<p>Each of these proposals offers unique advantages and challenges. The optimal approach may involve a
combination of these ideas, carefully adjusted to address the specific needs and dynamics of the
Polkadot and Kusama networks.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/46">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="proposed/0046-metadata-for-offline-signers.html#rfc-0046-metadata-for-offline-signers">RFC-0046: Metadata for Offline Signers</a>
<ul>
<li><a href="proposed/0046-metadata-for-offline-signers.html#summary">Summary</a></li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#motivation">Motivation</a>
<ul>
<li><a href="proposed/0046-metadata-for-offline-signers.html#background">Background</a></li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#solution-requirements">Solution requirements</a></li>
</ul>
</li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#stakeholders">Stakeholders</a></li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#explanation">Explanation</a>
<ul>
<li><a href="proposed/0046-metadata-for-offline-signers.html#definitions">Definitions</a></li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#general-flow">General flow</a></li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#metadata-modularization">Metadata modularization</a></li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#merging-protocol">Merging protocol</a></li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#complete-binary-merkle-tree-construction-protocol">Complete Binary Merkle Tree construction protocol</a></li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#digest">Digest</a></li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#chain-verification">Chain verification</a></li>
</ul>
</li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#drawbacks">Drawbacks</a>
<ul>
<li><a href="proposed/0046-metadata-for-offline-signers.html#increased-transaction-size">Increased transaction size</a></li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#transition-overhead">Transition overhead</a></li>
</ul>
</li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="proposed/0046-metadata-for-offline-signers.html#performance">Performance</a></li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#ergonomics">Ergonomics</a></li>
</ul>
</li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="proposed/0046-metadata-for-offline-signers.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0046-metadata-for-offline-signers"><a class="header" href="#rfc-0046-metadata-for-offline-signers">RFC-0046: Metadata for Offline Signers</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>2023-10-31</td></tr>
<tr><td><strong>Description</strong></td><td>Include merkelized metadata hash in extrinsic signature for trust-less metadata verification.</td></tr>
<tr><td><strong>Authors</strong></td><td>Alzymologist Oy, Zondax AG, Parity Technologies</td></tr>
</tbody></table>
</div>
<h2 id="summary-5"><a class="header" href="#summary-5">Summary</a></h2>
<p>To interact with chains in the Polkadot ecosystem it is required to know how transactions are encoded and how to read state. For doing this Substrate, the framework used by most of the chains in the Polkadot ecosystem, exposes metadata about the runtime to the outside. UIs, wallets, signers, etc can use this metadata to interact with the chains. This makes the metadata a crucial piece of the transaction encoding as users are relying on the interacting software to encode the transactions in the correct format. It gets even more important when the user signs the transaction on an offline signer as the latter by its nature can not get access to the metadata without relying on the online wallet to provide it the metadata. So, the idea is that the metadata is chunked and these chunks are put into a merkle tree. The root hash of this merkle tree represents the metadata. Cryptographically digested together small number of extra-metadata information, it allows the offline signers and wallets to decode transactions by getting proofs for the individual chunks of the metadata. This digest is also included into the signed data of the transaction (but not sent as part of the transaction). The runtime is then including its known metadata root hash and extra-metadata constants when verifying the transaction. If the digest known by the runtime differs from the one that the offline signer used, it very likely means that the online wallet provided some invalid data and the verification of the transaction fails thus making metadata substitution impossible. The user is depending on the offline wallet showing them the correct decoded transaction before signing and with the merkelized metadata they can be sure that it was the correct transaction or the runtime will reject the transaction.</p>
<p>Thus we propose following:</p>
<p>Add a metadata digest value to signed data to supplement signer party with proof of correct extrinsic interpretation. This would ensure that hardware wallets always use correct metadata to decode the information for the user.</p>
<p>The digest value is generated once before release and is well-known and deterministic. The digest mechanism is designed to be modular and flexible. It also supports partial metadata transfer as needed by the signing party's extrinsic decoding mechanism. This considers signing devices potentially limited communication bandwidth and/or memory capacity.</p>
<h2 id="motivation-5"><a class="header" href="#motivation-5">Motivation</a></h2>
<h3 id="background-2"><a class="header" href="#background-2">Background</a></h3>
<p>While all blockchain systems support (at least in some sense) offline signing used in air-gapped wallets and lightweight embedded devices, only few allow simultaneously complex upgradeable logic and full message decoding on the cold off-line signer side; Substrate is one of these heartening few, and therefore - we should build on this feature to greatly improve transaction security, and thus in general, network resilience.</p>
<p>As a starting point, it is important to recognise that prudence and due care are naturally required. As we build further reliance on this feature we should be very careful to make sure it works correctly every time so as not to create false sense of security.</p>
<p>In order to enable decoding that is small and optimized for chain storage transactions, a metadata entity is used, which is not at all small in itself (on the order of half-MB for most networks). This is a dynamic data chunk which completely describes chain interfaces and properties that could be made into a portable scale-encoded string for any given network version and passed along into an off-chain device to familiarize it with latest network updates. Of course, compromising this metadata anywhere in the path could result in differences between what user sees and signs, thus it is essential that we protect it.</p>
<p>Therefore, we have 2 problems to be solved:</p>
<ol>
<li>Metadata is large, takes long time to be passed into a cold storage device with memory insufficient for its storage; metadata SHOULD be shortened and transmission SHOULD be optimized.</li>
<li>Metadata authenticity SHOULD be ensured.</li>
</ol>
<p>As of now, there is no working solution for (1), as the whole metadata has to be passed to the device. On top of this, the solution for (2) heavily relies on a trusted party managing keys and ensuring metadata is indeed authentic: creating poorly decentralized points of potential failure.</p>
<h3 id="solution-requirements-2"><a class="header" href="#solution-requirements-2">Solution requirements</a></h3>
<h4 id="include-metadata-digest-into-signature"><a class="header" href="#include-metadata-digest-into-signature">Include metadata digest into signature</a></h4>
<p>Some cryptographically strong digest of metadata MAY be included into signable blob. There SHALL NOT be storage overhead for this blob, nor computational overhead; thus MUST be a constant within a given runtime, deterministically defined by metadata.</p>
<ul>
<li>Metadata information that could be used in signable extrinsic decoding MAY be included in digest, its inclusion MUST be indicated in signed extensions;</li>
<li>Digest MUST be deterministic with respect to metadata;</li>
<li>Digest MUST be cryptographically strong against pre-image, both first (finding an input that results in given digest) and second (finding an input that results in same digest as some other input given);</li>
<li>Extra-metadata information necessary for extrinsic decoding and constant within runtime version MUST be included in digest;</li>
<li>It SHOULD be possible to quickly withdraw offline signing mechanism without access to cold signing devices;</li>
<li>Digest format SHOULD be versioned.</li>
<li>Work necessary for proving metadata authenticity MAY be omitted at discretion of signer device design (to support automation tools).</li>
</ul>
<h4 id="reduce-metadata-size-1"><a class="header" href="#reduce-metadata-size-1">Reduce metadata size</a></h4>
<p>Metadata should be stripped from parts that are not necessary to parse a signable extrinsic, then it should be separated into a finite set of self-descriptive chunks. Thus, a subset of chunks necessary for signable extrinsic decoding and rendering could be sent, possibly in small portions (ultimately - one at a time), to cold device together with proof.</p>
<ul>
<li>Single chunk with proof payload size SHOULD fit within few kB;</li>
<li>Chunks handling mechanism SHOULD support chunks being sent in any order without memory utilization overhead;</li>
<li>Unused enum variants MUST be stripped (this has great impact on transmitted metadata size; examples: era enum, enum with all calls for call batching).</li>
</ul>
<h2 id="stakeholders-5"><a class="header" href="#stakeholders-5">Stakeholders</a></h2>
<ul>
<li>Runtime implementors</li>
<li>UI/wallet implementors</li>
<li>Offline wallet implementors</li>
</ul>
<p>All chain teams are stakeholders, as implementing this feature would require timely effort on their side and would impact compatibility with older tools.</p>
<p>This feature is essential for <strong>all</strong> offline signer tools; many regular signing tools might make use of it. In general, this RFC greatly improves security of any network implementing it, as many governing keys are used with offline signers.</p>
<p>Implementing this RFC would remove requirement to maintain metadata portals manually, as task of metadata verification would be effectively moved to consensus mechanism of the chain.</p>
<p>The idea for this RFC was brought up by runtime implementors and was extensively discussed with offline wallet implementors. It was designed in such a way that it can work easily with the existing offline wallet solutions in the Polkadot ecosystem.</p>
<h2 id="explanation-5"><a class="header" href="#explanation-5">Explanation</a></h2>
<p>The FRAME metadata provides a lot of information about a FRAME-based runtime. It contains information about the pallets, the calls per pallet, the storage entries per pallet, information about runtime APIs and type information about most of the types used in the runtime. For decoding transactions on an offline wallet, we mainly require this type of information. Most of the other information in the FRAME metadata is actually not required for the decoding, and thus, we can remove them. Thus, we have come up with a custom representation of the metadata and how this custom metadata is chunked, ensuring that we only need to send the chunks required for decoding a certain transaction to the offline wallet.</p>
<p>A reference implementation of this process is provided in <a href="https://docs.rs/metadata-shortener/latest/metadata_shortener/">metadata-shortener</a> crate.</p>
<h3 id="definitions"><a class="header" href="#definitions">Definitions</a></h3>
<h4 id="metadata-descriptor"><a class="header" href="#metadata-descriptor">Metadata descriptor</a></h4>
<p>Values for:</p>
<ol>
<li><code>u8</code> metadata shortening protocol version (as encoded enum variant), </li>
<li>The <code>Id</code> of type of the outermost Call enum (u32) as described in https://docs.rs/frame-metadata/latest/frame_metadata/v15/struct.ExtrinsicMetadata.html#structfield.call_ty</li>
<li>Vector of <code>SignedExtension</code> structs as defined in Metadata V15 (vector of named (<code>String</code>) pairs of <code>TypeId</code> (<code>u32</code>))</li>
<li><code>spec_version</code> <code>String</code> (see explanation below) as found in the <code>RuntimeVersion</code> at generation of the metadata. While this information can also be found in the metadata, it is hidden in a big blob of data. To avoid transferring this big blob of data, we directly add these values here,</li>
<li><code>spec_name</code> <code>String</code> as found in the <code>RuntimeVersion</code>,</li>
<li><code>u16</code> ss58 prefix,</li>
<li><code>u8</code> decimals value or <code>0u8</code> if no units are defined,</li>
<li><code>tokenSymbol</code> <code>String</code> defined on chain to identify the name of currency (available for example through <code>system.properties()</code> RPC call) or empty <code>String</code> if no base units are defined,</li>
</ol>
<pre><code>enum MetadataDescriptor {
V0,
V1(MetadataDescriptorV1),
}
struct MetadataDescriptorV1 {
call_ty: u32, // TypeId
signed_extensions: Vec&lt;SignedExtensionsMetadata&gt;,
spec_version: String,
spec_name: String,
ss58_prefix: u16,
decimals: u8,
token_symbol: String
}
struct SignedExtensionMetadata {
identifier: String,
ty: u32, // TypeId
additional_signed: u32, // TypeId
}
</code></pre>
<p>constitute metadata descriptor. This is minimal information that is, together with (shortened) types registry, sufficient to decode any signable transaction.</p>
<p><strong>Note on <code>spec_version</code></strong>: this type is described in metadata; it is indeed u32 in Polkadot and Kusama and most other networks and thus could be defined in a typesafe manner, but theoretically, it could be anything, as long as it is serializable. We saw a couple of networks using u16 there in the past. Not that it's all that important, but as long as our protocol would be the only limitation - it would be a limitation for our protocol; it would be unwise to enforce it here without first enforcing it upstream.</p>
<p>The version is not exactly an algebraic number - it has comparison operation, but no other algebra is defined; the primitive variable with these properties is really a <code>String</code>, not an integer. We've argued a lot about this internally, but with a naive JS-style solution, we have all the flexibility we might want and no real downsides (as from the point of view of our protocol, it's just an opaque constant - key for database search).</p>
<h4 id="merkle-tree"><a class="header" href="#merkle-tree">Merkle tree</a></h4>
<p>A <strong>Complete Binary Merkle Tree</strong> (<strong>CBMT</strong>) is proposed as digest structure.</p>
<p>Every node of the proposed tree has a 32-bit value.</p>
<p>A terminal node of the tree we call <strong>leaf</strong>. Its value is input for digest.</p>
<p>The top node of the tree we call <strong>root</strong>.</p>
<p>All node values for non-leave nodes are not terminal are computed through non-commutative <strong>merge</strong> procedure of child nodes.</p>
<p>In CBMT, all layers must be populated, except for the last one, that must have complete filling from the left.</p>
<p>Nodes are numbered top-down and left-to-right starting with 0 at the top of tree.</p>
<pre><code>Example 8-node tree
0
/ \
1 2
/ \ / \
3 4 5 6
/ \
7 8
Nodes 4, 5, 6, 7, 8 are leaves
Node 0 is root
</code></pre>
<h3 id="general-flow"><a class="header" href="#general-flow">General flow</a></h3>
<ol>
<li>The metadata is converted into lean modular form (vector of chunks)</li>
<li>A Merkle tree is constructed from the metadata chunks</li>
<li>A root of tree is merged with the hash of the <code>MetadataDescriptor</code></li>
<li>Resulting value is a constant to be included in <code>additionalSigned</code> to prove that the metadata seen by cold device is genuine</li>
</ol>
<h3 id="metadata-modularization"><a class="header" href="#metadata-modularization">Metadata modularization</a></h3>
<p>Structure of types in shortened metadata exactly matches structure of types in <code>scale-info</code> at MetadataV15 state, but <code>doc</code> field is always empty. This type system provides sufficient information for decoding with SCALE. For those not familiar with MetadataV15, a brief excerpt of types structure is presented below; however, it is important, that the protocol implemented in this RFC MUST agree directly to <code>scale-info</code> definitions, to ensure single source of truth and resolve possible diversions towards more adoptable and universal solution.</p>
<pre><code>struct Chunk {
id: u32,
type: Type,
}
struct Type {
path: Path, // vector of strings
type_params: Vec&lt;TypeParams&gt;,
type_def: TypeDef, // enum of various types
doc: Vec&lt;String&gt;,
}
struct TypeParams {
name: String,
ty: Option&lt;Type&gt;,
}
enum TypeDef {
Composite(TypeDefComposite),
Variant(TypeDefVariant),
Sequence(TypeDefSequence),
Array(TypeDefArray),
Tuple(TypeDefTuple),
Primitive(TypeDefPrimitive),
Compact(TypeDefCompact),
BitSequence(TypeDefBitSequence),
}
struct TypeDefComposite {
fields: Vec&lt;Field&gt;,
}
pub struct Field {
name: Option&lt;String&gt;,
ty: Type,
type_name: Option&lt;String&gt;,
docs: Vec&lt;String&gt;,
}
struct TypeDefVariant {
variants: Vec&lt;Variant&gt;,
}
struct Variant {
name: String,
fields: Vec&lt;Field&gt;,
index: u8,
docs: Vec&lt;String&gt;,
}
struct TypeDefSequence {
type_param: Type,
}
struct TypeDefArray {
len: u32,
type_param: Type,
}
struct TypeDefTuple {
fields: Vec&lt;Type&gt;,
}
enum TypeDefPrimitive {
bool,
char,
String,
u8,
u16,
u32,
u64,
u128,
u256,
i8,
i16,
i32,
i64,
i128,
i256,
}
struct TypeDefCompact {
type_param: Type,
}
struct TypeDefBitSequence {
pub bit_store_type: Type,
pub bit_order_type: Type,
}
</code></pre>
<p>Modularization happens thus:</p>
<ol>
<li>Types registry is stripped from <code>docs</code> fields.</li>
<li>Types records are separated into chunks, with enum variants being individual chunks differing by variant index; each chunk consisting of <code>id</code> (same as in full metadata registry) and SCALE-encoded 'Type' description (reduced to 1-variant enum for enum variants). Enums with 0 variants are treated as regular types.</li>
<li>Chunks are sorted by <code>id</code> in ascending order; chunks with same <code>id</code> are sorted by enum variant index in ascending order.</li>
</ol>
<pre><code>types_registry = metadataV15.types
modularized_registry = EmptyVector&lt;Chunk&gt;
for (id, type) in types.registry.iterate_enumerate {
type.doc = empty_vector
if (type is ReduceableEnum) { // false for 0-variant enums
for variant in type.variants.iterate {
variant_type = Type {
path: type.path,
type_params: empty_vector,
type_def: TypeDef::Variant(variants: [variant]),
docs: empty_vector,
}
modularized_registry.push(id, variant_type)
}
} else {
modularized_registry.push(id, type)
}
}
modularized_registry.sort(|a, b| {
if a.id == b.id { //only possible for variants
a.variant_index &gt; b.variant_index
} else { a.id &gt; b.id }
}
)
</code></pre>
<p>Chunks are scale-encoded and digested by <code>blake3</code> algorithm into 32-byte CBMT leaf values.</p>
<h3 id="merging-protocol"><a class="header" href="#merging-protocol">Merging protocol</a></h3>
<p><code>blake3</code> transformation of concatenated child nodes (<code>blake3(left + right)</code>) as merge procedure;</p>
<h3 id="complete-binary-merkle-tree-construction-protocol"><a class="header" href="#complete-binary-merkle-tree-construction-protocol">Complete Binary Merkle Tree construction protocol</a></h3>
<ol>
<li>Leaves are numbered in ascending order. Leaf index is associated with corresponding chunk.</li>
<li>Merge is performed using the leaf with highest index as right and leaf with second to highest index as left children; result is pushed to the end of nodes queue and leaves are discarded.</li>
<li>Step (2) is repeated until no leaves or just one leaf remains; in latter case, the last leaf is pushed to the front of the nodes queue.</li>
<li>Right node and then left node is popped from the front of the nodes queue and merged; the result is sent to the end of the queue.</li>
<li>Step (4) is repeated until only one node remains; this is tree root.</li>
</ol>
<pre><code>queue = empty_queue
while leaves.length &gt; 1 {
right = leaves.pop_last
left = leaves.pop_last
queue.push_back(merge(left, right))
}
if leaves.length == 1 {
queue.push_front(leaves.last)
}
while queue.len() &gt; 1 {
right = queue.pop_front
left = queue.pop_front
queue.push_back(merge(left, right))
}
return queue.pop
</code></pre>
<pre><code>Resulting tree for metadata consisting of 5 nodes (numbered from 0 to 4):
root
/ \
* *
/ \ / \
* 0 1 2
/ \
3 4
</code></pre>
<h3 id="digest"><a class="header" href="#digest">Digest</a></h3>
<ol>
<li>Blake3 hash is computed for each chunk of modular short metadata registry.</li>
<li>Complete Binary Merkle Tree is constructed as described above.</li>
<li>Root hash of this tree (left) is merged with metadata descriptor blake3 hash (right); this is metadata digest.</li>
</ol>
<p>Version number and corresponding resulting metadata digest MUST be included into Signed Extensions as specified in Chain Verification section below.</p>
<h3 id="chain-verification"><a class="header" href="#chain-verification">Chain verification</a></h3>
<p>The root of metadata computed by cold device MAY be included into Signed Extensions; if it is included, the transaction will pass as valid iff hash of metadata as seen by cold storage device is identical to consensus hash of metadata, ensuring fair signing protocol.</p>
<p>The Signed Extension representing metadata digest is a single byte representing both digest vaule inclusion and shortening protocol version; this MUST be included in Signed Extensions set. Depending on its value, a digest value is included as <code>additionalSigned</code> to signature computation according to following specification:</p>
<div class="table-wrapper"><table><thead><tr><th>signed extension value</th><th>digest value</th><th>comment</th></tr></thead><tbody>
<tr><td><code>0x00</code></td><td></td><td>digest is not included</td></tr>
<tr><td><code>0x01</code></td><td>32-byte digest</td><td>this represents protocol version 1</td></tr>
<tr><td><code>0x02</code> - <code>0xFF</code></td><td><em>reserved</em></td><td>reserved for future use</td></tr>
</tbody></table>
</div>
<h2 id="drawbacks-5"><a class="header" href="#drawbacks-5">Drawbacks</a></h2>
<h3 id="increased-transaction-size"><a class="header" href="#increased-transaction-size">Increased transaction size</a></h3>
<p>A 1-byte increase in transaction size due to signed extension value. Digest is not included in transferred transaction, only in signing process.</p>
<h3 id="transition-overhead"><a class="header" href="#transition-overhead">Transition overhead</a></h3>
<p>Some slightly out of spec systems might experience breaking changes as new content of signed extensions is added - tools that delay their transition instead of preparing ahead of time would break for the duration of delay. It is important to note, that there is no real overhead in processing time nor complexity, as the metadata checking mechanism is voluntary.</p>
<h2 id="testing-security-and-privacy-5"><a class="header" href="#testing-security-and-privacy-5">Testing, Security, and Privacy</a></h2>
<p>All implementations MUST follow the RFC to generate the metadata hash. This includes which hash function to use and how to construct the metadata types tree. So, all implementations are following the same security criteria. As the chains will calculate the metadata hash at compile time, the build process generally needs to be trusted. However, this is already a solved problem in the Polkadot ecosystem using reproducible builds. So, anyone can rebuild a chain runtime to ensure that a proposal is actually containing the changes as advertised.
Implementation can also be tested easily against each other by taking some metadata and ensuring that they all come to the same metadata hash.
Privacy of users should also not be impacted. This assumes that wallets will generate the metadata hash locally and don't leak any information to third party services about which chunks a user will send to its offline wallet. Besides that there is no leak of private information as getting the raw metadata from the chain is an operation that is done by almost everyone.</p>
<p>To be able to recall shortener protocol in case of vulnerability issues, a version byte is included.</p>
<h2 id="performance-ergonomics-and-compatibility-5"><a class="header" href="#performance-ergonomics-and-compatibility-5">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-6"><a class="header" href="#performance-6">Performance</a></h3>
<p>This is negligibly short pessimization during build time on the chain side. Cold wallets performance would improve mostly as metadata validity mechanism that was taking most of effort in cold wallet support would become trivial.</p>
<p>There should be no measurable impact on performance to Polkadot or any other chain using this feature. The metadata root hash is calculated at compile time and at runtime it is optionally used when checking the signature of a transaction. This means that at runtime no performance heavy operations are done.</p>
<h3 id="ergonomics-4"><a class="header" href="#ergonomics-4">Ergonomics</a></h3>
<p>The proposal alters the way a transaction is build, signed and verified. So, this imposes some required changes to any kind of developer that wants to construct transactions for Polkadot or any chain using this feature. As the developer can pass 0 for disabling the verification of the metadata root hash, it can be easily ignored.</p>
<h2 id="prior-art-and-references-4"><a class="header" href="#prior-art-and-references-4">Prior Art and References</a></h2>
<p>This project was developed upon a Polkadot Treasury grant; relevant development links are located in <a href="https://github.com/Alzymologist/metadata-offline-project">metadata-offline-project</a> repository.</p>
<p>There doesn't seem to exist a similar solution to what the RFC proposes right now. However, there are other solutions to the problem of trusted signing. Cosmos for example has a standardized way of transforming a transaction into some textual representation and this textual representation is included in the signed data. Basically achieving the same as what the RFC proposes, but it requires that for every transaction applied in a block, every node in the network always has to generate this textual representation to ensure the transaction signature is valid. Furthermore, since rendering of this textual representation is potentially non-trivial (with use of non-printed glyphs, direction control, escape characters, etc.), inclusion of arbitrary textual representation increases attack surface and potentially results in subtly false sense of security.</p>
<h2 id="unresolved-questions-5"><a class="header" href="#unresolved-questions-5">Unresolved Questions</a></h2>
<h2 id="future-directions-and-related-material-4"><a class="header" href="#future-directions-and-related-material-4">Future Directions and Related Material</a></h2>
<p>Changes to code of all cold signers to implement this mechanism SHOULD be done when this is enabled; non-cold signers may perform extra metadata check for better security. Ultimately, signing anything without decoding it with verifiable metadata should become discouraged in all situations where a decision-making mechanism is involved (that is, outside of fully automated blind signers like trade bots or staking rewards payout tools).</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/61">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="proposed/0061-allocator-inside-of-runtime.html#rfc-0061-support-allocator-inside-of-runtime">RFC-0061: Support allocator inside of runtime</a>
<ul>
<li><a href="proposed/0061-allocator-inside-of-runtime.html#summary">Summary</a></li>
<li><a href="proposed/0061-allocator-inside-of-runtime.html#motivation">Motivation</a></li>
<li><a href="proposed/0061-allocator-inside-of-runtime.html#stakeholders">Stakeholders</a></li>
<li><a href="proposed/0061-allocator-inside-of-runtime.html#explanation">Explanation</a>
<ul>
<li><a href="proposed/0061-allocator-inside-of-runtime.html#runtime-side-spec">Runtime side spec</a></li>
<li><a href="proposed/0061-allocator-inside-of-runtime.html#client-side-spec">Client side spec</a></li>
</ul>
</li>
<li><a href="proposed/0061-allocator-inside-of-runtime.html#drawbacks">Drawbacks</a></li>
<li><a href="proposed/0061-allocator-inside-of-runtime.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="proposed/0061-allocator-inside-of-runtime.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="proposed/0061-allocator-inside-of-runtime.html#performance">Performance</a></li>
<li><a href="proposed/0061-allocator-inside-of-runtime.html#ergonomics">Ergonomics</a></li>
<li><a href="proposed/0061-allocator-inside-of-runtime.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="proposed/0061-allocator-inside-of-runtime.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="proposed/0061-allocator-inside-of-runtime.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="proposed/0061-allocator-inside-of-runtime.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0061-support-allocator-inside-of-runtime"><a class="header" href="#rfc-0061-support-allocator-inside-of-runtime">RFC-0061: Support allocator inside of runtime</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>27 December 2023</td></tr>
<tr><td><strong>Description</strong></td><td>Supporting runtime built-in allocator makes the substrate runtime more versatile</td></tr>
<tr><td><strong>Authors</strong></td><td>Jiahao Ye</td></tr>
</tbody></table>
</div>
<h2 id="summary-6"><a class="header" href="#summary-6">Summary</a></h2>
<p>Currently, substrate runtime use an simple allocator defined by host side. Every runtime MUST
import these allocator functions for normal execution. This situation make runtime code not versatile enough.</p>
<p>So this RFC proposes to define a new spec for allocator part to make substrate runtime more generic.</p>
<h2 id="motivation-6"><a class="header" href="#motivation-6">Motivation</a></h2>
<p>Since this RFC define a new way for allocator, we now regard the old one as <code>legacy</code> allocator.
As we all know, since the allocator implementation details are defined by the substrate client, parachain/parathread cannot customize memory allocator algorithm, so the new specification allows the runtime to customize memory allocation, and then export the allocator function according to the specification for the client side to use.
Another benefit is that some new host functions can be designed without allocating memory on the client, which may have potential performance improvements. Also it will help provide a unified and clean specification if substrate runtime support multi-targets(e.g. RISC-V).
There is also a potential benefit. Many programming languages that support compilation to wasm may not be friendly to supporting external allocator. This is beneficial for other programming languages ​​to enter the substrate runtime ecosystem.
The last and most important benefit is that for offchain context execution, the runtime can fully support pure wasm. What this means here is that all imported host functions could not actually be called (as stub functions), then the various verification logic of the runtime can be converted into pure wasm, which provides the possibility for the substrate runtime to run block verification in other environments (such as in browsers and other non-substrate environments).</p>
<h2 id="stakeholders-6"><a class="header" href="#stakeholders-6">Stakeholders</a></h2>
<p>No attempt was made at convincing stakeholders.</p>
<h2 id="explanation-6"><a class="header" href="#explanation-6">Explanation</a></h2>
<h3 id="runtime-side-spec"><a class="header" href="#runtime-side-spec">Runtime side spec</a></h3>
<p>This section contains a list of functions should be exported by substrate runtime.</p>
<p>We define the spec as version 1, so the following <code>dummy</code> function <code>v1</code> MUST be exported to hint
client that runtime is using version 1 spec, otherwise rollback to <code>legacy</code> allocator.
The function should never be used, and its name is only for version checking.</p>
<pre><code class="language-wat"> (export &quot;v1&quot; (func $v1))
</code></pre>
<p>Choose this way is more generic than custom section since many other tools do not support custom section very well. But if an environment want to run it, it should always be possible to parse
the export section.</p>
<p>The allocator functions are:</p>
<pre><code class="language-wat">(export &quot;alloc&quot; (func $alloc))
(export &quot;dealloc&quot; (func $dealloc))
(export &quot;realloc&quot; (func $realloc))
</code></pre>
<p>Their signatures are:</p>
<pre><code class="language-wat">(func $alloc (param $size i32) (result i32))
(func $dealloc (param $addr i32) (param $size i32))
(func $realloc (param $addr i32) (param $size i32) (param $new_size i32) (result i32))
</code></pre>
<p>Note: <code>dealloc</code>/<code>realloc</code> is not used in substrate side currently, but for the functional integrity.</p>
<p>The following imports are disabled.</p>
<p>The two kind of allocators(<code>leagcy</code> and <code>v1</code>) cannot know each other, and importing them will cause abnormal memory allocation.</p>
<pre><code class="language-wat">(import &quot;env&quot; &quot;ext_allocator_free_version_1&quot; (func $ext_allocator_free_version_1 (type 0)))
(import &quot;env&quot; &quot;ext_allocator_malloc_version_1&quot; (func $ext_allocator_malloc_version_1 (type 1)))
</code></pre>
<p>The following export could be removed. The client side no need to know heap base.</p>
<pre><code class="language-wat">(export &quot;__heap_base&quot; (global 2))
</code></pre>
<h3 id="client-side-spec"><a class="header" href="#client-side-spec">Client side spec</a></h3>
<p>During instantiating time, add a version checking stage for wasm executor before any other wasm module checking.
Check if parsed wasm module contains a exported <code>v1</code> function:</p>
<ul>
<li>If not exist, we predicate it using legacy allocator, just do normal checking like before. Set legacy allocator be <code>Some</code> while set <code>v1</code> allocator be <code>None</code>.</li>
<li>If exist, we predicate it using <code>v1</code> allocator. And then we lookup and hold the exported <code>alloc</code> function for the total lifestyle of instance, return error if not exist. Set legacy allocator be <code>None</code> while set <code>v1</code> allocator be <code>Some</code>.</li>
<li>When wasm host functions or other entrypoint call(e.g. <code>runtime_apis</code>/<code>validate_block</code>) need to allocate memory, check if instance hold the <code>alloc</code>, if hold just call it otherwise call the legacy
allocator.</li>
</ul>
<p>Detail-heavy explanation of the RFC, suitable for explanation to an implementer of the changeset. This should address corner cases in detail and provide justification behind decisions, and provide rationale for how the design meets the solution requirements.</p>
<h2 id="drawbacks-6"><a class="header" href="#drawbacks-6">Drawbacks</a></h2>
<p>The allocator inside of the runtime will make code size bigger, but it's not obvious.
The allocator inside of the runtime maybe slow down(or speed up) the runtime, still not obvious.</p>
<p>We could ignore these drawbacks since they are not prominent. And the execution efficiency is highly decided by runtime developer. We could not prevent a poor efficiency if developer want to do it.</p>
<h2 id="testing-security-and-privacy-6"><a class="header" href="#testing-security-and-privacy-6">Testing, Security, and Privacy</a></h2>
<p>Keep the legacy allocator runtime test cases, and add new feature to compile test cases for <code>v1</code> allocator spec. And then update the test asserts.</p>
<p>Update template runtime to enable <code>v1</code> spec. Once the dev network runs well, it seems that the spec is implmented correctly.</p>
<h2 id="performance-ergonomics-and-compatibility-6"><a class="header" href="#performance-ergonomics-and-compatibility-6">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-7"><a class="header" href="#performance-7">Performance</a></h3>
<p>As the above says, not obvious impact about performance. And <code>polkadot-sdk</code> could offer the best practice allocator for all chains.
Third party also could customized by theirself. So the performance could be improved over time.</p>
<h3 id="ergonomics-5"><a class="header" href="#ergonomics-5">Ergonomics</a></h3>
<p>Only for runtime developer, Just need to import a new crate and enable a new feature. Maybe it's convienient for other wasm-target language to implment.</p>
<h3 id="compatibility-4"><a class="header" href="#compatibility-4">Compatibility</a></h3>
<p>It's 100% compatible. Only Some runtime configs and executor configs need to be depreacted.</p>
<p>For support new runtime spec, we MUST upgrade the client binary to support new spec of client part firstly.</p>
<p>We SHALL add an optional primtive crate to enable the version 1 spec and disable the legacy allocator by cargo feature.
For the first year, we SHALL disable the v1 by default, and enable it by default start in the next year.</p>
<h2 id="prior-art-and-references-5"><a class="header" href="#prior-art-and-references-5">Prior Art and References</a></h2>
<ul>
<li><a href="https://github.com/paritytech/substrate/issues/11883">Move the allocator inside of the runtime</a></li>
<li><a href="https://github.com/paritytech/polkadot-sdk/pull/1658">Add new allocator design</a></li>
</ul>
<h2 id="unresolved-questions-6"><a class="header" href="#unresolved-questions-6">Unresolved Questions</a></h2>
<p>None at this time.</p>
<h2 id="future-directions-and-related-material-5"><a class="header" href="#future-directions-and-related-material-5">Future Directions and Related Material</a></h2>
<p>The content discussed with <a href="https://github.com/polkadot-fellows/RFCs/pull/4">RFC-0004</a> is basically orthogonal, but it could still be considered together, and it is preferred that this rfc be implmentented first.</p>
<p>This feature could make substrate runtime be easier supported by other languages and integreted into other ecosystem.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/66">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#rfc-0066-add-evmink-contracts-pallets-to-asset-hub-for-polkadot">RFC-0066: Add EVM+ink! Contracts Pallets to Asset Hub for Polkadot</a>
<ul>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#summary">Summary</a></li>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#motivation">Motivation</a>
<ul>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#definft-applications-need-smart-contracts-on-assethub">Defi+NFT Applications need Smart Contracts on AssetHub</a></li>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#rollups-need-smart-contracts-on-assethub">Rollups need Smart Contracts on AssetHub</a></li>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#ink-on-assethub-will-lead-to-coreplay-developers--on-assethub">ink! on AssetHub will lead to CorePlay Developers on AssetHub</a></li>
</ul>
</li>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#stakeholders">Stakeholders</a></li>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#explanation">Explanation</a>
<ul>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#limit-smart-contract-weight-allocation">Limit Smart Contract Weight allocation</a></li>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#model-assethub-assets-inside-evm-smart-contracts-based-on-astar">Model AssetHub Assets inside EVM Smart Contracts based on Astar</a></li>
</ul>
</li>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#new-dot-revenue-sources">New DOT Revenue Sources</a>
<ul>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#new-revenue-from-assethub-evm-contracts">New Revenue from AssetHub EVM Contracts</a></li>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#new-revenue-for-inkcoreplay-contracts">New Revenue for ink!/Coreplay Contracts</a></li>
</ul>
</li>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#drawbacks-and-tradeoffs">Drawbacks and Tradeoffs</a></li>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#performance">Performance</a></li>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#ergonomics">Ergonomics</a></li>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="proposed/0066-add-smartcontracts-to-assethub.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0066-add-evmink-contracts-pallets-to-asset-hub-for-polkadot"><a class="header" href="#rfc-0066-add-evmink-contracts-pallets-to-asset-hub-for-polkadot">RFC-0066: Add EVM+ink! Contracts Pallets to Asset Hub for Polkadot</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>14 January 2024</td></tr>
<tr><td><strong>Description</strong></td><td>A proposal to add <em>EVM</em>+<em>ink!</em> Contracts to Asset Hub for Polkadot to support <strong>Polkadot Rollups</strong> and larger numbers of <em>EVM/Coreplay</em> smart contract developers and their users on <strong>Polkadot Rollups</strong> <em>and</em> <strong>AssetHub for Polkadot</strong>.</td></tr>
<tr><td><strong>Authors</strong></td><td><a href="https://github.com/sourabhniyogi">Sourabh Niyogi</a></td></tr>
</tbody></table>
</div>
<h2 id="summary-7"><a class="header" href="#summary-7">Summary</a></h2>
<p>This RFC proposes to add the two dominant smart contract programming languages in the Polkadot ecosystem to AssetHub:
EVM + ink!/Coreplay. The objective is to increase DOT Revenue by making AssetHub accessible to
(1) Polkadot Rollups;
(2) EVM smart contract programmers;
(3) Coreplay programmers who will benefit from easier-to-use smart contract environments.<br />
These changes in AssetHub are enabled by key Polkadot 2.0 technologies:
PolkaVM supporting Coreplay, and
hyper data availability in Blobs Chain.</p>
<h2 id="motivation-7"><a class="header" href="#motivation-7">Motivation</a></h2>
<p>EVM Contracts are pervasive in the Web3 blockchain ecosystem,
while Polkadot 2.0's Coreplay aims to surpass EVM Contracts in ease-of-use using PolkaVM's RISC architecture.</p>
<p>Asset Hub for Polkadot does <em>not</em> have smart contract capabilities,
even though dominant stablecoin assets such as USDC and USDT are originated there.<br />
In addition, in the <a href="https://github.com/polkadot-fellows/RFCs/blob/main/text/0032-minimal-relay.md">RFC #32 - Minimal Relay Chain architecture</a>,
DOT balances are planned to be shifted to Asset Hub, to support Polkadot 2.0's
<a href="https://github.com/polkadot-fellows/RFCs/blob/gav-coreplay/text/coreplay.md">CoreJam map-reduce architecture</a>.<br />
In this 2.0 architecture, there is no room for synchronous contracts on the Polkadot relay chain --
doing so would waste precious resources that should be dedicated to sync+async composability.<br />
However, while Polkadot fellows have concluded the Polkadot relay chain should <em>not</em> support
synchronous smart contracts, this is <em>not</em> applicable to AssetHub for Polkadot.</p>
<p>The following sections argue for the need for Smart Contracts on AssetHub.</p>
<h3 id="definft-applications-need-smart-contracts-on-assethub"><a class="header" href="#definft-applications-need-smart-contracts-on-assethub">Defi+NFT Applications need Smart Contracts on AssetHub</a></h3>
<p>EVM Smart Contract chains within Polkadot and outside are dominated by defi + NFT applications. While the assetConversion pallet (implementing Uniswap v1) is a <em>start</em> to having some basic defi on AssetHub,
many programmers may be surprised to find that synchronous EVM smart contract capabilities (e.g. uniswap v2+v3) on other chains are not possible on AssetHub.</p>
<p>Indeed, this is true for <em>many</em> Polkadot parachains, with the exception of the top 2 Polkadot parachains (by marketcap circa early 2024: Moonbeam + Astar) who <em>do</em> include the EVM pallets.<br />
This leads to a cumbersome Polkadot EVM smart contract programming experience between AssetHub and these 2 Polkadot parachains, making the Polkadot ecosystem hard to work with for asset-related applications from defi to NFTs.</p>
<p>The ink! defi ecosystem remains nascent, having only Astar as a potential home, and empirically has almost no defi/NFT activity. Although e.g. uniswap translations have been written,</p>
<p>An AssetHub for Polkadot deployment of EVM and ink! contracts, it is hoped, would likely support new applications for top assets (USDC and USDT) and spur many smart contract developers to develop end user applications with familiar synchronous programming constructs.</p>
<h3 id="rollups-need-smart-contracts-on-assethub"><a class="header" href="#rollups-need-smart-contracts-on-assethub">Rollups need Smart Contracts on AssetHub</a></h3>
<p>Polkadot Data Availability technology is extremely promising but underutilized.<br />
We envision a new class of customer, &quot;Polkadot Rollups&quot; that can utilize Polkadot DA far better than Ethereum and other technology platforms.<br />
Unlike Ethereum's DA which is capped at a fixed throughput now extending to EIP-4844, Polkadot data availability is <a href="https://forum.polkadot.network/t/polkadot-da-vs-competition/3403">linear in the number of cores</a>.<br />
This means Polkadot can support a much larger number of rollups than Ethereum <em>now</em>, and even more as the number of cores in Polkadot grows.
This performance difference has not been widely appreciated in the blockchain community.</p>
<p>Recently, a &quot;Blobs&quot; chain has been developed to expose Polkadot DA to rollups by senior Polkadot Fellows:</p>
<ul>
<li><a href="https://github.com/thrumdev/blobs">ThrumDev</a></li>
<li><a href="https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fblobs.colorfulnotion.com#/chainstate">Blobs on Kusama - ParaID 3338</a> )</li>
<li><a href="https://github.com/thrumdev/blobs/blob/e4c88182442c33f5d81dcfc9cc460dd81eab34a2/docs-site/docs/intro.md#integrations">Rollkit</a></li>
</ul>
<p>A rollup kit is mappable to widely used rollup platforms, such as OP Stack, Arbitrum Orbit or StarkNet Madara.<br />
A Blobs chain, currently deployed on Kusama (paraID 3338), enables rollups to utilize functionality <em>outside</em> the Polkadot 1.0 parachain architecture by having rollups submit transactions via a rollup kit abstraction. The Blobs chain write interface is simple <code>blobs.submitBlob(namespaceId, blob)</code> with a matching read interface.</p>
<p>However, simply sending blobs is not enough to power a rollup. End users need to interact with a &quot;settlement layer&quot;, while rollups require proof systems for security.</p>
<p>Key functionality for <em>optimistic</em> rollups (e.g. OP Stack, Arbitrum Orbit) are:</p>
<ul>
<li>enabling the users of the rollup to <em>deposit</em> and <em>withdraw</em> the L1 native token (DOT) into the rollup and from the rollup. In an AssetHub context:
<ul>
<li>Deposits: send DOT to the rollup from AssetHub, by calling an EVM Contract function on AssetHub;</li>
<li>Withdrawal: withdraw DOT from the rollup by submitting a EVM transaction on the rollup. After some of days (e.g. 7 days on OP Stack), the user submits a transaction on AssetHub to claim their DOT, using a Merkle proof.</li>
</ul>
</li>
<li>enabling interactive fraud proofs. While this has rarely happened in practice, it is critical to rollup security. In an AssetHub context:
<ul>
<li>Anyone monitoring a rollup, using the Blobs chain can access the recent history.</li>
<li>When detecting invalid state transitions, anyone can interact with rollup and AssetHub's EVM to generate a fraud proof.</li>
</ul>
</li>
</ul>
<p>Analogous functionality exist for ZK-rollup platforms (e.g. Polygon zkEVM, StarkNet Madara), with high potential for using the same Blobs+AssetHub chains.</p>
<p>While it is possible to have the operations in EVM Contracts translated in FRAME pallets (e.g. an &quot;opstack&quot; pallet), we do not believe a pallet translation confers significant benefits.<br />
Instead, we believe the translation would require regular updates from the rollup platform, which have proven difficult to implement in practice.</p>
<h3 id="ink-on-assethub-will-lead-to-coreplay-developers--on-assethub"><a class="header" href="#ink-on-assethub-will-lead-to-coreplay-developers--on-assethub">ink! on AssetHub will lead to CorePlay Developers on AssetHub</a></h3>
<p>While ink! WASM Smart Contracts have been promising technology, the adoption of ink! WASM Contracts amongst Polkadot parachains has been low, in practice just Astar to date, with nowhere near as many developers.<br />
This may be due to missing tooling, slow compile times, and/or simply because ink!/Rust is just harder to learn than Solidity, the dominant programming language of EVM Chains.</p>
<p>Fortunately, ink! can compile to <a href="https://forum.polkadot.network/t/announcing-polkavm-a-new-risc-v-based-vm-for-smart-contracts-and-possibly-more/3811">PolkaVM</a>, a new RISC based VM that has the special capability of suspending and resuming the registers, supporting long-running computations.<br />
This has the key new promise of making smart contract languages easier to use -- instead of smart contract developers worrying about what can be done within the gas limits of a specific block or a specific transaction, Coreplay smart contracts can be much easier to program on (see <a href="https://github.com/bkchr/coreplay-poc">here</a>).</p>
<p>We believe AssetHub should support ink! as a precursor to support CorePlay's capabilities as soon as possible.<br />
To the best of our knowledge, release times of this are <a href="https://forum.polkadot.network/t/announcing-polkavm-a-new-risc-v-based-vm-for-smart-contracts-and-possibly-more/3811/68">unknown</a> but having ink! inside AssetHub would be natural for Polkadot 2.0.</p>
<h2 id="stakeholders-7"><a class="header" href="#stakeholders-7">Stakeholders</a></h2>
<ul>
<li><strong>Asset Hub Users</strong>: Those who call any extrinsic on Asset Hub for Polkadot.</li>
<li><strong>DOT Token Holders</strong>: Those who hold DOT on any chain in the Polkadot ecosystem.</li>
<li><strong>AssetHub Smart Contract Developers</strong>: Those who utilize EVM Smart Contracts, ink! Contracts or Coreplay Contracts on AssetHub.</li>
<li><strong>Ethereum Rollups</strong>: Rollups who utilize Ethereum as a settlement layer and interactive fraud proofs or ZK proofs to secure their rollup and utilize Ethereum DA to record transactions, provide security for their rollup, and have rollup users settle on Ethereum.</li>
<li><strong>Polkadot Rollups</strong>: Rollups who utilize AssetHub as a settlement layer and interactive fraud proofs or ZK proofs on Assethub and Blobs to record rollup transactions, provide security for their rollup, and have rollup users settle on AssetHub for Polkadot.</li>
</ul>
<h2 id="explanation-7"><a class="header" href="#explanation-7">Explanation</a></h2>
<h3 id="limit-smart-contract-weight-allocation"><a class="header" href="#limit-smart-contract-weight-allocation">Limit Smart Contract Weight allocation</a></h3>
<p>AssetHub is a major component of the Polkadot 2.0 Minimal Relay Chain architecture. It is critical that smart contract developers not be able to clog AssetHub's blockspace for other mission critical applications, such as Staking and Governance.</p>
<p>As such, it is proposed that <em>at most</em> 50% of the available weight in AssetHub for Polkadot blocks be allocated to smart contracts pallets (EVM, ink! and/or Coreplay). While to date AssetHub has limited usage, it is believed (see <a href="https://forum.polkadot.network/t/permissioned-pallet-contracts-deployment-to-asset-hub-for-defi-primitives/3908/3">here</a>) that imposing this limit on smart contracts pallet would limit the effect on non-smart contract usage. A excessively small weight limit like 10% or 20% may limit the attractiveness of Polkadot as a platform for Polkadot rollups and EVM Contracts. A excessively large weight like 90% or 100% may threaten AssetHub usage.</p>
<p>In practice, this 50% weight limit would be used for 3 categories of smart contract usage:</p>
<ul>
<li>Defi/NFT applications: modern defi EVM Contracts would be expected to go beyond the capabilities of <code>assetConversion</code> pallet to support many common ERC20/ERC721/ERC1155 centric applications.</li>
<li>Polkadot Rollups users: deposit and withdrawal operations would be expected to dominate here. Note the operations of recording blocks would be done on the Blobs Chains, while interactive fraud proofs would be extremely rare by comparison.</li>
<li>Coreplay smart contract usage, at a future time.</li>
</ul>
<p>We expect the first category to dominate. If AssetHub smart contract usage increases so as to approach this 50% limit, the gas price will increase significantly. This likely motivates EVM contract developers to migrate to a EVM contract chain and/or rethink their application to work asynchronously within CoreJam, another major Polkadot 2.0 technology.</p>
<h3 id="model-assethub-assets-inside-evm-smart-contracts-based-on-astar"><a class="header" href="#model-assethub-assets-inside-evm-smart-contracts-based-on-astar">Model AssetHub Assets inside EVM Smart Contracts based on Astar</a></h3>
<p>It is essential to make AssetHub assets interface well with EVM Smart Contracts.
Polkadot parachains Astar and Moonbeam have a mapping between assetIDs and &quot;virtual&quot; EVM Contracts.</p>
<ul>
<li><a href="https://docs.astar.network/docs/learn/interoperability/asset-list">Astar XCM Asset List</a></li>
<li><a href="https://docs.moonbeam.network/builders/interoperability/xcm/xc20/overview/">Moonbeam XC-20</a></li>
</ul>
<p>We propose that AssetHub support a systemic mapping following Astar:
(a) Native Relay DOT + KSM - should be mapped to <code>0xFFfFfFffFFfffFFfFFfFFFFFffFFFffffFfFFFfF</code> on AssetHub for Polkadot and AssetHub for Kusama respectively
(b) Other Assethubs assets should map into an EVM address using a <code>0xffffffff</code> prefix
https://docs.astar.network/docs/learn/interoperability/xcm/integration/tools#xc20-address</p>
<p>The usage of the above has been made code-complete by Astar:</p>
<ul>
<li><a href="https://github.com/AstarNetwork/Astar/tree/master/pallets/xc-asset-config">xc-asset-config</a></li>
<li><a href="https://github.com/AstarNetwork/Astar/tree/master/precompiles/assets-erc20">assets-erc20</a></li>
<li><a href="https://github.com/AstarNetwork/Astar/tree/master/chain-extensions/pallet-assets">pallet-assets extensions</a></li>
</ul>
<p>Polkadot parachains Astar and Moonbeam adopted two very different approaches of how end users interact with EVM Contracts.<br />
We propose that AssetHub for Polkadot adopt the Astar solution, mirroring it as closely as possible.</p>
<h2 id="new-dot-revenue-sources"><a class="header" href="#new-dot-revenue-sources">New DOT Revenue Sources</a></h2>
<p>A substantial motivation in this proposal is to increase demand for DOT via two key chains:</p>
<ul>
<li>AssetHub - from defi/NFT users, Polkadot Rollup users and AssetHub Smart Contract Developers</li>
<li>Blobs - for Polkadot Rollups</li>
</ul>
<h3 id="new-revenue-from-assethub-evm-contracts"><a class="header" href="#new-revenue-from-assethub-evm-contracts">New Revenue from AssetHub EVM Contracts</a></h3>
<p>Enabling EVM Contracts on AssetHub will support DOT revenue from:</p>
<ul>
<li>defi/NFT users who use AssetHub directly</li>
<li>rollup operators who utilize Blobs chain</li>
<li>rollup users who buy DOT to utilize Polkadot Rollups</li>
</ul>
<h3 id="new-revenue-for-inkcoreplay-contracts"><a class="header" href="#new-revenue-for-inkcoreplay-contracts">New Revenue for ink!/Coreplay Contracts</a></h3>
<p>Enabling ink! contracts will pave the way to a new class of AssetHub Smart Contract Developers.<br />
Given PolkaVM's proven reduced compile time and RISC architecture enabling register snapshots, it is natural to utilize these new technical capabilities on a flagship system chain.<br />
To the extent these capabilities are attractive to smart contract developers, this has the potential for bringing in new DOT revenue from a system chain.</p>
<h2 id="drawbacks-and-tradeoffs"><a class="header" href="#drawbacks-and-tradeoffs">Drawbacks and Tradeoffs</a></h2>
<p>Supporting EVM Contracts in AssetHub is seen by some as undercutting Polkadot's 1.0 parachain architecture, both special purpose appchains and smart contract developer platform parachains.<br />
We believe the lack of growth of parachains in the last 12-18 months, and the high potential of CorePlay motivates new options be pursued in system chains.</p>
<p>Maintaining EVM Contracts on AssetHub may be seen as difficult and may require Substrate engineers to maintain EVM Pallets and manage the xcContracts.<br />
We believe this cost will be relatively small based on the proven deployment of Astar and Moonbeam.<br />
The cost will be justified compared to the potential upside of new DOT revenue from defi/NFT applications on AssetHub and the potential for utilizing Polkadot DA for Polkadot rollups.</p>
<h2 id="testing-security-and-privacy-7"><a class="header" href="#testing-security-and-privacy-7">Testing, Security, and Privacy</a></h2>
<p>Testing the mapping between assetIDs and EVM Contracts thoroughly will be critical.</p>
<p>Having a complete working OP Stack chain using AssetHub for Kusama (1000) and Blobs on Kusama (3338) would be highly desirable, but is unlikely to be required.</p>
<h2 id="performance-ergonomics-and-compatibility-7"><a class="header" href="#performance-ergonomics-and-compatibility-7">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-8"><a class="header" href="#performance-8">Performance</a></h3>
<p>The weight limit of 50% is expected to be adequate to limit excess smart contract usage at this time.</p>
<p>Storage bloat is expected to kept to a minimum with the nominal 0.01 Existential Deposit.</p>
<h3 id="ergonomics-6"><a class="header" href="#ergonomics-6">Ergonomics</a></h3>
<p>Note that the existential deposit is not 0 DOT but being lowered from 0.1 DOT to 0.01 DOT, which may pose problems for some developers.<br />
Many developers routinely deploy their EVM contracts on many different EVM Chains in parallel. This non-zero ED may pose problems for some developers</p>
<p>The 0.01 DOT (worth $0.075 USD) is unlikely to pose significant issue.</p>
<h3 id="compatibility-5"><a class="header" href="#compatibility-5">Compatibility</a></h3>
<p>It is believed that EVM pallet (as deployed on Moonbeam + Astar) is sufficiently compatible with Ethereum, and that the ED of 0.01 DOT pose negligible issues.</p>
<p>The messaging architecture for rollups are not compatible with Polkadot XCM.<br />
It is not clear if leading rollup platforms (OP Stack, Arbitrum Orbit, Polygon zkEVM) could be made compatible with XCM.</p>
<h2 id="unresolved-questions-7"><a class="header" href="#unresolved-questions-7">Unresolved Questions</a></h2>
<p>It is highly desirable to know the throughput of Polkadot DA with popular rollup architectures OP Stack and Arbitrum Orbit.<br />
This would enable CEXs and EVM L2 builders to choose Polkadot over Ethereum.</p>
<h2 id="future-directions-and-related-material-6"><a class="header" href="#future-directions-and-related-material-6">Future Directions and Related Material</a></h2>
<p>If accepted, this RFC could pave the way for CorePlay on Asset Hub for Polkadot/Kusama, a major component of Polkadot 2.0's smart contract future.</p>
<p>The importance of precompiles should</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/70">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="proposed/0070-x-track-kusamanetwork.html#rfc-0070-x-track-for-kusamanetwork">RFC-0070: X Track for @kusamanetwork</a>
<ul>
<li><a href="proposed/0070-x-track-kusamanetwork.html#summary">Summary</a></li>
<li><a href="proposed/0070-x-track-kusamanetwork.html#motivation">Motivation</a></li>
<li><a href="proposed/0070-x-track-kusamanetwork.html#stakeholders">Stakeholders</a></li>
<li><a href="proposed/0070-x-track-kusamanetwork.html#explanation">Explanation</a>
<ul>
<li><a href="proposed/0070-x-track-kusamanetwork.html#phase-1---track-configurations">Phase 1 - Track configurations</a></li>
<li><a href="proposed/0070-x-track-kusamanetwork.html#phase-2---establish-specs-for-x-post-track-referenda">Phase 2 - Establish Specs for X Post Track Referenda</a></li>
<li><a href="proposed/0070-x-track-kusamanetwork.html#phase-3---release-tooling--documentation">Phase 3 - Release, Tooling, &amp; Documentation</a></li>
</ul>
</li>
<li><a href="proposed/0070-x-track-kusamanetwork.html#drawbacks">Drawbacks</a></li>
<li><a href="proposed/0070-x-track-kusamanetwork.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="proposed/0070-x-track-kusamanetwork.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="proposed/0070-x-track-kusamanetwork.html#performance">Performance</a></li>
<li><a href="proposed/0070-x-track-kusamanetwork.html#ergonomics">Ergonomics</a></li>
<li><a href="proposed/0070-x-track-kusamanetwork.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="proposed/0070-x-track-kusamanetwork.html#references">References</a></li>
<li><a href="proposed/0070-x-track-kusamanetwork.html#unresolved-questions">Unresolved Questions</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0070-x-track-for-kusamanetwork"><a class="header" href="#rfc-0070-x-track-for-kusamanetwork">RFC-0070: X Track for <a href="https://x.com/kusamanetwork">@kusamanetwork</a></a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>January 29, 2024</td></tr>
<tr><td><strong>Description</strong></td><td>Add a governance track to facilitate posts on the @kusamanetwork's X account</td></tr>
<tr><td><strong>Author</strong></td><td>Adam Clay Steeber</td></tr>
</tbody></table>
</div>
<h2 id="summary-8"><a class="header" href="#summary-8">Summary</a></h2>
<p>This RFC proposes adding a trivial governance track on Kusama to facilitate X (formerly known as Twitter) posts on the <a href="https://x.com/kusamanetwork">@kusamanetwork</a> account. The technical aspect
of implementing this in the runtime is very inconsequential and straight-forward, though it might get more technical if the Fellowship wants to regulate this track
with a non-existent permission set. If this is implemented it would need to be followed up with:</p>
<ol>
<li>the establishment of specifications for proposing X posts via this track, and</li>
<li>the development of tools/processes to ensure that the content contained in referenda enacted in this track would be automatically posted on X.</li>
</ol>
<h2 id="motivation-8"><a class="header" href="#motivation-8">Motivation</a></h2>
<p>The overall motivation for this RFC is to decentralize the management of the Kusama brand/communication channel to KSM holders. This is necessary in my opinion primarily
because of the inactivity of the account in recent history, with posts spanning weeks or months apart. I am currently unaware of who/what entity manages the Kusama
X account, but if they are affiliated with Parity or W3F this proposed solution could also offload some of the legal ramifications of making (or not making)
announcements to the public regarding Kusama. While centralized control of the X account would still be present, it could become totally moot if this RFC is implemented
and the community becomes totally autonomous in the management of Kusama's X posts.</p>
<p>This solution does not cover every single communication front for Kusama, but it does cover one of the largest. It also establishes a precedent for other communication channels
that could be offloaded to openGov, provided this proof-of-concept is successful.</p>
<p>Finally, this RFC is the epitome of experimentation that Kusama is ideal for. This proposal may spark newfound excitement for Kusama and help us realize Kusama's potential
for pushing boundaries and trying new unconventional ideas.</p>
<h2 id="stakeholders-8"><a class="header" href="#stakeholders-8">Stakeholders</a></h2>
<p>This idea has not been formalized by any individual (or group of) KSM holder(s). To my knowledge the socialization of this idea is contained
entirely in <a href="https://twitter.com/AdamSteeber1/status/1750541362498302230">my recent X post here</a>, but it is possible that an idea like this one has been discussed in
other places. It appears to me that the ecosystem would welcome a change like this which is why I am taking action to formalize the discussion.</p>
<h2 id="explanation-8"><a class="header" href="#explanation-8">Explanation</a></h2>
<p>The implementation of this idea can be broken down into 3 primary phases:</p>
<h3 id="phase-1---track-configurations"><a class="header" href="#phase-1---track-configurations">Phase 1 - Track configurations</a></h3>
<p>First, we begin with this RFC to ensure all feedback can be discussed and implemented in the proposal. After the Fellowship and the community come to a reasonable
agreement on the changes necessary to make this happen, the Fellowship can merge changes into Kusama's runtime to include this new track with appropriate track configurations.
As a starting point, I recommend the following track configurations:</p>
<pre><code>const APP_X_POST: Curve = Curve::make_linear(7, 28, percent(50), percent(100));
const SUP_X_POST: Curve = Curve::make_reciprocal(?, ?, percent(?), percent(?), percent(?));
// I don't know how to configure the make_reciprocal variables to get what I imagine for support,
// but I recommend starting at 50% support and sharply decreasing such that 1% is sufficient quarterway
// through the decision period and hitting 0% at the end of the decision period, or something like that.
(
69,
pallet_referenda::TrackInfo {
name: &quot;x_post&quot;,
max_deciding: 50,
decision_deposit: 1 * UNIT,
prepare_period: 10 * MINUTES,
decision_period: 4 * DAYS,
confirm_period: 10 * MINUTES,
min_enactment_period: 1 * MINUTES,
min_approval: APP_X_POST,
min_support: SUP_X_POST,
},
),
</code></pre>
<p>I also recommend restricting permissions of this track to only submitting remarks or batches of remarks - that's all we'll need for its purpose. I'm not sure how
easy that is to configure, but it is important since we don't want such an agile track to be able to make highly consequential calls.</p>
<h3 id="phase-2---establish-specs-for-x-post-track-referenda"><a class="header" href="#phase-2---establish-specs-for-x-post-track-referenda">Phase 2 - Establish Specs for X Post Track Referenda</a></h3>
<p>It is important that we establish the specifications of referenda that will be submitted in this track to ensure that whatever automation tool is built can easily
make posts once a referendum is enacted. As stated above, we really only need a system.remark (or batch of remarks) to indicate the contents of a proposed X post.
The most straight-forward way to do this is to require remarks to adhere to X's requirements for making <a href="https://developer.twitter.com/en/docs/twitter-api/tweets/manage-tweets/api-reference/post-tweets">posts via their API</a>.</p>
<p>For example, if I wanted to propose a post that contained the text &quot;Hello World!&quot; I would propose a referendum in the X post track that contains the following call data:
<code>0x0000607b2274657874223a202248656c6c6f20576f726c6421227d</code> (i.e. <code>system.remark('{&quot;text&quot;: &quot;Hello World!&quot;}')</code>).</p>
<p>At first, we could support text posts only to prove the concept. Later on we could expand this spec to add support for media, likes, retweets, replies, polls, and
whatever other X features we want.</p>
<h3 id="phase-3---release-tooling--documentation"><a class="header" href="#phase-3---release-tooling--documentation">Phase 3 - Release, Tooling, &amp; Documentation</a></h3>
<p>Once we agree on track configurations and specs for referenda in this track, the Fellowship can move forward with merging these changes into Kusama's runtime and
include them in its next release. We could also move forward with developing the necessary tools that would listen for enacted referenda to post automatically on X.
This would require coordination with whoever controls the X account; they would either need to run the tools themselves or add a third party as an authorized user to
run the tools to make posts on the account's behalf. This is a bottleneck for decentralization, but as long as the tools are run by the X account manager or by a trusted third party
it should be fine. I'm open to more decentralized solutions, but those always come at a cost of complexity.</p>
<p>For the tools themselves, we could open a bounty on Kusama for developers/teams to bid on. We could also just ask the community to step up with a Treasury proposal
to have anyone fund the build. Or, the Fellowship could make the release of these changes contingent on their endorsement of developers/teams to build these tools. Lots of options!
For the record, me and my team could develop all the necessary tools, but all because I'm proposing these changes doesn't entitle me to funds to build the tools needed
to implement them. Here's what would be needed:</p>
<ul>
<li>a listener tool that would listen for enacted referenda in this track, verify the format of the remark(s), and submit to X's API with authenticating credentials</li>
<li>a UI to allow layman users to propose referenda on this track</li>
</ul>
<p>After everything is complete, we can update the Kusama wiki to include documentation on the X post specifications and include links to the tools/UI.</p>
<h2 id="drawbacks-7"><a class="header" href="#drawbacks-7">Drawbacks</a></h2>
<p>The main drawback to this change is that it requires a lot of off-chain coordination. It's easy enough to include the track on Kusama but it's a totally different
challenge to make it function as intended. The tools need to be built and the auth tokens need to be managed. It would certainly add an administrative burden to whoever
manages the X account since they would either need to run the tools themselves or manage auth tokens.</p>
<p>This change also introduces on-going costs to the Treasury since it would need to compensate people to support the tools necessary to facilitate this idea. The ultimate
question is whether these on-going costs would be worth the ability for KSM holders to make posts on Kusama's X account.</p>
<p>There's also the risk of misconfiguring the track to make referenda too easy to pass, potentially allowing a malicious actor to get content posted on X that violates X's ToS.
If that happens, we risk getting Kusama banned on X!</p>
<p>This change might also be outside the scope of the Fellowship/openGov. Perhaps the best solution for the X account is to have the Treasury pay for a professional
agency to manage posts. It wouldn't be decentralized but it would probably be more effective in terms of creating good content.</p>
<p>Finally, this solution is merely pseudo-decentralization since the X account manager would still have ultimate control of the account. It's decentralized insofar as
the auth tokens are given to people actually running the tools; a house of cards is required to facilitate X posts via this track. Not ideal.</p>
<h2 id="testing-security-and-privacy-8"><a class="header" href="#testing-security-and-privacy-8">Testing, Security, and Privacy</a></h2>
<p>There's major precedent for configuring tracks on openGov given the amount of power tracks have, so it shouldn't be hard to come up with a sound configuration.
That's why I recommend restricting permissions of this track to remarks and batches of remarks, or something equally inconsequential.</p>
<p>Building the tools for this implementation is really straight-forward and could be audited by Fellowship members, and the community at large, on Github.</p>
<p>The largest security concern would be the management of Kusama's X account's auth tokens. We would need to ensure that they aren't compromised.</p>
<h2 id="performance-ergonomics-and-compatibility-8"><a class="header" href="#performance-ergonomics-and-compatibility-8">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-9"><a class="header" href="#performance-9">Performance</a></h3>
<p>If a track on Kusama promises users that compliant referenda enacted therein would be posted on Kusama's X account, users would expect that track to perform as promised.
If the house of cards tumbles down and a compliant referendum doesn't actually get anything posted, users might think that Kusama is broken or unreliable. This
could be damaging to Kusama's image and cause people to question the soundness of other features on Kusama.</p>
<p>As mentioned in the drawbacks, the performance of this feature would depend on off-chain coordinations. We can reduce the administrative burden of these coordinations
by funding third parties with the Treasury to deal with it, but then we're relying on trusting these parties.</p>
<h3 id="ergonomics-7"><a class="header" href="#ergonomics-7">Ergonomics</a></h3>
<p>By adding a new track to Kusama, governance platforms like Polkassembly or Nova Wallet would need to include it on their applications. This shouldn't be too
much of a burden or overhead since they've already built the infrastructure for other openGov tracks.</p>
<h3 id="compatibility-6"><a class="header" href="#compatibility-6">Compatibility</a></h3>
<p>This change wouldn't break any compatibility as far as I know.</p>
<h2 id="references-1"><a class="header" href="#references-1">References</a></h2>
<p>One reference to a similar feature requiring on-chain/off-chain coordination would be the Kappa-Sigma-Mu Society. Nothing on-chain necessarily enforces the rules
or facilitates bids, challenges, defenses, etc. However, the Society has managed to maintain itself with integrity to its rules. So I don't think this is totally
out of Kusama's scope. But it will require some off-chain effort to maintain.</p>
<h2 id="unresolved-questions-8"><a class="header" href="#unresolved-questions-8">Unresolved Questions</a></h2>
<ul>
<li>Who will develop the tools necessary to implement this feature? How do we select them?</li>
<li>How can this idea be better implemented with on-chain/substrate features?</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/73">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="proposed/0073-referedum-deposit-track.html#rfc-0073-decision-deposit-referendum-track">RFC-0073: Decision Deposit Referendum Track</a>
<ul>
<li><a href="proposed/0073-referedum-deposit-track.html#summary">Summary</a></li>
<li><a href="proposed/0073-referedum-deposit-track.html#motivation">Motivation</a></li>
<li><a href="proposed/0073-referedum-deposit-track.html#explanation">Explanation</a>
<ul>
<li><a href="proposed/0073-referedum-deposit-track.html#referendum-track-parameters---polkadot">Referendum track parameters - Polkadot</a></li>
<li><a href="proposed/0073-referedum-deposit-track.html#referendum-track-parameters---kusama">Referendum track parameters - Kusama</a></li>
</ul>
</li>
<li><a href="proposed/0073-referedum-deposit-track.html#drawbacks">Drawbacks</a></li>
<li><a href="proposed/0073-referedum-deposit-track.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="proposed/0073-referedum-deposit-track.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="proposed/0073-referedum-deposit-track.html#performance">Performance</a></li>
<li><a href="proposed/0073-referedum-deposit-track.html#ergonomics">Ergonomics</a></li>
<li><a href="proposed/0073-referedum-deposit-track.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="proposed/0073-referedum-deposit-track.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="proposed/0073-referedum-deposit-track.html#unresolved-questions">Unresolved Questions</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0073-decision-deposit-referendum-track"><a class="header" href="#rfc-0073-decision-deposit-referendum-track">RFC-0073: Decision Deposit Referendum Track</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>12 February 2024</td></tr>
<tr><td><strong>Description</strong></td><td>Add a referendum track which can place the decision deposit on any other track</td></tr>
<tr><td><strong>Authors</strong></td><td>JelliedOwl</td></tr>
</tbody></table>
</div>
<h2 id="summary-9"><a class="header" href="#summary-9">Summary</a></h2>
<p>The current size of the decision deposit on some tracks is too high for many proposers. As a result, those needing to use it have to find someone else willing to put up the deposit for them - and a number of legitimate attempts to use the root track have timed out. This track would provide a more affordable (though slower) route for these holders to use the root track.</p>
<h2 id="motivation-9"><a class="header" href="#motivation-9">Motivation</a></h2>
<p>There have been recent attempts to use the Kusama root track which have timed out with no decision deposit placed. Usually, these referenda have been related to parachain registration related issues. </p>
<h2 id="explanation-9"><a class="header" href="#explanation-9">Explanation</a></h2>
<p>Propose to address this by adding a new referendum track <em><strong>[22] Referendum Deposit</strong></em> which can place the decision deposit on another referendum. This would require the following changes:</p>
<ul>
<li>[Referenda Pallet] Modify the <code>placeDecisionDesposit</code> function to additionally allow it to be called by root, with root call bypassing the requirements for a deposit payment.</li>
<li>[Runtime] Add a new referendum track which can only call <code>referenda-&gt;placeDecisionDeposit</code> and the utility functions.</li>
</ul>
<h3 id="referendum-track-parameters---polkadot"><a class="header" href="#referendum-track-parameters---polkadot">Referendum track parameters - Polkadot</a></h3>
<ul>
<li><strong>Decision deposit</strong>: 1000 DOT</li>
<li><strong>Decision period</strong>: 14 days</li>
<li><strong>Confirmation period</strong>: 12 hours</li>
<li><strong>Enactment period</strong>: 2 hour</li>
<li><strong>Approval &amp; Support curves</strong>: As per the root track, timed to match the decision period</li>
<li><strong>Maximum deciding</strong>: 10</li>
</ul>
<h3 id="referendum-track-parameters---kusama"><a class="header" href="#referendum-track-parameters---kusama">Referendum track parameters - Kusama</a></h3>
<ul>
<li><strong>Decision deposit</strong>: 33.333333 KSM</li>
<li><strong>Decision period</strong>: 7 days</li>
<li><strong>Confirmation period</strong>: 6 hours</li>
<li><strong>Enactment period</strong>: 1 hour</li>
<li><strong>Approval &amp; Support curves</strong>: As per the root track, timed to match the decision period</li>
<li><strong>Maximum deciding</strong>: 10</li>
</ul>
<h2 id="drawbacks-8"><a class="header" href="#drawbacks-8">Drawbacks</a></h2>
<p>This track would provide a route to starting a root referendum with a much-reduced slashable deposit. This might be undesirable but, assuming the decision deposit cost for this track is still high enough, slashing would still act as a disincentive.</p>
<p>An alternative to this might be to reduce the decision deposit size some of the more expensive tracks. However, part of the purpose of the high deposit - at least on the root track - is to prevent spamming the limited queue with junk referenda.</p>
<h2 id="testing-security-and-privacy-9"><a class="header" href="#testing-security-and-privacy-9">Testing, Security, and Privacy</a></h2>
<p>Will need additional tests case for the modified pallet and runtime. No security or privacy issues.</p>
<h2 id="performance-ergonomics-and-compatibility-9"><a class="header" href="#performance-ergonomics-and-compatibility-9">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-10"><a class="header" href="#performance-10">Performance</a></h3>
<p>No significant performance impact.</p>
<h3 id="ergonomics-8"><a class="header" href="#ergonomics-8">Ergonomics</a></h3>
<p>Only changes related to adding the track. Existing functionality is unchanged.</p>
<h3 id="compatibility-7"><a class="header" href="#compatibility-7">Compatibility</a></h3>
<p>No compatibility issues.</p>
<h2 id="prior-art-and-references-6"><a class="header" href="#prior-art-and-references-6">Prior Art and References</a></h2>
<ul>
<li>Recent discussion / referendum for an alternative way to address this issue: <a href="https://kusama.polkassembly.io/referenda/340">Kusama Referendum 340 - Funding a Decision Deposit Sponsor</a></li>
</ul>
<h2 id="unresolved-questions-9"><a class="header" href="#unresolved-questions-9">Unresolved Questions</a></h2>
<p>Feedback on whether my proposed implementation of this is the best way to address the issue - including which calls the track should be allowed to make. Are the track parameters correct or should be use something different? Alternative would be welcome.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/74">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="proposed/0074-stateful-multisig-pallet.html#rfc-0074-stateful-multisig-pallet">RFC-0074: Stateful Multisig Pallet</a>
<ul>
<li><a href="proposed/0074-stateful-multisig-pallet.html#summary">Summary</a></li>
<li><a href="proposed/0074-stateful-multisig-pallet.html#motivation">Motivation</a>
<ul>
<li><a href="proposed/0074-stateful-multisig-pallet.html#problem">Problem</a></li>
<li><a href="proposed/0074-stateful-multisig-pallet.html#requirements">Requirements</a></li>
<li><a href="proposed/0074-stateful-multisig-pallet.html#use-cases">Use Cases</a></li>
</ul>
</li>
<li><a href="proposed/0074-stateful-multisig-pallet.html#stakeholders">Stakeholders</a></li>
<li><a href="proposed/0074-stateful-multisig-pallet.html#explanation">Explanation</a>
<ul>
<li><a href="proposed/0074-stateful-multisig-pallet.html#state-transition-functions">State Transition Functions</a></li>
<li><a href="proposed/0074-stateful-multisig-pallet.html#storagestate">Storage/State</a></li>
<li><a href="proposed/0074-stateful-multisig-pallet.html#considerations--edge-cases">Considerations &amp; Edge cases</a></li>
</ul>
</li>
<li><a href="proposed/0074-stateful-multisig-pallet.html#drawbacks">Drawbacks</a></li>
<li><a href="proposed/0074-stateful-multisig-pallet.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="proposed/0074-stateful-multisig-pallet.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="proposed/0074-stateful-multisig-pallet.html#performance">Performance</a></li>
<li><a href="proposed/0074-stateful-multisig-pallet.html#ergonomics">Ergonomics</a></li>
<li><a href="proposed/0074-stateful-multisig-pallet.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="proposed/0074-stateful-multisig-pallet.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="proposed/0074-stateful-multisig-pallet.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="proposed/0074-stateful-multisig-pallet.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0074-stateful-multisig-pallet"><a class="header" href="#rfc-0074-stateful-multisig-pallet">RFC-0074: Stateful Multisig Pallet</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>15 February 2024</td></tr>
<tr><td><strong>Description</strong></td><td>Add Enhanced Multisig Pallet to System chains</td></tr>
<tr><td><strong>Authors</strong></td><td>Abdelrahman Soliman (Boda)</td></tr>
</tbody></table>
</div>
<h2 id="summary-10"><a class="header" href="#summary-10">Summary</a></h2>
<p>A pallet to facilitate enhanced multisig accounts. The main enhancement is that we store a multisig account in the state with related info (signers, threshold,..etc). The module affords enhanced control over administrative operations such as adding/removing signers, changing the threshold, account deletion, canceling an existing proposal. Each signer can approve/reject a proposal while still exists. The proposal is <strong>not</strong> intended for migrating or getting rid of existing multisig. It's to allow both options to coexist.</p>
<p>For the rest of the RFC We use the following terms:</p>
<ul>
<li><code>proposal</code> to refer to an extrinsic that is to be dispatched from a multisig account after getting enough approvals.</li>
<li><code>Stateful Multisig</code> to refer to the proposed pallet.</li>
<li><code>Stateless Multisig</code> to refer to the current multisig pallet in polkadot-sdk.</li>
</ul>
<h2 id="motivation-10"><a class="header" href="#motivation-10">Motivation</a></h2>
<h3 id="problem-2"><a class="header" href="#problem-2">Problem</a></h3>
<p>Entities in the Polkadot ecosystem need to have a way to manage their funds and other operations in a secure and efficient way. Multisig accounts are a common way to achieve this. Entities by definition change over time, members of the entity may change, threshold requirements may change, and the multisig account may need to be deleted. For even more enhanced hierarchical control, the multisig account may need to be controlled by other multisig accounts.</p>
<p>Current native solutions for multisig operations are less optimal, performance-wise (as we'll explain later in the RFC), and lack fine-grained control over the multisig account.</p>
<h4 id="stateless-multisig"><a class="header" href="#stateless-multisig">Stateless Multisig</a></h4>
<p>We refer to current <a href="https://github.com/paritytech/polkadot-sdk/tree/master/substrate/frame/multisig">multisig pallet in polkadot-sdk</a> because the multisig account is only derived and not stored in the state. Although deriving the account is determinsitc as it relies on exact users (sorted) and thershold to derive it. This does not allow for control over the multisig account. It's also tightly coupled to exact users and threshold. This makes it hard for an organization to manage existing accounts and to change the threshold or add/remove signers.</p>
<p>We believe as well that the stateless multisig is not efficient in terms of block footprint as we'll show in the performance section.</p>
<h4 id="pure-proxy"><a class="header" href="#pure-proxy">Pure Proxy</a></h4>
<p>Pure proxy can achieve having a stored and determinstic multisig account from different users but it's unneeded complexity as a way around the limitations of the current multisig pallet. It doesn't also have the same fine grained control over the multisig account.</p>
<p>Other points mentioned by @tbaut</p>
<ul>
<li>pure proxies aren't (yet) a thing cross chain</li>
<li>the end user complexity is much much higher with pure proxies, also for new users smart contract multisig are widely known while pure proxies are obscure.</li>
<li>you can shoot yourself in the foot by deleting the proxy, and effectively loosing access to funds with pure proxies.</li>
</ul>
<h3 id="requirements-2"><a class="header" href="#requirements-2">Requirements</a></h3>
<p>Basic requirements for the Stateful Multisig are:</p>
<ul>
<li>The ability to have concrete and permanent (unless deleted) multisig accounts in the state.</li>
<li>The ability to add/remove signers from an existing multisig account by the multisig itself.</li>
<li>The ability to change the threshold of an existing multisig account by the multisig itself.</li>
<li>The ability to delete an existing multisig account by the multisig itself.</li>
<li>The ability to cancel an existing proposal by the multisig itself.</li>
<li>Signers of multisig account can start a proposal on behalf of the multisig account which will be dispatched after getting enough approvals.</li>
<li>Signers of multisig account can approve/reject a proposal while still exists.</li>
</ul>
<h3 id="use-cases"><a class="header" href="#use-cases">Use Cases</a></h3>
<ul>
<li>
<p>Corporate Governance:
In a corporate setting, multisig accounts can be employed for decision-making processes. For example, a company may require the approval of multiple executives to initiate significant financial transactions.</p>
</li>
<li>
<p>Joint Accounts:
Multisig accounts can be used for joint accounts where multiple individuals need to authorize transactions. This is particularly useful in family finances or shared business accounts.</p>
</li>
<li>
<p>Decentralized Autonomous Organizations (DAOs):
DAOs can utilize multisig accounts to ensure that decisions are made collectively. Multiple key holders can be required to approve changes to the organization's rules or the allocation of funds.</p>
</li>
</ul>
<p>and much more...</p>
<h2 id="stakeholders-9"><a class="header" href="#stakeholders-9">Stakeholders</a></h2>
<ul>
<li>Polkadot holders</li>
<li>Polkadot developers</li>
</ul>
<h2 id="explanation-10"><a class="header" href="#explanation-10">Explanation</a></h2>
<p>I've created the stateful multisig pallet during my studies in Polkadot Blockchain Academy under supervision from @shawntabrizi and @ank4n. After that, I've enhanced it to be fully functional and this is a draft <a href="https://github.com/paritytech/polkadot-sdk/pull/3300">PR#3300</a> in polkadot-sdk. I'll list all the details and design decisions in the following sections. Note that the PR is not 1-1 exactly to the current RFC as the RFC is a more polished version of the PR after updating based on the feedback and discussions.</p>
<p>Let's start with a sequence diagram to illustrate the main operations of the Stateful Multisig.</p>
<p><img src="https://github.com/asoliman92/RFCs/assets/2677789/4f2e8972-f3b8-4250-b75f-1e4788b35752" alt="multisig operations" /></p>
<p>Notes on above diagram:</p>
<ul>
<li>It's a 3 step process to execute a proposal. (Start Proposal --&gt; Approvals --&gt; Execute Proposal)</li>
<li><code>Execute</code> is an explicit extrinsic for a simpler API. It can be optimized to be executed automatically after getting enough approvals.</li>
<li>Any user can create a multisig account and they don't need to be part of it. (Alice in the diagram)</li>
<li>A proposal is any extrinsic including control extrinsics (e.g. add/remove signer, change threshold,..etc).</li>
<li>Any multisig account signer can start a proposal on behalf of the multisig account. (Bob in the diagram)</li>
<li>Any multisig account owener can execute proposal if it's approved by enough signers. (Dave in the diagram)</li>
</ul>
<h3 id="state-transition-functions"><a class="header" href="#state-transition-functions">State Transition Functions</a></h3>
<p>having the following enum to store the call or the hash:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>enum CallOrHash&lt;T: Config&gt; {
Call(&lt;T as Config&gt;::RuntimeCall),
Hash(T::Hash),
}
<span class="boring">}</span></code></pre></pre>
<ul>
<li><code>create_multisig</code> - Create a multisig account with a given threshold and initial signers. (Needs Deposit)</li>
</ul>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> /// Creates a new multisig account and attach signers with a threshold to it.
///
/// The dispatch origin for this call must be _Signed_. It is expected to be a nomral AccountId and not a
/// Multisig AccountId.
///
/// T::BaseCreationDeposit + T::PerSignerDeposit * signers.len() will be held from the caller's account.
///
/// # Arguments
///
/// - `signers`: Initial set of accounts to add to the multisig. These may be updated later via `add_signer`
/// and `remove_signer`.
/// - `threshold`: The threshold number of accounts required to approve an action. Must be greater than 0 and
/// less than or equal to the total number of signers.
///
/// # Errors
///
/// * `TooManySignatories` - The number of signatories exceeds the maximum allowed.
/// * `InvalidThreshold` - The threshold is greater than the total number of signers.
pub fn create_multisig(
origin: OriginFor&lt;T&gt;,
signers: BoundedBTreeSet&lt;T::AccountId, T::MaxSignatories&gt;,
threshold: u32,
) -&gt; DispatchResult
<span class="boring">}</span></code></pre></pre>
<ul>
<li><code>start_proposal</code> - Start a multisig proposal. (Needs Deposit)</li>
</ul>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> /// Starts a new proposal for a dispatchable call for a multisig account.
/// The caller must be one of the signers of the multisig account.
/// T::ProposalDeposit will be held from the caller's account.
///
/// # Arguments
///
/// * `multisig_account` - The multisig account ID.
/// * `call_or_hash` - The enum having the call or the hash of the call to be approved and executed later.
///
/// # Errors
///
/// * `MultisigNotFound` - The multisig account does not exist.
/// * `UnAuthorizedSigner` - The caller is not an signer of the multisig account.
/// * `TooManySignatories` - The number of signatories exceeds the maximum allowed. (shouldn't really happen as it's the first approval)
pub fn start_proposal(
origin: OriginFor&lt;T&gt;,
multisig_account: T::AccountId,
call_or_hash: CallOrHash,
) -&gt; DispatchResult
<span class="boring">}</span></code></pre></pre>
<ul>
<li><code>approve</code> - Approve a multisig proposal.</li>
</ul>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> /// Approves a proposal for a dispatchable call for a multisig account.
/// The caller must be one of the signers of the multisig account.
///
/// If a signer did approve -&gt; reject -&gt; approve, the proposal will be approved.
/// If a signer did approve -&gt; reject, the proposal will be rejected.
///
/// # Arguments
///
/// * `multisig_account` - The multisig account ID.
/// * `call_or_hash` - The enum having the call or the hash of the call to be approved.
///
/// # Errors
///
/// * `MultisigNotFound` - The multisig account does not exist.
/// * `UnAuthorizedSigner` - The caller is not an signer of the multisig account.
/// * `TooManySignatories` - The number of signatories exceeds the maximum allowed.
/// This shouldn't really happen as it's an approval, not an addition of a new signer.
pub fn approve(
origin: OriginFor&lt;T&gt;,
multisig_account: T::AccountId,
call_or_hash: CallOrHash,
) -&gt; DispatchResult
<span class="boring">}</span></code></pre></pre>
<ul>
<li><code>reject</code> - Reject a multisig proposal.</li>
</ul>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> /// Rejects a proposal for a multisig account.
/// The caller must be one of the signers of the multisig account.
///
/// Between approving and rejecting, last call wins.
/// If a signer did approve -&gt; reject -&gt; approve, the proposal will be approved.
/// If a signer did approve -&gt; reject, the proposal will be rejected.
///
/// # Arguments
///
/// * `multisig_account` - The multisig account ID.
/// * `call_or_hash` - The enum having the call or the hash of the call to be rejected.
///
/// # Errors
///
/// * `MultisigNotFound` - The multisig account does not exist.
/// * `UnAuthorizedSigner` - The caller is not an signer of the multisig account.
/// * `SignerNotFound` - The caller has not approved the proposal.
#[pallet::call_index(3)]
#[pallet::weight(Weight::default())]
pub fn reject(
origin: OriginFor&lt;T&gt;,
multisig_account: T::AccountId,
call_or_hash: CallOrHash,
) -&gt; DispatchResult
<span class="boring">}</span></code></pre></pre>
<ul>
<li><code>execute_proposal</code> - Execute a multisig proposal. (Releases Deposit)</li>
</ul>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> /// Executes a proposal for a dispatchable call for a multisig account.
/// Poropsal needs to be approved by enough signers (exceeds or equal multisig threshold) before it can be executed.
/// The caller must be one of the signers of the multisig account.
///
/// This function does an extra check to make sure that all approvers still exist in the multisig account.
/// That is to make sure that the multisig account is not compromised by removing an signer during an active proposal.
///
/// Once finished, the withheld deposit will be returned to the proposal creator.
///
/// # Arguments
///
/// * `multisig_account` - The multisig account ID.
/// * `call_or_hash` - We should have gotten the RuntimeCall (preimage) and stored it in the proposal by the time the extrinsic is called.
///
/// # Errors
///
/// * `MultisigNotFound` - The multisig account does not exist.
/// * `UnAuthorizedSigner` - The caller is not an signer of the multisig account.
/// * `NotEnoughApprovers` - approvers don't exceed the threshold.
/// * `ProposalNotFound` - The proposal does not exist.
/// * `CallPreImageNotFound` - The proposal doesn't have the preimage of the call in the state.
pub fn execute_proposal(
origin: OriginFor&lt;T&gt;,
multisig_account: T::AccountId,
call_or_hash: CallOrHash,
) -&gt; DispatchResult
<span class="boring">}</span></code></pre></pre>
<ul>
<li><code>cancel_proposal</code> - Cancel a multisig proposal. (Releases Deposit)</li>
</ul>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> /// Cancels an existing proposal for a multisig account.
/// Poropsal needs to be rejected by enough signers (exceeds or equal multisig threshold) before it can be executed.
/// The caller must be one of the signers of the multisig account.
///
/// This function does an extra check to make sure that all rejectors still exist in the multisig account.
/// That is to make sure that the multisig account is not compromised by removing an signer during an active proposal.
///
/// Once finished, the withheld deposit will be returned to the proposal creator./
///
/// # Arguments
///
/// * `origin` - The origin multisig account who wants to cancel the proposal.
/// * `call_or_hash` - The call or hash of the call to be canceled.
///
/// # Errors
///
/// * `MultisigNotFound` - The multisig account does not exist.
/// * `ProposalNotFound` - The proposal does not exist.
pub fn cancel_proposal(
origin: OriginFor&lt;T&gt;,
multisig_account: T::AccountId,
call_or_hash: CallOrHash) -&gt; DispatchResult
<span class="boring">}</span></code></pre></pre>
<ul>
<li><code>cancel_own_proposal</code> - Cancel a multisig proposal started by the caller in case no other signers approved it yet. (Releases Deposit)</li>
</ul>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> /// Cancels an existing proposal for a multisig account Only if the proposal doesn't have approvers other than
/// the proposer.
///
/// This function needs to be called from a the proposer of the proposal as the origin.
///
/// The withheld deposit will be returned to the proposal creator.
///
/// # Arguments
///
/// * `multisig_account` - The multisig account ID.
/// * `call_or_hash` - The hash of the call to be canceled.
///
/// # Errors
///
/// * `MultisigNotFound` - The multisig account does not exist.
/// * `ProposalNotFound` - The proposal does not exist.
pub fn cancel_own_proposal(
origin: OriginFor&lt;T&gt;,
multisig_account: T::AccountId,
call_or_hash: CallOrHash,
) -&gt; DispatchResult
<span class="boring">}</span></code></pre></pre>
<ul>
<li><code>cleanup_proposals</code> - Cleanup proposals of a multisig account. (Releases Deposit)</li>
</ul>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> /// Cleanup proposals of a multisig account. This function will iterate over a max limit per extrinsic to ensure
/// we don't have unbounded iteration over the proposals.
///
/// The withheld deposit will be returned to the proposal creator.
///
/// # Arguments
///
/// * `multisig_account` - The multisig account ID.
///
/// # Errors
///
/// * `MultisigNotFound` - The multisig account does not exist.
/// * `ProposalNotFound` - The proposal does not exist.
pub fn cleanup_proposals(
origin: OriginFor&lt;T&gt;,
multisig_account: T::AccountId,
) -&gt; DispatchResult
<span class="boring">}</span></code></pre></pre>
<p>Note: Next functions need to be called from the multisig account itself. Deposits are reserved from the multisig account as well.</p>
<ul>
<li><code>add_signer</code> - Add a new signer to a multisig account. (Needs Deposit)</li>
</ul>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> /// Adds a new signer to the multisig account.
/// This function needs to be called from a Multisig account as the origin.
/// Otherwise it will fail with MultisigNotFound error.
///
/// T::PerSignerDeposit will be held from the multisig account.
///
/// # Arguments
///
/// * `origin` - The origin multisig account who wants to add a new signer to the multisig account.
/// * `new_signer` - The AccountId of the new signer to be added.
/// * `new_threshold` - The new threshold for the multisig account after adding the new signer.
///
/// # Errors
/// * `MultisigNotFound` - The multisig account does not exist.
/// * `InvalidThreshold` - The threshold is greater than the total number of signers or is zero.
/// * `TooManySignatories` - The number of signatories exceeds the maximum allowed.
pub fn add_signer(
origin: OriginFor&lt;T&gt;,
new_signer: T::AccountId,
new_threshold: u32,
) -&gt; DispatchResult
<span class="boring">}</span></code></pre></pre>
<ul>
<li><code>remove_signer</code> - Remove an signer from a multisig account. (Releases Deposit)</li>
</ul>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> /// Removes an signer from the multisig account.
/// This function needs to be called from a Multisig account as the origin.
/// Otherwise it will fail with MultisigNotFound error.
/// If only one signer exists and is removed, the multisig account and any pending proposals for this account will be deleted from the state.
///
/// # Arguments
///
/// * `origin` - The origin multisig account who wants to remove an signer from the multisig account.
/// * `signer_to_remove` - The AccountId of the signer to be removed.
/// * `new_threshold` - The new threshold for the multisig account after removing the signer. Accepts zero if
/// the signer is the only one left.kkk
///
/// # Errors
///
/// This function can return the following errors:
///
/// * `MultisigNotFound` - The multisig account does not exist.
/// * `InvalidThreshold` - The new threshold is greater than the total number of signers or is zero.
/// * `UnAuthorizedSigner` - The caller is not an signer of the multisig account.
pub fn remove_signer(
origin: OriginFor&lt;T&gt;,
signer_to_remove: T::AccountId,
new_threshold: u32,
) -&gt; DispatchResult
<span class="boring">}</span></code></pre></pre>
<ul>
<li><code>set_threshold</code> - Change the threshold of a multisig account.</li>
</ul>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> /// Sets a new threshold for a multisig account.
/// This function needs to be called from a Multisig account as the origin.
/// Otherwise it will fail with MultisigNotFound error.
///
/// # Arguments
///
/// * `origin` - The origin multisig account who wants to set the new threshold.
/// * `new_threshold` - The new threshold to be set.
/// # Errors
///
/// * `MultisigNotFound` - The multisig account does not exist.
/// * `InvalidThreshold` - The new threshold is greater than the total number of signers or is zero.
set_threshold(origin: OriginFor&lt;T&gt;, new_threshold: u32) -&gt; DispatchResult
<span class="boring">}</span></code></pre></pre>
<ul>
<li><code>delete_multisig</code> - Delete a multisig account. (Releases Deposit)</li>
</ul>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> /// Deletes a multisig account and all related proposals.
///
/// This function needs to be called from a Multisig account as the origin.
/// Otherwise it will fail with MultisigNotFound error.
///
/// # Arguments
///
/// * `origin` - The origin multisig account who wants to cancel the proposal.
///
/// # Errors
///
/// * `MultisigNotFound` - The multisig account does not exist.
pub fn delete_account(origin: OriginFor&lt;T&gt;) -&gt; DispatchResult
<span class="boring">}</span></code></pre></pre>
<h3 id="storagestate"><a class="header" href="#storagestate">Storage/State</a></h3>
<ul>
<li>Use 2 main storage maps to store mutlisig accounts and proposals.</li>
</ul>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>#[pallet::storage]
pub type MultisigAccount&lt;T: Config&gt; = StorageMap&lt;_, Twox64Concat, T::AccountId, MultisigAccountDetails&lt;T&gt;&gt;;
/// The set of open multisig proposals. A proposal is uniquely identified by the multisig account and the call hash.
/// (maybe a nonce as well in the future)
#[pallet::storage]
pub type PendingProposals&lt;T: Config&gt; = StorageDoubleMap&lt;
_,
Twox64Concat,
T::AccountId, // Multisig Account
Blake2_128Concat,
T::Hash, // Call Hash
MultisigProposal&lt;T&gt;,
&gt;;
<span class="boring">}</span></code></pre></pre>
<p>As for the values:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>pub struct MultisigAccountDetails&lt;T: Config&gt; {
/// The signers of the multisig account. This is a BoundedBTreeSet to ensure faster operations (add, remove).
/// As well as lookups and faster set operations to ensure approvers is always a subset from signers. (e.g. in case of removal of an signer during an active proposal)
pub signers: BoundedBTreeSet&lt;T::AccountId, T::MaxSignatories&gt;,
/// The threshold of approvers required for the multisig account to be able to execute a call.
pub threshold: u32,
pub deposit: BalanceOf&lt;T&gt;,
}
<span class="boring">}</span></code></pre></pre>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>pub struct MultisigProposal&lt;T: Config&gt; {
/// Proposal creator.
pub creator: T::AccountId,
pub creation_deposit: BalanceOf&lt;T&gt;,
/// The extrinsic when the multisig operation was opened.
pub when: Timepoint&lt;BlockNumberFor&lt;T&gt;&gt;,
/// The approvers achieved so far, including the depositor.
/// The approvers are stored in a BoundedBTreeSet to ensure faster lookup and operations (approve, reject).
/// It's also bounded to ensure that the size don't go over the required limit by the Runtime.
pub approvers: BoundedBTreeSet&lt;T::AccountId, T::MaxSignatories&gt;,
/// The rejectors for the proposal so far.
/// The rejectors are stored in a BoundedBTreeSet to ensure faster lookup and operations (approve, reject).
/// It's also bounded to ensure that the size don't go over the required limit by the Runtime.
pub rejectors: BoundedBTreeSet&lt;T::AccountId, T::MaxSignatories&gt;,
/// The block number until which this multisig operation is valid. None means no expiry.
pub expire_after: Option&lt;BlockNumberFor&lt;T&gt;&gt;,
}
<span class="boring">}</span></code></pre></pre>
<p>For optimization we're using BoundedBTreeSet to allow for efficient lookups and removals. Especially in the case of approvers, we need to be able to remove an approver from the list when they reject their approval. (which we do lazily when <code>execute_proposal</code> is called).</p>
<p>There's an extra storage map for the deposits of the multisig accounts per signer added. This is to ensure that we can release the deposits when the multisig removes them even if the constant deposit per signer changed in the runtime later on.</p>
<h3 id="considerations--edge-cases"><a class="header" href="#considerations--edge-cases">Considerations &amp; Edge cases</a></h3>
<h4 id="removing-an-signer-from-the-multisig-account-during-an-active-proposal"><a class="header" href="#removing-an-signer-from-the-multisig-account-during-an-active-proposal">Removing an signer from the multisig account during an active proposal</a></h4>
<p>We need to ensure that the approvers are always a subset from signers. This is also partially why we're using BoundedBTreeSet for signers and approvers. Once execute proposal is called we ensure that the proposal is still valid and the approvers are still a subset from current signers.</p>
<h4 id="multisig-account-deletion-and-cleaning-up-existing-proposals"><a class="header" href="#multisig-account-deletion-and-cleaning-up-existing-proposals">Multisig account deletion and cleaning up existing proposals</a></h4>
<p>Once the last signer of a multisig account is removed or the multisig approved the account deletion we delete the multisig accound from the state and keep the proposals until someone calls <code>cleanup_proposals</code> multiple times which iterates over a max limit per extrinsic. This is to ensure we don't have unbounded iteration over the proposals. Users are already incentivized to call <code>cleanup_proposals</code> to get their deposits back.</p>
<h4 id="multisig-account-deletion-and-existing-deposits"><a class="header" href="#multisig-account-deletion-and-existing-deposits">Multisig account deletion and existing deposits</a></h4>
<p>We currently just delete the account without checking for deposits (Would like to hear your thoughts here). We can either</p>
<ul>
<li>Don't make deposits to begin with and make it a fee.</li>
<li>Transfer to treasury.</li>
<li>Error on deletion. (don't like this)</li>
</ul>
<h4 id="approving-a-proposal-after-the-threshold-is-changed"><a class="header" href="#approving-a-proposal-after-the-threshold-is-changed">Approving a proposal after the threshold is changed</a></h4>
<p>We always use latest threshold and don't store each proposal with different threshold. This allows the following:</p>
<ul>
<li>In case threshold is lower than the number of approvers then the proposal is still valid.</li>
<li>In case threshold is higher than the number of approvers then we catch it during execute proposal and error.</li>
</ul>
<h2 id="drawbacks-9"><a class="header" href="#drawbacks-9">Drawbacks</a></h2>
<ul>
<li>New pallet to maintain.</li>
</ul>
<h2 id="testing-security-and-privacy-10"><a class="header" href="#testing-security-and-privacy-10">Testing, Security, and Privacy</a></h2>
<p>Standard audit/review requirements apply.</p>
<h2 id="performance-ergonomics-and-compatibility-10"><a class="header" href="#performance-ergonomics-and-compatibility-10">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-11"><a class="header" href="#performance-11">Performance</a></h3>
<p>Doing back of the envelop calculation to proof that the stateful multisig is more efficient than the stateless multisig given it's smaller footprint size on blocks.</p>
<p>Quick review over the extrinsics for both as it affects the block size:</p>
<p>Stateless Multisig:
Both <code>as_multi</code> and <code>approve_as_multi</code> has a similar parameters:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>origin: OriginFor&lt;T&gt;,
threshold: u16,
other_signatories: Vec&lt;T::AccountId&gt;,
maybe_timepoint: Option&lt;Timepoint&lt;BlockNumberFor&lt;T&gt;&gt;&gt;,
call_hash: [u8; 32],
max_weight: Weight,
<span class="boring">}</span></code></pre></pre>
<p>Stateful Multisig:
We have the following extrinsics:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>pub fn start_proposal(
origin: OriginFor&lt;T&gt;,
multisig_account: T::AccountId,
call_or_hash: CallOrHash,
)
<span class="boring">}</span></code></pre></pre>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>pub fn approve(
origin: OriginFor&lt;T&gt;,
multisig_account: T::AccountId,
call_or_hash: CallOrHash,
)
<span class="boring">}</span></code></pre></pre>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>pub fn execute_proposal(
origin: OriginFor&lt;T&gt;,
multisig_account: T::AccountId,
call_or_hash: CallOrHash,
)
<span class="boring">}</span></code></pre></pre>
<p>The main takeway is that we don't need to pass the threshold and other signatories in the extrinsics. This is because we already have the threshold and signatories in the state (only once).</p>
<p>So now for the caclulations, given the following:</p>
<ul>
<li>K is the number of multisig accounts.</li>
<li>N is number of signers in each multisig account.</li>
<li>For each proposal we need to have 2N/3 approvals.</li>
</ul>
<p>The table calculates if each of the K multisig accounts has one proposal and it gets approved by the 2N/3 and then executed. How much did the total Blocks and States sizes increased by the end of the day.</p>
<p>Note: We're not calculating the cost of proposal as both in statefull and stateless multisig they're almost the same and gets cleaned up from the state once the proposal is executed or canceled.</p>
<p>Stateless effect on blocksizes = 2/3<em>K</em>N^2 (as each user of the 2/3 users will need to call approve_as_multi with all the other signatories(N) in extrinsic body)</p>
<p>Stateful effect on blocksizes = K * N (as each user will need to call approve with the multisig account only in extrinsic body)</p>
<p>Stateless effect on statesizes = Nil (as the multisig account is not stored in the state)</p>
<p>Stateful effect on statesizes = K*N (as each multisig account (K) will be stored with all the signers (K) in the state)</p>
<div class="table-wrapper"><table><thead><tr><th>Pallet</th><th style="text-align: center">Block Size</th><th style="text-align: right">State Size</th></tr></thead><tbody>
<tr><td>Stateless</td><td style="text-align: center">2/3<em>K</em>N^2</td><td style="text-align: right">Nil</td></tr>
<tr><td>Stateful</td><td style="text-align: center">K*N</td><td style="text-align: right">K*N</td></tr>
</tbody></table>
</div>
<p>Simplified table removing K from the equation:
| Pallet | Block Size | State Size |
|----------------|:-------------:|-----------:|
| Stateless | N^2 | Nil |
| Stateful | N | N |</p>
<p>So even though the stateful multisig has a larger state size, it's still more efficient in terms of block size and total footprint on the blockchain.</p>
<h3 id="ergonomics-9"><a class="header" href="#ergonomics-9">Ergonomics</a></h3>
<p>The Stateful Multisig will have better ergonomics for managing multisig accounts for both developers and end-users.</p>
<h3 id="compatibility-8"><a class="header" href="#compatibility-8">Compatibility</a></h3>
<p>This RFC is compatible with the existing implementation and can be handled via upgrades and migration. It's not intended to replace the existing multisig pallet.</p>
<h2 id="prior-art-and-references-7"><a class="header" href="#prior-art-and-references-7">Prior Art and References</a></h2>
<p><a href="https://github.com/paritytech/polkadot-sdk/tree/master/substrate/frame/multisig">multisig pallet in polkadot-sdk</a></p>
<h2 id="unresolved-questions-10"><a class="header" href="#unresolved-questions-10">Unresolved Questions</a></h2>
<ul>
<li>On account deletion, should we transfer remaining deposits to treasury or remove signers' addition deposits completely and consider it as fees to start with?</li>
</ul>
<h2 id="future-directions-and-related-material-7"><a class="header" href="#future-directions-and-related-material-7">Future Directions and Related Material</a></h2>
<ul>
<li><input disabled="" type="checkbox"/>
Batch addition/removal of signers.</li>
<li><input disabled="" type="checkbox"/>
Add expiry to proposals. After a certain time, proposals will not accept any more approvals or executions and will be deleted.</li>
<li><input disabled="" type="checkbox"/>
Implement call filters. This will allow multisig accounts to only accept certain calls.</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/blob/main/text/0001-agile-coretime.md">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="approved/0001-agile-coretime.html#rfc-1-agile-coretime">RFC-1: Agile Coretime</a>
<ul>
<li><a href="approved/0001-agile-coretime.html#summary">Summary</a></li>
<li><a href="approved/0001-agile-coretime.html#motivation">Motivation</a>
<ul>
<li><a href="approved/0001-agile-coretime.html#present-system">Present System</a></li>
<li><a href="approved/0001-agile-coretime.html#problems">Problems</a></li>
</ul>
</li>
<li><a href="approved/0001-agile-coretime.html#requirements">Requirements</a></li>
<li><a href="approved/0001-agile-coretime.html#stakeholders">Stakeholders</a></li>
<li><a href="approved/0001-agile-coretime.html#explanation">Explanation</a>
<ul>
<li><a href="approved/0001-agile-coretime.html#overview">Overview</a></li>
<li><a href="approved/0001-agile-coretime.html#detail">Detail</a></li>
<li><a href="approved/0001-agile-coretime.html#specific-functions-of-the-coretime-chain">Specific functions of the Coretime-chain</a></li>
<li><a href="approved/0001-agile-coretime.html#notes-on-the-instantaneous-coretime-market">Notes on the Instantaneous Coretime Market</a></li>
<li><a href="approved/0001-agile-coretime.html#notes-on-economics">Notes on Economics</a></li>
<li><a href="approved/0001-agile-coretime.html#notes-on-types">Notes on Types</a></li>
<li><a href="approved/0001-agile-coretime.html#rollout">Rollout</a></li>
</ul>
</li>
<li><a href="approved/0001-agile-coretime.html#performance-ergonomics-and-compatibility">Performance, Ergonomics and Compatibility</a></li>
<li><a href="approved/0001-agile-coretime.html#testing-security-and-privacy">Testing, Security and Privacy</a></li>
<li><a href="approved/0001-agile-coretime.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
<li><a href="approved/0001-agile-coretime.html#drawbacks-alternatives-and-unknowns">Drawbacks, Alternatives and Unknowns</a></li>
<li><a href="approved/0001-agile-coretime.html#prior-art-and-references">Prior Art and References</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-1-agile-coretime"><a class="header" href="#rfc-1-agile-coretime">RFC-1: Agile Coretime</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>30 June 2023</td></tr>
<tr><td><strong>Description</strong></td><td>Agile periodic-sale-based model for assigning Coretime on the Polkadot Ubiquitous Computer.</td></tr>
<tr><td><strong>Authors</strong></td><td>Gavin Wood</td></tr>
</tbody></table>
</div>
<h2 id="summary-11"><a class="header" href="#summary-11">Summary</a></h2>
<p>This proposes a periodic, sale-based method for assigning Polkadot Coretime, the analogue of &quot;block space&quot; within the Polkadot Network. The method takes into account the need for long-term capital expenditure planning for teams building on Polkadot, yet also provides a means to allow Polkadot to capture long-term value in the resource which it sells. It supports the possibility of building rich and dynamic secondary markets to optimize resource allocation and largely avoids the need for parameterization.</p>
<h2 id="motivation-11"><a class="header" href="#motivation-11">Motivation</a></h2>
<h3 id="present-system"><a class="header" href="#present-system">Present System</a></h3>
<p>The <em>Polkadot Ubiquitous Computer</em>, or just <em>Polkadot UC</em>, represents the public service provided by the Polkadot Network. It is a trust-free, WebAssembly-based, multicore, internet-native omnipresent virtual machine which is highly resilient to interference and corruption.</p>
<p>The present system of allocating the limited resources of the Polkadot Ubiquitous Computer is through a process known as <em>parachain slot auctions</em>. This is a parachain-centric paradigm whereby a single core is long-term allocated to a single parachain which itself implies a Substrate/Cumulus-based chain secured and connected via the Relay-chain. Slot auctions are on-chain candle auctions which proceed for several days and result in the core being assigned to the parachain for six months at a time up to 24 months in advance. Practically speaking, we only see two year periods being bid upon and leased.</p>
<p>Funds behind the bids made in the slot auctions are merely locked, they are not consumed or paid and become unlocked and returned to the bidder on expiry of the lease period. A means of sharing the deposit trustlessly known as a <em>crowdloan</em> is available allowing token holders to contribute to the overall deposit of a chain without any counterparty risk.</p>
<h3 id="problems"><a class="header" href="#problems">Problems</a></h3>
<p>The present system is based on a model of one-core-per-parachain. This is a legacy interpretation of the Polkadot platform and is not a reflection of its present capabilities. By restricting ownership and usage to this model, more dynamic and resource-efficient means of utilizing the Polkadot Ubiquitous Computer are lost.</p>
<p>More specifically, it is impossible to lease out cores at anything less than six months, and apparently unrealistic to do so at anything less than two years. This removes the ability to dynamically manage the underlying resource, and generally experimentation, iteration and innovation suffer. It bakes into the platform an assumption of permanence for anything deployed into it and restricts the market's ability to find a more optimal allocation of the finite resource.</p>
<p>There is no ability to determine capital requirements for hosting a parachain beyond two years from the point of its initial deployment onto Polkadot. While it would be unreasonable to have perfect and indefinite cost predictions for any real-world platform, not having any clarity whatsoever beyond &quot;market rates&quot; two years hence can be a very off-putting prospect for teams to buy into.</p>
<p>However, quite possibly the most substantial problem is both a perceived and often real high barrier to entry of the Polkadot ecosystem. By forcing innovators to either raise seven-figure sums through investors or appeal to the wider token-holding community, Polkadot makes it difficult for a small band of innovators to deploy their technology into Polkadot. While not being actually permissioned, it is also far from the barrierless, permissionless ideal which an innovation platform such as Polkadot should be striving for.</p>
<h2 id="requirements-3"><a class="header" href="#requirements-3">Requirements</a></h2>
<ol>
<li>The solution SHOULD provide an acceptable value-capture mechanism for the Polkadot network.</li>
<li>The solution SHOULD allow parachains and other projects deployed on to the Polkadot UC to make long-term capital expenditure predictions for the cost of ongoing deployment.</li>
<li>The solution SHOULD minimize the barriers to entry in the ecosystem.</li>
<li>The solution SHOULD work well when the Polkadot UC has up to 1,000 cores.</li>
<li>The solution SHOULD work when the number of cores which the Polkadot UC can support changes over time.</li>
<li>The solution SHOULD facilitate the optimal allocation of work to cores of the Polkadot UC, including by facilitating the trade of regular core assignment at various intervals and for various spans.</li>
<li>The solution SHOULD avoid creating additional dependencies on functionality which the Relay-chain need not strictly provide for the delivery of the Polkadot UC.</li>
</ol>
<p>Furthermore, the design SHOULD be implementable and deployable in a timely fashion; three months from the acceptance of this RFC should not be unreasonable.</p>
<h2 id="stakeholders-10"><a class="header" href="#stakeholders-10">Stakeholders</a></h2>
<p>Primary stakeholder sets are:</p>
<ul>
<li>Protocol researchers and developers, largely represented by the Polkadot Fellowship and Parity Technologies' Engineering division.</li>
<li>Polkadot Parachain teams both present and future, and their users.</li>
<li>Polkadot DOT token holders.</li>
</ul>
<p><em>Socialization:</em></p>
<p>The essensials of this proposal were presented at Polkadot Decoded 2023 Copenhagen on the Main Stage. A small amount of socialization at the Parachain Summit preceeded it and some substantial discussion followed it. Parity Ecosystem team is currently soliciting views from ecosystem teams who would be key stakeholders.</p>
<h2 id="explanation-11"><a class="header" href="#explanation-11">Explanation</a></h2>
<h3 id="overview"><a class="header" href="#overview">Overview</a></h3>
<p>Upon implementation of this proposal, the parachain-centric slot auctions and associated crowdloans cease. Instead, Coretime on the Polkadot UC is sold by the Polkadot System in two separate formats: <em>Bulk Coretime</em> and <em>Instantaneous Coretime</em>.</p>
<p>When a Polkadot Core is utilized, we say it is dedicated to a <em>Task</em> rather than a &quot;parachain&quot;. The Task to which a Core is dedicated may change at every Relay-chain block and while one predominant type of Task is to secure a Cumulus-based blockchain (i.e. a parachain), other types of Tasks are envisioned.</p>
<p>Bulk Coretime is sold periodically on a specialised system chain known as the <em>Coretime-chain</em> and allocated in advance of its usage, whereas Instantaneous Coretime is sold on the Relay-chain immediately prior to usage on a block-by-block basis.</p>
<p>This proposal does not fix what should be done with revenue from sales of Coretime and leaves it for a further RFC process.</p>
<p>Owners of Bulk Coretime are tracked on the Coretime-chain and the ownership status and properties of the owned Coretime are exposed over XCM as a non-fungible asset.</p>
<p>At the request of the owner, the Coretime-chain allows a single Bulk Coretime asset, known as a <em>Region</em>, to be used in various ways including transferal to another owner, allocated to a particular task (e.g. a parachain) or placed in the Instantaneous Coretime Pool. Regions can also be split out, either into non-overlapping sub-spans or exactly-overlapping spans with less regularity.</p>
<p>The Coretime-Chain periodically instructs the Relay-chain to assign its cores to alternative tasks as and when Core allocations change due to new Regions coming into effect.</p>
<h4 id="renewal-and-migration"><a class="header" href="#renewal-and-migration">Renewal and Migration</a></h4>
<p>There is a renewal system which allows a Bulk Coretime assignment of a single core to be renewed unchanged with a known price increase from month to month. Renewals are processed in a period prior to regular purchases, effectively giving them precedence over a fixed number of cores available.</p>
<p>Renewals are only enabled when a core's assignment does not include an Instantaneous Coretime allocation and has not been split into shorter segments.</p>
<p>Thus, renewals are designed to ensure only that committed parachains get some guarantees about price for predicting future costs. This price-capped renewal system only allows cores to be reused for their same tasks from month to month. In any other context, Bulk Coretime would need to be purchased regularly.</p>
<p>As a migration mechanism, pre-existing leases (from the legacy lease/slots/crowdloan framework) are initialized into the Coretime-chain and cores assigned to them prior to Bulk Coretime sales. In the sale where the lease expires, the system offers a renewal, as above, to allow a priority sale of Bulk Coretime and ensure that the Parachain suffers no downtime when transitioning from the legacy framework.</p>
<h4 id="instantaneous-coretime"><a class="header" href="#instantaneous-coretime">Instantaneous Coretime</a></h4>
<p>Processing of Instantaneous Coretime happens in part on the Polkadot Relay-chain. Credit is purchased on the Coretime-chain for regular DOT tokens, and this results in a DOT-denominated Instantaneous Coretime Credit account on the Relay-chain being credited for the same amount.</p>
<p>Though the Instantaneous Coretime Credit account records a balance for an account identifier (very likely controlled by a collator), it is <em>non-transferable</em> and <em>non-refundable</em>. It can only be consumed in order to purchase some Instantaneous Coretime with immediate availability.</p>
<p>The Relay-chain reports this usage back to the Coretime-chain in order to allow it to reward the providers of the underlying Coretime, either the Polkadot System or owners of Bulk Coretime who contributed to the Instantaneous Coretime Pool.</p>
<p>Specifically the Relay-chain is expected to be responsible for:</p>
<ul>
<li>holding non-transferable, non-refundable DOT-denominated Instantaneous Coretime Credit balance information.</li>
<li>setting and adjusting the price of Instantaneous Coretime based on usage.</li>
<li>allowing collators to consume their Instantaneous Coretime Credit at the current pricing in exchange for the ability to schedule one PoV for near-immediate usage.</li>
<li>ensuring the Coretime-Chain has timely accounting information on Instantaneous Coretime Sales revenue.</li>
</ul>
<h4 id="coretime-chain"><a class="header" href="#coretime-chain">Coretime-chain</a></h4>
<p>The <em>Coretime-chain</em> is a new system parachain. It has the responsibility of providing the Relay-chain via UMP with information of:</p>
<ul>
<li>The number of cores which should be made available.</li>
<li>Which tasks should be running on which cores and in what ratios.</li>
<li>Accounting information for Instantaneous Coretime Credit.</li>
</ul>
<p>It also expects information from the Relay-chain via DMP:</p>
<ul>
<li>The number of cores available to be scheduled.</li>
<li>Account information on Instantaneous Coretime Sales.</li>
</ul>
<p>The specific interface is properly described in RFC-5.</p>
<h3 id="detail"><a class="header" href="#detail">Detail</a></h3>
<h4 id="parameters"><a class="header" href="#parameters">Parameters</a></h4>
<p>This proposal includes a number of parameters which need not necessarily be fixed. Their usage is explained below, but their values are suggested or specified in the later section <em>Parameter Values</em>.</p>
<h4 id="reservations-and-leases"><a class="header" href="#reservations-and-leases">Reservations and Leases</a></h4>
<p>The Coretime-chain includes some governance-set reservations of Coretime; these cover every System-chain. Additionally, governance is expected to initialize details of the pre-existing leased chains.</p>
<h4 id="regions"><a class="header" href="#regions">Regions</a></h4>
<p>A <em>Region</em> is an assignable period of Coretime with a known regularity.</p>
<p>All Regions are associated with a unique <em>Core Index</em>, to identify which core the assignment of which ownership of the Region controls.</p>
<p>All Regions are also associated with a <em>Core Mask</em>, an 80-bit bitmap, to denote the regularity at which it may be scheduled on the core. If all bits are set in the Core Mask value, it is said to be <em>Complete</em>. 80 is selected since this results in the size of the datatype used to identify any Region of Polkadot Coretime to be a very convenient 128-bit. Additionally, if <code>TIMESLICE</code> (the number of Relay-chain blocks in a Timeslice) is 80, then a single bit in the Core Mask bitmap represents exactly one Core for one Relay-chain block in one Timeslice.</p>
<p>All Regions have a span. Region spans are quantized into periods of <code>TIMESLICE</code> blocks; <code>BULK_PERIOD</code> divides into <code>TIMESLICE</code> a whole number of times.</p>
<p>The <code>Timeslice</code> type is a <code>u32</code> which can be multiplied by <code>TIMESLICE</code> to give a <code>BlockNumber</code> value representing the same quantity in terms of Relay-chain blocks.</p>
<p>Regions can be tasked to a <code>TaskId</code> (aka <code>ParaId</code>) or pooled into the Instantaneous Coretime Pool. This process can be <em>Provisional</em> or <em>Final</em>. If done only provisionally or not at all then they are fresh and have an <em>Owner</em> which is able to manipulate them further including reassignment. Once <em>Final</em>, then all ownership information is discarded and they cannot be manipulated further. Renewal is not possible when only provisionally tasked/pooled.</p>
<h4 id="bulk-sales"><a class="header" href="#bulk-sales">Bulk Sales</a></h4>
<p>A sale of Bulk Coretime occurs on the Coretime-chain every <code>BULK_PERIOD</code> blocks.</p>
<p>In every sale, a <code>BULK_LIMIT</code> of individual <em>Regions</em> are offered for sale.</p>
<p>Each Region offered for sale has a different Core Index, ensuring that they each represent an independently allocatable resource on the Polkadot UC.</p>
<p>The Regions offered for sale have the same span: they last exactly <code>BULK_PERIOD</code> blocks, and begin immediately following the span of the previous Sale's Regions. The Regions offered for sale also have the complete, non-interlaced, Core Mask.</p>
<p>The Sale Period ends immediately as soon as span of the Coretime Regions that are being sold begins. At this point, the next Sale Price is set according to the previous Sale Price together with the number of Regions sold compared to the desired and maximum amount of Regions to be sold. See Price Setting for additional detail on this point.</p>
<p>Following the end of the previous Sale Period, there is an <em>Interlude Period</em> lasting <code>INTERLUDE_PERIOD</code> of blocks. After this period is elapsed, regular purchasing begins with the <em>Purchasing Period</em>.</p>
<p>This is designed to give at least two weeks worth of time for the purchased regions to be partitioned, interlaced, traded and allocated.</p>
<h4 id="the-interlude"><a class="header" href="#the-interlude">The Interlude</a></h4>
<p>The Interlude period is a period prior to Regular Purchasing where renewals are allowed to happen. This has the effect of ensuring existing long-term tasks/parachains have a chance to secure their Bulk Coretime for a well-known price prior to general sales.</p>
<h4 id="regular-purchasing"><a class="header" href="#regular-purchasing">Regular Purchasing</a></h4>
<p>Any account may purchase Regions of Bulk Coretime if they have the appropriate funds in place during the Purchasing Period, which is from <code>INTERLUDE_PERIOD</code> blocks after the end of the previous sale until the beginning of the Region of the Bulk Coretime which is for sale as long as there are Regions of Bulk Coretime left for sale (i.e. no more than <code>BULK_LIMIT</code> have already been sold in the Bulk Coretime Sale). The Purchasing Period is thus roughly <code>BULK_PERIOD - INTERLUDE_PERIOD</code> blocks in length.</p>
<p>The Sale Price varies during an initial portion of the Purchasing Period called the <em>Leadin Period</em> and then stays stable for the remainder. This initial portion is <code>LEADIN_PERIOD</code> blocks in duration. During the Leadin Period the price decreases towards the Sale Price, which it lands at by the end of the Leadin Period. The actual curve by which the price starts and descends to the Sale Price is outside the scope of this RFC, though a basic suggestion is provided in the Price Setting Notes, below.</p>
<h4 id="renewals"><a class="header" href="#renewals">Renewals</a></h4>
<p>At any time when there are remaining Regions of Bulk Coretime to be sold, <em>including during the Interlude Period</em>, then certain Bulk Coretime assignmnents may be <em>Renewed</em>. This is similar to a purchase in that funds must be paid and it consumes one of the Regions of Bulk Coretime which would otherwise be placed for purchase. However there are two key differences.</p>
<p>Firstly, the price paid is the minimum of <code>RENEWAL_PRICE_CAP</code> more than what the purchase/renewal price was in the previous renewal and the current (or initial, if yet to begin) regular Sale Price.</p>
<p>Secondly, the purchased Region comes preassigned with exactly the same workload as before. It cannot be traded, repartitioned, interlaced or exchanged. As such unlike regular purchasing the Region never has an owner.</p>
<p>Renewal is only possible for either cores which have been assigned as a result of a previous renewal, which are migrating from legacy slot leases, or which fill their Bulk Coretime with an unsegmented, fully and finally assigned workload which does not include placement in the Instantaneous Coretime Pool. The renewed workload will be the same as this initial workload.</p>
<h4 id="manipulation"><a class="header" href="#manipulation">Manipulation</a></h4>
<p>Regions may be manipulated in various ways by its owner:</p>
<ol>
<li><em>Transferred</em> in ownership.</li>
<li><em>Partitioned</em> into quantized, non-overlapping segments of Bulk Coretime with the same ownership.</li>
<li><em>Interlaced</em> into multiple Regions over the same period whose eventual assignments take turns to be scheduled.</li>
<li><em>Assigned</em> to a single, specific task (identified by <code>TaskId</code> aka <code>ParaId</code>). This may be either <em>provisional</em> or <em>final</em>.</li>
<li><em>Pooled</em> into the Instantaneous Coretime Pool, in return for a pro-rata amount of the revenue from the Instantaneous Coretime Sales over its period.</li>
</ol>
<h4 id="enactment"><a class="header" href="#enactment">Enactment</a></h4>
<h3 id="specific-functions-of-the-coretime-chain"><a class="header" href="#specific-functions-of-the-coretime-chain">Specific functions of the Coretime-chain</a></h3>
<p>Several functions of the Coretime-chain SHALL be exposed through dispatchables and/or a <code>nonfungible</code> trait implementation integrated into XCM:</p>
<h4 id="1-transfer"><a class="header" href="#1-transfer">1. <code>transfer</code></a></h4>
<p>Regions may have their ownership transferred.</p>
<p>A <code>transfer(region: RegionId, new_owner: AccountId)</code> dispatchable shall have the effect of altering the current owner of the Region identified by <code>region</code> from the signed origin to <code>new_owner</code>.</p>
<p>An implementation of the <code>nonfungible</code> trait SHOULD include equivalent functionality. <code>RegionId</code> SHOULD be used for the <code>AssetInstance</code> value.</p>
<h4 id="2-partition"><a class="header" href="#2-partition">2. <code>partition</code></a></h4>
<p>Regions may be split apart into two non-overlapping interior Regions of the same Core Mask which together concatenate to the original Region.</p>
<p>A <code>partition(region: RegionId, pivot: Timeslice)</code> dispatchable SHALL have the effect of removing the Region identified by <code>region</code> and adding two new Regions of the same owner and Core Mask. One new Region will begin at the same point of the old Region but end at <code>pivot</code> timeslices into the Region, whereas the other will begin at this point and end at the end point of the original Region.</p>
<p>Also:</p>
<ul>
<li><code>owner</code> field of <code>region</code> must the equal to the Signed origin.</li>
<li><code>pivot</code> must equal neither the <code>begin</code> nor <code>end</code> fields of the <code>region</code>.</li>
</ul>
<h4 id="3-interlace"><a class="header" href="#3-interlace">3. <code>interlace</code></a></h4>
<p>Regions may be decomposed into two Regions of the same span whose eventual assignments take turns on the core by virtue of having complementary Core Masks.</p>
<p>An <code>interlace(region: RegionId, mask: CoreMask)</code> dispatchable shall have the effect of removing the Region identified by <code>region</code> and creating two new Regions. The new Regions will each have the same span and owner of the original Region, but one Region will have a Core Mask equal to <code>mask</code> and the other will have Core Mask equal to the XOR of <code>mask</code> and the Core Mask of the original Region.</p>
<p>Also:</p>
<ul>
<li><code>owner</code> field of <code>region</code> must the equal to the Signed origin.</li>
<li><code>mask</code> must have some bits set AND must not equal the Core Mask of the old Region AND must only have bits set which are also set in the old Region's' Core Mask.</li>
</ul>
<h4 id="4-assign"><a class="header" href="#4-assign">4. <code>assign</code></a></h4>
<p>Regions may be assigned to a core.</p>
<p>A <code>assign(region: RegionId, target: TaskId, finality: Finality)</code> dispatchable shall have the effect of placing an item in the workplan corresponding to the region's properties and assigned to the <code>target</code> task.</p>
<p>If the region's end has already passed (taking into account any advance notice requirements) then this operation is a no-op. If the region's begining has already passed, then it is effectively altered to become the next schedulable timeslice.</p>
<p><code>finality</code> may have the value of either <code>Final</code> or <code>Provisional</code>. If <code>Final</code>, then the operation is free, the <code>region</code> record is removed entirely from storage and renewal may be possible: if the Region's span is the entire <code>BULK_PERIOD</code>, then the Coretime-chain records in storage that the allocation happened during this period in order to facilitate the possibility for a renewal. (Renewal only becomes possible when the full Core Mask of a core is finally assigned for the full <code>BULK_PERIOD</code>.)</p>
<p>Also:</p>
<ul>
<li><code>owner</code> field of <code>region</code> must the equal to the Signed origin.</li>
</ul>
<h4 id="5-pool"><a class="header" href="#5-pool">5. <code>pool</code></a></h4>
<p>Regions may be consumed in exchange for a pro rata portion of the Instantaneous Coretime Sales Revenue from its period and regularity.</p>
<p>A <code>pool(region: RegionId, beneficiary: AccountId, finality: Finality)</code> dispatchable shall have the effect of placing an item in the workplan corresponding to the region's properties and assigned to the Instantaneous Coretime Pool. The details of the region will be recorded in order to allow for a pro rata share of the Instantaneous Coretime Sales Revenue at the time of the Region relative to any other providers in the Pool.</p>
<p>If the region's end has already passed (taking into account any advance notice requirements) then this operation is a no-op. If the region's begining has already passed, then it is effectively altered to become the next schedulable timeslice.</p>
<p><code>finality</code> may have the value of either <code>Final</code> or <code>Provisional</code>. If <code>Final</code>, then the operation is free and the <code>region</code> record is removed entirely from storage.</p>
<p>Also:</p>
<ul>
<li><code>owner</code> field of <code>region</code> must the equal to the Signed origin.</li>
</ul>
<h4 id="6-purchases"><a class="header" href="#6-purchases">6. Purchases</a></h4>
<p>A dispatchable <code>purchase(price_limit: Balance)</code> shall be provided. Any account may call <code>purchase</code> to purchase Bulk Coretime at the maximum price of <code>price_limit</code>.</p>
<p>This may be called successfully only:</p>
<ol>
<li>during the regular Purchasing Period;</li>
<li>when the caller is a Signed origin and their account balance is reducible by the current sale price;</li>
<li>when the current sale price is no greater than <code>price_limit</code>; and</li>
<li>when the number of cores already sold is less than <code>BULK_LIMIT</code>.</li>
</ol>
<p>If successful, the caller's account balance is reduced by the current sale price and a new Region item for the following Bulk Coretime span is issued with the owner equal to the caller's account.</p>
<h4 id="7-renewals"><a class="header" href="#7-renewals">7. Renewals</a></h4>
<p>A dispatchable <code>renew(core: CoreIndex)</code> shall be provided. Any account may call <code>renew</code> to purchase Bulk Coretime and renew an active allocation for the given <code>core</code>.</p>
<p>This may be called during the Interlude Period as well as the regular Purchasing Period and has the same effect as <code>purchase</code> followed by <code>assign</code>, except that:</p>
<ol>
<li>The price of the sale is the Renewal Price (see next).</li>
<li>The Region is allocated exactly the given <code>core</code> is currently allocated for the present Region.</li>
</ol>
<p>Renewal is only valid where a Region's span is assigned to Tasks (not placed in the Instantaneous Coretime Pool) for the entire unsplit <code>BULK_PERIOD</code> over all of the Core Mask and with Finality. There are thus three possibilities of a renewal being allowed:</p>
<ol>
<li>Purchased unsplit Coretime with final assignment to tasks over the full Core Mask.</li>
<li>Renewed Coretime.</li>
<li>A legacy lease which is ending.</li>
</ol>
<p><strong>Renewal Price</strong></p>
<p>The Renewal Price is the minimum of the current regular Sale Price (or the initial Sale Price if in the Interlude Period) and:</p>
<ul>
<li>If the workload being renewed came to be through the <em>Purchase and Assignment</em> of Bulk Coretime, then the price paid during that Purchase operation.</li>
<li>If the workload being renewed was previously renewed, then the price paid during this previous Renewal operation plus <code>RENEWAL_PRICE_CAP</code>.</li>
<li>If the workload being renewed is a migation from a legacy slot auction lease, then the nominal price for a Regular Purchase (outside of the Lead-in Period) of the Sale during which the legacy lease expires.</li>
</ul>
<h4 id="8-instantaneous-coretime-credits"><a class="header" href="#8-instantaneous-coretime-credits">8. Instantaneous Coretime Credits</a></h4>
<p>A dispatchable <code>purchase_credit(amount: Balance, beneficiary: RelayChainAccountId)</code> shall be provided. Any account with at least <code>amount</code> spendable funds may call this. This increases the Instantaneous Coretime Credit balance on the Relay-chain of the <code>beneficiary</code> by the given <code>amount</code>.</p>
<p>This Credit is consumable on the Relay-chain as part of the Task scheduling system and its specifics are out of the scope of this proposal. When consumed, revenue is recorded and provided to the Coretime-chain for proper distribution. The API for doing this is specified in RFC-5.</p>
<h3 id="notes-on-the-instantaneous-coretime-market"><a class="header" href="#notes-on-the-instantaneous-coretime-market">Notes on the Instantaneous Coretime Market</a></h3>
<p>For an efficient market to form around the provision of Bulk-purchased Cores into the pool of cores available for Instantaneous Coretime purchase, it is crucial to ensure that price changes for the purchase of Instantaneous Coretime are reflected well in the revenues of private Coretime providers during the same period.</p>
<p>In order to ensure this, then it is crucial that Instantaneous Coretime, once purchased, cannot be held indefinitely prior to eventual use since, if this were the case, a nefarious collator could purchase Coretime when cheap and utilize it some time later when expensive and deprive private Coretime providers of their revenue.</p>
<p>It must therefore be assumed that Instantaneous Coretime, once purchased, has a definite and short &quot;shelf-life&quot;, after which it becomes unusable. This incentivizes collators to avoid purchasing Coretime unless they expect to utilize it imminently and thus helps create an efficient market-feedback mechanism whereby a higher price will actually result in material revenues for private Coretime providers who contribute to the pool of Cores available to service Instantaneous Coretime purchases.</p>
<h3 id="notes-on-economics"><a class="header" href="#notes-on-economics">Notes on Economics</a></h3>
<p>The specific pricing mechanisms are out of scope for the present proposal. Proposals on economics should be properly described and discussed in another RFC. However, for the sake of completeness, I provide some basic illustration of how price setting could potentially work.</p>
<h4 id="bulk-price-progression"><a class="header" href="#bulk-price-progression">Bulk Price Progression</a></h4>
<p>The present proposal assumes the existence of a price-setting mechanism which takes into account several parameters:</p>
<ul>
<li><code>OLD_PRICE</code>: The price of the previous sale.</li>
<li><code>BULK_TARGET</code>: the target number of cores to be purchased as Bulk Coretime Regions or renewed during the previous sale.</li>
<li><code>BULK_LIMIT</code>: the maximum number of cores which could have been purchased/renewed during the previous sale.</li>
<li><code>CORES_SOLD</code>: the actual number of cores purchased/renewed in the previous sale.</li>
<li><code>SELLOUT_PRICE</code>: the price at which the most recent Bulk Coretime was purchased (<em>not</em> renewed) prior to selling more cores than <code>BULK_TARGET</code> (or immediately after, if none were purchased before). This may not have a value if no Bulk Coretime was purchased.</li>
</ul>
<p>In general we would expect the price to increase the closer <code>CORES_SOLD</code> gets to <code>BULK_LIMIT</code> and to decrease the closer it gets to zero. If it is exactly equal to <code>BULK_TARGET</code>, then we would expect the price to remain the same.</p>
<p>In the edge case that no cores were purchased yet more cores were sold (through renewals) than the target, then we would also avoid altering the price.</p>
<p>A simple example of this would be the formula:</p>
<pre><code>IF SELLOUT_PRICE == NULL AND CORES_SOLD &gt; BULK_TARGET THEN
RETURN OLD_PRICE
END IF
EFFECTIVE_PRICE := IF CORES_SOLD &gt; BULK_TARGET THEN
SELLOUT_PRICE
ELSE
OLD_PRICE
END IF
NEW_PRICE := IF CORES_SOLD &lt; BULK_TARGET THEN
EFFECTIVE_PRICE * MAX(CORES_SOLD, 1) / BULK_TARGET
ELSE
EFFECTIVE_PRICE + EFFECTIVE_PRICE *
(CORES_SOLD - BULK_TARGET) / (BULK_LIMIT - BULK_TARGET)
END IF
</code></pre>
<p>This exists only as a trivial example to demonstrate a basic solution exists, and should not be intended as a concrete proposal.</p>
<h4 id="intra-leadin-price-decrease"><a class="header" href="#intra-leadin-price-decrease">Intra-Leadin Price-decrease</a></h4>
<p>During the Leadin Period of a sale, the effective price starts higher than the Sale Price and falls to end at the Sale Price at the end of the Leadin Period. The price can thus be defined as a simple factor above one on which the Sale Price is multiplied. A function which returns this factor would accept a factor between zero and one specifying the portion of the Leadin Period which has passed.</p>
<p>Thus we assume <code>SALE_PRICE</code>, then we can define <code>PRICE</code> as:</p>
<pre><code>PRICE := SALE_PRICE * FACTOR((NOW - LEADIN_BEGIN) / LEADIN_PERIOD)
</code></pre>
<p>We can define a very simple progression where the price decreases monotonically from double the Sale Price at the beginning of the Leadin Period.</p>
<pre><code>FACTOR(T) := 2 - T
</code></pre>
<h4 id="parameter-values"><a class="header" href="#parameter-values">Parameter Values</a></h4>
<p>Parameters are either <em>suggested</em> or <em>specified</em>. If <em>suggested</em>, it is non-binding and the proposal should not be judged on the value since other RFCs and/or the governance mechanism of Polkadot is expected to specify/maintain it. If <em>specified</em>, then the proposal should be judged on the merit of the value as-is.</p>
<div class="table-wrapper"><table><thead><tr><th>Name</th><th>Value</th><th></th></tr></thead><tbody>
<tr><td><code>BULK_PERIOD</code></td><td><code>28 * DAYS</code></td><td>specified</td></tr>
<tr><td><code>INTERLUDE_PERIOD</code></td><td><code>7 * DAYS</code></td><td>specified</td></tr>
<tr><td><code>LEADIN_PERIOD</code></td><td><code>7 * DAYS</code></td><td>specified</td></tr>
<tr><td><code>TIMESLICE</code></td><td><code>8 * MINUTES</code></td><td>specified</td></tr>
<tr><td><code>BULK_TARGET</code></td><td><code>30</code></td><td>suggested</td></tr>
<tr><td><code>BULK_LIMIT</code></td><td><code>45</code></td><td>suggested</td></tr>
<tr><td><code>RENEWAL_PRICE_CAP</code></td><td><code>Perbill::from_percent(2)</code></td><td>suggested</td></tr>
</tbody></table>
</div>
<h4 id="instantaneous-price-progression"><a class="header" href="#instantaneous-price-progression">Instantaneous Price Progression</a></h4>
<p>This proposal assumes the existence of a Relay-chain-based price-setting mechanism for the Instantaneous Coretime Market which alters from block to block, taking into account several parameters: the last price, the size of the Instantaneous Coretime Pool (in terms of cores per Relay-chain block) and the amount of Instantaneous Coretime waiting for processing (in terms of Core-blocks queued).</p>
<p>The ideal situation is to have the size of the Instantaneous Coretime Pool be equal to some factor of the Instantaneous Coretime waiting. This allows all Instantaneous Coretime sales to be processed with some limited latency while giving limited flexibility over ordering to the Relay-chain apparatus which is needed for efficient operation.</p>
<p>If we set a factor of three, and thus aim to retain a queue of Instantaneous Coretime Sales which can be processed within three Relay-chain blocks, then we would increase the price if the queue goes above three times the amount of cores available, and decrease if it goes under.</p>
<p>Let us assume the values <code>OLD_PRICE</code>, <code>FACTOR</code>, <code>QUEUE_SIZE</code> and <code>POOL_SIZE</code>. A simple definition of the <code>NEW_PRICE</code> would be thus:</p>
<pre><code>NEW_PRICE := IF QUEUE_SIZE &lt; POOL_SIZE * FACTOR THEN
OLD_PRICE * 0.95
ELSE
OLD_PRICE / 0.95
END IF
</code></pre>
<p>This exists only as a trivial example to demonstrate a basic solution exists, and should not be intended as a concrete proposal.</p>
<h3 id="notes-on-types"><a class="header" href="#notes-on-types">Notes on Types</a></h3>
<p>This exists only as a short illustration of a potential technical implementation and should not be treated as anything more.</p>
<h4 id="regions-1"><a class="header" href="#regions-1">Regions</a></h4>
<p>This data schema achieves a number of goals:</p>
<ul>
<li>Coretime can be individually traded at a level of a single usage of a single core.</li>
<li>Coretime Regions, of arbitrary span and up to 1/80th interlacing can be exposed as NFTs and exchanged.</li>
<li>Any Coretime Region can be contributed to the Instantaneous Coretime Pool.</li>
<li>Unlimited number of individual Coretime contributors to the Instantaneous Coretime Pool. (Effectively limited only in number of cores and interlacing level; with current values this would allow 80,000 individual payees per timeslice).</li>
<li>All keys are self-describing.</li>
<li>Workload to communicate core (re-)assignments is well-bounded and low in weight.</li>
<li>All mandatory bookkeeping workload is well-bounded in weight.</li>
</ul>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>type Timeslice = u32; // 80 block amounts.
type CoreIndex = u16;
type CoreMask = [u8; 10]; // 80-bit bitmap.
// 128-bit (16 bytes)
struct RegionId {
begin: Timeslice,
core: CoreIndex,
mask: CoreMask,
}
// 296-bit (37 bytes)
struct RegionRecord {
end: Timeslice,
owner: AccountId,
}
map Regions = Map&lt;RegionId, RegionRecord&gt;;
// 40-bit (5 bytes). Could be 32-bit with a more specialised type.
enum CoreTask {
Off,
Assigned { target: TaskId },
InstaPool,
}
// 120-bit (15 bytes). Could be 14 bytes with a specialised 32-bit `CoreTask`.
struct ScheduleItem {
mask: CoreMask, // 80 bit
task: CoreTask, // 40 bit
}
/// The work we plan on having each core do at a particular time in the future.
type Workplan = Map&lt;(Timeslice, CoreIndex), BoundedVec&lt;ScheduleItem, 80&gt;&gt;;
/// The current workload of each core. This gets updated with workplan as timeslices pass.
type Workload = Map&lt;CoreIndex, BoundedVec&lt;ScheduleItem, 80&gt;&gt;;
enum Contributor {
System,
Private(AccountId),
}
struct ContributionRecord {
begin: Timeslice,
end: Timeslice,
core: CoreIndex,
mask: CoreMask,
payee: Contributor,
}
type InstaPoolContribution = Map&lt;ContributionRecord, ()&gt;;
type SignedTotalMaskBits = u32;
type InstaPoolIo = Map&lt;Timeslice, SignedTotalMaskBits&gt;;
type PoolSize = Value&lt;TotalMaskBits&gt;;
/// Counter for the total CoreMask which could be dedicated to a pool. `u32` so we don't ever get
/// an overflow.
type TotalMaskBits = u32;
struct InstaPoolHistoryRecord {
total_contributions: TotalMaskBits,
maybe_payout: Option&lt;Balance&gt;,
}
/// Total InstaPool rewards for each Timeslice and the number of core Mask which contributed.
type InstaPoolHistory = Map&lt;Timeslice, InstaPoolHistoryRecord&gt;;
<span class="boring">}</span></code></pre></pre>
<p><code>CoreMask</code> tracks unique &quot;parts&quot; of a single core. It is used with interlacing in order to give a unique identifier to each component of any possible interlacing configuration of a core, allowing for simple self-describing keys for all core ownership and allocation information. It also allows for each core's workload to be tracked and updated progressively, keeping ongoing compute costs well-bounded and low.</p>
<p>Regions are issued into the <code>Regions</code> map and can be transferred, partitioned and interlaced as the owner desires. Regions can only be tasked if they begin after the current scheduling deadline (if they have missed this, then the region can be auto-trimmed until it is).</p>
<p>Once tasked, they are removed from there and a record is placed in <code>Workplan</code>. In addition, if they are contributed to the Instantaneous Coretime Pool, then an entry is placing in <code>InstaPoolContribution</code> and <code>InstaPoolIo</code>.</p>
<p>Each timeslice, <code>InstaPoolIo</code> is used to update the current value of <code>PoolSize</code>. A new entry in <code>InstaPoolHistory</code> is inserted, with the <code>total_contributions</code> field of <code>InstaPoolHistoryRecord</code> being informed by the <code>PoolSize</code> value. Each core's has its <code>Workload</code> mutated according to its <code>Workplan</code> for the upcoming timeslice.</p>
<p>When Instantaneous Coretime Market Revenues are reported for a particular timeslice from the Relay-chain, this information gets placed in the <code>maybe_payout</code> field of the relevant record of <code>InstaPoolHistory</code>.</p>
<p>Payments can be requested made for any records in <code>InstaPoolContribution</code> whose <code>begin</code> is the key for a value in <code>InstaPoolHistory</code> whose <code>maybe_payout</code> is <code>Some</code>. In this case, the <code>total_contributions</code> is reduced by the <code>ContributionRecord</code>'s <code>mask</code> and a pro rata amount paid. The <code>ContributionRecord</code> is mutated by incrementing <code>begin</code>, or removed if <code>begin</code> becomes equal to <code>end</code>.</p>
<p>Example:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>// Simple example with a `u16` `CoreMask` and bulk sold in 100 timeslices.
Regions:
{ core: 0u16, begin: 100, mask: 0b1111_1111_1111_1111u16 } =&gt; { end: 200u32, owner: Alice };
// First split @ 50
Regions:
{ core: 0u16, begin: 100, mask: 0b1111_1111_1111_1111u16 } =&gt; { end: 150u32, owner: Alice };
{ core: 0u16, begin: 150, mask: 0b1111_1111_1111_1111u16 } =&gt; { end: 200u32, owner: Alice };
// Share half of first 50 blocks
Regions:
{ core: 0u16, begin: 100, mask: 0b1111_1111_0000_0000u16 } =&gt; { end: 150u32, owner: Alice };
{ core: 0u16, begin: 100, mask: 0b0000_0000_1111_1111u16 } =&gt; { end: 150u32, owner: Alice };
{ core: 0u16, begin: 150, mask: 0b1111_1111_1111_1111u16 } =&gt; { end: 200u32, owner: Alice };
// Sell half of them to Bob
Regions:
{ core: 0u16, begin: 100, mask: 0b1111_1111_0000_0000u16 } =&gt; { end: 150u32, owner: Alice };
{ core: 0u16, begin: 100, mask: 0b0000_0000_1111_1111u16 } =&gt; { end: 150u32, owner: Bob };
{ core: 0u16, begin: 150, mask: 0b1111_1111_1111_1111u16 } =&gt; { end: 200u32, owner: Alice };
// Bob splits first 10 and assigns them to himself.
Regions:
{ core: 0u16, begin: 100, mask: 0b1111_1111_0000_0000u16 } =&gt; { end: 150u32, owner: Alice };
{ core: 0u16, begin: 100, mask: 0b0000_0000_1111_1111u16 } =&gt; { end: 110u32, owner: Bob };
{ core: 0u16, begin: 110, mask: 0b0000_0000_1111_1111u16 } =&gt; { end: 150u32, owner: Bob };
{ core: 0u16, begin: 150, mask: 0b1111_1111_1111_1111u16 } =&gt; { end: 200u32, owner: Alice };
// Bob shares first 10 3 ways and sells smaller shares to Charlie and Dave
Regions:
{ core: 0u16, begin: 100, mask: 0b1111_1111_0000_0000u16 } =&gt; { end: 150u32, owner: Alice };
{ core: 0u16, begin: 100, mask: 0b0000_0000_1100_0000u16 } =&gt; { end: 110u32, owner: Charlie };
{ core: 0u16, begin: 100, mask: 0b0000_0000_0011_0000u16 } =&gt; { end: 110u32, owner: Dave };
{ core: 0u16, begin: 100, mask: 0b0000_0000_0000_1111u16 } =&gt; { end: 110u32, owner: Bob };
{ core: 0u16, begin: 110, mask: 0b0000_0000_1111_1111u16 } =&gt; { end: 150u32, owner: Bob };
{ core: 0u16, begin: 150, mask: 0b1111_1111_1111_1111u16 } =&gt; { end: 200u32, owner: Alice };
// Bob assigns to his para B, Charlie and Dave assign to their paras C and D; Alice assigns first 50 to A
Regions:
{ core: 0u16, begin: 150, mask: 0b1111_1111_1111_1111u16 } =&gt; { end: 200u32, owner: Alice };
Workplan:
(100, 0) =&gt; vec![
{ mask: 0b1111_1111_0000_0000u16, task: Assigned(A) },
{ mask: 0b0000_0000_1100_0000u16, task: Assigned(C) },
{ mask: 0b0000_0000_0011_0000u16, task: Assigned(D) },
{ mask: 0b0000_0000_0000_1111u16, task: Assigned(B) },
]
(110, 0) =&gt; vec![{ mask: 0b0000_0000_1111_1111u16, task: Assigned(B) }]
// Alice assigns her remaining 50 timeslices to the InstaPool paying herself:
Regions: (empty)
Workplan:
(100, 0) =&gt; vec![
{ mask: 0b1111_1111_0000_0000u16, task: Assigned(A) },
{ mask: 0b0000_0000_1100_0000u16, task: Assigned(C) },
{ mask: 0b0000_0000_0011_0000u16, task: Assigned(D) },
{ mask: 0b0000_0000_0000_1111u16, task: Assigned(B) },
]
(110, 0) =&gt; vec![{ mask: 0b0000_0000_1111_1111u16, task: Assigned(B) }]
(150, 0) =&gt; vec![{ mask: 0b1111_1111_1111_1111u16, task: InstaPool }]
InstaPoolContribution:
{ begin: 150, end: 200, core: 0, mask: 0b1111_1111_1111_1111u16, payee: Alice }
InstaPoolIo:
150 =&gt; 16
200 =&gt; -16
// Actual notifications to relay chain.
// Assumes:
// - Timeslice is 10 blocks.
// - Timeslice 0 begins at block #1000.
// - Relay needs 10 blocks notice of change.
//
Workload: 0 =&gt; vec![]
PoolSize: 0
// Block 990:
Relay &lt;= assign_core(core: 0u16, begin: 1000, assignment: vec![(A, 8), (C, 2), (D, 2), (B, 4)])
Workload: 0 =&gt; vec![
{ mask: 0b1111_1111_0000_0000u16, task: Assigned(A) },
{ mask: 0b0000_0000_1100_0000u16, task: Assigned(C) },
{ mask: 0b0000_0000_0011_0000u16, task: Assigned(D) },
{ mask: 0b0000_0000_0000_1111u16, task: Assigned(B) },
]
PoolSize: 0
// Block 1090:
Relay &lt;= assign_core(core: 0u16, begin: 1100, assignment: vec![(A, 8), (B, 8)])
Workload: 0 =&gt; vec![
{ mask: 0b1111_1111_0000_0000u16, task: Assigned(A) },
{ mask: 0b0000_0000_1111_1111u16, task: Assigned(B) },
]
PoolSize: 0
// Block 1490:
Relay &lt;= assign_core(core: 0u16, begin: 1500, assignment: vec![(Pool, 16)])
Workload: 0 =&gt; vec![
{ mask: 0b1111_1111_1111_1111u16, task: InstaPool },
]
PoolSize: 16
InstaPoolIo:
200 =&gt; -16
InstaPoolHistory:
150 =&gt; { total_contributions: 16, maybe_payout: None }
// Sometime after block 1500:
InstaPoolHistory:
150 =&gt; { total_contributions: 16, maybe_payout: Some(P) }
// Sometime after block 1990:
InstaPoolIo: (empty)
PoolSize: 0
InstaPoolHistory:
150 =&gt; { total_contributions: 16, maybe_payout: Some(P0) }
151 =&gt; { total_contributions: 16, maybe_payout: Some(P1) }
152 =&gt; { total_contributions: 16, maybe_payout: Some(P2) }
...
199 =&gt; { total_contributions: 16, maybe_payout: Some(P49) }
// Sometime later still Alice calls for a payout
InstaPoolContribution: (empty)
InstaPoolHistory: (empty)
// Alice gets rewarded P0 + P1 + ... P49.
<span class="boring">}</span></code></pre></pre>
<h3 id="rollout"><a class="header" href="#rollout">Rollout</a></h3>
<p>Rollout of this proposal comes in several phases:</p>
<ol>
<li>Finalise the specifics of implementation; this may be done through a design document or through a well-documented prototype implementation.</li>
<li>Implement the design, including all associated aspects such as unit tests, benchmarks and any support software needed.</li>
<li>If any new parachain is required, launch of this.</li>
<li>Formal audit of the implementation and any manual testing.</li>
<li>Announcement to the various stakeholders of the imminent changes.</li>
<li>Software integration and release.</li>
<li>Governance upgrade proposal(s).</li>
<li>Monitoring of the upgrade process.</li>
</ol>
<h2 id="performance-ergonomics-and-compatibility-11"><a class="header" href="#performance-ergonomics-and-compatibility-11">Performance, Ergonomics and Compatibility</a></h2>
<p>No specific considerations.</p>
<p>Parachains already deployed into the Polkadot UC must have a clear plan of action to migrate to an agile Coretime market.</p>
<p>While this proposal does not introduce documentable features per se, adequate documentation must be provided to potential purchasers of Polkadot Coretime. This SHOULD include any alterations to the Polkadot-SDK software collection.</p>
<h2 id="testing-security-and-privacy-11"><a class="header" href="#testing-security-and-privacy-11">Testing, Security and Privacy</a></h2>
<p>Regular testing through unit tests, integration tests, manual testnet tests, zombie-net tests and fuzzing SHOULD be conducted.</p>
<p>A regular security review SHOULD be conducted prior to deployment through a review by the Web3 Foundation economic research group.</p>
<p>Any final implementation MUST pass a professional external security audit.</p>
<p>The proposal introduces no new privacy concerns.</p>
<h2 id="future-directions-and-related-material-8"><a class="header" href="#future-directions-and-related-material-8">Future Directions and Related Material</a></h2>
<p>RFC-3 proposes a means of implementing the high-level allocations within the Relay-chain.</p>
<p>RFC-5 proposes the API for interacting with Relay-chain.</p>
<p>Additional work should specify the interface for the instantaneous market revenue so that the Coretime-chain can ensure Bulk Coretime placed in the instantaneous market is properly compensated.</p>
<h2 id="drawbacks-alternatives-and-unknowns"><a class="header" href="#drawbacks-alternatives-and-unknowns">Drawbacks, Alternatives and Unknowns</a></h2>
<p>Unknowns include the economic and resource parameterisations:</p>
<ul>
<li>The initial price of Bulk Coretime.</li>
<li>The price-change algorithm between Bulk Coretime sales.</li>
<li>The price increase per Bulk Coretime period for renewals.</li>
<li>The price decrease graph in the Leadin period for Bulk Coretime sales.</li>
<li>The initial price of Instantaneous Coretime.</li>
<li>The price-change algorithm for Instantaneous Coretime sales.</li>
<li>The percentage of cores to be sold as Bulk Coretime.</li>
<li>The fate of revenue collected.</li>
</ul>
<h2 id="prior-art-and-references-8"><a class="header" href="#prior-art-and-references-8">Prior Art and References</a></h2>
<p>Robert Habermeier initially wrote on the subject of Polkadot blockspace-centric in the article <a href="https://www.rob.tech/polkadot-blockspace-over-blockchains/">Polkadot Blockspace over Blockchains</a>. While not going into details, the article served as an early reframing piece for moving beyond one-slot-per-chain models and building out secondary market infrastructure for resource allocation.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/blob/main/text/0005-coretime-interface.md">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="approved/0005-coretime-interface.html#rfc-5-coretime-interface">RFC-5: Coretime Interface</a>
<ul>
<li><a href="approved/0005-coretime-interface.html#summary">Summary</a></li>
<li><a href="approved/0005-coretime-interface.html#motivation">Motivation</a></li>
<li><a href="approved/0005-coretime-interface.html#requirements">Requirements</a></li>
<li><a href="approved/0005-coretime-interface.html#stakeholders">Stakeholders</a></li>
<li><a href="approved/0005-coretime-interface.html#explanation">Explanation</a>
<ul>
<li><a href="approved/0005-coretime-interface.html#ump-message-types">UMP Message Types</a></li>
<li><a href="approved/0005-coretime-interface.html#dmp-message-types">DMP Message Types</a></li>
<li><a href="approved/0005-coretime-interface.html#realistic-limits-of-the-usage">Realistic Limits of the Usage</a></li>
</ul>
</li>
<li><a href="approved/0005-coretime-interface.html#performance-ergonomics-and-compatibility">Performance, Ergonomics and Compatibility</a></li>
<li><a href="approved/0005-coretime-interface.html#testing-security-and-privacy">Testing, Security and Privacy</a></li>
<li><a href="approved/0005-coretime-interface.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
<li><a href="approved/0005-coretime-interface.html#drawbacks-alternatives-and-unknowns">Drawbacks, Alternatives and Unknowns</a></li>
<li><a href="approved/0005-coretime-interface.html#prior-art-and-references">Prior Art and References</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-5-coretime-interface"><a class="header" href="#rfc-5-coretime-interface">RFC-5: Coretime Interface</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>06 July 2023</td></tr>
<tr><td><strong>Description</strong></td><td>Interface for manipulating the usage of cores on the Polkadot Ubiquitous Computer.</td></tr>
<tr><td><strong>Authors</strong></td><td>Gavin Wood, Robert Habermeier</td></tr>
</tbody></table>
</div>
<h2 id="summary-12"><a class="header" href="#summary-12">Summary</a></h2>
<p>In the Agile Coretime model of the Polkadot Ubiquitous Computer, as proposed in RFC-1 and RFC-3, it is necessary for the allocating parachain (envisioned to be one or more pallets on a specialised Brokerage System Chain) to communicate the core assignments to the Relay-chain, which is responsible for ensuring those assignments are properly enacted.</p>
<p>This is a proposal for the interface which will exist around the Relay-chain in order to communicate this information and instructions.</p>
<h2 id="motivation-12"><a class="header" href="#motivation-12">Motivation</a></h2>
<p>The background motivation for this interface is splitting out coretime allocation functions and secondary markets from the Relay-chain onto System parachains. A well-understood and general interface is necessary for ensuring the Relay-chain receives coretime allocation instructions from one or more System chains without introducing dependencies on the implementation details of either side.</p>
<h2 id="requirements-4"><a class="header" href="#requirements-4">Requirements</a></h2>
<ul>
<li>The interface MUST allow the Relay-chain to be scheduled on a low-latency basis.</li>
<li>Individual cores MUST be schedulable, both in full to a single task (a ParaId or the Instantaneous Coretime Pool) or to many unique tasks in differing ratios.</li>
<li>Typical usage of the interface SHOULD NOT overload the VMP message system.</li>
<li>The interface MUST allow for the allocating chain to be notified of all accounting information relevant for making accurate rewards for contributing to the Instantaneous Coretime Pool.</li>
<li>The interface MUST allow for Instantaneous Coretime Market Credits to be communicated.</li>
<li>The interface MUST allow for the allocating chain to instruct changes to the number of cores which it is able to allocate.</li>
<li>The interface MUST allow for the allocating chain to be notified of changes to the number of cores which are able to be allocated by the allocating chain.</li>
</ul>
<h2 id="stakeholders-11"><a class="header" href="#stakeholders-11">Stakeholders</a></h2>
<p>Primary stakeholder sets are:</p>
<ul>
<li>Developers of the Relay-chain core-management logic.</li>
<li>Developers of the Brokerage System Chain and its pallets.</li>
</ul>
<p><em>Socialization:</em></p>
<p>This content of this RFC was discussed in the Polkdot Fellows channel.</p>
<h2 id="explanation-12"><a class="header" href="#explanation-12">Explanation</a></h2>
<p>The interface has two sections: The messages which the Relay-chain is able to receive from the allocating parachain (the <em>UMP message types</em>), and messages which the Relay-chain is able to send to the allocating parachain (the <em>DMP message types</em>). These messages are expected to be able to be implemented in a well-known pallet and called with the XCM <code>Transact</code> instruction.</p>
<p>Future work may include these messages being introduced into the XCM standard.</p>
<h3 id="ump-message-types"><a class="header" href="#ump-message-types">UMP Message Types</a></h3>
<h4 id="request_core_count"><a class="header" href="#request_core_count"><code>request_core_count</code></a></h4>
<p>Prototype:</p>
<pre><code>fn request_core_count(
count: u16,
)
</code></pre>
<p>Requests the Relay-chain to alter the number of schedulable cores to <code>count</code>. Under normal operation, the Relay-chain SHOULD send a <code>notify_core_count(count)</code> message back.</p>
<h4 id="request_revenue_info_at"><a class="header" href="#request_revenue_info_at"><code>request_revenue_info_at</code></a></h4>
<p>Prototype:</p>
<pre><code>fn request_revenue_at(
when: BlockNumber,
)
</code></pre>
<p>Requests that the Relay-chain send a <code>notify_revenue</code> message back at or soon after Relay-chain block number <code>when</code> whose <code>until</code> parameter is equal to <code>when</code>.</p>
<p>The period in to the past which <code>when</code> is allowed to be may be limited; if so the limit should be understood on a channel outside of this proposal. In the case that the request cannot be serviced because <code>when</code> is too old a block then a <code>notify_revenue</code> message must still be returned, but its <code>revenue</code> field may be <code>None</code>.</p>
<h4 id="credit_account"><a class="header" href="#credit_account"><code>credit_account</code></a></h4>
<p>Prototype:</p>
<pre><code>fn credit_account(
who: AccountId,
amount: Balance,
)
</code></pre>
<p>Instructs the Relay-chain to add the <code>amount</code> of DOT to the Instantaneous Coretime Market Credit account of <code>who</code>.</p>
<p>It is expected that Instantaneous Coretime Market Credit on the Relay-chain is NOT transferrable and only redeemable when used to assign cores in the Instantaneous Coretime Pool.</p>
<h4 id="assign_core"><a class="header" href="#assign_core"><code>assign_core</code></a></h4>
<p>Prototype:</p>
<pre><code>type PartsOf57600 = u16;
enum CoreAssignment {
InstantaneousPool,
Task(ParaId),
}
fn assign_core(
core: CoreIndex,
begin: BlockNumber,
assignment: Vec&lt;(CoreAssignment, PartsOf57600)&gt;,
end_hint: Option&lt;BlockNumber&gt;,
)
</code></pre>
<p>Requirements:</p>
<pre><code>assert!(core &lt; core_count);
assert!(targets.iter().map(|x| x.0).is_sorted());
assert_eq!(targets.iter().map(|x| x.0).unique().count(), targets.len());
assert_eq!(targets.iter().map(|x| x.1).sum(), 57600);
</code></pre>
<p>Where:</p>
<ul>
<li><code>core_count</code> is assumed to be the sole parameter in the last received <code>notify_core_count</code> message.</li>
</ul>
<p>Instructs the Relay-chain to ensure that the core indexed as <code>core</code> is utilised for a number of assignments in specific ratios given by <code>assignment</code> starting as soon after <code>begin</code> as possible. Core assignments take the form of a <code>CoreAssignment</code> value which can either task the core to a <code>ParaId</code> value or indicate that the core should be used in the Instantaneous Pool. Each assignment comes with a ratio value, represented as the numerator of the fraction with a denominator of 57,600.</p>
<p>If <code>end_hint</code> is <code>Some</code> and the inner is greater than the current block number, then the Relay-chain should optimize in the expectation of receiving a new <code>assign_core(core, ...)</code> message at or prior to the block number of the inner value. Specific functionality should remain unchanged regardless of the <code>end_hint</code> value.</p>
<p>On the choice of denominator: 57,600 is a very composite number which factors into: 2 ** 8, 3 ** 2, 5 ** 2. By using it as the denominator we allow for various useful fractions to be perfectly represented including thirds, quarters, fifths, tenths, 80ths, percent and 256ths.</p>
<h3 id="dmp-message-types"><a class="header" href="#dmp-message-types">DMP Message Types</a></h3>
<h4 id="notify_core_count"><a class="header" href="#notify_core_count"><code>notify_core_count</code></a></h4>
<p>Prototype:</p>
<pre><code>fn notify_core_count(
count: u16,
)
</code></pre>
<p>Indicate that from this block onwards, the range of acceptable values of the <code>core</code> parameter of <code>assign_core</code> message is <code>[0, count)</code>. <code>assign_core</code> will be a no-op if provided with a value for <code>core</code> outside of this range.</p>
<h4 id="notify_revenue_info"><a class="header" href="#notify_revenue_info"><code>notify_revenue_info</code></a></h4>
<p>Prototype:</p>
<pre><code>fn notify_revenue_info(
until: BlockNumber,
revenue: Option&lt;Balance&gt;,
)
</code></pre>
<p>Provide the amount of revenue accumulated from Instantaneous Coretime Sales from Relay-chain block number <code>last_until</code> to <code>until</code>, not including <code>until</code> itself. <code>last_until</code> is defined as being the <code>until</code> argument of the last <code>notify_revenue</code> message sent, or zero for the first call. If <code>revenue</code> is <code>None</code>, this indicates that the information is no longer available.</p>
<p>This explicitly disregards the possibility of multiple parachains requesting and being notified of revenue information. The Relay-chain must be configured to ensure that only a single revenue information destination exists.</p>
<h3 id="realistic-limits-of-the-usage"><a class="header" href="#realistic-limits-of-the-usage">Realistic Limits of the Usage</a></h3>
<p>For <code>request_revenue_info</code>, a successful request should be possible if <code>when</code> is no less than the Relay-chain block number on arrival of the message less 100,000.</p>
<p>For <code>assign_core</code>, a successful request should be possible if <code>begin</code> is no less than the Relay-chain block number on arrival of the message plus 10 and <code>workload</code> contains no more than 100 items.</p>
<h2 id="performance-ergonomics-and-compatibility-12"><a class="header" href="#performance-ergonomics-and-compatibility-12">Performance, Ergonomics and Compatibility</a></h2>
<p>No specific considerations.</p>
<h2 id="testing-security-and-privacy-12"><a class="header" href="#testing-security-and-privacy-12">Testing, Security and Privacy</a></h2>
<p>Standard Polkadot testing and security auditing applies.</p>
<p>The proposal introduces no new privacy concerns.</p>
<h2 id="future-directions-and-related-material-9"><a class="header" href="#future-directions-and-related-material-9">Future Directions and Related Material</a></h2>
<p>RFC-1 proposes a means of determining allocation of Coretime using this interface.</p>
<p>RFC-3 proposes a means of implementing the high-level allocations within the Relay-chain.</p>
<h2 id="drawbacks-alternatives-and-unknowns-1"><a class="header" href="#drawbacks-alternatives-and-unknowns-1">Drawbacks, Alternatives and Unknowns</a></h2>
<p>None at present.</p>
<h2 id="prior-art-and-references-9"><a class="header" href="#prior-art-and-references-9">Prior Art and References</a></h2>
<p>None.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/blob/main/text/0007-system-collator-selection.md">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="approved/0007-system-collator-selection.html#rfc-0007-system-collator-selection">RFC-0007: System Collator Selection</a>
<ul>
<li><a href="approved/0007-system-collator-selection.html#summary">Summary</a></li>
<li><a href="approved/0007-system-collator-selection.html#motivation">Motivation</a>
<ul>
<li><a href="approved/0007-system-collator-selection.html#requirements">Requirements</a></li>
</ul>
</li>
<li><a href="approved/0007-system-collator-selection.html#stakeholders">Stakeholders</a></li>
<li><a href="approved/0007-system-collator-selection.html#explanation">Explanation</a>
<ul>
<li><a href="approved/0007-system-collator-selection.html#set-size">Set Size</a></li>
</ul>
</li>
<li><a href="approved/0007-system-collator-selection.html#drawbacks">Drawbacks</a></li>
<li><a href="approved/0007-system-collator-selection.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="approved/0007-system-collator-selection.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="approved/0007-system-collator-selection.html#performance">Performance</a></li>
<li><a href="approved/0007-system-collator-selection.html#ergonomics">Ergonomics</a></li>
<li><a href="approved/0007-system-collator-selection.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="approved/0007-system-collator-selection.html#prior-art-and-references">Prior Art and References</a>
<ul>
<li><a href="approved/0007-system-collator-selection.html#written-discussions">Written Discussions</a></li>
<li><a href="approved/0007-system-collator-selection.html#prior-feedback-and-input-from">Prior Feedback and Input From</a></li>
</ul>
</li>
<li><a href="approved/0007-system-collator-selection.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="approved/0007-system-collator-selection.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0007-system-collator-selection"><a class="header" href="#rfc-0007-system-collator-selection">RFC-0007: System Collator Selection</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>07 July 2023</td></tr>
<tr><td><strong>Description</strong></td><td>Mechanism for selecting collators of system chains.</td></tr>
<tr><td><strong>Authors</strong></td><td>Joe Petrowski</td></tr>
</tbody></table>
</div>
<h2 id="summary-13"><a class="header" href="#summary-13">Summary</a></h2>
<p>As core functionality moves from the Relay Chain into system chains, so increases the reliance on
the liveness of these chains for the use of the network. It is not economically scalable, nor
necessary from a game-theoretic perspective, to pay collators large rewards. This RFC proposes a
mechanism -- part technical and part social -- for ensuring reliable collator sets that are
resilient to attemps to stop any subsytem of the Polkadot protocol.</p>
<h2 id="motivation-13"><a class="header" href="#motivation-13">Motivation</a></h2>
<p>In order to guarantee access to Polkadot's system, the collators on its system chains must propose
blocks (provide liveness) and allow all transactions to eventually be included. That is, some
collators may censor transactions, but there must exist one collator in the set who will include a
given transaction. In fact, all collators may censor varying subsets of transactions, but as long
as no transaction is in the intersection of every subset, it will eventually be included. The
objective of this RFC is to propose a mechanism to select such a set on each system chain.</p>
<p>While the network as a whole uses staking (and inflationary rewards) to attract validators,
collators face different challenges in scale and have lower security assumptions than validators.
Regarding scale, there exist many system chains, and it is economically expensive to pay collators
a premium. Likewise, any staked DOT for collation is <em>not</em> staked for validation. Since collator
sets do not need to meet Byzantine Fault Tolerance criteria, staking as the primary mechanism for
collator selection would remove stake that is securing BFT assumptions, making the network less
secure.</p>
<p>Another problem with economic scalability relates to the increasing number of system chains, and
corresponding increase in need for collators (i.e., increase in collator slots). &quot;Good&quot; (highly
available, non-censoring) collators will not want to compete in elections on many chains when they
could use their resources to compete in the more profitable validator election. Such dilution
decreases the required bond on each chain, leaving them vulnerable to takeover by hostile
collator groups.</p>
<p>This RFC proposes a system whereby collation is primarily an infrastructure service, with the
on-chain Treasury reimbursing costs of semi-trusted node operators, referred to as &quot;Invulnerables&quot;.
The system need not trust the individual operators, only that as a <em>set</em> they would be resilient to
coordinated attempts to stop a single chain from halting or to censor a particular subset of
transactions.</p>
<p>In the case that users do not trust this set, this RFC also proposes that each chain always have
available collator positions that can be acquired by anyone by placing a bond.</p>
<h3 id="requirements-5"><a class="header" href="#requirements-5">Requirements</a></h3>
<ul>
<li>System MUST have at least one valid collator for every chain.</li>
<li>System MUST allow anyone to become a collator, provided they <code>reserve</code>/<code>hold</code> enough DOT.</li>
<li>System SHOULD select a set of collators with reasonable expectation that the set will not collude
to censor any subset of transactions.</li>
<li>Collators selected by governance SHOULD have a reasonable expectation that the Treasury will
reimburse their operating costs.</li>
</ul>
<h2 id="stakeholders-12"><a class="header" href="#stakeholders-12">Stakeholders</a></h2>
<ul>
<li>Infrastructure providers (people who run validator/collator nodes)</li>
<li>Polkadot Treasury</li>
</ul>
<h2 id="explanation-13"><a class="header" href="#explanation-13">Explanation</a></h2>
<p>This protocol builds on the existing
<a href="https://github.com/paritytech/cumulus/tree/b15da70/pallets/collator-selection">Collator Selection pallet</a>
and its notion of Invulnerables. Invulnerables are collators (identified by their <code>AccountId</code>s) who
will be selected as part of the collator set every session. Operations relating to the management
of the Invulnerables are done through privileged, governance origins. The implementation should
maintain an API for adding and removing Invulnerable collators.</p>
<p>In addition to Invulnerables, there are also open slots for &quot;Candidates&quot;. Anyone can register as a
Candidate by placing a fixed bond. However, with a fixed bond and fixed number of slots, there is
an obvious selection problem: The slots fill up without any logic to replace their occupants.</p>
<p>This RFC proposes that the collator selection protocol allow Candidates to increase (and decrease)
their individual bonds, sort the Candidates according to bond, and select the top <code>N</code> Candidates.
The selection and changeover should be coordinated by the session manager.</p>
<p>A FRAME pallet already exists for sorting (&quot;bagging&quot;) &quot;top N&quot; groups, the
<a href="https://github.com/paritytech/substrate/blob/5032b8d/frame/bags-list/src/lib.rs">Bags List pallet</a>.
This pallet's <code>SortedListProvider</code> should be integrated into the session manager of the Collator
Selection pallet.</p>
<p>Despite the lack of apparent economic incentives (i.e., inflation), several reasons exist why one
may want to bond funds to participate in the Candidates election, for example:</p>
<ul>
<li>They want to build credibility to be selected as Invulnerable;</li>
<li>They want to ensure availability of an application, e.g. a stablecoin issuer might run a collator
on Asset Hub to ensure transactions in its asset are included in blocks;</li>
<li>They fear censorship themselves, e.g. a voter might think their votes are being censored from
governance, so they run a collator on the governance chain to include their votes.</li>
</ul>
<p>Unlike the fixed-bond mechanism that fills up its Candidates, the election mechanism ensures that
anyone can join the collator set by placing the <code>Nth</code> highest bond.</p>
<h3 id="set-size"><a class="header" href="#set-size">Set Size</a></h3>
<p>In order to achieve the requirements listed under <em>Motivation</em>, it is reasonable to have
approximately:</p>
<ul>
<li>20 collators per system chain,</li>
<li>of which 15 are Invulnerable, and</li>
<li>five are elected by bond.</li>
</ul>
<h2 id="drawbacks-10"><a class="header" href="#drawbacks-10">Drawbacks</a></h2>
<p>The primary drawback is a reliance on governance for continued treasury funding of infrastructure
costs for Invulnerable collators.</p>
<h2 id="testing-security-and-privacy-13"><a class="header" href="#testing-security-and-privacy-13">Testing, Security, and Privacy</a></h2>
<p>The vast majority of cases can be covered by unit testing. Integration test should ensure that the
Collator Selection <code>UpdateOrigin</code>, which has permission to modify the Invulnerables and desired
number of Candidates, can handle updates over XCM from the system's governance location.</p>
<h2 id="performance-ergonomics-and-compatibility-13"><a class="header" href="#performance-ergonomics-and-compatibility-13">Performance, Ergonomics, and Compatibility</a></h2>
<p>This proposal has very little impact on most users of Polkadot, and should improve the performance
of system chains by reducing the number of missed blocks.</p>
<h3 id="performance-12"><a class="header" href="#performance-12">Performance</a></h3>
<p>As chains have strict PoV size limits, care must be taken in the PoV impact of the session manager.
Appropriate benchmarking and tests should ensure that conservative limits are placed on the number
of Invulnerables and Candidates.</p>
<h3 id="ergonomics-10"><a class="header" href="#ergonomics-10">Ergonomics</a></h3>
<p>The primary group affected is Candidate collators, who, after implementation of this RFC, will need
to compete in a bond-based election rather than a race to claim a Candidate spot.</p>
<h3 id="compatibility-9"><a class="header" href="#compatibility-9">Compatibility</a></h3>
<p>This RFC is compatible with the existing implementation and can be handled via upgrades and
migration.</p>
<h2 id="prior-art-and-references-10"><a class="header" href="#prior-art-and-references-10">Prior Art and References</a></h2>
<h3 id="written-discussions"><a class="header" href="#written-discussions">Written Discussions</a></h3>
<ul>
<li><a href="https://github.com/paritytech/roadmap/issues/34">GitHub: Collator Selection Roadmap</a></li>
<li><a href="https://github.com/paritytech/cumulus/issues/1159">GitHub: Revisit Collator Selection Mechanism</a></li>
<li><a href="https://forum.polkadot.network/t/economic-model-for-system-para-collators/1010">Polkadot Forum: Economic Model for System Para Collators</a></li>
</ul>
<h3 id="prior-feedback-and-input-from"><a class="header" href="#prior-feedback-and-input-from">Prior Feedback and Input From</a></h3>
<ul>
<li>Kian Paimani</li>
<li>Jeff Burdges</li>
<li>Rob Habermeier</li>
<li>SR Labs Auditors</li>
<li>Current collators including Paranodes, Stake Plus, Turboflakes, Peter Mensik, SIK, and many more.</li>
</ul>
<h2 id="unresolved-questions-11"><a class="header" href="#unresolved-questions-11">Unresolved Questions</a></h2>
<p>None at this time.</p>
<h2 id="future-directions-and-related-material-10"><a class="header" href="#future-directions-and-related-material-10">Future Directions and Related Material</a></h2>
<p>There may exist in the future system chains for which this model of collator selection is not
appropriate. These chains should be evaluated on a case-by-case basis.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/blob/main/text/0008-parachain-bootnodes-dht.md">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="approved/0008-parachain-bootnodes-dht.html#rfc-0008-store-parachain-bootnodes-in-relay-chain-dht">RFC-0008: Store parachain bootnodes in relay chain DHT</a>
<ul>
<li><a href="approved/0008-parachain-bootnodes-dht.html#summary">Summary</a></li>
<li><a href="approved/0008-parachain-bootnodes-dht.html#motivation">Motivation</a></li>
<li><a href="approved/0008-parachain-bootnodes-dht.html#stakeholders">Stakeholders</a></li>
<li><a href="approved/0008-parachain-bootnodes-dht.html#explanation">Explanation</a>
<ul>
<li><a href="approved/0008-parachain-bootnodes-dht.html#dht-provider-registration">DHT provider registration</a></li>
<li><a href="approved/0008-parachain-bootnodes-dht.html#new-networking-protocol">New networking protocol</a></li>
</ul>
</li>
<li><a href="approved/0008-parachain-bootnodes-dht.html#drawbacks">Drawbacks</a></li>
<li><a href="approved/0008-parachain-bootnodes-dht.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="approved/0008-parachain-bootnodes-dht.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="approved/0008-parachain-bootnodes-dht.html#performance">Performance</a></li>
<li><a href="approved/0008-parachain-bootnodes-dht.html#ergonomics">Ergonomics</a></li>
<li><a href="approved/0008-parachain-bootnodes-dht.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="approved/0008-parachain-bootnodes-dht.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="approved/0008-parachain-bootnodes-dht.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="approved/0008-parachain-bootnodes-dht.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0008-store-parachain-bootnodes-in-relay-chain-dht"><a class="header" href="#rfc-0008-store-parachain-bootnodes-in-relay-chain-dht">RFC-0008: Store parachain bootnodes in relay chain DHT</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>2023-07-14</td></tr>
<tr><td><strong>Description</strong></td><td>Parachain bootnodes shall register themselves in the DHT of the relay chain</td></tr>
<tr><td><strong>Authors</strong></td><td>Pierre Krieger</td></tr>
</tbody></table>
</div>
<h2 id="summary-14"><a class="header" href="#summary-14">Summary</a></h2>
<p>The full nodes of the Polkadot peer-to-peer network maintain a distributed hash table (DHT), which is currently used for full nodes discovery and validators discovery purposes.</p>
<p>This RFC proposes to extend this DHT to be used to discover full nodes of the parachains of Polkadot.</p>
<h2 id="motivation-14"><a class="header" href="#motivation-14">Motivation</a></h2>
<p>The maintenance of bootnodes has long been an annoyance for everyone.</p>
<p>When a bootnode is newly-deployed or removed, every chain specification must be updated in order to take the update into account. This has lead to various non-optimal solutions, such as pulling chain specifications from GitHub repositories.
When it comes to RPC nodes, UX developers often have trouble finding up-to-date addresses of parachain RPC nodes. With the ongoing migration from RPC nodes to light clients, similar problems would happen with chain specifications as well.</p>
<p>Furthermore, there exists multiple different possible variants of a certain chain specification: with the non-raw storage, with the raw storage, with just the genesis trie root hash, with or without checkpoint, etc. All of this creates confusion. Removing the need for parachain developers to be aware of and manage these different versions would be beneficial.</p>
<p>Since the PeerId and addresses of bootnodes needs to be stable, extra maintenance work is required from the chain maintainers. For example, they need to be extra careful when migrating nodes within their infrastructure. In some situations, bootnodes are put behind domain names, which also requires maintenance work.</p>
<p>Because the list of bootnodes in chain specifications is so annoying to modify, the consequence is that the number of bootnodes is rather low (typically between 2 and 15). In order to better resist downtimes and DoS attacks, a better solution would be to use every node of a certain chain as potential bootnode, rather than special-casing some specific nodes.</p>
<p>While this RFC doesn't solve these problems for relay chains, it aims at solving it for parachains by storing the list of all the full nodes of a parachain on the relay chain DHT.</p>
<p>Assuming that this RFC is implemented, and that light clients are used, deploying a parachain wouldn't require more work than registering it onto the relay chain and starting the collators. There wouldn't be any need for special infrastructure nodes anymore.</p>
<h2 id="stakeholders-13"><a class="header" href="#stakeholders-13">Stakeholders</a></h2>
<p>This RFC has been opened on my own initiative because I think that this is a good technical solution to a usability problem that many people are encountering and that they don't realize can be solved.</p>
<h2 id="explanation-14"><a class="header" href="#explanation-14">Explanation</a></h2>
<p>The content of this RFC only applies for parachains and parachain nodes that are &quot;Substrate-compatible&quot;. It is in no way mandatory for parachains to comply to this RFC.</p>
<p>Note that &quot;Substrate-compatible&quot; is very loosely defined as &quot;implements the same mechanisms and networking protocols as Substrate&quot;. The author of this RFC believes that &quot;Substrate-compatible&quot; should be very precisely specified, but there is controversy on this topic.</p>
<p>While a lot of this RFC concerns the implementation of parachain nodes, it makes use of the resources of the Polkadot chain, and as such it is important to describe them in the Polkadot specification.</p>
<p>This RFC adds two mechanisms: a registration in the DHT, and a new networking protocol.</p>
<h3 id="dht-provider-registration"><a class="header" href="#dht-provider-registration">DHT provider registration</a></h3>
<p>This RFC heavily relies on the functionalities of the Kademlia DHT already in use by Polkadot.
You can find a link to the specification <a href="https://github.com/libp2p/specs/tree/master/kad-dht">here</a>.</p>
<p>Full nodes of a parachain registered on Polkadot should register themselves onto the Polkadot DHT as the providers of a key corresponding to the parachain that they are serving, as described in <a href="https://github.com/libp2p/specs/tree/master/kad-dht#content-provider-advertisement">the <code>Content provider advertisement</code> section</a> of the specification. This uses the <code>ADD_PROVIDER</code> system of libp2p-kademlia.</p>
<p>This key is: <code>sha256(concat(scale_compact(para_id), randomness))</code> where the value of <code>randomness</code> can be found in the <code>randomness</code> field when calling the <code>BabeApi_currentEpoch</code> function.
For example, for a <code>para_id</code> equal to 1000, and at the time of writing of this RFC (July 14th 2023 at 09:13 UTC), it is <code>sha(0xa10f12872447958d50aa7b937b0106561a588e0e2628d33f81b5361b13dbcf8df708)</code>, which is equal to <code>0x483dd8084d50dbbbc962067f216c37b627831d9339f5a6e426a32e3076313d87</code>.</p>
<p>In order to avoid downtime when the key changes, parachain full nodes should also register themselves as a secondary key that uses a value of <code>randomness</code> equal to the <code>randomness</code> field when calling <code>BabeApi_nextEpoch</code>.</p>
<p>Implementers should be aware that their implementation of Kademlia might already hash the key before XOR'ing it. The key is not meant to be hashed twice.</p>
<p>The compact SCALE encoding has been chosen in order to avoid problems related to the number of bytes and endianness of the <code>para_id</code>.</p>
<h3 id="new-networking-protocol"><a class="header" href="#new-networking-protocol">New networking protocol</a></h3>
<p>A new request-response protocol should be added, whose name is <code>/91b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c3/paranode</code> (that hexadecimal number is the genesis hash of the Polkadot chain, and should be adjusted appropriately for Kusama and others).</p>
<p>The request consists in a SCALE-compact-encoded <code>para_id</code>. For example, for a <code>para_id</code> equal to 1000, this is <code>0xa10f</code>.</p>
<p>Note that because this is a request-response protocol, the request is always prefixed with its length in bytes. While the body of the request is simply the SCALE-compact-encoded <code>para_id</code>, the data actually sent onto the substream is both the length and body.</p>
<p>The response consists in a protobuf struct, defined as:</p>
<pre><code>syntax = &quot;proto2&quot;;
message Response {
// Peer ID of the node on the parachain side.
bytes peer_id = 1;
// Multiaddresses of the parachain side of the node. The list and format are the same as for the `listenAddrs` field of the `identify` protocol.
repeated bytes addrs = 2;
// Genesis hash of the parachain. Used to determine the name of the networking protocol to connect to the parachain. Untrusted.
bytes genesis_hash = 3;
// So-called &quot;fork ID&quot; of the parachain. Used to determine the name of the networking protocol to connect to the parachain. Untrusted.
optional string fork_id = 4;
};
</code></pre>
<p>The maximum size of a response is set to an arbitrary 16kiB. The responding side should make sure to conform to this limit. Given that <code>fork_id</code> is typically very small and that the only variable-length field is <code>addrs</code>, this is easily achieved by limiting the number of addresses.</p>
<p>Implementers should be aware that <code>addrs</code> might be very large, and are encouraged to limit the number of <code>addrs</code> to an implementation-defined value.</p>
<h2 id="drawbacks-11"><a class="header" href="#drawbacks-11">Drawbacks</a></h2>
<p>The <code>peer_id</code> and <code>addrs</code> fields are in theory not strictly needed, as the PeerId and addresses could be always equal to the PeerId and addresses of the node being registered as the provider and serving the response. However, the Cumulus implementation currently uses two different networking stacks, one of the parachain and one for the relay chain, using two separate PeerIds and addresses, and as such the PeerId and addresses of the other networking stack must be indicated. Asking them to use only one networking stack wouldn't feasible in a realistic time frame.</p>
<p>The values of the <code>genesis_hash</code> and <code>fork_id</code> fields cannot be verified by the requester and are expected to be unused at the moment. Instead, a client that desires connecting to a parachain is expected to obtain the genesis hash and fork ID of the parachain from the parachain chain specification. These fields are included in the networking protocol nonetheless in case an acceptable solution is found in the future, and in order to allow use cases such as discovering parachains in a not-strictly-trusted way.</p>
<h2 id="testing-security-and-privacy-14"><a class="header" href="#testing-security-and-privacy-14">Testing, Security, and Privacy</a></h2>
<p>Because not all nodes want to be used as bootnodes, implementers are encouraged to provide a way to disable this mechanism. However, it is very much encouraged to leave this mechanism on by default for all parachain nodes.</p>
<p>This mechanism doesn't add or remove any security by itself, as it relies on existing mechanisms.
However, if the principle of chain specification bootnodes is entirely replaced with the mechanism described in this RFC (which is the objective), then it becomes important whether the mechanism in this RFC can be abused in order to make a parachain unreachable.</p>
<p>Due to the way Kademlia works, it would become the responsibility of the 20 Polkadot nodes whose <code>sha256(peer_id)</code> is closest to the <code>key</code> (described in the explanations section) to store the list of bootnodes of each parachain.
Furthermore, when a large number of providers (here, a provider is a bootnode) are registered, only the providers closest to the <code>key</code> are kept, up to a certain implementation-defined limit.</p>
<p>For this reason, an attacker can abuse this mechanism by randomly generating libp2p PeerIds until they find the 20 entries closest to the <code>key</code> representing the target parachain. They are then in control of the parachain bootnodes.
Because the key changes periodically and isn't predictable, and assuming that the Polkadot DHT is sufficiently large, it is not realistic for an attack like this to be maintained in the long term.</p>
<p>Furthermore, parachain clients are expected to cache a list of known good nodes on their disk. If the mechanism described in this RFC went down, it would only prevent new nodes from accessing the parachain, while clients that have connected before would not be affected.</p>
<h2 id="performance-ergonomics-and-compatibility-14"><a class="header" href="#performance-ergonomics-and-compatibility-14">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-13"><a class="header" href="#performance-13">Performance</a></h3>
<p>The DHT mechanism generally has a low overhead, especially given that publishing providers is done only every 24 hours.</p>
<p>Doing a Kademlia iterative query then sending a provider record shouldn't take more than around 50 kiB in total of bandwidth for the parachain bootnode.</p>
<p>Assuming 1000 parachain full nodes, the 20 Polkadot full nodes corresponding to a specific parachain will each receive a sudden spike of a few megabytes of networking traffic when the <code>key</code> rotates. Again, this is relatively negligible. If this becomes a problem, one can add a random delay before a parachain full node registers itself to be the provider of the <code>key</code> corresponding to <code>BabeApi_next_epoch</code>.</p>
<p>Maybe the biggest uncertainty is the traffic that the 20 Polkadot full nodes will receive from light clients that desire knowing the bootnodes of a parachain. Light clients are generally encouraged to cache the peers that they use between restarts, so they should only query these 20 Polkadot full nodes at their first initialization.
If this every becomes a problem, this value of 20 is an arbitrary constant that can be increased for more redundancy.</p>
<h3 id="ergonomics-11"><a class="header" href="#ergonomics-11">Ergonomics</a></h3>
<p>Irrelevant.</p>
<h3 id="compatibility-10"><a class="header" href="#compatibility-10">Compatibility</a></h3>
<p>Irrelevant.</p>
<h2 id="prior-art-and-references-11"><a class="header" href="#prior-art-and-references-11">Prior Art and References</a></h2>
<p>None.</p>
<h2 id="unresolved-questions-12"><a class="header" href="#unresolved-questions-12">Unresolved Questions</a></h2>
<p>While it fundamentally doesn't change much to this RFC, using <code>BabeApi_currentEpoch</code> and <code>BabeApi_nextEpoch</code> might be inappropriate. I'm not familiar enough with good practices within the runtime to have an opinion here. Should it be an entirely new pallet?</p>
<h2 id="future-directions-and-related-material-11"><a class="header" href="#future-directions-and-related-material-11">Future Directions and Related Material</a></h2>
<p>It is possible that in the future a client could connect to a parachain without having to rely on a trusted parachain specification.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/blob/main/text/0010-burn-coretime-revenue.md">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="approved/0010-burn-coretime-revenue.html#rfc-0010-burn-coretime-revenue">RFC-0010: Burn Coretime Revenue</a>
<ul>
<li><a href="approved/0010-burn-coretime-revenue.html#summary">Summary</a></li>
<li><a href="approved/0010-burn-coretime-revenue.html#motivation">Motivation</a></li>
<li><a href="approved/0010-burn-coretime-revenue.html#stakeholders">Stakeholders</a></li>
<li><a href="approved/0010-burn-coretime-revenue.html#explanation">Explanation</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0010-burn-coretime-revenue"><a class="header" href="#rfc-0010-burn-coretime-revenue">RFC-0010: Burn Coretime Revenue</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>19.07.2023</td></tr>
<tr><td><strong>Description</strong></td><td>Revenue from Coretime sales should be burned</td></tr>
<tr><td><strong>Authors</strong></td><td>Jonas Gehrlein</td></tr>
</tbody></table>
</div>
<h2 id="summary-15"><a class="header" href="#summary-15">Summary</a></h2>
<p>The Polkadot UC will generate revenue from the sale of available Coretime. The question then arises: how should we handle these revenues? Broadly, there are two reasonable paths – burning the revenue and thereby removing it from total issuance or divert it to the Treasury. This Request for Comment (RFC) presents arguments favoring burning as the preferred mechanism for handling revenues from Coretime sales.</p>
<h2 id="motivation-15"><a class="header" href="#motivation-15">Motivation</a></h2>
<p>How to handle the revenue accrued from Coretime sales is an important economic question that influences the value of DOT and should be properly discussed before deciding for either of the options. Now is the best time to start this discussion.</p>
<h2 id="stakeholders-14"><a class="header" href="#stakeholders-14">Stakeholders</a></h2>
<p>Polkadot DOT token holders.</p>
<h2 id="explanation-15"><a class="header" href="#explanation-15">Explanation</a></h2>
<p>This RFC discusses potential benefits of burning the revenue accrued from Coretime sales instead of diverting them to Treasury. Here are the following arguments for it.</p>
<p>It's in the interest of the Polkadot community to have a consistent and predictable Treasury income, because volatility in the inflow can be damaging, especially in situations when it is insufficient. As such, this RFC operates under the presumption of a steady and sustainable Treasury income flow, which is crucial for the Polkadot community's stability. The assurance of a predictable Treasury income, as outlined in a prior discussion <a href="https://forum.polkadot.network/t/adjusting-the-current-inflation-model-to-sustain-treasury-inflow/3301">here</a>, or through other equally effective measures, serves as a baseline assumption for this argument. </p>
<p>Consequently, we need not concern ourselves with this particular issue here. This naturally begs the question - why should we introduce additional volatility to the Treasury by aligning it with the variable Coretime sales? It's worth noting that Coretime revenues often exhibit an inverse relationship with periods when Treasury spending should ideally be ramped up. During periods of low Coretime utilization (indicated by lower revenue), Treasury should spend more on projects and endeavours to increase the demand for Coretime. This pattern underscores that Coretime sales, by their very nature, are an inconsistent and unpredictable source of funding for the Treasury. Given the importance of maintaining a steady and predictable inflow, it's unnecessary to rely on another volatile mechanism. Some might argue that we could have both: a steady inflow (from inflation) and some added bonus from Coretime sales, but burning the revenue would offer further benefits as described below.</p>
<ul>
<li>
<p><strong>Balancing Inflation:</strong> While DOT as a utility token inherently profits from a (reasonable) net inflation, it also benefits from a deflationary force that functions as a counterbalance to the overall inflation. Right now, the only mechanism on Polkadot that burns fees is the one for underutilized DOT in the Treasury. Finding other, more direct target for burns makes sense and the Coretime market is a good option.</p>
</li>
<li>
<p><strong>Clear incentives:</strong> By burning the revenue accrued on Coretime sales, prices paid by buyers are clearly costs. This removes distortion from the market that might arise when the paid tokens occur on some other places within the network. In that case, some actors might have secondary motives of influencing the price of Coretime sales, because they benefit down the line. For example, actors that actively participate in the Coretime sales are likely to also benefit from a higher Treasury balance, because they might frequently request funds for their projects. While those effects might appear far-fetched, they could accumulate. Burning the revenues makes sure that the prices paid are clearly costs to the actors themselves.</p>
</li>
<li>
<p><strong>Collective Value Accrual:</strong> Following the previous argument, burning the revenue also generates some externality, because it reduces the overall issuance of DOT and thereby increases the value of each remaining token. In contrast to the aforementioned argument, this benefits all token holders collectively and equally. Therefore, I'd consider this as the preferrable option, because burns lets all token holders participate at Polkadot's success as Coretime usage increases.</p>
</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/blob/main/text/0012-process-for-adding-new-collectives.md">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="approved/0012-process-for-adding-new-collectives.html#rfc-0012-process-for-adding-new-system-collectives">RFC-0012: Process for Adding New System Collectives</a>
<ul>
<li><a href="approved/0012-process-for-adding-new-collectives.html#summary">Summary</a></li>
<li><a href="approved/0012-process-for-adding-new-collectives.html#motivation">Motivation</a></li>
<li><a href="approved/0012-process-for-adding-new-collectives.html#stakeholders">Stakeholders</a></li>
<li><a href="approved/0012-process-for-adding-new-collectives.html#explanation">Explanation</a>
<ul>
<li><a href="approved/0012-process-for-adding-new-collectives.html#removing-collectives">Removing Collectives</a></li>
</ul>
</li>
<li><a href="approved/0012-process-for-adding-new-collectives.html#drawbacks">Drawbacks</a></li>
<li><a href="approved/0012-process-for-adding-new-collectives.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="approved/0012-process-for-adding-new-collectives.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a></li>
<li><a href="approved/0012-process-for-adding-new-collectives.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="approved/0012-process-for-adding-new-collectives.html#unresolved-questions">Unresolved Questions</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0012-process-for-adding-new-system-collectives"><a class="header" href="#rfc-0012-process-for-adding-new-system-collectives">RFC-0012: Process for Adding New System Collectives</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>24 July 2023</td></tr>
<tr><td><strong>Description</strong></td><td>A process for adding new (and removing existing) system collectives.</td></tr>
<tr><td><strong>Authors</strong></td><td>Joe Petrowski</td></tr>
</tbody></table>
</div>
<h2 id="summary-16"><a class="header" href="#summary-16">Summary</a></h2>
<p>Since the introduction of the Collectives parachain, many groups have expressed interest in forming
new -- or migrating existing groups into -- on-chain collectives. While adding a new collective is
relatively simple from a technical standpoint, the Fellowship will need to merge new pallets into
the Collectives parachain for each new collective. This RFC proposes a means for the network to
ratify a new collective, thus instructing the Fellowship to instate it in the runtime.</p>
<h2 id="motivation-16"><a class="header" href="#motivation-16">Motivation</a></h2>
<p>Many groups have expressed interest in representing collectives on-chain. Some of these include:</p>
<ul>
<li>Parachain technical fellowship (new)</li>
<li>Fellowship(s) for media, education, and evangelism (new)</li>
<li>Polkadot Ambassador Program (existing)</li>
<li>Anti-Scam Team (existing)</li>
</ul>
<p>Collectives that form part of the core Polkadot protocol should have a mandate to serve the
Polkadot network. However, as part of the Polkadot protocol, the Fellowship, in its capacity of
maintaining system runtimes, will need to include modules and configurations for each collective.</p>
<p>Once a group has developed a value proposition for the Polkadot network, it should have a clear
path to having its collective accepted on-chain as part of the protocol. Acceptance should direct
the Fellowship to include the new collective with a given initial configuration into the runtime.
However, the network, not the Fellowship, should ultimately decide which collectives are in the
interest of the network.</p>
<h2 id="stakeholders-15"><a class="header" href="#stakeholders-15">Stakeholders</a></h2>
<ul>
<li>Polkadot stakeholders who would like to organize on-chain.</li>
<li>Technical Fellowship, in its role of maintaining system runtimes.</li>
</ul>
<h2 id="explanation-16"><a class="header" href="#explanation-16">Explanation</a></h2>
<p>The group that wishes to operate an on-chain collective should publish the following information:</p>
<ul>
<li>Charter, including the collective's mandate and how it benefits Polkadot. This would be similar
to the
<a href="https://github.com/polkadot-fellows/manifesto/blob/0c3df46/manifesto.pdf">Fellowship Manifesto</a>.</li>
<li>Seeding recommendation.</li>
<li>Member types, i.e. should members be individuals or organizations.</li>
<li>Member management strategy, i.e. how do members join and get promoted, if applicable.</li>
<li>How much, if at all, members should get paid in salary.</li>
<li>Any special origins this Collective should have outside its self. For example, the Fellowship
can whitelist calls for referenda via the <code>WhitelistOrigin</code>.</li>
</ul>
<p>This information could all be in a single document or, for example, a GitHub repository.</p>
<p>After publication, members should seek feedback from the community and Technical Fellowship, and
make any revisions needed. When the collective believes the proposal is ready, they should bring a
remark with the text <code>APPROVE_COLLECTIVE(&quot;{collective name}, {commitment}&quot;)</code> to a Root origin
referendum. The proposer should provide instructions for generating <code>commitment</code>. The passing of
this referendum would be unequivocal direction to the Fellowship that this collective should be
part of the Polkadot runtime.</p>
<p>Note: There is no need for a <code>REJECT</code> referendum. Proposals that have not been approved are simply
not included in the runtime.</p>
<h3 id="removing-collectives"><a class="header" href="#removing-collectives">Removing Collectives</a></h3>
<p>If someone believes that an existing collective is not acting in the interest of the network or in
accordance with its charter, they should likewise have a means to instruct the Fellowship to
<em>remove</em> that collective from Polkadot.</p>
<p>An on-chain remark from the Root origin with the text
<code>REMOVE_COLLECTIVE(&quot;{collective name}, {para ID}, [{pallet indices}]&quot;)</code> would instruct the
Fellowship to remove the collective via the listed pallet indices on <code>paraId</code>. Should someone want
to construct such a remark, they should have a reasonable expectation that a member of the
Fellowship would help them identify the pallet indices associated with a given collective, whether
or not the Fellowship member agrees with removal.</p>
<p>Collective removal may also come with other governance calls, for example voiding any scheduled
Treasury spends that would fund the given collective.</p>
<h2 id="drawbacks-12"><a class="header" href="#drawbacks-12">Drawbacks</a></h2>
<p>Passing a Root origin referendum is slow. However, given the network's investment (in terms of code
maintenance and salaries) in a new collective, this is an appropriate step.</p>
<h2 id="testing-security-and-privacy-15"><a class="header" href="#testing-security-and-privacy-15">Testing, Security, and Privacy</a></h2>
<p>No impacts.</p>
<h2 id="performance-ergonomics-and-compatibility-15"><a class="header" href="#performance-ergonomics-and-compatibility-15">Performance, Ergonomics, and Compatibility</a></h2>
<p>Generally all new collectives will be in the Collectives parachain. Thus, performance impacts
should strictly be limited to this parachain and not affect others. As the majority of logic for
collectives is generalized and reusable, we expect most collectives to be instances of similar
subsets of modules. That is, new collectives should generally be compatible with UIs and other
services that provide collective-related functionality, with little modifications to support new
ones.</p>
<h2 id="prior-art-and-references-12"><a class="header" href="#prior-art-and-references-12">Prior Art and References</a></h2>
<p>The launch of the Technical Fellowship, see the
<a href="https://forum.polkadot.network/t/calling-polkadot-core-developers/506">initial forum post</a>.</p>
<h2 id="unresolved-questions-13"><a class="header" href="#unresolved-questions-13">Unresolved Questions</a></h2>
<p>None at this time.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/blob/main/text/0014-improve-locking-mechanism-for-parachains.md">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="approved/0014-improve-locking-mechanism-for-parachains.html#rfc-0014-improve-locking-mechanism-for-parachains">RFC-0014: Improve locking mechanism for parachains</a>
<ul>
<li><a href="approved/0014-improve-locking-mechanism-for-parachains.html#summary">Summary</a></li>
<li><a href="approved/0014-improve-locking-mechanism-for-parachains.html#motivation">Motivation</a></li>
<li><a href="approved/0014-improve-locking-mechanism-for-parachains.html#requirements">Requirements</a></li>
<li><a href="approved/0014-improve-locking-mechanism-for-parachains.html#stakeholders">Stakeholders</a></li>
<li><a href="approved/0014-improve-locking-mechanism-for-parachains.html#explanation">Explanation</a>
<ul>
<li><a href="approved/0014-improve-locking-mechanism-for-parachains.html#status-quo">Status quo</a></li>
<li><a href="approved/0014-improve-locking-mechanism-for-parachains.html#proposed-changes">Proposed changes</a></li>
<li><a href="approved/0014-improve-locking-mechanism-for-parachains.html#migration">Migration</a></li>
</ul>
</li>
<li><a href="approved/0014-improve-locking-mechanism-for-parachains.html#drawbacks">Drawbacks</a></li>
<li><a href="approved/0014-improve-locking-mechanism-for-parachains.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="approved/0014-improve-locking-mechanism-for-parachains.html#performance">Performance</a></li>
<li><a href="approved/0014-improve-locking-mechanism-for-parachains.html#ergonomics">Ergonomics</a></li>
<li><a href="approved/0014-improve-locking-mechanism-for-parachains.html#compatibility">Compatibility</a></li>
<li><a href="approved/0014-improve-locking-mechanism-for-parachains.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="approved/0014-improve-locking-mechanism-for-parachains.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="approved/0014-improve-locking-mechanism-for-parachains.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0014-improve-locking-mechanism-for-parachains"><a class="header" href="#rfc-0014-improve-locking-mechanism-for-parachains">RFC-0014: Improve locking mechanism for parachains</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>July 25, 2023</td></tr>
<tr><td><strong>Description</strong></td><td>Improve locking mechanism for parachains</td></tr>
<tr><td><strong>Authors</strong></td><td>Bryan Chen</td></tr>
</tbody></table>
</div>
<h2 id="summary-17"><a class="header" href="#summary-17">Summary</a></h2>
<p>This RFC proposes a set of changes to the parachain lock mechanism. The goal is to allow a parachain manager to self-service the parachain without root track governance action.</p>
<p>This is achieved by remove existing lock conditions and only lock a parachain when:</p>
<ul>
<li>A parachain manager explicitly lock the parachain</li>
<li>OR a parachain block is produced successfully</li>
</ul>
<h2 id="motivation-17"><a class="header" href="#motivation-17">Motivation</a></h2>
<p>The manager of a parachain has permission to manage the parachain when the parachain is unlocked. Parachains are by default locked when onboarded to a slot. This requires the parachain wasm/genesis must be valid, otherwise a root track governance action on relaychain is required to update the parachain.</p>
<p>The current reliance on root track governance actions for managing parachains can be time-consuming and burdensome. This RFC aims to address this technical difficulty by allowing parachain managers to take self-service actions, rather than relying on general public voting.</p>
<p>The key scenarios this RFC seeks to improve are:</p>
<ol>
<li>Rescue a parachain with invalid wasm/genesis.</li>
</ol>
<p>While we have various resources and templates to build a new parachain, it is still not a trivial task. It is very easy to make a mistake and resulting an invalid wasm/genesis. With lack of tools to help detect those issues<sup class="footnote-reference"><a href="#1">1</a></sup>, it is very likely that the issues are only discovered after the parachain is onboarded on a slot. In this case, the parachain is locked and the parachain team has to go through a lengthy governance process to rescue the parachain.</p>
<ol start="2">
<li>Perform lease renewal for an existing parachain.</li>
</ol>
<p>One way to perform lease renewal for a parachain is by doing a least swap with another parachain with a longer lease. This requires the other parachain must be operational and able to perform XCM transact call into relaychain to dispatch the swap call. Combined with the overhead of setting up a new parachain, this is an time consuming and expensive process. Ideally, the parachain manager should be able to perform the lease swap call without having a running parachain<sup class="footnote-reference"><a href="#2">2</a></sup>.</p>
<h2 id="requirements-6"><a class="header" href="#requirements-6">Requirements</a></h2>
<ul>
<li>A parachain manager SHOULD be able to rescue a parachain by updating the wasm/genesis without root track governance action.</li>
<li>A parachain manager MUST NOT be able to update the wasm/genesis if the parachain is locked.</li>
<li>A parachain SHOULD be locked when it successfully produced the first block.</li>
<li>A parachain manager MUST be able to perform lease swap without having a running parachain.</li>
</ul>
<h2 id="stakeholders-16"><a class="header" href="#stakeholders-16">Stakeholders</a></h2>
<ul>
<li>Parachain teams</li>
<li>Parachain users</li>
</ul>
<h2 id="explanation-17"><a class="header" href="#explanation-17">Explanation</a></h2>
<h3 id="status-quo"><a class="header" href="#status-quo">Status quo</a></h3>
<p>A parachain can either be locked or unlocked<sup class="footnote-reference"><a href="#3">3</a></sup>. With parachain locked, the parachain manager does not have any privileges. With parachain unlocked, the parachain manager can perform following actions with the <code>paras_registrar</code> pallet:</p>
<ul>
<li><code>deregister</code>: Deregister a Para Id, freeing all data and returning any deposit.</li>
<li><code>swap</code>: Initiate or confirm lease swap with another parachain.</li>
<li><code>add_lock</code>: Lock the parachain.</li>
<li><code>schedule_code_upgrade</code>: Schedule a parachain upgrade to update parachain wasm.</li>
<li><code>set_current_head</code>: Set the parachain's current head.</li>
</ul>
<p>Currently, a parachain can be locked with following conditions:</p>
<ul>
<li>From <code>add_lock</code> call, which can be dispatched by relaychain Root origin, the parachain, or the parachain manager.</li>
<li>When a parachain is onboarded on a slot<sup class="footnote-reference"><a href="#4">4</a></sup>.</li>
<li>When a crowdloan is created.</li>
</ul>
<p>Only the relaychain Root origin or the parachain itself can unlock the lock<sup class="footnote-reference"><a href="#5">5</a></sup>.</p>
<p>This creates an issue that if the parachain is unable to produce block, the parachain manager is unable to do anything and have to rely on relaychain Root origin to manage the parachain.</p>
<h3 id="proposed-changes"><a class="header" href="#proposed-changes">Proposed changes</a></h3>
<p>This RFC proposes to change the lock and unlock conditions.</p>
<p>A parachain can be locked only with following conditions:</p>
<ul>
<li>Relaychain governance MUST be able to lock any parachain.</li>
<li>A parachain MUST be able to lock its own lock.</li>
<li>A parachain manager SHOULD be able to lock the parachain.</li>
<li>A parachain SHOULD be locked when it successfully produced a block for the first time.</li>
</ul>
<p>A parachain can be unlocked only with following conditions:</p>
<ul>
<li>Relaychain governance MUST be able to unlock any parachain.</li>
<li>A parachain MUST be able to unlock its own lock.</li>
</ul>
<p>Note that create crowdloan MUST NOT lock the parachain and onboard a parachain SHOULD NOT lock it until a new block is successfully produced.</p>
<h3 id="migration"><a class="header" href="#migration">Migration</a></h3>
<p>A one off migration is proposed in order to apply this change retrospectively so that existing parachains can also be benefited from this RFC. This migration will unlock parachains that confirms with following conditions:</p>
<ul>
<li>Parachain is locked.</li>
<li>Parachain never produced a block. Including from expired leases.</li>
<li>Parachain manager never explicitly lock the parachain.</li>
</ul>
<h2 id="drawbacks-13"><a class="header" href="#drawbacks-13">Drawbacks</a></h2>
<p>Parachain locks are designed in such way to ensure the decentralization of parachains. If parachains are not locked when it should be, it could introduce centralization risk for new parachains.</p>
<p>For example, one possible scenario is that a collective may decide to launch a parachain fully decentralized. However, if the parachain is unable to produce block, the parachain manager will be able to replace the wasm and genesis without the consent of the collective.</p>
<p>It is considered this risk is tolerable as it requires the wasm/genesis to be invalid at first place. It is not yet practically possible to develop a parachain without any centralized risk currently.</p>
<p>Another case is that a parachain team may decide to use crowdloan to help secure a slot lease. Previously, creating a crowdloan will lock a parachain. This means crowdloan participants will know exactly the genesis of the parachain for the crowdloan they are participating. However, this actually providers little assurance to crowdloan participants. For example, if the genesis block is determined before a crowdloan is started, it is not possible to have onchain mechanism to enforce reward distributions for crowdloan participants. They always have to rely on the parachain team to fulfill the promise after the parachain is alive.</p>
<p>Existing operational parachains will not be impacted.</p>
<h2 id="testing-security-and-privacy-16"><a class="header" href="#testing-security-and-privacy-16">Testing, Security, and Privacy</a></h2>
<p>The implementation of this RFC will be tested on testnets (Rococo and Westend) first.</p>
<p>An audit maybe required to ensure the implementation does not introduce unwanted side effects.</p>
<p>There is no privacy related concerns.</p>
<h2 id="performance-14"><a class="header" href="#performance-14">Performance</a></h2>
<p>This RFC should not introduce any performance impact.</p>
<h2 id="ergonomics-12"><a class="header" href="#ergonomics-12">Ergonomics</a></h2>
<p>This RFC should improve the developer experiences for new and existing parachain teams</p>
<h2 id="compatibility-11"><a class="header" href="#compatibility-11">Compatibility</a></h2>
<p>This RFC is fully compatibility with existing interfaces.</p>
<h2 id="prior-art-and-references-13"><a class="header" href="#prior-art-and-references-13">Prior Art and References</a></h2>
<ul>
<li>Parachain Slot Extension Story: https://github.com/paritytech/polkadot/issues/4758</li>
<li>Allow parachain to renew lease without actually run another parachain: https://github.com/paritytech/polkadot/issues/6685</li>
<li>Always treat parachain that never produced block for a significant amount of time as unlocked: https://github.com/paritytech/polkadot/issues/7539</li>
</ul>
<h2 id="unresolved-questions-14"><a class="header" href="#unresolved-questions-14">Unresolved Questions</a></h2>
<p>None at this stage.</p>
<h2 id="future-directions-and-related-material-12"><a class="header" href="#future-directions-and-related-material-12">Future Directions and Related Material</a></h2>
<p>This RFC is only intended to be a short term solution. Slots will be removed in future and lock mechanism is likely going to be replaced with a more generalized parachain manage &amp; recovery system in future. Therefore long term impacts of this RFC are not considered.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>https://github.com/paritytech/cumulus/issues/377
<sup class="footnote-reference"><a href="#2">2</a></sup>: https://github.com/paritytech/polkadot/issues/6685
<sup class="footnote-reference"><a href="#3">3</a></sup>: https://github.com/paritytech/polkadot/blob/994af3de79af25544bf39644844cbe70a7b4d695/runtime/common/src/paras_registrar.rs#L51-L52C15
<sup class="footnote-reference"><a href="#4">4</a></sup>: https://github.com/paritytech/polkadot/blob/994af3de79af25544bf39644844cbe70a7b4d695/runtime/common/src/paras_registrar.rs#L473-L475
<sup class="footnote-reference"><a href="#5">5</a></sup>: https://github.com/paritytech/polkadot/blob/994af3de79af25544bf39644844cbe70a7b4d695/runtime/common/src/paras_registrar.rs#L333-L340</p>
</div>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/blob/main/text/0022-adopt-encointer-runtime.md">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="approved/0022-adopt-encointer-runtime.html#rfc-0022-adopt-encointer-runtime">RFC-0022: Adopt Encointer Runtime</a>
<ul>
<li><a href="approved/0022-adopt-encointer-runtime.html#summary">Summary</a></li>
<li><a href="approved/0022-adopt-encointer-runtime.html#motivation">Motivation</a></li>
<li><a href="approved/0022-adopt-encointer-runtime.html#stakeholders">Stakeholders</a></li>
<li><a href="approved/0022-adopt-encointer-runtime.html#explanation">Explanation</a></li>
<li><a href="approved/0022-adopt-encointer-runtime.html#drawbacks">Drawbacks</a></li>
<li><a href="approved/0022-adopt-encointer-runtime.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="approved/0022-adopt-encointer-runtime.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a></li>
<li><a href="approved/0022-adopt-encointer-runtime.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="approved/0022-adopt-encointer-runtime.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="approved/0022-adopt-encointer-runtime.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0022-adopt-encointer-runtime"><a class="header" href="#rfc-0022-adopt-encointer-runtime">RFC-0022: Adopt Encointer Runtime</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>Aug 22nd 2023</td></tr>
<tr><td><strong>Description</strong></td><td>Permanently move the Encointer runtime into the Fellowship runtimes repo.</td></tr>
<tr><td><strong>Authors</strong></td><td>@brenzi for Encointer Association, 8000 Zurich, Switzerland</td></tr>
</tbody></table>
</div>
<h2 id="summary-18"><a class="header" href="#summary-18">Summary</a></h2>
<p>Encointer is a system chain on Kusama since Jan 2022 and has been developed and maintained by the Encointer association. This RFC proposes to treat Encointer like any other system chain and include it in the fellowship repo with <a href="https://github.com/polkadot-fellows/runtimes/pull/17">this PR</a>.</p>
<h2 id="motivation-18"><a class="header" href="#motivation-18">Motivation</a></h2>
<p>Encointer does not seek to be in control of its runtime repository. As a decentralized system, the fellowship has a more suitable structure to maintain a system chain runtime repo than the Encointer association does.</p>
<p>Also, Encointer aims to update its runtime in batches with other system chains in order to have consistency for interoperability across system chains. </p>
<h2 id="stakeholders-17"><a class="header" href="#stakeholders-17">Stakeholders</a></h2>
<ul>
<li>Fellowship: Will continue to take upon them the review and auditing work for the Encointer runtime, but the process is streamlined with other system chains and therefore less time-consuming compared to the separate repo and CI process we currently have.</li>
<li>Kusama Network: Tokenholders can easily see the changes of all system chains in one place.</li>
<li>Encointer Association: Further decentralization of the Encointer Network necessities like devops.</li>
<li>Encointer devs: Being able to work directly in the Fellowship runtimes repo to streamline and synergize with other developers. </li>
</ul>
<h2 id="explanation-18"><a class="header" href="#explanation-18">Explanation</a></h2>
<p><a href="https://github.com/polkadot-fellows/runtimes/pull/17">Our PR</a> has all details about our runtime and how we would move it into the fellowship repo.</p>
<p>Noteworthy: All Encointer-specific pallets will still be located in encointer's repo for the time being: https://github.com/encointer/pallets </p>
<p>It will still be the duty of the Encointer team to keep its runtime up to date and provide adequate test fixtures. Frequent dependency bumps with Polkadot releases would be beneficial for interoperability and could be streamlined with other system chains but that will not be a duty of fellowship. Whenever possible, all system chains could be upgraded jointly (including Encointer) with a batch referendum.</p>
<p>Further notes:</p>
<ul>
<li>Encointer will publish all its crates crates.io</li>
<li>Encointer does not carry out external auditing of its runtime nor pallets. It would be beneficial but not a requirement from our side if Encointer could join the auditing process of other system chains. </li>
</ul>
<h2 id="drawbacks-14"><a class="header" href="#drawbacks-14">Drawbacks</a></h2>
<p>Other than all other system chains, development and maintenance of the Encointer Network is mainly financed by the KSM Treasury and possibly the DOT Treasury in the future. Encointer is dedicated to maintaining its network and runtime code for as long as possible, but there is a dependency on funding which is not in the hands of the fellowship. The only risk in the context of funding, however, is that the Encointer runtime will see less frequent updates if there's less funding. </p>
<h2 id="testing-security-and-privacy-17"><a class="header" href="#testing-security-and-privacy-17">Testing, Security, and Privacy</a></h2>
<p>No changes to the existing system are proposed. Only changes to how maintenance is organized.</p>
<h2 id="performance-ergonomics-and-compatibility-16"><a class="header" href="#performance-ergonomics-and-compatibility-16">Performance, Ergonomics, and Compatibility</a></h2>
<p>No changes</p>
<h2 id="prior-art-and-references-14"><a class="header" href="#prior-art-and-references-14">Prior Art and References</a></h2>
<p><a href="https://github.com/encointer/encointer-parachain/tree/master/polkadot-parachains/encointer-runtime">Existing Encointer runtime repo</a></p>
<h2 id="unresolved-questions-15"><a class="header" href="#unresolved-questions-15">Unresolved Questions</a></h2>
<p>None identified</p>
<h2 id="future-directions-and-related-material-13"><a class="header" href="#future-directions-and-related-material-13">Future Directions and Related Material</a></h2>
<p>More info on Encointer: <a href="https://encointer.org">encointer.org</a></p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/blob/main/text/0032-minimal-relay.md">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="approved/0032-minimal-relay.html#rfc-0032-minimal-relay">RFC-0032: Minimal Relay</a>
<ul>
<li><a href="approved/0032-minimal-relay.html#summary">Summary</a></li>
<li><a href="approved/0032-minimal-relay.html#motivation">Motivation</a></li>
<li><a href="approved/0032-minimal-relay.html#stakeholders">Stakeholders</a></li>
<li><a href="approved/0032-minimal-relay.html#explanation">Explanation</a>
<ul>
<li><a href="approved/0032-minimal-relay.html#migrations">Migrations</a></li>
<li><a href="approved/0032-minimal-relay.html#interfaces">Interfaces</a></li>
<li><a href="approved/0032-minimal-relay.html#functional-architecture">Functional Architecture</a></li>
<li><a href="approved/0032-minimal-relay.html#resource-allocation">Resource Allocation</a></li>
<li><a href="approved/0032-minimal-relay.html#deployment">Deployment</a></li>
<li><a href="approved/0032-minimal-relay.html#kusama">Kusama</a></li>
</ul>
</li>
<li><a href="approved/0032-minimal-relay.html#drawbacks">Drawbacks</a></li>
<li><a href="approved/0032-minimal-relay.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="approved/0032-minimal-relay.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="approved/0032-minimal-relay.html#performance">Performance</a></li>
<li><a href="approved/0032-minimal-relay.html#ergonomics">Ergonomics</a></li>
<li><a href="approved/0032-minimal-relay.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="approved/0032-minimal-relay.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="approved/0032-minimal-relay.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="approved/0032-minimal-relay.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0032-minimal-relay"><a class="header" href="#rfc-0032-minimal-relay">RFC-0032: Minimal Relay</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>20 September 2023</td></tr>
<tr><td><strong>Description</strong></td><td>Proposal to minimise Relay Chain functionality.</td></tr>
<tr><td><strong>Authors</strong></td><td>Joe Petrowski, Gavin Wood</td></tr>
</tbody></table>
</div>
<h2 id="summary-19"><a class="header" href="#summary-19">Summary</a></h2>
<p>The Relay Chain contains most of the core logic for the Polkadot network. While this was necessary
prior to the launch of parachains and development of XCM, most of this logic can exist in
parachains. This is a proposal to migrate several subsystems into system parachains.</p>
<h2 id="motivation-19"><a class="header" href="#motivation-19">Motivation</a></h2>
<p>Polkadot's scaling approach allows many distinct state machines (known generally as parachains) to
operate with common guarantees about the validity and security of their state transitions. Polkadot
provides these common guarantees by executing the state transitions on a strict subset (a backing
group) of the Relay Chain's validator set.</p>
<p>However, state transitions on the Relay Chain need to be executed by <em>all</em> validators. If any of
those state transitions can occur on parachains, then the resources of the complement of a single
backing group could be used to offer more cores. As in, they could be offering more coretime (a.k.a.
blockspace) to the network.</p>
<p>By minimising state transition logic on the Relay Chain by migrating it into &quot;system chains&quot; -- a
set of parachains that, with the Relay Chain, make up the Polkadot protocol -- the Polkadot
Ubiquitous Computer can maximise its primary offering: secure blockspace.</p>
<h2 id="stakeholders-18"><a class="header" href="#stakeholders-18">Stakeholders</a></h2>
<ul>
<li>Parachains that interact with affected logic on the Relay Chain;</li>
<li>Core protocol and XCM format developers;</li>
<li>Tooling, block explorer, and UI developers.</li>
</ul>
<h2 id="explanation-19"><a class="header" href="#explanation-19">Explanation</a></h2>
<p>The following pallets and subsystems are good candidates to migrate from the Relay Chain:</p>
<ul>
<li>Identity</li>
<li>Balances</li>
<li>Staking
<ul>
<li>Staking</li>
<li>Election Provider</li>
<li>Bags List</li>
<li>NIS</li>
<li>Nomination Pools</li>
<li>Fast Unstake</li>
</ul>
</li>
<li>Governance
<ul>
<li>Treasury and Bounties</li>
<li>Conviction Voting</li>
<li>Referenda</li>
</ul>
</li>
</ul>
<p>Note: The Auctions and Crowdloan pallets will be replaced by Coretime, its system chain and
interface described in RFC-1 and RFC-5, respectively.</p>
<h3 id="migrations"><a class="header" href="#migrations">Migrations</a></h3>
<p>Some subsystems are simpler to move than others. For example, migrating Identity can be done by
simply preventing state changes in the Relay Chain, using the Identity-related state as the genesis
for a new chain, and launching that new chain with the genesis and logic (pallet) needed.</p>
<p>Other subsystems cannot experience any downtime like this because they are essential to the
network's functioning, like Staking and Governance. However, these can likely coexist with a
similarly-permissioned system chain for some time, much like how &quot;Gov1&quot; and &quot;OpenGov&quot; coexisted at
the latter's introduction.</p>
<p>Specific migration plans will be included in release notes of runtimes from the Polkadot Fellowship
when beginning the work of migrating a particular subsystem.</p>
<h3 id="interfaces"><a class="header" href="#interfaces">Interfaces</a></h3>
<p>The Relay Chain, in many cases, will still need to interact with these subsystems, especially
Staking and Governance. These subsystems will require making some APIs available either via
dispatchable calls accessible to XCM <code>Transact</code> or possibly XCM <code>Instruction</code>s in future versions.</p>
<p>For example, Staking provides a pallet-API to register points (e.g. for block production) and
offences (e.g. equivocation). With Staking in a system chain, that chain would need to allow the
Relay Chain to update validator points periodically so that it can correctly calculate rewards.</p>
<p>A pub-sub protocol may also lend itself to these types of interactions.</p>
<h3 id="functional-architecture"><a class="header" href="#functional-architecture">Functional Architecture</a></h3>
<p>This RFC proposes that system chains form individual components within the system's architecture and
that these components are chosen as functional groups. This approach allows synchronous
composibility where it is most valuable, but isolates logic in such a way that provides flexibility
for optimal resource allocation (see <a href="approved/0032-minimal-relay.html#resource-allocation">Resource Allocation</a>). For the
subsystems discussed in this RFC, namely Identity, Governance, and Staking, this would mean:</p>
<ul>
<li>People Chain, for identity and personhood logic, providing functionality related to the attributes
of single actors;</li>
<li>Governance Chain, for governance and system collectives, providing functionality for pluralities
to express their voices within the system;</li>
<li>Staking Chain, for Polkadot's staking system, including elections, nominations, reward
distribution, slashing, and non-interactive staking; and</li>
<li>Asset Hub, for fungible and non-fungible assets, including DOT.</li>
</ul>
<p>The Collectives chain and Asset Hub already exist, so implementation of this RFC would mean two new
chains (People and Staking), with Governance moving to the <em>currently-known-as</em> Collectives chain
and Asset Hub being increasingly used for DOT over the Relay Chain.</p>
<p>Note that one functional group will likely include many pallets, as we do not know how pallet
configurations and interfaces will evolve over time.</p>
<h3 id="resource-allocation"><a class="header" href="#resource-allocation">Resource Allocation</a></h3>
<p>The system should minimise wasted blockspace. These three (and other) subsystems may not each
consistently require a dedicated core. However, core scheduling is far more agile than functional
grouping. While migrating functionality from one chain to another can be a multi-month endeavour,
cores can be rescheduled almost on-the-fly.</p>
<p>Migrations are also breaking changes to some use cases, for example other parachains that need to
route XCM programs to particular chains. It is thus preferable to do them a single time in migrating
off the Relay Chain, reducing the risk of needing parachain splits in the future.</p>
<p>Therefore, chain boundaries should be based on functional grouping where synchronous composibility
is most valuable; and efficient resource allocation should be managed by the core scheduling
protocol.</p>
<p>Many of these system chains (including Asset Hub) could often share a single core in a semi-round
robin fashion (the coretime may not be uniform). When needed, for example during NPoS elections or
slashing events, the scheduler could allocate a dedicated core to the chain in need of more
throughput.</p>
<h3 id="deployment"><a class="header" href="#deployment">Deployment</a></h3>
<p>Actual migrations should happen based on some prioritization. This RFC proposes to migrate Identity,
Staking, and Governance as the systems to work on first. A brief discussion on the factors involved
in each one:</p>
<h4 id="identity"><a class="header" href="#identity">Identity</a></h4>
<p>Identity will be one of the simpler pallets to migrate into a system chain, as its logic is largely
self-contained and it does not &quot;share&quot; balances with other subsystems. As in, any DOT is held in
reserve as a storage deposit and cannot be simultaneously used the way locked DOT can be locked for
multiple purposes.</p>
<p>Therefore, migration can take place as follows:</p>
<ol>
<li>The pallet can be put in a locked state, blocking most calls to the pallet and preventing updates
to identity info.</li>
<li>The frozen state will form the genesis of a new system parachain.</li>
<li>Functions will be added to the pallet that allow migrating the deposit to the parachain. The
parachain deposit is on the order of 1/100th of the Relay Chain's. Therefore, this will result in
freeing up Relay State as well as most of each user's reserved balance.</li>
<li>The pallet and any leftover state can be removed from the Relay Chain.</li>
</ol>
<p>User interfaces that render Identity information will need to source their data from the new system
parachain.</p>
<p>Note: In the future, it may make sense to decommission Kusama's Identity chain and do all account
identities via Polkadot's. However, the Kusama chain will serve as a dress rehearsal for Polkadot.</p>
<h4 id="staking"><a class="header" href="#staking">Staking</a></h4>
<p>Migrating the staking subsystem will likely be the most complex technical undertaking, as the
Staking system cannot stop (the system MUST always have a validator set) nor run in parallel (the
system MUST have only <em>one</em> validator set) and the subsystem itself is made up of subsystems in the
runtime and the node. For example, if offences are reported to the Staking parachain, validator
nodes will need to submit their reports there.</p>
<p>Handling balances also introduces complications. The same balance can be used for staking and
governance. Ideally, all balances stay on Asset Hub, and only report &quot;credits&quot; to system chains like
Staking and Governance. However, staking mutates balances by issuing new DOT on era changes and for
rewards. Allowing DOT directly on the Staking parachain would simplify staking changes.</p>
<p>Given the complexity, it would be pragmatic to include the Balances pallet in the Staking parachain
in its first version. Any other systems that use overlapping locks, most notably governance, will
need to recognise DOT held on both Asset Hub and the Staking parachain.</p>
<p>There is more discussion about staking in a parachain in <a href="https://github.com/paritytech/polkadot-sdk/issues/491">Moving Staking off the Relay
Chain</a>.</p>
<h4 id="governance"><a class="header" href="#governance">Governance</a></h4>
<p>Migrating governance into a parachain will be less complicated than staking. Most of the primitives
needed for the migration already exist. The Treasury supports spending assets on remote chains and
collectives like the Polkadot Technical Fellowship already function in a parachain. That is, XCM
already provides the ability to express system origins across chains.</p>
<p>Therefore, actually moving the governance logic into a parachain will be simple. It can run in
parallel with the Relay Chain's governance, which can be removed when the parachain has demonstrated
sufficient functionality. It's possible that the Relay Chain maintain a Root-level emergency track
for situations like <a href="https://forum.polkadot.network/t/stalled-parachains-on-kusama-post-mortem/3998">parachains
halting</a>.</p>
<p>The only complication arises from the fact that both Asset Hub and the Staking parachain will have
DOT balances; therefore, the Governance chain will need to be able to credit users' voting power
based on balances from both locations. This is not expected to be difficult to handle.</p>
<h3 id="kusama"><a class="header" href="#kusama">Kusama</a></h3>
<p>Although Polkadot and Kusama both have system chains running, they have to date only been used for
introducing new features or bodies, for example fungible assets or the Technical Fellowship. There
has not yet been a migration of logic/state from the Relay Chain into a parachain. Given its more
realistic network conditions than testnets, Kusama is the best stage for rehearsal.</p>
<p>In the case of identity, Polkadot's system may be sufficient for the ecosystem. Therefore, Kusama
should be used to test the migration of logic and state from Relay Chain to parachain, but these
features may be (at the will of Kusama's governance) dropped from Kusama entirely after a successful
migration on Polkadot.</p>
<p>For Governance, Polkadot already has the Collectives parachain, which would become the Governance
parachain. The entire group of DOT holders is itself a collective (the legislative body), and
governance provides the means to express voice. Launching a Kusama Governance chain would be
sensible to rehearse a migration.</p>
<p>The Staking subsystem is perhaps where Kusama would provide the most value in its canary capacity.
Staking is the subsystem most constrained by PoV limits. Ensuring that elections, payouts, session
changes, offences/slashes, etc. work in a parachain on Kusama -- with its larger validator set --
will give confidence to the chain's robustness on Polkadot.</p>
<h2 id="drawbacks-15"><a class="header" href="#drawbacks-15">Drawbacks</a></h2>
<p>These subsystems will have reduced resources in cores than on the Relay Chain. Staking in particular
may require some optimizations to deal with constraints.</p>
<h2 id="testing-security-and-privacy-18"><a class="header" href="#testing-security-and-privacy-18">Testing, Security, and Privacy</a></h2>
<p>Standard audit/review requirements apply. More powerful multi-chain integration test tools would be
useful in developement.</p>
<h2 id="performance-ergonomics-and-compatibility-17"><a class="header" href="#performance-ergonomics-and-compatibility-17">Performance, Ergonomics, and Compatibility</a></h2>
<p>Describe the impact of the proposal on the exposed functionality of Polkadot.</p>
<h3 id="performance-15"><a class="header" href="#performance-15">Performance</a></h3>
<p>This is an optimization. The removal of public/user transactions on the Relay Chain ensures that its
primary resources are allocated to system performance.</p>
<h3 id="ergonomics-13"><a class="header" href="#ergonomics-13">Ergonomics</a></h3>
<p>This proposal alters very little for coretime users (e.g. parachain developers). Application
developers will need to interact with multiple chains, making ergonomic light client tools
particularly important for application development.</p>
<p>For existing parachains that interact with these subsystems, they will need to configure their
runtimes to recognize the new locations in the network.</p>
<h3 id="compatibility-12"><a class="header" href="#compatibility-12">Compatibility</a></h3>
<p>Implementing this proposal will require some changes to pallet APIs and/or a pub-sub protocol.
Application developers will need to interact with multiple chains in the network.</p>
<h2 id="prior-art-and-references-15"><a class="header" href="#prior-art-and-references-15">Prior Art and References</a></h2>
<ul>
<li><a href="https://github.com/paritytech/polkadot/issues/323">Transactionless Relay-chain</a></li>
<li><a href="https://github.com/paritytech/polkadot-sdk/issues/491">Moving Staking off the Relay Chain</a></li>
</ul>
<h2 id="unresolved-questions-16"><a class="header" href="#unresolved-questions-16">Unresolved Questions</a></h2>
<p>There remain some implementation questions, like how to use balances for both Staking and
Governance. See, for example, <a href="https://github.com/paritytech/polkadot-sdk/issues/491">Moving Staking off the Relay
Chain</a>.</p>
<h2 id="future-directions-and-related-material-14"><a class="header" href="#future-directions-and-related-material-14">Future Directions and Related Material</a></h2>
<p>Ideally the Relay Chain becomes transactionless, such that not even balances are represented there.
With Staking and Governance off the Relay Chain, this is not an unreasonable next step.</p>
<p>With Identity on Polkadot, Kusama may opt to drop its People Chain.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/blob/main/text/0042-extrinsics-state-version.md">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="approved/0042-extrinsics-state-version.html#rfc-0042-add-system-version-that-replaces-stateversion-on-runtimeversion">RFC-0042: Add System version that replaces StateVersion on RuntimeVersion</a>
<ul>
<li><a href="approved/0042-extrinsics-state-version.html#summary">Summary</a></li>
<li><a href="approved/0042-extrinsics-state-version.html#motivation">Motivation</a></li>
<li><a href="approved/0042-extrinsics-state-version.html#stakeholders">Stakeholders</a></li>
<li><a href="approved/0042-extrinsics-state-version.html#explanation">Explanation</a></li>
<li><a href="approved/0042-extrinsics-state-version.html#drawbacks">Drawbacks</a></li>
<li><a href="approved/0042-extrinsics-state-version.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="approved/0042-extrinsics-state-version.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="approved/0042-extrinsics-state-version.html#performance">Performance</a></li>
<li><a href="approved/0042-extrinsics-state-version.html#ergonomics">Ergonomics</a></li>
<li><a href="approved/0042-extrinsics-state-version.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="approved/0042-extrinsics-state-version.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="approved/0042-extrinsics-state-version.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="approved/0042-extrinsics-state-version.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0042-add-system-version-that-replaces-stateversion-on-runtimeversion"><a class="header" href="#rfc-0042-add-system-version-that-replaces-stateversion-on-runtimeversion">RFC-0042: Add System version that replaces StateVersion on RuntimeVersion</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>25th October 2023</td></tr>
<tr><td><strong>Description</strong></td><td>Add System Version and remove State Version</td></tr>
<tr><td><strong>Authors</strong></td><td>Vedhavyas Singareddi</td></tr>
</tbody></table>
</div>
<h2 id="summary-20"><a class="header" href="#summary-20">Summary</a></h2>
<p>At the moment, we have <code>system_version</code> field on <code>RuntimeVersion</code> that derives which state version is used for the
Storage.
We have a use case where we want extrinsics root is derived using <code>StateVersion::V1</code>. Without defining a new field
under <code>RuntimeVersion</code>,
we would like to propose adding <code>system_version</code> that can be used to derive both storage and extrinsic state version.</p>
<h2 id="motivation-20"><a class="header" href="#motivation-20">Motivation</a></h2>
<p>Since the extrinsic state version is always <code>StateVersion::V0</code>, deriving extrinsic root requires full extrinsic data.
This would be problematic when we need to verify the extrinsics root if the extrinsic sizes are bigger. This problem is
further explored in https://github.com/polkadot-fellows/RFCs/issues/19</p>
<p>For <code>Subspace</code> project, we have an enshrined rollups called <code>Domain</code> with optimistic verification and Fraud proofs are
used to detect malicious behavior.
One of the <code>Fraud proof</code> variant is to derive <code>Domain</code> block extrinsic root on <code>Subspace</code>'s consensus chain.
Since <code>StateVersion::V0</code> requires full extrinsic data, we are forced to pass all the extrinsics through the Fraud proof.
One of the main challenge here is some extrinsics could be big enough that this variant of Fraud proof may not be
included in the Consensus block due to Block's weight restriction.
If the extrinsic root is derived using <code>StateVersion::V1</code>, then we do not need to pass the full extrinsic data but
rather at maximum, 32 byte of extrinsic data.</p>
<h2 id="stakeholders-19"><a class="header" href="#stakeholders-19">Stakeholders</a></h2>
<ul>
<li>Technical Fellowship, in its role of maintaining system runtimes.</li>
</ul>
<h2 id="explanation-20"><a class="header" href="#explanation-20">Explanation</a></h2>
<p>In order to use project specific StateVersion for extrinsic roots, we proposed
an <a href="https://github.com/paritytech/polkadot-sdk/pull/1691">implementation</a> that introduced
parameter to <code>frame_system::Config</code> but that unfortunately did not feel correct.
So we would like to <a href="https://github.com/paritytech/polkadot-sdk/pull/1968">propose</a> adding this change to
the <code>RuntimeVersion</code>
object. The system version, if introduced, will be used to derive both storage and extrinsic state version.
If system version is <code>0</code>, then both Storage and Extrinsic State version would use V0.
If system version is <code>1</code>, then Storage State version would use V1 and Extrinsic State version would use V0.
If system version is <code>2</code>, then both Storage and Extrinsic State version would use V1.</p>
<p>If implemented, the new <code>RuntimeVersion</code> definition would look something similar to</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>/// Runtime version (Rococo).
#[sp_version::runtime_version]
pub const VERSION: RuntimeVersion = RuntimeVersion {
spec_name: create_runtime_str!(&quot;rococo&quot;),
impl_name: create_runtime_str!(&quot;parity-rococo-v2.0&quot;),
authoring_version: 0,
spec_version: 10020,
impl_version: 0,
apis: RUNTIME_API_VERSIONS,
transaction_version: 22,
system_version: 1,
};
<span class="boring">}</span></code></pre></pre>
<h2 id="drawbacks-16"><a class="header" href="#drawbacks-16">Drawbacks</a></h2>
<p>There should be no drawbacks as it would replace <code>state_version</code> with same behavior but documentation should be updated
so that chains know which <code>system_version</code> to use.</p>
<h2 id="testing-security-and-privacy-19"><a class="header" href="#testing-security-and-privacy-19">Testing, Security, and Privacy</a></h2>
<p>AFAIK, should not have any impact on the security or privacy.</p>
<h2 id="performance-ergonomics-and-compatibility-18"><a class="header" href="#performance-ergonomics-and-compatibility-18">Performance, Ergonomics, and Compatibility</a></h2>
<p>These changes should be compatible for existing chains if they use <code>state_version</code> value for <code>system_verision</code>.</p>
<h3 id="performance-16"><a class="header" href="#performance-16">Performance</a></h3>
<p>I do not believe there is any performance hit with this change.</p>
<h3 id="ergonomics-14"><a class="header" href="#ergonomics-14">Ergonomics</a></h3>
<p>This does not break any exposed Apis.</p>
<h3 id="compatibility-13"><a class="header" href="#compatibility-13">Compatibility</a></h3>
<p>This change should not break any compatibility.</p>
<h2 id="prior-art-and-references-16"><a class="header" href="#prior-art-and-references-16">Prior Art and References</a></h2>
<p>We <a href="https://github.com/paritytech/polkadot-sdk/pull/1691">proposed</a> introducing a similar change by introducing a
parameter to <code>frame_system::Config</code> but did not feel that
is the correct way of introducing this change.</p>
<h2 id="unresolved-questions-17"><a class="header" href="#unresolved-questions-17">Unresolved Questions</a></h2>
<p>I do not have any specific questions about this change at the moment.</p>
<h2 id="future-directions-and-related-material-15"><a class="header" href="#future-directions-and-related-material-15">Future Directions and Related Material</a></h2>
<p>IMO, this change is pretty self-contained and there won't be any future work necessary. </p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/blob/main/text/0043-storage-proof-size-hostfunction.md">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="approved/0043-storage-proof-size-hostfunction.html#rfc-0043-introduce-storage_proof_size-host-function-for-improved-parachain-block-utilization">RFC-0043: Introduce <code>storage_proof_size</code> Host Function for Improved Parachain Block Utilization</a>
<ul>
<li><a href="approved/0043-storage-proof-size-hostfunction.html#summary">Summary</a></li>
<li><a href="approved/0043-storage-proof-size-hostfunction.html#motivation">Motivation</a></li>
<li><a href="approved/0043-storage-proof-size-hostfunction.html#stakeholders">Stakeholders</a></li>
<li><a href="approved/0043-storage-proof-size-hostfunction.html#explanation">Explanation</a></li>
<li><a href="approved/0043-storage-proof-size-hostfunction.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="approved/0043-storage-proof-size-hostfunction.html#performance">Performance</a></li>
<li><a href="approved/0043-storage-proof-size-hostfunction.html#ergonomics">Ergonomics</a></li>
<li><a href="approved/0043-storage-proof-size-hostfunction.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="approved/0043-storage-proof-size-hostfunction.html#prior-art-and-references">Prior Art and References</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0043-introduce-storage_proof_size-host-function-for-improved-parachain-block-utilization"><a class="header" href="#rfc-0043-introduce-storage_proof_size-host-function-for-improved-parachain-block-utilization">RFC-0043: Introduce <code>storage_proof_size</code> Host Function for Improved Parachain Block Utilization</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>30 October 2023</td></tr>
<tr><td><strong>Description</strong></td><td>Host function to provide the storage proof size to runtimes.</td></tr>
<tr><td><strong>Authors</strong></td><td>Sebastian Kunert</td></tr>
</tbody></table>
</div>
<h2 id="summary-21"><a class="header" href="#summary-21">Summary</a></h2>
<p>This RFC proposes a new host function for parachains, <code>storage_proof_size</code>. It shall provide the size of the currently recorded storage proof to the runtime. Runtime authors can use the proof size to improve block utilization by retroactively reclaiming unused storage weight.</p>
<h2 id="motivation-21"><a class="header" href="#motivation-21">Motivation</a></h2>
<p>The number of extrinsics that are included in a parachain block is limited by two constraints: execution time and proof size. FRAME weights cover both concepts, and block-builders use them to decide how many extrinsics to include in a block. However, these weights are calculated ahead of time by benchmarking on a machine with reference hardware. The execution-time properties of the state-trie and its storage items are unknown at benchmarking time. Therefore, we make some assumptions about the state-trie:</p>
<ul>
<li><strong>Trie Depth:</strong> We assume a trie depth to account for intermediary nodes.</li>
<li><strong>Storage Item Size:</strong> We make a pessimistic assumption based on the <code>MaxEncodedLen</code> trait.</li>
</ul>
<p>These pessimistic assumptions lead to an overestimation of storage weight, negatively impacting block utilization on parachains.</p>
<p>In addition, the current model does not account for multiple accesses to the same storage items. While these repetitive accesses will not increase storage-proof size, the runtime-side weight monitoring will account for them multiple times. Since the proof size is completely opaque to the runtime, we can not implement retroactive storage weight correction.</p>
<p>A solution must provide a way for the runtime to track the exact storage-proof size consumed on a per-extrinsic basis.</p>
<h2 id="stakeholders-20"><a class="header" href="#stakeholders-20">Stakeholders</a></h2>
<ul>
<li><strong>Parachain Teams:</strong> They MUST include this host function in their runtime and node.</li>
<li><strong>Light-client Implementors:</strong> They SHOULD include this host function in their runtime and node.</li>
</ul>
<h2 id="explanation-21"><a class="header" href="#explanation-21">Explanation</a></h2>
<p>This RFC proposes a new host function that exposes the storage-proof size to the runtime. As a result, runtimes can implement storage weight reclaiming mechanisms that improve block utilization.</p>
<p>This RFC proposes the following host function signature:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>fn ext_storage_proof_size_version_1() -&gt; u64;
<span class="boring">}</span></code></pre></pre>
<p>The host function MUST return an unsigned 64-bit integer value representing the current proof size. In block-execution and block-import contexts, this function MUST return the current size of the proof. To achieve this, parachain node implementors need to enable proof recording for block imports. In other contexts, this function MUST return 18446744073709551615 (u64::MAX), which represents disabled proof recording.</p>
<h2 id="performance-ergonomics-and-compatibility-19"><a class="header" href="#performance-ergonomics-and-compatibility-19">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-17"><a class="header" href="#performance-17">Performance</a></h3>
<p>Parachain nodes need to enable proof recording during block import to correctly implement the proposed host function. Benchmarking conducted with balance transfers has shown a performance reduction of around 0.6% when proof recording is enabled. </p>
<h3 id="ergonomics-15"><a class="header" href="#ergonomics-15">Ergonomics</a></h3>
<p>The host function proposed in this RFC allows parachain runtime developers to keep track of the proof size. Typical usage patterns would be to keep track of the overall proof size or the difference between subsequent calls to the host function.</p>
<h3 id="compatibility-14"><a class="header" href="#compatibility-14">Compatibility</a></h3>
<p>Parachain teams will need to include this host function to upgrade.</p>
<h2 id="prior-art-and-references-17"><a class="header" href="#prior-art-and-references-17">Prior Art and References</a></h2>
<ul>
<li>Pull Request including proposed host function: <a href="https://github.com/paritytech/polkadot-sdk/pull/1462">PoV Reclaim (Clawback) Node Side</a>.</li>
<li>Issue with discussion: <a href="https://github.com/paritytech/polkadot-sdk/issues/209#top">[FRAME core] Clawback PoV Weights For Dispatchables</a></li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/blob/main/text/0047-assignment-of-availability-chunks.md">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="approved/0047-assignment-of-availability-chunks.html#rfc-0047-assignment-of-availability-chunks-to-validators">RFC-0047: Assignment of availability chunks to validators</a>
<ul>
<li><a href="approved/0047-assignment-of-availability-chunks.html#summary">Summary</a></li>
<li><a href="approved/0047-assignment-of-availability-chunks.html#motivation">Motivation</a></li>
<li><a href="approved/0047-assignment-of-availability-chunks.html#stakeholders">Stakeholders</a></li>
<li><a href="approved/0047-assignment-of-availability-chunks.html#explanation">Explanation</a>
<ul>
<li><a href="approved/0047-assignment-of-availability-chunks.html#systematic-erasure-codes">Systematic erasure codes</a></li>
<li><a href="approved/0047-assignment-of-availability-chunks.html#availability-recovery-at-present">Availability recovery at present</a></li>
<li><a href="approved/0047-assignment-of-availability-chunks.html#availability-recovery-from-systematic-chunks">Availability recovery from systematic chunks</a></li>
<li><a href="approved/0047-assignment-of-availability-chunks.html#chunk-assignment-function">Chunk assignment function</a></li>
<li><a href="approved/0047-assignment-of-availability-chunks.html#network-protocol">Network protocol</a></li>
<li><a href="approved/0047-assignment-of-availability-chunks.html#upgrade-path">Upgrade path</a></li>
</ul>
</li>
<li><a href="approved/0047-assignment-of-availability-chunks.html#drawbacks">Drawbacks</a></li>
<li><a href="approved/0047-assignment-of-availability-chunks.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="approved/0047-assignment-of-availability-chunks.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="approved/0047-assignment-of-availability-chunks.html#performance">Performance</a></li>
<li><a href="approved/0047-assignment-of-availability-chunks.html#ergonomics">Ergonomics</a></li>
<li><a href="approved/0047-assignment-of-availability-chunks.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="approved/0047-assignment-of-availability-chunks.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="approved/0047-assignment-of-availability-chunks.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="approved/0047-assignment-of-availability-chunks.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
<li><a href="approved/0047-assignment-of-availability-chunks.html#appendix-a">Appendix A</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0047-assignment-of-availability-chunks-to-validators"><a class="header" href="#rfc-0047-assignment-of-availability-chunks-to-validators">RFC-0047: Assignment of availability chunks to validators</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>03 November 2023</td></tr>
<tr><td><strong>Description</strong></td><td>An evenly-distributing indirection layer between availability chunks and validators.</td></tr>
<tr><td><strong>Authors</strong></td><td>Alin Dima</td></tr>
</tbody></table>
</div>
<h2 id="summary-22"><a class="header" href="#summary-22">Summary</a></h2>
<p>Propose a way of permuting the availability chunk indices assigned to validators, in the context of
<a href="https://github.com/paritytech/polkadot-sdk/issues/598">recovering available data from systematic chunks</a>, with the
purpose of fairly distributing network bandwidth usage.</p>
<h2 id="motivation-22"><a class="header" href="#motivation-22">Motivation</a></h2>
<p>Currently, the ValidatorIndex is always identical to the ChunkIndex. Since the validator array is only shuffled once
per session, naively using the ValidatorIndex as the ChunkIndex would pose an unreasonable stress on the first N/3
validators during an entire session, when favouring availability recovery from systematic chunks.</p>
<p>Therefore, the relay chain node needs a deterministic way of evenly distributing the first ~(N_VALIDATORS / 3)
systematic availability chunks to different validators, based on the relay chain block and core.
The main purpose is to ensure fair distribution of network bandwidth usage for availability recovery in general and in
particular for systematic chunk holders. </p>
<h2 id="stakeholders-21"><a class="header" href="#stakeholders-21">Stakeholders</a></h2>
<p>Relay chain node core developers.</p>
<h2 id="explanation-22"><a class="header" href="#explanation-22">Explanation</a></h2>
<h3 id="systematic-erasure-codes"><a class="header" href="#systematic-erasure-codes">Systematic erasure codes</a></h3>
<p>An erasure coding algorithm is considered systematic if it preserves the original unencoded data as part of the
resulting code.
<a href="https://github.com/paritytech/reed-solomon-novelpoly">The implementation of the erasure coding algorithm used for polkadot's availability data</a> is systematic.
Roughly speaking, the first N_VALIDATORS/3 chunks of data can be cheaply concatenated to retrieve the original data,
without running the resource-intensive and time-consuming reconstruction algorithm.</p>
<p>You can find the concatenation procedure of systematic chunks for polkadot's erasure coding algorithm
<a href="https://github.com/paritytech/reed-solomon-novelpoly/blob/be3751093e60adc20c19967f5443158552829011/reed-solomon-novelpoly/src/novel_poly_basis/mod.rs#L247">here</a></p>
<p>In a nutshell, it performs a column-wise concatenation with 2-byte chunks.
The output could be zero-padded at the end, so scale decoding must be aware of the expected length in bytes and ignore
trailing zeros (this assertion is already being made for regular reconstruction).</p>
<h3 id="availability-recovery-at-present"><a class="header" href="#availability-recovery-at-present">Availability recovery at present</a></h3>
<p>According to the <a href="https://spec.polkadot.network/chapter-anv#sect-candidate-recovery">polkadot protocol spec</a>:</p>
<blockquote>
<p>A validator should request chunks by picking peers randomly and must recover at least <code>f+1</code> chunks, where
<code>n=3f+k</code> and <code>k in {1,2,3}</code>.</p>
</blockquote>
<p>For parity's polkadot node implementation, the process was further optimised. At this moment, it works differently based
on the estimated size of the available data:</p>
<p>(a) for small PoVs (up to 128 Kib), sequentially try requesting the unencoded data from the backing group, in a random
order. If this fails, fallback to option (b).</p>
<p>(b) for large PoVs (over 128 Kib), launch N parallel requests for the erasure coded chunks (currently, N has an upper
limit of 50), until enough chunks were recovered. Validators are tried in a random order. Then, reconstruct the
original data.</p>
<p>All options require that after reconstruction, validators then re-encode the data and re-create the erasure chunks trie
in order to check the erasure root.</p>
<h3 id="availability-recovery-from-systematic-chunks"><a class="header" href="#availability-recovery-from-systematic-chunks">Availability recovery from systematic chunks</a></h3>
<p>As part of the effort of
<a href="https://github.com/paritytech/roadmap/issues/26">increasing polkadot's resource efficiency, scalability and performance</a>,
work is under way to modify the Availability Recovery protocol by leveraging systematic chunks. See
<a href="https://github.com/paritytech/polkadot-sdk/issues/598#issuecomment-1792007099">this comment</a> for preliminary
performance results.</p>
<p>In this scheme, the relay chain node will first attempt to retrieve the ~N/3 systematic chunks from the validators that
should hold them, before falling back to recovering from regular chunks, as before.</p>
<p>A re-encoding step is still needed for verifying the erasure root, so the erasure coding overhead cannot be completely
brought down to 0.</p>
<p>Not being able to retrieve even one systematic chunk would make systematic reconstruction impossible. Therefore, backers
can be used as a backup to retrieve a couple of missing systematic chunks, before falling back to retrieving regular
chunks.</p>
<h3 id="chunk-assignment-function"><a class="header" href="#chunk-assignment-function">Chunk assignment function</a></h3>
<h4 id="properties"><a class="header" href="#properties">Properties</a></h4>
<p>The function that decides the chunk index for a validator will be parameterized by at least
<code>(validator_index, core_index)</code>
and have the following properties:</p>
<ol>
<li>deterministic</li>
<li>relatively quick to compute and resource-efficient.</li>
<li>when considering a fixed <code>core_index</code>, the function should describe a permutation of the chunk indices</li>
<li>the validators that map to the first N/3 chunk indices should have as little overlap as possible for different cores.</li>
</ol>
<p>In other words, we want a uniformly distributed, deterministic mapping from <code>ValidatorIndex</code> to <code>ChunkIndex</code> per core.</p>
<p>It's desirable to not embed this function in the runtime, for performance and complexity reasons.
However, this means that the function needs to be kept very simple and with minimal or no external dependencies.
Any change to this function could result in parachains being stalled and needs to be coordinated via a runtime upgrade
or governance call.</p>
<h4 id="proposed-function"><a class="header" href="#proposed-function">Proposed function</a></h4>
<p>Pseudocode:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>pub fn get_chunk_index(
n_validators: u32,
validator_index: ValidatorIndex,
core_index: CoreIndex
) -&gt; ChunkIndex {
let threshold = systematic_threshold(n_validators); // Roughly n_validators/3
let core_start_pos = core_index * threshold;
(core_start_pos + validator_index) % n_validators
}
<span class="boring">}</span></code></pre></pre>
<h3 id="network-protocol"><a class="header" href="#network-protocol">Network protocol</a></h3>
<p>The request-response <code>/req_chunk</code> protocol will be bumped to a new version (from v1 to v2).
For v1, the request and response payloads are:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>/// Request an availability chunk.
pub struct ChunkFetchingRequest {
/// Hash of candidate we want a chunk for.
pub candidate_hash: CandidateHash,
/// The index of the chunk to fetch.
pub index: ValidatorIndex,
}
/// Receive a requested erasure chunk.
pub enum ChunkFetchingResponse {
/// The requested chunk data.
Chunk(ChunkResponse),
/// Node was not in possession of the requested chunk.
NoSuchChunk,
}
/// This omits the chunk's index because it is already known by
/// the requester and by not transmitting it, we ensure the requester is going to use his index
/// value for validating the response, thus making sure he got what he requested.
pub struct ChunkResponse {
/// The erasure-encoded chunk of data belonging to the candidate block.
pub chunk: Vec&lt;u8&gt;,
/// Proof for this chunk's branch in the Merkle tree.
pub proof: Proof,
}
<span class="boring">}</span></code></pre></pre>
<p>Version 2 will add an <code>index</code> field to <code>ChunkResponse</code>:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>#[derive(Debug, Clone, Encode, Decode)]
pub struct ChunkResponse {
/// The erasure-encoded chunk of data belonging to the candidate block.
pub chunk: Vec&lt;u8&gt;,
/// Proof for this chunk's branch in the Merkle tree.
pub proof: Proof,
/// Chunk index.
pub index: ChunkIndex
}
<span class="boring">}</span></code></pre></pre>
<p>An important thing to note is that in version 1, the <code>ValidatorIndex</code> value is always equal to the <code>ChunkIndex</code>.
Until the chunk rotation feature is enabled, this will also be true for version 2. However, after the feature is
enabled, this will generally not be true.</p>
<p>The requester will send the request to validator with index <code>V</code>. The responder will map the <code>V</code> validator index to the
<code>C</code> chunk index and respond with the <code>C</code>-th chunk. This mapping can be seamless, by having each validator store their
chunk by <code>ValidatorIndex</code> (just as before).</p>
<p>The protocol implementation MAY check the returned <code>ChunkIndex</code> against the expected mapping to ensure that
it received the right chunk.
In practice, this is desirable during availability-distribution and systematic chunk recovery. However, regular
recovery may not check this index, which is particularly useful when participating in disputes that don't allow
for easy access to the validator-&gt;chunk mapping. See <a href="approved/0047-assignment-of-availability-chunks.html#appendix-a">Appendix A</a> for more details.</p>
<p>In any case, the requester MUST verify the chunk's proof using the provided index.</p>
<p>During availability-recovery, given that the requester may not know (if the mapping is not available) whether the
received chunk corresponds to the requested validator index, it has to keep track of received chunk indices and ignore
duplicates. Such duplicates should be considered the same as an invalid/garbage response (drop it and move on to the
next validator - we can't punish via reputation changes, because we don't know which validator misbehaved).</p>
<h3 id="upgrade-path"><a class="header" href="#upgrade-path">Upgrade path</a></h3>
<h4 id="step-1-enabling-new-network-protocol"><a class="header" href="#step-1-enabling-new-network-protocol">Step 1: Enabling new network protocol</a></h4>
<p>In the beginning, both <code>/req_chunk/1</code> and <code>/req_chunk/2</code> will be supported, until all validators and
collators have upgraded to use the new version. V1 will be considered deprecated. During this step, the mapping will
still be 1:1 (<code>ValidatorIndex</code> == <code>ChunkIndex</code>), regardless of protocol.
Once all nodes are upgraded, a new release will be cut that removes the v1 protocol. Only once all nodes have upgraded
to this version will step 2 commence.</p>
<h4 id="step-2-enabling-the-new-validator-chunk-mapping"><a class="header" href="#step-2-enabling-the-new-validator-chunk-mapping">Step 2: Enabling the new validator-&gt;chunk mapping</a></h4>
<p>Considering that the Validator-&gt;Chunk mapping is critical to para consensus, the change needs to be enacted atomically
via governance, only after all validators have upgraded the node to a version that is aware of this mapping,
functionality-wise.
It needs to be explicitly stated that after the governance enactment, validators that run older client versions that
don't support this mapping will not be able to participate in parachain consensus.</p>
<p>Additionally, an error will be logged when starting a validator with an older version, after the feature was enabled.</p>
<p>On the other hand, collators will not be required to upgrade in this step (but are still require to upgrade for step 1),
as regular chunk recovery will work as before, granted that version 1 of the networking protocol has been removed.
Note that collators only perform availability-recovery in rare, adversarial scenarios, so it is fine to not optimise for
this case and let them upgrade at their own pace.</p>
<p>To support enabling this feature via the runtime, we will use the <code>NodeFeatures</code> bitfield of the <code>HostConfiguration</code>
struct (added in <code>https://github.com/paritytech/polkadot-sdk/pull/2177</code>). Adding and enabling a feature
with this scheme does not require a runtime upgrade, but only a referendum that issues a
<code>Configuration::set_node_feature</code> extrinsic. Once the feature is enabled and new configuration is live, the
validator-&gt;chunk mapping ceases to be a 1:1 mapping and systematic recovery may begin.</p>
<h2 id="drawbacks-17"><a class="header" href="#drawbacks-17">Drawbacks</a></h2>
<ul>
<li>Getting access to the <code>core_index</code> that used to be occupied by a candidate in some parts of the dispute protocol is
very complicated (See <a href="approved/0047-assignment-of-availability-chunks.html#appendix-a">appendix A</a>). This RFC assumes that availability-recovery processes initiated during
disputes will only use regular recovery, as before. This is acceptable since disputes are rare occurrences in practice
and is something that can be optimised later, if need be. Adding the <code>core_index</code> to the <code>CandidateReceipt</code> would
mitigate this problem and will likely be needed in the future for CoreJam and/or Elastic scaling.
<a href="https://forum.polkadot.network/t/pre-rfc-discussion-candidate-receipt-format-v2/3738">Related discussion about updating <code>CandidateReceipt</code></a></li>
<li>It's a breaking change that requires all validators and collators to upgrade their node version at least once.</li>
</ul>
<h2 id="testing-security-and-privacy-20"><a class="header" href="#testing-security-and-privacy-20">Testing, Security, and Privacy</a></h2>
<p>Extensive testing will be conducted - both automated and manual.
This proposal doesn't affect security or privacy.</p>
<h2 id="performance-ergonomics-and-compatibility-20"><a class="header" href="#performance-ergonomics-and-compatibility-20">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-18"><a class="header" href="#performance-18">Performance</a></h3>
<p>This is a necessary data availability optimisation, as reed-solomon erasure coding has proven to be a top consumer of
CPU time in polkadot as we scale up the parachain block size and number of availability cores.</p>
<p>With this optimisation, preliminary performance results show that CPU time used for reed-solomon coding/decoding can be
halved and total POV recovery time decrease by 80% for large POVs. See more
<a href="https://github.com/paritytech/polkadot-sdk/issues/598#issuecomment-1792007099">here</a>.</p>
<h3 id="ergonomics-16"><a class="header" href="#ergonomics-16">Ergonomics</a></h3>
<p>Not applicable.</p>
<h3 id="compatibility-15"><a class="header" href="#compatibility-15">Compatibility</a></h3>
<p>This is a breaking change. See <a href="approved/0047-assignment-of-availability-chunks.html#upgrade-path">upgrade path</a> section above.
All validators and collators need to have upgraded their node versions before the feature will be enabled via a
governance call.</p>
<h2 id="prior-art-and-references-18"><a class="header" href="#prior-art-and-references-18">Prior Art and References</a></h2>
<p>See comments on the <a href="https://github.com/paritytech/polkadot-sdk/issues/598">tracking issue</a> and the
<a href="https://github.com/paritytech/polkadot-sdk/pull/1644">in-progress PR</a></p>
<h2 id="unresolved-questions-18"><a class="header" href="#unresolved-questions-18">Unresolved Questions</a></h2>
<p>Not applicable.</p>
<h2 id="future-directions-and-related-material-16"><a class="header" href="#future-directions-and-related-material-16">Future Directions and Related Material</a></h2>
<p>This enables future optimisations for the performance of availability recovery, such as retrieving batched systematic
chunks from backers/approval-checkers.</p>
<h2 id="appendix-a"><a class="header" href="#appendix-a">Appendix A</a></h2>
<p>This appendix details the intricacies of getting access to the core index of a candidate in parity's polkadot node.</p>
<p>Here, <code>core_index</code> refers to the index of the core that a candidate was occupying while it was pending availability
(from backing to inclusion).</p>
<p>Availability-recovery can currently be triggered by the following phases in the polkadot protocol:</p>
<ol>
<li>During the approval voting process.</li>
<li>By other collators of the same parachain.</li>
<li>During disputes.</li>
</ol>
<p>Getting the right core index for a candidate can be troublesome. Here's a breakdown of how different parts of the
node implementation can get access to it:</p>
<ol>
<li>
<p>The approval-voting process for a candidate begins after observing that the candidate was included. Therefore, the
node has easy access to the block where the candidate got included (and also the core that it occupied).</p>
</li>
<li>
<p>The <code>pov_recovery</code> task of the collators starts availability recovery in response to noticing a candidate getting
backed, which enables easy access to the core index the candidate started occupying.</p>
</li>
<li>
<p>Disputes may be initiated on a number of occasions:</p>
<p>3.a. is initiated by the validator as a result of finding an invalid candidate while participating in the
approval-voting protocol. In this case, availability-recovery is not needed, since the validator already issued their
vote.</p>
<p>3.b is initiated by the validator noticing dispute votes recorded on-chain. In this case, we can safely
assume that the backing event for that candidate has been recorded and kept in memory.</p>
<p>3.c is initiated as a result of getting a dispute statement from another validator. It is possible that the dispute
is happening on a fork that was not yet imported by this validator, so the subsystem may not have seen this candidate
being backed.</p>
</li>
</ol>
<p>A naive attempt of solving 3.c would be to add a new version for the disputes request-response networking protocol.
Blindly passing the core index in the network payload would not work, since there is no way of validating that
the reported core_index was indeed the one occupied by the candidate at the respective relay parent.</p>
<p>Another attempt could be to include in the message the relay block hash where the candidate was included.
This information would be used in order to query the runtime API and retrieve the core index that the candidate was
occupying. However, considering it's part of an unimported fork, the validator cannot call a runtime API on that block.</p>
<p>Adding the <code>core_index</code> to the <code>CandidateReceipt</code> would solve this problem and would enable systematic recovery for all
dispute scenarios.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/blob/main/text/0050-fellowship-salaries.md">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="approved/0050-fellowship-salaries.html#rfc-0050-fellowship-salaries">RFC-0050: Fellowship Salaries</a>
<ul>
<li><a href="approved/0050-fellowship-salaries.html#summary">Summary</a></li>
<li><a href="approved/0050-fellowship-salaries.html#motivation">Motivation</a></li>
<li><a href="approved/0050-fellowship-salaries.html#stakeholders">Stakeholders</a></li>
<li><a href="approved/0050-fellowship-salaries.html#explanation">Explanation</a>
<ul>
<li><a href="approved/0050-fellowship-salaries.html#salary-asset">Salary Asset</a></li>
<li><a href="approved/0050-fellowship-salaries.html#projections">Projections</a></li>
<li><a href="approved/0050-fellowship-salaries.html#updates">Updates</a></li>
</ul>
</li>
<li><a href="approved/0050-fellowship-salaries.html#drawbacks">Drawbacks</a></li>
<li><a href="approved/0050-fellowship-salaries.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="approved/0050-fellowship-salaries.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="approved/0050-fellowship-salaries.html#performance">Performance</a></li>
<li><a href="approved/0050-fellowship-salaries.html#ergonomics">Ergonomics</a></li>
<li><a href="approved/0050-fellowship-salaries.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="approved/0050-fellowship-salaries.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="approved/0050-fellowship-salaries.html#unresolved-questions">Unresolved Questions</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0050-fellowship-salaries"><a class="header" href="#rfc-0050-fellowship-salaries">RFC-0050: Fellowship Salaries</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>15 November 2023</td></tr>
<tr><td><strong>Description</strong></td><td>Proposal to set rank-based Fellowship salary levels.</td></tr>
<tr><td><strong>Authors</strong></td><td>Joe Petrowski, Gavin Wood</td></tr>
</tbody></table>
</div>
<h2 id="summary-23"><a class="header" href="#summary-23">Summary</a></h2>
<p>The Fellowship Manifesto states that members should receive a monthly allowance on par with gross
income in OECD countries. This RFC proposes concrete amounts.</p>
<h2 id="motivation-23"><a class="header" href="#motivation-23">Motivation</a></h2>
<p>One motivation for the Technical Fellowship is to provide an incentive mechanism that can induct and
retain technical talent for the continued progress of the network.</p>
<p>In order for members to uphold their commitment to the network, they should receive support to
ensure that their needs are met such that they have the time to dedicate to their work on Polkadot.
Given the high expectations of Fellows, it is reasonable to consider contributions and requirements
on par with a full-time job. Providing a livable wage to those making such contributions makes it
pragmatic to work full-time on Polkadot.</p>
<p>Note: Goals of the Fellowship, expectations for each Dan, and conditions for promotion and demotion
are all explained in the Manifesto. This RFC is only to propose concrete values for allowances.</p>
<h2 id="stakeholders-22"><a class="header" href="#stakeholders-22">Stakeholders</a></h2>
<ul>
<li>Fellowship members</li>
<li>Polkadot Treasury</li>
</ul>
<h2 id="explanation-23"><a class="header" href="#explanation-23">Explanation</a></h2>
<p>This RFC proposes agreeing on salaries relative to a single level, the III Dan. As such, changes to
the amount or asset used would only be on a single value, and all others would adjust relatively. A
III Dan is someone whose contributions match the expectations of a full-time individual contributor.
The salary at this level should be reasonably close to averages in OECD countries.</p>
<div class="table-wrapper"><table><thead><tr><th style="text-align: center">Dan</th><th style="text-align: center">Factor</th></tr></thead><tbody>
<tr><td style="text-align: center">I</td><td style="text-align: center">0.125</td></tr>
<tr><td style="text-align: center">II</td><td style="text-align: center">0.25</td></tr>
<tr><td style="text-align: center">III</td><td style="text-align: center">1</td></tr>
<tr><td style="text-align: center">IV</td><td style="text-align: center">1.5</td></tr>
<tr><td style="text-align: center">V</td><td style="text-align: center">2.0</td></tr>
<tr><td style="text-align: center">VI</td><td style="text-align: center">2.5</td></tr>
<tr><td style="text-align: center">VII</td><td style="text-align: center">2.5</td></tr>
<tr><td style="text-align: center">VIII</td><td style="text-align: center">2.5</td></tr>
<tr><td style="text-align: center">IX</td><td style="text-align: center">2.5</td></tr>
</tbody></table>
</div>
<p>Note that there is a sizable increase between II Dan (Proficient) and III Dan (Fellow). By the third
Dan, it is generally expected that one is working on Polkadot as their primary focus in a full-time
capacity.</p>
<h3 id="salary-asset"><a class="header" href="#salary-asset">Salary Asset</a></h3>
<p>Although the Manifesto (Section 8) specifies a monthly allowance in DOT, this RFC proposes the use
of USDT instead. The allowance is meant to provide members stability in meeting their day-to-day
needs and recognize contributions. Using USDT provides more stability and less speculation.</p>
<p>This RFC proposes that a III Dan earn 80,000 USDT per year. The salary at this level is commensurate
with average salaries in OECD countries (note: 77,000 USD in the U.S., with an average engineer at
100,000 USD). The other ranks would thus earn:</p>
<div class="table-wrapper"><table><thead><tr><th style="text-align: center">Dan</th><th style="text-align: center">Annual Salary</th></tr></thead><tbody>
<tr><td style="text-align: center">I</td><td style="text-align: center">10,000</td></tr>
<tr><td style="text-align: center">II</td><td style="text-align: center">20,000</td></tr>
<tr><td style="text-align: center">III</td><td style="text-align: center">80,000</td></tr>
<tr><td style="text-align: center">IV</td><td style="text-align: center">120,000</td></tr>
<tr><td style="text-align: center">V</td><td style="text-align: center">160,000</td></tr>
<tr><td style="text-align: center">VI</td><td style="text-align: center">200,000</td></tr>
<tr><td style="text-align: center">VII</td><td style="text-align: center">200,000</td></tr>
<tr><td style="text-align: center">VIII</td><td style="text-align: center">200,000</td></tr>
<tr><td style="text-align: center">IX</td><td style="text-align: center">200,000</td></tr>
</tbody></table>
</div>
<p>The salary levels for Architects (IV, V, and VI Dan) are typical of senior engineers.</p>
<p>Allowances will be managed by the Salary pallet.</p>
<h3 id="projections"><a class="header" href="#projections">Projections</a></h3>
<p>Based on the current membership, the maximum yearly and monthly costs are shown below:</p>
<div class="table-wrapper"><table><thead><tr><th style="text-align: center">Dan</th><th style="text-align: center">Salary</th><th style="text-align: center">Members</th><th style="text-align: right">Yearly</th><th style="text-align: right">Monthly</th></tr></thead><tbody>
<tr><td style="text-align: center">I</td><td style="text-align: center">10,000</td><td style="text-align: center">27</td><td style="text-align: right">270,000</td><td style="text-align: right">22,500</td></tr>
<tr><td style="text-align: center">II</td><td style="text-align: center">20,000</td><td style="text-align: center">11</td><td style="text-align: right">220,000</td><td style="text-align: right">18,333</td></tr>
<tr><td style="text-align: center">III</td><td style="text-align: center">80,000</td><td style="text-align: center">8</td><td style="text-align: right">640,000</td><td style="text-align: right">53,333</td></tr>
<tr><td style="text-align: center">IV</td><td style="text-align: center">120,000</td><td style="text-align: center">3</td><td style="text-align: right">360,000</td><td style="text-align: right">30,000</td></tr>
<tr><td style="text-align: center">V</td><td style="text-align: center">160,000</td><td style="text-align: center">5</td><td style="text-align: right">800,000</td><td style="text-align: right">66,667</td></tr>
<tr><td style="text-align: center">VI</td><td style="text-align: center">200,000</td><td style="text-align: center">3</td><td style="text-align: right">600,000</td><td style="text-align: right">50,000</td></tr>
<tr><td style="text-align: center">&gt; VI</td><td style="text-align: center">200,000</td><td style="text-align: center">0</td><td style="text-align: right">0</td><td style="text-align: right">0</td></tr>
<tr><td style="text-align: center"></td><td style="text-align: center"></td><td style="text-align: center"></td><td style="text-align: right"></td><td style="text-align: right"></td></tr>
<tr><td style="text-align: center">Total</td><td style="text-align: center"></td><td style="text-align: center"></td><td style="text-align: right">2,890,000</td><td style="text-align: right">240,833</td></tr>
</tbody></table>
</div>
<p>Note that these are the maximum amounts; members may choose to take a passive (lower) level. On the
other hand, more people will likely join the Fellowship in the coming years.</p>
<h3 id="updates"><a class="header" href="#updates">Updates</a></h3>
<p>Updates to these levels, whether relative ratios, the asset used, or the amount, shall be done via
RFC.</p>
<h2 id="drawbacks-18"><a class="header" href="#drawbacks-18">Drawbacks</a></h2>
<p>By not using DOT for payment, the protocol relies on the stability of other assets and the ability
to acquire them. However, the asset of choice can be changed in the future.</p>
<h2 id="testing-security-and-privacy-21"><a class="header" href="#testing-security-and-privacy-21">Testing, Security, and Privacy</a></h2>
<p>N/A.</p>
<h2 id="performance-ergonomics-and-compatibility-21"><a class="header" href="#performance-ergonomics-and-compatibility-21">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-19"><a class="header" href="#performance-19">Performance</a></h3>
<p>N/A</p>
<h3 id="ergonomics-17"><a class="header" href="#ergonomics-17">Ergonomics</a></h3>
<p>N/A</p>
<h3 id="compatibility-16"><a class="header" href="#compatibility-16">Compatibility</a></h3>
<p>N/A</p>
<h2 id="prior-art-and-references-19"><a class="header" href="#prior-art-and-references-19">Prior Art and References</a></h2>
<ul>
<li><a href="https://github.com/polkadot-fellows/manifesto/blob/5e01eef15eded63f1db9be808b0f7c11bb9f4a12/manifesto.pdf">The Polkadot Fellowship
Manifesto</a></li>
<li><a href="https://data.oecd.org/earnwage/average-wages.htm#indicator-chart">OECD Average Wages</a></li>
<li><a href="https://www.indeed.com/career/engineer/salaries">Indeed: Average Salary for Engineers, United
States</a></li>
</ul>
<h2 id="unresolved-questions-19"><a class="header" href="#unresolved-questions-19">Unresolved Questions</a></h2>
<p>None at present.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/blob/main/text/0056-one-transaction-per-notification.md">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="approved/0056-one-transaction-per-notification.html#rfc-0056-enforce-only-one-transaction-per-notification">RFC-0056: Enforce only one transaction per notification</a>
<ul>
<li><a href="approved/0056-one-transaction-per-notification.html#summary">Summary</a></li>
<li><a href="approved/0056-one-transaction-per-notification.html#motivation">Motivation</a></li>
<li><a href="approved/0056-one-transaction-per-notification.html#stakeholders">Stakeholders</a></li>
<li><a href="approved/0056-one-transaction-per-notification.html#explanation">Explanation</a></li>
<li><a href="approved/0056-one-transaction-per-notification.html#drawbacks">Drawbacks</a></li>
<li><a href="approved/0056-one-transaction-per-notification.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="approved/0056-one-transaction-per-notification.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="approved/0056-one-transaction-per-notification.html#performance">Performance</a></li>
<li><a href="approved/0056-one-transaction-per-notification.html#ergonomics">Ergonomics</a></li>
<li><a href="approved/0056-one-transaction-per-notification.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="approved/0056-one-transaction-per-notification.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="approved/0056-one-transaction-per-notification.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="approved/0056-one-transaction-per-notification.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0056-enforce-only-one-transaction-per-notification"><a class="header" href="#rfc-0056-enforce-only-one-transaction-per-notification">RFC-0056: Enforce only one transaction per notification</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>2023-11-30</td></tr>
<tr><td><strong>Description</strong></td><td>Modify the transactions notifications protocol to always send only one transaction at a time</td></tr>
<tr><td><strong>Authors</strong></td><td>Pierre Krieger</td></tr>
</tbody></table>
</div>
<h2 id="summary-24"><a class="header" href="#summary-24">Summary</a></h2>
<p>When two peers connect to each other, they open (amongst other things) a so-called &quot;notifications protocol&quot; substream dedicated to gossiping transactions to each other.</p>
<p>Each notification on this substream currently consists in a SCALE-encoded <code>Vec&lt;Transaction&gt;</code> where <code>Transaction</code> is defined in the runtime.</p>
<p>This RFC proposes to modify the format of the notification to become <code>(Compact(1), Transaction)</code>. This maintains backwards compatibility, as this new format decodes as a <code>Vec</code> of length equal to 1.</p>
<h2 id="motivation-24"><a class="header" href="#motivation-24">Motivation</a></h2>
<p>There exists three motivations behind this change:</p>
<ul>
<li>
<p>It is technically impossible to decode a SCALE-encoded <code>Vec&lt;Transaction&gt;</code> into a list of SCALE-encoded transactions without knowing how to decode a <code>Transaction</code>. That's because a <code>Vec&lt;Transaction&gt;</code> consists in several <code>Transaction</code>s one after the other in memory, without any delimiter that indicates the end of a transaction and the start of the next. Unfortunately, the format of a <code>Transaction</code> is runtime-specific. This means that the code that receives notifications is necessarily tied to a specific runtime, and it is not possible to write runtime-agnostic code.</p>
</li>
<li>
<p>Notifications protocols are already designed to be optimized to send many items. Currently, when it comes to transactions, each item is a <code>Vec&lt;Transaction&gt;</code> that consists in multiple sub-items of type <code>Transaction</code>. This two-steps hierarchy is completely unnecessary, and was originally written at a time when the networking protocol of Substrate didn't have proper multiplexing.</p>
</li>
<li>
<p>It makes the implementation way more straight-forward by not having to repeat code related to back-pressure. See explanations below.</p>
</li>
</ul>
<h2 id="stakeholders-23"><a class="header" href="#stakeholders-23">Stakeholders</a></h2>
<p>Low-level developers.</p>
<h2 id="explanation-24"><a class="header" href="#explanation-24">Explanation</a></h2>
<p>To give an example, if you send one notification with three transactions, the bytes that are sent on the wire are:</p>
<pre><code>concat(
leb128(total-size-in-bytes-of-the-rest),
scale(compact(3)), scale(transaction1), scale(transaction2), scale(transaction3)
)
</code></pre>
<p>But you can also send three notifications of one transaction each, in which case it is:</p>
<pre><code>concat(
leb128(size(scale(transaction1)) + 1), scale(compact(1)), scale(transaction1),
leb128(size(scale(transaction2)) + 1), scale(compact(1)), scale(transaction2),
leb128(size(scale(transaction3)) + 1), scale(compact(1)), scale(transaction3)
)
</code></pre>
<p>Right now the sender can choose which of the two encoding to use. This RFC proposes to make the second encoding mandatory.</p>
<p>The format of the notification would become a SCALE-encoded <code>(Compact(1), Transaction)</code>.
A SCALE-compact encoded <code>1</code> is one byte of value <code>4</code>. In other words, the format of the notification would become <code>concat(&amp;[4], scale_encoded_transaction)</code>.
This is equivalent to forcing the <code>Vec&lt;Transaction&gt;</code> to always have a length of 1, and I expect the Substrate implementation to simply modify the sending side to add a <code>for</code> loop that sends one notification per item in the <code>Vec</code>.</p>
<p>As explained in the motivation section, this allows extracting <code>scale(transaction)</code> items without having to know how to decode them.</p>
<p>By &quot;flattening&quot; the two-steps hierarchy, an implementation only needs to back-pressure individual notifications rather than back-pressure notifications and transactions within notifications.</p>
<h2 id="drawbacks-19"><a class="header" href="#drawbacks-19">Drawbacks</a></h2>
<p>This RFC chooses to maintain backwards compatibility at the cost of introducing a very small wart (the <code>Compact(1)</code>).</p>
<p>An alternative could be to introduce a new version of the transactions notifications protocol that sends one <code>Transaction</code> per notification, but this is significantly more complicated to implement and can always be done later in case the <code>Compact(1)</code> is bothersome.</p>
<h2 id="testing-security-and-privacy-22"><a class="header" href="#testing-security-and-privacy-22">Testing, Security, and Privacy</a></h2>
<p>Irrelevant.</p>
<h2 id="performance-ergonomics-and-compatibility-22"><a class="header" href="#performance-ergonomics-and-compatibility-22">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-20"><a class="header" href="#performance-20">Performance</a></h3>
<p>Irrelevant.</p>
<h3 id="ergonomics-18"><a class="header" href="#ergonomics-18">Ergonomics</a></h3>
<p>Irrelevant.</p>
<h3 id="compatibility-17"><a class="header" href="#compatibility-17">Compatibility</a></h3>
<p>The change is backwards compatible if done in two steps: modify the sender to always send one transaction per notification, then, after a while, modify the receiver to enforce the new format.</p>
<h2 id="prior-art-and-references-20"><a class="header" href="#prior-art-and-references-20">Prior Art and References</a></h2>
<p>Irrelevant.</p>
<h2 id="unresolved-questions-20"><a class="header" href="#unresolved-questions-20">Unresolved Questions</a></h2>
<p>None.</p>
<h2 id="future-directions-and-related-material-17"><a class="header" href="#future-directions-and-related-material-17">Future Directions and Related Material</a></h2>
<p>None. This is a simple isolated change.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/blob/main/text/0059-nodes-capabilities-discovery.md">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="approved/0059-nodes-capabilities-discovery.html#rfc-0059-add-a-discovery-mechanism-for-nodes-based-on-their-capabilities">RFC-0059: Add a discovery mechanism for nodes based on their capabilities</a>
<ul>
<li><a href="approved/0059-nodes-capabilities-discovery.html#summary">Summary</a></li>
<li><a href="approved/0059-nodes-capabilities-discovery.html#motivation">Motivation</a></li>
<li><a href="approved/0059-nodes-capabilities-discovery.html#stakeholders">Stakeholders</a></li>
<li><a href="approved/0059-nodes-capabilities-discovery.html#explanation">Explanation</a>
<ul>
<li><a href="approved/0059-nodes-capabilities-discovery.html#capabilities">Capabilities</a></li>
<li><a href="approved/0059-nodes-capabilities-discovery.html#dht-provider-registration">DHT provider registration</a></li>
<li><a href="approved/0059-nodes-capabilities-discovery.html#secondary-dhts">Secondary DHTs</a></li>
<li><a href="approved/0059-nodes-capabilities-discovery.html#head-of-the-chain-providers">Head of the chain providers</a></li>
</ul>
</li>
<li><a href="approved/0059-nodes-capabilities-discovery.html#drawbacks">Drawbacks</a></li>
<li><a href="approved/0059-nodes-capabilities-discovery.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="approved/0059-nodes-capabilities-discovery.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="approved/0059-nodes-capabilities-discovery.html#performance">Performance</a></li>
<li><a href="approved/0059-nodes-capabilities-discovery.html#ergonomics">Ergonomics</a></li>
<li><a href="approved/0059-nodes-capabilities-discovery.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="approved/0059-nodes-capabilities-discovery.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="approved/0059-nodes-capabilities-discovery.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="approved/0059-nodes-capabilities-discovery.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0059-add-a-discovery-mechanism-for-nodes-based-on-their-capabilities"><a class="header" href="#rfc-0059-add-a-discovery-mechanism-for-nodes-based-on-their-capabilities">RFC-0059: Add a discovery mechanism for nodes based on their capabilities</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>2023-12-18</td></tr>
<tr><td><strong>Description</strong></td><td>Nodes having certain capabilities register themselves in the DHT to be discoverable</td></tr>
<tr><td><strong>Authors</strong></td><td>Pierre Krieger</td></tr>
</tbody></table>
</div>
<h2 id="summary-25"><a class="header" href="#summary-25">Summary</a></h2>
<p>This RFC proposes to make the mechanism of <a href="https://github.com/polkadot-fellows/RFCs/blob/main/text/0008-parachain-bootnodes-dht.md">RFC #8</a> more generic by introducing the concept of &quot;capabilities&quot;.</p>
<p>Implementations can implement certain &quot;capabilities&quot;, such as serving old block headers or being a parachain bootnode.</p>
<p>The discovery mechanism of RFC #8 is extended to be able to discover nodes of specific capabilities.</p>
<h2 id="motivation-25"><a class="header" href="#motivation-25">Motivation</a></h2>
<p>The Polkadot peer-to-peer network is made of nodes. Not all these nodes are equal. Some nodes store only the headers of recent blocks, some nodes store all the block headers and bodies since the genesis, some nodes store the storage of all blocks since the genesis, and so on.</p>
<p>It is currently not possible to know ahead of time (without connecting to it and asking) which nodes have which data available, and it is not easily possible to build a list of nodes that have a specific piece of data available.</p>
<p>If you want to download for example the header of block 500, you have to connect to a randomly-chosen node, ask it for block 500, and if it says that it doesn't have the block, disconnect and try another randomly-chosen node.
In certain situations such as downloading the storage of old blocks, nodes that have the information are relatively rare, and finding through trial and error a node that has the data can take a long time.</p>
<p>This RFC attempts to solve this problem by giving the possibility to build a list of nodes that are capable of serving specific data.</p>
<h2 id="stakeholders-24"><a class="header" href="#stakeholders-24">Stakeholders</a></h2>
<p>Low-level client developers.
People interested in accessing the archive of the chain.</p>
<h2 id="explanation-25"><a class="header" href="#explanation-25">Explanation</a></h2>
<p><em>Reading RFC #8 first might help with comprehension, as this RFC is very similar.</em></p>
<p>Please keep in mind while reading that everything below applies for both relay chains and parachains, except mentioned otherwise.</p>
<h3 id="capabilities"><a class="header" href="#capabilities">Capabilities</a></h3>
<p>This RFC defines a list of so-called <strong>capabilities</strong>:</p>
<ul>
<li><strong>Head of chain provider</strong>. An implementation with this capability must be able to serve to other nodes block headers, block bodies, justifications, calls proofs, and storage proofs of &quot;recent&quot; (see below) blocks, and, for relay chains, to serve to other nodes warp sync proofs where the starting block is a session change block and must participate in Grandpa and Beefy gossip.</li>
<li><strong>History provider</strong>. An implementation with this capability must be able to serve to other nodes block headers and block bodies of any block since the genesis, and must be able to serve to other nodes justifications of any session change block since the genesis up until and including their currently finalized block.</li>
<li><strong>Archive provider</strong>. This capability is a superset of <strong>History provider</strong>. In addition to the requirements of <strong>History provider</strong>, an implementation with this capability must be able to serve call proofs and storage proof requests of any block since the genesis up until and including their currently finalized block.</li>
<li><strong>Parachain bootnode</strong> (only for relay chains). An implementation with this capability must be able to serve the network request described in RFC 8.</li>
</ul>
<p>More capabilities might be added in the future.</p>
<p>In the context of the <em>head of chain provider</em>, the word &quot;recent&quot; means: any not-finalized-yet block that is equal to or an ancestor of a block that it has announced through a block announce, and any finalized block whose height is superior to its current finalized block minus <strong>16</strong>.
This does <em>not</em> include blocks that have been pruned because they're not a descendant of its current finalized block. In other words, blocks that aren't a descendant of the current finalized block can be thrown away.
A gap of blocks is required due to race conditions: when a node finalizes a block, it takes some time for its peers to be made aware of this, during which they might send requests concerning older blocks. The choice of the number of blocks in this gap is arbitrary.</p>
<p>Substrate is currently by default a <strong>head of chain provider</strong> provider. After it has finished warp syncing, it downloads the list of old blocks, after which it becomes a <strong>history provider</strong>.
If Substrate is instead configured as an archive node, then it downloads all blocks since the genesis and builds their state, after which it becomes an <strong>archive provider</strong>, <strong>history provider</strong>, and <strong>head of chain provider</strong>.
If blocks pruning is enabled and the chain is a relay chain, then Substrate unfortunately doesn't implement any of these capabilities, not even <strong>head of chain provider</strong>. This is considered as a bug that should be fixed, see <a href="https://github.com/paritytech/polkadot-sdk/issues/2733">https://github.com/paritytech/polkadot-sdk/issues/2733</a>.</p>
<h3 id="dht-provider-registration-1"><a class="header" href="#dht-provider-registration-1">DHT provider registration</a></h3>
<p>This RFC heavily relies on the functionalities of the Kademlia DHT already in use by Polkadot. You can find a link to the specification <a href="https://github.com/libp2p/specs/tree/master/kad-dht">here</a>.</p>
<p>Implementations that have the <strong>history provider</strong> capability should register themselves as providers under the key <code>sha256(concat(&quot;history&quot;, randomness))</code>.</p>
<p>Implementations that have the <strong>archive provider</strong> capability should register themselves as providers under the key <code>sha256(concat(&quot;archive&quot;, randomness))</code>.</p>
<p>Implementations that have the <strong>parachain bootnode</strong> capability should register themselves as provider under the key <code>sha256(concat(scale_compact(para_id), randomness))</code>, as described in RFC 8.</p>
<p>&quot;Register themselves as providers&quot; consists in sending <code>ADD_PROVIDER</code> requests to nodes close to the key, as described in <a href="https://github.com/libp2p/specs/tree/master/kad-dht#content-provider-advertisement">the <code>Content provider advertisement</code> section</a> of the specification.</p>
<p>The value of <code>randomness</code> can be found in the <code>randomness</code> field when calling the <code>BabeApi_currentEpoch</code> function.</p>
<p>In order to avoid downtimes when the key changes, nodes should also register themselves as a secondary key that uses a value of <code>randomness</code> equal to the <code>randomness</code> field when calling <code>BabeApi_nextEpoch</code>.</p>
<p>Implementers should be aware that their implementation of Kademlia might already hash the key before XOR'ing it. The key is not meant to be hashed twice.</p>
<p>Implementations must not register themselves if they don't fulfill the capability <em>yet</em>. For example, a node configured to be an archive node but that is still building its archive state in the background must register itself only after it has finished building its archive.</p>
<h3 id="secondary-dhts"><a class="header" href="#secondary-dhts">Secondary DHTs</a></h3>
<p>Implementations that have the <strong>history provider</strong> capability must also participate in a secondary DHT that comprises only of nodes with that capability. The protocol name of that secondary DHT must be <code>/&lt;genesis-hash&gt;/kad/history</code>.</p>
<p>Similarly, implementations that have the <strong>archive provider</strong> capability must also participate in a secondary DHT that comprises only of nodes with that capability and whose protocol name is <code>/&lt;genesis-hash&gt;/kad/archive</code>.</p>
<p>Just like implementations must not register themselves if they don't fulfill their capability yet, they must also not participate in the secondary DHT if they don't fulfill their capability yet.</p>
<h3 id="head-of-the-chain-providers"><a class="header" href="#head-of-the-chain-providers">Head of the chain providers</a></h3>
<p>Implementations that have the <strong>head of the chain provider</strong> capability do not register themselves as providers, but instead are the nodes that participate in the main DHT. In other words, they are the nodes that serve requests of the <code>/&lt;genesis_hash&gt;/kad</code> protocol.</p>
<p>Any implementation that isn't a head of the chain provider (read: light clients) must not participate in the main DHT. This is already presently the case.</p>
<p>Implementations must not participate in the main DHT if they don't fulfill the capability yet. For example, a node that is still in the process of warp syncing must not participate in the main DHT. However, assuming that warp syncing doesn't last more than a few seconds, it is acceptable to ignore this requirement in order to avoid complicating implementations too much.</p>
<h2 id="drawbacks-20"><a class="header" href="#drawbacks-20">Drawbacks</a></h2>
<p>None that I can see.</p>
<h2 id="testing-security-and-privacy-23"><a class="header" href="#testing-security-and-privacy-23">Testing, Security, and Privacy</a></h2>
<p><em>The content of this section is basically the same as the one in RFC 8.</em></p>
<p>This mechanism doesn't add or remove any security by itself, as it relies on existing mechanisms.</p>
<p>Due to the way Kademlia works, it would become the responsibility of the 20 Polkadot nodes whose <code>sha256(peer_id)</code> is closest to the <code>key</code> (described in the explanations section) to store the list of nodes that have specific capabilities.
Furthermore, when a large number of providers are registered, only the providers closest to the <code>key</code> are kept, up to a certain implementation-defined limit.</p>
<p>For this reason, an attacker can abuse this mechanism by randomly generating libp2p PeerIds until they find the 20 entries closest to the <code>key</code> representing the target capability. They are then in control of the list of nodes with that capability. While doing this can in no way be actually harmful, it could lead to eclipse attacks.</p>
<p>Because the key changes periodically and isn't predictable, and assuming that the Polkadot DHT is sufficiently large, it is not realistic for an attack like this to be maintained in the long term.</p>
<h2 id="performance-ergonomics-and-compatibility-23"><a class="header" href="#performance-ergonomics-and-compatibility-23">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-21"><a class="header" href="#performance-21">Performance</a></h3>
<p>The DHT mechanism generally has a low overhead, especially given that publishing providers is done only every 24 hours.</p>
<p>Doing a Kademlia iterative query then sending a provider record shouldn't take more than around 50 kiB in total of bandwidth for the parachain bootnode.</p>
<p>Assuming 1000 nodes with a specific capability, the 20 Polkadot full nodes corresponding to that capability will each receive a sudden spike of a few megabytes of networking traffic when the <code>key</code> rotates. Again, this is relatively negligible. If this becomes a problem, one can add a random delay before a node registers itself to be the provider of the <code>key</code> corresponding to <code>BabeApi_next_epoch</code>.</p>
<p>Maybe the biggest uncertainty is the traffic that the 20 Polkadot full nodes will receive from light clients that desire knowing the nodes with a capability. If this every becomes a problem, this value of 20 is an arbitrary constant that can be increased for more redundancy.</p>
<h3 id="ergonomics-19"><a class="header" href="#ergonomics-19">Ergonomics</a></h3>
<p>Irrelevant.</p>
<h3 id="compatibility-18"><a class="header" href="#compatibility-18">Compatibility</a></h3>
<p>Irrelevant.</p>
<h2 id="prior-art-and-references-21"><a class="header" href="#prior-art-and-references-21">Prior Art and References</a></h2>
<p>Unknown.</p>
<h2 id="unresolved-questions-21"><a class="header" href="#unresolved-questions-21">Unresolved Questions</a></h2>
<p>While it fundamentally doesn't change much to this RFC, using <code>BabeApi_currentEpoch</code> and <code>BabeApi_nextEpoch</code> might be inappropriate. I'm not familiar enough with good practices within the runtime to have an opinion here. Should it be an entirely new pallet?</p>
<h2 id="future-directions-and-related-material-18"><a class="header" href="#future-directions-and-related-material-18">Future Directions and Related Material</a></h2>
<p>This RFC would make it possible to reliably discover archive nodes, which would make it possible to reliably send archive node requests, something that isn't currently possible. This could solve the problem of finding archive RPC node providers by migrating archive-related request to using the native peer-to-peer protocol rather than JSON-RPC.</p>
<p>If we ever decide to break backwards compatibility, we could divide the &quot;history&quot; and &quot;archive&quot; capabilities in two, between nodes capable of serving older blocks and nodes capable of serving newer blocks.
We could even add to the peer-to-peer network nodes that are only capable of serving older blocks (by reading from a database) but do not participate in the head of the chain, and that just exist for historical purposes.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/4">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="stale/0004-remove-unnecessary-allocator-usage.html#rfc-0004-remove-the-host-side-runtime-memory-allocator">RFC-0004: Remove the host-side runtime memory allocator</a>
<ul>
<li><a href="stale/0004-remove-unnecessary-allocator-usage.html#summary">Summary</a></li>
<li><a href="stale/0004-remove-unnecessary-allocator-usage.html#motivation">Motivation</a></li>
<li><a href="stale/0004-remove-unnecessary-allocator-usage.html#stakeholders">Stakeholders</a></li>
<li><a href="stale/0004-remove-unnecessary-allocator-usage.html#explanation">Explanation</a>
<ul>
<li><a href="stale/0004-remove-unnecessary-allocator-usage.html#new-host-functions">New host functions</a></li>
<li><a href="stale/0004-remove-unnecessary-allocator-usage.html#other-changes">Other changes</a></li>
</ul>
</li>
<li><a href="stale/0004-remove-unnecessary-allocator-usage.html#drawbacks">Drawbacks</a></li>
<li><a href="stale/0004-remove-unnecessary-allocator-usage.html#prior-art">Prior Art</a></li>
<li><a href="stale/0004-remove-unnecessary-allocator-usage.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="stale/0004-remove-unnecessary-allocator-usage.html#future-possibilities">Future Possibilities</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0004-remove-the-host-side-runtime-memory-allocator"><a class="header" href="#rfc-0004-remove-the-host-side-runtime-memory-allocator">RFC-0004: Remove the host-side runtime memory allocator</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>2023-07-04</td></tr>
<tr><td><strong>Description</strong></td><td>Update the runtime-host interface to no longer make use of a host-side allocator</td></tr>
<tr><td><strong>Authors</strong></td><td>Pierre Krieger</td></tr>
</tbody></table>
</div>
<h2 id="summary-26"><a class="header" href="#summary-26">Summary</a></h2>
<p>Update the runtime-host interface to no longer make use of a host-side allocator.</p>
<h2 id="motivation-26"><a class="header" href="#motivation-26">Motivation</a></h2>
<p>The heap allocation of the runtime is currently controlled by the host using a memory allocator on the host side.</p>
<p>The API of many host functions consists in allocating a buffer. For example, when calling <code>ext_hashing_twox_256_version_1</code>, the host allocates a 32 bytes buffer using the host allocator, and returns a pointer to this buffer to the runtime. The runtime later has to call <code>ext_allocator_free_version_1</code> on this pointer in order to free the buffer.</p>
<p>Even though no benchmark has been done, it is pretty obvious that this design is very inefficient. To continue with the example of <code>ext_hashing_twox_256_version_1</code>, it would be more efficient to instead write the output hash to a buffer that was allocated by the runtime on its stack and passed by pointer to the function. Allocating a buffer on the stack in the worst case scenario simply consists in decreasing a number, and in the best case scenario is free. Doing so would save many Wasm memory reads and writes by the allocator, and would save a function call to <code>ext_allocator_free_version_1</code>.</p>
<p>Furthermore, the existence of the host-side allocator has become questionable over time. It is implemented in a very naive way, and for determinism and backwards compatibility reasons it needs to be implemented exactly identically in every client implementation. Runtimes make substantial use of heap memory allocations, and each allocation needs to go twice through the runtime &lt;-&gt; host boundary (once for allocating and once for freeing). Moving the allocator to the runtime side, while it would increase the size of the runtime, would be a good idea. But before the host-side allocator can be deprecated, all the host functions that make use of it need to be updated to not use it.</p>
<h2 id="stakeholders-25"><a class="header" href="#stakeholders-25">Stakeholders</a></h2>
<p>No attempt was made at convincing stakeholders.</p>
<h2 id="explanation-26"><a class="header" href="#explanation-26">Explanation</a></h2>
<h3 id="new-host-functions"><a class="header" href="#new-host-functions">New host functions</a></h3>
<p>This section contains a list of new host functions to introduce.</p>
<pre><code class="language-wat">(func $ext_storage_read_version_2
(param $key i64) (param $value_out i64) (param $offset i32) (result i64))
(func $ext_default_child_storage_read_version_2
(param $child_storage_key i64) (param $key i64) (param $value_out i64)
(param $offset i32) (result i64))
</code></pre>
<p>The signature and behaviour of <code>ext_storage_read_version_2</code> and <code>ext_default_child_storage_read_version_2</code> is identical to their version 1 counterparts, but the return value has a different meaning.
The new functions directly return the number of bytes that were written in the <code>value_out</code> buffer. If the entry doesn't exist, a value of <code>-1</code> is returned. Given that the host must never write more bytes than the size of the buffer in <code>value_out</code>, and that the size of this buffer is expressed as a 32 bits number, a 64bits value of <code>-1</code> is not ambiguous.</p>
<p>The runtime execution stops with an error if <code>value_out</code> is outside of the range of the memory of the virtual machine, even if the size of the buffer is 0 or if the amount of data to write would be 0 bytes.</p>
<pre><code class="language-wat">(func $ext_storage_next_key_version_2
(param $key i64) (param $out i64) (return i32))
(func $ext_default_child_storage_next_key_version_2
(param $child_storage_key i64) (param $key i64) (param $out i64) (return i32))
</code></pre>
<p>The behaviour of these functions is identical to their version 1 counterparts.
Instead of allocating a buffer, writing the next key to it, and returning a pointer to it, the new version of these functions accepts an <code>out</code> parameter containing <a href="https://spec.polkadot.network/chap-host-api#defn-runtime-pointer-size">a pointer-size</a> to the memory location where the host writes the output. The runtime execution stops with an error if <code>out</code> is outside of the range of the memory of the virtual machine, even if the function wouldn't write anything to <code>out</code>.
These functions return the size, in bytes, of the next key, or <code>0</code> if there is no next key. If the size of the next key is larger than the buffer in <code>out</code>, the bytes of the key that fit the buffer are written to <code>out</code> and any extra byte that doesn't fit is discarded.</p>
<p>Some notes:</p>
<ul>
<li>It is never possible for the next key to be an empty buffer, because an empty key has no preceding key. For this reason, a return value of <code>0</code> can unambiguously be used to indicate the lack of next key.</li>
<li>The <code>ext_storage_next_key_version_2</code> and <code>ext_default_child_storage_next_key_version_2</code> are typically used in order to enumerate keys that starts with a certain prefix. Given that storage keys are constructed by concatenating hashes, the runtime is expected to know the size of the next key and can allocate a buffer that can fit said key. When the next key doesn't belong to the desired prefix, it might not fit the buffer, but given that the start of the key is written to the buffer anyway this can be detected in order to avoid calling the function a second time with a larger buffer.</li>
</ul>
<pre><code class="language-wat">(func $ext_hashing_keccak_256_version_2
(param $data i64) (param $out i32))
(func $ext_hashing_keccak_512_version_2
(param $data i64) (param $out i32))
(func $ext_hashing_sha2_256_version_2
(param $data i64) (param $out i32))
(func $ext_hashing_blake2_128_version_2
(param $data i64) (param $out i32))
(func $ext_hashing_blake2_256_version_2
(param $data i64) (param $out i32))
(func $ext_hashing_twox_64_version_2
(param $data i64) (param $out i32))
(func $ext_hashing_twox_128_version_2
(param $data i64) (param $out i32))
(func $ext_hashing_twox_256_version_2
(param $data i64) (param $out i32))
(func $ext_trie_blake2_256_root_version_3
(param $data i64) (param $version i32) (param $out i32))
(func $ext_trie_blake2_256_ordered_root_version_3
(param $data i64) (param $version i32) (param $out i32))
(func $ext_trie_keccak_256_root_version_3
(param $data i64) (param $version i32) (param $out i32))
(func $ext_trie_keccak_256_ordered_root_version_3
(param $data i64) (param $version i32) (param $out i32))
(func $ext_default_child_storage_root_version_3
(param $child_storage_key i64) (param $out i32))
(func $ext_crypto_ed25519_generate_version_2
(param $key_type_id i32) (param $seed i64) (param $out i32))
(func $ext_crypto_sr25519_generate_version_2
(param $key_type_id i32) (param $seed i64) (param $out i32) (return i32))
(func $ext_crypto_ecdsa_generate_version_2
(param $key_type_id i32) (param $seed i64) (param $out i32) (return i32))
</code></pre>
<p>The behaviour of these functions is identical to their version 1 or version 2 counterparts. Instead of allocating a buffer, writing the output to it, and returning a pointer to it, the new version of these functions accepts an <code>out</code> parameter containing the memory location where the host writes the output. The output is always of a size known at compilation time. The runtime execution stops with an error if <code>out</code> is outside of the range of the memory of the virtual machine.</p>
<pre><code class="language-wat">(func $ext_default_child_storage_root_version_3
(param $child_storage_key i64) (param $out i32))
(func $ext_storage_root_version_3
(param $out i32))
</code></pre>
<p>The behaviour of these functions is identical to their version 1 and version 2 counterparts. Instead of allocating a buffer, writing the output to it, and returning a pointer to it, the new versions of these functions accepts an <code>out</code> parameter containing the memory location where the host writes the output. The output is always of a size known at compilation time. The runtime execution stops with an error if <code>out</code> is outside of the range of the memory of the virtual machine.</p>
<p>I have taken the liberty to take the version 1 of these functions as a base rather than the version 2, as a PPP deprecating the version 2 of these functions has previously been accepted: <a href="https://github.com/w3f/PPPs/pull/6">https://github.com/w3f/PPPs/pull/6</a>.</p>
<pre><code class="language-wat">(func $ext_storage_clear_prefix_version_3
(param $prefix i64) (param $limit i64) (param $removed_count_out i32)
(return i32))
(func $ext_default_child_storage_clear_prefix_version_3
(param $child_storage_key i64) (param $prefix i64)
(param $limit i64) (param $removed_count_out i32) (return i32))
(func $ext_default_child_storage_kill_version_4
(param $child_storage_key i64) (param $limit i64)
(param $removed_count_out i32) (return i32))
</code></pre>
<p>The behaviour of these functions is identical to their version 2 and 3 counterparts. Instead of allocating a buffer, writing the output to it, and returning a pointer to it, the version 3 and 4 of these functions accepts a <code>removed_count_out</code> parameter containing the memory location to a 8 bytes buffer where the host writes the number of keys that were removed in little endian. The runtime execution stops with an error if <code>removed_count_out</code> is outside of the range of the memory of the virtual machine. The functions return 1 to indicate that there are keys remaining, and 0 to indicate that all keys have been removed.</p>
<p>Note that there is an alternative proposal to add new host functions with the same names: <a href="https://github.com/w3f/PPPs/pull/7">https://github.com/w3f/PPPs/pull/7</a>. This alternative doesn't conflict with this one except for the version number. One proposal or the other will have to use versions 4 and 5 rather than 3 and 4.</p>
<pre><code class="language-wat">(func $ext_crypto_ed25519_sign_version_2
(param $key_type_id i32) (param $key i32) (param $msg i64) (param $out i32) (return i32))
(func $ext_crypto_sr25519_sign_version_2
(param $key_type_id i32) (param $key i32) (param $msg i64) (param $out i32) (return i32))
func $ext_crypto_ecdsa_sign_version_2
(param $key_type_id i32) (param $key i32) (param $msg i64) (param $out i32) (return i32))
(func $ext_crypto_ecdsa_sign_prehashed_version_2
(param $key_type_id i32) (param $key i32) (param $msg i64) (param $out i32) (return i64))
</code></pre>
<p>The behaviour of these functions is identical to their version 1 counterparts. The new versions of these functions accept an <code>out</code> parameter containing the memory location where the host writes the signature. The runtime execution stops with an error if <code>out</code> is outside of the range of the memory of the virtual machine, even if the function wouldn't write anything to <code>out</code>. The signatures are always of a size known at compilation time. On success, these functions return <code>0</code>. If the public key can't be found in the keystore, these functions return <code>1</code> and do not write anything to <code>out</code>.</p>
<p>Note that the return value is 0 on success and 1 on failure, while the previous version of these functions write 1 on success (as it represents a SCALE-encoded <code>Some</code>) and 0 on failure (as it represents a SCALE-encoded <code>None</code>). Returning 0 on success and non-zero on failure is consistent with common practices in the C programming language and is less surprising than the opposite.</p>
<pre><code class="language-wat">(func $ext_crypto_secp256k1_ecdsa_recover_version_3
(param $sig i32) (param $msg i32) (param $out i32) (return i64))
(func $ext_crypto_secp256k1_ecdsa_recover_compressed_version_3
(param $sig i32) (param $msg i32) (param $out i32) (return i64))
</code></pre>
<p>The behaviour of these functions is identical to their version 2 counterparts. The new versions of these functions accept an <code>out</code> parameter containing the memory location where the host writes the signature. The runtime execution stops with an error if <code>out</code> is outside of the range of the memory of the virtual machine, even if the function wouldn't write anything to <code>out</code>. The signatures are always of a size known at compilation time. On success, these functions return <code>0</code>. On failure, these functions return a non-zero value and do not write anything to <code>out</code>.</p>
<p>The non-zero value written on failure is:</p>
<ul>
<li>1: incorrect value of R or S</li>
<li>2: incorrect value of V</li>
<li>3: invalid signature</li>
</ul>
<p>These values are equal to the values returned on error by the version 2 (see <a href="https://spec.polkadot.network/chap-host-api#defn-ecdsa-verify-error">https://spec.polkadot.network/chap-host-api#defn-ecdsa-verify-error</a>), but incremented by 1 in order to reserve 0 for success.</p>
<pre><code class="language-wat">(func $ext_crypto_ed25519_num_public_keys_version_1
(param $key_type_id i32) (return i32))
(func $ext_crypto_ed25519_public_key_version_2
(param $key_type_id i32) (param $key_index i32) (param $out i32))
(func $ext_crypto_sr25519_num_public_keys_version_1
(param $key_type_id i32) (return i32))
(func $ext_crypto_sr25519_public_key_version_2
(param $key_type_id i32) (param $key_index i32) (param $out i32))
(func $ext_crypto_ecdsa_num_public_keys_version_1
(param $key_type_id i32) (return i32))
(func $ext_crypto_ecdsa_public_key_version_2
(param $key_type_id i32) (param $key_index i32) (param $out i32))
</code></pre>
<p>The functions superceded the <code>ext_crypto_ed25519_public_key_version_1</code>, <code>ext_crypto_sr25519_public_key_version_1</code>, and <code>ext_crypto_ecdsa_public_key_version_1</code> host functions.</p>
<p>Instead of calling <code>ext_crypto_ed25519_public_key_version_1</code> in order to obtain the list of all keys at once, the runtime should instead call <code>ext_crypto_ed25519_num_public_keys_version_1</code> in order to obtain the number of public keys available, then <code>ext_crypto_ed25519_public_key_version_2</code> repeatedly.
The <code>ext_crypto_ed25519_public_key_version_2</code> function writes the public key of the given <code>key_index</code> to the memory location designated by <code>out</code>. The <code>key_index</code> must be between 0 (included) and <code>n</code> (excluded), where <code>n</code> is the value returned by <code>ext_crypto_ed25519_num_public_keys_version_1</code>. Execution must trap if <code>n</code> is out of range.</p>
<p>The same explanations apply for <code>ext_crypto_sr25519_public_key_version_1</code> and <code>ext_crypto_ecdsa_public_key_version_1</code>.</p>
<p>Host implementers should be aware that the list of public keys (including their ordering) must not change while the runtime is running. This is most likely done by copying the list of all available keys either at the start of the execution or the first time the list is accessed.</p>
<pre><code class="language-wat">(func $ext_offchain_http_request_start_version_2
(param $method i64) (param $uri i64) (param $meta i64) (result i32))
</code></pre>
<p>The behaviour of this function is identical to its version 1 counterpart. Instead of allocating a buffer, writing the request identifier in it, and returning a pointer to it, the version 2 of this function simply returns the newly-assigned identifier to the HTTP request. On failure, this function returns <code>-1</code>. An identifier of <code>-1</code> is invalid and is reserved to indicate failure.</p>
<pre><code class="language-wat">(func $ext_offchain_http_request_write_body_version_2
(param $method i64) (param $uri i64) (param $meta i64) (result i32))
(func $ext_offchain_http_response_read_body_version_2
(param $request_id i32) (param $buffer i64) (param $deadline i64) (result i64))
</code></pre>
<p>The behaviour of these functions is identical to their version 1 counterpart. Instead of allocating a buffer, writing two bytes in it, and returning a pointer to it, the new version of these functions simply indicates what happened:</p>
<ul>
<li>For <code>ext_offchain_http_request_write_body_version_2</code>, 0 on success.</li>
<li>For <code>ext_offchain_http_response_read_body_version_2</code>, 0 or a non-zero number of bytes on success.</li>
<li>-1 if the deadline was reached.</li>
<li>-2 if there was an I/O error while processing the request.</li>
<li>-3 if the identifier of the request is invalid.</li>
</ul>
<p>These values are equal to the values returned on error by the version 1 (see <a href="https://spec.polkadot.network/chap-host-api#defn-http-error">https://spec.polkadot.network/chap-host-api#defn-http-error</a>), but tweaked in order to reserve positive numbers for success.</p>
<p>When it comes to <code>ext_offchain_http_response_read_body_version_2</code>, the host implementers must not read too much data at once in order to not create ambiguity in the returned value. Given that the size of the <code>buffer</code> is always inferior or equal to 4 GiB, this is not a problem.</p>
<pre><code class="language-wat">(func $ext_offchain_http_response_wait_version_2
(param $ids i64) (param $deadline i64) (param $out i32))
</code></pre>
<p>The behaviour of this function is identical to its version 1 counterpart. Instead of allocating a buffer, writing the output to it, and returning a pointer to it, the new version of this function accepts an <code>out</code> parameter containing the memory location where the host writes the output. The runtime execution stops with an error if <code>out</code> is outside of the range of the memory of the virtual machine.</p>
<p>The encoding of the response code is also modified compared to its version 1 counterpart and each response code now encodes to 4 little endian bytes as described below:</p>
<ul>
<li>100-999: the request has finished with the given HTTP status code.</li>
<li>-1 if the deadline was reached.</li>
<li>-2 if there was an I/O error while processing the request.</li>
<li>-3 if the identifier of the request is invalid.</li>
</ul>
<p>The buffer passed to <code>out</code> must always have a size of <code>4 * n</code> where <code>n</code> is the number of elements in the <code>ids</code>.</p>
<pre><code class="language-wat">(func $ext_offchain_http_response_header_name_version_1
(param $request_id i32) (param $header_index i32) (param $out i64) (result i64))
(func $ext_offchain_http_response_header_value_version_1
(param $request_id i32) (param $header_index i32) (param $out i64) (result i64))
</code></pre>
<p>These functions supercede the <code>ext_offchain_http_response_headers_version_1</code> host function.</p>
<p>Contrary to <code>ext_offchain_http_response_headers_version_1</code>, only one header indicated by <code>header_index</code> can be read at a time. Instead of calling <code>ext_offchain_http_response_headers_version_1</code> once, the runtime should call <code>ext_offchain_http_response_header_name_version_1</code> and <code>ext_offchain_http_response_header_value_version_1</code> multiple times with an increasing <code>header_index</code>, until a value of <code>-1</code> is returned.</p>
<p>These functions accept an <code>out</code> parameter containing <a href="https://spec.polkadot.network/chap-host-api#defn-runtime-pointer-size">a pointer-size</a> to the memory location where the header name or value should be written. The runtime execution stops with an error if <code>out</code> is outside of the range of the memory of the virtual machine, even if the function wouldn't write anything to <code>out</code>.</p>
<p>These functions return the size, in bytes, of the header name or header value. If request doesn't exist or is in an invalid state (as documented for <code>ext_offchain_http_response_headers_version_1</code>) or the <code>header_index</code> is out of range, a value of <code>-1</code> is returned. Given that the host must never write more bytes than the size of the buffer in <code>out</code>, and that the size of this buffer is expressed as a 32 bits number, a 64bits value of <code>-1</code> is not ambiguous.</p>
<p>If the buffer in <code>out</code> is too small to fit the entire header name of value, only the bytes that fit are written and the rest are discarded.</p>
<pre><code class="language-wat">(func $ext_offchain_submit_transaction_version_2
(param $data i64) (return i32))
(func $ext_offchain_http_request_add_header_version_2
(param $request_id i32) (param $name i64) (param $value i64) (result i32))
</code></pre>
<p>Instead of allocating a buffer, writing <code>1</code> or <code>0</code> in it, and returning a pointer to it, the version 2 of these functions return <code>0</code> or <code>1</code>, where <code>0</code> indicates success and <code>1</code> indicates failure. The runtime must interpret any non-<code>0</code> value as failure, but the client must always return <code>1</code> in case of failure.</p>
<pre><code class="language-wat">(func $ext_offchain_local_storage_read_version_1
(param $kind i32) (param $key i64) (param $value_out i64) (param $offset i32) (result i64))
</code></pre>
<p>This function supercedes the <code>ext_offchain_local_storage_get_version_1</code> host function, and uses an API and logic similar to <code>ext_storage_read_version_2</code>.</p>
<p>It reads the offchain local storage key indicated by <code>kind</code> and <code>key</code> starting at the byte indicated by <code>offset</code>, and writes the value to the <a href="https://spec.polkadot.network/chap-host-api#defn-runtime-pointer-size">pointer-size</a> indicated by <code>value_out</code>.</p>
<p>The function returns the number of bytes that were written in the <code>value_out</code> buffer. If the entry doesn't exist, a value of <code>-1</code> is returned. Given that the host must never write more bytes than the size of the buffer in <code>value_out</code>, and that the size of this buffer is expressed as a 32 bits number, a 64bits value of <code>-1</code> is not ambiguous.</p>
<p>The runtime execution stops with an error if <code>value_out</code> is outside of the range of the memory of the virtual machine, even if the size of the buffer is 0 or if the amount of data to write would be 0 bytes.</p>
<pre><code class="language-wat">(func $ext_offchain_network_peer_id_version_1
(param $out i64))
</code></pre>
<p>This function writes <a href="https://spec.polkadot.network/chap-networking#id-node-identities">the <code>PeerId</code> of the local node</a> to the memory location indicated by <code>out</code>. A <code>PeerId</code> is always 38 bytes long.
The runtime execution stops with an error if <code>out</code> is outside of the range of the memory of the virtual machine.</p>
<pre><code class="language-wat">(func $ext_input_size_version_1
(return i64))
(func $ext_input_read_version_1
(param $offset i64) (param $out i64))
</code></pre>
<p>When a runtime function is called, the host uses the allocator to allocate memory within the runtime where to write some input data. These two new host functions provide an alternative way to access the input that doesn't make use of the allocator.</p>
<p>The <code>ext_input_size_version_1</code> host function returns the size in bytes of the input data.</p>
<p>The <code>ext_input_read_version_1</code> host function copies some data from the input data to the memory of the runtime. The <code>offset</code> parameter indicates the offset within the input data where to start copying, and must be inferior or equal to the value returned by <code>ext_input_size_version_1</code>. The <code>out</code> parameter is <a href="https://spec.polkadot.network/chap-host-api#defn-runtime-pointer-size">a pointer-size</a> containing the buffer where to write to.
The runtime execution stops with an error if <code>offset</code> is strictly superior to the size of the input data, or if <code>out</code> is outside of the range of the memory of the virtual machine, even if the amount of data to copy would be 0 bytes.</p>
<h3 id="other-changes"><a class="header" href="#other-changes">Other changes</a></h3>
<p>In addition to the new host functions, this RFC proposes two changes to the runtime-host interface:</p>
<ul>
<li>The following function signature is now also accepted for runtime entry points: <code>(func (result i64))</code>.</li>
<li>Runtimes no longer need to expose a constant named <code>__heap_base</code>.</li>
</ul>
<p>All the host functions that are being superceded by new host functions are now considered deprecated and should no longer be used.
The following other host functions are similarly also considered deprecated:</p>
<ul>
<li><code>ext_storage_get_version_1</code></li>
<li><code>ext_default_child_storage_get_version_1</code></li>
<li><code>ext_allocator_malloc_version_1</code></li>
<li><code>ext_allocator_free_version_1</code></li>
<li><code>ext_offchain_network_state_version_1</code></li>
</ul>
<h2 id="drawbacks-21"><a class="header" href="#drawbacks-21">Drawbacks</a></h2>
<p>This RFC might be difficult to implement in Substrate due to the internal code design. It is not clear to the author of this RFC how difficult it would be.</p>
<h2 id="prior-art-1"><a class="header" href="#prior-art-1">Prior Art</a></h2>
<p>The API of these new functions was heavily inspired by API used by the C programming language.</p>
<h2 id="unresolved-questions-22"><a class="header" href="#unresolved-questions-22">Unresolved Questions</a></h2>
<p>The changes in this RFC would need to be benchmarked. This involves implementing the RFC and measuring the speed difference.</p>
<p>It is expected that most host functions are faster or equal speed to their deprecated counterparts, with the following exceptions:</p>
<ul>
<li>
<p><code>ext_input_size_version_1</code>/<code>ext_input_read_version_1</code> is inherently slower than obtaining a buffer with the entire data due to the two extra function calls and the extra copying. However, given that this only happens once per runtime call, the cost is expected to be negligible.</p>
</li>
<li>
<p>The <code>ext_crypto_*_public_keys</code>, <code>ext_offchain_network_state</code>, and <code>ext_offchain_http_*</code> host functions are likely slightly slower than their deprecated counterparts, but given that they are used only in offchain workers this is acceptable.</p>
</li>
<li>
<p>It is unclear how replacing <code>ext_storage_get</code> with <code>ext_storage_read</code> and <code>ext_default_child_storage_get</code> with <code>ext_default_child_storage_read</code> will impact performances.</p>
</li>
<li>
<p>It is unclear how the changes to <code>ext_storage_next_key</code> and <code>ext_default_child_storage_next_key</code> will impact performances.</p>
</li>
</ul>
<h2 id="future-possibilities"><a class="header" href="#future-possibilities">Future Possibilities</a></h2>
<p>After this RFC, we can remove from the source code of the host the allocator altogether in a future version, by removing support for all the deprecated host functions.
This would remove the possibility to synchronize older blocks, which is probably controversial and requires a some preparations that are out of scope of this RFC.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/6">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="stale/0006-dynamic-pricing-for-bulk-coretime-sales.html#rfc-0006-dynamic-pricing-for-bulk-coretime-sales">RFC-0006: Dynamic Pricing for Bulk Coretime Sales</a>
<ul>
<li><a href="stale/0006-dynamic-pricing-for-bulk-coretime-sales.html#summary">Summary</a></li>
<li><a href="stale/0006-dynamic-pricing-for-bulk-coretime-sales.html#motivation">Motivation</a>
<ul>
<li><a href="stale/0006-dynamic-pricing-for-bulk-coretime-sales.html#requirements">Requirements</a></li>
</ul>
</li>
<li><a href="stale/0006-dynamic-pricing-for-bulk-coretime-sales.html#stakeholders">Stakeholders</a></li>
<li><a href="stale/0006-dynamic-pricing-for-bulk-coretime-sales.html#explanation">Explanation</a>
<ul>
<li><a href="stale/0006-dynamic-pricing-for-bulk-coretime-sales.html#overview">Overview</a></li>
<li><a href="stale/0006-dynamic-pricing-for-bulk-coretime-sales.html#parameters">Parameters</a></li>
<li><a href="stale/0006-dynamic-pricing-for-bulk-coretime-sales.html#function">Function</a></li>
<li><a href="stale/0006-dynamic-pricing-for-bulk-coretime-sales.html#pseudo-code">Pseudo-code</a></li>
<li><a href="stale/0006-dynamic-pricing-for-bulk-coretime-sales.html#properties-of-the-curve">Properties of the Curve</a></li>
<li><a href="stale/0006-dynamic-pricing-for-bulk-coretime-sales.html#example-configurations">Example Configurations</a></li>
</ul>
</li>
<li><a href="stale/0006-dynamic-pricing-for-bulk-coretime-sales.html#drawbacks">Drawbacks</a></li>
<li><a href="stale/0006-dynamic-pricing-for-bulk-coretime-sales.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="stale/0006-dynamic-pricing-for-bulk-coretime-sales.html#future-possibilities">Future Possibilities</a></li>
<li><a href="stale/0006-dynamic-pricing-for-bulk-coretime-sales.html#references">References</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0006-dynamic-pricing-for-bulk-coretime-sales"><a class="header" href="#rfc-0006-dynamic-pricing-for-bulk-coretime-sales">RFC-0006: Dynamic Pricing for Bulk Coretime Sales</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>July 09, 2023</td></tr>
<tr><td><strong>Description</strong></td><td>A dynamic pricing model to adapt the regular price for bulk coretime sales</td></tr>
<tr><td><strong>Authors</strong></td><td>Tommi Enenkel (Alice und Bob)</td></tr>
<tr><td><strong>License</strong></td><td>MIT</td></tr>
</tbody></table>
</div>
<h2 id="summary-27"><a class="header" href="#summary-27">Summary</a></h2>
<p>This RFC proposes a dynamic pricing model for the sale of Bulk Coretime on the Polkadot UC. The proposed model updates the regular price of cores for each sale period, by taking into account the number of cores sold in the previous sale, as well as a limit of cores and a target number of cores sold. It ensures a minimum price and limits price growth to a maximum price increase factor, while also giving govenance control over the steepness of the price change curve. It allows governance to address challenges arising from changing market conditions and should offer predictable and controlled price adjustments.</p>
<p>Accompanying visualizations are provided at [1].</p>
<h2 id="motivation-27"><a class="header" href="#motivation-27">Motivation</a></h2>
<p>RFC-1 proposes periodic Bulk Coretime Sales as a mechanism to sell continouos regions of blockspace (suggested to be 4 weeks in length). A number of Blockspace Regions (compare RFC-1 &amp; RFC-3) are provided for sale to the Broker-Chain each period and shall be sold in a way that provides value-capture for the Polkadot network. The exact pricing mechanism is out of scope for RFC-1 and shall be provided by this RFC. </p>
<p>A dynamic pricing model is needed. A limited number of Regions are offered for sale each period. The model needs to find the price for a period based on supply and demand of the previous period.</p>
<p>The model shall give Coretime consumers predictability about upcoming price developments and confidence that Polkadot governance can adapt the pricing model to changing market conditions.</p>
<h3 id="requirements-7"><a class="header" href="#requirements-7">Requirements</a></h3>
<ol>
<li>The solution SHOULD provide a dynamic pricing model that increases price with growing demand and reduces price with shrinking demand.</li>
<li>The solution SHOULD have a slow rate of change for price if the number of Regions sold is close to a given sales target and increase the rate of change as the number of sales deviates from the target.</li>
<li>The solution SHOULD provide the possibility to always have a minimum price per Region.</li>
<li>The solution SHOULD provide a maximum factor of price increase should the limit of Regions sold per period be reached.</li>
<li>The solution should allow governance to control the steepness of the price function</li>
</ol>
<h2 id="stakeholders-26"><a class="header" href="#stakeholders-26">Stakeholders</a></h2>
<p>The primary stakeholders of this RFC are:</p>
<ul>
<li>Protocol researchers and evelopers</li>
<li>Polkadot DOT token holders</li>
<li>Polkadot parachains teams</li>
<li>Brokers involved in the trade of Bulk Coretime</li>
</ul>
<h2 id="explanation-27"><a class="header" href="#explanation-27">Explanation</a></h2>
<h3 id="overview-1"><a class="header" href="#overview-1">Overview</a></h3>
<p>The dynamic pricing model sets the new price based on supply and demand in the previous period. The model is a function of the number of Regions sold, piecewise-defined by two power functions.</p>
<ul>
<li>The left side ranges from 0 to the target. It represents situations where demand was lower than the target.</li>
<li>The right side ranges from the target to limit. It represents situations where demand was higher than the target.</li>
</ul>
<p>The curve of the function forms a plateau around the target and then falls off to the left and rises up to the right. The shape of the plateau can be controlled via a scale factor for the left side and right side of the function respectively.</p>
<h3 id="parameters-1"><a class="header" href="#parameters-1">Parameters</a></h3>
<p>From here on, we will also refer to Regions sold as 'cores' to stay congruent with RFC-1.</p>
<div class="table-wrapper"><table><thead><tr><th>Name</th><th>Suggested Value</th><th>Description</th><th>Constraints</th></tr></thead><tbody>
<tr><td><code>BULK_LIMIT</code></td><td>45</td><td>The maximum number of cores being sold</td><td><code>0 &lt; BULK_LIMIT</code></td></tr>
<tr><td><code>BULK_TARGET</code></td><td>30</td><td>The target number of cores being sold</td><td><code>0 &lt; BULK_TARGET &lt;= BULK_LIMIT</code></td></tr>
<tr><td><code>MIN_PRICE</code></td><td>1</td><td>The minimum price a core will always cost.</td><td><code>0 &lt; MIN_PRICE</code></td></tr>
<tr><td><code>MAX_PRICE_INCREASE_FACTOR</code></td><td>2</td><td>The maximum factor by which the price can change.</td><td><code>1 &lt; MAX_PRICE_INCREASE_FACTOR</code></td></tr>
<tr><td><code>SCALE_DOWN</code></td><td>2</td><td>The steepness of the left side of the function.</td><td><code>0 &lt; SCALE_DOWN</code></td></tr>
<tr><td><code>SCALE_UP</code></td><td>2</td><td>The steepness of the right side of the function.</td><td><code>0 &lt; SCALE_UP</code></td></tr>
</tbody></table>
</div>
<h3 id="function"><a class="header" href="#function">Function</a></h3>
<pre><code class="language-math">P(n) = \begin{cases}
(P_{\text{old}} - P_{\text{min}}) \left(1 - \left(\frac{T - n}{T}\right)^d\right) + P_{\text{min}} &amp; \text{if } n \leq T \\
((F - 1) \cdot P_{\text{old}} \cdot \left(\frac{n - T}{L - T}\right)^u) + P_{\text{old}} &amp; \text{if } n &gt; T
\end{cases}
</code></pre>
<ul>
<li>$P_{\text{old}}$ is the <code>old_price</code>, the price of a core in the previous period.</li>
<li>$P_{\text{min}}$ is the <code>MIN_PRICE</code>, the minimum price a core will always cost.</li>
<li>$F$ is the <code>MAX_PRICE_INCREASE_FACTOR</code>, the factor by which the price maximally can change from one period to another.</li>
<li>$d$ is the <code>SCALE_DOWN</code>, the steepness of the left side of the function.</li>
<li>$u$ is the <code>SCALE_UP</code>, the steepness of the right side of the function.</li>
<li>$T$ is the <code>BULK_TARGET</code>, the target number of cores being sold.</li>
<li>$L$ is the <code>BULK_LIMIT</code>, the maximum number of cores being sold.</li>
<li>$n$ is <code>cores_sold</code>, the number of cores being sold.</li>
</ul>
<h4 id="left-side"><a class="header" href="#left-side">Left side</a></h4>
<p>The left side is a power function that describes an increasing concave downward curvature that approaches <code>old_price</code>. We realize this by using the form $y = a(1 - x^d)$, usually used as a downward sloping curve, but in our case flipped horizontally by letting the argument $x = \frac{T-n}{T}$ decrease with $n$, doubly inversing the curve.</p>
<p>This approach is chosen over a decaying exponential because it let's us a better control the shape of the plateau, especially allowing us to get a straight line by setting <code>SCALE_DOWN</code> to $1$.</p>
<h4 id="ride-side"><a class="header" href="#ride-side">Ride side</a></h4>
<p>The right side is a power function of the form $y = a(x^u)$.</p>
<h3 id="pseudo-code"><a class="header" href="#pseudo-code">Pseudo-code</a></h3>
<pre><code>NEW_PRICE := IF CORES_SOLD &lt;= BULK_TARGET THEN
(OLD_PRICE - MIN_PRICE) * (1 - ((BULK_TARGET - CORES_SOLD)^SCALE_DOWN / BULK_TARGET^SCALE_DOWN)) + MIN_PRICE
ELSE
((MAX_PRICE_INCREASE_FACTOR - 1) * OLD_PRICE * ((CORES_SOLD - BULK_TARGET)^SCALE_UP / (BULK_LIMIT - BULK_TARGET)^SCALE_UP)) + OLD_PRICE
END IF
</code></pre>
<h3 id="properties-of-the-curve"><a class="header" href="#properties-of-the-curve">Properties of the Curve</a></h3>
<h4 id="minimum-price"><a class="header" href="#minimum-price">Minimum Price</a></h4>
<p>We introduce <code>MIN_PRICE</code> to control the minimum price.</p>
<p>The left side of the function shall be allowed to come close to 0 if cores sold approaches 0. The rationale is that if there are actually 0 cores sold, the previous sale price was too high and the price needs to adapt quickly.</p>
<h4 id="price-forms-a-plateau-around-the-target"><a class="header" href="#price-forms-a-plateau-around-the-target">Price forms a plateau around the target</a></h4>
<p>If the number of cores is close to <code>BULK_TARGET</code>, less extreme price changes might be sensible. This ensures that a drop in sold cores or an increase doesn’t lead to immediate price changes, but rather slowly adapts. Only if more extreme changes in the number of sold cores occur, does the price slope increase.</p>
<p>We introduce <code>SCALE_DOWN</code> and <code>SCALE_UP</code> to control for the steepness of the left and the right side of the function respectively.</p>
<h4 id="max-price-increase-factor"><a class="header" href="#max-price-increase-factor">Max price increase factor</a></h4>
<p>We introduce <code>MAX_PRICE_INCREASE_FACTOR</code> as the factor that controls how much the price may increase from one period to another.</p>
<p>Introducing this variable gives governance an additional control lever and avoids the necessity for a future runtime upgrade.</p>
<h3 id="example-configurations"><a class="header" href="#example-configurations">Example Configurations</a></h3>
<h4 id="baseline"><a class="header" href="#baseline">Baseline</a></h4>
<p>This example proposes the baseline parameters. If not mentioned otherwise, other examples use these values. </p>
<p>The minimum price of a core is 1 DOT, the price can double every 4 weeks. Price change around <code>BULK_TARGET</code> is dampened slightly.</p>
<pre><code>BULK_TARGET = 30
BULK_LIMIT = 45
MIN_PRICE = 1
MAX_PRICE_INCREASE_FACTOR = 2
SCALE_DOWN = 2
SCALE_UP = 2
OLD_PRICE = 1000
</code></pre>
<h4 id="more-aggressive-pricing"><a class="header" href="#more-aggressive-pricing">More aggressive pricing</a></h4>
<p>We might want to have a more aggressive price growth, allowing the price to triple every 4 weeks and have a linear increase in price on the right side.</p>
<pre><code>BULK_TARGET = 30
BULK_LIMIT = 45
MIN_PRICE = 1
MAX_PRICE_INCREASE_FACTOR = 3
SCALE_DOWN = 2
SCALE_UP = 1
OLD_PRICE = 1000
</code></pre>
<h4 id="conservative-pricing-to-ensure-quick-corrections-in-an-affluent-market"><a class="header" href="#conservative-pricing-to-ensure-quick-corrections-in-an-affluent-market">Conservative pricing to ensure quick corrections in an affluent market</a></h4>
<p>If governance considers the risk that a sudden surge in DOT price might price chains out from bulk coretime markets, it can ensure the model quickly reacts to a quick drop in demand, by setting 0 &lt; SCALE_DOWN &lt; 1 and setting the max price increase factor more conservatively.</p>
<pre><code>BULK_TARGET = 30
BULK_LIMIT = 45
MIN_PRICE = 1
MAX_PRICE_INCREASE_FACTOR = 1.5
SCALE_DOWN = 0.5
SCALE_UP = 2
OLD_PRICE = 1000
</code></pre>
<h4 id="linear-pricing"><a class="header" href="#linear-pricing">Linear pricing</a></h4>
<p>By setting the scaling factors to 1 and potentially adapting the max price increase, we can achieve a linear function</p>
<pre><code>BULK_TARGET = 30
BULK_LIMIT = 45
MIN_PRICE = 1
MAX_PRICE_INCREASE_FACTOR = 1.5
SCALE_DOWN = 1
SCALE_UP = 1
OLD_PRICE = 1000
</code></pre>
<h2 id="drawbacks-22"><a class="header" href="#drawbacks-22">Drawbacks</a></h2>
<p>None at present.</p>
<h2 id="prior-art-and-references-22"><a class="header" href="#prior-art-and-references-22">Prior Art and References</a></h2>
<p>This pricing model is based on the requirements from the basic linear solution proposed in RFC-1, which is a simple dynamic pricing model and only used as proof. The present model adds additional considerations to make the model more adaptable under real conditions. </p>
<h2 id="future-possibilities-1"><a class="header" href="#future-possibilities-1">Future Possibilities</a></h2>
<p>This RFC, if accepted, shall be implemented in conjunction with RFC-1.</p>
<h2 id="references-2"><a class="header" href="#references-2">References</a></h2>
<ul>
<li>[1] Polkadot forum post with visualizations: <a href="https://forum.polkadot.network/t/dynamic-pricing-for-bulk-coretime-sales/3359">Dynamic Pricing for Bulk Coretime Sales</a></li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/9">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="stale/0009-improved-net-light-client-requests.html#rfc-0009-improved-light-client-requests-networking-protocol">RFC-0009: Improved light client requests networking protocol</a>
<ul>
<li><a href="stale/0009-improved-net-light-client-requests.html#summary">Summary</a></li>
<li><a href="stale/0009-improved-net-light-client-requests.html#motivation">Motivation</a></li>
<li><a href="stale/0009-improved-net-light-client-requests.html#stakeholders">Stakeholders</a></li>
<li><a href="stale/0009-improved-net-light-client-requests.html#explanation">Explanation</a></li>
<li><a href="stale/0009-improved-net-light-client-requests.html#drawbacks">Drawbacks</a></li>
<li><a href="stale/0009-improved-net-light-client-requests.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="stale/0009-improved-net-light-client-requests.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="stale/0009-improved-net-light-client-requests.html#performance">Performance</a></li>
<li><a href="stale/0009-improved-net-light-client-requests.html#ergonomics">Ergonomics</a></li>
<li><a href="stale/0009-improved-net-light-client-requests.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="stale/0009-improved-net-light-client-requests.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="stale/0009-improved-net-light-client-requests.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="stale/0009-improved-net-light-client-requests.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0009-improved-light-client-requests-networking-protocol"><a class="header" href="#rfc-0009-improved-light-client-requests-networking-protocol">RFC-0009: Improved light client requests networking protocol</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>2023-07-19</td></tr>
<tr><td><strong>Description</strong></td><td>Modify the networking storage read requests to solve some problems with the existing one</td></tr>
<tr><td><strong>Authors</strong></td><td>Pierre Krieger</td></tr>
</tbody></table>
</div>
<h2 id="summary-28"><a class="header" href="#summary-28">Summary</a></h2>
<p>Improve the networking messages that query storage items from the remote, in order to reduce the bandwidth usage and number of round trips of light clients.</p>
<h2 id="motivation-28"><a class="header" href="#motivation-28">Motivation</a></h2>
<p>Clients on the Polkadot peer-to-peer network can be divided into two categories: full nodes and light clients. So-called full nodes are nodes that store the content of the chain locally on their disk, while light clients are nodes that don't. In order to access for example the balance of an account, a full node can do a disk read, while a light client needs to send a network message to a full node and wait for the full node to reply with the desired value. This reply is in the form of a Merkle proof, which makes it possible for the light client to verify the exactness of the value.</p>
<p>Unfortunately, this network protocol is suffering from some issues:</p>
<ul>
<li>It is not possible for the querier to check whether a key exists in the storage of the chain except by querying the value of that key. The reply will thus include the value of the key, only for that value to be discarded by the querier that isn't interested by it. This is a waste of bandwidth.</li>
<li>It is not possible for the querier to know whether a value in the storage of the chain has been modified between two blocks except by querying this value for both blocks and comparing them. Only a few storage values get modified in a block, and thus most of the time the comparison will be equal. This leads to a waste of bandwidth as the values have to be transferred.</li>
<li>While it is possible to ask for multiple specific storage keys at the same time, it is not possible to ask for a list of keys that start with a certain prefix. Due to the way FRAME works, storage keys are grouped by &quot;prefix&quot;, for example all account balances start with the same prefix. It is thus a common necessity for a light client to obtain the list of all keys (and possibly their values) that start with a specific prefix. This is currently not possible except by performing multiple queries serially that &quot;walk down&quot; the trie.</li>
</ul>
<p>Once Polkadot and Kusama will have transitioned to <code>state_version = 1</code>, which modifies the format of the trie entries, it will be possible to generate Merkle proofs that contain only the hashes of values in the storage. Thanks to this, it is already possible to prove the existence of a key without sending its entire value (only its hash), or to prove that a value has changed or not between two blocks (by sending just their hashes).
Thus, the only reason why aforementioned issues exist is because the existing networking messages don't give the possibility for the querier to query this. This is what this proposal aims at fixing.</p>
<h2 id="stakeholders-27"><a class="header" href="#stakeholders-27">Stakeholders</a></h2>
<p>This is the continuation of https://github.com/w3f/PPPs/pull/10, which itself is the continuation of https://github.com/w3f/PPPs/pull/5.</p>
<h2 id="explanation-28"><a class="header" href="#explanation-28">Explanation</a></h2>
<p>The protobuf schema of the networking protocol can be found here: https://github.com/paritytech/substrate/blob/5b6519a7ff4a2d3cc424d78bc4830688f3b184c0/client/network/light/src/schema/light.v1.proto</p>
<p>The proposal is to modify this protocol in this way:</p>
<pre><code class="language-diff">@@ -11,6 +11,7 @@ message Request {
RemoteReadRequest remote_read_request = 2;
RemoteReadChildRequest remote_read_child_request = 4;
// Note: ids 3 and 5 were used in the past. It would be preferable to not re-use them.
+ RemoteReadRequestV2 remote_read_request_v2 = 6;
}
}
@@ -48,6 +49,21 @@ message RemoteReadRequest {
repeated bytes keys = 3;
}
+message RemoteReadRequestV2 {
+ required bytes block = 1;
+ optional ChildTrieInfo child_trie_info = 2; // Read from the main trie if missing.
+ repeated Key keys = 3;
+ optional bytes onlyKeysAfter = 4;
+ optional bool onlyKeysAfterIgnoreLastNibble = 5;
+}
+
+message ChildTrieInfo {
+ enum ChildTrieNamespace {
+ DEFAULT = 1;
+ }
+
+ required bytes hash = 1;
+ required ChildTrieNamespace namespace = 2;
+}
+
// Remote read response.
message RemoteReadResponse {
// Read proof. If missing, indicates that the remote couldn't answer, for example because
@@ -65,3 +81,8 @@ message RemoteReadChildRequest {
// Storage keys.
repeated bytes keys = 6;
}
+
+message Key {
+ required bytes key = 1;
+ optional bool skipValue = 2; // Defaults to `false` if missing
+ optional bool includeDescendants = 3; // Defaults to `false` if missing
+}
</code></pre>
<p>Note that the field names aren't very important as they are not sent over the wire. They can be changed at any time without any consequence. I would invite people to not discuss these field names as they are implementation details.</p>
<p>This diff adds a new type of request (<code>RemoteReadRequestV2</code>).</p>
<p>The new <code>child_trie_info</code> field in the request makes it possible to specify which trie is concerned by the request. The current networking protocol uses two different structs (<code>RemoteReadRequest</code> and <code>RemoteReadChildRequest</code>) for main trie and child trie queries, while this new request would make it possible to query either. This change doesn't fix any of the issues mentioned in the previous section, but is a side change that has been done for simplicity.
An alternative could have been to specify the <code>child_trie_info</code> for each individual <code>Key</code>. However this would make it necessary to send the child trie hash many times over the network, which leads to a waste of bandwidth, and in my opinion makes things more complicated for no actual gain. If a querier would like to access more than one trie at the same time, it is always possible to send one query per trie.</p>
<p>If <code>skipValue</code> is <code>true</code> for a <code>Key</code>, then the value associated with this key isn't important to the querier, and the replier is encouraged to replace the value with its hash provided that the storage item has a <code>state_version</code> equal to 1. If the storage value has a <code>state_version</code> equal to 0, then the optimization isn't possible and the replier should behave as if <code>skipValue</code> was <code>false</code>.</p>
<p>If <code>includeDescendants</code> is <code>true</code> for a <code>Key</code>, then the replier must also include in the proof all keys that are descendant of the given key (in other words, its children, children of children, children of children of children, etc.). It must do so even if <code>key</code> itself doesn't have any storage value associated to it. The values of all of these descendants are replaced with their hashes if <code>skipValue</code> is <code>true</code>, similarly to <code>key</code> itself.</p>
<p>The optional <code>onlyKeysAfter</code> and <code>onlyKeysAfterIgnoreLastNibble</code> fields can provide a lower bound for the keys contained in the proof. The responder must not include in its proof any node whose key is strictly inferior to the value in <code>onlyKeysAfter</code>. If <code>onlyKeysAfterIgnoreLastNibble</code> is provided, then the last 4 bits for <code>onlyKeysAfter</code> must be ignored. This makes it possible to represent a trie branch node that doesn't have an even number of nibbles. If no <code>onlyKeysAfter</code> is provided, it is equivalent to being empty, meaning that the response must start with the root node of the trie.</p>
<p>If <code>onlyKeysAfterIgnoreLastNibble</code> is missing, it is equivalent to <code>false</code>. If <code>onlyKeysAfterIgnoreLastNibble</code> is <code>true</code> and <code>onlyKeysAfter</code> is missing or empty, then the request is invalid.</p>
<p>For the purpose of this networking protocol, it should be considered as if the main trie contained an entry for each default child trie whose key is <code>concat(&quot;:child_storage:default:&quot;, child_trie_hash)</code> and whose value is equal to the trie root hash of that default child trie. This behavior is consistent with what the host functions observe when querying the storage. This behavior is present in the existing networking protocol, in other words this proposal doesn't change anything to the situation, but it is worth mentioning.
Also note that child tries aren't considered as descendants of the main trie when it comes to the <code>includeDescendants</code> flag. In other words, if the request concerns the main trie, no content coming from child tries is ever sent back.</p>
<p>This protocol keeps the same maximum response size limit as currently exists (16 MiB). It is not possible for the querier to know in advance whether its query will lead to a reply that exceeds the maximum size. If the reply is too large, the replier should send back only a limited number (but at least one) of requested items in the proof. The querier should then send additional requests for the rest of the items. A response containing none of the requested items is invalid.</p>
<p>The server is allowed to silently discard some keys of the request if it judges that the number of requested keys is too high. This is in line with the fact that the server might truncate the response.</p>
<h2 id="drawbacks-23"><a class="header" href="#drawbacks-23">Drawbacks</a></h2>
<p>This proposal doesn't handle one specific situation: what if a proof containing a single specific item would exceed the response size limit? For example, if the response size limit was 1 MiB, querying the runtime code (which is typically 1.0 to 1.5 MiB) would be impossible as it's impossible to generate a proof less than 1 MiB. The response size limit is currently 16 MiB, meaning that no single storage item must exceed 16 MiB.</p>
<p>Unfortunately, because it's impossible to verify a Merkle proof before having received it entirely, parsing the proof in a streaming way is also not possible.</p>
<p>A way to solve this issue would be to Merkle-ize large storage items, so that a proof could include only a portion of a large storage item. Since this would require a change to the trie format, it is not realistically feasible in a short time frame.</p>
<h2 id="testing-security-and-privacy-24"><a class="header" href="#testing-security-and-privacy-24">Testing, Security, and Privacy</a></h2>
<p>The main security consideration concerns the size of replies and the resources necessary to generate them. It is for example easily possible to ask for all keys and values of the chain, which would take a very long time to generate. Since responses to this networking protocol have a maximum size, the replier should truncate proofs that would lead to the response being too large. Note that it is already possible to send a query that would lead to a very large reply with the existing network protocol. The only thing that this proposal changes is that it would make it less complicated to perform such an attack.</p>
<p>Implementers of the replier side should be careful to detect early on when a reply would exceed the maximum reply size, rather than inconditionally generate a reply, as this could take a very large amount of CPU, disk I/O, and memory. Existing implementations might currently be accidentally protected from such an attack thanks to the fact that requests have a maximum size, and thus that the list of keys in the query was bounded. After this proposal, this accidental protection would no longer exist.</p>
<p>Malicious server nodes might truncate Merkle proofs even when they don't strictly need to, and it is not possible for the client to (easily) detect this situation. However, malicious server nodes can already do undesirable things such as throttle down their upload bandwidth or simply not respond. There is no need to handle unnecessarily truncated Merkle proofs any differently than a server simply not answering the request.</p>
<h2 id="performance-ergonomics-and-compatibility-24"><a class="header" href="#performance-ergonomics-and-compatibility-24">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-22"><a class="header" href="#performance-22">Performance</a></h3>
<p>It is unclear to the author of the RFC what the performance implications are. Servers are supposed to have limits to the amount of resources they use to respond to requests, and as such the worst that can happen is that light client requests become a bit slower than they currently are.</p>
<h3 id="ergonomics-20"><a class="header" href="#ergonomics-20">Ergonomics</a></h3>
<p>Irrelevant.</p>
<h3 id="compatibility-19"><a class="header" href="#compatibility-19">Compatibility</a></h3>
<p>The prior networking protocol is maintained for now. The older version of this protocol could get removed in a long time.</p>
<h2 id="prior-art-and-references-23"><a class="header" href="#prior-art-and-references-23">Prior Art and References</a></h2>
<p>None. This RFC is a clean-up of an existing mechanism.</p>
<h2 id="unresolved-questions-23"><a class="header" href="#unresolved-questions-23">Unresolved Questions</a></h2>
<p>None</p>
<h2 id="future-directions-and-related-material-19"><a class="header" href="#future-directions-and-related-material-19">Future Directions and Related Material</a></h2>
<p>The current networking protocol could be deprecated in a long time. Additionally, the current &quot;state requests&quot; protocol (used for warp syncing) could also be deprecated in favor of this one.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/17">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="stale/0015-market-design-revisit.html#rfc-0015-market-design-revisit">RFC-0015: Market Design Revisit</a>
<ul>
<li><a href="stale/0015-market-design-revisit.html#summary">Summary</a></li>
<li><a href="stale/0015-market-design-revisit.html#motivation">Motivation</a></li>
<li><a href="stale/0015-market-design-revisit.html#stakeholders">Stakeholders</a></li>
<li><a href="stale/0015-market-design-revisit.html#explanation">Explanation</a>
<ul>
<li><a href="stale/0015-market-design-revisit.html#bulk-markets">Bulk Markets</a></li>
<li><a href="stale/0015-market-design-revisit.html#benefits-of-this-system">Benefits of this system</a></li>
<li><a href="stale/0015-market-design-revisit.html#further-discussion-points">Further Discussion Points</a></li>
</ul>
</li>
<li><a href="stale/0015-market-design-revisit.html#drawbacks">Drawbacks</a></li>
<li><a href="stale/0015-market-design-revisit.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="stale/0015-market-design-revisit.html#unresolved-questions">Unresolved Questions</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0015-market-design-revisit"><a class="header" href="#rfc-0015-market-design-revisit">RFC-0015: Market Design Revisit</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>05.08.2023</td></tr>
<tr><td><strong>Description</strong></td><td>This RFC refines the previously proposed mechanisms involving the various Coretime markets and presents an integrated framework for harmonious interaction between all markets.</td></tr>
<tr><td><strong>Authors</strong></td><td>Jonas Gehrlein</td></tr>
</tbody></table>
</div>
<h2 id="summary-29"><a class="header" href="#summary-29">Summary</a></h2>
<p>This document is a proposal for restructuring the bulk markets in the Polkadot UC's coretime allocation system to improve efficiency and fairness. The proposal suggests separating the <code>BULK_PERIOD</code> into <code>MARKET_PERIOD</code> and <code>RENEWAL_PERIOD</code>, allowing for a market-driven price discovery through a clearing price Dutch auction during the <code>MARKET_PERIOD</code> followed by renewal offers at the <code>MARKET_PRICE</code> during the <code>RENEWAL_PERIOD</code>. The new system ensures synchronicity between renewal and market prices, fairness among all current tenants, and efficient price discovery, while preserving price caps to provide security for current tenants. It seeks to start a discussion about the possibility of long-term leases.</p>
<h2 id="motivation-29"><a class="header" href="#motivation-29">Motivation</a></h2>
<p>While the initial <a href="https://github.com/polkadot-fellows/RFCs/blob/6f29561a4747bbfd95307ce75cd949dfff359e39/text/0001-agile-coretime.md">RFC-1</a> has provided a robust framework for Coretime allocation within the Polkadot UC, this proposal builds upon its strengths and uses many provided building blocks to address some areas that could be further improved. </p>
<p>In particular, this proposal introduces the following changes:</p>
<ul>
<li>It introduces a <code>RESERVE_PRICE</code> that anchors all markets, promoting price synchronicity within the Bulk markets (flexible + renewals).
<ul>
<li>This reduces complexity.</li>
<li>This makes sure all consumers pay a closely correlated price for coretime within a <code>BULK_PERIOD</code>.</li>
</ul>
</li>
<li>It reverses the order of the market and renewal phase.
<ul>
<li>This allows to fine-tune the price through market forces.</li>
</ul>
</li>
<li>It exposes the renewal prices, while still being beneficial for longterm tenants, more to market forces. </li>
<li>It removes the LeadIn period and introduces a (from the perspective of the coretime systemchain) passive Settlement Phase, that allows the secondary market to exert it's force.</li>
</ul>
<p>The premise of this proposal is to reduce complexity by introducing a common price (that develops releative to capacity consumption of Polkadot UC), while still allowing for market forces to add efficiency. Longterm lease owners still receive priority <strong>IF</strong> they can pay (close to) the market price. This prevents a situation where the renewal price significantly diverges from renewal prices which allows for core captures. While maximum price increase certainty might seem contradictory to efficient price discovery, the proposed model aims to balance these elements, utilizing market forces to determine the price and allocate cores effectively within certain bounds. It must be stated, that potential price increases remain predictable (in the worst-case) but could be higher than in the originally proposed design. The argument remains, however, that we need to allow market forces to affect all prices for an efficient Coretime pricing and allocation.</p>
<p>Ultimately, this the framework proposed here adheres to all requirements stated in RFC-1.</p>
<h2 id="stakeholders-28"><a class="header" href="#stakeholders-28">Stakeholders</a></h2>
<p>Primary stakeholder sets are:</p>
<ul>
<li>Protocol researchers and developers, largely represented by the Polkadot Fellowship and Parity Technologies' Engineering division.</li>
<li>Polkadot Parachain teams both present and future, and their users.</li>
<li>Polkadot DOT token holders.</li>
</ul>
<h2 id="explanation-29"><a class="header" href="#explanation-29">Explanation</a></h2>
<h3 id="bulk-markets"><a class="header" href="#bulk-markets">Bulk Markets</a></h3>
<p>The <code>BULK_PERIOD</code> has been restructured into two primary segments: the <code>MARKET_PERIOD</code> and <code>RENEWAL_PERIOD</code>, along with an auxiliary <code>SETTLEMENT_PERIOD</code>. This latter period doesn't necessitate any actions from the coretime system chain, but it facilitates a more efficient allocation of coretime in secondary markets. A significant departure from the original proposal lies in the timing of renewals, which now occur post-market phase. This adjustment aims to harmonize renewal prices with their market counterparts, ensuring a more consistent and equitable pricing model.</p>
<h4 id="market-period-14-days"><a class="header" href="#market-period-14-days">Market Period (14 days)</a></h4>
<p>During the market period, core sales are conducted through a well-established <strong>clearing price Dutch auction</strong> that features a <code>RESERVE_PRICE</code>. The price initiates at a premium, designated as <code>PRICE_PREMIUM</code> (for instance, 30%) and descends linearly to the <code>RESERVE_PRICE</code> throughout the duration of the <code>MARKET_PERIOD</code>. Each bidder is expected to submit both their desired price and the quantity (that is, the amount of Coretime) they wish to purchase. To secure these acquisitions, bidders must make a deposit equivalent to their bid multiplied by the chosen quantity, in DOT. </p>
<p>The market achieves resolution once all quantities have been sold, or the <code>RESERVE_PRICE</code> has been reached. This situation leads to determining the <code>MARKET_PRICE</code> either by the lowest bid that was successful in clearing the entire market or by the <code>RESERVE_PRICE</code>. This mechanism yields a uniform price, shaped by market forces (refer to the following discussion for an explanation of its benefits). In other words, all buyers pay the same price (per unit of Coretime). Further down the benefits of this variant of a Dutch auction is discussed.</p>
<p><strong>Note:</strong> In cases where some cores remain unsold in the market, all buyers are obligated to pay the <code>RESERVE_PRICE</code>.</p>
<h4 id="renewal-period-7-days"><a class="header" href="#renewal-period-7-days">Renewal Period (7 days)</a></h4>
<p>As the <code>RENEWAL_PERIOD</code> commences, all current tenants are granted the opportunity to renew their cores at a slight discount of <code>MARKET_PRICE * RENEWAL_DISCOUNT</code> (for instance, 10%). This provision affords marginal benefits to existing tenants, balancing out the non-transferability aspect of renewals.</p>
<p>At the end of the period, all available cores are allocated to the current tenants who have opted for renewal and the participants who placed bids during the market period. If the demand for cores exceeds supply, the cores left unclaimed from renewals may be awarded to bidders who placed their bids early in the auction, thereby subtly incentivizing early participation. If the supply exceeds the demand, all unsold cores are transferred to the Instantanous Market.</p>
<h4 id="reserve-price-adjustment"><a class="header" href="#reserve-price-adjustment">Reserve Price Adjustment</a></h4>
<p>After all cores are allocated, the <code>RESERVE_PRICE</code> is adjusted following the process described in RFC-1 and serves as baseline price in the next <code>BULK_PERIOD</code>. </p>
<p><strong>Note:</strong> The particular price curve is outside the scope of the proposal. The <code>MARKET_PRICE</code> (as a function of <code>RESERVE_PRICE</code>), however, is able to capture higher demand very well while being capped downwards. That means, the curve that adjusts the <code>RESERVE_PRICE</code> should be more sensitive to undercapacity.</p>
<h4 id="price-predictability"><a class="header" href="#price-predictability">Price Predictability</a></h4>
<p>Tasks that are in the &quot;renewal-pipeline&quot; can determine the upper bound for the price they will pay in any future period. The main driver of any price increase over time is the adjustment of the <code>RESERVE_PRICE</code>, that occurs at the end of each <code>BULK_PERIOD</code> after determining the capacity fillment of Polkadot UC. To calculate the maximum price in some future period, a task could assume maximum capacity in all upcoming periods and track the resulting price increase of <code>RESERVE_PRICE</code>. In the final period, that price can get a maximum premium of <code>PRICE_PREMIUM</code> and after deducting a potential <code>RENEWAL_DISCOUNT</code>, the maximum price can be determined.</p>
<h4 id="settlement-period-7-days"><a class="header" href="#settlement-period-7-days">Settlement Period (7 days)</a></h4>
<p>During the settlement period, participants have ample time to trade Coretime on secondary markets before the onset of the next <code>BULK_PERIOD</code>. This allows for trading with full Coretime availability. Trading transferrable Coretime naturally continues during each <code>BULK_PERIOD</code>, albeit with cores already in use.</p>
<h3 id="benefits-of-this-system"><a class="header" href="#benefits-of-this-system">Benefits of this system</a></h3>
<ul>
<li>The introduction of a single price, the <code>RESERVE_PRICE</code>, provides an anchor for all Coretime markets. This is a preventative measure against the possible divergence and mismatch of prices, which could inadvertently lead to a situation where existing tenants secure cores at significantly below-market rates.</li>
<li>With a more market-responsive pricing system, we can achieve a more efficient price discovery process. Any price increases will be less arbitrary and more dynamic.</li>
<li>The ideal strategy for existing tenants is to maintain passivity, i.e., refrain from active market participation and simply accept the offer presented to them during the renewal phase. This approach lessens the organizational overhead for long-term projects.</li>
<li>In the two-week market phase, the maximum price increase is known well in advance, providing ample time for tenants to secure necessary funds to meet the potential price escalation.</li>
<li>All existing tenants pay an equal amount for Coretime, reflecting our intent to price the Coretime itself and not the relative timing of individual projects.</li>
</ul>
<h4 id="discussion-clearing-price-dutch-auctions"><a class="header" href="#discussion-clearing-price-dutch-auctions">Discussion: Clearing Price Dutch Auctions</a></h4>
<p>Having all bidders pay the market clearing price offers some benefits and disadvantages.</p>
<ul>
<li>Advantages:
<ul>
<li><strong>Fairness</strong>: All bidders pay the same price.</li>
<li><strong>Active participation</strong>: Because bidders are protected from overbidding (winner's curse), they are more likely to engage and reveal their true valuations.</li>
<li><strong>Simplicity</strong>: A single price is easier to work with for pricing renewals later.</li>
<li><strong>Truthfulness</strong>: There is no need to try to game the market by waiting with bidding. Bidders can just bid their valuations.</li>
</ul>
</li>
<li>Disadvantages:
<ul>
<li><strong>(Potentially) Lower Revenue</strong>: While the theory predicts revenue-equivalence between a uniform price and pay-as-bid type of auction, slightly lower revenue for the former type is observed empirically. Arguably, revenue maximization (i.e., squeezing out the maximum willingness to pay from bidders) is not the priority for Polkadot UC. Instead, it is interested in efficient allocation and the other benefits illustrated above.</li>
<li><strong>(Technical) Complexity</strong>: Instead of making a final purchase within the auction, the bid is only a deposit. Some refunds might happen after the auction is finished. This might pose additional challenges from the technical side (e.g., storage requirements).</li>
</ul>
</li>
</ul>
<h3 id="further-discussion-points"><a class="header" href="#further-discussion-points">Further Discussion Points</a></h3>
<ul>
<li><strong>Long-term Coretime</strong>: The Polkadot UC is undergoing a transition from two-year leases without an instantaneous market to a model encompassing instantaneous and one-month leases. This shift seems to pivot from one extreme to another. While the introduction of short-term leases, both instantaneous and for one month, is a constructive move to lower barriers to entry and promote experimentation, it seems to be the case that established projects might benefit from more extended lease options. We could consider offering another product, such as a six-month Coretime lease, using the same mechanism described herein. Although the majority of leases would still be sold on a one-month basis, the addition of this option would enhance market efficiency as it would <strong>strengthen the impact of a secondary market</strong>.</li>
</ul>
<h2 id="drawbacks-24"><a class="header" href="#drawbacks-24">Drawbacks</a></h2>
<p>There are trade-offs that arise from this proposal, compared to the initial model. The most notable one is that here, I prioritize requirement 6 over requirement 2. The price, in the very &quot;worst-case&quot; (meaning a huge explosion in demand for coretime) could lead to a much larger increase of prices in Coretime. From an economic perspective, this (rare edgecase) would also mean that we'd vastly underprice Coretime in the original model, leading to highly inefficient allocations.</p>
<h2 id="prior-art-and-references-24"><a class="header" href="#prior-art-and-references-24">Prior Art and References</a></h2>
<p>This RFC builds extensively on the available ideas put forward in <a href="https://github.com/polkadot-fellows/RFCs/blob/6f29561a4747bbfd95307ce75cd949dfff359e39/text/0001-agile-coretime.md">RFC-1</a>. </p>
<p>Additionally, I want to express a special thanks to <a href="https://samuelhaefner.github.io/">Samuel Haefner</a> and <a href="https://sites.google.com/site/dobzin/">Shahar Dobzinski</a> for fruitful discussions and helping me structure my thoughts. </p>
<h2 id="unresolved-questions-24"><a class="header" href="#unresolved-questions-24">Unresolved Questions</a></h2>
<p>The technical feasability needs to be assessed.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/20">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="stale/0020-treasurer-track-confirmation-period-duration-modification.html#rfc-0020-treasurer-track-confirmation-period-duration-modification">RFC-0020: Treasurer Track Confirmation Period Duration Modification</a>
<ul>
<li><a href="stale/0020-treasurer-track-confirmation-period-duration-modification.html#summary">Summary</a></li>
<li><a href="stale/0020-treasurer-track-confirmation-period-duration-modification.html#motivation">Motivation</a></li>
<li><a href="stale/0020-treasurer-track-confirmation-period-duration-modification.html#stakeholders">Stakeholders</a></li>
<li><a href="stale/0020-treasurer-track-confirmation-period-duration-modification.html#explanation">Explanation</a></li>
<li><a href="stale/0020-treasurer-track-confirmation-period-duration-modification.html#drawbacks">Drawbacks</a></li>
<li><a href="stale/0020-treasurer-track-confirmation-period-duration-modification.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="stale/0020-treasurer-track-confirmation-period-duration-modification.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="stale/0020-treasurer-track-confirmation-period-duration-modification.html#performance">Performance</a></li>
<li><a href="stale/0020-treasurer-track-confirmation-period-duration-modification.html#ergonomics--compatibility">Ergonomics &amp; Compatibility</a></li>
</ul>
</li>
<li><a href="stale/0020-treasurer-track-confirmation-period-duration-modification.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="stale/0020-treasurer-track-confirmation-period-duration-modification.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="stale/0020-treasurer-track-confirmation-period-duration-modification.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0020-treasurer-track-confirmation-period-duration-modification"><a class="header" href="#rfc-0020-treasurer-track-confirmation-period-duration-modification">RFC-0020: Treasurer Track Confirmation Period Duration Modification</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>August 10, 2023</td></tr>
<tr><td><strong>Description</strong></td><td>Treasurer Track Confirmation Period Duration Modification</td></tr>
<tr><td><strong>Authors</strong></td><td>ChaosDAO</td></tr>
</tbody></table>
</div>
<h2 id="summary-30"><a class="header" href="#summary-30">Summary</a></h2>
<p>This RFC proposes a change to the duration of the confirmation period for the treasurer track from 3 hours to at least 48 hours.</p>
<h2 id="motivation-30"><a class="header" href="#motivation-30">Motivation</a></h2>
<p>Track parameters for Polkadot OpenGov should be configured in a way that their &quot;difficulty&quot; increases relative to the power associated with their respective origin. When we look at the confirmation periods for treasury based tracks, we can see that this is clearly the case - with the one notable exception to the trend being the treasurer track:</p>
<div class="table-wrapper"><table><thead><tr><th>Track Description</th><th>Confirmation Period Duration</th></tr></thead><tbody>
<tr><td>Small Tipper</td><td>10 Min</td></tr>
<tr><td>Big Tipper</td><td>1 Hour</td></tr>
<tr><td>Small Spender</td><td>12 Hours</td></tr>
<tr><td>Medium Spender</td><td>24 Hours</td></tr>
<tr><td>Big Spender</td><td>48 Hours</td></tr>
<tr><td>Treasurer</td><td><strong>3 Hours</strong></td></tr>
</tbody></table>
</div>
<p>The confirmation period is one of the last lines of defence for the collective Polkadot stakeholders to react to a potentially bad referendum and vote NAY in order for its confirmation period to be aborted. </p>
<p>Since the power / privilege level of the treasurer track is greater than that of the the big spender track – their confirmation period should be either equal, or the treasurer track's should be higher (note: currently the big spender track has a longer confirmation period than even the root track).</p>
<h2 id="stakeholders-29"><a class="header" href="#stakeholders-29">Stakeholders</a></h2>
<p>The primary stakeholders of this RFC are:</p>
<ul>
<li>DOT token holders – as this affects the protocol's treasury</li>
<li>Entities wishing to submit a referendum via the treasurer track - as this affects the referendum timeline</li>
<li>Projects with governance app integrations - see Performance, Ergonomics, and Compatibility section below.</li>
<li><a href="https://twitter.com/lolmcshizz/status/1681896333349736448">lolmcshizz</a> - expressed interest to change this parameter</li>
<li><a href="https://twitter.com/LeemoXD/status/1687408369147998208">Leemo</a> - expressed interest to change this parameter</li>
<li><a href="https://twitter.com/ParaNodes/status/1681963024842731520">Paradox</a> - expressed interest to change this parameter</li>
</ul>
<h2 id="explanation-30"><a class="header" href="#explanation-30">Explanation</a></h2>
<p>This RFC proposes to change the duration of the confirmation period for the treasurer track. In order to achieve that, the <code>confirm_period</code> parameter for the treasurer track in <code>runtime/polkadot/src/governance/tracks.rs</code> must be changed.</p>
<p><a href="https://github.com/paritytech/polkadot/blob/a1c8d720e05624d5f2ac43d89dcedd3d0d2e7342/runtime/polkadot/src/governance/tracks.rs#L119C1-L119C30">Currently it is set to</a> <code>confirm_period: 3 * HOURS</code> </p>
<p>It should be changed to <code>confirm_period: 48 * HOURS</code> as a minimum.</p>
<p>It may make sense for it to be changed to a value greater than 48 hours since the treasurer track has more power than the big spender track (48 hour confirmation period); however, the root track's confirmation period is 24 hours. 48 hours may be on the upper bounds of a trade-off between security and flexibility.</p>
<h2 id="drawbacks-25"><a class="header" href="#drawbacks-25">Drawbacks</a></h2>
<p>The drawback of changing the treasurer track's confirmation period would be that the lifecycle of a referendum submitted on the treasurer track would ultimately be longer. However, the security of the protocol and its treasury should take priority here.</p>
<h2 id="testing-security-and-privacy-25"><a class="header" href="#testing-security-and-privacy-25">Testing, Security, and Privacy</a></h2>
<p>This change will enhance / improve the security of the protocol as it relates to its treasury. The confirmation period is one of the last lines of defence for the collective Polkadot stakeholders to react to a potentially bad referendum and vote NAY in order for its confirmation period to be aborted. It makes sense for the treasurer track's confirmation period duration to be either equal to, or higher than, the big spender track confirmation period.</p>
<h2 id="performance-ergonomics-and-compatibility-25"><a class="header" href="#performance-ergonomics-and-compatibility-25">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-23"><a class="header" href="#performance-23">Performance</a></h3>
<p>This is a simple change (code wise) which should not affect the performance of the Polkadot protocol, outside of increasing the duration of the confirmation period on the treasurer track.</p>
<h3 id="ergonomics--compatibility-1"><a class="header" href="#ergonomics--compatibility-1">Ergonomics &amp; Compatibility</a></h3>
<p>If the proposal alters exposed interfaces to developers or end-users, which types of usage patterns have been optimized for?</p>
<p>I have confirmed with the following projects that this is not a breaking change for their governance apps:</p>
<ul>
<li>Nova Wallet - directly uses on-chain data, and change will be automatically reflected.</li>
<li>Polkassembly - directly uses on-chain data via rpc to fetch trackInfo so the change will be automatically reflected.</li>
<li>SubSquare - scan script will update their app to the latest parameters and it will be automatically reflected in their app.</li>
</ul>
<h2 id="prior-art-and-references-25"><a class="header" href="#prior-art-and-references-25">Prior Art and References</a></h2>
<p>N/A</p>
<h2 id="unresolved-questions-25"><a class="header" href="#unresolved-questions-25">Unresolved Questions</a></h2>
<p>The proposed change to the confirmation period duration for the treasurer track is to set it to 48 hours. This is equal to the current confirmation period for the big spender track.</p>
<p>Typically it seems that track parameters increase in difficulty (duration, etc.) based on the power level of their associated origin. </p>
<p>The longest confirmation period is that of the big spender, at 48 hours. There may be value in discussing whether or not the treasurer track confirmation period should be longer than 48 hours – a discussion of the trade-offs between security vs flexibility/agility.</p>
<p>As a side note, the root track confirmation period is 24 hours.</p>
<h2 id="future-directions-and-related-material-20"><a class="header" href="#future-directions-and-related-material-20">Future Directions and Related Material</a></h2>
<p>This RFC hopefully reminds the greater Polkadot community that it is possible to submit changes to the parameters of Polkadot OpenGov, and the greater protocol as a whole through the RFC process.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/26">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="stale/0026-sassafras-consensus.html#rfc-0026-sassafras-consensus-protocol">RFC-0026: Sassafras Consensus Protocol</a>
<ul>
<li><a href="stale/0026-sassafras-consensus.html#abstract">Abstract</a></li>
<li><a href="stale/0026-sassafras-consensus.html#1-motivation">1. Motivation</a>
<ul>
<li><a href="stale/0026-sassafras-consensus.html#11-relevance-to-implementors">1.1. Relevance to Implementors</a></li>
<li><a href="stale/0026-sassafras-consensus.html#12-supporting-sassafras-for-polkadot">1.2. Supporting Sassafras for Polkadot</a></li>
</ul>
</li>
<li><a href="stale/0026-sassafras-consensus.html#2-stakeholders">2. Stakeholders</a>
<ul>
<li><a href="stale/0026-sassafras-consensus.html#21-blockchain-developers">2.1. Blockchain Developers</a></li>
<li><a href="stale/0026-sassafras-consensus.html#22-polkadot-ecosystem-contributors">2.2. Polkadot Ecosystem Contributors</a></li>
</ul>
</li>
<li><a href="stale/0026-sassafras-consensus.html#3-notation-and-convention">3. Notation and Convention</a>
<ul>
<li><a href="stale/0026-sassafras-consensus.html#31-data-structures-definitions-and-encoding">3.1. Data Structures Definitions and Encoding</a></li>
<li><a href="stale/0026-sassafras-consensus.html#32-pseudo-code">3.2. Pseudo-Code</a></li>
<li><a href="stale/0026-sassafras-consensus.html#33-incremental-introduction-of-types-and-functions">3.3. Incremental Introduction of Types and Functions</a></li>
</ul>
</li>
<li><a href="stale/0026-sassafras-consensus.html#4-protocol-introduction">4. Protocol Introduction</a>
<ul>
<li><a href="stale/0026-sassafras-consensus.html#41-submission-of-candidate-tickets">4.1. Submission of Candidate Tickets</a></li>
<li><a href="stale/0026-sassafras-consensus.html#42-validation-of-candidate-tickets">4.2. Validation of Candidate Tickets</a></li>
<li><a href="stale/0026-sassafras-consensus.html#43-tickets-and-slots-binding">4.3. Tickets and Slots Binding</a></li>
<li><a href="stale/0026-sassafras-consensus.html#44-claim-of-ticket-ownership">4.4. Claim of Ticket Ownership</a></li>
<li><a href="stale/0026-sassafras-consensus.html#45-validation-of-ticket-ownership">4.5. Validation of Ticket Ownership</a></li>
</ul>
</li>
<li><a href="stale/0026-sassafras-consensus.html#5-bandersnatch-vrfs-cryptographic-primitives">5. Bandersnatch VRFs Cryptographic Primitives</a>
<ul>
<li><a href="stale/0026-sassafras-consensus.html#51-vrf-input">5.1. VRF Input</a></li>
<li><a href="stale/0026-sassafras-consensus.html#52-vrf-preoutput">5.2. VRF PreOutput</a></li>
<li><a href="stale/0026-sassafras-consensus.html#53-vrf-signature-data">5.3. VRF Signature Data</a></li>
<li><a href="stale/0026-sassafras-consensus.html#54-vrf-signature">5.4. VRF Signature</a></li>
</ul>
</li>
<li><a href="stale/0026-sassafras-consensus.html#6-sassafras-protocol">6. Sassafras Protocol</a>
<ul>
<li><a href="stale/0026-sassafras-consensus.html#61-epochs-first-block">6.1. Epoch's First Block</a></li>
<li><a href="stale/0026-sassafras-consensus.html#62-creation-and-submission-of-candidate-tickets">6.2. Creation and Submission of Candidate Tickets</a></li>
<li><a href="stale/0026-sassafras-consensus.html#63-validation-of-candidate-tickets">6.3. Validation of candidate tickets</a></li>
<li><a href="stale/0026-sassafras-consensus.html#64-ticket-slot-binding">6.4. Ticket-Slot Binding</a></li>
<li><a href="stale/0026-sassafras-consensus.html#65-slot-claim-production">6.5. Slot Claim Production</a></li>
<li><a href="stale/0026-sassafras-consensus.html#66-slot-claim-verification">6.6. Slot Claim Verification</a></li>
<li><a href="stale/0026-sassafras-consensus.html#661-primary-method">6.6.1. Primary Method</a></li>
<li><a href="stale/0026-sassafras-consensus.html#67-randomness-accumulator">6.7. Randomness Accumulator</a></li>
</ul>
</li>
<li><a href="stale/0026-sassafras-consensus.html#7-drawbacks">7. Drawbacks</a></li>
<li><a href="stale/0026-sassafras-consensus.html#8-testing-security-and-privacy">8. Testing, Security, and Privacy</a></li>
<li><a href="stale/0026-sassafras-consensus.html#9-performance-ergonomics-and-compatibility">9. Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="stale/0026-sassafras-consensus.html#91-performance">9.1. Performance</a></li>
<li><a href="stale/0026-sassafras-consensus.html#92-ergonomics">9.2. Ergonomics</a></li>
<li><a href="stale/0026-sassafras-consensus.html#93-compatibility">9.3. Compatibility</a></li>
</ul>
</li>
<li><a href="stale/0026-sassafras-consensus.html#10-prior-art-and-references">10. Prior Art and References</a></li>
<li><a href="stale/0026-sassafras-consensus.html#11-unresolved-questions">11. Unresolved Questions</a></li>
<li><a href="stale/0026-sassafras-consensus.html#12-future-directions-and-related-material">12. Future Directions and Related Material</a>
<ul>
<li><a href="stale/0026-sassafras-consensus.html#121-interactions-with-on-chain-code">12.1. Interactions with On-Chain Code</a></li>
<li><a href="stale/0026-sassafras-consensus.html#122-deployment-strategies">12.2. Deployment Strategies</a></li>
<li><a href="stale/0026-sassafras-consensus.html#123-zk-snark-srs-initialization">12.3. ZK-SNARK SRS Initialization</a></li>
<li><a href="stale/0026-sassafras-consensus.html#124-anonymous-submission-of-tickets">12.4. Anonymous Submission of Tickets.</a></li>
</ul>
</li>
</ul>
</li>
</ul>
<h1 id="rfc-0026-sassafras-consensus-protocol"><a class="header" href="#rfc-0026-sassafras-consensus-protocol">RFC-0026: Sassafras Consensus Protocol</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>September 06, 2023</td></tr>
<tr><td><strong>Description</strong></td><td>Sassafras consensus protocol description and structures</td></tr>
<tr><td><strong>Authors</strong></td><td>Davide Galassi</td></tr>
</tbody></table>
</div>
<h2 id="abstract"><a class="header" href="#abstract">Abstract</a></h2>
<p>Sassafras is a novel consensus protocol designed to address the recurring
fork-related challenges encountered in other lottery-based protocols.</p>
<p>The protocol aims to create a mapping between each epoch's slots and the
validators set while ensuring that the identity of validators assigned to
the slots remains undisclosed until the slot is actively claimed during block
production.</p>
<h2 id="1-motivation"><a class="header" href="#1-motivation">1. Motivation</a></h2>
<p>Sassafras Protocol has been rigorously detailed in a comprehensive
<a href="https://eprint.iacr.org/2023/031.pdf">research paper</a> authored by the
<a href="https://web3.foundation">Web3 foundation</a> research team.</p>
<p>This RFC is primarily intended to detail the critical implementation aspects
vital for ensuring interoperability and to clarify certain aspects that are
left open by the research paper and thus subject to interpretation during
implementation.</p>
<h3 id="11-relevance-to-implementors"><a class="header" href="#11-relevance-to-implementors">1.1. Relevance to Implementors</a></h3>
<p>This RFC focuses on providing implementors with the necessary insights into the
protocol's operation.</p>
<p>In instances of inconsistency between this document and the research paper,
this RFC should be considered authoritative to eliminate ambiguities and ensure
interoperability.</p>
<h3 id="12-supporting-sassafras-for-polkadot"><a class="header" href="#12-supporting-sassafras-for-polkadot">1.2. Supporting Sassafras for Polkadot</a></h3>
<p>Beyond promoting interoperability, this RFC also aims to facilitate the
implementation of Sassafras within the Polkadot ecosystem.</p>
<p>Although the specifics of deployment strategies are beyond the scope of this
document, it lays the groundwork for the integration of Sassafras into the
Polkadot network.</p>
<h2 id="2-stakeholders"><a class="header" href="#2-stakeholders">2. Stakeholders</a></h2>
<h3 id="21-blockchain-developers"><a class="header" href="#21-blockchain-developers">2.1. Blockchain Developers</a></h3>
<p>Developers responsible for creating blockchains who intend to leverage the
benefits offered by the Sassafras Protocol.</p>
<h3 id="22-polkadot-ecosystem-contributors"><a class="header" href="#22-polkadot-ecosystem-contributors">2.2. Polkadot Ecosystem Contributors</a></h3>
<p>Developers contributing to the Polkadot ecosystem, both relay-chain and
para-chains.</p>
<p>The protocol will have a central role in the next generation block authoring
consensus systems.</p>
<h2 id="3-notation-and-convention"><a class="header" href="#3-notation-and-convention">3. Notation and Convention</a></h2>
<p>This section outlines the notation and conventions adopted throughout this
document to ensure clarity and consistency.</p>
<h3 id="31-data-structures-definitions-and-encoding"><a class="header" href="#31-data-structures-definitions-and-encoding">3.1. Data Structures Definitions and Encoding</a></h3>
<p>Data structures are primarily defined using standard <a href="https://en.wikipedia.org/wiki/ASN.1">ASN.1</a>,
syntax with few exceptions:</p>
<ul>
<li>Fixed width integer types are not explicitly defined by ASN.1 standard.
Within this document, <code>U&lt;n&gt;</code> denotes a <code>n</code>-bit unsigned integer.</li>
</ul>
<p>Unless explicitly noted, all types must be serialized using
<a href="https://github.com/paritytech/parity-scale-codec">SCALE</a> codec.</p>
<p>To ensure interoperability of serialized structures, the order of the fields
must match the structures definitions found within this document.</p>
<h3 id="32-pseudo-code"><a class="header" href="#32-pseudo-code">3.2. Pseudo-Code</a></h3>
<p>It is advantageous to make use of code snippets as part of the protocol
description. As a convention, the code is formatted in a style similar to
<em>Rust</em>, and can make use of the following set of predefined functions:</p>
<ul>
<li>
<p><code>BYTES(x: T)</code>: returns an <code>OCTET_STRING</code> that represents the raw byte array of
the object x with type T.</p>
<ul>
<li>If <code>T</code> is a <code>VisibleString</code> (ASCII string), it returns the sequence
of octets of its ASCII representation.</li>
<li>If <code>T</code> is <code>U&lt;n&gt;</code>, it returns the little-endian encoding of the integer
<code>U&lt;n&gt;</code> as <code>n/8</code> octets.</li>
</ul>
</li>
<li>
<p><code>U&lt;n&gt;(x: OCTET_STRING)</code>: returns a <code>U&lt;n&gt;</code> interpreting <code>x</code> as the
little-endian encoding of a <code>n</code> bits unsigned integer.</p>
</li>
<li>
<p><code>SCALE(x: T)</code>: returns an <code>OCTET_STRING</code> representing the SCALE encoding of
<code>x</code> with type <code>T</code>.</p>
</li>
<li>
<p><code>BLAKE2(n: U32, x: OCTET_STRING)</code>: returns the standard <em>Blake2b</em> <code>n</code>
bytes hash of <code>x</code> as an <code>OCTET_STRING</code> (note this is not equivalent to the
truncation of the full 64 bytes <em>Blake2b</em> hash).</p>
</li>
<li>
<p><code>CONCAT(x₀: OCTET_STRING, ..., xₖ: OCTET_STRING)</code>: returns the concatenation
of the inputs as an <code>OCTET_STRING</code>.</p>
</li>
<li>
<p><code>LENGTH(x: OCTET_STRING)</code>: returns the number of octets in <code>x</code> as an <code>U32</code>.</p>
</li>
</ul>
<h3 id="33-incremental-introduction-of-types-and-functions"><a class="header" href="#33-incremental-introduction-of-types-and-functions">3.3. Incremental Introduction of Types and Functions</a></h3>
<p>More types and helper functions are introduced incrementally as they become
relevant within the document's context.</p>
<p>We find this approach more agile, especially given that the set of types used is
not overly complex.</p>
<h2 id="4-protocol-introduction"><a class="header" href="#4-protocol-introduction">4. Protocol Introduction</a></h2>
<p>The timeline is segmented into a sequentially ordered sequence of <strong>slots</strong>.
This entire sequence of slots is then further partitioned into distinct segments
known as <strong>epochs</strong>.</p>
<p>The Sassafras protocol aims to map each slot within an epoch to the designated
validators for that epoch, utilizing a ticketing system.</p>
<p>The protocol operation can be roughly divided into five phases:</p>
<h3 id="41-submission-of-candidate-tickets"><a class="header" href="#41-submission-of-candidate-tickets">4.1. Submission of Candidate Tickets</a></h3>
<p>Each of the validators associated to the target epoch generates and submits
a set of candidate tickets to the blockchain. Every ticket is bundled with an
anonymous proof of validity.</p>
<h3 id="42-validation-of-candidate-tickets"><a class="header" href="#42-validation-of-candidate-tickets">4.2. Validation of Candidate Tickets</a></h3>
<p>Each candidate ticket undergoes a validation process for the associated validity
proof and compliance with other protocol-specific constraints.</p>
<h3 id="43-tickets-and-slots-binding"><a class="header" href="#43-tickets-and-slots-binding">4.3. Tickets and Slots Binding</a></h3>
<p>After collecting all valid candidate tickets, a deterministic method is used to
uniquely associate a subset of these tickets with the slots of the target epoch.</p>
<h3 id="44-claim-of-ticket-ownership"><a class="header" href="#44-claim-of-ticket-ownership">4.4. Claim of Ticket Ownership</a></h3>
<p>During the block production phase of the target epoch, validators are required
to demonstrate their ownership of tickets. This step discloses the identity of
the ticket owners.</p>
<h3 id="45-validation-of-ticket-ownership"><a class="header" href="#45-validation-of-ticket-ownership">4.5. Validation of Ticket Ownership</a></h3>
<p>During block verification, the claim of ticket ownership is validated.</p>
<h2 id="5-bandersnatch-vrfs-cryptographic-primitives"><a class="header" href="#5-bandersnatch-vrfs-cryptographic-primitives">5. Bandersnatch VRFs Cryptographic Primitives</a></h2>
<p>This chapter provides a high-level overview of the Bandersnatch VRF primitive as
it relates to the Sassafras protocol.</p>
<p>It's important to note that this section is not intended to serve as an
exhaustive exploration of the mathematically intensive foundations of the
cryptographic primitive. Rather, its primary aim is to offer a concise and
accessible explanation of the primitive's role and usage which is relevant
within the scope of this RFC.</p>
<p>For an in-depth explanation, refer to the Ring-VRF
<a href="https://eprint.iacr.org/2023/002.pdf">paper</a> authored by the Web3 foundation
research team.</p>
<h3 id="51-vrf-input"><a class="header" href="#51-vrf-input">5.1. VRF Input</a></h3>
<p>The VRF Input, denoted as <code>VrfInput</code>, is constructed by combining a domain
identifier with arbitrary data through the <code>vrf_input</code> function:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> fn vrf_input(domain: OCTET_STRING, data: OCTET_STRING) -&gt; VrfInput;
<span class="boring">}</span></code></pre></pre>
<p>The specific implementation details of this function are intentionally omitted.
A reference implementation is provided by the
<a href="https://github.com/w3f/ring-vrf/tree/master/bandersnatch_vrfs"><code>bandersnatch_vrfs</code></a>
project.</p>
<TODO>
The above link points to some temporary code (Transcript label set to "TemporaryDoNotDeploy").
Also replace with docs.rs link once published to crates.io.
</TODO>
<p>Helper function to construct a <code>VrfInput</code> from a sequence of <code>data</code> items:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> fn vrf_input_from_items(domain: OCTET_STRING, items: SEQUENCE_OF OCTET_STRING) -&gt; VrfInput {
let data = OCTET_STRING(SIZE = 0); // empty octet string
for item in items {
data.append(item);
data.append(LENGTH(item) as U8);
}
return vrf_input(domain, data);
}
<span class="boring">}</span></code></pre></pre>
<p>Note that each item length is safely casted to an <code>U8</code> as:</p>
<ol>
<li>In the context of this protocol all items lengths are less than 256.</li>
<li>The function is internal and not designed for generic use.</li>
</ol>
<h3 id="52-vrf-preoutput"><a class="header" href="#52-vrf-preoutput">5.2. VRF PreOutput</a></h3>
<p>Functionally, the <code>VrfPreOutput</code> can be considered as a <em>seed</em> for a PRNG to
produce an arbitrary number of output bytes.</p>
<p>It is computed as function of a <code>VrfInput</code> and a <code>BandersnatchSecretKey</code>.</p>
<p>Two different approaches can be used to generate it: as a standalone object
or as part of a signature. While the resulting <code>VrfPreOutput</code> is identical
in both cases, the legitimacy of the latter can be confirmed by verifying the
signature using the <code>BandersnatchPublicKey</code> of the expected signer.</p>
<p>When constructed as a standalone object, <code>VrfPreOutput</code> is primarily employed
in situations where the secret key owner needs to check if the generated output
bytes fulfill some context specific criteria before applying the signature.</p>
<p>To facilitate the construction, the following helper function is provided:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> fn vrf_pre_output(secret: BandernatchSecretKey, input: VrfInput) -&gt; VrfPreOutput;
<span class="boring">}</span></code></pre></pre>
<p>An additional helper function is provided for producing an arbitrary number of
output bytes from <code>VrfInput</code> and <code>VrfPreOutput</code>:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> fn vrf_bytes(len: U32, input: VrfInput, pre_output: VrfPreOuput) -&gt; OCTET_STRING;
<span class="boring">}</span></code></pre></pre>
<p>Similar to the <code>vrf_input</code> function, the details about the implementation
of these functions is omitted. Reference implementations are provided by the
<a href="https://github.com/w3f/ring-vrf/tree/master/dleq_vrfs"><code>dleq_vrfs</code></a> project</p>
<ul>
<li><a href="https://docs.rs/dleq_vrf/0.0.1/dleq_vrf/keys/struct.SecretKey.html#method.vrf_preout"><code>vrf_pre_output</code></a></li>
<li><a href="https://docs.rs/dleq_vrf/0.0.1/dleq_vrf/vrf/struct.VrfInOut.html#method.vrf_preoutput_bytes"><code>vrf_bytes</code></a></li>
</ul>
<h3 id="53-vrf-signature-data"><a class="header" href="#53-vrf-signature-data">5.3. VRF Signature Data</a></h3>
<p>This section outlines the data to be signed utilizing the VRF primitive:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> VrfSignatureData ::= SEQUENCE {
transcript: Transcript,
inputs: SEQUENCE_OF VrfInput
}
<span class="boring">}</span></code></pre></pre>
<p>Where:</p>
<ul>
<li><code>transcript</code>: a <a href="https://docs.rs/ark-transcript/0.0.1/ark_transcript/struct.Transcript.html"><code>Transcript</code></a>
instance. In practice, this is a <em>special</em> hash of some protocol-specific data
to sign which doesn't influence the <code>VrfPreOutput</code>.</li>
<li><code>inputs</code>: sequence of <code>VrfInputs</code> to be signed.</li>
</ul>
<p>To simplify the construction of <code>VrfSignatureData</code> objects, a helper function is defined:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> fn vrf_signature_data(
transcript_label: OCTET_STRING,
transcript_data: SEQUENCE_OF OCTET_STRING,
inputs: SEQUENCE_OF VrfInput
) -&gt; VrfSignatureData {
let mut transcript = Transcript::new_labeled(transcript_label);
for data in transcript_data {
transcript.append(data);
}
VrfSignatureData { transcript, inputs }
}
<span class="boring">}</span></code></pre></pre>
<h3 id="54-vrf-signature"><a class="header" href="#54-vrf-signature">5.4. VRF Signature</a></h3>
<p>Bandersnatch VRF offers two signature flavors:</p>
<ul>
<li><em>plain</em> signature: much like a traditional <em>Schnorr</em> signature,</li>
<li><em>ring</em> signature: leverages a <em>zk-SNARK</em> to allows for anonymous signatures
using a key from a predefined set of enabled keys, known as the ring.</li>
</ul>
<h4 id="541-plain-vrf-signature"><a class="header" href="#541-plain-vrf-signature">5.4.1. Plain VRF Signature</a></h4>
<p>This section describes the signature process for <code>VrfSignatureData</code> using the
<em>plain</em> signature flavor.</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> PlainSignature ::= OCTET_STRING;
VrfSignature ::= SEQUENCE {
signature: PlainSignature,
pre_outputs: SEQUENCE-OF VrfPreOutput
}
<span class="boring">}</span></code></pre></pre>
<p>Where:</p>
<ul>
<li><code>signature</code>: the actual plain signature.</li>
<li><code>pre_outputs</code>: sequence of <code>VrfPreOutput</code>s corresponding to the <code>VrfInput</code>s
found within the <code>VrfSignatureData</code>.</li>
</ul>
<p>Helper function to construct <code>VrfPlainSignature</code> from <code>VrfSignatureData</code>:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> BandersnatchSecretKey ::= OCTET_STRING;
fn vrf_sign(
secret: BandernatchSecretKey,
signature_data: VrfSignatureData
) -&gt; VrfSignature
<span class="boring">}</span></code></pre></pre>
<p>Helper function for signature verification returning a <code>BOOLEAN</code> value
indicating the validity of the signature (<code>true</code> on success):</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> BandersnatchPublicKey ::= OCTET_STRING;
fn vrf_verify(
public: BandersnatchPublicKey,
signature: VrfSignature
) -&gt; BOOLEAN;
<span class="boring">}</span></code></pre></pre>
<p>In this document, the types <code>BandersnatchSecretKey</code>, <code>BandersnatchPublicKey</code>
and <code>PlainSignature</code> are intentionally left undefined. Their definitions can be
found in the <code>bandersnatch_vrfs</code> reference implementation.</p>
<h4 id="542-ring-vrf-signature"><a class="header" href="#542-ring-vrf-signature">5.4.2. Ring VRF Signature</a></h4>
<p>This section describes the signature process for <code>VrfSignatureData</code> using the
<em>ring</em> signature flavor.</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> RingSignature ::= OCTET_STRING;
RingVrfSignature ::= SEQUENCE {
signature: RingSignature,
pre_outputs: SEQUENCE_OF VrfPreOutput
}
<span class="boring">}</span></code></pre></pre>
<ul>
<li><code>signature</code>: the actual ring signature.</li>
<li><code>pre_outputs</code>: sequence of <code>VrfPreOutput</code>s corresponding to the <code>VrfInput</code>s
found within the <code>VrfSignatureData</code>.</li>
</ul>
<p>Helper function to construct <code>RingVrfSignature</code> from <code>VrfSignatureData</code>:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> BandersnatchRingProverKey ::= OCTET_STRING;
fn ring_vrf_sign(
secret: BandersnatchRingProverKey,
signature_data: VrfSignatureData,
) -&gt; RingVrfSignature;
<span class="boring">}</span></code></pre></pre>
<p>Helper function for signature verification returning a <code>BOOLEAN</code> value
indicating the validity of the signature (<code>true</code> on success).</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> BandersnatchRingVerifierKey ::= OCTET_STRING;
fn ring_vrf_verify(
verifier: BandersnatchRingVerifierKey,
signature: RingVrfSignature,
) -&gt; BOOLEAN;
<span class="boring">}</span></code></pre></pre>
<p>Note that this function doesn't require the signer's public key.</p>
<p>In this document, the types <code>BandersnatchRingProverKey</code>,
<code>BandersnatchRingVerifierKey</code>, and <code>RingSignature</code> are intentionally left
undefined. Their definitions can be found in the <code>bandersnatch_vrfs</code> reference
implementation.</p>
<h2 id="6-sassafras-protocol"><a class="header" href="#6-sassafras-protocol">6. Sassafras Protocol</a></h2>
<h3 id="61-epochs-first-block"><a class="header" href="#61-epochs-first-block">6.1. Epoch's First Block</a></h3>
<p>For epoch <code>N</code>, the first block produced must include a descriptor for some of
the subsequent epoch (<code>N+1</code>) parameters. This descriptor is defined as:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> NextEpochDescriptor ::= SEQUENCE {
randomness: OCTET_STRING(SIZE(32)),
authorities: SEQUENCE_OF BandersnatchPublicKey,
configuration: ProtocolConfiguration OPTIONAL
}
<span class="boring">}</span></code></pre></pre>
<p>Where:</p>
<ul>
<li><code>randomness</code>: 32-bytes pseudo random value.</li>
<li><code>authorities</code>: list of authorities.</li>
<li><code>configuration</code>: optional protocol configuration.</li>
</ul>
<p>This descriptor must be encoded using the <code>SCALE</code> encoding system and embedded
in the block header's digest log. The identifier for the digest element is
<code>BYTES(&quot;SASS&quot;)</code>.</p>
<p>A special case arises for the first block for epoch <code>0</code>, which each node produces
independently during the genesis phase. In this case, the <code>NextEpochDescriptor</code>
relative to epoch <code>1</code> is shared within the second block, as outlined in section
<a href="stale/0026-sassafras-consensus.html#613-startup-parameters">6.1.3</a>.</p>
<h4 id="611-epoch-randomness"><a class="header" href="#611-epoch-randomness">6.1.1. Epoch Randomness</a></h4>
<p>The randomness in the <code>NextEpochDescriptor</code> <code>randomness</code> is computed as:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> randomness = BLAKE2(32, CONCAT(randomness_accumulator, BYTES(next_epoch.index)));
<span class="boring">}</span></code></pre></pre>
<p>Here, <code>randomness_accumulator</code> refers to a 32-byte <code>OCTET_STRING</code> stored
on-chain and computed through a process that incorporates verifiable random
elements from all previously imported blocks. The exact procedure is described
in section <a href="stale/0026-sassafras-consensus.html#67-randomness-accumulator">6.7</a>.</p>
<h4 id="612-protocol-configuration"><a class="header" href="#612-protocol-configuration">6.1.2. Protocol Configuration</a></h4>
<p>The <code>ProtocolConfiguration</code> primarily influences certain checks carried out
during tickets validation. It is defined as:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> ProtocolConfiguration ::= SEQUENCE {
attempts_number: U32,
redundancy_factor: U32
}
<span class="boring">}</span></code></pre></pre>
<p>Where:</p>
<ul>
<li><code>attempts_number</code>: maximum number of tickets that each authority for the next
epoch is allowed to submit.</li>
<li><code>redundancy_factor</code>: expected ratio between epoch's slots and the cumulative
number of tickets which can be submitted by the set of epoch validators.</li>
</ul>
<p>The <code>attempts_number</code> influences the anonymity of block producers. As all
published tickets have a <strong>public</strong> attempt number less than <code>attempts_number</code>,
all the tickets which share the attempt number value must belong to different
block producers, which reduces anonymity late as we approach the epoch tail.
Bigger values guarantee more anonymity but also more computation.</p>
<p>Details about how exactly these parameters drives the ticket validity
probability can be found in section <a href="stale/0026-sassafras-consensus.html#622-tickets-threshold">6.2.2</a>.</p>
<p><code>ProtocolConfiguration</code> values can be adjusted via a dedicated on-chain call
which should have origin set to <code>Root</code>. Any proposed changes to
<code>ProtocolConfiguration</code> that are submitted in epoch <code>K</code> will be included in the
<code>NextEpochDescriptor</code> at the start of epoch <code>K+1</code> and will come into effect in
epoch <code>K+2</code>.</p>
<h4 id="613-startup-parameters"><a class="header" href="#613-startup-parameters">6.1.3. Startup Parameters</a></h4>
<p>Some of the initial parameters for the first epoch, Epoch <code>#0</code>, are set through
the genesis configuration, which is defined as:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> GenesisConfig ::= SEQUENCE {
authorities: SEQUENCE_OF BandersnatchPublicKey,
configuration: ProtocolConfiguration,
}
<span class="boring">}</span></code></pre></pre>
<p>The on-chain randomness accumulator is initialized only <strong>after</strong> the genesis
block is produced. It starts with the hash of the genesis block:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> randomness_accumulator = genesis_hash
<span class="boring">}</span></code></pre></pre>
<p>Since block <code>#0</code> is generated locally by each node as part of the genesis
process, the first block that a validator explicitly produces for Epoch
<code>#0</code> is block <code>#1</code>. Therefore, block <code>#1</code> is required to contain the
<code>NextEpochDescriptor</code> for the following epoch, Epoch <code>#1</code>.</p>
<p>The <code>NextEpochDescriptor</code> for Epoch <code>#1</code>:</p>
<ul>
<li><code>randomness</code>: computed using the <code>randomness_accumulator</code> established
post-genesis, as mentioned above.</li>
<li><code>authorities</code>: the same as those specified in the genesis configuration.</li>
<li><code>configuration</code>: not set (i.e., <code>None</code>), implying the reuse of the
one found in the genesis configuration.</li>
</ul>
<h3 id="62-creation-and-submission-of-candidate-tickets"><a class="header" href="#62-creation-and-submission-of-candidate-tickets">6.2. Creation and Submission of Candidate Tickets</a></h3>
<p>After the beginning of a new epoch <code>N</code>, each validator associated to the next
epoch (<code>N+1</code>) constructs a set of tickets which may be eligible (<a href="stale/0026-sassafras-consensus.html#622-tickets-threshold">6.2.2</a>)
to be submitted on-chain. These tickets aim to secure ownership of one or more
slots in the upcoming epoch <code>N+1</code>.</p>
<p>Each validator is allowed to submit a maximum number of tickets, as specified by
the <code>attempts_number</code> field in the <code>ProtocolConfiguration</code> for the next epoch.</p>
<p>The ideal timing for a validator to start creating the tickets is subject to
strategy. A recommended approach is to initiate tickets creation once the block
containing the <code>NextEpochDescriptor</code> is either probabilistically or, preferably,
deterministically finalized. This timing is suggested to prevent to waste
resources on tickets that might become obsolete if a different chain branch
is finally chosen as the best one by the distributed system.</p>
<p>However, validators are also advised to avoid submitting tickets too late,
as tickets submitted during the second half of the epoch must be discarded.</p>
<h4 id="621-ticket-identifier-value"><a class="header" href="#621-ticket-identifier-value">6.2.1. Ticket Identifier Value</a></h4>
<p>Each ticket has an associated 128-bit unique identifier defined as:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> TicketId ::= U128;
<span class="boring">}</span></code></pre></pre>
<p>The value of the <code>TicketId</code> is determined by the output of the Bandersnatch VRF
with the following input:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> ticket_id_vrf_input = vrf_input_from_items(
BYTES(&quot;sassafras-ticket-v1.0&quot;),
[
next_epoch.randomness,
BYTES(next_epoch.index),
BYTES(attempt_index)
]
);
ticket_id_vrf_pre_output = vrf_pre_output(AUTHORITY_SECRET_KEY, ticket_id_vrf_input);
ticket_bytes = vrf_bytes(16, ticket_id_vrf_input, ticket_id_vrf_pre_output);
ticket_id = U128(ticket_bytes);
<span class="boring">}</span></code></pre></pre>
<p>Where:</p>
<ul>
<li><code>next_epoch.randomness</code>: randomness associated to the target epoch.</li>
<li><code>next_epoch.index</code>: index of the target epoch as a <code>U64</code>.</li>
<li><code>attempt_index</code>: value going from <code>0</code> to <code>attempts_number</code> as a <code>U32</code>.</li>
</ul>
<h4 id="622-tickets-threshold"><a class="header" href="#622-tickets-threshold">6.2.2. Tickets Threshold</a></h4>
<p>A <code>TicketId</code> value is valid if its value is less than the ticket threshold:</p>
<pre><code>T = (r·s)/(a·v)
</code></pre>
<p>Where:</p>
<ul>
<li><code>v</code>: epoch's authorities (aka validators) number</li>
<li><code>s</code>: epoch's slots number</li>
<li><code>r</code>: redundancy factor</li>
<li><code>a</code>: attempts number</li>
<li><code>T</code>: ticket threshold value (<code>0 ≤ T ≤ 1</code>)</li>
</ul>
<h5 id="6221-formula-derivation"><a class="header" href="#6221-formula-derivation">6.2.2.1 Formula Derivation</a></h5>
<p>In an epoch with <code>s</code> slots, the goal is to achieve an expected number of tickets
for block production equal to <code>r·s</code>.</p>
<p>It's crucial to ensure that the probability of having fewer than <code>s</code> winning
tickets is very low, even in scenarios where up to <code>1/3</code> of the authorities
might be offline.</p>
<p>To accomplish this, we first define the winning probability of a single ticket
as <code>T = (r·s)/(a·v)</code>.</p>
<p>Let <code>n</code> be the actual number of participating validators, where <code>v·2/3 ≤ n ≤ v</code>.</p>
<p>These <code>n</code> validators each make <code>a</code> attempts, for a total of <code>a·n</code> attempts.</p>
<p>Let <code>X</code> be the random variable associated to the number of winning tickets, then
its expected value is:</p>
<pre><code>E[X] = T·a·n = (r·s·n)/v
</code></pre>
<p>By setting <code>r = 2</code>, we get</p>
<pre><code>s·4/3 ≤ E[X] ≤ s·2
</code></pre>
<p>Using <em>Bernestein's inequality</em> we get <code>Pr[X &lt; s] ≤ e^(-s/21)</code>.</p>
<p>For instance, with <code>s = 600</code> this results in <code>Pr[X &lt; s] &lt; 4·10⁻¹³</code>.
Consequently, this approach offers considerable tolerance for offline nodes and
ensures that all slots are likely to be filled with tickets.</p>
<p>For more details about threshold formula please refer to the
<a href="https://research.web3.foundation/Polkadot/protocols/block-production/SASSAFRAS#probabilities-and-parameters">probabilities and parameters</a>
paragraph in the Web3 foundation description of the protocol.</p>
<h4 id="623-ticket-body"><a class="header" href="#623-ticket-body">6.2.3. Ticket Body</a></h4>
<p>Every candidate ticket identifier has an associated body, defined as:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> TicketBody ::= SEQUENCE {
attempt_index: U32,
erased_pub: Ed25519PublicKey,
revealed_pub: Ed25519PublicKey
}
<span class="boring">}</span></code></pre></pre>
<p>Where:</p>
<ul>
<li><code>attempt_index</code>: attempt index used to generate the associated <code>TicketId</code>.</li>
<li><code>erased_pub</code>: Ed25519 ephemeral public key which gets erased as soon as the
ticket is claimed. This key can be used to encrypt data for the validator.</li>
<li><code>revealed_pub</code>: Ed25519 ephemeral public key which gets exposed as soon as the
ticket is claimed.</li>
</ul>
<p>The process of generating an erased key pair is intentionally left undefined,
allowing the implementor the freedom to choose the most suitable strategy.</p>
<p>Revealed key pair is generated using the bytes produced by the VRF with input
parameters equal to those employed in <code>TicketId</code> generation, only the label
is different.</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> revealed_vrf_input = vrf_input_from_items(
domain: BYTES(&quot;sassafras-revealed-v1.0&quot;),
data: [
next_epoch.randomness,
BYTES(next_epoch.index),
BYTES(attempt_index)
]
);
revealed_vrf_pre_output = vrf_pre_output(AUTHORITY_SECRET_KEY, revealed_vrf_input);
revealed_seed = vrf_bytes(32, revealed_vrf_input, revealed_vrf_pre_output);
revealed_pub = ed25519_secret_from_seed(revealed_seed).public();
<span class="boring">}</span></code></pre></pre>
<p>Where:</p>
<ul>
<li><code>next_epoch.randomness</code>: randomness associated to the target epoch.</li>
<li><code>next_epoch.index</code>: index of the target epoch as a <code>U64</code>.</li>
<li><code>attempt_index</code>: value going from <code>0</code> to <code>attempts_number</code> as a <code>U32</code>.</li>
</ul>
<p>The ephemeral public keys are also used for claiming the tickets on block production.
Refer to section <a href="stale/0026-sassafras-consensus.html#65-slot-claim-production">6.5</a> for details.</p>
<h4 id="624-ring-signature-production"><a class="header" href="#624-ring-signature-production">6.2.4. Ring Signature Production</a></h4>
<p><code>TicketBody</code> must be signed using the Bandersnatch ring VRF flavor (<a href="stale/0026-sassafras-consensus.html#542-ring-vrf-signature">5.4.2</a>).</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> sign_data = vrf_signature_data(
transcript_label: BYTES(&quot;sassafras-ticket-body-v1.0&quot;),
transcript_data: [
SCALE(ticket_body)
],
inputs: [
ticket_id_vrf_input
]
)
ring_signature = ring_vrf_sign(AUTHORITY_SECRET_KEY, RING_PROVER_KEY, sign_data)
<span class="boring">}</span></code></pre></pre>
<p><code>RING_PROVER_KEY</code> object is constructed using the set of public keys which
belong to the target epoch's authorities and the <em>zk-SNARK</em> context parameters
(for more details refer to the
<a href="https://github.com/w3f/ring-vrf/blob/18614458ca4cb335c88d4e710c13906a76f51e43/bandersnatch_vrfs/src/ring.rs#L91-L93">bandersnatch_vrfs</a>
reference implementation).</p>
<p>The body and the ring signature are combined in the <code>TicketEnvelope</code> structure:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> TicketEnvelope ::= SEQUENCE {
ticket_body: TicketBody,
ring_signature: RingVrfSignature
}
<span class="boring">}</span></code></pre></pre>
<p>All the envelopes corresponding to valid tickets can be submitted on-chain via a
dedicated on-chain call (extrinsic).</p>
<h3 id="63-validation-of-candidate-tickets"><a class="header" href="#63-validation-of-candidate-tickets">6.3. Validation of candidate tickets</a></h3>
<p>All the actions in the steps described by this paragraph are executed by
on-chain code.</p>
<p>Validation rules:</p>
<ul>
<li>Tickets submissions must occur within a block part of the first half of the epoch.</li>
<li>Ring signature is verified using the on-chain <code>RING_VERIFIER_KEY</code>.</li>
<li>Ticket identifier is locally (re)computed from the <code>VrfPreOutput</code> contained in the
<code>RingVrfSignature</code> and its value is checked to be less than the tickets' threshold.</li>
</ul>
<p>Valid tickets bodies are all persisted on-chain.</p>
<h3 id="64-ticket-slot-binding"><a class="header" href="#64-ticket-slot-binding">6.4. Ticket-Slot Binding</a></h3>
<p>Before the beginning of the next epoch, the on-chain list of tickets must be
associated with the next epoch's slots such that there must be at most one
ticket per slot.</p>
<p>The assignment process happens in the second half of the submission epoch and
follows these steps:</p>
<ul>
<li>Sorting: The complete list of tickets is sorted based on their <code>TicketId</code>
value, with smaller values coming first.</li>
<li>Trimming: In scenarios where there are more tickets than available slots, the
list is trimmed to fit the epoch's slots by removing the larger value.</li>
<li>Assignment: Tickets are assigned to the epoch's slots following an
<em>outside-in</em> strategy.</li>
</ul>
<h4 id="641-outside-in-assignment"><a class="header" href="#641-outside-in-assignment">6.4.1. Outside-In Assignment</a></h4>
<p>Given an ordered sequence of tickets <code>[t0, t1, t2, ..., tk]</code> to be assigned to
<code>n</code> slots, where <code>n ≥ k</code>, the tickets are allocated according to the following
strategy:</p>
<pre><code> slot-index : [ 0, 1, 2, ............ , n ]
tickets : [ t1, t3, t5, ... , t4, t2, t0 ]
</code></pre>
<p>Here <code>slot-index</code> is a relative value computed as:</p>
<pre><code>slot-index = absolute_slot - epoch_start_slot
</code></pre>
<p>The association between each ticket and a slot is recorded on-chain and thus
is public. What remains confidential is the identity of the ticket's author, and
consequently, who possesses the authority to claim the corresponding slot. This
information is known only to the author of the ticket.</p>
<p>In case the number of available tickets is less than the number of epoch slots,
some <em>orphan</em> slots in the middle of the epoch will remain unbounded to any
ticket. For claiming strategy refer to <a href="stale/0026-sassafras-consensus.html#652-secondary-method">6.5.2</a>.</p>
<h3 id="65-slot-claim-production"><a class="header" href="#65-slot-claim-production">6.5. Slot Claim Production</a></h3>
<p>With tickets bound to epoch slots, every validator acquires information about
the slots for which they are supposed to produce a block.</p>
<p>The procedure for slot claiming depends on whether a given slot has an
associated ticket according to the on-chain state.</p>
<p>If a slot is associated with a ticket, the primary authoring method is used.
Conversely, the protocol resorts to the secondary method as a fallback.</p>
<h4 id="651-primary-method"><a class="header" href="#651-primary-method">6.5.1. Primary Method</a></h4>
<p>Let <code>ticket_body</code> be the <code>TicketBody</code> that has been committed to the on-chain
state, <code>curr_epoch</code> denote an object containing information about the current
epoch, and <code>slot</code> represent the slot number (absolute).</p>
<p>Follows the construction of <code>VrfSignatureData</code>:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> randomness_vrf_input = vrf_input_from_items(
domain: BYTES(&quot;sassafras-randomness-v1.0&quot;),
data: [
curr_epoch.randomness,
BYTES(curr_epoch.index),
BYTES(slot)
]
);
revealed_vrf_input = vrf_input_from_items(
domain: BYTES(&quot;sassafras-revealed-v1.0&quot;),
data: [
curr_epoch.randomness,
BYTES(curr_epoch.index),
BYTES(ticket_body.attempt_index)
]
);
sign_data = vrf_signature_data(
transcript_label: BYTES(&quot;sassafras-claim-v1.0&quot;),
transcript_data: [
SCALE(ticket_body)
],
inputs: [
randomness_vrf_input,
revealed_vrf_input
]
);
<span class="boring">}</span></code></pre></pre>
<h5 id="6511-ephemeral-key-claim"><a class="header" href="#6511-ephemeral-key-claim">6.5.1.1. Ephemeral Key Claim</a></h5>
<p><em>Fiat-Shamir</em> transform is used to obtain a 32-byte challenge associated with
the <code>VrfSignData</code> transcript.</p>
<p>Validators employ the secret key associated with <code>erased_pub</code>, which has been
committed in the <code>TicketBody</code>, to sign the challenge.</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> challenge = sign_data.transcript.challenge();
erased_signature = ed25519_sign(ERASED_SECRET_KEY, challenge);
<span class="boring">}</span></code></pre></pre>
<p>As ticket's ownership can be claimed by reconstructing the <code>revealed_pub</code> entry
of the committed <code>TicketBody</code>, this step is considered optional.</p>
<TODO>
Is this step really necessary?
- Isn't better to keep it simple if this step doesn't offer any extra security?
- We already have a strong method to claim ticket ownership using the vrf output
- What if a validator provides both the proofs?
More weight for the branch (i.e. used to decide what is the best branch by validators)?
E.g.
- primary method + ed25519 erased signature => score 2
- primary method => score 1
- fallback method => score 0
</TODO>
<h4 id="652-secondary-method"><a class="header" href="#652-secondary-method">6.5.2. Secondary Method</a></h4>
<p>By noting that the authorities registered on-chain are kept in an ordered list,
the index of the authority which has the privilege to claim an <em>orphan</em> slot is:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> index_bytes = BLAKE2(4, CONCAT(epoch_randomness, BYTES(slot)));
index = U32(index_bytes) mod authorities_number;
<span class="boring">}</span></code></pre></pre>
<p>Given <code>randomness_vrf_input</code> constructed as shown for the primary method (<a href="stale/0026-sassafras-consensus.html#651-primary-method">6.5.1</a>),
the <code>VrfSignatureData</code> is constructed as:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> sign_data = vrf_signature_data(
transcript_label: BYTES(&quot;sassafras-claim-v1.0&quot;),
transcript_data: [ ],
inputs: [
randomness_vrf_input
]
)
<span class="boring">}</span></code></pre></pre>
<h4 id="653-slot-claim-object"><a class="header" href="#653-slot-claim-object">6.5.3. Slot Claim Object</a></h4>
<p>The <code>SlotClaim</code> structure is used to contain all the necessary information to
assess ownership of a slot.</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> SlotClaim ::= SEQUENCE {
authority_index: U32,
slot: U64,
signature: VrfSignature,
erased_signature: Ed25519Signature OPTIONAL
}
<span class="boring">}</span></code></pre></pre>
<p>The claim is constructed as follows:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> signature = vrf_sign(AUTHORITY_SECRET_KEY, sign_data);
claim = SlotClaim {
authority_index,
slot,
signature,
erased_signature
}
<span class="boring">}</span></code></pre></pre>
<p>Where:</p>
<ul>
<li><code>authority_index</code>: index of the block author in the on-chain authorities list.</li>
<li><code>slot</code>: slot number (absolute, not relative to the epoch start)</li>
<li><code>signature</code>: signature relative to the <code>sign_data</code> constructed via the
primary <a href="stale/0026-sassafras-consensus.html#651-primary-method">6.5.1</a> or secondary (<a href="stale/0026-sassafras-consensus.html#652-secondary-method">6.5.2</a>) method.</li>
<li><code>erased_signature</code>: optional signature providing an additional proof of ticket
ownership (<a href="stale/0026-sassafras-consensus.html#6511-ephemeral-key-claim">6.5.1.1</a>).</li>
</ul>
<p>The signature includes one or two <code>VrfPreOutputs</code>.</p>
<ul>
<li>The first is always present and is used to generate per-block randomness
to feed the randomness accumulator (<a href="stale/0026-sassafras-consensus.html#67-randomness-accumulator">6.7</a>).</li>
<li>The second is included if the slot is bound to a ticket. This is relevant to
claim ticket ownership (<a href="stale/0026-sassafras-consensus.html#661-primary-method">6.6.1</a>).</li>
</ul>
<p>The <code>claim</code> object is <em>SCALE</em> encoded and sent in the block's header digest log.</p>
<h3 id="66-slot-claim-verification"><a class="header" href="#66-slot-claim-verification">6.6. Slot Claim Verification</a></h3>
<p>The signature within the <code>SlotClaim</code> is verified using a <code>VrfSignData</code>
constructed as specified in <a href="stale/0026-sassafras-consensus.html#65-slot-claim-production">6.5</a>.</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> public_key = authorities[claim.authority_index];
result = vrf_verify(public_key, sign_data, claim.signature);
assert(result == true);
<span class="boring">}</span></code></pre></pre>
<p>With:</p>
<ul>
<li><code>authorities</code>: list of authorities for the epoch, as recorded on-chain.</li>
<li><code>sign_data</code>: data that has been signed, constructed as specified in <a href="stale/0026-sassafras-consensus.html#65-slot-claim-production">6.5</a>.</li>
</ul>
<p>If signature verification is successful, the validation process then diverges
based on whether the slot is associated with a ticket according to the on-chain
state.</p>
<p>For slots tied to a ticket, the primary verification method is employed. Otherwise,
the secondary method is utilized.</p>
<h3 id="661-primary-method"><a class="header" href="#661-primary-method">6.6.1. Primary Method</a></h3>
<p>This method verifies ticket ownership using the second <code>VrfPreOutput</code> from the
<code>SlotClaim</code> signature</p>
<p>The process involves comparing the <code>revealed_pub</code> key from the committed
<code>TicketBody</code> with a reconstructed key using the <code>VrfPreOutput</code> and the expected
<code>VrfInput</code>. A mismatch indicates an illegitimate claim.</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> revealed_vrf_input = vrf_input_from_items(
domain: BYTES(&quot;sassafras-revealed-v1.0&quot;),
data: [
curr_epoch.randomness,
BYTES(curr_epoch.index),
BYTES(ticket_body.attempt_index)
]
);
reveled_vrf_pre_output = claim.signature.pre_outputs[1];
revealed_seed = vrf_bytes(32, revealed_vrf_input, revealed_vrf_pre_output);
revealed_pub = ed25519_secret_from_seed(revealed_seed).public();
assert(revealed_pub == ticket_body.revealed_pub);
<span class="boring">}</span></code></pre></pre>
<h5 id="6611-ephemeral-key-signature-check"><a class="header" href="#6611-ephemeral-key-signature-check">6.6.1.1. Ephemeral Key Signature Check</a></h5>
<p>If the <code>erased_signature</code> is present in <code>SlotClaim</code>, the <code>erased_pub</code> within the
committed <code>TicketBody</code> key is used to verify it.</p>
<p>The signed challenge is generated as outlined in section <a href="stale/0026-sassafras-consensus.html#6511-ephemeral-key-claim">6.5.1.1</a>.</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> challenge = sign_data.transcript.challenge();
result = ed25519_verify(ticket_body.erased_pub, challenge, claim.erased_signature);
assert(result == true);
<span class="boring">}</span></code></pre></pre>
<h4 id="662-secondary-method"><a class="header" href="#662-secondary-method">6.6.2. Secondary Method</a></h4>
<p>If the slot doesn't have any associated ticket then the validator index contained in
the claim should match the one given by the rule outlined in section <a href="stale/0026-sassafras-consensus.html#652-secondary-method">6.5.2</a>.</p>
<h3 id="67-randomness-accumulator"><a class="header" href="#67-randomness-accumulator">6.7. Randomness Accumulator</a></h3>
<p>The first <code>VrfPreOutput</code> which ships within the block's <code>SlotClaim</code> signature
is mandatory and must be used as entropy source for the randomness which gets
accumulated on-chain <strong>after</strong> block transactions execution.</p>
<p>Given <code>claim</code> the instance of <code>SlotClaim</code> found within the block header, and
<code>randomness_accumulator</code> the current value for the randomness accumulator, the
<code>randomness_accumulator</code> value is updated as follows:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span> randomness_vrf_input = vrf_input_from_items(
domain: BYTES(&quot;sassafras-randomness-v1.0&quot;),
data: [
curr_epoch.randomness,
BYTES(curr_epoch.index),
BYTES(slot)
]
);
randomness_vrf_pre_output = claim.signature.pre_outputs[0];
randomness = vrf_bytes(32, randomness_vrf_input, randomness_vrf_pre_output);
randomness_accumulator = BLAKE2(32, CONCAT(randomness_accumulator, randomness));
<span class="boring">}</span></code></pre></pre>
<p>The <code>randomness_accumulator</code> never resets and is a continuously evolving value.
It primarily serves as a basis for calculating the randomness associated to the
epochs as outlined on section <a href="stale/0026-sassafras-consensus.html#61-epochs-first-block">6.1</a>, but custom usages
from the user are not excluded.</p>
<h2 id="7-drawbacks"><a class="header" href="#7-drawbacks">7. Drawbacks</a></h2>
<p>None</p>
<h2 id="8-testing-security-and-privacy"><a class="header" href="#8-testing-security-and-privacy">8. Testing, Security, and Privacy</a></h2>
<p>It is critical that implementations of this RFC undergo thorough testing on
test networks.</p>
<p>A security audit may be desirable to ensure the implementation does not
introduce unwanted side effects.</p>
<h2 id="9-performance-ergonomics-and-compatibility"><a class="header" href="#9-performance-ergonomics-and-compatibility">9. Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="91-performance"><a class="header" href="#91-performance">9.1. Performance</a></h3>
<p>Adopting Sassafras consensus marks a significant improvement in reducing the
frequency of short-lived forks.</p>
<p>Forks are eliminated by design. Forks may only result from network disruptions
or protocol attacks. In such cases, the choice of which fork to follow upon
recovery is clear-cut, with only one valid option.</p>
<h3 id="92-ergonomics"><a class="header" href="#92-ergonomics">9.2. Ergonomics</a></h3>
<p>No specific considerations.</p>
<h3 id="93-compatibility"><a class="header" href="#93-compatibility">9.3. Compatibility</a></h3>
<p>The adoption of Sassafras affects the native client and thus can't be introduced
just via a runtime upgrade.</p>
<p>A deployment strategy should be carefully engineered for live networks.</p>
<p>This subject is left open for a dedicated RFC.</p>
<h2 id="10-prior-art-and-references"><a class="header" href="#10-prior-art-and-references">10. Prior Art and References</a></h2>
<ul>
<li><a href="https://research.web3.foundation/Polkadot/protocols/block-production/SASSAFRAS">Web3 Foundation research page</a></li>
<li><a href="https://eprint.iacr.org/2023/031.pdf">Sassafras research paper</a></li>
<li><a href="https://eprint.iacr.org/2023/002.pdf">Ring-VRF research paper</a></li>
<li><a href="https://github.com/paritytech/substrate/issues/11515">Sassafras reference implementation tracking issue</a></li>
<li><a href="https://github.com/paritytech/substrate/pull/11879">Sassafras reference implementation main PR</a></li>
<li><a href="https://github.com/w3f/ring-vrf/tree/master/bandersnatch_vrfs">Bandersnatch VRFS reference implementation</a></li>
</ul>
<TODO replace bandersnatch-vrfs with docs.rs link once published />
<h2 id="11-unresolved-questions"><a class="header" href="#11-unresolved-questions">11. Unresolved Questions</a></h2>
<p>None</p>
<h2 id="12-future-directions-and-related-material"><a class="header" href="#12-future-directions-and-related-material">12. Future Directions and Related Material</a></h2>
<p>While this RFC lays the groundwork and outlines the core aspects of the
protocol, several crucial topics remain to be addressed in future RFCs.
These include:</p>
<h3 id="121-interactions-with-on-chain-code"><a class="header" href="#121-interactions-with-on-chain-code">12.1. Interactions with On-Chain Code</a></h3>
<ul>
<li>
<p><strong>Outbound Interfaces</strong>: Interfaces that the host environment provides to the
on-chain code, typically known as <em>Host Functions</em>.</p>
</li>
<li>
<p><strong>Unrecorded Inbound Interfaces</strong>. Interfaces that the on-chain code provides
to the host code, typically known as <em>Runtime APIs</em>.</p>
</li>
<li>
<p><strong>Transactional Inbound Interfaces</strong>. Interfaces that the on-chain code provides
to the world to alter the chain state, typically known as <em>Transactions</em>
(or <em>extrinsics</em> in the <em>Polkadot</em> ecosystem)</p>
</li>
</ul>
<h3 id="122-deployment-strategies"><a class="header" href="#122-deployment-strategies">12.2. Deployment Strategies</a></h3>
<ul>
<li><strong>Protocol Migration</strong>. Exploring how this protocol can seamlessly replace an
already operational instance of another protocol. Future RFCs should focus on
deployment strategies to facilitate a smooth transition.</li>
</ul>
<h3 id="123-zk-snark-srs-initialization"><a class="header" href="#123-zk-snark-srs-initialization">12.3. ZK-SNARK SRS Initialization</a></h3>
<ul>
<li>
<p><strong>Procedure</strong>: Determining the procedure for the <em>zk-SNARK</em> SRS (Structured
Reference String) initialization. Future RFCs should provide insights into
whether this process should include an ad-hoc initialization ceremony or if
we can reuse an SRS from another ecosystem (e.g. Zcash or Ethereum).</p>
</li>
<li>
<p><strong>Sharing with Para-chains</strong>: Considering the complexity of the process, we
must understand whether the SRS is shared with system para-chains or
maintained independently.</p>
</li>
</ul>
<h3 id="124-anonymous-submission-of-tickets"><a class="header" href="#124-anonymous-submission-of-tickets">12.4. Anonymous Submission of Tickets.</a></h3>
<ul>
<li><strong>Mixnet Integration</strong>: Submitting tickets directly can pose a risk of
potential deanonymization through traffic analysis. Subsequent RFCs should
investigate the potential for incorporating Mixnet protocol or other
privacy-enhancing mechanisms to address this concern.</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/34">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="stale/0034-xcm-absolute-location-account-derivation.html#rfc-34-xcm-absolute-location-account-derivation">RFC-34: XCM Absolute Location Account Derivation</a>
<ul>
<li><a href="stale/0034-xcm-absolute-location-account-derivation.html#summary">Summary</a></li>
<li><a href="stale/0034-xcm-absolute-location-account-derivation.html#motivation">Motivation</a></li>
<li><a href="stale/0034-xcm-absolute-location-account-derivation.html#stakeholders">Stakeholders</a></li>
<li><a href="stale/0034-xcm-absolute-location-account-derivation.html#explanation">Explanation</a></li>
<li><a href="stale/0034-xcm-absolute-location-account-derivation.html#drawbacks">Drawbacks</a></li>
<li><a href="stale/0034-xcm-absolute-location-account-derivation.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="stale/0034-xcm-absolute-location-account-derivation.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="stale/0034-xcm-absolute-location-account-derivation.html#performance">Performance</a></li>
<li><a href="stale/0034-xcm-absolute-location-account-derivation.html#ergonomics">Ergonomics</a></li>
<li><a href="stale/0034-xcm-absolute-location-account-derivation.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="stale/0034-xcm-absolute-location-account-derivation.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="stale/0034-xcm-absolute-location-account-derivation.html#unresolved-questions">Unresolved Questions</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-34-xcm-absolute-location-account-derivation"><a class="header" href="#rfc-34-xcm-absolute-location-account-derivation">RFC-34: XCM Absolute Location Account Derivation</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>05 October 2023</td></tr>
<tr><td><strong>Description</strong></td><td>XCM Absolute Location Account Derivation</td></tr>
<tr><td><strong>Authors</strong></td><td>Gabriel Facco de Arruda</td></tr>
</tbody></table>
</div>
<h2 id="summary-31"><a class="header" href="#summary-31">Summary</a></h2>
<p>This RFC proposes changes that enable the use of absolute locations in AccountId derivations, which allows protocols built using XCM to have static account derivations in any runtime, regardless of its position in the family hierarchy.</p>
<h2 id="motivation-31"><a class="header" href="#motivation-31">Motivation</a></h2>
<p>These changes would allow protocol builders to leverage absolute locations to maintain the exact same derived account address across all networks in the ecosystem, thus enhancing user experience.</p>
<p>One such protocol, that is the original motivation for this proposal, is InvArch's Saturn Multisig, which gives users a unifying multisig and DAO experience across all XCM connected chains.</p>
<h2 id="stakeholders-30"><a class="header" href="#stakeholders-30">Stakeholders</a></h2>
<ul>
<li>Ecosystem developers</li>
</ul>
<h2 id="explanation-31"><a class="header" href="#explanation-31">Explanation</a></h2>
<p>This proposal aims to make it possible to derive accounts for absolute locations, enabling protocols that require the ability to maintain the same derived account in any runtime. This is done by deriving accounts from the hash of described absolute locations, which are static across different destinations.</p>
<p>The same location can be represented in relative form and absolute form like so:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>// Relative location (from own perspective)
{
parents: 0,
interior: Here
}
// Relative location (from perspective of parent)
{
parents: 0,
interior: [Parachain(1000)]
}
// Relative location (from perspective of sibling)
{
parents: 1,
interior: [Parachain(1000)]
}
// Absolute location
[GlobalConsensus(Kusama), Parachain(1000)]
<span class="boring">}</span></code></pre></pre>
<p>Using <code>DescribeFamily</code>, the above relative locations would be described like so:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>// Relative location (from own perspective)
// Not possible.
// Relative location (from perspective of parent)
(b&quot;ChildChain&quot;, Compact::&lt;u32&gt;::from(*index)).encode()
// Relative location (from perspective of sibling)
(b&quot;SiblingChain&quot;, Compact::&lt;u32&gt;::from(*index)).encode()
<span class="boring">}</span></code></pre></pre>
<p>The proposed description for absolute location would follow the same pattern, like so:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>(
b&quot;GlobalConsensus&quot;,
network_id,
b&quot;Parachain&quot;,
Compact::&lt;u32&gt;::from(para_id),
tail
).encode()
<span class="boring">}</span></code></pre></pre>
<p>This proposal requires the modification of two XCM types defined in the <code>xcm-builder</code> crate: The <code>WithComputedOrigin</code> barrier and the <code>DescribeFamily</code> MultiLocation descriptor.</p>
<h4 id="withcomputedorigin"><a class="header" href="#withcomputedorigin">WithComputedOrigin</a></h4>
<p>The <code>WtihComputedOrigin</code> barrier serves as a wrapper around other barriers, consuming origin modification instructions and applying them to the message origin before passing to the inner barriers. One of the origin modifying instructions is <code>UniversalOrigin</code>, which serves the purpose of signaling that the origin should be a Universal Origin that represents the location as an absolute path prefixed by the <code>GlobalConsensus</code> junction.</p>
<p>In it's current state the barrier transforms locations with the <code>UniversalOrigin</code> instruction into relative locations, so the proposed changes aim to make it return absolute locations instead.</p>
<h4 id="describefamily"><a class="header" href="#describefamily">DescribeFamily</a></h4>
<p>The <code>DescribeFamily</code> location descriptor is part of the <code>HashedDescription</code> MultiLocation hashing system and exists to describe locations in an easy format for encoding and hashing, so that an AccountId can be derived from this MultiLocation.</p>
<p>This implementation contains a match statement that does not match against absolute locations, so changes to it involve matching against absolute locations and providing appropriate descriptions for hashing.</p>
<h2 id="drawbacks-26"><a class="header" href="#drawbacks-26">Drawbacks</a></h2>
<p>No drawbacks have been identified with this proposal.</p>
<h2 id="testing-security-and-privacy-26"><a class="header" href="#testing-security-and-privacy-26">Testing, Security, and Privacy</a></h2>
<p>Tests can be done using simple unit tests, as this is not a change to XCM itself but rather to types defined in <code>xcm-builder</code>.</p>
<p>Security considerations should be taken with the implementation to make sure no unwanted behavior is introduced.</p>
<p>This proposal does not introduce any privacy considerations.</p>
<h2 id="performance-ergonomics-and-compatibility-26"><a class="header" href="#performance-ergonomics-and-compatibility-26">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-24"><a class="header" href="#performance-24">Performance</a></h3>
<p>Depending on the final implementation, this proposal should not introduce much overhead to performance.</p>
<h3 id="ergonomics-21"><a class="header" href="#ergonomics-21">Ergonomics</a></h3>
<p>The ergonomics of this proposal depend on the final implementation details.</p>
<h3 id="compatibility-20"><a class="header" href="#compatibility-20">Compatibility</a></h3>
<p>Backwards compatibility should remain unchanged, although that depend on the final implementation.</p>
<h2 id="prior-art-and-references-26"><a class="header" href="#prior-art-and-references-26">Prior Art and References</a></h2>
<ul>
<li><code>DescirbeFamily</code> type: https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/xcm/xcm-builder/src/location_conversion.rs#L122</li>
<li><code>WithComputedOrigin</code> type: https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/xcm/xcm-builder/src/barriers.rs#L153</li>
</ul>
<h2 id="unresolved-questions-26"><a class="header" href="#unresolved-questions-26">Unresolved Questions</a></h2>
<p>Implementation details and overall code is still up to discussion.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/35">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="stale/0035-conviction-voting-delegation-modifications.html#rfc-0035-conviction-voting-delegation-modifications">RFC-0035: Conviction Voting Delegation Modifications</a>
<ul>
<li><a href="stale/0035-conviction-voting-delegation-modifications.html#summary">Summary</a></li>
<li><a href="stale/0035-conviction-voting-delegation-modifications.html#motivation">Motivation</a></li>
<li><a href="stale/0035-conviction-voting-delegation-modifications.html#stakeholders">Stakeholders</a></li>
<li><a href="stale/0035-conviction-voting-delegation-modifications.html#explanation">Explanation</a></li>
<li><a href="stale/0035-conviction-voting-delegation-modifications.html#drawbacks">Drawbacks</a></li>
<li><a href="stale/0035-conviction-voting-delegation-modifications.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="stale/0035-conviction-voting-delegation-modifications.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="stale/0035-conviction-voting-delegation-modifications.html#performance">Performance</a></li>
<li><a href="stale/0035-conviction-voting-delegation-modifications.html#ergonomics--compatibility">Ergonomics &amp; Compatibility</a></li>
</ul>
</li>
<li><a href="stale/0035-conviction-voting-delegation-modifications.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="stale/0035-conviction-voting-delegation-modifications.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="stale/0035-conviction-voting-delegation-modifications.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0035-conviction-voting-delegation-modifications"><a class="header" href="#rfc-0035-conviction-voting-delegation-modifications">RFC-0035: Conviction Voting Delegation Modifications</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>October 10, 2023</strong></td><td></td></tr>
<tr><td><strong>Conviction Voting Delegation Modifications</strong></td><td></td></tr>
<tr><td><strong>ChaosDAO</strong></td><td></td></tr>
</tbody></table>
</div>
<h2 id="summary-32"><a class="header" href="#summary-32">Summary</a></h2>
<p>This RFC proposes to make modifications to voting power delegations as part of the Conviction Voting pallet. The changes being proposed include:</p>
<ol>
<li>Allow a Delegator to vote independently of their Delegate if they so desire.</li>
<li>Allow nested delegations – for example Charlie delegates to Bob who delegates to Alice – when Alice votes then both Bob and Charlie vote alongside Alice (in the current implementation Charlie will not vote when Alice votes).</li>
<li>Make a change so that when a delegate votes abstain their delegated votes also vote abstain.</li>
<li>Allow a Delegator to delegate/ undelegate their votes for all tracks with a single call. </li>
</ol>
<h2 id="motivation-32"><a class="header" href="#motivation-32">Motivation</a></h2>
<p>It has become clear since the launch of OpenGov that there are a few common tropes which pop up time and time again:</p>
<ol>
<li>The frequency of referenda is often too high for network participants to have sufficient time to review, comprehend, and ultimately vote on each individual referendum. This means that these network participants end up being inactive in on-chain governance.</li>
<li>There are active network participants who are reviewing every referendum and are providing feedback in an attempt to help make the network thrive – but often time these participants do not control enough voting power to influence the network with their positive efforts.</li>
<li>Delegating votes for all tracks currently requires long batched calls which result in high fees for the Delegator - resulting in a reluctance from many to delegate their votes.</li>
</ol>
<p>We believe (based on feedback from token holders with a larger stake in the network) that if there were some changes made to delegation mechanics, these larger stake holders would be more likely to delegate their voting power to active network participants – thus greatly increasing the support turnout.</p>
<h2 id="stakeholders-31"><a class="header" href="#stakeholders-31">Stakeholders</a></h2>
<p>The primary stakeholders of this RFC are:</p>
<ul>
<li>The Polkadot Technical Fellowship who will have to research and implement the technical aspects of this RFC</li>
<li>DOT token holders in general </li>
</ul>
<h2 id="explanation-32"><a class="header" href="#explanation-32">Explanation</a></h2>
<p>This RFC proposes to make 4 changes to the convictionVoting pallet logic in order to improve the user experience of those delegating their voting power to another account. </p>
<ol>
<li>
<p><strong>Allow a Delegator to vote independently of their Delegate if they so desire</strong> – this would empower network participants to more actively delegate their voting power to active voters, removing the tedious steps of having to undelegate across an entire track every time they do not agree with their delegate's voting direction for a particular referendum.</p>
</li>
<li>
<p><strong>Allow nested delegations – for example Charlie delegates to Bob who delegates to Alice – when Alice votes then both Bob and Charlie vote alongside Alice (in the current runtime Charlie will not vote when Alice votes)</strong> – This would allow network participants who control multiple (possibly derived) accounts to be able to delegate all of their voting power to a single account under their control, which would in turn delegate to a more active voting participant. Then if the delegator wishes to vote independently of their delegate they can control all of their voting power from a single account, which again removes the pain point of having to issue multiple undelegate extrinsics in the event that they disagree with their delegate.</p>
</li>
<li>
<p><strong>Have delegated votes follow their delegates abstain votes</strong> – there are times where delegates may vote abstain on a particular referendum and adding this functionality will increase the support of a particular referendum. It has a secondary benefit of meaning that Validators who are delegating their voting power do not lose points in the 1KV program in the event that their delegate votes abstain (another pain point which may be preventing those network participants from delegating).</p>
</li>
<li>
<p><strong>Allow a Delegator to delegate/ undelegate their votes for all tracks with a single call</strong> - in order to delegate votes across all tracks, a user must batch 15 calls - resulting in high costs for delegation. A single call for <code>delegate_all</code>/ <code>undelegate_all</code> would reduce the complexity and therefore costs of delegations considerably for prospective Delegators.</p>
</li>
</ol>
<h2 id="drawbacks-27"><a class="header" href="#drawbacks-27">Drawbacks</a></h2>
<p>We do not foresee any drawbacks by implementing these changes. If anything we believe that this should help to increase overall voter turnout (via the means of delegation) which we see as a net positive.</p>
<h2 id="testing-security-and-privacy-27"><a class="header" href="#testing-security-and-privacy-27">Testing, Security, and Privacy</a></h2>
<p>We feel that the Polkadot Technical Fellowship would be the most competent collective to identify the testing requirements for the ideas presented in this RFC.</p>
<h2 id="performance-ergonomics-and-compatibility-27"><a class="header" href="#performance-ergonomics-and-compatibility-27">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-25"><a class="header" href="#performance-25">Performance</a></h3>
<p>This change may add extra chain storage requirements on Polkadot, especially with respect to nested delegations. </p>
<h3 id="ergonomics--compatibility-2"><a class="header" href="#ergonomics--compatibility-2">Ergonomics &amp; Compatibility</a></h3>
<p>The change to add nested delegations may affect governance interfaces such as Nova Wallet who will have to apply changes to their indexers to support nested delegations. It may also affect the Polkadot Delegation Dashboard as well as Polkassembly &amp; SubSquare.</p>
<p>We want to highlight the importance for ecosystem builders to create a mechanism for indexers and wallets to be able to understand that changes have occurred such as increasing the pallet version, etc.</p>
<h2 id="prior-art-and-references-27"><a class="header" href="#prior-art-and-references-27">Prior Art and References</a></h2>
<p>N/A</p>
<h2 id="unresolved-questions-27"><a class="header" href="#unresolved-questions-27">Unresolved Questions</a></h2>
<p>N/A</p>
<h2 id="future-directions-and-related-material-21"><a class="header" href="#future-directions-and-related-material-21">Future Directions and Related Material</a></h2>
<p>Additionally we would like to re-open the conversation about the potential for there to be free delegations. This was discussed by Dr Gavin Wood at Sub0 2022 and we feel like this would go a great way towards increasing the amount of network participants that are delegating: https://youtu.be/hSoSA6laK3Q?t=526</p>
<p>Overall, we strongly feel that delegations are a great way to increase voter turnout, and the ideas presented in this RFC would hopefully help in that aspect.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/44">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="stale/0044-rent-based-registration.html#rfc-0044-rent-based-registration-model">RFC-0044: Rent based registration model</a>
<ul>
<li><a href="stale/0044-rent-based-registration.html#summary">Summary</a></li>
<li><a href="stale/0044-rent-based-registration.html#motivation">Motivation</a></li>
<li><a href="stale/0044-rent-based-registration.html#requirements">Requirements</a></li>
<li><a href="stale/0044-rent-based-registration.html#stakeholders">Stakeholders</a></li>
<li><a href="stale/0044-rent-based-registration.html#explanation">Explanation</a>
<ul>
<li><a href="stale/0044-rent-based-registration.html#registering-an-on-demand-parachain">Registering an on-demand parachain</a></li>
<li><a href="stale/0044-rent-based-registration.html#on-demand-parachain-pruning">On-demand parachain pruning</a></li>
<li><a href="stale/0044-rent-based-registration.html#ensuring-rent-is-paid">Ensuring rent is paid</a></li>
<li><a href="stale/0044-rent-based-registration.html#on-demand-para-re-registration">On-demand para re-registration</a></li>
</ul>
</li>
<li><a href="stale/0044-rent-based-registration.html#drawbacks">Drawbacks</a></li>
<li><a href="stale/0044-rent-based-registration.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="stale/0044-rent-based-registration.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="stale/0044-rent-based-registration.html#performance">Performance</a></li>
<li><a href="stale/0044-rent-based-registration.html#ergonomics">Ergonomics</a></li>
<li><a href="stale/0044-rent-based-registration.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="stale/0044-rent-based-registration.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="stale/0044-rent-based-registration.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="stale/0044-rent-based-registration.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0044-rent-based-registration-model"><a class="header" href="#rfc-0044-rent-based-registration-model">RFC-0044: Rent based registration model</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>6 November 2023</td></tr>
<tr><td><strong>Description</strong></td><td>A new rent based parachain registration model</td></tr>
<tr><td><strong>Authors</strong></td><td>Sergej Sakac</td></tr>
</tbody></table>
</div>
<h2 id="summary-33"><a class="header" href="#summary-33">Summary</a></h2>
<p>This RFC proposes a new model for a sustainable on-demand parachain registration, involving a smaller initial deposit and periodic rent payments. The new model considers that on-demand chains may be unregistered and later re-registered. The proposed solution also ensures a quick startup for on-demand chains on Polkadot in such cases.</p>
<h2 id="motivation-33"><a class="header" href="#motivation-33">Motivation</a></h2>
<p>With the support of on-demand parachains on Polkadot, there is a need to explore a new, more cost-effective model for registering validation code. In the current model, the parachain manager is responsible for reserving a unique <code>ParaId</code> and covering the cost of storing the validation code of the parachain. These costs can escalate, particularly if the validation code is large. We need a better, sustainable model for registering on-demand parachains on Polkadot to help smaller teams deploy more easily.</p>
<p>This RFC suggests a new payment model to create a more financially viable approach to on-demand parachain registration. In this model, a lower initial deposit is required, followed by recurring payments upon parachain registration.</p>
<p>This new model will coexist with the existing one-time deposit payment model, offering teams seeking to deploy on-demand parachains on Polkadot a more cost-effective alternative.</p>
<h2 id="requirements-8"><a class="header" href="#requirements-8">Requirements</a></h2>
<ol>
<li>The solution SHOULD NOT affect the current model for registering validation code.</li>
<li>The solution SHOULD offer an easily configurable way for governance to adjust the initial deposit and recurring rent cost.</li>
<li>The solution SHOULD provide an incentive to prune validation code for which rent is not paid.</li>
<li>The solution SHOULD allow anyone to re-register validation code under the same <code>ParaId</code> without the need for redundant pre-checking if it was already verified before.</li>
<li>The solution MUST be compatible with the Agile Coretime model, as described in RFC#0001</li>
<li>The solution MUST allow anyone to pay the rent.</li>
<li>The solution MUST prevent the removal of validation code if it could still be required for disputes or approval checking.</li>
</ol>
<h2 id="stakeholders-32"><a class="header" href="#stakeholders-32">Stakeholders</a></h2>
<ul>
<li>Future Polkadot on-demand Parachains</li>
</ul>
<h2 id="explanation-33"><a class="header" href="#explanation-33">Explanation</a></h2>
<p>This RFC proposes a set of changes that will enable the new rent based approach to registering and storing validation code on-chain.
The new model, compared to the current one, will require periodic rent payments. The parachain won't be pruned automatically if the rent is not paid, but by permitting anyone to prune the parachain and rewarding the caller, there will be an incentive for the removal of the validation code.</p>
<p>On-demand parachains should still be able to utilize the current one-time payment model. However, given the size of the deposit required, it's highly likely that most on-demand parachains will opt for the new rent-based model.</p>
<p>Importantly, this solution doesn't require any storage migrations in the current system nor does it introduce any breaking changes. The following provides a detailed description of this solution.</p>
<h3 id="registering-an-on-demand-parachain"><a class="header" href="#registering-an-on-demand-parachain">Registering an on-demand parachain</a></h3>
<p>In the current implementation of the registrar pallet, there are two constants that specify the necessary deposit for parachains to register and store their validation code:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>trait Config {
// -- snip --
/// The deposit required for reserving a `ParaId`.
#[pallet::constant]
type ParaDeposit: Get&lt;BalanceOf&lt;Self&gt;&gt;;
/// The deposit to be paid per byte stored on chain.
#[pallet::constant]
type DataDepositPerByte: Get&lt;BalanceOf&lt;Self&gt;&gt;;
}
<span class="boring">}</span></code></pre></pre>
<p>This RFC proposes the addition of three new constants that will determine the payment amount and the frequency of the recurring rent payment:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>trait Config {
// -- snip --
/// Defines how frequently the rent needs to be paid.
///
/// The duration is set in sessions instead of block numbers.
#[pallet::constant]
type RentDuration: Get&lt;SessionIndex&gt;;
/// The initial deposit amount for registering validation code.
///
/// This is defined as a proportion of the deposit that would be required in the regular
/// model.
#[pallet::constant]
type RentalDepositProportion: Get&lt;Perbill&gt;;
/// The recurring rental cost defined as a proportion of the initial rental registration deposit.
#[pallet::constant]
type RentalRecurringProportion: Get&lt;Perbill&gt;;
}
<span class="boring">}</span></code></pre></pre>
<p>Users will be able to reserve a <code>ParaId</code> and register their validation code for a proportion of the regular deposit required. However, they must also make additional rent payments at intervals of <code>T::RentDuration</code>.</p>
<p>For registering using the new rental system we will have to make modifications to the <code>paras-registrar</code> pallet. We should expose two new extrinsics for this:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>mod pallet {
// -- snip --
pub fn register_rental(
origin: OriginFor&lt;T&gt;,
id: ParaId,
genesis_head: HeadData,
validation_code: ValidationCode,
) -&gt; DispatchResult { /* ... */ }
pub fn pay_rent(origin: OriginFor&lt;T&gt;, id: ParaId) -&gt; DispatchResult {
/* ... */
}
}
<span class="boring">}</span></code></pre></pre>
<p>A call to <code>register_rental</code> will require the reservation of only a percentage of the deposit that would otherwise be required to register the validation code when using the regular model.
As described later in the <em>Quick para re-registering</em> section below, we will also store the code hash of each parachain to enable faster re-registration after a parachain has been pruned. For this reason the total initial deposit amount is increased to account for that.</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>// The logic for calculating the initial deposit for parachain registered with the
// new rent-based model:
let validation_code_deposit = per_byte_fee.saturating_mul((validation_code.0.len() as u32).into());
let head_deposit = per_byte_fee.saturating_mul((genesis_head.0.len() as u32).into())
let hash_deposit = per_byte_fee.saturating_mul(HASH_SIZE);
let deposit = T::RentalDepositProportion::get().mul_ceil(validation_code_deposit)
.saturating_add(T::ParaDeposit::get())
.saturating_add(head_deposit)
.saturating_add(hash_deposit)
<span class="boring">}</span></code></pre></pre>
<p>Once the <code>ParaId</code> is reserved and the validation code is registered the rent must be periodically paid to ensure the on-demand parachain doesn't get removed from the state. The <code>pay_rent</code> extrinsic should be callable by anyone, removing the need for the parachain to depend on the parachain manager for rent payments.</p>
<h3 id="on-demand-parachain-pruning"><a class="header" href="#on-demand-parachain-pruning">On-demand parachain pruning</a></h3>
<p>If the rent is not paid, anyone has the option to prune the on-demand parachain and claim a portion of the initial deposit reserved for storing the validation code. This type of 'light' pruning only removes the validation code, while the head data and validation code hash are retained. The validation code hash is stored to allow anyone to register it again as well as to enable quicker re-registration by skipping the pre-checking process.</p>
<p>The moment the rent is no longer paid, the parachain won't be able to purchase on-demand access, meaning no new blocks are allowed. This stage is called the &quot;hibernation&quot; stage, during which all the parachain-related data is still stored on-chain, but new blocks are not permitted. The reason for this is to ensure that the validation code is available in case it is needed in the dispute or approval checking subsystems. Waiting for one entire session will be enough to ensure it is safe to deregister the parachain.</p>
<p>This means that anyone can prune the parachain only once the &quot;hibernation&quot; stage is over, which lasts for an entire session after the moment that the rent is not paid.</p>
<p>The pruning described here is a light form of pruning, since it only removes the validation code. As with all parachains, the parachain or para manager can use the <code>deregister</code> extrinsic to remove all associated state.</p>
<h3 id="ensuring-rent-is-paid"><a class="header" href="#ensuring-rent-is-paid">Ensuring rent is paid</a></h3>
<p>The <code>paras</code> pallet will be loosely coupled with the <code>para-registrar</code> pallet. This approach enables all the pallets tightly coupled with the <code>paras</code> pallet to have access to the rent status information.</p>
<p>Once the validation code is stored without having its rent paid the <code>assigner_on_demand</code> pallet will ensure that an order for that parachain cannot be placed. This is easily achievable given that the <code>assigner_on_demand</code> pallet is tightly coupled with the <code>paras</code> pallet.</p>
<h3 id="on-demand-para-re-registration"><a class="header" href="#on-demand-para-re-registration">On-demand para re-registration</a></h3>
<p>If the rent isn't paid on time, and the parachain gets pruned, the new model should provide a quick way to re-register the same validation code under the same <code>ParaId</code>. This can be achieved by skipping the pre-checking process, as the validation code hash will be stored on-chain, allowing us to easily verify that the uploaded code remains unchanged.</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>/// Stores the validation code hash for parachains that successfully completed the
/// pre-checking process.
///
/// This is stored to enable faster on-demand para re-registration in case its pvf has been earlier
/// registered and checked.
///
/// NOTE: During a runtime upgrade where the pre-checking rules change this storage map should be
/// cleared appropriately.
#[pallet::storage]
pub(super) type CheckedCodeHash&lt;T: Config&gt; =
StorageMap&lt;_, Twox64Concat, ParaId, ValidationCodeHash&gt;;
<span class="boring">}</span></code></pre></pre>
<p>To enable parachain re-registration, we should introduce a new extrinsic in the <code>paras-registrar</code> pallet that allows this. The logic of this extrinsic will be same as regular registration, with the distinction that it can be called by anyone, and the required deposit will be smaller since it only has to cover for the storage of the validation code.</p>
<h2 id="drawbacks-28"><a class="header" href="#drawbacks-28">Drawbacks</a></h2>
<p>This RFC does not alter the process of reserving a <code>ParaId</code>, and therefore, it does not propose reducing it, even though such a reduction could be beneficial.</p>
<p>Even though this RFC doesn't delve into the specifics of the configuration values for parachain registration but rather focuses on the mechanism, configuring it carelessly could lead to potential problems.</p>
<p>Since the validation code hash and head data are not removed when the parachain is pruned but only when the <code>deregister</code> extrinsic is called, the <code>T::DataDepositPerByte</code> must be set to a higher value to create a strong enough incentive for removing it from the state.</p>
<h2 id="testing-security-and-privacy-28"><a class="header" href="#testing-security-and-privacy-28">Testing, Security, and Privacy</a></h2>
<p>The implementation of this RFC will be tested on Rococo first.</p>
<p>Proper research should be conducted on setting the configuration values of the new system since these values can have great impact on the network.</p>
<p>An audit is required to ensure the implementation's correctness.</p>
<p>The proposal introduces no new privacy concerns.</p>
<h2 id="performance-ergonomics-and-compatibility-28"><a class="header" href="#performance-ergonomics-and-compatibility-28">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-26"><a class="header" href="#performance-26">Performance</a></h3>
<p>This RFC should not introduce any performance impact.</p>
<h3 id="ergonomics-22"><a class="header" href="#ergonomics-22">Ergonomics</a></h3>
<p>This RFC does not affect the current parachains, nor the parachains that intend to use the one-time payment model for parachain registration.</p>
<h3 id="compatibility-21"><a class="header" href="#compatibility-21">Compatibility</a></h3>
<p>This RFC does not break compatibility.</p>
<h2 id="prior-art-and-references-28"><a class="header" href="#prior-art-and-references-28">Prior Art and References</a></h2>
<p>Prior discussion on this topic: https://github.com/paritytech/polkadot-sdk/issues/1796</p>
<h2 id="unresolved-questions-28"><a class="header" href="#unresolved-questions-28">Unresolved Questions</a></h2>
<p>None at this time.</p>
<h2 id="future-directions-and-related-material-22"><a class="header" href="#future-directions-and-related-material-22">Future Directions and Related Material</a></h2>
<p>As noted in <a href="https://github.com/paritytech/polkadot-sdk/issues/1796">this GitHub issue</a>, we want to raise the per-byte cost of on-chain data storage. However, a substantial increase in this cost would make it highly impractical for on-demand parachains to register on Polkadot.
This RFC offers an alternative solution for on-demand parachains, ensuring that the per-byte cost increase doesn't overly burden the registration process.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/48">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="stale/0048-session-keys-runtime-api.html#rfc-0048-generate-ownership-proof-for-sessionkeys">RFC-0048: Generate ownership proof for <code>SessionKeys</code></a>
<ul>
<li><a href="stale/0048-session-keys-runtime-api.html#summary">Summary</a></li>
<li><a href="stale/0048-session-keys-runtime-api.html#motivation">Motivation</a></li>
<li><a href="stale/0048-session-keys-runtime-api.html#stakeholders">Stakeholders</a></li>
<li><a href="stale/0048-session-keys-runtime-api.html#explanation">Explanation</a></li>
<li><a href="stale/0048-session-keys-runtime-api.html#drawbacks">Drawbacks</a></li>
<li><a href="stale/0048-session-keys-runtime-api.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="stale/0048-session-keys-runtime-api.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="stale/0048-session-keys-runtime-api.html#performance">Performance</a></li>
<li><a href="stale/0048-session-keys-runtime-api.html#ergonomics">Ergonomics</a></li>
<li><a href="stale/0048-session-keys-runtime-api.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="stale/0048-session-keys-runtime-api.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="stale/0048-session-keys-runtime-api.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="stale/0048-session-keys-runtime-api.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0048-generate-ownership-proof-for-sessionkeys"><a class="header" href="#rfc-0048-generate-ownership-proof-for-sessionkeys">RFC-0048: Generate ownership proof for <code>SessionKeys</code></a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>13 November 2023</td></tr>
<tr><td><strong>Description</strong></td><td>Change <code>SessionKeys</code> runtime api to also create a proof of ownership for on chain registration.</td></tr>
<tr><td><strong>Authors</strong></td><td>Bastian Köcher</td></tr>
</tbody></table>
</div>
<h2 id="summary-34"><a class="header" href="#summary-34">Summary</a></h2>
<p>When rotating/generating the <code>SessionKeys</code> of a node, the node calls into the runtime using the
<code>SessionKeys::generate_session_keys</code> runtime api. This runtime api function needs to be changed
to add an extra parameter <code>owner</code> and to change the return value to also include the <code>proof</code> of
ownership. The <code>owner</code> should be the account id of the account setting the <code>SessionKeys</code> on chain
to allow the on chain logic the verification of the proof. The on chain logic is then able to proof
the possession of the private keys of the <code>SessionKeys</code> using the <code>proof</code>.</p>
<h2 id="motivation-34"><a class="header" href="#motivation-34">Motivation</a></h2>
<p>When a user sets new <code>SessionKeys</code> on chain the chain can currently not ensure that the user
actually has control over the private keys of the <code>SessionKeys</code>. With the RFC applied the chain is able
to ensure that the user actually is in possession of the private keys.</p>
<h2 id="stakeholders-33"><a class="header" href="#stakeholders-33">Stakeholders</a></h2>
<ul>
<li>Polkadot runtime implementors</li>
<li>Polkadot node implementors</li>
<li>Validator operators</li>
</ul>
<h2 id="explanation-34"><a class="header" href="#explanation-34">Explanation</a></h2>
<p>We are first going to explain the <code>proof</code> format being used:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>type Proof = (Signature, Signature, ..);
<span class="boring">}</span></code></pre></pre>
<p>The <code>proof</code> being a SCALE encoded tuple over all signatures of each private session
key signing the <code>owner</code>. The actual type of each signature depends on the
corresponding session key cryptographic algorithm. The order of the signatures in
the <code>proof</code> is the same as the order of the session keys in the <code>SessionKeys</code> type.</p>
<p>The version of the <code>SessionKeys</code> needs to be bumped to <code>1</code> to reflect the changes to the
signature of <code>SessionKeys_generate_session_keys</code>:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>pub struct OpaqueGeneratedSessionKeys {
pub keys: Vec&lt;u8&gt;,
pub proof: Vec&lt;u8&gt;,
}
fn SessionKeys_generate_session_keys(owner: Vec&lt;u8&gt;, seed: Option&lt;Vec&lt;u8&gt;&gt;) -&gt; OpaqueGeneratedSessionKeys;
<span class="boring">}</span></code></pre></pre>
<p>The default calling convention for runtime apis is applied, meaning the parameters
passed as SCALE encoded array and the length of the encoded array. The return value
being the SCALE encoded return value as <code>u64</code> (<code>array_ptr | length &lt;&lt; 32</code>). So, the
actual exported function signature looks like:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>fn SessionKeys_generate_session_keys(array: *const u8, len: usize) -&gt; u64;
<span class="boring">}</span></code></pre></pre>
<p>The on chain logic for setting the <code>SessionKeys</code> needs to be changed as well. It
already gets the <code>proof</code> passed as <code>Vec&lt;u8&gt;</code>. This <code>proof</code> needs to be decoded to
the actual <code>Proof</code> type as explained above. The <code>proof</code> and the SCALE encoded
<code>account_id</code> of the sender are used to verify the ownership of the <code>SessionKeys</code>.</p>
<h2 id="drawbacks-29"><a class="header" href="#drawbacks-29">Drawbacks</a></h2>
<p>Validator operators need to pass the their account id when rotating their session keys in a node.
This will require updating some high level docs and making users familiar with the slightly changed ergonomics.</p>
<h2 id="testing-security-and-privacy-29"><a class="header" href="#testing-security-and-privacy-29">Testing, Security, and Privacy</a></h2>
<p>Testing of the new changes is quite easy as it only requires passing an appropriate <code>owner</code>
for the current testing context. The changes to the proof generation and verification got
audited to ensure they are correct.</p>
<h2 id="performance-ergonomics-and-compatibility-29"><a class="header" href="#performance-ergonomics-and-compatibility-29">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-27"><a class="header" href="#performance-27">Performance</a></h3>
<p>Does not have any impact on the overall performance, only setting <code>SessionKeys</code> will require more weight.</p>
<h3 id="ergonomics-23"><a class="header" href="#ergonomics-23">Ergonomics</a></h3>
<p>If the proposal alters exposed interfaces to developers or end-users, which types of usage patterns have been optimized for?</p>
<h3 id="compatibility-22"><a class="header" href="#compatibility-22">Compatibility</a></h3>
<p>Introduces a new version of the <code>SessionKeys</code> runtime api. Thus, nodes should be updated before
a runtime is enacted that contains these changes otherwise they will fail to generate session keys.</p>
<h2 id="prior-art-and-references-29"><a class="header" href="#prior-art-and-references-29">Prior Art and References</a></h2>
<p>None.</p>
<h2 id="unresolved-questions-29"><a class="header" href="#unresolved-questions-29">Unresolved Questions</a></h2>
<p>None.</p>
<h2 id="future-directions-and-related-material-23"><a class="header" href="#future-directions-and-related-material-23">Future Directions and Related Material</a></h2>
<p>Substrate implementation of the <a href="https://github.com/paritytech/polkadot-sdk/pull/1739">RFC</a>.</p>
<div style="break-before: page; page-break-before: always;"></div><p><a href="https://github.com/polkadot-fellows/RFCs/pull/54">(source)</a></p>
<p><strong>Table of Contents</strong></p>
<ul>
<li><a href="stale/0054-remove-heap-pages.html#rfc-0054-remove-the-concept-of-heap-pages-from-the-client">RFC-0054: Remove the concept of &quot;heap pages&quot; from the client</a>
<ul>
<li><a href="stale/0054-remove-heap-pages.html#summary">Summary</a></li>
<li><a href="stale/0054-remove-heap-pages.html#motivation">Motivation</a></li>
<li><a href="stale/0054-remove-heap-pages.html#stakeholders">Stakeholders</a></li>
<li><a href="stale/0054-remove-heap-pages.html#explanation">Explanation</a></li>
<li><a href="stale/0054-remove-heap-pages.html#drawbacks">Drawbacks</a></li>
<li><a href="stale/0054-remove-heap-pages.html#testing-security-and-privacy">Testing, Security, and Privacy</a></li>
<li><a href="stale/0054-remove-heap-pages.html#performance-ergonomics-and-compatibility">Performance, Ergonomics, and Compatibility</a>
<ul>
<li><a href="stale/0054-remove-heap-pages.html#performance">Performance</a></li>
<li><a href="stale/0054-remove-heap-pages.html#ergonomics">Ergonomics</a></li>
<li><a href="stale/0054-remove-heap-pages.html#compatibility">Compatibility</a></li>
</ul>
</li>
<li><a href="stale/0054-remove-heap-pages.html#prior-art-and-references">Prior Art and References</a></li>
<li><a href="stale/0054-remove-heap-pages.html#unresolved-questions">Unresolved Questions</a></li>
<li><a href="stale/0054-remove-heap-pages.html#future-directions-and-related-material">Future Directions and Related Material</a></li>
</ul>
</li>
</ul>
<h1 id="rfc-0054-remove-the-concept-of-heap-pages-from-the-client"><a class="header" href="#rfc-0054-remove-the-concept-of-heap-pages-from-the-client">RFC-0054: Remove the concept of &quot;heap pages&quot; from the client</a></h1>
<div class="table-wrapper"><table><thead><tr><th></th><th></th></tr></thead><tbody>
<tr><td><strong>Start Date</strong></td><td>2023-11-24</td></tr>
<tr><td><strong>Description</strong></td><td>Remove the concept of heap pages from the client and move it to the runtime.</td></tr>
<tr><td><strong>Authors</strong></td><td>Pierre Krieger</td></tr>
</tbody></table>
</div>
<h2 id="summary-35"><a class="header" href="#summary-35">Summary</a></h2>
<p>Rather than enforce a limit to the total memory consumption on the client side by loading the value at <code>:heappages</code>, enforce that limit on the runtime side.</p>
<h2 id="motivation-35"><a class="header" href="#motivation-35">Motivation</a></h2>
<p>From the early days of Substrate up until recently, the runtime was present in two forms: the wasm runtime (wasm bytecode passed through an interpreter) and the native runtime (native code directly run by the client).</p>
<p>Since the wasm runtime has a lower amount of available memory (4 GiB maximum) compared to the native runtime, and in order to ensure sure that the wasm and native runtimes always produce the same outcome, it was necessary to clamp the amount of memory available to both runtimes to the same value.</p>
<p>In order to achieve this, a special storage key (a &quot;well-known&quot; key) <code>:heappages</code> was introduced and represents the number of &quot;wasm pages&quot; (one page equals 64kiB) of memory that are available to the memory allocator of the runtimes. If this storage key is absent, it defaults to 2048, which is 128 MiB.</p>
<p>The native runtime has since then been disappeared, but the concept of &quot;heap pages&quot; still exists. This RFC proposes a simplification to the design of Polkadot by removing the concept of &quot;heap pages&quot; as is currently known, and proposes alternative ways to achieve the goal of limiting the amount of memory available.</p>
<h2 id="stakeholders-34"><a class="header" href="#stakeholders-34">Stakeholders</a></h2>
<p>Client implementers and low-level runtime developers.</p>
<h2 id="explanation-35"><a class="header" href="#explanation-35">Explanation</a></h2>
<p>This RFC proposes the following changes to the client:</p>
<ul>
<li>The client no longer considers <code>:heappages</code> as special.</li>
<li>The memory allocator of the runtime is no longer bounded by the value of <code>:heappages</code>.</li>
</ul>
<p>With these changes, the memory available to the runtime is now only bounded by the available memory space (4 GiB), and optionally by the maximum amount of memory specified in the Wasm binary (see https://webassembly.github.io/spec/core/bikeshed/#memories%E2%91%A0). In Rust, the latter can be controlled during compilation with the flag <code>-Clink-arg=--max-memory=...</code>.</p>
<p>Since the client-side change is strictly more tolerant than before, we can perform the change immediately after the runtime has been updated, and without having to worry about backwards compatibility.</p>
<p>This RFC proposes three alternative paths (different chains might choose to follow different paths):</p>
<ul>
<li>
<p>Path A: add back the same memory limit to the runtime, like so:</p>
<ul>
<li>At initialization, the runtime loads the value of <code>:heappages</code> from the storage (using <code>ext_storage_get</code> or similar), and sets a global variable to the decoded value.</li>
<li>The runtime tracks the total amount of memory that it has allocated using its instance of <code>#[global_allocator]</code> (https://github.com/paritytech/polkadot-sdk/blob/e3242d2c1e2018395c218357046cc88caaed78f3/substrate/primitives/io/src/lib.rs#L1748-L1762). This tracking should also be added around the host functions that perform allocations.</li>
<li>If an allocation is attempted that would go over the value in the global variable, the memory allocation fails.</li>
</ul>
</li>
<li>
<p>Path B: define the memory limit using the <code>-Clink-arg=--max-memory=...</code> flag.</p>
</li>
<li>
<p>Path C: don't add anything to the runtime. This is effectively the same as setting the memory limit to ~4 GiB (compared to the current default limit of 128 MiB). This solution is viable only because we're compiling for 32bits wasm rather than for example 64bits wasm. If we ever compile for 64bits wasm, this would need to be revisited.</p>
</li>
</ul>
<p>Each parachain can choose the option that they prefer, but the author of this RFC strongly suggests either option C or B.</p>
<h2 id="drawbacks-30"><a class="header" href="#drawbacks-30">Drawbacks</a></h2>
<p>In case of path A, there is one situation where the behaviour pre-RFC is not equivalent to the one post-RFC: when a host function that performs an allocation (for example <code>ext_storage_get</code>) is called, without this RFC this allocation might fail due to reaching the maximum heap pages, while after this RFC this will always succeed.
This is most likely not a problem, as storage values aren't supposed to be larger than a few megabytes at the very maximum.</p>
<p>In the unfortunate event where the runtime runs out of memory, path B would make it more difficult to relax the memory limit, as we would need to re-upload the entire Wasm, compared to updating only <code>:heappages</code> in path A or before this RFC.
In the case where the runtime runs out of memory only in the specific event where the Wasm runtime is modified, this could brick the chain. However, this situation is no different than the thousands of other ways that a bug in the runtime can brick a chain, and there's no reason to be particularily worried about this situation in particular.</p>
<h2 id="testing-security-and-privacy-30"><a class="header" href="#testing-security-and-privacy-30">Testing, Security, and Privacy</a></h2>
<p>This RFC would reduce the chance of a consensus issue between clients.
The <code>:heappages</code> are a rather obscure feature, and it is not clear what happens in some corner cases such as the value being too large (error? clamp?) or malformed. This RFC would completely erase these questions.</p>
<h2 id="performance-ergonomics-and-compatibility-30"><a class="header" href="#performance-ergonomics-and-compatibility-30">Performance, Ergonomics, and Compatibility</a></h2>
<h3 id="performance-28"><a class="header" href="#performance-28">Performance</a></h3>
<p>In case of path A, it is unclear how performances would be affected. Path A consists in moving client-side operations to the runtime without changing these operations, and as such performance differences are expected to be minimal. Overall, we're talking about one addition/subtraction per malloc and per free, so this is more than likely completely negligible.</p>
<p>In case of path B and C, the performance gain would be a net positive, as this RFC strictly removes things.</p>
<h3 id="ergonomics-24"><a class="header" href="#ergonomics-24">Ergonomics</a></h3>
<p>This RFC would isolate the client and runtime more from each other, making it a bit easier to reason about the client or the runtime in isolation.</p>
<h3 id="compatibility-23"><a class="header" href="#compatibility-23">Compatibility</a></h3>
<p>Not a breaking change. The runtime-side changes can be applied immediately (without even having to wait for changes in the client), then as soon as the runtime is updated, the client can be updated without any transition period. One can even consider updating the client before the runtime, as it corresponds to path C.</p>
<h2 id="prior-art-and-references-30"><a class="header" href="#prior-art-and-references-30">Prior Art and References</a></h2>
<p>None.</p>
<h2 id="unresolved-questions-30"><a class="header" href="#unresolved-questions-30">Unresolved Questions</a></h2>
<p>None.</p>
<h2 id="future-directions-and-related-material-24"><a class="header" href="#future-directions-and-related-material-24">Future Directions and Related Material</a></h2>
<p>This RFC follows the same path as https://github.com/polkadot-fellows/RFCs/pull/4 by scoping everything related to memory allocations to the runtime.</p>
</main>
<nav class="nav-wrapper" aria-label="Page navigation">
<!-- Mobile navigation buttons -->
<div style="clear: both"></div>
</nav>
</div>
</div>
<nav class="nav-wide-wrapper" aria-label="Page navigation">
</nav>
</div>
<script>
window.playground_copyable = true;
</script>
<script src="elasticlunr.min.js"></script>
<script src="mark.min.js"></script>
<script src="searcher.js"></script>
<script src="clipboard.min.js"></script>
<script src="highlight.js"></script>
<script src="book.js"></script>
<!-- Custom JS scripts -->
<script>
window.addEventListener('load', function() {
window.setTimeout(window.print, 100);
});
</script>
</div>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink