Code coverage (#529)

Adds `make coverage` (cargo-llvm-cov over the full workspace including
revive-llvm-builder), a new "Code coverage" book chapter that briefly
explains how to run it, and a PR workflow that runs coverage and
comments on the PR (it updates the same comment if it already exists)
when the `measure code coverage` label is applied (see e.g. [this
comment](https://github.com/paritytech/revive/pull/529#issuecomment-4966136441)).

Building instrumented LLVM and generating an LLVM C++ coverage report is
also possible, see accompanying [PR description
#575](https://github.com/paritytech/revive/pull/575) for more details.

---------

Signed-off-by: Cyrill Leutwiler <bigcyrill@hotmail.com>
Signed-off-by: xermicus <bigcyrill@hotmail.com>
Co-authored-by: Cyrill Leutwiler <bigcyrill@hotmail.com>
Co-authored-by: xermicus <cyrill@parity.io>
Co-authored-by: elle-j <linnea@parity.io>
Co-authored-by: LJ <81748770+elle-j@users.noreply.github.com>
This commit is contained in:
authored and GitHub committed 2026-07-16 14:48:44 +02:00
1 parent ca3c770228
commit ed95ffbdfe
40 files changed
+882 -123

No files matched your search

+64 -2
View File
@@ -36,10 +36,10 @@
const path_to_root = "";
const default_light_theme = "light";
const default_dark_theme = "navy";
window.path_to_searchindex_js = "searchindex-339ac392.js";
window.path_to_searchindex_js = "searchindex-57ec751f.js";
</script>
<!-- Start loading toc.js asap -->
<script src="toc-1290cc30.js"></script>
<script src="toc-f74aae8e.js"></script>
</head>
<body>
<div id="mdbook-help-container">
@@ -4819,6 +4819,68 @@ export PATH=/path/to/go-ethereum/build/bin/:$PATH
<p>A lot of nuanced bugs caused by tiny implementation details inside the <code>revive</code> compiler <em>and</em> the <code>pallet-revive</code> runtime could be identified and eliminated early on thanks to the differential testing strategy. Thus, we decided to take this approach further and created a comprehensive test runner and a large suite of more complex test cases.</p>
<p>The <a href="https://github.com/paritytech/revive-differential-tests/">Revive Differential Tests</a> follow the exact same strategy but implement a much more powerful test spec format, spec runner and reports. This allows differentially testing much more complex test cases (for example testing Uniswap pair creations and swaps), executed via transactions sent to actual blockchain nodes.</p>
<div style="break-before: page; page-break-before: always;"></div>
<h1 id="code-coverage"><a class="header" href="#code-coverage">Code coverage</a></h1>
<p>Revive measures Rust and LLVM C++ line and region coverage with <a href="https://github.com/taiki-e/cargo-llvm-cov"><code>cargo-llvm-cov</code></a> over the same tests as <code>make test-workspace</code>.</p>
<p>Two ways to obtain a report:</p>
<ol>
<li><strong>Locally:</strong> <code>make coverage</code> and <code>make coverage-llvm-report</code> (described below).</li>
<li><strong>Per PR:</strong> apply the <code>measure code coverage</code> label and read the bot-generated PR comment once the triggered workflow completes.</li>
</ol>
<h2 id="running-locally"><a class="header" href="#running-locally">Running locally</a></h2>
<pre><code class="language-bash"># Install LLVM.
make install-llvm
# Or install instrumented LLVM.
make install-llvm-coverage
# ..set LLVM_SYS_221_PREFIX to a compatible LLVM build..
# Run coverage and generate revive Rust report.
make coverage
# Generate a separate LLVM C++ report if needed, if
# instrumented LLVM was used for `make coverage`.
make coverage-llvm-report
# Print clickable links to all accumulated HTML reports detected
# (they're grouped by timestamps) to easily view them in the browser.
make coverage-browse
</code></pre>
<p>The target:</p>
<ol>
<li>Builds and tests the workspace under cargo-llvm-cov with <code>--ignore-run-fail</code> so partial coverage lands even when test binaries exit non-zero.</li>
<li>Writes a browsable HTML report into a timestamped run directory (<code>coverage-reports/&lt;stamp&gt;/revive/</code> or <code>coverage-reports/&lt;stamp&gt;/llvm/</code>).</li>
</ol>
<h2 id="resource-requirements-and-troubleshooting"><a class="header" href="#resource-requirements-and-troubleshooting">Resource requirements and troubleshooting</a></h2>
<p><code>make coverage</code> bakes in two defaults that keep instrumented runs tractable:</p>
<ul>
<li>It drops DWARF with <code>CARGO_PROFILE_DEV_DEBUG=0</code> since coverage line data lives in the binaries’ <code>__llvm_covmap</code> sections.</li>
<li>It bounds profile output with <code>LLVM_PROFILE_FILE_NAME=revive-%4m.profraw</code>. The default pattern contains <code>%p</code>, writing one raw profile per process. With an instrumented LLVM this could exceed tens of gigabytes of raw profiles in a full run. The <code>%4m</code> merge pool makes processes merge their counters into a handful of files instead.</li>
</ul>
<p>Even with those defaults, plan for:</p>
<ul>
<li>Disk space:
<ul>
<li>Building instrumented LLVM and running an instrumented <code>make coverage</code> still needs tens of gigabytes of disk space.</li>
</ul>
</li>
<li>LLVM link memory (<code>make install-llvm-coverage</code>):
<ul>
<li>If a link step gets OOM-killed, rerun with capped parallelism (e.g. <code>JOBS=1</code>) to serialize the links. The rerun resumes incrementally.</li>
</ul>
</li>
<li>Test-binary link memory (<code>make coverage</code>):
<ul>
<li>If a link gets OOM-killed, rerun with <code>RUSTFLAGS="-C link-arg=-fuse-ld=lld"</code> (or <em>additionally</em> with capped parallelism, e.g. <code>CARGO_BUILD_JOBS=1</code>). GNU ld holds every input object and the coverage mapping of the entire dependency closure in memory at once. lld links with a smaller peak.</li>
</ul>
</li>
<li>Test-run memory (<code>make coverage</code>):
<ul>
<li>Each concurrently running test spawns an instrumented resolc. On memory-constrained machines cap it with <code>RUST_TEST_THREADS=4</code>. This throttles test execution only, not compilation.</li>
</ul>
</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div>
<h1 id="cross-compilation"><a class="header" href="#cross-compilation">Cross compilation</a></h1>
<p>We cross-compile the <code>resolc.js</code> frontend executable to Wasm for running it in a Node.js or browser environment.</p>
<p>The <a href="https://www.musl-libc.org/">musl</a> target is used to obtain statically linked ELF binaries for Linux.</p>