Implement verification of reproducible builds and run differential E2E tests (#468)

## Description

This adds verification for reproducible builds (via
[revive-differential-tests](https://github.com/paritytech/revive-differential-tests)
testing framework) by:

1. Compiling
[contracts](https://github.com/paritytech/resolc-compiler-tests/tree/main/fixtures/solidity)
and generating hashes from each bytecode
    * With the resolc builds:
      * linux musl
      * macos universal
      * windows
      * (future todo: wasm32 emscripten)
    * With optimization levels `0`, `3`, `z` for each contract
2. Exporting the hashes
3. Comparing the hashes
    * For each optimization level:
* Checks if each hash for a specific contract is identical across the
platforms

This PR also runs the differential E2E tests (via `retester`) with the
resolc linux musl build.

All of the above only runs if our `test` workflow succeeds.

## Related

This PR initially implemented the above in this repo but has now been
implemented into the separate testing framework instead. The most
relevant PRs implemented alongside this PR as prerequisites are:

* `revive-differential-tests`:
  * [Generate bytecode hashes included in the report
](https://github.com/paritytech/revive-differential-tests/pull/238)
* [Fix handling of resolc's optimizer `mode` and allow setting solc
optimizer
enabled](https://github.com/paritytech/revive-differential-tests/pull/243)
* [Fix solc and resolc optimization
settings](https://github.com/paritytech/revive-differential-tests/pull/253)
* [Implement a `compile` context/subcommand for standalone compilation
mode](https://github.com/paritytech/revive-differential-tests/pull/239)
* [Implement support for `--mode` CLI argument for
`compile`](https://github.com/paritytech/revive-differential-tests/pull/246)
* [Implement `export-hashes` and `compare-hashes` subcommands for report
processor
](https://github.com/paritytech/revive-differential-tests/pull/250)
* [Implement GH actions for compiling, exporting and comparing
hashes](https://github.com/paritytech/revive-differential-tests/pull/251)
* [Gate zombienet-sdk behind `cfg(unix)` to fix Windows
builds](https://github.com/paritytech/revive-differential-tests/pull/254)
* [Link library-using contracts with dummy addresses for `--compile`
command](https://github.com/paritytech/revive-differential-tests/pull/267)
* `revive`:
* [Build LLVM for Windows using MSVC-compatible
Clang](https://github.com/paritytech/revive/pull/497)
* [Apply LLVM patches for deterministic operand order and correct
peephole optimization for `minsize`
](https://github.com/paritytech/revive/pull/500)
* Investigation of comparison mismatches in third-party dependencies:
  * `llvm-project`:
* [`SelectionDAG` type legalization produces different operand ordering
for `i256` multiply across
platforms](https://github.com/llvm/llvm-project/issues/190148)
  * `solidity`:
* [Windows absolute paths in standard JSON break import resolution and
library linking](https://github.com/argotorg/solidity/issues/16579)

## Notes

### Failed compilations

Since
[resolc-compiler-tests](https://github.com/paritytech/resolc-compiler-tests)
currently includes contracts that are expected to fail resolc
compilation (e.g. due to unsupported instructions, contracts that are
expected to work with solc for evm/revm), the compilation workflow job
is not expected to fail if compilation fails for a contract.

Instead, if a contract compiles with one build but not with another, a
hash will be missing and will thereby generate a mismatch when comparing
all hashes, failing at the verification step.

## Resolved Issues

Closes [#175](https://github.com/paritytech/revive/issues/175)
This commit is contained in:
LJ authored and GitHub committed 2026-04-14 16:58:58 +02:00
1 parent f887e75545
commit 733e01c78e
7 files changed
+1251 -62

No files matched your search

+176
View File
@@ -0,0 +1,176 @@
name: Verify Reproducible Builds
on:
workflow_call:
env:
# The compiler modes to use for compilation by retester.
MODES: "Y M0 S+, Y M3 S+, Y Mz S+"
SOLC_VERSION: "0.8.34"
jobs:
build:
uses: ./.github/workflows/reusable-build.yml
with:
is_release: false
retention_days: 1
compile-and-export-hashes-native:
needs: build
strategy:
fail-fast: false
matrix:
include:
- platform: linux-x86_64
target: x86_64-unknown-linux-musl
runner: ubuntu-24.04
- platform: macos-arm64
target: universal-apple-darwin
runner: macos-15
- platform: macos-x86_64
target: universal-apple-darwin
runner: macos-15-intel
- platform: windows-x86_64
target: x86_64-pc-windows-msvc
runner: windows-2022
runs-on: ${{ matrix.runner }}
steps:
# For some deeply nested contracts in `resolc-compiler-tests`, Windows
# will generate a "Filename too long" error. This step enables long paths.
- name: Enable Git Long Paths (Windows)
if: ${{ runner.os == 'Windows' }}
run: git config --system core.longpaths true
- name: Checkout revive
uses: actions/checkout@v4
with:
path: revive
submodules: recursive
- name: Download Binary
uses: actions/download-artifact@v4
with:
name: resolc-${{ matrix.target }}
path: resolc-bin
- name: Make resolc Executable
shell: bash
run: chmod +x "resolc-bin/resolc-${{ matrix.target }}${{ runner.os == 'Windows' && '.exe' || '' }}"
- name: Compile Contracts
id: compile
uses: ./revive/revive-differential-tests/.github/actions/compile-contracts
with:
revive-differential-tests-path: revive/revive-differential-tests
resolc-path: resolc-bin/resolc-${{ matrix.target }}${{ runner.os == 'Windows' && '.exe' || '' }}
solc-version: ${{ env.SOLC_VERSION }}
modes: ${{ env.MODES }}
upload-artifact-name: compilation-report-${{ matrix.platform }}
- name: Export Bytecode Hashes
uses: ./revive/revive-differential-tests/.github/actions/export-bytecode-hashes
with:
revive-differential-tests-path: revive/revive-differential-tests
report-path: ${{ steps.compile.outputs.report-path }}
remove-prefix: ${{ steps.compile.outputs.contracts-source-prefix }}
platform-label: ${{ matrix.platform }}
upload-artifact-name: hashes-${{ matrix.platform }}
compare-hashes:
needs: compile-and-export-hashes-native
runs-on: ubuntu-24.04
steps:
- name: Checkout revive
uses: actions/checkout@v4
with:
path: revive
submodules: true
- name: Compare Bytecode Hashes
uses: ./revive/revive-differential-tests/.github/actions/compare-bytecode-hashes
with:
revive-differential-tests-path: revive/revive-differential-tests
hash-artifact-names: "hashes-linux-x86_64, hashes-macos-arm64, hashes-macos-x86_64, hashes-windows-x86_64"
modes: ${{ env.MODES }}
upload-artifact-name: hash-comparison-result
run-e2e-tests:
needs: build
runs-on: parity-large
timeout-minutes: 75
env:
# Commit SHA that produced the polkadot-sdk crate pinned in Cargo.toml,
# obtained from the crate's .cargo_vcs_info.json via:
# curl -sL https://crates.io/api/v1/crates/polkadot-sdk/<VERSION>/download | tar xzO '*/.cargo_vcs_info.json'
# Current version: 2603.0.0
POLKADOT_SDK_REF: "c56b086a3cfcb3a48b75fd1ef9c77ad9b7408b95"
RUSTFLAGS: "-Awarnings"
steps:
- name: Checkout revive
uses: actions/checkout@v4
with:
path: revive
submodules: recursive
- name: Download Binary
uses: actions/download-artifact@v4
with:
name: resolc-x86_64-unknown-linux-musl
path: ./resolc-bin
- name: Make resolc Executable
shell: bash
run: chmod +x ./resolc-bin/resolc-x86_64-unknown-linux-musl
- name: Checkout Polkadot SDK
uses: actions/checkout@v4
with:
repository: paritytech/polkadot-sdk
ref: ${{ env.POLKADOT_SDK_REF }}
path: polkadot-sdk
submodules: recursive
- name: Install Dependencies
run: |
sudo apt-get update
sudo apt-get install -y build-essential pkg-config libssl-dev protobuf-compiler llvm clang libclang-dev
- name: Download forklift
run: |
curl -fsSLo forklift https://github.com/paritytech/forklift/releases/download/0.15.0/forklift_0.15.0_linux_amd64
chmod +x forklift
sudo mv forklift /usr/local/bin/
- name: Configure forklift
run: |
mkdir -p .forklift
cp polkadot-sdk/.forklift/config.toml .forklift/config.toml
- name: Set Up Rust Toolchain
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: "1.90.0"
target: "wasm32-unknown-unknown"
components: "rust-src"
rustflags: ""
- name: Run Differential E2E Tests
uses: ./revive/revive-differential-tests/.github/actions/run-differential-tests
with:
revive-differential-tests-path: revive/revive-differential-tests
polkadot-sdk-path: ./polkadot-sdk
cargo-command: "forklift cargo"
platform: revive-dev-node-polkavm-resolc
resolc-path: ./resolc-bin/resolc-x86_64-unknown-linux-musl
solc-version: ${{ env.SOLC_VERSION }}
expectations-file-path: ./revive/.github/assets/revive-dev-node-polkavm-resolc.json
# Sentinel job that depends on the jobs we want to allow completing (instead of
# being canceled once the other job chains complete). Thus, run-e2e-tests
# will always run even if compare-hashes finishes first, and vice versa.
wait-for-jobs:
if: ${{ always() }}
needs: [compare-hashes, run-e2e-tests]
runs-on: ubuntu-24.04
steps:
- run: echo "All jobs finished"
+1 -38
View File
@@ -53,46 +53,11 @@ jobs:
is_release: false
retention_days: 40
create-macos-fat-binary:
if: ${{ needs.check_commits.outputs.has_commits == 'true' }}
needs: [build]
outputs:
resolc-universal-apple-darwin_url: ${{ steps.set-output.outputs.resolc-universal-apple-darwin_url }}
resolc-universal-apple-darwin_sha: ${{ steps.set-output.outputs.resolc-universal-apple-darwin_sha }}
runs-on: macos-15
steps:
- uses: actions/download-artifact@v4
with:
merge-multiple: true
- name: Create macOS Fat Binary
run: |
lipo resolc-aarch64-apple-darwin resolc-x86_64-apple-darwin -create -output resolc-universal-apple-darwin
- name: Make Executable
run: |
chmod +x resolc-universal-apple-darwin
- uses: actions/upload-artifact@v4
id: artifact-upload-step
with:
name: resolc-universal-apple-darwin
path: resolc-universal-apple-darwin
retention-days: 40
- name: Set output variables
id: set-output
env:
TARGET: resolc-universal-apple-darwin
run: |
echo "${TARGET}_url=${{ steps.artifact-upload-step.outputs.artifact-url }}" >> "$GITHUB_OUTPUT"
echo "${TARGET}_sha=${{ steps.artifact-upload-step.outputs.artifact-digest }}" >> "$GITHUB_OUTPUT"
generate-nightly-json:
runs-on: ubuntu-24.04
if: ${{ needs.check_commits.outputs.has_commits == 'true' }}
environment: tags
needs: [build, create-macos-fat-binary, check_commits]
needs: [build, check_commits]
steps:
- name: Checkout revive
uses: actions/checkout@v4
@@ -129,8 +94,6 @@ jobs:
run: |
echo '[' > data.json
echo '${{ toJSON(needs.build.outputs) }}' >> data.json
echo ',' >> data.json
echo '${{ toJSON(needs.create-macos-fat-binary.outputs) }}' >> data.json
echo ']' >> data.json
chmod +x bins/resolc-x86_64-unknown-linux-musl
export FIRST_SOLC_VERSION=$(./bins/resolc-x86_64-unknown-linux-musl --supported-solc-versions | cut -f 1 -d "," | tr -d ">=")
+8 -12
View File
@@ -69,7 +69,7 @@ jobs:
create-release:
if: startsWith(github.ref_name, 'v')
needs: [check-version-changed, build]
runs-on: macos-15
runs-on: ubuntu-24.04
environment: tags
steps:
- name: Download Artifacts
@@ -77,10 +77,6 @@ jobs:
with:
merge-multiple: true
- name: Create macOS Fat Binary
run: |
lipo resolc-aarch64-apple-darwin resolc-x86_64-apple-darwin -create -output resolc-universal-apple-darwin
- name: Make Executable
run: |
chmod +x resolc-x86_64-unknown-linux-musl
@@ -88,12 +84,12 @@ jobs:
- name: Create sha-256 checksum
run: |
shasum -a 256 resolc-x86_64-unknown-linux-musl > checksums.txt
shasum -a 256 resolc-universal-apple-darwin >> checksums.txt
shasum -a 256 resolc-x86_64-pc-windows-msvc.exe >> checksums.txt
shasum -a 256 resolc.js >> checksums.txt
shasum -a 256 resolc.wasm >> checksums.txt
shasum -a 256 resolc_web.js >> checksums.txt
sha256sum resolc-x86_64-unknown-linux-musl > checksums.txt
sha256sum resolc-universal-apple-darwin >> checksums.txt
sha256sum resolc-x86_64-pc-windows-msvc.exe >> checksums.txt
sha256sum resolc.js >> checksums.txt
sha256sum resolc.wasm >> checksums.txt
sha256sum resolc_web.js >> checksums.txt
- uses: actions/create-github-app-token@v1
id: app-token
@@ -126,7 +122,7 @@ jobs:
npm-release:
needs: [create-release]
runs-on: macos-15
runs-on: ubuntu-24.04
environment: tags
steps:
- uses: actions/checkout@v4
+121 -11
View File
@@ -25,6 +25,10 @@ on:
value: ${{ jobs.build.outputs.resolc-x86_64-apple-darwin_url }}
resolc-x86_64-apple-darwin_sha:
value: ${{ jobs.build.outputs.resolc-x86_64-apple-darwin_sha }}
resolc-universal-apple-darwin_url:
value: ${{ jobs.create-macos-universal.outputs.resolc-universal-apple-darwin_url }}
resolc-universal-apple-darwin_sha:
value: ${{ jobs.create-macos-universal.outputs.resolc-universal-apple-darwin_sha }}
resolc-x86_64-pc-windows-msvc_url:
value: ${{ jobs.build.outputs.resolc-x86_64-pc-windows-msvc_url }}
resolc-x86_64-pc-windows-msvc_sha:
@@ -37,6 +41,9 @@ on:
env:
CARGO_TERM_COLOR: always
RUST_MUSL_CROSS_IMAGE: messense/rust-musl-cross@sha256:2a8837c43bf12e246f1ebd05191de9ee27fcd22f9ca81511ccd4cf75dc16d71c
# An empty string is interpreted as the latest published release (default behavior).
# To test a specific draft release, use the full name (e.g. "llvm-21.1.8-revive.5789e11").
LLVM_VERSION: ""
jobs:
build:
@@ -51,6 +58,23 @@ jobs:
resolc-x86_64-apple-darwin_sha: ${{ steps.set-output.outputs.resolc-x86_64-apple-darwin_sha }}
resolc-x86_64-pc-windows-msvc_url: ${{ steps.set-output.outputs.resolc-x86_64-pc-windows-msvc_url }}
resolc-x86_64-pc-windows-msvc_sha: ${{ steps.set-output.outputs.resolc-x86_64-pc-windows-msvc_sha }}
# Queue jobs for the same target and commit instead of running in parallel or cancelling
# the first job. This allows the second workflow to use cached builds from the first if
# it is triggered by the same commit.
#
# Example: Workflow A and Workflow B both call this workflow for the same commit:
#
# Time | Workflow A | Workflow B
# -----|---------------------------|---------------------------
# T1 | Job starts, cache miss |
# T2 | Building... | Job queued (same group)
# T3 | Build done, cache saved | Job queued
# T4 | Job finishes | Job starts, cache hit
# T5 | | Skips build, job finishes
concurrency:
# Use `reusable-build` and not `${{ github.workflow }}` here so that callers share the same concurrency group.
group: reusable-build-${{ matrix.target }}-${{ github.sha }}
cancel-in-progress: false
strategy:
matrix:
target:
@@ -81,18 +105,31 @@ jobs:
if: runner.os == 'Windows'
run: git config --system core.longpaths true
- uses: actions/checkout@v4
# Cache build artifacts by target and commit (restores if available, saves on cache miss).
- name: Check Build Cache
id: cache
uses: actions/cache@v5
with:
# Use glob to match binaries with and without extensions.
path: resolc-${{ matrix.target }}*
key: build-${{ matrix.target }}-${{ github.sha }}
- uses: actions-rust-lang/setup-rust-toolchain@v1
if: ${{ steps.cache.outputs.cache-hit != 'true' }}
with:
rustflags: ""
cache-key: ${{ matrix.target }}
- name: Download LLVM
if: ${{ steps.cache.outputs.cache-hit != 'true' }}
uses: ./.github/actions/get-llvm
with:
target: ${{ matrix.target }}
version: ${{ env.LLVM_VERSION }}
- name: Build (Native)
if: ${{ matrix.type == 'native' }}
if: ${{ steps.cache.outputs.cache-hit != 'true' && matrix.type == 'native' }}
shell: bash
run: |
export LLVM_SYS_211_PREFIX=$PWD/llvm-${{ matrix.target }}
@@ -100,7 +137,7 @@ jobs:
mv target/release/resolc resolc-${{ matrix.target }} || mv target/release/resolc.exe resolc-${{ matrix.target }}.exe
- name: Build (MUSL)
if: ${{ matrix.type == 'musl' }}
if: ${{ steps.cache.outputs.cache-hit != 'true' && matrix.type == 'musl' }}
run: |
docker run -v $PWD:/opt/revive $RUST_MUSL_CROSS_IMAGE /bin/bash -c "
cd /opt/revive
@@ -137,37 +174,106 @@ jobs:
echo "resolc-${{ matrix.target }}_url=${{ steps.artifact-upload-step.outputs.artifact-url }}" >> "$GITHUB_OUTPUT"
echo "resolc-${{ matrix.target }}_sha=${{ steps.artifact-upload-step.outputs.artifact-digest }}" >> "$GITHUB_OUTPUT"
create-macos-universal:
needs: build
runs-on: macos-15
concurrency:
# Use `reusable-build` and not `${{ github.workflow }}` here so that callers share the same concurrency group.
group: reusable-build-universal-apple-darwin-${{ github.sha }}
cancel-in-progress: false
outputs:
resolc-universal-apple-darwin_url: ${{ steps.set-output.outputs.resolc-universal-apple-darwin_url }}
resolc-universal-apple-darwin_sha: ${{ steps.set-output.outputs.resolc-universal-apple-darwin_sha }}
steps:
- name: Check Build Cache
id: cache
uses: actions/cache@v5
with:
path: resolc-universal-apple-darwin
key: build-universal-apple-darwin-${{ github.sha }}
- name: Download macOS Artifacts
if: ${{ steps.cache.outputs.cache-hit != 'true' }}
uses: actions/download-artifact@v4
with:
pattern: resolc-*-apple-darwin
merge-multiple: true
- name: Create macOS Fat Binary
if: ${{ steps.cache.outputs.cache-hit != 'true' }}
run: |
lipo resolc-aarch64-apple-darwin resolc-x86_64-apple-darwin -create -output resolc-universal-apple-darwin
chmod +x resolc-universal-apple-darwin
- name: Upload Universal Binary
uses: actions/upload-artifact@v4
id: artifact-upload-step
with:
name: resolc-universal-apple-darwin
path: resolc-universal-apple-darwin
retention-days: ${{ inputs.retention_days }}
- name: Set output variables
if: ${{ !inputs.is_release }}
id: set-output
run: |
echo "resolc-universal-apple-darwin_url=${{ steps.artifact-upload-step.outputs.artifact-url }}" >> "$GITHUB_OUTPUT"
echo "resolc-universal-apple-darwin_sha=${{ steps.artifact-upload-step.outputs.artifact-digest }}" >> "$GITHUB_OUTPUT"
build-wasm:
runs-on: ubuntu-24.04
outputs:
resolc_web_js_url: ${{ steps.set-output.outputs.resolc_web_js_url }}
resolc_web_js_sha: ${{ steps.set-output.outputs.resolc_web_js_sha }}
# Queue jobs for the same target and commit instead of running in parallel or cancelling
# the first job. This allows the second workflow to use cached builds from the first if
# it is triggered by the same commit.
concurrency:
# Use `reusable-build` and not `${{ github.workflow }}` here so that callers share the same concurrency group.
group: reusable-build-wasm32-unknown-emscripten-${{ github.sha }}
cancel-in-progress: false
env:
RELEASE_RESOLC_WASM_URI: https://github.com/paritytech/revive/releases/download/${{ github.ref_name }}/resolc.wasm
steps:
- uses: actions/checkout@v4
# Cache build artifacts by commit (restores if available, saves on cache miss).
- name: Check Build Cache
id: cache
uses: actions/cache@v5
with:
path: resolc-wasm32-unknown-emscripten
key: build-wasm32-unknown-emscripten-${{ github.sha }}
# Pin to 1.92.0 until LLVM WASM libraries are rebuilt with -fwasm-exceptions
# See: https://github.com/paritytech/revive/issues/XXX
- uses: actions-rust-lang/setup-rust-toolchain@v1
if: ${{ steps.cache.outputs.cache-hit != 'true' }}
with:
toolchain: "1.92.0"
target: wasm32-unknown-emscripten
rustflags: ""
- name: Download Host LLVM
if: ${{ steps.cache.outputs.cache-hit != 'true' }}
uses: ./.github/actions/get-llvm
with:
target: x86_64-unknown-linux-gnu
version: ${{ env.LLVM_VERSION }}
- name: Download Wasm LLVM
if: ${{ steps.cache.outputs.cache-hit != 'true' }}
uses: ./.github/actions/get-llvm
with:
target: wasm32-unknown-emscripten
version: ${{ env.LLVM_VERSION }}
- name: Download EMSDK
if: ${{ steps.cache.outputs.cache-hit != 'true' }}
uses: ./.github/actions/get-emsdk
- name: Build
if: ${{ steps.cache.outputs.cache-hit != 'true' }}
run: |
export LLVM_SYS_211_PREFIX=$PWD/llvm-x86_64-unknown-linux-gnu
export REVIVE_LLVM_TARGET_PREFIX=$PWD/llvm-wasm32-unknown-emscripten
@@ -175,6 +281,14 @@ jobs:
make install-wasm
chmod -x ./target/wasm32-unknown-emscripten/release/resolc.wasm
- name: Compress Artifact
if: ${{ steps.cache.outputs.cache-hit != 'true' }}
run: |
mkdir -p resolc-wasm32-unknown-emscripten
mv ./target/wasm32-unknown-emscripten/release/resolc.js ./resolc-wasm32-unknown-emscripten/
mv ./target/wasm32-unknown-emscripten/release/resolc.wasm ./resolc-wasm32-unknown-emscripten/
mv ./target/wasm32-unknown-emscripten/release/resolc_web.js ./resolc-wasm32-unknown-emscripten/
- name: Set Up Node.js
uses: actions/setup-node@v3
with:
@@ -185,8 +299,11 @@ jobs:
mkdir -p solc
curl -sSLo solc/soljson.js https://github.com/argotorg/solidity/releases/download/v0.8.34/soljson.js
node -e "
const soljson = require('solc/soljson');
const createRevive = require('./target/wasm32-unknown-emscripten/release/resolc.js');
// Make sure to require './solc/soljson' and not 'solc/soljson' in order to use
// the explicitly downloaded solc, rather than resolving an npm package potentially
// installed in prior workflow steps.
const soljson = require('./solc/soljson');
const createRevive = require('./resolc-wasm32-unknown-emscripten/resolc.js');
const compiler = createRevive();
compiler.soljson = soljson;
@@ -224,13 +341,6 @@ jobs:
if(!bytecode.startsWith('50564d')) { process.exit(1); }
"
- name: Compress Artifact
run: |
mkdir -p resolc-wasm32-unknown-emscripten
mv ./target/wasm32-unknown-emscripten/release/resolc.js ./resolc-wasm32-unknown-emscripten/
mv ./target/wasm32-unknown-emscripten/release/resolc.wasm ./resolc-wasm32-unknown-emscripten/
mv ./target/wasm32-unknown-emscripten/release/resolc_web.js ./resolc-wasm32-unknown-emscripten/
- name: Upload artifacts (Release)
if: ${{ inputs.is_release }}
uses: actions/upload-artifact@v4
+5
View File
@@ -60,3 +60,8 @@ jobs:
- name: Test docs
run: make doc
run-differential-tests:
# Only run this if `test` successfully completes.
needs: test
uses: ./.github/workflows/differential-tests.yml