## Description
We started seeing rate-limit `429` errors when cloning LLVM (as seen in
e.g. [this failed
run](https://github.com/paritytech/revive/actions/runs/28924583524/job/85809154518)
and [this failed
run](https://github.com/paritytech/revive/actions/runs/28926259175/job/85821336465?pr=564)).
`llvm/llvm-project` now seems to be heavily rate-limited by GitHub.
(Authenticating the clone did not seem to have much effect.)
Even though our resolc build steps only do a treeless clone, all build
steps still did the same clone.
### Changes
* I've instead introduced a `.llvm-version` file that the `get-llvm`
action reads from, eliminating the network request all together.
* `.llvm-version` is meant to be kept in sync with the LLVM submodule
version we use. This is enforced by:
* Introduced a `check-llvm-version.yml` workflow
* Runs once to check that the recorded version in `.llvm-version`
coincides with the version detected from the submodule.
* The submodule version detection *does* perform a network request as
before, but this is only run once now (when relevant files change),
rather than for all our resolc build jobs.
* Since the differential test jobs require checking out our
revive-differential-tests (rdt) submodule, those checkout steps cloned
not only rdt, but also LLVM. I've updated this to not clone LLVM at all.
## Description
The resolc long version stopped being populated on the standard JSON and
combined JSON output. This PR restores these fields and populates them.
## Description
Extends our differential tests such that the compile+compare bytecode
hashes and the E2E tests also exercise the newyork IR pipeline.
The requested compiler modes that E2E test cases run with are still
controlled by the metadata in the corpus, but the related PR listed
below also enables support for an allow-list to optionally narrow the
set of modes run, via an input to the GH action used.
The changes to the workflow in this PR allows all newyork retester
compiler modes (all LLVM opt levels + solc optimizer on/off) to run for
the E2E tests since they still complete fast.
## Related
The PRs implemented as prerequisites alongside this PR are:
* `revive-differential-tests`
* [Support optimized newyork IR compiler
pipeline](https://github.com/paritytech/revive-differential-tests/pull/308)
* `resolc-compiler-tests`
* [Add newyork IR pipeline counterparts for explicitly-listed Yul
modes](https://github.com/paritytech/resolc-compiler-tests/pull/15)
## Resolved issues
Closes#553
## Description
Adds a CI workflow that builds and tests a small Foundry project with
`foundry-polkadot`'s `forge`, exercising the standard JSON interface
between `resolc` and `foundry-polkadot` end-to-end.
## Resolved Issues
Closes#513
## Description
Before the CI refactor, the Wasm builds initiated by `release.yml` and
`test-wasm.yml` ran in parallel, building duplicate builds. After the CI
refactor, those workflows shared the build cache key to instead reuse
the build. In both cases, the resolc Wasm URI baked into `resolc_web.js`
depends on whether it is a release or not.
If the Wasm build step initiated by `release.yml` started *before*
`test-wasm.yml`, `test-wasm.yml` would use an incorrect cache due to the
invalid resolc Wasm URI (seen by [this failed
run](https://github.com/paritytech/revive/actions/runs/27178950108/job/80235255116?pr=529)).
This PR updates the workflow input `is_release` to only pertain to ref
names starting with `v` (which is what the `create-release` step guards
on), and updates the cache key and concurrency group to include
`is_release`, solving the cache collision. (Thus, the workflows needing
a Wasm build for non-v-releases will still reuse the now-correct cache.)
## Description
Improves overall time spent for our CI checks to complete. For cache
hits, total time decreases from ~70min to ~29min.
## Overview
```mermaid
gantt
title CI on a PR — BEFORE ~70m vs NOW ~29m
dateFormat HH:mm
axisFormat %M
section BEFORE ~70m
[release] build-wasm 20m :r1, 00:00, 20m
[release] build-x86_64-apple-darwin 17m :r2, 00:00, 17m
[release] build-x86_64-pc-windows-msvc 11m :r3, 00:00, 11m
[release] build-x86_64-unknown-linux-musl 10m :r4, 00:00, 10m
[release] build-aarch64-apple-darwin 8m :r5, 00:00, 8m
[release] create-macos-universal 14s :r6, 00:17, 1m
[test-wasm] own duplicate wasm build 19m :w1, 00:00, 19m
[test-wasm] test matrix 3m :w2, 00:19, 3m
[test] required 12m :crit, t1, 00:00, 12m
[differential-tests] build matrix waits 8m for release :crit, t2, 00:12, 8m
[differential-tests] macos-x86_64 compile and hash 44m :crit, t3, 00:20, 44m
[differential-tests] windows-x86_64 compile and hash 26m :t4, 00:20, 26m
[differential-tests] linux-x86_64 compile and hash 20m :t5, 00:20, 20m
[differential-tests] macos-arm64 compile and hash 15m :t6, 00:20, 15m
[differential-tests] run-e2e-tests parallel 43m :t7, 00:22, 43m
[differential-tests] compare-hashes 5m :crit, t8, 01:04, 5m
[hardhat] use-hardhat chained 1m :crit, t9, 01:09, 1m
section NOW ~29m
[release] build-windows 3m :nr1, 00:00, 3m
[release] build-x86_64-apple-darwin 7m :nr2, 00:00, 7m
[release] build-linux 2m :nr3, 00:00, 2m
[release] build-aarch64-apple-darwin 2m :nr4, 00:00, 2m
[release] create-macos-universal 23s :nr5, 00:07, 1m
[release] build-wasm cache hit ~11s :nr6, 00:15, 1m
[test-wasm] shared wasm build 15m :nw1, 00:00, 15m
[test-wasm] test matrix 3m :nw2, 00:15, 3m
[test] required 6m :crit, nt1, 00:00, 6m
[test] check-changes 2s parallel :nt2, 00:00, 1m
[differential-tests] per-target cache hits ~1m 30s :crit, nt3, 00:06, 2m
[differential-tests] windows-x86_64 compile and hash 19m :crit, nt4, 00:08, 19m
[differential-tests] linux-x86_64 compile and hash 16m :nt5, 00:08, 16m
[differential-tests] macos-arm64 compile and hash 13m :nt6, 00:08, 13m
[differential-tests] run-e2e-tests 16m :nt7, 00:08, 16m
[differential-tests] compare-hashes 2m :crit, nt8, 00:27, 2m
[hardhat] use-hardhat parallel 1m :nt9, 00:07, 1m
```
## Measurements
| Metric | Before | Now | Improvement |
|---|---|---|---|
| **PR CI wall-clock** | 70m 15s | 28m 47s | **−41m 28s (−59%)** |
| `test` job (required check) | 12m 10s | 6m 17s | −5m 53s (faster LLVM
download) |
| `compile-and-hash` build phase | ~7m 30s (waiting on release builds) |
~30s (builds already available) to ~1m 30s (if waiting for macos) |
−6-7m (per-target build split + faster LLVM download) |
| `run-e2e-tests` build phase | ~7m 30s (waiting on release builds) |
~30s (`linux-x86_64` already available) | −7m (per-target build split) |
| Longest `compile-and-hash` job | 44m 05s (`macos-x86_64`) | 19m 08s
(`windows-x86_64`) | −24m 57s (`macos-x86_64` excluded on PRs) |
| `compile-and-hash` matrix start time | 20m 13s | 7m 54s | −12m 19s
(per-target build split + faster LLVM download) |
| `run-e2e-tests` start time | 21m 50s | 7m 36s | −14m 14s (per-target
build split + faster LLVM download) |
| `run-e2e-tests` duration | 43m 28s | 16m 22s | −27m 06s (concurrency
bump + flakiness fix) |
## Changes
| Change | Before | Now |
|---|---|---|
| **Per-target build sub-workflows** | <ul><li>Getting a reusable build
depended on all builds completing.</li><li>Even if e.g. a macOS build is
cached, the waiting workflow with macOS runners could be queued for a
long time just to get the status and cache of the other workflow, since
macOS runners are more scarce than e.g. Ubuntu.</li></ul> |
<ul><li>Split umbrella `reusable-build.yml` into
`reusable-build-{linux,macos,windows,wasm}.yml`.</li><li>Consumers can
wait on a specific target and start earlier instead of blocking on the
whole matrix.</li><li>E.g. E2E and the compile-and-hash matrix start
earlier (gate only on the targets needed).</li></ul> |
| **Bottleneck macOS Intel only running on `main`** | macOS Intel
runners are 2-3× slower than macOS ARM for the contract compilation
tests. Ran on each PR, being the bottleneck. | This build is now
excluded from the matrix for PR runs, and only included for pushes to
`main`. |
| **E2E tests use higher concurrency** | `run-e2e-tests` duration:
~43min.</li></ul> | <ul><li>Uses higher number of concurrent tasks, made
possible by retester flakiness fix.</li><li>`run-e2e-tests` duration:
~16min.</li></ul> |
| **Faster LLVM downloads** | Building resolc took ~8-13min (~16 min on
macOS intel). | `get-llvm/action.yml` uses more efficient version
detection. Building resolc now takes ~2min (~4-7 min on macOS intel). |
| **Wasm tests use cached artifacts** | `test-wasm.yml` ran its own
duplicate build. | `test-wasm.yml` uses the reusable Wasm build. |
| **Skip unnecessary polkadot-sdk package** | For E2E tests,
`polkadot-omni-node` was unconditionally downloaded (~3-4min). | Only
`pallet-revive-eth-rpc` and `revive-dev-node` are downloaded from the
cache. |
| **Cache `report-processor`** | Only `retester` was cached. |
`report-processor` is cached for the compilations and export hashes
jobs. |
| **Hardhat unblocked from differential tests** | Started after the
differential tests succeeded in order to reuse the Linux build. | Starts
as soon as the required `test` job succeeds, using the reusable Linux
build. |
| **Conditionally skip workflows** | <ul><li>Some workflows
conditionally skipped for md-only changes.</li><li>Tooling-only changes
also ran e.g. `test-wasm.yml` and `differential-tests.yml`.</ul></li> |
<ul><li>Most workflows are skipped for doc-only
updates.</li><li>Tooling-only updates trigger only the respective
tooling test (Hardhat), apart from the required `test`
job.</li><li>Differential tests are skipped for js-only
updates.</li></ul> |
| **Miscellaneous** | <ul><li>Used actions running on soon-deprecated
Node 20.</li><li>Difficult to debug Playwright `test-wasm`
failures.</li></ul> | <ul><li>Upgraded actions that used Node
20.</li><li>Added Playwright debug traces uploaded on `test-wasm`
failures.</li></ul> |
## Resolved Issues
Closes#533
Bumps `@playwright/test` from ^1.59.1 to ^1.60.0 in an attempt to fix
the flaky install-step hang in test-wasm.yml.
Co-authored-by: xermicus <cyrill@parity.io>
We experienced flakiness in the e2e tests after bumping the
revive-differential-tests submodule which included changes to the global
RPC concurrency limiter and batching layer. This PR attempts to fix the
flakiness by lowering the number of concurrent tasks spawned.
## Description
Adds a CI workflow that builds a small Hardhat project with
`hardhat-polkadot`, exercising the standard JSON interface between
`resolc` and `hardhat-polkadot` end-to-end.
## Resolved Issues
Closes#514
## Description
We previously had a workaround to strip `minsize` attributes for
functions with 256-bit integer division due to a crash in the LLVM
backend. We now [patch](https://github.com/paritytech/revive/pull/500)
LLVM with the fix, allowing us to use `minsize` with correct backend
optimization, making the workaround unnecessary.
## Description
Applies LLVM patches that have not been backported to the major version
used in our submodule:
* [LLVM PR #163679](https://github.com/llvm/llvm-project/pull/163679)
* *"[RISCV] Allow large div peephole optimization for minsize"*
* Fixes peephole optimization of large integers when `minsize` function
attribute is set.
* (Our revive codebase used a workaround that can be removed once this
500 PR is merged and we publish a new LLVM release.)
* [LLVM PR #190219](https://github.com/llvm/llvm-project/pull/190219)
* *"[TargetLowering] Speculative fix for a non-determinism issue between
different compilers"*
* Operand ordering of `mul i256` were different depending on the target
ABI when building LLVM, producing different RISC-V and PVM bytecode for
our Windows vs macOS/linux builds.
* The bytecode divergence (yet functionally equivalent) affected
Solidity contracts containing a value [multiplied by
itself](https://github.com/llvm/llvm-project/issues/190148) (e.g. via
explicit squaring or exponentiation), manifesting primarily when using
optimization level `-O0`.
## Description
Switches the Windows LLVM build from Visual Studio to `clang-cl` +
`lld-link` (Clang's MSVC-compatible drivers) with Ninja.
This is in order to better match the Clang toolchain used on our Linux
and macOS variants, while keeping the MSVC ABI.
It also speeds up the build time by at least 2x on Windows.
# Description
Compilation times for resolc were significantly slower than solc when:
* Multiple files were included in the JSON input file via
`--standard-json`
* And none of the files requested code generation
* (If only a subset of files requested code generation, both resolc and
solc were very slow)
Reason:
* Resolc unconditionally generated bytecode for all contracts, even if
not requested.
* When compiling to PolkaVM bytecode, we need to pass additional flags
(e.g. `irOptimized`) to solc. If these flags were not explicitly
requested, resolc also added these to the requested output since PolkaVM
bytecode was always generated.
## Breaking Changes
The following changes are breaking, but fix the semantics to match its
intended behavior:
* Compiling with `--standard-json` now compiles to PolkaVM bytecode
***only*** for the contracts explicitly requested in the
`outputSelection`.
* Compiling with `--standard-json` now outputs ***only*** the explicitly
requested output (e.g. `evm.assembly`, `evm.bytecode`, `ast`, etc.).
```jsonc
// Example:
{
"settings": {
// ...
"outputSelection": {
"src/common/GasService.sol": {
// Contracts in this file will generate bytecode.
// Only these fields of the JSON output selection will be in the `contracts` output.
"*": ["abi", "evm.methodIdentifiers", "metadata", "evm.bytecode"],
// Only this field of the JSON output selection will be in the `sources` output.
"": ["ast"]
},
"src/common/Gateway.sol": {
// No contracts in this file will generate bytecode.
"*": ["abi", "evm.methodIdentifiers", "metadata"],
// No `ast` will be in the `sources` output (only the automatically added `id`,
// similar to solc as this is not a configurable output selection).
"": []
},
}
}
}
```
## Performance Improvements
When e.g. the standard JSON input file includes multiple files and
requests output for every contract as per this setting:
```jsonc
{
"settings": {
// ...
"optimizer": { "enabled": true, "runs": 1 },
"viaIR": true,
"outputSelection": {
"src/common/GasService.sol": {
"*": ["abi", "evm.methodIdentifiers", "metadata", "evm.bytecode"],
},
// Rest of files..
}
}
}
```
The differences in compilation times depending on whether `evm.bytecode`
is selected in every file, one file, or no files can be seen below:
| Solc Optimizer (runs=1) | ViaIR | Request `abi`,
`evm.methodIdentifiers`, `metadata` | Request `evm.bytecode` | resolc
before | resolc after | solc v0.8.33 |
|:-----------------------:|:-----:|:---------:|:---------:|:-------------:|:------------:|:----------:|
| ✅ | ✅ | All files | All files | `~14.5s` | `~14.5s` | `~13.5s` |
| ✅ | ✅ | All files | One file | `~14.5s` | `~450ms` | `~13.5s` |
| ✅ | ✅ | All files | No files | `~14.5s` | `~390ms` | `~350ms` |
**Observations:**
* `resolc before` is roughly the same no matter the request
* This is due to unconditionally generating bytecode
* When not requesting bytecode for any contracts:
* `resolc after` now compiles in ~390ms (vs ~14.5s), similar to `solc`
* Neither `solc` nor `resolc after` generates bytecode when none of the
contracts request it
* When requesting bytecode for all contracts in _only one_ file:
* `resolc after` now compiles in ~450ms (vs ~14.5s)
* `solc` still compiles in the same time as when requesting bytecode for
all contracts in all files
* This is something they have tried to improve (see
[Solidity#15373](https://github.com/argotorg/solidity/issues/15373) and
[Solidity#15615](https://github.com/argotorg/solidity/issues/15615)),
and the intended behavior seems to now be that bytecode should only be
generated when explicitly requested for each contract.
* Even though PRs have merged to try to fix it, their issue seems to
persist as of v0.8.33.
# Open Questions
* Foundry:
* [x] Pruning of the AST:
* Just saw [this past
commit](https://github.com/paritytech/revive/pull/385) to not prune the
AST for Foundry compatibility.
* Can Foundry not explicitly request `"": "ast"`?
* **Answer:**
* Pruning the AST when not requested should be fine, as long as Foundry
explicitly requests it when needed.
* Will be tested by Foundry maintainer.
# Review Notes
Changes in `docs/` (~25 files) can be skipped as they are automatically
updated after adding content to `book/` and rebuilding it.
# Resolved Issues
Closes#326
---------
Co-authored-by: xermicus <cyrill@parity.io>
Differentially tests the behavior of deleting a struct with a member
that do not occupy the full 32-byte slot. From [this
issue](https://github.com/paritytech/contract-issues/issues/118) it was
unclear whether the `delete` was expected to reset only the bytes
occupied or the entire slot. The current behavior of resetting the
entire slot is expected.
Fixes the rendering of the Markdown tables for `resolc-bin` which do not
render inside the `details` tags for GH pages. Removes `details` and
instead adds a table of contents at the top for fast navigation to
respective platform.
## Description
Adds default values to the compiler arguments below when parsed by
`clap`. This does not change previous user-facing behavior as these
defaults were implemented using different logic, but improves the UX and
the visibility of the exact defaults used.
**Example output from `resolc --help`:**
(Only affected arguments are shown here, and some of their descriptions
have been omitted here.)
```
Options:
-O, --optimization <OPTIMIZATION>
Set the optimization parameter -O[0 | 1 | 2 | 3 | s | z]. Use `3` for best performance and `z` for minimal size
[default: z]
--metadata-hash <METADATA_HASH>
Set the metadata hash type. Available types: `none`, `ipfs`, `keccak256`
[default: keccak256]
--heap-size <HEAP_SIZE>
The emulated EVM linear heap memory static buffer size in bytes.
...
[default: 131072]
--stack-size <STACK_SIZE>
The contracts total stack size in bytes.
...
[default: 131072]
```
## Resolved Issues
Closes#446
## Description
Adds a workflow executing the resolc and yul benchmarks.
* The workflow can be manually triggered via the Actions UI.
* User input:
* Branch, tag, or SHA before changes (e.g. `main`) _(required)_
* Branch, tag, or SHA after changes (e.g. `lj/ci-benchmarks`)
_(required)_
* PR number _(optional)_
* If provided, the benchmarking result will be posted as a PR comment
* The comment is updated if it exists, otherwise it is created
* If omitted, the result is uploaded as an artifact
This PR originally implemented the workflow to automatically run on each
PR once its tests passed successfully. As we likely do not need to run
this workflow for each PR, the workflow has been rewritten to manually
trigger it.
Providing user input also allows for comparisons between any commits,
such as releases.
## Notes
### File must exist on the default branch
* This workflow is triggered by a `workflow_dispatch` event
[only](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#workflow_dispatch)
if the workflow file exists on the default branch. **Therefore, the
workflow cannot be run for this PR.** (See section below on how it has
still been tested.)
* Once merged, for any subsequent PRs that modify this workflow, those
changes will be testable before merge (by choosing to run the workflow
from the PR branch).
## Testing the Workflow
I've tested a similar enough workflow in a personal repo in order to
push it to `main` and trigger it. Will verify again once this PR is
merged.
- [x] Runs when provided with branch names
- [x] Runs when provided with SHAs
- [x] A PR comment is created if a PR number is provided and the comment
does not exist
- [x] A PR comment is updated if a PR number is provided and the comment
already exists
- [x] An artifact is created if a PR number is omitted
- [x] An in-progress run is cancelled if another one is triggered
comparing the same refs
## Example PR Comment
<details>
<summary>Expand to see an example PR comment</summary>
# Benchmarks
* **Ref Before:** `main`
* **Ref After:** `lj/ci-benchmarks`
## Compile E2E
<details>
<summary>Results</summary>
```
group resolc_before resolc_changes
----- ------------- --------------
Dependency/resolc 1.00 144.0±6.94ms ? ?/sec 1.00 144.6±5.97ms ? ?/sec
Dependency/solc 1.04 65.9±10.73ms ? ?/sec 1.00 63.5±10.10ms ? ?/sec
Empty/resolc 1.03 68.8±4.40ms ? ?/sec 1.00 66.6±4.61ms ? ?/sec
Empty/solc 1.13 11.2±2.01ms ? ?/sec 1.00 10.0±2.28ms ? ?/sec
LargeDivRem/resolc 1.00 119.3±5.62ms ? ?/sec 1.02 121.3±17.03ms ? ?/sec
LargeDivRem/solc 1.15 24.7±3.15ms ? ?/sec 1.00 21.5±1.03ms ? ?/sec
Memset (`--yul`)/resolc 1.02 62.7±3.92ms ? ?/sec 1.00 61.7±3.76ms ? ?/sec
Memset (`--yul`)/solc 1.00 9.0±0.84ms ? ?/sec 1.12 10.1±2.23ms ? ?/sec
Multiple Contracts (`--standard-json`)/resolc 1.01 1559.2±68.95ms ? ?/sec 1.00 1539.3±64.92ms ? ?/sec
Multiple Contracts (`--standard-json`)/solc 1.01 652.0±19.38ms ? ?/sec 1.00 648.4±31.83ms ? ?/sec
Return (`--yul`)/resolc 1.00 60.1±17.02ms ? ?/sec 1.02 61.2±14.50ms ? ?/sec
Return (`--yul`)/solc 1.09 8.8±2.37ms ? ?/sec 1.00 8.0±3.08ms ? ?/sec
```
</details>
## Parse Yul
<details>
<summary>Results</summary>
```
group yul_parse_before yul_parse_changes
----- ---------------- -----------------
Baseline/parse 1.07 8.9±3.64µs ? ?/sec 1.00 8.4±0.09µs ? ?/sec
ERC20/parse 1.00 162.5±34.25µs ? ?/sec 1.07 173.3±123.25µs ? ?/sec
SHA1/parse 1.01 80.8±33.04µs ? ?/sec 1.00 80.3±52.82µs ? ?/sec
Storage/parse 1.00 16.9±8.15µs ? ?/sec 1.01 17.2±7.54µs ? ?/sec
Transfer/parse 1.00 18.5±0.23µs ? ?/sec 1.23 22.7±18.50µs ? ?/sec
```
</details>
## Lower Yul
<details>
<summary>Results</summary>
```
group yul_lower_before yul_lower_changes
----- ---------------- -----------------
Baseline/lower 1.00 122.1±5.95µs ? ?/sec 1.09 133.6±35.12µs ? ?/sec
ERC20/lower 1.00 641.7±13.24µs ? ?/sec 1.15 735.6±665.92µs ? ?/sec
SHA1/lower 1.00 361.4±78.47µs ? ?/sec 1.09 392.3±114.48µs ? ?/sec
Storage/lower 1.00 167.2±50.34µs ? ?/sec 1.01 168.2±32.45µs ? ?/sec
Transfer/lower 1.00 167.5±8.70µs ? ?/sec 1.03 173.1±16.32µs ? ?/sec
```
</details>
</details>
## Resolved Issues
Closes#405
## Description
For more easily finding the details of the resolc binary releases for
the various supported platforms, the plain JSON information will be
displayed on [resolc-bin](https://github.com/paritytech/resolc-bin)'s
[GitHub Pages website](https://paritytech.github.io/resolc-bin/).
Changes:
* Whenever there is a regular or nightly release:
* The corresponding workflow files will run a script to update
resolc-bin's `index.md` file (new file) with the updated JSON.
* `index.md` will be the file GitHub Pages serves as HTML (by default it
is built with Jekyll and the Markdown is processed by kramdown, hence
the syntax seen in the script).
* Documentation updates
Thus, this will run on the next nightly release after this PR is merged.
## Resolved Issues
Closes#443
---------
Co-authored-by: xermicus <cyrill@parity.io>
Closes#416.
Workflows using x86_64-apple-darwin target with a `macos-13` runner
image have been migrated to `macos-15-intel` due to deprecation (see
#416), and `macos-14` images have been migrated to `macos-15`.
When updating documentation Markdown files, long-running tests still run
unnecessarily. With these changes, it will skip running test workflows
when PRs only containing changes to such files are opened.
Co-authored-by: xermicus <cyrill@parity.io>
LJ