mirror of
https://github.com/pezkuwichain/revive.git
synced 2026-10-20 18:56:17 +00:00
ci: Add GHA workflow executing benchmarks (#422)
## Description
Adds a workflow executing the resolc and yul benchmarks.
* The workflow can be manually triggered via the Actions UI.
* User input:
* Branch, tag, or SHA before changes (e.g. `main`) _(required)_
* Branch, tag, or SHA after changes (e.g. `lj/ci-benchmarks`)
_(required)_
* PR number _(optional)_
* If provided, the benchmarking result will be posted as a PR comment
* The comment is updated if it exists, otherwise it is created
* If omitted, the result is uploaded as an artifact
This PR originally implemented the workflow to automatically run on each
PR once its tests passed successfully. As we likely do not need to run
this workflow for each PR, the workflow has been rewritten to manually
trigger it.
Providing user input also allows for comparisons between any commits,
such as releases.
## Notes
### File must exist on the default branch
* This workflow is triggered by a `workflow_dispatch` event
[only](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#workflow_dispatch)
if the workflow file exists on the default branch. **Therefore, the
workflow cannot be run for this PR.** (See section below on how it has
still been tested.)
* Once merged, for any subsequent PRs that modify this workflow, those
changes will be testable before merge (by choosing to run the workflow
from the PR branch).
## Testing the Workflow
I've tested a similar enough workflow in a personal repo in order to
push it to `main` and trigger it. Will verify again once this PR is
merged.
- [x] Runs when provided with branch names
- [x] Runs when provided with SHAs
- [x] A PR comment is created if a PR number is provided and the comment
does not exist
- [x] A PR comment is updated if a PR number is provided and the comment
already exists
- [x] An artifact is created if a PR number is omitted
- [x] An in-progress run is cancelled if another one is triggered
comparing the same refs
## Example PR Comment
<details>
<summary>Expand to see an example PR comment</summary>
# Benchmarks
* **Ref Before:** `main`
* **Ref After:** `lj/ci-benchmarks`
## Compile E2E
<details>
<summary>Results</summary>
```
group resolc_before resolc_changes
----- ------------- --------------
Dependency/resolc 1.00 144.0±6.94ms ? ?/sec 1.00 144.6±5.97ms ? ?/sec
Dependency/solc 1.04 65.9±10.73ms ? ?/sec 1.00 63.5±10.10ms ? ?/sec
Empty/resolc 1.03 68.8±4.40ms ? ?/sec 1.00 66.6±4.61ms ? ?/sec
Empty/solc 1.13 11.2±2.01ms ? ?/sec 1.00 10.0±2.28ms ? ?/sec
LargeDivRem/resolc 1.00 119.3±5.62ms ? ?/sec 1.02 121.3±17.03ms ? ?/sec
LargeDivRem/solc 1.15 24.7±3.15ms ? ?/sec 1.00 21.5±1.03ms ? ?/sec
Memset (`--yul`)/resolc 1.02 62.7±3.92ms ? ?/sec 1.00 61.7±3.76ms ? ?/sec
Memset (`--yul`)/solc 1.00 9.0±0.84ms ? ?/sec 1.12 10.1±2.23ms ? ?/sec
Multiple Contracts (`--standard-json`)/resolc 1.01 1559.2±68.95ms ? ?/sec 1.00 1539.3±64.92ms ? ?/sec
Multiple Contracts (`--standard-json`)/solc 1.01 652.0±19.38ms ? ?/sec 1.00 648.4±31.83ms ? ?/sec
Return (`--yul`)/resolc 1.00 60.1±17.02ms ? ?/sec 1.02 61.2±14.50ms ? ?/sec
Return (`--yul`)/solc 1.09 8.8±2.37ms ? ?/sec 1.00 8.0±3.08ms ? ?/sec
```
</details>
## Parse Yul
<details>
<summary>Results</summary>
```
group yul_parse_before yul_parse_changes
----- ---------------- -----------------
Baseline/parse 1.07 8.9±3.64µs ? ?/sec 1.00 8.4±0.09µs ? ?/sec
ERC20/parse 1.00 162.5±34.25µs ? ?/sec 1.07 173.3±123.25µs ? ?/sec
SHA1/parse 1.01 80.8±33.04µs ? ?/sec 1.00 80.3±52.82µs ? ?/sec
Storage/parse 1.00 16.9±8.15µs ? ?/sec 1.01 17.2±7.54µs ? ?/sec
Transfer/parse 1.00 18.5±0.23µs ? ?/sec 1.23 22.7±18.50µs ? ?/sec
```
</details>
## Lower Yul
<details>
<summary>Results</summary>
```
group yul_lower_before yul_lower_changes
----- ---------------- -----------------
Baseline/lower 1.00 122.1±5.95µs ? ?/sec 1.09 133.6±35.12µs ? ?/sec
ERC20/lower 1.00 641.7±13.24µs ? ?/sec 1.15 735.6±665.92µs ? ?/sec
SHA1/lower 1.00 361.4±78.47µs ? ?/sec 1.09 392.3±114.48µs ? ?/sec
Storage/lower 1.00 167.2±50.34µs ? ?/sec 1.01 168.2±32.45µs ? ?/sec
Transfer/lower 1.00 167.5±8.70µs ? ?/sec 1.03 173.1±16.32µs ? ?/sec
```
</details>
</details>
## Resolved Issues
Closes #405
This commit is contained in:
1 file changed
+158
@@ -0,0 +1,158 @@
|
||||
name: Benchmark
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
ref_before:
|
||||
description: "Branch, tag, or SHA before changes"
|
||||
default: "main"
|
||||
type: string
|
||||
required: true
|
||||
ref_after:
|
||||
description: "Branch, tag, or SHA after changes"
|
||||
type: string
|
||||
required: true
|
||||
pr_number:
|
||||
description: "PR number (for posting the result as a comment)"
|
||||
type: number
|
||||
required: false
|
||||
|
||||
# Cancels in-progress runs for runs comparing the same refs if the workflow is re-triggered.
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ inputs.ref_before }}-${{ inputs.ref_after }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
benchmark:
|
||||
runs-on: parity-large
|
||||
permissions:
|
||||
pull-requests: write
|
||||
steps:
|
||||
- name: Set Up Rust Toolchain
|
||||
uses: actions-rust-lang/setup-rust-toolchain@v1
|
||||
with:
|
||||
# Without this it will override our rust flags.
|
||||
rustflags: ""
|
||||
|
||||
- name: Install Benchmarking Tools
|
||||
run: |
|
||||
cargo install critcmp
|
||||
|
||||
- name: Checkout Commit Before Changes
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
ref: ${{ inputs.ref_before }}
|
||||
|
||||
- name: Install Solc
|
||||
uses: ./.github/actions/get-solc
|
||||
|
||||
- name: Download LLVM
|
||||
uses: ./.github/actions/get-llvm
|
||||
with:
|
||||
target: x86_64-unknown-linux-gnu
|
||||
|
||||
- name: Set LLVM Environment Variables
|
||||
run: |
|
||||
echo "LLVM_SYS_181_PREFIX=$(pwd)/llvm-x86_64-unknown-linux-gnu" >> $GITHUB_ENV
|
||||
|
||||
- name: Build resolc
|
||||
run: |
|
||||
make install
|
||||
|
||||
- name: Run Benchmarks on Commit Before Changes
|
||||
run: |
|
||||
cargo bench --package resolc --bench compile -- --save-baseline resolc_before
|
||||
cargo bench --package revive-benchmarks --bench parse -- --save-baseline yul_parse_before
|
||||
cargo bench --package revive-benchmarks --bench lower -- --save-baseline yul_lower_before
|
||||
timeout-minutes: 20
|
||||
|
||||
- name: Checkout Commit After Changes
|
||||
run: |
|
||||
git fetch origin ${{ inputs.ref_after }}
|
||||
git checkout -f ${{ inputs.ref_after }}
|
||||
|
||||
- name: Install Solc
|
||||
uses: ./.github/actions/get-solc
|
||||
|
||||
- name: Download LLVM
|
||||
uses: ./.github/actions/get-llvm
|
||||
with:
|
||||
target: x86_64-unknown-linux-gnu
|
||||
|
||||
- name: Build resolc
|
||||
run: |
|
||||
make install
|
||||
|
||||
- name: Run Benchmarks on Commit After Changes
|
||||
run: |
|
||||
cargo bench --package resolc --bench compile -- --save-baseline resolc_changes
|
||||
cargo bench --package revive-benchmarks --bench parse -- --save-baseline yul_parse_changes
|
||||
cargo bench --package revive-benchmarks --bench lower -- --save-baseline yul_lower_changes
|
||||
timeout-minutes: 20
|
||||
|
||||
- name: Compare Benchmarks
|
||||
run: |
|
||||
critcmp resolc_before resolc_changes > benchmarks_resolc
|
||||
critcmp yul_parse_before yul_parse_changes > benchmarks_yul_parse
|
||||
critcmp yul_lower_before yul_lower_changes > benchmarks_yul_lower
|
||||
|
||||
- name: Create Report
|
||||
env:
|
||||
REF_BEFORE: ${{ inputs.ref_before }}
|
||||
REF_AFTER: ${{ inputs.ref_after }}
|
||||
run: |
|
||||
cat > BENCHMARK_REPORT.md << EOF
|
||||
# Benchmarks
|
||||
|
||||
* **Ref Before:** \`$REF_BEFORE\`
|
||||
* **Ref After:** \`$REF_AFTER\`
|
||||
|
||||
## Compile E2E
|
||||
|
||||
<details>
|
||||
<summary>Results</summary>
|
||||
|
||||
\`\`\`
|
||||
$(cat benchmarks_resolc)
|
||||
\`\`\`
|
||||
|
||||
</details>
|
||||
|
||||
## Parse Yul
|
||||
|
||||
<details>
|
||||
<summary>Results</summary>
|
||||
|
||||
\`\`\`
|
||||
$(cat benchmarks_yul_parse)
|
||||
\`\`\`
|
||||
|
||||
</details>
|
||||
|
||||
## Lower Yul
|
||||
|
||||
<details>
|
||||
<summary>Results</summary>
|
||||
|
||||
\`\`\`
|
||||
$(cat benchmarks_yul_lower)
|
||||
\`\`\`
|
||||
|
||||
</details>
|
||||
EOF
|
||||
|
||||
- name: Post PR Comment with Benchmark Report
|
||||
if: ${{ inputs.pr_number }}
|
||||
uses: marocchino/sticky-pull-request-comment@v2
|
||||
with:
|
||||
header: benchmark
|
||||
number: ${{ inputs.pr_number }}
|
||||
path: BENCHMARK_REPORT.md
|
||||
|
||||
- name: Upload Benchmark Results
|
||||
if: ${{ !inputs.pr_number }}
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: benchmark-report
|
||||
path: BENCHMARK_REPORT.md
|
||||
retention-days: 7
|
||||
Reference in new issue
Block a user