The indexer lockfile still held wasm-crypto 7.5.17, which implements
only sr25519 and hashing; ed25519, secp256k1, derivation and VRF throw.
The bots move from 7.5.20 to 7.5.21, the same outputs on a crate set
without known advisories.
deploy-vps reached subquery1 as root, through appleboy/ssh-action referenced
by tag, with a key whose comment (mamostehp@MamostePC) made it look like a
person's. It now uses plain ssh with SUBQUERY_DEPLOY_KEY, which the host pins
(authorized_keys restrict,command=) to ops/host/subquery-deploy-ssh: the
commit SHA is its only input. subquery-deploy, run through sudo, does what
the workflow's script did, with three changes:
- It deploys exactly the SHA CI was triggered for, and only a commit on main.
- The host's local edits to tracked files (the indexer manifests) are put
back whether the deploy succeeds or fails. Before, a failed build exited
before `stash pop` and left them stashed.
- Only tracked files are stashed. Stashing untracked ones too (.yarn/, which
yarn install recreates) made every pop fail half-way; that is how 13 stash
entries piled up on the host.
test-deploy.sh covers this against a scratch clone (red with the restore
trap removed); CI runs it with shellcheck. subquery1's host key is pinned.
The indexer endpoints themselves are unchanged. Actions pinned to SHAs; the
secret scanner's own AGPL licence is allowed in dependency review.
@pezkuwi/rpc-provider lists @bizinikiwi/connect as an optional dependency,
so npm installed it, with smoldot, into both images. Both are GPL-3.0, and
the bots never load them: they reach the chains over WsProvider. The images
now install with --omit=optional; with that install both bots load and read
the relay, Asset Hub and People chains.
The lockfiles still list the packages, so Dependency Review, which denies
GPL-3.0, flagged them on the new payout-bot lockfile (the noter's had them
before). It allows exactly those three packages, and the bots job fails if
an image contains @bizinikiwi or smoldot, so the allowance cannot outlive
the reason for it.
Both images ran `npm install` with version ranges and ignored any lockfile,
so what production ran depended on the day the image was built. The live
containers, built on 2026-10-03, run @pezkuwi/wasm-crypto 7.5.17, whose
ed25519, ecdsa and mnemonic generation are stubs that throw. Both bots sign
with sr25519 only, which is why they kept working.
Each bot now has a package.json and package-lock.json (@pezkuwi/api 16.5.37,
keyring, util and util-crypto 14.0.26, wasm-crypto 7.5.20) and its image
installs exactly that with `npm ci`. Node 20 is past its end of life; the
images move to node:24-alpine.
CI gains a `bots` job that builds each image and runs inside it:
scripts/bot-smoke.mjs checks the installed versions against the lockfile and
signs and verifies with sr25519, ed25519 and ecdsa, and the bot itself must
load and stop at the mnemonic it is not given. Against the previous lockfile
the smoke check fails on mnemonic generation, ed25519 and ecdsa.
yarn.lock: transitive bumps within declared ranges for @babel/core, ajv,
axios (1.x), bn.js, body-parser, brace-expansion, browserslist, defu,
dottie, fast-uri, flatted, follow-redirects, form-data, h3, handlebars,
ip-address, js-yaml, lodash, minimatch, moment, path-to-regexp, picomatch,
qs, simple-git, tar, ws, yaml and @modelcontextprotocol/sdk where in range.
noter/package-lock.json: bn.js 5.2.2 -> 5.2.5, ws 8.19.0 -> 8.22.0.
scripts/python_scripts and scripts/update_projects_description.sh are
removed rather than patched (27 alerts on requirements.txt). They came in
with the initial commit from the Nova project and were never changed: they
call SubQuery's managed service (api.subquery.network) and write
descriptions for the "nova-wallet" organisation. Nothing in CI, package.json
or the other scripts calls them, and our indexers are self-hosted.
Left for upstream SubQuery releases, all build/codegen tooling rather than
the indexer runtime: axios 0.27 via @subql/network-clients, ws 8.18 and
elliptic via its ethers 5, serialize-javascript 6 via terser-webpack-plugin,
uuid 8 via @subql/x-sequelize, tmp via @inquirer/editor, got 9 via
update-notifier.
Checked: yarn install --immutable; subql codegen and build succeed for
pezkuwi.yaml, pezkuwi-assethub.yaml and pezkuwi-governance.yaml.
The organisation's code security configuration enables CodeQL default
setup, and GitHub refuses to apply it while a workflow runs CodeQL itself:
the configuration failed on this repository with "Code scanning conflict".
Default setup covers what this job did. The job ran the default query
suite with no config file, path filters or packs, and the languages it
analysed are among those default setup detects here. Default setup takes
over the scan; this workflow keeps its other jobs.
security-events: write served only the SARIF upload, so it goes with it.
contents: read is all the remaining jobs need.
This module is ESM, so `require` is not defined at runtime. `node --check`
only parses, so it passed while both helpers would have thrown: the write
was swallowed by its catch, and the read fell through to `Infinity`, which
the watchdog reads as a permanently stale heartbeat — it would have exited
the process a minute after every scan and left the bot restarting in a loop.
`fs` is already imported at the top of the file; use it.
The bot pointed at the block-producing nodes' RPC ports. Those bind to
localhost, so once they stopped being exposed to the internet the bot lost
every chain at once and has submitted nothing since. `staking_score` gates
the entire trust score, so a noter that cannot reach the chains does not
degrade the score — it zeroes it for every tracked account.
Point it at the public endpoints instead, which is what an external client
should use and what the code already defaulted to. They are TLS-terminated
rather than plaintext ws:// across the internet, and this drops a hardcoded
node address from the repository.
Reaching the chains again is not enough on its own: the outage lasted three
weeks because reconnecting forever looks identical to working. The bot now
writes a heartbeat when a scan *completes*, a container healthcheck reads it,
and a watchdog exits once it goes stale so the restart policy turns a silent
stall into a visibly crash-looping container.
hasDataChanged() only compared fresh data against CachedStakingDetails -
which stays empty until a submission matures and gets finalized. For any
account whose real stake was genuinely stable, every 5-minute scan still
saw "changed from empty cache", resubmitted, and reset submitted_at to the
current block. Confirmed live: Serok's and QaziM's real, correctly-computed
stake (490,000 / 100,000 HEZ) never once reached CachedStakingDetails/
TrustScores despite multiple successful submissions - each new scan arrived
before the previous one's 300-block dispute window elapsed and overwrote it
with a fresh timer, a self-perpetuating loop.
getPendingData() now reads PendingStakingDetails first; hasDataChanged()
compares against that when present, only falling back to the cache once
nothing is pending. Applied to both the AssetHub and legacy RelayChain-clear
comparisons. Also reordered main()'s cycle to finalizeMaturedPending()
before fullScan(), so anything that matured this cycle gets promoted before
a new scan pass runs at all.
The staking-score pallet's audit-fix (commit 9afd99ae) added a dispute-window
model: receive_staking_details() only writes to PendingStakingDetails, and a
separate, permissionless finalize_staking_details() call promotes it to
CachedStakingDetails once DisputeWindow blocks have passed. No caller of that
promotion step existed anywhere in this bot (or anywhere else in this repo) -
confirmed live: Serok's and QaziM's real stake was correctly submitted to
Pending after the earlier NotRegisteredNoter/nonce fixes, but sat there
indefinitely with no path to ever reaching CachedStakingDetails/TrustScores.
Adds finalizeMaturedPending(), called once per periodic scan cycle (and once
at startup): scans PendingStakingDetails.entries(), finalizes whichever have
matured, batched into one utility.batchAll like the existing submission path,
routed through the same submitQueue so it can never race with a concurrent
submitStakingDetails call.
fullScan() processes accounts in concurrent batches of 10, and each account's
own data-gathering takes variable time, so multiple accounts' submitStakingDetails
calls landed close enough together to race on nonce assignment - the pool then
rejected most of them as duplicate-priority ("Priority is too low ... already in
the pool"). Observed live: 37 of 50 tracked accounts silently dropped in a single
scan cycle, including real accounts with genuine stake (Serok, QaziM) whose trust
scores stayed stuck at 0 as a result even after the separate NotRegisteredNoter
bond issue was fixed.
Routes every submission (both fullScan's batches and the live event listener)
through a single queue so exactly one signAndSend is ever in flight at a time,
in call order - data-gathering stays concurrent (unaffected), only the actual
chain submission is serialized.
Both governance and AH indexers were crashing at pruned blocks (RPC
nodes only retain last ~256 blocks of state). Bumping startBlock to
near-tip allows clean indexing forward from current state.
- governance: 57100 → 1228500 (relay tip - ~200)
- AH: 582592 → 1029200 (AH tip - ~200)
Loss: governance 57100-1228500 (mostly empty, pre-governance-activity);
AH 582592-1029200 (~410k blocks of staking/transfer data — postgres
state will be dropped, accept loss).
Long-term fix: provision Asset Hub archive node so reindexing from
genesis is possible without recurring pruning issues.
Both AH (56400→579775) and Relay (57100→708687) startBlocks were
pointing to blocks whose state has been pruned by the chain nodes.
This caused a crash loop: SubQuery couldn't index those blocks because
api.query calls returned 'State already discarded' errors.
Root cause: AH node has ~200 block state window (default pruning).
Fix: set startBlock to current tip - 50 blocks (safely within window).
Also added --state-pruning=archive to AH collator azad on VPS3 so
this doesn't recur as SubQuery re-syncs.
When SubQuery restarts and hits pruned blocks, events are empty (from patch 4).
This caused two downstream crashes:
- getExtrinsicSuccess(undefined) → TypeError on findIndex
- wrapBlock with events.toArray() → TypeError on undefined
Patches 5-6 handle undefined events gracefully.
- handleBlock now checks activeEra every block as fallback when
StakersElected event is dropped due to pruned state patch
- Clear all existing ActiveStaker entries before saving fresh data
from current era exposure set (prevents stale nominators)
- Re-save pool stash accounts after cleanup
- Robust exposure parsing (handle Option and non-Option types)
- Better error logging with stack traces for APY computation failures
processAccount() was comparing direct staking against cache BEFORE
checking pool data. This caused a flip-flop: when cache=121K and
direct=100K, the bot submitted 100K (missing pool). Next scan it
detected the difference and submitted 121K. Every 5 minutes, forever.
Worse, when direct staking went to 0, the bot submitted 0 which
caused the pallet to delete StakingStartBlock — permanently removing
the user from tracking and zeroing their trust score.
Fix: collect ALL staking data (direct + pool) first, combine into
a single total, then compare against cache. Also adds queryFailed
flag to prevent downgrading stake when pool RPC query fails.
When SubQuery restarts and the last processed block's state has been
pruned by the RPC node, fetchEventsRange would crash the indexer in a
loop. Now returns empty events for pruned blocks so the indexer can
skip forward gracefully.
- payout-bot/bot.js: Periodically calls payoutStakersByPage for all
validators in completed eras. Runs every 10 minutes.
- payout-bot/Dockerfile: Node 20 alpine with @pezkuwi/api
- docker-compose.prod.yml: Add payout-bot service with secret mnemonic
Previously only pool stash accounts were saved as activeStakers.
Direct nominators/validators on Asset Hub were missing, causing
the wallet dashboard to show INACTIVE for direct stakers.
Staking pallet (index 9), FastUnstake (15), and VoterBagsList (100)
were removed from the relay chain. The relay indexer now only tracks
transfers and signed extrinsics. Added safety try-catch to staking
queries in case handlers are accidentally re-enabled.
RPC nodes run in pruned mode (~256 blocks state history).
Setting startBlock to recent blocks ensures all required
state is available. Old blocks have no significant user
transactions to index.
The _getBlockRegistryViaHash method in @polkadot/api calls
getRuntimeVersion.raw(header.parentHash) which fails for genesis
and other pruned blocks. Add fallback to current runtime version.
Substrate nodes prune historical state by default (~256 blocks).
When SubQuery restarts and tries to fetch runtime version for old
blocks, it crashes with "State already discarded". This patch adds
a fallback to current runtime version when historical state is
unavailable, preventing crash loops on pruned RPC endpoints.
Also removes --pull from deploy workflow to avoid unnecessary
full rebuilds on every push.
Index convictionVoting and referenda pallet data to support PezWallet
governance UI — referendum tracking, casting votes, delegation stats,
and delegator vote propagation.
The wallet queries activeStakers with the pool's bonded stash address
and stakingType="relaychain" (unwrapped from nomination-pool). Derive
pool stash accounts from bondedPools and save with correct type.
The event-based StakersElected handler had a bug where paged exposure
data wasn't properly extracted. Add handleRelayBlock that queries
erasStakersPaged directly from live chain state on first block to
ensure all validators and nominators are captured.
