The first scan after the Zagros People upgrade re-confirmed all 98
accounts, one transaction per account, a block each. It ran past the
15-minute staleness window and the watchdog killed it halfway; the next
interval had meanwhile started a second scan beside it, which read the
same accounts before the first one's reports landed and sent them again
(124 submissions for 98 accounts). On mainnet every tracked citizen's
re-confirmation falls due in the same scan, every three days.
- A scan group's updates go out as one utility.forceBatch. The accounts
are unrelated, so one refused report no longer takes the others with
it; each refused item is logged with its decoded error.
- The heartbeat records progress -- a group read and, if it had anything
to report, reported -- instead of a finished scan. A group that cannot
be read, or whose report is refused outright, is not progress, so a
mute noter still goes stale.
- A periodic scan does not start while the previous one is running.
People Chain now counts a staking duration only up to the last confirmation
of the stake plus MaxDataAge (fourteen days), so a stopped noter cannot leave
scores climbing on unchecked data. The bot submitted only on change, which
under that rule would freeze every account whose stake stays put.
confirm.js decides when unchanged stake is sent again: not while a
submission is pending (each one restarts the dispute window), not on
half-read data, and only once the last confirmation is CONFIRM_EVERY_BLOCKS
old (default three days). On a runtime without stakingScore.lastConfirmed
the bot behaves as before.
The direct-staking query failed silently into zeros, and a zero that lands
clears the account's stake clock on chain. It now reports the failure, and a
failure in either half skips the account for that scan.
Every report waits behind the previous one, and a submission only settled
when it reached a block. One the pool dropped, rejected or replaced never
settled, so every report queued after it waited forever. The watchdog meant
to catch a silent bot was armed only after the first scan returned, so when
that first scan was the one that hung there was no watchdog at all: the
production container logged its last line one minute after starting on
2026-10-13 and stayed up, silent, for more than a day.
- submit.js: a submission settles on a terminal status (Dropped, Invalid,
Usurped, FinalityTimeout) or after SUBMIT_TIMEOUT_MS, and releases its
status subscription.
- The watchdog is armed before connecting and counts from the process start
until a scan completes, so a heartbeat left by an earlier process in the
same container does not count for this one.
- The noter's own account is skipped: the chain refuses a noter's report
about themselves, so it failed in every scan.
- Unit tests for both rules, run inside the noter image in CI.
The indexer lockfile still held wasm-crypto 7.5.17, which implements
only sr25519 and hashing; ed25519, secp256k1, derivation and VRF throw.
The bots move from 7.5.20 to 7.5.21, the same outputs on a crate set
without known advisories.
@pezkuwi/rpc-provider lists @bizinikiwi/connect as an optional dependency,
so npm installed it, with smoldot, into both images. Both are GPL-3.0, and
the bots never load them: they reach the chains over WsProvider. The images
now install with --omit=optional; with that install both bots load and read
the relay, Asset Hub and People chains.
The lockfiles still list the packages, so Dependency Review, which denies
GPL-3.0, flagged them on the new payout-bot lockfile (the noter's had them
before). It allows exactly those three packages, and the bots job fails if
an image contains @bizinikiwi or smoldot, so the allowance cannot outlive
the reason for it.
Both images ran `npm install` with version ranges and ignored any lockfile,
so what production ran depended on the day the image was built. The live
containers, built on 2026-10-03, run @pezkuwi/wasm-crypto 7.5.17, whose
ed25519, ecdsa and mnemonic generation are stubs that throw. Both bots sign
with sr25519 only, which is why they kept working.
Each bot now has a package.json and package-lock.json (@pezkuwi/api 16.5.37,
keyring, util and util-crypto 14.0.26, wasm-crypto 7.5.20) and its image
installs exactly that with `npm ci`. Node 20 is past its end of life; the
images move to node:24-alpine.
CI gains a `bots` job that builds each image and runs inside it:
scripts/bot-smoke.mjs checks the installed versions against the lockfile and
signs and verifies with sr25519, ed25519 and ecdsa, and the bot itself must
load and stop at the mnemonic it is not given. Against the previous lockfile
the smoke check fails on mnemonic generation, ed25519 and ecdsa.
yarn.lock: transitive bumps within declared ranges for @babel/core, ajv,
axios (1.x), bn.js, body-parser, brace-expansion, browserslist, defu,
dottie, fast-uri, flatted, follow-redirects, form-data, h3, handlebars,
ip-address, js-yaml, lodash, minimatch, moment, path-to-regexp, picomatch,
qs, simple-git, tar, ws, yaml and @modelcontextprotocol/sdk where in range.
noter/package-lock.json: bn.js 5.2.2 -> 5.2.5, ws 8.19.0 -> 8.22.0.
scripts/python_scripts and scripts/update_projects_description.sh are
removed rather than patched (27 alerts on requirements.txt). They came in
with the initial commit from the Nova project and were never changed: they
call SubQuery's managed service (api.subquery.network) and write
descriptions for the "nova-wallet" organisation. Nothing in CI, package.json
or the other scripts calls them, and our indexers are self-hosted.
Left for upstream SubQuery releases, all build/codegen tooling rather than
the indexer runtime: axios 0.27 via @subql/network-clients, ws 8.18 and
elliptic via its ethers 5, serialize-javascript 6 via terser-webpack-plugin,
uuid 8 via @subql/x-sequelize, tmp via @inquirer/editor, got 9 via
update-notifier.
Checked: yarn install --immutable; subql codegen and build succeed for
pezkuwi.yaml, pezkuwi-assethub.yaml and pezkuwi-governance.yaml.
This module is ESM, so `require` is not defined at runtime. `node --check`
only parses, so it passed while both helpers would have thrown: the write
was swallowed by its catch, and the read fell through to `Infinity`, which
the watchdog reads as a permanently stale heartbeat — it would have exited
the process a minute after every scan and left the bot restarting in a loop.
`fs` is already imported at the top of the file; use it.
The bot pointed at the block-producing nodes' RPC ports. Those bind to
localhost, so once they stopped being exposed to the internet the bot lost
every chain at once and has submitted nothing since. `staking_score` gates
the entire trust score, so a noter that cannot reach the chains does not
degrade the score — it zeroes it for every tracked account.
Point it at the public endpoints instead, which is what an external client
should use and what the code already defaulted to. They are TLS-terminated
rather than plaintext ws:// across the internet, and this drops a hardcoded
node address from the repository.
Reaching the chains again is not enough on its own: the outage lasted three
weeks because reconnecting forever looks identical to working. The bot now
writes a heartbeat when a scan *completes*, a container healthcheck reads it,
and a watchdog exits once it goes stale so the restart policy turns a silent
stall into a visibly crash-looping container.
hasDataChanged() only compared fresh data against CachedStakingDetails -
which stays empty until a submission matures and gets finalized. For any
account whose real stake was genuinely stable, every 5-minute scan still
saw "changed from empty cache", resubmitted, and reset submitted_at to the
current block. Confirmed live: Serok's and QaziM's real, correctly-computed
stake (490,000 / 100,000 HEZ) never once reached CachedStakingDetails/
TrustScores despite multiple successful submissions - each new scan arrived
before the previous one's 300-block dispute window elapsed and overwrote it
with a fresh timer, a self-perpetuating loop.
getPendingData() now reads PendingStakingDetails first; hasDataChanged()
compares against that when present, only falling back to the cache once
nothing is pending. Applied to both the AssetHub and legacy RelayChain-clear
comparisons. Also reordered main()'s cycle to finalizeMaturedPending()
before fullScan(), so anything that matured this cycle gets promoted before
a new scan pass runs at all.
The staking-score pallet's audit-fix (commit 9afd99ae) added a dispute-window
model: receive_staking_details() only writes to PendingStakingDetails, and a
separate, permissionless finalize_staking_details() call promotes it to
CachedStakingDetails once DisputeWindow blocks have passed. No caller of that
promotion step existed anywhere in this bot (or anywhere else in this repo) -
confirmed live: Serok's and QaziM's real stake was correctly submitted to
Pending after the earlier NotRegisteredNoter/nonce fixes, but sat there
indefinitely with no path to ever reaching CachedStakingDetails/TrustScores.
Adds finalizeMaturedPending(), called once per periodic scan cycle (and once
at startup): scans PendingStakingDetails.entries(), finalizes whichever have
matured, batched into one utility.batchAll like the existing submission path,
routed through the same submitQueue so it can never race with a concurrent
submitStakingDetails call.
fullScan() processes accounts in concurrent batches of 10, and each account's
own data-gathering takes variable time, so multiple accounts' submitStakingDetails
calls landed close enough together to race on nonce assignment - the pool then
rejected most of them as duplicate-priority ("Priority is too low ... already in
the pool"). Observed live: 37 of 50 tracked accounts silently dropped in a single
scan cycle, including real accounts with genuine stake (Serok, QaziM) whose trust
scores stayed stuck at 0 as a result even after the separate NotRegisteredNoter
bond issue was fixed.
Routes every submission (both fullScan's batches and the live event listener)
through a single queue so exactly one signAndSend is ever in flight at a time,
in call order - data-gathering stays concurrent (unaffected), only the actual
chain submission is serialized.
processAccount() was comparing direct staking against cache BEFORE
checking pool data. This caused a flip-flop: when cache=121K and
direct=100K, the bot submitted 100K (missing pool). Next scan it
detected the difference and submitted 121K. Every 5 minutes, forever.
Worse, when direct staking went to 0, the bot submitted 0 which
caused the pallet to delete StakingStartBlock — permanently removing
the user from tracking and zeroing their trust score.
Fix: collect ALL staking data (direct + pool) first, combine into
a single total, then compare against cache. Also adds queryFailed
flag to prevent downgrading stake when pool RPC query fails.
SatoshiQaziMuhammed