hasDataChanged() only compared fresh data against CachedStakingDetails -
which stays empty until a submission matures and gets finalized. For any
account whose real stake was genuinely stable, every 5-minute scan still
saw "changed from empty cache", resubmitted, and reset submitted_at to the
current block. Confirmed live: Serok's and QaziM's real, correctly-computed
stake (490,000 / 100,000 HEZ) never once reached CachedStakingDetails/
TrustScores despite multiple successful submissions - each new scan arrived
before the previous one's 300-block dispute window elapsed and overwrote it
with a fresh timer, a self-perpetuating loop.
getPendingData() now reads PendingStakingDetails first; hasDataChanged()
compares against that when present, only falling back to the cache once
nothing is pending. Applied to both the AssetHub and legacy RelayChain-clear
comparisons. Also reordered main()'s cycle to finalizeMaturedPending()
before fullScan(), so anything that matured this cycle gets promoted before
a new scan pass runs at all.
The staking-score pallet's audit-fix (commit 9afd99ae) added a dispute-window
model: receive_staking_details() only writes to PendingStakingDetails, and a
separate, permissionless finalize_staking_details() call promotes it to
CachedStakingDetails once DisputeWindow blocks have passed. No caller of that
promotion step existed anywhere in this bot (or anywhere else in this repo) -
confirmed live: Serok's and QaziM's real stake was correctly submitted to
Pending after the earlier NotRegisteredNoter/nonce fixes, but sat there
indefinitely with no path to ever reaching CachedStakingDetails/TrustScores.
Adds finalizeMaturedPending(), called once per periodic scan cycle (and once
at startup): scans PendingStakingDetails.entries(), finalizes whichever have
matured, batched into one utility.batchAll like the existing submission path,
routed through the same submitQueue so it can never race with a concurrent
submitStakingDetails call.
fullScan() processes accounts in concurrent batches of 10, and each account's
own data-gathering takes variable time, so multiple accounts' submitStakingDetails
calls landed close enough together to race on nonce assignment - the pool then
rejected most of them as duplicate-priority ("Priority is too low ... already in
the pool"). Observed live: 37 of 50 tracked accounts silently dropped in a single
scan cycle, including real accounts with genuine stake (Serok, QaziM) whose trust
scores stayed stuck at 0 as a result even after the separate NotRegisteredNoter
bond issue was fixed.
Routes every submission (both fullScan's batches and the live event listener)
through a single queue so exactly one signAndSend is ever in flight at a time,
in call order - data-gathering stays concurrent (unaffected), only the actual
chain submission is serialized.
Both governance and AH indexers were crashing at pruned blocks (RPC
nodes only retain last ~256 blocks of state). Bumping startBlock to
near-tip allows clean indexing forward from current state.
- governance: 57100 → 1228500 (relay tip - ~200)
- AH: 582592 → 1029200 (AH tip - ~200)
Loss: governance 57100-1228500 (mostly empty, pre-governance-activity);
AH 582592-1029200 (~410k blocks of staking/transfer data — postgres
state will be dropped, accept loss).
Long-term fix: provision Asset Hub archive node so reindexing from
genesis is possible without recurring pruning issues.
Both AH (56400→579775) and Relay (57100→708687) startBlocks were
pointing to blocks whose state has been pruned by the chain nodes.
This caused a crash loop: SubQuery couldn't index those blocks because
api.query calls returned 'State already discarded' errors.
Root cause: AH node has ~200 block state window (default pruning).
Fix: set startBlock to current tip - 50 blocks (safely within window).
Also added --state-pruning=archive to AH collator azad on VPS3 so
this doesn't recur as SubQuery re-syncs.
When SubQuery restarts and hits pruned blocks, events are empty (from patch 4).
This caused two downstream crashes:
- getExtrinsicSuccess(undefined) → TypeError on findIndex
- wrapBlock with events.toArray() → TypeError on undefined
Patches 5-6 handle undefined events gracefully.
- handleBlock now checks activeEra every block as fallback when
StakersElected event is dropped due to pruned state patch
- Clear all existing ActiveStaker entries before saving fresh data
from current era exposure set (prevents stale nominators)
- Re-save pool stash accounts after cleanup
- Robust exposure parsing (handle Option and non-Option types)
- Better error logging with stack traces for APY computation failures
processAccount() was comparing direct staking against cache BEFORE
checking pool data. This caused a flip-flop: when cache=121K and
direct=100K, the bot submitted 100K (missing pool). Next scan it
detected the difference and submitted 121K. Every 5 minutes, forever.
Worse, when direct staking went to 0, the bot submitted 0 which
caused the pallet to delete StakingStartBlock — permanently removing
the user from tracking and zeroing their trust score.
Fix: collect ALL staking data (direct + pool) first, combine into
a single total, then compare against cache. Also adds queryFailed
flag to prevent downgrading stake when pool RPC query fails.
When SubQuery restarts and the last processed block's state has been
pruned by the RPC node, fetchEventsRange would crash the indexer in a
loop. Now returns empty events for pruned blocks so the indexer can
skip forward gracefully.
- payout-bot/bot.js: Periodically calls payoutStakersByPage for all
validators in completed eras. Runs every 10 minutes.
- payout-bot/Dockerfile: Node 20 alpine with @pezkuwi/api
- docker-compose.prod.yml: Add payout-bot service with secret mnemonic
Previously only pool stash accounts were saved as activeStakers.
Direct nominators/validators on Asset Hub were missing, causing
the wallet dashboard to show INACTIVE for direct stakers.
Staking pallet (index 9), FastUnstake (15), and VoterBagsList (100)
were removed from the relay chain. The relay indexer now only tracks
transfers and signed extrinsics. Added safety try-catch to staking
queries in case handlers are accidentally re-enabled.
RPC nodes run in pruned mode (~256 blocks state history).
Setting startBlock to recent blocks ensures all required
state is available. Old blocks have no significant user
transactions to index.
The _getBlockRegistryViaHash method in @polkadot/api calls
getRuntimeVersion.raw(header.parentHash) which fails for genesis
and other pruned blocks. Add fallback to current runtime version.
Substrate nodes prune historical state by default (~256 blocks).
When SubQuery restarts and tries to fetch runtime version for old
blocks, it crashes with "State already discarded". This patch adds
a fallback to current runtime version when historical state is
unavailable, preventing crash loops on pruned RPC endpoints.
Also removes --pull from deploy workflow to avoid unnecessary
full rebuilds on every push.
Index convictionVoting and referenda pallet data to support PezWallet
governance UI — referendum tracking, casting votes, delegation stats,
and delegator vote propagation.
The wallet queries activeStakers with the pool's bonded stash address
and stakingType="relaychain" (unwrapped from nomination-pool). Derive
pool stash accounts from bondedPools and save with correct type.
The event-based StakersElected handler had a bug where paged exposure
data wasn't properly extracted. Add handleRelayBlock that queries
erasStakersPaged directly from live chain state on first block to
ensure all validators and nominators are captured.
- Add handleBlock to query live chain state for existing pool members
- Add second GraphQL engine container for Asset Hub schema (port 3001)
- Register block handler in pezkuwi-assethub.yaml
Add handlePoolBonded/handlePoolUnbonded handlers to Asset Hub SubQuery.
When a user bonds to a nomination pool, they are saved as an ActiveStaker
with networkId=AssetHub and stakingType=nomination-pool. This fixes the
wallet showing INACTIVE for HEZ stakers.
Node.js checks package.json exports before filesystem, so the
mkdir+echo shim didn't work. Now patches the exports map with
node -e to add ./substrate pointing to pezkuwi augmentation.
pezkuwichain