The first scan after the Zagros People upgrade re-confirmed all 98
accounts, one transaction per account, a block each. It ran past the
15-minute staleness window and the watchdog killed it halfway; the next
interval had meanwhile started a second scan beside it, which read the
same accounts before the first one's reports landed and sent them again
(124 submissions for 98 accounts). On mainnet every tracked citizen's
re-confirmation falls due in the same scan, every three days.
- A scan group's updates go out as one utility.forceBatch. The accounts
are unrelated, so one refused report no longer takes the others with
it; each refused item is logged with its decoded error.
- The heartbeat records progress -- a group read and, if it had anything
to report, reported -- instead of a finished scan. A group that cannot
be read, or whose report is refused outright, is not progress, so a
mute noter still goes stale.
- A periodic scan does not start while the previous one is running.
Every report waits behind the previous one, and a submission only settled
when it reached a block. One the pool dropped, rejected or replaced never
settled, so every report queued after it waited forever. The watchdog meant
to catch a silent bot was armed only after the first scan returned, so when
that first scan was the one that hung there was no watchdog at all: the
production container logged its last line one minute after starting on
2026-10-13 and stayed up, silent, for more than a day.
- submit.js: a submission settles on a terminal status (Dropped, Invalid,
Usurped, FinalityTimeout) or after SUBMIT_TIMEOUT_MS, and releases its
status subscription.
- The watchdog is armed before connecting and counts from the process start
until a scan completes, so a heartbeat left by an earlier process in the
same container does not count for this one.
- The noter's own account is skipped: the chain refuses a noter's report
about themselves, so it failed in every scan.
- Unit tests for both rules, run inside the noter image in CI.
SatoshiQaziMuhammed