Stop one lost extrinsic from silencing the noter bot (#30)

Every report waits behind the previous one, and a submission only settled
when it reached a block. One the pool dropped, rejected or replaced never
settled, so every report queued after it waited forever. The watchdog meant
to catch a silent bot was armed only after the first scan returned, so when
that first scan was the one that hung there was no watchdog at all: the
production container logged its last line one minute after starting on
2026-10-13 and stayed up, silent, for more than a day.

- submit.js: a submission settles on a terminal status (Dropped, Invalid,
  Usurped, FinalityTimeout) or after SUBMIT_TIMEOUT_MS, and releases its
  status subscription.
- The watchdog is armed before connecting and counts from the process start
  until a scan completes, so a heartbeat left by an earlier process in the
  same container does not count for this one.
- The noter's own account is skipped: the chain refuses a noter's report
  about themselves, so it failed in every scan.
- Unit tests for both rules, run inside the noter image in CI.
This commit is contained in:
SatoshiQaziMuhammed authored and GitHub committed 2026-10-15 06:03:40 +04:00
1 parent d25afe25bb
commit 1e8d5e3675
5 files changed
+209 -40

No files matched your search

+67
View File
@@ -0,0 +1,67 @@
// Watching one extrinsic, and the liveness rule for the scan loop.
//
// Kept apart from bot.js, which connects to three chains the moment it is imported, so that
// both can be tested without a chain.
/**
* Statuses after which an extrinsic will never reach a block. Each of them used to be ignored:
* the promise below only settled on `isInBlock`, so a submission the pool dropped, rejected
* or replaced left it pending forever, and since every submission waits its turn behind the
* previous one, one lost extrinsic stopped every report after it.
*/
const TERMINAL = ['isDropped', 'isInvalid', 'isUsurped', 'isFinalityTimeout'];
/**
* Sign and submit `tx`, settling once it is in a block or can never be.
*
* Resolves on `isInBlock` without a dispatch error, as before: an extrinsic on a fork that is
* later retracted is not lost, because the next scan finds neither a pending nor a cached
* entry and submits again. Rejects on a dispatch error, on a terminal status, and when nothing
* has settled after `timeoutMs`, which also covers a status subscription the node stopped
* answering.
*/
export function watchSubmission(tx, signer, { decodeError, timeoutMs }) {
return new Promise((resolve, reject) => {
let unsub = null;
let settled = false;
const settle = (fn, value) => {
if (settled) return;
settled = true;
clearTimeout(timer);
if (typeof unsub === 'function') unsub();
fn(value);
};
const timer = setTimeout(
() => settle(reject, new Error(`no block after ${timeoutMs} ms`)),
timeoutMs,
);
tx.signAndSend(signer, ({ status, dispatchError }) => {
if (status.isInBlock) {
if (dispatchError) settle(reject, new Error(decodeError(dispatchError)));
else settle(resolve, status.asInBlock.toHex());
return;
}
const terminal = TERMINAL.find((flag) => status[flag]);
if (terminal) settle(reject, new Error(`extrinsic ${status.type ?? terminal.slice(2)}`));
})
.then((u) => {
unsub = u;
if (settled) u();
})
.catch((err) => settle(reject, err));
});
}
/**
* Whether the bot has gone too long without completing a scan.
*
* Counted from the later of the last completed scan and the process start. Counting only from
* the last scan left a process whose *first* scan hung with no watchdog at all, because the
* watchdog was armed after that scan returned: on 2026-10-13 the production container stopped
* logging one minute after it started and stayed up, silent, for more than a day.
*/
export function isStale({ lastScanAt, startedAt, now, staleAfter }) {
const since = Math.max(Number.isFinite(lastScanAt) ? lastScanAt : 0, startedAt);
return now - since > staleAfter;
}