Make this monorepo's tracked tree identical to the deployed github repo
(github.com/pezkuwichain/pezkuwi-DKS): all RC/AH/People runtimes and framework
carry the mainnet fixes (incl. U128CurrencyToVote everywhere, so the staking
election stall cannot recur), and tools/usdt-bridge matches the current,
public-safe custody tooling (real config/deploy units gitignored, secret-file
locations and server IPs stripped, sanitized *.example templates committed).
Only exception to the mirror: GitLab-native CI is preserved (.gitlab-ci.yml and
.gitlab/), to be rebuilt to match the github CI pipeline in a follow-up.
Same silent-loss pattern as the previous connect-failure fix, found while
auditing further: amount > max_single_tx (both deposit and withdrawal)
warned and Telegram-alerted but never called db.add_deposit/add_withdrawal
- "not auto-recorded" was literally in the log line. Now always recorded
(pending_multisig_approval_amount_exceeds_cap), same as every other
manual-review path.
Backfill throttle: catching up many blocks after real downtime fires a
tight burst of RPC calls that can trip a metered/shared endpoint's rate
limit hard enough to poison the whole connection - observed live. A 150ms
pause between backfilled blocks avoids it.
Telegram notification threshold: at real user volume, a message for every
small successful auto-pay would bury the ones that need a human (failures,
over-cap, reserve problems, anything falling to manual 3-of-5) under
noise. Added notify_threshold (default 500 USDT) gating ONLY the routine
"auto-paid successfully" message on both legs - everything else keeps
notifying unconditionally regardless of amount. Also found and fixed a
real asymmetry: the withdrawal leg's "awaiting multisig approval" case
never notified Telegram at all (the deposit leg always did) - a
withdrawal needing manual review must always page someone.
process_withdrawal_block's reserve/circuit-breaker check used to `continue`
past a Polkadot connect failure without ever calling db.add_withdrawal() -
a transient RPC error (confirmed live: a public-endpoint rate limit) meant
an already-finalized Pezkuwi withdrawal was silently untracked, with no
pending_multisig_approval entry for the signatories to ever see. It now
always records the withdrawal; an unreachable reserve check is treated as
"unverified" (never auto-paid, flagged distinctly from a confirmed
shortfall) rather than as a reason to drop the record.
Also add bounded exponential backoff (2s..32s, 6 attempts) to
connect_to_chain instead of failing on the first error - every caller
either bubbles the failure through a listener's outer reconnect loop or
out of main() to a systemd restart, so a single blip was tearing down and
fully re-initializing both listeners instead of just waiting it out.
Finally, support POLKADOT_RPC_URL/PEZKUWI_RPC_URL env var overrides for
the configured RPC endpoints, so a dedicated/API-keyed provider URL (e.g.
Dwellir, adopted after the public Polkadot Asset Hub endpoint started
rate-limiting us) can live in a tightly-permissioned EnvironmentFile
instead of the git-tracked bridge_config.json - same pattern already used
for PEZBRIDGE_TELEGRAM_TOKEN and AUTOMATION_KEY_MNEMONIC.
The automation key's signing clients (pezkuwi_signing_client/
polkadot_signing_client) are created once at startup and reused for every
auto-pay call. Same failure class as the listener's live subscription
(silent WS death, no error until the next real use, see
LIVE_SUBSCRIPTION_STALL_TIMEOUT's comment): a real withdrawal auto-pay hit
"Connection reset by peer... restart required" twice in a row today,
sending both to manual 3-of-5 review even though the automation key's
Polkadot approval had just been granted and should have covered them.
Both execute_wusdt_auto_payout/execute_usdt_auto_payout call sites now
retry once against a genuinely fresh client on failure before giving up -
cheap for an occasional per-event call (unlike the listener's persistent
subscription, no reason to avoid reconnecting every time). Falls back to
manual review exactly as before if the retry also fails - never a new
failure mode, just a narrower one.
A real 1 USDT deposit landed on-chain at block #18250967 and was never
detected: subscribe_finalized()'s stream silently stopped yielding new
blocks after startup - no error, no closed stream, the process stayed up
and "Listening for deposits..." was the last line logged for 3+ hours while
the deposit sat undetected the whole time. Checkpoint + backfill (added
earlier) only covers a gap across a process restart; it does nothing for a
stall with no restart at all, which is exactly what happened here.
Both listen_deposits and listen_withdrawals now wrap their subscribe loop in
an outer reconnect loop: each blocks.next() call is bounded by a 120s
timeout (Asset Hub blocks land every ~6-12s, so this is a wide margin), and
a timeout, a closed stream, or a stream-yielded error all trigger a full
reconnect (fresh client + fresh legacy RPC) rather than silently hanging or
crashing the whole process. The existing checkpoint-vs-head backfill then
catches up whatever was missed during the stall, same as it already does
for an ordinary restart.
listen_deposits/listen_withdrawals previously used ONLY a live finalized-
block subscription with no persisted cursor - the last_polkadot_block/
last_pezkuwi_block fields existed in BridgeStats but were never read or
written anywhere. A transient subscription gap (no logging, no crash, just
a silently skipped block) could make a real deposit vanish from detection
entirely. Confirmed live: a real user's 1 USDT deposit landed correctly at
the multisig custody address (verified via direct chain balance query) but
was never once logged as detected across 7+ hours of continuous uptime with
zero errors - see res/validators-tiki.md 2026-07-15 for the full incident.
Both listeners now:
1. On startup, compare the persisted checkpoint against current chain head
and explicitly backfill (fetch block-by-hash via the legacy RPC's
chain_get_block_hash, then process identically to the live path) any gap
before starting the live subscription.
2. During live operation, detect if the subscription itself ever jumps ahead
unexpectedly and backfill that gap too, paging Telegram when it happens.
3. Persist the checkpoint after every block processed, live or backfilled.
process_deposit_block/process_withdrawal_block are the single shared
per-block handlers - backfilled and live blocks run through identical logic,
so there's no way for the two paths to silently diverge in behavior.
wallet-android's Bridge screen has always advertised a 1 USDT minimum
(MIN_USDT constant), but usdt-bridge's actual min_deposit/min_withdraw
defaulted to 10 USDT since the tool's original scaffold commit - nobody had
ever reconciled the two. Confirmed live: a real user swap of ~1 USDT landed
correctly at the multisig custody address but was silently skipped by
listen_deposits (amount < min_deposit), so the user never received wUSDT
despite the wallet telling them 1 USDT was a valid amount.
Backend now matches the value already promised to users, not the other way
around - changing the wallet's displayed minimum after users have already
relied on it would be the wrong direction to reconcile this in.
Small/mid-size swaps no longer have to wait on 3-of-5 for every single
transaction. usdt-bridge can now execute Assets.transfer_approved directly via
the automation key's existing delegated allowance when a deposit/withdrawal
passes all three independent guards:
- max_single_tx (50,000 USDT): any single transaction above this always queues
for manual multisig review, no exceptions.
- auto_pay_daily_cap (200,000 USDT): a rolling 24h budget tracked separately
per leg (wUSDT payouts vs USDT payouts draw on separate on-chain allowances),
not a per-tx ceiling - lets many legitimate sub-cap swaps clear same-day.
- circuit breaker: if >=50% of the daily cap is consumed within 30 minutes,
auto-pay halts entirely until a human runs reset-circuit-breaker - a burst
that fast reads as a compromised key or a bug, not organic demand.
The automation key can only ever move funds via its pre-approved allowance
(never mint/burn/set_team/transfer_ownership - those stay multisig-only), so
its worst case is bounded by whatever allowance + daily cap remain, not
unbounded custody. Withdrawal auto-pay releases real USDT immediately but
defers the corresponding wUSDT burn (Owner/Admin-only, no delegation exists
for it) to a periodic manual multisig batch - tracked via a distinct
auto_paid_burn_pending status so it can't be silently forgotten.
Auto-pay is opt-in at the infra level: AUTOMATION_KEY_MNEMONIC unset means
100% manual behavior, unchanged from before. Bumped pezbridge-bot's
renewal_threshold/topup_amount so the next 3-of-5-approved allowance renewal
scales to the new daily cap.
- show_status/show_balances/show_reserves checked the wrong account:
load_wallet_addresses(&config.seed_path), the old, explicitly-retired
single-operator key, never the multisig custody addresses. These
commands were actively misleading, not just unfinished.
- listen_deposits/listen_withdrawals ran as concurrent tasks sharing one
BridgeDatabase JSON file with no locking and a non-atomic write - two
events landing close together could produce a lost update, and a
mid-write crash left a truncated file that hard-crashed the next
start. Added a process-wide async mutex around every load-mutate-save
critical section, and made save() write-temp-then-rename so a crash
mid-write can never corrupt the file in place. load() now refuses to
silently treat a corrupt file as empty (would erase replay-protection
history) - only a genuinely missing file gets a fresh database.
- The reserve-shortfall circuit breaker - the single most safety-critical
signal this tool can produce - only called a local error!() log macro.
Nobody was notified. Added real Telegram alerting (reusing
PezbridgeBot's exact chats/send pattern rather than a second channel)
for reserve shortfalls, max-single-tx-cap deposits/withdrawals needing
manual review, and every new pending_multisig_approval row.
Also removed ProcessDeposit/MintWusdt/TransferUsdt - dead-but-dangerous
CLI subcommands that still loaded the retired operator key and attempted
direct mint/transfer/burn calls that only happened to fail today because
that key lost asset authority. A latent trap, not a safe no-op by
design; production flow is listen-all -> pending_multisig_approval ->
pwap-web signing, never a single-key path.
cargo check: 0 errors, 0 warnings (both binaries).
Renewing the automation key's spending allowance always needs 3-of-5
multisig approval regardless, so having this bot hold one signer's
(Noter's) private key on a VPS to auto-submit one of those three
signatures was never actually saving a real step - the other two
signers still had to sign manually every time. Once pwap-web shipped
a real multisig operations UI (and now wallet-android's Bridge screen
too), the bot's only remaining job is to detect a low allowance and
notify all 5 signers via Telegram - they each sign with their own
wallet through one of pwap-web, pezbridge-sign.pex.mom, or the wallet
app directly. Noter's mnemonic was removed from the VPS entirely
(shred -u) after this rewrite was deployed and verified.
Also adds gen_automation_key.rs (one-off key generation helper) and
the updated systemd unit with LoadCredential removed.
listen_deposits/listen_withdrawals matched the bridge address by
checking whether a Debug-formatted string of the decoded dynamic Value
contained a 10-char slice of the address's SS58 text - flagged earlier
this session as unverified and provably broken (that same Debug string
was also fed into convert_ss58_prefix, which only parses real SS58
text and could never have succeeded on it).
Added account_id_from_dynamic(), which decodes an AccountId32's actual
32 bytes out of the event field (peeling the single-item composite
wrapper subxt's dynamic API adds, same nesting seen in Tiki storage
this session) and account_bytes_to_ss58() to re-encode them properly.
Both listeners now compare raw bytes against the target address
instead of matching strings, and derive the sender's real SS58 address
for logging and convert_ss58_prefix() instead of a Debug string.
Verified against a live event, not just read: sent a real 1 wUSDT
transfer from Treasury_1 to the multisig and confirmed the decoder
correctly extracted and matched the multisig's AccountId32 bytes from
the resulting Assets.Transferred event.
The single-key custodial model (see prior commit's own doc comment:
"Custodial: A single keypair controls both sides") was replaced with
a 3-of-5 multisig spanning the Serok/SerokiMeclise/Xezinedar/Berdevk/
Noter tiki role-holders (full record in res/validators-tiki.md,
outside this repo). Discovered along the way: wUSDT's ~100M supply
had zero real USDT backing on Polkadot Asset Hub - Treasury_1 (the
actual asset owner, not the seed_path wallet this tool assumed) held
0 USDT/DOT there, with ~99M of the 100M supply sitting undistributed
in Treasury_1's own account. Migrated Pezkuwi Asset Hub custody in one
atomic batch (transfer 1M wUSDT to the multisig, set_team, then
transfer_ownership last) signed by Treasury_1 - tx
0xd5dc550c5cf04f323e5c1d9c13f2056acafc9f56404c657b9743c15ff3a3aacb,
verified on-chain.
listen_deposits/listen_withdrawals no longer sign and submit
mint/burn/transfer - no key this process holds has authority over the
multisig account. They now only detect on-chain events and record
status=pending_multisig_approval with the exact call parameters an
operator needs; actual approval happens through pwap-web's existing
multisig UI (shared/lib/multisig.ts, shared/lib/usdt.ts), not this
relayer. BridgeConfig gained custody_multisig_polkadot/_pezkuwi
fields.
Added pezkuwi-subxt/pezkuwi-subxt-signer alongside the existing
vanilla subxt dependency: vanilla subxt's codegen hardcodes the
`sp_runtime` module path and its default signed-extension handling
produced "bad signature" on submission against this chain's actual
metadata - pezkuwi-subxt (following vendor/pezkuwi-subxt's own
grant_noter_tiki.rs example: PezkuwiConfig, XCM v3, sudo_unchecked_weight)
is what actually signs and submits successfully. Also used to grant
the Xezinedar and Berdevk tiki roles (previously missing signatories
for the multisig) to two validators via a relay-chain sudo XCM Transact,
since the People chain has no local Sudo pallet.
The withdrawal direction (burn wUSDT -> release real USDT) had no
listener at all - nothing watched Pezkuwi for redemption deposits, and
the manual ProcessWithdrawals command was an empty stub. The deposit
listener also had two real bugs: it minted to the bridge's own address
instead of the sender's (a leftover "for demo" placeholder), and it
never checked whether a deposit had already been processed before
minting again.
- Add is_deposit_processed/is_withdrawal_processed idempotency checks,
enforced before every mint/burn/release, not just recorded after.
- Fix listen_deposits() to mint to the sender's actual Pezkuwi address
(same AccountId32, re-encoded under Pezkuwi's SS58 prefix - both
chains share one derivation, so no memo/registration is needed) and
to persist processed deposits to the database, which it never did.
- Add listen_withdrawals(), mirroring the deposit listener: detects
wUSDT sent to bridge custody, burns it, verifies the Polkadot
custody wallet actually holds enough real USDT before releasing
(circuit breaker - an under-collateralized peg must halt loudly,
not silently short-pay), then releases to the sender's Polkadot
address.
- Add max_single_tx cap so one bad/anomalous event can't move an
unbounded amount automatically.
- Add a Reserves command reporting real custody balance vs circulating
wUSDT supply - the exact invariant that, when broken, is what the
wallet's "waiting for liquidity" banner was masking.
- load_keypair() now prefers BRIDGE_SEED_MNEMONIC (meant to be
Vault-injected, matching how the exchange's hot wallets are already
provisioned) over the plaintext seed_path file, which remains only
as a local/dev fallback with a loud warning.
Flagged, not fixed here: the event-matching in both listeners
(Debug-formatted Value string containing a slice of the bridge's SS58
text) is unverified against a live node in this environment and should
be replaced with subxt's typed/codegen API before this handles real
value. Custody is still single-keypair (no multisig/MPC threshold
signing) - that is an infrastructure/key-ceremony decision, not a code
change, and is the highest-priority follow-up.
chain-spec-tool:
- CLI utility for chain spec manipulation and validation
- Used for mainnet chain spec generation and verification
usdt-bridge:
- Custodial bridge for wUSDT token operations
- Handles USDT wrapping/unwrapping between external chains and Asset Hub
- Configuration in bridge_config.json
Satoshi Qazi Muhammed