Embedded PezkuwiChain light client with RPC fallback (Zagros) (#30)

* Add pezkuwi-light: embedded PezkuwiChain light client binding A JNI binding around the PezkuwiChain fork of smoldot (pezkuwichain/smoldot), which verifies PezkuwiChain blocks under the chain's sovereign constants. Upstream smoldot cannot. The wallet reaches its chains through JSON-RPC WebSockets, so the light client is served the same way: a WebSocket server bound to 127.0.0.1 on a random port. Every path carries a random per-process token: ws://127.0.0.1:<port>/<token>/<chain key>. - A wrong token, a missing token or an unregistered chain is refused during the handshake with 404. - Each connection adds its own smoldot chain instance. smoldot shares the sync between instances of the same chain. - Each registered chain's finalized state is written to disk every minute, through a temporary file and a rename. The next start resumes from it. Kotlin surface: PezkuwiLight.start(dir), addChain(key, spec, relayKey), chainUrl(base, key). The module is not yet a dependency of the app. Tested against the live Zagros testnet, with the relay chain and People specs: - dks.kurd resolves through NavnisanApi_resolve to the same bytes the full node returns. - An unregistered name returns 0x00. - The saved state lets a new client resolve. - Reverting the token check turns the routing test red. The library builds for arm, arm64, x86 and x86_64 with 16 KB aligned LOAD segments (0x4000). * pezkuwi-light: secure WebSockets and the app's resolver for peers smoldot's DefaultPlatform cannot open wss connections and resolves names through the system resolver. Where our domains are blocked, both fail: a poisoned DNS answer decides where the client connects, and plain libp2p TCP is recognisable on the wire. PezkuwiPlatform replaces it: - Opens /dns/<host>/tcp/<port>/wss over TLS, using rustls with the ring provider and the Mozilla roots, so a connection looks like any HTTPS WebSocket. - Resolves every name through a Resolve trait. On Android this is the app's HostResolver, called over JNI, so the light client follows the same DNS policy as the rest of the app (EncryptedFirstDns: DoH first, system resolver last). - Tries IPv4 before IPv6, and gives up on an address after 10 s. - With setSecureOnly(true) it opens nothing but secure WebSockets. Peers found through the DHT advertise plain TCP too, and none of them is used. The peer's identity is still checked end to end by libp2p Noise. Tested against live Zagros with only the /wss boot nodes in the specs and secure-only on: relay and People sync and dks.kurd resolves to the full node's bytes. Nail: with secure-only on and the specs' plain TCP boot nodes, nothing syncs and the test times out red. * pezkuwi-light: measure a start from the saved database in its own process An app launch resumes from the database in a fresh process. The in-process warm measurement ran beside a client that was still syncing, so it did not show that case. The new ignored test resolves dks.kurd after a start from the database in PEZKUWI_LIGHT_DB_DIR. On the first run, when no database exists yet, it syncs and waits for a save. The tests also log through env_logger, so RUST_LOG shows smoldot's own timeline when a measurement needs explaining. * Fall back to the embedded light client when every Zagros RPC node fails ChainConnection stays on RPC first. A light client loads the peer-to-peer nodes, and those nodes have a fixed number of light client slots, so RPC is the default. When every RPC node of a supported chain has failed in a row, the connection moves to the light client, served at ws://127.0.0.1:<port>/<token>/<genesis>. RPC is retried after one minute. The wait doubles up to 15 minutes while RPC keeps failing. The connection returns to RPC as soon as a retry connects. If the light client cannot start, or its own connection fails, the connection stays on RPC. LightFallbackPolicy holds these rules. Its tests cover each one, and five mutations of the policy each turn a test red. ChainConnection handles RPC node changes, socket states and a 15 s tick as one event stream, one event at a time, so a node change and a fallback cannot race. LightClients starts the client on first use. A wallet whose RPC answers never runs it. LightClients registers Zagros and Zagros People, the chains the fork can verify, from specifications bundled at build time from pezkuwi-wallet-utils light-client/: the genesis state root, the /wss boot nodes on 443 first, and the relay checkpoint. Peer names resolve through EncryptedFirstDns, which CommonApi now exposes to the runtime module. * pezkuwi-light: strip in the cargo profile The Android Gradle plugin packages a native library it cannot strip as it is. With debug info, libpezkuwi_light.so is 147 MB per ABI. Stripped it is 8.6 MB. The first CI build of this library ran out of disk: four targets with debug info filled the runner. Stripping in the release profile, with debug info off, fixes both: - The library is 8.6 MB for arm64 and keeps its three JNI symbols. - The build directory is 389 MB per target. * Log the light client fallback's decisions Each RPC failure the fallback sees, the light client URL it got, and each RPC retry are logged under LightFallback. The first device run on a blocked Zagros RPC showed no fallback and no trace of where it stopped.
2026-11-11 13:17:15 +00:00 · 2026-11-03 03:30:15 +04:00
1 parent 7b8fd731f8
commit 8549501a62
@@ -6,6 +6,7 @@ import dagger.Module
 import dagger.Provides
 import io.novafoundation.nova.common.BuildConfig
 import io.novafoundation.nova.common.data.network.NetworkApiCreator
+import io.novafoundation.nova.common.data.network.dns.EncryptedFirstDns
 import io.novafoundation.nova.common.data.network.rpc.BulkRetriever
 import io.novafoundation.nova.common.data.storage.Preferences
 import io.novafoundation.nova.common.di.scope.ApplicationScope
@@ -25,6 +26,7 @@ import io.novafoundation.nova.runtime.multiNetwork.connection.ConnectionPool
 import io.novafoundation.nova.runtime.multiNetwork.connection.ConnectionSecrets
 import io.novafoundation.nova.runtime.multiNetwork.connection.Web3ApiPool
 import io.novafoundation.nova.runtime.multiNetwork.connection.autobalance.NodeAutobalancer
+import io.novafoundation.nova.runtime.multiNetwork.connection.light.LightClients
 import io.novafoundation.nova.runtime.multiNetwork.connection.autobalance.strategy.NodeSelectionStrategyProvider
 import io.novafoundation.nova.runtime.multiNetwork.connection.node.healthState.NodeHealthStateTesterFactory
 import io.novafoundation.nova.runtime.multiNetwork.runtime.AsyncChainSyncDispatcher
@@ -179,16 +181,25 @@ class ChainRegistryModule {
            .build()
    )

+    @Provides
+    @ApplicationScope
+    fun provideLightClients(
+        context: Context,
+        encryptedFirstDns: EncryptedFirstDns,
+    ) = LightClients(context, encryptedFirstDns)
+
    @Provides
    @ApplicationScope
    fun provideChainConnectionFactory(
        socketProvider: Provider<SocketService>,
        externalRequirementsFlow: MutableStateFlow<ChainConnection.ExternalRequirement>,
        nodeAutobalancer: NodeAutobalancer,
+        lightClients: LightClients,
    ) = ChainConnectionFactory(
        externalRequirementsFlow,
        nodeAutobalancer,
        socketProvider,
+        lightClients,
    )

    @Provides
@@ -3,6 +3,7 @@ package io.novafoundation.nova.runtime.di
 import android.content.Context
 import com.google.gson.Gson
 import io.novafoundation.nova.common.data.network.NetworkApiCreator
+import io.novafoundation.nova.common.data.network.dns.EncryptedFirstDns
 import io.novafoundation.nova.common.data.repository.AssetsIconModeRepository
 import io.novafoundation.nova.common.data.storage.Preferences
 import io.novafoundation.nova.common.interfaces.FileProvider
@@ -32,4 +33,6 @@ interface RuntimeDependencies {
    fun chainAssetDao(): ChainAssetDao

    fun assetsIconModeService(): AssetsIconModeRepository
+
+    fun encryptedFirstDns(): EncryptedFirstDns
 }
@@ -5,6 +5,8 @@ import io.novafoundation.nova.common.utils.LOG_TAG
 import io.novafoundation.nova.common.utils.share
 import io.novafoundation.nova.runtime.multiNetwork.chain.model.Chain
 import io.novafoundation.nova.runtime.multiNetwork.connection.autobalance.NodeAutobalancer
+import io.novafoundation.nova.runtime.multiNetwork.connection.light.LightClients
+import io.novafoundation.nova.runtime.multiNetwork.connection.light.LightFallbackPolicy
 import io.novasama.substrate_sdk_android.wsrpc.SocketService
 import io.novasama.substrate_sdk_android.wsrpc.interceptor.WebSocketResponseInterceptor
 import io.novasama.substrate_sdk_android.wsrpc.interceptor.WebSocketResponseInterceptor.ResponseDelivery
@@ -14,6 +16,7 @@ import io.novasama.substrate_sdk_android.wsrpc.state.SocketStateMachine.State
 import kotlinx.coroutines.CoroutineScope
 import kotlinx.coroutines.Dispatchers
 import kotlinx.coroutines.cancel
+import kotlinx.coroutines.delay
 import kotlinx.coroutines.flow.Flow
 import kotlinx.coroutines.flow.MutableSharedFlow
 import kotlinx.coroutines.flow.MutableStateFlow
@@ -21,6 +24,7 @@ import kotlinx.coroutines.flow.SharingStarted
 import kotlinx.coroutines.flow.distinctUntilChanged
 import kotlinx.coroutines.flow.filter
 import kotlinx.coroutines.flow.first
+import kotlinx.coroutines.flow.flow
 import kotlinx.coroutines.flow.launchIn
 import kotlinx.coroutines.flow.map
 import kotlinx.coroutines.flow.mapNotNull
@@ -28,12 +32,14 @@ import kotlinx.coroutines.flow.merge
 import kotlinx.coroutines.flow.onEach
 import kotlinx.coroutines.flow.shareIn
 import kotlinx.coroutines.flow.stateIn
+import kotlinx.coroutines.flow.transform
 import javax.inject.Provider

 class ChainConnectionFactory(
    private val externalRequirementFlow: Flow<ChainConnection.ExternalRequirement>,
    private val nodeAutobalancer: NodeAutobalancer,
    private val socketServiceProvider: Provider<SocketService>,
+    private val lightClients: LightClients,
 ) {

    suspend fun create(chain: Chain): ChainConnection {
@@ -41,7 +47,8 @@ class ChainConnectionFactory(
            socketService = socketServiceProvider.get(),
            externalRequirementFlow = externalRequirementFlow,
            nodeAutobalancer = nodeAutobalancer,
-            initialChain = chain
+            initialChain = chain,
+            lightClients = lightClients.takeIf { it.supports(chain.id) },
        )

        connection.setup()
@@ -58,6 +65,11 @@ private const val BLUST_RATE_LIMIT_ERROR_CODE = -32097

 private const val ON_FINALITY_RATE_LIMIT_ERROR_CODE = -32029

+/** How often a connection on the light client checks whether to retry RPC. */
+private const val LIGHT_FALLBACK_TICK_MILLIS = 15_000L
+
+private const val LIGHT_FALLBACK_TAG = "LightFallback"
+
 private val RATE_LIMIT_ERROR_CODES = listOf(
    INFURA_ERROR_CODE,
    ALCHEMY_ERROR_CODE,
@@ -71,6 +83,7 @@ class ChainConnection internal constructor(
    private val externalRequirementFlow: Flow<ExternalRequirement>,
    private val nodeAutobalancer: NodeAutobalancer,
    initialChain: Chain,
+    private val lightClients: LightClients? = null,
 ) : CoroutineScope by CoroutineScope(Dispatchers.Default),
    WebSocketResponseInterceptor {

@@ -94,11 +107,14 @@ class ChainConnection internal constructor(
        .distinctUntilChanged()
        .share(SharingStarted.Eagerly)

-    val currentUrl = nodeAutobalancer.connectionUrlFlow(
+    private val rpcUrl = nodeAutobalancer.connectionUrlFlow(
        chainId = initialChain.id,
        changeConnectionEventFlow = nodeChangeSignal,
        availableNodesFlow = availableNodes,
-    ).share(SharingStarted.Eagerly)
+    )
+
+    val currentUrl = (lightClients?.let { rpcUrl.withLightFallback(it) } ?: rpcUrl)
+        .share(SharingStarted.Eagerly)

    suspend fun setup() {
        socketService.setInterceptor(this)
@@ -138,15 +154,7 @@ class ChainConnection internal constructor(
        socketService.stop()
    }

-    private suspend fun actualUrl(): String? {
-        return when (val stateSnapshot = state.first()) {
-            is State.WaitingForReconnect -> stateSnapshot.url
-            is State.Connecting -> stateSnapshot.url
-            is State.Connected -> stateSnapshot.url
-            State.Disconnected -> null
-            is State.Paused -> stateSnapshot.url
-        }
-    }
+    private suspend fun actualUrl(): String? = state.first().urlOrNull()

    private fun Flow<State>.nodeChangeEvents(): Flow<Unit> {
        return mapNotNull { stateValue ->
@@ -156,6 +164,86 @@ class ChainConnection internal constructor(

    private fun State.needsAutobalance() = this is State.WaitingForReconnect && attempt > 1

+    private sealed interface FallbackEvent {
+        class Rpc(val node: NodeWithSaturatedUrl?) : FallbackEvent
+        class Socket(val state: State) : FallbackEvent
+        object Tick : FallbackEvent
+    }
+
+    /**
+     * Moves the connection to the light client once every RPC node has failed in a row, and back
+     * to RPC when a retry connects (see [LightFallbackPolicy]). Events are handled one at a time,
+     * in the order they arrive.
+     */
+    private fun Flow<NodeWithSaturatedUrl?>.withLightFallback(lightClients: LightClients): Flow<NodeWithSaturatedUrl?> {
+        Log.d(LIGHT_FALLBACK_TAG, "${chain.value.name}: light client fallback armed")
+        val policy = LightFallbackPolicy()
+        var latestRpc: NodeWithSaturatedUrl? = null
+        var light: NodeWithSaturatedUrl? = null
+        val ticks = flow {
+            while (true) {
+                delay(LIGHT_FALLBACK_TICK_MILLIS)
+                emit(FallbackEvent.Tick)
+            }
+        }
+        val rpcNodeCount = { chain.value.nodes.nodes.count { it.connectionType == Chain.Node.ConnectionType.WSS } }
+
+        return merge(map(FallbackEvent::Rpc), state.map(FallbackEvent::Socket), ticks).transform { event ->
+            when (event) {
+                is FallbackEvent.Rpc -> {
+                    latestRpc = event.node
+                    if (policy.source == LightFallbackPolicy.Source.RPC) emit(event.node)
+                }
+
+                is FallbackEvent.Socket -> {
+                    val onLight = light != null && event.state.urlOrNull() == light?.saturatedUrl
+                    if (event.state is State.Connected && !onLight) policy.onRpcConnected()
+
+                    if (event.state.needsAutobalance()) {
+                        Log.d(LIGHT_FALLBACK_TAG, "${chain.value.name}: ${event.state} failed, source ${policy.source}, on light $onLight")
+                        if (onLight) {
+                            policy.onLightUnavailable(rpcNodeCount())
+                            emit(latestRpc)
+                        } else if (policy.onRpcFailure(rpcNodeCount(), System.currentTimeMillis())) {
+                            val url = lightClients.url(chain.value.id)
+                            Log.d(LIGHT_FALLBACK_TAG, "${chain.value.name}: light client url $url")
+                            if (url == null) {
+                                policy.onLightUnavailable(rpcNodeCount())
+                            } else {
+                                Log.d(LIGHT_FALLBACK_TAG, "Every RPC node of ${chain.value.name} failed, using the light client")
+                                light = lightClientNode(url).also { emit(it) }
+                            }
+                        }
+                    }
+                }
+
+                FallbackEvent.Tick -> if (policy.onTick(rpcNodeCount(), System.currentTimeMillis())) {
+                    Log.d(LIGHT_FALLBACK_TAG, "${chain.value.name}: retrying RPC")
+                    emit(latestRpc)
+                }
+            }
+        }
+    }
+
+    private fun lightClientNode(url: String) = NodeWithSaturatedUrl(
+        node = Chain.Node(
+            chainId = chain.value.id,
+            unformattedUrl = url,
+            name = "Light client",
+            orderId = Int.MAX_VALUE,
+            isCustom = false,
+        ),
+        saturatedUrl = url,
+    )
+
+    private fun State.urlOrNull(): String? = when (this) {
+        is State.WaitingForReconnect -> url
+        is State.Connecting -> url
+        is State.Connected -> url
+        State.Disconnected -> null
+        is State.Paused -> url
+    }
+
    override fun onRpcResponseReceived(rpcResponse: RpcResponse): ResponseDelivery {
        val error = rpcResponse.error

@@ -0,0 +1,76 @@
+package io.novafoundation.nova.runtime.multiNetwork.connection.light
+
+import android.content.Context
+import android.util.Log
+import io.novafoundation.nova.common.utils.LOG_TAG
+import io.novafoundation.nova.lightclient.HostResolver
+import io.novafoundation.nova.lightclient.PezkuwiLight
+import io.novafoundation.nova.runtime.multiNetwork.chain.model.ChainId
+import kotlinx.coroutines.Dispatchers
+import kotlinx.coroutines.sync.Mutex
+import kotlinx.coroutines.sync.withLock
+import kotlinx.coroutines.withContext
+import okhttp3.Dns
+import java.io.File
+
+/**
+ * The embedded light client (`bindings/pezkuwi-light`) for the chains it can verify.
+ *
+ * Started on first use, so a wallet whose RPC nodes answer never runs it. Chain specifications
+ * come from the APK (`light-client/` assets, built from pezkuwi-wallet-utils): the genesis state
+ * root, boot nodes and, for a relay chain, a checkpoint. Peer host names resolve through [dns],
+ * the app's DNS-over-HTTPS-first resolver.
+ */
+class LightClients(
+    private val context: Context,
+    private val dns: Dns,
+) {
+
+    private class LightChain(val asset: String, val relay: ChainId?)
+
+    private val chains = mapOf(
+        ZAGROS to LightChain("light-client/zagros.json", relay = null),
+        ZAGROS_PEOPLE to LightChain("light-client/people-zagros.json", relay = ZAGROS),
+    )
+
+    private val mutex = Mutex()
+    private var baseUrl: String? = null
+    private val registered = mutableSetOf<ChainId>()
+
+    fun supports(chainId: ChainId): Boolean = chainId in chains
+
+    /** Loopback JSON-RPC URL of [chainId], or null when the light client cannot serve it. */
+    suspend fun url(chainId: ChainId): String? = mutex.withLock {
+        withContext(Dispatchers.IO) {
+            runCatching {
+                if (!supports(chainId)) return@runCatching null
+                val base = baseUrl ?: PezkuwiLight.start(File(context.filesDir, DATABASE_DIR), resolver()).also { baseUrl = it }
+                register(chainId)
+                PezkuwiLight.chainUrl(base, chainId)
+            }.onFailure {
+                Log.w(LOG_TAG, "Light client unavailable for $chainId", it)
+            }.getOrNull()
+        }
+    }
+
+    private fun register(chainId: ChainId) {
+        if (chainId in registered) return
+        val chain = chains.getValue(chainId)
+        chain.relay?.let(::register)
+
+        val spec = context.assets.open(chain.asset).bufferedReader().use { it.readText() }
+        PezkuwiLight.addChain(chainId, spec, chain.relay)
+        registered += chainId
+    }
+
+    private fun resolver() = HostResolver { host ->
+        dns.lookup(host).mapNotNull { it.hostAddress }.toTypedArray()
+    }
+
+    private companion object {
+        const val ZAGROS: ChainId = "1b0b4727bbb5e44ed587d1056b4ad537ffb330034509a58866a088d77e792cd5"
+        const val ZAGROS_PEOPLE: ChainId = "c72b38f36affaeb2661018b201c2be02e8c09fc699640525d39200a10fd77878"
+
+        const val DATABASE_DIR = "light-client"
+    }
+}
@@ -0,0 +1,69 @@
+package io.novafoundation.nova.runtime.multiNetwork.connection.light
+
+/**
+ * When a chain's connection moves from its RPC nodes to the embedded light client, and back.
+ *
+ * RPC comes first: a light client puts load on the peer-to-peer nodes that RPC does not, and
+ * those nodes have a fixed number of light client slots. Once every RPC node has failed in a row,
+ * the light client takes over. RPC is retried after [initialBackoffMillis], doubling up to
+ * [maxBackoffMillis] while it keeps failing, and takes back the connection once it connects.
+ *
+ * Not thread-safe: one connection feeds it its events in order.
+ */
+class LightFallbackPolicy(
+    private val initialBackoffMillis: Long = 60_000,
+    private val maxBackoffMillis: Long = 15 * 60_000,
+) {
+
+    enum class Source { RPC, LIGHT }
+
+    var source = Source.RPC
+        private set
+
+    private var consecutiveRpcFailures = 0
+    private var backoffMillis = initialBackoffMillis
+    private var rpcRetryAt: Long? = null
+
+    /** An RPC node failed. True when the light client should take over now. */
+    fun onRpcFailure(rpcNodeCount: Int, nowMillis: Long): Boolean {
+        if (source != Source.RPC) return false
+
+        consecutiveRpcFailures++
+        if (consecutiveRpcFailures < rpcNodeCount.coerceAtLeast(1)) return false
+
+        source = Source.LIGHT
+        rpcRetryAt = nowMillis + backoffMillis
+        backoffMillis = (backoffMillis * 2).coerceAtMost(maxBackoffMillis)
+        return true
+    }
+
+    fun onRpcConnected() {
+        source = Source.RPC
+        consecutiveRpcFailures = 0
+        backoffMillis = initialBackoffMillis
+        rpcRetryAt = null
+    }
+
+    /**
+     * True when RPC should be tried again now. One more RPC failure then hands the connection
+     * straight back to the light client, without walking every node again.
+     */
+    fun onTick(rpcNodeCount: Int, nowMillis: Long): Boolean {
+        val retryAt = rpcRetryAt ?: return false
+        if (source != Source.LIGHT || nowMillis < retryAt) return false
+
+        returnToRpc(rpcNodeCount)
+        return true
+    }
+
+    /** The light client could not start or its connection failed: RPC is all there is. */
+    fun onLightUnavailable(rpcNodeCount: Int) {
+        if (source == Source.LIGHT) returnToRpc(rpcNodeCount)
+    }
+
+    private fun returnToRpc(rpcNodeCount: Int) {
+        source = Source.RPC
+        consecutiveRpcFailures = (rpcNodeCount - 1).coerceAtLeast(0)
+        rpcRetryAt = null
+    }
+}
@@ -0,0 +1,94 @@
+package io.novafoundation.nova.runtime.multiNetwork.connection.light
+
+import io.novafoundation.nova.runtime.multiNetwork.connection.light.LightFallbackPolicy.Source
+import org.junit.Assert.assertEquals
+import org.junit.Assert.assertFalse
+import org.junit.Assert.assertTrue
+import org.junit.Test
+
+class LightFallbackPolicyTest {
+
+    private val minute = 60_000L
+    private val policy = LightFallbackPolicy(initialBackoffMillis = minute, maxBackoffMillis = 4 * minute)
+
+    @Test
+    fun `light client takes over only after every RPC node failed in a row`() {
+        assertFalse(policy.onRpcFailure(rpcNodeCount = 3, nowMillis = 0))
+        assertFalse(policy.onRpcFailure(rpcNodeCount = 3, nowMillis = 0))
+        assertEquals(Source.RPC, policy.source)
+
+        assertTrue(policy.onRpcFailure(rpcNodeCount = 3, nowMillis = 0))
+        assertEquals(Source.LIGHT, policy.source)
+    }
+
+    @Test
+    fun `a connected RPC node resets the count`() {
+        policy.onRpcFailure(rpcNodeCount = 2, nowMillis = 0)
+        policy.onRpcConnected()
+
+        assertFalse(policy.onRpcFailure(rpcNodeCount = 2, nowMillis = 0))
+        assertTrue(policy.onRpcFailure(rpcNodeCount = 2, nowMillis = 0))
+    }
+
+    @Test
+    fun `a single RPC node hands over on its first failure`() {
+        assertTrue(policy.onRpcFailure(rpcNodeCount = 1, nowMillis = 0))
+    }
+
+    @Test
+    fun `RPC is retried after the backoff and one more failure returns to the light client`() {
+        policy.onRpcFailure(rpcNodeCount = 3, nowMillis = 0)
+        policy.onRpcFailure(rpcNodeCount = 3, nowMillis = 0)
+        policy.onRpcFailure(rpcNodeCount = 3, nowMillis = 0)
+
+        assertFalse(policy.onTick(rpcNodeCount = 3, nowMillis = minute - 1))
+        assertTrue(policy.onTick(rpcNodeCount = 3, nowMillis = minute))
+        assertEquals(Source.RPC, policy.source)
+
+        assertTrue(policy.onRpcFailure(rpcNodeCount = 3, nowMillis = minute))
+        assertEquals(Source.LIGHT, policy.source)
+    }
+
+    @Test
+    fun `the backoff doubles while RPC keeps failing, up to the maximum`() {
+        var now = 0L
+        policy.onRpcFailure(rpcNodeCount = 1, nowMillis = now)
+
+        for (expectedWait in listOf(minute, 2 * minute, 4 * minute, 4 * minute)) {
+            assertFalse(policy.onTick(rpcNodeCount = 1, nowMillis = now + expectedWait - 1))
+            now += expectedWait
+            assertTrue(policy.onTick(rpcNodeCount = 1, nowMillis = now))
+            assertTrue(policy.onRpcFailure(rpcNodeCount = 1, nowMillis = now))
+        }
+    }
+
+    @Test
+    fun `a connected RPC retry resets the backoff`() {
+        policy.onRpcFailure(rpcNodeCount = 1, nowMillis = 0)
+        policy.onTick(rpcNodeCount = 1, nowMillis = minute)
+        policy.onRpcFailure(rpcNodeCount = 1, nowMillis = minute)
+        policy.onTick(rpcNodeCount = 1, nowMillis = 3 * minute)
+        policy.onRpcConnected()
+
+        assertTrue(policy.onRpcFailure(rpcNodeCount = 1, nowMillis = 10 * minute))
+        assertFalse(policy.onTick(rpcNodeCount = 1, nowMillis = 11 * minute - 1))
+        assertTrue(policy.onTick(rpcNodeCount = 1, nowMillis = 11 * minute))
+    }
+
+    @Test
+    fun `an unavailable light client leaves the connection on RPC`() {
+        policy.onRpcFailure(rpcNodeCount = 2, nowMillis = 0)
+        policy.onRpcFailure(rpcNodeCount = 2, nowMillis = 0)
+        policy.onLightUnavailable(rpcNodeCount = 2)
+
+        assertEquals(Source.RPC, policy.source)
+        assertFalse(policy.onTick(rpcNodeCount = 2, nowMillis = 10 * minute))
+    }
+
+    @Test
+    fun `no failure is counted while on the light client`() {
+        policy.onRpcFailure(rpcNodeCount = 1, nowMillis = 0)
+        assertFalse(policy.onRpcFailure(rpcNodeCount = 1, nowMillis = 0))
+        assertEquals(Source.LIGHT, policy.source)
+    }
+}