Skip to content

multinode: opt-in fallback for FinalizedBlockOutOfSync liveness loss - #127

Draft
Krish-vemula wants to merge 1 commit into
mainfrom
fix/finalized-block-out-of-sync-liveness
Draft

Krish-vemula wants to merge 1 commit into
mainfrom
fix/finalized-block-out-of-sync-liveness

Conversation

@Krish-vemula

Copy link
Copy Markdown
Contributor

Problem

All tnet pods share the same RPC set, and HighestHead sends all of them to the same RPC. On chains that finalize in
batches (OP stack, Chiado, etc.), the backup RPCs are often one batch behind and get marked FinalizedBlockOutOfSync.
If the active RPC fails during that window, no RPC is routable and SelectRPC returns ErrNodeError.
FinalizedBlockOutOfSync is the only health check without a last-node-standing rule (#126 added one to the others).

Change

Adds an opt-in optional config interface, FinalizedBlockOutOfSyncFallbackConfig (off by default; only applies when
EnforceRepeatableRead is on). When enabled, a node's finalized block is compared against:

reference = min(highest finalized block observed by users, best finalized block among healthy peers)
  • A healthy peer is at the high mark: same as today.
  • No healthy peer is at the high mark: the freshest lagging node(s) stay routable, serving a delayed finalized view.
  • There are no healthy peers: the node stays routable (last node standing).

StateForMetrics still reports FinalizedBlockOutOfSync. A Warn is logged when a node starts serving a delayed
finalized view, and an Info when it stops.

Lock safety

State() now reads peer health, so each node keeps a lock-free atomic.Bool health snapshot, refreshed under stateMu
at every state or health change (setStateLocked, modifyHealthCheckFailure, modifySyncIssue). Peers never take each
other's stateMu, which rules out recursion and RWMutex deadlock cycles. A deadlock stress test and an AST test that
forbids direct n.state writes guard this.

Compatibility

  • The exported interfaces (Node, PoolChainInfoProvider) are unchanged and no mocks change. The new pool capability is an unexported optional interface.
  • Fails closed: if the pool or any primary node can't report health (for example, a wrapper around a Node), the whole pool stays strict and one Warn is logged.
  • With the flag off, behaviour matches main. Existing tests are untouched, and TestFinalizedBlockOutOfSync_FallbackDisabled covers it.

Trade-off

With the fallback on, a finalized read can return a lower block than one already served, in exchange for liveness.
Detecting stuck RPCs is still the job of NoNewFinalizedHeadsThreshold, NoNewHeadsThreshold and SyncThreshold.

Follow-ups

  • chainlink-evm: a NodePool TOML field (default false) plus docs, then enable it for tnets.
  • Spread pods across RPCs: random tie-break in HighestHead and random pick in PriorityLevel.

@github-actions

Copy link
Copy Markdown
Contributor

✅ API Diff Results - github.com/smartcontractkit/chainlink-framework/multinode

✅ Compatible Changes (4)

./ (1)
  • FinalizedBlockOutOfSyncFallbackConfig — ➕ Added
(*MultiNode[CHAIN_ID, RPC]) (3)
  • HealthyPeersChainInfo — ➕ Added

  • HealthyPeersChainInfo — ➕ Added

  • HealthyPeersChainInfo — ➕ Added


📄 View full apidiff report

@Krish-vemula
Krish-vemula marked this pull request as ready for review November 16, 2026 15:00
@Krish-vemula
Krish-vemula requested a review from a team as a code owner November 16, 2026 15:00
@Krish-vemula
Krish-vemula marked this pull request as draft November 16, 2026 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant