Repository navigation
multinode: opt-in fallback for FinalizedBlockOutOfSync liveness loss - #127
Draft
Krish-vemula wants to merge 1 commit into
Draft
Krish-vemula wants to merge 1 commit into
Krish-vemula wants to merge 1 commit into
Conversation
Contributor
✅ API Diff Results -
|
Krish-vemula
marked this pull request as ready for review
November 16, 2026 15:00
Krish-vemula
marked this pull request as draft
November 16, 2026 18:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
All tnet pods share the same RPC set, and HighestHead sends all of them to the same RPC. On chains that finalize in
batches (OP stack, Chiado, etc.), the backup RPCs are often one batch behind and get marked
FinalizedBlockOutOfSync.If the active RPC fails during that window, no RPC is routable and
SelectRPCreturnsErrNodeError.FinalizedBlockOutOfSync is the only health check without a last-node-standing rule (#126 added one to the others).
Change
Adds an opt-in optional config interface,
FinalizedBlockOutOfSyncFallbackConfig(off by default; only applies whenEnforceRepeatableReadis on). When enabled, a node's finalized block is compared against:StateForMetricsstill reportsFinalizedBlockOutOfSync. A Warn is logged when a node starts serving a delayedfinalized view, and an Info when it stops.
Lock safety
State()now reads peer health, so each node keeps a lock-freeatomic.Boolhealth snapshot, refreshed understateMuat every state or health change (
setStateLocked,modifyHealthCheckFailure,modifySyncIssue). Peers never take eachother's
stateMu, which rules out recursion and RWMutex deadlock cycles. A deadlock stress test and an AST test thatforbids direct
n.statewrites guard this.Compatibility
Node,PoolChainInfoProvider) are unchanged and no mocks change. The new pool capability is an unexported optional interface.Node), the whole pool stays strict and one Warn is logged.main. Existing tests are untouched, andTestFinalizedBlockOutOfSync_FallbackDisabledcovers it.Trade-off
With the fallback on, a
finalizedread can return a lower block than one already served, in exchange for liveness.Detecting stuck RPCs is still the job of
NoNewFinalizedHeadsThreshold,NoNewHeadsThresholdandSyncThreshold.Follow-ups
NodePoolTOML field (default false) plus docs, then enable it for tnets.