Skip to content

fix(indexer-common): reconcile full RAV backlogs and escrow snapshots - #1259

Merged
tmigone merged 1 commit into
graphprotocol:mainfrom
chriswessels:chris/collector-reconciliation
Oct 20, 2026
Merged

tmigone merged 1 commit into
graphprotocol:mainfrom
chriswessels:chris/collector-reconciliation

Conversation

@chriswessels

@chriswessels chriswessels commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

A bounded reconciliation pass currently keeps selecting the same highest-value RAVs. If those rows stay pending, lower-value rows never get reconciled. Separately, Horizon escrow reads use Graph Node's default first page, so an omitted collected-token record is treated as zero and an omitted payer has no balance.

This change:

  • Paginates both Horizon escrow entities independently, with ordered ID cursors and one block hash for the complete snapshot. Partial GraphQL failures, missing/inconsistent metadata and non-advancing pages reject the read instead of returning incomplete balances or collected amounts.
  • Advances both collectors through a descending-value keyset sweep, using every composite primary-key field to break ties. The existing 1,000-row reconciliation limit remains; a full prefix no longer needs to settle before the tail is visited. Redemption eligibility continues to include all unredeemed terminal RAVs.
  • Awaits observed-redemption database writes before finality and eligibility reload. Transaction queries also pin _meta to the row snapshot: previously, floating metadata could advance the block used by later pages/chunks and supply a newer timestamp for finality decisions.

The scan cursor is process-local. Restart starts a new sweep; rows inserted or moved ahead of the cursor are revisited on the next sweep. Failed downstream reconciliation leaves durable rows pending for a later sweep. The redemption threshold, finality/reorg intervals, filter chunk sizes, contract validation and transaction batch sizes are unchanged. This does not make below-threshold dust eligible for redemption.

Validation: 44 focused tests pass on Node 22.23.1 with PostgreSQL 13. New cases reproduce the original failures before repair and cover multiple full batches, all composite-key ties, wrap/deletion, failed reads, numeric value movement, unrestricted eligibility, gated real database writes, complete escrow accounting and moving-head GraphQL pagination. All existing ordering/chunking assertions remain. The common package compiles; scoped ESLint, Prettier and git diff --check pass. The repository Node 20/22 matrix runs in CI; pre-existing provider-dependent skips do not establish live-chain coverage.

Related to #1248 and complementary to #1247's partial-collection settlement repair. The Horizon await repair overlaps #1247. A merge analysis found one textual overlap at that call: when combining them, retain #1247's settledRavKeys argument and await. This PR otherwise preserves main's settlement semantics; it does not replace #1247.

The initial fork CI run also exposed an existing test-fixture bug: GitHub supplies an unavailable RPC secret as an empty string, but loadTestYamlConfig() used ?? and preserved that invalid URL instead of its public default. The helper now treats an empty override like an absent one. Five regressions cover absent/empty/explicit/invalid/whitespace overrides; all 17 focused parser/configuration cases pass, and explicit invalid URLs still fail schema validation. This change weakens no schema validation or integration assertion and adds no skipped tests. The subsequent test workflow passed on both Node 20 and Node 22; lint/typecheck/format, CodeQL and all applicable PR image builds also passed.

The passing CI matrix retains the repository's pre-existing skips: 3 suites and 43 tests in indexer-common. All 163 executed common tests, 11 Agent tests and 82 CLI tests pass on each Node version.

@github-project-automation github-project-automation Bot moved this to 🗃️ Inbox in Indexer Oct 17, 2026

@tmigone tmigone left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, I think there is a potential edge case for TAPv1 where a RAV that's redeemed on chain but failed the agent db update could force batch reverts for a bit, but it's a non issue since TAPv1 is pretty much deprecated anyway.

@github-project-automation github-project-automation Bot moved this from 🗃️ Inbox to ✅ Approved in Indexer Oct 19, 2026
@tmigone

tmigone commented Oct 19, 2026

Copy link
Copy Markdown
Member

@chriswessels the repo requires signed commits could you do that?

@chriswessels
chriswessels force-pushed the chris/collector-reconciliation branch from 32b6880 to bb6e572 Compare October 19, 2026 20:27
Advance bounded reconciliation through pending TAP v1 and v2 RAVs with
stable keyset cursors, and await redeemed-state updates.

Paginate Horizon escrow balances and collected amounts at a consistent
block snapshot. Reject partial queries and inconsistent metadata across
escrow and transaction pages.

Cover backlog traversal and snapshot pagination, and default empty test
provider overrides to the public endpoint.
@chriswessels
chriswessels force-pushed the chris/collector-reconciliation branch from bb6e572 to beead28 Compare October 19, 2026 20:29
@chriswessels

Copy link
Copy Markdown
Contributor Author

@tmigone Done! Thank you

@tmigone
tmigone merged commit 8d8d806 into graphprotocol:main Oct 20, 2026
11 checks passed
@github-project-automation github-project-automation Bot moved this from ✅ Approved to 🚗 Merged in Indexer Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 🚗 Merged

Development

Successfully merging this pull request may close these issues.

2 participants