Skip to content

Fix claim trie rebuild exhausting memory after unclean shutdown - #5

Open
bitcoinbrisbane wants to merge 1 commit into
LBRYFoundation:master-oldfrom
bitcoinbrisbane:claimtrie-rebuild-fix
Open

bitcoinbrisbane wants to merge 1 commit into
LBRYFoundation:master-oldfrom
bitcoinbrisbane:claimtrie-rebuild-fix

Conversation

@bitcoinbrisbane

Copy link
Copy Markdown

Summary

After an unclean shutdown, lbcd can fail to restart: RAM climbs steadily until the machine runs out of memory, and the node never gets back to syncing. This PR fixes the two problems behind that.

1. The claim trie rebuild replayed the whole chain in memory

When the claim trie is behind the block chain, rebuildMissingClaimTrieData replayed every block from genesis into a single UtxoViewpoint. The database UTXO set only reflects the tip, so this was the only way to recover the outputs spent by historical blocks. The view ends up holding the entire historical UTXO set. On a mainnet node with a claim trie only 277 blocks behind, it grew from 5 GB to 54 GB in about 10 minutes and ran out of memory around block 977k. The existing TODO noted the problem and that spent inputs were needed to fix it.

Each block's spend journal entry already records the outputs that block spent, and claim processing only needs the scripts of spent inputs. So the rebuild now:

  • replays only claimTrie.Height()+1 .. tip, not everything from genesis
  • builds a small per-block view from dbFetchSpendJournalEntry, with a sanity check against countSpentOutputs

Memory use is constant. Each replayed block is still checked against the claim trie hash in its header.

2. Stale node changes left by the crash were applied twice

With memory fixed, the replay hit computed hash ... != header's ClaimTrie at the second block. The claim trie's pebble repos (blocks, node changes, temporal) are written independently with NoSync, so a crash can leave them inconsistent. On the affected node:

repo height after crash
block repo (committed trie height) 1,119,213
block DB (ffldb) 1,119,489
node change repo changes up to 1,119,740 (103,925 changes across 45,175 names above the committed height)

Replaying a block whose changes were already persisted appended them a second time. This adds ClaimTrie.DropChangesAboveHeight(), which goes through node.Manager and node.Repo. It scans the node repo, drops changes above the committed height (using the same criteria as DropChanges), and evicts those names from the node cache. The rebuild calls it before replaying. Being behind the chain implies an unclean shutdown, so the full scan only runs on that path.

Test plan

  • go build ./..., go vet, and go test . ./blockchain/ ./claimtrie/... pass
  • New TestDropChangesAbove in claimtrie/node/noderepo
  • On the affected mainnet node (chain at 1,119,489, claim trie at 1,119,213 after an unclean shutdown):
    • the stale-change scan dropped changes from 45,175 names in 27 s
    • 276 blocks were rebuilt in 9.6 s, all matching their header claim trie hashes
    • the node then resumed normal sync; RAM has stayed flat at about 6 GB through 30k+ further blocks

Known gap

If a crash leaves the claim trie height equal to the chain tip but with extra node changes above it, this path isn't taken. The next block would then fail the hash check and be marked invalid. Covering that needs a cheap way to detect an unclean shutdown in the claim trie, which is left for a follow-up.

🤖 Generated with Claude Code

When the claim trie is behind the block chain, rebuildMissingClaimTrieData
replayed every block from genesis into a single UtxoViewpoint to recover
the historical outputs spent by the missing blocks. On mainnet that grows
to tens of GB and runs the node out of memory, even when the trie is only
a few hundred blocks behind.

Build the view for each missing block from its spend journal entry
instead, which already records the outputs the block spent, and only
replay the blocks above the claim trie height. Memory use is now constant
and catching up 276 blocks takes seconds.

The replay also exposed a second problem: the claim trie's repos are
written independently, so an unclean shutdown can leave the node repo
holding changes for blocks the block repo never recorded. Replaying those
blocks applied their changes twice and failed the claim hash check. Drop
node changes above the claim trie height before replaying.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant