Repository navigation
Fix claim trie rebuild exhausting memory after unclean shutdown - #5
Open
bitcoinbrisbane wants to merge 1 commit into
Open
bitcoinbrisbane wants to merge 1 commit into
bitcoinbrisbane wants to merge 1 commit into
Conversation
When the claim trie is behind the block chain, rebuildMissingClaimTrieData replayed every block from genesis into a single UtxoViewpoint to recover the historical outputs spent by the missing blocks. On mainnet that grows to tens of GB and runs the node out of memory, even when the trie is only a few hundred blocks behind. Build the view for each missing block from its spend journal entry instead, which already records the outputs the block spent, and only replay the blocks above the claim trie height. Memory use is now constant and catching up 276 blocks takes seconds. The replay also exposed a second problem: the claim trie's repos are written independently, so an unclean shutdown can leave the node repo holding changes for blocks the block repo never recorded. Replaying those blocks applied their changes twice and failed the claim hash check. Drop node changes above the claim trie height before replaying. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
After an unclean shutdown, lbcd can fail to restart: RAM climbs steadily until the machine runs out of memory, and the node never gets back to syncing. This PR fixes the two problems behind that.
1. The claim trie rebuild replayed the whole chain in memory
When the claim trie is behind the block chain,
rebuildMissingClaimTrieDatareplayed every block from genesis into a singleUtxoViewpoint. The database UTXO set only reflects the tip, so this was the only way to recover the outputs spent by historical blocks. The view ends up holding the entire historical UTXO set. On a mainnet node with a claim trie only 277 blocks behind, it grew from 5 GB to 54 GB in about 10 minutes and ran out of memory around block 977k. The existingTODOnoted the problem and that spent inputs were needed to fix it.Each block's spend journal entry already records the outputs that block spent, and claim processing only needs the scripts of spent inputs. So the rebuild now:
claimTrie.Height()+1 .. tip, not everything from genesisdbFetchSpendJournalEntry, with a sanity check againstcountSpentOutputsMemory use is constant. Each replayed block is still checked against the claim trie hash in its header.
2. Stale node changes left by the crash were applied twice
With memory fixed, the replay hit
computed hash ... != header's ClaimTrieat the second block. The claim trie's pebble repos (blocks, node changes, temporal) are written independently withNoSync, so a crash can leave them inconsistent. On the affected node:Replaying a block whose changes were already persisted appended them a second time. This adds
ClaimTrie.DropChangesAboveHeight(), which goes throughnode.Managerandnode.Repo. It scans the node repo, drops changes above the committed height (using the same criteria asDropChanges), and evicts those names from the node cache. The rebuild calls it before replaying. Being behind the chain implies an unclean shutdown, so the full scan only runs on that path.Test plan
go build ./...,go vet, andgo test . ./blockchain/ ./claimtrie/...passTestDropChangesAboveinclaimtrie/node/noderepoKnown gap
If a crash leaves the claim trie height equal to the chain tip but with extra node changes above it, this path isn't taken. The next block would then fail the hash check and be marked invalid. Covering that needs a cheap way to detect an unclean shutdown in the claim trie, which is left for a follow-up.
🤖 Generated with Claude Code