Repository navigation
plugin: Add FORGET_CHANNEL to stuck HSM types - #754
Merged
Merged
Conversation
Type 34 (WIRE_HSMD_FORGET_CHANNEL) is sent synchronously by lightningd's delete_channel() once onchaind completes for a closed channel. In bgsync mode no signer is attached and the StagingHsmServer only answers the canned startup messages, so the request sits in the stager forever and blocks lightningd's main loop. Because the type was not sticky the node did not report itself as stuck, so update_state called getinfo on the blocked lightningd and the state poll stalled: no BlockSync self-preempt, no 60s stuck-grace kill and no stuck webhook to summon the signer. The session then idled until the 1h session wall. On EU bgsync this accounted for ~95% of the ~2,029 cap-hit sessions/day, with 1,936 "forgetting peer" events/day repeating on every visit to the same nodes (Loki, 2026-10-12). Marking the type stuck makes update_state skip the hanging getinfo, fires the stuck webhook so the signer gets summoned, and kills the session after ~60s instead of ~3,540s. The forget then commits for good once a user session attaches the signer.
cdecker
force-pushed
the
2026w42-stuck-forget-channel
branch
from
October 13, 2026 16:18
330d951 to
72ef9e1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nodes running in bgsync mode (no signer attached) deadlock when a closed channel's onchaind finishes and lightningd's
delete_channel()issues a synchronousWIRE_HSMD_FORGET_CHANNEL(msgtype 34). TheStagingHsmServeronly answers the canned startup messages, so the request sits in the stager forever and blocks lightningd's main loop. Because type 34 was not instuck_request_types(), the node never reported itself stuck:update_statecalledgetinfoon the blocked lightningd (which hangs, no timeout), stalling the 10s state poll just beforeblockheight == headheightwould have triggered the BlockSync self-preempt. No stuck webhook fired and the 60s stuck-grace kill never engaged, so the session idled until the 1h session wall.Prod numbers (EU bgsync, Loki 2026-10-12): ~95% of the ~2,029 cap-hit sessions/day were already-caught-up nodes stuck this way; 1,936 "onchaind complete, forgetting peer" events/day, repeating on every visit to the same nodes because the forget never commits before the session dies.
The fix mirrors 30325ff (type 18): add 34 to the sticky list, plus a unit test asserting a queued type-34 request makes the stage report stuck (and that a non-sticky type such as 27/DERIVE_SECRET does not). With the type sticky,
update_stateskips the hanginggetinfo, the stuck webhook (NotifyStuck) summons the signer, and the session is killed after ~60s instead of ~3,540s; the forget then commits for good in the next user session with an attached signer.Known residual race (not addressed here): a
getinfoalready in flight when the type-34 request arrives still hangs the state poll; an RPC timeout watchdog inupdate_statewould be a possible follow-up.cargo test -p gl-plugin: 34 passed, 0 failed.🤖 Generated with Claude Code