Skip to content

plugin: Add FORGET_CHANNEL to stuck HSM types - #754

Merged
cdecker merged 1 commit into
mainfrom
2026w42-stuck-forget-channel
Oct 14, 2026
Merged

cdecker merged 1 commit into
mainfrom
2026w42-stuck-forget-channel

Conversation

@cdecker

@cdecker cdecker commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

Nodes running in bgsync mode (no signer attached) deadlock when a closed channel's onchaind finishes and lightningd's delete_channel() issues a synchronous WIRE_HSMD_FORGET_CHANNEL (msgtype 34). The StagingHsmServer only answers the canned startup messages, so the request sits in the stager forever and blocks lightningd's main loop. Because type 34 was not in stuck_request_types(), the node never reported itself stuck: update_state called getinfo on the blocked lightningd (which hangs, no timeout), stalling the 10s state poll just before blockheight == headheight would have triggered the BlockSync self-preempt. No stuck webhook fired and the 60s stuck-grace kill never engaged, so the session idled until the 1h session wall.

Prod numbers (EU bgsync, Loki 2026-10-12): ~95% of the ~2,029 cap-hit sessions/day were already-caught-up nodes stuck this way; 1,936 "onchaind complete, forgetting peer" events/day, repeating on every visit to the same nodes because the forget never commits before the session dies.

The fix mirrors 30325ff (type 18): add 34 to the sticky list, plus a unit test asserting a queued type-34 request makes the stage report stuck (and that a non-sticky type such as 27/DERIVE_SECRET does not). With the type sticky, update_state skips the hanging getinfo, the stuck webhook (NotifyStuck) summons the signer, and the session is killed after ~60s instead of ~3,540s; the forget then commits for good in the next user session with an attached signer.

Known residual race (not addressed here): a getinfo already in flight when the type-34 request arrives still hangs the state poll; an RPC timeout watchdog in update_state would be a possible follow-up.

cargo test -p gl-plugin: 34 passed, 0 failed.

🤖 Generated with Claude Code

Type 34 (WIRE_HSMD_FORGET_CHANNEL) is sent synchronously by
lightningd's delete_channel() once onchaind completes for a closed
channel. In bgsync mode no signer is attached and the StagingHsmServer
only answers the canned startup messages, so the request sits in the
stager forever and blocks lightningd's main loop.

Because the type was not sticky the node did not report itself as
stuck, so update_state called getinfo on the blocked lightningd and
the state poll stalled: no BlockSync self-preempt, no 60s stuck-grace
kill and no stuck webhook to summon the signer. The session then
idled until the 1h session wall. On EU bgsync this accounted for
~95% of the ~2,029 cap-hit sessions/day, with 1,936 "forgetting
peer" events/day repeating on every visit to the same nodes (Loki,
2026-10-12).

Marking the type stuck makes update_state skip the hanging getinfo,
fires the stuck webhook so the signer gets summoned, and kills the
session after ~60s instead of ~3,540s. The forget then commits for
good once a user session attaches the signer.
@cdecker
cdecker force-pushed the 2026w42-stuck-forget-channel branch from 330d951 to 72ef9e1 Compare October 13, 2026 16:18
@cdecker
cdecker merged commit 1499d7d into main Oct 14, 2026
16 checks passed
@cdecker
cdecker deleted the 2026w42-stuck-forget-channel branch October 14, 2026 10:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant