Repository navigation
fix(tracing): run the Stop hook in the background with a single-flight lock - #130
CNSeniorious000 wants to merge 4 commits into
Conversation
…t lock The Stop hook runs synchronously, so a backlog of turns keeps the Codex UI on "Uploading Codex trace to Langfuse" for minutes, and the 30s timeout kills the process before the flush finishes, leaving turns unmarked so the next Stop replays an ever-growing backlog. Run the hook with async: true (timeout 300, below the 600s default). Background hooks can overlap across fast turns (Codex runs up to eight concurrently per session), and two runs over the same rollout would export the same unmarked turns twice, so guard runs with a <rollout>.langfuse.lock created with O_EXCL. Codex cancels background hooks at timeout and session end, which leaves stale locks behind; a lock whose owner pid is gone is stolen on the next run. The flush-then-mark order in onTurnExported is unchanged: a killed run still loses at most the turn it was exporting.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c53dc86b69
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Two review findings on the lock introduced in the parent commit: - A stale lock (owner killed by hook timeout or session end) was never removed before the retry, so the second O_EXCL create hit the same EEXIST and every later run bailed out until the file was deleted by hand. - Bailing out while another run held the lock dropped turns that completed after the running export's transcript snapshot: convertRollout never reloads the transcript, so only a later run can deliver them. A queued hook now waits for the lock and then reads the transcript fresh.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 887cdb8757
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Three review findings on the lock's file protocol: - Publish the pid atomically: create the lock as a temp file plus a hard link so contenders never read a half-written lock and mistake a live owner for dead. - Reclaim a stale lock only while it still holds the pid that was inspected, so one contender cannot unlink a replacement lock another contender just acquired (the compare-and-unlink gap has no atomic POSIX primitive; worst case is duplicate exports, folded server-side by the seeded span ids). - Tolerate the lock vanishing while inspecting its owner (ENOENT) and retry instead of aborting the run, and release only a lock that still carries our own pid.
Problem
The Stop hook is a synchronous command hook, so every turn ends with the Codex UI sitting on "Uploading Codex trace to Langfuse" until the export finishes. On a session with a backlog this takes minutes — one real run posted 611 media-upload requests for a 31-turn backlog — and the 30 s hook timeout kills the process before the flush completes. The sidecar then holds only the first few turns, so the next Stop picks up a slightly smaller backlog and the session never catches up.
Changes
1. Run the hook in the background (
"async": true,timeout: 300).timeout: 300is a third of the 600 s default. Background hooks are cancelled at timeout and at session end, which is why the flush-then-mark order below has to stay.2. Single-flight lock per rollout (
<rollout>.langfuse.lock, published atomically as a temp file plus a hard link, owner pid inside).timeoutbounds the wait.Unchanged semantics:
onTurnExportedstill flushes beforemarkTurnUploaded, so a cancelled run loses at most the turn it was exporting. Background hooks deliver output at the next safe point and can't block the turn — the hook doesn't rely on hook output.Follow-up: the request-count reduction itself depends on upstream media dedup (langfuse/langfuse-js#1098); this PR only stops the uploads from blocking the turn and keeps overlapping turns from being dropped.