Skip to content

fix(tracing): run the Stop hook in the background with a single-flight lock - #130

Open
CNSeniorious000 wants to merge 4 commits into
langfuse:mainfrom
CNSeniorious000:async-stop-hook-single-flight
Open

CNSeniorious000 wants to merge 4 commits into
langfuse:mainfrom
CNSeniorious000:async-stop-hook-single-flight

Conversation

@CNSeniorious000

@CNSeniorious000 CNSeniorious000 commented Nov 19, 2026 •

Copy link
Copy Markdown

Problem

The Stop hook is a synchronous command hook, so every turn ends with the Codex UI sitting on "Uploading Codex trace to Langfuse" until the export finishes. On a session with a backlog this takes minutes — one real run posted 611 media-upload requests for a 31-turn backlog — and the 30 s hook timeout kills the process before the flush completes. The sidecar then holds only the first few turns, so the next Stop picks up a slightly smaller backlog and the session never catches up.

Changes

1. Run the hook in the background ("async": true, timeout: 300).

  • The turn no longer waits on uploads; Codex continues while the export runs (hooks docs).
  • timeout: 300 is a third of the 600 s default. Background hooks are cancelled at timeout and at session end, which is why the flush-then-mark order below has to stay.

2. Single-flight lock per rollout (<rollout>.langfuse.lock, published atomically as a temp file plus a hard link, owner pid inside).

  • Background hooks overlap across fast turns — Codex runs up to eight concurrently per session — and two hook processes over the same rollout would export the same unmarked turns twice (the sidecar read-then-mark is not atomic), doubling exactly the requests this change is about.
  • A run that finds the lock held waits for it and then reads the transcript fresh: the running export's snapshot may predate the new turn, so only a later run can deliver it. The hook's own timeout bounds the wait.
  • Codex kills background hooks at timeout and session end, which leaves stale locks behind: the next run steals the lock when the owner pid is gone.

Unchanged semantics: onTurnExported still flushes before markTurnUploaded, so a cancelled run loses at most the turn it was exporting. Background hooks deliver output at the next safe point and can't block the turn — the hook doesn't rely on hook output.

Follow-up: the request-count reduction itself depends on upstream media dedup (langfuse/langfuse-js#1098); this PR only stops the uploads from blocking the turn and keeps overlapping turns from being dropped.

…t lock

The Stop hook runs synchronously, so a backlog of turns keeps the Codex UI on "Uploading Codex trace to Langfuse" for minutes, and the 30s timeout kills the process before the flush finishes, leaving turns unmarked so the next Stop replays an ever-growing backlog.

Run the hook with async: true (timeout 300, below the 600s default). Background hooks can overlap across fast turns (Codex runs up to eight concurrently per session), and two runs over the same rollout would export the same unmarked turns twice, so guard runs with a <rollout>.langfuse.lock created with O_EXCL. Codex cancels background hooks at timeout and session end, which leaves stale locks behind; a lock whose owner pid is gone is stolen on the next run. The flush-then-mark order in onTurnExported is unchanged: a killed run still loses at most the turn it was exporting.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T07:02:12.181289Z c7e1427 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c53dc86b69

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/tracing/src/sidecar.ts Resolved
Comment thread plugins/tracing/src/index.ts Resolved
Two review findings on the lock introduced in the parent commit:

- A stale lock (owner killed by hook timeout or session end) was never
  removed before the retry, so the second O_EXCL create hit the same EEXIST
  and every later run bailed out until the file was deleted by hand.
- Bailing out while another run held the lock dropped turns that completed
  after the running export's transcript snapshot: convertRollout never
  reloads the transcript, so only a later run can deliver them. A queued
  hook now waits for the lock and then reads the transcript fresh.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 887cdb8757

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/tracing/src/sidecar.ts Resolved
Comment thread plugins/tracing/src/sidecar.ts Resolved
Comment thread plugins/tracing/src/sidecar.ts Resolved
Three review findings on the lock's file protocol:

- Publish the pid atomically: create the lock as a temp file plus a hard
  link so contenders never read a half-written lock and mistake a live
  owner for dead.
- Reclaim a stale lock only while it still holds the pid that was
  inspected, so one contender cannot unlink a replacement lock another
  contender just acquired (the compare-and-unlink gap has no atomic POSIX
  primitive; worst case is duplicate exports, folded server-side by the
  seeded span ids).
- Tolerate the lock vanishing while inspecting its owner (ENOENT) and
  retry instead of aborting the run, and release only a lock that still
  carries our own pid.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant