Skip to content

Reopen seiload's log follow and read the pod before draining it - #617

Merged
masih merged 2 commits into
mainfrom
masih/1792350368-seiload-follow-reconnect
Oct 18, 2026
Merged

masih merged 2 commits into
mainfrom
masih/1792350368-seiload-follow-reconnect

Conversation

@masih

@masih masih commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

This picks up the two seidroid suggestions on #616, which merged before they landed. First, assertSeiloadRun used to receive the follower's lines as an argument, so runSeiload drained the follower before the pod was read. If the stream had not ended yet, that drain could hold things up for as long as summaryStreamDrain (30s) while Karpenter is free to reclaim the finished pod. assertSeiloadRun now takes the *summaryFollower and calls lines() only once jobPod and the tail read have happened.

Second, a followed stream that ended before seiload exited, such as a reset connection partway through a saturation run, stopped the follower for good and put the check back on the tail read. followSummary now reopens the stream after logStreamRetry until seiloadExited sees every container terminated. A reopened stream re-reads the current log file, but the repeated lines do no harm because summaryCounts takes the first match.

@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access that start with 'Devin'.

  • Disable automatic comment, CI, and merge conflict monitoring

@masih
masih marked this pull request as ready for review October 18, 2026 19:06
@cursor

cursor Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

PR Summary

Low Risk
Changes only affect integration test harness timing and log following; no production runtime behavior.

Overview
Hardens the seiload integration test’s log capture so run verification stays reliable when kubelet rotates logs, streams drop, or Karpenter reclaims nodes.

followSummary now loops: after each log stream ends it reopens the follow (after logStreamRetry) until seiloadExited reports all containers terminated, with line collection moved into collect. Duplicate summary lines from re-reads are safe because summaryCounts uses the first regex match.

assertSeiloadRun takes a *summaryFollower instead of a pre-built string: it reads the pod and log tail first, then drains summary.lines(), avoiding up to summaryStreamDrain (30s) of blocking before the pod read while the finished workload’s node might be reclaimed.

Reviewed by Cursor Bugbot for commit 055bb33. Bugbot is set up for automated code reviews on this repo. Configure here.

@seidroid seidroid Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moves the follower drain in assertSeiloadRun so it runs after the pod and tail have been read, and reopens a followed seiload log stream until every container has terminated. Nothing blocks: the follower's state is only read after done closes, each loop path ends on exit, cancel or drain timeout, and repeated lines are harmless because summaryCounts takes the first match. The only reader output, codex, reported nothing, which matches my reading. No Go toolchain was available, so this was not built or vetted.

1 nit, not posted on the code
  • test/integration/seiload_test.go:442 — summary.lines() is now the only thing that cancels the follower, and assertSeiloadRun returns earlier than this when jobPod fails. On that path the follower is never cancelled. With the new reopen loop, a missing pod makes seiloadExited false and followPodLog retry, so the goroutine polls the API every 5s until the scenario ctx is cancelled, which can be several minutes later through assertChainLive. A defer summary.cancel() in runSeiload would close this.

seidroid review · decision approve · session 171db4d6cddf431ab32f8c73536e8213 · turn resp_claude_c0e732408bdae79b91232f4029b10b9a · item 0189a424ec86543dbe1334280e6ec9b8

Findings: 0 blocking | 0 non-blocking | 0 posted inline

@masih
masih merged commit 713a6c2 into main Oct 18, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant