Skip to content

fix: make while read over a redirected file linear in its size - #378

Open
ferdingler wants to merge 3 commits into
strands-agents:mainfrom
ferdingler:fix/338-read-quadratic
Open

ferdingler wants to merge 3 commits into
strands-agents:mainfrom
ferdingler:fix/338-read-quadratic

Conversation

@ferdingler

Copy link
Copy Markdown
Collaborator

Description

while read …; done < file got slower with the square of the file size. A 1 MB file took about 11 seconds and a 2 MB file hit the 30 second timeout, while bash reads it in a fraction of a second. read takes one byte at a time, and every byte it took from a redirected file shifted the rest of the unread file along in memory.

Now the unread bytes are kept in a form where taking from the front is cheap, so the loop runs in linear time. One thing to know: this changes the type of the unread-bytes field on the pipe reader that custom kernels build, so a custom kernel that creates one has to say Bytes::new() instead of Vec::new(). The built-in kernels and the custom kernel test are updated. It touches the same corner of read as the in-flight whitespace fix for #343, but only the timeout path that puts bytes back, not the splitting.

On main the new test over a 1.5 MB file took about 29 seconds; with the change it takes about 3 seconds in a debug build. In a release build, the 2 MB file from the issue went from timing out to about half a second. I also added a differential case that reads a few thousand lines, including empty ones, and checks that each read takes exactly one line and leaves the rest for the next command. It matches bash before and after. The full test suite passes, apart from the known flaky here-doc deadline test, which failed once and passed when I ran it alone.

Related Issues

Fixes #338

Documentation PR

None

Type of Change

Bug fix

Testing

How have you tested the change? Verify that the changes do not break functionality or introduce new warnings.

  • I ran the relevant test suites for the bindings I touched (cargo test --workspace --all-targets, pytest tests/python, npm test)
  • If I touched Rust, I ran cargo fmt and cargo clippy

Checklist

  • I have read the CONTRIBUTING document
  • I have reviewed and understand every line of code in this PR, including any generated by AI tools, and I can explain why it works
  • My change is focused and reasonably small; I have split unrelated work into separate PRs
  • I have added any necessary tests that prove my fix is effective or my feature works
  • I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…est (strands-agents#338)

A redirected file reaches the reader as one chunk, and read takes one
byte per call. The unread part was a Vec<u8>, so each byte drained from
its front moved every byte after it, making while read over a file
quadratic. Keep it as Bytes, where split_to is O(1).
@ferdingler

Copy link
Copy Markdown
Collaborator Author

@strandly-the-agent Please review this PR

@strandly-the-agent strandly-the-agent left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. The fix is correct and the loop is linear now. One non-blocking suggestion inline, on the timing test.

  • Speed (release, the command from #338): the 2.07 MB file takes 0.43 s here, against the 30 s timeout on main and 0.19 s in bash. An 8.3 MB file takes 1.72 s.
  • Behaviour: no difference from main in any probe. That covers 2,000 random scripts mixing read, head, cat and others on one fd, binary input, and the timeout put-back path.
  • 🟡 Inline: the 20 s bound in the new test passes on main on a machine about 1.5 times faster than the one I used.
  • Pre-existing, filed: #388 (>> in a loop is quadratic, so the while read loop from #338 still times out if its body appends to a file) and #389 (head -n consumes the rest of a redirected file).
What I verified, one question, appendix

Verified at head 3cde1aa against base dbb6189, Debian 12 aarch64, Rust 1.100.0:

  • cargo fmt --check and cargo clippy --workspace --all-targets exit 0. cargo test --workspace --all-targets: 2734 passed, 0 failed, 0 ignored across 20 test binaries. The here-doc deadline test passed.
  • The new timing test in a debug build: 2.8–3.1 s here, and it fails on base at 28.6–29.5 s.
  • The differential case's .expected matches bash 5.2.15. It passes on base too, so it guards behaviour, not speed.
  • src/os.rs:1200 (*remainder = bytes.slice(n..)) cannot drop bytes: rx.poll_recv is only reached when remainder is empty (early return at 1190-1194). A model test over random chunk and read sizes (3,000 iterations, 66 MB) passes on both builds.
  • Other inputs that arrive as one chunk got the same speed-up at 1 MB (here-doc, exec <, echo "$x" |, one 1 MB line): from 11–12 s to 0.2–0.7 s.
  • Peak memory is lower, not higher: wc -c < on a 64 MB file peaks at 71 MB against 136 MB on base.
  • None of the 8 other open PRs that touch these files adds a ChannelReader with the old Vec type, so this merge does not break their builds.

Not built: wasm32, --features node, --features python. Only read.rs, os.rs and vfs_kernel.rs reference the changed field.

Question (non-blocking): all 10 places that build a ChannelReader (9 in src, 1 in tests/custom_kernel.rs) pass an empty buf. Since custom kernels have to change this line anyway, would a constructor that takes only rx be worth adding, so that the next change to buf does not break them again?

Appendix — non-blocking (3)

  • ⚪ src/builtins/read.rs:112: no test reaches the put-back with unread bytes in buf. Deleting the line leaves the whole suite green. It needs a multi-thread runtime, which no in-tree binding uses; in a scratch build with one, this PR's code kept every byte in 8 of 8 runs. Nothing to change here.
  • Pre-existing, filed #388 and #389 (see above).
  • Pre-existing, not filed: { cat f; } > out turns 3 non-UTF-8 bytes into 7. This looks like the byte replacement already described in #382.

Comment on lines +1513 to +1537
#[test]
fn while_read_over_a_large_redirected_file_takes_linear_time() {
let (rt, local) = rt();
rt.block_on(local.run_until(async {
let mut shell = Shell::builder().build().unwrap();
// 20,000 lines of 77 bytes: about 1.5 MB, which took tens of seconds
// when each one-byte read shifted the rest of the file.
let line = format!("{}\n", "a".repeat(76));
let data = line.repeat(20_000);
shell.write_file("/tmp/big", data.as_bytes()).await.unwrap();
let started = std::time::Instant::now();
let out = shell
.run("n=0; while read -r l; do n=$((n+1)); done < /tmp/big; echo $n")
.await;
let elapsed = started.elapsed();
assert_eq!(
(out.status, out.stdout.as_str(), out.stderr.as_str()),
(0, "20000\n", "")
);
assert!(
elapsed < std::time::Duration::from_secs(20),
"took {elapsed:?}"
);
}));
}

@strandly-the-agent strandly-the-agent Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 20 s bound passes on main on a machine about 1.5 times faster than this one

This test is the only guard against #338 coming back, since the differential case passes on main too. On main it takes 28.6–29.5 s against the 20 s bound, so on a runner about 1.5 times faster it passes with the quadratic read still in place. A slow runner gets close from the other side: with 24 copies of the test running on 4 cores, the fixed code took 11–18 s.

Suggestion: time 2,500 and 20,000 lines in one run and assert long < short * 16 + 1 s, as tests/path_resolution_cost.rs:39-46 does. In a debug build this PR gives a ratio of 8.0 and passes; main gives 38.7 and fails.

Full replacement for the test
Suggested change
#[test]
fn while_read_over_a_large_redirected_file_takes_linear_time() {
let (rt, local) = rt();
rt.block_on(local.run_until(async {
let mut shell = Shell::builder().build().unwrap();
// 20,000 lines of 77 bytes: about 1.5 MB, which took tens of seconds
// when each one-byte read shifted the rest of the file.
let line = format!("{}\n", "a".repeat(76));
let data = line.repeat(20_000);
shell.write_file("/tmp/big", data.as_bytes()).await.unwrap();
let started = std::time::Instant::now();
let out = shell
.run("n=0; while read -r l; do n=$((n+1)); done < /tmp/big; echo $n")
.await;
let elapsed = started.elapsed();
assert_eq!(
(out.status, out.stdout.as_str(), out.stderr.as_str()),
(0, "20000\n", "")
);
assert!(
elapsed < std::time::Duration::from_secs(20),
"took {elapsed:?}"
);
}));
}
#[test]
fn while_read_over_a_large_redirected_file_takes_linear_time() {
// Eight times the lines should take about eight times as long; the quadratic read took about
// forty. Comparing two sizes in one run keeps the bound independent of machine speed.
async fn time_to_read(lines: usize) -> std::time::Duration {
let mut shell = Shell::builder().build().unwrap();
let data = format!("{}\n", "a".repeat(76)).repeat(lines);
shell.write_file("/tmp/big", data.as_bytes()).await.unwrap();
let started = std::time::Instant::now();
let out = shell
.run("n=0; while read -r l; do n=$((n+1)); done < /tmp/big; echo $n")
.await;
let elapsed = started.elapsed();
assert_eq!(
(out.status, out.stdout, out.stderr),
(0, format!("{lines}\n"), String::new())
);
elapsed
}
let (rt, local) = rt();
rt.block_on(local.run_until(async {
let short = time_to_read(2_500).await;
let long = time_to_read(20_000).await;
assert!(
long < short * 16 + std::time::Duration::from_secs(1),
"20000 lines took {long:?}, 2500 took {short:?}"
);
}));
}
Reproduction details

Head 3cde1aa, base dbb6189 (main), debug builds, Debian 12 aarch64 (4 cores), Rust 1.100.0.

Current test. cargo test --test shell_integration while_read_over_a_large:

Build Result
head, idle passes, 2.8–3.1 s (9 runs)
head, 12 copies at once passes, 5.6–9.1 s
head, 24 copies at once passes, 11.3–17.8 s
base source with this test fails, took 28.56s to 29.45s (4 runs)

Suggested test. I ran the same body as a separate test file on both builds, and the block above in place in tests/shell_integration.rs on head (passes in 3.5 s; cargo fmt --check reports nothing for the file).

Build 2,500 lines 20,000 lines Ratio Bound Result
head (3 runs) 0.35 s 2.81–2.83 s 8.0 6.6 s passes
base (2 runs) 0.74 s 28.7–28.9 s 38.6–39.0 12.9 s fails

Limit. The current test does not catch a smaller slowdown: a scratch change that copies the unread bytes once per read call (once per line, not per byte) takes 3.8 s in it and still passes. I did not run that change against the suggested test; going by those times its ratio would be about 10, so it would pass there too.

@ferdingler ferdingler Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks. I switched to your two-size comparison, so the bound no longer depends on machine speed. It passes here in about 4 s, and going by your numbers on main it fails with a ratio near 39. Fixed in b96afbe.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at b96afbe. On this PR the test passes in about 3.5 s (3 of 3 runs). On main's source with this test file it fails (2 of 2 runs): 20,000 lines took 28.7–28.9 s and 2,500 took 0.77 s, against a bound of 13.3 s.

@ferdingler

Copy link
Copy Markdown
Collaborator Author

@strandly-the-agent Please review this PR

@strandly-the-agent strandly-the-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at b96afbe. The timing-test finding is fixed, and the API pass I skipped in the first review found nothing to change.

  • Previous finding (20 s bound): fixed. The committed two-size test passes on this PR and fails on main's source; numbers are in the thread.
  • Type change on FdKind::ChannelReader.buf: fine as is. The alternative that keeps Vec<u8> (a read cursor private to FdReader) is not linear, and Bytes is already in this variant through rx.
  • I withdraw my constructor question. proc.alloc_fd(FdReader::from_receiver(rx).into_fd_kind()) already builds a reader without naming the buffer type, and buf: Default::default() compiles before and after this PR.
  • One consumer breaks: strands-agents/box, crates/shell/src/host.rs:425 (at c70fec5), has buf: Vec::new() and needs that one token changed at its next shell bump.
  • Optional, and only possible before merge: the squash commit keeps only the PR title, and release notes are built from titles, so the kernel break will show in neither. fix(os)!: … would carry it. Your call; no title in this repo has used ! so far.
What I verified, which passes ran, appendix

Passes. The change since 3cde1aa is my own suggestion applied, so I checked it directly. Two more passes ran because the first review fell short of the routing:

  • API pass (advanced tier, at b96afbe): FdKind is public through pub mod os, so it was due on the first review and I skipped it there.
  • Reviewer pass (default tier, at b96afbe): the first review's reviewer pass ran on the fast tier, so I ran it again. It found nothing new.
  • The adversarial pass from the first review (advanced tier, at 3cde1aa) still applies: src is identical at both commits.

Verified at head b96afbe against base dbb6189, Debian 12 aarch64, Rust 1.100.0:

  • The change since 3cde1aa is tests/shell_integration.rs only (14 insertions, 10 deletions) and is the suggested test unchanged.

  • cargo fmt --check and cargo clippy --workspace --all-targets exit 0. cargo test --workspace --all-targets: 2734 passed, 0 failed, 0 ignored across 20 test binaries. CI is green on b96afbe for Rust, Python and Node.

  • All 10 places that build a ChannelReader are converted. Only poll_read (src/os.rs:1190-1201) and read's put-back (src/builtins/read.rs:111-114) touch buf; every other path moves the whole FdKind.

  • Cost of the break: a kernel outside the crate with buf: Vec::new() fails with E0308, and rustc's own hint (Vec::new().into()) compiles.

  • bytes is not re-exported (strands_shell::os::Bytes is private, E0603), so a kernel that writes Bytes::new() needs its own bytes dependency. That was already true for rx; Default::default() avoids it.

  • A buf that already holds bytes is read before anything from rx, the same at base and head (17 of 17 scratch cases pass on both).

  • The private-cursor alternative, in a scratch build that keeps Vec<u8> (debug, 2 runs, seconds):

    Lines of 77 bytes 10,000 20,000 40,000 80,000
    this PR 1.26 2.52–2.55 5.03–5.07 10.07–10.18
    private cursor 1.29–1.33 2.81–2.84 6.31–6.41 15.42–15.62

    read hands fd 0 back once per line (src/builtins/read.rs:122), and each hand-back would have to move the unread tail.

  • For the title: the repo squashes with the PR title and a blank message, and .github/workflows/release.yml:595 feeds only commit subjects to the release notes. .github/scripts/group-release-notes.py accepts ! and lists the commit under Fixes. I did not run the PR title check with !.

Not built locally: wasm32, --features node, --features python.

Appendix — non-blocking (2)

  • ⚪ The PR description still describes the single-size test ("about 29 seconds … about 3 seconds"). The numbers are still about right.
  • Cross-repo, not filed: the strands-agents/box line above.

Comment on lines +1513 to +1537
#[test]
fn while_read_over_a_large_redirected_file_takes_linear_time() {
let (rt, local) = rt();
rt.block_on(local.run_until(async {
let mut shell = Shell::builder().build().unwrap();
// 20,000 lines of 77 bytes: about 1.5 MB, which took tens of seconds
// when each one-byte read shifted the rest of the file.
let line = format!("{}\n", "a".repeat(76));
let data = line.repeat(20_000);
shell.write_file("/tmp/big", data.as_bytes()).await.unwrap();
let started = std::time::Instant::now();
let out = shell
.run("n=0; while read -r l; do n=$((n+1)); done < /tmp/big; echo $n")
.await;
let elapsed = started.elapsed();
assert_eq!(
(out.status, out.stdout.as_str(), out.stderr.as_str()),
(0, "20000\n", "")
);
assert!(
elapsed < std::time::Duration::from_secs(20),
"took {elapsed:?}"
);
}));
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at b96afbe. On this PR the test passes in about 3.5 s (3 of 3 runs). On main's source with this test file it fails (2 of 2 runs): 20,000 lines took 28.7–28.9 s and 2,500 took 0.77 s, against a bound of 13.3 s.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] while read …; done < file takes time quadratic in the file size, so a 2 MB file hits the 30 s timeout

2 participants