Repository navigation
fix: make while read over a redirected file linear in its size - #378
ferdingler wants to merge 3 commits into
Conversation
…taking quadratic time
…est (strands-agents#338) A redirected file reaches the reader as one chunk, and read takes one byte per call. The unread part was a Vec<u8>, so each byte drained from its front moved every byte after it, making while read over a file quadratic. Keep it as Bytes, where split_to is O(1).
|
@strandly-the-agent Please review this PR |
There was a problem hiding this comment.
Approve. The fix is correct and the loop is linear now. One non-blocking suggestion inline, on the timing test.
- Speed (release, the command from #338): the 2.07 MB file takes 0.43 s here, against the 30 s timeout on
mainand 0.19 s in bash. An 8.3 MB file takes 1.72 s. - Behaviour: no difference from
mainin any probe. That covers 2,000 random scripts mixingread,head,catand others on one fd, binary input, and the timeout put-back path. - 🟡 Inline: the 20 s bound in the new test passes on
mainon a machine about 1.5 times faster than the one I used. - Pre-existing, filed: #388 (
>>in a loop is quadratic, so thewhile readloop from #338 still times out if its body appends to a file) and #389 (head -nconsumes the rest of a redirected file).
What I verified, one question, appendix
Verified at head 3cde1aa against base dbb6189, Debian 12 aarch64, Rust 1.100.0:
cargo fmt --checkandcargo clippy --workspace --all-targetsexit 0.cargo test --workspace --all-targets: 2734 passed, 0 failed, 0 ignored across 20 test binaries. The here-doc deadline test passed.- The new timing test in a debug build: 2.8–3.1 s here, and it fails on base at 28.6–29.5 s.
- The differential case's
.expectedmatches bash 5.2.15. It passes on base too, so it guards behaviour, not speed. src/os.rs:1200(*remainder = bytes.slice(n..)) cannot drop bytes:rx.poll_recvis only reached whenremainderis empty (early return at 1190-1194). A model test over random chunk and read sizes (3,000 iterations, 66 MB) passes on both builds.- Other inputs that arrive as one chunk got the same speed-up at 1 MB (here-doc,
exec <,echo "$x" |, one 1 MB line): from 11–12 s to 0.2–0.7 s. - Peak memory is lower, not higher:
wc -c <on a 64 MB file peaks at 71 MB against 136 MB on base. - None of the 8 other open PRs that touch these files adds a
ChannelReaderwith the oldVectype, so this merge does not break their builds.
Not built: wasm32, --features node, --features python. Only read.rs, os.rs and vfs_kernel.rs reference the changed field.
Question (non-blocking): all 10 places that build a ChannelReader (9 in src, 1 in tests/custom_kernel.rs) pass an empty buf. Since custom kernels have to change this line anyway, would a constructor that takes only rx be worth adding, so that the next change to buf does not break them again?
Appendix — non-blocking (3)
- ⚪
src/builtins/read.rs:112: no test reaches the put-back with unread bytes inbuf. Deleting the line leaves the whole suite green. It needs a multi-thread runtime, which no in-tree binding uses; in a scratch build with one, this PR's code kept every byte in 8 of 8 runs. Nothing to change here. - Pre-existing, filed #388 and #389 (see above).
- Pre-existing, not filed:
{ cat f; } > outturns 3 non-UTF-8 bytes into 7. This looks like the byte replacement already described in #382.
| #[test] | ||
| fn while_read_over_a_large_redirected_file_takes_linear_time() { | ||
| let (rt, local) = rt(); | ||
| rt.block_on(local.run_until(async { | ||
| let mut shell = Shell::builder().build().unwrap(); | ||
| // 20,000 lines of 77 bytes: about 1.5 MB, which took tens of seconds | ||
| // when each one-byte read shifted the rest of the file. | ||
| let line = format!("{}\n", "a".repeat(76)); | ||
| let data = line.repeat(20_000); | ||
| shell.write_file("/tmp/big", data.as_bytes()).await.unwrap(); | ||
| let started = std::time::Instant::now(); | ||
| let out = shell | ||
| .run("n=0; while read -r l; do n=$((n+1)); done < /tmp/big; echo $n") | ||
| .await; | ||
| let elapsed = started.elapsed(); | ||
| assert_eq!( | ||
| (out.status, out.stdout.as_str(), out.stderr.as_str()), | ||
| (0, "20000\n", "") | ||
| ); | ||
| assert!( | ||
| elapsed < std::time::Duration::from_secs(20), | ||
| "took {elapsed:?}" | ||
| ); | ||
| })); | ||
| } |
There was a problem hiding this comment.
The 20 s bound passes on main on a machine about 1.5 times faster than this one
This test is the only guard against #338 coming back, since the differential case passes on main too. On main it takes 28.6–29.5 s against the 20 s bound, so on a runner about 1.5 times faster it passes with the quadratic read still in place. A slow runner gets close from the other side: with 24 copies of the test running on 4 cores, the fixed code took 11–18 s.
Suggestion: time 2,500 and 20,000 lines in one run and assert long < short * 16 + 1 s, as tests/path_resolution_cost.rs:39-46 does. In a debug build this PR gives a ratio of 8.0 and passes; main gives 38.7 and fails.
Full replacement for the test
| #[test] | |
| fn while_read_over_a_large_redirected_file_takes_linear_time() { | |
| let (rt, local) = rt(); | |
| rt.block_on(local.run_until(async { | |
| let mut shell = Shell::builder().build().unwrap(); | |
| // 20,000 lines of 77 bytes: about 1.5 MB, which took tens of seconds | |
| // when each one-byte read shifted the rest of the file. | |
| let line = format!("{}\n", "a".repeat(76)); | |
| let data = line.repeat(20_000); | |
| shell.write_file("/tmp/big", data.as_bytes()).await.unwrap(); | |
| let started = std::time::Instant::now(); | |
| let out = shell | |
| .run("n=0; while read -r l; do n=$((n+1)); done < /tmp/big; echo $n") | |
| .await; | |
| let elapsed = started.elapsed(); | |
| assert_eq!( | |
| (out.status, out.stdout.as_str(), out.stderr.as_str()), | |
| (0, "20000\n", "") | |
| ); | |
| assert!( | |
| elapsed < std::time::Duration::from_secs(20), | |
| "took {elapsed:?}" | |
| ); | |
| })); | |
| } | |
| #[test] | |
| fn while_read_over_a_large_redirected_file_takes_linear_time() { | |
| // Eight times the lines should take about eight times as long; the quadratic read took about | |
| // forty. Comparing two sizes in one run keeps the bound independent of machine speed. | |
| async fn time_to_read(lines: usize) -> std::time::Duration { | |
| let mut shell = Shell::builder().build().unwrap(); | |
| let data = format!("{}\n", "a".repeat(76)).repeat(lines); | |
| shell.write_file("/tmp/big", data.as_bytes()).await.unwrap(); | |
| let started = std::time::Instant::now(); | |
| let out = shell | |
| .run("n=0; while read -r l; do n=$((n+1)); done < /tmp/big; echo $n") | |
| .await; | |
| let elapsed = started.elapsed(); | |
| assert_eq!( | |
| (out.status, out.stdout, out.stderr), | |
| (0, format!("{lines}\n"), String::new()) | |
| ); | |
| elapsed | |
| } | |
| let (rt, local) = rt(); | |
| rt.block_on(local.run_until(async { | |
| let short = time_to_read(2_500).await; | |
| let long = time_to_read(20_000).await; | |
| assert!( | |
| long < short * 16 + std::time::Duration::from_secs(1), | |
| "20000 lines took {long:?}, 2500 took {short:?}" | |
| ); | |
| })); | |
| } |
Reproduction details
Head 3cde1aa, base dbb6189 (main), debug builds, Debian 12 aarch64 (4 cores), Rust 1.100.0.
Current test. cargo test --test shell_integration while_read_over_a_large:
| Build | Result |
|---|---|
| head, idle | passes, 2.8–3.1 s (9 runs) |
| head, 12 copies at once | passes, 5.6–9.1 s |
| head, 24 copies at once | passes, 11.3–17.8 s |
| base source with this test | fails, took 28.56s to 29.45s (4 runs) |
Suggested test. I ran the same body as a separate test file on both builds, and the block above in place in tests/shell_integration.rs on head (passes in 3.5 s; cargo fmt --check reports nothing for the file).
| Build | 2,500 lines | 20,000 lines | Ratio | Bound | Result |
|---|---|---|---|---|---|
| head (3 runs) | 0.35 s | 2.81–2.83 s | 8.0 | 6.6 s | passes |
| base (2 runs) | 0.74 s | 28.7–28.9 s | 38.6–39.0 | 12.9 s | fails |
Limit. The current test does not catch a smaller slowdown: a scratch change that copies the unread bytes once per read call (once per line, not per byte) takes 3.8 s in it and still passes. I did not run that change against the suggested test; going by those times its ratio would be about 10, so it would pass there too.
There was a problem hiding this comment.
Good catch, thanks. I switched to your two-size comparison, so the bound no longer depends on machine speed. It passes here in about 4 s, and going by your numbers on main it fails with a ratio near 39. Fixed in b96afbe.
There was a problem hiding this comment.
Verified at b96afbe. On this PR the test passes in about 3.5 s (3 of 3 runs). On main's source with this test file it fails (2 of 2 runs): 20,000 lines took 28.7–28.9 s and 2,500 took 0.77 s, against a bound of 13.3 s.
…t depend on machine speed
|
@strandly-the-agent Please review this PR |
strandly-the-agent
left a comment
There was a problem hiding this comment.
Approve at b96afbe. The timing-test finding is fixed, and the API pass I skipped in the first review found nothing to change.
- Previous finding (20 s bound): fixed. The committed two-size test passes on this PR and fails on
main's source; numbers are in the thread. - Type change on
FdKind::ChannelReader.buf: fine as is. The alternative that keepsVec<u8>(a read cursor private toFdReader) is not linear, andBytesis already in this variant throughrx. - I withdraw my constructor question.
proc.alloc_fd(FdReader::from_receiver(rx).into_fd_kind())already builds a reader without naming the buffer type, andbuf: Default::default()compiles before and after this PR. - One consumer breaks: strands-agents/box,
crates/shell/src/host.rs:425(at c70fec5), hasbuf: Vec::new()and needs that one token changed at its next shell bump. - Optional, and only possible before merge: the squash commit keeps only the PR title, and release notes are built from titles, so the kernel break will show in neither.
fix(os)!: …would carry it. Your call; no title in this repo has used!so far.
What I verified, which passes ran, appendix
Passes. The change since 3cde1aa is my own suggestion applied, so I checked it directly. Two more passes ran because the first review fell short of the routing:
- API pass (advanced tier, at b96afbe):
FdKindis public throughpub mod os, so it was due on the first review and I skipped it there. - Reviewer pass (default tier, at b96afbe): the first review's reviewer pass ran on the fast tier, so I ran it again. It found nothing new.
- The adversarial pass from the first review (advanced tier, at 3cde1aa) still applies:
srcis identical at both commits.
Verified at head b96afbe against base dbb6189, Debian 12 aarch64, Rust 1.100.0:
-
The change since 3cde1aa is
tests/shell_integration.rsonly (14 insertions, 10 deletions) and is the suggested test unchanged. -
cargo fmt --checkandcargo clippy --workspace --all-targetsexit 0.cargo test --workspace --all-targets: 2734 passed, 0 failed, 0 ignored across 20 test binaries. CI is green on b96afbe for Rust, Python and Node. -
All 10 places that build a
ChannelReaderare converted. Onlypoll_read(src/os.rs:1190-1201) andread's put-back (src/builtins/read.rs:111-114) touchbuf; every other path moves the wholeFdKind. -
Cost of the break: a kernel outside the crate with
buf: Vec::new()fails with E0308, and rustc's own hint (Vec::new().into()) compiles. -
bytesis not re-exported (strands_shell::os::Bytesis private, E0603), so a kernel that writesBytes::new()needs its ownbytesdependency. That was already true forrx;Default::default()avoids it. -
A
bufthat already holds bytes is read before anything fromrx, the same at base and head (17 of 17 scratch cases pass on both). -
The private-cursor alternative, in a scratch build that keeps
Vec<u8>(debug, 2 runs, seconds):Lines of 77 bytes 10,000 20,000 40,000 80,000 this PR 1.26 2.52–2.55 5.03–5.07 10.07–10.18 private cursor 1.29–1.33 2.81–2.84 6.31–6.41 15.42–15.62 readhands fd 0 back once per line (src/builtins/read.rs:122), and each hand-back would have to move the unread tail. -
For the title: the repo squashes with the PR title and a blank message, and
.github/workflows/release.yml:595feeds only commit subjects to the release notes..github/scripts/group-release-notes.pyaccepts!and lists the commit under Fixes. I did not run the PR title check with!.
Not built locally: wasm32, --features node, --features python.
Appendix — non-blocking (2)
- ⚪ The PR description still describes the single-size test ("about 29 seconds … about 3 seconds"). The numbers are still about right.
- Cross-repo, not filed: the strands-agents/box line above.
| #[test] | ||
| fn while_read_over_a_large_redirected_file_takes_linear_time() { | ||
| let (rt, local) = rt(); | ||
| rt.block_on(local.run_until(async { | ||
| let mut shell = Shell::builder().build().unwrap(); | ||
| // 20,000 lines of 77 bytes: about 1.5 MB, which took tens of seconds | ||
| // when each one-byte read shifted the rest of the file. | ||
| let line = format!("{}\n", "a".repeat(76)); | ||
| let data = line.repeat(20_000); | ||
| shell.write_file("/tmp/big", data.as_bytes()).await.unwrap(); | ||
| let started = std::time::Instant::now(); | ||
| let out = shell | ||
| .run("n=0; while read -r l; do n=$((n+1)); done < /tmp/big; echo $n") | ||
| .await; | ||
| let elapsed = started.elapsed(); | ||
| assert_eq!( | ||
| (out.status, out.stdout.as_str(), out.stderr.as_str()), | ||
| (0, "20000\n", "") | ||
| ); | ||
| assert!( | ||
| elapsed < std::time::Duration::from_secs(20), | ||
| "took {elapsed:?}" | ||
| ); | ||
| })); | ||
| } |
There was a problem hiding this comment.
Verified at b96afbe. On this PR the test passes in about 3.5 s (3 of 3 runs). On main's source with this test file it fails (2 of 2 runs): 20,000 lines took 28.7–28.9 s and 2,500 took 0.77 s, against a bound of 13.3 s.
Description
while read …; done < filegot slower with the square of the file size. A 1 MB file took about 11 seconds and a 2 MB file hit the 30 second timeout, while bash reads it in a fraction of a second.readtakes one byte at a time, and every byte it took from a redirected file shifted the rest of the unread file along in memory.Now the unread bytes are kept in a form where taking from the front is cheap, so the loop runs in linear time. One thing to know: this changes the type of the unread-bytes field on the pipe reader that custom kernels build, so a custom kernel that creates one has to say
Bytes::new()instead ofVec::new(). The built-in kernels and the custom kernel test are updated. It touches the same corner ofreadas the in-flight whitespace fix for #343, but only the timeout path that puts bytes back, not the splitting.On main the new test over a 1.5 MB file took about 29 seconds; with the change it takes about 3 seconds in a debug build. In a release build, the 2 MB file from the issue went from timing out to about half a second. I also added a differential case that reads a few thousand lines, including empty ones, and checks that each read takes exactly one line and leaves the rest for the next command. It matches bash before and after. The full test suite passes, apart from the known flaky here-doc deadline test, which failed once and passed when I ran it alone.
Related Issues
Fixes #338
Documentation PR
None
Type of Change
Bug fix
Testing
How have you tested the change? Verify that the changes do not break functionality or introduce new warnings.
cargo test --workspace --all-targets,pytest tests/python,npm test)cargo fmtandcargo clippyChecklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.