Repository navigation
fix: keep a function's output inside $( ), redirects and pipes when a group is piped into it - #384
ferdingler wants to merge 4 commits into
Conversation
|
@strandly-the-agent Please review this PR |
…tack Box the captured tail so the redirect form fits a 1 MiB stack, and check the substitution form in process: any function call inside $( ) already overflows the Windows debug CLI's main thread stack on main.
|
Pushed f6a79a4 for the Windows failure. Two of the new CLI tests ran out of stack on the 1 MiB Windows main thread. That is pre-existing: on main, |
strandly-the-agent
left a comment
There was a problem hiding this comment.
Approve at f6a79a4. The fix does what it says: the three #383 scripts and the subshell form match bash 5.2.15 in the CLI and through Shell::run, and I found no CLI regression. Two non-blocking findings, both only for callers that install a stdout writer:
| Finding | Where | |
|---|---|---|
| 🟡 | With a stdout writer, the captured tail now arrives after a later nested group (a, n on main, n, a here). Same hole as #390, which is already on main; a 4-line change fixes both |
src/exec.rs:2198 |
| 🟡 | No test covers the lines that set stdout aside | tests/shell_integration.rs:3186 |
The Windows follow-up checks out. The in-process test still fails on main ([][]), and boxing the tail lowers stack use below main's in every script I measured. On main, g() { echo a; }; x=$( g ) does need more than 1 MiB in a debug build (measured on Linux, not Windows).
What I verified, side effects, appendix
What I ran
Head f6a79a4, base dbb6189, Linux aarch64 debug builds, bash 5.2.15 as the reference.
cargo test --workspace --all-targetson head: 2736 passed, 0 failed, 0 ignored.cargo fmt --checkclean;cargo clippy --workspace --all-targetsexits 0 with no warning on a changed line. CI is green on f6a79a4, including Rust (windows-latest).- Head's tests on base's source: the three
compound_pipe_into_function_*tests fail, and the differential case fails on the redirect line (file:[a]thenb). The.expectedfile matches what bash prints. - CLI, bash vs base vs head: 39 hand-written scripts (stderr and redirects on the tail,
exit/return,read, nested functions,!, pipefail,set -e, backticks, nesting, ordering around the construct, a 300-iteration loop, 300 KB through the function). 36 match bash on head; the other 3 are #370 twice andwcpadding. An adversarial pass ran 760 generated CLI scripts with 0 regressions. - Library: 26 scripts through
runandexecutewith four writer setups (208 runs), 0 regressions; 800 generated runs found the writer finding below and nothing else. Output limit, deadline and kernel command accounting behave like the non-compound form (echo a | g).
Smallest ulimit -s (KiB, ±16) at which the debug CLI finishes, Linux aarch64:
| Script | main | c4d0185 | f6a79a4 |
|---|---|---|---|
g() { echo a; }; x=$( g ) |
1238 | 1238 | 1191 |
g() { cat; }; x=$( { echo a; } | g ); echo "[$x]" |
1159 | 1159 | 1111 |
g() { cat; }; { { echo a; } | g; } > f; echo "file:$(cat f)" |
1048 | 1064 | 1000 |
| the pipe test's script | 794 | 810 | 730 |
Frame sizes differ on Windows, so read these as direction only.
Not tested: Windows and macOS locally, release builds, WASM, the Python and Node.js bindings, the MCP server, the REPL.
Side effects of the fix (not findings)
On main the tail bypassed the capture, which happened to be right when only stderr was redirected. It now behaves like the non-compound form already does on main:
x=$( { echo a; } | g ):g's stderr is dropped, like all stderr inside$( )(#370). On main it reached the terminal.{ { …; } | g; } 2>/dev/nullin the CLI: the tail is now under the 1 MiB output limit, output of a background job started ingis lost, and non-UTF-8 bytes are replaced.{ echo a | g; } 2>/dev/nulldoes the same on main.- A call cancelled while the tail runs leaves
proc.captureon. Other capture sites already do this on main; filed #391.
Still open, same on main
- Output of an
if/whilecondition skips the capture (#371), sox=$( if { echo a; } | g; then :; fi )still leavesxempty. eval,commandorsourceas a later pipeline stage is "command not found", with or without a group on the left.- Pre-existing, filed #390 and #391.
Reading order
src/exec.rs:2193-2204, then capture_pipeline_checked (1918-1933) which it mirrors, then the function-call path in run_pipeline (4069-4128), then the tests.
Appendix, non-blocking (2)
- ⚪
tests/shell_integration.rs:3155-3156: "with or without this fix" is temporal context (AGENTS.md:115). "…of the CLI." is enough. - ⚪
src/exec.rs:2204: the leftover append has one reachable effect I could find. UnderShell::run,echo bg & { { echo a; } | wait; echo in; } > f; echo "f:$(cat f)"givesf:bg,in; main and bash givebg,f:in. Piping intowaitis contrived, so I'd leave it.
| let mut saved_stdout = Process::empty(); | ||
| proc.transfer_fd(STDOUT, &mut saved_stdout); |
There was a problem hiding this comment.
🟡 With a stdout writer installed, the captured tail now arrives after a later nested group
A Rust caller that installs a stdout writer (shell.proc.set_channel_writer(STDOUT, tx), as the expect_writers! tests do) and runs { { echo a; } | cat; { echo n; } 2>/dev/null; } 2>/dev/null now receives n, a. On main it receives a, n, as bash prints them.
The tail is now buffered with the rest of the outer group, but the inner { echo n; } 2>/dev/null still writes straight to the writer: its arm (lines 2280-2284) turns proc.capture on and leaves fd 1 in place. That hole is already on main (#390), so this is one more script shape reaching it. The CLI and calls without a writer are not affected.
Suggestion: set stdout aside in that arm too. If you'd rather keep this PR to one arm, #390 can carry it.
let saved_capture = std::mem::replace(&mut proc.capture, true);
let saved_output = std::mem::take(&mut proc.captured_output);
+ let mut saved_stdout = Process::empty();
+ proc.transfer_fd(STDOUT, &mut saved_stdout);
let result = Box::pin(execute_item(os.clone(), proc, &cr)).await;
proc.capture = saved_capture;
+ proc.close(STDOUT);
+ saved_stdout.transfer_fd(STDOUT, proc);
let out = std::mem::replace(&mut proc.captured_output, saved_output);Reproduction details
Head f6a79a4, base dbb6189, Linux aarch64, debug build. A test file in tests/ that installs a writer the way run_with_writers does, runs each script with Shell::run, and prints what the writer received:
let mut shell = Shell::builder().build().unwrap();
let (tx, mut rx) = strands_shell::os::pipe(1024);
shell.proc.set_channel_writer(strands_shell::os::STDOUT, tx);
shell.run(script).await;
// then drain rx with try_recv| Script | bash | main | f6a79a4 | f6a79a4 + diff above |
|---|---|---|---|---|
{ { echo a; } | cat; { echo n; } 2>/dev/null; } 2>/dev/null |
a n |
a n |
n a |
a n |
g() { cat; }; { { echo a; } | g; { echo n; } 2>/dev/null; } 2>/dev/null |
a n |
a n |
n a |
a n |
{ echo A; { echo B; } 2>&1; echo C; } 2>/dev/null (#390) |
A B C |
B A C |
B A C |
A B C |
g() { cat; }; { echo m1; { echo a; } | g; echo m2; } 2>/dev/null |
m1 a m2 |
a m1 m2 |
m1 a m2 |
m1 a m2 |
The last row is the writer case this PR fixes. With the diff applied to f6a79a4, cargo test --workspace --all-targets gives 2736 passed, 0 failed. I did not run the diff on main alone.
There was a problem hiding this comment.
| "g() { cat; }; { echo a; } | g | cat; for i in 1 2; do { echo $i; } | g; done | tr 12 xy", | ||
| ); | ||
| assert_eq!(out, "a\nx\ny\n"); | ||
| } |
There was a problem hiding this comment.
🟡 No test covers the lines that set stdout aside
Deleting the four saved_stdout lines in src/exec.rs (2197-2198 and 2202-2203) leaves these three tests and the differential case green. Those lines only matter when the caller installed a stdout writer: without them the function's output goes straight to the writer, ahead of the group's earlier output.
Suggestion: add a case next to the other expect_writers! tests. This one fails on main and with those lines deleted (the writer gets a, m1, m2), and passes on f6a79a4:
expect_writers!(
writer_receives_function_piped_from_group_in_order,
"g() { cat; }; { echo m1; { echo a; } | g; echo m2; } 2>/dev/null",
"m1\na\nm2\n",
|stderr: &str| assert_eq!(stderr, "")
);Reproduction details
Linux aarch64, debug build. I appended the test above to tests/shell_integration.rs and ran cargo test --test shell_integration writer_receives_function_piped_from_group_in_order on three trees:
| Tree | Result |
|---|---|
| f6a79a4 | ok |
f6a79a4 without the four saved_stdout lines |
FAILED, left "a\nm1\nm2\n", right "m1\na\nm2\n" |
| dbb6189 source with f6a79a4's tests | FAILED, same values |
On the tree without those four lines, cargo test --test shell_integration compound_pipe_into_function gives 3 passed and cargo test --test differential passes.
There was a problem hiding this comment.
Added your writer test in dec1ca7. It passes with the fix and fails with the four stdout-aside lines removed: the writer gets a, m1, m2. I also dropped "with or without this fix" from the Windows comment.
There was a problem hiding this comment.
Verified on dec1ca7: the test passes, and fails on main and with the four saved_stdout lines removed (a, m1, m2).
|
@strandly-the-agent Please review this PR |
strandly-the-agent
left a comment
There was a problem hiding this comment.
Approve at dec1ca7. The delta since f6a79a4 is test-only (src/ is unchanged), so the earlier review stands.
| Previous finding | Status |
|---|---|
| 🟡 No test covers the lines that set stdout aside | Fixed. writer_receives_function_piped_from_group_in_order passes here, and fails on main and with the four saved_stdout lines removed |
🟡 Writer ordering after a later nested group (n, a) |
Still open in this PR, deferred to #390 by the author. That is the option I offered, and #390 already carries the repro script |
| ⚪ "with or without this fix" comment | Fixed |
What I ran on dec1ca7
Linux aarch64, debug build.
cargo test --workspace --all-targets: 2737 passed, 0 failed, 0 ignored.cargo fmt --checkclean;cargo clippy --workspace --all-targetsexits 0 with no warning on a changed line. CI is green, including Rust (windows-latest).- dec1ca7 without the four
saved_stdoutlines: the writer test fails (left"a\nm1\nm2\n", right"m1\na\nm2\n"); the threecompound_pipe_into_function_*tests pass. - dbb6189 source with dec1ca7's tests: all four fail.
Description
When a group or subshell was piped into a shell function, the function's output skipped whatever was wrapped around it. So
x=$( { echo a; } | g )leftxempty and printedato the terminal, and the same thing happened with a redirect or a pipe around it. This showed up mostly in the CLI. Through the library, the redirect form also went wrong: the function's output went to the terminal and the file got earlier output.Now that last pipeline stage gets its output captured the same way other captured pipelines already do, so it ends up in the substitution, file or pipe. The fix is about a dozen lines in the executor; the rest is tests.
I added tests for the substitution, redirect and pipe forms, plus a differential case recorded from bash 5.2.15. On main they all failed, and the differential case failed on the redirect line. With the fix they pass, and the full test suite passes. The substitution test runs in process rather than through the CLI. On Windows, the debug CLI already runs out of stack for any function call inside
$( ), on main too, so I left that for a separate issue.Related Issues
Fixes #383
Unblocks #359
Documentation PR
N/A
Type of Change
Bug fix
Testing
How have you tested the change? Verify that the changes do not break functionality or introduce new warnings.
cargo test --workspace --all-targets,pytest tests/python,npm test)cargo fmtandcargo clippyChecklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.