Skip to content

fix: keep a function's output inside $( ), redirects and pipes when a group is piped into it - #384

Open
ferdingler wants to merge 4 commits into
strands-agents:mainfrom
ferdingler:fix/383-compound-pipe-into-function
Open

ferdingler wants to merge 4 commits into
strands-agents:mainfrom
ferdingler:fix/383-compound-pipe-into-function

Conversation

@ferdingler

@ferdingler ferdingler commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Description

When a group or subshell was piped into a shell function, the function's output skipped whatever was wrapped around it. So x=$( { echo a; } | g ) left x empty and printed a to the terminal, and the same thing happened with a redirect or a pipe around it. This showed up mostly in the CLI. Through the library, the redirect form also went wrong: the function's output went to the terminal and the file got earlier output.

Now that last pipeline stage gets its output captured the same way other captured pipelines already do, so it ends up in the substitution, file or pipe. The fix is about a dozen lines in the executor; the rest is tests.

I added tests for the substitution, redirect and pipe forms, plus a differential case recorded from bash 5.2.15. On main they all failed, and the differential case failed on the redirect line. With the fix they pass, and the full test suite passes. The substitution test runs in process rather than through the CLI. On Windows, the debug CLI already runs out of stack for any function call inside $( ), on main too, so I left that for a separate issue.

Related Issues

Fixes #383

Unblocks #359

Documentation PR

N/A

Type of Change

Bug fix

Testing

How have you tested the change? Verify that the changes do not break functionality or introduce new warnings.

  • I ran the relevant test suites for the bindings I touched (cargo test --workspace --all-targets, pytest tests/python, npm test)
  • If I touched Rust, I ran cargo fmt and cargo clippy

Checklist

  • I have read the CONTRIBUTING document
  • I have reviewed and understand every line of code in this PR, including any generated by AI tools, and I can explain why it works
  • My change is focused and reasonably small; I have split unrelated work into separate PRs
  • I have added any necessary tests that prove my fix is effective or my feature works
  • I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@ferdingler

Copy link
Copy Markdown
Collaborator Author

@strandly-the-agent Please review this PR

@github-actions github-actions Bot added the bug Something isn't working label Nov 21, 2026
…tack

Box the captured tail so the redirect form fits a 1 MiB stack, and check
the substitution form in process: any function call inside $( ) already
overflows the Windows debug CLI's main thread stack on main.
@ferdingler

Copy link
Copy Markdown
Collaborator Author

Pushed f6a79a4 for the Windows failure. Two of the new CLI tests ran out of stack on the 1 MiB Windows main thread. That is pre-existing: on main, g() { echo a; }; x=$( g ) already overflows a 1 MiB debug stack. I boxed the tail call, which fixes the redirect form, and moved the substitution test in process through Shell::execute. It still fails on main. @strandly-the-agent Please review this PR

@strandly-the-agent strandly-the-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at f6a79a4. The fix does what it says: the three #383 scripts and the subshell form match bash 5.2.15 in the CLI and through Shell::run, and I found no CLI regression. Two non-blocking findings, both only for callers that install a stdout writer:

Finding Where
🟡 With a stdout writer, the captured tail now arrives after a later nested group (a, n on main, n, a here). Same hole as #390, which is already on main; a 4-line change fixes both src/exec.rs:2198
🟡 No test covers the lines that set stdout aside tests/shell_integration.rs:3186

The Windows follow-up checks out. The in-process test still fails on main ([][]), and boxing the tail lowers stack use below main's in every script I measured. On main, g() { echo a; }; x=$( g ) does need more than 1 MiB in a debug build (measured on Linux, not Windows).

What I verified, side effects, appendix

What I ran

Head f6a79a4, base dbb6189, Linux aarch64 debug builds, bash 5.2.15 as the reference.

  • cargo test --workspace --all-targets on head: 2736 passed, 0 failed, 0 ignored. cargo fmt --check clean; cargo clippy --workspace --all-targets exits 0 with no warning on a changed line. CI is green on f6a79a4, including Rust (windows-latest).
  • Head's tests on base's source: the three compound_pipe_into_function_* tests fail, and the differential case fails on the redirect line (file:[a] then b). The .expected file matches what bash prints.
  • CLI, bash vs base vs head: 39 hand-written scripts (stderr and redirects on the tail, exit/return, read, nested functions, !, pipefail, set -e, backticks, nesting, ordering around the construct, a 300-iteration loop, 300 KB through the function). 36 match bash on head; the other 3 are #370 twice and wc padding. An adversarial pass ran 760 generated CLI scripts with 0 regressions.
  • Library: 26 scripts through run and execute with four writer setups (208 runs), 0 regressions; 800 generated runs found the writer finding below and nothing else. Output limit, deadline and kernel command accounting behave like the non-compound form (echo a | g).

Smallest ulimit -s (KiB, ±16) at which the debug CLI finishes, Linux aarch64:

Script main c4d0185 f6a79a4
g() { echo a; }; x=$( g ) 1238 1238 1191
g() { cat; }; x=$( { echo a; } | g ); echo "[$x]" 1159 1159 1111
g() { cat; }; { { echo a; } | g; } > f; echo "file:$(cat f)" 1048 1064 1000
the pipe test's script 794 810 730

Frame sizes differ on Windows, so read these as direction only.

Not tested: Windows and macOS locally, release builds, WASM, the Python and Node.js bindings, the MCP server, the REPL.

Side effects of the fix (not findings)

On main the tail bypassed the capture, which happened to be right when only stderr was redirected. It now behaves like the non-compound form already does on main:

  • x=$( { echo a; } | g ): g's stderr is dropped, like all stderr inside $( ) (#370). On main it reached the terminal.
  • { { …; } | g; } 2>/dev/null in the CLI: the tail is now under the 1 MiB output limit, output of a background job started in g is lost, and non-UTF-8 bytes are replaced. { echo a | g; } 2>/dev/null does the same on main.
  • A call cancelled while the tail runs leaves proc.capture on. Other capture sites already do this on main; filed #391.

Still open, same on main

  • Output of an if/while condition skips the capture (#371), so x=$( if { echo a; } | g; then :; fi ) still leaves x empty.
  • eval, command or source as a later pipeline stage is "command not found", with or without a group on the left.
  • Pre-existing, filed #390 and #391.

Reading order

src/exec.rs:2193-2204, then capture_pipeline_checked (1918-1933) which it mirrors, then the function-call path in run_pipeline (4069-4128), then the tests.

Appendix, non-blocking (2)

  • ⚪ tests/shell_integration.rs:3155-3156: "with or without this fix" is temporal context (AGENTS.md:115). "…of the CLI." is enough.
  • ⚪ src/exec.rs:2204: the leftover append has one reachable effect I could find. Under Shell::run, echo bg & { { echo a; } | wait; echo in; } > f; echo "f:$(cat f)" gives f:bg, in; main and bash give bg, f:in. Piping into wait is contrived, so I'd leave it.

Comment thread src/exec.rs
Comment on lines +2197 to +2198
let mut saved_stdout = Process::empty();
proc.transfer_fd(STDOUT, &mut saved_stdout);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 With a stdout writer installed, the captured tail now arrives after a later nested group

A Rust caller that installs a stdout writer (shell.proc.set_channel_writer(STDOUT, tx), as the expect_writers! tests do) and runs { { echo a; } | cat; { echo n; } 2>/dev/null; } 2>/dev/null now receives n, a. On main it receives a, n, as bash prints them.

The tail is now buffered with the rest of the outer group, but the inner { echo n; } 2>/dev/null still writes straight to the writer: its arm (lines 2280-2284) turns proc.capture on and leaves fd 1 in place. That hole is already on main (#390), so this is one more script shape reaching it. The CLI and calls without a writer are not affected.

Suggestion: set stdout aside in that arm too. If you'd rather keep this PR to one arm, #390 can carry it.

                         let saved_capture = std::mem::replace(&mut proc.capture, true);
                         let saved_output = std::mem::take(&mut proc.captured_output);
+                        let mut saved_stdout = Process::empty();
+                        proc.transfer_fd(STDOUT, &mut saved_stdout);
                         let result = Box::pin(execute_item(os.clone(), proc, &cr)).await;
                         proc.capture = saved_capture;
+                        proc.close(STDOUT);
+                        saved_stdout.transfer_fd(STDOUT, proc);
                         let out = std::mem::replace(&mut proc.captured_output, saved_output);
Reproduction details

Head f6a79a4, base dbb6189, Linux aarch64, debug build. A test file in tests/ that installs a writer the way run_with_writers does, runs each script with Shell::run, and prints what the writer received:

let mut shell = Shell::builder().build().unwrap();
let (tx, mut rx) = strands_shell::os::pipe(1024);
shell.proc.set_channel_writer(strands_shell::os::STDOUT, tx);
shell.run(script).await;
// then drain rx with try_recv
Script bash main f6a79a4 f6a79a4 + diff above
{ { echo a; } | cat; { echo n; } 2>/dev/null; } 2>/dev/null a n a n n a a n
g() { cat; }; { { echo a; } | g; { echo n; } 2>/dev/null; } 2>/dev/null a n a n n a a n
{ echo A; { echo B; } 2>&1; echo C; } 2>/dev/null (#390) A B C B A C B A C A B C
g() { cat; }; { echo m1; { echo a; } | g; echo m2; } 2>/dev/null m1 a m2 a m1 m2 m1 a m2 m1 a m2

The last row is the writer case this PR fixes. With the diff applied to f6a79a4, cargo test --workspace --all-targets gives 2736 passed, 0 failed. I did not run the diff on main alone.

@ferdingler ferdingler Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this one for #390. The fix goes in the redirected-group arm, not the arm this PR changes, and it is the fix for #390 itself, so I would rather not bundle it here. Once #390 is fixed, the n/a ordering here is fixed too. Your first table row would make a good test for that fix.

"g() { cat; }; { echo a; } | g | cat; for i in 1 2; do { echo $i; } | g; done | tr 12 xy",
);
assert_eq!(out, "a\nx\ny\n");
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 No test covers the lines that set stdout aside

Deleting the four saved_stdout lines in src/exec.rs (2197-2198 and 2202-2203) leaves these three tests and the differential case green. Those lines only matter when the caller installed a stdout writer: without them the function's output goes straight to the writer, ahead of the group's earlier output.

Suggestion: add a case next to the other expect_writers! tests. This one fails on main and with those lines deleted (the writer gets a, m1, m2), and passes on f6a79a4:

expect_writers!(
    writer_receives_function_piped_from_group_in_order,
    "g() { cat; }; { echo m1; { echo a; } | g; echo m2; } 2>/dev/null",
    "m1\na\nm2\n",
    |stderr: &str| assert_eq!(stderr, "")
);
Reproduction details

Linux aarch64, debug build. I appended the test above to tests/shell_integration.rs and ran cargo test --test shell_integration writer_receives_function_piped_from_group_in_order on three trees:

Tree Result
f6a79a4 ok
f6a79a4 without the four saved_stdout lines FAILED, left "a\nm1\nm2\n", right "m1\na\nm2\n"
dbb6189 source with f6a79a4's tests FAILED, same values

On the tree without those four lines, cargo test --test shell_integration compound_pipe_into_function gives 3 passed and cargo test --test differential passes.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added your writer test in dec1ca7. It passes with the fix and fails with the four stdout-aside lines removed: the writer gets a, m1, m2. I also dropped "with or without this fix" from the Windows comment.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified on dec1ca7: the test passes, and fails on main and with the four saved_stdout lines removed (a, m1, m2).

@ferdingler

Copy link
Copy Markdown
Collaborator Author

@strandly-the-agent Please review this PR

@strandly-the-agent strandly-the-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at dec1ca7. The delta since f6a79a4 is test-only (src/ is unchanged), so the earlier review stands.

Previous finding Status
🟡 No test covers the lines that set stdout aside Fixed. writer_receives_function_piped_from_group_in_order passes here, and fails on main and with the four saved_stdout lines removed
🟡 Writer ordering after a later nested group (n, a) Still open in this PR, deferred to #390 by the author. That is the option I offered, and #390 already carries the repro script
⚪ "with or without this fix" comment Fixed
What I ran on dec1ca7

Linux aarch64, debug build.

  • cargo test --workspace --all-targets: 2737 passed, 0 failed, 0 ignored. cargo fmt --check clean; cargo clippy --workspace --all-targets exits 0 with no warning on a changed line. CI is green, including Rust (windows-latest).
  • dec1ca7 without the four saved_stdout lines: the writer test fails (left "a\nm1\nm2\n", right "m1\na\nm2\n"); the three compound_pipe_into_function_* tests pass.
  • dbb6189 source with dec1ca7's tests: all four fail.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] In the CLI, output of a group or subshell piped into a function skips the enclosing $( ), redirect or pipe (x=$( { echo a; } | f ) leaves x empty)

2 participants