fix(flows): avoid StackOverflowError after many steps in BaseLlmFlow

[innoprej]

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

• Closes: BaseLlmFlow nests each step inside the previous one and overflows the stack after a few hundred LLM calls #1564
• Related: fix: avoid StackOverflowError in PersistBarrier.awaitPersisted for large steps #1336, which fixed a different StackOverflowError inside PersistBarrier.awaitPersisted (a single step with many events)

2. Or, if no issue exists, describe the change:

Problem:

BaseLlmFlow.run subscribes to each step from inside the previous step's completion (concatWith → toList → flatMapPublisher → PersistBarrier.awaitPersisted(...).andThen(run(...))). When a step completes on the subscribing thread, the next step starts on the same call stack, 23 frames deeper. The built-in models and session services complete on the subscribing thread (Gemini through Flowable.fromFuture, Claude and the non-streaming LangChain4j and SpringAI paths through Flowable.just after a blocking call, the session services through Single.just or Single.fromCallable), so with a 1 MB thread stack, the default on x86-64, an agent that keeps calling tools fails with StackOverflowError after roughly 250–300 LLM calls, before the default maxLlmCalls of 500 can end the run. On aarch64 the default thread stack is 2 MB; on a 2 MB stack the same agent made 635–663 calls in my runs, so there the default limit usually comes first. The error never reaches onError: it is thrown out of subscribe(), or, when the run is subscribed on a scheduler thread, it goes to RxJavaPlugins as an UndeliverableException and the subscriber never receives a terminal signal (in my runs, blockingGet() returned only when a 15-second timeout fired). The issue has the measurements.

Solution:

Drive the steps with repeatUntil, as LoopAgent loops over its sub-agents on its resumable path:

• run(InvocationContext) now runs one step per subscription of an inner Flowable.defer(...) and lets repeatUntil subscribe again until the flow ends. When a step completes synchronously, repeatUntil resubscribes from a loop instead of from the completion callback (FlowableRepeatUntil.RepeatSubscriber.subscribeNext in RxJava 3.1.12), so every step starts at the same stack depth. This is the Java counterpart of the while True: loop in adk-python's BaseLlmFlow.run_async.
• The private method that runs one step is renamed from run to runStep, since it no longer runs the rest of the flow, and gets a short Javadoc. Its end-of-flow checks, the pause on a pending long-running call and the maxSteps cut-off are unchanged. Where it used to subscribe to the next step, it now sets a continueFlow flag and ends the step with the PersistBarrier wait.
• The step counter and the flag are created per subscription, inside the outer Flowable.defer.

Events are still emitted as each step produces them, the next step still starts only after the Runner has persisted the previous step's events, and an error still ends the run. One difference: subscribing twice to the same Flowable returned by run(ctx) now runs the flow twice from the start, where the old code replayed the first step from its cache() and ran the later steps again. Agents do not do this: BaseAgent calls runAsyncImpl again for every subscription (through Flowable.defer), and LlmAgent.runAsyncImpl calls run each time.

Testing Plan

Unit Tests:

• I have added or updated unit tests for my change.
• All unit tests pass locally. (core, see the table below)

BaseLlmFlowTest.run_modelRespondingOnSubscribingThread_reachesMaxLlmCallsWithoutStackOverflow runs the flow with a model that returns a function call through Flowable.just every time and maxLlmCalls set to 2,000, and expects the run to end with LlmCallsLimitExceededException after 2,000 requests. The model also records the StackWalker depth at every call, and the test expects the same depth for all 2,000 calls, so whether it fails on main does not depend on the thread stack size: in my runs it fails with StackOverflowError on the default 1 MB stack, and on 8 MB and 16 MB stacks, where main reaches 2,000 calls, the depth check fails because the depth grows by 23 frames per step. It runs the flow without a Runner, so the history does not grow, and it takes about a second.

Windows 11 on x86-64, Microsoft OpenJDK 17.0.19, Maven 4.0.0-rc-3 via mvnw:

Run	main (BaseLlmFlow.java from main, with the new test)	This change
./mvnw -pl core test -Dtest=BaseLlmFlowTest	28 run, 1 error: the new test, java.lang.StackOverflowError	28 run, 0 failures
./mvnw -pl core clean test	not run	default-test and basic each run 2,205 tests, 24 skipped, with no failures. Of the other four surefire executions, three pass and apigee-llm-proxy-url runs no tests (#1557).

Manual End-to-End (E2E) Tests:

Not tested against a live model. The reproduction in the issue runs an LlmAgent with an EchoTool in an InMemoryRunner, with a TestLlm that asks for the tool on every call and answers on the subscribing thread:

	main	This change
1 MB thread stack (the x86-64 default)	StackOverflowError after 253–265 LLM calls (14 runs)	LlmCallsLimitExceededException after 500
-Xss256k	StackOverflowError after 42	LlmCallsLimitExceededException after 500
-Xss2m (the aarch64 default size)	stops at the 500-call limit; with maxLlmCalls raised to 2,000, StackOverflowError after 635–663 calls	LlmCallsLimitExceededException at the limit (500, or 2,000 when raised)
Run subscribed with .subscribeOn(Schedulers.io()) and .timeout(15, SECONDS)	stack overflow after 254–263 calls, an UndeliverableException to RxJavaPlugins, no signal until the timeout fires	LlmCallsLimitExceededException after 500, in under 2 s

Recording the StackWalker depth at each model response shows 23 more frames per step on main, and the same depth at every step with this change, also over 20,000 steps on a 256 KB stack.

Checklist

• I have read the CONTRIBUTING.md document.
• My pull request contains a single commit.
• I have performed a self-review of my own code.
• I have commented my code, particularly in hard-to-understand areas.
• I have added tests that prove my fix is effective or that my feature works.
• New and existing unit tests pass locally with my changes. (core)
• I have manually tested my changes end-to-end.
• Any dependent changes have been merged and published in downstream modules.

Additional context

Rebased onto main at dbc973b without conflicts. main already imports AtomicBoolean and has the legacyResumption check, which runStep keeps.

[hemasekhar-p]

Hi @innoprej, thank you for your contribution We appreciate you taking the time to submit this pull request. Currently this PR is under review by our team we will keep you posted if any additional information is required. thank you.

[MiloszSobczyk]

Helllo, could you please merge the newest changes that were introduced in the code? Afterwards, please ensure that the code still works correctly.

[innoprej]

Done. I rebased the single commit onto the current main (2d4a59e). The only conflict was in the imports: main now imports AtomicBoolean itself, so this change only adds AtomicInteger; the rest of the diff is unchanged. main's new resumption logic is kept as is: StepResume runs inside runOneStep, which this change doesn't touch, and the legacyResumption pause in runStep returns without setting continueFlow, so the loop ends there, as it does at the other end conditions.

On Windows 11 with JDK 17, BaseLlmFlowTest, PersistBarrierTest, RunnerResumabilityTest and RunnerLegacyResumabilityTest pass (95 tests in both the default-test and basic executions). With only the test file applied to the same main, the new regression test still fails with StackOverflowError. The full core suite (2,016 tests in each of those executions) has one failure, LocalSkillSourceTest.testListResources, the Windows-only failure that is also on main and that #1542 fixes.

[innoprej]

@MiloszSobczyk Could you take another look when you have a moment? Your approval is on the previous commit (5d92339), and the rebased commit (7470b43) has no import/copybara status yet, while #1534 and #1542, rebased at the same time, have since been merged. The only difference from the commit you approved is the AtomicBoolean import described in my comment above. CI on 7470b43 passed on October 6, and the PR still merges cleanly into main as of 7d11442, with the full core suite passing on that merge locally. #1621 changes the same code and plans to rebase onto this PR once it lands.

[MiloszSobczyk]

Hello, sorry for the delay. Thank you for the rebase, the only change since my approval is the import, so this still LGTM. One optional idea, please feel free to skip it: the regression test passes on the old code with an 8 MB stack. Recording the StackWalker depth at each model call and asserting it stays the same would make the test independent of -Xss

[MiloszSobczyk]

No need for that comment. Reword to say // Flowable.just answers on the subscribing thread, like the built-in models.

[innoprej]

Replaced the three lines with that comment.

[innoprej]

Thanks for taking another look. I added it: the test model records the StackWalker depth on every call, and the test asserts that all 2,000 are equal. With the old BaseLlmFlow the depth grows by 23 frames per step, so in my runs the test now fails there on 8 MB and 16 MB stacks too, where the existing checks passed; on the default 1 MB stack it still fails with StackOverflowError.

Since 7470b43 only BaseLlmFlowTest.java has changed: your comment wording and this check. The branch still merges cleanly into main (dbc973b), and BaseLlmFlowTest passes on that merge. I updated the test paragraph in the description to match.

[innoprej]

Rebased onto the current main (dbc973b), since the GitHub Actions Scan flagged the old base's workflows (mostly unpinned actions), which main has since fixed or removed. There were no conflicts, and the patch is unchanged from 9a98d4d. On the new base the full core suite passes, and with only the test file applied the regression test still fails with StackOverflowError. I updated the rebase note and the test numbers in the description.

[MiloszSobczyk]

Thank you for your contribution @innoprej. The PR has been merged after some internal changes and will be part of the next non bugfix release.