Run portable ValidatesRunner tests one at a time

[aIbrahiim]

fixes: #38934
caused by: #40324
#38934 (comment)

[github-actions]

📋 Review Summary

This PR limits in-JVM test concurrency for Flink Portable ValidatesRunner tasks to 1 thread (beam.validatesRunner.parallelThreads=1), matching the existing configuration in the Spark job server. This resolves OOMs and test hangs caused by spawning up to 8 concurrent in-process job servers and Flink mini-clusters within a single 4GB test worker JVM.

🔍 General Feedback

• The fix is clean, well-targeted, and directly addresses the resource contention introduced when beam.test.parallelThreads was defaulted to 8 in #40324 while preserving fork-level parallelism (maxParallelForks = 2).

---

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

[Abacn]

Thanks!

[Abacn]

However it seems the test still stuck, probably other issues introduced since then

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[aIbrahiim]

However it seems the test still stuck, probably other issues introduced since then

yeah the 1.20 job is the old DataSet suite and those 7 tests already hang on master until they hit the 20 min timeout but the other Flink versions passed. i excluded them for DataSet only so that job can actually finish

[Abacn]

Thanks, lgtm for now, however we still need to fix these tests as Flink 1.20 is LTS. Could you please open an Issue as a release block?

[Abacn]

actually it seems still stuck

[aIbrahiim]

actually it seems still stuck

yeah the hang was from the portable stage context sticking around across jobs in the same jvm so I cleaned that up per job and had to keep the env cache delay so teardown still gets a chance and PostCommit Flink Batch is successful now

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.05%. Comparing base (76f6e6d) to head (15c3b9f).
⚠️ Report is 221 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #40632      +/-   ##
============================================
+ Coverage     56.16%   65.05%   +8.88%     
- Complexity     2288    13993   +11705     
============================================
  Files          1124     1729     +605     
  Lines        178336   108515   -69821     
  Branches       1489    11313    +9824     
============================================
- Hits         100170    70599   -29571     
+ Misses        75645    33704   -41941     
- Partials       2521     4212    +1691     

Flag	Coverage Δ
go	?
java	65.05% <ø> (-10.73%)	⬇️
python	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Assigning reviewers:

R: @tvalentyn added as fallback since no labels match configuration
Backup reviewer: jrmccluskey (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[aIbrahiim]

@Abacn PTAL

[Abacn]

Thanks!