Skip to content

[SDK] fix: respect shutdown timeout in span processor - #4767

Open
proost wants to merge 7 commits into
open-telemetry:mainfrom
proost:fix-respect-shutdown-timeout
Open

proost wants to merge 7 commits into
open-telemetry:mainfrom
proost:fix-respect-shutdown-timeout

Conversation

@proost

@proost proost commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Fixes #4532

Changes

Using conditional variable, return early even if background thread is not joined.

This is half solution for only Shutdown.

limitation: current destructor call InternalShutdown without timeout. So if exporter Export runs indefinitely, then destructor never finished. This is same after this PR is merged.

  • CHANGELOG.md updated for non-trivial changes
  • Unit tests have been added
  • Changes in public API reviewed

@proost
proost requested a review from a team as a code owner November 6, 2026 15:24
@proost proost changed the title Fix respect shutdown timeout fix: respect shutdown timeout in span processor Nov 6, 2026
@proost proost changed the title fix: respect shutdown timeout in span processor [SDK] fix: respect shutdown timeout in span processor Nov 6, 2026
@codecov

codecov Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.64%. Comparing base (d05c6a8) to head (4e08f27).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4767      +/-   ##
==========================================
+ Coverage   86.61%   86.64%   +0.04%     
==========================================
  Files         530      530              
  Lines       21169    21181      +12     
==========================================
+ Hits        18333    18350      +17     
+ Misses       2836     2831       -5     
Files with missing lines Coverage Δ
...ude/opentelemetry/sdk/trace/batch_span_processor.h 100.00% <ø> (ø)
sdk/src/trace/batch_span_processor.cc 94.48% <100.00%> (+3.03%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@proost
proost marked this pull request as draft November 7, 2026 14:36

@mateenali66 mateenali66 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch.

The timeout works, and batch_span_processor_test is clean under TSan at 4e337fe7.

BatchLogRecordProcessor::InternalShutdown has the same untimed join() (batch_log_record_processor.cc:410), and #4532 names both, so please use "Part of #4532" in the description.

See comments about what the caller still waits for, and the Shutdown contract.

Comment thread sdk/src/trace/batch_span_processor.cc Resolved
Comment thread sdk/src/trace/batch_span_processor.cc Resolved
@proost
proost marked this pull request as ready for review November 8, 2026 12:36
@proost
proost requested a review from mateenali66 November 8, 2026 12:36

@mateenali66 mateenali66 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update.

The repeated Shutdown now matches the contract, and batch_span_processor_test passes plain and under TSan at 4e08f27d.

The Part of #4532 request is still open, since BatchLogRecordProcessor::InternalShutdown keeps the untimed join().

See comment about a crash when Shutdown overlaps teardown.

Will approve once comments are fixed.

// A Shutdown() that timed out left the worker running and the exporter up.
if (worker_thread_.joinable())
{
worker_thread_.join();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This join() can run while another thread is still inside the join() in InternalShutdown, and std::thread::join then throws from a destructor.

With shutdown_m no longer taken, a second Shutdown returns before the first has finished, so its caller can release the last reference.

thread A  provider->Shutdown()
thread B  provider->Shutdown(200ms), then release the last reference

macOS, 12 runs each
main      B returns at +3003 ms, 12 clean
head      B returns at +55 ms, 12 abort
          thread::join failed: Invalid argument

Please take shutdown_m again, after the exchange in InternalShutdown and before this joinable().

std::lock_guard<std::mutex> shutdown_guard{synchronization_data_->shutdown_m};

Tested with both, B still returns at +55 ms, 12 clean, and the suite passes TSan over 20 repeats.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] BatchSpanProcessor::Shutdown ignores its timeout when the worker thread is blocked in Export()

2 participants