Skip to content

Recover Azure App Service agents after overlapping restarts - #12899

Open
Lewis-E wants to merge 30 commits into
masterfrom
lewis.lewis/svls-9057-aas-agent-supervision
Open

Lewis-E wants to merge 30 commits into
masterfrom
lewis.lewis/svls-9057-aas-agent-supervision

Conversation

@Lewis-E

@Lewis-E Lewis-E commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

This makes the tracer's agent supervisor in Azure App Service recover from overlapping restarts, and makes its lifecycle readable in default-level logs. A followup to #12898.

  1. ProcessSupervisor logs each agent start, exit (exit code and run time), and stop at info, with the tracer PID and, (on Java 9+) the agent PID. It also logs when it uses an agent that another process started, when that agent disappears, and when shutdown leaves an agent running. After repeated quick failures these lines drop to debug until the agent stays up, or an agent another process started stays healthy, for 60 s.
  2. Restarts back off from 2 s, doubling to a 60 s cap, and never stop. The fault count resets only after the agent has stayed up and passed its health checks for 60 s, or an agent another process started has stayed healthy for 60 s.
  3. While an agent this tracer started is running, the supervisor health-checks it every 10 s and restarts it after 3 failed checks in a row. It never starts a second copy while its own is still running. Next to an agent another process started, it starts its own only after that agent fails 3 checks in a row, as in .NET.
  4. A pipe whose instances are all busy counts as present, but the trace agent's /info check fails if the pipe is still busy after 500 ms.
  5. The trace agent health check now has an option to send GET /info over a pipe handle of its own, with tracing muted, on one reused thread, and waits at most 2 s.

Motivation

On Azure App Service (Windows), an overlapped IIS recycle leaves Java apps without telemetry until someone restarts them (SVLS-9057, escalation SLES-2826).

Background. In App Service, the Datadog site extension doesn't run an agent. Each tracer starts datadog-trace-agent.exe and dogstatsd.exe itself and sends to them over Windows named pipes. The extension picks the two pipe names once, at install, so every worker process on the site uses the same names. Only one process can own a pipe name. An agent started while another agent still holds the name fails to create its pipe, and the trace agent exits at once with "Access is denied". So before starting an agent, a tracer checks whether the pipe already exists, and if it does, it uses the agent that's already there.

During an overlapped IIS recycle, the old and new worker run at the same time, and each worker start also runs several JVMs in a row. So several tracers are checking the same pipes and starting agents on them at once. When one of those agents couldn't start, hung, or kept exiting, the tracer's ProcessSupervisor handled it badly:

  • An agent that couldn't start was given up on for good. After 5 failed starts the supervisor stopped trying, so the app sent nothing until it was restarted, even after the cause went away. For example, the agents run from %HOME%, so they can't start while Azure Storage is unavailable. Restarts now back off up to 60 s and never stop (change 2).
  • A hung agent was never noticed. After starting an agent, the supervisor only waited for it to exit. An agent that stopped answering but kept running also kept its pipe, so traces stopped until the app restarted. A hung agent is the leading explanation for one customer's 27-hour outage. The supervisor now health-checks agents it started and restarts them (changes 3 to 5).
  • An agent that kept exiting was restarted every 2 s, forever. The fault count reset on every start, so there was no backoff. For example, while the old worker's agent still owns the pipe during the overlap, each agent the new worker's tracer starts fails to create the pipe and exits right away. That tracer started a new process every 2 s for as long as the old agent ran. Restarts now back off up to 60 s (change 2).

Additionally:

  • The default logs couldn't show what happened. Starts, exits, exit codes, and use of another worker's agent were debug-only or not logged. Improved logging helps the customer figure out what's happening immediately, and helps us debug and determine a fix.
  • A new JVM truncated the shared tracer log file from 3,522 bytes to 0, so the old JVM's last lines, including its shutdown, were lost. A separate PR Write a per-process tracer log file in Azure App Service #13227 will address this if accepted.

Additional Notes

  • Busy pipes are recognized by the English text of the open error. Note that Azure always gives errors in English as far as I can tell. This still feels a bit fragile and better approaches would be welcome.
  • Any HTTP status from /info counts as healthy.
  • Pipe reads can't time out, so an /info request to a hung agent stays blocked until that agent exits. Later checks fail without sending another request.
  • A trace agent started by another worker that keeps failing its checks causes a duplicate start after every 3 failed checks in a row, as in .NET. The duplicate can't create the pipe and exits.
  • The supervisor restarts only an agent it started, and never kills one it didn't. If an adopted trace agent hangs while holding the pipe, the tracer logs it once and keeps retrying on the capped backoff, but traces stay down until that agent exits.
  • On Java 8 the logs show the child agent's PID as unknown, because Java 8 has no public API for it.
  • Tests: ProcessSupervisorTest covers the backoff, the fault-count reset, restarting after failed health checks, adopting an agent it didn't start, waiting for 3 failed checks before starting next to one, and not starting a second agent while one won't stop. NamedPipeHealthCheckTest covers the /info check's timeout and late answers on any OS.
  • The pipe health checks and the busy-pipe check ran in the App Service repro on Windows and have no unit tests. The 500 ms busy-pipe retry added during review has not run there yet. Tests could maybe be added after Experiment with testing of dd-trace-java on GitLab on Windows OS #12311 -- right now there's no windows CI where named pipes could even potentially be created.

Contributor Checklist

Jira ticket: SVLS-9057

@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 60.28%
• Overall Coverage: 60.33% (-0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5907ed2 | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.06 s 14.05 s [-0.6%; +0.8%] (no difference)
startup:insecure-bank:tracing:Agent 12.98 s 12.98 s [-0.7%; +0.7%] (no difference)
startup:petclinic:appsec:Agent 17.87 s 17.68 s [+0.2%; +2.0%] (maybe worse)
startup:petclinic:iast:Agent 17.69 s 17.64 s [-0.4%; +1.0%] (no difference)
startup:petclinic:profiling:Agent 17.37 s 17.24 s [-0.3%; +1.9%] (no difference)
startup:petclinic:sca:Agent 17.83 s 17.69 s [-0.1%; +1.7%] (no difference)
startup:petclinic:tracing:Agent 16.71 s 16.38 s [-2.3%; +6.3%] (no difference)

Commit: 5907ed24 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@Lewis-E
Lewis-E force-pushed the lewis.lewis/java-named-pipe-statsd-fixes branch 3 times, most recently from 2254f7d to 0c39b52 Compare October 30, 2026 16:45
@Lewis-E
Lewis-E force-pushed the lewis.lewis/svls-9057-aas-agent-supervision branch from 391843b to 1c4bed4 Compare October 30, 2026 16:48
Base automatically changed from lewis.lewis/java-named-pipe-statsd-fixes to master November 3, 2026 22:13
@Lewis-E
Lewis-E force-pushed the lewis.lewis/svls-9057-aas-agent-supervision branch from 1c4bed4 to 42dd7e4 Compare November 4, 2026 18:14
@Lewis-E
Lewis-E force-pushed the lewis.lewis/svls-9057-aas-agent-supervision branch from 1823a62 to d3b4fd3 Compare November 4, 2026 20:01
@Lewis-E
Lewis-E marked this pull request as ready for review November 5, 2026 16:40
@Lewis-E
Lewis-E requested a review from a team as a code owner November 5, 2026 16:40
@Lewis-E
Lewis-E requested review from bric3 and removed request for a team November 5, 2026 16:40
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-05T16:47:21.609076Z d3b4fd3 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d3b4fd3ff9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Localized Windows missing-pipe errors are treated as proof that an agent is running, preventing trace-agent and DogStatsD startup or recovery on affected hosts.

Open Bits AI session

🤖 Bits Code Review · Commit d3b4fd3

@Lewis-E

Lewis-E commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

@DataDog review

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Health-triggered restarts remain restricted to agents started by this tracer, preserving agents adopted from overlapping workers. Confidence is limited to static analysis; Windows named-pipe recovery was not exercised at runtime.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 92c63d2

In Azure App Service the tracer starts and supervises the trace agent and
DogStatsD. Overlapping worker processes during platform restarts can leave
those agents missing or shared, but starts, exits, and stops were only
logged at debug level, so default customer logs could not show what
happened.

- Log each agent start, exit (with exit code and run time), and stop at
  info, with the tracer PID and, on Java 9+, the agent PID.
- Log when an already-running agent not started by this process is used,
  and when that agent goes away.
- After repeated quick failures, move lifecycle lines to debug until the
  agent stays up, so a crash loop does not flood the log.
- Warn when shutdown leaves an agent running.
ProcessSupervisor reset its fault count on every successful start, so an
agent that kept exiting was restarted every 2 seconds forever, while a
start that kept throwing gave up for good after 5 attempts.

- Forget faults only after the process stays up for 60 seconds.
- Back off after each fault, doubling from 2 seconds to a cap of 60
  seconds, and never stop retrying.
- Move repeated start exceptions to debug after the first few, so a
  permanently failing start does not log a stack trace every minute.
Once ProcessSupervisor started an agent it blocked in waitFor() until the
agent exited, so an agent that hung but stayed alive was never checked or
replaced.

- While its own agent runs, wait in keep-alive steps of 10 seconds and
  run the health check after each one.
- After 3 failed health checks in a row, stop that agent and go back
  through the normal start path, after the fault backoff. Log the first
  failed check and the restart once.
- Keep tracking an agent that is still running after a stop request, so
  a restart never starts a second copy next to it.
- Supervisors without a health check keep only waiting for the exit.
On Windows, File.exists() can report false for a named pipe whose
instances are all busy. The supervisor then took a busy agent for a
missing one and started a competing agent, which lost the pipe race and
exited.

When File.exists() reports false on Windows, open the pipe once and close
it again. Only "cannot find the file" means the pipe is absent; "All pipe
instances are busy" and other open errors mean a process is serving it.
Lewis-E and others added 2 commits November 18, 2026 11:11
Co-authored-by: Brice Dutheil <brice.dutheil@gmail.com>
…r.java

Co-authored-by: Brice Dutheil <brice.dutheil@gmail.com>

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The PR improves Azure agent recovery with backoff, health checks, and lifecycle logging. Its latest PID-lookup refactor prevents compilation because of missing JDK imports, the only open review finding; compilation and Windows named-pipe behavior have not been exercised.

Open Bits AI session

🤖 Bits Code Review · Commit e81efaa

PidHelper now calls Process.pid() through a MethodHandle, but the
java.lang.invoke imports were missing, so internal-api didn't compile.
Add them, drop the unused Function import, and delete
JDK9ProcessPidFunction and its test, which nothing uses anymore.
The start-exception, restart, and exit paths each counted failures in a
row and chose the log level themselves, and only the exit path said when
the log went quiet. logFailure() and recordFailure() now do both, so the
"logged at debug level" notice is written whichever path reaches the
fifth failure.
@Lewis-E

Lewis-E commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

@bric3 thank you for taking a look -- I have applied both of your changes. I also refactored a small helper for some of the added logging.

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The PR improves Azure agent recovery with persistent backoff, health checks, and lifecycle logging. The missing-import compilation blocker is fixed and failure logging is consolidated, leaving no open review findings; compilation, tests, and Windows named-pipe behavior remain unexecuted.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit d4604f1

@bric3 bric3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pre-approving, I'd like another pair of eyes like @PerfectSlayer

Removing JDK9ProcessPidFunctionTest dropped the child PID cases, so a
broken Process.pid() handle would only show up as "unknown" in the logs.
Check PidHelper.getPid(Process) on Java 9+: the PID of a process that
reports one, and an empty string for one that can't.
The /info health check counted a busy pipe as an answer, so an agent
that stopped accepting connections passed every check and was never
restarted. The check now retries opening the pipe for up to 500 ms,
like .NET's NamedPipeClientStream.Connect and the tracer's own pipe
socket, and fails if the pipe is still busy. The pipe-present check
still counts a busy pipe as present.

A failed check next to an agent this tracer did not start used to start
a duplicate at once. Like dd-trace-dotnet, the supervisor now waits for
3 failed checks in a row first, and counts again after each start.
It also catches health-check exceptions from a process that is already
running, so "Exception starting process" was misleading.
An unexpected IOException, such as a broken pipe or access denied, looked
the same as a hung agent.
ProcessSupervisor is a general internal-api class. On Windows, where the
AAS agents run, destroy() and destroyForcibly() both call TerminateProcess,
but elsewhere destroy() may only ask the process to exit. The total wait
stays 400 ms.
A new thread for every /info check, every 10 s, added up to thousands of
short-lived threads a day. A new request is sent only once the previous
one has finished, so one thread is enough.
agentResponds now takes the request as a Callable, so its timeout logic
runs without a named pipe: an answer within the timeout, a late answer
counted at the next check, a late failure followed by a new request, and
no new request while one is still waiting.
A single healthy check of an agent this tracer did not start reset the
failure count, so the logs of an agent that was only sometimes healthy
never went quiet. Like a process this tracer started, and like
dd-trace-dotnet, it now has to stay healthy for 60 s. That also resets
the restart backoff, which an adopted agent never reset before.
The backoff and the quiet-logging count follow the same 60 s rule, which
three places applied by hand. One helper now does both.
An agent that was hung and did not stop when asked stays tracked, and once
its start was over 60 s ago every wait reset the backoff and the quiet
logging. It was then retried about every 12 s, with two warnings each
time, for as long as it stayed hung.
The backoff and the quiet logging kept the same count in two counters,
reset by hand in each place that records uptime. FaultBackoff now owns
the one count: it spaces out retries and says when logging goes quiet.
The repository's guidance asks for Javadoc on field documentation.
The health check now also judges a running process, and ALWAYS_READY is
recognized by identity, so write both down.

@PerfectSlayer PerfectSlayer left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two final findings and we should be all good :)

Nothing shut the /info request thread down. HealthCheck gets a close()
that the supervisor thread calls when it stops, and the pipe check shuts
its request thread down there. A request still blocked on a read of a
hung agent's pipe may stay blocked until that agent exits.
"Stopping process", "Stopped process" and "was still running after the
stop request" bypassed the quiet mode, so an agent that kept hanging and
getting restarted still logged them at info or warn each cycle.

@bric3 bric3 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe all the points have been addressed for me, thanks a lot !
Ill let @PerfectSlayer validate his points though.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: core Tracer core comp: logging Tracer internal logging type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants