Repository navigation
Conversation
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
2254f7d to
0c39b52
Compare
391843b to
1c4bed4
Compare
1c4bed4 to
42dd7e4
Compare
1823a62 to
d3b4fd3
Compare
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d3b4fd3ff9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
Localized Windows missing-pipe errors are treated as proof that an agent is running, preventing trace-agent and DogStatsD startup or recovery on affected hosts.
🤖 Bits Code Review · Commit d3b4fd3
|
@DataDog review |
There was a problem hiding this comment.
More details
Health-triggered restarts remain restricted to agents started by this tracer, preserving agents adopted from overlapping workers. Confidence is limited to static analysis; Windows named-pipe recovery was not exercised at runtime.
🤖 Bits Code Review · Commit 92c63d2
In Azure App Service the tracer starts and supervises the trace agent and DogStatsD. Overlapping worker processes during platform restarts can leave those agents missing or shared, but starts, exits, and stops were only logged at debug level, so default customer logs could not show what happened. - Log each agent start, exit (with exit code and run time), and stop at info, with the tracer PID and, on Java 9+, the agent PID. - Log when an already-running agent not started by this process is used, and when that agent goes away. - After repeated quick failures, move lifecycle lines to debug until the agent stays up, so a crash loop does not flood the log. - Warn when shutdown leaves an agent running.
ProcessSupervisor reset its fault count on every successful start, so an agent that kept exiting was restarted every 2 seconds forever, while a start that kept throwing gave up for good after 5 attempts. - Forget faults only after the process stays up for 60 seconds. - Back off after each fault, doubling from 2 seconds to a cap of 60 seconds, and never stop retrying. - Move repeated start exceptions to debug after the first few, so a permanently failing start does not log a stack trace every minute.
Once ProcessSupervisor started an agent it blocked in waitFor() until the agent exited, so an agent that hung but stayed alive was never checked or replaced. - While its own agent runs, wait in keep-alive steps of 10 seconds and run the health check after each one. - After 3 failed health checks in a row, stop that agent and go back through the normal start path, after the fault backoff. Log the first failed check and the restart once. - Keep tracking an agent that is still running after a stop request, so a restart never starts a second copy next to it. - Supervisors without a health check keep only waiting for the exit.
On Windows, File.exists() can report false for a named pipe whose instances are all busy. The supervisor then took a busy agent for a missing one and started a competing agent, which lost the pipe race and exited. When File.exists() reports false on Windows, open the pipe once and close it again. Only "cannot find the file" means the pipe is absent; "All pipe instances are busy" and other open errors mean a process is serving it.
Co-authored-by: Brice Dutheil <brice.dutheil@gmail.com>
…r.java Co-authored-by: Brice Dutheil <brice.dutheil@gmail.com>
There was a problem hiding this comment.
The PR improves Azure agent recovery with backoff, health checks, and lifecycle logging. Its latest PID-lookup refactor prevents compilation because of missing JDK imports, the only open review finding; compilation and Windows named-pipe behavior have not been exercised.
🤖 Bits Code Review · Commit e81efaa
PidHelper now calls Process.pid() through a MethodHandle, but the java.lang.invoke imports were missing, so internal-api didn't compile. Add them, drop the unused Function import, and delete JDK9ProcessPidFunction and its test, which nothing uses anymore.
The start-exception, restart, and exit paths each counted failures in a row and chose the log level themselves, and only the exit path said when the log went quiet. logFailure() and recordFailure() now do both, so the "logged at debug level" notice is written whichever path reaches the fifth failure.
|
@bric3 thank you for taking a look -- I have applied both of your changes. I also refactored a small helper for some of the added logging. |
There was a problem hiding this comment.
More details
The PR improves Azure agent recovery with persistent backoff, health checks, and lifecycle logging. The missing-import compilation blocker is fixed and failure logging is consolidated, leaving no open review findings; compilation, tests, and Windows named-pipe behavior remain unexecuted.
🤖 Bits Code Review · Commit d4604f1
bric3
left a comment
There was a problem hiding this comment.
Pre-approving, I'd like another pair of eyes like @PerfectSlayer
Removing JDK9ProcessPidFunctionTest dropped the child PID cases, so a broken Process.pid() handle would only show up as "unknown" in the logs. Check PidHelper.getPid(Process) on Java 9+: the PID of a process that reports one, and an empty string for one that can't.
The /info health check counted a busy pipe as an answer, so an agent that stopped accepting connections passed every check and was never restarted. The check now retries opening the pipe for up to 500 ms, like .NET's NamedPipeClientStream.Connect and the tracer's own pipe socket, and fails if the pipe is still busy. The pipe-present check still counts a busy pipe as present. A failed check next to an agent this tracer did not start used to start a duplicate at once. Like dd-trace-dotnet, the supervisor now waits for 3 failed checks in a row first, and counts again after each start.
It also catches health-check exceptions from a process that is already running, so "Exception starting process" was misleading.
An unexpected IOException, such as a broken pipe or access denied, looked the same as a hung agent.
ProcessSupervisor is a general internal-api class. On Windows, where the AAS agents run, destroy() and destroyForcibly() both call TerminateProcess, but elsewhere destroy() may only ask the process to exit. The total wait stays 400 ms.
A new thread for every /info check, every 10 s, added up to thousands of short-lived threads a day. A new request is sent only once the previous one has finished, so one thread is enough.
agentResponds now takes the request as a Callable, so its timeout logic runs without a named pipe: an answer within the timeout, a late answer counted at the next check, a late failure followed by a new request, and no new request while one is still waiting.
A single healthy check of an agent this tracer did not start reset the failure count, so the logs of an agent that was only sometimes healthy never went quiet. Like a process this tracer started, and like dd-trace-dotnet, it now has to stay healthy for 60 s. That also resets the restart backoff, which an adopted agent never reset before.
The backoff and the quiet-logging count follow the same 60 s rule, which three places applied by hand. One helper now does both.
An agent that was hung and did not stop when asked stays tracked, and once its start was over 60 s ago every wait reset the backoff and the quiet logging. It was then retried about every 12 s, with two warnings each time, for as long as it stayed hung.
The backoff and the quiet logging kept the same count in two counters, reset by hand in each place that records uptime. FaultBackoff now owns the one count: it spaces out retries and says when logging goes quiet.
The repository's guidance asks for Javadoc on field documentation.
The health check now also judges a running process, and ALWAYS_READY is recognized by identity, so write both down.
PerfectSlayer
left a comment
There was a problem hiding this comment.
The two final findings and we should be all good :)
Nothing shut the /info request thread down. HealthCheck gets a close() that the supervisor thread calls when it stops, and the pipe check shuts its request thread down there. A request still blocked on a read of a hung agent's pipe may stay blocked until that agent exits.
"Stopping process", "Stopped process" and "was still running after the stop request" bypassed the quiet mode, so an agent that kept hanging and getting restarted still logged them at info or warn each cycle.
There was a problem hiding this comment.
I believe all the points have been addressed for me, thanks a lot !
Ill let @PerfectSlayer validate his points though.
What Does This Do
This makes the tracer's agent supervisor in Azure App Service recover from overlapping restarts, and makes its lifecycle readable in default-level logs. A followup to #12898.
ProcessSupervisorlogs each agent start, exit (exit code and run time), and stop at info, with the tracer PID and, (on Java 9+) the agent PID. It also logs when it uses an agent that another process started, when that agent disappears, and when shutdown leaves an agent running. After repeated quick failures these lines drop to debug until the agent stays up, or an agent another process started stays healthy, for 60 s./infocheck fails if the pipe is still busy after 500 ms.GET /infoover a pipe handle of its own, with tracing muted, on one reused thread, and waits at most 2 s.Motivation
On Azure App Service (Windows), an overlapped IIS recycle leaves Java apps without telemetry until someone restarts them (SVLS-9057, escalation SLES-2826).
Background. In App Service, the Datadog site extension doesn't run an agent. Each tracer starts
datadog-trace-agent.exeanddogstatsd.exeitself and sends to them over Windows named pipes. The extension picks the two pipe names once, at install, so every worker process on the site uses the same names. Only one process can own a pipe name. An agent started while another agent still holds the name fails to create its pipe, and the trace agent exits at once with "Access is denied". So before starting an agent, a tracer checks whether the pipe already exists, and if it does, it uses the agent that's already there.During an overlapped IIS recycle, the old and new worker run at the same time, and each worker start also runs several JVMs in a row. So several tracers are checking the same pipes and starting agents on them at once. When one of those agents couldn't start, hung, or kept exiting, the tracer's
ProcessSupervisorhandled it badly:%HOME%, so they can't start while Azure Storage is unavailable. Restarts now back off up to 60 s and never stop (change 2).Additionally:
Additional Notes
/infocounts as healthy./inforequest to a hung agent stays blocked until that agent exits. Later checks fail without sending another request.unknown, because Java 8 has no public API for it.ProcessSupervisorTestcovers the backoff, the fault-count reset, restarting after failed health checks, adopting an agent it didn't start, waiting for 3 failed checks before starting next to one, and not starting a second agent while one won't stop.NamedPipeHealthCheckTestcovers the/infocheck's timeout and late answers on any OS.dd-trace-javaon GitLab on Windows OS #12311 -- right now there's no windows CI where named pipes could even potentially be created.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: SVLS-9057