Skip to content

Release 2.1.3: retry lifecycle results on transient errors, directory symlinks, non-UTF-8 file names, and Windows install permissions - #436

Merged
nathan-inge merged 8 commits into
mainfrom
2.1.3
Nov 4, 2026
Merged

nathan-inge merged 8 commits into
mainfrom
2.1.3

Conversation

@nathan-inge

Copy link
Copy Markdown
Contributor

Agent release 2.1.3.

Added

  • Optional preserve_directory_symlinks (default false, Linux and macOS). A symlink to a directory in the revision is installed as a symlink instead of copying the directory it points to
    (#152). The permissions section is not applied through a preserved link.

Fixed

  • Lifecycle event results are retried on transient errors. When reporting a result fails with a 5xx or network error, the agent retries with exponential backoff for up to 65 minutes after the command was acknowledged,
    instead of dropping the result. A hook that succeeded no longer fails with "CodeDeploy agent was not able to notify CodeDeploy server the execution result of this lifecycle event". Stopping the agent during retries can take as
    long as the retry period, bounded by kill_agent_max_wait_time_seconds.
  • Installing a symlink over an existing file or symlink replaces it (#143). A symlink over an existing directory is refused before any file is copied, and cleanup
    removes dangling symlinks.
  • File names that are not valid UTF-8 no longer hang the Install event (#284). They are installed and cleaned up on the next deployment. A destination path that
    contains a line break fails the deployment before any files are copied.
  • On Windows, standard users keep read access to the agent's deployment folders (deployment-logs, ongoing-deployment, and others), matching the 0755 folders on Linux and macOS. They cannot modify them.

Security

  • Improved file system permissions on the agent installation folder on Windows Server.

Also in 2.1.3 (already on main)

  • Concurrent deployments that create the same missing parent directory no longer fail Install (#429).
  • GitHub deployments that use an access token no longer fail with HTTP 401 (#434, #435).
    By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

The MSI now sets an explicit ACL on C:\ProgramData\Amazon\CodeDeploy
instead of leaving the folder to inherit its permissions from
%ProgramData%. As in the 1.x MSI, a single CreateFolder/Permission
grants Administrators full control on the install root, and the
subfolders inherit it. Windows Installer adds LocalSystem full control
to every LockPermissions descriptor.

tests/msi_install_dir_acl.rs reads wix/main.wxs and fails if the
install-root ACL is removed or loosened, or if any other Permission is
added.
…adline

A completion that hit a 5xx or a network error was logged and dropped:
only throttling was retried. The lifecycle event then stayed InProgress
until the service's 70-minute acknowledgement timeout failed it with
"CodeDeploy agent was not able to notify CodeDeploy server the execution
result of this lifecycle event", even when the hook had succeeded.

report_completion now retries retryable errors until 65 minutes after
the command was acknowledged, with full-jitter exponential backoff (1 s,
doubling, capped at 30 s). Every attempt sends the same status and
diagnostics, so a transport failure never changes the reported result.
Resending is safe because the service keeps the first completion it
records and discards later ones, and ignores completions after its
timeout. The deadline is taken before the ack is sent so it can only
fall early. Non-retryable errors (4xx, signing, serialization) are still
logged and dropped.

- codedeploy-commands Error keeps the HTTP status (status(),
  with_status()). A 503 ServiceUnavailableException and a bodyless 500
  both classify as Http, so the status is what tells them apart from a
  4xx.
- CodeDeployClientError::is_retryable covers throttling, any 5xx, server
  exceptions, network errors and an unavailable IMDS.
- Before retrying after a 5xx response (other than throttling), the
  client drops its pooled connections (Client::reset_http_connections,
  which rebuilds only the reqwest client and keeps the credentials), so
  retries don't keep going to the backend host that failed. Resets are
  rate-limited to one per minute per client. Only the processor's client
  is reset; the poller keeps its connection.
- HTTP calls no longer hold the client lock for the whole request. Each
  call copies the client under a brief read lock and sends on the copy,
  so a reset or credential refresh no longer waits for in-flight
  requests.
- Retries continue through agent shutdown, bounded by the worker's drain
  timeout (kill_agent_max_wait_time_seconds, as for a running hook), so
  a stop during an outage can take up to the 65-minute deadline.
- CompletionRetryPolicy is configurable for tests.
Adds an opt-in preserve_directory_symlinks setting (default false,
Linux and macOS only) that installs a symlink to a directory in the
revision as a symlink instead of copying the directory it points to
(#152). The permissions section is never applied through a preserved
link; a warning is logged instead.

Installing a symlink over an existing file or symlink now replaces it
atomically (#143), using a temporary link and rename. A symlink over an
existing directory is refused before any file is copied. Cleanup also
removes dangling symlinks.
A file name that isn't valid UTF-8 panicked the command thread, so the
Install event never completed (#284).

- install.json paths use to_string_lossy().
- The cleanup file stores raw path bytes on Unix and is read back as
  bytes, so such files and directories are removed on the next
  deployment. Windows is unchanged (text lines).
- A destination containing a line break fails planning, and the cleanup
  writer refuses one too, so a file name can't add cleanup entries.
install_non_utf8_file_name, install_non_utf8_directory_name_redeploys
and execute_cleanup_removes_non_utf8_path create a file or directory
whose name is not valid UTF-8. APFS rejects such names with EILSEQ
("Illegal byte sequence", os error 92), so on macOS the tests fail in
setup, before any installer code runs.

Run the three tests on Unix other than macOS. No revision on macOS can
hold such a name, so the case they cover doesn't arise there. The
in-memory non-UTF-8 tests create no files and still run on macOS.
The install root now carries an explicit Administrators/SYSTEM ACL that
its subfolders inherit, so agent dirs created under the default
restrict_agent_dir_permissions: false no longer inherit read access for
BUILTIN\Users from %ProgramData%.

- create_dir_world_readable on Windows grants BUILTIN\Users
  ReadAndExecute on the dir, inherited by files and subdirs, keeping the
  dir's other ACEs and its protection state. This is the Windows
  counterpart of the 0755 dirs on Linux and macOS. Users never get
  write. It's a no-op when Users can already read the dir, so
  re-touching a dir doesn't re-propagate ACLs through its subtree on
  every deployment.
- New create_root_dir, used by ensure_dirs for root_dir. On Windows
  root_dir is the install root, which also holds bin\, conf.yml and
  conf.onpremises.yml, so its installer ACL is left as is. Elsewhere,
  and with the flag set, it's create_deployment_dir.
- restrict_agent_dir_permissions: true is unchanged.
@gmiliaras
gmiliaras self-requested a review November 4, 2026 10:02
@nathan-inge
nathan-inge merged commit b1c56fb into main Nov 4, 2026
11 checks passed
@nathan-inge
nathan-inge deleted the 2.1.3 branch November 4, 2026 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants