Skip to content

Automated Sync from main to stable - #40

Open
rhods-devops-app[bot] wants to merge 341 commits into
stablefrom
main
Open

rhods-devops-app[bot] wants to merge 341 commits into
stablefrom
main

Conversation

@rhods-devops-app

@rhods-devops-app rhods-devops-app Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Automated Sync from main to stable

This PR automatically syncs the main branch to the stable branch by opening a pull request from main into stable.

Sync Summary

  • Latest commit: 743d2977 Merge remote-tracking branch 'upstream/main'

  • Total commits to sync: 328

  • Source: https://github.com/red-hat-data-services/OpenShell.git @ main

  • Target: https://github.com/red-hat-data-services/OpenShell.git @ stable

  • PR head: main

Commits to be synced

Merging

GitHub automerge is enabled for this pull request once required checks pass.

benoitf and others added 30 commits October 16, 2026 07:56
…3463)

* feat(vm): add host bind mount support via virtiofs (libkrun)

Introduce host directory bind mounts for libkrun VM sandboxes using
the virtiofs4 API with simplified permission semantics. The gateway
accepts `enable_bind_mounts` in the `[openshell.drivers.vm]` TOML
table and forwards it to the VM driver subprocess.

Mounts are specified per-sandbox in driver_config.mounts[] with
source, target, optional read_only (default: read-only), and an
optional `type` that must be `bind`, so Docker and Podman bind entries
(without `selinux_label`) parse unchanged.

At sandbox validation the driver rejects mounts unless bind mounts
are enabled and resource admission is disabled. It rejects GPU and
other QEMU launches, sources with control characters, and more mounts
than libkrun has interrupt lines for (32 on aarch64, 3 on x86_64), and
checks targets against reserved VM-internal directories and each other
(no nested targets). The QEMU check runs again once the launch backend
is resolved.

Host source directories are checked at provisioning, before image
preparation, so a briefly missing source fails provisioning visibly
instead of denying recovery of a persisted sandbox. Missing,
inaccessible, symlinked, non-directory, and unreadable sources are
rejected: libkrun does not follow a final symlink, and on macOS it
panics during guest boot when it cannot open the share root.

The driver generates virtiofs tags and always writes a guest-side
manifest into the overlay disk, together with the init drop-ins, so an
image-supplied manifest can never drive guest mounts. The launcher
parses its tab-separated --vm-mount arguments from the right, so a
source can never shift the tag or mode, and the driver logs each
shared directory at launch.

Guest init mounts the shares after every /sandbox ownership fixup
and the init drop-ins, so recursive chowns never reach host files.
It refuses targets that traverse symlinks in the image, hands newly
created mount point parents under /sandbox to the sandbox user, and
skips an empty manifest.

krun_add_virtiofs4 is resolved optionally, so older libkrun runtimes
still boot sandboxes without mounts.

The kernel config lists CONFIG_FUSE_FS and CONFIG_VIRTIO_FS
explicitly; the pinned libkrunfw base config already enables both.

A vm_bind_mount e2e test checks a read-write share round-trips data
with the host, accepts a new file, directory, and nested file that the
host then sees, and that a read-only share rejects writes. libkrun on
rootless Linux refuses file creation by a guest user whose UID/GID
differ from its own, so the VM e2e gateway runs the sandbox as the
gateway user there. It enables bind mounts like the Docker and Podman
e2e gateways.

The runtimes docs cover the mount limit, the symlink restriction, the
guest-kernel exposure, and the rootless Linux UID/GID note. The
debug-openshell-cluster skill lists VM bind mount checks.

Signed-off-by: Florent Benoit <fbenoit@redhat.com>

* fix(nix): refresh vm-runtime hashes to the libkrun v1.19.4 runtime

The Nix vm-runtime package fetches the rolling vm-runtime release with
fixed hashes pinned on 2026-09-16. The release was rebuilt on
2026-09-21 after the libkrun v1.19.4 / libkrunfw v5.6.1 bump (NVIDIA#3451),
but the hashes were not updated, so CI kept substituting the
2026-09-15 runtime (libkrun v1.17.4) from the binary cache.

That libkrun lacks krun_add_virtiofs4, which VM bind mounts need.
Point the hashes at the current release assets.

Signed-off-by: Florent Benoit <fbenoit@redhat.com>

---------

Signed-off-by: Florent Benoit <fbenoit@redhat.com>
…IDIA#4632)

Requests the gateway rejects before consulting any authenticator now emit
an OCSF Authentication [3002] event: no credential presented, mTLS
required without a client certificate, or an operator certificate
combined with an authorization header. Each event carries a reason
category in status_detail and never the presented credential. The
response returned to the caller is unchanged.

openshell-ocsf gains the Authentication class with user and service
objects, and a shared OcsfCapture test layer. Schema validation now
rejects attributes the class does not define and profile attributes
whose profile the event does not declare.

Refs NVIDIA#2912

Signed-off-by: Kris Hicks <khicks@nvidia.com>
…on (NVIDIA#4524)

The endpoint-status startup sweep ran before the gateway installed its
Prometheus recorder, so the lock waits, timeouts, errors and hold times
it recorded went to the no-op recorder and only its warnings reached
the log. The sweep retries lock timeouts for up to three lock waits
before startup fails, and a slow sweep that still succeeds is what
operators want to see in the lock metrics.

When metrics are enabled, the recorder is now installed before startup
reconciliation, and the metrics listener still binds with the other
listeners. A new pod's first scrape therefore already holds the sweep's
samples, so they show in the raw counter and histogram count values,
not in rate() or increase(). A sweep that fails still exits before any
listener binds, so its timeouts remain log-only.

The startup lock-timeout test now waits for the sweep's timeout counter
instead of a fixed sleep and checks its timeout and wait samples, and
the debug skill says which startup timeouts reach the metrics and how
to read them.

Part of NVIDIA#4307
Closes NVIDIA#4523

Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(cli): add --annotation to sandbox create

Signed-off-by: Eric Curtin <eric.curtin@docker.com>

* fix(cli): drop reserved annotation check

Signed-off-by: Eric Curtin <eric.curtin@docker.com>

* test(cli): adapt annotation tests to env helper

Signed-off-by: Eric Curtin <eric.curtin@docker.com>

---------

Signed-off-by: Eric Curtin <eric.curtin@docker.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>
…er (NVIDIA#4424)

* fix(ci): make conformance-only e2e selection explicit

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(ci): use mapped NodePort for kind gateway sessions

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(ci): waive SSH transport recovery only on Kubernetes

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(e2e): log a notice when runners skip Rust e2e tests

Emit a GitHub Actions notice from the Docker, Podman, Kubernetes, and VM
runners when e2e-conformance-only skips the Rust tests, so a narrowed lane
is visible in the job summary instead of silently reporting success.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* docs(testing): move e2e conformance-only selector to shared section

The selector applies to all four driver runners, so document it with the
Rust CLI e2e suites instead of under the Kubernetes task variants.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

---------

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Co-authored-by: Matthew Grossman <mgrossman@nvidia.com>
…teway traits (NVIDIA#4619)

* perf(server): drop async_trait from mutation replay

Mutation is never used as a trait object, so the boxed futures from
#[tonic::async_trait] buy nothing. Declare its methods as
-> impl Future + Send instead. This keeps the Send guarantee, removes a
heap allocation per call, and cuts a cold openshell-server lib build from
~60s to ~36s.

Box the approve_all_draft_chunks future at its call sites, since it is no
longer boxed implicitly and trips clippy::large_futures.

Closes NVIDIA#4618

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* perf(supervisor): drop async_trait from generic-only gateway traits

StartupGateway, PolicyGatewayClient, and ReportClient are only used
through generic parameters, never as trait objects. Declare their methods
as -> impl Future + Send instead of boxing each future.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

---------

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…3898)

* feat(isolation): publish backend protocol contracts independently of runtime

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(isolation): expose trusted launch and registry data contracts

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(isolation): name the required capability sandbox

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(sandbox): align protocol and compute capability names

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(sandbox): finalize public protocol contract names

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(isolation)!: activate backend contracts in sandbox runtime and driver selection

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(sandbox): simplify runtime and fix backend example

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(examples): organize WASI backend in Rust

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(sandbox): defer protocol conformance fixtures

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(examples): add supervised V8 backend demo

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(examples): stream V8 guest output with default policy

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(examples): handle V8 rejections and roll back failed launches

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(examples): align V8 crypto dependencies with CI AWS-LC

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(sandbox): represent Sandbox Protocol operations in protobuf

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(isolation): replace backend traits with RPC clients

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(proto): remove legacy isolation protocol reservations

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(proto): clarify Sandbox Protocol names

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(isolation): clarify authentication bootstrap and supervisor APIs

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(isolation): simplify policy enforcement confirmation

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): rotate launch generations and stabilize retry sessions

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(sandbox): remove redundant backend adapters and TCP flags

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): repair Docker restarts and macOS lint

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(isolation): simplify RPC contracts and backend registration

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): deliver TCP denials before closing the RPC

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
…vents (NVIDIA#4655)

Authenticators now return the mechanism and a reason category with the
status they reject a request with, and the gateway router records both
in the Authentication event. Callers still receive the original status.

Refs NVIDIA#2912

Signed-off-by: Kris Hicks <khicks@nvidia.com>
…#4660)

* fix(agents): validate sandbox names before Gator launch setup

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* chore(agents): remove standalone launcher test script

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

---------

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
…on (NVIDIA#4320)

Supervisors poll the gateway for configuration every 10 seconds. This PR adds an opt-in push mode (config_delivery_mode = "push"): supervisors still start by polling, and once their ConnectSupervisor session is accepted, the gateway delivers every later change over it and supervisors apply it and report the result. poll stays the default, and older supervisors and gateways keep polling.

---------

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: divesh <dgude@nvidia.com>
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
…ts (NVIDIA#4673)

Authentication events now carry the connection's immediate peer as
src_endpoint, or edge-tunnel for WebSocket-tunneled requests, and the
request's x-request-id as metadata.correlation_uid when it is a UUID.

Refs NVIDIA#2912

Signed-off-by: Kris Hicks <khicks@nvidia.com>
OCSF 1.8 requires type_id on the os object, but device.os carried only
a name. Events now report Linux (200), Windows (100), macOS (300), or
Other (99) with the platform name as the type label. The os object
schema is vendored so tests validate it.

Refs NVIDIA#2912

Signed-off-by: Kris Hicks <khicks@nvidia.com>
Closes NVIDIA#4253

Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* fix(tmachine): link libstdc++ statically and verify static musl binaries

On macOS hosts, the aarch64 musl openshell-prover that
build-artifacts-binaries produced was not static. z3-sys asks for a
dynamic libstdc++ (-Wl,-Bdynamic -lstdc++), and the musl cross compiler
wrapper resolved libstdc++.so. The result was a static executable with a
DT_NEEDED entry for libstdc++.so.6 and no interpreter, which crashed
during C++ static initialization. The local Debian package from NVIDIA#4598
ships this prover without running it.

Build the musl binaries with CXXSTDLIB=static:-bundle=stdc++ so z3-sys
requests a static libstdc++. CXXSTDLIB applies to every target and
Darwin has no libstdc++, so set it only on this musl build rather than in
the shared toolchain environment. The bundled form, static=stdc++, fails
because rustc cannot find libstdc++.a on its own search path.

Run tasks/scripts/verify-static-binary.sh on the musl CLI, sandbox and
prover, as the release does, so a binary that is not fully static fails
the local build.

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* feat(tmachine): build local RPM artifacts

Add build-artifacts-rpm, which builds the CLI and gateway RPMs that
tmachine's rpm installer expects at artifacts/packages/rpm/openshell.rpm
and artifacts/packages/rpm/openshell-gateway.rpm. It runs
`packit build locally` with openshell.spec in the Fedora 44 image that
build-rpm.yml uses, for the host's Linux architecture, and packages the
binaries staged by build-artifacts-binaries. The package version defaults
to the development version that branch E2E CI computes.

The aggregate build-artifacts app runs the RPM step after the DEB step.

Signed-off-by: Evan Lezar <elezar@nvidia.com>

---------

Signed-off-by: Evan Lezar <elezar@nvidia.com>
* ci(release): promote qualified prereleases to stable

Signed-off-by: Simon Scatton <sscatton@nvidia.com>

* fix(release): preserve qualified images during stable publication

Signed-off-by: Simon Scatton <sscatton@nvidia.com>

* ci(release): run shared release jobs with Nix

Signed-off-by: Simon Scatton <sscatton@nvidia.com>

* fix(release): recheck latest before publishing shared channels

Signed-off-by: Simon Scatton <sscatton@nvidia.com>

---------

Signed-off-by: Simon Scatton <sscatton@nvidia.com>
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
…4719)

* ci(release): name the failed stage and failing jobs in pre-release failure notices

The pre-release failure notification said only that the run failed.
Name the failed stage (build, publication, or release qualification),
link the tag, and list each failing job of the run attempt with a link.
Skip the aggregate Release Qualification job unless it is the only
failure, and still send the message if the jobs lookup fails.

Closes NVIDIA#4718

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* ci(release): simplify the failure notice to a list of failed jobs

Drop the stage headline and the aggregate-job filtering. Keep the
existing message and append a link to each failed job.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

---------

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…VIDIA#4578)

helm push uploads the chart manifest by digest, then fetches it back to
apply the version tag. GHCR sometimes returns 404 for a manifest it just
accepted, failing the push with 'failed to perform "Tag" on
destination: <digest>: not found'. This failed Release Dev on Oct 5 and
the v0.1.5-pre.2 Release Tag run after qualification passed.

Retry each chart push up to four times with a short backoff. A retry
finds the blobs and manifest already present and only applies the tag.

Closes NVIDIA#4577

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…and refactor AWS-LC (NVIDIA#3351)

* refactor(crypto): centralize AWS-LC behind backend interface

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* refactor(crypto): route new host proxy tests through facade

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): make backend boundary check portable

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): route Google JWT signing through facade and verify context dispatch

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* refactor(crypto): propagate digest backend failures

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* refactor(crypto): route Vault TLS fixtures through facade

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* refactor(crypto): adapt isolation runtime callers after main rebase

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): declare sandbox dependency and remove boundary lint

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* refactor(crypto): support independent backends and backend-free CA import

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* chore(crypto): refresh feature suite lockfile after rebase

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): route runtime integrity and entropy through backend

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* chore(crypto): refresh lockfile after upstream rebase

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): route VM socket entropy through backend after rebase

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* docs(crypto): document architecture and contributor boundaries

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* test(crypto): adapt upstream fixtures to backend facade

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* test(crypto): route relay TLS fixture through facade

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* test(crypto): adapt service bearer TLS fixture after rebase

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): route provider profile TLS initialization through facade

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): adapt upstream VM and supervisor paths to facade

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* ci(crypto): enforce backend independence in branch checks

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* test(crypto): adapt upstream token grant TLS fixture

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): avoid format collect in executable hashing

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): address provider selection and decryption review

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): adapt upstream operator TLS paths after rebase

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): adapt upstream TLS fixtures and protocol tests

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

---------

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
* ci(release): generate PR-based stable release notes

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(release): clarify PR release note length

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* ci(release): route note generation through inference hub

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* ci(release): configure inference endpoint and model with actions variables

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* ci(release): add a notes-only workflow preview

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* ci(release): retain prerelease notes as advisory artifacts

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(ci): shorten release notes workflow overview

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(release): resolve note comparisons within the generator

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(ci): simplify release notes workflow setup

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* ci: refine release notes guidance and upgrade section

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* ci: prioritize compatibility requirements in release notes

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* ci: simplify public compatibility release guidance

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* ci: clarify release-facing upgrade guidance

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(server): warn on dangling provider refs at startup

Signed-off-by: Eric Curtin <eric.curtin@docker.com>

* fix(server): load providers concurrently at startup

Signed-off-by: Eric Curtin <eric.curtin@docker.com>

---------

Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* fix(ocsf): report https scheme for TLS-terminated requests

Closes NVIDIA#4562

Signed-off-by: Eric Curtin <eric.curtin@docker.com>

* fix(ocsf): use transport scheme on passthrough relay

Signed-off-by: Eric Curtin <eric.curtin@docker.com>

---------

Signed-off-by: Eric Curtin <eric.curtin@docker.com>
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* fix(snap): only enable gateway user service for one user

Signed-off-by: Oliver Calder <oliver.calder@canonical.com>

* docs(snap): correct snap docs for the gateway user service

Signed-off-by: Oliver Calder <oliver.calder@canonical.com>

* fix(snap): add documentation of global to per-user enablement migration

Signed-off-by: Oliver Calder <oliver.calder@canonical.com>

---------

Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
…ance (NVIDIA#4676)

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
…#4195)

* fix(vm): flush guest filesystem before a planned sandbox stop

A planned stop of a MicroVM sandbox ended the VM before the guest wrote its dirty pages to disk, so files written shortly before the stop were missing after the next start.

The driver closed the host supervisor's parent-liveness pipe before it sent SIGTERM. The supervisor exits at once on that end-of-file, so it never ran its orderly shutdown: terminate the workload, then ask the guest to terminate. Signal the supervisor first, keep the pipe open until it has exited, and give it a grace period derived from the supervisor's own bounds and kept below the gateway's reclaim deadline.

In the guest runtime, syncfs the root filesystem after the process tree is terminated and before the termination is acknowledged, bounded to 3 seconds and best effort. Only the VM backend, which uses a vsock boundary listener, flushes; the container drivers share this runtime and are unchanged.

A sandbox whose main process has already exited is not covered.

Closes NVIDIA#4185

Signed-off-by: Shiju <shiju@nvidia.com>

* fix(vm): warn about unconfirmed flush after supervisor kill

Preserve the supervisor exit status so a planned stop warns when a signal
killed the supervisor and guest writeback was not confirmed. Keep normal
workload exit codes quiet, including 143 after a handled SIGTERM.

Extend the existing process tests to check the warning after a forced kill
and its absence after an orderly shutdown.

Refs NVIDIA#4185

Signed-off-by: Shiju <shiju@nvidia.com>

* fix(vm): satisfy shutdown timeout control-flow lint

Use if let for the single successful timeout pattern while preserving
the supervisor exit status and forced-kill error handling.

Signed-off-by: Shiju <shiju@nvidia.com>

* fix(vm): preserve shutdown merge resolutions during rebase

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* docs(vm): explain planned stop durability and flush limits

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

---------

Signed-off-by: Shiju <shiju@nvidia.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>
EmilienM and others added 2 commits October 29, 2026 23:50
…3-followups

CARRY: feat(odh): pin runtime images by digest and add Renovate coverage
FrostGod and others added 21 commits October 30, 2026 05:11
* fix(supervisor): batch buffered log replay after reconnect

Signed-off-by: divesh <dgude@nvidia.com>

* fix(gateway): accept the full reconnect log buffer

Signed-off-by: divesh <dgude@nvidia.com>

* fix(logging): batch reconnect replay and warn on truncation

Signed-off-by: divesh <dgude@nvidia.com>

---------

Signed-off-by: divesh <dgude@nvidia.com>
…gest to 5f08e80

Signed-off-by: red-hat-konflux <126015336+red-hat-konflux[bot]@users.noreply.github.com>
Signed-off-by: red-hat-konflux <126015336+red-hat-konflux[bot]@users.noreply.github.com>
…tmaker/main/registry.access.redhat.com-ubi9-ubi-9.8

chore(deps): update registry.access.redhat.com/ubi9/ubi:9.8 docker digest to 5f08e80
Signed-off-by: Bobbins228 <mcampbel@redhat.com>
Move ubi-minimal, ubi-micro and nodejs-24-minimal to current digests so runtime images ship ca-certificates 2026.2.92_v9.0.319 and Node 24.22.1, which distrust the compromised eMudhra intermediates.

Signed-off-by: Andre Lustosa <alustosa@redhat.com>
…udhra-ca/andre-motta

CARRY: fix(konflux): bump UBI 9.8 bases for eMudhra CA distrust
…obs (NVIDIA#5036)

Remove the OCSF schema downgrade. OpenShell now writes native OCSF 1.8.0
records to supervisor JSONL and stderr, gateway JSONL, and OTLP log
export regardless of configuration.

The downgrade did not produce valid records: events it labelled 1.1 or
1.3 failed validation against those versions, because OpenShell's
emitters rely on requirements that only 1.8 relaxed (NVIDIA#4284). It was also
not shown to be required by any SIEM: of ten sinks and pipelines tested
with real OpenShell records, none rejected native 1.8.0 or gated on
metadata.version.

The version knobs stay accepted so existing configuration still loads,
but they are deprecated and ignored:

- The ocsf_schema_version sandbox setting accepts any value; the
  supervisor reports a newly set value as a Device Config State Change
  event.
- schema_version under [openshell.gateway.ocsf_log] and
  [openshell.gateway.otlp.ocsf], and the server.ocsfLog.schemaVersion
  Helm value, accept any value; the gateway logs a warning at startup.
- The compute drivers' OPENSHELL_OTLP_OCSF_SCHEMA_VERSION flag and
  environment variable are hidden and ignored, with a warning, and are
  no longer forwarded to sandboxes.

BREAKING CHANGE: Deployments that set an OCSF schema version now receive
native OCSF 1.8.0 records instead of records labelled 1.1 or 1.3.

Closes NVIDIA#4284

Signed-off-by: Kris Hicks <khicks@nvidia.com>
Signed-off-by: Bobbins228 <mcampbel@redhat.com>
…VIDIA#4843)

* feat(crypto): add an opt-in system OpenSSL backend

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* ci(crypto): enforce crypto dependency boundaries with cargo-deny

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): reject encrypted PEM keys without prompting

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): validate OpenSSL keys and propagate TLS setup errors

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): address OpenSSL backend review findings

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* test(crypto): verify OpenSSL integration and linkage

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): reject inconsistent RSA private keys

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(ci): isolate OpenSSL checks and clear deny wrappers

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

---------

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
…es (NVIDIA#4873)

* feat(gateway): add storage-preserving sandbox image updates

Add image-only UpdateSandbox with conditional versions, durable replay, fenced pending intent, and Kubernetes storage retention. Enable non-destructive image repair after completed timeout cleanup and exercise both Kubernetes controller API lanes.

Closes NVIDIA#4623

Signed-off-by: sjenning <493891+sjenning@users.noreply.github.com>

* feat(gateway): add storage-preserving sandbox image updates

OpenClaw-Publication: b2036b09-b697-4d65-aba9-5c96a3198f9c
Signed-off-by: sjenning <493891+sjenning@users.noreply.github.com>

* fix(gateway): preserve image update patch-release compatibility

Move image updates to a separate authenticated service and retain the baseline public sandbox status. Persist image intent and revisions independently with atomic baseline-known lifecycle fencing and explicit post-upgrade activation.

Preserve concurrent provisioning clocks, fence replaced attempts and delayed callbacks, and verify private-state transactions plus both Kubernetes controller API lanes.

Signed-off-by: sjenning <493891+sjenning@users.noreply.github.com>

* fix(gateway): preserve optional driver compatibility and repair CI

Move image updates to a separate optional compute-driver service, preserving the baseline generated trait and streaming associated type. Add source and TCP/UDS transport regressions, capability negotiation coverage, and a default-unsupported in-process extension.

Keep Go compatibility tests outside generated output and update production reflection expectations for the separate sandbox update service.

Signed-off-by: sjenning <493891+sjenning@users.noreply.github.com>

* fix(gateway): preserve baseline factory and descriptor APIs

Restore the original in-process driver trait object and existing instance enum. Register optional image updates through a default factory hook without requiring changes to existing drivers or factories, and require registration plus capability before mutation.

Accept OpenShell-only descriptors while retaining strict validation of present update services. Cover baseline registration, missing-updater no-write behavior, descriptor validation, and both live Kubernetes retention lanes.

Signed-off-by: sjenning <493891+sjenning@users.noreply.github.com>

* chore(ci): use upstream Kubernetes E2E defaults

Drop the task-level fallback superseded by main's runner defaults and explicit e2e-conformance-only selector. Preserve current upstream task definitions and recovery waiver unchanged.

Signed-off-by: sjenning <493891+sjenning@users.noreply.github.com>

* fix(gateway): integrate image updates with backend protocol

Adapt non-launch image updates to the upstream remote admission API while retaining resource-policy verification before forwarding. Cover both negotiated extensions and preservation of backend pins and persisted launch identity.

Signed-off-by: sjenning <493891+sjenning@users.noreply.github.com>

* fix(gateway): remove sandbox image state on deletion

Signed-off-by: Gordon Sim <gsim@redhat.com>

* feat(gateway): expand UpdateSandbox configuration updates

Signed-off-by: Gordon Sim <gsim@redhat.com>

* fix(sandbox): recover pending updates and address review feedback

Signed-off-by: Gordon Sim <gsim@redhat.com>

* perf(server): index pending sandbox updates for recovery

Signed-off-by: Gordon Sim <gsim@redhat.com>

* docs(sandbox): clarify stop behavior when updates are disabled

Signed-off-by: Gordon Sim <gsim@redhat.com>

* fix(server): scope deletion bypass to the matching sandbox update

Signed-off-by: Gordon Sim <gsim@redhat.com>

* fix(server): back off pending sandbox update recovery

Signed-off-by: Gordon Sim <gsim@redhat.com>

* ci: restore combined-chart Kubernetes e2e coverage

Signed-off-by: Gordon Sim <gsim@redhat.com>

* test(kubernetes): use uuid for update request IDs

Signed-off-by: Gordon Sim <gsim@redhat.com>

* fix(kubernetes): reject oversized annotations before accepting sandbox updates

Signed-off-by: Gordon Sim <gsim@redhat.com>

* docs(skills): correct pending sandbox update recovery guidance

Signed-off-by: Gordon Sim <gsim@redhat.com>

* fix(sandbox): preserve terminal results when stopping missing compute

Signed-off-by: Gordon Sim <gsim@redhat.com>

* fix(server): bound sandbox update attempts to unblock recovery and deletion

Signed-off-by: Gordon Sim <gsim@redhat.com>

* fix(server): remove unintended sandbox label count limit

Signed-off-by: Gordon Sim <gsim@redhat.com>

* fix(kubernetes): check known pod names during sandbox update preflight

Signed-off-by: Gordon Sim <gsim@redhat.com>

* docs: clarify sandbox update permissions and driver support

Signed-off-by: Gordon Sim <gsim@redhat.com>

---------

Signed-off-by: sjenning <493891+sjenning@users.noreply.github.com>
Signed-off-by: Gordon Sim <gsim@redhat.com>
Co-authored-by: sjenning <493891+sjenning@users.noreply.github.com>
…nvidia-main/Bobbins228

chore(sync): merge NVIDIA OpenShell main into ODH
Generate and package gRPC type stubs with concrete request and response types. Verify consumer typing, native async transport behavior, and wheel contents while preserving the existing synchronous APIs.

Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
…DIA#4947)

In a git worktree .git is a file, so the hard-coded ../../.git/* rerun-if-changed
paths never exist and cargo rebuilds openshell-core and every dependent crate
on each invocation. Resolve HEAD and logs/HEAD from the git dir and tag refs,
packed-refs, and the checked-out branch ref from the common git dir, and only
emit paths that exist.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
…VIDIA#4744)

* refactor(gator): separate runtime assignments from tooling images (WIP)

Signed-off-by: John Myers <johntmyers@users.noreply.github.com>

* feat(agents): support Kubernetes image delivery and reusable tooling

Signed-off-by: John Myers <johntmyers@users.noreply.github.com>

* fix(agents): use pushed Podman manifest digest

Signed-off-by: John Myers <johntmyers@users.noreply.github.com>

---------

Signed-off-by: John Myers <johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
NVIDIA#5065)

* feat(cli): support fixed callback port for browser login in containers

Browser login (OIDC Authorization Code + PKCE and the edge/mTLS browser
flow) binds its callback listener to an ephemeral port on 127.0.0.1.
When the CLI runs inside a container, the wrapper cannot know a random
port in advance, so it cannot publish it to the host and the browser
never reaches the callback.

Add a shared bind helper honoring two environment variables:

- OPENSHELL_CALLBACK_PORT pins the listener to a fixed port (validated
  integer 1..=65535) so a wrapper can publish
  -p 127.0.0.1:<port>:<port> before login starts.
- OPENSHELL_CALLBACK_BIND_ADDRESS overrides the bind address (validated
  IP address) so user-mode container networking (pasta/slirp4netns),
  which can only reach 0.0.0.0 listeners, can forward to it.

Defaults are unchanged: with both variables unset the listener binds
127.0.0.1:0 exactly as before, and the bind address never widens to
0.0.0.0 without the explicit variable, since a wildcard bind on a host
machine would expose the callback server to the local network. The
browser-facing redirect URL stays http://127.0.0.1:<port>/callback
even for a wildcard bind, so no identity-provider allowlist change is
needed. A busy fixed port or an invalid value fails with a clear error
instead of falling back silently.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(auth): support IPv6 callback targets in browser login flows

An IPv6-only callback bind (OPENSHELL_CALLBACK_BIND_ADDRESS=::1) was
accepted but both browser flows still directed callbacks at
127.0.0.1, so the listener was unreachable and login timed out.

Route callbacks to the address the listener actually bound:

- The CLI derives the browser-facing callback target from the bound
  address. Wildcard binds map to the matching loopback (0.0.0.0 to
  127.0.0.1, :: to [::1]) because a wildcard listener accepts
  loopback connections of its family; IPv6 addresses render
  bracketed per RFC 3986/RFC 8252.
- The OIDC redirect URI therefore becomes
  http://[::1]:<port>/callback for ::1-bound listeners, and the
  bundled Keycloak realm now allows http://[::1]:* redirects for
  the CLI client alongside the existing IPv4 entries.
- The edge flow passes the unbracketed loopback literal to the
  gateway connect page as callback_host. The page strictly
  validates it to a loopback IP literal (anything missing,
  non-loopback, or malformed falls back to 127.0.0.1) before
  embedding it in the fetch URL and CSP connect-src, so a crafted
  parameter can never direct the embedded edge token to an
  external host. The host is preserved across the waiting-page
  refresh, and older gateways that ignore the new parameter keep
  the historical 127.0.0.1 behavior.

Defaults are unchanged: the listener still binds 127.0.0.1 on an
ephemeral port unless the environment variables say otherwise.

Adds a callback-reachability test (connecting to the advertised
[::1] target succeeds while 127.0.0.1 is refused for a ::1 bind),
host-derivation table tests, OIDC redirect tests for ::1 and ::
binds, edge URL tests, and server-side validation and render tests.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* docs(auth): scope IPv6 support around Chrome's CSP limitation

Review feedback on b676200: Chrome does not support IPv6 literals
in CSP host-source expressions, so the edge flow's connect page
blocks the callback POST to http://[::1]:<port> with a connect-src
violation even though the policy allows it. The CLI listener is
reachable; the browser refuses to fetch.

Keep the callback routing as-is (correct per the CSP specification,
and the failure mode is an explicit page error rather than a hang)
and scope the claim precisely instead:

- The connect page CSP always keeps the IPv4 loopback source so a
  dropped IPv6 source expression cannot degrade the policy to empty.
- The auth reference and PR description now state that IPv6
  callback binds are fully supported for OIDC login but are
  best-effort for edge login until browsers implement IPv6 CSP
  source expressions; all IPv4 paths, including the container use
  case, are unaffected.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

---------

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
…ing with Keycloak (NVIDIA#4891)

* feat(testkit): pass extra environment variables to the CLI

Add Cli::run_with_env and Context::cli_with_env so tests can hand
credentials to the CLI by name instead of on its command line.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* feat(tmachine): add an Ubuntu Docker environment that runs Keycloak

Environment setup can now take inputs, like installers do, and hashes
them only when present so existing cached disks stay valid.
ubuntu-docker-rootful-keycloak runs the Docker setup plus a Keycloak
playbook that starts the container with a restart policy, since setup is
cached as a disk, and writes how to reach it as OPENSHELL_TEST_* lines in
/etc/openshell-test/env.d/keycloak.env.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* feat(test-runner): pass the variables an environment publishes to tests

After boot, read every /etc/openshell-test/env.d/*.env file, accept only
OPENSHELL_TEST_* names, and add the variables to the testkit-ready and
nextest commands through guest_env. Errors name the key, never the value.
The runner doesn't know which services an environment runs; suites read
the variables, so they also run against a deployment that sets them.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(features): port Keycloak provider refresh to testkit

Add a keycloak features crate with a suite-local KeycloakContext fixture
that reads OPENSHELL_TEST_KEYCLOAK_*, waits for the realm to answer, and
provides password grants, revocation and the token endpoint. Port the
provider refresh test onto it. Align the features lockfile with the root
and add it to the alignment check. The old provider-refresh crate stays
until the required lanes switch.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* ci(tests): run the Keycloak features suite in E2E Suites

Add a matrix row for features/keycloak on ubuntu-docker-rootful-keycloak,
include the Keycloak realm in the tmachine disk cache key, and document
environments that provide services in TESTING.md.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* docs(testing): describe how to add a feature suite

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* refactor(tmachine): publish test variables through a shared task file

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(test-runner): keep published test variables out of SSH commands

The runner passed the variables in env.d on the nextest command line, so
an SSH timeout or launch failure printed their values, which can be
credentials, in the error context. The runner now only checks the files;
the guest shell exports them before starting nextest, so only the gateway
name and CLI path appear in the command.

Lines are read literally on both sides: no trimming, and only empty lines
and lines starting with # are skipped.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(features): fit Keycloak readiness in the per-test timeout

The fixture waited up to three minutes for Keycloak inside the test, but
the runner's nextest profile kills a test after two. Wait up to 90 seconds
so a slow Keycloak fails with a readiness error instead.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(test-runner): run the export script test with shell builtins only

The Nix build runs the runner's tests in a sandbox without env on PATH,
so the test failed there and the runner didn't build. Print the exported
variables with printf instead.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* refactor(features): name the Keycloak suite after what it tests

The suite was named after the service it needs rather than what it
tests. Rename it to keycloak-provider-refresh and move the Keycloak
fixture into a support crate at tests/support/keycloak, so other suites
that need an OIDC issuer can depend on it. TESTING.md now asks for a
descriptive suite name instead of prescribing one. Lint and format the
new crates in CI and mise.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(features): bound each Keycloak request in the fixture

poll_until checks its deadline only between attempts, and curl had no
time limit, so a Keycloak that accepts connections but stalls could hold
one attempt until nextest killed the test. Give each request 10 seconds.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* ci(tests): key the tmachine disk cache by environment and installer

Both E2E suites rows computed the same cache key. Only the first job to
save it got its disks in, and actions/cache never saves after an exact
hit, so the other row rebuilt its setup and install layers on every run.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(testkit): keep the gateway selection over caller CLI env

run_with_env applied the caller's variables last, so one named like the
gateway, endpoint or workspace variable could silently point the CLI at
another gateway. Apply them first so testkit's selection wins.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(features): leave the test body room after Keycloak readiness

poll_until checks its deadline only between attempts, so a 90 second
wait could run for about 102 seconds with the request time limit and the
poll interval, leaving the test body under 20 seconds of the 2 minute
budget. Wait 75 seconds and spell out the margin.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* ci(tests): include config.nix in the tmachine disk cache key

tmachine's setup hash covers each environment's playbook list and setup
inputs, which config.nix declares. Changing only config.nix left the
cache key unchanged, so the rebuilt layers were never saved again.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* docs(testing): add the lint step for new feature suites

CI formats and lints feature suites one package at a time, so a suite
added by following TESTING.md got no fmt or clippy coverage. Also say
what test_env actually guarantees about published values.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(features): report Keycloak suite cleanup failures

The provider and profile cleanup discarded both errors and failed exit
codes, so leftovers went unnoticed. Print them without failing the test.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

---------

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
konflux-internal-p02 Bot and others added 5 commits October 31, 2026 02:23
Signed-off-by: konflux-internal-p02 <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
Co-authored-by: konflux-internal-p02[bot] <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
NVIDIA#5089)

Keycloak 24 rejects http://[::1]:* (and every other port wildcard on an
IPv6 literal) when importing the realm, so it fails to start with
"Invalid client openshell-cli: A redirect URI is not a valid URI". That
broke every lane that loads scripts/keycloak-realm.json, including the
features/keycloak suite and the OIDC Docker E2E jobs.

Restore the IPv4 loopback and localhost entries only, and correct the
authentication docs: IPv6 callback binds against the bundled realm need
an exact http://[::1]:<port>/callback redirect URI.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment