Skip to content

feat(dgw): run AI analysis of a recording as a provisioner task - #2044

Open
irvingouj@Devolutions (irvingoujAtDevolution) wants to merge 50 commits into
masterfrom
feat/recording-ai-analysis
Open

irvingouj@Devolutions (irvingoujAtDevolution) wants to merge 50 commits into
masterfrom
feat/recording-ai-analysis

Conversation

@irvingoujAtDevolution

@irvingoujAtDevolution irvingouj@Devolutions (irvingoujAtDevolution) commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

AI analysis of a recording, started by DVLS with a plain HTTP request (no new token type), tracked as a provisioner Task and run as a job on the existing job queue.

Provisioner tasks run through a ProvisionerTaskRunner, the one place that keeps a task's record (#2041's ProvisionerTaskRecord) and its job in step. Each kind of work is a ProvisionerTask (what one attempt does, what to drop once its record ends); the runner creates the record, queues its job, runs each attempt with its timeout, retries transient failures and records how the Task ends. The Task is the source of truth and the job only wakes it up: each Task names the one job allowed to run it (#2041), so a stale or duplicate job does nothing. A Task is reported created only once its job is stored in the job queue. At startup, before the job runner claims anything, an unfinished Task whose job is no longer stored (such as after a crash between recording the Task and storing its job) gets a new one; a Task whose job is still stored keeps it. The analysis itself is the AI crate's analyze_recording (#2043); Gateway hands it the manifest of the finished recording and the language of the request, and adds the log it returns to the session as its own file.

API (stable, no enable_unstable)

  • POST /jet/jrec/{session_id}/ai-analysis, scope gateway.tasks.recording-ai-analysis.start. Body: taskId, provider, model, baseUrl (needed for openai-compatible), maxOutputTokens, language, apiKey.
    • 202 + Task when created, 200 + Task for the same request again
    • 400 bad settings, body or language, 404 no recording
    • 409 task_id_conflict / recording_active (a clip is still pushed) / analysis_in_progress (+ the running taskId)
  • GET /jet/jrec/{session_id}/ai-analysis/{task_id}, scope gateway.tasks.recording-ai-analysis.read: the Task (state, payload, attempts, times), or 404. The 202 answer of the POST names it in its Location header. A Task of another kind, or the analysis of another session, is not found under that path.

Each provisioner task declares its own scopes, gateway.tasks.<kind>.start and gateway.tasks.<kind>.read, so a token for one kind of Task can neither start nor read another kind. The kind of AI analysis is recording-ai-analysis.

The two new scopes in Devolutions.Gateway.Utils and the regenerated OpenAPI clients are in #2054, stacked on this PR.

The first commit hands the provisioner task store (#2041) to DgwState on its own gateway.db connection; DgwState::mock becomes async for that, hence the .await in a few tests. It also passes job_queue::MAX_ATTEMPTS (5, same as the current default) to the job queue explicitly, so the retry check in the AI analysis job can't drift from it.

How it behaves

  • The API key and base URL live in memory only, encrypted, as the Task's secret in the SecretStore (feat(dgw): add an encrypted in-memory secret store #2058); never in the job, gateway.db or our logs. After a Gateway restart the Task fails with "the API key is no longer available".
  • Transient provider errors are retried by the job queue (up to 5 attempts); permanent ones fail the Task. 30 min per attempt, 2 h per Task.
  • Terminal (.cast, .trp) and video (.webm) recordings are analyzed (feat(dgw): analyze any recording in the AI crate #2043); video needs XMF v2026.9.25.0 or later.
  • language is any text of 1 to 10 characters, such as fr-CA or français; anything else is a 400. Without it the analysis is in English. It's part of the Task parameters, so the same taskId with another language is a task_id_conflict.
  • The result is its own log, ai-analysis-0.slog under artifacts.ai-analysis, with the summary on its session.end (feat(ai): write the AI analysis in the requested language, with a summary #2118). Gateway reads nothing the producer pushed.
  • Every attempt of a new Task first takes the session lock and removes the earlier AI analysis, even a succeeded one. So a session holds at most one, and a Task that fails, even on a lost key, leaves none.
  • The Task params keep the AI settings without the key and the base URL, plus the language; the OpenAPI document describes them as AiAnalysisParams.
  • The Task payload names the log, the model, the prompt versions and the token usage.

Where to look hardest

  • provisioner_tasks/runner/mod.rs: run_attempt (cancel on timeout, retry vs. fail, on_end on every end) and resume (called from service.rs before the job runner starts)
  • provisioner_tasks/recording_ai_analysis/mod.rs: the AI ProvisionerTask, and lock_session: the working files belong to the session, so analyses of one session run one at a time (a Task past its deadline may still have an attempt running when the next Task starts)
  • start() in the same file: the order of the checks

Known, accepted

  • At VerbosityProfile All, reqwest/hyper's own TRACE lines show the provider's address (not the key). Our code never logs it.
  • If Gateway crashes right after adding the log to the manifest, and before moving it in, the manifest lists a missing ai-analysis-0.slog.
  • A Task that fails before any attempt runs (its job can't be queued, or no time is left for one) keeps the earlier analysis, since only an attempt holds the session lock.

Tests

  • cargo test -p devolutions-gateway --lib: 249 passed.
  • cargo test -p testsuite --test integration_tests ai_analysis: 10 passed. These endpoint tests drive the Gateway binary with a mock AI provider, and check that the key and the base URL reach neither the logs nor gateway.db. The testsuite gains a time dev-dependency to check the RFC 3339 times.
  • clippy (--features openapi --tests) and nightly rustfmt are clean.
  • Not run against a real AI provider at this head.

🤖 Generated with Claude Code

@irvingoujAtDevolution irvingouj@Devolutions (irvingoujAtDevolution) changed the title feat/recording ai analysis feat(dgw): run AI analysis of a recording as a provisioner task Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

Let maintainers know that an action is required on their side

  • Add the label release-required Please cut a new release (Devolutions Gateway, Devolutions Agent, Jetsocat, PowerShell module) when you request a maintainer to cut a new release (Devolutions Gateway, Devolutions Agent, Jetsocat, PowerShell module)

  • Add the label release-blocker Follow-up is required before cutting a new release if a follow-up is required before cutting a new release

  • Add the label publish-required Please publish libraries (`Devolutions.Gateway.Utils`, OpenAPI clients, etc) when you request a maintainer to publish libraries (Devolutions.Gateway.Utils, OpenAPI clients, etc.)

  • Add the label publish-blocker Follow-up is required before publishing libraries if a follow-up is required before publishing libraries

@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2045 October 6, 2026 21:26
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2046 October 7, 2026 20:36
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2049 October 7, 2026 22:19
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2051 October 7, 2026 22:20
The caller of `POST /jet/jrec/{session_id}/ai-analysis` now picks the
language in an optional `language` field: any text of 1 to 10
characters, such as `fr-CA` or `français`; anything else is a 400.
Without it, the analysis is in English. The language is part of the
Task parameters, so the same Task ID with another language is a
`task_id_conflict`.

Gateway no longer reads anything the producer pushed: there is no
events log to take a `locale` from.

- The tests count the summary request the AI crate now sends.
- "AI analysis queued" is a debug line, since the runner already logs
  the queued Task.
- The jrec and provisioner tasks intents state the language, how many
  analyses a session keeps, and how retries and restarts behave.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The working files of an analysis now always name their language, so a fixture without one is not a valid workspace.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Clearing on the first attempt only missed two paths. A Task that failed before its first attempt cleared, such as on a
lost key, kept the earlier analysis. A first attempt cancelled while waiting for the session retried as attempt 2 and
never cleared, so the session ended with two analyses. Each attempt now takes the session lock, then clears, before
anything else can fail.

The Task params keep the language as a `Language`, so it's checked once, when the request comes in.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…document

`params` of a `recording-ai-analysis` Task now has the `AiAnalysisParams` schema, like its payloads. The lint attributes
on these schema structs are gone, since the lint never fires on them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The tests move from the crate's own `tests/` to the testsuite, as AGENTS.md asks, and talk to a running Gateway and a
mock AI provider instead of linking Gateway internals. They read the key leak checks from the log files and
`gateway.db` in the config folder.

The check that a Task of another kind is not found at this endpoint is gone: the binary only knows one kind.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…st does

The full path made the OpenAPI document reference a schema that does not exist.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…a failed removal

A session without recording has no AI analysis to remove, so clearing it succeeds and the attempt then reports
"session has no recording".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Windows can't delete the database and log files a running Gateway holds open, so each test left its folder behind.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…er accepts

The server reads a `u32`, which an `int32` schema can't hold, so the clients now get a 64-bit integer from 0 to
4294967295.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The AI access of a Task now lives in the standalone `SecretStore`,
keyed by the Task ID, instead of the provisioning store, which is
about sessions only. Gateway holds one store in `DgwState`, hands it
to the provisioner tasks, and runs its expiry cleanup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Task secrets live in the standalone `SecretStore`, not in the
provisioning store, which is about sessions only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record the owner's rulings from the intent review of the AI analysis
Task: a new Task replaces the earlier analysis only once its success is
recorded, a failed Task leaves no file behind, attempt errors are
recoverable or permanent, token usage is always reported, a running
Task fails after 30 minutes without progress, and the base URL is a
secret like the API key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Task keeps its API key until its deadline, and that deadline now
moves while the Task makes progress, so the key must follow it.
An expired or removed secret stays gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Errors: an attempt fails with a closed kind and no text, recoverable
or permanent. A failed Task's payload is `{ error, usage }`, so the
caller words the error.

Usage: every payload but `queued` carries the tokens used so far, over
all attempts. Requests whose usage the provider did not report are
counted apart in `unreportedRequests`.

Time: a Task fails if it does not start within 2 hours. Once it runs,
each attempt start and each progress report moves its deadline 30
minutes ahead, and the runner cancels an attempt past it. Waiting for
a remux is not progress. The store fails an overdue Task with
`deadlinePassed` and keeps the usage of its last payload.

Success: the new log sits in the session unlisted until the success is
recorded. Ending the Task then lists it and drops the earlier analysis.
If Gateway stops in between, the job runs again at startup and redoes
that step. A failed Task removes everything it made and leaves the
earlier analysis in place.

The session manifest is written to a file of its own, then renamed
over the old one, so a crash never leaves half of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The end of a Task, such as listing its AI analysis log, ran from the
Task's own job, whose runs are shared with the attempts. Once those
runs were used up, a failed end was never done again, not even at the
next start.

A failed end now hands the Task to a new job that ends it again five
minutes later. The job is stored, so a restart runs it too.

Deadlines are also cut to whole seconds, as the store keeps them, so
the runner and the store agree on when one passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A request was counted only once its answer came. One cut off by the
deadline or a restart was never counted, so the usage looked complete
when it was not. Each request now counts as unreported from the moment
it is sent, and its reported tokens replace that count on answer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Task whose end failed could list its log later, over the analysis of
a newer Task of the session. A new Task now removes the logs earlier
Tasks left unlisted, so such a late end finds nothing to list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The manifest is written to a temporary file first. Its name now starts
with a dot, which jrec never serves, so a file a crash leaves behind is
never downloaded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The OpenAPI document writes the error kinds out by hand, since utoipa
drops the descriptions of variants without fields. An exhaustive match
now stops the build until a new kind is documented too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new Task removed every log earlier Tasks left unlisted. When the end
of a succeeded Task was still due, the new Task dropped an analysis
whose success was recorded, and a new Task that then failed left the
session without it.

A new Task now does what the end of the earlier Task would: it lists
the log of a succeeded Task, and removes the log of a failed or unknown
one. The runner gives an attempt the state of another record for this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the success of a Task could not be recorded, any error that came
after the failure was recorded was only logged. An end that could not
be queued again was then lost. That error now reaches the job queue,
whose retry finds the Task finished and only ends it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants