Skip to content

feat(dgw): page recording search with a cursor - #2137

Open
irvingouj@Devolutions (irvingoujAtDevolution) wants to merge 20 commits into
feat/recording-searchfrom
feat/recording-search-pagination
Open

irvingouj@Devolutions (irvingoujAtDevolution) wants to merge 20 commits into
feat/recording-searchfrom
feat/recording-search-pagination

Conversation

@irvingoujAtDevolution

@irvingoujAtDevolution irvingouj@Devolutions (irvingoujAtDevolution) commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Searching every recording doesn't fit in one response, so POST /jet/jrec/search now returns pages.

A page ends after reading about 16 MiB of logs or after maxHits hits, from 1 to 1000 and 1000 by default; 0 or more than 1000 is a 400. nextCursor replaces limitReached: sending the same query, recordings and startTime with it as cursor gets the next page, until a page has no nextCursor.

Gateway keeps no state between pages. The cursor holds the recording, file, byte offset and line number to resume at, plus a hash of the search, so a cursor sent with another search is a 400. A cursor whose position cannot exist is a 400 too, and its file name is only matched against the recording's manifest, never joined into a path. Pages never go back, so a recording that starts behind the cursor while paging isn't searched. The paging rules are in search.intent.md.

🤖 Generated with Claude Code

Issue: DGW-463

Add `POST /jet/jrec/search`, so DVLS can search the `.slog` logs of
recordings without downloading every log.

The body is `{ "query": "...", "recordings": "all" | ["<id>", ...] }`,
guarded by a new `gateway.recordings.search` scope. The existing
`gateway.recordings.read` only lists IDs, so reusing it would let every
read token see log contents.

The search reads the `.slog` files and AI analysis logs listed in
`recording.json`, with the rules of the log viewer's parser: a
case-insensitive substring over the shown fields and parameters,
entries the viewer drops are skipped, and long strings match on their
shown part only. Hits are the original lines. A search stops at 1000
hits, 16 MiB of hits or 256 MiB read, and reports it with
`limitReached`.

`api/jrec.rs` moves to `api/jrec/mod.rs` to hold the `search`
submodule.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The log viewer strips a trailing `\r` before it checks a line against
the 256 KiB limit, so a CRLF line of exactly 256 KiB is shown there.
The search now strips it first too, also on a last line without `\n`.

Mark the search invariants with `INVARIANT:` and give each recording
selection case its own test.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The derived schema put `"all"` in an `AllRecordings` schema of its
own, and the C# client generator (7.9.0) turns that `oneOf` member
into a constructor that does not compile. The schema now lists
`"all"` inline, like `MonitoringProbeType`; the JSON is unchanged.

Also test the line limit one byte over it and on a last line ending
with `\r`, and keep `INVARIANT:` for definitions only.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Scanning costs about 16 ms per MiB of logs, so a 256 MiB budget kept
one request busy for about 4 seconds. A search now reads at most
16 MiB.

Every hit is a line the search has read, so the read budget bounds the
response too, and the separate 16 MiB hit-bytes limit goes away.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Search no longer reads `.slog` lines the way the log viewer's parser
does. It matches the query, case-insensitively, against the raw text of
each line, so Gateway does not need to understand a log to search it
and any kind of artifact can be searched. It reads only the `.slog`
files listed under `artifacts` in `recording.json`.

A hit is now the line as a string, with its line number and byte
offset, since a line is not guaranteed to be JSON. A line over 256 KiB
is matched and returned on its first 256 KiB, marked `truncated`, and
the rest of it is skipped within the read budget, so a log without line
breaks cannot make one request read all of it.

The request takes an optional `startTime` range of RFC 3339 times,
matched on the start time in `recording.json`, with `from` included and
`to` excluded. Recordings that start together are ordered by ID, so the
order is the same on every request.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record what the search reads, how it matches, what a hit holds, who may
search, and its limits, with the reason behind each.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@irvingoujAtDevolution

irvingouj@Devolutions (irvingoujAtDevolution) commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Lines changed, against #2112:

Added Removed
Production code (search.rs) 245 63
Tests (search.rs unit tests, recording_search.rs) 334 25
Other (search.intent.md, hex for the testsuite) 14 3

@irvingoujAtDevolution

Copy link
Copy Markdown
Contributor Author

Intent drafts: handoff

The Pages section of search.intent.md replaces the Limits section of #2112. The rules come from the same interview, which read #2112 at 4de203f.

Provenance

Levels: 1 = my words verbatim, 2 = my words tidied, 3 = agent wording I adopted, 4 = agent fill-in, not intent.

# Commitment as written My words (if tidied) Level
1 Results come in pages, so that searching every recording takes a bounded time per request. "I efel like some sort of pagination is needed"; "there might be thouands of videos" 2
2 A page ends after reading about 16 MiB of logs or after maxHits hits, whichever comes first. "so one page, internally, for gateway to read is 16mb at most, but we return at max 1000 line as long as 16mb not hit, correct?" 2
3 maxHits is from 1 to 1,000 and defaults to 1,000; a value outside that range is rejected, never clamped. "reject 0 and > 1000, just reject"; "default 1000" 2
4 A page carries a nextCursor until every log is read, and a page with fewer hits than maxHits may still have one, since the read limit can end it first. "acceptable, we can do this" (cursor design) 3
5 Gateway keeps no state between pages; the cursor holds where the next page starts. "acceptable, we can do this" 3
6 A cursor is accepted only with the same query, recordings and startTime as the page that gave it. "acceptable, we can do this" (a hash of the request in the cursor) 3
7 The hit that does not fit in a page opens the next one, so that every page moves the search forward and returns each hit once. "acceptable, we can do this" 3
8 Pages never go back: a recording that starts, or a log that grows, behind the cursor while paging is not searched. "acceptable, we can do this" (no catch-up while paging) 3
9 A log replaced while paging, as an AI analysis rerun does, is read from the cursor's byte offset on, so lines can be missed or misnumbered; this is accepted for now. "let's keep it simple for today, just accept it" 2
10 "the same on every request" becomes "the same on every page". tidy only 3

Agent fill-in (level 4, not in the intent file): the cursor is hex-encoded JSON, not signed. A changed cursor can only move the read position inside the logs the same search reads, since its recording ID and file name are matched against the listing and never joined into a path.

Left out on purpose

  • Skip-N offset paging and server-side search sessions were rejected for the stateless cursor.
  • A file-version check in the cursor is not built, per commitment 9.

Placement

devolutions-gateway/src/api/jrec/search.intent.md, same file as #2112.

Pending experiments

None.

The search request refers to `StartTimeRange`, but the schema was not
registered, so the generated document held a dangling reference and
the client generators rejected it.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The query is no longer trimmed, so a hit is a line that holds the
query as sent; only an empty query is rejected.

A long line is now cut at a character boundary even when an invalid
byte comes earlier in it. `limitReached` stays false when the read
budget runs out exactly at the end of the last log, since nothing is
left unread. Manifest read failures are traced rather than logged at
debug level, because they repeat on every search.

Tests now cover recordings that start together, a log listed in the
manifest but gone, and a recording folder without a manifest. The
search schemas get descriptions.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`str::to_lowercase` turns a capital sigma that ends a word into a final
sigma, so the query "ΧΡΗΣ" missed the line "ΧΡΗΣΤΗΣ". Each character
is now lowercased on its own and a final sigma is read as a sigma, so
case never decides whether a line holds the query.

A manifest that lists the same log twice no longer gets it searched
twice. A missing log is traced rather than logged at debug level,
since it repeats on every search while the manifest lists it.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Searching all recordings now skips folders not named by the
hyphenated recording ID, the only spelling Gateway writes, so a copy
under another spelling of the same ID is not searched as the same
recording twice.

Lines are lowercased with `str::to_lowercase` again, keeping its fast
path for ASCII, and every final sigma then becomes a middle sigma.
The schema now states that `query` is not empty and that `lineNumber`
starts at 1.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) force-pushed the feat/recording-search-pagination branch 2 times, most recently from 3a2b5fd to 0122f4c Compare October 21, 2026 04:11
A search request, and its `startTime` range, now fail on a field they
do not know. An option added later, such as a regular expression mode,
then fails on a Gateway that predates it instead of being dropped
while a plain substring search runs.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Matching the raw text of a line made a field name such as
"description" match every line, and missed text a JSON writer
escapes: "DOMAIN\user" is stored as "DOMAIN\user".

A line holding a JSON object is now matched against its string
values, at any depth and once unescaped. Any other line, such as a
half-written last line or one cut at 256 KiB, is still matched as
raw text, so non-JSON artifacts stay searchable.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Numeric page sizes exceeding u32 return 422 instead of the promised 400.

1 open issue
What changed in this PR

Adds stateless cursor pagination to recording search so clients can retrieve results across bounded pages.

Changes:

  • Replaces limitReached with nextCursor and adds maxHits.
  • Documents paging behavior and cursor constraints.
  • Adds pagination and cursor-validation tests.
File Description
testsuite/​tests/​cli/​dgw/​recording_search.rs Adds HTTP pagination and cursor tests.
testsuite/​Cargo.toml Adds hex support for cursor tests.
devolutions-gateway/​src/​api/​jrec/​search.rs Implements cursor pagination and unit tests.
devolutions-gateway/​src/​api/​jrec/​search.intent.md Defines pagination semantics.
Cargo.lock Records the test dependency.

Review effort: Balanced

/// Most hits in the page, from 1 to 1000; 1000 when missing.
#[serde(default)]
#[cfg_attr(feature = "openapi", schema(minimum = 1, maximum = 1000))]
max_hits: Option<u32>,
Benoit's review moved search to matching the string values of JSON
lines, which the intent still described as raw-text matching. The
intent now gives the rule and its reasons, says why line numbers
count from 1, names the provisioner rather than DVLS as the party
that decides who gets a search token, and marks choosing artifact
kinds or other sources as wanted later.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The unicode escape case held a literal "é", so it passed with raw
matching too; it now holds the JSON escape it is named after. New
cases cover a string inside an array, which must match, and a null
value, which must not. The docs also name null among the values that
never match.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No test told apart the rule that a cut line is matched by its string
values when what was read parses from the rejected rule that never
parses a cut line. A cut object followed by spaces now must not match
its field name. The docs say a hit matches the search text rather
than holds it, since a field name holding the query is not a hit.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cut line test only asserted that nothing matched, so it also
passed if the line was never matched. It now searches an escaped
value that only a parsed line can match, and checks the hit is cut.
The endpoint docs say that a line over 256 KiB follows the same rule
on its first 256 KiB, so a cut JSON object is usually raw text.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reason for matching values described what raw matching did, and
"a hit is a line that holds the query" was not true of a JSON line
holding the query only in a field name.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Searching every recording doesn't fit in one response, so
`POST /jet/jrec/search` now returns pages.

A page ends after reading about 16 MiB of logs or after `maxHits`
hits, from 1 to 1000 and 1000 by default; 0 or more than 1000 is a
400. `nextCursor` replaces `limitReached`: sending the same `query`,
`recordings` and `startTime` with it as `cursor` gets the next page,
until a page has no `nextCursor`.

Gateway keeps no state between pages. The cursor holds the recording,
file, byte offset and line number to resume at, plus a hash of the
search, so a cursor sent with another search is a 400. A cursor whose
position cannot exist is a 400 too, and its file name is only matched
against the recording's manifest, never joined into a path. Pages
never go back, so a recording that starts behind the cursor while
paging isn't searched. The paging rules are in `search.intent.md`.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants