Repository navigation
feat(dgw): page recording search with a cursor - #2137
irvingouj@Devolutions (irvingoujAtDevolution) wants to merge 20 commits into
Conversation
Add `POST /jet/jrec/search`, so DVLS can search the `.slog` logs of
recordings without downloading every log.
The body is `{ "query": "...", "recordings": "all" | ["<id>", ...] }`,
guarded by a new `gateway.recordings.search` scope. The existing
`gateway.recordings.read` only lists IDs, so reusing it would let every
read token see log contents.
The search reads the `.slog` files and AI analysis logs listed in
`recording.json`, with the rules of the log viewer's parser: a
case-insensitive substring over the shown fields and parameters,
entries the viewer drops are skipped, and long strings match on their
shown part only. Hits are the original lines. A search stops at 1000
hits, 16 MiB of hits or 256 MiB read, and reports it with
`limitReached`.
`api/jrec.rs` moves to `api/jrec/mod.rs` to hold the `search`
submodule.
Issue: DGW-463
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The log viewer strips a trailing `\r` before it checks a line against the 256 KiB limit, so a CRLF line of exactly 256 KiB is shown there. The search now strips it first too, also on a last line without `\n`. Mark the search invariants with `INVARIANT:` and give each recording selection case its own test. Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The derived schema put `"all"` in an `AllRecordings` schema of its own, and the C# client generator (7.9.0) turns that `oneOf` member into a constructor that does not compile. The schema now lists `"all"` inline, like `MonitoringProbeType`; the JSON is unchanged. Also test the line limit one byte over it and on a last line ending with `\r`, and keep `INVARIANT:` for definitions only. Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Scanning costs about 16 ms per MiB of logs, so a 256 MiB budget kept one request busy for about 4 seconds. A search now reads at most 16 MiB. Every hit is a line the search has read, so the read budget bounds the response too, and the separate 16 MiB hit-bytes limit goes away. Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Search no longer reads `.slog` lines the way the log viewer's parser does. It matches the query, case-insensitively, against the raw text of each line, so Gateway does not need to understand a log to search it and any kind of artifact can be searched. It reads only the `.slog` files listed under `artifacts` in `recording.json`. A hit is now the line as a string, with its line number and byte offset, since a line is not guaranteed to be JSON. A line over 256 KiB is matched and returned on its first 256 KiB, marked `truncated`, and the rest of it is skipped within the read budget, so a log without line breaks cannot make one request read all of it. The request takes an optional `startTime` range of RFC 3339 times, matched on the start time in `recording.json`, with `from` included and `to` excluded. Recordings that start together are ordered by ID, so the order is the same on every request. Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record what the search reads, how it matches, what a hit holds, who may search, and its limits, with the reason behind each. Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Lines changed, against #2112:
|
Intent drafts: handoffThe ProvenanceLevels: 1 = my words verbatim, 2 = my words tidied, 3 = agent wording I adopted, 4 = agent fill-in, not intent.
Agent fill-in (level 4, not in the intent file): the cursor is hex-encoded JSON, not signed. A changed cursor can only move the read position inside the logs the same search reads, since its recording ID and file name are matched against the listing and never joined into a path. Left out on purpose
Placement
Pending experimentsNone. |
The search request refers to `StartTimeRange`, but the schema was not registered, so the generated document held a dangling reference and the client generators rejected it. Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5224627 to
100d71d
Compare
The query is no longer trimmed, so a hit is a line that holds the query as sent; only an empty query is rejected. A long line is now cut at a character boundary even when an invalid byte comes earlier in it. `limitReached` stays false when the read budget runs out exactly at the end of the last log, since nothing is left unread. Manifest read failures are traced rather than logged at debug level, because they repeat on every search. Tests now cover recordings that start together, a log listed in the manifest but gone, and a recording folder without a manifest. The search schemas get descriptions. Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
100d71d to
70ada1a
Compare
`str::to_lowercase` turns a capital sigma that ends a word into a final sigma, so the query "ΧΡΗΣ" missed the line "ΧΡΗΣΤΗΣ". Each character is now lowercased on its own and a final sigma is read as a sigma, so case never decides whether a line holds the query. A manifest that lists the same log twice no longer gets it searched twice. A missing log is traced rather than logged at debug level, since it repeats on every search while the manifest lists it. Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
70ada1a to
bd82377
Compare
Searching all recordings now skips folders not named by the hyphenated recording ID, the only spelling Gateway writes, so a copy under another spelling of the same ID is not searched as the same recording twice. Lines are lowercased with `str::to_lowercase` again, keeping its fast path for ASCII, and every final sigma then becomes a middle sigma. The schema now states that `query` is not empty and that `lineNumber` starts at 1. Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3a2b5fd to
0122f4c
Compare
A search request, and its `startTime` range, now fail on a field they do not know. An option added later, such as a regular expression mode, then fails on a Gateway that predates it instead of being dropped while a plain substring search runs. Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0122f4c to
c3292a5
Compare
Matching the raw text of a line made a field name such as "description" match every line, and missed text a JSON writer escapes: "DOMAIN\user" is stored as "DOMAIN\user". A line holding a JSON object is now matched against its string values, at any depth and once unescaped. Any other line, such as a half-written last line or one cut at 256 KiB, is still matched as raw text, so non-JSON artifacts stay searchable. Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
c3292a5 to
4ea8c83
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Numeric page sizes exceeding u32 return 422 instead of the promised 400.
1 open issue
What changed in this PR
Adds stateless cursor pagination to recording search so clients can retrieve results across bounded pages.
Changes:
- Replaces
limitReachedwithnextCursorand addsmaxHits. - Documents paging behavior and cursor constraints.
- Adds pagination and cursor-validation tests.
| File | Description |
|---|---|
| testsuite/tests/cli/dgw/recording_search.rs | Adds HTTP pagination and cursor tests. |
| testsuite/Cargo.toml | Adds hex support for cursor tests. |
| devolutions-gateway/src/api/jrec/search.rs | Implements cursor pagination and unit tests. |
| devolutions-gateway/src/api/jrec/search.intent.md | Defines pagination semantics. |
| Cargo.lock | Records the test dependency. |
Review effort: Balanced
| /// Most hits in the page, from 1 to 1000; 1000 when missing. | ||
| #[serde(default)] | ||
| #[cfg_attr(feature = "openapi", schema(minimum = 1, maximum = 1000))] | ||
| max_hits: Option<u32>, |
Benoit's review moved search to matching the string values of JSON lines, which the intent still described as raw-text matching. The intent now gives the rule and its reasons, says why line numbers count from 1, names the provisioner rather than DVLS as the party that decides who gets a search token, and marks choosing artifact kinds or other sources as wanted later. Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4ea8c83 to
27d9951
Compare
The unicode escape case held a literal "é", so it passed with raw matching too; it now holds the JSON escape it is named after. New cases cover a string inside an array, which must match, and a null value, which must not. The docs also name null among the values that never match. Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
27d9951 to
9ea50fc
Compare
No test told apart the rule that a cut line is matched by its string values when what was read parses from the rejected rule that never parses a cut line. A cut object followed by spaces now must not match its field name. The docs say a hit matches the search text rather than holds it, since a field name holding the query is not a hit. Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9ea50fc to
2db02f6
Compare
The cut line test only asserted that nothing matched, so it also passed if the line was never matched. It now searches an escaped value that only a parsed line can match, and checks the hit is cut. The endpoint docs say that a line over 256 KiB follows the same rule on its first 256 KiB, so a cut JSON object is usually raw text. Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2db02f6 to
f56cb84
Compare
Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
f56cb84 to
175d6db
Compare
The reason for matching values described what raw matching did, and "a hit is a line that holds the query" was not true of a JSON line holding the query only in a field name. Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
175d6db to
f05df42
Compare
Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Searching every recording doesn't fit in one response, so `POST /jet/jrec/search` now returns pages. A page ends after reading about 16 MiB of logs or after `maxHits` hits, from 1 to 1000 and 1000 by default; 0 or more than 1000 is a 400. `nextCursor` replaces `limitReached`: sending the same `query`, `recordings` and `startTime` with it as `cursor` gets the next page, until a page has no `nextCursor`. Gateway keeps no state between pages. The cursor holds the recording, file, byte offset and line number to resume at, plus a hash of the search, so a cursor sent with another search is a 400. A cursor whose position cannot exist is a 400 too, and its file name is only matched against the recording's manifest, never joined into a path. Pages never go back, so a recording that starts behind the cursor while paging isn't searched. The paging rules are in `search.intent.md`. Issue: DGW-463 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
f05df42 to
170f536
Compare

Searching every recording doesn't fit in one response, so
POST /jet/jrec/searchnow returns pages.A page ends after reading about 16 MiB of logs or after
maxHitshits, from 1 to 1000 and 1000 by default; 0 or more than 1000 is a 400.nextCursorreplaceslimitReached: sending the samequery,recordingsandstartTimewith it ascursorgets the next page, until a page has nonextCursor.Gateway keeps no state between pages. The cursor holds the recording, file, byte offset and line number to resume at, plus a hash of the search, so a cursor sent with another search is a 400. A cursor whose position cannot exist is a 400 too, and its file name is only matched against the recording's manifest, never joined into a path. Pages never go back, so a recording that starts behind the cursor while paging isn't searched. The paging rules are in
search.intent.md.🤖 Generated with Claude Code
Issue: DGW-463