Repository navigation
fix: integrate storage, workflow, retention and evaluation repairs - #4772
Merged
ScriptedAlchemy merged 74 commits intoNov 5, 2026
Merged
Conversation
A store whose workflow schema this binary refuses reported only that its workflow tables were "absent, incomplete, or not exact", so four refused stores could not be told apart or traced to a cause. The refusal still fails closed through the existing typed reset path but now says what differs: the missing and altered contract tables, the stored workflow identity next to the required one, the differing identity fields, or the first differing column. Only contract names and numbers are echoed; stored SQL, column text, and unrecognized digests never reach the message, and lists are bounded. RefusedAuthorityV1 carries the reason as a Cow so static refusals stay constant. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Refs #4513. Keep each physical SQLite pin under one shared descriptor so tearing down a reader or writer owner cannot release POSIX locks that belong to another live SQLite connection. Defer pin release on file loss until all affected workers have stopped. Cover rollback and WAL locking with independent-process regression probes.
Preserve the concurrent shared-pin release and formatting repair. Invoke that release directly from the attachment that already owns the shared pin and verifies both worker groups have exited, including final close. Remove the now-unneeded reader pool and locator forwarding methods. Strengthen the held-writer regression with a retained pin handle: it must remain usable while the writer is held and be revoked after exit. Keep the held-reader-pool descriptor regression and existing deadlines. Native Bazel validation is still required.
Refs #4513. Require a live shared pin before reader path selection and connection verification so retained locators cannot regain authority from a surviving canonical pathname after shutdown. Preserve the concurrent attachment-owned teardown. Add a real reader fixture regression for both the original locator and its retained clone; release through an attachment pin only after all SQLite connections exit. Keep the historical identity for diagnostics. Source-reviewed; Bazel/Rust execution requires hosted CI.
A backend with no executable bound and a store refusing until its reset cannot recover by repeating work, yet both retried on every tick. Report the unbound executable as a typed backend_unconfigured skip and scheduler status, before any run is admitted, so nothing accumulates and the next tick after a configuration change admits the task. The retry ladder also stops at one attempt for it. Park the loop on a reset refusal until a project open or reconcile wakes it. Log tick and task failures as state changes, which also quiets failures that stay on cadence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4540. Context Scout previously loaded tool_calls/tool_events metadata from user and tool-result rows, then treated each as observed tool activity. An untrusted result could therefore qualify a hint despite a successful negative semantic decision. Keep metadata-origin calls only on assistant messages and enforce that provenance at the intent selector too. Add registered-store and pure-window regressions covering forged user/tool metadata, genuine assistant calls, negative model answers, and absence of authentic tool invocations. Full Kev evaluation and legacy hint-route retirement remain outstanding.
The analytics diagnostics summary answered three facts it never measured: an unmounted session-message authority produced `message_count: 0` and four `0.0` per-message rates, an unmounted readiness projection counted every row as skipped and left the fold permanently unavailable, and a window with no observable span served `events_per_hour: 0.0`. The readiness projection was also only mounted by the dashboard MCP handler, so the CLI and MCP admin surfaces always reported it unmounted. It now lives once in `tracedecay-agent-hosts` (where the aggregation already lives), is exported through one idempotent install, and is mounted by the admin-CLI dispatch every diagnostics surface crosses plus the dashboard entry points. `message_count` and every ratio member are now optional on the wire: absent authority and measured zero both serve `null` rather than a fabricated zero, and `events_per_hour` is `null` when no span was observed. Unavailable readiness fallbacks report only genuinely non-`hook_completed` rows as skipped. Dashboard contracts are regenerated and consumers state the absence.
Refs #4513 and #4099. Daemon-backed storage reports opened and closed a raw SQLite database descriptor to inspect its free-page header. On Unix, that close can release process-wide record locks of live SQLite writers. Skip raw header reads in the Unix daemon path, retaining size information and representing free pages as unavailable instead of zero. Preserve the separate-process offline CLI and Windows measurements. Test the registered-store report while a transaction holds a lock: an independent SQLite process must remain excluded before and after it, and the committed database must pass integrity_check.
Route project-scoped daemon reports through the same no-raw-descriptor policy as paginated daemon reports on Unix. Keep offline header sampling unchanged. Verify an independent rollback writer stays excluded through both live report entry points, with free-page metrics truthfully unknown.
Correct the exact repository-gates formatting diff on the current SQLite lock-lifetime repair without changing its behavior or tests.
Apply the exact rustfmt delta reported by Repository gates job 127025791417 in CI run 41960601791. Preserve all assertions and production behavior.
Refs #4535. The decoder already requires an exact option map but still accepts probabilities that do not sum to one, or a chosen label that is not the most probable. Validate the declared SystemOne choice contract with a small decimal-rounding tolerance. Preserve explicit none choices, optional picks, and typed failure outcomes. Exercise malformed and rounded-success responses through the real HTTP decision client. The broader Kev evaluation and routing decision remain outstanding.
Shorten the shared probability-tolerance name to avoid wrapping the selected-label guard; preserve the checks and all regression cases.
Apply the exact two formatting changes required by the current-head repository-gates log without altering the independently verified SQLite writer-exclusion regression.
Refs #4535. Remove the automatic cross-run hybrid and its invented 680 ms latency surcharge. Do not replace a recorded hybrid's labels, probabilities, usage, or latency. Reject duplicate configuration files before replacing a summary rather than silently choosing a filename. Add CLI regressions for matching and disjoint arm datasets, preservation of actual hybrid measurements and input files, and duplicate refusal. All 16 standalone Python checks pass on hash-verified source copies; the new cases fail against the original scorer. Bazel CI is outstanding.
Refs #4535. An explicit null, boolean, number, object or array type tag was erased by and_then(as_str), bypassing the answer-kind checks as if the tag had been omitted. Preserve omitted-tag compatibility but reject malformed declared tags before decoding any question kind. Cover Noul, Choice and Score through the real loopback HTTP client with valid omitted tags and all five non-string JSON shapes. Preserve the existing probability, alias, range and mismatch checks. Independently source-reviewed; Rust/Bazel execution requires hosted CI.
Refs #4535. The runner opened the completed JSON destination before serializing it, so disk/serialization errors could leave a truncated final file that blocked exact-identity resume despite a complete .partial log. Render and fsync a same-directory staging file before publishing with an exclusive hard link. Preserve the previous completed result during races; leave the validated partial for retries on staging or link failure, and remove staging after every finished attempt. Add real runner tests for full-byte visibility, stage sync/link failures, no-new-inference retry, competing publications, and staging cleanup. Retain the single-writer/no-crash-durability limits.
Every tagged release through 1.0.0-beta.92 created the same nine workflow tables and identity digest. beta.93 relaxed the artifact table's CHECK and changed the digest with no path forward, so those stores were refused and could only be reset, losing their workflow records. Admission now recognizes exactly that shape: byte-exact stored SQL for all nine tables, the single published identity row, no other schema object referring to the artifact table, and rows that satisfy the published constraints. It re-proves the shape under the write lock, then moves the table aside, creates the final table, copies rows with their rowids, verifies them byte for byte, drops the old table, compare-and-swaps the digest, and requires the final contract, all in the admission transaction. Anything else keeps the typed refusal, and a store another authority refuses is not touched. The published digest is an opaque label rather than a hash of the stored SQL, so the stored text is the evidence. A failure that recurs on every open is a typed reset refusal; one that can clear, such as a held lock or full disk, stays an operation failure the next open retries. The refusal's identity read gets the same split: a transient read failure stays typed instead of reading as an unreadable identity. The explicit operator request for this upgrade overrides the repository's no-data-migration default for this one published shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Detect hard-link incompatibility before issuing HTTP inference. Keep no-clobber and resumable evidence unchanged, and verify sample destinations, interrupted retries and dangling output links.
The two native-runtime diagnostics tests failed in the full application suite while passing alone: `language_refresh` reported `LSP server '/usr/bin/python3' initialize timed out after 5000 ms` and `idle_backfill` never published. Both tests time a 5s initialize budget, and the first interpreter spawn in a test process costs far more than every later one. Measured in this target, the first start against the fixture took 0.49-1.66s while the second and third took 27-90ms for the same script. Under load that one-time cost consumed the whole budget, so the assertion timed the process's first spawn rather than the refresh. Warm the interpreter once before the authority is built, so the measured refresh starts from the steady state. Verified against the configuration that reproduced both failures (24 CPU burners on 18 cores): the full suite now passes 567/567, and the two tests pass 5/5 under 40 burners. The idle backfill also discarded its refresh error entirely, so a backfill that produced nothing looked like one with no work to do. Log it; the broker already records it as the language's engine error.
Refs #4535. Exclusive-publication preflight must not delete an already-existing path when hard-link creation fails with EEXIST. Only unlink the probe link when this invocation created it. Also clean owned staging after close or cleanup failures and preserve underlying permission and filesystem error details. Add four fault-injection regressions for collision, permissions, close failure and cleanup failure; preserve existing runner tests. The exact patch passed 41 standalone Python regressions previously; Bazel CI is required on the published head.
The reset census rehearsed the whole published-shape upgrade and rolled it back, so a 276 MB store wrote 267 MB to the WAL per census. A census now decides from read-only classification plus the one catalog-only rename no query can predict: 19 ms and no WAL growth at any payload size. One copy statement and one byte-compare query could outrun the transaction deadline, which renews only between statements, so a large store could fail every open and never upgrade, and shutdown could not stop the copy. Rows now move in batches of at most 128 MiB, and the classification scan is paged. The byte compare is gone: SQLite's transaction already guarantees the copy, so rows are checked by count. Classification also missed shapes the copy would have mishandled: a trigger named after the artifact table, triggers on workflow_schema, rows of the wrong storage type, and an identity row that does not decode. A schema that refuses the rename now refuses workflow features and serves the store's other authorities, as other shape refusals do. The upgrade runs last, so a store refused by observations keeps its published shape. A real attach still copies once, since the exact-SQL guard denies the catalog edit that would avoid it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`records_truncated` was `records.len() >= AUTOMATION_RECORD_LIMIT`, which reports the newest page as a truncated window whenever the ledger holds exactly that many rows. A project whose automation ledger held exactly 200 runs was told its counts were partial when the page was the whole ledger. The page read already answers this: `AutomationRunLedgerPageV1::is_complete` is false only when a row was left behind or a row could not be read. Read the page instead of the row list so the flag comes from that authority rather than from an inference about the row count. The new test derives the page bound from the service, then asserts the boundary from both sides: a ledger of exactly that many rows is complete, and one more row is truncated. Against the old predicate it fails with `left: Bool(true), right: false`.
A held lock or full disk rolled the background upgrade back and left workflow storage recovering until the daemon restarted: schedule() runs a store's task once, and a mounted store is not reopened, so 1dba4d9's "next open retries" did not hold for a store that stayed mounted. The store's one convergence task now tries again while its authority is still recovering, about once a minute. It keeps the foreground-open gate, the single convergence permit, and abort on retire and shutdown, and between attempts it holds no permit and only the weak lease issuer, so a waiting store cannot block its owner's retirement. No periodic owner fits: the maintenance tick is hours apart and does not run at all with retention disabled. A store mounted again over a failed upgrade replaces the old worker, which could only find its owner gone. A refusal stays the typed reset; shutdown stops the retries. workflow_storage_problem no longer erases recovering to workflow.unavailable: workflow tools, work retry rebinding, and fan-out recovery answer the retryable workflow.authority_recovering. No wire shape changes. A production-composition journey proves it over a real SQLite write lock and the MCP server: code search serves and workflow tools answer the typed problem while the attempt is parked and after the lock fails it, then the attempt after the lock clears commits on the same mount with every row intact. With the retry disabled this journey fails. SQLITE_FULL is proven as an injected fault at every upgrade step, not a real full disk: the SQLite guard refuses max_page_count, and it stays in place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
//:rustfmt_check, part of the required Repository gates job, rejects the bounded retention scan merged from #4769. Apply the pinned rustfmt to the seven affected files. The diff is whitespace, import order and closure braces only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reword comments and HTTP_RUNNER.md text added by this integration. Remove change narration, replace figurative phrasing and symbol shorthand, and name the actor in the runner's publication and hard-link probe behavior. Only comments, one Python docstring and the runner doc change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4535. The evaluator followed redirects and honored ambient proxy settings, allowing a transcript-bearing request or its model answer to cross the operator-selected serving boundary. Disable environment proxies and all redirects for each decision request. Test actual local HTTP delivery with an injected dead proxy and verify that a cross-endpoint redirect is refused at the originating response. Preserve byte-exact request-hash coverage. An independent Python transport reproduction passed. The integrated Bazel Python tests still require hosted CI.
A unix test in the SQLite attachment called a Linux-only helper, so that crate's unit tests did not compile on macOS. The check is now Linux-only, matching the helper. The rest are lint fixes with no behavior change: the daemon reset report builds its section without format-push, and two tests match the outcome directly and show the value when they fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Doctor called a store waiting on its next workflow upgrade attempt stopped, and the only difference between it and a real failure was text in a message. Schema convergence now has a typed retrying state that keeps the last failure, separate from degraded, which now means nothing retries on this mount. Status, the dashboard, and the CLI show it. CLI doctor warns rather than fails, since the daemon clears it by itself. A refusal stays terminal, and the retry delay is unchanged. Three paths could leave a store recovering with no worker, or report retrying with none running: - A retirement refused by a client lease had already stopped the worker, so the restored mount answered recovering until restart. The interrupted plan now runs again on the restored owner. - A retry that woke during a retirement fence exited, though the fence could still roll back. It now waits for the next attempt. - A worker that found its owner gone left retrying on display, and so did a missing timer. Both now report degraded, without overwriting a successor. A retry also takes its strong lease only after the foreground gate and the convergence permit, so queueing no longer holds a client that blocks the store's retirement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4535. Refuse nonnumeric, out-of-range or nonfinite probability values before evaluating accuracy or calibration. Also reject invalid labels, duplicate cases, missing category coverage, invalid choices and elapsed times, including corrupted raw probability telemetry. Preserve legitimate Noul and Choice result shapes (including an explicit none option). A malformed result must not replace an existing summary. Add focused direct and CLI regressions. The standalone Python validation probe passed; Bazel scoring tests and integrated CI remain outstanding.
Refuse versioned HTTP scoring rows whose calibrated probabilities disagree with recorded raw Noul probabilities or calibration temperature. Move the existing runner calibration function verbatim into shared common.py so runtime validation and scoring use one authority. Cover valid calibration, corrupted output, missing raw evidence, temperature conflicts, and the CLI summary no-clobber path. Refs #4535. The standalone/Bazel tests have not run in this environment; full integration CI on the exact head is required before merge to master.
Reject score inputs whose recorded HTTP experiment identity, per-case request fingerprint aggregate, case order, dataset identity, or arm name does not match the completed result. Preserve local-only scoring and calibration checks. Add regression cases for each mismatch and the last-good summary preservation path. Refs #4535. Focused Python fingerprint validation passed; full Bazel tests and integrated CI are required before merging to master.
Restore real Markdown newlines and the intended newline-terminated last-good summary fixture in the HTTP provenance regression. Refs #4535.
Refs #4535. Lock each partial output for its writer lifetime and reject redirected or replaced partial paths before writing. Preserve crash-resumable files and refuse concurrent same-tag inference without duplicating results. Add a barrier regression for two concurrent runs and an alias-isolation test. Full Kev qualification, labels and runtime parity remain open.
Refs #4535. Allow operators to supply a bearer token through an environment variable without recording the credential or its variable name in experiment identity, partial rows or completed results. Require HTTPS for authenticated remote inference, allowing direct loopback HTTP for local serving. Keep proxies and redirects disabled. Cover early denial of missing, malformed and insecure configuration, credential-free evidence, and actual loopback transport behavior. This enables remote authentication only; independent labels, runtime parity and model qualification are still outstanding.
Use binary-search ranks for tied AUC and subtract a held-out row from precomputed threshold confusion counts. Keep threshold tie-breaking and undefined-class semantics unchanged. Add exhaustive-oracle comparison and a large-cohort scoring regression. Refs #4535
Refs #4535. A second same-tag evaluator could acquire the partial lock after the first released it but before the first published the result, racing to publish the same evidence. Keep the lock until the no-clobber final result is linked, recheck an output that appeared before lock admission, and cover both races with behavioral tests.
Avoid indefinite FIFO reads during resumable evaluation admission. Use nonblocking, no-follow opening and validate the descriptor type before taking the existing exclusive lock and parsing the log. Retain existing pathname-identity verification and regular resume behavior. Add a bounded behavioral regression that proves a named pipe is refused without waiting for a writer or issuing model requests. Refs #4535
HTTP measurement rows retain request fingerprints. If the experiment field disappears, the scorer previously treated these results as an unprovenanced local run, bypassing identity and calibration checks. Require the HTTP identity whenever HTTP fingerprint markers remain. Preserve historical local results with their dataset digest, and test both refusal and preservation of the last scored summary. Refs #4535
The scorer validated request and dataset fingerprints but never checked that each output row retained its original ground-truth labels or context-needed annotation. Relabelling completed rows could silently change evaluation accuracy with otherwise valid experiment identity. Bind those annotations to the captured experiment and refuse changed or unbound evidence before scoring. Add tests for label/context drift, incomplete identity, and preserving the last valid summary. Refs #4535
Reject malformed case labels, metadata and rendered inputs before opening results or spending tokens, rather than persisting unscorable evidence. Cover invalid cases and preserve the existing valid dataset behavior. Refs #4535
…chema-diagnostics fix: recover workflow stores and repair agent runtime behavior
ScriptedAlchemy
changed the title
CI runs clippy with -D warnings, and the #4769 code fails it in two ways. The retention planner passed `&mut usize::MAX` for its entry budgets, which const_item_mutation rejects, and two doc comments left `ReadDir` unquoted. Name a local budget at each call. Each call still starts from its own usize::MAX, so behavior is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
deleted_descriptors reads /proc/self/fd and exists only on Linux, but a #[cfg(unix)] test called it, so this crate's unit tests could not compile on macOS. The assertion still runs on Linux. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This PR combines all eight open application PRs into one integration PR that is ready for review. The maintainer requested this to cut duplicate CI queues. Release PR #4763 stays out of this PR and is unchanged.
The integration branch starts at master commit
8505b451976093dd52526e62ecf3839937b1f40b. The consolidation retargeted each source PR tointegration/application-repairs-20261105, merged it through GitHub's ordinary merge endpoint with its exact expected head SHA, then checked separately that it closed as merged into this staging branch, not master. It used no force pushes or source squashes, changed no branch protection, and discarded no commits.Included source revisions
13959bc85dedb7f4587c659039e762201e37b5fa7784423279a8dd9f3f48531b775abedab7b957094d7cd9543b28141547cfdf54188674c4f77799d4ad525ff9449c69c7f1aa47eb54ccca5a258bd9bdc4ed744789e341a44592a7f7b44c479982cc50be8cd33915f984d2edf3c7ed2cee533787488325f8c4f087192889515bc69a4aadc500569f7b77790d29196f83399e91201b1c9db24ee730ca90ed6b09All eight merges completed without textual conflicts and preserve the original commit ancestry and regression tests. The initial combined head is
a6fd1ad8e5eda5dc02a8b900b6fd8a93e23644b9.Later commits on this branch add follow-up repairs. Two of them do not change behavior:
8b4c5f05afapplies rustfmt to the #4769 retention scan code, which failed//:rustfmt_check, anda0fb660cderewords comments andHTTP_RUNNER.md.Additional application consolidation (November 5)
master, marked ready and merged only into this integration branch, preserving its exact source headefcab24483e59f923a2f77bd0861907a498be80fand merge commit45d50ccc720d7833007f753af54a02c79de50065. Its 93 changed paths, generated contracts and regression suites require validation as part of this combined PR; source-only or draft checks do not establish combined readiness.mastervia normal expected-head merges. This integration branch incorporated exactly their two lockfiles with a non-force two-parent commit1846fdfeacc5adcc0dddca1cf6665f92f0674b15, using master028026b35174435b6b69e68c37169b9fa7e74c90as its second parent. No application source was overwritten.Validation and outstanding review
Combining these PRs did not validate them and did not merge anything to master. This PR claims no combined Rust or Bazel test pass.
Run the full applicable CI matrix on this PR's current head, integrated with current master. Passing runs on the source branches do not count as proof for the combined revision. Keep native platform coverage for SQLite lock lifetime and for Windows directory-handle and quarantine behavior, and keep the evaluation runner and scoring tests.
This consolidation does not clear earlier source CI failures, including the thread-exhaustion failure on #4761. Review the current combined logs and repair the defects they trace to. The table above links the source PRs, and their review discussions still apply. Unresolved findings and required reviews block the merge to master.
While this PR is open, make follow-up application repairs on this integration branch instead of starting separate long-running CI runs. Do not reopen the superseded #4760 and #4765 or the eight integrated source PRs. Push non-force updates against the exact current head, so concurrent edits survive.
This PR includes no releases, version bumps, tags or publication, and no production, infrastructure or credential changes. The completed beta.95 recovery stays as it is. Issues #4513, #4535, #4540, #4099, #4756 and #4374 stay open where their original acceptance work is outstanding.
🤖 Generated with Claude Code