Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,14 +94,14 @@ interpretation and do not count as additional benchmark-quality gains.
### Public numeric evidence registry

Every exact public aggregate retained below comes from the checked-in, public-safe
[`offline-fixtures-pr262-git-bytes-20261014.json`](docs/benchmark-evidence/offline-fixtures-pr262-git-bytes-20261014.json) artifact. Its
[`offline-fixtures-pr263-review-boundaries-20261016.json`](docs/benchmark-evidence/offline-fixtures-pr263-review-boundaries-20261016.json) artifact. Its
SHA-256 is
`1b4ba1d95ae003bab4c5d15a7d619fc7a7b813d66387bc9530a74babad5221eb`, also recorded in the
`87fed662d344f5cce1640452f21058b5c5eb1e76a6e6df397378a0731182f18e`, also recorded in the
adjacent `.sha256` file. The artifact contains no raw questions, answers, prompts, customer data,
or per-record content fingerprints.

The fixture-suite digest is
`e5f78cc61a584b64051cc8650c8e2f68cff9a3ccb096df57e8a621d376f829b8`. The artifact defines
`d10e174ef0768ffe70dd2d004cecd4d4a2eb6056c4f3a50bcff05b15b0b826ae`. The artifact defines
the digest algorithm and records the SHA-256 of every suite and dataset file. Each evidence ID
also binds its exact command through `sha256(UTF-8 exact command)`:

Expand All @@ -126,13 +126,13 @@ grounded checks in separate panels; provider billing and MCP transport are not m
the SVG and matching PNG with:

```bash
python scripts/render_benchmark_report.py --report docs/benchmark-evidence/offline-fixtures-pr262-git-bytes-20261014.json --output docs/images/context-efficiency.svg --png-output docs/images/context-efficiency.png
python scripts/render_benchmark_report.py --report docs/benchmark-evidence/offline-fixtures-pr263-review-boundaries-20261016.json --output docs/images/context-efficiency.svg --png-output docs/images/context-efficiency.png
```

The companion examples are also generated from that artifact with:

```bash
python -m scripts.render_benchmark_examples --report docs/benchmark-evidence/offline-fixtures-pr262-git-bytes-20261014.json --output docs/images/evidence-backed-agent-examples.svg --png-output docs/images/evidence-backed-agent-examples.png
python -m scripts.render_benchmark_examples --report docs/benchmark-evidence/offline-fixtures-pr263-review-boundaries-20261016.json --output docs/images/evidence-backed-agent-examples.svg --png-output docs/images/evidence-backed-agent-examples.png
```
The historical-to-executable mapping is in
[`docs/BENCHMARK_CHANGE_COVERAGE.md`](docs/BENCHMARK_CHANGE_COVERAGE.md).
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Connect a coding agent over MCP with the [agent setup guide](https://github.com/
The current registered artifact contains three deterministic offline fixture runs. It separates context size, retrieval quality, and grounded decision checks; these small fixtures do not establish general task performance.

<p align="center">
<img src="https://raw.githubusercontent.com/Coding-Dev-Tools/engraphis/dfa3151d06ca3c226b50bc2e43305f4b23d42903/docs/images/context-efficiency.svg" alt="Three registered offline fixtures: structure-aware chunking reduced mean retrieved context from 740.3 to 214.3 tokens per question (71.1%, 18 questions); the serialized JSON-shape proxy fell from 24,590 to 11,138 tokens across 26 payload samples and 260 recalls. Candidate and packed retrieval quality (Recall@5, Hit@5, and answer-token recall) are shown separately (each 1.000). Grounded checks show 5/5 answerable queries grounded and 6/6 abstention queries rejected, including a 1/1 quarantined-evidence probe; 11/11 decisions were correct. MCP transport and provider billing were not measured. Artifact SHA-256 prefix 1b4ba1d95ae0; see the benchmark guide for the full checksum." width="100%">
<img src="https://raw.githubusercontent.com/Coding-Dev-Tools/engraphis/e574baa806eef561b24ac1318fb3abebb4432956/docs/images/context-efficiency.svg" alt="Three registered offline fixtures: structure-aware chunking reduced mean retrieved context from 740.3 to 214.3 tokens per question (71.1%, 18 questions); the serialized JSON-shape proxy fell from 24,590 to 11,138 tokens across 26 payload samples and 260 recalls. Candidate and packed retrieval quality (Recall@5, Hit@5, and answer-token recall) are shown separately (each 1.000). Grounded checks show 5/5 answerable queries grounded and 6/6 abstention queries rejected, including a 1/1 quarantined-evidence probe; 11/11 decisions were correct. MCP transport and provider billing were not measured. Artifact SHA-256 prefix 87fed662d344; see the benchmark guide for the full checksum." width="100%">
<br>
<sup>Three offline fixtures separate context reduction, candidate and packed retrieval quality, and grounded behavior. The chart shows the artifact checksum prefix; see the benchmark guide for the full checksum and reproduction steps.</sup>
</p>
Expand All @@ -61,7 +61,7 @@ The current registered artifact contains three deterministic offline fixture run
| Recall payload proxy | JSON-shape proxy: 24,590 → 11,138 tokens (54.71% lower, 26 samples; 260 timed recalls) | Candidate and packed Recall@5, Hit@5, and answer-token recall are each 1.000 |
| Grounded decisions | 5/5 answerable queries grounded; 6/6 abstention queries rejected, including 1/1 quarantined-evidence check | 11/11 decisions correct |

The payload figure is a serialized JSON-shape estimate, not an MCP transport measurement or provider billing total. See the [Benchmark methodology](https://github.com/Coding-Dev-Tools/engraphis/blob/dfa3151d06ca3c226b50bc2e43305f4b23d42903/BENCHMARKS.md) for artifact identity, counting methods, reproduction commands, external-evaluation boundaries, and limitations.
The payload figure is a serialized JSON-shape estimate, not an MCP transport measurement or provider billing total. See the [Benchmark methodology](https://github.com/Coding-Dev-Tools/engraphis/blob/5f94cd2a1f01c5627cb62104015d184ca9c30f01/BENCHMARKS.md) for artifact identity, counting methods, reproduction commands, external-evaluation boundaries, and limitations.

## Optional Jev assistance

Expand Down
4 changes: 4 additions & 0 deletions docs/DOCUMENT_IMPORT.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,10 @@ rather than silently split. Containers are additionally capped at 2,000 members
declared decompressed content. Invalid UTF-8/UTF-16 in permitted text is replaced explicitly and
reported as a parsing warning.

Library Quick import applies the same credential filename exclusions to uploaded files and
folder paths before extraction or optional fact derivation. Hidden and dependency directories
are skipped as well, and a symlink's target cannot bypass the filename exclusions.

Markdown uses the Obsidian adapter's stricter 2 MB raw-note limit before decoding.

Review a preview before importing sensitive material. The local importer makes no network request
Expand Down
11 changes: 9 additions & 2 deletions docs/SPEC_CRAWL.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ The unified dashboard at `http://127.0.0.1:8700` includes a dedicated **Spec Stu
Beyond inspecting isolated single-text prompt specifications, Spec Crawl provides cluster-level coherence auditing across multiple active memory records (`engraphis/core/spec_crawl.py::analyze_memory_nodes`):
1. **Contradiction Detection**:
- Parameter collisions for positively matching, nonempty subject keys (e.g. `port: 5432` vs `port: 5433` for the same database).
- Sentence-final periods on numeric port, timeout, and retry values do not create a parameter conflict; decimal values remain intact.
- Opposing modal directives (e.g. `must enforce <target>` vs `never allow / forbid <target>`).
- Divergences for the same `(subject_key, claim_kind)` are review suggestions; different keyed subjects do not establish parameter conflicts.
- A keeper suggestion requires distinct, finite `valid_from` dates. Ingestion order never determines which fact is current. Missing, invalid, or equal effective dates require explicit clarification without a retirement target.
Expand All @@ -111,6 +112,10 @@ Beyond inspecting isolated single-text prompt specifications, Spec Crawl provide

Memory bodies are used during analysis but are not returned in node summaries. The workspace audit analyzes at most 150 eligible memories per request and reports its eligible total and truncation state; Spec Studio warns when its health score and findings describe only a partial sample.

Title aliases and body signals match whole words, so titles such as “Preview configuration” do not count as review policies. Similarity comparisons use at most 512 distinct tokens per memory, selected deterministically by token hash, and reuse each pair's result across link, orphan, divergence, and redundancy analysis. Reports expose `similarity_token_limit` and `similarity_sampled_node_count`; each node includes `similarity_token_count` and `similarity_sampled`. Sampled similarity is approximate and produces suggestions for review. Parameter and directive conflict checks inspect the complete memory prose outside fenced and inline code examples. Code examples are also excluded from prompt injection scoring.

Word classification reads at most `word_limit` (5,000) tokens per memory. Each node reports `word_analysis_truncated` and `word_count_is_lower_bound`; a truncated `word_count` is the number analyzed, not the full memory length. Reports include `word_analysis_truncated_node_count`, and partial nodes receive no clean-node quality credit. HTTP and MCP audits set `truncated` and `analysis_truncated` for either omitted memories or incomplete word analysis; `node_selection_truncated` identifies an omitted-memory sample separately.

---

## 7. Command Line Interface (CLI)
Expand Down Expand Up @@ -170,12 +175,12 @@ Executes an explicit remediation on two live, approved memories after checking s

## 9. Smart MCP Tools

The two read-only crawls are available to hosted/team viewers; clarification and resolution require member access.
In hosted/team Classic MCP, the two read-only crawls allow viewer access; clarification and resolution require member access. In remote Smart MCP, crawls use the viewer-accessible `engraphis_execute_read`, while clarification and resolution use `engraphis_execute_action` and require admin access. Local stdio retains the operator's full capability.
Omit `workspace` when passing a `session_id` to inherit its workspace and repository; explicitly mismatched scope selections are rejected. Default cluster audits return at most 150 eligible nodes and report `truncated`, `node_limit`, and `count_semantics`; the dashboard labels truncated health scores as a sample. Inferred similarity links are suggestions and do not count as stored connectivity or remove orphan remedies.

These four Classic MCP tools are also discoverable and callable through Smart MCP actions:
- **`engraphis_spec_crawl`**: Read-only prompt spec crawler returning scores, radar coverage, classified tokens, connection tentacles, and flags.
- **`engraphis_spec_crawl_answer`**: Clarification answer submission updating specifications and writing pending procedural memories.
- **`engraphis_spec_crawl_answer`**: Clarification answer submission updating specifications and optionally saving pending procedural memories.
- **`engraphis_spec_crawl_memories`**: Audits active workspace memory nodes for contradictions, orphans, redundancies, and cluster health.
- **`engraphis_spec_crawl_resolve`**: Applies an explicitly requested link or confirmed supersession to selected memory nodes.

Expand All @@ -185,6 +190,8 @@ These four Classic MCP tools are also discoverable and callable through Smart MC

CI verifies determinism, axis classification, flag recall, qualified-token avoidance, section counts, and score boundaries in both the full stack and NumPy-only jobs. Qualified-token avoidance measures the explicitly labelled negative examples; it is not a general false-positive precision estimate:

The same gate checks eight synthetic memory-audit contracts in both input orders: unrelated-subject noise, agreeing and opposing nested directives, subject and claim-kind separation, stored orphan connectivity, unknown effective dates, and effective-time ordering. It compares exact conflict pairs and retirement targets, requires compact node responses, and reports the fixture digest. These bounded regression fixtures do not establish general classifier quality.

```bash
python -m eval.spec_crawl
```
Expand Down
Loading