Sync ako/mxcli: incremental source-catalog rebuilds (re-describe only what changed)

.claude/skills/fix-issue/findings/mdl-other/2026-10-30-source-catalog-rebuild-redescribes-every-document-after-each-save.json

@@ -0,0 +1 @@
+{"area":"mdl/catalog","date":"2026-10-30","symptom":"On a project with a source index, every catalog rebuild — `refresh catalog`, and the implicit ones `search` / `show references` start after any save — described every document again: 38 s on a 4,753-unit app on macOS, hours on Windows before ako/mxcli#1789, repeated after every edit in an agent loop","cause":"By design a refresh rebuilds at least the cached level (#1081, `cmd_catalog.go`), and `buildSource` had no memory of what it had described before, so the cheapest rebuild that could still be saved was the most expensive one","file":"`mdl/catalog/builder_source_reuse.go` (basis tables `source_units` / `source_elements`, `planSourceReuse`, `sourceDependents`, `sourceBasis`), `mdl/catalog/builder_source.go`, `modelsdk/mpr/reader.go` (`UnitContentHashes`), `mdl/executor/cmd_catalog.go` (`sourceFingerprint`, `catalogForce`)","insight":"**A row is reusable only if everything its describe read is unchanged, and the describe reads more than its own unit.** Three dependencies turned up, each found by a step of the equality test (incremental vs forced full build, row for row, after each kind of edit): other documents named in the refs (a page prints its data-source flow's return entity: `-- Context: $currentObject (M.ProbeA)`); the unit's container, which is a Unit-table column and not in the bytes, so a move into an existing folder kept the stale `folder` line; and the describer itself (fingerprint = executable path, size, mtime). **The equality test needs a control that breaks the dependency logic**: with `sourceDependents` stubbed to return nothing the first version of the test still passed, because none of its edits changed a dependent's text; the page-context step is what made it fail. Anything the plan cannot follow (a changed unit that is not a described document: folder, module, settings, Java action) falls back to describing everything, so reuse is an optimisation of the full build and never a different answer. FactoryManagement: one attribute added, plain refresh 11.2 s vs 38.2 s, 3,525 of 3,718 rows reused, byte-identical to `force`","refs":["ako/mxcli#1790","ako/mxcli#1789","mendixlabs/mxcli#1081"]}

CHANGELOG.md

@@ -17,6 +17,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ### Fixed
 
+- **Rebuilding a catalog that has a source index describes only what changed** (ako/mxcli#1790) — once a project had a source index (`refresh catalog full source`), every later rebuild described every document again. That includes the rebuilds `search` and `show references` start after each save. On a 4,753-unit app that was 38 s per rebuild, and much longer on Windows before #1789. A rebuild now re-describes the documents whose unit changed (an entity's unit is its module's domain model) and the documents whose references name one of them. The rest are carried over from the cache. After adding one attribute, the same app rebuilds in 11 s: 3,525 rows reused, 193 described, and the index is byte-identical to a forced full rebuild. A change outside the described documents (a folder, a module, the project settings, a Java action), a cache written by another mxcli build, and `force` still describe everything. The cache records each unit's content hash and container, so a document moved into another folder is described again too.
 - **Commands on an MPR v2 project open only the files they need** (ako/mxcli#1791) — on a 4,753-unit app, `show modules` opened 4,511 unit files, `describe microflow` 2,996 and `describe page` 3,176, in every process. The commands read every folder to build module paths and every microflow to name or find one, and `show modules` decoded every page, snippet, flow, constant and Java action just to count them. Each open is cheap on macOS and is scanned by Defender on Windows, so this added time to every command an agent runs. The unit index in `.mxcli/` now also holds each unit's name and `Excluded` flag, keyed by content hash like its type, so it never goes stale. Folders and microflows are named from the index, a microflow is found by name and read by ID, and `show modules` counts from unit types. The same commands now open 509, 93 and 274 files, with output identical across three projects and all 229 page describes. The index moves to `unit-index-2.idx`, so an older mxcli on the same project keeps its own `unit-types.idx` instead of the two rebuilding each other's.
 - **`refresh catalog full source` reads each unit file once instead of millions of times** (ako/mxcli#1789) — the source index describes every document, and each microflow describe found its microflow, and every microflow it calls, by listing every microflow of the project from disk. On a 4,753-unit MPR v2 app one refresh opened 7,532,649 `.mxunit` files: 5.5 minutes on macOS, and 90 minutes or more on Windows, where Defender scans every file open. The catalog build now keeps the units it reads in memory, and a describe reads its microflow and the microflows it calls by ID. The same refresh opens 4,753 files and takes 31 s, and its source text is byte-identical. The rebuilds that `show references` and `search` run on a project with a source index are the same build, so they speed up too.
 - **Moving a Java or JavaScript action to another module takes its source along** — `move java action A.X to B` moved the document and left `javasource/a/actions/X.java` behind (likewise a JavaScript action's `.js`). The build still succeeded, but the app called a generated stub that throws "Java action was not implemented", and `describe` printed a placeholder body that re-executing would have stored. The source now moves to the target module's directory, with a Java source's `package` line rewritten exactly as mxbuild regenerates it; an occupied destination refuses the move before anything changes.

docs-site/src/tools/refresh-catalog.md

@@ -71,6 +71,8 @@ mxcli -p app.mpr -c "REFRESH CATALOG FULL FORCE"
 
 The catalog is cached in `.mxcli/catalog.db` next to the MPR file. On subsequent refreshes, mxcli checks whether the project has changed and only rebuilds if necessary (unless `FORCE` is specified).
 
+A catalog with the MDL source index (`REFRESH CATALOG FULL SOURCE`) keeps it on every later rebuild, including the ones `SEARCH` and `SHOW REFERENCES` start when the project has changed. Those rebuilds describe again only the documents that changed, and the documents that reference them. Every other row is carried over from the cache, so a rebuild after one edit costs about as much as `REFRESH CATALOG FULL`. Everything is described again when something outside the described documents changes (a folder, a module, the project settings, a Java action), when another mxcli build wrote the cache, and with `FORCE`.
+
 ## When to Refresh
 
 - **After opening a project** -- Run `REFRESH CATALOG` to enable queries

mdl/backend/modelsdk/units.go

@@ -92,6 +92,13 @@ func (b *Backend) ListFolders() ([]*types.FolderInfo, error) {
 	return out, nil
 }
 
+// UnitContentHashes returns a hash of every unit's current contents, keyed by
+// unit ID, so a caller can tell which units changed since it last looked (the
+// catalog's incremental source index, ako/mxcli#1790).
+func (b *Backend) UnitContentHashes() (map[string]string, error) {
+	return b.reader.UnitContentHashes()
+}
+
 // AddRawUnit inserts a new unit verbatim.
 //
 // Exposed on the backend because the marketplace transplant copies a module's

mdl/catalog/builder.go

@@ -96,6 +96,12 @@ type Builder struct {
 	resolution      float64   // Leiden resolution for the graph-analysis pass
 	describeFunc    DescribeFunc
 
+	// prevSource is the last source build's rows and basis, and
+	// sourceFingerprint the mxcli build writing this one: together they let
+	// unchanged documents keep their rows (builder_source_reuse.go).
+	prevSource        *PreviousSource
+	sourceFingerprint string
+
 	// The project's DefaultLanguageCode, resolved once per build by
 	// defaultLanguage(). Texts (enumeration value captions today) are read in
 	// this language: see language.go.

mdl/catalog/builder_source.go

@@ -87,14 +87,17 @@ func (b *Builder) buildSource() error {
 	// Phase 1: Collect all items to describe (fast — uses cached lists)
 	var items []sourceItem
 
-	// Entities
+	// Entities. An entity is described from its module's domain model, the
+	// unit whose hash says whether it changed.
+	entityUnit := map[string]string{}
 	dms, err := b.cachedDomainModels()
 	if err == nil {
 		for _, dm := range dms {
 			moduleID := b.hierarchy.findModuleID(dm.ContainerID)
 			moduleName := b.hierarchy.getModuleName(moduleID)
 			for _, ent := range dm.Entities {
 				items = append(items, sourceItem{SourceEntity, moduleName + "." + ent.Name, moduleName, string(ent.ID)})
+				entityUnit[string(ent.ID)] = string(dm.ID)
 			}
 		}
 	}
@@ -204,11 +207,26 @@ func (b *Builder) buildSource() error {
 		return nil
 	}
 
+	// Carry over the rows whose basis has not changed (#1790); describe the
+	// rest. Without unit hashes nothing is reused and nothing is recorded.
+	units := make([]string, len(items))
+	for i, it := range items {
+		units[i] = sourceUnitOf(entityUnit, it)
+	}
+	hashes := b.sourceBasis()
+	plan := b.planSourceReuse(items, units, hashes)
+	toDescribe := make([]sourceItem, 0, len(items))
+	for i, it := range items {
+		if _, ok := plan.reused[i]; !ok {
+			toDescribe = append(toDescribe, it)
+		}
+	}
+
 	// Phase 2: Generate MDL source in parallel. This is the slow phase on large
 	// projects (one describe per document), so report progress periodically —
 	// from a single ticker goroutine, the only caller of report() during the
 	// run, so the progress sink isn't written concurrently.
-	total := len(items)
+	total := len(toDescribe)
 	stop := make(chan struct{})
 	var done atomic.Int64
 	go func() {
@@ -224,7 +242,7 @@ func (b *Builder) buildSource() error {
 		}
 	}()
 
-	results, failures := runDescribes(items, b.describeFunc,
+	results, failures := runDescribes(toDescribe, b.describeFunc,
 		max(min(runtime.NumCPU(), 8), 1),
 		func(n int) { done.Store(int64(n)) })
 
@@ -248,7 +266,18 @@ func (b *Builder) buildSource() error {
 		stmt.Exec(res.item.qn, res.item.objType, res.text, res.item.moduleName, res.item.id)
 		count++
 	}
+	for i, row := range plan.reused {
+		it := items[i]
+		stmt.Exec(it.qn, it.objType, row.text, it.moduleName, it.id)
+		count++
+	}
+	if err := b.persistSourceBasis(items, units, hashes); err != nil {
+		return err
+	}
 
+	if len(plan.reused) > 0 {
+		b.report(sourceReuseSummary(len(plan.reused), len(toDescribe)), len(toDescribe))
+	}
 	b.report("source", count)
 
 	// Report what produced no row. Silence here is what let #912 sit unnoticed: