diff --git a/.claude/commands/mendix/validate-project.md b/.claude/commands/mendix/validate-project.md index 7158e03003..87f8bc6d41 100644 --- a/.claude/commands/mendix/validate-project.md +++ b/.claude/commands/mendix/validate-project.md @@ -55,6 +55,21 @@ Found 1 error(s) and 1 warning(s) | CW0xxx | Warning | Warning - should be reviewed | | CD0xxx | Deprecation | Deprecated feature usage | +## Reading CE0463 + +`mxcli docker check` checks the project as stored, and labels each CE0463 ("The +definition of this widget has changed") after re-checking a temporary copy with the +widgets resynced: + +- `stale widget definition — mxcli fix widgets fixes it`: the widget is older than its + installed package. Run `mxcli fix widgets -p app.mpr` (never bare `mx update-widgets` + on an MPR v2 project: it deletes `mprcontents/`). +- `persists after update-widgets`: a resync does not clear it; something in the stored + widget is wrong. If mxcli wrote the widget, report it. + +`--update-widgets` checks only the resynced copy and hides the stale ones, which still +fail MxBuild and `mxcli run --local`; do not use it to decide that the project builds. + ## When to Use - After making changes with mxcli to verify they're valid diff --git a/.claude/skills/debug-bson.md b/.claude/skills/debug-bson.md index f2971957bb..6d06a27c82 100644 --- a/.claude/skills/debug-bson.md +++ b/.claude/skills/debug-bson.md @@ -359,7 +359,7 @@ for t, props in crash_props.items(): **Investigation methodology used for v0.10 CE0463 fixes** — see [WIDGET_BSON_VERSION_COMPATIBILITY.md](../../docs/03-development/WIDGET_BSON_VERSION_COMPATIBILITY.md) for the full case study and version-resilience model. -**Quick workaround** (if you can't fix the root cause): normalize with `mxcli fix widgets -p app.mpr`, which runs the widget update on the project **and preserves MPRv2 storage**. Not `mxcli docker check`/`build`: they run the update on a **temporary copy**, so a CE0463 the normalization clears is not reported while the stored project still fails MxBuild — use `docker check --no-update-widgets` to see the project as stored (and [`diagnose-ce0463.md`](./diagnose-ce0463.md) before calling it widget drift). Do **not** run bare `mx update-widgets` on a v2 project you care about — it converts to single-file v1 and deletes `mprcontents/` (corrupts git, breaks `mxcli run --local`). Raw `mx update-widgets` is fine only on a throwaway diagnostic copy or a v1 project. +**Quick workaround** (if you can't fix the root cause): normalize with `mxcli fix widgets -p app.mpr`, which runs the widget update on the project **and preserves MPRv2 storage**. `mxcli docker check` does not fix anything: it checks the project as stored and labels each CE0463 the normalization would clear as a *stale widget definition* (it runs the update on a **temporary copy** to find out), and one it would not as *persists after update-widgets* — read [`diagnose-ce0463.md`](./diagnose-ce0463.md) before calling either widget drift. `docker build` and `docker check --update-widgets` check only the normalized copy, so they pass while the stored project still fails MxBuild. Do **not** run bare `mx update-widgets` on a v2 project you care about — it converts to single-file v1 and deletes `mprcontents/` (corrupts git, breaks `mxcli run --local`). Raw `mx update-widgets` is fine only on a throwaway diagnostic copy or a v1 project. ### CE0642: Property X Is Required @@ -439,7 +439,7 @@ for _, pm := range a.ParameterMappings { 2. **Mode-dependent properties must be consistent**: When changing a mode-switching property (e.g., `showContentAs`), all dependent properties must be updated to match. -3. **Widget normalization is the safety net — via `mxcli fix widgets`**: it applies the update-widgets normalization to the project *and* preserves MPRv2 storage. `mxcli docker check`/`build` normalize only a temporary copy, which hides CE0463 without fixing the stored project. Bare `mx update-widgets` does the same normalization but rewrites a v2 project to v1 and deletes `mprcontents/` — only use it on a v1 project or a throwaway diagnostic copy. +3. **Widget normalization is the safety net — via `mxcli fix widgets`**: it applies the update-widgets normalization to the project *and* preserves MPRv2 storage. `mxcli docker build` (and `docker check --update-widgets`) normalize only a temporary copy, which hides CE0463 without fixing the stored project; a default `docker check` reports the stored project's CE0463 and labels the ones `fix widgets` clears. Bare `mx update-widgets` does the same normalization but rewrites a v2 project to v1 and deletes `mprcontents/` — only use it on a v1 project or a throwaway diagnostic copy. 4. **The mpk is the source of truth**: The XML schema defines property types/defaults, the editorConfig.js defines visibility rules. Together they specify the complete expected Object structure. diff --git a/.claude/skills/diagnose-ce0463.md b/.claude/skills/diagnose-ce0463.md index ff7b2e84c8..6ba15ae9a9 100644 --- a/.claude/skills/diagnose-ce0463.md +++ b/.claude/skills/diagnose-ce0463.md @@ -9,21 +9,35 @@ Related: [`debug-bson.md`](debug-bson.md) for the general BSON diff workflow; [`WIDGET_BSON_VERSION_COMPATIBILITY.md`](../../docs/03-development/WIDGET_BSON_VERSION_COMPATIBILITY.md) for what is version-fragile and the per-minor onboarding record. -## Before you measure anything: `docker check` clears CE0463 by default +## Before you measure anything: which `docker check` you are reading -`mxcli docker check` runs **`mx update-widgets` before `mx check`**, deliberately — it -normalizes pluggable widget definitions to suppress the Case A noise below. So the plain -command reports **0 errors** on a project that genuinely has a CE0463, and the repair -also *mutates the project*, leaving it clean for any later check too. +`mxcli docker check` checks the project **as stored** (ako/mxcli#568, #646), and when +it finds CE0463 it already runs control 2 below for you: it re-checks its temporary +copy after `mx update-widgets` and labels each CE0463 — + +``` + at Data grid 2 'dgOrders' (4×): stale widget definition — `mxcli fix widgets` fixes it + at Combo box 'cbStatus': persists after update-widgets +``` + +A *stale* one is Case A; one that *persists* is the candidate for Case B. The project +itself is never touched (everything runs on a copy). Two flags change what it shows: ```bash -mxcli docker check -p app.mpr --no-update-widgets # the only form that can see CE0463 +mxcli docker check -p app.mpr # stored project + labels (default) +mxcli docker check -p app.mpr --no-update-widgets # stored project, no re-check, no labels +mxcli docker check -p app.mpr --update-widgets # ONLY the resynced copy: hides Case A ``` -This is documented in `docker check --help` and is not a defect, but it is the first -thing that goes wrong in a CE0463 investigation: you check, you see zero errors, and you -conclude there is nothing to find. Measured on 11.13.0: a page mxcli wrote reported -`0 errors / Project check passed` with the default and **1 error, CE0463** with the flag. +`--update-widgets` was the default until #646, and it is the first thing that went wrong +in CE0463 investigations: it reports 0 errors on a project that genuinely has a CE0463 +(measured on 11.13.0: a page mxcli wrote, `0 errors / Project check passed` that way and +**1 error, CE0463** as stored). Findings and bug reports written before #646 that say +"`docker check` passed" mean the resynced copy passed. + +mx check's text output does not name the page, so same-named widgets on different pages +print the same line; the labels count them (`(4×)`, `(2 of 3)`). For page names, run +`mx check -j out.json` on a copy (10.24+; 9.24's mx has no `-j`). **`mxcli check` is not an alternative.** It validates an MDL *script* — syntax, references, creation order, expression types — and never invokes mxbuild, so it cannot @@ -47,7 +61,8 @@ These look identical in `mx check` output and have unrelated causes. 2. **Does `mx update-widgets` clear it?** If yes, the BSON was structurally valid and correct for the version it was written against → Case A. Genuine template bugs do *not* clear this way (the Image stale default and the number-filter markerless - array both needed template fixes). + array both needed template fixes). A default `mxcli docker check` answers this per + CE0463 (*stale widget definition* vs *persists after update-widgets*); by hand: ```bash cp -r proj proj-ref && mx update-widgets proj-ref/App.mpr # ALWAYS on a copy diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-29-docker-check-passed-a-widget-normalised-copy-not-the-stored.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-29-docker-check-passed-a-widget-normalised-copy-not-the-stored.json new file mode 100644 index 0000000000..cdb621c137 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-29-docker-check-passed-a-widget-normalised-copy-not-the-stored.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli/docker", "date": "2026-10-29", "symptom": "`mxcli docker check` reports 0 errors and exits 0, then `mxcli run --local` refuses to start with CE0463 \"The definition of this widget has changed\" at the same widgets; `docker check --no-update-widgets` reports them. A gate that says yes about a model it did not examine", "cause": "`Check` ran `mx update-widgets` on its temporary copy before `mx check` by default, to suppress \"false CE0463\" — so it validated a widget-normalised model, never the stored one. The CE0463s it cleared were real: MxBuild and `run --local` build the project as stored. An earlier fix only added a closing note saying so; the verdict and exit status stayed the normalised copy's", "file": "`cmd/mxcli/docker/check.go` (`Check`, `recheckCE0463`), flags in `cmd/mxcli/docker.go` (`--update-widgets`, `--no-update-widgets`)", "insight": "Disclosing a normalisation in the output does not fix a gate: callers read the exit status, not the note. The fix moves diagnose-ce0463's second control into the gate: check the stored project (its verdict and exit status), and only when that found CE0463 run update-widgets on the same copy and re-check, labelling each \"stale widget definition — `mxcli fix widgets` fixes it\" or \"persists after update-widgets\" — no second check on a clean project. mx check's text output does not name the page and a blank 11.13 app has four `dataGrid2_1` and four `dataGrid21` on different pages, so the lines are compared as counts (`(4×)`, `(2 of 3)`); `-j` would give module/document, but 9.24's mx rejects it (\"Option 'j' is unknown\") before checking anything. A stale instance needs no second package version: add one property to the widget XML inside the project's own .mpk (Data grid 2 3.4.0 + a boolean → 11 CE0463 on a blank 11.13.0 app, all cleared by update-widgets). Measured: old default 0 errors / exit 0; new default 11 errors / exit 1, all 11 labelled stale; `--update-widgets` 0 / exit 0 (old behaviour kept as opt-in); after `mxcli fix widgets` (11 units, MPR v2 kept) the default reports 0. The project tree's hash was unchanged by every run. Tests that pinned update-widgets-before-check as the default now pass `UpdateWidgetsFirst`", "test": "`cmd/mxcli/docker/check_stored_test.go` (`TestCheck_ChecksTheStoredProjectByDefault`, `TestCheck_LabelsCE0463ThatPersistsAfterUpdateWidgets`, `TestCheck_NoCE0463NoSecondCheck`), `cmd/mxcli/docker/check_stored_integration_test.go` (`TestCheck_ReportsStaleWidgetCE0463OfTheStoredProject`, real mx), `cmd/mxcli/docker_help_test.go` (`TestDockerCheckHelp_StoredProjectIsTheDefault`)", "refs": ["ako/mxcli#568", "ako/mxcli#646"], "ce": ["CE0463"]} diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-29-legacy-runner-retrieve-declare-variable-not-renamed.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-29-legacy-runner-retrieve-declare-variable-not-renamed.json new file mode 100644 index 0000000000..bdc452399a --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-29-legacy-runner-retrieve-declare-variable-not-renamed.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-10-29", "symptom": "On the after-startup runner (every Docker run, `--local --legacy-runner`) two tests in one file that define the same variable with `retrieve $x`, `declare $x`, a list operation or aggregate (`$x = head(…)`, `$x = count(…)`), `send rest request`, `execute database query` or a loop iterator fail injection with MDL063 (MDL052 for the iterator), CE0111 \"Duplicate variable name\" from mxbuild. Measured by sweeping every non-.fail test file through GenerateTestRunner + ValidateProgram: setup-annotation.test.mdl (`retrieve $Brands` x3) and 1103-test-file-checks-as-microflow-bodies.test.mdl (`retrieve $Request` + `$Request = head(…)`) failed; the endpoint runner was never affected", "cause": "The runner joins every test into one microflow and renames each test's variables with `_N`, but the rename set came from `assignPattern`, a regex matching `$x = call|create` only. Every other way to define a variable kept its name and was declared once per test", "file": "`cmd/mxcli/testrunner/generator.go` (`extractVariableNames` now parses the body and takes `executor.FlowDefinedVariables`, new in `mdl/executor/validate_microflow_ce_gaps.go`); `generator_endpoint.go` (`bodyParsePrefix`, shared with `attachOnError`)", "insight": "Ask the validator what a body defines instead of keeping a second list: `statementProducedVars` (reflective on `OutputVariable`, plus declare/create/create list/retrieve/loop iterator) walked with `ast.StatementBodies` is exactly what MDL063 counts, so the rename set and the gate cannot drift. The wrong turn the previous finding suggested is `visitor.FlowStatements`: it yields grammar-rule names and offsets, not variable names, so it would need a hand-kept list of producing rules, a regex by another name. Sweep on MDL052 as well as MDL063: a reused loop iterator is CE0111 but reported by MDL052, and a sweep filtered on MDL063 alone shows it green. Over-renaming costs nothing (a void call's output, a String contains/find the builder turns into Change Variable) while under-renaming is CE0111, so the rename set deliberately skips MDL063's type-dependent exemptions. `$x = …` without `set` is a Change Variable (setStatement is the last alternative), not a definition, and keeps the old \\b guard's intent for free. The sweep is now a test (TestLegacyRunnerRepositoryTestFilesDefineEachVariableOnce); control: the regex restored fails 15 shape cases and 3 repository files with MDL063/MDL052", "refs": ["ako/mxcli#1775", "ako/mxcli#1753"], "rules": ["MDL063", "MDL052"], "ce": ["CE0111"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-alter-members-of-list-op-java-self-reference-outputs.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-alter-members-of-list-op-java-self-reference-outputs.json new file mode 100644 index 0000000000..812740c427 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-alter-members-of-list-op-java-self-reference-outputs.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-10-29","symptom":"ako/mxcli#1777 (follow-up to #1769): `alter microflow … { insert after $Related begin change $First (Status = 'Seen'); end; }` with $First = head $Lines in the stored flow was refused, nothing written: \"cannot qualify member \\\"Status\\\": the entity its variable holds is not known here\". The same for any member (find/filter by, sort, change) of a stored filter/sort/tail/union/intersect/subtract/range/head/find output, a Java action result (entity, list, list of a type parameter), or a retrieve over a self-reference. `create microflow` types all of them.","cause":"alterFlowContext.storedVariables, the scope a spliced fragment starts from, left ListOperationAction, JavaActionCallAction and a self-reference AssociationRetrieveSource 'Unknown'; associationRetrieveType returned \"\" for a self-reference because 'which end it starts from is not known'.","file":"mdl/executor/cmd_alter_flow.go (storedVariables, storedDependent, storedDependents.typeOf), mdl/executor/cmd_microflows_builder_actions.go (listOperationResult, listResult.typeFrom, listOperationInput, aggregateResultType, associationRetrieveType), mdl/executor/cmd_microflows_builder_calls.go (javaActionResultType, returnsTypeParameterList, firstListType, javaActionArgVariables)","fix":"The full build's typing was pulled out of addListOperationAction, addAggregateListAction and addCallJavaActionAction into functions over the stored shapes (the microflows.ListOperation, the AggregateFunction + ReduceReturnType, the JavaAction definition + argument variables), and storedVariables calls the same ones: a list operation's output and a Java action's result are storedDependents typed in the fixed point, so one reading a list another output holds is typed whatever the stored order, and a name two origins give different derivations stays untyped (a head and a find of one list are one origin). Aggregates get the primitive the full build declares. A self-reference is followed from its FROM end.","test":"mdl/executor/flow_splice_stored_outputs_test.go (TestStoredVariables_ListOperationJavaAndSelfReferenceAreTyped, TestSpliceFragment_MembersOfStoredOutputsAreQualified: 13 shapes, TestSpliceFragment_MembersOfConflictingOutputsAreRefused); flow_splice_assoc_retrieve_test.go $Parent now Mod.Order. Controls: storedVariables and associationRetrieveType reverted -> all 12 new shapes refused 'cannot qualify member' (the self reference set was already typed); conflict guard off -> $Dup typed Mod.Tag, $Mixed Mod.OrderLine, both written. Platform 11.13.0: 7 alters refused before, all 8 written after, mx check 0 errors, describe -> exec Unchanged.","insight":"The stored AssociationRetrieveSource has no direction field (only StartVariableName and the association), so 'which end a self-reference starts from' is not a missing fact to be guessed but a platform rule to be measured: on 11.13.0 `retrieve $P from $Order/M.Order_Parent; change $P (…)` builds and `count($P)` is CE0013 \"Input variable 'P' must be of type 'List'\" — Mendix follows it from the FROM end. Owner Both on a self-reference is CE0002, so that case does not exist. The full build types a self-reference by use (a list unless an object use follows), which disagrees with the platform for a list use; the fragment takes the platform's reading. To type stored outputs, refactor the builder's typing to take the stored action rather than the AST statement, then call it from both sides — the builder's switch on ast.ListOpX was the only copy and would have drifted.","ce":["CE0013","CE0002"],"refs":["ako/mxcli#1777","ako/mxcli#1769","ako/mxcli#885"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-exec-silent-about-files-beside-the-model.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-exec-silent-about-files-beside-the-model.json new file mode 100644 index 0000000000..8b13e1560a --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-exec-silent-about-files-beside-the-model.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-10-29", "symptom": "`mxcli exec` of `create java action Probe.JA_Hello () returns String as $$ return \"hi\"; $$;` printed only \"Created java action: Probe.JA_Hello\" while also writing javasource/probe/actions/JA_Hello.java; `git status` then showed the .java (and a JavaScript action's .js under javascriptsource/) untracked, so a workflow that commits the model (.mpr + mprcontents/) left the body behind and a clone got an action with no logic", "cause": "Nothing reported writes beside the model. The backend already knew each one — `WriteSourceIfChanged` returns whether the content changed and `noteFileWrite` counted it, but only as a number feeding the Unchanged decision; the path never left the backend, and each handler printed its model-element sentence on its own", "file": "`mdl/backend/source_files.go` (SourceFileChange, SourceFileReporter); `mdl/backend/modelsdk/backend.go` (noteSourceFile, TakeSourceFileChanges), `java_write.go`, `javascript_write.go`, `module_write.go` (removeModuleSourceDirs); `mdl/executor/source_file_report.go`, drained in `executor_dispatch.go` executeInner, summary in `flushTally`", "insight": "Record the change where the bytes move, not in the handler: only the writer knows the path and whether the content actually differed, so a handler-side `Wrote …` would print for an identical rewrite (the reports-that-outrun-the-write shape). One drain after every statement in executeInner then covers every writer — create/modify, rename (Moved a to b), drop (Removed), DROP MODULE's folder removal — with no per-handler print to forget. Writers found by grepping os.WriteFile/Rename/Remove/RemoveAll under mdl/ and sdk/: java_write.go, javascript_write.go, module_write.go; widget_defs.go also writes during exec (.mxcli/widgets/*.def.json, and widget docs on refresh catalog), deliberately not reported — gitignored, regenerable caches that `mxcli git commit` unstages anyway. `exec` has no JSON mode, so the reporter's third suggestion does not apply. Control: with the drain disabled, an 11.13.0 app shows the issue exactly (model elements only, both source files untracked). Not fixed here, measured on 11.13.0: `move java action MvA.JA_Mv to MvB` leaves the body at javasource/mva/actions/JA_Mv.java and `describe` of MvB.JA_Mv says the Java source is not available; and exec's rollback restores only .mpr/mprcontents, so a failed run's 'Nothing was written' is untrue of these files", "refs": ["ako/mxcli#1618", "mendixlabs/mxcli#1634"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-move-action-source-left-behind.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-move-action-source-left-behind.json new file mode 100644 index 0000000000..3721ebd586 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-move-action-source-left-behind.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-10-29", "symptom": "`move java action A.X to B` (and `move javascript action`) reparented the document and left javasource/a/actions/X.java (javascriptsource/a/actions/X.js) behind. DESCRIBE then printed the 'Java source not available' placeholder, so re-executing its output replaced the code. Measured on 11.13.0: mx check passed and `docker build` succeeded, but the class the app calls, jamb/actions/JA_Hello, was mxbuild's generated stub throwing 'Java action was not implemented', and the author's code compiled only as an orphaned jama.actions class; the page bundle for a JS action carried 'JavaScript action was not implemented'", "cause": "execMove's movers change containment only; RenameJavaSourceFile existed for a rename within the module, but nothing moved the source between module directories", "file": "`mdl/executor/cmd_move.go` (`moveActionSource`, run before the document moves, undone by a deferred move back if the statement fails), `mdl/backend/modelsdk/java_write.go` (`MoveJavaSourceFile`, `moveSourceFile`), `javascript_write.go` (`MoveJavaScriptSourceFile`), `sdk/javaactions/rename.go` (`MoveSource`, `rewriteGenerated`)", "insight": "A green build is not evidence here: mxbuild regenerates every action stub where the model puts it and compiles the old file too, so the move only shows as a runtime throw - read the compiled class out of project.jar (`strings jamb/actions/JA_Hello.class`) or grep the page bundle, and a JS action is only bundled when a nanoflow on a page calls it. The oracle for the source rewrite is mxbuild run IN PLACE on a source moved with nothing else changed: it rewrote exactly one line, `package jama.actions;` -> `package jamb.actions;`, and kept user code that mentions the old package (testdata/mxbuild/JA_Moved*). Move the source before the document so an occupied destination refuses the statement with the model unchanged", "refs": []} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-odata-metadata-url-rewritten-per-checkout-path.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-odata-metadata-url-rewritten-per-checkout-path.json new file mode 100644 index 0000000000..0d5c4ecdf0 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-odata-metadata-url-rewritten-per-checkout-path.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-10-29","symptom":"Re-executing an unchanged script from a checkout at another path rewrote every OData client: `create or modify odata client … ( MetadataUrl: './contracts/f1-metadata.xml' )` had stored `file:///workspaces/…/contracts/f1-metadata.xml` and now stored `file:///home/user/…/contracts/f1-metadata.xml` (5 of 20 writes in the reported project). `describe` printed the absolute form, so describe → exec spread the authoring machine's path into scripts. mx check, build and runtime all clean","cause":"The modify branch normalized the statement's relative path to this checkout's absolute file:// URL and assigned it unconditionally, so the stored value always followed whichever checkout ran the script. describe printed the stored absolute URL as is","file":"`mdl/executor/cmd_odata.go` (`metadataURLToStore`, `namesSameMetadataFile`, `isRelativeMetadataPath`, `describedMetadataURL`; `refreshCachedMetadata` now takes the fetch URL); test `mdl/executor/cmd_odata_metadata_url_checkout_test.go`; repro `mdl-examples/bug-tests/1499-odata-metadata-url-checkout-path.mdl`","insight":"**mx check cannot arbitrate the stored form**: on 11.13.0 it reports 0 errors for './contracts/x.xml', 'contracts/x.xml' AND a dead 'file:///nonexistent/…' alike, so it never reads the value; mx has no headless contract refresh; no Studio Pro-authored ConsumedODataService exists in testdata/, PedApp or mdl-examples. Storing a relative URL would trade a measured defect for an unmeasured Studio Pro one (the skill says its metadata editor keys URL-vs-File on the file:// scheme), so the absolute form stays and the write is elided instead. **'Same file' across checkouts needs the filesystem as tiebreak**: the stored path must end with the statement's project-relative path, and the directory it leaves must either not exist here (another machine, a moved checkout) or hold this project's .mpr (a git worktree); a directory that exists without it (a shared contracts folder) is a different file and the statement moves the client off it. Without the worktree rule two worktrees on one machine flip-flop forever, which is the reported bug again. **Read the contract from this checkout's file, not the kept URL**, or keeping a foreign path costs the refresh. Platform: fresh 11.13 app, exec at P1, mv to P2 and cp to P3: 'documents already in sync' both times (old binary: 'Modified OData client'), mx check 0 errors. describe is project-relative only when './rel' normalizes back to exactly the stored URL, so exec(describe) stays Unchanged","refs":["ako/mxcli#1499"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-source-catalog-build-rereads-every-microflow-per-describe.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-source-catalog-build-rereads-every-microflow-per-describe.json new file mode 100644 index 0000000000..8766a81013 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-source-catalog-build-rereads-every-microflow-per-describe.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-10-29","symptom":"`refresh catalog full source` took 5.5 min on a 4,753-unit MPR v2 app on macOS and 90+ min for users on Windows, where `full` alone takes 9 s; agent-driven builds on Windows ran hours behind the same input on a Mac. The cost grows with the square of the project","cause":"Each source describe (up to 8 in parallel) found its microflow with `microflowsNamed`, which lists every microflow unit, and the duplicate-output-variable warning resolved each called flow with `GetRawUnitByName`, which lists every unit of the type; `GetMicroflow(id)` listed them all too. Nothing held the reads, so one refresh opened 7,532,649 .mxunit files. An open costs microseconds on a warm macOS cache and around a millisecond under Windows Defender's real-time scan","file":"`mdl/executor/cmd_catalog.go` (`buildCatalog`: `CacheUnitReads` scope over the whole build), `modelsdk/mpr/reader_units.go` (`Reader.UnitByID`), `modelsdk/mprread/generic.go` (`UnitByID[T]`) behind `mdl/backend/modelsdk/microflow.go` `GetMicroflow`, `mdl/executor/helpers.go` (`pinnedMicroflow`, used by `describeMicroflowMode` when the build pinned `describeID`), `mdl/executor/validate_void_code_calls.go` (`microflowID`: called microflows read by ID)","insight":"**Count file opens, not seconds: the platform gap is per open.** The same refresh is minutes on a Mac and hours on Windows, so a wall-time report from one platform says nothing about the other, and a Mac benchmark hides the bug. A temporary build with a global counter next to `r.fileReads.Add(1)` gave the platform-free number; the CPU profile (83% syscall under `listUnitsByTypeV2In`) named the two callers. The narrow lookups cut the reads 9,818 → 4,092 on pedapp, and the profile then showed a tail of other per-describe listings (Java action lookups via `containerSetForModule`, `describeNanoflow`'s `ListMicroflows`, the XPath enricher's `ListDomainModels`); fixing them one by one would not end, so the build-wide `CacheUnitReads` is what bounds opens at one per unit (11,017 → 409 on pedapp). A unit's ID is its document's $ID on both formats (checked on a v1 and a v2 project), which is what makes a by-ID read possible without any listing. Guard: `TestSourceCatalogBuild_ReadsUnitFilesLinearly` (reads ≤ units, failing at 11,017 without the fix) and `TestUnitByID_AgreesWithTheListing` on v1 and v2","refs":["ako/mxcli#1789","ako/mxcli#1790","ako/mxcli#1791","mendixlabs/mxcli#1272"]} diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-mpr-v1-unit-write-leaves-contents-hash-stale.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-mpr-v1-unit-write-leaves-contents-hash-stale.json new file mode 100644 index 0000000000..0e2df11c3b --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-mpr-v1-unit-write-leaves-contents-hash-stale.json @@ -0,0 +1 @@ +{"area":"modelsdk/mpr","date":"2026-10-29","symptom":"On an MPR v1 project (unit contents inside the .mpr, Mendix below 10.18) a unit write replaced Unit.Contents and left Unit.ContentsHash describing the old bytes, and an inserted unit got ContentsHash ''. `mx check` 0 errors and MxBuild clean; Studio Pro reads the hash to detect what changed. `mxcli fix hashes` said v1 'has no index to drift' and exited 0","cause":"The v2 branches of updateUnit, WriteTransaction.WriteUnit and insertUnit computed base64(SHA-256) and stored it; the v1 branches wrote `UPDATE Unit SET Contents = ?` and `INSERT … ContentsHash = ''`, on the assumption that the column only indexes mprcontents files. It does not: Studio Pro fills it on every v1 unit too","file":"`modelsdk/mpr/writer_core.go` (v1 branches of `updateUnit`, `WriteTransaction.WriteUnit`, `insertUnit` now set ContentsHash = `contentsHashOf(contents)`), `modelsdk/mpr/contents_hash.go` (`verifyContentsHashesV1`, report `Version`, issue `Unit`), `cmd/mxcli/cmd_fix_hashes.go` (`reportV1Hashes`; the exec/docker drift warning still skips v1); tests `modelsdk/mpr/writer_v1_contents_hash_test.go`, `contents_hash_test.go`, `cmd/mxcli/cmd_fix_hashes_test.go`; repro `mdl-examples/bug-tests/1159-mpr-v1-contents-hash.mdl`","insight":"**Measure the formula on a Studio Pro-authored v1 file before writing it**: a fresh `mxcli new --version 9.24.30.51827 --skip-build` app stores standard padded base64 of SHA-256 over the Contents blob exactly, 308 of 308 units, and `testdata/v1-project/App.mpr` 20 of 20, so that fixture doubles as the control (`TestV1FixtureHashesMatchContents`). Platform run: the old binary left 2 stale rows after `create entity` + `create microflow` (DomainModel old hash, new microflow ''), the fixed one 0; `fix hashes --repair` repairs the stale copy. **The reporter's `add jar dependency` cannot reproduce on 9.24** (JarDependencies is 10.0+, refused up front), so use any rewrite plus any insert. **Testing WriteTransaction.WriteUnit on v1 through `NewWriter` hangs forever**: the reconcile reads the stored bytes through the reader's pool while the transaction holds its only connection (`SetMaxOpenConns(1)`), which on v1 (bytes in the same DB) deadlocks in `GetRawUnitBytes`. That is a separate defect of codec.Store.SaveUnit/FlushUnits on v1; the test uses an unlimited pool. **Keep the per-exec drift warning v2-only**: verifying v1 reads every blob in the .mpr (140 MB on a large app), and nothing but a writer can make a v1 row drift","refs":["ako/mxcli#1159","ako/mxcli#972"]} diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-property-floor-was-lowest-measured-version.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-property-floor-was-lowest-measured-version.json new file mode 100644 index 0000000000..3ff6e8d1ff --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-property-floor-was-lowest-measured-version.json @@ -0,0 +1 @@ +{"area": "modelsdk/canon", "date": "2026-10-29", "symptom": "On Mendix 10.24.18 (and every 10.24 patch below 10.24.24) a nanoflow re-created with `drop nanoflow` + `create nanoflow` made `mx diff` exit 129: `Objects with ID 71e6faf6-… of type Microflows$Nanoflow do not have the same properties. baseNames = …, ExportLevel, …, ReturnVariableName; newNames = …`; a fresh microflow and entity lacked AggregateAction UseExpression/Expression/ReduceInitialValueExpression/ReduceReturnDataType, DeleteAction.ErrorHandlingType, WhileLoopCondition.Caption, DatabaseRetrieveSource.XpathConstraint and EntityImpl Image/Source/ImageData; `mxcli fix property-sets --dry-run` said `Every unit already has the property names Studio Pro writes.` mx check 0 errors throughout (ako/mxcli#1505, mendixlabs/mxcli#1595)", "cause": "Every one of those keys was a row of studiopro_property_defaults.json with since 10.24.24, the lowest version the table had been MEASURED on (CI's 10.24 is 10.24.24), and propertyDefault.appliesTo reads since as the version that introduced the key, so 10.24.0–10.24.23 got none of them. fix property-sets judged completeness by the same table, so it could not see what the table did not cover", "file": "`modelsdk/canon/studiopro_property_defaults.json` (94 rows to since 10.24.0, measured), `modelsdk/mpr/property_set_census.go` + `cmd/mxcli/cmd_fix_property_sets.go` (MIXED report); tests `modelsdk/canon/propertyset_issue1505_test.go`, `TestFixPropertySets_ReportsAKeyNoMeasuredValueFills`, `modelsdk/mpr/property_set_census_test.go`; bug-test `mdl-examples/bug-tests/1505-property-sets-below-10-24-24.mdl`", "insight": "**A table floor is only as low as the oldest version measured: measure the lowest patch of a line (10.24.0), not the one CI runs.** **Strip-and-convert measures a row without needing a writer to leave it out**: delete the row's key from every element of a blank Studio Pro app (plus probe content for types the template lacks), `fix hashes --repair`, `mx convert -p -s` a copy, and every key Mendix writes back is a measurement with its value. It measured 53 rows on 10.24.0 and 10.24.18 in minutes, including ~30 the doctype gate cannot see because today's writers emit them (gaps there come only from units an older mxcli stored). Two traps: never strip a key whose value holds elements (a non-empty Widgets or Items list), which deletes the subtree being measured so convert re-adds only empty lists; and convert without -s fails its check on stripped values. Feed the result to propdefaults as a Recording line, so Merge refuses a value that varies. **The doctype gate is blind on a script with a marketplace dependency on 10.24**: after `mx module-import` it logs `MPR v1 project, nothing to compare per unit` and passes; 13-business-events did, while its BusinessEventService lacked Document. **The project is its own reference for property NAMES, not values**: Studio Pro writes the same key set on every element of a $Type (0 mixed of 115–252 types on pedapp, expr-checker, testapp-views and blank 10.24.0/10.24.18 apps), so fix property-sets now reports any name the elements of a type disagree on (MIXED). With the old table it named 7 of the issue's 10 type.keys on 10.24.18 (the other 3 had no sibling element in a blank app), and it fills none, since a value from another version or element is a guess. Suggestion 3 (carry the dropped nanoflow's header) was skipped: once the completion writes the names, drop + create is a new document and Hidden is its right ExportLevel. After the fix `mx convert` still rewrites mxcli's units with 0 key differences: key ORDER only, which the merge does not compare", "refs": ["ako/mxcli#1505", "mendixlabs/mxcli#1595", "mendixlabs/mxcli#1373"]} diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-type-newer-than-project-written.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-type-newer-than-project-written.json new file mode 100644 index 0000000000..af8eb4a1ba --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-type-newer-than-project-written.json @@ -0,0 +1 @@ +{"area":"modelsdk","date":"2026-10-29","symptom":"A $Type newer than the project was written with no version gate, and the project then did not load: exec reported success, then mx check 9.24.30 died with `TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$WaitForNotificationActivity` (a workflow with `wait for notification`) and `… Microflows$GetWorkflowsAction` (a microflow with `get workflows`). The fourth-plus recurrence of one class: #1754, #1209, #1163, #1771","cause":"The generated version data (modelsdk/version, from modelsdk/gen/*/version.go) holds property versions only: version.TypeVersionInfo has no field for a type's own introduction, and the generated packages cannot be regenerated (hand fixes, #1163). So the write path compared every key with the project's version and never an element's $Type","file":"`modelsdk/canon/typefloor.go` (UndeclaredTypeError, TypeIntroduced, TypeDeclared), `modelsdk/canon/mendixmodelsdk_type_versions.json` + `gen_type_versions.go` (extracted from mendixmodelsdk 4.117.0), `modelsdk/mpr/writer_property_sets.go` (normalizePropertySetsFor: after the lowering and the property floor, on the bytes that land), `mdl/executor/version_gates.go` (workflowTypeGates, workflowDefinitionSelectionGate)","insight":"**Key the type floor by STORAGE name, and let an alias only loosen it.** The SDK names types by qualified name; the codec's storage aliases (modelsdk/gen/*/types.go SetTypeName) map them, but a storage name only an alias reaches can belong to an older type the SDK does not list: the first extraction floored Forms$FormCallArgument at 9.7 (alias of the SDK's PageParameterMapping) and refused 311 units Studio Pro 9.1.1 and 9.4.0 wrote. Each row is now the union of every SDK type stored under its name, and a name only an alias reaches gets no row. **The oracle for the data is Studio Pro-authored bytes across versions, and the Marketplace has a ladder of them**: Workflow Commons publishes a version per Mendix minor (9.24, 10.0, 10.3, 10.11, 10.12, 10.14, 10.17, 11.0, 11.2, 11.8, 11.9, 11.11, 11.13, 11.16), each .mpk holding a project.mpr; plus samples 117680 (9.4.0), 117785 (9.1.1), 221741 (9.24.3) and blank apps from the cached mxbuilds. `MXCLI_TYPEFLOOR_PROJECTS=: go test ./modelsdk/mpr -run TestTypeFloor_StoredProjectsHoldNoUndeclaredType` scans them: 23 projects, 0 refusals, after the alias fix. **A cached mxbuild is a second oracle, and needs no project**: `strings -a Mendix*.dll | tr -c 'A-Za-z0-9_\\n' '\\n' | sort -u` lists every class name a version has. WaitForNotificationActivity, WaitForTimerActivity, the three timer boundary events, both WorkflowDefinition*Selection types, GetWorkflowsAction and NotifyWorkflowAction are absent from 9.24.30's and present in 10.24.24's; of the 255 rows floored above 9.24, 224 short names are absent there. The 31 'present' are short-name collisions across a domain move (Rest$EntitySet is ODataPublish$EntitySet from 10.4, measured unloadable on 9.24 in #1163) or generic names (Line, Value, Source), so key on the domain before trusting a hit. **Do not enforce the SDK's deletion versions.** The same class-name list shows 56 SDK-deleted types still classes in 11.15.0's mxbuild, Workflows$UserTask, XPathBasedUserSource and TimerBoundaryEvent among them (kept to load and convert older models). A rename keeps the storage name (CallMicroflowTask 'deleted' 11.9 is still how every call microflow activity is stored), and mxbuild 11.x still loads Workflows$TimerBoundaryEvent, 'deleted' 10.20 (MDL-WF07 exists because only the RUNTIME lacks it). An introduction floor cannot misfire on a rename (the new name is never stored), a deletion floor can. **Order matters for the messages**: the type floor runs after the property floor, so a key a version lacks is still named as the key (CallMicroflowTask.BoundaryEvents on 10.13), and only a type with no older shape is refused as a type. Its first catch in the existing tests was a CONTROL that was wrong: a user task's InterruptingTimerBoundaryEvent was asserted writable on 10.14, but that is a 10.20 type (10.14–10.19 store TimerBoundaryEvent). The second: TestCreateWorkflow_AIAgentTaskRoundTrip wrote Workflows$AIAgentTaskActivity (11.9, absent from the 11.6 mxbuild) to the 11.6.6 fixture; it now sets the fixture to 11.14.0, and the 11.6.6 refusal is its own test. Measured: 9.24.30 fresh apps, before → TypeCacheUnknownTypeException for WaitForNotificationActivity and GetWorkflowsAction; after → check -p MDL-VERSION01 for the first, write refusal 'cannot store Microflows$GetWorkflowsAction (Mendix 10.4.0+)' for the second, App.mpr byte-identical, mx check 0 errors. Bug-test on 10.24.24 and 11.13.0: exec and mx check identical before/after, 0 errors. Doctype suite on 9.24.30 before and after: 61 pass, the same 17 fail at the same first failure (none of them a type), and no TypeCacheUnknownTypeException either way: every script that would write a newer type is @version-gated or fails earlier, which is how the class kept shipping. Run UNFILTERED instead (every doctype script into one fresh app, exec --continue-on-error --no-check, so gates still refuse what they cover): on 9.24.30 the floor refuses Rest$ODataEntityTypeSource (10.8, external entities from OData), Microflows$RestOperationCallAction (10.1), NotifyWorkflowAction (10.0), GetWorkflowsAction and GetWorkflowActivityRecordsAction (10.4), none of them gated; on 10.24.24 it refuses no type, and none of the new gates fires. **The SDK introduction can be later than the platform class**: 9.24.30 loads a RestOperationCallAction (mx check: CE7053 \"No REST operation specified\", no load failure) though the SDK says 10.1; its target (Rest$ConsumedRestService, 10.1) is absent from 9.24, so the activity cannot be configured there and refusing it costs nothing, but a floor is a claim about loading, so measure before lowering one. Control: replacing the UndeclaredTypeError call with nil fails the five new/changed tests with the write accepted. Tests: writer_type_floor_1774_test.go, type_floor_projects_test.go, canon/typefloor_test.go, version_gates_workflow_types_1774_test.go; mdl-examples/bug-tests/1774-workflow-types-below-their-version.mdl","refs":["#1774","#1771","#1163","#1754","#1209"]} diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-v1-write-transaction-deadlock.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-v1-write-transaction-deadlock.json new file mode 100644 index 0000000000..a908403929 --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-v1-write-transaction-deadlock.json @@ -0,0 +1 @@ +{"area": "modelsdk", "date": "2026-10-29", "symptom": "On an MPR v1 project (Mendix below 10.18, contents in the .mpr) every write through the Go library's own path hung forever: codec.Store.SaveUnit, codec.Store.FlushUnits and modelsdk.Model.Flush never returned. Measured on a fresh Studio Pro-authored 9.24.30 app: SaveUnit of a changed DomainModels$DomainModel did not return within 15s. mxcli exec was unaffected, because the MDL executor writes through Writer.UpdateRawUnit, which is why none of the 9.24 doctype runs ever saw it", "cause": "WriteTransaction.WriteUnit reconciles the new bytes with the stored ones (canon.Reconcile, ADR-0008) before writing, and read the stored ones with reader.GetRawUnitBytes, i.e. through reader.db. The reader opens SQLite with SetMaxOpenConns(1), and the transaction the caller began already holds that one connection, so the read waited for it forever. On v2 the stored bytes come from mprcontents/ and the read never touched the pool, so every v2 test passed", "file": "`modelsdk/mpr/writer_core.go` (`WriteTransaction.WriteUnit` reads through `wt.tx`; `reconcileAgainst` takes the stored-bytes reader), `modelsdk/mpr/reader.go` (`rawUnitBytes(q rowQuerier, id)` behind `GetRawUnitBytes`)", "insight": "With a one-connection pool, any read through the pool while a transaction is open is a deadlock, not a race, and it only shows on the storage format whose reads touch the database. A v2-only test suite cannot see it: the test that catches it must open a real v1 file through NewWriter (so the pool is the real one) and run Begin/WriteUnit/Commit under a deadline, otherwise the hang takes the whole package down instead of failing one test. Revert proof: the deadline test and a SaveUnit probe on a 9.24 app both time out on the old code and return in under 0.1s on the fix; mx check passes on the written project", "refs": []} diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-30-v2-commands-open-every-folder-and-microflow-for-names.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-30-v2-commands-open-every-folder-and-microflow-for-names.json new file mode 100644 index 0000000000..5beb462353 --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-30-v2-commands-open-every-folder-and-microflow-for-names.json @@ -0,0 +1 @@ +{"area":"modelsdk/mpr","date":"2026-10-30","symptom":"On MPR v2 nearly every command opened most unit files in a fresh process: on a 4,753-unit app `show modules` 4,511, `describe microflow` 2,996, `describe page` 3,176, `show microflows in M` 3,072. Fast enough on macOS, a per-call floor of seconds on Windows, where Defender scans each open","cause":"A unit's Name lives only in its BSON, and the type index (mendixlabs/mxcli#1603) held only $Type. The container hierarchy read every Projects$Folder for its name; `getMicroflowNames` / `ListMicroflowHeaders` read every microflow for Name and Excluded; `ListMicroflowsNamed` filtered every microflow on the stored Name; a page describe's `flowReturnEntity` decoded every microflow per flow-sourced widget; `show modules` decoded seven document types to count them; `populateViewEntityOql` listed every module per view-entity document","file":"`modelsdk/mpr/reader_units_v2.go` (index entry: Type, Name, Excluded; file `unit-index-2.idx`), `modelsdk/mpr/reader_units.go` (`ListUnitHeaders`), `modelsdk/mpr/types.go` (`ListFolders`), `modelsdk/mprread/generic.go` (`ListUnitHeaders`, `ListUnitsNamed`), `mdl/executor/cmd_modules.go` (`listModules` counts by unit type plus `meta` System built-ins), `mdl/executor/helpers.go` (`microflowsInModule`), `mdl/executor/cmd_pages_describe_flowcontext.go`, `mdl/backend/modelsdk/domainmodel.go`","insight":"**Trace reads by caller, not by count.** A temporary hook in `readMprContents` that records the unit's $Type and the first non-reader frames gave a table like `1908 Microflows$Microflow <- ListMicroflowHeaders < getMicroflowNames` per command, which named every fix directly; the raw count only said 'too many'. Anything derived from a unit's top level belongs in the ContentsHash index beside its type: it cannot go stale for the same reason. **Changing a persisted cache's format needs a new file name, not only a new header**: the old and new binaries each read the other's file as corrupt, rebuilt it and wrote their own, so two mxcli versions on one project (VS Code extension plus CLI) reopened every file on every call. That showed up only as nonsense numbers while benchmarking old against new on one project. Counting from unit types missed the System module's built-in enumerations and Java actions, which the listings add from `modelsdk/meta`; an agreement test against the full listings caught it (System Enums 0 vs 15). Guards: `TestFreshProcess_CommandsOpenOnlyWhatTheyShow` (pedapp: 196/113 reads before, 56/18 after), `TestShowModules_CountsAgreeWithTheListings`, `TestListUnitHeaders_*`, `TestUnitIndex_PreviousFormatIsLeftAlone`","refs":["ako/mxcli#1791","ako/mxcli#1789","mendixlabs/mxcli#1603"]} diff --git a/.claude/skills/mendix/create-page/reference/widgets.md b/.claude/skills/mendix/create-page/reference/widgets.md index 845e399576..fa7bf58ecc 100644 --- a/.claude/skills/mendix/create-page/reference/widgets.md +++ b/.claude/skills/mendix/create-page/reference/widgets.md @@ -1292,9 +1292,12 @@ Notes: widget-version drift (embedded template vs the installed `Charts.mpk`), even for a chart with no series. Clear it with **`mxcli fix widgets -p app.mpr`**, which applies the widget normalization to the project and keeps your storage format. - `mxcli docker check`/`build` normalize only a **temporary copy**: the check then - passes while the stored project still fails MxBuild and `mxcli run --local` with - CE0463. `docker check --no-update-widgets` checks the project as stored. + `mxcli docker check` checks the stored project and labels each such CE0463 + `stale widget definition — mxcli fix widgets fixes it`; a CE0463 labelled + `persists after update-widgets` is not drift, and `fix widgets` will not clear it. + `docker build` and `docker check --update-widgets` normalize only a **temporary + copy**, so they pass while the stored project still fails MxBuild and + `mxcli run --local` with CE0463. **Do NOT run bare `mx update-widgets` on an MPRv2 project** (one with an `mprcontents/` folder — everything `mxcli new` creates). `mx update-widgets` rewrites the project into the single-file v1 format and **deletes `mprcontents/`**, which diff --git a/.claude/skills/mendix/custom-widgets/SKILL.md b/.claude/skills/mendix/custom-widgets/SKILL.md index 4e0947793c..08efda6fb0 100644 --- a/.claude/skills/mendix/custom-widgets/SKILL.md +++ b/.claude/skills/mendix/custom-widgets/SKILL.md @@ -326,8 +326,8 @@ MDL032). **CE0463 "update this widget" is EXPECTED after generating charts.** mxcli writes the WidgetType from an embedded 11.6 baseline; the installed Charts.mpk is a different version, so Studio Pro/mxbuild flags drift. Clear it with **`mxcli fix -widgets`** (keeps your storage format); `docker check` only normalizes a temp copy, -so check the stored project with `--no-update-widgets`. The whole +widgets`** (keeps your storage format); `docker check` labels each one it fixes as a +*stale widget definition* (one that *persists after update-widgets* is not drift). The whole `mdl-examples/doctype-tests/34-chart-widget-examples.mdl` builds **0 errors** after. **Do NOT run bare `mx update-widgets` on an MPRv2 project** (an `mprcontents/`-folder project — what `mxcli new` creates): it converts the project to single-file v1 and diff --git a/.claude/skills/mendix/download-marketplace-content/SKILL.md b/.claude/skills/mendix/download-marketplace-content/SKILL.md index 0bc6226614..f2f5ef97b8 100644 --- a/.claude/skills/mendix/download-marketplace-content/SKILL.md +++ b/.claude/skills/mendix/download-marketplace-content/SKILL.md @@ -285,7 +285,7 @@ An MPR v1 project is passed straight through, since these tools write v1 nativel |---|---|---| | `mxcli fix widgets` | **yes** | the fix — after any headless install | | `mxcli fix design-properties` | **yes** | the fix — after any headless install | -| `mxcli docker check` | no | runs the widget resync under a snapshot so the *check* is not tripped by CE0463; the stored model stays stale | +| `mxcli docker check` | no | checks the stored project, so it reports the CE0463 a headless install leaves, and labels each one the resync clears as a stale widget definition (the resync runs on a temporary copy only) | | `mxcli widget sync` | yes, partial | reconciles widget schemas in mxcli's own code; clears 7 of 40 on the reference fixture | ## Step 5 — Before updating: has the module been edited? diff --git a/.claude/skills/mendix/java-actions/SKILL.md b/.claude/skills/mendix/java-actions/SKILL.md index 80377cc0f7..ea6220ed97 100644 --- a/.claude/skills/mendix/java-actions/SKILL.md +++ b/.claude/skills/mendix/java-actions/SKILL.md @@ -13,12 +13,12 @@ description: "Create and call custom Java actions — extending Mendix with serv | 28-39 | Reference files | | 40-48 | When to Use This Skill | | 49-55 | Overview | -| 56-385 | Part 2.5: Creating Java Actions in MDL | -| 386-579 | Part 3: Calling Java Actions from MDL | -| 580-593 | Validation Checklist | -| 594-604 | Common Errors | -| 605-612 | Related Documentation | -| 613-694 | Quick Reference | +| 56-394 | Part 2.5: Creating Java Actions in MDL | +| 395-588 | Part 3: Calling Java Actions from MDL | +| 589-603 | Validation Checklist | +| 604-614 | Common Errors | +| 615-622 | Related Documentation | +| 623-704 | Quick Reference | # Mendix Java Actions Skill @@ -57,6 +57,15 @@ Java actions allow you to extend Mendix with custom Java code. The workflow is: MDL supports defining Java actions with inline Java code using `create java action`. +**The body is not in the model.** The statement stores the action (parameters, +return type) in a unit and writes the `as $$ … $$` body to +`javasource//actions/.java`. Commit that file with the `.mpr` and +`mprcontents/`: a commit of the model alone gives a clone an action with no logic. +`exec` names each file it writes there on its own line — +`Wrote javasource//actions/.java (Java action body; commit it with the model)` — +and ends with `Files changed outside the model: N (…)`. `mxcli git commit` stages +them for you. + ### Basic Syntax ```mdl @@ -585,6 +594,7 @@ Before deploying Java actions, verify: - [ ] All parameters are defined with correct types - [ ] Return type matches what you return in Java - [ ] Code is only between `begin user CODE` / `end user CODE` markers +- [ ] `javasource//actions/.java` is committed with the model - [ ] Proper null checks for all parameters - [ ] Exception handling with logging - [ ] No hardcoded credentials or sensitive data diff --git a/.claude/skills/mendix/javascript-actions/SKILL.md b/.claude/skills/mendix/javascript-actions/SKILL.md index 62311b2d5b..38f2da32d6 100644 --- a/.claude/skills/mendix/javascript-actions/SKILL.md +++ b/.claude/skills/mendix/javascript-actions/SKILL.md @@ -11,15 +11,15 @@ description: "Create, call and drop JavaScript actions — client-side logic cal | lines | section | |---|---| | 31-39 | When to Use This Skill | -| 40-51 | Overview | -| 52-97 | Syntax | -| 98-159 | Examples | -| 160-171 | Calling from a nanoflow | -| 172-196 | The generated .js file | -| 197-205 | Inspecting | -| 206-214 | Validation Checklist | -| 215-224 | Common Errors | -| 225-229 | Related Documentation | +| 40-57 | Overview | +| 58-103 | Syntax | +| 104-165 | Examples | +| 166-177 | Calling from a nanoflow | +| 178-202 | The generated .js file | +| 203-211 | Inspecting | +| 212-220 | Validation Checklist | +| 221-230 | Common Errors | +| 231-235 | Related Documentation | # Mendix JavaScript Actions Skill @@ -49,6 +49,12 @@ A JavaScript action is two things on disk: function body (between the `BEGIN USER CODE` / `END USER CODE` markers), so it normally returns a `Promise`. +The body lives only in the `.js` file, so commit `javascriptsource/` with the +`.mpr` and `mprcontents/`; a commit of the model alone gives a clone an action +with no logic. `exec` names each file it writes there +(`Wrote javascriptsource//actions/.js (JavaScript action body; commit it with the model)`) +and ends with `Files changed outside the model: N (…)`. + ## Syntax ```sql diff --git a/.claude/skills/mendix/odata-data-sharing/SKILL.md b/.claude/skills/mendix/odata-data-sharing/SKILL.md index a814e88f36..e4932f9825 100644 --- a/.claude/skills/mendix/odata-data-sharing/SKILL.md +++ b/.claude/skills/mendix/odata-data-sharing/SKILL.md @@ -12,15 +12,15 @@ description: "Share data between Mendix apps over OData — published services, |---|---| | 30-42 | Reference files | | 43-52 | When to Use This Skill | -| 53-71 | MetadataUrl Formats | -| 72-123 | Refreshing the Cached Contract | -| 124-152 | ServiceUrl Must Be a Constant | -| 153-191 | Architecture Overview | -| 192-224 | API Versioning | -| 225-270 | Folder Organization | -| 271-282 | Module Organization Conventions | -| 283-304 | Checklist | -| 305-332 | Exploration Commands | +| 53-73 | MetadataUrl Formats | +| 74-125 | Refreshing the Cached Contract | +| 126-154 | ServiceUrl Must Be a Constant | +| 155-193 | Architecture Overview | +| 194-226 | API Versioning | +| 227-272 | Folder Organization | +| 273-284 | Module Organization Conventions | +| 285-306 | Checklist | +| 307-334 | Exploration Commands | # OData Data Sharing Between Mendix Apps @@ -64,6 +64,8 @@ long build-outs are next door: - Relative paths (with or without `./`) are **automatically converted** to absolute `file://` URLs in the Mendix model - This ensures Studio Pro can properly detect local file vs HTTP metadata sources (radio button in UI) - Example: `./metadata/service.xml` → `file:///absolute/path/to/project/metadata/service.xml` +- Re-running the script from a checkout at another path (another machine, a moved folder, a git worktree) leaves that stored URL alone when it names the same project file, so the client reports `Unchanged` instead of being rewritten to the new path +- `describe` prints a stored URL inside the project directory as `./`, so a script generated from the model stays portable; executing it leaves the client unchanged **Path Resolution (before normalization):** - With project loaded (`-p` flag or REPL): relative paths are resolved against the `.mpr` file's directory diff --git a/.claude/skills/mendix/teamserver-git/SKILL.md b/.claude/skills/mendix/teamserver-git/SKILL.md index 3b2af2afdb..e064d43702 100644 --- a/.claude/skills/mendix/teamserver-git/SKILL.md +++ b/.claude/skills/mendix/teamserver-git/SKILL.md @@ -12,11 +12,11 @@ description: "Commit, branch and push a Mendix project in git the way Studio Pro |---|---| | 35-41 | The rule: let Studio Pro do version control when it can | | 42-55 | Branches: create them on the server | -| 56-97 | Commit from the agent | -| 98-112 | Push the branch and the notes together, and soon | -| 113-125 | Never rewrite or merge model history with git | -| 126-136 | Reading what Studio Pro recorded | -| 137-145 | Checklist | +| 56-101 | Commit from the agent | +| 102-116 | Push the branch and the notes together, and soon | +| 117-129 | Never rewrite or merge model history with git | +| 130-140 | Reading what Studio Pro recorded | +| 141-149 | Checklist | # Working with Git on a Mendix Team Server Project @@ -78,7 +78,11 @@ mxcli git note -p app.mpr --write # attach the mx_metadata note ``` - Stage `App.mpr`, `mprcontents/` and the source you changed (`theme/`, - `javasource/`, `javascriptsource/`, `widgets/`, `mdlsource/`). Never stage + `javasource/`, `javascriptsource/`, `widgets/`, `mdlsource/`). A Java or + JavaScript action's body is in `javasource/` or `javascriptsource/`, not in + the model; `exec` names every such file it wrote, moved or removed + (`Wrote javasource/…`) and ends with `Files changed outside the model: N (…)` + — stage each of them. Never stage `project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, `.mendix-cache/`, `deployment/`, `theme-cache/` or `.mxcli/`; Mendix's `.gitignore` already excludes most of them. diff --git a/CHANGELOG.md b/CHANGELOG.md index 1084df4b74..fc5801b326 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,8 +11,23 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, and the project's own `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/` beside the `.mpr` — a folder of the same name further down, such as `resources/deployment/`, is committed) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` reports what it would commit without committing — and without repairing anything, hash drift included, because a dry run writes nothing: it stages into a copy of the index, so what the user had staged is left as it was. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. - **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS — judged as configured, so a `url..insteadOf` rule in the user's git config (https → `git@` is a common one) is reported rather than "repaired" on every run — and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. A clone whose notes ref lacks the remote's notes — `git clone` never fetches `refs/notes/*`, so every fresh clone and CI checkout — is reported as exactly that, with the fetch command, rather than as a history of unannotated commits; `--fix` fetches the notes first, as a fast-forward only. Docs: "Working Outside Studio Pro" › *Auditing a repository*. +### Changed + +- **`docker check` checks the project as stored by default, and says which CE0463 `mxcli fix widgets` clears** (ako/mxcli#568, #646) — this changes the default. Until now `docker check` ran `mx update-widgets` on its temporary copy before `mx check`, so it validated a widget-normalised model: a project whose stored widgets were stale (CE0463 "The definition of this widget has changed") got "0 errors" and exit 0, and then `mxcli run --local` and MxBuild, which build the project as stored, refused it. Now the verdict and the exit status are the stored project's. When the stored project has CE0463, the same copy is re-checked after `mx update-widgets` and each CE0463 is labelled `stale widget definition — mxcli fix widgets fixes it` or `persists after update-widgets`, with the `mxcli fix widgets -p .mpr` command and the error count after the resync. A project without CE0463 is checked once, as before. `--update-widgets` restores the old behaviour (check only the normalised copy); `--no-update-widgets` is still accepted, and checks as stored without the re-check. Measured on Mendix 11.13.0 with a blank app whose Data grid 2 package gained a property: the old default reported 0 errors, the new one 11 errors, all labelled stale; after `mxcli fix widgets` it reports 0. The project is still never modified. `docker build`'s pre-build check is unchanged and still runs on a normalised copy. + ### Fixed +- **Commands on an MPR v2 project open only the files they need** (ako/mxcli#1791) — on a 4,753-unit app, `show modules` opened 4,511 unit files, `describe microflow` 2,996 and `describe page` 3,176, in every process. The commands read every folder to build module paths and every microflow to name or find one, and `show modules` decoded every page, snippet, flow, constant and Java action just to count them. Each open is cheap on macOS and is scanned by Defender on Windows, so this added time to every command an agent runs. The unit index in `.mxcli/` now also holds each unit's name and `Excluded` flag, keyed by content hash like its type, so it never goes stale. Folders and microflows are named from the index, a microflow is found by name and read by ID, and `show modules` counts from unit types. The same commands now open 509, 93 and 274 files, with output identical across three projects and all 229 page describes. The index moves to `unit-index-2.idx`, so an older mxcli on the same project keeps its own `unit-types.idx` instead of the two rebuilding each other's. +- **`refresh catalog full source` reads each unit file once instead of millions of times** (ako/mxcli#1789) — the source index describes every document, and each microflow describe found its microflow, and every microflow it calls, by listing every microflow of the project from disk. On a 4,753-unit MPR v2 app one refresh opened 7,532,649 `.mxunit` files: 5.5 minutes on macOS, and 90 minutes or more on Windows, where Defender scans every file open. The catalog build now keeps the units it reads in memory, and a describe reads its microflow and the microflows it calls by ID. The same refresh opens 4,753 files and takes 31 s, and its source text is byte-identical. The rebuilds that `show references` and `search` run on a project with a source index are the same build, so they speed up too. +- **Moving a Java or JavaScript action to another module takes its source along** — `move java action A.X to B` moved the document and left `javasource/a/actions/X.java` behind (likewise a JavaScript action's `.js`). The build still succeeded, but the app called a generated stub that throws "Java action was not implemented", and `describe` printed a placeholder body that re-executing would have stored. The source now moves to the target module's directory, with a Java source's `package` line rewritten exactly as mxbuild regenerates it; an occupied destination refuses the move before anything changes. +- **Writing an MPR v1 project through the Go library no longer hangs** — on a project whose contents live in the `.mpr` (Mendix below 10.18), `codec.Store.SaveUnit`, `FlushUnits` and `modelsdk.Model.Flush` never returned: the write transaction read the stored unit through the reader's single pooled connection, which the transaction itself held. It now reads through the transaction. `mxcli exec` writes through a different path and was not affected. +- **Re-running an OData client script from a checkout at another path no longer rewrites the client** (ako/mxcli#1499) — a relative `MetadataUrl` such as `./contracts/f1-metadata.xml` is stored as the absolute `file://` URL of the checkout that ran it, and `create or modify consumed odata service` replaced that URL with its own on every run. The same unchanged script, run from another machine, a moved folder or a git worktree, rewrote every client: 5 of the 20 writes of a full re-exec in the reported project. The stored URL is now kept when it names the same project file, and the contract is read from this checkout's file. The stored form stays absolute: `mx check` accepts a relative value, but it also accepts a dead absolute path, so it shows nothing about what Studio Pro's metadata editor accepts. `describe` prints a stored URL inside the project directory as `./contracts/…`, so a script generated from the model carries no machine path. Measured on 11.13.0: the same script run after moving the app, and from a copy, reported the client unchanged (before: "Modified OData client"), with `mx check` 0 errors. +- **On an MPR v1 project, a unit write updates `Unit.ContentsHash` with the contents** (ako/mxcli#1159) — below Mendix 10.18 the unit contents live in the `.mpr`, and every write replaced a unit's `Contents` but left its `ContentsHash` describing the old bytes; an inserted unit got an empty hash. `mx check` and MxBuild never read the column, so nothing failed, but Studio Pro reads it to detect what changed. Studio Pro fills it on every v1 unit with the same formula as v2, `base64(SHA-256(Contents))`: 308 of 308 units of a fresh 9.24.30 app. Update, insert and the transactional write now store it. `mxcli fix hashes` covers v1 too: it compares each hash with the contents in its row, and `--repair` rewrites the ones an earlier mxcli left stale. `exec` and `docker check` still verify v2 only, because on v1 that reads every unit. Measured on 9.24.30: `create entity` plus `create microflow` left 2 stale rows before the fix and none after, `mx check` 0 errors, and `fix hashes --repair` repaired the stale copy. +- **Workflow activities and events an older Mendix does not have are refused up front** (ako/mxcli#1774) — `mxcli check -p` and `exec`'s pre-flight report MDL-VERSION01 before anything is written, naming the clause: `wait for notification` below 10.0, `wait for timer` below 10.6, a `boundary event timer` of no kind below 10.14, `boundary event [non] interrupting timer` below 10.20, and `lock workflow` or `unlock workflow` naming a workflow below 10.0. Each floor is the type's own introduction in the Mendix Model SDK, and the write refuses the type itself too. Before, a workflow with `wait for notification` was written to 9.24.30, and the project did not load ("TypeCacheUnknownTypeException … Workflows$WaitForNotificationActivity"). Measured on 9.24.30 (refused, and the project is byte-identical) and on 10.24.24 and 11.13.0 (written, 0 errors, as before). +- **A type newer than the project is refused instead of making the project unloadable** (ako/mxcli#1774) — the version data held property floors only, so no write compared an element's own `$Type` with the project's version. `exec` reported success, and then `mx check` and Studio Pro could not load the project: "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name …". This shipped four times (#1754, #1209, #1163, #1771), each fixed for its one type. Every type's introduction version is now taken from the Mendix Model SDK (`modelsdk/canon/mendixmodelsdk_type_versions.json`, extracted from mendixmodelsdk 4.117.0 by `gen_type_versions.go`). Every unit write refuses an element whose type the project does not have yet, wherever it sits in the unit, naming the type, its version and the project's: "this Mendix 9.24.30 project cannot store Microflows$GetWorkflowsAction (Mendix 10.4.0+)". The check runs after the older shapes are written, so the parallel split's end marker and user tasks are still lowered, not refused. Measured on 9.24.30: a microflow with `get workflows` made the project unloadable before. Now the write is refused, and the project is byte-identical afterwards, with 0 errors. 23 Studio Pro-authored projects from 9.1.1 to 11.16.0 hold no type the floor refuses. The SDK's deletion versions are not enforced, because a type it renames keeps its storage name. +- **Documents written to Mendix 10.24 below 10.24.24 have the property names Studio Pro writes** (ako/mxcli#1505, mendixlabs/mxcli#1595) — on 10.24.18 a nanoflow re-created with `drop` and `create` lacked `ExportLevel` and `ReturnVariableName`, so `mx diff` and Studio Pro's change view threw "Objects with ID … do not have the same properties". A new microflow lacked the aggregate's four expression properties, a delete's error handling, a while loop's caption and a retrieve's XPath constraint, and a new entity its image properties. mxcli fills these from a table of values measured with `mx convert`, and the table had been measured from 10.24.24 up only; that version was then treated as the one that introduced each property. Measured again on 10.24.0 and 10.24.18, 94 of its rows now apply from 10.24.0, including open user task, REST operation call, timer boundary event and business event service properties. Run `mxcli fix property-sets` to complete the documents an earlier mxcli wrote. +- **`mxcli fix property-sets` no longer reports a project complete when an element lacks a property the others of its type carry** (ako/mxcli#1505) — on 10.24.18 it said "Every unit already has the property names Studio Pro writes" for a nanoflow without `ExportLevel` while `mx diff` threw on it, because it knew only the properties its measured table holds for the project's version. Studio Pro writes the same property names on every element of a type, so the command now also lists each name some elements of a type carry and others lack, as `MIXED .` with the units concerned, and exits 1. It does not fill those: no value measured on that Mendix version says what to write. +- **An `alter microflow` fragment can name a member of a list operation's, a Java action's or a self-reference retrieve's output** (ako/mxcli#1777) — `alter microflow … { insert after … begin change $First (Status = 'Seen'); end; }`, with `$First = head $Lines` in the stored flow, was refused with nothing written: "cannot qualify member "Status": the entity its variable holds is not known here". The same for a member of any stored filter, sort, tail, union, intersect, subtract, range, head or find output, of a Java action's result, or of a retrieve over a self-reference, though `create microflow` types them all. A fragment now starts from those types, taken from the same code the full build uses: a list operation's output has its input list's type (an element of it for a head or find), a Java action's result its declared return type (for a list of its type parameter, the list it is given), and an aggregate the primitive the full build declares. A retrieve over a self-reference holds the object the reference holds (the list, for a reference set): the stored retrieve records no direction, and Mendix follows it from the FROM end — measured on 11.13.0, a change on it builds and a count of it is CE0013. A name two origins give different types stays untyped, its member refused. Measured on 11.13.0: seven alters refused before; all written after, mx check 0 errors, describe → exec "Unchanged". - **User-task features an older Mendix cannot store are refused up front** (ako/mxcli#1771) — `mxcli check -p` and `exec` report MDL-VERSION01 before anything is written, naming the clause. The floors are group targeting (`targeting groups …`) 11.2, a multi-user task 9.22, `decide by majority` or `threshold` 10.0, `await all users` 10.2, and `decide by microflow` 10.3. Before, group targeting was written on 10.24 and failed `mx check` with CE1859. The others were refused only at write time, after the statements before them were written. Measured on 9.24.30 and 10.24.24 (each refused, nothing written) and on 11.13.0 (all written, 0 errors). - **`alter workflow … set (Targeting: …)` changes a user task's targeting on Mendix 9 to 11.1** (ako/mxcli#1771) — it reported "Altered workflow" and changed nothing. Before 11.2 a user task stores its targeting under another key, and the edit only replaced a key that was already there. The edit now replaces the stored targeting, whichever key holds it, and the write stores it in the version's shape. Measured on 9.24.30 and 10.24.24: `describe` shows the new targeting, with 0 errors. 11.2 and later are unchanged. - **A workflow with a user task no longer makes a Mendix 9 or 10.0–10.11 project unloadable** (ako/mxcli#1771) — `exec` reported success, then `mx check` and Studio Pro could not load the project at all: "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity". mxcli wrote a user task as a `SingleUserTaskActivity` or `MultiUserTaskActivity`, types introduced in 10.12. Before 10.12 both are a `Workflows$UserTask`, as Studio Pro 9.24 stores it. Its completion says which kind it is, and holds a multi-user task's rule, participants and, from 10.2, `await all users`. That is what is written now, for `create`, `create or modify` and a task `alter workflow` inserts. `describe` reads a multi-user task in that shape, and `create or modify` no longer resets its rule silently. What the older type cannot hold is refused, naming the task: a multi-user task below 9.22, `decide by majority` or `threshold` below 10.0, `decide by microflow` below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14. Measured on 9.24.30 and 10.11.0: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. Converted up to 10.24, the tasks are the ones mxcli writes there directly. 10.12 and later are unchanged. @@ -25,7 +40,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A find in a loop an `alter microflow` writes is stored as a find by attribute, not as a find by an invalid expression** (ako/mxcli#1769, mendixlabs/mxcli#1673) — `alter microflow … { replace loop … with begin … end; }` whose loop body held `$Line = find $Lines by Code = $Lookup/Code;`, with `$Lines` retrieved over an association before the loop, stored `Microflows$FindByExpression` with the bare text `Code = $Lookup/Code`: `check --references` and `exec` reported nothing, `describe` printed `find $Lines where …`, and mx check failed CE0117 "Error(s) in expression." `create microflow` stores the same find as `Microflows$Find` on `MyModule.OrderLine.Code`. Two causes. The guard that refuses a member a spliced fragment cannot qualify (ako/mxcli#891) applied on the fragment's main line only, not in a loop, while loop or error handler inside it, and so did not cover `replace body of loop` either: there any find, filter, sort, aggregate or change member was written bare. And the fragment did not know what a stored retrieve over an association holds, nor a stored loop's iterator. Both are now typed as `create microflow` types them: from the association's TO end a retrieve yields the list of its FROM entity, from the FROM end the one TO object. So the reported alter writes `Microflows$Find`, and what still cannot be typed (a self-reference, a list operation's output) is refused with nothing written, wherever the statement sits. Measured on Mendix 11.13.0: CE0117 before; `Microflows$Find` on `MyModule.OrderLine.Code`, mx check 0 errors, and describe → exec "Unchanged" after. - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. -- **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. +- **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation was not renamed until ako/mxcli#1775 (next entry). +- **Two tests that define the same variable with `retrieve`, `declare`, a list operation or any other activity run on the after-startup runner** (ako/mxcli#1775) — that runner joins every test into one microflow and renames each test's variables, but which ones it renamed came from a pattern that recognised only `$x = call …` and `$x = create …`, in any case. A variable defined by `retrieve`, `declare`, a list operation or aggregate, `send rest request`, `execute database query` or a loop iterator kept its name, so two tests defining it declared it twice and injection failed with MDL063 or MDL052 (CE0111 "Duplicate variable name" from mxbuild); `setup-annotation.test.mdl` and the #1103 example still did, under Docker or `--legacy-runner`. The names now come from the parsed test body through the definition MDL063 itself counts, including those nested in an `if` branch, a loop or an error handler. The endpoint runner (`--local`) gives each test its own microflow and was never affected. +- **`exec` names every file it writes beside the model** (ako/mxcli#1618, mendixlabs/mxcli#1634) — `create java action … as $$ … $$` stores the action in the model and writes its body to `javasource//actions/.java` (a JavaScript action's to `javascriptsource/`), but `exec` printed only "Created java action: …", so a workflow that commits the model (`.mpr` and `mprcontents/`) left the body behind and a clone got an action with no logic. Each file written, moved (`rename java action`) or removed (`drop java action`, `drop javascript action`, the source folders `drop module` deletes) is now named on its own line as a project-relative path, e.g. `Wrote javasource/probe/actions/JA_Hello.java (Java action body; commit it with the model)`, and a run that changed any ends with `Files changed outside the model: N (javasource/, …). Stage them with the .mpr.` A rewrite that leaves a file's content as it was does not touch it and is not reported. - **`create published odata service` on Mendix 9 and 10.0–10.3 is refused up front, naming the real reason** (ako/mxcli#1163) — it was refused at write time for "ODataPublish$PublishedODataService2.IncludeMetadataByDefault (Mendix 10.8.0+)". That property was not the problem: the whole document mxcli writes is a 10.4 type, and Mendix 9.24 cannot load a project that holds one. Before 10.4 a published OData service is a different document that mxcli does not write. `check -p` and `exec` now report MDL-VERSION01 before writing anything. - **A page passing `$currentObject` to a microflow or nanoflow is written on Mendix 9 and on 10.0–10.9** (ako/mxcli#1163) — `call microflow M.F(P = $currentObject)` on a button or a data grid column was refused with "cannot store Forms$MicroflowParameterMapping.Expression (Mendix 10.10.0+)". An argument held as an expression arrived in 10.10. Before it, Studio Pro stores no argument for the enclosing object and infers it. Without the refusal, 9.24 could not even load the page, because the mapping had no `Variable`. Below 10.10 that argument is now left out, and a page parameter argument is stored as before. Any other expression argument is refused, naming the parameter. - **A constant override in a configuration is written on Mendix 9 and on 10.0–10.8** (ako/mxcli#1163) — `alter settings constant @M.C value '…' in configuration 'Default'` was refused with "cannot store Settings$ConstantValue.SharedOrPrivateValue (Mendix 10.9.0+)". Before 10.9 an override's value is a flat `Value` string; Studio Pro 9.24 does not know the nested form and empties the override on load. A new override is now written flat there, and an existing one is updated in its own shape. A private value has no form before 10.9 and is refused, naming the constant. diff --git a/cmd/mxcli/cmd_exec.go b/cmd/mxcli/cmd_exec.go index a1e3d2ab00..73d1579f35 100644 --- a/cmd/mxcli/cmd_exec.go +++ b/cmd/mxcli/cmd_exec.go @@ -43,6 +43,15 @@ writes through it with --mcp. --force writes anyway (for a lock left behind by a crash); MXCLI_ALLOW_STUDIO_PRO_OPEN=1 does the same for every command. Reads, and re-running a script whose statements change nothing, are never refused. +Some statements also write files beside the model: a Java or JavaScript +action's body goes to javasource/ or javascriptsource/, and DROP MODULE removes +the module's source folders. Each file written, moved or removed is named on its +own line after the statement's report, as a path relative to the project +("Wrote javasource/probe/actions/JA_Hello.java ..."), and a run that changed any +ends with "Files changed outside the model: N (...)". Commit them with the .mpr +and mprcontents/: they are not part of the model. A rewrite that leaves a +file's content as it was does not touch it and is not reported. + A script is applied all or nothing: if a statement still fails while it is written (after the checks passed), the writes of the statements before it are rolled back and the project is left as it was. Only the model (.mpr and @@ -87,6 +96,9 @@ Example: exec, logger := newLoggedExecutor("exec") defer logger.Close() defer exec.Close() + if noReport, _ := cmd.Flags().GetBool("no-source-file-report"); noReport { + exec.SetSourceFileReport(false) + } // A relative path inside the script (a toolbox icon, an image) names a // file next to the script, not next to the caller. Empty for stdin. @@ -218,4 +230,10 @@ func init() { "Write even though Studio Pro appears to have the project open (its .mpr.lock is present) — e.g. a lock left behind by a crash") execCmd.Flags().Bool("continue-on-error", false, "Run every statement, reporting each failure instead of halting at the first (exits non-zero if any failed) — makes a partially-applied script re-runnable") + // Hidden: for a caller whose writes are temporary. mxcli test injects a Java + // action and deletes its source at cleanup; telling its user to commit that + // file would be wrong (ako/mxcli#1618). + execCmd.Flags().Bool("no-source-file-report", false, + "Do not name the files written beside the model (for a caller that removes them again, such as mxcli test)") + _ = execCmd.Flags().MarkHidden("no-source-file-report") } diff --git a/cmd/mxcli/cmd_fix.go b/cmd/mxcli/cmd_fix.go index 15571b790f..c042030fd5 100644 --- a/cmd/mxcli/cmd_fix.go +++ b/cmd/mxcli/cmd_fix.go @@ -71,10 +71,11 @@ CE0463 "The definition of this widget has changed" is what a project reports when its stored widget instances are older than the widget packages installed beside them — the normal state after any headless module or widget install. -'mxcli docker check' and 'mxcli docker build' already run this step, but on a -temporary copy of the project, so the check passes and the stored model stays -stale. This persists the resync instead, which is what Studio Pro's "Update all widgets" -does. It is also more complete than 'mxcli widget sync', which reconciles widget +'mxcli docker check' reports these against the stored project and labels each +one this command clears as a "stale widget definition" (it runs the resync on a +temporary copy to find out). 'mxcli docker build' runs the resync on its copy +too, so it builds while the stored model stays stale. This persists the resync +instead, which is what Studio Pro's "Update all widgets" does. It is also more complete than 'mxcli widget sync', which reconciles widget schemas itself and clears only part of the same errors.`, Example: ` mxcli fix widgets -p app.mpr`, Args: cobra.NoArgs, diff --git a/cmd/mxcli/cmd_fix_hashes.go b/cmd/mxcli/cmd_fix_hashes.go index 28647742dd..5366809ee1 100644 --- a/cmd/mxcli/cmd_fix_hashes.go +++ b/cmd/mxcli/cmd_fix_hashes.go @@ -14,14 +14,14 @@ import ( "go.mongodb.org/mongo-driver/v2/bson" ) -// cmd_fix_hashes.go verifies and repairs the MPR v2 ContentsHash index -// (ako/mxcli#972). Unlike its siblings in cmd_fix.go it needs no mx: the index +// cmd_fix_hashes.go verifies and repairs the .mpr's ContentsHash index +// (ako/mxcli#972; MPR v1 since ako/mxcli#1159). Unlike its siblings in cmd_fix.go it needs no mx: the index // is mxcli's own to read and write. var fixHashesCmd = &cobra.Command{ Use: "hashes", - Short: "Verify (and with --repair, rewrite) the MPR v2 ContentsHash index", - Long: `Verify that every unit's ContentsHash in the .mpr matches its .mxunit file. + Short: "Verify (and with --repair, rewrite) the .mpr's ContentsHash index", + Long: `Verify that every unit's ContentsHash in the .mpr matches its contents. In an MPR v2 project the .mpr indexes each mprcontents/**/*.mxunit file by base64(SHA-256(file)). mxcli and Studio Pro keep the two in step, but anything @@ -41,9 +41,13 @@ truth. MISSING and ORPHAN have no correct hash to write and are only reported ('mxcli diag --check-units --fix' removes orphan files). Like every mxcli write, the repair is refused while Studio Pro has the project open. -Exits 1 when an issue remains after the command (so the verify can gate CI). -An MPR v1 project keeps unit contents inside the .mpr and has no index to -drift; the command says so and exits 0.`, +An MPR v1 project (Mendix below 10.18) keeps each unit's contents inside the +.mpr, in the same row as its ContentsHash, so there are no files to drift. The +hash is still Studio Pro's to read, and earlier mxcli versions rewrote contents +without it (ako/mxcli#1159): on v1 the command compares each hash with the +contents stored beside it, and --repair rewrites it from them. + +Exits 1 when an issue remains after the command (so the verify can gate CI).`, Example: ` mxcli fix hashes -p app.mpr mxcli fix hashes -p app.mpr --repair`, Args: cobra.NoArgs, @@ -100,13 +104,12 @@ func fixHashes(mprPath string, repair bool, out io.Writer) error { rep, err = r.VerifyContentsHashes() _ = r.Close() } - if errors.Is(err, mpr.ErrNotMPRv2) { - fmt.Fprintf(out, "%s is an MPR v1 project: unit contents live inside the .mpr, so there is no ContentsHash index to verify.\n", filepath.Base(mprPath)) - return nil - } if rep == nil { return err } + if rep.Version == mpr.MPRVersionV1 { + return reportV1Hashes(mprPath, rep, repair, repaired, err, out) + } fmt.Fprintf(out, "Checked %d unit(s) in %s against %d .mxunit file(s).\n", rep.Units, filepath.Base(mprPath), rep.Files) contentsDir := filepath.Join(filepath.Dir(mprPath), "mprcontents") @@ -153,6 +156,34 @@ func fixHashes(mprPath string, repair bool, out io.Writer) error { return nil } +// reportV1Hashes is fixHashes' report for an MPR v1 project, where each hash +// is compared with the contents in its own row: nothing can be missing or +// orphaned, and the contents are what a repair writes the hash from. +func reportV1Hashes(mprPath string, rep *mpr.ContentsHashReport, repair bool, repaired int, err error, out io.Writer) error { + fmt.Fprintf(out, "Checked %d unit(s) in %s (MPR v1) against the contents stored beside them.\n", rep.Units, filepath.Base(mprPath)) + for _, m := range rep.Mismatches { + label := "" + if m.Unit != "" { + label = " [" + m.Unit + "]" + } + fmt.Fprintf(out, " MISMATCH %s%s\n stored %s, contents %s\n", m.UnitID, label, orNone(m.Stored), m.Actual) + } + fmt.Fprintf(out, "\n%d mismatch(es).\n", len(rep.Mismatches)) + if err != nil { // the repair itself failed (Studio Pro guard, SQLite) + return err + } + switch { + case repair && repaired > 0: + fmt.Fprintf(out, "Repaired %d ContentsHash value(s) from the stored contents.\n", repaired) + case len(rep.Mismatches) > 0: + fmt.Fprintf(out, "Run 'mxcli fix hashes -p %s --repair' to rewrite the mismatched hashes from the stored contents.\n", mprPath) + return errHashIssuesRemain + default: + fmt.Fprintln(out, "Every unit's ContentsHash matches its stored contents.") + } + return nil +} + func orNone(s string) string { if s == "" { return "(none)" @@ -183,8 +214,13 @@ func unitLabel(path string) string { // when the ContentsHash index disagrees with the unit files, or "" when it // agrees. Measured on testapp (900 units): the whole `mxcli fix hashes` run, // process start included, takes ~0.2 s, so it is cheap enough to run on every -// exec and check. It never fails the caller: an MPR v1 project or any error -// opening or reading the project yields "". +// exec and check. It never fails the caller: any error opening or reading the +// project yields "". +// +// MPR v1 is skipped. Its hashes live in the rows with the contents, so a +// restored file cannot make them drift, and verifying them reads every blob in +// the .mpr: far more than 0.2 s on a large v1 app, on every command. `fix +// hashes` checks them when asked (ako/mxcli#1159). func contentsHashDriftWarning(mprPath string) string { if mprPath == "" { return "" @@ -194,6 +230,9 @@ func contentsHashDriftWarning(mprPath string) string { return "" } defer r.Close() + if r.Version() != mpr.MPRVersionV2 { + return "" + } rep, err := r.VerifyContentsHashes() if err != nil { return "" diff --git a/cmd/mxcli/cmd_fix_hashes_test.go b/cmd/mxcli/cmd_fix_hashes_test.go index 7f6fd4bf15..a961a4a677 100644 --- a/cmd/mxcli/cmd_fix_hashes_test.go +++ b/cmd/mxcli/cmd_fix_hashes_test.go @@ -4,6 +4,7 @@ package main import ( "bytes" + "database/sql" "errors" "os" "path/filepath" @@ -94,25 +95,64 @@ func TestFixHashes_DetectsAndRepairsRevertedUnit(t *testing.T) { } } -func TestFixHashes_V1NotApplicable(t *testing.T) { +// ako/mxcli#1159: on MPR v1 the hash sits beside the contents in the same row, +// and every mxcli before the fix rewrote Contents without it. `fix hashes` +// compares against the Contents column there. +func TestFixHashes_V1ComparesAgainstContents(t *testing.T) { src := filepath.Join("..", "..", "modelsdk", "mpr", "testdata", "v1-project") - files, _ := filepath.Glob(filepath.Join(src, "*.mpr")) - if len(files) == 0 { - t.Skip("no v1 fixture") - } dst := t.TempDir() if err := copyTree(src, dst); err != nil { t.Fatal(err) } - mprPath := filepath.Join(dst, filepath.Base(files[0])) + mprPath := filepath.Join(dst, "App.mpr") + + // Control: the Studio Pro-authored fixture is clean. var out bytes.Buffer if err := fixHashes(mprPath, false, &out); err != nil { - t.Fatalf("v1: %v", err) + t.Fatalf("v1 control: %v\n%s", err, out.String()) + } + if !strings.Contains(out.String(), "MPR v1") || !strings.Contains(out.String(), "matches its stored contents") { + t.Fatalf("v1 control not reported clean:\n%s", out.String()) } - if !strings.Contains(out.String(), "MPR v1") { - t.Fatalf("v1 not explained:\n%s", out.String()) + + // What an mxcli before #1159 left behind: new Contents, old hash. + db, err := sql.Open("sqlite", mprPath) + if err != nil { + t.Fatal(err) } + var id []byte + var contents []byte + if err := db.QueryRow(`SELECT UnitID, Contents FROM Unit ORDER BY UnitID LIMIT 1`).Scan(&id, &contents); err != nil { + t.Fatal(err) + } + if _, err := db.Exec(`UPDATE Unit SET Contents = ? WHERE UnitID = ?`, append(contents, 0), id); err != nil { + t.Fatal(err) + } + _ = db.Close() + + out.Reset() + err = fixHashes(mprPath, false, &out) + if !errors.Is(err, errHashIssuesRemain) { + t.Fatalf("v1 verify: want errHashIssuesRemain, got %v\n%s", err, out.String()) + } + if !strings.Contains(out.String(), "MISMATCH") || !strings.Contains(out.String(), "1 mismatch(es)") { + t.Fatalf("v1 mismatch not reported:\n%s", out.String()) + } + // exec and docker check stay silent on v1: the warning targets files + // restored with git, and would read every blob of the .mpr on every run. if w := contentsHashDriftWarning(mprPath); w != "" { t.Fatalf("v1 warned: %s", w) } + + out.Reset() + if err := fixHashes(mprPath, true, &out); err != nil { + t.Fatalf("v1 repair: %v\n%s", err, out.String()) + } + if !strings.Contains(out.String(), "Repaired 1 ContentsHash") { + t.Fatalf("v1 repair not reported:\n%s", out.String()) + } + out.Reset() + if err := fixHashes(mprPath, false, &out); err != nil { + t.Fatalf("v1 after repair: %v\n%s", err, out.String()) + } } diff --git a/cmd/mxcli/cmd_fix_property_sets.go b/cmd/mxcli/cmd_fix_property_sets.go index 8df0dd7b71..d6388748bc 100644 --- a/cmd/mxcli/cmd_fix_property_sets.go +++ b/cmd/mxcli/cmd_fix_property_sets.go @@ -9,6 +9,7 @@ import ( "os" "path/filepath" "sort" + "strings" mpr "github.com/mendixlabs/mxcli/modelsdk/mpr" "github.com/spf13/cobra" @@ -59,7 +60,15 @@ A unit holding a value in a key the project's version does not declare is reported and left alone — dropping the value would change the model. Like every mxcli write, the repair is refused while Studio Pro has the project open. -Exits 1 when a unit could not be repaired, or, with --dry-run, when one needs it.`, +The keys it adds come from values measured on Mendix versions. A key no +measurement covers on the project's version is not filled, but it is found: +Studio Pro writes the same property names on every element of a type, so a +name some elements of a type carry and others lack is listed as MIXED, with the +units on the smaller side, and left as it is. Report those with your Mendix +version so the value can be measured. + +Exits 1 when a unit could not be repaired, when a MIXED name remains, or, with +--dry-run, when a unit needs the repair.`, Example: ` mxcli fix property-sets -p app.mpr --dry-run mxcli fix property-sets -p app.mpr`, Args: cobra.NoArgs, @@ -128,6 +137,7 @@ func fixPropertySets(mprPath string, dryRun bool, out io.Writer) error { changed []mpr.PropertySetRepair failed []string added, removed int + census = mpr.NewPropertySetCensus() ) for _, id := range ids { var rep mpr.PropertySetRepair @@ -140,12 +150,14 @@ func fixPropertySets(mprPath string, dryRun bool, out io.Writer) error { failed = append(failed, fmt.Sprintf(" FAILED %s%s: %v", id, repairLabel(rep), err)) continue } + census.Count(rep) if rep.Changed() { changed = append(changed, rep) added += rep.Added removed += rep.Stripped } } + mixed := census.Mixed() verb := "Completed" if dryRun { @@ -161,7 +173,7 @@ func fixPropertySets(mprPath string, dryRun bool, out io.Writer) error { fmt.Fprintln(out, f) } switch { - case len(changed) == 0 && len(failed) == 0: + case len(changed) == 0 && len(failed) == 0 && len(mixed) == 0: fmt.Fprintln(out, "Every unit already has the property names Studio Pro writes.") case len(changed) > 0: fmt.Fprintf(out, "\n%s %d unit(s): %d key(s) added, %d stripped.\n", verb, len(changed), added, removed) @@ -172,12 +184,42 @@ func fixPropertySets(mprPath string, dryRun bool, out io.Writer) error { if len(failed) > 0 { fmt.Fprintf(out, "%d unit(s) could not be normalized and were left as they are.\n", len(failed)) } - if len(failed) > 0 || (dryRun && len(changed) > 0) { + reportMixedPropertyNames(out, mixed, pv.MajorVersion, pv.MinorVersion, pv.PatchVersion) + if len(failed) > 0 || len(mixed) > 0 || (dryRun && len(changed) > 0) { return errPropertySetsRemain } return nil } +// reportMixedPropertyNames lists the property names the elements of one $Type +// disagree on after the repair (ako/mxcli#1505): Studio Pro writes the same +// names on every element of a type, so each is a pair of revisions Mendix's +// merge engine refuses to compare. The repair fills only keys with a value +// measured on a Mendix version, so these are reported and left alone. +func reportMixedPropertyNames(out io.Writer, mixed []mpr.MixedPropertyName, major, minor, patch int) { + if len(mixed) == 0 { + return + } + fmt.Fprintf(out, "\nProperty names that differ between elements of one type (no value measured on Mendix %d.%d.%d fills them):\n", + major, minor, patch) + for _, m := range mixed { + side := "only on" + if m.Without() { + side = "without it" + } + var units []string + for _, u := range m.Examples { + units = append(units, u.UnitID+repairLabel(u)) + } + fmt.Fprintf(out, " MIXED %s.%s: on %d of %d element(s); %s: %s\n", + m.Type, m.Key, m.With, m.Elements, side, strings.Join(units, ", ")) + } + fmt.Fprintf(out, "%d property name(s) left as they are. Mendix's merge engine cannot compare a revision of these "+ + "elements with one that has the other set of names. mxcli fills a key only with the value Studio Pro was "+ + "measured writing on a Mendix version; report the names above with your Mendix version so it can be measured.\n", + len(mixed)) +} + // repairLabel names the unit (" [Microflows$Microflow Name]"), or "". func repairLabel(rep mpr.PropertySetRepair) string { switch { diff --git a/cmd/mxcli/cmd_fix_property_sets_test.go b/cmd/mxcli/cmd_fix_property_sets_test.go index 7f842465af..7bc6af3b26 100644 --- a/cmd/mxcli/cmd_fix_property_sets_test.go +++ b/cmd/mxcli/cmd_fix_property_sets_test.go @@ -126,3 +126,44 @@ func TestFixPropertySets_CompletesAUnitNothingRewrites(t *testing.T) { t.Fatalf("the repair left the ContentsHash index stale: %s", w) } } + +// ako/mxcli#1505: on Mendix 10.24.18 `fix property-sets --dry-run` said "Every +// unit already has the property names Studio Pro writes" while a nanoflow mxcli +// had re-created lacked ExportLevel and ReturnVariableName and `mx diff` threw on +// it. The check only knew the keys the measured table fills on the project's +// version, and the table did not cover 10.24.18. +// +// The project itself shows such a gap whatever the table covers: Studio Pro +// writes the full property set on every element of a $Type, so one element of +// the type without a key the others carry is a name set Mendix's merge engine +// cannot compare. With no measured value to fill it from, the command reports +// it and writes nothing. The subject here is a key the table has no row for at +// all, on the Studio Pro-authored fixture (11.13.0); the control is +// TestFixPropertySets_StudioProProjectIsClean, where every type is uniform. +func TestFixPropertySets_ReportsAKeyNoMeasuredValueFills(t *testing.T) { + mprPath := copyPedApp(t) + f := dropKeyFromUnit(t, mprPath, "Microflows$Nanoflow", "ACT_Login", "MarkAsUsed") + var sink bytes.Buffer + if err := fixHashes(mprPath, true, &sink); err != nil { + t.Fatalf("re-index the subject: %v\n%s", err, sink.String()) + } + edited, _ := os.ReadFile(f) + + for _, dryRun := range []bool{true, false} { + var out bytes.Buffer + err := fixPropertySets(mprPath, dryRun, &out) + if !errors.Is(err, errPropertySetsRemain) { + t.Fatalf("dry-run=%v: want errPropertySetsRemain, got %v\n%s", dryRun, err, out.String()) + } + got := out.String() + if strings.Contains(got, "Every unit already has") { + t.Fatalf("dry-run=%v: reported complete with a nanoflow missing MarkAsUsed:\n%s", dryRun, got) + } + if !strings.Contains(got, "Microflows$Nanoflow.MarkAsUsed") || !strings.Contains(got, "[Microflows$Nanoflow ACT_Login]") { + t.Fatalf("dry-run=%v: the gap is not named with the unit that has it:\n%s", dryRun, got) + } + if b, _ := os.ReadFile(f); !bytes.Equal(b, edited) { + t.Fatalf("dry-run=%v: wrote the unit — a key with no measured value must not be filled", dryRun) + } + } +} diff --git a/cmd/mxcli/cmd_marketplace_update.go b/cmd/mxcli/cmd_marketplace_update.go index 2b6ff903f1..7ebf518bdd 100644 --- a/cmd/mxcli/cmd_marketplace_update.go +++ b/cmd/mxcli/cmd_marketplace_update.go @@ -316,10 +316,10 @@ func reportUpdate(out io.Writer, r *marketplace.UpdateResult) { // Saying so here is the difference between a two-command fix and a day in // diagnose-ce0463.md, which is where that error normally leads. // - // The resync is named as `mxcli docker check`, not as bare `mx update-widgets`: + // The resync is named as `mxcli fix widgets`, not as bare `mx update-widgets`: // the latter rewrites an MPR v2 project as v1 (measured on 11.12.1 — 200 - // .mxunit files to 0, a 69 KB index to 14 MB), while docker check runs the - // same step under a storage-format snapshot (#808). + // .mxunit files to 0, a 69 KB index to 14 MB), while fix widgets writes the + // result back as v2 (#808). docker check then checks the project as stored. fmt.Fprintln(out, "\n Next, repair what a headless update leaves behind (expected, not a fault):") fmt.Fprintln(out, " mxcli fix widgets -p # CE0463") fmt.Fprintln(out, " mxcli fix design-properties -p # CE6087") diff --git a/cmd/mxcli/docker.go b/cmd/mxcli/docker.go index 5414f03143..5917e06ba4 100644 --- a/cmd/mxcli/docker.go +++ b/cmd/mxcli/docker.go @@ -185,19 +185,30 @@ compiles the theme into theme-cache/ and deployment/, and update-widgets rewrites the model. Build output, caches and VCS folders (deployment/, releases/, theme-cache/, .git/, node_modules/, ...) are not copied. -By default, 'mx update-widgets' runs before 'mx check', on that copy, to -normalize pluggable widget definitions and prevent false CE0463 errors. -A CE0463 that normalisation clears is then not reported, although it still -fails MxBuild and 'mxcli run --local' on the stored project. Use ---no-update-widgets to check the project as stored, and 'mxcli fix widgets' -to apply the normalisation to the project. - -If update-widgets fails, the check still runs, without the normalisation, and -says so. update-widgets refuses a project whose Mendix version is not exactly -mx's own ("Project is in different Mendix version", exit status 4) where mx -check accepts it, so an mx of the project's minor but another patch checks the -project and only skips the normalisation. 'mxcli setup mxbuild -p app.mpr' -fetches the project's exact version. +It checks the project as stored, which is what MxBuild and 'mxcli run --local' +build, and the exit status is the stored project's. When the check finds CE0463 +("The definition of this widget has changed"), it runs 'mx update-widgets' on +the same copy, checks it again, and labels each CE0463 in the stored project: + + at Data grid 2 'dgOrders': stale widget definition — ` + "`mxcli fix widgets`" + ` fixes it + at Combo box 'cbStatus': persists after update-widgets + +A stale definition is a widget instance older than its installed package; +'mxcli fix widgets -p app.mpr' updates it in the project (as Studio Pro's +"Update all widgets" does). One that persists is not cleared by a resync. The +re-check costs a second mx check, so it runs only when there is a CE0463. + + --no-update-widgets check as stored and skip the re-check (no labels) + --update-widgets normalise the copy first and check only that copy, as + earlier releases did by default; it hides every CE0463 + a resync clears, which the stored project still fails on + +update-widgets refuses a project whose Mendix version is not exactly mx's own +("Project is in different Mendix version", exit status 4) where mx check +accepts it, so with an mx of the project's minor but another patch the check +runs and the CE0463s go unlabelled (with --update-widgets: unnormalised), and +it says so. 'mxcli setup mxbuild -p app.mpr' fetches the project's exact +version. There is no Mendix version floor for check: Mendix 10 works (tested on 10.24). The 11.6.1 requirement is docker build's, for the Portable App Distribution. @@ -212,6 +223,7 @@ Examples: mxcli docker check -p app.mpr mxcli docker check -p app.mpr --mxbuild-path /path/to/mendix mxcli docker check -p app.mpr --no-update-widgets + mxcli docker check -p app.mpr --update-widgets `, Run: func(cmd *cobra.Command, args []string) { projectPath, _ := cmd.Flags().GetString("project") @@ -222,13 +234,15 @@ Examples: mxbuildPath, _ := cmd.Flags().GetString("mxbuild-path") noUpdateWidgets, _ := cmd.Flags().GetBool("no-update-widgets") + updateWidgets, _ := cmd.Flags().GetBool("update-widgets") opts := docker.CheckOptions{ - ProjectPath: projectPath, - MxBuildPath: mxbuildPath, - SkipUpdateWidgets: noUpdateWidgets, - Stdout: os.Stdout, - Stderr: os.Stderr, + ProjectPath: projectPath, + MxBuildPath: mxbuildPath, + SkipUpdateWidgets: noUpdateWidgets, + UpdateWidgetsFirst: updateWidgets, + Stdout: os.Stdout, + Stderr: os.Stderr, } // Neither of these is something mx check reports (#972). @@ -547,7 +561,9 @@ func init() { // Check command flags dockerCheckCmd.Flags().String("mxbuild-path", "", "Path to MxBuild/Mendix installation (used to find mx)") - dockerCheckCmd.Flags().Bool("no-update-widgets", false, "Skip 'mx update-widgets' before check") + dockerCheckCmd.Flags().Bool("no-update-widgets", false, "Check the project as stored without the update-widgets re-check that labels its CE0463 errors") + dockerCheckCmd.Flags().Bool("update-widgets", false, "Run 'mx update-widgets' on the temporary copy first and check only that normalised copy (hides CE0463 the stored project has)") + dockerCheckCmd.MarkFlagsMutuallyExclusive("no-update-widgets", "update-widgets") // Init command flags dockerInitCmd.Flags().StringP("output", "o", "", "Output directory (default: .docker/ next to MPR)") diff --git a/cmd/mxcli/docker/check.go b/cmd/mxcli/docker/check.go index 855811d361..d5c45f546f 100644 --- a/cmd/mxcli/docker/check.go +++ b/cmd/mxcli/docker/check.go @@ -3,12 +3,15 @@ package docker import ( + "bytes" "fmt" "io" "os" "os/exec" "path/filepath" + "regexp" "runtime" + "strconv" "strings" ) @@ -20,12 +23,24 @@ type CheckOptions struct { // MxBuildPath is an explicit path to the mxbuild executable (used to find mx). MxBuildPath string - // SkipUpdateWidgets skips the 'mx update-widgets' step before checking. - // By default, update-widgets runs first, on the temporary copy the check - // uses, to normalize pluggable widget definitions and prevent false CE0463 - // errors. + // By default the project is checked as stored, and when that finds CE0463 + // ("the definition of this widget has changed") the same temporary copy is + // re-checked after 'mx update-widgets' to label each CE0463 as a stale + // widget definition or as one that persists (ako/mxcli#568, #646). The + // verdict is always the stored project's. + + // SkipUpdateWidgets checks the project as stored and never runs + // 'mx update-widgets', not even for the re-check (--no-update-widgets). + // The verdict is the default's; only the labels are missing. SkipUpdateWidgets bool + // UpdateWidgetsFirst runs 'mx update-widgets' on the temporary copy before + // the check and checks only that widget-normalised copy (--update-widgets). + // It was the default until #646: it hides every CE0463 a resync clears, + // which the stored project still has and which fails MxBuild and + // `mxcli run --local`. + UpdateWidgetsFirst bool + // Stdout for output messages. Stdout io.Writer @@ -33,6 +48,12 @@ type CheckOptions struct { Stderr io.Writer } +// The two labels a CE0463 of the stored project gets from the re-check. +const ( + staleCE0463Label = "stale widget definition — `mxcli fix widgets` fixes it" + persistingCE0463Label = "persists after update-widgets" +) + // copyFile copies a single file from src to dst, preserving the source file mode. func copyFile(src, dst string) error { info, err := os.Stat(src) @@ -71,11 +92,22 @@ var mxCheckCmd = func(mxPath, mprPath string, args []string, w, stderr io.Writer // Check runs 'mx check' on the project to validate it before building. // +// It checks the project as stored: a CE0463 there fails MxBuild and +// `mxcli run --local`, so the check must report it (ako/mxcli#568, #646). When +// it finds one, it re-checks the copy after `mx update-widgets` and says which +// CE0463s a widget resync (`mxcli fix widgets`) clears; the verdict stays the +// stored project's. opts.UpdateWidgetsFirst restores the old behaviour of +// checking only the normalised copy. +// // It never modifies the project. `mx update-widgets` rewrites the model (and // turns an MPRv2 project into MPRv1), and `mx check` itself writes theme-cache/ // and deployment/sass/ — so both run on a temporary copy, and what mx prints is // reported against the project's own paths (ako/mxcli#951). func Check(opts CheckOptions) error { + if opts.SkipUpdateWidgets && opts.UpdateWidgetsFirst { + return fmt.Errorf("--no-update-widgets and --update-widgets cannot be combined: " + + "the first checks the project as stored, the second only a widget-normalised copy") + } w := opts.Stdout if w == nil { w = os.Stdout @@ -117,12 +149,12 @@ func Check(opts CheckOptions) error { defer errOut.Flush() fmt.Fprintln(w, "Checking a temporary copy (mx writes into the project it checks; the project on disk is not changed).") - // Normalize pluggable widget definitions so `mx check` does not report - // CE0463 ("widget definition changed") for definitions that only need a - // resync. On the copy, so neither the model nor its storage format changes. + // --update-widgets: normalise pluggable widget definitions first and check + // only the normalised copy, as docker check did by default before #646. On + // the copy, so neither the model nor its storage format changes. normalised := false - if !opts.SkipUpdateWidgets { - fmt.Fprintln(w, "Normalising widget definitions on the temporary copy...") + if opts.UpdateWidgetsFirst { + fmt.Fprintln(w, "Normalising widget definitions on the temporary copy (--update-widgets)...") if err := updateWidgetsCmd(mxPath, workMpr, out, errOut); err != nil { out.Flush() fmt.Fprintf(w, "Warning: update-widgets failed (continuing): %v\n", err) @@ -131,9 +163,10 @@ func Check(opts CheckOptions) error { } } - // Run mx check + // Run mx check. Its output is kept as well as shown, to find the CE0463s. fmt.Fprintf(w, "Checking project %s...\n", opts.ProjectPath) - checkErr := mxCheckCmd(mxPath, workMpr, nil, out, errOut) + var checkOut, checkErrOut bytes.Buffer + checkErr := mxCheckCmd(mxPath, workMpr, nil, io.MultiWriter(out, &checkOut), io.MultiWriter(errOut, &checkErrOut)) out.Flush() errOut.Flush() @@ -141,18 +174,20 @@ func Check(opts CheckOptions) error { case normalised: // #568 / #646: the normalisation can hide a CE0463 the stored project // still has, which then fails `run --local` and MxBuild. - fmt.Fprintln(w, "Note: widget definitions were normalised on a temporary copy before checking, so a") - fmt.Fprintln(w, " CE0463 the stored project still has is not reported here; it fails MxBuild and") - fmt.Fprintln(w, " `mxcli run --local`. Use --no-update-widgets to check the project as stored, and") - fmt.Fprintln(w, " `mxcli fix widgets` to apply the normalisation to the project.") - case !opts.SkipUpdateWidgets: + fmt.Fprintln(w, "Note: widget definitions were normalised on a temporary copy before checking") + fmt.Fprintln(w, " (--update-widgets), so a CE0463 the stored project still has is not reported here;") + fmt.Fprintln(w, " it fails MxBuild and `mxcli run --local`. Without --update-widgets docker check") + fmt.Fprintln(w, " checks the project as stored; `mxcli fix widgets` applies the normalisation to it.") + case opts.UpdateWidgetsFirst: // mendixlabs/mxcli#1629: update-widgets refuses a project of another // Mendix version than mx's own (exit status 4) where mx check accepts it. fmt.Fprintln(w, "Note: widget definitions were not normalised (update-widgets failed above), so this") fmt.Fprintln(w, " checked the project as stored, and a CE0463 may be one normalisation would clear.") - fmt.Fprintln(w, " If mx said \"Project is in different Mendix version\", update-widgets needs an mx of") - fmt.Fprintln(w, " the project's exact version, which mx check does not: `mxcli setup mxbuild -p") - fmt.Fprintln(w, " .mpr` fetches it, and --mxbuild-path ~/.mxcli/mxbuild/ selects it.") + fmt.Fprintln(w, versionMismatchHint) + case checkErr != nil && !opts.SkipUpdateWidgets: + if stored := ce0463Lines(checkOut.String() + "\n" + checkErrOut.String()); len(stored) > 0 { + recheckCE0463(w, out, mxPath, workMpr, opts.ProjectPath, stored, checkOut.String()) + } } if checkErr != nil { return fmt.Errorf("project check failed: %w", checkErr) @@ -162,6 +197,152 @@ func Check(opts CheckOptions) error { return nil } +// versionMismatchHint explains the commonest update-widgets failure +// (mendixlabs/mxcli#1629). +const versionMismatchHint = " If mx said \"Project is in different Mendix version\", update-widgets needs an mx of\n" + + " the project's exact version, which mx check does not: `mxcli setup mxbuild -p\n" + + " .mpr` fetches it, and --mxbuild-path ~/.mxcli/mxbuild/ selects it." + +// recheckCE0463 runs `mx update-widgets` on the copy the stored project was just +// checked on, checks it again, and labels each CE0463 of the stored project: a +// stale widget definition, which the resync clears and `mxcli fix widgets` +// applies to the project, or one that persists after it. The re-check's own +// output is not shown — the stored project's result is the one that counts. +// +// mx check's text output does not name a CE0463's page, so two same-named +// widgets on different pages print the same line; the lines are compared as +// counts, and a line that is partly cleared says how many of it were. +func recheckCE0463(w io.Writer, out *pathRewriter, mxPath, workMpr, projectPath string, stored []string, storedOutput string) { + fmt.Fprintf(w, "\nThe stored project has %d CE0463 %s. Re-checking the temporary copy after\n", + len(stored), plural(len(stored), "error", "errors")) + fmt.Fprintln(w, " `mx update-widgets` to tell stale widget definitions from other causes...") + + var uwOut bytes.Buffer + if err := updateWidgetsCmd(mxPath, workMpr, &uwOut, &uwOut); err != nil { + // Its output says why (mendixlabs/mxcli#1629: another patch version). + io.Copy(out, &uwOut) + out.Flush() + fmt.Fprintf(w, "The re-check could not tell which CE0463s are stale widget definitions: update-widgets failed: %v\n", err) + fmt.Fprintln(w, versionMismatchHint) + return + } + var after bytes.Buffer + _ = mxCheckCmd(mxPath, workMpr, nil, &after, &after) // non-zero whenever errors remain + afterTotal, ok := appErrorCount(after.String()) + if !ok { + fmt.Fprintln(w, "The re-check could not tell which CE0463s are stale widget definitions: mx check did") + fmt.Fprintln(w, " not complete on the copy after update-widgets.") + return + } + + remaining := map[string]int{} + for _, l := range ce0463Lines(after.String()) { + remaining[l]++ + } + total := map[string]int{} + var order []string + for _, l := range stored { + if total[l] == 0 { + order = append(order, l) + } + total[l]++ + } + stale, persisting := 0, 0 + fmt.Fprintln(w, "CE0463 in the stored project:") + for _, l := range order { + n := total[l] + p := min(remaining[l], n) + s := n - p + stale += s + persisting += p + at := ce0463Location(l) + if s > 0 { + fmt.Fprintf(w, " at %s%s: %s\n", at, countOf(s, n), staleCE0463Label) + } + if p > 0 { + fmt.Fprintf(w, " at %s%s: %s\n", at, countOf(p, n), persistingCE0463Label) + } + } + switch { + case persisting == 0 && stale == 1: + fmt.Fprintln(w, "It is a stale widget definition. This updates it in the project, as Studio Pro's") + fmt.Fprintln(w, " \"Update all widgets\" does:") + case persisting == 0: + fmt.Fprintf(w, "All %d are stale widget definitions. This updates them in the project, as Studio\n", stale) + fmt.Fprintln(w, " Pro's \"Update all widgets\" does:") + case stale == 0 && persisting == 1: + fmt.Fprintln(w, "It is not a stale widget definition: it persists after update-widgets, so a widget") + fmt.Fprintln(w, " resync (`mxcli fix widgets`) does not clear it.") + case stale == 0: + fmt.Fprintf(w, "None of the %d is a stale widget definition: all persist after update-widgets, so a\n", persisting) + fmt.Fprintln(w, " widget resync (`mxcli fix widgets`) does not clear them.") + default: + fmt.Fprintf(w, "%d of %d are stale widget definitions; %d persist after update-widgets, which a\n", + stale, stale+persisting, persisting) + fmt.Fprintln(w, " widget resync does not clear. This updates the stale ones in the project:") + } + if stale > 0 { + fmt.Fprintf(w, " mxcli fix widgets -p %s\n", projectPath) + } + if storedTotal, ok := appErrorCount(storedOutput); ok { + fmt.Fprintf(w, "As stored: %d %s. ", storedTotal, plural(storedTotal, "error", "errors")) + } + fmt.Fprintf(w, "After update-widgets: %d %s.\n", afterTotal, plural(afterTotal, "error", "errors")) +} + +// countOf prints "(k of n)" when only part of n identical lines share a label. +func countOf(k, n int) string { + switch { + case n == 1: + return "" + case k == n: + return fmt.Sprintf(" (%d×)", n) + default: + return fmt.Sprintf(" (%d of %d)", k, n) + } +} + +func plural(n int, one, many string) string { + if n == 1 { + return one + } + return many +} + +// ce0463Lines returns mx check's CE0463 lines, trimmed, in the order printed: +// +// [error] [CE0463] "The definition of this widget has changed. ..." at Data grid 2 'dgRequests' +func ce0463Lines(output string) []string { + var lines []string + for _, l := range strings.Split(output, "\n") { + if l = strings.TrimSpace(l); strings.Contains(l, "[CE0463]") { + lines = append(lines, l) + } + } + return lines +} + +// ce0463Location is the widget a CE0463 line names ("Data grid 2 'dgRequests'"), +// or the whole line when it names none. +func ce0463Location(line string) string { + if i := strings.LastIndex(line, `" at `); i >= 0 { + return line[i+len(`" at `):] + } + return line +} + +var appErrorCountRe = regexp.MustCompile(`The app contains: (\d+) errors?\.`) + +// appErrorCount reads mx check's closing "The app contains: N errors." line. +func appErrorCount(output string) (int, bool) { + m := appErrorCountRe.FindStringSubmatch(output) + if m == nil { + return 0, false + } + n, err := strconv.Atoi(m[1]) + return n, err == nil +} + // MxCheckOnCopy runs `mx check args...` on a temporary copy of the // project, so the check cannot write into it, and returns mx's error (a non-zero // exit when the project has errors). Output naming the copy is rewritten to name diff --git a/cmd/mxcli/docker/check_integration_test.go b/cmd/mxcli/docker/check_integration_test.go index 3a668b61fe..2f00eb7f0d 100644 --- a/cmd/mxcli/docker/check_integration_test.go +++ b/cmd/mxcli/docker/check_integration_test.go @@ -28,6 +28,10 @@ import ( // Requires a resolvable mx (provided by the CI integration job's // `mxcli setup mxbuild`); skips otherwise. Uses `mx create-project`, whose output // is MPRv2, as the fixture — the same scaffolding the executor roundtrip suite uses. +// +// Since ako/mxcli#646 update-widgets runs before the check only with +// --update-widgets (UpdateWidgetsFirst), which is what this test now passes so +// that it still runs the converting tool. func TestCheck_PreservesMPRv2StorageFormat(t *testing.T) { mxPath, err := ResolveMx("") if err != nil { @@ -54,9 +58,10 @@ func TestCheck_PreservesMPRv2StorageFormat(t *testing.T) { var stdout, stderr bytes.Buffer if err := Check(CheckOptions{ - ProjectPath: mprPath, - Stdout: &stdout, - Stderr: &stderr, + ProjectPath: mprPath, + UpdateWidgetsFirst: true, + Stdout: &stdout, + Stderr: &stderr, }); err != nil { t.Fatalf("Check failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout.String(), stderr.String()) } @@ -83,10 +88,12 @@ func mprStorageVersion(t *testing.T, mprPath string) types.MPRVersion { } // TestCheck_LeavesProjectUntouched is the end-to-end guard for ako/mxcli#951: -// with real mx, a default `docker check` and a --no-update-widgets one leave -// every file of an MPRv1 and an MPRv2 project byte-identical with the same -// mtime. Before the fix, update-widgets rewrote a v1 .mpr permanently, and mx -// check wrote theme-cache/ and deployment/sass/ in both formats. +// with real mx, a default `docker check`, a --no-update-widgets one and an +// --update-widgets one leave every file of an MPRv1 and an MPRv2 project +// byte-identical with the same mtime. Before the fix, update-widgets rewrote a +// v1 .mpr permanently, and mx check wrote theme-cache/ and deployment/sass/ in +// both formats. (The default ran update-widgets until #646; --update-widgets +// is the mode that runs it on a clean project now.) func TestCheck_LeavesProjectUntouched(t *testing.T) { mxPath, err := ResolveMx("") if err != nil { @@ -109,9 +116,17 @@ func TestCheck_LeavesProjectUntouched(t *testing.T) { return filepath.Join(dir, "App.mpr") } + modes := []struct { + name string + opts CheckOptions + }{ + {"default", CheckOptions{}}, + {"no-update-widgets", CheckOptions{SkipUpdateWidgets: true}}, + {"update-widgets", CheckOptions{UpdateWidgetsFirst: true}}, + } for _, format := range []string{"v2", "v1"} { - for _, skip := range []bool{false, true} { - t.Run(fmt.Sprintf("%s/no-update-widgets=%v", format, skip), func(t *testing.T) { + for _, mode := range modes { + t.Run(fmt.Sprintf("%s/%s", format, mode.name), func(t *testing.T) { mprPath := scaffold(t) if format == "v1" { // update-widgets converts a v2 project to v1 — the fixture we want. @@ -125,7 +140,9 @@ func TestCheck_LeavesProjectUntouched(t *testing.T) { dir := filepath.Dir(mprPath) before := treeState(t, dir) var stdout bytes.Buffer - if err := Check(CheckOptions{ProjectPath: mprPath, SkipUpdateWidgets: skip, Stdout: &stdout, Stderr: &stdout}); err != nil { + opts := mode.opts + opts.ProjectPath, opts.Stdout, opts.Stderr = mprPath, &stdout, &stdout + if err := Check(opts); err != nil { t.Fatalf("Check: %v\n%s", err, stdout.String()) } if d := diffStates(before, treeState(t, dir)); len(d) > 0 { diff --git a/cmd/mxcli/docker/check_readonly_test.go b/cmd/mxcli/docker/check_readonly_test.go index bc7ec574fe..52160b355a 100644 --- a/cmd/mxcli/docker/check_readonly_test.go +++ b/cmd/mxcli/docker/check_readonly_test.go @@ -97,8 +97,9 @@ func addProjectDirs(t *testing.T, dir string) { // stubTools replaces mx resolution and both mx invocations with stubs that // mutate the project they are pointed at the way the real tools do, and record -// the paths they were given. -func stubTools(t *testing.T) (seen *[]string) { +// the paths they were given. Each widget named in staleWidgets gets a CE0463 +// from mx check until update-widgets has rewritten the model it checks. +func stubTools(t *testing.T, staleWidgets ...string) (seen *[]string) { t.Helper() var paths []string origResolve, origUW, origCheck := resolveMxForCheck, updateWidgetsCmd, mxCheckCmd @@ -131,42 +132,63 @@ func stubTools(t *testing.T) (seen *[]string) { return err } // mx reports the project it loaded by path in some messages. - fmt.Fprintf(w, "Loading %s\nThe app contains: 0 errors.\n", mprPath) + fmt.Fprintf(w, "Loading %s\n", mprPath) + if b, _ := os.ReadFile(mprPath); string(b) != "rewritten by update-widgets" && len(staleWidgets) > 0 { + for _, el := range staleWidgets { + fmt.Fprintln(w, ce0463Line(el)) + } + fmt.Fprintf(w, "The app contains: %d errors.\n", len(staleWidgets)) + return fmt.Errorf("exit status 1") + } + fmt.Fprintln(w, "The app contains: 0 errors.") return nil } return &paths } +// The project is untouched in every mode: the default on a clean project (one +// check), the default on a project with a stale widget (the re-check runs +// update-widgets — on the copy), --no-update-widgets, and --update-widgets. func TestCheck_DoesNotModifyProject(t *testing.T) { for _, tc := range []struct { name string fixture func(*testing.T) string skipUpdate bool + updateFirst bool + staleWidgets []string wantUWOnACopy bool }{ - {"v1", v1Fixture, false, true}, - {"v2", v2Fixture, false, true}, - {"v1 --no-update-widgets", v1Fixture, true, false}, - {"v2 --no-update-widgets", v2Fixture, true, false}, + {"v1", v1Fixture, false, false, nil, false}, + {"v2", v2Fixture, false, false, nil, false}, + {"v1 stale widget", v1Fixture, false, false, []string{"Data grid 2 'dg'"}, true}, + {"v2 stale widget", v2Fixture, false, false, []string{"Data grid 2 'dg'"}, true}, + {"v1 --no-update-widgets", v1Fixture, true, false, []string{"Data grid 2 'dg'"}, false}, + {"v2 --no-update-widgets", v2Fixture, true, false, []string{"Data grid 2 'dg'"}, false}, + {"v1 --update-widgets", v1Fixture, false, true, []string{"Data grid 2 'dg'"}, true}, + {"v2 --update-widgets", v2Fixture, false, true, []string{"Data grid 2 'dg'"}, true}, } { t.Run(tc.name, func(t *testing.T) { mprPath := tc.fixture(t) projectDir := filepath.Dir(mprPath) addProjectDirs(t, projectDir) - seen := stubTools(t) + seen := stubTools(t, tc.staleWidgets...) tmpRoot := t.TempDir() t.Setenv("TMPDIR", tmpRoot) before := treeState(t, projectDir) var out bytes.Buffer - if err := Check(CheckOptions{ - ProjectPath: mprPath, - SkipUpdateWidgets: tc.skipUpdate, - Stdout: &out, - Stderr: io.Discard, - }); err != nil { - t.Fatalf("Check: %v\n%s", err, out.String()) + err := Check(CheckOptions{ + ProjectPath: mprPath, + SkipUpdateWidgets: tc.skipUpdate, + UpdateWidgetsFirst: tc.updateFirst, + Stdout: &out, + Stderr: io.Discard, + }) + // The stored project fails exactly when it has a CE0463 and the + // mode checks it as stored. + if wantErr := len(tc.staleWidgets) > 0 && !tc.updateFirst; (err != nil) != wantErr { + t.Fatalf("Check error = %v, want error %v\n%s", err, wantErr, out.String()) } after := treeState(t, projectDir) if d := diffStates(before, after); len(d) > 0 { @@ -182,10 +204,16 @@ func TestCheck_DoesNotModifyProject(t *testing.T) { t.Errorf("mx was pointed at the project itself: %s", s) } } - ranUW := strings.HasPrefix((*seen)[0], "update-widgets ") + ranUW := false + for _, s := range *seen { + ranUW = ranUW || strings.HasPrefix(s, "update-widgets ") + } if ranUW != tc.wantUWOnACopy { t.Errorf("update-widgets ran = %v, want %v (%v)", ranUW, tc.wantUWOnACopy, *seen) } + if first := strings.Fields((*seen)[0])[0]; tc.updateFirst != (first == "update-widgets") { + t.Errorf("first mx step = %q; update-widgets comes first only with --update-widgets (%v)", first, *seen) + } // The output names the project, not the copy, and says what happened. got := out.String() @@ -195,9 +223,12 @@ func TestCheck_DoesNotModifyProject(t *testing.T) { if !strings.Contains(got, "Loading "+mprPath) { t.Errorf("mx output not reported against the original path:\n%s", got) } - if tc.wantUWOnACopy && !strings.Contains(got, "normalised on a temporary copy") { + if tc.updateFirst && !strings.Contains(got, "normalised on a temporary copy") { t.Errorf("output does not say widgets were normalised on a copy:\n%s", got) } + if tc.wantUWOnACopy && !tc.updateFirst && !strings.Contains(got, staleCE0463Label) { + t.Errorf("output does not label the stale widget's CE0463:\n%s", got) + } // The copy is cleaned up. if entries, _ := os.ReadDir(tmpRoot); len(entries) != 0 { @@ -359,7 +390,9 @@ func TestWithProjectCopy_SkipsNamedFoldersButNeverTheProject(t *testing.T) { // in different Mendix version - 10.24.13.86719" and exits 4; mx check reports 0 // errors. The check carried on, as it should — and then closed with a note that // widget definitions "were normalised on a temporary copy", a few lines after -// the warning that they were not. +// the warning that they were not. Since #646 update-widgets runs first only with +// --update-widgets; the default's re-check is TestCheck_FailedRecheckLeaves +// CE0463Unlabelled. func TestCheck_FailedUpdateWidgetsIsNotReportedAsNormalised(t *testing.T) { mprPath := v1Fixture(t) stubTools(t) @@ -370,7 +403,7 @@ func TestCheck_FailedUpdateWidgetsIsNotReportedAsNormalised(t *testing.T) { t.Setenv("TMPDIR", t.TempDir()) var out bytes.Buffer - if err := Check(CheckOptions{ProjectPath: mprPath, Stdout: &out, Stderr: io.Discard}); err != nil { + if err := Check(CheckOptions{ProjectPath: mprPath, UpdateWidgetsFirst: true, Stdout: &out, Stderr: io.Discard}); err != nil { t.Fatalf("a failed update-widgets must not stop the check: %v\n%s", err, out.String()) } got := out.String() diff --git a/cmd/mxcli/docker/check_stored_integration_test.go b/cmd/mxcli/docker/check_stored_integration_test.go new file mode 100644 index 0000000000..cad1621de4 --- /dev/null +++ b/cmd/mxcli/docker/check_stored_integration_test.go @@ -0,0 +1,148 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build integration + +package docker + +import ( + "archive/zip" + "bytes" + "io" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +// TestCheck_ReportsStaleWidgetCE0463OfTheStoredProject is ako/mxcli#568 and +// #646 end to end, with real mx. A widget package that gained a property after +// its widgets were authored — what any widget upgrade does — leaves every +// stored instance stale: CE0463, which fails MxBuild and `mxcli run --local`, +// and which `mx update-widgets` clears. Until #646 the default docker check ran +// update-widgets on its copy first and passed such a project. +// +// Measured on 11.13.0 with Data grid 2 3.4.0 plus one boolean property: mx check +// on the stored project reports 11 CE0463, the old default 0 errors. +func TestCheck_ReportsStaleWidgetCE0463OfTheStoredProject(t *testing.T) { + mxPath, err := ResolveMx("") + if err != nil { + t.Skipf("mx not resolvable: %v", err) + } + // Not t.TempDir(): create-project's template extraction hits + // PathTooLongException under a long test name. + dir, err := os.MkdirTemp("", "stl") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { os.RemoveAll(dir) }) + scaffold := exec.Command(mxPath, "create-project") + scaffold.Dir = dir + PrepareMxCommand(scaffold) + if out, err := scaffold.CombinedOutput(); err != nil { + t.Skipf("mx create-project failed: %v\n%s", err, out) + } + mprPath := filepath.Join(dir, "App.mpr") + mpk := filepath.Join(dir, "widgets", "com.mendix.widget.web.Datagrid.mpk") + if _, err := os.Stat(mpk); err != nil { + t.Skipf("the scaffolded project has no Data grid 2 package to make stale: %v", err) + } + addWidgetProperty(t, mpk, "Datagrid.xml") + before := treeState(t, dir) + + check := func(opts CheckOptions) (string, error) { + var out bytes.Buffer + opts.ProjectPath, opts.Stdout, opts.Stderr = mprPath, &out, &out + err := Check(opts) + return out.String(), err + } + + // Precondition, and the ground truth: the stored project has CE0463. + stored, err := check(CheckOptions{SkipUpdateWidgets: true}) + n := len(ce0463Lines(stored)) + if n == 0 { + t.Skipf("no CE0463 after making Data grid 2 stale (no instance in this template?):\n%s", stored) + } + if err == nil { + t.Fatalf("--no-update-widgets passed a project with %d stored CE0463:\n%s", n, stored) + } + if strings.Contains(stored, staleCE0463Label) { + t.Errorf("--no-update-widgets labelled the CE0463s; it must not run update-widgets:\n%s", stored) + } + + // The default reports them, labels each as stale, and fails. + got, err := check(CheckOptions{}) + if err == nil { + t.Fatalf("docker check passed a project whose stored model has %d CE0463 (#568):\n%s", n, got) + } + if c := len(ce0463Lines(got)); c != n { + t.Errorf("default reported %d CE0463, the stored project has %d:\n%s", c, n, got) + } + if !strings.Contains(got, staleCE0463Label) || strings.Contains(got, persistingCE0463Label) { + t.Errorf("the CE0463s are not all labelled %q:\n%s", staleCE0463Label, got) + } + if !strings.Contains(got, "After update-widgets: 0 errors.") { + t.Errorf("the re-check does not report the resynced copy clean:\n%s", got) + } + + // --update-widgets is the old behaviour: it checks the normalised copy only. + old, err := check(CheckOptions{UpdateWidgetsFirst: true}) + if err != nil { + t.Errorf("--update-widgets failed on a project whose only errors a resync clears: %v\n%s", err, old) + } + + if d := diffStates(before, treeState(t, dir)); len(d) > 0 { + t.Errorf("docker check modified the project:\n %v", d) + } +} + +// addWidgetProperty adds a boolean property to the widget definition xmlName in +// the package at mpkPath, the way a newer package version does: stored +// instances of the widget lack it from then on. +func addWidgetProperty(t *testing.T, mpkPath, xmlName string) { + t.Helper() + zr, err := zip.OpenReader(mpkPath) + if err != nil { + t.Fatal(err) + } + var buf bytes.Buffer + zw := zip.NewWriter(&buf) + found := false + for _, f := range zr.File { + rc, err := f.Open() + if err != nil { + t.Fatal(err) + } + data, err := io.ReadAll(rc) + rc.Close() + if err != nil { + t.Fatal(err) + } + if f.Name == xmlName { + i := bytes.Index(data, []byte(`Stale probe`) + data = append(append(append([]byte{}, data[:i]...), probe...), data[i:]...) + found = true + } + w, err := zw.CreateHeader(&zip.FileHeader{Name: f.Name, Method: zip.Deflate, Modified: f.Modified}) + if err != nil { + t.Fatal(err) + } + if _, err := w.Write(data); err != nil { + t.Fatal(err) + } + } + zr.Close() + if !found { + t.Fatalf("%s not found in %s", xmlName, mpkPath) + } + if err := zw.Close(); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(mpkPath, buf.Bytes(), 0o644); err != nil { + t.Fatal(err) + } +} diff --git a/cmd/mxcli/docker/check_stored_test.go b/cmd/mxcli/docker/check_stored_test.go new file mode 100644 index 0000000000..211494d86b --- /dev/null +++ b/cmd/mxcli/docker/check_stored_test.go @@ -0,0 +1,261 @@ +// SPDX-License-Identifier: Apache-2.0 + +// ako/mxcli#568 and #646: `docker check` ran `mx update-widgets` on its copy +// before `mx check`, so it validated a widget-normalised model, never the +// stored one. A CE0463 the stored project had was cleared on the copy and the +// check reported 0 errors, while `mxcli run --local` built the project as stored +// and failed: +// +// | command | result | +// | mxcli docker check | 0 errors | +// | mxcli docker check --no-update-widgets | 3 errors (CE0463) | +// | mxcli run --local | build fails, same 3 | +// +// docker check now checks the stored project by default. When that finds +// CE0463, it re-checks the same temporary copy after update-widgets and labels +// each CE0463 as a stale widget definition (`mxcli fix widgets` clears it) or +// as one that persists. The old behaviour is --update-widgets. +package docker + +import ( + "bytes" + "fmt" + "io" + "os" + "path/filepath" + "strings" + "testing" +) + +// ce0463Line is how mx check prints a CE0463 (measured on 11.13.0). The text +// does not name the page, so two widgets of the same name on different pages +// print the same line. +func ce0463Line(element string) string { + return `[error] [CE0463] "The definition of this widget has changed. Update this widget by right-clicking it ` + + `and selecting 'Update widget', or select 'Update all widgets' to update all widgets in the app." at ` + element +} + +// stubStaleWidgets makes the stubbed mx behave like a project whose stored +// widgets need a resync: mx check reports every line in stale and persist until +// update-widgets has run on the copy it is given, and only persist afterwards, +// plus other errors either way. It records each invocation. +func stubStaleWidgets(t *testing.T, stale, persist, other []string) (calls *[]string) { + t.Helper() + var seen []string + origResolve, origUW, origCheck := resolveMxForCheck, updateWidgetsCmd, mxCheckCmd + t.Cleanup(func() { resolveMxForCheck, updateWidgetsCmd, mxCheckCmd = origResolve, origUW, origCheck }) + + marker := func(mprPath string) string { return filepath.Join(filepath.Dir(mprPath), "widgets-updated") } + resolveMxForCheck = func(string, string) (string, error) { return "mx", nil } + updateWidgetsCmd = func(_, mprPath string, w, _ io.Writer) error { + seen = append(seen, "update-widgets") + fmt.Fprintln(w, "Updating widgets...") + return os.WriteFile(marker(mprPath), nil, 0o644) + } + mxCheckCmd = func(_, mprPath string, _ []string, w, _ io.Writer) error { + seen = append(seen, "check") + fmt.Fprintln(w, "Checking app for errors...") + var lines []string + if _, err := os.Stat(marker(mprPath)); err != nil { + for _, e := range stale { + lines = append(lines, ce0463Line(e)) + } + } + for _, e := range persist { + lines = append(lines, ce0463Line(e)) + } + lines = append(lines, other...) + for _, l := range lines { + fmt.Fprintln(w, l) + } + fmt.Fprintf(w, "The app contains: %d errors.\n", len(lines)) + if len(lines) > 0 { + return fmt.Errorf("exit status 1") + } + return nil + } + return &seen +} + +// The reported symptom: a project whose stored widgets carry CE0463 that a +// resync clears. The default must fail on it, as MxBuild and `run --local` do. +func TestCheck_ChecksTheStoredProjectByDefault(t *testing.T) { + mprPath := v1Fixture(t) + calls := stubStaleWidgets(t, + []string{"Tree node 'tnCustomer'", "Tree node 'tnPart'", "Data grid 2 'dgRequests'"}, nil, nil) + t.Setenv("TMPDIR", t.TempDir()) + + var out bytes.Buffer + err := Check(CheckOptions{ProjectPath: mprPath, Stdout: &out, Stderr: io.Discard}) + got := out.String() + if err == nil { + t.Fatalf("docker check passed a project whose stored model has 3 CE0463 errors "+ + "(the #568 false pass: it checked a widget-normalised copy):\n%s", got) + } + if !strings.Contains(got, "The app contains: 3 errors.") { + t.Errorf("the stored project's own check result is not reported:\n%s", got) + } + if len(*calls) == 0 || (*calls)[0] != "check" { + t.Errorf("the first mx step must be a check of the project as stored, got %v", *calls) + } + for _, el := range []string{"Tree node 'tnCustomer'", "Tree node 'tnPart'", "Data grid 2 'dgRequests'"} { + want := "at " + el + ": " + staleCE0463Label + if !strings.Contains(got, want) { + t.Errorf("CE0463 at %s not labelled %q:\n%s", el, staleCE0463Label, got) + } + } + if !strings.Contains(got, "mxcli fix widgets -p "+mprPath) { + t.Errorf("output does not name the command that clears them for this project:\n%s", got) + } + if !strings.Contains(got, "After update-widgets: 0 errors.") { + t.Errorf("output does not say what the check reports once the widgets are resynced:\n%s", got) + } +} + +// A CE0463 a resync does not clear is not a stale definition, and must not be +// labelled as one. Same-named widgets on different pages print identical lines, +// so the labels are counted per line. +func TestCheck_LabelsCE0463ThatPersistsAfterUpdateWidgets(t *testing.T) { + mprPath := v1Fixture(t) + stubStaleWidgets(t, + []string{"Data grid 2 'dataGrid2_1'", "Data grid 2 'dataGrid2_1'", "Combo box 'cb'"}, + []string{"Data grid 2 'dataGrid2_1'", "Image 'img'"}, + []string{`[error] [CE0117] "Error(s) in expression." at Microflow 'ACT_X'`}) + t.Setenv("TMPDIR", t.TempDir()) + + var out bytes.Buffer + if err := Check(CheckOptions{ProjectPath: mprPath, Stdout: &out, Stderr: io.Discard}); err == nil { + t.Fatalf("Check passed a project with errors:\n%s", out.String()) + } + got := out.String() + for _, want := range []string{ + "at Data grid 2 'dataGrid2_1' (2 of 3): " + staleCE0463Label, + "at Data grid 2 'dataGrid2_1' (1 of 3): " + persistingCE0463Label, + "at Combo box 'cb': " + staleCE0463Label, + "at Image 'img': " + persistingCE0463Label, + "3 of 5 are stale widget definitions; 2 persist after update-widgets", + " mxcli fix widgets -p " + mprPath, + "As stored: 6 errors. After update-widgets: 3 errors.", + } { + if !strings.Contains(got, want) { + t.Errorf("output lacks %q:\n%s", want, got) + } + } + if strings.Contains(got, "at ACT_X") || strings.Contains(got, "Microflow 'ACT_X': ") { + t.Errorf("a non-CE0463 error was labelled:\n%s", got) + } +} + +// The re-check costs a second mx check, so it runs only when there is a CE0463 +// to explain: a clean project, or one whose errors are all something else, is +// checked once and update-widgets never runs. +func TestCheck_NoCE0463NoSecondCheck(t *testing.T) { + for _, tc := range []struct { + name string + other []string + wantErr bool + }{ + {"clean", nil, false}, + {"other errors only", []string{`[error] [CE0117] "Error(s) in expression." at Microflow 'ACT_X'`}, true}, + } { + t.Run(tc.name, func(t *testing.T) { + mprPath := v1Fixture(t) + calls := stubStaleWidgets(t, nil, nil, tc.other) + t.Setenv("TMPDIR", t.TempDir()) + var out bytes.Buffer + err := Check(CheckOptions{ProjectPath: mprPath, Stdout: &out, Stderr: io.Discard}) + if (err != nil) != tc.wantErr { + t.Fatalf("Check error = %v, want error %v\n%s", err, tc.wantErr, out.String()) + } + if strings.Join(*calls, ",") != "check" { + t.Errorf("mx steps = %v, want one check and no update-widgets", *calls) + } + if strings.Contains(out.String(), "update-widgets") { + t.Errorf("output talks about update-widgets although nothing needed it:\n%s", out.String()) + } + }) + } +} + +// --update-widgets is the old behaviour, kept for whoever wants it: normalise +// the copy, then check only that. On the #568 project it reports 0 errors and +// passes — with a note saying what that hides. +func TestCheck_UpdateWidgetsFlagChecksTheNormalisedCopy(t *testing.T) { + mprPath := v1Fixture(t) + calls := stubStaleWidgets(t, []string{"Tree node 'tnCustomer'"}, nil, nil) + t.Setenv("TMPDIR", t.TempDir()) + + var out bytes.Buffer + if err := Check(CheckOptions{ProjectPath: mprPath, UpdateWidgetsFirst: true, Stdout: &out, Stderr: io.Discard}); err != nil { + t.Fatalf("--update-widgets: %v\n%s", err, out.String()) + } + if strings.Join(*calls, ",") != "update-widgets,check" { + t.Errorf("mx steps = %v, want update-widgets then one check", *calls) + } + got := out.String() + if !strings.Contains(got, "The app contains: 0 errors.") { + t.Errorf("--update-widgets did not check the normalised copy:\n%s", got) + } + if !strings.Contains(got, "normalised on a temporary copy") { + t.Errorf("output does not say the check ran on a normalised copy:\n%s", got) + } +} + +// --no-update-widgets keeps working: it checks the project as stored, as the +// default now does, and never runs update-widgets — not even to label. +func TestCheck_NoUpdateWidgetsChecksStoredWithoutRecheck(t *testing.T) { + mprPath := v1Fixture(t) + calls := stubStaleWidgets(t, []string{"Tree node 'tnCustomer'"}, nil, nil) + t.Setenv("TMPDIR", t.TempDir()) + + var out bytes.Buffer + if err := Check(CheckOptions{ProjectPath: mprPath, SkipUpdateWidgets: true, Stdout: &out, Stderr: io.Discard}); err == nil { + t.Fatalf("--no-update-widgets passed a project with a stored CE0463:\n%s", out.String()) + } + if strings.Join(*calls, ",") != "check" { + t.Errorf("mx steps = %v, want one check and no update-widgets", *calls) + } +} + +func TestCheck_UpdateWidgetsFlagsAreExclusive(t *testing.T) { + mprPath := v1Fixture(t) + calls := stubStaleWidgets(t, nil, nil, nil) + err := Check(CheckOptions{ProjectPath: mprPath, SkipUpdateWidgets: true, UpdateWidgetsFirst: true, Stdout: io.Discard, Stderr: io.Discard}) + if err == nil { + t.Fatal("Check accepted both --no-update-widgets and --update-widgets") + } + if len(*calls) != 0 { + t.Errorf("mx ran although the options contradict each other: %v", *calls) + } +} + +// mendixlabs/mxcli#1629 in the re-check: update-widgets refuses a project of +// another patch version. The verdict is still the stored project's, and the +// output says the CE0463s could not be told apart rather than labelling them. +func TestCheck_FailedRecheckLeavesCE0463Unlabelled(t *testing.T) { + mprPath := v1Fixture(t) + calls := stubStaleWidgets(t, []string{"Tree node 'tnCustomer'"}, nil, nil) + updateWidgetsCmd = func(_, _ string, w, _ io.Writer) error { + *calls = append(*calls, "update-widgets") + fmt.Fprintln(w, "Project is in different Mendix version - 10.24.13.86719") + return fmt.Errorf("exit status 4") + } + t.Setenv("TMPDIR", t.TempDir()) + + var out bytes.Buffer + if err := Check(CheckOptions{ProjectPath: mprPath, Stdout: &out, Stderr: io.Discard}); err == nil { + t.Fatalf("Check passed a project with a stored CE0463:\n%s", out.String()) + } + got := out.String() + if strings.Contains(got, staleCE0463Label) || strings.Contains(got, persistingCE0463Label) { + t.Errorf("CE0463 labelled although update-widgets failed:\n%s", got) + } + for _, want := range []string{"Project is in different Mendix version", "could not tell"} { + if !strings.Contains(got, want) { + t.Errorf("output lacks %q:\n%s", want, got) + } + } + if strings.Join(*calls, ",") != "check,update-widgets" { + t.Errorf("mx steps = %v, want the check, then the failed update-widgets and no re-check", *calls) + } +} diff --git a/cmd/mxcli/docker/check_test.go b/cmd/mxcli/docker/check_test.go index 5ef6fc560e..21789d6baf 100644 --- a/cmd/mxcli/docker/check_test.go +++ b/cmd/mxcli/docker/check_test.go @@ -150,7 +150,12 @@ echo "$1" >> ` + logFile + "\n" return dir, logFile } -func TestCheck_UpdateWidgetsBeforeCheck(t *testing.T) { +// TestCheck_DefaultChecksStoredProject: since ako/mxcli#646 the default runs no +// update-widgets before the check — the fake mx reports no CE0463, so it runs +// none at all. (This test asserted update-widgets-then-check for the default +// until then; that order is now TestCheck_UpdateWidgetsBeforeCheck's, behind +// --update-widgets.) +func TestCheck_DefaultChecksStoredProject(t *testing.T) { if runtime.GOOS == "windows" { t.Skip("shell script test not supported on Windows") } @@ -158,11 +163,38 @@ func TestCheck_UpdateWidgetsBeforeCheck(t *testing.T) { mxDir, logFile := createFakeMxDir(t) var stdout, stderr bytes.Buffer - opts := CheckOptions{ + if err := Check(CheckOptions{ ProjectPath: fakeProject(t), MxBuildPath: mxDir, Stdout: &stdout, Stderr: &stderr, + }); err != nil { + t.Fatalf("Check: %v\n%s", err, stdout.String()) + } + + logBytes, err := os.ReadFile(logFile) + if err != nil { + t.Fatalf("failed to read command log: %v", err) + } + if string(logBytes) != "check\n" { + t.Errorf("want exactly one mx check and no update-widgets, got log:\n%s", logBytes) + } +} + +func TestCheck_UpdateWidgetsBeforeCheck(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("shell script test not supported on Windows") + } + + mxDir, logFile := createFakeMxDir(t) + + var stdout, stderr bytes.Buffer + opts := CheckOptions{ + ProjectPath: fakeProject(t), + MxBuildPath: mxDir, + UpdateWidgetsFirst: true, // --update-widgets: the default before #646 + Stdout: &stdout, + Stderr: &stderr, } Check(opts) @@ -238,6 +270,7 @@ func TestUpdateWidgetsPathArg_AbsolutizesBareFilename(t *testing.T) { // TestCheck_UpdateWidgetsReceivesAbsolutePath verifies the invocation passes an // absolute path to `mx update-widgets` even when ProjectPath is a bare filename // (as with `mxcli docker check -p app.mpr` run from the project directory). +// --update-widgets is what runs it unconditionally since #646. func TestCheck_UpdateWidgetsReceivesAbsolutePath(t *testing.T) { if runtime.GOOS == "windows" { t.Skip("shell script test not supported on Windows") @@ -254,7 +287,7 @@ func TestCheck_UpdateWidgetsReceivesAbsolutePath(t *testing.T) { var stdout, stderr bytes.Buffer // Bare filename — the crash trigger. t.Chdir(filepath.Dir(fakeProject(t))) - Check(CheckOptions{ProjectPath: "fake.mpr", MxBuildPath: dir, Stdout: &stdout, Stderr: &stderr}) + Check(CheckOptions{ProjectPath: "fake.mpr", MxBuildPath: dir, UpdateWidgetsFirst: true, Stdout: &stdout, Stderr: &stderr}) logBytes, err := os.ReadFile(logFile) if err != nil { diff --git a/cmd/mxcli/docker_help_test.go b/cmd/mxcli/docker_help_test.go index b44f727ea2..0053bee289 100644 --- a/cmd/mxcli/docker_help_test.go +++ b/cmd/mxcli/docker_help_test.go @@ -28,3 +28,46 @@ func TestDockerHelp_VersionFloorIsBuildsNotChecks(t *testing.T) { } } } + +// ako/mxcli#568, #646: docker check's help said update-widgets ran first "to +// prevent false CE0463 errors" — the CE0463s it cleared were real, and failed +// the build of the stored project. The default now checks the project as +// stored; the help has to say so, name the flag that restores the old +// behaviour, and keep accepting --no-update-widgets. +func TestDockerCheckHelp_StoredProjectIsTheDefault(t *testing.T) { + long := flat(dockerCheckCmd.Long) + for _, want := range []string{ + "It checks the project as stored", + "the exit status is the stored project's", + "stale widget definition — `mxcli fix widgets` fixes it", + "persists after update-widgets", + "--update-widgets", + "--no-update-widgets", + } { + if !strings.Contains(long, want) { + t.Errorf("docker check --help does not say %q", want) + } + } + if strings.Contains(long, "false CE0463") { + t.Error("docker check --help still calls the CE0463s update-widgets clears false") + } + for _, name := range []string{"update-widgets", "no-update-widgets"} { + if dockerCheckCmd.Flags().Lookup(name) == nil { + t.Errorf("docker check has no --%s flag", name) + } + } + // Both at once is a contradiction cobra refuses. + cmd := dockerCheckCmd + t.Cleanup(func() { + for _, name := range []string{"update-widgets", "no-update-widgets"} { + f := cmd.Flags().Lookup(name) + f.Value.Set("false") + f.Changed = false + } + }) + cmd.Flags().Set("update-widgets", "true") + cmd.Flags().Set("no-update-widgets", "true") + if err := cmd.ValidateFlagGroups(); err == nil { + t.Error("docker check accepts --update-widgets together with --no-update-widgets") + } +} diff --git a/cmd/mxcli/exec_source_files_1618_test.go b/cmd/mxcli/exec_source_files_1618_test.go new file mode 100644 index 0000000000..a9176acd2d --- /dev/null +++ b/cmd/mxcli/exec_source_files_1618_test.go @@ -0,0 +1,88 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/executor" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// ako/mxcli#1618 (mendixlabs/mxcli#1634): `exec` of `create java action … as +// $$ … $$` wrote javasource//actions/.java and named only the +// model element, so a commit of the model left the body behind. Through the +// real backend, on an MPR v1 and a v2 project: the file is named on its own +// line, project-relative, and the run ends with the summary. +func TestExecNamesTheSourceFilesItWritesBesideTheModel(t *testing.T) { + const script = "create module Probe1618;\n" + + "create or modify java action Probe1618.JA_Hello () returns String as $$\nreturn \"hi\";\n$$;\n" + + "create or modify javascript action Probe1618.JS_Hello () returns String as $$\nreturn \"hi\";\n$$;\n" + for _, fx := range atomicFixtures { + t.Run(fx.name, func(t *testing.T) { + dir := copyFixtureTree(t, fx.src) + out := runScriptCapturing(t, filepath.Join(dir, fx.mpr), script) + for _, want := range []string{ + "Created java action: Probe1618.JA_Hello\n" + + "Wrote javasource/probe1618/actions/JA_Hello.java (Java action body; commit it with the model)\n", + "Created javascript action: Probe1618.JS_Hello\n" + + "Wrote javascriptsource/probe1618/actions/JS_Hello.js (JavaScript action body; commit it with the model)\n", + } { + if !strings.Contains(out, want) { + t.Errorf("exec output lacks\n%s\ngot:\n%s", want, out) + } + } + if !strings.HasSuffix(out, "Files changed outside the model: 2 (javasource/, javascriptsource/). Stage them with the .mpr.\n") { + t.Errorf("exec output does not end with the summary:\n%s", out) + } + for _, p := range []string{"javasource/probe1618/actions/JA_Hello.java", "javascriptsource/probe1618/actions/JS_Hello.js"} { + if _, err := os.Stat(filepath.Join(dir, filepath.FromSlash(p))); err != nil { + t.Errorf("the reported file is not there: %v", err) + } + } + + // The control: the same script again rewrites nothing beside the + // model, and says nothing about it. + again := runScriptCapturing(t, filepath.Join(dir, fx.mpr), script) + if strings.Contains(again, "Wrote ") || strings.Contains(again, "outside the model") { + t.Errorf("a re-run with unchanged bodies reported a file:\n%s", again) + } + }) + } +} + +// `mxcli test` passes --no-source-file-report to the exec that injects its +// endpoint, whose Java source it deletes again at cleanup. +func TestExecHasAHiddenNoSourceFileReportFlag(t *testing.T) { + f := execCmd.Flags().Lookup("no-source-file-report") + if f == nil { + t.Fatal("exec has no --no-source-file-report flag; mxcli test's injection passes it") + } + if !f.Hidden { + t.Error("--no-source-file-report is for mxcli test's own injection runs and should be hidden") + } +} + +func runScriptCapturing(t *testing.T, mprPath, script string) string { + t.Helper() + var out bytes.Buffer + exec := executor.New(&out) + exec.SetBackendFactory(newBackendFactory()) + conn, _ := visitor.Build("connect local '" + visitor.QuoteString(mprPath) + "';") + if err := exec.ExecuteProgram(conn); err != nil { + t.Fatalf("connect: %v", err) + } + out.Reset() + prog, errs := visitor.Build(script) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + if err := runScriptAtomically(exec, prog); err != nil { + t.Fatalf("exec: %v", err) + } + return out.String() +} diff --git a/cmd/mxcli/testrunner/generator.go b/cmd/mxcli/testrunner/generator.go index 0e347a7f9b..5e2e8fa40f 100644 --- a/cmd/mxcli/testrunner/generator.go +++ b/cmd/mxcli/testrunner/generator.go @@ -7,7 +7,10 @@ import ( "regexp" "strings" + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/executor" "github.com/mendixlabs/mxcli/mdl/langver" + "github.com/mendixlabs/mxcli/mdl/visitor" ) // suiteVersion is the language version a generated script is written in: the @@ -158,7 +161,7 @@ func writeTestBlock(b *strings.Builder, tc TestCase, index int) { } // Collect variable names from the MDL body and rename them with suffix - varNames := extractVariableNames(tc.MDL) + varNames := extractVariableNames(tc.MDL, tc.Version) renamedMDL := renameVariables(tc.MDL, varNames, suffix) // Every assertion counts itself when evaluated; see assertionsEvaluatedVar. @@ -223,7 +226,7 @@ func writeThrowsTestBlock(b *strings.Builder, tc TestCase, suffix string) { b.WriteString(fmt.Sprintf(" DECLARE %s Boolean = false;\n", didThrowVar)) // Rename variables in the body - varNames := extractVariableNames(tc.MDL) + varNames := extractVariableNames(tc.MDL, tc.Version) renamedMDL := renameVariables(tc.MDL, varNames, suffix) lines := strings.Split(renamedMDL, "\n") @@ -273,26 +276,39 @@ func writeExpectAssertion(b *strings.Builder, testID string, exp Expect, evaluat // varPattern matches $VariableName in MDL ($ followed by word characters). var varPattern = regexp.MustCompile(`\$([A-Za-z_][A-Za-z0-9_]*)`) -// assignPattern matches "$var = call …" or "$var = create …" at the start of any -// statement line inside the test body. +// extractVariableNames returns the variables a test body defines, without the +// `$`: every name a statement in it creates — a call, create, retrieve, declare, +// list operation or aggregate, send rest request, execute database query, loop +// iterator, anything with an output variable — nested bodies included. // -// The keywords match in any case. MDL keywords are case-insensitive, and mdl 1 -// and `fmt` write them in lower case; matching only `CALL|CREATE` left -// `$r = call microflow …` unrenamed, so two tests that both assigned `$r` declared -// it twice in the one runner flow and injection failed with MDL063 (CE0111). -// The `\b` keeps a value that merely starts with those letters from matching. -var assignPattern = regexp.MustCompile(`(?mi)^\s*\$([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(?:call|create)\b`) - -// extractVariableNames finds all user-defined variable names in the MDL body. -// Returns the set of variable names (without $ prefix). -func extractVariableNames(mdl string) map[string]bool { +// The after-startup runner joins every test into one microflow, whose variable +// namespace is flat (an IF branch, a loop body or an error handler does not +// open a scope), so each of these is renamed per test. The set used to come +// from a pattern over the text that matched `$x = call|create` only: two tests +// that defined one name any other way — even `retrieve $x` — declared it twice, +// and injection failed with MDL063 (CE0111 "Duplicate variable name") +// (ako/mxcli#1775). It is now read from the parsed body through the definition +// MDL063 itself counts, so what the runner renames and what the validator +// refuses cannot drift apart. A `$x = …` that changes a variable (with or +// without `set`) defines nothing and is not in the set; its references are +// renamed with the definition they change. +// +// A body that does not parse defines nothing here, as attachOnError leaves it +// unwrapped: no renaming makes it parse, and the error the author then sees is +// about their own text. +func extractVariableNames(body string, v langver.Version) map[string]bool { names := make(map[string]bool) - - // Find assignment targets: $var = CALL MICROFLOW ... or $var = CREATE ... - for _, m := range assignPattern.FindAllStringSubmatch(mdl, -1) { - names[m[1]] = true + prog, errs := visitor.Build(bodyParsePrefix(v) + body + bodyParseSuffix) + if len(errs) > 0 || prog == nil { + return names + } + for _, stmt := range prog.Statements { + if mf, ok := stmt.(*ast.CreateMicroflowStmt); ok { + for _, name := range executor.FlowDefinedVariables(mf.Body) { + names[name] = true + } + } } - return names } diff --git a/cmd/mxcli/testrunner/generator_endpoint.go b/cmd/mxcli/testrunner/generator_endpoint.go index c9ff710e28..f7dc93bcb5 100644 --- a/cmd/mxcli/testrunner/generator_endpoint.go +++ b/cmd/mxcli/testrunner/generator_endpoint.go @@ -305,14 +305,8 @@ func quoteForMessage(s string) string { // parse, and the error the author then sees is about their own text. func attachOnError(lines []string, v langver.Version, handler []string) []string { body := strings.Join(lines, "\n") - - // Parsed inside the same wrapper the generators put it in, under the same - // language version, so it reads exactly as the generated script will. - var wrap strings.Builder - writeScriptHeader(&wrap, v) - wrap.WriteString(createFlow(v) + " " + mxTestModule + ".Body ()\nBEGIN\n") - prefix := wrap.String() - stmts, ok := visitor.FlowStatements(prefix + body + "\nEND;\n") + prefix := bodyParsePrefix(v) + stmts, ok := visitor.FlowStatements(prefix + body + bodyParseSuffix) if !ok { return lines } @@ -369,6 +363,19 @@ func attachOnError(lines []string, v langver.Version, handler []string) []string return strings.Split(string(runes), "\n") } +// bodyParsePrefix is the text a test body is parsed after, wherever the +// generators have to read it rather than copy it: inside the same kind of +// wrapper they put it in and under the same language version, so it reads +// exactly as the generated script will. bodyParseSuffix closes it. +func bodyParsePrefix(v langver.Version) string { + var wrap strings.Builder + writeScriptHeader(&wrap, v) + wrap.WriteString(createFlow(v) + " " + mxTestModule + ".Body ()\nBEGIN\n") + return wrap.String() +} + +const bodyParseSuffix = "\nEND;\n" + // escapeMDLComment keeps a test name from closing the javadoc block it sits in. func escapeMDLComment(s string) string { return strings.ReplaceAll(s, "*/", "* /") diff --git a/cmd/mxcli/testrunner/injection_args_test.go b/cmd/mxcli/testrunner/injection_args_test.go new file mode 100644 index 0000000000..462c45f97f --- /dev/null +++ b/cmd/mxcli/testrunner/injection_args_test.go @@ -0,0 +1,21 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "slices" + "testing" +) + +// The runner injects a Java action (the test endpoint) with `mxcli exec` and +// deletes its javasource/ folder again at cleanup. Since ako/mxcli#1618 `exec` +// names every file it writes beside the model and says to commit it — advice +// that is wrong for a file the runner is about to remove, so its injection runs +// turn the report off. +func TestInjectionExecTurnsOffTheSourceFileReport(t *testing.T) { + args := injectionExecArgs("/tmp/mxtest-endpoint.mdl", "/work/App.mpr") + want := []string{"exec", "/tmp/mxtest-endpoint.mdl", "-p", "/work/App.mpr", "--no-source-file-report"} + if !slices.Equal(args, want) { + t.Errorf("injection exec args = %q, want %q", args, want) + } +} diff --git a/cmd/mxcli/testrunner/on_error_coverage_test.go b/cmd/mxcli/testrunner/on_error_coverage_test.go index 669a898584..62f4dc0390 100644 --- a/cmd/mxcli/testrunner/on_error_coverage_test.go +++ b/cmd/mxcli/testrunner/on_error_coverage_test.go @@ -6,6 +6,7 @@ import ( "strings" "testing" + "github.com/mendixlabs/mxcli/mdl/langver" "github.com/mendixlabs/mxcli/mdl/visitor" ) @@ -81,9 +82,10 @@ func assertHandled(t *testing.T, runner, mdl, stmt string) { // legacyStmt is stmt as the legacy runner writes it for the suite's first test: // that runner compiles every test into one flow, so it renames the variable a -// call assigns with the test's suffix. These statements used to be matched +// call defines with the test's suffix. These statements used to be matched // unrenamed, which held only because a lower-case `call` was not renamed -// (TestLegacyRunnerRenamesLowerCaseAssignments). +// (TestLegacyRunnerRenamesLowerCaseAssignments) — nor, in any case, `send rest +// request` or `execute database query` (ako/mxcli#1775). func legacyStmt(stmt string) string { - return renameVariables(stmt, extractVariableNames(stmt), "_1") + return renameVariables(stmt, extractVariableNames(stmt, langver.V0), "_1") } diff --git a/cmd/mxcli/testrunner/rename_keyword_case_test.go b/cmd/mxcli/testrunner/rename_keyword_case_test.go index ef63847244..cd4bb3af1f 100644 --- a/cmd/mxcli/testrunner/rename_keyword_case_test.go +++ b/cmd/mxcli/testrunner/rename_keyword_case_test.go @@ -3,13 +3,41 @@ package testrunner import ( + "io/fs" + "os" + "path/filepath" "strings" "testing" "github.com/mendixlabs/mxcli/mdl/executor" + "github.com/mendixlabs/mxcli/mdl/langver" "github.com/mendixlabs/mxcli/mdl/visitor" ) +// collisionRules are the checks that report one variable name defined twice in +// a flow: MDL063 for any definition, MDL052 for two loop iterators. Both are +// CE0111 "Duplicate variable name" from mxbuild, and both are what the legacy +// runner's per-test renaming exists to prevent. +var collisionRules = map[string]bool{"MDL063": true, "MDL052": true} + +// legacyRunnerCollisions generates the after-startup runner for tests and +// returns every duplicate-variable violation ValidateProgram reports on it. +func legacyRunnerCollisions(t *testing.T, tests []TestCase) (string, []string) { + t.Helper() + src := GenerateTestRunner(&TestSuite{Name: "s", Tests: tests}) + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("generated runner does not parse: %v\n%s", errs[0], src) + } + var out []string + for _, v := range executor.ValidateProgram(prog, "") { + if collisionRules[v.RuleID] { + out = append(out, v.RuleID+": "+v.Message) + } + } + return src, out +} + // TestLegacyRunnerRenamesLowerCaseAssignments: the after-startup runner compiles // every test into one microflow, so a variable two tests both assign has to be // renamed per test. Which variables are renamed was decided by a pattern that @@ -18,57 +46,108 @@ import ( // not renamed. Two tests that both assign `$r` then declare it twice, and // injection fails with MDL063 (CE0111 "Duplicate variable name" from mxbuild). // Found alongside ako/mxcli#1753. +// +// The pattern never matched any other statement that defines a variable either +// — retrieve, declare, a list operation or aggregate, send rest request, execute +// database query, a loop iterator — in any case, so two tests defining the same +// name one of those ways still collided (ako/mxcli#1775). The names now come from +// the parsed body, so every definition MDL063 counts is renamed, nested ones +// included: a Mendix flow's variable namespace is flat, and the runner joins +// every test into one flow. func TestLegacyRunnerRenamesLowerCaseAssignments(t *testing.T) { + const items = "retrieve $items from Probe.Item;\n" cases := []struct { - name, header, body, renamed string + name, header, body, expect string + renamed []string }{ - {"call microflow, mdl 1", "mdl 1;\n", - "$r = call microflow Probe.SUB_Make();", "$r_%d = call microflow"}, - {"call microflow, mdl 0", "", - "$r = call microflow Probe.SUB_Make();", "$r_%d = call microflow"}, - {"call java action", "mdl 1;\n", - "$r = call java action Probe.JA_Upper(Text = 'a');", "$r_%d = call java action"}, - {"create object", "mdl 1;\n", - "$r = create Probe.Item (Name = 'a');", "$r_%d = create Probe.Item"}, - {"create list", "mdl 1;\n", - "$r = create list of Probe.Item;", "$r_%d = create list of"}, - {"mixed case", "mdl 1;\n", - "$r = Call Microflow Probe.SUB_Make();", "$r_%d = Call Microflow"}, + {name: "call microflow, mdl 1", header: "mdl 1;\n", + body: "$r = call microflow Probe.SUB_Make();", renamed: []string{"$r_%d = call microflow"}}, + {name: "call microflow, mdl 0", + body: "$r = call microflow Probe.SUB_Make();", renamed: []string{"$r_%d = call microflow"}}, + {name: "call java action", header: "mdl 1;\n", + body: "$r = call java action Probe.JA_Upper(Text = 'a');", renamed: []string{"$r_%d = call java action"}}, + {name: "create object", header: "mdl 1;\n", + body: "$r = create Probe.Item (Name = 'a');", renamed: []string{"$r_%d = create Probe.Item"}}, + {name: "create list", header: "mdl 1;\n", + body: "$r = create list of Probe.Item;", renamed: []string{"$r_%d = create list of"}}, + {name: "mixed case", header: "mdl 1;\n", + body: "$r = Call Microflow Probe.SUB_Make();", renamed: []string{"$r_%d = Call Microflow"}}, + + // ako/mxcli#1775: definitions the pattern never matched, in any case. + {name: "retrieve, mdl 1", header: "mdl 1;\n", + body: "retrieve $r from Probe.Item;", renamed: []string{"retrieve $r_%d from"}}, + {name: "retrieve, mdl 0", + body: "retrieve $r from Probe.Item;", renamed: []string{"retrieve $r_%d from"}}, + {name: "retrieve first", header: "mdl 1;\n", + body: "retrieve $r from Probe.Item first;", renamed: []string{"retrieve $r_%d from"}}, + {name: "RETRIEVE upper case", + body: "RETRIEVE $r FROM Probe.Item;", renamed: []string{"RETRIEVE $r_%d FROM"}}, + {name: "declare", header: "mdl 1;\n", + body: "declare $r String = 'a';", renamed: []string{"declare $r_%d String"}}, + {name: "declare, mdl 0", + body: "DECLARE $r String = 'a';", renamed: []string{"DECLARE $r_%d String"}}, + {name: "list operation head", header: "mdl 1;\n", + body: items + "$r = head($items);", + renamed: []string{"retrieve $items_%d from", "$r_%d = head($items_%d)"}}, + {name: "list operation, statement form", header: "mdl 1;\n", + body: items + "$r = head $items;", + renamed: []string{"$r_%d = head $items_%d;"}}, + {name: "list operation filter", header: "mdl 1;\n", expect: "count($r) >= 0", + body: items + "$r = filter($items, Name = 'a');", + renamed: []string{"$r_%d = filter($items_%d"}}, + {name: "aggregate", header: "mdl 1;\n", expect: "$r >= 0", + body: items + "$r = count($items);", + renamed: []string{"$r_%d = count($items_%d)"}}, + {name: "send rest request", header: "mdl 1;\n", + body: "$r = send rest request Probe.Api.GetThing;", renamed: []string{"$r_%d = send rest request"}}, + {name: "execute database query", header: "mdl 1;\n", expect: "count($r) >= 0", + body: "$r = execute database query Probe.Db.ListRows;", renamed: []string{"$r_%d = execute database query"}}, + + // Nested definitions share the flow's one namespace: an IF branch, a + // loop body and an error handler do not open a scope (MDL063), and a + // loop iterator is flow-wide too (MDL052). + {name: "declare in an if branch", header: "mdl 1;\n", + body: "declare $r String = 'a';\nif $r = 'a' then\n declare $inner String = 'b';\n set $r = $inner;\nend if;", + renamed: []string{"declare $inner_%d String", "set $r_%d = $inner_%d"}}, + {name: "loop iterator and a definition in its body", header: "mdl 1;\n", expect: "count($items) >= 0", + body: items + "loop $it in $items begin\n retrieve $r from Probe.Item first;\nend loop;", + renamed: []string{"loop $it_%d in $items_%d", "retrieve $r_%d from"}}, + {name: "declare in a custom error handler", header: "mdl 1;\n", expect: "1 = 1", + body: "call microflow Probe.SUB_Make() on error begin\n declare $msg String = 'failed';\n log error node 'T' $msg;\nend error;", + renamed: []string{"declare $msg_%d String", "node 'T' $msg_%d"}}, + // Control: the spelling the pattern always matched. - {"upper case (control)", "", - "$r = CALL MICROFLOW Probe.SUB_Make();", "$r_%d = CALL MICROFLOW"}, + {name: "upper case (control)", + body: "$r = CALL MICROFLOW Probe.SUB_Make();", renamed: []string{"$r_%d = CALL MICROFLOW"}}, } for _, c := range cases { t.Run(c.name, func(t *testing.T) { + expect := c.expect + if expect == "" { + expect = "$r != empty" + } block := func(n string) string { - return "/**\n * @test " + n + "\n * @expect $r != empty\n */\n" + c.body + "\n/\n" + return "/**\n * @test " + n + "\n * @expect " + expect + "\n */\n" + c.body + "\n/\n" } tests, err := parseMDLTests(c.header+block("one")+"\n"+block("two"), "rename.test.mdl") if err != nil { t.Fatalf("parse: %v", err) } - src := GenerateTestRunner(&TestSuite{Name: "s", Tests: tests}) + src, collisions := legacyRunnerCollisions(t, tests) - for i, want := range []string{ - strings.Replace(c.renamed, "%d", "1", 1), - strings.Replace(c.renamed, "%d", "2", 1), - } { - if !strings.Contains(src, want) { - t.Errorf("test %d: no %q — the assignment was not renamed", i+1, want) + for _, pattern := range c.renamed { + for i, n := range []string{"1", "2"} { + want := strings.ReplaceAll(pattern, "%d", n) + if !strings.Contains(src, want) { + t.Errorf("test %d: no %q — the definition was not renamed", i+1, want) + } } } - if strings.Contains(src, "($r != empty)") { + if strings.Contains(expect, "$r") && strings.Contains(src, "($r ") { t.Errorf("an assertion still reads the unrenamed $r") } - - prog, errs := visitor.Build(src) - if len(errs) > 0 { - t.Fatalf("generated runner does not parse: %v\n%s", errs[0], src) - } - for _, v := range executor.ValidateProgram(prog, "") { - if v.RuleID == "MDL063" { - t.Errorf("%s: %s", v.RuleID, v.Message) - } + for _, v := range collisions { + t.Error(v) } if t.Failed() { t.Logf("generated runner:\n%s", src) @@ -77,18 +156,95 @@ func TestLegacyRunnerRenamesLowerCaseAssignments(t *testing.T) { } } -// TestAssignPatternNeedsAWholeKeyword: matching case-insensitively must not turn -// an assignment whose value merely starts with the letters of a keyword into one. +// TestLegacyRunnerRenamesTheThrowsPath: an @throws test goes through its own +// block writer, which renames the body too. Two of them defining one name must +// not collide either. +func TestLegacyRunnerRenamesTheThrowsPath(t *testing.T) { + block := func(n string) string { + return "/**\n * @test " + n + "\n * @throws\n */\nretrieve $r from Probe.Item first;\n" + + "call microflow Probe.SUB_Fail(Item = $r);\n/\n" + } + tests, err := parseMDLTests("mdl 1;\n"+block("one")+"\n"+block("two"), "throws.test.mdl") + if err != nil { + t.Fatalf("parse: %v", err) + } + src, collisions := legacyRunnerCollisions(t, tests) + for _, n := range []string{"1", "2"} { + for _, want := range []string{"retrieve $r_" + n + " from", "(Item = $r_" + n + ")"} { + if !strings.Contains(src, want) { + t.Errorf("no %q", want) + } + } + } + for _, v := range collisions { + t.Error(v) + } + if t.Failed() { + t.Logf("generated runner:\n%s", src) + } +} + +// TestAssignPatternNeedsAWholeKeyword: an assignment whose value merely starts +// with the letters of a keyword defines nothing — it changes a variable — so it +// must not be renamed as if it were a call or a create. Nor does `set`, the +// spelled-out form of the same Change Variable. func TestAssignPatternNeedsAWholeKeyword(t *testing.T) { for _, body := range []string{ "$r = callback;", "$r = createdDate;", + "set $r = 1;", } { - if names := extractVariableNames(body); len(names) != 0 { + if names := extractVariableNames(body, langver.V0); len(names) != 0 { t.Errorf("%q: renamed %v, want nothing", body, names) } } - if names := extractVariableNames("$r = call microflow M.F();"); !names["r"] { + if names := extractVariableNames("$r = call microflow M.F();", langver.V0); !names["r"] { t.Errorf("control: $r = call microflow was not picked up: %v", names) } } + +// TestLegacyRunnerRepositoryTestFilesDefineEachVariableOnce runs every test +// file in the repository through the after-startup runner's generator and the +// validator `exec` applies at injection. A test file the endpoint runner runs +// must not fail on this one because two of its tests define a name the same +// way. The sweep is what found ako/mxcli#1775: before it, 2 of these files drew +// MDL063 (`retrieve $Brands` in three tests; `retrieve $Request` and +// `$Request = head(…)`). +// +// A `.fail.` file is a negative fixture whose point is another refusal, and is +// left out. +func TestLegacyRunnerRepositoryTestFilesDefineEachVariableOnce(t *testing.T) { + var files []string + for _, root := range []string{"../../../mdl-examples", "../../../docs-site"} { + _ = filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error { + if err == nil && !d.IsDir() && isTestFile(d.Name()) && !strings.Contains(d.Name(), ".fail.") { + files = append(files, p) + } + return nil + }) + } + if len(files) == 0 { + t.Fatal("no test files found") + } + for _, p := range files { + t.Run(filepath.Base(p), func(t *testing.T) { + if _, err := os.Stat(p); err != nil { + t.Fatal(err) + } + suite, err := ParseTestFile(p) + if err != nil { + t.Fatalf("ParseTestFile: %v", err) + } + if len(suite.Tests) < 2 { + t.Skip("fewer than two tests: nothing can collide") + } + src, collisions := legacyRunnerCollisions(t, suite.Tests) + for _, v := range collisions { + t.Error(v) + } + if t.Failed() { + t.Logf("generated runner:\n%s", src) + } + }) + } +} diff --git a/cmd/mxcli/testrunner/runner.go b/cmd/mxcli/testrunner/runner.go index bd30506970..f6349f0022 100644 --- a/cmd/mxcli/testrunner/runner.go +++ b/cmd/mxcli/testrunner/runner.go @@ -804,7 +804,16 @@ func execMDLScript(projectPath, mdl, namePattern string) error { } f.Close() - return execMxcli(projectPath, "exec", path, "-p", projectPath) + return execMxcli(projectPath, injectionExecArgs(path, projectPath)...) +} + +// injectionExecArgs are the arguments of the `mxcli exec` that injects a +// generated script. Everything it writes is removed again at cleanup — the +// endpoint's Java source included (removeGeneratedJavaSource) — so exec must +// not tell the user to commit the files it writes beside the model +// (ako/mxcli#1618). +func injectionExecArgs(scriptPath, projectPath string) []string { + return []string{"exec", scriptPath, "-p", projectPath, "--no-source-file-report"} } // cleanupCommands returns the MDL statements that put the project back the way it diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 83fc73373d..a6992e720c 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -125,6 +125,24 @@ body on Mendix 10 too, and Studio Pro could not open the project. | Deciding by a microflow | `DECIDE BY MICROFLOW ...` | No | 10.3+ | | Group targeting | `TARGETING GROUPS XPATH ...`, `TARGETING GROUPS MICROFLOW ...` | No | 11.2+ | | Parallel splits | `PARALLEL SPLIT` | Yes | Yes | +| Waiting for a notification | `WAIT FOR NOTIFICATION` | No | Yes | +| Waiting for a timer | `WAIT FOR TIMER` | No | 10.6+ | +| A timer boundary event of no kind | `BOUNDARY EVENT TIMER ...` | No | 10.14+ (MDL-WF07 refuses it from 11.0) | +| Interrupting and non-interrupting timer boundary events | `BOUNDARY EVENT [NON] INTERRUPTING TIMER ...` | No | 10.20+ | +| Locking and unlocking a workflow | `LOCK WORKFLOW ...`, `UNLOCK WORKFLOW ...` | No | Yes | + +Every write refuses an element whose type the project's Mendix version does not +have yet, naming the type, the version that introduced it and the project's +version ("this Mendix 9.24.30 project cannot store +Workflows$WaitForNotificationActivity (Mendix 10.0.0+)"). The versions are each +type's own, from the Mendix Model SDK. Below their versions, the wait +activities, the timer boundary events and a lock or unlock naming a workflow are +also refused up front, by `check -p` and by `exec` before anything is written, +with MDL-VERSION01. Before this, such a type was written, and the project did not load +(`TypeCacheUnknownTypeException`, [ako/mxcli#1774](https://github.com/ako/mxcli/issues/1774)). +Where an older version stores the same thing in another shape (the parallel +split's end marker, user tasks), that shape is written instead, as listed under +"Properties Stored in an Older Shape". Below Mendix 11.3 a workflow's main flow has no stored start or end activity: 11.1 and 11.2 keep them implicit, and 10.x and 11.0 do not have the start diff --git a/docs-site/src/guides/marketplace.md b/docs-site/src/guides/marketplace.md index bf2703e68e..8a528abe29 100644 --- a/docs-site/src/guides/marketplace.md +++ b/docs-site/src/guides/marketplace.md @@ -402,7 +402,7 @@ Re-running is free: a second run reports 0 units changed, because [idempotent wr |---|---|---| | `mxcli fix widgets` | **yes** | the fix — after any headless install | | `mxcli fix design-properties` | **yes** | the fix — after any headless install | -| `mxcli docker check` | no | runs the widget resync on a temporary copy so the *check* is not tripped by CE0463; the stored model stays stale (`--no-update-widgets` checks it as stored) | +| `mxcli docker check` | no | checks the project as stored, so it reports the CE0463 a headless install leaves, and labels each one the widget resync clears as a stale widget definition (the resync runs on a temporary copy only) | | `mxcli widget sync` | yes, partial | reconciles widget schemas in mxcli's own code; clears 7 of 40 on the reference fixture | CE6087 is distinct from `CE6083`, which is a *missing* design-property declaration and is fixed by installing everything the package ships — something `install` and `update` already do. diff --git a/docs-site/src/guides/pluggable-widgets.md b/docs-site/src/guides/pluggable-widgets.md index 6d0812a4db..4374598ddb 100644 --- a/docs-site/src/guides/pluggable-widgets.md +++ b/docs-site/src/guides/pluggable-widgets.md @@ -103,9 +103,11 @@ This changed during the beta. Earlier builds rebuilt every pluggable widget of a rewritten page from the installed `.mpk`, so re-running `describe` output also upgraded it — and usually cleared its CE0463 — while silently resetting whatever the statement could not say (a specialized column binding, a row click). If a script -relied on that side effect to clear CE0463, run `mxcli fix widgets` instead; -`mxcli docker check` passing is not enough, because it updates the widgets of a -temporary copy only (`--no-update-widgets` checks the project as stored). +relied on that side effect to clear CE0463, run `mxcli fix widgets` instead. +`mxcli docker check` checks the project as stored and labels each CE0463 a widget +resync would clear as a *stale widget definition*, so it shows which ones +`mxcli fix widgets` fixes (`--update-widgets` checks a resynced copy only, and +hides them). **Two widgets do not yet author cleanly against Data Widgets 3.10+.** Freshly created `gallery` and `dropdownfilter` widgets produce CE0463 on those packages; `datagrid2`, diff --git a/docs-site/src/internals/idempotent-writes.md b/docs-site/src/internals/idempotent-writes.md index f9aef48c3e..da8ba70ada 100644 --- a/docs-site/src/internals/idempotent-writes.md +++ b/docs-site/src/internals/idempotent-writes.md @@ -108,6 +108,16 @@ lacks, from `modelsdk/canon/studiopro_property_floors.json`. Writers that derive a value for a newer key leave it out on older versions themselves, so the refusal only fires for something the statement asked for. +An element whose `$Type` the project's version does not have yet is refused the +same way, wherever it sits in the unit (`canon.UndeclaredTypeError`, +ako/mxcli#1774). Each type's introduction version is in +`modelsdk/canon/mendixmodelsdk_type_versions.json`, extracted from the Mendix +Model SDK by `modelsdk/canon/gen_type_versions.go`. The check runs after the +property shapes are lowered for the version, so a type with an older shape is +written in that shape instead of being refused. The SDK's deletion versions are +in the file too, but they are not enforced: a type the SDK renamed or moved +keeps its storage name. + The doctype integration gate measures this on every script. A new gap fails it. To record a new measurement into the table, run the gate with `MXCLI_PROPERTY_SETS_RECORD=` and then run diff --git a/docs-site/src/reference/microflow/create-java-action.md b/docs-site/src/reference/microflow/create-java-action.md index 214d7f01bf..3f88449e56 100644 --- a/docs-site/src/reference/microflow/create-java-action.md +++ b/docs-site/src/reference/microflow/create-java-action.md @@ -44,7 +44,16 @@ clears a description. A new action gets no descriptions and the return value nam ### The Java Source File The statement writes `javasource//actions/.java` the way mxbuild -generates it, byte for byte, so the next build leaves it unchanged. The action's +generates it, byte for byte, so the next build leaves it unchanged. + +The body is only in this file, not in the model, so commit `javasource/` +together with the `.mpr` and `mprcontents/`. A commit of the model alone gives a +clone an action with no logic. `mxcli exec` names each file it writes there on +its own line, for example +`Wrote javasource/sales/actions/JA_Ping.java (Java action body; commit it with the model)`, +and ends the run with `Files changed outside the model: N (…)`. A rewrite that +leaves the file's content as it was is not reported. `RENAME JAVA ACTION` moves +the file and `DROP JAVA ACTION` removes it; both are reported the same way. The action's documentation -- the `/** ... */` before `CREATE` -- is the class javadoc above `public class`; a rewrite without one keeps the stored documentation, in the model and in the file. (Before ako/mxcli#1700 the file had no javadoc, so every rewrite diff --git a/docs-site/src/reference/microflow/create-javascript-action.md b/docs-site/src/reference/microflow/create-javascript-action.md index 49f8b1bdad..e8c0779747 100644 --- a/docs-site/src/reference/microflow/create-javascript-action.md +++ b/docs-site/src/reference/microflow/create-javascript-action.md @@ -23,6 +23,14 @@ Two things are written: the model unit the JSDoc header, the exported `async function`, and the supplied body placed between the `BEGIN USER CODE` / `END USER CODE` markers. +The body is only in the `.js` file, not in the model, so commit +`javascriptsource/` together with the `.mpr` and `mprcontents/`. A commit of +the model alone gives a clone an action with no logic. `mxcli exec` names each +file it writes there on its own line, for example +`Wrote javascriptsource/sales/actions/JS_Ping.js (JavaScript action body; commit it with the model)`, +and ends the run with `Files changed outside the model: N (…)`. A rewrite that +leaves the file's content as it was is not reported. + > **The `AS $$ ... $$` body is mandatory.** Like Java actions, every `CREATE` > needs a code body; omitting it reports `no viable alternative at input '...'`. > Use a stub such as `AS $$ return Promise.resolve(false); $$` when the diff --git a/docs-site/src/reference/organization/move.md b/docs-site/src/reference/organization/move.md index e3afdc77dd..cddd34c865 100644 --- a/docs-site/src/reference/organization/move.md +++ b/docs-site/src/reference/organization/move.md @@ -12,6 +12,15 @@ Moves a document to a different folder or module. Every top-level document type Entity moves only support moving to a module (not to a folder), because entities are embedded in domain model documents. +A Java or JavaScript action keeps its source in a directory named after its module +(`javasource//actions/`, `javascriptsource//actions/`). A +cross-module move takes the source file along, and rewrites a Java source's +`package` line the way mxbuild would, so the action keeps its code. A source left +behind builds without error, but the app then calls a generated stub that throws +"not implemented". If the target module's directory already holds a source file +of that name, the move is refused before anything changes. A move within the +module, into a folder, leaves the source where it is. + A `FOLDER` clause on `CREATE OR MODIFY` moves an existing document as well, so a script that already creates a document does not need a separate `MOVE` to place it. diff --git a/docs-site/src/tools/docker-build.md b/docs-site/src/tools/docker-build.md index d7d2244785..3c472cbd94 100644 --- a/docs-site/src/tools/docker-build.md +++ b/docs-site/src/tools/docker-build.md @@ -26,7 +26,9 @@ project into MPRv1), mx check compiles the theme into `theme-cache/` and So `docker build` copies the project to a temporary directory (in `$TMPDIR`) and runs all three there. Widget definitions are normalised on that copy, so the build uses the normalised model while the project keeps its own; run -`mxcli fix widgets` to apply the normalisation to the project. The only thing +`mxcli fix widgets` to apply the normalisation to the project. (So the build's +pre-build check does not report a stale widget definition; `mxcli docker check`, +which checks the project as stored, does.) The only thing written is the output directory (`.docker/build/` by default, or `-o`). Build output, caches and VCS folders (`deployment/`, `.docker/`, `.git/`, `node_modules/`, ...) are not copied. diff --git a/docs-site/src/tools/docker-check.md b/docs-site/src/tools/docker-check.md index 66cae4f903..63882a9154 100644 --- a/docs-site/src/tools/docker-check.md +++ b/docs-site/src/tools/docker-check.md @@ -11,10 +11,11 @@ mxcli docker check -p app.mpr ## What It Does 1. **Resolves the `mx` binary** -- Takes the `mx` next to `--mxbuild-path`, else the first it finds on PATH, in a Studio Pro installation, or in the download cache at `~/.mxcli/mxbuild//modeler/mx`. It does not download one (see [Getting mx](#getting-mx)) -2. **Runs `mx check`** -- Executes the Mendix project checker against a temporary copy of the `.mpr` file, as a local process; despite the `docker` in its name, it does not use Docker +2. **Runs `mx check`** -- Executes the Mendix project checker against a temporary copy of the project **as stored**, as a local process; despite the `docker` in its name, it does not use Docker 3. **Reports results** -- Outputs errors, warnings, and deprecations to the console +4. **Explains CE0463** -- Only when the check found a CE0463, re-checks the copy after `mx update-widgets` and labels each one (see [CE0463 and stale widget definitions](#ce0463-and-stale-widget-definitions)) -If the project has errors, the command exits with a non-zero status code. +If the stored project has errors, the command exits with a non-zero status code. Before `mx check` it also warns, without failing, about two problems `mx check` cannot see: an MPR v2 `ContentsHash` index that disagrees with the `.mxunit` @@ -45,26 +46,83 @@ another patch of the same minor; `mx update-widgets` does not and the check then runs without it (see `mxcli docker check --help`). Pass `--mxbuild-path ~/.mxcli/mxbuild/` to choose one. +## CE0463 and stale widget definitions + +CE0463 ("The definition of this widget has changed") means a stored pluggable +widget instance does not match its widget package. Its commonest cause is +harmless to fix: the package was upgraded after the widget was placed (any +Marketplace or widget install does this), and Studio Pro's "Update all widgets" +resyncs it. But until it is resynced, MxBuild and `mxcli run --local` refuse +the project, so `docker check` reports it. + +When the check of the stored project finds a CE0463, `docker check` runs +`mx update-widgets` on the same temporary copy, checks it again, and labels each +CE0463 of the stored project: + +| Label | Meaning | Fix | +|---|---|---| +| `` stale widget definition — `mxcli fix widgets` fixes it `` | the resync clears it | `mxcli fix widgets -p app.mpr` updates the project's widgets and keeps MPR v2 | +| `persists after update-widgets` | the resync does not clear it: something in the stored widget is wrong in a way a resync does not repair (if mxcli wrote the widget, that is an mxcli defect worth reporting) | not a resync; see [Pluggable Widgets](../guides/pluggable-widgets.md) | + +The verdict and the exit status are always the stored project's. The re-check +costs a second `mx check`, so it runs only when there is a CE0463 to explain. + +`mx check` does not print the page a CE0463 is on, so widgets of the same name on +different pages print the same line; the labels count them (`(4×)`, or +`(2 of 3)` when a resync clears only some of them). + +| Flag | Checks | Exit status | Labels | +|---|---|---|---| +| *(none)* | the project as stored | the stored project's | yes, when there is a CE0463 | +| `--no-update-widgets` | the project as stored | the stored project's | no: update-widgets never runs | +| `--update-widgets` | only a copy normalised by `mx update-widgets` first | the normalised copy's | no | + +`--update-widgets` is how `docker check` behaved by default in earlier +releases. It reports 0 errors for a project whose only errors are stale widget +definitions, which then still fails to build: use it only when you mean to ask +"would this pass after `mxcli fix widgets`?". `--no-update-widgets` is kept for +scripts that pass it; its verdict is the default's. + +If `mx update-widgets` fails during the re-check (it refuses a project whose +Mendix version is not exactly its own, `exit status 4`), the CE0463s are left +unlabelled and the output says so. + ## Example Output A clean project: ``` -Using mx: /home/vscode/.mxcli/mxbuild/11.6.3.12345/modeler/mx +Using mx: /home/vscode/.mxcli/mxbuild/11.13.0/modeler/mx +Checking a temporary copy (mx writes into the project it checks; the project on disk is not changed). Checking project /workspaces/app/app.mpr... +Checking app for errors... The app contains: 0 errors. Project check passed. ``` -A project with errors: +A project whose Data grid 2 package was upgraded after its grids were placed +(measured on Mendix 11.13.0; the long CE0463 message is shortened here): ``` -Using mx: /home/vscode/.mxcli/mxbuild/11.6.3.12345/modeler/mx Checking project /workspaces/app/app.mpr... -CE0001: Entity 'MyModule.Customer' has no access rules configured. -CE0463: Widget definition changed for 'DataGrid2'. -The app contains: 2 errors. -project check failed: exit status 1 +Checking app for errors... +[error] [CE0463] "The definition of this widget has changed. ..." at Data grid 2 'dataGrid2_1' +[error] [CE0463] "The definition of this widget has changed. ..." at Data grid 2 'dataGrid21' +... +The app contains: 11 errors. + +The stored project has 11 CE0463 errors. Re-checking the temporary copy after + `mx update-widgets` to tell stale widget definitions from other causes... +CE0463 in the stored project: + at Data grid 2 'dataGrid2_1' (4×): stale widget definition — `mxcli fix widgets` fixes it + at Data grid 2 'dataGrid21' (4×): stale widget definition — `mxcli fix widgets` fixes it + at Data grid 2 'dataGrid2_2' (2×): stale widget definition — `mxcli fix widgets` fixes it + at Data grid 2 'dataGrid2_3': stale widget definition — `mxcli fix widgets` fixes it +All 11 are stale widget definitions. This updates them in the project, as Studio + Pro's "Update all widgets" does: + mxcli fix widgets -p /workspaces/app/app.mpr +As stored: 11 errors. After update-widgets: 0 errors. +Error: project check failed: exit status 1 ``` ## When to Use @@ -72,7 +130,7 @@ project check failed: exit status 1 `mxcli docker check` is faster than a full build (`mxcli docker build`) because it only validates the project structure without compiling or packaging. This makes it useful for: - **Quick validation** after making MDL changes -- **Pre-build gate** -- `mxcli docker build` runs `mx check` automatically before building (use `--skip-check` to bypass) +- **Pre-build gate** -- `mxcli docker build` runs `mx check` automatically before building (use `--skip-check` to bypass). That check runs on the build's widget-normalised copy, as `docker check --update-widgets` does, so it does not report stale widget definitions; `docker check` does - **CI pipelines** -- fail fast on invalid projects before spending time on a full build - **Iterative development** -- check after each MDL script execution to catch errors early diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index 8219bcf627..e63779ea77 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -27,13 +27,20 @@ mxcli fix hashes -p app.mpr --repair # rewrite mismatched hashes from the file The repair treats the files as the source of truth and never changes them. It updates every hash in one SQLite transaction, and, like every mxcli write, is -refused while Studio Pro has the project open (the `.mpr.lock` beside it). An -MPR v1 project keeps unit contents inside the `.mpr`, has no index to drift, -and the command says so. +refused while Studio Pro has the project open (the `.mpr.lock` beside it). + +An MPR v1 project (below Mendix 10.18) keeps each unit's contents inside the +`.mpr`, in the same row as its `ContentsHash`, and Studio Pro fills the hash +there too, with the same formula over the stored contents. No file can drift +from it, but mxcli before this release replaced a unit's contents and left the +old hash in place, or stored none for a new unit. On v1 the command compares +each hash with the contents beside it, reports `MISMATCH` only, and `--repair` +rewrites the hash from the contents. `mxcli exec` and `mxcli docker check` run the verify too and print a one-line warning when it finds a mismatch or a missing file. It never fails them; -measured on a 900-unit project, the whole verify takes about 0.2 s. +measured on a 900-unit project, the whole verify takes about 0.2 s. They skip +MPR v1, where the verify reads every unit's contents. ## Property names Studio Pro's merge compares: `mxcli fix property-sets` @@ -73,6 +80,19 @@ there still deletes its row and the user assignments that are not demo users. A key its version does not declare is reported and left alone, and, like every mxcli write, the command is refused while Studio Pro has the project open. +The values the command adds were measured with Mendix's own tools on specific +Mendix versions. A key that no measurement covers on your version is not +filled, but the command still finds it. Studio Pro writes the same property +names on every element of a type, so the command also lists each name that +some elements of a type carry and others lack: + +``` + MIXED Microflows$Nanoflow.ExportLevel: on 14 of 15 element(s); without it: 71e6faf6-… [Microflows$Nanoflow ACT_Login] +``` + +It leaves those names as they are and exits 1. Report them with your Mendix +version so the value can be measured. + ### The repair reaches forward, not back A merge compares each document both branches changed with its revision in the diff --git a/docs-site/src/tutorial/validation.md b/docs-site/src/tutorial/validation.md index 151ab5ce92..3fe1551298 100644 --- a/docs-site/src/tutorial/validation.md +++ b/docs-site/src/tutorial/validation.md @@ -98,7 +98,7 @@ This runs Mendix's own `mx check` tool on a temporary copy of the project. It pe - BSON serialization errors (malformed internal data) - Security configuration problems (missing access rules, CE0066 errors) -- Widget definition mismatches (CE0463 errors) +- Widget definition mismatches (CE0463 errors), each labelled either a *stale widget definition*, which `mxcli fix widgets` clears, or one that *persists after update-widgets* - Missing required properties on widgets - Broken cross-references between documents diff --git a/docs-wiki/bug-patterns/widget-type-object-drift.md b/docs-wiki/bug-patterns/widget-type-object-drift.md index d0f31a2541..903b886da6 100644 --- a/docs-wiki/bug-patterns/widget-type-object-drift.md +++ b/docs-wiki/bug-patterns/widget-type-object-drift.md @@ -51,13 +51,16 @@ tell them apart: whether Studio Pro's *own* template widgets fail alongside your and whether `mx update-widgets` clears it. Skipping that step produces a confident wrong answer, and has. -**The default measurement hides the class.** `mxcli docker check` runs -`mx update-widgets` first, on purpose, to suppress exactly the package-drift noise -above — so it reports 0 errors on a project that genuinely has a CE0463, *and* -repairs the project on the way past, leaving it clean for any later check too. -Both of this cycle's real CE0463s were invisible until someone passed -`--no-update-widgets`. (`mxcli check` is not an alternative: it validates an MDL -script and never invokes mxbuild, so it cannot produce CE0463 at all.) +**The default measurement hid the class.** Until ako/mxcli#646, `mxcli docker +check` ran `mx update-widgets` on its copy first, to suppress exactly the +package-drift noise above — so it reported 0 errors on a project that genuinely had +a CE0463, which then failed `run --local`. A gate that normalises its input before +judging it answers a different question from the one asked. It now checks the +stored project and runs the second control itself, labelling each CE0463 *stale* +(the resync clears it) or *persists*; `--update-widgets` is the old behaviour, so +"docker check passed" in an older finding meant the resynced copy passed. +(`mxcli check` is not an alternative: it validates an MDL script and never invokes +mxbuild, so it cannot produce CE0463 at all.) **The cause is usually a value, not the schema** — four of five fixes in one release cycle were value-shaped while the error named the version: an empty diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index a50d69d8a8..d6442be9ad 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -420,7 +420,7 @@ CREATE CONSUMED ODATA SERVICE MyModule.LocalService2 ( **Note:** `MetadataUrl` supports three formats: - `https://...` or `http://...` — fetches from HTTP(S) endpoint - `file:///abs/path` — reads from local absolute path -- `./path` or `path/file.xml` — reads from local relative path, **normalized to absolute `file://` in the model** for Studio Pro compatibility +- `./path` or `path/file.xml` — reads from local relative path, **normalized to absolute `file://` in the model** for Studio Pro compatibility. A re-run from a checkout at another path keeps the stored URL when it names the same project file, and `describe` prints a stored URL inside the project as `./path` **Important:** `ServiceUrl` must always be a constant reference starting with `@` (e.g., `@Module.ConstantName`). Create a constant first: ```sql diff --git a/mdl-examples/bug-tests/1159-mpr-v1-contents-hash.mdl b/mdl-examples/bug-tests/1159-mpr-v1-contents-hash.mdl new file mode 100644 index 0000000000..5d5697e28d --- /dev/null +++ b/mdl-examples/bug-tests/1159-mpr-v1-contents-hash.mdl @@ -0,0 +1,41 @@ +mdl 1; +-- ============================================================================ +-- ako/mxcli#1159: on an MPR v1 project (unit contents inside the .mpr, Mendix +-- below 10.18) a unit write replaced Unit.Contents and left Unit.ContentsHash +-- describing the old bytes; an inserted unit got ContentsHash ''. mx check and +-- MxBuild never read the column, so the project checks and builds clean, but +-- Studio Pro uses it to detect what changed. +-- +-- Studio Pro fills ContentsHash = base64(SHA-256(Contents)) (standard base64, +-- padded) on every v1 unit: 308 of 308 in a fresh 9.24.30 app. +-- +-- Repro (needs a v1 project; any 9.x or 10.x below 10.18): +-- +-- mxcli new App924 --version 9.24.30.51827 --theme none --layout none \ +-- --skip-init --skip-build +-- mxcli exec mdl-examples/bug-tests/1159-mpr-v1-contents-hash.mdl -p App924/App924.mpr +-- +-- then compare every row (the reporter's check): +-- +-- import sqlite3, hashlib, base64 +-- c = sqlite3.connect('App924/App924.mpr') +-- for uid, h, b in c.execute('select UnitID, ContentsHash, Contents from Unit'): +-- if h != base64.b64encode(hashlib.sha256(bytes(b)).digest()).decode(): +-- print('stale:', bytes(uid).hex()) +-- +-- Before the fix it printed two rows: the DomainModel the entity rewrote +-- (old hash) and the new microflow (empty hash). After it, none. A project an +-- older mxcli left stale is repaired with `mxcli fix hashes -p App.mpr --repair`, +-- which now compares v1 hashes against the Contents column. +-- ============================================================================ + +-- rewrites an existing unit: DomainModels$DomainModel +create persistent entity MyFirstModule.Customer1159 ( + "Name": String(100) +); + +-- inserts a new unit +create microflow MyFirstModule.ACT_1159 () +begin + log info node 'T1159' 'hello'; +end; diff --git a/mdl-examples/bug-tests/1499-odata-metadata-url-checkout-path.mdl b/mdl-examples/bug-tests/1499-odata-metadata-url-checkout-path.mdl new file mode 100644 index 0000000000..6384ff7225 --- /dev/null +++ b/mdl-examples/bug-tests/1499-odata-metadata-url-checkout-path.mdl @@ -0,0 +1,47 @@ +mdl 1; +-- ============================================================================ +-- ako/mxcli#1499: a relative MetadataUrl is stored as the absolute file:// URL +-- of the checkout that ran the script. Re-executing the same unchanged script +-- from a checkout at another path (another machine, a moved folder, a git +-- worktree) rewrote every OData client to its own path: +-- +-- - MetadataUrl: 'file:///workspaces/mxcli-formula1/Formula1Frontend/contracts/f1-metadata.xml', +-- + MetadataUrl: 'file:///home/user/mxcli-formula1/Formula1Frontend/contracts/f1-metadata.xml', +-- +-- 5 of the 20 writes of a full re-exec in the reported project. Nothing failed: +-- mx check accepts the value whatever it is (a relative one, a dead absolute +-- one; measured on 11.13.0), so the churn was only visible in the diff. +-- +-- The stored form stays the absolute file:// URL. A create or modify whose +-- relative path names the same project file now leaves the stored URL alone, +-- and the contract is read from this checkout's file. describe prints a stored +-- URL inside the project directory as './contracts/…', so a script generated +-- from the model carries no machine path. +-- +-- Repro (MetadataUrl resolves against the .mpr's directory): +-- +-- mkdir -p A/app/contracts +-- cp mdl-examples/odata-local-metadata/sample-metadata.xml A/app/contracts/f1-metadata.xml +-- mxcli exec mdl-examples/bug-tests/1499-odata-metadata-url-checkout-path.mdl -p A/app/App.mpr +-- mv A B # or cp -r A B: a second checkout of the same project +-- mxcli exec mdl-examples/bug-tests/1499-odata-metadata-url-checkout-path.mdl -p B/app/App.mpr +-- +-- Before the fix the second exec printed "Modified OData client"; after it the +-- client is unchanged ("documents already in sync"). Executing the script twice +-- from the same checkout is the control: unchanged before and after. +-- +-- Regression coverage: mdl/executor/cmd_odata_metadata_url_checkout_test.go +-- ============================================================================ + +create or modify module F1Cached; + +create or modify constant F1Cached.ApiLocation + ( Type: String, DefaultValue: 'http://localhost:8080/odata/cached/' ); + +create or modify consumed odata service F1Cached.CachedApi ( + Version: '1.0.0', + ODataVersion: OData4, + MetadataUrl: './contracts/f1-metadata.xml', + ServiceUrl: '@F1Cached.ApiLocation', + Timeout: '300' +); diff --git a/mdl-examples/bug-tests/1505-property-sets-below-10-24-24.mdl b/mdl-examples/bug-tests/1505-property-sets-below-10-24-24.mdl new file mode 100644 index 0000000000..a7cdeff279 --- /dev/null +++ b/mdl-examples/bug-tests/1505-property-sets-below-10-24-24.mdl @@ -0,0 +1,44 @@ +mdl 1; +-- ako/mxcli#1505 (mendixlabs/mxcli#1595): on Mendix 10.24 below 10.24.24 mxcli +-- wrote documents without properties Studio Pro writes. A re-created nanoflow +-- lacked ExportLevel and ReturnVariableName, so `mx diff` against the original +-- threw: +-- +-- Objects with ID … of type Microflows$Nanoflow do not have the same +-- properties. baseNames = …, ExportLevel, …, ReturnVariableName; newNames = … +-- +-- and a fresh microflow and entity lacked AggregateAction's four expression +-- keys, DeleteAction.ErrorHandlingType, WhileLoopCondition.Caption, +-- DatabaseRetrieveSource.XpathConstraint and EntityImpl Image/Source/ImageData. +-- `mxcli fix property-sets --dry-run` reported every unit complete. +-- +-- The table of values (modelsdk/canon/studiopro_property_defaults.json) was +-- measured from 10.24.24 up, and its lowest measured version was read as the +-- version that introduced each key. Re-measured on 10.24.0 and 10.24.18. +-- +-- Run it on a fresh 10.24.18 (or 10.24.0) app, keeping a copy as the base: +-- `mx diff base/App.mpr App/App.mpr out.json` exits 0, `mx convert -p` on a +-- copy adds no key to any element mxcli wrote, `fix property-sets --dry-run` +-- reports every unit complete, and `mx check` reports 0 errors. + +drop nanoflow Atlas_Web_Content.ACT_Login; +create nanoflow Atlas_Web_Content.ACT_Login () begin end; + +create module Probe1505; + +create persistent entity Probe1505.Item (Code: String(50), Status: String(20), Amount: Integer); + +create microflow Probe1505.ACT_Do () +begin + retrieve $List from Probe1505.Item; + $Cnt = count $List; + $Sum = sum $List by Amount; + $New = create Probe1505.Item (Code = 'a', Status = 'b'); + change $New (Amount = 5); + delete $New; + declare $i Integer = 0; + while $i < 3 + begin + set $i = $i + 1; + end while; +end; diff --git a/mdl-examples/bug-tests/1618-exec-names-files-beside-the-model.mdl b/mdl-examples/bug-tests/1618-exec-names-files-beside-the-model.mdl new file mode 100644 index 0000000000..65d7274f67 --- /dev/null +++ b/mdl-examples/bug-tests/1618-exec-names-files-beside-the-model.mdl @@ -0,0 +1,37 @@ +-- ako/mxcli#1618 (mendixlabs/mxcli#1634): `exec` did not say it wrote files +-- outside the model. +-- +-- A code action's body is not in the model. `create java action … as $$ … $$` +-- stores the action in a unit AND writes the body to +-- javasource//actions/.java; a JavaScript action's goes to +-- javascriptsource//actions/.js. `exec` printed only +-- "Created java action: Probe1618.JA_Hello", so a workflow that commits the model +-- (.mpr and mprcontents/) left the body untracked, and a clone got an action +-- with no logic. +-- +-- mxcli new JA --version 11.13.0 --theme none --layout none --skip-init --skip-build +-- cd JA && git init -q && git add -A && git commit -qm base +-- mxcli exec mdl-examples/bug-tests/1618-exec-names-files-beside-the-model.mdl -p JA.mpr +-- +-- Expected: after each "Created … action" line, the file it wrote: +-- +-- Wrote javasource/probe1618/actions/JA_Hello.java (Java action body; commit it with the model) +-- Wrote javascriptsource/probe1618/actions/JS_Hello.js (JavaScript action body; commit it with the model) +-- +-- and, last, "Files changed outside the model: 2 (javasource/, +-- javascriptsource/). Stage them with the .mpr." — the two paths `git status` +-- lists as untracked. Run it a second time: the files are unchanged, so neither +-- line appears. `mxcli docker check` reports 0 errors. +mdl 1; + +create module Probe1618; + +create or modify java action Probe1618.JA_Hello () returns String +as $$ +return "hi"; +$$; + +create or modify javascript action Probe1618.JS_Hello () returns String +as $$ +return "hi"; +$$; diff --git a/mdl-examples/bug-tests/1774-workflow-types-below-their-version.mdl b/mdl-examples/bug-tests/1774-workflow-types-below-their-version.mdl new file mode 100644 index 0000000000..15d43dce7f --- /dev/null +++ b/mdl-examples/bug-tests/1774-workflow-types-below-their-version.mdl @@ -0,0 +1,76 @@ +mdl 1; +-- ako/mxcli#1774: a $Type newer than the project was written with no version +-- gate, and the project then did not load. `exec` reported success; mx check +-- 9.24.30 (and Studio Pro) died before checking anything: +-- +-- TypeCacheUnknownTypeException: The type cache does not contain a type +-- with qualified name Workflows$WaitForNotificationActivity. +-- +-- The generated version data held PROPERTY floors only, so no write ever +-- compared an element's own $Type with the project's version. It shipped four +-- times (#1754, #1209, #1163, #1771) before every type's introduction version +-- was taken from the Model SDK (modelsdk/canon/mendixmodelsdk_type_versions.json) +-- and every unit write refused an element whose type the project does not +-- have yet. `check -p` and exec's pre-flight name the statements below first +-- (MDL-VERSION01), each at its type's floor (mendixmodelsdk 4.117.0): +-- +-- wait for notification 10.0 WaitForNotificationActivity +-- wait for timer 10.6 WaitForTimerActivity +-- boundary event [non] interrupting timer 10.20 (Non)InterruptingTimerBoundaryEvent +-- lock / unlock workflow on a workflow 10.0 WorkflowDefinition*Selection +-- +-- and `get workflows` (Microflows$GetWorkflowsAction, 10.4), which no gate +-- names, is refused by the write itself. +-- +-- Measured on fresh apps (`mxcli new … --skip-build`), exec then `mxcli docker +-- check`: +-- 9.24.30.51827 before: a workflow with `wait for notification` and a +-- microflow with `get workflows` each made the project +-- unloadable (TypeCacheUnknownTypeException naming the type). +-- After: the first is refused by check -p and exec's +-- pre-flight, the second at write time, and the project is +-- byte-identical to before exec; mx check 0 errors. +-- 10.24.24, 11.13.0 this script: exec and mx check as before the fix. + +create module Probe1774; + +create persistent entity Probe1774.Request ( + Title: String(100) +); + +create microflow Probe1774.ACT_Validate ($WorkflowContext: Probe1774.Request) +begin + log info node 'Probe1774' 'validate'; +end; + +create microflow Probe1774.ACT_Expired ($WorkflowContext: Probe1774.Request) +begin + log info node 'Probe1774' 'expired'; +end; + +create workflow Probe1774.WF_Wait + parameter $WorkflowContext: Probe1774.Request + display 'Wait' +begin + call microflow Probe1774.ACT_Validate as validate; + wait for timer WaitAnHour addHours([%CurrentDateTime%], 1); + wait for notification WaitForApproval; +end workflow; + +-- The boundary event is inserted through the raw mutator, outside CREATE's +-- encoder: the type floor runs on every unit write, so it covers both. +alter workflow Probe1774.WF_Wait { + insert into validate { boundary event interrupting timer addDays([%CurrentDateTime%], 1) { + call microflow Probe1774.ACT_Expired; + } } +}; + +create microflow Probe1774.MF_Workflows ( + $Request: Probe1774.Request, + $WfDef: System.WorkflowDefinition +) +begin + $Workflows = get workflows for $Request; + lock workflow $WfDef; + unlock workflow Probe1774.WF_Wait; +end; diff --git a/mdl-examples/bug-tests/1775-legacy-runner-retrieve-in-two-tests.test.mdl b/mdl-examples/bug-tests/1775-legacy-runner-retrieve-in-two-tests.test.mdl new file mode 100644 index 0000000000..e24fa217f2 --- /dev/null +++ b/mdl-examples/bug-tests/1775-legacy-runner-retrieve-in-two-tests.test.mdl @@ -0,0 +1,65 @@ +-- ako/mxcli#1775: on the after-startup runner (every Docker run, and --local +-- --legacy-runner) two tests in one file that define the same variable with +-- `retrieve` failed injection with MDL063 (CE0111 "Duplicate variable name" +-- from mxbuild). That runner joins every test into one microflow and renames +-- each test's variables with a `_N` suffix, but which ones it renamed came from +-- a pattern matching `$x = call|create` only, so a retrieve, declare, list +-- operation, aggregate, send rest request, execute database query or loop +-- iterator kept its name and was declared once per test. The names now come +-- from the parsed body, nested definitions included. The endpoint runner +-- (--local alone) gives each test its own microflow and was never affected. +-- +-- Setup: 1775-setup.mdl. Then: +-- +-- mxcli test mdl-examples/bug-tests/1775-legacy-runner-retrieve-in-two-tests.test.mdl -p app.mpr --local --legacy-runner +-- +-- Expected: 4 tests PASS. Before the fix, injection was refused with MDL063 +-- for $Items, $First, $Count and $Item before any test ran. The generator side +-- is pinned by cmd/mxcli/testrunner/rename_keyword_case_test.go. +mdl 1; + +/** + * @setup Probe.SUB_SeedItems + */ + +/** + * @test 1 a retrieve defines $Items + * @expect count($Items) >= 2 + */ +retrieve $Items from Probe.Item; +/ + +/** + * @test 2 a second test retrieves the same name + * @expect count($Items) >= 2 + */ +retrieve $Items from Probe.Item; +/ + +/** + * @test 3 a list operation, a declare and a loop iterator + * @expect $First != empty + * @expect $Count = count($Items) + */ +retrieve $Items from Probe.Item; +$First = head $Items; +declare $Count Integer = 0; +loop $Item in $Items begin + set $Count = $Count + 1; +end loop; +/ + +/** + * @test 4 the same names again, the loop iterator nested in an if branch + * @expect $First != empty + * @expect $Count = count($Items) + */ +retrieve $Items from Probe.Item; +$First = head $Items; +declare $Count Integer = 0; +if $First != empty then + loop $Item in $Items begin + set $Count = $Count + 1; + end loop; +end if; +/ diff --git a/mdl-examples/bug-tests/1775-setup.mdl b/mdl-examples/bug-tests/1775-setup.mdl new file mode 100644 index 0000000000..de1b389191 --- /dev/null +++ b/mdl-examples/bug-tests/1775-setup.mdl @@ -0,0 +1,14 @@ +-- ako/mxcli#1775: the setup script for +-- 1775-legacy-runner-retrieve-in-two-tests.test.mdl. Apply it before running +-- the tests: +-- +-- mxcli exec mdl-examples/bug-tests/1775-setup.mdl -p app.mpr +mdl 1; +create module Probe; +create persistent entity Probe.Item ( Name: string(100) ); +create microflow Probe.SUB_SeedItems () begin + $One = create Probe.Item ( Name = 'one' ); + $Two = create Probe.Item ( Name = 'two' ); + commit $One; + commit $Two; +end; diff --git a/mdl-examples/bug-tests/1777-alter-members-of-list-op-java-self-reference.mdl b/mdl-examples/bug-tests/1777-alter-members-of-list-op-java-self-reference.mdl new file mode 100644 index 0000000000..2557355708 --- /dev/null +++ b/mdl-examples/bug-tests/1777-alter-members-of-list-op-java-self-reference.mdl @@ -0,0 +1,119 @@ +mdl 1; +-- ako/mxcli#1777 — a fragment an alter splices names a member of a stored +-- list operation's, Java action's or self-reference retrieve's output +-- +-- A fragment spliced by `alter microflow` (or `create or modify`) starts from +-- the variables the stored flow declares, typed as `create microflow` types +-- them. #1769 typed a retrieve over an association and a loop's iterator; the +-- output of a list operation, of a Java action call and of a retrieve over a +-- self-reference stayed untyped, so a member of one was refused with nothing +-- written: "cannot qualify member "Code": the entity its variable holds is +-- not known here …". +-- +-- Now a list operation's output has its input list's type (the list for a +-- filter, sort, tail, union, intersect, subtract or range; an element for a +-- head or find), a Java action's result its declared return type (a list of +-- its type parameter: the list its argument holds), and a self-reference +-- retrieve what Mendix reads it as: the stored retrieve records no direction, +-- and Mendix follows it from the FROM end, so a reference gives the one +-- object and a reference set the list (measured on 11.13.0: a change on a +-- self-reference retrieve builds, a count of it is CE0013). +-- +-- Run on any project: +-- mxcli exec mdl-examples/bug-tests/1777-alter-members-of-list-op-java-self-reference.mdl -p app.mpr +-- mxcli bson dump -p app.mpr --type microflow --object MyModule.SUB_Shapes +-- Expected: every alter reports "Altered microflow MyModule.SUB_Shapes"; the +-- dump shows each find, filter and change on MyModule.OrderLine.Code, +-- MyModule.OrderLine.Status or MyModule.Order.Code; mx check reports 0 errors, +-- and exec of `describe microflow MyModule.SUB_Shapes` reports Unchanged. +-- Before the fix: the first seven alters are refused, "cannot qualify member" +-- (the last, over a self reference set, was typed by #1769) — measured on +-- 11.13.0. + +create module MyModule; +create persistent entity MyModule.Order (Code: String(50)); +create persistent entity MyModule.OrderLine (Code: String(50), Status: String(20)); +create association MyModule.OrderLine_Order from MyModule.OrderLine to MyModule.Order; +create association MyModule.Order_Parent from MyModule.Order to MyModule.Order; +create association MyModule.Order_Related from MyModule.Order to MyModule.Order type ReferenceSet; + +create java action MyModule.JA_GetOrder (Code: String) returns MyModule.Order as $$ + return null; +$$; + +create java action MyModule.JA_GetLines (Code: String) returns list of MyModule.OrderLine as $$ + return new java.util.ArrayList<>(); +$$; + +create java action MyModule.JA_Shuffle (Items: list of T) returns list of T as $$ + return Items; +$$; + +create microflow MyModule.SUB_Shapes ($Order: MyModule.Order, $Lines: List of MyModule.OrderLine) +begin + $Open = filter $Lines by Status = 'Open'; + $Sorted = sort $Lines by Code asc; + $First = head $Lines; + $Got = call java action MyModule.JA_GetOrder(Code = 'A'); + $GotLines = call java action MyModule.JA_GetLines(Code = 'A'); + $Shuffled = call java action MyModule.JA_Shuffle(Items = $Lines); + retrieve $Parent from $Order/MyModule.Order_Parent; + retrieve $Related from $Order/MyModule.Order_Related; + log info node 'Shapes' 'done'; +end; + +-- A filter's output: the list it filters. +alter microflow MyModule.SUB_Shapes { + insert after $Related begin + $OpenA = find $Open by Code = 'A'; + end; +}; + +-- A sort's output: the list it sorts. +alter microflow MyModule.SUB_Shapes { + insert after $Related begin + $SortedB = filter $Sorted by Code = 'B'; + end; +}; + +-- A head's output: one object of the list. +alter microflow MyModule.SUB_Shapes { + insert after $Related begin + change $First (Status = 'Seen'); + end; +}; + +-- A Java action returning an entity. +alter microflow MyModule.SUB_Shapes { + insert after $Related begin + change $Got (Code = 'X'); + end; +}; + +-- A Java action returning a list of an entity. +alter microflow MyModule.SUB_Shapes { + insert after $Related begin + $GotOpen = find $GotLines by Status = 'Open'; + end; +}; + +-- A Java action returning a list of its type parameter: the list it is given. +alter microflow MyModule.SUB_Shapes { + insert after $Related begin + $ShuffledA = find $Shuffled by Code = 'A'; + end; +}; + +-- A retrieve over a self-reference: the one object at the FROM end. +alter microflow MyModule.SUB_Shapes { + insert after $Related begin + change $Parent (Code = 'P'); + end; +}; + +-- A retrieve over a self reference set: the list (typed since #1769). +alter microflow MyModule.SUB_Shapes { + insert after $Related begin + $RelatedA = find $Related by Code = 'A'; + end; +}; diff --git a/mdl-examples/bug-tests/move-java-action-source-across-modules.mdl b/mdl-examples/bug-tests/move-java-action-source-across-modules.mdl new file mode 100644 index 0000000000..9a5b7842d4 --- /dev/null +++ b/mdl-examples/bug-tests/move-java-action-source-across-modules.mdl @@ -0,0 +1,32 @@ +-- A cross-module MOVE of a Java or JavaScript action takes its source file along. +-- +-- Before: the document moved and its source stayed under the old module's +-- directory. Measured on Mendix 11.13.0 with `mxcli docker build`: the build +-- succeeded, the Java class the app calls (jamb/actions/JA_Hello) was +-- mxbuild's generated stub throwing "Java action was not implemented", and the +-- author's code compiled only as an orphaned jama.actions class. The +-- JavaScript action's bundle held "JavaScript action was not implemented". +-- DESCRIBE printed the "Java source not available" placeholder, so re-executing +-- it replaced the code. +-- +-- After: exec reports "Moved javasource/jama/actions/JA_Hello.java to +-- javasource/jamb/actions/JA_Hello.java" (and the .js likewise), the Java source's +-- package line reads `package jamb.actions;` (what mxbuild regenerates, run +-- in place), DESCRIBE shows both bodies, and the build carries the author's code. +mdl 1; + +create module JamA; +create module JamB; + +create java action JamA.JA_Hello () returns String +as $$ +return "hi from the moved action"; +$$; + +create javascript action JamA.JSA_Hello () returns String +as $$ +return "hi from the moved js action"; +$$; + +move java action JamA.JA_Hello to JamB; +move javascript action JamA.JSA_Hello to JamB; diff --git a/mdl-examples/odata-local-metadata/README.md b/mdl-examples/odata-local-metadata/README.md index d1654caf16..a9ec6f22d2 100644 --- a/mdl-examples/odata-local-metadata/README.md +++ b/mdl-examples/odata-local-metadata/README.md @@ -11,7 +11,7 @@ This example demonstrates how to create consumed OData services using local meta ## Important Notes -1. **Relative paths are normalized** — Any relative path is automatically converted to an absolute `file://` URL in the Mendix model for Studio Pro compatibility +1. **Relative paths are normalized** — Any relative path is automatically converted to an absolute `file://` URL in the Mendix model for Studio Pro compatibility. Re-running the script from a checkout at another path keeps the stored URL when it names the same project file, and `describe` prints a stored URL inside the project as `./` (ako/mxcli#1499) 2. **ServiceUrl must be a constant** — Always use `@Module.ConstantName` format, not direct URLs ## Supported Formats diff --git a/mdl/backend/java.go b/mdl/backend/java.go index 49f77c192e..8b069b44f5 100644 --- a/mdl/backend/java.go +++ b/mdl/backend/java.go @@ -26,6 +26,11 @@ type JavaBackend interface { WriteJavaSourceFile(moduleName, actionName string, javaCode string, params []*javaactions.JavaActionParameter, returnType javaactions.CodeActionReturnType, extraImports []string, extraCode *string, documentation string) error DeleteJavaSourceFile(moduleName, actionName string) error RenameJavaSourceFile(moduleName, oldName, newName string) error + // MoveJavaSourceFile moves an action's source to another module's actions + // directory and rewrites its package line, refusing an occupied destination + // before touching anything. It returns the new project-relative path, or "" + // when there was no source to move. + MoveJavaSourceFile(fromModule, toModule, actionName string) (string, error) ReadJavaSourceFile(moduleName, actionName string) (string, error) CreateJavaScriptAction(jsa *types.JavaScriptAction) error @@ -34,4 +39,7 @@ type JavaBackend interface { WriteJavaScriptSourceFile(moduleName, actionName string, jsCode string, params []*types.JavaActionParameter, returnType types.CodeActionReturnType) error DeleteJavaScriptSourceFile(moduleName, actionName string) error RenameJavaScriptSourceFile(moduleName, oldName, newName string) error + // MoveJavaScriptSourceFile is MoveJavaSourceFile for a JavaScript action; + // its source names no module, so only the file moves. + MoveJavaScriptSourceFile(fromModule, toModule, actionName string) (string, error) } diff --git a/mdl/backend/mcp/unsupported_gen.go b/mdl/backend/mcp/unsupported_gen.go index a88608fbd7..dceed4e6cc 100644 --- a/mdl/backend/mcp/unsupported_gen.go +++ b/mdl/backend/mcp/unsupported_gen.go @@ -977,6 +977,16 @@ func (unsupportedBackend) MoveJavaAction(_ *javaactions.JavaAction) (err0 error) return } +func (unsupportedBackend) MoveJavaScriptSourceFile(_ string, _ string, _ string) (r0 string, err1 error) { + err1 = errUnsupported("MoveJavaScriptSourceFile") + return +} + +func (unsupportedBackend) MoveJavaSourceFile(_ string, _ string, _ string) (r0 string, err1 error) { + err1 = errUnsupported("MoveJavaSourceFile") + return +} + func (unsupportedBackend) MoveMicroflow(_ *microflows.Microflow) (err0 error) { err0 = errUnsupported("MoveMicroflow") return diff --git a/mdl/backend/mock/backend.go b/mdl/backend/mock/backend.go index b8b360aac4..78a387b733 100644 --- a/mdl/backend/mock/backend.go +++ b/mdl/backend/mock/backend.go @@ -265,6 +265,7 @@ type MockBackend struct { WriteJavaSourceFileFunc func(moduleName, actionName string, javaCode string, params []*javaactions.JavaActionParameter, returnType javaactions.CodeActionReturnType, extraImports []string, extraCode *string, documentation string) error DeleteJavaSourceFileFunc func(moduleName, actionName string) error RenameJavaSourceFileFunc func(moduleName, oldName, newName string) error + MoveJavaSourceFileFunc func(fromModule, toModule, actionName string) (string, error) ReadJavaSourceFileFunc func(moduleName, actionName string) (string, error) CreateJavaScriptActionFunc func(jsa *types.JavaScriptAction) error UpdateJavaScriptActionFunc func(jsa *types.JavaScriptAction) error @@ -272,6 +273,7 @@ type MockBackend struct { WriteJavaScriptSourceFileFunc func(moduleName, actionName string, jsCode string, params []*types.JavaActionParameter, returnType types.CodeActionReturnType) error DeleteJavaScriptSourceFileFunc func(moduleName, actionName string) error RenameJavaScriptSourceFileFunc func(moduleName, oldName, newName string) error + MoveJavaScriptSourceFileFunc func(fromModule, toModule, actionName string) (string, error) // WorkflowBackend ListWorkflowsFunc func() ([]*workflows.Workflow, error) diff --git a/mdl/backend/mock/mock_java.go b/mdl/backend/mock/mock_java.go index 63eca805c4..662d946b7a 100644 --- a/mdl/backend/mock/mock_java.go +++ b/mdl/backend/mock/mock_java.go @@ -85,6 +85,13 @@ func (m *MockBackend) RenameJavaSourceFile(moduleName, oldName, newName string) return nil } +func (m *MockBackend) MoveJavaSourceFile(fromModule, toModule, actionName string) (string, error) { + if m.MoveJavaSourceFileFunc != nil { + return m.MoveJavaSourceFileFunc(fromModule, toModule, actionName) + } + return "", nil +} + func (m *MockBackend) ReadJavaSourceFile(moduleName, actionName string) (string, error) { if m.ReadJavaSourceFileFunc != nil { return m.ReadJavaSourceFileFunc(moduleName, actionName) @@ -133,3 +140,10 @@ func (m *MockBackend) RenameJavaScriptSourceFile(moduleName, oldName, newName st } return nil } + +func (m *MockBackend) MoveJavaScriptSourceFile(fromModule, toModule, actionName string) (string, error) { + if m.MoveJavaScriptSourceFileFunc != nil { + return m.MoveJavaScriptSourceFileFunc(fromModule, toModule, actionName) + } + return "", nil +} diff --git a/mdl/backend/modelsdk/backend.go b/mdl/backend/modelsdk/backend.go index 383fc1f0fe..2fea532e8a 100644 --- a/mdl/backend/modelsdk/backend.go +++ b/mdl/backend/modelsdk/backend.go @@ -19,6 +19,8 @@ package modelsdkbackend import ( "fmt" + "path/filepath" + "strings" "github.com/mendixlabs/mxcli/mdl/backend" "github.com/mendixlabs/mxcli/mdl/types" @@ -32,6 +34,7 @@ import ( // embedded `unimplemented` for every method it doesn't override). var _ backend.FullBackend = (*Backend)(nil) var _ backend.WriteStatsReporter = (*Backend)(nil) +var _ backend.SourceFileReporter = (*Backend)(nil) // Backend reads and writes a Mendix project through the modelsdk codec engine. // It embeds `unimplemented` (generated, see gen_unimplemented.go) so any @@ -49,6 +52,10 @@ type Backend struct { // javasource/ or javascriptsource/ rather than in its unit. They are folded // into WriteStats so a body-only edit is not reported as "Unchanged". fileWrites backend.WriteStats + + // sourceFiles lists the files and folders beside the model that writes + // changed since the executor last took them (TakeSourceFileChanges). + sourceFiles []backend.SourceFileChange } // New constructs a modelsdk backend. @@ -95,6 +102,46 @@ func (b *Backend) noteFileWrite(changed bool) { } } +// Readers' names for what the files beside the model hold. +const ( + javaActionBody = "Java action body" + javaScriptActionBody = "JavaScript action body" +) + +// noteSourceFile records a file or folder beside the model that a write just +// changed, by its absolute path (from, for a move, likewise; empty otherwise). +// Call it only once the change is on disk. +func (b *Backend) noteSourceFile(path, from string, removed bool, what string) { + c := backend.SourceFileChange{Path: b.projectRelative(path), Removed: removed, What: what} + if from != "" { + c.From = b.projectRelative(from) + } + b.sourceFiles = append(b.sourceFiles, c) +} + +// projectRelative turns an absolute path under the project directory into the +// slash-separated path a reader would stage. A folder's trailing separator is +// kept, as "/". +func (b *Backend) projectRelative(path string) string { + folder := strings.HasSuffix(path, string(filepath.Separator)) + if rel, err := filepath.Rel(filepath.Dir(b.path), path); err == nil { + path = rel + } + path = filepath.ToSlash(path) + if folder && !strings.HasSuffix(path, "/") { + path += "/" + } + return path +} + +// TakeSourceFileChanges returns the files and folders beside the model that +// writes changed since the previous call, and forgets them. +func (b *Backend) TakeSourceFileChanges() []backend.SourceFileChange { + changes := b.sourceFiles + b.sourceFiles = nil + return changes +} + // --- ConnectionBackend --- // Connect opens the project read-write through the modelsdk reader/writer diff --git a/mdl/backend/modelsdk/domainmodel.go b/mdl/backend/modelsdk/domainmodel.go index ec4af54ea2..0627f0f8ad 100644 --- a/mdl/backend/modelsdk/domainmodel.go +++ b/mdl/backend/modelsdk/domainmodel.go @@ -69,17 +69,27 @@ func (b *Backend) populateViewEntityOql(dms []*domainmodel.DomainModel) { if err != nil { return } + // The modules are listed once: GetModule lists them all per call, so a + // lookup per view-entity document read every module unit that many times. + modules, err := b.reader.ListModules() + if err != nil { + return + } + moduleNames := make(map[string]string, len(modules)) + for _, m := range modules { + moduleNames[m.ID] = m.Name + } oqlByQN := make(map[string]string, len(units)) for _, u := range units { name := u.Element.Name() if name == "" { continue } - mi, _ := b.reader.GetModule(string(u.ContainerID)) - if mi == nil { + modName, ok := moduleNames[string(u.ContainerID)] + if !ok { continue } - oqlByQN[mi.Name+"."+name] = u.Element.Oql() + oqlByQN[modName+"."+name] = u.Element.Oql() } for _, dm := range dms { for _, e := range dm.Entities { diff --git a/mdl/backend/modelsdk/java_move_source_test.go b/mdl/backend/modelsdk/java_move_source_test.go new file mode 100644 index 0000000000..7960dcde5c --- /dev/null +++ b/mdl/backend/modelsdk/java_move_source_test.go @@ -0,0 +1,123 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// A cross-module MOVE of a Java or JavaScript action must take its source file +// along: mxbuild looks for it under the new module's directory and generates a +// stub that throws when it finds nothing there (measured on 11.13.0; see +// sdk/javaactions/move_test.go). + +func writeFile(t *testing.T, path, content string) { + t.Helper() + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(content), 0o644); err != nil { + t.Fatal(err) + } +} + +func TestMoveJavaSourceFileTakesTheCodeAlong(t *testing.T) { + root := t.TempDir() + from := filepath.Join(root, "javasource", "jama", "actions", "JA_Hello.java") + writeFile(t, from, "package jama.actions;\r\n\r\npublic class JA_Hello {\r\n\t// BEGIN USER CODE\r\n\treturn \"jama.actions\";\r\n\t// END USER CODE\r\n}\r\n") + + b := &Backend{path: filepath.Join(root, "App.mpr")} + moved, err := b.MoveJavaSourceFile("JamA", "JamB", "JA_Hello") + if err != nil { + t.Fatal(err) + } + if moved != "javasource/jamb/actions/JA_Hello.java" { + t.Errorf("reported %q, want the project-relative destination", moved) + } + if _, err := os.Stat(from); !os.IsNotExist(err) { + t.Errorf("the source is still under the old module (stat err %v)", err) + } + got, err := os.ReadFile(filepath.Join(root, "javasource", "jamb", "actions", "JA_Hello.java")) + if err != nil { + t.Fatal(err) + } + if !strings.HasPrefix(string(got), "package jamb.actions;\r\n") { + t.Errorf("package line not rewritten:\n%s", got) + } + if !strings.Contains(string(got), "return \"jama.actions\";") { + t.Errorf("the user code was rewritten:\n%s", got) + } +} + +// The destination already holding a file is refused before anything moves: it +// is the author's code too, and nothing says which of the two is wanted. +func TestMoveJavaSourceFileRefusesAnOccupiedDestination(t *testing.T) { + root := t.TempDir() + from := filepath.Join(root, "javasource", "jama", "actions", "JA_Hello.java") + to := filepath.Join(root, "javasource", "jamb", "actions", "JA_Hello.java") + writeFile(t, from, "package jama.actions; // A\n") + writeFile(t, to, "package jamb.actions; // B\n") + + b := &Backend{path: filepath.Join(root, "App.mpr")} + if _, err := b.MoveJavaSourceFile("JamA", "JamB", "JA_Hello"); err == nil || !strings.Contains(err.Error(), "already exists") { + t.Fatalf("want an already-exists refusal, got %v", err) + } + for path, want := range map[string]string{from: "// A", to: "// B"} { + got, err := os.ReadFile(path) + if err != nil || !strings.Contains(string(got), want) { + t.Errorf("%s changed (err %v): %q", path, err, got) + } + } +} + +// An action with no source on disk yet has nothing to move. +func TestMoveJavaSourceFileWithoutSource(t *testing.T) { + b := &Backend{path: filepath.Join(t.TempDir(), "App.mpr")} + moved, err := b.MoveJavaSourceFile("JamA", "JamB", "JA_Hello") + if err != nil || moved != "" { + t.Errorf("got (%q, %v), want nothing moved and no error", moved, err) + } +} + +func TestMoveJavaScriptSourceFileTakesTheCodeAlong(t *testing.T) { + root := t.TempDir() + from := filepath.Join(root, "javascriptsource", "jama", "actions", "JSA_Hello.js") + const src = "// This file was generated by Mendix Studio Pro.\nexport async function JSA_Hello() {\n\t// BEGIN USER CODE\n\treturn 1;\n\t// END USER CODE\n}\n" + writeFile(t, from, src) + + b := &Backend{path: filepath.Join(root, "App.mpr")} + moved, err := b.MoveJavaScriptSourceFile("JamA", "JamB", "JSA_Hello") + if err != nil { + t.Fatal(err) + } + if moved != "javascriptsource/jamb/actions/JSA_Hello.js" { + t.Errorf("reported %q", moved) + } + got, err := os.ReadFile(filepath.Join(root, "javascriptsource", "jamb", "actions", "JSA_Hello.js")) + if err != nil || string(got) != src { + t.Errorf("the source did not arrive unchanged (err %v): %q", err, got) + } + if _, err := os.Stat(from); !os.IsNotExist(err) { + t.Errorf("the source is still under the old module (stat err %v)", err) + } +} + +// The move is one of the files beside the model a statement touched, so exec +// names it with the rest (ako/mxcli#1618): as a move, from the old path. +func TestMoveJavaSourceFileIsReported(t *testing.T) { + root := t.TempDir() + writeFile(t, filepath.Join(root, "javasource", "jama", "actions", "JA_Hello.java"), "package jama.actions;\n") + + b := &Backend{path: filepath.Join(root, "App.mpr")} + if _, err := b.MoveJavaSourceFile("JamA", "JamB", "JA_Hello"); err != nil { + t.Fatal(err) + } + changes := b.TakeSourceFileChanges() + if len(changes) != 1 || changes[0].Path != "javasource/jamb/actions/JA_Hello.java" || + changes[0].From != "javasource/jama/actions/JA_Hello.java" || changes[0].Removed { + t.Errorf("reported %+v, want one move jama -> jamb", changes) + } +} diff --git a/mdl/backend/modelsdk/java_write.go b/mdl/backend/modelsdk/java_write.go index b28b9fe07f..6ec59a22ed 100644 --- a/mdl/backend/modelsdk/java_write.go +++ b/mdl/backend/modelsdk/java_write.go @@ -115,6 +115,9 @@ func (b *Backend) WriteJavaSourceFile(moduleName, actionName string, javaCode st if err != nil { return fmt.Errorf("WriteJavaSourceFile: write: %w", err) } + if changed { + b.noteSourceFile(path, "", false, javaActionBody) + } return nil } @@ -154,12 +157,75 @@ func (b *Backend) RenameJavaSourceFile(moduleName, oldName, newName string) erro } // The move is a write whether or not the text needed rewriting. b.noteFileWrite(true) + b.noteSourceFile(newPath, oldPath, false, javaActionBody) if _, err := javaactions.WriteSourceIfChanged(newPath, javaactions.RenameSource(string(content), oldName, newName)); err != nil { return fmt.Errorf("RenameJavaSourceFile: rewrite class name: %w", err) } return nil } +// MoveJavaSourceFile moves javasource//actions/.java into 's +// actions directory and rewrites its package line, as mxbuild regenerates it +// there. Left behind, the source was not where mxbuild looks: measured on +// 11.13.0, the build succeeded with a stub that throws "Java action was not +// implemented" in the class the app calls, and compiled the author's code as an +// orphaned class nothing calls. An occupied destination is refused before +// anything moves; a missing source is not an error (the action may have no stub +// yet). It returns the new project-relative path, or "" when nothing moved. +func (b *Backend) MoveJavaSourceFile(fromModule, toModule, actionName string) (string, error) { + if b.path == "" { + return "", fmt.Errorf("MoveJavaSourceFile: no project path") + } + dir := func(module string) string { + return filepath.Join(filepath.Dir(b.path), "javasource", strings.ToLower(module), "actions") + } + moved, content, err := b.moveSourceFile(dir(fromModule), dir(toModule), actionName+".java", javaActionBody) + if moved == "" || err != nil { + return moved, err + } + dst := filepath.Join(filepath.Dir(b.path), filepath.FromSlash(moved)) + if _, err := javaactions.WriteSourceIfChanged(dst, javaactions.MoveSource(content, fromModule, toModule)); err != nil { + return moved, fmt.Errorf("MoveJavaSourceFile: rewrite package: %w", err) + } + return moved, nil +} + +// moveSourceFile moves fromDir/name to toDir/name, records the move for exec's +// report as what, and returns the new project-relative path and the moved +// file's content. It refuses an existing destination before touching anything, +// and moves nothing — returning "" — when there is no source or the two +// directories are the same. +func (b *Backend) moveSourceFile(fromDir, toDir, name, what string) (string, string, error) { + if filepath.Clean(fromDir) == filepath.Clean(toDir) { + return "", "", nil + } + src, dst := filepath.Join(fromDir, name), filepath.Join(toDir, name) + content, err := os.ReadFile(src) + if os.IsNotExist(err) { + return "", "", nil + } + if err != nil { + return "", "", err + } + rel, err := filepath.Rel(filepath.Dir(b.path), dst) + if err != nil { + return "", "", err + } + rel = filepath.ToSlash(rel) + if _, err := os.Stat(dst); err == nil { + return "", "", fmt.Errorf("%s already exists; move or delete it first", rel) + } + if err := os.MkdirAll(toDir, 0o755); err != nil { + return "", "", err + } + if err := os.Rename(src, dst); err != nil { + return "", "", err + } + b.noteFileWrite(true) + b.noteSourceFile(dst, src, false, what) + return rel, string(content), nil +} + // DeleteJavaSourceFile removes javasource//actions/.java. A missing // file is not an error. func (b *Backend) DeleteJavaSourceFile(moduleName, actionName string) error { @@ -167,7 +233,10 @@ func (b *Backend) DeleteJavaSourceFile(moduleName, actionName string) error { return fmt.Errorf("DeleteJavaSourceFile: no project path") } filePath := filepath.Join(filepath.Dir(b.path), "javasource", strings.ToLower(moduleName), "actions", actionName+".java") - if err := os.Remove(filePath); err != nil && !os.IsNotExist(err) { + switch err := os.Remove(filePath); { + case err == nil: + b.noteSourceFile(filePath, "", true, javaActionBody) + case !os.IsNotExist(err): return fmt.Errorf("DeleteJavaSourceFile: %w", err) } return nil diff --git a/mdl/backend/modelsdk/javascript_write.go b/mdl/backend/modelsdk/javascript_write.go index 5d0e547555..4fcc632559 100644 --- a/mdl/backend/modelsdk/javascript_write.go +++ b/mdl/backend/modelsdk/javascript_write.go @@ -184,6 +184,9 @@ func (b *Backend) WriteJavaScriptSourceFile(moduleName, actionName string, jsCod if err != nil { return fmt.Errorf("WriteJavaScriptSourceFile: write: %w", err) } + if changed { + b.noteSourceFile(path, "", false, javaScriptActionBody) + } return nil } @@ -193,19 +196,39 @@ func (b *Backend) DeleteJavaScriptSourceFile(moduleName, actionName string) erro return fmt.Errorf("DeleteJavaScriptSourceFile: no project path") } filePath := filepath.Join(b.jsActionSourceDir(moduleName), actionName+".js") - if err := os.Remove(filePath); err != nil && !os.IsNotExist(err) { + switch err := os.Remove(filePath); { + case err == nil: + b.noteSourceFile(filePath, "", true, javaScriptActionBody) + case !os.IsNotExist(err): return fmt.Errorf("DeleteJavaScriptSourceFile: %w", err) } return nil } +// MoveJavaScriptSourceFile moves javascriptsource//actions/.js into +// 's actions directory when the action changes module, for the reason +// MoveJavaSourceFile gives: mxbuild looks for the source under the action's +// module and bundles a stub that throws when it finds none. The source names no +// module, so it moves as it is. +func (b *Backend) MoveJavaScriptSourceFile(fromModule, toModule, actionName string) (string, error) { + if b.path == "" { + return "", fmt.Errorf("MoveJavaScriptSourceFile: no project path") + } + moved, _, err := b.moveSourceFile(b.jsActionSourceDir(fromModule), b.jsActionSourceDir(toModule), actionName+".js", javaScriptActionBody) + return moved, err +} + // RenameJavaScriptSourceFile renames the .js file when a JS action is renamed. func (b *Backend) RenameJavaScriptSourceFile(moduleName, oldName, newName string) error { if b.path == "" { return fmt.Errorf("RenameJavaScriptSourceFile: no project path") } dir := b.jsActionSourceDir(moduleName) - if err := os.Rename(filepath.Join(dir, oldName+".js"), filepath.Join(dir, newName+".js")); err != nil && !os.IsNotExist(err) { + oldPath, newPath := filepath.Join(dir, oldName+".js"), filepath.Join(dir, newName+".js") + switch err := os.Rename(oldPath, newPath); { + case err == nil: + b.noteSourceFile(newPath, oldPath, false, javaScriptActionBody) + case !os.IsNotExist(err): return fmt.Errorf("RenameJavaScriptSourceFile: %w", err) } return nil diff --git a/mdl/backend/modelsdk/microflow.go b/mdl/backend/modelsdk/microflow.go index 2541020ec9..c2965b1b48 100644 --- a/mdl/backend/modelsdk/microflow.go +++ b/mdl/backend/modelsdk/microflow.go @@ -39,7 +39,7 @@ func (b *Backend) ListMicroflows() ([]*microflows.Microflow, error) { // name, in every module: a lookup by name decodes only those // (mendixlabs/mxcli#1272). func (b *Backend) ListMicroflowsNamed(name string) ([]*microflows.Microflow, error) { - units, err := mprread.ListUnitsWithContainerWhere[*genMf.Microflow](b.reader, mprread.HasName(name)) + units, err := mprread.ListUnitsNamed[*genMf.Microflow](b.reader, name) if err != nil { return nil, err } @@ -65,16 +65,11 @@ func (b *Backend) ListMicroflowHeaders() ([]types.DocumentHeader, error) { } func (b *Backend) GetMicroflow(id model.ID) (*microflows.Microflow, error) { - units, err := mprread.ListUnitsWithContainerWhere[*genMf.Microflow](b.reader, mprread.HasID(id)) - if err != nil { + u, ok, err := mprread.UnitByID[*genMf.Microflow](b.reader, id) + if err != nil || !ok { return nil, err } - for _, u := range units { - if model.ID(u.Element.ID()) == id { - return microflowFromGen(u.Element, u.ContainerID), nil - } - } - return nil, nil + return microflowFromGen(u.Element, u.ContainerID), nil } // IsRule reports whether qualifiedName refers to a rule (Microflows$Rule). diff --git a/mdl/backend/modelsdk/microflow_get_cost_test.go b/mdl/backend/modelsdk/microflow_get_cost_test.go new file mode 100644 index 0000000000..2db046e13a --- /dev/null +++ b/mdl/backend/modelsdk/microflow_get_cost_test.go @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "reflect" + "testing" +) + +// ako/mxcli#1789: GetMicroflow(id) listed every microflow of the project and +// kept the one with that $ID. The catalog's source build reads each microflow +// it describes by ID, so the build read every microflow once per microflow. +// A unit's ID is its document's $ID, so the lookup reads that unit alone. + +func TestGetMicroflow_ReadsOneUnitFile(t *testing.T) { + b := New() + if err := b.Connect(copyPedApp(t)); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = b.Disconnect() }) + + all, err := b.ListMicroflows() + if err != nil { + t.Fatalf("list microflows: %v", err) + } + if len(all) < 4 { + t.Fatalf("control: the fixture has %d microflows; the bound below would prove nothing", len(all)) + } + before := b.UnitFileReads() + if _, err := b.GetMicroflow(all[len(all)-1].ID); err != nil { + t.Fatalf("GetMicroflow: %v", err) + } + if reads := b.UnitFileReads() - before; reads != 1 { + t.Errorf("GetMicroflow of one microflow read %d unit files; the project has %d microflows, and it needs one", reads, len(all)) + } +} + +// The narrowing must not change the answer: every microflow is what the +// listing says it is, and an ID that names no microflow — another document's, +// or none at all — finds nothing. No MPR v1 fixture holds a microflow; v1's +// by-ID read is pinned in modelsdk/mpr (TestUnitByID_AgreesWithTheListing). +func TestGetMicroflow_AgreesWithTheListing(t *testing.T) { + for _, fixture := range []struct { + name string + path func(*testing.T) string + }{ + {"pedapp", copyPedApp}, + } { + t.Run(fixture.name, func(t *testing.T) { + b := New() + if err := b.Connect(fixture.path(t)); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = b.Disconnect() }) + all, err := b.ListMicroflows() + if err != nil { + t.Fatalf("list microflows: %v", err) + } + if len(all) == 0 { + t.Fatalf("control: the fixture has no microflow") + } + for _, want := range all { + got, err := b.GetMicroflow(want.ID) + if err != nil { + t.Fatalf("GetMicroflow(%s): %v", want.ID, err) + } + if !reflect.DeepEqual(got, want) { + t.Errorf("GetMicroflow(%s) differs from the listing's %s", want.ID, want.Name) + } + } + mods, err := b.ListModules() + if err != nil || len(mods) == 0 { + t.Fatalf("list modules: %v", err) + } + if got, err := b.GetMicroflow(mods[0].ID); err != nil || got != nil { + t.Errorf("GetMicroflow(a module's ID) = %v, %v; want nil, nil", got, err) + } + if got, err := b.GetMicroflow("00000000-0000-0000-0000-000000000000"); err != nil || got != nil { + t.Errorf("GetMicroflow(unknown ID) = %v, %v; want nil, nil", got, err) + } + }) + } +} diff --git a/mdl/backend/modelsdk/module_write.go b/mdl/backend/modelsdk/module_write.go index ba61a6ed30..0ea3cb1e42 100644 --- a/mdl/backend/modelsdk/module_write.go +++ b/mdl/backend/modelsdk/module_write.go @@ -168,24 +168,34 @@ func (b *Backend) DeleteModuleWithCleanup(id model.ID, moduleName string) error if err := b.DeleteModule(id); err != nil { return err } + b.removeModuleSourceDirs(moduleName) + return nil +} + +// removeModuleSourceDirs removes a module's source folders beside the model. +func (b *Backend) removeModuleSourceDirs(moduleName string) { if b.path == "" { - return nil + return } projectDir := filepath.Dir(b.path) lower := strings.ToLower(moduleName) // themesource and javasource use the lowercased module name; javascriptsource // uses the original casing (with a lowercase fallback). - for _, dir := range []string{ - filepath.Join(projectDir, "themesource", lower), - filepath.Join(projectDir, "javasource", lower), - filepath.Join(projectDir, "javascriptsource", moduleName), - filepath.Join(projectDir, "javascriptsource", lower), + for _, d := range []struct{ dir, what string }{ + {filepath.Join(projectDir, "themesource", lower), "module theme source"}, + {filepath.Join(projectDir, "javasource", lower), "module Java source"}, + {filepath.Join(projectDir, "javascriptsource", moduleName), "module JavaScript source"}, + {filepath.Join(projectDir, "javascriptsource", lower), "module JavaScript source"}, } { - if stat, err := os.Stat(dir); err == nil && stat.IsDir() { - _ = os.RemoveAll(dir) + if stat, err := os.Stat(d.dir); err == nil && stat.IsDir() { + // Recorded only once gone: a folder that is still there needs no + // staging. On a case-insensitive filesystem the two javascriptsource + // spellings are one folder, and the second stat finds nothing. + if os.RemoveAll(d.dir) == nil { + b.noteSourceFile(d.dir+string(filepath.Separator), "", true, d.what) + } } } - return nil } // collectDescendantUnitIDs returns every unit ID transitively contained under diff --git a/mdl/backend/modelsdk/source_file_changes_test.go b/mdl/backend/modelsdk/source_file_changes_test.go new file mode 100644 index 0000000000..ba68d0337f --- /dev/null +++ b/mdl/backend/modelsdk/source_file_changes_test.go @@ -0,0 +1,156 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "os" + "path/filepath" + "reflect" + "testing" + + "github.com/mendixlabs/mxcli/mdl/backend" +) + +// ako/mxcli#1618 (mendixlabs/mxcli#1634): a code action's body is not in the +// model. `create java action … as $$ … $$` stores the action in a unit and +// writes the body to javasource//actions/.java, and `exec` named +// only the model element — so a workflow that commits the model (.mpr and +// mprcontents/) left the body behind, and a clone got an action with no logic. +// The backend now records every file it changes beside the model, at the point +// it touches disk, so `exec` can name each one. + +var _ backend.SourceFileReporter = (*Backend)(nil) + +func newSourceFileBackend(t *testing.T) (*Backend, string) { + t.Helper() + root := t.TempDir() + return &Backend{path: filepath.Join(root, "App.mpr")}, root +} + +func takeChanges(t *testing.T, b *Backend) []backend.SourceFileChange { + t.Helper() + return b.TakeSourceFileChanges() +} + +func TestWriteJavaSourceFileRecordsTheFile(t *testing.T) { + b, _ := newSourceFileBackend(t) + if err := b.WriteJavaSourceFile("Probe", "JA_Hello", `return "hi";`, nil, nil, nil, nil, ""); err != nil { + t.Fatal(err) + } + want := []backend.SourceFileChange{{Path: "javasource/probe/actions/JA_Hello.java", What: "Java action body"}} + if got := takeChanges(t, b); !reflect.DeepEqual(got, want) { + t.Errorf("changes = %+v, want %+v", got, want) + } + if got := takeChanges(t, b); len(got) != 0 { + t.Errorf("a second take returned %+v; taking must forget what it returned", got) + } + + // An identical rewrite does not touch the file, so there is nothing to + // commit and nothing is recorded. + if err := b.WriteJavaSourceFile("Probe", "JA_Hello", `return "hi";`, nil, nil, nil, nil, ""); err != nil { + t.Fatal(err) + } + if got := takeChanges(t, b); len(got) != 0 { + t.Errorf("identical rewrite recorded %+v, want nothing", got) + } + + // Control: a changed body is recorded again. + if err := b.WriteJavaSourceFile("Probe", "JA_Hello", `return "bye";`, nil, nil, nil, nil, ""); err != nil { + t.Fatal(err) + } + if got := takeChanges(t, b); !reflect.DeepEqual(got, want) { + t.Errorf("changed body: changes = %+v, want %+v", got, want) + } +} + +func TestWriteJavaScriptSourceFileRecordsTheFile(t *testing.T) { + b, _ := newSourceFileBackend(t) + if err := b.WriteJavaScriptSourceFile("Probe", "JS_Hello", `return "hi";`, nil, nil); err != nil { + t.Fatal(err) + } + want := []backend.SourceFileChange{{Path: "javascriptsource/probe/actions/JS_Hello.js", What: "JavaScript action body"}} + if got := takeChanges(t, b); !reflect.DeepEqual(got, want) { + t.Errorf("changes = %+v, want %+v", got, want) + } + if err := b.WriteJavaScriptSourceFile("Probe", "JS_Hello", `return "hi";`, nil, nil); err != nil { + t.Fatal(err) + } + if got := takeChanges(t, b); len(got) != 0 { + t.Errorf("identical rewrite recorded %+v, want nothing", got) + } +} + +// A rename moves the body, and a drop deletes it: both are changes the commit +// has to carry, or a clone keeps the old file and lacks the new one. +func TestRenameAndDeleteOfASourceFileAreRecorded(t *testing.T) { + b, _ := newSourceFileBackend(t) + if err := b.WriteJavaSourceFile("Probe", "JA_Old", `return "hi";`, nil, nil, nil, nil, ""); err != nil { + t.Fatal(err) + } + if err := b.WriteJavaScriptSourceFile("Probe", "JS_Old", `return "hi";`, nil, nil); err != nil { + t.Fatal(err) + } + takeChanges(t, b) + + if err := b.RenameJavaSourceFile("Probe", "JA_Old", "JA_New"); err != nil { + t.Fatal(err) + } + if err := b.RenameJavaScriptSourceFile("Probe", "JS_Old", "JS_New"); err != nil { + t.Fatal(err) + } + if err := b.DeleteJavaSourceFile("Probe", "JA_New"); err != nil { + t.Fatal(err) + } + if err := b.DeleteJavaScriptSourceFile("Probe", "JS_New"); err != nil { + t.Fatal(err) + } + want := []backend.SourceFileChange{ + {Path: "javasource/probe/actions/JA_New.java", From: "javasource/probe/actions/JA_Old.java", What: "Java action body"}, + {Path: "javascriptsource/probe/actions/JS_New.js", From: "javascriptsource/probe/actions/JS_Old.js", What: "JavaScript action body"}, + {Path: "javasource/probe/actions/JA_New.java", Removed: true, What: "Java action body"}, + {Path: "javascriptsource/probe/actions/JS_New.js", Removed: true, What: "JavaScript action body"}, + } + if got := takeChanges(t, b); !reflect.DeepEqual(got, want) { + t.Errorf("changes =\n%+v\nwant\n%+v", got, want) + } + + // A file that is not there is not renamed or removed, and nothing is recorded. + if err := b.RenameJavaSourceFile("Probe", "JA_Missing", "JA_Other"); err != nil { + t.Fatal(err) + } + if err := b.RenameJavaScriptSourceFile("Probe", "JS_Missing", "JS_Other"); err != nil { + t.Fatal(err) + } + if err := b.DeleteJavaSourceFile("Probe", "JA_Missing"); err != nil { + t.Fatal(err) + } + if err := b.DeleteJavaScriptSourceFile("Probe", "JS_Missing"); err != nil { + t.Fatal(err) + } + if got := takeChanges(t, b); len(got) != 0 { + t.Errorf("missing files recorded %+v, want nothing", got) + } +} + +// DROP MODULE removes the module's source folders beside the model. +func TestRemovingAModulesSourceFoldersIsRecorded(t *testing.T) { + b, root := newSourceFileBackend(t) + for _, dir := range []string{"themesource/probe/web", "javasource/probe/actions"} { + if err := os.MkdirAll(filepath.Join(root, filepath.FromSlash(dir)), 0o755); err != nil { + t.Fatal(err) + } + } + b.removeModuleSourceDirs("Probe") + want := []backend.SourceFileChange{ + {Path: "themesource/probe/", Removed: true, What: "module theme source"}, + {Path: "javasource/probe/", Removed: true, What: "module Java source"}, + } + if got := takeChanges(t, b); !reflect.DeepEqual(got, want) { + t.Errorf("changes = %+v, want %+v", got, want) + } + for _, dir := range []string{"themesource/probe", "javasource/probe"} { + if _, err := os.Stat(filepath.Join(root, filepath.FromSlash(dir))); !os.IsNotExist(err) { + t.Errorf("%s still exists (stat err %v)", dir, err) + } + } +} diff --git a/mdl/backend/modelsdk/unimplemented_gen.go b/mdl/backend/modelsdk/unimplemented_gen.go index 9eab3b463c..ed7db0a0b2 100644 --- a/mdl/backend/modelsdk/unimplemented_gen.go +++ b/mdl/backend/modelsdk/unimplemented_gen.go @@ -880,6 +880,16 @@ func (unimplemented) MoveJavaAction(_ *javaactions.JavaAction) error { return errUnimplemented("MoveJavaAction") } +func (unimplemented) MoveJavaScriptSourceFile(_ string, _ string, _ string) (string, error) { + var r0 string + return r0, errUnimplemented("MoveJavaScriptSourceFile") +} + +func (unimplemented) MoveJavaSourceFile(_ string, _ string, _ string) (string, error) { + var r0 string + return r0, errUnimplemented("MoveJavaSourceFile") +} + func (unimplemented) MoveMicroflow(_ *microflows.Microflow) error { return errUnimplemented("MoveMicroflow") } diff --git a/mdl/backend/modelsdk/workflow_agent_test.go b/mdl/backend/modelsdk/workflow_agent_test.go index b8e66fa0a7..0393eeb505 100644 --- a/mdl/backend/modelsdk/workflow_agent_test.go +++ b/mdl/backend/modelsdk/workflow_agent_test.go @@ -4,6 +4,7 @@ package modelsdkbackend import ( "fmt" + "strings" "testing" "github.com/mendixlabs/mxcli/sdk/workflows" @@ -15,6 +16,10 @@ import ( // call-microflow activity it resembles. func TestCreateWorkflow_AIAgentTaskRoundTrip(t *testing.T) { proj := copyFixture(t) + // Workflows$AIAgentTaskActivity is an 11.9 type (absent from the 11.6 + // mxbuild); the fixture is 11.6.6, which cannot store it (ako/mxcli#1774: + // TestCreateWorkflow_AIAgentTaskRefusedBelow11_9). Written where it exists. + setProductVersion(t, proj, "11.14.0") b := New() if err := b.Connect(proj); err != nil { t.Fatalf("connect: %v", err) @@ -107,3 +112,42 @@ func TestCreateWorkflow_AIAgentTaskRoundTrip(t *testing.T) { } } } + +// On the 11.6.6 fixture the same workflow is refused: the 11.6 mxbuild has no +// Workflows$AIAgentTaskActivity, so the project would not load, and the write +// names the type instead (ako/mxcli#1774). A plain microflow call is written. +func TestCreateWorkflow_AIAgentTaskRefusedBelow11_9(t *testing.T) { + proj := copyFixture(t) + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = b.Disconnect() }) + mod, err := b.GetModuleByName("MyFirstModule") + if err != nil || mod == nil { + t.Fatalf("GetModuleByName: %v", err) + } + workflow := func(name string, agent bool) *workflows.Workflow { + return &workflows.Workflow{ + ContainerID: mod.ID, + Name: name, + Parameter: &workflows.WorkflowParameter{EntityRef: "MyFirstModule.Ctx"}, + Flow: &workflows.Flow{Activities: []workflows.WorkflowActivity{ + &workflows.StartWorkflowActivity{BaseWorkflowActivity: workflows.BaseWorkflowActivity{Name: "Start"}}, + &workflows.CallMicroflowTask{ + BaseWorkflowActivity: workflows.BaseWorkflowActivity{Name: "call", Caption: "Call"}, + IsAgent: agent, + Microflow: "MyFirstModule.InvokeAgent", + }, + &workflows.EndWorkflowActivity{BaseWorkflowActivity: workflows.BaseWorkflowActivity{Name: "End"}}, + }}, + } + } + err = b.CreateWorkflow(workflow("ZzAgent", true)) + if err == nil || !strings.Contains(err.Error(), "Workflows$AIAgentTaskActivity (Mendix 11.9.0+)") { + t.Errorf("an agent task on 11.6.6 = %v, want it refused naming the type", err) + } + if err := b.CreateWorkflow(workflow("ZzPlain", false)); err != nil { + t.Errorf("control: a plain microflow call on 11.6.6 is refused: %v", err) + } +} diff --git a/mdl/backend/source_files.go b/mdl/backend/source_files.go new file mode 100644 index 0000000000..c48bfe8fe6 --- /dev/null +++ b/mdl/backend/source_files.go @@ -0,0 +1,33 @@ +// SPDX-License-Identifier: Apache-2.0 + +package backend + +// SourceFileChange is a file or folder beside the model — in the project +// directory, but not the .mpr or mprcontents/ — that a write created, changed, +// moved or removed. +// +// It exists because the model is not the whole project. A Java or JavaScript +// action's body lives in javasource/ or javascriptsource/, not in its unit, so +// a workflow that commits "the model" leaves it behind, and a clone gets an +// action with no logic (ako/mxcli#1618, mendixlabs/mxcli#1634). The writer is +// the only code that knows what it touched; it records each change at the +// moment it touches disk, and `exec` names them. +type SourceFileChange struct { + // Path is project-relative and slash-separated, e.g. + // "javasource/probe/actions/JA_Hello.java". A folder ends in "/". + Path string + // From is the path the file was moved from, for a rename; empty otherwise. + From string + // Removed is set when the file or folder was deleted. + Removed bool + // What names the content for a reader, e.g. "Java action body". + What string +} + +// SourceFileReporter is implemented by backends that write files beside the +// model and can say which. A backend that writes none need not implement it. +type SourceFileReporter interface { + // TakeSourceFileChanges returns the changes made since the previous call, + // in the order they were made, and forgets them. + TakeSourceFileChanges() []SourceFileChange +} diff --git a/mdl/executor/catalog_source_unit_reads_pedapp_test.go b/mdl/executor/catalog_source_unit_reads_pedapp_test.go new file mode 100644 index 0000000000..7b7315d6aa --- /dev/null +++ b/mdl/executor/catalog_source_unit_reads_pedapp_test.go @@ -0,0 +1,56 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// ako/mxcli#1789: `refresh catalog full source` describes every document, and +// each microflow describe found its microflow — and every flow it calls, for +// the void-call warning — by listing every microflow of the project from disk. +// One refresh of a 4,753-unit project opened 7,532,649 unit files: minutes on +// macOS, hours on Windows, where Defender scans each open. The build now holds +// the units it reads in memory, and a describe reads the document it was +// pinned to and the microflows it calls by ID rather than by listing. + +func TestSourceCatalogBuild_ReadsUnitFilesLinearly(t *testing.T) { + exec, out, dir := openPedAppCopy(t) + // Every microflow calls the one before it, so each describe also resolves + // a called flow: both of the issue's lookups are exercised. + const n = 40 + if err := agreeExec(t, exec, redeclareChain(n, "create")); err != nil { + t.Fatalf("setup: %v\n%s", err, out.String()) + } + units := 0 + _ = filepath.Walk(filepath.Join(dir, "mprcontents"), func(p string, info os.FileInfo, err error) error { + if err == nil && !info.IsDir() && strings.HasSuffix(p, ".mxunit") { + units++ + } + return nil + }) + + before := unitFileReads(t, exec) + if err := agreeExec(t, exec, "refresh catalog full source force;"); err != nil { + t.Fatalf("source build: %v\n%s", err, out.String()) + } + // Each unit file is read from disk at most once per build, however often + // the describes look a document up. Before the fix this build read 11,017 + // files of these 409. + if reads := unitFileReads(t, exec) - before; reads > int64(units) { + t.Errorf("a source catalog build read %d unit files; the project has %d", reads, units) + } + + // The narrowing must not change the text: the pinned describe still + // renders the body, call included. + out.Reset() + if err := agreeExec(t, exec, "select SourceText from catalog.source where QualifiedName = 'MyFirstModule.MF_Chain040';"); err != nil { + t.Fatalf("query source: %v\n%s", err, out.String()) + } + if !strings.Contains(out.String(), "call microflow MyFirstModule.MF_Chain039") { + t.Errorf("the source row of MF_Chain040 lacks its call:\n%s", out.String()) + } +} diff --git a/mdl/executor/check_unit_reads_pedapp_test.go b/mdl/executor/check_unit_reads_pedapp_test.go index 92dce85da6..00b077a46b 100644 --- a/mdl/executor/check_unit_reads_pedapp_test.go +++ b/mdl/executor/check_unit_reads_pedapp_test.go @@ -70,12 +70,15 @@ func TestCheckPasses_ReadEachUnitFileOnce(t *testing.T) { t.Fatalf("parse: %v", errs[0]) } - // Control: the verdict pass without the cache reads the project's - // microflows over and over — more files than the project has. + // Control: the verdict pass without the cache reads the re-declared flows + // over and over. It used to read more files than the project has; since + // the unit index names flows without opening them (ako/mxcli#1791) it + // reads only flows it looks up, but each of them several times — 103 + // reads for these 12 flows, against 12 with the cache. before := unitFileReads(t, exec) CheckFlowVerdicts(exec.newExecContext(context.Background()), prog) - if reads := unitFileReads(t, exec) - before; reads <= int64(units) { - t.Fatalf("control: the uncached verdict pass read %d files of %d; the bound below would prove nothing", reads, units) + if reads := unitFileReads(t, exec) - before; reads <= 2*n { + t.Fatalf("control: the uncached verdict pass read %d files for %d flows; the bound below would prove nothing", reads, n) } passes := []struct { @@ -93,6 +96,9 @@ func TestCheckPasses_ReadEachUnitFileOnce(t *testing.T) { if reads := unitFileReads(t, exec) - before; reads > int64(units) { t.Errorf("%s read %d unit files for %d re-declared flows; the project has %d", p.name, reads, n, units) } + if reads := unitFileReads(t, exec) - before; p.name == "CheckFlowVerdicts" && reads > n { + t.Errorf("%s read %d unit files for %d re-declared flows; each needs reading once", p.name, reads, n) + } } before = unitFileReads(t, exec) diff --git a/mdl/executor/cmd_alter_flow.go b/mdl/executor/cmd_alter_flow.go index 1205acecfa..335587e73d 100644 --- a/mdl/executor/cmd_alter_flow.go +++ b/mdl/executor/cmd_alter_flow.go @@ -14,6 +14,7 @@ import ( mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" "github.com/mendixlabs/mxcli/mdl/types" "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/javaactions" "github.com/mendixlabs/mxcli/sdk/microflows" ) @@ -988,14 +989,15 @@ func (a *alterFlowContext) storedVariables(ctx *ExecContext) (varTypes, declared add(p.Name, p.Type) } // What is typed from another variable — a retrieve over an association - // from its start, a loop's iterator from its list — is typed once the - // rest are, below. + // from its start, a loop's iterator or a list operation's output from its + // list — is typed once the rest are, below; so is a Java action's result, + // which a list argument can type and another origin can contradict. deps := storedDependents{} fb := &flowBuilder{backend: ctx.Backend} for _, c := range a.cands { if loop, ok := c.Object.(*microflows.LoopedActivity); ok { if src, ok := loop.LoopSource.(*microflows.IterableList); ok { - deps.add(src.VariableName, storedDependent{from: src.ListVariableName}) + deps.add(src.VariableName, storedDependent{how: dependsOnList, list: listResultElement, from: src.ListVariableName}) } continue } @@ -1020,7 +1022,7 @@ func (a *alterFlowContext) storedVariables(ctx *ExecContext) (varTypes, declared // and is typed below (ako/mxcli#1769). if src, ok := x.Source.(*microflows.AssociationRetrieveSource); ok { declared[c.OutputVariable] = "Unknown" - deps.add(c.OutputVariable, storedDependent{from: src.StartVariable, assoc: src.AssociationQualifiedName}) + deps.add(c.OutputVariable, storedDependent{how: dependsOnAssociation, from: src.StartVariable, assoc: src.AssociationQualifiedName}) break } src, ok := x.Source.(*microflows.DatabaseRetrieveSource) @@ -1051,6 +1053,34 @@ func (a *alterFlowContext) storedVariables(ctx *ExecContext) (varTypes, declared default: declared[c.OutputVariable] = "Unknown" } + case *microflows.ListOperationAction: + // The output holds what the operation keeps of its input list, + // as the full build types it (ako/mxcli#1777); a contains or an + // equals holds a Boolean. + declared[c.OutputVariable] = "Unknown" + if r := listOperationResult(x.Operation); r != listResultNone { + deps.add(c.OutputVariable, storedDependent{how: dependsOnList, list: r, from: listOperationInput(x.Operation)}) + } + case *microflows.AggregateListAction: + declared[c.OutputVariable] = aggregateResultType(x.Function, x.ReduceReturnType) + case *microflows.JavaActionCallAction: + // The result has the action's declared return type, read the + // way the full build reads it (ako/mxcli#1777); a list of a type + // parameter takes the list an argument holds. + declared[c.OutputVariable] = "Unknown" + var ja *javaactions.JavaAction + if fb.backend != nil && x.JavaAction != "" { + ja, _ = fb.backend.ReadJavaActionByName(x.JavaAction) + } + switch { + case ja == nil: + case returnsTypeParameterList(ja): + deps.add(c.OutputVariable, storedDependent{how: dependsOnFirstList, from: strings.Join(storedArgVariables(x), ",")}) + default: + if t := javaActionResultType(ja, nil, nil); t != "" { + deps.add(c.OutputVariable, storedDependent{how: dependsOnNothing, typ: t}) + } + } default: declared[c.OutputVariable] = "Unknown" } @@ -1059,18 +1089,66 @@ func (a *alterFlowContext) storedVariables(ctx *ExecContext) (varTypes, declared return varTypes, declared } -// storedDependent is a stored variable whose type follows from another's: the -// output of a retrieve over assoc from the variable from, or — assoc empty — -// the iterator of a loop over the list from. +// storedArgVariable is an argument a stored Java action call passes as a +// variable as it stands: `$Lines`. +var storedArgVariable = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)$`) + +// storedArgVariables are the variables a stored Java action call passes as +// arguments as they stand, in the order it stores them: what +// javaActionArgVariables is to the statement the full build reads. +func storedArgVariables(x *microflows.JavaActionCallAction) []string { + var vars []string + for _, m := range x.ParameterMappings { + if v, ok := m.Value.(*microflows.BasicCodeActionParameterValue); ok { + if sub := storedArgVariable.FindStringSubmatch(strings.TrimSpace(v.Argument)); sub != nil { + vars = append(vars, sub[1]) + } + } + } + return vars +} + +// storedDependent is a stored variable whose type follows from another's, or +// is one a different origin of the same name can contradict; how says which. type storedDependent struct { - from, assoc string + how storedDependence + // from is the variable it follows from; for dependsOnFirstList, the + // variables a Java action call passes, comma-separated. + from string + // assoc is the association a retrieve follows (dependsOnAssociation). + assoc string + // list is what it holds of the list from holds (dependsOnList). + list listResult + // typ is its type outright (dependsOnNothing). + typ string } +// storedDependence is how a stored dependent's type follows. +type storedDependence int + +const ( + // dependsOnList: from a list, by list — a loop's iterator holds an + // element of the list it walks, a list operation's output the list or + // an element of it (listOperationResult). + dependsOnList storedDependence = iota + // dependsOnAssociation: the output of a retrieve over assoc from the + // variable from (flowBuilder.associationRetrieveType). + dependsOnAssociation + // dependsOnFirstList: the result of a Java action returning a list of + // its type parameter, the list the first of from that holds one holds + // (javaActionResultType). + dependsOnFirstList + // dependsOnNothing: typ, a Java action's declared return type. + dependsOnNothing +) + // storedDependents are the stored flow's dependent variables by name. A name // two of them give different origins is nil: which one a fragment means -// depends on where it goes (two loops can share an iterator name), and a guess -// would qualify a member with the wrong entity. It stays untyped and the -// member is refused. +// depends on where it goes (two loops can share an iterator name, two loop +// bodies a list operation's output), and a guess would qualify a member with +// the wrong entity. It stays untyped and the member is refused. Two origins +// that say the same — a head and a find of one list, two calls returning one +// entity — are one. type storedDependents map[string]*storedDependent func (d storedDependents) add(name string, dep storedDependent) { @@ -1100,19 +1178,7 @@ func (d storedDependents) typeAll(fb *flowBuilder, varTypes, declared map[string if _, typed := varTypes[name]; typed { continue } - from, ok := varTypes[dep.from] - if !ok { - continue - } - t := "" - if dep.assoc == "" { - if elem, isList := strings.CutPrefix(from, "List of "); isList { - t = elem - } - } else { - t = fb.associationRetrieveType(from, dep.assoc) - } - if t != "" { + if t := d.typeOf(fb, dep, varTypes); t != "" { varTypes[name] = t delete(declared, name) progress = true @@ -1121,6 +1187,34 @@ func (d storedDependents) typeAll(fb *flowBuilder, varTypes, declared map[string } } +// typeOf is the type dep gives its variable with varTypes as typed so far, +// through the full build's own typing; "" while what it follows from is not +// typed, or when it holds no entity. +func (d storedDependents) typeOf(fb *flowBuilder, dep *storedDependent, varTypes map[string]string) string { + switch dep.how { + case dependsOnNothing: + return dep.typ + case dependsOnFirstList: + // The first argument that holds a list, once each of them that + // another dependent may still type is typed. + args := strings.Split(dep.from, ",") + for _, v := range args { + if _, typed := varTypes[v]; !typed && d[v] != nil { + return "" + } + } + return firstListType(args, varTypes) + } + from, ok := varTypes[dep.from] + if !ok { + return "" + } + if dep.how == dependsOnAssociation { + return fb.associationRetrieveType(from, dep.assoc) + } + return dep.list.typeFrom(from) +} + // systemVariables are in scope everywhere they exist at all; the platform // reports a misuse (a $latestError outside an error handler) itself. var systemVariables = map[string]bool{ diff --git a/mdl/executor/cmd_catalog.go b/mdl/executor/cmd_catalog.go index 6362c9e763..26d39e6cec 100644 --- a/mdl/executor/cmd_catalog.go +++ b/mdl/executor/cmd_catalog.go @@ -482,6 +482,12 @@ func buildCatalog(ctx *ExecContext, full, isSource, communities bool, resolution if isSource || communities { full = true // both imply full (they read refs/source built in full mode) } + // A build writes nothing to the project, and its passes read the same + // documents again and again — the source describes list whole document + // types per document. Each unit file is read from disk once (#1789): an + // open is cheap on macOS and expensive where every one is scanned + // (Windows Defender), which made one refresh take hours there. + defer ctx.CacheUnitReads()() if carry, res := carryGraphAnalysis(cachedCatalogInfo(ctx), full, communities); carry { communities, resolution = true, res diff --git a/mdl/executor/cmd_microflows_builder_actions.go b/mdl/executor/cmd_microflows_builder_actions.go index 55dd0c3c8b..403ebb3270 100644 --- a/mdl/executor/cmd_microflows_builder_actions.go +++ b/mdl/executor/cmd_microflows_builder_actions.go @@ -1941,21 +1941,10 @@ func (fb *flowBuilder) addListOperationAction(s *ast.ListOperationStmt) model.ID ErrorHandlingType: fb.ehType(s.ErrorHandling), } - // Track output variable type for operations that preserve/produce list types + // The output holds what the operation keeps of its input list. if fb.varTypes != nil && s.OutputVariable != "" && s.InputVariable != "" { - inputType := fb.varTypes[s.InputVariable] - switch s.Operation { - case ast.ListOpFilter, ast.ListOpSort, ast.ListOpTail, ast.ListOpUnion, ast.ListOpIntersect, ast.ListOpSubtract, ast.ListOpRange: - // These operations preserve the list type - if inputType != "" { - fb.varTypes[s.OutputVariable] = inputType - } - case ast.ListOpHead, ast.ListOpFind: - // These return a single element (remove "List of " prefix) - if after, ok := strings.CutPrefix(inputType, "List of "); ok { - fb.varTypes[s.OutputVariable] = after - } - // CONTAINS and EQUALS return Boolean, no need to track + if t := listOperationResult(operation).typeFrom(fb.varTypes[s.InputVariable]); t != "" { + fb.varTypes[s.OutputVariable] = t } } @@ -1977,6 +1966,80 @@ func (fb *flowBuilder) addListOperationAction(s *ast.ListOperationStmt) model.ID return activity.ID } +// listResult is what a list operation's output holds, as against the list it +// reads. +type listResult int + +const ( + listResultNone listResult = iota // a Boolean: contains, equals + listResultList // the list itself: filter, sort, tail, union, intersect, subtract, range + listResultElement // one object of it: head, find +) + +// listOperationResult says what op's output holds. It reads the operation, +// not the statement, so a stored list operation's output can be typed the way +// the full build (addListOperationAction) types it (ako/mxcli#1777). +func listOperationResult(op microflows.ListOperation) listResult { + switch op.(type) { + case *microflows.FilterOperation, *microflows.FilterByAttributeOperation, *microflows.SortOperation, + *microflows.TailOperation, *microflows.UnionOperation, *microflows.IntersectOperation, + *microflows.SubtractOperation, *microflows.ListRangeOperation: + return listResultList + case *microflows.HeadOperation, *microflows.FindOperation, *microflows.FindByAttributeOperation: + return listResultElement + } + return listResultNone +} + +// typeFrom is the type an output holding r has when the list it reads has +// inputType ("List of Module.Entity"); "" when that is not known or the +// output holds no entity. +func (r listResult) typeFrom(inputType string) string { + switch r { + case listResultList: + return inputType + case listResultElement: + if elem, ok := strings.CutPrefix(inputType, "List of "); ok { + return elem + } + } + return "" +} + +// listOperationInput is the list op reads: the first of the two a union, +// intersect, subtract or equals reads, whose type the output takes. +func listOperationInput(op microflows.ListOperation) string { + switch o := op.(type) { + case *microflows.HeadOperation: + return o.ListVariable + case *microflows.TailOperation: + return o.ListVariable + case *microflows.FindOperation: + return o.ListVariable + case *microflows.FilterOperation: + return o.ListVariable + case *microflows.FindByAttributeOperation: + return o.ListVariable + case *microflows.FilterByAttributeOperation: + return o.ListVariable + case *microflows.SortOperation: + return o.ListVariable + case *microflows.ListRangeOperation: + return o.ListVariable + case *microflows.ContainsOperation: + return o.ListVariable + case *microflows.UnionOperation: + return o.ListVariable1 + case *microflows.IntersectOperation: + return o.ListVariable1 + case *microflows.SubtractOperation: + return o.ListVariable1 + case *microflows.EqualsOperation: + return o.ListVariable1 + } + return "" +} + func (fb *flowBuilder) listAttributeOperation(s *ast.ListOperationStmt, filter bool) microflows.ListOperation { // `find $L where …` / `filter $L where …` is always the by-expression // operation, even when the expression happens to read `Member = value`. @@ -2117,23 +2180,9 @@ func (fb *flowBuilder) addAggregateListAction(s *ast.AggregateListStmt) model.ID } // Declare the result so a later SET on it is accepted, as the validate path - // already does (mendixlabs/mxcli#1413). SUM/MIN/MAX follow the attribute's - // type, which is not resolved here, so they stay Unknown. + // already does (mendixlabs/mxcli#1413). if fb.declaredVars != nil && s.OutputVariable != "" { - resultType := "Unknown" - switch s.Operation { - case ast.AggregateCount: - resultType = "Integer" - case ast.AggregateAverage: - resultType = "Decimal" - case ast.AggregateAll, ast.AggregateAny: - resultType = "Boolean" - case ast.AggregateReduce: - if s.ReturnType != nil { - resultType = s.ReturnType.Kind.String() - } - } - fb.declaredVars[s.OutputVariable] = resultType + fb.declaredVars[s.OutputVariable] = aggregateResultType(action.Function, action.ReduceReturnType) } activity := µflows.ActionActivity{ @@ -2154,6 +2203,28 @@ func (fb *flowBuilder) addAggregateListAction(s *ast.AggregateListStmt) model.ID return activity.ID } +// aggregateResultType is the primitive an aggregate's output holds: Integer +// for a count, Decimal for an average, Boolean for all and any, a reduce's +// return type. A sum, minimum or maximum follows the attribute's type, which +// is not resolved here, so it is Unknown. It reads the action, not the +// statement, so a stored aggregate's output can be typed the way the full +// build (addAggregateListAction) types it (ako/mxcli#1777). +func aggregateResultType(fn microflows.AggregateFunction, reduceReturnType microflows.DataType) string { + switch fn { + case microflows.AggregateFunctionCount: + return "Integer" + case microflows.AggregateFunctionAverage: + return "Decimal" + case microflows.AggregateFunctionAll, microflows.AggregateFunctionAny: + return "Boolean" + case microflows.AggregateFunctionReduce: + if reduceReturnType != nil { + return reduceReturnType.GetTypeName() + } + } + return "Unknown" +} + // addCreateListAction creates a CREATE LIST OF statement. func (fb *flowBuilder) addCreateListAction(s *ast.CreateListStmt) model.ID { entityQN := "" @@ -2838,10 +2909,14 @@ func (fb *flowBuilder) lookupAssociation(moduleName, assocName string) *assocLoo // other end, by associationRetrieveYieldsList — the answer addRetrieveAction // and check --references take. Mind the inversion: an association's parent is // its FROM end and its child the TO end, so a retrieve from the TO end yields -// the FROM entity. "" when it cannot be established: the association or the -// start unknown, the start a list, or the start no end of it — or, over a -// reference, both ends (a self-reference), where which end it starts from is -// not known. +// the FROM entity. A self-reference is followed from its FROM end: the stored +// retrieve records no direction, and Mendix reads it that way — the object a +// reference holds, the list a reference set holds (measured on 11.13.0, +// ako/mxcli#1777: a change on it builds, a count of it is CE0013 "Input +// variable 'P' must be of type 'List'."). "" when it cannot be established: the +// association or the start unknown, the start a list, the start no end of it, +// or a specialization of both ends of an association between two entities, +// where which end it starts from is not known. func (fb *flowBuilder) associationRetrieveType(startType, assocQN string) string { if startType == "" || strings.HasPrefix(startType, "List of ") { return "" @@ -2859,12 +2934,10 @@ func (fb *flowBuilder) associationRetrieveType(startType, assocQN string) string fromFrom := fb.entityIsSubtypeOf(startType, info.parentEntityQN) switch { case fromTo && fromFrom: - // Either end: a reference set to itself is a list of it whichever - // end it starts from, and nothing else is known. - if refSet && info.parentEntityQN == info.childEntityQN { - return "List of " + info.parentEntityQN + if info.parentEntityQN != info.childEntityQN { + return "" } - return "" + fromTo = false // a self-reference: from the FROM end case !fromTo && !fromFrom: return "" } diff --git a/mdl/executor/cmd_microflows_builder_calls.go b/mdl/executor/cmd_microflows_builder_calls.go index bbca7c291c..576205447f 100644 --- a/mdl/executor/cmd_microflows_builder_calls.go +++ b/mdl/executor/cmd_microflows_builder_calls.go @@ -442,11 +442,9 @@ func (fb *flowBuilder) addCallJavaActionAction(s *ast.CallJavaActionStmt) model. ResultVariableName: s.OutputVariable, UseReturnVariable: s.OutputVariable != "", } - if s.OutputVariable != "" && jaDef != nil && fb.varTypes != nil { - if varType := javaActionReturnVarType(jaDef.ReturnType); varType != "" { + if s.OutputVariable != "" && fb.varTypes != nil { + if varType := javaActionResultType(jaDef, javaActionArgVariables(s), fb.varTypes); varType != "" { fb.varTypes[s.OutputVariable] = varType - } else if inferred := fb.inferGenericJavaActionReturnType(jaDef, s); inferred != "" { - fb.varTypes[s.OutputVariable] = inferred } } @@ -485,34 +483,58 @@ func javaActionReturnVarType(returnType javaactions.CodeActionReturnType) string return "" } -// inferGenericJavaActionReturnType infers the element type of a generic -// `ListType{Entity: ""}` Java action return by inspecting the caller's -// variable-typed arguments. When the action parameters don't determine an -// element type, the function returns "" and the caller records the declaration -// as Unknown. -func (fb *flowBuilder) inferGenericJavaActionReturnType(jaDef *javaactions.JavaAction, s *ast.CallJavaActionStmt) string { - if jaDef == nil || fb.varTypes == nil || s == nil { +// javaActionResultType is the type a call of jaDef binds its result to, +// "Module.Entity" or "List of Module.Entity": the declared return type, or — +// for a list of a type parameter — the list the first of argVars that holds +// one holds (varTypes), argVars being the variables the call passes as they +// stand. "" when the result holds no entity or that is not known, and the +// caller records the declaration as Unknown. It reads the definition and the +// argument variables, not the statement, so a stored call's result can be +// typed the way the full build (addCallJavaActionAction) types it +// (ako/mxcli#1777). +func javaActionResultType(jaDef *javaactions.JavaAction, argVars []string, varTypes map[string]string) string { + if jaDef == nil { return "" } - // ListType is always stored as a pointer by the parser; there is no value - // form in the SDK. Only the generic (Entity == "") case reaches the - // variable-type lookup below. - t, ok := jaDef.ReturnType.(*javaactions.ListType) - if !ok || t.Entity != "" { + if varType := javaActionReturnVarType(jaDef.ReturnType); varType != "" { + return varType + } + if !returnsTypeParameterList(jaDef) { return "" } - for _, arg := range s.Arguments { - varExpr, ok := arg.Value.(*ast.VariableExpr) - if !ok { - continue - } - if typ := fb.varTypes[varExpr.Name]; strings.HasPrefix(typ, "List of ") { + return firstListType(argVars, varTypes) +} + +// returnsTypeParameterList reports whether jaDef returns a list of one of its +// type parameters, whose entity is the one a list argument holds. ListType is +// always stored as a pointer by the parser; there is no value form in the SDK. +func returnsTypeParameterList(jaDef *javaactions.JavaAction) bool { + t, ok := jaDef.ReturnType.(*javaactions.ListType) + return ok && t.Entity == "" +} + +// firstListType is the type of the first of vars that holds a list, or "". +func firstListType(vars []string, varTypes map[string]string) string { + for _, v := range vars { + if typ := varTypes[v]; strings.HasPrefix(typ, "List of ") { return typ } } return "" } +// javaActionArgVariables are the variables s passes as arguments as they +// stand (`P = $Lines`), in the order it states them. +func javaActionArgVariables(s *ast.CallJavaActionStmt) []string { + var vars []string + for _, arg := range s.Arguments { + if varExpr, ok := arg.Value.(*ast.VariableExpr); ok { + vars = append(vars, varExpr.Name) + } + } + return vars +} + // addCallJavaScriptActionAction creates a CALL JAVASCRIPT ACTION statement. func (fb *flowBuilder) addCallJavaScriptActionAction(s *ast.CallJavaScriptActionStmt) model.ID { actionQN := s.ActionName.Module + "." + s.ActionName.Name diff --git a/mdl/executor/cmd_microflows_show.go b/mdl/executor/cmd_microflows_show.go index 3f01780377..8832b8a955 100644 --- a/mdl/executor/cmd_microflows_show.go +++ b/mdl/executor/cmd_microflows_show.go @@ -33,8 +33,7 @@ func listMicroflows(ctx *ExecContext, moduleName string) error { } } - // Get all microflows - microflows, err := ctx.Backend.ListMicroflows() + microflows, err := microflowsInModule(ctx, h, moduleName) if err != nil { return mdlerrors.NewBackend("list microflows", err) } @@ -229,8 +228,13 @@ func describeMicroflowMode(ctx *ExecContext, name ast.QualifiedName, opts descri microflowNames := getMicroflowNames(ctx, h) // Find the microflow: getMicroflowNames above already named every - // microflow, so only the ones with this name are decoded. - allMicroflows, err := microflowsNamed(ctx, name.Name) + // microflow, so only the ones with this name are decoded. A describe the + // catalog's source build pinned to a document reads that document alone: + // the name lookup reads every microflow, once per describe (#1789). + allMicroflows, err := pinnedMicroflow(ctx, h, name) + if err == nil && allMicroflows == nil { + allMicroflows, err = microflowsNamed(ctx, name.Name) + } if err != nil { return mdlerrors.NewBackend("list microflows", err) } @@ -1698,6 +1702,9 @@ func microflowBodyWarnings( var voids *voidCodeActions if ctx != nil && ctx.Backend != nil { voids = newVoidCodeActions(nil, func() backend.FullBackend { return ctx.Backend }) + if h, err := getHierarchy(ctx); err == nil && h != nil { + voids.microflowNames = func() map[model.ID]string { return getMicroflowNames(ctx, h) } + } } out = append(out, duplicateOutputVariableWarnings(mf.ObjectCollection, voids.actionIsVoidCall)...) out = append(out, irreducibleGraphWarnings(mf.ObjectCollection, declaredCrossed)...) diff --git a/mdl/executor/cmd_modules.go b/mdl/executor/cmd_modules.go index e968d9fb82..222ac1efb5 100644 --- a/mdl/executor/cmd_modules.go +++ b/mdl/executor/cmd_modules.go @@ -14,6 +14,7 @@ import ( mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" "github.com/mendixlabs/mxcli/mdl/types" "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/modelsdk/meta" "github.com/mendixlabs/mxcli/sdk/domainmodel" "github.com/mendixlabs/mxcli/sdk/security" ) @@ -467,9 +468,28 @@ func listModules(ctx *ExecContext) error { bizEventCounts := make(map[model.ID]int) extDbCounts := make(map[model.ID]int) - // Count from units by type prefix (efficient single pass) + // Count from the unit types in one pass. Every count but entities is a + // count of units of one type, which the unit listing knows without opening + // a document; listing each type through its decoder read every page, + // snippet, flow, constant and Java action of the project (ako/mxcli#1791). for _, u := range units { modID := h.FindModuleID(u.ContainerID) + switch u.Type { + case "Enumerations$Enumeration": + enumCounts[modID]++ + case "Forms$Page": + pageCounts[modID]++ + case "Forms$Snippet": + snippetCounts[modID]++ + case "Microflows$Microflow": + microflowCounts[modID]++ + case "Microflows$Nanoflow": + nanoflowCounts[modID]++ + case "Constants$Constant": + constantCounts[modID]++ + case "JavaActions$JavaAction": + javaActionCounts[modID]++ + } switch { case strings.HasPrefix(u.Type, "Rest$PublishedRestService"): pubRestCounts[modID]++ @@ -486,7 +506,17 @@ func listModules(ctx *ExecContext) error { } } - // Count entities from domain models + // The System module's enumerations and Java actions are platform + // built-ins with no unit; the backend's listings add them, so count them + // the same way. + for _, e := range meta.BuildSystemEnumerations() { + enumCounts[h.FindModuleID(e.ContainerID)]++ + } + for _, ja := range meta.BuildSystemJavaActions() { + javaActionCounts[h.FindModuleID(ja.ContainerID)]++ + } + + // Count entities from domain models: one unit per module holds them all. if dms, err := ctx.Backend.ListDomainModels(); err == nil { for _, dm := range dms { modID := h.FindModuleID(dm.ContainerID) @@ -494,62 +524,6 @@ func listModules(ctx *ExecContext) error { } } - // Count enumerations - if enums, err := ctx.Backend.ListEnumerations(); err == nil { - for _, enum := range enums { - modID := h.FindModuleID(enum.ContainerID) - enumCounts[modID]++ - } - } - - // Count pages - if pages, err := ctx.Backend.ListPages(); err == nil { - for _, p := range pages { - modID := h.FindModuleID(p.ContainerID) - pageCounts[modID]++ - } - } - - // Count snippets - if snippets, err := ctx.Backend.ListSnippets(); err == nil { - for _, s := range snippets { - modID := h.FindModuleID(s.ContainerID) - snippetCounts[modID]++ - } - } - - // Count microflows - if mfs, err := ctx.Backend.ListMicroflows(); err == nil { - for _, mf := range mfs { - modID := h.FindModuleID(mf.ContainerID) - microflowCounts[modID]++ - } - } - - // Count nanoflows - if nfs, err := ctx.Backend.ListNanoflows(); err == nil { - for _, nf := range nfs { - modID := h.FindModuleID(nf.ContainerID) - nanoflowCounts[modID]++ - } - } - - // Count constants - if constants, err := ctx.Backend.ListConstants(); err == nil { - for _, c := range constants { - modID := h.FindModuleID(c.ContainerID) - constantCounts[modID]++ - } - } - - // Count Java actions - if jas, err := ctx.Backend.ListJavaActions(); err == nil { - for _, ja := range jas { - modID := h.FindModuleID(ja.ContainerID) - javaActionCounts[modID]++ - } - } - // Sort modules alphabetically by name sort.Slice(modules, func(i, j int) bool { return strings.ToLower(modules[i].Name) < strings.ToLower(modules[j].Name) diff --git a/mdl/executor/cmd_move.go b/mdl/executor/cmd_move.go index ff2049dcdf..dcad689226 100644 --- a/mdl/executor/cmd_move.go +++ b/mdl/executor/cmd_move.go @@ -14,7 +14,7 @@ import ( ) // execMove handles MOVE PAGE/MICROFLOW/SNIPPET/NANOFLOW/ENTITY/ENUMERATION statements. -func execMove(ctx *ExecContext, s *ast.MoveStmt) error { +func execMove(ctx *ExecContext, s *ast.MoveStmt) (moveErr error) { if !ctx.ConnectedForWrite() { return mdlerrors.NewNotConnected() } @@ -66,6 +66,27 @@ func execMove(ctx *ExecContext, s *ast.MoveStmt) error { // ("microflow not found" on a second move of the same document). defer invalidateHierarchy(ctx) + // A Java or JavaScript action's source lives under its module's directory, + // and mxbuild looks for it under the module the model now names: left + // behind, the build succeeded with a stub that throws where the app calls + // the action (measured on 11.13.0). The source moves first, so an occupied + // destination refuses the statement before the model changes, and moves + // back if the rest of the statement fails. exec names the move with the + // other files beside the model a statement touched. + if isCrossModuleMove { + moved, err := moveActionSource(ctx, s.DocumentType, s.Name.Module, targetModule.Name, s.Name.Name) + if err != nil { + return err + } + if moved != "" { + defer func() { + if moveErr != nil { + _, _ = moveActionSource(ctx, s.DocumentType, targetModule.Name, s.Name.Module, s.Name.Name) + } + }() + } + } + // Execute move based on document type switch s.DocumentType { case ast.DocumentTypePage: @@ -133,6 +154,28 @@ func execMove(ctx *ExecContext, s *ast.MoveStmt) error { return nil } +// moveActionSource moves a Java or JavaScript action's source file from one +// module's directory to another's and returns its new project-relative path, or +// "" when the document type has no source file or there was none to move. +func moveActionSource(ctx *ExecContext, docType ast.DocumentType, fromModule, toModule, name string) (string, error) { + var ( + moved string + err error + ) + switch docType { + case ast.DocumentTypeJavaAction: + moved, err = ctx.Backend.MoveJavaSourceFile(fromModule, toModule, name) + case ast.DocumentTypeJavaScriptAction: + moved, err = ctx.Backend.MoveJavaScriptSourceFile(fromModule, toModule, name) + default: + return "", nil + } + if err != nil { + return "", mdlerrors.NewBackend("move "+docType.CanonicalSpelling()+" source", err) + } + return moved, nil +} + // alreadyPlaced reports, as Unchanged, a move whose document already sits in // the target container. Anything it cannot establish — no such document, a // different doctype under that name — is left to the mover, which reports it. diff --git a/mdl/executor/cmd_move_action_source_test.go b/mdl/executor/cmd_move_action_source_test.go new file mode 100644 index 0000000000..40537adcb5 --- /dev/null +++ b/mdl/executor/cmd_move_action_source_test.go @@ -0,0 +1,122 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "errors" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/javaactions" +) + +// A cross-module MOVE JAVA ACTION reparented the document and left its source +// under the old module. mxbuild then built a stub that throws "Java action was +// not implemented" where the app calls the action, and compiled the author's +// code as an orphaned class (measured on 11.13.0). The move must take the source +// along — first, so an occupied destination refuses the statement before the +// model changes, and back again if the rest of the statement fails. + +type sourceMoveProbe struct { + calls [][3]string // from, to, action + unitMoved bool +} + +func javaActionMoveBackend(t *testing.T, probe *sourceMoveProbe, moveUnitErr, moveSourceErr error) (*mock.MockBackend, *ContainerHierarchy, *model.Module, *model.Module) { + t.Helper() + srcMod, dstMod := mkModule("JamA"), mkModule("JamB") + ja := &javaactions.JavaAction{Name: "JA_Hello"} + ja.ID = nextID("ja") + ja.ContainerID = srcMod.ID + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{srcMod, dstMod}, nil }, + ListJavaActionsFullFunc: func() ([]*javaactions.JavaAction, error) { return []*javaactions.JavaAction{ja}, nil }, + MoveJavaActionFunc: func(*javaactions.JavaAction) error { + if moveUnitErr != nil { + return moveUnitErr + } + probe.unitMoved = true + return nil + }, + MoveJavaSourceFileFunc: func(from, to, action string) (string, error) { + probe.calls = append(probe.calls, [3]string{from, to, action}) + if moveSourceErr != nil { + return "", moveSourceErr + } + return "javasource/" + to + "/actions/" + action + ".java", nil + }, + UpdateQualifiedNameInAllUnitsFunc: func(string, string) (int, error) { return 0, nil }, + } + h := mkHierarchy(srcMod, dstMod) + return mb, h, srcMod, dstMod +} + +func TestMoveJavaActionAcrossModulesMovesItsSource(t *testing.T) { + var probe sourceMoveProbe + mb, h, _, _ := javaActionMoveBackend(t, &probe, nil, nil) + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execMove(ctx, &ast.MoveStmt{ + DocumentType: ast.DocumentTypeJavaAction, + Name: ast.QualifiedName{Module: "JamA", Name: "JA_Hello"}, + TargetModule: "JamB", + })) + if !probe.unitMoved { + t.Fatal("the java action was not reparented") + } + if len(probe.calls) != 1 || probe.calls[0] != [3]string{"JamA", "JamB", "JA_Hello"} { + t.Fatalf("source moves = %v, want one move JamA -> JamB of JA_Hello", probe.calls) + } +} + +// An occupied destination is refused before the document moves. +func TestMoveJavaActionRefusedBeforeTheModelChanges(t *testing.T) { + var probe sourceMoveProbe + mb, h, _, _ := javaActionMoveBackend(t, &probe, nil, errors.New("javasource/jamb/actions/JA_Hello.java already exists")) + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + err := execMove(ctx, &ast.MoveStmt{ + DocumentType: ast.DocumentTypeJavaAction, + Name: ast.QualifiedName{Module: "JamA", Name: "JA_Hello"}, + TargetModule: "JamB", + }) + if err == nil { + t.Fatal("want the move refused") + } + if probe.unitMoved { + t.Error("the document moved although its source could not") + } +} + +// If the document cannot move, the source goes back where it was. +func TestMoveJavaActionPutsTheSourceBackOnFailure(t *testing.T) { + var probe sourceMoveProbe + mb, h, _, _ := javaActionMoveBackend(t, &probe, errors.New("disk full"), nil) + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + if err := execMove(ctx, &ast.MoveStmt{ + DocumentType: ast.DocumentTypeJavaAction, + Name: ast.QualifiedName{Module: "JamA", Name: "JA_Hello"}, + TargetModule: "JamB", + }); err == nil { + t.Fatal("want the move to fail") + } + want := [][3]string{{"JamA", "JamB", "JA_Hello"}, {"JamB", "JamA", "JA_Hello"}} + if len(probe.calls) != 2 || probe.calls[0] != want[0] || probe.calls[1] != want[1] { + t.Fatalf("source moves = %v, want %v", probe.calls, want) + } +} + +// A move within the module (into a folder) leaves the source where it is. +func TestMoveJavaActionWithinItsModuleLeavesTheSource(t *testing.T) { + var probe sourceMoveProbe + mb, h, _, _ := javaActionMoveBackend(t, &probe, nil, nil) + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execMove(ctx, &ast.MoveStmt{ + DocumentType: ast.DocumentTypeJavaAction, + Name: ast.QualifiedName{Module: "JamA", Name: "JA_Hello"}, + })) + if len(probe.calls) != 0 { + t.Errorf("source moved on a move within the module: %v", probe.calls) + } +} diff --git a/mdl/executor/cmd_odata.go b/mdl/executor/cmd_odata.go index 280dfb3c88..3435b1f369 100644 --- a/mdl/executor/cmd_odata.go +++ b/mdl/executor/cmd_odata.go @@ -4,8 +4,10 @@ package executor import ( "crypto/sha256" + "errors" "fmt" "io" + "io/fs" "net/http" "os" "path/filepath" @@ -153,7 +155,7 @@ func outputConsumedODataServiceMDL(ctx *ExecContext, svc *model.ConsumedODataSer props = append(props, fmt.Sprintf(" ODataVersion: %s", svc.ODataVersion)) } if svc.MetadataUrl != "" { - props = append(props, fmt.Sprintf(" MetadataUrl: %s", mdlQuote(ctx, svc.MetadataUrl))) + props = append(props, fmt.Sprintf(" MetadataUrl: %s", mdlQuote(ctx, describedMetadataURL(ctx, svc.MetadataUrl)))) } if svc.TimeoutExpression != "" { props = append(props, fmt.Sprintf(" Timeout: %s", svc.TimeoutExpression)) @@ -1033,12 +1035,14 @@ func createODataClient(ctx *ExecContext, stmt *ast.CreateODataClientStmt) error if stmt.ODataVersion != "" { svc.ODataVersion = stmt.ODataVersion } + fetchURL := "" if stmt.MetadataUrl != "" { - normalized, nerr := normalizeMetadataURL(ctx, stmt.MetadataUrl) + fetch, store, nerr := metadataURLToStore(ctx, stmt.MetadataUrl, svc.MetadataUrl) if nerr != nil { return fmt.Errorf("failed to normalize MetadataUrl: %w", nerr) } - svc.MetadataUrl = normalized + svc.MetadataUrl = store + fetchURL = fetch } if stmt.TimeoutExpression != "" { svc.TimeoutExpression = stmt.TimeoutExpression @@ -1113,7 +1117,7 @@ func createODataClient(ctx *ExecContext, stmt *ast.CreateODataClientStmt) error // (mxcli-formula1: F1LiveNowApi gained Trace and Stints, and // the only way out was DROP + recreate, which invalidates the // client ID the external entities point at). - refreshed := refreshCachedMetadata(ctx, svc, stmt) + refreshed := refreshCachedMetadata(ctx, svc, stmt, fetchURL) if err := ctx.Backend.UpdateConsumedODataService(svc); err != nil { return mdlerrors.NewBackend("update OData client", err) @@ -1277,20 +1281,125 @@ func contractSummary(metadata string) string { return fmt.Sprintf("%d entity types, %d actions", entityCount, len(doc.Actions)) } -// refreshCachedMetadata re-reads the contract at svc.MetadataUrl into -// svc.Metadata / svc.MetadataHash, reporting whether it actually changed. +// metadataURLToStore resolves a create-or-modify's MetadataUrl against the one +// the client already stores. fetch is the statement's value as this checkout's +// absolute URL, which the contract is read from; store is what the client +// keeps: the stored URL when it already names the statement's file, else fetch. +// +// A relative path is stored as the absolute file:// URL of whichever checkout +// ran the script, so without this the same unchanged script rewrote every +// client it touched when run from a checkout at another path (ako/mxcli#1499). +// The absolute form itself stays: mx check accepts a relative value, but it +// accepts a dead absolute one too, and nothing that can run here shows what +// Studio Pro's metadata editor makes of a relative one. +func metadataURLToStore(ctx *ExecContext, stmtURL, stored string) (fetch, store string, err error) { + fetch, err = normalizeMetadataURL(ctx, stmtURL) + if err != nil { + return "", "", err + } + if stored != "" && namesSameMetadataFile(ctx, stmtURL, fetch, stored) { + return fetch, stored, nil + } + return fetch, fetch, nil +} + +// namesSameMetadataFile reports whether stored, a MetadataUrl in the model, +// names the file the statement's stmtURL (resolved here to want) names. +// +// Beyond the plain cases (equal, or equal once normalized) this answers for a +// relative path inside the project, the form a script uses: a stored file URL +// that reaches the same project-relative path from another directory names +// this file in another checkout of the project, when that directory does not +// exist here (another machine, or a checkout since moved) or holds this +// project's .mpr (another worktree). A directory that exists here without it +// is somewhere else, and the statement moves the client off it. +func namesSameMetadataFile(ctx *ExecContext, stmtURL, want, stored string) bool { + if stored == want { + return true + } + if norm, err := normalizeMetadataURL(ctx, stored); err == nil && norm == want { + return true + } + if !isRelativeMetadataPath(stmtURL) || ctx == nil || ctx.MprPath == "" { + return false + } + projectDir, err := filepath.Abs(filepath.Dir(ctx.MprPath)) + if err != nil { + return false + } + rel, err := filepath.Rel(projectDir, pathutil.PathFromURL(want)) + if err != nil || !filepath.IsLocal(rel) { + return false + } + storedPath := filepath.ToSlash(pathutil.PathFromURL(stored)) + suffix := "/" + filepath.ToSlash(rel) + if storedPath == "" || !strings.HasSuffix(storedPath, suffix) { + return false + } + otherDir := filepath.FromSlash(strings.TrimSuffix(storedPath, suffix)) + if _, err := os.Stat(otherDir); err != nil { + return errors.Is(err, fs.ErrNotExist) + } + _, err = os.Stat(filepath.Join(otherDir, filepath.Base(ctx.MprPath))) + return err == nil +} + +// isRelativeMetadataPath reports whether a MetadataUrl as written is a path +// relative to the project, the one form whose stored URL depends on where the +// project is checked out. +func isRelativeMetadataPath(raw string) bool { + for _, scheme := range []string{"http://", "https://", "file://"} { + if strings.HasPrefix(raw, scheme) { + return false + } + } + return !filepath.IsAbs(raw) +} + +// describedMetadataURL is the MetadataUrl describe prints: a stored file URL +// inside the project directory as the project-relative path a script would +// write (ako/mxcli#1499), so a script generated from the model carries no +// machine path. Only when that path resolves back to exactly the stored URL, +// so executing the output leaves the client as it is; anything else as stored. +func describedMetadataURL(ctx *ExecContext, stored string) string { + path := pathutil.PathFromURL(stored) + if path == "" || ctx == nil || ctx.MprPath == "" { + return stored + } + projectDir, err := filepath.Abs(filepath.Dir(ctx.MprPath)) + if err != nil { + return stored + } + rel, err := filepath.Rel(projectDir, path) + if err != nil || !filepath.IsLocal(rel) { + return stored + } + relative := "./" + filepath.ToSlash(rel) + if back, err := normalizeMetadataURL(ctx, relative); err != nil || back != stored { + return stored + } + return relative +} + +// refreshCachedMetadata re-reads the contract into svc.Metadata / +// svc.MetadataHash, reporting whether it actually changed. It reads fetchURL, +// the statement's MetadataUrl as resolved in this checkout, or svc.MetadataUrl +// when the statement named none. // // A fetch that fails is a warning, not an error, and leaves the cached contract // in place: losing a contract that was merely unreachable is worse than serving // a stale one, and the caller has nothing else to fall back on. An unchanged // contract leaves svc untouched, so the write is elided and the statement // reports "Unchanged" rather than churning the document. -func refreshCachedMetadata(ctx *ExecContext, svc *model.ConsumedODataService, stmt *ast.CreateODataClientStmt) bool { - if svc == nil || svc.MetadataUrl == "" { +func refreshCachedMetadata(ctx *ExecContext, svc *model.ConsumedODataService, stmt *ast.CreateODataClientStmt, fetchURL string) bool { + if fetchURL == "" && svc != nil { + fetchURL = svc.MetadataUrl + } + if svc == nil || fetchURL == "" { return false } auth := metadataAuthFromStmt(ctx, stmt) - metadata, hash, err := fetchODataMetadata(svc.MetadataUrl, auth) + metadata, hash, err := fetchODataMetadata(fetchURL, auth) if err != nil { fmt.Fprintf(ctx.Output, "Warning: could not refresh $metadata: %v\n", err) for _, hint := range auth.hints() { diff --git a/mdl/executor/cmd_odata_metadata_url_checkout_test.go b/mdl/executor/cmd_odata_metadata_url_checkout_test.go new file mode 100644 index 0000000000..7dbcb42a48 --- /dev/null +++ b/mdl/executor/cmd_odata_metadata_url_checkout_test.go @@ -0,0 +1,209 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "os" + "path/filepath" + "reflect" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/model" +) + +// ako/mxcli#1499: a relative MetadataUrl is stored as the absolute file:// URL +// of whichever checkout ran the script. Re-executing the same unchanged script +// from a checkout at another path rewrote every client to its own path — 5 of +// the 20 writes a full re-exec of the reported project produced, on a run that +// should have written nothing: +// +// - MetadataUrl: 'file:///workspaces/mxcli-formula1/Formula1Frontend/contracts/f1-metadata.xml', +// + MetadataUrl: 'file:///home/user/mxcli-formula1/Formula1Frontend/contracts/f1-metadata.xml', +// +// The stored URL keeps its absolute file:// form (mx check is indifferent to +// the value, so nothing here can show Studio Pro accepts a relative one), and a +// statement whose relative path names the same project file leaves it alone. + +// checkoutProject lays out //app.mpr with contracts/f1.xml holding +// contract, and returns the project directory. +func checkoutProject(t *testing.T, root, name, contract string) string { + t.Helper() + dir := filepath.Join(root, name) + if err := os.MkdirAll(filepath.Join(dir, "contracts"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, "app.mpr"), nil, 0o600); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, "contracts", "f1.xml"), []byte(contract), 0o600); err != nil { + t.Fatal(err) + } + return dir +} + +func fileURL(path string) string { return "file://" + filepath.ToSlash(path) } + +// modifyFromCheckout builds the fixture in project dir `here`, with the client +// stored as another run left it (storedURL), runs the statement, and returns +// the service before the statement and the one handed to the backend. +func modifyFromCheckout(t *testing.T, here, storedURL, stmtURL string) (before, after model.ConsumedODataService, f *odataModifyFixture) { + t.Helper() + f = newODataModifyFixture(t, here, filepath.Join(here, "contracts", "f1.xml")) + f.svc.MetadataUrl = storedURL + before = *f.svc + f.modify(t, stmtURL) + return before, **f.updated, f +} + +// The reported case: the stored path is from a checkout on another machine, +// where this project sat somewhere that does not exist here. +func TestCreateOrModifyODataClient_KeepsMetadataUrlFromAnotherMachine(t *testing.T) { + root := t.TempDir() + here := checkoutProject(t, root, "home-user", edmxWith("Session")) + foreign := fileURL(filepath.Join(root, "workspaces", "Formula1Frontend", "contracts", "f1.xml")) + + before, after, _ := modifyFromCheckout(t, here, foreign, "./contracts/f1.xml") + + if after.MetadataUrl != foreign { + t.Errorf("an unchanged statement rewrote the client:\n- MetadataUrl: %q\n+ MetadataUrl: %q", foreign, after.MetadataUrl) + } + if !reflect.DeepEqual(after, before) { + t.Errorf("the client handed to the backend differs from the stored one, so the write is not elided:\nstored %+v\nnow %+v", before, after) + } +} + +// A second checkout on the same machine (a git worktree): the stored path +// exists, in another directory holding the same project. +func TestCreateOrModifyODataClient_KeepsMetadataUrlFromAnotherWorktree(t *testing.T) { + root := t.TempDir() + first := checkoutProject(t, root, "main", edmxWith("Session")) + here := checkoutProject(t, root, "worktree", edmxWith("Session")) + stored := fileURL(filepath.Join(first, "contracts", "f1.xml")) + + before, after, _ := modifyFromCheckout(t, here, stored, "./contracts/f1.xml") + + if !reflect.DeepEqual(after, before) { + t.Errorf("re-exec from a worktree rewrote the client:\n- MetadataUrl: %q\n+ MetadataUrl: %q", before.MetadataUrl, after.MetadataUrl) + } +} + +// Control: a stored path that only shares the tail, in a directory that exists +// here and is not a checkout of this project, is a different file. The +// statement moves the client onto the project's copy. +func TestCreateOrModifyODataClient_RewritesMetadataUrlNamingAnotherFile(t *testing.T) { + root := t.TempDir() + here := checkoutProject(t, root, "app", edmxWith("Session")) + shared := filepath.Join(root, "shared") + if err := os.MkdirAll(filepath.Join(shared, "contracts"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(shared, "contracts", "f1.xml"), []byte(edmxWith("Session")), 0o600); err != nil { + t.Fatal(err) + } + stored := fileURL(filepath.Join(shared, "contracts", "f1.xml")) + + _, after, _ := modifyFromCheckout(t, here, stored, "./contracts/f1.xml") + + if want := fileURL(filepath.Join(here, "contracts", "f1.xml")); after.MetadataUrl != want { + t.Errorf("MetadataUrl = %q, want the project's own file %q", after.MetadataUrl, want) + } +} + +// Control: a statement naming a different project file is a change, wherever +// the stored path came from. +func TestCreateOrModifyODataClient_RewritesMetadataUrlForAnotherPath(t *testing.T) { + root := t.TempDir() + here := checkoutProject(t, root, "home-user", edmxWith("Session")) + if err := os.WriteFile(filepath.Join(here, "contracts", "f2.xml"), []byte(edmxWith("Session")), 0o600); err != nil { + t.Fatal(err) + } + foreign := fileURL(filepath.Join(root, "workspaces", "app", "contracts", "f1.xml")) + + _, after, _ := modifyFromCheckout(t, here, foreign, "./contracts/f2.xml") + + if want := fileURL(filepath.Join(here, "contracts", "f2.xml")); after.MetadataUrl != want { + t.Errorf("MetadataUrl = %q, want %q", after.MetadataUrl, want) + } +} + +// Keeping another checkout's path must not cost the contract refresh: the +// statement's file is the one in this checkout, and it is what gets read. +func TestCreateOrModifyODataClient_RefreshesFromThisCheckoutsFile(t *testing.T) { + root := t.TempDir() + here := checkoutProject(t, root, "home-user", edmxWith("Session")) + f := newODataModifyFixture(t, here, filepath.Join(here, "contracts", "f1.xml")) + foreign := fileURL(filepath.Join(root, "workspaces", "home-user", "contracts", "f1.xml")) + f.svc.MetadataUrl = foreign + + if err := os.WriteFile(filepath.Join(here, "contracts", "f1.xml"), []byte(edmxWith("Session", "Trace")), 0o600); err != nil { + t.Fatal(err) + } + f.modify(t, "./contracts/f1.xml") + + got := *f.updated + if !strings.Contains(got.Metadata, `Name="Trace"`) { + t.Errorf("the contract was not refreshed from this checkout's file; output:\n%s", f.out.String()) + } + if got.MetadataUrl != foreign { + t.Errorf("MetadataUrl = %q, want the stored %q kept", got.MetadataUrl, foreign) + } +} + +// describe prints a stored URL inside the project directory as the +// project-relative path a script uses, so a generated script carries no +// machine path; executing it leaves the client unchanged. +func TestDescribeODataClient_PrintsProjectRelativeMetadataUrl(t *testing.T) { + root := t.TempDir() + here := checkoutProject(t, root, "app", edmxWith("Session")) + f := newODataModifyFixture(t, here, filepath.Join(here, "contracts", "f1.xml")) + before := *f.svc + + var out bytes.Buffer + f.ctx.Output = &out + if err := outputConsumedODataServiceMDL(f.ctx, f.svc, "MyModule", ""); err != nil { + t.Fatal(err) + } + if !strings.Contains(out.String(), "MetadataUrl: './contracts/f1.xml'") { + t.Fatalf("describe did not print the project-relative path:\n%s", out.String()) + } + if strings.Contains(out.String(), root) { + t.Fatalf("describe printed this machine's path:\n%s", out.String()) + } + + var stmt *ast.CreateODataClientStmt + for _, s := range parseMDL(t, out.String()).Statements { + if c, ok := s.(*ast.CreateODataClientStmt); ok { + stmt = c + } + } + if stmt == nil { + t.Fatalf("no create odata client in:\n%s", out.String()) + } + f.ctx.Output = f.out + f.modify(t, stmt.MetadataUrl) + if after := **f.updated; after.MetadataUrl != before.MetadataUrl || after.Metadata != before.Metadata { + t.Errorf("exec(describe) changed the client: MetadataUrl %q -> %q", before.MetadataUrl, after.MetadataUrl) + } +} + +// Control: a stored URL outside the project directory has no project-relative +// form, and describe prints it as stored. +func TestDescribeODataClient_PrintsMetadataUrlOutsideProjectAsStored(t *testing.T) { + root := t.TempDir() + here := checkoutProject(t, root, "app", edmxWith("Session")) + f := newODataModifyFixture(t, here, filepath.Join(here, "contracts", "f1.xml")) + outside := fileURL(filepath.Join(root, "elsewhere", "f1.xml")) + f.svc.MetadataUrl = outside + + var out bytes.Buffer + f.ctx.Output = &out + if err := outputConsumedODataServiceMDL(f.ctx, f.svc, "MyModule", ""); err != nil { + t.Fatal(err) + } + if !strings.Contains(out.String(), "MetadataUrl: '"+outside+"'") { + t.Fatalf("describe did not print the stored URL %q:\n%s", outside, out.String()) + } +} diff --git a/mdl/executor/cmd_pages_describe_flowcontext.go b/mdl/executor/cmd_pages_describe_flowcontext.go index 46ae4b4db7..dc3da1deb6 100644 --- a/mdl/executor/cmd_pages_describe_flowcontext.go +++ b/mdl/executor/cmd_pages_describe_flowcontext.go @@ -97,7 +97,11 @@ func flowReturnEntity(ctx *ExecContext, kind, qualifiedName string) string { } switch kind { case "microflow": - mfs, err := ctx.Backend.ListMicroflows() + // Only the microflows of this name are decoded: a page describe asks + // once per flow-sourced widget, and listing them all read every + // microflow of the project each time (ako/mxcli#1791). + _, local, _ := strings.Cut(qualifiedName, ".") + mfs, err := microflowsNamed(ctx, local) if err != nil { return "" } diff --git a/mdl/executor/executor.go b/mdl/executor/executor.go index 558d4e6f5c..f16cf084dc 100644 --- a/mdl/executor/executor.go +++ b/mdl/executor/executor.go @@ -307,6 +307,12 @@ type Executor struct { untimedMu sync.Mutex // protects untimed untimed *untimedWork // the running statement's untimed-work tracker (see untimed_work.go) catalogGen uint64 // monotonic generation counter for catalog swaps + + // Files changed beside the model (ako/mxcli#1618): a program run's, for + // its summary line, and whether the caller wants them reported at all — + // not when its writes are temporary (SetSourceFileReport). + sourceFiles sourceFileLog + noSourceFileReport bool } // New creates a new executor with the given output writer. @@ -817,11 +823,18 @@ func (e *Executor) beginTally() bool { if e.tally == nil { e.tally = &mutationTally{} } - return e.tally.begin() + owns := e.tally.begin() + if owns { + e.sourceFiles.begin() + } + return owns } -// flushTally writes the run's one-line summary and stands the tally down. +// flushTally writes the run's one-line summaries and stands the tally down. +// The files changed beside the model come last: what to stage is the line a +// reader should finish on. func (e *Executor) flushTally() { e.tally.flush(e.output) + e.sourceFiles.flush(e.output) e.tally.end() } diff --git a/mdl/executor/executor_dispatch.go b/mdl/executor/executor_dispatch.go index 8b05cd15df..40ac9691a7 100644 --- a/mdl/executor/executor_dispatch.go +++ b/mdl/executor/executor_dispatch.go @@ -22,6 +22,11 @@ func (e *Executor) executeInner(ctx context.Context, stmt ast.Statement) error { // session (next statement fails with "not connected to a project"). before := e.backend err := e.registry.Dispatch(ectx, stmt) + // What the statement wrote beside the model, after its own report and + // whether or not it then failed: a file on disk is there either way. + if ctx.Err() == nil { + e.reportSourceFileChanges(ectx.Backend, ectx.Output) + } // Only sync back when the context has not been cancelled. Execute() runs // executeInner in a goroutine with a wall-clock timeout; if the timeout // fires, the goroutine keeps running but Execute() has already returned. diff --git a/mdl/executor/flow_splice_assoc_retrieve_test.go b/mdl/executor/flow_splice_assoc_retrieve_test.go index be5dd7aadc..7986719baf 100644 --- a/mdl/executor/flow_splice_assoc_retrieve_test.go +++ b/mdl/executor/flow_splice_assoc_retrieve_test.go @@ -97,7 +97,7 @@ func orderFixture(t *testing.T) (*ExecContext, *alterFlowContext) { // candidates is not the order a variable is typed in. over("SiblingLines", "LineOrder", "OrderLine_Order"), over("LineOrder", "Line", "OrderLine_Order"), - over("Parent", "Order", "Order_Parent"), // self-reference: which end is unknown + over("Parent", "Order", "Order_Parent"), // self-reference: from the FROM end (#1777) over("Nowhere", "Order", "Order_Missing"), // no such association over("Unstarted", "NoSuchVariable", "OrderLine_Order"), // A loop's iterator is an element of its list, and a retrieve in @@ -118,8 +118,9 @@ func orderFixture(t *testing.T) (*ExecContext, *alterFlowContext) { // retrieve over an association has the type the association gives it from // the end it starts at — a list of the FROM entity from the TO end, the one // TO object from the FROM end, a list either way over a reference set, one -// object either way over a one-to-one. What cannot be established stays -// untyped, so a member of it is refused, not guessed. +// object either way over a one-to-one, and over a self-reference what the FROM +// end holds, which is how Mendix reads it (ako/mxcli#1777). What cannot be +// established stays untyped, so a member of it is refused, not guessed. func TestStoredVariables_AssociationRetrieveIsTyped(t *testing.T) { ctx, a := orderFixture(t) varTypes, declared := a.storedVariables(ctx) @@ -134,13 +135,14 @@ func TestStoredVariables_AssociationRetrieveIsTyped(t *testing.T) { "LineOrder": "Mod.Order", "EachLine": "Mod.OrderLine", "EachOrder": "Mod.Order", + "Parent": "Mod.Order", } for v, w := range want { if got := varTypes[v]; got != w { t.Errorf("$%s: typed %q, want %q", v, got, w) } } - for _, v := range []string{"Parent", "Nowhere", "Unstarted", "ItOrder"} { + for _, v := range []string{"Nowhere", "Unstarted", "ItOrder"} { if got, ok := varTypes[v]; ok { t.Errorf("$%s: typed %q, want it left untyped", v, got) } diff --git a/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go b/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go index 11680b697c..a60d7ef366 100644 --- a/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go +++ b/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go @@ -24,10 +24,14 @@ create persistent entity MyModule.Order (Code: String(50)); create persistent entity MyModule.OrderLine (Code: String(50), Status: String(20)); create association MyModule.OrderLine_Order from MyModule.OrderLine to MyModule.Order; create non-persistent entity MyModule.Lookup (Code: String(50)); +create java action MyModule.JA_Items (EntityType: entity not null) returns list of T as $$ + return new java.util.ArrayList<>(); +$$; create microflow MyModule.SUB_MatchLines ($Order: MyModule.Order, $Lookups: List of MyModule.Lookup) returns Integer as $Found begin declare $Found Integer = 0; retrieve $Lines from $Order/MyModule.OrderLine_Order; + $Items = call java action MyModule.JA_Items(EntityType = MyModule.OrderLine); $Open = filter $Lines by Status = 'open'; loop $Lookup in $Lookups begin @@ -88,15 +92,17 @@ func openMatchLinesFixture(t *testing.T) (*Executor, *bytes.Buffer, string) { } // TestAlterMicroflow_FindOverUntypedListIsRefused: over a list whose entity -// the fragment does not know — the stored flow does not type a list -// operation's output — every alter is refused and writes nothing. On the main +// the fragment does not know — the result of a Java action returning a list of +// its type parameter, bound by an entity argument, which `create microflow` +// does not type either (a list operation's output, which this used, is typed +// since ako/mxcli#1777) — every alter is refused and writes nothing. On the main // line and in a stored loop body it always was (ako/mxcli#891); in a loop the // fragment brings, it was written as a find by expression. Check does not // pass it silently: like the main-line refusal, it names the change as not // predicted with exec's reason. func TestAlterMicroflow_FindOverUntypedListIsRefused(t *testing.T) { const want = `cannot qualify member "Code"` - for name, src := range matchLinesFindIn("$Open") { + for name, src := range matchLinesFindIn("$Items") { t.Run(name, func(t *testing.T) { exec, _, dir := openMatchLinesFixture(t) before := projectFiles(t, dir) diff --git a/mdl/executor/flow_splice_stored_outputs_1777_pedapp_test.go b/mdl/executor/flow_splice_stored_outputs_1777_pedapp_test.go new file mode 100644 index 0000000000..11260976d2 --- /dev/null +++ b/mdl/executor/flow_splice_stored_outputs_1777_pedapp_test.go @@ -0,0 +1,126 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "context" + "testing" + + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// ako/mxcli#1777 on the Studio Pro-authored PedApp, through the real codec: a +// fragment an alter inserts names a member of a stored list operation's, Java +// action's or self-reference retrieve's output, unqualified. It was refused +// ("cannot qualify member …") with nothing written; it is written qualified, +// as `create microflow` writes it, and check predicts it. + +const storedShapesSetup = `mdl 1; +create module MyModule; +create persistent entity MyModule.Order (Code: String(50)); +create persistent entity MyModule.OrderLine (Code: String(50), Status: String(20)); +create association MyModule.OrderLine_Order from MyModule.OrderLine to MyModule.Order; +create association MyModule.Order_Parent from MyModule.Order to MyModule.Order; +create association MyModule.Order_Related from MyModule.Order to MyModule.Order type ReferenceSet; +create java action MyModule.JA_GetOrder (Code: String) returns MyModule.Order as $$ + return null; +$$; +create java action MyModule.JA_GetLines (Code: String) returns list of MyModule.OrderLine as $$ + return new java.util.ArrayList<>(); +$$; +create java action MyModule.JA_Shuffle (Items: list of T) returns list of T as $$ + return Items; +$$; +create microflow MyModule.SUB_Shapes ($Order: MyModule.Order, $Lines: List of MyModule.OrderLine) +begin + $Open = filter $Lines by Status = 'Open'; + $Sorted = sort $Lines by Code asc; + $First = head $Lines; + $Got = call java action MyModule.JA_GetOrder(Code = 'A'); + $GotLines = call java action MyModule.JA_GetLines(Code = 'A'); + $Shuffled = call java action MyModule.JA_Shuffle(Items = $Lines); + retrieve $Parent from $Order/MyModule.Order_Parent; + retrieve $Related from $Order/MyModule.Order_Related; + log info node 'Shapes' 'done'; +end;` + +// storedShapesFlow returns the stored MyModule.SUB_Shapes. +func storedShapesFlow(t *testing.T, exec *Executor) *microflows.Microflow { + t.Helper() + ctx := exec.newExecContext(context.Background()) + h, err := getHierarchy(ctx) + if err != nil { + t.Fatal(err) + } + all, err := ctx.Backend.ListMicroflows() + if err != nil { + t.Fatal(err) + } + for _, m := range all { + if m.Name == "SUB_Shapes" && h.GetModuleName(h.FindModuleID(m.ContainerID)) == "MyModule" { + mf, err := ctx.Backend.GetMicroflow(m.ID) + if err != nil { + t.Fatal(err) + } + return mf + } + } + t.Fatal("MyModule.SUB_Shapes not found") + return nil +} + +// memberWrittenFor is the member the stored activity outputting (a list +// operation) or changing (a change) variable writes, or "". +func memberWrittenFor(mf *microflows.Microflow, variable string) string { + for _, obj := range mf.ObjectCollection.Objects { + act, ok := obj.(*microflows.ActionActivity) + if !ok { + continue + } + switch x := act.Action.(type) { + case *microflows.ChangeObjectAction: + if x.ChangeVariable == variable && len(x.Changes) == 1 { + return x.Changes[0].AttributeQualifiedName + } + case *microflows.ListOperationAction: + if x.OutputVariable != variable { + continue + } + switch op := x.Operation.(type) { + case *microflows.FindByAttributeOperation: + return op.Attribute + case *microflows.FilterByAttributeOperation: + return op.Attribute + } + } + } + return "" +} + +func TestAlterMicroflow_MembersOfStoredOutputsAreQualified(t *testing.T) { + cases := []struct { + name, stmt, variable, want string + }{ + {"a filter's output", `$OpenA = find $Open by Code = 'A';`, "OpenA", "MyModule.OrderLine.Code"}, + {"a sort's output", `$SortedB = filter $Sorted by Code = 'B';`, "SortedB", "MyModule.OrderLine.Code"}, + {"a head's output", `change $First (Status = 'Seen');`, "First", "MyModule.OrderLine.Status"}, + {"a Java action's object", `change $Got (Code = 'X');`, "Got", "MyModule.Order.Code"}, + {"a Java action's list", `$GotOpen = find $GotLines by Status = 'Open';`, "GotOpen", "MyModule.OrderLine.Status"}, + {"a Java action's list of its type parameter", `$ShuffledA = find $Shuffled by Code = 'A';`, "ShuffledA", "MyModule.OrderLine.Code"}, + {"a self-reference's object", `change $Parent (Code = 'P');`, "Parent", "MyModule.Order.Code"}, + {"a self reference set's list", `$RelatedA = find $Related by Code = 'A';`, "RelatedA", "MyModule.Order.Code"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + exec, out, _ := openPedAppCopy(t) + if err := agreeExec(t, exec, storedShapesSetup); err != nil { + t.Fatalf("setup: %v\n%s", err, out.String()) + } + predictedAgree(t, exec, "mdl 1;\nalter microflow MyModule.SUB_Shapes {\n insert after $Related begin "+ + c.stmt+" end;\n};", "") + if got := memberWrittenFor(storedShapesFlow(t, exec), c.variable); got != c.want { + t.Errorf("%s wrote %q, want %q", c.stmt, got, c.want) + } + }) + } +} diff --git a/mdl/executor/flow_splice_stored_outputs_test.go b/mdl/executor/flow_splice_stored_outputs_test.go new file mode 100644 index 0000000000..68fc9a441b --- /dev/null +++ b/mdl/executor/flow_splice_stored_outputs_test.go @@ -0,0 +1,320 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mfmutator" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/domainmodel" + "github.com/mendixlabs/mxcli/sdk/javaactions" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// ako/mxcli#1777, the follow-up to #1769: a fragment spliced into a stored +// flow starts from the variables the stored flow declares (storedVariables), +// and three kinds were left untyped there although `create microflow` types +// them — the output of a list operation, of a Java action call, and of a +// retrieve over a self-reference. A member of one was refused ("cannot qualify +// member …: the entity its variable holds is not known here") and nothing was +// written. + +// storedOutputsFixture is a stored flow over Mod.Order (Code) and +// Mod.OrderLine (Code, Status). Order_Parent is a reference from Order to +// itself, Order_Related a reference set from Order to itself. The Java actions +// return an Order (GetOrder), a list of lines (GetLines), a list of their type +// parameter (Shuffle, typed from its list argument) and an Integer (CountLines). +func storedOutputsFixture(t *testing.T) (*ExecContext, *alterFlowContext) { + t.Helper() + mod := mkModule("Mod") + attr := func(name string) *domainmodel.Attribute { + return &domainmodel.Attribute{BaseElement: model.BaseElement{ID: nextID("attr")}, Name: name} + } + order := mkEntity(mod.ID, "Order") + order.Attributes = []*domainmodel.Attribute{attr("Code")} + line := mkEntity(mod.ID, "OrderLine") + line.Attributes = []*domainmodel.Attribute{attr("Code"), attr("Status")} + tag := mkEntity(mod.ID, "Tag") + tag.Attributes = []*domainmodel.Attribute{attr("Label")} + dm := mkDomainModel(mod.ID, order, line, tag) + related := mkAssociation(dm.ID, "Order_Related", order.ID, order.ID) + related.Type = domainmodel.AssociationTypeReferenceSet + dm.Associations = []*domainmodel.Association{ + mkAssociation(dm.ID, "OrderLine_Order", line.ID, order.ID), + mkAssociation(dm.ID, "Order_Parent", order.ID, order.ID), + related, + } + javaActions := map[string]*javaactions.JavaAction{ + "Mod.GetOrder": {Name: "GetOrder", ReturnType: &javaactions.EntityType{Entity: "Mod.Order"}}, + "Mod.GetLines": {Name: "GetLines", ReturnType: &javaactions.ListType{Entity: "Mod.OrderLine"}}, + "Mod.Shuffle": {Name: "Shuffle", ReturnType: &javaactions.ListType{TypeParameter: "T"}}, + "Mod.CountLines": {Name: "CountLines", ReturnType: &javaactions.IntegerType{}}, + } + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + GetModuleByNameFunc: func(name string) (*model.Module, error) { + if name == "Mod" { + return mod, nil + } + return nil, nil + }, + GetDomainModelFunc: func(model.ID) (*domainmodel.DomainModel, error) { return dm, nil }, + ReadJavaActionByNameFunc: func(qn string) (*javaactions.JavaAction, error) { + if ja, ok := javaActions[qn]; ok { + return ja, nil + } + return nil, fmt.Errorf("java action %s not found", qn) + }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(mkHierarchy(mod))) + + act := func(out string, a microflows.MicroflowAction) mfmutator.Candidate { + return mfmutator.Candidate{OutputVariable: out, Object: µflows.ActionActivity{Action: a}} + } + listOp := func(out string, op microflows.ListOperation) mfmutator.Candidate { + return act(out, µflows.ListOperationAction{OutputVariable: out, Operation: op}) + } + aggregate := func(out string, fn microflows.AggregateFunction, reduce microflows.DataType) mfmutator.Candidate { + return act(out, µflows.AggregateListAction{InputVariable: "Lines", OutputVariable: out, Function: fn, ReduceReturnType: reduce}) + } + java := func(out, action string, args ...string) mfmutator.Candidate { + var mappings []*microflows.JavaActionParameterMapping + for i, arg := range args { + mappings = append(mappings, µflows.JavaActionParameterMapping{ + Parameter: fmt.Sprintf("%s.P%d", action, i), + Value: µflows.BasicCodeActionParameterValue{Argument: arg}, + }) + } + return act(out, µflows.JavaActionCallAction{JavaAction: action, ParameterMappings: mappings, + ResultVariableName: out, UseReturnVariable: true}) + } + over := func(out, start, assoc string) mfmutator.Candidate { + return act(out, µflows.RetrieveAction{OutputVariable: out, + Source: µflows.AssociationRetrieveSource{StartVariable: start, AssociationQualifiedName: "Mod." + assoc}}) + } + a := &alterFlowContext{ + stmt: &ast.AlterFlowStmt{}, + mf: µflows.Microflow{Parameters: []*microflows.MicroflowParameter{ + {Name: "Order", Type: µflows.ObjectType{EntityQualifiedName: "Mod.Order"}}, + {Name: "Lines", Type: µflows.ListType{EntityQualifiedName: "Mod.OrderLine"}}, + {Name: "More", Type: µflows.ListType{EntityQualifiedName: "Mod.OrderLine"}}, + {Name: "Tags", Type: µflows.ListType{EntityQualifiedName: "Mod.Tag"}}, + }}, + entityNames: map[model.ID]string{order.ID: "Mod.Order", line.ID: "Mod.OrderLine", tag.ID: "Mod.Tag"}, + cands: []mfmutator.Candidate{ + // List operations that keep the list. + listOp("Filtered", µflows.FilterByAttributeOperation{ListVariable: "Lines", Attribute: "Mod.OrderLine.Status", Expression: "'Open'"}), + listOp("FilteredExpr", µflows.FilterOperation{ListVariable: "Lines", Expression: "true"}), + listOp("Rest", µflows.TailOperation{ListVariable: "Lines"}), + listOp("Page", µflows.ListRangeOperation{ListVariable: "Lines", LimitExpression: "10"}), + listOp("Joined", µflows.UnionOperation{ListVariable1: "Lines", ListVariable2: "More"}), + listOp("Common", µflows.IntersectOperation{ListVariable1: "Lines", ListVariable2: "More"}), + listOp("Diff", µflows.SubtractOperation{ListVariable1: "Lines", ListVariable2: "More"}), + // List operations that pick one object. + listOp("First", µflows.HeadOperation{ListVariable: "Lines"}), + listOp("Found", µflows.FindByAttributeOperation{ListVariable: "Lines", Attribute: "Mod.OrderLine.Code", Expression: "'A'"}), + listOp("FoundExpr", µflows.FindOperation{ListVariable: "Lines", Expression: "true"}), + // Reads a list a later candidate types: the order of the + // candidates is not the order a variable is typed in. + listOp("FirstSorted", µflows.HeadOperation{ListVariable: "Sorted"}), + listOp("Sorted", µflows.SortOperation{ListVariable: "Lines"}), + // A Boolean, which holds no entity. + listOp("Has", µflows.ContainsOperation{ListVariable: "Lines", ObjectVariable: "First"}), + listOp("Same", µflows.EqualsOperation{ListVariable1: "Lines", ListVariable2: "More"}), + // Aggregates hold a primitive. + aggregate("Count", microflows.AggregateFunctionCount, nil), + aggregate("Mean", microflows.AggregateFunctionAverage, nil), + aggregate("Total", microflows.AggregateFunctionSum, nil), + aggregate("AnyOpen", microflows.AggregateFunctionAny, µflows.BooleanType{}), + aggregate("Folded", microflows.AggregateFunctionReduce, µflows.StringType{}), + // Java action calls. + java("Got", "Mod.GetOrder"), + java("GotLines", "Mod.GetLines"), + java("Shuffled", "Mod.Shuffle", "$Lines"), + java("Number", "Mod.CountLines", "$Lines"), + java("Lost", "Mod.NoSuchAction"), + // Retrieves over a self-reference, and one started from it. + over("Parent", "Order", "Order_Parent"), + over("Grandparent", "Parent", "Order_Parent"), + over("Related", "Order", "Order_Related"), + over("ParentLines", "Parent", "OrderLine_Order"), + // Two origins that give one type. + listOp("Pick", µflows.HeadOperation{ListVariable: "Lines"}), + listOp("Pick", µflows.FindOperation{ListVariable: "Lines", Expression: "true"}), + // Two origins that give different types: which one a fragment + // means depends on where it goes (two loop bodies can each declare + // it). + listOp("Dup", µflows.HeadOperation{ListVariable: "Lines"}), + listOp("Dup", µflows.HeadOperation{ListVariable: "Tags"}), + java("Mixed", "Mod.GetOrder"), + listOp("Mixed", µflows.HeadOperation{ListVariable: "Lines"}), + listOp("DupFirst", µflows.HeadOperation{ListVariable: "Dup"}), + }, + } + return ctx, a +} + +// TestStoredVariables_ListOperationJavaAndSelfReferenceAreTyped: each output +// has the type `create microflow` gives it — a list operation's that of its +// input list (the list, or for head and find its element), a Java action's its +// declared return type (or, for a list of its type parameter, its list +// argument's), a retrieve over a self-reference what Mendix reads it as — and +// what cannot be established stays untyped. +func TestStoredVariables_ListOperationJavaAndSelfReferenceAreTyped(t *testing.T) { + ctx, a := storedOutputsFixture(t) + varTypes, declared := a.storedVariables(ctx) + want := map[string]string{ + "Filtered": "List of Mod.OrderLine", + "FilteredExpr": "List of Mod.OrderLine", + "Rest": "List of Mod.OrderLine", + "Page": "List of Mod.OrderLine", + "Joined": "List of Mod.OrderLine", + "Common": "List of Mod.OrderLine", + "Diff": "List of Mod.OrderLine", + "Sorted": "List of Mod.OrderLine", + "First": "Mod.OrderLine", + "Found": "Mod.OrderLine", + "FoundExpr": "Mod.OrderLine", + "FirstSorted": "Mod.OrderLine", + "Got": "Mod.Order", + "GotLines": "List of Mod.OrderLine", + "Shuffled": "List of Mod.OrderLine", + "Parent": "Mod.Order", + "Grandparent": "Mod.Order", + "Related": "List of Mod.Order", + "ParentLines": "List of Mod.OrderLine", + "Pick": "Mod.OrderLine", + } + for v, w := range want { + if got := varTypes[v]; got != w { + t.Errorf("$%s: typed %q, want %q", v, got, w) + } + if d, ok := declared[v]; ok { + t.Errorf("$%s: typed and also declared %q", v, d) + } + } + wantDeclared := map[string]string{ + "Has": "Unknown", + "Same": "Unknown", + "Count": "Integer", + "Mean": "Decimal", + "Total": "Unknown", + "AnyOpen": "Boolean", + "Folded": "String", + "Number": "Unknown", + "Lost": "Unknown", + "Dup": "Unknown", + "Mixed": "Unknown", + "DupFirst": "Unknown", + } + for v, w := range wantDeclared { + if got, ok := varTypes[v]; ok { + t.Errorf("$%s: typed %q, want it left untyped", v, got) + } + if declared[v] != w { + t.Errorf("$%s: declared %q, want %q", v, declared[v], w) + } + } +} + +// fragmentMembers builds src's body as a spliced fragment and returns the +// member each list operation and change in it writes, in order. +func fragmentMembers(t *testing.T, ctx *ExecContext, a *alterFlowContext, src string) ([]string, error) { + t.Helper() + frag, err := a.buildFragment(ctx, parsedFlowBody(t, "mdl 1;\ncreate microflow Mod.Fragment ()\nbegin\n"+src+"\nend;")) + if err != nil { + return nil, err + } + var members []string + for _, obj := range frag.Objects { + act, ok := obj.(*microflows.ActionActivity) + if !ok { + continue + } + switch x := act.Action.(type) { + case *microflows.ChangeObjectAction: + for _, m := range x.Changes { + members = append(members, m.AttributeQualifiedName) + } + case *microflows.ListOperationAction: + switch op := x.Operation.(type) { + case *microflows.FindByAttributeOperation: + members = append(members, op.Attribute) + case *microflows.FilterByAttributeOperation: + members = append(members, op.Attribute) + case *microflows.SortOperation: + for _, s := range op.Sorting { + members = append(members, s.AttributeQualifiedName) + } + default: + members = append(members, fmt.Sprintf("%T", op)) + } + } + } + return members, nil +} + +// TestSpliceFragment_MembersOfStoredOutputsAreQualified is the reported +// shape: a fragment naming a member of a stored list operation's, Java +// action's or self-reference retrieve's output, unqualified, writes it +// qualified with the entity the variable holds, as `create microflow` does. +func TestSpliceFragment_MembersOfStoredOutputsAreQualified(t *testing.T) { + cases := []struct { + name, src string + want []string + }{ + {"find in a filtered list", `$F = find $Filtered by Code = 'A';`, []string{"Mod.OrderLine.Code"}}, + {"filter a sorted list", `$F = filter $Sorted by Status = 'Open';`, []string{"Mod.OrderLine.Status"}}, + {"sort a union", `$S = sort $Joined by Code asc;`, []string{"Mod.OrderLine.Code"}}, + {"find in a tail", `$F = find $Rest by Code = 'A';`, []string{"Mod.OrderLine.Code"}}, + {"change a head", `change $First (Code = 'B');`, []string{"Mod.OrderLine.Code"}}, + {"change a find", `change $Found (Status = 'Done');`, []string{"Mod.OrderLine.Status"}}, + {"change a head of a sorted list", `change $FirstSorted (Code = 'B');`, []string{"Mod.OrderLine.Code"}}, + {"change a Java action's object", `change $Got (Code = 'B');`, []string{"Mod.Order.Code"}}, + {"find in a Java action's list", `$F = find $GotLines by Status = 'Open';`, []string{"Mod.OrderLine.Status"}}, + {"find in a Java action's list of its type parameter", `$F = find $Shuffled by Code = 'A';`, []string{"Mod.OrderLine.Code"}}, + {"change a self-reference's object", `change $Parent (Code = 'B');`, []string{"Mod.Order.Code"}}, + {"change over a self-reference twice", `change $Grandparent (Code = 'B');`, []string{"Mod.Order.Code"}}, + {"find in a self reference set", `$F = find $Related by Code = 'A';`, []string{"Mod.Order.Code"}}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + ctx, a := storedOutputsFixture(t) + got, err := fragmentMembers(t, ctx, a, c.src) + if err != nil { + t.Fatalf("%s: %v", c.src, err) + } + if strings.Join(got, ",") != strings.Join(c.want, ",") { + t.Errorf("%s wrote %v, want %v", c.src, got, c.want) + } + }) + } +} + +// TestSpliceFragment_MembersOfConflictingOutputsAreRefused: a name two stored +// origins give different types stays untyped, and a member of it is refused +// rather than qualified with a guessed entity — as is one read from it. +// Control: the same member of a name two origins give one type is written. +func TestSpliceFragment_MembersOfConflictingOutputsAreRefused(t *testing.T) { + for _, src := range []string{ + `change $Dup (Code = 'B');`, + `change $Mixed (Code = 'B');`, + `change $DupFirst (Code = 'B');`, + `change $Lost (Code = 'B');`, + } { + ctx, a := storedOutputsFixture(t) + if _, err := fragmentMembers(t, ctx, a, src); err == nil || !strings.Contains(err.Error(), "cannot qualify member") { + t.Errorf("%s: want a refusal, got %v", src, err) + } + } + ctx, a := storedOutputsFixture(t) + got, err := fragmentMembers(t, ctx, a, `change $Pick (Code = 'B');`) + if err != nil || strings.Join(got, ",") != "Mod.OrderLine.Code" { + t.Errorf("change $Pick: got %v, %v; want [Mod.OrderLine.Code]", got, err) + } +} diff --git a/mdl/executor/helpers.go b/mdl/executor/helpers.go index 62d078b3cd..bd16cc82f8 100644 --- a/mdl/executor/helpers.go +++ b/mdl/executor/helpers.go @@ -563,6 +563,54 @@ func microflowsNamed(ctx *ExecContext, name string) ([]*microflows.Microflow, er return out, nil } +// pinnedMicroflow is the microflow a describe is pinned to (ctx.describeID), +// read by ID, when it is the named one; nil when there is no pin or the pinned +// document is not that microflow, so the caller falls back to the name lookup. +func pinnedMicroflow(ctx *ExecContext, h *ContainerHierarchy, name ast.QualifiedName) ([]*microflows.Microflow, error) { + if ctx.describeID == "" { + return nil, nil + } + mf, err := ctx.Backend.GetMicroflow(ctx.describeID) + if err != nil || mf == nil { + return nil, err + } + if mf.Name != name.Name || h.GetModuleName(h.FindModuleID(mf.ContainerID)) != name.Module { + return nil, nil + } + return []*microflows.Microflow{mf}, nil +} + +// microflowsInModule is the microflows of one module, or of the project when +// moduleName is "". A module's are found by their headers and decoded one by +// one, so `show microflows in M` decodes M's microflows, not the project's +// (ako/mxcli#1791). +func microflowsInModule(ctx *ExecContext, h *ContainerHierarchy, moduleName string) ([]*microflows.Microflow, error) { + if moduleName == "" { + return ctx.Backend.ListMicroflows() + } + if _, ok := ctx.Backend.(microflowLookup); !ok { + return ctx.Backend.ListMicroflows() + } + hs, err := microflowHeaders(ctx) + if err != nil { + return nil, err + } + var out []*microflows.Microflow + for _, hd := range hs { + if h.GetModuleName(h.FindModuleID(hd.ContainerID)) != moduleName { + continue + } + mf, err := ctx.Backend.GetMicroflow(hd.ID) + if err != nil { + return nil, err + } + if mf != nil { + out = append(out, mf) + } + } + return out, nil +} + // microflowHeaders is every microflow's ID, container, name and Excluded flag, // for a name set or an ID-to-name map: no flow is decoded where the backend // can avoid it. diff --git a/mdl/executor/source_file_report.go b/mdl/executor/source_file_report.go new file mode 100644 index 0000000000..2b5062d121 --- /dev/null +++ b/mdl/executor/source_file_report.go @@ -0,0 +1,114 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "io" + "strings" + + "github.com/mendixlabs/mxcli/mdl/backend" +) + +// Files beside the model (ako/mxcli#1618, mendixlabs/mxcli#1634). +// +// `create java action … as $$ … $$` stores the action in the model and writes +// its body to javasource//actions/.java; `exec` named only the +// model element. A workflow that commits "the model" — the .mpr and +// mprcontents/ — left the body behind, and a clone got an action with no logic, +// with nothing in the output to say a second place had been written. +// +// So every file or folder a statement changes beside the model is named on its +// own line after the statement's report, and a program run that changed any +// ends with one line saying to stage them. The changes come from the backend, +// which records each at the moment it touches disk (backend.SourceFileReporter): +// the report cannot claim a write that did not happen, and an identical rewrite, +// which leaves the file alone, is not reported at all. + +// SetSourceFileReport turns the report of files changed beside the model on +// (the default) or off. Off is for a caller whose writes are temporary — `mxcli +// test` injects a Java action and deletes its source again at cleanup, and +// telling its user to commit that file would be wrong. +func (e *Executor) SetSourceFileReport(on bool) { + e.noSourceFileReport = !on +} + +// reportSourceFileChanges prints, one per line, the files and folders beside +// the model that the statement just run changed, and keeps them for the run's +// summary. A backend that writes none does not implement the reporter. +func (e *Executor) reportSourceFileChanges(b backend.FullBackend, w io.Writer) { + r, ok := b.(backend.SourceFileReporter) + if !ok { + return + } + // Taken even when not reported, so a later statement does not report it. + changes := r.TakeSourceFileChanges() + if e.noSourceFileReport { + return + } + for _, c := range changes { + fmt.Fprintln(w, sourceFileChangeLine(c)) + } + e.sourceFiles.add(changes) +} + +// sourceFileChangeLine is the line one change is reported with. +func sourceFileChangeLine(c backend.SourceFileChange) string { + switch { + case c.Removed: + return fmt.Sprintf("Removed %s (%s; commit the deletion with the model)", c.Path, c.What) + case c.From != "": + return fmt.Sprintf("Moved %s to %s (%s; commit both paths with the model)", c.From, c.Path, c.What) + default: + return fmt.Sprintf("Wrote %s (%s; commit it with the model)", c.Path, c.What) + } +} + +// sourceFileLog collects a program run's changes beside the model for the +// summary line at its end. Its lifetime is the mutation tally's: the run that +// owns the tally owns this, so a nested EXECUTE SCRIPT adds to its caller's +// summary instead of printing its own. +type sourceFileLog struct { + active bool + changes []backend.SourceFileChange +} + +// begin starts collecting for a program run. +func (l *sourceFileLog) begin() { + l.active = true + l.changes = nil +} + +// add records changes for the summary. Outside a program run — one statement +// in the REPL — there is no summary, and nothing is kept. +func (l *sourceFileLog) add(changes []backend.SourceFileChange) { + if l.active { + l.changes = append(l.changes, changes...) + } +} + +// flush writes the summary, if the run changed anything beside the model, and +// stands the log down. A path written more than once counts once; the folders +// are the top-level ones, in the order first touched. +func (l *sourceFileLog) flush(w io.Writer) { + defer func() { l.active, l.changes = false, nil }() + if !l.active || len(l.changes) == 0 { + return + } + paths := map[string]bool{} + var roots []string + seenRoot := map[string]bool{} + for _, c := range l.changes { + paths[c.Path] = true + root := c.Path + if i := strings.Index(root, "/"); i >= 0 { + root = root[:i+1] + } + if !seenRoot[root] { + seenRoot[root] = true + roots = append(roots, root) + } + } + fmt.Fprintf(w, "Files changed outside the model: %d (%s). Stage them with the .mpr.\n", + len(paths), strings.Join(roots, ", ")) +} diff --git a/mdl/executor/source_file_report_test.go b/mdl/executor/source_file_report_test.go new file mode 100644 index 0000000000..1084995a82 --- /dev/null +++ b/mdl/executor/source_file_report_test.go @@ -0,0 +1,213 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/javaactions" +) + +// ako/mxcli#1618 (mendixlabs/mxcli#1634): `create java action … as $$ … $$` +// stores the action in the model AND writes its body to +// javasource//actions/.java, and `exec` said only "Created java +// action: Probe.JA_Hello". A workflow that commits the model (.mpr and +// mprcontents/) left the .java behind, and a clone got an action with no body. +// Every file a statement changes beside the model is now named on its own line, +// and a run that changed any ends with one line saying to stage them. + +// sourceFileMock records the files its java-action writer "writes", the way the +// modelsdk backend does, and hands them over on TakeSourceFileChanges. +type sourceFileMock struct { + *mock.MockBackend + pending []backend.SourceFileChange +} + +func (m *sourceFileMock) TakeSourceFileChanges() []backend.SourceFileChange { + c := m.pending + m.pending = nil + return c +} + +// newSourceFileExecutor returns an executor whose backend creates Java actions +// in module Probe. A body equal to unchangedBody is "already on disk" and +// records nothing, as WriteSourceIfChanged does for an identical rewrite. +func newSourceFileExecutor(t *testing.T, out *bytes.Buffer, unchangedBody string) *Executor { + t.Helper() + mod := mkModule("Probe") + m := &sourceFileMock{} + m.MockBackend = &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListJavaActionsFunc: func() ([]*types.JavaAction, error) { return nil, nil }, + ReadJavaActionByNameFunc: func(string) (*javaactions.JavaAction, error) { return nil, nil }, + CreateJavaActionFunc: func(*javaactions.JavaAction) error { return nil }, + WriteJavaSourceFileFunc: func(module, action, code string, _ []*javaactions.JavaActionParameter, + _ javaactions.CodeActionReturnType, _ []string, _ *string, _ string) error { + if strings.TrimSpace(code) != unchangedBody { + m.pending = append(m.pending, backend.SourceFileChange{ + Path: "javasource/" + strings.ToLower(module) + "/actions/" + action + ".java", + What: "Java action body", + }) + } + return nil + }, + } + e := New(out) + e.backend = m + return e +} + +func buildProgram(t *testing.T, src string) *ast.Program { + t.Helper() + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs[0]) + } + return prog +} + +const twoJavaActions = `create java action Probe.JA_Hello () returns String as $$ +return "hi"; +$$; +create java action Probe.JA_Bye () returns String as $$ +return "bye"; +$$; +` + +func TestExecNamesEveryFileItWritesBesideTheModel(t *testing.T) { + var out bytes.Buffer + e := newSourceFileExecutor(t, &out, "") + if err := e.ExecuteProgram(buildProgram(t, twoJavaActions)); err != nil { + t.Fatal(err) + } + want := "Created java action: Probe.JA_Hello\n" + + "Wrote javasource/probe/actions/JA_Hello.java (Java action body; commit it with the model)\n" + + "Created java action: Probe.JA_Bye\n" + + "Wrote javasource/probe/actions/JA_Bye.java (Java action body; commit it with the model)\n" + + "Files changed outside the model: 2 (javasource/). Stage them with the .mpr.\n" + if got := out.String(); got != want { + t.Errorf("exec output:\n%s\nwant:\n%s", got, want) + } +} + +// The control: a body already on disk is not rewritten, so there is nothing to +// stage and nothing is said — not a "Wrote" line, not a summary. +func TestExecSaysNothingAboutAnUnchangedSourceFile(t *testing.T) { + var out bytes.Buffer + e := newSourceFileExecutor(t, &out, `return "hi";`) + if err := e.ExecuteProgram(buildProgram(t, twoJavaActions)); err != nil { + t.Fatal(err) + } + got := out.String() + if strings.Contains(got, "JA_Hello.java") { + t.Errorf("an unchanged file was reported:\n%s", got) + } + if !strings.Contains(got, "Wrote javasource/probe/actions/JA_Bye.java") { + t.Errorf("control: the changed file was not reported:\n%s", got) + } + if !strings.Contains(got, "Files changed outside the model: 1 (javasource/)") { + t.Errorf("the summary does not count only the changed file:\n%s", got) + } + + out.Reset() + e = newSourceFileExecutor(t, &out, `return "hi";`) + if err := e.ExecuteProgram(buildProgram(t, "create java action Probe.JA_Hello () returns String as $$\nreturn \"hi\";\n$$;\n")); err != nil { + t.Fatal(err) + } + if got := out.String(); strings.Contains(got, "Wrote") || strings.Contains(got, "outside the model") { + t.Errorf("a run that changed no file beside the model mentioned one:\n%s", got) + } +} + +// One statement run on its own (the REPL) names its file; the summary belongs to +// a program run, where the per-file lines may have scrolled away. +func TestASingleStatementNamesItsFileWithoutASummary(t *testing.T) { + var out bytes.Buffer + e := newSourceFileExecutor(t, &out, "") + prog := buildProgram(t, "create java action Probe.JA_Hello () returns String as $$\nreturn \"hi\";\n$$;\n") + if err := e.Execute(prog.Statements[0]); err != nil { + t.Fatal(err) + } + want := "Created java action: Probe.JA_Hello\n" + + "Wrote javasource/probe/actions/JA_Hello.java (Java action body; commit it with the model)\n" + if got := out.String(); got != want { + t.Errorf("output:\n%s\nwant:\n%s", got, want) + } +} + +// `mxcli test` injects a Java action into the project and deletes it again at +// cleanup; telling its user to commit that file would be wrong. A caller whose +// writes are temporary turns the report off — and what the backend recorded is +// still taken, so it is not reported by a later statement either. +func TestSourceFileReportCanBeTurnedOff(t *testing.T) { + var out bytes.Buffer + e := newSourceFileExecutor(t, &out, "") + e.SetSourceFileReport(false) + if err := e.ExecuteProgram(buildProgram(t, twoJavaActions)); err != nil { + t.Fatal(err) + } + want := "Created java action: Probe.JA_Hello\nCreated java action: Probe.JA_Bye\n" + if got := out.String(); got != want { + t.Errorf("output with the report off:\n%s\nwant:\n%s", got, want) + } + + // Turned back on, the next run reports only its own writes. + out.Reset() + e.SetSourceFileReport(true) + if err := e.ExecuteProgram(buildProgram(t, "create java action Probe.JA_Third () returns String as $$\nreturn \"3\";\n$$;\n")); err != nil { + t.Fatal(err) + } + if got := out.String(); strings.Contains(got, "JA_Hello") || strings.Contains(got, "JA_Bye") || + !strings.Contains(got, "Files changed outside the model: 1 (javasource/)") { + t.Errorf("a later run reported writes the silenced run made:\n%s", got) + } +} + +func TestSourceFileChangeLines(t *testing.T) { + for _, c := range []struct { + change backend.SourceFileChange + want string + }{ + {backend.SourceFileChange{Path: "javascriptsource/probe/actions/JS_Hello.js", What: "JavaScript action body"}, + "Wrote javascriptsource/probe/actions/JS_Hello.js (JavaScript action body; commit it with the model)"}, + {backend.SourceFileChange{Path: "javasource/probe/actions/JA_New.java", From: "javasource/probe/actions/JA_Old.java", What: "Java action body"}, + "Moved javasource/probe/actions/JA_Old.java to javasource/probe/actions/JA_New.java (Java action body; commit both paths with the model)"}, + {backend.SourceFileChange{Path: "javasource/probe/", Removed: true, What: "module Java source"}, + "Removed javasource/probe/ (module Java source; commit the deletion with the model)"}, + } { + if got := sourceFileChangeLine(c.change); got != c.want { + t.Errorf("%+v:\n got %q\nwant %q", c.change, got, c.want) + } + } +} + +func TestSourceFileSummaryCountsPathsOnceAndNamesEveryFolder(t *testing.T) { + var log sourceFileLog + log.begin() + log.add([]backend.SourceFileChange{ + {Path: "javasource/probe/actions/JA_Hello.java"}, + {Path: "javascriptsource/probe/actions/JS_Hello.js"}, + {Path: "javasource/probe/actions/JA_Hello.java"}, // written twice in one run + {Path: "themesource/probe/", Removed: true}, + }) + var out bytes.Buffer + log.flush(&out) + want := "Files changed outside the model: 3 (javasource/, javascriptsource/, themesource/). Stage them with the .mpr.\n" + if got := out.String(); got != want { + t.Errorf("summary %q, want %q", got, want) + } + out.Reset() + log.flush(&out) + if out.Len() != 0 { + t.Errorf("a second flush repeated the summary: %q", out.String()) + } +} diff --git a/mdl/executor/unit_reads_per_command_pedapp_test.go b/mdl/executor/unit_reads_per_command_pedapp_test.go new file mode 100644 index 0000000000..e855d1c0d4 --- /dev/null +++ b/mdl/executor/unit_reads_per_command_pedapp_test.go @@ -0,0 +1,165 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "context" + "encoding/json" + "path/filepath" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + modelsdkbackend "github.com/mendixlabs/mxcli/mdl/backend/modelsdk" + "github.com/mendixlabs/mxcli/model" +) + +// ako/mxcli#1791: on MPR v2 nearly every command opened most of the project's +// unit files — `show modules` 4,511 of a 4,753-unit app's, `describe +// microflow` 2,996 — to read names (every folder for the hierarchy, every +// microflow for the name map and the by-name lookup) or to count documents. +// Each command runs in a fresh process, and an open costs a scan on Windows, +// so this was a floor under every call an agent makes. A fresh process now +// names what the persisted unit index has seen without opening it. + +// openPedAppAgain connects a second executor to dir, as the next command's +// process would: nothing is shared but what is on disk. +func openPedAppAgain(t *testing.T, dir string) (*Executor, *bytes.Buffer) { + t.Helper() + out := &bytes.Buffer{} + exec := New(out) + exec.SetBackendFactory(func() backend.FullBackend { return modelsdkbackend.New() }) + if err := exec.Execute(&ast.ConnectStmt{Path: filepath.Join(dir, "PedApp.mpr")}); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = exec.Execute(&ast.DisconnectStmt{}) }) + return exec, out +} + +func TestFreshProcess_CommandsOpenOnlyWhatTheyShow(t *testing.T) { + first, out, dir := openPedAppCopy(t) + // The first process learns every unit and persists the index. + if err := agreeExec(t, first, "show modules;"); err != nil { + t.Fatalf("warm: %v\n%s", err, out.String()) + } + hs, err := microflowHeaders(first.newExecContext(context.Background())) + if err != nil || len(hs) < 10 { + t.Fatalf("control: the fixture has %d microflows (%v); the bounds below would prove nothing", len(hs), err) + } + h, err := getHierarchy(first.newExecContext(context.Background())) + if err != nil { + t.Fatal(err) + } + target := h.GetQualifiedName(hs[0].ContainerID, hs[0].Name) + + // Measured on this fixture (375 units, 16 microflows): before the fix + // `show modules` opened 196 files and the describe 113; after, 56 and 18. + // The bounds sit between, so a regression to either listing fails. + for _, c := range []struct { + stmt string + max int64 + }{ + // The modules, their settings and domain models: no folder, no flow, + // no page is opened to name or count it. + {"show modules;", 80}, + // The microflow itself and the domain models its names resolve + // against: not every microflow to find it by name. + {"describe microflow " + target + ";", 30}, + } { + exec, out := openPedAppAgain(t, dir) + before := unitFileReads(t, exec) + if err := agreeExec(t, exec, c.stmt); err != nil { + t.Fatalf("%s: %v\n%s", c.stmt, err, out.String()) + } + if reads := unitFileReads(t, exec) - before; reads > c.max { + t.Errorf("%s opened %d unit files in a fresh process, want at most %d; the project has %d microflows", c.stmt, reads, c.max, len(hs)) + } + } +} + +// `show modules` counts documents from the unit types instead of decoding +// every document; the counts must be the ones the full listings give, +// including the System module's built-in enumerations and Java actions, which +// have no unit. +func TestShowModules_CountsAgreeWithTheListings(t *testing.T) { + exec, out, _ := openPedAppCopy(t) + exec.SetFormat(FormatJSON) + out.Reset() // the connect message + if err := agreeExec(t, exec, "show modules;"); err != nil { + t.Fatalf("show modules: %v\n%s", err, out.String()) + } + var rows []map[string]any + if err := json.Unmarshal(out.Bytes(), &rows); err != nil { + t.Fatalf("parse: %v\n%s", err, out.String()) + } + shown := map[string]map[string]int{} + for _, r := range rows { + m := map[string]int{} + for k, v := range r { + if n, ok := v.(float64); ok { + m[k] = int(n) + } + } + shown[r["Module"].(string)] = m + } + + ctx := exec.newExecContext(context.Background()) + h, err := getHierarchy(ctx) + if err != nil { + t.Fatal(err) + } + b := ctx.Backend + want := map[string]map[string]int{} + add := func(col string, container model.ID) { + mod := h.GetModuleName(h.FindModuleID(container)) + if want[mod] == nil { + want[mod] = map[string]int{} + } + want[mod][col]++ + } + enums, _ := b.ListEnumerations() + for _, e := range enums { + add("Enums", e.ContainerID) + } + pages, _ := b.ListPages() + for _, p := range pages { + add("Pages", p.ContainerID) + } + snippets, _ := b.ListSnippets() + for _, s := range snippets { + add("Snippets", s.ContainerID) + } + mfs, _ := b.ListMicroflows() + for _, mf := range mfs { + add("Microflows", mf.ContainerID) + } + nfs, _ := b.ListNanoflows() + for _, nf := range nfs { + add("Nanoflows", nf.ContainerID) + } + constants, _ := b.ListConstants() + for _, c := range constants { + add("Constants", c.ContainerID) + } + jas, _ := b.ListJavaActions() + for _, ja := range jas { + add("JavaActions", ja.ContainerID) + } + if want["System"]["Enums"] == 0 || want["System"]["JavaActions"] == 0 { + t.Fatalf("control: the listings give System no built-ins; the System case is not exercised") + } + + checked := 0 + for mod, cols := range want { + for col, n := range cols { + if shown[mod][col] != n { + t.Errorf("show modules: %s %s = %d, the listing has %d", mod, col, shown[mod][col], n) + } + checked++ + } + } + if checked < 10 { + t.Fatalf("control: only %d module/column counts compared", checked) + } +} diff --git a/mdl/executor/validate_microflow_ce_gaps.go b/mdl/executor/validate_microflow_ce_gaps.go index 4d078c2b19..b7958cb37f 100644 --- a/mdl/executor/validate_microflow_ce_gaps.go +++ b/mdl/executor/validate_microflow_ce_gaps.go @@ -317,6 +317,37 @@ func (v *microflowValidator) checkDuplicateVariableNames(params []ast.MicroflowP walk(body) } +// FlowDefinedVariables returns every variable name the statements define, +// nested bodies included, each once and in the order first defined: the +// definitions MDL063 and MDL052 count. A flow's variable namespace is flat, so +// two of these with one name are CE0111. +// +// It leaves out MDL063's two exemptions, which need resolved types (a void +// call's output name, and a String contains/find that builds as a Change +// Variable): its caller renames a body's variables per copy — the `mxcli test` +// after-startup runner, which joins every test into one flow (ako/mxcli#1775) — +// and renaming one name too many costs nothing, where missing one is CE0111. +func FlowDefinedVariables(body []ast.MicroflowStatement) []string { + var names []string + seen := map[string]bool{} + var walk func(stmts []ast.MicroflowStatement) + walk = func(stmts []ast.MicroflowStatement) { + for _, s := range stmts { + for _, p := range statementProducedVars(s) { + if !seen[p.name] { + seen[p.name] = true + names = append(names, p.name) + } + } + for _, nested := range ast.StatementBodies(s) { + walk(nested) + } + } + } + walk(body) + return names +} + // statementProducedVars returns the variables a statement CREATES. // // The `OutputVariable` field is read by reflection rather than from a switch: diff --git a/mdl/executor/validate_void_code_calls.go b/mdl/executor/validate_void_code_calls.go index ca00923749..e14e3269bf 100644 --- a/mdl/executor/validate_void_code_calls.go +++ b/mdl/executor/validate_void_code_calls.go @@ -63,6 +63,11 @@ type voidCodeActions struct { unknownIsVoid bool // shared, when set, carries project resolutions across runs (the editor). shared *CodeActionCache + // microflowNames, when set, names every stored microflow by ID, so a + // called microflow is read by ID rather than found by a name lookup that + // reads every microflow — once per call, per describe (#1789). + microflowNames func() map[model.ID]string + microflowIDs map[string]model.ID // declared types: which qualified names are enumerations and which are // entities, for MDL043's wording (validate_declare_enum.go). The script's @@ -171,6 +176,17 @@ func (r *voidCodeActions) resolveFlow(nanoflow bool, qn string) voidness { if nanoflow { objectType = "nanoflow" } + if id, ok := r.microflowID(nanoflow, qn); ok { + contents, err := b.GetRawUnitBytes(id) + if err != nil || len(contents) == 0 { + return voidness{} + } + mf, err := b.ParseMicroflowBSON(contents, id, "") + if err != nil || mf == nil { + return voidness{} + } + return voidness{void: flowReturnsVoid(mf.ReturnType), known: true} + } raw, err := b.GetRawUnitByName(objectType, qn) if err != nil || raw == nil || len(raw.Contents) == 0 { return voidness{} @@ -183,6 +199,27 @@ func (r *voidCodeActions) resolveFlow(nanoflow bool, qn string) voidness { }) } +// microflowID is the ID of the stored microflow named qn, from microflowNames +// inverted on first use. A name two microflows share (an excluded twin) has no +// answer here, so the by-name lookup decides between them as before. +func (r *voidCodeActions) microflowID(nanoflow bool, qn string) (model.ID, bool) { + if nanoflow || r.microflowNames == nil { + return "", false + } + if r.microflowIDs == nil { + r.microflowIDs = map[string]model.ID{} + for id, name := range r.microflowNames() { + if _, dup := r.microflowIDs[name]; dup { + r.microflowIDs[name] = "" + } else { + r.microflowIDs[name] = id + } + } + } + id := r.microflowIDs[qn] + return id, id != "" +} + // lookup answers key from the script, the caches, and finally the project // through read. func (r *voidCodeActions) lookup(key string, read func(backend.FullBackend) voidness) voidness { diff --git a/mdl/executor/version_gates.go b/mdl/executor/version_gates.go index e2afaec8ec..24fd772fdb 100644 --- a/mdl/executor/version_gates.go +++ b/mdl/executor/version_gates.go @@ -159,6 +159,9 @@ func statementVersionGates(stmt ast.Statement) []versionGate { gates = append(gates, versionGate{"microflows", "deep_link_url", "a microflow URL (deep link)", "upgrade your project to 10.6+, or set the URL in Studio Pro"}) } + if bodyContains(s.Body, selectsWorkflowDefinition) { + gates = append(gates, workflowDefinitionSelectionGate) + } return gates case *ast.CreateNanoflowStmt: // SYNCHRONIZE UNSYNCHRONIZED is the one mode with a floor inside @@ -201,6 +204,7 @@ func statementVersionGates(stmt ast.Statement) []versionGate { if workflowUsesNotificationEvents(activities) { gates = append(gates, notificationEventsGate) } + gates = append(gates, workflowTypeGates(activities, nil)...) return append(gates, userTaskGates(activities)...) case *ast.AlterWorkflowStmt: // Workflows require Mendix 9.12+. @@ -213,11 +217,96 @@ func statementVersionGates(stmt ast.Statement) []versionGate { if workflowUsesNotificationEvents(added) { gates = append(gates, notificationEventsGate) } + var inserted []string + for _, op := range s.Operations { + if o, ok := op.(*ast.InsertBoundaryEventOp); ok { + inserted = append(inserted, o.EventType) + } + } + gates = append(gates, workflowTypeGates(added, inserted)...) return append(gates, userTaskGates(added)...) } return nil } +// The gates of the workflow activity and event TYPES mxcli writes that an +// older Mendix does not have (ako/mxcli#1774). Each type's introduction is in +// the Model SDK's version data (mendixmodelsdk 4.117.0), and the write path +// refuses the type below it (canon.UndeclaredTypeError); these name the +// statement first. +var ( + waitForNotificationGate = versionGate{"workflows", "wait_for_notification", "wait for notification", + "Workflows$WaitForNotificationActivity arrived in Mendix 10.0: upgrade the project"} + waitForTimerGate = versionGate{"workflows", "wait_for_timer", "wait for timer", + "Workflows$WaitForTimerActivity arrived in Mendix 10.6: upgrade the project"} + timerBoundaryEventGate = versionGate{"workflows", "timer_boundary_event", "boundary event timer", + "boundary events arrived in Mendix 10.14: upgrade the project"} + interruptingTimerBoundaryEventGate = versionGate{"workflows", "interrupting_timer_boundary_event", + "boundary event interrupting timer / non interrupting timer", + "the (non-)interrupting timer boundary events are Mendix 10.20 types; 10.14–10.19 store a plain timer " + + "boundary event, which mxcli writes for `boundary event timer …`, or upgrade the project"} + workflowDefinitionSelectionGate = versionGate{"workflows", "workflow_definition_selection", + "lock workflow / unlock workflow on a workflow", + "before Mendix 10.0 the lock and unlock actions name their workflow differently, which mxcli does not write: " + + "set it in Studio Pro, or upgrade the project"} +) + +// workflowTypeGates are the gates of the wait activities and timer boundary +// events among activities, and of the boundary events an `alter workflow` +// inserts (their EventTypes), each once. +func workflowTypeGates(activities []ast.WorkflowActivityNode, insertedEvents []string) []versionGate { + var notification, timer, plainTimer, interruptingTimer bool + event := func(eventType string) { + switch eventType { + case "Timer": + plainTimer = true + case "", "InterruptingTimer", "NonInterruptingTimer": + interruptingTimer = true + } + } + walkWorkflowActivities(activities, func(a ast.WorkflowActivityNode) { + switch a.(type) { + case *ast.WorkflowWaitForNotificationNode: + notification = true + case *ast.WorkflowWaitForTimerNode: + timer = true + } + for _, e := range workflowActivityBoundaryEvents(a) { + event(e.EventType) + } + }) + for _, e := range insertedEvents { + event(e) + } + var gates []versionGate + if notification { + gates = append(gates, waitForNotificationGate) + } + if timer { + gates = append(gates, waitForTimerGate) + } + if plainTimer { + gates = append(gates, timerBoundaryEventGate) + } + if interruptingTimer { + gates = append(gates, interruptingTimerBoundaryEventGate) + } + return gates +} + +// selectsWorkflowDefinition reports a lock or unlock that names its workflow +// (by name or by variable): the action stores it as a +// Workflows$WorkflowDefinition*Selection, a 10.0 type. +func selectsWorkflowDefinition(st ast.MicroflowStatement) bool { + switch s := st.(type) { + case *ast.LockWorkflowStmt: + return s.Workflow != "" || s.WorkflowVariable != "" + case *ast.UnlockWorkflowStmt: + return s.Workflow != "" || s.WorkflowVariable != "" + } + return false +} + // userTaskGates are the gates of what the user tasks among activities state // that an older user task cannot store, each once (ako/mxcli#1771). Below 11.2 // a task targets users only, and a multi-user task, its later rules and diff --git a/mdl/executor/version_gates_workflow_types_1774_test.go b/mdl/executor/version_gates_workflow_types_1774_test.go new file mode 100644 index 0000000000..ba12ab67f2 --- /dev/null +++ b/mdl/executor/version_gates_workflow_types_1774_test.go @@ -0,0 +1,100 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// ako/mxcli#1774: the workflow activity and event TYPES older Mendix does not +// have are refused by `check -p` and by exec's pre-flight with MDL-VERSION01, +// before anything is written. The write path refuses the types themselves +// (canon.UndeclaredTypeError); these name the statement. Each floor is the +// type's own introduction in mendixmodelsdk 4.117.0: +// - Workflows$WaitForNotificationActivity 10.0.0; +// - Workflows$WaitForTimerActivity 10.6.0; +// - Workflows$TimerBoundaryEvent 10.14.0 (a bare `boundary event timer`); +// - Workflows$(Non)InterruptingTimerBoundaryEvent 10.20.0; +// - Workflows$WorkflowDefinition(Name|Object)Selection 10.0.0 (lock/unlock). +func TestCheckVersionGates_WorkflowTypes(t *testing.T) { + onTask := func(eventType string) *ast.WorkflowUserTaskNode { + return &ast.WorkflowUserTaskNode{Name: "Review", Caption: "Review", + BoundaryEvents: []ast.WorkflowBoundaryEventNode{{EventType: eventType, Delay: "addDays([%CurrentDateTime%], 1)"}}} + } + inSplit := func(a ast.WorkflowActivityNode) ast.WorkflowActivityNode { + return &ast.WorkflowParallelSplitNode{Paths: []ast.WorkflowParallelPathNode{{Activities: []ast.WorkflowActivityNode{a}}}} + } + for _, c := range []struct { + name string + act ast.WorkflowActivityNode + below, floor [3]int + want string + }{ + {"wait for notification", &ast.WorkflowWaitForNotificationNode{Name: "w"}, [3]int{9, 24, 30}, [3]int{10, 0, 0}, "wait for notification"}, + {"wait for timer", &ast.WorkflowWaitForTimerNode{Name: "w", DelayExpression: "addHours([%CurrentDateTime%], 1)"}, [3]int{10, 5, 0}, [3]int{10, 6, 0}, "wait for timer"}, + {"nested wait for timer", inSplit(&ast.WorkflowWaitForTimerNode{Name: "w"}), [3]int{10, 5, 0}, [3]int{10, 6, 0}, "wait for timer"}, + {"interrupting timer", onTask("InterruptingTimer"), [3]int{10, 19, 0}, [3]int{10, 20, 0}, "interrupting timer"}, + {"non interrupting timer", onTask("NonInterruptingTimer"), [3]int{10, 19, 0}, [3]int{10, 20, 0}, "interrupting timer"}, + {"bare timer", onTask("Timer"), [3]int{10, 13, 0}, [3]int{10, 14, 0}, "boundary event timer"}, + {"timer on a wait for notification", &ast.WorkflowWaitForNotificationNode{Name: "w", + BoundaryEvents: []ast.WorkflowBoundaryEventNode{{EventType: "InterruptingTimer"}}}, [3]int{10, 19, 0}, [3]int{10, 20, 0}, "interrupting timer"}, + } { + for _, stmt := range []ast.Statement{ + &ast.CreateWorkflowStmt{Activities: []ast.WorkflowActivityNode{c.act}}, + &ast.AlterWorkflowStmt{Operations: []ast.AlterWorkflowOp{ + &ast.InsertAfterOp{NewActivities: []ast.WorkflowActivityNode{c.act}}}}, + } { + prog := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, stmt}} + vs := versionGateViolations(versionCtx(t, c.below[0], c.below[1], c.below[2]), prog) + if len(vs) != 1 || vs[0].RuleID != "MDL-VERSION01" || !strings.Contains(vs[0].Message, c.want) || vs[0].Suggestion == "" { + t.Errorf("%s (%T) below its floor: want one MDL-VERSION01 naming %q with a hint, got %+v", c.name, stmt, c.want, vs) + } + if vs := versionGateViolations(versionCtx(t, c.floor[0], c.floor[1], c.floor[2]), prog); len(vs) != 0 { + t.Errorf("%s (%T): reported at its floor %v: %+v", c.name, stmt, c.floor, vs) + } + } + } + + // A boundary event `alter workflow` inserts is the op's own, not an + // activity's; an empty kind is an interrupting timer. + for _, eventType := range []string{"", "InterruptingTimer", "NonInterruptingTimer"} { + prog := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, + &ast.AlterWorkflowStmt{Operations: []ast.AlterWorkflowOp{&ast.InsertBoundaryEventOp{ActivityRef: "Review", EventType: eventType}}}}} + vs := versionGateViolations(versionCtx(t, 10, 19, 0), prog) + if len(vs) != 1 || !strings.Contains(vs[0].Message, "interrupting timer") { + t.Errorf("insert boundary event %q on 10.19: %+v", eventType, vs) + } + if vs := versionGateViolations(versionCtx(t, 10, 20, 0), prog); len(vs) != 0 { + t.Errorf("insert boundary event %q on 10.20: %+v", eventType, vs) + } + } +} + +// A lock or unlock naming its workflow stores a 10.0 selection type; one +// naming none (`lock workflow all`) is refused elsewhere, and is not this gate. +func TestCheckVersionGates_WorkflowDefinitionSelection(t *testing.T) { + for _, st := range []ast.MicroflowStatement{ + &ast.LockWorkflowStmt{Workflow: "M.WF"}, + &ast.LockWorkflowStmt{WorkflowVariable: "WF", PauseAllWorkflows: true}, + &ast.UnlockWorkflowStmt{Workflow: "M.WF"}, + &ast.IfStmt{ThenBody: []ast.MicroflowStatement{&ast.UnlockWorkflowStmt{WorkflowVariable: "WF"}}}, + } { + prog := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, + &ast.CreateMicroflowStmt{Body: []ast.MicroflowStatement{st}}}} + vs := versionGateViolations(versionCtx(t, 9, 24, 30), prog) + if len(vs) != 1 || !strings.Contains(vs[0].Message, "lock workflow / unlock workflow on a workflow") || vs[0].Suggestion == "" { + t.Errorf("%T on 9.24: %+v", st, vs) + } + if vs := versionGateViolations(versionCtx(t, 10, 0, 0), prog); len(vs) != 0 { + t.Errorf("%T on 10.0: %+v", st, vs) + } + } + bare := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, + &ast.CreateMicroflowStmt{Body: []ast.MicroflowStatement{&ast.LockWorkflowStmt{PauseAllWorkflows: true}}}}} + if vs := versionGateViolations(versionCtx(t, 9, 24, 30), bare); len(vs) != 0 { + t.Errorf("a lock naming no workflow reported by the selection gate: %+v", vs) + } +} diff --git a/modelsdk/canon/gen_type_versions.go b/modelsdk/canon/gen_type_versions.go new file mode 100644 index 0000000000..82bb3ac534 --- /dev/null +++ b/modelsdk/canon/gen_type_versions.go @@ -0,0 +1,425 @@ +//go:build ignore + +// SPDX-License-Identifier: Apache-2.0 + +// gen_type_versions extracts every metamodel TYPE's own introduction and +// deletion version from the Mendix Model SDK into +// mendixmodelsdk_type_versions.json, which the type floor (typefloor.go) reads +// (ako/mxcli#1774). +// +// The generated version data (modelsdk/version, from modelsdk/gen/*/version.go) +// holds property versions only, and the generated packages cannot be +// regenerated here (they carry hand fixes, ako/mxcli#1163), so the type data +// comes straight from the SDK package. It is read as TEXT — never executed — +// from each `X.versionInfo = new exports.StructureVersionInfo({...})` in +// src/gen/*.js: the object's top-level `introduced` and `deleted` (the nested +// ones under `properties` are property versions, already in modelsdk/version). +// +// # Storage names +// +// The SDK names a type by its qualified name; a document stores it under its +// storage name, which differs for some (Pages$ is stored as Forms$, a +// CreateObjectAction as Microflows$CreateChangeAction, …). The storage names are +// taken from modelsdk/gen/*/types.go (each initX's SetTypeName). A storage name +// can then belong to more than one SDK type — a type the SDK renamed keeps its +// storage name (Workflows$CallMicroflowTask is the storage name of both the +// SDK's CallMicroflowTask, deleted 11.9, and CallMicroflowActivity, introduced +// 11.9). So each row is the UNION of every SDK type that can be stored under its +// name: introduced is the earliest of them (none if any of them has no +// introduction), deleted only if all of them are deleted, the latest. Both +// directions can only loosen a floor, never tighten one. A storage name only an +// alias reaches gets no row at all (see main). +// +// The rows are checked against Studio Pro-authored projects by +// modelsdk/mpr's TestTypeFloor_StoredProjectsHoldNoUndeclaredType; run it with +// MXCLI_TYPEFLOOR_PROJECTS after a refresh. +// +// Refresh: +// +// npm pack mendixmodelsdk@ # in a scratch directory +// cd modelsdk/canon +// go run gen_type_versions.go -sdk /path/to/mendixmodelsdk-.tgz +package main + +import ( + "archive/tar" + "bytes" + "compress/gzip" + "encoding/json" + "flag" + "fmt" + "io" + "os" + "path" + "path/filepath" + "regexp" + "sort" + "strconv" + "strings" +) + +// storageDomain is the storage domain of an SDK domain stored under another +// name as a whole. +var storageDomain = map[string]string{"Pages": "Forms"} + +type sdkType struct { + file, qualified string + introduced, deleted string + hasVersionInfo, used bool +} + +func main() { + sdk := flag.String("sdk", "", "mendixmodelsdk .tgz (npm pack) or its extracted package directory") + gen := flag.String("gen", "../gen", "the generated metamodel packages (storage names)") + out := flag.String("o", "mendixmodelsdk_type_versions.json", "output file") + flag.Parse() + if *sdk == "" { + fail("-sdk is required") + } + files, pkgVersion, err := readSDK(*sdk) + if err != nil { + fail("%v", err) + } + var types []*sdkType + names := make([]string, 0, len(files)) + for n := range files { + names = append(names, n) + } + sort.Strings(names) + for _, n := range names { + ts, err := parseFile(n, files[n]) + if err != nil { + fail("%s: %v", n, err) + } + types = append(types, ts...) + } + aliases, err := storageAliases(*gen) + if err != nil { + fail("%v", err) + } + + // candidates: storage name → every SDK type that can be stored under it. + candidates := map[string][]*sdkType{} + add := func(storage string, t *sdkType) { + for _, c := range candidates[storage] { + if c == t { + return + } + } + candidates[storage] = append(candidates[storage], t) + } + for _, t := range types { + add(defaultStorage(t.qualified), t) + } + // An aliased type joins the row of its storage name only where that name + // is also some SDK type's own: a name only an alias reaches can be the + // storage of an older type the SDK does not list at all, and a row from + // the alias alone would be a floor that type never had. Forms$FormCallArgument + // is the measured case: the alias of the SDK's PageParameterMapping (9.7), + // stored by Studio Pro 9.1.1 and 9.4.0 (Marketplace 117785, 117680). + for _, t := range types { + _, name, _ := strings.Cut(t.qualified, "$") + if s, ok := aliases[t.file+"."+name]; ok && s != defaultStorage(t.qualified) { + if _, own := candidates[s]; own { + add(s, t) + } + } + } + + type row struct { + Type string `json:"type"` + Introduced string `json:"introduced,omitempty"` + Deleted string `json:"deleted,omitempty"` + SDK []string `json:"sdk,omitempty"` + } + var rows []row + for storage, cs := range candidates { + r := row{Type: storage} + intro, del := "", "" + always, live := false, false + for _, c := range cs { + if c.qualified != storage { + r.SDK = append(r.SDK, c.qualified) + } + if !c.hasVersionInfo || c.introduced == "" { + always = true + } else if intro == "" || less(c.introduced, intro) { + intro = c.introduced + } + if c.deleted == "" { + live = true + } else if del == "" || less(del, c.deleted) { + del = c.deleted + } + } + if !always { + r.Introduced = intro + } + if !live { + r.Deleted = del + } + if r.Introduced == "" && r.Deleted == "" { + continue + } + sort.Strings(r.SDK) + rows = append(rows, r) + } + sort.Slice(rows, func(i, j int) bool { return rows[i].Type < rows[j].Type }) + + var b bytes.Buffer + fmt.Fprintf(&b, "{\n \"source\": %q,\n \"types\": [\n", "mendixmodelsdk@"+pkgVersion) + for i, r := range rows { + line, err := json.Marshal(r) + if err != nil { + fail("%v", err) + } + sep := "," + if i == len(rows)-1 { + sep = "" + } + fmt.Fprintf(&b, " %s%s\n", line, sep) + } + b.WriteString(" ]\n}\n") + if err := os.WriteFile(*out, b.Bytes(), 0o644); err != nil { + fail("%v", err) + } + fmt.Fprintf(os.Stderr, "gen_type_versions: %d SDK types, %d rows from mendixmodelsdk %s\n", len(types), len(rows), pkgVersion) +} + +// defaultStorage is the storage name of an SDK type with no alias: its +// qualified name, in its domain's storage name. +func defaultStorage(qualified string) string { + dom, name, _ := strings.Cut(qualified, "$") + if d, ok := storageDomain[dom]; ok { + dom = d + } + return dom + "$" + name +} + +// readSDK returns the package's src/gen/*.js files by base name, and its +// version, from a .tgz or an extracted directory. +func readSDK(p string) (map[string][]byte, string, error) { + files := map[string][]byte{} + var pkgJSON []byte + st, err := os.Stat(p) + if err != nil { + return nil, "", err + } + if st.IsDir() { + matches, _ := filepath.Glob(filepath.Join(p, "src", "gen", "*.js")) + for _, m := range matches { + b, err := os.ReadFile(m) + if err != nil { + return nil, "", err + } + files[strings.TrimSuffix(filepath.Base(m), ".js")] = b + } + if pkgJSON, err = os.ReadFile(filepath.Join(p, "package.json")); err != nil { + return nil, "", err + } + } else { + f, err := os.Open(p) + if err != nil { + return nil, "", err + } + defer f.Close() + gz, err := gzip.NewReader(f) + if err != nil { + return nil, "", err + } + tr := tar.NewReader(gz) + for { + h, err := tr.Next() + if err == io.EOF { + break + } + if err != nil { + return nil, "", err + } + name := path.Clean(h.Name) + switch { + case name == "package/package.json": + if pkgJSON, err = io.ReadAll(tr); err != nil { + return nil, "", err + } + case path.Dir(name) == "package/src/gen" && strings.HasSuffix(name, ".js"): + b, err := io.ReadAll(tr) + if err != nil { + return nil, "", err + } + files[strings.TrimSuffix(path.Base(name), ".js")] = b + } + } + } + var pkg struct { + Name, Version string + } + if err := json.Unmarshal(pkgJSON, &pkg); err != nil || pkg.Name != "mendixmodelsdk" || pkg.Version == "" { + return nil, "", fmt.Errorf("%s is not a mendixmodelsdk package (%v)", p, err) + } + if len(files) == 0 { + return nil, "", fmt.Errorf("%s: no src/gen/*.js", p) + } + return files, pkg.Version, nil +} + +var ( + reTypeName = regexp.MustCompile(`(\w+)\.structureTypeName = "([^"]+)";`) + reVersionInfo = regexp.MustCompile(`(\w+)\.versionInfo = new exports\.StructureVersionInfo\(`) +) + +// parseFile returns the types one src/gen/.js declares, with the +// top-level introduced/deleted of each one's StructureVersionInfo. +func parseFile(file string, src []byte) ([]*sdkType, error) { + byClass := map[string]*sdkType{} + var order []*sdkType + for _, m := range reTypeName.FindAllSubmatch(src, -1) { + cls, q := string(m[1]), string(m[2]) + if !strings.Contains(q, "$") { + return nil, fmt.Errorf("structure type name %q has no domain", q) + } + if _, dup := byClass[cls]; dup { + return nil, fmt.Errorf("class %s declared twice", cls) + } + t := &sdkType{file: file, qualified: q} + byClass[cls] = t + order = append(order, t) + } + for _, loc := range reVersionInfo.FindAllSubmatchIndex(src, -1) { + cls := string(src[loc[2]:loc[3]]) + t, ok := byClass[cls] + if !ok { + return nil, fmt.Errorf("versionInfo for %s, which has no structureTypeName", cls) + } + top, err := topLevelStrings(src[loc[1]:]) + if err != nil { + return nil, fmt.Errorf("%s.versionInfo: %w", cls, err) + } + t.hasVersionInfo = true + t.introduced, t.deleted = top["introduced"], top["deleted"] + for _, v := range []string{t.introduced, t.deleted} { + if v != "" && !validVersion(v) { + return nil, fmt.Errorf("%s: version %q", t.qualified, v) + } + } + } + return order, nil +} + +// topLevelStrings reads the object literal src starts with and returns its +// top-level keys whose values are string literals. +func topLevelStrings(src []byte) (map[string]string, error) { + out := map[string]string{} + depth := 0 + key := "" + for i := 0; i < len(src); i++ { + c := src[i] + switch { + case c == '"' || c == '\'': + j := i + 1 + for ; j < len(src) && src[j] != c; j++ { + if src[j] == '\\' { + j++ + } + } + if j >= len(src) { + return nil, fmt.Errorf("unterminated string") + } + if depth == 1 && key != "" { + out[key] = string(src[i+1 : j]) + } + key = "" + i = j + case c == '{' || c == '[': + depth++ + key = "" + case c == '}' || c == ']': + depth-- + key = "" + if depth == 0 { + return out, nil + } + case depth == 1 && isIdentStart(c) && (i == 0 || !isIdent(src[i-1])): + j := i + for j < len(src) && isIdent(src[j]) { + j++ + } + k := j + for k < len(src) && (src[k] == ' ' || src[k] == '\t' || src[k] == '\n' || src[k] == '\r') { + k++ + } + if k < len(src) && src[k] == ':' { + key = string(src[i:j]) + i = k + } else { + key = "" + i = j - 1 + } + case c == ',': + key = "" + } + } + return nil, fmt.Errorf("unterminated object") +} + +func isIdentStart(c byte) bool { + return c == '_' || c == '$' || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') +} +func isIdent(c byte) bool { return isIdentStart(c) || (c >= '0' && c <= '9') } + +var reInit = regexp.MustCompile(`(?m)^func init(\w+)\(\) \*\w+ \{\n\to := &\w+\{\}\n\to\.SetTypeName\("([^"]+)"\)`) + +// storageAliases maps "." to the storage name the generated +// codec gives that class (initX's SetTypeName). +func storageAliases(genDir string) (map[string]string, error) { + matches, err := filepath.Glob(filepath.Join(genDir, "*", "types.go")) + if err != nil || len(matches) == 0 { + return nil, fmt.Errorf("no %s/*/types.go (%v)", genDir, err) + } + out := map[string]string{} + for _, m := range matches { + src, err := os.ReadFile(m) + if err != nil { + return nil, err + } + pkg := filepath.Base(filepath.Dir(m)) + for _, sm := range reInit.FindAllSubmatch(src, -1) { + out[pkg+"."+string(sm[1])] = string(sm[2]) + } + } + if len(out) == 0 { + return nil, fmt.Errorf("no storage names found in %s", genDir) + } + return out, nil +} + +func validVersion(v string) bool { + parts := strings.Split(v, ".") + if len(parts) != 3 { + return false + } + for _, p := range parts { + if _, err := strconv.Atoi(p); err != nil { + return false + } + } + return true +} + +// less compares two dotted versions numerically. +func less(a, b string) bool { + pa, pb := strings.Split(a, "."), strings.Split(b, ".") + for i := 0; i < len(pa) && i < len(pb); i++ { + x, _ := strconv.Atoi(pa[i]) + y, _ := strconv.Atoi(pb[i]) + if x != y { + return x < y + } + } + return len(pa) < len(pb) +} + +func fail(format string, args ...any) { + fmt.Fprintf(os.Stderr, "gen_type_versions: "+format+"\n", args...) + os.Exit(1) +} diff --git a/modelsdk/canon/mendixmodelsdk_type_versions.json b/modelsdk/canon/mendixmodelsdk_type_versions.json new file mode 100644 index 0000000000..0001796493 --- /dev/null +++ b/modelsdk/canon/mendixmodelsdk_type_versions.json @@ -0,0 +1,736 @@ +{ + "source": "mendixmodelsdk@4.117.0", + "types": [ + {"type":"AppServices$AppServiceAction","deleted":"9.0.2"}, + {"type":"AppServices$AppServiceActionParameter","deleted":"9.0.2"}, + {"type":"AppServices$ConsumedAppService","deleted":"9.0.2"}, + {"type":"AppServices$Msd","deleted":"9.0.2"}, + {"type":"AppServices$MsdAssociation","deleted":"9.0.2"}, + {"type":"AppServices$MsdAttribute","deleted":"9.0.2"}, + {"type":"AppServices$MsdDomainModel","deleted":"9.0.2"}, + {"type":"AppServices$MsdEntity","deleted":"9.0.2"}, + {"type":"AppServices$MsdEnumeration","deleted":"9.0.2"}, + {"type":"AppServices$MsdEnumerationContainer","deleted":"9.0.2"}, + {"type":"AppServices$MsdEnumerationValue","deleted":"9.0.2"}, + {"type":"AppServices$MsdMetadata","deleted":"9.0.2"}, + {"type":"AppServices$MsdMicroflow","deleted":"9.0.2"}, + {"type":"AppServices$MsdMicroflowParameter","deleted":"9.0.2"}, + {"type":"AppServices$MsdText","deleted":"9.0.2"}, + {"type":"AppServices$MsdVersion","deleted":"9.0.2"}, + {"type":"Authentication$Authentication","introduced":"11.3.0"}, + {"type":"Authentication$AuthenticationDetails","introduced":"11.3.0"}, + {"type":"Authentication$BasicAuthenticationDetails","introduced":"11.3.0"}, + {"type":"Authentication$OAuth20AuthCodeDetails","introduced":"11.3.0"}, + {"type":"Authentication$OAuth20AuthenticationDetails","introduced":"11.3.0"}, + {"type":"Authentication$OAuth20ClientCredentialsDetails","introduced":"11.3.0"}, + {"type":"BusinessEvents$AttributeEnumeration","introduced":"10.0.0"}, + {"type":"BusinessEvents$AttributeEnumerationItem","introduced":"10.0.0"}, + {"type":"BusinessEvents$BusinessEventDefinition","introduced":"9.24.0"}, + {"type":"BusinessEvents$BusinessEventService","introduced":"9.24.0"}, + {"type":"BusinessEvents$Channel","introduced":"9.24.0"}, + {"type":"BusinessEvents$ConsumedBusinessEvent","introduced":"9.11.0","deleted":"9.24.0"}, + {"type":"BusinessEvents$ConsumedBusinessEventService","introduced":"9.8.0","deleted":"9.24.0"}, + {"type":"BusinessEvents$Message","introduced":"9.24.0"}, + {"type":"BusinessEvents$MessageAttribute","introduced":"9.24.0"}, + {"type":"BusinessEvents$PublishedBusinessEventService","introduced":"9.11.0","deleted":"9.24.0"}, + {"type":"BusinessEvents$PublishedChannel","introduced":"9.11.0","deleted":"9.24.0"}, + {"type":"BusinessEvents$PublishedMessage","introduced":"9.11.0","deleted":"9.24.0"}, + {"type":"BusinessEvents$PublishedMessageAttribute","introduced":"9.11.0","deleted":"9.24.0"}, + {"type":"BusinessEvents$ServiceOperation","introduced":"9.24.0"}, + {"type":"ChangeDataCapture$AssociationProperty","introduced":"11.9.0"}, + {"type":"ChangeDataCapture$AttributeProperty","introduced":"11.9.0"}, + {"type":"ChangeDataCapture$EntityChangeDataCaptureService","introduced":"11.5.0"}, + {"type":"ChangeDataCapture$EntityChangeProperty","introduced":"11.9.0"}, + {"type":"ChangeDataCapture$EntityChangeSource","introduced":"11.5.0"}, + {"type":"ChangeDataCapture$Version","introduced":"11.10.0"}, + {"type":"ChangeDataCapture$VersionedEntityChangeDataCaptureService","introduced":"11.10.0"}, + {"type":"ChangeDataCapture$VersionedEntityChangeSource","introduced":"11.10.0"}, + {"type":"ChangeDataCapture$VersionedProperty","introduced":"11.10.0"}, + {"type":"CodeActions$BasicParameterType","introduced":"7.21.0"}, + {"type":"CodeActions$BooleanType","introduced":"7.21.0"}, + {"type":"CodeActions$ConcreteEntityType","introduced":"7.21.0"}, + {"type":"CodeActions$CustomBlobDocumentParameterType","introduced":"11.6.0"}, + {"type":"CodeActions$DateTimeType","introduced":"7.21.0"}, + {"type":"CodeActions$DecimalType","introduced":"7.21.0"}, + {"type":"CodeActions$EntityType","introduced":"7.21.0"}, + {"type":"CodeActions$EntityTypeParameterType","introduced":"7.21.0"}, + {"type":"CodeActions$EnumerationType","introduced":"7.21.0"}, + {"type":"CodeActions$FloatType","introduced":"7.21.0"}, + {"type":"CodeActions$IntegerType","introduced":"7.21.0"}, + {"type":"CodeActions$ListType","introduced":"7.21.0"}, + {"type":"CodeActions$MicroflowActionInfo","introduced":"7.21.0"}, + {"type":"CodeActions$ParameterType","introduced":"7.21.0"}, + {"type":"CodeActions$ParameterizedEntityType","introduced":"7.21.0"}, + {"type":"CodeActions$PrimitiveType","introduced":"7.21.0"}, + {"type":"CodeActions$StringTemplateParameterType","introduced":"8.4.0"}, + {"type":"CodeActions$StringType","introduced":"7.21.0"}, + {"type":"CodeActions$Type","introduced":"7.21.0"}, + {"type":"CodeActions$TypeParameter","introduced":"7.21.0"}, + {"type":"CodeActions$VoidType","introduced":"8.3.0"}, + {"type":"ConnectorKit$Connector","introduced":"11.0.0"}, + {"type":"ConnectorKit$ConnectorAction","introduced":"11.0.0"}, + {"type":"ConnectorKit$ConnectorActionCallAction","introduced":"11.0.0"}, + {"type":"ConnectorKit$ConnectorActionParameterMapping","introduced":"11.0.0"}, + {"type":"ConnectorKit$ConnectorProperty","introduced":"11.0.0"}, + {"type":"ConnectorKit$ConnectorPropertyMapping","introduced":"11.0.0"}, + {"type":"ConnectorKit$ConnectorType","introduced":"11.0.0"}, + {"type":"CustomBlobDocuments$CustomBlobDocument","introduced":"11.4.0"}, + {"type":"CustomBlobDocuments$CustomBlobDocumentMetadata","introduced":"11.4.0"}, + {"type":"CustomIcons$CustomIcon","introduced":"9.20.0"}, + {"type":"CustomIcons$CustomIconCollection","introduced":"9.20.0"}, + {"type":"CustomWidgets$CustomWidgetDatabaseSource","introduced":"8.5.0","deleted":"10.5.0"}, + {"type":"CustomWidgets$CustomWidgetXPathSource","introduced":"8.5.0"}, + {"type":"CustomWidgets$WidgetActionVariable","introduced":"10.21.0"}, + {"type":"DataSets$DataSetAccess","deleted":"11.13.0"}, + {"type":"DataSets$DataSetColumn","deleted":"11.15.0"}, + {"type":"DataSets$DataSetConstraintAccess","deleted":"11.13.0"}, + {"type":"DataSets$DataSetDateTimeConstraint","deleted":"11.15.0"}, + {"type":"DataSets$DataSetModuleRoleAccess","deleted":"11.13.0"}, + {"type":"DataSets$DataSetNumericConstraint","deleted":"11.15.0"}, + {"type":"DataSets$DataSetObjectConstraint","deleted":"11.13.0"}, + {"type":"DataSets$DataSetParameterAccess","deleted":"11.13.0"}, + {"type":"DataSets$DataSetParameterConstraint","deleted":"11.15.0"}, + {"type":"DataSets$JavaDataSetSource","deleted":"11.15.0"}, + {"type":"DataTransformers$BooleanValueType","introduced":"11.1.0"}, + {"type":"DataTransformers$DataTransformer","introduced":"11.1.0"}, + {"type":"DataTransformers$DateTimeValueType","introduced":"11.1.0"}, + {"type":"DataTransformers$DecimalValueType","introduced":"11.1.0"}, + {"type":"DataTransformers$JsltAction","introduced":"11.1.0"}, + {"type":"DataTransformers$JsonSource","introduced":"11.1.0"}, + {"type":"DataTransformers$LongValueType","introduced":"11.1.0"}, + {"type":"DataTransformers$Source","introduced":"11.1.0"}, + {"type":"DataTransformers$Step","introduced":"11.1.0"}, + {"type":"DataTransformers$StringValueType","introduced":"11.1.0"}, + {"type":"DataTransformers$StructureAttribute","introduced":"11.1.0"}, + {"type":"DataTransformers$StructureElement","introduced":"11.1.0"}, + {"type":"DataTransformers$StructureObject","introduced":"11.1.0"}, + {"type":"DataTransformers$StructureValue","introduced":"11.1.0"}, + {"type":"DataTransformers$StructureValueType","introduced":"11.1.0"}, + {"type":"DataTransformers$TransformerAction","introduced":"11.1.0"}, + {"type":"DataTypes$BinaryType","introduced":"7.9.0"}, + {"type":"DataTypes$BooleanType","introduced":"7.9.0"}, + {"type":"DataTypes$DataType","introduced":"7.9.0"}, + {"type":"DataTypes$DateTimeType","introduced":"7.9.0"}, + {"type":"DataTypes$DecimalType","introduced":"7.9.0"}, + {"type":"DataTypes$EmptyType","introduced":"7.9.0"}, + {"type":"DataTypes$EntityType","introduced":"7.9.0"}, + {"type":"DataTypes$EnumerationType","introduced":"7.9.0"}, + {"type":"DataTypes$FloatType","introduced":"7.9.0"}, + {"type":"DataTypes$IntegerType","introduced":"7.9.0"}, + {"type":"DataTypes$ListType","introduced":"7.9.0"}, + {"type":"DataTypes$ObjectType","introduced":"7.9.0"}, + {"type":"DataTypes$StringType","introduced":"7.9.0"}, + {"type":"DataTypes$UnknownType","introduced":"7.9.0"}, + {"type":"DataTypes$VoidType","introduced":"7.9.0"}, + {"type":"DatabaseConnector$AdditionalProperty","introduced":"10.12.0"}, + {"type":"DatabaseConnector$AdditionalPropertyValue","introduced":"10.12.0"}, + {"type":"DatabaseConnector$ColumnMapping","introduced":"9.22.0"}, + {"type":"DatabaseConnector$ConnectionDetails","introduced":"9.22.0","deleted":"10.0.0"}, + {"type":"DatabaseConnector$ConnectionInput","introduced":"10.0.0"}, + {"type":"DatabaseConnector$ConnectionParameterMapping","introduced":"11.1.0"}, + {"type":"DatabaseConnector$ConnectionParts","introduced":"10.0.0"}, + {"type":"DatabaseConnector$ConnectionString","introduced":"10.0.0"}, + {"type":"DatabaseConnector$DatabaseConnection","introduced":"9.22.0"}, + {"type":"DatabaseConnector$DatabaseQuery","introduced":"9.22.0"}, + {"type":"DatabaseConnector$ExecuteDatabaseQueryAction","introduced":"9.24.0"}, + {"type":"DatabaseConnector$LimitedLengthSqlDataType","introduced":"9.22.0"}, + {"type":"DatabaseConnector$QueryParameter","introduced":"9.22.0"}, + {"type":"DatabaseConnector$QueryParameterMapping","introduced":"9.24.0"}, + {"type":"DatabaseConnector$SimpleSqlDataType","introduced":"9.22.0"}, + {"type":"DatabaseConnector$SqlDataType","introduced":"9.22.0"}, + {"type":"DatabaseConnector$TableMapping","introduced":"9.22.0"}, + {"type":"DatabaseConnector$ValueAsConstant","introduced":"10.12.0"}, + {"type":"DatabaseConnector$ValueAsString","introduced":"10.12.0"}, + {"type":"DomainModels$AssociationCapabilities","introduced":"8.11.0","deleted":"9.0.1"}, + {"type":"DomainModels$AssociationRef","introduced":"7.11.0"}, + {"type":"DomainModels$AssociationSource","introduced":"8.10.0"}, + {"type":"DomainModels$AttributeCapabilities","introduced":"8.13.0","deleted":"9.0.1"}, + {"type":"DomainModels$AttributeRef","introduced":"7.11.0"}, + {"type":"DomainModels$CachedMappedValue","introduced":"9.0.2","deleted":"10.2.0"}, + {"type":"DomainModels$CurrencyAttributeType","deleted":"9.1.0"}, + {"type":"DomainModels$DirectEntityRef","introduced":"7.11.0"}, + {"type":"DomainModels$EntityCapabilities","introduced":"8.12.0","deleted":"9.0.1"}, + {"type":"DomainModels$EntityKey","introduced":"8.9.0"}, + {"type":"DomainModels$EntityKeyPart","introduced":"8.9.0"}, + {"type":"DomainModels$EntityRef","introduced":"7.11.0"}, + {"type":"DomainModels$EntityRefStep","introduced":"7.11.0"}, + {"type":"DomainModels$EntitySource","introduced":"8.10.0"}, + {"type":"DomainModels$FloatAttributeType","deleted":"9.1.0"}, + {"type":"DomainModels$FloatAttributeTypeBase","deleted":"9.1.0"}, + {"type":"DomainModels$IndirectEntityRef","introduced":"7.11.0"}, + {"type":"DomainModels$MappedValue","introduced":"8.10.0"}, + {"type":"DomainModels$MaterializedRemoteEntitySource","introduced":"8.11.0","deleted":"10.2.0"}, + {"type":"DomainModels$MemberRef","introduced":"7.11.0"}, + {"type":"DomainModels$MultiLanguageAttributeType","introduced":"9.22.0","deleted":"9.24.0"}, + {"type":"DomainModels$OqlViewAssociationSource","introduced":"10.9.0"}, + {"type":"DomainModels$OqlViewEntitySource","introduced":"10.9.0"}, + {"type":"DomainModels$OqlViewValue","introduced":"10.9.0"}, + {"type":"DomainModels$QueryBasedRemoteEntitySource","introduced":"8.10.0"}, + {"type":"DomainModels$RemoteAssociationSource","introduced":"8.10.0"}, + {"type":"DomainModels$RemoteEntitySource","introduced":"8.10.0"}, + {"type":"DomainModels$RemoteEntitySourceDocument","introduced":"7.18.0"}, + {"type":"DomainModels$ViewEntitySource","introduced":"10.9.0"}, + {"type":"DomainModels$ViewEntitySourceDocument","introduced":"10.21.0"}, + {"type":"Enumerations$RemoteEnumerationSource","introduced":"10.2.0"}, + {"type":"Enumerations$RemoteEnumerationValue","introduced":"10.12.0"}, + {"type":"ExcelDataImporter$CSVSheet","introduced":"10.6.0"}, + {"type":"ExcelDataImporter$CSVTemplateContents","introduced":"10.6.0"}, + {"type":"ExcelDataImporter$ColumnAttributeMapping","introduced":"9.24.0"}, + {"type":"ExcelDataImporter$CsvSheetMappingSourceReference","introduced":"10.16.0"}, + {"type":"ExcelDataImporter$DataImporterElement","introduced":"10.15.0"}, + {"type":"ExcelDataImporter$ExcelSheet","introduced":"10.6.0"}, + {"type":"ExcelDataImporter$ExcelSheetMappingSourceReference","introduced":"10.16.0"}, + {"type":"ExcelDataImporter$ExcelTemplateContents","introduced":"10.6.0"}, + {"type":"ExcelDataImporter$ImportExcelDataAction","introduced":"10.0.0"}, + {"type":"ExcelDataImporter$IndexReference","introduced":"9.24.0"}, + {"type":"ExcelDataImporter$NameReference","introduced":"9.24.0"}, + {"type":"ExcelDataImporter$Reference","introduced":"9.24.0"}, + {"type":"ExcelDataImporter$Sheet","introduced":"9.24.0","deleted":"10.6.0"}, + {"type":"ExcelDataImporter$Template","introduced":"9.24.0"}, + {"type":"ExcelDataImporter$TemplateContents","introduced":"10.6.0"}, + {"type":"Expressions$BinaryExpression","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$BooleanLiteral","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$ConstantRefExpression","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$EmptyLiteral","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$EnumerationValueRefExpression","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$Expression","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$FloatLiteral","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$FunctionCallExpression","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$GlobalVariableRef","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$IfExpression","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$IntegerLiteral","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$LiteralExpression","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$NoExpression","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$NoVariableRef","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$ParenthesisExpression","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$StringLiteral","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$UnaryExpression","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$UnparsableExpression","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$VariableRef","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Expressions$VariableRefExpression","introduced":"7.9.0","deleted":"9.8.0"}, + {"type":"Forms$AbstractDesignPropertyValue","introduced":"10.2.0","sdk":["Pages$AbstractDesignPropertyValue"]}, + {"type":"Forms$AccessibilitySettings","introduced":"9.22.0","sdk":["Pages$AccessibilitySettings"]}, + {"type":"Forms$Appearance","introduced":"8.0.0","sdk":["Pages$Appearance"]}, + {"type":"Forms$BackButton","deleted":"7.0.2","sdk":["Pages$BackButton"]}, + {"type":"Forms$BuildingBlock","introduced":"7.7.0","sdk":["Pages$BuildingBlock"]}, + {"type":"Forms$CallNanoflowClientAction","introduced":"7.8.0","sdk":["Pages$CallNanoflowClientAction"]}, + {"type":"Forms$CallWorkflowClientAction","introduced":"9.0.2","sdk":["Pages$CallWorkflowClientAction"]}, + {"type":"Forms$CancelButton","deleted":"7.0.2","sdk":["Pages$CancelButton"]}, + {"type":"Forms$CancelChangesClientAction","introduced":"7.0.2","sdk":["Pages$CancelChangesClientAction"]}, + {"type":"Forms$CancelSynchronizationClientAction","introduced":"11.8.0","sdk":["Pages$CancelSynchronizationClientAction"]}, + {"type":"Forms$ClosePageClientAction","introduced":"7.0.2","sdk":["Pages$ClosePageClientAction"]}, + {"type":"Forms$CompoundDesignPropertyValue","introduced":"10.2.0","sdk":["Pages$CompoundDesignPropertyValue"]}, + {"type":"Forms$CreateObjectClientAction","introduced":"7.17.0","sdk":["Pages$CreateObjectClientAction"]}, + {"type":"Forms$CustomDesignPropertyValue","introduced":"10.2.0","sdk":["Pages$CustomDesignPropertyValue"]}, + {"type":"Forms$DataViewActionButton","deleted":"6.7.0","sdk":["Pages$DataViewActionButton"]}, + {"type":"Forms$DataViewCancelButton","deleted":"6.7.0","sdk":["Pages$DataViewCancelButton"]}, + {"type":"Forms$DataViewCloseButton","deleted":"6.7.0","sdk":["Pages$DataViewCloseButton"]}, + {"type":"Forms$DataViewControlBar","deleted":"6.7.0","sdk":["Pages$DataViewControlBar"]}, + {"type":"Forms$DataViewControlBarButton","deleted":"6.7.0","sdk":["Pages$DataViewControlBarButton"]}, + {"type":"Forms$DataViewSaveButton","deleted":"6.7.0","sdk":["Pages$DataViewSaveButton"]}, + {"type":"Forms$DatabaseConstraint","deleted":"10.5.0","sdk":["Pages$DatabaseConstraint"]}, + {"type":"Forms$DatabaseSourceBase","deleted":"10.5.0","sdk":["Pages$DatabaseSourceBase"]}, + {"type":"Forms$DeleteClientAction","introduced":"7.17.0","sdk":["Pages$DeleteClientAction"]}, + {"type":"Forms$DesignPropertyValue","introduced":"8.0.0","sdk":["Pages$DesignPropertyValue"]}, + {"type":"Forms$EditPageTemplateType","introduced":"8.13.0","sdk":["Pages$EditPageTemplateType"]}, + {"type":"Forms$GridDatabaseSource","deleted":"10.5.0","sdk":["Pages$GridDatabaseSource"]}, + {"type":"Forms$GridDeleteButton","deleted":"7.17.0","sdk":["Pages$GridDeleteButton"]}, + {"type":"Forms$GridEditButton","deleted":"7.17.0","sdk":["Pages$GridEditButton"]}, + {"type":"Forms$HorizontalSplitPane","deleted":"8.0.0","sdk":["Pages$HorizontalSplitPane"]}, + {"type":"Forms$IconCollectionIcon","introduced":"9.20.0","sdk":["Pages$IconCollectionIcon"]}, + {"type":"Forms$LayoutContent","introduced":"8.0.0","sdk":["Pages$LayoutContent"]}, + {"type":"Forms$LinkButton","deleted":"7.3.0","sdk":["Pages$LinkButton"]}, + {"type":"Forms$ListViewDatabaseSource","deleted":"10.5.0","sdk":["Pages$ListViewDatabaseSource"]}, + {"type":"Forms$LocalVariable","introduced":"10.17.0","sdk":["Pages$LocalVariable"]}, + {"type":"Forms$LoginButton","introduced":"7.0.2","sdk":["Pages$LoginButton"]}, + {"type":"Forms$LoginIdTextBox","introduced":"7.0.2","sdk":["Pages$LoginIdTextBox"]}, + {"type":"Forms$LoginTextBox","introduced":"7.0.2","sdk":["Pages$LoginTextBox"]}, + {"type":"Forms$LogoutButton","deleted":"7.1.0","sdk":["Pages$LogoutButton"]}, + {"type":"Forms$MasterDetail","introduced":"7.1.0","deleted":"7.15.0","sdk":["Pages$MasterDetail"]}, + {"type":"Forms$MasterDetailDetailRegion","introduced":"7.1.0","deleted":"7.15.0","sdk":["Pages$MasterDetailDetailRegion"]}, + {"type":"Forms$MasterDetailMasterRegion","introduced":"7.1.0","deleted":"7.15.0","sdk":["Pages$MasterDetailMasterRegion"]}, + {"type":"Forms$MasterDetailRegion","introduced":"7.1.0","deleted":"7.15.0","sdk":["Pages$MasterDetailRegion"]}, + {"type":"Forms$MicroflowParameterMapping","introduced":"7.19.0","sdk":["Pages$MicroflowParameterMapping"]}, + {"type":"Forms$NamedValue","introduced":"9.24.0","deleted":"10.1.0","sdk":["Pages$NamedValue"]}, + {"type":"Forms$NanoflowParameterMapping","introduced":"7.19.0","sdk":["Pages$NanoflowParameterMapping"]}, + {"type":"Forms$NanoflowSource","introduced":"7.12.0","sdk":["Pages$NanoflowSource"]}, + {"type":"Forms$NativeLayoutContent","introduced":"8.0.0","sdk":["Pages$NativeLayoutContent"]}, + {"type":"Forms$NewButton","deleted":"7.17.0","sdk":["Pages$NewButton"]}, + {"type":"Forms$OfflineSchema","deleted":"7.22.0","sdk":["Pages$OfflineSchema"]}, + {"type":"Forms$OfflineSchemaFetchInstruction","deleted":"7.22.0","sdk":["Pages$OfflineSchemaFetchInstruction"]}, + {"type":"Forms$OnClickBehavior","deleted":"7.18.0","sdk":["Pages$OnClickBehavior"]}, + {"type":"Forms$OnClickEnlarge","deleted":"7.18.0","sdk":["Pages$OnClickEnlarge"]}, + {"type":"Forms$OnClickMicroflow","deleted":"7.18.0","sdk":["Pages$OnClickMicroflow"]}, + {"type":"Forms$OnClickNothing","deleted":"7.18.0","sdk":["Pages$OnClickNothing"]}, + {"type":"Forms$OpenLinkClientAction","introduced":"7.3.0","sdk":["Pages$OpenLinkClientAction"]}, + {"type":"Forms$OpenUserTaskClientAction","introduced":"9.0.2","sdk":["Pages$OpenUserTaskClientAction"]}, + {"type":"Forms$OpenWorkflowClientAction","introduced":"9.0.2","sdk":["Pages$OpenWorkflowClientAction"]}, + {"type":"Forms$OptionDesignPropertyValue","introduced":"10.2.0","sdk":["Pages$OptionDesignPropertyValue"]}, + {"type":"Forms$OutputMapping","introduced":"11.5.0","sdk":["Pages$OutputMapping"]}, + {"type":"Forms$OverviewPageTemplateType","introduced":"11.4.0","sdk":["Pages$OverviewPageTemplateType"]}, + {"type":"Forms$PageParameter","introduced":"9.4.0","sdk":["Pages$PageParameter"]}, + {"type":"Forms$PageParameterMapping","introduced":"9.7.0","sdk":["Pages$PageParameterMapping"]}, + {"type":"Forms$PagePrimitiveParameterUrlSegment","introduced":"11.9.0","sdk":["Pages$PagePrimitiveParameterUrlSegment"]}, + {"type":"Forms$PageTemplate","introduced":"7.7.0","sdk":["Pages$PageTemplate"]}, + {"type":"Forms$PageVariable","introduced":"8.4.0","sdk":["Pages$PageVariable"]}, + {"type":"Forms$ParameterAttributeUrlSegment","introduced":"10.0.0","sdk":["Pages$ParameterAttributeUrlSegment"]}, + {"type":"Forms$ParameterIdUrlSegment","introduced":"10.0.0","sdk":["Pages$ParameterIdUrlSegment"]}, + {"type":"Forms$PasswordTextBox","introduced":"7.0.2","sdk":["Pages$PasswordTextBox"]}, + {"type":"Forms$RegularPageTemplateType","introduced":"8.13.0","sdk":["Pages$RegularPageTemplateType"]}, + {"type":"Forms$RetrievalQuery","introduced":"7.12.0","deleted":"10.1.0","sdk":["Pages$RetrievalQuery"]}, + {"type":"Forms$RetrievalQueryParameter","introduced":"8.6.0","deleted":"10.1.0","sdk":["Pages$RetrievalQueryParameter"]}, + {"type":"Forms$RetrievalSchema","introduced":"6.2.0","deleted":"8.4.0","sdk":["Pages$RetrievalSchema"]}, + {"type":"Forms$RuntimeOperation","introduced":"9.17.0","deleted":"10.1.0","sdk":["Pages$RuntimeOperation"]}, + {"type":"Forms$SaveButton","deleted":"7.0.2","sdk":["Pages$SaveButton"]}, + {"type":"Forms$SaveChangesClientAction","introduced":"7.0.2","sdk":["Pages$SaveChangesClientAction"]}, + {"type":"Forms$SelectPageTemplateType","introduced":"8.13.0","sdk":["Pages$SelectPageTemplateType"]}, + {"type":"Forms$SelectorDatabaseSource","introduced":"6.1.0","deleted":"10.5.0","sdk":["Pages$SelectorDatabaseSource"]}, + {"type":"Forms$SetTaskOutcomeClientAction","introduced":"9.0.2","sdk":["Pages$SetTaskOutcomeClientAction"]}, + {"type":"Forms$SignOutClientAction","introduced":"7.1.0","sdk":["Pages$SignOutClientAction"]}, + {"type":"Forms$SnippetParameter","introduced":"9.21.0","sdk":["Pages$SnippetParameter"]}, + {"type":"Forms$SnippetParameterMapping","introduced":"9.21.0","sdk":["Pages$SnippetParameterMapping"]}, + {"type":"Forms$SplitPane","deleted":"8.0.0","sdk":["Pages$SplitPane"]}, + {"type":"Forms$StaticUrlSegment","introduced":"10.0.0","deleted":"10.1.0","sdk":["Pages$StaticUrlSegment"]}, + {"type":"Forms$SyncButton","deleted":"7.0.2","sdk":["Pages$SyncButton"]}, + {"type":"Forms$SyncClientAction","introduced":"7.0.2","sdk":["Pages$SyncClientAction"]}, + {"type":"Forms$TemplateFormBase","introduced":"7.7.0","sdk":["Pages$TemplateFormBase"]}, + {"type":"Forms$TemplatePlaceholder","introduced":"8.13.0","sdk":["Pages$TemplatePlaceholder"]}, + {"type":"Forms$TemplateType","introduced":"8.13.0","sdk":["Pages$TemplateType"]}, + {"type":"Forms$ToggleDesignPropertyValue","introduced":"10.2.0","sdk":["Pages$ToggleDesignPropertyValue"]}, + {"type":"Forms$UrlSegment","introduced":"10.0.0","deleted":"10.1.0","sdk":["Pages$UrlSegment"]}, + {"type":"Forms$UserRoleSet","introduced":"9.21.0","deleted":"10.1.0","sdk":["Pages$UserRoleSet"]}, + {"type":"Forms$UserTaskTemplateType","introduced":"9.0.2","sdk":["Pages$UserTaskTemplateType"]}, + {"type":"Forms$ValidationMessage","introduced":"7.0.2","sdk":["Pages$ValidationMessage"]}, + {"type":"Forms$VerticalFlow","deleted":"7.15.0","sdk":["Pages$VerticalFlow"]}, + {"type":"Forms$VerticalSplitPane","deleted":"8.0.0","sdk":["Pages$VerticalSplitPane"]}, + {"type":"Forms$WebLayoutContent","introduced":"8.0.0","sdk":["Pages$WebLayoutContent"]}, + {"type":"Forms$WidgetValidation","introduced":"7.6.0","sdk":["Pages$WidgetValidation"]}, + {"type":"Forms$WorkflowOverviewTemplateType","introduced":"9.0.2","sdk":["Pages$WorkflowOverviewTemplateType"]}, + {"type":"Forms$WorkflowTemplateType","introduced":"9.0.2","sdk":["Pages$WorkflowTemplateType"]}, + {"type":"IntegrationOverview$CatalogApi","introduced":"10.21.0"}, + {"type":"IntegrationOverview$SourceApi","introduced":"10.21.0"}, + {"type":"JavaActions$BasicParameterType","introduced":"6.7.0","deleted":"7.21.0"}, + {"type":"JavaActions$BooleanType","introduced":"6.6.0","deleted":"7.21.0"}, + {"type":"JavaActions$ConcreteEntityType","introduced":"6.6.0","deleted":"7.21.0"}, + {"type":"JavaActions$DateTimeType","introduced":"6.6.0","deleted":"7.21.0"}, + {"type":"JavaActions$DecimalType","introduced":"6.6.0","deleted":"7.21.0"}, + {"type":"JavaActions$EntityType","introduced":"6.6.0","deleted":"7.21.0"}, + {"type":"JavaActions$EntityTypeParameterType","introduced":"6.7.0","deleted":"7.21.0"}, + {"type":"JavaActions$EnumerationType","introduced":"6.6.0","deleted":"7.21.0"}, + {"type":"JavaActions$ExportMappingJavaActionParameterType","introduced":"7.21.0"}, + {"type":"JavaActions$ExportMappingParameterType","introduced":"7.2.0","deleted":"7.21.0"}, + {"type":"JavaActions$FloatType","introduced":"6.6.0","deleted":"7.21.0"}, + {"type":"JavaActions$ImportMappingJavaActionParameterType","introduced":"7.21.0"}, + {"type":"JavaActions$ImportMappingParameterType","introduced":"7.2.0","deleted":"7.21.0"}, + {"type":"JavaActions$IntegerType","introduced":"6.6.0","deleted":"7.21.0"}, + {"type":"JavaActions$ListType","introduced":"6.6.0","deleted":"7.21.0"}, + {"type":"JavaActions$MicroflowActionInfo","introduced":"6.6.0","deleted":"7.21.0"}, + {"type":"JavaActions$MicroflowJavaActionParameterType","introduced":"7.21.0"}, + {"type":"JavaActions$MicroflowParameterType","introduced":"6.9.0","deleted":"7.21.0"}, + {"type":"JavaActions$ParameterType","introduced":"6.7.0","deleted":"7.21.0"}, + {"type":"JavaActions$ParameterizedEntityType","introduced":"6.6.0","deleted":"7.21.0"}, + {"type":"JavaActions$PrimitiveType","introduced":"6.6.0","deleted":"7.21.0"}, + {"type":"JavaActions$StringType","introduced":"6.6.0","deleted":"7.21.0"}, + {"type":"JavaActions$Type","introduced":"6.6.0","deleted":"7.21.0"}, + {"type":"JavaActions$TypeParameter","introduced":"6.6.0","deleted":"7.21.0"}, + {"type":"JavaScriptActions$JavaScriptAction","introduced":"7.21.0"}, + {"type":"JavaScriptActions$JavaScriptActionParameter","introduced":"7.21.0"}, + {"type":"JavaScriptActions$MicroflowJavaScriptActionParameterType","introduced":"10.21.0"}, + {"type":"JavaScriptActions$NanoflowJavaScriptActionParameterType","introduced":"8.4.0"}, + {"type":"JsonStructures$JsonElement","introduced":"6.6.0"}, + {"type":"JsonStructures$JsonStructure","introduced":"6.2.0"}, + {"type":"Kafka$ConsumedKafkaService","introduced":"8.11.0","deleted":"10.2.0"}, + {"type":"Kafka$KafkaMappedValue","introduced":"8.11.0","deleted":"10.2.0"}, + {"type":"Kafka$KafkaRemoteEntitySource","introduced":"8.11.0","deleted":"10.2.0"}, + {"type":"Kafka$PublishedKafkaResource","introduced":"8.14.0","deleted":"10.2.0"}, + {"type":"Kafka$PublishedKafkaResourceAttribute","introduced":"9.0.1","deleted":"10.2.0"}, + {"type":"Kafka$PublishedKafkaService","introduced":"8.14.0","deleted":"10.2.0"}, + {"type":"MLMappings$MLMappingDocument","introduced":"9.17.0"}, + {"type":"MLMappings$MLModelEntityMappings","introduced":"9.17.0"}, + {"type":"MLMappings$MLModelMappings","introduced":"9.17.0"}, + {"type":"MLMappings$MLModelMetadata","introduced":"9.22.0"}, + {"type":"MLMappings$TensorDimension","introduced":"9.17.0"}, + {"type":"MLMappings$TensorMappingElement","introduced":"9.17.0"}, + {"type":"Mappings$Element","introduced":"6.6.0"}, + {"type":"Mappings$MappingSource","introduced":"10.6.0"}, + {"type":"Mappings$MappingSourceDocument","introduced":"9.24.0"}, + {"type":"Mappings$MappingSourceReference","introduced":"10.16.0"}, + {"type":"MessageDefinitions$AssociationElement","introduced":"7.10.0"}, + {"type":"MessageDefinitions$AttributeElement","introduced":"7.6.0"}, + {"type":"MessageDefinitions$EntityElement","introduced":"7.6.0"}, + {"type":"MessageDefinitions$EntityMessageDefinition","introduced":"7.6.0","deleted":"11.15.0"}, + {"type":"MessageDefinitions$ExposedAssociation","introduced":"7.6.0"}, + {"type":"MessageDefinitions$ExposedAttribute","introduced":"7.6.0"}, + {"type":"MessageDefinitions$ExposedEntity","introduced":"7.6.0"}, + {"type":"MessageDefinitions$ExposedEntityBase","introduced":"7.6.0"}, + {"type":"MessageDefinitions$ExposedMember","introduced":"7.6.0"}, + {"type":"MessageDefinitions$MessageDefinition","introduced":"7.6.0","deleted":"11.15.0"}, + {"type":"MessageDefinitions$MessageDefinition2","introduced":"11.10.0"}, + {"type":"MessageDefinitions$MessageDefinitionCollection","introduced":"7.6.0","deleted":"11.15.0"}, + {"type":"Microflows$AbortOperation","introduced":"9.2.0"}, + {"type":"Microflows$AdditionalAttribute","introduced":"11.5.0"}, + {"type":"Microflows$AppServiceCallAction","deleted":"9.0.2"}, + {"type":"Microflows$AppServiceCallParameterMapping","deleted":"9.0.2"}, + {"type":"Microflows$ApplyJumpToOptionAction","introduced":"9.15.0"}, + {"type":"Microflows$AuthenticationConfig","introduced":"11.2.0"}, + {"type":"Microflows$AuthenticationDocumentConfig","introduced":"11.5.0"}, + {"type":"Microflows$BasicAuthConfig","introduced":"11.2.0"}, + {"type":"Microflows$BasicCodeActionParameterValue","introduced":"7.21.0"}, + {"type":"Microflows$BasicJavaActionParameterValue","introduced":"6.7.0","deleted":"7.21.0"}, + {"type":"Microflows$BezierCurve","introduced":"10.16.0"}, + {"type":"Microflows$BinaryRequestHandling","introduced":"6.9.0"}, + {"type":"Microflows$BodyVariable","introduced":"10.5.0"}, + {"type":"Microflows$CallExternalAction","introduced":"9.19.0"}, + {"type":"Microflows$CancelSynchronizationAction","introduced":"11.8.0"}, + {"type":"Microflows$ClearFromClientAction","introduced":"10.0.0"}, + {"type":"Microflows$CodeActionParameterValue","introduced":"7.21.0"}, + {"type":"Microflows$ContinueOperation","introduced":"9.3.0"}, + {"type":"Microflows$CounterMeterAction","introduced":"9.5.0"}, + {"type":"Microflows$CustomBlobDocumentCodeActionParameterValue","introduced":"11.6.0"}, + {"type":"Microflows$DeleteExternalObject","introduced":"9.12.0"}, + {"type":"Microflows$EmailConnectionConfig","introduced":"11.2.0"}, + {"type":"Microflows$EmailCustomHeader","introduced":"11.11.0"}, + {"type":"Microflows$EmailMessage","introduced":"11.2.0"}, + {"type":"Microflows$EntityTypeCodeActionParameterValue","introduced":"7.21.0"}, + {"type":"Microflows$EntityTypeJavaActionParameterValue","introduced":"6.8.0","deleted":"7.21.0"}, + {"type":"Microflows$ExportMappingJavaActionParameterValue","introduced":"7.2.0","deleted":"7.21.0"}, + {"type":"Microflows$ExportMappingParameterValue","introduced":"7.21.0"}, + {"type":"Microflows$ExpressionBasedCodeActionParameterValue","introduced":"7.21.0"}, + {"type":"Microflows$ExpressionListOperation","introduced":"9.0.3"}, + {"type":"Microflows$ExternalActionParameterMapping","introduced":"10.2.0"}, + {"type":"Microflows$FilterByExpression","introduced":"9.0.3"}, + {"type":"Microflows$FindByExpression","introduced":"9.0.3"}, + {"type":"Microflows$FormDataPart","introduced":"8.0.0"}, + {"type":"Microflows$FormDataRequestHandling","introduced":"8.0.0"}, + {"type":"Microflows$GaugeMeterAction","introduced":"9.5.0"}, + {"type":"Microflows$GenerateJumpToOptionsAction","introduced":"9.14.0"}, + {"type":"Microflows$GetWorkflowActivityRecordsAction","introduced":"10.4.0"}, + {"type":"Microflows$GetWorkflowDataAction","introduced":"9.10.0"}, + {"type":"Microflows$GetWorkflowsAction","introduced":"10.4.0"}, + {"type":"Microflows$ImportMappingJavaActionParameterValue","introduced":"7.2.0","deleted":"7.21.0"}, + {"type":"Microflows$ImportMappingParameterValue","introduced":"7.21.0"}, + {"type":"Microflows$IncludedAssociation","introduced":"10.11.0"}, + {"type":"Microflows$IncrementCounterMeterAction","introduced":"9.5.0"}, + {"type":"Microflows$IterableList","introduced":"9.0.4"}, + {"type":"Microflows$JavaActionParameterValue","introduced":"6.7.0","deleted":"7.21.0"}, + {"type":"Microflows$JavaScriptActionCallAction","introduced":"7.21.0"}, + {"type":"Microflows$JavaScriptActionParameterMapping","introduced":"7.21.0"}, + {"type":"Microflows$Line","introduced":"10.16.0"}, + {"type":"Microflows$ListRange","introduced":"9.24.0"}, + {"type":"Microflows$LockWorkflowAction","introduced":"9.21.0"}, + {"type":"Microflows$LoopSource","introduced":"9.0.4"}, + {"type":"Microflows$MLModelCall","introduced":"9.10.0","deleted":"9.18.0"}, + {"type":"Microflows$MLModelCallAction","introduced":"9.10.0"}, + {"type":"Microflows$MLModelCallParameterMapping","introduced":"9.10.0","deleted":"9.18.0"}, + {"type":"Microflows$MeterAction","introduced":"9.5.0"}, + {"type":"Microflows$MeterTagMapping","introduced":"9.5.0"}, + {"type":"Microflows$MicroflowJavaActionParameterValue","introduced":"6.9.0","deleted":"7.21.0"}, + {"type":"Microflows$MicroflowParameterAttributeUrlSegment","introduced":"10.1.0"}, + {"type":"Microflows$MicroflowParameterValue","introduced":"7.21.0"}, + {"type":"Microflows$MicroflowPrimitiveParameterUrlSegment","introduced":"10.3.0"}, + {"type":"Microflows$Nanoflow","introduced":"7.10.0"}, + {"type":"Microflows$NanoflowCall","introduced":"8.0.0"}, + {"type":"Microflows$NanoflowCallAction","introduced":"8.0.0"}, + {"type":"Microflows$NanoflowCallParameterMapping","introduced":"8.0.0"}, + {"type":"Microflows$NanoflowParameter","introduced":"7.19.0"}, + {"type":"Microflows$NotifyWorkflowAction","introduced":"10.0.0"}, + {"type":"Microflows$OpenUserTaskAction","introduced":"9.0.2"}, + {"type":"Microflows$OpenWorkflowAction","introduced":"9.0.2"}, + {"type":"Microflows$OrthogonalPath","introduced":"10.16.0"}, + {"type":"Microflows$OutputVariable","introduced":"10.4.0"}, + {"type":"Microflows$ParameterIdUrlSegment","introduced":"10.1.0"}, + {"type":"Microflows$PauseOperation","introduced":"9.3.0"}, + {"type":"Microflows$PrimitiveTypedTemplateArgument","introduced":"8.6.0"}, + {"type":"Microflows$ProxyConfiguration","introduced":"7.15.0"}, + {"type":"Microflows$PushToClientAction","introduced":"8.4.0"}, + {"type":"Microflows$QueryParameterMapping","introduced":"11.0.0"}, + {"type":"Microflows$RestCallAction","introduced":"6.6.0"}, + {"type":"Microflows$RestOperationCallAction","introduced":"10.1.0"}, + {"type":"Microflows$RestOperationParameterMapping","introduced":"10.2.0"}, + {"type":"Microflows$RestParameterMapping","introduced":"10.17.0"}, + {"type":"Microflows$RestartOperation","introduced":"9.2.0"}, + {"type":"Microflows$ResumeOperation","introduced":"9.3.0","deleted":"9.12.0"}, + {"type":"Microflows$RetryOperation","introduced":"9.12.0"}, + {"type":"Microflows$SendEmailAction","introduced":"11.2.0"}, + {"type":"Microflows$SendExternalObject","introduced":"9.6.0"}, + {"type":"Microflows$SetTaskOutcomeAction","introduced":"9.0.2"}, + {"type":"Microflows$StringTemplateParameterValue","introduced":"8.4.0"}, + {"type":"Microflows$SynchronizeAction","introduced":"7.22.0"}, + {"type":"Microflows$TransformJsonAction","introduced":"11.4.0"}, + {"type":"Microflows$TypedTemplate","introduced":"8.6.0"}, + {"type":"Microflows$TypedTemplateArgument","introduced":"8.6.0"}, + {"type":"Microflows$UnlockWorkflowAction","introduced":"9.21.0"}, + {"type":"Microflows$UnpauseOperation","introduced":"9.12.0"}, + {"type":"Microflows$WebServiceOperationParameterMapping","introduced":"6.3.0"}, + {"type":"Microflows$WhileLoopCondition","introduced":"9.0.4"}, + {"type":"Microflows$WorkflowCallAction","introduced":"9.0.2"}, + {"type":"Microflows$WorkflowOperation","introduced":"9.2.0"}, + {"type":"Microflows$WorkflowOperationAction","introduced":"9.2.0"}, + {"type":"Nanoflows$NanoflowParameterValue","introduced":"8.4.0"}, + {"type":"NativePages$BottomBarItem","introduced":"8.0.0"}, + {"type":"NativePages$NativeLayout","introduced":"7.21.0","deleted":"8.0.0"}, + {"type":"NativePages$NativeLayoutCallArgument","introduced":"7.23.0","deleted":"8.0.0"}, + {"type":"NativePages$NativePage","introduced":"7.21.0","deleted":"8.0.0"}, + {"type":"NativePages$NativePageClientAction","introduced":"7.23.0","deleted":"8.0.0"}, + {"type":"NativePages$NativePlaceholder","introduced":"7.23.0","deleted":"8.0.0"}, + {"type":"Navigation$EmbeddedHomePage","introduced":"11.12.0"}, + {"type":"Navigation$EmbeddedNavigationProfile","introduced":"11.12.0"}, + {"type":"Navigation$NativeHomePage","introduced":"9.4.0"}, + {"type":"Navigation$NativeHomePageBase","introduced":"9.4.0"}, + {"type":"Navigation$NativeNavigationProfile","introduced":"7.22.0"}, + {"type":"Navigation$NotFoundHomePage","introduced":"10.13.0"}, + {"type":"Navigation$OfflineEntityConfig","introduced":"7.22.0"}, + {"type":"Navigation$ProgressiveWebAppSettings","introduced":"9.0.3"}, + {"type":"Navigation$RoleBasedNativeHomePage","introduced":"8.0.0"}, + {"type":"ODataPublish$CallMicroflowToChange","introduced":"10.4.0"}, + {"type":"ODataPublish$CallMicroflowToRead","introduced":"10.4.0"}, + {"type":"ODataPublish$ChangeMode","introduced":"10.4.0"}, + {"type":"ODataPublish$ChangeNotSupported","introduced":"10.4.0"}, + {"type":"ODataPublish$ChangeSource","introduced":"10.4.0"}, + {"type":"ODataPublish$EntitySet","introduced":"10.4.0"}, + {"type":"ODataPublish$EntityType","introduced":"10.4.0"}, + {"type":"ODataPublish$PublishedAssociationEnd","introduced":"10.4.0"}, + {"type":"ODataPublish$PublishedAttribute","introduced":"10.4.0"}, + {"type":"ODataPublish$PublishedContract","introduced":"10.4.0"}, + {"type":"ODataPublish$PublishedEnumeration","introduced":"10.4.0"}, + {"type":"ODataPublish$PublishedEnumerationValue","introduced":"10.4.0"}, + {"type":"ODataPublish$PublishedId","introduced":"10.4.0"}, + {"type":"ODataPublish$PublishedMember","introduced":"10.4.0"}, + {"type":"ODataPublish$PublishedMicroflow","introduced":"10.4.0"}, + {"type":"ODataPublish$PublishedMicroflowParameter","introduced":"10.4.0"}, + {"type":"ODataPublish$PublishedODataService2","introduced":"10.4.0"}, + {"type":"ODataPublish$QueryOptions","introduced":"10.4.0"}, + {"type":"ODataPublish$ReadMode","introduced":"10.4.0"}, + {"type":"ODataPublish$ReadSource","introduced":"10.4.0"}, + {"type":"ODataPublish$ServiceFeed","introduced":"10.4.0"}, + {"type":"Projects$Dependency","introduced":"11.12.0"}, + {"type":"Projects$GuidMappingEntry","introduced":"11.12.0"}, + {"type":"Projects$JarDependency","introduced":"10.0.0"}, + {"type":"Projects$JarDependencyExclusion","introduced":"10.12.0"}, + {"type":"Projects$ModuleDependencies","introduced":"11.12.0"}, + {"type":"Projects$ModuleGuidMapping","introduced":"11.12.0"}, + {"type":"Projects$ModuleSettings","introduced":"9.8.0"}, + {"type":"Projects$SetupFolder","introduced":"11.16.0"}, + {"type":"Projects$SetupItem","introduced":"11.16.0"}, + {"type":"Projects$SetupSettings","introduced":"11.16.0"}, + {"type":"Projects$SetupShortcut","introduced":"11.16.0"}, + {"type":"Queues$BasicQueueConfig","introduced":"8.16.0"}, + {"type":"Queues$Queue","introduced":"8.16.0"}, + {"type":"Queues$QueueConfig","introduced":"8.16.0"}, + {"type":"Queues$QueueExponentialRetry","introduced":"9.10.0"}, + {"type":"Queues$QueueFixedRetry","introduced":"9.10.0"}, + {"type":"Queues$QueueRetry","introduced":"9.10.0"}, + {"type":"Queues$QueueSettings","introduced":"9.10.0"}, + {"type":"Reports$BasicReport","deleted":"11.13.0"}, + {"type":"Reports$BasicReportAggregate","deleted":"11.13.0"}, + {"type":"Reports$BasicReportColumn","deleted":"11.13.0"}, + {"type":"Reports$ReportButton","deleted":"11.13.0"}, + {"type":"Reports$ReportChart","deleted":"9.0.1"}, + {"type":"Reports$ReportChartSeries","deleted":"9.0.1"}, + {"type":"Reports$ReportDateRangeField","deleted":"11.13.0"}, + {"type":"Reports$ReportDateRangeSelector","deleted":"11.13.0"}, + {"type":"Reports$ReportDropDown","deleted":"11.13.0"}, + {"type":"Reports$ReportPane","deleted":"6.10.0"}, + {"type":"Reports$ReportParameter","deleted":"11.13.0"}, + {"type":"Reports$ReportWidget","deleted":"11.13.0"}, + {"type":"Reports$ReportZoomInfo","deleted":"11.13.0"}, + {"type":"Reports$ReportZoomMapping","deleted":"11.13.0"}, + {"type":"Rest$AuthenticationScheme","introduced":"10.2.0"}, + {"type":"Rest$BasicAuthenticationScheme","introduced":"10.2.0"}, + {"type":"Rest$Body","introduced":"10.4.0"}, + {"type":"Rest$CallMicroflowToChange","introduced":"9.11.0","deleted":"10.4.0"}, + {"type":"Rest$CallMicroflowToRead","introduced":"9.14.0","deleted":"10.4.0"}, + {"type":"Rest$ChangeMode","introduced":"9.11.0","deleted":"10.4.0"}, + {"type":"Rest$ChangeNotSupported","introduced":"9.11.0","deleted":"10.4.0"}, + {"type":"Rest$ChangeSource","introduced":"9.11.0","deleted":"10.4.0"}, + {"type":"Rest$ConstantValue","introduced":"10.2.0"}, + {"type":"Rest$ConsumedODataService","introduced":"7.18.0"}, + {"type":"Rest$ConsumedRestService","introduced":"10.1.0"}, + {"type":"Rest$CorsConfiguration","introduced":"7.18.0"}, + {"type":"Rest$HeaderWithValueTemplate","introduced":"10.1.0"}, + {"type":"Rest$ImplicitMappingBody","introduced":"10.4.0"}, + {"type":"Rest$ImplicitMappingResponseHandling","introduced":"10.3.0"}, + {"type":"Rest$JsonBody","introduced":"11.0.0"}, + {"type":"Rest$MetadataReference","introduced":"8.6.0"}, + {"type":"Rest$NoResponseHandling","introduced":"10.3.0"}, + {"type":"Rest$ODataEntityTypeSource","introduced":"10.8.0"}, + {"type":"Rest$ODataKey","introduced":"8.9.0"}, + {"type":"Rest$ODataKeyPart","introduced":"8.9.0"}, + {"type":"Rest$ODataMappedPrimitiveCollectionValue","introduced":"11.6.0"}, + {"type":"Rest$ODataMappedValue","introduced":"8.10.0"}, + {"type":"Rest$ODataPrimitiveCollectionAssociationSource","introduced":"11.6.0"}, + {"type":"Rest$ODataPrimitiveCollectionEntitySource","introduced":"11.6.0"}, + {"type":"Rest$ODataRemoteAssociationSource","introduced":"8.10.0"}, + {"type":"Rest$ODataRemoteEntitySource","introduced":"8.10.0"}, + {"type":"Rest$ODataRemoteEnumerationSource","introduced":"10.2.0"}, + {"type":"Rest$ODataRemoteEnumerationValue","introduced":"10.12.0"}, + {"type":"Rest$OpenApiFile","introduced":"10.21.0"}, + {"type":"Rest$OperationParameter","introduced":"10.1.0"}, + {"type":"Rest$OptionalQueryParameterUsage","introduced":"11.0.0"}, + {"type":"Rest$PublishedODataContract","introduced":"9.14.0","deleted":"10.4.0"}, + {"type":"Rest$PublishedODataEnumeration","introduced":"9.21.0","deleted":"10.4.0"}, + {"type":"Rest$PublishedODataEnumerationValue","introduced":"9.21.0","deleted":"10.4.0"}, + {"type":"Rest$PublishedODataMicroflow","introduced":"9.19.0","deleted":"10.4.0"}, + {"type":"Rest$PublishedODataMicroflowParameter","introduced":"10.0.0","deleted":"10.4.0"}, + {"type":"Rest$PublishedODataService","deleted":"10.4.0"}, + {"type":"Rest$PublishedRestResource","deleted":"10.4.0"}, + {"type":"Rest$PublishedRestService","introduced":"7.6.0"}, + {"type":"Rest$PublishedRestServiceOperation","introduced":"7.7.0"}, + {"type":"Rest$PublishedRestServiceResource","introduced":"7.7.0"}, + {"type":"Rest$QueryOptions","introduced":"9.17.0","deleted":"10.4.0"}, + {"type":"Rest$QueryParameter","introduced":"11.0.0"}, + {"type":"Rest$QueryParameterUsage","introduced":"11.0.0"}, + {"type":"Rest$ReadMode","introduced":"9.14.0","deleted":"10.4.0"}, + {"type":"Rest$ReadSource","introduced":"9.14.0","deleted":"10.4.0"}, + {"type":"Rest$RequiredQueryParameterUsage","introduced":"11.0.0"}, + {"type":"Rest$RestOperation","introduced":"10.1.0"}, + {"type":"Rest$RestOperationMethod","introduced":"10.4.0"}, + {"type":"Rest$RestOperationMethodWithBody","introduced":"10.4.0"}, + {"type":"Rest$RestOperationMethodWithoutBody","introduced":"10.4.0"}, + {"type":"Rest$RestOperationParameter","introduced":"7.8.0"}, + {"type":"Rest$RestOperationResponseHandling","introduced":"10.3.0"}, + {"type":"Rest$RestParameter","introduced":"10.17.0"}, + {"type":"Rest$ServiceFeed","introduced":"9.14.0","deleted":"10.4.0"}, + {"type":"Rest$StringBody","introduced":"10.4.0"}, + {"type":"Rest$StringValue","introduced":"10.1.0"}, + {"type":"Rest$Value","introduced":"10.1.0"}, + {"type":"Rest$ValueTemplate","introduced":"10.1.0"}, + {"type":"ScheduledEvents$DaySchedule","introduced":"9.12.0"}, + {"type":"ScheduledEvents$HourSchedule","introduced":"9.12.0"}, + {"type":"ScheduledEvents$MinuteSchedule","introduced":"9.12.0"}, + {"type":"ScheduledEvents$MonthDateSchedule","introduced":"9.12.0"}, + {"type":"ScheduledEvents$MonthSchedule","introduced":"9.12.0"}, + {"type":"ScheduledEvents$MonthWeekdaySchedule","introduced":"9.12.0"}, + {"type":"ScheduledEvents$Schedule","introduced":"9.12.0"}, + {"type":"ScheduledEvents$WeekSchedule","introduced":"9.12.0"}, + {"type":"ScheduledEvents$YearDateSchedule","introduced":"9.12.0"}, + {"type":"ScheduledEvents$YearSchedule","introduced":"9.12.0"}, + {"type":"ScheduledEvents$YearWeekdaySchedule","introduced":"9.12.0"}, + {"type":"Settings$ActionActivityDefaultColor","introduced":"8.6.0"}, + {"type":"Settings$DistributionSettings","introduced":"9.24.0"}, + {"type":"Settings$JarDeploymentSettings","introduced":"9.10.0"}, + {"type":"Settings$JarLocationBase","introduced":"9.10.0"}, + {"type":"Settings$JavaActionsSettings","introduced":"6.9.0","deleted":"8.0.0"}, + {"type":"Settings$KafkaConfiguration","introduced":"11.16.0"}, + {"type":"Settings$OpenTelemetryConfiguration","introduced":"11.12.0"}, + {"type":"Settings$OpenTelemetryLogsConfiguration","introduced":"11.12.0"}, + {"type":"Settings$OpenTelemetryTracesConfiguration","introduced":"11.12.0"}, + {"type":"Settings$PrivateValue","introduced":"10.9.0"}, + {"type":"Settings$ProtectedModuleJarLocation","introduced":"9.10.0"}, + {"type":"Settings$SharedOrPrivateValue","introduced":"10.9.0"}, + {"type":"Settings$SharedValue","introduced":"10.9.0"}, + {"type":"Settings$ThemeModuleEntry","introduced":"9.3.0"}, + {"type":"Settings$TracingConfiguration","introduced":"10.21.0","deleted":"11.12.0"}, + {"type":"Settings$UserLibJarLocation","introduced":"9.10.0"}, + {"type":"Settings$WorkflowGroup","introduced":"11.2.0"}, + {"type":"Settings$WorkflowsProjectSettingsPart","introduced":"8.8.0"}, + {"type":"Url$StaticUrlSegment","introduced":"10.1.0"}, + {"type":"Url$UrlSegment","introduced":"10.1.0"}, + {"type":"WebServices$PublishedAppService","deleted":"9.0.2"}, + {"type":"WebServices$SystemIdDataAttribute","introduced":"6.7.0"}, + {"type":"Workflows$AIAgentTaskActivity","introduced":"11.9.0"}, + {"type":"Workflows$AbsoluteAmountUserInput","introduced":"9.22.0"}, + {"type":"Workflows$AllUserInput","introduced":"9.22.0"}, + {"type":"Workflows$Annotation","introduced":"9.15.0"}, + {"type":"Workflows$BezierCurve","introduced":"11.1.0"}, + {"type":"Workflows$BooleanCase","introduced":"11.5.0"}, + {"type":"Workflows$BooleanConditionOutcome","introduced":"9.0.2"}, + {"type":"Workflows$BoundaryEvent","introduced":"10.14.0"}, + {"type":"Workflows$CallMicroflowActivity","introduced":"11.9.0"}, + {"type":"Workflows$CallMicroflowTask","introduced":"9.0.2","sdk":["Workflows$CallMicroflowActivity"]}, + {"type":"Workflows$CallWorkflowActivity","introduced":"9.0.2"}, + {"type":"Workflows$CaseValue","introduced":"11.5.0"}, + {"type":"Workflows$ConditionOutcome","introduced":"9.0.2"}, + {"type":"Workflows$ConditionOutcomeActivity","introduced":"9.0.2"}, + {"type":"Workflows$ConsensusCompletionCriteria","introduced":"9.22.0"}, + {"type":"Workflows$EmptyUserSource","introduced":"9.10.0","deleted":"11.2.0"}, + {"type":"Workflows$EndOfBoundaryEventPathActivity","introduced":"10.14.0"}, + {"type":"Workflows$EndOfParallelSplitPathActivity","introduced":"10.21.0"}, + {"type":"Workflows$EndWorkflowActivity","introduced":"9.0.2"}, + {"type":"Workflows$EnumerationValueConditionOutcome","introduced":"9.0.2"}, + {"type":"Workflows$EventSubProcess","introduced":"11.8.0"}, + {"type":"Workflows$ExclusiveSplitActivity","introduced":"9.0.2"}, + {"type":"Workflows$FloatingAnnotation","introduced":"11.1.0"}, + {"type":"Workflows$Flow","introduced":"9.0.2"}, + {"type":"Workflows$FlowLine","introduced":"11.1.0"}, + {"type":"Workflows$InterruptingNotificationBoundaryEvent","introduced":"11.11.0"}, + {"type":"Workflows$InterruptingNotificationEventSubProcessStartActivity","introduced":"11.10.0"}, + {"type":"Workflows$InterruptingNotificationEventSubProcessStartActivityTarget","introduced":"11.10.0"}, + {"type":"Workflows$InterruptingTimerBoundaryEvent","introduced":"10.20.0"}, + {"type":"Workflows$InterruptingTimerEventSubProcessStartActivity","introduced":"11.13.0"}, + {"type":"Workflows$JumpToActivity","introduced":"9.0.5"}, + {"type":"Workflows$Line","introduced":"11.1.0"}, + {"type":"Workflows$LinearRecurrence","introduced":"10.20.0"}, + {"type":"Workflows$MajorityCompletionCriteria","introduced":"10.0.0"}, + {"type":"Workflows$MergeActivity","introduced":"11.5.0"}, + {"type":"Workflows$MicroflowBasedActivity","introduced":"11.9.0"}, + {"type":"Workflows$MicroflowBasedEvent","introduced":"9.0.5"}, + {"type":"Workflows$MicroflowBasedTargeting","introduced":"11.2.0"}, + {"type":"Workflows$MicroflowBasedUserSource","introduced":"9.0.2","deleted":"11.2.0"}, + {"type":"Workflows$MicroflowCallParameterMapping","introduced":"9.0.2"}, + {"type":"Workflows$MicroflowCompletionCriteria","introduced":"10.3.0"}, + {"type":"Workflows$MicroflowEventHandler","introduced":"9.12.0"}, + {"type":"Workflows$MicroflowGroupTargeting","introduced":"11.2.0"}, + {"type":"Workflows$MicroflowUserTargeting","introduced":"11.2.0"}, + {"type":"Workflows$MultiInputCompletion","introduced":"9.22.0","deleted":"10.12.0"}, + {"type":"Workflows$MultiUserTaskActivity","introduced":"10.12.0"}, + {"type":"Workflows$NoEvent","introduced":"9.0.5"}, + {"type":"Workflows$NoUserTargeting","introduced":"11.2.0"}, + {"type":"Workflows$NonInterruptingNotificationBoundaryEvent","introduced":"11.11.0"}, + {"type":"Workflows$NonInterruptingNotificationEventSubProcessStartActivity","introduced":"11.8.0"}, + {"type":"Workflows$NonInterruptingNotificationEventSubProcessStartActivityTarget","introduced":"11.8.0"}, + {"type":"Workflows$NonInterruptingTimerBoundaryEvent","introduced":"10.20.0"}, + {"type":"Workflows$NonInterruptingTimerEventSubProcessStartActivity","introduced":"11.13.0"}, + {"type":"Workflows$NotificationActivity","introduced":"11.11.0"}, + {"type":"Workflows$NotificationBoundaryEvent","introduced":"11.11.0"}, + {"type":"Workflows$NotificationEventSubProcessStartActivity","introduced":"11.8.0"}, + {"type":"Workflows$NotifyNotificationActivityTarget","introduced":"11.11.0"}, + {"type":"Workflows$NotifyNotificationBoundaryEventTarget","introduced":"11.11.0"}, + {"type":"Workflows$NotifyTarget","introduced":"11.7.0"}, + {"type":"Workflows$NotifyWaitForNotificationActivityTarget","introduced":"11.7.0"}, + {"type":"Workflows$OrthogonalPath","introduced":"11.1.0"}, + {"type":"Workflows$Outcome","introduced":"9.0.2"}, + {"type":"Workflows$PageParameterMapping","introduced":"9.11.0"}, + {"type":"Workflows$PageReference","introduced":"9.11.0"}, + {"type":"Workflows$ParallelSplitActivity","introduced":"9.0.2"}, + {"type":"Workflows$ParallelSplitOutcome","introduced":"9.0.2"}, + {"type":"Workflows$Parameter","introduced":"9.6.0"}, + {"type":"Workflows$PercentageAmountUserInput","introduced":"9.22.0"}, + {"type":"Workflows$Recurrence","introduced":"10.20.0"}, + {"type":"Workflows$SingleInputCompletion","introduced":"9.22.0","deleted":"10.12.0"}, + {"type":"Workflows$SingleUserTaskActivity","introduced":"10.12.0"}, + {"type":"Workflows$StartWorkflowActivity","introduced":"11.1.0"}, + {"type":"Workflows$StringCase","introduced":"11.5.0"}, + {"type":"Workflows$TargetUserInput","introduced":"9.22.0"}, + {"type":"Workflows$ThresholdCompletionCriteria","introduced":"10.0.0"}, + {"type":"Workflows$TimerBoundaryEvent","introduced":"10.14.0","deleted":"10.20.0"}, + {"type":"Workflows$TimerEventSubProcessStartActivity","introduced":"11.13.0"}, + {"type":"Workflows$UserSource","introduced":"9.0.2","deleted":"11.2.0"}, + {"type":"Workflows$UserTargeting","introduced":"11.2.0"}, + {"type":"Workflows$UserTask","introduced":"9.0.2","deleted":"10.12.0"}, + {"type":"Workflows$UserTaskActivity","introduced":"10.12.0"}, + {"type":"Workflows$UserTaskCompletion","introduced":"9.22.0","deleted":"10.12.0"}, + {"type":"Workflows$UserTaskCompletionCriteria","introduced":"9.22.0"}, + {"type":"Workflows$UserTaskEvent","introduced":"9.0.5"}, + {"type":"Workflows$UserTaskOutcome","introduced":"9.0.2"}, + {"type":"Workflows$VetoCompletionCriteria","introduced":"9.22.0"}, + {"type":"Workflows$VoidCase","introduced":"11.5.0"}, + {"type":"Workflows$VoidConditionOutcome","introduced":"9.0.2"}, + {"type":"Workflows$WaitForNotificationActivity","introduced":"10.0.0"}, + {"type":"Workflows$WaitForTimerActivity","introduced":"10.6.0"}, + {"type":"Workflows$Workflow","introduced":"9.0.2"}, + {"type":"Workflows$WorkflowActivity","introduced":"9.0.2"}, + {"type":"Workflows$WorkflowCallParameterMapping","introduced":"9.18.0"}, + {"type":"Workflows$WorkflowDefinitionNameSelection","introduced":"10.0.0"}, + {"type":"Workflows$WorkflowDefinitionObjectSelection","introduced":"10.0.0"}, + {"type":"Workflows$WorkflowDefinitionSelection","introduced":"10.0.0"}, + {"type":"Workflows$WorkflowEventHandler","introduced":"10.7.0"}, + {"type":"Workflows$WorkflowMetaData","introduced":"11.1.0"}, + {"type":"Workflows$WorkflowType","introduced":"9.7.0","deleted":"9.10.0"}, + {"type":"Workflows$XPathBasedTargeting","introduced":"11.2.0"}, + {"type":"Workflows$XPathBasedUserSource","introduced":"9.0.2","deleted":"11.2.0"}, + {"type":"Workflows$XPathGroupTargeting","introduced":"11.2.0"}, + {"type":"Workflows$XPathUserTargeting","introduced":"11.2.0"}, + {"type":"XmlSchemas$XmlElement","introduced":"6.6.0"} + ] +} diff --git a/modelsdk/canon/propertyset.go b/modelsdk/canon/propertyset.go index d2cd625d8e..952034d04e 100644 --- a/modelsdk/canon/propertyset.go +++ b/modelsdk/canon/propertyset.go @@ -59,6 +59,14 @@ import ( // exists in versions it was not measured on is left out on them: the gap this // closes, not a new one. A nil version adds nothing. // +// So Since is only as low as the oldest version anyone measured, and every +// version below it keeps its gaps. Measured from 10.24.24 up, the table gave +// 10.24.0–10.24.23 two of its rows, and a nanoflow re-created on 10.24.18 +// broke `mx diff` (ako/mxcli#1505); measure the lowest patch of a version line, +// not the newest. Stripping a row's key from Studio Pro-authored elements and +// letting `mx convert` write it back measures the row without any mxcli writer +// leaving it out. +// // # Limits // // It only ADDS. A key mxcli writes that Mendix's metamodel does not have is a diff --git a/modelsdk/canon/propertyset_issue1097_test.go b/modelsdk/canon/propertyset_issue1097_test.go index 380682aa78..fcefc73c04 100644 --- a/modelsdk/canon/propertyset_issue1097_test.go +++ b/modelsdk/canon/propertyset_issue1097_test.go @@ -16,6 +16,11 @@ import ( // mappings; testdata/expr-checker (11.6.6) holds the same EntityRef and no // MessageDefinition2 at all, which gen's Mappings$MappingDocument records as // Introduced 11.10.0. +// +// EntityRef's floor was the lowest fixture that held it, not the version that +// introduced it. ako/mxcli#1505 measured it lower: Studio Pro 10.24.0 stores +// EntityRef null on all 75 AttributeRefs of a blank app, and `mx convert` +// 10.24.0 and 10.24.18 write the same null back where it is stripped. func TestCompletePropertySets_Issue1097Rows(t *testing.T) { cases := []struct { typ, key string @@ -23,7 +28,7 @@ func TestCompletePropertySets_Issue1097Rows(t *testing.T) { since string below string }{ - {"DomainModels$AttributeRef", "EntityRef", nil, "11.6.6", "11.6.5"}, + {"DomainModels$AttributeRef", "EntityRef", nil, "10.24.0", "10.23.0"}, {"ImportMappings$ImportMapping", "MessageDefinition2", "", "11.10.0", "11.9.9"}, {"ExportMappings$ExportMapping", "MessageDefinition2", "", "11.10.0", "11.9.9"}, } diff --git a/modelsdk/canon/propertyset_issue1505_test.go b/modelsdk/canon/propertyset_issue1505_test.go new file mode 100644 index 0000000000..98ed036a52 --- /dev/null +++ b/modelsdk/canon/propertyset_issue1505_test.go @@ -0,0 +1,76 @@ +// SPDX-License-Identifier: Apache-2.0 + +package canon + +import ( + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +// ako/mxcli#1505: on Mendix 10.24.18 a nanoflow mxcli re-created, and a fresh +// microflow and entity, lacked keys Studio Pro writes, so `mx diff` threw +// +// Objects with ID … of type Microflows$Nanoflow do not have the same +// properties. baseNames = …, ExportLevel, …, ReturnVariableName; newNames = … +// +// The table held every one of them, with "since": "10.24.24" — the lowest +// version anyone had MEASURED, read by appliesTo as the version that introduced +// the key. Each value below is what `mx convert --in-place` wrote on 10.24.18 +// and on 10.24.0, the lowest 10.24 patch, for the element mxcli left without +// it (the key-set diff of every $ID before and after the conversion). +func TestCompletePropertySets_Issue1505RowsBelow10_24_24(t *testing.T) { + cases := []struct { + typ, key string + check func(any) bool + }{ + {"Microflows$Nanoflow", "ExportLevel", eq("Hidden")}, + {"Microflows$Nanoflow", "ReturnVariableName", eq("")}, + {"Microflows$AggregateAction", "UseExpression", eq(false)}, + {"Microflows$AggregateAction", "Expression", eq("")}, + {"Microflows$AggregateAction", "ReduceInitialValueExpression", eq("")}, + {"Microflows$AggregateAction", "ReduceReturnDataType", isElement("DataTypes$BooleanType")}, + {"Microflows$DeleteAction", "ErrorHandlingType", eq("Rollback")}, + {"Microflows$WhileLoopCondition", "Caption", eq("")}, + {"Microflows$DatabaseRetrieveSource", "XpathConstraint", eq("")}, + {"DomainModels$EntityImpl", "Image", eq("")}, + {"DomainModels$EntityImpl", "Source", eq(nil)}, + {"DomainModels$EntityImpl", "ImageData", isEmptyBinary}, + } + for _, ver := range []string{"10.24.18", "10.24.0"} { + for _, c := range cases { + t.Run(ver+"/"+c.typ+"."+c.key, func(t *testing.T) { + in := mustMarshal(t, bson.D{ + {Key: "$ID", Value: freshID()}, + {Key: "$Type", Value: "Test$Holder"}, + {Key: "El", Value: bson.D{{Key: "$ID", Value: freshID()}, {Key: "$Type", Value: c.typ}}}, + }) + got, ok := keysOf(t, CompletePropertySets(in, v(ver)), "El")[c.key] + if !ok { + t.Fatalf("a Mendix %s project got %s without %s; Studio Pro %s writes it", ver, c.typ, c.key, ver) + } + if !c.check(got) { + t.Fatalf("%s.%s = %#v on %s, not the value mx convert wrote", c.typ, c.key, got, ver) + } + }) + } + } +} + +func eq(want any) func(any) bool { return func(got any) bool { return got == want } } + +func isElement(typ string) func(any) bool { + return func(got any) bool { + d, ok := got.(bson.D) + if !ok { + return false + } + s, _ := lookupString(d, "$Type") + return s == typ && elementIDBytes(d) != nil + } +} + +func isEmptyBinary(got any) bool { + b, ok := got.(bson.Binary) + return ok && len(b.Data) == 0 +} diff --git a/modelsdk/canon/studiopro_property_defaults.json b/modelsdk/canon/studiopro_property_defaults.json index f05785e613..9af1dd8033 100644 --- a/modelsdk/canon/studiopro_property_defaults.json +++ b/modelsdk/canon/studiopro_property_defaults.json @@ -1,113 +1,113 @@ [ - {"type":"BusinessEvents$BusinessEventService","key":"Document","since":"10.24.24","seen":"10.24.24","value":{"v":""}}, - {"type":"BusinessEvents$MessageAttribute","key":"EnumerationDefinition","since":"10.24.24","seen":"10.24.24","value":{"v":null}}, + {"type":"BusinessEvents$BusinessEventService","key":"Document","since":"10.24.0","seen":"10.24.24","value":{"v":""}}, + {"type":"BusinessEvents$MessageAttribute","key":"EnumerationDefinition","since":"10.24.0","seen":"10.24.24","value":{"v":null}}, {"type":"DatabaseConnector$SimpleSqlDataType","key":"DataTypeName","since":"10.24.24","seen":"10.24.24","value":{"v":""}}, - {"type":"DomainModels$AccessRule","key":"Documentation","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"DomainModels$AccessRule","key":"XPathConstraintCaption","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"DomainModels$AttributeRef","key":"EntityRef","since":"11.6.6","seen":"11.13.0","value":{"v":null}}, - {"type":"DomainModels$EntityImpl","key":"Image","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"DomainModels$EntityImpl","key":"ImageData","since":"10.24.24","seen":"11.15.0","value":{"v":{"$binary":{"base64":"","subType":"00"}}}}, - {"type":"DomainModels$EntityImpl","key":"Source","since":"10.24.24","seen":"11.15.0","value":{"v":null}}, - {"type":"DomainModels$RangeRuleInfo","key":"MaxAttribute","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"DomainModels$RangeRuleInfo","key":"MaxValue","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"DomainModels$RangeRuleInfo","key":"MinAttribute","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"DomainModels$RangeRuleInfo","key":"MinValue","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"DomainModels$ValidationRule","key":"Message","since":"10.24.24","seen":"11.15.0","value":{"v":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"Texts$Text","Items":[{"$numberInt":"3"}]}}}, + {"type":"DomainModels$AccessRule","key":"Documentation","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"DomainModels$AccessRule","key":"XPathConstraintCaption","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"DomainModels$AttributeRef","key":"EntityRef","since":"10.24.0","seen":"11.13.0","value":{"v":null}}, + {"type":"DomainModels$EntityImpl","key":"Image","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"DomainModels$EntityImpl","key":"ImageData","since":"10.24.0","seen":"11.15.0","value":{"v":{"$binary":{"base64":"","subType":"00"}}}}, + {"type":"DomainModels$EntityImpl","key":"Source","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, + {"type":"DomainModels$RangeRuleInfo","key":"MaxAttribute","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"DomainModels$RangeRuleInfo","key":"MaxValue","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"DomainModels$RangeRuleInfo","key":"MinAttribute","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"DomainModels$RangeRuleInfo","key":"MinValue","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"DomainModels$ValidationRule","key":"Message","since":"10.24.0","seen":"11.15.0","value":{"v":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"Texts$Text","Items":[{"$numberInt":"3"}]}}}, {"type":"ExportMappings$ExportMapping","key":"MessageDefinition2","since":"11.10.0","seen":"11.13.0","value":{"v":""}}, - {"type":"Forms$CheckBox","key":"AriaRequired","since":"10.24.24","seen":"11.15.0","value":{"v":false}}, - {"type":"Forms$CheckBox","key":"LabelPosition","since":"10.24.24","seen":"11.15.0","value":{"v":"Default"}}, - {"type":"Forms$CheckBox","key":"NativeRenderMode","since":"10.24.24","seen":"11.15.0","value":{"v":"Switch"}}, - {"type":"Forms$ClosePageClientAction","key":"NumberOfPagesToClose","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"Forms$DeleteClientAction","key":"SourceVariable","since":"10.24.24","seen":"11.15.0","value":{"v":null}}, - {"type":"Forms$DivContainer","key":"Widgets","since":"10.24.24","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, - {"type":"Forms$FormCallArgument","key":"Widgets","since":"10.24.24","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, - {"type":"Forms$LayoutGridColumn","key":"Widgets","since":"10.24.24","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, - {"type":"Forms$ScrollContainer","key":"Alignment","since":"10.24.24","seen":"11.15.0","value":{"v":"Center"}}, - {"type":"Forms$ScrollContainer","key":"Bottom","since":"10.24.24","seen":"11.15.0","value":{"v":null}}, - {"type":"Forms$ScrollContainer","key":"LayoutMode","since":"10.24.24","seen":"11.15.0","value":{"v":"Headline"}}, - {"type":"Forms$ScrollContainer","key":"Left","since":"10.24.24","seen":"11.15.0","value":{"v":null}}, - {"type":"Forms$ScrollContainer","key":"NativeHideScrollbars","since":"10.24.24","seen":"11.15.0","value":{"v":false}}, - {"type":"Forms$ScrollContainer","key":"Right","since":"10.24.24","seen":"11.15.0","value":{"v":null}}, - {"type":"Forms$ScrollContainer","key":"ScrollBehavior","since":"10.24.24","seen":"11.15.0","value":{"v":"PerRegion"}}, - {"type":"Forms$ScrollContainer","key":"Top","since":"10.24.24","seen":"11.15.0","value":{"v":null}}, - {"type":"Forms$ScrollContainer","key":"Width","since":"10.24.24","seen":"11.15.0","value":{"v":{"$numberLong":"960"}}}, - {"type":"Forms$ScrollContainer","key":"WidthMode","since":"10.24.24","seen":"11.15.0","value":{"v":"Auto"}}, - {"type":"Forms$ScrollContainerRegion","key":"ToggleMode","since":"10.24.24","seen":"11.15.0","value":{"v":"None"}}, - {"type":"Forms$TabControl","key":"ActivePageAttributeRef","since":"10.24.24","seen":"11.15.0","value":{"v":null}}, - {"type":"Forms$TabControl","key":"ActivePageSourceVariable","since":"10.24.24","seen":"11.15.0","value":{"v":null}}, - {"type":"Forms$TabControl","key":"ConditionalVisibilitySettings","since":"10.24.24","seen":"11.15.0","value":{"v":null}}, - {"type":"Forms$TabControl","key":"DefaultPagePointer","since":"10.24.24","seen":"11.15.0","value":{"v":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}}}}, - {"type":"Forms$TabPage","key":"Badge","since":"10.24.24","seen":"11.15.0","value":{"v":null}}, - {"type":"Forms$TabPage","key":"ConditionalVisibilitySettings","since":"10.24.24","seen":"11.15.0","value":{"v":null}}, - {"type":"Forms$TextArea","key":"Autocomplete","since":"10.24.24","seen":"11.15.0","value":{"v":true}}, - {"type":"Forms$WebLayoutContent","key":"LayoutCall","since":"10.24.24","seen":"11.15.0","value":{"v":null}}, + {"type":"Forms$CheckBox","key":"AriaRequired","since":"10.24.0","seen":"11.15.0","value":{"v":false}}, + {"type":"Forms$CheckBox","key":"LabelPosition","since":"10.24.0","seen":"11.15.0","value":{"v":"Default"}}, + {"type":"Forms$CheckBox","key":"NativeRenderMode","since":"10.24.0","seen":"11.15.0","value":{"v":"Switch"}}, + {"type":"Forms$ClosePageClientAction","key":"NumberOfPagesToClose","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"Forms$DeleteClientAction","key":"SourceVariable","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, + {"type":"Forms$DivContainer","key":"Widgets","since":"10.24.0","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, + {"type":"Forms$FormCallArgument","key":"Widgets","since":"10.24.0","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, + {"type":"Forms$LayoutGridColumn","key":"Widgets","since":"10.24.0","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, + {"type":"Forms$ScrollContainer","key":"Alignment","since":"10.24.0","seen":"11.15.0","value":{"v":"Center"}}, + {"type":"Forms$ScrollContainer","key":"Bottom","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, + {"type":"Forms$ScrollContainer","key":"LayoutMode","since":"10.24.0","seen":"11.15.0","value":{"v":"Headline"}}, + {"type":"Forms$ScrollContainer","key":"Left","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, + {"type":"Forms$ScrollContainer","key":"NativeHideScrollbars","since":"10.24.0","seen":"11.15.0","value":{"v":false}}, + {"type":"Forms$ScrollContainer","key":"Right","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, + {"type":"Forms$ScrollContainer","key":"ScrollBehavior","since":"10.24.0","seen":"11.15.0","value":{"v":"PerRegion"}}, + {"type":"Forms$ScrollContainer","key":"Top","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, + {"type":"Forms$ScrollContainer","key":"Width","since":"10.24.0","seen":"11.15.0","value":{"v":{"$numberLong":"960"}}}, + {"type":"Forms$ScrollContainer","key":"WidthMode","since":"10.24.0","seen":"11.15.0","value":{"v":"Auto"}}, + {"type":"Forms$ScrollContainerRegion","key":"ToggleMode","since":"10.24.0","seen":"11.15.0","value":{"v":"None"}}, + {"type":"Forms$TabControl","key":"ActivePageAttributeRef","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, + {"type":"Forms$TabControl","key":"ActivePageSourceVariable","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, + {"type":"Forms$TabControl","key":"ConditionalVisibilitySettings","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, + {"type":"Forms$TabControl","key":"DefaultPagePointer","since":"10.24.0","seen":"11.15.0","value":{"v":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}}}}, + {"type":"Forms$TabPage","key":"Badge","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, + {"type":"Forms$TabPage","key":"ConditionalVisibilitySettings","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, + {"type":"Forms$TextArea","key":"Autocomplete","since":"10.24.0","seen":"11.15.0","value":{"v":true}}, + {"type":"Forms$WebLayoutContent","key":"LayoutCall","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, {"type":"ImportMappings$ImportMapping","key":"MessageDefinition2","since":"11.10.0","seen":"11.13.0","value":{"v":""}}, - {"type":"Microflows$AggregateAction","key":"Expression","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"Microflows$AggregateAction","key":"ReduceInitialValueExpression","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"Microflows$AggregateAction","key":"ReduceReturnDataType","since":"10.24.24","seen":"11.15.0","value":{"v":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"DataTypes$BooleanType"}}}, - {"type":"Microflows$AggregateAction","key":"UseExpression","since":"10.24.24","seen":"11.15.0","value":{"v":false}}, - {"type":"Microflows$CallExternalAction","key":"IncludedAssociations","since":"10.24.24","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, - {"type":"Microflows$CallWebServiceAction","key":"ErrorHandlingType","since":"10.24.24","seen":"11.15.0","value":{"v":"Rollback"}}, - {"type":"Microflows$CallWebServiceAction","key":"HttpConfiguration","since":"10.24.24","seen":"11.15.0","value":{"v":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"Microflows$HttpConfiguration","ClientCertificate":"","CustomLocation":"","CustomLocationTemplate":null,"HttpAuthenticationPassword":"","HttpAuthenticationUserName":"","HttpHeaderEntries":[{"$numberInt":"3"}],"HttpMethod":"Post","OverrideLocation":false,"UseHttpAuthentication":false}}}, - {"type":"Microflows$CallWebServiceAction","key":"IsValidationRequired","since":"10.24.24","seen":"11.15.0","value":{"v":false}}, - {"type":"Microflows$CallWebServiceAction","key":"NewResultHandling","since":"10.24.24","seen":"11.15.0","value":{"v":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"Microflows$ResultHandling","Bind":false,"ImportMappingCall":null,"ResultVariableName":"","VariableType":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"DataTypes$VoidType"}}}}, - {"type":"Microflows$CallWebServiceAction","key":"ProxyConfiguration","since":"10.24.24","seen":"11.15.0","value":{"v":null}}, - {"type":"Microflows$CallWebServiceAction","key":"RequestBodyHandling","since":"10.24.24","seen":"11.15.0","value":{"v":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"Microflows$SimpleRequestHandling","NullValueOption":"LeaveOutElement","ParameterMappings":[{"$numberInt":"2"}]}}}, - {"type":"Microflows$CallWebServiceAction","key":"RequestHeaderHandling","since":"10.24.24","seen":"11.15.0","value":{"v":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"Microflows$SimpleRequestHandling","NullValueOption":"LeaveOutElement","ParameterMappings":[{"$numberInt":"2"}]}}}, - {"type":"Microflows$CallWebServiceAction","key":"RequestProxyType","since":"10.24.24","seen":"11.15.0","value":{"v":"DefaultProxy"}}, - {"type":"Microflows$CallWebServiceAction","key":"ServiceName","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"Microflows$ChangeAction","key":"Items","since":"10.24.24","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, - {"type":"Microflows$ChangeActionItem","key":"Attribute","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"Microflows$CloseFormAction","key":"NumberOfPagesToClose","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"Microflows$CreateChangeAction","key":"Items","since":"10.24.24","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, - {"type":"Microflows$DatabaseRetrieveSource","key":"XpathConstraint","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"Microflows$DeleteAction","key":"ErrorHandlingType","since":"10.24.24","seen":"11.15.0","value":{"v":"Rollback"}}, - {"type":"Microflows$HttpConfiguration","key":"CustomLocationTemplate","since":"10.24.24","seen":"11.15.0","value":{"v":null}}, - {"type":"Microflows$JavaActionCallAction","key":"ParameterMappings","since":"10.24.24","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, - {"type":"Microflows$JavaScriptActionCallAction","key":"ParameterMappings","since":"10.24.24","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, - {"type":"Microflows$LoopedActivity","key":"Documentation","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"Microflows$Nanoflow","key":"ExportLevel","since":"10.24.24","seen":"11.15.0","value":{"v":"Hidden"}}, - {"type":"Microflows$Nanoflow","key":"ReturnVariableName","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, + {"type":"Microflows$AggregateAction","key":"Expression","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"Microflows$AggregateAction","key":"ReduceInitialValueExpression","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"Microflows$AggregateAction","key":"ReduceReturnDataType","since":"10.24.0","seen":"11.15.0","value":{"v":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"DataTypes$BooleanType"}}}, + {"type":"Microflows$AggregateAction","key":"UseExpression","since":"10.24.0","seen":"11.15.0","value":{"v":false}}, + {"type":"Microflows$CallExternalAction","key":"IncludedAssociations","since":"10.24.0","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, + {"type":"Microflows$CallWebServiceAction","key":"ErrorHandlingType","since":"10.24.0","seen":"11.15.0","value":{"v":"Rollback"}}, + {"type":"Microflows$CallWebServiceAction","key":"HttpConfiguration","since":"10.24.0","seen":"11.15.0","value":{"v":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"Microflows$HttpConfiguration","ClientCertificate":"","CustomLocation":"","CustomLocationTemplate":null,"HttpAuthenticationPassword":"","HttpAuthenticationUserName":"","HttpHeaderEntries":[{"$numberInt":"3"}],"HttpMethod":"Post","OverrideLocation":false,"UseHttpAuthentication":false}}}, + {"type":"Microflows$CallWebServiceAction","key":"IsValidationRequired","since":"10.24.0","seen":"11.15.0","value":{"v":false}}, + {"type":"Microflows$CallWebServiceAction","key":"NewResultHandling","since":"10.24.0","seen":"11.15.0","value":{"v":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"Microflows$ResultHandling","Bind":false,"ImportMappingCall":null,"ResultVariableName":"","VariableType":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"DataTypes$VoidType"}}}}, + {"type":"Microflows$CallWebServiceAction","key":"ProxyConfiguration","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, + {"type":"Microflows$CallWebServiceAction","key":"RequestBodyHandling","since":"10.24.0","seen":"11.15.0","value":{"v":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"Microflows$SimpleRequestHandling","NullValueOption":"LeaveOutElement","ParameterMappings":[{"$numberInt":"2"}]}}}, + {"type":"Microflows$CallWebServiceAction","key":"RequestHeaderHandling","since":"10.24.0","seen":"11.15.0","value":{"v":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"Microflows$SimpleRequestHandling","NullValueOption":"LeaveOutElement","ParameterMappings":[{"$numberInt":"2"}]}}}, + {"type":"Microflows$CallWebServiceAction","key":"RequestProxyType","since":"10.24.0","seen":"11.15.0","value":{"v":"DefaultProxy"}}, + {"type":"Microflows$CallWebServiceAction","key":"ServiceName","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"Microflows$ChangeAction","key":"Items","since":"10.24.0","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, + {"type":"Microflows$ChangeActionItem","key":"Attribute","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"Microflows$CloseFormAction","key":"NumberOfPagesToClose","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"Microflows$CreateChangeAction","key":"Items","since":"10.24.0","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, + {"type":"Microflows$DatabaseRetrieveSource","key":"XpathConstraint","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"Microflows$DeleteAction","key":"ErrorHandlingType","since":"10.24.0","seen":"11.15.0","value":{"v":"Rollback"}}, + {"type":"Microflows$HttpConfiguration","key":"CustomLocationTemplate","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, + {"type":"Microflows$JavaActionCallAction","key":"ParameterMappings","since":"10.24.0","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, + {"type":"Microflows$JavaScriptActionCallAction","key":"ParameterMappings","since":"10.24.0","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, + {"type":"Microflows$LoopedActivity","key":"Documentation","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"Microflows$Nanoflow","key":"ExportLevel","since":"10.24.0","seen":"11.15.0","value":{"v":"Hidden"}}, + {"type":"Microflows$Nanoflow","key":"ReturnVariableName","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, {"type":"Microflows$Nanoflow","key":"UseListParameterByReference","since":"11.10.0","seen":"11.15.0","value":{"v":true}}, - {"type":"Microflows$OpenUserTaskAction","key":"AssignOnOpen","since":"11.6.8","seen":"11.15.0","value":{"v":true}}, - {"type":"Microflows$OpenUserTaskAction","key":"OpenWhenAssigned","since":"11.6.8","seen":"11.15.0","value":{"v":false}}, - {"type":"Microflows$RestOperationCallAction","key":"ErrorHandlingType","since":"11.6.8","seen":"11.15.0","value":{"v":"Rollback"}}, - {"type":"Microflows$ValidationFeedbackAction","key":"Association","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"Microflows$WhileLoopCondition","key":"Caption","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"ODataPublish$PublishedODataService2","key":"ExportLevel","since":"10.24.24","seen":"11.15.0","value":{"v":"Hidden"}}, - {"type":"Projects$JarDependency","key":"Exclusions","since":"10.24.24","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, - {"type":"Projects$ModuleImpl","key":"AppStorePackageId","since":"10.24.24","until":"10.25.0","seen":"10.24.24","value":{"v":{"$numberLong":"0"}}}, + {"type":"Microflows$OpenUserTaskAction","key":"AssignOnOpen","since":"10.24.0","seen":"11.15.0","value":{"v":true}}, + {"type":"Microflows$OpenUserTaskAction","key":"OpenWhenAssigned","since":"10.24.0","seen":"11.15.0","value":{"v":false}}, + {"type":"Microflows$RestOperationCallAction","key":"ErrorHandlingType","since":"10.24.0","seen":"11.15.0","value":{"v":"Rollback"}}, + {"type":"Microflows$ValidationFeedbackAction","key":"Association","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"Microflows$WhileLoopCondition","key":"Caption","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"ODataPublish$PublishedODataService2","key":"ExportLevel","since":"10.24.0","seen":"11.15.0","value":{"v":"Hidden"}}, + {"type":"Projects$JarDependency","key":"Exclusions","since":"10.24.0","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, + {"type":"Projects$ModuleImpl","key":"AppStorePackageId","since":"10.24.0","until":"10.25.0","seen":"10.24.24","value":{"v":{"$numberLong":"0"}}}, {"type":"Projects$ModuleSettings","key":"Checksum","since":"11.12.0","seen":"11.15.0","value":{"v":""}}, {"type":"Projects$ModuleSettings","key":"ConvertedChecksum","since":"11.12.0","seen":"11.15.0","value":{"v":""}}, {"type":"Projects$ModuleSettings","key":"EnableDetailedTroubleshooting","since":"11.12.0","seen":"11.15.0","value":{"v":true}}, {"type":"Projects$ModuleSettings","key":"ModuleDependencies","since":"11.12.0","seen":"11.15.0","value":{"v":null}}, {"type":"Projects$ModuleSettings","key":"OriginalPackageId","since":"11.12.0","seen":"11.15.0","value":{"v":""}}, {"type":"Projects$ModuleSettings","key":"PackageId","since":"11.12.0","seen":"11.15.0","value":{"v":""}}, - {"type":"Rest$ConsumedODataService","key":"ApplicationId","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"Rest$ConsumedODataService","key":"CatalogUrl","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, + {"type":"Rest$ConsumedODataService","key":"ApplicationId","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"Rest$ConsumedODataService","key":"CatalogUrl","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, {"type":"Rest$ConsumedODataService","key":"ConfigurationEntityMicroflow","since":"11.10.0","seen":"11.15.0","value":{"v":""}}, - {"type":"Rest$ConsumedODataService","key":"ConfigurationMicroflow","since":"10.24.24","until":"11.7.0","seen":"11.6.8","value":{"v":""}}, - {"type":"Rest$ConsumedODataService","key":"EndpointId","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"Rest$ConsumedODataService","key":"EnvironmentType","since":"10.24.24","seen":"11.15.0","value":{"v":"Unknown"}}, - {"type":"Rest$ConsumedODataService","key":"ErrorHandlingMicroflow","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, + {"type":"Rest$ConsumedODataService","key":"ConfigurationMicroflow","since":"10.24.0","until":"11.7.0","seen":"11.6.8","value":{"v":""}}, + {"type":"Rest$ConsumedODataService","key":"EndpointId","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"Rest$ConsumedODataService","key":"EnvironmentType","since":"10.24.0","seen":"11.15.0","value":{"v":"Unknown"}}, + {"type":"Rest$ConsumedODataService","key":"ErrorHandlingMicroflow","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, {"type":"Rest$ConsumedODataService","key":"HeaderListMicroflow","since":"11.10.0","seen":"11.15.0","value":{"v":""}}, - {"type":"Rest$ConsumedODataService","key":"Icon","since":"10.24.24","seen":"11.15.0","value":{"v":{"$binary":{"base64":"","subType":"00"}}}}, - {"type":"Rest$ConsumedODataService","key":"ProxyHost","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"Rest$ConsumedODataService","key":"ProxyPassword","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"Rest$ConsumedODataService","key":"ProxyPort","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"Rest$ConsumedODataService","key":"ProxyUsername","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"Rest$ConsumedRestService","key":"OpenApiFile","since":"10.24.24","seen":"11.15.0","value":{"v":null}}, - {"type":"Rest$NoResponseHandling","key":"ContentType","since":"10.24.24","seen":"11.15.0","value":{"v":"application/json"}}, - {"type":"Rest$NoResponseHandling","key":"StatusCode","since":"10.24.24","seen":"11.15.0","value":{"v":{"$numberLong":"200"}}}, + {"type":"Rest$ConsumedODataService","key":"Icon","since":"10.24.0","seen":"11.15.0","value":{"v":{"$binary":{"base64":"","subType":"00"}}}}, + {"type":"Rest$ConsumedODataService","key":"ProxyHost","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"Rest$ConsumedODataService","key":"ProxyPassword","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"Rest$ConsumedODataService","key":"ProxyPort","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"Rest$ConsumedODataService","key":"ProxyUsername","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"Rest$ConsumedRestService","key":"OpenApiFile","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, + {"type":"Rest$NoResponseHandling","key":"ContentType","since":"10.24.0","seen":"11.15.0","value":{"v":"application/json"}}, + {"type":"Rest$NoResponseHandling","key":"StatusCode","since":"10.24.0","seen":"11.15.0","value":{"v":{"$numberLong":"200"}}}, {"type":"Rest$OperationParameter","key":"TestValue","since":"11.6.8","seen":"11.15.0","value":{"v":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"Rest$StringValue","Value":""}}}, {"type":"Rest$PublishedRestService","key":"PublicDocumentation","since":"11.10.0","seen":"11.15.0","value":{"v":""}}, {"type":"Rest$QueryParameter","key":"TestValue","since":"11.6.8","seen":"11.15.0","value":{"v":""}}, - {"type":"Security$UserRole","key":"CheckSecurity","since":"10.24.24","seen":"11.15.0","value":{"v":true}}, - {"type":"Security$UserRole","key":"Description","since":"10.24.24","seen":"11.15.0","value":{"v":""}}, - {"type":"Security$UserRole","key":"ManageUsersWithoutRoles","since":"10.24.24","seen":"11.15.0","value":{"v":false}}, - {"type":"Security$UserRole","key":"ManageableRoles","since":"10.24.24","seen":"11.15.0","value":{"v":[{"$numberInt":"1"}]}}, - {"type":"Workflows$InterruptingTimerBoundaryEvent","key":"Annotation","since":"11.6.8","seen":"11.15.0","value":{"v":null}}, - {"type":"Workflows$NonInterruptingTimerBoundaryEvent","key":"Annotation","since":"11.6.8","seen":"11.15.0","value":{"v":null}}, + {"type":"Security$UserRole","key":"CheckSecurity","since":"10.24.0","seen":"11.15.0","value":{"v":true}}, + {"type":"Security$UserRole","key":"Description","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"Security$UserRole","key":"ManageUsersWithoutRoles","since":"10.24.0","seen":"11.15.0","value":{"v":false}}, + {"type":"Security$UserRole","key":"ManageableRoles","since":"10.24.0","seen":"11.15.0","value":{"v":[{"$numberInt":"1"}]}}, + {"type":"Workflows$InterruptingTimerBoundaryEvent","key":"Annotation","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, + {"type":"Workflows$NonInterruptingTimerBoundaryEvent","key":"Annotation","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, {"type":"Workflows$NonInterruptingTimerEventSubProcessStartActivity","key":"Recurrence","since":"11.15.0","seen":"11.15.0","value":{"v":null}}, {"type":"Workflows$Workflow","key":"UsertaskOnStateChangeEvent","since":"9.24.30","until":"10.25.0","seen":"10.24.24","value":{"v":null}}, {"type":"Workflows$Workflow","key":"WorkflowOnStateChangeEvent","since":"9.24.30","until":"10.25.0","seen":"10.24.24","value":{"v":null}} diff --git a/modelsdk/canon/typefloor.go b/modelsdk/canon/typefloor.go new file mode 100644 index 0000000000..3847cfed11 --- /dev/null +++ b/modelsdk/canon/typefloor.go @@ -0,0 +1,210 @@ +// SPDX-License-Identifier: Apache-2.0 + +package canon + +import ( + _ "embed" + "encoding/json" + "fmt" + "sort" + "strings" + "sync" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/modelsdk/version" +) + +// UndeclaredTypeError refuses a unit that holds an element whose $Type the +// project's Mendix version does not have yet: introduced in a later version, by +// the Model SDK's own version data (ako/mxcli#1774). Every element is checked, +// the unit's root and everything nested in it. Nil when there is none, and for +// a nil or zero version. +// +// # Why +// +// The property floor (StripUndeclaredProperties) compares property versions; +// the generated version data holds nothing else, so a type newer than the +// project went through it. Such a document does not load: mx check and Studio +// Pro die with `TypeCacheUnknownTypeException: The type cache does not contain +// a type with qualified name …`, naming no document. It shipped four times +// before this check existed: workflows below 11.3 (#1754), page shapes (#1209), +// the parallel split end marker, a 10.21 type written to 9.x and 10.0–10.20 +// (#1163), and user tasks, 10.12 types written to 9.x (#1771). +// +// # Refused, not dropped +// +// Dropping the element would write a different model from the one the +// statement described, and the parent often requires it (an activity in a +// flow, an event on a task). Where an older version stores the same fact in an +// older shape, LowerToProjectShape writes that shape first — this runs after +// it, so it sees only what could not be lowered. +// +// # Deletions are not checked +// +// The data carries the SDK's deletion versions too, but they are not enforced. +// A type the SDK renamed keeps its storage name: the SDK deletes +// Workflows$CallMicroflowTask in 11.9 and introduces CallMicroflowActivity, +// which the codec stores as Workflows$CallMicroflowTask still. And a deleted +// type can still load: the SDK deletes Workflows$TimerBoundaryEvent in 10.20, +// and mxbuild 11.x loads a project holding one (only the runtime lacks the +// class, which is why MDL-WF07 exists). Some deletions are real +// (MessageDefinitions$MessageDefinitionCollection in 11.15, which mxcli refuses +// there on its own), but telling them apart takes a measurement per row. An +// introduction floor cannot misfire on a rename (the new name is simply never +// stored); a deletion floor can. +func UndeclaredTypeError(contents []byte, pv *version.Version) error { + if pv == nil || pv.IsZero() || len(contents) == 0 { + return nil + } + floors := typeFloors() + if floors.newest.Compare(*pv) <= 0 { + return nil // every type the data knows of is declared + } + var refused []string + seen := map[string]bool{} + walkTypes(bson.Raw(contents), func(typ string) { + if seen[typ] { + return + } + seen[typ] = true + if since, ok := floors.introduced[typ]; ok && pv.Compare(since) < 0 { + refused = append(refused, fmt.Sprintf("%s (Mendix %s+)", typ, since)) + } + }) + if len(refused) == 0 { + return nil + } + sort.Strings(refused) + return fmt.Errorf("this Mendix %s project cannot store %s: the type%s did not exist yet in this version, "+ + "and Studio Pro cannot open a document that holds one. Remove what needs it from the statement, or upgrade the project", + pv, strings.Join(refused, "; "), map[bool]string{true: "", false: "s"}[len(refused) == 1]) +} + +// TypeIntroduced returns the version that introduced a storage $Type, by the +// Model SDK's version data. ok is false for a type the data has no +// introduction for: one Mendix had before the SDK's history starts, or one it +// does not know. +func TypeIntroduced(typeName string) (version.Version, bool) { + v, ok := typeFloors().introduced[typeName] + return v, ok +} + +// TypeDeleted returns the version the Model SDK says deleted a storage $Type. +// It is data, not a floor: UndeclaredTypeError does not refuse on it (see +// there for why). +func TypeDeleted(typeName string) (version.Version, bool) { + v, ok := typeFloors().deleted[typeName] + return v, ok +} + +// TypeDeclared reports whether a project of version pv has the storage $Type +// typeName, by the data UndeclaredTypeError refuses on. A type with no data is +// declared, as it is to the check. +func TypeDeclared(typeName string, pv version.Version) bool { + since, ok := TypeIntroduced(typeName) + return !ok || pv.Compare(since) >= 0 +} + +// walkTypes calls fn with the $Type of every element in raw, nested ones +// included. +func walkTypes(raw bson.Raw, fn func(string)) { + elems, err := raw.Elements() + if err != nil { + return + } + for _, e := range elems { + v := e.Value() + switch v.Type { + case bson.TypeString: + if e.Key() == "$Type" { + fn(v.StringValue()) + } + case bson.TypeEmbeddedDocument: + walkTypes(v.Document(), fn) + case bson.TypeArray: + walkTypes(bson.Raw(v.Array()), fn) + } + } +} + +//go:embed mendixmodelsdk_type_versions.json +var typeVersionsJSON []byte + +type typeFloorData struct { + introduced map[string]version.Version + deleted map[string]version.Version + // newest is the latest introduction in the data: a project at or past it + // has every type, and the walk is skipped. + newest version.Version +} + +var ( + typeFloorsOnce sync.Once + typeFloorsData typeFloorData +) + +func typeFloors() typeFloorData { + typeFloorsOnce.Do(func() { + d, err := parseTypeVersions(typeVersionsJSON) + if err != nil { + // The embedded file is checked by TestTypeVersionsParse; a broken + // one must not take every write down with it. + d = typeFloorData{} + } + typeFloorsData = d + }) + return typeFloorsData +} + +// typeVersionsFile is mendixmodelsdk_type_versions.json, generated by +// gen_type_versions.go from the Model SDK. +type typeVersionsFile struct { + Source string `json:"source"` + Types []struct { + Type string `json:"type"` + Introduced string `json:"introduced"` + Deleted string `json:"deleted"` + SDK []string `json:"sdk"` + } `json:"types"` +} + +func parseTypeVersions(data []byte) (typeFloorData, error) { + var f typeVersionsFile + if err := json.Unmarshal(data, &f); err != nil { + return typeFloorData{}, fmt.Errorf("type versions: %w", err) + } + if !strings.HasPrefix(f.Source, "mendixmodelsdk@") || len(f.Types) == 0 { + return typeFloorData{}, fmt.Errorf("type versions: no source or no types") + } + d := typeFloorData{introduced: map[string]version.Version{}, deleted: map[string]version.Version{}} + seen := make(map[string]bool, len(f.Types)) + for _, t := range f.Types { + if !strings.Contains(t.Type, "$") { + return typeFloorData{}, fmt.Errorf("type versions: %q is not a $Type", t.Type) + } + if seen[t.Type] { + return typeFloorData{}, fmt.Errorf("type versions: %s listed twice", t.Type) + } + seen[t.Type] = true + if t.Deleted != "" { + v := version.Parse(t.Deleted) + if v.IsZero() { + return typeFloorData{}, fmt.Errorf("type versions: %s deleted %q", t.Type, t.Deleted) + } + d.deleted[t.Type] = v + } + if t.Introduced == "" { + continue + } + v := version.Parse(t.Introduced) + if v.IsZero() { + return typeFloorData{}, fmt.Errorf("type versions: %s introduced %q", t.Type, t.Introduced) + } + d.introduced[t.Type] = v + if d.newest.Compare(v) < 0 { + d.newest = v + } + } + return d, nil +} diff --git a/modelsdk/canon/typefloor_test.go b/modelsdk/canon/typefloor_test.go new file mode 100644 index 0000000000..4d36f88df8 --- /dev/null +++ b/modelsdk/canon/typefloor_test.go @@ -0,0 +1,140 @@ +// SPDX-License-Identifier: Apache-2.0 + +package canon + +import ( + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/modelsdk/version" +) + +// The embedded data parses, and a broken file is not silently empty: a parse +// error would disable the type floor for every write (typeFloors falls back to +// no data rather than failing writes). +func TestTypeVersionsParse(t *testing.T) { + d, err := parseTypeVersions(typeVersionsJSON) + if err != nil { + t.Fatal(err) + } + if len(d.introduced) < 500 { + t.Errorf("only %d type introductions; the SDK 4.117.0 extraction has over 600", len(d.introduced)) + } + for _, bad := range []string{ + `{"source":"mendixmodelsdk@1","types":[{"type":"NoDollar","introduced":"1.0.0"}]}`, + `{"source":"mendixmodelsdk@1","types":[{"type":"A$B","introduced":"x"}]}`, + `{"source":"mendixmodelsdk@1","types":[{"type":"A$B"},{"type":"A$B"}]}`, + `{"source":"elsewhere","types":[{"type":"A$B","introduced":"1.0.0"}]}`, + } { + if _, err := parseTypeVersions([]byte(bad)); err == nil { + t.Errorf("parsed %s", bad) + } + } +} + +// The versions the lowerings hold as constants are the data's: each was read +// from the same SDK by hand (ako/mxcli#1163, #1771) before the data existed. +func TestTypeFloor_AgreesWithTheLowerings(t *testing.T) { + for typ, want := range map[string]version.Version{ + "Workflows$EndOfParallelSplitPathActivity": endOfParallelSplitPathSince, + "Workflows$SingleUserTaskActivity": userTaskActivitySince, + "Workflows$MultiUserTaskActivity": userTaskActivitySince, + "Workflows$EmptyUserSource": emptyUserSourceSince, + "Workflows$MajorityCompletionCriteria": completionCriteriaSince["Workflows$MajorityCompletionCriteria"], + "Workflows$ThresholdCompletionCriteria": completionCriteriaSince["Workflows$ThresholdCompletionCriteria"], + "Workflows$MicroflowCompletionCriteria": completionCriteriaSince["Workflows$MicroflowCompletionCriteria"], + } { + got, ok := TypeIntroduced(typ) + if !ok || got.Compare(want) != 0 { + t.Errorf("%s: data says %v (%v), the lowering %v", typ, got, ok, want) + } + } +} + +// The rows #1774 names, as the SDK has them. +func TestTypeFloor_IssueRows(t *testing.T) { + for typ, want := range map[string]string{ + "Workflows$WaitForTimerActivity": "10.6.0", + "Workflows$WaitForNotificationActivity": "10.0.0", + "Workflows$WorkflowDefinitionNameSelection": "10.0.0", + "Workflows$WorkflowDefinitionObjectSelection": "10.0.0", + "Workflows$TimerBoundaryEvent": "10.14.0", + "Workflows$InterruptingTimerBoundaryEvent": "10.20.0", + "Workflows$NonInterruptingTimerBoundaryEvent": "10.20.0", + "Workflows$NotificationActivity": "11.11.0", + "Workflows$StartWorkflowActivity": "11.1.0", + "Forms$FormCallArgument": "", // Studio Pro 9.1.1 and 9.4.0 store it; only an alias of the 9.7 PageParameterMapping reaches it + "Forms$ToggleDesignPropertyValue": "10.2.0", + "Microflows$CreateChangeAction": "", // CreateObjectAction's storage name: always there + "Workflows$CallMicroflowTask": "9.0.2", + "Workflows$NonInterruptingTimerEventSubProcessStartActivity": "11.13.0", + } { + got, ok := TypeIntroduced(typ) + switch { + case want == "" && ok: + t.Errorf("%s: introduced %v, want no floor", typ, got) + case want != "" && (!ok || got.String() != want): + t.Errorf("%s: introduced %v (%v), want %s", typ, got, ok, want) + } + } +} + +// A type the SDK renamed keeps its storage name: the SDK deletes +// CallMicroflowTask in 11.9 and introduces CallMicroflowActivity, which the +// generated codec still stores as Workflows$CallMicroflowTask. The extraction +// keeps the stored name valid on both sides, and deletions are not enforced. +func TestTypeFloor_RenamedTypeKeepsItsStorageName(t *testing.T) { + for _, v := range []string{"9.24.30", "10.24.24", "11.9.0", "11.15.0"} { + if !TypeDeclared("Workflows$CallMicroflowTask", version.Parse(v)) { + t.Errorf("%s: Workflows$CallMicroflowTask not declared", v) + } + } + if d, ok := TypeDeleted("Workflows$CallMicroflowTask"); ok { + t.Errorf("Workflows$CallMicroflowTask carries a deletion (%v) though CallMicroflowActivity is stored under it", d) + } + doc, _ := bson.Marshal(bson.D{{Key: "$Type", Value: "Workflows$UserTask"}}) + if err := UndeclaredTypeError(doc, ver("11.13.0")); err != nil { + t.Errorf("a deletion was enforced: %v", err) + } +} + +func ver(s string) *version.Version { v := version.Parse(s); return &v } + +// The walk sees elements in documents and in lists, at any depth, and names +// each refused type once. +func TestUndeclaredTypeError_WalksTheWholeUnit(t *testing.T) { + wait := bson.D{{Key: "$Type", Value: "Workflows$WaitForTimerActivity"}} + doc, _ := bson.Marshal(bson.D{ + {Key: "$Type", Value: "Workflows$Workflow"}, + {Key: "Flow", Value: bson.D{{Key: "$Type", Value: "Workflows$Flow"}, + {Key: "Activities", Value: bson.A{int32(3), wait, bson.D{ + {Key: "$Type", Value: "Workflows$ParallelSplitActivity"}, + {Key: "Outcomes", Value: bson.A{int32(3), bson.D{{Key: "$Type", Value: "Workflows$ParallelSplitOutcome"}, + {Key: "Flow", Value: bson.D{{Key: "$Type", Value: "Workflows$Flow"}, + {Key: "Activities", Value: bson.A{int32(3), wait, + bson.D{{Key: "$Type", Value: "Workflows$WaitForNotificationActivity"}}}}}}}}}}}}}}, + }) + err := UndeclaredTypeError(doc, ver("9.24.30")) + if err == nil { + t.Fatal("not refused") + } + msg := err.Error() + for _, want := range []string{"this Mendix 9.24.30 project cannot store", + "Workflows$WaitForNotificationActivity (Mendix 10.0.0+); Workflows$WaitForTimerActivity (Mendix 10.6.0+)", + "the types did not exist yet"} { + if !strings.Contains(msg, want) { + t.Errorf("%q does not say %q", msg, want) + } + } + if strings.Count(msg, "WaitForTimerActivity") != 1 { + t.Errorf("a type held twice is named twice: %s", msg) + } + if err := UndeclaredTypeError(doc, ver("10.6.0")); err != nil { + t.Errorf("control 10.6.0: %v", err) + } + if err := UndeclaredTypeError(doc, nil); err != nil { + t.Errorf("no version: %v", err) + } +} diff --git a/modelsdk/canon/versionshape.go b/modelsdk/canon/versionshape.go index fbf5097cd3..ecaadd7e5c 100644 --- a/modelsdk/canon/versionshape.go +++ b/modelsdk/canon/versionshape.go @@ -58,7 +58,8 @@ import ( // Before, a path is its activities and nothing more (Studio Pro 9.4.0, the // Marketplace's "Example project workflow extension" 2.0.0), and a project // holding the marker does not load (TypeCacheUnknownTypeException). The -// floor checks property versions, never a type's own. +// property floor never saw it; the type floor (UndeclaredTypeError, +// ako/mxcli#1774) refuses it, so it is lowered first. // - Workflows$ user task targeting: UserTargeting from 11.2.0, declared on // the abstract UserTaskActivity; before, UserSource (ako/mxcli#1771). See // lowerUserTargeting. @@ -124,9 +125,9 @@ type shapeLowering struct { // endOfParallelSplitPathSince is the version that introduced the type // Workflows$EndOfParallelSplitPathActivity (mendixmodelsdk 4.117.0: -// EndOfParallelSplitPathActivity.versionInfo introduced "10.21.0"). It is -// written down here because the generated version data holds property -// versions only, never a type's own. +// EndOfParallelSplitPathActivity.versionInfo introduced "10.21.0"), read by +// hand before the type data existed; TestTypeFloor_AgreesWithTheLowerings +// keeps it equal to mendixmodelsdk_type_versions.json. var endOfParallelSplitPathSince = version.Version{Major: 10, Minor: 21} var placeholderWidgets = map[string]bool{"Forms$TextBox": true, "Forms$TextArea": true, "Forms$DatePicker": true} diff --git a/modelsdk/canon/versionshape_usertask.go b/modelsdk/canon/versionshape_usertask.go index d436e4e4b1..a6583f07a1 100644 --- a/modelsdk/canon/versionshape_usertask.go +++ b/modelsdk/canon/versionshape_usertask.go @@ -41,8 +41,9 @@ var emptyUserSourceSince = version.Version{Major: 9, Minor: 10} // userTaskActivitySince is the version that introduced the types // Workflows$SingleUserTaskActivity and MultiUserTaskActivity, and deleted -// Workflows$UserTask (mendixmodelsdk 4.117.0). Written down here because the -// generated version data holds property versions only, never a type's own. +// Workflows$UserTask (mendixmodelsdk 4.117.0), read by hand before the type +// data existed; TestTypeFloor_AgreesWithTheLowerings keeps it equal to +// mendixmodelsdk_type_versions.json. var userTaskActivitySince = version.Version{Major: 10, Minor: 12} // completionCriteriaSince holds the multi-user task rules introduced after the diff --git a/modelsdk/mpr/contents_hash.go b/modelsdk/mpr/contents_hash.go index 6058b01c95..c6fa4bcae4 100644 --- a/modelsdk/mpr/contents_hash.go +++ b/modelsdk/mpr/contents_hash.go @@ -5,15 +5,16 @@ package mpr import ( "crypto/sha256" "encoding/base64" - "errors" "fmt" "os" "path/filepath" "sort" "strings" + + "go.mongodb.org/mongo-driver/v2/bson" ) -// ContentsHash verification (ako/mxcli#972). +// ContentsHash verification (ako/mxcli#972, #1159). // // In MPR v2 every Unit row carries ContentsHash = base64(SHA-256(.mxunit)). // The writer keeps the two in step (WriteTransaction.WriteUnit, updateUnit), but @@ -21,28 +22,33 @@ import ( // `git checkout` leaves the row describing bytes that are no longer on disk, and // `mx check` does not notice. VerifyContentsHashes finds the drift and // RepairContentsHashes rewrites the index from the files. - -// ErrNotMPRv2 is returned for an MPR v1 project, which has no ContentsHash index -// to drift: the unit contents live in the Unit table itself. -var ErrNotMPRv2 = errors.New("not an MPR v2 project: ContentsHash only indexes mprcontents files") - -// ContentsHashIssue is one unit whose stored hash does not describe its file. +// +// MPR v1 keeps the contents in the row, and Studio Pro fills ContentsHash there +// too, with the same formula over the Contents column (all 308 units of a fresh +// 9.24.30 app). Nothing outside the row can make the two drift, but a writer +// that forgets the column does: every mxcli before #1159 replaced Contents and +// left the hash describing the old bytes, and stored '' for an inserted unit. + +// ContentsHashIssue is one unit whose stored hash does not describe its file +// (v2) or its Contents column (v1). type ContentsHashIssue struct { UnitID string // Mendix UUID form, as everywhere else in this package - Path string // the .mxunit the row points at + Path string // the .mxunit the row points at ("" on MPR v1, which has none) Stored string // ContentsHash in the .mpr ("" when the row has none) - Actual string // hash of the file on disk ("" when the file is missing) + Actual string // hash of the file on disk ("" when the file is missing), or of Contents on v1 + Unit string // "<$Type> " of the unit on MPR v1, where no file names it blob []byte // the row's UnitID, for the repair's UPDATE } // ContentsHashReport is the result of VerifyContentsHashes. type ContentsHashReport struct { + Version MPRVersion // format checked: v1 compares against Contents, v2 against files Units int // rows in the Unit table - Files int // .mxunit files under mprcontents + Files int // .mxunit files under mprcontents (0 on v1) Mismatches []ContentsHashIssue // file present, hash differs - MissingFiles []ContentsHashIssue // row present, file absent - OrphanFiles []string // file present, no row + MissingFiles []ContentsHashIssue // row present, file absent (v2 only) + OrphanFiles []string // file present, no row (v2 only) } // Clean reports whether the index and the files agree completely. @@ -63,11 +69,15 @@ func unitFilePath(contentsDir string, unitIDBlob []byte) string { } // VerifyContentsHashes compares every Unit.ContentsHash with the SHA-256 of the -// .mxunit file it indexes, and lists .mxunit files no row indexes. Read-only. -// Returns ErrNotMPRv2 for an MPR v1 project. +// .mxunit file it indexes, and lists .mxunit files no row indexes. On an MPR +// v1 project it compares each row's hash with its own Contents column instead. +// Read-only. func (r *Reader) VerifyContentsHashes() (*ContentsHashReport, error) { - if r.version != MPRVersionV2 || r.contentsDir == "" { - return nil, ErrNotMPRv2 + if r.version != MPRVersionV2 { + return r.verifyContentsHashesV1() + } + if r.contentsDir == "" { + return nil, fmt.Errorf("MPR v2 project %s has no mprcontents folder", r.path) } rows, err := r.db.Query(`SELECT UnitID, COALESCE(ContentsHash, '') FROM Unit`) if err != nil { @@ -91,7 +101,7 @@ func (r *Reader) VerifyContentsHashes() (*ContentsHashReport, error) { return nil, fmt.Errorf("query unit hashes: %w", err) } - rep := &ContentsHashReport{Units: len(units)} + rep := &ContentsHashReport{Version: MPRVersionV2, Units: len(units)} indexed := make(map[string]bool, len(units)) for _, u := range units { if len(u.blob) != 16 { @@ -131,15 +141,63 @@ func (r *Reader) VerifyContentsHashes() (*ContentsHashReport, error) { return rep, nil } +// verifyContentsHashesV1 is VerifyContentsHashes for MPR v1: each row's hash +// against its own Contents. There are no files, so nothing can be missing or +// orphaned. Reads every blob, so it is for an explicit check, not every command. +func (r *Reader) verifyContentsHashesV1() (*ContentsHashReport, error) { + rows, err := r.db.Query(`SELECT UnitID, COALESCE(ContentsHash, ''), Contents FROM Unit`) + if err != nil { + return nil, fmt.Errorf("query unit hashes: %w", err) + } + defer rows.Close() + rep := &ContentsHashReport{Version: MPRVersionV1} + for rows.Next() { + var blob, contents []byte + var hash string + if err := rows.Scan(&blob, &hash, &contents); err != nil { + return nil, fmt.Errorf("scan unit row: %w", err) + } + rep.Units++ + if len(blob) != 16 { + continue + } + if actual := contentsHashOf(contents); actual != hash { + rep.Mismatches = append(rep.Mismatches, ContentsHashIssue{ + UnitID: blobToUUID(blob), Stored: hash, Actual: actual, + Unit: unitTypeAndName(contents), blob: blob, + }) + } + } + if err := rows.Err(); err != nil { + return nil, fmt.Errorf("query unit hashes: %w", err) + } + sort.Slice(rep.Mismatches, func(i, j int) bool { return rep.Mismatches[i].UnitID < rep.Mismatches[j].UnitID }) + return rep, nil +} + +// unitTypeAndName is "<$Type> " of a unit document, or as much of it as +// the document has. +func unitTypeAndName(contents []byte) string { + typ := getTypeFromContents(contents) + if typ == "" { + return "" + } + if name, ok := bson.Raw(contents).Lookup("Name").StringValueOK(); ok && name != "" { + return typ + " " + name + } + return typ +} + func sortIssues(issues []ContentsHashIssue) { sort.Slice(issues, func(i, j int) bool { return issues[i].Path < issues[j].Path }) } // RepairContentsHashes rewrites every mismatched ContentsHash from the file on -// disk in one SQLite transaction, and bumps _Transaction.LastTransactionID in -// that transaction as every other write does (Studio Pro reads it to detect an -// external change). It takes the writer's Studio Pro open-project guard: a -// running Studio Pro would overwrite the index on its next save. +// disk (on MPR v1, from the row's Contents) in one SQLite transaction, and +// bumps _Transaction.LastTransactionID in that transaction as every other write +// does (Studio Pro reads it to detect an external change; v1 has no such +// table). It takes the writer's Studio Pro open-project guard: a running +// Studio Pro would overwrite the index on its next save. // // Missing files and orphan files are reported, not repaired: neither has a // correct hash to write (`mxcli diag --check-units --fix` removes orphans). diff --git a/modelsdk/mpr/contents_hash_test.go b/modelsdk/mpr/contents_hash_test.go index 2af0dadf34..6972a561fe 100644 --- a/modelsdk/mpr/contents_hash_test.go +++ b/modelsdk/mpr/contents_hash_test.go @@ -119,9 +119,95 @@ func TestRepairContentsHashesRefusedWhileStudioProOpen(t *testing.T) { } } -func TestVerifyContentsHashesV1NotApplicable(t *testing.T) { - r := &Reader{version: MPRVersionV1} - if _, err := r.VerifyContentsHashes(); !errors.Is(err, ErrNotMPRv2) { - t.Fatalf("want ErrNotMPRv2, got %v", err) +// ako/mxcli#1159: an MPR v1 project has no files, but it does have the +// column, and Studio Pro fills it on every unit. A writer that forgot it (every +// mxcli before the fix) left it describing the previous contents, or '' on an +// inserted unit; the comparison is against the Contents column. + +// staleV1Unit replaces one Projects$Folder's Contents the way an mxcli before +// #1159 did, leaving its ContentsHash alone. Returns the unit ID and the new bytes. +func staleV1Unit(t *testing.T, w *Writer) (string, []byte) { + t.Helper() + id, stored := v1UnitOfType(t, w, "Projects$Folder") + changed := withFields(t, stored, map[string]any{"Name": "WrittenByOldMxcli"}) + if _, err := w.reader.db.Exec(`UPDATE Unit SET Contents = ? WHERE UnitID = ?`, changed, uuidToBlob(id)); err != nil { + t.Fatal(err) + } + return id, changed +} + +func TestVerifyContentsHashesV1CleanControl(t *testing.T) { + w := openV1FixtureWriter(t) + rep, err := w.ConcreteReader().VerifyContentsHashes() + if err != nil { + t.Fatal(err) + } + if !rep.Clean() || rep.Units != 20 || rep.Files != 0 || rep.Version != MPRVersionV1 { + t.Fatalf("Studio Pro-authored v1 fixture not clean: %+v", rep) } } + +func TestVerifyContentsHashesV1DetectsStaleHash(t *testing.T) { + w := openV1FixtureWriter(t) + before := storedHash(t, w, v1FolderID(t, w)) + id, changed := staleV1Unit(t, w) + + rep, err := w.ConcreteReader().VerifyContentsHashes() + if err != nil { + t.Fatal(err) + } + if len(rep.Mismatches) != 1 || len(rep.MissingFiles) != 0 || len(rep.OrphanFiles) != 0 { + t.Fatalf("want exactly 1 mismatch, got %+v", rep) + } + m := rep.Mismatches[0] + if m.UnitID != id || m.Stored != before || m.Actual != hashOf(changed) || m.Path != "" { + t.Fatalf("wrong mismatch: %+v", m) + } + if m.Unit != "Projects$Folder WrittenByOldMxcli" { + t.Fatalf("mismatch does not name its unit: %q", m.Unit) + } +} + +func TestVerifyContentsHashesV1ReportsEmptyHash(t *testing.T) { + w := openV1FixtureWriter(t) + id := v1FolderID(t, w) + if _, err := w.reader.db.Exec(`UPDATE Unit SET ContentsHash = '' WHERE UnitID = ?`, uuidToBlob(id)); err != nil { + t.Fatal(err) + } + rep, err := w.ConcreteReader().VerifyContentsHashes() + if err != nil { + t.Fatal(err) + } + if len(rep.Mismatches) != 1 || rep.Mismatches[0].UnitID != id || rep.Mismatches[0].Stored != "" { + t.Fatalf("an inserted unit's empty hash is not reported: %+v", rep) + } +} + +func TestRepairContentsHashesV1RewritesFromContents(t *testing.T) { + w := openV1FixtureWriter(t) + id, changed := staleV1Unit(t, w) + + rep, n, err := w.RepairContentsHashes() + if err != nil { + t.Fatal(err) + } + if n != 1 || len(rep.Mismatches) != 1 { + t.Fatalf("want 1 row repaired, got %d (%+v)", n, rep) + } + if got := storedHash(t, w, id); got != hashOf(changed) { + t.Fatalf("ContentsHash = %q, want %q", got, hashOf(changed)) + } + if stale := staleV1Hashes(t, w); len(stale) != 0 { + t.Fatalf("still stale after repair: %v", stale) + } + // The contents are the source of truth and must be left alone. + if got, _ := w.reader.GetRawUnitBytes(id); string(got) != string(changed) { + t.Fatal("repair modified the unit's contents") + } +} + +func v1FolderID(t *testing.T, w *Writer) string { + t.Helper() + id, _ := v1UnitOfType(t, w, "Projects$Folder") + return id +} diff --git a/modelsdk/mpr/property_set_census.go b/modelsdk/mpr/property_set_census.go new file mode 100644 index 0000000000..4518f321cc --- /dev/null +++ b/modelsdk/mpr/property_set_census.go @@ -0,0 +1,211 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "sort" + "strings" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +// PropertySetCensus counts, across a whole project, which property names the +// elements of each $Type carry, and reports the names some elements of a type +// carry and others do not (ako/mxcli#1505). +// +// # Why +// +// The property-set repair fills only what the measured table +// (canon.CompletePropertySets) holds for the project's Mendix version. A key +// the table has no value for on that version — a version nobody measured, or a +// writer gap no measurement exercised — is invisible to it, so on Mendix +// 10.24.18 `fix property-sets` reported every unit complete while `mx diff` +// threw on a nanoflow without ExportLevel. +// +// The project is its own reference for that question, if not for the value: +// Studio Pro serializes the full property set of an element's type on every +// element of it (on the Studio Pro-authored fixtures pedapp 11.13.0, +// expr-checker 11.6.6, testapp-views 11.14.0 and a blank 10.24.18 app, every +// one of 115–251 types is uniform). So an element of a type without a name the +// others carry — or with one they lack — is a pair of name sets Mendix's merge +// engine refuses to compare. +// +// # What it does not do +// +// It reports; it never fills. Which side is right, and what value Mendix gives +// an absent key on this version, is a measurement (`mx convert` on that +// version), and a value taken from another version or from another element is +// a guess that changes the model when it is wrong. Count the units as the +// repair leaves them, so what the table fills is not reported. +type PropertySetCensus struct { + types map[string]*typeCensus +} + +type typeCensus struct { + elements int + sets map[string]*keySetCensus // by the sorted, NUL-joined key set +} + +type keySetCensus struct { + keys map[string]bool + elements int + units []PropertySetRepair // the first few units holding such an element +} + +// censusExamples bounds the units remembered per key set. +const censusExamples = 3 + +// NewPropertySetCensus returns an empty census. +func NewPropertySetCensus() *PropertySetCensus { + return &PropertySetCensus{types: map[string]*typeCensus{}} +} + +// Count adds every element of one unit, as the repair (StoredPropertySetGaps or +// CompleteStoredPropertySets) left it. A unit that could not be normalized adds +// nothing; the repair reports it. +func (c *PropertySetCensus) Count(rep PropertySetRepair) { + if rep.normalized == nil { + return + } + var doc bson.D + if err := bson.Unmarshal(rep.normalized, &doc); err != nil { + return + } + label := rep + label.normalized = nil + var walk func(v any) + walk = func(v any) { + switch x := v.(type) { + case bson.D: + typ := "" + for _, e := range x { + if e.Key == "$Type" { + typ, _ = e.Value.(string) + } + walk(e.Value) + } + if typ != "" { + c.add(typ, x, label) + } + case bson.A: + for _, e := range x { + walk(e) + } + } + } + walk(doc) +} + +func (c *PropertySetCensus) add(typ string, d bson.D, unit PropertySetRepair) { + names := make([]string, 0, len(d)) + for _, e := range d { + names = append(names, e.Key) + } + sort.Strings(names) + sig := strings.Join(names, "\x00") + + tc := c.types[typ] + if tc == nil { + tc = &typeCensus{sets: map[string]*keySetCensus{}} + c.types[typ] = tc + } + tc.elements++ + ks := tc.sets[sig] + if ks == nil { + ks = &keySetCensus{keys: make(map[string]bool, len(names))} + for _, n := range names { + ks.keys[n] = true + } + tc.sets[sig] = ks + } + ks.elements++ + if len(ks.units) < censusExamples { + for _, u := range ks.units { + if u.UnitID == unit.UnitID { + return + } + } + ks.units = append(ks.units, unit) + } +} + +// MixedPropertyName is one property name that some elements of a $Type carry +// and others do not. +type MixedPropertyName struct { + Type, Key string + Elements int // elements of Type in the project + With int // of which carry Key + // Examples are up to three units holding an element on the smaller side: + // without the key when most elements carry it, with it otherwise. + Examples []PropertySetRepair +} + +// Without reports whether Examples are units lacking the key (most elements +// carry it) rather than units holding it. +func (m MixedPropertyName) Without() bool { return m.With*2 >= m.Elements } + +// Mixed returns every property name the elements of one type disagree on, +// sorted by type and key. +func (c *PropertySetCensus) Mixed() []MixedPropertyName { + var out []MixedPropertyName + for typ, tc := range c.types { + if len(tc.sets) < 2 { + continue + } + with := map[string]int{} + for _, ks := range tc.sets { + for k := range ks.keys { + with[k] += ks.elements + } + } + for k, n := range with { + if n == tc.elements { + continue + } + m := MixedPropertyName{Type: typ, Key: k, Elements: tc.elements, With: n} + wantKey := !m.Without() + for _, ks := range sortedSets(tc.sets) { + if ks.keys[k] != wantKey { + continue + } + for _, u := range ks.units { + if len(m.Examples) < censusExamples && !hasUnit(m.Examples, u.UnitID) { + m.Examples = append(m.Examples, u) + } + } + } + out = append(out, m) + } + } + sort.Slice(out, func(i, j int) bool { + if out[i].Type != out[j].Type { + return out[i].Type < out[j].Type + } + return out[i].Key < out[j].Key + }) + return out +} + +// sortedSets orders a type's key sets by signature, so the examples a report +// names do not depend on map order. +func sortedSets(sets map[string]*keySetCensus) []*keySetCensus { + sigs := make([]string, 0, len(sets)) + for s := range sets { + sigs = append(sigs, s) + } + sort.Strings(sigs) + out := make([]*keySetCensus, len(sigs)) + for i, s := range sigs { + out[i] = sets[s] + } + return out +} + +func hasUnit(us []PropertySetRepair, id string) bool { + for _, u := range us { + if u.UnitID == id { + return true + } + } + return false +} diff --git a/modelsdk/mpr/property_set_census_test.go b/modelsdk/mpr/property_set_census_test.go new file mode 100644 index 0000000000..ff05481cbe --- /dev/null +++ b/modelsdk/mpr/property_set_census_test.go @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +// censusUnit is a unit holding one nanoflow-like element with the given keys, +// as the repair leaves it. +func censusUnit(t *testing.T, id, name string, keys ...string) PropertySetRepair { + t.Helper() + d := bson.D{{Key: "$ID", Value: bson.Binary{Subtype: 0, Data: uuidToBlob(id)}}, {Key: "$Type", Value: "Microflows$Nanoflow"}} + for _, k := range keys { + d = append(d, bson.E{Key: k, Value: ""}) + } + b, err := bson.Marshal(d) + if err != nil { + t.Fatal(err) + } + return PropertySetRepair{UnitID: id, Type: "Microflows$Nanoflow", Name: name, normalized: b} +} + +// ako/mxcli#1505: one nanoflow without the ExportLevel every other nanoflow of +// the project carries is reported, with that nanoflow as the example; a key the +// minority carries names the minority instead. +func TestPropertySetCensus_NamesTheElementsThatDiffer(t *testing.T) { + c := NewPropertySetCensus() + c.Count(censusUnit(t, "11111111-1111-4111-8111-111111111111", "A", "Name", "ExportLevel")) + c.Count(censusUnit(t, "22222222-2222-4222-8222-222222222222", "B", "Name", "ExportLevel")) + c.Count(censusUnit(t, "33333333-3333-4333-8333-333333333333", "ACT_Login", "Name", "Extra")) + + got := c.Mixed() + if len(got) != 2 { + t.Fatalf("Mixed() = %+v, want ExportLevel and Extra", got) + } + el, extra := got[0], got[1] + if el.Key != "ExportLevel" || el.With != 2 || el.Elements != 3 || !el.Without() || + len(el.Examples) != 1 || el.Examples[0].Name != "ACT_Login" { + t.Errorf("ExportLevel = %+v, want on 2 of 3, without it: ACT_Login", el) + } + if extra.Key != "Extra" || extra.With != 1 || extra.Without() || + len(extra.Examples) != 1 || extra.Examples[0].Name != "ACT_Login" { + t.Errorf("Extra = %+v, want on 1 of 3, only on: ACT_Login", extra) + } +} + +// The control: elements of a type that all carry the same names report +// nothing, and a unit the repair could not normalize is not counted. +func TestPropertySetCensus_UniformTypeIsClean(t *testing.T) { + c := NewPropertySetCensus() + c.Count(censusUnit(t, "11111111-1111-4111-8111-111111111111", "A", "Name", "ExportLevel")) + c.Count(censusUnit(t, "22222222-2222-4222-8222-222222222222", "B", "Name", "ExportLevel")) + c.Count(PropertySetRepair{UnitID: "33333333-3333-4333-8333-333333333333"}) + if got := c.Mixed(); len(got) != 0 { + t.Fatalf("Mixed() = %+v on a uniform type", got) + } +} diff --git a/modelsdk/mpr/reader.go b/modelsdk/mpr/reader.go index 6456f7faab..57fc3e754a 100644 --- a/modelsdk/mpr/reader.go +++ b/modelsdk/mpr/reader.go @@ -47,7 +47,7 @@ type Reader struct { // Unit table held at the last rebuild — what a persisted index keeps. // v2TypesMu guards these and the unit cache rebuild. v2TypesMu sync.Mutex - v2Types map[string]string + v2Types map[string]unitIndexEntry v2TypesLoaded bool v2TypesDirty bool v2Hashes map[string]struct{} @@ -91,6 +91,10 @@ type cachedUnit struct { ContainerID string ContainmentName string Type string + // Name and Excluded are the unit's top-level properties, known without + // opening its file once the index has seen its bytes (reader_units_v2.go). + Name string + Excluded bool } // OpenOptions configures how the MPR file is opened. @@ -408,6 +412,19 @@ func (r *Reader) ClearAllOverlays() { // GetRawUnitBytes returns the raw BSON bytes for a unit identified by its UUID string. func (r *Reader) GetRawUnitBytes(unitID string) ([]byte, error) { + return r.rawUnitBytes(r.db, unitID) +} + +// rowQuerier is what reading one unit's v1 contents needs: *sql.DB and *sql.Tx +// both satisfy it. +type rowQuerier interface { + QueryRow(query string, args ...any) *sql.Row +} + +// rawUnitBytes is GetRawUnitBytes with the v1 query run on q. A caller holding a +// transaction must pass it: the pool has one connection (OpenWithOptions), the +// transaction owns it, and a read through r.db would wait for it forever. +func (r *Reader) rawUnitBytes(q rowQuerier, unitID string) ([]byte, error) { // Fast path: return buffered bytes injected by BufferedUnitStore if present. if len(r.overlay) > 0 { if data, ok := r.overlay[unitID]; ok { @@ -433,7 +450,7 @@ func (r *Reader) GetRawUnitBytes(unitID string) ([]byte, error) { return nil, fmt.Errorf("invalid unit ID: %s", unitID) } var contents []byte - err := r.db.QueryRow("SELECT Contents FROM Unit WHERE UnitID = ?", blob).Scan(&contents) + err := q.QueryRow("SELECT Contents FROM Unit WHERE UnitID = ?", blob).Scan(&contents) if err != nil { return nil, fmt.Errorf("unit not found: %s", unitID) } diff --git a/modelsdk/mpr/reader_unit_by_id_test.go b/modelsdk/mpr/reader_unit_by_id_test.go new file mode 100644 index 0000000000..926bb9a72b --- /dev/null +++ b/modelsdk/mpr/reader_unit_by_id_test.go @@ -0,0 +1,63 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "bytes" + "testing" +) + +// ako/mxcli#1789: UnitByID reads one unit where a lookup by ID used to list +// every unit of the type. It must find what the listing finds, on both storage +// formats, and nothing for an ID the project does not hold. +func TestUnitByID_AgreesWithTheListing(t *testing.T) { + v2, _ := seedProject(t, 6) + for _, fixture := range []struct { + name string + r *Reader + }{ + {"v1", newTypedListingFixture(t, []string{"Microflows$Microflow", "Forms$Page", "Projects$Folder", "Microflows$Microflow"}).r}, + {"v2", v2.reader()}, + } { + t.Run(fixture.name, func(t *testing.T) { + all, err := fixture.r.listUnitsByType("") + if err != nil { + t.Fatal(err) + } + if len(all) < 4 { + t.Fatalf("control: the fixture lists %d units", len(all)) + } + for _, want := range all { + got, ok, err := fixture.r.UnitByID(want.ID) + if err != nil || !ok { + t.Fatalf("UnitByID(%s) = ok %v, err %v", want.ID, ok, err) + } + if got.ID != want.ID || got.ContainerID != want.ContainerID || got.Type != want.Type || !bytes.Equal(got.Contents, want.Contents) { + t.Errorf("UnitByID(%s) = {%s %s %s %d bytes}, the listing has {%s %s %s %d bytes}", + want.ID, got.ID, got.ContainerID, got.Type, len(got.Contents), + want.ID, want.ContainerID, want.Type, len(want.Contents)) + } + } + if _, ok, err := fixture.r.UnitByID("99999999-9999-9999-9999-999999999999"); ok || err != nil { + t.Errorf("UnitByID(unknown) = ok %v, err %v; want not found", ok, err) + } + }) + } +} + +// On v2 the lookup opens the one file. +func TestUnitByID_V2OpensOneFile(t *testing.T) { + p, _ := seedProject(t, 40) + r := p.reader() + units, err := r.listUnitTypes() // the unit index, as any earlier call builds it + if err != nil { + t.Fatal(err) + } + before := r.UnitFileReads() + if _, ok, err := r.UnitByID(units[len(units)-1].ID); !ok || err != nil { + t.Fatalf("UnitByID = ok %v, err %v", ok, err) + } + if reads := r.UnitFileReads() - before; reads != 1 { + t.Errorf("UnitByID opened %d unit files of %d; it needs one", reads, len(units)) + } +} diff --git a/modelsdk/mpr/reader_unit_headers_test.go b/modelsdk/mpr/reader_unit_headers_test.go new file mode 100644 index 0000000000..b00fae643f --- /dev/null +++ b/modelsdk/mpr/reader_unit_headers_test.go @@ -0,0 +1,168 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "os" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +// ako/mxcli#1791: naming every folder (the container hierarchy, built by +// nearly every command) or every microflow (a describe's name map) opened each +// of their files on every call, because a unit's Name is only in its BSON. The +// unit index records Name and Excluded beside the $Type, keyed by the same +// ContentsHash, so a fresh process names the units it has seen without +// opening them. + +// headerDoc is a unit with a Name and, when excluded, Excluded = true. +func headerDoc(t *testing.T, typ, name string, excluded bool) []byte { + t.Helper() + d := bson.D{ + {Key: "$ID", Value: bson.Binary{Subtype: 0x00, Data: uuidToBlob("11111111-1111-1111-1111-111111111111")}}, + {Key: "$Type", Value: typ}, + {Key: "Name", Value: name}, + } + if excluded { + d = append(d, bson.E{Key: "Excluded", Value: true}) + } + b, err := bson.Marshal(d) + if err != nil { + t.Fatalf("marshal unit: %v", err) + } + return b +} + +func TestListUnitHeaders_SecondProcessOpensNoFile(t *testing.T) { + const n = 40 + p, _ := seedProject(t, n) + for i := n + 1; i <= n+5; i++ { + p.add(t, i, "Projects$Folder") + } + + first := p.reader() + want, err := first.ListUnitHeaders("Microflows$Microflow") + if err != nil { + t.Fatal(err) + } + if len(want) != n { + t.Fatalf("first reader listed %d microflow headers, want %d", len(want), n) + } + if got := first.UnitFileReads(); got < n { + t.Fatalf("control: the cold listing read %d files, want >= %d (the fixture does not exercise the index)", got, n) + } + + second := p.reader() + got, err := second.ListUnitHeaders("Microflows$Microflow") + if err != nil { + t.Fatal(err) + } + folders, err := second.ListFolders() + if err != nil { + t.Fatal(err) + } + if reads := second.UnitFileReads(); reads != 0 { + t.Errorf("a fresh process naming %d microflows and %d folders opened %d unit files, want 0", len(got), len(folders), reads) + } + if len(got) != len(want) { + t.Fatalf("second reader listed %d headers, the first %d", len(got), len(want)) + } + for i := range want { + if got[i] != want[i] { + t.Errorf("header %d from the index = %+v, from the file = %+v", i, got[i], want[i]) + } + } + for _, f := range folders { + if f.Name == "" { + t.Errorf("folder %s has no name from the index", f.ID) + } + } +} + +// What the index answers is what the files say, on both storage formats, +// including a name no line-based format could hold unescaped and the +// Excluded flag. +func TestListUnitHeaders_AgreesWithTheContents(t *testing.T) { + awkward := "Tab\there \"quoted\"\nnewline ü" + + p := newV2Project(t) + ids := []string{p.add(t, 1, "Microflows$Microflow")} + id := p.add(t, 2, "Microflows$Microflow") + p.rewrite(t, id, headerDoc(t, "Microflows$Microflow", awkward, true)) + ids = append(ids, id) + p.add(t, 3, "Forms$Page") + + // Warm the persisted index with one reader, answer from it with another. + if _, err := p.reader().ListUnitHeaders("Microflows$Microflow"); err != nil { + t.Fatal(err) + } + if _, err := os.Stat(p.reader().unitTypeIndexPath()); err != nil { + t.Fatalf("control: the index was not persisted: %v", err) + } + r := p.reader() + hs, err := r.ListUnitHeaders("Microflows$Microflow") + if err != nil { + t.Fatal(err) + } + if reads := r.UnitFileReads(); reads != 0 { + t.Fatalf("control: the listing opened %d files, so it did not answer from the persisted index", reads) + } + if len(hs) != len(ids) { + t.Fatalf("listed %d microflow headers, want %d", len(hs), len(ids)) + } + for _, h := range hs { + contents, err := os.ReadFile(p.unitPath(h.ID)) + if err != nil { + t.Fatal(err) + } + e := unitIndexEntryOf(contents) + if h.Name != e.Name || h.Excluded != e.Excluded { + t.Errorf("header %s = {%q %v}, the file says {%q %v}", h.ID, h.Name, h.Excluded, e.Name, e.Excluded) + } + } + + v1 := newTypedListingFixture(t, []string{"Microflows$Microflow", "Forms$Page", "Microflows$Microflow"}) + units, err := v1.r.listUnitsByType("Microflows$Microflow") + if err != nil { + t.Fatal(err) + } + v1hs, err := v1.r.ListUnitHeaders("Microflows$Microflow") + if err != nil { + t.Fatal(err) + } + if len(v1hs) != len(units) || len(units) != 2 { + t.Fatalf("v1: %d headers for %d microflows, want 2", len(v1hs), len(units)) + } + for i, u := range units { + e := unitIndexEntryOf(u.Contents) + if v1hs[i].ID != u.ID || v1hs[i].Name != e.Name || v1hs[i].Excluded != e.Excluded { + t.Errorf("v1 header %d = %+v, the blob says {%s %q %v}", i, v1hs[i], u.ID, e.Name, e.Excluded) + } + } +} + +// An index in the previous format (type only, unit-types.idx) is neither read +// nor overwritten: an older mxcli working on the same project keeps its own. +func TestUnitIndex_PreviousFormatIsLeftAlone(t *testing.T) { + p, _ := seedProject(t, 3) + r := p.reader() + dir := r.unitTypeIndexPath()[:len(r.unitTypeIndexPath())-len(unitTypeIndexName)] + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + old := []byte("mxcli-unit-types 1\nsomehash\tMicroflows$Microflow\n") + if err := os.WriteFile(dir+"unit-types.idx", old, 0o644); err != nil { + t.Fatal(err) + } + if _, err := r.ListUnitHeaders("Microflows$Microflow"); err != nil { + t.Fatal(err) + } + got, err := os.ReadFile(dir + "unit-types.idx") + if err != nil || string(got) != string(old) { + t.Errorf("the previous format's index was changed: %q, %v", got, err) + } + if _, err := os.Stat(r.unitTypeIndexPath()); err != nil { + t.Errorf("the current index was not written beside it: %v", err) + } +} diff --git a/modelsdk/mpr/reader_units.go b/modelsdk/mpr/reader_units.go index 931451565b..99385d4fab 100644 --- a/modelsdk/mpr/reader_units.go +++ b/modelsdk/mpr/reader_units.go @@ -6,7 +6,9 @@ package mpr import ( "bytes" "crypto/sha256" + "database/sql" "encoding/base64" + "errors" "fmt" "os" "path/filepath" @@ -79,6 +81,103 @@ func (r *Reader) ListUnitsByType(typePrefix string) ([]UnitRef, error) { return result, nil } +// UnitHeaderRef is a unit's identity and its top-level Name and Excluded. +type UnitHeaderRef struct { + ID string + ContainerID string + Name string + Excluded bool +} + +// ListUnitHeaders lists the units of exactly typeName with their Name and +// Excluded. On MPR v2 these come from the unit index, so a unit whose bytes the +// index has seen is not opened: naming every folder or microflow used to open +// every one of their files on each call (ako/mxcli#1791). MPR v1 reads the +// blobs, which live in the one database file. +func (r *Reader) ListUnitHeaders(typeName string) ([]UnitHeaderRef, error) { + if r.version != MPRVersionV2 { + units, err := r.listUnitsByType(typeName) + if err != nil { + return nil, err + } + out := make([]UnitHeaderRef, 0, len(units)) + for _, u := range units { + e := unitIndexEntryOf(u.Contents) + out = append(out, UnitHeaderRef{ID: u.ID, ContainerID: u.ContainerID, Name: e.Name, Excluded: e.Excluded}) + } + return out, nil + } + if !r.unitCacheValid { + if err := r.buildUnitCache(); err != nil { + return nil, err + } + } + var out []UnitHeaderRef + for _, cu := range r.unitCache { + if cu.Type != typeName { + continue + } + name, excluded := cu.Name, cu.Excluded + if held, ok := r.overlaid(cu.ID); ok { + // Held after the cache was built: the held bytes are the unit now. + e := unitIndexEntryOf(held) + if e.Type != typeName { + continue + } + name, excluded = e.Name, e.Excluded + } + out = append(out, UnitHeaderRef{ID: cu.ID, ContainerID: cu.ContainerID, Name: name, Excluded: excluded}) + } + return out, nil +} + +// UnitByID returns the unit whose ID is id, reading that unit's contents only. +// ok is false when the project has no such unit. A unit's ID is its +// document's $ID, so a lookup of one document by ID needs no listing: the +// catalog's source build described each microflow through a listing of them +// all and opened millions of files on a large project (ako/mxcli#1789). +func (r *Reader) UnitByID(id string) (unit UnitRef, ok bool, err error) { + if r.version != MPRVersionV2 { + return r.unitByIDV1(id) + } + if !r.unitCacheValid { + if err := r.buildUnitCache(); err != nil { + return UnitRef{}, false, err + } + } + for _, cu := range r.unitCache { + if cu.ID != id { + continue + } + contents, err := r.readMprContents(cu.ID) + if err != nil { + return UnitRef{}, false, err + } + return UnitRef{ID: cu.ID, ContainerID: cu.ContainerID, Type: cu.Type, Contents: contents}, true, nil + } + return UnitRef{}, false, nil +} + +// unitByIDV1 is UnitByID on MPR v1, where the contents are a column. +func (r *Reader) unitByIDV1(id string) (UnitRef, bool, error) { + blob := uuidToBlob(id) + if blob == nil { + return UnitRef{}, false, nil + } + var containerID, contents []byte + err := r.db.QueryRow("SELECT ContainerID, Contents FROM Unit WHERE UnitID = ?", blob).Scan(&containerID, &contents) + if errors.Is(err, sql.ErrNoRows) { + return UnitRef{}, false, nil + } + if err != nil { + return UnitRef{}, false, err + } + if held, ok := r.overlaid(id); ok { + contents = held + } + return UnitRef{ID: id, ContainerID: blobToUUID(containerID), Type: getTypeFromContents(contents), Contents: contents}, true, nil +} + // ListUnitsByTypeInContainer returns the units of exactly typeName whose // ContainerID is containerID, reading the contents of those units only: a // lookup of one module's domain model must not read every module's @@ -201,21 +300,21 @@ func (r *Reader) buildUnitCache() error { } unitUUID := blobToUUID(unitID) - var typeName string + var entry unitIndexEntry if held, ok := r.overlaid(unitUUID); ok { // Bytes held in memory are what the unit is now; its stored hash // describes the file it will replace. - typeName = getTypeFromContents(held) - } else if known, ok := r.knownUnitType(hash); ok { - typeName = known + entry = unitIndexEntryOf(held) + } else if known, ok := r.knownUnit(hash); ok { + entry = known } else { contents, err := r.readMprContents(unitUUID) if err != nil { continue } opened++ - typeName = getTypeFromContents(contents) - r.learnUnitType(contentsHash(contents), typeName) + entry = unitIndexEntryOf(contents) + r.learnUnit(contentsHash(contents), entry) } if hash != "" { hashes[hash] = struct{}{} @@ -224,7 +323,9 @@ func (r *Reader) buildUnitCache() error { ID: unitUUID, ContainerID: blobToUUID(containerID), ContainmentName: containmentName, - Type: typeName, + Type: entry.Type, + Name: entry.Name, + Excluded: entry.Excluded, }) } if err := rows.Err(); err != nil { diff --git a/modelsdk/mpr/reader_units_v2.go b/modelsdk/mpr/reader_units_v2.go index 25bc23de37..6178b76962 100644 --- a/modelsdk/mpr/reader_units_v2.go +++ b/modelsdk/mpr/reader_units_v2.go @@ -7,7 +7,10 @@ import ( "bytes" "os" "path/filepath" + "strconv" "strings" + + "go.mongodb.org/mongo-driver/v2/bson" ) // The MPR v2 unit-type index. @@ -26,7 +29,14 @@ import ( // change — including a re-type in place (#503) — has a new hash and is read // again; unchanged bytes have the type they had. // -// The index is persisted to /.mxcli/unit-types.idx so the next +// Beside the $Type the index holds the unit's top-level Name and Excluded, +// which are as much a function of the bytes as the type is. They are what a +// name listing needs: the container hierarchy names every folder, and a +// describe names every microflow, so without them each of those opened every +// folder or microflow file on every call — 2,904 of a 4,753-unit app's files +// for one `describe microflow` (ako/mxcli#1791). +// +// The index is persisted to /.mxcli/unit-index-2.idx so the next // process starts warm. The file is a pure cache: absent, unreadable, corrupt // or unwritable all mean "type from the file", never a wrong answer. A unit // with no ContentsHash cannot be keyed and is read every time, as before. @@ -39,11 +49,34 @@ import ( // verify-hashes` finds it, ako/mxcli#972), and a lookup by that stale hash // returns the type of the bytes it was last true of. -const unitTypeIndexName = "unit-types.idx" +// The file name carries the format: an mxcli that wrote the previous format +// (unit-types.idx, type only) and one that writes this one each read their +// own, rather than discarding each other's as corrupt on every call when two +// versions work on one project — the VS Code extension's and the CLI's. +const unitTypeIndexName = "unit-index-2.idx" // unitTypeIndexHeader is the first line of the persisted index; a file // without it is ignored, so a format change only costs one cold start. -const unitTypeIndexHeader = "mxcli-unit-types 1" +const unitTypeIndexHeader = "mxcli-unit-index 2" + +// unitIndexEntry is what the index knows of the unit whose bytes hash to its +// key: everything derived from the top level of the document. +type unitIndexEntry struct { + Type string + Name string + Excluded bool +} + +// unitIndexEntryOf derives an entry from a unit's bytes. +func unitIndexEntryOf(contents []byte) unitIndexEntry { + e := unitIndexEntry{Type: getTypeFromContents(contents)} + raw := bson.Raw(contents) + if raw.Validate() == nil { + e.Name, _ = raw.Lookup("Name").StringValueOK() + e.Excluded, _ = raw.Lookup("Excluded").BooleanOK() + } + return e +} // unitTypeIndexPersistAfter is how many files a later rebuild must have // opened before it rewrites the persisted index straight away. A reader's @@ -61,30 +94,30 @@ func (r *Reader) unitTypeIndexPath() string { return filepath.Join(filepath.Dir(r.path), ".mxcli", unitTypeIndexName) } -// knownUnitType returns the indexed $Type for a ContentsHash, loading the +// knownUnit returns the indexed entry for a ContentsHash, loading the // persisted index on first use. Caller holds v2TypesMu. -func (r *Reader) knownUnitType(hash string) (string, bool) { +func (r *Reader) knownUnit(hash string) (unitIndexEntry, bool) { if !r.v2TypesLoaded { r.v2TypesLoaded = true r.v2Types = loadUnitTypeIndex(r.unitTypeIndexPath()) } if hash == "" { - return "", false + return unitIndexEntry{}, false } - typ, ok := r.v2Types[hash] - return typ, ok + e, ok := r.v2Types[hash] + return e, ok } -// learnUnitType records a hash's $Type. Caller holds v2TypesMu. -func (r *Reader) learnUnitType(hash, typ string) { +// learnUnit records a hash's entry. Caller holds v2TypesMu. +func (r *Reader) learnUnit(hash string, e unitIndexEntry) { if hash == "" { return } if r.v2Types == nil { - r.v2Types = make(map[string]string) + r.v2Types = make(map[string]unitIndexEntry) } - if old, ok := r.v2Types[hash]; !ok || old != typ { - r.v2Types[hash] = typ + if old, ok := r.v2Types[hash]; !ok || old != e { + r.v2Types[hash] = e r.v2TypesDirty = true } } @@ -103,13 +136,23 @@ func (r *Reader) persistUnitTypeIndex() { buf.WriteString(unitTypeIndexHeader) buf.WriteByte('\n') for hash := range r.v2Hashes { - typ, ok := r.v2Types[hash] + e, ok := r.v2Types[hash] if !ok { continue } + // hash, type, excluded, name: the name last and quoted, so no + // character a name can hold splits the line. + excluded := "0" + if e.Excluded { + excluded = "1" + } buf.WriteString(hash) buf.WriteByte('\t') - buf.WriteString(typ) + buf.WriteString(e.Type) + buf.WriteByte('\t') + buf.WriteString(excluded) + buf.WriteByte('\t') + buf.WriteString(strconv.Quote(e.Name)) buf.WriteByte('\n') } if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { @@ -130,8 +173,8 @@ func (r *Reader) persistUnitTypeIndex() { } // loadUnitTypeIndex reads a persisted index; any problem yields an empty one. -func loadUnitTypeIndex(path string) map[string]string { - out := make(map[string]string) +func loadUnitTypeIndex(path string) map[string]unitIndexEntry { + out := make(map[string]unitIndexEntry) if path == "" { return out } @@ -145,14 +188,18 @@ func loadUnitTypeIndex(path string) map[string]string { return out } for sc.Scan() { - hash, typ, ok := strings.Cut(sc.Text(), "\t") - if !ok || hash == "" || strings.Contains(typ, "\t") { - return make(map[string]string) // corrupt: start cold rather than half-trust it + f := strings.SplitN(sc.Text(), "\t", 4) + if len(f) != 4 || f[0] == "" || (f[2] != "0" && f[2] != "1") { + return make(map[string]unitIndexEntry) // corrupt: start cold rather than half-trust it + } + name, err := strconv.Unquote(f[3]) + if err != nil { + return make(map[string]unitIndexEntry) } - out[hash] = typ + out[f[0]] = unitIndexEntry{Type: f[1], Excluded: f[2] == "1", Name: name} } if sc.Err() != nil { - return make(map[string]string) + return make(map[string]unitIndexEntry) } return out } diff --git a/modelsdk/mpr/type_floor_projects_test.go b/modelsdk/mpr/type_floor_projects_test.go new file mode 100644 index 0000000000..63cbd6618d --- /dev/null +++ b/modelsdk/mpr/type_floor_projects_test.go @@ -0,0 +1,113 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "os" + "path/filepath" + "sort" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/modelsdk/canon" + mversion "github.com/mendixlabs/mxcli/modelsdk/version" +) + +// typeFloorProjectsEnv lists more projects, path-list separated, for +// TestTypeFloor_StoredProjectsHoldNoUndeclaredType to scan: the measurement +// behind the type floor's data was this test run over Studio Pro-authored +// projects (blank apps from `mx create-project` on 9.24.30, 10.24.24, 11.13.0 +// and 11.15.0, and Marketplace modules) — ako/mxcli#1774. +const typeFloorProjectsEnv = "MXCLI_TYPEFLOOR_PROJECTS" + +// A project of version V holds only types V has, so the type floor must accept +// every unit stored in one: a refusal here is a row of +// mendixmodelsdk_type_versions.json that is wrong for storage, and on the +// write path it would refuse rewriting what Studio Pro wrote. Types the SDK +// says were deleted at or before V are logged: they are the evidence for not +// enforcing the deletion data. +func TestTypeFloor_StoredProjectsHoldNoUndeclaredType(t *testing.T) { + projects := []string{ + filepath.Join("testdata", "v1-project", "App.mpr"), // 9.24.30 + filepath.Join("..", "..", "testdata", "expr-checker", "minimal.mpr"), // 11.6.6 + } + if extra := os.Getenv(typeFloorProjectsEnv); extra != "" { + projects = append(projects, filepath.SplitList(extra)...) + } + for _, path := range projects { + if path == "" { + continue + } + t.Run(filepath.Base(filepath.Dir(path))+"/"+filepath.Base(path), func(t *testing.T) { + r, err := Open(path) + if err != nil { + t.Fatal(err) + } + defer r.Close() + pv := r.ProjectVersion() + v := mversion.Version{Major: pv.MajorVersion, Minor: pv.MinorVersion, Patch: pv.PatchVersion} + ids, err := r.ListAllUnitIDs() + if err != nil { + t.Fatal(err) + } + types := map[string]bool{} + deletedHeld := map[string]bool{} + for _, id := range ids { + raw, err := r.GetRawUnitBytes(id) + if err != nil || len(raw) == 0 { + continue + } + if err := canon.UndeclaredTypeError(raw, &v); err != nil { + t.Errorf("stored unit %s: %v", id, err) + } + collectTypes(bson.Raw(raw), types) + } + for typ := range types { + if d, ok := canon.TypeDeleted(typ); ok && v.Compare(d) >= 0 { + deletedHeld[typ+" (SDK: deleted "+d.String()+")"] = true + } + } + // Positive control: the walk reaches the data. Two majors down, + // a real project holds types that did not exist. + low := mversion.Version{Major: v.Major - 2} + refusedLow := 0 + for _, id := range ids { + if raw, err := r.GetRawUnitBytes(id); err == nil && canon.UndeclaredTypeError(raw, &low) != nil { + refusedLow++ + } + } + if refusedLow == 0 { + t.Errorf("control: no unit of this %s project is refused at %s — the floor sees nothing", v, low) + } + held := make([]string, 0, len(deletedHeld)) + for k := range deletedHeld { + held = append(held, k) + } + sort.Strings(held) + t.Logf("Mendix %s: %d units, %d distinct $Types; stored types the SDK calls deleted: %s", + v, len(ids), len(types), strings.Join(held, ", ")) + }) + } +} + +func collectTypes(raw bson.Raw, into map[string]bool) { + elems, err := raw.Elements() + if err != nil { + return + } + for _, e := range elems { + val := e.Value() + switch val.Type { + case bson.TypeString: + if e.Key() == "$Type" { + into[val.StringValue()] = true + } + case bson.TypeEmbeddedDocument: + collectTypes(val.Document(), into) + case bson.TypeArray: + collectTypes(bson.Raw(val.Array()), into) + } + } +} diff --git a/modelsdk/mpr/types.go b/modelsdk/mpr/types.go index fa60868d07..57506c8416 100644 --- a/modelsdk/mpr/types.go +++ b/modelsdk/mpr/types.go @@ -98,25 +98,16 @@ func (r *Reader) ListUnits() ([]*UnitInfo, error) { // ListFolders returns all project folders with their names. func (r *Reader) ListFolders() ([]*FolderInfo, error) { - units, err := r.listUnitsByType("Projects$Folder") + units, err := r.ListUnitHeaders("Projects$Folder") if err != nil { return nil, err } - var result []*FolderInfo + result := make([]*FolderInfo, 0, len(units)) for _, u := range units { - name := "" - if len(u.Contents) > 0 { - var raw map[string]any - if err := bsoncompat.UnmarshalDocMap(u.Contents, &raw); err == nil { - if n, ok := raw["Name"].(string); ok { - name = n - } - } - } result = append(result, &FolderInfo{ ID: u.ID, ContainerID: u.ContainerID, - Name: name, + Name: u.Name, }) } return result, nil diff --git a/modelsdk/mpr/writer_core.go b/modelsdk/mpr/writer_core.go index 15cb1d6c54..393ab264bf 100644 --- a/modelsdk/mpr/writer_core.go +++ b/modelsdk/mpr/writer_core.go @@ -5,9 +5,7 @@ package mpr import ( "bytes" "crypto/rand" - "crypto/sha256" "database/sql" - "encoding/base64" "encoding/hex" "fmt" "os" @@ -226,7 +224,12 @@ func (wt *WriteTransaction) WriteUnit(unitID string, contents []byte) error { // same storage (codec.Store.SaveUnit / FlushUnits reach it), so leaving it // out would mean the codec engine's own document writes kept churning while // UpdateRawUnit's stopped — a difference no caller could reason about. - contents, unchanged, err := wt.writer.reconcileWithStored(unitID, contents) + // + // The stored bytes are read through the transaction. On v1 they are in the + // database, and the pool's one connection is the one wt.tx holds: reading + // through the reader waited for it forever. + stored := func(id string) ([]byte, error) { return wt.writer.reader.rawUnitBytes(wt.tx, id) } + contents, unchanged, err := wt.writer.reconcileAgainst(unitID, contents, stored) if err != nil { return err } @@ -266,18 +269,17 @@ func (wt *WriteTransaction) WriteUnit(unitID string, contents []byte) error { }) // Update hash in DB - hash := sha256.Sum256(contents) - contentsHash := base64.StdEncoding.EncodeToString(hash[:]) _, err := wt.tx.Exec(` UPDATE Unit SET ContentsHash = ? WHERE UnitID = ? - `, contentsHash, unitIDBlob) + `, contentsHashOf(contents), unitIDBlob) return err } - // V1: Update in database directly + // V1: contents live in the row, and so does their hash. Studio Pro fills + // ContentsHash on every v1 unit too (ako/mxcli#1159), so it moves with them. _, err = wt.tx.Exec(` - UPDATE Unit SET Contents = ? WHERE UnitID = ? - `, contents, unitIDBlob) + UPDATE Unit SET Contents = ?, ContentsHash = ? WHERE UnitID = ? + `, contents, contentsHashOf(contents), unitIDBlob) return err } @@ -588,15 +590,11 @@ func (w *Writer) insertUnit(unitID, containerID, containmentName, unitType strin return fmt.Errorf("failed to write unit file: %w", err) } - // Compute content hash (base64-encoded SHA256) - hash := sha256.Sum256(contents) - contentsHash := base64.StdEncoding.EncodeToString(hash[:]) - // Insert reference to database _, err := w.reader.db.Exec(` INSERT INTO Unit (UnitID, ContainerID, ContainmentName, TreeConflict, ContentsHash, ContentsConflicts) VALUES (?, ?, ?, 0, ?, '') - `, unitIDBlob, containerIDBlob, containmentName, contentsHash) + `, unitIDBlob, containerIDBlob, containmentName, contentsHashOf(contents)) if err != nil { // Clean up the file we just wrote — otherwise it becomes an orphan os.Remove(filePath) @@ -608,12 +606,13 @@ func (w *Writer) insertUnit(unitID, containerID, containmentName, unitType strin return nil } - // MPR v1: Store directly in database + // MPR v1: Store directly in database, with the hash Studio Pro stores + // beside the contents on every v1 unit (ako/mxcli#1159; this wrote ''). // Try new schema first (without Type column - Mendix 11.6.2+) _, err := w.reader.db.Exec(` INSERT INTO Unit (UnitID, ContainerID, ContainmentName, TreeConflict, ContentsHash, ContentsConflicts, Contents) - VALUES (?, ?, ?, 0, '', '', ?) - `, unitIDBlob, containerIDBlob, containmentName, contents) + VALUES (?, ?, ?, 0, ?, '', ?) + `, unitIDBlob, containerIDBlob, containmentName, contentsHashOf(contents), contents) if err != nil { // Try old schema with Type column _, err = w.reader.db.Exec(` @@ -702,11 +701,9 @@ func (w *Writer) updateUnit(unitID string, contents []byte, opts ...canon.Option } // Update ContentsHash in database - hash := sha256.Sum256(contents) - contentsHash := base64.StdEncoding.EncodeToString(hash[:]) _, err := w.reader.db.Exec(` UPDATE Unit SET ContentsHash = ? WHERE UnitID = ? - `, contentsHash, unitIDBlob) + `, contentsHashOf(contents), unitIDBlob) if err == nil { w.reader.InvalidateCache() w.updateTransactionID() @@ -714,16 +711,24 @@ func (w *Writer) updateUnit(unitID string, contents []byte, opts ...canon.Option return err } - // MPR v1: Update in database + // MPR v1: Update in database. ContentsHash goes in the same statement: + // Studio Pro fills it on every v1 unit and reads it to see what changed, + // and mx check never looks at it (ako/mxcli#1159). _, err = w.reader.db.Exec(` - UPDATE Unit SET Contents = ? WHERE UnitID = ? - `, contents, unitIDBlob) + UPDATE Unit SET Contents = ?, ContentsHash = ? WHERE UnitID = ? + `, contents, contentsHashOf(contents), unitIDBlob) return err } // reconcileWithStored applies the shared no-op-elision policy (canon.Reconcile, // ADR-0008 decision 1) to a write against this project. func (w *Writer) reconcileWithStored(unitID string, contents []byte, opts ...canon.Option) (out []byte, unchanged bool, err error) { + return w.reconcileAgainst(unitID, contents, w.reader.GetRawUnitBytes, opts...) +} + +// reconcileAgainst is reconcileWithStored reading the stored unit with +// readStored, so a write inside a transaction can read through it. +func (w *Writer) reconcileAgainst(unitID string, contents []byte, readStored func(string) ([]byte, error), opts ...canon.Option) (out []byte, unchanged bool, err error) { w.writesOffered++ // Normalised before comparing, so a rebuild of a Studio Pro document that // differed from it only by the properties the writer leaves out is elided. @@ -731,7 +736,7 @@ func (w *Writer) reconcileWithStored(unitID string, contents []byte, opts ...can if err != nil { return nil, false, err } - stored, readErr := w.reader.GetRawUnitBytes(unitID) + stored, readErr := readStored(unitID) if readErr != nil { w.writesLanded++ return contents, false, nil // new unit, or unreadable — write it diff --git a/modelsdk/mpr/writer_property_sets.go b/modelsdk/mpr/writer_property_sets.go index 354d6e6e2f..e16ccc6731 100644 --- a/modelsdk/mpr/writer_property_sets.go +++ b/modelsdk/mpr/writer_property_sets.go @@ -24,6 +24,11 @@ type PropertySetRepair struct { Name string // the unit's Name, "" when it has none Added int Stripped int + + // normalized is the unit as the repair leaves it (the stored bytes when + // nothing changes), for a PropertySetCensus; nil when it could not be + // normalized. + normalized []byte } // Changed reports whether the unit's property-name sets differ from what is @@ -103,6 +108,7 @@ func planPropertySetRepair(r *Reader, unitID string) (rep PropertySetRepair, sto if err != nil { return rep, stored, nil, err } + rep.normalized = out if len(out) == len(stored) && string(out) == string(stored) { return rep, stored, out, nil } @@ -135,6 +141,10 @@ func planPropertySetRepair(r *Reader, unitID string) (rep PropertySetRepair, sto return rep, stored, out, nil } +// lowerToProjectShape is canon.LowerToProjectShape, a variable so a test can +// stub the lowering and show what the checks after it see without it. +var lowerToProjectShape = canon.LowerToProjectShape + // normalizePropertySetsFor is normalizePropertySets for a given project // version, shared by the write path and the stored-unit repair. func normalizePropertySetsFor(pv *version.ProjectVersion, unitID string, contents []byte) ([]byte, error) { @@ -145,7 +155,7 @@ func normalizePropertySetsFor(pv *version.ProjectVersion, unitID string, content // Properties whose shape changed at a version are written in the project's // shape first, so the floor sees the keys this version declares // (ako/mxcli#1209). - lowered, err := canon.LowerToProjectShape(contents, &v) + lowered, err := lowerToProjectShape(contents, &v) if err != nil { return nil, fmt.Errorf("unit %s: %w", unitID, err) } @@ -155,7 +165,15 @@ func normalizePropertySetsFor(pv *version.ProjectVersion, unitID string, content } // A GUID-less user role gets its $ID as GUID — absent only, never changed // (canon.FillMissingUserRoleGUIDs). Not a table row: the value is per role. - return canon.FillMissingUserRoleGUIDs(canon.CompletePropertySets(stripped, &v)), nil + out := canon.FillMissingUserRoleGUIDs(canon.CompletePropertySets(stripped, &v)) + // An element whose $Type this version does not have yet makes the project + // unloadable (TypeCacheUnknownTypeException). What has an older shape was + // lowered above; what is left is refused, on the bytes that would land + // (ako/mxcli#1774). + if err := canon.UndeclaredTypeError(out, &v); err != nil { + return nil, fmt.Errorf("unit %s: %w", unitID, err) + } + return out, nil } // elementKeySets maps every document node in raw, by its containment path, to diff --git a/modelsdk/mpr/writer_type_floor_1774_test.go b/modelsdk/mpr/writer_type_floor_1774_test.go new file mode 100644 index 0000000000..51c78f811c --- /dev/null +++ b/modelsdk/mpr/writer_type_floor_1774_test.go @@ -0,0 +1,161 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "bytes" + "os" + "path/filepath" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/modelsdk/mpr/version" + mversion "github.com/mendixlabs/mxcli/modelsdk/version" +) + +// ako/mxcli#1774: the version data held property floors only, so a $Type +// newer than the project went through every write, and the project then did +// not load (`TypeCacheUnknownTypeException: The type cache does not contain a +// type with qualified name …`). It shipped four times: #1754, #1209, #1163 +// (Workflows$EndOfParallelSplitPathActivity, 10.21) and #1771 +// (Workflows$SingleUserTaskActivity, 10.12). Each type's own introduction +// version is now in mendixmodelsdk_type_versions.json, and every unit write +// refuses an element whose type the project does not have yet. + +// waitActivity is a wait activity as the workflow encoder writes one. +func waitActivity(typeName, name string) bson.D { + return bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: typeName}, + {Key: "Annotation", Value: nil}, {Key: "BoundaryEvents", Value: bson.A{int32(2)}}, + {Key: "Caption", Value: name}, {Key: "Name", Value: name}} +} + +// workflowHolding is a workflow whose main flow holds the given activities. +func workflowHolding(acts ...bson.D) bson.D { + list := bson.A{int32(3)} + for _, a := range acts { + list = append(list, a) + } + return bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$Workflow"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$Flow"}, + {Key: "Activities", Value: list}}}, + {Key: "Name", Value: "WF"}, {Key: "Title", Value: "WF"}} +} + +// parallelSplitHolding is a parallel split whose one path holds act, nested +// two flows down from the workflow. +func parallelSplitHolding(act bson.D) bson.D { + path := bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$ParallelSplitOutcome"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$Flow"}, + {Key: "Activities", Value: bson.A{int32(3), act}}}}} + return bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$ParallelSplitActivity"}, + {Key: "Annotation", Value: nil}, {Key: "Caption", Value: "Parallel split"}, {Key: "Name", Value: "split"}, + {Key: "Outcomes", Value: bson.A{int32(3), path}}} +} + +func label(v *version.ProjectVersion) string { + return mversion.Version{Major: v.MajorVersion, Minor: v.MinorVersion, Patch: v.PatchVersion}.String() +} + +// Workflows$WaitForTimerActivity is a 10.6.0 type and +// Workflows$WaitForNotificationActivity a 10.0.0 one (mendixmodelsdk 4.117.0, +// each type's versionInfo). Neither has an older shape, so below its version +// the write is refused, naming the type, its version and the project's. +func TestNormalize_RefusesATypeNewerThanTheProject(t *testing.T) { + for _, tt := range []struct { + v *version.ProjectVersion + typ string + refuse string // "" = accepted + }{ + {pv(10, 5, 0), "Workflows$WaitForTimerActivity", "Workflows$WaitForTimerActivity (Mendix 10.6.0+)"}, + {pv(10, 6, 0), "Workflows$WaitForTimerActivity", ""}, + {pv(9, 24, 30), "Workflows$WaitForNotificationActivity", "Workflows$WaitForNotificationActivity (Mendix 10.0.0+)"}, + {pv(10, 0, 0), "Workflows$WaitForNotificationActivity", ""}, + {pv(11, 13, 0), "Workflows$WaitForTimerActivity", ""}, // control + } { + _, err := normalizeShape(t, tt.v, workflowHolding(waitActivity(tt.typ, "wait"))) + switch { + case tt.refuse == "" && err != nil: + t.Errorf("%s: %s is refused: %v", label(tt.v), tt.typ, err) + case tt.refuse != "" && err == nil: + t.Errorf("%s: %s was accepted; want it refused", label(tt.v), tt.typ) + case tt.refuse != "": + for _, want := range []string{"this Mendix " + label(tt.v) + " project cannot store", tt.refuse} { + if !strings.Contains(err.Error(), want) { + t.Errorf("%s: refusal %q does not say %q", label(tt.v), err, want) + } + } + } + } +} + +// The check walks the whole unit: a wait activity on a parallel split path, +// two flows below the workflow, is refused as surely as one in the main flow. +func TestNormalize_RefusesANestedTypeNewerThanTheProject(t *testing.T) { + wf := workflowHolding(parallelSplitHolding(waitActivity("Workflows$WaitForTimerActivity", "nested"))) + _, err := normalizeShape(t, pv(10, 5, 0), wf) + if err == nil || !strings.Contains(err.Error(), "Workflows$WaitForTimerActivity (Mendix 10.6.0+)") { + t.Errorf("a nested 10.6 type on 10.5 was not refused naming it: %v", err) + } +} + +// The check runs after LowerToProjectShape, so what has an older shape is +// lowered and not refused: the parallel split end marker (10.21) on 9.24. With +// the lowering stubbed out, the same unit is refused — the check is what +// stands behind every lowering, and would have caught #1163 with none. +func TestNormalize_TypeFloorRunsAfterTheLowering(t *testing.T) { + if _, err := normalizeShape(t, pv(9, 24, 30), workflowWithParallelSplit()); err != nil { + t.Fatalf("a lowered parallel split is refused on 9.24: %v", err) + } + + saved := lowerToProjectShape + lowerToProjectShape = func(contents []byte, _ *mversion.Version) ([]byte, error) { return contents, nil } + defer func() { lowerToProjectShape = saved }() + + // The marker without its PersistentId (10.21, a property the floor refuses + // on its own), so what is left to refuse it is its type. + marker := bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$EndOfParallelSplitPathActivity"}, + {Key: "Annotation", Value: nil}, {Key: "Caption", Value: "End of parallel split path"}, + {Key: "Name", Value: "EndOfParallelSplitPath"}} + _, err := normalizeShape(t, pv(9, 24, 30), workflowHolding(parallelSplitHolding(marker))) + if err == nil || !strings.Contains(err.Error(), "Workflows$EndOfParallelSplitPathActivity (Mendix 10.21.0+)") { + t.Errorf("with the lowering stubbed, the 10.21 end marker on 9.24 was not refused: %v", err) + } +} + +// The refusal is a refusal of the write: the stored unit is left as it was, +// byte for byte, and nothing of the statement lands. +func TestWriter_TypeNewerThanTheProjectIsNotWritten(t *testing.T) { + stored, err := bson.Marshal(workflowHolding()) + if err != nil { + t.Fatal(err) + } + unitID := "8a6f3c1e-1774-4d2b-9a5e-0c3b2f1e7d64" + w, path := newV2WriterForCommitTest(t, unitID, stored) + w.reader.projectVersion = pv(10, 5, 0) + + next, err := bson.Marshal(workflowHolding(waitActivity("Workflows$WaitForTimerActivity", "wait"))) + if err != nil { + t.Fatal(err) + } + err = w.UpdateRawUnit(unitID, next) + if err == nil || !strings.Contains(err.Error(), "Workflows$WaitForTimerActivity (Mendix 10.6.0+)") { + t.Fatalf("UpdateRawUnit of a 10.6 type on 10.5 = %v, want it refused", err) + } + if after, _ := os.ReadFile(path); !bytes.Equal(after, stored) { + t.Error("the refused write changed the stored unit") + } + + // A new unit is refused the same way, and leaves no file behind. + newID := "8a6f3c1e-1774-4d2b-9a5e-0c3b2f1e7d65" + err = w.InsertUnit(newID, unitID, "Documents", "Workflows$Workflow", next) + if err == nil || !strings.Contains(err.Error(), "Workflows$WaitForTimerActivity (Mendix 10.6.0+)") { + t.Fatalf("InsertUnit of a 10.6 type on 10.5 = %v, want it refused", err) + } + swapped := blobToUUIDSwapped(uuidToBlob(newID)) + if _, statErr := os.Stat(filepath.Join(filepath.Dir(filepath.Dir(filepath.Dir(path))), + swapped[0:2], swapped[2:4], swapped+".mxunit")); !os.IsNotExist(statErr) { + t.Errorf("the refused insert left a unit file (stat: %v)", statErr) + } +} diff --git a/modelsdk/mpr/writer_v1_contents_hash_test.go b/modelsdk/mpr/writer_v1_contents_hash_test.go new file mode 100644 index 0000000000..9c02f4bff1 --- /dev/null +++ b/modelsdk/mpr/writer_v1_contents_hash_test.go @@ -0,0 +1,202 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "bytes" + "os" + "path/filepath" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +// ako/mxcli#1159: on an MPR v1 project (unit contents inside the .mpr, Mendix +// below 10.18) every unit write replaced Unit.Contents and left +// Unit.ContentsHash describing the old bytes; an insert stored ''. Studio Pro +// fills the column on every unit, base64(SHA-256(Contents)) — measured on a +// fresh 9.24.30 app (308 of 308 units) and on testdata/v1-project — and uses +// it to detect what changed. mx check and MxBuild do not read it, so nothing +// but this check notices. + +// copyV1Fixture copies the Studio Pro-authored v1 fixture into a temp dir and +// returns the copy's path. +func copyV1Fixture(t *testing.T) string { + t.Helper() + b, err := os.ReadFile(filepath.Join("testdata", "v1-project", "App.mpr")) + if err != nil { + t.Fatalf("read v1 fixture: %v", err) + } + dst := filepath.Join(t.TempDir(), "App.mpr") + if err := os.WriteFile(dst, b, 0o644); err != nil { + t.Fatal(err) + } + return dst +} + +// staleV1Hashes is the reporter's check: every Unit row whose ContentsHash is +// not base64(SHA-256(Contents)). +func staleV1Hashes(t *testing.T, w *Writer) []string { + t.Helper() + rows, err := w.reader.db.Query(`SELECT UnitID, COALESCE(ContentsHash, ''), Contents FROM Unit`) + if err != nil { + t.Fatal(err) + } + defer rows.Close() + var stale []string + for rows.Next() { + var id, contents []byte + var hash string + if err := rows.Scan(&id, &hash, &contents); err != nil { + t.Fatal(err) + } + if hash != hashOf(contents) { + stale = append(stale, blobToUUID(id)) + } + } + if err := rows.Err(); err != nil { + t.Fatal(err) + } + return stale +} + +// v1UnitOfType returns the ID and contents of the first unit of typeName. +func v1UnitOfType(t *testing.T, w *Writer, typeName string) (string, []byte) { + t.Helper() + rows, err := w.reader.db.Query(`SELECT UnitID, Contents FROM Unit`) + if err != nil { + t.Fatal(err) + } + defer rows.Close() + for rows.Next() { + var id, contents []byte + if err := rows.Scan(&id, &contents); err != nil { + t.Fatal(err) + } + if getTypeFromContents(contents) == typeName { + return blobToUUID(id), contents + } + } + t.Fatalf("no %s unit in the fixture", typeName) + return "", nil +} + +// withFields returns doc with the given top-level fields replaced. +func withFields(t *testing.T, doc []byte, fields map[string]any) []byte { + t.Helper() + var d bson.D + if err := bson.Unmarshal(doc, &d); err != nil { + t.Fatal(err) + } + for i := range d { + if v, ok := fields[d[i].Key]; ok { + d[i].Value = v + } + } + out, err := bson.Marshal(d) + if err != nil { + t.Fatal(err) + } + return out +} + +func openV1FixtureWriter(t *testing.T) *Writer { + t.Helper() + w, err := NewWriter(copyV1Fixture(t)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = w.Close() }) + if w.reader.version != MPRVersionV1 { + t.Fatalf("fixture opened as MPR v%d, want v1", w.reader.version) + } + return w +} + +// TestV1FixtureHashesMatchContents is the control: what Studio Pro stores is +// exactly the formula the writer must reproduce. If this fails, the formula +// is wrong, not the writer. +func TestV1FixtureHashesMatchContents(t *testing.T) { + w := openV1FixtureWriter(t) + if stale := staleV1Hashes(t, w); len(stale) != 0 { + t.Fatalf("Studio Pro-authored fixture has %d unit(s) whose ContentsHash is not base64(SHA-256(Contents)): %v", len(stale), stale) + } +} + +func TestV1UpdateUnitUpdatesContentsHash(t *testing.T) { + w := openV1FixtureWriter(t) + id, stored := v1UnitOfType(t, w, "Projects$Folder") + updated := withFields(t, stored, map[string]any{"Name": "RenamedBy1159"}) + + if err := w.UpdateRawUnit(id, updated); err != nil { + t.Fatalf("UpdateRawUnit: %v", err) + } + got, err := w.reader.GetRawUnitBytes(id) + if err != nil { + t.Fatal(err) + } + if bytes.Equal(got, stored) { + t.Fatal("update did not land: Contents unchanged, so the test proves nothing") + } + if stale := staleV1Hashes(t, w); len(stale) != 0 { + t.Fatalf("after an update, ContentsHash is stale for %v (stored %q, want %q)", + stale, storedHash(t, w, id), hashOf(got)) + } +} + +// Through the real one-connection pool: the transaction now reads the stored +// bytes through itself (writer_v1_transaction_test.go), so it no longer needs a +// pool of its own to avoid waiting on the connection it holds. +func TestV1WriteTransactionUpdatesContentsHash(t *testing.T) { + w := openV1FixtureWriter(t) + id, stored := v1UnitOfType(t, w, "Projects$Folder") + updated := withFields(t, stored, map[string]any{"Name": "RenamedInTransaction"}) + + wt, err := w.BeginWriteTransaction() + if err != nil { + t.Fatal(err) + } + if err := wt.WriteUnit(id, updated); err != nil { + _ = wt.Rollback() + t.Fatalf("WriteUnit: %v", err) + } + if err := wt.Commit(); err != nil { + t.Fatalf("Commit: %v", err) + } + got, err := w.reader.GetRawUnitBytes(id) + if err != nil { + t.Fatal(err) + } + if bytes.Equal(got, stored) { + t.Fatal("write did not land: Contents unchanged, so the test proves nothing") + } + if stale := staleV1Hashes(t, w); len(stale) != 0 { + t.Fatalf("after a transactional write, ContentsHash is stale for %v (stored %q, want %q)", + stale, storedHash(t, w, id), hashOf(got)) + } +} + +func TestV1InsertUnitStoresContentsHash(t *testing.T) { + w := openV1FixtureWriter(t) + moduleID, _ := v1UnitOfType(t, w, "Projects$ModuleImpl") + _, folder := v1UnitOfType(t, w, "Projects$Folder") + + newID := generateUUID() + contents := withFields(t, folder, map[string]any{ + "$ID": bson.Binary{Subtype: 0x00, Data: uuidToBlob(newID)}, + "Name": "InsertedBy1159", + }) + if err := w.InsertUnit(newID, moduleID, "Folders", "Projects$Folder", contents); err != nil { + t.Fatalf("InsertUnit: %v", err) + } + got, err := w.reader.GetRawUnitBytes(newID) + if err != nil { + t.Fatalf("inserted unit not readable: %v", err) + } + if h := storedHash(t, w, newID); h != hashOf(got) { + t.Fatalf("inserted unit's ContentsHash = %q, want %q", h, hashOf(got)) + } + if stale := staleV1Hashes(t, w); len(stale) != 0 { + t.Fatalf("after an insert, ContentsHash is stale for %v", stale) + } +} diff --git a/modelsdk/mpr/writer_v1_transaction_test.go b/modelsdk/mpr/writer_v1_transaction_test.go new file mode 100644 index 0000000000..e76214ab5b --- /dev/null +++ b/modelsdk/mpr/writer_v1_transaction_test.go @@ -0,0 +1,119 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "bytes" + "database/sql" + "path/filepath" + "testing" + "time" + + "go.mongodb.org/mongo-driver/v2/bson" + _ "modernc.org/sqlite" +) + +// WriteTransaction.WriteUnit reconciles the new contents with the stored ones +// before writing. On MPR v1 the stored contents are in the database, and the +// reader's pool holds one connection (reader.go, SetMaxOpenConns(1)) — the one +// the open transaction owns. Reading them through the reader instead of the +// transaction waited for that connection forever, so codec.Store.SaveUnit and +// FlushUnits never returned on a v1 project. v2 reads the stored bytes from +// mprcontents/ and never touched the pool, which is why no v2 test saw it. + +// newV1ProjectFile writes a minimal MPR v1 project — the Unit table with its +// Contents column, and the metadata the reader detects the version from — and +// returns its path. It is opened afterwards through NewWriter, the path every +// real write takes, so the pool is the real one-connection pool. +func newV1ProjectFile(t *testing.T, unitID string, stored []byte) string { + t.Helper() + path := filepath.Join(t.TempDir(), "app.mpr") + db, err := sql.Open("sqlite", path) + if err != nil { + t.Fatal(err) + } + defer db.Close() + for _, stmt := range []string{ + `CREATE TABLE _MetaData (_ProductVersion TEXT, _BuildVersion TEXT, _SchemaHash TEXT)`, + `INSERT INTO _MetaData VALUES ('9.24.30', '9.24.30.51827', '')`, + `CREATE TABLE Unit (UnitID BLOB PRIMARY KEY NOT NULL, ContainerID BLOB, ContainmentName TEXT, + TreeConflict LONG, ContentsHash TEXT, ContentsConflicts TEXT, Contents BLOB)`, + } { + if _, err := db.Exec(stmt); err != nil { + t.Fatal(err) + } + } + if _, err := db.Exec(`INSERT INTO Unit (UnitID, ContainerID, ContainmentName, Contents) VALUES (?, ?, ?, ?)`, + uuidToBlob(unitID), uuidToBlob(v1UnitID(0)), "Documents", stored); err != nil { + t.Fatal(err) + } + return path +} + +func v1TestUnit(t *testing.T, name string) []byte { + t.Helper() + b, err := bson.Marshal(bson.D{ + {Key: "$ID", Value: bson.Binary{Subtype: 0, Data: uuidToBlob(v1UnitID(1))}}, + {Key: "$Type", Value: "Test$Unit"}, + {Key: "Name", Value: name}, + }) + if err != nil { + t.Fatal(err) + } + return b +} + +// writeInTransaction runs Begin → WriteUnit → Commit the way codec.Store.SaveUnit +// does, and fails the test if it does not return in time instead of hanging the +// whole package. On timeout the writer is deliberately left open: closing it +// would wait on the connection the stuck transaction holds. +func writeInTransaction(t *testing.T, w *Writer, unitID string, contents []byte) { + t.Helper() + done := make(chan error, 1) + go func() { + wt, err := w.BeginWriteTransaction() + if err != nil { + done <- err + return + } + if err := wt.WriteUnit(unitID, contents); err != nil { + _ = wt.Rollback() + done <- err + return + } + done <- wt.Commit() + }() + select { + case err := <-done: + if err != nil { + t.Fatalf("write in transaction: %v", err) + } + case <-time.After(10 * time.Second): + t.Fatal("WriteTransaction.WriteUnit on an MPR v1 project did not return within 10s: " + + "the reconcile read waits for the pool's only connection, which the transaction holds") + } +} + +func TestWriteTransactionWriteUnitV1DoesNotDeadlock(t *testing.T) { + id := v1UnitID(1) + path := newV1ProjectFile(t, id, v1TestUnit(t, "Before")) + w, err := NewWriter(path) + if err != nil { + t.Fatal(err) + } + + writeInTransaction(t, w, id, v1TestUnit(t, "After")) + got, err := w.reader.GetRawUnitBytes(id) + if err != nil { + t.Fatal(err) + } + if !bytes.Contains(got, []byte("After")) { + t.Fatalf("stored contents were not updated: %q", got) + } + + // The unchanged write is elided, which takes the same reconcile read. + writeInTransaction(t, w, id, v1TestUnit(t, "After")) + if err := w.Close(); err != nil { + t.Fatal(err) + } +} diff --git a/modelsdk/mpr/writer_version_shape_1771_type_test.go b/modelsdk/mpr/writer_version_shape_1771_type_test.go index 21a21ae5b0..da3b8fc932 100644 --- a/modelsdk/mpr/writer_version_shape_1771_type_test.go +++ b/modelsdk/mpr/writer_version_shape_1771_type_test.go @@ -169,6 +169,9 @@ func TestNormalize_RefusesWhatAnOlderUserTaskCannotHold(t *testing.T) { withBoundary := userTaskDoc("Single", false, xpathUserTargeting("[Name = 'admin']")) withBoundary[indexOf(withBoundary, "BoundaryEvents")].Value = bson.A{int32(2), bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$InterruptingTimerBoundaryEvent"}, {Key: "Caption", Value: "Late"}}} + withTimerBoundary := userTaskDoc("Single", false, xpathUserTargeting("[Name = 'admin']")) + withTimerBoundary[indexOf(withTimerBoundary, "BoundaryEvents")].Value = bson.A{int32(2), + bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$TimerBoundaryEvent"}, {Key: "Caption", Value: "Late"}}} for _, tt := range []struct { name string v *version.ProjectVersion @@ -198,12 +201,21 @@ func TestNormalize_RefusesWhatAnOlderUserTaskCannotHold(t *testing.T) { {pv(10, 2, 0), awaiting}, {pv(10, 0, 0), withCriteria("Workflows$MajorityCompletionCriteria", bson.E{Key: "CompletionType", Value: "Absolute"})}, {pv(10, 3, 0), withCriteria("Workflows$MicroflowCompletionCriteria", bson.E{Key: "Microflow", Value: "M.Decide"})}, - {pv(10, 14, 0), withBoundary}, + {pv(10, 14, 0), withTimerBoundary}, + {pv(10, 20, 0), withBoundary}, } { if _, err := normalizeShape(t, tt.v, workflowWithUserTasks(tt.doc)); err != nil { t.Errorf("control %s: refused: %v", vlabel(tt.v), err) } } + // A user task holds boundary events from 10.14, but the (Non)Interrupting + // timer types mxcli writes are 10.20 types; 10.14–10.19 store a + // Workflows$TimerBoundaryEvent (mendixmodelsdk 4.117.0). The type floor + // refuses the newer type there (ako/mxcli#1774). + if _, err := normalizeShape(t, pv(10, 14, 0), workflowWithUserTasks(withBoundary)); err == nil || + !strings.Contains(err.Error(), "Workflows$InterruptingTimerBoundaryEvent (Mendix 10.20.0+)") { + t.Errorf("an InterruptingTimerBoundaryEvent on 10.14 was not refused naming the type: %v", err) + } // The empty list the codec writes on every task is dropped below 10.14. d, err := normalizeShape(t, pv(10, 13, 0), workflowWithUserTasks(userTaskDoc("Single", false, xpathUserTargeting("[Name = 'admin']")))) if err != nil { diff --git a/modelsdk/mprread/generic.go b/modelsdk/mprread/generic.go index 588022227c..fa643e0843 100644 --- a/modelsdk/mprread/generic.go +++ b/modelsdk/mprread/generic.go @@ -112,6 +112,46 @@ func ListUnitsInContainer[T element.Element](r *mmpr.Reader, containerID model.I return decodeUnits[T](refs, typeName, nil) } +// UnitByID decodes the unit of type T whose ID is id, reading that unit only. +// ok is false when there is no such unit or it is not a T. +func UnitByID[T element.Element](r *mmpr.Reader, id model.ID) (unit Unit[T], ok bool, err error) { + typeName := bsonTypeName[T]() + ref, found, err := r.UnitByID(string(id)) + if err != nil || !found || ref.Type != typeName { + return Unit[T]{}, false, err + } + units, err := decodeUnits[T]([]mmpr.UnitRef{ref}, typeName, nil) + if err != nil || len(units) == 0 { + return Unit[T]{}, false, err + } + return units[0], true, nil +} + +// ListUnitsNamed decodes the units of type T whose top-level Name is name. +// The names come from the reader's unit index, so on MPR v2 only the matching +// units are opened, where filtering the listing on the stored Name opened +// every unit of the type to read it (ako/mxcli#1791). +func ListUnitsNamed[T element.Element](r *mmpr.Reader, name string) ([]Unit[T], error) { + hs, err := r.ListUnitHeaders(bsonTypeName[T]()) + if err != nil { + return nil, err + } + var out []Unit[T] + for _, h := range hs { + if h.Name != name { + continue + } + u, ok, err := UnitByID[T](r, model.ID(h.ID)) + if err != nil { + return nil, err + } + if ok { + out = append(out, u) + } + } + return out, nil +} + // UnitHeader is what a name listing needs of a unit, read without decoding it. type UnitHeader struct { ID model.ID @@ -124,27 +164,17 @@ type UnitHeader struct { // type T without decoding any of them: for callers that build a name set or an // ID-to-name map and would otherwise decode the whole flow of every document. func ListUnitHeaders[T element.Element](r *mmpr.Reader) ([]UnitHeader, error) { - typeName := bsonTypeName[T]() - refs, err := r.ListUnitsByType(typeName) + hs, err := r.ListUnitHeaders(bsonTypeName[T]()) if err != nil { return nil, err } - out := make([]UnitHeader, 0, len(refs)) - for _, ref := range refs { - if ref.Type != typeName { - continue - } - raw := bson.Raw(ref.Contents) - if err := raw.Validate(); err != nil { - return nil, fmt.Errorf("decode unit %s: %w", ref.ID, err) - } - name, _ := raw.Lookup("Name").StringValueOK() - excluded, _ := raw.Lookup("Excluded").BooleanOK() + out := make([]UnitHeader, 0, len(hs)) + for _, h := range hs { out = append(out, UnitHeader{ - ID: model.ID(codec.PeekID(raw)), - ContainerID: model.ID(ref.ContainerID), - Name: name, - Excluded: excluded, + ID: model.ID(h.ID), + ContainerID: model.ID(h.ContainerID), + Name: h.Name, + Excluded: h.Excluded, }) } return out, nil diff --git a/sdk/javaactions/move_test.go b/sdk/javaactions/move_test.go new file mode 100644 index 0000000000..d5bb2fa2a7 --- /dev/null +++ b/sdk/javaactions/move_test.go @@ -0,0 +1,58 @@ +// SPDX-License-Identifier: Apache-2.0 + +package javaactions + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// MOVE JAVA ACTION A.X TO B left javasource/a/actions/X.java behind. mxbuild +// generates the stub where the model now puts the action, and that is the class +// the app calls: measured on 11.13.0, the build succeeded, jamb/actions/JA_Hello +// threw "Java action was not implemented", and the author's code compiled only as +// an orphaned jama.actions class nothing calls. +// +// The goldens are measured, not written: JA_Moved.before is the source moved to +// javasource/jamb/actions/ with nothing else changed — hand-edited first to add +// an import, extra code, and user code that mentions jama.actions in a comment +// and a string — and JA_Moved is the same file after mxbuild 11.13.0 ran on the +// project in place. mxbuild changed exactly one line, the package declaration, +// and kept everything the author wrote, CRLF line endings included. +func TestMoveSourceMatchesMxbuild(t *testing.T) { + read := func(name string) string { + b, err := os.ReadFile(filepath.Join("testdata", "mxbuild", name)) + if err != nil { + t.Fatal(err) + } + return string(b) + } + before := read("JA_Moved.before.java.golden") + want := read("JA_Moved.java.golden") + + got := MoveSource(before, "JamA", "JamB") + if got != want { + gl, wl := strings.Split(got, "\n"), strings.Split(want, "\n") + for i := 0; i < len(gl) && i < len(wl); i++ { + if gl[i] != wl[i] { + t.Fatalf("line %d differs from mxbuild's regeneration:\n got: %q\nwant: %q", i+1, gl[i], wl[i]) + } + } + t.Fatalf("length differs from mxbuild's regeneration: %d lines, want %d", len(gl), len(wl)) + } +} + +// A move within one module directory changes nothing, and the package match is +// the whole module name: moving out of "Sales" must not touch "salesextra". +func TestMoveSourceRewritesOnlyThePackageItNames(t *testing.T) { + src := "package salesextra.actions;\n\npublic class X {}\n" + if got := MoveSource(src, "Sales", "Billing"); got != src { + t.Errorf("rewrote another module's package:\n%s", got) + } + same := "package sales.actions;\n" + if got := MoveSource(same, "Sales", "SALES"); got != same { + t.Errorf("a move to the same lowercased directory rewrote the source: %q", got) + } +} diff --git a/sdk/javaactions/rename.go b/sdk/javaactions/rename.go index 73081dc116..5554b7f9af 100644 --- a/sdk/javaactions/rename.go +++ b/sdk/javaactions/rename.go @@ -28,14 +28,37 @@ func RenameSource(source, oldName, newName string) string { } replacements := []string{"${1}" + newName, "${1}" + newName + "${2}", `${1}"` + newName + `"${2}`} - rewrite := func(generated string) string { + return rewriteGenerated(source, func(generated string) string { for i, re := range patterns { generated = re.ReplaceAllString(generated, replacements[i]) } return generated + }) +} + +// MoveSource rewrites a Java action's source for a move from fromModule to +// toModule the way mxbuild regenerates it in the new module: the generated +// `package .actions;` line, which names the lowercased module, and +// nothing else. No other generated line names the action's own module — a +// proxy type names its entity's module, which did not move — and the author's +// sections are left byte-for-byte, a mention of the old package included. +// +// Measured against mxbuild 11.13.0 (testdata/mxbuild/JA_Moved*): run in place on +// a source moved to the new module's directory, it changed exactly that line. +func MoveSource(source, fromModule, toModule string) string { + from, to := strings.ToLower(fromModule), strings.ToLower(toModule) + if from == to || from == "" { + return source } + pkg := regexp.MustCompile(`(\bpackage\s+)` + regexp.QuoteMeta(from) + `(\.actions\s*;)`) + return rewriteGenerated(source, func(generated string) string { + return pkg.ReplaceAllString(generated, "${1}"+to+"${2}") + }) +} - // Rewrite only the generated stretches between the author's sections. +// rewriteGenerated applies rewrite to the generated stretches of a Java action's +// source, between the author's sections, and leaves those sections as written. +func rewriteGenerated(source string, rewrite func(string) string) string { var b strings.Builder rest := source for { diff --git a/sdk/javaactions/testdata/mxbuild/JA_Moved.before.java.golden b/sdk/javaactions/testdata/mxbuild/JA_Moved.before.java.golden new file mode 100644 index 0000000000..5f41f1573d --- /dev/null +++ b/sdk/javaactions/testdata/mxbuild/JA_Moved.before.java.golden @@ -0,0 +1,47 @@ +// This file was generated by Mendix Studio Pro. +// +// WARNING: Only the following code will be retained when actions are regenerated: +// - the import list +// - the code between BEGIN USER CODE and END USER CODE +// - the code between BEGIN EXTRA CODE and END EXTRA CODE +// Other code you write will be lost the next time you deploy the project. +// Special characters, e.g., é, ö, à, etc. are supported in comments. + +package jama.actions; + +import com.mendix.systemwideinterfaces.core.IContext; +import com.mendix.systemwideinterfaces.core.UserAction; +import java.util.List; + +public class JA_Hello extends UserAction +{ + public JA_Hello(IContext context) + { + super(context); + } + + @java.lang.Override + public java.lang.String executeAction() throws Exception + { + // BEGIN USER CODE + + // was in package jama.actions before the move + return greeting("jama.actions"); + + // END USER CODE + } + + /** + * Returns a string representation of this action + * @return a string representation of this action + */ + @java.lang.Override + public java.lang.String toString() + { + return "JA_Hello"; + } + + // BEGIN EXTRA CODE + private static java.lang.String greeting(java.lang.String pkg) { return "hi from " + pkg + " " + List.of(1).size(); } + // END EXTRA CODE +} diff --git a/sdk/javaactions/testdata/mxbuild/JA_Moved.java.golden b/sdk/javaactions/testdata/mxbuild/JA_Moved.java.golden new file mode 100644 index 0000000000..bfbd5c1178 --- /dev/null +++ b/sdk/javaactions/testdata/mxbuild/JA_Moved.java.golden @@ -0,0 +1,47 @@ +// This file was generated by Mendix Studio Pro. +// +// WARNING: Only the following code will be retained when actions are regenerated: +// - the import list +// - the code between BEGIN USER CODE and END USER CODE +// - the code between BEGIN EXTRA CODE and END EXTRA CODE +// Other code you write will be lost the next time you deploy the project. +// Special characters, e.g., é, ö, à, etc. are supported in comments. + +package jamb.actions; + +import com.mendix.systemwideinterfaces.core.IContext; +import com.mendix.systemwideinterfaces.core.UserAction; +import java.util.List; + +public class JA_Hello extends UserAction +{ + public JA_Hello(IContext context) + { + super(context); + } + + @java.lang.Override + public java.lang.String executeAction() throws Exception + { + // BEGIN USER CODE + + // was in package jama.actions before the move + return greeting("jama.actions"); + + // END USER CODE + } + + /** + * Returns a string representation of this action + * @return a string representation of this action + */ + @java.lang.Override + public java.lang.String toString() + { + return "JA_Hello"; + } + + // BEGIN EXTRA CODE + private static java.lang.String greeting(java.lang.String pkg) { return "hi from " + pkg + " " + List.of(1).size(); } + // END EXTRA CODE +} diff --git a/sdk/versions/mendix-10.yaml b/sdk/versions/mendix-10.yaml index 96a25823d4..bcfdf63ad2 100644 --- a/sdk/versions/mendix-10.yaml +++ b/sdk/versions/mendix-10.yaml @@ -227,6 +227,26 @@ features: min_version: "10.3.0" mdl: "MULTI USER TASK … DECIDE BY MICROFLOW M.Decide" notes: "MicroflowCompletionCriteria introduced in 10.3.0 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." + wait_for_notification: + min_version: "10.0.0" + mdl: "WAIT FOR NOTIFICATION Name;" + notes: "Workflows$WaitForNotificationActivity introduced in 10.0.0 (mendixmodelsdk 4.117.0, the type's versionInfo). The write refuses the type below it (canon.UndeclaredTypeError, ako/mxcli#1774)." + wait_for_timer: + min_version: "10.6.0" + mdl: "WAIT FOR TIMER Name addHours([%CurrentDateTime%], 1);" + notes: "Workflows$WaitForTimerActivity introduced in 10.6.0 (mendixmodelsdk 4.117.0; ako/mxcli#1774)." + timer_boundary_event: + min_version: "10.14.0" + mdl: "BOUNDARY EVENT TIMER addDays([%CurrentDateTime%], 1) { … }" + notes: "Workflows$TimerBoundaryEvent and the boundaryEvents of an activity introduced in 10.14.0 (mendixmodelsdk 4.117.0; ako/mxcli#1774). From 11.0 the bare form is MDL-WF07: no 11.x runtime has the class." + interrupting_timer_boundary_event: + min_version: "10.20.0" + mdl: "BOUNDARY EVENT [NON] INTERRUPTING TIMER addDays([%CurrentDateTime%], 1) { … }" + notes: "Workflows$InterruptingTimerBoundaryEvent and NonInterruptingTimerBoundaryEvent introduced in 10.20.0; 10.14–10.19 store a Workflows$TimerBoundaryEvent (mendixmodelsdk 4.117.0; ako/mxcli#1774)." + workflow_definition_selection: + min_version: "10.0.0" + mdl: "LOCK WORKFLOW Module.Workflow; UNLOCK WORKFLOW $Workflow;" + notes: "Lock/UnlockWorkflowAction.workflowSelection and the Workflows$WorkflowDefinition(Name|Object)Selection it holds introduced in 10.0.0; 9.21–9.24 store the workflow as the action's own Workflow (mendixmodelsdk 4.117.0; ako/mxcli#1774)." event_handlers: min_version: "10.7.0" mdl: "CREATE WORKFLOW … ON WORKFLOW EVENTS (…) MICROFLOW Module.Handler AS '…'" diff --git a/sdk/versions/mendix-11.yaml b/sdk/versions/mendix-11.yaml index ef0935f85e..2a551c8b60 100644 --- a/sdk/versions/mendix-11.yaml +++ b/sdk/versions/mendix-11.yaml @@ -268,6 +268,26 @@ features: min_version: "11.9.0" mdl: "CALL AGENT MICROFLOW Module.InvokeAgent WITH (Param = '$WorkflowContext')" notes: "Workflows$AIAgentTaskActivity, introduced when Mendix 11.9 split MicroflowBasedActivity into CallMicroflowActivity + AIAgentTaskActivity. Absent from the 11.6 mxbuild, present from 11.10." + wait_for_notification: + min_version: "10.0.0" + mdl: "WAIT FOR NOTIFICATION Name;" + notes: "Workflows$WaitForNotificationActivity introduced in 10.0.0 (mendixmodelsdk 4.117.0, the type's versionInfo). The write refuses the type below it (canon.UndeclaredTypeError, ako/mxcli#1774)." + wait_for_timer: + min_version: "10.6.0" + mdl: "WAIT FOR TIMER Name addHours([%CurrentDateTime%], 1);" + notes: "Workflows$WaitForTimerActivity introduced in 10.6.0 (mendixmodelsdk 4.117.0; ako/mxcli#1774)." + timer_boundary_event: + min_version: "10.14.0" + mdl: "BOUNDARY EVENT TIMER addDays([%CurrentDateTime%], 1) { … }" + notes: "Workflows$TimerBoundaryEvent and the boundaryEvents of an activity introduced in 10.14.0 (mendixmodelsdk 4.117.0; ako/mxcli#1774). From 11.0 the bare form is MDL-WF07: no 11.x runtime has the class." + interrupting_timer_boundary_event: + min_version: "10.20.0" + mdl: "BOUNDARY EVENT [NON] INTERRUPTING TIMER addDays([%CurrentDateTime%], 1) { … }" + notes: "Workflows$InterruptingTimerBoundaryEvent and NonInterruptingTimerBoundaryEvent introduced in 10.20.0; 10.14–10.19 store a Workflows$TimerBoundaryEvent (mendixmodelsdk 4.117.0; ako/mxcli#1774)." + workflow_definition_selection: + min_version: "10.0.0" + mdl: "LOCK WORKFLOW Module.Workflow; UNLOCK WORKFLOW $Workflow;" + notes: "Lock/UnlockWorkflowAction.workflowSelection and the Workflows$WorkflowDefinition(Name|Object)Selection it holds introduced in 10.0.0; 9.21–9.24 store the workflow as the action's own Workflow (mendixmodelsdk 4.117.0; ako/mxcli#1774)." event_handlers: min_version: "10.7.0" mdl: "CREATE WORKFLOW … ON WORKFLOW EVENTS (…) MICROFLOW Module.Handler AS '…'"