diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-30-debug-break-reports-set-runtime-refused-missing-condition.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-30-debug-break-reports-set-runtime-refused-missing-condition.json new file mode 100644 index 0000000000..de38672b05 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-30-debug-break-reports-set-runtime-refused-missing-condition.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli/docker", "date": "2026-10-30", "symptom": "ako/mxcli#1782 (mendixlabs/mxcli#1676): \"`mxcli debug break` prints \\\"Breakpoint set\\\" but the runtime never registers the breakpoint\" — the microflow runs to the end; a logging proxy showed add_breakpoint sent without `condition` and answered {\"message\":\"Missing property: condition\",\"status\":2}. Same session: `debug inspect ` printed {} for a String variable, `debug continue` after a step printed \"Continued.\" while the flow stayed paused until `continue --all`, and some debug calls seemed to hang while a flow was paused", "cause": "The /debugger/ endpoint answers every request with HTTP 200 and reports the outcome in the envelope: {\"result\":…,\"status\":0} on success, {\"result\":{},\"message\":…,\"status\":2} on a refusal (every reply measured on 11.13 and 11.15). DebuggerClient.post checked only the HTTP status, so every refusal came back as success with an empty result — and three malformed requests stayed invisible behind it: add_breakpoint omitted `condition` when empty (required: \"Missing property: condition\"), `continue` sent {} (required debug_id: \"Missing property: debug_id\"), and `inspect` sent a primitive to get_object, which takes objects only (\"The variable 'Greeting' is not a Mendix object.\"). The 'hang' is poll_events, a long poll that returns after ~25 s when no event is pending, called by every paused/step/inspect/continue that auto-resolves the flow", "file": "`cmd/mxcli/docker/debugger.go` (`post`, `DebuggerError`, `AddBreakpoint`, `Continue`/`ContinueAll`); `cmd/mxcli/cmd_debug.go` (`continue --flow`, inspect of a primitive, SilenceUsage in RunE); `cmd/mxcli/debug_resolve.go` (`pausedVariable`)", "insight": "An API that puts the outcome in the body turns a status-code-only client into one where every refusal is a success, and then each malformed request it sends is invisible: one missing envelope check hid three protocol bugs. Surface the envelope first; the other bugs then fail on their own (the existing continue test went red the moment the stub answered like the runtime). The wrong turn is the proposal's protocol table (`condition?`, `continue {}`): it was never read against the envelope. Probe the runtime directly — a 30-line client posting each action through a logging proxy — and keep the stub answering the runtime's way (status, plus a 'Missing property' refusal per required param); the old stub replied {\"result\":{}} to everything, so the tests passed against a protocol that does not exist. Not fixed, both measured: poll_events consumes events, and an event delivered to an abandoned poll (client timeout, a `timeout` wrapper) is lost — a short poll timeout is NOT a safe fix for the 25 s wait (control: with no poll in flight the same pause's event arrives); and queued paused_microflow events outlive the pause (one for a finished flow came back minutes later). Controls: stubbing the status guard fails 17 tests with 'the client reported success'; the pre-fix binary prints 'Breakpoint set' and exits 0 for a condition the runtime refuses", "refs": ["ako/mxcli#1782", "mendixlabs/mxcli#1676"]} diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-30-playwright-check-mxdevtools-502-host-name-behind-proxy.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-30-playwright-check-mxdevtools-502-host-name-behind-proxy.json new file mode 100644 index 0000000000..5b1dc1d809 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-30-playwright-check-mxdevtools-502-host-name-behind-proxy.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-10-30", "symptom": "`mxcli playwright check http://frontend.local:8180/ -p app.mpr --role R` (frontend.local an /etc/hosts name for 127.0.0.1 and the app's configured root URL, app started with `run --local --detach`) printed a rendered page (right title, text=2021) with `console-errors=2`, two `ERR WebSocket connection to 'ws://frontend.local:8180/mxdevtools/' failed: Error during WebSocket handshake: Unexpected response code: 502` and `FAIL 2 console error(s)`, exit 1; the same check on http://127.0.0.1:8180/ passed with 0 console errors", "cause": "Not the runtime and not run --local: the runtime answers the /mxdevtools/ handshake 101 on the configured host name (curl --noproxy '*' with Host and Origin vm:18180, 11.15.0), and run --local puts nothing between browser and runtime. The 502 is the environment's HTTPS proxy. Chromium on Linux takes its proxy from the environment and, for a WebSocket, prefers the HTTPS proxy even for ws:// (RFC 6455 4.1.3). With only https_proxy set, every http:// load of the app went direct, while ws://frontend.local:8180/mxdevtools/ went to the proxy because no_proxy listed localhost and 127.0.0.1 but not the app's own name; the Claude Code agent proxy cannot reach this machine's loopback and answered `502 upstream request failed` (curl -p -x $HTTPS_PROXY: 200 Connection Established, then 502)", "file": "cmd/mxcli/docker/browser_proxy.go (directRouteEnv), called where mxcli starts a browser at the app: cmd/mxcli/docker/pagecheck.go (ProbePages), screenshot_login.go (LoginAndSaveStorage, SignOut), screenshot.go (CaptureScreenshot)", "fix": "the browser mxcli starts gets every host among its URLs that resolves only to this machine (loopback, or an address of one of its interfaces) appended to no_proxy and NO_PROXY, when the environment configures a proxy; the proxy settings themselves are untouched, so a remote --hub URL still uses it. No console error is ignored: the channel connects, and a genuine console error still fails the page", "insight": "The report called /mxdevtools/ mxcli's own channel and offered two fixes (ignore it in the verdict, or make run --local serve it on the configured host); both assumed the server answered 502 on that host. One curl straight at the runtime with the configured Host (101) and one through the proxy (502 'upstream request failed') settled who answers, and three standalone Playwright runs of the same page settled the route: env as is -> 502; the name appended to no_proxy -> 0 errors, 2 frames received; https_proxy unset -> 0 errors. The tell was the shape of the failure: the page loads, only the WebSocket fails, and only on a name not in no_proxy. Ignoring the channel in the classifier would have passed the page while the channel stayed broken in that browser, and an app's own ws:// to its host would still have failed the check. Repro without /etc/hosts edits: a Claude Code container maps `vm` and `runsc` to 127.0.0.1 and neither is in no_proxy, so `ApplicationRootUrl http://vm:/` reproduces the report byte for byte. A unit test needs no host name either: an httptest page on the machine's own non-loopback address (Chromium bypasses only loopback by itself) plus a fake CONNECT proxy that answers the tunnelled request 502 reproduces the message verbatim. Scope: env proxies are what Chromium reads on Linux; macOS and Windows Chromium read the OS proxy settings, which this does not touch", "test": "cmd/mxcli/docker/browser_proxy_test.go (TestDirectRouteEnv_*: loopback name and interface address added; remote, split, unresolvable, already-listed, no-proxy and unparsable left alone; both no_proxy spellings; IPv6 bracketed); cmd/mxcli/docker/pagecheck_proxy_test.go (TestProbeReachesALocalAppsDevChannelDirectly: the real probe against a fake dev-mode page behind a refusing proxy, with an in-test control that reproduces the 502; skips without Node/Playwright). Control: directRouteEnv stubbed to return env fails all of them, the probe test with `Unexpected response code: 502`. Runtime A/B on 11.15.0 (mdl-examples/bug-tests/pagecheck-1800-host-name-behind-proxy.mdl, root URL http://vm:18180/): control binary / FAIL 2 console errors (2x 502), /p/brokenprobe 3; fixed / OK console-errors=0, /p/brokenprobe FAIL on its genuine error only", "refs": ["ako/mxcli#1800", "ako/mxcli#1216", "mdl-examples/bug-tests/pagecheck-1800-host-name-behind-proxy.mdl"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-30-self-reference-retrieve-typed-as-list.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-30-self-reference-retrieve-typed-as-list.json new file mode 100644 index 0000000000..ea5acbc782 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-30-self-reference-retrieve-typed-as-list.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-10-30","symptom":"ako/mxcli#1793: for a self-referencing Reference M.Order_Parent (M.Order to M.Order), `retrieve $P from $Order/M.Order_Parent; $n = count $P;` passed `check -p --references` and exec (\"Created microflow\"), then mx check failed CE0013 \"Input variable 'P' must be of type 'List'.\" The full build typed $P `List of M.Order` unless an object use followed; the splice path (#1777) already typed it M.Order.","cause":"Three answers to 'which end does this association retrieve start from': addRetrieveAction asked only 'is the start the child (TO) entity', which a self-reference always is, so it read as a reverse traversal; flowBuilder.associationRetrieveType (#1777) special-cased the self-reference as the FROM end; check's associationRetrieveShape returned known=false for from == to and took the nearest generalization link otherwise; validateFlowBody's forwardReferenceTarget kept a self-association a list. And the check judged only a list used as an object, never one object used as a list, so even a forward Reference `count` (CE0013) was silent.","file":"mdl/executor/validate_retrieve_association_list.go (associationRetrieveEnd, associationRetrieveShape, listActivityInput, the MDL-RETRIEVE01 half of validateAssociationRetrieveListUse), mdl/executor/cmd_microflows_builder_actions.go (associationRetrieveEndFrom, addRetrieveAction, associationRetrieveType), mdl/executor/cmd_microflows_builder_validate.go (forwardReferenceTarget), mdl/executor/validate_flow_member_refs.go (scriptGeneralizationChain)","fix":"One rule, associationRetrieveEnd(isFrom, isTo, self): both ends + self-reference = FROM, both ends of two different entities = ambiguous (the build keeps its old TO reading, the splice and the check stay silent), else the end it is. The build, the splice, check --references and validateFlowBody all ask it. check --references now reports MDL-RETRIEVE01 for the one object a Reference followed from its FROM end binds (self-reference or not) taken by an aggregate (CE0013), a list operation but range (CE0097), a loop (CE0100) or a list parameter (CE0117); exec's pre-flight refuses the same. A one-to-one from its TO end is not judged (the build rewrites a list use into a database retrieve). The start's chain is read through the script's declarations first, so a specialization the script creates counts.","test":"cmd_microflows_builder_retrieve_self_reference_test.go (TestBuildFlowGraph_SelfReferenceIsFollowedFromItsFromEnd, TestBuildFlowGraph_AssociationDirectionControls, TestAssociationRetrieveType_AgreesWithTheFullBuild); validate_retrieve_self_reference_test.go (TestAssociationRetrieveObjectUsedAsList: 11 refused shapes, 10 controls; _ScriptDeclaredSpecialization; SelfReferenceSetUsedAsObject; TestCheckSetOnSelfReferenceObject). mdl-examples/bug-tests/1793-self-reference-retrieve-is-the-object.mdl. Control: the fix's source files reverted to main -> $P typed List of Sample.Node, the owner-Both self-reference expanded to a DatabaseRetrieveSource, the check returns no error for every list use. Knock-on: bug-tests 1323 (item 2, set over a self-reference Reference) and 1100 (SUB_LoopAssoc looped over a FROM-end Reference: exec as written is CE0100 on 11.13.0) held shapes the rule now refuses; both rewritten to the measured-valid shape.","insight":"Measure every activity kind before refusing, not one: over the same object on 11.13.0 every aggregate is CE0013, every list operation CE0097 EXCEPT range (Microflows$ListRange, 0 errors), a loop CE0100, a list parameter CE0117, and add/remove/clear/set (Change list) 0 errors — mx check does not type a Change list's variable or a range's. That last gap is why #1323's bug-test claimed 'Mendix types a self-association retrieve as a list': its `set` was written as Change list Replace and passed; mx check's CE0100 text names the variable's type outright (\"'P' is of type M.Order\"), which is the measurement to trust. When a fix adds a helper that answers a question another site answers too, grep for every site (here four) and route them through one function with a test that the two builder paths agree.","ce":["CE0013","CE0097","CE0100","CE0117","CE7247"],"rules":["MDL-RETRIEVE01","MDL-RETRIEVE03","MDL-SET01"],"refs":["ako/mxcli#1793","ako/mxcli#1777","ako/mxcli#1323","mendixlabs/mxcli#1516"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-30-workflow-microflow-actions-gated-at-their-type-floor.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-30-workflow-microflow-actions-gated-at-their-type-floor.json new file mode 100644 index 0000000000..c1a8634eb6 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-30-workflow-microflow-actions-gated-at-their-type-floor.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-10-30", "symptom": "`mxcli check -p` passed a script with workflow microflow actions the project's Mendix version does not have, and exec then refused it at write time, part-way: on 9.24.30 `this Mendix 9.24.30 project cannot store Microflows$NotifyWorkflowAction (Mendix 10.0.0+)` after five statements (rolled back), and with --continue-on-error the script ran on without MF_Notify, MF_GetWorkflows, MF_ActivityRecords and an `alter microflow` that inserted `get workflows`", "cause": "#1774 added the write-time type floor (canon.UndeclaredTypeError) for every type, but check-time MDL-VERSION01 gates only for the workflow activity and event types. No gate named a workflow microflow action, and statementVersionGates had no case for `alter microflow|nanoflow` at all, so nothing a splice inserted was ever version-gated (send email and a named lock included)", "file": "`mdl/executor/version_gates.go` (typeGate, workflowActionGates, microflowBodyGates, nanoflowBodyGates, the AlterFlowStmt case); `mdl/executor/cmd_alter_flow.go` (execAlterFlow calls checkVersionGates); `sdk/versions/mendix-9|10|11.yaml` (workflows.workflow_microflow_actions 9.0.2, workflow_operations 9.2.0, workflow_operation_pause_continue 9.3.0, get_workflow_data 9.10.0, workflow_operation_retry_unpause 9.12.0, notify_workflow 10.0.0, get_workflows 10.4.0, get_workflow_activity_records 10.4.0)", "insight": "**A write-time floor with no check-time gate is a check/exec disagreement, and the sweep is mechanical**: list the $Types the statement's converter writes (mdl/backend/modelsdk/microflow_workflow_write.go: workflowMicroflowActionToGen, workflowOperationToGen), look each up in modelsdk/canon/mendixmodelsdk_type_versions.json, and gate every floor above the lowest supported version. That is 9.0.0 (sdk/versions supported_range), and it is real: the CDN lists mxbuild 9.0.0 and 9.0.1 builds (`curl 'https://cdn.mendix.com/?list-type=2&prefix=runtime/mxbuild-9.0.'`), so the 9.0.2 actions (call workflow, set task outcome, open user task, open workflow) are gated too. A workflow operation writes two types, the action (9.2) and the operation (abort/restart 9.2, pause/continue 9.3, retry/unpause 9.12): the floor is the later. **Keep the floor in sdk/versions and pin it, rather than reading canon at check time**: every gate reads the registry, `show features` lists it, and the message's version comes from it; a gate that read canon would be a second mechanism, and the yaml row could still drift from it. TestTypeGates_FloorIsTheTypeIntroduction pins each type gate's row (the #1774 ones too) to canon.TypeIntroduced of the types it names, at the exact boundary (available at the floor, not just below it). The migration reference does not read the yaml, so `check-migration-reference` is unaffected. **Lock and unlock (9.21) need no gate of their own**: every lock or unlock that check and exec accept names its workflow, and that selection is the 10.0 workflowDefinitionSelectionGate. **`alter microflow` had no version gate at all**: a body gate added for `create microflow` must reach the splice, so share one function (microflowBodyGates) and apply it to every operation's fragment; execAlterFlow calls it too, for exec without the pre-flight. `create or modify` needs nothing new: it is a CreateMicroflowStmt, gated on its whole declared body, and its splices do not go through execAlterFlow. **The mxbuild class-name oracle agrees**: `strings -a Mendix*.dll | tr -c 'A-Za-z0-9_\\n' '\\n' | sort -u` on 9.24.30 lacks NotifyWorkflowAction, GetWorkflowsAction and GetWorkflowActivityRecordsAction and has every other action and operation (ResumeOperation too, kept to load older models); 10.24.24 has all. Measured with the bug-test: on 9.24.30 before, check -p passed the script without its 10.0 workflow and exec refused it at write time; after, check -p and exec's pre-flight report MF_Notify (10.0), MF_GetWorkflows, MF_ActivityRecords and the alter (10.4), App.mpr is byte-identical, mx check 0 errors; the 9.0.2-9.12 actions, written with --no-check, load on 9.24 with 0 errors. 10.24.24 before and after: check -p clean, exec writes everything, mx check 0 errors. Control: without the table's gates in microflowBodyGates, the 70 create and 140 alter below-floor cases report nothing; a row moved off its type's introduction fails the pin test; without execAlterFlow's call, exec reports `microflow not found` instead of the refusal. Tests: version_gates_workflow_actions_1787_test.go; mdl-examples/bug-tests/1787-workflow-actions-below-their-version.mdl", "refs": ["#1787", "#1774", "#1771", "#1434"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-30-workflows-basic-floor-below-the-workflow-type.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-30-workflows-basic-floor-below-the-workflow-type.json new file mode 100644 index 0000000000..594a0037c6 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-30-workflows-basic-floor-below-the-workflow-type.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-10-30", "symptom": "On a Mendix 9.0.0 or 9.0.1 project `mxcli check -p` passed `create workflow` (and `create or modify workflow`), and exec then refused it at write time (\"cannot store … Workflows$Workflow (Mendix 9.0.2+)\", or, for a workflow with a parameter, the 9.6–9.18 Workflows$Parameter properties) after the statements before it; `alter workflow`'s hint said \"upgrade your project to Mendix 9.12+\"", "cause": "sdk/versions workflows.basic was typed as 9.0.0 when Workflows$Workflow is 9.0.2 in mendixmodelsdk 4.117.0, and the CreateWorkflowStmt case of statementVersionGates applied only the sub-feature gates; only AlterWorkflowStmt applied basic, with a comment and hint naming 9.12, a third version", "file": "`mdl/executor/version_gates.go` (workflowBasicGate, the CreateWorkflowStmt and AlterWorkflowStmt cases); `sdk/versions/mendix-9|10|11.yaml` (workflows.basic 9.0.2); `mdl/executor/version_gates_workflow_actions_1787_test.go` (pinned in TestTypeGates_FloorIsTheTypeIntroduction)", "insight": "**A `basic` row is a floor like any other, so pin it to the type the statement writes.** It looked exempt because 9.0.0 is the bottom of supported_range and the gate could never fire, which is also why nobody noticed that `create workflow` had no basic gate at all: a gate that cannot fire and a missing gate behave the same until the floor moves. **A hint that names a version is a second copy of the floor**, and this one disagreed with both the row (9.0.0) and the type (9.0.2); checkFeature already puts the registry's version in the message, so a hint should say what to do, not when. Measured on a 9.24.30 app whose _MetaData version was set to 9.0.1.0 (no 9.0.x mxbuild is cached; the gate and the write read only that field): before, check -p passed and exec created the microflow and refused the workflow at write time; after, check -p and the pre-flight report MDL-VERSION01 for create, create or modify and alter workflow, and App.mpr is byte-identical. Control: the row moved back to 9.0.0 fails the pin test (\"available at 9.0.1, where the write refuses [Workflows$Workflow]\") and the gate test; the create case without workflowBasicGate fails the gate test for both create forms. The same stamped run shows how much more is refused below 9.12 that no gate names, all by property floors: `create module` (Projects$ModuleSettings.ProtectedModuleType 9.12), `create entity` (ExportLevel 9.3) and a workflow parameter (Workflows$Parameter.Name 9.18)", "refs": ["#1809", "#1787", "#1774"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-30-write-floors-swept-into-check-time-gates.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-30-write-floors-swept-into-check-time-gates.json new file mode 100644 index 0000000000..59fe7b7c67 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-30-write-floors-swept-into-check-time-gates.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-10-30", "symptom": "`mxcli check -p` passed statements the write then refused part-way through exec (\"this Mendix 9.24.30 project cannot store Microflows$RestOperationCallAction (Mendix 10.1.0+)\"; \"… DatabaseConnector$DatabaseConnection.ConnectionInput (Mendix 10.0.0+)\"; on 10.24.24 \"… Microflows$TransformJsonAction (Mendix 11.4.0+)\"): send rest request, range, call external action, transform … with, find/filter … where, and on the sweep loops, show page arguments, queued calls, database queries and connections, workflow parameters and user tasks, page variables, snippet parameters, navigation and REST client shapes; `create view entity` was gated at 10.18 and refused at write time on 10.18-10.20", "cause": "Gates were added statement by statement from memory of what was new; the write's floors (canon.UndeclaredTypeError for types, canon.StripUndeclaredProperties for a property holding a value) cover everything mxcli writes, so every written type or property newer than 9.0.0 without a gate at or above its floor is a check/exec disagreement. Several existing rows were typed by hand below the write's floor (view_entities 10.18 vs the 10.21 source document) or above it (execute_database_query 10.6, business_events 10.0)", "file": "`mdl/executor/version_gates_flow_actions.go` (flowActionGates, flowGates), `version_gates_workflow_shapes.go` (workflowShapeGates, statementTypeGate), `version_gates_documents.go` (documentTypeGates), `version_gates.go` (typeGate.properties, viewEntityGate, statementCaseGates + documentGates); tests `version_gates_flow_actions_1811_test.go`, `version_gates_workflow_shapes_1811_test.go`, `version_gates_documents_1811_test.go`, `version_gates_coverage_1811_test.go` (TestTypeGates_CoverEveryWrittenType), the property-aware pin in `version_gates_workflow_actions_1787_test.go`; `sdk/versions/mendix-9|10|11.yaml`", "insight": "**Measure the write's floor, don't read it off the type.** A shape's floor is the latest of the types AND the properties it writes a value to, and the properties usually decide: Workflows$Parameter is 9.6 but its Name is 9.18, a user task's types are 9.0.2-9.11 but UserTaskOutcome.Value is 9.19, a database query's type is 9.22 but its QueryType is 10.12, every REST operation's Method is 10.4. The cheap oracle needs no old mxbuild: copy a 9.24.30 app.mpr, `update _MetaData set _ProductVersion='X'` (sqlite3), and `exec --no-check --continue-on-error` a probe script with the unfixed build — the gate and the write read only that field, so each refusal is the write's floor at X (stored 9.24 content rewritten wholesale shows up as artifacts, e.g. Settings$JarDeploymentSettings; create module and create entity are themselves refused below 9.12 and 9.3, so probe in the stored MyFirstModule over System entities). typeGate.properties pins those floors via canon.PropertyDeclared. **Make the sweep a test, not a list**: TestTypeGates_CoverEveryWrittenType scans the write paths for `\"Module$Type\"` literals and modelsdk/gen constructors (resolved through each package's `init…()` SetTypeName) and fails for a written type above 9.0.0 that no type gate names, no broader gate covers at or above its floor, and typeCoverageExceptions does not explain (lowered / unreachable / RESIDUAL); stale entries fail too. Its first run found the view entity gap and five database connector types nothing listed. **Do not read `mx check` 0 errors as the platform accepting a newer property**: on 9.24.30 a database connection with ConnectionInput and a page with Variables both check clean, because mxbuild tolerates undeclared properties (propertyfloor.go); only a newer TYPE in a known property fails to load. The SDK and the platform agreed wherever a type could be measured on 9.24.30 (business events and execute database query load, as their 9.24 floors say; send rest request loads with CE7053 but its 10.1 service cannot exist). Residuals no statement-level gate can decide, left to the write: design-property kinds (theme), external entity source kinds ($metadata), a send rest request argument that is not a path parameter, and module creation (auto-create; below 9.12 the ModuleSettings.ProtectedModuleType property refuses every create module, an out-of-scope property-floor gap, as is create entity's ExportLevel below 9.3). Measured with bug-tests 1811-*.mdl: 9.24.30 before, check -p clean and exec refused the database connection at write time, with --continue-on-error each 10.x statement refused; after, MDL-VERSION01 for each, App.mpr byte-identical, the 9.x-floor statements write and mx check 0 errors; 10.24.24 the same for the 11.x shapes, and the 10.x script writes with mx check 0 errors. Controls: flowGates/workflowShapeGatesOf/documentGates stubbed fail 205/13/25 below-floor cases; a row moved below its floor fails the pin; a new `newElem(\"ChangeDataCapture$Version\")` or a removed exception fails the coverage test; rest_client_basic moved to 10.0 fails it as a broader gate below its types. Found on the way: `execute database query X connection (…)` with no query argument list writes the connection arguments as query parameter mappings (visitor CallArgumentList(0)), not fixed here", "refs": ["#1811", "#1809", "#1787", "#1774"]} diff --git a/.claude/skills/fix-issue/findings/mdl-other/2026-10-30-source-reuse-redescribes-a-whole-module-for-one-entity.json b/.claude/skills/fix-issue/findings/mdl-other/2026-10-30-source-reuse-redescribes-a-whole-module-for-one-entity.json new file mode 100644 index 0000000000..b6839e467a --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-other/2026-10-30-source-reuse-redescribes-a-whole-module-for-one-entity.json @@ -0,0 +1 @@ +{"area":"mdl/catalog","date":"2026-10-30","symptom":"After ako/mxcli#1790, adding one attribute to one entity still made the incremental source rebuild describe every entity of its module and every document that used any of them: 346 of 3,718 documents for an unreferenced entity in a 154-entity module","cause":"The reuse basis was the unit, and an entity is not a unit: it lives in its module's domain model, so the domain model's hash stood for every entity in it, and the dependents query was seeded with all of their names","file":"`mdl/catalog/builder_source_reuse.go` (`entityBasis`, `splitDomainModel`, the parents argument of `planSourceReuse` / `persistSourceBasis`), `mdl/catalog/builder_source.go`","insight":"**Pick the basis by what the describe reads, not by the storage unit.** `describeEntity` reads only the entity's own subtree (attributes, rules, indexes, handlers and access rules are all inside it; associations are not printed), so the subtree hash is a sound basis and the rest of the domain model becomes a parent whose change still re-describes the module. Raw BSON is the right thing to hash: element order is stored order, so hashing the array element bytes is deterministic without a canonical form. Observing *which* documents were described (a nil-by-default hook on the describe func) is what makes the control bite: comparing rows with a full build passes with the coarse basis too, since over-describing is never wrong, only slow","fix":"Hash each entity's subtree (plus its domain model's ID, so a move between modules is a change) under the entity's own ID, hash the domain model without `Entities` under the unit's ID, and make the domain model each entity's parent in the plan. Test: `TestSourceCatalog_IncrementalBuildEqualsFullBuild` asserts the sibling entity is carried over; with `entityBasis` stubbed out it fails naming the sibling"} diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-30-every-refresh-opens-every-unit-file.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-30-every-refresh-opens-every-unit-file.json new file mode 100644 index 0000000000..8ec648f3ef --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-30-every-refresh-opens-every-unit-file.json @@ -0,0 +1 @@ +{"area":"modelsdk/mpr","date":"2026-10-30","symptom":"On MPR v2 every catalog refresh opened every .mxunit (4,753 on a large app) after every save; Windows users measured the virus scanner's per-open cost as the bulk of their wait (77% faster with the project excluded from scanning)","cause":"The reader's only persistent knowledge of a unit was its type and name (the unit index); its bytes were always re-read from the unit's own file, and a refresh needs every unit's bytes","file":"`modelsdk/mpr/reader_unit_store.go` (`unitStore`: lookup / record / flush), `modelsdk/mpr/reader_units.go` (`readMprContents`, `buildUnitCache` row hashes, flush on the outermost `CacheUnitReads` release), `modelsdk/mpr/reader.go` (`Close`)","insight":"**On Windows the cost is the open, not the bytes, so cache the bytes behind a check that does not open the file.** A stat (size + modification time, taken BEFORE the read so a file caught mid-write is stored under the old stat) plus the Unit row's ContentsHash decides a hit; the hash alone is not enough because a plain git checkout can leave it stale, and the stat alone misses a copy with preserved timestamps. Each safeguard has its own test and each test fails with its safeguard removed — the first stat control passed anyway because the changed name had another length and the post-decode size check caught it, and the first hash control 'passed' because stubbing it left a variable unused and the package did not compile: grep for `--- FAIL` and `FAIL` lines and a compile error, not only the former. The store sits in `.mxcli/cache/` with its own `.gitignore` because `.mxcli/` is not ignored in every project","fix":"Persist zstd-compressed unit bytes in `.mxcli/cache/unit-contents.db`, serve a unit from it while size, mtime and row hash match, record misses and flush at the end of a full pass and on Close, drop units the Unit table no longer has. Tests: `TestUnitStore_*` (modelsdk/mpr) and `TestCatalogRefresh_NextProcessOpensOnlyChangedUnits` (executor: 369 files → 0, 1 after one file changes, catalog row-identical to a build that read every file)"} diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-30-mixed-names-writer-branch-vs-older-stored-elements.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-30-mixed-names-writer-branch-vs-older-stored-elements.json new file mode 100644 index 0000000000..f7502b4731 --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-30-mixed-names-writer-branch-vs-older-stored-elements.json @@ -0,0 +1 @@ +{"area":"modelsdk/canon","date":"2026-10-30","symptom":"On a Mendix 11.14.0 project written only by mxcli, `mxcli fix property-sets` reported seven MIXED names and filled none: `MIXED ODataPublish$EntitySet.ReadMode: on 42 of 60 element(s); without it: … [ODataPublish$PublishedODataService2 F1AdminApi]`, `MIXED ODataPublish$PublishedMicroflow.Parameters: on 2 of 5 element(s); only on: …`, and on one domain model `DomainModels$DateTimeAttributeType.LocalizeDate` (15 of 16) plus `DomainModels$NoGeneralization.HasOwnerAttr` / HasChangedByAttr / HasChangedDateAttr / HasCreatedDateAttr (81 of 84). mx check was clean (ako/mxcli#1805)","cause":"Two different causes behind one report. (1) A current writer branch: publishedEntitySetToGen wrote each of the four modes only `if es.XMode != \"\"`, and publishedMicroflowToGen wrote Parameters only `if len(pm.Parameters) > 0`; the doctype gate never took those branches (10-odata states all four modes on every set and its one action has parameters). (2) Stored state: no writer of today omits the domain-model keys (entityToGen writes all four flags and LocalizeDate since #743/#1373), but an older mxcli wrote only the true flags and no LocalizeDate, and every later domain-model write rebuilds only the entity it names and passes the siblings through as stored bytes. The table had no row for any of the ten keys, because it is measured from what today's writers leave out, so neither the next write nor the repair could fill them","file":"`mdl/backend/modelsdk/odata_write.go` (publishedEntitySetToGen writes all four modes; publishedMicroflowToGen always writes Parameters, RegisterMandatoryPropertyList for the empty [3]); `modelsdk/canon/studiopro_property_defaults.json` (10 rows, since 10.24.24); tests `mdl/backend/modelsdk/issue1805_odata_property_sets_test.go`, `modelsdk/canon/propertyset_issue1805_test.go`, `modelsdk/mpr/property_set_issue1805_test.go`, `cmd/mxcli/cmd_fix_property_sets_issue1805_test.go`; bug-test `mdl-examples/bug-tests/1805-property-sets-mixed-odata-and-domain-model.mdl`","insight":"**Tell a writer bug from stored state by running every write path on a fresh app before reading any code**: one MDL script through create, create or modify, add/modify attribute, association, alter of a sibling, rename and move, plus the OData service with stated and unstated clauses, then `fix property-sets --dry-run`. What reproduces (the OData names, on 10.24.24, 11.13.0 and 11.15.0) is a writer; what does not (the domain-model names) is an element an older build stored, and only a measured table row reaches it — on the next write of its unit (completion at the choke point) or by the repair. **A writer that branches on an optional clause is invisible to the doctype gate** unless a script leaves the clause out: grep the writer for `if x != \"\"` / `if len(x) > 0` around a part, not only for the key name. **Strip-and-convert measures a row no writer leaves out** (#1505's method): strip the key from every element of the type, `fix hashes --repair`, `mx convert -p -s` a copy, and what convert writes back is the row; ReadSource, ChangeNotSupported, [3], false and true were identical on all three versions, and testdata/testapp-views' 11.14.0 Studio Pro entity set carries the same four modes. **Once a table row exists, completion hides the writer bug from every storage-level test at or above its floor**: the census test through the backend passed with the writer fix reverted; only the encoded-document test (codec.Encoder on the converter, no storage) failed. Keep both: the writer fix is what covers 10.4–10.24.23, below the floor. Side effect worth knowing: describe now prints all four modes for a set created without them (as it always did for a Studio Pro-authored one), and that output re-checks clean","refs":["ako/mxcli#1805","ako/mxcli#1505","mendixlabs/mxcli#1373"]} diff --git a/.claude/skills/mendix/debug-microflows/SKILL.md b/.claude/skills/mendix/debug-microflows/SKILL.md index 6587e1c14c..42b8979dfb 100644 --- a/.claude/skills/mendix/debug-microflows/SKILL.md +++ b/.claude/skills/mendix/debug-microflows/SKILL.md @@ -16,9 +16,9 @@ description: "Drive the Mendix runtime's microflow and nanoflow debugger from th | 62-81 | The loop | | 82-101 | Commands | | 102-132 | Nanoflows (client-side) | -| 133-149 | Gotchas | -| 150-157 | Validation checklist | -| 158-160 | Microflow/nanoflow debugger (`mxcli debug`) | +| 133-156 | Gotchas | +| 157-164 | Validation checklist | +| 165-167 | Microflow/nanoflow debugger (`mxcli debug`) | # Debug Microflows — `mxcli debug` @@ -90,9 +90,9 @@ mxcli debug disable -p app.mpr | `mxcli debug unbreak --activity <#n\|caption>` | Clear a breakpoint | | `mxcli debug breaks` | List the breakpoints mxcli has set this session (name → object ID) | | `mxcli debug paused` | Show paused microflows + full state (variables) | -| `mxcli debug inspect [--list] [--flow ]` | Inspect one variable of a paused flow (`--list` for a list variable → `get_list`) | +| `mxcli debug inspect [--list] [--flow ]` | Inspect one variable of a paused flow: an object's members, a primitive's value (String, enumeration, …) (`--list` for a list variable → `get_list`) | | `mxcli debug step [over\|into\|out] [--flow ]` | Advance one step (default `over`) | -| `mxcli debug continue [--all]` | Resume the paused flow (or all with `--all`) | +| `mxcli debug continue [--flow ] [--all]` | Resume the paused flow (or all with `--all`) | Selecting an activity: `--activity '#2'` (the index from `activities`) or a caption substring like `--activity 'Retrieve'` (must match exactly one, case- @@ -146,6 +146,13 @@ name. `--debug-pass` (or `MXCLI_APP_URL` / `MXCLI_ADMIN_PASS` / `MXCLI_DEBUG_PASS`) default to a `run --local` runtime; set them to debug a differently-configured or remote runtime. +6. **A refused request fails with the runtime's reason** — e.g. `the runtime + refused add_breakpoint (status 2): Failed to parse condition: …`. The usual + ones: a bad `--if` expression; an activity or flow the *running* app does not + have yet (rebuild/restart after changing the model); `Invalid session token` + (the runtime restarted, or another debugger such as Studio Pro started a + session — run `mxcli debug enable`). A command that printed success did reach + the runtime and was accepted. ## Validation checklist diff --git a/.claude/skills/mendix/docker-workflow/SKILL.md b/.claude/skills/mendix/docker-workflow/SKILL.md index 82b1b0e839..a12cc9ac30 100644 --- a/.claude/skills/mendix/docker-workflow/SKILL.md +++ b/.claude/skills/mendix/docker-workflow/SKILL.md @@ -17,9 +17,9 @@ description: "Build, run and validate a Mendix app in Docker from inside a devco | 86-113 | Creating an Empty Mendix App | | 114-274 | Step-by-Step Workflow | | 275-305 | Common Workflow: Edit, Rebuild, Test | -| 306-417 | Hot Reload | -| 418-436 | Environment Variables | -| 437-457 | Troubleshooting | +| 306-419 | Hot Reload | +| 420-438 | Environment Variables | +| 439-459 | Troubleshooting | # Docker Build & Run Skill @@ -387,6 +387,8 @@ The Mendix runtime exposes a WebSocket at `ws://localhost:8080/mxdevtools/` (on When `reload_model` is called, a new `set_deployment_id` is pushed to browsers, triggering automatic refresh. This is why you don't need to manually refresh the browser after `docker reload`. +**A 502 on this channel is a proxy, not the runtime.** The runtime serves it on every host name the app is reached by. But a browser in an environment that sets `https_proxy` sends `ws://` to that proxy (Chromium prefers the HTTPS proxy for a WebSocket) unless the host is in `no_proxy`. An app checked by its own name — `http://frontend.local:8180/`, an `/etc/hosts` entry for `127.0.0.1` — then loads directly while `ws://frontend.local:8180/mxdevtools/` fails with `Unexpected response code: 502`, and auto-refresh and styling hot reload never reach that tab. The browser `mxcli playwright check`, `--page-check` and `--screenshot` start adds such a host to `no_proxy` itself (ako/mxcli#1800). A browser or `playwright-cli` session of your own needs the name added to `no_proxy`. + ### When to Use `reload` vs `run` | Scenario | Command | Why | diff --git a/CHANGELOG.md b/CHANGELOG.md index 5313377b9c..eb98810d9c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,23 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **`playwright check` no longer fails a page checked by the app's own host name behind a proxy** (ako/mxcli#1800) — with `https_proxy` set and an app checked as `http://frontend.local:8180/` (an `/etc/hosts` name for `127.0.0.1`), a page that rendered failed on two console errors, `WebSocket connection to 'ws://frontend.local:8180/mxdevtools/' failed: … Unexpected response code: 502`, while the same check on `127.0.0.1` passed. The runtime serves its dev channel on that name; the 502 came from the proxy, because Chromium sends `ws://` to the HTTPS proxy and `no_proxy` listed only the loopback names. The browser that `playwright check`, `run --page-check` and `run --screenshot` start now reaches a host that resolves to this machine directly. No console error is ignored: the channel connects, and a genuine console error still fails the page. +- **Published OData entity sets, OData actions and older domain models have the property names Studio Pro writes** (ako/mxcli#1805) — on a Mendix 11.14.0 project `mxcli fix property-sets` reported seven `MIXED` names and could fill none of them. An entity set published without stating its modes was written without `ReadMode`, `InsertMode`, `UpdateMode` and `DeleteMode`, and an action whose microflow takes no parameters without `Parameters`; they are now written, as the values Mendix already read the missing ones as (read from the database, changes not supported, no parameters), so the service behaves as before and `describe` now prints the four modes. The domain-model names (`LocalizeDate` on a DateTime attribute, the four system-attribute flags on an entity without a generalization) are not left out by any current statement: they belong to entities an older mxcli wrote, which every later write of the domain model kept as they were. Both are now filled from values measured with `mx convert` on 10.24.24, 11.13.0 and 11.15.0, on the next write of the document or by `mxcli fix property-sets`. Run `mxcli fix property-sets` once to complete what an earlier mxcli wrote. +- **A retrieve over a self-reference is the one object Mendix makes it, and a list use of it is refused by `check -p`** (ako/mxcli#1793) — for a Reference from an entity to itself (`M.Order_Parent`), `create microflow` typed `retrieve $P from $Order/M.Order_Parent` as a list of `M.Order` unless `$P` was later used as an object, so `$n = count $P;` passed `check -p` and `exec` and failed `mx check` with CE0013 "Input variable 'P' must be of type 'List'.". Mendix follows a self-reference from its FROM end: a Reference gives the one object, a Reference set the list. The full build, `alter`'s splice and `check -p` now take that one rule, from the entity and from a specialization of it alike, and `check -p` and `exec`'s pre-flight report MDL-RETRIEVE01 for the one object a Reference gives from its FROM entity — a self-reference or not — taken by an aggregate (CE0013), a list operation other than `range` (CE0097), a `loop` (CE0100) or a list parameter (CE0117). `set` on a self-reference's object is refused like `set` on any object (CE7247); the Change list Replace it used to be written as passed `mx check` only because mx check does not type a Change list's variable. Measured on 11.13.0: before, `count $P` passed `check -p` and `exec` and was CE0013; now `check -p` refuses it, and the object uses and the Reference set's list uses build at 0 errors. +- **Pages, navigation, module settings, scheduled events, database connections, business events, REST clients and view entities are refused by `check -p` below what their write needs** (ako/mxcli#1811) — the same sweep found document shapes the writers produce in a type or property newer than the project, with no check-time gate or with one below it: snippet parameters (9.21), `alter page … add parameters` (9.4), page variables (10.20, their default value), `action: complete task` (9.19) and the other workflow button actions (9.0.2), a menu item `Icon:` (9.20), progressive web app settings (9.0.3), a native profile's home page (9.4), a not-found page (10.13), JAR dependencies (10.0) and their exclusions (10.12), `create scheduled event` (9.12), `create database connection` (10.0; with a `query`, 10.12), `create business event service` (9.24), and on a consumed REST service basic authentication (10.2), any operation (10.4: its method and response handling), query parameters (11.0) and an OpenAPI document (10.21). `create view entity` was gated at 10.18, where view entities arrived, but mxcli stores the query in a separate source document that is a 10.21 type: on 10.18 to 10.20 `check -p` passed it and the write refused it. Its floor is now 10.21. Each floor is pinned to the type and property data the write refuses on; the `sdk/versions` rows that said otherwise (database connector 10.6, business events 10.0) now hold it, and mx 9.24.30 loads a business event service mxcli writes. +- **A workflow parameter, a user task and the other workflow parts an older Mendix stores differently are refused by `check -p`, not part-way through `exec`** (ako/mxcli#1811) — past the workflow's own 9.0.2 (#1809), what a workflow states has later floors, which the write enforced and no gate named. Measured on 9.x projects whose recorded version was set below each floor: every workflow with a `parameter` is refused below 9.18 (`Workflows$Parameter.Name`) and every `user task` below 9.19 (`Workflows$UserTaskOutcome.Value`); a `jump to` needs 9.0.5, a `call workflow` activity 9.6 (with parameter mappings 9.18), an `overview page` 9.11, an annotation 9.15 and an event sub-process started by an interrupting notification 11.10. `create workflow` and `alter workflow` now report each with MDL-VERSION01. The `user task` row of `sdk/versions` said 9.0.0, so `show features` on 9.0 to 9.18 listed user tasks the write could not store. +- **Loops, `show page` with arguments, queued calls and `execute database query` are refused by `check -p` below their Mendix version** (ako/mxcli#1811) — the sweep behind the previous entry listed every `$Type` the converters construct against the Model SDK's type data and found more flow activities the write refused with no check-time gate: `loop … in $List` and `while` (9.0.4: before it, a loop stores its list on itself), `show page … (Param = $Value)` (9.7), `call microflow`/`call java action … in queue` (9.10), `execute database query` (9.24) and its `connection ( … )` arguments (11.1). They are gated in `create`/`alter microflow` and `nanoflow` like the previous entry's. The `sdk/versions` rows for the database query said 10.6 and 11.0; they now hold the type floors, 9.24 and 11.1, and mx 9.24.30 loads the action, so `show features` on 9.24 lists it. +- **`send rest request`, `range`, `call external action`, `transform … with` and `find`/`filter … where` are refused by `check -p` below their Mendix version, not part-way through `exec`** (ako/mxcli#1811) — none of them had a check-time gate, so `mxcli check -p` passed them and the write refused them ("this Mendix 9.24.30 project cannot store Microflows$RestOperationCallAction (Mendix 10.1.0+)") after the statements before it, or, with `--continue-on-error`, ran on without that microflow. `check -p` and `exec`'s pre-flight now report MDL-VERSION01 naming the statement and the version, in `create microflow`/`nanoflow` and in what `alter microflow`/`nanoflow` inserts, wherever it sits: `find`/`filter … where` (or a condition that is not `Member = value`) 9.0.3, `call external action` 9.19 (with arguments 10.2), `range` 9.24, `send rest request` 10.1 (with arguments 10.2, a result variable 10.4, `body $Var` 10.5) and `transform … with` 11.4. Each floor is the type's introduction in the Mendix Model SDK, pinned by a test to the data the write refuses on. `send rest request` is gated at 10.1 although mx 9.24.30 loads the action: the consumed REST service it calls is a 10.1 type too, so a 9.24 author cannot set one up. A `find` on a String the statement declares is a String `find()`, not a list operation, and is not gated. +- **`create workflow` is refused by `check -p` on Mendix 9.0.0 and 9.0.1, not part-way through `exec`** (ako/mxcli#1809) — the `workflows basic` floor was 9.0.0, while `Workflows$Workflow` arrived in 9.0.2 (the Mendix Model SDK's version data), and only `alter workflow` applied it. On 9.0.0 or 9.0.1, `create workflow` and `create or modify workflow` passed `check -p` and the write then refused them ("cannot store Workflows$Workflow (Mendix 9.0.2+)"). The floor is now 9.0.2, pinned by a test to the same data the write refuses on, and all three statements report MDL-VERSION01 in `check -p` and `exec`'s pre-flight. `alter workflow`'s hint said "upgrade your project to Mendix 9.12+", a version the gate never stood for; it names none now, and the message states the floor. +- **A catalog refresh on an MPR v2 project opens only the files that changed** (ako/mxcli#1804) — every refresh read every `.mxunit` of the project, 4,753 files on a large app, after every save; on Windows each open is scanned by the virus scanner. mxcli now keeps a compressed copy of the units in `.mxcli/cache/unit-contents.db` and reads a unit from its file only when the file's size, modification time or recorded hash has changed, so a refresh after a save opens one file or none. The store is git-ignored and safe to delete. +- **Changing one entity no longer re-describes its whole module in the source index** (ako/mxcli#1803) — an incremental source rebuild treated an entity as changed whenever anything in its module's domain model did, so one new attribute re-described every entity of the module and every document using any of them: 346 documents for an unreferenced entity in a 154-entity module of a 4,753-unit app. An entity's basis is now its own subtree, and the same edit re-describes 1. A change to an association or annotation still re-describes the module's entities. +- **Workflow actions an older Mendix does not have are refused by `check -p`, not part-way through `exec`** (ako/mxcli#1787) — `notify workflow` (10.0), `get workflows` and `get workflow activity records` (10.4), and the other workflow actions of a microflow (`call workflow`, `set task outcome`, `open user task` and `open workflow` 9.0.2, `workflow operation abort` and `restart` 9.2, `pause` and `continue` 9.3, `get workflow data` 9.10, `retry` and `unpause` 9.12) had no check-time gate: `mxcli check -p` passed the script, and the write then refused it ("this Mendix 9.24.30 project cannot store Microflows$NotifyWorkflowAction (Mendix 10.0.0+)") after the statements before it, or, with `--continue-on-error`, ran on without that microflow. `check -p` and `exec`'s pre-flight now report MDL-VERSION01 naming the statement and the version, wherever the action sits: a branch, a loop, an error handler. Each floor is the type's introduction in the Mendix Model SDK, and a test pins the `sdk/versions` rows to the same data the write refuses on (canon.TypeIntroduced), so the two cannot disagree. Measured on 9.24.30: before, `check -p` passed and `exec` refused at write time; now `check -p` reports each, `exec` writes nothing, and `App.mpr` is byte-identical. On 10.24.24 the script executes as before, with 0 errors. +- **`alter microflow` and `alter nanoflow` are version-gated like `create`** (ako/mxcli#1787) — no version gate reached what an `insert` or `replace` splices in, so `mxcli check -p` passed an `alter microflow … { insert after … begin $Wfs = get workflows for $Obj; end; }` on 9.24.30, and the write refused it ("cannot store Microflows$GetWorkflowsAction (Mendix 10.4.0+)"). What an alter inserts is now gated as the same statements in a `create microflow` or `create nanoflow` are: the workflow actions, `send email` (11.13), a lock or unlock naming its workflow (10.0) and `synchronize unsynchronized` (9.4). `check -p` and `exec`'s pre-flight report MDL-VERSION01, and `exec` without the pre-flight refuses the statement before reading the stored flow. A drop, or an insert of nothing gated, is not affected. +- **`mxcli debug enable`, `status` and `break` no longer say continuing and stepping arrive "in a later slice"** (ako/mxcli#1813) — they work since ako/mxcli#1782; `enable` now names `debug break`, and `status`/`break` name `debug continue`. The debugger proposal's protocol table now records the measured shapes: `add_breakpoint` requires `condition` and `continue` takes `{debug_id}`. +- **`mxcli debug break` sets the breakpoint, and no `mxcli debug` command reports success when the runtime refuses it** (ako/mxcli#1782, mendixlabs/mxcli#1676) — `debug break` without `--if` sent `add_breakpoint` with no `condition`, which Mendix 11.13 and 11.15 (both measured) refuse with `{"message":"Missing property: condition","status":2}` and register nothing; mxcli printed "Breakpoint set", `debug breaks` listed it, and the microflow ran to the end without pausing. The runtime answers a refused debugger request with HTTP 200, and mxcli read only the HTTP status. `condition` is now always sent (empty for an unconditional breakpoint), and every debugger command fails with the runtime's message when the reply's status is non-zero — a bad `--if` expression, an unknown activity or microflow, a variable that does not exist, or a session token the runtime no longer knows (with the `mxcli debug enable` remedy). Verified on a running 11.15 app: the breakpoint now pauses the flow, and `paused`, `step` and `continue` drive it to the end. +- **`mxcli debug` prints the runtime's reason for a refused request without the usage text after it** (ako/mxcli#1782) — once a `debug` subcommand's arguments are accepted, an error is the runtime refusing a request or no flow being paused, not a usage mistake, and thirty lines of flags buried the reason. A missing argument or unknown flag still prints the usage. +- **`mxcli debug inspect` prints a String or enumeration variable's value instead of `{}`** (ako/mxcli#1782, mendixlabs/mxcli#1676) — `inspect` asks the runtime's `get_object`, which takes only an object variable and refuses a primitive ("The variable 'Greeting' is not a Mendix object."); the refusal's empty result was printed as the value. A primitive's type and value are in the paused flow's state already, and `inspect` now prints them from there; an object variable still shows its members. +- **`mxcli debug continue` resumes the paused flow instead of leaving it paused** (ako/mxcli#1782, mendixlabs/mxcli#1676) — it sent `continue` with no `debug_id`, which the runtime refuses ("Missing property: debug_id", measured on Mendix 11.13 and 11.15), and printed "Continued." while the flow stayed paused until `continue --all`. It now resumes the paused flow by its `debug_id`: the only paused one, or the one named with the new `--flow `, as `step` and `inspect` already did. - **Rebuilding a catalog that has a source index describes only what changed** (ako/mxcli#1790) — once a project had a source index (`refresh catalog full source`), every later rebuild described every document again. That includes the rebuilds `search` and `show references` start after each save. On a 4,753-unit app that was 38 s per rebuild, and much longer on Windows before #1789. A rebuild now re-describes the documents whose unit changed (an entity's unit is its module's domain model) and the documents whose references name one of them. The rest are carried over from the cache. After adding one attribute, the same app rebuilds in 11 s: 3,525 rows reused, 193 described, and the index is byte-identical to a forced full rebuild. A change outside the described documents (a folder, a module, the project settings, a Java action), a cache written by another mxcli build, and `force` still describe everything. The cache records each unit's content hash and container, so a document moved into another folder is described again too. - **Commands on an MPR v2 project open only the files they need** (ako/mxcli#1791) — on a 4,753-unit app, `show modules` opened 4,511 unit files, `describe microflow` 2,996 and `describe page` 3,176, in every process. The commands read every folder to build module paths and every microflow to name or find one, and `show modules` decoded every page, snippet, flow, constant and Java action just to count them. Each open is cheap on macOS and is scanned by Defender on Windows, so this added time to every command an agent runs. The unit index in `.mxcli/` now also holds each unit's name and `Excluded` flag, keyed by content hash like its type, so it never goes stale. Folders and microflows are named from the index, a microflow is found by name and read by ID, and `show modules` counts from unit types. The same commands now open 509, 93 and 274 files, with output identical across three projects and all 229 page describes. The index moves to `unit-index-2.idx`, so an older mxcli on the same project keeps its own `unit-types.idx` instead of the two rebuilding each other's. - **`refresh catalog full source` reads each unit file once instead of millions of times** (ako/mxcli#1789) — the source index describes every document, and each microflow describe found its microflow, and every microflow it calls, by listing every microflow of the project from disk. On a 4,753-unit MPR v2 app one refresh opened 7,532,649 `.mxunit` files: 5.5 minutes on macOS, and 90 minutes or more on Windows, where Defender scans every file open. The catalog build now keeps the units it reads in memory, and a describe reads its microflow and the microflows it calls by ID. The same refresh opens 4,753 files and takes 31 s, and its source text is byte-identical. The rebuilds that `show references` and `search` run on a project with a source index are the same build, so they speed up too. diff --git a/cmd/mxcli/check_version_gates_test.go b/cmd/mxcli/check_version_gates_test.go index c8a20d7074..48280ae053 100644 --- a/cmd/mxcli/check_version_gates_test.go +++ b/cmd/mxcli/check_version_gates_test.go @@ -36,7 +36,7 @@ end; ` // The same script plus a statement the project's version cannot store: exec's -// handler refuses `create view entity` below 10.18. +// handler refuses `create view entity` below 10.21 (ako/mxcli#1811). const issue1434Gated = issue1434Flows + `create view entity R1434.ItemView (Name: String(200)) as select cast('a' as String) as Name; ` @@ -84,7 +84,7 @@ func TestCheck_RefusesWhatTheProjectVersionCannotStore(t *testing.T) { if code == 0 || strings.Contains(out, "Check passed!") { t.Errorf("check passed a statement the 9.24 project cannot store (exit %d):\n%s", code, out) } - for _, want := range []string{"statement 5: create view entity requires Mendix 10.18.0+", "MDL-VERSION01"} { + for _, want := range []string{"statement 5: create view entity requires Mendix 10.21.0+", "MDL-VERSION01"} { if !strings.Contains(out, want) { t.Errorf("check output must name %q:\n%s", want, out) } @@ -134,7 +134,7 @@ func TestExecPreflight_AgreesWithCheckOnVersionGates(t *testing.T) { if refusal == "" || !strings.Contains(refusal, "Nothing was written") { t.Errorf("want the pre-flight to refuse the version-gated script, got %q\n%s", refusal, out) } - if !strings.Contains(out, "create view entity requires Mendix 10.18.0+") { + if !strings.Contains(out, "create view entity requires Mendix 10.21.0+") { t.Errorf("pre-flight output must name the gate:\n%s", out) } diff --git a/cmd/mxcli/cmd_debug.go b/cmd/mxcli/cmd_debug.go index e19736d115..0288b0fea6 100644 --- a/cmd/mxcli/cmd_debug.go +++ b/cmd/mxcli/cmd_debug.go @@ -14,10 +14,10 @@ import ( "github.com/spf13/cobra" ) -// cmd_debug.go is slice 1 of the microflow debugger (see +// cmd_debug.go is the microflow debugger's CLI (see // docs/11-proposals/PROPOSAL_microflow_debugger.md): connect to a runtime's -// debugger and toggle it (status/enable/disable), starting a session on enable. -// Breakpoints, paused-microflow inspection, and stepping are later slices. +// debugger and toggle it (status/enable/disable), starting a session on enable; +// set breakpoints by name, list and inspect paused flows, step and continue. var debugCmd = &cobra.Command{ Use: "debug", @@ -55,7 +55,7 @@ var debugStatusCmd = &cobra.Command{ fmt.Printf("Debug client connected: %v\n", st.ClientConnected) fmt.Printf("Paused microflows: %d\n", st.NumberOfPausedMicroflows) if st.NumberOfPausedMicroflows > 0 { - fmt.Println("\nNote: paused microflows hold their requests open until 'mxcli debug continue' (a later slice) or 'mxcli debug disable'.") + fmt.Println("\nNote: paused microflows hold their requests open until 'mxcli debug continue' (or 'continue --all') or 'mxcli debug disable'.") } return nil }, @@ -76,7 +76,7 @@ var debugEnableCmd = &cobra.Command{ return fmt.Errorf("debugger enabled but starting a session failed: %w", err) } fmt.Println("Debugger enabled; debug session started.") - fmt.Println("Breakpoints/stepping arrive in a later slice; for now the session is ready.") + fmt.Println("Set a breakpoint with 'mxcli debug break --activity <#n|caption>'.") fmt.Println("Remember to run 'mxcli debug disable' when done — a breakpoint pauses whoever hits it, the browser included.") return nil }, @@ -176,7 +176,7 @@ var debugBreakCmd = &cobra.Command{ if condition != "" { fmt.Printf(" condition: %s\n", condition) } - fmt.Println("Any execution that reaches it — including a browser request — pauses until 'continue' (a later slice) or 'mxcli debug disable'.") + fmt.Println("Any execution that reaches it — including a browser request — pauses until 'mxcli debug continue' or 'mxcli debug disable'.") return nil }, } @@ -289,6 +289,20 @@ var debugInspectCmd = &cobra.Command{ return err } asList, _ := cmd.Flags().GetBool("list") + if !asList { + // get_object takes only an object variable; a primitive (measured: + // a String, an enumeration) is refused ("The variable 'X' is not a + // Mendix object."), which printed as {} (ako/mxcli#1782). Its value + // is in the paused flow's state already, so print it from there. + // A variable not in that state (a nanoflow's, which lives in + // poll_events) still goes to get_object, refusal and all. + if paused, perr := c.PausedMicroflows(); perr == nil { + if v, typ, ok := pausedVariable(paused, debugID, args[0]); ok && typ != "object" && typ != "list" { + printJSON(v) + return nil + } + } + } var raw []byte if asList { raw, err = c.GetList(debugID, args[0]) @@ -331,7 +345,7 @@ var debugStepCmd = &cobra.Command{ } var debugContinueCmd = &cobra.Command{ - Use: "continue [--all]", + Use: "continue [--flow ] [--all]", Short: "Resume a paused microflow (or all with --all)", RunE: func(cmd *cobra.Command, args []string) error { c, err := resolveDebuggerClient(cmd) @@ -339,14 +353,24 @@ var debugContinueCmd = &cobra.Command{ return err } all, _ := cmd.Flags().GetBool("all") - if err := c.Continue(all); err != nil { - return err - } if all { + if err := c.ContinueAll(); err != nil { + return err + } fmt.Println("Continued all paused microflows.") - } else { - fmt.Println("Continued.") + return nil } + // The runtime resumes one flow only by its debug_id; without one it + // refuses the request and the flow stays paused. + flow, _ := cmd.Flags().GetString("flow") + debugID, err := resolveDebugID(c, flow) + if err != nil { + return err + } + if err := c.Continue(debugID); err != nil { + return err + } + fmt.Printf("Continued (debug_id: %s).\n", debugID) return nil }, } @@ -508,6 +532,7 @@ func init() { debugInspectCmd.Flags().String("flow", "", "Which paused microflow (debug_id); defaults to the only paused one") debugInspectCmd.Flags().Bool("list", false, "Inspect a list variable (uses get_list instead of get_object)") debugStepCmd.Flags().String("flow", "", "Which paused microflow (debug_id); defaults to the only paused one") + debugContinueCmd.Flags().String("flow", "", "Which paused microflow (debug_id); defaults to the only paused one") debugContinueCmd.Flags().Bool("all", false, "Continue all paused microflows, not just one") debugCmd.AddCommand(debugStatusCmd) @@ -521,5 +546,17 @@ func init() { debugCmd.AddCommand(debugContinueCmd) debugCmd.AddCommand(debugEnableCmd) debugCmd.AddCommand(debugDisableCmd) + // Once a subcommand runs, its arguments were right: an error from there on + // is the runtime refusing a request or a flow that is not paused, and the + // usage text would bury the runtime's reason. Argument and flag mistakes + // fail before RunE and still print the usage. + for _, sub := range debugCmd.Commands() { + if run := sub.RunE; run != nil { + sub.RunE = func(cmd *cobra.Command, args []string) error { + cmd.SilenceUsage = true + return run(cmd, args) + } + } + } rootCmd.AddCommand(debugCmd) } diff --git a/cmd/mxcli/cmd_debug_test.go b/cmd/mxcli/cmd_debug_test.go new file mode 100644 index 0000000000..e1843cbcd6 --- /dev/null +++ b/cmd/mxcli/cmd_debug_test.go @@ -0,0 +1,198 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "encoding/json" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "testing" +) + +// pausedACTDebug is get_paused_microflows as a Mendix 11.15 runtime answered it +// with MyFirstModule.ACT_Debug paused on its first activity (measured for +// ako/mxcli#1782): a String, two objects and an enumeration in scope. +const pausedACTDebug = `{"result":{"paused_microflows":[{"debug_id":"dbg-1","variables":{` + + `"currentSession":{"id":"13229323905401380","state":"NORMAL","type":"object","entity":"System.Session"},` + + `"Greeting":{"type":"string","value":"hello"},` + + `"currentUser":{"id":"13792273858822669","state":"NORMAL","type":"object","entity":"System.User"},` + + `"currentDeviceType":{"enumeration_name":"System.DeviceType","type":"enum","value":"Desktop"}},` + + `"microflow_id":"c5b44d8c-defc-42dd-aa6c-65acc647ce17","object_name":"CreateAndChange",` + + `"microflow_name":"MyFirstModule.ACT_Debug","object_id":"11276f01-9157-4cff-82b8-cdb78636802e"}]},"status":0}` + +// debugRuntimeStub is a /debugger/ endpoint that records each request and +// answers through reply (action, params) -> body. +type debugRuntimeStub struct { + *httptest.Server + actions []string + params []map[string]any +} + +func newDebugRuntimeStub(t *testing.T, reply func(action string, params map[string]any) string) *debugRuntimeStub { + t.Helper() + s := &debugRuntimeStub{} + s.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var body struct { + Action string `json:"action"` + Params map[string]any `json:"params"` + } + _ = json.NewDecoder(r.Body).Decode(&body) + s.actions = append(s.actions, body.Action) + s.params = append(s.params, body.Params) + _, _ = w.Write([]byte(reply(body.Action, body.Params))) + })) + t.Cleanup(s.Close) + return s +} + +// runDebug runs `mxcli debug ` against the stub, with a cached session +// token beside a scratch project path, and returns what it printed. +func runDebug(t *testing.T, stub *debugRuntimeStub, args ...string) (string, error) { + t.Helper() + dir := t.TempDir() + if err := os.MkdirAll(filepath.Join(dir, ".mxcli"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, ".mxcli", "debug-session.token"), []byte("tok-1"), 0o600); err != nil { + t.Fatal(err) + } + full := append([]string{"debug"}, args...) + full = append(full, "--app-url", stub.URL, "-p", filepath.Join(dir, "app.mpr")) + return captureStdout(t, func() error { + rootCmd.SetArgs(full) + return rootCmd.Execute() + }) +} + +// `debug continue` resumes the one paused flow by its debug_id. Sent without +// one, the runtime refuses it ("Missing property: debug_id") and the flow +// stays paused, which printed "Continued." (ako/mxcli#1782). +func TestDebugContinue_ResumesThePausedFlowByDebugID(t *testing.T) { + stub := newDebugRuntimeStub(t, func(action string, params map[string]any) string { + switch action { + case "get_paused_microflows": + return pausedACTDebug + case "poll_events": + return `{"result":{"events":[]},"status":0}` + case "continue": + if _, ok := params["debug_id"]; !ok { + return `{"result":{},"message":"Missing property: debug_id","status":2}` + } + } + return `{"result":{},"status":0}` + }) + out, err := runDebug(t, stub, "continue") + if err != nil { + t.Fatalf("debug continue: %v\n%s", err, out) + } + last := len(stub.actions) - 1 + if stub.actions[last] != "continue" || stub.params[last]["debug_id"] != "dbg-1" { + t.Errorf("last request = %s %v, want continue {debug_id: dbg-1}", stub.actions[last], stub.params[last]) + } + if !strings.Contains(out, "dbg-1") { + t.Errorf("output = %q, want it to name the flow it continued", out) + } +} + +// inspectStub answers get_object the way 11.15 does: an object variable gets +// its members, anything else is refused as "not a Mendix object". +func inspectStub(t *testing.T) *debugRuntimeStub { + return newDebugRuntimeStub(t, func(action string, params map[string]any) string { + switch action { + case "get_paused_microflows": + return pausedACTDebug + case "poll_events": + return `{"result":{"events":[]},"status":0}` + case "get_object": + if params["variable_name"] == "currentUser" { + return `{"result":{"object":{"members":{"Name":{"type":"string","value":"MxAdmin"}},"id":"13792273858822669","state":"NORMAL","entity":"System.User"}},"status":0}` + } + return `{"result":{},"message":"The variable '` + params["variable_name"].(string) + `' is not a Mendix object.","status":2}` + } + return `{"result":{},"status":0}` + }) +} + +// `debug inspect` of a String variable printed {} (ako/mxcli#1782): get_object +// refuses a primitive ("The variable 'Greeting' is not a Mendix object."), and +// the refusal's empty result was printed as the value. A primitive's value is +// in the paused flow's state already; inspect prints it from there. +func TestDebugInspect_PrimitiveVariablePrintsItsValue(t *testing.T) { + stub := inspectStub(t) + out, err := runDebug(t, stub, "inspect", "Greeting", "--flow", "dbg-1") + if err != nil { + t.Fatalf("debug inspect Greeting: %v\n%s", err, out) + } + if !strings.Contains(out, `"value": "hello"`) || !strings.Contains(out, `"type": "string"`) { + t.Errorf("output = %q, want the String variable's type and value", out) + } + for _, a := range stub.actions { + if a == "get_object" { + t.Errorf("a primitive must not be sent to get_object, which refuses it; requests = %v", stub.actions) + } + } +} + +// An object variable still goes to get_object, which returns its members. +func TestDebugInspect_ObjectVariableUsesGetObject(t *testing.T) { + stub := inspectStub(t) + out, err := runDebug(t, stub, "inspect", "currentUser", "--flow", "dbg-1") + if err != nil { + t.Fatalf("debug inspect currentUser: %v\n%s", err, out) + } + if !strings.Contains(out, `"MxAdmin"`) { + t.Errorf("output = %q, want the object's members from get_object", out) + } + if last := stub.actions[len(stub.actions)-1]; last != "get_object" { + t.Errorf("last request = %q, want get_object", last) + } +} + +// A variable the paused flow does not have is the runtime's call to make: +// get_object refuses it, and that refusal is the error. +func TestDebugInspect_UnknownVariableIsTheRuntimesRefusal(t *testing.T) { + stub := inspectStub(t) + out, err := runDebug(t, stub, "inspect", "Nope", "--flow", "dbg-1") + if err == nil { + t.Fatalf("debug inspect Nope succeeded, want the runtime's refusal\n%s", out) + } + if !strings.Contains(err.Error(), "The variable 'Nope'") { + t.Errorf("error = %v, want the runtime's message", err) + } +} + +// A request the runtime refused is not a usage mistake: the command line was +// right, so the refusal is printed without the usage text, which used to bury +// it under thirty lines of flags. Misuse — a missing argument — still prints +// the usage. +func TestDebug_RefusalPrintsNoUsage(t *testing.T) { + // cobra writes the error to the err writer and the usage to the out one. + var stderr bytes.Buffer + rootCmd.SetErr(&stderr) + rootCmd.SetOut(&stderr) + t.Cleanup(func() { rootCmd.SetErr(nil); rootCmd.SetOut(nil) }) + + stub := inspectStub(t) + if _, err := runDebug(t, stub, "inspect", "Nope", "--flow", "dbg-1"); err == nil { + t.Fatal("debug inspect Nope succeeded, want the runtime's refusal") + } + if strings.Contains(stderr.String(), "Usage:") { + t.Errorf("a runtime refusal printed the usage text:\n%s", stderr.String()) + } + + // The silencing is set on the command as its RunE starts, which is per + // process in the CLI (one command per run); undo it to stand for a new run. + debugInspectCmd.SilenceUsage = false + t.Cleanup(func() { debugInspectCmd.SilenceUsage = false }) + stderr.Reset() + if _, err := runDebug(t, stub, "inspect"); err == nil { + t.Fatal("debug inspect with no variable succeeded, want an argument error") + } + if !strings.Contains(stderr.String(), "Usage:") { + t.Errorf("a missing argument should still print the usage, got:\n%s", stderr.String()) + } +} diff --git a/cmd/mxcli/cmd_fix_property_sets_issue1805_test.go b/cmd/mxcli/cmd_fix_property_sets_issue1805_test.go new file mode 100644 index 0000000000..a97bbcbbb2 --- /dev/null +++ b/cmd/mxcli/cmd_fix_property_sets_issue1805_test.go @@ -0,0 +1,130 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "errors" + "os" + "path/filepath" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +// ako/mxcli#1805: on a Mendix 11.14.0 project mxcli wrote, `fix property-sets` +// reported +// +// MIXED DomainModels$NoGeneralization.HasOwnerAttr: on 81 of 84 element(s); without it: bf8b2a01-… [DomainModels$DomainModel] +// +// (with HasChangedByAttr, HasChangedDateAttr, HasCreatedDateAttr and +// DateTimeAttributeType.LocalizeDate) and wrote nothing: the elements were +// stored by an older mxcli, and the table had no measured value for the keys. +// The subject is the same gap in the Studio Pro-authored fixture (11.13.0): +// one entity's NoGeneralization and the DateTime attribute type of the same +// domain model as the older mxcli left them. + +// stripNestedKeys removes keys from the first element of typ in the unit file +// f and returns how many it removed. +func stripNestedKeys(t *testing.T, f, typ string, keys ...string) int { + t.Helper() + b, err := os.ReadFile(f) + if err != nil { + t.Fatal(err) + } + var doc bson.D + if err := bson.Unmarshal(b, &doc); err != nil { + t.Fatal(err) + } + drop := map[string]bool{} + for _, k := range keys { + drop[k] = true + } + removed, done := 0, false + var walk func(v any) any + walk = func(v any) any { + switch x := v.(type) { + case bson.D: + isSubject := false + for _, e := range x { + if e.Key == "$Type" && e.Value == typ && !done { + isSubject = true + } + } + out := make(bson.D, 0, len(x)) + for _, e := range x { + if isSubject && drop[e.Key] { + removed++ + continue + } + out = append(out, bson.E{Key: e.Key, Value: walk(e.Value)}) + } + if isSubject { + done = true + } + return out + case bson.A: + out := make(bson.A, len(x)) + for i, e := range x { + out[i] = walk(e) + } + return out + } + return v + } + nb, err := bson.Marshal(walk(doc)) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(f, nb, 0o644); err != nil { + t.Fatal(err) + } + return removed +} + +func TestFixPropertySets_Issue1805FillsAnOlderEntitysFlags(t *testing.T) { + mprPath := copyPedApp(t) + // The pedapp domain model holding both a NoGeneralization and the + // project's DateTime attribute. + f := filepath.Join(filepath.Dir(mprPath), "mprcontents", "35", "2a", "352a0695-fe3e-4b42-8e50-529188236707.mxunit") + flags := []string{"HasChangedByAttr", "HasChangedDateAttr", "HasCreatedDateAttr", "HasOwnerAttr"} + if n := stripNestedKeys(t, f, "DomainModels$NoGeneralization", flags...); n != len(flags) { + t.Fatalf("subject: removed %d system-attribute flag(s), want %d — the fixture changed", n, len(flags)) + } + if n := stripNestedKeys(t, f, "DomainModels$DateTimeAttributeType", "LocalizeDate"); n != 1 { + t.Fatalf("subject: removed %d LocalizeDate key(s), want 1 — the fixture changed", n) + } + var sink bytes.Buffer + if err := fixHashes(mprPath, true, &sink); err != nil { + t.Fatalf("re-index the subject: %v\n%s", err, sink.String()) + } + edited, _ := os.ReadFile(f) + + var out bytes.Buffer + err := fixPropertySets(mprPath, true, &out) + if !errors.Is(err, errPropertySetsRemain) { + t.Fatalf("dry run: want errPropertySetsRemain, got %v\n%s", err, out.String()) + } + if got := out.String(); strings.Contains(got, "MIXED") || !strings.Contains(got, "[DomainModels$DomainModel]: +5 key(s), -0 key(s)") { + t.Fatalf("dry run: want the domain model offered for the 5 missing keys and no MIXED name:\n%s", got) + } + if b, _ := os.ReadFile(f); !bytes.Equal(b, edited) { + t.Fatal("the dry run wrote the unit") + } + + out.Reset() + if err := fixPropertySets(mprPath, false, &out); err != nil { + t.Fatalf("repair: %v\n%s", err, out.String()) + } + if !strings.Contains(out.String(), "Completed 1 unit(s): 5 key(s) added, 0 stripped") { + t.Fatalf("repair not reported:\n%s", out.String()) + } + out.Reset() + if err := fixPropertySets(mprPath, true, &out); err != nil { + t.Fatalf("after the repair: %v\n%s", err, out.String()) + } + if !strings.Contains(out.String(), "Every unit already has") { + t.Fatalf("after the repair the project is not complete:\n%s", out.String()) + } +} diff --git a/cmd/mxcli/debug_resolve.go b/cmd/mxcli/debug_resolve.go index 121a6b7dce..cb0641b2ba 100644 --- a/cmd/mxcli/debug_resolve.go +++ b/cmd/mxcli/debug_resolve.go @@ -224,6 +224,36 @@ func extractPausedFromEvents(raw []byte) []pausedFlowSummary { return out } +// pausedVariable finds one in-scope variable of a paused flow in the +// get_paused_microflows result: {"paused_microflows":[{"debug_id":…, +// "variables":{"":{"type":"string","value":"hello"},…}}]} on 11.15. It +// returns the variable's entry and its "type" ("object", "string", "enum", …). +// Parsed as defensively as extractPausedFlows; not found is (nil, "", false). +func pausedVariable(raw []byte, debugID, name string) (json.RawMessage, string, bool) { + var v any + if json.Unmarshal(raw, &v) != nil { + return nil, "", false + } + for _, it := range firstList(v) { + m, ok := it.(map[string]any) + if !ok || firstString(m, "debug_id", "debugId", "id") != debugID { + continue + } + vars, _ := m["variables"].(map[string]any) + entry, ok := vars[name].(map[string]any) + if !ok { + return nil, "", false + } + b, err := json.Marshal(entry) + if err != nil { + return nil, "", false + } + typ, _ := entry["type"].(string) + return b, typ, true + } + return nil, "", false +} + // firstList returns v if it is an array, else the first array value found among a // map's values (one level deep). func firstList(v any) []any { diff --git a/cmd/mxcli/debug_resolve_test.go b/cmd/mxcli/debug_resolve_test.go index e7a23345d4..617f43e46b 100644 --- a/cmd/mxcli/debug_resolve_test.go +++ b/cmd/mxcli/debug_resolve_test.go @@ -139,6 +139,30 @@ func TestExtractPausedFromEvents(t *testing.T) { } } +func TestPausedVariable(t *testing.T) { + // get_paused_microflows' result with two flows paused (11.15 shape). + raw := []byte(`{"paused_microflows":[ + {"debug_id":"d-1","microflow_name":"M.A","variables":{"Greeting":{"type":"string","value":"hello"},"Thing":{"id":"1","type":"object","entity":"M.Thing"}}}, + {"debug_id":"d-2","microflow_name":"M.B","variables":{"Greeting":{"type":"string","value":"other flow"}}} + ]}`) + v, typ, ok := pausedVariable(raw, "d-1", "Greeting") + if !ok || typ != "string" || string(v) != `{"type":"string","value":"hello"}` { + t.Errorf("Greeting of d-1 = %s %q %v, want its own string entry", v, typ, ok) + } + if _, typ, ok := pausedVariable(raw, "d-1", "Thing"); !ok || typ != "object" { + t.Errorf("Thing type = %q %v, want object", typ, ok) + } + if _, _, ok := pausedVariable(raw, "d-1", "Nope"); ok { + t.Error("an unknown variable must not be found") + } + if _, _, ok := pausedVariable(raw, "d-3", "Greeting"); ok { + t.Error("a variable of another flow must not be found") + } + if _, _, ok := pausedVariable([]byte(`not json`), "d-1", "Greeting"); ok { + t.Error("invalid json must not be found") + } +} + func TestBreakpointRegistry_UpsertRemove(t *testing.T) { var bps []localBreakpoint bps = upsertBreakpoint(bps, localBreakpoint{Microflow: "M.F", Activity: "A", ObjectID: "g1"}) diff --git a/cmd/mxcli/docker/browser_proxy.go b/cmd/mxcli/docker/browser_proxy.go new file mode 100644 index 0000000000..9789ebc264 --- /dev/null +++ b/cmd/mxcli/docker/browser_proxy.go @@ -0,0 +1,197 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "context" + "net" + "net/url" + "strings" + "time" +) + +// browser_proxy.go keeps the browsers mxcli launches against an app from +// routing a connection to this machine through an egress proxy. +// +// Chromium on Linux takes its proxy from the environment (http_proxy, +// https_proxy, all_proxy, no_proxy). For a WebSocket it prefers the HTTPS +// proxy, ws:// included (RFC 6455 §4.1.3). So where only https_proxy is set — +// the Claude Code agent proxy is one — every http:// load of a local app goes +// direct while the Mendix runtime's dev channel, ws:///mxdevtools/, goes +// to the proxy unless is in no_proxy. localhost and 127.0.0.1 are; the +// app's own host name usually is not: an /etc/hosts entry such as +// frontend.local, which is how two local apps get separate cookie jars. A proxy +// elsewhere cannot reach this machine's loopback, and the agent proxy answers +// the handshake "502 upstream request failed". Chromium logs that as a console +// error, twice, and `playwright check` failed a page that had rendered +// (ako/mxcli#1800). With http_proxy set too, the page itself goes the same way. +// +// The runtime is not the cause: it answers the handshake 101 on the configured +// host name (measured on 11.15.0, curl with the proxy bypassed). Nor is +// `run --local`, which puts nothing between the browser and the runtime. +// +// So a host that resolves only to this machine is added to no_proxy for the +// browser mxcli starts. Nothing is hidden by it: the channel then connects as +// it does on 127.0.0.1, and any console error the page does have still counts. + +// lookupHostIPs resolves a host name. A seam for tests. +var lookupHostIPs = func(host string) ([]net.IP, error) { + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + addrs, err := net.DefaultResolver.LookupIPAddr(ctx, host) + if err != nil { + return nil, err + } + ips := make([]net.IP, len(addrs)) + for i, a := range addrs { + ips[i] = a.IP + } + return ips, nil +} + +// thisMachineIPs lists the addresses assigned to this machine's interfaces. +// A seam for tests. +var thisMachineIPs = func() []net.IP { + addrs, err := net.InterfaceAddrs() + if err != nil { + return nil + } + var ips []net.IP + for _, a := range addrs { + if n, ok := a.(*net.IPNet); ok { + ips = append(ips, n.IP) + } + } + return ips +} + +// directRouteEnv returns env for a browser that will load urls, with every +// host among them that resolves only to this machine added to no_proxy and +// NO_PROXY. env is returned unchanged when it configures no proxy, or when no +// such host needs adding; the proxy settings themselves are never touched, so +// a remote page (a --hub preview URL) is still fetched through the proxy. +func directRouteEnv(env []string, urls ...string) []string { + if !envConfiguresProxy(env) { + return env + } + // Chromium reads no_proxy, and NO_PROXY only when that is unset. + effective, ok := envGet(env, "no_proxy") + if !ok { + effective, _ = envGet(env, "NO_PROXY") + } + var hosts []string + for _, raw := range urls { + u, err := url.Parse(raw) + if err != nil { + continue + } + host := strings.ToLower(u.Hostname()) + if host == "" || containsFold(hosts, host) || withBypassHosts(effective, []string{host}) == effective || + !resolvesToThisMachine(host) { + continue + } + hosts = append(hosts, host) + } + if len(hosts) == 0 { + return env + } + out := append([]string(nil), env...) + for _, key := range []string{"no_proxy", "NO_PROXY"} { + cur, _ := envGet(env, key) + if merged := withBypassHosts(cur, hosts); merged != cur { + // Appended: os/exec gives a child the last value of a duplicate key. + out = append(out, key+"="+merged) + } + } + return out +} + +// envConfiguresProxy reports whether env sets any proxy Chromium reads. +func envConfiguresProxy(env []string) bool { + for _, key := range []string{"http_proxy", "https_proxy", "all_proxy"} { + for _, k := range []string{key, strings.ToUpper(key)} { + if v, _ := envGet(env, k); strings.TrimSpace(v) != "" { + return true + } + } + } + return false +} + +// resolvesToThisMachine reports whether every address host resolves to is +// loopback or assigned to this machine. A host that does not resolve, or that +// also resolves elsewhere, is left to the proxy. +func resolvesToThisMachine(host string) bool { + var ips []net.IP + if ip := net.ParseIP(host); ip != nil { + ips = []net.IP{ip} + } else { + var err error + if ips, err = lookupHostIPs(host); err != nil || len(ips) == 0 { + return false + } + } + local := thisMachineIPs() + for _, ip := range ips { + if ip.IsLoopback() { + continue + } + mine := false + for _, l := range local { + if l.Equal(ip) { + mine = true + break + } + } + if !mine { + return false + } + } + return true +} + +// withBypassHosts appends hosts to a comma-separated no_proxy list, skipping +// any it already names. An IPv6 literal is bracketed, as bypass lists write it. +func withBypassHosts(list string, hosts []string) string { + var have []string + for _, e := range strings.Split(list, ",") { + if e = strings.TrimSpace(e); e != "" { + have = append(have, e) + } + } + out := strings.TrimRight(strings.TrimSpace(list), ",") + for _, h := range hosts { + if ip := net.ParseIP(h); ip != nil && ip.To4() == nil { + h = "[" + h + "]" + } + if containsFold(have, h) { + continue + } + if out != "" { + out += "," + } + out += h + have = append(have, h) + } + return out +} + +// envGet returns the value of key in env, the last one when it repeats. +func envGet(env []string, key string) (string, bool) { + val, ok := "", false + for _, kv := range env { + if k, v, found := strings.Cut(kv, "="); found && k == key { + val, ok = v, true + } + } + return val, ok +} + +func containsFold(list []string, s string) bool { + for _, x := range list { + if strings.EqualFold(x, s) { + return true + } + } + return false +} diff --git a/cmd/mxcli/docker/browser_proxy_test.go b/cmd/mxcli/docker/browser_proxy_test.go new file mode 100644 index 0000000000..03b6a86c7c --- /dev/null +++ b/cmd/mxcli/docker/browser_proxy_test.go @@ -0,0 +1,141 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "net" + "strings" + "testing" +) + +// ako/mxcli#1800: `mxcli playwright check http://frontend.local:8180/` failed a +// page that rendered, on two console errors: +// +// WebSocket connection to 'ws://frontend.local:8180/mxdevtools/' failed: +// Error during WebSocket handshake: Unexpected response code: 502 +// +// frontend.local is an /etc/hosts name for 127.0.0.1. The runtime answers that +// handshake with 101 on the name; the 502 is the environment's HTTPS proxy, +// which Chromium uses for ws:// (RFC 6455 §4.1.3) and which cannot reach this +// machine's loopback. 127.0.0.1 and localhost are in no_proxy, which is why the +// same check on 127.0.0.1 passed. The browser mxcli launches must reach an app +// on this machine directly, whatever the name. + +// stubResolver makes host resolution and the machine's own addresses +// deterministic for the duration of a test. +func stubResolver(t *testing.T, hosts map[string][]string, local ...string) { + t.Helper() + oldLookup, oldLocal := lookupHostIPs, thisMachineIPs + t.Cleanup(func() { lookupHostIPs, thisMachineIPs = oldLookup, oldLocal }) + lookupHostIPs = func(host string) ([]net.IP, error) { + var ips []net.IP + for _, s := range hosts[host] { + ips = append(ips, net.ParseIP(s)) + } + if len(ips) == 0 { + return nil, &net.DNSError{Err: "no such host", Name: host, IsNotFound: true} + } + return ips, nil + } + thisMachineIPs = func() []net.IP { + var ips []net.IP + for _, s := range local { + ips = append(ips, net.ParseIP(s)) + } + return ips + } +} + +// The environment of the report: an HTTPS proxy, and a no_proxy that lists the +// loopback names but not the app's own host name. +var agentProxyEnv = []string{ + "PATH=/usr/bin", + "https_proxy=http://127.0.0.1:40291", + "HTTPS_PROXY=http://127.0.0.1:40291", + "no_proxy=localhost,127.0.0.1,::1", + "NO_PROXY=localhost,127.0.0.1,::1", +} + +func TestDirectRouteEnv_ALoopbackHostNameBypassesTheProxy(t *testing.T) { + stubResolver(t, map[string][]string{"frontend.local": {"127.0.0.1"}}) + env := directRouteEnv(agentProxyEnv, "http://frontend.local:8180/", "http://frontend.local:8180/p/narrate") + for _, key := range []string{"no_proxy", "NO_PROXY"} { + got, _ := envGet(env, key) + if got != "localhost,127.0.0.1,::1,frontend.local" { + t.Errorf("%s = %q: the browser sends ws://frontend.local:8180/mxdevtools/ to the HTTPS proxy, "+ + "which answers 502", key, got) + } + } + // The proxy itself is left as it was: a remote page still needs it. + if got, _ := envGet(env, "https_proxy"); got != "http://127.0.0.1:40291" { + t.Errorf("https_proxy = %q, want it untouched", got) + } +} + +// An address assigned to this machine is as unreachable for a proxy elsewhere +// as loopback is — the case of an app on --listen-addr 0.0.0.0 checked through +// the machine's own name or address. +func TestDirectRouteEnv_AnAddressOfThisMachineBypassesTheProxy(t *testing.T) { + stubResolver(t, map[string][]string{"devbox": {"192.0.2.2"}}, "192.0.2.2") + env := directRouteEnv(agentProxyEnv, "http://devbox:8080/", "http://192.0.2.2:8080/") + if got, _ := envGet(env, "no_proxy"); got != "localhost,127.0.0.1,::1,devbox,192.0.2.2" { + t.Errorf("no_proxy = %q, want devbox and 192.0.2.2 added", got) + } +} + +// Controls: what is not on this machine keeps the proxy, and nothing changes +// where there is no proxy to bypass. +func TestDirectRouteEnv_LeavesEverythingElseAlone(t *testing.T) { + stubResolver(t, map[string][]string{ + "frontend.local": {"127.0.0.1"}, + "preview.example": {"203.0.113.7"}, + "split.example": {"127.0.0.1", "203.0.113.7"}, + "localhost": {"127.0.0.1", "::1"}, + "already.listed.dev": {"127.0.0.1"}, + }) + cases := []struct { + name string + env []string + urls []string + }{ + {"a remote host", agentProxyEnv, []string{"https://preview.example/"}}, + {"a host that also resolves elsewhere", agentProxyEnv, []string{"http://split.example:8080/"}}, + {"a host that does not resolve", agentProxyEnv, []string{"http://nowhere.invalid:8080/"}}, + {"a host no_proxy already lists", []string{"https_proxy=http://p:1", "no_proxy=already.listed.dev"}, []string{"http://ALREADY.listed.dev:8080/"}}, + {"no proxy configured", []string{"PATH=/usr/bin", "no_proxy=localhost"}, []string{"http://frontend.local:8180/"}}, + {"an unparsable URL", agentProxyEnv, []string{"http://[::1"}}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + got := directRouteEnv(c.env, c.urls...) + if strings.Join(got, "\n") != strings.Join(c.env, "\n") { + t.Errorf("env changed:\n got %q\nwant %q", got, c.env) + } + }) + } + // Control for the cases above: the same env does change for a local name. + if got := directRouteEnv(agentProxyEnv, "http://frontend.local:8180/"); len(got) == len(agentProxyEnv) { + t.Errorf("control: a loopback host name did not change the env") + } +} + +// Chromium reads no_proxy, falling back to NO_PROXY; an environment that sets a +// proxy and neither gets both. +func TestDirectRouteEnv_SetsNoProxyWhenThereIsNone(t *testing.T) { + stubResolver(t, map[string][]string{"frontend.local": {"127.0.0.1"}}) + env := directRouteEnv([]string{"HTTPS_PROXY=http://127.0.0.1:40291"}, "http://frontend.local:8180/") + for _, key := range []string{"no_proxy", "NO_PROXY"} { + if got, ok := envGet(env, key); !ok || got != "frontend.local" { + t.Errorf("%s = %q (set %v), want frontend.local", key, got, ok) + } + } +} + +// An IPv6 literal is written the way a proxy bypass list spells one. +func TestDirectRouteEnv_IPv6LiteralIsBracketed(t *testing.T) { + stubResolver(t, nil, "fd00::2") + env := directRouteEnv([]string{"https_proxy=http://p:1"}, "http://[fd00::2]:8080/") + if got, _ := envGet(env, "no_proxy"); got != "[fd00::2]" { + t.Errorf("no_proxy = %q, want [fd00::2]", got) + } +} diff --git a/cmd/mxcli/docker/debugger.go b/cmd/mxcli/docker/debugger.go index 710e1893a0..df71b7c877 100644 --- a/cmd/mxcli/docker/debugger.go +++ b/cmd/mxcli/docker/debugger.go @@ -14,11 +14,11 @@ import ( "time" ) -// debugger.go is slice 1 of the Mendix microflow debugger (see -// docs/11-proposals/PROPOSAL_microflow_debugger.md). It covers the two-plane -// wiring only: the M2EE admin plane toggles the debugger on/off and reports -// status, and the app's /debugger/ endpoint starts a debug session. Breakpoints, -// paused-microflow inspection, and stepping are later slices. +// debugger.go is the client of the Mendix microflow debugger (see +// docs/11-proposals/PROPOSAL_microflow_debugger.md). It speaks two planes: the +// M2EE admin plane toggles the debugger on/off and reports status, and the +// app's /debugger/ endpoint runs the session — breakpoints, paused-flow +// inspection, stepping and continue. // // Two APIs, two auth schemes: // @@ -42,9 +42,10 @@ type DebuggerOptions struct { // TokenPath, when set, caches the session token across invocations (the CLI // is one-shot per command). Empty keeps the token in memory only. TokenPath string - // Timeout bounds a single debugger HTTP request (default 30s). Note: a - // breakpoint-driven call in a later slice can block far longer while paused — - // those will use their own timeout. + // Timeout bounds a single debugger HTTP request (default 30s). It must stay + // above poll_events' long poll, which the runtime holds for about 25 s when + // nothing is pending: a poll the client abandons loses the event delivered + // into it (measured for ako/mxcli#1782; see ako/mxcli#1812). Timeout time.Duration } @@ -151,6 +152,11 @@ func (c *DebuggerClient) StartSession() (string, error) { // // A nanoflow breakpoint uses the nanoflow_name param, not microflow_name — the // runtime NPEs on the wrong key (findings — nanoflow debugging). +// +// condition is always sent, as "" for an unconditional breakpoint: the runtime +// (11.13 and 11.15, measured) answers a request without it with +// {"message":"Missing property: condition","status":2} and registers nothing +// (ako/mxcli#1782). func (c *DebuggerClient) AddBreakpoint(flowName, objectID, condition string, nanoflow bool) error { nameKey := "microflow_name" if nanoflow { @@ -159,9 +165,7 @@ func (c *DebuggerClient) AddBreakpoint(flowName, objectID, condition string, nan params := map[string]any{ nameKey: flowName, "object_id": objectID, - } - if condition != "" { - params["condition"] = condition + "condition": condition, } _, err := c.post("add_breakpoint", true, params) return err @@ -219,14 +223,20 @@ func (c *DebuggerClient) Step(kind, debugID string) error { return err } -// Continue resumes execution: the current paused flow, or all paused flows when -// all is true. -func (c *DebuggerClient) Continue(all bool) error { - action := "continue" - if all { - action = "continue_all" - } - _, err := c.post(action, true, nil) +// Continue resumes one paused flow (by its debug_id). The runtime requires the +// debug_id: continue without it is refused with "Missing property: debug_id" +// and the flow stays paused (11.13 and 11.15, measured — ako/mxcli#1782). +func (c *DebuggerClient) Continue(debugID string) error { + if debugID == "" { + return fmt.Errorf("continue needs the paused flow's debug_id (see 'mxcli debug paused')") + } + _, err := c.post("continue", true, map[string]any{"debug_id": debugID}) + return err +} + +// ContinueAll resumes every paused flow. +func (c *DebuggerClient) ContinueAll() error { + _, err := c.post("continue_all", true, nil) return err } @@ -266,7 +276,8 @@ func (c *DebuggerClient) clearToken() { // post drives the /debugger/ plane. The envelope is {action, session_token?, // params}; params is always serialized (the runtime rejects a missing params -// with "Missing property"). Returns the raw "result" object from the response. +// with "Missing property"). Returns the raw "result" object from the response, +// or a *DebuggerError when the reply's status says the runtime refused it. func (c *DebuggerClient) post(action string, withToken bool, params map[string]any) (json.RawMessage, error) { if params == nil { params = map[string]any{} @@ -311,10 +322,45 @@ func (c *DebuggerClient) post(action string, withToken bool, params map[string]a } var env struct { - Result json.RawMessage `json:"result"` + Result json.RawMessage `json:"result"` + Status *int `json:"status"` + Message string `json:"message"` } if err := json.Unmarshal(respBody, &env); err != nil { return nil, fmt.Errorf("decoding debugger response: %w (body: %s)", err, strings.TrimSpace(string(respBody))) } + // The runtime answers a refusal with HTTP 200 too; only the envelope says so. + // A missing status is not a refusal (see DebuggerError). + if env.Status != nil && *env.Status != 0 { + return nil, &DebuggerError{Action: action, Status: *env.Status, Message: env.Message} + } return env.Result, nil } + +// DebuggerError is the runtime refusing a /debugger/ request. The endpoint +// answers HTTP 200 either way, so only the reply envelope tells them apart: +// success is {"result":…,"status":0}, a refusal {"result":{},"message":…, +// "status":2} — every reply measured on Mendix 11.13 and 11.15 carries a status, +// and every refusal seen was 2. Any non-zero status is treated as a refusal; a +// reply with no status at all is not, so a runtime that leaves it out still +// works. Ignoring the envelope is how `debug break` printed "Breakpoint set" +// over "Missing property: condition" (ako/mxcli#1782). +type DebuggerError struct { + Action string // the debugger action that was refused + Status int // the envelope status (non-zero) + Message string // the runtime's reason, verbatim; may be empty +} + +func (e *DebuggerError) Error() string { + msg := e.Message + if msg == "" { + msg = "no message" + } + s := fmt.Sprintf("the runtime refused %s (status %d): %s", e.Action, e.Status, msg) + // A token the runtime does not know: it restarted, or another debug session + // started since (one session at a time — starting one invalidates the last). + if strings.Contains(e.Message, "session token") { + s += " -- the debug session is gone (the runtime restarted, or another debugger started a session); run 'mxcli debug enable' for a new one" + } + return s +} diff --git a/cmd/mxcli/docker/debugger_test.go b/cmd/mxcli/docker/debugger_test.go index bcacef36e4..fcccc1d0f2 100644 --- a/cmd/mxcli/docker/debugger_test.go +++ b/cmd/mxcli/docker/debugger_test.go @@ -15,15 +15,42 @@ import ( // debuggerTestServer stubs BOTH planes on one httptest server, routed by path: // "/" is the M2EE admin plane, "/debugger/" is the debugger endpoint. It records // what it received so tests can assert the envelope and auth. +// +// The /debugger/ half answers the way a Mendix 11.13/11.15 runtime does, measured +// against both (ako/mxcli#1782): every reply is HTTP 200, success is +// {"result":…,"status":0}, and a rejection is {"result":{},"message":…,"status":2}. +// A request missing a property the runtime requires is rejected with the +// runtime's own "Missing property: " — add_breakpoint without condition and +// continue without debug_id among them. type debuggerTestServer struct { *httptest.Server adminActions []string enablePass string // password seen on enable_debugger dbgActions []string - dbgAuth string // X-Debugger-Authentication seen on the last /debugger/ call - dbgHadParams bool // whether the last /debugger/ body had a params key - dbgToken string // session_token seen on the last /debugger/ call - dbgParams map[string]any // params of the last /debugger/ call + dbgAuth string // X-Debugger-Authentication seen on the last /debugger/ call + dbgHadParams bool // whether the last /debugger/ body had a params key + dbgToken string // session_token seen on the last /debugger/ call + dbgParams map[string]any // params of the last /debugger/ call + reject map[string]string // action -> message to reject it with (status 2) +} + +// debuggerRequiredParams are the properties the runtime refuses a request +// without (each measured: omitting it yields "Missing property: "). +var debuggerRequiredParams = map[string][]string{ + "add_breakpoint": {"object_id", "condition"}, + "remove_breakpoint": {"object_id"}, + "get_object": {"debug_id", "variable_name"}, + "get_list": {"debug_id", "variable_name"}, + "step_over": {"debug_id"}, + "step_into": {"debug_id"}, + "step_out": {"debug_id"}, + "continue": {"debug_id"}, +} + +// debuggerRejection is the runtime's rejection envelope, verbatim in shape. +func debuggerRejection(message string) []byte { + b, _ := json.Marshal(map[string]any{"result": map[string]any{}, "message": message, "status": 2}) + return b } func newDebuggerTestServer(t *testing.T) (*debuggerTestServer, DebuggerOptions) { @@ -45,11 +72,21 @@ func newDebuggerTestServer(t *testing.T) (*debuggerTestServer, DebuggerOptions) if tok, ok := body["session_token"]; ok { _ = json.Unmarshal(tok, &ts.dbgToken) } + if msg, ok := ts.reject[action]; ok { + _, _ = w.Write(debuggerRejection(msg)) + return + } + for _, key := range debuggerRequiredParams[action] { + if _, ok := ts.dbgParams[key]; !ok { + _, _ = w.Write(debuggerRejection("Missing property: " + key)) + return + } + } switch action { case "start_session": - _, _ = w.Write([]byte(`{"result":{"session_token":"tok-123","runtime_version":"11.6"}}`)) + _, _ = w.Write([]byte(`{"result":{"session_token":"tok-123","runtime_version":"11.15.0","paused_microflows":[]},"status":0}`)) default: - _, _ = w.Write([]byte(`{"result":{}}`)) + _, _ = w.Write([]byte(`{"result":{},"status":0}`)) } return } @@ -189,7 +226,12 @@ func TestDebugger_AddBreakpoint(t *testing.T) { } } -func TestDebugger_AddBreakpointOmitsEmptyCondition(t *testing.T) { +// An unconditional breakpoint still sends condition, as the empty string. The +// runtime (11.13 and 11.15, measured) refuses add_breakpoint without it — +// {"message":"Missing property: condition","status":2} — and registers nothing, +// so a breakpoint set this way never paused (ako/mxcli#1782, +// mendixlabs/mxcli#1676). +func TestDebugger_AddBreakpointSendsEmptyCondition(t *testing.T) { ts, opts := newDebuggerTestServer(t) c := NewDebuggerClient(opts) if _, err := c.StartSession(); err != nil { @@ -198,8 +240,12 @@ func TestDebugger_AddBreakpointOmitsEmptyCondition(t *testing.T) { if err := c.AddBreakpoint("M.F", "guid-2", "", false); err != nil { t.Fatalf("AddBreakpoint: %v", err) } - if _, ok := ts.dbgParams["condition"]; ok { - t.Errorf("empty condition must be omitted, got %v", ts.dbgParams["condition"]) + cond, ok := ts.dbgParams["condition"] + if !ok { + t.Fatalf("add_breakpoint params = %v: condition is missing, which the runtime rejects with \"Missing property: condition\"", ts.dbgParams) + } + if cond != "" { + t.Errorf("condition = %v, want the empty string for an unconditional breakpoint", cond) } } @@ -257,23 +303,33 @@ func TestDebugger_StepActions(t *testing.T) { } } +// continue names the flow it resumes. The runtime (11.13 and 11.15, measured) +// rejects continue without debug_id — {"message":"Missing property: debug_id", +// "status":2} — and the flow stays paused, which `debug continue` reported as +// "Continued." (ako/mxcli#1782). continue_all takes no flow. func TestDebugger_Continue(t *testing.T) { ts, opts := newDebuggerTestServer(t) c := NewDebuggerClient(opts) if _, err := c.StartSession(); err != nil { t.Fatalf("StartSession: %v", err) } - if err := c.Continue(false); err != nil { + if err := c.Continue("dbg-9"); err != nil { t.Fatalf("Continue: %v", err) } if got := ts.dbgActions[len(ts.dbgActions)-1]; got != "continue" { - t.Errorf("Continue(false) = %q, want continue", got) + t.Errorf("Continue = %q, want continue", got) + } + if ts.dbgParams["debug_id"] != "dbg-9" { + t.Errorf("continue params = %v, want debug_id dbg-9 (the runtime rejects continue without it: \"Missing property: debug_id\")", ts.dbgParams) } - if err := c.Continue(true); err != nil { - t.Fatalf("Continue(all): %v", err) + if err := c.ContinueAll(); err != nil { + t.Fatalf("ContinueAll: %v", err) } if got := ts.dbgActions[len(ts.dbgActions)-1]; got != "continue_all" { - t.Errorf("Continue(true) = %q, want continue_all", got) + t.Errorf("ContinueAll = %q, want continue_all", got) + } + if err := c.Continue(""); err == nil { + t.Error("Continue without a debug_id should fail before reaching the runtime") } } @@ -363,3 +419,130 @@ func TestDebugger_AuthFailure(t *testing.T) { t.Errorf("error = %q, want it to mention auth + how to enable", got) } } + +// The reported symptom (ako/mxcli#1782, mendixlabs/mxcli#1676): the runtime +// refuses a debugger request with HTTP 200 and a non-zero envelope status, and +// `debug break` printed "Breakpoint set" over it. Every client call must return +// that refusal as an error carrying the runtime's message, so no `mxcli debug` +// command can report success over it. +func TestDebugger_RejectionReachesTheCaller(t *testing.T) { + // Each message is one the runtime gave for that action, measured on 11.15. + const ( + badToken = "Could not read session token: Invalid session token." + badFlowID = "Could not find microflow/nanoflow in debug with id: dbg-9" + ) + cases := []struct { + action string + msg string + call func(c *DebuggerClient) error + }{ + {"add_breakpoint", "Missing property: condition", func(c *DebuggerClient) error { return c.AddBreakpoint("M.F", "guid-1", "", false) }}, + {"remove_breakpoint", badToken, func(c *DebuggerClient) error { return c.RemoveBreakpoint("guid-1") }}, + {"get_paused_microflows", badToken, func(c *DebuggerClient) error { _, err := c.PausedMicroflows(); return err }}, + {"poll_events", badToken, func(c *DebuggerClient) error { _, err := c.PollEvents(); return err }}, + {"get_object", "The variable 'Greeting' is not a Mendix object.", func(c *DebuggerClient) error { _, err := c.GetObject("dbg-9", "Greeting"); return err }}, + {"get_list", "The variable 'Greeting' is not a list of Mendix objects.", func(c *DebuggerClient) error { _, err := c.GetList("dbg-9", "Greeting"); return err }}, + {"step_over", badFlowID, func(c *DebuggerClient) error { return c.Step("over", "dbg-9") }}, + {"step_into", badFlowID, func(c *DebuggerClient) error { return c.Step("into", "dbg-9") }}, + {"step_out", badFlowID, func(c *DebuggerClient) error { return c.Step("out", "dbg-9") }}, + {"continue", badFlowID, func(c *DebuggerClient) error { return c.Continue("dbg-9") }}, + {"continue_all", "Could not find any paused microflows/nanoflows to continue.", func(c *DebuggerClient) error { return c.ContinueAll() }}, + } + for _, tc := range cases { + t.Run(tc.action, func(t *testing.T) { + ts, opts := newDebuggerTestServer(t) + c := NewDebuggerClient(opts) + if _, err := c.StartSession(); err != nil { + t.Fatalf("StartSession: %v", err) + } + ts.reject = map[string]string{tc.action: tc.msg} + err := tc.call(c) + if err == nil { + t.Fatalf("%s: the runtime answered {\"message\":%q,\"status\":2} and the client reported success", tc.action, tc.msg) + } + if !strings.Contains(err.Error(), tc.msg) { + t.Errorf("%s error = %q, want the runtime's message %q", tc.action, err, tc.msg) + } + if !strings.Contains(err.Error(), tc.action) { + t.Errorf("%s error = %q, want it to name the action", tc.action, err) + } + }) + } +} + +// start_session's refusal surfaces the runtime's message too, rather than a +// generic "returned no session_token". +func TestDebugger_StartSessionRejection(t *testing.T) { + ts, opts := newDebuggerTestServer(t) + ts.reject = map[string]string{"start_session": "Debugger is not enabled"} + _, err := NewDebuggerClient(opts).StartSession() + if err == nil || !strings.Contains(err.Error(), "Debugger is not enabled") { + t.Fatalf("StartSession error = %v, want the runtime's message", err) + } +} + +// A session token the runtime no longer knows (it restarted, or another debug +// session started since) is refused with no result at all, verbatim below as +// measured on 11.13. The error names the runtime's message and how to get a +// fresh session. +func TestDebugger_InvalidSessionToken(t *testing.T) { + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write([]byte(`{"message":"Could not read session token: Invalid session token.","status":2}`)) + })) + defer ts.Close() + tokenPath := filepath.Join(t.TempDir(), "debug-session.token") + if err := os.WriteFile(tokenPath, []byte("0c1b2d3e-0000-4000-8000-000000000000"), 0o600); err != nil { + t.Fatal(err) + } + c := NewDebuggerClient(DebuggerOptions{AppURL: ts.URL, TokenPath: tokenPath}) + if err := c.LoadToken(); err != nil { + t.Fatal(err) + } + err := c.AddBreakpoint("M.F", "guid-1", "", false) + if err == nil { + t.Fatal("a refused session token must fail the call") + } + if got := err.Error(); !strings.Contains(got, "Invalid session token") || !strings.Contains(got, "debug enable") { + t.Errorf("error = %q, want the runtime's message and the 'mxcli debug enable' remedy", got) + } +} + +// What success looks like: status 0 (every reply measured on 11.13 and 11.15 +// carries it), or no status at all. A reply without a status is not a refusal, +// so a runtime that omits it keeps working; any non-zero status is one, with +// or without a message. +func TestDebugger_EnvelopeStatus(t *testing.T) { + cases := []struct { + name string + body string + wantErr string // "" = success + }{ + {"status 0", `{"result":{"paused_microflows":[]},"status":0}`, ""}, + {"no status", `{"result":{"paused_microflows":[]}}`, ""}, + {"status 2 with message", `{"result":{},"message":"Unknown action: frobnicate","status":2}`, "Unknown action: frobnicate"}, + {"non-zero status without message", `{"result":{},"status":1}`, "status 1"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write([]byte(tc.body)) + })) + defer ts.Close() + c := NewDebuggerClient(DebuggerOptions{AppURL: ts.URL}) + c.token = "tok-123" + raw, err := c.PausedMicroflows() + if tc.wantErr == "" { + if err != nil { + t.Fatalf("PausedMicroflows: %v", err) + } + if !strings.Contains(string(raw), "paused_microflows") { + t.Errorf("result = %s, want the result object", raw) + } + return + } + if err == nil || !strings.Contains(err.Error(), tc.wantErr) { + t.Errorf("error = %v, want it to contain %q", err, tc.wantErr) + } + }) + } +} diff --git a/cmd/mxcli/docker/pagecheck.go b/cmd/mxcli/docker/pagecheck.go index 6dfef0ca55..342737f5ce 100644 --- a/cmd/mxcli/docker/pagecheck.go +++ b/cmd/mxcli/docker/pagecheck.go @@ -318,7 +318,13 @@ func ProbePages(opts ProbeOptions) ([]PageSignals, error) { timeout = 60*time.Second + time.Duration(len(opts.Pages))*perPage } cmd := exec.Command(node, scriptPath, reqPath) - cmd.Env = append(os.Environ(), "NODE_PATH="+playwrightNodePath(node)) + // An app on this machine is reached directly, whatever name it is checked + // by: its dev channel must not be sent to an egress proxy (ako/mxcli#1800). + pageURLs := make([]string, len(opts.Pages)) + for i, pg := range opts.Pages { + pageURLs[i] = pg.URL + } + cmd.Env = directRouteEnv(append(os.Environ(), "NODE_PATH="+playwrightNodePath(node)), pageURLs...) out := &syncBuffer{} cmd.Stdout, cmd.Stderr = out, out diff --git a/cmd/mxcli/docker/pagecheck_proxy_test.go b/cmd/mxcli/docker/pagecheck_proxy_test.go new file mode 100644 index 0000000000..5faed3c4b5 --- /dev/null +++ b/cmd/mxcli/docker/pagecheck_proxy_test.go @@ -0,0 +1,192 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "bufio" + "crypto/sha1" + "encoding/base64" + "fmt" + "io" + "net" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" +) + +// ako/mxcli#1800, end to end through the real probe: a page on this machine, +// reached by a name the environment's no_proxy does not list, opens the +// runtime's dev channel ws:///mxdevtools/. Chromium sends a ws:// URL to +// the HTTPS proxy (RFC 6455 §4.1.3) — here a fake one that, like the agent +// proxy of the report, cannot reach the app and answers the handshake 502 — +// while the page's own http:// loads go direct and render. The check failed +// the page on the two handshake errors. +// +// The host is this machine's own non-loopback address, an IP literal: Chromium +// bypasses localhost and 127.0.0.0/8 by itself, and a host name that resolves +// to loopback would need an /etc/hosts entry. The routing decision is the same +// one: the host is not in no_proxy. Host-name resolution is covered by +// browser_proxy_test.go. + +func TestProbeReachesALocalAppsDevChannelDirectly(t *testing.T) { + host := nonLoopbackAddrOrSkip(t) + proxy := newRefusingProxy(t) + app := fakeDevModePage(t, host) + proxyEnv(t, proxy.url) + + // Control: with the address not known as this machine's, the channel goes + // through the proxy and the reported symptom appears. + stubResolver(t, nil) + sig := probeOrSkip(t, ProbeOptions{Pages: []PageTarget{{URL: app.URL + "/"}}, WaitMs: 3000}) + if !has502Handshake(sig.ConsoleErrors) || proxy.connects.Load() == 0 { + t.Fatalf("control: the harness did not reproduce the proxied handshake (connects=%d): %q", + proxy.connects.Load(), sig.ConsoleErrors) + } + + // The fix: the same page, with the address known as this machine's. + stubResolver(t, nil, host) + proxy.connects.Store(0) + sig = probeOrSkip(t, ProbeOptions{Pages: []PageTarget{{URL: app.URL + "/"}}, WaitMs: 3000}) + if len(sig.ConsoleErrors) != 0 { + t.Errorf("console errors for a page that rendered: %q", sig.ConsoleErrors) + } + if n := proxy.connects.Load(); n != 0 { + t.Errorf("the browser sent %d request(s) for an app on this machine to the proxy", n) + } + if fails := EvaluatePage(sig, CheckSpec{}); len(fails) != 0 { + t.Errorf("the page fails the check: %v", fails) + } +} + +func has502Handshake(errs []string) bool { + for _, e := range errs { + if strings.Contains(e, "/mxdevtools/") && strings.Contains(e, "Unexpected response code: 502") { + return true + } + } + return false +} + +// nonLoopbackAddrOrSkip returns an IPv4 address of this machine that is not +// loopback, to serve the fake app on. +func nonLoopbackAddrOrSkip(t *testing.T) string { + t.Helper() + addrs, err := net.InterfaceAddrs() + if err != nil { + t.Skipf("no interface addresses: %v", err) + } + for _, a := range addrs { + if n, ok := a.(*net.IPNet); ok && n.IP.To4() != nil && !n.IP.IsLoopback() && !n.IP.IsLinkLocalUnicast() { + return n.IP.String() + } + } + t.Skip("this machine has no non-loopback IPv4 address to serve the fake app on") + return "" +} + +// proxyEnv gives the probe the environment of the report: an HTTPS proxy and +// nothing else, with only the loopback names exempt. +func proxyEnv(t *testing.T, proxyURL string) { + for _, k := range []string{"https_proxy", "HTTPS_PROXY"} { + t.Setenv(k, proxyURL) + } + for _, k := range []string{"http_proxy", "HTTP_PROXY", "all_proxy", "ALL_PROXY"} { + t.Setenv(k, "") + } + for _, k := range []string{"no_proxy", "NO_PROXY"} { + t.Setenv(k, "localhost,127.0.0.1,::1") + } +} + +// refusingProxy accepts a CONNECT and answers whatever is sent through the +// tunnel with the agent proxy's 502, as measured: `curl -p -x +// http://frontend.local:8180/mxdevtools/` → 200 Connection Established, then +// 502 Bad Gateway "upstream request failed". +type refusingProxy struct { + url string + connects atomic.Int32 +} + +func newRefusingProxy(t *testing.T) *refusingProxy { + t.Helper() + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + p := &refusingProxy{url: "http://" + ln.Addr().String()} + t.Cleanup(func() { ln.Close() }) + go func() { + for { + c, err := ln.Accept() + if err != nil { + return + } + go p.serve(c) + } + }() + return p +} + +func (p *refusingProxy) serve(c net.Conn) { + defer c.Close() + br := bufio.NewReader(c) + req, err := http.ReadRequest(br) + if err != nil { + return + } + if req.Method != http.MethodConnect { + fmt.Fprint(c, "HTTP/1.1 405 Method Not Allowed\r\nContent-Length: 0\r\n\r\n") + return + } + p.connects.Add(1) + fmt.Fprint(c, "HTTP/1.1 200 Connection Established\r\n\r\n") + if _, err := http.ReadRequest(br); err != nil { + return + } + body := "upstream request failed" + fmt.Fprintf(c, "HTTP/1.1 502 Bad Gateway\r\nContent-Type: text/plain; charset=utf-8\r\n"+ + "Content-Length: %d\r\nConnection: close\r\n\r\n%s", len(body), body) +} + +// fakeDevModePage serves a rendered page on host that opens the dev channel +// the way the Mendix client does, and accepts the WebSocket handshake as the +// runtime does (101) — so any failure of the channel is the route's. +func fakeDevModePage(t *testing.T, host string) *httptest.Server { + t.Helper() + mux := http.NewServeMux() + mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/html") + fmt.Fprint(w, `Mendix - Narrate +

Narrate

The page rendered.

+`) + }) + mux.HandleFunc("/mxdevtools/", func(w http.ResponseWriter, r *http.Request) { + if !strings.EqualFold(r.Header.Get("Upgrade"), "websocket") { + http.Error(w, "websocket only", http.StatusBadRequest) + return + } + sum := sha1.Sum([]byte(r.Header.Get("Sec-WebSocket-Key") + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11")) + conn, rw, err := http.NewResponseController(w).Hijack() + if err != nil { + return + } + fmt.Fprintf(rw, "HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n"+ + "Sec-WebSocket-Accept: %s\r\n\r\n", base64.StdEncoding.EncodeToString(sum[:])) + _ = rw.Flush() + // Hold the socket open, as the runtime's server-push channel does, + // until the browser closes the page. + go func() { _, _ = io.Copy(io.Discard, conn); conn.Close() }() + }) + ln, err := net.Listen("tcp", net.JoinHostPort(host, "0")) + if err != nil { + t.Skipf("cannot listen on %s: %v", host, err) + } + srv := httptest.NewUnstartedServer(mux) + srv.Listener.Close() + srv.Listener = ln + srv.Start() + t.Cleanup(srv.Close) + return srv +} diff --git a/cmd/mxcli/docker/screenshot.go b/cmd/mxcli/docker/screenshot.go index 581bacdaa3..4ed2147136 100644 --- a/cmd/mxcli/docker/screenshot.go +++ b/cmd/mxcli/docker/screenshot.go @@ -87,7 +87,9 @@ func CaptureScreenshot(opts ScreenshotOptions) error { args := append(append([]string{}, prefix...), screenshotArgs(opts)...) cmd := exec.Command(bin, args...) - cmd.Env = os.Environ() // carries PLAYWRIGHT_BROWSERS_PATH + // os.Environ carries PLAYWRIGHT_BROWSERS_PATH; a page on this machine is + // reached directly, not through an egress proxy (ako/mxcli#1800). + cmd.Env = directRouteEnv(os.Environ(), opts.URL) out := &syncBuffer{} cmd.Stdout = out cmd.Stderr = out diff --git a/cmd/mxcli/docker/screenshot_login.go b/cmd/mxcli/docker/screenshot_login.go index 745368f997..1878e1a5b2 100644 --- a/cmd/mxcli/docker/screenshot_login.go +++ b/cmd/mxcli/docker/screenshot_login.go @@ -222,7 +222,7 @@ func LoginAndSaveStorage(opts LoginOptions) error { } cmd := exec.Command(node, scriptPath, pkgDir, opts.AppURL, opts.Username, opts.Password, opts.StoragePath) // PLAYWRIGHT_BROWSERS_PATH resolves Chromium; NODE_PATH resolves Playwright. - cmd.Env = append(os.Environ(), "NODE_PATH="+playwrightNodePath(node)) + cmd.Env = directRouteEnv(append(os.Environ(), "NODE_PATH="+playwrightNodePath(node)), opts.AppURL) if exe := headlessShellPath(); exe != "" { cmd.Env = append(cmd.Env, "MXCLI_CHROMIUM_EXE="+exe) } @@ -313,7 +313,7 @@ func SignOut(opts SignOutOptions) error { timeout = 60 * time.Second } cmd := exec.Command(node, scriptPath, opts.AppURL, opts.StoragePath) - cmd.Env = append(os.Environ(), "NODE_PATH="+playwrightNodePath(node)) + cmd.Env = directRouteEnv(append(os.Environ(), "NODE_PATH="+playwrightNodePath(node)), opts.AppURL) if exe := headlessShellPath(); exe != "" { cmd.Env = append(cmd.Env, "MXCLI_CHROMIUM_EXE="+exe) } diff --git a/cmd/mxcli/syntax/features_microflow.go b/cmd/mxcli/syntax/features_microflow.go index 505b2bcde6..9ec83d8eff 100644 --- a/cmd/mxcli/syntax/features_microflow.go +++ b/cmd/mxcli/syntax/features_microflow.go @@ -198,6 +198,11 @@ func init() { "-- CHANGE, an object parameter) is CE0117 / CE7247 — check --references\n" + "-- reports it as MDL-RETRIEVE03. For one object, retrieve it instead:\n" + "-- RETRIEVE $O FROM MyModule.Order WHERE [MyModule.Order_Customer = $Customer] FIRST;\n" + + "-- A self-reference (MyModule.Order_Parent from Order to Order) is followed\n" + + "-- from its FROM end: a Reference gives the one object, a ReferenceSet the\n" + + "-- list. Using the one object as a list (an aggregate, a list operation, a\n" + + "-- LOOP, a list parameter) is CE0013 / CE0097 / CE0100 / CE0117 —\n" + + "-- check --references reports it as MDL-RETRIEVE01.\n" + "-- * $Obj/System.owner and $Obj/System.changedBy are not associations a\n" + "-- retrieve can follow (CE0136, MDL-RETRIEVE02). Use\n" + "-- RETRIEVE $U FROM System.User WHERE [id = $Obj/System.owner] FIRST;\n" + diff --git a/docs-site/src/appendixes/error-messages.md b/docs-site/src/appendixes/error-messages.md index 89142f7bc7..de2c405290 100644 --- a/docs-site/src/appendixes/error-messages.md +++ b/docs-site/src/appendixes/error-messages.md @@ -422,6 +422,22 @@ and a list parameter build. For one object, retrieve it from the database: retrieve $L from Mod.Line where [Mod.Line_Header = $h] first; ``` +The reverse is reported as **MDL-RETRIEVE01** (ako/mxcli#1793): the one object a +Reference gives from its FROM entity, taken as a list. An aggregate of it (`count`, `sum`, +…) is CE0013 "Input variable 'P' must be of type 'List'.", a list operation (`head`, +`filter`, …) CE0097, a `loop` CE0100 and a list parameter CE0117 — measured on 11.13.0. +A **self-reference** (`Mod.Order_Parent` from `Mod.Order` to `Mod.Order`) is always +followed from its FROM end, from an `Order` and from a specialization of it alike: a +Reference gives the one object, a Reference set the list. `range`, `add`, `remove` and +`clear` over the object are not reported — mx check does not type their variable. + +``` +$P = retrieve over Mod.Order_Parent from $Order (a Mod.Order) is one Mod.Order — a +Reference followed from its FROM end, which is how Mendix follows a self-reference — so +`count` cannot take it: mxbuild rejects it with CE0013 "Input variable 'P' must be of +type 'List'." [MDL-RETRIEVE01] +``` + A flow already stored with such a member describes with a `-- WARNING` on the change line. `create or modify` will not splice over it: without the header it rebuilds the flow (which repairs it), and under `mdl 1;` it refuses with the drop-and-create advice. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index a6992e720c..9f8d93d2cd 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -54,7 +54,7 @@ The tables below show exactly which features are available on each Mendix versio | Calculated attributes | `CALCULATED BY Module.Microflow` | Yes | Yes | Yes | Yes | | Entity generalization | `EXTENDS Module.ParentEntity` | Yes | Yes | Yes | Yes | | ALTER ENTITY | `ALTER ENTITY ... ADD/DROP/RENAME` | -- | Yes | Yes | Yes | -| View entities | `CREATE VIEW ENTITY ... AS SELECT` | -- | -- | Yes | Yes | +| View entities | `CREATE VIEW ENTITY ... AS SELECT` | -- | -- | 10.21+ | Yes | ### Microflows @@ -62,11 +62,26 @@ The tables below show exactly which features are available on each Mendix versio |---------|-----------|-----|-------|-------|-------| | Basic microflows | `CREATE MICROFLOW ... BEGIN ... END` | Yes | Yes | Yes | Yes | | Loop in branches | `LOOP inside IF/ELSE` | Yes | Yes | Yes | Yes | +| Loops over a list, while loops | `LOOP $Item IN $List ...`, `WHILE ...` | 9.0.4+ | Yes | Yes | Yes | +| Show page with arguments | `SHOW PAGE Module.Page (Param = $Value)` | 9.7+ | Yes | Yes | Yes | +| Calls in a task queue | `CALL MICROFLOW ... IN QUEUE ...`, `CALL JAVA ACTION ... IN QUEUE ...` | 9.10+ | Yes | Yes | Yes | +| Find and filter by expression | `FIND $List WHERE ...`, `FILTER $List WHERE ...` | 9.0.3+ | Yes | Yes | Yes | +| Call external action | `CALL EXTERNAL ACTION Module.Service.Action (...)` | 9.19+ (with arguments: --) | Yes (with arguments: 10.2+) | Yes | Yes | +| Range of a list | `RANGE $List OFFSET ... LIMIT ...` | 9.24+ | Yes | Yes | Yes | | SEND REST REQUEST | `SEND REST REQUEST Module.Service.Op` | -- | 10.1+ | Yes | Yes | -| Execute database query | `EXECUTE DATABASE QUERY ...` | -- | -- | Yes | Yes | -| Show page with params | `SHOW PAGE ... WITH PARAMS` | -- | -- | -- | Yes | +| SEND REST REQUEST arguments, result, body | `... WITH (...)`, `$R = SEND REST REQUEST ...`, `... BODY $Var` | -- | 10.2+, 10.4+, 10.5+ | Yes | Yes | +| Transform JSON with a data transformer | `$Out = TRANSFORM $Json WITH Module.Transformer` | -- | -- | -- | 11.4+ | +| Execute database query | `EXECUTE DATABASE QUERY ...` | 9.24+ | Yes | Yes | Yes | | REST query parameters | `SEND REST REQUEST with QUERY params` | -- | -- | -- | Yes | -| DB query runtime connection | `EXECUTE DATABASE QUERY CONNECTION ...` | -- | -- | -- | Yes | +| DB query runtime connection | `EXECUTE DATABASE QUERY ... CONNECTION (...)` | -- | -- | -- | 11.1+ | + +Below its version, each of these is refused by `check -p` and by `exec` before +anything is written, with MDL-VERSION01, wherever it sits in the flow and in what +an `alter microflow` or `alter nanoflow` inserts. The versions are the Mendix +Model SDK's own for the type each writes; before this, `check -p` passed them and +the write refused them part-way through the script +([ako/mxcli#1811](https://github.com/ako/mxcli/issues/1811)). A `FIND` or +`FILTER` `BY Member = value` writes the older find and filter every version has. ### Pages @@ -81,6 +96,8 @@ The tables below show exactly which features are available on each Mendix versio | Page parameters (entity) | `Params: ( $Item: Module.Entity )` | 9.4+ | Yes | Yes | Yes | | Page parameters (primitive) | `Params: ( $Qty: Integer )` | -- | -- | -- | 11.6+ | | Page variables | `Variables: ( ... )` | -- | 10.20+ | 10.20+ | Yes | +| Snippet parameters | `CREATE SNIPPET ... (Params: ( ... ))` | 9.21+ | Yes | Yes | Yes | +| Workflow actions on a button | `action: open user task`, `open workflow`, `call workflow ...`; `complete task '...'` | 9.0.2+; 9.19+ | Yes | Yes | Yes | | Design properties (Atlas v3) | `DesignProperties: (...)` | -- | -- | -- | Yes | ::: tip Widget Templates @@ -101,10 +118,16 @@ Pluggable widget templates are currently extracted from Mendix 11.6. When used o | Feature | MDL Syntax | 10.0+ | 10.1+ | 10.4+ | 10.6+ | 11.0+ | |---------|-----------|-------|-------|-------|-------|-------| | OData client | `CREATE CONSUMED ODATA SERVICE` | Yes | Yes | Yes | Yes | Yes | -| Business events | `CREATE BUSINESS EVENT SERVICE` | Yes | Yes | Yes | Yes | Yes | -| REST client (basic) | `CREATE CONSUMED REST SERVICE ... ( ... ) { OPERATION ... }` | -- | Yes | Yes | Yes | Yes | +| Business events (9.24+) | `CREATE BUSINESS EVENT SERVICE` | Yes | Yes | Yes | Yes | Yes | +| REST client with no operation | `CREATE CONSUMED REST SERVICE ... ( ... ) { }` | -- | Yes | Yes | Yes | Yes | +| REST client basic authentication | `authentication: basic (...)` | -- | 10.2+ | Yes | Yes | Yes | +| REST client operations | `... { OPERATION ... }` | -- | -- | Yes | Yes | Yes | +| REST client from an OpenAPI document | `CREATE CONSUMED REST SERVICE ... ( OpenAPI: ... )` | -- | -- | -- | 10.21+ | Yes | | REST client headers | `Headers: ('Name': 'Value')` | -- | -- | Yes | Yes | Yes | -| Database Connector | `CREATE DATABASE CONNECTION` | -- | -- | -- | Yes | Yes | +| Database Connector | `CREATE DATABASE CONNECTION` | Yes | Yes | Yes | Yes | Yes | +| Database Connector queries | `CREATE DATABASE CONNECTION ... { QUERY ... }` | -- | -- | -- | 10.12+ | Yes | +| Scheduled events | `CREATE SCHEDULED EVENT ... ( Repeat: ... )` | Yes (9.12+) | Yes | Yes | Yes | Yes | +| Module JAR dependencies, exclusions | `ALTER MODULE ... ADD JAR DEPENDENCY`, `... ADD EXCLUSION` | Yes; 10.12+ | Yes | Yes | Yes | Yes | | REST client query params | `QUERY $param: Type` | -- | -- | -- | -- | Yes | A REST client operation's `body: json from $X` is stored as Mendix 11's JSON @@ -117,8 +140,13 @@ body on Mendix 10 too, and Studio Pro could not open the project. | Feature | MDL Syntax | 9.x | 10.0+ | |---------|-----------|-----|-------| -| Basic workflows | `CREATE WORKFLOW` | Yes | Yes | -| User tasks | `USER TASK ... OUTCOMES (...)` | Yes | Yes | +| Basic workflows | `CREATE WORKFLOW`, `ALTER WORKFLOW` | 9.0.2+ | Yes | +| User tasks | `USER TASK ... OUTCOMES (...)` | 9.19+ | Yes | +| A workflow parameter | `PARAMETER $WorkflowContext: Module.Entity` | 9.18+ | Yes | +| A jump | `JUMP TO ...` | 9.0.5+ | Yes | +| Calling a workflow from a workflow | `CALL WORKFLOW ...`; with `(WorkflowContext = ...)` | 9.6+; 9.18+ | Yes | +| An overview page | `OVERVIEW PAGE ...` | 9.11+ | Yes | +| Annotations | `ANNOTATION '...'` | 9.15+ | Yes | | Multi-user tasks | `MULTI USER TASK ...` | 9.22+ | Yes | | Multi-user task rules majority and threshold | `DECIDE BY MAJORITY ...`, `DECIDE BY THRESHOLD ...` | No | Yes | | Waiting for all users | `AWAIT ALL USERS` | No | 10.2+ | @@ -130,15 +158,29 @@ body on Mendix 10 too, and Studio Pro could not open the project. | A timer boundary event of no kind | `BOUNDARY EVENT TIMER ...` | No | 10.14+ (MDL-WF07 refuses it from 11.0) | | Interrupting and non-interrupting timer boundary events | `BOUNDARY EVENT [NON] INTERRUPTING TIMER ...` | No | 10.20+ | | Locking and unlocking a workflow | `LOCK WORKFLOW ...`, `UNLOCK WORKFLOW ...` | No | Yes | +| Starting a workflow, setting a task outcome, opening a user task or a workflow (microflow) | `CALL WORKFLOW`, `SET TASK OUTCOME`, `OPEN USER TASK`, `OPEN WORKFLOW` | 9.0.2+ | Yes | +| Aborting and restarting a workflow (microflow) | `WORKFLOW OPERATION ABORT ...`, `WORKFLOW OPERATION RESTART ...` | 9.2+ | Yes | +| Pausing and continuing a workflow (microflow) | `WORKFLOW OPERATION PAUSE ...`, `WORKFLOW OPERATION CONTINUE ...` | 9.3+ | Yes | +| Getting workflow data (microflow) | `GET WORKFLOW DATA ... AS ...` | 9.10+ | Yes | +| Retrying and unpausing a workflow (microflow) | `WORKFLOW OPERATION RETRY ...`, `WORKFLOW OPERATION UNPAUSE ...` | 9.12+ | Yes | +| Notifying a workflow (microflow) | `NOTIFY WORKFLOW ... TARGET ...` | No | Yes | +| Getting an object's workflows, or a workflow's activity records (microflow) | `GET WORKFLOWS FOR ...`, `GET WORKFLOW ACTIVITY RECORDS ...` | No | 10.4+ | Every write refuses an element whose type the project's Mendix version does not have yet, naming the type, the version that introduced it and the project's version ("this Mendix 9.24.30 project cannot store Workflows$WaitForNotificationActivity (Mendix 10.0.0+)"). The versions are each type's own, from the Mendix Model SDK. Below their versions, the wait -activities, the timer boundary events and a lock or unlock naming a workflow are -also refused up front, by `check -p` and by `exec` before anything is written, -with MDL-VERSION01. Before this, such a type was written, and the project did not load +activities, the timer boundary events, a lock or unlock naming a workflow, and +the workflow actions of a microflow (in `create microflow` and in what `alter +microflow` inserts) are also refused up front, by `check -p` and by `exec` +before anything is written, with MDL-VERSION01 +([ako/mxcli#1787](https://github.com/ako/mxcli/issues/1787)). So is a workflow itself on 9.0.0 and 9.0.1, which +predate Workflows$Workflow (9.0.2): `create workflow`, `create or modify +workflow` and `alter workflow` ([ako/mxcli#1809](https://github.com/ako/mxcli/issues/1809)), and what a +workflow states whose version is later still, listed above: on a stamped 9.x +project the write refuses every workflow with a parameter below 9.18 and every +user task below 9.19 ([ako/mxcli#1811](https://github.com/ako/mxcli/issues/1811)). Before this, such a type was written, and the project did not load (`TypeCacheUnknownTypeException`, [ako/mxcli#1774](https://github.com/ako/mxcli/issues/1774)). Where an older version stores the same thing in another shape (the parallel split's end marker, user tasks), that shape is written instead, as listed under @@ -160,6 +202,15 @@ longer loaded, and 11.1 and 11.2 failed `mx check` with CE6671 and CE6689 | Navigation profiles | `ALTER NAVIGATION ...` | Yes | Yes | | Menu items | `MENU ITEM ...` | Yes | Yes | | Home pages | `HOME PAGE Module.Page` | Yes | Yes | +| A progressive web app's settings | `PROGRESSIVE WEB APP ( ... )` | 9.0.3+ | Yes | +| A native profile's home page | `CREATE OR MODIFY NAVIGATION NativePhone HOME PAGE ...` | 9.4+ | Yes | +| Menu item icons | `MENU ITEM '...' ( ..., Icon: Module.Collection.icon )` | 9.20+ | Yes | +| A not-found page | `NOT FOUND PAGE Module.Page` | No | 10.13+ | + +Each "+" version above that is not a whole major is the Model SDK's version of a +type or property mxcli writes for the feature, with no older shape; below it, +`check -p` and `exec` refuse the statement with MDL-VERSION01 before anything is +written ([ako/mxcli#1811](https://github.com/ako/mxcli/issues/1811)). ### OQL (View Entity queries) @@ -185,7 +236,7 @@ The Mendix metamodel evolves across versions. The reflection data shows ~42% typ ### View Entities (10.x vs 11.x) -View entities exist in both 10.18+ and 11.x, but the BSON structure differs. In Mendix 10.x, the `OqlViewEntitySource` object has an `Oql` field that stores the OQL query inline (in addition to the separate `ViewEntitySourceDocument`). Mendix 11.0 removed the inline `Oql` field. The writer detects the project version and includes the inline field for 10.x projects. +View entities arrived in Mendix 10.18 with the query stored inline. mxcli stores it in a separate `ViewEntitySourceDocument`, which the entity's source names; both are 10.21 types, so `create view entity` needs 10.21 and is refused by `check -p` below it ([ako/mxcli#1811](https://github.com/ako/mxcli/issues/1811)); before, `check -p` passed it on 10.18 to 10.20 and the write refused it. The BSON structure also differs between 10.x and 11.x. In Mendix 10.x, the `OqlViewEntitySource` object has an `Oql` field that stores the OQL query inline (in addition to the separate `ViewEntitySourceDocument`). Mendix 11.0 removed the inline `Oql` field. The writer detects the project version and includes the inline field for 10.x projects. ### Page Parameters (9.4+, with an 11.5 tail) diff --git a/docs-site/src/language/activity-types.md b/docs-site/src/language/activity-types.md index 9eb8edcec3..7ab66076b6 100644 --- a/docs-site/src/language/activity-types.md +++ b/docs-site/src/language/activity-types.md @@ -116,6 +116,13 @@ RETRIEVE $Lines FROM $Order/Sales.OrderLine_Order; RETRIEVE $Customer FROM $Order/Sales.Order_Customer; ``` +What it binds follows the association: from the FROM entity of a Reference, the one +object it points at; from its TO entity, the list of objects pointing at it (one object +when the owner is Both); over a Reference set, a list from either end. A self-reference +is followed from its FROM end, so a Reference from an entity to itself gives the one +object and a Reference set the list. `mxcli check -p --references` reports a list used +as an object (MDL-RETRIEVE03) and an object used as a list (MDL-RETRIEVE01). + The association path uses the format `$Variable/Module.AssociationName`. A bare name (`$Order/Order_Customer`) is resolved against the variable's entity — through its generalizations, from either end of the association — when exactly one diff --git a/docs-site/src/reference/capabilities.md b/docs-site/src/reference/capabilities.md index 34984d86d1..a3c6a078e1 100644 --- a/docs-site/src/reference/capabilities.md +++ b/docs-site/src/reference/capabilities.md @@ -32,7 +32,7 @@ Everything mxcli can do, organized by use case. |---|---|---| | Create entity | `CREATE PERSISTENT ENTITY` | Full support | | Create non-persistent entity | `CREATE NON-PERSISTENT ENTITY` | Full support | -| Create view entity (OQL) | `CREATE VIEW ENTITY ... AS SELECT` | 10.18+ | +| Create view entity (OQL) | `CREATE VIEW ENTITY ... AS SELECT` | 10.21+ | | Modify entity | `ALTER ENTITY ... ADD/MODIFY/DROP ATTRIBUTE` | Full support | | Create association | `CREATE ASSOCIATION ... FROM ... TO` | Full support | | Create enumeration | `CREATE ENUMERATION` | Full support | diff --git a/docs-site/src/tools/debug-microflows.md b/docs-site/src/tools/debug-microflows.md index b04db8ecad..27e9882552 100644 --- a/docs-site/src/tools/debug-microflows.md +++ b/docs-site/src/tools/debug-microflows.md @@ -49,9 +49,9 @@ Under the hood the debugger spans two runtime APIs, which `mxcli debug` hides: | `mxcli debug unbreak --activity <#n\|caption>` | Clear a breakpoint | | `mxcli debug breaks` | List the breakpoints set this session (name → object ID) | | `mxcli debug paused` | Paused microflows + full state (variables) | -| `mxcli debug inspect [--list] [--flow ]` | Inspect one variable of a paused flow; `--list` inspects a list variable (`get_list`) | +| `mxcli debug inspect [--list] [--flow ]` | Inspect one variable of a paused flow — an object's members, or a primitive's value (String, enumeration, …); `--list` inspects a list variable (`get_list`) | | `mxcli debug step [over\|into\|out] [--flow ]` | Advance one step (default `over`) | -| `mxcli debug continue [--all]` | Resume the paused flow (or all with `--all`) | +| `mxcli debug continue [--flow ] [--all]` | Resume the paused flow (or all with `--all`) | **Selecting an activity:** `--activity '#2'` (the index from `activities`) or a caption substring like `--activity 'Retrieve'` (case-insensitive; must match one). @@ -108,3 +108,10 @@ server-side (browser console only) — only `INFO`/`WARNING`/`ERROR` reach `runt shutdown; a manual `enable` is your responsibility. - **Use the same `-p` for every command.** The session token and breakpoint record live under `/.mxcli/`. +- **A request the runtime refuses is an error, with the runtime's reason** — for + example `the runtime refused add_breakpoint (status 2): Failed to parse + condition: …` for a bad `--if`. An activity or flow the *running* app does not + have yet (rebuild or restart after changing the model) and `Invalid session + token` (the runtime restarted, or another debugger such as Studio Pro started a + session — run `mxcli debug enable`) are the other common ones. A command that + printed success was accepted by the runtime. diff --git a/docs-site/src/tools/playwright.md b/docs-site/src/tools/playwright.md index ef7cd331e4..baf092155b 100644 --- a/docs-site/src/tools/playwright.md +++ b/docs-site/src/tools/playwright.md @@ -129,6 +129,31 @@ Module.Page"`) against the user role's (`describe user role R`). A warning banner or validation message is shown but does not fail the page. +### An app on this machine, by its own host name, behind a proxy + +An app on this machine is reached directly, whatever name the URL gives it. This +matters where the environment sets a proxy — a dev container, a Claude Code +session — and the app is checked by a host name of its own, such as an +`/etc/hosts` entry `frontend.local` for `127.0.0.1` (the usual way to give two +local apps separate cookie jars). Chromium takes its proxy from `http_proxy` / +`https_proxy`, and for a WebSocket it prefers the HTTPS proxy even for `ws://`. +With only `https_proxy` set and `no_proxy` listing `localhost` and `127.0.0.1`, +the page loaded directly while the runtime's dev channel, +`ws://frontend.local:8180/mxdevtools/`, went to the proxy, which cannot reach +this machine and answered 502. A page that rendered failed on two console errors +(ako/mxcli#1800): + +``` + ERR WebSocket connection to 'ws://frontend.local:8180/mxdevtools/' failed: Error during WebSocket handshake: Unexpected response code: 502 +``` + +The browser that `playwright check`, `run --page-check` and `run --screenshot` +start now adds a host that resolves only to this machine (loopback, or an address +of one of its interfaces) to `no_proxy`. Nothing is ignored: the channel connects, +as it does on `127.0.0.1`, and a console error the page does have still fails it. +A remote URL, such as a `--hub` preview, still goes through the proxy. A browser +of your own needs the same entry in `no_proxy`. + ### Signing in | Flags | Signs in as | diff --git a/docs-site/src/tools/refresh-catalog.md b/docs-site/src/tools/refresh-catalog.md index b94798624f..15ae297e5a 100644 --- a/docs-site/src/tools/refresh-catalog.md +++ b/docs-site/src/tools/refresh-catalog.md @@ -71,7 +71,13 @@ mxcli -p app.mpr -c "REFRESH CATALOG FULL FORCE" The catalog is cached in `.mxcli/catalog.db` next to the MPR file. On subsequent refreshes, mxcli checks whether the project has changed and only rebuilds if necessary (unless `FORCE` is specified). -A catalog with the MDL source index (`REFRESH CATALOG FULL SOURCE`) keeps it on every later rebuild, including the ones `SEARCH` and `SHOW REFERENCES` start when the project has changed. Those rebuilds describe again only the documents that changed, and the documents that reference them. Every other row is carried over from the cache, so a rebuild after one edit costs about as much as `REFRESH CATALOG FULL`. Everything is described again when something outside the described documents changes (a folder, a module, the project settings, a Java action), when another mxcli build wrote the cache, and with `FORCE`. +A catalog with the MDL source index (`REFRESH CATALOG FULL SOURCE`) keeps it on every later rebuild, including the ones `SEARCH` and `SHOW REFERENCES` start when the project has changed. Those rebuilds describe again only the documents that changed, and the documents that reference them. An entity counts as changed when its own definition does, not when another entity of its module does; a change to an association or annotation describes every entity of that module again. Every other row is carried over from the cache, so a rebuild after one edit costs about as much as `REFRESH CATALOG FULL`. Everything is described again when something outside the described documents changes (a folder, a module, the project settings, a Java action), when another mxcli build wrote the cache, and with `FORCE`. + +### The unit store (MPR v2) + +An MPR v2 project keeps each document in its own file under `mprcontents/`, and a refresh reads all of them. mxcli keeps a compressed copy of those files in `.mxcli/cache/unit-contents.db` and reads a document from its own file only when the file has changed since mxcli last read it (its size or modification time differs, or the `.mpr` records another content hash). A refresh after a save therefore opens one or two files instead of thousands. That matters most on Windows, where a virus scanner inspects every file a process opens. + +The store is a cache: deleting it costs one slower refresh, and a store that cannot be read or written is ignored. It is about half the size of `mprcontents/`, and its directory has its own `.gitignore`, so it is never committed even where `.mxcli/` is not ignored. ## When to Refresh diff --git a/docs-site/src/tutorial/validation.md b/docs-site/src/tutorial/validation.md index 3fe1551298..60b357d165 100644 --- a/docs-site/src/tutorial/validation.md +++ b/docs-site/src/tutorial/validation.md @@ -44,7 +44,7 @@ This catches everything Level 1 catches, plus: - Page layouts that don't exist in the project - Association endpoints pointing to missing entities - A plain `create` whose name already exists in the project -- A statement the project's Mendix version cannot store (`MDL-VERSION01`, e.g. `create view entity` below 10.18) — the same version gates `exec` refuses with +- A statement the project's Mendix version cannot store (`MDL-VERSION01`, e.g. `create view entity` below 10.21) — the same version gates `exec` refuses with This is the check you should run before executing a script. It's fast (reads the project but doesn't modify it) and catches most mistakes. diff --git a/docs-wiki/bug-patterns/duplicate-resolver-drift.md b/docs-wiki/bug-patterns/duplicate-resolver-drift.md index b651590b81..9de8538025 100644 --- a/docs-wiki/bug-patterns/duplicate-resolver-drift.md +++ b/docs-wiki/bug-patterns/duplicate-resolver-drift.md @@ -116,6 +116,28 @@ source: the AST package's own declarations, `generated/metamodel`, a `go/parser` scan of the builders for the keys they read. The reliable version of this test is not "the lists agree" but "the list agrees with something nobody edits". +**A version floor is two answers too.** The write refuses a type newer than the +project from the Model SDK's data (`canon.UndeclaredTypeError`), and `check -p` +refuses a statement from an `sdk/versions` row. Every type with a write floor and +no gate is a script `check -p` passes and `exec` refuses part-way. That class has +come back four times (#1434, #1771, #1774, #1787). In the last one, `alter +microflow` had no `case` in the gate switch at all. The sweep starts from the +types the converter writes, not from the statements someone remembered. The +guard is the third source above: a test pins each type gate's row to +`canon.TypeIntroduced`, which nobody edits by hand. + +It came back twice more the same day (#1809, #1811), and the second time the +sweep was made a test: `TestTypeGates_CoverEveryWrittenType` scans the write +paths for every storage type they construct and fails for one above 9.0.0 that +no gate covers and no listed reason explains. Two lessons came with it. The +write's floor is the latest of the types *and the properties* a shape stores a +value in — the properties usually decide (a workflow parameter's 9.6 type has a +9.18 `Name`) — so the pin reads `canon.PropertyDeclared` too. And the floor can +be measured without an old Mendix: stamp a newer app's `_MetaData` version down +and exec a probe with the unfixed build, since the gate and the write read only +that field. `mx check` is no witness for a property: it tolerates one the +version does not declare. + **The tell is that the fix for the reported instance is obviously incomplete.** When a symptom's cause is "this switch was missing a case", the next question is how many other switches answer the same question — the answer has repeatedly been diff --git a/docs/11-proposals/PROPOSAL_microflow_debugger.md b/docs/11-proposals/PROPOSAL_microflow_debugger.md index 1f2acf3a78..e6a4cbcc01 100644 --- a/docs/11-proposals/PROPOSAL_microflow_debugger.md +++ b/docs/11-proposals/PROPOSAL_microflow_debugger.md @@ -56,12 +56,19 @@ debugger state only: | Action | Params | Notes | |---|---|---| | `start_session` | `{breakpoints: []}` | **no token**; response has `result.session_token` | -| `add_breakpoint` | `{microflow_name, object_id, condition?}` | `object_id` = activity model GUID | +| `add_breakpoint` | `{microflow_name, object_id, condition}` | `object_id` = activity model GUID; `condition` is required, `""` for none (a nanoflow uses `nanoflow_name`) | | `remove_breakpoint` | `{object_id}` | | | `get_paused_microflows` | `{}` | flows + all in-scope variables | -| `get_object` | `{debug_id, variable_name}` | inspect one variable | +| `get_object` | `{debug_id, variable_name}` | inspect one **object** variable; a primitive is refused, its value is in `get_paused_microflows` | | `step_over` / `step_into` / `step_out` | `{debug_id}` | | -| `continue` / `continue_all` | `{}` | resume | +| `continue` | `{debug_id}` | resume one paused flow | +| `continue_all` | `{}` | resume every paused flow; refused when none is paused | + +Every reply is HTTP 200; success or refusal is only in the envelope — +`{"result":…,"status":0}` or `{"result":{},"message":…,"status":2}`. Measured on runtimes +11.13.0 and 11.15.0 (ako/mxcli#1782): the table above first listed `condition` as +optional and `continue` with `{}`, and both are refused ("Missing property: condition", +"Missing property: debug_id"). Auth quirk (documented): the debugger endpoint accepts only `X-Debugger-Authentication: base64(pass)` — raw password, `Basic`, and `Bearer` all 401, diff --git a/mdl-examples/bug-tests/1100-loop-variable-typing.mdl b/mdl-examples/bug-tests/1100-loop-variable-typing.mdl index 9a247584c4..7fe4935858 100644 --- a/mdl-examples/bug-tests/1100-loop-variable-typing.mdl +++ b/mdl-examples/bug-tests/1100-loop-variable-typing.mdl @@ -86,15 +86,21 @@ end; -- A loop over an association retrieve. The iterator is typed from the entity at -- the far end of Request_Reporter, which is resolved rather than read off the -- statement — an expression path does not spell its intermediate entity. -create or modify microflow Probe.SUB_LoopAssoc ($Req: Probe.Request) +-- +-- It starts from the Reporter, the association's TO entity, which is what makes +-- it a list: the Requests pointing at it. This loop used to start from $Req, the +-- FROM entity, which gives the ONE Reporter — mx check failed it with CE0100 +-- "'reps' is of type Probe.Reporter, but should be of type List." (measured on +-- 11.13.0), and check -p reports it as MDL-RETRIEVE01 since ako/mxcli#1793. +create or modify microflow Probe.SUB_LoopAssoc ($Rep: Probe.Reporter) returns String begin - declare $emails String = ''; - retrieve $reps from $Req/Probe.Request_Reporter; - loop $rep in $reps begin - set $emails = $emails + $rep/Email; + declare $codes String = ''; + retrieve $reqs from $Rep/Probe.Request_Reporter; + loop $r in $reqs begin + set $codes = $codes + $r/Code; end loop; - return $emails; + return $codes; end; -- A loop over a filtered list: FILTER carries the element type through, so the diff --git a/mdl-examples/bug-tests/1323-set-object-variable.mdl b/mdl-examples/bug-tests/1323-set-object-variable.mdl index 160cb1f532..60200bce5e 100644 --- a/mdl-examples/bug-tests/1323-set-object-variable.mdl +++ b/mdl-examples/bug-tests/1323-set-object-variable.mdl @@ -37,11 +37,23 @@ begin end if; end; --- 2. A self-association retrieve is typed as a LIST by Mendix, so `set` on it --- is a Change list Replace (ako/mxcli#949) and stays accepted. +-- 2. A self-association REFERENCE SET retrieve is a list, so `set` on it is a +-- Change list Replace (ako/mxcli#949) and stays accepted. +-- +-- This item used to show the same over F1323.Node_Parent, a Reference, as +-- "typed as a LIST by Mendix". It is not: Mendix follows a self-reference +-- from its FROM end, so a Reference retrieve is the one object (measured on +-- 11.13.0, ako/mxcli#1793: a loop over one is CE0100 "'P' is of type +-- M.Order, but should be of type List."). The Change list Replace mxcli +-- wrote passed mx check only because mx check does not type a Change list's +-- variable (an `add` to the object is clean too). `set` over it is now +-- refused like `set` over any object variable (CE7247). This form, over the +-- reference set, was measured on 11.13.0 with mx check reporting 0 errors. +create association F1323.Node_Siblings from F1323.Node to F1323.Node type ReferenceSet; + create microflow F1323.ReplaceList ($Start: F1323.Node, $Other: F1323.Node) begin - retrieve $Cursor from $Start/F1323.Node_Parent; - retrieve $Next from $Other/F1323.Node_Parent; + retrieve $Cursor from $Start/F1323.Node_Siblings; + retrieve $Next from $Other/F1323.Node_Siblings; set $Cursor = $Next; end; diff --git a/mdl-examples/bug-tests/1787-workflow-actions-below-their-version.mdl b/mdl-examples/bug-tests/1787-workflow-actions-below-their-version.mdl new file mode 100644 index 0000000000..870d27ee90 --- /dev/null +++ b/mdl-examples/bug-tests/1787-workflow-actions-below-their-version.mdl @@ -0,0 +1,116 @@ +mdl 1; +-- ako/mxcli#1787: the workflow microflow actions an older Mendix does not +-- have passed `mxcli check -p`, and only the write refused them +-- (canon.UndeclaredTypeError, #1774) — part-way through the script, after the +-- statements before them were written; with --continue-on-error the rest ran +-- without that microflow. On 9.24.30: +-- +-- this Mendix 9.24.30 project cannot store Microflows$GetWorkflowsAction +-- (Mendix 10.4.0+): the type did not exist yet in this version … +-- +-- `check -p` and exec's pre-flight now refuse each with MDL-VERSION01, naming +-- the statement and the version, wherever it sits in the microflow and in +-- what an `alter microflow` inserts. Each floor is the type's introduction in +-- mendixmodelsdk 4.117.0, pinned to the type data by +-- TestTypeGates_FloorIsTheTypeIntroduction: +-- +-- call workflow, set task outcome, open user task, open workflow 9.0.2 +-- workflow operation abort / restart 9.2 +-- workflow operation pause / continue 9.3 +-- get workflow data 9.10 +-- workflow operation retry / unpause 9.12 +-- notify workflow 10.0 +-- get workflows, get workflow activity records 10.4 +-- +-- (lock / unlock workflow, 9.21, are gated at 10.0 by the selection that +-- names their workflow: 1774-workflow-types-below-their-version.mdl.) +-- +-- EXPECTED: `mxcli check` passes this file. Measured on fresh apps +-- (`mxcli new … --skip-build`), check -p, exec, then mx check: +-- 9.24.30.51827 MDL-VERSION01 for WF_Notify (wait for notification, +-- #1774), MF_Notify (10.0), MF_GetWorkflows (10.4), +-- MF_ActivityRecords (10.4) and the alter microflow +-- (10.4); exec writes nothing (App.mpr byte-identical). +-- Before: only WF_Notify. Without WF_Notify, check -p +-- passed the script, and exec refused MF_Notify at write +-- time; --continue-on-error --no-check wrote the other +-- five statements, mx check 0 errors, so MF_Older's +-- actions (9.0.2 to 9.12) load on 9.24. +-- 10.24.24.119653 none, before and after; exec writes everything, mx +-- check 0 errors. + +create module Probe1787; + +create persistent entity Probe1787.Request ( + Title: String(100) +); + +create microflow Probe1787.ACT_Validate ($WorkflowContext: Probe1787.Request) +begin + log info node 'Probe1787' 'validate'; +end; + +create workflow Probe1787.WF_Approve + parameter $WorkflowContext: Probe1787.Request + display 'Approve' +begin + call microflow Probe1787.ACT_Validate as validate; +end workflow; + +-- The target of `notify workflow`: a wait for notification is itself 10.0. +create workflow Probe1787.WF_Notify + parameter $WorkflowContext: Probe1787.Request + display 'Notify' +begin + wait for notification WaitForApproval; +end workflow; + +-- 9.0.2 to 9.12: accepted on 9.24 and later. `set task outcome` draws the +-- MDL-WORKFLOW10 warning; the claim it suggests +-- (System.WorkflowUserTask_Assignees) is CE1613 on 9.24.30, so it is left out. +create microflow Probe1787.MF_Older ( + $Request: Probe1787.Request, + $Wf: System.Workflow, + $UserTask: System.WorkflowUserTask +) +begin + $Started = call workflow Probe1787.WF_Approve ($Request); + $Context = get workflow data $Wf as Probe1787.WF_Approve; + set task outcome $UserTask 'Approve'; + open user task $UserTask; + open workflow $Wf; + workflow operation pause $Wf; + workflow operation continue $Wf; + workflow operation restart $Wf; + workflow operation abort $Wf reason 'no longer needed'; + workflow operation retry $Wf; + workflow operation unpause $Wf; +end; + +create microflow Probe1787.MF_Notify ($Wf: System.Workflow) +returns Boolean as $Notified +begin + $Notified = notify workflow $Wf target Probe1787.WF_Notify.WaitForApproval; + return $Notified; +end; + +create microflow Probe1787.MF_GetWorkflows ($Request: Probe1787.Request) +begin + $Workflows = get workflows for $Request; +end; + +-- Nested in a decision: the gate walks branches, loops and error handlers. +create microflow Probe1787.MF_ActivityRecords ($Wf: System.Workflow) +begin + if $Wf != empty then + $Records = get workflow activity records $Wf; + end if; +end; + +-- What an alter inserts is gated as the same statement in a create: MF_Older +-- is accepted on 9.24, this insert into it is not. +alter microflow Probe1787.MF_Older { + insert after $Started begin + $Running = get workflows for $Request; + end; +}; diff --git a/mdl-examples/bug-tests/1793-self-reference-retrieve-is-the-object.mdl b/mdl-examples/bug-tests/1793-self-reference-retrieve-is-the-object.mdl new file mode 100644 index 0000000000..6e4b3d4098 --- /dev/null +++ b/mdl-examples/bug-tests/1793-self-reference-retrieve-is-the-object.mdl @@ -0,0 +1,56 @@ +mdl 1; +-- ako/mxcli#1793: a retrieve over a self-reference — a Reference from an entity +-- to itself — binds the ONE object at its FROM end. The stored retrieve records +-- no direction, and Mendix follows a self-reference from its FROM end. The full +-- build used to type it as a list (the start variable is an instance of the TO +-- entity too, so it read as a reverse traversal) unless an object use followed, +-- and check --references said nothing, so a list use was accepted, written, and +-- failed the build. Measured on 11.13.0: +-- +-- retrieve $P from $Order/MyFirstModule.Bug1793Order_Parent; +-- $n = count $P; -- CE0013 "Input variable 'P' must be of type 'List'." +-- $h = head $P; -- CE0097 "The selected 'P' variable must be of type List." +-- loop $x in $P … -- CE0100 "'P' is of type MyFirstModule.Bug1793Order, but should be of type List." +-- +-- check -p now refuses those as MDL-RETRIEVE01, naming the CE code, and the full +-- build types $P as the object whatever the use. A Reference SET from an entity to +-- itself is a list, and stays one. +-- +-- This script is the working spelling: the object used as an object (from the +-- entity and from a specialization of it), the reference set used as a list. +-- Expected: check -p passes, exec creates everything, mx check reports 0 errors. +create persistent entity MyFirstModule.Bug1793Order ( Code: String(50), Amount: Integer ); +create persistent entity MyFirstModule.Bug1793SpecialOrder extends MyFirstModule.Bug1793Order ( Extra: String(20) ); +create association MyFirstModule.Bug1793Order_Parent + from MyFirstModule.Bug1793Order to MyFirstModule.Bug1793Order; +create association MyFirstModule.Bug1793Order_Related + from MyFirstModule.Bug1793Order to MyFirstModule.Bug1793Order type ReferenceSet; + +-- The parent: one object. Change it, read a member of it. +create or modify microflow MyFirstModule.MF_Bug1793_Parent ($Order: MyFirstModule.Bug1793Order) +returns Integer as $a +begin + retrieve $Parent from $Order/MyFirstModule.Bug1793Order_Parent; + change $Parent ( Code = 'P' ); + declare $a Integer = $Parent/Amount; + return $a; +end; + +-- The same from a specialization of the entity. +create or modify microflow MyFirstModule.MF_Bug1793_ParentOfSpecial ($Special: MyFirstModule.Bug1793SpecialOrder) +begin + retrieve $Parent from $Special/MyFirstModule.Bug1793Order_Parent; + change $Parent ( Code = 'S' ); +end; + +-- The related orders: a reference set is a list. Count it, loop it. +create or modify microflow MyFirstModule.MF_Bug1793_Related ($Order: MyFirstModule.Bug1793Order) +returns Integer as $n +begin + retrieve $Related from $Order/MyFirstModule.Bug1793Order_Related; + loop $Item in $Related begin + change $Item ( Amount = 1 ); + end loop; + $n = count $Related; + return $n; +end; diff --git a/mdl-examples/bug-tests/1805-property-sets-mixed-odata-and-domain-model.mdl b/mdl-examples/bug-tests/1805-property-sets-mixed-odata-and-domain-model.mdl new file mode 100644 index 0000000000..e6983d0602 --- /dev/null +++ b/mdl-examples/bug-tests/1805-property-sets-mixed-odata-and-domain-model.mdl @@ -0,0 +1,98 @@ +mdl 1; +-- ako/mxcli#1805: on a Mendix 11.14.0 project mxcli wrote, `mxcli fix +-- property-sets` reported seven property names the elements of a type +-- disagree on, among them +-- +-- MIXED ODataPublish$EntitySet.ReadMode: on 42 of 60 element(s); without it: … [ODataPublish$PublishedODataService2 F1AdminApi] +-- MIXED ODataPublish$PublishedMicroflow.Parameters: on 2 of 5 element(s); only on: … [ODataPublish$PublishedODataService2 F1OpsApi] +-- MIXED DomainModels$NoGeneralization.HasOwnerAttr: on 81 of 84 element(s); without it: … [DomainModels$DomainModel] +-- MIXED DomainModels$DateTimeAttributeType.LocalizeDate: on 15 of 16 element(s); without it: … [DomainModels$DomainModel] +-- +-- An entity set whose statement states no mode was written without ReadMode, +-- InsertMode, UpdateMode and DeleteMode, and an action whose microflow takes +-- no parameters without Parameters. Studio Pro writes all five on every +-- element; `mx convert -p -s` fills an absent one with ReadSource, +-- ChangeNotSupported and the empty list, which is what is now written. +-- +-- The domain-model names are not written by this script: no writer of today +-- leaves them out. They are elements an older mxcli stored, which every later +-- write passed through; measured rows now fill them, on the next write of their +-- domain model or with `mxcli fix property-sets`. The statements below run +-- every domain-model write path, so a regression in one of them shows here. +-- +-- Run on a fresh app (10.24.24, 11.13.0 or 11.15.0), then: +-- mxcli fix property-sets --dry-run -p App.mpr -> no MIXED line +-- mx convert -p -s on a copy -> adds no key to anything mxcli wrote +-- mx check -> 0 errors + +create module Bug1805; +create module Bug1805b; + +-- Domain model: create, create or modify, add / modify attribute, association, +-- an alter that rewrites a sibling, rename, move. +create persistent entity Bug1805.Plain (Code: String(50) unique error message 'Code is unique', At: DateTime); +create persistent entity Bug1805.Audited (Code: String(50) unique error message 'Code is unique', CreatedDate: AutoCreatedDate, ChangedDate: AutoChangedDate); +create non-persistent entity Bug1805.Transient (WhenAt: DateTime); +create or modify persistent entity Bug1805.Plain (Code: String(50) unique error message 'Code is unique', At: DateTime, Extra: Integer); +alter entity Bug1805.Plain add attribute Later: DateTime; +alter entity Bug1805.Plain modify attribute Later: DateTime not localized; +create association Bug1805.Plain_Audited from Bug1805.Plain to Bug1805.Audited; +alter entity Bug1805.Audited set documentation 'Rewrites the domain model around Plain.'; +rename entity Bug1805.Transient to Transient2; +create persistent entity Bug1805.Mover (D: DateTime); +move entity Bug1805.Mover to Bug1805b; + +-- Published OData: an entity set with every mode stated and one with none; an +-- action with a parameter and one without. +create microflow Bug1805.Act_NoParams () returns Boolean as $Ok +begin + declare $Ok Boolean = true; + return $Ok; +end; + +create microflow Bug1805.Act_Params ($Note: String) returns Boolean as $Ok +begin + declare $Ok Boolean = true; + return $Ok; +end; + +create published odata service Bug1805.Api ( + path: 'odata/bug1805/', + version: '1.0.0', + ODataVersion: OData4, + namespace: 'Bug1805.Api', + Authentication: (basic) +) +{ + publish entity Bug1805.Plain as 'Plains' ( + ReadMode: source, InsertMode: source, UpdateMode: source, DeleteMode: source + ) + expose ( Code as 'code' (KEY, Filterable, Sortable) ); + + publish entity Bug1805.Audited as 'Auditeds' + expose ( Code as 'code' (KEY, Filterable, Sortable) ); + + publish microflow Bug1805.Act_NoParams as 'NoParams'; + publish microflow Bug1805.Act_Params as 'WithParams'; +}; + +-- The modify path writes the same converter's output over the stored service. +create or modify published odata service Bug1805.Api ( + path: 'odata/bug1805/', + version: '1.0.1', + ODataVersion: OData4, + namespace: 'Bug1805.Api', + Authentication: (basic) +) +{ + publish entity Bug1805.Plain as 'Plains' ( + ReadMode: source, InsertMode: source, UpdateMode: source, DeleteMode: source + ) + expose ( Code as 'code' (KEY, Filterable, Sortable) ); + + publish entity Bug1805.Audited as 'Auditeds' + expose ( Code as 'code' (KEY, Filterable, Sortable) ); + + publish microflow Bug1805.Act_NoParams as 'NoParams'; + publish microflow Bug1805.Act_Params as 'WithParams'; +}; diff --git a/mdl-examples/bug-tests/1809-workflow-below-its-version.mdl b/mdl-examples/bug-tests/1809-workflow-below-its-version.mdl new file mode 100644 index 0000000000..c1cb524681 --- /dev/null +++ b/mdl-examples/bug-tests/1809-workflow-below-its-version.mdl @@ -0,0 +1,50 @@ +mdl 1; +-- ako/mxcli#1809: Workflows$Workflow is a 9.0.2 type (mendixmodelsdk +-- 4.117.0), and the write refuses it below that (canon.UndeclaredTypeError, +-- #1774). The `workflows basic` floor said 9.0.0 and only `alter workflow` +-- applied it, so on 9.0.0 or 9.0.1 `create workflow` passed `mxcli check -p` +-- and the write then refused it part-way through the script: +-- +-- this Mendix 9.0.1 project cannot store Workflows$Flow (Mendix 9.0.2+); +-- Workflows$Workflow (Mendix 9.0.2+): the types did not exist yet … +-- +-- `check -p` and exec's pre-flight now report MDL-VERSION01 for each of the +-- three statements below ("create workflow requires Mendix 9.0.2+"), and the +-- floor is pinned to the type data by TestTypeGates_FloorIsTheTypeIntroduction. +-- +-- EXPECTED: `mxcli check` passes this file. Measured (no 9.0.x mxbuild is +-- cached, so the 9.0.1 run is a 9.24.30 app whose _MetaData version was set to +-- 9.0.1.0, which is all the gate and the write read): +-- 9.0.1 (stamped) before: check -p clean; exec refused WF_Basic at write +-- time ("cannot store … Workflows$Workflow (Mendix +-- 9.0.2+)"). After: MDL-VERSION01 for statements 4, 5 and +-- 6; exec writes nothing (App.mpr byte-identical). +-- 9.24.30.51827 none, before and after; exec writes everything, mx +-- check 0 errors. + +create module Probe1809; + +create persistent entity Probe1809.Request ( + Title: String(100) +); + +create microflow Probe1809.ACT_Validate ($WorkflowContext: Probe1809.Request) +begin + log info node 'Probe1809' 'validate'; +end; + +create workflow Probe1809.WF_Basic + parameter $WorkflowContext: Probe1809.Request + display 'Basic' +begin + call microflow Probe1809.ACT_Validate as validate; +end workflow; + +create or modify workflow Probe1809.WF_Basic + parameter $WorkflowContext: Probe1809.Request + display 'Basic' +begin + call microflow Probe1809.ACT_Validate as validate; +end workflow; + +alter workflow Probe1809.WF_Basic { set (Display: 'Basic, altered'); }; diff --git a/mdl-examples/bug-tests/1811-actions-below-their-version.mdl b/mdl-examples/bug-tests/1811-actions-below-their-version.mdl new file mode 100644 index 0000000000..c516ba5570 --- /dev/null +++ b/mdl-examples/bug-tests/1811-actions-below-their-version.mdl @@ -0,0 +1,199 @@ +mdl 1; +-- ako/mxcli#1811: microflow activities and document shapes the write refuses +-- below their Mendix version (canon.UndeclaredTypeError and the property +-- floor, #1774) passed `mxcli check -p`, and exec refused them part-way +-- through the script. On 9.24.30, before: +-- +-- this Mendix 9.24.30 project cannot store Rest$RestOperation.Method +-- (Mendix 10.4.0+); Rest$RestOperation.ResponseHandling (Mendix 10.3.0+) … +-- +-- `check -p` and exec's pre-flight now report each with MDL-VERSION01. The +-- floors are the write's — the latest type or property the shape stores — +-- pinned to the Model SDK's data by TestTypeGates_FloorIsTheTypeIntroduction, +-- and TestTypeGates_CoverEveryWrittenType fails for a written type above 9.0.0 +-- that no gate covers. The Mendix 11 shapes are in +-- 1811-mendix-11-shapes-below-their-version.mdl. +-- +-- find / filter … where 9.0.3 +-- loop over a list, while 9.0.4 +-- show page with arguments 9.7 +-- create scheduled event 9.12 +-- a workflow parameter / a user task 9.18 / 9.19 +-- range 9.24 +-- create database connection (with a query) 10.0 (10.12) +-- send rest request (result variable) 10.1 (10.4) +-- a consumed REST service's operation 10.4 +-- a not-found page 10.13 +-- page variables 10.20 +-- create view entity (its source document) 10.21 +-- +-- EXPECTED: `mxcli check` passes this file. Measured on fresh apps +-- (`mxcli new …`), check -p, exec, then mx check: +-- 9.24.30.51827 MDL-VERSION01 for statements 15 (database connection, +-- 10.0), 16 (REST service, 10.1), 17 and 19 (send rest +-- request, 10.1), 21 (not-found page, 10.13), 22 (page +-- variable, 10.20) and 23 (view entity, 10.21); exec writes +-- nothing (App.mpr byte-identical). Before: only 16 and 23 +-- (at 10.18); without those two, check -p passed the script +-- and exec refused the database connection at write time +-- ("cannot store DatabaseConnector$DatabaseConnection. +-- ConnectionInput (Mendix 10.0.0+); …DatabaseQuery.QueryType +-- (Mendix 10.12.0+) …"). Statements 1 to 14 alone (the 9.x +-- floors) pass check -p, exec writes them, mx check 0 errors. +-- 10.24.24.119653 none, before and after; exec writes everything, mx check +-- 0 errors. + +create module Probe1811; + +create persistent entity Probe1811.Sale ( + Code: String(20), + Amount: Decimal +); + +create non-persistent entity Probe1811.Row ( + Code: String(20) +); + +create constant Probe1811.DbUrl ( Type: String, DefaultValue: 'jdbc:postgresql://localhost:5432/probe' ); + +create microflow Probe1811.MF_FindFilter ($Sales: List of Probe1811.Sale) +begin + $Big = find $Sales where $currentObject/Amount > 100; + $Small = filter $Sales where $currentObject/Amount <= 100; +end; + +create microflow Probe1811.MF_Loops ($Sales: List of Probe1811.Sale) +begin + declare $Count Integer = 0; + loop $Sale in $Sales + begin + set $Count = $Count + 1; + end loop; + while $Count > 0 + begin + set $Count = $Count - 1; + end while; +end; + +create page Probe1811.P_Sale (Title: 'Sale', Layout: Atlas_Core.Atlas_Default, Params: ($Sale: Probe1811.Sale)) { + layoutgrid lgSale { + row { + column (DesktopWidth: autofill) { + dataview dvSale (DataSource: $Sale) { + textbox txtCode (Attribute: Code) + } + } + } + } +}; + +create microflow Probe1811.MF_ShowSale ($Sale: Probe1811.Sale) +begin + show page Probe1811.P_Sale(Sale = $Sale); +end; + +create microflow Probe1811.MF_Job () +begin + log info node 'Probe1811' 'job'; +end; + +create scheduled event Probe1811.SE_Nightly ( + Microflow: Probe1811.MF_Job, + Repeat: Daily, + HourOfDay: 4, + MinuteOfHour: 0, + TimeZone: Server, + Enabled: false +); + +create microflow Probe1811.ACT_Review ($WorkflowContext: Probe1811.Sale) +begin + log info node 'Probe1811' 'review'; +end; + +create page Probe1811.P_Task (Title: 'Task', Layout: Atlas_Core.Atlas_Default, Params: ($WorkflowUserTask: System.WorkflowUserTask)) { + dynamictext txtTask (Content: 'Review the sale') + actionbutton btnDone (Caption: 'Done', Action: complete task 'Done') +}; + +create workflow Probe1811.WF_Review + parameter $WorkflowContext: Probe1811.Sale + display 'Review' +begin + call microflow Probe1811.ACT_Review as review; + user task approve 'Approve' + page Probe1811.P_Task + targeting users xpath [id = '[%CurrentUser%]'] + outcomes 'Done' { }; +end workflow; + +create microflow Probe1811.MF_Range ($Sales: List of Probe1811.Sale) +begin + $Page = range $Sales offset 0 limit 10; +end; + +-- 10.0: the connection's input is a DatabaseConnector$ConnectionString; +-- 10.12: a query's QueryType. (execute database query, 9.24, is left out: +-- mx check wants the External Database Connector module for it.) +create database connection Probe1811.Db ( + Type: 'PostgreSQL', + ConnectionString: @Probe1811.DbUrl, + Username: @Probe1811.DbUrl, + Password: @Probe1811.DbUrl +) { + query AllRows ( + Sql: 'SELECT code FROM rows', + Returns: Probe1811.Row, + Map: ( + Code = code + ) + ) +}; + +-- 10.4: every operation stores its method and response handling. +create consumed rest service Probe1811.Api ( + BaseUrl: 'https://example.com', + authentication: none +) +{ + operation GetStatus ( + method: get, + path: '/status', + headers: ('Accept': '*/*'), + response: none + ) +}; + +create microflow Probe1811.MF_Rest () +begin + send rest request Probe1811.Api.GetStatus; +end; + +create microflow Probe1811.MF_Spliced () +begin + declare $Marker String = 'before'; +end; + +alter microflow Probe1811.MF_Spliced { + insert after $Marker begin + send rest request Probe1811.Api.GetStatus; + end; +}; + +create page Probe1811.P_NotFound (Title: 'Not found', Layout: Atlas_Core.Atlas_Default) { + dynamictext txtNotFound (Content: 'Not found') +}; + +alter navigation Responsive { + set not found page Probe1811.P_NotFound; +}; + +alter page Probe1811.P_NotFound { + add variables $ShowHelp: boolean = false; +}; + +create view entity Probe1811.V_SaleCodes ( + Code: String(20) +) as ( + select s.Code as Code from Probe1811.Sale as s +); diff --git a/mdl-examples/bug-tests/1811-mendix-11-shapes-below-their-version.mdl b/mdl-examples/bug-tests/1811-mendix-11-shapes-below-their-version.mdl new file mode 100644 index 0000000000..aa69e10219 --- /dev/null +++ b/mdl-examples/bug-tests/1811-mendix-11-shapes-below-their-version.mdl @@ -0,0 +1,71 @@ +mdl 1; +-- ako/mxcli#1811: the Mendix 11 shapes of the sweep, which a 10.24 project +-- cannot store. Before, `mxcli check -p` passed each of them on 10.24.24 and +-- the write refused it part-way through exec; now check -p and exec's +-- pre-flight report MDL-VERSION01 for each, naming the version: +-- +-- a REST operation's query parameter 11.0 +-- execute database query … connection ( … ) 11.1 +-- transform … with (a data transformer) 11.4 +-- +-- (The 10.x and 9.x shapes are in 1811-actions-below-their-version.mdl.) +-- +-- EXPECTED: `mxcli check` passes this file. Measured on a fresh 10.24.24.119653 +-- app: check -p reports statements 6, 7 and 8; exec writes nothing (the +-- project is byte-identical); mx check 0 errors. Before: check -p clean, and exec refused the +-- REST service at write time (Rest$RestOperation.QueryParameters, Mendix +-- 11.0.0+), after writing the module and the entities. + +create module Probe1811m; + +create non-persistent entity Probe1811m.Row ( + Code: String(20) +); + +create constant Probe1811m.DbUrl ( Type: String, DefaultValue: 'jdbc:postgresql://localhost:5432/probe' ); + +create database connection Probe1811m.Db ( + Type: 'PostgreSQL', + ConnectionString: @Probe1811m.DbUrl, + Username: @Probe1811m.DbUrl, + Password: @Probe1811m.DbUrl +) { + query ByCode ( + Sql: 'SELECT code FROM rows WHERE code = {code}', + Parameters: ( code: string default 'x' ), + Returns: Probe1811m.Row, + Map: ( + Code = code + ) + ) +}; + +create microflow Probe1811m.MF_Job () +begin + log info node 'Probe1811m' 'job'; +end; + +create consumed rest service Probe1811m.Search ( + BaseUrl: 'https://example.com', + authentication: none +) +{ + operation Find ( + method: get, + path: '/search', + query: ($q: string), + response: none + ) +}; + +create microflow Probe1811m.MF_QueryOverride () +begin + -- The query argument list is written out: with none, the connection + -- arguments are read as the query's (a parser issue outside this fix). + $Rows = execute database query Probe1811m.Db.ByCode (code = 'x') connection (Url = 'jdbc:postgresql://other:5432/probe'); +end; + +create microflow Probe1811m.MF_Transform ($Json: String) +begin + $Out = transform $Json with Probe1811m.Flatten; +end; diff --git a/mdl-examples/bug-tests/pagecheck-1800-host-name-behind-proxy.mdl b/mdl-examples/bug-tests/pagecheck-1800-host-name-behind-proxy.mdl new file mode 100644 index 0000000000..e320fa9c40 --- /dev/null +++ b/mdl-examples/bug-tests/pagecheck-1800-host-name-behind-proxy.mdl @@ -0,0 +1,65 @@ +-- ako/mxcli#1800: `mxcli playwright check http://frontend.local:8180/` failed a +-- page that rendered, on two console errors, while the same check against +-- http://127.0.0.1:8180/ passed: +-- +-- ERR WebSocket connection to 'ws://frontend.local:8180/mxdevtools/' failed: Error during WebSocket handshake: Unexpected response code: 502 +-- +-- frontend.local is an /etc/hosts name for 127.0.0.1 and the app's configured +-- root URL. The runtime answers the dev channel's handshake 101 on that name +-- (curl with the proxy bypassed). The 502 is the environment's HTTPS proxy: +-- Chromium sends ws:// to the HTTPS proxy (RFC 6455 4.1.3), no_proxy lists +-- localhost and 127.0.0.1 but not the app's name, and a proxy elsewhere cannot +-- reach this machine's loopback ("502 upstream request failed"). The page's +-- http:// loads went direct, because only https_proxy is set. +-- +-- No /etc/hosts edit is needed where the machine already has a name for +-- 127.0.0.1 that no_proxy does not list (a Claude Code container has `vm`). +-- Against a blank 11.15.0 app, in an environment that sets https_proxy: +-- +-- mxcli exec mdl-examples/bug-tests/pagecheck-1800-host-name-behind-proxy.mdl -p app.mpr +-- mxcli run --local --detach -p app.mpr --db-type hsqldb --app-port 18180 +-- mxcli playwright check http://vm:18180/ http://vm:18180/p/brokenprobe -p app.mpr +-- +-- Before the fix: / FAILs with the two 502 handshake errors; /p/brokenprobe +-- FAILs with three (one 502, one genuine). After it: / passes with +-- console-errors=0; /p/brokenprobe still FAILs, on its genuine error only: +-- +-- ERR [Client] genuine console error from JSA_LogError (ako/mxcli#1800 control) +-- FAIL 1 console error(s) +-- +-- The browser the check starts now adds a host that resolves only to this +-- machine to no_proxy; nothing is ignored, the channel connects. +mdl 1; + +create or modify configuration 'Default' ( + ApplicationRootUrl: 'http://vm:18180/' +); + +-- The genuine-error control: a page that renders and logs a real console error. +create non-persistent entity MyFirstModule.ProbeItem ( + Label: String(50) +); + +create or modify javascript action MyFirstModule.JSA_LogError () returns Boolean +as $$ + console.error('genuine console error from JSA_LogError (ako/mxcli#1800 control)'); + return Promise.resolve(true); +$$; + +create or modify nanoflow MyFirstModule.DS_ProbeItem () +returns MyFirstModule.ProbeItem as $Item +begin + $Logged = call javascript action MyFirstModule.JSA_LogError(); + $Item = create MyFirstModule.ProbeItem (Label = 'probe item'); + return $Item; +end; + +create or modify page MyFirstModule.BrokenProbe ( + Title: 'Broken probe', + Layout: Atlas_Core.Atlas_TopBar, + Url: 'brokenprobe' +) { + dataview dvProbe (DataSource: nanoflow MyFirstModule.DS_ProbeItem) { + dynamictext txtProbe (Content: 'Broken probe page rendered') + } +}; diff --git a/mdl/backend/modelsdk/issue1805_odata_property_sets_test.go b/mdl/backend/modelsdk/issue1805_odata_property_sets_test.go new file mode 100644 index 0000000000..d39a63e126 --- /dev/null +++ b/mdl/backend/modelsdk/issue1805_odata_property_sets_test.go @@ -0,0 +1,233 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "os" + "path/filepath" + "sort" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/modelsdk/codec" + mmpr "github.com/mendixlabs/mxcli/modelsdk/mpr" +) + +// ako/mxcli#1805: on a Mendix 11.14.0 project mxcli wrote, `mxcli fix +// property-sets` reported +// +// MIXED ODataPublish$EntitySet.ReadMode: on 42 of 60 element(s); without it: … [ODataPublish$PublishedODataService2 F1AdminApi] +// MIXED ODataPublish$PublishedMicroflow.Parameters: on 2 of 5 element(s); only on: … [ODataPublish$PublishedODataService2 F1OpsApi] +// +// The published-service writer wrote an entity set's ReadMode, InsertMode, +// UpdateMode and DeleteMode only when the statement stated them, and an +// action's Parameters only when the microflow had any. Studio Pro writes all +// five on every element (the 11.14.0 service in testdata/testapp-views), and +// `mx convert -p -s` on 10.24.24, 11.13.0 and 11.15.0 fills an absent one with +// ReadSource, ChangeNotSupported and an empty marker-3 list — so the writer now +// writes those when the statement leaves them out, and the stored meaning of a +// service mxcli wrote before does not change. + +// studioProEntitySetKeys is the property-name set Studio Pro 11.14.0 wrote on +// the entity set of MyFirstModule.CarOdataApi in testdata/testapp-views. +func studioProEntitySetKeys(t *testing.T) []string { + t.Helper() + b, err := os.ReadFile(filepath.Join("..", "..", "..", "testdata", "testapp-views", "mprcontents", + "6e", "16", "6e16f1f7-1207-4d70-8847-c369a7dd360a.mxunit")) + if err != nil { + t.Skipf("testdata/testapp-views not available: %v", err) + } + sets := bson.Raw(b).Lookup("EntitySets").Array() + vals, err := sets.Values() + if err != nil || len(vals) != 2 { + t.Fatalf("subject: the Studio Pro service should hold one entity set, got %d (%v)", len(vals)-1, err) + } + return rawKeyNames(t, vals[1].Document()) +} + +func rawKeyNames(t *testing.T, d bson.Raw) []string { + t.Helper() + elems, err := d.Elements() + if err != nil { + t.Fatal(err) + } + keys := make([]string, 0, len(elems)) + for _, e := range elems { + keys = append(keys, e.Key()) + } + sort.Strings(keys) + return keys +} + +// listItems returns the documents of a stored Mendix list (marker first). +func listItems(t *testing.T, doc bson.Raw, key string) (marker bson.RawValue, items []bson.Raw) { + t.Helper() + v, err := doc.LookupErr(key) + if err != nil { + t.Fatalf("%s is absent", key) + } + vals, err := v.Array().Values() + if err != nil || len(vals) == 0 { + t.Fatalf("%s is not a marked list: %v", key, err) + } + for _, x := range vals[1:] { + items = append(items, x.Document()) + } + return vals[0], items +} + +func encodedPublishedService(t *testing.T, svc *model.PublishedODataService) bson.Raw { + t.Helper() + raw, err := (&codec.Encoder{}).Encode(publishedODataServiceToGen(svc, true)) + if err != nil { + t.Fatalf("encode: %v", err) + } + return raw +} + +func TestIssue1805_EntitySetWritesStudioProPropertySet(t *testing.T) { + want := studioProEntitySetKeys(t) + raw := encodedPublishedService(t, &model.PublishedODataService{ + Name: "ZzApi", + EntityTypes: []*model.PublishedEntityType{ + {Entity: "MyFirstModule.Stated", ExposedName: "Stated"}, + {Entity: "MyFirstModule.Unstated", ExposedName: "Unstated"}, + }, + EntitySets: []*model.PublishedEntitySet{ + {ExposedName: "Stated", EntityTypeName: "MyFirstModule.Stated", + ReadMode: "source", InsertMode: "source", UpdateMode: "source", DeleteMode: "source"}, + // `publish entity M.E as 'Unstated'` with no mode stated. + {ExposedName: "Unstated", EntityTypeName: "MyFirstModule.Unstated"}, + }, + }) + _, sets := listItems(t, raw, "EntitySets") + if len(sets) != 2 { + t.Fatalf("got %d entity sets, want 2", len(sets)) + } + for _, es := range sets { + name := es.Lookup("ExposedName").StringValue() + if got := rawKeyNames(t, es); strings.Join(got, ",") != strings.Join(want, ",") { + t.Errorf("entity set %s has property names\n %v\nStudio Pro writes\n %v\n"+ + "— `fix property-sets` reports ODataPublish$EntitySet as MIXED", name, got, want) + } + } + unstated := sets[1] + for key, typ := range map[string]string{ + "ReadMode": "ODataPublish$ReadSource", + "InsertMode": "ODataPublish$ChangeNotSupported", + "UpdateMode": "ODataPublish$ChangeNotSupported", + "DeleteMode": "ODataPublish$ChangeNotSupported", + } { + v, err := unstated.LookupErr(key) + if err != nil { + t.Errorf("unstated entity set has no %s", key) + continue + } + if got := v.Document().Lookup("$Type").StringValue(); got != typ { + t.Errorf("unstated %s = %s, want %s — what Mendix gives the absent property", key, got, typ) + } + } +} + +func TestIssue1805_PublishedMicroflowWritesItsParametersWhenThereAreNone(t *testing.T) { + raw := encodedPublishedService(t, &model.PublishedODataService{ + Name: "ZzApi", + Microflows: []*model.PublishedMicroflow{ + {Microflow: "MyFirstModule.Act_NoParams", ExposedName: "NoParams", ReturnTypeKind: "Boolean"}, + {Microflow: "MyFirstModule.Act_Params", ExposedName: "WithParams", ReturnTypeKind: "Boolean", + Parameters: []*model.PublishedMicroflowParameter{{ + MicroflowParameter: "MyFirstModule.Act_Params.Note", ExposedName: "note", DataTypeKind: "String", + }}}, + }, + }) + _, actions := listItems(t, raw, "Microflows") + if len(actions) != 2 { + t.Fatalf("got %d actions, want 2", len(actions)) + } + none, some := actions[0], actions[1] + if a, b := rawKeyNames(t, none), rawKeyNames(t, some); strings.Join(a, ",") != strings.Join(b, ",") { + t.Fatalf("an action without parameters has property names\n %v\nand one with them\n %v\n"+ + "— `fix property-sets` reports ODataPublish$PublishedMicroflow.Parameters as MIXED", a, b) + } + marker, params := listItems(t, none, "Parameters") + if m, ok := marker.Int32OK(); !ok || m != 3 || len(params) != 0 { + t.Errorf("Parameters of a parameterless action = marker %v + %d item(s), want the empty marker-3 list Mendix writes", marker, len(params)) + } +} + +// The census over a project written through both forms of each element, the +// way the reporter's services were: a fresh Mendix 11.14.0 project +// (testdata/testapp-views, Studio Pro-authored, every type uniform) gets a +// service whose entity sets state their modes and do not, and whose actions +// take parameters and do not. `fix property-sets` must find no ODataPublish +// name the elements of a type disagree on. +func TestIssue1805_PublishedServiceCensusHasNoMixedNames(t *testing.T) { + dst := t.TempDir() + if err := os.CopyFS(dst, os.DirFS("../../../testdata/testapp-views")); err != nil { + t.Fatalf("copy fixture: %v", err) + } + proj := filepath.Join(dst, "TestApp.mpr") + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + mod, err := b.GetModuleByName("MyFirstModule") + if err != nil || mod == nil { + t.Fatalf("GetModuleByName: %v", err) + } + svc := &model.PublishedODataService{ + ContainerID: mod.ID, Name: "Zz1805Api", Path: "odata/zz1805/", Version: "1.0.0", + ODataVersion: "OData4", Namespace: "MyFirstModule.Zz1805", + EntityTypes: []*model.PublishedEntityType{ + {Entity: "MyFirstModule.Car", ExposedName: "StatedCars"}, + {Entity: "MyFirstModule.VCar", ExposedName: "UnstatedCars"}, + }, + EntitySets: []*model.PublishedEntitySet{ + {ExposedName: "StatedCars", EntityTypeName: "MyFirstModule.Car", + ReadMode: "source", InsertMode: "source", UpdateMode: "source", DeleteMode: "source"}, + {ExposedName: "UnstatedCars", EntityTypeName: "MyFirstModule.VCar"}, + }, + Microflows: []*model.PublishedMicroflow{ + {Microflow: "MyFirstModule.Act_NoParams", ExposedName: "NoParams", ReturnTypeKind: "Boolean"}, + {Microflow: "MyFirstModule.Act_Params", ExposedName: "WithParams", ReturnTypeKind: "Boolean", + Parameters: []*model.PublishedMicroflowParameter{{ + MicroflowParameter: "MyFirstModule.Act_Params.Note", ExposedName: "note", DataTypeKind: "String", + }}}, + }, + } + if err := b.CreatePublishedODataService(svc); err != nil { + t.Fatalf("CreatePublishedODataService: %v", err) + } + // The modify path writes the same converter's output over the stored unit. + svc.Version = "1.0.1" + if err := b.UpdatePublishedODataService(svc); err != nil { + t.Fatalf("UpdatePublishedODataService: %v", err) + } + if err := b.Disconnect(); err != nil { + t.Fatalf("disconnect: %v", err) + } + + r, err := mmpr.Open(proj) + if err != nil { + t.Fatalf("open: %v", err) + } + t.Cleanup(func() { _ = r.Close() }) + ids, err := r.ListAllUnitIDs() + if err != nil { + t.Fatal(err) + } + census := mmpr.NewPropertySetCensus() + for _, id := range ids { + rep, err := r.StoredPropertySetGaps(id) + if err != nil { + t.Fatalf("unit %s: %v", id, err) + } + census.Count(rep) + } + for _, m := range census.Mixed() { + t.Errorf("MIXED %s.%s: on %d of %d element(s)", m.Type, m.Key, m.With, m.Elements) + } +} diff --git a/mdl/backend/modelsdk/odata_write.go b/mdl/backend/modelsdk/odata_write.go index c6b04ab876..08ef7bb96e 100644 --- a/mdl/backend/modelsdk/odata_write.go +++ b/mdl/backend/modelsdk/odata_write.go @@ -34,6 +34,10 @@ func init() { // every other part list in this document. codec.RegisterListMarker("ODataPublish$PublishedMicroflow", 3) codec.RegisterListMarker("ODataPublish$PublishedMicroflowParameter", 3) + // An action without parameters still writes the list, empty: Studio Pro + // serializes it on every action, and `mx convert -p -s` fills an absent one + // with [3] on 10.24.24, 11.13.0 and 11.15.0 (ako/mxcli#1805). + codec.RegisterMandatoryPropertyList("ODataPublish$PublishedMicroflow", "Parameters", 3) // AllowedModuleRoles empties as marker 1, not 3 — it is a BY_NAME role list, // not a part list. Without the entry, a service with no grants omits the key // entirely, and since the document is serialized wholesale that DELETES the @@ -464,13 +468,15 @@ func publishedMicroflowToGen(pm *model.PublishedMicroflow) element.Element { addStr(g, "ExposedName", pm.ExposedName) addStr(g, "AlternativeExposedName", "") addStr(g, "Microflow", pm.Microflow) - if len(pm.Parameters) > 0 { - params := make([]element.Element, 0, len(pm.Parameters)) - for _, p := range pm.Parameters { - params = append(params, publishedMicroflowParameterToGen(p)) - } - addPartList(g, "Parameters", params) - } + // Written with no parameters too, as the empty list (registered mandatory + // in init): an action without the key and one with it are two + // property-name sets Mendix's merge engine refuses to compare + // (ako/mxcli#1805). + params := make([]element.Element, 0, len(pm.Parameters)) + for _, p := range pm.Parameters { + params = append(params, publishedMicroflowParameterToGen(p)) + } + addPartList(g, "Parameters", params) if rt := dataTypeElement(pm.ReturnTypeKind, pm.ReturnTypeRef); rt != nil { addPart(g, "ReturnType", rt) } @@ -552,18 +558,17 @@ func publishedEntitySetToGen(es *model.PublishedEntitySet, entityTypeID string) if entityTypeID != "" { addIDRef(g, "EntityTypePointer", model.ID(entityTypeID)) } - if es.ReadMode != "" { - addPart(g, "ReadMode", odataReadModeToGen(es.ReadMode)) - } - if es.InsertMode != "" { - addPart(g, "InsertMode", odataChangeModeToGen(es.InsertMode)) - } - if es.UpdateMode != "" { - addPart(g, "UpdateMode", odataChangeModeToGen(es.UpdateMode)) - } - if es.DeleteMode != "" { - addPart(g, "DeleteMode", odataChangeModeToGen(es.DeleteMode)) - } + // All four modes, stated or not. Studio Pro writes every one on every entity + // set, so a set whose statement left one out had a property-name set Mendix's + // merge engine refuses to compare with its siblings' (ako/mxcli#1805). An + // unstated mode is the value Mendix gives the absent property — ReadSource + // and ChangeNotSupported, measured with `mx convert -p -s` on 10.24.24, + // 11.13.0 and 11.15.0, and what Studio Pro 11.14.0 creates — so writing it + // changes nothing the service does. + addPart(g, "ReadMode", odataReadModeToGen(es.ReadMode)) + addPart(g, "InsertMode", odataChangeModeToGen(es.InsertMode)) + addPart(g, "UpdateMode", odataChangeModeToGen(es.UpdateMode)) + addPart(g, "DeleteMode", odataChangeModeToGen(es.DeleteMode)) return g } diff --git a/mdl/catalog/builder_source.go b/mdl/catalog/builder_source.go index 786636d0eb..97df5cb7ae 100644 --- a/mdl/catalog/builder_source.go +++ b/mdl/catalog/builder_source.go @@ -87,8 +87,8 @@ func (b *Builder) buildSource() error { // Phase 1: Collect all items to describe (fast — uses cached lists) var items []sourceItem - // Entities. An entity is described from its module's domain model, the - // unit whose hash says whether it changed. + // Entities. An entity is described from its own subtree of its module's + // domain model (entityBasis), which is the unit that holds it. entityUnit := map[string]string{} dms, err := b.cachedDomainModels() if err == nil { @@ -210,11 +210,13 @@ func (b *Builder) buildSource() error { // Carry over the rows whose basis has not changed (#1790); describe the // rest. Without unit hashes nothing is reused and nothing is recorded. units := make([]string, len(items)) + parents := make([]string, len(items)) for i, it := range items { - units[i] = sourceUnitOf(entityUnit, it) + units[i] = it.id + parents[i] = sourceParentOf(entityUnit, it) } hashes := b.sourceBasis() - plan := b.planSourceReuse(items, units, hashes) + plan := b.planSourceReuse(items, units, parents, hashes) toDescribe := make([]sourceItem, 0, len(items)) for i, it := range items { if _, ok := plan.reused[i]; !ok { @@ -271,7 +273,7 @@ func (b *Builder) buildSource() error { stmt.Exec(it.qn, it.objType, row.text, it.moduleName, it.id) count++ } - if err := b.persistSourceBasis(items, units, hashes); err != nil { + if err := b.persistSourceBasis(items, units, parents, hashes); err != nil { return err } diff --git a/mdl/catalog/builder_source_reuse.go b/mdl/catalog/builder_source_reuse.go index be3bc174b8..4c41d01766 100644 --- a/mdl/catalog/builder_source_reuse.go +++ b/mdl/catalog/builder_source_reuse.go @@ -3,8 +3,12 @@ package catalog import ( + "crypto/sha256" "database/sql" + "encoding/hex" "fmt" + + "go.mongodb.org/mongo-driver/v2/bson" ) // Incremental CATALOG.SOURCE (ako/mxcli#1790). @@ -16,8 +20,10 @@ import ( // next build carries a document's row over when nothing its text depends on // has changed: // -// - its own unit's bytes are unchanged (an entity's unit is its module's -// domain model, so any change there re-describes all its entities); +// - its own unit's bytes are unchanged. An entity is not a unit: its basis +// is its own subtree of the domain model (#1803), and the rest of the +// domain model is a unit of its own that, when it changes, re-describes +// every entity in it; // - no document it references changed: a describe resolves names against // other documents (a called flow's return type, a data source's entity), // so a document whose refs name a changed one is described again; @@ -166,9 +172,91 @@ func (b *Builder) sourceBasis() map[string]string { hashes[id] = hash + "|" + string(u.ContainerID) } } + b.entityBasis(hashes) return hashes } +// entityBasis splits each domain model's basis per entity (#1803). Adding an +// attribute to one entity changes its domain model's unit, and with the unit +// as the basis every entity of the module was described again, along with +// everything that references any of them. An entity's describe reads only +// its own subtree — attributes, rules, indexes, handlers and access rules all +// live in it, and associations are not printed — so the entity is keyed on a +// hash of that subtree and its container, and the domain model's own entry +// covers the rest of it. A change there (an association, an annotation) +// still re-describes every entity of the module: planSourceReuse treats the +// domain model as each entity's parent. +// +// An entity whose subtree cannot be found keeps the whole domain model's +// hash, so it is described again whenever anything in its module changes. +func (b *Builder) entityBasis(hashes map[string]string) { + dms, err := b.cachedDomainModels() + if err != nil || len(dms) == 0 { + return + } + units, err := b.reader.ListRawUnitsByType("DomainModels$DomainModel") + if err != nil { + return + } + raw := map[string][]byte{} + for _, u := range units { + raw[string(u.ID)] = u.Contents + } + for _, dm := range dms { + dmID := string(dm.ID) + unitHash, ok := hashes[dmID] + if !ok { + continue + } + residual, entities, ok := splitDomainModel(raw[dmID]) + for _, ent := range dm.Entities { + h, found := entities[ent.Name] + if !ok || !found { + h = unitHash + } + hashes[string(ent.ID)] = h + "|" + dmID + } + if ok { + hashes[dmID] = residual + "|" + string(dm.ContainerID) + } + } +} + +// splitDomainModel hashes a domain model's entities by name, and everything +// else in it as one residual. ok is false when the contents do not parse. +func splitDomainModel(contents []byte) (residual string, entities map[string]string, ok bool) { + elems, err := bson.Raw(contents).Elements() + if err != nil { + return "", nil, false + } + rest := sha256.New() + entities = map[string]string{} + for _, e := range elems { + if e.Key() != "Entities" { + rest.Write(e) + continue + } + arr, isArr := e.Value().ArrayOK() + if !isArr { + return "", nil, false + } + vals, err := arr.Values() + if err != nil { + return "", nil, false + } + for _, v := range vals { + doc, isDoc := v.DocumentOK() + if !isDoc { + continue // the array's leading version marker + } + name, _ := doc.Lookup("Name").StringValueOK() + sum := sha256.Sum256(doc) + entities[name] = hex.EncodeToString(sum[:]) + } + } + return hex.EncodeToString(rest.Sum(nil)), entities, true +} + // sourceReusePlan says, per item, whether its previous row can be carried // over; reused[i] is that row's text. type sourceReusePlan struct { @@ -177,9 +265,11 @@ type sourceReusePlan struct { } // planSourceReuse decides which items keep their previous row. units maps -// each item to the unit it is described from; hashes is every unit's content -// hash now. -func (b *Builder) planSourceReuse(items []sourceItem, units []string, hashes map[string]string) sourceReusePlan { +// each item to the basis it is described from (its unit, or for an entity the +// entity itself), parents to the unit that holds it when that is another one +// ("" otherwise): a change to the parent changes the item. hashes is every +// basis now. +func (b *Builder) planSourceReuse(items []sourceItem, units, parents []string, hashes map[string]string) sourceReusePlan { prev := b.prevSource if prev == nil { return sourceReusePlan{reason: "no previous source index"} @@ -189,8 +279,11 @@ func (b *Builder) planSourceReuse(items []sourceItem, units []string, hashes map } itemUnit := map[string]bool{} - for _, u := range units { + for i, u := range units { itemUnit[u] = true + if parents[i] != "" { + itemUnit[parents[i]] = true + } } changed := map[string]bool{} for id, h := range hashes { @@ -203,6 +296,9 @@ func (b *Builder) planSourceReuse(items []sourceItem, units []string, hashes map changed[id] = true } } + changedItem := func(i int) bool { + return changed[units[i]] || (parents[i] != "" && changed[parents[i]]) + } for id := range changed { if !itemUnit[id] && !prev.described[id] { return sourceReusePlan{reason: "a unit other than a described document changed"} @@ -213,7 +309,7 @@ func (b *Builder) planSourceReuse(items []sourceItem, units []string, hashes map // from a changed unit, now and before (a rename or a delete). var names []string for i, it := range items { - if changed[units[i]] { + if changedItem(i) { names = append(names, it.qn) } } @@ -227,7 +323,7 @@ func (b *Builder) planSourceReuse(items []sourceItem, units []string, hashes map plan := sourceReusePlan{reused: map[int]previousRow{}} for i, it := range items { - if changed[units[i]] || dependents[it.qn] { + if changedItem(i) || dependents[it.qn] { continue } row, ok := prev.rows[it.id] @@ -278,13 +374,16 @@ func (b *Builder) sourceDependents(names []string) (map[string]bool, error) { // persistSourceBasis records what this build described from, so the next one // can tell what changed. -func (b *Builder) persistSourceBasis(items []sourceItem, units []string, hashes map[string]string) error { +func (b *Builder) persistSourceBasis(items []sourceItem, units, parents []string, hashes map[string]string) error { if hashes == nil || b.sourceFingerprint == "" { return nil } described := map[string]bool{} - for _, u := range units { + for i, u := range units { described[u] = true + if parents[i] != "" { + described[parents[i]] = true + } } ustmt, err := b.tx.Prepare(`INSERT OR REPLACE INTO source_units (UnitId, Hash, Described) VALUES (?, ?, ?)`) if err != nil { @@ -319,13 +418,12 @@ func sourceReuseSummary(reused, described int) string { return fmt.Sprintf("source: %d reused, %d described", reused, described) } -// sourceUnitOf maps entity element IDs to their domain model's unit; every -// other source document is its own unit. -func sourceUnitOf(entityUnit map[string]string, it sourceItem) string { +// sourceParentOf is the unit that holds an item when that is not the item +// itself: an entity's domain model. Every other source document is its own +// unit, and its basis is that unit. +func sourceParentOf(entityUnit map[string]string, it sourceItem) string { if it.objType == SourceEntity { - if u, ok := entityUnit[it.id]; ok { - return u - } + return entityUnit[it.id] } - return it.id + return "" } diff --git a/mdl/executor/catalog_source_incremental_pedapp_test.go b/mdl/executor/catalog_source_incremental_pedapp_test.go index 1c470b17ed..0907b81011 100644 --- a/mdl/executor/catalog_source_incremental_pedapp_test.go +++ b/mdl/executor/catalog_source_incremental_pedapp_test.go @@ -10,6 +10,7 @@ import ( "sort" "strconv" "strings" + "sync" "testing" ) @@ -154,8 +155,16 @@ end; `}, // A rename rewrites the callers too; the old name's row must go. {"rename a flow", "rename microflow MyFirstModule.MF_Chain004 to MF_Chain004b;"}, - // A domain model change re-describes its entities and what uses them. + // A domain model change re-describes the changed entity and what uses + // it, and not the module's other entities. {"alter an entity", "alter entity FeedbackModule.Feedback add attribute IncrementalProbe: String;"}, + {"alter one of two entities", "alter entity MyFirstModule.ProbeA add attribute Extra: String;"}, + // An association lives in the domain model outside every entity: its + // change re-describes every entity of the module. + {"add an association", "create association MyFirstModule.ProbeA_ProbeB from MyFirstModule.ProbeA to MyFirstModule.ProbeB;"}, + {"drop an association", "drop association MyFirstModule.ProbeA_ProbeB;"}, + // The old name's row must go, and what printed it must follow. + {"rename an entity", "rename entity MyFirstModule.ProbeB to ProbeC;"}, // A new folder is a unit no describe covers: everything is described. {"move a flow to a new folder", "move microflow MyFirstModule.MF_Chain001 to folder 'Probes';"}, // A move into an existing folder changes no unit's bytes, only its @@ -164,6 +173,25 @@ end; // A deleted document's row must not survive. {"drop a flow", "drop microflow MyFirstModule.MF_Added;"}, } + // What a step's build must describe again, and what it must carry over: + // an entity's row depends on its own subtree only, so a change to one + // entity leaves its siblings alone (#1803), while a change to the rest of + // the domain model describes them all. + expect := map[string]struct{ described, kept []string }{ + "alter an entity": {[]string{"FeedbackModule.Feedback"}, []string{"FeedbackModule.ResponseHelper"}}, + "alter one of two entities": {[]string{"MyFirstModule.ProbeA"}, []string{"MyFirstModule.ProbeB"}}, + "add an association": {[]string{"MyFirstModule.ProbeA", "MyFirstModule.ProbeB"}, nil}, + "rename an entity": {[]string{"MyFirstModule.ProbeC"}, nil}, + } + var mu sync.Mutex + describedNow := map[string]bool{} + sourceDescribeHook = func(_, qn string) { + mu.Lock() + describedNow[qn] = true + mu.Unlock() + } + t.Cleanup(func() { sourceDescribeHook = nil }) + reusedAny := false for _, s := range steps { if s.mdl != "" { @@ -171,7 +199,18 @@ end; t.Fatalf("%s: %v\n%s", s.name, err, out.String()) } } + clear(describedNow) got, buildOut := sourceBuild(t, exec, false) + for _, qn := range expect[s.name].described { + if !describedNow[qn] { + t.Errorf("%s: %s was carried over, want it described again", s.name, qn) + } + } + for _, qn := range expect[s.name].kept { + if describedNow[qn] { + t.Errorf("%s: %s was described again, want its row carried over", s.name, qn) + } + } n := reusedCount(buildOut) if n > 0 { reusedAny = true diff --git a/mdl/executor/catalog_unit_store_pedapp_test.go b/mdl/executor/catalog_unit_store_pedapp_test.go new file mode 100644 index 0000000000..7eb3f339c6 --- /dev/null +++ b/mdl/executor/catalog_unit_store_pedapp_test.go @@ -0,0 +1,159 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "context" + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "testing" + "time" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// ako/mxcli#1804: a catalog refresh opened every unit file of the project, +// after every save, and on Windows each open is scanned. The reader keeps a +// copy of the units in one store file and serves an unchanged unit from it, +// so the next process's refresh opens only what changed — and builds the +// same catalog. + +// forcedFullBuild builds the catalog from scratch and returns every row of +// every table, by table. +func forcedFullBuild(t *testing.T, exec *Executor) map[string][]string { + t.Helper() + ctx := exec.newExecContext(context.Background()) + var out bytes.Buffer + ctx.Output = &out + ctx.catalogForce = true + if err := buildCatalog(ctx, true, false, false, 0); err != nil { + t.Fatalf("build: %v\n%s", err, out.String()) + } + defer func() { ctx.Catalog.Close(); ctx.Catalog = nil }() + tables, err := ctx.Catalog.Query(`SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%'`) + if err != nil { + t.Fatal(err) + } + rows := map[string][]string{} + for _, tr := range tables.Rows { + name := fmt.Sprint(tr[0]) + // What records when the build ran, not what it found. + if name == "catalog_meta" || name == "snapshots" || name == "projects" || strings.Contains(name, "_fts") { + continue + } + res, err := ctx.Catalog.Query("SELECT * FROM " + name) + if err != nil { + continue // a virtual table's shadow tables + } + for _, r := range res.Rows { + rows[name] = append(rows[name], fmt.Sprint(r...)) + } + sort.Strings(rows[name]) + } + return rows +} + +func TestCatalogRefresh_NextProcessOpensOnlyChangedUnits(t *testing.T) { + first, out, dir := openPedAppCopy(t) + before := unitFileReads(t, first) + want := forcedFullBuild(t, first) + cold := unitFileReads(t, first) - before + if cold < 100 { + t.Fatalf("control: the first refresh read %d unit files; the fixture does not exercise the store", cold) + } + if err := first.Execute(&ast.DisconnectStmt{}); err != nil { + t.Fatalf("disconnect: %v\n%s", err, out.String()) + } + + second, _ := openPedAppAgain(t, dir) + before = unitFileReads(t, second) + got := forcedFullBuild(t, second) + if reads := unitFileReads(t, second) - before; reads != 0 { + t.Errorf("a refresh in the next process read %d unit files, want 0 (the first read %d)", reads, cold) + } + diffCatalogs(t, "unchanged project", got, want) + + // A save through mxcli: the writing process reads the unit back and + // keeps the new copy, so the next refresh opens nothing. + if err := agreeExec(t, second, "alter entity FeedbackModule.Feedback add attribute StoreProbe: String;"); err != nil { + t.Fatalf("edit: %v", err) + } + if err := second.Execute(&ast.DisconnectStmt{}); err != nil { + t.Fatal(err) + } + third, _ := openPedAppAgain(t, dir) + before = unitFileReads(t, third) + afterSave := forcedFullBuild(t, third) + if reads := unitFileReads(t, third) - before; reads != 0 { + t.Errorf("a refresh after a save through mxcli read %d unit files, want 0", reads) + } + if err := third.Execute(&ast.DisconnectStmt{}); err != nil { + t.Fatal(err) + } + + // A file changed outside mxcli (Studio Pro, a checkout): the next + // refresh opens that file, and only it. + touchOneUnit(t, dir) + fourth, _ := openPedAppAgain(t, dir) + before = unitFileReads(t, fourth) + forcedFullBuild(t, fourth) + if reads := unitFileReads(t, fourth) - before; reads != 1 { + t.Errorf("a refresh after one file changed read %d unit files, want 1", reads) + } + if err := fourth.Execute(&ast.DisconnectStmt{}); err != nil { + t.Fatal(err) + } + + removeUnitStore(t, dir) + fresh, _ := openPedAppAgain(t, dir) + diffCatalogs(t, "after a save", afterSave, forcedFullBuild(t, fresh)) +} + +// touchOneUnit moves one unit file's modification time on, as a save that +// rewrote it would. +func touchOneUnit(t *testing.T, dir string) { + t.Helper() + var path string + _ = filepath.WalkDir(filepath.Join(dir, "mprcontents"), func(p string, d os.DirEntry, err error) error { + if err == nil && path == "" && strings.HasSuffix(p, ".mxunit") { + path = p + } + return nil + }) + fi, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + later := fi.ModTime().Add(2 * time.Second) + if err := os.Chtimes(path, later, later); err != nil { + t.Fatal(err) + } +} + +func diffCatalogs(t *testing.T, step string, got, want map[string][]string) { + t.Helper() + for name, w := range want { + g := got[name] + if strings.Join(g, "\n") != strings.Join(w, "\n") { + t.Errorf("%s: table %s differs from a build that read every file (%d rows, want %d)", step, name, len(g), len(w)) + } + } + for name := range got { + if _, ok := want[name]; !ok { + t.Errorf("%s: table %s is not in a build that read every file", step, name) + } + } +} + +// removeUnitStore deletes the persisted unit copies, so the next process +// reads every file: the reference build. +func removeUnitStore(t *testing.T, dir string) { + t.Helper() + if err := os.RemoveAll(filepath.Join(dir, ".mxcli", "cache")); err != nil { + t.Fatal(err) + } +} diff --git a/mdl/executor/cmd_alter_flow.go b/mdl/executor/cmd_alter_flow.go index 335587e73d..26cc285c73 100644 --- a/mdl/executor/cmd_alter_flow.go +++ b/mdl/executor/cmd_alter_flow.go @@ -36,6 +36,11 @@ func execAlterFlow(ctx *ExecContext, s *ast.AlterFlowStmt) error { if err := validateAlterFlowRules(s); err != nil { return err } + // An action the project's version does not have, in what the statement + // splices in: statementVersionGates (ako/mxcli#1787). + if err := checkVersionGates(ctx, s); err != nil { + return err + } a, err := loadAlterFlow(ctx, s) if err != nil { return err diff --git a/mdl/executor/cmd_alter_workflow.go b/mdl/executor/cmd_alter_workflow.go index 78a62cfa97..b47dd8e88c 100644 --- a/mdl/executor/cmd_alter_workflow.go +++ b/mdl/executor/cmd_alter_workflow.go @@ -23,8 +23,8 @@ func execAlterWorkflow(ctx *ExecContext, s *ast.AlterWorkflowStmt) error { return mdlerrors.NewNotConnectedWrite() } - // Version pre-check: workflows require Mendix 9.12+, and the added - // activities need their own features (statementVersionGates). + // Version pre-check: the workflow document's own floor, and the added + // activities' features (statementVersionGates). if err := checkVersionGates(ctx, s); err != nil { return err } diff --git a/mdl/executor/cmd_catalog.go b/mdl/executor/cmd_catalog.go index 3c1e1d9cf2..a3c2e9bd49 100644 --- a/mdl/executor/cmd_catalog.go +++ b/mdl/executor/cmd_catalog.go @@ -527,6 +527,9 @@ func buildCatalog(ctx *ExecContext, full, isSource, communities bool, resolution builder.SetSourceMode(true) preWarmCache(ctx) builder.SetDescribeFunc(func(objectType string, qualifiedName string, id string) (string, error) { + if hook := sourceDescribeHook; hook != nil { + hook(objectType, qualifiedName) + } return captureDescribeParallel(ctx, objectType, qualifiedName, model.ID(id)) }) // Rows of documents nothing has changed under are carried over from @@ -769,6 +772,12 @@ func execRefreshCatalogStmt(ctx *ExecContext, stmt *ast.RefreshCatalogStmt) erro return buildCatalog(ctx, full, source, stmt.Communities, stmt.Resolution) } +// sourceDescribeHook, when set, sees every document a source build describes +// rather than carries over. Tests use it to name what an incremental build +// described again; it runs on the describe workers, so it must be safe for +// concurrent use. +var sourceDescribeHook func(objectType, qualifiedName string) + // sourceFingerprint identifies the running mxcli build, so source rows are // only reused by the build that described them: another version may describe // the same document differently. The executable's path, size and modification diff --git a/mdl/executor/cmd_microflows_builder_actions.go b/mdl/executor/cmd_microflows_builder_actions.go index 403ebb3270..cf49ee0fd1 100644 --- a/mdl/executor/cmd_microflows_builder_actions.go +++ b/mdl/executor/cmd_microflows_builder_actions.go @@ -1257,9 +1257,20 @@ func (fb *flowBuilder) addRetrieveAction(s *ast.RetrieveStmt) model.ID { // — where HttpRequest extends HttpMessage and HttpHeaders has child // HttpMessage — are still classified as reverse traversal (which returns // a list, so the output variable is typed accordingly below). + // + // Which end it starts from is the rule a spliced fragment and check + // --references take (associationRetrieveEndFrom). A self-reference is + // followed from its FROM end, never the child side: it used to be read + // as a reverse traversal, typed as a list unless an object use followed, + // and a list use of it was written and failed the build with CE0013 / + // CE0097 / CE0100 (ako/mxcli#1793, measured on 11.13.0). A specialization + // of both ends of an association between two entities stays read from + // the child side, as before: which end Mendix takes there is not + // measured, and the fragment and the check leave it untyped. + startEnd := fb.associationRetrieveEndFrom(startVarType, assocInfo) startsFromChildSide := assocInfo != nil && assocInfo.childEntityQN != "" && - fb.entityIsSubtypeOf(startVarType, assocInfo.childEntityQN) + (startEnd == assocEndTo || startEnd == assocEndAmbiguous) // The `$var/Module.Assoc` syntax is a retrieve-by-association (in-memory). // Keep it as an AssociationRetrieveSource — that is what preserves the @@ -1272,6 +1283,8 @@ func (fb *flowBuilder) addRetrieveAction(s *ast.RetrieveStmt) model.ID { // SINGLE object in Mendix (mxbuild reports CE0100 when it feeds a loop or // aggregate), so there is no in-memory way to obtain the list — fall back // to a DatabaseRetrieveSource. Object usage stays an association retrieve. + // A self-reference is never a reverse traversal (startsFromChildSide), so + // it never takes this branch. expandReverseReference := assocInfo != nil && assocInfo.Type == domainmodel.AssociationTypeReference && assocInfo.Owner == domainmodel.AssociationOwnerBoth && @@ -2904,19 +2917,31 @@ func (fb *flowBuilder) lookupAssociation(moduleName, assocName string) *assocLoo return nil } +// associationRetrieveEndFrom is the end of info a retrieve over it starts from +// when its start variable holds startType, by associationRetrieveEnd — the +// rule the full build, a spliced fragment and check --references share. A +// self-reference is followed from its FROM end (ako/mxcli#1793). +func (fb *flowBuilder) associationRetrieveEndFrom(startType string, info *assocLookupResult) assocRetrieveEnd { + if info == nil { + return assocEndNone + } + return associationRetrieveEnd( + fb.entityIsSubtypeOf(startType, info.parentEntityQN), + fb.entityIsSubtypeOf(startType, info.childEntityQN), + info.parentEntityQN != "" && info.parentEntityQN == info.childEntityQN) +} + // associationRetrieveType is what `retrieve $X from $Start/assocQN` binds when // $Start holds startType ("Module.Entity"): the list or the object at the -// other end, by associationRetrieveYieldsList — the answer addRetrieveAction -// and check --references take. Mind the inversion: an association's parent is -// its FROM end and its child the TO end, so a retrieve from the TO end yields -// the FROM entity. A self-reference is followed from its FROM end: the stored -// retrieve records no direction, and Mendix reads it that way — the object a -// reference holds, the list a reference set holds (measured on 11.13.0, -// ako/mxcli#1777: a change on it builds, a count of it is CE0013 "Input -// variable 'P' must be of type 'List'."). "" when it cannot be established: the -// association or the start unknown, the start a list, the start no end of it, -// or a specialization of both ends of an association between two entities, -// where which end it starts from is not known. +// other end, by associationRetrieveEndFrom and associationRetrieveYieldsList — +// the answer addRetrieveAction and check --references take. Mind the +// inversion: an association's parent is its FROM end and its child the TO end, +// so a retrieve from the TO end yields the FROM entity. A self-reference is +// followed from its FROM end: the object a reference holds, the list a +// reference set holds (measured on 11.13.0, ako/mxcli#1777). "" when it cannot +// be established: the association or the start unknown, the start a list, the +// start no end of it, or a specialization of both ends of an association +// between two entities, where which end it starts from is not known. func (fb *flowBuilder) associationRetrieveType(startType, assocQN string) string { if startType == "" || strings.HasPrefix(startType, "List of ") { return "" @@ -2930,15 +2955,13 @@ func (fb *flowBuilder) associationRetrieveType(startType, assocQN string) string return "" } refSet := info.Type == domainmodel.AssociationTypeReferenceSet - fromTo := fb.entityIsSubtypeOf(startType, info.childEntityQN) - fromFrom := fb.entityIsSubtypeOf(startType, info.parentEntityQN) - switch { - case fromTo && fromFrom: - if info.parentEntityQN != info.childEntityQN { - return "" - } - fromTo = false // a self-reference: from the FROM end - case !fromTo && !fromFrom: + var fromTo bool + switch fb.associationRetrieveEndFrom(startType, info) { + case assocEndFrom: + fromTo = false + case assocEndTo: + fromTo = true + default: return "" } other := info.childEntityQN diff --git a/mdl/executor/cmd_microflows_builder_retrieve_self_reference_test.go b/mdl/executor/cmd_microflows_builder_retrieve_self_reference_test.go new file mode 100644 index 0000000000..d0eccf66b4 --- /dev/null +++ b/mdl/executor/cmd_microflows_builder_retrieve_self_reference_test.go @@ -0,0 +1,221 @@ +// SPDX-License-Identifier: Apache-2.0 + +// ako/mxcli#1793: the full microflow build typed `retrieve $P from +// $Order/M.Order_Parent` — a Reference from M.Order to M.Order — as a LIST of +// M.Order unless $P was later used as an object, because the start variable is +// an instance of the association's TO entity (as well as its FROM entity), so +// the retrieve was classified as a reverse traversal. Mendix reads it the +// other way: the stored AssociationRetrieveSource records no direction and a +// self-reference is followed from its FROM end. Measured on 11.13.0 with +// M.Order_Parent (Reference, owner Default) and M.Order_Related (Reference +// set), each shape in its own microflow: +// +// retrieve $P from $Order/M.Order_Parent; change $P (…) 0 errors +// … $n = count $P; CE0013 "Input variable 'P' must be of type 'List'." +// (sum, average, minimum, maximum, all, any, reduce: CE0013 too) +// … $h = head $P; CE0097 "The selected 'P' variable must be of type List." +// (tail, find, filter, sort, union, intersect, subtract, contains, equals: CE0097 too) +// … $r = range $P limit 1; 0 errors (Microflows$ListRange: mx check does not type it) +// … add / remove / clear $P 0 errors (Change list: mx check does not type it) +// … loop $x in $P CE0100 "'P' is of type M.Order, but should be of type List." +// … call M.SubList(Orders = $P) CE0117 "Error(s) in expression." +// the same from a $S: M.SpecialOrder (a specialization) the same +// retrieve $R from $Order/M.Order_Related; count / loop 0 errors +// … declare $c String = $R/Code; CE0117 (the reference set is a list) +// retrieve $C from $Order/M.Order_Customer; count $C CE0013 (a forward Reference) +// +// check --references was silent on every one of them, so a list use was +// accepted, written by exec, and failed the build. +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/domainmodel" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// selfReferenceBackend is a project with Sample.Node, which has a Reference +// (Node_Parent), a Reference set (Node_Related) and a one-to-one (Node_Twin, +// owner Both — CE0002 in Studio Pro, but a stored model is what the builder +// reads) from itself to itself, Sample.SpecialNode a specialization of it, +// Sample.Line with a Reference Line_Node to Node, and Sample.Animal with its +// specialization Sample.Dog, with a Reference Dog_Animal from Dog to Animal and +// Animal_Dog the other way: a Dog is a specialization of both ends of either. +func selfReferenceBackend() *mock.MockBackend { + mod := &model.Module{BaseElement: model.BaseElement{ID: "mod-sample"}, Name: "Sample"} + ent := func(id, name, general string) *domainmodel.Entity { + return &domainmodel.Entity{BaseElement: model.BaseElement{ID: model.ID(id)}, ContainerID: mod.ID, + Name: name, Persistable: true, GeneralizationRef: general} + } + assoc := func(name, parent, child string, typ domainmodel.AssociationType, owner domainmodel.AssociationOwner) *domainmodel.Association { + return &domainmodel.Association{BaseElement: model.BaseElement{ID: model.ID("a-" + name)}, ContainerID: mod.ID, + Name: name, ParentID: model.ID(parent), ChildID: model.ID(child), Type: typ, Owner: owner} + } + dm := &domainmodel.DomainModel{ + BaseElement: model.BaseElement{ID: "dm-sample"}, ContainerID: mod.ID, + Entities: []*domainmodel.Entity{ + ent("e-node", "Node", ""), + ent("e-special", "SpecialNode", "Sample.Node"), + ent("e-line", "Line", ""), + ent("e-animal", "Animal", ""), + ent("e-dog", "Dog", "Sample.Animal"), + }, + Associations: []*domainmodel.Association{ + assoc("Node_Parent", "e-node", "e-node", domainmodel.AssociationTypeReference, domainmodel.AssociationOwnerDefault), + assoc("Node_Related", "e-node", "e-node", domainmodel.AssociationTypeReferenceSet, domainmodel.AssociationOwnerDefault), + assoc("Node_Twin", "e-node", "e-node", domainmodel.AssociationTypeReference, domainmodel.AssociationOwnerBoth), + assoc("Line_Node", "e-line", "e-node", domainmodel.AssociationTypeReference, domainmodel.AssociationOwnerDefault), + assoc("Dog_Animal", "e-dog", "e-animal", domainmodel.AssociationTypeReference, domainmodel.AssociationOwnerDefault), + assoc("Animal_Dog", "e-animal", "e-dog", domainmodel.AssociationTypeReference, domainmodel.AssociationOwnerDefault), + }, + } + return &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + GetModuleByNameFunc: func(name string) (*model.Module, error) { + if name == mod.Name { + return mod, nil + } + return nil, nil + }, + ListDomainModelsFunc: func() ([]*domainmodel.DomainModel, error) { return []*domainmodel.DomainModel{dm}, nil }, + GetDomainModelFunc: func(id model.ID) (*domainmodel.DomainModel, error) { + if id == mod.ID { + return dm, nil + } + return nil, nil + }, + } +} + +func selfReferenceFlowBuilder() *flowBuilder { + return &flowBuilder{ + varTypes: map[string]string{ + "Node": "Sample.Node", + "Special": "Sample.SpecialNode", + "Line": "Sample.Line", + "Dog": "Sample.Dog", + }, + backend: selfReferenceBackend(), + posX: 200, + posY: 200, + spacing: HorizontalSpacing, + } +} + +// TestBuildFlowGraph_SelfReferenceIsFollowedFromItsFromEnd is the reported +// shape: a list use after a retrieve over a self-referencing Reference. The +// output is the one object the FROM end holds, whatever the use, and it stays +// an association retrieve — a self-reference never takes the owner-Both +// reverse fallback to a database retrieve. +func TestBuildFlowGraph_SelfReferenceIsFollowedFromItsFromEnd(t *testing.T) { + count := func(list string) ast.MicroflowStatement { + return &ast.AggregateListStmt{OutputVariable: "N", Operation: ast.AggregateCount, InputVariable: list} + } + change := func(obj string) ast.MicroflowStatement { + return &ast.ChangeObjectStmt{Variable: obj, Changes: []ast.ChangeItem{{Attribute: "Name", Value: &ast.LiteralExpr{Value: "x", Kind: ast.LiteralString}}}} + } + loop := func(list string) ast.MicroflowStatement { + return &ast.LoopStmt{LoopVariable: "It", ListVariable: list, Body: []ast.MicroflowStatement{}} + } + for _, tc := range []struct { + name, start, assoc string + use ast.MicroflowStatement + want string + }{ + {"Reference, a list use (count)", "Node", "Node_Parent", count("P"), "Sample.Node"}, + {"Reference, a list use (loop)", "Node", "Node_Parent", loop("P"), "Sample.Node"}, + {"Reference, unused", "Node", "Node_Parent", nil, "Sample.Node"}, + {"Reference, an object use", "Node", "Node_Parent", change("P"), "Sample.Node"}, + {"Reference from a specialization, a list use", "Special", "Node_Parent", count("P"), "Sample.Node"}, + {"owner Both, a list use: no database fallback", "Node", "Node_Twin", count("P"), "Sample.Node"}, + {"Reference set: the list", "Node", "Node_Related", count("P"), "List of Sample.Node"}, + {"Reference set from a specialization: the list", "Special", "Node_Related", nil, "List of Sample.Node"}, + } { + t.Run(tc.name, func(t *testing.T) { + fb := selfReferenceFlowBuilder() + stmts := []ast.MicroflowStatement{&ast.RetrieveStmt{ + Variable: "P", + StartVariable: tc.start, + Source: ast.QualifiedName{Module: "Sample", Name: tc.assoc}, + }} + if tc.use != nil { + stmts = append(stmts, tc.use) + } + fb.buildFlowGraph(stmts, nil) + + action := firstRetrieveAction(t, fb) + source, ok := action.Source.(*microflows.AssociationRetrieveSource) + if !ok { + t.Fatalf("source = %T, want an AssociationRetrieveSource", action.Source) + } + if source.StartVariable != tc.start || source.AssociationQualifiedName != "Sample."+tc.assoc { + t.Fatalf("association source = %#v", source) + } + if got := fb.varTypes["P"]; got != tc.want { + t.Errorf("$P typed %q, want %q", got, tc.want) + } + }) + } +} + +// TestBuildFlowGraph_AssociationDirectionControls: the shapes the fix must not +// move. A Reference from its TO end is the list of FROM objects, and from its +// FROM end the one TO object; a start that is a specialization of both ends of +// an association between two different entities stays read from the TO end, +// as before #1793 (which end Mendix starts from there is not measured). +func TestBuildFlowGraph_AssociationDirectionControls(t *testing.T) { + for _, tc := range []struct { + name, start, assoc, want string + }{ + {"Reference from its TO end", "Node", "Line_Node", "List of Sample.Line"}, + {"Reference from its FROM end", "Line", "Line_Node", "Sample.Node"}, + {"a specialization of both ends of two entities", "Dog", "Dog_Animal", "List of Sample.Dog"}, + } { + t.Run(tc.name, func(t *testing.T) { + fb := selfReferenceFlowBuilder() + fb.buildFlowGraph([]ast.MicroflowStatement{ + &ast.RetrieveStmt{Variable: "P", StartVariable: tc.start, Source: ast.QualifiedName{Module: "Sample", Name: tc.assoc}}, + &ast.AggregateListStmt{OutputVariable: "N", Operation: ast.AggregateCount, InputVariable: "P"}, + }, nil) + if got := fb.varTypes["P"]; got != tc.want { + t.Errorf("$P typed %q, want %q", got, tc.want) + } + }) + } +} + +// TestAssociationRetrieveType_AgreesWithTheFullBuild: the fragment a splice +// builds (associationRetrieveType, ako/mxcli#1777) and the full build +// (addRetrieveAction) take one rule for which end a retrieve starts from, so +// they type every unambiguous shape alike. The full build's type is read with +// no use following the retrieve, where it is the association's own answer. +func TestAssociationRetrieveType_AgreesWithTheFullBuild(t *testing.T) { + for _, tc := range []struct{ start, assoc string }{ + {"Node", "Node_Parent"}, + {"Special", "Node_Parent"}, + {"Node", "Node_Related"}, + {"Special", "Node_Related"}, + {"Node", "Node_Twin"}, + {"Node", "Line_Node"}, + {"Line", "Line_Node"}, + } { + fb := selfReferenceFlowBuilder() + startType := fb.varTypes[tc.start] + spliced := fb.associationRetrieveType(startType, "Sample."+tc.assoc) + fb.addRetrieveAction(&ast.RetrieveStmt{Variable: "P", StartVariable: tc.start, + Source: ast.QualifiedName{Module: "Sample", Name: tc.assoc}}) + if built := fb.varTypes["P"]; spliced == "" || built != spliced { + t.Errorf("$%s/Sample.%s: the full build types %q, a splice %q", tc.start, tc.assoc, built, spliced) + } + } + // The ambiguous shape is the one the splice leaves untyped. + fb := selfReferenceFlowBuilder() + if got := fb.associationRetrieveType("Sample.Dog", "Sample.Dog_Animal"); got != "" { + t.Errorf("a specialization of both ends of two entities: a splice typed %q, want it untyped", got) + } +} diff --git a/mdl/executor/cmd_microflows_builder_validate.go b/mdl/executor/cmd_microflows_builder_validate.go index edff58e343..3da408d806 100644 --- a/mdl/executor/cmd_microflows_builder_validate.go +++ b/mdl/executor/cmd_microflows_builder_validate.go @@ -466,16 +466,22 @@ func ValidateRuleBody(s *ast.CreateRuleStmt) []string { } // forwardReferenceTarget reports the entity a retrieve over association yields -// when that is one object: a Reference followed from its FROM entity, the same -// reading the builder makes. A self-association, a reverse traversal, a -// ReferenceSet or an association check cannot resolve stays a list, so the -// rules keyed on it never see an object that is not one. +// when that is one object: a Reference followed from its FROM entity, by the +// rule the builder takes (associationRetrieveEnd). A self-reference is +// followed from its FROM end, so it is that object too: it used to stay a +// list here, and `set` on it passed as a Change list Replace over what Mendix +// types as an object (ako/mxcli#1793). A reverse traversal, a ReferenceSet or +// an association check cannot resolve stays a list, so the rules keyed on it +// never see an object that is not one. func (fb *flowBuilder) forwardReferenceTarget(s *ast.RetrieveStmt) (string, bool) { shape, ok := fb.checkAssocShapes[strings.ToLower(s.Source.Module+"."+s.Source.Name)] - if !ok || shape.refSet || strings.EqualFold(shape.from, shape.to) { + if !ok || shape.refSet { return "", false } - if !strings.EqualFold(fb.varTypes[s.StartVariable], shape.from) { + start := fb.varTypes[s.StartVariable] + end := associationRetrieveEnd(strings.EqualFold(start, shape.from), strings.EqualFold(start, shape.to), + strings.EqualFold(shape.from, shape.to)) + if end != assocEndFrom { return "", false } return shape.to, true diff --git a/mdl/executor/cmd_rest_openapi_mock_test.go b/mdl/executor/cmd_rest_openapi_mock_test.go index 9597f107cf..c47bc9edad 100644 --- a/mdl/executor/cmd_rest_openapi_mock_test.go +++ b/mdl/executor/cmd_rest_openapi_mock_test.go @@ -43,15 +43,17 @@ const minimalOpenAPISpec = `{ // newOpenAPIMockBackend returns a MockBackend pre-wired for OpenAPI import // tests: write-connected, a project version that satisfies the REST client -// feature gate, and no pre-existing REST clients. +// feature gates, and no pre-existing REST clients. 10.21 is the floor of the +// stored OpenAPI document (Rest$OpenApiFile): the write refuses one on 10.6, +// measured on a stamped project, and so does the gate (ako/mxcli#1811). func newOpenAPIMockBackend() *mock.MockBackend { return &mock.MockBackend{ IsConnectedFunc: func() bool { return true }, ProjectVersionFunc: func() *types.ProjectVersion { return &types.ProjectVersion{ - ProductVersion: "10.6.0", + ProductVersion: "10.21.0", MajorVersion: 10, - MinorVersion: 6, + MinorVersion: 21, PatchVersion: 0, } }, diff --git a/mdl/executor/noop_reporting_pedapp_test.go b/mdl/executor/noop_reporting_pedapp_test.go index 0e387fbefe..851fb3eda5 100644 --- a/mdl/executor/noop_reporting_pedapp_test.go +++ b/mdl/executor/noop_reporting_pedapp_test.go @@ -94,11 +94,17 @@ func openPedAppCopy(t *testing.T) (*Executor, *bytes.Buffer, string) { return exec, out, dir } -// projectFiles hashes every file of the project by content. +// projectFiles hashes every file of the project by content. The unit store +// under .mxcli/cache/ is left out: it is a cache of what was read, which a +// read-only command refreshes, not part of the project (ako/mxcli#1804). func projectFiles(t *testing.T, dir string) map[string]string { t.Helper() files := map[string]string{} + cache := filepath.Join(dir, ".mxcli", "cache") err := filepath.Walk(dir, func(p string, info os.FileInfo, err error) error { + if err == nil && info.IsDir() && p == cache { + return filepath.SkipDir + } if err != nil || info.IsDir() { return err } diff --git a/mdl/executor/roundtrip_mxcheck_test.go b/mdl/executor/roundtrip_mxcheck_test.go index 9e9c5e8411..ec125ae5d8 100644 --- a/mdl/executor/roundtrip_mxcheck_test.go +++ b/mdl/executor/roundtrip_mxcheck_test.go @@ -640,7 +640,7 @@ func TestMxCheck_ViewEntitySimple(t *testing.T) { env := setupTestEnv(t) defer env.teardown() - env.requireMinVersion(t, 10, 18) // VIEW ENTITY requires 10.18+ + env.requireMinVersion(t, 10, 21) // VIEW ENTITY requires 10.21+ (its source document, ako/mxcli#1811) mod := testModule @@ -695,7 +695,7 @@ func TestMxCheck_ViewEntityWithAggregates(t *testing.T) { env := setupTestEnv(t) defer env.teardown() - env.requireMinVersion(t, 10, 18) // VIEW ENTITY requires 10.18+ + env.requireMinVersion(t, 10, 21) // VIEW ENTITY requires 10.21+ (its source document, ako/mxcli#1811) mod := testModule diff --git a/mdl/executor/validate.go b/mdl/executor/validate.go index 855963945b..66d8fb7e0f 100644 --- a/mdl/executor/validate.go +++ b/mdl/executor/validate.go @@ -529,7 +529,9 @@ func validateProgramWithWarnings(ctx *ExecContext, prog *ast.Program) ([]error, // MDL-RETRIEVE03 over an association: whether `retrieve $L from $h/M.A` // binds a list depends on the association and on which end $h is, so the // no-project rule cannot judge it. Using that list as an object built as - // CE0117 / CE7247 (mendixlabs/mxcli#1516). + // CE0117 / CE7247 (mendixlabs/mxcli#1516); using the one object a + // Reference gives from its FROM end — a self-reference always — as a list + // is MDL-RETRIEVE01, CE0013 / CE0097 / CE0100 (ako/mxcli#1793). errors = append(errors, validateAssociationRetrieveListUse(ctx, prog, sc)...) // Resolve the MEMBERS inside a widget's XPath constraint. The entity in // `database from Mod.Entity` was resolved and the `where […]` was not, so a diff --git a/mdl/executor/validate_flow_member_refs.go b/mdl/executor/validate_flow_member_refs.go index 9150917df8..0eb8d79ccb 100644 --- a/mdl/executor/validate_flow_member_refs.go +++ b/mdl/executor/validate_flow_member_refs.go @@ -70,6 +70,14 @@ func (r *flowMemberRefs) catalog() associationCatalog { // chain is entityQN's generalization chain through the script's declarations // and then the project. func (r *flowMemberRefs) chain(entityQN string) ([]string, bool) { + return scriptGeneralizationChain(r.ctx, r.sc, entityQN) +} + +// scriptGeneralizationChain is entityQN's generalization chain, itself first, +// through the entities the script declares and then the project's: a check +// runs before the script's entities exist, so the project alone does not +// know a specialization the same script creates. +func scriptGeneralizationChain(ctx *ExecContext, sc *scriptContext, entityQN string) ([]string, bool) { var chain []string seen := map[string]bool{} for qn := entityQN; qn != ""; { @@ -77,12 +85,12 @@ func (r *flowMemberRefs) chain(entityQN string) ([]string, bool) { return chain, false } seen[qn] = true - if r.sc != nil && r.sc.entities[qn] { + if sc != nil && sc.entities[qn] { chain = append(chain, qn) - qn = r.sc.entityGeneralizations[strings.ToLower(qn)] + qn = sc.entityGeneralizations[strings.ToLower(qn)] continue } - rest, complete := generalizationChain(r.ctx, qn) + rest, complete := generalizationChain(ctx, qn) return append(chain, rest...), complete } return chain, true diff --git a/mdl/executor/validate_retrieve_association_list.go b/mdl/executor/validate_retrieve_association_list.go index 47806d2391..43f7a9127d 100644 --- a/mdl/executor/validate_retrieve_association_list.go +++ b/mdl/executor/validate_retrieve_association_list.go @@ -32,13 +32,57 @@ func associationRetrieveYieldsList(refSet, ownerBoth, fromToSide bool) bool { return fromToSide && !ownerBoth } +// assocRetrieveEnd is the end of an association a retrieve over it starts from. +type assocRetrieveEnd int + +const ( + assocEndNone assocRetrieveEnd = iota // the start is no end of it, as far as is known + assocEndFrom // the FROM end: the association's parent, the foreign-key owner + assocEndTo // the TO end: its child + assocEndAmbiguous // a specialization of both ends of an association between two entities +) + +// associationRetrieveEnd decides which end `retrieve $X from $Start/Assoc` +// starts from, given whether $Start holds (a specialization of) the +// association's FROM entity and its TO entity. It is the one answer the full +// build (addRetrieveAction), a spliced fragment (associationRetrieveType) and +// check --references (associationRetrieveShape) take, each asking "is it a" of +// its own view of the project. +// +// A self-reference has the start at both ends, and the stored +// AssociationRetrieveSource records no direction (only the start variable and +// the association): Mendix follows it from its FROM end, so a Reference gives +// the object it holds and a Reference set the list. Measured on 11.13.0 +// (ako/mxcli#1777, #1793): over M.Order_Parent from M.Order to M.Order a +// `change` of the output builds, while `count` is CE0013 "Input variable 'P' +// must be of type 'List'.", a loop CE0100 "'P' is of type M.Order, but should +// be of type List.", from an M.Order and from a specialization of it alike. +// Owner Both on a self-reference is CE0002, so the one-to-one case does not +// arise. A start that is a specialization of both ends of an association +// between two different entities is ambiguous: which end Mendix takes there is +// not measured. +func associationRetrieveEnd(isFrom, isTo, selfReference bool) assocRetrieveEnd { + switch { + case isFrom && isTo: + if selfReference { + return assocEndFrom + } + return assocEndAmbiguous + case isTo: + return assocEndTo + case isFrom: + return assocEndFrom + } + return assocEndNone +} + // validateAssociationRetrieveListUse is MDL-RETRIEVE03 for a retrieve over an -// association. The no-project rule judges only database retrieves, because -// whether `retrieve $L from $h/M.A` binds a list or an object depends on the -// association's type, owner and which end $h is — the project knows, the AST -// does not. Without it, using the list as an object passed check --references, -// exec wrote it, and the build failed (mendixlabs/mxcli#1516, measured on -// 11.6.6 and 11.15.0): +// association, and MDL-RETRIEVE01 the other way round. The no-project rules +// judge only database retrieves, because whether `retrieve $L from $h/M.A` +// binds a list or an object depends on the association's type, owner and +// which end $h is — the project knows, the AST does not. Without it, using the +// list as an object passed check --references, exec wrote it, and the build +// failed (mendixlabs/mxcli#1516, measured on 11.6.6 and 11.15.0): // // $L/Qty in any expression CE0117 "Error(s) in expression." // change $L (…) CE7247 "Variable 'L' is of type List of …, but should be of type Object." @@ -47,10 +91,28 @@ func associationRetrieveYieldsList(refSet, ownerBoth, fromToSide bool) bool { // The bare list retrieve and list uses of it (count, loop, a list parameter) // build clean on both versions, so they are not reported. // +// The reverse — the one object a Reference followed from its FROM end binds, +// used as a list — passed the same way (ako/mxcli#1793, measured on 11.13.0). +// A self-reference is always followed from its FROM end, and the full build +// used to type it as a list, so that was where it showed: +// +// $n = count $P; (any aggregate) CE0013 "Input variable 'P' must be of type 'List'." +// $h = head $P; (any list operation) CE0097 "The selected 'P' variable must be of type List." +// loop $x in $P CE0100 "'P' is of type M.Order, but should be of type List." +// call M.Sub(list = $P) CE0117 (the parameter takes a list) +// +// `add`, `remove`, `clear` and `set` over the object are Change list +// activities and a `range` is a Microflows$ListRange; mx check types none of +// their variables (each measured clean), so they are not reported here — `set` +// on an object is MDL-SET01's. +// // Anything not established is silence: an association neither the project nor -// the script defines, an owner mxcli does not write, a self-association, a start -// variable whose entity this walk did not track, or one that is no end of the -// association by its generalization chain as far as it could be read. +// the script defines, an owner mxcli does not write, a start variable whose +// entity this walk did not track, one that is no end of the association by its +// generalization chain as far as it could be read, or a specialization of both +// ends of an association between two entities (associationRetrieveEnd). The +// object a one-to-one gives from its TO end is not judged as a list either: the +// full build rewrites a list use of it into a database retrieve. func validateAssociationRetrieveListUse(ctx *ExecContext, prog *ast.Program, sc *scriptContext) []error { if prog == nil || !ctx.Connected() { return nil @@ -86,16 +148,24 @@ func validateAssociationRetrieveListUse(ctx *ExecContext, prog *ast.Program, sc if shapes == nil { shapes = checkAssociationShapes(ctx, sc) } - msgs := associationListObjectUses(ctx, shapes, signature, params, body) - if len(msgs) == 0 { - continue - } - if kind, name, excluded := excludedFlow(stmt); excluded { - sc.warnExcluded(kind, name, msgs) - continue + uses := associationListObjectUses(ctx, sc, shapes, signature, params, body) + for _, group := range []struct { + msgs []string + what string + }{ + {uses.listAsObject, "that yields a list is used as one object"}, + {uses.objectAsList, "that yields one object is used as a list"}, + } { + if len(group.msgs) == 0 { + continue + } + if kind, name, excluded := excludedFlow(stmt); excluded { + sc.warnExcluded(kind, name, group.msgs) + continue + } + errs = append(errs, mdlerrors.NewValidationf("statement %d: a retrieve over an association "+ + "%s:\n - %s", i+1, group.what, strings.Join(group.msgs, "\n - "))) } - errs = append(errs, mdlerrors.NewValidationf("statement %d: a retrieve over an association "+ - "that yields a list is used as one object:\n - %s", i+1, strings.Join(msgs, "\n - "))) } return errs } @@ -110,16 +180,25 @@ func hasAssociationRetrieve(body []ast.MicroflowStatement) bool { return found } -// assocListBinding is what one association retrieve bound to a list. +// assocListBinding is what one association retrieve bound: a list of elem, or +// one elem. type assocListBinding struct { assoc, start, startEntity, elem string + selfReference bool +} + +// assocRetrieveUses are the misuses of association retrieves in one flow body. +type assocRetrieveUses struct { + listAsObject []string // MDL-RETRIEVE03: a list taken as one object + objectAsList []string // MDL-RETRIEVE01: one object taken as a list } // associationListObjectUses walks one flow body in statement order, tracking the // entity each object variable holds (so a retrieve's start end can be decided) -// and which variables an association retrieve bound to a list. -func associationListObjectUses(ctx *ExecContext, shapes map[string]assocShape, - signature func(string) *flowSignature, params []ast.MicroflowParam, body []ast.MicroflowStatement) []string { +// and which variables an association retrieve bound to a list, or to the one +// object a Reference followed from its FROM end holds. +func associationListObjectUses(ctx *ExecContext, sc *scriptContext, shapes map[string]assocShape, + signature func(string) *flowSignature, params []ast.MicroflowParam, body []ast.MicroflowStatement) assocRetrieveUses { objects := map[string]string{} // object variable -> entity QN lists := map[string]string{} // list variable -> element entity QN @@ -132,9 +211,10 @@ func associationListObjectUses(ctx *ExecContext, shapes map[string]assocShape, } } } - bound := map[string]assocListBinding{} + bound := map[string]assocListBinding{} // bound to a list + single := map[string]assocListBinding{} // bound to the object at a Reference's TO end - var out []string + var out assocRetrieveUses walkListObjectUses(body, func(name string) bool { _, ok := bound[name]; return ok }, func(s ast.MicroflowStatement, name string, use listObjectUse) { b := bound[name] @@ -148,11 +228,18 @@ func associationListObjectUses(ctx *ExecContext, shapes map[string]assocShape, how, rejection = fmt.Sprintf("a member path `$%s/…` has nothing to navigate", name), "CE0117 \"Error(s) in expression.\"" } - out = append(out, fmt.Sprintf("%s, so %s — mxbuild rejects it with %s [MDL-RETRIEVE03]. %s", + out.listAsObject = append(out.listAsObject, fmt.Sprintf("%s, so %s — mxbuild rejects it with %s [MDL-RETRIEVE03]. %s", what, how, rejection, assocListSuggestion(name, b))) }, func(s ast.MicroflowStatement) { - // Object arguments: a list passed to a parameter that takes one object. + // One object taken by a list activity. + if name, op, rejection := listActivityInput(s); name != "" { + if b, ok := single[name]; ok { + out.objectAsList = append(out.objectAsList, assocObjectAsList(name, b, op, rejection)) + } + } + // Arguments: a list passed to a parameter that takes one object, + // and one object passed to a parameter that takes a list. var callee string var args []ast.CallArgument switch c := s.(type) { @@ -169,8 +256,9 @@ func associationListObjectUses(ctx *ExecContext, shapes map[string]assocShape, if !ok { continue } - b, ok := bound[v.Name] - if !ok { + listB, isList := bound[v.Name] + oneB, isOne := single[v.Name] + if !isList && !isOne { continue } sig := signature(callee) @@ -178,12 +266,21 @@ func associationListObjectUses(ctx *ExecContext, shapes map[string]assocShape, return } for _, p := range sig.Params { - if p.Name == a.Name && p.Object && !p.List { - out = append(out, fmt.Sprintf("$%s = retrieve over %s from $%s (a %s) is a list of %s, "+ + if p.Name != a.Name || !p.Object { + continue + } + switch { + case isList && !p.List: + b := listB + out.listAsObject = append(out.listAsObject, fmt.Sprintf("$%s = retrieve over %s from $%s (a %s) is a list of %s, "+ "so it cannot be passed to parameter %s of %s, which takes one %s — mxbuild rejects it "+ "with CE0117 \"Error(s) in expression.\" [MDL-RETRIEVE03]. %s", v.Name, b.assoc, b.start, b.startEntity, b.elem, p.Name, callee, p.Entity, assocListSuggestion(v.Name, b))) + case isOne && p.List: + out.objectAsList = append(out.objectAsList, assocObjectAsList(v.Name, oneB, + fmt.Sprintf("parameter %s of %s, which takes a list of %s,", p.Name, callee, p.Entity), + "CE0117 \"Error(s) in expression.\"")) } } } @@ -192,6 +289,7 @@ func associationListObjectUses(ctx *ExecContext, shapes map[string]assocShape, // Rebinding, after the statement's uses were judged. for _, p := range statementProducedVars(s) { delete(bound, p.name) + delete(single, p.name) delete(objects, p.name) delete(lists, p.name) } @@ -229,24 +327,76 @@ func associationListObjectUses(ctx *ExecContext, shapes map[string]assocShape, if !ok { return } - other, isList, known := associationRetrieveShape(ctx, shapes, st.Source, startEntity) + shape, known := associationRetrieveShape(ctx, sc, shapes, st.Source, startEntity) if !known { return } - if isList { - lists[st.Variable] = other - bound[st.Variable] = assocListBinding{ - assoc: shapes[strings.ToLower(qualifyAssoc(ctx, shapes, st.Source, startEntity))].qn, - start: start, startEntity: startEntity, elem: other, - } - } else { - objects[st.Variable] = other + b := assocListBinding{assoc: shape.qn, start: start, startEntity: startEntity, elem: shape.other, + selfReference: shape.selfReference} + switch { + case shape.isList: + lists[st.Variable] = shape.other + bound[st.Variable] = b + case !shape.fromTo: + objects[st.Variable] = shape.other + single[st.Variable] = b + default: + // One object of a one-to-one from its TO end: the full build + // turns a list use of it into a database retrieve. + objects[st.Variable] = shape.other } } }) return out } +// listActivityInput is the variable a list activity takes as its list — an +// aggregate's, a list operation's first operand, a loop's — with what takes it +// and the CE code mxbuild reports when it holds one object. "" for any other +// statement. Measured on 11.13.0 over a self-reference's object +// (ako/mxcli#1793): every aggregate (count, sum, average, minimum, maximum, +// all, any, reduce) CE0013; every list operation but range (head, tail, find, +// filter, sort, union, intersect, subtract, contains, equals) CE0097; a loop +// CE0100. A range is Microflows$ListRange and mx check reports nothing on it, +// so it is not judged; nor is a list operation's second operand, which union, +// intersect, subtract and contains take as an object. +func listActivityInput(s ast.MicroflowStatement) (name, op, rejection string) { + switch st := s.(type) { + case *ast.AggregateListStmt: + if st.InputVariable != "" { + return st.InputVariable, "`" + strings.ToLower(st.Operation.String()) + "`", + fmt.Sprintf("CE0013 \"Input variable '%s' must be of type 'List'.\"", st.InputVariable) + } + case *ast.ListOperationStmt: + if st.InputVariable != "" && st.Operation != ast.ListOpRange { + return st.InputVariable, "`" + strings.ToLower(st.Operation.String()) + "`", + fmt.Sprintf("CE0097 \"The selected '%s' variable must be of type List.\"", st.InputVariable) + } + case *ast.LoopStmt: + if st.ListVariable != "" { + return st.ListVariable, "a loop", "" + } + } + return "", "", "" +} + +// assocObjectAsList is the MDL-RETRIEVE01 message for one object an association +// retrieve bound, taken by what as a list. An empty rejection is a loop's, which +// names the entity. +func assocObjectAsList(name string, b assocListBinding, what, rejection string) string { + if rejection == "" { + rejection = fmt.Sprintf("CE0100 \"'%s' is of type %s, but should be of type List.\"", name, b.elem) + } + why := "a Reference followed from its FROM end" + if b.selfReference { + why += ", which is how Mendix follows a self-reference" + } + return fmt.Sprintf("$%s = retrieve over %s from $%s (a %s) is one %s — %s — so %s cannot take it: "+ + "mxbuild rejects it with %s [MDL-RETRIEVE01]. Use $%s as the one object it is (`$%s/…`, `change $%s (…)`); "+ + "a list of %s needs a Reference set, or a retrieve from the database: `retrieve $%s from %s where […];`.", + name, b.assoc, b.start, b.startEntity, b.elem, why, what, rejection, name, name, name, b.elem, name, b.elem) +} + // qualifyAssoc is the association a retrieve names. A bare name resolves the // way the flow builder resolves it (resolveBareAssociation, ako/mxcli#1690), // not by assuming the start entity's module; one that does not resolve is "", @@ -266,34 +416,60 @@ func qualifyAssoc(ctx *ExecContext, shapes map[string]assocShape, assoc ast.Qual return qn } +// assocRetrieveShape is what a retrieve over an association binds. +type assocRetrieveShape struct { + qn string // the association, qualified as the project spells it + other string // the entity at the end the retrieve does not start from + fromTo bool // it starts from the association's TO end + isList bool // it binds a list, not one object + selfReference bool +} + // associationRetrieveShape decides what a retrieve over assoc from an object of -// startEntity binds: the entity at the other end, and whether it is a list. -// known is false whenever that was not established. -func associationRetrieveShape(ctx *ExecContext, shapes map[string]assocShape, - assoc ast.QualifiedName, startEntity string) (other string, isList, known bool) { +// startEntity binds, by associationRetrieveEnd and associationRetrieveYieldsList +// — the rules the full build takes. known is false whenever that was not +// established. +func associationRetrieveShape(ctx *ExecContext, sc *scriptContext, shapes map[string]assocShape, + assoc ast.QualifiedName, startEntity string) (shape assocRetrieveShape, known bool) { - shape, ok := shapes[strings.ToLower(qualifyAssoc(ctx, shapes, assoc, startEntity))] - if !ok || shape.owner == "" || strings.EqualFold(shape.from, shape.to) { - return "", false, false + s, ok := shapes[strings.ToLower(qualifyAssoc(ctx, shapes, assoc, startEntity))] + if !ok || s.owner == "" { + return assocRetrieveShape{}, false } - // The start may be a specialization of an end; walk as far as the - // generalization chain can be read, and stay silent if no link is an end. - chain, _ := generalizationChain(ctx, startEntity) + // The start may be a specialization of an end; read as much of the + // generalization chain as can be read, through the script's entities first + // (a specialization the same script creates is not in the project yet). + chain, _ := scriptGeneralizationChain(ctx, sc, startEntity) if len(chain) == 0 { chain = []string{startEntity} } - for _, qn := range chain { - switch { - case strings.EqualFold(qn, shape.from): - return shape.to, associationRetrieveYieldsList(shape.refSet, shape.owner == "Both", false), true - case strings.EqualFold(qn, shape.to): - return shape.from, associationRetrieveYieldsList(shape.refSet, shape.owner == "Both", true), true + isA := func(qn string) bool { + for _, link := range chain { + if strings.EqualFold(link, qn) { + return true + } } + return false } - return "", false, false + self := strings.EqualFold(s.from, s.to) + ownerBoth := s.owner == "Both" + switch associationRetrieveEnd(isA(s.from), isA(s.to), self) { + case assocEndFrom: + return assocRetrieveShape{qn: s.qn, other: s.to, selfReference: self, + isList: associationRetrieveYieldsList(s.refSet, ownerBoth, false)}, true + case assocEndTo: + return assocRetrieveShape{qn: s.qn, other: s.from, fromTo: true, selfReference: self, + isList: associationRetrieveYieldsList(s.refSet, ownerBoth, true)}, true + } + return assocRetrieveShape{}, false } func assocListSuggestion(name string, b assocListBinding) string { + if b.selfReference { + // Which way a database constraint over a self-reference runs is its own + // question; keep to the list the retrieve already has. + return fmt.Sprintf("Keep the list and work on its items (`loop $item in $%s`, or `$item = head($%s);`).", name, name) + } return fmt.Sprintf("Keep the list and work on its items (`loop $item in $%s`, or `$item = head($%s);`), "+ "or retrieve one %s from the database: `retrieve $%s from %s where [%s = $%s] first;`.", name, name, b.elem, name, b.elem, b.assoc, b.start) diff --git a/mdl/executor/validate_retrieve_self_reference_test.go b/mdl/executor/validate_retrieve_self_reference_test.go new file mode 100644 index 0000000000..daa03dd20d --- /dev/null +++ b/mdl/executor/validate_retrieve_self_reference_test.go @@ -0,0 +1,222 @@ +// SPDX-License-Identifier: Apache-2.0 + +// ako/mxcli#1793 for check --references: what the full build now types a +// self-reference retrieve as, the check says too. The project is +// selfReferenceBackend (cmd_microflows_builder_retrieve_self_reference_test.go), +// whose header holds the 11.13.0 measurements. +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// selfRefCheckErrors runs check --references' association-retrieve pass over +// flow, against the selfReferenceBackend project. +func selfRefCheckErrors(t *testing.T, flow string) []string { + t.Helper() + ctx, _ := newMockCtx(t, withBackend(selfReferenceBackend())) + prog := parseMDL(t, flow) + sc := newScriptContext() + sc.collectDefinitions(prog) + var out []string + for _, err := range validateAssociationRetrieveListUse(ctx, prog, sc) { + out = append(out, err.Error()) + } + return out +} + +const selfRefSubFlows = ` +create microflow Sample.SubList ($Nodes: list of Sample.Node) +begin + log info node 'S' 'sub'; +end; +create microflow Sample.SubOne ($Node: Sample.Node) +begin + log info node 'S' 'sub'; +end; +` + +// TestAssociationRetrieveObjectUsedAsList: check --references refuses a list +// use of a retrieve that binds one object — a Reference followed from its FROM +// end, which a self-reference always is — naming the CE code the build +// reports. The controls are the shapes that build. +func TestAssociationRetrieveObjectUsedAsList(t *testing.T) { + for _, tc := range []struct { + name, body string + want []string // nil = no error + }{ + {"self-reference, count (the report)", `retrieve $P from $Node/Sample.Node_Parent; + $n = count $P;`, []string{"count", "CE0013", "Input variable 'P' must be of type 'List'."}}, + {"self-reference, sum", `retrieve $P from $Node/Sample.Node_Parent; + $n = sum $P by Amount;`, []string{"sum", "CE0013"}}, + {"self-reference, head", `retrieve $P from $Node/Sample.Node_Parent; + $h = head $P;`, []string{"head", "CE0097", "The selected 'P' variable must be of type List."}}, + {"self-reference, filter", `retrieve $P from $Node/Sample.Node_Parent; + $f = filter $P by Amount = 1;`, []string{"filter", "CE0097"}}, + {"self-reference, the first operand of contains", `retrieve $P from $Node/Sample.Node_Parent; + $b = contains $Node in $P;`, []string{"contains", "CE0097"}}, + {"self-reference, reduce", `retrieve $P from $Node/Sample.Node_Parent; + $r = reduce $P from 0 as Integer using $currentResult + 1;`, []string{"reduce", "CE0013"}}, + {"self-reference, loop", `retrieve $P from $Node/Sample.Node_Parent; + loop $x in $P begin + log info node 'S' 'x'; + end loop;`, []string{"a loop", "CE0100", "'P' is of type Sample.Node, but should be of type List."}}, + {"self-reference, a list parameter", `retrieve $P from $Node/Sample.Node_Parent; + call microflow Sample.SubList(Nodes = $P);`, []string{"parameter Nodes of Sample.SubList", "CE0117"}}, + {"self-reference from a specialization, count", `retrieve $P from $Special/Sample.Node_Parent; + $n = count $P;`, []string{"CE0013", "a Sample.SpecialNode"}}, + {"self-reference in a loop body, count", `loop $it in $Nodes begin + retrieve $P from $it/Sample.Node_Parent; + $n = count $P; + end loop;`, []string{"CE0013"}}, + {"a forward Reference, count", `retrieve $C from $Line/Sample.Line_Node; + $n = count $C;`, []string{"Sample.Line_Node", "CE0013", "Input variable 'C' must be of type 'List'."}}, + + // Controls: the shapes that build (measured on 11.13.0), and the ones + // whose shape is not established. + {"control: self-reference, change", `retrieve $P from $Node/Sample.Node_Parent; + change $P (Amount = 1);`, nil}, + {"control: self-reference, a member path", `retrieve $P from $Node/Sample.Node_Parent; + declare $a Integer = $P/Amount;`, nil}, + {"control: self-reference, an object parameter", `retrieve $P from $Node/Sample.Node_Parent; + call microflow Sample.SubOne(Node = $P);`, nil}, + {"control: self-reference set, count and loop", `retrieve $R from $Node/Sample.Node_Related; + $n = count $R; + loop $x in $R begin + change $x (Amount = 1); + end loop;`, nil}, + {"control: add, remove, clear (Change list activities mx check does not type)", `retrieve $P from $Node/Sample.Node_Parent; + add $Node to $P; + remove $Node from $P; + clear $P;`, nil}, + {"control: range (mx check reports nothing on it)", `retrieve $P from $Node/Sample.Node_Parent; + $r = range $P limit 1;`, nil}, + {"control: the object as a list operation's second operand", `retrieve $P from $Node/Sample.Node_Parent; + $u = union $Nodes with $P; + $b = contains $P in $Nodes;`, nil}, + {"control: a Reference from its TO end is the list", `retrieve $L from $Node/Sample.Line_Node; + $n = count $L;`, nil}, + {"control: a rebinding clears the object", `retrieve $P from $Node/Sample.Node_Parent; + retrieve $P from Sample.Node; + $n = count $P;`, nil}, + {"control: a specialization of both ends of two entities is not judged", `retrieve $A from $Dog/Sample.Dog_Animal; + $n = count $A; + change $A (Name = 'x'); + retrieve $B from $Dog/Sample.Animal_Dog; + $m = count $B; + change $B (Name = 'x');`, nil}, + } { + t.Run(tc.name, func(t *testing.T) { + errs := selfRefCheckErrors(t, selfRefSubFlows+` +create microflow Sample.F ($Node: Sample.Node, $Special: Sample.SpecialNode, $Line: Sample.Line, + $Dog: Sample.Dog, $Nodes: list of Sample.Node) +begin + `+tc.body+` +end;`) + if tc.want == nil { + if len(errs) != 0 { + t.Fatalf("want no error, got %v", errs) + } + return + } + if len(errs) != 1 { + t.Fatalf("want 1 error, got %d: %v", len(errs), errs) + } + for _, w := range append(tc.want, "MDL-RETRIEVE01", "one Sample.") { + if !strings.Contains(errs[0], w) { + t.Errorf("error should mention %q:\n%s", w, errs[0]) + } + } + }) + } +} + +// TestAssociationRetrieveObjectUsedAsList_ScriptDeclaredSpecialization: the +// specialization and the self-reference are created by the script being +// checked, so neither is in the project yet; the start end is decided through +// the script's declarations, as exec decides it after creating them. +func TestAssociationRetrieveObjectUsedAsList_ScriptDeclaredSpecialization(t *testing.T) { + ctx, _ := newMockCtx(t) + prog := parseMDL(t, `create module Tmp; +create persistent entity Tmp.Order ( Code: String(50) ); +create persistent entity Tmp.SpecialOrder extends Tmp.Order ( Extra: String(20) ); +create association Tmp.Order_Parent from Tmp.Order to Tmp.Order; +create microflow Tmp.F ($S: Tmp.SpecialOrder) returns Integer as $n +begin + retrieve $P from $S/Tmp.Order_Parent; + $n = count $P; + return $n; +end; +create microflow Tmp.G ($S: Tmp.SpecialOrder) +begin + retrieve $P from $S/Tmp.Order_Parent; + change $P (Code = 'x'); +end;`) + sc := newScriptContext() + sc.collectDefinitions(prog) + errs := validateAssociationRetrieveListUse(ctx, prog, sc) + if len(errs) != 1 { + t.Fatalf("want 1 error (the count in Tmp.F), got %d: %v", len(errs), errs) + } + for _, w := range []string{"statement 5", "a Tmp.SpecialOrder", "one Tmp.Order", "CE0013", "MDL-RETRIEVE01"} { + if !strings.Contains(errs[0].Error(), w) { + t.Errorf("error should mention %q:\n%s", w, errs[0]) + } + } +} + +// TestAssociationRetrieveSelfReferenceSetUsedAsObject: a self-reference is no +// longer outside MDL-RETRIEVE03. Over a reference set it is a list from the +// FROM end as from any other, and a member path through it is CE0117 (measured +// on 11.13.0) — reported, with a suggestion that does not guess which way a +// database constraint over the self-reference runs. +func TestAssociationRetrieveSelfReferenceSetUsedAsObject(t *testing.T) { + errs := selfRefCheckErrors(t, `create microflow Sample.F ($Node: Sample.Node) returns Integer as $a +begin + retrieve $R from $Node/Sample.Node_Related; + declare $a Integer = $R/Amount; + return $a; +end;`) + if len(errs) != 1 { + t.Fatalf("want 1 error, got %d: %v", len(errs), errs) + } + for _, w := range []string{"MDL-RETRIEVE03", "member path `$R/…`", "is a list of Sample.Node", "CE0117"} { + if !strings.Contains(errs[0], w) { + t.Errorf("error should mention %q:\n%s", w, errs[0]) + } + } + if strings.Contains(errs[0], "where [") { + t.Errorf("the suggestion should not spell a database constraint over a self-reference:\n%s", errs[0]) + } +} + +// TestCheckSetOnSelfReferenceObject: check --references' flow validator types +// a self-reference retrieve as the object the full build now writes, so `set` +// on it is refused as it is for every object (MDL-SET01's CE7247), where it +// used to pass as a Change list Replace; over a self reference set it is still +// a list, and `set` is still a Change list Replace. +func TestCheckSetOnSelfReferenceObject(t *testing.T) { + ctx, _ := newMockCtx(t, withBackend(selfReferenceBackend())) + shapes := checkAssociationShapes(ctx, nil) + flow := func(assoc string) *ast.CreateMicroflowStmt { + prog := parseMDL(t, `create microflow Sample.F ($A: Sample.Node, $B: Sample.Node) +begin + retrieve $Cursor from $A/Sample.`+assoc+`; + retrieve $Next from $B/Sample.`+assoc+`; + set $Cursor = $Next; +end;`) + return prog.Statements[0].(*ast.CreateMicroflowStmt) + } + ref := flow("Node_Parent") + if got := strings.Join(validateFlowBody(ref.Parameters, nil, ref.Body, true, shapes), "\n"); !strings.Contains(got, "CE7247") || + !strings.Contains(got, "$Cursor") { + t.Errorf("set on a self-reference retrieve: want the CE7247 refusal, got %q", got) + } + set := flow("Node_Related") + if got := validateFlowBody(set.Parameters, nil, set.Body, true, shapes); len(got) != 0 { + t.Errorf("set on a self reference set retrieve (a list): want no error, got %v", got) + } +} diff --git a/mdl/executor/version_gates.go b/mdl/executor/version_gates.go index 24fd772fdb..930bb901c4 100644 --- a/mdl/executor/version_gates.go +++ b/mdl/executor/version_gates.go @@ -33,6 +33,12 @@ type versionGate struct { // (requireMessageDefinitionDocuments on a drop/alter of a document that is not // there), stays with its handler: check cannot know it from the script. func statementVersionGates(stmt ast.Statement) []versionGate { + return append(statementCaseGates(stmt), documentGates(stmt)...) +} + +// statementCaseGates are the gates statementVersionGates names case by case; +// documentTypeGates add the document shapes (ako/mxcli#1811). +func statementCaseGates(stmt ast.Statement) []versionGate { switch s := stmt.(type) { case *ast.CreateRestClientStmt: return []versionGate{{"integration", "rest_client_basic", "create rest client", @@ -74,8 +80,7 @@ func statementVersionGates(stmt ast.Statement) []versionGate { "On an earlier version, model the buckets as your own entity and target user tasks with a microflow."}} } case *ast.CreateViewEntityStmt: - gates := []versionGate{{"domain_model", "view_entities", "create view entity", - "upgrade your project to 10.18+ or use a regular entity with a microflow data source"}} + gates := []versionGate{viewEntityGate.versionGate} // An OQL comment — `--` or `/* */`, anywhere in the query — is CE0174 // "Error(s) in OQL query: The '-' part is incomplete or incorrect" below // the floor. The query now keeps a comment before its first token or @@ -144,14 +149,7 @@ func statementVersionGates(stmt ast.Statement) []versionGate { case *ast.CreateSnippetStmtV3, *ast.AlterPageStmt: return flowReturnsGates(s) case *ast.CreateMicroflowStmt: - var gates []versionGate - // SEND EMAIL writes the 11.13+ shape (StringTemplate subject and - // bodies); an older project stores a different EmailMessage, or no - // such activity. - if bodyContains(s.Body, func(st ast.MicroflowStatement) bool { _, ok := st.(*ast.SendEmailStmt); return ok }) { - gates = append(gates, versionGate{"microflows", "send_email", "send email", - "use the Email Connector module, or upgrade the project to 11.13+"}) - } + gates := microflowBodyGates(s.Parameters, s.Body) // Deep links are Mendix 10.6+. Gated on the clause being STATED, not on // the resulting value: a rewrite that carries a stored URL forward on // an older project preserves what is already there. @@ -159,21 +157,26 @@ func statementVersionGates(stmt ast.Statement) []versionGate { gates = append(gates, versionGate{"microflows", "deep_link_url", "a microflow URL (deep link)", "upgrade your project to 10.6+, or set the URL in Studio Pro"}) } - if bodyContains(s.Body, selectsWorkflowDefinition) { - gates = append(gates, workflowDefinitionSelectionGate) - } return gates case *ast.CreateNanoflowStmt: - // SYNCHRONIZE UNSYNCHRONIZED is the one mode with a floor inside - // Mendix 9: SynchronizationType.Unsynchronized was introduced in 9.4.0, - // while the activity and its other two modes go back further. Gate the - // mode, not the statement, so a 9.0-9.3 project keeps `synchronize all`. - if bodyUsesUnsynchronized(s.Body) { - return []versionGate{{"microflows", "synchronize_unsynchronized", "synchronize unsynchronized", - "use `synchronize all` or `synchronize $Var`, or upgrade the project to 9.4+"}} + return nanoflowBodyGates(s.Parameters, s.Body) + case *ast.AlterFlowStmt: + // What an insert or a replace splices in is gated as the same + // statements in a `create` are; a drop writes nothing new. Without + // this, check -p passed an alter that the write then refused + // (ako/mxcli#1787). + var inserted []ast.MicroflowStatement + for _, op := range s.Operations { + inserted = append(inserted, op.Body...) + } + if s.Nanoflow { + return nanoflowBodyGates(nil, inserted) } + return microflowBodyGates(nil, inserted) case *ast.CreateWorkflowStmt: - var gates []versionGate + // `create or modify workflow` is this statement too: it writes the + // same Workflows$Workflow (ako/mxcli#1809). + gates := []versionGate{workflowBasicGate("create workflow")} if len(s.EventHandlers) > 0 { gates = append(gates, versionGate{"workflows", "event_handlers", "on workflow events", "remove the `on … workflow event` clauses, or upgrade the project"}) @@ -205,11 +208,10 @@ func statementVersionGates(stmt ast.Statement) []versionGate { gates = append(gates, notificationEventsGate) } gates = append(gates, workflowTypeGates(activities, nil)...) - return append(gates, userTaskGates(activities)...) + gates = append(gates, userTaskGates(activities)...) + return append(gates, workflowShapeGatesOf(s)...) case *ast.AlterWorkflowStmt: - // Workflows require Mendix 9.12+. - gates := []versionGate{{"workflows", "basic", "alter workflow", - "upgrade your project to Mendix 9.12+ to use workflows"}} + gates := []versionGate{workflowBasicGate("alter workflow")} added := alterWorkflowAddedActivities(s) if workflowUsesAgentTask(added) { gates = append(gates, agentTaskGate) @@ -224,11 +226,33 @@ func statementVersionGates(stmt ast.Statement) []versionGate { } } gates = append(gates, workflowTypeGates(added, inserted)...) - return append(gates, userTaskGates(added)...) + gates = append(gates, userTaskGates(added)...) + return append(gates, workflowShapeGatesOf(s)...) } return nil } +// viewEntityGate is the gate of `create view entity`. Its floor is the latest +// of what the write stores for one: the query in a separate +// DomainModels$ViewEntitySourceDocument, which the entity's source names +// (OqlViewEntitySource.SourceDocument), both 10.21 — not 10.18, where the +// OQL was inline and the write refuses the document (ako/mxcli#1811). +var viewEntityGate = typeGate{ + versionGate: versionGate{"domain_model", "view_entities", "create view entity", + "use a regular entity with a microflow data source, or upgrade the project"}, + types: []string{"DomainModels$ViewEntitySourceDocument", "DomainModels$OqlViewEntitySource"}, + properties: []string{"DomainModels$OqlViewEntitySource.SourceDocument"}, +} + +// workflowBasicGate is the gate of every statement that writes a workflow +// document: Workflows$Workflow is a 9.0.2 type, and the write refuses it below +// that (ako/mxcli#1809). The hint names no version; the message states the +// registry's floor. +func workflowBasicGate(statement string) versionGate { + return versionGate{"workflows", "basic", statement, + "this version has no workflows (Workflows$Workflow): upgrade the project"} +} + // The gates of the workflow activity and event TYPES mxcli writes that an // older Mendix does not have (ako/mxcli#1774). Each type's introduction is in // the Model SDK's version data (mendixmodelsdk 4.117.0), and the write path @@ -294,6 +318,144 @@ func workflowTypeGates(activities []ast.WorkflowActivityNode, insertedEvents []s return gates } +// microflowBodyGates are the gates of the activities in a microflow body, +// nested ones included (bodyContains walks branches, loops and error +// handlers): what `create microflow` declares and what `alter microflow` +// splices in. params are the microflow's when the statement declares them. +func microflowBodyGates(params []ast.MicroflowParam, body []ast.MicroflowStatement) []versionGate { + var gates []versionGate + // SEND EMAIL writes the 11.13+ shape (StringTemplate subject and + // bodies); an older project stores a different EmailMessage, or no + // such activity. + if bodyContains(body, func(st ast.MicroflowStatement) bool { _, ok := st.(*ast.SendEmailStmt); return ok }) { + gates = append(gates, versionGate{"microflows", "send_email", "send email", + "use the Email Connector module, or upgrade the project to 11.13+"}) + } + if bodyContains(body, selectsWorkflowDefinition) { + gates = append(gates, workflowDefinitionSelectionGate) + } + for _, g := range workflowActionGates { + if bodyContains(body, g.match) { + gates = append(gates, g.versionGate) + } + } + return append(gates, flowGates(params, body)...) +} + +// nanoflowBodyGates are the gates of the activities in a nanoflow body, as +// microflowBodyGates are a microflow's. +func nanoflowBodyGates(params []ast.MicroflowParam, body []ast.MicroflowStatement) []versionGate { + var gates []versionGate + // SYNCHRONIZE UNSYNCHRONIZED is the one mode with a floor inside + // Mendix 9: SynchronizationType.Unsynchronized was introduced in 9.4.0, + // while the activity and its other two modes go back further. Gate the + // mode, not the statement, so a 9.0-9.3 project keeps `synchronize all`. + if bodyUsesUnsynchronized(body) { + gates = append(gates, versionGate{"microflows", "synchronize_unsynchronized", "synchronize unsynchronized", + "use `synchronize all` or `synchronize $Var`, or upgrade the project to 9.4+"}) + } + return append(gates, flowGates(params, body)...) +} + +// typeGate is a gate whose floor is the introduction of the storage $Types +// its statement writes: the version below which the write refuses them +// (canon.UndeclaredTypeError). Its sdk/versions row holds that version, and +// TestTypeGates_FloorIsTheTypeIntroduction pins the row to the types, so +// check -p and the write cannot disagree about a version. +type typeGate struct { + versionGate + // types are the storage $Types the statement writes; the floor is the + // latest of their introductions and of properties'. + types []string + // properties are the "Type.Key" properties the statement writes a value + // to whose introduction is later than the types': the write refuses a + // value below it (canon.StripUndeclaredProperties), as it refuses a type. + properties []string + // match reports a statement in a flow body that writes them. + match func(ast.MicroflowStatement) bool +} + +// workflowActionGates are the gates of the workflow microflow actions Mendix +// added after 9.0 (ako/mxcli#1787), each at its types' introduction in +// mendixmodelsdk 4.117.0. Newest first, so a microflow holding several of +// these is refused naming the one whose version covers the rest. A lock or +// unlock is not here: every one check and exec accept names its workflow, +// and the selection that names it is 10.0, past the action's own 9.21 +// (workflowDefinitionSelectionGate). +var workflowActionGates = []typeGate{ + { + versionGate: versionGate{"workflows", "get_workflow_activity_records", "get workflow activity records", + "this version has no Microflows$GetWorkflowActivityRecordsAction: upgrade the project"}, + types: []string{"Microflows$GetWorkflowActivityRecordsAction"}, + match: func(st ast.MicroflowStatement) bool { _, ok := st.(*ast.GetWorkflowActivityRecordsStmt); return ok }, + }, + { + versionGate: versionGate{"workflows", "get_workflows", "get workflows", + "this version has no Microflows$GetWorkflowsAction: upgrade the project"}, + types: []string{"Microflows$GetWorkflowsAction"}, + match: func(st ast.MicroflowStatement) bool { _, ok := st.(*ast.GetWorkflowsStmt); return ok }, + }, + { + versionGate: versionGate{"workflows", "notify_workflow", "notify workflow", + "this version has no Microflows$NotifyWorkflowAction, nor the wait for notification it resumes: upgrade the project"}, + types: []string{"Microflows$NotifyWorkflowAction"}, + match: func(st ast.MicroflowStatement) bool { _, ok := st.(*ast.NotifyWorkflowStmt); return ok }, + }, + workflowOperationGate("retry", "Microflows$RetryOperation", "workflow_operation_retry_unpause", resumeOperationHint), + workflowOperationGate("unpause", "Microflows$UnpauseOperation", "workflow_operation_retry_unpause", resumeOperationHint), + { + versionGate: versionGate{"workflows", "get_workflow_data", "get workflow data", + "this version has no Microflows$GetWorkflowDataAction: upgrade the project"}, + types: []string{"Microflows$GetWorkflowDataAction"}, + match: func(st ast.MicroflowStatement) bool { _, ok := st.(*ast.GetWorkflowDataStmt); return ok }, + }, + workflowOperationGate("pause", "Microflows$PauseOperation", "workflow_operation_pause_continue", + "this version has no Microflows$PauseOperation: upgrade the project"), + workflowOperationGate("continue", "Microflows$ContinueOperation", "workflow_operation_pause_continue", + "this version has no Microflows$ContinueOperation: upgrade the project"), + workflowOperationGate("abort", "Microflows$AbortOperation", "workflow_operations", + "this version has no workflow operation action (Microflows$WorkflowOperationAction): upgrade the project"), + workflowOperationGate("restart", "Microflows$RestartOperation", "workflow_operations", + "this version has no workflow operation action (Microflows$WorkflowOperationAction): upgrade the project"), + workflowMicroflowActionGate("call workflow", "Microflows$WorkflowCallAction", + func(st ast.MicroflowStatement) bool { _, ok := st.(*ast.CallWorkflowStmt); return ok }), + workflowMicroflowActionGate("set task outcome", "Microflows$SetTaskOutcomeAction", + func(st ast.MicroflowStatement) bool { _, ok := st.(*ast.SetTaskOutcomeStmt); return ok }), + workflowMicroflowActionGate("open user task", "Microflows$OpenUserTaskAction", + func(st ast.MicroflowStatement) bool { _, ok := st.(*ast.OpenUserTaskStmt); return ok }), + workflowMicroflowActionGate("open workflow", "Microflows$OpenWorkflowAction", + func(st ast.MicroflowStatement) bool { _, ok := st.(*ast.OpenWorkflowStmt); return ok }), +} + +// resumeOperationHint is the hint of retry and unpause, which replaced one +// Resume operation (introduced 9.3.0, deleted 9.12.0 in mendixmodelsdk 4.117.0). +const resumeOperationHint = "before this version, retry and unpause are one Resume operation (Microflows$ResumeOperation, " + + "Mendix 9.3 to 9.11), which mxcli does not write: set it in Studio Pro, or upgrade the project" + +// workflowOperationGate is the gate of `workflow operation `, which +// writes a Microflows$WorkflowOperationAction holding operation. +func workflowOperationGate(kind, operation, row, hint string) typeGate { + return typeGate{ + versionGate: versionGate{"workflows", row, "workflow operation " + kind, hint}, + types: []string{"Microflows$WorkflowOperationAction", operation}, + match: func(st ast.MicroflowStatement) bool { + s, ok := st.(*ast.WorkflowOperationStmt) + return ok && s.OperationType == kind + }, + } +} + +// workflowMicroflowActionGate is the gate of a microflow action Mendix added +// with workflows themselves. +func workflowMicroflowActionGate(statement, typ string, match func(ast.MicroflowStatement) bool) typeGate { + return typeGate{ + versionGate: versionGate{"workflows", "workflow_microflow_actions", statement, + "this version has no " + typ + ": the microflow actions arrived with workflows, so upgrade the project"}, + types: []string{typ}, + match: match, + } +} + // selectsWorkflowDefinition reports a lock or unlock that names its workflow // (by name or by variable): the action stores it as a // Workflows$WorkflowDefinition*Selection, a 10.0 type. diff --git a/mdl/executor/version_gates_coverage_1811_test.go b/mdl/executor/version_gates_coverage_1811_test.go new file mode 100644 index 0000000000..21f91ac426 --- /dev/null +++ b/mdl/executor/version_gates_coverage_1811_test.go @@ -0,0 +1,321 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "os" + "path/filepath" + "regexp" + "sort" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/modelsdk/canon" + "github.com/mendixlabs/mxcli/modelsdk/version" + "github.com/mendixlabs/mxcli/sdk/versions" +) + +// allTypeGates are the gates pinned to the types (and properties) their +// statements write: TestTypeGates_FloorIsTheTypeIntroduction holds each row to +// the latest of them, and TestTypeGates_CoverEveryWrittenType counts a type one +// of them names as gated. +func allTypeGates() []typeGate { + gates := []typeGate{ + // ako/mxcli#1809: every statement that writes a workflow document. + {versionGate: workflowBasicGate("create workflow"), types: []string{"Workflows$Workflow"}}, + {versionGate: waitForNotificationGate, types: []string{"Workflows$WaitForNotificationActivity"}}, + {versionGate: waitForTimerGate, types: []string{"Workflows$WaitForTimerActivity"}}, + {versionGate: timerBoundaryEventGate, types: []string{"Workflows$TimerBoundaryEvent"}}, + {versionGate: interruptingTimerBoundaryEventGate, types: []string{ + "Workflows$InterruptingTimerBoundaryEvent", "Workflows$NonInterruptingTimerBoundaryEvent"}}, + {versionGate: workflowDefinitionSelectionGate, types: []string{ + "Workflows$WorkflowDefinitionNameSelection", "Workflows$WorkflowDefinitionObjectSelection"}}, + viewEntityGate, + } + gates = append(gates, workflowActionGates...) + gates = append(gates, flowActionGates...) + for _, g := range workflowShapeGates { + gates = append(gates, g.typeGate) + } + for _, g := range documentTypeGates { + gates = append(gates, g.typeGate) + } + return gates +} + +// broaderTypeGates are written types a gate covers without naming them: the +// gate's statement is the only one that writes the type, and its floor is at +// or above the type's (TestTypeGates_CoverEveryWrittenType checks the floor). +var broaderTypeGates = []struct { + gate versionGate + types []string + why string +}{ + {workflowBasicGate("create workflow"), []string{"Workflows$Flow", "Workflows$BooleanConditionOutcome", + "Workflows$CallMicroflowTask", "Workflows$EndWorkflowActivity", "Workflows$EnumerationValueConditionOutcome", + "Workflows$ExclusiveSplitActivity", "Workflows$MicroflowCallParameterMapping", "Workflows$ParallelSplitActivity", + "Workflows$ParallelSplitOutcome", "Workflows$UserTaskOutcome", "Workflows$VoidConditionOutcome"}, + "written only inside a workflow"}, + {versionGate{"workflows", "multi_user_task", "multi user task", ""}, []string{"Workflows$ConsensusCompletionCriteria", + "Workflows$VetoCompletionCriteria", "Workflows$AllUserInput", "Workflows$AbsoluteAmountUserInput", + "Workflows$PercentageAmountUserInput"}, "a multi-user task's rule and participants"}, + {versionGate{"workflows", "multi_user_task_majority_threshold", "decide by majority", ""}, + []string{"Workflows$MajorityCompletionCriteria", "Workflows$ThresholdCompletionCriteria"}, "decide by majority / threshold"}, + {versionGate{"workflows", "multi_user_task_microflow_rule", "decide by microflow", ""}, + []string{"Workflows$MicroflowCompletionCriteria"}, "decide by microflow"}, + {versionGate{"workflows", "event_handlers", "on workflow events", ""}, + []string{"Workflows$WorkflowEventHandler", "Workflows$MicroflowEventHandler"}, "written only in a workflow event handler"}, + {versionGate{"workflows", "group_targeting", "targeting groups on a user task", ""}, + []string{"Workflows$XPathGroupTargeting", "Workflows$MicroflowGroupTargeting"}, "targeting groups"}, + {versionGate{"workflows", "event_subprocesses", "event subprocess", ""}, []string{"Workflows$EventSubProcess", + "Workflows$NonInterruptingNotificationEventSubProcessStartActivity"}, "an event sub-process and its notification start"}, + {versionGate{"workflows", "timer_event_subprocesses", "event subprocess … on … timer", ""}, + []string{"Workflows$InterruptingTimerEventSubProcessStartActivity", "Workflows$NonInterruptingTimerEventSubProcessStartActivity"}, + "a timer-started event sub-process"}, + {notificationEventsGate, []string{"Workflows$NotificationActivity", "Workflows$InterruptingNotificationBoundaryEvent", + "Workflows$NonInterruptingNotificationBoundaryEvent"}, "notification activities and boundary events"}, + {agentTaskGate, []string{"Workflows$AIAgentTaskActivity"}, "call agent microflow"}, + {timerBoundaryEventGate, []string{"Workflows$EndOfBoundaryEventPathActivity"}, + "ends a boundary event's path; the earliest boundary event is the 10.14 timer"}, + {interruptingTimerBoundaryEventGate, []string{"Workflows$LinearRecurrence"}, + "a non-interrupting timer boundary event's `repeat every`, or an event sub-process timer's (11.15)"}, + {flowReturnsGate, []string{"Forms$OutputMapping"}, "returns ( … ) on a flow call"}, + {versionGate{"integration", "published_odata_service", "create published odata service", ""}, []string{ + "ODataPublish$PublishedODataService2", "ODataPublish$EntityType", "ODataPublish$EntitySet", + "ODataPublish$PublishedAttribute", "ODataPublish$PublishedAssociationEnd", "ODataPublish$PublishedId", + "ODataPublish$PublishedMicroflow", "ODataPublish$PublishedMicroflowParameter", "ODataPublish$QueryOptions", + "ODataPublish$ReadSource", "ODataPublish$ChangeSource", "ODataPublish$ChangeNotSupported", + "ODataPublish$CallMicroflowToRead", "ODataPublish$CallMicroflowToChange"}, "a published OData service's document"}, + {versionGate{"integration", "rest_client_basic", "create rest client", ""}, []string{"Rest$ConsumedRestService", + "Rest$ValueTemplate", "Rest$StringValue", "Rest$RestOperation", "Rest$OperationParameter", "Rest$HeaderWithValueTemplate"}, + "a consumed REST service's document"}, + {versionGate{"integration", "rest_client_authentication", "authentication: basic", ""}, + []string{"Rest$ConstantValue"}, "a basic authentication's constant user name or password"}, + {versionGate{"integration", "rest_client_operations", "an operation of a consumed REST service", ""}, + []string{"Rest$ImplicitMappingBody", "Rest$ImplicitMappingResponseHandling"}, "an operation's body and response mapping"}, + {versionGate{"integration", "data_transformer", "create data transformer", ""}, []string{"DataTransformers$DataTransformer", + "DataTransformers$JsltAction", "DataTransformers$JsonSource", "DataTransformers$Step", "DataTransformers$StructureObject"}, + "a data transformer's document"}, + {agentDocumentGate("agent", "create agent"), []string{"CustomBlobDocuments$CustomBlobDocument", + "CustomBlobDocuments$CustomBlobDocumentMetadata"}, "the agent-editor documents (all four rows are 11.9)"}, + {messageDefinitionDocumentGate("create message definition"), []string{"MessageDefinitions$MessageDefinition2"}, + "the per-document message definition"}, + {versionGate{"workflows", "groups", "alter settings workflows add group", ""}, []string{"Settings$WorkflowGroup"}, "a workflow group"}, + {viewEntityGate.versionGate, []string{"DomainModels$OqlViewValue", "DomainModels$OqlViewAssociationSource"}, + "a view entity's attributes and associations"}, + {versionGate{"microflows", "send_email", "send email", ""}, []string{"Microflows$SendEmailAction", "Microflows$EmailMessage", + "Microflows$BasicAuthConfig", "Microflows$EmailConnectionConfig", "Microflows$EmailCustomHeader"}, + "send email writes the 11.13 shape"}, + {versionGate{"integration", "database_connector_basic", "create database connection", ""}, []string{ + "DatabaseConnector$DatabaseQuery", "DatabaseConnector$QueryParameter", "DatabaseConnector$TableMapping", + "DatabaseConnector$ColumnMapping", "DatabaseConnector$SimpleSqlDataType"}, "a database connection's queries"}, + {workflowDefinitionSelectionGate, []string{"Microflows$LockWorkflowAction", "Microflows$UnlockWorkflowAction"}, + "every lock or unlock check and exec accept names its workflow (#1787)"}, +} + +// typeCoverageExceptions are the written types above 9.0.0 that no gate +// names, each with why no statement reaches a project below the type's floor +// with it — or, marked RESIDUAL, why the statement cannot tell and the write +// alone refuses it there. +var typeCoverageExceptions = map[string]string{ + // Lowered: an older version stores the same thing in another shape, which + // the writer or canon.LowerToProjectShape writes instead. + "Microflows$BezierCurve": "lowered: below 10.16 a flow stores Origin/DestinationBezierVector (microflow_write.go hasLine)", + "Workflows$SingleUserTaskActivity": "lowered: below 10.12 a user task is a Workflows$UserTask (canon lowerUserTaskType)", + "Workflows$MultiUserTaskActivity": "lowered: below 10.12 a user task is a Workflows$UserTask (canon lowerUserTaskType)", + "Workflows$EndOfParallelSplitPathActivity": "lowered: below 10.21 a parallel path holds its activities only (canon unmarkedPaths)", + "Workflows$StartWorkflowActivity": "lowered: the writer stores a start activity from 11.3 only (workflowStoresStartAndEnd)", + "Workflows$CallMicroflowActivity": "lowered: below 11.9 the activity is stored as Workflows$CallMicroflowTask (usesCallMicroflowActivityName)", + "Workflows$XPathUserTargeting": "lowered: below 11.2 a user task's targeting is its user source (canon userSource)", + "Workflows$MicroflowUserTargeting": "lowered: below 11.2 a user task's targeting is its user source (canon userSource)", + "Workflows$NoUserTargeting": "lowered: below 11.2 a user task's targeting is its user source (canon userSource)", + "Settings$PrivateValue": "lowered: below 10.9 a configuration's constant value is flat (canon flatConstant)", + "Settings$SharedValue": "lowered: below 10.9 a configuration's constant value is flat (canon flatConstant)", + "Rest$JsonBody": "lowered: below 11.0 a JSON body is a Rest$StringBody (restBodyForms)", + "Workflows$NotifyWaitForNotificationActivityTarget": "lowered: below 11.7 notify workflow stores its target as the action's Activity (setNotifyTarget)", + // Unreachable: the type is written only for something a project below + // its floor cannot hold. + "Workflows$NonInterruptingNotificationEventSubProcessStartActivityTarget": "unreachable: a notify target of that kind, which the stored workflow holds only from 11.8 (notifyTargetType)", + "Workflows$InterruptingNotificationEventSubProcessStartActivityTarget": "unreachable: a notify target of that kind, which the stored workflow holds only from 11.10 (notifyTargetType)", + "Workflows$NotifyNotificationActivityTarget": "unreachable: a notify target of that kind, which the stored workflow holds only from 11.11 (notifyTargetType)", + "Workflows$NotifyNotificationBoundaryEventTarget": "unreachable: a notify target of that kind, which the stored workflow holds only from 11.11 (notifyTargetType)", + "Forms$SnippetParameterMapping": "unreachable: a snippet call maps only the parameters the called snippet declares, and a snippet holds parameters from 9.21 (Forms$Snippet.Parameters)", + "CodeActions$CustomBlobDocumentParameterType": "unreachable: carried only from a stored Java action parameter of that type, which a project below 11.6 cannot hold", + "Microflows$CustomBlobDocumentCodeActionParameterValue": "unreachable: written only for a Java action parameter of a custom document type, which a project below 11.6 cannot hold", + // Residual: the statement does not say which shape the writer picks, so + // no gate can; the write refuses it below its floor. + "Forms$ToggleDesignPropertyValue": "lowered below 10.2 to a flat DesignPropertyValue (canon lowerDesignProperty)", + "Forms$OptionDesignPropertyValue": "lowered below 10.2 to a flat DesignPropertyValue (canon lowerDesignProperty)", + "Forms$CustomDesignPropertyValue": "RESIDUAL: below 10.2 the lowering refuses it at write time; which kind a design property is comes from the theme's definitions, not the statement", + "Forms$CompoundDesignPropertyValue": "RESIDUAL: below 10.2 the lowering refuses it at write time; which kind a design property is comes from the theme's definitions, not the statement", + "Rest$ODataEntityTypeSource": "RESIDUAL: an external entity's source kind comes from the consumed service's $metadata (cmd_contract.go: a type with no entity set), not the statement", + "Rest$ODataPrimitiveCollectionEntitySource": "RESIDUAL: chosen by the consumed service's $metadata (a primitive collection), not the statement", + "Rest$ODataPrimitiveCollectionAssociationSource": "RESIDUAL: chosen by the consumed service's $metadata (a primitive collection), not the statement", + "Rest$ODataMappedPrimitiveCollectionValue": "RESIDUAL: chosen by the consumed service's $metadata (a primitive collection), not the statement", + "Microflows$QueryParameterMapping": "RESIDUAL: written for an argument that is not one of the operation's path parameters: a query parameter (Rest$QueryParameter, itself 11.0) or a name the stored operation lacks", + "Projects$ModuleSettings": "RESIDUAL: every module creation writes it — create module, and the module any statement auto-creates — so no one statement can be gated, and below 9.12 the write refuses the module on ModuleSettings.ProtectedModuleType anyway: the fix is to lower the settings unit, not a gate", +} + +// The write paths: the converters and the builders that name storage types. +var writtenTypeSources = []string{ + "../backend/modelsdk/*.go", + "../backend/pagemutator/*.go", + "../scheduledevents/*.go", + "../settingsoverlay/*.go", + "../../sdk/*/*.go", + "cmd_microflows_builder*.go", + "cmd_pages_builder*.go", +} + +var ( + typeLiteralRe = regexp.MustCompile(`"([A-Z][A-Za-z]+\$[A-Za-z0-9_]+)"`) + genImportRe = regexp.MustCompile(`(\w+)\s+"github.com/mendixlabs/mxcli/modelsdk/gen/(\w+)"`) + genCallRe = regexp.MustCompile(`\b(\w+)\.New([A-Z]\w*)\(`) + genTypeNameRe = regexp.MustCompile(`func init(\w+)\(\) \*\w+ \{\n\to := &\w+\{\}\n\to\.SetTypeName\("([^"]+)"\)`) + readOnlySource = regexp.MustCompile(`_test\.go$|_read|describe|format|mermaid|diff`) +) + +// writtenTypes are the storage $Types the write paths construct — a +// "Module$Type" literal, or a modelsdk/gen constructor resolved to the type it +// stamps — with the files that do, leaving out readers and describers. +func writtenTypes(t *testing.T) map[string][]string { + t.Helper() + genTypes := map[string]map[string]string{} + genType := func(pkg, name string) string { + if genTypes[pkg] == nil { + genTypes[pkg] = map[string]string{} + files, _ := filepath.Glob(filepath.Join("..", "..", "modelsdk", "gen", pkg, "*.go")) + for _, f := range files { + b, err := os.ReadFile(f) + if err != nil { + t.Fatal(err) + } + for _, m := range genTypeNameRe.FindAllStringSubmatch(string(b), -1) { + genTypes[pkg][m[1]] = m[2] + } + } + } + return genTypes[pkg][name] + } + out := map[string][]string{} + for _, pattern := range writtenTypeSources { + files, err := filepath.Glob(pattern) + if err != nil || len(files) == 0 { + t.Fatalf("no write path matches %s", pattern) + } + for _, f := range files { + if readOnlySource.MatchString(filepath.Base(f)) { + continue + } + b, err := os.ReadFile(f) + if err != nil { + t.Fatal(err) + } + src := string(b) + aliases := map[string]string{} + for _, m := range genImportRe.FindAllStringSubmatch(src, -1) { + aliases[m[1]] = m[2] + } + seen := map[string]bool{} + for _, line := range strings.Split(src, "\n") { + if strings.HasPrefix(strings.TrimSpace(line), "//") { + continue + } + var types []string + for _, m := range typeLiteralRe.FindAllStringSubmatch(line, -1) { + types = append(types, m[1]) + } + for _, m := range genCallRe.FindAllStringSubmatch(line, -1) { + if pkg, ok := aliases[m[1]]; ok { + if typ := genType(pkg, m[2]); typ != "" { + types = append(types, typ) + } + } + } + for _, typ := range types { + if !seen[typ] { + seen[typ] = true + out[typ] = append(out[typ], filepath.ToSlash(f)) + } + } + } + } + } + return out +} + +// ako/mxcli#1811: every storage $Type a write path constructs whose +// introduction is above the lowest supported version (9.0.0) is gated — named +// by a type gate, or covered by a gate whose floor is at or above it — or +// listed in typeCoverageExceptions with why no statement reaches a project +// below its floor. A converter that starts writing a newer type fails here +// until it is one or the other, instead of passing `check -p` and being +// refused by the write part-way through a script, which is how #1774, #1787, +// #1809 and #1811 shipped. +func TestTypeGates_CoverEveryWrittenType(t *testing.T) { + reg, err := versions.Load() + if err != nil { + t.Fatal(err) + } + lowest := version.Version{Major: 9} + named := map[string]bool{} + for _, g := range allTypeGates() { + for _, typ := range g.types { + named[typ] = true + } + } + broader := map[string]versionGate{} + for _, b := range broaderTypeGates { + if b.why == "" { + t.Errorf("%s.%s covers %v with no reason", b.gate.area, b.gate.name, b.types) + } + for _, typ := range b.types { + broader[typ] = b.gate + } + } + written := writtenTypes(t) + var types []string + for typ := range written { + types = append(types, typ) + } + sort.Strings(types) + counted := 0 + for _, typ := range types { + since, ok := canon.TypeIntroduced(typ) + if !ok || since.Compare(lowest) <= 0 { + continue + } + counted++ + at := versions.SemVer{Major: since.Major, Minor: since.Minor, Patch: since.Patch} + switch gate, ok := broader[typ]; { + case named[typ]: + case ok: + if reg.IsAvailable(gate.area, gate.name, justBelow(at)) { + t.Errorf("%s (Mendix %s+) is covered by %s.%s, which is available at %s, where the write refuses the type", + typ, since, gate.area, gate.name, justBelow(at)) + } + case typeCoverageExceptions[typ] != "": + default: + t.Errorf("%s (Mendix %s+) is written by %v and no gate names it: gate the statement that writes it "+ + "(a typeGate pinned to the type), or list it in typeCoverageExceptions with why no project below %s reaches it", + typ, since, written[typ], since) + } + } + if counted < 100 { + t.Fatalf("found only %d written types above %s: the scan of the write paths is not reading them", counted, lowest) + } + // Nothing listed is stale: each type is still written, above 9.0.0. + listed := map[string]string{} + for typ := range typeCoverageExceptions { + listed[typ] = "typeCoverageExceptions" + } + for typ := range broader { + listed[typ] = "broaderTypeGates" + } + for typ, where := range listed { + since, ok := canon.TypeIntroduced(typ) + if len(written[typ]) == 0 || !ok || since.Compare(lowest) <= 0 { + t.Errorf("%s in %s is no longer written above %s: remove it", typ, where, lowest) + } + } +} diff --git a/mdl/executor/version_gates_documents.go b/mdl/executor/version_gates_documents.go new file mode 100644 index 0000000000..f75c2245d9 --- /dev/null +++ b/mdl/executor/version_gates_documents.go @@ -0,0 +1,424 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/types" +) + +// documentTypeGates are the gates of the document shapes whose storage +// $Types (or the properties that hold them) Mendix added after 9.0 and the +// writers produce with no older shape (ako/mxcli#1811). Each floor is the +// latest introduction among its types and properties in mendixmodelsdk +// 4.117.0, the version below which the write refuses the statement +// (canon.UndeclaredTypeError, canon.StripUndeclaredProperties) — pinned by +// TestTypeGates_FloorIsTheTypeIntroduction. The workflow shapes are +// workflowShapeGates. +var documentTypeGates = []statementTypeGate{ + { + // create page has the same row in its own case; an alter adds one + // to a page as surely (ako/mxcli#1121's floor, pinned here). + typeGate: typeGate{ + versionGate: versionGate{"pages", "page_parameters", "add a page parameter", + "pass data via a non-persistent entity or microflow parameter instead"}, + types: []string{"Forms$PageParameter"}, + properties: []string{"Forms$Page.Parameters"}, + }, + applies: func(stmt ast.Statement) bool { + s, ok := stmt.(*ast.AlterPageStmt) + return ok && !strings.EqualFold(s.ContainerType, "SNIPPET") && alterPageAddsParameter(s) + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"pages", "snippet_parameters", "a snippet parameter", + "this version stores no snippet parameters (Forms$SnippetParameter): pass the object through the data context, or upgrade the project"}, + types: []string{"Forms$SnippetParameter"}, + properties: []string{"Forms$Snippet.Parameters"}, + }, + applies: func(stmt ast.Statement) bool { + switch s := stmt.(type) { + case *ast.CreateSnippetStmtV3: + return len(s.Parameters) > 0 + case *ast.AlterPageStmt: + return strings.EqualFold(s.ContainerType, "SNIPPET") && alterPageAddsParameter(s) + } + return false + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"pages", "page_variables", "a page or snippet variable", + "this version stores no page variables (Forms$LocalVariable): keep the value in a non-persistent object, or upgrade the project"}, + types: []string{"Forms$LocalVariable"}, + // MDL writes every variable with its default expression, and the + // DefaultValue property is three minors later than the type. + properties: []string{"Forms$Page.Variables", "Forms$LocalVariable.DefaultValue"}, + }, + applies: func(stmt ast.Statement) bool { + switch s := stmt.(type) { + case *ast.CreatePageStmtV3: + return len(s.Variables) > 0 + case *ast.CreateSnippetStmtV3: + return len(s.Variables) > 0 + case *ast.AlterPageStmt: + for _, op := range s.Operations { + if _, ok := op.(*ast.AddVariableOp); ok { + return true + } + } + } + return false + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"pages", "complete_task_action", "action: complete task", + "this version stores no outcome on a page's complete task action (Forms$SetTaskOutcomeClientAction.OutcomeValue): set the outcome from a microflow, or upgrade the project"}, + types: []string{"Forms$SetTaskOutcomeClientAction"}, + properties: []string{"Forms$SetTaskOutcomeClientAction.OutcomeValue"}, + }, + applies: func(stmt ast.Statement) bool { return pageActionTypes(stmt)["completeTask"] }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"pages", "workflow_page_actions", "action: open user task / open workflow / call workflow", + "this version has no workflow page actions (Forms$OpenUserTaskClientAction, OpenWorkflowClientAction, CallWorkflowClientAction): upgrade the project"}, + types: []string{"Forms$OpenUserTaskClientAction", "Forms$OpenWorkflowClientAction", "Forms$CallWorkflowClientAction"}, + }, + applies: func(stmt ast.Statement) bool { + t := pageActionTypes(stmt) + return t["openUserTask"] || t["openWorkflow"] || t["callWorkflow"] + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"navigation", "menu_item_icons", "Icon: on a menu item", + "this version has no icon collections (Forms$IconCollectionIcon): leave the icon out, or upgrade the project"}, + types: []string{"Forms$IconCollectionIcon"}, + }, + applies: statesMenuIcon, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"navigation", "progressive_web_app", "progressive web app ( … )", + "this version has no progressive web app settings (Navigation$ProgressiveWebAppSettings): leave the clause out, or upgrade the project"}, + types: []string{"Navigation$ProgressiveWebAppSettings"}, + }, + applies: func(stmt ast.Statement) bool { + s, ok := stmt.(*ast.CreateNavigationStmt) + return ok && s.ProgressiveWebApp != nil && !s.ProgressiveWebApp.Off + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"navigation", "native_home_page", "home page on a native profile", + "this version stores a native profile's home page differently (Navigation$NativeHomePage), which mxcli does not write: set it in Studio Pro, or upgrade the project"}, + types: []string{"Navigation$NativeHomePage"}, + }, + applies: setsNativeHomePage, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"navigation", "not_found_page", "not found page", + "this version has no not-found page (Navigation$NotFoundHomePage): leave the clause out, or upgrade the project"}, + types: []string{"Navigation$NotFoundHomePage"}, + properties: []string{"Navigation$NavigationProfile.NotFoundHomepage"}, + }, + applies: func(stmt ast.Statement) bool { + switch s := stmt.(type) { + case *ast.CreateNavigationStmt: + return s.NotFoundPage != nil + case *ast.AlterNavigationStmt: + for _, op := range s.Operations { + if o, ok := op.(*ast.SetNavPageClauseOp); ok && o.Clause == "notfound" { + return true + } + } + } + return false + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"modules", "jar_dependencies", "add jar dependency", + "this version stores no JAR dependencies on a module (Projects$JarDependency): put the JAR in userlib/, or upgrade the project"}, + types: []string{"Projects$JarDependency"}, + properties: []string{"Projects$ModuleSettings.JarDependencies"}, + }, + applies: func(stmt ast.Statement) bool { + return jarDepActions(stmt, func(a ast.JarDepAction) bool { _, ok := a.(*ast.AddJarDepAction); return ok }) + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"modules", "jar_dependency_exclusions", "add exclusion on a jar dependency", + "this version has no JAR dependency exclusions (Projects$JarDependencyExclusion): upgrade the project"}, + types: []string{"Projects$JarDependencyExclusion"}, + }, + applies: func(stmt ast.Statement) bool { + return jarDepActions(stmt, func(a ast.JarDepAction) bool { _, ok := a.(*ast.AddJarDepExclusionAction); return ok }) + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"scheduled_events", "schedules", "create scheduled event", + "this version stores a scheduled event's interval, not a schedule (ScheduledEvents$*Schedule), which mxcli does not write: create it in Studio Pro, or upgrade the project"}, + types: []string{"ScheduledEvents$MinuteSchedule", "ScheduledEvents$HourSchedule", "ScheduledEvents$DaySchedule", + "ScheduledEvents$WeekSchedule", "ScheduledEvents$MonthDateSchedule", "ScheduledEvents$MonthWeekdaySchedule", + "ScheduledEvents$YearDateSchedule", "ScheduledEvents$YearWeekdaySchedule"}, + properties: []string{"ScheduledEvents$ScheduledEvent.Schedule", "ScheduledEvents$ScheduledEvent.OnOverlap"}, + }, + applies: func(stmt ast.Statement) bool { + s, ok := stmt.(*ast.CreateScheduledEventStmt) + return ok && s.Repeat != "" + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"integration", "database_connector_basic", "create database connection", + "this version stores a database connection's connection input differently (DatabaseConnector$ConnectionString), which mxcli does not write: create it in Studio Pro, or upgrade the project"}, + types: []string{"DatabaseConnector$DatabaseConnection", "DatabaseConnector$ConnectionString"}, + properties: []string{"DatabaseConnector$DatabaseConnection.ConnectionInput"}, + }, + applies: func(stmt ast.Statement) bool { _, ok := stmt.(*ast.CreateDatabaseConnectionStmt); return ok }, + }, + { + // Measured on stamped projects: any query is refused below 10.12, on + // its QueryType; one returning an entity on its TableMappings too. + typeGate: typeGate{ + versionGate: versionGate{"integration", "database_connector_queries", "a query of a database connection", + "this version stores a database query differently (DatabaseConnector$DatabaseQuery.QueryType), which mxcli does not write: create the connection without queries and add them in Studio Pro, or upgrade the project"}, + types: []string{"DatabaseConnector$DatabaseQuery"}, + properties: []string{"DatabaseConnector$DatabaseQuery.QueryType", "DatabaseConnector$DatabaseQuery.TableMappings"}, + }, + applies: func(stmt ast.Statement) bool { + s, ok := stmt.(*ast.CreateDatabaseConnectionStmt) + return ok && len(s.Queries) > 0 + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"integration", "business_events", "create business event service", + "this version has no business event services (BusinessEvents$BusinessEventService): upgrade the project"}, + types: []string{"BusinessEvents$BusinessEventService", "BusinessEvents$BusinessEventDefinition", + "BusinessEvents$Channel", "BusinessEvents$Message", "BusinessEvents$MessageAttribute", "BusinessEvents$ServiceOperation"}, + }, + applies: func(stmt ast.Statement) bool { _, ok := stmt.(*ast.CreateBusinessEventServiceStmt); return ok }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"integration", "rest_client_authentication", "authentication: basic (on a consumed REST service)", + "this version has no authentication scheme on a consumed REST service (Rest$BasicAuthenticationScheme): set the header yourself, or upgrade the project"}, + types: []string{"Rest$BasicAuthenticationScheme"}, + properties: []string{"Rest$ConsumedRestService.AuthenticationScheme"}, + }, + applies: func(stmt ast.Statement) bool { + switch s := stmt.(type) { + case *ast.CreateRestClientStmt: + return s.Authentication != nil + case *ast.AlterRestClientStmt: + for _, a := range s.Actions { + if set, ok := a.(*ast.RestClientSetAction); ok && set.AuthenticationSet && set.Authentication != nil { + return true + } + } + } + return false + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"integration", "rest_client_operations", "an operation of a consumed REST service", + "this version stores a REST operation's method and response differently (Rest$RestOperationMethodWithBody / WithoutBody, Rest$NoResponseHandling), which mxcli does not write: create it in Studio Pro, or upgrade the project"}, + types: []string{"Rest$RestOperationMethodWithBody", "Rest$RestOperationMethodWithoutBody", + "Rest$NoResponseHandling", "Rest$StringBody"}, + properties: []string{"Rest$RestOperation.Method", "Rest$RestOperation.ResponseHandling"}, + }, + applies: func(stmt ast.Statement) bool { + return len(restClientOperations(stmt)) > 0 || openAPIRestClient(stmt) + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"integration", "rest_client_openapi", "create consumed rest service from an OpenAPI document", + "this version stores no OpenAPI document on a consumed REST service (Rest$OpenApiFile): declare the operations, or upgrade the project"}, + types: []string{"Rest$OpenApiFile"}, + }, + applies: openAPIRestClient, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"integration", "rest_client_query_params", "query: ( … ) on a REST operation", + "this version has no query parameters on a REST operation (Rest$QueryParameter): put them in the path, or upgrade the project"}, + types: []string{"Rest$QueryParameter", "Rest$RequiredQueryParameterUsage"}, + properties: []string{"Rest$RestOperation.QueryParameters"}, + }, + applies: func(stmt ast.Statement) bool { + for _, op := range restClientOperations(stmt) { + if len(op.QueryParameters) > 0 { + return true + } + } + return false + }, + }, +} + +// documentGates are the documentTypeGates stmt states, in table order. +func documentGates(stmt ast.Statement) []versionGate { + var gates []versionGate + for _, g := range documentTypeGates { + if g.applies(stmt) { + gates = append(gates, g.versionGate) + } + } + return gates +} + +func alterPageAddsParameter(s *ast.AlterPageStmt) bool { + for _, op := range s.Operations { + if _, ok := op.(*ast.AddParameterOp); ok { + return true + } + } + return false +} + +// pageActionTypes are the action types (ast.ActionV3.Type) the widgets a page, +// snippet or alter page statement writes carry, then-actions included. +func pageActionTypes(stmt ast.Statement) map[string]bool { + out := map[string]bool{} + visit := func(v any) { + for a, _ := v.(*ast.ActionV3); a != nil; a = a.ThenAction { + out[a.Type] = true + } + } + forEachWidget(stmt, func(w *ast.WidgetV3, _ string) { + for _, v := range w.Properties { + visit(v) + } + }) + if s, ok := stmt.(*ast.AlterPageStmt); ok { + for _, op := range s.Operations { + if o, ok := op.(*ast.SetPropertyOp); ok { + for _, v := range o.Properties { + visit(v) + } + } + } + } + return out +} + +// statesMenuIcon reports a menu item the statement writes with an icon from an +// icon collection: the only icon `Icon:` can state. +func statesMenuIcon(stmt ast.Statement) bool { + var items []ast.NavMenuItemDef + var ops []ast.MenuAlterOp + switch s := stmt.(type) { + case *ast.CreateMenuStmt: + items = s.Items + case *ast.CreateNavigationStmt: + items = s.MenuItems + case *ast.AlterMenuStmt: + ops = s.Operations + case *ast.AlterNavigationStmt: + ops = s.Operations + } + for _, op := range ops { + switch o := op.(type) { + case *ast.InsertMenuItemsOp: + items = append(items, o.Items...) + case *ast.ReplaceMenuItemOp: + items = append(items, o.Items...) + case *ast.SetMenuItemOp: + if o.SetIcon { + items = append(items, o.Item) + } + } + } + var walk func([]ast.NavMenuItemDef) bool + walk = func(items []ast.NavMenuItemDef) bool { + for _, it := range items { + if it.IconKind == types.MenuIconCollection || walk(it.Items) { + return true + } + } + return false + } + return walk(items) +} + +// setsNativeHomePage reports a default home page set on the native profile. +// The writer picks the native shape from the stored profile's type; the native +// profile is the one Studio Pro names NativePhone, which is how a statement +// addresses it. +func setsNativeHomePage(stmt ast.Statement) bool { + native := func(profile string) bool { return strings.HasPrefix(strings.ToLower(profile), "native") } + switch s := stmt.(type) { + case *ast.CreateNavigationStmt: + if !native(s.ProfileName) { + return false + } + for _, hp := range s.HomePages { + if hp.ForRole == nil { + return true + } + } + case *ast.AlterNavigationStmt: + if !native(s.ProfileName) { + return false + } + for _, op := range s.Operations { + if o, ok := op.(*ast.SetNavPageClauseOp); ok && o.Clause == "home" && o.Home.ForRole == nil { + return true + } + } + } + return false +} + +func jarDepActions(stmt ast.Statement, match func(ast.JarDepAction) bool) bool { + s, ok := stmt.(*ast.AlterModuleJarDepStmt) + if !ok { + return false + } + for _, a := range s.Actions { + if match(a) { + return true + } + } + return false +} + +// restClientOperations are the operations a create or alter of a consumed REST +// service declares (added or rewritten). +func restClientOperations(stmt ast.Statement) []*ast.RestOperationDef { + switch s := stmt.(type) { + case *ast.CreateRestClientStmt: + return s.Operations + case *ast.AlterRestClientStmt: + var ops []*ast.RestOperationDef + for _, a := range s.Actions { + switch x := a.(type) { + case *ast.RestClientAddOperationAction: + ops = append(ops, x.Operation) + case *ast.RestClientModifyOperationAction: + ops = append(ops, x.Operation) + } + } + return ops + } + return nil +} + +// openAPIRestClient reports a consumed REST service created from an OpenAPI +// document: its operations come from the document, and the document is stored. +func openAPIRestClient(stmt ast.Statement) bool { + s, ok := stmt.(*ast.CreateRestClientStmt) + return ok && s.OpenApiPath != "" +} diff --git a/mdl/executor/version_gates_documents_1811_test.go b/mdl/executor/version_gates_documents_1811_test.go new file mode 100644 index 0000000000..fc31a1bc76 --- /dev/null +++ b/mdl/executor/version_gates_documents_1811_test.go @@ -0,0 +1,123 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/types" +) + +// ako/mxcli#1811 sweep: the document shapes whose storage types (or the +// properties holding them) arrived after 9.0, which the writers produce with +// no older shape and no statement gated. Each floor is the write's: the latest +// introduction among the types and properties the shape writes, measured on +// stamped projects and pinned to the type and property data by +// TestTypeGates_FloorIsTheTypeIntroduction. +func TestCheckVersionGates_DocumentShapes1811(t *testing.T) { + qn := func(module, name string) ast.QualifiedName { return ast.QualifiedName{Module: module, Name: name} } + param := []ast.PageParameter{{Name: "Order", EntityType: qn("M", "Order")}} + variable := []ast.PageVariable{{Name: "Flag", DataType: "Boolean", DefaultValue: "true"}} + button := func(action *ast.ActionV3) []*ast.WidgetV3 { + return []*ast.WidgetV3{{Type: "ACTIONBUTTON", Name: "b", Properties: map[string]any{"Action": action}}} + } + iconItem := ast.NavMenuItemDef{Caption: "Home", Icon: "Atlas_Core.Atlas_Filled.home", IconKind: types.MenuIconCollection} + getOp := &ast.RestOperationDef{Name: "Get", Method: "GET", Path: "/x"} + queryOp := &ast.RestOperationDef{Name: "Search", Method: "GET", Path: "/s", QueryParameters: []ast.RestParamDef{{Name: "q", DataType: "String"}}} + checkStatementGateCases(t, []statementGateCase{ + {"create snippet with a parameter", &ast.CreateSnippetStmtV3{Name: qn("M", "S"), Parameters: param}, + [3]int{9, 20, 0}, [3]int{9, 21, 0}, "a snippet parameter"}, + {"alter snippet add a parameter", &ast.AlterPageStmt{ContainerType: "SNIPPET", PageName: qn("M", "S"), + Operations: []ast.AlterPageOperation{&ast.AddParameterOp{Parameter: param[0]}}}, + [3]int{9, 20, 0}, [3]int{9, 21, 0}, "a snippet parameter"}, + {"alter page add a parameter", &ast.AlterPageStmt{ContainerType: "PAGE", PageName: qn("M", "P"), + Operations: []ast.AlterPageOperation{&ast.AddParameterOp{Parameter: param[0]}}}, + [3]int{9, 3, 0}, [3]int{9, 4, 0}, "add a page parameter"}, + {"create page with variables", &ast.CreatePageStmtV3{Name: qn("M", "P"), Variables: variable}, + [3]int{10, 19, 0}, [3]int{10, 20, 0}, "a page or snippet variable"}, + {"create snippet with variables", &ast.CreateSnippetStmtV3{Name: qn("M", "S"), Variables: variable}, + [3]int{10, 19, 0}, [3]int{10, 20, 0}, "a page or snippet variable"}, + {"alter page add a variable", &ast.AlterPageStmt{ContainerType: "PAGE", PageName: qn("M", "P"), + Operations: []ast.AlterPageOperation{&ast.AddVariableOp{Variable: variable[0]}}}, + [3]int{10, 16, 0}, [3]int{10, 20, 0}, "a page or snippet variable"}, + {"a complete task button", &ast.CreatePageStmtV3{Name: qn("M", "P"), Widgets: button(&ast.ActionV3{Type: "completeTask", OutcomeValue: "Approve"})}, + [3]int{9, 18, 0}, [3]int{9, 19, 0}, "action: complete task"}, + {"an open user task button", &ast.CreateSnippetStmtV3{Name: qn("M", "S"), Widgets: button(&ast.ActionV3{Type: "openUserTask"})}, + [3]int{9, 0, 1}, [3]int{9, 0, 2}, "action: open user task / open workflow / call workflow"}, + {"a call workflow button, inserted", &ast.AlterPageStmt{ContainerType: "PAGE", PageName: qn("M", "P"), + Operations: []ast.AlterPageOperation{&ast.InsertWidgetOp{Widgets: button(&ast.ActionV3{Type: "callWorkflow", Target: "M.WF"})}}}, + [3]int{9, 0, 1}, [3]int{9, 0, 2}, "action: open user task / open workflow / call workflow"}, + {"create menu with an icon", &ast.CreateMenuStmt{Name: qn("M", "Menu"), Items: []ast.NavMenuItemDef{ + {Caption: "More", Items: []ast.NavMenuItemDef{iconItem}}}}, + [3]int{9, 19, 0}, [3]int{9, 20, 0}, "Icon: on a menu item"}, + {"alter navigation set an icon", &ast.AlterNavigationStmt{ProfileName: "Responsive", Operations: []ast.MenuAlterOp{ + &ast.SetMenuItemOp{SetIcon: true, Item: iconItem}}}, + [3]int{9, 19, 0}, [3]int{9, 20, 0}, "Icon: on a menu item"}, + {"progressive web app settings", &ast.CreateNavigationStmt{ProfileName: "PhoneOffline", ProgressiveWebApp: &types.NavPWASpec{}}, + [3]int{9, 0, 2}, [3]int{9, 0, 3}, "progressive web app ( … )"}, + {"a native home page", &ast.CreateNavigationStmt{ProfileName: "NativePhone", HomePages: []ast.NavHomePageDef{{IsPage: true, Target: qn("M", "Home")}}}, + [3]int{9, 3, 0}, [3]int{9, 4, 0}, "home page on a native profile"}, + {"create navigation not found page", &ast.CreateNavigationStmt{ProfileName: "Responsive", NotFoundPage: &ast.QualifiedName{Module: "M", Name: "NotFound"}}, + [3]int{10, 12, 0}, [3]int{10, 13, 0}, "not found page"}, + {"alter navigation set not found page", &ast.AlterNavigationStmt{ProfileName: "Responsive", Operations: []ast.MenuAlterOp{ + &ast.SetNavPageClauseOp{Clause: "notfound", Page: qn("M", "NotFound")}}}, + [3]int{10, 12, 0}, [3]int{10, 13, 0}, "not found page"}, + {"add jar dependency", &ast.AlterModuleJarDepStmt{ModuleName: "M", Actions: []ast.JarDepAction{ + &ast.AddJarDepAction{Group: "g", Artifact: "a", Version: "1"}}}, + [3]int{9, 24, 30}, [3]int{10, 0, 0}, "add jar dependency"}, + {"add a jar dependency exclusion", &ast.AlterModuleJarDepStmt{ModuleName: "M", Actions: []ast.JarDepAction{ + &ast.AddJarDepExclusionAction{}}}, + [3]int{10, 11, 0}, [3]int{10, 12, 0}, "add exclusion on a jar dependency"}, + {"create scheduled event", &ast.CreateScheduledEventStmt{Name: qn("M", "E"), Microflow: "M.Job", Repeat: "Daily"}, + [3]int{9, 11, 0}, [3]int{9, 12, 0}, "create scheduled event"}, + {"create database connection", &ast.CreateDatabaseConnectionStmt{Name: qn("M", "Db"), DatabaseType: "MSSQL"}, + [3]int{9, 24, 30}, [3]int{10, 0, 0}, "create database connection"}, + {"a database connection with a query", &ast.CreateDatabaseConnectionStmt{Name: qn("M", "Db"), DatabaseType: "MSSQL", + Queries: []ast.DatabaseQueryDef{{Name: "All", SQL: "SELECT 1"}}}, + [3]int{10, 11, 0}, [3]int{10, 12, 0}, "a query of a database connection"}, + {"create business event service", &ast.CreateBusinessEventServiceStmt{Name: qn("M", "Events")}, + [3]int{9, 23, 0}, [3]int{9, 24, 0}, "create business event service"}, + {"a REST client with basic authentication", &ast.CreateRestClientStmt{Name: qn("M", "R"), Authentication: &ast.RestAuthDef{Scheme: "BASIC"}}, + [3]int{10, 1, 0}, [3]int{10, 2, 0}, "authentication: basic (on a consumed REST service)"}, + {"a REST client with an operation", &ast.CreateRestClientStmt{Name: qn("M", "R"), Operations: []*ast.RestOperationDef{getOp}}, + [3]int{10, 3, 0}, [3]int{10, 4, 0}, "an operation of a consumed REST service"}, + {"alter REST client add an operation", &ast.AlterRestClientStmt{Name: qn("M", "R"), Actions: []ast.RestClientAlterAction{ + &ast.RestClientAddOperationAction{Operation: getOp}}}, + [3]int{10, 3, 0}, [3]int{10, 4, 0}, "an operation of a consumed REST service"}, + {"a REST operation with a query parameter", &ast.CreateRestClientStmt{Name: qn("M", "R"), Operations: []*ast.RestOperationDef{queryOp}}, + [3]int{10, 24, 24}, [3]int{11, 0, 0}, "query: ( … ) on a REST operation"}, + {"a REST client from OpenAPI", &ast.CreateRestClientStmt{Name: qn("M", "R"), OpenApiPath: "spec.json"}, + [3]int{10, 20, 0}, [3]int{10, 21, 0}, "create consumed rest service from an OpenAPI document"}, + // The view entity's floor moves: its source document is 10.21. + {"create view entity", &ast.CreateViewEntityStmt{Query: ast.OQLQuery{RawQuery: "select s.Code as Code\nfrom M.Sale as s"}}, + [3]int{10, 20, 0}, [3]int{10, 21, 0}, "create view entity"}, + }) +} + +// A statement that states none of the shapes is not gated by them on 9.0.2: +// a workflow with no parameter, a page and snippet without parameters or +// variables, a menu item without an icon, a REST client with no operation. +func TestCheckVersionGates_PlainShapesAreNotGated(t *testing.T) { + qn := func(module, name string) ast.QualifiedName { return ast.QualifiedName{Module: module, Name: name} } + for _, stmt := range []ast.Statement{ + &ast.CreateWorkflowStmt{Name: qn("M", "WF"), Activities: []ast.WorkflowActivityNode{&ast.WorkflowCallMicroflowNode{Microflow: qn("M", "Act")}}}, + &ast.CreateSnippetStmtV3{Name: qn("M", "S")}, + &ast.CreatePageStmtV3{Name: qn("M", "P"), Widgets: []*ast.WidgetV3{{Type: "ACTIONBUTTON", Name: "b", Properties: map[string]any{"Action": &ast.ActionV3{Type: "close"}}}}}, + &ast.CreateMenuStmt{Name: qn("M", "Menu"), Items: []ast.NavMenuItemDef{{Caption: "Home"}}}, + &ast.CreateNavigationStmt{ProfileName: "Responsive", ProgressiveWebApp: &types.NavPWASpec{Off: true}}, + &ast.CreateNavigationStmt{ProfileName: "Responsive", HomePages: []ast.NavHomePageDef{{IsPage: true, Target: qn("M", "Home")}}}, + &ast.AlterModuleJarDepStmt{ModuleName: "M", Actions: []ast.JarDepAction{&ast.DropJarDepAction{}}}, + } { + prog := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, stmt}} + if vs := versionGateViolations(versionCtx(t, 9, 0, 2), prog); len(vs) != 0 { + t.Errorf("%T reported on 9.0.2: %+v", stmt, vs) + } + } + // A REST client is 10.1 (rest_client_basic): one with no operation, no + // authentication and no OpenAPI document is reported for that alone. + prog := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, &ast.CreateRestClientStmt{Name: qn("M", "R")}}} + if vs := versionGateViolations(versionCtx(t, 10, 1, 0), prog); len(vs) != 0 { + t.Errorf("a REST client with no operation reported on 10.1: %+v", vs) + } +} diff --git a/mdl/executor/version_gates_flow_actions.go b/mdl/executor/version_gates_flow_actions.go new file mode 100644 index 0000000000..18cd77c096 --- /dev/null +++ b/mdl/executor/version_gates_flow_actions.go @@ -0,0 +1,215 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import "github.com/mendixlabs/mxcli/mdl/ast" + +// flowActionGates are the gates of the microflow and nanoflow activities whose +// storage $Types Mendix added after 9.0 and the converters write with no older +// shape (ako/mxcli#1811), each at its types' introduction in mendixmodelsdk +// 4.117.0. The write refuses each type below it (canon.UndeclaredTypeError); +// these name the statement first, in `create microflow|nanoflow` and in what +// `alter microflow|nanoflow` splices in, wherever it sits (bodyContains). +// Newest first, so a body holding several is refused naming the one whose +// version covers the rest; the loops come last (see there). A statement a nanoflow cannot hold is refused there +// by the nanoflow validation; matching it here as well costs nothing. +var flowActionGates = []typeGate{ + { + versionGate: versionGate{"microflows", "transform_json", "transform … with (a data transformer)", + "this version has no Microflows$TransformJsonAction: map the JSON with an import mapping, or upgrade the project"}, + types: []string{"Microflows$TransformJsonAction"}, + match: func(st ast.MicroflowStatement) bool { _, ok := st.(*ast.TransformJsonStmt); return ok }, + }, + { + versionGate: versionGate{"microflows", "execute_database_query_runtime_connection", "execute database query … connection ( … )", + "this version has no connection arguments on a database query (DatabaseConnector$ConnectionParameterMapping): set the connection on the database connection, or upgrade the project"}, + types: []string{"DatabaseConnector$ConnectionParameterMapping"}, + properties: []string{"DatabaseConnector$ExecuteDatabaseQueryAction.ConnectionParameterMappings"}, + match: func(st ast.MicroflowStatement) bool { + s, ok := st.(*ast.ExecuteDatabaseQueryStmt) + return ok && len(s.ConnectionArguments) > 0 + }, + }, + { + // The body variable is written for an operation whose body is an export + // mapping, or one the builder cannot find; for a JSON, template or file + // body the body lives on the operation and the clause is dropped. Gated + // on the clause as written, which is what an older project refuses. + versionGate: versionGate{"microflows", "send_rest_request_body", "send rest request … body $Variable", + "this version has no Microflows$BodyVariable: leave the body to the operation, or upgrade the project"}, + types: []string{"Microflows$BodyVariable"}, + properties: []string{"Microflows$RestOperationCallAction.BodyVariable"}, + match: func(st ast.MicroflowStatement) bool { + s, ok := st.(*ast.SendRestRequestStmt) + return ok && s.BodyVariable != "" + }, + }, + { + versionGate: versionGate{"microflows", "send_rest_request_output", "$Result = send rest request …", + "this version has no Microflows$OutputVariable: call the operation without storing its result, or upgrade the project"}, + types: []string{"Microflows$OutputVariable"}, + properties: []string{"Microflows$RestOperationCallAction.OutputVariable"}, + match: func(st ast.MicroflowStatement) bool { + s, ok := st.(*ast.SendRestRequestStmt) + return ok && s.OutputVariable != "" + }, + }, + { + // A parameter is a path parameter mapping below 11.0: an operation + // there has no query parameters (Rest$QueryParameter is 11.0), and + // the query parameter mapping, also 11.0, is written only for one. + versionGate: versionGate{"microflows", "send_rest_request_parameters", "send rest request … with ( … )", + "this version has no Microflows$RestOperationParameterMapping: upgrade the project"}, + types: []string{"Microflows$RestOperationParameterMapping"}, + properties: []string{"Microflows$RestOperationCallAction.ParameterMappings"}, + match: func(st ast.MicroflowStatement) bool { + s, ok := st.(*ast.SendRestRequestStmt) + return ok && len(s.Parameters) > 0 + }, + }, + { + // mx 9.24.30 loads the action (CE7053 "No REST operation specified", + // not a load failure), but the consumed REST service it calls, + // Rest$ConsumedRestService, is 10.1 too: a 9.24 author cannot set one up. + versionGate: versionGate{"microflows", "send_rest_request", "send rest request", + "this version has no consumed REST service (Rest$ConsumedRestService) or operation call " + + "(Microflows$RestOperationCallAction): use `call rest`, or upgrade the project"}, + types: []string{"Microflows$RestOperationCallAction"}, + match: func(st ast.MicroflowStatement) bool { _, ok := st.(*ast.SendRestRequestStmt); return ok }, + }, + { + versionGate: versionGate{"microflows", "call_external_action_arguments", "call external action … ( arguments )", + "this version has no Microflows$ExternalActionParameterMapping: upgrade the project"}, + types: []string{"Microflows$ExternalActionParameterMapping"}, + properties: []string{"Microflows$CallExternalAction.ParameterMappings"}, + match: func(st ast.MicroflowStatement) bool { + s, ok := st.(*ast.CallExternalActionStmt) + return ok && len(s.Arguments) > 0 + }, + }, + { + versionGate: versionGate{"microflows", "call_external_action", "call external action", + "this version has no Microflows$CallExternalAction: upgrade the project"}, + types: []string{"Microflows$CallExternalAction"}, + match: func(st ast.MicroflowStatement) bool { _, ok := st.(*ast.CallExternalActionStmt); return ok }, + }, + { + versionGate: versionGate{"microflows", "list_range", "range (a list operation)", + "this version has no Microflows$ListRange: retrieve the range from the database with limit and offset, or upgrade the project"}, + types: []string{"Microflows$ListRange"}, + match: func(st ast.MicroflowStatement) bool { + s, ok := st.(*ast.ListOperationStmt) + return ok && s.Operation == ast.ListOpRange + }, + }, + { + versionGate: versionGate{"microflows", "execute_database_query", "execute database query", + "this version has no database query action (DatabaseConnector$ExecuteDatabaseQueryAction): upgrade the project"}, + types: []string{"DatabaseConnector$ExecuteDatabaseQueryAction", "DatabaseConnector$QueryParameterMapping"}, + match: func(st ast.MicroflowStatement) bool { _, ok := st.(*ast.ExecuteDatabaseQueryStmt); return ok }, + }, + { + versionGate: versionGate{"microflows", "queued_calls", "call … in queue", + "this version has no task queues on a call (Queues$QueueSettings): leave out `in queue`, or upgrade the project"}, + types: []string{"Queues$QueueSettings"}, + properties: []string{"Microflows$MicroflowCall.QueueSettings", "Microflows$JavaActionCallAction.QueueSettings"}, + match: func(st ast.MicroflowStatement) bool { + switch s := st.(type) { + case *ast.CallMicroflowStmt: + return s.Queue != nil + case *ast.CallJavaActionStmt: + return s.Queue != nil + } + return false + }, + }, + { + versionGate: versionGate{"microflows", "show_page_with_params", "show page … ( arguments )", + "this version passes no arguments to a page from a flow (Forms$PageParameterMapping): pass the object with `for $Object`, or upgrade the project"}, + types: []string{"Forms$PageParameterMapping"}, + properties: []string{"Forms$FormSettings.ParameterMappings"}, + match: func(st ast.MicroflowStatement) bool { + s, ok := st.(*ast.ShowPageStmt) + return ok && len(s.Arguments) > 0 + }, + }, + { + versionGate: versionGate{"microflows", "list_find_filter_by_expression", "find / filter by expression (a list operation)", + "this version has no Microflows$FindByExpression or FilterByExpression: find or filter by a member " + + "(`find($List, Member = value)`), or upgrade the project"}, + types: []string{"Microflows$FindByExpression", "Microflows$FilterByExpression"}, + match: writesListExpressionOperation, + }, + // Last, out of version order: a loop holds the activities it repeats, and + // one of those whose own floor is the later is the more useful refusal. + { + versionGate: versionGate{"microflows", "loop_over_list", "loop … in $List", + "this version stores a loop's list on the loop itself (before Microflows$IterableList), which mxcli does not write: upgrade the project"}, + types: []string{"Microflows$IterableList"}, + properties: []string{"Microflows$LoopedActivity.LoopSource"}, + match: func(st ast.MicroflowStatement) bool { _, ok := st.(*ast.LoopStmt); return ok }, + }, + { + versionGate: versionGate{"microflows", "while_loop", "while", + "this version has no while loop (Microflows$WhileLoopCondition): loop over a list, or upgrade the project"}, + types: []string{"Microflows$WhileLoopCondition"}, + properties: []string{"Microflows$LoopedActivity.LoopSource"}, + match: func(st ast.MicroflowStatement) bool { _, ok := st.(*ast.WhileStmt); return ok }, + }, +} + +// writesListExpressionOperation reports a find or filter the builder writes as +// Microflows$FindByExpression / FilterByExpression: one written with `where`, +// or one whose condition is not `Member = value` (listAttributeOperation). +// A `find` the builder rewrites into a String find() call writes no list +// operation (stringOverloadedListOp); flowGates excludes those it can see. +func writesListExpressionOperation(st ast.MicroflowStatement) bool { + s, ok := st.(*ast.ListOperationStmt) + if !ok || (s.Operation != ast.ListOpFind && s.Operation != ast.ListOpFilter) { + return false + } + return s.ByExpression || !ast.IsMemberEquality(s.Condition) +} + +// flowGates are the gates of the activities in a microflow or nanoflow body +// that flowActionGates names. params are the flow's parameters when the +// statement declares them (create); an alter's fragment has none. +func flowGates(params []ast.MicroflowParam, body []ast.MicroflowStatement) []versionGate { + stringVars := stringVariables(params, body) + var gates []versionGate + for _, g := range flowActionGates { + match := g.match + if g.name == "list_find_filter_by_expression" { + match = func(st ast.MicroflowStatement) bool { + s, ok := st.(*ast.ListOperationStmt) + if ok && stringOverloadedListOp(s.Operation) && stringVars[s.InputVariable] { + return false // a String find(): a change variable, no list operation + } + return writesListExpressionOperation(st) + } + } + if bodyContains(body, match) { + gates = append(gates, g.versionGate) + } + } + return gates +} + +// stringVariables are the names the builder knows as String when it reaches a +// list operation: a String parameter, or a variable declared String anywhere in +// the body — the declaredVars a `find` on a String is rewritten from. +func stringVariables(params []ast.MicroflowParam, body []ast.MicroflowStatement) map[string]bool { + out := map[string]bool{} + for _, p := range params { + if p.Type.Kind == ast.TypeString { + out[p.Name] = true + } + } + bodyContains(body, func(st ast.MicroflowStatement) bool { + if d, ok := st.(*ast.DeclareStmt); ok && d.Type.Kind == ast.TypeString { + out[d.Variable] = true + } + return false + }) + return out +} diff --git a/mdl/executor/version_gates_flow_actions_1811_test.go b/mdl/executor/version_gates_flow_actions_1811_test.go new file mode 100644 index 0000000000..6f4d525874 --- /dev/null +++ b/mdl/executor/version_gates_flow_actions_1811_test.go @@ -0,0 +1,233 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/modelsdk/version" +) + +// flowGateCase is one activity a flow body gate names: the statement, the +// last version that cannot store it, its floor, and how check names it. +type flowGateCase struct { + stmt ast.MicroflowStatement + below, floor [3]int + want string + // nanoflow: the activity is also a nanoflow's (a list operation, a loop), + // so create nanoflow and alter nanoflow are gated on it as well. + nanoflow bool +} + +// orInALoop names the statement a placement inside a loop may be reported for +// instead of the activity, when the loop's floor is the activity's own: a while +// in a loop, both 9.0.4. +var orInALoop = map[string]string{"while": "loop … in $List"} + +// flowGatePlaces are where an activity sits in a body: every one of them is +// found (bodyContains walks branches, loops and error handlers). +func flowGatePlaces(st ast.MicroflowStatement) []struct { + name string + body []ast.MicroflowStatement +} { + return []struct { + name string + body []ast.MicroflowStatement + }{ + {"top level", []ast.MicroflowStatement{st}}, + {"in a loop", []ast.MicroflowStatement{&ast.LoopStmt{LoopVariable: "It", ListVariable: "List", Body: []ast.MicroflowStatement{st}}}}, + {"in an else branch", []ast.MicroflowStatement{&ast.IfStmt{HasElse: true, ElseBody: []ast.MicroflowStatement{st}}}}, + {"in a while loop", []ast.MicroflowStatement{&ast.WhileStmt{Body: []ast.MicroflowStatement{st}}}}, + {"in an error handler", []ast.MicroflowStatement{&ast.RetrieveStmt{Variable: "X", Source: ast.QualifiedName{Module: "M", Name: "E"}, + ErrorHandling: &ast.ErrorHandlingClause{Type: ast.ErrorHandlingCustom, Body: []ast.MicroflowStatement{st}}}}}, + } +} + +// flowGateStatements are the statements that write a body: create, an alter's +// insert, and an alter's replace beside a drop — for a nanoflow too when the +// activity is one. +func flowGateStatements(body []ast.MicroflowStatement, nanoflow bool) []ast.Statement { + name := ast.QualifiedName{Module: "M", Name: "Flow"} + stmts := []ast.Statement{ + &ast.CreateMicroflowStmt{Name: name, Body: body}, + &ast.AlterFlowStmt{Name: name, Operations: []*ast.AlterFlowOperation{{Op: ast.AlterFlowInsertAfter, Target: "$X", Body: body}}}, + &ast.AlterFlowStmt{Name: name, Operations: []*ast.AlterFlowOperation{ + {Op: ast.AlterFlowDrop, Target: "$Y"}, {Op: ast.AlterFlowReplace, Target: "$X", Body: body}}}, + } + if nanoflow { + stmts = append(stmts, + &ast.CreateNanoflowStmt{Name: name, Body: body}, + &ast.AlterFlowStmt{Name: name, Nanoflow: true, Operations: []*ast.AlterFlowOperation{{Op: ast.AlterFlowInsertBefore, Target: "$X", Body: body}}}) + } + return stmts +} + +// checkFlowGateCases asserts, for every place and statement, that check -p +// reports the activity below its floor with MDL-VERSION01 naming it and the +// version, that exec's handler refuses with the same text, and that the +// activity is not reported at its floor. (At its floor something else may +// be: a loop is itself gated below 9.0.4.) +func checkFlowGateCases(t *testing.T, cases []flowGateCase) { + t.Helper() + for _, c := range cases { + floor := version.Version{Major: c.floor[0], Minor: c.floor[1], Patch: c.floor[2]} + wantVersion := "requires Mendix " + floor.String() + "+" + for _, place := range flowGatePlaces(c.stmt) { + for _, stmt := range flowGateStatements(place.body, c.nanoflow) { + prog := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, stmt}} + vs := versionGateViolations(versionCtx(t, c.below[0], c.below[1], c.below[2]), prog) + named := len(vs) == 1 && (strings.Contains(vs[0].Message, c.want+" requires") || + orInALoop[c.want] != "" && place.name == "in a loop" && strings.Contains(vs[0].Message, orInALoop[c.want]+" requires")) + if !named || vs[0].RuleID != "MDL-VERSION01" || !strings.Contains(vs[0].Message, wantVersion) || vs[0].Suggestion == "" { + t.Errorf("%s %s (%T) below its floor %v: want one MDL-VERSION01 naming %q and %q with a hint, got %+v", + c.want, place.name, stmt, c.below, c.want, wantVersion, vs) + } + if err := checkVersionGates(versionCtx(t, c.below[0], c.below[1], c.below[2]), stmt); err == nil || + len(vs) == 0 || !strings.Contains(vs[0].Message, strings.SplitN(err.Error(), "\n", 2)[0]) { + t.Errorf("%s %s (%T): exec's refusal %v differs from check's %+v", c.want, place.name, stmt, err, vs) + } + for _, v := range versionGateViolations(versionCtx(t, c.floor[0], c.floor[1], c.floor[2]), prog) { + if strings.Contains(v.Message, c.want+" requires") { + t.Errorf("%s %s (%T): reported at its floor %v: %+v", c.want, place.name, stmt, c.floor, v) + } + } + } + } + } +} + +// ako/mxcli#1811: the microflow activities whose storage types Mendix added +// after 9.0 had no check-time gate, so `check -p` passed them and the write +// refused them part-way through exec (canon.UndeclaredTypeError). Each floor is +// its type's introduction in mendixmodelsdk 4.117.0: +// +// find / filter by expression 9.0.3 +// call external action 9.19.0 +// range 9.24.0 +// send rest request 10.1.0 +// … with ( … ), call external action ( arguments ) 10.2.0 +// $Result = … 10.4.0 +// … body $Variable 10.5.0 +// transform … with 11.4.0 +func TestCheckVersionGates_FlowActions1811(t *testing.T) { + cond := &ast.BinaryExpr{Left: &ast.AttributePathExpr{Variable: "currentObject", Path: []string{"Amount"}}, + Operator: ">", Right: &ast.LiteralExpr{Value: int64(0), Kind: ast.LiteralInteger}} + op := func(kind ast.ListOperationType, byExpression bool) *ast.ListOperationStmt { + return &ast.ListOperationStmt{OutputVariable: "Out", Operation: kind, InputVariable: "List", Condition: cond, ByExpression: byExpression} + } + rest := ast.QualifiedName{Module: "M", Name: "Service.Operation"} + checkFlowGateCases(t, []flowGateCase{ + {op(ast.ListOpFind, true), [3]int{9, 0, 2}, [3]int{9, 0, 3}, "find / filter by expression (a list operation)", true}, + {op(ast.ListOpFilter, true), [3]int{9, 0, 2}, [3]int{9, 0, 3}, "find / filter by expression (a list operation)", true}, + // The function form whose condition is not `Member = value`. + {op(ast.ListOpFilter, false), [3]int{9, 0, 2}, [3]int{9, 0, 3}, "find / filter by expression (a list operation)", true}, + {&ast.CallExternalActionStmt{ServiceName: ast.QualifiedName{Module: "M", Name: "Svc"}, ActionName: "Reset"}, + [3]int{9, 18, 0}, [3]int{9, 19, 0}, "call external action", false}, + {&ast.ListOperationStmt{OutputVariable: "Page", Operation: ast.ListOpRange, InputVariable: "List", + LimitExpr: &ast.LiteralExpr{Value: int64(10), Kind: ast.LiteralInteger}}, + [3]int{9, 23, 0}, [3]int{9, 24, 0}, "range (a list operation)", true}, + {&ast.SendRestRequestStmt{Operation: rest}, [3]int{9, 24, 30}, [3]int{10, 1, 0}, "send rest request", false}, + {&ast.SendRestRequestStmt{Operation: rest, Parameters: []ast.SendRestParamDef{{Name: "Id", Expression: "1"}}}, + [3]int{10, 1, 0}, [3]int{10, 2, 0}, "send rest request … with ( … )", false}, + {&ast.CallExternalActionStmt{ServiceName: ast.QualifiedName{Module: "M", Name: "Svc"}, ActionName: "Get", + Arguments: []ast.CallArgument{{Name: "Id", Value: &ast.LiteralExpr{Value: int64(1), Kind: ast.LiteralInteger}}}}, + [3]int{10, 1, 0}, [3]int{10, 2, 0}, "call external action … ( arguments )", false}, + {&ast.SendRestRequestStmt{Operation: rest, OutputVariable: "Response"}, [3]int{10, 3, 0}, [3]int{10, 4, 0}, + "$Result = send rest request …", false}, + {&ast.SendRestRequestStmt{Operation: rest, BodyVariable: "Order"}, [3]int{10, 4, 0}, [3]int{10, 5, 0}, + "send rest request … body $Variable", false}, + {&ast.TransformJsonStmt{OutputVariable: "Out", InputVariable: "Json", Transformation: ast.QualifiedName{Module: "M", Name: "T"}}, + [3]int{11, 3, 0}, [3]int{11, 4, 0}, "transform … with (a data transformer)", false}, + }) +} + +// A find or filter BY a member writes Microflows$Find / Filter, which every +// version has, and a `find` on a String is a String find() call, not a list +// operation: neither is gated, on 9.0.0. A `find` on a String the statement +// cannot see is (an alter's fragment over a parameter of the stored flow): the +// gate cannot tell it from a list. +func TestCheckVersionGates_FindFilterByMemberIsNotGated(t *testing.T) { + member := &ast.BinaryExpr{Left: &ast.IdentifierExpr{Name: "Status"}, Operator: "=", + Right: &ast.LiteralExpr{Value: "Open", Kind: ast.LiteralString}} + text := &ast.LiteralExpr{Value: "needle", Kind: ast.LiteralString} + stringFind := &ast.ListOperationStmt{OutputVariable: "Pos", Operation: ast.ListOpFind, InputVariable: "Text", Condition: text} + for _, c := range []struct { + name string + stmt ast.Statement + }{ + {"find by member", &ast.CreateMicroflowStmt{Body: []ast.MicroflowStatement{ + &ast.ListOperationStmt{OutputVariable: "Out", Operation: ast.ListOpFind, InputVariable: "List", Condition: member}}}}, + {"filter by member", &ast.CreateNanoflowStmt{Body: []ast.MicroflowStatement{ + &ast.ListOperationStmt{OutputVariable: "Out", Operation: ast.ListOpFilter, InputVariable: "List", Condition: member}}}}, + {"find on a String parameter", &ast.CreateMicroflowStmt{ + Parameters: []ast.MicroflowParam{{Name: "Text", Type: ast.DataType{Kind: ast.TypeString}}}, + Body: []ast.MicroflowStatement{stringFind}}}, + {"find on a declared String", &ast.AlterFlowStmt{Operations: []*ast.AlterFlowOperation{{Op: ast.AlterFlowInsertAfter, Target: "$X", + Body: []ast.MicroflowStatement{&ast.DeclareStmt{Variable: "Text", Type: ast.DataType{Kind: ast.TypeString}}, stringFind}}}}}, + } { + prog := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, c.stmt}} + if vs := versionGateViolations(versionCtx(t, 9, 0, 0), prog); len(vs) != 0 { + t.Errorf("%s reported on 9.0.0: %+v", c.name, vs) + } + } + // Control: the same find over a list is the by-expression operation. + prog := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, + &ast.CreateMicroflowStmt{Body: []ast.MicroflowStatement{stringFind}}}} + if vs := versionGateViolations(versionCtx(t, 9, 0, 0), prog); len(vs) != 1 { + t.Errorf("a find over an unknown variable whose condition is not Member = value: want one MDL-VERSION01 on 9.0.0, got %+v", vs) + } +} + +// exec without the pre-flight (the REPL, --no-check) refuses an alter that +// inserts one of them before reading the stored flow, as it does #1787's. +func TestExecAlterFlow_RefusesAGatedFlowActionBeforeReading(t *testing.T) { + stmt := &ast.AlterFlowStmt{Name: ast.QualifiedName{Module: "M", Name: "MF"}, Operations: []*ast.AlterFlowOperation{ + {Op: ast.AlterFlowInsertAfter, Target: "$X", Body: []ast.MicroflowStatement{ + &ast.SendRestRequestStmt{Operation: ast.QualifiedName{Module: "M", Name: "Service.Operation"}}}}}} + err := execAlterFlow(versionCtx(t, 9, 24, 30), stmt) + if err == nil || !strings.Contains(err.Error(), "send rest request requires Mendix 10.1.0+ (project is") { + t.Fatalf("alter microflow inserting send rest request on 9.24.30: %v", err) + } +} + +// ako/mxcli#1811 sweep: the other flow activities a converter writes in a +// type no gate named, found by listing every $Type the converters construct +// against the type data. Each floor is the write's, measured on stamped +// projects: +// +// loop … in $List, while 9.0.4 (Microflows$IterableList / WhileLoopCondition) +// show page … ( arguments ) 9.7.0 (Forms$PageParameterMapping) +// call microflow | java action … in queue 9.10.0 (Queues$QueueSettings) +// execute database query 9.24.0 (DatabaseConnector$ExecuteDatabaseQueryAction) +// … connection ( … ) 11.1.0 (DatabaseConnector$ConnectionParameterMapping) +func TestCheckVersionGates_FlowSweep1811(t *testing.T) { + queue := &ast.QualifiedName{Module: "M", Name: "Queue"} + arg := []ast.CallArgument{{Name: "Url", Value: &ast.LiteralExpr{Value: "jdbc:x", Kind: ast.LiteralString}}} + checkFlowGateCases(t, []flowGateCase{ + {&ast.LoopStmt{LoopVariable: "It", ListVariable: "List"}, [3]int{9, 0, 3}, [3]int{9, 0, 4}, "loop … in $List", true}, + {&ast.WhileStmt{Condition: &ast.LiteralExpr{Value: false, Kind: ast.LiteralBoolean}}, [3]int{9, 0, 3}, [3]int{9, 0, 4}, "while", true}, + {&ast.ShowPageStmt{PageName: ast.QualifiedName{Module: "M", Name: "P"}, Arguments: []ast.ShowPageArg{{ParamName: "Order", Value: &ast.VariableExpr{Name: "Order"}}}}, + [3]int{9, 6, 0}, [3]int{9, 7, 0}, "show page … ( arguments )", true}, + {&ast.CallMicroflowStmt{MicroflowName: ast.QualifiedName{Module: "M", Name: "Job"}, Queue: queue}, + [3]int{9, 9, 0}, [3]int{9, 10, 0}, "call … in queue", false}, + {&ast.CallJavaActionStmt{ActionName: ast.QualifiedName{Module: "M", Name: "JA"}, Queue: queue}, + [3]int{9, 9, 0}, [3]int{9, 10, 0}, "call … in queue", false}, + {&ast.ExecuteDatabaseQueryStmt{OutputVariable: "Rows", QueryName: "M.Db.Q"}, [3]int{9, 23, 0}, [3]int{9, 24, 0}, "execute database query", false}, + {&ast.ExecuteDatabaseQueryStmt{OutputVariable: "Rows", QueryName: "M.Db.Q", ConnectionArguments: arg}, + [3]int{11, 0, 0}, [3]int{11, 1, 0}, "execute database query … connection ( … )", false}, + }) + // A show page with no arguments, or passing the object `for $Object`, and a + // call outside a queue write none of these types: not gated on 9.0.4. + for _, st := range []ast.MicroflowStatement{ + &ast.ShowPageStmt{PageName: ast.QualifiedName{Module: "M", Name: "P"}, ForObject: "Order"}, + &ast.CallMicroflowStmt{MicroflowName: ast.QualifiedName{Module: "M", Name: "Job"}}, + } { + prog := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, + &ast.CreateMicroflowStmt{Body: []ast.MicroflowStatement{st}}}} + if vs := versionGateViolations(versionCtx(t, 9, 0, 4), prog); len(vs) != 0 { + t.Errorf("%T %+v reported on 9.0.4: %+v", st, st, vs) + } + } +} diff --git a/mdl/executor/version_gates_test.go b/mdl/executor/version_gates_test.go index 7b320b2e05..7d780e4bbe 100644 --- a/mdl/executor/version_gates_test.go +++ b/mdl/executor/version_gates_test.go @@ -159,7 +159,7 @@ func TestCheckFeatureIsOnlyCalledFromTheGateTable(t *testing.T) { // The comment gate fires on a COMMENT, not on the characters: a query with // none, or with `--` and `/*` inside a string literal, passes on 10.24.25 — the -// view-entity floor is 10.18, so nothing else gates these. +// view-entity floor is 10.21, so nothing else gates these. func TestCheckVersionGates_ViewQueryWithoutCommentPasses(t *testing.T) { prog := &ast.Program{Statements: []ast.Statement{ &ast.CreateModuleStmt{Name: "M"}, diff --git a/mdl/executor/version_gates_workflow_actions_1787_test.go b/mdl/executor/version_gates_workflow_actions_1787_test.go new file mode 100644 index 0000000000..72e7e38dad --- /dev/null +++ b/mdl/executor/version_gates_workflow_actions_1787_test.go @@ -0,0 +1,273 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/modelsdk/canon" + "github.com/mendixlabs/mxcli/modelsdk/version" + "github.com/mendixlabs/mxcli/sdk/versions" +) + +// ako/mxcli#1787: the workflow microflow actions Mendix added after 9.0 were +// refused only by the write (canon.UndeclaredTypeError), part-way through a +// script that `mxcli check -p` had passed. Each is now refused by check -p and +// by exec's pre-flight with MDL-VERSION01, naming the statement and the +// version, wherever it sits in the body, in `create microflow` and in what +// `alter microflow` inserts. Each floor is its type's introduction in +// mendixmodelsdk 4.117.0: +// +// call workflow, set task outcome, open user task, open workflow 9.0.2 +// workflow operation abort / restart 9.2.0 +// workflow operation pause / continue 9.3.0 +// get workflow data 9.10.0 +// workflow operation retry / unpause 9.12.0 +// notify workflow 10.0.0 +// get workflows, get workflow activity records 10.4.0 +func TestCheckVersionGates_WorkflowMicroflowActions(t *testing.T) { + op := func(kind string) ast.MicroflowStatement { + return &ast.WorkflowOperationStmt{OperationType: kind, WorkflowVariable: "Wf"} + } + for _, c := range []struct { + stmt ast.MicroflowStatement + below, floor [3]int + want string + }{ + {&ast.CallWorkflowStmt{Workflow: ast.QualifiedName{Module: "M", Name: "WF"}, ContextVariable: "Ctx"}, [3]int{9, 0, 1}, [3]int{9, 0, 2}, "call workflow"}, + {&ast.SetTaskOutcomeStmt{WorkflowTaskVariable: "Task", OutcomeValue: "Approve"}, [3]int{9, 0, 1}, [3]int{9, 0, 2}, "set task outcome"}, + {&ast.OpenUserTaskStmt{UserTaskVariable: "Task"}, [3]int{9, 0, 1}, [3]int{9, 0, 2}, "open user task"}, + {&ast.OpenWorkflowStmt{WorkflowVariable: "Wf"}, [3]int{9, 0, 1}, [3]int{9, 0, 2}, "open workflow"}, + {op("abort"), [3]int{9, 1, 0}, [3]int{9, 2, 0}, "workflow operation abort"}, + {op("restart"), [3]int{9, 1, 0}, [3]int{9, 2, 0}, "workflow operation restart"}, + {op("pause"), [3]int{9, 2, 0}, [3]int{9, 3, 0}, "workflow operation pause"}, + {op("continue"), [3]int{9, 2, 0}, [3]int{9, 3, 0}, "workflow operation continue"}, + {&ast.GetWorkflowDataStmt{OutputVariable: "Data", WorkflowVariable: "Wf", Workflow: ast.QualifiedName{Module: "M", Name: "WF"}}, [3]int{9, 9, 0}, [3]int{9, 10, 0}, "get workflow data"}, + {op("retry"), [3]int{9, 11, 0}, [3]int{9, 12, 0}, "workflow operation retry"}, + {op("unpause"), [3]int{9, 11, 0}, [3]int{9, 12, 0}, "workflow operation unpause"}, + {&ast.NotifyWorkflowStmt{WorkflowVariable: "Wf", Target: "M.WF.Wait"}, [3]int{9, 24, 30}, [3]int{10, 0, 0}, "notify workflow"}, + {&ast.GetWorkflowsStmt{OutputVariable: "Wfs", WorkflowContextVariableName: "Ctx"}, [3]int{10, 3, 0}, [3]int{10, 4, 0}, "get workflows"}, + {&ast.GetWorkflowActivityRecordsStmt{OutputVariable: "Records", WorkflowVariable: "Wf"}, [3]int{10, 3, 0}, [3]int{10, 4, 0}, "get workflow activity records"}, + } { + floor := version.Version{Major: c.floor[0], Minor: c.floor[1], Patch: c.floor[2]} + wantVersion := "requires Mendix " + floor.String() + "+" + for _, place := range []struct { + name string + body []ast.MicroflowStatement + }{ + {"top level", []ast.MicroflowStatement{c.stmt}}, + {"in a loop", []ast.MicroflowStatement{&ast.LoopStmt{LoopVariable: "It", ListVariable: "List", Body: []ast.MicroflowStatement{c.stmt}}}}, + {"in an else branch", []ast.MicroflowStatement{&ast.IfStmt{HasElse: true, ElseBody: []ast.MicroflowStatement{c.stmt}}}}, + {"in a while loop", []ast.MicroflowStatement{&ast.WhileStmt{Body: []ast.MicroflowStatement{c.stmt}}}}, + {"in an error handler", []ast.MicroflowStatement{&ast.RetrieveStmt{Variable: "X", Source: ast.QualifiedName{Module: "M", Name: "E"}, + ErrorHandling: &ast.ErrorHandlingClause{Type: ast.ErrorHandlingCustom, Body: []ast.MicroflowStatement{c.stmt}}}}}, + } { + for _, stmt := range []ast.Statement{ + &ast.CreateMicroflowStmt{Name: ast.QualifiedName{Module: "M", Name: "MF"}, Body: place.body}, + &ast.AlterFlowStmt{Name: ast.QualifiedName{Module: "M", Name: "MF"}, Operations: []*ast.AlterFlowOperation{ + {Op: ast.AlterFlowInsertAfter, Target: "$X", Body: place.body}}}, + &ast.AlterFlowStmt{Name: ast.QualifiedName{Module: "M", Name: "MF"}, Operations: []*ast.AlterFlowOperation{ + {Op: ast.AlterFlowDrop, Target: "$Y"}, + {Op: ast.AlterFlowReplace, Target: "$X", Body: place.body}}}, + } { + prog := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, stmt}} + vs := versionGateViolations(versionCtx(t, c.below[0], c.below[1], c.below[2]), prog) + if len(vs) != 1 || vs[0].RuleID != "MDL-VERSION01" || !strings.Contains(vs[0].Message, c.want+" requires") || + !strings.Contains(vs[0].Message, wantVersion) || vs[0].Suggestion == "" { + t.Errorf("%s %s (%T) below its floor %v: want one MDL-VERSION01 naming %q and %q with a hint, got %+v", + c.want, place.name, stmt, c.below, c.want, wantVersion, vs) + } + // exec's handler refuses with the same text. + if err := checkVersionGates(versionCtx(t, c.below[0], c.below[1], c.below[2]), stmt); err == nil || + len(vs) == 0 || !strings.Contains(vs[0].Message, strings.SplitN(err.Error(), "\n", 2)[0]) { + t.Errorf("%s %s (%T): exec's refusal %v differs from check's %+v", c.want, place.name, stmt, err, vs) + } + // At its floor the action is not reported. A loop placement + // below 9.0.4 is — for the loop, gated itself since + // ako/mxcli#1811 — and nothing else. + for _, v := range versionGateViolations(versionCtx(t, c.floor[0], c.floor[1], c.floor[2]), prog) { + loop := strings.Contains(place.name, "loop") && c.floor[0] == 9 && c.floor[1] == 0 && c.floor[2] < 4 + if !loop || !strings.Contains(v.Message, "loop … in $List requires") && !strings.Contains(v.Message, "while requires") { + t.Errorf("%s %s (%T): reported at its floor %v: %+v", c.want, place.name, stmt, c.floor, v) + } + } + } + } + } +} + +// An `alter microflow` is gated on what it inserts, and on nothing else: a +// drop, or a fragment with no gated action, passes on 9.0.0 — and an alter +// nanoflow that inserts `synchronize unsynchronized` is gated as create +// nanoflow is. +func TestCheckVersionGates_AlterFlowGatesWhatItInserts(t *testing.T) { + plain := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, + &ast.AlterFlowStmt{Operations: []*ast.AlterFlowOperation{ + {Op: ast.AlterFlowDrop, Target: "$X"}, + {Op: ast.AlterFlowInsertBefore, Target: "$Y", Body: []ast.MicroflowStatement{&ast.LogStmt{}}}, + }}}} + if vs := versionGateViolations(versionCtx(t, 9, 0, 0), plain); len(vs) != 0 { + t.Errorf("an alter microflow inserting no gated action reported on 9.0.0: %+v", vs) + } + unsync := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, + &ast.AlterFlowStmt{Nanoflow: true, Operations: []*ast.AlterFlowOperation{ + {Op: ast.AlterFlowReplaceBody, Target: "loop $It in $List", Body: []ast.MicroflowStatement{ + &ast.SynchronizeStmt{SyncType: "Unsynchronized"}}}}}}} + if vs := versionGateViolations(versionCtx(t, 9, 3, 0), unsync); len(vs) != 1 || !strings.Contains(vs[0].Message, "synchronize unsynchronized") { + t.Errorf("alter nanoflow inserting synchronize unsynchronized on 9.3: %+v", vs) + } + if vs := versionGateViolations(versionCtx(t, 9, 4, 0), unsync); len(vs) != 0 { + t.Errorf("alter nanoflow inserting synchronize unsynchronized on 9.4: %+v", vs) + } + // The other microflow body gates reach an alter too: send email (11.13) + // and a lock naming its workflow (10.0). + for _, c := range []struct { + stmt ast.MicroflowStatement + below [3]int + want string + }{ + {&ast.SendEmailStmt{}, [3]int{11, 12, 0}, "send email"}, + {&ast.LockWorkflowStmt{Workflow: "M.WF"}, [3]int{9, 24, 30}, "lock workflow / unlock workflow on a workflow"}, + } { + prog := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, + &ast.AlterFlowStmt{Operations: []*ast.AlterFlowOperation{{Op: ast.AlterFlowInsertAfter, Target: "$X", + Body: []ast.MicroflowStatement{c.stmt}}}}}} + if vs := versionGateViolations(versionCtx(t, c.below[0], c.below[1], c.below[2]), prog); len(vs) != 1 || !strings.Contains(vs[0].Message, c.want) { + t.Errorf("alter microflow inserting %T on %v: %+v", c.stmt, c.below, vs) + } + } +} + +// exec reaches `alter microflow` without the pre-flight too (the REPL, +// --no-check): its handler refuses the gated insert before it reads, let +// alone writes, the stored flow — the mock backend's microflow calls are not +// configured, so reaching one would fail with another error. +func TestExecAlterFlow_RefusesAGatedInsertBeforeReading(t *testing.T) { + stmt := &ast.AlterFlowStmt{Name: ast.QualifiedName{Module: "M", Name: "MF"}, Operations: []*ast.AlterFlowOperation{ + {Op: ast.AlterFlowInsertAfter, Target: "$X", Body: []ast.MicroflowStatement{ + &ast.GetWorkflowsStmt{OutputVariable: "Wfs", WorkflowContextVariableName: "Ctx"}}}}} + err := execAlterFlow(versionCtx(t, 9, 24, 30), stmt) + if err == nil || !strings.Contains(err.Error(), "get workflows requires Mendix 10.4.0+ (project is") { + t.Fatalf("alter microflow inserting get workflows on 9.24.30: %v", err) + } +} + +// The floor check -p applies to a type gate is the version the write refuses +// the type below, so the two cannot disagree: each gate's sdk/versions row +// opens exactly at canon.TypeIntroduced of the types the statement writes, or +// at the version canon.PropertyDeclared first declares a property it writes a +// value to, whichever is latest (ako/mxcli#1811: a 9.0.2 type can sit in a +// 9.19 property). A row typed with another version fails here, not as a +// script check passed and exec then refused part-way (ako/mxcli#1787). The +// #1774 workflow type gates are pinned with them. +func TestTypeGates_FloorIsTheTypeIntroduction(t *testing.T) { + reg, err := versions.Load() + if err != nil { + t.Fatal(err) + } + pinned := allTypeGates() + if len(workflowActionGates) == 0 || len(flowActionGates) == 0 || len(workflowShapeGates) == 0 || len(documentTypeGates) == 0 { + t.Fatal("no workflow action, flow action, workflow shape or document gates") + } + for _, g := range pinned { + if len(g.types) == 0 { + t.Errorf("%s: a type gate names no type", g.statement) + continue + } + var floor version.Version + for _, typ := range g.types { + v, ok := canon.TypeIntroduced(typ) + if !ok { + t.Errorf("%s: %s has no introduction in the Model SDK data", g.statement, typ) + continue + } + if floor.Compare(v) < 0 { + floor = v + } + } + for _, prop := range g.properties { + typ, key, _ := strings.Cut(prop, ".") + v, ok := propertyIntroduction(typ, key) + if !ok { + t.Errorf("%s: %s is declared by no version the data knows", g.statement, prop) + continue + } + if floor.Compare(v) < 0 { + floor = v + } + } + at := versions.SemVer{Major: floor.Major, Minor: floor.Minor, Patch: floor.Patch} + if !reg.IsAvailable(g.area, g.name, at) { + t.Errorf("%s: %s.%s is not available at %s, where the write accepts %v %v", g.statement, g.area, g.name, at, g.types, g.properties) + } + if below := justBelow(at); reg.IsAvailable(g.area, g.name, below) { + t.Errorf("%s: %s.%s is available at %s, where the write refuses %v %v (introduced %s)", g.statement, g.area, g.name, below, g.types, g.properties, at) + } + } +} + +// propertyIntroduction is the first version canon.PropertyDeclared declares +// typ.key in: the floor the write refuses a value of it below. +func propertyIntroduction(typ, key string) (version.Version, bool) { + if canon.PropertyDeclared(typ, key, version.Version{Major: 7}) { + return version.Version{}, false // no data: declared everywhere + } + for major := 7; major <= 12; major++ { + for minor := 0; minor <= 40; minor++ { + for patch := 0; patch <= 60; patch++ { + if v := (version.Version{Major: major, Minor: minor, Patch: patch}); canon.PropertyDeclared(typ, key, v) { + return v, true + } + } + } + } + return version.Version{}, false +} + +// justBelow is the last version before v. +func justBelow(v versions.SemVer) versions.SemVer { + switch { + case v.Patch > 0: + v.Patch-- + case v.Minor > 0: + v.Minor, v.Patch = v.Minor-1, 999 + default: + v.Major, v.Minor, v.Patch = v.Major-1, 999, 999 + } + return v +} + +// Every workflow microflow statement either has a gate or is covered by a +// broader one. Lock and unlock (Microflows$Lock/UnlockWorkflowAction, 9.21) are +// the second: every lock or unlock check and exec accept names its workflow, +// and the selection that names it is 10.0 (workflowDefinitionSelectionGate). +func TestWorkflowActionGates_CoverEveryWorkflowMicroflowStatement(t *testing.T) { + op := func(kind string) ast.MicroflowStatement { return &ast.WorkflowOperationStmt{OperationType: kind} } + for _, st := range []ast.MicroflowStatement{ + &ast.CallWorkflowStmt{}, &ast.GetWorkflowDataStmt{}, &ast.GetWorkflowsStmt{}, &ast.GetWorkflowActivityRecordsStmt{}, + op("abort"), op("continue"), op("pause"), op("restart"), op("retry"), op("unpause"), + &ast.SetTaskOutcomeStmt{}, &ast.OpenUserTaskStmt{}, &ast.NotifyWorkflowStmt{}, &ast.OpenWorkflowStmt{}, + } { + n := 0 + for _, g := range workflowActionGates { + if g.match(st) { + n++ + } + } + if n != 1 { + t.Errorf("%T %+v: matched by %d workflow action gates, want 1", st, st, n) + } + } + for _, st := range []ast.MicroflowStatement{&ast.LockWorkflowStmt{Workflow: "M.WF"}, &ast.UnlockWorkflowStmt{WorkflowVariable: "Def"}} { + if !selectsWorkflowDefinition(st) { + t.Errorf("%T naming a workflow is not gated by the 10.0 selection", st) + } + for _, g := range workflowActionGates { + if g.match(st) { + t.Errorf("%T matched by %q as well as by the selection gate", st, g.statement) + } + } + } +} diff --git a/mdl/executor/version_gates_workflow_shapes.go b/mdl/executor/version_gates_workflow_shapes.go new file mode 100644 index 0000000000..82f1dea06c --- /dev/null +++ b/mdl/executor/version_gates_workflow_shapes.go @@ -0,0 +1,194 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// statementTypeGate is a typeGate on what a whole statement states rather than +// on an activity in a flow body: applies reports a statement that writes its +// types and properties. +type statementTypeGate struct { + typeGate + applies func(ast.Statement) bool +} + +// workflowShapeGates are the gates of what a workflow statement states whose +// floor is above the workflow document's own (9.0.2, workflowBasicGate): the +// types and properties mxcli writes for them with no older shape +// (ako/mxcli#1811). Measured on stamped 9.x projects, a workflow with a +// parameter is refused below 9.18 (Workflows$Parameter.Name) and one with a +// user task below 9.19 (Workflows$UserTaskOutcome.Value), so most workflows +// meet those two first. Newest first. +var workflowShapeGates = []statementTypeGate{ + { + typeGate: typeGate{ + versionGate: versionGate{"workflows", "interrupting_notification_event_subprocess", + "event subprocess … on interrupting notification", + "this version has no interrupting notification start (Workflows$InterruptingNotificationEventSubProcessStartActivity): start it with a non-interrupting notification, or upgrade the project"}, + types: []string{"Workflows$InterruptingNotificationEventSubProcessStartActivity"}, + }, + applies: func(stmt ast.Statement) bool { + s, ok := stmt.(*ast.CreateWorkflowStmt) + if !ok { + return false + } + for _, esp := range s.EventSubProcesses { + if esp.Interrupting && !esp.Timer { + return true + } + } + return false + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"workflows", "user_task", "user task", + "this version stores a user task differently (its outcome's value, Workflows$UserTaskOutcome.Value, and task page), which mxcli does not write: create it in Studio Pro, or upgrade the project"}, + types: []string{"Workflows$NoEvent", "Workflows$MicroflowBasedEvent", "Workflows$PageReference"}, + properties: []string{"Workflows$UserTask.OnCreatedEvent", "Workflows$UserTask.TaskPage", "Workflows$UserTaskOutcome.Value"}, + }, + applies: func(stmt ast.Statement) bool { + return workflowStatesActivity(stmt, func(a ast.WorkflowActivityNode) bool { _, ok := a.(*ast.WorkflowUserTaskNode); return ok }) + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"workflows", "workflow_parameter", "parameter $WorkflowContext: …", + "this version stores a workflow's parameter differently (Workflows$Parameter and its Name), which mxcli does not write: create the workflow in Studio Pro, or upgrade the project"}, + types: []string{"Workflows$Parameter"}, + properties: []string{"Workflows$Workflow.Parameter", "Workflows$Parameter.Entity", "Workflows$Parameter.Name"}, + }, + applies: func(stmt ast.Statement) bool { + switch s := stmt.(type) { + case *ast.CreateWorkflowStmt: + return s.ParameterVar != "" || s.ParameterEntity.Name != "" + case *ast.AlterWorkflowStmt: + return alterWorkflowSets(s, "PARAMETER") + } + return false + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"workflows", "call_workflow_parameter_mappings", "call workflow … ( mappings ) (an activity)", + "this version has no parameter mappings on a call workflow activity (Workflows$WorkflowCallParameterMapping): pass the context object, or upgrade the project"}, + types: []string{"Workflows$WorkflowCallParameterMapping"}, + properties: []string{"Workflows$CallWorkflowActivity.ParameterMappings"}, + }, + applies: func(stmt ast.Statement) bool { + return workflowStatesActivity(stmt, func(a ast.WorkflowActivityNode) bool { + n, ok := a.(*ast.WorkflowCallWorkflowNode) + return ok && len(n.ParameterMappings) > 0 + }) + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"workflows", "annotations", "an annotation in a workflow", + "this version has no workflow annotations (Workflows$Annotation): keep the note as an MDL comment, or upgrade the project"}, + types: []string{"Workflows$Annotation"}, + properties: []string{"Workflows$Workflow.Annotation"}, + }, + applies: func(stmt ast.Statement) bool { + if s, ok := stmt.(*ast.CreateWorkflowStmt); ok { + if s.Annotation != "" { + return true + } + for _, esp := range s.EventSubProcesses { + if esp.Annotation != "" { + return true + } + } + } + return workflowStatesActivity(stmt, func(a ast.WorkflowActivityNode) bool { + if _, ok := a.(*ast.WorkflowAnnotationActivityNode); ok { + return true + } + an, ok := a.(ast.AnnotatedWorkflowActivity) + return ok && an.ActivityAnnotation() != "" + }) + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"workflows", "overview_page", "overview page (of a workflow)", + "this version stores a workflow's overview page differently (Workflows$Workflow.AdminPage), which mxcli does not write: set it in Studio Pro, or upgrade the project"}, + types: []string{"Workflows$PageReference"}, + properties: []string{"Workflows$Workflow.AdminPage"}, + }, + applies: func(stmt ast.Statement) bool { + switch s := stmt.(type) { + case *ast.CreateWorkflowStmt: + return s.OverviewPage.Name != "" + case *ast.AlterWorkflowStmt: + return alterWorkflowSets(s, "OVERVIEW_PAGE") + } + return false + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"workflows", "call_workflow_activity", "call workflow (an activity)", + "this version stores a call workflow activity differently (Workflows$CallWorkflowActivity.ParameterExpression), which mxcli does not write: upgrade the project"}, + types: []string{"Workflows$CallWorkflowActivity"}, + properties: []string{"Workflows$CallWorkflowActivity.ParameterExpression"}, + }, + applies: func(stmt ast.Statement) bool { + return workflowStatesActivity(stmt, func(a ast.WorkflowActivityNode) bool { _, ok := a.(*ast.WorkflowCallWorkflowNode); return ok }) + }, + }, + { + typeGate: typeGate{ + versionGate: versionGate{"workflows", "jump_to", "jump to", + "this version has no jump activity (Workflows$JumpToActivity): upgrade the project"}, + types: []string{"Workflows$JumpToActivity"}, + }, + applies: func(stmt ast.Statement) bool { + return workflowStatesActivity(stmt, func(a ast.WorkflowActivityNode) bool { _, ok := a.(*ast.WorkflowJumpToNode); return ok }) + }, + }, +} + +// workflowStatesActivity reports an activity a create or alter workflow writes +// that match accepts, nested and event sub-process ones included. +func workflowStatesActivity(stmt ast.Statement, match func(ast.WorkflowActivityNode) bool) bool { + var activities []ast.WorkflowActivityNode + switch s := stmt.(type) { + case *ast.CreateWorkflowStmt: + activities = workflowStatementActivities(s) + case *ast.AlterWorkflowStmt: + activities = alterWorkflowAddedActivities(s) + default: + return false + } + found := false + walkWorkflowActivities(activities, func(a ast.WorkflowActivityNode) { + found = found || match(a) + }) + return found +} + +// alterWorkflowSets reports an `alter workflow … set (Property: …)`. +func alterWorkflowSets(s *ast.AlterWorkflowStmt, property string) bool { + for _, op := range s.Operations { + if o, ok := op.(*ast.SetWorkflowPropertyOp); ok && strings.EqualFold(o.Property, property) { + return true + } + } + return false +} + +// workflowShapeGatesOf are the workflowShapeGates stmt states. +func workflowShapeGatesOf(stmt ast.Statement) []versionGate { + var gates []versionGate + for _, g := range workflowShapeGates { + if g.applies(stmt) { + gates = append(gates, g.versionGate) + } + } + return gates +} diff --git a/mdl/executor/version_gates_workflow_shapes_1811_test.go b/mdl/executor/version_gates_workflow_shapes_1811_test.go new file mode 100644 index 0000000000..2f5c54898e --- /dev/null +++ b/mdl/executor/version_gates_workflow_shapes_1811_test.go @@ -0,0 +1,95 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/modelsdk/version" +) + +// statementGateCase is one statement a document or workflow shape gate names. +type statementGateCase struct { + name string + stmt ast.Statement + below, floor [3]int + want string +} + +// checkStatementGateCases asserts that check -p reports the statement below +// the floor with MDL-VERSION01 naming it and the version, that exec's handler +// refuses with the same text, and that the statement is not reported for it +// at its floor. +func checkStatementGateCases(t *testing.T, cases []statementGateCase) { + t.Helper() + for _, c := range cases { + floor := version.Version{Major: c.floor[0], Minor: c.floor[1], Patch: c.floor[2]} + wantVersion := "requires Mendix " + floor.String() + "+" + prog := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, c.stmt}} + vs := versionGateViolations(versionCtx(t, c.below[0], c.below[1], c.below[2]), prog) + if len(vs) != 1 || vs[0].RuleID != "MDL-VERSION01" || !strings.Contains(vs[0].Message, c.want+" requires") || + !strings.Contains(vs[0].Message, wantVersion) || vs[0].Suggestion == "" { + t.Errorf("%s below its floor %v: want one MDL-VERSION01 naming %q and %q with a hint, got %+v", + c.name, c.below, c.want, wantVersion, vs) + } + if err := checkVersionGates(versionCtx(t, c.below[0], c.below[1], c.below[2]), c.stmt); err == nil || + len(vs) == 0 || !strings.Contains(vs[0].Message, strings.SplitN(err.Error(), "\n", 2)[0]) { + t.Errorf("%s: exec's refusal %v differs from check's %+v", c.name, err, vs) + } + for _, v := range versionGateViolations(versionCtx(t, c.floor[0], c.floor[1], c.floor[2]), prog) { + if strings.Contains(v.Message, c.want+" requires") { + t.Errorf("%s: reported at its floor %v: %+v", c.name, c.floor, v) + } + } + } +} + +// ako/mxcli#1811 sweep, the workflow shapes: a workflow's own floor is 9.0.2 +// (#1809); what it states can be later. Measured on stamped 9.x projects: a parameter is refused below +// 9.18 and a user task below 9.19. +func TestCheckVersionGates_WorkflowShapes1811(t *testing.T) { + wf := ast.QualifiedName{Module: "M", Name: "WF"} + call := &ast.WorkflowCallMicroflowNode{Microflow: ast.QualifiedName{Module: "M", Name: "Act"}} + create := func(acts ...ast.WorkflowActivityNode) *ast.CreateWorkflowStmt { + return &ast.CreateWorkflowStmt{Name: wf, Activities: acts} + } + withParam := create(call) + withParam.ParameterVar, withParam.ParameterEntity = "$WorkflowContext", ast.QualifiedName{Module: "M", Name: "Request"} + withOverview := create(call) + withOverview.OverviewPage = ast.QualifiedName{Module: "M", Name: "Overview"} + withHeaderNote := create(call) + withHeaderNote.Annotation = "note" + annotated := &ast.WorkflowCallMicroflowNode{Microflow: ast.QualifiedName{Module: "M", Name: "Act"}} + annotated.Annotation = "note" + userTask := &ast.WorkflowUserTaskNode{Name: "Review", Outcomes: []ast.WorkflowUserTaskOutcomeNode{{Caption: "Done"}}} + callWf := &ast.WorkflowCallWorkflowNode{Workflow: ast.QualifiedName{Module: "M", Name: "Sub"}} + callWfMapped := &ast.WorkflowCallWorkflowNode{Workflow: ast.QualifiedName{Module: "M", Name: "Sub"}, + ParameterMappings: []ast.WorkflowParameterMappingNode{{Parameter: "WorkflowContext", Expression: "$WorkflowContext"}}} + esp := create(call) + esp.EventSubProcesses = []ast.WorkflowEventSubProcessNode{{Name: "Cancel", Interrupting: true, Activities: []ast.WorkflowActivityNode{call}}} + insert := func(act ast.WorkflowActivityNode) *ast.AlterWorkflowStmt { + return &ast.AlterWorkflowStmt{Name: wf, Operations: []ast.AlterWorkflowOp{&ast.InsertAfterOp{NewActivities: []ast.WorkflowActivityNode{act}}}} + } + checkStatementGateCases(t, []statementGateCase{ + {"a jump", create(call, &ast.WorkflowJumpToNode{Target: "act"}), [3]int{9, 0, 4}, [3]int{9, 0, 5}, "jump to"}, + {"a call workflow activity", create(callWf), [3]int{9, 5, 0}, [3]int{9, 6, 0}, "call workflow (an activity)"}, + {"an inserted call workflow activity", insert(callWf), [3]int{9, 5, 0}, [3]int{9, 6, 0}, "call workflow (an activity)"}, + {"an overview page", withOverview, [3]int{9, 10, 0}, [3]int{9, 11, 0}, "overview page (of a workflow)"}, + {"alter workflow set the overview page", &ast.AlterWorkflowStmt{Name: wf, Operations: []ast.AlterWorkflowOp{ + &ast.SetWorkflowPropertyOp{Property: "OVERVIEW_PAGE", Entity: ast.QualifiedName{Module: "M", Name: "Overview"}}}}, + [3]int{9, 10, 0}, [3]int{9, 11, 0}, "overview page (of a workflow)"}, + {"the workflow's annotation", withHeaderNote, [3]int{9, 14, 0}, [3]int{9, 15, 0}, "an annotation in a workflow"}, + {"an activity's annotation", create(annotated), [3]int{9, 14, 0}, [3]int{9, 15, 0}, "an annotation in a workflow"}, + {"a parameter", withParam, [3]int{9, 17, 0}, [3]int{9, 18, 0}, "parameter $WorkflowContext: …"}, + {"alter workflow set the parameter", &ast.AlterWorkflowStmt{Name: wf, Operations: []ast.AlterWorkflowOp{ + &ast.SetWorkflowPropertyOp{Property: "PARAMETER", Entity: ast.QualifiedName{Module: "M", Name: "Request"}}}}, + [3]int{9, 17, 0}, [3]int{9, 18, 0}, "parameter $WorkflowContext: …"}, + {"call workflow with mappings", create(callWfMapped), [3]int{9, 17, 0}, [3]int{9, 18, 0}, "call workflow … ( mappings ) (an activity)"}, + {"a user task", create(userTask), [3]int{9, 18, 0}, [3]int{9, 19, 0}, "user task"}, + {"an inserted user task", insert(userTask), [3]int{9, 18, 0}, [3]int{9, 19, 0}, "user task"}, + {"an interrupting notification event sub-process", esp, [3]int{11, 9, 0}, [3]int{11, 10, 0}, + "event subprocess … on interrupting notification"}, + }) +} diff --git a/mdl/executor/version_gates_workflows_basic_1809_test.go b/mdl/executor/version_gates_workflows_basic_1809_test.go new file mode 100644 index 0000000000..d69d805930 --- /dev/null +++ b/mdl/executor/version_gates_workflows_basic_1809_test.go @@ -0,0 +1,57 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// ako/mxcli#1809: Workflows$Workflow is a 9.0.2 type (mendixmodelsdk 4.117.0), +// and the write refuses it below that (canon.UndeclaredTypeError, #1774). The +// `workflows basic` row said 9.0.0, and only `alter workflow` applied it, so +// on 9.0.0 or 9.0.1 `create workflow` passed `check -p` and was refused part-way +// through exec. The row is now the type's introduction, and every statement +// that writes a workflow applies it: `create workflow`, `create or modify +// workflow` and `alter workflow`. The hint names no version: the registry's +// floor is the one the message states, and the old hint's "9.12+" was not it. +func TestCheckVersionGates_WorkflowsBasic(t *testing.T) { + name := ast.QualifiedName{Module: "M", Name: "WF"} + for _, c := range []struct { + stmt ast.Statement + want string + }{ + {&ast.CreateWorkflowStmt{Name: name}, "create workflow requires Mendix 9.0.2+"}, + {&ast.CreateWorkflowStmt{Name: name, CreateOrModify: true}, "create workflow requires Mendix 9.0.2+"}, + {&ast.AlterWorkflowStmt{Name: name}, "alter workflow requires Mendix 9.0.2+"}, + } { + prog := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, c.stmt}} + for _, below := range [][3]int{{9, 0, 0}, {9, 0, 1}} { + vs := versionGateViolations(versionCtx(t, below[0], below[1], below[2]), prog) + if len(vs) != 1 || vs[0].RuleID != "MDL-VERSION01" || !strings.Contains(vs[0].Message, c.want) { + t.Errorf("%T (create or modify: %v) on %v: want one MDL-VERSION01 %q, got %+v", + c.stmt, isCreateOrModify(c.stmt), below, c.want, vs) + continue + } + if strings.Contains(vs[0].Suggestion, "9.") || vs[0].Suggestion == "" { + t.Errorf("%T: the hint must say what to do without naming a version (the message has the floor): %q", + c.stmt, vs[0].Suggestion) + } + // exec's handler refuses with the same text. + if err := checkVersionGates(versionCtx(t, below[0], below[1], below[2]), c.stmt); err == nil || + !strings.Contains(err.Error(), c.want) { + t.Errorf("%T on %v: exec's handler refusal %v, want %q", c.stmt, below, err, c.want) + } + } + if vs := versionGateViolations(versionCtx(t, 9, 0, 2), prog); len(vs) != 0 { + t.Errorf("%T reported at its floor 9.0.2: %+v", c.stmt, vs) + } + } +} + +func isCreateOrModify(stmt ast.Statement) bool { + s, ok := stmt.(*ast.CreateWorkflowStmt) + return ok && s.CreateOrModify +} diff --git a/modelsdk/canon/propertyset_issue1805_test.go b/modelsdk/canon/propertyset_issue1805_test.go new file mode 100644 index 0000000000..ece20f3b5d --- /dev/null +++ b/modelsdk/canon/propertyset_issue1805_test.go @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: Apache-2.0 + +package canon + +import ( + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +// ako/mxcli#1805: on a Mendix 11.14.0 project mxcli wrote, `fix property-sets` +// reported seven names the elements of a type disagree on, among them +// +// MIXED DomainModels$NoGeneralization.HasOwnerAttr: on 81 of 84 element(s); without it: bf8b2a01-… [DomainModels$DomainModel] +// MIXED DomainModels$DateTimeAttributeType.LocalizeDate: on 15 of 16 element(s); without it: bf8b2a01-… [DomainModels$DomainModel] +// +// and left them, because the table had no row to fill them from: the writers of +// today emit every one of them, so no measurement of what they write had seen +// one missing. The elements without them were stored by an older mxcli and +// passed through every later rewrite of their unit untouched. +// +// Each value is what `mx convert -p -s` wrote back on 10.24.24, 11.13.0 and +// 11.15.0 after the key was stripped from Studio Pro- and mxcli-authored +// elements of a fresh app (the strip-and-convert method of ako/mxcli#1505), and +// the same everywhere. +func TestCompletePropertySets_Issue1805Rows(t *testing.T) { + cases := []struct { + typ, key string + check func(any) bool + }{ + {"DomainModels$NoGeneralization", "HasOwnerAttr", eq(false)}, + {"DomainModels$NoGeneralization", "HasChangedByAttr", eq(false)}, + {"DomainModels$NoGeneralization", "HasCreatedDateAttr", eq(false)}, + {"DomainModels$NoGeneralization", "HasChangedDateAttr", eq(false)}, + {"DomainModels$DateTimeAttributeType", "LocalizeDate", eq(true)}, + {"ODataPublish$EntitySet", "ReadMode", isElement("ODataPublish$ReadSource")}, + {"ODataPublish$EntitySet", "InsertMode", isElement("ODataPublish$ChangeNotSupported")}, + {"ODataPublish$EntitySet", "UpdateMode", isElement("ODataPublish$ChangeNotSupported")}, + {"ODataPublish$EntitySet", "DeleteMode", isElement("ODataPublish$ChangeNotSupported")}, + {"ODataPublish$PublishedMicroflow", "Parameters", isEmptyList(3)}, + } + // 11.14.0 is the reporter's version: inside the measured range. + for _, ver := range []string{"10.24.24", "11.13.0", "11.14.0", "11.15.0"} { + for _, c := range cases { + t.Run(ver+"/"+c.typ+"."+c.key, func(t *testing.T) { + in := mustMarshal(t, bson.D{ + {Key: "$ID", Value: freshID()}, + {Key: "$Type", Value: "Test$Holder"}, + {Key: "El", Value: bson.D{{Key: "$ID", Value: freshID()}, {Key: "$Type", Value: c.typ}}}, + }) + got, ok := keysOf(t, CompletePropertySets(in, v(ver)), "El")[c.key] + if !ok { + t.Fatalf("a Mendix %s project kept %s without %s; Studio Pro writes it on every element", ver, c.typ, c.key) + } + if !c.check(got) { + t.Fatalf("%s.%s = %#v on %s, not the value mx convert wrote", c.typ, c.key, got, ver) + } + }) + } + } +} + +// The floor is the lowest version measured, and a stated value always wins: +// below it nothing is added, and an element that has the key keeps its value. +func TestCompletePropertySets_Issue1805RowsKeepTheirFloorAndTheStoredValue(t *testing.T) { + in := mustMarshal(t, bson.D{ + {Key: "$ID", Value: freshID()}, + {Key: "$Type", Value: "Test$Holder"}, + {Key: "El", Value: bson.D{{Key: "$ID", Value: freshID()}, {Key: "$Type", Value: "DomainModels$NoGeneralization"}}}, + }) + if got := keysOf(t, CompletePropertySets(in, v("10.24.23")), "El"); len(got) != 2 { + t.Errorf("10.24.23, below every measurement, got %v; want nothing added", got) + } + + owned := mustMarshal(t, bson.D{ + {Key: "$ID", Value: freshID()}, + {Key: "$Type", Value: "Test$Holder"}, + {Key: "El", Value: bson.D{ + {Key: "$ID", Value: freshID()}, + {Key: "$Type", Value: "DomainModels$NoGeneralization"}, + {Key: "HasOwnerAttr", Value: true}, + }}, + }) + if got := keysOf(t, CompletePropertySets(owned, v("11.14.0")), "El")["HasOwnerAttr"]; got != true { + t.Errorf("a stored HasOwnerAttr true became %#v", got) + } +} + +func isEmptyList(marker int32) func(any) bool { + return func(got any) bool { + a, ok := got.(bson.A) + return ok && len(a) == 1 && a[0] == marker + } +} diff --git a/modelsdk/canon/studiopro_property_defaults.json b/modelsdk/canon/studiopro_property_defaults.json index 9af1dd8033..be4b521a15 100644 --- a/modelsdk/canon/studiopro_property_defaults.json +++ b/modelsdk/canon/studiopro_property_defaults.json @@ -5,9 +5,14 @@ {"type":"DomainModels$AccessRule","key":"Documentation","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, {"type":"DomainModels$AccessRule","key":"XPathConstraintCaption","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, {"type":"DomainModels$AttributeRef","key":"EntityRef","since":"10.24.0","seen":"11.13.0","value":{"v":null}}, + {"type":"DomainModels$DateTimeAttributeType","key":"LocalizeDate","since":"10.24.24","seen":"11.15.0","value":{"v":true}}, {"type":"DomainModels$EntityImpl","key":"Image","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, {"type":"DomainModels$EntityImpl","key":"ImageData","since":"10.24.0","seen":"11.15.0","value":{"v":{"$binary":{"base64":"","subType":"00"}}}}, {"type":"DomainModels$EntityImpl","key":"Source","since":"10.24.0","seen":"11.15.0","value":{"v":null}}, + {"type":"DomainModels$NoGeneralization","key":"HasChangedByAttr","since":"10.24.24","seen":"11.15.0","value":{"v":false}}, + {"type":"DomainModels$NoGeneralization","key":"HasChangedDateAttr","since":"10.24.24","seen":"11.15.0","value":{"v":false}}, + {"type":"DomainModels$NoGeneralization","key":"HasCreatedDateAttr","since":"10.24.24","seen":"11.15.0","value":{"v":false}}, + {"type":"DomainModels$NoGeneralization","key":"HasOwnerAttr","since":"10.24.24","seen":"11.15.0","value":{"v":false}}, {"type":"DomainModels$RangeRuleInfo","key":"MaxAttribute","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, {"type":"DomainModels$RangeRuleInfo","key":"MaxValue","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, {"type":"DomainModels$RangeRuleInfo","key":"MinAttribute","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, @@ -74,6 +79,11 @@ {"type":"Microflows$RestOperationCallAction","key":"ErrorHandlingType","since":"10.24.0","seen":"11.15.0","value":{"v":"Rollback"}}, {"type":"Microflows$ValidationFeedbackAction","key":"Association","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, {"type":"Microflows$WhileLoopCondition","key":"Caption","since":"10.24.0","seen":"11.15.0","value":{"v":""}}, + {"type":"ODataPublish$EntitySet","key":"DeleteMode","since":"10.24.24","seen":"11.15.0","value":{"v":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"ODataPublish$ChangeNotSupported"}}}, + {"type":"ODataPublish$EntitySet","key":"InsertMode","since":"10.24.24","seen":"11.15.0","value":{"v":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"ODataPublish$ChangeNotSupported"}}}, + {"type":"ODataPublish$EntitySet","key":"ReadMode","since":"10.24.24","seen":"11.15.0","value":{"v":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"ODataPublish$ReadSource"}}}, + {"type":"ODataPublish$EntitySet","key":"UpdateMode","since":"10.24.24","seen":"11.15.0","value":{"v":{"$ID":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAA==","subType":"00"}},"$Type":"ODataPublish$ChangeNotSupported"}}}, + {"type":"ODataPublish$PublishedMicroflow","key":"Parameters","since":"10.24.24","seen":"11.15.0","value":{"v":[{"$numberInt":"3"}]}}, {"type":"ODataPublish$PublishedODataService2","key":"ExportLevel","since":"10.24.0","seen":"11.15.0","value":{"v":"Hidden"}}, {"type":"Projects$JarDependency","key":"Exclusions","since":"10.24.0","seen":"11.15.0","value":{"v":[{"$numberInt":"2"}]}}, {"type":"Projects$ModuleImpl","key":"AppStorePackageId","since":"10.24.0","until":"10.25.0","seen":"10.24.24","value":{"v":{"$numberLong":"0"}}}, diff --git a/modelsdk/mpr/property_set_issue1805_test.go b/modelsdk/mpr/property_set_issue1805_test.go new file mode 100644 index 0000000000..d28a3ae682 --- /dev/null +++ b/modelsdk/mpr/property_set_issue1805_test.go @@ -0,0 +1,194 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "os" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +// ako/mxcli#1805: on a Mendix 11.14.0 project mxcli wrote, `fix property-sets` +// reported +// +// MIXED DomainModels$DateTimeAttributeType.LocalizeDate: on 15 of 16 element(s); without it: bf8b2a01-… [DomainModels$DomainModel] +// MIXED DomainModels$NoGeneralization.HasChangedByAttr: on 81 of 84 element(s); without it: bf8b2a01-… [DomainModels$DomainModel] +// (and HasChangedDateAttr, HasCreatedDateAttr, HasOwnerAttr) +// +// No writer of today leaves those keys out — create, create or modify, alter, +// rename, move and the association writes all emit them (measured: a script +// through every one gives no such MIXED line on 10.24.24, 11.13.0 or 11.15.0). +// The elements without them were written by an older mxcli, which wrote only +// the system-attribute flags that were true and no LocalizeDate, and they +// survive every later write of their domain model: a write rebuilds the entity +// it names and passes its siblings through as their stored bytes. The table had +// no row for the keys, because no measurement of today's writers saw one +// missing, so neither a rewrite nor the repair filled them. + +const ( + issue1805DMID = "18051805-0000-4000-8000-000000000001" + issue1805NewID = "18051805-0000-4000-8000-000000000002" + issue1805OlderID = "18051805-0000-4000-8000-000000000003" +) + +// issue1805DomainModel is a domain model holding an entity as today's writer +// writes it and one as an older mxcli wrote it: a NoGeneralization with only +// Persistable, and a DateTime attribute type without LocalizeDate. +func issue1805DomainModel(t *testing.T, newDoc string) []byte { + t.Helper() + bin := func(id string) bson.Binary { return bson.Binary{Subtype: 0, Data: uuidToBlob(id)} } + sub := func(id string, n byte) bson.Binary { + b := uuidToBlob(id) + b[14] = n + return bson.Binary{Subtype: 0, Data: b} + } + entity := func(id, name, doc string, gen bson.D, dt bson.D) bson.D { + return bson.D{ + {Key: "$ID", Value: bin(id)}, + {Key: "$Type", Value: "DomainModels$EntityImpl"}, + {Key: "Documentation", Value: doc}, + {Key: "Image", Value: ""}, + {Key: "ImageData", Value: bson.Binary{Subtype: 0, Data: []byte{}}}, + {Key: "MaybeGeneralization", Value: gen}, + {Key: "Name", Value: name}, + {Key: "Source", Value: nil}, + {Key: "Attributes", Value: bson.A{int32(3), bson.D{ + {Key: "$ID", Value: sub(id, 0xa0)}, + {Key: "$Type", Value: "DomainModels$Attribute"}, + {Key: "Name", Value: "At"}, + {Key: "NewType", Value: dt}, + }}}, + } + } + current := entity(issue1805NewID, "Current", newDoc, + bson.D{ + {Key: "$ID", Value: sub(issue1805NewID, 0xb0)}, + {Key: "$Type", Value: "DomainModels$NoGeneralization"}, + {Key: "HasChangedByAttr", Value: false}, + {Key: "HasChangedDateAttr", Value: false}, + {Key: "HasCreatedDateAttr", Value: false}, + {Key: "HasOwnerAttr", Value: false}, + {Key: "Persistable", Value: true}, + }, + bson.D{ + {Key: "$ID", Value: sub(issue1805NewID, 0xc0)}, + {Key: "$Type", Value: "DomainModels$DateTimeAttributeType"}, + {Key: "LocalizeDate", Value: true}, + }) + older := entity(issue1805OlderID, "Older", "", + bson.D{ + {Key: "$ID", Value: sub(issue1805OlderID, 0xb0)}, + {Key: "$Type", Value: "DomainModels$NoGeneralization"}, + {Key: "Persistable", Value: true}, + }, + bson.D{ + {Key: "$ID", Value: sub(issue1805OlderID, 0xc0)}, + {Key: "$Type", Value: "DomainModels$DateTimeAttributeType"}, + }) + b, err := bson.Marshal(bson.D{ + {Key: "$ID", Value: bin(issue1805DMID)}, + {Key: "$Type", Value: "DomainModels$DomainModel"}, + {Key: "Entities", Value: bson.A{int32(3), current, older}}, + }) + if err != nil { + t.Fatal(err) + } + return b +} + +var issue1805Keys = map[string][]string{ + "DomainModels$NoGeneralization": {"HasChangedByAttr", "HasChangedDateAttr", "HasCreatedDateAttr", "HasOwnerAttr"}, + "DomainModels$DateTimeAttributeType": {"LocalizeDate"}, +} + +// olderEntityKeys returns, per $Type, the keys the Older entity's +// generalization and attribute type carry in raw. +func olderEntityKeys(t *testing.T, raw []byte) map[string]map[string]any { + t.Helper() + ents, err := bson.Raw(raw).Lookup("Entities").Array().Values() + if err != nil { + t.Fatal(err) + } + for _, e := range ents[1:] { + d := e.Document() + if d.Lookup("Name").StringValue() != "Older" { + continue + } + out := map[string]map[string]any{} + collect := func(sub bson.Raw) { + var m bson.D + if err := bson.Unmarshal(sub, &m); err != nil { + t.Fatal(err) + } + typ := "" + vals := map[string]any{} + for _, x := range m { + if x.Key == "$Type" { + typ, _ = x.Value.(string) + } + vals[x.Key] = x.Value + } + out[typ] = vals + } + collect(d.Lookup("MaybeGeneralization").Document()) + attrs, _ := d.Lookup("Attributes").Array().Values() + collect(attrs[1].Document().Lookup("NewType").Document()) + return out + } + t.Fatal("no Older entity") + return nil +} + +func TestIssue1805_CensusFindsNoMixedNameOnAnOlderEntity(t *testing.T) { + stored := issue1805DomainModel(t, "") + w, _ := newV2WriterForCommitTest(t, issue1805DMID, stored) + w.reader.projectVersion = pv(11, 14, 0) + + rep, err := w.reader.StoredPropertySetGaps(issue1805DMID) + if err != nil { + t.Fatal(err) + } + c := NewPropertySetCensus() + c.Count(rep) + for _, m := range c.Mixed() { + t.Errorf("MIXED %s.%s: on %d of %d element(s)", m.Type, m.Key, m.With, m.Elements) + } + if rep.Added != 5 { + t.Errorf("the repair would add %d key(s), want the 5 the older entity lacks", rep.Added) + } +} + +// Any write of the domain model — here one that changes only the CURRENT +// entity, as `alter entity Current set documentation` does, with the older one +// passed through as its stored bytes — completes the older entity on the way. +func TestIssue1805_AnyRewriteOfTheDomainModelCompletesAnOlderEntity(t *testing.T) { + w, path := newV2WriterForCommitTest(t, issue1805DMID, issue1805DomainModel(t, "")) + w.reader.projectVersion = pv(11, 14, 0) + + if err := w.UpdateRawUnit(issue1805DMID, issue1805DomainModel(t, "changed")); err != nil { + t.Fatal(err) + } + after, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + got := olderEntityKeys(t, after) + for typ, keys := range issue1805Keys { + for _, k := range keys { + if _, ok := got[typ][k]; !ok { + t.Errorf("after a rewrite of its domain model the older entity's %s still has no %s", typ, k) + } + } + } + // The values are Mendix's for the absent key, so the model says what it + // said before: no system member appears, the date stays localized. + for _, k := range issue1805Keys["DomainModels$NoGeneralization"] { + if v, ok := got["DomainModels$NoGeneralization"][k]; ok && v != false { + t.Errorf("%s = %#v, want false", k, v) + } + } + if v, ok := got["DomainModels$DateTimeAttributeType"]["LocalizeDate"]; ok && v != true { + t.Errorf("LocalizeDate = %#v, want true", v) + } +} diff --git a/modelsdk/mpr/reader.go b/modelsdk/mpr/reader.go index b252e23afa..44c8f079db 100644 --- a/modelsdk/mpr/reader.go +++ b/modelsdk/mpr/reader.go @@ -67,6 +67,11 @@ type Reader struct { // cache saves. Read by tests (mendixlabs/mxcli#1272). fileReads atomic.Int64 + // store is MPR v2's persisted copy of the units (reader_unit_store.go), + // made on first use; nil for v1 and for a reader with no project path. + storeOnce sync.Once + store *unitStore + // overlay holds unit bytes injected by BufferedUnitStore so that reads // within the same import file see buffered (uncommitted) writes. // nil means no overlay is active — zero cost on the normal path. @@ -290,6 +295,9 @@ func (r *Reader) Close() error { r.v2TypesMu.Lock() r.persistUnitTypeIndex() r.v2TypesMu.Unlock() + if store := r.unitStore(); store != nil { + store.close() + } if r.db != nil && r.ownsDB { return r.db.Close() } diff --git a/modelsdk/mpr/reader_cache_reads_test.go b/modelsdk/mpr/reader_cache_reads_test.go index b1d40ae092..45731f58fd 100644 --- a/modelsdk/mpr/reader_cache_reads_test.go +++ b/modelsdk/mpr/reader_cache_reads_test.go @@ -56,10 +56,12 @@ func TestCacheUnitReads_ReadsEachFileOnce(t *testing.T) { t.Errorf("cached: two listings read the file %d time(s), want 1", n) } + // Released, each read checks the file again: it is opened, or served by + // the persisted store after a stat says it is unchanged (#1804). release() - before = r.fileReads.Load() + before = r.fileReads.Load() + r.UnitStoreHits() listTwice(t, r) - if n := r.fileReads.Load() - before; n < 2 { + if n := r.fileReads.Load() + r.UnitStoreHits() - before; n < 2 { t.Errorf("released: two listings read the file %d time(s), want >= 2 (the cache must not outlive the pass)", n) } } diff --git a/modelsdk/mpr/reader_unit_store.go b/modelsdk/mpr/reader_unit_store.go new file mode 100644 index 0000000000..5d69432921 --- /dev/null +++ b/modelsdk/mpr/reader_unit_store.go @@ -0,0 +1,319 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "database/sql" + "os" + "path/filepath" + "sync" + + "github.com/klauspost/compress/zstd" +) + +// The unit store keeps a copy of every MPR v2 unit's bytes in one file, +// /.mxcli/cache/unit-contents.db, so a process that reads the whole +// project opens that file instead of every .mxunit (ako/mxcli#1804). +// +// Opening a file is what Windows Defender scans, and a v2 project is one file +// per document: a catalog refresh of a 4,753-unit app opened 4,753 of them, +// after every save, and the same users who reported hours of waiting measured +// a 77% speed-up from excluding the project from scanning. Reading a unit's +// size and modification time does not open it, so the store is checked with +// a stat and a unit is read from its file only when it changed. +// +// A stored copy is served only while all three of these still hold: +// +// - the file's size and modification time are what they were when it was +// read. A write by Studio Pro, by mxcli, or by a git checkout changes +// them; the stat is taken before the read, so a file caught mid-write is +// stored under the old stat and never served for the new one; +// - the Unit table's ContentsHash for it is the one it had then, which +// catches a project copied over another with its timestamps preserved; +// - the store was written by this format (unitStoreVersion). +// +// Anything else reads the file, as before. The store is a pure cache: +// absent, unreadable, locked by another process or unwritable all mean +// "read the file", never a wrong answer. It holds the bytes zstd-compressed +// (145 MB of units take about 60), and it lives in a directory of its own +// whose .gitignore ignores it, because .mxcli/ itself is not ignored in every +// project and a copy of the whole model must never be committed. + +const ( + unitStoreDir = "cache" + unitStoreName = "unit-contents.db" + unitStoreVersion = "1" +) + +// unitStoreMeta is what a stored copy was read under. +type unitStoreMeta struct { + size int64 + mtime int64 // UnixNano + hash string +} + +type unitStorePending struct { + meta unitStoreMeta + data []byte +} + +// unitStore is a reader's view of the persisted unit copies. All methods are +// safe for concurrent use: describes read units from several goroutines. +type unitStore struct { + path string + + mu sync.Mutex + loaded bool + db *sql.DB + stored map[string]unitStoreMeta // unit ID -> what its stored copy was read under + rowHash map[string]string // unit ID -> the Unit table's ContentsHash now + pending map[string]unitStorePending // read from a file since the last flush + removed bool // rowHash shows units the store still holds + foreign bool // the file is a store of another format + hits int64 + dec *zstd.Decoder + enc *zstd.Encoder +} + +func newUnitStore(projectPath string) *unitStore { + if projectPath == "" { + return nil + } + return &unitStore{ + path: filepath.Join(filepath.Dir(projectPath), ".mxcli", unitStoreDir, unitStoreName), + rowHash: map[string]string{}, + pending: map[string]unitStorePending{}, + } +} + +// open loads the stored copies' metadata on first use. Caller holds mu. +func (s *unitStore) open() { + if s.loaded { + return + } + s.loaded = true + s.stored = map[string]unitStoreMeta{} + if _, err := os.Stat(s.path); err != nil { + return + } + db, err := sql.Open("sqlite", "file:"+s.path+"?_pragma=busy_timeout(3000)") + if err != nil { + return + } + var version string + if err := db.QueryRow(`SELECT value FROM meta WHERE key = 'version'`).Scan(&version); err != nil || version != unitStoreVersion { + // Only a store this build can tell is another format is replaced; one + // it could not read (locked by another process) is left alone. + s.foreign = err == nil + db.Close() + return + } + rows, err := db.Query(`SELECT id, size, mtime, hash FROM units`) + if err != nil { + db.Close() + return + } + for rows.Next() { + var id string + var m unitStoreMeta + if rows.Scan(&id, &m.size, &m.mtime, &m.hash) != nil { + rows.Close() + db.Close() + s.stored = map[string]unitStoreMeta{} + return + } + s.stored[id] = m + } + rows.Close() + s.db = db +} + +// noteRowHash records the Unit table's ContentsHash of a unit, which a stored +// copy must match. buildUnitCache calls it for every row before it reads any. +func (s *unitStore) noteRowHash(id, hash string) { + s.mu.Lock() + s.rowHash[id] = hash + s.mu.Unlock() +} + +// setRowHashes replaces the row hashes with the Unit table's as a whole, so a +// unit that has gone is neither served nor kept. +func (s *unitStore) setRowHashes(hashes map[string]string) { + s.mu.Lock() + s.rowHash = hashes + s.removed = true + s.mu.Unlock() +} + +// lookup returns the stored copy of a unit if its file is unchanged, and the +// file's stat for recording a read when it is not (nil when the file cannot +// be stat'ed, which the read then reports). +func (s *unitStore) lookup(id, path string) ([]byte, os.FileInfo, bool) { + fi, err := os.Stat(path) + if err != nil { + return nil, nil, false + } + s.mu.Lock() + s.open() + m, stored := s.stored[id] + hash, known := s.rowHash[id] + db := s.db + s.mu.Unlock() + if !stored || !known || db == nil || + m.size != fi.Size() || m.mtime != fi.ModTime().UnixNano() || m.hash != hash { + return nil, fi, false + } + var blob []byte + if db.QueryRow(`SELECT data FROM units WHERE id = ?`, id).Scan(&blob) != nil { + return nil, fi, false + } + data, err := s.decoder().DecodeAll(blob, nil) + if err != nil || int64(len(data)) != fi.Size() { + return nil, fi, false + } + s.mu.Lock() + s.hits++ + s.mu.Unlock() + return data, fi, true +} + +// record keeps the bytes of a unit read from its file under the stat taken +// before the read, for the next flush. +func (s *unitStore) record(id string, fi os.FileInfo, data []byte) { + if fi == nil { + return + } + s.mu.Lock() + defer s.mu.Unlock() + hash, known := s.rowHash[id] + if !known { + return + } + s.pending[id] = unitStorePending{ + meta: unitStoreMeta{size: fi.Size(), mtime: fi.ModTime().UnixNano(), hash: hash}, + data: data, + } +} + +func (s *unitStore) decoder() *zstd.Decoder { + s.mu.Lock() + defer s.mu.Unlock() + if s.dec == nil { + s.dec, _ = zstd.NewReader(nil) + } + return s.dec +} + +// flush writes what was read from files since the last flush, and drops the +// copies of units the project no longer has. Best effort: a failure leaves +// the store as it was, and the next process reads those files again. +func (s *unitStore) flush() { + s.mu.Lock() + defer s.mu.Unlock() + s.open() + var gone []string + if s.removed { + for id := range s.stored { + if _, ok := s.rowHash[id]; !ok { + gone = append(gone, id) + } + } + } + if len(s.pending) == 0 && len(gone) == 0 { + return + } + if s.db == nil && !s.create() { + return + } + if s.enc == nil { + enc, err := zstd.NewWriter(nil, zstd.WithEncoderLevel(zstd.SpeedFastest)) + if err != nil { + return + } + s.enc = enc + } + tx, err := s.db.Begin() + if err != nil { + return + } + put, err := tx.Prepare(`INSERT OR REPLACE INTO units (id, size, mtime, hash, data) VALUES (?, ?, ?, ?, ?)`) + if err != nil { + tx.Rollback() + return + } + for id, p := range s.pending { + if _, err := put.Exec(id, p.meta.size, p.meta.mtime, p.meta.hash, s.enc.EncodeAll(p.data, nil)); err != nil { + put.Close() + tx.Rollback() + return + } + } + put.Close() + for _, id := range gone { + if _, err := tx.Exec(`DELETE FROM units WHERE id = ?`, id); err != nil { + tx.Rollback() + return + } + } + if tx.Commit() != nil { + return + } + for id, p := range s.pending { + s.stored[id] = p.meta + } + for _, id := range gone { + delete(s.stored, id) + } + s.pending = map[string]unitStorePending{} + s.removed = false +} + +// create makes an empty store. Caller holds mu. +func (s *unitStore) create() bool { + dir := filepath.Dir(s.path) + if err := os.MkdirAll(dir, 0o755); err != nil { + return false + } + ignore := filepath.Join(dir, ".gitignore") + if _, err := os.Stat(ignore); err != nil { + _ = os.WriteFile(ignore, []byte("# mxcli caches; never commit\n*\n"), 0o644) + } + if _, err := os.Stat(s.path); err == nil { + if !s.foreign { + return false + } + _ = os.Remove(s.path) + } + db, err := sql.Open("sqlite", "file:"+s.path+"?_pragma=busy_timeout(3000)") + if err != nil { + return false + } + for _, stmt := range []string{ + `CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT)`, + `CREATE TABLE units (id TEXT PRIMARY KEY, size INTEGER, mtime INTEGER, hash TEXT, data BLOB)`, + `INSERT INTO meta (key, value) VALUES ('version', '` + unitStoreVersion + `')`, + } { + if _, err := db.Exec(stmt); err != nil { + db.Close() + return false + } + } + s.db = db + return true +} + +// close flushes and releases the store's file. +func (s *unitStore) close() { + s.flush() + s.mu.Lock() + defer s.mu.Unlock() + if s.db != nil { + s.db.Close() + s.db = nil + } + s.loaded = false + if s.dec != nil { + s.dec.Close() + s.dec = nil + } +} diff --git a/modelsdk/mpr/reader_unit_store_test.go b/modelsdk/mpr/reader_unit_store_test.go new file mode 100644 index 0000000000..98830a9384 --- /dev/null +++ b/modelsdk/mpr/reader_unit_store_test.go @@ -0,0 +1,233 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "bytes" + "database/sql" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +// ako/mxcli#1804: a process that reads the whole project (a catalog refresh) +// opened every .mxunit, after every save; on Windows each open is scanned. +// The unit store keeps the bytes in one file and serves a unit from it while +// the unit's file is unchanged. + +// readEvery reads every unit of the project in one pass, the way a catalog +// build does, and returns the bytes by ID. +func readEvery(t *testing.T, r *Reader, ids []string) map[string][]byte { + t.Helper() + release := r.CacheUnitReads() + defer release() + if err := r.buildUnitCache(); err != nil { + t.Fatalf("build unit cache: %v", err) + } + out := map[string][]byte{} + for _, id := range ids { + b, err := r.readMprContents(blobToUUID(uuidToBlob(id))) + if err != nil { + t.Fatalf("read %s: %v", id, err) + } + out[id] = b + } + return out +} + +func storeProject(t *testing.T, n int) (*v2Project, []string) { + t.Helper() + p := newV2Project(t) + ids := []string{p.add(t, 0, "Projects$ModuleImpl")} + for i := 1; i <= n; i++ { + ids = append(ids, p.add(t, i, "Microflows$Microflow")) + } + return p, ids +} + +// warm runs one process over the project and closes it, leaving the store. +func warm(t *testing.T, p *v2Project, ids []string) { + t.Helper() + r := p.reader() + readEvery(t, r, ids) + if got := r.UnitFileReads(); got < int64(len(ids)) { + t.Fatalf("control: the cold pass read %d files, want >= %d", got, len(ids)) + } + if err := r.Close(); err != nil { + t.Fatal(err) + } +} + +func TestUnitStore_SecondProcessReadsNoUnitFile(t *testing.T) { + const n = 40 + p, ids := storeProject(t, n) + warm(t, p, ids) + + second := p.reader() + got := readEvery(t, second, ids) + if reads := second.UnitFileReads(); reads != 0 { + t.Errorf("a second pass over an unchanged project read %d unit files, want 0", reads) + } + if hits := second.UnitStoreHits(); hits < int64(len(ids)) { + t.Errorf("the store served %d reads, want >= %d", hits, len(ids)) + } + for _, id := range ids { + want, err := os.ReadFile(p.unitPath(id)) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(got[id], want) { + t.Errorf("unit %s: the store served bytes that differ from its file", id) + } + } +} + +// A save changes the file and its row: the next pass reads that one file. +func TestUnitStore_ChangedUnitIsReadFromItsFile(t *testing.T) { + p, ids := storeProject(t, 10) + warm(t, p, ids) + + changed := typedDoc(t, "Microflows$Microflow", "Renamed") + p.rewrite(t, ids[3], changed) + bumpMtime(t, p.unitPath(ids[3])) + + r := p.reader() + got := readEvery(t, r, ids) + if !bytes.Equal(got[ids[3]], changed) { + t.Fatalf("the changed unit was served from the store, not its file") + } + if reads := r.UnitFileReads(); reads != 1 { + t.Errorf("a pass after one save read %d unit files, want 1", reads) + } +} + +// A plain git checkout rewrites the file and can leave the Unit table's hash +// as it was (`mxcli git doctor` exists for that drift). The file's stat still +// changes, and that is enough. +func TestUnitStore_FileChangedBehindTheHashIsReadFromTheFile(t *testing.T) { + p, ids := storeProject(t, 10) + warm(t, p, ids) + + changed := typedDoc(t, "Microflows$Microflow", "UnitX") // the length of "Unit2": only the stat tells them apart + if err := os.WriteFile(p.unitPath(ids[2]), changed, 0o644); err != nil { + t.Fatal(err) + } + bumpMtime(t, p.unitPath(ids[2])) + + got := readEvery(t, p.reader(), ids) + if !bytes.Equal(got[ids[2]], changed) { + t.Fatalf("a file changed behind its row's hash was served from the store") + } +} + +// A project copied over another with its timestamps kept has the same stat +// and another hash. +func TestUnitStore_RowHashChangeIsReadFromTheFile(t *testing.T) { + p, ids := storeProject(t, 10) + warm(t, p, ids) + + if _, err := p.db.Exec(`UPDATE Unit SET ContentsHash = 'other' WHERE UnitID = ?`, uuidToBlob(ids[4])); err != nil { + t.Fatal(err) + } + r := p.reader() + readEvery(t, r, ids) + if reads := r.UnitFileReads(); reads != 1 { + t.Errorf("a pass after one row's hash changed read %d unit files, want 1", reads) + } +} + +// A deleted unit's copy does not outlive it. +func TestUnitStore_DeletedUnitIsDropped(t *testing.T) { + p, ids := storeProject(t, 10) + warm(t, p, ids) + + gone := ids[5] + if _, err := p.db.Exec(`DELETE FROM Unit WHERE UnitID = ?`, uuidToBlob(gone)); err != nil { + t.Fatal(err) + } + if err := os.Remove(p.unitPath(gone)); err != nil { + t.Fatal(err) + } + rest := append(append([]string{}, ids[:5]...), ids[6:]...) + warmAgain := p.reader() + readEvery(t, warmAgain, rest) + if err := warmAgain.Close(); err != nil { + t.Fatal(err) + } + + if n := storedRows(t, p, blobToUUID(uuidToBlob(gone))); n != 0 { + t.Errorf("the store still holds %d copies of a deleted unit", n) + } +} + +// A store an older or newer mxcli wrote is replaced, not misread. +func TestUnitStore_AnotherFormatIsReplaced(t *testing.T) { + p, ids := storeProject(t, 5) + path := storePath(p) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + db, err := sql.Open("sqlite", path) + if err != nil { + t.Fatal(err) + } + if _, err := db.Exec(`CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT); INSERT INTO meta VALUES ('version', '0')`); err != nil { + t.Fatal(err) + } + db.Close() + + warm(t, p, ids) + second := p.reader() + readEvery(t, second, ids) + if reads := second.UnitFileReads(); reads != 0 { + t.Errorf("after replacing another format's store a second pass read %d files, want 0", reads) + } +} + +// The store is a copy of the whole model: it must never be committed, and +// .mxcli/ is not ignored in every project. +func TestUnitStore_IsGitIgnored(t *testing.T) { + p, ids := storeProject(t, 3) + warm(t, p, ids) + b, err := os.ReadFile(filepath.Join(filepath.Dir(storePath(p)), ".gitignore")) + if err != nil { + t.Fatalf("the store's directory has no .gitignore: %v", err) + } + if !strings.Contains(string(b), "\n*\n") { + t.Errorf("the store's .gitignore does not ignore everything:\n%s", b) + } +} + +func storePath(p *v2Project) string { + return filepath.Join(p.root, ".mxcli", unitStoreDir, unitStoreName) +} + +func storedRows(t *testing.T, p *v2Project, id string) int { + t.Helper() + db, err := sql.Open("sqlite", storePath(p)) + if err != nil { + t.Fatal(err) + } + defer db.Close() + var n int + if err := db.QueryRow(`SELECT count(*) FROM units WHERE id = ?`, id).Scan(&n); err != nil { + t.Fatal(err) + } + return n +} + +// bumpMtime moves a file's modification time on, so a test does not depend +// on the file system's timestamp resolution to see a rewrite. +func bumpMtime(t *testing.T, path string) { + t.Helper() + fi, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + later := fi.ModTime().Add(2 * time.Second) + if err := os.Chtimes(path, later, later); err != nil { + t.Fatal(err) + } +} diff --git a/modelsdk/mpr/reader_units.go b/modelsdk/mpr/reader_units.go index 99385d4fab..25f3523a71 100644 --- a/modelsdk/mpr/reader_units.go +++ b/modelsdk/mpr/reader_units.go @@ -290,6 +290,8 @@ func (r *Reader) buildUnitCache() error { r.unitCache = nil hashes := make(map[string]struct{}) + store := r.unitStore() + rowHashes := make(map[string]string) opened := 0 for rows.Next() { var unitID, containerID []byte @@ -300,6 +302,10 @@ func (r *Reader) buildUnitCache() error { } unitUUID := blobToUUID(unitID) + rowHashes[unitUUID] = hash + if store != nil { + store.noteRowHash(unitUUID, hash) + } var entry unitIndexEntry if held, ok := r.overlaid(unitUUID); ok { // Bytes held in memory are what the unit is now; its stored hash @@ -332,6 +338,9 @@ func (r *Reader) buildUnitCache() error { return fmt.Errorf("failed to scan units: %w", err) } + if store != nil { + store.setRowHashes(rowHashes) + } first := r.v2Hashes == nil r.v2Hashes = hashes if first || opened >= unitTypeIndexPersistAfter { @@ -390,12 +399,18 @@ func (r *Reader) CacheUnitReads() (release func()) { return func() { once.Do(func() { r.contentMu.Lock() - defer r.contentMu.Unlock() - if r.cacheScopes <= 0 { - return + last := false + if r.cacheScopes > 0 { + if r.cacheScopes--; r.cacheScopes == 0 { + r.contentCache = nil + last = true + } } - if r.cacheScopes--; r.cacheScopes == 0 { - r.contentCache = nil + r.contentMu.Unlock() + // A pass over the project has just read what it needed: keep it + // for the next process (reader_unit_store.go). + if store := r.unitStore(); last && store != nil { + store.flush() } }) } @@ -405,6 +420,29 @@ func (r *Reader) CacheUnitReads() (release func()) { // mprcontents/. For tests. func (r *Reader) UnitFileReads() int64 { return r.fileReads.Load() } +// UnitStoreHits is how many unit reads this reader served from the persisted +// store instead of a file. For tests. +func (r *Reader) UnitStoreHits() int64 { + if store := r.unitStore(); store != nil { + store.mu.Lock() + defer store.mu.Unlock() + return store.hits + } + return 0 +} + +// unitStore is the reader's persisted unit store, or nil when there is none: +// MPR v1 holds its units in the .mpr, and a reader without a project path +// has nowhere to keep one. +func (r *Reader) unitStore() *unitStore { + r.storeOnce.Do(func() { + if r.version == MPRVersionV2 && r.contentsDir != "" { + r.store = newUnitStore(r.path) + } + }) + return r.store +} + // readMprContents reads content from the mprcontents folder for v2 format. // The path is: mprcontents/XX/YY/UUID.mxunit where XX and YY are first two chars of UUID. // @@ -439,19 +477,44 @@ func (r *Reader) readMprContents(unitUUID string) ([]byte, error) { unitUUID[2:4], unitUUID+".mxunit", ) + // An unchanged unit is read from the persisted store rather than its + // own file, which on Windows is what a virus scanner inspects (#1804). + store := r.unitStore() + var fi os.FileInfo + if store != nil { + stored, info, ok := store.lookup(unitUUID, path) + if ok { + if caching { + r.contentMu.Lock() + if r.contentCache != nil { + r.contentCache[unitUUID] = bytes.Clone(stored) + } + r.contentMu.Unlock() + } + return stored, nil + } + fi = info + } + data, err := os.ReadFile(path) if err != nil { return nil, err } r.fileReads.Add(1) + // One copy serves both the in-memory cache and the store: neither hands + // out the bytes it holds, only copies of them. + held := bytes.Clone(data) if caching { r.contentMu.Lock() if r.contentCache != nil { - r.contentCache[unitUUID] = bytes.Clone(data) + r.contentCache[unitUUID] = held } r.contentMu.Unlock() } + if store != nil { + store.record(unitUUID, fi, held) + } return data, nil } diff --git a/sdk/versions/mendix-10.yaml b/sdk/versions/mendix-10.yaml index bcfdf63ad2..81e4af2b87 100644 --- a/sdk/versions/mendix-10.yaml +++ b/sdk/versions/mendix-10.yaml @@ -12,9 +12,9 @@ features: min_version: "10.0.0" mdl: "CREATE NON-PERSISTENT ENTITY Module.Name (...)" view_entities: - min_version: "10.18.0" + min_version: "10.21.0" mdl: "CREATE VIEW ENTITY Module.Name (...) AS SELECT ..." - notes: "OQL stored inline on OqlViewEntitySource in 10.x" + notes: "View entities arrived in 10.18 with their OQL inline; mxcli stores the query in a separate DomainModels$ViewEntitySourceDocument that the entity's OqlViewEntitySource.SourceDocument names, both 10.21.0 (mendixmodelsdk 4.117.0), and the write refuses them below. 10.x also stores the OQL inline. Was 10.18.0 (ako/mxcli#1811)." calculated_attributes: min_version: "10.0.0" mdl: "CALCULATED BY Module.Microflow" @@ -76,18 +76,60 @@ features: send_rest_request: min_version: "10.1.0" mdl: "SEND REST REQUEST ..." + notes: "Microflows$RestOperationCallAction introduced in 10.1.0 (mendixmodelsdk 4.117.0). mx 9.24.30 loads the action (CE7053 \"No REST operation specified\", not a load failure), but the consumed REST service it calls, Rest$ConsumedRestService, is 10.1 as well, so a 9.24 author cannot set one up. check -p names the statement before the write refuses it (ako/mxcli#1811)." send_rest_query_params: min_version: "11.0.0" notes: "Query parameters in REST requests" execute_database_query: - min_version: "10.6.0" + min_version: "9.24.0" mdl: "EXECUTE DATABASE QUERY ..." + notes: "DatabaseConnector$ExecuteDatabaseQueryAction and QueryParameterMapping introduced in 9.24.0 (mendixmodelsdk 4.117.0); mx 9.24.30 loads the action (CE1613 for a query that does not exist, not a load failure). Was 10.6.0, a floor nothing measured (ako/mxcli#1811)." execute_database_query_runtime_connection: - min_version: "11.0.0" - notes: "Runtime connection override for database queries" + min_version: "11.1.0" + notes: "DatabaseConnector$ConnectionParameterMapping and ExecuteDatabaseQueryAction.ConnectionParameterMappings introduced in 11.1.0 (mendixmodelsdk 4.117.0). Was 11.0.0 (ako/mxcli#1811)." loop_in_branch: min_version: "10.0.0" notes: "LOOP inside IF/ELSE branches" + list_find_filter_by_expression: + min_version: "9.0.3" + mdl: "$Found = FIND $List WHERE expression; $Kept = FILTER $List WHERE expression;" + notes: "Microflows$FindByExpression and FilterByExpression introduced in 9.0.3 (mendixmodelsdk 4.117.0); `BY Member = value` writes Microflows$Find/Filter, which every version has. The write refuses each type below it (canon.UndeclaredTypeError); check -p names the statement first (ako/mxcli#1811)." + call_external_action: + min_version: "9.19.0" + mdl: "CALL EXTERNAL ACTION Module.Service.Action ();" + notes: "Microflows$CallExternalAction introduced in 9.19.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + list_range: + min_version: "9.24.0" + mdl: "$Page = RANGE $List OFFSET $Offset LIMIT $Amount;" + notes: "Microflows$ListRange introduced in 9.24.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + send_rest_request_parameters: + min_version: "10.2.0" + mdl: "SEND REST REQUEST Module.Service.Operation WITH (Id = $Order/Id);" + notes: "Microflows$RestOperationParameterMapping introduced in 10.2.0 (mendixmodelsdk 4.117.0). Below 11.0 an operation has only path parameters (Rest$QueryParameter is 11.0), so every argument is one (ako/mxcli#1811)." + call_external_action_arguments: + min_version: "10.2.0" + mdl: "CALL EXTERNAL ACTION Module.Service.Action (Name = $Value);" + notes: "Microflows$ExternalActionParameterMapping introduced in 10.2.0 (mendixmodelsdk 4.117.0); an action called with no arguments writes none (ako/mxcli#1811)." + send_rest_request_output: + min_version: "10.4.0" + mdl: "$Response = SEND REST REQUEST Module.Service.Operation;" + notes: "Microflows$OutputVariable introduced in 10.4.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + send_rest_request_body: + min_version: "10.5.0" + mdl: "SEND REST REQUEST Module.Service.Operation BODY $Order;" + notes: "Microflows$BodyVariable introduced in 10.5.0 (mendixmodelsdk 4.117.0). Written for an operation whose body is an export mapping; the clause is gated as written (ako/mxcli#1811)." + loop_over_list: + min_version: "9.0.4" + mdl: "LOOP $Item IN $List BEGIN … END LOOP;" + notes: "Microflows$IterableList and LoopedActivity.LoopSource introduced in 9.0.4 (mendixmodelsdk 4.117.0); before, a loop stored its list on the activity itself, which mxcli does not write. The write refuses the loop below it; check -p names the statement first (ako/mxcli#1811)." + while_loop: + min_version: "9.0.4" + mdl: "WHILE condition BEGIN … END WHILE;" + notes: "Microflows$WhileLoopCondition introduced in 9.0.4 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + queued_calls: + min_version: "9.10.0" + mdl: "CALL MICROFLOW Module.Flow() IN QUEUE Module.Queue; CALL JAVA ACTION Module.Action() IN QUEUE Module.Queue;" + notes: "Queues$QueueSettings and the calls' QueueSettings introduced in 9.10.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." pages: basic: @@ -124,6 +166,18 @@ features: alter_page: min_version: "10.0.0" mdl: "ALTER PAGE Module.Name SET/INSERT/DROP/REPLACE ..." + workflow_page_actions: + min_version: "9.0.2" + mdl: "action: open user task | open workflow | call workflow Module.WF" + notes: "Forms$OpenUserTaskClientAction, OpenWorkflowClientAction and CallWorkflowClientAction introduced in 9.0.2 with workflows (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + complete_task_action: + min_version: "9.19.0" + mdl: "action: complete task 'Outcome'" + notes: "Forms$SetTaskOutcomeClientAction is 9.0.2, but the OutcomeValue mxcli writes is 9.19.0 (mendixmodelsdk 4.117.0); below, the write refuses the page (ako/mxcli#1811)." + snippet_parameters: + min_version: "9.21.0" + mdl: "CREATE SNIPPET Module.Name (Params: ($P: Module.Entity)) { … }" + notes: "Forms$SnippetParameter and Snippet.Parameters introduced in 9.21.0 (mendixmodelsdk 4.117.0). A snippet call writes a mapping only for a parameter the called snippet declares, so it cannot reach an older project (ako/mxcli#1811)." security: module_roles: @@ -156,15 +210,17 @@ features: min_version: "11.9.0" mdl: "CREATE DATA TRANSFORMER Module.Name SOURCE JSON '...' { JSLT '...'; }" database_connector_basic: - min_version: "10.6.0" + min_version: "10.0.0" mdl: "CREATE DATABASE CONNECTION Module.Name ..." + notes: "DatabaseConnector$DatabaseConnection is 9.22.0, but the ConnectionInput (a DatabaseConnector$ConnectionString) mxcli writes is 10.0.0 (mendixmodelsdk 4.117.0); below, the write refuses the connection. Was 10.6.0, a floor nothing measured (ako/mxcli#1811)." database_connector_execute: - min_version: "10.6.0" + min_version: "9.24.0" mdl: "EXECUTE DATABASE QUERY in microflows" - notes: "Full BSON format requires 11.0+" + notes: "The action microflows.execute_database_query gates: 9.24.0 (ako/mxcli#1811)." business_events: - min_version: "10.0.0" + min_version: "9.24.0" mdl: "CREATE BUSINESS EVENT SERVICE Module.Name ..." + notes: "BusinessEvents$BusinessEventService and its children introduced in 9.24.0 (mendixmodelsdk 4.117.0). mx 9.24.30 loads a service mxcli writes; its errors there are the content (the Business Events module, an attribute's length). Was 10.0.0 (ako/mxcli#1811)." odata_client: min_version: "10.0.0" published_odata_service: @@ -202,15 +258,33 @@ features: # (ako/mxcli#987). min_version: "10.0.0" mdl: "CREATE MESSAGE DEFINITION COLLECTION Module.Name { DEFINITION ... }" + rest_client_authentication: + min_version: "10.2.0" + mdl: "authentication: basic (username: '…', password: '…')" + notes: "Rest$BasicAuthenticationScheme and ConsumedRestService.AuthenticationScheme introduced in 10.2.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + rest_client_operations: + min_version: "10.4.0" + mdl: "OPERATION Name ( method: get, path: '…', response: … )" + notes: "Rest$RestOperation.Method (RestOperationMethodWithBody / WithoutBody) is 10.4.0 and ResponseHandling 10.3.0 (mendixmodelsdk 4.117.0); every operation mxcli writes holds both, so a consumed REST service with an operation needs 10.4 (ako/mxcli#1811)." + rest_client_openapi: + min_version: "10.21.0" + mdl: "CREATE CONSUMED REST SERVICE Module.Name ( OpenAPI: '…' )" + notes: "Rest$OpenApiFile and ConsumedRestService.OpenApiFile introduced in 10.21.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + database_connector_queries: + min_version: "10.12.0" + mdl: "CREATE DATABASE CONNECTION Module.Name ( … ) { QUERY Name ( Sql: '…' ) }" + notes: "DatabaseConnector$DatabaseQuery is 9.22.0, but its QueryType, which mxcli writes for every query, and TableMappings are 10.12.0 (mendixmodelsdk 4.117.0); measured on stamped projects, the write refuses a connection with a query below 10.12 (ako/mxcli#1811)." workflows: basic: - min_version: "9.0.0" + min_version: "9.0.2" mdl: "CREATE WORKFLOW Module.Name ..." + notes: "Workflows$Workflow introduced in 9.0.2 (mendixmodelsdk 4.117.0); the CDN lists mxbuild 9.0.0 and 9.0.1, which have no workflows. The write refuses the type below it (canon.UndeclaredTypeError); create workflow, create or modify workflow and alter workflow are refused by check -p first (ako/mxcli#1809)." parallel_split: min_version: "9.0.0" user_task: - min_version: "9.0.0" + min_version: "9.19.0" + notes: "Workflows$UserTask is 9.0.2, but the outcome value (Workflows$UserTaskOutcome.Value) mxcli writes is 9.19.0, its task page 9.11.0 and its on-created event 9.0.5 (mendixmodelsdk 4.117.0). Measured on stamped 9.x projects: the write refuses a user task below 9.19 (ako/mxcli#1811)." multi_user_task: min_version: "9.22.0" mdl: "MULTI USER TASK Vote 'Vote' PAGE M.TaskPage OUTCOMES 'Yes' { } 'No' { };" @@ -247,10 +321,66 @@ features: min_version: "10.0.0" mdl: "LOCK WORKFLOW Module.Workflow; UNLOCK WORKFLOW $Workflow;" notes: "Lock/UnlockWorkflowAction.workflowSelection and the Workflows$WorkflowDefinition(Name|Object)Selection it holds introduced in 10.0.0; 9.21–9.24 store the workflow as the action's own Workflow (mendixmodelsdk 4.117.0; ako/mxcli#1774)." + workflow_microflow_actions: + min_version: "9.0.2" + mdl: "CALL WORKFLOW Module.WF ($Ctx); SET TASK OUTCOME $Task 'Approve'; OPEN USER TASK $Task; OPEN WORKFLOW $Wf;" + notes: "Microflows$WorkflowCallAction, SetTaskOutcomeAction, OpenUserTaskAction and OpenWorkflowAction introduced in 9.0.2 with workflows (mendixmodelsdk 4.117.0). The write refuses each type below it (canon.UndeclaredTypeError); check -p names the statement first (ako/mxcli#1787)." + workflow_operations: + min_version: "9.2.0" + mdl: "WORKFLOW OPERATION ABORT $Wf REASON '…'; WORKFLOW OPERATION RESTART $Wf;" + notes: "Microflows$WorkflowOperationAction, AbortOperation and RestartOperation introduced in 9.2.0 (mendixmodelsdk 4.117.0; ako/mxcli#1787)." + workflow_operation_pause_continue: + min_version: "9.3.0" + mdl: "WORKFLOW OPERATION PAUSE $Wf; WORKFLOW OPERATION CONTINUE $Wf;" + notes: "Microflows$PauseOperation and ContinueOperation introduced in 9.3.0 (mendixmodelsdk 4.117.0; ako/mxcli#1787)." + get_workflow_data: + min_version: "9.10.0" + mdl: "$Data = GET WORKFLOW DATA $Wf AS Module.WF;" + notes: "Microflows$GetWorkflowDataAction introduced in 9.10.0 (mendixmodelsdk 4.117.0; ako/mxcli#1787)." + workflow_operation_retry_unpause: + min_version: "9.12.0" + mdl: "WORKFLOW OPERATION RETRY $Wf; WORKFLOW OPERATION UNPAUSE $Wf;" + notes: "Microflows$RetryOperation and UnpauseOperation introduced in 9.12.0, replacing ResumeOperation (9.3.0 to 9.11; mendixmodelsdk 4.117.0; ako/mxcli#1787)." + notify_workflow: + min_version: "10.0.0" + mdl: "$Notified = NOTIFY WORKFLOW $Wf TARGET Module.WF.WaitForApproval;" + notes: "Microflows$NotifyWorkflowAction introduced in 10.0.0, with the wait for notification it resumes (mendixmodelsdk 4.117.0); absent from the 9.24.30 mxbuild (ako/mxcli#1787)." + get_workflows: + min_version: "10.4.0" + mdl: "$Workflows = GET WORKFLOWS FOR $Ctx;" + notes: "Microflows$GetWorkflowsAction introduced in 10.4.0 (mendixmodelsdk 4.117.0); absent from the 9.24.30 mxbuild (ako/mxcli#1787)." + get_workflow_activity_records: + min_version: "10.4.0" + mdl: "$Records = GET WORKFLOW ACTIVITY RECORDS $Wf;" + notes: "Microflows$GetWorkflowActivityRecordsAction introduced in 10.4.0 (mendixmodelsdk 4.117.0); absent from the 9.24.30 mxbuild (ako/mxcli#1787)." event_handlers: min_version: "10.7.0" mdl: "CREATE WORKFLOW … ON WORKFLOW EVENTS (…) MICROFLOW Module.Handler AS '…'" notes: "Workflow.onWorkflowEvent introduced in 10.7.0 (modelsdk gen). `ON ANY WORKFLOW EVENT` expands to the measured per-version type list and needs 11.6.0+." + jump_to: + min_version: "9.0.5" + mdl: "JUMP TO Activity;" + notes: "Workflows$JumpToActivity introduced in 9.0.5 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + call_workflow_activity: + min_version: "9.6.0" + mdl: "CALL WORKFLOW Module.WF; (an activity of a workflow)" + notes: "Workflows$CallWorkflowActivity is 9.0.2, but the ParameterExpression mxcli writes is 9.6.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + overview_page: + min_version: "9.11.0" + mdl: "OVERVIEW PAGE Module.Page" + notes: "Workflow.AdminPage, a Workflows$PageReference, introduced in 9.11.0 (mendixmodelsdk 4.117.0); before, the property was OverviewPage, which mxcli does not write (ako/mxcli#1811)." + annotations: + min_version: "9.15.0" + mdl: "ANNOTATION '…' (on a workflow or an activity)" + notes: "Workflows$Annotation and Workflow.Annotation introduced in 9.15.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + workflow_parameter: + min_version: "9.18.0" + mdl: "PARAMETER $WorkflowContext: Module.Entity" + notes: "Workflows$Parameter is 9.6.0, its Entity 9.10.0 and its Name 9.18.0 (mendixmodelsdk 4.117.0); mxcli writes all three. Measured on stamped 9.x projects: below 9.18 the write refuses every workflow with a parameter (ako/mxcli#1811)." + call_workflow_parameter_mappings: + min_version: "9.18.0" + mdl: "CALL WORKFLOW Module.WF (WorkflowContext = $Expr); (an activity of a workflow)" + notes: "Workflows$WorkflowCallParameterMapping and CallWorkflowActivity.ParameterMappings introduced in 9.18.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." navigation: profiles: @@ -259,6 +389,22 @@ features: min_version: "10.0.0" home_pages: min_version: "10.0.0" + progressive_web_app: + min_version: "9.0.3" + mdl: "CREATE OR MODIFY NAVIGATION Profile PROGRESSIVE WEB APP ( Precaching: true )" + notes: "Navigation$ProgressiveWebAppSettings introduced in 9.0.3 (mendixmodelsdk 4.117.0); `progressive web app off` writes none (ako/mxcli#1811)." + native_home_page: + min_version: "9.4.0" + mdl: "CREATE OR MODIFY NAVIGATION NativePhone HOME PAGE Module.Page" + notes: "Navigation$NativeHomePage and NativeNavigationProfile.NativeHomePage introduced in 9.4.0 (mendixmodelsdk 4.117.0); the native profile itself is older (ako/mxcli#1811)." + menu_item_icons: + min_version: "9.20.0" + mdl: "MENU ITEM 'Home' ( OnClick: …, Icon: Module.Collection.icon )" + notes: "Forms$IconCollectionIcon introduced in 9.20.0 with icon collections (mendixmodelsdk 4.117.0); `Icon:` names an icon collection entry and nothing else (ako/mxcli#1811)." + not_found_page: + min_version: "10.13.0" + mdl: "NOT FOUND PAGE Module.Page" + notes: "Navigation$NotFoundHomePage and NavigationProfile.NotFoundHomepage introduced in 10.13.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." mpr_format: mpr_v2: @@ -268,6 +414,22 @@ features: min_version: "11.6.0" notes: "New portable app deployment format" + modules: + jar_dependencies: + min_version: "10.0.0" + mdl: "ALTER MODULE Module ADD JAR DEPENDENCY (group = '…', artifact = '…', version = '…')" + notes: "Projects$JarDependency and ModuleSettings.JarDependencies introduced in 10.0.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + jar_dependency_exclusions: + min_version: "10.12.0" + mdl: "ALTER MODULE Module SET JAR DEPENDENCY 'group:artifact' ADD EXCLUSION '…'" + notes: "Projects$JarDependencyExclusion introduced in 10.12.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + + scheduled_events: + schedules: + min_version: "9.12.0" + mdl: "CREATE SCHEDULED EVENT Module.Name ( Microflow: …, Repeat: Daily, … )" + notes: "ScheduledEvents$MinuteSchedule … YearWeekdaySchedule, ScheduledEvent.Schedule and OnOverlap introduced in 9.12.0 (mendixmodelsdk 4.117.0); before, an event has only an interval, which mxcli does not write (ako/mxcli#1811)." + deprecated: - id: "DEP001" pattern: "Persistable: false on view entities" diff --git a/sdk/versions/mendix-11.yaml b/sdk/versions/mendix-11.yaml index 2a551c8b60..bd37439884 100644 --- a/sdk/versions/mendix-11.yaml +++ b/sdk/versions/mendix-11.yaml @@ -10,8 +10,9 @@ features: min_version: "10.0.0" mdl: "CREATE NON-PERSISTENT ENTITY Module.Name (...)" view_entities: - min_version: "10.18.0" + min_version: "10.21.0" mdl: "CREATE VIEW ENTITY Module.Name (...) AS SELECT ..." + notes: "View entities arrived in 10.18 with their OQL inline; mxcli stores the query in a separate DomainModels$ViewEntitySourceDocument that the entity's OqlViewEntitySource.SourceDocument names, both 10.21.0 (mendixmodelsdk 4.117.0), and the write refuses them below. 10.x also stores the OQL inline. Was 10.18.0 (ako/mxcli#1811)." calculated_attributes: min_version: "10.0.0" mdl: "CALCULATED BY Module.Microflow" @@ -77,18 +78,65 @@ features: send_rest_request: min_version: "10.1.0" mdl: "SEND REST REQUEST ..." + notes: "Microflows$RestOperationCallAction introduced in 10.1.0 (mendixmodelsdk 4.117.0). mx 9.24.30 loads the action (CE7053 \"No REST operation specified\", not a load failure), but the consumed REST service it calls, Rest$ConsumedRestService, is 10.1 as well, so a 9.24 author cannot set one up. check -p names the statement before the write refuses it (ako/mxcli#1811)." send_rest_query_params: min_version: "11.0.0" mdl: "SEND REST REQUEST with query parameters" execute_database_query: - min_version: "10.6.0" + min_version: "9.24.0" mdl: "EXECUTE DATABASE QUERY ..." + notes: "DatabaseConnector$ExecuteDatabaseQueryAction and QueryParameterMapping introduced in 9.24.0 (mendixmodelsdk 4.117.0); mx 9.24.30 loads the action (CE1613 for a query that does not exist, not a load failure). Was 10.6.0, a floor nothing measured (ako/mxcli#1811)." execute_database_query_runtime_connection: - min_version: "11.0.0" + min_version: "11.1.0" mdl: "EXECUTE DATABASE QUERY with runtime connection override" + notes: "DatabaseConnector$ConnectionParameterMapping and ExecuteDatabaseQueryAction.ConnectionParameterMappings introduced in 11.1.0 (mendixmodelsdk 4.117.0). Was 11.0.0 (ako/mxcli#1811)." loop_in_branch: min_version: "10.0.0" notes: "LOOP inside IF/ELSE branches" + list_find_filter_by_expression: + min_version: "9.0.3" + mdl: "$Found = FIND $List WHERE expression; $Kept = FILTER $List WHERE expression;" + notes: "Microflows$FindByExpression and FilterByExpression introduced in 9.0.3 (mendixmodelsdk 4.117.0); `BY Member = value` writes Microflows$Find/Filter, which every version has. The write refuses each type below it (canon.UndeclaredTypeError); check -p names the statement first (ako/mxcli#1811)." + call_external_action: + min_version: "9.19.0" + mdl: "CALL EXTERNAL ACTION Module.Service.Action ();" + notes: "Microflows$CallExternalAction introduced in 9.19.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + list_range: + min_version: "9.24.0" + mdl: "$Page = RANGE $List OFFSET $Offset LIMIT $Amount;" + notes: "Microflows$ListRange introduced in 9.24.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + send_rest_request_parameters: + min_version: "10.2.0" + mdl: "SEND REST REQUEST Module.Service.Operation WITH (Id = $Order/Id);" + notes: "Microflows$RestOperationParameterMapping introduced in 10.2.0 (mendixmodelsdk 4.117.0). Below 11.0 an operation has only path parameters (Rest$QueryParameter is 11.0), so every argument is one (ako/mxcli#1811)." + call_external_action_arguments: + min_version: "10.2.0" + mdl: "CALL EXTERNAL ACTION Module.Service.Action (Name = $Value);" + notes: "Microflows$ExternalActionParameterMapping introduced in 10.2.0 (mendixmodelsdk 4.117.0); an action called with no arguments writes none (ako/mxcli#1811)." + send_rest_request_output: + min_version: "10.4.0" + mdl: "$Response = SEND REST REQUEST Module.Service.Operation;" + notes: "Microflows$OutputVariable introduced in 10.4.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + send_rest_request_body: + min_version: "10.5.0" + mdl: "SEND REST REQUEST Module.Service.Operation BODY $Order;" + notes: "Microflows$BodyVariable introduced in 10.5.0 (mendixmodelsdk 4.117.0). Written for an operation whose body is an export mapping; the clause is gated as written (ako/mxcli#1811)." + transform_json: + min_version: "11.4.0" + mdl: "$Out = TRANSFORM $Json WITH Module.Transformer;" + notes: "Microflows$TransformJsonAction introduced in 11.4.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + loop_over_list: + min_version: "9.0.4" + mdl: "LOOP $Item IN $List BEGIN … END LOOP;" + notes: "Microflows$IterableList and LoopedActivity.LoopSource introduced in 9.0.4 (mendixmodelsdk 4.117.0); before, a loop stored its list on the activity itself, which mxcli does not write. The write refuses the loop below it; check -p names the statement first (ako/mxcli#1811)." + while_loop: + min_version: "9.0.4" + mdl: "WHILE condition BEGIN … END WHILE;" + notes: "Microflows$WhileLoopCondition introduced in 9.0.4 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + queued_calls: + min_version: "9.10.0" + mdl: "CALL MICROFLOW Module.Flow() IN QUEUE Module.Queue; CALL JAVA ACTION Module.Action() IN QUEUE Module.Queue;" + notes: "Queues$QueueSettings and the calls' QueueSettings introduced in 9.10.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." pages: basic: @@ -136,6 +184,18 @@ features: alter_page: min_version: "10.0.0" mdl: "ALTER PAGE Module.Name SET/INSERT/DROP/REPLACE ..." + workflow_page_actions: + min_version: "9.0.2" + mdl: "action: open user task | open workflow | call workflow Module.WF" + notes: "Forms$OpenUserTaskClientAction, OpenWorkflowClientAction and CallWorkflowClientAction introduced in 9.0.2 with workflows (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + complete_task_action: + min_version: "9.19.0" + mdl: "action: complete task 'Outcome'" + notes: "Forms$SetTaskOutcomeClientAction is 9.0.2, but the OutcomeValue mxcli writes is 9.19.0 (mendixmodelsdk 4.117.0); below, the write refuses the page (ako/mxcli#1811)." + snippet_parameters: + min_version: "9.21.0" + mdl: "CREATE SNIPPET Module.Name (Params: ($P: Module.Entity)) { … }" + notes: "Forms$SnippetParameter and Snippet.Parameters introduced in 9.21.0 (mendixmodelsdk 4.117.0). A snippet call writes a mapping only for a parameter the called snippet declares, so it cannot reach an older project (ako/mxcli#1811)." security: module_roles: @@ -169,14 +229,17 @@ features: min_version: "11.9.0" mdl: "CREATE DATA TRANSFORMER Module.Name SOURCE JSON '...' { JSLT '...'; }" database_connector_basic: - min_version: "10.6.0" + min_version: "10.0.0" mdl: "CREATE DATABASE CONNECTION Module.Name ..." + notes: "DatabaseConnector$DatabaseConnection is 9.22.0, but the ConnectionInput (a DatabaseConnector$ConnectionString) mxcli writes is 10.0.0 (mendixmodelsdk 4.117.0); below, the write refuses the connection. Was 10.6.0, a floor nothing measured (ako/mxcli#1811)." database_connector_execute: - min_version: "10.6.0" + min_version: "9.24.0" mdl: "EXECUTE DATABASE QUERY in microflows" + notes: "The action microflows.execute_database_query gates: 9.24.0 (ako/mxcli#1811)." business_events: - min_version: "10.0.0" + min_version: "9.24.0" mdl: "CREATE BUSINESS EVENT SERVICE Module.Name ..." + notes: "BusinessEvents$BusinessEventService and its children introduced in 9.24.0 (mendixmodelsdk 4.117.0). mx 9.24.30 loads a service mxcli writes; its errors there are the content (the Business Events module, an attribute's length). Was 10.0.0 (ako/mxcli#1811)." odata_client: min_version: "10.0.0" published_odata_service: @@ -213,6 +276,22 @@ features: # definition, the 11.15 replacement for the collection (ako/mxcli#987). min_version: "11.15.0" mdl: "CREATE MESSAGE DEFINITION Module.Name FOR Module.Entity { ... }" + rest_client_authentication: + min_version: "10.2.0" + mdl: "authentication: basic (username: '…', password: '…')" + notes: "Rest$BasicAuthenticationScheme and ConsumedRestService.AuthenticationScheme introduced in 10.2.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + rest_client_operations: + min_version: "10.4.0" + mdl: "OPERATION Name ( method: get, path: '…', response: … )" + notes: "Rest$RestOperation.Method (RestOperationMethodWithBody / WithoutBody) is 10.4.0 and ResponseHandling 10.3.0 (mendixmodelsdk 4.117.0); every operation mxcli writes holds both, so a consumed REST service with an operation needs 10.4 (ako/mxcli#1811)." + rest_client_openapi: + min_version: "10.21.0" + mdl: "CREATE CONSUMED REST SERVICE Module.Name ( OpenAPI: '…' )" + notes: "Rest$OpenApiFile and ConsumedRestService.OpenApiFile introduced in 10.21.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + database_connector_queries: + min_version: "10.12.0" + mdl: "CREATE DATABASE CONNECTION Module.Name ( … ) { QUERY Name ( Sql: '…' ) }" + notes: "DatabaseConnector$DatabaseQuery is 9.22.0, but its QueryType, which mxcli writes for every query, and TableMappings are 10.12.0 (mendixmodelsdk 4.117.0); measured on stamped projects, the write refuses a connection with a query below 10.12 (ako/mxcli#1811)." agent_documents: # Agent Editor documents (Model, Knowledge Base, Consumed MCP Service, @@ -234,8 +313,9 @@ features: workflows: basic: - min_version: "9.0.0" + min_version: "9.0.2" mdl: "CREATE WORKFLOW Module.Name ..." + notes: "Workflows$Workflow introduced in 9.0.2 (mendixmodelsdk 4.117.0); the CDN lists mxbuild 9.0.0 and 9.0.1, which have no workflows. The write refuses the type below it (canon.UndeclaredTypeError); create workflow, create or modify workflow and alter workflow are refused by check -p first (ako/mxcli#1809)." groups: min_version: "11.2.0" mdl: "ALTER SETTINGS WORKFLOWS ADD [OR MODIFY] GROUP 'Approvers' (Description: '...')" @@ -243,7 +323,8 @@ features: parallel_split: min_version: "9.0.0" user_task: - min_version: "9.0.0" + min_version: "9.19.0" + notes: "Workflows$UserTask is 9.0.2, but the outcome value (Workflows$UserTaskOutcome.Value) mxcli writes is 9.19.0, its task page 9.11.0 and its on-created event 9.0.5 (mendixmodelsdk 4.117.0). Measured on stamped 9.x projects: the write refuses a user task below 9.19 (ako/mxcli#1811)." multi_user_task: min_version: "9.22.0" mdl: "MULTI USER TASK Vote 'Vote' PAGE M.TaskPage OUTCOMES 'Yes' { } 'No' { };" @@ -288,6 +369,38 @@ features: min_version: "10.0.0" mdl: "LOCK WORKFLOW Module.Workflow; UNLOCK WORKFLOW $Workflow;" notes: "Lock/UnlockWorkflowAction.workflowSelection and the Workflows$WorkflowDefinition(Name|Object)Selection it holds introduced in 10.0.0; 9.21–9.24 store the workflow as the action's own Workflow (mendixmodelsdk 4.117.0; ako/mxcli#1774)." + workflow_microflow_actions: + min_version: "9.0.2" + mdl: "CALL WORKFLOW Module.WF ($Ctx); SET TASK OUTCOME $Task 'Approve'; OPEN USER TASK $Task; OPEN WORKFLOW $Wf;" + notes: "Microflows$WorkflowCallAction, SetTaskOutcomeAction, OpenUserTaskAction and OpenWorkflowAction introduced in 9.0.2 with workflows (mendixmodelsdk 4.117.0). The write refuses each type below it (canon.UndeclaredTypeError); check -p names the statement first (ako/mxcli#1787)." + workflow_operations: + min_version: "9.2.0" + mdl: "WORKFLOW OPERATION ABORT $Wf REASON '…'; WORKFLOW OPERATION RESTART $Wf;" + notes: "Microflows$WorkflowOperationAction, AbortOperation and RestartOperation introduced in 9.2.0 (mendixmodelsdk 4.117.0; ako/mxcli#1787)." + workflow_operation_pause_continue: + min_version: "9.3.0" + mdl: "WORKFLOW OPERATION PAUSE $Wf; WORKFLOW OPERATION CONTINUE $Wf;" + notes: "Microflows$PauseOperation and ContinueOperation introduced in 9.3.0 (mendixmodelsdk 4.117.0; ako/mxcli#1787)." + get_workflow_data: + min_version: "9.10.0" + mdl: "$Data = GET WORKFLOW DATA $Wf AS Module.WF;" + notes: "Microflows$GetWorkflowDataAction introduced in 9.10.0 (mendixmodelsdk 4.117.0; ako/mxcli#1787)." + workflow_operation_retry_unpause: + min_version: "9.12.0" + mdl: "WORKFLOW OPERATION RETRY $Wf; WORKFLOW OPERATION UNPAUSE $Wf;" + notes: "Microflows$RetryOperation and UnpauseOperation introduced in 9.12.0, replacing ResumeOperation (9.3.0 to 9.11; mendixmodelsdk 4.117.0; ako/mxcli#1787)." + notify_workflow: + min_version: "10.0.0" + mdl: "$Notified = NOTIFY WORKFLOW $Wf TARGET Module.WF.WaitForApproval;" + notes: "Microflows$NotifyWorkflowAction introduced in 10.0.0, with the wait for notification it resumes (mendixmodelsdk 4.117.0); absent from the 9.24.30 mxbuild (ako/mxcli#1787)." + get_workflows: + min_version: "10.4.0" + mdl: "$Workflows = GET WORKFLOWS FOR $Ctx;" + notes: "Microflows$GetWorkflowsAction introduced in 10.4.0 (mendixmodelsdk 4.117.0); absent from the 9.24.30 mxbuild (ako/mxcli#1787)." + get_workflow_activity_records: + min_version: "10.4.0" + mdl: "$Records = GET WORKFLOW ACTIVITY RECORDS $Wf;" + notes: "Microflows$GetWorkflowActivityRecordsAction introduced in 10.4.0 (mendixmodelsdk 4.117.0); absent from the 9.24.30 mxbuild (ako/mxcli#1787)." event_handlers: min_version: "10.7.0" mdl: "CREATE WORKFLOW … ON WORKFLOW EVENTS (…) MICROFLOW Module.Handler AS '…'" @@ -312,6 +425,34 @@ features: min_version: "11.7.0" mdl: "$Notified = NOTIFY WORKFLOW $Workflow TARGET Module.Workflow.espCancelStart;" notes: "NotifyWorkflowAction.notifyTarget introduced in 11.7.0 (modelsdk gen), replacing Activity. Not a refusal: before 11.7 mxcli writes the Activity form. A notify with no target is CE0166 on 11.6, 11.10 and 11.13." + jump_to: + min_version: "9.0.5" + mdl: "JUMP TO Activity;" + notes: "Workflows$JumpToActivity introduced in 9.0.5 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + call_workflow_activity: + min_version: "9.6.0" + mdl: "CALL WORKFLOW Module.WF; (an activity of a workflow)" + notes: "Workflows$CallWorkflowActivity is 9.0.2, but the ParameterExpression mxcli writes is 9.6.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + overview_page: + min_version: "9.11.0" + mdl: "OVERVIEW PAGE Module.Page" + notes: "Workflow.AdminPage, a Workflows$PageReference, introduced in 9.11.0 (mendixmodelsdk 4.117.0); before, the property was OverviewPage, which mxcli does not write (ako/mxcli#1811)." + annotations: + min_version: "9.15.0" + mdl: "ANNOTATION '…' (on a workflow or an activity)" + notes: "Workflows$Annotation and Workflow.Annotation introduced in 9.15.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + workflow_parameter: + min_version: "9.18.0" + mdl: "PARAMETER $WorkflowContext: Module.Entity" + notes: "Workflows$Parameter is 9.6.0, its Entity 9.10.0 and its Name 9.18.0 (mendixmodelsdk 4.117.0); mxcli writes all three. Measured on stamped 9.x projects: below 9.18 the write refuses every workflow with a parameter (ako/mxcli#1811)." + call_workflow_parameter_mappings: + min_version: "9.18.0" + mdl: "CALL WORKFLOW Module.WF (WorkflowContext = $Expr); (an activity of a workflow)" + notes: "Workflows$WorkflowCallParameterMapping and CallWorkflowActivity.ParameterMappings introduced in 9.18.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + interrupting_notification_event_subprocess: + min_version: "11.10.0" + mdl: "EVENT SUBPROCESS Name ON INTERRUPTING NOTIFICATION { … };" + notes: "Workflows$InterruptingNotificationEventSubProcessStartActivity introduced in 11.10.0 (mendixmodelsdk 4.117.0); the non-interrupting start is 11.8 with event sub-processes themselves (ako/mxcli#1811)." navigation: profiles: @@ -320,6 +461,22 @@ features: min_version: "10.0.0" home_pages: min_version: "10.0.0" + progressive_web_app: + min_version: "9.0.3" + mdl: "CREATE OR MODIFY NAVIGATION Profile PROGRESSIVE WEB APP ( Precaching: true )" + notes: "Navigation$ProgressiveWebAppSettings introduced in 9.0.3 (mendixmodelsdk 4.117.0); `progressive web app off` writes none (ako/mxcli#1811)." + native_home_page: + min_version: "9.4.0" + mdl: "CREATE OR MODIFY NAVIGATION NativePhone HOME PAGE Module.Page" + notes: "Navigation$NativeHomePage and NativeNavigationProfile.NativeHomePage introduced in 9.4.0 (mendixmodelsdk 4.117.0); the native profile itself is older (ako/mxcli#1811)." + menu_item_icons: + min_version: "9.20.0" + mdl: "MENU ITEM 'Home' ( OnClick: …, Icon: Module.Collection.icon )" + notes: "Forms$IconCollectionIcon introduced in 9.20.0 with icon collections (mendixmodelsdk 4.117.0); `Icon:` names an icon collection entry and nothing else (ako/mxcli#1811)." + not_found_page: + min_version: "10.13.0" + mdl: "NOT FOUND PAGE Module.Page" + notes: "Navigation$NotFoundHomePage and NavigationProfile.NotFoundHomepage introduced in 10.13.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." mpr_format: mpr_v2: @@ -331,3 +488,19 @@ features: portable_app: min_version: "11.6.0" notes: "New portable app deployment format" + + modules: + jar_dependencies: + min_version: "10.0.0" + mdl: "ALTER MODULE Module ADD JAR DEPENDENCY (group = '…', artifact = '…', version = '…')" + notes: "Projects$JarDependency and ModuleSettings.JarDependencies introduced in 10.0.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + jar_dependency_exclusions: + min_version: "10.12.0" + mdl: "ALTER MODULE Module SET JAR DEPENDENCY 'group:artifact' ADD EXCLUSION '…'" + notes: "Projects$JarDependencyExclusion introduced in 10.12.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + + scheduled_events: + schedules: + min_version: "9.12.0" + mdl: "CREATE SCHEDULED EVENT Module.Name ( Microflow: …, Repeat: Daily, … )" + notes: "ScheduledEvents$MinuteSchedule … YearWeekdaySchedule, ScheduledEvent.Schedule and OnOverlap introduced in 9.12.0 (mendixmodelsdk 4.117.0); before, an event has only an interval, which mxcli does not write (ako/mxcli#1811)." diff --git a/sdk/versions/mendix-9.yaml b/sdk/versions/mendix-9.yaml index 433c275314..19a8dd4565 100644 --- a/sdk/versions/mendix-9.yaml +++ b/sdk/versions/mendix-9.yaml @@ -35,6 +35,34 @@ features: min_version: "9.7.0" mdl: "SHOW PAGE Module.PageName WITH PARAMS ..." notes: "Pages$PageSettings.parameterMappings introduced in 9.7.0 (mendixmodelsdk StructureVersionInfo)" + list_find_filter_by_expression: + min_version: "9.0.3" + mdl: "$Found = FIND $List WHERE expression; $Kept = FILTER $List WHERE expression;" + notes: "Microflows$FindByExpression and FilterByExpression introduced in 9.0.3 (mendixmodelsdk 4.117.0); `BY Member = value` writes Microflows$Find/Filter, which every version has. The write refuses each type below it (canon.UndeclaredTypeError); check -p names the statement first (ako/mxcli#1811)." + call_external_action: + min_version: "9.19.0" + mdl: "CALL EXTERNAL ACTION Module.Service.Action ();" + notes: "Microflows$CallExternalAction introduced in 9.19.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + list_range: + min_version: "9.24.0" + mdl: "$Page = RANGE $List OFFSET $Offset LIMIT $Amount;" + notes: "Microflows$ListRange introduced in 9.24.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + loop_over_list: + min_version: "9.0.4" + mdl: "LOOP $Item IN $List BEGIN … END LOOP;" + notes: "Microflows$IterableList and LoopedActivity.LoopSource introduced in 9.0.4 (mendixmodelsdk 4.117.0); before, a loop stored its list on the activity itself, which mxcli does not write. The write refuses the loop below it; check -p names the statement first (ako/mxcli#1811)." + while_loop: + min_version: "9.0.4" + mdl: "WHILE condition BEGIN … END WHILE;" + notes: "Microflows$WhileLoopCondition introduced in 9.0.4 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + queued_calls: + min_version: "9.10.0" + mdl: "CALL MICROFLOW Module.Flow() IN QUEUE Module.Queue; CALL JAVA ACTION Module.Action() IN QUEUE Module.Queue;" + notes: "Queues$QueueSettings and the calls' QueueSettings introduced in 9.10.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + execute_database_query: + min_version: "9.24.0" + mdl: "$Rows = EXECUTE DATABASE QUERY Module.Connection.Query;" + notes: "DatabaseConnector$ExecuteDatabaseQueryAction and QueryParameterMapping introduced in 9.24.0 (mendixmodelsdk 4.117.0); mx 9.24.30 loads the action (CE1613 for a query that does not exist, not a load failure). The 10 and 11 rows said 10.6.0, a floor nothing measured (ako/mxcli#1811)." pages: basic: @@ -44,6 +72,18 @@ features: min_version: "9.4.0" mdl: "CREATE PAGE Module.Name (Params: ($P: Module.Entity)) { ... }" notes: "Pages$PageParameter and Page.parameters introduced in 9.4.0 (mendixmodelsdk StructureVersionInfo)" + workflow_page_actions: + min_version: "9.0.2" + mdl: "action: open user task | open workflow | call workflow Module.WF" + notes: "Forms$OpenUserTaskClientAction, OpenWorkflowClientAction and CallWorkflowClientAction introduced in 9.0.2 with workflows (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + complete_task_action: + min_version: "9.19.0" + mdl: "action: complete task 'Outcome'" + notes: "Forms$SetTaskOutcomeClientAction is 9.0.2, but the OutcomeValue mxcli writes is 9.19.0 (mendixmodelsdk 4.117.0); below, the write refuses the page (ako/mxcli#1811)." + snippet_parameters: + min_version: "9.21.0" + mdl: "CREATE SNIPPET Module.Name (Params: ($P: Module.Entity)) { … }" + notes: "Forms$SnippetParameter and Snippet.Parameters introduced in 9.21.0 (mendixmodelsdk 4.117.0). A snippet call writes a mapping only for a parameter the called snippet declares, so it cannot reach an older project (ako/mxcli#1811)." security: module_roles: @@ -55,16 +95,62 @@ features: workflows: basic: - min_version: "9.0.0" + min_version: "9.0.2" mdl: "CREATE WORKFLOW Module.Name ..." + notes: "Workflows$Workflow introduced in 9.0.2 (mendixmodelsdk 4.117.0); the CDN lists mxbuild 9.0.0 and 9.0.1, which have no workflows. The write refuses the type below it (canon.UndeclaredTypeError); create workflow, create or modify workflow and alter workflow are refused by check -p first (ako/mxcli#1809)." parallel_split: min_version: "9.0.0" user_task: - min_version: "9.0.0" + min_version: "9.19.0" + notes: "Workflows$UserTask is 9.0.2, but the outcome value (Workflows$UserTaskOutcome.Value) mxcli writes is 9.19.0, its task page 9.11.0 and its on-created event 9.0.5 (mendixmodelsdk 4.117.0). Measured on stamped 9.x projects: the write refuses a user task below 9.19 (ako/mxcli#1811)." multi_user_task: min_version: "9.22.0" mdl: "MULTI USER TASK Vote 'Vote' PAGE M.TaskPage OUTCOMES 'Yes' { } 'No' { };" notes: "Before 10.12 a Workflows$UserTask whose UserTaskCompletion is a MultiInputCompletion, introduced with consensus and veto in 9.22.0 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." + workflow_microflow_actions: + min_version: "9.0.2" + mdl: "CALL WORKFLOW Module.WF ($Ctx); SET TASK OUTCOME $Task 'Approve'; OPEN USER TASK $Task; OPEN WORKFLOW $Wf;" + notes: "Microflows$WorkflowCallAction, SetTaskOutcomeAction, OpenUserTaskAction and OpenWorkflowAction introduced in 9.0.2 with workflows (mendixmodelsdk 4.117.0). The write refuses each type below it (canon.UndeclaredTypeError); check -p names the statement first (ako/mxcli#1787)." + workflow_operations: + min_version: "9.2.0" + mdl: "WORKFLOW OPERATION ABORT $Wf REASON '…'; WORKFLOW OPERATION RESTART $Wf;" + notes: "Microflows$WorkflowOperationAction, AbortOperation and RestartOperation introduced in 9.2.0 (mendixmodelsdk 4.117.0; ako/mxcli#1787)." + workflow_operation_pause_continue: + min_version: "9.3.0" + mdl: "WORKFLOW OPERATION PAUSE $Wf; WORKFLOW OPERATION CONTINUE $Wf;" + notes: "Microflows$PauseOperation and ContinueOperation introduced in 9.3.0 (mendixmodelsdk 4.117.0; ako/mxcli#1787)." + get_workflow_data: + min_version: "9.10.0" + mdl: "$Data = GET WORKFLOW DATA $Wf AS Module.WF;" + notes: "Microflows$GetWorkflowDataAction introduced in 9.10.0 (mendixmodelsdk 4.117.0; ako/mxcli#1787)." + workflow_operation_retry_unpause: + min_version: "9.12.0" + mdl: "WORKFLOW OPERATION RETRY $Wf; WORKFLOW OPERATION UNPAUSE $Wf;" + notes: "Microflows$RetryOperation and UnpauseOperation introduced in 9.12.0, replacing ResumeOperation (9.3.0 to 9.11; mendixmodelsdk 4.117.0; ako/mxcli#1787)." + jump_to: + min_version: "9.0.5" + mdl: "JUMP TO Activity;" + notes: "Workflows$JumpToActivity introduced in 9.0.5 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + call_workflow_activity: + min_version: "9.6.0" + mdl: "CALL WORKFLOW Module.WF; (an activity of a workflow)" + notes: "Workflows$CallWorkflowActivity is 9.0.2, but the ParameterExpression mxcli writes is 9.6.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + overview_page: + min_version: "9.11.0" + mdl: "OVERVIEW PAGE Module.Page" + notes: "Workflow.AdminPage, a Workflows$PageReference, introduced in 9.11.0 (mendixmodelsdk 4.117.0); before, the property was OverviewPage, which mxcli does not write (ako/mxcli#1811)." + annotations: + min_version: "9.15.0" + mdl: "ANNOTATION '…' (on a workflow or an activity)" + notes: "Workflows$Annotation and Workflow.Annotation introduced in 9.15.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." + workflow_parameter: + min_version: "9.18.0" + mdl: "PARAMETER $WorkflowContext: Module.Entity" + notes: "Workflows$Parameter is 9.6.0, its Entity 9.10.0 and its Name 9.18.0 (mendixmodelsdk 4.117.0); mxcli writes all three. Measured on stamped 9.x projects: below 9.18 the write refuses every workflow with a parameter (ako/mxcli#1811)." + call_workflow_parameter_mappings: + min_version: "9.18.0" + mdl: "CALL WORKFLOW Module.WF (WorkflowContext = $Expr); (an activity of a workflow)" + notes: "Workflows$WorkflowCallParameterMapping and CallWorkflowActivity.ParameterMappings introduced in 9.18.0 (mendixmodelsdk 4.117.0; ako/mxcli#1811)." navigation: profiles: @@ -73,3 +159,27 @@ features: min_version: "9.0.0" home_pages: min_version: "9.0.0" + progressive_web_app: + min_version: "9.0.3" + mdl: "CREATE OR MODIFY NAVIGATION Profile PROGRESSIVE WEB APP ( Precaching: true )" + notes: "Navigation$ProgressiveWebAppSettings introduced in 9.0.3 (mendixmodelsdk 4.117.0); `progressive web app off` writes none (ako/mxcli#1811)." + native_home_page: + min_version: "9.4.0" + mdl: "CREATE OR MODIFY NAVIGATION NativePhone HOME PAGE Module.Page" + notes: "Navigation$NativeHomePage and NativeNavigationProfile.NativeHomePage introduced in 9.4.0 (mendixmodelsdk 4.117.0); the native profile itself is older (ako/mxcli#1811)." + menu_item_icons: + min_version: "9.20.0" + mdl: "MENU ITEM 'Home' ( OnClick: …, Icon: Module.Collection.icon )" + notes: "Forms$IconCollectionIcon introduced in 9.20.0 with icon collections (mendixmodelsdk 4.117.0); `Icon:` names an icon collection entry and nothing else (ako/mxcli#1811)." + + scheduled_events: + schedules: + min_version: "9.12.0" + mdl: "CREATE SCHEDULED EVENT Module.Name ( Microflow: …, Repeat: Daily, … )" + notes: "ScheduledEvents$MinuteSchedule … YearWeekdaySchedule, ScheduledEvent.Schedule and OnOverlap introduced in 9.12.0 (mendixmodelsdk 4.117.0); before, an event has only an interval, which mxcli does not write (ako/mxcli#1811)." + + integration: + business_events: + min_version: "9.24.0" + mdl: "CREATE BUSINESS EVENT SERVICE Module.Name ..." + notes: "BusinessEvents$BusinessEventService and its children introduced in 9.24.0 (mendixmodelsdk 4.117.0). mx 9.24.30 loads a service mxcli writes; its errors there are the content (the Business Events module, an attribute's length). The 10 and 11 rows said 10.0.0 (ako/mxcli#1811)." diff --git a/sdk/versions/registry_test.go b/sdk/versions/registry_test.go index 655326b15e..e90ca882c6 100644 --- a/sdk/versions/registry_test.go +++ b/sdk/versions/registry_test.go @@ -73,9 +73,10 @@ func TestIsAvailable(t *testing.T) { version SemVer want bool }{ - // View entities require 10.18+ - {"domain_model", "view_entities", SemVer{10, 18, 0}, true}, - {"domain_model", "view_entities", SemVer{10, 17, 0}, false}, + // View entities require 10.21+: the source document mxcli writes is + // a 10.21 type (ako/mxcli#1811), though Mendix had them from 10.18. + {"domain_model", "view_entities", SemVer{10, 21, 0}, true}, + {"domain_model", "view_entities", SemVer{10, 20, 0}, false}, {"domain_model", "view_entities", SemVer{11, 0, 0}, true}, // Basic entities available in 9.x+ {"domain_model", "entities", SemVer{9, 0, 0}, true},