Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
425b6f3
test: add unit tests for BundleSchemaOrder to verify import behavior …
dschwartz-ftadvisory Nov 4, 2026
acb10c6
Refactor Python code generation and tests for improved schema handling
dschwartz-ftadvisory Nov 5, 2026
367315c
fix: single value extraction and enum handling
dschwartz-ftadvisory Nov 5, 2026
56110cd
fix: resolve count operations issues
dschwartz-ftadvisory Nov 5, 2026
6979160
fix: resolve circular import errors and improve function import handl…
dschwartz-ftadvisory Nov 5, 2026
68ce676
fix: resolve circular import issues and improve module import handlin…
dschwartz-ftadvisory Nov 5, 2026
490e49d
fix: add round-trip testing for FPML 5.10 samples with detailed error…
dschwartz-ftadvisory Nov 6, 2026
9c7029a
fix: add unit tests for bundled types and improve import handling for…
dschwartz-ftadvisory Nov 6, 2026
bb4ef61
fix: update enum type handling to include namespace prefix in generat…
dschwartz-ftadvisory Nov 6, 2026
3f7db09
fix: enhance enum handling and import resolution in generated Python …
dschwartz-ftadvisory Nov 6, 2026
fe73cb3
fix: add tests for single value filtering and ensure correct handling…
dschwartz-ftadvisory Nov 6, 2026
dc5ecf1
fix: phase 1 emf/ecore removal
dschwartz-ftadvisory Nov 6, 2026
3c58e83
Fix: Phase 2 EMF removal
dschwartz-ftadvisory Nov 6, 2026
ac4bdd3
fix: relax to to-enum processing to match the DSL defined behavior
dschwartz-ftadvisory Nov 6, 2026
3198d7b
fix: match switch cases by type and nested choice option, bind the ma…
dschwartz-ftadvisory Nov 6, 2026
e52504f
fix: implement handling for circular function imports in Python code …
dschwartz-ftadvisory Nov 6, 2026
3b1b70f
fix: Fix NameError in class conditions that use as to narrow to a dat…
dschwartz-ftadvisory Nov 6, 2026
4e01564
fix: remove unused EMF dependency and update test to reflect runtime …
dschwartz-ftadvisory Nov 6, 2026
20f35f8
fix: Attach metadata serializer and validator to bundled fields so @k…
dschwartz-ftadvisory Nov 6, 2026
bb6ae20
fix: Add unit tests and serialization logic for type-level keys in Py…
dschwartz-ftadvisory Nov 6, 2026
c6888af
fix: use the flattened approach for choice serialization and deserial…
dschwartz-ftadvisory Nov 9, 2026
f9f36fb
fix: add _same_number function to compare number and string values fo…
dschwartz-ftadvisory Nov 9, 2026
8f1b4f8
fix: update runtime directory path and enhance error handling in seri…
dschwartz-ftadvisory Nov 10, 2026
504cd4b
fix: add error-handling directory to .gitignore to prevent unwanted t…
dschwartz-ftadvisory Nov 10, 2026
ef0d660
Merge remote-tracking branch 'origin/fix-use-runtime-rune-sorted' int…
dschwartz-ftadvisory Nov 12, 2026
e47bfa2
Merge remote-tracking branch 'origin/fix-use-runtime-rune-sorted' int…
dschwartz-ftadvisory Nov 12, 2026
1022ef4
feat: implement PythonChoiceDispatchProcessor
dschwartz-ftadvisory Nov 12, 2026
87647e4
fix: prune redundant choice dispatch entries
plamen-neykov Nov 13, 2026
a699427
Merge remote-tracking branch 'origin/main' into fix-circular-import-e…
dschwartz-ftadvisory Nov 13, 2026
a2e199c
fix: comment out RUNE_RUNTIME_DIR assignment in setup script
dschwartz-ftadvisory Nov 13, 2026
6ce8fba
fix: apply prune redundant choice dispatch entries (#334), missing fr…
dschwartz-ftadvisory Nov 13, 2026
873ebc8
fix: document known Maven build warnings in README and POM file
dschwartz-ftadvisory Nov 13, 2026
79d943a
feat: add serialization tests for various error handling scenarios
dschwartz-ftadvisory Nov 13, 2026
2a23a83
chore: remove unused build/ files and track the error-handling test m…
dschwartz-ftadvisory Nov 13, 2026
8c80444
fix: clean up .gitignore by removing unnecessary entries
dschwartz-ftadvisory Nov 13, 2026
edde74d
fix: run the license header check before packaging and add missing he…
dschwartz-ftadvisory Nov 13, 2026
003be74
feat: add CDM analysis for both Python and Java comparing results
dschwartz-ftadvisory Nov 13, 2026
2107b59
feat: implement handling for attribute name clashes with built-in typ…
dschwartz-ftadvisory Nov 13, 2026
37494f5
fix: exclude ChoiceDeepNested from round-trip tests due to model conf…
dschwartz-ftadvisory Nov 13, 2026
7b6199a
fix: key enum choice options by Rune FQN
dschwartz-ftadvisory Nov 13, 2026
6fbc3db
chore: clean up checkstyle warnings
dschwartz-ftadvisory Nov 13, 2026
8d2babd
fix: require runtime version 3.1.0 or greater
dschwartz-ftadvisory Nov 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 0 additions & 8 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,3 @@
# xtend
**/src-gen/
**/xtend-gen/
# Compiled Java class files
*.class
# Compiled Python bytecode
Expand Down Expand Up @@ -41,8 +38,6 @@ Thumbs.db
*.wmv
# Jupyter Notebook
.ipynb_checkpoints
# some other generated files
*._trace
# Python
__pycache__
.pyenv/
Expand All @@ -53,12 +48,9 @@ __pycache__
**/.checkstyle
# project
**/*.egg-info
**/rune.runtime-*.*.*-py3-none-any.whl
**/test_helper-0.0.0-py3-none-any.whl
**/python-test/serialization-tests/rune-common
**/python-test/cdm-tests/rosetta/
src/generated/
*.todo
/build/
test/***
**/python-test/serialization-tests/error-handling
77 changes: 0 additions & 77 deletions build/example_function/AddBusinessDays.py

This file was deleted.

25 changes: 0 additions & 25 deletions build/github_actions_cleanup.sh

This file was deleted.

12 changes: 12 additions & 0 deletions docs/BUILD_AND_TEST.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,18 @@ mvn clean package

The JUnit tests compare generated Python output against expected results. All tests should pass.

### Known Build Warnings

These warnings come from upstream dependencies and don't affect the build. They are left visible rather than suppressed.

- **`jvnet-nexus-releases` (Maven 3.10 and later):**

```
[WARNING] Could not transfer metadata /.meta/prefixes.txt from/to jvnet-nexus-releases (https://maven.java.net/content/repositories/releases/): (certificate_expired) PKIX path validation failed
```

`maven-checkstyle-plugin` 3.6.0 depends on `javax.xml.bind:jaxb-api` 2.3.1, whose parent POM declares the retired `maven.java.net` repository. Maven 3.10 checks that repository and reports the failure; every file still resolves from Maven Central. The dependency is removed upstream for `maven-checkstyle-plugin` 3.6.1 ([apache/maven-checkstyle-plugin#736](https://github.com/apache/maven-checkstyle-plugin/issues/736)), which is not yet released. Tracked in [#335](https://github.com/finos/rune-python-generator/issues/335).

---

## Run Python Unit Tests
Expand Down
18 changes: 13 additions & 5 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -269,10 +269,6 @@
<version>1.5.3</version>
</dependency>
<!-- extra dependencies -->
<dependency>
<groupId>org.eclipse.emf</groupId>
<artifactId>org.eclipse.emf.codegen.ecore</artifactId>
</dependency>
<dependency>
<groupId>com.google.inject</groupId>
<artifactId>guice</artifactId>
Expand Down Expand Up @@ -377,6 +373,10 @@
</includes>
<excludes>
<exclude>src/generated/**</exclude>
<!-- Local virtual environments and caches; not part of the repository -->
<exclude>**/.pyenv/**</exclude>
<exclude>**/.profile-pyenv/**</exclude>
<exclude>**/__pycache__/**</exclude>
</excludes>
</licenseSet>
</licenseSets>
Expand All @@ -385,7 +385,9 @@
<executions>
<execution>
<id>check-license-headers</id>
<phase>verify</phase>
<!-- Run before package: maven-shade-plugin's dependency-reduced POM moves the
project base directory to target/, after which no includes match (#337) -->
<phase>validate</phase>
<goals>
<goal>check</goal>
</goals>
Expand Down Expand Up @@ -419,6 +421,12 @@
</execution>
</executions>
</plugin>
<!--
Known warning with Maven 3.10+: "Could not transfer metadata /.meta/prefixes.txt from/to
jvnet-nexus-releases". This plugin's jaxb-api 2.3.1 dependency inherits that retired
repository; the build is unaffected. Fixed upstream for 3.6.1, not yet released:
https://github.com/apache/maven-checkstyle-plugin/issues/736. Tracked in https://github.com/finos/rune-python-generator/issues/335.
-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
Expand Down
59 changes: 59 additions & 0 deletions python-test/cdm-tests/conftest.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
#
# Copyright (c) 2023-2026 CLOUDRISK Limited and FT Advisory LLC
# SPDX-License-Identifier: Apache-2.0
#

'''Collects round-trip results and prints a summary grouped by error type.'''
from collections import Counter, defaultdict

import pytest

_ROUNDTRIP_KEY = pytest.StashKey[dict]()


def pytest_configure(config):
config.stash[_ROUNDTRIP_KEY] = {}


@pytest.fixture
def roundtrip_results(request) -> dict:
'''Maps each sample to its problems, as (error type, example) pairs: the
field for a difference, or the sample for a read or write failure. An empty
list means the sample came back unchanged.'''
return request.config.stash[_ROUNDTRIP_KEY]


def pytest_terminal_summary(terminalreporter, exitstatus, config):
'''Prints one row per error type, with the number of samples it affects.'''
del exitstatus # unused
results = config.stash.get(_ROUNDTRIP_KEY, {})
if not results:
return

files = defaultdict(set)
occurrences = Counter()
details = defaultdict(Counter)
for sample, problems in results.items():
for category, detail in problems:
files[category].add(sample)
occurrences[category] += 1
details[category][detail] += 1

unchanged = sum(1 for problems in results.values() if not problems)
write = terminalreporter.write_line
terminalreporter.write_sep('=', 'FpML 5.10 round-trip summary')
write(f'Samples: {len(results)} unchanged: {unchanged} '
f'with problems: {len(results) - unchanged}')
if not files:
return

rows = [(category, str(len(files[category])), str(occurrences[category]),
details[category].most_common(1)[0][0])
for category in sorted(files, key=lambda c: (-len(files[c]), c))]
headers = ('Error type', 'Samples', 'Occurrences', 'Most common field or example sample')
widths = [max(len(row[i]) for row in rows + [headers]) for i in range(3)]
write('')
write(' '.join(h.ljust(w) for h, w in zip(headers, widths)) + ' ' + headers[3])
write(' '.join('-' * w for w in widths) + ' ' + '-' * len(headers[3]))
for row in rows:
write(' '.join(cell.ljust(w) for cell, w in zip(row, widths)) + ' ' + row[3])
56 changes: 56 additions & 0 deletions python-test/cdm-tests/java-validation/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# CDM ingestion samples: Java validation

Reads, round-trips and validates CDM's FpML ingestion samples with CDM's own Java code, to produce a Java baseline for comparison with the generated Python.

The samples are the ones listed in CDM's test-pack configs, `rosetta-source/src/main/resources/ingest/config/test-pack-*.json`. They cover FpML 5.10, 5.12 and 5.13, trade states and workflow steps, including the `incomplete` and `invalid` packs.

Run it by hand; it isn't part of the build or CI. It builds CDM Java from source, which takes a few minutes and needs network access the first time.

## Requirements

- Java 21 and Maven
- Python 3.11+
- A local git clone of [finos/common-domain-model](https://github.com/finos/common-domain-model)

## Run

From the project root:

```sh
python-test/cdm-tests/java-validation/run_java_validation.sh <cdm-repo> <git-ref> <work-dir>
```

- `<cdm-repo>`: path to the CDM clone.
- `<git-ref>`: the CDM commit, branch or tag to test, for example `upstream/master`. Fetch it first.
- `<work-dir>`: a directory **outside this repository**. The script builds CDM there and writes the results to `<work-dir>/results/`.

For each sample, `ValidateSamples`:

1. reads the sample's output JSON with `RuneJsonObjectMapper`, as a `TradeState`, or a `WorkflowStep` for the `workflow-step` packs;
2. writes it back and compares it with the original, ignoring `@version` and comparing numbers by value;
3. resolves its references with `ReferenceResolverProcessStep`;
4. validates it with `RosettaTypeValidator`.

## Results

| File | One row per | Columns |
|---|---|---|
| `all-samples.tsv` | sample | test pack, output path, `modelValidationFailures` and `runtimeError` from the config |
| `java-summary.tsv` | sample | test pack, sample, config failure count, Java failure count, round trip (`same` or `DIFF`), read error |
| `java-failures.tsv` | validation failure | test pack, sample, type (`CARDINALITY` or `DATA_RULE`), object, rule, path, reason |

A sample with a read error has no failure count or round-trip result. An empty output file in CDM shows up as a read error.

## Other tools

- **`undefined_keys.py <sample.json> ...`** lists the references in a sample whose key isn't defined anywhere in the same file. Java leaves such a reference's value empty without reporting a failure, so a sample can pass Java validation with data missing.
- **`java/ShowReferences.java`** prints what Java holds, after resolving references, for the `dateRelativeTo` and business center references of a trade state's first interest rate payout. Compile and run it with the classpath `run_java_validation.sh` builds:

```sh
CP="<work-dir>/cdm-java/rosetta-source/target/classes:$(cat <work-dir>/cp.txt)"
java -cp "<work-dir>/classes:$CP" ShowReferences <sample.json>
```

## Comparing with Python

`python-test/cdm-tests/run_fpml510_tests.sh` and `test_fpml510_samples.py` read and round-trip the same samples with the generated Python. Use the same CDM commit for both, then compare per sample: whether it reads, whether the round trip is unchanged, and the validation failures.
22 changes: 22 additions & 0 deletions python-test/cdm-tests/java-validation/java/ShowReferences.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
/*
* Copyright (c) 2023-2026 CLOUDRISK Limited and FT Advisory LLC
* SPDX-License-Identifier: Apache-2.0
*/
import com.google.inject.*;
import com.regnosys.rosetta.common.hashing.ReferenceResolverProcessStep;
import com.rosetta.model.lib.RosettaModelObject;
import org.finos.cdm.CdmRuntimeModule;
import org.finos.cdm.reference.CdmReferenceConfig;
import org.finos.rune.mapper.RuneJsonObjectMapper;
import java.nio.file.*;
public class ShowReferences { public static void main(String[] a) throws Exception {
Guice.createInjector(new CdmRuntimeModule());
cdm.event.common.TradeState m=new RuneJsonObjectMapper().readValue(Files.readString(Paths.get(a[0])),cdm.event.common.TradeState.class);
RosettaModelObject b=m.toBuilder(); new ReferenceResolverProcessStep(CdmReferenceConfig.get()).runProcessStep(b.getType(),b);
var t=(cdm.event.common.TradeState)b.build();
var p=t.getTrade().getProduct().getEconomicTerms().getPayout().get(0).getInterestRatePayout();
var r=p.getResetDates().getFixingDates().getDateRelativeTo();
System.out.println("dateRelativeTo: "+r+" | value="+r.getValue());
var bc=p.getCalculationPeriodDates().getCalculationPeriodDatesAdjustments().getBusinessCenters();
System.out.println("businessCenters: reference="+bc.getBusinessCentersReference()+" value="+bc.getBusinessCentersReference().getValue()+" businessCenter="+bc.getBusinessCenter());
}}
60 changes: 60 additions & 0 deletions python-test/cdm-tests/java-validation/java/ValidateSamples.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
/*
* Copyright (c) 2023-2026 CLOUDRISK Limited and FT Advisory LLC
* SPDX-License-Identifier: Apache-2.0
*/
import com.fasterxml.jackson.databind.*;
import com.google.inject.*;
import com.regnosys.rosetta.common.hashing.ReferenceResolverProcessStep;
import com.regnosys.rosetta.common.validation.*;
import com.rosetta.model.lib.RosettaModelObject;
import com.rosetta.model.lib.validation.ValidationResult;
import org.finos.cdm.CdmRuntimeModule;
import org.finos.cdm.reference.CdmReferenceConfig;
import org.finos.rune.mapper.RuneJsonObjectMapper;
import java.nio.file.*; import java.io.*; import java.util.*;
/** args: resourcesDir samples.tsv summary.tsv failures.tsv */
public class ValidateSamples {
static final Comparator<JsonNode> NUM = (x, y) -> {
if (x.equals(y)) return 0;
if (x.isNumber() && y.isNumber()) return x.decimalValue().compareTo(y.decimalValue());
return 1;
};
public static void main(String[] a) throws Exception {
Path base = Paths.get(a[0]);
Injector inj = Guice.createInjector(new CdmRuntimeModule());
RosettaTypeValidator val = new RosettaTypeValidator(); inj.injectMembers(val);
ObjectMapper mapper = new RuneJsonObjectMapper();
ObjectMapper plain = new ObjectMapper();
try (PrintWriter sum = new PrintWriter(a[2]); PrintWriter fail = new PrintWriter(a[3])) {
sum.println("pack\tsample\texpected\tjava_failures\troundtrip\terror");
for (String line : Files.readAllLines(Paths.get(a[1]))) {
String[] t = line.split("\t");
String pack = t[0], f = t[1], expected = t[2];
Class<? extends RosettaModelObject> c = pack.startsWith("workflow-step")
? cdm.event.workflow.WorkflowStep.class : cdm.event.common.TradeState.class;
String rt = "", fails = "", err = "";
try {
String json = Files.readString(base.resolve(f));
RosettaModelObject m = mapper.readValue(json, c);
JsonNode in = plain.readTree(json), out = plain.readTree(mapper.writeValueAsString(m));
((com.fasterxml.jackson.databind.node.ObjectNode) in).remove("@version");
((com.fasterxml.jackson.databind.node.ObjectNode) out).remove("@version");
rt = in.equals(NUM, out) ? "same" : "DIFF";
RosettaModelObject b = m.toBuilder();
new ReferenceResolverProcessStep(CdmReferenceConfig.get()).runProcessStep(b.getType(), b);
ValidationReport r = val.runProcessStep(c, b.build());
List<ValidationResult<?>> vf = r.validationFailures();
fails = String.valueOf(vf.size());
for (ValidationResult<?> v : vf)
fail.println(String.join("\t", pack, f, String.valueOf(v.getValidationType()), v.getModelObjectName(),
v.getName(), String.valueOf(v.getPath()), v.getFailureReason().orElse("").replaceAll("\\s+", " ")));
} catch (Exception e) {
err = e.getClass().getSimpleName() + ": " + String.valueOf(e.getMessage()).replaceAll("\\s+", " ");
if (err.length() > 300) err = err.substring(0, 300);
}
sum.println(String.join("\t", pack, f, expected, fails, rt, err));
sum.flush();
}
}
}
}
Loading
Loading