Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 47 additions & 52 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,95 +1,90 @@
[![FINOS - Incubating](https://cdn.jsdelivr.net/gh/finos/contrib-toolbox@master/images/badge-incubating.svg)](https://community.finos.org/docs/governance/Software-Projects/stages/incubating)
[![Run Unit Tests](https://github.com/regnosys/rune-python-runtime/actions/workflows/run-tests.yml/badge.svg?branch=main)](https://github.com/regnosys/rune-python-runtime/actions/workflows/run-tests.yml)
[![OpenSSF Best Practices](https://www.bestpractices.dev/projects/10733/badge)](https://www.bestpractices.dev/projects/10733)[![Python CI](https://github.com/regnosys/rune-python-runtime/actions/workflows/cve-scanning-python.yml/badge.svg)](https://github.com/regnosys/rune-python-runtime/actions/workflows/cve-scanning-python.yml)
[![Run Unit Tests](https://github.com/finos/rune-python-runtime/actions/workflows/run-tests.yml/badge.svg?branch=main)](https://github.com/finos/rune-python-runtime/actions/workflows/run-tests.yml)
[![OpenSSF Best Practices](https://www.bestpractices.dev/projects/10733/badge)](https://www.bestpractices.dev/projects/10733)
[![Python CI](https://github.com/finos/rune-python-runtime/actions/workflows/cve-scanning-python.yml/badge.svg)](https://github.com/finos/rune-python-runtime/actions/workflows/cve-scanning-python.yml)
<img align="right" width="15%" src="https://www.finos.org/hubfs/FINOS/finos-logo/FINOS_Icon_Wordmark_Name_RGB_horizontal.png">

# Rune Python Runtime

*Rune Python Runtime* - the runtime supports and is an integral part of Python code generated from a [Rune DSL](https://github.com/finos/rune-dsl) defined model. Rune DSL is a Domain-Specific Language used to model Financial Markets activities including the [Common Domain Model (CDM)](https://github.com/finos/common-domain-model).
The Rune Python Runtime provides shared support for Python models and functions
generated from [Rune DSL](https://github.com/finos/rune-dsl), including the
[Common Domain Model (CDM)](https://github.com/finos/common-domain-model).

The Rune Python Runtime is used in collaboration with the [Rune Python Code Generator](https://github.com/REGnosys/rune-python-generator) to translate a Rune DSL model into a fully usable Python package.
The [Rune Python Code Generator](https://github.com/finos/rune-python-generator)
translates Rune models into Python packages. This runtime supplies their shared
validation, JSON serialization, metadata, reference handling, and function helpers.

## Installation

The runtime is not generally installed on a stand alone basis but rather comes as part of a generated package such as CDM.
However, pip install `rune-runtime` would install it from PyPi.

## Development setup
### Setup for developers
This guide is meant for everyone who wants to contribute to the Rune Python Runtime and needs to get things up and running.
### 1. Clone the Project
Start by cloning the project: `git clone https://github.com/regnosys/rune-python-generator`

### 2. Setup the Development Environment
Use [dev_clean_setup.sh](https://github.com/Cloudrisk/rune-python-runtime/blob/main/dev_clean_setup.sh) to setup a development environment.
Python 3.11 or later is required. The runtime is normally installed as a dependency
of a generated model package. To install it directly:

```sh
./dev_clean_setup.sh
python -m pip install rune-runtime
```

This script installs all dev dependencies and registers the pre-commit hooks, including the license-header check. Every `.py` file under `src/` and `test/` must carry the Apache 2.0 SPDX header:
## Design

```python
# Copyright (c) 2023-2026 CLOUDRISK Limited and FT Advisory LLC
# SPDX-License-Identifier: Apache-2.0
```
Start with the [runtime design overview](https://github.com/finos/rune-python-runtime/blob/main/docs/design.md) for the generator/runtime
boundary, the main components, and links to the detailed design documents.

## Development setup

To add missing headers to all files at any time, run:
With Python 3.11 or later and Bash available (for example, Git Bash on Windows):

```sh
src/license/apply_headers.sh
git clone https://github.com/finos/rune-python-runtime.git
cd rune-python-runtime
./dev_clean_setup.sh
```

### 3. Build
Use [build_wheel.sh](https://github.com/Cloudrisk/rune-python-runtime/blob/main/build_wheel.sh) to build the package
The [setup script](https://github.com/finos/rune-python-runtime/blob/main/dev_clean_setup.sh) creates `.pydevenv`, installs development
dependencies, and registers pre-commit hooks, including license-header checks.

Build a wheel, then run the tests against it in an isolated environment:

```sh
./build_wheel.sh
```
To run the unit tests:
```sh
test/run_runtime_tests.sh
./test/run_runtime_tests.sh
```

## Roadmap
See [build_wheel.sh](https://github.com/finos/rune-python-runtime/blob/main/build_wheel.sh) and the
[test runner](https://github.com/finos/rune-python-runtime/blob/main/test/run_runtime_tests.sh) for details.

The Roadmap will be aligned to the Rune DSL and [CDM](https://github.com/finos/common-domain-model/blob/master/ROADMAP.md) roadmaps.
## Roadmap

In addition, the intention is to make future releases available at [PyPi](https://pypi.org)
The roadmap is aligned with Rune DSL and the
[CDM roadmap](https://github.com/finos/common-domain-model/blob/master/ROADMAP.md).

## Contributors

- [CLOUDRISK Limited](https://www.cloudrisk.uk), email: info@cloudrisk.co.uk
- [FT Advisory LLC](https://www.ftadvisory.co), email: info@ftadvisory.co
- [TradeHeader SL](https://www.tradeheader.com), email: info@tradeheader.com

## Contributing
For any questions, bugs or feature requests please open an [issue](https://github.com/regnosys/rune-python-runtime/issues)
For anything else please send an email to {project mailing list}.

To submit a contribution:
1. Fork it (<https://github.com/regnosys/rune-python-runtime/fork>)
2. Create your feature branch (`git checkout -b feature/fooBar`)
3. Read our [contribution guidelines](.github/CONTRIBUTING.md) and [Community Code of Conduct](https://www.finos.org/code-of-conduct)
4. Commit your changes (`git commit -am 'Add some fooBar'`)
5. Push to the branch (`git push origin feature/fooBar`)
6. Create a new Pull Request

_NOTE:_ Commits and pull requests to FINOS repositories will only be accepted from those contributors with an active, executed Individual Contributor License Agreement (ICLA) with FINOS OR who are covered under an existing and active Corporate Contribution License Agreement (CCLA) executed with FINOS. Commits from individuals not covered under an ICLA or CCLA will be flagged and blocked by the FINOS Clabot tool (or [EasyCLA](https://community.finos.org/docs/governance/Software-Projects/easycla)). Please note that some CCLAs require individuals/employees to be explicitly named on the CCLA.

*Need an ICLA? Unsure if you are covered under an existing CCLA? Email [help@finos.org](mailto:help@finos.org)*

## Get in touch with the Rune Python Runtime Team
For questions, bug reports, or feature requests, open a
[GitHub issue](https://github.com/finos/rune-python-runtime/issues).

Get in touch with the Rune team by creating a [GitHub issue](https://github.com/REGnosys/rune-python-runtime/issues/new) and labelling it with "help wanted".
Read the [contribution guidelines](https://github.com/finos/rune-python-runtime/blob/main/CONTRIBUTING.md) for the contribution process
and FINOS contributor license agreement requirements, and follow the
[Community Code of Conduct](https://github.com/finos/rune-python-runtime/blob/main/.github/CODE_OF_CONDUCT.md).
To contribute, [fork the runtime](https://github.com/finos/rune-python-runtime/fork),
create a branch, and submit a pull request.

We encourage the community to get in touch via the [FINOS Slack](https://www.finos.org/blog/finos-announces-new-community-slack).
You can also get in touch through
[FINOS Slack](https://www.finos.org/blog/finos-announces-new-community-slack).

## Governance

This project implements https://community.finos.org/docs/governance/#open-source-software-projects
This project follows the
[FINOS governance framework](https://community.finos.org/docs/governance/#open-source-software-projects).

## License

Copyright 2023-2025 CLOUDRISK Limited and FT Advisory LLC
Copyright 2023-2026 CLOUDRISK Limited and FT Advisory LLC

Distributed under the [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0).
Distributed under the [Apache License, Version 2.0](https://github.com/finos/rune-python-runtime/blob/main/LICENSE).

SPDX-License-Identifier: [Apache-2.0](https://spdx.org/licenses/Apache-2.0)
148 changes: 148 additions & 0 deletions docs/choice-serialization.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
# Choice serialization

## Principles

1. Python objects retain their choice option fields. JSON flattens nested choices
to the selected leaf, preserving required metadata and reference wrappers.
2. Discriminators use Rune identifiers: data types and enums use their Rune FQN;
primitive options use their option name. Python package prefixes and Java
implementation class names are excluded.
3. The generator supplies `_RUNE_CHOICE` and `_CHOICE_DISPATCH`. Runtime dispatch
follows that table without searching the complete choice tree.
4. The DSL prohibits metadata on choice types. Metadata on choice-valued fields
and on individual options remains valid and must survive serialization.
5. References remain reference occurrences, preserving their keys and targets.
Flattening must respect the distinction between a reference to a choice and
a reference to its selected option.
6. JSON readers require the flat format. Native Python constructors and builders
continue to use option field names. Ordinary types with `one-of` conditions
retain their existing representation.

## Wire format

Examples omit the document's `@model` and `@version` envelope fields.

| Selected option | JSON choice value |
| --- | --- |
| Data type | `{"@type": "example.A", "value": "foo"}` |
| Primitive | `{"@type": "string", "@data": "foo"}` |
| Enum | `{"@type": "example.Currency", "@data": "USD"}` |
| Primitive with field metadata | `{"@type": "string", "@scheme": "urn:test", "@data": "foo"}` |

Metadata does not turn a primitive discriminator into a Java class name such as
`java.lang.String`. Enums use their Rune FQN with or without metadata. Their
Python field names and annotations remain unchanged.

**Java interoperability check:** the [Java enum example](examples/java-enum-choice/README.md)
records Rune Common 12.29.0 emitting an option name for a plain enum and a Rune FQN
for a metadata-bearing enum. That differs from the uniform enum-FQN generator
rule above and requires alignment with the intended interoperability contract.
The [atomic-choice example](examples/java-atomic-choices/README.md) records Java's
numeric and temporal representations and its unsupported `pattern` option.

## Generator contract

Each generated choice declares a marker and a discriminator-to-field map:

```python
_RUNE_CHOICE: ClassVar[bool] = True
_CHOICE_DISPATCH: ClassVar[dict[str, str]] = {
"example.A": "ChoiceAB",
"example.B": "ChoiceAB",
"example.C": "ChoiceCD",
"example.D": "ChoiceCD",
}
```

Values are immediate Python field names, including any name mangling. Each nested
choice repeats the lookup at its own level. For example, a deeper path
`Outer.ChoiceABCD.ChoiceAB.A` is reconstructed one field at a time; the table
does not store that complete path.

The generator strips type aliases, propagates descendant discriminators, excludes
Python namespace prefixes, and rejects ambiguous routes to the same key. It adds
a nested choice's own FQN only when metadata on that option requires a wrapper or
reference discriminator. Plain nested choices contribute their leaves only.

Discriminator-based reads use the map. Primitive and enum writes reverse it to obtain
`@type`; data writes use the declared class's `_fqrtn()`. Nested writes forward
the selected leaf representation. Generated enums currently have no `_fqrtn()`.
The generator therefore owns the correctness of their dispatch keys.

## Metadata and references

Metadata permissions are additive: field permissions union type permissions.
For a valid generated choice, the type contribution is empty, so permitted
metadata comes from its containing field. The inherited `serialise_meta` handles
it; no choice-specific metadata blacklist is needed. Manually bypassing metadata
validation is outside this contract.

A choice-valued field with location metadata retains a wrapper:

```json
{"@key:scoped": "c1", "@data": {"@type": "example.A", "value": "foo"}}
```

When that field is itself a selected choice option, the enclosing choice adds
the option's own type:

```json
{"@type": "example.Inner", "@key:scoped": "c1", "@data": {"@type": "example.A", "value": "foo"}}
```

Likewise, `{"@ref:scoped": "c1"}` references the whole choice, while
`{"@type": "example.Inner", "@ref:scoped": "c1"}` selects a referenced option.
Keys inside selected leaves remain available to graph reference resolution.
Lists retain reference occurrence metadata, and COW proxies retain their normal
behavior; see [Object identity and references](object-identity.md).

A standalone choice has no containing-field metadata. The root serializer omits
any such metadata retained on a reused choice instance, while preserving metadata
on its selected leaf or option, including a primitive's `@scheme`.

## Runtime flow and entry points

- **Read:** unwrap field metadata, dispatch `@type` into native option fields,
validate those fields, then initialize containment parents, keys and references.
- **Write:** serialize fields, substitute reference tokens, flatten the selected
option, then merge the document envelope.

The shared Pydantic hooks are `_deserialize_rune_model` and
`_serialize_rune_model`. Choice handling lives in `_choice_to_fields` and
`_flatten_choice`; field wrappers are handled by `ComplexTypeMetaDataMixin`.

`rune_deserialize`, `model_validate_json` and `TypeAdapter.validate_json` enter
wire-validation mode. Constructors, ordinary `model_validate` calls and
`ObjectBuilder` accept native field dictionaries. A scoped context flag, the model
metaclass and the builder's explicit native scope keep constructors and builders
working inside JSON-validation default factories.

Inside a graph, the declared field type supplies the enclosing choice class.
For standalone round trips, call `KnownChoice.rune_deserialize(payload)`. The
generic `BaseDataClass` reader cannot infer the enclosing choice from a leaf tag:
a data FQN identifies the leaf class; primitive and enum tags require choice
context. `rune_serialize()` adds the normal document envelope in either case.

## Errors and maintenance

- Missing `_CHOICE_DISPATCH`: `_choice_dispatch()` raises `RuntimeError` with
regeneration guidance on read and write. An empty table is distinct from a
missing table. The lookup is deliberately uncached.
- Wrong discriminator or wrapper Python type: `TypeError`. Missing or unsupported
wire values: `ValueError`, which Pydantic wraps as `ValidationError` when reading.
- Serializer failures are wrapped in `PydanticSerializationError`. Multiple
selected options and data values whose runtime type differs from the declared
option class are rejected on write. Rune constraint validation remains separate
from a bare `model_dump()` call.
- Option-class inspection is cached; `model_rebuild()` clears that cache.
- Regenerate older models to obtain dispatch tables. Older JSON using nested
option field names must be regenerated into the flat format.

Implementation: [base_data_class.py](../src/rune/runtime/base_data_class.py),
[metadata.py](../src/rune/runtime/metadata.py), and
[object_builder.py](../src/rune/runtime/object_builder.py). Generator counterpart:
`PythonChoiceDispatchProcessor` in `rune-python-generator`.

Regression coverage: [test_choice_serialization.py](../test/test_choice_serialization.py)
covers nesting, root and field use, metadata, references and lists, builders, COW,
filtering, aliases, rebuilding, and error diagnostics.
Loading
Loading