diff --git a/README.md b/README.md index 754fc67..54d1030 100644 --- a/README.md +++ b/README.md @@ -1,95 +1,90 @@ [![FINOS - Incubating](https://cdn.jsdelivr.net/gh/finos/contrib-toolbox@master/images/badge-incubating.svg)](https://community.finos.org/docs/governance/Software-Projects/stages/incubating) -[![Run Unit Tests](https://github.com/regnosys/rune-python-runtime/actions/workflows/run-tests.yml/badge.svg?branch=main)](https://github.com/regnosys/rune-python-runtime/actions/workflows/run-tests.yml) -[![OpenSSF Best Practices](https://www.bestpractices.dev/projects/10733/badge)](https://www.bestpractices.dev/projects/10733)[![Python CI](https://github.com/regnosys/rune-python-runtime/actions/workflows/cve-scanning-python.yml/badge.svg)](https://github.com/regnosys/rune-python-runtime/actions/workflows/cve-scanning-python.yml) +[![Run Unit Tests](https://github.com/finos/rune-python-runtime/actions/workflows/run-tests.yml/badge.svg?branch=main)](https://github.com/finos/rune-python-runtime/actions/workflows/run-tests.yml) +[![OpenSSF Best Practices](https://www.bestpractices.dev/projects/10733/badge)](https://www.bestpractices.dev/projects/10733) +[![Python CI](https://github.com/finos/rune-python-runtime/actions/workflows/cve-scanning-python.yml/badge.svg)](https://github.com/finos/rune-python-runtime/actions/workflows/cve-scanning-python.yml) + # Rune Python Runtime -*Rune Python Runtime* - the runtime supports and is an integral part of Python code generated from a [Rune DSL](https://github.com/finos/rune-dsl) defined model. Rune DSL is a Domain-Specific Language used to model Financial Markets activities including the [Common Domain Model (CDM)](https://github.com/finos/common-domain-model). +The Rune Python Runtime provides shared support for Python models and functions +generated from [Rune DSL](https://github.com/finos/rune-dsl), including the +[Common Domain Model (CDM)](https://github.com/finos/common-domain-model). -The Rune Python Runtime is used in collaboration with the [Rune Python Code Generator](https://github.com/REGnosys/rune-python-generator) to translate a Rune DSL model into a fully usable Python package. +The [Rune Python Code Generator](https://github.com/finos/rune-python-generator) +translates Rune models into Python packages. This runtime supplies their shared +validation, JSON serialization, metadata, reference handling, and function helpers. ## Installation -The runtime is not generally installed on a stand alone basis but rather comes as part of a generated package such as CDM. -However, pip install `rune-runtime` would install it from PyPi. - -## Development setup -### Setup for developers -This guide is meant for everyone who wants to contribute to the Rune Python Runtime and needs to get things up and running. -### 1. Clone the Project -Start by cloning the project: `git clone https://github.com/regnosys/rune-python-generator` - -### 2. Setup the Development Environment -Use [dev_clean_setup.sh](https://github.com/Cloudrisk/rune-python-runtime/blob/main/dev_clean_setup.sh) to setup a development environment. +Python 3.11 or later is required. The runtime is normally installed as a dependency +of a generated model package. To install it directly: ```sh -./dev_clean_setup.sh +python -m pip install rune-runtime ``` -This script installs all dev dependencies and registers the pre-commit hooks, including the license-header check. Every `.py` file under `src/` and `test/` must carry the Apache 2.0 SPDX header: +## Design -```python -# Copyright (c) 2023-2026 CLOUDRISK Limited and FT Advisory LLC -# SPDX-License-Identifier: Apache-2.0 -``` +Start with the [runtime design overview](https://github.com/finos/rune-python-runtime/blob/main/docs/design.md) for the generator/runtime +boundary, the main components, and links to the detailed design documents. + +## Development setup -To add missing headers to all files at any time, run: +With Python 3.11 or later and Bash available (for example, Git Bash on Windows): ```sh -src/license/apply_headers.sh +git clone https://github.com/finos/rune-python-runtime.git +cd rune-python-runtime +./dev_clean_setup.sh ``` -### 3. Build -Use [build_wheel.sh](https://github.com/Cloudrisk/rune-python-runtime/blob/main/build_wheel.sh) to build the package +The [setup script](https://github.com/finos/rune-python-runtime/blob/main/dev_clean_setup.sh) creates `.pydevenv`, installs development +dependencies, and registers pre-commit hooks, including license-header checks. + +Build a wheel, then run the tests against it in an isolated environment: + ```sh ./build_wheel.sh -``` -To run the unit tests: -```sh -test/run_runtime_tests.sh +./test/run_runtime_tests.sh ``` -## Roadmap +See [build_wheel.sh](https://github.com/finos/rune-python-runtime/blob/main/build_wheel.sh) and the +[test runner](https://github.com/finos/rune-python-runtime/blob/main/test/run_runtime_tests.sh) for details. -The Roadmap will be aligned to the Rune DSL and [CDM](https://github.com/finos/common-domain-model/blob/master/ROADMAP.md) roadmaps. +## Roadmap -In addition, the intention is to make future releases available at [PyPi](https://pypi.org) +The roadmap is aligned with Rune DSL and the +[CDM roadmap](https://github.com/finos/common-domain-model/blob/master/ROADMAP.md). ## Contributors + - [CLOUDRISK Limited](https://www.cloudrisk.uk), email: info@cloudrisk.co.uk - [FT Advisory LLC](https://www.ftadvisory.co), email: info@ftadvisory.co - [TradeHeader SL](https://www.tradeheader.com), email: info@tradeheader.com ## Contributing -For any questions, bugs or feature requests please open an [issue](https://github.com/regnosys/rune-python-runtime/issues) -For anything else please send an email to {project mailing list}. - -To submit a contribution: -1. Fork it () -2. Create your feature branch (`git checkout -b feature/fooBar`) -3. Read our [contribution guidelines](.github/CONTRIBUTING.md) and [Community Code of Conduct](https://www.finos.org/code-of-conduct) -4. Commit your changes (`git commit -am 'Add some fooBar'`) -5. Push to the branch (`git push origin feature/fooBar`) -6. Create a new Pull Request - -_NOTE:_ Commits and pull requests to FINOS repositories will only be accepted from those contributors with an active, executed Individual Contributor License Agreement (ICLA) with FINOS OR who are covered under an existing and active Corporate Contribution License Agreement (CCLA) executed with FINOS. Commits from individuals not covered under an ICLA or CCLA will be flagged and blocked by the FINOS Clabot tool (or [EasyCLA](https://community.finos.org/docs/governance/Software-Projects/easycla)). Please note that some CCLAs require individuals/employees to be explicitly named on the CCLA. - -*Need an ICLA? Unsure if you are covered under an existing CCLA? Email [help@finos.org](mailto:help@finos.org)* -## Get in touch with the Rune Python Runtime Team +For questions, bug reports, or feature requests, open a +[GitHub issue](https://github.com/finos/rune-python-runtime/issues). - Get in touch with the Rune team by creating a [GitHub issue](https://github.com/REGnosys/rune-python-runtime/issues/new) and labelling it with "help wanted". +Read the [contribution guidelines](https://github.com/finos/rune-python-runtime/blob/main/CONTRIBUTING.md) for the contribution process +and FINOS contributor license agreement requirements, and follow the +[Community Code of Conduct](https://github.com/finos/rune-python-runtime/blob/main/.github/CODE_OF_CONDUCT.md). +To contribute, [fork the runtime](https://github.com/finos/rune-python-runtime/fork), +create a branch, and submit a pull request. - We encourage the community to get in touch via the [FINOS Slack](https://www.finos.org/blog/finos-announces-new-community-slack). +You can also get in touch through +[FINOS Slack](https://www.finos.org/blog/finos-announces-new-community-slack). ## Governance -This project implements https://community.finos.org/docs/governance/#open-source-software-projects +This project follows the +[FINOS governance framework](https://community.finos.org/docs/governance/#open-source-software-projects). ## License -Copyright 2023-2025 CLOUDRISK Limited and FT Advisory LLC +Copyright 2023-2026 CLOUDRISK Limited and FT Advisory LLC -Distributed under the [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0). +Distributed under the [Apache License, Version 2.0](https://github.com/finos/rune-python-runtime/blob/main/LICENSE). SPDX-License-Identifier: [Apache-2.0](https://spdx.org/licenses/Apache-2.0) diff --git a/docs/choice-serialization.md b/docs/choice-serialization.md new file mode 100644 index 0000000..8e6ecd5 --- /dev/null +++ b/docs/choice-serialization.md @@ -0,0 +1,148 @@ +# Choice serialization + +## Principles + +1. Python objects retain their choice option fields. JSON flattens nested choices + to the selected leaf, preserving required metadata and reference wrappers. +2. Discriminators use Rune identifiers: data types and enums use their Rune FQN; + primitive options use their option name. Python package prefixes and Java + implementation class names are excluded. +3. The generator supplies `_RUNE_CHOICE` and `_CHOICE_DISPATCH`. Runtime dispatch + follows that table without searching the complete choice tree. +4. The DSL prohibits metadata on choice types. Metadata on choice-valued fields + and on individual options remains valid and must survive serialization. +5. References remain reference occurrences, preserving their keys and targets. + Flattening must respect the distinction between a reference to a choice and + a reference to its selected option. +6. JSON readers require the flat format. Native Python constructors and builders + continue to use option field names. Ordinary types with `one-of` conditions + retain their existing representation. + +## Wire format + +Examples omit the document's `@model` and `@version` envelope fields. + +| Selected option | JSON choice value | +| --- | --- | +| Data type | `{"@type": "example.A", "value": "foo"}` | +| Primitive | `{"@type": "string", "@data": "foo"}` | +| Enum | `{"@type": "example.Currency", "@data": "USD"}` | +| Primitive with field metadata | `{"@type": "string", "@scheme": "urn:test", "@data": "foo"}` | + +Metadata does not turn a primitive discriminator into a Java class name such as +`java.lang.String`. Enums use their Rune FQN with or without metadata. Their +Python field names and annotations remain unchanged. + +**Java interoperability check:** the [Java enum example](examples/java-enum-choice/README.md) +records Rune Common 12.29.0 emitting an option name for a plain enum and a Rune FQN +for a metadata-bearing enum. That differs from the uniform enum-FQN generator +rule above and requires alignment with the intended interoperability contract. +The [atomic-choice example](examples/java-atomic-choices/README.md) records Java's +numeric and temporal representations and its unsupported `pattern` option. + +## Generator contract + +Each generated choice declares a marker and a discriminator-to-field map: + +```python +_RUNE_CHOICE: ClassVar[bool] = True +_CHOICE_DISPATCH: ClassVar[dict[str, str]] = { + "example.A": "ChoiceAB", + "example.B": "ChoiceAB", + "example.C": "ChoiceCD", + "example.D": "ChoiceCD", +} +``` + +Values are immediate Python field names, including any name mangling. Each nested +choice repeats the lookup at its own level. For example, a deeper path +`Outer.ChoiceABCD.ChoiceAB.A` is reconstructed one field at a time; the table +does not store that complete path. + +The generator strips type aliases, propagates descendant discriminators, excludes +Python namespace prefixes, and rejects ambiguous routes to the same key. It adds +a nested choice's own FQN only when metadata on that option requires a wrapper or +reference discriminator. Plain nested choices contribute their leaves only. + +Discriminator-based reads use the map. Primitive and enum writes reverse it to obtain +`@type`; data writes use the declared class's `_fqrtn()`. Nested writes forward +the selected leaf representation. Generated enums currently have no `_fqrtn()`. +The generator therefore owns the correctness of their dispatch keys. + +## Metadata and references + +Metadata permissions are additive: field permissions union type permissions. +For a valid generated choice, the type contribution is empty, so permitted +metadata comes from its containing field. The inherited `serialise_meta` handles +it; no choice-specific metadata blacklist is needed. Manually bypassing metadata +validation is outside this contract. + +A choice-valued field with location metadata retains a wrapper: + +```json +{"@key:scoped": "c1", "@data": {"@type": "example.A", "value": "foo"}} +``` + +When that field is itself a selected choice option, the enclosing choice adds +the option's own type: + +```json +{"@type": "example.Inner", "@key:scoped": "c1", "@data": {"@type": "example.A", "value": "foo"}} +``` + +Likewise, `{"@ref:scoped": "c1"}` references the whole choice, while +`{"@type": "example.Inner", "@ref:scoped": "c1"}` selects a referenced option. +Keys inside selected leaves remain available to graph reference resolution. +Lists retain reference occurrence metadata, and COW proxies retain their normal +behavior; see [Object identity and references](object-identity.md). + +A standalone choice has no containing-field metadata. The root serializer omits +any such metadata retained on a reused choice instance, while preserving metadata +on its selected leaf or option, including a primitive's `@scheme`. + +## Runtime flow and entry points + +- **Read:** unwrap field metadata, dispatch `@type` into native option fields, + validate those fields, then initialize containment parents, keys and references. +- **Write:** serialize fields, substitute reference tokens, flatten the selected + option, then merge the document envelope. + +The shared Pydantic hooks are `_deserialize_rune_model` and +`_serialize_rune_model`. Choice handling lives in `_choice_to_fields` and +`_flatten_choice`; field wrappers are handled by `ComplexTypeMetaDataMixin`. + +`rune_deserialize`, `model_validate_json` and `TypeAdapter.validate_json` enter +wire-validation mode. Constructors, ordinary `model_validate` calls and +`ObjectBuilder` accept native field dictionaries. A scoped context flag, the model +metaclass and the builder's explicit native scope keep constructors and builders +working inside JSON-validation default factories. + +Inside a graph, the declared field type supplies the enclosing choice class. +For standalone round trips, call `KnownChoice.rune_deserialize(payload)`. The +generic `BaseDataClass` reader cannot infer the enclosing choice from a leaf tag: +a data FQN identifies the leaf class; primitive and enum tags require choice +context. `rune_serialize()` adds the normal document envelope in either case. + +## Errors and maintenance + +- Missing `_CHOICE_DISPATCH`: `_choice_dispatch()` raises `RuntimeError` with + regeneration guidance on read and write. An empty table is distinct from a + missing table. The lookup is deliberately uncached. +- Wrong discriminator or wrapper Python type: `TypeError`. Missing or unsupported + wire values: `ValueError`, which Pydantic wraps as `ValidationError` when reading. +- Serializer failures are wrapped in `PydanticSerializationError`. Multiple + selected options and data values whose runtime type differs from the declared + option class are rejected on write. Rune constraint validation remains separate + from a bare `model_dump()` call. +- Option-class inspection is cached; `model_rebuild()` clears that cache. +- Regenerate older models to obtain dispatch tables. Older JSON using nested + option field names must be regenerated into the flat format. + +Implementation: [base_data_class.py](../src/rune/runtime/base_data_class.py), +[metadata.py](../src/rune/runtime/metadata.py), and +[object_builder.py](../src/rune/runtime/object_builder.py). Generator counterpart: +`PythonChoiceDispatchProcessor` in `rune-python-generator`. + +Regression coverage: [test_choice_serialization.py](../test/test_choice_serialization.py) +covers nesting, root and field use, metadata, references and lists, builders, COW, +filtering, aliases, rebuilding, and error diagnostics. diff --git a/docs/design.md b/docs/design.md new file mode 100644 index 0000000..4f58852 --- /dev/null +++ b/docs/design.md @@ -0,0 +1,70 @@ +# Rune Python Runtime design + +This document describes the shared runtime architecture and the boundary between +generated Python code and runtime support. The detailed design documents linked +below define the contracts for individual areas. + +## Generator and runtime boundary + +The [Rune Python Code Generator](https://github.com/finos/rune-python-generator) +translates Rune DSL models into Python classes, enums, constraints, and functions. +Generated packages depend on the +[Rune Python Runtime](https://github.com/finos/rune-python-runtime) for common +behavior; model-specific definitions and business logic remain in generated code. + +The generator supplies field annotations, metadata permissions, Rune type +identifiers, and choice dispatch tables. The runtime uses those declarations to +construct and validate objects, execute shared operations, manage references, and +read and write Rune JSON. Changes to this contract may require corresponding +generator changes and regeneration of model packages. + +## Main components + +| Component | Responsibility | +| --- | --- | +| [BaseDataClass](../src/rune/runtime/base_data_class.py) | Pydantic-based foundation for generated models, graph initialization, explicit Rune validation, and JSON serialization and deserialization. | +| [Metadata and references](../src/rune/runtime/metadata.py) | Metadata-bearing values, identity keys, reference tokens, key registries, and containment parent links. | +| [Reference lists](../src/rune/runtime/reference_list.py) | Mutable lists that expose targets while retaining reference metadata for each occurrence. | +| [ObjectBuilder](../src/rune/runtime/object_builder.py) | Incremental construction of nested values before materializing a generated model. | +| [Copy-on-write wrappers](../src/rune/runtime/cow.py) | Deferred copying for supported mutations of function inputs and intermediate values. | +| [Expression helpers](../src/rune/runtime/utils.py) | Shared Rune operations for attribute access, collections, comparisons, cardinality, and sorting. | +| [Conditions](../src/rune/runtime/conditions.py) | Registration and execution support for generated constraints. | +| [Function proxies](../src/rune/runtime/func_proxy.py) and [native registry](../src/rune/runtime/native_registry.py) | Replaceable function calls, return-value finalization, and registration and dispatch of native implementations. | + +## Model lifecycle + +Python constructors and `ObjectBuilder` work with native option field names. +JSON entry points interpret the wire representation, including metadata wrappers +and choice discriminators. Pydantic validates field structure and types; runtime +initialization establishes containment parents, registers keys, and resolves +references. + +Generated functions use the shared expression helpers, builders, and copy-on-write +support. Copy-on-write wrappers defer copying until a supported mutation requires +it. The object identity design below describes the ownership rules and remaining +limitations around writes through borrowed references. + +Rune model validation is explicit: `validate_model()` performs attribute, +cardinality, and condition checks. Ordinary construction or a Pydantic validation +call does not replace this step. + +`rune_serialize()` writes the Rune JSON representation and its document envelope. +Reference occurrences serialize as keys rather than embedded targets; choices +serialize as the selected leaf while preserving required metadata and wrappers. +`rune_deserialize()` reads that representation into generated model classes. + +## Detailed designs + +| Design document | Scope | +| --- | --- | +| [Object identity and references](object-identity.md) | Identity independently of keys, `as-key` selection, reference occurrences, resolution scopes, ownership, and copy-on-write boundaries. It also identifies follow-up ownership enforcement work. | +| [Choice serialization](choice-serialization.md) | Flat JSON choices, Rune discriminators, generator dispatch tables, metadata and reference wrappers, native versus JSON entry points, errors, and interoperability considerations. | + +These contracts interact: choice flattening must preserve reference occurrences, +and builders and collection helpers must preserve occurrence metadata. A single +target can appear inline and by reference in the same list, so reference intent +cannot be inferred from object identity. Use `rune_sorted` when sorting must retain +that metadata; built-in `sorted` returns a plain list. + +Keep detailed rules and implementation status in the individual design documents. +Use this overview as the entry point when adding further design topics. diff --git a/docs/home.md b/docs/home.md index d5c0f86..c01e4e3 100644 --- a/docs/home.md +++ b/docs/home.md @@ -6,4 +6,8 @@ draft: false weight: 2 --- -See [README.md](../README.md) for the project description +See the [README](../README.md) for the project overview, installation, and +development setup. + +The [runtime design overview](design.md) introduces the architecture and links to +the detailed design documents. diff --git a/pyproject.toml b/pyproject.toml index b5884c4..3e2aacd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -3,7 +3,7 @@ # SPDX-License-Identifier: Apache-2.0 # [build-system] -requires = ["setuptools>=64", "setuptools_scm>=8", "wheel>=0.45.1", "pytest-runner>=6.0.1"] +requires = ["setuptools>=77.0.3", "setuptools_scm>=8", "wheel>=0.45.1", "pytest-runner>=6.0.1"] build-backend = "setuptools.build_meta" [project] @@ -13,7 +13,7 @@ requires-python = ">=3.11" dependencies = [ "pydantic>=2.10.3", "python-dateutil>=2.9.0.post0", - "tzdata>=2025.2" + "tzdata>=2026.6" ] optional-dependencies.dev = [ "pytest>=9.0.2", @@ -23,7 +23,7 @@ optional-dependencies.dev = [ "licenseheaders>=0.8.8", "pre-commit>=3.0.0" ] -description = "rune-runtime: the Rune DSL runtime for Python" +description = "Runtime support for Python models and functions generated from Rune DSL" readme = "README.md" keywords = [ "rune", @@ -33,7 +33,7 @@ license = "Apache-2.0" authors = [ { name = "Daniel Schwartz" }, { name = "Plamen Neykov" }, - { name = "Others (See AUTHORS)" } + { name = "Rune Python Runtime contributors" } ] classifiers = [ "Development Status :: 6 - Mature", @@ -46,10 +46,17 @@ classifiers = [ "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", + "Programming Language :: Python :: 3.14", "Topic :: Software Development :: Libraries", "Topic :: Software Development :: Libraries :: Python Modules" ] +[project.urls] +Homepage = "https://github.com/finos/rune-python-runtime" +Documentation = "https://github.com/finos/rune-python-runtime/blob/main/README.md" +Repository = "https://github.com/finos/rune-python-runtime" +Issues = "https://github.com/finos/rune-python-runtime/issues" + [tool.setuptools.packages.find] where = ["src"]