diff --git a/.github/actions/conformance/action.yml b/.github/actions/conformance/action.yml new file mode 100644 index 0000000..60270d3 --- /dev/null +++ b/.github/actions/conformance/action.yml @@ -0,0 +1,50 @@ +# Checks a codec against fspm-TF's conformance files, in a codec's own CI, once the codec has run: +# +# - uses: ./.github/actions/conformance (in this repository; elsewhere openalea/fspm-TF/.github/actions/conformance@main) +# with: +# dumps: out/dumps # what the codec read: one JSON dump per conformance GLB, x.json for x.expected.json +# written: out/written # what the codec wrote: the conformance scenes, as .glb files (optional) +# +# Reading passes when every dump equals its expected values; writing, when every written file passes fspm-tf validate +# and Khronos' glTF-Validator with no error. +name: fspm-TF conformance +description: Compare a codec's reading of the conformance files with their expected values, and validate what it wrote +inputs: + dumps: + description: the folder of the codec's JSON dumps of the conformance files + required: true + written: + description: the folder of the .glb files the codec wrote for the conformance scenes (empty to skip) + required: false + default: "" + fspm-tf: + description: the path of an fspm-TF checkout (its conformance folder and Python package) + required: false + default: "." + gltf-validator: + description: the version of Khronos' glTF-Validator + required: false + default: "2.0.0-dev.3.10" +runs: + using: composite + steps: + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - name: Install openalea.fspm_tf + shell: bash + run: python -m pip install numpy jsonschema "${{ inputs.fspm-tf }}" + - name: Reading + shell: bash + run: fspm-tf compare-all "${{ inputs.dumps }}" "${{ inputs.fspm-tf }}/conformance" + - name: Writing, fspm-TF validator + if: inputs.written != '' + shell: bash + run: find "${{ inputs.written }}" -name "*.glb" -print0 | xargs -0 fspm-tf validate + - name: Writing, Khronos' glTF-Validator + if: inputs.written != '' && runner.os == 'Linux' + shell: bash + run: | + curl -sSL -o validator.tar.xz "https://github.com/KhronosGroup/glTF-Validator/releases/download/${{ inputs.gltf-validator }}/gltf_validator-${{ inputs.gltf-validator }}-linux64.tar.xz" + tar xf validator.tar.xz gltf_validator + find "${{ inputs.written }}" -name "*.glb" -print0 | xargs -0 -n1 ./gltf_validator diff --git a/.github/workflows/conformance.yml b/.github/workflows/conformance.yml index 9052634..dae2e39 100644 --- a/.github/workflows/conformance.yml +++ b/.github/workflows/conformance.yml @@ -46,3 +46,20 @@ jobs: run: python -m pip install numpy jsonschema sphinx pydata-sphinx-theme myst-parser sphinx-favicon sphinx-copybutton . - name: Build the documentation, warnings as errors run: python -m sphinx -b html docs docs/_build -W --keep-going + + # The reference codec through the action other codecs use: its dumps of the conformance files, and the files it writes + python-codec-conformance: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - name: Dump and write the conformance scenes with the Python codec + run: | + python -m pip install numpy jsonschema . + python bindings/python/conformance_run.py conformance out + - uses: ./.github/actions/conformance + with: + dumps: out/dumps + written: out/written diff --git a/README.md b/README.md index d04a55b..11e0f41 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# fspm-TF +# FSPM-TF - Functional Structural Plant Model Transmission Format - based on GLTF 2.0 An exchange format for whole functional-structural plant model (FSPM) scenes, across platforms and languages (openalea.metafspm, CPlantBox, GroIMP, VirtualPlantLab). It holds: @@ -30,7 +30,7 @@ change there, released as a new version. Everything else implements, checks or d | `spec/specification.md` | the specification: data model, glTF layout, sequences, conformance | **yes** | | `spec/schema/` | the JSON Schemas of the `FSPM_scene` extension and of a sequence's index | **yes** | | `spec/vocabulary/` | the standard names (units, kinds, definitions, and each platform's names for them), the labels with their Plant Ontology terms, the enumerations | **yes** | -| `conformance/` | reference files, each a GLB with its expected values as JSON, written by `generate.py`: the specification as test cases for every platform's code | follows `spec/` | +| `conformance/` | reference files, each a GLB with its expected values as JSON, written by `generate.py`: the specification as test cases for every platform's code (`conformance/README.md`; checked with `fspm-tf compare-all` and the action `.github/actions/conformance`) | follows `spec/` | | `bindings//` | each language's codec, with its own build, tests and release; `bindings/python/`: `openalea.fspm_tf`, the reference (codec, validator, display copies, command line), which reads `spec/` and ships it | no | | `paraview/` | the ParaView reader plugin, and the script checking it in a `pvpython` | no | | `docs/design_note.md` | how the decisions were reached | no | diff --git a/bindings/python/conformance_run.py b/bindings/python/conformance_run.py new file mode 100644 index 0000000..16e4904 --- /dev/null +++ b/bindings/python/conformance_run.py @@ -0,0 +1,38 @@ +""" +The Python codec's run of the conformance check, as every codec does it in its CI: read each conformance GLB and dump +it as JSON (OUT/dumps/x.json for x.glb), and write the conformance scenes again as GLB files (OUT/written/x.glb). +Then .github/actions/conformance compares the dumps with the expected values and validates the written files. + + python bindings/python/conformance_run.py CONFORMANCE OUT +""" +import json +import os +import shutil +import sys + +from openalea import fspm_tf +from openalea.fspm_tf.dump import frame_to_dict + + +def main(conformance: str, out: str) -> int: + for folder, _, files in os.walk(conformance): + for name in sorted(files): + if not name.endswith(".glb"): + continue + relative = os.path.relpath(os.path.join(folder, name), conformance) + frame = fspm_tf.read(os.path.join(conformance, relative)) + dump = os.path.join(out, "dumps", relative[:-len(".glb")] + ".json") + written = os.path.join(out, "written", relative) + for path in (dump, written): + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(dump, "w", encoding="utf-8") as file: + json.dump(frame_to_dict(frame), file, indent=1) + fspm_tf.write(written, frame) # written again from what was read + index = os.path.join(conformance, "growing_sequence", "scene.fspm.json") + if os.path.exists(index): + shutil.copy(index, os.path.join(out, "written", "growing_sequence", "scene.fspm.json")) + return 0 + + +if __name__ == "__main__": + sys.exit(main(*sys.argv[1:3])) diff --git a/bindings/python/src/openalea/fspm_tf/cli.py b/bindings/python/src/openalea/fspm_tf/cli.py index 7e31f39..247223e 100644 --- a/bindings/python/src/openalea/fspm_tf/cli.py +++ b/bindings/python/src/openalea/fspm_tf/cli.py @@ -1,10 +1,14 @@ """ -The fspm-tf command line: +The fspm-tf command line:: fspm-tf validate FILE... check .glb frames or sequences (scene.fspm.json); exit 1 on errors fspm-tf info FILE what a frame holds fspm-tf display FILE NAME... [-o OUT] add display copies of variables (in place, or to OUT) fspm-tf display FILE --remove [-o OUT] remove them + fspm-tf compare ACTUAL EXPECTED a codec's JSON dump (or a .glb) against expected values; exit 1 if they + differ + fspm-tf compare-all DUMPS CONFORMANCE every conformance file's expected values against the dump at the same + path (x.expected.json: DUMPS/x.json); exit 1 if any differ """ import argparse import sys @@ -58,6 +62,31 @@ def _display(arguments) -> int: return 0 +def _print_differences(name: str, differences: list, shown: int = 20) -> None: + print(f"{name}: {'equal' if not differences else f'{len(differences)} difference(s)'}") + for line in differences[:shown]: + print(f" {line}") + if len(differences) > shown: + print(f" ... {len(differences) - shown} more") + + +def _compare(arguments) -> int: + from openalea.fspm_tf.compare import compare_files + differences = compare_files(arguments.actual, arguments.expected) + _print_differences(arguments.actual, differences) + return 1 if differences else 0 + + +def _compare_all(arguments) -> int: + from openalea.fspm_tf.compare import compare_all + results = compare_all(arguments.dumps, arguments.conformance) + for name, differences in results.items(): + _print_differences(name, differences) + failed = sum(1 for differences in results.values() if differences) + print(f"{len(results) - failed} of {len(results)} conformance files read as expected") + return 1 if failed or not results else 0 + + def main(argv=None) -> int: parser = argparse.ArgumentParser(prog="fspm-tf", description="fspm-TF files: validate, inspect, display") commands = parser.add_subparsers(dest="command", required=True) @@ -74,6 +103,15 @@ def main(argv=None) -> int: display.add_argument("--remove", action="store_true", help="remove display copies (all, or the names given)") display.add_argument("-o", "--output", default=None, help="write to this file instead of in place") display.set_defaults(run=_display) + compare = commands.add_parser("compare", help="a codec's JSON dump (or a .glb) against expected values") + compare.add_argument("actual") + compare.add_argument("expected") + compare.set_defaults(run=_compare) + compare_all = commands.add_parser("compare-all", help="dumps of every conformance file against their expected " + "values") + compare_all.add_argument("dumps", help="the folder of the codec's dumps (x.json for x.expected.json)") + compare_all.add_argument("conformance", help="fspm-TF's conformance folder") + compare_all.set_defaults(run=_compare_all) arguments = parser.parse_args(argv) return arguments.run(arguments) diff --git a/bindings/python/src/openalea/fspm_tf/codec.py b/bindings/python/src/openalea/fspm_tf/codec.py index 4d9b5ed..84ef88b 100644 --- a/bindings/python/src/openalea/fspm_tf/codec.py +++ b/bindings/python/src/openalea/fspm_tf/codec.py @@ -30,6 +30,9 @@ ARRAY_BUFFER, ELEMENT_ARRAY_BUFFER = 34962, 34963 FLOAT, UNSIGNED_INT = 5126, 5125 MAX_FLOAT_FEATURE = 2 ** 24 # feature rows are stored as float32 vertex attributes +# The scene's frame is z up (metres); glTF viewers expect y up: the root node turns the scene for them (-90° about x, +# column-major). Readers of the tables use the scene's frame, not this node. +Z_UP_TO_Y_UP = [1., 0., 0., 0., 0., 0., -1., 0., 0., 1., 0., 0., 0., 0., 0., 1.] COMPONENT_TYPES = {np.dtype(name): name.upper() for name in ( "int8", "uint8", "int16", "uint16", "int32", "uint32", "int64", "uint64", "float32", "float64")} @@ -54,7 +57,8 @@ def to_bytes(frame: Frame) -> bytes: for value, name in enumerate(EDGE_TYPES)]}}} tables, variables = [], {} document = {"asset": {"version": "2.0", "generator": f"fspm-tf {SPECIFICATION_VERSION}"}, - "scene": 0, "scenes": [{"nodes": []}], "nodes": []} + "scene": 0, "scenes": [{"nodes": [0]}], + "nodes": [{"name": "fspm_scene", "matrix": Z_UP_TO_Y_UP, "children": []}]} accessors, meshes = [], [] def add_table(table: Table, class_id: str) -> int: @@ -95,7 +99,7 @@ def add_table(table: Table, class_id: str) -> int: if not np.array_equal(data_structure.origin, np.eye(4)): node["matrix"] = entry["origin"] document["nodes"].append(node) - document["scenes"][0]["nodes"].append(len(document["nodes"]) - 1) + document["nodes"][0]["children"].append(len(document["nodes"]) - 1) entry["node"] = len(document["nodes"]) - 1 if data_structure.meshes: node["children"], entry["meshes"] = [], [] diff --git a/bindings/python/src/openalea/fspm_tf/compare.py b/bindings/python/src/openalea/fspm_tf/compare.py new file mode 100644 index 0000000..e904afc --- /dev/null +++ b/bindings/python/src/openalea/fspm_tf/compare.py @@ -0,0 +1,79 @@ +""" +Comparing what a codec reads with the conformance files' expected values (``*.expected.json``), whatever the codec's +language: the codec dumps each GLB it reads as JSON (the layout of fspm_tf.dump), and this compares the dumps:: + + compare(actual, expected) -> [differences] two JSON values: objects by key, arrays in order, numbers + exactly (1 and 1.0 equal), null, "Infinity", "-Infinity", text + compare_all(dumps, conformance) -> {file: [differences]} + every *.expected.json of the conformance folder against the + dump at the same relative path (x.expected.json: x.json) +""" +import json +import math +import os + +from openalea.fspm_tf.dump import frame_to_dict + + +def _number(value) -> bool: + return isinstance(value, (int, float)) and not isinstance(value, bool) + + +def compare(actual, expected, path: str = "$") -> list: + """The differences between two JSON values, as "path: what differs" lines (none when equal).""" + if isinstance(expected, dict): + if not isinstance(actual, dict): + return [f"{path}: an object expected, got {type(actual).__name__}"] + found = [f"{path}: missing key '{key}'" for key in expected if key not in actual] + found += [f"{path}: unexpected key '{key}'" for key in actual if key not in expected] + for key in expected: + if key in actual: + found += compare(actual[key], expected[key], f"{path}.{key}") + return found + if isinstance(expected, list): + if not isinstance(actual, list): + return [f"{path}: an array expected, got {type(actual).__name__}"] + if len(actual) != len(expected): + return [f"{path}: {len(actual)} items, {len(expected)} expected"] + found = [] + for index, (left, right) in enumerate(zip(actual, expected)): + found += compare(left, right, f"{path}[{index}]") + return found + if _number(expected): + if not _number(actual) or float(actual) != float(expected) or (math.isnan(float(actual))): + return [f"{path}: {actual!r}, {expected!r} expected"] + return [] + if actual != expected or type(actual) is not type(expected): + return [f"{path}: {actual!r}, {expected!r} expected"] + return [] + + +def load(path: str): + """A JSON dump, or a GLB read by this package and dumped (to check this package itself).""" + if path.endswith(".glb"): + from openalea.fspm_tf import read + return json.loads(json.dumps(frame_to_dict(read(path)))) + with open(path, encoding="utf-8") as file: + return json.load(file) + + +def compare_files(actual_path: str, expected_path: str) -> list: + return compare(load(actual_path), load(expected_path)) + + +def compare_all(dumps: str, conformance: str) -> dict: + """{expected file (relative): its differences} for every *.expected.json under *conformance*, against the dump at + the same relative path under *dumps* (a missing dump is a difference).""" + results = {} + for folder, _, files in sorted(os.walk(conformance)): + for name in sorted(files): + if not name.endswith(".expected.json"): + continue + expected = os.path.join(folder, name) + relative = os.path.relpath(expected, conformance) + actual = os.path.join(dumps, relative[:-len(".expected.json")] + ".json") + if not os.path.exists(actual): + results[relative] = [f"no dump {os.path.relpath(actual, dumps)}"] + continue + results[relative] = compare_files(actual, expected) + return results diff --git a/bindings/python/src/openalea/fspm_tf/ucum.py b/bindings/python/src/openalea/fspm_tf/ucum.py index 959c064..0ad27b0 100644 --- a/bindings/python/src/openalea/fspm_tf/ucum.py +++ b/bindings/python/src/openalea/fspm_tf/ucum.py @@ -25,6 +25,7 @@ def _dims(**exponents) -> tuple: "lm": _dims(J=1), "lx": _dims(J=1, L=-2), "Bq": _dims(T=-1), "Gy": _dims(L=2, T=-2), "Sv": _dims(L=2, T=-2), "L": _dims(L=3), "l": _dims(L=3), "bar": _dims(M=1, L=-1, T=-2), "ar": _dims(L=2), "t": _dims(M=1), "eq": _dims(N=1), "Cel": _dims(Th=1), + "m[H2O]": _dims(M=1, L=-1, T=-2), "m[Hg]": _dims(M=1, L=-1, T=-2), # pressures as heights: cm[H2O], mm[Hg] } ATOMS = dict(METRIC_ATOMS, **{ "min": _dims(T=1), "h": _dims(T=1), "d": _dims(T=1), "wk": _dims(T=1), "mo": _dims(T=1), "a": _dims(T=1), @@ -33,7 +34,7 @@ def _dims(**exponents) -> tuple: SPECIAL = {"Cel"} # an offset scale: not allowed in products, quotients, powers PREFIXES = ("da", "Y", "Z", "E", "P", "T", "G", "M", "k", "h", "d", "c", "m", "u", "n", "p", "f", "a", "z", "y") -_TOKEN = re.compile(r"\{[^{}]*\}|\(|\)|\.|/|[+-]?\d+|\[[^\]]+\]|%|[A-Za-z]+") +_TOKEN = re.compile(r"\{[^{}]*\}|\(|\)|\.|/|[+-]?\d+|[A-Za-z]+\[[^\]]+\]|\[[^\]]+\]|%|[A-Za-z]+") class UnitError(ValueError): diff --git a/bindings/python/src/openalea/fspm_tf/validate.py b/bindings/python/src/openalea/fspm_tf/validate.py index 4ec1ece..921b6fb 100644 --- a/bindings/python/src/openalea/fspm_tf/validate.py +++ b/bindings/python/src/openalea/fspm_tf/validate.py @@ -16,7 +16,8 @@ import numpy as np from openalea.fspm_tf import schema as schemas, ucum, vocabulary -from openalea.fspm_tf.codec import (DTYPES, EDGE_TYPE_ENUM, EXTENSION, FEATURES, METADATA, from_bytes) +from openalea.fspm_tf.codec import (DTYPES, EDGE_TYPE_ENUM, EXTENSION, FEATURES, METADATA, Z_UP_TO_Y_UP, + from_bytes) from openalea.fspm_tf.glb import unpack from openalea.fspm_tf.model import EDGE_TYPES, NO_ID @@ -95,6 +96,11 @@ def _check_document(document: dict, binary: bytes, report: Report) -> bool: report.error("extensionsUsed", f"must list {name}") if EXTENSION in document.get("extensionsRequired", []): report.warning("extensionsRequired", f"{EXTENSION} should not be required, so that any viewer opens the file") + roots = [document.get("nodes", [])[index] for index in document.get("scenes", [{}])[0].get("nodes", []) + if index < len(document.get("nodes", []))] + if not any(node.get("name") == "fspm_scene" and node.get("matrix") == Z_UP_TO_Y_UP for node in roots): + report.warning("scenes", "no root node fspm_scene turning the scene (z up) to glTF's y up: viewers will show it " + "lying down") extensions = document.get("extensions", {}) if EXTENSION not in extensions: report.error("extensions", f"no {EXTENSION} extension: not an fspm-TF file") diff --git a/bindings/python/test/test_compare.py b/bindings/python/test/test_compare.py new file mode 100644 index 0000000..8d4502f --- /dev/null +++ b/bindings/python/test/test_compare.py @@ -0,0 +1,58 @@ +"""Comparing a codec's dumps with the conformance files' expected values (fspm-tf compare, compare-all).""" +import json +import os +import shutil + +import pytest + +from conftest import CONFORMANCE +from openalea.fspm_tf.cli import main +from openalea.fspm_tf.compare import compare, compare_all + + +@pytest.mark.parametrize("actual, expected, equal", [ + ({"a": [1, 2.5, None, "Infinity"]}, {"a": [1., 2.5, None, "Infinity"]}, True), # 1 and 1.0 are equal + ({"a": 1}, {"a": 1.0000000000000002}, False), # floats exactly + ({"a": True}, {"a": 1}, False), # a boolean is not a number + ({"a": [1, 2]}, {"a": [2, 1]}, False), # arrays in order + ({"b": 1, "a": 2}, {"a": 2, "b": 1}, True), # objects by key + ({"a": 1}, {"a": 1, "b": 2}, False), + ({"a": "-Infinity"}, {"a": "Infinity"}, False), +]) +def test_compare(actual, expected, equal): + assert (compare(actual, expected) == []) is equal + + +def test_the_differences_say_where(): + differences = compare({"x": {"y": [1, 2, 4]}, "z": 1}, {"x": {"y": [1, 2, 3]}, "w": 1}) + assert "$.x.y[2]: 4, 3 expected" in differences + assert "$: missing key 'w'" in differences and "$: unexpected key 'z'" in differences + + +def test_compare_all(tmp_path): + dumps = tmp_path / "dumps" + for folder, _, files in os.walk(CONFORMANCE): + for name in files: + if name.endswith(".expected.json"): + relative = os.path.relpath(os.path.join(folder, name), CONFORMANCE) + target = dumps / (relative[:-len(".expected.json")] + ".json") + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy(os.path.join(folder, name), target) + results = compare_all(str(dumps), CONFORMANCE) + assert len(results) == 9 and not any(results.values()) + assert main(["compare-all", str(dumps), CONFORMANCE]) == 0 + with open(dumps / "root_system.json") as file: + dump = json.load(file) + dump["step"] = 1 + with open(dumps / "root_system.json", "w") as file: + json.dump(dump, file) + (dumps / "wheat_shoot.json").unlink() + results = compare_all(str(dumps), CONFORMANCE) + assert results["root_system.expected.json"] == ["$.step: 1, 0 expected"] + assert results["wheat_shoot.expected.json"] == ["no dump wheat_shoot.json"] + assert main(["compare-all", str(dumps), CONFORMANCE]) == 1 + + +def test_compare_a_glb_with_its_expected_values(): + glb = os.path.join(CONFORMANCE, "section_cells.glb") + assert main(["compare", glb, glb.replace(".glb", ".expected.json")]) == 0 diff --git a/bindings/python/test/test_tools.py b/bindings/python/test/test_tools.py index bf6a5bd..6027372 100644 --- a/bindings/python/test/test_tools.py +++ b/bindings/python/test/test_tools.py @@ -10,7 +10,8 @@ @pytest.mark.parametrize("code, other", [("mm3/(s.MPa)", "m3/(s.Pa)"), ("cm4/(hPa.d)", "m4/(s.Pa)"), ("umol/(m2.s)", "mol.m-2.s-1"), ("Cel", "K"), ("m3/m3", "1"), - ("{leaf}", "1"), ("K.d", "s.K"), ("%", "1")]) + ("{leaf}", "1"), ("K.d", "s.K"), ("%", "1"), ("cm[H2O]", "hPa"), + ("mm[Hg]", "Pa"), ("cm[H2O]/d", "Pa/s")]) def test_convertible(code, other): assert ucum.convertible(code, other) diff --git a/conformance/README.md b/conformance/README.md new file mode 100644 index 0000000..fa9c75a --- /dev/null +++ b/conformance/README.md @@ -0,0 +1,58 @@ +# Conformance files + +The specification as test cases for every codec. Each scene is a frame (`x.glb`) and its expected values +(`x.expected.json`); `growing_sequence/` is a sequence with its index. `generate.py` writes them all from fixed +numbers, the same bytes on every run; the CI checks that the committed files are what it writes. + +| Scene | Covers | +|---|---| +| `root_system` | Plant, Axis and SubOrgan scales, MTG relations, segment variables, a mesh representing three scales, scalars | +| `wheat_shoot` | Plant, Axis, Organ and SubOrgan scales, leaf quads, NaN through `noData`, a translated origin | +| `section_cells` | a cell network: an entity set and a relation with ids, labels and conductances; text and vector columns; extras | +| `soil_and_roots` | a root system, a regular soil grid with z pointing down, and their mapping | +| `adaptive_soil` | an adaptive grid's leaf cells (`level`, `index`) | +| `parameter_set` | a `table` data structure: parameters per plant and in the scalars, references, an infinite value | +| `growing_sequence/` | three frames of a growing root system, ids kept, and their index | + +## The expected values + +`x.expected.json` is the frame `x.glb` holds, as plain JSON (written by `openalea.fspm_tf.dump.frame_to_dict`): + +```text +{"time": 0.0, "time_unit": "d", "step": 0, + "data_structures": [ + {"id": "...", "kind": "graph", "origin": [[4 numbers] x 4, row by row], "platform": "...", "model": "...", + "extras": {...}, (when present) + "entity_sets": [{"name": "SubOrgan", "rank": 6, "count": 10, + "columns": {"id": {"dtype": "int64", "values": [...]}, + "label": {"dtype": "string", "values": [...]}, (labels and enums as their names) + "end": {"dtype": "float64", "values": [[x, y, z], ...]}, ...}, + "variables": {"length": {"unit": "m", "role": "state", "standard_name": "length", + "kind": "extensive", "description": "..."}, ...}}], + "relations": [... like entity sets, with "source_set" and "target_set"], + "scalars": {... like an entity set, one row}, + "grid": {"kind", "shape", "origin", "spacing", "axes", "periodic", "z_up", "max_level", "cells"}, + "meshes": [{"name": "...", "positions_float32": [[x, y, z], ...], "feature_rows": {"SubOrgan": [...]}, + "indices": [...], "display": {"name": [...]}}]}], + "mappings": [... like entity sets, with "source", "target" ([data structure, entity set]) and "method"]} +``` + +- **Floats** are the shortest decimals giving the same double, compared exactly. +- **NaN** is `null`; +∞ and −∞ are the strings `"Infinity"` and `"-Infinity"`. +- **Positions** are the FLOAT32 values of the file, widened to doubles. +- **`dtype`** is the column's numpy type name (`float64`, `int64`, `uint8`, `bool`...), or `string` for text and enums. +- Keys only appear when the frame has them (no `extras`, `relations`, `scalars`, `grid` or `meshes` when absent). + +## Checking a codec + +1. The codec reads each `x.glb` and dumps what it read in this layout as `DUMPS/x.json` (same relative path). +2. It writes the same scenes back as GLB files, in `WRITTEN/`. +3. Then: + + ```bash + fspm-tf compare-all DUMPS conformance # reading: every dump equals its expected values + fspm-tf validate WRITTEN/*.glb # writing: valid fspm-TF (and Khronos' glTF-Validator: valid glTF) + ``` + +In GitHub Actions, the composite action `.github/actions/conformance` runs both. The Python codec's own run is +`bindings/python/conformance_run.py`, used by the `python-codec-conformance` job of `.github/workflows/conformance.yml`. diff --git a/conformance/adaptive_soil.glb b/conformance/adaptive_soil.glb index 71acf51..80dd7a0 100644 Binary files a/conformance/adaptive_soil.glb and b/conformance/adaptive_soil.glb differ diff --git a/conformance/growing_sequence/frame_000000.glb b/conformance/growing_sequence/frame_000000.glb index ab404ba..7d09190 100644 Binary files a/conformance/growing_sequence/frame_000000.glb and b/conformance/growing_sequence/frame_000000.glb differ diff --git a/conformance/growing_sequence/frame_000001.glb b/conformance/growing_sequence/frame_000001.glb index 7441258..ecb3a69 100644 Binary files a/conformance/growing_sequence/frame_000001.glb and b/conformance/growing_sequence/frame_000001.glb differ diff --git a/conformance/growing_sequence/frame_000002.glb b/conformance/growing_sequence/frame_000002.glb index 3d46a06..5af5667 100644 Binary files a/conformance/growing_sequence/frame_000002.glb and b/conformance/growing_sequence/frame_000002.glb differ diff --git a/conformance/parameter_set.glb b/conformance/parameter_set.glb index f525e97..8ac363c 100644 Binary files a/conformance/parameter_set.glb and b/conformance/parameter_set.glb differ diff --git a/conformance/root_system.glb b/conformance/root_system.glb index bbd7cb7..3b9b857 100644 Binary files a/conformance/root_system.glb and b/conformance/root_system.glb differ diff --git a/conformance/section_cells.glb b/conformance/section_cells.glb index bc8f299..2272cc3 100644 Binary files a/conformance/section_cells.glb and b/conformance/section_cells.glb differ diff --git a/conformance/soil_and_roots.glb b/conformance/soil_and_roots.glb index 6f193d2..8a17a94 100644 Binary files a/conformance/soil_and_roots.glb and b/conformance/soil_and_roots.glb differ diff --git a/conformance/wheat_shoot.glb b/conformance/wheat_shoot.glb index 7b8cefb..6c059ce 100644 Binary files a/conformance/wheat_shoot.glb and b/conformance/wheat_shoot.glb differ diff --git a/docs/api.md b/docs/api.md index d56e450..fc35111 100644 --- a/docs/api.md +++ b/docs/api.md @@ -41,6 +41,13 @@ the vocabulary and units. :members: dimension, is_valid, convertible ``` +## Comparing with expected values + +```{eval-rst} +.. automodule:: openalea.fspm_tf.compare + :members: compare, compare_files, compare_all +``` + ## Expected values ```{eval-rst} diff --git a/docs/platforms.md b/docs/platforms.md index 10e4458..ae671e0 100644 --- a/docs/platforms.md +++ b/docs/platforms.md @@ -76,16 +76,26 @@ mapping, an adaptive grid, a parameter set, and a growing sequence. Each GLB com - **Writing:** what it writes, for the same scenes, passes `fspm-tf validate` with no error. It also passes Khronos' glTF-Validator with no error, so that every viewer opens it. -In the codec's CI, both run against the shared `conformance/` folder: +`conformance/README.md` gives the layout of the expected JSON. A codec dumps what it reads from each conformance +GLB in that layout (`x.json` for `x.glb`), and writes the same scenes back as GLB files. Then: + +```bash +fspm-tf compare-all dumps/ conformance/ # reading: every dump equals its expected values, or the differences +fspm-tf validate written/*.glb # writing +``` + +In GitHub Actions, the composite action `.github/actions/conformance` runs both, plus Khronos' glTF-Validator, +after the codec's own steps: ```yaml -- name: Read the conformance files - run: # compares each conformance GLB with its *.expected.json -- name: Write the conformance scenes, then validate them - run: | - conformance-out/ - pip install numpy jsonschema && pip install -e . - fspm-tf validate conformance-out/*.glb +- name: Dump and write the conformance scenes + run: conformance out # out/dumps/x.json, out/written/x.glb +- uses: ./.github/actions/conformance + with: + dumps: out/dumps + written: out/written ``` +The Python codec does exactly this (`bindings/python/conformance_run.py`, the `python-codec-conformance` job). + A codec that passes is listed in [Implementations](implementations.md) with the version it passes. diff --git a/spec/specification.md b/spec/specification.md index 2bddf3b..a91ec32 100644 --- a/spec/specification.md +++ b/spec/specification.md @@ -31,6 +31,10 @@ The language-neutral files of this version are part of the specification: | `table` | entity sets without topology, e.g. a parameter set | Its `origin` is a 4×4 matrix placing its coordinates in the scene's frame. +- **The scene's frame** is right-handed, in metres, with **z up**: z = 0 at the soil surface, negative below it. + Every position (meshes, the segment ends of the vocabulary's `proximal_end_*` and `distal_end_*`, grid origins and + spacings) is in metres. A platform with other axes or units (e.g. y up, or centimetres) converts its coordinates, or + puts the conversion in the data structure's origin. - An **entity set** is a table: one row per entity, one column per variable. A graph has one entity set per scale, with that scale's `rank` (coarser scales have smaller ranks). A grid's cells form one entity set, with no rank. - A **relation** is an edge table between two entity sets of the same data structure. Examples: a cell network, or @@ -52,8 +56,11 @@ A frame MUST be a GLB file (glTF 2.0 binary): spaces and the binary chunk with zeros to 8 bytes, as `EXT_structural_metadata` requires. 3. `extensionsUsed` MUST list `FSPM_scene` and `EXT_structural_metadata`, plus `EXT_mesh_features` when there are meshes. `FSPM_scene` SHOULD NOT be listed in `extensionsRequired`, so that every glTF viewer opens the file. -4. The glTF scene MUST have one root node per data structure, named after its `id`. That node carries the data - structure's origin as its `matrix` when the origin is not the identity. The node's children each hold one of the +4. The glTF scene's root node, `fspm_scene`, turns the scene's frame (z up) into glTF's (y up), so that glTF viewers + show the scene upright: its `matrix` is a rotation of −90° about x (column-major + `[1,0,0,0, 0,0,-1,0, 0,1,0,0, 0,0,0,1]`). It is for viewers only: readers of the tables, and `FSPM_scene`'s + origins, use the scene's frame. Under it, one node per data structure, named after its `id`, carries the data + structure's origin as its `matrix` when the origin is not the identity; that node's children each hold one of the data structure's meshes. 5. The file SHOULD be valid glTF for Khronos' glTF-Validator: no errors. @@ -158,6 +165,11 @@ Rules on these columns: - `parent` and `edge_type` go together. `edge_type` is `none` exactly where `parent` is −1. The parents MUST form no cycle. - Every id referred to MUST exist in the entity set referred to. +- `parent` links entities of the same scale. When an axis's first entity grows on an entity of another scale (in an + MTG, a branch axis's first internode, linked to its bearer only through the axes), its `parent` is −1. Its bearer + MAY be written as a relation named `bearer`, from the entity to the one it is borne on. +- Other links between entities are relations, one per kind of link. A pair of entities linked in several ways (e.g. a + successor edge that is also a platform's own edge) appears in each of these relations. - Each label column or relation column is an `ENUM` of its own, listing the values used. The enum `FSPM_edge_type` is fixed: `none` = 0, `successor` = 1, `branch` = 2. - A grid's cells MAY omit `id`; their ids are then their row numbers. A regular grid's rows are its cells in C order @@ -190,6 +202,7 @@ Units are UCUM codes, with: - SI and common atoms (`m`, `g`, `s`, `K`, `mol`, `Pa`, `J`, `W`, `L`, `Cel`, `min`, `h`, `d`, `a`, `deg`, `%`, ...); - metric prefixes on the metric atoms (`mm`, `umol`, `MPa`, `hPa`, ...); +- pressures as heights of water or mercury: `m[H2O]`, `m[Hg]`, with prefixes (`cm[H2O]`, `mm[Hg]`); - products `.`, quotients `/`, integer exponents (`m2`, `s-1`), parentheses, annotations `{...}`, and `1`. `Cel` cannot be combined with other units: thermal time is in `K.d`. No spaces are allowed: @@ -248,7 +261,8 @@ A sequence is a folder holding one GLB per frame and an index, `scene.fspm.json` index's. - **Ids across frames.** An entity keeps its id from frame to frame. An id whose entity has disappeared MUST NOT be given to another entity in a later frame of the same data structure and entity set. Labels and parents MAY change, - e.g. an apex moved by growth. + e.g. an apex moved by growth. A platform whose own ids may come back (a counter restarted, a deleted maximum's id + given again) keeps, across a sequence, a map from its entities to the ids it writes. - **Packing.** Packing a sequence into one GLB is not part of 0.1. ## 9. Conformance diff --git a/spec/vocabulary/standard_names.csv b/spec/vocabulary/standard_names.csv index f07d609..5eaae3b 100644 --- a/spec/vocabulary/standard_names.csv +++ b/spec/vocabulary/standard_names.csv @@ -5,8 +5,8 @@ proximal_end_z,m,descriptor,position z,entity proximal end,,scene frame,SubOrgan distal_end_x,m,descriptor,position x,entity distal end,,scene frame,SubOrgan,"x coordinate of the end of the entity farthest from the plant base, in the data structure's frame",x2,,,,metafspm/ArchiSimple/Adel: x2 distal_end_y,m,descriptor,position y,entity distal end,,scene frame,SubOrgan,"y coordinate of the end of the entity farthest from the plant base, in the data structure's frame",y2,,,,metafspm/ArchiSimple/Adel: y2 distal_end_z,m,descriptor,position z,entity distal end,,scene frame,SubOrgan,"z coordinate (up) of the end of the entity farthest from the plant base, in the data structure's frame",z2,,,,metafspm/ArchiSimple/Adel: z2 -length,m,extensive,length,entity,,,SubOrgan Organ,length of the entity along its axis,length,,,,metafspm/ArchiSimple/Adel/GRANAP/MECHA: length; Caribu: element_length; CPlantBox: segment length -radius,m,intensive,radius,entity,,,SubOrgan,"radius of the entity's cross section (a root segment's, a stem's)",radius,,,,ArchiSimple/GRANAP: radius; CPlantBox: radius +length,m,extensive,length,entity,,,SubOrgan Organ,length of the entity along its axis,length,length,,,"metafspm/ArchiSimple/Adel/GRANAP/MECHA: length; Caribu: element_length; CPlantBox: segment length; cplantbox: proposed from Organ::getParameter (organ level, cm and d), to be confirmed by the CPlantBox team" +radius,m,intensive,radius,entity,,,SubOrgan,"radius of the entity's cross section (a root segment's, a stem's)",radius,radius,,,"ArchiSimple/GRANAP: radius; CPlantBox: radius; cplantbox: proposed from Organ::getParameter (organ level, cm and d), to be confirmed by the CPlantBox team" surface_area,m2,extensive,area,entity,,one side,SubOrgan Organ,"area of the entity's surface (one side of a leaf, the lateral surface of a cylinder)",area,,,,Adel: area; Caribu: element area green_area,m2,extensive,area,entity green part,,one side,SubOrgan Organ,area of the photosynthetically active (non-senescent) part of the entity,green_area,,,,Adel: green_area volume,m3,extensive,volume,entity,,,SubOrgan Cell,volume of the entity,volume,,,,ArchiSimple: volume; GRANAP: cell_volume; MECHA: node_volume @@ -14,9 +14,9 @@ distance_from_tip,m,intensive,distance,entity,,along its axis,SubOrgan,distance azimuth,deg,descriptor,angle,entity axis,,from x towards y,SubOrgan Organ,azimuth of the entity's axis,azimuth,,,,ArchiSimple/Caribu: azimuth; Adel: organ_azimuth inclination,deg,descriptor,angle,entity axis,,from the vertical,SubOrgan Organ,inclination of the entity's axis from the vertical,inclination,,,,Adel/Caribu: inclination structural_mass,g,extensive,mass,structural dry matter,entity,,SubOrgan Plant,dry mass of the entity's structural matter,struct_mass,,,,ArchiSimple: struct_mass; metafspm example: struct_mass -age,s,intensive,duration,entity,,since its emergence,SubOrgan Organ,time since the entity emerged,age,,,,ArchiSimple: age; CPlantBox: age +age,s,intensive,duration,entity,,since its emergence,SubOrgan Organ,time since the entity emerged,age,age,,,"ArchiSimple: age; CPlantBox: age; cplantbox: proposed from Organ::getParameter (organ level, cm and d), to be confirmed by the CPlantBox team" thermal_time,K.d,intensive,thermal time,entity,,since its emergence,SubOrgan Organ Plant,"temperature-weighted time since the entity emerged, above the model's base temperature",thermal_time,,,,Adel: thermal_time (°Cd); ArchiSimple: thermal_time_since_emergence (s at a reference temperature) -branching_order,1,descriptor,order,entity's axis,,,Axis SubOrgan,"branching order of the entity's axis (1: an axis from the seed or the plant base, 2: its laterals, ...)",root_order,,,,ArchiSimple: root_order; CPlantBox: order +branching_order,1,descriptor,order,entity's axis,,,Axis SubOrgan,"branching order of the entity's axis (1: an axis from the seed or the plant base, 2: its laterals, ...)",root_order,order,,,"ArchiSimple: root_order; CPlantBox: order; cplantbox: proposed from Organ::getParameter (organ level, cm and d), to be confirmed by the CPlantBox team" water_potential,Pa,intensive,water potential,water,entity,,SubOrgan Cell Compartment,total water potential in the entity,water_potential,,,,metafspm: water_potential (MPa); MECHA: psi (hPa); CPlantBox: psi_x (cm) osmotic_potential,Pa,intensive,osmotic potential,water,entity,,SubOrgan Cell Compartment,osmotic component of the water potential in the entity,osmotic_potential,,,,MECHA: osmotic_potential (hPa) soil_water_potential,Pa,intensive,water potential,water,soil,,Voxel,total water potential of the soil water,soil_water_potential,,,,metafspm: soil_water_potential (MPa); MECHA: soil_psi (hPa)