Skip to content
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion src/about/whats-new-23.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

DataJoint 2.3 adds a first-class **upstream read surface** — `Diagram.trace` and `self.upstream` — which make "a computed row derives only from its declared upstream inputs" easy to follow inside `make()` and easy to query afterward. It also ships the **SparkAdapter Codec Protocol** for typed rendering to Spark-native types, **`dj.deploy.set_replica_identity`** for PostgreSQL change-data-capture, and a **cascade fix** for Part-of-Part and renamed-foreign-key chains.

Later releases on the line added **redesigned diagram rendering** on the DataJoint brand palette, with a light/dark [`display.diagram_theme`](../reference/configuration.md#display-settings) setting, **S3 stores that resolve an ambient AWS identity** instead of requiring static keys, and **extrinsic provenance at the pipeline's entry points** — a hidden `_prov` attribute the framework records on every row arriving from outside.
Later releases on the line added **redesigned diagram rendering** on the DataJoint brand palette, with a light/dark [`display.diagram_theme`](../reference/configuration.md#display-settings) setting, **S3 stores that resolve an ambient AWS identity** instead of requiring static keys, and **extrinsic provenance at the pipeline's entry points** — a hidden `_prov` attribute the framework records on every row arriving from outside. 2.3.5 fixes **cascade deletes that removed rows outside the delete**, and a restricted distributed `populate` that removed other workers' jobs.

> **Upgrading from 2.0, 2.1, or 2.2?** No API breaks — every feature on the 2.3 line is additive.
> Two fixes in 2.3.3 do, however, reject input that previously passed silently: a misspelled native
Expand All @@ -13,6 +13,25 @@ Later releases on the line added **redesigned diagram rendering** on the DataJoi

> **Citation:** Yatsenko D, Nguyen TT. *DataJoint 2.0: A Computational Substrate for Agentic Scientific Workflows.* arXiv:2602.16585. 2026. [doi:10.48550/arXiv.2602.16585](https://doi.org/10.48550/arXiv.2602.16585)

## Changes in 2.3.5

2.3.5 is a patch release on the 2.3 line, and it is recommended for every 2.3 deployment: several of its fixes stop operations from removing data they should have kept. If you are upgrading from **2.3.4**:

- **Cascade deletes remove only the rows they reach.** In three cases, a delete removed rows outside the cascade, without an error:
- `(T & dj.Top(n)).delete()` ignored the `Top` and deleted every row matching the other conditions, together with their descendants. `delete_quick()` did the same. The delete now starts from the rows the `Top` selects ([#1589](https://github.com/datajoint/datajoint-python/issues/1589)).
- A child table whose nullable foreign key shares a primary-key column with the parent lost the rows whose foreign key is NULL, although those rows reference nothing. In a diamond, a child reached through a secondary foreign key could instead be missed, and the delete failed with a misleading "unloaded schema" error. The cascade now copies a parent's condition to a child only when the foreign key carries every attribute the condition names, without renaming, and lies in the child's primary key. Otherwise it restricts the child by the restricted parent ([#1590](https://github.com/datajoint/datajoint-python/issues/1590)).
- With `part_integrity="cascade"`, deleting part rows deleted masters that had no matching part rows, and everything downstream of them. A master now joins the cascade only if one of its part rows did ([#1591](https://github.com/datajoint/datajoint-python/issues/1591)).

`dj.Diagram.cascade(...).counts()` previews the corrected set. On earlier releases, restrict a delete by the primary keys it should remove: `(T & (T & cond).keys()).delete()`.

- **On PostgreSQL, dropping a table no longer drops other tables' columns.** An enum attribute is declared with a PostgreSQL enum type named by its values, so tables whose enum attributes have the same values share one type. Dropping a table dropped its enum types with `CASCADE`, which also dropped every other column of that type, with its data. Every jobs table has the same `status` enum, so dropping one jobs table removed `status` from all the others in the schema. An enum type is now dropped only when no column uses it ([#1630](https://github.com/datajoint/datajoint-python/issues/1630)).
- **`alter()` keeps an attribute's values when it renames it.** A comment that opens with `{old_name}` renames that attribute. For a core-type attribute (`float32`, `int32`, and the like), `alter()` missed the marker and dropped the old attribute before adding the new one: on MySQL every value became `0.0`, or NULL for a nullable attribute; on PostgreSQL a nullable attribute lost its values. MySQL now renames with `CHANGE`. PostgreSQL refuses the rename before changing anything; renaming there comes in 2.4 ([#1631](https://github.com/datajoint/datajoint-python/issues/1631)). See [Alter Tables](../how-to/alter-tables.md).
- **A restricted `populate(reserve_jobs=True)` keeps other workers' jobs.** `populate(*restrictions, reserve_jobs=True)` refreshed the jobs table against the restricted `key_source`, so every job outside the restriction looked obsolete. Reservations held by other workers and error records were deleted once they were older than `stale_timeout`. Cleanup now checks keys against the full `key_source`; the restrictions only limit which jobs are added. Success and error records of rows that were deleted upstream are now removed on every refresh instead of lingering for up to `stale_timeout` ([#1594](https://github.com/datajoint/datajoint-python/issues/1594); part of [#1577](https://github.com/datajoint/datajoint-python/issues/1577)). See [Distributed Computing](../how-to/distributed-computing.md).
- **Jobs record `completed_time` to the millisecond.** `reserved_time` was recorded with milliseconds but `completed_time` was truncated to the second on MySQL, so a job finishing within the second it was reserved appeared to complete before it started. Both are now taken from the server clock at the column's precision ([#1600](https://github.com/datajoint/datajoint-python/issues/1600)).
- **`alter()` resolves foreign keys in the table's own module.** Called from another module (a migration script, or a notebook that imports the pipeline), `Table.alter()` looked up foreign-key references in the caller's namespace and failed with `Foreign key reference ... could not be resolved`. It now uses the namespace the table was declared in, as `Part.alter()` already did. Pass `context=` to override ([#1602](https://github.com/datajoint/datajoint-python/issues/1602)). See [Alter Tables](../how-to/alter-tables.md).
- **MySQL column-level privilege errors raise `AccessError`.** MySQL error 1143 (SELECT denied on a column) came through as a raw `pymysql` error, while the table-level 1142 already became `dj.errors.AccessError`. Both now raise `AccessError`, as on PostgreSQL ([#1586](https://github.com/datajoint/datajoint-python/issues/1586)).
- **Python 3.15 support.** `requires-python` now spans `>=3.10,<3.16`, and CI runs the suite on 3.15. On Python 3.15 only, `deepdiff` is held below 9 until its dependencies publish 3.15 wheels. DataJoint uses it only to check that a three-part `make` reads the same data twice, which behaves the same on 8.6.

## Changes in 2.3.4

2.3.4 is a patch release on the 2.3 line. If you are upgrading from **2.3.3**:
Expand Down
Loading