From 900f3cf721e5579a78ac786ff65e8ba5cdd17e59 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Sat, 17 Oct 2026 18:25:23 -0500 Subject: [PATCH 1/4] docs(whats-new): changes in 2.3.5 Adds the 2.3.5 section to What's New in 2.3: the cascade-delete fixes, deletes restricted by another query, restricted populate keeping other workers' jobs, jobs completed_time precision, drop() removing the jobs table, alter() resolving foreign keys in the declaring module, MySQL error 1143, and Python 3.15. --- src/about/whats-new-23.md | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/src/about/whats-new-23.md b/src/about/whats-new-23.md index 4581266b..1f3895f4 100644 --- a/src/about/whats-new-23.md +++ b/src/about/whats-new-23.md @@ -2,7 +2,7 @@ DataJoint 2.3 adds a first-class **upstream read surface** — `Diagram.trace` and `self.upstream` — which make "a computed row derives only from its declared upstream inputs" easy to follow inside `make()` and easy to query afterward. It also ships the **SparkAdapter Codec Protocol** for typed rendering to Spark-native types, **`dj.deploy.set_replica_identity`** for PostgreSQL change-data-capture, and a **cascade fix** for Part-of-Part and renamed-foreign-key chains. -Later releases on the line added **redesigned diagram rendering** on the DataJoint brand palette, with a light/dark [`display.diagram_theme`](../reference/configuration.md#display-settings) setting, **S3 stores that resolve an ambient AWS identity** instead of requiring static keys, and **extrinsic provenance at the pipeline's entry points** — a hidden `_prov` attribute the framework records on every row arriving from outside. +Later releases on the line added **redesigned diagram rendering** on the DataJoint brand palette, with a light/dark [`display.diagram_theme`](../reference/configuration.md#display-settings) setting, **S3 stores that resolve an ambient AWS identity** instead of requiring static keys, and **extrinsic provenance at the pipeline's entry points** — a hidden `_prov` attribute the framework records on every row arriving from outside. 2.3.5 fixes **cascade deletes that removed rows outside the delete**, and a restricted distributed `populate` that removed other workers' jobs. > **Upgrading from 2.0, 2.1, or 2.2?** No API breaks — every feature on the 2.3 line is additive. > Two fixes in 2.3.3 do, however, reject input that previously passed silently: a misspelled native @@ -13,6 +13,25 @@ Later releases on the line added **redesigned diagram rendering** on the DataJoi > **Citation:** Yatsenko D, Nguyen TT. *DataJoint 2.0: A Computational Substrate for Agentic Scientific Workflows.* arXiv:2602.16585. 2026. [doi:10.48550/arXiv.2602.16585](https://doi.org/10.48550/arXiv.2602.16585) +## Changes in 2.3.5 + +2.3.5 is a patch release on the 2.3 line, and it is recommended for every 2.3 deployment: three of its fixes stop deletes from removing rows they should have kept. If you are upgrading from **2.3.4**: + +- **Cascade deletes remove only the rows they reach.** In three cases, a delete removed rows outside the cascade, without an error: + - `(T & dj.Top(n)).delete()` ignored the `Top` and deleted every row matching the other conditions, together with their descendants. `delete_quick()` did the same. The delete now starts from the rows the `Top` selects ([#1589](https://github.com/datajoint/datajoint-python/issues/1589)). + - A child table whose nullable foreign key shares a primary-key column with the parent lost the rows whose foreign key is NULL, although those rows reference nothing. In a diamond, a child reached through a secondary foreign key could instead be missed, and the delete failed with a misleading "unloaded schema" error. The cascade now copies a parent's condition to a child only when the foreign key carries every attribute the condition names, without renaming, and lies in the child's primary key. Otherwise it restricts the child by the restricted parent ([#1590](https://github.com/datajoint/datajoint-python/issues/1590)). + - With `part_integrity="cascade"`, deleting part rows deleted masters that had no matching part rows, and everything downstream of them. A master now joins the cascade only if one of its part rows did ([#1591](https://github.com/datajoint/datajoint-python/issues/1591)). + + `dj.Diagram.cascade(...).counts()` previews the corrected set. On earlier releases, restrict a delete by the primary keys it should remove: `(T & (T & cond).keys()).delete()`. + +- **A delete restricted by another query works.** `(Subject & (Session & "quality = 0")).delete()` raised `Attribute ... is not found in query` before deleting anything. On PostgreSQL, `(Subject & Session).delete()` deleted nothing and returned 0. A delete whose restriction contains a subquery now starts from the primary keys that restriction selects when the delete begins ([#1610](https://github.com/datajoint/datajoint-python/issues/1610)). +- **A restricted `populate(reserve_jobs=True)` keeps other workers' jobs.** `populate(*restrictions, reserve_jobs=True)` refreshed the jobs table against the restricted `key_source`, so every job outside the restriction looked obsolete. Reservations held by other workers and error records were deleted once they were older than `stale_timeout`. Cleanup now checks keys against the full `key_source`; the restrictions only limit which jobs are added. Success and error records of rows that were deleted upstream are now removed on every refresh instead of lingering for up to `stale_timeout` ([#1594](https://github.com/datajoint/datajoint-python/issues/1594); part of [#1577](https://github.com/datajoint/datajoint-python/issues/1577)). See [Distributed Computing](../how-to/distributed-computing.md). +- **Jobs record `completed_time` to the millisecond.** `reserved_time` was recorded with milliseconds but `completed_time` was truncated to the second on MySQL, so a job finishing within the second it was reserved appeared to complete before it started. Both are now taken from the server clock at the column's precision ([#1600](https://github.com/datajoint/datajoint-python/issues/1600)). +- **`drop()` removes a table's jobs table.** Dropping a `Computed` or `Imported` table left its `~~table_name` jobs table behind. A table later declared under the same name inherited it, and if its primary key differed, `populate(reserve_jobs=True)` silently computed nothing. `drop()` and `drop_quick()` now drop the jobs table too. A jobs table left by an earlier release, whose primary key does not match the table's, now raises on first use with the command that removes it ([#1601](https://github.com/datajoint/datajoint-python/issues/1601)). +- **`alter()` resolves foreign keys in the table's own module.** Called from another module (a migration script, or a notebook that imports the pipeline), `Table.alter()` looked up foreign-key references in the caller's namespace and failed with `Foreign key reference ... could not be resolved`. It now uses the namespace the table was declared in, as `Part.alter()` already did. Pass `context=` to override ([#1602](https://github.com/datajoint/datajoint-python/issues/1602)). See [Alter Tables](../how-to/alter-tables.md). +- **MySQL column-level privilege errors raise `AccessError`.** MySQL error 1143 (SELECT denied on a column) came through as a raw `pymysql` error, while the table-level 1142 already became `dj.errors.AccessError`. Both now raise `AccessError`, as on PostgreSQL ([#1586](https://github.com/datajoint/datajoint-python/issues/1586)). +- **Python 3.15 support.** `requires-python` now spans `>=3.10,<3.16`, and CI runs the suite on 3.15. On Python 3.15 only, `deepdiff` is held below 9 until its dependencies publish 3.15 wheels. DataJoint uses it only to check that a three-part `make` reads the same data twice, which behaves the same on 8.6. + ## Changes in 2.3.4 2.3.4 is a patch release on the 2.3 line. If you are upgrading from **2.3.3**: From 3b06ac2f1d1c66dd71de5ea4eaa546499810a221 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Sat, 17 Oct 2026 19:40:33 -0500 Subject: [PATCH 2/4] docs(whats-new): add the enum, rename and percent fixes to 2.3.5 --- src/about/whats-new-23.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/about/whats-new-23.md b/src/about/whats-new-23.md index 1f3895f4..f3d0ca6a 100644 --- a/src/about/whats-new-23.md +++ b/src/about/whats-new-23.md @@ -15,7 +15,7 @@ Later releases on the line added **redesigned diagram rendering** on the DataJoi ## Changes in 2.3.5 -2.3.5 is a patch release on the 2.3 line, and it is recommended for every 2.3 deployment: three of its fixes stop deletes from removing rows they should have kept. If you are upgrading from **2.3.4**: +2.3.5 is a patch release on the 2.3 line, and it is recommended for every 2.3 deployment: several of its fixes stop operations from removing data they should have kept. If you are upgrading from **2.3.4**: - **Cascade deletes remove only the rows they reach.** In three cases, a delete removed rows outside the cascade, without an error: - `(T & dj.Top(n)).delete()` ignored the `Top` and deleted every row matching the other conditions, together with their descendants. `delete_quick()` did the same. The delete now starts from the rows the `Top` selects ([#1589](https://github.com/datajoint/datajoint-python/issues/1589)). @@ -24,7 +24,9 @@ Later releases on the line added **redesigned diagram rendering** on the DataJoi `dj.Diagram.cascade(...).counts()` previews the corrected set. On earlier releases, restrict a delete by the primary keys it should remove: `(T & (T & cond).keys()).delete()`. -- **A delete restricted by another query works.** `(Subject & (Session & "quality = 0")).delete()` raised `Attribute ... is not found in query` before deleting anything. On PostgreSQL, `(Subject & Session).delete()` deleted nothing and returned 0. A delete whose restriction contains a subquery now starts from the primary keys that restriction selects when the delete begins ([#1610](https://github.com/datajoint/datajoint-python/issues/1610)). +- **On PostgreSQL, dropping a table no longer drops other tables' columns.** An enum attribute is declared with a PostgreSQL enum type named by its values, so tables whose enum attributes have the same values share one type. Dropping a table dropped its enum types with `CASCADE`, which also dropped every other column of that type, with its data. Every jobs table has the same `status` enum, so dropping one jobs table removed `status` from all the others in the schema. An enum type is now dropped only when no column uses it ([#1630](https://github.com/datajoint/datajoint-python/issues/1630)). +- **`alter()` keeps an attribute's values when it renames it.** A comment that opens with `{old_name}` renames that attribute. For a core-type attribute (`float32`, `int32`, and the like), `alter()` missed the marker and dropped the old attribute before adding the new one: on MySQL every value became `0.0`, or NULL for a nullable attribute; on PostgreSQL a nullable attribute lost its values. MySQL now renames with `CHANGE`. PostgreSQL refuses the rename before changing anything; renaming there comes in 2.4 ([#1631](https://github.com/datajoint/datajoint-python/issues/1631)). See [Alter Tables](../how-to/alter-tables.md). +- **A delete restricted by another query works.** `(Subject & (Session & "quality = 0")).delete()` raised `Attribute ... is not found in query` before deleting anything. On PostgreSQL, `(Subject & Session).delete()` deleted nothing and returned 0. A delete whose restriction contains a subquery now starts from the primary keys that restriction selects when the delete begins ([#1610](https://github.com/datajoint/datajoint-python/issues/1610)). A delete whose restriction contains `%`, such as `name = '50%'`, likewise deleted nothing, and now selects the same rows as the restriction on its own ([#1636](https://github.com/datajoint/datajoint-python/issues/1636)). - **A restricted `populate(reserve_jobs=True)` keeps other workers' jobs.** `populate(*restrictions, reserve_jobs=True)` refreshed the jobs table against the restricted `key_source`, so every job outside the restriction looked obsolete. Reservations held by other workers and error records were deleted once they were older than `stale_timeout`. Cleanup now checks keys against the full `key_source`; the restrictions only limit which jobs are added. Success and error records of rows that were deleted upstream are now removed on every refresh instead of lingering for up to `stale_timeout` ([#1594](https://github.com/datajoint/datajoint-python/issues/1594); part of [#1577](https://github.com/datajoint/datajoint-python/issues/1577)). See [Distributed Computing](../how-to/distributed-computing.md). - **Jobs record `completed_time` to the millisecond.** `reserved_time` was recorded with milliseconds but `completed_time` was truncated to the second on MySQL, so a job finishing within the second it was reserved appeared to complete before it started. Both are now taken from the server clock at the column's precision ([#1600](https://github.com/datajoint/datajoint-python/issues/1600)). - **`drop()` removes a table's jobs table.** Dropping a `Computed` or `Imported` table left its `~~table_name` jobs table behind. A table later declared under the same name inherited it, and if its primary key differed, `populate(reserve_jobs=True)` silently computed nothing. `drop()` and `drop_quick()` now drop the jobs table too. A jobs table left by an earlier release, whose primary key does not match the table's, now raises on first use with the command that removes it ([#1601](https://github.com/datajoint/datajoint-python/issues/1601)). From 9cc12c1208fc0a80d56d2fd7d87d7ff03c670a54 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Tue, 20 Oct 2026 18:57:49 -0500 Subject: [PATCH 3/4] docs(whats-new): the delete-seed fixes move to 2.4 --- src/about/whats-new-23.md | 1 - 1 file changed, 1 deletion(-) diff --git a/src/about/whats-new-23.md b/src/about/whats-new-23.md index f3d0ca6a..a5ce172b 100644 --- a/src/about/whats-new-23.md +++ b/src/about/whats-new-23.md @@ -26,7 +26,6 @@ Later releases on the line added **redesigned diagram rendering** on the DataJoi - **On PostgreSQL, dropping a table no longer drops other tables' columns.** An enum attribute is declared with a PostgreSQL enum type named by its values, so tables whose enum attributes have the same values share one type. Dropping a table dropped its enum types with `CASCADE`, which also dropped every other column of that type, with its data. Every jobs table has the same `status` enum, so dropping one jobs table removed `status` from all the others in the schema. An enum type is now dropped only when no column uses it ([#1630](https://github.com/datajoint/datajoint-python/issues/1630)). - **`alter()` keeps an attribute's values when it renames it.** A comment that opens with `{old_name}` renames that attribute. For a core-type attribute (`float32`, `int32`, and the like), `alter()` missed the marker and dropped the old attribute before adding the new one: on MySQL every value became `0.0`, or NULL for a nullable attribute; on PostgreSQL a nullable attribute lost its values. MySQL now renames with `CHANGE`. PostgreSQL refuses the rename before changing anything; renaming there comes in 2.4 ([#1631](https://github.com/datajoint/datajoint-python/issues/1631)). See [Alter Tables](../how-to/alter-tables.md). -- **A delete restricted by another query works.** `(Subject & (Session & "quality = 0")).delete()` raised `Attribute ... is not found in query` before deleting anything. On PostgreSQL, `(Subject & Session).delete()` deleted nothing and returned 0. A delete whose restriction contains a subquery now starts from the primary keys that restriction selects when the delete begins ([#1610](https://github.com/datajoint/datajoint-python/issues/1610)). A delete whose restriction contains `%`, such as `name = '50%'`, likewise deleted nothing, and now selects the same rows as the restriction on its own ([#1636](https://github.com/datajoint/datajoint-python/issues/1636)). - **A restricted `populate(reserve_jobs=True)` keeps other workers' jobs.** `populate(*restrictions, reserve_jobs=True)` refreshed the jobs table against the restricted `key_source`, so every job outside the restriction looked obsolete. Reservations held by other workers and error records were deleted once they were older than `stale_timeout`. Cleanup now checks keys against the full `key_source`; the restrictions only limit which jobs are added. Success and error records of rows that were deleted upstream are now removed on every refresh instead of lingering for up to `stale_timeout` ([#1594](https://github.com/datajoint/datajoint-python/issues/1594); part of [#1577](https://github.com/datajoint/datajoint-python/issues/1577)). See [Distributed Computing](../how-to/distributed-computing.md). - **Jobs record `completed_time` to the millisecond.** `reserved_time` was recorded with milliseconds but `completed_time` was truncated to the second on MySQL, so a job finishing within the second it was reserved appeared to complete before it started. Both are now taken from the server clock at the column's precision ([#1600](https://github.com/datajoint/datajoint-python/issues/1600)). - **`drop()` removes a table's jobs table.** Dropping a `Computed` or `Imported` table left its `~~table_name` jobs table behind. A table later declared under the same name inherited it, and if its primary key differed, `populate(reserve_jobs=True)` silently computed nothing. `drop()` and `drop_quick()` now drop the jobs table too. A jobs table left by an earlier release, whose primary key does not match the table's, now raises on first use with the command that removes it ([#1601](https://github.com/datajoint/datajoint-python/issues/1601)). From b1b4f41dce95820d13301a519f35bb27abc67e6d Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Wed, 21 Oct 2026 10:21:24 -0500 Subject: [PATCH 4/4] docs(whats-new): the drop() jobs-table fix moves to 2.4 --- src/about/whats-new-23.md | 1 - 1 file changed, 1 deletion(-) diff --git a/src/about/whats-new-23.md b/src/about/whats-new-23.md index a5ce172b..828301dd 100644 --- a/src/about/whats-new-23.md +++ b/src/about/whats-new-23.md @@ -28,7 +28,6 @@ Later releases on the line added **redesigned diagram rendering** on the DataJoi - **`alter()` keeps an attribute's values when it renames it.** A comment that opens with `{old_name}` renames that attribute. For a core-type attribute (`float32`, `int32`, and the like), `alter()` missed the marker and dropped the old attribute before adding the new one: on MySQL every value became `0.0`, or NULL for a nullable attribute; on PostgreSQL a nullable attribute lost its values. MySQL now renames with `CHANGE`. PostgreSQL refuses the rename before changing anything; renaming there comes in 2.4 ([#1631](https://github.com/datajoint/datajoint-python/issues/1631)). See [Alter Tables](../how-to/alter-tables.md). - **A restricted `populate(reserve_jobs=True)` keeps other workers' jobs.** `populate(*restrictions, reserve_jobs=True)` refreshed the jobs table against the restricted `key_source`, so every job outside the restriction looked obsolete. Reservations held by other workers and error records were deleted once they were older than `stale_timeout`. Cleanup now checks keys against the full `key_source`; the restrictions only limit which jobs are added. Success and error records of rows that were deleted upstream are now removed on every refresh instead of lingering for up to `stale_timeout` ([#1594](https://github.com/datajoint/datajoint-python/issues/1594); part of [#1577](https://github.com/datajoint/datajoint-python/issues/1577)). See [Distributed Computing](../how-to/distributed-computing.md). - **Jobs record `completed_time` to the millisecond.** `reserved_time` was recorded with milliseconds but `completed_time` was truncated to the second on MySQL, so a job finishing within the second it was reserved appeared to complete before it started. Both are now taken from the server clock at the column's precision ([#1600](https://github.com/datajoint/datajoint-python/issues/1600)). -- **`drop()` removes a table's jobs table.** Dropping a `Computed` or `Imported` table left its `~~table_name` jobs table behind. A table later declared under the same name inherited it, and if its primary key differed, `populate(reserve_jobs=True)` silently computed nothing. `drop()` and `drop_quick()` now drop the jobs table too. A jobs table left by an earlier release, whose primary key does not match the table's, now raises on first use with the command that removes it ([#1601](https://github.com/datajoint/datajoint-python/issues/1601)). - **`alter()` resolves foreign keys in the table's own module.** Called from another module (a migration script, or a notebook that imports the pipeline), `Table.alter()` looked up foreign-key references in the caller's namespace and failed with `Foreign key reference ... could not be resolved`. It now uses the namespace the table was declared in, as `Part.alter()` already did. Pass `context=` to override ([#1602](https://github.com/datajoint/datajoint-python/issues/1602)). See [Alter Tables](../how-to/alter-tables.md). - **MySQL column-level privilege errors raise `AccessError`.** MySQL error 1143 (SELECT denied on a column) came through as a raw `pymysql` error, while the table-level 1142 already became `dj.errors.AccessError`. Both now raise `AccessError`, as on PostgreSQL ([#1586](https://github.com/datajoint/datajoint-python/issues/1586)). - **Python 3.15 support.** `requires-python` now spans `>=3.10,<3.16`, and CI runs the suite on 3.15. On Python 3.15 only, `deepdiff` is held below 9 until its dependencies publish 3.15 wheels. DataJoint uses it only to check that a three-part `make` reads the same data twice, which behaves the same on 8.6.