Skip to content

ci(release): publish to PyPI from an annotated tag - #2

Draft
dimitri-yatsenko wants to merge 5 commits into
mainfrom
ci/release-from-tags
Draft

dimitri-yatsenko wants to merge 5 commits into
mainfrom
ci/release-from-tags

Conversation

@dimitri-yatsenko

Copy link
Copy Markdown
Member

Releases of dj-plotly-codecs become annotated tags, the same process datajoint/datajoint-python#1588 adopts: a release is a tag vX.Y.Z on main whose message is the release notes, and pushing it builds, publishes to PyPI, and creates the GitHub release from that message. The package has no release yet, so this lands before the first one. The PR also makes the package ready for that release: it publishes to pypi.org, requires the DataJoint version its codec needs, and fixes the failing Format job.

Changes

Each item is one commit.

Publish a release from its annotated tag

  • cd.yml runs on a push of a vX.Y.Z tag instead of on a published GitHub release. Pull requests and pushes to main still build the package.
  • A check job runs .github/scripts/check_release_tag.sh, the script datajoint-python uses, unchanged. It refuses a tag that is not vX.Y.Z, a lightweight tag (and prints the git tag -a command that replaces it), and a tag whose commit is not on main.
  • The CI workflow runs as the test gate: ci.yml can now be called from cd.yml, and publishing waits for its lint and test jobs.
  • Before publishing, the publish job checks that the wheel and sdist carry the tag's version.
  • A release job creates the GitHub release from the tag's message, with the wheel and sdist attached.
  • A push to main or a pull request still replaces a build in progress, but a release run is never cancelled.
  • ci.yml names its concurrency group CI explicitly. In a run called from CD, github.workflow is CD, and a group shared with the caller makes GitHub cancel the run.
  • CONTRIBUTING.md has a new "Releasing" section.

Publish to PyPI, not TestPyPI: the publish step drops repository-url: https://test.pypi.org/legacy/.

Require datajoint 2.3.4: the codec calls self._extract_context(key, context) and self._codec_config(key, context), which DataJoint adds in 2.3.4. Under 2.3.0 to 2.3.3, which datajoint>=2.3 allowed, every encode and decode would fail. The README states the requirement, and uv.lock changes only in its recorded specifier.

Read the codec context only: DataJoint 2.3.4 and later pass context to every codec that declares it: schema, table, field, and config on insert, and config on fetch. The codec now reads them there and nowhere else, with no fallback to the underscore keys in key, which DataJoint 2.4 removes (datajoint/datajoint-python#1557). Without a context, it raises a DataJointError that says DataJoint 2.3.4 and later pass one.

Run the Format job on Python 3.14: "3.x" now resolves to 3.15. The job's pylint session installs datajoint, which pulls in deepdiff 9.1, and deepdiff 9.1 needs cachebox 5, which has no Python 3.15 build. The pin stays until datajoint 2.3.5 is on PyPI. No other job resolves to 3.15.

Why cd.yml and the pypi environment keep their names

A PyPI trusted publisher names the repository, the workflow file, and the environment. If either the file or the environment were renamed, PyPI would refuse the upload until its publisher was changed to match.

Before the first release

  • Pending trusted publisher on PyPI, under the DataJoint organization: project dj-plotly-codecs, repository datajoint/dj-plotly-codecs, workflow cd.yml, environment pypi. The project is created, owned by the organization, on the first upload.

  • Optionally, protect the pypi environment (Settings, Environments): add required reviewers, or limit deployments to tags matching v*. The environment does not exist yet. Without this step, GitHub creates it, unprotected, on the first release.

  • Merge this PR, then tag main with the release notes in notes.md and push the tag:

    git checkout main && git pull
    git tag -a v0.1.0 --cleanup=verbatim -F notes.md && git push origin v0.1.0

    The notes start with one sentence that says what the release is about. After a blank line come the changes, grouped by area, each naming its pull request. --cleanup=verbatim keeps Markdown headings, which git would otherwise drop as comments. The version comes from the tag through hatch-vcs, so nothing in the repository is edited.

Testing

  • actionlint, with shellcheck, passes on cd.yml and ci.yml, and so do the repository's pre-commit hooks.
  • check_release_tag.sh is byte-identical to datajoint-python's. On a scratch repository with main, it accepts an annotated tag on main and refuses a lightweight tag, a tag off main, and a tag that is not vX.Y.Z. It accepts a tag with a Retroactive: line but does not publish it.
  • A build at a trial tag v0.1.0 produces dj_plotly_codecs-0.1.0.tar.gz and dj_plotly_codecs-0.1.0-py3-none-any.whl, which the version check accepts. It refuses them for a v0.1.1 tag.
  • pytest with datajoint 2.3.4 and MySQL 8 in Docker: 30 passed, with warnings treated as errors, so no deprecation warnings either. The round trips through a real table exercise the context on insert and on fetch, and new tests check the error when there is no context.

A release is now an annotated tag vX.Y.Z on main whose message is the
release notes. Pushing the tag runs the CD workflow, which checks that the
tag is annotated and on main, runs the CI workflow as the test gate, checks
that the built files carry the tag's version, publishes them, and creates
the GitHub release with the tag's message as its notes.

The tag push replaces the "release published" trigger. The workflow file
keeps its name, cd.yml, and the publish job keeps the pypi environment,
because PyPI trusted publishing is bound to both. Pull requests and pushes
to main still build the package, and a release run is never cancelled by a
newer push.

CI can now be called from CD. Its concurrency group is named "CI"
explicitly: in a called run, github.workflow is the caller's name, and the
shared group would make GitHub cancel the run.

.github/scripts/check_release_tag.sh is the same script datajoint-python
uses. CONTRIBUTING describes how to cut a release.
The publish step no longer sets repository-url, so a release tag publishes
to pypi.org. Trusted publishing is unchanged: the same workflow file and the
same pypi environment.
The codec calls self._extract_context(key, context) and
self._codec_config(key, context), which datajoint adds in 2.3.4. Under
2.3.0 to 2.3.3, which datajoint>=2.3 allowed, _extract_context takes only
the key and _codec_config does not exist, so every encode and decode would
fail.

The README and the docs index state the requirement. uv.lock already locked
datajoint 2.3.4; only its recorded specifier changes.
DataJoint 2.3.4 and later pass context to every codec that declares it:
schema, table, field and config on insert, config on fetch. The codec now
reads them there and nowhere else. It no longer calls _extract_context and
_codec_config, which fall back to the underscore keys in key (_schema,
_table, _field, _config) when context lacks a value; DataJoint 2.4 removes
those keys. The primary key still comes from key, without its underscore
entries.

Called without a context, or with one that lacks a value it needs, the
codec raises a DataJointError that names the missing values and says that
DataJoint 2.3.4 and later pass them, before it reads or writes any file.

New tests call encode and decode without a context, with key filled the
way DataJoint 2.3 fills it, and expect that error. The round trips through
a real table check that DataJoint 2.3.4 passes the context on insert and on
fetch.
The job set up "3.x", which now resolves to Python 3.15. Its pylint session
installs the package and so datajoint, whose dependency deepdiff resolves
to 9.1, which needs cachebox 5. cachebox 5 has no Python 3.15 build and
fails to compile, so the job fails. It runs on 3.14 until datajoint 2.3.5,
which supports Python 3.15 and keeps deepdiff below 9 there, is on PyPI.

No other job resolves to 3.15: the test matrix names its versions, and the
build action in CD defaults to 3.14.
@dimitri-yatsenko dimitri-yatsenko added the enhancement New feature or request label Oct 20, 2026
@dimitri-yatsenko
dimitri-yatsenko marked this pull request as draft October 21, 2026 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant