Skip to content

Check and backfill notarization of darwin tarballs - #4549

Open
UlisesGascon wants to merge 1 commit into
mainfrom
macos-tarball-notarization
Open

UlisesGascon wants to merge 1 commit into
mainfrom
macos-tarball-notarization

Conversation

@UlisesGascon

Copy link
Copy Markdown
Member

Build side of nodejs/node#67140, pairs with nodejs/node#67143. Adds a script that checks the published darwin tarballs for an Apple notarization ticket and, with credentials, submits the ones that lack it, so already shipped releases get fixed in place without changing any artifact or checksum. Documents how signing and notarization are wired across the pkg and tarball jobs and defines a nightly ci-release job to run it.

What the script does, per tarball:

  • Resolve the versions: explicit, or --all-supported --latest N from the Release schedule and index.json. Releases only.
  • Download the tarball, extract bin/node.
  • codesign --verify --strict. Invalid: report, no submission.
  • Ask Apple's ticket service for the binary's CDHash (the lookup macOS itself does; codesign -R=notarized only reads the local cache). Ticket present: skip.
  • Without --submit: report the missing ticket.
  • With --submit: check the signing team, zip with ditto, notarytool submit --wait through the NODE_RELEASE_PROFILE keychain profile, require Apple's Accepted verdict (print the notary log otherwise), poll until the ticket is visible.
  • Exit 0 all ticketed, 1 a published binary has no ticket, 3 a check could not complete (download, lookup, notary service).

Testing: check mode was run on an Intel Mac in all three forms, explicit version, --all-supported --latest 3, and a nightly via NODE_DIST_URL. The Apple ticket lookup was verified against 18 published tarballs (table below) and the CDHash extraction against codesign -dvvv; the script's check mode on the Mac reports the same 15 present and 3 missing. The submit path was exercised with stubbed codesign/ditto/notarytool only. I have no Apple credentials locally, so the plan is to try --submit after merge over SSH on an ephemeral release-ci VM, which has the keychain profile, and shape the script from whatever that first session surfaces. Expect a follow-up PR.

Ticket status of the 18 most recent tarballs, from Apple's ticket service, 2026-10-31
Tarball Release Ticket created (UTC)
v22.23.2 x64 / arm64 2026-07-26 2026-08-08 / 2026-07-29
v22.23.3 x64 / arm64 2026-09-22 2026-09-29 / 2026-09-28
v22.23.4 x64 / arm64 2026-10-20 2026-10-22 / 2026-10-21
v24.22.0 x64 / arm64 2026-10-11 2026-10-14 / 2026-10-14
v24.22.1 x64 / arm64 2026-10-20 2026-10-21 / 2026-10-21
v24.22.2 x64 / arm64 2026-10-27 2026-10-30 / 2026-10-29
v26.12.0 x64 / arm64 2026-10-13 none / 2026-10-15
v26.12.1 x64 / arm64 2026-10-20 2026-10-23 / 2026-10-22
v26.13.0 x64 / arm64 built 2026-10-28 none / none

Our own .pkg tickets are created within a minute of the pkg signature. Tarball tickets appear days later and get re-issued often, consistent with third parties notarizing apps that bundle our signed bin/node. The first days after a release, and any version nobody bundles, are uncovered.

Actions for release admins, not in this diff: Backfill the published tarballs (three binaries today: v26.13.0 x64 and arm64, v26.12.0 x64). Two options: run the script once over SSH on a release VM, or, if we agree the check is worth keeping, create the nightly notarize-darwin-tarballs job described in doc/jenkins-jobs.md, whose first run is the backfill and which afterwards detects a tarball axis that stopped notarizing. The doc section can be dropped if we settle on the one-off run.

Ref

Add tools/macos-tarball-notarization.sh. Given release versions, or
--all-supported for the lines currently in their support window, it
downloads the darwin tarballs, verifies the bin/node signature and asks
Apple's Gatekeeper ticket service, by code directory hash, whether a
notarization ticket exists. That is the lookup macOS itself performs;
`codesign --verify -R=notarized` only reads the local ticket cache and
reports a notarized binary as unticketed on a machine that never fetched
its ticket. A failed lookup is reported as such, never as a missing
ticket. With --submit it zips and submits the binaries that lack one
through the NODE_RELEASE_PROFILE keychain profile, requires Apple's
Accepted verdict, and waits for the ticket. --latest N limits the run to
the N most recent releases per line, since every check downloads a
tarball. Notarizing a published binary does not modify it, so checksums
stay valid; this is how already shipped tarballs get backfilled.

Document in release-overview.md how macOS signing and notarization are
wired, that tarball binaries usually receive a ticket only days after a
release when third parties notarize apps that bundle them, that the
NOTARIZATION_* variables in iojs+release must be exported on the tarball
axes too, and how to check and backfill a release. Describe in
jenkins-jobs.md the scheduled ci-release job that runs the script
nightly, which both backfills and detects a tarball axis that stopped
notarizing. Note in the orka README that pkg and tarball nodes share the
release image.

Refs: nodejs/node#67140
Refs: nodejs/node#67143
Refs: #3538
Refs: #4545
Assisted-by: Claude
Assisted-by: Codex
Signed-off-by: Ulises Gascon <ulisesgascongonzalez@gmail.com>
@UlisesGascon

Copy link
Copy Markdown
Member Author

After nodejs/node#67140 (comment) probably the pipeline is not needed, I am checking how many versions actually require submission. Based on that we can decide how relevant is this

@UlisesGascon

Copy link
Copy Markdown
Member Author

Full inventory done. I ran ./tools/macos-tarball-notarization.sh --all-supported to the check against every darwin tarball of the supported lines (v22, v24, v26; 89 releases, both architectures, 178 binaries): 136 already have a notarization ticket and 42 do not, (29 x64 and 13 arm64).

The 42 tarballs without a ticket
node-v26.13.0-darwin-x64
node-v26.13.0-darwin-arm64
node-v26.12.0-darwin-x64
node-v26.11.1-darwin-x64
node-v26.11.0-darwin-x64
node-v26.11.0-darwin-arm64
node-v26.10.0-darwin-x64
node-v26.9.0-darwin-x64
node-v26.8.1-darwin-x64
node-v26.8.0-darwin-x64
node-v26.8.0-darwin-arm64
node-v26.7.0-darwin-x64
node-v26.6.0-darwin-x64
node-v26.6.0-darwin-arm64
node-v26.5.1-darwin-x64
node-v26.2.0-darwin-x64
node-v26.2.0-darwin-arm64
node-v26.1.0-darwin-x64
node-v24.8.0-darwin-x64
node-v24.7.0-darwin-x64
node-v24.6.0-darwin-x64
node-v24.5.0-darwin-x64
node-v24.4.1-darwin-x64
node-v24.4.1-darwin-arm64
node-v24.4.0-darwin-x64
node-v24.4.0-darwin-arm64
node-v24.3.0-darwin-x64
node-v24.2.0-darwin-x64
node-v24.0.2-darwin-x64
node-v24.0.2-darwin-arm64
node-v24.0.1-darwin-x64
node-v24.0.1-darwin-arm64
node-v22.9.0-darwin-x64
node-v22.7.0-darwin-x64
node-v22.6.0-darwin-x64
node-v22.6.0-darwin-arm64
node-v22.5.0-darwin-x64
node-v22.5.0-darwin-arm64
node-v22.4.1-darwin-x64
node-v22.4.0-darwin-x64
node-v22.4.0-darwin-arm64
node-v22.3.0-darwin-arm64

Note: I ignored EOL lines, but happy to check if makes sense.

IMO we can just ssh into the machine and submit them and remove the new pipeline idea from the table. WDYT?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant