darwin tarball binaries are not notarized at release time #67140

Thanks @sxa for bringing this to my attention. It has been a while since I last touched the macOS notarization, so I went back to it with fresh eyes.

Update 2026-10-31: the "no notarization ticket" findings below were produced with codesign --verify -R=notarized, which only reads the local ticket cache. Asking Apple's ticket service directly shows that 15 of the 18 most recent tarballs do have a ticket, created days after release by third parties that bundle our binary. Only v26.13.0 (both arches) and v26.12.0 x64 lack one. The root cause and the fix are unchanged; the gap is the first days after every release. Details in this comment. Struck text below is kept for context.

Context

We have had reports about badly signed or non notarized macOS binaries for a long time:

• macOS refuses to start node when downloaded from the tarball build#3538 (2023): Gatekeeper refuses to start node from the tarball, "Notary Ticket Missing".
• tools: add osx notarization support for tarballs #52610 (2024): my first attempt to notarize the tarballs, stalled on the Jenkins side and closed as stale.
• error: Signature will not be accepted by Gatekeeper! in macos15-release-pkg build build#4527 (2026-10-19): pkg notarization ticket not visible right after Accepted, fixed with a retry loop in tools: retry Gatekeeper check and stapling after notarization #66855.
• v26.13.0 Darwin tarball binaries are not notarized build#4545 (2026-10-29): v26.13.0 tarball binaries have no ticket, syspolicyd slows every spawn.
• macOS 26.3 arm64: Node v24.19.0 checksum-matched artifacts fail signature verification help#5173, Node.js Foundation Core Technical Committee (CTC) Meeting 2016-02-10 #5176, test: mark test flaky on windows again #5178 (October 2026): "invalid signature" reports on Node 24 artifacts that collaborators cannot reproduce.

Historically this was hard to reproduce locally, which is why #52610 never got the testing it needed.

[Solved] local reproduction at least to debug

I wrote a script that downloads the darwin tarball and the pkg for a version, verifies signatures, asks Apple whether a notarization ticket exists, and compares the tarball binary with the one inside the pkg. It needs no credentials and executes nothing it downloads, so anyone can run it on any modern macOS version and architecture: https://gist.github.com/UlisesGascon/ddce84a3f80521324e1c77bbfba12dba.

sh macos-validate.sh v26.12.1 v26.13.0 v24.22.1
DIST=https://nodejs.org/download/nightly ARCH=arm64 sh macos-validate.sh v28.0.0-nightly202610304e48b9b06b

Relevant output from my execution, same result for all three releases and for the nightly:

$ codesign --verify --strict --verbose=2 node-v26.13.0-darwin-x64/bin/node
node-v26.13.0-darwin-x64/bin/node: valid on disk
node-v26.13.0-darwin-x64/bin/node: satisfies its Designated Requirement
  exit=0
$ codesign --verify --verbose=2 -R=notarized node-v26.13.0-darwin-x64/bin/node
test-requirement: code failed to satisfy specified code requirement(s)
  exit=3

$ pkgutil --check-signature node-v26.13.0.pkg
   Status: signed by a developer certificate issued by Apple for distribution
   Notarization: trusted by the Apple notary service
  exit=0
$ xcrun stapler validate node-v26.13.0.pkg
The validate action worked!
  exit=0

  CDHash DIFFER: tarball=f2a61f63d9bcc07d74e1f67c443c85cff7a5cdd5 pkg=940e23498f45871c88f11bdf04d07760a4e243a7

So every tarball bin/node is correctly signed with our Developer ID but has no notarization ticket, on x64 for 24.22.1, 26.12.1 and 26.13.0, and on arm64 for today's nightly. Correction: the signatures are fine, but the "no ticket" results came from codesign -R=notarized, which only reads the local cache. Apple's ticket service shows tickets for 24.22.1 and 26.12.1 on both arches (created 2026-10-21 to 10-23, by third parties) and none for 26.13.0 and the nightly. Every pkg is notarized and stapled. The pkg's bin/node is a different build from the tarball's, so the pkg ticket does not cover it. The "invalid signature" errors in the help issues do not reproduce on 15.0.1, 26.4.1 or 27.0.1.

What is wrong here?

1. The binary target in the Makefile only runs tools/osx-codesign.sh. tools/osx-notarize.sh is only called by the pkg target, takes a version id and hardcodes node-$pkgid.pkg as the thing to submit. Tarball binaries have never been submitted to Apple.
2. On the Jenkins side the NOTARIZATION_* variables that gate the script are only exported on the pkg axis, so even a Makefile change would be a no-op on the tar nodes until the job is adjusted.
3. There is no verification step for tarballs. The pkg gets spctl --assess and a stapler check; the tar job has nothing, so an unticketed binary ships silently.
4. The one tarball known to carry a ticket (26.12.1 arm64, per v26.13.0 Darwin tarball binaries are not notarized build#4545) is the exception, probably from a manual submission during the test-http-regr-gh-2928 is flakey in parallel runs #4527 debugging, not evidence of a working path. Correction: most tarball binaries do end up with a ticket, one to seven days after release, because third parties notarize apps that bundle our signed bin/node and Apple issues a ticket for every nested binary. That is accidental and late, not a working path; 26.13.0 shows what the first days after a release look like. The pkg's bin/node cannot cover it: the pkg target configures with the default prefix and the tarball target with --prefix=/ and PORTABLE=1, so the two binaries always differ.

Proposal

Two PRs plus one Jenkins change:

1. nodejs/node: make tools/osx-notarize.sh take a path, keep the pkg flow as is, and add a binary flow: zip bin/node with ditto, submit with notarytool, verify that Apple's ticket service returns a ticket for the binary's cdhash, using the existing retry helper, fail the build if no ticket appears. Call it from the binary target after codesign. The binary is not modified, so the published checksums do not change. Backport to 22, 24 and 26.
2. nodejs/build: a tools/macos-tarball-notarization.sh that checks any published version for a ticket against Apple's ticket service (codesign -R=notarized is not a reliable check, see the update) and, with --submit and the release keychain profile, backfills the ones that lack it. Plus docs in release-overview.md describing the signing and notarization flow and how to enable it on a job axis.
3. ci-release iojs+release: export NOTARIZATION_ID and NOTARIZATION_TEAM_ID on the macOS tarball axes. NOTARIZATION_PASSWORD already reaches every axis through the job-level credentials binding apple-notarization-password, and the release image already has the cert and the NODE_RELEASE_PROFILE keychain profile on every macOS release VM, so nothing changes in ansible or orka. The job has one conditional step per axis type; the pkg one injects the variables, the tar one does not. The change is two lines in the tar step's "Inject environment variables" block:

 <org.jenkinsci.plugins.conditionalbuildstep.ConditionalBuilder>
   <runCondition class="org.jenkins_ci.plugins.run_condition.core.ExpressionCondition">
     <expression>osx1(\d|\d\d\d)-release-tar|macos1(\d|\d\d\d)-release-tar</expression>
     <label>${ENV, var="nodes"}</label>
   </runCondition>
   <conditionalbuilders>
     <EnvInjectBuilder>
       <info>
-        <propertiesContent>CODESIGN_CERT=Developer ID Application: Node.js Foundation</propertiesContent>
+        <propertiesContent>CODESIGN_CERT=Developer ID Application: Node.js Foundation
+NOTARIZATION_ID=REDACTED
+NOTARIZATION_TEAM_ID=REDACTED</propertiesContent>
       </info>
     </EnvInjectBuilder>
     <hudson.tasks.Shell>
       <command>... security unlock-keychain -p ${SSH_RELEASE_PASSWORD} ... make -j $JOBS binary-upload ...</command>
     </hudson.tasks.Shell>
   </conditionalbuilders>
 </org.jenkinsci.plugins.conditionalbuildstep.ConditionalBuilder>

Backlog

We can split into 2 goals (notarize on release and backfill) as they are independent and the priorities differ too.

Notarize on release

• @UlisesGascon create the Node PR: tools: notarize bin/node in the darwin tarballs #67143
  • land the PR
• @UlisesGascon fork the existing iojs+release job into a new one and execute it
• @UlisesGascon validate the results and decide if we apply the changes on iojs+release and remove the temporal job
• @UlisesGascon check the nighthly buils now are properly notarize

Backfill

• @UlisesGascon create the Build PR: Check and backfill notarization of darwin tarballs build#4549
  • land the PR
• @nodejs/build to decide if we want a pipeline job just to backfill
  • @UlisesGascon create the job to check and backfill recent releases in Jenkins release
• @UlisesGascon backfill the missing binaries without a ticket via ssh on a release VM
  • @UlisesGascon to check the total affected packages: see Check and backfill notarization of darwin tarballs build#4549 (comment). In non-EOL 136 already have a notarization ticket and 42 do not, (29 x64 and 13 arm64).