Some CI jobs are failing on Apple

🐛 Describe the bug

See https://hud.pytorch.org/hud/pytorch/executorch/main
The failures started yesterday afternoon

By looking at the logs I can say that those previous logs that succeeded mentioned macos-14-arm64 runners, and the failed ones are using macos-15-arm64.

There's a common error fetching a prebuilt torch wheel:

Example of a failing job:
https://github.com/pytorch/executorch/actions/runs/17279168142/job/49043504059

Versions

Current CI

Looks like issue is related to torchaudio:

MD -MT src/libtorchaudio/sox/CMakeFiles/libtorchaudio_sox.dir/types.cpp.o -MF src/libtorchaudio/sox/CMakeFiles/libtorchaudio_sox.dir/types.cpp.o.d -o src/libtorchaudio/sox/CMakeFiles/libtorchaudio_sox.dir/types.cpp.o -c /private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-x_jy67c1/src/libtorchaudio/sox/types.cpp
      ninja: build stopped: subcommand failed.
      
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 35, in <module>
        File "/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-x_jy67c1/setup.py", line 144, in <module>
          _main()
        File "/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-x_jy67c1/setup.py", line 99, in _main
          setup(
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 186, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
          dist.run_commands()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
          self.run_command(cmd)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/command/bdist_wheel.py", line 370, in run
          self.run_command("build")
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-x_jy67c1/tools/setup_helpers/extension.py", line 99, in run
          super().run()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 99, in run
          _build_ext.run(self)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 368, in run
          self.build_extensions()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 484, in build_extensions
          self._build_extensions_serial()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 510, in _build_extensions_serial
          self.build_extension(ext)
        File "/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-x_jy67c1/tools/setup_helpers/extension.py", line 187, in build_extension
          subprocess.check_call(["cmake", "--build", "."] + build_args, cwd=self.build_temp)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for torchaudio
  Running setup.py clean for torchaudio
Failed to build torchaudio
ERROR: Failed to build installable wheels for some pyproject.toml based projects (torchaudio)
ERROR conda.cli.main_run:execute(125): `conda run .ci/scripts/setup-macos.sh --build-tool cmake` failed. (See above for error)

Another error I see:

/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-485m887y/src/libtorio/ffmpeg/pybind/pybind.cpp
      ninja: build stopped: subcommand failed.
      
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 35, in <module>
        File "/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-485m887y/setup.py", line 144, in <module>
          _main()
        File "/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-485m887y/setup.py", line 99, in _main
          setup(
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 186, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
          dist.run_commands()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
          self.run_command(cmd)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/command/bdist_wheel.py", line 370, in run
          self.run_command("build")
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-485m887y/tools/setup_helpers/extension.py", line 99, in run
          super().run()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 99, in run
          _build_ext.run(self)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 368, in run
          self.build_extensions()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 484, in build_extensions
          self._build_extensions_serial()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 510, in _build_extensions_serial
          self.build_extension(ext)
        File "/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-485m887y/tools/setup_helpers/extension.py", line 187, in build_extension
          subprocess.check_call(["cmake", "--build", "."] + build_args, cwd=self.build_temp)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for torchaudio
  Running setup.py clean for torchaudio
Failed to build torchaudio
ERROR: Failed to build installable wheels for some pyproject.toml based projects (torchaudio)
ERROR conda.cli.main_run:execute(125): `conda run .ci/scripts/setup-macos.sh --build-tool cmake` failed. (See above for error)
Traceback (most recent call last):
  File "/Users/runner/work/executorch/executorch/test-infra/.github/scripts/run_with_env_secrets.py", line 102, in <module>
    main()
    ~~~~^^
  File "/Users/runner/work/executorch/executorch/test-infra/.github/scripts/run_with_env_secrets.py", line 61, in main
    run_cmd_or_die(f"bash { os.environ.get('RUNNER_TEMP', '') }/exec_script")
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/executorch/executorch/test-infra/.github/scripts/run_with_env_secrets.py", line 39, in run_cmd_or_die
    raise RuntimeError(f"Command {cmd} failed with exit code {exit_code}")
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1

Use macos-14-xlarge instead of macos-latest-xlarge which has switched… (

Fixed by #13821, which moved these jobs back to macOS 14 runners. The Apple jobs on main build torch and torchaudio fine today. Closing.

Navigation Menu

Repository navigation

Some CI jobs are failing on Apple #13773

🐛 Describe the bug

Versions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Some CI jobs are failing on Apple #13773

Description

🐛 Describe the bug

Versions

Activity

atalman commented on Aug 28, 2025

atalman commented on Aug 28, 2025

shoumikhin commented on Oct 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions