Skip to content

gh-119696: Install build-details.json with relative paths - #63

Closed
itamaro wants to merge 133 commits into
mainfrom
build-details-relative-paths
Closed

itamaro wants to merge 133 commits into
mainfrom
build-details-relative-paths

Conversation

@itamaro

@itamaro itamaro commented Nov 7, 2026

Copy link
Copy Markdown
Owner

make install writes build-details.json (PEP 739) with absolute
paths, so the file is wrong once the installation is moved. Fixing
that is one of the steps towards relocatable CPython installations.

PEP 739 allows the paths to be relative to base_prefix, and
base_prefix to be relative to the file's directory;
generate-build-details.py already implements that behind
--relative-paths --config-file-path. Pass those flags from the
Makefile.

The installed-file tests now resolve the paths like a consumer would,
instead of skipping on a relocated installation.

vstinner and others added 30 commits November 7, 2026 22:35
Filter test_io.test_free_threading (need Free Threading),
test_io.test_winconsoleio (need Windows),
and test_io.test_file_eintr (need subprocess).
…ython#148928)

`_PyHamt_Assoc` allocates HAMT nodes, which can trigger GC. A weakref callback or finaliser fired during GC could re-enter `contextvar_set`, replacing `ctx->ctx_vars` via `Py_SETREF` and freeing the old HAMT while `_PyHamt_Assoc` was still traversing it, causing a use-after-free.

Hold a strong reference to `ctx->ctx_vars` during the `_PyHamt_Assoc` call, so that a re-entrant free cannot affect the HAMT being traversed.
…ython#161382)

The parser allocated an empty dict for **kwargs when there were no
keyword arguments.  Pass NULL instead, like the caller does, and handle
NULL in the impls that used the dict unconditionally.

All **kwargs impls were audited: _csv and zoneinfo already accept NULL,
and the rest (structseq and SimpleNamespace `__replace__`,
_interpreters.new_config, atexit.register, SubElement) now check for it.
atexit stores None, which the run loop already maps back to NULL.

pyperf, PGO+LTO, `t = time.gmtime(0)`, `ns = SimpleNamespace(a=1)`:

| Benchmark                       | main   | this change           |
|---------------------------------|:------:|:---------------------:|
| struct_time.`__replace__`()     | 229 ns | 47.0 ns: 4.87x faster |
| SimpleNamespace.`__replace__`() | 72.2 ns | 65.5 ns: 1.10x faster |

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ython#161156)

* pythongh-160872: Keep PyLong "x_divrem" inner loop carry in 64 bits

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
…ameError (python#161462)

These exceptions and their subclasses had no vectorcall, and `__init__`
parsed keywords through PyArg_ParseTupleAndKeywords with a temporary
empty tuple on every call.  Error messages are unchanged except when more
keywords than parameters are given, which now names the unexpected
keyword.

The clinic inputs have no docstring, so docstrings are unchanged and
these exceptions don't gain a `__text_signature__`.

pyperf microbenchmark (PGO+LTO, run serially):

| Benchmark                        | base    | vectorcall            |
|----------------------------------|:-------:|:---------------------:|
| AttributeError(s, name=s, obj=s) | 127 ns  | 41.9 ns: 3.02x faster |
| ImportError(s, name=s, path=s)   | 129 ns  | 43.4 ns: 2.96x faster |
| NameError(s, name=s)             | 88.3 ns | 39.3 ns: 2.24x faster |
| ModuleNotFoundError(s, name=s)   | 89.9 ns | 40.6 ns: 2.21x faster |
| AttributeError(s)                | 39.4 ns | 32.1 ns: 1.23x faster |
| UnboundLocalError(s)             | 38.6 ns | 31.8 ns: 1.22x faster |
| NameError(s)                     | 38.5 ns | 32.2 ns: 1.19x faster |
| ImportError(s)                   | 40.7 ns | 34.4 ns: 1.18x faster |
| attribute_error_to_none()        | 176 ns  | 173 ns: 1.02x faster  |

Benchmark code:

```python
import pyperf

runner = pyperf.Runner()
setup = """s = 'x'
class O: pass
o = O()
def attribute_error_to_none():
    try:
        o.x
    except AttributeError:
        pass
"""
for stmt in ["AttributeError(s)", "AttributeError(s, name=s, obj=s)",
             "NameError(s)", "NameError(s, name=s)",
             "ImportError(s)", "ImportError(s, name=s, path=s)",
             "ModuleNotFoundError(s, name=s)", "UnboundLocalError(s)",
             "attribute_error_to_none()"]:
    runner.timeit(stmt, stmt, setup)
```

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… faster binding (python#160727)

Pass the cached ' ' and '\n' strings for the default sep and end directly, instead of converting them from C strings with PyFile_WriteString(), which runs strlen() and the UTF-8 decoder on every call to get back the same cached string.

Rebinding file.write during print() keeps working as before. pythonGH-160803 significantly lowered binding cost. Could save ~5ns/call on x64 linux by binding only once, but that is left for work in https://discuss.python.org/t/make-print-use-a-single-write-call/109886

| Benchmark                                      | main    | this PR               |
|------------------------------------------------|:-------:|:---------------------:|
| print(file=devnull)                            | 51.2 ns | 47.4 ns: 1.08x faster |
| print('hello world', file=devnull)             | 81.0 ns | 79.2 ns: 1.02x faster |
| print('a', 1, 2.5, file=devnull)               | 281 ns  | 278 ns: 1.01x faster  |
| print('a', 'b', sep='-', end='', file=devnull) | 124 ns  | 126 ns: 1.02x slower  |

Benchmark hidden because not significant (1): print(*ten_ints, file=devnull)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
With `--with-lto` on macOS, linker flags include Make's `$@` target variable.
Immediate expansion in `PY_CORE_EXE_LDFLAGS` erased the target name,
causing executable links to share an LTO object path. Parallel links
could then produce incorrect executables and cause `test_embed` failures.

Use recursive assignment (`=` instead of `:=`) so `$@` expands when each
link recipe runs.
…59122)

* Move port validation logic to parsing time

* Fix CI. Handle the special case of file://C:\\  parsed by url2pathname
function where ":" should not be treated as port separator.

---------

Co-authored-by: Miguel Mendes <miguel.mdebrito@gmail.com>
…-compiling (python#161321)

When cross-compiling, `PYTHON_FOR_BUILD` uses the interpreter specified
by `--with-build-python`. With `--enable-shared`, running it with
`RUNSHARED` adds the target build directory to `LD_LIBRARY_PATH`. A
shared build Python can then load the target's `libpython` instead of
its own, failing if the target uses an incompatible libc or architecture.

Introduce `RUNSHARED_FOR_BUILD`, set to `$(RUNSHARED)` for native builds
and empty for cross builds, and use it for `PYTHON_FOR_BUILD` invocations
during building and installation.

Leave `RUNSHARED` unchanged for PGO profiling (pythongh-141926) and
`TESTPYTHON`, which uses `HOSTRUNNER` to launch the target interpreter.
Fix the warning on Windows when building Python in 32-bit:

    Modules\_uuidmodule.c(302,50): warning C4244: 'initializing':
    conversion from 'uint64_t' to 'Py_ssize_t', possible loss of data
…61626)

Fix compiler warnings when Python is built in 32-bit mode on Windows.
…161632)

We don't provide backward compatibility in the internal C API.
Fix warnigs:

* Include/internal/pycore_global_objects_fini.h:45:51: warning:
  comparison of integer expressions of different signedness:
  'digit' {aka 'unsigned int'} and 'long int' [-Wsign-compare]
* Modules/_testlimitedcapi/heaptype_relative.c:71:28: warning: cast
  from pointer to integer of different size [-Wpointer-to-int-cast]
* Objects/moduleobject.c:412:38: warning: cast from pointer to
  integer of different size [-Wpointer-to-int-cast]
* Objects/moduleobject.c:474:44: warning: cast from pointer to
  integer of different size [-Wpointer-to-int-cast]
* Python/getopt.c:163:49: warning: format '%lc' expects argument of
  type 'wint_t', but argument 3 has type 'wchar_t' {aka 'long int'}
  [-Wformat=]
* Python/getopt.c:177:55: warning: format '%lc' expects argument of
  type 'wint_t', but argument 3 has type 'wchar_t' {aka 'long int'}
  [-Wformat=]

In getopt.c, pass wint_t instead of wchar_t to printf("%lc").
BytesIO is commonly constructed in the test suite and gets significantly
faster to construct. Argument error messages are unchanged.

pyperf microbenchmark (PGO+LTO, run serially):

| Benchmark                    | base    | bytesio               |
|------------------------------|:-------:|:---------------------:|
| BytesIO(bytes)               | 54.2 ns | 27.0 ns: 2.01x faster |
| BytesIO(bytearray)           | 76.7 ns | 38.6 ns: 1.99x faster |
| BytesIO(initial_bytes=bytes) | 56.5 ns | 34.8 ns: 1.63x faster |
| BytesIO()                    | 40.9 ns | 25.4 ns: 1.61x faster |
| Geometric mean               | (ref)   | 1.48x faster          |

Benchmark hidden because not significant (2): `BytesIO.__new__`,
subclass(bytes)

Benchmark code:

```python
"""BytesIO() construction: AC @vectorcall vs tp_new + tp_init."""
import pyperf

runner = pyperf.Runner()

setup = "from io import BytesIO; b = b'abcdefgh'"
runner.timeit('BytesIO()', 'BytesIO()', setup)
runner.timeit('BytesIO(bytes)', 'BytesIO(b)', setup)
runner.timeit('BytesIO(bytearray)', 'BytesIO(x)', setup + '; x = bytearray(b)')
runner.timeit('BytesIO(initial_bytes=bytes)', 'BytesIO(initial_bytes=b)', setup)
runner.timeit('subclass(bytes)', 'S(b)', setup + '\nclass S(BytesIO): pass')
runner.timeit('BytesIO.__new__', 'new(BytesIO)',
              setup + '; new = BytesIO.__new__')
```

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Handle failures of update_slot(), since it can fail in the
default build due to non-str keys.
Bump default Emscripten version to 7.0.0.
…60175)

* pythongh-159739: Preserve large years in time conversions

The time module added 1900 to the C library year using int arithmetic, which overflowed for timestamps whose calendar year exceeds INT_MAX. On platforms that accept these timestamps, gmtime(), localtime(), and ctime() could report a negative year; perform the addition in a wider type. Backport to 3.15 and 3.14.

* pythongh-159739: Round-trip large years and validate datetime timestamps

Time tuple consumers parsed calendar years as C int before subtracting 1900, so they rejected valid struct_time results. Date and datetime timestamp conversion could also overflow before reporting the invalid year. Preserve the full time tuple range and report oversized datetime years correctly. Backport to 3.15 and 3.14.
)

Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
…ranch-free `find_max_char()` (python#122902)

Co-authored-by: Stan Ulbrych <stan@python.org>
Co-authored-by: Victor Stinner <vstinner@python.org>
…1409)

A hook added with PySys_AddAuditHook() before Py_Initialize() was
allocated with PyMem_RawMalloc() before the startup configuration was
applied, and freed with PyMem_RawFree() at finalization. When
PYTHONMALLOC=mimalloc, -X dev or a debug allocator changed the raw
allocator in between, the entry was freed with a different allocator
and the process crashed at exit.

Allocate and free the entries with plain malloc() and free(), which
nothing replaces at startup.
vstinner and others added 27 commits November 14, 2026 00:59
Check that no reference is leaked when _PyTuple_Resize() fails.
)

ECH hasn't been included in FIPS mode in OpenSSL so far.
If the UTF-8 mode is disabled and/or if PYTHONCOERCECLOCALE
environment variable is set to "0": mention it in the "== encodings"
regrtest header.

If the locale encoding and the filesystem encoding are equal,
log "locale=FS=<encoding>" to have a shorter output.

On Windows, log ANSI and OEM code pages.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
…umber_InPlaceMultiply` (python#161969)

Co-authored-by: Stan Ulbrych <stan@python.org>
… urlparse with non numeric port. (python#162011)

pythongh-158956: Update whatsnew section for the parsing difference in
urlparse with non numeric port.
…161789)

Resolve command objects through the path protocol before selecting text or
bytes handling. This preserves the documented bytes return type for
path-like objects backed by bytes.

Ref python#161788
…_` (python#161933)

When a class body binds __classdict__ (assignment, for target, global,
del), analyze_name() treated the name as global in the annotation and
type parameter scopes that can see the class, so they no longer got the
__classdict__ cell the class creates for them. Compiling such a class
with a generic class inside crashed, and with an annotated method or a
type alias raised SystemError.

Skip that rule for __classdict__ itself.
…15 (python#162022)

Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>
…1464)

Under Node, NODEFS takes a new file's mtime from the host kernel but
utime(path, None) uses Date.now() written back via fs.utimesSync, which
can land a millisecond before the creation time.  Take the baseline from
utime(path, None) too, so both timestamps come from the same source.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
python#161147)

Long ASCII runs now OR eight aligned words and test the high bits once,
one branch per 64 bytes (32 on 32-bit) instead of one per word.  When
a block contains a non-ASCII byte, the existing one-word loop finds it.
It is plain C: GCC and Clang compile it to SSE2/NEON loads, other
compilers to ORed word loads with the same single branch.

Multiple compiler and architecture codegen sample: https://godbolt.org/z/8bM83vx3a

pyperf, PGO+LTO, 2 interleaved rounds:

| Benchmark | main | this |
|---|---:|---:|
| (b'a' * 4096).decode() | 165 ns | 108 ns: 1.53x faster |
| (b'a' * 200).decode() | 28.9 ns | 26.5 ns: 1.09x faster |
| b'a' * 200 + 'é' + ..., .decode() | 60.9 ns | 56.0 ns: 1.09x faster |
| b'a' * 136 + 'é' + ..., .decode() | 53.8 ns | 51.3 ns: 1.05x faster |
| 8-100 byte decodes, 'é' within 100 | | within 1.2 ns |

```python
import pyperf

runner = pyperf.Runner()
tail = 'é'.encode() + b'a' * 50
for n in (8, 100, 200, 4096):
    runner.timeit(f"(b'a' * {n}).decode()", "s.decode()",
                  globals={"s": b'a' * n})
for n in (16, 100, 136, 200):
    runner.timeit(f"(b'a' * {n} + 'é'.encode() + b'a' * 50).decode()",
                  "s.decode()", globals={"s": b'a' * n + tail})
```

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
…w-up to pythonGH-24180) (python#154041)

bpo-42861: Added prev_network method, required refactor of next_network to prevent duplication.
…ory pressure (python#152255)

Co-authored-by: Victor Stinner <vstinner@python.org>
….prefix on Windows venvs (pythonGH-158465)

When a venv interpreter is invoked through a "\\?\" extended-length path, the
venv launcher reports its own path via GetModuleFileNameW, which preserves
that prefix. getpath.py consumed the value verbatim, so sys.prefix,
sys.exec_prefix and sys.executable all kept the prefix.

Windows does not resolve ".." or any other relative component in an
extended-length path, so tools that join them onto a sysconfig path break.
That is what makes "pip install" of a wheel with a "scripts=" entry fail with
"OSError: [Errno 22] Invalid argument".

Strip the prefix back off the launcher-provided path on Windows. A path that
would still exceed MAX_PATH once unprefixed keeps the prefix, since the
extended-length form is the only usable one for it.
Co-authored-by: Stan Ulbrych <stan@python.org>
…hon#161956)

Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
`make install` writes `build-details.json` (PEP 739) with absolute
paths, so the file is wrong once the installation is moved. Fixing
that is one of the steps towards relocatable CPython installations.

PEP 739 allows the paths to be relative to `base_prefix`, and
`base_prefix` to be relative to the file's directory;
`generate-build-details.py` already implements that behind
`--relative-paths --config-file-path`. Pass those flags from the
Makefile.

The installed-file tests now resolve the paths like a consumer would,
instead of skipping on a relocated installation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@itamaro
itamaro force-pushed the build-details-relative-paths branch from ff84334 to 22c061e Compare November 16, 2026 01:32
`make install prefix=...` (which the Android build uses) changes
`$(LIBDEST)` but not the prefix that `sysconfig` reports, so
`base_prefix` was computed between two unrelated paths. Add an
`--installed-location` flag that uses `sysconfig.get_path('stdlib')`.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@itamaro itamaro closed this Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.