Python 3.14: crash or hang at exit while daemon threads scan object/Arrow data (not fixed by #709) #714

What happens?

On Python 3.14, a process whose daemon threads are still running queries that scan Python objects (an object-dtype pandas column, or a pyarrow table) crashes or hangs at interpreter exit. The same scripts exit cleanly on Python 3.13.

These are the daemon_threads_scanning_object_frame_at_exit and daemon_threads_scanning_one_chunk_arrow_table_at_exit cases in tests/slow/test_interpreter_shutdown.py. They fail intermittently on 3.14. Daemon threads that scan numeric or string[pyarrow] frames, or that only run range() queries, never fail.

#709 (end every open result before the interpreter finalizes) helps the object-frame case (37 → 11 failures per 100) but not the Arrow-table case (7 → 9), so this looks like a separate path: DuckDB's worker threads still enter the Python C API during 3.14 finalization, while the daemon threads' queries are running.

Failure signatures (a mix, from run to run):

• SIGSEGV, exit status -11, usually right after nanobind's leak report
• double free or corruption (out) followed by Fatal Python error: PyInterpreterState_Delete: remaining threads (Python runtime state: finalizing)
• Fatal Python error: PyGILState_Release: auto-releasing thread-state, but no thread-state for this thread
• Fatal Python error: gilstate_tss_set: failed to set current tstate (TSS)
• On Windows (CI): exit status 0xC0000409 (STATUS_STACK_BUFFER_OVERRUN)
• A hang at exit: the process never terminates and has to be killed

Failures per 100 runs of each test case, using the test's own pass rule. Linux aarch64, 12 runs in parallel:

With 3 runs in parallel, duckdb==2.0.0 on 3.14.7 failed 19 (object_frame) and 17 (one_chunk_arrow_table) out of 100. We have also seen both cases fail on macOS arm64 and Windows amd64 CI runners with the cp312-abi3 wheel on 3.14.

Possibly related: #601 (don't touch Python state from destructors that outlive the interpreter), #638 and #531.

One possible direction: before a scan callback takes the GIL from a DuckDB worker thread, check Py_IsFinalizing() (or _Py_IsFinalizing()). If finalization has started, fail the task with an interrupt instead of calling PyGILState_Ensure. Together with #709's atexit hook, that would cover a query that starts or resumes after the hook has run.

To Reproduce

import threading
import time

import duckdb
import pandas as pd

frame = pd.DataFrame({"a": pd.Series([f"v{i % 100}" for i in range(200_000)], dtype=object)})
con = duckdb.connect()


def work():
    cursor = con.cursor()
    while True:
        cursor.execute("select count(a), min(a) from frame").fetchall()


for _ in range(4):
    threading.Thread(target=work, daemon=True).start()
time.sleep(0.5)
print("done", flush=True)

Run it repeatedly on Python 3.14 with a timeout, because some runs hang:

for i in $(seq 1 100); do timeout 20 python repro.py >/dev/null 2>err.txt; echo $?; done | sort | uniq -c

With duckdb==2.0.0 from PyPI on Linux aarch64, one run at a time, the script above hung at exit in 4 of 100 runs on Python 3.14.7 and in 0 of 50 on Python 3.13.15. With 12 runs in parallel, as in the table above, the crashes appear as well.

OS:

Amazon Linux 2023 (aarch64); also seen on macOS arm64 and Windows amd64 GitHub Actions runners

DuckDB Package Version:

2.0.0

Python Version:

3.14.7 (3.13.15 is not affected)

Full Name:

Rusty Conover

Affiliation:

Query.Farm

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I tested with the latest release.

Did you include all relevant data sets for reproducing the issue?

Not applicable - the reproduction does not require a data set

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration to reproduce the issue?

• Yes, I have

🤖 Generated with Claude Code