[napetrov]

Part of #432

Summary

Three tests failed for timing or rounding reasons, not because of a regression. Between 09-16 and 10-16 they cost about 2.8k runner-minutes in failed jobs and reruns.

 // Testing InnerProduct Distance (inner_product.cpp)
-margin = 0.03
+margin = max(0.03, N * FLT_EPSILON * sum(|a_i * b_i|))   // forward error bound of the fp32 sum

 // Timing (timing.cpp)
-require elapsed < sleep + 1 ms
+require elapsed < 5 * sleep          // guards against double counting, not scheduler noise

 // Basics / Spin While (threads/thread.cpp)
-sleep 1 ms; require worker progressed
+poll until worker progressed (10 s deadline)
+join worker, then assert            // a failed REQUIRE no longer std::terminates the binary

Evidence

CI history (09-16..10-16):

Test	Where it failed	Rate	Notes
InnerProduct	ARM, macOS (never x86)	18 ARM jobs (4.2%), 11 macOS (7.5%)	19 branches plus 3 main pushes; all 10 reruns of the same job passed
Timing	macOS	5 jobs (3.4%)	measured 20.0 ms for a 10 ms busy sleep
Spin While	ARM	1 job	aborted the whole binary (Subprocess aborted), passed on rerun

InnerProduct:

• The failure is always in the Float-Int8 section, on the generic (non-AVX) kernel generic_ip, which sums the fp32 products sequentially.
• It started after 03876d0 fixed RelWithDebugInfo. With NDEBUG defined, NTESTS went from 1,000 to 100,000.
• I reproduced it with a standalone copy of generic_ip<N> (float × int8, inputs in [-128, 127]), 4M trials per N, at -O2 and at -O2 -mfma -ffp-contract=fast:

before: N=160 fail=2/4M   N=223 fail=1/4M   (~4e-7 per compute; ~200k computes per test run -> ~5-10% of runs)
after:  N=160 fail=0/4M   N=223 fail=0/4M   worst error = 1.7% of the new margin

Locally on x86, RelWithDebInfo:

• ctest -R "^(Testing InnerProduct Distance|Timing|Basics)$" --repeat until-fail:10 passes.
• Timing and Basics also pass ×5 pinned to a single core (taskset -c 0).

Merge Danger

Door: two-way

Blast Radius: tests-only

• The InnerProduct margin is wider only when terms cancel. For typical inputs the 2% relative epsilon still decides the check.
• The Timing upper bound is looser: an accounting bug that inflates time by less than 5x would no longer be caught. The lower bounds are unchanged.
• No library code changes.

🤖 Generated with Claude Code