Repository navigation
Draft
Conversation
… tiers, applicability-matrix gate, all-8-facade surface gates, writeable-flag schema + 6 real Core bugs and 3 leaks the new cells found Executes docs/plans/oracle-coverage-expansion.md (authored earlier today) end to end: every phase lands except the cells that require NEW NumSharp feature work (reduction-family out=, sum-family tuple-axis), which are now WARN cells in the new applicability gate rather than silent absences. Full ledger: test/NumSharp.Tests.Oracle/Fuzz/README.md → "The 2026-09-18 coverage expansion". Plan checkboxes updated in place with a status header. NEW TIERS ========= * instance.jsonl (5,414; gen_oracle.py `instance` mode; OpRegistry.Instance.cs) — the whole `ndarray.*` INSTANCE surface (plan row G0). Op keys carry the "ndarray." prefix (the ma. convention); MisalignedRegistry strips it at Classify entry so the shared excuse branches (float var/std order, complex envelopes, repeat-offset known bug…) cover the instance spellings. Covers: the dual-form methods through their INSTANCE defaults (a.max(axis), a.reshape(-1), a.round(n), a.astype/view/byteswap/getfield, argsort/argmax, cumsum/cumprod, clip/take/compress/searchsorted/choose/dot, flatten orders, transpose perms, diagonal/trace); instance-only members (item→scalar kind with NumPy's Python-scalar dtype mapping, tobytes→uint8 vector for C AND F order, __len__, view(dtype) same-size reinterprets, getfield windows); the §D4 property reads (T/mT/real/imag/flat/nbytes/itemsize/ndim/size/strides); nonzero's tuple (arity == ndim asserted); and the IN-PLACE mutators sort/fill/put/resize — §D3's operand-after comparator expressed with the EXISTING tuple machinery as [post-call view, post-call WHOLE base buffer], so a mutator that writes outside a strided view's window is caught (resize is array-kind: after resize the old base no longer exists; the registry resizes an OWNING copy because reconstructed operands don't own their buffer — the ownership/refcheck contract stays unit-suite territory). partition/argpartition are deliberately ABSENT: whole-output bytes between kth anchors are introselect-implementation- specific on BOTH sides (the sort tier pins DERIVED kth-values; the instance spelling delegates to the same kernel) — first generation proved it with 40+ spurious reds. * emath.jsonl (329; `emath` mode) — np.emath promoted into the differential corpus (§A2/E5): the real→complex promotion DECISION (any(x<0), |x|>1) + promoted values over the complex128/float64 lanes; int8/16/uint16/float32/float16 promote to NumPy complex64 (numpy _scimath _tocomplex) which NumSharp cannot represent (#569) — those lanes stay sibling-owned. Host-pinned (RunHostLibmCorpus). logn/power ride pair operands. 329/329 bit-exact first run. WIDENED TIERS ============= * out_where.jsonl 6,243→6,463: §B2 — out= beyond the elementwise ufuncs, for the surface NumSharp actually implements: out_scan (cumsum/cumprod; complex128 cumPROD carved exactly as nanscan carves it — host-FMA product chain + NumPy copies element 0 where a 1*z seed poisons both lanes through NaN), out_round (round_ decimals×out), out_clip (scalar 0-D bounds as operands; the read-only broadcast-out refusal included — np.clip refuses first, verbatim), out_nanarg (nanargmax/nanargmin axis×out). Same two-slot contract (returned view + ENTIRE out base buffer). The sum/prod/min/max family has NO out= overload yet — a tracked FEATURE gap the applicability matrix warns on, deliberately not generated. * params.jsonl 1,208→1,496: §C1 — multi-axis (tuple-axis) reduction cells, "axes" int[] param binding the int[] overloads, for exactly the reductions that HAVE them (median/average/ nanmedian × {[0,1],[-2,-1],[0,2],[0,1,2]} × keepdims × 3 layouts × 4 dtypes). Bit-exact. OpRegistry median/average/nanmedian gained the axes branch. * errors_full.jsonl 714→801 over 22→50 distinct NumPy messages: §B1 curated recipes over ops with verbatim-parity claims — reshape rejection family (wrong size, multi-unknown), expand_dims OOB both signs, flip axis-OOB + repeated-axis, matrix_transpose ndim, take/put OOB + float-index dtype rejection (same_kind vs safe rule split), partition kth OOB, percentile/quantile q-range, linalg validation (1-D / non-square / float16-unsupported — backend-free cells), fft n-guard. The regeneration ALSO picked up 29 fmod×complex cells the tier had never generated (fmod postdates the last regen) — exposing the missing guard fixed below. * nan.jsonl 120→176: §B3 — the binary special-pair CROSS grid (FLOAT_VALS × FLOAT_VALS, BOTH NaN signs), closing the audit's finding that the pair layouts align A[i] with B[i] so cross-operand NaN interactions (nan+finite, ±inf vs both NaN signs, mod-by-0, copysign-from-NaN, fmax NaN priority) never occurred: 19 ops at f64/f32, the 13-op widen-compute-narrow/bit-level subset at f16 (power/arctan2/hypot/logaddexp* are f32/f64-only — their f16 loops bridge differently), plus 5 complex128 binaries against a rolled grid (divide/multiply byte-exact ports; power under the registry's complex-binary excuse; complex-binary NaN sign stays tokenized — only the unary NDComplexMath family carries the sign pin). SCHEMA — operand.writeable (closed a SILENTLY-RED gate hole) =========================================================== Since 60024b4 deleted the K10 excuse, the OutWhere tier had been red on 293 cells and nobody had run it whole: a SAME-SHAPE np.broadcast_to keeps its source's ordinary strides, so the (dtype,shape,strides,offset,bytes) serialization reconstructed the read-only out as WRITEABLE and NumSharp computed a result where NumPy raises ValueError("output array is read-only"). Bisected via a worktree walk (green at d41eb4a, red from 60024b4 on; NOT caused by either in-flight session). layout_catalog.describe() now emits "writeable": false ONLY when the view is read-only beyond what its strides convey, and FuzzCorpus.Reconstruct honors it through the public setflags(write:false) route — emitted-only-when-False keeps every pre-flag corpus row byte-identical. This also UNBLOCKS the shift/logical out_where oracle wiring 59f9932 deferred on exactly this reconstruction gap. REAL BUGS THE NEW CELLS FOUND (all fixed, cells retained as regression proof) ============================================================================ * np.trace / ndarray.trace promoted the UNSIGNED narrow lanes (uint8/uint16/Char) to int64 where NumPy's add.reduce default-integer rule gives uint64 — wrong result dtype AND wrong wrap point. Invisible until now because the pre-existing trace corpus covered only int16/int32/int64/floats/ complex. Both maps fixed (DirectILKernelGenerator.Trace: TraceTypeInfo + GetTraceAccumTypeCode; Char→UInt64 per the uint16-proxy rule); np.trace docs corrected. * ndarray.flat over a 0-d array returned 0-d; NumPy yields shape (1,) (== np.ravel(0-d), probed 2.4.2). The IsScalar branch now reshapes to (1,); the two stale test pins in NDArray.flat.Test.cs updated with the parity note. * np.fmod with a complex operand leaked the kernel NotSupportedException — fmod landed AFTER the K4/K5 sweep and missed the verbatim no-loop TypeError guard every sibling has. Default.Fmod now raises NumPy's exact "ufunc 'fmod' not supported for the input types…" (dtype-keyed, so zero-size complex is rejected too). * np.take mode='raise' OOB text matched NEITHER NumPy spelling: an axis take says "index N is out of bounds for axis {k} with size M", the flat form drops the axis clause ("for size M"); NumSharp said "for axis with size". axis threaded into ExecuteTakeKernel (full docs added); both spellings now exact. * ndarray.item() on size>1 and __len__() on 0-d now raise NumPy's verbatim texts ("can only convert an array of size 1 to a Python scalar" / "len() of unsized object"); IncorrectSizeException added to the ValueError type map. * THREE pooled-buffer leaks (UndisposedIntermediateTests' zero-leak rule — fixed, never allowlisted; all PRE-EXISTED, the instance tier was simply the first corpus to replay those exact op×layout families): tobytes' materialized reorder copy on every non-directable layout; the f16 SUM's ascontiguousarray temp (flat strided/negstride) and float32 wide/halfResult intermediates (axis path); the f16 MEAN axis path's double intermediate before the narrowing Cast. The leak gate is back to exactly its pre-existing baseline (logspace/geomspace/piecewise/ loadtxt/fromfile/frombuffer — another session's claimed work per b14ff3a's commit message). * net8-ONLY: SIMD hypot returned |other|·√2 for hypot(NaN, finite). .NET 8's Vector256.Max/Min lower to the raw x86 vmaxpd/vminpd, which return the SECOND operand on an unordered compare — so a NaN lane produced a FINITE hi inside the safe band and skipped the scalar completion (.NET 9+ Max propagates NaN and hid it; found by the new binary NaN grid, which pairs NaN with finite for the first time). All three safe-gates in NDHypotMath.Simd.cs now AND in Vector256.Equals(v,v) ordered-operand guards — false only for NaN on every TFM. NEW GATES (plan §A / §M — the durability levers) ================================================ * OracleSurfaceCoverageTests now reflects ALL 8 [ModuleName] facades: `ndarray` (lowercase = NumPy-named, must be ndarray.*-corpus-covered or classified; PascalCase = C# infra by convention; the 25-property row included, T exempted from the uppercase rule), `np.emath` (100% corpus-covered — no classifications permitted), `np.ma` (ma.* keys + 13 aliases + the sibling ledger), `np.dtypes` (the 29 class properties, DTypes-suite-owned) — each with the stale-classification self-retirement rule (§A3), so a renamed member or newly-covered op fails instead of rotting. * OracleApplicabilityTests — the §M1-M3 mechanization: a 22-row declarative manifest of (taxonomy-group × assertion-kind) cells. Enforced cells assert op-count floors (MinCases-style ratchets — e.g. G1×outwhere ≥10 ops, G2×nan ≥15, G0×inplace ≥4); Warn cells PRINT their missing ops (the driven checklist for the next session: reduction out=, nan-reduce errors, G3/G9/G13/ma error recipes, precision widening); NotApplicable REQUIRES a reason and fails if coverage appears anyway; kinds a row omits entirely are covered by its one-line InapplicableNote. Applied kinds are computed from the corpus itself (out_* vehicle keys credit the ufunc named in their params; masked/masked_tuple normalize to array/tuple; zerod from 0-d operands/layouts; multiaxis from the axes param; inplace from the mutator set). EXCUSE BRANCHES (MisalignedRegistry) ==================================== * ndarray./emath. prefixes stripped at Classify entry (shallow-copy recurse — the ma. pattern). * argsort tie order: NumPy's default introsort is UNSTABLE, so ±0.0/duplicate ties resolve arbitrarily while NumSharp's radix argsort is stable. Excused ONLY when ArgsortPermutationsEquivalent re-derives from the operand bytes that both index vectors are in-range permutations selecting pairwise IEEE-equal values (self-limiting; the main sort tier uses distinct values and never reaches it). Trap encoded: identical indices must short-circuit BEFORE the IEEE compare, or an agreed NaN position vetoes the excuse (NaN != NaN). * K13 signed-zero extended to out_clip: clip IS maximum(minimum(x,hi),lo), so a ±0 result's sign inherits the same non-contractual SIMD-lane dependence; still guarded by every-diff-is-a-pure- ±0-flip. VERIFICATION ============ FuzzMatrix net10.0: FuzzCorpusTests 88/88 green (all tiers incl. the 6 touched + ma), gate classes green (surface ×5, applicability, strength, index, metamorphic, harness-self, random- isolation, journey3, BLAS delta); leak gate at its pre-existing baseline with ZERO new families. net8.0: 90/90 (the corpus + gate classes; the binary NaN grid found and now pins the net8 hypot fix). Main suite (CI filter, net10.0): 16,193 passed / 11 skipped / 1 failed — the one failure was the Indexing.KernelRoute pin of the OLD flat-take wording, updated to NumPy's verbatim flat spelling (with the two flat-0-d pins in NDArray.flat.Test.cs) — 193/193 green on the affected classes after. MinCases floors added/bumped (instance 4400, emath 260, out_where 6200, params 1190, errors_full 720, nan 140). The known teardown-crash flake (AccessViolation after all-passed) reproduced twice on full-category runs; per-class reruns are clean, per the documented protocol.
…harp's 15 dtypes, widen the narrow axes, and gate the spread (>=4 dtypes/op + per-dtype floors + a 61-entry reviewed ledger) Answers "do all oracles cover all dtypes NumSharp supports?" by measurement, closes the gaps that were generator artifacts, and turns the answer into a ratcheted gate so it stays true. Model: the NumPy corpus expresses 13 dtypes directly, Char rides the uint16 proxy weave, Decimal rides the independent C# oracle (decimal_*.jsonl); a dtype is COVERED for an op when it appears as an operand of a value OR error cell (a gated rejection IS coverage), with the params/expected dtype as the fallback for zero-operand creation ops. Ledger section: test/NumSharp.Tests.Oracle/Fuzz/README.md -> "The dtype-spread validation". THE SCAN ======== 612 op keys x 15 dtypes across every committed tier (+ the sibling oracles). Pre-pass spread: char 173 ops, decimal 124, uint64 217, uint32 245, int16 252 ... float64 579. The thin tail (< 4 dtypes: 75 ops) split cleanly into generator artifacts (fixable) and by-design narrowness (reviewable) — the two are now mechanically separated by the new gate. AXES WIDENED (all bit-exact on regeneration — ZERO new NumSharp divergences, itself a result: the char instance weave re-proves Char == uint16 across ~50 instance methods incl. the in-place mutators; the modf integer lanes gate 59f9932's per-width promotion differentially for the first time) ============================================================================================ * instance.jsonl 5,414 -> 9,463: INSTANCE_DTYPES 8 -> the 13 NumPy-expressible dtypes, plus char_tier("instance") — gen_instance is now parameterized (dtypes=None) so the whole tier re-runs on the uint16 proxy, dedicated dot/searchsorted/choose and resize jobs included; ndarray.view gains the int16/uint16/uint32/uint64 same-size reinterpret pairs. FIX in _relabel_dtype: it never recursed into kind=tuple SLOTS, so every char in-place mutator (sort/fill/put — slots [post-call view, base buffer], both proxy-dtyped) kept "uint16" slot dtypes after the relabel and failed on a dtype mismatch. Slots now relabel. * modf.jsonl 128 -> 208: MODF_DTYPES was the stale pre-promotion list (f16/f32/f64/int32); 59f9932's per-width promotion (bool/i8/u8->f16, i16/u16->f32, i32+->f64) makes every non-complex lane computable — all 12 generated + char via the new char_tier("modf") branch. * nanreduce.jsonl 10,266 -> 10,416: nanpercentile/nanquantile gain the integer/bool lanes (legal NumPy — no NaN can occur, so it degenerates to percentile; the degenerate path is exactly what was ungated). Generator trap fixed: the NaN-laced float pool RAISES at array construction for unsigned/bool (NumPy 2.x bounds-checks python ints) and a float->uint astype is C-UNDEFINED — integer lanes build an int64 pool and modular-astype it instead. * emath.jsonl 329 -> 482: the unsigned lanes (uint16/uint32/uint64) — safe by construction: no negatives means the any(x<0) promotion never fires, and |x|>1 cells that would land complex64 are auto-skipped by the existing result-dtype guard. * out_where.jsonl 6,463 -> 6,631: float16 + uint8 lanes through out_scan/out_round/out_clip/ out_nanarg (clip keeps its complex skip, cumprod its complex carve). THE GATE — OracleCoverageStrengthTests.EveryOrdinaryOp_MeetsItsDtypeSpreadFloor ============================================================================== Same file scope as the case-count gate (host pins, index_, ma_ excluded — ma has its own coverage model). Two rules, both ratchets: 1. every op reaches >= 4 distinct dtypes OR carries a one-line reviewed entry in FixedDtypeOps (61 entries in five classes: fixed-output generators — windows/fftfreq/ index-coordinate builders; dtype-axis-in-PARAMS — can_cast/promote_types/min_scalar_type sweep dtype PAIRS the operand metric cannot see; PRNG protocol; the host-pinned LAPACK/CBLAS f32/f64/c128 family; variant keys whose primary op carries the axis, e.g. modf-tuple vs modf_frac/modf_int's 13). Self-retiring BOTH ways: an entry whose op vanished, or whose spread grew past the floor, fails — the ledger tracks reality. 2. fifteen per-dtype GLOBAL op-count floors (~95% of the post-widening spread: char 216, decimal 117, uint64 261 ... float64 417), so a regeneration that silently drops a dtype axis turns the gate red instead of shrinking quietly. SIBLING ORACLES (validated as-is, posture recorded in the README) ================================================================= .npy format oracle: full dtype-map coverage incl. <U1/Char, big-endian variants and the <c8 widening; Decimal's rejection is itself a gated case. Advanced-indexing oracle: a dedicated 13-dtype tier (index_dtype.jsonl) + setter-dtype tier. Flags + layout-parity oracles: 13-dtype spot sweeps around an int64/f64 bulk — correct for their contract (flags/view semantics depend on itemsize, not lane type; the flags oracle is under active parallel work and untouched here). Decimal stays gated exclusively by decimal_*.jsonl (124 op keys) by design. VERIFICATION ============ net10.0: FuzzCorpusTests 88/88 (all widened tiers green), the full gate family green (surface x5, applicability, strength incl. the new dtype gate, case-count, index floors), leak gate exactly at its pre-existing baseline (logspace/geomspace/piecewise/loadtxt/fromfile/frombuffer — another session's claimed work) with ZERO new families from the ~5,700 new rows. net8.0: corpus + gates green (dtype gate + Instance/Modf/NanReduce re-run explicitly after the known teardown-abort flake truncated the batch run). MinCases floors bumped: instance 7500, emath 385, modf 165, out_where 6300, nanreduce 8300.
…lections.Concurrent An API-proposal + design doc pitching OrderedDict as a new BCL collection: the concurrent, insertion-ordered, index-addressable dictionary the framework currently lacks. .NET 9's OrderedDictionary<TKey,TValue> is ordered+indexed but NOT thread-safe; ConcurrentDictionary is thread-safe but unordered — no member is all three at once. This is the reference implementation's case for filling that cell. docs/proposals/ConcurrentOrderedDictionary.md: - Gap-analysis table (Dictionary / OrderedDictionary<,> / ConcurrentDictionary vs proposed) + the "manually locking an OrderedDictionary serializes reads too" motivation. - Goals/non-goals — deliberately scopes out lock-free writers, arbitrary positional insert, and O(1) interior remove, so reviewers are not surprised. - Full ref-assembly API surface proposed as System.Collections.Concurrent.ConcurrentOrderedDictionary<TKey,TValue>: the ConcurrentDictionary atomic combinators (TryAdd / TryUpdate / TryRemove / GetOrAdd / AddOrUpdate incl. the TArg overloads) fused with the OrderedDictionary positional read surface (this[int], IndexOf, GetKeyAt/GetValueAt), the right interface set, and the int-key indexer footgun noted. - Concurrency specification (the load-bearing section): lock-free reads, lock-free atomic value replace, serialized structural writes, Volatile/release-acquire (Arm64-safe), snapshot enumeration, a per-op complexity table including the honest O(n) order-preserving remove, and a "what is NOT a fault" list (last-writer-wins, add-not-yet-visible, indexer-races-remove). - Measured evidence: vs OrderedDictionary<,> reads ~parity + enum 1.53x (build/replace the modest thread-safety tax); vs ConcurrentDictionary 2.3-41x single-thread + 2.8x concurrent-read throughput (~914M vs ~325M lookups/sec); concurrency gun ~280M ops with 0 faults + a focused concurrent-get-plus-write test (195.7M reads + 6.0M writes on the SAME keys under resize churn, 0 faulty reads, 0 lost writes). - Alternative designs, risks, open questions (naming, positional mutation, swap-back removal). Reference implementation already on master: src/NumSharp.Core/Collections/Concurrent/OrderedDict.cs (6cf21af).
Upgrades the coverage pipeline to schema v5 by adding `objectTypes` reflection data (including inherited member ownership), object-surface cataloging, and stricter scope projection for dashboard/API counts. It also introduces an offline NumPy docs-link snapshot with anchor-aware auditing, new CI contract tests (Python + Node), and refreshed coverage docs/dashboard behavior. Additionally, the flags oracle was extended with a new recipe wave (87 -> 100 recipes) and regenerated corpus/test expectations to cover previously missing view/copy and boolean-mask flag scenarios.
…harp2 Reference-only clone (pinned at 7334cc3b, PR #218) matching the refs/ convention — every reference tree here is a committed submodule (numpy, pocketfft, OpenBLAS, onnxruntime, parquet-dotnet, data). Added via `git submodule add https://github.com/SciSharp/TensorSharp2.git refs/TensorSharp2`. WHAT IT IS (for future reference): Zhongkai Fu's native .NET LLM inference engine for GGUF models — the Cceusor -> Seq2SeqSharp -> TensorSharp lineage grown into a full local-inference + agent stack (~519K LOC C#, 43 projects). Ships a CLI, browser chat UI, and Ollama/OpenAI-compatible HTTP APIs; runs a dozen autoregressive families (DeepSeek V4/V4.1, GLM 5.x, Qwen 3.5/3.6/3.8, Gemma 4, GPT-OSS, Nemotron-H, Mistral 3, Muse-Glimmer) plus text-diffusion, image edit, and video+audio generation. Benchmarked head-to-head against llama.cpp on identical GGUFs. WHY IT IS A USEFUL REFERENCE FOR NUMSHARP: it is the OTHER SciSharp tensor library, and its core (TensorSharp.Core, only ~11K LOC) is the opposite design point to NumSharp's NumPy port — a Torch/TH-style tensor worth studying by contrast: - Tensor = mutable `class : IDisposable` (sizes[] + strides[] + refcounted Storage + storageOffset), NOT a value-ish NumPy NDArray; `RefCounted` base is CAS+SpinWait AddRef/Release with a finalizer safety net (cf. NumSharp ARC). - DType is a 6-entry enum — Float32/Float16/Float64/Int32/UInt8 + GGML block quants Q8_0/Q4_0 — i.e. exactly what GGUF inference needs, vs NumSharp's 15 dtypes + full NEP50 promotion. No Complex/Decimal/Char/bool, no promotion table. - Ops are result-tensor-IN (`Ops.Add(result, lhs, rhs)`, Torch convention) routed through a reflection-based OpRegistry: backends tag classes `[OpsClass]` and methods `[RegisterOp*]`/`[ArgConstraint]`; dispatch matches op-name + storage type. Device = IAllocator (BlasEnum DotNet/MKL/CUDA/GGML_METAL/GGML_CPU/MLX), so one Tensor type transparently lives on CPU/CUDA/Metal. Contrast NumSharp's TensorEngine + IL-emitted SIMD kernels + bit-exact-NumPy-parity goal. - Backends: managed CPU (from-scratch SGEMM under Cpu/LinearAlgebra, LSAME/XERBLA BLAS-reference lineage), direct CUDA (Driver API + cuBLAS + PTX), Apple MLX, and a native C++ ggml bridge (TensorSharp.GGML.Native). AGENTS.md: keep upstream ggml unmodified; TensorSharp-owned code implements anything ggml lacks. Layering above Core: Runtime (GGUF/tokenizers/sampling, paged KV cache, continuous-batching scheduler, speculative decoding) -> Models (13 architectures + multimodal) -> AgentHost (skills + sandboxed code exec) -> Chat (host-neutral pipeline) -> Server/Cli/TensorAgent(iOS). Not built or referenced by NumSharp; it sits in refs/ purely for study, like the other reference trees.
…ng[]/Shape overloads)
np.indices and np.indices_sparse accepted ONLY int[] for the grid dimensions,
which is at odds with the whole house shape vocabulary — NDArray.shape is long[],
Shape.Dimensions is long[], and np.zeros/np.ones/np.empty already offer the full
int[]/long[]/Shape overload set. A caller holding their own array's shape had to
manually narrow it to 32 bits to reach np.indices, risking silent truncation of a
dimension that only fits in 64 bits.
This mirrors the ndindex/zeros overload pattern: one long-native core with thin
spellings on top.
## API surface (both dense np.indices and sparse np.indices_sparse)
- indices(int[] dimensions, DType dtype = null) — KEPT for back-compat + the
oracle harness; widens to the long core (int[] does not convert to long[] by
array covariance, so it needs its own entry).
- indices(long[] dimensions, DType dtype = null) — NEW, THE house shape type.
`np.indices(arr.shape)` now binds here with no down-cast — the friction this
fixes. Carries `[NDScoped]` (the astype temp is born here).
- indices(Shape dimensions, DType dtype = null) — NEW. Accepts an NDArray.Shape
directly, and lets a C# value tuple read like Python: `np.indices((2, 3))`.
Overload resolution is unambiguous: an exact-type parameter (int[]->int[],
long[]->long[]) always wins over Shape's implicit int[]/long[] conversions, so
every existing `np.indices(intArray)` call still binds the int[] overload (verified:
OpRegistry, benchmark, MaskedArray all recompile clean).
## Core made long-native
BuildDenseIndices / BuildSparseIndicesAsArray now take long[]. The IL fill kernel
(DirectILKernelGenerator.Indices) was ALREADY fully long-typed (dimStrides/dims/
prod/result are long), so nothing changed there — only the public param and the
`int dim` reads were the 32-bit chokepoint. The dense builder now pins the long[]
dims array directly for the kernel (dropping the per-call int->long scratch copy),
and the sparse builder uses arange(0L, dim). Overflow detection on the running
element-count product is retained (long-native).
Behaviour parity: values, dtype, empty-dims (-> (0,)), and negative-dim rejection
all match the former int[] path bit-for-bit (differential-fuzz creation tier green).
A dimension > int.MaxValue paired with a 0 now yields shape (2, 2147483648, 0),
size 0, allocation-free — the old API could not even express it, and a silent
(int) narrowing would have flipped it negative and thrown.
Deliberate, documented convenience divergence (Shape overload only): because a
scalar converts implicitly to a 1-D Shape, `np.indices(5)` is accepted as the grid
(1, 5); NumPy's `np.indices(5)` raises (an int is not a sequence). Spell a 1-D grid
as `np.indices((5,))` to stay verbatim. Same class as zeros' scalar convenience.
## Internal consumers cleaned up (they existed BECAUSE of the int[]-only limit)
- np.mgrid.Build: was `int[] sizes = checked((int)MeshAxisSize(...))` then re-widened
to `long[] full` right after — a pure long->int->long round-trip forced by the
int[]-only indices, plus an artificial 2^31 grid ceiling. Now `long[] sizes`
straight through; `fullShape = new Shape(sizes)`. Negative-size and zero-step
behaviour unchanged (indices still rejects negatives; MeshAxisSize still throws
on a zero real step).
- MaskedArray.notmasked_edges: `np.indices(d.shape.Select(x => (int)x).ToArray())`
-> `np.indices(d.shape)` (d.shape is long[]).
- MaskedArray.indices wrapper: added long[] and Shape overloads beside the int[] one.
## Tests
- test/NumSharp.Tests/Indexing/np.indices.Test.cs (NEW, 14 tests): spelling parity
(int[]==long[]==tuple==arr.shape==arr.Shape), known 2x3 values, dtype, empty,
negative-throws, the >int.MaxValue non-truncation case, sparse parity + shapes,
the mgrid regression, and the ma.indices long/Shape overloads.
- Green: np_indices (14), mgrid+ogrid+MaskedArray (123, incl. notmasked_edges with
axis), creation FuzzMatrix tier (indices/indices_sparse differential), and
OracleSurfaceCoverageTests (5). Core/test/Oracle build 0W/0E.
…ples, dtype-safe
Convert the DocFX documentation examples from `new[]{...}` / `new int[,]{...}`
array-creation syntax to C# 12 collection expressions `[...]` (and `()` tuples
where a shape/tuple overload exists) wherever it is BEHAVIORALLY IDENTICAL —
so the examples read like NumPy's own `np.array([...])` — while carefully
preserving every case where `[]` would change semantics or dtype.
Every replacement class was verified empirically against NumSharp.Core
(compile + run), not assumed:
np.array(new[]{1,2,3}) -> np.array([1,2,3]) int32, identical
np.array(new int[,]{{..}}) -> np.array([[..],[..]]) 2-D, identical
np.array(new long[]{1,2,3}) -> np.array([1L,2L,3L]) int64 PRESERVED (L suffix)
np.array(new long[,]{..}) -> np.array([[10L],[20L]]) int64 PRESERVED
np.array(new float[,]{..}) -> np.array([[5.1f,..],..]) float32 PRESERVED (f suffix)
np.transpose(x, new[]{..}) -> np.transpose(x, [..]) int[] arg (no long[] twin)
np.hsplit(x, new[]{3,4}) -> np.hsplit(x, [3,4])
np.indices(new[]{3,3}) -> np.indices([3,3])
np.loadtxt(usecols:new[]{..})-> usecols:[..]
nd.AsDataView(new[]{"a",..}) -> nd.AsDataView(["a",..]) string[] overload
np.concatenate(new[]{a,b}) -> np.concatenate([a,b]) still an array (prose stays valid)
DELIBERATELY KEPT AS `new[]` — these are NOT convertible (verified):
- Fancy indexing x[new[]{0,2}] stays: a collection expr `[0,2]` binds to the
COORDINATE indexer (x[0,2]), not the fancy int[] one — it returns a scalar or
throws "too many indices", NOT the row-select. Only the np.array(...) DATA
wrapped inside such index expressions was converted; the raw index was not.
- Narrow integers new sbyte[]/byte[]/short[] stay: no C# literal suffix, so
`[-1,0,1]` infers int (int32), not int8. Added a note in dtypes.md and
array-creation.md documenting the suffix rule (L/u/f/m widen inline; the
narrow types need `new T[]` or a per-element cast).
- var names = new[]{...} stays: `var x = [...]` has no target type (won't compile).
- np.zeros(new[]{3,4}) in NDArray.md stays: it teaches int[]->Shape, and
`np.zeros([3,4])` is C#12-ambiguous between the int[]/long[] overloads.
- Empty typed (new double[0]); NDIter / nested_iters operand/dtype/flag arrays;
reflection Type[]; np.block object[][]; ORT session.Run; System.Numerics.Tensors;
and byte[] buffers — all left as-is.
Method: 148 bulk 1-D conversions applied by a tightly-scoped, dry-run-reviewed
regex anchored to `np.array(` (so it can never reach a fancy index, np.concatenate,
or an NDIter operand array), plus ~16 hand-edited special cases (2-D literals,
long/float suffixes, int[]-only args, concatenate, the dtype-table comments).
Comments that named the now-hidden array type were reworded to be literal-accurate
(e.g. "from int[]" -> "int literals").
Pre-existing uncommitted changes left untouched (not part of this work):
docs/website-src/docs/toc.yml, templates/numsharp/public/main.js,
docs/extensions/bitmap.md, docs/advanced/index.md.
…invalid (5,) in doc
Follow-up to the long-friendly np.indices overloads: prove the API is idiomatic from
C#/.NET, and bring the one DocFX example that uses it in line with the new surface.
## .NET-friendliness tests (np.indices.Test.cs, +7)
Exercise the API through real C# idioms, all verified compiling AND agreeing:
- Collection expressions: np.indices([2, 3]) binds int[] (unambiguous — an identity
element conversion beats int->long, so adding the long[] overload did NOT make the
documented [3, 3] spelling ambiguous), np.indices([2L, 3L]) binds long[].
- Value tuple: np.indices((2, 3)) reads like NumPy.
- The `row, col = np.indices(...)` unpack idiom via the leading-axis indexer.
- Result is a normal NDArray: fluent chaining (…[1].reshape(6L)).
- Sparse result is a plain NDArray[]: foreach + LINQ (Select over shapes, sum of sizes).
- dtype via every .NET spelling: typeof(int), NPTypeCode.Int32, "i4", np.int32.
- The documented scalar convenience: np.indices(5) -> (1, 5) grid (NumPy raises), with
the valid C# 1-D spellings np.indices([5]) / np.indices(new long[]{5}).
Green on BOTH net8.0 and net10.0 (collection expressions / tuples compile on both).
## DocFX example coverage + accuracy
docs/website-src/docs/fundamentals/array-creation.md was the ONLY page with an np.indices
code example, and its comment ("takes an int[] of dimensions (not a tuple)") became false
with the new overloads. Updated it to show every .NET shape spelling (collection expr,
tuple, long[], an array's own .shape) plus the sparse form.
FundamentalsArrayCreationDocTests: the mirror test Intrinsic_Indices_TakesIntArray ->
Intrinsic_Indices_AcceptsAllShapeSpellings, asserting all four spellings agree on the
(2,3,3) grid, the documented grid[0]/grid[1] values, and the sparse (3,1)/(1,3) shapes.
## Doc fix
The np.indices(Shape) XML <remarks> suggested spelling a 1-D grid as np.indices((5,)) —
INVALID C# (no 1-element tuple). Corrected to np.indices([5]) / np.indices(new long[]{5}),
which is what the new scalar-convenience test pins.
Green: np_indices (20) + FundamentalsArrayCreationDocTests (15) on net8.0 AND net10.0;
whole Documentation namespace (192). Core/test build 0W/0E.
Convert every em-dash (U+2014) to an ASCII hyphen-minus (-) in the hand-authored docfx source under docs/website-src/. 1574 occurrences across 42 tracked markdown/YAML files. Scope: only git-tracked *.md / *.yml content in docs/website-src/. The generated api/*.yml (gitignored) and the refs/data report content pulled in at build time are untouched. Method: byte-level replacement of the UTF-8 encoding E2 80 94 -> 2D, so line endings (CRLF/LF), any BOM, and every other byte are preserved exactly. The diff is a balanced 1438/1438 (each touched line is one delete + one add), so there is no whitespace or EOL churn. UTF-8 is self-synchronizing, so the 3-byte sequence cannot collide with part of another character. Safety checks made before converting: - No em-dash starts any line (after leading whitespace), so none could turn into a stray "- " markdown list item. - The only YAML occurrence was mid-value (docs/toc.yml, "Under the hood - internals"); the list-item dash sits at line start and is unaffected. - Em-dashes inside fenced code blocks were all prose comments (// ... - ...), never executable syntax. En-dashes (U+2013, 27 occurrences) are intentionally left as-is; only em-dashes were requested.
Nucs
changed the title
…p.ma.masked stays as NumPy alias
NumSharp's port of numpy.ma.MaskedArray now follows the ND* type convention of NDArray.
Renames (symbols, done with the IDE rename)
- class MaskedArray -> NDMaskedArray (file Ma/MaskedArray.cs -> Ma/NDMaskedArray.cs)
- class MaskedConstant -> NDMaskedConstant
- tests MaskedArrayTests -> NDMaskedArrayTests (file Ma/MaskedArrayTests.cs -> Ma/NDMaskedArrayTests.cs)
- prop np.ma.masked -> np.ma.NDMasked
- prop np.ma.masked_singleton -> np.ma.NDMaskedSingleton
- the ma oracle harness (FuzzCorpusTests.Ma / OpRegistry.Ma) and three gate comments follow the type rename.
MaskedArrayModule (the np.ma facade) and MaskedPrintOption keep their names.
NumPy API parity kept: np.ma.masked / np.ma.masked_singleton
- NumPy 2.4.2 (probed): np.ma.masked is the ONE numpy.ma.core.MaskedConstant instance (a MaskedArray
subclass), a 0-D float64 0.0 under a True mask, `masked is masked_singleton` is True, repr 'masked',
and a fully-masked reduction returns it (`ma.array([1.,2.], mask=[1,1]).sum() is masked`).
- Both NumPy names are re-added on MaskedArrayModule as EXPRESSION-BODIED aliases of NDMasked, so all four
spellings (NDMasked, NDMaskedSingleton, masked, masked_singleton) are the same object. A second stored
`new NDMaskedConstant()` would have broken the reference-identity contract: the ufuncs/reductions return
NDMasked, and `ReferenceEquals(r, np.ma.masked)` (NumPy's `r is np.ma.masked`) must hold. Ported NumPy
code (`x[i] = np.ma.masked`) compiles unchanged; NumSharp code can use the ND* name.
- New test NDMaskedArrayTests.MaskedConstant_NumPyNames_AliasTheNDMaskedSingleton pins identity across all
four names, the NDMaskedConstant type, the repr token, the 0-D float64 0.0 / True-mask payload, the
fully-masked sum returning np.ma.masked, and `y[1] = np.ma.masked` masking without touching data.
Repaired: the IDE's "text occurrences" pass damaged prose, NumPy-verbatim strings, and the oracle gate
The rename ran with text-occurrence replacement, which rewrote EVERY whole word `masked` -> `NDMasked`
(1,104 comment/string hits) and `MaskedArray` -> `NDMaskedArray` even where the text names NumPy's own
class. Left as-is it would have shipped:
1. A red oracle gate. FuzzCorpusTests.Ma's `case "masked":` became `case "NDMasked":`, but the committed
ma_* corpus spells the kind "masked" (~62K cases), so every masked case fell to `default` ->
"unknown ma expected.kind 'masked'". Verified: MaReduce / MaScan / MaExtras all failed that way.
2. Broken NumPy-verbatim messages: "A masked array does not own its data and therefore cannot be
resized.", "Cannot process masked data." (x2), "Unavailable for masked array.", NumPy's own
NotImplementedError "MaskedArray.tofile() not implemented yet.", the view() itemsize message, and the
NDMaskedConstant.ToString() token "masked" (NumPy's repr(np.ma.masked)). Their tests had been rewritten
in lockstep, so the unit suite stayed green while parity broke.
3. English turned into nonsense, including in files the rename never needed to touch: "masked-off slots
keep prior contents" -> "NDMasked-off slots" in Default.Fabs/Frexp/Ldexp/Positive/Reciprocal and
DefaultEngine.CompareOp, "NumPy's MaskedArray._insert_masked_print" -> "NDMaskedArray._insert_…" in
ArrayFormatter, the masked-corpus schema docs in FuzzCorpus.cs, "masked positives" in
UndisposedIntermediateTests, and the pythonnet decode tests that describe the PYTHON-side numpy
MaskedArray. Those 9 files outside the ma rename are byte-identical to HEAD again (not in this commit);
UndisposedIntermediateTests keeps only its correct NDMaskedArray type mention.
How the repair was done (mechanical, then reviewed)
- HEAD and the renamed tree were tokenized (words + single non-word chars). The rename was strictly 1:1
word-for-word, so both token streams have equal length and zip into exactly 1,889 substitutions.
- Each substitution was classified with Roslyn at its position:
code identifier / doc-comment cref ....... 701 kept (real references to the renamed symbols)
string / interpolated-string literal ....... 19 reverted (NumPy-verbatim text, corpus kind, repr)
comment / XML doc text: `masked` ......... 1,087 reverted (the English word / NumPy's constant)
comment / XML doc text: masked_singleton ..... 3 reverted (NumPy's name)
comment / XML doc text: MaskedArray ......... 79 split by context: 29 name NumSharp's own class
(kept as NDMaskedArray / NDMaskedArrayTests);
50 name NumPy's class (reverted: numpy.ma.MaskedArray,
"NumPy's <c>MaskedArray.x</c>", MaskedArray._mask, …).
10 of those 50 were caught in manual review (the "NumPy's" sat on the previous doc line, which the
context heuristic did not see).
- The +2-char rename had pushed aligned trailing `//` comments right; the 7 affected lines are put back
on HEAD's comment column. Two articles fixed to house style ("an NDMaskedArray"; the codebase writes
"an NDArray" 196x vs "a NDArray" 2x).
Docs
- docs/MA_ORACLE_DESIGN.md, docs/MA_MODULE_AUDIT.md: current-state pointers now name
Ma/NDMaskedArray.cs / NDMaskedArrayTests, plus a dated rename note; the historical pass logs keep the
names that were in use at the time.
Verification (isolated detached worktree at b779d95 + exactly this changeset; the shared tree carries
an unrelated in-flight collections rename that is NOT part of this commit)
- Build: NumSharp.Tests + NumSharp.Tests.Oracle (Core, OpenBLAS interop) - 0 errors.
- NumSharp.Tests `NumSharp.Tests.Ma.*`: 91/91 on net10.0 and net8.0 (90 existing + 1 new).
- Oracle (net10.0), each in its own test host: MaUnary, MaBinary, MaReduce, MaScan, MaManip,
MaConstruct, MaSelect, MaSortSetops, MaExtras, MaSurface_IsCoveredOrExplicitlyClassified,
OracleCoverageStrengthTests (3), OracleApplicabilityTests (1) - all pass.
- No remaining code references to the old identifiers in any tracked .cs; no API snapshot, coverage
mapping (coverage/object_surfaces.py maps ma.MaskedArray with no CLR owner) or website page names them.
Pre-existing, NOT introduced here (reproduced 3/3 on a clean b779d95 worktree)
- Running several ma oracle tiers in ONE test host intermittently kills the host: it vanishes mid-test
(the crash point varies from after MaReduce to during OracleApplicabilityTests), with no managed
exception, no stderr, and no Application event-log entry. It is timing-sensitive: a --diag run of the
9 tiers completed with exit code 0. Every test passes in its own host.
Not included (other work present in the tree): the staged OrderedDict/ConcurrentOrderedDict ->
*Dictionary collections rename and ConcurrentPointingDict.cs deletion, benchmark BenchmarkBase.cs /
Program.cs / ConcurrentOrderedDictBenchmarks.cs edits, and the dirty refs/data submodule.
…entPointingDict removed, docs and crefs fixed
Lands the rename of NumSharp's ordered concurrent maps (an IDE rename, finished here) and deletes the experimental
ConcurrentPointingDict, with every doc, cref, test, benchmark and probe brought in line.
TYPE / FILE RENAMES (namespace NumSharp.Collections unless noted)
- ConcurrentOrderedDict<TKey,TValue> -> ConcurrentOrderedDictionary<TKey,TValue> (COD)
- ConcurrentOrderedCompactDict<TKey,TValue> -> ConcurrentOrderedCompactDictionary<TKey,TValue> (COCD)
- OrderedDict<TKey,TValue> -> OrderedDictionary<TKey,TValue> (lock-free value-once variant; the
proposal's reference implementation)
- ConcurrentOrderedDict.{TODO,COMPACT}.md -> ConcurrentOrderedDictionary.{TODO,COMPACT}.md (design ledgers)
- ConcurrentDictionary.NumSharp.cs -> ConcurrentDictionary.RefAccessors.cs (the vendored clone's one
NumSharp-only member). GetValueRefOrNullRef is now PUBLIC (was internal) - deliberate part of the rename; its
remarks now say the three caller obligations (lifetime inside one serialized section, single-field <=word writes
only, caller-side write serialization) bind EVERY caller, and that it bypasses the tear-free discipline the REST of
the public surface keeps.
- All 11 test classes/files now read ...Dictionary... (the IDE rename had covered only the two primary suites;
the Advanced/Concurrency/Memory/Randomized mirrors of both types and COCD's Specific suite still said "Dict").
- ConcurrentOrderedDictBenchmarks -> ConcurrentOrderedDictionaryBenchmarks. LOAD-BEARING, not cosmetic: the rename
changed menu option 15 to `--filter "*ConcurrentOrderedDictionary*"`, and BenchmarkDotNet's --filter globs
`Namespace.Class.Method` (categories are not matched), so with the class still named ...DictBenchmarks the menu
entry matched ZERO benchmarks. Verified: `--list flat` now lists all 43.
REMOVED
- ConcurrentPointingDict<TKey,TValue> (src/NumSharp.Core/Collections/Concurrent/ConcurrentPointingDict.cs,
1,030 lines). Experimental "hash -> int32 index -> value list" variant on the vendored ConcurrentDictionary; it had
no tests, no benchmark rows and no in-tree consumer, and its own /optimize run already recommended not shipping it.
It also carried two concurrency defects reproduced before removal (isolated worktree, x64): its validated read
checks keys[idx] BEFORE loading values[idx], so an in-place TryRemoveSwapBack let a reader of the removed key
return the moved entry's value (31,268 wrong-value hits / 6 s with the default comparer; COD/COCD 0) - the exact
chained-layout tear ConcurrentOrderedDictionary.COMPACT.md section 7 warns about - and its "lock-free" read spun
until an interior removal finished re-indexing (4.1-4.5 ms GC-free stall at N=1M vs ~0.35 ms for COD/COCD).
Nothing else referenced it; docs/MA_MODULE_AUDIT.md keeps its dated pass-11 mention (a historical log).
DOC / CREF FIXES
- The IDE rename double-applied inside crefs: 22 `<see cref="ConcurrentOrderedDictionaryionary{TKey,TValue}"/>` /
`ConcurrentOrderedCompactDictionaryionary` (src 4, benchmark 1, tests 17). All collapsed to the real names. The
csproj NoWarn suppresses the whole CS1574 family, so the compiler never flags a broken cref - docfx is the check.
- Crefs docfx cannot bind on net10.0, where the BCL signatures are `ref readonly`:
* `<see cref="Volatile.Read(ref int)"/>` x3 (OrderedDictionary class doc - rendered as unlinked plain text on the
public API page - plus COD/COCD private count docs) -> `<c>Volatile.Read</c>` (the paragraphs already link the
Volatile class; a `ref readonly` cref would break the net8.0 doc build instead).
* `<see cref="Unsafe.IsNullRef{T}(ref T)"/>` -> `(ref readonly T)` on GetValueRefOrNullRef - newly visible because
the method is now public, so docfx renders it.
- Ledgers (ConcurrentOrderedDictionary.TODO.md / .COMPACT.md): type, file, test-suite, benchmark-class and
menu-filter names; the seam's new file name and public signature.
- docs/proposals/ConcurrentOrderedDictionary.md: reference implementation is now
`NumSharp.Collections.OrderedDictionary<TKey, TValue>`; added a naming note because the rename makes three names
collide in one document - the reference impl shares its simple name with the BCL
`System.Collections.Generic.OrderedDictionary<TKey, TValue>` the proposal compares against, and NumSharp's own
node-based `NumSharp.Collections.ConcurrentOrderedDictionary<TKey, TValue>` shares the PROPOSED type's name while
NOT being the reference implementation. Section 8's comparator is now fully qualified.
- benchmark/collections/probes/*.cs (4 `dotnet run` reproduction scripts the ledgers cite; outside every build, so
nothing else would have caught them): type names updated - they no longer compiled against the renamed types.
NAMING CAVEAT (not changed, flagged): NumSharp.Collections.OrderedDictionary<TKey,TValue> now shares its simple name
with System.Collections.Generic.OrderedDictionary<TKey,TValue> (.NET 9+). A net10.0 consumer importing both
namespaces gets CS0104 and must alias/qualify (net8.0 has no generic BCL OrderedDictionary). Same situation as the
pre-existing public NumSharp.Collections.Concurrent.ConcurrentDictionary. Nothing in-tree is affected.
VERIFICATION (isolated `git worktree` = HEAD + exactly the paths in this commit; the shared tree holds a parallel
session's unrelated pending edits)
- Build: NumSharp.Core + NumSharp.Tests (net8.0 + net10.0) + NumSharp.Benchmark.CSharp - 0 errors. The benchmark
project needed that parallel session's still-uncommitted BenchmarkBase.cs fix to build at all: at HEAD it has
CS0160 (`catch (IncorrectTypeException)` after `catch (TypeError)`, and IncorrectTypeException : TypeError). That
fix is NOT part of this commit.
- Tests: `FullyQualifiedName~ConcurrentOrdered` -> 167/167 passed on net10.0 AND net8.0 (Release).
- BenchmarkDotNet: `--filter "*ConcurrentOrderedDictionary*" --list flat` -> 43 benchmarks (= every method).
- docfx metadata: 0 warnings for any collection file (repo total 423, all pre-existing elsewhere); the API
reference regenerates 11 collection pages under the new names.
- Probes: `dotnet build -c Release <probe>.cs` -> all 4 compile.
- Leftover sweep: no ConcurrentOrderedDict / ConcurrentOrderedCompactDict / OrderedDict / ConcurrentPointingDict /
ConcurrentDictionary.NumSharp / "Dictionaryionary" in tracked files outside the dated MA audit log and the
gitignored optimize-runs/.
NOT IN THIS COMMIT: the parallel session's pending work in the shared tree (Ma MaskedArray -> NDMaskedArray rename,
Default.* math / ArrayFormatter / Oracle fuzz / interop test edits, BenchmarkBase.cs, refs/data).
…n list, probe self-name, blame-ignore the rename A second, independent verification pass over 5635182 (the *Dict -> *Dictionary rename), running checks the first pass did not, plus the three small fixes it surfaced. VERIFIED, NO CHANGE NEEDED - Mirror-suite integrity. For each of the 5 COD/COCD test-suite pairs, substituting ConcurrentOrderedDictionary -> ConcurrentOrderedCompactDictionary in the COD file and diffing against its COCD mirror leaves exactly the documented residual deltas (the MIRROR header, cod-/cocd- thread names, the memory-doc pointer TODO -> COMPACT, the `== 0` vs `<= 72` presized-append bound, the AddRange comment variant) - and those residuals are IDENTICAL (name-normalized) before and after the rename, so the IDE rename kept both suites in lockstep. - Compiler-level cref validation, stronger than docfx (every member incl. private, BOTH TFMs): rebuilt NumSharp.Tests (pulls Core) in a clean HEAD worktree with `-p:NoWarn=NONE -p:GenerateDocumentationFile=true`. The override surfaces 782 CS1574 repo-wide, and ZERO XML-doc diagnostics (CS1574/1580/1581/1584/0419/1570-1573/1711/ 1712/1734) in any src or test collection file - their only warnings are 668 CS1591 (undocumented test members). - Cross-reference integrity: every file path / file name cited by the 26 files of 5635182 resolves to a tracked file (the only non-resolving tokens are upstream dotnet/runtime paths) - except one pre-existing stale self-reference, fixed below. - Broader leftover sweep (`Dict(?!ionary)` in scope; case-insensitive `ordered[ _-]?dict` repo-wide): only BCL Dictionary shorthand ("COD vs Dict", the DictC adapter), Python-dict semantics (AddRange_UpsertsWithPythonDictSemantics) and generic "ordered dict" prose remain - none name these types. - Rename-collateral audit (the Rider text-occurrence hazard): every changed string literal is the type naming itself (reentrancy messages, nameof, BDN category, menu text/filter); the rename touched nothing outside the collection scope (no Python OrderedDict exists in any tracked .py). FIXED - ConcurrentDictionary.cs header: its "deviations from the upstream source" list said the public key-API is otherwise unchanged, but the clone now carries a PUBLIC NumSharp-only member (GetValueRefOrNullRef in ConcurrentDictionary.RefAccessors.cs, made public in 5635182). Added the `partial` + public-accessor bullet and scoped the closing "unchanged" line to the upstream members. - benchmark/collections/probes/compact_ordered_dict_complexity.cs: its title comment named the file "complexity_probe.cs" (stale since it was committed under its real name). - .git-blame-ignore-revs: records 5635182, per the file's convention for mechanical renames (test-project rename, Npy* -> ND*, src -> refs relocation). Verified: a renamed test line (ConcurrentOrderedDictionaryConcurrencyTests.cs:110) blames to 5635182 without the file and back to its origin 1fcaea0 with it. The entry states the caveat measured while verifying: git also re-attributes the commit's few genuinely NEW doc lines to neighbouring older lines (the proposal's naming note blames to e3d5b77 with the file on); `git blame --no-ignore-revs-file` shows their true origin. Every earlier entry's SHA is an ancestor of origin/master (branches merge, not squash), and an unknown SHA in the file is harmless (probed: blame still exits 0). BUILD: NumSharp.Core (net8.0 + net10.0) and the edited probe (`dotnet build -c Release <probe>.cs`) compile in a clean HEAD worktree. LEFT AS IS, DELIBERATELY: ledger line widths (already 19/43 lines over 120 chars - tables and repo paths; the rename added one each), generic "ordered dict" prose, and the dated history entries in docs/MA_MODULE_AUDIT.md.
… bool-mask crash, leaks, host-pinned tests PR #631 (journey4) had never run on Linux/macOS CI. Run 35818966511 failed in five jobs, and CI stops at the first red step, so later steps hid more. Every job was replicated locally to find all of it: Windows directly, Linux in WSL Ubuntu 24.04 (every step of the `test` job plus the docs job's benchmark smoke), and ARM64 via DOTNET_EnableSSE41=0 (Vector128 stays accelerated, Sse41.IsSupported turns false, the same shape the kernel emitter sees on Apple silicon). Six root causes, two of them real Core bugs. 1. CORE BUG: np.argsort wrote past a zero-byte buffer (native heap corruption) AxisSort.ArgSortInto guarded only an empty AXIS (N == 0), not an empty ARRAY. For (0,3) along the last axis (or (2,0,3)/-1) there are zero lines, but the all-but-axis NDIter still makes one kernel call, and the radix line kernels ignore the per-call count: each call writes a full line of N int64 indices. Those 24 bytes landed in dst's ZERO-byte buffer. glibc never noticed. Windows' heap manager killed the test host with 0xC0000374 (STATUS_HEAP_CORRUPTION) in whatever test ran next, which aborted the whole Windows Oracle run inside IndexOracleTests.Index_Curated/Index_Dtype. Bisected 88 FuzzCorpusTests tiers -> MaSortSetops -> the single case ma.sort/empty_2d/bool.some/748 -> np.argsort (np.ma.sort = argsort of the filled keys + take_along_axis), then isolated in a standalone repro: every dtype, not just bool, at any zero-lines shape with a length >= 2 sort axis. This is the "OPEN pre-existing Ma-tier testhost crash" noted with 2ea7ba6. SortInPlace and AxisPartition (partition/argpartition) already carried the size guard; argsort was the one sibling without it. Fix: `if (N == 0 || src.size == 0) return;` in ArgSortInto, plus a defensive `ops[0].size == 0` early-out in DriveAllButAxis so a future caller cannot repeat it. Gate: Sorting/ArgsortZeroLinesTests (all 15 dtypes, covering every argsort kernel family, x 6 empty shapes/axes, axis=None, the ma.sort corpus case) with native-heap churn after each call. Verified to have teeth: the unfixed build crashes the test host; the fixed build passes 3/3 in 0.6 s. 2. CORE BUG: np.evaluate threw PlatformNotSupportedException on ARM64 (macOS) The fused NDExpr shell (vector v2) widens a STREAMED bool operand into a Vector<W> lane mask with DirectILKernelGenerator.EmitInlineMaskCreation, whose 2/4/8-byte path calls the x86-only Sse41/Avx2 ConvertToVector*Int*. The plan/shell gated bool operands on vector WIDTH only (FusedBoolLanesAvailable: 128 or 256 bits), and ARM64 has 128-bit AdvSimd Vector128 with every x86 ISA unsupported. The DynamicMethod compiles fine and throws only when it runs, so every tree mixing a bool array with a 2/4/8-byte lane (where(m1,a,b), a*m1, a+m1, where(m1&(a>b),a,-b)) threw on the macos-latest runner (30 cases across 7 NDEvaluateVectorTests). np.where / np.select / np.piecewise gate the same helper with their own needsX86 checks; only the NDExpr shell lacked one. Fix: new DirectILKernelGenerator.InlineMaskCreationSupported(simdBits, elementSize) (the helper's real ISA contract) + FusedBoolInputMasksAvailable, applied in both NDExprVectorPlan.TryPlan and FusedSimdViable. Such trees take the scalar shell on ARM64, which is the byte-for-byte oracle the vector path must reproduce, so results are unchanged. The gate is precise: a bool OUTPUT is packed portably and stays vectorized, and so does a bool 0-d PARAMETER (a constant Zero/AllBitsSet mask, no x86). TryPlan now receives isParam so it can tell the two apart. x86 behaviour is unchanged (SSE4.1/AVX2 present). Tests: VectorPlan_DecidesLaneAndScalarFallback now states the plan against the host capability, plus param/output cases and an InlineMaskCreationSupported truth table. Verified: with DOTNET_EnableSSE41=0 the 30 failures reproduce before the fix and all NDEvaluate tests pass after it. 3. Undisposed intermediates (Oracle Corpus_AllOps_LeaveNoUndisposedIntermediates; 38 families / 54 cases, all on this branch) - np.logspace / np.geomspace: CastSpacingResult dropped the float64 compute array after astype -> exactly 1 escaped buffer per non-float64 call. It now disposes the replaced array (ReferenceEquals-guarded). Ownership transfer is documented on the parameter. - np.piecewise (1-3 escaped per call): the bool-condition aliases, ToBoolCondition's astype temp, ComputeElseCondition's partial ORs and its logical_not result, the coerced 0-d scalar constants (composition path), and the fused path's scalar + astype pair were all dropped live. PiecewiseCore now owns and releases them in a finally, and also releases the half-written result on a throw. The bare-array and scalar-bool overloads release the condition arrays they build. Deliberately NOT [NDScoped]: an ambient scope would reclaim arrays a user callback allocates or keeps (the reason apply_along_axis stays unscoped too), so disposal is explicit and never touches caller-owned or user-callback arrays. asanyarray can hand back an EXISTING array for NDArray input, so the scalar-constant dispose is safe only because that case returns earlier. That is documented at the site. 4. Stale tests / benchmark gates behind the earlier failures - benchmark CS0160 (Deploy Docs build): IncorrectTypeException now derives from TypeError (3ad4360), so `catch (IncorrectTypeException)` after `catch (TypeError)` was unreachable. Reordered derived-before-base and kept it explicit so the benchmark survives a future re-parenting. - ModfBenchmarks (next failure in the same job once it compiled; benchmark smoke: "expected to be rejected, but the operation succeeded"): np.modf accepts every REAL dtype since 59f9932 (bool/int -> narrowest float per width, float16 kernel); measured 14/15 accept, only Complex is rejected, the same as NumPy 2.4.2 (per-dtype probe on both sides, result dtypes identical). The C# gate now rejects only Complex, and the NumPy twin (numpy_benchmark.py) times every non-complex dtype so the new C# rows have peers. The old comment explaining the float32/float64-only restriction was describing a C# rejection that no longer exists. - MisalignedRegistryTightnessTests B2/B3 asserted that two excuses STILL existed, but 074d4d0 (complex multiply now bit-exact) and 7fd70a3 (cumprod size<=1 widening) removed them on purpose. Inverted to *_NoLongerExcused so they pin the tight contract. 5. Tests that need an optional dependency CI does not provide - 25 Examples tests (Gist/Karpathy demos + polyfit companions) called OpenBlasEngine.Enable unconditionally, and CI's test job stages no native binary, so they threw DllNotFoundException on all 3 OSes. New Examples/ExampleBlasBackend follows the house LapackEigTests.RequireLapack pattern (Inconclusive where no CBLAS/LAPACK loads). KarpathyLstmTests binds in [TestInitialize] without failing: its 4 semantic/tolerance tests now run everywhere (verified passing with no backend), and only the 2 exact-value tests require the backend. With the binary staged, all 104 example tests still pass (0 skipped). - 9 interop tests import SciPy, which the interop job does not install ("No module named 'scipy'"). They now call the existing SkipUnless("scipy") as their FIRST statement: their mid-test failures had leaked export pins, which cascaded into NumpyNet_BootsOnTheSharedEngine / Lifetime_TheirWrapperIsTheOnlyHolder (LiveExports 9 != 0). 6. Host-dependent exact values (the tests, not the product) - np.emath arctanh(0.5) (glibc) and arccos(0.5)/arccos(0.3) (Apple libm), plus np.sinc(0.25) (both), differ from the win-amd64 ucrtbase bits by 1 ULP. The matching corpus tiers are already host-pinned (RunHostLibmCorpus: Inconclusive off-Windows). New Utilities/HostLibm gives unit tests the same pin but keeps checking: bit-exact on Windows, <= 4 ULP elsewhere, with NaN-ness, infinities and the sign bit (incl. signed zero) always strict. - 4 interop live-parity tests (NES final weights/trace, forecast mpe, ranking dcg) measured 1 ULP off live NumPy on macos-latest arm64 only. Their authors deliberately refuse to widen per-host budgets ("Other hosts deliberately expose a disagreement instead of silently widening the gate"), so they use the existing SkipByteExactOnArm64 x64 pin (the NEON 128-bit reduction-width class) with the measured drift in their docs. Verification (all local; macOS arm64 itself is only reachable through CI). Other sessions landed 2ea7ba6, 5635182 and 4499d3e while this was in progress (none touch these files), so each figure names the base it ran on: - Windows (CI filter): NumSharp.Tests net8.0 16022/0 (b779d95) and net10.0 16024/0 (2ea7ba6); Oracle net8.0/net10.0 182/0 without OpenBLAS (2ea7ba6; was: test-host crash) and 206/206 with it staged (4499d3e); Interop net8.0 733/733 with SciPy and 724 passed / 9 skipped / 0 failed with SciPy hidden, the CI path (2ea7ba6). At 4499d3e all four affected projects build and the 220 targeted tests pass. - Linux (WSL Ubuntu 24.04, CI test-job replica, every step run even after a failure, b779d95): NumSharp.Tests net8.0 15958/0, net10.0 15959/0; Oracle net8/net10 167/0; Analyzer net8/net10 321/0 (baseline before the fix: 28 + 28 + 3 + 3 failures). The docs job's benchmark body smoke runs all 499 bodies OK on net8.0, and the unary twin join check is OK. - ARM64 simulation (DOTNET_EnableSSE41=0): NDEvaluate 30 failures -> 0. Not changed here (noted for follow-up): benchmark/scenarios.html is a generated artifact that already disagrees with its own (non-CI) test_scenarios_audit.py totals (python 1296 vs 1699); regenerate it with benchmark/scripts/generate_scenarios_html.py. NumpyNet lifetime tests assert ABSOLUTE LiveExports counts, so any interop test that leaks on failure cascades into them; comparing deltas would localise such failures.
Nucs
added a commit
that referenced
this pull request
Sep 23, 2026
… bool-mask crash, leaks, host-pinned tests PR #631 (journey4) had never run on Linux/macOS CI. Run 35818966511 failed in five jobs, and CI stops at the first red step, so later steps hid more. Every job was replicated locally to find all of it: Windows directly, Linux in WSL Ubuntu 24.04 (every step of the `test` job plus the docs job's benchmark smoke), and ARM64 via DOTNET_EnableSSE41=0 (Vector128 stays accelerated, Sse41.IsSupported turns false, the same shape the kernel emitter sees on Apple silicon). Six root causes, two of them real Core bugs. 1. CORE BUG: np.argsort wrote past a zero-byte buffer (native heap corruption) AxisSort.ArgSortInto guarded only an empty AXIS (N == 0), not an empty ARRAY. For (0,3) along the last axis (or (2,0,3)/-1) there are zero lines, but the all-but-axis NDIter still makes one kernel call, and the radix line kernels ignore the per-call count: each call writes a full line of N int64 indices. Those 24 bytes landed in dst's ZERO-byte buffer. glibc never noticed. Windows' heap manager killed the test host with 0xC0000374 (STATUS_HEAP_CORRUPTION) in whatever test ran next, which aborted the whole Windows Oracle run inside IndexOracleTests.Index_Curated/Index_Dtype. Bisected 88 FuzzCorpusTests tiers -> MaSortSetops -> the single case ma.sort/empty_2d/bool.some/748 -> np.argsort (np.ma.sort = argsort of the filled keys + take_along_axis), then isolated in a standalone repro: every dtype, not just bool, at any zero-lines shape with a length >= 2 sort axis. This is the "OPEN pre-existing Ma-tier testhost crash" noted with 2ea7ba6. SortInPlace and AxisPartition (partition/argpartition) already carried the size guard; argsort was the one sibling without it. Fix: `if (N == 0 || src.size == 0) return;` in ArgSortInto, plus a defensive `ops[0].size == 0` early-out in DriveAllButAxis so a future caller cannot repeat it. Gate: Sorting/ArgsortZeroLinesTests (all 15 dtypes, covering every argsort kernel family, x 6 empty shapes/axes, axis=None, the ma.sort corpus case) with native-heap churn after each call. Verified to have teeth: the unfixed build crashes the test host; the fixed build passes 3/3 in 0.6 s. 2. CORE BUG: np.evaluate threw PlatformNotSupportedException on ARM64 (macOS) The fused NDExpr shell (vector v2) widens a STREAMED bool operand into a Vector<W> lane mask with DirectILKernelGenerator.EmitInlineMaskCreation, whose 2/4/8-byte path calls the x86-only Sse41/Avx2 ConvertToVector*Int*. The plan/shell gated bool operands on vector WIDTH only (FusedBoolLanesAvailable: 128 or 256 bits), and ARM64 has 128-bit AdvSimd Vector128 with every x86 ISA unsupported. The DynamicMethod compiles fine and throws only when it runs, so every tree mixing a bool array with a 2/4/8-byte lane (where(m1,a,b), a*m1, a+m1, where(m1&(a>b),a,-b)) threw on the macos-latest runner (30 cases across 7 NDEvaluateVectorTests). np.where / np.select / np.piecewise gate the same helper with their own needsX86 checks; only the NDExpr shell lacked one. Fix: new DirectILKernelGenerator.InlineMaskCreationSupported(simdBits, elementSize) (the helper's real ISA contract) + FusedBoolInputMasksAvailable, applied in both NDExprVectorPlan.TryPlan and FusedSimdViable. Such trees take the scalar shell on ARM64, which is the byte-for-byte oracle the vector path must reproduce, so results are unchanged. The gate is precise: a bool OUTPUT is packed portably and stays vectorized, and so does a bool 0-d PARAMETER (a constant Zero/AllBitsSet mask, no x86). TryPlan now receives isParam so it can tell the two apart. x86 behaviour is unchanged (SSE4.1/AVX2 present). Tests: VectorPlan_DecidesLaneAndScalarFallback now states the plan against the host capability, plus param/output cases and an InlineMaskCreationSupported truth table. Verified: with DOTNET_EnableSSE41=0 the 30 failures reproduce before the fix and all NDEvaluate tests pass after it. 3. Undisposed intermediates (Oracle Corpus_AllOps_LeaveNoUndisposedIntermediates; 38 families / 54 cases, all on this branch) - np.logspace / np.geomspace: CastSpacingResult dropped the float64 compute array after astype -> exactly 1 escaped buffer per non-float64 call. It now disposes the replaced array (ReferenceEquals-guarded). Ownership transfer is documented on the parameter. - np.piecewise (1-3 escaped per call): the bool-condition aliases, ToBoolCondition's astype temp, ComputeElseCondition's partial ORs and its logical_not result, the coerced 0-d scalar constants (composition path), and the fused path's scalar + astype pair were all dropped live. PiecewiseCore now owns and releases them in a finally, and also releases the half-written result on a throw. The bare-array and scalar-bool overloads release the condition arrays they build. Deliberately NOT [NDScoped]: an ambient scope would reclaim arrays a user callback allocates or keeps (the reason apply_along_axis stays unscoped too), so disposal is explicit and never touches caller-owned or user-callback arrays. asanyarray can hand back an EXISTING array for NDArray input, so the scalar-constant dispose is safe only because that case returns earlier. That is documented at the site. 4. Stale tests / benchmark gates behind the earlier failures - benchmark CS0160 (Deploy Docs build): IncorrectTypeException now derives from TypeError (3ad4360), so `catch (IncorrectTypeException)` after `catch (TypeError)` was unreachable. Reordered derived-before-base and kept it explicit so the benchmark survives a future re-parenting. - ModfBenchmarks (next failure in the same job once it compiled; benchmark smoke: "expected to be rejected, but the operation succeeded"): np.modf accepts every REAL dtype since 59f9932 (bool/int -> narrowest float per width, float16 kernel); measured 14/15 accept, only Complex is rejected, the same as NumPy 2.4.2 (per-dtype probe on both sides, result dtypes identical). The C# gate now rejects only Complex, and the NumPy twin (numpy_benchmark.py) times every non-complex dtype so the new C# rows have peers. The old comment explaining the float32/float64-only restriction was describing a C# rejection that no longer exists. - MisalignedRegistryTightnessTests B2/B3 asserted that two excuses STILL existed, but 074d4d0 (complex multiply now bit-exact) and 7fd70a3 (cumprod size<=1 widening) removed them on purpose. Inverted to *_NoLongerExcused so they pin the tight contract. 5. Tests that need an optional dependency CI does not provide - 25 Examples tests (Gist/Karpathy demos + polyfit companions) called OpenBlasEngine.Enable unconditionally, and CI's test job stages no native binary, so they threw DllNotFoundException on all 3 OSes. New Examples/ExampleBlasBackend follows the house LapackEigTests.RequireLapack pattern (Inconclusive where no CBLAS/LAPACK loads). KarpathyLstmTests binds in [TestInitialize] without failing: its 4 semantic/tolerance tests now run everywhere (verified passing with no backend), and only the 2 exact-value tests require the backend. With the binary staged, all 104 example tests still pass (0 skipped). - 9 interop tests import SciPy, which the interop job does not install ("No module named 'scipy'"). They now call the existing SkipUnless("scipy") as their FIRST statement: their mid-test failures had leaked export pins, which cascaded into NumpyNet_BootsOnTheSharedEngine / Lifetime_TheirWrapperIsTheOnlyHolder (LiveExports 9 != 0). 6. Host-dependent exact values (the tests, not the product) - np.emath arctanh(0.5) (glibc) and arccos(0.5)/arccos(0.3) (Apple libm), plus np.sinc(0.25) (both), differ from the win-amd64 ucrtbase bits by 1 ULP. The matching corpus tiers are already host-pinned (RunHostLibmCorpus: Inconclusive off-Windows). New Utilities/HostLibm gives unit tests the same pin but keeps checking: bit-exact on Windows, <= 4 ULP elsewhere, with NaN-ness, infinities and the sign bit (incl. signed zero) always strict. - 4 interop live-parity tests (NES final weights/trace, forecast mpe, ranking dcg) measured 1 ULP off live NumPy on macos-latest arm64 only. Their authors deliberately refuse to widen per-host budgets ("Other hosts deliberately expose a disagreement instead of silently widening the gate"), so they use the existing SkipByteExactOnArm64 x64 pin (the NEON 128-bit reduction-width class) with the measured drift in their docs. Verification (all local; macOS arm64 itself is only reachable through CI). Other sessions landed 2ea7ba6, 5635182 and 4499d3e while this was in progress (none touch these files), so each figure names the base it ran on: - Windows (CI filter): NumSharp.Tests net8.0 16022/0 (b779d95) and net10.0 16024/0 (2ea7ba6); Oracle net8.0/net10.0 182/0 without OpenBLAS (2ea7ba6; was: test-host crash) and 206/206 with it staged (4499d3e); Interop net8.0 733/733 with SciPy and 724 passed / 9 skipped / 0 failed with SciPy hidden, the CI path (2ea7ba6). At 4499d3e all four affected projects build and the 220 targeted tests pass. - Linux (WSL Ubuntu 24.04, CI test-job replica, every step run even after a failure, b779d95): NumSharp.Tests net8.0 15958/0, net10.0 15959/0; Oracle net8/net10 167/0; Analyzer net8/net10 321/0 (baseline before the fix: 28 + 28 + 3 + 3 failures). The docs job's benchmark body smoke runs all 499 bodies OK on net8.0, and the unary twin join check is OK. - ARM64 simulation (DOTNET_EnableSSE41=0): NDEvaluate 30 failures -> 0. Not changed here (noted for follow-up): benchmark/scenarios.html is a generated artifact that already disagrees with its own (non-CI) test_scenarios_audit.py totals (python 1296 vs 1699); regenerate it with benchmark/scripts/generate_scenarios_html.py. NumpyNet lifetime tests assert ABSOLUTE LiveExports counts, so any interop test that leaks on failure cascades into them; comparing deltas would localise such failures.
…ntOrderedDictionary ([OpenBugs]), pin the compact sibling's comparer exception safety Two new test classes. Every [OpenBugs] test asserts the CORRECT outcome promised by the type's own docs and by docs/proposals/ConcurrentOrderedDictionary.md, and fails today (CI excludes the category until each fix lands; remove the attribute together with the fix). Five passing controls pin the behaviour the compact sibling and OrderedDictionary already get right. Tests only - no product code changed. == test/NumSharp.Tests/Collections/OrderedDictionaryOpenBugsTests.cs (6 tests, all [OpenBugs]) == 1. Enumerate_WhileAWriterRegrowsTheTable_NeverThrowsAndNeverYieldsAnUnwrittenSlot GetEnumerator() = new Enumerator(_t._values, Volatile.Read(ref _t._count)) reads the volatile generation field TWICE: a growth published between the reads pairs the old (shorter) values array with the new count -> IndexOutOfRangeException or phantom default values. Violates "snapshot enumeration never throws / consistent prefix" (proposal 7.1). Fix: read `_t` once into a local. Storm (writer Clear+refill 64 keys, readers foreach); fails in ~3 ms; ~160K IOOR/s measured on the shipped code. 2. WideValueReplaceThenAppend_ReaderHoldingTheOlderGeneration_NeverReadsAnUnwrittenValue (DETERMINISTIC) A replace of a non-atomically-writable TValue (decimal/Guid/wide struct/long on 32-bit) clones ONLY the values array; the new generation SHARES _index/_keys with the old one, so the next in-place append writes a word + key into those shared arrays and a reader still holding the old generation resolves the new key and returns its own never-written values[slot] = default for a present key. Exactly the aliasing rule ConcurrentOrderedDictionary.COMPACT.md 4.7 forbids (the compact sibling copies the whole generation). Fix: full generation copy-on-write. Made deterministic with a comparer hook: TryGetValue captures its generation and only THEN calls GetHashCode, so the hook runs the writer (wide replace + append, on another thread) inside that window. Result today: found=True, value=0 for a key whose value is 150.25 (~47K/s with a trivial comparer). 3. LockFreeReplace_RacingCopyingResizes_NeverLosesACompletedWrite The lock-free SetByKey does a plain value store then volatile-reads _resizing/_t; a resize volatile-writes _resizing = true then copies the values array. That is the store-buffering (Dekker) litmus: release/acquire cannot forbid it on x86-TSO or ARM64, so the resize can copy the OLD value while the replacer sees the flag clear -> the completed write is lost when the resize publishes. Violates "no update is ever lost" (proposal 7.1). Fix: Interlocked.MemoryBarrier() after the store AND after every `_resizing = true` - verified in a scratch A/B: 3.26% lost -> 0 (0.84% -> 0 with one slot per cache line), +3.3 ns per replace (4.9 -> 8.2 ns), still ~5x a locked replace. The test uses one WRITER PER KEY with read-your-own-write re-reads: a storm with several writers per key (the original ~280M-op gun) cannot see this - last-writer-wins hides the lost intermediate write. 4. NullKey_IsRejectedOnEveryKeyEntryPoint_LikeTheSiblingsAndTheBcl No null check anywhere; EqualityComparer<string>.Default.GetHashCode(null) is 0, so a null key is stored, found and enumerated. All 11 key entry points accept it (COD/COCD/Dictionary/ConcurrentDictionary throw ArgumentNullException). Fix: the siblings' NullCheck (with the typeof(TKey).IsValueType Debug-boxing guard). 5. HugeCapacity_IsRefusedWithArgumentOutOfRange_InsteadOfSpinningForever `while (len < cap * 100 / 70 + 1) len <<= 1;` on an int: from capacity ~752M the shift wraps to int.MinValue then 0 and `0 << 1 == 0` spins forever (AppendGrow's `newLen <<= 1` has the same overflow). The compact sibling throws ArgumentOutOfRangeException above ~715M. Runs the ctor on a lowest-priority background thread with a 10 s join (a spinning managed thread cannot be stopped; no memory is allocated before the spin). 6. ComparerThrowDuringAResize_DoesNotLeaveTheReplacePermanentlyLocked Every generation-replacing resize sets _resizing = true, then allocates and rebuilds the index through the comparer, then clears the flag - with no try/finally. A comparer/OOM throw in that window leaves the collection consistent (nothing published) but _resizing stuck true forever, so every later "lock-free" replace fails its re-check and silently takes the write lock. Observable: SetByKey on an existing key no longer completes while another thread holds the lock (an AddRange whose source blocks inside the locked foreach). A healthy-instance baseline check in the same test proves the harness. Fix: clear the flag in a finally. == test/NumSharp.Tests/Collections/OrderedCollectionsComparerExceptionSafetyTests.cs == ConcurrentOrderedDictionary mutates its key map FIRST and then re-indexes the shifted/moved keys THROUGH the map (one comparer call per key) BEFORE publishing the new list-path store, in: the order-preserving interior removal (RemoveCoreUnderLock), both swap-back branches (in place for word-sized types, copying for wide ones) and RemoveWhere (whose finally re-runs the throwing re-index, so its "publish the consistent prefix" never happens). A comparer that throws there - including the collection's OWN LockRecursionException raised by the documented "evil comparer write-back" defence, which the docs say is "refused ... instead of being allowed to corrupt" - aborts the operation half-applied. 4 x [OpenBugs], deterministic, single-threaded: - interior TryRemove(3), poison key 7: key 3 still enumerates but ContainsKey(3) is false; IndexOf(4..6) wrong - in-place TryRemoveSwapBack(3), poison 9 (the moved key): key 9 enumerates TWICE - copying TryRemoveSwapBack(3) on <int,decimal>: key 3 enumerates but no longer resolves - RemoveWhere(even), poison 7: keys 0/2/4/6 enumerate but no longer resolve (15 disagreements) Fix: resolve every node ref to be re-indexed BEFORE the first map mutation, then mutate throw-free (the same "allocate before touching the map" discipline the type already applies to allocations). The existing Reentrancy_EvilComparerWritingDuringALockedLookup test only covers the refusal on the FIRST lookup (before any mutation), which is why this was not caught. 5 passing controls (run in CI, regression guards): ConcurrentOrderedCompactDictionary never consults the comparer after its probe (tags travel with the index words) and probes BOTH swap-back keys before mutating (in-place and copying branches), so the same attacks either never fire or change nothing; OrderedDictionary rebuilds a private generation and publishes last, so its published state survives the throw (its stuck-flag defect is test 6 above). The audit is generic over the value type: count/keys/values agree, no key enumerates twice, IndexOf/TryGetValue/ ContainsKey agree with every enumerated position, the value law holds, and no key resolves without enumerating. == Notes == - Both files put `using NumSharp.Collections;` INSIDE the namespace block: on net10.0 a file-level using next to `using System.Collections.Generic;` makes `OrderedDictionary<,>` ambiguous with the BCL's .NET 9 generic OrderedDictionary (CS0104 - verified with a compile check). The using attached to the namespace declaration is consulted first. Any net9+ consumer with ImplicitUsings that imports NumSharp.Collections and names OrderedDictionary<,> hits the same error. - Storm tests stop at the first anomaly (a broken build fails in milliseconds; a fixed one runs its 2-3 s budget). They are statistical by nature: on a 1-2 core box the races may not reproduce within the budget. Verification (isolated worktree, Release): - HEAD b7d9093 + these two files: net10.0 -> 10 failed / 5 passed (exactly the [OpenBugs] set fails). - At 5635182 + these files: net10.0 and net8.0 each 10 failed / 5 passed, every failure message checked for the right reason; CI filter (FullyQualifiedName~NumSharp.Tests.Collections & TestCategory!=OpenBugs) -> 172/172 on net10.0 and net8.0 (167 existing + 5 new controls). - A single-threaded differential fuzz of all three types against a List-of-pairs oracle (765K ops over int/string keys, constant-hash / mod-8 / case-insensitive comparers, int/long/decimal/Guid/string values, growth, tail pops, interior removals, AddRange, Clear, every read surface) found NO functional divergence: the defects are concurrency, exception-safety and edge-contract ones. Related findings NOT covered by these tests (for reference): - ConcurrentOrderedDictionary.ReplaceExistingUnderLock, non-atomic TValue: the key-map node is swapped BEFORE the values-array Clone() allocation, so an OutOfMemoryException there leaves the key path holding a value the list path never publishes - contrary to the type's own strand-proofing rule (AddRange's wide-value branch does it in the right order). Inspection only (no deterministic in-process OOM). - docs/proposals/ConcurrentOrderedDictionary.md still says the reference implementation is "concurrency-verified" and "complete" and that 7.1 holds (no lost updates / no garbage reads / never-throwing enumeration); tests 1-3 contradict it, and the proposed surface (KeyValuePair indexer, IDictionary family, TryUpdate/AddOrUpdate) is a superset of what OrderedDictionary implements. - AddRange semantics differ inside the family: OrderedDictionary = add-if-absent (first value wins), ConcurrentOrderedDictionary / ConcurrentOrderedCompactDictionary = upsert (last value wins). Both documented. - OrderedDictionary accepts a negative capacity (documented: <= 0 -> default); both siblings and the BCL throw ArgumentOutOfRangeException. - OrderedDictionary.TryRemove is O(n) even for the LAST entry (no O(1) tail path, so a pop-back drain is quadratic) and both TryRemove and growth re-hash every key through the comparer (no stored hashes/tags) - the compact sibling pops the tail in O(1) and rebuilds from stored tags. - GetValueRefOrNullRef on the public vendored ConcurrentDictionary is now public (documented as intentional in 4499d3e); it bypasses the class's "all public members are thread-safe" guarantee by design.
…ed-collection bugs (+1 by inspection) docs/ORDERED_COLLECTIONS_FIX_HANDOVER.md - a self-contained brief for a fresh session: revalidate each problem first (the fe756f9 [OpenBugs] tests; expected 10 failed / 5 passed on net8+net10, CI filter 172/172), and only if confirmed apply the fix and flip the test to a CI gate in the same commit. Covers, with exact file:line sites at fe756f9, root cause, fix design and acceptance gates: - OrderedDictionary B1 enumerator double-read of the generation field; B2 wide-value replace sharing index/keys across generations (fix = whole-generation copy, COMPACT.md 4.7); B3 lock-free replace lost updates (store-buffering; fix = full fence on BOTH sides, measured 3.26% -> 0 at +3.3 ns/replace, with a proof sketch); B4 null keys; B5 int overflow in the capacity/growth loops (mirror COCD IndexLengthFor); B6 _resizing stuck after a throw (try/finally). - ConcurrentOrderedDictionary B7-B10: comparer called during the post-map re-index of interior removal, both swap-back branches and RemoveWhere; recommended fix = node-handle seam in the vendored clone's NumSharp partial (FindNode / NodeValue / TryRemoveNode by reference) so every comparer call happens before the first mutation; weaker pre-pass fallback documented. B11 (inspection): wide-value replace swaps the map node before the Clone() allocation. - Ground rules for this multi-session repo (isolated worktree, commit --only, no attribution lines, full XML docs, docfx as the only cref check, CS0104-safe test usings), acceptance gates (both TFMs, 20x storm loop, 182 green, measured costs, proposal 7.1 update) and the user decisions NOT to take unilaterally (OrderedDictionary name vs the BCL type, AddRange semantics split, negative capacity, O(n) tail removal, public GetValueRefOrNullRef).
…y_gauss, host-libm sin(2^32) out_where cell After b7d9093, run 35824898562 was green on Windows and Ubuntu (test + interop + every other job) and Deploy Docs passed. macOS arm64 had two remaining reds, both test-harness classes, no product change. (The 4 Karpathy interop failures were ALREADY in the first run 35818966495; the previous pass missed them.) 1. interop-test (macos-latest): 4 Karpathy seeded-initialization tests PackedInitialization_ExactSeededNumpy, OriginalPinnedClassAndBothOriginalChecks_RunThroughPythonNet, Full200By6400_SeededInitializationForwardAndBackward_Exact, ShortPong_AllStepsAndElevenEpisodes_ByteExactAgainstPinnedOriginal. Root cause: NumPy's OWN seeded Gaussian stream differs on arm64. legacy_gauss (numpy/random/src/legacy/legacy-distributions.c) computes `r2 = x1*x1 + x2*x2`. NumPy's `min` CPU baseline is ASIMD on aarch64, where fused multiply-add is base ISA, and clang's default contraction turns that line into fma(x1, x1, x2*x2). The x86-64 baseline is X86_V2 (no FMA3), so every x86-64 wheel rounds both products, and so does NumSharp (RyuJIT never contracts). log(r2) amplifies that 1 ULP near the unit circle: ~14% of draws differ, by 1 to ~50,000 ULP. Evidence: a C# replica that fuses exactly as fma(x1,x1,x2*x2) reproduces the FIRST mismatching element macos-latest reported for each seeded test: seed 7 -> element 24 (LSTM, row 0 is overwritten), seed 23 -> 10 (short Pong), seed 3 -> 44 (full Pong). The other operand order gives 20/22/0. For all these seeds the accept/reject decisions and the number of MT19937 doubles consumed are identical (316, 624, 65276, 1629594), so the stream position and every later uniform() draw still agree. NumSharp keeps the literal (x86-64) stream on every architecture on purpose. It is what every x64 NumPy returns and keeps seeded NumSharp output portable. The win-amd64-authored random_parity_host corpus and the NumSharp.Tests random unit tests pin it; making NumSharp fuse on arm64 would break those, and would still leave gamma/beta/... differing, since clang contracts other lines there too. Fix (test harness, zero coverage loss): InteropTestBase.NumPyLegacyGaussianIsLiteral (true for an X64/X86 PROCESS, since the in-process CPython loads the wheel built for it) and DefineLegacyRandn(), which defines Python `legacy_randn(rs, *shape)`: - where NumPy is literal (x64): exactly rs.randn(*shape), so x64 keeps comparing against NumPy's own output; - elsewhere: re-evaluates legacy_gauss literally over rs's OWN live MT19937 doubles (random_sample = the same next_double), with separately rounded NumPy-ufunc products and sum, and math.log from the platform libm (the one .NET's Math.Log calls). It honours and leaves rs's cached Gaussian and advances rs by exactly the doubles the literal sampler consumes, so later draws on rs still come from NumPy's generator. It never patches NumPy, per KarpathyOriginalSource's stated rule. The pinned original LSTM's own init keeps its direct comparison on x64. On a fusing host that check is split: the original's deterministic bias row must match exactly, and the matrix is compared against the same formula over legacy_randn. The short Pong driver's pg_initialize draws through legacy_randn, so all 44 ticks / 11 episodes / RMSprop update stay byte-exact on arm64 too. The same root cause behind two tests pinned last pass: Nes_Complete300IterationOptimization and Nes_EveryStateAndRewardAcrossOriginal300Updates were marked SkipByteExactOnArm64 as a "1-ULP NEON reduction-width drift". That diagnosis was WRONG. NumSharp's NES trajectory is bit-identical with AVX2, with 128-bit vectors only (DOTNET_EnableAVX2=0) and with hardware intrinsics off; its final weights[1] is 4F2FA1136C6CBB3F, the exact C# value macos-latest printed, while macOS NumPy printed 502F... A fused r2 moves a typical draw ~1 ULP, which the 0.0002 step rounds away almost every iteration. That is why the trace first diverged only at iteration 135. Both tests now draw through legacy_randn and the arm64 skip is REMOVED, restoring their arm64 coverage. Nes_SeededNoiseAndOneStep and NesOriginalObjective_TwelveCompleteUpdates make the same seeded-stream claim and passed on macOS only by luck (8 unaffected draws; 12 steps end before the first visible effect), so they use legacy_randn too. The rule is now uniform: a test that claims NumSharp's seeded Gaussians equal NumPy's draws its expectation through legacy_randn. Tests that feed NumPy-drawn (or NumSharp-exported) inputs are untouched. 2. test (macos-latest) Oracle: OutWhere, 12/6631 comparisons (6 cases x 2 slots) out_unary/sin/2x3x4/float32 at input 2^32: expected 0xbeec8981 (win-amd64 ucrtbase), actual 0xbeec8982 (Apple libm), 1 ULP. (The report prints little-endian BYTES, 81 89 ec be, which read like a factor-4 error; they are not.) |x| = 2^32 is past NumPy's sin Cody-Waite limit (117435.992). There the simd_sincos_f32 port calls MathF.Sin, exactly as NumPy's own kernel calls npy_sinf, so on every host both libraries return that host's libm answer, and the corpus holds ucrtbase's. The whole-tier libm pin (RunHostLibmCorpus) would throw away 6,631 portable out=/where= cells for one element, and the generic "unary ~ULP" excuse cannot reach out_unary (two operands) and deliberately excludes the float32 sin/cos port. Fix: MisalignedRegistry branch (H1) + internal predicate IsHostLibmSinCosHandoff and the IsLibmReferenceHost flag (the same Windows check RunHostLibmCorpus uses). It is narrow by construction: - never on the reference host, so Windows stays strict; - only float32 sin/cos, the plain op or out_unary params.ufunc; - only a float32 input; - every differing element must BE this host's MathF.Sin/Cos answer for a past-limit input in the case's input buffer, and within 2 ULP of the reference. An in-range lane (the bit-exact port) therefore cannot qualify, and a port regression still fails on every host. The host is a parameter so both sides are pinned anywhere. New MisalignedRegistryTightnessTests: - H1_PastLimitHostLibmAnswer_OneUlp_ExcusedOnlyOffTheReferenceHost: sin+cos x plain+out tier; excused off-host, strict on-host, and Classify follows this host. - H1_InRangeLane_GrossMiss_OtherOpOrDtype_NotExcused: an in-range 1-ULP port miss, with and without a past-limit input in the case; a 64-ULP miss; a value within 1 ULP that is NOT the host libm answer; tan; float64. Verification: - Python reference: rs.randn == legacy_randn with the literal path forced on win-amd64 NumPy 2.4.2. 126 draws in 63 seeded sequences over 9 seeds (odd counts, pre-cached Gaussians, empty, 1.28M-element), values + full state tuple + continuation byte-identical. - Interop (Windows, net8.0, OpenBLAS staged, SciPy present): the 32 tests in the 5 touched classes pass on the x64 path, with only the Python flag forced literal-off, and with NumPyLegacyGaussianIsLiteral itself forced false (the arm64 branch, confirmed by reading the property back from the built DLL by reflection). Full suite 733/733 on the final build. - Oracle (Windows): net8.0 and net10.0 184 passed / 24 skipped / 0 failed (208 incl. the 2 new H1 tests). - Oracle (Linux, WSL Ubuntu x64, clone at 035f24f + this diff, where H1 is LIVE): net8.0 and net10.0 169 passed / 39 skipped / 0 failed; both H1 tests pass (Classify != null off Windows). - fe756f9 (another session's tests-only commit that rides along in this push): Collections tests with CI's filter, 175/175 on net8.0 and net10.0. macOS arm64 itself is only reachable through CI; this push is its check. Still pinned to x64, cause NOT the Gaussian stream (no randn involved), left as is: AllThirtyForecastScores (mpe) and RankingDiscounts (method-0 dcg), each 1 ULP on macos-latest.
Nucs
added a commit
that referenced
this pull request
Sep 23, 2026
…y_gauss, host-libm sin(2^32) out_where cell After b7d9093, run 35824898562 was green on Windows and Ubuntu (test + interop + every other job) and Deploy Docs passed. macOS arm64 had two remaining reds, both test-harness classes, no product change. (The 4 Karpathy interop failures were ALREADY in the first run 35818966495; the previous pass missed them.) 1. interop-test (macos-latest): 4 Karpathy seeded-initialization tests PackedInitialization_ExactSeededNumpy, OriginalPinnedClassAndBothOriginalChecks_RunThroughPythonNet, Full200By6400_SeededInitializationForwardAndBackward_Exact, ShortPong_AllStepsAndElevenEpisodes_ByteExactAgainstPinnedOriginal. Root cause: NumPy's OWN seeded Gaussian stream differs on arm64. legacy_gauss (numpy/random/src/legacy/legacy-distributions.c) computes `r2 = x1*x1 + x2*x2`. NumPy's `min` CPU baseline is ASIMD on aarch64, where fused multiply-add is base ISA, and clang's default contraction turns that line into fma(x1, x1, x2*x2). The x86-64 baseline is X86_V2 (no FMA3), so every x86-64 wheel rounds both products, and so does NumSharp (RyuJIT never contracts). log(r2) amplifies that 1 ULP near the unit circle: ~14% of draws differ, by 1 to ~50,000 ULP. Evidence: a C# replica that fuses exactly as fma(x1,x1,x2*x2) reproduces the FIRST mismatching element macos-latest reported for each seeded test: seed 7 -> element 24 (LSTM, row 0 is overwritten), seed 23 -> 10 (short Pong), seed 3 -> 44 (full Pong). The other operand order gives 20/22/0. For all these seeds the accept/reject decisions and the number of MT19937 doubles consumed are identical (316, 624, 65276, 1629594), so the stream position and every later uniform() draw still agree. NumSharp keeps the literal (x86-64) stream on every architecture on purpose. It is what every x64 NumPy returns and keeps seeded NumSharp output portable. The win-amd64-authored random_parity_host corpus and the NumSharp.Tests random unit tests pin it; making NumSharp fuse on arm64 would break those, and would still leave gamma/beta/... differing, since clang contracts other lines there too. Fix (test harness, zero coverage loss): InteropTestBase.NumPyLegacyGaussianIsLiteral (true for an X64/X86 PROCESS, since the in-process CPython loads the wheel built for it) and DefineLegacyRandn(), which defines Python `legacy_randn(rs, *shape)`: - where NumPy is literal (x64): exactly rs.randn(*shape), so x64 keeps comparing against NumPy's own output; - elsewhere: re-evaluates legacy_gauss literally over rs's OWN live MT19937 doubles (random_sample = the same next_double), with separately rounded NumPy-ufunc products and sum, and math.log from the platform libm (the one .NET's Math.Log calls). It honours and leaves rs's cached Gaussian and advances rs by exactly the doubles the literal sampler consumes, so later draws on rs still come from NumPy's generator. It never patches NumPy, per KarpathyOriginalSource's stated rule. The pinned original LSTM's own init keeps its direct comparison on x64. On a fusing host that check is split: the original's deterministic bias row must match exactly, and the matrix is compared against the same formula over legacy_randn. The short Pong driver's pg_initialize draws through legacy_randn, so all 44 ticks / 11 episodes / RMSprop update stay byte-exact on arm64 too. The same root cause behind two tests pinned last pass: Nes_Complete300IterationOptimization and Nes_EveryStateAndRewardAcrossOriginal300Updates were marked SkipByteExactOnArm64 as a "1-ULP NEON reduction-width drift". That diagnosis was WRONG. NumSharp's NES trajectory is bit-identical with AVX2, with 128-bit vectors only (DOTNET_EnableAVX2=0) and with hardware intrinsics off; its final weights[1] is 4F2FA1136C6CBB3F, the exact C# value macos-latest printed, while macOS NumPy printed 502F... A fused r2 moves a typical draw ~1 ULP, which the 0.0002 step rounds away almost every iteration. That is why the trace first diverged only at iteration 135. Both tests now draw through legacy_randn and the arm64 skip is REMOVED, restoring their arm64 coverage. Nes_SeededNoiseAndOneStep and NesOriginalObjective_TwelveCompleteUpdates make the same seeded-stream claim and passed on macOS only by luck (8 unaffected draws; 12 steps end before the first visible effect), so they use legacy_randn too. The rule is now uniform: a test that claims NumSharp's seeded Gaussians equal NumPy's draws its expectation through legacy_randn. Tests that feed NumPy-drawn (or NumSharp-exported) inputs are untouched. 2. test (macos-latest) Oracle: OutWhere, 12/6631 comparisons (6 cases x 2 slots) out_unary/sin/2x3x4/float32 at input 2^32: expected 0xbeec8981 (win-amd64 ucrtbase), actual 0xbeec8982 (Apple libm), 1 ULP. (The report prints little-endian BYTES, 81 89 ec be, which read like a factor-4 error; they are not.) |x| = 2^32 is past NumPy's sin Cody-Waite limit (117435.992). There the simd_sincos_f32 port calls MathF.Sin, exactly as NumPy's own kernel calls npy_sinf, so on every host both libraries return that host's libm answer, and the corpus holds ucrtbase's. The whole-tier libm pin (RunHostLibmCorpus) would throw away 6,631 portable out=/where= cells for one element, and the generic "unary ~ULP" excuse cannot reach out_unary (two operands) and deliberately excludes the float32 sin/cos port. Fix: MisalignedRegistry branch (H1) + internal predicate IsHostLibmSinCosHandoff and the IsLibmReferenceHost flag (the same Windows check RunHostLibmCorpus uses). It is narrow by construction: - never on the reference host, so Windows stays strict; - only float32 sin/cos, the plain op or out_unary params.ufunc; - only a float32 input; - every differing element must BE this host's MathF.Sin/Cos answer for a past-limit input in the case's input buffer, and within 2 ULP of the reference. An in-range lane (the bit-exact port) therefore cannot qualify, and a port regression still fails on every host. The host is a parameter so both sides are pinned anywhere. New MisalignedRegistryTightnessTests: - H1_PastLimitHostLibmAnswer_OneUlp_ExcusedOnlyOffTheReferenceHost: sin+cos x plain+out tier; excused off-host, strict on-host, and Classify follows this host. - H1_InRangeLane_GrossMiss_OtherOpOrDtype_NotExcused: an in-range 1-ULP port miss, with and without a past-limit input in the case; a 64-ULP miss; a value within 1 ULP that is NOT the host libm answer; tan; float64. Verification: - Python reference: rs.randn == legacy_randn with the literal path forced on win-amd64 NumPy 2.4.2. 126 draws in 63 seeded sequences over 9 seeds (odd counts, pre-cached Gaussians, empty, 1.28M-element), values + full state tuple + continuation byte-identical. - Interop (Windows, net8.0, OpenBLAS staged, SciPy present): the 32 tests in the 5 touched classes pass on the x64 path, with only the Python flag forced literal-off, and with NumPyLegacyGaussianIsLiteral itself forced false (the arm64 branch, confirmed by reading the property back from the built DLL by reflection). Full suite 733/733 on the final build. - Oracle (Windows): net8.0 and net10.0 184 passed / 24 skipped / 0 failed (208 incl. the 2 new H1 tests). - Oracle (Linux, WSL Ubuntu x64, clone at 035f24f + this diff, where H1 is LIVE): net8.0 and net10.0 169 passed / 39 skipped / 0 failed; both H1 tests pass (Classify != null off Windows). - fe756f9 (another session's tests-only commit that rides along in this push): Collections tests with CI's filter, 175/175 on net8.0 and net10.0. macOS arm64 itself is only reachable through CI; this push is its check. Still pinned to x64, cause NOT the Gaussian stream (no randn involved), left as is: AllThirtyForecastScores (mpe) and RankingDiscounts (method-0 dcg), each 1 ULP on macos-latest.
…u leg (SVML runners broke byte-exact tests) Run 35828348090 (6ee94ae): interop-test macOS and Windows went GREEN (macOS 632 passed / 101 skipped / 0 failed on net8.0 and net10.0; before: 626 / 103 / 4 failed; the two NES tests un-skipped by 6ee94ae now run and pass on arm64). But interop-test (ubuntu-latest) failed 18 byte-exact tests that NO commit touched: GoogLeNet_LocalResponseNormalization_ByteExact, GoogLeNet_BothSourceLrnStageShapes x2, StableWalk_All200FullSizeLatentFrames, WalkOriginalSlerp (False), SourceSizedForwardAllIntermediatesAndFourGradients x3, OriginalPinnedClassAndBothOriginalChecks, SixUpdateRecurrentTrainingShortRun, CompleteTransformer (16,4,1), AdamFourUpdates, ShortTrainingLedger, ShortPong_AllStepsAndElevenEpisodes, FullForwardLossAndBptt x2, BoundedTraining_AllFortyLosses, FourSourceSizedUpdates. All of them passed on the ubuntu leg of the previous run (35824898562, b7d9093). Root cause: runner hardware changes the LIVE NumPy oracle, not NumSharp. The job prints numpy.show_config(). The failing runner's "SIMD Extensions" found X86_V3 + X86_V4 + AVX512_ICL; the passing runner found only X86_V3. NumPy's Linux x86-64 wheel links Intel SVML (numpy/_core/meson.build: use_svml = linux AND x86_64 AND X86_V4 in dispatch). On AVX512_SKX hardware its float64 exp/log dispatch to __svml_exp8_ha/__svml_log8_ha (loops_exponent_log.dispatch.c.src), and ~20 more f32/f64 ufuncs go to SVML (loops_umath_fp.dispatch.c.src: power, arccos, tan, cbrt, expm1, log1p, ...). The AVX512F float32 exp/log instantiations also replace the AVX2-FMA3 ones NumSharp ports bit-exactly. On an AVX2 runner, and in the win-amd64 reference wheel, the same ufuncs call the platform libm, which NumSharp's .NET Math also calls. That is why the failures are LSTM/RNN/transformer/Pong training (exp/log/tanh chains), LRN (power) and slerp (arccos): every one is a transcendental that SVML answers differently. The Windows wheel is immune by construction: meson_cpu disables X86_V4 ("Considered broken by Highway on MSVC") and AVX512_ICL under MSVC, so it dispatches nothing past X86_V3. The macOS arm64 wheel has none of these features. So only the ubuntu leg can draw an AVX-512 runner and change answers, and whether a run is red depended on which runner GitHub assigned. Fix: interop-test job env NPY_DISABLE_CPU_FEATURES: ${{ matrix.os == 'ubuntu-latest' && 'X86_V4 AVX512_ICL AVX512_SPR' || '' }} NumPy's runtime override (npy_cpu_features.c npy__cpu_check_env): a dispatched feature the CPU HAS is turned off, and one it LACKS is silently accepted for "disable", so AVX2 runners are unaffected. An empty value counts as unset. Both Windows and macOS get the empty value; on Windows the names would only earn an ImportWarning ("not part of the dispatched optimizations (X86_V3)", measured). The pin sits at job level, so "Install numpy" prints show_config() WITH it applied: on an AVX-512 runner X86_V4/AVX512_ICL now appear under "not found", making the pin visible in every log. Verified locally (no AVX-512 hardware here, so the AVX-512 branch itself rests on NumPy's source): - the workflow YAML parses; interop-test env carries both variables; - Linux x86-64 numpy 2.4.2 (WSL, AVX2) with the pin under `python -W error`: imports cleanly, __cpu_features__ X86_V3 True / X86_V4 False / AVX512_ICL False, and np.exp bytes identical with and without the pin (a no-op on AVX2, as intended); - Windows numpy 2.4.2: the names would only trigger the ImportWarning above, and the import succeeds. NumSharp's own AVX-512 code paths are deliberately NOT pinned here. The Oracle test job replays the committed win-amd64 corpus on whatever runner it draws, so it keeps covering them, and nothing in this run implicates them. The nightly fuzz-soak generates fresh expectations with live NumPy on ubuntu runners and can be affected the same way; it is not part of PR CI and is left for a follow-up.
…spatch Run 35828946169 (33a4d2c) is the first FULLY GREEN Build and Release on PR #631: all 16 jobs, the release-only jobs skipped on a PR as designed, and Deploy Docs is green too. - interop-test ubuntu: 638 passed / 95 skipped / 0 failed, net8.0 and net10.0, with numpy.show_config() reporting found [X86_V3] and not found [X86_V4, AVX512_ICL, AVX512_SPR]. - The 6ee94ae run before it: macOS test job green (NumSharp.Tests 15964/15965, Oracle 168 + 40 host-pinned skips, Analyzer 321, net8.0 and net10.0). Its Oracle TRX records exactly "12x float32 sin/cos past NumPy's Cody-Waite limit ... [host-libm pin]" in both frameworks. macOS interop: 632 passed / 101 skipped / 0 failed. What that green run could NOT show is whether its ubuntu runner even had AVX-512. show_config() lists NumPy's EFFECTIVE dispatch, which the NPY_DISABLE_CPU_FEATURES pin (33a4d2c) makes read "X86_V3 only" on every runner, and nothing else in the job names the hardware. So the pin had not been observed working on the hardware it exists for. Hosted ubuntu-latest runners are a mix: run 35824898562 drew an AVX2 box, 35828348090 an AVX-512 (ICL) one, and whether the 18 byte-exact live-NumPy tests went red depended on that. This adds one line to "Report Python host" (all three OSes): cpu AVX512F=<hw> AVX2=<hw> ASIMD=<hw> | numpy dispatch X86_V4=<eff> AVX512_ICL=<eff> read from numpy._core._multiarray_umath.__cpu_features__, the same private module numpy.show_config() reads. NumPy's runtime override clears only the named dispatch GROUPS (npy__cpu_check_env sets npy__cpu_have[feature_id] for each listed dispatch name), so the raw per-instruction flags survive it. "AVX512F=True ... X86_V4=False" is therefore the pin at work, and a future byte-exact red on this leg can be attributed to hardware (or ruled out) from the log alone. f.get() tolerates a missing key, and on the macOS arm64 wheel the x86 flags simply read False, with ASIMD=True. Verified: extracted the step body from the YAML (yaml.safe_load) and ran it under Git Bash on Windows and under WSL Ubuntu python3, with and without the pin. Every run printed the line; this AVX2 host reads AVX512F=False AVX2=True, dispatch X86_V4=False AVX512_ICL=False.
Nucs
added a commit
that referenced
this pull request
Sep 23, 2026
…spatch Run 35828946169 (33a4d2c) is the first FULLY GREEN Build and Release on PR #631: all 16 jobs, the release-only jobs skipped on a PR as designed, and Deploy Docs is green too. - interop-test ubuntu: 638 passed / 95 skipped / 0 failed, net8.0 and net10.0, with numpy.show_config() reporting found [X86_V3] and not found [X86_V4, AVX512_ICL, AVX512_SPR]. - The 6ee94ae run before it: macOS test job green (NumSharp.Tests 15964/15965, Oracle 168 + 40 host-pinned skips, Analyzer 321, net8.0 and net10.0). Its Oracle TRX records exactly "12x float32 sin/cos past NumPy's Cody-Waite limit ... [host-libm pin]" in both frameworks. macOS interop: 632 passed / 101 skipped / 0 failed. What that green run could NOT show is whether its ubuntu runner even had AVX-512. show_config() lists NumPy's EFFECTIVE dispatch, which the NPY_DISABLE_CPU_FEATURES pin (33a4d2c) makes read "X86_V3 only" on every runner, and nothing else in the job names the hardware. So the pin had not been observed working on the hardware it exists for. Hosted ubuntu-latest runners are a mix: run 35824898562 drew an AVX2 box, 35828348090 an AVX-512 (ICL) one, and whether the 18 byte-exact live-NumPy tests went red depended on that. This adds one line to "Report Python host" (all three OSes): cpu AVX512F=<hw> AVX2=<hw> ASIMD=<hw> | numpy dispatch X86_V4=<eff> AVX512_ICL=<eff> read from numpy._core._multiarray_umath.__cpu_features__, the same private module numpy.show_config() reads. NumPy's runtime override clears only the named dispatch GROUPS (npy__cpu_check_env sets npy__cpu_have[feature_id] for each listed dispatch name), so the raw per-instruction flags survive it. "AVX512F=True ... X86_V4=False" is therefore the pin at work, and a future byte-exact red on this leg can be attributed to hardware (or ruled out) from the log alone. f.get() tolerates a missing key, and on the macOS arm64 wheel the x86 flags simply read False, with ASIMD=True. Verified: extracted the step body from the YAML (yaml.safe_load) and ran it under Git Bash on Windows and under WSL Ubuntu python3, with and without the pin. Every run printed the line; this AVX2 host reads AVX512F=False AVX2=True, dispatch X86_V4=False AVX512_ICL=False.
…G1-G13) + 4 upstream ORT issues
A full review of NumSharp.Interop.OnnxRuntime against the complete ONNX Runtime surface: the ORT C# API
and the native C API in refs/onnxruntime (a full checkout at v1.29.0, tags through v1.30.0), with NuGet
current = Microsoft.ML.OnnxRuntime.Managed 1.30.0. Written to docs/plans/onnxruntime-gap-review.md as the
companion of docs/plans/onnxruntime.md (design, decisions not re-litigated) and the business review.
Method: every finding was EXECUTED, not inferred. The existing suite is 159/159 on ORT 1.16.0 (floor) and
on 1.30.0. Three scratch dotnet-run probes (real InferenceSessions, 10 purpose-built .onnx models + the
committed ones) ran on 1.29.0 and 1.16.0 with identical outcomes except the version-dependent cells noted
in the doc. The NumSharp ownership analyzer was run over the package sources (0 diagnostics) and over a
consumer file.
High:
- G1 Fortran-order AsDenseTensor<T> builds a reverseStride DenseTensor that ORT's only DenseTensor->OrtValue
projection (PinAsTensor, OrtValue.shared.cs:1567) refuses ("Tensor of reverseStride is not supported"),
and the documented OrtTensor<T>.Memory -> CreateTensorValueFromMemory route feeds it TRANSPOSED with no
error (identity output row0 [0,4,8,1] for source [0,1,2,3]). The existing test never fed it to ORT.
- G2 ORT's managed GetTensorBufferRawData narrows the byte length with (int) (OrtValue.shared.cs:712-716):
AsNDArray/ToNDArray throw ArgumentOutOfRangeException for 2-4 GB tensors and silently wrap at >= 4 GB
(lease memory pressure 42x too small at 4.4 GB). Export (AsOrtValue) is fine; the README/website/
TooLargeForSpanMessage advice "share it zero-copy instead" is wrong for import.
- G3 OrtIoBinding.BindInput/Output (native IOBinding copies the OrtValue), SessionOptions.AddInitializer
(session reads NumSharp memory for its lifetime) and OrtValue.CreateSequence (TensorSeq::Add shares data)
keep dereferencing NumSharp memory after the OrtTensor handle and its ARC pin are gone (LiveExports == 0
while the binding still reads the buffer) - a use-after-free once the NDArray dies. Proposes one "keeper"
concept: NDArrayIoBinding, AddInitializer(name, NDArray), ToOrtSequence(NDArray[]).
Medium:
- G4 Run(dict) with default outputNames throws on any non-tensor output (skl2onnx classifiers: label +
ZipMap); a string label output is unreadable through the tier at all.
- G5 ToMap/ToMaps refuse string-keyed maps (every string-labelled ZipMap) though ORT reads them.
- G6 the Run tier accepts only NDArray inputs: no string/sequence/map/raw-OrtValue, so a model with one
string + one float input cannot go through it.
- G7 PrepareInput refuses optional(tensor) inputs (with a wrong CreateSequence/CreateMap hint) and
overridable initializers - both of which ORT accepts (inference_session.cc:3016, LookupInputMetadata).
Low / docs:
- G8 no RunAsync tier; ORT's own RunAsync pins its arrays until completion only from 1.30.0 (#32015).
- G9 ToNDArrays/ToMap(s) copy twice (GetValue already returns a fresh ORT copy) - 1.26x on 16 MB.
- G10 IsCpuAccessible refuses QnnHtpShared (CPU host-accessible, MemTypeDefault) and CpuAligned4K on newer
ORT; plain Run outputs are always CPU, device tensors only arise through IoBinding.
- G11 the multi-output Run result is a non-disposable IReadOnlyDictionary the analyzer cannot see.
- G12 BFloat16 bridgeable today by widen/RNE-narrow without a Core dtype.
- G13 CI "current" pin 1.29.0 (NuGet 1.30.0, suite green), "148 tests" (now 159), the 2 GB and F-order
claims, and the csproj FileLoadException rationale (ORT 1.16.0's DLL has AssemblyVersion 0.0.0.0).
Upstream ORT (documented, not fixable here): a None optional graph output AVs inside ORT's own C#
session.Run (onnxruntime_typeinfo.cc:169-176 null deref); the (int) truncation above; float8/int4 graph
I/O unreachable from C# (metadata lookup throws "Unregistered TensorElementType ... DataTypeMax"); no
sparse-tensor API in C#.
Also records what was verified NOT to be a gap and a recommended implementation order (G1 first).
Documentation only - no code, test or CI change in this commit.
…ce, all corpus families, catalogue + completeness gate (NOT merged: gate red until np.ma/histogram leaks are fixed) Side branch off journey4 (256224d). This is work in progress, parked here so the shared PR branch stays green: the extended gate intentionally fails until the leaks it finds are fixed. Full state, audit numbers and ordered next steps are in docs/plans/leak-audit-completion.md. WHY The leak oracle (UndisposedIntermediateTests + ScopeAudit, [FuzzMatrix]+[ScopeAudit]) was green, but its "every op gated at zero" held only for ops with ordinary corpus rows. Cross-referencing the ApiInventory surface (coverage/NumSharp.Tools.ApiInventory: 8 [ModuleName] modules, 847 methods + 91 properties + 42 fields) against the op keys the sweep actually MEASURED found 265 of 961 module members never leak-measured: ndarray 94, np 75, np.ma 62, np.linalg 21, np.random 13. Measured with the gate's own protocol: - np.ma (the ma_* tiers were excluded) leaks in 36,696 of 68,860 cases: 84 of 149 ops, up to 29 buffers per call (std 29, median 28, var 22, average 12, anom 11, mean 9). The replay takes 12m47s while leaking, over CI's 10-minute Oracle step. - The histogram family has no corpus rows and leaks 4-13 buffers per call; apply_along_axis 1. - 2,441 error paths (expects_throw), now measured: 0 leaks. - 276 LAPACK cases always threw (backend suppressed), so they were never measured. - NDW012 already flags 214 sites statically (176 in NDMaskedArray.cs, 34 in histogram). WHAT (test/NumSharp.Tests.Oracle/Fuzz/) - ScopeAudit: MeasureConfirmedTraffic / MeasureConfirmed (the screen, settle, confirm protocol, shared by every replay). - UndisposedIntermediateTests (now partial): - SharedSweep, a Lazy with one sweep per process, shared with the completeness gate. - SweepResult/SweepAccumulator gain per-family counts, MeasuredByOp (+ rnd:/grnd: coverage keys), ErrorMeasuredByOp and ThrewByOp. - Error paths are measured, not skipped. Per-family non-vacuity floors; the index and error-path floors are estimates to verify on the first run. - FIX: the bypass freshness range now starts at Storage.InternalArray.Address. It used `Address - Offset*isz`, but a corpus operand's Storage.Address IS the base, so the range sat shifted down by the offset. That was a latent false bypass for contiguous slices in the last `offset` elements (0 disagreements today). - AssertNoUnclassifiedEscapes extracted so every gate shares one verdict. - UndisposedIntermediateTests.Families: ma_* replay (NDMaskedArray operands; the keep-set is recomputed per execution because mutators swap operand masks) and index_* replay (indexer get/set, bases built under a harness NDScope). Also DisposeAny, and MaskedSingletons(): np.ma.nomask and the masked constant's arrays are returned BY mask_or/getmask/.mask, and disposing them would recycle a shared scalar slot process-wide. - UndisposedIntermediateTests.Backend: the whole ordinary corpus under OpenBLAS (threads=1), Inconclusive where none loads; BackendOnlyOpKeys must be measured when it does. - LeakCatalogue(.NDArray/.Masked): ~360 direct invocations for members no corpus row reaches, plus the LeakFixture. NOT YET COMPILED. - LeakSurfaceCoverageTests: LeakSurface.Enumerate() (the inventory tool's discovery rule, operators, the object_surfaces.py owners, NDMaskedArray, NDArray<T>) and EveryInventoryMember_IsLeakAudited with self-retiring mappings. NOT YET COMPILED. - IndexOracleTests helpers and OracleSurfaceCoverageTests alias maps widened to internal (with docs) for reuse. NEXT: see docs/plans/leak-audit-completion.md. In order: compile fixes, the catalogue and reflective-read runner, np.ma hand-scopes (the weaver refuses class carriers), the histogram [NDScoped], the README and chokepoint pin, then merge to journey4.
…whole-generation wide replace, single-read enumerator, null keys, capacity guard, unstickable resize flag Fixes the six OrderedDictionary<TKey,TValue> bugs pinned as [OpenBugs] in fe756f9 (handover docs/ORDERED_COLLECTIONS_FIX_HANDOVER.md, B1-B6) and flips their tests to CI tests. Every bug was first revalidated in an isolated worktree at 035f24f: the 15-test revalidation filter failed 10 / passed 5 on net10.0 AND net8.0 with exactly the failures the handover predicted (the other 4 failures are ConcurrentOrderedDictionary's, fixed in the next commit). B1 - enumerator read the volatile generation field twice GetEnumerator() was `new Enumerator(_t._values, Volatile.Read(ref _t._count))`: a growth published between the two reads paired the OLD value array with the NEW count -> IndexOutOfRangeException or phantom default values mid-foreach. Now reads `_t` once into a local and takes both fields from it (as ToArray/AsValuesSpan already did). Grepped the type: no other expression reads `_t` twice (SetByKey's `ReferenceEquals(_t, t)` re-read is the intentional generation check). B2 - a non-atomic (wide TValue) replace shared the index and keys across generations The wide SetByKey cloned ONLY the value array and published `new Tables(t._index, t._keys, values, ...)`. The next in-place append wrote its index word and key into those SHARED arrays, so a reader still holding the older generation resolved the new key to its OWN never-written value slot: `(true, default)` for a present key. Fix (ConcurrentOrderedDictionary.COMPACT.md section 4 rule 7): ReplaceWideUnderLock publishes a WHOLE new generation - index cloned word-for-word (same length/shift/slots, so nothing is re-hashed and no user code runs), fresh keys, fresh values. No array is ever shared by two generations now, which is what the count-gate-free read relies on. Cost (measured, below): 1.95x at N=1K, 2.59x at N=100K for the O(n) wide replace - three arrays instead of one; the correctness price, same O(n) class. B3 - the lock-free replace lost completed writes (store-buffering / Dekker race) Replacer: plain value store, then volatile loads of _resizing/_t. Resizer: volatile store of `_resizing = true`, then copy the value array. Release/acquire cannot forbid "each side's load misses the other side's store" on x86-TSO or ARM64, so the copy could read the old value while the replacer saw the flag down -> the resize published a generation without the completed write. Fix, both sides fenced: * replacer: Interlocked.MemoryBarrier() between the value store and the re-check; the flag is read with Volatile.Read BEFORE the generation (a flag already lowered by a finished resize then guarantees its publish is visible, so the generation check cannot miss it); * _resizing is now an int raised by BeginResize() = Interlocked.Exchange(ref _resizing, 1) (full fence) before any read of the live value array, lowered by EndResize() = Volatile.Write(ref _resizing, 0); * TryRemove reads the value it RETURNS inside the fenced window too - read before the flag it could be the value a lock-free replace had just overwritten, a replace that then committed: a lost-update variant on the removal's result that the handover did not list. Evidence (in-process A/B, pre-fix vs fixed source embedded in one script, B3's single-writer-per-key read-your-own-write storm, 4 replacers + 1 churn thread, 3 s x 2 reps each, i9-13900K): pre-fix: 3.43 % / 3.79 % / 1.40 % / 3.83 % / 4.23 % / 1.61 % of completed replaces lost (6 runs, 68M replaces) fixed: 0 lost - 18.3M replaces under 9.0M copying resizes with this commit's code, and 53M more under 11.7M resizes with the build-first TryRemove variant described below. B4 - null keys were accepted EqualityComparer<string>.Default.GetHashCode(null) is 0, so a null key was silently stored, found and enumerated. Added the siblings' NullCheck (`!typeof(TKey).IsValueType && key is null` - the typeof guard keeps Debug codegen from boxing value-type keys; Release folds it) to TryGetValue, ContainsKey, IndexOf, TryAdd, SetByKey, TryRemove and per pair in AddRange; GetByKey / this[key] get / GetOrAdd get it through TryGetValue (their first call), this[key] set through SetByKey. All 11 entry points of the test now throw ArgumentNullException. B5 - a capacity of ~752M or more hung the constructor `int len = 8; while (len < cap*100/70 + 1) len <<= 1;` wrapped len to int.MinValue then 0 (`0 << 1 == 0`) and spun forever; AppendGrow's `newLen <<= 1` had the same overflow. Now IndexLengthFor(int) computes the length in long (RoundUpToPowerOf2 of max(8, cap*100/70+1)) and refuses > 2^30 words (the largest power-of-two int[]) with ArgumentOutOfRangeException BEFORE allocating; AppendGrow refuses to double a 2^30-word index the same way. The limit is MaxCount = 751,619,276 entries (2^30 x 70 %), identical for the constructor and growth; ThrowTooManyEntries mirrors the compact sibling's refusal (AOOR, paramName "capacity"). The test's `new OrderedDictionary<byte,byte>(800_000_000)` now throws in microseconds. B6 - a throw during a resize left _resizing stuck true forever Every resize set the flag, re-hashed survivors through the user comparer, then cleared it with no try/finally, so a comparer throw silently turned every later "lock-free" replace into a locked one. Now every generation replacement (TryRemove, Clear, AppendGrow, the wide replace) lowers the flag in a finally. Clearing after a failed resize is safe: nothing was published, so a replacer that stored into the live generation stored into the generation that stays live. Resize shape, per operation (all under the write lock): * AppendGrow and ReplaceWideUnderLock build their new keys + index BEFORE raising the flag (GrownWith / the clone), so their fenced window holds only the live-value copy and the publish; a lock-free replace that overlaps the (comparer-calling) rebuild commits without waiting. Measured: growth path 1.01x. * TryRemove keeps the handover's prescribed order - flag (full fence) -> removed-value read -> copy keys and values -> rebuild -> publish -> lower in finally. Building the index before the flag was tried and measured 1.16-1.21x slower single-threaded (the values must be copied right after the keys; allocating the value array inside the window, dropping the try/finally or the fence each changed nothing - diagnostic variants), while the prescribed order measured 1.00x (3 processes). The trade-off it gives up (a replace overlapping the rebuild falls back and waits, exactly as before the fix) was measured too: in the churn storm the build-first order completed ~60 % more replaces but ~38 % fewer removals. Measured cost (in-process A/B, pre-fix vs fixed, pinned P-core, 2.5 s spin-up, 150 ms warm-up per op, best-of-15 x 5 interleaved blocks, median of 3 processes; A/A harness noise on the same rows 0.99-1.04): SetByKey existing <int,int> lock-free 1.86 -> 4.88-5.08 ns (+3.0-3.2 ns: the fence; handover expected ~+3.3; +2.6-3.2 across every run of the session) SetByKey existing <int,long> lock-free 1.86 -> 4.88 ns SetByKey existing <string,int> 5.27 -> 6.74 ns (+1.5 ns) TryGetValue <int,int> / <string,int> 1.46 -> 1.46 / 5.18 -> 5.18 ns (null check folded / one compare) wide SetByKey <int,decimal> N=1K / 100K 1.95x / 2.59x (B2's whole-generation copy) TryRemove interior@0 + re-add 1.00x at 1K (prescribed order; see above) build 100K from empty via TryAdd 1.01x Context: the lock-free replace stays ~3x faster than the sibling maps' locked replace (14.6 ns). Documentation (every edited member fully XML-documented, WHY comments on the protocol): * type remarks rewritten for the fenced handshake, generation ownership (no shared arrays), the measured fence cost, null keys + the 751,619,276-entry limit, and TWO NAMED RELAXATIONS of the lock-free replace that the store-then-verify design cannot avoid while a replace is retrying under the lock: (1) another reader can see new -> old -> new for that key; (2) an overlapping TryRemove can return the retrying replace's value while the retry (an upsert) re-adds the key. Once SetByKey returns every later read sees its value or newer. Neither sibling map has them (their replace takes the lock). * docs/proposals/ConcurrentOrderedDictionary.md: section 7.1 describes the fenced protocol (and why release/acquire is insufficient), whole-generation replace, single-read snapshots; 7.2 replaces "per key, operations are linearizable" with the precise contract + the two relaxations, fixes the memory-model claim (release/acquire PLUS full fences), adds null keys and the capacity limit; 7.3 notes the retry flicker; section 8 stops calling the implementation "complete" (it lists what the reference implementation lacks of the proposed API), says its perf numbers predate these fixes, and replaces the "0 lost updates" gun claim with the five defects + fixes + the storm evidence, citing the contract tests. Tests: test/NumSharp.Tests/Collections/OrderedDictionaryOpenBugsTests.cs -> OrderedDictionaryContractTests.cs (git mv, history kept; the class name no longer says "OpenBugs" now that every test in it passes), the six [OpenBugs] attributes removed, each test's doc rewritten from "Bug:" to "Defect caught:" + the fix. Verification (isolated worktree; the final code): * the 15 revalidation tests: 15/15 on net10.0 and net8.0, Release AND Debug; * storm tests (enumerator regrow, lost-update, stuck-flag) looped 20x per TFM twice (once before the TryRemove order decision, once on the final binaries): 80/80 runs, zero flakes; * Collections CI filter: 172 -> 182 on both TFMs (the +10 are this commit's 6 and the next commit's 4); * full NumSharp.Tests CI filter green on both TFMs (net10.0 16,039 passed / 0 failed; net8.0 16,038 / 0). Not changed (handover section 4 - the user's decisions): the OrderedDictionary<,> name vs the .NET 9 BCL type (CS0104), AddRange's keep-first semantics vs the siblings' upsert, negative capacity = default, the O(n) tail removal / re-hash-on-resize perf opportunity.
…xception-safe (node handles, trusted one-pass path), wide replace allocates before it swaps Fixes the four ConcurrentOrderedDictionary (COD, the shipped node type) bugs pinned as [OpenBugs] in fe756f9 (handover docs/ORDERED_COLLECTIONS_FIX_HANDOVER.md, B7-B10) plus B11 (by inspection), flips the four tests to CI tests, and marks the handover executed. Revalidated first in an isolated worktree at 035f24f: all four failed on net10.0 and net8.0 with exactly the predicted audits (key 3 enumerates but does not resolve; key 9 enumerates twice; 0/2/4/6 enumerate but do not resolve); the five COCD/OrderedDictionary controls passed. Root cause (B7-B10) Interior removal (RemoveCoreUnderLock), both TryRemoveSwapBack branches and RemoveWhere removed the key from the map FIRST, then re-indexed the shifted/moved keys through the map - one call into the user's key comparer per key - and only then published the new Store. Any comparer throw in that window aborted the removal half-applied, permanently. That includes COD's OWN LockRecursionException: the documented "evil comparer" defence refuses a write-back from inside GetHashCode, and the refusal itself corrupted the collection. RemoveWhere was worse: its finally re-ran the comparer and skipped the publish altogether. Fix: resolve every node first, then mutate without the comparer New internal members in the NumSharp partial of the vendored clone (ConcurrentDictionary.RefAccessors.cs, never the vendored body): * NodeHandle - a readonly struct around one live Node (typed object: Node is private and may not appear in an internal signature); Value = ref to the node's value field via Unsafe.As (FindNode is the only producer). * FindNode(key) - GetValueRefOrNullRef's exact walk (same hash + comparer calls, same value-type/default- comparer split), returning the node; AggressiveInlining like the seam. * TryRemoveNode(handle) - unlink BY IDENTITY: bucket from the node's stored _hashcode, chain compared by reference, TryRemoveInternal's stripe lock + tables-changed retry + _countPerLock bookkeeping, no comparer. Handles share the ref seam's lifetime rule (valid inside one serialized critical section; a growth re-creates nodes, so a stale handle is reported absent by TryRemoveNode). COD then: * TryRemove / RemoveAt resolve the removed key's node once (FindNode) and unlink it by handle - one hash walk fewer than before (GetValueRefOrNullRef + TryRemove(key)); * TryRemoveSwapBack resolves the MOVED key's node before any mutation (both branches); * RemoveCoreUnderLock / RemoveWhere resolve every node they will unlink or re-index before the first mutation, then write indices through the handles (RewriteIndexUnderLock now takes a handle); * RemoveWhere pre-allocates the Store its finally publishes (an OOM there, after unlinking keys, would strand them), so the finally can no longer throw. A refused write-back now aborts the removal with nothing changed; the four tests' full two-path audits pass. Two paths, because resolve-first is NOT neutral (the handover predicted ~neutral; measured otherwise) Micro-decomposition on the vendored map, per shifted key: one-pass lookup+write 1.18-1.25 ns; resolve-first (lookup + write-barriered handle store, then a second pass over the nodes) 2.0 ns cache-resident, 3.1 ns at 100K (the second pass is memory-bound). Pooling the handle array (ArrayPool) made small N worse and changed nothing at 100K, so allocation was not the cost - the extra traffic is intrinsic. But the hazard only exists when a lookup can run USER code, so: * _lookupsRunNoUserCode = TKey is a framework type whose default equality/hashing no user code can override (primitive, enum, string, decimal) AND the map uses the default comparer: those lookups cannot throw or write back, so the re-index keeps the one-pass form (pre-fix order), which is exception-safe there; * everything else (any custom comparer - even a StringComparer - and every user key type) takes the resolve-first path. Placement trap found on the way: the trusted one-pass loop was byte-identical in the tier-1 JIT output (DOTNET_JitDisasm) to the pre-fix loop yet ran ~0.4 ns/key slower inlined in the now-larger RemoveCoreUnderLock; as its own method (ReindexOnePassUnderLock) it runs at or above the pre-fix speed. B11 - out-of-memory could strand a wide-value replace ReplaceExistingUnderLock (non-atomic TValue) swapped the map node BEFORE the O(n) values.Clone() and the new Store, so an OOM left the key path ahead of the list path. Now: clone + fill + new Store first, then the node swap (the vendored indexer hashes and allocates the replacement node before linking, so it is strand-proof), then the throw-free publish - the order AddRange's wide branch already used. Measured (in-process A/B: pre-fix and fixed COD sources embedded in one script, pinned P-core, 2.5 s spin-up, 150 ms warm-up, best-of-15 x 5 interleaved blocks; median of 3 processes; A/A harness noise 0.99-1.01): interior TryRemove@0 + re-add, default comparer N=1K 0.87x N=10K 0.89x N=100K 0.98x (faster) same, custom comparer (resolve-first) N=1K 1.53x N=10K 1.62x N=100K 2.37x (the safety price) RemoveWhere(even) N=100K default 0.95x custom 1.33x TryRemoveSwapBack drain N=100K default 0.95x custom 0.97x TryRemoveSwapBack drain <int,decimal> (copying) 1.00x tail-pop drain N=100K 0.99x SetByKey existing <int,decimal> (B11 reorder) 0.99x Recorded in ConcurrentOrderedDictionary.TODO.md ("Comparer exception safety"). New finding, reported not fixed (OOM-only, hot add path, wants its own measurement) The vendored map grows its table AFTER inserting a key (TryAddInternal -> GrowTable), and AddRange allocates its published Store in its finally, so an out-of-memory raised by either can leave the key just added resolving without enumerating. The class remarks and the TODO's thread-safety contract now say so instead of claiming full OOM-tightness; the fix sketch (pre-grow before the insert via a partial member; allocate AddRange's holder when it goes fresh) is in the TODO. Docs (every new/edited member fully XML-documented, WHY comments on each non-obvious body): * COD class remarks: exception-tightness now covers comparer calls, names the two OOM-only gaps; RemoveWhere remarks: a throwing comparer aborts the pass with nothing removed. * ConcurrentOrderedDictionary.TODO.md: design paragraph, Gap B3 (re-index made exception-safe), thread-safety contract (comparer clause, corrected; the open OOM gap), a new "Comparer exception safety" section with the decomposition + A/B table + the placement trap, memory-churn rows (+8 B per resolved entry on the resolve-first path only), clone deviations (the partial's four members and the internals they read), gates. * ConcurrentDictionary.cs header (NumSharp-authored) + RefAccessors header list the new partial members. * docs/ORDERED_COLLECTIONS_FIX_HANDOVER.md: an "EXECUTED" status block with the gates, the deviations from the plan and the new findings. Tests: OrderedCollectionsComparerExceptionSafetyTests - the four [OpenBugs] attributes removed, class remarks and each test's doc rewritten as regression guards; the five COCD/OrderedDictionary controls unchanged. Verification (isolated worktree; final code): the 15 target tests 15/15 on net10.0 + net8.0, Release and Debug; Collections CI filter 182/182 on both TFMs; full NumSharp.Tests CI filter green on both TFMs (net10.0 16,039 / net8.0 16,038 passed, 0 failed); docfx metadata: no new cref warnings in Collections.
…ates run; 9 Core leak/bug fixes (shape setter, ReplaceData ARC, ToJaggedArray OOB, poly1d, mvn, tofile, ogrid, apply_*, vectorize) Continues 8df2598 on the side branch (NOT merged to journey4: the catalogue gate is still red on the np.ma and histogram families by design until those leaks are fixed - see docs/plans/leak-audit-completion.md "Session 2 progress" + "Next steps"). Harness (test/NumSharp.Tests.Oracle/Fuzz): - Catalogue/gate files now COMPILE (masked-indexer casts, GetData(int[]), FromMultiDimArray<double>). - UndisposedIntermediateTests.Catalogue.cs (new): SharedCatalogue + Catalogue_EveryEntry_Leaves- NoUndisposedIntermediates - warm + ScopeAudit.MeasureConfirmedTraffic per entry, pool-bypass check via FreshBytesAny against the fixture's base-buffer ranges (LeakFixture.Ranges), backend-only entries under OpenBLAS threads=1 (skipped + reported where none loads), unrunnable entries fail as harness errors. LeakCase.Throws + T(...): an always-throwing member is measured as an ERROR path and a Throws entry that starts returning fails as stale. - UndisposedIntermediateTests.Properties.cs (new): SharedPropertyReads + EveryPropertyAndField_Read_ LeavesNoUndisposedIntermediates. Every surface property/field (288 members) is read - settable ones round-trip written - on targets built INSIDE the measured region (so a getter caching an allocation on its owner is caught at the owner's disposal); two reads per region tell owner-held parts (same reference) from fresh ones (disposed, also when the setter throws); MissingBackendException reads re-run with OpenBLAS. GREEN: 539 reads + 15 error paths. - CollectDisposables: the one result walk DisposeAny, the freshness check and the stable-part detection share (a lazy IEnumerable<NDArray> is enumerated exactly once). SweepAccumulator.ToResult + Direct count. Core fixes (each found by the new gates, documented in place): - ndarray.shape / Shape setters -> SetShapeInPlace = NumPy's array_shape_set over the existing ReshapeCore port: adopt the no-copy view's dims/strides (WRITEABLE/ALIGNED kept), else AttributeError("Incompatible shape for in-place modification. Use `.reshape()` to make a copy with the desired shape.") verbatim. The old UnmanagedStorage.Reshape route COPIED a non-contiguous array in place and swapped its buffer under the NDArray's ARC reference: 2 pooled buffers stranded per assignment + a NumPy divergence (probed 2.4.2: T.shape=(4,3) ok strides (8,32); (12,)/(2,6) AttributeError; [:, ::2] (2,6) ok). - NDArray.ReplaceData (6 overloads): MoveArcReference re-points the array's counted reference to the new buffer. Before: the old buffer stayed referenced forever AND Dispose released a reference on the adopted buffer it never took - x.ReplaceData(nd) could free nd's buffer under nd. - ToJaggedArray: no densified Storage.GetData<T>() temp (leaked for views); rank-2 now reads by coordinate (GetValue<T>(i, j)). It returned WRONG values for a transposed view (0,5,10 vs 0,4,8) and read OUT OF BOUNDS on a column slice (-3.0e-241). - ndarray.tofile (binary): the C-order copy of a non-contiguous view is released in a finally. - np.ogrid: [NDScoped] NdGridLines (each line's wrapper stranded one buffer per slice). - poly1d: the constructor DETACHES its field (Returns re-tracked it into the caller's scope, which released a live polynomial's coefficients); FromFresh/WrapQuotient release every intermediate behind the operators, deriv, integ; Call(double), ==, indexer get/set, np.polyval(poly1d, poly1d) seed fixed; np.polymul disposes its temporary polynomials. - np.random.multivariate_normal (both overloads) + ComputeSvdTransform: 7 raw UnmanagedMemoryBlock scratch buffers per call freed in finally (FreeScratch, the dirichlet DangerousFree idiom), throw paths included. - np.apply_along_axis releases buff/inarr; np.apply_over_axes releases superseded expand_dims views; np.vectorize signature mode releases its broadcast views. Ownership contract (documented in each XML doc): arrays a user CALLBACK returns, and slices handed to it, stay the caller's - never disposed - so the catalogue drives these APIs with non-allocating callbacks. Found, not fixed (measured as error paths, reason in the entry label): NDArray.AsStringArray (legacy ToString layout), GetStringAt/SetStringAt (ndim vs ndim-1 coordinates), NDArray.Normalize (NotImplemented). State: catalogue 443/451 measured (8 by-design throwers on error paths); the only remaining catalogue escapes are np.ma (48 families) and histogram (4 ops). Corpus sweep / backend pass / completeness gate not yet rerun.
…netGhActionsCicd - two projects that run NumPy, pandas, SciPy, scikit-learn, Matplotlib and PyTorch over NumSharp arrays through NumSharp.Interop.pythonnet: a console app that embeds the Python on the local machine (virtual environments included), and its asserted MSTest twin with the GitHub Actions workflow that runs it on Windows, Linux and macOS
WHAT
examples/NumSharpPythonnetLocalExample (console app, net10.0)
- Program.cs starts the engine, prints which Python it embedded and the versions of the six
libraries, runs the six examples and prints where Matplotlib saved its figure.
- PythonHost.cs finds a Python and owns the engine. Discovery order: the interpreter that
NUMSHARP_PYTHONNET_PYTHON names (binding), then .venv in the current directory or beside the
project file, then python3 / python on PATH (py on Windows). It runs the interpreter once to
read sys.version_info, sys.executable, sys.prefix, sys.base_prefix, INSTSONAME and LIBDIR as one
JSON object (ASCII-safe for any code page), checks the version against
PythonEngine.Min/MaxSupportedVersion, and finds the shared library of the BASE install
(pythonXY.dll, LIBDIR/INSTSONAME, the macOS framework binary, lib/libpythonX.Y.so).
- Libraries/{NumPy,Pandas,SciPy,ScikitLearn,Matplotlib,PyTorch}.cs are written like the twelve
tutorials of examples/NumSharp.Interop.pythonnet.Examples: a dynamic Python namespace, one
NDScope for each example, Py.GIL() around the Python parts, no output, and the outcome of each
line in the comment beside it.
- requirements.txt pins numpy 2.4.2, pandas 3.0.5, scipy 1.16.3, scikit-learn 1.9.1,
matplotlib 3.11.2, torch 2.13.0 and adds PyTorch's CPU-only index.
examples/NumSharpPythonnetGhActionsCicd (MSTest 3.8.3, net10.0, 18 tests)
- The same code as the six examples, with each comment turned into an assertion:
NumPyTests (3), PandasTests (4), SciPyTests (3), ScikitLearnTests (3), MatplotlibTests (2),
PyTorchTests (3).
- PythonEnvironment.cs ([AssemblyInitialize] / [AssemblyCleanup]) embeds exactly the interpreter
that NUMSHARP_PYTHONNET_PYTHON names. Unset under CI=true is a FAILURE with the line to add to
the workflow; unset on a developer machine is Inconclusive. It imports the six libraries before
the first test and writes a version table to GITHUB_STEP_SUMMARY. [assembly: DoNotParallelize]
because the MSTest template turns parallel test methods on.
- PythonTest.cs gives each test a new Python namespace and fails a test whose
NDArrayPythonInterop.LiveExports / LiveImports do not return to their value before the test
(it pumps both collectors for up to 15 seconds).
.github/workflows/example-pythonnet.yml
- ubuntu-latest, windows-latest, macos-latest; fail-fast off; triggers: workflow_dispatch, and
push / pull_request on master / main limited by a paths filter to the example, the interop
package and the workflow file.
- setup-dotnet 10.0.x, setup-python 3.12 with a pip cache keyed on the example's
requirements.txt, pip install into the job's interpreter, dotnet build, dotnet test with
NUMSHARP_PYTHONNET_PYTHON = steps.python.outputs.python-path, upload of TestResults (the .trx
report and the figure that the Matplotlib test attached).
- -p:WarningLevel=0 on the build step: inside this repository the project builds NumSharp from
source, and that build prints 557 warnings (measured, from-scratch Release build).
Both project files refer to src/NumSharp.Interop.pythonnet when that path exists and restore the
NuGet package NumSharp.Interop.pythonnet otherwise, so a copied folder needs no edit. Both projects
are in SciSharp.NumSharp.sln under "examples". Pointers were added to the interop package README,
the tutorials README and the "See also" list of docs/interop/pythonnet-numpy.md. .claude/CLAUDE.md
records the workflow and the facts below in "CI Pipeline".
WHY THESE DESIGN CHOICES (each one measured in this session)
- PythonEngine.ProgramName is set for EVERY interpreter, a virtual environment or not.
Windows, no program name: sys.executable is the .NET host, and a venv is not adopted (numpy was
imported from the base install). Linux, no program name: the engine takes its paths from the
first python3 on PATH; with another Python there (the system 3.12.3 against a 3.12.14 library)
the process died with a segmentation fault. With the name set, both platforms give the right
sys.executable, sys.prefix and packages whatever PATH holds. test/NumSharp.Tests.Interop's
PythonSession sets it for virtual environments only; this goes one step further.
- pythonnet 3.0.5's Runtime.Shutdown runs 87 full PyGC_Collect passes (3 + 20 + 20 attempts of two
passes each, plus one; the loop does not stop early: read from the v3.0.5 source). With the six
libraries imported one pass takes about 0.1 s, so the shutdown took 7.3 s. The Local example
runs gc.collect(); gc.freeze() first: 0.2 s, exit code 0, and the interop's export sweep still
brings LiveExports to 0. The test fixture keeps the plain shutdown (speed does not matter
there, and it is the sequence the interop suite runs on all three systems).
- torch.optim.SGD lazily imports about 800 more modules (torch._dynamo, sympy): 1.3 to 2.5 s in
plain Python, and the first run of that test took 19 s in the test host. The autograd example
updates the weights by hand: the same result (loss 1.2e-31), 0.05 s, and autograd stays visible.
- torch.__version__ is an instance of a str subclass: PyObject.As<string>() throws
("'TorchVersion' value cannot be converted to System.String"); ToString() is Python's str().
- The .NET app host (<app>.exe, testhost.exe) is not long-path aware on Windows. A module file
past 260 characters failed to import as "No module named
sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode" while python.exe and
"dotnet <app>.dll" imported it. Both READMEs say so.
- The pandas view example uses a Series that pandas owns alone. Under Copy-on-Write a write to a
frame does not reach a column that was taken from it before (the first version asserted that it
does, and failed).
- macOS, not run here: the macOS arm64 wheels of torch 2.13.0 and scikit-learn 1.9.1 each
contain a libomp.dylib (listed from the wheels). LLVM's runtime aborts when a second copy
starts (OMP: Error #15) unless KMP_DUPLICATE_LIB_OK is set, and sklearn/__init__.py sets it at
import (line 56 of 1.9.1). Both hosts import all six libraries before any computation.
VERIFIED
Windows 11 x64, CPython 3.12.12, pythonnet 3.0.5, the six pins:
- 18/18 tests, three runs in a row, Release; 18/18 in Debug; 0 warnings for both projects at the
default warning level.
- The Local example runs to completion in 5.6 s (exit 0) and writes a correct 400 x 300 figure.
- Fixture policies: with the variable unset the tests report Inconclusive; unset with CI=true,
every test fails with the message; a missing interpreter and a numpy-only environment fail
with the cause named.
- The leak check fails a test that parks a NumPy view in builtins ("LiveExports 0 -> 1").
- Discovery: .venv in the current directory, PATH, no Python, the Microsoft Store alias, a
named interpreter that does not exist.
Linux (WSL Ubuntu 24.04), GitHub's own Python 3.12.14 build (actions/python-versions) with the
libraries installed by pip as the workflow does (torch 2.13.0+cpu, 0 nvidia packages):
- The workflow's build and test commands with CI=true: 18/18, the job summary written, the
.trx and the PNG under TestResults; also 18/18 with another python3 first on PATH.
- The Local example by its README (python3 -m venv .venv, pip install, dotnet run): exit 0,
3.7 s for a second run; every discovery branch, including "started from another directory".
Copied out of the repository, both projects restore NumSharp.Interop.pythonnet 0.70.0,
NumSharp 0.70.0 and pythonnet 3.0.5 from nuget.org: dotnet run exits 0 and dotnet test is 18/18.
The pins resolve as binary wheels for Python 3.11, 3.12 and 3.13 on win_amd64,
manylinux_2_28 x86_64 and macOS arm64 (uv pip compile); without the CPU index the Linux
resolution adds 19 nvidia / cuda / triton packages.
NOT VERIFIED
The workflow has not run on GitHub: this commit is local. The macOS leg in particular was not run
anywhere. Its interpreter discovery is the logic the interop suite already uses on macOS runners;
the two-libomp fact above is the known risk.
…nstructor (arrays, matrices, "1 2; 3 4" literals, sequences), T / H / I / A / A1, the row / column / scalar rule of indexing, * as the matrix product and ** as the matrix power, the two-axis reductions and every member ndarray hands down, held to NumPy 2.4.2 by a 26,027-case zero-excuse oracle tier (values, result class, view / copy / owner, error type and text); np.asmatrix and np.bmat return NDMatrix; index-array and boolean-mask assignment take NumPy's value rules library-wide
WHY
NumSharp had no numpy.matrix: np.asmatrix returned a plain 2-D NDArray and np.bmat a plain array, so ported code
that relies on matrix semantics (m * m is the matrix product, m[0] keeps two axes, m.sum(0) is a row, m.I, m.H,
np.matrix("1 2; 3 4")) either did not compile or silently computed something else. The coverage inventory listed
the 83 numpy.matrix rows as missing.
NAMING (how numpy.matrix points to NDMatrix)
- The CLR type is NumSharp.NDMatrix (the house ND* prefix, as numpy.ma.MaskedArray is NDMaskedArray):
`public partial class NDMatrix : NDArray, IEnumerable` in src/NumSharp.Core/Matrixlib/.
- The NumPy names reach it: np.matrix(data, dtype=None, copy=True) (Matrixlib/np.matrix.cs, four overloads:
NDArray / string / Array / object), np.asmatrix(...) and np.bmat(...) return NDMatrix.
- coverage/object_surfaces.py maps the object surface: ObjectSurface("matrix", "Matrix objects",
"NumSharp.NDMatrix"), so every numpy.matrix.<member> row of the inventory is credited by the NDMatrix member of
the same name: 80 of 83 available (ctypes, dump and dumps are missing, as on ndarray).
DESIGN
- Matrix semantics follow the STATIC type. NumPy finds the subclass at run time; here the compiler binds the
members and operators declared on NDMatrix. The same object through an NDArray reference is a plain 2-D array
(* is elementwise there, an integer index drops an axis). Reason: every kernel and helper of the library is
written against NDArray, and run-time dispatch would change that code's meaning for a caller who passed a matrix
in. Consequence, documented as a divergence: the np.* functions return NDArray where NumPy returns a matrix
(np.sum(m, 0), np.transpose(m), np.linalg.inv(m)); use the member (m.sum(0), m.T, m.I) or np.asmatrix(result),
an O(1) view. NDMatrix.Clone() HIDES NDArray.Clone() for the same reason (as an override it made np.positive(m)
return a matrix while np.copy(m) did not).
- A 0-d NDMatrix is a scalar, not a matrix. The 15 C# scalar types convert implicitly to a 0-d NDMatrix, so
m * 2, m + 1.5 and m[mask] = 0 compile and bind the (NDMatrix, NDMatrix) operators; * with a 0-d operand scales.
- Result forms (NDMatrix.cs): ViewOf (a.view(matrix), never an owner), From (a result NumPy allocates as a
matrix: the matrix takes the result's storage over and owns when it did), FromWrapped (a ufunc result that
__array_wrap__ returns: a view of the hidden result, never an owner), FromIndexing (matrix.__getitem__'s rule;
an index-array result is a view of the gathered array, NumPy's wrap_out_array; a 0-d integer array index gives
an owning K-order copy, HAS_SCALAR_ARRAY). All apply the finalize rule Shape.AsMatrix() (View/Shape.Matrix.cs):
0-d -> (1,1), 1-d -> (1,N), more than two axes longer than 1 -> ValueError "shape too large to be a matrix.".
WHAT IS IN THE CLASS (69 member names, 18 operator symbols, 9 indexers, 15 scalar conversions)
- Construction (NDMatrix.Construction.cs): the four branches of matrix.__new__ in NumPy's order. A literal is
_convert_from_string plus a port of ast.literal_eval for one token (the four number bases, "_", floats,
imaginary numbers, signs, parentheses, True / False; SyntaxError and "malformed node" errors; new house
exception Exceptions/SyntaxError.cs). A sequence goes through PolySequence (NumPy's array coercion); with a
dtype each element is stored by PyArray_Pack (ReductionInitial.TryPack): a Python int out of range raises
OverflowError, a complex number into a real dtype raises TypeError.
- T, H, I, A, A1 (and getT / getH / getI / getA / getA1), mT, real, imag, shape.
- Indexing (NDMatrix.Indexing.cs): a 1-d result is a COLUMN when len(index) > 1 and isscalar(index[1]), else a
row; other ranks are returned as they are (m[newaxis] is a 3-d matrix object, as in NumPy). m[int, int] is a
0-d NDArray view. Index errors carry NumPy's texts.
- Assignment: ndarray's assignment gets the view of a subclass through its own __getitem__ for an index of
integers and slices, so the value is broadcast into the row or COLUMN: m[:, 0] = [7, 8, 9] raises "could not
broadcast input array from shape (3,) into shape (3,1)". One element stores through the dtype's setitem.
- * and ** (NDMatrix.Operators.cs): m * x is np.dot(m, asmatrix(x)) (a 1-d x becomes a ROW), x * m is
np.dot(x, m); __mul__ / __rmul__ / __imul__ / __pow__ / __ipow__ / __rpow__. Every binary operator has five
forms (matrix/matrix, matrix/array, array/matrix, matrix/object, object/matrix). The other operators are
ndarray's ufuncs, wrapped back into a matrix; reflected comparisons evaluate with the matrix as the first ufunc
operand (the order of NumPy's broadcast error text).
- Scaling is np.dot's scalar route (cblas_matrixproduct): the dtype is promote_types of both operands (not the
weak-scalar ufunc rule); for float32 / float64 / complex128 the result is a new C-contiguous owner and each
element is 0 + scalar * x (?axpy into zeros), so a product of -0.0 is +0.0, except a ONE-element result (the
direct product); every other dtype is the multiply ufunc, wrapped. A plain 0-d NDArray on the LEFT stands for a
NumPy scalar and is the elementwise ufunc. Three routes with one result: (1) a C-contiguous float operand of the
loop dtype runs ONE direct pass over raw memory (ScaleContiguous: the polynomial package's PolyAffineDouble /
PolyAffineSingle, multiply then add, never an FMA; GC.KeepAlive on the operands after the kernel); (2) another
operand of the loop dtype runs one fused np.evaluate pass from the measured crossover on (4,096 elements for a
float view, 512 for complex128); (3) the rest runs the multiply ufunc, then adds the zero in place with the same
raw kernel (a complex buffer as twice as many float64 values).
- Reductions (NDMatrix.Reductions.cs): sum / prod / mean / var / std / any / all / max / min keep two axes
(keepdims then _collapse), argmax / argmin / ptp use _align; (int axis, ...) returns NDMatrix, (int? axis = null,
...) returns NDArray (the 0-d scalar without an axis). A matrix reduction is always NumPy's ARRAY form (a
float16 mean rounds twice; std(dtype=int) raises the sqrt cast error also without an axis). With out= the loop
is promote_types(out.dtype, operand dtype) without the small-integer widening (reducelike_promote_and_resolve),
stored with an unsafe cast; var / std hand out to their LAST sum only.
- Methods ndarray hands down, each returning NumPy's class and ownership (NDMatrix.Methods.cs): reshape, ravel,
flatten, squeeze, transpose, swapaxes, diagonal, trace, take, repeat, compress, choose, clip, round (PyArray_
Round's class rules: a float matrix rounded to decimals != 0 is a plain array in NumPy), conj / conjugate,
astype, copy, view, cumsum, cumprod, dot, argsort, argpartition, byteswap, getfield, to_device; repr / str
("matrix([[1, 2], ...])" through NDArray.ReprClassName) and iteration over rows.
LIBRARY FIXES THIS NEEDED (they change NDArray)
- BREAKING: np.asmatrix and np.bmat return NDMatrix (they returned NDArray). `var m = np.asmatrix(a); m * m` is
now the matrix product and m[0] keeps two axes; type the variable NDArray or use .A for the array semantics.
np.asmatrix also returns a NEW view where it returned a 2-D operand itself (the array-object oracle's gap VF15,
retired: view_flags.jsonl regenerated, 7,294 gate + 610 carved cases, 46 gap ids). np.r_'s "r" / "c" directive
result object is an NDMatrix (typed NDArray by the indexer).
- BREAKING: NumPy's MapIter value rule for every index-array assignment (new NDArray.PrepareAssignedValue,
Selection/NDArray.Indexing.AssignedValue.cs; three call sites in the setter and the mask setter). A value with
more axes than the selection loses its leading axes when the trailing ones hold every element (a (1, N) matrix
row fits an (N,) selection; it threw), and a value that does not fit raises ValueError "shape mismatch: value
array of shape (1,4) could not be broadcast to indexing result of shape (2,)" BEFORE anything is written (it was
IncorrectShapeException with another text).
- BREAKING: a boolean mask of the array's own shape follows array_assign_boolean_subscript
(Backends/Default/Indexing/Default.BooleanMask.cs): a value with two or more axes is TypeError "NumPy boolean
array indexing assignment requires a 0 or 1-dimensional input, input has 2 dimensions"; a 1-d value whose length
is neither 1 nor the true count is ValueError "NumPy boolean array indexing assignment cannot assign 3 input
values to the 6 output values where the mask is true" (also for a true count of 0). A prefix mask takes the
MapIter rule.
- np.dot(1-D, 2-D) and np.matmul with a 1-D operand return an OWNER (the inserted axis is dropped in place,
DefaultEngine.DropInsertedAxis; they returned a squeezed view). np.dot(N-D, 1-D) reports a mismatch in np.dot's
words ("shapes (3,4) and (5,) not aligned: 4 (dim 1) != 5 (dim 0)").
- DefaultEngine.VarStdNumPy gains outDtype / sumOnly for the matrix out= route (no behavior change for np.var).
- NDArray.Equals(object) no longer leaks its comparison array (found by the leak gate through NDMatrix.Equals).
ORACLE (test/oracle/gen_matrix_oracle.py -> corpus/matrix.jsonl; needs numpy==2.4.2; deterministic)
- 26,027 gate cases over 87 op keys ("matrix", "asmatrix", "matrix.<member>", Python's dunder names for the
operators; params.sym / params.via say which C# form the replay calls). 5,339 of them are NumPy errors, held to
type and text. Sections: construction (arrays of every rank, matrices, 62 literals, nested sequences, dtype
conversions), properties, getitem / setitem (integers, slices, new axes, ellipses, index arrays, masks, 0-d
arrays; values of every shape), * and the elementwise operators (matrix / array / sequence / Python scalar /
NumPy scalar on either side), **, the reductions (axis, dtype=, out= of seven dtypes, ddof), the methods, text
and iteration; 14 source layouts (C, F, transposed, stepped, reversed, a C view at an offset, read-only,
broadcast, a row, a column, 1 x 1, two empty shapes, square) and the dtypes.
- Three harness additions: an operand descriptor with "matrix": true rebuilds as an NDMatrix over the same storage;
expected.cls (matrix / ndarray / scalar: 18,221 / 349 / 1,866 cases) holds NumSharp to the class NumPy returned;
the memory facet of the array-object oracle (five flags, byte strides, shares / is each operand) is recorded for
every array result, so a matrix that owns where NumPy's is a view of the hidden ufunc result fails.
- Zero excuses (valueclass "matrix": MisalignedRegistry returns null). MinCases floor 24,700.
- Three replays, one per float scaling route: Matrix (production; asserts the direct pass ran),
Matrix_FusedScaling (NDMatrix.DisableDirectScale + FusedScaleMinElementsTestOverride = 2),
Matrix_TwoPassScaling (both single-pass routes off).
- Files: OpRegistry.Matrix.cs (the op-name -> NDMatrix call map), FuzzCorpusTests.Matrix.cs, FuzzCorpus.cs,
FuzzCorpusTests.Kinds.cs (class comparison), OpRegistry.cs / .Kinds.cs (routing), MisalignedRegistry.cs.
THE CARVE (corpus/gaps/matrix_gaps.jsonl, 172 cells, 11 ids; exact in both directions: every carved case
diverges, no kept case does - checked against an uncarved replay's failure dump)
The first replay had 2,441 divergences; fixes in NDMatrix and in the array library closed 2,269. Every one of the
172 left is an NDArray member or np.* function BELOW the class:
MX01 56 a[:, index_array]: NumPy's transposed MapIter view (F-contiguous); NumSharp a C-ordered array
MX02 36 np.cumsum / np.cumprod result layout for an F, transposed or broadcast-bool operand
MX03 15 ndarray.imag of a real F-contiguous array is C-ordered
MX04 5 ndarray.round of a broadcast operand: result layout
MX05 2 np.conjugate of a transposed strided / reversed complex128 view writes C order (reached through m.H)
MX06 16 reshape with -1 on a broadcast view throws NotSupportedException
MX07 8 a two-entry fancy assignment into a C-contiguous view at an offset throws IndexOutOfRangeException
MX08 15 np.prod in a cast loop: flat dtype=bool throws, flat complex128 / float16 loops return NaN or garbage
MX09 4 complex128 np.prod over a reversed view: the sign of a zero part
MX10 3 np.dot of an N-D operand with an M-D one computes in float64: complex128 loses its imaginary parts
MX11 12 np.repeat of a C-contiguous view at an offset reads from the parent's start
Recorded, not fixed. Seven have repros through public calls in test/NumSharp.Tests/OpenBugs.Matrix.cs (all
failing, as [OpenBugs]); MX01, MX06, MX09 and MX11 are pinned by older files. Matrix_KnownGaps ([OpenBugs]) reports
172 / 172 still open.
TESTS
- test/NumSharp.Tests/Matrixlib/NDMatrix.Tests.cs (42): NumPy's test_defmatrix.py ported, plus the design pins
(static typing, the scalar stand-in, ownership of each result form) and Scaling_EveryRoute_GivesTheUfuncBits (the
three scaling routes against np.add(np.multiply(x, s, dtype: loop), 0) at 70 x 70, bit for bit, with the route
counters). Two planted bugs (the direct pass adding -0.0; the complex add-zero pass covering half the buffer)
are caught.
- Eight older tests pinned the old IncorrectShapeException / texts of setter value mismatches and now hold NumPy's
ValueError / TypeError texts (BooleanMasking.MatrixTests, BooleanMasking.UsingTests, CombinedIndexing.MatrixTests,
Indexing.KernelRoute.Tests, np.asmatrix.Tests), each checked against NumPy 2.4.2.
- A route-equality probe (not committed): 30,378 comparisons of the three scaling routes with the ufunc
reference over float32 / float64 / complex128, nine layouts (a C-contiguous np.unstack child at an offset
included), both operand orders, scalars of every dtype that keeps the loop (0-d views included), special
values, 2 to 300,300 elements: 0 value / shape / dtype differences; 12 differ only in the payload of a NaN that
meets a NaN scalar (NumPy's comes from the BLAS ?axpy kernel, which no route models).
LEAK AND COVERAGE GATES
- LeakSurfaceCoverageTests: NDMatrix joins ObjectOwners and OperatorOwners (MatrixOperatorKeys maps the C#
operators to the corpus's dunder keys); a member is credited by its corpus key matrix.<name>.
LeakCatalogue.Matrix.cs measures the members no corpus row reaches (Clone, Equals, GetHashCode, GetEnumerator,
the indexers, argpartition, choose, getfield, to_device, the conversions, !).
UndisposedIntermediateTests.Properties.cs reads every property over three matrix targets.
- coverage/oracle_map.json: the prefix rule "matrix" -> numpy.matrix.{rest} and 37 explicit keys;
coverage/test_object_surfaces.py: test_matrix_members_map_to_ndmatrix.
PERFORMANCE (NPY/NS, both sides on two pinned P-cores, best of 9 rounds, every C# cell warmed in four rounds;
97 cells: 33 operations x 4 / 64 / 512 square, plus three literals)
- geomean 2.26. Construction 2.6-18x, m[i] / m[:, j] 5x, m + m 1.8-5x, m * 2.0 2.6x / 1.9x / 25x, m.sum()
2.9-6.2x, m * m at 4x4 6.6x, m.I at 4x4 8.6x, literals 8-13x.
- 31 cells are under 1.5x, each a cost of the array library below the class: two index arrays (m[rows, cols]
0.33-0.69x, its assignment 0.18-0.27x: the fancy delegate route); m > scalar from 4,096 elements (0.25-0.52x:
the scalar comparison is a scalar loop on journey4; vectorized on branch strings, plan X24 / X25); products at
64x64 without the OpenBLAS backend (m * m 0.58x, m ** 3 0.50x, m.I 1.12x); axis reductions (m.sum(0) 1.1-1.4x,
m.mean(0) and m.argmax(1) at 512 1.0-1.1x, m.cumsum(0) 1.2-1.3x); m[mask] (0.84x at 512); members at the
NDArray allocation floor (m.copy() 1.1-1.3x, m[i, j] and m.A 1.4-1.5x).
- A C# literal operand costs 0.2-1.2 microseconds more than a held 0-d operand (m * 2.0 at 256 elements: 1.5
against 0.58): the conversion's 0-d temporary is left to the finalizer, as for a * 2.0 on a plain array (plan
X26, open).
- Measuring trap recorded in the code and CLAUDE.md: a harness that warmed a cell once read 4.3 microseconds for
a fused np.evaluate call that takes 1.4 at tier 1, which first put the fused threshold four times too high.
DIVERGENCES FROM NUMPY (documented in the class remarks and .claude/CLAUDE.md)
- The static typing above (np.* functions return NDArray).
- A C# literal keeps its C# type (matrix(int8) * 2 is int32; NumPy's Python int gives int64).
- NumPy raises for m * np.array(2); NumSharp cannot tell that 0-d array from a NumPy scalar and scales.
- The typed setters take a matrix or a number (__setitem__(index, object) takes any value); no tuple axes.
- The object address in ast's error texts is zero; complex128 scaling by a complex scalar is the ufunc product,
which can differ from OpenBLAS zaxpy in the last bit.
DOCS
.claude/CLAUDE.md (a new "np.matrix ... NDMatrix" section, the creation list, the Core Source Files row, the
OpenBugs files entry, "The matrix tier" oracle bullet), test/NumSharp.Tests.Oracle/Fuzz/README.md ("The matrix
tier": schema additions, the three replays, the gap table, the rules for redrawing the carve).
VERIFIED (Release, after the last change)
- NumSharp.Tests (TestCategory!=OpenBugs&TestCategory!=HighMemory): 18,443 passed on net10.0, 18,442 on net8.0,
0 failed. One earlier full net8.0 run of the same sources reported a single failure that did not recur in six
more runs and was not identified (that run's output was not saved); the matrix and indexing classes passed 15
loops of 953 tests.
- NumSharp.Tests.Oracle: 280 passed, 1 skipped (RandomApiSoak) on both frameworks; Matrix_KnownGaps 172 / 172 open.
- NumSharp.Tests.Interop: 736 passed on both frameworks.
- python -m pytest coverage: 33 passed; coverage/generate_coverage.py --check: current.
…rojects, and two OpenBLAS example projects join them - NumSharpOpenBlasDefaultExample (one package reference, no configuration) and NumSharpOpenBlasAdvancedExample (a pinned OpenBLAS build, one thread, pinned kernels, results that reproduce bit for bit and equal NumPy 2.4.4's)
WHAT
1. The project files of the two pythonnet examples are simplified (owner's request).
examples/NumSharpPythonnetLocalExample and examples/NumSharpPythonnetGhActionsCicd had two
custom MSBuild properties (NumSharpInteropVersion, NumSharpInteropProject), a conditional
ProjectReference / PackageReference pair, and SignAssembly / IsPackable lines. Each file is now
exactly what `dotnet new` and `dotnet add package NumSharp.Interop.pythonnet` write: the
template's properties and one plain <PackageReference ... Version="0.70.0" />. The reference
line was written by `dotnet add package` itself.
Consequences:
- Both projects build against the PUBLISHED package, in the repository too. A build takes 2
to 5 seconds instead of 40 to 80, and prints 0 warnings, so the workflow's build step lost
its -p:WarningLevel=0.
- .github/workflows/example-pythonnet.yml no longer starts on changes under
src/NumSharp.Interop.pythonnet: a source change cannot change the result of a project that
restores the released package. It starts on changes to the example or to itself.
- The READMEs say what to copy in terms of the two commands, not of a reference switch.
- Inside the repository Directory.Build.props still strong-names the two assemblies (no
opt-out line any more); that produces no warning.
2. examples/NumSharpOpenBlasDefaultExample (console app, net10.0, new).
The project file is `dotnet new console` + `dotnet add package NumSharp.Interop.OpenBLAS`.
- Program.cs: OpenBlasEngine.Enable() as the first statement, then a status report
(OpenBlasEngine.Info, IsBundledLibrary, LapackAvailable).
- Products.cs: np.dot / np.matmul / np.inner / np.outer / np.vdot, a stack of matrices,
float32 and complex128, np.einsum / np.tensordot / np.linalg.multi_dot / matrix_power, and
an integer product that stays on NumSharp's own kernels. Silent; the outcome of each line
is in the comment beside it.
- LinearAlgebra.cs: solve, inv, det, slogdet, cholesky, qr, eigh, svd, lstsq, pinv,
matrix_rank, cond, norm, eig. It returns the largest residual of the identities it shows
(4.3e-15 on Windows and on Linux), which Program.cs prints.
- Speed.cs: one 1024 x 1024 float64 product by OpenBLAS and, after OpenBlasEngine.Disable(),
by NumSharp's own kernels, with the largest difference between the two results.
3. examples/NumSharpOpenBlasAdvancedExample (console app, net10.0, new).
- The project file pins the OpenBLAS build as an attribute of the same regular reference:
<PackageReference Include="NumSharp.Interop.OpenBLAS" Version="0.70.0"
OpenBlasVersion="0.3.31.188.0" />. That is the scipy-openblas release that NumPy 2.4.4
pins. A comment lists the other settings (OpenBlasSha256, OpenBlasPath,
OpenBlasDistribution, OpenBlasFeed, OpenBlasDelivery); the README explains each.
- Program.cs names no type of the package and sets NUMSHARP_OPENBLAS_BUNDLE_AUTOINSTALL=0
first, so that the first load of the library is the call in Backend.Start.
- Backend.cs: OpenBlasEngine.Enable(threads: 1, coreType: OpenBlasEngine.CoreType) with a
fallback for a processor that cannot run the Haswell kernels and for a library that was
loaded earlier; a report that reads the build's openblas.source.json marker.
- Reproducibility.cs: the SHA-256 of one float32 (512,784) @ (784,512) product with 1 thread
(twice), 2 threads, all threads, and NumSharp's own kernels.
4. Both OpenBLAS projects are in SciSharp.NumSharp.sln under "examples". Pointers were added to
src/NumSharp.Interop.OpenBLAS/README.md and to the "See also" list of
docs/website-src/docs/interop/openblas.md. .claude/CLAUDE.md records the regular-csproj rule,
the two new examples and the measured facts below.
FINDINGS ABOUT THE RELEASED PACKAGE (measured against NumSharp.Interop.OpenBLAS 0.70.0; reported,
not fixed here - the examples and their READMEs work around them and say so)
A. A package reference alone does not install the backend.
The backend is installed by a [ModuleInitializer], and the runtime runs it when it first
prepares a method that names a type of the package. NumSharp.Core never loads the assembly.
A program that references the package and calls only np.* functions:
- has TensorEngine.Blas == null before and after np.dot (the assembly is not even loaded),
- computes dot / matmul / solve / inv / det / slogdet / matrix_power(-1) on managed code,
- gets OpenBlasMissingBackendException from cholesky, qr, svd, svdvals, eig, eigvals, eigh,
eigvalsh, lstsq, pinv, matrix_rank, cond and norm with ord 2 / 'nuc'. The text of that
exception tells the user to reference the package, which the project already does.
The quick start of the docs works because its method names OpenBlasEngine.Info: the
initializer runs at the ENTRY of the first method that names the package (probed: the backend
is installed before that method's first statement; same result with tiered compilation off,
PGO off, QuickJit off, ReadyToRun off and JitStress=1). tools/verify_package_consumer.sh's
consumer names the type as well, so no gate sees the gap.
The default example therefore starts with OpenBlasEngine.Enable(), and its README has a
section "Name the package before you compute".
B. On glibc Linux the build-time pin cannot stage scipy-openblas 0.3.33 or later.
The fetch task selects the wheel with name.Contains("-" + tag + "."), where tag is the first
component of the platform tag of the manifest's wheel (manylinux2014_x86_64). From 0.3.33.0.0
the glibc wheels are named manylinux_2_27_x86_64.manylinux_2_28_x86_64, so the build fails:
NumSharp.Interop.OpenBLAS: expected exactly one scipy-openblas64 0.3.33.112.0 wheel for
platform tag 'manylinux2014_x86_64', found 0.
(measured on Ubuntu 24.04 for 0.3.33.112.0, the OpenBLAS of NumPy 2.5.0 / 2.5.1, and for
0.3.34.0.0). The same versions build, load and compute on Windows (0.3.33.112.0 and
0.3.34.237.0 were run: BLAS and LAPACK). 0.3.29.0.0 fails on glibc Linux for the mirror-image
reason: its compound tag lists manylinux_2_17 first. Versions 0.3.29.265.1 to 0.3.31.188.0
have a file with the expected tag for all eight RIDs, which is why the example pins
0.3.31.188.0.
C. Smaller facts that the examples and READMEs encode:
- Kernel request (coreType): applied when the call is the first load; accepted when the
loaded kernels are the same; InvalidOperationException when the library is loaded with
other kernels; PlatformNotSupportedException when the processor cannot run them ("SkylakeX"
on this machine); an UNKNOWN name raises nothing - OpenBLAS ignores it (read CoreName).
- OPENBLAS_CORETYPE and OPENBLAS_NUM_THREADS in the environment of the process work with no
code (Info showed "threads 3 ... Sandybridge").
- A build output holds the libraries of all eight RIDs: 201 MB. `dotnet publish -r <rid>
--self-contained false` gives 23 MB on Windows and 29 MB on Linux; the pinned library and
its marker are then beside the application, and the program runs from there.
- The released NumSharp package ships the NDW012 analyzer: example code that creates arrays
outside an NDScope gets a warning for each one. The examples open one scope per method.
- The download of a pin is 7 to 10 MB (the wheel), not the 20 MB of the extracted library.
PARITY, MEASURED (one Core i9-13900K; Windows 11 and WSL Ubuntu 24.04)
- The bundled library is byte-identical with the file inside the NumPy 2.4.2 wheel
(sha256 74a408729250596b0973e69fdd954eea07a70ff527a1dbaccf9ae21247b80377), and the pinned
0.3.31.188.0 library with the file inside the NumPy 2.4.4 wheel
(63c857e738469261263c764a36be9436ebdeaa272e340a828f42047a97131080). On Linux NumPy's copy has
other bytes (auditwheel rewrites the ELF header) and carries the hash of the staged file in
its name (libscipy_openblas64_-32a4b2a6.so).
- The float32 product of the advanced example (np.random.seed(7) inputs, which NumSharp's
generator reproduces) has the same SHA-256 in NumSharp and in NumPy:
threads pinned 0.3.31.188.0 = NumPy 2.4.4 bundled 0.3.31.22.0 = NumPy 2.4.2
1 d02ca290eb70d865 d02ca290eb70d865
2 67cdb5ec364fa7df 67cdb5ec364fa7df
4 e3433600e14af1cd
24 d16b7257b4864151 cc2acd3eb7be1ab1
NumSharp's own kernels give b5560c0b1370343c. The 1- and 2-thread hashes are the same on
Windows and on Linux (NumPy 2.4.4 on Linux was compared at 1 and 2 threads). Every hash
repeats from run to run.
- NumPy's pins, read from requirements/ci_requirements.txt at each tag: 2.2.6 / 2.3.0 ->
0.3.29.0.0; 2.3.2 to 2.3.4 -> 0.3.30.0.1; 2.3.5 / 2.4.0 / 2.4.1 -> 0.3.30.0.7; 2.4.2 ->
0.3.31.22.0; 2.4.3 -> 0.3.31.126.1; 2.4.4 -> 0.3.31.188.0; 2.5.0 / 2.5.1 -> 0.3.33.112.0.
VERIFIED
Windows 11 x64:
- All four projects build from scratch with 0 warnings and 0 errors (Debug and Release).
- pythonnet: 18/18 tests (Release with CI=true, and Debug); 18 Inconclusive without the
variable; the local example exits 0.
- OpenBLAS default: exit 0; 1024 x 1024 float64 matmul 8.3 ms against 89.8 to 190 ms for
NumSharp's own kernels across runs on a loaded machine; exits 1 with the package's reason
when no library can be loaded (NUMSHARP_OPENBLAS_USE_BUNDLED=0); works with the automatic
start turned off in the environment.
- OpenBLAS advanced: the build downloads, caches and stages the pin; "bundled False", "pin
scipy-openblas64 0.3.31.188.0, required: True", "kernels Haswell (pinned)"; exits 1 with
OpenBlasRequiredOverrideException's text when the pinned library is removed from the
output; the published (-r win-x64) copy runs.
Linux (WSL Ubuntu 24.04 x64), from copies of the four folders only (no repository):
- pythonnet: the workflow's build and test commands with GitHub's own Python 3.12.14 build:
0 warnings, 18/18, job summary and artifacts written; the local example exits 0.
- OpenBLAS default and advanced: build with 0 warnings and run to completion; the advanced
build stages the manylinux2014 wheel with its vendored libgfortran / libquadmath; the
published (-r linux-x64) copy runs from another directory.
NOT VERIFIED
- macOS and ARM: none of the four projects was run there. On ARM the advanced example takes its
PlatformNotSupportedException branch (no Haswell kernels) and pins the thread count only.
- The workflow has still not run on GitHub: journey4 is not pushed.
…he scalar-broadcast comparison kernels of branch strings (d0a6f4c) brought to journey4 unchanged: m > 0.75 on a 512x512 float64 matrix 0.25x -> 1.0x NumPy and on 64x64 0.52x -> 1.2x, a float64 array against a 0-d operand 6.8x faster at 100K elements; with the loop-bound fix the same files carry - the comparison and predicate SIMD loops no longer read one element past their inputs and write one bool past their result Why === The numpy.matrix benchmark (201e9c7) left the row "m > scalar" under NumPy: m > 0.75 at 4x4 / 64x64 / 512x512 was 1.39x / 0.52x / 0.25x. The class adds almost nothing to that call. The time was in the array library: on journey4 the comparison of an array with a one-element operand (ExecutionPath SimdScalarRight / SimdScalarLeft) was a scalar loop, 0.7 ns an element for float64 (0.36 for the integers, 1.4 for bool), where the array-against-array kernel is vectorized. Branch strings had already replaced those loops in d0a6f4c (plan api-parameter-parity X24 + X25). What was brought, and what was not ================================== Three files are now BYTE-IDENTICAL on journey4 and strings (git blob ids in brackets), so the later merge of strings does not conflict in them: - src/NumSharp.Core/Backends/Kernels/Direct/DirectILKernelGenerator.Comparison.cs [dfc0ef0] - src/NumSharp.Core/Backends/Kernels/Direct/DirectILKernelGenerator.Unary.Predicate.cs [66db492] - test/NumSharp.Tests/Backends/Kernels/ComparisonBroadcastKernelTests.cs [647ae48] (new on journey4; the same file strings adds) Both kernel files were changed on strings by d0a6f4c only, and not at all on journey4 since the two branches split (d70cbb0), so "the file as strings has it" is exactly "journey4's file plus the kernel half of d0a6f4c". NOT brought: the rest of d0a6f4c. That is the NEP 50 weak-scalar pack of the comparisons (X24), the weak-int range check of the binary ufuncs (X2 / X21), the numpy.ma strong-scalar rule, the NDExpr change, and the 15,050-case weak_scalar oracle tier. They are a breaking change of results, they sit on top of the string-dtype commits of that branch, and journey4 has four oracle tier families that strings does not have yet (relayout, surface, ma surface, matrix). They arrive with the merge. What the kernels do =================== - array op scalar / scalar op array compare a whole vector per step with the scalar broadcast into a vector once (EmitComparisonBroadcastSimdLoop, CanUseComparisonBroadcastSimd). The condition is that the array already has the comparison dtype: the vector body loads the elements as they are. - The 4x-unrolled bodies store their four masks PACKED into result bytes on AVX2 hosts (EmitMasksToBools / EmitPackedMasksToBools, NumPy's npyv_pack_b8_b16 / _b32 / _b64). 1-byte lanes store mask & 1 as one vector. The array-against-array kernel (SimdFull) uses the same stores. - The single-mask store reads the lane bits as 64 bits (a Vector512 mask of 1-byte lanes has 64; the old 32-bit local could not hold them). The loop bound (silent memory corruption on journey4 until this commit) ====================================================================== The comparison SimdFull kernel and the contiguous predicate kernel (isnan, isinf, isfinite, isposinf, isneginf, signbit) ran their 4x-unrolled and one-vector loops while i <= n - step + 1. So they ran one body too many when n % step == step - 1: one element read past both inputs and one bool written past the result. A fresh result's extra byte landed past its allocation (the pool's buckets have exact sizes); an out= view's extra byte landed in the next slot of its parent buffer. The loops now run while i <= n - step. Proof on journey4: with the old two kernel files and the new test file, three of its six tests fail: ComparisonKernels_NeverWritePastTheResult "Boolean Equal SimdFull n=31 wrote result slot 31", found 0x01 PredicateKernel_NeverWritesPastTheResult "IsNan Single n=7 wrote result slot 7", found 0x00 Comparisons_KeepTheParentBufferPastAnOutView "Boolean n=31 array: slot 31 past the out view changed" With the new kernels all six pass. A value-only gate (the oracle, the unit tests) cannot see a write past the result, which is why this stayed hidden. Measured (NPY/NS, higher is better; two pinned P-cores, Release) ================================================================ The matrix benchmark's row, re-run with the harness of 201e9c7 (97 cells, two rounds a side, the minimum; "before" is that commit's last run): before (us) after (us) cell NumPy NumSharp NumPy NumSharp NPY/NS m > 0.75 @4 1.68 1.21 1.89 0.94 1.39x -> 2.02x m > 0.75 @64 2.34 4.54 2.57 2.13 0.52x -> 1.20x m > 0.75 @512 46.7 185.4 46.3 45.9 0.25x -> 1.01x The whole benchmark: geomean 2.26x -> 2.35x, cells under 1.5x 31 -> 29. A second harness (87 cells, best of three interleaved rounds a side; "before" is journey4's head, measured in the same harness): float64, n x n NumPy us before us after us after a > held 0-d @4 0.83 0.44 0.43 1.95x a > held 0-d @64 1.37 3.39 0.96 1.43x a > held 0-d @512 41.5 187.8 40.1 1.03x m > held 0-d @64 2.49 3.70 1.16 2.14x greater(a, s, out) @64 1.25 3.60 1.09 1.14x a > b (two arrays) @512 94.9 118.7 100.1 0.95x a > s, s a 0-d of the array's dtype, 100K elements (us) dtype NumPy before after dtype NumPy before after float64 11.07 70.5 10.31 int16 5.10 35.6 5.39 float32 6.11 70.2 6.04 uint16 6.26 35.6 5.57 int64 34.2 36.2 18.3 int8 3.87 35.5 3.63 uint64 33.7 53.1 18.2 uint8 4.17 35.6 4.05 int32 7.81 35.6 9.03 bool 4.16 139.3 3.58 uint32 9.90 52.9 7.16 the same at 10M (ms): float64 6.05 / 8.08 / 4.46, float32 4.27 / 7.13 / 2.17, int32 4.49 / 4.23 / 2.22, int16 3.12 / 3.61 / 1.13, int8 2.63 / 3.58 / 0.71, bool 2.66 / 13.99 / 0.67. Over the 87 cells: geomean 0.52x -> 1.62x, minimum 0.03x -> 0.82x. At 1K every dtype is 1.9-2.4x, at 10M 1.4-4.0x. At 100K both libraries run the same packed loop at the speed of the cache (0.86-1.9x). Why two cells of the row stay under 1.5x ======================================== 1. 64x64: the C# literal. m > 0.75 converts 0.75 into a 0-d matrix that no one disposes. With a held operand the cell takes 1.16 us (2.14x NumPy); the literal adds 0.9 us. Measured over one second of calls: a dropped "NDArray s = 0.75" costs 1.13 us (0.32 when it is disposed; a dropped object with an empty finalizer costs 0.18), and (a > 0.75) at 4x4 takes 1.75 us against 0.47 with a held operand. This is plan item X26 (open, library-wide). It needs an operand that is not a finalizable object, which is a design decision and not part of this change. 2. 512x512: memory. The kernel already runs at the cost of READING its input. Called directly on aligned buffers it takes 0.36 us for 4,096 float64 elements and 18.5 us for 196,608; a hand-written loop that only loads and compares (it writes nothing) takes 0.37 and 17.7. A 512x512 float64 matrix is 2 MB, the size of this CPU's L2 cache (i9-13900K, P-core): there the read-only loop takes 30 us, the kernel 39 and NumPy 40-48. In the same hand-written loop (38.1 us) a software prefetch changes nothing (37.7 us) and a non-temporal result store gains 6 % (35.9 us), at the cost of every later read of the mask. So 1.5x NumPy is not reachable at that size. What still runs the scalar loop on journey4 (until strings merges) ================================================================= The vector loop needs the array to have the comparison dtype, and journey4 still promotes a 0-d operand strongly (X24 is not here). So a literal that promotes the comparison stays slow. At 100K elements: float32 > 50.5 (double) 72.4 us float32 > 50.5f 7.9 us uint8 > 50 (int) 39.3 us uint8 > (byte)50 3.9 us int16 > 50 (int) 36.7 us int16 > (short)50 4.7 us int32 > 50L (long) 39.1 us int32 > 50 7.3 us A float64 array takes the vector loop with an int, float or double literal (10.8 us), an int32 / int64 array with an int literal. Finding, not acted on: buffer alignment is a 20-35 % lottery ============================================================ A buffer whose address is 16 mod 32 makes every second 32-byte load cross a cache line. The kernel called directly on 100K elements: float32 4.8 us aligned and 6.4 us not, float64 9.3 and 12.5, int32 6.0 and 8.2. For 1- and 2-byte lanes the RESULT's alignment costs as much (int16 at 262,144 elements: 9.0 and 12.8 us). Pooled buffers are 16-byte aligned, so each array wins or loses by its address. Through the public operator, the same 100K float64 values take 10.1 us in a view that starts on a 32-byte boundary and 13.3 us in a view that starts 16 bytes later (float32: 5.8 and 7.3). It applies to every SIMD kernel of the library, so the fix belongs in the allocator (a 64-byte aligned allocation), not in this kernel. The 2 MB matrix of the row is not affected: the operating system returns a large block 64-byte aligned. Validation ========== All in a private worktree of 07aa1d1 with the four files of this commit, Release: - NumSharp.Tests (CI filter): net10.0 18,449 passed, 0 failed, 11 skipped; net8.0 18,448 passed, 0 failed, 11 skipped. - NumSharp.Tests.Oracle (the NumPy 2.4.2 gate): 280 passed, 0 failed, 1 skipped (RandomApiSoak, which needs its environment variable) on net10.0 and on net8.0. - ComparisonBroadcastKernelTests: 6 of 6. With the old kernels: 3 fail (above). - Other SIMD configurations, 231 comparison / predicate tests each, all pass: DOTNET_EnableAVX=0 (128-bit vectors and no BMI2: the mask-store branch an arm64 host takes), DOTNET_EnableAVX2=0, DOTNET_EnableHWIntrinsic=0 (no vectors). A probe confirmed that the switches change the kernels' vector width (256 / 128 / 0). The only vector method the new code resolves by itself, Vector{N}.BitwiseAnd, is the cross-platform one on every host. - NOT run: an arm64 host and a 512-bit host (this machine has neither). Docs ==== .claude/CLAUDE.md, the numpy.matrix paragraph: the row's new numbers, what was brought and what stays on strings, the two floors, the literal's measured cost, the alignment finding. Merge note for strings -> journey4 ================================== - The three files above are identical on both sides: no conflict. - d0a6f4c's other files merge as they would have before this commit. - .claude/CLAUDE.md: this commit edits only the numpy.matrix paragraph, which strings does not have. The paragraph's sentences about what "stays on strings" and about literals that promote become stale at the merge.
…pauses - a second run of the one-second probe of 19a0057 with pause and lock-contention counters; the NDScope numbers of that commit's paragraph are replaced by the range of both runs 19a0057 measured the 0-d temporary of a C# literal operand (plan X26, open) and wrote in .claude/CLAUDE.md that a scope around the call "does not make it cheap either", with the numbers of ONE run (2.27 us in an NDScope, 2.43 without, 1.45 with a held operand). A second run of the same probe read 1.86 / 2.30 / 1.38. The paragraph now gives both runs as ranges, and says what the scope does: it removes the collector's pauses, not the scalar. The second run also printed GC.GetTotalPauseDuration and Monitor.LockContentionCount per case (each case is about one second of calls, Release, two pinned P-cores): case mean ns gen0/gen1 GC pause lock contentions object with a finalizer, dropped 253 5 / 4 182 ms 0 NDArray.Scalar(0.75), disposed 338 90 / 0 109 ms 0 NDArray s = 0.75, dropped 1133 27 / 26 454 ms 8 NDMatrix s = 0.75, dropped 1120 27 / 26 435 ms 5 (a > held) 4x4 475 59 / 0 82 ms 0 (a > 0.75) 4x4 1682 34 / 33 394 ms 11 (m > held) 4x4 764 59 / 0 81 ms 0 (m > 0.75) 4x4 2025 37 / 36 401 ms 2 (a64 > held) 64x64 982 249 / 0 48 ms 0 (a64 > 0.75) 64x64 1871 132 / 131 282 ms 0 (m64 > held) 64x64 1381 180 / 0 53 ms 0 (m64 > 0.75) 64x64 2296 108 / 107 284 ms 1 (m64 > 0.75) in an NDScope 1863 131 / 0 69 ms 0 So the cost is not a lock and not the scalar pool: a dropped scalar makes every collection a generation-1 collection of about 17 ms (the finalizable array and its memory block's disposer are promoted with the storage object, the boxed slice, the closure and the delegate they reference), against about 1.4 ms a collection when the operand is held or disposed. The thread is paused for 0.28 to 0.45 s of each second. The scalar pool running dry (about 25,000 extra slots a second) is a symptom of the same lag. This also explains why a best-of-rounds harness reads the literal's cost 0.2 to 0.9 us low: it reports a round with few collections. No code change.
… element past the view - np.zeros(163840, np.uint8)[1::2].copy() ended the process with an access violation, because seven SIMD deinterleave loops loaded one element beyond a view that ends at its buffer's last element; the second load now starts one element early and shifts (same speed), and a fenced-memory test puts the source of all 15 x 15 casts against inaccessible pages THE DEFECT (in the library since 2026-06-20, commits 32353c3 / 4a77991 / f21876c) The strided cast kernels for one- and two-byte elements deinterleave a step of two (a[::2]) with two vector loads per group of N results: N elements at s[2i] and N elements at s[2i+N]. Together they cover s[2i .. 2i+2N-1]. The group needs s[2i], s[2i+2], ..., s[2i+2N-2]. The loads therefore read ONE element past the last element of the group. For the last group of a row that element is one past the view's last element. When the view's last element is its buffer's last element (a[1::2] of an even-length buffer, a[::2] of an odd-length one), it is outside the buffer. Values were right (the element is masked away), so no value test and no oracle case could see it. It became a crash when the buffer ended where mapped memory ends: var z = np.zeros(new Shape(40 * 4096), NPTypeCode.Byte); // 163,840 bytes var c = z["1::2"].copy(); // AccessViolationException, the process ends A zeroed array of 128 KB or more is its own VirtualAlloc region (OsVirtualMemory). 40 pages are two and a half allocation granules, so nothing can be mapped behind the buffer. NumPy returns 81,920 zeros. Seven loops had the pattern: DirectILKernelGenerator.Cast.SubwordCopy.cs SubwordCopyStrided1B, SubwordCopyStrided2B (same-type copies of bool / int8 / uint8 / int16 / uint16 / char / float16, and same-width casts such as uint8 -> int8) DirectILKernelGenerator.Cast.SubwordNarrow.cs SubwordNarrowInt2to1, SubwordNarrowBool2to1 (2-byte -> 1-byte / bool) DirectILKernelGenerator.Cast.SubwordWiden.cs SubwordWidenSigned1to2, SubwordWidenUnsigned1to2 (1-byte -> 2-byte) DirectILKernelGenerator.Cast.ToBool.cs StridedHalfToBool (float16 -> bool) The 4- and 8-byte kernels (Cast.WordCopy.cs) bound their loop for this already (i + 9 <= n, i + 5 <= n). THE FIX The second load starts one element EARLY, at s[2i+N-1]. Its lanes then hold the wanted elements in their HIGH half (the unwanted s[2i+N-1+2k] below, the wanted s[2i+N+2k] above), so a logical right shift by one element width replaces the mask of that load. The instruction count is the same, and the load ends exactly on s[2i+2N-2]. Speed, old loop against new loop in one process, interleaved, two pinned P-cores, (1000, 1000) -> (1000, 500): 1-byte 26.60 us -> 26.80 us (1.008) 2-byte 87.73 us -> 87.83 us (1.001) A loop bound of one element less would have sent the last 16 or 32 results of every row whose length is a multiple of the group through the scalar tail, which is the common case (rows of 512). HOW IT WAS FOUND The comparison engine of the next commits stages stepped operands through these kernels. Its tests place every operand against an inaccessible page; the first run ended in SubwordCopyStrided1B. TESTS test/NumSharp.Tests/Utilities/FencedMemory.cs (new, test utility) Native memory with an inaccessible page directly below its first byte and directly above its last one. The region is reserved whole and only its data pages are committed, so a read or a write one byte outside ends the process. Place(bytes, atEnd) puts a block against either page. Windows (VirtualAlloc); a test that needs it reports Inconclusive elsewhere. test/NumSharp.Tests/Backends/Kernels/StridedCastFenceTests.cs (new, 3 tests) - OneDimensionalViews_OfAFencedBuffer_CopyAndCastWithoutLeavingIt: for every dtype, a buffer against the low or the high page, the views [:], [::2], [1::2], [::3], [1::3], [2::3], [::-1], [::-2], [-2::-2], [::-3] at 18 lengths around each kernel's vector group; each view is copied and cast to all 15 dtypes. - TwoDimensionalViews_OfAFencedBuffer_CopyAndCastWithoutLeavingIt: stepped and reversed columns and rows and the transpose of a (3, n) buffer. - SteppedViewEndingAtTheLastElementOfAZeroedRegion_IsCopied: the np.zeros call above for the seven sub-word dtypes (copy, narrow, widen, same-width cast). Values are checked too: the copy against the base's elements, each cast against the same cast of a contiguous copy. A violation is not a failed assertion; the test host ends with the kernel's stack. Do the tests see each loop? Each of the seven fixes was reverted alone and the class run again: SubwordCopy 1-byte test host crashed in SubwordCopyStrided1B SubwordCopy 2-byte test host crashed in SubwordCopyStrided2B SubwordNarrow loop 1 test host crashed in SubwordNarrowInt2to1 SubwordNarrow loop 2 test host crashed in SubwordNarrowBool2to1 SubwordWiden loop 1 test host crashed in SubwordWidenSigned1to2 SubwordWiden loop 2 test host crashed in SubwordWidenUnsigned1to2 half -> bool test host crashed in StridedHalfToBool With all seven fixed the class passes, and no other cast kernel leaves its source on any of the 225 dtype pairs. VERIFICATION OF THE WHOLE TREE (with the two following commits; the three were developed and tested together) NumSharp.Tests net10.0 and net8.0, filter TestCategory!=OpenBugs&TestCategory!=HighMemory: 0 failures NumSharp.Tests.Oracle net10.0 and net8.0: 0 failures The whole unit suite and the whole oracle replay under NUMSHARP_DEBUG_GUARD_PAGES=1 (every pooled buffer ends at an inaccessible page): no crash. Only the tests that count pool takes and returns fail there, because the mode bypasses the pool. DOCUMENTATION .claude/CLAUDE.md: "A strided copy or cast never reads outside its view" (after the ILKernel coverage table) and the sentence at the end of the Cast row. BRANCH NOTE The four kernel files are byte-identical on journey4 and on strings before this commit, so the change merges without a conflict.
…rgument - the typed overload installed the Shape as given, so the Shape of a view, a transposed or a broadcast array put that array's strides over this array's buffer (strides (16,8,1) over 40 elements read up to element 75: other heap memory, or an access violation); it now passes newShape.AsClean() on, which is what the untyped overload always amounted to
THE DEFECT
NumPy's reshape takes a shape TUPLE. The untyped NDArray.reshape(Shape) (ReshapeCore, the port of
_reshape_with_copy_arg) reads the dimensions of its argument and nothing else.
The typed overload NDArray<T>.reshape(Shape) / reshape(ref Shape) did not go through ReshapeCore. It relabelled
its storage with the Shape as given: UnmanagedStorage.Reshape(ref Shape) -> Shape.Reshape(Shape), which builds the
result from the argument's dimensions AND strides. A Shape taken from another array carries that array's layout:
var view = np.arange(80).reshape(5, 2, 8)[":, :, :4"]; // dims (5,2,4), strides (16,8,1), buffer of 80
NDArray<bool> m = np.zeros(new Shape(40), NPTypeCode.Boolean).AsGeneric<bool>();
var r = m.reshape(view.Shape);
// before: dims (5,2,4), strides (16,8,1), bufferSize 40 -> element [4,1,3] is read at offset 75 of 40
// now: dims (5,2,4), strides (8,4,1), bufferSize 40 (NumPy: np.zeros(40, bool).reshape((5, 2, 4)))
The result addressed memory outside its own buffer: wrong values read from whatever lies behind it, writes into
it, and an access violation when the buffer ends where mapped memory ends. The Shapes of a stepped view, a
reversed view (negative stride and offset), a transposed array (F strides) and a broadcast array (stride 0) were
installed the same way.
A comparison returns NDArray<bool>, so the natural port of NumPy's `(a % 3 == 0).reshape(x.shape)` hit it. An
existing test did exactly that: Indexing_KernelRoute_Tests.Where_NarrowRows_3D_LeadingAxisWalkedPerBlock (added
2026-09-03) built its where= mask with `.reshape(sv.Shape)`, sv being x[":, :, :4"]. Its assertion compared
np.add(..., where: mask) with np.where(mask, ...): both sides read the same mask outside its buffer, so the test
passed. Under NUMSHARP_DEBUG_GUARD_PAGES=1 it ended the test host in the masked 2-D kernel, which is how this was
found.
THE FIX (src/NumSharp.Core/Generics/NDArray`1.ReShape.cs)
reshape(ref Shape newShape) relabels with newShape.AsClean(): a Shape that already is a plain dimension list
(offset 0, the C strides of its dimensions, a buffer size equal to its element count) is passed on as it is, so
a.reshape(2, 3) and a.reshape(new Shape(2, 3)) allocate what they did; any other layout is re-laid out in C order
first. The -1 inference and NumPy's size-mismatch text are unchanged (pinned by the new tests).
NOT CHANGED, ON PURPOSE
- reshape_unsafe(Shape), typed and untyped: the unchecked relabel that installs the Shape it is given.
Broadcasting uses it for exactly that.
- The typed overloads still COPY a non-contiguous source where the untyped route returns NumPy's strided view
(_attempt_nocopy_reshape). That is a view / copy difference, not an addressing error. Routing the typed
overloads through ReshapeCore would close it; ReshapeCore lives in Creation/NDArray.ReShape.cs, which branch
strings also edits, so it is left for after that merge.
TESTS (test/NumSharp.Tests/Manipulation/TypedReshapeViewShapeTests.cs, new, 3 tests; expected values are NumPy
2.4.2's: np.arange(40).reshape((5, 2, 4))[i, j, k] == (i*2 + j)*4 + k)
- TypedReshape_WithTheShapeOfAnotherArraysView_UsesItsDimensionsOnly: the Shapes of a stepped view, a column
view, a reversed view, a transposed array and a broadcast array. The result has dims (5,2,4), strides (8,4,1),
offset 0, bufferSize 40, the untyped overload's strides, and NumPy's 40 values.
- TypedReshape_WithAViewsShape_IsAViewOfItsOwnBuffer: a write through the result shows in the source; a
comparison's NDArray<bool> reshaped with a view's Shape counts NumPy's 14 True values.
- TypedReshape_WithAPlainShape_IsUnchanged: a plain Shape, the -1 inference, NumPy's error text.
Before the fix the first test reads strides (16,8,2) / (16,8,1) / ... where (8,4,1) is expected.
Where_NarrowRows_3D_LeadingAxisWalkedPerBlock now runs over a mask inside its buffer and passes under guard pages.
VERIFICATION (the tree of this commit and its two neighbours, which were developed together)
NumSharp.Tests net10.0 and net8.0, filter TestCategory!=OpenBugs&TestCategory!=HighMemory: 0 failures
(no test depended on the old relabel)
NumSharp.Tests.Oracle net10.0 and net8.0: 0 failures
Under NUMSHARP_DEBUG_GUARD_PAGES=1 the whole unit suite now runs to its end without a crash (it stopped at the
test above before), and so does the whole oracle replay.
DOCUMENTATION
.claude/CLAUDE.md, "Critical: View Semantics": "NDArray<T>.reshape(Shape) reads the DIMENSIONS of its argument".
BRANCH NOTE
Generics/NDArray`1.ReShape.cs is byte-identical on journey4 and on strings before this commit.
…>, >= runs the vector kernels - a ComparisonRun turns any chunk (stepped, reversed, another dtype, a stepped out) into calls of the contiguous SIMD kernels, a two-level route runs a matrix against a row, a column or an outer product without an iterator, and complex128 and int64-against-uint64 get vector kernels: 1,279 benchmark cells geomean 1.15x -> 2.05x NumPy, cells under 1.0x 445 -> 43, the lowest cell 0.03x -> 0.81x WHY Commit 19a0057 gave array-against-scalar comparisons a vector loop. Every other variation of the six comparisons still ran one element at a time: a matrix against a row or a column, a stepped or reversed operand, two dtypes, a stepped or float64 out=, complex128, int64 against uint64. On a 1,279-cell benchmark 445 cells were slower than NumPy and 88 of them more than twice as slow (the lowest 0.03x). WHAT NUMPY DOES (numpy/_core/src/umath/loops_comparison.dispatch.c.src, PyUFunc_SimpleBinaryComparisonTypeResolver) Its comparison loops are vectorized for three operand forms only: two contiguous inputs, or one input a single value, each with a contiguous bool output. A stepped or reversed input, an input of another dtype (cast through a buffer first) and a stepped output run its scalar loop at 0.6 - 1 ns an element; so do complex128 and float16. int64 against uint64 has exact loops. WHAT THIS COMMIT DOES 1. ComparisonRun (src/NumSharp.Core/Backends/Kernels/ComparisonRun.cs, new) One cached instance per (operator, left dtype, right dtype). It takes one inner-loop chunk (three pointers, three byte strides, a count: NumPy's inner-loop contract) and runs it through the three contiguous vector kernels (SimdFull, SimdScalarRight, SimdScalarLeft), whatever the strides and dtypes are: direct an input that already is a contiguous run of the dtype the kernel compares in, or one value, is read in place. A long run that starts off a vector boundary gets a short first kernel call (AlignmentPeel: the input stream for lanes of two bytes or more, the output for one-byte lanes). staged any other input is copied and converted 8 KB at a time into a stack block, and the kernel reads the block while it is in L1. The house SIMD copy removes a step of two or a reversal, the new strided move any other stride, the house SIMD cast converts. flipped inputs that run backwards in the kernel's dtype are read forwards from their last element and the bool results are stored reversed (one byte an element moved, not 8 or 16). scattered a bool output that is not contiguous receives each block through a strided byte store. adopted an array of another dtype against ONE value is compared in the array's own dtype (point 3). A chunk under 64 elements runs the per-element kernel the iterator routes always used. 2. Three callers - DefaultEngine.TryTrivialContiguousComparisonOp -> Run: two contiguous arrays of one shape, or an array and one value, of any two dtypes; also two one-element operands of different rank. - DefaultEngine.TryRowsComparisonOp (DefaultEngine.CompareOp.Rows.cs, new) -> RunRows: an iteration of R runs of N elements, without an iterator. The structure is read from the strides with NumPy's coalescing rule (stride[d] == stride[d+1] * extent[d+1]). It declines an F-ordered result, an iteration of three or more levels, and many rows under 16 elements whose inputs are contiguous or one value (the iterator's 2-D block kernel loops those inside one call). Measured: an iterator costs 428 ns to build and dispose and 23 ns for each chunk it hands out (40 float64 values); a kernel call per run costs 7.8 ns. - NDIterRef.TryExecuteComparisonRun (NDIter.Execution.ComparisonRun.cs, new) -> one Chunk call per inner-loop chunk: every other iteration, and every out= call (DefaultEngine.ExecuteComparisonUfuncInto; an out of another dtype is filled through the iterator's Boolean buffer). where= keeps its masked route (1.5 - 11x NumPy). 3. Scalar adoption is exact, never a cast (ComparisonRun.ScalarAdoption) NumPy's comparison is convert(a) op convert(v) in the promoted dtype. It has the same answer as a op a' for ONE value a' of the array's dtype, or the same answer for every element: integer array, integer value a' = v inside the array dtype's range; outside it the chunk is decided integer array <= 32 bits, float t a > t is a > floor(t), a >= t is a >= ceil(t), a < t is a < ceil(t), a <= t is a <= floor(t); == can hold only for an integral t in range float16 / float32 array, wider t > and <= use the largest array-dtype value not above t, >= and < the smallest not below it; == can hold only when t is an array-dtype value a NaN value decides the chunk (False; True for !=) So float32_array > 0.1 still compares against the double 0.1. Not adopted: a 64-bit integer array against a float, complex, decimal, char. 4. New kernels (src/NumSharp.Core/Backends/Kernels/Direct/) - DirectILKernelGenerator.Comparison.Complex.cs: complex128, four elements a step. Two loads are split into real and imaginary lanes, compared by NumPy's CLT rule (a NaN in an imaginary part makes the ordered operators False), and the mask bits become bytes (pdep, or integer arithmetic without BMI2). 0.66 ns an element for <, NumPy 0.8 - 1.45. - DirectILKernelGenerator.Comparison.MixedSign.cs: int64 against uint64, either order, as machine compares (a negative lane is decided by its sign, the others compare unsigned), eight elements a step. The emitted loop compared in System.Decimal. - DirectILKernelGenerator.StridedMove.cs: an unrolled IL strided move for elements of 1 / 2 / 4 / 8 / 16 bytes. DirectILKernelGenerator.Comparison.cs hooks the two comparison kernels into GenerateComparisonKernel. NDPolySeries.cs gains PolySeriesView.RawStrided (a run of raw memory at any byte stride, for the staging copy). FIXED ON THE WAY - np.less(int64, uint64, out=o) compared in float64 (inexact past 2^53). Every route is exact now. - Two one-element operands of different rank ((1,) < 0-d) took 1.03 us through an iterator; 0.44 us now. RESULTS ARE UNCHANGED Every route gives the bytes of the per-element comparison at NumPy's comparison dtype. TESTS (test/NumSharp.Tests/Backends/Kernels/ComparisonRunTests.cs, new, 19 tests) - every dtype pair x operator x 1-D layout (contiguous, steps of 2 and 3, reversed, negative steps, one value, views that end at the buffer's last element) against an element reference written from NumPy's rules; - two- and three-dimensional layouts (row, column, outer product, C against F, reversed and stepped rows and columns, partial broadcasts of a 3-D operand); - the four out= forms (a stepped out's skipped slots are not written), long runs off a vector boundary, an out that is an input, int64 against uint64 on every route, result order, one-element operands of different rank; - scalar adoption over every uint8 / int8 value, every float16 bit pattern and the float32 values around each threshold, both sides, six operators; - Run and RunRows on raw memory with sentinel margins, and with every operand against an inaccessible page (FencedMemory): no byte outside an operand is read or written; - the complex, mixed-sign and strided-move kernels and the bit helpers; - the run against the routes it replaced (ComparisonRun.Disabled). The class takes about 4.5 s. MUTATION PASS: 35 planted defects (kernels, adoption, staging, stores, the rows route, dispatch). 33 fail one test or more. The two survivors give the same results: a range test at the boundary (the restated comparison is all False as well), and a branch of the bit-pattern step that its caller never reaches. The pass also showed that the first value generator of the tests drew ONE value for pools of seven entries (a step of 7); it is a multiplicative hash now, and the scratch sweep that shared the flaw was run again. SWEEP: a scratch program against an independent reference (15 x 15 dtype pairs x 6 operators x layouts x out= forms x 2-D and 3-D broadcasts): 373,543,380 checks, 0 failures, with AVX2, with DOTNET_EnableAVX2=0 and with DOTNET_EnableHWIntrinsic=0. The unit classes also pass with EnableAVX=0, EnableBMI2=0 and EnableSSE41=0. SUITES NumSharp.Tests net10.0 18,474 passed, net8.0 18,473 passed, 0 failed (CI filter) NumSharp.Tests.Oracle net10.0 280 passed, net8.0 280 passed, 0 failed Both under NUMSHARP_DEBUG_GUARD_PAGES=1: no crash (the pool-accounting tests fail there by design). BENCHMARK (NPY/NS; two pinned P-cores; tier-up forced with DOTNET_TC_CallCountingDelayMs=0; per-cell minimum over three NumSharp runs and two NumPy runs; in brackets the build before this commit, measured the same way) 1,279 cells: A contiguous (13 dtypes x 6 operators x array / value forms x 1K / 100K / 10M), B 20 layouts, C 12 dtype pairs, D out= and where= forms, E C# literal operands, F tiny and edge operands. all cells geomean 2.05x (1.15x) under 1.5x: 463 (904) under 1.0x: 43 (445) under 0.5x: 0 (88) lowest 0.81x (0.03x) A 1.81x (1.62x) B 2.12x (0.75x) C 3.06x (1.20x) D 2.24x (1.75x) E 1.47x (0.73x) F 1.43x (1.37x) float64, 100K elements, NumPy us | before -> after: a matrix against a row 37.1 | 75.8 -> 14.6 (0.49x -> 2.55x) against a column 38.1 | 107.6 -> 11.2 (0.35x -> 3.41x) an outer product 58.3 | 107.9 -> 10.6 (0.54x -> 5.49x) reversed columns 133.2 | 113.1 -> 25.4 (1.18x -> 5.25x) a reversed 1-D pair 63.4 | 107.6 -> 18.5 (0.59x -> 3.42x) a stepped array against a value 63.9 | 109.5 -> 30.9 (0.58x -> 2.06x) a fully broadcast operand 11.6 | 107.7 -> 10.5 (0.11x -> 1.10x) C against F 78.2 | 117.8 -> 61.3 (0.66x -> 1.28x) other dtypes, 100K: int32 against a column 20.6 | 108.5 -> 8.0 (0.19x -> 2.58x) uint8 reversed columns 85.0 | 112.1 -> 12.5 (0.76x -> 6.79x) complex128 < 143.1 | 256.0 -> 90.7 (0.56x -> 1.58x) complex128 < against a value 158.1 | 273.6 -> 73.6 (0.58x -> 2.15x) two dtypes, 100K: float32 array, float64 value 58.1 | 70.0 -> 5.95 (0.83x -> 9.77x) int32 array, float64 array 63.8 | 70.4 -> 24.0 (0.91x -> 2.65x) uint8 array, int32 value 45.5 | 35.3 -> 3.7 (1.29x -> 12.2x) int64 against uint64 (1K) 1.21 | 4.36 -> 0.71 (0.28x -> 1.72x) out=, 100K: a stepped bool out 63.7 | 104.7 -> 38.1 (0.61x -> 1.67x) a float64 out 107.1 | 95.2 -> 56.9 (1.13x -> 1.88x) C# literals, 100K: float32 > 50.5 6.8 | 70.3 -> 7.1 (0.10x -> 0.95x) uint8 > 50 4.5 | 45.3 -> 4.7 (0.10x -> 0.96x) int32 > 50.5 59.4 | 70.4 -> 8.1 (0.84x -> 7.3x) float16 > 50.5 261.8 | 275.2 -> 15.7 (0.95x -> 16.7x) uint64 > 50 34.2 | 89.1 -> 19.6 (0.38x -> 1.74x) The numpy.matrix benchmark (97 cells) reads geomean 2.30x; m > 0.75: 1.99x / 1.29x / 1.20x at 4 / 64 / 512. THE CELLS STILL UNDER 1.5x ARE MEASURED FLOORS (every cell under 1.0x is between 0.81x and 0.99x) - Contiguous operands of one dtype at 100K elements, and their literal, F-ordered and offset forms (26 of the 43 cells under 1.0x; most of the 163 contiguous 100K cells under 1.5x). Both libraries run a vector loop at the L2 transfer bound and the buffer's address decides the rest. Kernel only, ns an element, input offset / output offset in bytes past a 64-byte boundary: float64 two arrays 0.141 at 0/0 and 0.250 at 16/0; float32 0.074 and 0.126; uint8 against a value 0.026 at 0/0 and 0.036 at 16/16. A prefix aligns ONE stream. Pooled buffers are 16-byte aligned, so each array wins or loses by its address; NumPy's are too (ctypes.data % 64 is uniform over 0 / 16 / 32 / 48 up to 800 KB). OPEN LEVER, not in this commit: 64-byte aligned pool buffers from one page up (SizeBucketedBufferPool is the only source of Native blocks, so the change is contained there). - Two stepped float64 / int64 operands, and complex128 == / !=, at 100K (10 cells, 0.85 - 0.97x): 3.2 MB of source, above this CPU's 2 MB L2. - Six 1K cells at parity (0.90 - 0.99x): two stepped float64 / int64 operands (two staging passes), bool C against F. - The empty result (0,3) < (3,) (0.85x): branch strings has an empty-result guard that arrives with the merge. - Between 1.0x and 1.5x (420 cells): contiguous at 100K (the same transfer bound), contiguous at 1K and tiny operands (the NDArray allocation floor: about 0.35 us of a 0.45 - 0.55 us call), C against F, literal operands at 1K (the literal's 0-d temporary, plan X26). MERGING strings The edits stay out of the hunks strings changed in DefaultEngine.CompareOp.cs and repeat its comparison-type lines of DefaultEngine.UfuncOut.cs byte for byte. git merge-tree reports the same conflict list with and without this commit (the ten files that conflicted before). DOCUMENTATION .claude/CLAUDE.md: "The comparison engine" under Comparison & Logic, "Memory-safety gates: fenced memory and guard pages" under Build & Test, and a row in the Core Source Files table.
…ta science library and their C# ports through a Python.* tree; the first pair is scikit-learn supervised learning (split, scale, fit, predict, score), and the port prints the same seven lines as its Python program on Windows and on Linux
WHAT THIS IS
A new example project for the "pythonic" way to drive a Python library from C# (.claude/skills/pythonic):
a root `static class Python` whose nested classes follow the Python module paths, so that a call reads in
C# as it reads in Python and the compiler checks it. The two existing pythonnet example projects
(NumSharpPythonnetLocalExample, NumSharpPythonnetGhActionsCicd) show the other style, `dynamic` and a
codec. This project shows the style for a bridge.
Each example is a PAIR:
- a real Python program, kept in the folder as the reference (ScikitLearn/classification.py);
- its C# port (ScikitLearn/Classification.cs), statement by statement. The Python lines stand as a
comment above the C# lines that port them.
The two programs print the same lines. verify.py runs each pair and compares the outputs.
This commit holds ONE pair, for review. The owner asked for one library and one feature first. The plan
for the other five pairs is at the end of this message.
THE FIRST PAIR: scikit-learn, supervised learning
The program loads the Wisconsin breast cancer table (569 tumors, 30 numbers each), takes the ten "mean"
columns, keeps a stratified quarter apart (train_test_split), scales with a StandardScaler that learns from
the training part only, fits a LogisticRegression, and reports accuracy, area under the ROC curve and the
confusion matrix on the test part. Then it answers a question with array code: which probability limit
finds every malignant tumor of the test, and how many benign tumors does that limit refer?
tumors 569 with 30 numbers each: 212 malignant, 357 benign
accuracy 0.9161
roc auc 0.9759
[[46 7]
[ 5 85]]
missed 7 of 53 malignant tumors at the limit 0.5
limit 0.9528 finds all of them and refers 28 of 90 benign tumors
Three rules give every line of the port:
1. A library call is a member of the tree with the same name:
Python model = LogisticRegression(max_iter=1000); model.fit(X_train_scaled, y_train)
C# using PyObject model = Python.sklearn.linear_model.LogisticRegression.New(max_iter: 1000);
Python.sklearn.linear_model.LogisticRegression.fit(model, X_train_scaled, y_train);
A keyword argument is a C# named argument with Python's name (test_size: 0.25, stratify: y_np,
random_state: 0). A method of a Python object is an unbound member that takes the object first, as
Python itself can write it. A constructor is `New`.
2. A NumPy line is a NumSharp line with the same operators: `y_test == 0`, `a & b`, `~mask`, `p[mask]`,
`.max()`, np.count_nonzero. Only a slice is spelled as a string: X[:, :10] is X[":, :10"].
3. An array changes sides in one visible call: nd.ToNumpy() (NumPy views NumSharp's memory, no copy,
the strided column view included) and pyobj.ToNDArray() (one copy of a Python result). A name that
ends in `_np` is NumPy's side of the array that NumSharp holds under the name without the ending.
Two things are C#'s and have no line in the Python program: PythonRuntime.Run(() => { ... }) holds the GIL
for the Python part and returns .NET values only, and every PyObject is a `using` local (no call is chained
on the result of another one). One NDScope releases the NumSharp arrays of a port.
THE FILES (examples/Pythonic)
Python.cs the root `static partial class Python` and the seven rules of the tree
ScikitLearn/Python.sklearn.cs the scikit-learn branch, twelve members:
Python.sklearn.datasets.load_breast_cancer(return_X_y)
Python.sklearn.model_selection.train_test_split(X, y, test_size, random_state, stratify)
Python.sklearn.preprocessing.StandardScaler.New / fit_transform / transform
Python.sklearn.linear_model.LogisticRegression.New(max_iter) / fit / predict / predict_proba
Python.sklearn.metrics.accuracy_score_t / roc_auc_score_t / confusion_matrix
ScikitLearn/classification.py the Python program
ScikitLearn/Classification.cs its port
PythonRuntime.cs the backend under the tree: Initialize (idempotent, one engine for each
process), Run / Run<T> (the one place that takes the GIL), Import (cached,
process-owned handles, the loser of a race disposes), Shutdown; and
PythonInteropException, the base type for bridge errors
PythonInstall.cs finds the Python to embed: NUMSHARP_PYTHONNET_PYTHON (binding), then a
.venv in the current directory or beside the project file, then PATH
Program.cs `dotnet run` (every port under a heading), `dotnet run -- <name>` (one
port, the reference's lines and no more), and the lifetime check
verify.py builds once, runs each */*.py and its port, compares line by line; the
exit code is the number of ports whose output is different
requirements.txt numpy 2.4.2, scipy 1.16.3, scikit-learn 1.9.1
README.md, .gitignore, Pythonic.csproj
CONVENTIONS OF THE TREE (the ones that the skill does not spell out, decided here)
- The members carry Python's parameter names in Python's spelling (test_size, random_state, max_iter,
return_X_y), so a C# named argument reads like the Python keyword.
- A keyword parameter is nullable, and `null` means "do not send it": Python's own default applies and
the tree never restates a default that a later library version can change.
- A member returns a PyObject that the caller owns. A `_t` member returns a .NET value in its place
(accuracy_score_t returns a double).
- fit() returns void: Python returns the model itself from fit, and the member drops that second
reference, so a caller cannot leak it.
- Strict disposal inside a member: the bound function, the keyword dictionary and each boxed argument
are `using` locals (`using (PyObject value = new PyInt(seed)) kwargs["random_state"] = value;`).
- One file for each library branch (`static partial class Python`). The module classes are lower case,
as Python has them, which raises CS8981. A `#pragma warning disable CS8981` at the top of the branch
file silences it, because the project file must stay in the regular form (no NoWarn).
- The tree holds the calls that its programs use and no more.
- No member moves array data. Arrays cross at the call site with the verbs of NumSharp.Interop.pythonnet.
The codec is NOT registered: it is process-global, and a bridge must not set it for other code.
THE BACKEND
PythonRuntime is library-agnostic. It keeps what the measurements of 2026-10-09 found for the Local
example: PythonEngine.ProgramName is set for every interpreter (it is what makes CPython adopt a virtual
environment; without it, Windows reports the .NET program as sys.executable and Linux takes its paths from
the first python3 on PATH), sys.prefix is compared with the selected environment after the start, and the
shutdown runs gc.collect() and gc.freeze() first (pythonnet 3.0.5 runs 87 full collections otherwise) and
sets RuntimeData.FormatterType to NoopFormatter. A missing library is a PythonInteropException whose
message names the interpreter in use and the pip command; Program.cs prints it without a stack trace and
ends with exit code 1.
THE LIFETIME CHECK
After the ports, Program.cs reads NDArrayPythonInterop.LiveExports and LiveImports. Both must be 0: each
port disposed every Python object it made, so no NumPy array views NumSharp memory any more and no NumSharp
array views Python memory. Otherwise the program prints the two counts and ends with exit code 2. The check
has an effect: a scratch copy of the port with one `using` removed (means_np) ended with exit code 2 and
"A port left memory shared with Python: 1 NumSharp buffers, 0 Python buffers."
THE PROJECT FILE
Pythonic.csproj is what `dotnet new console` and `dotnet add package NumSharp.Interop.pythonnet` write, and
nothing more (the owner's rule for copy-ready examples of 2026-10-10). So the project builds against the
PUBLISHED NumSharp 0.70.0 packages and uses released API only. It is in SciSharp.NumSharp.sln under the
`examples` solution folder.
VERIFIED
Windows 11 x64, Python 3.12.12, numpy 2.4.2, scipy 1.16.3, scikit-learn 1.9.1, pythonnet 3.0.5, .NET SDK 10:
- dotnet build, Debug and Release: 0 warnings, 0 errors (inside the repository, with its
Directory.Build.props).
- `python ScikitLearn/classification.py` and `dotnet run -- classification` print the same seven lines.
Unrounded, the accuracy (0.916083916083916), the area under the curve (0.9758909853249477) and the limit
(0.9527624954311276) are equal with all their digits.
- `dotnet run`: the heading and the seven lines, exit code 0. `dotnet run --project examples/Pythonic`
from the repository root finds the .venv beside the project file.
- `python verify.py`: "ScikitLearn/classification.py 7 lines same output", exit code 0.
- An unknown port name: a message that lists the ports, exit code 1.
- NUMSHARP_PYTHONNET_PYTHON that names a file that does not exist: "... cannot be used: not found",
exit code 1, no other Python is tried.
- An environment without scikit-learn: "Python cannot import 'sklearn.datasets' (No module named
'sklearn'). The interpreter in use is ... Install the libraries of this project into it: ...", exit 1.
- A base install in place of a virtual environment works.
- Wall time of one run: 2.6 to 3.8 seconds.
Ubuntu 24.04 x64 on WSL 2, Python 3.12.14 (GitHub's build), the folder copied OUTSIDE the repository
(so without Directory.Build.props: the project stands alone):
- the two README commands create the environment: 282 MB (258 MB on Windows);
- dotnet build: 0 warnings, 0 errors;
- the port prints the same seven lines as the Python program, verify.py reports "same output", and the
Linux lines are equal to the Windows lines.
NOT VERIFIED: macOS. No workflow runs this project yet.
FACTS ABOUT NumSharp 0.70.0 THAT THE PORT RELIES ON (measured)
- NDArray.ToString() prints a 2-D integer array exactly as NumPy does ("[[46 7]\n [ 5 85]]").
- np.count_nonzero returns long. `~mask` on a Boolean array is the logical not. `mask & other`,
`p[mask]` and `(double)nd.max()` work as in NumPy. nd.shape is long[].
- nd[":, 1"] of a (143, 2) float64 array is a strided view (16-byte strides), and ToNumpy() exports it
as a view.
- np.sum of that strided view gives 87.20879206270375 where NumPy gives 87.2087920627038. A port must
print a NumSharp float result with a fixed number of digits. The programs of this folder print counts,
comparisons and a maximum, which have no such last digit.
- NDArray<T> is in NumSharp.Generic. The port declares its masks as plain NDArray.
HELD BACK UNTIL THE OWNER ACCEPTS THE SHAPE
No paragraph in .claude/CLAUDE.md, and no link from the website, from the package README or from the
sibling example READMEs.
THE PLAN FOR THE OTHER FIVE PAIRS (not started)
The three libraries are scikit-learn, SciPy and Matplotlib: NumPy is what NumSharp ports, and pandas and
PyTorch already have adapters in NumSharp.Interop.pythonnet. Two features for each library:
scikit-learn 2 Model selection: a Pipeline (scaler and classifier) searched with GridSearchCV under
cross-validation; best_params_, best_score_ and cv_results_ come back as arrays. The
pipeline and its parameter grid stay Python text (the snippet runner, `evalStmts`).
SciPy 1 scipy.optimize: curve_fit and minimize on measurements that NumSharp makes; the objective
is a C# function that Python calls back (the "a callback does not hold the GIL" rule).
SciPy 2 scipy.stats: a t-test and a linear regression; the result objects become `_t` members
that return C# tuples.
Matplotlib 1 A line chart: subplots, plot, labels, legend, savefig on the Agg backend; a style context
(`with plt.style.context(...)`) becomes an IDisposable scope. Both programs print the
figure size and a hash of the RGBA canvas.
Matplotlib 2 A histogram and a scatter plot in one figure; the canvas comes back as an NDArray view
without a copy.
Shared additions when a pair first needs them: the snippet runner (evalStmts), a context-manager scope,
Python.builtins members, more pins in requirements.txt.
…e buffer pool hands out 64-byte aligned buffers from 4096 bytes up (the C allocator aligns to 16, so half of all arrays started in the middle of a 32-byte vector and every second vector load of a SIMD kernel crossed a cache line): two-array comparisons of 100,000 elements run 12-25 % faster (float64 20.0 -> 15.1 us), 82 of 614 benchmark cells at that size gain 10 % or more, the comparison benchmark reads 2.08x NumPy with 23 cells under 1.0x (2.05x and 43) WHY The comparison engine of e2ef7f3 left its contiguous two-array cells at 100,000 elements between 0.81x and 1.3x of NumPy, and the reason was the buffer's address. Kernel only, ns an element, input offset / output offset in bytes past a 64-byte boundary: float64 two arrays 0.141 at 0/0 0.250 at 16/0 float32 two arrays 0.074 0.126 uint8 against a value 0.026 0.036 at 16/16 A kernel can align ONE of its streams with a short first call (ComparisonRun.AlignmentPeel does). Only the allocator can align all of them. Pooled buffers came straight from the C allocator, which aligns to 16 bytes, so each array won or lost by its address. NumPy's buffers are 16-byte aligned as well: `a.ctypes.data % 64` is uniform over 0 / 16 / 32 / 48 for arrays up to 800 KB (measured, NumPy 2.4.2 on this host). WHAT CHANGES (src/NumSharp.Core/Backends/Unmanaged/Pooling/SizeBucketedBufferPool.cs) A buffer of AlignedMinBytes (4096, one page) or more starts on a BufferAlignment (64) byte boundary, whether it is fresh (Take), zeroed (TakeZeroed), reused from a bucket, or outside the pooled window (64 MiB and more). - AllocAligned allocates a block AlignedOverheadBytes (80) larger, hands out the first boundary at least a header past its start, and stores the block's own pointer and the requested size in the 16 bytes in front of the buffer. FreeFresh reads the pointer back and frees the block. - Below one page nothing changes: the buffer is the allocator's own block. Such a buffer stays in the L1 cache while a kernel works on it, and 80 bytes would be a large part of it. - A header, not NativeMemory.AlignedAlloc: the zeroed path must stay calloc. For a large block the C runtime hands out pages that the operating system zeroes when they are first written, which is what makes np.zeros of a huge array cost microseconds. There is no aligned calloc, and an aligned allocation followed by a clear would touch every page. With the header both paths end in the same NativeMemory.Free on every operating system; writing the header touches the block's first page only. - A cache line, not a page: a page boundary costs up to 4 KB a buffer, and buffers that all start at one page offset put every stream of a many-operand kernel in the same cache set at each step. A cache line is enough for the kernels measured: the distance between an output's and an input's offset inside a page does not change their speed (bool == value over 100,000 elements: 2.55 - 2.70 us at twelve distances from 0 to 4032 bytes; uint8 < uint8: 3.23 - 3.40 us). THE RULE IT ADDS Return(ptr, bytes) must get the size the buffer was taken with. The size already selected the bucket; it now also tells whether the buffer has a header. A Debug.Assert compares it with the size in the header. A pool pointer must never be freed directly. Every Native block comes from Take / TakeZeroed (two sites in UnmanagedMemoryBlock), and the other callers (np.copyto, the reduction iterator, the exact min / max engine, the polynomial arena and Vandermonde scratch) pair Take and Return with one size each, so the change is contained in the pool. MEASURED (two pinned P-cores, tier-up forced; per-cell minimum of three runs of each build) The comparison benchmark, 1,279 cells, aligned / unaligned time: 100,000 elements geomean 0.967 82 of 614 cells 10 % or more faster, 4 slower by 10 % or more (two one-byte offset views, bool == True with its literal's temporary, complex128 > a value) 1,000 elements geomean 0.997 10,000,000 geomean 0.986 in interleaved runs, no cell moves 10 % Two arrays of 100,000 elements, us (and NPY/NS): float64 20.0 -> 15.1 (0.98x -> 1.30x) float32 10.6 -> 8.5 (1.01x -> 1.25x) int32 11.4 -> 9.4 (1.19x -> 1.43x) int16 6.4 -> 5.7 (1.14x -> 1.29x) An array against a value moved by 0 - 4 % (the prefix aligned its one stream already), one-byte dtypes by 1 - 6 %. The whole benchmark against NumPy: geomean 2.08x (2.05x), under 1.5x: 451 (463), under 1.0x: 23 (43), the lowest cell 0.86x (0.81x); section A (contiguous operands) has no cell under 1.05x. The numpy.matrix benchmark (97 cells, three interleaved runs of each build): geomean 0.986 of the unaligned time, no cell 10 % slower; m.cumsum(0) at 64 x 64 7.9 -> 4.5 us, an int32 matrix times 2 at 512 x 512 142 -> 129 us. THE 23 CELLS STILL UNDER 1.0x (0.86x - 1.00x) - two stepped float64 / int64 operands at 100,000 elements (8 cells): 3.2 MB of source, above this CPU's 2 MB L2; - a C# literal operand at 100,000 elements (9 cells): the literal's 0-d temporary costs about 1 us of a 4.6 us call (plan X26); the same comparison with a held 0-d operand is 1.07x or more; - four cells at 1,000 elements at parity (bool C against F, two stepped int64 operands); - np.less(a, b) of two float64 arrays at 100,000 (0.98x: both libraries at 15.3 - 15.6 us); - the empty result (0,3) < (3,) (branch strings has a guard that arrives with the merge). TESTS (test/NumSharp.Tests/Backends/Unmanaged/PoolBufferAlignmentTests.cs, new, 4 tests) - TakeAndTakeZeroed_FromOnePageUp_AreAligned: eleven sizes from 4096 to 64 MiB + 1, several buffers of a size alive at once, first and last byte written and read back, a zeroed buffer is zero, a buffer taken again after its return is aligned. - BelowOnePage_BuffersAreTheAllocatorsOwn: sizes 1 to 4095 are usable from the first byte to the last. - Arrays_FromOnePageUp_StartOnACacheLine: np.empty, np.zeros, np.arange, a + b, a < b, astype and copy. - Clear_FreesAlignedBuffers. Under NUMSHARP_DEBUG_GUARD_PAGES=1 the two alignment tests report Inconclusive (that mode places a buffer's END at a page boundary instead). SUITES NumSharp.Tests net10.0 18,478 passed, net8.0 18,477 passed, 0 failed (CI filter) NumSharp.Tests.Oracle net10.0 280 passed, net8.0 280 passed, 0 failed NumSharp.Tests in a DEBUG build, net10.0: 18,478 passed. The Debug.Assert on the returned size is live there, so every Return of the suite gave the size it took. ALSO ComparisonRun.cs: the remarks of AlignmentPeel said that NumPy's large arrays are 64-byte aligned on Windows. The measurement above shows that this holds from about 1.6 MB only; the text now says what the prefix is still for (a view that starts inside its buffer, memory the pool did not allocate). DOCUMENTATION .claude/CLAUDE.md: a row in Key Design Decisions; "The pool aligns its buffers" and the rewritten "Floors" in the comparison section, with the benchmark numbers of this build; the alignment finding of the matrix section marked as acted on.
…- an explicit 0-d ARRAY (np.array(2), np.asarray(2)) is an ndarray to matrix.__mul__, so it raises "shapes (2,3) and (1,1) not aligned" unless the matrix has one column, and np.array(2) * m takes np.dot's scalar route; every computed 0-d value (m.sum(), np.dot(v, v), np.linalg.norm(a), a[0, 0], NDArray.Scalar) still scales WHY 201e9c7 documented one divergence of NDMatrix as [Misaligned]: m * np.array(2) returned the scaled matrix, where NumPy 2.4.2 raises ValueError. The owner decided for NumPy's behavior, with the condition that nothing else is hurt. There is no mathematics in NumPy's error, only the order of two type tests (numpy/matrixlib/defmatrix.py): def __mul__(self, other): if isinstance(other, (N.ndarray, list, tuple)): return N.dot(self, asmatrix(other)) # np.array(2) lands here: asmatrix makes it [[2]] if isscalar(other) or not hasattr(other, '__rmul__'): return N.dot(self, other) # 2, np.float64(2), m.sum() land here A 0-d ndarray passes the first test, becomes a 1 x 1 matrix, and the matrix product of (2,3) with (1,1) has unequal inner dimensions. For a matrix with one column the same call works and returns the scaled column. On the left, np.array(2) * m is matrix.__rmul__ = N.dot(other, self), where np.dot takes a 0-d array as a scalar. THE DOUBLE-CHECK THAT SHAPED THE DESIGN NumSharp has one 0-d type for both roles (a NumPy scalar and a 0-d ndarray). A rule that reads the flags cannot tell them apart. Probe of 39 ways to get a 0-d NDArray, with what NumPy returns from the same call: - writeable + owning, NumPy returns an ARRAY: np.array(2.5), np.asarray(2.5), NDArray.Scalar(2.5), np.zeros(()), z.copy(), z.astype(float32); - writeable + owning, NumPy returns a SCALAR: np.dot(v, v), np.vdot, np.inner, np.linalg.det, np.linalg.norm, np.polyval(v, 2.0), np.add(z, 1.0), z + z, np.sqrt(z), np.multiply(z, z), np.negative(z), np.float_power(z, 2) (13 of the 39); - read-only, NumPy returns a SCALAR: sum, mean, max, prod, std, median, average, ptp, trace, trapezoid, nansum; - writeable + not owning: a[1, 2], m[1, 2], v[2] (NumPy: scalars) and x.reshape(()), np.squeeze(x) (NumPy: arrays). So "writeable means array" would make m * np.linalg.norm(a) and m * np.dot(v, v) raise, where NumPy scales. WHAT CHANGED - New per-object mark NDArray.IsExplicitZeroDimArray (Backends/NDArray.ZeroDimArray.cs), set by the two explicit spellings only: np.array<T>(T scalar) and np.asarray<T>(T data). np.array(nd) and np.asarray(nd) of a marked array carry it to the new object (NumPy returns a 0-d ndarray from both). UNMARKED MEANS SCALAR, so nothing that scaled before stops scaling except those spellings. The mark belongs to the object, not to its storage: a view, a .copy(), an .astype() or an operation result does not have it. Branch strings marks C# literals the same way (NDArray.WasPythonLiteral), in another file, so the merge does not conflict. - NDMatrix.Product reads the mark: a marked plain 0-d operand on the RIGHT of a matrix is From(np.dot(m, ViewOf(a0))), the 1 x 1 product (NumPy's alignment text, or the scaled column); on the LEFT it is Scale(a0, m), np.dot's scalar route (a new C-contiguous owner, 0 + s * x for the float dtypes; the multiply ufunc, wrapped, for the others). A 0-d NDMatrix (the stand-in a C# number converts to) is a scalar as before. - Eight internal call sites built COMPUTED scalars with np.array(x): the results of np.all / np.any (np.all.cs, np.any.cs, Default.LogicalReduction.cs) and a temporary in np.histogram.cs. They now use NDArray.Scalar(x), the same function without the mark. Found by planting [Obsolete(error: true)] on the two overloads and reading the compiler's list: np.asarray<T>(T) has no internal caller. - Not marked on purpose: np.zeros / np.ones / np.empty / np.full with a 0-d shape. NumSharp's own kernels allocate their 0-d results with those functions, so marking there would turn computed scalars into arrays. A 0-d array built that way, or by x.reshape(()) / np.squeeze(x), still scales a matrix where NumPy raises ([Misaligned], documented in the class remarks and in .claude/CLAUDE.md). BREAKING m * np.array(2) and m * np.asarray(2) raise unless m has one column (they returned the scaled matrix since 201e9c7, which is not released). To scale, write m * 2, m * NDArray.Scalar(2.0) or m * (NDMatrix)2. ORACLE (test/oracle/gen_matrix_oracle.py -> matrix.jsonl, NumPy 2.4.2) 735 new cases, zero excuses, zero carves: m * a0 and a0 * m for the 13 dtypes x 5 scalar dtypes x the shapes (2,3), (3,1), (1,3), (1,1), seven source layouts, the object and method forms, and the sign of a zero through the left route (float32 / float64 / complex128 with -0.5, 0.5, -0.0, 0.0). NumPy's answers: 213 ValueErrors ("shapes (2,3) and (1,1) not aligned: 3 (dim 1) != 1 (dim 0)"), 148 products on the right, 373 results on the left (owners for the float dtypes, views of the ufunc result for the others). params.array0d lists the operand positions the replay marks (OpRegistry.Matrix.cs); a 0-d operand without it is a NumPy scalar, as before. The values are non-zero on the right route, because np.dot's size-1 branch is a known gap of the array library for the sign of a zero (pinned in OpenBugs.Reductions.cs). Tier: 26,761 gate cases over 87 op keys (was 26,027), 172 carved cases unchanged; the three replays (production, fused, two-pass) pass on both frameworks. MinCases floor 24,700 -> 25,400. TESTS NDMatrixTests.ExplicitZeroDimArray_IsAnArrayOperand_NotAScalar: the four spellings that raise (np.array, np.asarray, np.array(np.array(...)), the object overload), the one-column product, the left route (owner, +0.0), six kinds of 0-d value that still scale (a C# number, NDArray.Scalar, a reduction result, an element, a norm, a vector dot), and where the mark is and is not. ALSO FOUND (no change): the unit test that failed once on net8.0 A hunt (the net8.0 suite in a loop from a copy of the build output of 201e9c7) failed in its third run: AuditV2_LogicShapeStorage.T2_NonZero_Is_Much_Slower_Than_NumPy, "Expected sw.ElapsedMilliseconds to be less than 32L ... but found 33L". It is a wall-clock limit (10 np.nonzero calls over 1M elements in under 32 ms; 4-6 ms on an idle host) that a concurrent build pushed over. A host-load signal, not a product defect; the test's own remarks say so. VERIFIED (Release) - NumSharp.Tests (TestCategory!=OpenBugs&TestCategory!=HighMemory): 18,479 passed on net10.0, 18,478 on net8.0, 0 failed. - NumSharp.Tests.Oracle: 280 passed, 1 skipped (RandomApiSoak) on both frameworks.
…asses are C# classes with instance methods (var scaler = new StandardScaler(); scaler.fit_transform(X_train)), a module is imported with using static, arrays are NumSharp arrays on both ends of every call, one PythonScope replaces every using, and print($"...{x:.4f}") takes Python's own format specs
WHY
The owner's review of the first pair (b10ac7e), in his words: ".cs with pythonic should be using classes
and not statics necessarily which will allow scaler = StandardScaler() / X_train_scaled =
scaler.fit_transform(X_train) / X_test_scaled = scaler.transform(X_test). We need these fixed because the
Python.* are very long in this example", and "Can we also make our pythonic code look more like actual
python?". The first version wrote every object member as an unbound static member behind its full module
path, and every Python object as a `using PyObject` local:
using PyObject scaler = Python.sklearn.preprocessing.StandardScaler.New();
using PyObject X_train_scaled = Python.sklearn.preprocessing.StandardScaler.fit_transform(scaler, X_train);
using PyObject X_test_scaled = Python.sklearn.preprocessing.StandardScaler.transform(scaler, X_test);
WHAT A PORT LOOKS LIKE NOW (ScikitLearn/Classification.cs, beside ScikitLearn/classification.py)
# Python // C#
from sklearn.preprocessing import StandardScaler using static Python.sklearn.preprocessing;
X, y = load_breast_cancer(return_X_y=True) var (X, y) = load_breast_cancer(return_X_y: true);
means = X[:, :10] var means = X[":, :10"];
scaler = StandardScaler() var scaler = new StandardScaler();
X_train_scaled = scaler.fit_transform(X_train) var X_train_scaled = scaler.fit_transform(X_train);
model = LogisticRegression(max_iter=1000) var model = new LogisticRegression(max_iter: 1000);
model.fit(X_train_scaled, y_train) model.fit(X_train_scaled, y_train);
limit = p_benign[malignant].max() var limit = p_benign[malignant].max();
print(f"accuracy {accuracy_score(y_test, y_pred):.4f}") print($"accuracy {accuracy_score(y_test, y_pred):.4f}");
The port has the statements of the Python program in the same order, with the same names and the same
comments. C# writes `var`, `new`, `:` for the `=` of a keyword, `$` for the `f` of an f-string, `true`, and
a slice as a string. One line has no line in Python:
using var scope = new PythonScope(NDScope.Open());
No PyObject, no `_np` twin name, no ToNumpy() / ToNDArray() and no GIL block is left in a port.
PythonRuntime.Run(() => { ... }) is gone.
THE DESIGN
1. A module is a static class; `using static` is its import. The directive brings in the functions AND the
nested classes of the module, as `from module import ...` does. The `using` lines stand below the
file-scoped `namespace Pythonic;` line, where the name `Python` is the class of the project (above that
line it is pythonnet's namespace and the directive needs `Pythonic.Python...`). An alias imports one
class (`using StandardScaler = Pythonic.Python.sklearn.preprocessing.StandardScaler;`), and the full
path works without a directive (`Python.sklearn.metrics.accuracy_score(y, p)`). All three were compiled
and run.
2. A Python class is a C# class that derives from PythonObject (new file). The base owns the one Python
reference, registers itself with the PythonScope of its thread, gives Python's `str(obj)` from
ToString() ("LogisticRegression(max_iter=1000)"), and disposes under the GIL. `fit` returns `this`, so
`new LogisticRegression(max_iter: 1000).fit(X, y)` chains as in Python.
3. A member takes and returns .NET values. Each member is three lines: take the GIL
(PythonRuntime.GIL(), new; re-entrant), Call(...), convert the result. Three private helpers of the root
class do the work (Python.cs):
- Call(target, name, args, kwargs): the one place where .NET arguments become Python objects (an
NDArray is ToNumpy(), a view; a tree object is the reference it holds; bool, integers, double, float,
string are boxed) and where those temporaries are released. A keyword that is null is not sent.
- Create(module, type, kwargs): the constructor call for a base initializer (it takes the GIL itself).
- AsArray(result): `result.AsNDArray(allowReadonly: true)`, a NumSharp VIEW of the memory that NumPy
allocated. So nothing is copied in either direction. In the first version a result was one copy.
The `_t` suffix is gone: accuracy_score returns a double.
load_breast_cancer(return_X_y) has the pair form only and refuses `false` (Python returns another type
there, and a C# method has one return type).
4. PythonScope (new file) is the lifetime of a port: `new PythonScope(NDScope.Open())`. NumSharp's NDScope
collects the arrays, the views of Python memory included; the PythonScope collects the PythonObjects of
its thread and closes both, objects first. The NDScope is opened in the port's own line ON PURPOSE:
NumSharp's analyzer exempts a method that calls NDScope.Open() and reports NDW012 for the arrays of a
method that does not. A PythonScope that opened the NDScope inside its constructor produced 3 NDW012
warnings in the port; the alternatives were [NDScopedCovered] on each port (it also silences a port that
forgot the scope), two scope lines, or an .editorconfig that switches the rule off.
5. print and f-strings (new file Python.builtins.cs). `print(fstring)` takes an
[InterpolatedStringHandler] type, so the compiler passes each hole to it. A hole with a spec goes to
Python's own format(), a double without a spec to Python's str(); integers, text and bool are written
in C# (the digits are the same), and an array by NumSharp, which prints as NumPy does. So the specs are
Python's: .4f, >10.3f, +.2f, 08.3f, the thousands comma, _, .1%, .3e, .3g, d, x, #x, b, o, .2s.
`print(object)` writes one value, `print()` an empty line. There is no print(a, b): a `$"..."` that
is not written directly as the argument is formatted by .NET, and a two-argument form would take that
path without a warning. `str` and `format` are members too. C#'s alignment syntax ({x,10}) does not
compile against the handler.
6. Python.gc.collect (new file Python.gc.cs), which the scope uses.
FOUND AND SOLVED ON THE WAY (each one measured)
- Python garbage holds the training table. After the first run the check at the end reported "1 NumSharp
buffers, 1 Python buffers", also after a 5 second wait. LogisticRegression.fit leaves four unreachable
objects in a reference cycle (a tuple, a function, scipy.optimize._differentiable_functions.
_ScalarHessWrapper and a bound method); the tuple is the `args` of the optimizer and refers to X, which
is now a NumPy view of NumSharp memory. Python frees a cycle only when its collector runs. So after its
release the scope compares NDArrayPythonInterop.LiveExports with the count at its start and, while more
are alive, runs gc.collect(0), then (1), then (2). Generation 0 is enough here and took 0.8 ms; a full
collection took 23 ms. The first version did not see this, because the arrays that it passed to fit were
Python's own.
- A view of Python memory is given back on another thread. NumSharp.Interop.pythonnet 0.70.0 queues the
release of an imported buffer and a thread-pool thread drains the queue under the GIL (or the next
conversion call does). Right after a scope closed, 7 to 9 of its 11 imported buffers were still counted,
and 300 ms later only the one that Python's garbage pinned. So the check in Program.cs waits up to 5
seconds for both counts to reach 0 before it judges.
- A caught Python error kept the arguments of the failed call shared. After one failed
train_test_split(X, y[":5"]) two NumSharp buffers stayed exported: after the scope closed, after a .NET
collection with finalizers, after another Python call and after a full Python collection. They were
released only by a call of pythonnet's Finalizer.Instance.Collect() (its Threshold property reads 200).
The frames of the traceback hold the arguments, and the exception holds the traceback. Disposing the
exception's Type, Value and Traceback did not help. Python's own traceback.clear_frames did: Call
catches PythonException, empties the frames (ReleaseFrames) and rethrows the same exception. The exports
read 0 right after a failed function call, a failed method call and a failed format; the message, the
type name, the formatted traceback (13 lines) and the Python lines of the stack trace are still there.
- NumSharp 0.70.0 prints arrays as NumPy does: 19 arrays (integers, floats of mixed magnitude, float32,
bool, nan and inf, 3-D, a 2000-element summary, a wrapped line, 0-d, empty) gave the same text as
NumPy's str(). That is why print(array) needs no Python call.
- Facts of the published interop that the tree's documentation states, each probed: AsNDArray of a
read-only NumPy array is a non-writeable view; of a 0-d array a 0-d view; of a NumPy scalar a
non-writeable 0-d view; of a reversed float32 array a view; of a complex64 array NotSupportedException
("ToNDArray widens it to Complex (complex128) as a copy"); of an int, a str, a list, a dict or None
NotSupportedException ("the object does not export a PEP 3118 buffer"). copy() plus NDScope.Detach
takes an array out of a scope.
VERIFIED
Windows 11 x64, Python 3.12.12, numpy 2.4.2, scipy 1.16.3, scikit-learn 1.9.1, pythonnet 3.0.5, against the
published NumSharp.Interop.pythonnet 0.70.0:
- dotnet build from a clean folder, Debug and Release: 0 warnings, 0 errors.
- `dotnet run -- classification` prints the seven lines of `python ScikitLearn/classification.py`;
verify.py: "7 lines same output", exit code 0. Three runs: exit code 0 each (the check at the end
passes), 2.6 to 2.7 seconds.
- print against real Python: a scratch port of 26 lines (every spec above, floats 1.0 / 1e-05 / 1e+16 /
1e15 / inf / nan / -0.0 / 5e-324, integers up to 2**64 - 1, bool, None, text, a 1-D array, a 0-d array
with and without a spec, an estimator, str() and format()) printed the same 26 lines as its Python twin.
- A port without its scope line: 3 NDW012 warnings at build, and "A port left memory shared with Python:
1 NumSharp buffers, 11 Python buffers", exit code 2, at run.
- Misuse: a tree object after its scope gives ObjectDisposedException and "<released StandardScaler>" from
ToString(); a scope closed out of order gives InvalidOperationException; a second Dispose does nothing;
a scope on a second thread works and leaves nothing shared.
- Python errors arrive as PythonException with Python's text ("Found input variables with inconsistent
numbers of samples: [569, 5]", sklearn's InvalidParameterError, "Unknown format code 'f' for object of
type 'str'"), and a hole of a type without a Python form gives NotSupportedException.
- Error paths of the program are unchanged: unknown port (exit 1), an interpreter that does not exist
(exit 1), an environment without scikit-learn (the message with the pip command, exit 1).
Ubuntu 24.04 x64 on WSL 2, Python 3.12.14, the folder copied outside the repository: 0 build warnings, the
port prints the lines of the Python program, exit code 0, verify.py "same output", and the Linux lines are
equal to the Windows lines.
NOT VERIFIED: macOS. No workflow runs the project. The 26-line print test ran on Windows only (the
formatting is done by Python, so the platform should not matter).
BREAKING (inside the example; nothing outside it used the first form)
The tree of b10ac7e is replaced: the unbound members (StandardScaler.New / fit_transform(scaler, X)), the
`_t` members, PythonRuntime.Run and the PyObject-typed signatures do not exist any more.
OPEN
- The skill file .claude/skills/pythonic/SKILL.md still teaches the unbound static form that this commit
replaces. It is not changed here: the owner decides.
- The plan for the other five pairs (b10ac7e's message) stands, and each pair now takes this form. For
them the tree needs, when a pair first uses it: text parameters typed as the f-string type, a `with`
block as an IDisposable scope, a C# function as a Python callback, attributes of fitted objects.
…70 rules for a port, the tree, arrays, the GIL, lifetime, errors, print, the backend, verification and the project, each one met by examples/Pythonic at ebcc245, with the measurement behind it and a list of what is not specified yet WHY The owner asked: "Kindly write down a new PYTHONIC_SPEC.md". The form of pythonic code changed in ebcc245 (classes with instance methods, `using static` imports, NumSharp arrays on both ends of a call, one scope line, print with Python's format specs). Until now that form was written down only in the README, which is a tour, and in commit messages. The skill file .claude/skills/pythonic/SKILL.md still describes the form that the owner's review rejected. WHAT IT IS examples/Pythonic/PYTHONIC_SPEC.md, version 2 (version 1 is the form of b10ac7e). Each rule has an identifier, so that a review can name it: PORT-1..12 the port: a pair is two files, one method, the namespace line first and the `using` lines below it, one C# form for each Python import form, the scope line as the first statement, one C# statement for each Python statement, Python's names, `var`, Python's comments, a translation table, what a port must not contain, the same printed lines TREE-1..13 the tree: the root class, one file for each package, a module is a static class, a function is a static method, Python's parameter names with `null` = not sent, .NET values in and out (a type table), one return type, a Python class is a sealed class over PythonObject, the three steps of a member, the three helpers and the argument types of Call, only fixed calls, the documentation of a member, accessibility ARR-1..7 arrays: NDArray on the C# side, an argument is a view, a result is a view, a copy only when a view is not possible, no element loops, no codec, NumSharp computes on Python's memory GIL-1..5 each member takes the GIL through PythonRuntime.GIL(); a port never sees it LIFE-1..9 a tree object owns one reference; the scope line; NDScope.Open() stands in the method (the analyzer); the close order; Python's collector for garbage that holds an export; one thread, innermost first; what leaves a scope; the two counts after a scope; the shutdown ERR-1..4 a Python error is pythonnet's PythonException, unchanged; Call empties the frames of the traceback; a setup problem is PythonInteropException; the tree's own refusals PRINT-1..6 the three forms of print; which side writes which hole; the `$"..."` directly as the argument; no C# alignment; str and format BACK-1..6 one engine, the interpreter search, ProgramName, the import cache, the shutdown VER-1..5 the same lines (verify.py), the lifetime check and the exit codes, no warning, a new format or conversion path is tested against Python, where a pair ran PROJ-1..3 the regular project file, the pinned requirements, the folder Section 1 states five goals. Section 13 lists thirteen constructs that this version does not specify (an attribute of an object, a dict or list argument, a `with` block, a C# function that Python calls, a text parameter with a format spec, a keyword that is None on purpose, operators and indexers, a second form of one function, control flow, a catch by the Python type, Python source as text, await, a shared tree), each with a proposal that is marked as a proposal. Section 14 has the checklists for a new member, a new pair and a review. Appendix A gives the nine measurements behind the rules. VERIFIED WHILE WRITING IT (scratch ports against the published 0.70.0 package and real Python; nothing in the project's code changed) - PORT-4, the import forms. All of them were compiled and run, and each printed what its Python twin printed: `using static Python.pkg.mod;`; a class alias (`using C = Python.pkg.mod.C;`); a module alias (`using metrics = Python.sklearn.metrics;`, for `from sklearn import metrics`); `using static Python;` and then `sklearn.datasets.load_breast_cancer(...)`, for `import sklearn.datasets`; and `using xp = NumSharp.np;`, for `import numpy as xp`. A lower-case alias gives the warning CS8981, which the rule states. - ARR-2. A reversed array (y["::-1"]), a stepped array (y["::3"]) and one column of a table as arguments of accuracy_score, confusion_matrix and roc_auc_score gave Python's results (0.4798, the matrix [[64 148] [148 209]], 0.061684). - PRINT-5. `print($"[{12.5,10}]")` does not compile: error CS1739, "The best overload for 'AppendFormatted' does not have a parameter named 'alignment'". It does not fall back to the print(object) overload and .NET's formatting. - PRINT-3. A C# float hole prints the exact value of the float: 0.1f gives 0.10000000149011612, as float(np.float32(0.1)) does in Python. - A script checked that each rule identifier that the text refers to exists (70 rules, 9 appendix items) and that each file that the spec and the README name exists. The README links the specification and lists it in its file table. The project builds with 0 warnings and verify.py reports "same output", as before. NOT DONE The skill file is not changed. Whether it gets this form, or points to this document, is the owner's decision. The same holds for the bridges in the sibling repository OptunaSharp, which are written to version 1.
…ed tile by tile, and a matrix against one short row several rows per kernel call - the F operand's rows are no longer gathered one strided row at a time (a new AVX2 kernel copies a tile of several rows, reading each column's rows as one contiguous piece): float64 (250, 400) 63.5 -> 35.3 us (NumPy 78.2), uint8 32.9 -> 9.3 us (NumPy 46.5), a (1000, 1000) matrix against its transpose 1111 -> 548 us (NumPy 1104), a (25000, 4) matrix against a row 85 -> 12.8 us (NumPy 140); 1,279 benchmark cells geomean 2.08x -> 2.12x NumPy, 54 new cells 1.17x -> 3.61x WHY After e2ef7f3 (the comparison engine) and 5b15137 (the aligned pool) 451 of 1,279 benchmark cells were under 1.5x NumPy. Most of them are floors both libraries share (memory transfer at 100,000 elements, the array allocation at 1,000). Three classes had a lever that was not built: - a C-ordered operand against an F-ordered one (24 cells, 0.89 - 1.54x), which is also `m == m.T`; - a one-byte matrix against a row (1.2 - 2.0x), and any matrix against a row of a few elements; - two stepped operands at 1,000 elements (1.0 - 1.4x). This commit builds the first two. The third was probed and has no cheap lever (see FLOORS). 1. TILES: AN OPERAND WHOSE ROWS ARE ADJACENT ELEMENTS An F-ordered matrix read in C order has rows whose elements are one COLUMN apart. A row of N elements touches N cache lines, and N pages once a column is 4 KB long. The rows route (ComparisonRun.RunRows) staged such a row with one strided move a row: 0.3 - 0.6 ns an element for a (250, 400) matrix, 0.65 - 1.3 for (1000, 1000), 3.8 - 8.6 for (3162, 3162). NumPy's scalar loop pays the same. The elements of one column ARE adjacent, so a tile of B rows reads B contiguous elements of each column and deals them out to B rows of a stack block. src/NumSharp.Core/Backends/Kernels/Direct/DirectILKernelGenerator.TransposeTile.cs (new) TransposeTile(src, srcColumnStride, dst, dstRowStride, rows, cols, itemSize): dst[b * dstRowStride + w * itemSize] = src[w * srcColumnStride + b * itemSize] AVX2 groups of 2 x 4 elements of 8 bytes, 4 x 8 of 4 bytes, 8 x 16 of 2 bytes, 16 x 32 of one byte. - Each 256-bit register is built from two columns: the first column's piece in the low lane, the piece of the column `far` further in the high lane (vinserti128 with a memory operand, which uses no shuffle port). The in-lane transposition then gives rows of twice the columns and needs no lane-crossing shuffle. - Integer unpacks for every width. Current Intel cores run them on two ports and the floating-point ones (vunpcklpd, vunpcklps) on one: the 2-byte group runs at 0.13 cycles an element where its 24 shuffles on one port would take 0.19, and the 4-byte kernel went from 0.13 to 0.08 ns an element with them. - The 8-byte loop takes two row groups a step (0.18 -> 0.13 ns an element). - A tile that is not a whole number of groups ends with a group that OVERLAPS the one before it, so there is no scalar tail. That needs a destination that is not the source. - Below one group, for 16-byte elements and without AVX2 the tile is moved by the strided move kernel along its longer side (a plain loop without runtime IL generation). L1 speed, ns an element: 0.13 / 0.08 / 0.05 / 0.03 for 8 / 4 / 2 / 1 bytes (a strided move: 0.2; a memcpy of the same bytes: 0.05 / 0.026 / 0.013 / 0.007). It reads exactly the tile's elements and writes exactly the tile's elements. src/NumSharp.Core/Backends/Kernels/ComparisonRun.cs RunTiles (called by RunRows before its row-by-row loop) and TileOperand. - An operand is tiled when its successive runs are one element apart, in either direction, and the elements of one run are further apart (IsTiled). Rows that go downwards in memory are read from the tile's last row and stored from the block's last row upwards. - The tile is at most 8 KB of the kernel dtype. It is a whole row wide when 8 rows of that width fit, else the row is cut evenly; then it is as many rows high as fit. Widths and heights are multiples of the vector group. From a row SPAN of 4 MB (elements a row times the column stride) the fewest rows are 16 (4- and 8-byte elements) or 32: each column piece is then a cache miss, and further up a TLB miss. - The last tile of each direction is moved back to end on the operand's edge and computes some results a second time. RunRows never has an output that is an input (its contract). - The operand beside the tiled one: contiguous rows are read in place, or converted into the other block (one cast for the tile when the rows follow each other); one value a row is read in place; a second tiled operand gets its own block. - One kernel call for a tile when every operand's rows of the tile follow each other and the output's do; otherwise one call a row. - Declined (the row-by-row loop runs): 16-byte elements with rows over 64 elements, and conversions to complex128. Two F operands never arrive (the result is F-ordered: the iterator route), nor do out= calls (they arrive as chunks). Geometry, measured through the public operator (one process, the configurations in turn, five passes, minimum; us a call): stage of 8 KB against 16 KB (400, 250) uint8 8.9 / 13.6 (250, 400) int16 11.7 / 14.4 (250, 400) float64 40.5 / 36.8 (1000, 1000) float64 791 / 880 fewest rows 4 / 8 / 16 (250, 400) float64 40.5 / 35.3 / 43.2 float32 19.2 / 21.2 / 27.0 (1000, 1000) float64 791 / 641 / 753 float32 414 / 340 / 334 rows 8 / 16 / 32 on big ones (1448, 1448) float32 1550 / 1355 / 1456 int16 625 / 449 / 514 (3162, 3162) float64 17330 / 15261 / 17221, float32 11993 / 8726 / 8855, int16 9350 / 5545 / 4686, uint8 (16 / 32 / 64 rows) 3557 / 2926 / 2643 16-byte elements: a tile of them only saves the calls of the row-by-row loop (a 16-byte element is one move whatever the route). Rows of 3 to 40 complex128 elements gain 1.4 - 2.9x, rows of 400 lose (0.93x), and a conversion to complex128 through the house cast is slower than the per-element kernel (0.70x). Hence the rule. 2. A MATRIX AGAINST ONE ROW A kernel call costs about 8 ns whatever it compares, and a row that ends in a partial vector pays a scalar tail: together most of a short row (400 one-byte elements take 10 ns). ComparisonRun.RunRepeatedRow / RepeatsRow When the matrix's rows follow each other in memory, there are 16 rows or more, a row is at most 2,048 bytes and neither operand is converted, the row is repeated in an 8 KB stack block (filled by doubling: 12 copies for a row of two elements) and a block of rows is compared per kernel call. Kernel time of the repeated form against a call a row: 0.31x for (25, 40) of one byte, 0.52x for (250, 400) of one byte, 0.86x for (1000, 100) of eight bytes (800 bytes a row), 0.95x for (250, 400) of four bytes (1,600), 1.05x for (250, 400) of eight bytes (3,200: the limit is under it). With 7 rows the copy costs about 10 % more than the calls it saves. DefaultEngine.CompareOp.Rows.cs TryRowsComparisonOp sent many rows of fewer than 16 elements to the iterator's 2-D block kernel (1.5 ns a row plus 1 ns an element). A matrix against one row now stays here whatever the row's length. RESULTS ARE UNCHANGED Both routes give the bytes the row-by-row loop gives. Test switches: ComparisonRun.TilesDisabled, ComparisonRun.RepeatedRowDisabled. Counters: ComparisonRun.TileRuns, ComparisonRun.RepeatedRowRuns. TESTS test/NumSharp.Tests/Backends/Kernels/TransposeTileTests.cs (new, 3 tests) - EveryTileShape_IsCopiedExactly: 285 tile shapes (heights and widths on both sides of every group size) x 5 widths x 3 column strides (columns adjacent, with a gap, in descending order) x rows stored upwards and downwards; the destination holds the transposed tile and its sentinel margins are untouched. - Tile_BetweenInaccessiblePages_TouchesNoByteOutsideIt: the same shapes with the source and the destination against inaccessible pages (FencedMemory). - Groups_FollowTheInstructionSet_AndDegenerateTilesAreSafe. test/NumSharp.Tests/Backends/Kernels/ComparisonRunTests.cs (5 more tests, 24 in all; new helper RawTiled) - TiledOperands_MatchAnElementReference: 19 dtype pairs; an F operand on either side, with reversed rows, with reversed columns, two F slices, a column against an F operand; shapes from (2, 2) to (9, 600). - TilesAndRepeatedRows_AgreeWithTheRowByRowRoute: the routes against the loop they replace and against C-ordered copies up to 4.4 million elements (rows cut into several tiles, the taller tiles); the counters prove which route ran, and that long complex rows, a conversion to complex128 and two F operands are declined. - MatrixAgainstOneRow_MatchesAnElementReference: 8 dtype pairs, rows of 2 to 100 elements, the row on either side, and matrices whose rows do NOT follow each other (every second row, reversed rows). - RunRows_Tiles_OnRawMemory_TouchNoByteOutsideTheirOperands: a tiled operand on either side, each form of the operand beside it, gaps after the column pieces, downward rows and columns, every operand and the output against inaccessible pages. - RunRows_RepeatedRow_OnRawMemory_TouchesNoByteOutsideItsOperands: the same for a matrix against one row, with the exact number of calls that must take the route. The class takes about 6 s (4.5 s before). MUTATION PASS: 32 planted defects in the kernel and the two routes. 31 fail: 26 by an assertion, 5 by ending the test host (they read or write outside a buffer). The survivor gives the same results: it drops `width == count` from the test whether an in-place operand's rows are joined, which the output's stride already implies. A trap of the script: a test host that ends in the middle of a run still prints a summary line for the tests that had finished ("Passed! - Failed: 0, Passed: 1, Total: 1"), so five out-of-bounds mutants first read as survivors. The script now compares the summary's total with a clean run's. SWEEP: the scratch program against an independent reference, with a section for these forms over all 15 x 15 dtype pairs and six operators: 453,687,480 checks, 0 failures, with AVX2, with DOTNET_EnableAVX2=0 and with DOTNET_EnableHWIntrinsic=0 (373,543,380 before the section). The two test classes also pass with EnableAVX2=0, EnableAVX=0, EnableHWIntrinsic=0 and under NUMSHARP_DEBUG_GUARD_PAGES=1. SUITES (Release, CI filter TestCategory!=OpenBugs&TestCategory!=HighMemory) On 5b15137 with this change: NumSharp.Tests net10.0 18,486 passed, net8.0 18,485 passed, 0 failed NumSharp.Tests.Oracle net10.0 280 passed, net8.0 280 passed, 0 failed On 8aba70c with this change (see BASE): NumSharp.Tests net10.0 18,487 passed, net8.0 18,486 passed, 0 failed NumSharp.Tests.Oracle net10.0 280 passed, net8.0 280 passed, 0 failed The first net10.0 run there had one failure: AuditV2_LogicShapeStorage.T2_NonZero_Is_Much_Slower_Than_NumPy, a wall-clock limit on np.nonzero (115 ms where it allows 32). It passed in three runs alone and in a second run of the whole suite; the host had other work at that time. This commit does not touch np.nonzero. BENCHMARK (NPY/NS; two pinned P-cores; tier-up forced; per-cell minimum over three NumSharp runs and two NumPy runs) The 1,279 cells of sections A - F, the build of 5b15137 -> this commit: geomean 2.08x -> 2.12x under 1.5x: 451 -> 425 under 1.0x: 23 -> 22 lowest 0.86x -> 0.87x section B (layouts) 2.16x -> 2.27x; the other sections did not move A new section G (54 cells: F against C, a matrix against its transpose, C against F at 1,000,000 elements, a matrix of rows of four elements against a row), this executable with the routes off -> on: geomean 1.17x -> 3.61x under 1.5x: 48 -> 0 NumPy us | routes off -> on (one executable; CMP_OLD_ROUTES=1 switches the routes off): C against F, 1,000 elements float64 2.28 | 1.65 -> 0.84 (1.38x -> 2.72x) bool 1.87 | 2.11 -> 0.67 (0.89x -> 2.80x) C against F, 100,000 float64 78.2 | 63.5 -> 35.3 (1.23x -> 2.21x) float32 51.4 | 50.3 -> 21.2 (1.02x -> 2.42x) int32 54.0 | 52.7 -> 23.8 (1.03x -> 2.27x) int64 83.1 | 67.1 -> 39.6 (1.24x -> 2.10x) uint8 46.5 | 32.9 -> 9.3 (1.41x -> 5.01x) C against F, 1,000,000 float64 1281 | 1290 -> 680 (0.99x -> 1.88x) uint8 616 | 656 -> 151 (0.94x -> 4.08x) m < m.T, 316 x 316 float64 74.2 | 63.7 -> 36.6 (1.16x -> 2.03x) uint8 46.8 | 35.1 -> 8.5 (1.33x -> 5.50x) m < m.T, 1000 x 1000 float64 1104 | 1111 -> 548 (0.99x -> 2.02x) a matrix against a row, 100,000 uint8 8.97 | 7.23 -> 3.84 (1.24x -> 2.33x) float32 20.3 | 8.46 -> 6.82 (2.40x -> 2.97x) (250, 4) against a row float64 2.69 | 2.14 -> 0.72 (1.26x -> 3.75x) (25000, 4) against a row float64 140 | 85.1 -> 12.8 (1.65x -> 10.99x) uint8 122 | 64.5 -> 3.95 (1.89x -> 30.8x) The other cells did not move: geomean 0.981 of the earlier build's time. Four cells read 10 % slower (a contiguous array against a value, code this commit does not touch); in interleaved runs of the two builds they read 0.97 - 1.02. FLOORS THAT REMAIN (22 cells under 1.0x, 0.87 - 1.00x; 403 between 1.0x and 1.5x) - Two stepped float64 / int64 operands at 100,000 elements (8 cells): 3.2 MB of source, above this CPU's 2 MB L2. - A C# literal operand at 100,000 elements (9 cells): the literal's 0-d temporary, about 1 us of a 4.6 us call (plan X26). - Two stepped operands at 1,000 elements (3 cells under 1.0x; 1.3 - 1.5x for 4-byte elements). Probed: the two staging copies are 0.42 - 0.48 us of the 0.61 us the run takes (0.2 ns an element for each stream; a direct two-load deinterleave written for the probe measured 0.18), and the call's fixed part is 0.65 us where contiguous operands take 0.43. No cheap lever was found. - np.less(a, b, out) of two float64 arrays at 100,000 (0.97x) and the empty result (0,3) < (3,) (0.87x; branch strings has a guard). - Between 1.0x and 1.5x: contiguous operands at 100,000 elements in any order (the L2 transfer bound), contiguous and tiny operands at 1,000 elements (the array allocation: about 0.35 us of a 0.45 - 0.55 us call), a one-byte matrix against a column at 100,000 (1.3x: 250 calls with a value each), literal operands at 1,000 (plan X26). MEASUREMENT NOTES - The host's clock read 2.8 GHz during this work (a chain of dependent 64-bit multiplies took 1.06 ns a step; 3.4 GHz during the engine's work). Cycle counts above use the measured clock. - The host was not quiet (about 17 % background load). Routes were compared in one process, in turn, and effects under 1.3x were not acted on. MERGING strings ComparisonRun.cs, DefaultEngine.CompareOp.Rows.cs, the two test files and the kernel file exist on journey4 only (they were added by e2ef7f3 and by this commit). git merge-tree reports the same 13 conflicting files with and without this commit (10 at 5b15137; np.asarray.cs, np.all.cs and np.any.cs joined with daa5882, another session's commit). BASE Written and measured on 5b15137; moved onto 8aba70c (three commits of other sessions: daa5882 fix(matrix), ebcc245 and 8aba70c for examples/Pythonic) and the unit suite and the oracle replay run again there. DOCUMENTATION .claude/CLAUDE.md, "The comparison engine": the two forms of a rows iteration, "Tiles", "A matrix against one row", the kernel, the gates and the mutation-script trap, the benchmark lines, the floors, the clock note; the Core Source Files row.
…on that receives a numpy.matrix through its NDArray parameter returns NumPy's class and shape (np.sum(m, 0) is a 1 x N matrix, np.negative(m) and np.linalg.inv(m) are matrices, np.ravel(m) a plain array, np.stack((m, m)) raises), while the operators, the indexers and the NDArray members keep the plain-array meaning; 363 rule statements and 203 plain-array regions, a 4,645-case oracle tier, and an audit that replays 311,725 corpus cases with matrix operands and finds no library method that reaches a matrix rule
WHAT CHANGED FOR A CALLER
Before, the matrix semantics followed the static type everywhere: np.sum(m, 0) of an NDMatrix returned a plain
(N,) array, where NumPy returns a 1 x N matrix. Now a public np.* function looks at the OBJECT. The result is an
NDMatrix typed NDArray; a cast gives the matrix members back, no copy is made.
NDMatrix m = np.matrix("1 2 3; 4 5 6");
NDArray a = m; // the same object
np.sum(a, 0) // matrix([[5, 7, 9]]) was array([5, 7, 9])
np.negative(a), np.transpose(a) // matrices were plain arrays
np.ravel(a), np.where(a > 2, a, 0) // plain arrays (NumPy's asarray), unchanged
np.stack(new[] { a, a }) // ValueError: shape too large to be a matrix.
np.sum(a, 0, keepdims: true) // TypeError: matrix.sum() got an unexpected keyword argument 'keepdims'
a * a, a[0], a.sum(0) // operators, indexers, members: plain, unchanged
BREAKING: code that passed an NDMatrix to an np.* function and relied on the plain result now gets NumPy's
matrix result (a different shape for reductions and 1-d results). Pass m.A for the plain answer.
THE DESIGN (src/NumSharp.Core/Matrixlib/NDMatrixBoundary.cs)
NumPy decides per function, by three mechanisms, and each has a rule here:
* Wrapped __array_wrap__ or allocation through the operand's class: ufuncs, np.transpose, np.concatenate,
np.dot, np.linalg.inv. A 1-d result becomes a 1 x N row; more than two long axes raise.
* Reduced the fromnumeric wrappers call the operand's METHOD: a scalar without an axis, a row for axis 0, a
column for axis 1; keepdims: true raises the matrix method's TypeError.
* Plain asarray inside: np.cross, np.einsum, np.linalg.norm, every np.fft function.
* WrappedAll / WrappedPair / WrappedTriple / WrappedFirstTwoOfFour for several results (np.split, np.unstack,
np.divmod, np.frexp, np.modf, np.linalg.qr / eig / eigh / svd / lstsq), and WrappedIf for np.block, whose
class follows NumPy's ROUTE (repeated concatenation: a matrix; the single-allocation route past 2*512*512
elements: a new ndarray).
One statement per public overload:
if (NDMatrixBoundary.Sees(a))
return NDMatrixBoundary.Wrapped((a, axis), static s => flip(s.a, s.axis));
Sees is true only for a matrix object at depth zero. A rule raises a per-thread depth counter around a SECOND call
of the function, so that call and everything below it compute with plain-array semantics; the rule then shapes the
result. The arguments travel in a tuple to a static lambda: a capturing lambda would allocate a closure at method
entry on every call, plain arrays included.
A function WITHOUT a rule opens a region (using var plain = NDMatrixBoundary.PlainFor(a, ...)): its result is the
plain array NumPy's asarray-based functions return, and the safe answer where NumPy's is not implemented. An
object that KEEPS an array for later calls stores the plain view (PlainOf): numpy.ma (the masked array's data and
mask, AsData, MaskOperand) and the samplers' converted parameters (RandomParam.Convert). The NDArray members that
are implemented by np.* functions, the NDMatrix members and the engine methods that call back up into np.* open
regions too (PlainFor(this) / Inside()).
Facts NumPy's source decides, each probed on 2.4.2: np.linalg.solve and lstsq take their class from b; np.clip
from the array OR a bound; np.iscomplex is a matrix for a complex matrix and a plain array otherwise; np.r_ /
np.c_ / np.block cannot give a matrix entry a third axis (np.block([[[m]]]) raises "axis -3 is out of bounds for
array of dimension 2", np.r_['0,3', m, m] is a 4 x 3 matrix); np.average over a tuple of axes keeps every reduced
axis; np.apply_along_axis and np.piecewise return a matrix; np.real of a complex matrix is a matrix over the real
lane; power(in, in) and reshape(ref Shape) carry the rule of their by-value overloads.
THE CONTRACT IS MEASURED
No method of the library may reach the matrix rule of a public function at depth zero: library code is written
for plain results. FuzzCorpusTests.MatrixOperand_LibraryNeverReachesAMatrixRule turns that into a measurement. It
replays every corpus case that has a two-dimensional operand (311,725: the ordinary tiers, the numpy.ma tiers, the
index tiers) with those operands typed as matrices while NDMatrixBoundary.AuditInternalCallers is on, and requires
the set of library callers to stay empty. The audit reads the stack by NAME (the public function's frame can be
inlined). It found 57 call paths when it was written:
* NDArray members and the ~ operator that call np.* with `this`;
* engine methods that call back up (IsClose, NonZero / FlatNonZero / Argwhere, TryClipFlat, the bool add loop,
the variance helpers, Vdot, np.evaluate's arg stream, the float16 sum fallback);
* public functions without a rule (np.place, np.putmask, np.block, np.r_, np.bmat, np.iscomplex, fftshift,
hfft, the average with several axes);
* numpy.polynomial (mapdomain, the array arithmetic of PolyNumber, the calculus scale) and numpy.random (the
ufunc steps of uniform, hypergeometric and negative_binomial).
Each is closed by a region, by a rule for the calling function, or by PlainOf.
THE ORACLE TIER (test/oracle/gen_matrix_operand_oracle.py)
Existing corpus cases that a plain NumPy call reproduces byte for byte are re-emitted with one two-dimensional
operand typed as a matrix, and NumPy's answer is recomputed: class, dtype, shape, values, or the error type and
text. 4,645 answers over 260 functions, 680 of them errors.
matrix_operand.jsonl 3,885 + matrix_operand_host.jsonl 420 (RunHostLibmCorpus)
gaps/matrix_operand_gaps.jsonl 330 + gaps/matrix_operand_host_gaps.jsonl 10, carved exactly (every carved case
diverges, no kept case does):
MO01 24 np.isclose returns NDArray<bool>, which cannot be a matrix object
MO02 32 np.count_nonzero with an axis: NumPy hands keepdims to matrix.sum and raises
MO03 211 the median / percentile / quantile family: NumPy's keepdims and ravel paths meet matrix methods
MO04 46 np.roll, np.ediff1d, np.vdot, np.union1d, a refused np.append: ravel() of a matrix stays 1 x N
MO05 27 np.polyval, np.i0, narrow-float np.sinc, np.trapezoid with a matrix x: `*` is the matrix product
MO02 to MO05 are NumPy artifacts (matrix semantics leak into NumPy's own Python code); NumSharp returns the
plain result shaped by the function's rule.
The class is compared without a registry call. MisalignedRegistry returns null for a shape, threw, error-text or
expected-error divergence of a case whose layout starts with matrix@, and the two NEP50 0-d blankets skip the
tier. Operator keys replay through the function form (OpRegistry.MatrixOperand.cs).
Functions outside the truth table (tuple results, nested operands, non-deterministic arrangements, LAPACK) took
their rule from NumPy's source and two direct probes; they are pinned by unit tests with the probed class and
shape.
LIMITS ([Misaligned], pinned)
* a typed return cannot carry the class (np.isclose; the typed np.all / np.any overloads);
* np.gradient and np.meshgrid return plain arrays in their result structs (MO06 / MO07, OpenBugs.MatrixOperand.cs);
* numpy.ma has no masked_matrix (NumPy's ma.array(m).sum(axis=0) is 1 x N; here the plain (N,));
* the numpy.polynomial package computes plain (NumPy's mapdomain of a matrix is a matrix);
* a result is always a matrix VIEW of the plain result: no flags.owndata parity where NumPy allocates the
subclass;
* artifacts not reproduced: np.linalg.vector_norm(m) (an array of |x| in NumPy), np.average(m, axis=1,
weights=ndarray) (an N x N matrix in NumPy), np.apply_along_axis when func1d depends on the slice's rank
(NumPy hands it a 1 x N matrix), np.lexsort(m).
TESTS
NumSharp.Tests net10.0 18,505 passed / net8.0 18,504 passed (TestCategory!=OpenBugs&!=HighMemory)
NumSharp.Tests.Oracle net10.0 283 passed / net8.0 283 passed
NumSharp.Tests.Interop net10.0 741 passed (OpenBLAS backend)
new: Matrixlib/NDMatrixBoundary.Tests.cs (24), OpenBugs.MatrixOperand.cs (2 open gaps),
NumSharp.Tests.Interop/MatrixOperandLapackTests.cs (5), FuzzCorpusTests.MatrixOperand.cs (2 tiers, 2 gap
replays, the audit)
NDMatrix.Tests: Semantics_FollowTheStaticType became
Semantics_OperatorsAndMembersFollowTheStaticType_FunctionsFollowTheObject.
One run of the net10.0 unit suite failed AuditV2_LogicShapeStorage.T2_NonZero_Is_Much_Slower_Than_NumPy (108 ms
against a 32 ms wall-clock limit while other builds ran); it passes 3 of 3 times alone and in every later run.
PERF (two pinned P-cores, best of 15 batches, a host with other load: runs differ by about 3 %)
Plain arrays pay a type test per call. 44 cells (22 functions at 4 and 1,024 float64 elements), the tree before
against this one, each measured twice: geomean new / old 1.02, inside the noise but for np.asarray (2.1 -> 2.8
ns) and np.diagonal (176 -> 187 ns: two regions, each a try / finally).
A matrix operand pays the rule, about 270 to 350 ns a call (the scope that reclaims the hidden plain result, the
second call, the matrix view): np.add of 2 x 2 matrices 681 ns (plain arrays 413 ns, the NDMatrix operator 687
ns). Against NumPy's own matrix calls, 32 cells: geomean 3.0x, min 1.1x (m.sum(0) at 2 x 2 and the managed
np.dot at 32 x 32); the np.* cells 2.2x to 12x.
TRAPS FOUND
* NumPy 2.4.2's np.stack / np.dstack with a matrix operand raise and CORRUPT NumPy's heap (a later call dies in
PyDataMem_UserFREE). The generator makes every matrix call in a worker process and retires it after such an
error.
* CheckError reports "NumPy raises but NumSharp produced a result" as a VALUE divergence, so a source tier's
value excuse hid error-parity gaps; expected-error cases are strict for this tier.
* A function with a rule that is called by a function with a REGION loses its rule (the depth is raised). A
region therefore goes only on a function that has no rule on any overload.
* System.Collections.IList of a rectangular CLR array throws on a single index: the nested-operand scan
(np.block, np.r_) reads only one-dimensional arrays of references.
* An indexer's [CallerMemberName] is "Item" while its stack frame is get_Item: np.r_ passes the name.
* The benchmark project (benchmark/NumSharp.Benchmark.CSharp) does not build: CS0121 on np.amin / np.amax /
np.min / np.max in MinMaxBenchmarks.cs. It fails the same way on the parent commit; not touched here.
FILES
src/NumSharp.Core/Matrixlib/NDMatrixBoundary.cs (new); one statement in 275 source files of NumSharp.Core;
Matrixlib/NDMatrix.cs and NDMatrix.Tests.cs documentation; .claude/CLAUDE.md (the np.matrix section, the tier,
the source-file row, the OpenBugs list); test/NumSharp.Tests.Oracle/Fuzz/README.md (the tier's section).
…try under New APIs (construction, T / H / I / A / A1, the matrix product and power, the two-axis reductions, np.* functions that return NumPy's class and shape for a matrix operand) and three breaking-change lines (np.asmatrix / np.bmat return NDMatrix, the index-array assignment rule, the boolean-mask assignment rule) The release notes were swept at 7310f3f, before the numpy.matrix work. Its three commits (201e9c7 the class, daa5882 the explicit 0-d array, 51bd91f the np.* functions) had no line. New APIs & Modules: one feature bullet with a sub-list, the hashes on the parent line (the sub-bullets are facets of one feature, CHANGELOG_STYLE section 6). The oracle counts are the two tiers': 26,761 cases for the class (matrix.jsonl) and 4,645 NumPy answers for the functions with a matrix operand (matrix_operand*.jsonl with the carved gaps). Breaking Changes, each with old -> new and the migration: * np.asmatrix / np.bmat return NDMatrix (was a plain 2-D NDArray): `*` is the matrix product, an index keeps two axes, and np.* functions return NumPy's matrix results for it (np.sum(m, 0) has shape (1, N)); `.A` gives the plain array. Note for a reader of the earlier plan: typing the variable NDArray keeps the operators and members plain, but since 51bd91f it does not change what an np.* function returns, so the line names `.A` only. * An index-array assignment follows NumPy's value rule: a value that does not fit raises ValueError with NumPy's "shape mismatch" text before anything is written (was IncorrectShapeException), and a (1, N) value fits an (N,) selection (it threw). * A boolean-mask assignment takes a 0-d or 1-d value: a 2-d value raises NumPy's TypeError, a 1-d value whose length is neither 1 nor the number of true elements NumPy's ValueError. Not added: a Testing & Tooling line (the tiers are named in the feature bullet; the section's corpus total is a sweep-time figure and was left as swept), and a line for `m * np.array(2)` (a fix inside a feature that ships for the first time in this release, folded as a hash).
…heir matrix rule adds - TypeError for keepdims with a numpy.matrix operand (46 overloads with a Reduced rule) and ValueError "shape too large to be a matrix." for a result that cannot be a matrix (256 overloads with a Wrapped rule) 51bd91f gave the public np.* functions NumPy's rule for a numpy.matrix operand. Two of those rules can raise where the function did not raise before, and the XML documentation of the overloads did not say so: * a Reduced rule (np.sum, np.prod, np.mean, np.std, np.var, np.any, np.all, np.max, np.min, np.average, the nan twins) raises TypeError for keepdims: true, because NumPy calls the matrix METHOD, which has no such keyword ("matrix.sum() got an unexpected keyword argument 'keepdims'"); * a Wrapped rule (the ufuncs, the shape functions, np.concatenate, np.dot, np.linalg.inv ...) raises ValueError when the result has more than two axes longer than 1 ("shape too large to be a matrix."), for example np.stack((m, m)) or a ufunc of a matrix with a 3-d array. 302 <exception> lines in 142 files, each added after the last line of the overload's existing /// block. The lines were placed by a script that reads which rule the overload's body applies: * Reduced with a method name, a KeywordError throw or np.average's matrix helper -> the TypeError line; a Reduced rule whose method is null (the argmax family, the median family, np.trapezoid) accepts keepdims and gets no line; * Wrapped / WrappedAll / WrappedPair / WrappedTriple / WrappedFirstTwoOfFour / WrappedIf -> the ValueError line. Not documented: 4 overloads that have a rule and no /// block at all (np.mod, np.remainder, np.fmod with an object operand in Math/np.math.cs, and one np.ptp overload). A lone <exception> line would be their whole documentation; they need a full block, which is a separate change. Comments only: no code changed, NumSharp.Core builds for net8.0 and net10.0.
…and 39 pairs chosen by GitHub measurements (Matplotlib, SciPy and scikit-learn confirmed; Pillow, OpenCV and TensorFlow new), and 52 planned rules for the C# forms that the plan needs, each tried in a scratch port against a Python program on Windows and on Ubuntu, or marked open
The owner's review of the plan that came with version 2 (five more pairs: scikit-learn model selection
with the pipeline as Python text, scipy.optimize, scipy.stats, a Matplotlib line chart, a Matplotlib
histogram and scatter chart): "These are not great or enough, figure more, also use github advanced
search to find separate top 3 python libraries". This commit is the answer, in the specification. It
changes two documents and no code. No pair is implemented: the owner reviews the plan first.
WHAT IS NEW IN examples/Pythonic/PYTHONIC_SPEC.md (562 -> about 1,300 lines)
- Section 13, "The pairs". Nine rules for a pair (PAIR-1 to PAIR-9: one question with real data, a
feature that many programs use, NumSharp does a part of the work, the pair prints its proof, no
person / display / network, the same lines on each run, short, it states what it adds, plain Python).
The table of the six libraries. One table of pairs for each library: scikit-learn 7 (SK-1 is the pair
that exists), SciPy 11, Matplotlib 5, Pillow 5, OpenCV 6, TensorFlow with Keras 5. Each row has the
feature with its counts, the program as a question with its data, and what the pair adds to the
approach. Then the order of the work, a comparison with the plan of five pairs, and what the plan
needs from the project.
- Section 14, "The planned rules" (it replaces "Not specified yet"). PLAN-1 to PLAN-52: the statements
of a port (literals, loops, slices, operators, builtins, with, lambdas, the standard modules), the
parameters of a member (overloads, an optional positional parameter, a keyword of several forms, text,
functions, a tuple of one item), the results (a table of 14 kinds), the objects (attributes,
iterators, callable objects, objects and constants of a module, a module and its class under one
name, an object that NumPy converts, operators and indexes, one name for different classes, lists and
grids of objects, a function that writes into the array of the caller), and arrays / print / backend /
lifetime. Each has a state: "tried (C-n)" with the round of Appendix C, or "open". Section 14.6 has
nine open questions and a table of the rules of sections 3 to 12 that change when a planned rule
becomes a rule.
- Appendix B, "The measurements of GitHub": the method (code search for an import phrase or a name,
the dependents page, the stars, the topic search for the pool of candidates), the limits of the
counts, the table of 36 libraries, the ten topics, the counts of more than 200 names of the six
libraries, and the packages.
- Appendix C, "The scratch ports": six rounds (C-1 to C-6) and 16 findings (C-7 to C-22).
- Section 15 is the checklists (it was 14), with the steps for the plan and for a scratch port.
- The header table: version 3, the plan, the planned rules, the history. Sections 1 to 12 keep each
rule. Two sentences of PORT-10 now point to section 14.
README.md of the example: three sentences that point to the plan.
THE MEASUREMENT (2026-10-10, gh api, the legacy code search: "phrase" language:Python -> total_count)
Python files that import the library / repositories that depend on it:
numpy 47.4 M / 3.56 M, pandas 18.2 M / 2.79 M, torch 18.0 M / 0.85 M (not in the plan: NumSharp is
the NumPy of a port; pandas and PyTorch have their own conversions in the interop package)
matplotlib 9.88 M / 1.86 M, scipy 6.72 M / 1.42 M, sklearn 6.08 M / 1.35 M (the three of version 2:
confirmed)
PIL 3.82 M / 2.46 M, cv2 3.75 M / not available, tensorflow 2.97 M / 0.54 M (the separate three)
then transformers 2.95 M (PyTorch, downloads models), sympy 1.30 M, networkx 1.00 M, seaborn 0.92 M,
plotly 0.85 M, keras 0.80 M (inside TensorFlow), skimage 0.38 M, statsmodels 0.31 M, xgboost 0.17 M.
requests, tqdm, PyYAML and Beautiful Soup have high counts and exchange no arrays.
The features of a library are ranked the same way, for example scipy.stats 1.64 M, scipy.sparse 1.47 M,
scipy.optimize 1.31 M; plt.figure 3.41 M, plt.subplots 3.22 M; Image.open 2.61 M, Image.fromarray
1.01 M; cv2.cvtColor 1.48 M, cv2.imread 1.16 M; keras.layers 0.97 M, tf.keras 0.73 M. The five modules
of scikit-learn with the highest counts (metrics, model_selection, preprocessing, datasets,
linear_model) are all in the pair that exists.
Limits that the appendix states: the search cuts a phrase at each point and does not tell capital
letters; a name of several libraries counts for all (marked); a name with a variable in it is a lower
bound (marked); the table is not a full list of GitHub, only of the candidates that were named. After
about 150 requests GitHub answered about one request in one minute.
WHAT WAS TRIED (scratch ports: a copy of the project with more tree members, a port, and a Python
program with the same statements; "tried" = the same lines and exit code 0; not in the repository)
C-1 scikit-learn, SciPy 18 lines literals, attributes, dict results, an iterator, with,
C# functions that curve_fit and minimize call, named results
C-2 scikit-learn, SciPy 27 lines one name for two classes, None, range / enumerate / zip, a
callable object, an object at module level, a lambda for quad,
operators and an index on a sparse matrix, output=
C-3 Matplotlib, Pillow, cv2 17 lines the canvas as an NDArray view, hist, a grid of charts, a photo
to an array and back, OpenCV that draws into NumSharp memory
C-4 none 4 lines a line that Python prints by itself
C-5 Matplotlib, cv2 20 lines random numbers, computed slices, rows, @ and **, sorted,
a comprehension, bytes, an error by its type
C-6 TensorFlow 2.21, Keras 17 lines a Keras model, GradientTape, model.summary()
Rounds C-1, C-2, C-3 and C-5 also ran on Ubuntu 24.04 (WSL 2) with the same lines as on Windows.
Two Python scripts made the main calls of the 34 pairs that the rounds do not cover.
FINDINGS THAT CHANGED THE PLAN OR THE RULES
- A line that a Python library prints by itself is LOST when the output of the process goes to a file
or a pipe (2 of 4 lines, on Windows and on Ubuntu). The backend must send sys.stdout to the console
of .NET (PLAN-47) before the first pair whose library prints (model.summary(), verbose=1).
- OpenCV 5.0 (opencv-python-headless 5.0.0.93) has no cv2.CascadeClassifier and no detector files. The
face detector pair became a QR code and a template search (CV-6). OpenCV 4.14.0.94 still has it.
- A C# function that Python calls needs three lifetime rules (PLAN-49): the result goes back through
a Python list (27 arrays stayed exported with a return value), each call has a plain NDScope (15 s
-> 3 s), and a .NET exception stays out of Python (1 array stayed exported).
- A C# array converts to an NDArray by itself, so len(names) is CS0121 without a T[] overload.
- Two overloads for one optional positional argument bind plot(x, y, "r--") to the keyword `label`
without an error. One method with the optional parameter at Python's position does not.
- n / 2 with two C# integers truncates, Math.Round(2.675, 2) is 2.68 where Python gives 2.67, and
np.array([1, 2]) is int32 where NumPy gives int64.
- Environment.SetEnvironmentVariable does not reach TensorFlow; os.environ through Python does.
- np.random.seed(0) gives the same normal / uniform / randint / permutation numbers in NumSharp 0.70.0
and NumPy 2.4.2, so a pair can make its own noise.
- TensorFlow loads into the .NET process in 2.2 s and gives the same numbers after set_random_seed(0);
its environment has 1.9 GB (445 MB with the other five libraries).
- Matplotlib copies the data of a line: a chart that is not closed does not fail the lifetime check.
NOT VERIFIED
- No pair of the plan is implemented, and no planned rule is in the reference implementation
(examples/Pythonic at ebcc245). "Tried" is less than "verified": the scratch members have no
documentation and no review.
- The round with TensorFlow ran on Windows only. Nothing ran on macOS.
- Open, in section 14.6: TensorFlow in the plan or not; an error by its Python type (the exact-name
filter misses subclasses; proposal: NumSharp's ValueError / TypeError classes); with ... as name when
__enter__ returns another value; setters; integers (int / long); a name that needs a type; keywords
by position; a result that can be None; BigInteger; an object whose class Python decides at run time.
- The skill file .claude/skills/pythonic/SKILL.md still describes version 1 (the owner's decision).
…ithout a lock - GetCastingImpl 24 -> 10 ns on one thread and 229-315 -> 7-10 ns per call with 8 threads querying at once; the within-dtype slot is claimed by one compare-exchange, because its null check + store let two racing registrations both succeed
What changed (src/NumSharp.Core/DTypes/DTypeMeta.cs)
- `_castingImpls` was a Dictionary<DTypeMeta, CastingImpl> behind `lock (_castingImpls)` in both GetCastingImpl and
AddCastingImpl. It is now a System.Collections.Concurrent.ConcurrentDictionary (the house type: 83 Core files use
it; NumSharp's vendored NumSharp.Collections.Concurrent copy is only the base of the ordered collections).
- GetCastingImpl reads it with TryGetValue, which takes none of the dictionary's locks, and reads the within-dtype
slot with Volatile.Read. WithinDTypeCastingImpl reads the slot the same way.
- AddCastingImpl stores with TryAdd: the ContainsKey + Add pair as one atomic step, the same
InvalidOperationException text when the pair is taken ("A cast from <class ...> to <class ...> is already
registered."). The within-dtype slot is claimed by Interlocked.CompareExchange(ref slot, impl, null), same text
("A cast within <class ...> is already registered.").
- Constructor arguments `(concurrencyLevel: 1, capacity: 31)`: writes are rare (the registry's static constructor
and the public DTypeRegistry.AddCastingImpl), so one write lock is enough, and reads never take it; capacity 31
(37 buckets after the prime round-up, and a per-lock budget of 37) holds every class's registrations (16 at most
today: 14 numeric + 2 datetime per storage class, 16 for each datetime class) without a resize. The default constructor allocates one lock object per CPU for each of the 21 classes, and
on this 32-thread host (37 buckets, budget 1 per lock) it would also grow the table and the lock array during the
registry's static constructor (checked against the vendored clone of the BCL source,
Collections/Concurrent/ConcurrentDictionary.cs: DefaultConcurrencyLevel = ProcessorCount, growLockArray: true,
GrowTable).
- Keys keep the default comparer: no DTypeMeta subclass overrides Equals/GetHashCode, so a key is the class object
itself, as in NumPy's castingimpls dict.
Why
- GetCastingImpl is on the path of every np.can_cast over descriptors (DTypeCasting.CheckCastSafety / GetCastInfo /
CanCastTypeTo), of the DType < <= > >= operators, and of DTypePromotion.CastDescrToDType whenever the common class
is parametric (datetime64 / timedelta64). The lock made every such query from every thread take the same monitor.
A race this fixes
- The within-dtype branch was `if (slot != null) throw; slot = impl;` with no lock. Two racing registrations of a
class's within-dtype cast could both pass the check: both returned, the later replacing the earlier. Restoring that
code as a mutant fails the new race test in 8 of 8 runs.
Measured (A/B, old = f608647 vs this change, two separate builds in detached worktrees; dotnet run -c Release,
P-cores (affinity 0xFFFF), DOTNET_TC_CallCountingDelayMs=0, best of 5-7 rounds; two runs each; ns per call; with
threads: wall time per call of one thread while the others run):
old run 1 / run 2 new run 1 / run 2
GetCastingImpl 1 thread 24.25 / 24.14 10.08 / 10.31
CanCastTypeTo 1 thread 37.81 / 39.08 22.47 / 28.46
GetCastingImpl 4 threads 110.03 / 119.64 11.58 / 11.01
CanCastTypeTo 4 threads 115.35 / 124.25 26.26 / 29.68
GetCastingImpl 8 threads 228.90 / 315.14 9.72 / 6.96
CanCastTypeTo 8 threads 233.50 / 302.15 36.30 / 31.16
(256 fixed cross-class builtin pairs; CanCastTypeTo asks NPY_SAFE_CASTING.)
Tests (test/NumSharp.Tests/DTypes/DTypeMetaCastingImplConcurrencyTests.cs, 4 tests, ~260 ms together)
- Every class there is a private probe never added to DTypeRegistry's lookup tables, so the builtin classes' cast
tables, which every other test shares, are untouched.
- Registration_Contract: an empty table answers null; a cast is found on its source class only; a second
registration of the pair, or of the within-dtype cast, raises with the exact text and leaves the first in place;
GetCastingImpl(null) / DTypeRegistry.AddCastingImpl(null) raise ArgumentNullException; a cast filed on the wrong
class raises ArgumentException naming "impl".
- SamePair_RegisteredByRacingThreads_ExactlyOneWins / WithinDType_RegisteredByRacingThreads_ExactlyOneWins: 1,500
rounds, min(ProcessorCount, 8) threads released together by a Barrier, each registering its own cast for that
round's fresh pair: exactly one returns, every other raises the exact text, the pair resolves to the one that
returned.
- Lookups_DuringRegistrationAndGrowth_SeeNullOrTheRegisteredCast: one writer registers 48 x 160 casts (160 per class,
past the 37 initial buckets - capacity 31 rounded up to the next prime of the BCL table - so the table grows twice
per class under the readers, at the 38th and the 90th registration) while up to 4 readers sweep the class being
filled: every lookup is null or exactly the registered cast, a cast once seen never disappears, an unregistered pair
stays null, nothing throws.
- Threads are joined with a 1-minute timeout, so a hang fails instead of stalling the run.
Mutation check (detached worktree, one planted edit at a time, 8 runs of the class each; baseline 8/8 clean before
and after the campaign): 5 of 5 killed.
- M1 within-dtype slot: null check + plain store (the old code): WithinDType race test 8/8.
- M2 table: indexer overwrite instead of TryAdd: Registration_Contract + SamePair 8/8.
- M3 table: plain Dictionary with no lock at all: SamePair 8/8, Lookups 5/8.
- M4 table: ContainsKey, then TryAdd with its result ignored: SamePair 8/8.
- M5 within-dtype lookup reads the table instead of the slot: Registration_Contract + WithinDType 8/8.
Verification
- NumSharp.Tests, CI filter (TestCategory!=OpenBugs&TestCategory!=HighMemory): net10.0 18,517 passed, 11 skipped,
0 failed; net8.0 18,516 passed, 11 skipped, 0 failed.
- After the last edits (doc text only): the DTypes namespace 269/269 on net10.0 and net8.0.
- Crefs: Core built in Release with the csproj NoWarn replaced (-p:NoWarn=NU1603;NU1902 -p:WarningLevel=4
--no-incremental) on net10.0 and net8.0: 0 warnings in DTypeMeta.cs. `Volatile.Read{T}(ref T)` does not resolve
(CS1574) on .NET 8+, whose parameter is `ref readonly T`; the arity-only `Volatile.Read{T}` does (one generic
overload).
Found, not changed here (pre-existing; the old lock never covered it)
- BuiltinCastingImpl.Casting fills `NPY_CASTING? _casting` lazily with `_casting ??= ComputeCasting(...)`. Roslyn
compiles that to GetValueOrDefault() on the field, THEN HasValue on the field (two loads, value first), and the
JIT stores the computed Nullable as one 8-byte write. A reader that loads the value before another thread's store
and HasValue after it returns NPY_NO_CASTING (0), and CheckCastSafety then answers true for every casting rule
(MinCastSafety(NO, rule) == rule). Probe: two threads read .Casting of a fresh float64 -> int32 cast at the same
moment, 1,000,000 rounds: 32 of 2,000,000 reads returned NPY_NO_CASTING with tiering on, 46 of 2,000,000 at tier 0
(DOTNET_TC_CallCounting=0); its true value is NPY_UNSAFE_CASTING. In the library each builtin cast computes its
value once per process, so only threads racing that first computation can see it. The fix is an int field with a
-1 sentinel read once with Volatile.Read; left for a separate change.
Merge note: branch `strings` edits DTypeMeta.cs too (the class remarks and new members ahead of the "Casting
implementations" section); those hunks do not overlap these.
…n broke point at their headings again - a heading "A — B" has the id "a--b" and "A - B" has "a---b", and the links kept "a--b" (DocFX InvalidBookmark: NDIter.md 14, numsharp-build-compiler.md 6, iterating-and-enumerating.md 3, NDArray.md 2, getting-and-setting-values.md 2) ## What broke 538dd00 ("docs: normalize em-dashes to hyphens across docfx content") rewrote headings such as "### Typed iteration — unboxed elements and Span chunks" to "### Typed iteration - unboxed elements and Span chunks". Markdig's automatic heading id drops the em-dash but keeps a hyphen, so the id gains one hyphen: `typed-iteration--unboxed-elements-and-span-chunks` became `typed-iteration---unboxed-elements-and-span-chunks`. The links to those headings kept the old ids, and DocFX 2.81.0 (the version docs.yml installs) reports each one as InvalidBookmark. Master's build has none of these warnings: all 27 came with the em-dash commit. ## The fix Each broken fragment is replaced by the id DocFX generates for its heading now, read from the built HTML of the target page: the one id whose hyphen runs collapse to the same text as the old fragment. Each of the 27 had exactly one such id. Two keep their inner double hyphens, because only the em-dash separator changed: - `net-interop--spant--memoryt--arrays--ienumerablet` -> `net-interop---spant--memoryt--arrays--ienumerablet` - `leak-detection--ndw012--ndw013` -> `leak-detection---ndw012--ndw013` Only a fragment that ends a link target is replaced, on the line DocFX named, and every other byte of the files is kept (LF line endings included). ## Files (27 links on 26 lines) - docs/website-src/docs/NDIter.md: 14 (its contents list at lines 13-53, and lines 89, 136, 1134, 1663) - docs/website-src/docs/numsharp-build-compiler.md: 6 (`#inheritance--...` five times, `#leak-detection--...` once) - docs/website-src/docs/iterating-and-enumerating.md: 3 (two on line 63) - docs/website-src/docs/NDArray.md: 2 - docs/website-src/docs/getting-and-setting-values.md: 2 ## Verification DocFX 2.81.0 over the docs-only branch journey4-docfx (master's source for the API reference, the data branch's latest snapshots, rendered as docs.yml renders them), with this commit and the two that follow it: 391 warnings against 440 before and 0 new ones. The 27 InvalidBookmark warnings are gone; the other 22 removed warnings are the next commit's InvalidFileLink ones.
…e dashboards' Full Reports lists open the data branch's published reports again - the code/data split (c423450) moved docfx.json's sources to ../../refs/data/<type>/latest but left 20 links on the old source paths (../../../benchmark/history/latest/*.md, ../../../benchmark/coverage/generated/summary.md, ../../../test/inventory/generated/*), which DocFX cannot resolve and ships as raw links that 404 on scisharp.github.io; NDIter.md's unified-report URL and two IterationBenchmarks.cs links are fixed with them (DocFX InvalidFileLink 22 -> 0) ## How these links worked DocFX resolves a link by the SOURCE path of its target and writes the target's OUTPUT path. Before the split, docfx.json published `../../benchmark/history/latest/*.md` as `docs/reports/`, `../../benchmark/coverage/generated/summary.md` as `docs/reports/coverage/`, `../../test/inventory/generated/summary.md` as `docs/reports/tests-oracle/` and the inventory CSV as `docs/data/`. The TOC and the two dashboards linked those source paths, so the site showed `reports/MANIFEST.html` and so on. ## What broke c423450 ("build(data): split code from data") moved master's generated data to the `data` branch. docs.yml clones it into refs/data, and docfx.json's sources became `../../refs/data/<type>/latest`. The 20 links kept the old source paths, which no longer exist. DocFX reports each one as InvalidFileLink (master's build has these warnings) and writes it unchanged, so it points above the site root. Measured on 2026-10-14: the live toc.json still carries `../../../benchmark/history/latest/MANIFEST.md`, which from /NumSharp/docs/ lands on https://scisharp.github.io/benchmark/history/latest/MANIFEST.md (404), while https://scisharp.github.io/NumSharp/docs/reports/MANIFEST.html (the page the data branch's MANIFEST.md becomes) returns 200. Every TOC entry under "Raw Reports" and "Generated Summary" and every link in the two "Full Reports" lists was dead. ## The fix Each link names the same file in the data checkout, so DocFX resolves it to the published output again: | Old link (repo path) | New link (data branch source) | Published as | |---|---|---| | ../../../benchmark/history/latest/MANIFEST.md | ../../../refs/data/benchmark/latest/MANIFEST.md | docs/reports/MANIFEST.html | | ../../../benchmark/history/latest/benchmark-report.md | ../../../refs/data/benchmark/latest/benchmark-report.md | docs/reports/benchmark-report.md (raw resource, as before the split) | | ../../../benchmark/history/latest/{cast,fusion,layout,nditer,operand,openblas}_results.md | ../../../refs/data/benchmark/latest/... | docs/reports/*_results.html | | ../../../benchmark/coverage/generated/summary.md | ../../../refs/data/benchmark-coverage/latest/summary.md | docs/reports/coverage/summary.html | | ../../../test/inventory/generated/summary.md | ../../../refs/data/tests-oracle/latest/summary.md | docs/reports/tests-oracle/summary.html | | ../../../test/inventory/generated/tests-oracle-report.csv | ../../../refs/data/tests-oracle/latest/tests-oracle-report.csv | docs/data/tests-oracle-report.csv | A link to the output path (`reports/MANIFEST.md`) does not work: DocFX looks it up as a source file and reports `Invalid file link:(~/docs/reports/MANIFEST.md)` (tried first, 18 such warnings). Fixed with them: - NDIter.md linked https://scisharp.github.io/NumSharp/docs/reports/benchmark-report.html, which never existed: the unified report is a raw resource, and docs.yml renders its HTML form as docs/benchmark-matrix.md. The link now opens https://scisharp.github.io/NumSharp/docs/benchmark-matrix.html (200 on the live site). - iterating-and-enumerating.md linked `IterationBenchmarks.cs` by repository path (twice), which the site cannot serve (InvalidFileLink). It now opens the file on GitHub's master branch, where it exists. ## Files (23 links) - docs/website-src/docs/toc.yml: 10 (Generated Summary + the 9 Raw Reports entries) - docs/website-src/docs/benchmarks-dashboard.md: 8 (Full Reports) - docs/website-src/docs/tests-oracle-dashboard.md: 2 (Full Reports) - docs/website-src/docs/NDIter.md: 1 - docs/website-src/docs/iterating-and-enumerating.md: 2 ## Verification DocFX 2.81.0 over the docs-only branch journey4-docfx, with refs/data holding origin/data's tip as docs.yml clones it: the 22 InvalidFileLink warnings are gone and no warning is new (391 warnings with the commits around this one, 440 before). In the built site, toc.json's entries are `reports/tests-oracle/summary.html`, `reports/MANIFEST.html`, `reports/benchmark-report.md`, `reports/coverage/summary.html` and `reports/{cast,fusion,layout,nditer,operand,openblas}_results.html`; the dashboards' lists resolve the same way plus `data/tests-oracle-report.csv`; every target exists in the built site and on the live site. benchmarks-dashboard.md's own text test (benchmark/scripts/tests/test_o1_exclusions.py) passes.
…ventory that carries no oracle evidence - master's CI publishes the inventory its own generator (1.6.0) writes to the data branch, whose rows have no `oracle` record, and the page read that as "no contracts" for every API: an "Oracle-verified parity 0.0% (0 of 560)" card, "0 oracle-verified" on every surface card and tooltip, an "Available, no oracle contracts" filter, and a "no committed NumPy-oracle contract exercises this API" block on every available API; journey4's inventory (generator 1.10.0) renders as before ## Why the page meets such an artifact The site renders whatever `refs/data/inventory/latest/coverage.json` holds, and docs.yml publishes that file on every push to master from master's own `coverage/generate_coverage.py`. The oracle join (`coverage/oracle_evidence.py`, generator 1.10.0) exists on journey4 only, so a docs change that reaches master before journey4's coverage tooling meets master's 1.6.0 inventory: 834 rows, none with an `oracle` record, no `oracle_verified` in `summary.default_scope`. The page already treated such an artifact as older (the headline oracle line stayed empty), but every other oracle element read the missing record as "this API has no contracts". ## The fix `state.hasOracle` records whether the artifact carries oracle evidence at all: `initializeMetadata` sets it from `summary.default_scope.oracle_verified` (which generator 1.10+ always publishes) before it recounts the headline. Without evidence: - `parityClass` returns "none" for every row, so no row reads as "unverified", the explorer shows no oracle badge and the detail pane no "no contracts" block; - the "Oracle-verified parity" metric card is omitted; - the surface cards' and tooltips' "N oracle-verified" counts are omitted; - the explorer's "NumPy oracle" filter control is hidden; - the detail pane's "NumPy oracle" fact is omitted; - the lede's "and the committed NumPy-oracle contracts that prove each API's parity" and the reading guide's "Oracle-verified is that claim's evidence ..." sentence are hidden (new spans `cov-lede-oracle` and `cov-oracle-explainer`). The flag defaults to true, so a caller that renders without metadata (the node test harness) keeps the evidence view. ## Tests coverage/test_dashboard.cjs gains two tests (journey4: 17 of 17 pass): - "an artifact without oracle evidence (generator < 1.10) hides every oracle element": the oracle fixture rows without their records and a summary without `oracle_verified`; - "an artifact with oracle evidence keeps the oracle elements after metadata". Both fail against the page before this commit. ## Measured with the two real inventories The page's own script, run in node over both artifacts: - master's (data branch `inventory/latest`, generator 1.6.0): before, the metrics read "Oracle-verified parity 0.0% 0 of 560 ... 491 available without"; after, "Scoped API availability 87.3% 489 of 560 comparable NumPy APIs", "Exact names 474", "Open gaps 69", "Entries in scope 560" and no oracle element; no error. - journey4's (generator 1.10.0, 2,851 rows): unchanged, "Oracle-verified parity 37.4% 941 of 2519" and "Headline oracle-verified: 88.2% (494/560 APIs; 43 available APIs without contracts)."
…xact names, and a NumPy 2.4.2 oracle of 4,683 probed records per C long model gates them - DTypeMeta.DefaultDescr() is default_descr(), IsAbstract is _abstract, ScalarType is type, DTypeFlags.Legacy is NPY_DT_LEGACY, DType.NativeByteOrder is NPY_NATIVE; the probes found and this fixes newbyteorder dropping an explicit host byte order (surface gap SU17), ensure_canonical handing out the singleton, promote_types' fast path matching by equality instead of identity, and np.dtype('M') not being the builtin singleton; ports dtype.isalignedstruct, len(dtype), dtype[key], dtype.metadata and np.dtype's align / copy / metadata keywords
The owner's rule (2026-10-14): every member NumPy names is spelled exactly as NumPy spells it, case for case; only class
names take the house ND* style. DType already followed it for its Python attributes; DTypeMeta, DTypeFlags and DType's
byte-order constants did not. A first check (NumPy's dir() of numpy.dtype, numpy._DTypeMeta and numpy.dtypes against the C#
members) found DType 22 of 23 attributes present and DTypeMeta's names all PascalCase.
Renames (DTypeMeta has not shipped in a release yet; DType's renamed constants are new names for the same values):
- DTypeMeta, Python class attributes: ScalarType -> type, IsAbstract -> _abstract, IsParametric -> _parametric,
IsLegacy -> _legacy, IsNumeric -> _is_numeric.
- DTypeMeta, C fields (PyArray_DTypeMeta / NPY_DType_Slots): Singleton -> singleton, TypeNum -> type_num, Flags -> flags,
WithinDTypeCastingImpl -> within_dtype_castingimpl, and NEW castingimpls (a read-only live view of the cast table,
wrapped once per class so it cannot be cast back to a writable dictionary).
- DTypeMeta, NEP 42 slots (dtype_api.h): DefaultDescr -> default_descr, CommonDType -> common_dtype,
CommonInstance -> common_instance, EnsureCanonical -> ensure_canonical, DiscoverDescrFromObject ->
discover_descr_from_pyobject, IsKnownScalarType -> is_known_scalar_type, GetClearLoop -> get_clear_loop,
GetFillZeroLoop -> get_fill_zero_loop. Overrides in LegacyBuiltinDTypeMeta, DatetimeDTypeMeta, PyScalarDTypeMeta and
the two test DTypeMeta subclasses follow.
- DTypeFlags: Legacy / Abstract / Parametric / Numeric -> NPY_DT_LEGACY / NPY_DT_ABSTRACT / NPY_DT_PARAMETRIC /
NPY_DT_NUMERIC (values unchanged, NumPy's bits).
- DType: NativeByteOrder -> NPY_NATIVE, NotApplicableByteOrder -> NPY_IGNORE, HostByteOrder -> NPY_NATBYTE,
SwappedByteOrder -> NPY_OPPBYTE, plus NumPy's NPY_LITTLE, NPY_BIG and NPY_SWAP.
- Kept, because they ARE NumPy's spelling: GetCastingImpl (PyArray_GetCastingImpl), AddCastingImpl
(PyArray_AddCastingImplementation), DTypePromotion.CommonDType (PyArray_CommonDType) - NumPy's C FUNCTION names are
PascalCase after the PyArray_ prefix. Kept, because NumPy has no name for them: Name, ModulePath, FullName, ScalarName,
TypeCode, Kind, TypeChar, ItemSize, Alignment, Aliases, HasStorage, HasReferences, Instantiate() (the class call),
DType.Meta, DType.DatetimeMetadata.
Ported numpy.dtype members:
- isalignedstruct: False (no NumSharp dtype is structured; NumPy answers False for every non-structured dtype).
- __len__(): 0 fields (NumPy's len(dtype) is the FIELD count, not the item size).
- __getitem__(key): NumPy's KeyError for any key, 'There are no fields in dtype float64.' naming str(dtype), quoted as
Python's str(KeyError) shows it (np.typename's KeyError convention).
- metadata: the dict np.dtype(..., metadata=...) attached (read-only; an empty attachment is an empty dict, not None),
carried by every copy (newbyteorder, ensure_canonical), ignored by equality, hashing, str and repr, dropped by
promotion of non-parametric classes - all as in NumPy.
- np.dtype(dtype, align=False, copy=False, metadata=None) on the string / Type / NPTypeCode / DType overloads, in
arraydescr_new's statement order: align is accepted and has no effect on a non-structured dtype (NumPy:
np.dtype('f8', align=True) is np.dtype('f8')); copy copies only a class singleton (NumPy's fields == None test: an
already-fresh descriptor such as '>f8' comes back as itself); metadata always yields a fresh descriptor whose dict is the
existing metadata merged with the keyword's WITHOUT overwriting existing keys (PyDict_Merge override=0).
Behaviour fixes the probes found (each pinned by the oracle and by DTypeMemberSemanticsTests):
- newbyteorder is now PyArray_ByteorderConverter + PyArray_DescrNewByteorder statement for statement. A requested host
order is STORED explicitly: np.dtype('f8').newbyteorder('<') has byteorder '<' on a little-endian host, is native, equals
and hashes like the '=' singleton, prints str() 'float64' but repr() "dtype('<f8')". A swap of a non-native descriptor
gives the host's explicit order, so repr(np.dtype('>i4').newbyteorder()) is "dtype('<i4')" - NumSharp returned
"dtype('int32')" (surface gap SU17, now fixed: its 2 cases return to surface.jsonl, its OpenBugs pin moves to the DTypes
tests). The error quotes the WHOLE request with Python's repr ("byteorder not recognized (got 'Xyz')"; NumSharp quoted
only the first character) and None is NumPy's TypeError "byteorder must be str, not NoneType".
- isnative is PyArray_ISNBO (byteorder != NPY_OPPBYTE), so an explicit host order is native; equality and GetHashCode read
'=' as the host's order (NumPy's _normalize_byteorder); repr takes the short '<f8' form whenever the STORED byte order is
explicit, the long 'float64' form only for '=' (isnative cannot decide it).
- The parser (_convert_from_str) folds an explicit host order into '=' before it builds anything, so np.dtype('<f8') stays
the float64 singleton and np.dtype('<M8[ns]') stores '='; and the datetime LETTERS np.dtype('M') / np.dtype('m') return
the static builtin descriptor (NumPy: isbuiltin 1, the same object every call), where NumSharp returned a fresh copy.
- ensure_canonical (NumPy's ensure_native_byteorder) returns a FRESH native copy of a non-native descriptor, never the class
singleton: np.result_type(np.dtype('>f8')).isbuiltin is 0 in NumPy and was 1 here. It now also refuses a descriptor of
another class (ArgumentException), as the datetime override already did.
- DTypePromotion.PromoteTypes' fast path is POINTER identity (NumPy's type1 == type2): two equal but distinct descriptors
(a copy, one with metadata, one with an explicit host order) go through the common class and come back as the singleton.
NumSharp matched by Equals and returned the first argument (repr "dtype('<f8')", isbuiltin 0, metadata kept).
The oracle (test/oracle/gen_dtype_oracle.py -> test/NumSharp.Tests/DTypes/corpus/dtype_oracle.<model>.jsonl, replayed by
test/NumSharp.Tests/DTypes/DTypeOracleTests.cs, no Python at test time). One file per C long model, because NumPy's type
numbers / characters of the 4- and 8-byte integers and which classes the C-name aliases are follow the platform's C long,
and so does DTypeRegistry: dtype_oracle.llp64.jsonl from Windows NumPy 2.4.2, dtype_oracle.lp64.jsonl from WSL's Linux
NumPy 2.4.2 (~/np242); the replay picks its host's model, so Windows, Linux and macOS CI are each gated in full. 4,683
records per file:
- members (1): numpy.dtype's 23 public attributes and 22 dunders, numpy._DTypeMeta's 5 own attributes, numpy.dtypes' 33
names, and - parsed from NumPy's own headers (refs/numpy dtype_api.h / dtypemeta.h, or NUMPY_SRC) - PyArray_DTypeMeta's
fields, NPY_DType_Slots and the 12 NEP 42 slot ids, the NPY_DT_* flag values and the byte-order macros. The replay
asserts each exists in C# under NumPy's spelling. Exemptions are listed with reasons and self-retire (each must stay
absent / divergent, so porting one fails the test until the entry is removed): __bool__, pickle, typing and Python
object-model dunders, __mul__ / __rmul__ (sub-array dtypes); the slots setitem, getitem, finalize_descr, get_constant,
the ArrFuncs table and 2.4's sort_meth / argsort_meth (NumSharp's kernels select on NPTypeCode); dt_slots / reserved
(C layout); the absent classes Complex64DType, StrDType, BytesDType, StringDType, ObjectDType, VoidDType.
- descr (1,014): names, sized and character codes, every byte-order prefix, every datetime unit and multiples, every
newbyteorder code (incl. 'little', 'swap', 'lx', 'Sx') and two-step chains over 16 bases, the copy / align / metadata
keywords and their combinations, and each class call - every attribute plus str, repr, len, the class name and d[0]'s
KeyError. Specs whose NumPy class is not the canonical class of its width (IntDType on Windows, LongLongDType on Linux,
LongDoubleDType) are left out: NumSharp has one class per width and kind.
- error (98): the recipes NumPy refuses (newbyteorder's invalid codes over every base), type and str(e).
- compare (784): ==, !=, <, <=, >, >= and hash equality over 28 descriptors incl. explicit-host, non-native, metadata and
copy descriptors and datetime units. One pair is a documented, self-retiring divergence: NumPy's
M8[1000ms] == M8[s] is True one way only (PyArray_EquivTypes' metric-prefix fold); NumSharp's Equals stays symmetric
(the Equals contract dictionaries rely on) and DTypeCasting.EquivTypes is NumPy's relation.
- promote (812): np.promote_types over every pair and np.result_type of each - repr, isbuiltin, byteorder, metadata, or
the DTypePromotionError.
- class (33): every numpy.dtypes class - __name__, __module__, repr, type.__name__, the four flag attributes, the class
call (instance repr and singleton identity, or the error). The C-integer / long-double classes NumPy keeps apart are
asserted to be NumSharp's canonical class instead.
- cast (1,941): every ordered pair of the 15 classes - whether NumPy has a casting implementation, its name, and what its
_resolve_descriptors makes of given descriptor pairs (casting level with NumPy 2.4's NPY_SAME_VALUE_CASTING_FLAG
masked, the loop descriptors, view_offset) against GetCastingImpl(to).ResolveDescriptors. All matched on the first run.
Mutation check: 8 planted bugs (isnative ignoring an explicit host order, a swap to '=', the equality fast path, a copied
datetime letter, a key-overriding metadata merge, a time cast safe instead of same_kind, a shifted flag value, repr's
short form keyed on isnative) - all 8 killed.
Also:
- The surface tier (gen_surface_oracle.py) loses the SU17 carve: regenerated deterministically, the diff is exactly the two
SU17 cases moving from gaps/surface_gaps.jsonl (984 left under 41 ids) to surface.jsonl (15,484); counts updated in
FuzzCorpusTests, the Fuzz README and CLAUDE.md.
- LeakCatalogue covers DType.__len__ and DType.__getitem__ (the leak-audit inventory gate requires every public member).
- Twelve crefs naming np.dtype(string) (6 in DType.cs, np.dtype.cs, 2 in np.frombuffer.cs, three test files) now name the new signature
- adding optional parameters broke them silently, CS1574 being in Core's NoWarn - and DType.cs' Type.Name cref and
DTypeFlags' NPY_CASTING cref resolve; a rebuild with NoWarn replaced shows no doc warning in the edited files.
- Updated: .claude/CLAUDE.md (the dtype section: NumPy names, the gate, the fixed behaviours, the current C long
type-number model), docs/plans/dtype-system.md (section 5).
Verification: NumSharp.Tests Release net10.0 18,287 / net8.0 18,286 passed (CI filter; 258 skipped = OpenBLAS binaries not
staged in the verification worktree); NumSharp.Tests.Oracle Release net10.0 255 / net8.0 173 passed (CI filter); the DTypes
namespace under Linux .NET in WSL, replaying dtype_oracle.lp64.jsonl, 286/286.
Merge note: a parallel session committed dd9a2c1 (DTypeMeta's lock-free cast table) this morning and has a follow-up
planned (BuiltinCastingImpl.Casting's torn Nullable read); code written against DTypeMeta must use the new names. Branch
`strings` adds DTypeMeta subclasses (StringDType and friends) and needs the same renames on its next rebase.
…type oracle of 6f11dc5 - under New APIs, the DType system entry lists isalignedstruct, metadata, len(dtype) / dtype[key] and np.dtype(dtype, align, copy, metadata), and the dtype classes under NumPy's own names (type, _abstract, _parametric, _legacy, _is_numeric, singleton, type_num, default_descr(), common_dtype(), ensure_canonical(), castingimpls, DTypeFlags.NPY_DT_*, DType.NPY_NATIVE and the other byte-order macros); under Testing & Tooling, the numpy.dtype / dtype-class oracle (4,683 NumPy 2.4.2 records per C long model) No Breaking Changes line: DTypeMeta, DTypeFlags and DType's byte-order constants first appear in 0.71.0 (0.70.0's DType had no byte-order constants and its newbyteorder threw), so the renames in 6f11dc5 change nothing a released version shipped.
Nucs
added a commit
that referenced
this pull request
Oct 14, 2026
…e, MLNet, System.Numerics.Tensors, ParquetNet) - merged code, but no NuGet package, no release-notes line and no public docs page until the owner releases them; CI now builds and tests all four WHY The owner decided on 2026-10-02 that NumSharp.Interop.OnnxRuntime, NumSharp.Interop.MLNet, NumSharp.Interop.System.Numerics.Tensors and NumSharp.Interop.ParquetNet stay in the codebase and merge to master with journey4, but are not released: no NuGet package is created, and no release or public documentation mentions them, until the owner releases them. Three of them were already on master, and public. On 2026-09-10 journey3's tip (07881ef, the code/data split) was fast-forward-pushed straight to master, and that push carried all 38 journey3 commits made after the v0.70.0 merge, the onnxruntime branch merge (112dba9), MLNet (bec868c) and System.Numerics.Tensors (b3a69d3) among them. Before this commit: - CI would pack and publish OnnxRuntime and MLNet on the next tag, and the GitHub release body hardcoded an install line and a badge row for each; - docs.yml had been deploying interop/onnxruntime.md, mlnet.md and system-numerics-tensors.md to scisharp.github.io/NumSharp since 09-10 (all three answered HTTP 200 on 2026-10-02); - no workflow built or tested System.Numerics.Tensors or ParquetNet at all (a 2026-10-02 probe found five open bugs in the Tensors bridge, recorded in the catalog). Removing the three from master instead was considered and rejected: they are interleaved with ~30 commits master needs, journey4 sits on master's tip, and a plain removal commit would make PR #631's merge silently delete the 81 of their 99 package files that journey4 never edited. WHAT Packaging, one switch per package: - The four csproj files set <IsPackable>false</IsPackable> beside their PackageId, with a comment. Probed: `dotnet pack` of such a project exits 0, writes no .nupkg and does not even compile (a project holding a compile error packs cleanly). `dotnet msbuild -getProperty:IsPackable` reads false, so nothing in Directory.Build.props overrides it, and all four pack to nothing in Release. - build-and-release.yml keeps naming them in the signing check and in build-nuget's Build and Pack steps, adding System.Numerics.Tensors and ParquetNet there, so the day one is released by flipping that property it is built, packed, signed, published and listed with no workflow edit. - create-release's "Compose release notes" step no longer hardcodes the package list. It reads each packed .nupkg's nuspec <id> and lists exactly those, NumSharp first, NumSharp.Build last, the rest in ordinal order, and it fails the step when it reads no id or fewer ids than there are .nupkg files (an unreadable package). Run under an ubuntu bash against fake packages: the five released ids are listed, and the unreadable and empty cases fail the step. Docs: - docfx.json excludes docs/interop/{onnxruntime,mlnet,system-numerics-tensors}.md. DocFX publishes every .md under docs/website-src whether the toc links it or not, so leaving a page out of the toc does not hide it. The page files stay in the repo. - toc.yml drops their three entries (a YAML comment says why; DocFX does not render it), interop/index.md drops their three table rows and "See also" links, advanced/index.md drops them from its list of bridges, and absolute-basics.md drops the ML.NET link from the mixed-CSV note, which would otherwise 404. ParquetNet has no page. - What is left is not a mention of the packages: the two website hits for "System.Numerics.Tensors" are code samples calling the BCL's own TensorPrimitives, and Core names the packages only in // comments, which the API reference does not render. CI gate: - interop-test gains a System.Numerics.Tensors section (build, then net8.0 and net10.0) and a ParquetNet section (build against the floor Parquet.Net 6.1.0, then net8.0 and net10.0) after the ML.NET one, each test run filtered TestCategory!=OpenBugs so a future known-bug reproduction does not fail the job. Local Release run: Tensors 54/54 and ParquetNet 11/11 on both TFMs. OnnxRuntime and MLNet were already gated there. These steps are the only thing in CI that compiles the hidden packages. - The job's header comment, its env comment and validate-release's comment now list six suites, and the signing check's timing note says only pythonnet compiles there now. .claude/CLAUDE.md: a new "Unreleased packages" section (what hidden means, the rules, how to release one), and the OnnxRuntime and MLNet table rows are marked UNRELEASED. AGENTS.md is a symlink to it. Master gets a matching commit for the three packages it already carries, which takes their pages off the public site. journey4 then records that commit with `git merge -s ours`, because this commit already does everything it does. Verified: the workflow parses (python yaml), every steps.<id> reference resolves and step ids are unique; the four hidden projects pack to nothing; the Tensors and ParquetNet suites pass on net8.0 and net10.0. Files: src/NumSharp.Interop.{OnnxRuntime,MLNet,System.Numerics.Tensors,ParquetNet}/*.csproj, .github/workflows/build-and-release.yml, docs/website-src/{docfx.json, docs/toc.yml, docs/interop/index.md, docs/advanced/index.md, docs/absolute-basics.md}, .claude/CLAUDE.md
…gistration leaves no trace, and NpzFile is safe to share between threads - the registry's three tables are ConcurrentDictionary (a plain Dictionary read during Register's writes lost registered classes in a resize: "FromName lost probe 632 after it was found"), every key is checked before the first write (a class refused for its storage code stayed half-registered), and NpzFile reads its ZipArchive under one gate with lock-free cache hits (concurrent first loads decoded each other's bytes)
DTypeRegistry (src/NumSharp.Core/DTypes/DTypeRegistry.cs)
- The race. Register wrote _byTypeNum / _byName / _byScalarType (plain Dictionary) under _lock, while FromTypeNum,
FromName and FromScalarType read them with no lock. FromScalarType runs on every Type -> DType conversion
(DType.From(Type), the implicit Type -> DType operator, np.zeros(3, typeof(int))). A Dictionary read during a resize
misses every key whose bucket the rehash has not reached yet. Restoring plain Dictionary tables makes the new test
report "FromName lost probe 632 after it was found" (11 of 12 runs).
- Now the three tables are ConcurrentDictionary with concurrencyLevel 1: _lock already serializes every write, so the
dictionary's own write lock is never contended. Capacities 31 / 61 / 31 hold the builtins without a resize: 18 type
numbers, 33 names and aliases, 16 scalar types, read from the live tables.
- The _byCode slots are written with Volatile.Write and read with Volatile.Read.
- _all is an array that each registration replaces with a longer copy (AppendToAll), so All copies an array without
the lock, and the caller still gets its own copy.
- Only Register locks, and no lookup waits for it. The lock stays on purpose. Add checks the type number, the name,
every alias and the storage code before it writes any table, and concurrent collections alone cannot make that
check-and-write one step: a reader could see a class that a refused registration removes again, or two
registrations could both fail where a serial order lets one succeed. A lock-free writer would need one immutable
snapshot swapped by compare-exchange; not done.
- Pre-existing bug fixed in the same method: Add wrote the type number and the names, and only then checked the
storage code. A class refused for a taken or out-of-range code stayed findable by FromTypeNum and FromName, and a
corrected retry failed with "A DType with type number N is already registered". Every check now runs before the first
write.
NpzFile (src/NumSharp.Core/IO/NpzFile.cs)
- The race. _cache (a Dictionary) was written by every first load with no lock, and the first load, GetRawBytes and
IsArray's magic check all read the ZipArchive. A ZipArchive is not thread-safe: each entry stream seeks and reads the
one underlying stream. Two threads loading members of one archive decoded each other's bytes (the mutant without the
gate fails the new test 5 of 5 runs), and two first loads of one member returned two different arrays.
- Now:
- _cache is a ConcurrentDictionary (concurrencyLevel 1, capacity = the member count, so it never resizes), read
without a lock.
- Every read of the archive and Dispose run under one gate (_archiveGate).
- A first load re-checks the cache under the gate, so every access to a member returns one instance, as the cache
promised single-threaded.
- _disposed is volatile and re-checked under the gate, so a Dispose racing a read ends it with
ObjectDisposedException. Dispose waits for a load in progress. Arrays already handed out stay valid: they belong
to their readers.
- Zip's docs say that reading the archive through it bypasses the gate.
- NumPy's NpzFile is not thread-safe; this one now is.
PythonRuntimeInterop._orphanedExports (pythonnet): checked, not changed
- A List<ExportKeeper> under its own lock, used only around engine shutdown. CaptureOrphanedExports appends every live
keeper; ReleaseOrphanedExports takes a snapshot and clears under the lock, then releases outside it.
- Nothing looks a keeper up by key, so a dictionary does not fit.
- The snapshot matters. A lock-free drain (ConcurrentQueue, as _pendingDisposals uses) would let the sweep for one dead
session release keepers that the next session captures during its own shutdown. Its interpreter may still read those
buffers while it dies.
- The lock is cold (shutdown only), and is never held across a Python call or on a finalizer thread.
Tests
- test/NumSharp.Tests/DTypes/DTypeRegistryConcurrencyTests.cs (3):
- A class refused for an alias, for a taken code or for an out-of-range code leaves no type number, name, alias or
All entry, and the corrected class registers.
- All returns a copy, in registration order.
- Lookups during 1,024 registrations: 4 writer and 4 reader threads; the writers start once every reader is looping,
and the builtins are checked between every probe lookup. Every lookup returns null or the right class, a class once
found is never lost, All never shrinks, nothing throws. With 96 classes no reader hit a resize in 10 runs (a resize
of the builtin-sized tables lasts about 100 ns, and the builtin keys are re-filled first). 1,024 classes grow each
table through five or six resizes, the late ones over thousands of entries.
- Registrations are permanent, so each test takes type numbers from 40,000 up and names built from them, and probe
scalar types ScalarProbe<T1, T2, T3, T4> (10,000 distinct types). The tests already reading All tolerate extra
classes (Contain, Count >= 21, unique type numbers).
- test/NumSharp.Tests/IO/NpzFileConcurrencyTests.cs (2):
- 24 rounds of a fresh compressed archive, 4-8 threads, two starting on each member: values, raw bytes and IsArray
are right, and one instance per member per round.
- Dispose while threads read raw bytes and members: every read returns its bytes or raises ObjectDisposedException,
nothing else, and arrays handed out before keep their values. Members of 10,000 float64 (80,128 bytes), so every
GetRawBytes copy stays under the large-object threshold.
- Both dispose every array they allocate. A first version left about 20 MB of pooled buffers to the finalizer
(expected arrays, arange / astype intermediates, the members handed out), and the wall-clock test
T2_NonZero_Is_Much_Slower_Than_NumPy, which runs later in the suite, then measured 127-145 ms against its 32 ms
limit on both frameworks. It passed with the npz tests excluded and passes with them disposing everything.
- test/NumSharp.Tests/DTypes/DTypeMetaCastingImplConcurrencyTests.cs: one comment corrected. Barrier.Dispose does not
throw while a thread waits (only when called from the post-phase action); the waiting thread gets an
ObjectDisposedException on its own thread, which nothing catches there, so it would end the test host.
Mutation check (detached worktrees; 5 runs per mutant unless noted; the unmutated tests passed all 40 baseline runs
before and after the mutants)
- R1 storage code checked after the writes (the old order): 5/5 caught.
- R2 plain Dictionary tables (the original race): 0/15 with 96 classes; 11/12 with 1,024.
- R3 All returns the registry's own array: 5/5.
- R4 Register without the lock: 15/15 (racing appends lose classes from All).
- N1 a first load without the gate: 5/5. N2 no re-check of the cache under the gate: 5/5 (two instances).
N3 GetRawBytes without the gate: 5/5. N4 Dispose without the gate: 5/5. All four again 5/5 on the final test
(smaller members, everything disposed).
- N5 GetRawBytes without the disposal re-check under the gate: 0/5, equivalent here. ZipArchiveEntry.Open on a
disposed archive raises ObjectDisposedException itself; the explicit check keeps the contract independent of that.
- N6 _disposed not volatile: 0/5, equivalent on x64 (the lock orders it).
Verification
- NumSharp.Tests with CI's filter, on the final tests: net10.0 18,539 passed, 11 skipped, 0 failed; net8.0 18,538
passed, 11 skipped, 0 failed.
- DTypes and IO namespaces plus T2_NonZero_Is_Much_Slower_Than_NumPy: 422/422 on net10.0 and net8.0.
- Crefs: Core built in Release with the csproj NoWarn replaced, on net10.0 and net8.0. No warning from the changed
lines; the CS1591s on DTypeRegistry.Bool ... Complex128 were there before.
Docs: .claude/CLAUDE.md, the DTypeRegistry sentence (lookups take no lock, only Register locks, nothing is written
before every check passes) and the NpzFile sentence (safe to share between threads, unlike NumPy's).
…ake the field names of the NumPy structs they port - name, nin, nout, casting, flags, dtypes, resolve_descriptors() and get_strided_loop() (PyArrayMethodObject / PyBoundArrayMethodObject), method and descriptors (PyArrayMethod_Context), @base and num (PyArray_DatetimeMetaData); the dtype oracle now records each struct's fields from NumPy's headers and its replay holds the C# classes to them The owner's rule, applied in 6f11dc5 to DType, DTypeMeta and DTypeFlags, now covers the other ports of src/NumSharp.Core/DTypes: a member NumPy names takes NumPy's exact spelling (only class names keep the house style). Renamed (old -> new): - ArrayMethod: Name -> name, NIn -> nin, NOut -> nout, Casting -> casting, Flags -> flags, DTypes -> dtypes, ResolveDescriptors(...) -> resolve_descriptors(...), GetStridedLoop(...) -> get_strided_loop(...). These are the fields of NumPy's PyArrayMethodObject (numpy/_core/src/multiarray/array_method.h) and, for dtypes, of the PyBoundArrayMethodObject NumPy wraps around a method to hold its operand classes. NumSharp keeps both in one object. The same names are the fields of the public PyArrayMethod_Spec (dtype_api.h) that an extension fills in. The overrides follow: BuiltinCastingImpl, TimeToTimeCastingImpl, DatetimeTimedeltaCastingImpl, LegacyWrappingCastingImpl, and the test probe in DTypeMetaCastingImplConcurrencyTests. - ArrayMethodContext: Method -> method, Descriptors -> descriptors (PyArrayMethod_Context, dtype_api.h). - DatetimeMetaData: Base -> @base, Num -> num (PyArray_DatetimeMetaData, ndarraytypes.h). C# spells the field `meta.@base` because base is a keyword. Reflection and nameof read the name as `base`, the same precedent as NDArray.@base. Kept, with the reason: - CastingImpl.From / To / IsWithinDType and ArrayMethod.NoView have no NumPy name. NumPy reads the two classes as dtypes[0] / dtypes[1] and writes NPY_MIN_INTP for "not a view" inline. - The static helpers that port NumPy-internal C functions keep their names, as in 6f11dc5: DatetimeMetaData.Parse, ParseUnit, Divides, GreatestCommonDivisor and CanCast*, and DTypeCasting.ParseCasting / CastingToString / KindToOrdering. Ports of C API functions already have NumPy's spelling after the PyArray_ prefix. - Parameter names stay C# camelCase (givenDescrs, loopDescrs, viewOffset): no NumPy caller names a C slot's arguments. The constructors whose parameters now carry the fields' names write every field through `this.`. The ArrayMethod, ArrayMethodContext, CastingImpl and DatetimeMetaData constructors gained the XML documentation they lacked. The gate (test/oracle/gen_dtype_oracle.py -> DTypes/corpus/dtype_oracle.{llp64,lp64}.jsonl): - struct_fields() reads a declaration of several fields (`int nin, nout;`) as each of them, by splitting at top-level commas. It drops comments and preprocessor lines first, so a field inside `#if ... #endif` (PyArrayMethod_Context's NumPy 2.4 fields) is read. The two structs it already parsed give the same names as before. - The members record gains struct_fields: PyArrayMethodObject, PyBoundArrayMethodObject, PyArrayMethod_Context and PyArray_DatetimeMetaData, each with the field names of NumPy's headers. - Both files were regenerated, the llp64 one with Windows NumPy 2.4.2 and the lp64 one with Linux NumPy 2.4.2 under WSL. Only the members line changed; the struct fields are the same in both models. - DTypeOracleTests.StructPorts maps each struct to its C# port and lists every field the port does not port, with the reason. PyArrayMethodObject: static_data, resolve_descriptors_with_scalars, get_reduction_initial, the five default loops, the four wrapping fields, legacy_initial. PyBoundArrayMethodObject: method. PyArrayMethod_Context: caller, _reserved, flags, parameters. - Members_ExistUnderNumPysNames requires every other field to be a public member of the port under the exact name. An exemption that becomes present fails the test until its entry is removed. A struct in the record with no mapping fails, and so does a mapping the record no longer has. - Checked by mutation: - an extra field in the record fails ("no DatetimeMetaData member 'nonexistent_field'"); - an unmapped struct fails; - a `caller` member added to ArrayMethodContext fails ("is now ported"). Verified: - The solution builds in Release. The only errors are the 11 pre-existing CS0121 ambiguities of np.amax / np.amin in benchmark/NumSharp.Benchmark.CSharp on net8.0; they are unrelated and not touched. - NumSharp.Tests with the CI filter: net10.0 18,287 passed, net8.0 18,286 passed, 0 failed. - The DTypes tests (286) pass under Linux .NET with the lp64 corpus. - No new unresolvable cref (built with CS1574 visible). Docs: .claude/CLAUDE.md (the dtype paragraph and the names paragraph name the new members and what is kept); docs/plans/dtype-system.md (the class map and a new entry under "NumPy's names and a probed oracle"). Breaking, and released nowhere: the DTypes folder first ships in 0.71.0 (v0.70.0 has no src/NumSharp.Core/DTypes). (cherry picked from commit 5ffdc84697f20e398f95edd54d73bb643e372125)
… ufunc-like export - the method list is read from numpy.ufunc (adding resolve_dtypes to the hand-written reduce / accumulate / reduceat / outer / at), and the scan covers the ufuncs numpy.strings and numpy.char export and numpy.ma's masked ufunc wrappers, which had no rows at all Question this answers: does the coverage inventory cover the ufunc-object methods (np.maximum.accumulate, np.maximum.reduce, ...) and similar ones? The top-level combinations were catalogued (numpy.<ufunc>.<method> rows, out of the headline scope, all missing), but the per-export loop in coverage/object_surfaces.py hard-coded five method names and scanned only vars(np). Three gaps followed, each confirmed against the NumPy 2.4.2 pin: 1. resolve_dtypes. numpy.ufunc has six public methods (accumulate, at, outer, reduce, reduceat, resolve_dtypes - NumPy 1.24+). numpy.add.resolve_dtypes had no row; only the generic numpy.ufunc.resolve_dtypes member row (from the np.add sample) existed. 2. Submodule ufuncs. numpy.strings exports 17 ufunc objects (7 identical to top-level ones, 10 string-only: str_len, isalpha, isdigit, ...) and numpy.char 11. None had method rows, although np.strings.add.reduce concatenates a StringDType array and np.strings.str_len.at is the same protocol as np.add.at. 3. numpy.ma's masked ufunc wrappers. 19 exports (_MaskedBinaryOperation, _extrema_operation) implement part of the protocol: ma.add / multiply / subtract / hypot / arctan2 / the bitwise and logical ones have reduce, accumulate and outer; the six comparisons only accumulate and outer (NumPy sets equal.reduce etc. to None on the instance); ma.maximum / ma.minimum only reduce and outer; the domained and unary wrappers none. No row existed for any of them. The per-ufunc attributes (identity, nin, nout, nargs, ntypes, types, signature) stay generic numpy.ufunc.<name> rows: each is the same contract on every ufunc, while a method is a distinct operation per ufunc (add.reduce is a sum, maximum.reduce a max). The change: - object_surfaces.ufunc_protocol_methods(np): the method list, read from numpy.ufunc with inspect.getattr_static (callable descriptors only, so properties are excluded and nothing is evaluated). A method a later NumPy adds gets rows on the next generation. - object_surfaces.ufunc_method_rejection(ufunc, method): the shape rules the old loop applied, per method and only for the five methods they were reviewed for (gufunc core signature; binary for reduce/accumulate/reduceat/outer; single output for reduce/accumulate/reduceat/at; unary or binary for at). resolve_dtypes has no shape rule: probed, it resolves for unary (sqrt), multi-output (divmod) and core-signature (matmul) ufuncs; its failures (bitwise_* on float64, isnat on non-datetimes) are dtype support, so its rows are "conditional". An unknown method is never rejected by guesswork. - object_surfaces.ufunc_protocol_rows(np, seen_ids, namespaces, ...): walks numpy itself plus the given namespaces. A numpy.ufunc instance gets a row per method (applicability not_applicable when NumPy rejects every call, else conditional). Any other non-class, non-module export gets a row per protocol method it actually implements: getattr_static sees the instance override numpy.ma uses to remove a method (equal.reduce = None), so numpy.ma.equal.reduce and numpy.ma.maximum.accumulate correctly have no row. Top-level rows keep the "ufunc" surface and their ids; submodule rows use the namespace's display surface (numpy.strings.add.reduce on surface "strings", numpy.ma.add.reduce on "ma"). A namespace that does not resolve raises ValueError, so a removed submodule fails the run instead of silently dropping rows. Wrapper method rows link the nearest documented page (numpy.ma's), never a fabricated URL. - generate_coverage.py: HEADLINE_SUBMODULES (random, linalg, fft; now also used by public_exports instead of an inline tuple, same iteration) and UFUNC_PROTOCOL_NAMESPACES (the headline submodules plus every EXTENDED_SUBMODULES entry), passed to object_surface_rows as ufunc_namespaces. The scan therefore walks exactly the namespaces the catalog compares; today only numpy, numpy.strings, numpy.char and numpy.ma hold ufunc-like exports. summary.md's protocol sentence says what the count covers. GENERATOR_VERSION 1.10.0 -> 1.11.0. - README.md describes the discovery rule and the wrapper subsets. Artifact, before -> after (generated from this tree, NumPy 2.4.2): - catalog rows 2,851 -> 3,174 (+323: 106 resolve_dtypes, 102 numpy.strings, 66 numpy.char, 49 numpy.ma wrapper rows); no row removed; no pre-existing row changed except the note text of the 530 top-level protocol rows. - ufunc protocol rows 530 -> 853 (conditional 272 -> 515, not_applicable 258 -> 338: the 80 new not_applicable rows are the unary string ufuncs' reductions and outer). - headline unchanged: 533 of 560 available (+4 partial), 537 oracle-verified. - expanded API scope 2,519 -> 2,842 APIs, 1,213 available (48.2% -> 42.7%): the new rows are real NumPy contracts NumSharp lacks (it exposes ufuncs as functions, so no object member can match), previously invisible. Tests (coverage/test_object_surfaces.py, now built from the generator's namespace list): - the method list equals numpy.ufunc's six public methods and excludes the seven properties, which keep their numpy.ufunc.<name> rows; - all 106 top-level ufunc exports carry every method (was: five); - numpy.strings (17) and numpy.char (11) ufunc exports carry every method, on their surface; np.strings.add.reduce is conditional and NumPy concatenates a StringDType array but raises TypeError for fixed-width <U1 (dtype resolution), and isalpha.reduce is not_applicable (ValueError); - numpy.ma: exactly 49 rows over 19 wrappers, the per-wrapper method sets pinned (add: reduce / accumulate / outer; equal and less: accumulate / outer; maximum and minimum: reduce / outer), no rows for equal.reduce, maximum.accumulate, divide, sqrt, add.at or add.resolve_dtypes, and NumPy's own behaviour checked (ma.add.reduce of [1, --, 3] is 4, equal.reduce is None, maximum has no accumulate); - resolve_dtypes is conditional and resolves for add, sqrt, divmod and matmul; - numpy.add.resolve_dtypes, numpy.ma.add.reduce and numpy.strings.add.reduce are never credited to a NumSharp namesake. Results: python -m unittest discover -s coverage -p "test_*.py" 37 OK; node --test coverage/test_dashboard.cjs 17 OK; python coverage/generate_coverage.py writes the artifact (oracle join clean, 690,681 contracts). Context: GitHub usage counts measured the same day (gh api search/code, vendored library copies removed) put np.add.at (54.5K files), np.maximum.accumulate (42.7K), np.maximum.reduce (11.7K) and np.add.reduce (9.0K) on top; per method at 63.6K, accumulate 53.3K, reduce 49.5K, outer 12.0K, reduceat 9.2K files. NumSharp has no spelling for add.at, running max/min, reduceat or N-D outer.
1046439 and quote the coverage dashboard's current figures - the expanded scope is 1,213 of 2,842 APIs (was 1,137 of 2,519, which 1046439's 323 new ufunc-method contracts and the later surface tiers had made stale), the headline 533 of 560 available with 4 more partial and 537 oracle-verified (was 535 available and 494 verified, written on 2026-10-02 before the surface tier) What moved, and why the old figures were wrong rather than the artifact: - Testing & Tooling gains one line for 1046439: the coverage inventory lists every numpy.ufunc method (reduce, accumulate, reduceat, outer, at, resolve_dtypes, read from the type) for every ufunc, including those of numpy.strings and numpy.char, and the methods numpy.ma's masked ufunc wrappers implement - 323 more API contracts. - The Supported Features Dashboard's expanded-scope sub-bullet now names the ufunc methods next to the submodules and object members, and quotes 1,213 of 2,842 APIs (the artifact generated from journey4 at 1046439 with NumPy 2.4.2). The denominator grew by exactly the 323 rows of 1046439; the 1,137 -> 1,213 rise in available APIs predates it (the surface tiers and the APIs added since 2026-10-02). - The headline (Overview and the dashboard bullet) quotes 533 of 560 available (95.2%), 4 more partial, and 537 of 560 oracle-verified (95.9%: every available and partial headline API has contracts). The 535 written on 2026-10-02 dropped to 533 deliberately: 2de4328 (the surface tier, 2026-10-08) reclassified numpy.cumulative_sum / numpy.cumulative_prod from available to partial in coverage/overrides.json, because their np.cumsum / np.cumprod aliases lack include_initial and flatten an N-D input without axis where NumPy raises (surface gaps SU36 / SU37). The other two partial rows are numpy.ndarray.data and numpy.shape. The 494 -> 537 verified count is the surface tiers' contracts. Figures checked against coverage/generate_coverage.py's output for this tree (generator 1.11.0): default_scope total 560, available 533, partial 4, missing 23, oracle_verified 537; api_scope total 2,842, available 1,213. ASCII hyphens only (CHANGELOG_STYLE section 4).
…t their NumSharp owner types NDMaskedArray and Vectorized - the object surfaces declared no owner, so all 94 MaskedArray members and the vectorize members read missing although NumSharp implements 77 of them and Vectorized.signature; 79 rows become available (expanded scope 1,213 -> 1,291 of 2,842)
Found while preparing a usage ranking of everything the inventory reports missing (the owner's
request: rank all missing NumPy inventory by GitHub code-search counts). Before counting, the
"missing" list had to mean missing: coverage/object_surfaces.py declared
ObjectSurface("ma.MaskedArray", disposition="subsystem") and ObjectSurface("vectorize",
disposition="tooling") without a clr_type, and object members match only their explicitly mapped
CLR owner, so nothing could ever credit them. The reflection inventory has both owners
(objectTypes "NumSharp.NDMaskedArray", "NumSharp.Vectorized"); the gap was already noted in the
memory of the 2026-09-25 oracle join ("map it to NDMaskedArray, a crediting change, not done").
The change: ma.MaskedArray -> NumSharp.NDMaskedArray, vectorize -> NumSharp.Vectorized (both now
the default "candidate" disposition: NumSharp has the subsystem, and the remaining members are
implementable). Checked before mapping:
- NDMaskedArray is a standalone class (no NDArray base), and every one of the 77 matched members
is declared on NDMaskedArray itself (inventory declaringTypes), so no credit comes from an
inherited ndarray member without masked semantics.
- Exact names, and method/property shape agreeing (the tool's existing rule); 0 name-but-wrong-
shape cases.
- The 17 MaskedArray members NumSharp lacks stay missing: base, ctypes, device, dump, dumps,
flags, get_fill_value, get_imag, get_real, getfield, set_fill_value, setfield, setflags,
sharedmask, to_device, toflex, torecords. vectorize keeps pyfunc, otypes, excluded, cache and
the __call__ protocol missing.
Artifact (generator 1.11.0 -> 1.12.0, NumPy 2.4.2): 78 numpy.ma.MaskedArray rows (77 members +
the class row) and numpy.vectorize.signature move missing -> available; expanded scope 1,213 ->
1,291 available of 2,842 (42.7% -> 45.4%); headline unchanged (533 of 560, 537 verified); the
oracle's "uncredited" rows (contracts exercising an API the catalog calls missing) drop 1 -> 0,
so README's example of that class now says none remain.
Tests: test_object_surfaces gains test_masked_array_and_vectorize_members_credit_their_numsharp_owners
(94 member rows, 77 credited to NumSharp.NDMaskedArray.<name> with source paths, the genuine gaps
still missing, filled -> NDMaskedArray.filled, vectorize.signature -> Vectorized.signature,
vectorize.pyfunc missing); the "not credited" list swaps numpy.ma.MaskedArray.reshape (now
correctly credited to NDMaskedArray's own reshape) for numpy.ma.MaskedArray.ctypes. Results:
python -m unittest discover -s coverage -p "test_*.py" 38 OK; node --test
coverage/test_dashboard.cjs 17 OK.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NumSharp 0.71.0
🧭 Overview
np.polynomial.*- thenumpy.polynomialpackage, 172 of its 193 functions for all six bases, bit-exact.np.*- +48 functions.np.random.*- every NumPy bit generator, the wholeGeneratordistribution surface, and a legacyRandomStatethat is NumPy's own sampler code line for line, all byte-identical streams.np.ma.*- the masked-array module;np.emath.*- the automatic-domain math module.DTypeis NumPy's two-level dtype model -NDArray.dtypereturns anumpy.dtype-shaped descriptor, withnp.dtypes, byte order anddatetime64/timedelta64descriptors.max,min,ptp,nanmax,nanmin,argmaxand the fusednp.evaluatereductions match NumPy bit for bit, down to the sign of a ±0 tie and a NaN's payload, and most of them run faster.Tensor<T>/TensorSpan<T>.Detailed Breakdown
Show All
📦 New NuGet Packages
One optional companion package ships for the first time, co-versioned with NumSharp 0.71.0 (it depends on NumSharp).
NDArrayand System.Numerics.Tensors (Tensor<T>,TensorSpan<T>,ReadOnlyTensorSpan<T>); all 15 NumSharp dtypes cross as their own CLR type,Half,Complex,decimalandcharincluded.AsTensorSpan<T>()views anNDArrayof any non-negative-stride layout (contiguous, sliced, transposed, strided, broadcast) as aTensorSpan<T>over the same unmanaged buffer, through a disposable handle that keeps the buffer pinned;ToTensor<T>()copies into a denseTensor<T>.AsNDArray()views aTensor<T>as anNDArraywithout copying (writes go through to the tensor);ToNDArray()copies aTensor<T>,TensorSpan<T>orReadOnlyTensorSpan<T>.TensorPrimitivesdirectly, with no fill loop.📊 Dashboards & Docs
numpy.ma,numpy.emath,numpy.polynomial,numpy.char,numpy.strings, ...) and the members of its objects (nditer,dtype,finfo, the random classes): 1,137 of 2,519 APIs available in that expanded scope, the dashboard's default view.✨ New APIs & Modules
np.polynomial- thenumpy.polynomialpackage for all six bases (polynomial,chebyshev,legendre,laguerre,hermite,hermite_e) pluspolyutils: 172 of its 193 functions, bit-exact with NumPy 2.4.2 against generated oracles (139,882 cases, 0 excused);{p}below ispoly/cheb/leg/lag/herm/herme-57545200,38ed8da7,ac8608d9,38526155,e2ff9477,0c02740c(+0896cbf8,d494e83d).{p}val,{p}val2d,{p}val3d,{p}grid2d,{p}grid3d,{p}valnd- evaluation.{p}add,{p}sub,{p}mulx,{p}mul,{p}div,{p}pow,{p}fromroots- series arithmetic.{p}der,{p}int- calculus, with everym/k/lbnd/scl/axisform.{p}vander,{p}vander2d,{p}vander3d- Vandermonde matrices.{p}companion,{p}roots- companion matrices and roots (the roots compute through the OpenBLAS backend'seigvals).{p}trim,{p}line,{p}domain,{p}zero,{p}one,{p}x, the fiveX2poly/poly2Xpairs - helpers, module constants and basis conversions.as_series,trimseq,trimcoef,getdomain,mapparms,mapdomain-polyutils.chebpts/chebinterpolate,polyvalfromroots,format_floatand the six series classes.np.randombit-generator family - NumPy's engines, seeding and stream splitting, byte-identical to NumPy 2.4.2 and replayed overload by overload (493 members, 5 engines x 10 seeds) -c1e0feb5,65bdea7b(+0e39714b,eebe85a6).PCG64DXSM,Philox,SFC64- new bit generators;PCG64.advance,jumped,random_raw,state- engine controls;MT19937is now a NumPyBitGeneratortoo.SeedSequence.spawn,spawn_key,pool_size,state,BitGenerator.spawn,Generator.spawn,ISeedSequence- seeding and independent streams.default_rng- every NumPy seed form (integers,BigInteger, arrays,SeedSequence, a bit generator, aGenerator, aRandomState,nullfor fresh entropy).np.random.PCG64(42),np.random.Generator(...),np.random.SeedSequence(...)and the other classes - NumPy's module spellings;get_bit_generator,set_bit_generator.Generatordistributions - NumPy's wholenumpy.random.Generatorsurface on NumPy's modern algorithms, 29 samplers byte-identical todefault_rng-c0613b2f.beta,chisquare,f,noncentral_chisquare,noncentral_f,standard_t,standard_cauchy,vonmises- the gamma family and friends.pareto,weibull,power,laplace,gumbel,logistic,lognormal,rayleigh,wald,triangular- one transform per draw.binomial,negative_binomial,poisson,zipf,geometric,hypergeometric,logseries- discrete.multinomial,dirichlet,multivariate_normal(svd/eigh/cholesky),multivariate_hypergeometric(marginals/count) - multivariate.pareto/powerstay within 2 ULP /ceil(2/a)+3ULP of NumPy, whose C runtimeexpm1has no portable twin;multivariate_normalis byte-identical with the OpenBLAS backend and may pick the other eigenvector sign without it.RandomStateandGeneratortakeNDArrayparameters, broadcast againstsizeas NumPy does, byte-identical streams -39d446e0(+5f204884).integers,randint- array bounds andBigIntegerbounds, sog.integers(0, BigInteger.Pow(2, 64), dtype: np.uint64)ports verbatim -0e39714b.RandomStategains NumPy's form over any bit generator (np.random.RandomState(new PCG64(42))), the dictget_state(legacy: false)/set_state,tomaxint,ranfandsample-f87b4d04.np.ma- NumPy's masked-array module on a newNDMaskedArraytype, bit-exact vs NumPy 2.4.2 and gated by a 68,860-case differential corpus -5f1476f5(+ba15bad4,bf66bc0f,b33b6515,3fb1d897,e0d6a81c,67f8f96e,7a262afc).average,median,dot, the set operations,sort,unique,cov,corrcoef,convolve,correlate,polyfit,apply_along_axis,apply_over_axes- the extras.str/repr; only the structured/record-dtype functions are missing.np.emath- automatic-domainsqrt,log,log2,log10,logn,power,arccos,arcsin,arctanh(an out-of-domain input gives the complex result instead of NaN) -a3666a5f.np.*functions, NumPy 2.4.2 parity, each gated by the differential-fuzz oracle -hypot(correctly rounded: float32/float16 bit-exact, float64 bit-exact on ~91% of inputs and 1 ULP more accurate on the rest),divmod,fmod,remainder,float_power,heaviside,spacing,frexp,ldexp,gcd,lcm,fabs,signbit,fix- elementwise math ufuncs -5c667b09,df232348,44636747,6e0e5ea2,ade8d6b5,83e0e43e,a15f0ee4,68a84d3c,46f5f651.sinc,i0,real_if_close,unwrap- special functions and signal helpers -c3be0b7f,a0c6e2fe,877a36b2,b1937c51.bartlett,blackman,hamming,hanning,kaiser- window functions, bit-identical -0e9dbc42.trapezoid,gradient- numerical integration and differentiation, bit-exact -22d5f4d3(+f6037bc8,4898b40a).histogram,histogram_bin_edges,histogramdd,histogram2d- histograms -2693d83e(+0e80bc89).packbits,unpackbits,bitwise_count- bit packing and population count -3936fd0d,bab51edb.piecewise,putmask,put_along_axis- indexed and conditional assignment -76b2ab9d,86cc28b8,f417827e.vectorize,frompyfunc,apply_along_axis,apply_over_axes,broadcast_shapes- functional and shape helpers -b1841830,53b7a149.array_equiv,may_share_memory,shares_memory(withTooHardError) - array predicates -38f9a14e,4cd3b72f.logspace,geomspace- log-scale ranges, bit-exact -d48737ec(+3a367520).binary_repr,base_repr,typename- string helpers -c2f129b5,7cd38a97.maximum,minimum,fmax,fmin-out=/where=-e5afe84c.left_shift,right_shift,logical_and,logical_or,logical_xor,logical_not,modf-out=/where=/dtype=-3384a0b5.array_equal-equal_nan=-38f9a14e.indices,indices_sparse-long[]/Shapedimensions (np.indices(arr.shape)), andmgridpast 2^31 elements -07c7b94c.np.evaluate/NDExpr- the fused-expression DSL covers most of the ufunc surface and every reduction, NumPy-typed and bit-exact (29,563-case oracle tier) -6175a017,bd0091fb,e1400974,2f02d65d,eb1921e1,fbef1c4e,c5422cfc,dcbe7079.Maximum,Minimum,FMax,FMin,Fmod,CopySign,NextAfter,LogAddExp,LogAddExp2,Hypot,Heaviside,Gcd,Lcm,LeftShift,RightShift- binary nodes.Positive,Conjugate,Fabs,Spacing,SignBit,IsPosInf,IsNegInf,BitwiseCount,Rint,Real,Imag,Angle,Cast,Round(decimals)- unary nodes.LogicalAnd,LogicalOr,LogicalXor,Select,Clip, and the</>/<=/>=operators onNDExpr- logic and selection.Any,All,CountNonzero,NanSum,NanProd,Ptp,NanMin,NanMax,ArgMax,ArgMin,NanMean,Var,Std(ddof), the weightedAverage, axis forms andkeepdims- reductions.dtype=,casting=,order=,where=- keywords;expr.Compile()returns a reusableCompiledExpression.If,Switch,Mux,Saturate,NanTo,Coalesce,Relu,LeakyRelu,Elu,Sigmoid,Swish,Softplus,Gelu,Step,Nand,Implies,IsClose,Between,Bucketize,Lerpand 18 more - 38 decision and activation combinators that fuse like any node -d3057b39(+7aba9b4a).DTypesystem - NumPy's NEP 40-50 two-level dtype model (dtype classes and dtype instances), the descriptor every dtype parameter andNDArray.dtypenow use -896f02b2,c916c579,72e4d29c.np.dtypes(Float64DTypeand the other dtype classes),np.datetime_data- the class module.num,kind,char,itemsize,str,descr,byteorder,newbyteorder,isbuiltin,isnative,flags,type_num(NPY_TYPES) - the fullnumpy.dtypesurface.np.dtype(">i4")) anddatetime64/timedelta64descriptors that parse, print, promote and cast-check (no datetime storage yet).promote_types,result_type,can_cast,issubdtype,isdtype-DTypeoverloads with NEP 50 weak scalars.np.flat<T>(a)/a.flatiter.AsTyped<T>()- typed by-refC-order flat iteration -fa2e8521.np.ndenumerate<T>(a).AsRef(),np.ndindex(...).AsSpans()- allocation-free enumeration -afb0d5ce.nd.Unsafe.Span<T>(),Memory<T>(),Bytes(),Pointer<T>(),TryGetSpan<T>(),TryGetMemory<T>()- zero-copy BCL views of the buffer (they do not keep it alive: hold theNDArray);nd.AsEnumerable<T>()and the typed iterators'ToArray()/CopyTo(Span<T>)- safe copies -37a112be.Py.RegisterNumSharpCodec()(NumSharp.Interop.pythonnet) - the one-call codec registration, spelled on pythonnet's ownPy(a C# 14 extension member) -f18aaa23(+b648ee53).NumSharp.Collections-ConcurrentOrderedDictionary<TKey,TValue>,ConcurrentOrderedCompactDictionary<TKey,TValue>(about a third of the memory) and the lock-freeOrderedDictionary<TKey,TValue>: insertion-ordered, index-addressable dictionaries, plus a vendoredNumSharp.Collections.Concurrent.ConcurrentDictionary<TKey,TValue>with by-ref access -39e5ba17,47c12604,ba5c347d(+04677203,873900b1,478b5310).🧩 ndarray surface
ToArray<T>()- aTother than the dtype converts likeastype(it threwArrayTypeMismatchException), and every layout copies in one pass (~5.8x faster than before on the measured set) -73514657.ToMuliDimArray<T>()/(Array)nd- one pass over any layout; float16, complex128, aTother than the dtype (converted likeastype), 0-d and empty decimal arrays now work (they threw) -c243201a.ToJaggedArray<T>()- any rank (rank 6 came back wrong, deeper ranks threw), any layout (a transposed view gave wrong values, a column slice read out of bounds) and aTother than the dtype; 10.7x faster with 7.7x less memory -4231d2d0(+6f2ec473,ac89847a).⚡ Performance
Ratios are NumPy ÷ NumSharp - higher is better (
x2= twice NumPy's speed);xLOW->xHIGHspans the worst→best measured cell across sizes and dtypes.x0.70->x3.1-np.max/np.min/np.ptpon NumPy's exact reduction schedules, slab reductions folding eight reduced indices per pass (axis cells on non-C layouts were x0.28-x1.0; L3-bound 4M cells of 1- and 4-byte lanes still trail) -1d79ce61,f8f91b3d.x0.41->x4.7-np.nanmax/np.nanmin(float32/float64 axis cells were x0.15-x0.50, flat C now x2.89 / float32 x3.33; float16/complex128 non-C axes run NumPy's scalar loop at x0.41-x0.69) -de6333dd,268d91fd.x0.77->x106-np.argmax/np.argminalong an axis walk the strides instead of a per-output div/mod walk (axis 0: float64 x4.4-x8.8, integers x10-x16, bool x95-x106; memory-bound contiguous rows at 4M x0.77-x1.04), and the 64-bit flat argmax runs NumPy's single-pass tournament -ceac36f2.x0.89->x2.65-np.mean/np.sumalong an axis with a large output: the identity seed and the mean's division run as typed block passes (np.mean(a, axis=0)of 3x1M float32 was x0.29;mean(axis=1)over 3-wide rows stays below NumPy) -3f22d30a.x1.43->x1.83-np.convolvelong products without the OpenBLAS backend (1000x1000 float64 was ~x0.9, complex128 x0.25) -38526155.x1.15->x5.9- complex128divide,multiplyand integer-exponentpower, now on NumPy's own formulas -dc9caff5,338d260a,3e7d6b33.x1.26->x4.0-np.modfplain,out=and promoted calls (out=at 10M was x0.40) -3384a0b5.x1.0->x16-np.array_equal/np.array_equivfused early-exit compare (x15-x16 at N=1000, parity at 100K, instant on an early mismatch) -44604193.x1.05->x6.4- everynp.randombit generator fills through a bulk layer: all 105 engine x op x size cells beat NumPy -aa9905b7.x1.08->x5.58- legacyRandomStatesamplers (hypergeometric was x0.96, zipf x1.12, multinomial x0.97; the samplers bound by the C runtime'spow/log/expstay at x1.17-x1.49) -910a6480.x0.65->x17.6-np.evaluatecomparison, logical,whereand bool trees vectorize instead of running the whole tree scalar (a > 0.5was x0.16) -939d0636.x1.0->x22.2-np.evaluatemixed-dtype trees vectorize through lane groups (i4*2+f8x14.2 andf4*f8+f4x22.2 at 100K; 1K cells sit at the per-call floor) -3631fcc7.x0.86->x53-np.evaluatereductions stream block by block instead of materializing their child, andAny/Allstop at the deciding block (any(a > b, axis=0)was x0.37;count_nonzero(a > b)at 100K sits at the read-bandwidth ceiling) -ee639862,0f8819b7,129fbaba,78692d91,ceac36f2,0f78f6e2.x2.06- a smallnp.evaluatecall skips the iterator like NumPy's trivial loop:a*b+cat n=8 runs in 255 ns vs NumPy's 526 (out=: 75 ns) -30dbfe0d(+df799381).🎯 Parity & Fixes
np.max/np.min/np.ptp, flat and along an axis, run NumPy's own reduction schedules - the sign of a ±0 tie and a canonical-vs-payload NaN now match on every non-broadcast layout -1d79ce61(+129fbaba,f8f91b3d).np.nanmax/np.nanminare NumPy'sfmax.reduce/fmin.reducebit for bit: complex128 skips NaN (it propagated it), float16 on non-C layouts read the wrong elements, and an all-NaN float16 slice returns its first NaN -de6333dd(+268d91fd).divide(1 ULP off on ~33% of operands),multiply(NumPy's fusedsimd_cmul),powerwith an integer exponent (up to ~1e14 ULP off) andabs(NumPy's fusedsimd_cabsolute; 35.5% of values were off) -dc9caff5,338d260a,3e7d6b33,d1b89c11.percentile/quantile/medianbit-exact - NumPy's two-branch_lerpin each dtype's arithmetic,medianas the mean of the middle pair, and a view at a non-zero offset no longer reads the wrong data -d2c8d26e.np.round/np.aroundwithdecimals != 0portPyArray_Round- negative decimals threw, complex values stayed unrounded, float32 rounded in double, and the result dtypes follow NumPy -6db8238a.sum/prodof a size-0 or size-1 array widen the accumulator by NEP 50 like any other size -23ea2ad9.np.array_equal(a, a)with a NaN inareturnedTrue; NumPy returnsFalseunlessequal_nan-38f9a14e.53602da9.mod/floor_divide/fmodby zero give ±inf like NumPy (they gave NaN), and float16add/subtract/multiply/divideon theout=/where=/ strided paths compute in float32 like NumPy's HALF loops (they double-rounded through double) -df232348,b53a0cc1.maximum/minimumof float16 with float32/float64 keep the NaN's bits -ac89847a.np.linalg.eig/eigvalsof a float32 matrix with complex eigenvalues return NumPy's complex64 values (they kept double precision) -0c02740c.np.real/np.imag/.real/.imag,view(dtype)andgetfielddid not keep their owner's buffer alive, so after the owner was disposed (as insideeigvals) the next allocation overwrote them;resize(refcheck: true)now sees such views -59890501.np.convolve/np.correlatewithout the backend are byte-exact on float32 positions under 32 terms and complex128 positions under 8 (NumPy's scalarsdot/zdoturegimes), and complex products with inf/NaN follow NumPy (convolve([1+0j], [inf+0j])isnan+nanj) -38526155,058f4f27.np.arctan2(complex)corrupted the heap and crashed the process; it raises NumPy'sTypeError-dc2fc075.cbrt,floor,ceil,trunc,deg2rad,rad2deg,floor_divideandmodraises NumPy'sTypeError(it raisedNotSupportedException, and a zero-size complex input returned an empty array);IncorrectTypeExceptionnow derives fromTypeError-9a1eeb7e,2e205a1f.np.modfaccepts int, float16 and bool input like NumPy (it rejected them), float16 byte-exact -3384a0b5.np.traceof uint8/uint16/char accumulates in uint64 like NumPy;ndarray.flatof a 0-d array has shape(1,);np.take,ndarray.item()andlen()of a 0-d array raise NumPy's verbatim texts -e84fd2f1.np.argsortof an empty array with sort axis >= 2 overran the heap -74656f13.flags.owndata), like NumPy -32162b64.!(logical not) on a non-contiguous array returned wrong values -76b2ab9d.np.choose,tofile,tobytes,ogrid,poly1d,polyfit,multivariate_normal,apply_along_axis/apply_over_axes,flatiterslice/fancy access, cross-dtypeSetDataand the float16sum/meantemporaries;NDArray.ReplaceDatacould free another array's buffer -1dd23bbe,6f2ec473,7e60327d,e84fd2f1.Generatorparity (it shipped in 0.70.0) -1855ea9e.integerswith ahighpast the dtype returned constant data (integers(0, 257, dtype: np.uint8)was all zeros); it raises NumPy's error.lock, as in NumPy.choicewithout asizereturns one element (it returned shape(1,));permutedfollows memory order and casts intoout=safely; a-0.0scale, shape or range is rejected; a()size keeps float32.RandomStateparity -c1e0feb5,f87b4d04.RandomState()took its seed fromEnvironment.TickCount, so instances made in the same millisecond were identical; it uses OS entropy.choiceignoredreplace=Falseand searched the probability CDF from the wrong side;randint(dtype: bool)crashed.legacy-distributions.cline for line:gamma(shape<1),binomial,negative_binomial,poisson,f,pareto,power,standard_cauchy,multinomial,hypergeometric,zipf,geometricandvonmisesused other algorithms.SeedSequence- a typednullseed array seeded the empty sequence (the same stream on every run; now fresh entropy, NumPy'sNone), andSeedSequence(new[] { -1 })silently seeded0xFFFFFFFFinstead of raising -0e39714b,65bdea7b.result_typeof four or more operands dropped every other pair (result_type(int8, int8, float64, int8)was int8; it is float64) -896f02b2.np.evaluate- flat and axisSum/Mean/Prod/Min/Maxfollow NumPy's own schedules bit for bit (they matched the value only), a flatMeanof an integer child is NumPy's buffered float64 sum, NEP 50 typing gaps are closed, and two closures over one lambda no longer share a kernel -53602da9,e1400974,129fbaba,df799381.nd["1:3"]) re-parsed its pattern on every call because the staticRegexcache thrashed - 9-18 µs -> 0.6-2.8 µs per parse -28fa9d79.🧰 Testing & Tooling
numpy.polynomialunit (139,882 cases),np.ma(68,860),np.evaluate(29,563),np.emath,ndarrayinstance members, and complex128 / Decimal coverage, with a dtype-spread gate (at least 4 dtypes per op) and an applicability gate -57545200,67f8f96e,7a262afc,e84fd2f1,55a2a6a7,192f456c,1dd23bbe.np.randomoverload against NumPy 2.4.2 by its exact C# signature (493 members, 5 engines x 10 seeds, the full post-call state), with coverage gates and a nightly soak under 10 fresh seeds -20a5b60d,1c7ac693,8f70af7c,ecdab2fb.fd058f03.💥 Breaking Changes
DTypereplacesSystem.Typeas the dtype currency -c916c579,68a02258.NDArray.dtypereturnsDType(wasType), andnp.float64,np.int32and the rest areDTypedescriptors (wereType);DType->Typeis explicit (a.dtype.type), anda.dtype.Nameisa.dtype.name.DType: theType/NPTypeCodetwins are gone (aType,NPTypeCodeor dtype string still converts implicitly), andlinspace/nancumsum'stypeCode:isdtype:.result_type,promote_types,find_common_type,common_type, ...) returnsDType, andisdtyperejectsissubdtype's vocabulary ("floating"), as NumPy does.finfo.dtype/iinfo.dtypereturnDType(wereNPTypeCode), and the pythonnet dtype maps takeDType-ea080d8e."Safe"raisesValueError) -896f02b2.np.int32/np.int64report the local NumPy's LLP64 type numbers and chars (7/'l'and9/'q'; they were hard-coded to Linux's5/'i'and7/'l') -72e4d29c.ValueError(wasNumSharpException); it derives fromArgumentException, socatch (NumSharpException)no longer catches it -f319b657.np.maximum/np.minimum/np.fmax/np.fmintakeoutas the third positional argument (NumPy's order), notdtype; passdtype:by name -e5afe84c.np.logical_and/logical_or/logical_xor/logical_notreturnNDArray(wasNDArray<bool>), andnp.modf's tuple fieldIntergralisIntegral-3384a0b5.ndarray.shape = ...raises NumPy'sAttributeErrorwhen the layout cannot be reshaped in place (it silently copied); usereshape-6f2ec473.np.nanmax/np.nanminof an empty reduction raise NumPy's "zero-size array to reduction operation fmax which has no identity" (they returned a 0-d NaN) -de6333dd.new MT19937(42)seeds like NumPy'snp.random.MT19937(42)(throughSeedSequence), not likeRandomState(42);mt._legacy_seeding(42)is the old stream.MT19937's non-NumPy members (NextUInt32,NextDouble,NextInt,NextLong,Next,NextBytes,Seed,SeedByArray,SetState,Key,Pos,Clone) are gone, andPcg64StateDataisPCG64.State-c1e0feb5.RandomStatestreams ofgamma(shape<1),binomial,negative_binomial,poisson,f,pareto,power,standard_cauchy,multinomial,hypergeometric,zipf,geometric,vonmises(andgumbel/logistic/rayleighat scale 0) change to NumPy's values;binomial(int n)isbinomial(long n); inputs NumPy accepts (dirichlet([]), a NaN alpha,negative_binomial(p: 0)) no longer throw -f87b4d04.long:randintdefaults to int64 (was int32),random_integers/permutation/choice/multinomialreturn int64,NumPyRandom.Seedis auint(wasint), andSeedSequence.generate_statereturns anNDArray(wasArray) -eebe85a6.Generator(bit_generator)(wasbitGenerator),PCG64(seed)and the other bit generators (wasseedSeq, now anyISeedSequence), anddefault_rng(seed)(wasbitGenerator/generator) - named-argument calls must be renamed -eebe85a6,1c7ac693.NumPyRandom.uniform(NDArray low, NDArray high, DType dtype)is removed; two-array calls bind NumPy's broadcasting overload (float64;.astype()the result for another dtype) -39d446e0.