MSVC strictly enforces const on tritonNumCTAs/tritonMaxNreg/
tritonEnableFpFusion/tritonDisableLineInfo calls. Use non-const
reference since these methods aren't const-qualified.

MSVC's STL marks string_view::substr() as non-constexpr (it can throw
via _Xran), which breaks llvm/Support/TypeName.h static_assert at
lines 74 and 98. Since Triton JIT also requires Linux-only PTX/NVRTC
toolchain, skip Triton entirely on Windows builds.

…, ccache optimizations

- Replace mingw-w64-x86_64-SDL (dropped from MSYS2) with SDL2 in all
  Windows workflows and msys2-base-setup action, matching the CPU
  workflow that was already updated
- Wire --oom-critical-threshold through libnd4j/pom.xml (default 97%)
  so the critical SIGKILL threshold is above the graceful threshold,
  fixing inverted thresholds where critical (92%) fired before
  graceful (95%) making graceful shutdown unreachable
- Add file_stat_matches to SmartCcache.cmake CCACHE_SLOPPINESS to
  match the CI action config and avoid unnecessary cache misses when
  file stats change but content doesn't
- Disable CCACHE_LOGFILE in CI setup actions (Linux, Windows, macOS) —
  was marked "temporary" diagnostic but left enabled, adding I/O
  overhead on every compilation

…y-based assertion

Test now verifies routing switches when device 0 cannot fit the
allocation (100MB free vs 500MB request) rather than relying on a
soft-limit percentage threshold.

Cross-platform floating point differences in LSTM computation can exceed
1e-7 precision. Relaxing to 1e-5 which is still a tight tolerance for
neural network output comparison.

All 3 matrix variants (base, compile, cudnn) were saving separate
ccache entries under different keys, creating 12 saves of 3-7 GB each
against GitHub Actions' 10 GB per-repo cache limit. This caused
constant eviction — most variants had cold caches on every run.

Changes:
- Drop ${{ matrix.helper }} from cache-suffix so all variants share
  one cache key per platform/CUDA version (4 total instead of 12)
- Remove cache-family (no longer needed when all variants share a key)
- Reduce max-size from 8G to 2G so 4 caches fit within the 10 GB limit
- Add continue-on-error to save step (2nd/3rd variant gets "key exists")

ccache's content-based hashing ensures correctness: identical object
files get cache hits regardless of which variant compiled them.

Includes updates to GGML quantizers, convolution utils, YOLO output
layer, masked reduction, SameDiff serialization, regression eval tests,
and CUDA memory pool.

…ingTests

NDArray.muli handles broadcasting automatically, making the explicit
BroadcastMulOp unnecessary.

Updates to op codegen, preprocessors, MultiLayerNetwork, reduce ops,
clip helpers, BroadcastableOp, MKL-DNN reshape/softmax, SDMath,
serialization, BaseNDArray, DynamicCustomOp, Nd4j factory, and tests.

…ind (#12 batch divergence)

Root cause of testFreshInputCloseBetween[bgeEncoder][5] diverging ~7% ONLY in
the full-class batch: NativePlanCache returns a REPLAYING plan (warmed by an
earlier test with identical shape key on the same thread) to a NEW Java
executor. The Java side syncs straight into frozen state (native phase>=1), and
performPreReplaySync's broad-H2D gate 'firstFrozenExec = !slotBySlot &&
executeCount_==0' is FALSE for the cached plan — so only the variable-filter
(placeholder) subset is H2D-prepared and the new executor's WEIGHT DataBuffers
are never synced; slot-by-slot rewarmup after refreshProtectedWeightBuffers'
unseal reads stale device memory. Isolated runs compile a fresh plan
(executeCount_==0 -> broad sync) and pass — a textbook cross-test lifecycle
leak.

Fix: encode the missing invariant — when refreshProtectedWeightBuffers detects
weight DataBuffer rebinds (added/removed>0), the NEXT external-input H2D
prepare must be broad. Implemented as an encapsulated, DSP_DIAG-logged flag
(never poked directly, same contract as the GraphSegmentExec state methods):
markWeightRebindNeedsBroadSync() at the rebind site,
consumeBroadPreReplaySync() OR-ed into performPreReplaySync's broad gate
(consumes + logs), read-only weightRebindBroadSyncPending() widening the
execute-level broadPrepare (consumption stays owned by prereplay).

Verified: full DspSlotLifecycleAuditTest batch — [5] no longer diverges.
Remaining 3 deterministic failures (testSlotPhaseVisibility[46/47/49],
longViewChain/JIT 'null output slot 0 in REPLAYING') are PRE-EXISTING and
unrelated: instrumented run shows 0 BROAD_SYNC firings and 0 workspace-detach
firings in that fixture; they reproduce in isolation and are tracked as a
separate fix. Mechanism credit: instrumented A/B isolation-vs-batch
investigation (findings in /tmp/n12-findings.md at time of fix).

…lose-path lifetime contracts

Root (ground-truth traced, /tmp/n52-gt.json): InferenceSession closed the CAST
placeholder copies immediately after DSP execution — but DSP modes that do not
stage externals (EMULATED_REPLAY: segments never freeze, staging never
allocates) install view slots DIRECTLY over the cast copy's DataBuffer. The
close fired between AUTO_SEAL and the DspHandle read (DB_DELETE_BUFFERS of the
view's parent is the event immediately preceding GET_SLOT_OUTPUT dbClosed=1),
leaving a dangling wrapper: shapeInfo reads clean, isValid()=false,
getOpaqueNDArrayLength returns 0, Java getSlotOutput -> null ->
'1 null output slots in REPLAYING phase' (testSlotPhaseVisibility
longViewChain/EMULATED_REPLAY, deterministic; NVRTC/PTX siblings via the
borrower-switch path).

Fixes (lifetime contracts, no guard relaxation):
- InferenceSession: DEFER cast-copy close to the START of the next DSP
  execution (views re-mint over fresh casts by then); flush at session reset so
  nothing outlives the session — preserves the eb949f1 leak guarantee
  (bounded to ONE call's copies, no unbounded tl_castCache-style growth).
- invalidateExternalViewSlotsOnReacquire (borrower switch): full
  SegmentLifecycle::invalidateSegmentCaptures (covers CUDA-graph, JIT AND
  emulated handles) before resetForWarmup — reset alone left the emulated
  handle replaying past freshly-nulled slots.
- refreshStaleViewWrappersInSegment: when the shape cache is absent
  (never-frozen segments) re-mint from the stale view's own
  ConstantShapeHelper-owned shapeInfo instead of demoting to a null slot;
  guard the dtype-fix cache write-back against OOB on that path.
- closeOwnedArrays: never close protected external buffers (current inputs /
  protected constants) — zero-copy outputs of view chains share them.
- DynamicShapePlanExecutor.getSlotOutput + JNI: permanent diagnostics for this
  family (OPAQUE_LEN_ZERO with db validity, DB_DELETE_BUFFERS ground-truth
  close trace, GET_SLOT_OUTPUT_NULL, Java null-branch reasons, WRITE_SLOT
  plan/value pointers, pass-3 first-slot detail).

Verified: testSlotPhaseVisibility 49/49 (was 3F deterministic); full class +
DSP core gate: cascade ERRORS 10 -> 0. Remaining failures are a DIFFERENT,
partially pre-existing family unmasked by the error fix: bgeEncoder NVRTC_JIT
numeric divergence ~4% (worstRel=0.041 vs rtol=0.01) — tracked separately.
Includes the borrower-switch view-slot invalidation this strengthens (developed
in the parallel session's #70).

…all-green campaign

Two verified arcs, gated by the full 3-step DSP regression suite:
step1 3/3, step2 51/51, step3 1593/0F/0E/0S (exceeds 1590/0/0/0 milestone);
DspSlotLifecycleAuditTest 448/448, DspBufferAliasAccuracyTest 189/189;
whisper JFK transcript word-perfect via incremental KV decode, espeak oracle
PASS, WhisperMergedBranchParityTest green at 2e-5.

ONNX If / merged-decoder import (whisper STT end-to-end):
- If.kt: implicit branch captures materialized as declared inputs; two-axis
  branch namespacing (variables + op own-names) with linkage validation;
  multi-output If via switch/merge (16 dropped present.* outputs); eager
  suspension across composition; OnnxIRGraph node-name dedup (_dedupN);
  suggestDynamicVariables contract warning; 1D-conv importer rule.
- Interpreted-If execution: null-marker discipline (dead switch sides publish
  explicit markers; transitive skip stops at Switch AND Merge; merges defer on
  ABSENT vs skip on all-markers with bounded post-pass); strided-view
  placeholder materialization; Switch tolerates null data; Identity dead-path
  passthrough; ':1'-suffix consumer fallback removed (poisoned live frames).
- Decode: optimum empty-present retention contract (isEmpty() || length()==0
  at 3 sites — encoder presents in with-past branch are zero-length dummies
  meaning caller RETAINS pasts); dynamicMergedIfDecodeLoop (position from
  token count, not shape(past)); minNewTokens stop-suppression; DSP compiler
  If-frame gates; concatCpuGeneric empty-input filter + bounds precondition.

CUDA DSP campaign (from 9F/58E + deterministic JVM crashes to all-green),
every root convicted by direct evidence before the fix:
- #62 pinned arg tables: Triton per-kernel/consolidated arg-table pinned host
  blocks are baked as H2D sources of captured merged graphs; teardown freed
  them under the live graph (host-side SIGSEGV inside cudaGraphLaunch,
  free-log pointer-match 697/852). Capture-ownership transfer:
  tl_capturedHostPtrs -> replay handle, CudaMemoryPool::relinquishPinnedHost,
  7 guarded free sites.
- #63v2 GPU modules: TritonGraphBackend is a singleton whose compiled cache
  outlives plans; unloading modules at teardown killed baked kernel nodes
  (CUDA 98, driver-API node autopsy + unload logs), and handle-death unloads
  killed cache-shared modules (order-dependent '[cast] failed').
  CapturedModuleRegistry.h: owner+handleRefs refcount; unload only when last
  holder releases.
- #65 writeOutputSlot: oldBufAddr capture ran NDArray::specialBuffer()
  SELF-HEAL on the previous borrower's closed view (MALLOC_PERTURB_ registers
  RBX=0xaaaa..; allocateSpecial ws=0xaaaa.. bytes=8). Raw special() peek
  behind the same isValid() contract as the delete guard.
- #66 stream race: prereplay Step2 H2Ds fresh externals on the LC stream,
  ensureAndSyncStagingBuffers D2D-read them on the DSP stream with NO
  ordering (VERIFY-syncs masked it; ~70-param load exposed it). Thread-local
  LC->DSP order event, capture-gated via DebugHelper::streamIsCapturing.
- #67 plan cache: full-planBytes FNV in NativePlanCache::Key — two graphs
  sharing outputs/shapes/count/mode/thread can no longer serve each other's
  plans (acquire-time hash, hot path untouched).
- #69 borrower switch: dispatchNativePlan(+newBorrower) — Java executor's
  first dispatch flags a potential borrower switch; native invalidates
  ext-fed view slots + resetForWarmup/markArgsStale on affected segments.
  (Cache-HIT-triggered invalidation was WRONG — same-borrower re-dispatch
  cleared live captured-graph slots, 67F regression, reverted.)
- #72/#74 view re-mint gates: FF var-ext break exempts view-capable slots
  (they must reach view-verify every exec — dead views over closed inputs
  survived into REPLAYING as the '[cast]'/'Owner died'/Workspace-SIGSEGV
  family); validateReusableSlotArray re-mints ext-fed views when the cached
  DataBuffer OBJECT differs from the current external's (ABA-proof; blanket
  re-mint broke frozen baked-address contracts, 9F/19E, refined same cycle).
- #77 staging currency: resolveViewInput preferred staging unconditionally;
  once exec target flips to direct SBS staging is never refreshed — views
  object-matched to exec-1 staging read iteration-stale values forever
  (deepAttentionQKV varying#4 bit-identical across NVRTC/PTX).
  stagingMaintainedThisExec_ gates the preference.
- JIT_TOL {1e-3,1e-3} for NVRTC/PTX/EMULATED in the alias test (TRITON_TOL
  precedent): verified sign-alternating ~1e-4 softmax-amplified scatter =
  cross-kernel-algorithm float32 accumulation variance; staleness-class bugs
  are coherent and >=5e-3. Applied only AFTER the 5e-3 staleness fix.

Permanent diagnostics: merged-capture NODE[] autopsy (driver-API
cuGraphKernelNodeGetParams + cuFuncGetModule), pinned/module lifecycle logs,
GET_SLOT_OUTPUT buffer-state fingerprint, VIEW_MINT_SRC, DataBuffer
workspace-attach fingerprints, graphExec destroy logs, PARAM_BANNER +
audit.fixture/audit.mode + alias.fixture/alias.mode narrow-run filters.
New tests: WhisperMergedBranchParityTest (branch-parity oracle, audio-free),
DspAliasCrashReproTest (negative-space record of the crash isolation).

…BLAS (#53 both roots)

Root 1 - false device-actuality after merged CUDA-graph replays: the
merged tick loop blanket-tickWriteDevice()'d every slot in the group
range, marking never-graph-written buffers device-actual; later host
reads D2H'd uninitialized device memory (initcheck: 18,485 findings via
DataBuffer::syncToPrimary). Merged captures now populate the handle's
per-capture slot audit (exact per-slot node deltas for captured gap
slots, range-level entries for fused islands) and merged replay routes
through postReplayFixupRange - the segment fixup core refactored to
take (handle, slotRange) - ticking only audit-confirmed graph-written
slots, re-executing host-only ones, and blanket-ticking LOUDLY
(FALLBACK diag) when no in-range audit exists. An in-range coverage
guard prevents a stale foreign-segment lastCaptureAudit_ from
suppressing ticks.

All audit state is encapsulated behind CudaGraphHandle's API
(resetCaptureAudit/recordSlotAudit/recordAuditEntry/hasCaptureAudit/
getCaptureAudit; private container). Merged-capture TLS transitions,
island filter, capture-workspace bind/rewind, capture-scoped cache
clears, and merged-handle release are consolidated into logged helpers
(enterMergedCaptureTls/exitMergedCaptureTls, set/expandIslandFilterTls,
bindCaptureWorkspaceTls/rewindCaptureWorkspaceTls,
clearCaptureScopedCachesTls/clearCaptureReplicateCacheTls,
releaseMergedHandle).

Root 2 - GEM_NVRTC_JIT/GEM_PTX_JIT were the only ModeContract modes
without requiresDeterministicCublas: their cuBLAS gap/fallback GEMMs
ran CUBLAS_DEFAULT_MATH (tensor cores) against the PEDANTIC
SLOT_BY_SLOT reference - deterministic 1e-3..1e-2 drift on norm-heavy
graphs, expressed whenever async JIT compilation lost the race to
replay (load-dependent: full-suite failed [4]/[12] bit-identically,
every narrowed subset passed). JIT modes now require PEDANTIC math
like every sibling mode.

Test harness: audit.fixture/audit.mode narrowing accepts comma lists
for order-dependent bisects.

Verification: testReplayAccuracy* 3x 49/49 (was 1F/2F/0F lottery, then
2F/2F/2F deterministic after root 1 alone); initcheck 0 findings (was
18,485); DspSlotLifecycleAuditTest 448/448; gate steps 1-2 green; core
batch 1593 run / 1 failure - that failure is a distinct pre-existing
close-weight Java-readback mapping bug (native output proven correct
in the same diag window), tracked separately.

… recalibration)

The {1e-3,1e-3} JIT calibration was mostly absorbing CUBLAS_DEFAULT_MATH
tensor-core drift in NVRTC/PTX gap GEMMs, fixed in 374a69c
(requiresDeterministicCublas for JIT modes). Probe at {1e-5,1e-5}:
188/189 pass; sole residual is deepAttentionQKV/NVRTC_JIT varying#4 at
6.6e-5 abs / 1.1e-3 rel - genuine accumulation-order scatter on the
deepest softmax chain. New JIT_TOL {2e-4,1e-3} gives 3x headroom over
the measured residual and is strictly tighter than before on both axes.
Verified: DspBufferAliasAccuracyTest 189/189.

…Ms ran DEFAULT/TF32 (#55)

cuBLAS handles are thread-local, so platformBeginExecution's
CUBLAS_PEDANTIC_MATH only ever covered the plan thread's handle. Gap and
fallback GEMMs dispatched from executor/pool threads acquired their own
fresh handles, which CublasHelper::handle()'s lazy-TF32 logic then set to
DEFAULT/TF32 - producing bit-identical ~1e-2 drift against the PEDANTIC
SLOT_BY_SLOT reference on norm-heavy graphs. Expression was load/order
dependent (thread routing), so it surfaced only in batch JVMs: the
recurring attnSingleLayer/PTX_JIT L0_ln2 failure family
(testReplayAccuracy and testWarmupRecycleInput, ref=-0.290945 vs
actual=-0.295453 across runs and methods).

Fix: depth-counted atomic deterministic window
(CublasHelper::enter/exit/inDeterministicWindow). handle() now converges
the ACQUIRING thread's handle to PEDANTIC while any deterministic plan
execution is active, keeps caller-managed state untouched when
tl_cublasLtDisabled is set, and lazily returns handles to TF32/DEFAULT
after the window closes. platformBeginExecution opens the window in its
requiresDeterministicCublas branch; platformEndExecution closes it; both
crash-leak cleanup paths (begin-side and end-side tl_cublasLtDisabled
leak resets) balance the depth, which clamps at zero.

Also ships the frozen-readback address-trace instrument for the open
close-weight investigation: ND4JSystemProperties.DSP_READBACK_TRACE
gates READBACK_TRACE/CACHE_BUILD lines in DynamicShapePlanExecutor's
zero-copy refresh path (src/dst device addresses per output copy, for
correlation with native DB_DELETE_BUFFERS pool-reuse traces).

Verification: 2-class repro (DspBufferAliasAccuracyTest +
DspSlotLifecycleAuditTest, ~1-in-3 failure rate pre-fix) 3x 637/637;
full DSP core gate 3/3 + 51/51 + 1593/0/0/0.

…ssion B)

The FP16 cast cache's skip-assign guard keys on raw source DataBuffer
pointers (sourcePtrs). Those objects are freed at plan teardown and the
heap reuses their addresses, so a successor plan's same-shape constant
at a recycled address matched a retired guard entry and skipped the
cast refresh - its matmuls then consumed the PREVIOUS plan's cast
weight values. Signature: bit-identical ~1e-2 drift on attnSingleLayer
L0_ln2 at the freeze boundary (#4/#5), hopping across test methods
(testReplayAccuracy/testWarmupRecycleInput/testFreshInputCloseBetween)
and execution modes (NVRTC/PTX/EMULATED - which ruled out GEMM math),
inputs proven clean, batch-only (~1/2 of 11-class core-batch rounds),
never reproducible isolated.

Fix: global cast-cache epoch (atomic) bumped in
platformFreePlanResources when a plan's constants are freed; each
thread's CastCacheSide lazily drops its pointer-identity guards on
epoch mismatch at the next cast (cached cast arrays stay - content
refreshes via assign once the guard is gone). Steady-state decode has
no teardowns, so the skip-assign optimization is untouched there.
DSP_DIAG(MEMORY) on bump and per-thread sync; DSP_DIAG(EXECUTE) added
to the #55 deterministic-window enter/exit and to cuBLAS handle mode
transitions.

Also: permanent recycled-input integrity assertion in
testWarmupRecycleInput (execution must never mutate caller inputs) -
the discriminator that proved inputs clean.

Verified: 4x 11-class core batch - expression B zero occurrences
(pre-fix ~1/2 rate). The remaining close-weight readback mis-map
(expression A, pre-existing, values = raw pipeline stages) became
frequently reproducible (3/4) under the new allocation pattern and is
tracked separately with the READBACK_TRACE instrument.

…ky/batch-only protocol (#29)

…mily)

A plan checked out from the shape-keyed native cache retained raw
pointers into its donor's pool-freed device buffers, in two places:

1. Arg/staging identity: the EMULATED/replay fast path trusted the arg
   generation counter, which only advances when NATIVE code observes a
   change. When the Java resolver replaced a placeholder (new INDArray +
   new device buffer) the gen stayed current and the captured args kept
   reading the freed address. The fast path now recomputes the (cheap,
   pointer-hash) input addr key and demotes to the full path on baseline
   mismatch (segments.cpp; the v1 compare against
   lastExternalInputAddrs_ was vacuous - those are re-recorded at every
   execute entry).

2. Slot readback identity: the step-3 cached-reuse path installed the
   plan-cache wrapper into outputSlots_ while pooled/frozen op contexts
   kept their own output arrays - kernel wrote the ctx array, readback
   D2H'd from the wrapper's stale device pointer. cached-reuse now
   rebinds the pooled context output to the reused wrapper on DataBuffer
   mismatch (CACHED_REUSE_CTX_REBIND, slotexec.cpp).

Also: constant DataBuffer death now bumps the cast-cache epoch directly
from deleteSpecial (extern-linkage hook - SameDiff.close() frees
constants AFTER plan destruction, outside the previous bump site).

Why it looked flaky for two days: the stale pointers were dereferenced
on EVERY run, but the freed blocks usually still held the donor's data,
and same-shaped sibling tests compute identical values - reading the
corpse gave the right answer by luck. It only asserted when the pool
re-handed a block with different content.

New deterministic regression guard:
DspSlotLifecycleAuditTest#testCachedPlanReuseAfterDonorCloseWithPoison -
donor runs to frozen/replay, closes; freed pool blocks are immediately
repoisoned with sentinel-filled allocations; a same-shape borrower then
checks out the cached plan and must match the reference. Turns the
whole bug class into an 11-second always-on check (49 params).

Verification: poison test 49/49; hot-fixture narrowed sweep 287/287;
full DSP core gate 1642/0/0/0 (1593 + 49 new).

The audit-driven postReplayFixupRange (374a69c) re-executed every
0-node audited slot per replay. View-capable ops (reshape/expand_dims/
squeeze/strided_slice) landed in that set: ~26 wrapped executeSlot calls
per decode token (sync guards + prepare/register ceremony) costing
~2.3s per 250-token decode while the ops themselves total ~3ms — GPU
sat at 38% utilization. View outputs ALIAS their source DataBuffer and
device-actuality lives on that shared buffer, which the audit tick of
the producing slot already maintains; re-execution buys nothing.

Found by diffing --op-timing output against op-timing/SMOLDOC_IDEAL.csv:
reshape x4344 + expand_dims x2197 live executions present vs ZERO in the
reference profile.

OPTIMAL 250-tok lateSteady: 38.54 -> 65.39 tok/s (old TF32-era band was
62.3-64.1); SLOT_BY_SLOT 30.04 -> 39.25. Poison test 49/49; DSP gate
3/3 + 51/51 + 1642/0.

getOpNum resolves by NAME via DifferentialFunctionClassHolder (one class
per name). BooleanNot (legacy TRANSFORM_BOOL, opNum 7) collides with
LogicalNot (CUSTOM "boolean_not") since 3a167f2 — the writer picked
the custom class and serialized its HASH into a legacy-typed FlatNode,
which LegacyOpMapper cannot read ("No known transform bool op for op
number: 2090978343"). This broke SDNB round-trip cloning and therefore
ALL model import (Benchmark phase IMPORT_MODELS failed).

asFlatNode now takes node.opNum() from the actual instance for legacy
types; name lookup remains for CUSTOM/LOGIC/control kinds where the
number is a hash or fixed table.

Trims (per-token driver-call reductions, #38):
- plan-owned ownedCrossStreamEvent_ reused across executions (was
  cudaEventCreateWithFlags + cudaEventDestroy per token); mirrors
  executionCompleteEvent_ lifecycle incl. device-change re-create and
  teardown.
- single cudaGetDevice in platformBeginExecution (reused for capture
  gate, ctx->deviceId, event device); platformEndExecution reuses
  ctx->deviceId for the paired exec-counter decrement.
- NativeOps_dsp.cu: WARNING comment on the input-validation loop — its
  setSpecialBuffer(nullptr) recovery is LOAD-BEARING for cached-plan
  checkout (gating it regressed the poison test in 36s); do not skip it
  until checkout-time sanitize exists.

Policy knob: the #55 deterministic-math window now yields to the
EXISTING explicit fast-math opt-in (nd4j.cublas.tf32 / ND4J_CUBLAS_TF32)
— a deliberate, documented speed-over-reproducibility choice; parity
tests never set it. run-benchmark.sh grows --cublas-tf32/--no-cublas-tf32
feeding that property. A/B measured: deterministic == TF32 on this
model within noise (30.50 vs 30.73 SBS pre-view-fix), so the default
stays deterministic.

Verification: OPTIMAL 250-tok 65.39 tok/s; poison 49/49; DSP gate
3/3 + 51/51 + 1642/0.

…57, closes #46 WS-O3)

bge-base [32x512] warmup under memory pressure poisoned the CUDA context with
error 700 within two failover events (kompile embedding lane: LFM resident,
then bge). Root: failover-served buffers (managed/pinned direct allocations)
were freed with SYNCHRONOUS cudaFree/cudaFreeHost — stream-oblivious — while
async cuBLAS GEMM consumers were still in flight; the UVM unmap under a
running kernel is a driver-level fault invisible to memcheck. Pool buffers
survive the identical dtor-after-enqueue pattern because cudaFreeAsync is
stream-ordered behind the consumer.

No syncs added (WS-N mandate) — all ordering is async:
- Event-deferred direct frees: record an event on the consumer stream at free
  time, reap via non-blocking cudaEventQuery at allocate/failover entry and
  releaseAll (DSP_DIAG MEMORY: DEFER_DIRECT_FREE / DRAIN_DEFERRED_FREES /
  IMMEDIATE_DIRECT_FREE).
- Managed fallback is HOST-RESIDENT (preferred location CPU + SetAccessedBy
  pre-mapping + prefetch-to-host enqueued on the consumer stream): prefetching
  into the full device is a best-effort no-op and demand faults there are
  unserviceable (zero evictable frames).
- memsetSync memsets on the pool's resolved alloc stream (was
  cudaStreamPerThread — ordered against nothing; 906 use-before-alloc races
  flagged by compute-sanitizer --track-stream-ordered-races); the trailing
  host sync is dropped on that path.
- Plan-wide tl_dspGapStream pin: slot-by-slot warmup ops now share ONE stream
  with pool alloc/free resolution (previously compositeReplay-only), making
  the free-time event ordering exact.

Repro: Dsp700BgeWarmupReproTest — deterministic bounded failover method
(device-only ballast to the soft-limit floor, 8x256 plan, ~11s) plus the
production [32x512] shape with its capacity requirement documented.
Verified: pressure method green (exec #0 6.4s / #1 3.7s, no 700), poison
canary 49/49, SMOLDOC 250-token lateSteady 65.86 tok/s (65.39 baseline held).

…evice budget (#58)

The bound was decorative on the CUDA pool: Environment.setMaxDeviceMemory
stored _maxDeviceMemory but CoreConfig had NO getter (unlike maxPrimaryMemory/
maxSpecialMemory) and the pool never read it — a 110M-param encoder's single
DSP plan reserved 22.4 GB against a delivered 13.5 GB bound (1.66x), blocking
any multi-tenant GPU packing.

- CoreConfig.h / Environment.h: add maxDeviceMemory() getter (int64_t so the
  -1 "unbounded" sentinel survives).
- CudaMemoryPool::allocate: when a bound is set (>0), read this process's OWN
  pool usage (cudaMemPoolAttrUsedMemCurrent via getStats — NOT cudaMemGetInfo,
  so co-tenant processes don't distort the accounting) and route any allocation
  that would push past the budget to allocateFailover (host-resident managed /
  peer — safe since #57) instead of growing device reservation. DSP_DIAG MEMORY
  DEVICE_BUDGET_EXCEEDED. Skipped during capture; off by default (-1 → a single
  atomic load, byte-identical behavior for every existing caller).

Enforcement is via Nd4j.getEnvironment().setMaxDeviceMemory(bytes) or the
SD_MAX_DEVICE_BYTES env var. NOTE: -Dnd4j.environment.maxDeviceMemory is NOT a
real property — no dl4j code reads it; callers must use the setter/env var.

Test DspDeviceBudgetEnforcementTest drives the real setter: budget 3 GB,
allocate 6 GB device-only, pool usage plateaus at 2848 MB (all 24 allocs spill,
none fail). Poison canary 49/49. Perf A/B vs reverted HEAD: 61.1 vs 61.0 —
neutral (the block is gated off when unbounded).

…ph replay

Enable genuine device-aware DSP sharding across non-peer GPUs (RTX 4090 dev0
+ RTX 3070 Ti dev1). assignDevices() partitions op-slots by available memory;
each device's contiguous run becomes a captured CUDA-graph island, and
cross-device boundaries are handled by async gap migration. All 5
DspMultiGpuShardingTest cases (7 instances) pass byte-exact (maxAbsDiff=0.0)
vs the single-GPU reference, including the 10-iteration captured-graph replay
across the non-peer boundary.

Five root fixes for one class of bug: "assignDevices split a group of slots
that some pass assumed lived on one device."

1. FusionPass (FusionPass.cpp): never fuse an elementwise chain whose slots
   span a device boundary. A cross-device fused *tail* gets no independent
   output allocation, leaving an uninitialized NDArray that crashes at freeze
   (buildSegments -> estimateSlotOutputBytes -> lengthOf()). Covers all three
   fusion types (ELEMENTWISE_CHAIN / BIAS_ACTIVATION / MATMUL_BIAS_ACTIVATION).
2. Batched-GEMM grouping (NativeDynamicShapePlan_batchgemm.cu): a matmul may
   only join a batched group on the same device.
3. In-place buffer aliasing (NativePlanCompiler.cpp): alias a unary op's output
   to its input slot only when both are on the same device.
4. Input migration (DynamicShapePlanExecutor.java): route each external weight
   to its consuming segment's device instead of the single primary device.
   Fixes the weight ping-pong (dev1->dev0->dev1) that broke captured replay
   with CUDA error 700.
5. Capture stream (NativeDynamicShapePlan_gpubackend.cu): run a secondary-device
   island's whole compile/warmup/capture/replay/direct pipeline on its own
   cudaStreamPerThread so its Triton/cuBLAS kernels launch in the correct device
   context (was: device-0 stream -> cuLaunchKernel "invalid resource handle" /
   cuBLAS err13 during capture).

Plus the device-aware infrastructure they build on: per-segment device-bind
guard, per-device cuBLAS / JIT / NVRTC / PTX caches, cross-device constant
replication, and async cross-device input/output migration.

Tested: DspMultiGpuShardingTest 7/7 green (exact); focused single-GPU
regression DspConcurrentPlanSharingTest#testOutputBufferIsolation 3/3 green.

NOT run (flagging for follow-up): the full DSP core regression batch (11
classes per AGENTS.md), the DSP benchmark/perf sweep, and the broader
single-GPU DSP suites. All five fixes are gated on targetDeviceId>0 /
numDevices>1, so single-GPU takes no-op paths, but the broad gate was not run.

Known leftover: pre-existing temporary ground-truth diagnostics remain
(MMUL_DIAG / PRE_GEMM_DIAG gated to the secondary device, CONSTANT_DIAG,
SETZERO_DIAG); these do not affect single-GPU runs and should be stripped in a
follow-up. One edge case is deferred: identity/view ops aliasing a *migrated*
cross-segment input (a lifetime dangling the MLP does not exercise, but a
reshape/permute-heavy transformer could).

…re fixed

Remove the temporary diagnostics added while root-causing the multi-GPU
sharding failures (roots now fixed in the prior commit):
- MmulHelper.cu: MMUL_DIAG, PRE_GEMM_DIAG (incl. a cudaStreamSynchronize +
  stream-device probe), MMUL_DIAG_FAIL, and the mmEntryErr peek. Kept the
  labeled cuBLAS error throws and the per-device usableWorkspaceSize guard.
- ConstantHelper.cu: CONSTANT_DIAG in replicatePointer.
- DataBuffer.cu: SETZERO_DIAG in setToZeroBuffers (kept the device-1 stream fix).

No logic change. DspMultiGpuShardingTest still 7/7 green (maxAbsDiff=0.0,
all 10 replay iterations).

…layout

platformGetOutputForDevice0 migrated a device-N plan output to device 0 with
cudaMemcpyPeerAsync — a raw byte copy that ignores strides. A permute/transpose
output is F-strided (e.g. [1,16]) even when labeled order='c', so building the
device-0 copy with canonical strides relabeled those bytes as a different layout
(the transpose): a deterministic wrong result on the sharded path (the earlier
device/host "coherence"/flaky symptoms were all downstream of it). Build the copy
with arr's exact shapeInfo (copyStrides=true) so the migrated output is
bit-identical to single-GPU. No dup is used (arr's bytes are already correct),
which also avoids an async dup-on-unknown-stream ordering hazard (all-zero output).

Also: materializeViewSlot materializes a secondary-device view output ON that
device (uses the producing slot's targetDeviceId — DataBuffer deviceId() can be
stale 0) and drops a dup() double-wrap that leaked the dup result; same dup() leak
fixed in TritonGraphBackend_execute.

New: DspMultiGpuShardingTest.testRepeatedShardedViewOpsMatchSingleGpu (permute
output across the non-peer boundary, 10-iteration replay). Full class 8/8 green.

platformMigrateSegmentInputs materializes a cross-segment VIEW input via
arr->dup() (a view shares its parent's buffer, so unlike an owned output it can't
be raw-copied / copyStrides-cloned). NDArray::dup fills the copy on srcMat's
context stream (new NDArray(...,getContext()); assign(this) — NDArray.hXX ~5273),
NOT the target peer-copy stream, so the async cudaMemcpyPeerAsync raced the
still-filling buffer and would migrate zeros. Record a reusable event (never
destroyed — a per-call destroy invalidates the enqueued wait and hangs) on
srcMat->getContext()'s stream and have the target copy stream wait on it.

Gated on isView(): the non-view boundary inputs the MLP tests exercise are
unaffected (full DspMultiGpuShardingTest 8/8 green). The view-INPUT path has no
dedicated repro yet (the memory-proportion shard split lands the dev0->dev1
boundary on a non-view op in buildViewMlp); the fix is grounded in the dup impl
and mirrors the verified view-OUTPUT handling (6b3a44f), which instead uses
copyStrides because an owned output needs no dup.

testShardedViewInputBoundarySweep moves the dev0->dev1 shard boundary across op
positions by sweeping nd4j.dsp.nonP2pBudgetFraction (0.1..0.9) — assignDevices
assigns ops in execution order proportional to the per-device budget. Sharded
output stays bit-identical to single-GPU at every split ratio (maxAbsDiff=0.0),
adding split-ratio robustness coverage. Full class 9/9 green.

The migration-point MULTI_DEVICE diag reports slot/dev/isView/rank/len per
cross-segment input migration. The sweep confirms EVERY boundary migration is
isView=0: the memory-proportional split lands boundaries on memory-heavy ops
(matmul/tanh), never on a low-memory permute (view), so a view AS a cross-segment
input does not arise under the current algorithm — the view-input dup-ordering
fix (6fb68ca) is sound defensive code. The reachable view cases remain
covered: view-as-output (6b3a44f, bit-identical) and a view aliasing a
migrated input (buildViewMlp dev1 permutes).

…tead of demote-crash

#52 (0765462) added invalidateExternalViewSlotsOnReacquire() to null dangling
view slots minted over a previous borrower's post-exec-closed external buffers. It
resets the affected segments to WARMUP but left the PLAN phase at REPLAYING — so
execute()'s pre-execute steady-state check (isReplaying() && a segment not fully
replaying -> demotePlanPhase THROW, ~2842) fires BEFORE the anySegmentNeedsWarmup()
re-warm dispatch and throws "segment no longer satisfies replay steady state ...
kind=demotion". Deterministic on a recurrent GDN-hybrid model driven across many
generates on one shared pipeline: each generate is a new borrower over fresh
external KV/state buffers, so these external-fed views are invalidated every gen.

Complete the transition: unseal() (REPLAYING -> SHAPES_FROZEN) when slots were
cleared and the plan is replaying, so the next execute() re-warms + re-captures the
freshly-nulled views gracefully instead of throwing. SHAPES_FROZEN (not SLOT_BY_SLOT
/ unfreeze) — shapes are unchanged; only the views re-mint over new borrower memory.

Regression gate green (105 tests, 0F/0E): DspSlotLifecycleAuditTest (98, incl. #52's
testSlotPhaseVisibility + the borrower-switch poison test), DspLifecycleExhaustive-
Test#testMarkVariableAfterReplaying, DspConcurrentPlanSharingTest#testConcurrent-
RecaptureAfterMarkVariable, TestNativeDecodeLoopRegression#testReplayStateAfter-
NativeDecode (replay held). On the fixed-buffer GDN decode repro: no demotion,
coherent output. (The re-warm-per-generate throughput cost is addressed separately
by a buffer-reuse change to the fixed-buffer decode path.)

…100% token match

INT8-V2 makes the INT8 KV cache the live decode storage (float KV freed after
prefill quantize). Verified: leading-window 10/10 + full-horizon 100/100 token
match vs STATIC; eager kernel isolation 0.3-0.76% relErr incl. model geometry
(GQA 8:2, headDim 256, masked partial cache); V1 unregressed 32/32; DSP core
regression gate 1642/0/0 across all 11 classes.

Core design:
- ROW-INLINE scale layout [B, maxKvLen, kvH, headDim+4]: each row carries hd
  int8 values + its float32 scale INSIDE the logical tensor, so DSP ext-input
  staging, forced placeholder H2D, CUDA-graph capture and device migration all
  preserve it by construction (an appended-tail layout was proven to be dropped
  by staging, which copies only the logical tensor bytes).
- Null-scale contract across kvCacheQuantize / kvInPlaceWriteQuantisedBSHD /
  fusedGQADecodeQuantised (CUDA + CPU) + Triton MLIR emitFusedAttentionKernel
  INT8 path (i8 loads, SIToFP, per-row scale multiply, hd+4 row pitch).
- KV placeholders converted to INT8 (+ trailing dim relaxed) at prefill and
  RESTORED on pipeline close — the decoder SameDiff is shared across pipelines
  and a leaked INT8 declaration broke later non-quantized pipelines.
- Triton SegmentCacheKey now hashes cross-segment INPUT dtypes so INT8 and
  float compile distinct kernels for identical shapes; the three key-recovery
  sites (refreshArgTablesForReplay / copyConsolidatedArgTable / getGapSlots)
  gain a loose-match fallback (findCompiledSegmentAnyDtype) — a dtypeIndex_
  miss previously skipped the arg-table refresh silently and replayed with
  stale pointers (caught by the poison-guard gate test).
- Match-rate gate asserts the leading 10-token window; long-horizon identity
  is informational (per-step INT8 KV writes can legitimately flip greedy
  repetition attractors; numerical closeness is pinned by the eager tests).

Also lands the accumulated, tested decode-campaign surface this work builds
on (oMLX port waves): runtime quantized-weight matmul, lm_head last-position
prefill, typical-p/XTC sampling, rotating KV + sinks, paged KV + radix prefix
cache, MoE weighted-sum epilogue, ADR-0106 Phase-1 masked decode substrate,
GDN chunked prefill, generation session continuation (ADR-0105), and the
supporting ADRs (0105-0108) and platform tests.

…pe precision fixes

Promote typical-p and XTC logit filtering to first-class ops (typical_p_filter,
xtc_filter) via the op-codegen path instead of raw DynamicCustomOp builder
strings in SamplerUtils:
- headers/nn.h declarations + generic/nn/sampling_penalties.cpp op bodies
  (copy logits -> apply helper); CPU + CUDA helpers already present.
- op-codegen: NeuralNetwork.kt Op() defs -> generated NDNN/SDNN; hand op
  classes TypicalPFilter / XtcFilter registered in DifferentialFunctionClass-
  Holder; SamplerUtils routes through Nd4j.nn().typicalPFilter / xtcFilter.

Precision / correctness:
- Mask filtered logits to -inf via DataTypeUtils::infOrMax<T> (CUDA + CPU
  sampling_penalties + token_sample) so filtered tokens are exactly excluded
  (was -FLT_MAX, a finite sentinel); drop the over-strict xtcThreshold < 0.5
  REQUIRE_TRUE that rejected valid no-op configs.
- Accumulate sampling reductions in AccType (double stays double) rather than a
  hardcoded float, so double inputs are no longer silently narrowed.
- distillation_kl_loss CUDA backprop now uses the canonical helpers::softmax at
  rank-1 (hand-rolled softmax gave wrong grad on reshaped rank-3).

Verified on CUDA (nd4j-cuda-12.9): TestTypicalPAndXtcSampling 19/19,
TestAccTypeDoublePrecisionOps 12/12, TestBugFixes20260702 distillation family
6/6; DSP core regression gate 1643/0/0 across all 11 classes.

Snapshot the current CUDA/Triton DSP lifecycle fixes together with the Vulkan, SDX AOT, native-image, and backend test work.

Validated the DSP core gate at 1643/0/0 twice and the MTP oracle at 60/60.

Wire the reusable source-SDZ compile/cache module into Android accelerator and Google Tensor G5 provider packaging.

Require the model cache/profile classes in mobile AAR verification and remove an unused cache import.

Replace the legacy shell bundle construction path with the backend-neutral Java compiler/cache launcher and align the mobile guide and deployment ADR.

Recursive MTP drafting feeds the predictor its own output hidden — out of
distribution for single-module heads trained on trunk hidden only (measured
on Qwen3.5-0.8B: 41% acceptance at draft position 0, 0/51 at position 1 even
when position 0's token was correct; chained drafts collapse to filler
tokens). Cap the chain at the first position whose evaluations never accept
(unconditional per-row draft-vs-argmax counting, >=12 evals), in both CUDA
and CPU decode helpers; n-gram proposals untouched. KV_CACHE-gated
MTP_CHAIN_PROBE samples carry-in/token/hidden-out per chain exec riding the
existing acceptance sync.

Bench 250-tok: acceptance 10.6% -> 44.2% (197 proposed / 87 accepted / 161
steps), 24.8 tok/s, token-identical to greedy; 60-tok oracle stays lossless
and engaged.

refreshStaleViewWrappersInSegment guarded the cached OUTPUT wrapper with
safeHasValidShapeInfo but deref'd resolveViewInput results raw. On a
plan-cache reuse the previous borrower's teardown can leave a destructed
NDArray in outputSlots_, and input0->dataBuffer() on it SIGSEGVs — hit
deterministically by run-benchmark-mtp.sh --tokens 250 in the MTP prefill
plan's freeze-time refresh (prepareBundledMtp). Probe each resolved input;
dead inputs null out into the existing refresh-input-invalid demote path.
Bench-250 0/2 -> 2/2 green; bench-60 and MTP oracle stay green.