Summary

PR 09 of 22 PRs in the ag_new_release_updates_2 branch split. Merge after Layer 2 (native core ops + helpers).

• DSP execution engine: 35 new .cpp/.cu files + 40 new headers implementing the full native plan executor
• Dispatch priority: Triton MLIR → NVRTC JIT → PTX templates → CUDA Graphs → slot-by-slot
• GraphExecutionMode: 19-variant C++ enum (GEM_AUTO through GEM_SHAPE_INFERENCE_ONLY); settable from Java via setGraphExecutionMode()
• CUDA graph capture: Per-device atomics (g_captureActive[16]) serialize concurrent captures; CaptureStateGuard RAII prevents illegal cudaStreamQuery calls during capture (avoids CUDA error 900)
• argTableStable fast-path: Skips external input refresh and H2D sync when all pointer addresses match captured values — dominant hot path during autoregressive decode
• Thread-isolated plan cache: NativePlanCache keys on (outputSetHash, phShapeContentHash, phCount, graphExecutionMode, threadId); each thread gets its own plan instance, mutable slot state never shared
• Dual eviction policy: Hard count cap + memory soft cap (planCacheBudgetFraction * freeDeviceMem); pin/unpin prevents eviction of active plans
• DSP diagnostics: 20-category bitfield, 3 detail levels (SUMMARY/DETAILED/FULL), 65,536-entry DspDiagEvent ring buffer; FULL level echoes every event to stderr in real time
• Multi-backend replay: GraphReplayFactory selects from 8 GraphReplayHandle subclasses: CUDA, HIP/ROCm, Metal, Vulkan, Level Zero, TPU, Hexagon/QNN, CPU functional
• CPU graph backends: OneDNN, ACL, MLX, MLIR, ARM Hybrid, NNAPI, OpenVINO, FunctionalReplay
• Control flow removed: All 10 LogicXxx files (LogicWhile, LogicSwitch, etc.) deleted; TF-graph-style control flow replaced by plan-based compilation (resolved at Java level before serialization)
• Dual DSP streams: tl_dspExecutionStream routes H2D through DSP stream; tl_dspGapStream (via ScopedDspGapStream RAII) routes cuBLAS gap ops onto the same stream as island replay

What Changed

Core DSP execution engine — graph/impl/ (35 new files, 10 modified)

• NativeDynamicShapePlan.cpp — main executor: plan deserialization, NativeSlot construction, executeSlot() dispatcher routing to slot-by-slot/CUDA graph/NVRTC/Triton; DSP_BUF(arr) macro for portable CUDA/CPU buffer access
• NativeDynamicShapePlan_segments.cpp — segment shape key hashing, CPU graph backend dispatch, slot-by-slot fallback
• NativeDynamicShapePlan_cudagraph.cu — CUDA graph state machine: warmup→capture→instantiate→replay; per-GPU atomics for concurrent capture serialization; ScopedDspGapStream RAII
• NativeDynamicShapePlan_cuda.cu — device-side slot execution; tl_dspExecutionStream and tl_dspGapStream thread-local stream management; cuBLAS workspace gate for graph capture
• NativeDynamicShapePlan_slotexec.cpp / _slotexec_cuda.cu — op-by-op dispatch using LegacyTransformSameOp, LegacyScalarOp, LegacyReduceFloatOp, etc.
• NativeDynamicShapePlan_cublas.cu — batched GEMM via cublasSgemmStridedBatched; cuBLASLt matmul with workspace; stream-capture-safe allocation
• NativeDynamicShapePlan_batchgemm.cu — groups consecutive matmul slots into single cublasGemmStridedBatchedEx call
• NativeDynamicShapePlan_batchzero.cu — zeros output buffers via cudaMemsetAsync before segment execution
• NativeDynamicShapePlan_gpubackend.cpp/.cu — GPU backend selection: Triton→NVRTC→PTX→CUDA Graphs; handles segment splitting for register pressure limits
• NativeDynamicShapePlan_prereplay.cu — pointer stability validation, placeholder H2D sync, argTableStable fast-path detection
• NativeDynamicShapePlan_cuda_stubs.cpp — CPU-side stubs so CPU build links cleanly
• NativePlanCache.cpp — LRU with content-based key; dual eviction: hard count cap + memory soft cap; pin/unpin support
• NativePlanCompiler.cpp — deserializes FlatBuffers from Java, constructs NativeSlot array, resolves variable references
• PlanDefinition.cpp, PlanTopology.cpp — plan data model and topological ordering
• SegmentExecutor.cpp — segment lifecycle: build→seal→replace for FrozenPlan segments
• SlotArray.cpp, SlotBufferOwnership.cpp — slot array management and buffer ownership for CUDA graph address stability
• FrozenPlan.cpp — pointer table captured at freeze; validated on each replay attempt
• FusionPass.cpp — identity elimination, cast chain collapsing, element-wise chain grouping
• GraphAnalysisUtils.cpp — convex subgraph detection, segment boundary inference
• GraphReplayFactory.cpp — selects correct GraphReplayHandle subclass based on detected hardware
• ReplayCacheManager.cpp — per-segment compiled graph handles with LRU eviction
• ResourceBinder.cpp — binds placeholders/variables/constants to native slot pointers pre-execution
• DspBufferColorMap.cpp, DspDiagnostics.cpp — buffer reuse analysis and diagnostics framework
• DspBufferPool.cpp — reusable intermediate buffer pool
• ExecutionState.cpp/.cu, DeviceExecutionContext.cpp/.cu — per-execution mutable state, per-device context
• SdnbReader.cpp, SdzReader.cpp — plan bundle deserializers (single bytes and zip format)
• Node.cpp, Graph.cpp, Context.cpp, FlatUtils.cpp and others — updated for DSP compatibility

Core DSP headers — graph/ (40 new, 6 modified, 12 deleted)

• NativeDynamicShapePlan.h — GraphExecutionMode enum (19 modes); JitMode enum; NativeSlot struct; plan lifecycle methods
• NativePlanCache.h — LRU cache with content-based key; thread-per-plan isolation
• DspDiagnostics.h — 20-category bitfield (all → DSP_DIAG_ALL=0xFFFFF); SUMMARY/DETAILED/FULL levels; DspDiagEvent ring buffer (65,536 entries)
• GraphBackend.h — abstract backend interface: isAvailable(), canFuseSegment(), compileSegment(), executeSegment(), invalidateCache(), getLastCompilationAudit()
• GraphReplayHandle.h — abstract replay handle: ReplayState enum, ReplayStatistics, workspace management API
• DspGraphTypes.h — GraphSegment struct, NativeSlot struct with op classification and buffer pointers
• DspPhaseUtils.h — phase transition guards; SLOT_BY_SLOT→SHAPES_FROZEN→REPLAYING lifecycle checks
• CaptureStateGuard.h — RAII guard setting streamCaptureActive flag to block illegal API calls during CUDA graph capture
• DspDiagnostics.h, DspBufferColorMap.h, DspBufferPool.h, DspExecutionTrace.h, DspStreamGuard.h, DspThreadState.h, FrozenPlan.h, FusionPass.h, PlanDefinition.h, PlanTopology.h, ReplayCacheManager.h, ResourceBinder.h, SegmentExecutor.h, SlotArray.h, DeviceExecutionContext.h, SdnbReader.h, SdzReader.h — supporting subsystem headers
• Deleted: FlowPath.h, GraphExecutioner.h, GraphHolder.h, all 10 LogicXxx.h/.cpp files (LogicConditional, LogicWhile, LogicSwitch, etc.)

Multi-backend GraphReplayHandle — graph/ (12 new files)

• cuda/CudaGraphReplayHandle.h/.cu — wraps CudaGraphHandle; beginCapture/endCapture/finalize/replay via CUDA graph API
• hip/HipGraphReplayHandle.h/.cpp — HIP (ROCm) graph capture/replay mirror
• metal/MetalReplayHandle.h/.mm — Metal command buffer capture/replay
• vulkan/VulkanReplayHandle.h/.cpp — Vulkan command buffer recording/replay
• levelzero/LevelZeroReplayHandle.h/.cpp — Intel Level Zero graph backend
• tpu/TpuReplayHandle.h/.cpp, TpuGraphBackend.h/.cpp, HloIRBuilder.h/.cpp, PjrtClientManager.h/.cpp — TPU via PJRT; HloIRBuilder generates XLA HLO from NativeSlots
• hexagon/HexagonGraphBackend.h/.cpp, HexagonIRBuilder.h/.cpp, HexagonReplayHandle.h/.cpp, HexagonRuntimeManager.h/.cpp — Qualcomm Hexagon/QNN backend

CPU graph backends — graph/cpu/ (14 new files)

• OneDnnGraphBackend.h/.cpp — oneDNN Graph API: builds and executes compiled_partition from segment slots
• AclGraphBackend.h/.cpp — ARM Compute Library Dynamic Fusion backend
• MlxGraphBackend.h/.cpp, MlxIRBuilder.h/.cpp — Apple MLX compute graph backend
• MlirCpuGraphBackend.h/.cpp — MLIR Linalg/arith JIT for CPU
• ArmHybridGraphBackend.h/.cpp — MLIR CPU + Vulkan for Android/AArch64
• NnapiGraphBackend.h/.cpp — Android NNAPI backend
• OpenVinoGraphBackend.h/.cpp — Intel OpenVINO backend
• FunctionalReplayHandle.h/.cpp — CPU fallback: re-runs compiled segment function on replay
• CpuIRBuilder.h/.cpp — platform-neutral IR generator for CPU backends

Profiling

• profiling/OpTimingTracker.h/.cpp — per-op timing with CSV export; integrates with TIMING diagnostic category
• profiling/impl/GraphProfile.cpp — updated to use OpTimingTracker
• Deleted: profiling/impl/GraphProfilingHelper.cpp — replaced by OpTimingTracker

Dependencies

• Depends on: PR05, PR06, PR07 (libnd4j core); PR10 (Triton/NVRTC/PTX backends referenced by NativeDynamicShapePlan_gpubackend)
• Required by: PR12, PR13 (Java DSP executor layer requires this native plan infrastructure)

Merge Order

These 22 PRs must merge in layer order. Each layer depends on the layers above it being merged first. PRs within the same layer are independent and can merge in parallel.

This PR: Merge after Layer 2 (native core ops + helpers).