Reference chains for surviving live-heap samples via a bounded manual heap walk (PROF-15341) - #644

What does this PR do?:
Implements reference-chain reconstruction for live-heap samples that survive
past their allocation window (PROF-15341). A ReferenceChainTracker runs a
dedicated BFS thread that tags reachable objects with JVMTI object tags,
walks the heap incrementally across GC epochs, and records the referrer-type
chain back toward a GC root (labelled with its GC-root kind). It is bridged
to LivenessTracker::selectLeakCandidates()'s population-slope leak
detection: when a still-live sampled object starts to look leak-shaped, its
chain is reconstructed from the frontier table and emitted to JFR.

flowchart TD
    LT["LivenessTracker::selectLeakCandidates (population-slope ranking)"] -->|"ranked klass candidates"| PWT["ReferenceChainTracker::pollWatchedTargets"]
    BFS["BFS thread: threadLoop"] -->|"shouldRunPass gate: GC epoch advanced or cadence elapsed"| RP["runPass"]
    RP -->|"every pass"| MW["runPassManualWalk: IterateOverReachableObjects seeds roots"]
    MW --> EF["expandFrontier: batched array-holder FollowReferences"]
    EF --> FT["FrontierTable: FRONTIER / EXPANDED / EDGE / ABANDONED"]
    FT -->|"tag already set on a leak-candidate instance"| PWT
    PWT -->|"buildChainEvent"| RC["cacheResolvedChain: one entry per klass id, cap 256"]
    RC -->|"Profiler::dump, snapshot without clearing"| DR["drainPendingChainEvents"]
    DR --> JFR["datadog.ReferenceChain / datadog.ReferenceChainAbandoned"]

Heap-walk mechanism (the main design decision in this branch): every pass
takes a bounded manual walk (runPassManualWalk()) — pure JVMTI
(IterateOverReachableObjects for root enumeration + expandFrontier() for
incremental frontier expansion), resumable across passes, and bounded so its
safepoint pauses stay small on every collector, including ZGC. The walk
issues only JVMTI heap calls, which run inside the VM_HeapWalkOperation
safepoint and honor ZGC's load barriers, so concurrent relocation cannot
corrupt it — it reads no raw oop.

expandFrontier()'s batching, concretely: instead of one FollowReferences
call per frontier entry, up to budget pending tags are resolved in one
GetObjectsWithTags call, packed into a single JNI array (holder), and
expanded via exactly one FollowReferences(initial_object = holder) call —
so one BFS level is discovered per VM-safepoint operation rather than one
safepoint per entry:

sequenceDiagram
    participant BFS as BFS thread
    participant JVMTI as JVMTI
    participant JNI as JNI holder array
    participant CB as heapReferenceCallback

    BFS->>BFS: pull up to budget tags from front of _pending_expand
    BFS->>JVMTI: GetObjectsWithTags, resolve which tags are still live
    JVMTI-->>BFS: live jobject references
    BFS->>JNI: EnsureLocalCapacity, then NewObjectArray to build holder
    alt exception, EnsureLocalCapacity failure, or null holder
        BFS->>BFS: ctx.truncated = true, retry this batch next pass
    else holder built successfully
        BFS->>JNI: SetObjectArrayElement per resolved object
        BFS->>JVMTI: FollowReferences, initial_object = holder
        JVMTI->>CB: heapReferenceCallback per outgoing edge
        CB-->>JVMTI: descend only for batch_tags boundary objects
        JVMTI-->>BFS: one BFS hop expanded for the whole batch
        BFS->>BFS: markExpanded, admitObject appends children to _pending_expand
    end

expandFrontier()'s JNI/JVMTI error handling is defensive by construction: a
null holder, a pending JNI exception after NewObjectArray/
SetObjectArrayElement, or an EnsureLocalCapacity failure all set
ctx.truncated = true (retry next pass) instead of marking the batch
permanently EXPANDED; a failed java/lang/Object class resolution with
pending work also forces truncated = true, so runPass() cannot mistake
it for SearchState::COMPLETED.

Termination and pacing — an unbounded traversal could otherwise stall a
GC safepoint or run forever:

stateDiagram-v2
    direction LR
    [*] --> RUNNING
    RUNNING --> COMPLETED: frontier drained,<br/>no truncation this pass
    RUNNING --> ABANDONED: frontier-size cap hit,<br/>or wall-clock TTL exceeded<br/>with work still pending
    COMPLETED --> RUNNING: restartSearch
    ABANDONED --> RUNNING: restartSearch

    note right of ABANDONED
        SearchAbandonReason (FRONTIER_CAP/TTL)
        records which cutoff fired, for the
        T_REFERENCE_CHAIN_ABANDONED JFR event
    end note
    note right of RUNNING
        restartSearch only fires once a leak
        candidate is seen and PainBudget allows it
    end note

• Per-pass budget, hop cap, wall-clock TTL, and frontier-size cap (the last
  treated as immediate search abandonment, not per-pass truncation) —
  configured via referencechains=true:hops=N:budget=N:ttl=N:framecap=N
  (plus firstpassbudget=N for the first pass's own budget). Negative
  hops/budget/framecap values are floored (an unfloored negative
  hops would otherwise wrap to ~4e9 as a u32, silently disabling the cap
  it's meant to enforce); all three are also ceiling-clamped.
• A PID pause-time pacing controller (updatePacing(), pausetarget=N ms)
  adapts the effective per-pass budget and cadence toward a configured
  safepoint-pause SLO.
• Search restarts are gated on an actual leak indication existing (so a
  restarted walk is very likely to still find the object) plus a
  PainBudget cooldown (painbudget=N, 0-100) — a leaky bucket over
  cumulative safepoint time spent, so a cheap search can restart sooner than
  an expensive one. This closes a structural gap where a one-shot walk could
  finish before population-trend detection had accumulated enough GC epochs
  to flag a candidate, leaving anything allocated afterward permanently
  undiscoverable.
• Leak-candidate selection (LivenessTracker::selectLeakCandidates()) requires
  a klass's population growth to clear both a magnitude bar and a floor-rise
  bar for LEAK_TREND_HYSTERESIS_BASE (5) consecutive qualifying epochs —
  lowered to LEAK_TREND_HYSTERESIS_CORROBORATED (3) when the aggregate
  post-GC heap floor is also rising — rather than trusting a single epoch's
  positive slope. This closes a false-positive gap where an oscillating
  ("see-saw") population could otherwise trigger a search with no real
  longer-term growth.

JFR persistence across dumps: a resolved chain is cached per klass id
(_resolved_chains, capped at 256 entries, drop-not-evict once full — see
REFERENCE_CHAIN_EVENTS_DROPPED) and re-stamped into every subsequent dump
the sample survives into (drainPendingChainEvents() snapshots without
clearing, mirroring how LivenessTracker re-emits live-object samples), so a
long-lived leak's chain is present in each JFR chunk rather than only the
chunk active when it was first reconstructed. Verified end-to-end in
ExternalProcessReferenceChainTest.

Motivation:
PROF-15341 — give live-heap samples that survive long enough to look
leak-shaped an actual referrer chain, not just "this object is still alive",
without regressing safepoint-pause behaviour on low-pause collectors.

Additional Notes:
See doc/reference-chains-design.md and doc/reference-chains-collection-summary.md
for the full design and leak-detection mechanism. Per-pass
budget/hop-cap/TTL/frontier-cap/pause-target/pain-budget defaults are round,
unbenchmarked placeholders pending a measurement pass — no JMH harness for
this feature exists yet, though utils/ has repro, parameter-sweep, and
JFR-report shell/Python tooling used to characterise pause behaviour ad hoc.
Earlier planning/proposal docs (implementation plan, remaining-work plan,
benchmark plan, and an alternative VMStructs-walk design) described approaches
or work that were superseded or shipped differently than planned, so they were
dropped from this PR (kept locally, not committed) rather than left to drift
further from the code.

How to test the change?:

• ./gradlew :ddprof-lib:gtestDebug — native unit suite, including
  ReferenceChainsTest/ReferenceChainsBfsTest/ReferenceChainsTagTest/
  FrontierTableTest/PollWatchedTargetsTest/ResolvedChainCacheTest,
  PainBudgetTest, SearchRestartTest (restart gate),
  ReferenceChainJfrRoundtripTest (JFR encode/decode), ArgumentsTest
  (referencechains= sub-option parsing, including hops/budget/framecap
  clamping behaviour), and the LivenessTracker/SelectLeakCandidates
  bridging tests.
• ./gradlew :ddprof-test:testDebug --tests "*ExternalProcessReferenceChainTest*" —
  genuinely separate-process end-to-end test: runs a real leaking-cache Java
  app in a child JVM, asserts the reconstructed chain's leaf class, and
  confirms the chain re-emits into three fresh JFR dumps (across-dumps
  persistence).
• ./gradlew :ddprof-test:testDebug --tests "*ReferenceChainTrackingTest*" —
  in-process coverage for the walk engine, target-selection bridging, and
  abandonment reporting.

For Datadog employees:

• If this PR touches code that signs or publishes builds or packages, or handles
  credentials of any kind, I've requested a security review (run the dd:platform-security-review
  skill, or file a request via the PSEC review form).
  bewaire also runs automatically on every PR.
• This PR doesn't touch any of that.
• JIRA: PROF-15341

Unsure? Have a question? Request a review!

🤖 Generated with Claude Code