Skip to content

Restore priming and fix races on acquire/release ProfiledThread - #713

Open
zhengyu123 wants to merge 69 commits into
mainfrom
zgu/thread_priming
Open

Restore priming and fix races on acquire/release ProfiledThread#713
zhengyu123 wants to merge 69 commits into
mainfrom
zgu/thread_priming

Conversation

@zhengyu123

@zhengyu123 zhengyu123 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?:
Problem
ProfiledThread::current() returns nullptr for any thread the profiler hasn't explicitly registered yet. That's fine for normal application threads (they go through JVMTI's ThreadStart callback before doing anything interesting), but several JVM-internal threads — most notably JIT compiler threads (C1/C2 CompilerThread on HotSpot, JIT Compilation Thread on OpenJ9) — start very early during JVM bootstrap, often before the profiler agent has attached at all. Those threads never go through the normal registration path, so every profiling signal that lands on them is silently dropped: no ProfiledThread, no sample, no visibility into compiler-thread CPU cost.

Solution
Add TLS priming: the ability to attach a ProfiledThread to a thread on demand, from inside the signal handler itself, the first time that thread is sampled.

ThreadLocalDataPool(new threadLocalDataPool.h/.cpp): a fixed-capacity (64), pre-allocated pool of ProfiledThread slots. Slots are claimed/released via atomic CAS on a FLAG_CLAIMED bit — no locks, no allocation, safe to call from a signal handler.
ProfiledThread::acquireCurrent() (new threadLocalData.inline.h): get-or-prime. Returns the existing TLS value if set; otherwise claims a pool slot and attaches it via pthread_setspecific, right there in the signal handler.
ProfiledThread::supportPriming(): gates whether priming is safe at all. On glibc, pthread_setspecific can malloc internally unless the thread's pthread_key_t falls in the NPTL's pre-allocated first-level array (< PTHREAD_KEY_2NDLEVEL_SIZE) — priming is only enabled when that's guaranteed. Always enabled on musl. Fails safe (disabled) on any other libc (e.g. macOS), since the glibc-specific check doesn't apply there.
stackWalker.cpp / hotspotSupport.cpp (walkFP, walkDwarf, walkVM, walkJavaStack) switched from current() to acquireCurrent(), with an early return + SAMPLES_DROPPED_THREAD_LOCAL counter bump when priming isn't possible (pool exhausted, or unsupported on this libc) — replacing the old != nullptr ternary chains.

Supporting changes
threadLocalData.h's current()/acquireCurrent() moved to a new threadLocalData.inline.h (needed to avoid a circular include with ThreadLocalDataPool); ~15 .cpp files updated to include it.
New INJECT_FAULT_BOOL_HIGH fault-injection tier (10% firing rate) used to exercise the supportPriming() false-path in fault-injection builds.
UnwindFailures gained a reset() so a recycled pool slot can be reused without a fresh malloc.
New SAMPLES_DROPPED_TLS_POOL_EXHAUSTED counter for observability when the pool runs out of slots.

Motivation:

  • Improve profiling coverage (priming threads)
  • Reduce number of unknown frames in recording
  • Improve reliability (siglongjmp protection)

Additional Notes:
Rules to enforce (AGENT.md)

Thread Safety and Performance

  • Thread termination: A thread must call blockProfilingForExit() before releasing its ProfiledThread on exit — otherwise a profiling signal can race the release and allocate a new ProfiledThread that can never be freed (leak).

Sampler Safety

  • Sampled thread: The sampled thread must have a ProfiledThread in TLS, since it owns the sigjmp_buf used to recover via siglongjmp() if the stack walker crashes. If none is available, the sampler must drop the sample and report SAMPLES_DROPPED_THREAD_LOCAL.
  • Stack walker: HotspotSupport::walkVM(), StackWalker::walkDwarf(), and StackWalker::walkFP() must be protected by sigsetjmp()/siglongjmp().
  • Samplers: Every sampler must set up the ProfiledThread TLS before sampling, and skip the sample if it isn't available. Signal-based samplers use ProfiledThread::acquireCurrent(); non-signal-based samplers use ProfiledThread::initCurrentThreadSignalSafe().
  • JNI/JVMTI callbacks: Use ProfiledThread::initCurrentThreadSignalSafe() to set up ProfiledThread for the thread.

How to test the change?:

threadLocalDataPool_ut.cpp (new): boundary tests for contains() (first/last/one-past-end/one-before/null slot), plus a targeted regression test for the used >= _capacity vs. used > _capacity off-by-one in claim() — it checks the SAMPLES_DROPPED_TLS_POOL_EXHAUSTED counter rather than the return value, since both variants return nullptr at capacity but only the buggy one falls through to the exhaustion-counting scan.
TlsPrimingTest.java (new): end-to-end validation — forces sustained JIT compilation via a dynamically-generated class, then asserts datadog.ExecutionSample events include samples whose eventThread is a compiler thread. This is the test that actually proves priming works: if it silently broke, compiler threads would just never show up as eventThread and this test would fail.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-15601

Unsure? Have a question? Request a review!

Copilot AI review requested due to automatic review settings August 3, 2026 19:56
@dd-octo-sts

dd-octo-sts Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Scan-Build Report

User:runner@runnervmzvulz
Working Directory:/home/runner/work/java-profiler/java-profiler/ddprof-lib/src/test/make
Command Line:make -j4 all
Clang Version:Ubuntu clang version 18.1.3 (1ubuntu1)
Date:Fri Aug 14 21:57:47 2026

Bug Summary

Bug TypeQuantityDisplay?
All Bugs1
Logic error
Dereference of null pointer1

Reports

Bug Group Bug Type ▾ File Function/Method Line Path Length
Logic errorDereference of null pointerprofiler.hfindLibraryByAddress52328

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reintroduces “priming” for ProfiledThread access in async/signal-handling stack-walk paths by adding a preallocated ThreadLocalDataPool and switching key sampling sites to acquire and cache a ProfiledThread without per-signal allocation.

Changes:

  • Added ThreadLocalDataPool plus ProfiledThread::acquire_current() to acquire/carry a reusable ProfiledThread in signal context.
  • Updated stack walking and sampling code paths (StackWalker/HotSpot) to use acquire_current() and track drops when TLS cannot be acquired.
  • Rewired a number of translation units to include threadLocalData.inline.h (moving the inline TLS accessors out of threadLocalData.h).

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
ddprof-lib/src/main/cpp/wallClock.h Switch to threadLocalData.inline.h include for inlined TLS access.
ddprof-lib/src/main/cpp/threadLocalDataPool.h New pool API for reusing ProfiledThread instances.
ddprof-lib/src/main/cpp/threadLocalDataPool.cpp New pool implementation: allocate/claim/unclaim pooled ProfiledThread slots.
ddprof-lib/src/main/cpp/threadLocalData.inline.h New header providing inline definitions of ProfiledThread::current() and acquire_current().
ddprof-lib/src/main/cpp/threadLocalData.h Adds claimed flag/state and declares new inline TLS accessors.
ddprof-lib/src/main/cpp/threadLocalData.cpp Routes TLS destructor cleanup through the pool when applicable.
ddprof-lib/src/main/cpp/stackWalker.cpp Uses acquire_current() and increments drop counter when TLS cannot be acquired.
ddprof-lib/src/main/cpp/refCountGuard.cpp Switch include to threadLocalData.inline.h.
ddprof-lib/src/main/cpp/perfEvents_linux.cpp Switch include to threadLocalData.inline.h.
ddprof-lib/src/main/cpp/jvmThread.h Adds supportPriming() decision helper (musl vs glibc TLS key range).
ddprof-lib/src/main/cpp/jvmSupport.cpp Initializes ThreadLocalDataPool when priming is supported.
ddprof-lib/src/main/cpp/javaApi.cpp Switch include to threadLocalData.inline.h.
ddprof-lib/src/main/cpp/itimer.cpp Switch include to threadLocalData.inline.h.
ddprof-lib/src/main/cpp/hotspot/vmStructs.cpp Switch include to threadLocalData.inline.h.
ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp Uses acquire_current() in HotSpot stack-walk paths.
ddprof-lib/src/main/cpp/guards.cpp Switch include to threadLocalData.inline.h.
ddprof-lib/src/main/cpp/flightRecorder.cpp Switch include to threadLocalData.inline.h.
ddprof-lib/src/main/cpp/ctimer_linux.cpp Switch include to threadLocalData.inline.h.
ddprof-lib/src/main/cpp/context_api.cpp Switch include to threadLocalData.inline.h.
Suppressed comments (1)

ddprof-lib/src/main/cpp/threadLocalDataPool.cpp:64

  • initialize() publishes the pool pointer unconditionally; if construction failed (e.g., _threads == nullptr), subsequent acquire()/release() calls can hit UB. Only publish the pool if it is usable; otherwise keep _pool null.
void ThreadLocalDataPool::initialize() {
    ThreadLocalDataPool* pool = new ThreadLocalDataPool();
    __atomic_store_n(&_pool, pool, __ATOMIC_RELEASE);
}

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread ddprof-lib/src/main/cpp/threadLocalDataPool.h Outdated
Comment thread ddprof-lib/src/main/cpp/threadLocalDataPool.cpp Outdated
Comment thread ddprof-lib/src/main/cpp/threadLocalDataPool.cpp
Comment thread ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 3, 2026 20:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Suppressed comments (7)

ddprof-lib/src/main/cpp/threadLocalDataPool.cpp:18

  • ThreadLocalDataPool doesn’t initialize _threads when malloc fails, leaving it indeterminate. That can lead to invalid free() in the destructor and crashes in claim()/contains(). Initialize _threads to nullptr in the ctor initializer list.
ThreadLocalDataPool::ThreadLocalDataPool(uint64_t capacity) : _capacity(capacity), _used(0) {
    size_t malloc_size = capacity * sizeof(ProfiledThread);
    void* p = malloc(malloc_size);
    if (p != nullptr) {
      _threads = reinterpret_cast<ProfiledThread*>(p);

ddprof-lib/src/main/cpp/threadLocalDataPool.cpp:48

  • On probe failure, claim() currently assert(false) after scanning the whole pool. Under races (or if _threads is unexpectedly null), this can abort the process from a signal handler. Prefer to roll back _used and return nullptr (dropping the sample) instead of asserting.
    do {
        if (_threads[index].claim_acquire(tid)) {
            return &_threads[index];
        }
        index = (index + 1) % _capacity;
    } while (index != start_pos);
    assert(false && "Should not reach here");
    return nullptr;

ddprof-lib/src/main/cpp/threadLocalDataPool.cpp:56

  • unclaim() has the same unsigned-decrement issue as claim() (using __atomic_fetch_add(..., -1, ...) on a uint16_t). Use __atomic_fetch_sub(..., 1, ...) so _used doesn’t wrap.
        new (t)ProfiledThread(0);
        uint16_t used = __atomic_fetch_add(&_used, -1, __ATOMIC_RELEASE);
        assert(used > 0);
        return true;

ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp:1226

  • The previous code restored ThreadLocalData::_unwinding_Java after a recovered siglongjmp because siglongjmp bypasses AsyncSampleMutex destructors. That restore was removed, so a crash recovery can leave _unwinding_Java stuck true, preventing future Java stack walks on the thread.
  if (sigsetjmp(crash_protection_ctx, 1) != 0) {
    // checkFault() does a siglongjmp from inside segvHandler, bypassing
    // segvHandler's SignalHandlerScope destructor. Compensate.
    SIGNAL_HANDLER_UNWIND_AFTER_LONGJMP();
    prof_thread->setJmpCtx(prev_jmp_buf);

ddprof-lib/src/main/cpp/threadLocalDataPool.cpp:55

  • unclaim() reconstructs ProfiledThread via placement-new while other threads may concurrently probe/claim the same slot. Because ProfiledThread uses atomic operations on _misc_flags, reinitializing it with non-atomic stores (constructor/placement-new) can race with those atomics (UB) and also makes it possible to observe a partially-reset object. Consider adding an explicit “reset for pool reuse” routine that keeps the slot in a claimed state while resetting fields, then clears FLAG_CLAIMED with a release store as the final step.
bool ThreadLocalDataPool::unclaim(ProfiledThread* t) {
    if (contains(t)) {
        new (t)ProfiledThread(0);
        uint16_t used = __atomic_fetch_add(&_used, -1, __ATOMIC_RELEASE);
        assert(used > 0);

ddprof-lib/src/main/cpp/threadLocalData.inline.h:18

  • ProfiledThread::current() is defined here, but if it’s also defined inline in threadLocalData.h (to keep existing include sites compiling), this becomes a duplicate definition across TUs. Keep only one definition (e.g., define current() in threadLocalData.h and leave only acquire_current() here).
ProfiledThread* ProfiledThread::current() {
    if (!isThreadKeyValid()) {
      return nullptr;
    }
    return _current_thread.get();
}

ddprof-lib/src/main/cpp/threadLocalDataPool.cpp:37

  • __atomic_fetch_add(&_used, -1, ...) is performed on a uint16_t. The -1 is converted to uint16_t (65535), so this increments by 65535 (wraps) rather than decrementing. Use __atomic_fetch_sub(..., 1, ...) (and similarly in unclaim).

This issue also appears on line 53 of the same file.

    uint16_t used = __atomic_fetch_add(&_used, 1, __ATOMIC_RELAXED);
    if (used >= _capacity) {
        __atomic_fetch_add(&_used, -1, __ATOMIC_RELAXED);
        return nullptr;

Comment thread ddprof-lib/src/main/cpp/threadLocalData.h
Comment thread ddprof-lib/src/main/cpp/threadLocalDataPool.cpp Outdated
Comment thread ddprof-lib/src/main/cpp/threadLocalDataPool.cpp
@dd-octo-sts

dd-octo-sts Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #31844540234 | Commit: a6912ff | Duration: 14m 34s (longest job)

All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - - -
8-ibm - - -
8-j9 - -
8-librca - -
8-orcl - - -
11 - - -
11-j9 - -
11-librca - -
17 - -
17-graal - -
17-j9 - -
17-librca - -
21 - -
21-graal - -
21-librca - -
25 - -
25-graal - -
25-librca - -

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-08-14 22:13:38 UTC

@dd-octo-sts

dd-octo-sts Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 856ee13)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128699821 Commit: 856ee133a68fe770ffcfa01dd89dffccb4305ee2

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 21): runtime -4.5% (2778→2653 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10326 ms (21 iters) ✅ 10362 ms (21 iters) ≈ +0.3% (±11.4%) — / —
finagle-chirper 21 ✅ 5952 ms (33 iters) ✅ 5955 ms (33 iters) ≈ +0.1% (±25.5%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5484 ms (36 iters) ✅ 5471 ms (36 iters) ≈ -0.2% (±24.1%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2778 ms (67 iters) ✅ 2653 ms (71 iters) 🟢 -4.5% — / —
fj-kmeans 25 ✅ 2764 ms (68 iters) ✅ 2759 ms (68 iters) ≈ -0.2% (±2.8%) — / —
future-genetic 21 ✅ 2060 ms (90 iters) ✅ 2114 ms (87 iters) ≈ +2.6% (±2.7%) — / —
future-genetic 25 ✅ 2053 ms (90 iters) ✅ 2009 ms (93 iters) ≈ -2.1% (±2.5%) — / —
naive-bayes 21 ✅ 1268 ms (135 iters) ✅ 1298 ms (132 iters) ≈ +2.4% (±33.2%) — / —
reactors 21 ✅ 16232 ms (15 iters) ✅ 16628 ms (16 iters) ≈ +2.4% (±8.8%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 5 / 3 2054 / 1956 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 2 / 2 8800 / 8383 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 1 8585 / 8202 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1279 / 1277 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 1 / 4 2914 / 2956 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 2797 / 2885 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 7 / 2 3545 / 3557 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ 1 / ✅ 1581 / 1852 ✅ / ✅ ✅ / ✅

Copilot AI review requested due to automatic review settings August 3, 2026 20:57
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@datadog-datadog-us1-prod

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

Suppressed comments (3)

ddprof-lib/src/main/cpp/threadLocalDataPool.cpp:58

  • unclaim() reconstructs the slot with placement-new (new (t) ProfiledThread(0)), which writes _misc_flags (and other fields) non-atomically while other threads may concurrently read _misc_flags via __atomic_* in claim_acquire(). This mixes atomic and non-atomic accesses to the same object and can also momentarily clear FLAG_CLAIMED before the slot reset is fully complete.
    return nullptr;
}

bool ThreadLocalDataPool::unclaim(ProfiledThread* t) {
    if (contains(t)) {

ddprof-lib/src/main/cpp/threadLocalDataPool.cpp:15

  • If malloc() fails in ThreadLocalDataPool's constructor, _threads is left uninitialized, but later code (including the destructor and contains()) assumes it is either a valid pointer or nullptr. This can lead to undefined behavior.
ThreadLocalDataPool::ThreadLocalDataPool(uint64_t capacity)

ddprof-lib/src/main/cpp/threadLocalData.h:125

  • The assertion message in ProfiledThread::unclaim() is inverted: if the assert fires, the slot was not claimed, but the message says it "has been claimed".
    assert(isClaimed() && "Slot has been claimed");

Comment thread ddprof-lib/src/main/cpp/threadLocalDataPool.cpp
Comment thread ddprof-lib/src/main/cpp/jvmThread.h Outdated
Copilot AI review requested due to automatic review settings August 3, 2026 21:03
@datadog-datadog-us1-prod

datadog-datadog-us1-prod Bot commented Aug 3, 2026

Copy link
Copy Markdown

Bits has a CI fix ready

🟢 Investigated · 🟢 Fix prepared · ⚪ Validation skipped · 🟠 Ready

faultInjection_ut.cpp still referenced deleted INJECT_FAULT_INT_* and INJECT_FAULT_LONG_* macros after the fault-injection API refactor. Removed those obsolete assertions and added disabled-build identity coverage for INJECT_FAULT_BOOL_HIGH.

Commit fix to this PR


View in Datadog | Reviewed commit 40c69d4 · Any feedback? Reach out in #deveng-pr-agent

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.

Suppressed comments (4)

ddprof-lib/src/main/cpp/threadLocalDataPool.cpp:43

  • ThreadLocalDataPool::claim() currently has a broken/missing capacity guard: the code unconditionally decrements _used and returns nullptr, and the braces are unbalanced, so this won’t compile and the pool can never hand out slots. Add the intended used >= _capacity check and close the block correctly.
    uint16_t used = __atomic_fetch_add(&_used, 1, __ATOMIC_RELAXED);
        __atomic_fetch_add(&_used, -1, __ATOMIC_RELAXED);
        return nullptr;
    }

ddprof-lib/src/main/cpp/threadLocalDataPool.cpp:60

  • unclaim() reconstructs the slot with placement-new (ProfiledThread(0)), which clears FLAG_CLAIMED via a non-atomic write to _misc_flags. That allows another thread to observe the slot as unclaimed and race in while the object is mid-reset. Prefer clearing the claimed bit atomically (the class already provides ProfiledThread::unclaim() for this) and let the next acquire() reinitialize the object.
    if (contains(t)) {
        new (t)ProfiledThread(0);
        uint16_t used = __atomic_fetch_add(&_used, -1, __ATOMIC_RELEASE);

ddprof-lib/src/main/cpp/threadLocalData.inline.h:20

  • ProfiledThread::acquire_current() is defined in a header included by multiple translation units but is not marked inline, which can produce multiple-definition linker errors. Mark it inline.
ProfiledThread* ProfiledThread::acquire_current() {

ddprof-lib/src/main/cpp/threadLocalDataPool.cpp:12

  • This file uses placement-new (new (&_threads[index]) ...) but does not include <new>, which is required to declare placement-new in standard C++. Add the missing include to avoid build failures on stricter toolchains.
#include <cassert>
#include <stdlib.h>

Comment thread ddprof-lib/src/main/cpp/threadLocalData.inline.h Outdated
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 3, 2026 21:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

Suppressed comments (8)

ddprof-lib/src/main/cpp/threadLocalDataPool.cpp:62

  • _used was widened in the header, but this local variable is still uint16_t, which will truncate the atomic counter and can underflow/wrap incorrectly.
        uint16_t used = __atomic_fetch_add(&_used, -1, __ATOMIC_RELEASE);

ddprof-lib/src/main/cpp/threadLocalDataPool.cpp:63

  • unclaim() reconstructs the ProfiledThread in-place with placement-new while other threads may concurrently read/modify the slot (via claim_acquire() / _misc_flags). Re-ending/restarting an object’s lifetime and doing non-atomic writes to the same storage other threads touch is undefined behavior and can lead to double-claim or corrupted state. Consider keeping slot ownership state separate from the ProfiledThread object (e.g., a dedicated std::atomic<uint32_t> claim word per slot) and avoid placement-new on shared objects; reset per-thread state only after exclusive ownership is established.
bool ThreadLocalDataPool::unclaim(ProfiledThread* t) {
    if (contains(t)) {
        new (t)ProfiledThread(0);
        uint16_t used = __atomic_fetch_add(&_used, -1, __ATOMIC_RELEASE);
        assert(used > 0);

ddprof-lib/src/main/cpp/threadLocalDataPool.cpp:83

  • acquire() reconstructs a claimed ProfiledThread in-place. Even though the slot is "claimed", other threads may still probe its claim state concurrently (via _misc_flags), and reconstructing the object restarts its lifetime while concurrent reads are possible. This is undefined behavior in C++ and can manifest as intermittent races. Prefer a separate per-slot claim flag (outside the object being reconstructed), or avoid placement-new and instead reset fields under exclusive ownership without touching memory concurrently accessed by other threads.
        ProfiledThread* t = pool->claim(tid);
        if (t != nullptr) {
            new (t)ProfiledThread(tid, true /* claimed */);
        }
        return t;

ddprof-lib/src/main/cpp/threadLocalData.h:183

  • ProfiledThread::current() is declared inline here but no longer defined in this header. Several existing translation units still include threadLocalData.h (not threadLocalData.inline.h) and call ProfiledThread::current(), which will fail to compile. Either keep current() defined here (as before) or make threadLocalData.h include the inline definitions.
  // Signal-handler friendly (no allocation): returns existing TLS or nullptr.
  static inline ProfiledThread *current();
  // signal-handler friendly with priming: return existing TLS or acquire and set
  // ProfiledThread from ThreadLocalDataPool.
  static inline ProfiledThread* acquire_current();

ddprof-lib/src/main/cpp/threadLocalData.inline.h:13

  • With ProfiledThread::current() defined back in threadLocalData.h, this out-of-class definition becomes a duplicate definition when threadLocalData.inline.h is included (directly or indirectly). Remove it to avoid redefinition errors.
inline ProfiledThread* ProfiledThread::current() {

ddprof-lib/src/main/cpp/threadLocalDataPool.h:20

  • _used is a 16-bit counter but _capacity is 64-bit; if the pool capacity is ever increased beyond 65535, _used will wrap and the full/empty checks become incorrect. Use a wider counter type that can represent _capacity.
    const uint64_t      _capacity;
    volatile uint16_t   _used;
    ProfiledThread*     _threads;

ddprof-lib/src/main/cpp/threadLocalDataPool.cpp:41

  • _used was widened in the header, but this local variable is still uint16_t, which will truncate the atomic counter and break capacity checks once _used exceeds 65535.

This issue also appears on line 62 of the same file.

    uint16_t used = __atomic_fetch_add(&_used, 1, __ATOMIC_RELAXED);

ddprof-lib/src/main/cpp/threadLocalDataPool.cpp:72

  • This PR introduces a new concurrent, signal-path critical allocation strategy (ThreadLocalDataPool + ProfiledThread::acquire_current()), but there are no accompanying C++ unit tests validating pool exhaustion behavior, claim/release correctness, or the interaction with TLS teardown (ProfiledThread::freeValue). The repo has an existing gtest suite under ddprof-lib/src/test/cpp/; please add targeted tests to lock in correctness.
void ThreadLocalDataPool::initialize() {
    ThreadLocalDataPool* pool = new ThreadLocalDataPool();
    __atomic_store_n(&_pool, pool, __ATOMIC_RELEASE);
}

Comment thread ddprof-lib/src/main/cpp/threadLocalDataPool.cpp Outdated
Comment thread ddprof-lib/src/main/cpp/jvmSupport.cpp Outdated
@jbachorik

Copy link
Copy Markdown
Collaborator

🗿 🤖 🔴

Heads up @zhengyu123 — commit a21096c ("sphinx: address review feedback on PR #713") on this branch was pushed automatically by an assistant-run /sphinx address pass, without your review beforehand. Flagging it explicitly since you didn't get a chance to weigh in first.

It changes guards.cpp/guards.h: CriticalSection's constructor/destructor now use ProfiledThread::acquireCurrent() instead of current() + assert(_thread_ptr != nullptr), treating a null result (priming unsupported, e.g. macOS, or pool exhausted) as "did not enter" rather than dereferencing. This addresses the HIGH-severity finding on guards.cpp:87 from the review.

Please review this commit on its own merits — revert it if you disagree with the approach, no automated process should make that call for you.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d576acf488

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ddprof-lib/src/main/cpp/perfEvents_linux.cpp Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

https://github.com/DataDog/java-profiler/blob/329fed0ee9962932ae9529f3c93114f169ce7f0c//tmp/review-23e3c0e/ddprof-lib/src/main/cpp/guards.h#L115
P1 Badge Restore the removed signal guard macro

When the C++ gtest suite is built, ddprof-lib/src/test/cpp/signalOrigin_ut.cpp still invokes SIGNAL_HANDLER_GUARD() in WallclockGuardContract_ForeignSignalReleasesGuard, but this header now only defines the new SIGNAL_HANDLER_GUARD_OR_DROP / SIGNAL_HANDLER_GUARD_NO_SAMPLE forms. That leaves the test target failing to compile; either update that test to the new macro or keep a compatibility alias for the old guard.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9f03a2f3fb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ddprof-lib/src/main/cpp/mallocTracer.cpp Outdated
@zhengyu123

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a5ea581ca5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ddprof-lib/src/main/cpp/threadLocalData.cpp
Comment thread ddprof-test/src/test/java/com/datadoghq/profiler/cpu/TlsPrimingTest.java Outdated
@zhengyu123
zhengyu123 requested a review from jbachorik August 13, 2026 20:35

@zhengyu123 zhengyu123 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

@jbachorik jbachorik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗿 🤖 🔴

Sphinx Review found 2 critical/high severity finding(s) that must be addressed.

Comment thread AGENTS.md Outdated
Comment thread AGENTS.md Outdated
Comment thread ddprof-lib/src/main/cpp/ctimer_linux.cpp Outdated
Comment thread ddprof-lib/src/main/cpp/guards.cpp Outdated
Comment thread ddprof-lib/src/main/cpp/guards.h Outdated
Comment thread ddprof-lib/src/test/cpp/signalOrigin_ut.cpp Outdated
Comment thread ddprof-lib/src/test/cpp/stress_callTraceStorage.cpp Outdated
Comment thread ddprof-lib/src/test/cpp/thread_teardown_safety_ut.cpp
Comment thread ddprof-test/src/test/java/com/datadoghq/profiler/cpu/TlsPrimingTest.java Outdated
Comment thread ddprof-test/src/test/java/com/datadoghq/profiler/cpu/TlsPrimingTest.java Outdated

@jbachorik jbachorik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗿 🤖 🔴

Sphinx Review found 2 critical/high severity finding(s) that must be addressed.

Comment thread AGENTS.md Outdated
Comment thread AGENTS.md Outdated
Comment thread ddprof-lib/src/main/cpp/ctimer_linux.cpp Outdated
Comment thread ddprof-lib/src/main/cpp/guards.cpp Outdated
Comment thread ddprof-lib/src/main/cpp/guards.h Outdated
Comment thread ddprof-lib/src/test/cpp/signalOrigin_ut.cpp Outdated
Comment thread ddprof-lib/src/test/cpp/stress_callTraceStorage.cpp Outdated
Comment thread ddprof-lib/src/test/cpp/thread_teardown_safety_ut.cpp
Comment thread ddprof-test/src/test/java/com/datadoghq/profiler/cpu/TlsPrimingTest.java Outdated
Comment thread ddprof-test/src/test/java/com/datadoghq/profiler/cpu/TlsPrimingTest.java Outdated
@jbachorik

Copy link
Copy Markdown
Collaborator

🗿 🤖 🔴

[Sphinx Review — MEDIUM] ThreadLocalDataPool::claim() increments SAMPLES_DROPPED_TLS_POOL_EXHAUSTED on pool exhaustion, and every caller of ProfiledThread::acquireCurrent() (which wraps claim()) also increments SAMPLES_DROPPED_THREAD_LOCAL when acquireCurrent() returns nullptr for the same event — every pool-exhaustion drop is now counted under two separate counters, undermining the new counter's value as a distinguishing diagnostic.

Suggestion: Pick one owner for the drop accounting: either have ThreadLocalDataPool::claim()/acquireCurrent() be the sole incrementer of a single counter (retiring the duplicate caller-side SAMPLES_DROPPED_THREAD_LOCAL increments when the cause is specifically pool exhaustion), or have callers suppress their own SAMPLES_DROPPED_THREAD_LOCAL increment when acquireCurrent() already signalled TLS_POOL_EXHAUSTED (e.g. by returning an enum/reason code instead of a bare nullptr).

zhengyu123 and others added 4 commits August 14, 2026 20:13
Co-authored-by: Jaroslav Bachorik <jaroslav.bachorik@datadoghq.com>
Co-authored-by: Jaroslav Bachorik <jaroslav.bachorik@datadoghq.com>
Co-authored-by: Jaroslav Bachorik <jaroslav.bachorik@datadoghq.com>
Co-authored-by: Jaroslav Bachorik <jaroslav.bachorik@datadoghq.com>
@zhengyu123

Copy link
Copy Markdown
Contributor Author

@jbachorik It seems that Sphnix inserted the same review to the PR twice -- very confusing.

@zhengyu123

Copy link
Copy Markdown
Contributor Author

🗿 🤖 🔴

[Sphinx Review — MEDIUM] ThreadLocalDataPool::claim() increments SAMPLES_DROPPED_TLS_POOL_EXHAUSTED on pool exhaustion, and every caller of ProfiledThread::acquireCurrent() (which wraps claim()) also increments SAMPLES_DROPPED_THREAD_LOCAL when acquireCurrent() returns nullptr for the same event — every pool-exhaustion drop is now counted under two separate counters, undermining the new counter's value as a distinguishing diagnostic.

Suggestion: Pick one owner for the drop accounting: either have ThreadLocalDataPool::claim()/acquireCurrent() be the sole incrementer of a single counter (retiring the duplicate caller-side SAMPLES_DROPPED_THREAD_LOCAL increments when the cause is specifically pool exhaustion), or have callers suppress their own SAMPLES_DROPPED_THREAD_LOCAL increment when acquireCurrent() already signalled TLS_POOL_EXHAUSTED (e.g. by returning an enum/reason code instead of a bare nullptr).

@jbachorik Added comment. But there is no simple way to distinguish the counters.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5172370084

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ddprof-lib/src/main/cpp/threadLocalData.cpp
@zhengyu123
zhengyu123 requested a review from jbachorik August 14, 2026 22:36
@zhengyu123

Copy link
Copy Markdown
Contributor Author

@jbachorik I think I addressed all review comments, let me know if I missed any.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sphinx:critical Sphinx: critical — human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants