Skip to content

make CTD GenerativeSampler visibility-aware - #3305

Merged
deruyter92 merged 2 commits into
mainfrom
jaap/update_ctd_generative_sampler
May 12, 2026
Merged

make CTD GenerativeSampler visibility-aware#3305
deruyter92 merged 2 commits into
mainfrom
jaap/update_ctd_generative_sampler

Conversation

@deruyter92

@deruyter92 deruyter92 commented May 5, 2026

Copy link
Copy Markdown
Collaborator

Summary
A regression analysis revealed that performance for ctd_coam drops since rc12, and more specifically since #2995. This PR applied some improvements to the PyTorch DLCLoader, and amongs other changes it introduces a method _remove_nans() that sanitizes the ground truth keypoints.

Cause:
The CTD GenerativeSampler at deeplabcut/pose_estimation_pytorch/data/generative_sampling.py relied on NaN propagation as an implicit safety net: invisible/unlabeled GT joints carried (NaN, NaN, vis=0) and NaN-based math made every synth path (good/jitter/miss/inv/swap) produce vis=0, and a final nan_mask zero’d visibility. Net effect: invisible joints stayed invisible in the conditional input. Since this implicit assumption is now not met anymore, we should explicitly make the GenerativeSampler visibility aware.

Changes:
This PR introduces a simple fix, where for invisible keypoints the generative sampling logic is skipped and a visibility of 0 is always propagated:

for j in range(self.num_keypoints):
      if keypoints[j, 2] <= 0:
          synth_joints[j] = 0  # (x, y, vis) = (0, 0, 0)
          continue
          
       ....

Note that this fix restores the observed regression.

image

@deruyter92
deruyter92 marked this pull request as ready for review May 11, 2026 14:26
@deruyter92
deruyter92 merged commit 7f5fb45 into main May 12, 2026
11 checks passed
@deruyter92
deruyter92 deleted the jaap/update_ctd_generative_sampler branch May 12, 2026 13:27
@deruyter92 deruyter92 mentioned this pull request May 21, 2026
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants