-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Description
Is there an existing issue for this?
- I have searched the existing issues
Operating System
AlmaLinux 9.6 (Sage Margay) (HPC)
DeepLabCut version
3.0rc10/rc12
What engine are you using?
pytorch
DeepLabCut mode
multi animal
Device type
Nvidia L40S 48GB
Bug description 🐛
After training up a PyTorch model, I ran the analysis step in order to extract outlier frames.
The analysis step run to 100% and then threw an error.
100%|██████████| 1296914/1296914 [15:18:47<00:00, 23.53it/s]
Traceback (most recent call last):
File "/home/ts/scripts/DLC_analyze_and_extract.py", line 71, in <module>
sys.exit(main())
File "/home/ts/scripts/DLC_analyze_and_extract.py", line 37, in main
deeplabcut.analyze_videos(
File "/home/ts/miniforge3/envs/dlc/lib/python3.12/site-packages/deeplabcut/compat.py", line 953, in analyze_videos
return analyze_videos(
File "/home/ts/miniforge3/envs/dlc/lib/python3.12/site-packages/deeplabcut/pose_estimation_pytorch/apis/videos.py", line 689, in analyze_videos
_generate_assemblies_file(
File "/home/ts/miniforge3/envs/dlc/lib/python3.12/site-packages/deeplabcut/pose_estimation_pytorch/apis/videos.py", line 883, in _generate_assemblies_file
unique_bpts = np.stack(keypoint_preds[num_bodyparts:])
File "/home/ts/miniforge3/envs/dlc/lib/python3.12/site-packages/numpy/core/shape_base.py", line 445, in stack
raise ValueError('need at least one array to stack')
ValueError: need at least one array to stackI tried with both autotrack = true and false, tried stitching the tracklets after the analysis.
All resulted in the same error.
Even a short 2 minute clip caused the same error.
With some trial and error I figured out that the _full.pickle file is not written, but the _full.pickle.dat, _full.pickle.dir, _full.pickle.bak, and _meta.pickle exist.
Steps To Reproduce
- Train pytorch model
- run analysis
- see failure once 100% is reached
- No
_full.pickle
Relevant log output
Anything else?
Tried it both with rc10 and rc12, both with the same error.
Also both dekr_w48 and dlcrnet_stride32_ms5 show the same behavior.
Pytorch is 2.8.0+cu128
I can share a reproducible example via email with a 2 min video.
Code of Conduct
- I agree to follow this project's Code of Conduct