Skip to content

PyTorch maDLC analysis crashes post-inference with ValueError: need at least one array to stack #3108

@tsievert

Description

@tsievert

Is there an existing issue for this?

  • I have searched the existing issues

Operating System

AlmaLinux 9.6 (Sage Margay) (HPC)

DeepLabCut version

3.0rc10/rc12

What engine are you using?

pytorch

DeepLabCut mode

multi animal

Device type

Nvidia L40S 48GB

Bug description 🐛

After training up a PyTorch model, I ran the analysis step in order to extract outlier frames.
The analysis step run to 100% and then threw an error.

100%|██████████| 1296914/1296914 [15:18:47<00:00, 23.53it/s]

Traceback (most recent call last):
  File "/home/ts/scripts/DLC_analyze_and_extract.py", line 71, in <module>
    sys.exit(main())
  File "/home/ts/scripts/DLC_analyze_and_extract.py", line 37, in main
    deeplabcut.analyze_videos(
  File "/home/ts/miniforge3/envs/dlc/lib/python3.12/site-packages/deeplabcut/compat.py", line 953, in analyze_videos
    return analyze_videos(
  File "/home/ts/miniforge3/envs/dlc/lib/python3.12/site-packages/deeplabcut/pose_estimation_pytorch/apis/videos.py", line 689, in analyze_videos
    _generate_assemblies_file(
  File "/home/ts/miniforge3/envs/dlc/lib/python3.12/site-packages/deeplabcut/pose_estimation_pytorch/apis/videos.py", line 883, in _generate_assemblies_file
    unique_bpts = np.stack(keypoint_preds[num_bodyparts:])
  File "/home/ts/miniforge3/envs/dlc/lib/python3.12/site-packages/numpy/core/shape_base.py", line 445, in stack
    raise ValueError('need at least one array to stack')

ValueError: need at least one array to stack

I tried with both autotrack = true and false, tried stitching the tracklets after the analysis.
All resulted in the same error.
Even a short 2 minute clip caused the same error.

With some trial and error I figured out that the _full.pickle file is not written, but the _full.pickle.dat, _full.pickle.dir, _full.pickle.bak, and _meta.pickle exist.

Steps To Reproduce

  1. Train pytorch model
  2. run analysis
  3. see failure once 100% is reached
  4. No _full.pickle

Relevant log output

Anything else?

Tried it both with rc10 and rc12, both with the same error.
Also both dekr_w48 and dlcrnet_stride32_ms5 show the same behavior.
Pytorch is 2.8.0+cu128

I can share a reproducible example via email with a 2 min video.

Code of Conduct

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions