Is there an existing issue for this?
Operating System
Manjaro Linux
DeepLabCut version
3.0.1
What engine are you using?
pytorch
DeepLabCut mode
multi animal
Device type
NVIDIA RTX 4000 Ada Generation
Bug description 🐛
COCOLoader.get_dataset_parameters derives individuals from train.json and ignores model_cfg["metadata"]["individuals"]. DLCLoader does the opposite and treats the config as authoritative. As a result, max_individuals has no effect on COCO projects, and adaptation_train crashes partway through training if any evaluation image holds more individuals than the busiest training image.
Steps To Reproduce
- Build a COCO project where the busiest train image has 3 instances and the busiest test image has 4.
- PoseConfig.build_for_superanimal_inference(..., max_individuals=4) and write it to pytorch_config.yaml.
- Call adaptation_train(project_root=..., train_file="train.json", test_file="test.json", model_config_path=, eval_interval=5, ...).
Relevant log output
Traceback (most recent call last):
File "train_dlc.py", line 149, in <module>
train_dlc(project_root)
File "train_dlc.py", line 109, in train_dlc
adaptation_train(
File ".venv/lib/python3.12/site-packages/deeplabcut/pose_estimation_pytorch/modelzoo/train_from_coco.py", line 79, in adaptation_train
train(
File ".venv/lib/python3.12/site-packages/deeplabcut/pose_estimation_pytorch/apis/training.py", line 194, in train
runner.fit(
File ".venv/lib/python3.12/site-packages/deeplabcut/pose_estimation_pytorch/runners/train.py", line 230, in fit
valid_loss = self._epoch(valid_loader, mode="eval", display_iters=display_iters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".venv/lib/python3.12/site-packages/deeplabcut/pose_estimation_pytorch/runners/train.py", line 275, in _epoch
for i, batch in enumerate(loader):
^^^^^^^^^^^^^^^^^
File ".venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 725, in __next__
data = self._next_data()
^^^^^^^^^^^^^^^^^
File ".venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 785, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".venv/lib/python3.12/site-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
~~~~~~~~~~~~^^^^^
File ".venv/lib/python3.12/site-packages/deeplabcut/pose_estimation_pytorch/data/dataset.py", line 276, in __getitem__
return self._prepare_final_data_dict(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".venv/lib/python3.12/site-packages/deeplabcut/pose_estimation_pytorch/data/dataset.py", line 313, in _prepare_final_data_dict
"annotations": self._prepare_final_annotation_dict(keypoints, keypoints_unique, bboxes, annotations_merged),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".venv/lib/python3.12/site-packages/deeplabcut/pose_estimation_pytorch/data/dataset.py", line 347, in _prepare_final_annotation_dict
"keypoints": pad_to_length(keypoints[..., :3], num_animals, 0).astype(np.single),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".venv/lib/python3.12/site-packages/deeplabcut/pose_estimation_pytorch/data/utils.py", line 528, in pad_to_length
raise ValueError(f"Cannot pad! data.shape={data.shape} > length={length}")
ValueError: Cannot pad! data.shape=(4, 27, 3) > length=3Anything else?
I would suggest to validate individuals before any training run to avoid hanging out at validation after X epochs.
Code of Conduct
Is there an existing issue for this?
Operating System
Manjaro Linux
DeepLabCut version
3.0.1
What engine are you using?
pytorch
DeepLabCut mode
multi animal
Device type
NVIDIA RTX 4000 Ada Generation
Bug description 🐛
COCOLoader.get_dataset_parameters derives individuals from train.json and ignores model_cfg["metadata"]["individuals"]. DLCLoader does the opposite and treats the config as authoritative. As a result, max_individuals has no effect on COCO projects, and adaptation_train crashes partway through training if any evaluation image holds more individuals than the busiest training image.
Steps To Reproduce
Relevant log output
Anything else?
I would suggest to validate individuals before any training run to avoid hanging out at validation after X epochs.
Code of Conduct