Migrate to `pathlib.Path` instead of `str` (first phase) by deruyter92 · Pull Request #3350 · DeepLabCut/DeepLabCut

deruyter92 · 2026-05-29T08:51:14Z

Motivation

We currently use pathlib.Path and str mixed inconsistently throughout the codebase. Settling on pathlib.Path could make the codebase more uniform, and fix issues with brittle path resolvement, (e.g. see #3348).

This PR builds on top of #3349, which introduces safer path resolution.

Policy

Internal: use pathlib.Path for path math and state.
Public API: accept str | Path at entry; normalize with Path(...).absolute() internally.
Boundaries: convert with str() / os.fspath() only where required:
- Qt widgets (QLineEdit, QFileDialog, list item text)
- video_sets dict keys in config.yaml
- OpenCV / PIL / VideoReader / subprocess / pandas paths
- (legacy helpers that still expect str -> should be replaced on long-term as well)

Scope

This PR starts with addressing many places to clearly enforce Path-centric API, but leaves quite some legacy helpers untouched for now, to avoid a single big switch. See details (autogenerated) below:

Done in this PR

`create_project` (2D core)

new.py, add.py, demo_data.py: Path internally; os.fspath at video_sets / VideoReader / copy-mklink boundaries
create_new_project: returns Path | Literal["nothingcreated"] (was str)
Removed legacy realpath / readlink fallback for video_sets keys; use Path(video).absolute()

GUI

window.py: config_path: Path | None, files: set[Path]; derived folders (project_folder, models_folder, …) return Path; os.fspath for Qt line edits / recent files
widgets.py: ConfigEditor stores Path; DragDropListView keeps str in Qt items (by design), normalizes on add
components.py: video/snapshot/conditions selection uses Path; os.fspath at QFileDialog
Tabs updated: open_project, manage_project, create_project, analyze_videos, modelzoo
Other tabs mostly pass root.config_path (Path) through to DLC APIs — no per-tab refactor

`pose_estimation_pytorch` (partial)

Loaders / metadata: Loader.project_root, DLCLoader.project_path, PoseMetadata.project_path are Path
APIs: use cfg.project_path via ProjectConfig instead of Path(cfg["project_path"]) in apis/utils, analyze_images, tracklets, evaluation, visualization; dlcloader.load_ground_truth / load_split
Model zoo: modelzoo/config.py reads super-animal project config via ProjectConfig.from_yaml
Boundaries kept as str: COCO file_name, cv2.imread, Image.open, VideoReader, etc.
Public functions still typed config: str | Path at entry

Partially done / not in this PR

Area	Status
`create_project/modelzoo.py`	Mixed: still `str()` for train paths and some config writes
`create_project/new_3d.py`	`Path` for dirs; `project_path` written as `str` to YAML
`pose_estimation_pytorch` CTD `conditions` typing	Separate PR (heterogeneous YAML shape)
`pose_estimation_pytorch/metrics/scoring.py`	Still `str`-only hints
`deeplabcut.utils.*`	Largely unchanged; many functions still `str`-centric internally
`deeplabcut.compat`	Accepts `str \| Path` at API, coerces internally — not fully Path-native
`generate_training_dataset`, `refine_training_dataset`	Some `Path` usage, no systematic migration
`pose_estimation_tensorflow`	Out of scope — still `os.path` / `str` patterns
Top-level `analyze_videos`, `create_labeled_video`, etc.	Not refactored in this slice

Intentional exceptions

Qt list / line-edit text stays str
video_sets keys in YAML stay str
create_new_project return type changes from str → Path (callers using read_config / path math are fine; strict isinstance(x, str) would break)

…h Path(..).absolute()

C-Achard · 2026-05-29T11:29:26Z

Strongly approve that this is a worthwhile effort, let me know if I can help!

Copilot

Pull request overview

This PR advances the codebase-wide shift toward consistent pathlib.Path usage (reducing brittle string-based path handling) and enables Ruff’s PTH rules to help enforce that direction.

Changes:

Enable Ruff PTH lint rules in pyproject.toml.
Replace many os.path/open() call sites with Path operations (joins, exists checks, mkdir, open, glob) across utilities, TensorFlow/PyTorch pipelines, model zoo, GUI, and benchmark code.
Update several public API type hints/signatures to accept Path (and convert inputs to Path early in functions).

Reviewed changes

Copilot reviewed 107 out of 107 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
pyproject.toml	Enable Ruff `PTH` rules to encourage `pathlib` usage
deeplabcut/utils/visualization.py	Convert joins/saves to `Path`-based operations
deeplabcut/utils/skeleton.py	Replace `os` filesystem ops with `Path`
deeplabcut/utils/pseudo_label.py	Switch JSON/path/glob handling to `Path`
deeplabcut/utils/plotting.py	Use `Path` for output paths; accept `Path` inputs in API
deeplabcut/utils/make_labeled_video.py	Convert path handling to `Path`; accept `Path` inputs
deeplabcut/utils/conversioncode.py	Use `Path` for labeled-data file operations; update type hints
deeplabcut/utils/auxiliaryfunctions_3d.py	Replace glob/path ops with `Path` where applicable
deeplabcut/utils/auxfun_videos.py	Replace `os.path` usage with `Path` in video utilities
deeplabcut/utils/auxfun_multianimal.py	Use `Path.open` and `Path` joins for multi-animal utils
deeplabcut/utils/auxfun_models.py	Replace directory scanning with `Path.iterdir` and `Path` joins
deeplabcut/refine_training_dataset/tracklets.py	Use `Path.open` for pickle loading
deeplabcut/refine_training_dataset/stitch.py	Use `Path.open` and `Path`-based paths; update type hints
deeplabcut/refine_training_dataset/outlier_frames.py	Use `Path` for file IO/paths; update type hints
deeplabcut/post_processing/filtering.py	Update function type hints to accept `Path`
deeplabcut/post_processing/analyze_skeleton.py	Use `Path` for filesystem checks; update type hints
deeplabcut/pose_tracking_pytorch/train_dlctransreid.py	Replace glob/path logic with `Path.glob`
deeplabcut/pose_tracking_pytorch/processor/processor.py	Use `Path` for checkpoint/results output
deeplabcut/pose_tracking_pytorch/create_dataset.py	Use `Path` for pickles/feature files and checks
deeplabcut/pose_tracking_pytorch/config/init.py	Use `Path` for locating `reid_cfg.yaml`
deeplabcut/pose_tracking_pytorch/apis.py	Switch checkpoint handling to `Path`
deeplabcut/pose_estimation_tensorflow/visualizemaps.py	Replace many joins/cwd/exists with `Path`
deeplabcut/pose_estimation_tensorflow/util/logging.py	Remove redundant `os.path.join` for log filename
deeplabcut/pose_estimation_tensorflow/training.py	Use `Path` for model/config paths and CWD handling
deeplabcut/pose_estimation_tensorflow/predict_videos.py	Replace many joins/checks with `Path`
deeplabcut/pose_estimation_tensorflow/predict_multianimal.py	Use `Path.open` for pickles
deeplabcut/pose_estimation_tensorflow/modelzoo/api/superanimal_inference.py	Replace glob/os.path with `Path` for configs/weights/outputs
deeplabcut/pose_estimation_tensorflow/modelzoo/api/spatiotemporal_adapt.py	Replace glob/os.path with `Path`; mkdir via `Path.mkdir`
deeplabcut/pose_estimation_tensorflow/export.py	Use `Path` for export directory/files and checkpoint copying
deeplabcut/pose_estimation_tensorflow/datasets/pose_tensorpack.py	Use `Path` joins for dataset/image paths
deeplabcut/pose_estimation_tensorflow/datasets/pose_multianimal_imgaug.py	Use `Path` for config/dataset paths and IO
deeplabcut/pose_estimation_tensorflow/datasets/pose_imgaug.py	Use `Path` for dataset paths and pickle reading
deeplabcut/pose_estimation_tensorflow/datasets/pose_deterministic.py	Use `Path` for dataset/image paths and IO
deeplabcut/pose_estimation_tensorflow/core/train.py	Use `Path.cwd()` and `Path.open` for stats CSV
deeplabcut/pose_estimation_tensorflow/core/train_multianimal.py	Use `Path.cwd()` and `Path.open`; simplify snapshot prefix
deeplabcut/pose_estimation_tensorflow/core/test.py	Replace `os` directory/file ops with `Path`
deeplabcut/pose_estimation_tensorflow/core/openvino/session.py	Replace `os.path` usage with `Path` joins/parents
deeplabcut/pose_estimation_tensorflow/core/evaluate.py	Use `Path` for model/data/evaluation paths
deeplabcut/pose_estimation_tensorflow/core/evaluate_multianimal.py	Use `Path` for evaluation/data paths
deeplabcut/pose_estimation_tensorflow/config.py	Use `Path.open` when reading YAML config
deeplabcut/pose_estimation_tensorflow/backbones/mobilenet.py	Replace string path splitting with `Path` operations
deeplabcut/pose_estimation_tensorflow/backbones/efficientnet_builder.py	Use `Path` for model param file path
deeplabcut/pose_estimation_pytorch/utils.py	Use `Path` for folder creation
deeplabcut/pose_estimation_pytorch/runners/logger.py	Use `Path.open` for YAML/CSV logging IO
deeplabcut/pose_estimation_pytorch/modelzoo/utils.py	Treat DLC root as `Path` consistently
deeplabcut/pose_estimation_pytorch/modelzoo/memory_replay.py	Use `Path.open` and `Path` joins for JSON IO
deeplabcut/pose_estimation_pytorch/modelzoo/inference.py	Use `Path.open` for JSON writing
deeplabcut/pose_estimation_pytorch/modelzoo/fmpose_3d/inference.py	Use `Path.open` for JSON writing
deeplabcut/pose_estimation_pytorch/modelzoo/config.py	Use `Path.mkdir`/`Path.open` for config output
deeplabcut/pose_estimation_pytorch/metrics/scoring.py	Use `Path.open` for pickle IO
deeplabcut/pose_estimation_pytorch/data/dlcloader.py	Use `Path.open` for dataset pickle IO
deeplabcut/pose_estimation_pytorch/data/ctd.py	Use `Path.open` for conditions JSON/pickle
deeplabcut/pose_estimation_pytorch/data/cocoloader.py	Use `Path` for COCO JSON path resolution and IO
deeplabcut/pose_estimation_pytorch/config/utils.py	Treat DLC root as `Path` consistently
deeplabcut/pose_estimation_pytorch/apis/videos.py	Accept/coerce `Path` inputs; use `Path.open` for pickles
deeplabcut/pose_estimation_pytorch/apis/utils.py	Update type hints to accept `Path`
deeplabcut/pose_estimation_pytorch/apis/tracklets.py	Accept `Path` inputs; use `Path.cwd()` and `Path.open`
deeplabcut/pose_estimation_pytorch/apis/tracking_dataset.py	Accept `Path` inputs; update destfolder type hints
deeplabcut/pose_estimation_pytorch/apis/analyze_images.py	Use `Path.open` and `Path` for output folders/paths
deeplabcut/pose_estimation_3d/triangulation.py	Use `Path` for existence checks and output paths; update type hints
deeplabcut/pose_estimation_3d/plotting3D.py	Replace glob/os.path with `Path` and `Path.glob`
deeplabcut/pose_estimation_3d/camera_calibration.py	Replace glob/os.path/symlink ops with `Path`
deeplabcut/modelzoo/video_inference.py	Accept `Path` dest folder; use `Path.open` for JSON IO
deeplabcut/modelzoo/utils.py	Replace glob/os.path with `Path` and `Path.glob`
deeplabcut/modelzoo/generalized_data_converter/utils.py	Replace glob/os.path with `Path` and `Path` joins
deeplabcut/modelzoo/generalized_data_converter/datasets/single_dlc.py	Replace joins with `Path` for image paths
deeplabcut/modelzoo/generalized_data_converter/datasets/single_dlc_dataframe.py	Replace joins with `Path` for labeled-data access
deeplabcut/modelzoo/generalized_data_converter/datasets/materialize.py	Replace joins/symlink/mkdir with `Path`
deeplabcut/modelzoo/generalized_data_converter/datasets/ma_dlc.py	Replace joins with `Path` for image paths
deeplabcut/modelzoo/generalized_data_converter/datasets/ma_dlc_dataframe.py	Replace joins with `Path` for labeled-data access
deeplabcut/modelzoo/generalized_data_converter/datasets/coco.py	Use `Path` for annotation/image path handling
deeplabcut/modelzoo/generalized_data_converter/datasets/base.py	Replace `os.sep` splitting with `Path`-based name extraction
deeplabcut/modelzoo/generalized_data_converter/datasets/base_dlc.py	Use `Path` for config/dataset doc IO
deeplabcut/gui/window.py	Replace some `os.path` usage with `Path` in GUI paths
deeplabcut/gui/widgets.py	Replace file/dir checks and path joins with `Path`
deeplabcut/gui/tracklet_toolbox.py	Use `Path` for labeled-data export paths and IO
deeplabcut/gui/tabs/train_network.py	Replace logo path derivation with `Path`
deeplabcut/gui/tabs/refine_tracklets.py	Use `Path` for destination/datafile paths
deeplabcut/gui/tabs/open_project.py	Use `Path.cwd()` and `Path` for logo paths
deeplabcut/gui/tabs/modelzoo.py	Replace icon path joins with `Path`
deeplabcut/gui/tabs/manage_project.py	Use `Path.cwd()` for file dialogs
deeplabcut/gui/tabs/label_frames.py	Accept `Path` image_folder; use `Path` for directory iteration
deeplabcut/gui/tabs/evaluate_network.py	Use `Path` for evaluation maps directory and listing
deeplabcut/gui/tabs/create_training_dataset.py	Use `Path` for training artifact checks
deeplabcut/gui/tabs/create_project.py	Use `Path` for video suffix checks and default paths
deeplabcut/gui/launch_script.py	Use `Path` for asset paths and stylesheet loading
deeplabcut/gui/displays/shuffle_metadata_viewer.py	Use `Path.open` for reading metadata
deeplabcut/gui/displays/selected_shuffle_display.py	Pass `Path` to `read_plainconfig` consistently
deeplabcut/gui/components.py	Replace abspath/basename with `Path.resolve()`/`.name`
deeplabcut/gui/init.py	Represent `BASE_DIR` as a `Path`
deeplabcut/generate_training_dataset/multiple_individuals_trainingsetmanipulation.py	Use `Path` joins and `Path.unlink` for dataset recreation
deeplabcut/generate_training_dataset/metadata.py	Use `Path.open` and `Path.exists` consistently
deeplabcut/generate_training_dataset/frame_extraction.py	Accept `Path` config; use `Path` glob/iterdir/unlink
deeplabcut/create_project/new.py	Update API to accept `Path`; use `Path` for config/symlinks
deeplabcut/create_project/new_3d.py	Use `Path` for config output path
deeplabcut/create_project/modelzoo.py	Use `Path.open` for template read/write; use `Path.cwd()`
deeplabcut/create_project/demo_data.py	Accept `Path` config; use `Path` for demo video paths
deeplabcut/create_project/add.py	Accept `Path`; use `Path` for symlink/readlink
deeplabcut/core/inferenceutils.py	Use `Path.open` for assembly pickle IO
deeplabcut/core/crossvalutils.py	Replace path splitting/joining and pickle IO with `Path`
deeplabcut/core/config.py	Use `Path.open` for YAML config read/write
deeplabcut/compat.py	Coerce config/videos/destfolder to `Path`; update API type hints
deeplabcut/benchmark/utils.py	Use `Path.open` for `/dev/null` redirection
deeplabcut/benchmark/metrics.py	Use `Path` for file IO/path assembly in benchmarking
deeplabcut/benchmark/init.py	Use `Path` for data/cache locations and IO

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Fix path handling when loading the dataset pickle. The code previously attempted Path(project_path) / file_name even though file_name already included the project path, resulting in an incorrect path. This changes the context manager to call file_name.open('rb') directly, removing the redundant join and preventing path errors when loading the pickle.

Replace direct ruamel_file.dump(...open('w')) call with a with-statement to open pose_cfg.yaml. This ensures the file handle is properly closed after writing and avoids potential resource leaks.

Convert video_path to str when constructing cv2.VideoCapture to ensure pathlib.Path objects are handled correctly and avoid type errors when opening videos. This makes VideoReader more robust when given Path instances.

Replace the list-comprehension side-effect used to delete temporary frame files with an explicit for-loop that checks p.is_file() before calling unlink. This avoids building an unused list, prevents errors from trying to unlink non-file entries (e.g., directories), and improves readability and safety.

C-Achard

Here is a first pass, I addressed a few things I thought were not too discussion-worthy directly, please see the latest commits!

PoseConfig.model.backbone is currently a dict-type but freeze_bn_stats were set as attribute. This commit fixes that and treats backbone correctly as dict.

deruyter92 · 2026-07-09T12:47:16Z

Important

Merge #3349 before this PR.

I've changed the base to main to trigger all functional tests, but this PR builds on top of #3349 which should be merged first.

deruyter92 · 2026-07-09T15:08:32Z

@C-Achard, I tried to scope it a bit to a stable first phase (see PR description). The migration can be a bit tricky still if downstream code receives Path now instead of str. But those cases will be easy to fix in case they pop up.

Let me know what you think.

We can also try to go all the way all at once, and do everything (even including the TensorFlow branch)

C-Achard

Apologies, I thought I had approved from VSCode, not sure where that went. I think the current scope is good, thanks for the effort! We can keep converting later, better to chunk it down imo.

deruyter92 added 3 commits May 29, 2026 09:51

fix path resolution on network drives. Replace Path(..).resolve() wit…

3b8c44a

…h Path(..).absolute()

Add safe_resolve helper for path resolution

fddcbaa

replace str with pathlib.Path

2403b72

deruyter92 changed the base branch from main to jaap/safe_resolve_paths May 29, 2026 08:51

deruyter92 mentioned this pull request May 29, 2026

read_config() project path resolution can break path #3348

Closed

2 tasks

deruyter92 added 4 commits May 29, 2026 11:16

fix redundant resolution of resolved path in memory_replay.py

65f8dfb

fix open -> os.stat

6bbe26d

replace os.stat -> open for files, listdir for dirs.

3f93a76

fix TestReadConfigProjectPath

87cd6aa

deruyter92 and others added 3 commits May 29, 2026 15:10

Update deeplabcut/utils/auxiliaryfunctions.py

8be1c26

additional replacements: str ->Path

f2bb35d

Update function signatures str -> str | Path

810406a

deruyter92 force-pushed the jaap/pathlib_path_policy branch from b9de7c2 to 810406a Compare June 4, 2026 14:55

Add ruff linting rule PTH

c7835bc

C-Achard reviewed Jun 5, 2026

View reviewed changes

Comment thread pyproject.toml Outdated

C-Achard assigned deruyter92 Jun 15, 2026

C-Achard added the enhancement New feature or request label Jun 15, 2026

C-Achard mentioned this pull request Jun 15, 2026

Update GUI assets import #3370

Merged

C-Achard requested a review from Copilot June 15, 2026 16:05

Copilot started reviewing on behalf of C-Achard June 15, 2026 16:06 View session

Copilot AI reviewed Jun 15, 2026

View reviewed changes

C-Achard added 4 commits June 15, 2026 13:04

Use context manager when writing pose_cfg.yaml

156c25f

Replace direct ruamel_file.dump(...open('w')) call with a with-statement to open pose_cfg.yaml. This ensures the file handle is properly closed after writing and avoids potential resource leaks.

Pass string path to cv2.VideoCapture

d854aa5

Convert video_path to str when constructing cv2.VideoCapture to ensure pathlib.Path objects are handled correctly and avoid type errors when opening videos. This makes VideoReader more robust when given Path instances.

C-Achard reviewed Jun 15, 2026

View reviewed changes

Fix CircleCI failure

e896398

C-Achard mentioned this pull request Jun 15, 2026

Fix PySide6 stylesheet usage #3371

Merged

fix Matplotlib Path import shadowing ->MPLPath

132b882

deruyter92 added 3 commits July 7, 2026 15:51

GUI use os.fspath for str conversions where required

af613f3

use pathlib Path in create_project

1586e2a

use pathlib Path across pose_estimation_pytorch internally

c49d392

deruyter92 removed request for AlexEMG and MMathisLab July 7, 2026 14:43

deruyter92 changed the title ~~Enforce pathlib.Path instead of str~~ Migrate to pathlib.Path instead of str (first phase) Jul 7, 2026

deruyter92 added 7 commits July 9, 2026 13:39

use centralized yaml dumper (which normalizes Path -> str)

6a85abb

fix materialize.py downstream old str.split -> Path

33c7b58

fix testsscripts str usage

efebf8d

fix old str usage in trainingsetmanipulation

971f381

allow str type at TF boundary

292b251

allow str formatting in tensorflow code

1330eab

fix config update in train_from_coco.py

84b5c83

PoseConfig.model.backbone is currently a dict-type but freeze_bn_stats were set as attribute. This commit fixes that and treats backbone correctly as dict.

deruyter92 changed the base branch from jaap/safe_resolve_paths to main July 9, 2026 12:42

deruyter92 requested a review from C-Achard July 9, 2026 15:03

C-Achard approved these changes Jul 14, 2026

View reviewed changes

deruyter92 requested review from AlexEMG and MMathisLab July 15, 2026 12:15

deruyter92 assigned AlexEMG and MMathisLab Jul 15, 2026

MMathisLab approved these changes Jul 15, 2026

View reviewed changes

deruyter92 mentioned this pull request Jul 15, 2026

GUI: Config validation errors handler dialog & layout fixes #3397

Merged

3 tasks

deruyter92 merged commit 30a8155 into main Jul 15, 2026
30 checks passed

deruyter92 deleted the jaap/pathlib_path_policy branch July 15, 2026 15:53

C-Achard mentioned this pull request Jul 16, 2026

GUI: Fix redundant signal calls, streamline path selections #3395

Merged

This was referenced Jul 17, 2026

Fix network-drive path corruption in read_config() and related call sites #3349

Closed

Harden path-handling after pathlib migration #3421

Merged

version bump 3.0.1 #3423

Merged

Uh oh!

Uh oh!

Conversation

deruyter92 commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Policy

Scope

Done in this PR

create_project (2D core)

GUI

pose_estimation_pytorch (partial)

Partially done / not in this PR

Intentional exceptions

Uh oh!

C-Achard commented May 29, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

C-Achard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

deruyter92 commented Jul 9, 2026

Uh oh!

deruyter92 commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

C-Achard left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

deruyter92 commented May 29, 2026 •

edited

Loading

`create_project` (2D core)

`pose_estimation_pytorch` (partial)

deruyter92 commented Jul 9, 2026 •

edited

Loading

C-Achard left a comment •

edited

Loading