Guard CoW read-only .to_numpy()/.values mutations under pandas 3 - #3445
Open
AxelNoun wants to merge 9 commits into
Open
Guard CoW read-only .to_numpy()/.values mutations under pandas 3#3445AxelNoun wants to merge 9 commits into
AxelNoun wants to merge 9 commits into
Conversation
Under pandas 3.0 Copy-on-Write, .to_numpy()/.values return read-only arrays for single-dtype selections; in-place mutation raises ValueError. Use copy=True at the five affected sites. Pin unchanged.
Follow-up to the pandas 3.0 CoW sweep (DeepLabCut#3362). Three more sites extract a NumPy array from a pandas object and mutate it in place; under pandas 3 CoW these can be read-only views, raising "assignment destination is read-only". - refine_training_dataset/tracklets.py: self.data backs self.xy/self.prob, which swap_tracklets and the refine GUI mutate in place (HDF load path; flagged in review). - utils/make_labeled_video.py: coords is a view into xyp, masked in place in both the first-frame and per-frame branches. - pose_estimation_tensorflow/.../pose_multianimal_imgaug.py: a single-row float Series is read-only; kpts is masked in place when mask_kpts_below_thresh is set. Co-authored-by: Cursor <cursoragent@cursor.com>
Guards against the in-place write on the next line producing a read-only view when the intermediate frame becomes single-block.
C-Achard
approved these changes
Aug 13, 2026
C-Achard
left a comment
C-Achard
left a comment
Collaborator
There was a problem hiding this comment.
@deruyter92 As mentioned, PR is identical to previously, so I'm re-approving for convenience
deruyter92
approved these changes
Aug 13, 2026
deruyter92
left a comment
deruyter92
left a comment
Collaborator
There was a problem hiding this comment.
@AxelNoun thanks for re-opening. Again, great PR, thanks for your contribution!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Recreates #3416, which was closed automatically on 12 Aug when the fork
hosting its branch was detached during an account cleanup on my side —
my mistake, apologies for the noise.
The head commit is byte-identical to the original PR (
a3977ff926), so itstill includes your own commits on that branch, @deruyter92 — nothing was
rebased or dropped. The original review thread remains readable at
#3416, where the PR was approved.
Guard in-place mutations of
.to_numpy()/.valuesresults (Copy-on-Write, pandas 3.0)Follow-up to #3360. Related to the tracking issue #3362.
Summary
Under pandas 3.0's Copy-on-Write,
.to_numpy()and.valuesreturn aread-only array for single-dtype selections; mutating that array in place
raises
ValueError: assignment destination is read-only. This PR guards the fivesites where such an array is mutated, using
to_numpy(copy=True).The
pandas<3upper bound is intentionally left unchanged — this is aforward-compatibility fix, not the pin removal discussed in #3362. It complements
the preparatory work already merged in #3360 and is safe on the currently pinned
pandas 2.x.
The pattern
copy=Truereturns a writable array and is behavior-preserving on pandas 2.x, sothe change is correct under both major versions.
Changes
to_numpy(copy=True)at the sites that mutate the extracted array in place:pose_estimation_pytorch/data/ctd.pypose_estimation_3d/triangulation.pypose_estimation_3d/plotting3D.pyrefine_training_dataset/tracklets.pypost_processing/filtering.py— spline-filter branch (same interpolationpattern as
tracklets.py; only thetrackletscopy is covered by the currenttests, so this one was located by a static scan rather than a failing test)
Considered and left unchanged:
pose_estimation_tensorflow/core/evaluate_multianimal.pyextracts a mixed-dtype selection (
["sample", "y", "x", "bodyparts"]), whichyields a writable object array and is therefore not affected.
Testing
the fix, and the spline-filter path is smoke-tested (writable arrays, correct
gap-filling).
behavior-preserving.
Notes for #3362
runtime under pandas 3.0, so it isn't caught by the 2.3
future-mode tooling.Migration to Pandas 3.0 #3362 ("PyTables cannot serialize a MultiIndex whose levels use extension
dtypes -> change all dataframes back to
objectbefore saving") does notreproduce: pandas 3.0 explicitly whitelists
StringDtypein that guard(
io/pytables.py,write_multi_index:isinstance(lev.dtype, ExtensionDtype) and not isinstance(lev.dtype, StringDtype)),and DeepLabCut's string index/column levels serialize fine (verified for the
df_with_missing/tracks/predictions/asskeys). So noto_hdfobject-conversion wrapper appears necessary on the released 3.0. (The pandas 2.3
future.infer_stringmode does raise there, since the exemption only lands in3.0 — so that mode shouldn't be used to validate HDF writes.)