Skip to content

Can no longer deepcopy LogNorm objects on master #18119

Closed
sunpy/sunpy
#4877

Description

@dstansby

Bug report

Bug summary

As of 0d1d95c on the master branch it is no longer possible to deepcopy a LogNorm instance, which is a regression from 3.3.x.

Code for reproduction

import matplotlib.colors as mcolor
import copy
norm = mcolor.LogNorm()
copy.deepcopy(norm)

Actual outcome

Traceback (most recent call last):
  File "test.py", line 4, in <module>
    copy.deepcopy(norm)
  File "/Users/dstansby/miniconda3/envs/dev/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Users/dstansby/miniconda3/envs/dev/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Users/dstansby/miniconda3/envs/dev/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Users/dstansby/miniconda3/envs/dev/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Users/dstansby/miniconda3/envs/dev/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Users/dstansby/miniconda3/envs/dev/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Users/dstansby/miniconda3/envs/dev/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Users/dstansby/miniconda3/envs/dev/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Users/dstansby/miniconda3/envs/dev/lib/python3.8/copy.py", line 153, in deepcopy
    y = copier(memo)
  File "/Users/dstansby/github/matplotlib/lib/matplotlib/transforms.py", line 143, in __copy__
    raise NotImplementedError(
NotImplementedError: TransformNode instances can not be copied. Consider using frozen() instead.

Expected outcome

No error

Matplotlib version

  • Operating system:
  • Matplotlib version:
  • Matplotlib backend (print(matplotlib.get_backend())):
  • Python version:
  • Jupyter version (if applicable):
  • Other libraries:

Activity

  1. jklymak commented on Jul 29, 2020

    @jklymak
    Member

    So norms now derive from scales (#16457), and scales have probably never been deep copyable.

    import matplotlib.scale as mscale
    import copy
    
    sc = mscale.LogTransform(10)
    sc2 = copy.deepcopy(sc)
    Traceback (most recent call last):
      File "testScaleCopy.py", line 5, in <module>
        sc2 = copy.deepcopy(sc)
      File "/Users/jklymak/opt/anaconda3/envs/matplotlibdev/lib/python3.8/copy.py", line 153, in deepcopy
        y = copier(memo)
      File "/Users/jklymak/matplotlib/lib/matplotlib/transforms.py", line 143, in __copy__
        raise NotImplementedError(
    NotImplementedError: TransformNode instances can not be copied. Consider using frozen() instead.
    

    Previously norms did not use the transform stack, but now that they are based on scales, they use it.

    I guess a "fix" is to write a __deepcopy__ method for the scales if there is a strong need to call deepcopy (I guess I can sort of see the use)

  2. added this to the v3.3.1 milestone on Jul 29, 2020
  3. dstansby commented on Jul 29, 2020

    @dstansby
    MemberAuthor

    The specific use case here is passing a norm to a plot method that autoscales vmin/vmax in place, but taking a copy to avoid modifying the vmin/vmax on the original norm object.

  4. jklymak commented on Jul 29, 2020

    @jklymak
    Member

    Implementing a __deepcopy__ of the scale seems straightforward enough, but it does require keeping a lot more of the scale init info around as extra private variables. Before doing all that (to all the scales) I wonder if anyone else has solutions...

    ie for LogScale we would need to keep track of axis_name and nonpositive from the __init__.

        def __deepcopy__(self, memo):
            return self.__init__(self._axis_name, base=self.base, subs=self.subs,
                                 nonpositive=self._nonpositive)
  5. jklymak commented on Jul 29, 2020

    @jklymak
    Member

    Ugh, but the same song and dance would have to be done for the norm as well because it holds its transform now. I guess the other alternative is implementing __deepcopy__ for the transform.

  6. dstansby commented on Jul 29, 2020

    @dstansby
    MemberAuthor

    Does one need to implement a custom __deepcopy__ instead of just relying on the python default? I just removed the "Cannot be copied" error from TransformNode and at least my simple example worked fine.

  7. jklymak commented on Jul 29, 2020

    @jklymak
    Member

    Yeah, I don't know why deepcopy might be deemed problematic for transforms, or why the frozen was used instead.

  8. dstansby commented on Jul 29, 2020

    @dstansby
    MemberAuthor

    Looks like it's been around since forever (2007), with no explanation as to why the restriction on copying is there. Maybe I'll remove it and see what happens to the CI...

  9. jklymak commented on Jul 29, 2020

    @jklymak
    Member

    I doubt it will universally work - the TransformNode subclasses all have frozen defined with special hooks into internal state. But maybe its all obsolete and deepcopy is more robust these days?

  10. tacaswell commented on Jul 29, 2020

    @tacaswell
    Member

    frozen is an internal thing.

    The issue with transforms is that they are a branching partially-cached call graph. That is the ax.transData both has a cached version of the transform from data space -> physical screen space, it knows that it is a compound of the figure transfrom, the dpi, transform, and the axes transform. If any of it's parents get changed it is notified that it has to be re-computed. I suspect that the number of links in that chain is why it is forbidden (along with the theory that it is better to not allowing something we don't need internally than it is to expose a wrong functionality).

    A way to implement deepcopy using things we do test is new = pickle.loads(pickle.dumps(obj))

  11. jklymak commented on Jul 29, 2020

    @jklymak
    Member

    Thanks @tacaswell

    It seems to me that scales can use frozen transforms? i.e. they never depend on any children, do they? It seems to me that if a transform was frozen then copying it would be safe. But maybe I'm not following the subtleties.

  12. QuLogic commented on Jul 30, 2020

    @QuLogic
    Member

    They could be frozen, but also, the transforms in scale.py could override __copy__? Just like LogTransform overrides .inverted to return InvertedLogTransform.

    I don't think scales use an transforms outside of the scale.py ones, except for the identity.

  13. QuLogic commented on Jul 30, 2020

    @QuLogic
    Member

    So frozen just returns self, which seems the opposite of what you want for copy, or at least deepcopy, doesn't it?

  14. jklymak commented on Jul 30, 2020

    @jklymak
    Member

    So if I add

        def __deepcopy__(self, memo):
            return self.frozen()

    to LogTransform and InvertedLogTransform in scale then

    import matplotlib.scale as mscale
    import copy
    
    sc = mscale.LogScale(axis='x', base=10)
    sc2 = copy.deepcopy(sc)
    
    print(sc2)
    print('deon')
    import matplotlib.colors as mcolor
    norm = mcolor.LogNorm()
    norm.vmin = -10
    norm2 = copy.deepcopy(norm)
    
    print(norm2, norm2.vmin)

    returns what I'd expect:

    <matplotlib.scale.LogScale object at 0x7fdf522a5970>
    deon
    <matplotlib.colors.LogNorm object at 0x7fdf522d6f40> -10
    

    I somewhat hesitate to do all this because the transform stack seems delicate, but given that scale.py is at the end, I think its OK...

  15. modified the milestones: v3.3.1, v3.4.0 on Jul 30, 2020
  16. added
    Release criticalFor bugs that make the library unusable (segfaults, incorrect plots, etc) and major regressions.
    on Dec 31, 2020
  17. dstansby commented on Dec 31, 2020

    @dstansby
    MemberAuthor

    I'm going to be a little bit greedy, and mark this as Release critical. I think plotting multiple images with the same norm, but automatically autoscaling each one (which requires individual (deep)copies of the original norm object), is a reasonable use case, and this is a regression from 3.3.x.

  18. jklymak commented on Dec 31, 2020

    @jklymak
    Member

    I mean there is a PR for it. Does it not fix the problem? @tacaswell seemed to have some issue with it but I frankly did not fully understand what it was.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    API: consistencyConsistency of the matplotlib API, including naming, behavior, defaults, …Release criticalFor bugs that make the library unusable (segfaults, incorrect plots, etc) and major regressions.topic: transforms and scales

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions