You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: plt.hist takes significantly more time with torch and jax arrays #25882
Time taken to plot plt.hist directly on jax or torch arrays is significantly more than combined time taken to first convert them to numpy and then using plt.hist. Shouldn't matplotlib internally convert them to numpy arrays before plotting?
To reproduce the bug, directly run the following snippet on Google Colab.
"""Internal helper to extract data from e.g. pandas and xarray objects."""
ifisinstance(x, np.ndarray):
# If numpy, return directly
returnx
ifhasattr(x, 'to_numpy'):
# Assume that any to_numpy() method actually returns a numpy array
returnx.to_numpy()
ifhasattr(x, 'values'):
xtmp=x.values
# For example a dict has a 'values' attribute, but it is not a property
# so in this case we do not want to return a function
ifisinstance(xtmp, np.ndarray):
returnxtmp
returnx
The pytorch tensor does not support any of the conversion methods, so Matplotlib doesn't really know what to do with it. There is a discussion in #22645 about this, but if I remember correctly we expect the libraries to support the to_numpy method (but still support the values attribute).
(I could not install jax, but I suppose something similar goes on there.)
Thanks for the quick response, @oscargus! Given that both these libraries support .__array__() method for conversion to numpy array, wouldn't it be easier to add one more if condition in _unpack_to_numpy to include them?
Yes, I also noted that. It probably can make sense.
(I think the reason why we do this somewhat carefully is for unit information to not get lost.)
Would you be interested in submitting a patch? I think that if this goes last in the conversion chain, it shouldn't break too many things... (A problem here is that we do not, yet, test for "all" types that possibly can be used and "works". There's been a discussion of having a special test suite for that, but it has not yet been implemented.)
Even tensorflow supports __array__() method. I guess these 3 libraries account for almost 99% of the machine learning codebase available online :) It'd be great if this conversion passes without breaking many things!
Sure, I'd submit a patch. I guess I need to only change the _unpack_to_numpy to the following, right?
def_unpack_to_numpy(x):
"""Internal helper to extract data from e.g. pandas and xarray objects."""ifisinstance(x, np.ndarray):
# If numpy, return directly returnxifhasattr(x, 'to_numpy'):
# Assume that any to_numpy() method actually returns a numpy array returnx.to_numpy()
ifhasattr(x, 'values'):
xtmp=x.values# For example a dict has a 'values' attribute, but it is not a property # so in this case we do not want to return a function ifisinstance(xtmp, np.ndarray):
returnxtmpifhasattr(x, '__array__'):
# Assume that any to __array__() method returns a numpy array (e.g. TensorFlow, JAX or PyTorch arrays)returnx.__array__()
returnx
Thank you for the important suggestion, @timhoffm. Now, __array__ method check works in theory for the cases I imagined but np.float32 type objects get stuck into that check. When __array__ method is called on np.float32 object, it gets converted to ndarray type and eventually this leads to an infinite recursion.
A temporary fix I could figure out is to add two more if conditions to check if object is of type np.floating (includes all float types) and type np.integer (includes all integer types including uint). I can also include a boolean check. Will it be all or this already looks unpythonic?
More directions to solve this issue could be the following:
Raise an issue to add to_numpy() methods in JAX and PyTorch repos.
Raise an issue to have a universal numpy object checker type in NumPy library so that we can replace ndarray check with that. After this, any numpy object will be captured in the first check.
Add hard-coded checks for JAX and PyTorch like the following:
Edit1: np.generic works for most (all?) scalars, so we can add if isinstance(x, np.generic) as the second check just after ndarray check like the following:
def_unpack_to_numpy(x):
"""Internal helper to extract data from e.g. pandas and xarray objects."""ifisinstance(x, np.ndarray):
# If numpy array, return directlyreturnxifisinstance(x, np.generic):
# If numpy scalar, return directlyreturnxifhasattr(x, 'to_numpy'):
# Assume that any to_numpy() method actually returns a numpy arrayreturnx.to_numpy()
ifhasattr(x, 'values'):
xtmp=x.values# For example a dict has a 'values' attribute, but it is not a property# so in this case we do not want to return a functionifisinstance(xtmp, np.ndarray):
returnxtmpifhasattr(x, '__array__'):
# Assume that any to __array__() method returns a numpy array (e.g. TensorFlow, JAX or PyTorch arrays)x=x.__array__()
# Anything that doesn't return ndarray via __array__() method will be filtered by the following checkifisinstance(x, np.ndarray):
returnxreturnx
I am not convinced this is a high-priority problem as there is a very simple user-side solution (cast to numpy before passing to Matplotlib!).
Given that, to my understanding, some of these arrays may actually be in GPU memory, forcing them back to CPU memory before passing to Matplotlib is probably a good idea anyway.
I've just ran into the issue and Googled my way to here. While the solution is simple, it took me a while to find which is the offending function, so I'd support fixing it.
To address the issue of the tensor being in GPU, a simple check on what device the tensor is on could be added. My execution crashed with 10K elements on CPU, which I expected to take less than a second to plot.
To address the issue of the tensor being in GPU, a simple check on what device the tensor is on could be added. My execution crashed with 10K elements on CPU, which I expected to take less than a second to plot.
Just to set expectations, that is a request to torch, or something the user needs to take care of, not to Matplotlib. We don't officially support torch or jax arrays, and we aren't going to have extra checks to support them beyond what __array__ might support
Activity
oscargus commentedon May 13, 2023
The unpacking happens here:
matplotlib/lib/matplotlib/cbook.py
Lines 2237 to 2251 in b61bb0b
The pytorch tensor does not support any of the conversion methods, so Matplotlib doesn't really know what to do with it. There is a discussion in #22645 about this, but if I remember correctly we expect the libraries to support the
to_numpymethod (but still support thevaluesattribute).(I could not install jax, but I suppose something similar goes on there.)
oscargus commentedon May 13, 2023
And when the conversion doesn't work, it ends up in this loop:
matplotlib/lib/matplotlib/cbook.py
Lines 1332 to 1348 in b61bb0b
which is where most of the time is spent.
patel-zeel commentedon May 13, 2023
Thanks for the quick response, @oscargus! Given that both these libraries support
.__array__()method for conversion to numpy array, wouldn't it be easier to add one moreifcondition in_unpack_to_numpyto include them?oscargus commentedon May 13, 2023
Yes, I also noted that. It probably can make sense.
(I think the reason why we do this somewhat carefully is for unit information to not get lost.)
Would you be interested in submitting a patch? I think that if this goes last in the conversion chain, it shouldn't break too many things... (A problem here is that we do not, yet, test for "all" types that possibly can be used and "works". There's been a discussion of having a special test suite for that, but it has not yet been implemented.)
patel-zeel commentedon May 13, 2023
Even
tensorflowsupports__array__()method. I guess these 3 libraries account for almost 99% of the machine learning codebase available online :) It'd be great if this conversion passes without breaking many things!Sure, I'd submit a patch. I guess I need to only change the
_unpack_to_numpyto the following, right?timhoffm commentedon May 13, 2023
Yes, but please verify that
__array__actually returns a numpy array, like we do withvaluesabove.patel-zeel commentedon May 13, 2023
Thank you for the important suggestion, @timhoffm. Now,
__array__method check works in theory for the cases I imagined butnp.float32type objects get stuck into that check. When__array__method is called onnp.float32object, it gets converted tondarraytype and eventually this leads to an infinite recursion.A temporary fix I could figure out is to add two more
ifconditions to check if object is of typenp.floating(includes all float types) and typenp.integer(includes all integer types includinguint). I can also include a boolean check. Will it be all or this already looks unpythonic?More directions to solve this issue could be the following:
to_numpy()methods in JAX and PyTorch repos.numpyobject checker type in NumPy library so that we can replacendarraycheck with that. After this, any numpy object will be captured in the first check.I am open to your suggestions.
Edit1:
np.genericworks for most (all?) scalars, so we can addif isinstance(x, np.generic)as the second check just afterndarraycheck like the following:tacaswell commentedon May 18, 2023
I am not convinced this is a high-priority problem as there is a very simple user-side solution (cast to numpy before passing to Matplotlib!).
Given that, to my understanding, some of these arrays may actually be in GPU memory, forcing them back to CPU memory before passing to Matplotlib is probably a good idea anyway.
jklymak commentedon May 18, 2023
I agree this isn't high priority, but if an object has a numpy array representation, why would we ignore it?
xstreck1 commentedon Oct 19, 2023
I've just ran into the issue and Googled my way to here. While the solution is simple, it took me a while to find which is the offending function, so I'd support fixing it.
To address the issue of the tensor being in GPU, a simple check on what device the tensor is on could be added. My execution crashed with 10K elements on CPU, which I expected to take less than a second to plot.
jklymak commentedon Oct 22, 2023
Just to set expectations, that is a request to torch, or something the user needs to take care of, not to Matplotlib. We don't officially support torch or jax arrays, and we aren't going to have extra checks to support them beyond what
__array__might supporttimhoffm commentedon Dec 30, 2023
Sorry this slipped.
The code suggested in Edit1 of #25882 (comment) looks reasonable to me.
Tests should include the numpy scalar case and a mock object with an
__array__method (we don’t want to depend on torch or JAX).@patel-zeel do you want to pick this up again.
patel-zeel commentedon Dec 30, 2023
Sure, @timhoffm! I will try this fix in my existing PR #25887.