Repository navigation
Closed
Milestone
Description
Bug summary
matplotlib refuses to make a plot because it thinks the input is 2-dimensional.
Code for reproduction
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
x, y = np.random.randn(2, 100)
x = pd.Series(x, dtype="Float32") # <-- pandas nullable type
y = pd.Series(y, dtype="Float32") # <-- pandas nullable type
plt.plot(list(x), list(y)) # <-- works fine
plt.plot(x, y) # <-- throws errorActual outcome
`matplotlib` throws an error
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/series.py in _get_values(self, indexer)
1025 try:
-> 1026 new_mgr = self._mgr.getitem_mgr(indexer)
1027 return self._constructor(new_mgr).__finalize__(self)
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/internals/managers.py in getitem_mgr(self, indexer)
1635 blk = self._block
-> 1636 array = blk._slice(indexer)
1637 if array.ndim > 1:
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/internals/blocks.py in _slice(self, slicer)
1556
-> 1557 return self.values[slicer]
1558
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/arrays/masked.py in __getitem__(self, item)
145
--> 146 return type(self)(self._data[item], self._mask[item])
147
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/arrays/floating.py in __init__(self, values, mask, copy)
254 )
--> 255 super().__init__(values, mask, copy=copy)
256
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/arrays/masked.py in __init__(self, values, mask, copy)
122 if values.ndim != 1:
--> 123 raise ValueError("values must be a 1D array")
124 if mask.ndim != 1:
ValueError: values must be a 1D array
During handling of the above exception, another exception occurred:
ValueError Traceback (most recent call last)
/tmp/ipykernel_409521/2612272364.py in <module>
7 y = pd.Series(y, dtype="Float32") # <-- pandas nullable type
8 plt.plot(list(x), list(y)) # <-- works fine
----> 9 plt.plot(x, y) # <-- throws error
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/matplotlib/pyplot.py in plot(scalex, scaley, data, *args, **kwargs)
2755 @_copy_docstring_and_deprecators(Axes.plot)
2756 def plot(*args, scalex=True, scaley=True, data=None, **kwargs):
-> 2757 return gca().plot(
2758 *args, scalex=scalex, scaley=scaley,
2759 **({"data": data} if data is not None else {}), **kwargs)
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/matplotlib/axes/_axes.py in plot(self, scalex, scaley, data, *args, **kwargs)
1630 """
1631 kwargs = cbook.normalize_kwargs(kwargs, mlines.Line2D)
-> 1632 lines = [*self._get_lines(*args, data=data, **kwargs)]
1633 for line in lines:
1634 self.add_line(line)
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/matplotlib/axes/_base.py in __call__(self, data, *args, **kwargs)
310 this += args[0],
311 args = args[1:]
--> 312 yield from self._plot_args(this, kwargs)
313
314 def get_next_color(self):
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/matplotlib/axes/_base.py in _plot_args(self, tup, kwargs, return_kwargs)
485
486 if len(xy) == 2:
--> 487 x = _check_1d(xy[0])
488 y = _check_1d(xy[1])
489 else:
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/matplotlib/cbook/__init__.py in _check_1d(x)
1325 message='Support for multi-dimensional indexing')
1326
-> 1327 ndim = x[:, None].ndim
1328 # we have definitely hit a pandas index or series object
1329 # cast to a numpy array.
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/series.py in __getitem__(self, key)
964 return self._get_values(key)
965
--> 966 return self._get_with(key)
967
968 def _get_with(self, key):
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/series.py in _get_with(self, key)
979 )
980 elif isinstance(key, tuple):
--> 981 return self._get_values_tuple(key)
982
983 elif not is_list_like(key):
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/series.py in _get_values_tuple(self, key)
1009 # mpl hackaround
1010 if com.any_none(*key):
-> 1011 result = self._get_values(key)
1012 deprecate_ndim_indexing(result, stacklevel=5)
1013 return result
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/series.py in _get_values(self, indexer)
1030 # see tests.series.timeseries.test_mpl_compat_hack
1031 # the asarray is needed to avoid returning a 2D DatetimeArray
-> 1032 return np.asarray(self._values[indexer])
1033
1034 def _get_value(self, label, takeable: bool = False):
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/arrays/masked.py in __getitem__(self, item)
144 item = check_array_indexer(self, item)
145
--> 146 return type(self)(self._data[item], self._mask[item])
147
148 @doc(ExtensionArray.fillna)
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/arrays/floating.py in __init__(self, values, mask, copy)
253 "the 'pd.array' function instead"
254 )
--> 255 super().__init__(values, mask, copy=copy)
256
257 @classmethod
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/arrays/masked.py in __init__(self, values, mask, copy)
121 )
122 if values.ndim != 1:
--> 123 raise ValueError("values must be a 1D array")
124 if mask.ndim != 1:
125 raise ValueError("mask must be a 1D array")
ValueError: values must be a 1D array
Expected outcome
It should plot the array normally in this case. No missing values are present and at the very least, I would expect matplotlib to try plot(list(x), list(y)) as a fallback mechanism.
On a secondary level, it would be kind of nice if matplotlib could detect and treat pandas.NA as if it were numpy.nan. (e.g., the list approach currently does not work if any missing values are present. But I think this should probably be a seperate issue.)
Activity
jklymak commentedon Jan 6, 2022
@tacaswell, pandas 1.0 is 2 years old in a couple of weeks . How far back does newish matplotlib need to support? With my pandas and matplotlib, if I replace
cbook._check_1dbynp.atleast_1dI think all the tests pass (and it fixes this incompatibility with a newish data type)jklymak commentedon Jan 6, 2022
.. the other practical fix here is to expand the except in
_check_1dto includeValueError.jklymak commentedon Jan 7, 2022
I really think this is a pandas bug, with
FloatingArraynot supportingndarray-like slicing. See #22141Personally I am skeptical of pandas approach here; it seems they have decided they need a NA in addition to NaN, and instead of getting numpy to agree they have made their own Array type. I don't think that Matplotlib should feel obliged to support whatever downstream throws at us. (So far as I can tell, pandas.arrays.FloatingArray is not even documented).
randolf-scholz commentedon Jan 7, 2022
@jklymak
I completely agree with this sentiment, yet, do you not agree it would be useful to have a fallback mechanism like I proposed?
In my example,
plot(list(x), list(y))works. I do not see the hurt in having something like this as a fallback mechanism if everything else fails. E.g. before throwing an error and refusing to plot, try casting the inputs as lists and create the plots from these lists, possibly accompanied with a warningfailed to interpret inputs, using fallback modeor something. This would be in line with the Robustness principle, wouldn't it?jklymak commentedon Jan 7, 2022
Well, see the code around #22141 - it is fallback after fallback after fallback. Just converting to a list isn't particularly robust if the list elements are objects we can't recognize. They are so far, but it seems pandas is even considering
x.to_numpy()to return numpy object arrays full of their ownFloat32instead of numpyfloat64arrays.tacaswell commentedon Jan 10, 2022
Per https://matplotlib.org/stable/devel/min_dep_policy.html 24 months so pandas 1.0 is about to fall out of support for the main branch but not for 3.5.x. I would argue that simplifying work-arounds to pandas issues counts as "we need new features".
I think I agree, I think they are raising a different exception than we are expecting and inconsistent by dtype. Opening a bug with them now.