Skip to content

[Bug]: plt.plot thinks pandas.Series is 2-dimensional when nullable data type is used #22125

Closed

Description

@randolf-scholz

Bug summary

matplotlib refuses to make a plot because it thinks the input is 2-dimensional.

Code for reproduction

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

x, y = np.random.randn(2, 100)
x = pd.Series(x, dtype="Float32")  # <-- pandas nullable type
y = pd.Series(y, dtype="Float32")  # <-- pandas nullable type
plt.plot(list(x), list(y)) # <-- works fine
plt.plot(x, y)             # <-- throws error

Actual outcome

`matplotlib` throws an error
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/series.py in _get_values(self, indexer)
   1025         try:
-> 1026             new_mgr = self._mgr.getitem_mgr(indexer)
   1027             return self._constructor(new_mgr).__finalize__(self)

~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/internals/managers.py in getitem_mgr(self, indexer)
   1635         blk = self._block
-> 1636         array = blk._slice(indexer)
   1637         if array.ndim > 1:

~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/internals/blocks.py in _slice(self, slicer)
   1556 
-> 1557         return self.values[slicer]
   1558 

~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/arrays/masked.py in __getitem__(self, item)
    145 
--> 146         return type(self)(self._data[item], self._mask[item])
    147 

~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/arrays/floating.py in __init__(self, values, mask, copy)
    254             )
--> 255         super().__init__(values, mask, copy=copy)
    256 

~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/arrays/masked.py in __init__(self, values, mask, copy)
    122         if values.ndim != 1:
--> 123             raise ValueError("values must be a 1D array")
    124         if mask.ndim != 1:

ValueError: values must be a 1D array

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
/tmp/ipykernel_409521/2612272364.py in <module>
      7 y = pd.Series(y, dtype="Float32")  # <-- pandas nullable type
      8 plt.plot(list(x), list(y)) # <-- works fine
----> 9 plt.plot(x, y)             # <-- throws error

~/miniconda3/envs/kiwi/lib/python3.9/site-packages/matplotlib/pyplot.py in plot(scalex, scaley, data, *args, **kwargs)
   2755 @_copy_docstring_and_deprecators(Axes.plot)
   2756 def plot(*args, scalex=True, scaley=True, data=None, **kwargs):
-> 2757     return gca().plot(
   2758         *args, scalex=scalex, scaley=scaley,
   2759         **({"data": data} if data is not None else {}), **kwargs)

~/miniconda3/envs/kiwi/lib/python3.9/site-packages/matplotlib/axes/_axes.py in plot(self, scalex, scaley, data, *args, **kwargs)
   1630         """
   1631         kwargs = cbook.normalize_kwargs(kwargs, mlines.Line2D)
-> 1632         lines = [*self._get_lines(*args, data=data, **kwargs)]
   1633         for line in lines:
   1634             self.add_line(line)

~/miniconda3/envs/kiwi/lib/python3.9/site-packages/matplotlib/axes/_base.py in __call__(self, data, *args, **kwargs)
    310                 this += args[0],
    311                 args = args[1:]
--> 312             yield from self._plot_args(this, kwargs)
    313 
    314     def get_next_color(self):

~/miniconda3/envs/kiwi/lib/python3.9/site-packages/matplotlib/axes/_base.py in _plot_args(self, tup, kwargs, return_kwargs)
    485 
    486         if len(xy) == 2:
--> 487             x = _check_1d(xy[0])
    488             y = _check_1d(xy[1])
    489         else:

~/miniconda3/envs/kiwi/lib/python3.9/site-packages/matplotlib/cbook/__init__.py in _check_1d(x)
   1325                     message='Support for multi-dimensional indexing')
   1326 
-> 1327                 ndim = x[:, None].ndim
   1328                 # we have definitely hit a pandas index or series object
   1329                 # cast to a numpy array.

~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/series.py in __getitem__(self, key)
    964             return self._get_values(key)
    965 
--> 966         return self._get_with(key)
    967 
    968     def _get_with(self, key):

~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/series.py in _get_with(self, key)
    979             )
    980         elif isinstance(key, tuple):
--> 981             return self._get_values_tuple(key)
    982 
    983         elif not is_list_like(key):

~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/series.py in _get_values_tuple(self, key)
   1009         # mpl hackaround
   1010         if com.any_none(*key):
-> 1011             result = self._get_values(key)
   1012             deprecate_ndim_indexing(result, stacklevel=5)
   1013             return result

~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/series.py in _get_values(self, indexer)
   1030             #  see tests.series.timeseries.test_mpl_compat_hack
   1031             # the asarray is needed to avoid returning a 2D DatetimeArray
-> 1032             return np.asarray(self._values[indexer])
   1033 
   1034     def _get_value(self, label, takeable: bool = False):

~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/arrays/masked.py in __getitem__(self, item)
    144         item = check_array_indexer(self, item)
    145 
--> 146         return type(self)(self._data[item], self._mask[item])
    147 
    148     @doc(ExtensionArray.fillna)

~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/arrays/floating.py in __init__(self, values, mask, copy)
    253                 "the 'pd.array' function instead"
    254             )
--> 255         super().__init__(values, mask, copy=copy)
    256 
    257     @classmethod

~/miniconda3/envs/kiwi/lib/python3.9/site-packages/pandas/core/arrays/masked.py in __init__(self, values, mask, copy)
    121             )
    122         if values.ndim != 1:
--> 123             raise ValueError("values must be a 1D array")
    124         if mask.ndim != 1:
    125             raise ValueError("mask must be a 1D array")

ValueError: values must be a 1D array

Expected outcome

It should plot the array normally in this case. No missing values are present and at the very least, I would expect matplotlib to try plot(list(x), list(y)) as a fallback mechanism.

On a secondary level, it would be kind of nice if matplotlib could detect and treat pandas.NA as if it were numpy.nan. (e.g., the list approach currently does not work if any missing values are present. But I think this should probably be a seperate issue.)

Operating system Ubuntu 20.04.3 LTS

Matplotlib Version 3.5.1

Activity

  1. jklymak commented on Jan 6, 2022

    @jklymak
    Member

    @tacaswell, pandas 1.0 is 2 years old in a couple of weeks . How far back does newish matplotlib need to support? With my pandas and matplotlib, if I replace cbook._check_1d by np.atleast_1d I think all the tests pass (and it fixes this incompatibility with a newish data type)

  2. jklymak commented on Jan 6, 2022

    @jklymak
    Member

    .. the other practical fix here is to expand the except in _check_1d to include ValueError.

  3. jklymak commented on Jan 7, 2022

    @jklymak
    Member

    I really think this is a pandas bug, with FloatingArray not supporting ndarray-like slicing. See #22141

    Personally I am skeptical of pandas approach here; it seems they have decided they need a NA in addition to NaN, and instead of getting numpy to agree they have made their own Array type. I don't think that Matplotlib should feel obliged to support whatever downstream throws at us. (So far as I can tell, pandas.arrays.FloatingArray is not even documented).

  4. randolf-scholz commented on Jan 7, 2022

    @randolf-scholz
    ContributorAuthor

    @jklymak

    I don't think that Matplotlib should feel obliged to support whatever downstream throws at us.

    I completely agree with this sentiment, yet, do you not agree it would be useful to have a fallback mechanism like I proposed?

    In my example, plot(list(x), list(y)) works. I do not see the hurt in having something like this as a fallback mechanism if everything else fails. E.g. before throwing an error and refusing to plot, try casting the inputs as lists and create the plots from these lists, possibly accompanied with a warning failed to interpret inputs, using fallback mode or something. This would be in line with the Robustness principle, wouldn't it?

  5. jklymak commented on Jan 7, 2022

    @jklymak
    Member

    Well, see the code around #22141 - it is fallback after fallback after fallback. Just converting to a list isn't particularly robust if the list elements are objects we can't recognize. They are so far, but it seems pandas is even considering x.to_numpy() to return numpy object arrays full of their own Float32 instead of numpy float64 arrays.

  6. added this to the v3.5.2 milestone on Jan 10, 2022
  7. tacaswell commented on Jan 10, 2022

    @tacaswell
    Member

    How far back does newish matplotlib need to support?

    Per https://matplotlib.org/stable/devel/min_dep_policy.html 24 months so pandas 1.0 is about to fall out of support for the main branch but not for 3.5.x. I would argue that simplifying work-arounds to pandas issues counts as "we need new features".

    I really think this is a pandas bug,

    I think I agree, I think they are raising a different exception than we are expecting and inconsistent by dtype. Opening a bug with them now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions