Missing data handling with categorical units

Bug report

Bug summary

Handing of missing data (represented by np.nan) is inconsistent and surprising.

Code for reproduction

import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
print(mpl.__version__)
x, y = ["a", np.nan, "c"], [1, 2, 3]

plt.figure()
plt.plot(x, y)  # runs

plt.figure()
plt.bar(x, y)  # raises

plt.figure()
plt.bar(np.array(x), y)  # runs

Actual outcome

The first issue is that plt.bar (and plt.scatter, probably others, but notably not plt.plot) raise when they get a list containing np.nan (but not an array containing np.nan?):

Details

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-119-77b75fdcb3b5> in <module>
      9 
     10 plt.figure()
---> 11 plt.bar(x, y)
     12 
     13 plt.figure()

~/miniconda3/envs/seaborn-py38-latest/lib/python3.8/site-packages/matplotlib/pyplot.py in bar(x, height, width, bottom, align, data, **kwargs)
   2485         x, height, width=0.8, bottom=None, *, align='center',
   2486         data=None, **kwargs):
-> 2487     return gca().bar(
   2488         x, height, width=width, bottom=bottom, align=align,
   2489         **({"data": data} if data is not None else {}), **kwargs)

~/miniconda3/envs/seaborn-py38-latest/lib/python3.8/site-packages/matplotlib/__init__.py in inner(ax, data, *args, **kwargs)
   1436     def inner(ax, *args, data=None, **kwargs):
   1437         if data is None:
-> 1438             return func(ax, *map(sanitize_sequence, args), **kwargs)
   1439 
   1440         bound = new_sig.bind(ax, *args, **kwargs)

~/miniconda3/envs/seaborn-py38-latest/lib/python3.8/site-packages/matplotlib/axes/_axes.py in bar(self, x, height, width, bottom, align, **kwargs)
   2405 
   2406         if orientation == 'vertical':
-> 2407             self._process_unit_info(xdata=x, ydata=height, kwargs=kwargs)
   2408             if log:
   2409                 self.set_yscale('log', nonpositive='clip')

~/miniconda3/envs/seaborn-py38-latest/lib/python3.8/site-packages/matplotlib/axes/_base.py in _process_unit_info(self, xdata, ydata, kwargs)
   2187             return kwargs
   2188 
-> 2189         kwargs = _process_single_axis(xdata, self.xaxis, 'xunits', kwargs)
   2190         kwargs = _process_single_axis(ydata, self.yaxis, 'yunits', kwargs)
   2191         return kwargs

~/miniconda3/envs/seaborn-py38-latest/lib/python3.8/site-packages/matplotlib/axes/_base.py in _process_single_axis(data, axis, unit_name, kwargs)
   2170                 # We only need to update if there is nothing set yet.
   2171                 if not axis.have_units():
-> 2172                     axis.update_units(data)
   2173 
   2174             # Check for units in the kwargs, and if present update axis

~/miniconda3/envs/seaborn-py38-latest/lib/python3.8/site-packages/matplotlib/axis.py in update_units(self, data)
   1461         neednew = self.converter != converter
   1462         self.converter = converter
-> 1463         default = self.converter.default_units(data, self)
   1464         if default is not None and self.units is None:
   1465             self.set_units(default)

~/miniconda3/envs/seaborn-py38-latest/lib/python3.8/site-packages/matplotlib/category.py in default_units(data, axis)
    105         # the conversion call stack is default_units -> axis_info -> convert
    106         if axis.units is None:
--> 107             axis.set_units(UnitData(data))
    108         else:
    109             axis.units.update(data)

~/miniconda3/envs/seaborn-py38-latest/lib/python3.8/site-packages/matplotlib/category.py in __init__(self, data)
    174         self._counter = itertools.count()
    175         if data is not None:
--> 176             self.update(data)
    177 
    178     @staticmethod

~/miniconda3/envs/seaborn-py38-latest/lib/python3.8/site-packages/matplotlib/category.py in update(self, data)
    209         for val in OrderedDict.fromkeys(data):
    210             # OrderedDict just iterates over unique values in data.
--> 211             cbook._check_isinstance((str, bytes), value=val)
    212             if convertible:
    213                 # this will only be called so long as convertible is True.

~/miniconda3/envs/seaborn-py38-latest/lib/python3.8/site-packages/matplotlib/cbook/__init__.py in _check_isinstance(_types, **kwargs)
   2244     for k, v in kwargs.items():
   2245         if not isinstance(v, types):
-> 2246             raise TypeError(
   2247                 "{!r} must be an instance of {}, not a {}".format(
   2248                     k,

TypeError: 'value' must be an instance of str or bytes, not a float

~/miniconda3/envs/seaborn-py38-latest/lib/python3.8/site-packages/matplotlib/cbook/__init__.py in _check_isinstance(_types, **kwargs)
   2244     for k, v in kwargs.items():
   2245         if not isinstance(v, types):
-> 2246             raise TypeError(
   2247                 "{!r} must be an instance of {}, not a {}".format(
   2248                     k,

TypeError: 'value' must be an instance of str or bytes, not a float

The second issue is that when the code does run, it includes "nan" as a category in the plot, whereas (consistent with the rest of matplotlib) I would have expected the nans to propagate through and not appear in the plot:

Expected outcome

Consistent behavior across input objects (seems straightforward)
Treating nans in categorical datasets as missing, rather than as a different category (possibly controversial?)

Matplotlib version

Operating system: macos
Matplotlib version: 3.3.2
Matplotlib backend (print(matplotlib.get_backend())): pylab inline
Python version: 3.8

The "a" makes that a categorical plot, and everything after that, including numbers, is coerced to string. I'm not sure we should change that, though it has been discussed.

Right, I mean I get that. But in the context of categorical data analysis, nan is pretty strongly associated with "missing", so it's surprising to have it show up as a separate category.

Sure if there is a standard I guess we should try and respect it.

Also should have mentioned that plt.bar and plt.scatter and others also raise when they get a pandas.Series that contain strings and nans.

(All matplotlib functions raise when they get the new pd.NA but that's probably a different conversation...)

Having tried to special case for nans and the ilk before, the fix is probably to change if val not in self._mapping: to if val not in self._mapping and not in missing or explicitly if val in missing: self._mapping[val] = np.nan
but would have to decide what falls under the missing category.

code is here for anyone who want to tackle it https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/category.py#L218

 for val in OrderedDict.fromkeys(data):
            # OrderedDict just iterates over unique values in data.
            cbook._check_isinstance((str, bytes), value=val)
            if convertible:
                # this will only be called so long as convertible is True.
                convertible = self._str_is_convertible(val)
            if val not in self._mapping:
                self._mapping[val] = next(self._counter)

OK, so the root of the problem is that np.asarray(['a', np.NaN, 'b']) returns ['a', 'nan', 'b'].

We very often (i.e. in plot, but apparently not bar) coerce to a numpy array before passing to the units machinery. So the units machinery never sees the np.NaN, but rather three strings and do not raise an error. For bar, it raises an error, because this step is missed, and the book._check_isinstance raises when the np.NaN is hit.

I'm not sure how Matplotlib should handle this:

special case the "nan" string. I'm personally not a fan of this, since "Nan" was my Aunt's name, and not that I was planning to make a plot using her name, but just to say it is a valid string that does not unambiguosly mean "no data".
delaying np.asarray to after unit conversion, or making that part of unit conversion. This should be all re-engineered because unit conversion has a lot of inconsistencies. But that is a long-term project.
at least make bar consistent by making it coerce to an array.

I'm not sure what else we can do. At the lowest level we plot numpy arrays of floats. The unit machinery is kludged on top of that, and if someone is going to mix data types in an input array, I think its hard to guarantee consistency.

oh, so in the units machinary this is the conversion to preserve the types:

data = np.atleast_1d(np.array(data, dtype=object))

but might be too computationally expensive to use everywhere. I think the last time this discussion came up, the consensus was to expect the user to drop missing data before plotting it as a category, but I'm sympathetic to that not being feasible.

_process_unit_info should probably just always do the right thing and always be called before any manipulation to the incoming object.

Yes you definitely don't want to use object typed arrays everywhere, but once you know you're working with categorical data you probably do. I don't think you want to use numpy string dtypes for categorical data...

(It's much cheaper to use a pandas categorical dtype. Obviously that's not possible in base matplotlib, but I want to make more use of categorical units in seaborn and it's possible I'll end up writing a subclass that does that).

I came to this thread looking for a related issue, but having read this thread I thought I would comment that: There is a valid case for having nan be listed as its own category (it feels like the most natural behavior to me).

Say you have a survey where you ask for participants' voting preference, candidate A, candidate B, and some skip the question (refuse to answer): It's highly informative to have a bar showing the count of missing answers. If you don't like it, you can drop the nans before making the plot, whereas having to do something like dropna=False to keep them in feels contrived. And without the nans any percentage labels would be off too.

IMO it's easier to fill nans that represent valid observations than to just "drop the nans before making the plot".

e.g.

df.fillna({"response": "missing"})

The alternative workflow is cumbersome, and varies across two situations:

One: You have a dataframe with many columns, only a couple of which will participate in this plot. If you do df.dropna() it will remove rows that have a null anywhere, which may drop observations that should otherwise appear in the plot. So you need to do something awkward like

plt.scatter(data=df[df[["x", "y"]].notna().all(axis=1), x="x", y="y")

Two: you have separate vectors, meaning that you need to do something like

use = x.notna() & y.notna()
plt.scatter(x=x[use], y=y[use])

Beyond usability concerns, in your example

Say you have a survey where you ask for participants' voting preference, candidate A, candidate B, and some skip the question (refuse to answer): It's highly informative to have a bar showing the count of missing answers.

I agree that it can be informative to show the count of missing answers but I wonder whether using a "nan" label for them is informative in most situations? If you're going to present that plot to anyone, you will probably need to take the extra step of changing changing the label into something interpretable anyway. So you might as well do that upstream.

If you load a spreadsheet as a pandas dataframe with df = read_data(filepath), the empty cells automatically get filled with NaN, so it feels natural to me (I guess I see NaN as a synonym for missing). NaNs appear for many reasons, including errors and bugs. I don't think it's too useful to brush them under the carpet, as it were. And removing NaNs selectively is not a big problem, in my opinion, since it's a natural step in data manipulation. If a NaN category unexpectedly appears in your plot, and you don't like it, you deal with it, whereas if the NaNs are automagically thrown away, the plot won't tell you the story behind. Anyways, it's a subjective issue.

Edit: For a simple histplot and a dataframe, which is what I had in mind coming here, you can just df[col].dropna().histplot().

NaNs appear for many reasons, including errors and bugs.

I agree, which is why I think it is better for the analysis logic to convert them into a different representation when they are intended to be meaningful, like "respondent declined to answer".

And removing NaNs selectively is not a big problem, in my opinion, since it's a natural step in data manipulation.

But that doesn't seem responsive to the point that you may not want to drop all rows that have a nan from the canonical representation of your dataframe. Which means that if you are responsible for dropping nans in analysis logic, it needs to be done in an ad hoc way for each plot / model, and that can be pretty awkward (particularly when using more than one variable) as demonstrated by my code snippets.

This issue has been marked "inactive" because it has been 365 days since the last comment. If this issue is still present in recent Matplotlib releases, or the feature request is still wanted, please leave a comment and this label will be removed. If there are no updates in another 30 days, this issue will be automatically closed, but you are free to re-open or create a new issue if needed. We value issue reports, and this procedure is meant to help us resurface and prioritize issues that have not been addressed yet, not make them disappear. Thanks for your help!

This still raises.

Would converting categoricals to use the new numpy stringdtype and matching their missing data semantics work? That would be consistent with how we handle numerical arrays and their missing values and therefore push categoricals more inline w/ other behavior.

Navigation Menu

Repository navigation

Missing data handling with categorical units #19139

Bug report

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Missing data handling with categorical units #19139

Description

Bug report

Activity

jklymak commented on Dec 17, 2020

mwaskom commented on Dec 17, 2020

jklymak commented on Dec 17, 2020

mwaskom commented on Dec 17, 2020

story645 commented on Dec 17, 2020

jklymak commented on Dec 17, 2020

story645 commented on Dec 17, 2020

jklymak commented on Dec 17, 2020

mwaskom commented on Dec 17, 2020

ptoche commented on Dec 28, 2021

mwaskom commented on Dec 28, 2021

ptoche commented on Dec 28, 2021

mwaskom commented on Dec 28, 2021

github-actions commented on Sep 22, 2023

mwaskom commented on Sep 22, 2023

story645 commented on May 24, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions