Skip to content

Range determination for data with NaNs #6483

Closed

Description

@TrishGillett

I notice that this code will plot with warnings:

from matplotlib import pyplot as plt
from numpy import NaN
plt.hist([1, NaN, 4], range=[0,5])
/Users/trish/anaconda/lib/python2.7/site-packages/numpy/lib/function_base.py:229: RuntimeWarning: invalid value encountered in greater_equal
  keep = (tmp_a >= mn)
/Users/trish/anaconda/lib/python2.7/site-packages/numpy/lib/function_base.py:230: RuntimeWarning: invalid value encountered in less_equal
  keep &= (tmp_a <= mx)

Whereas this code ends up with an error because of how the range is determined:

from matplotlib import pyplot as plt
from numpy import NaN
plt.hist([1, NaN, 4])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/trish/anaconda/lib/python2.7/site-packages/matplotlib/pyplot.py", line 2958, in hist
    stacked=stacked, data=data, **kwargs)
  File "/Users/trish/anaconda/lib/python2.7/site-packages/matplotlib/__init__.py", line 1812, in inner
    return func(ax, *args, **kwargs)
  File "/Users/trish/anaconda/lib/python2.7/site-packages/matplotlib/axes/_axes.py", line 6010, in hist
    m, bins = np.histogram(x[i], bins, weights=w[i], **hist_kwargs)
  File "/Users/trish/anaconda/lib/python2.7/site-packages/numpy/lib/function_base.py", line 176, in histogram
    'max must be larger than min in range parameter.')
AttributeError: max must be larger than min in range parameter.

If we're willing to forgive NaNs in the first case and just plot while ignoring them, wouldn't it also make sense to ignore them in the range computation? Or if that's a bad idea, maybe we could return an error that makes it clear to the user that the problem was NaNs in the data?

The friend who showed this to me was stumped by the error message because his data was huge and he had no idea it contained NaNs. When he saw that AttributeError he just thought "But I'm letting matplotlib decide the range, how can it be wrong?"

Activity

  1. tacaswell commented on May 26, 2016

    @tacaswell
    Member

    Some of these are coming directly out of numpy

    In [3]: np.histogram([1, np.nan, 5])
    /home/tcaswell/.virtualenvs/dd35/lib/python3.5/site-packages/numpy/lib/function_base.py:229: RuntimeWarning: invalid value encountered in greater_equal
      keep = (tmp_a >= mn)
    /home/tcaswell/.virtualenvs/dd35/lib/python3.5/site-packages/numpy/lib/function_base.py:230: RuntimeWarning: invalid value encountered in less_equal
      keep &= (tmp_a <= mx)
    Out[3]: 
    (array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),
     array([ nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan]))

    In other context we mask out invalid values, might be worth doing the same in hist.

  2. added this to the milestone on May 26, 2016
  3. efiring commented on Jul 15, 2016

    @efiring
    Member

    This would be quite easy via a modification of the internal function _normalize_input to always return a list of 1-d ndarrays after application of np.ma.masked_invalid(arr).compressed() to each arr (list entry or column) of the original input. @discardthree, would you like to submit a PR for this? It would be a nice contribution, addressing a longstanding shortcoming of hist().

  4. TrishGillett commented on Jul 16, 2016

    @TrishGillett
    ContributorAuthor

    Sure, I'll give it a try. Thanks for the hint about np.ma.masked_invalid(arr).compressed(). It looks to me like it's just a touch more complicated than that since _normalize_input is applied to the weights as well (if used). The same mask (the one based on x's validity) needs to be applied to both x and weights.

    I'm thinking of something like this:

    x_mask = np.ma.masked_invalid(x).mask
    inp = np.ma.masked_array(inp, mask=x_mask).compressed()

    Am I on the right track?

    Edit to add: I'm not taking 2D arrays into account yet but I'll get there...

  5. efiring commented on Jul 16, 2016

    @efiring
    Member

    Good point!
    Yes, I think you are on the right track.
    I think you can treat a 2D array input as a list of columns by transposing, and then iterating over it just as you iterate over the list of arrays.
    There is a function, matplotlib.cbook.delete_masked_points() that might be usable for the mask handling, given a 1-D array and its weights as inputs, but it's pretty complicated because it's more general than needed here.

  6. TrishGillett commented on Jul 16, 2016

    @TrishGillett
    ContributorAuthor

    Do you think it's important that the masking happen inside _normalize_input? To mask weights we require access to x after any reshaping but before compressed() is applied. I'm thinking of tweaking _normalize_input so that it always returns a list of one or more arrays, and then doing the masking after the shaping finishes like this (last bit mine, rest included for context):

            # Massage 'x' for processing.
            if input_empty:
                x = np.array([[]])
            else:
                x = _normalize_input(x, 'x')
            nx = len(x)  # number of datasets
    
            # Weights need to be shaped appropriately.
            if weights is not None:
                w = _normalize_input(weights, 'weights')
            else:
                w = [None]*nx
    
            if len(w) != nx:
                raise ValueError('weights should have the same shape as x')
    
            # Masking invalid data:
            # Where an element of x[i] is invalid (NaN or inf), it and the
            # corresponding element of weights[i] (if weights used) will be masked.
            for i in range(len(x)):
                xi_mask = np.ma.masked_invalid(x[i]).mask
                x[i] = np.ma.masked_array(x[i], mask=xi_mask).compressed()
                if weights[i]:
                    weights[i] = np.ma.masked_array(weights[i], mask=xi_mask).compressed()
  7. efiring commented on Jul 16, 2016

    @efiring
    Member

    This raises the question of whether weights should support nans/masked values, or at least check for them. If so, it could be handled by modifying _normalize_input to return a sequence of masked arrays by using cbook.safe_masked_invalid in place of asarray. Then the combined mask, if any, could be used for both prior to .compressed(). If not, it will still be necessary to change asarray to asanyarray or an equivalent, and it might be wise to put in a check. Also, regardless, use cbook.safe_masked_invalid unless the array is guaranteed to be floating point.

  8. TrishGillett commented on Jul 17, 2016

    @TrishGillett
    ContributorAuthor

    Okay, this sounds pretty good for the most part. If it's okay with you I'm going to press pause on this for the moment because I'm preparing to defend my thesis, but I'll circle back around to this in a little over a week.

  9. efiring commented on Jul 17, 2016

    @efiring
    Member

    No rush. I hope your defense goes well, as I expect it will.

  10. TrishGillett commented on Sep 17, 2016

    @TrishGillett
    ContributorAuthor

    FYI, I'm alive and starting on this again. (And my defense went okay! 🎉)

  11. tritemio commented on Jun 6, 2017

    @tritemio
  12. tritemio commented on Jun 6, 2017

    @tritemio

    It is worth noting that, in numpy 1.12, NaNs in the input data will cause the error even if the array of bin edges is fully specified. I don't think this was the case with numpy 1.11. Here an example raising the error:

    from matplotlib import pyplot as plt
    from numpy import NaN
    plt.hist([1, NaN, 3], bins=[1, 2, 3])
  13. choldgraf commented on Aug 8, 2017

    @choldgraf
    Contributor

    hey folks - any movement on this? dealing with some student confusion since I have to tell them to drop nans whenever we plot histograms, but that we don't have to do this with other plotting functions...

  14. tritemio commented on Aug 8, 2017

    @tritemio

    @choldgraf, as a workaround you can also specify the range argument.

  15. choldgraf commented on Aug 8, 2017

    @choldgraf
    Contributor

    oh nice! didn't know about that workaround. In that case, if it's going to take some time for this fix to get merged, I think we should add a check for NaNs whenever people call hist that returns a human readable error, e.g. ValueError('NaNs found in input data. To plot histogram with NaNs, specify the range parameter.'). That way in case this PR slips through the cracks it will at least be more clear to users. WDYT?

  16. IreneCrisologo commented on Sep 11, 2017

    @IreneCrisologo

    another workaround is pylab.hist(A[~np.isnan(A)]) from here

  17. modified the milestones: , on Sep 19, 2017
  18. eric-wieser commented on Dec 26, 2017

    @eric-wieser
    Contributor

    As of numpy/numpy#10271 (1.15), there is no longer a problem with NaNs if explicit bin edges are given.

    The error persists if the range must be determined automatically

  19. choldgraf commented on Dec 26, 2017

    @choldgraf
    Contributor

    as in, if you do plt.hist(xyz, bins=[2, 3, 4]) that will work? If so that's a definite improvement

  20. eric-wieser commented on Dec 26, 2017

    @eric-wieser
    Contributor

    Correct, that now works (although still gives warnings about Nan)

  21. tacaswell commented on Mar 31, 2019

    @tacaswell
    Member

    This works as of #11267 which went in for mpl 3.0

  22. removed this from the needs sorting milestone on Mar 31, 2019
  23. added this to the v3.0.0 milestone on Mar 31, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions