You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/trish/anaconda/lib/python2.7/site-packages/matplotlib/pyplot.py", line 2958, in hist
stacked=stacked, data=data, **kwargs)
File "/Users/trish/anaconda/lib/python2.7/site-packages/matplotlib/__init__.py", line 1812, in inner
return func(ax, *args, **kwargs)
File "/Users/trish/anaconda/lib/python2.7/site-packages/matplotlib/axes/_axes.py", line 6010, in hist
m, bins = np.histogram(x[i], bins, weights=w[i], **hist_kwargs)
File "/Users/trish/anaconda/lib/python2.7/site-packages/numpy/lib/function_base.py", line 176, in histogram
'max must be larger than min in range parameter.')
AttributeError: max must be larger than min in range parameter.
If we're willing to forgive NaNs in the first case and just plot while ignoring them, wouldn't it also make sense to ignore them in the range computation? Or if that's a bad idea, maybe we could return an error that makes it clear to the user that the problem was NaNs in the data?
The friend who showed this to me was stumped by the error message because his data was huge and he had no idea it contained NaNs. When he saw that AttributeError he just thought "But I'm letting matplotlib decide the range, how can it be wrong?"
This would be quite easy via a modification of the internal function _normalize_input to always return a list of 1-d ndarrays after application of np.ma.masked_invalid(arr).compressed() to each arr (list entry or column) of the original input. @discardthree, would you like to submit a PR for this? It would be a nice contribution, addressing a longstanding shortcoming of hist().
Sure, I'll give it a try. Thanks for the hint about np.ma.masked_invalid(arr).compressed(). It looks to me like it's just a touch more complicated than that since _normalize_input is applied to the weights as well (if used). The same mask (the one based on x's validity) needs to be applied to both x and weights.
Good point!
Yes, I think you are on the right track.
I think you can treat a 2D array input as a list of columns by transposing, and then iterating over it just as you iterate over the list of arrays.
There is a function, matplotlib.cbook.delete_masked_points() that might be usable for the mask handling, given a 1-D array and its weights as inputs, but it's pretty complicated because it's more general than needed here.
Do you think it's important that the masking happen inside _normalize_input? To mask weights we require access to x after any reshaping but before compressed() is applied. I'm thinking of tweaking _normalize_input so that it always returns a list of one or more arrays, and then doing the masking after the shaping finishes like this (last bit mine, rest included for context):
# Massage 'x' for processing.ifinput_empty:
x=np.array([[]])
else:
x=_normalize_input(x, 'x')
nx=len(x) # number of datasets# Weights need to be shaped appropriately.ifweightsisnotNone:
w=_normalize_input(weights, 'weights')
else:
w= [None]*nxiflen(w) !=nx:
raiseValueError('weights should have the same shape as x')
# Masking invalid data:# Where an element of x[i] is invalid (NaN or inf), it and the# corresponding element of weights[i] (if weights used) will be masked.foriinrange(len(x)):
xi_mask=np.ma.masked_invalid(x[i]).maskx[i] =np.ma.masked_array(x[i], mask=xi_mask).compressed()
ifweights[i]:
weights[i] =np.ma.masked_array(weights[i], mask=xi_mask).compressed()
This raises the question of whether weights should support nans/masked values, or at least check for them. If so, it could be handled by modifying _normalize_input to return a sequence of masked arrays by using cbook.safe_masked_invalid in place of asarray. Then the combined mask, if any, could be used for both prior to .compressed(). If not, it will still be necessary to change asarray to asanyarray or an equivalent, and it might be wise to put in a check. Also, regardless, use cbook.safe_masked_invalid unless the array is guaranteed to be floating point.
Okay, this sounds pretty good for the most part. If it's okay with you I'm going to press pause on this for the moment because I'm preparing to defend my thesis, but I'll circle back around to this in a little over a week.
It is worth noting that, in numpy 1.12, NaNs in the input data will cause the error even if the array of bin edges is fully specified. I don't think this was the case with numpy 1.11. Here an example raising the error:
hey folks - any movement on this? dealing with some student confusion since I have to tell them to drop nans whenever we plot histograms, but that we don't have to do this with other plotting functions...
oh nice! didn't know about that workaround. In that case, if it's going to take some time for this fix to get merged, I think we should add a check for NaNs whenever people call hist that returns a human readable error, e.g. ValueError('NaNs found in input data. To plot histogram with NaNs, specify the range parameter.'). That way in case this PR slips through the cracks it will at least be more clear to users. WDYT?
Reacted by Irene, Antonino Ingargiola, Karl Lapo and jtlz2
Activity
tacaswell commentedon May 26, 2016
Some of these are coming directly out of numpy
In other context we mask out invalid values, might be worth doing the same in
hist.efiring commentedon Jul 15, 2016
This would be quite easy via a modification of the internal function
_normalize_inputto always return a list of 1-d ndarrays after application ofnp.ma.masked_invalid(arr).compressed()to eacharr(list entry or column) of the original input. @discardthree, would you like to submit a PR for this? It would be a nice contribution, addressing a longstanding shortcoming ofhist().TrishGillett commentedon Jul 16, 2016
Sure, I'll give it a try. Thanks for the hint about
np.ma.masked_invalid(arr).compressed(). It looks to me like it's just a touch more complicated than that since_normalize_inputis applied to the weights as well (if used). The same mask (the one based on x's validity) needs to be applied to both x and weights.I'm thinking of something like this:
Am I on the right track?
Edit to add: I'm not taking 2D arrays into account yet but I'll get there...
efiring commentedon Jul 16, 2016
Good point!
Yes, I think you are on the right track.
I think you can treat a 2D array input as a list of columns by transposing, and then iterating over it just as you iterate over the list of arrays.
There is a function,
matplotlib.cbook.delete_masked_points()that might be usable for the mask handling, given a 1-D array and its weights as inputs, but it's pretty complicated because it's more general than needed here.TrishGillett commentedon Jul 16, 2016
Do you think it's important that the masking happen inside
_normalize_input? To maskweightswe require access toxafter any reshaping but before compressed() is applied. I'm thinking of tweaking_normalize_inputso that it always returns a list of one or more arrays, and then doing the masking after the shaping finishes like this (last bit mine, rest included for context):efiring commentedon Jul 16, 2016
This raises the question of whether weights should support nans/masked values, or at least check for them. If so, it could be handled by modifying
_normalize_inputto return a sequence of masked arrays by usingcbook.safe_masked_invalidin place ofasarray. Then the combined mask, if any, could be used for both prior to.compressed(). If not, it will still be necessary to changeasarraytoasanyarrayor an equivalent, and it might be wise to put in a check. Also, regardless, usecbook.safe_masked_invalidunless the array is guaranteed to be floating point.TrishGillett commentedon Jul 17, 2016
Okay, this sounds pretty good for the most part. If it's okay with you I'm going to press pause on this for the moment because I'm preparing to defend my thesis, but I'll circle back around to this in a little over a week.
efiring commentedon Jul 17, 2016
No rush. I hope your defense goes well, as I expect it will.
TrishGillett commentedon Sep 17, 2016
FYI, I'm alive and starting on this again. (And my defense went okay! 🎉)
tritemio commentedon Jun 6, 2017
xref numpy/numpy#7503
tritemio commentedon Jun 6, 2017
It is worth noting that, in numpy 1.12, NaNs in the input data will cause the error even if the array of bin edges is fully specified. I don't think this was the case with numpy 1.11. Here an example raising the error:
choldgraf commentedon Aug 8, 2017
hey folks - any movement on this? dealing with some student confusion since I have to tell them to drop
nans whenever we plot histograms, but that we don't have to do this with other plotting functions...tritemio commentedon Aug 8, 2017
@choldgraf, as a workaround you can also specify the range argument.
choldgraf commentedon Aug 8, 2017
oh nice! didn't know about that workaround. In that case, if it's going to take some time for this fix to get merged, I think we should add a check for
NaNs whenever people callhistthat returns a human readable error, e.g.ValueError('NaNs found in input data. To plot histogram with NaNs, specify the range parameter.'). That way in case this PR slips through the cracks it will at least be more clear to users. WDYT?IreneCrisologo commentedon Sep 11, 2017
another workaround is
pylab.hist(A[~np.isnan(A)])from hereeric-wieser commentedon Dec 26, 2017
As of numpy/numpy#10271 (1.15), there is no longer a problem with NaNs if explicit bin edges are given.
The error persists if the range must be determined automatically
choldgraf commentedon Dec 26, 2017
as in, if you do
plt.hist(xyz, bins=[2, 3, 4])that will work? If so that's a definite improvementeric-wieser commentedon Dec 26, 2017
Correct, that now works (although still gives warnings about Nan)
tacaswell commentedon Mar 31, 2019
This works as of #11267 which went in for mpl 3.0
Implemented dropna bug fix for plt.hist