You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
plt.hist density argument does not function as described. #10398
The histogram works as expected. The density argument is explained in the documentation
density : boolean, optional
If True, the first element of the return tuple will be the counts normalized to form a probability density, i.e., the area (or integral) under the histogram will sum to 1. [...]
Hence if you calculate the integral,
print (np.sum(n*np.diff(bins)))
the result is indeed 1.0, as expected.
You might want to search stackoverflow.com on questions where people got confused by the density or normed keyword argument.
Reacted by Thomas A Caswell, Delweng, tianlongfei, Axel Jacobsen, Jian Guo and Seyoung Park
I guess I don’t understand why np.diff(bins) is ever relevant for the summation, “the counts normalized to form a pdf, ie, the area under the histogram will sum to one” literally means “[the sum of] the first element of the tupple will sum to 1” as the number of bins in a histogram is more like a resolution for the integral than it is an xvalue.
I don’t understand how to generate the desired functionality. Maybe there should be an option for this if it’s such a common misunderstanding.
The area of each bar is (height * width), “the counts normalized to form a pdf, ie, the area (or integral) under the histogram will sum to one” translate to np.sum(n*np.diff(bins)) == 1 (not np.sum(n) == 1).
We changed the kwarg from 'normed', which could reasonably be either one, to 'density' to avoid this ambiguity.
If you want your bars to sum to one (independent of the bin widths) then pass in weights=np.ones(len(data)) / len(data) and density=False.
Reacted by Elan Ernest, Vincent Déhaye, Delweng, Zichao Yan, Akshay Maheshwari, wxiaoning, Rion Brattig Correia, Yang Wang , Qi Wang, Erik Poppleton and 24 more
The numpy.histogram documentation is a bit more verbose on this part:
density : bool, optional
If False, the result will contain the number of samples in each bin. If True, the result is the value of the probability density function at the bin, normalized such that the integral over the range is 1. Note that the sum of the histogram values will not be equal to 1 unless bins of unity width are chosen; it is not a probability mass function.
It also directly provides an example case which makes this case clear:
Maybe the matplotlib documentation can link to the numpy docs or take this over?
The comment above
If you want your bars to sum to one (independent of the bin widths) then pass in weights=np.ones(len(data)) / len(data) and density=False.
is also a good one which could be added to the docs.
Alternatively, in order not to blow up the docstrings, what about a dedicated histogram example? It seems there is currently only the pyplot-text example around showing how to use the hist function?
@ryanpeach"a pdf doesn’t include the xvalue in integrating to find the normalization constant" is not correct; any integral takes the independent variable into account, i.e. in ∫ f(x) dx you have dx, which is exactly the bin width.
Ok, so I just learned the distinction between a pdf and a pmf. Why not just add normed as a flag and make it the pmf? This is a very common feature, and shouldn’t require hacks to perform.
I guess it will be hard to add another functionality to an argument which has been there for years, even though it being deprecated by now. So instead of normed one could surely think of another name.
However, one problem would be that the histogram can take unequal bin sizes; in this case normalizing to the sum of the counts is rather dangerous. Any such plot would essentially need a label "Attention, this plot has been normalized to the sum of the frequencies independent of the bin width" or so. Hence I'm not sure if adding this normalization will not lead to even more confusion.
At the end why not stick to the above suggestion of
For a PMF perhaps you should use a different calculation and plotting method entirely: instead of bins, find the counts of values for all unique (discrete) values in the array, and plot each count at each discrete value, either with a marker or with a vertical line up from the origin.
Eric, that’s a good idea, except for when you are dealing with semi-continuous things like integers, or continuous values that you still want divided into bins. Like, if you have 100 samples of a continuous distribution, your distribution will not look continuous without bins.
Because I can’t hardly think of a time when a pdf would be useful in the context of a histogram, whereas pmfs are very natural use cases for them, yet we have a flag for the pdf and not one for the pmf.
On top of that, weighting the values individually seems unintuitive, though I can see why it’s correct. It just seems like something the average user shouldn’t have to hack around. I’ll add it if that’s what it takes.
This conversation has gotten more contentious than necessary.
Make those two operations mutually exclusive (illegal to use unequal bin sizes during pmf)
This is pushing on a tension we have through out Matplotlib between methods with simple signatures that do one thing (and try to do it well) and methods with very complex signatures (and internals) that do many different things. Having kwargs that are mutually exclusive or have complex conditional dependencies between them quickly becomes hard to document and maintain (we already have things like this so this is not a theoretical concern). The signature and implementation of hist are already pretty complex, I am extremely wary of adding yet more.
Because I can’t hardly think of a time when a pdf would be useful in the context of a histogram, whereas pmfs are very natural use cases for them, yet we have a flag for the pdf and not one for the pmf.
On top of that, weighting the values individually seems unintuitive, though I can see why it’s correct. It just seems like something the average user shouldn’t have to hack around.
Matplotlib has a wide enough user base that we have to be careful about generalization and appeals to 'intuitive'. From my background (physics), the density normalization seems 'natural' and normalizing by mass seems like a corner case where I am fine using the 'weights' method (but definitely get that my experience is not universal).
I’ll add it if that’s what it takes.
As I said above, I am wary of adding complexity to hist. If you do want to add a mass kwarg, how will it interact with density, normed (which is still there, we did not set a deprecation version), and 'stacked' histograms?
It may also be worth taking this question to numpy (a year or so ago, we had a request for optimal bin estimation in hist which got pushed to numpy and now we have bins='auto' on np.histogram). Currently (I think) in the case of non-stacked histograms we just forward all of the computation down to numpy.
The issue about the pdf and the pmf is slightly pedantic anyway because there is no such thing as a continuous function in numpy/matplotlib. We are always dealing with discrete floating point numbers, and thus a distribution either with bins=len(np.unique(x)) (stem) or bins<len(np.unique(x)) (bar). Then there is the independent option of density with or without scale invariance. Am I right?
No, this is not a matter of being pedantic. There is a big difference between looking at frequencies of discrete events (e.g., counts of Fords, Hondas, etc. going through a tollboth in a day) and quantities that are binned such that there are several or many different values being grouped together in each of several bins. The fact that there is a finite number of floating point numbers that can be represented in a computer with a single or double-precision float is irrelevant.
There is a big difference between looking at frequencies of discrete events (e.g., counts of Fords, Hondas, etc. going through a tollboth in a day) and quantities that are binned such that there are several or many different values being grouped together in each of several bins.
But these aren't mutually exclusive. For instance, my example and @ImportanceOfBeingErnest 's example are binned, continuous, and thus demand a bar plot. However, they do not require normalization via the absolute area under the curve, rather by the normalized xaxis area under the curve. A stem chart won't work for them, because they aren't a selection of a few discrete things, yet the density function won't work for them either, because it cares about the absolute area under the curve. Is that a pdf or a pmf? You tell me.
I think this inevitably comes down, on a technical level, to an argument over limiting choice within a function that was designed for convienience. All of this could be done by hand, plt.hist could be removed and let everyone just use bar plots or stem plots with np.hist till the end of time. But it's obvious that someone thought including plt.hist was important for convenience, but didn't take into account all the popular ways of using it. Now we are stuck looking at wikipedia definitions to mathematical terms that have at best a passing relation to the data visualization tequnique of a histogram, and the normalization constants commonly associated with one, and are arguing over how a user "should" do normalization, how they "should" visualize probabilities and frequencies, how they "should" bin things. Call those normalization options whatever you want, make pretty and "cannonical" defaults, but not having them because they don't fit a mold (playing favorites) even though plenty of people use them every day is what is pedantic.
There is a good principle when it comes to plotting: Keep data aggregation and manipulation separate from data visualization.
Matplotlib provides functions to draw data as lines, points, shapes, bars etc. Once you have aggregated your data, you can decide on which kind of plot works best to visualize this data. This completely lies in the user's responsibility. If the user wants to draw a pmf of his data as a bar plot, he can do so.
So following this principle there should not even be a hist function in the first place - essentially it only saves some typing and is hence convenient to use. Once you start adding all possible use cases of this function as individual arguments you will loose much of the convenience. It may look as if this function is easier to use if some argument does exactly what you want, but you'll soon end up spending more time understanding all the arguments and their implications on the drawn result than you would have spend on aggregating the data in the way you need it and simply calling the line/bar/stem/scatter function.
In view of this, I am currently not sure if there is actually anything matplotlib should do, other than the two points mentioned above: (1) improve documentation (2) decide upon adding an additional frequency normalization argument. @ryanpeach Despite all the comments added from your side, we are still at exactly the point we were yesterday. Be assured that people did understand that there might be a need to do (1) and (2) and they will decide on that at some point in the future.
Ok, just let me know. I just hope you don't choose to ignore (2) while maintaining the density kwarg, that's just biased. I'll let the thread linger till you all make a decision but still offer my services in making the pull request.
The two definitions are the same if you have equally spaced bins (which is the most usual situation). I’d agree that if numpy accepts mass we can revisit this and follow their lead. @ryanpeach if you want to continue discussion, you should chime in on the numpy issue linked above, or of course another dev is free to re-open this if they disagree...
A more or less recent decision was to make plt.hist a thinner (plotting) wrapper around np.histogram than it used to be (because the first one predated the latter IIRC), was it not? So I would suggest that if at some point we want to provide a way to plot PMF-like objects through plt.hist, then it may be a reasonable idea to work with Numpy devs to see if it can make it in np.histogram.
FWIW, after a bit of googling, it seems like some (and not most of those that I checked¹) plotting softwares offer the feature that is discussed here through a keyword argument like “norm”/“density”/etc. that can take string inputs among “pdf”, “cdf”, “counts”, “probability” (usually what is discussed here), “sf”, etc.
I don't think it's numpy's job to do this because normalization is easy to do mathematically post-hoc. It's just the graphing that's hard. It's matplotlibs wrapper that makes it difficult to do intermediate steps on the binning like normalization. But I digress.
So I came here because I have a histogram with log=True and density=True and it is giving me weird results. Note that I have logarithmic spaced bins
Note how on some histograms the bins, for which supposedly there are no counts, still have some height to them (is appearing above 0). Ideally this is not what I was looking for. I feel this discussion is related to this problem. And I am unsure if this is a bug or intended.
@nunocalaim Thanks for reaching out. However I do not think your issue is in any way related. Would you mind opening a new issue and include a self-contained example (i.e. a code one can copy, paste and run) to allow people to reproduce your problem? This is the only way one can find out if the outcome is expected or whether there is a problem in the matplotlib code base.
Note that we now have stair which allows folks to normalize whatever way is most appropriate for their field. #18275
I'm pretty concerned with Matplotlib mediating multiple normalization conventions among fields. But if someone can come up with a clean way to accommodate everyone, or write their own wrapper, I imagine we could consider that.
Activity
ImportanceOfBeingErnest commentedon Feb 8, 2018
The histogram works as expected. The density argument is explained in the documentation
Hence if you calculate the integral,
the result is indeed 1.0, as expected.
You might want to search stackoverflow.com on questions where people got confused by the
densityornormedkeyword argument.tacaswell commentedon Feb 8, 2018
@ryanpeach Which documentation did you find misleading?
ryanpeach commentedon Feb 15, 2018
I guess I don’t understand why np.diff(bins) is ever relevant for the summation, “the counts normalized to form a pdf, ie, the area under the histogram will sum to one” literally means “[the sum of] the first element of the tupple will sum to 1” as the number of bins in a histogram is more like a resolution for the integral than it is an xvalue.
I don’t understand how to generate the desired functionality. Maybe there should be an option for this if it’s such a common misunderstanding.
ryanpeach commentedon Feb 15, 2018
I guess what I’m saying is, a pdf doesn’t include the xvalue in integrating to find the normalization constant. It’s the y value that must sum to 1.
tacaswell commentedon Feb 15, 2018
The area of each bar is (height * width), “the counts normalized to form a pdf, ie, the area (or integral) under the histogram will sum to one” translate to
np.sum(n*np.diff(bins)) == 1(notnp.sum(n) == 1).We changed the kwarg from 'normed', which could reasonably be either one, to 'density' to avoid this ambiguity.
If you want your bars to sum to one (independent of the bin widths) then pass in
weights=np.ones(len(data)) / len(data)anddensity=False.tacaswell commentedon Feb 15, 2018
@ryanpeach How would you re-write that docstring to be clearer to you?
ImportanceOfBeingErnest commentedon Feb 15, 2018
The numpy.histogram documentation is a bit more verbose on this part:
It also directly provides an example case which makes this case clear:
Maybe the matplotlib documentation can link to the numpy docs or take this over?
The comment above
is also a good one which could be added to the docs.
Alternatively, in order not to blow up the docstrings, what about a dedicated histogram example? It seems there is currently only the pyplot-text example around showing how to use the
histfunction?@ryanpeach "a pdf doesn’t include the xvalue in integrating to find the normalization constant" is not correct; any integral takes the independent variable into account, i.e. in
∫ f(x) dxyou havedx, which is exactly the bin width.ryanpeach commentedon Feb 17, 2018
Ok, so I just learned the distinction between a pdf and a pmf. Why not just add normed as a flag and make it the pmf? This is a very common feature, and shouldn’t require hacks to perform.
diff = np.diff(bin_edge) if not normed else 1.ImportanceOfBeingErnest commentedon Feb 17, 2018
I guess it will be hard to add another functionality to an argument which has been there for years, even though it being deprecated by now. So instead of
normedone could surely think of another name.However, one problem would be that the histogram can take unequal bin sizes; in this case normalizing to the sum of the counts is rather dangerous. Any such plot would essentially need a label "Attention, this plot has been normalized to the sum of the frequencies independent of the bin width" or so. Hence I'm not sure if adding this normalization will not lead to even more confusion.
At the end why not stick to the above suggestion of
efiring commentedon Feb 18, 2018
For a PMF perhaps you should use a different calculation and plotting method entirely: instead of bins, find the counts of values for all unique (discrete) values in the array, and plot each count at each discrete value, either with a marker or with a vertical line up from the origin.
ryanpeach commentedon Feb 18, 2018
ryanpeach commentedon Feb 18, 2018
Make those two operations mutually exclusive (illegal to use unequal bin sizes during pmf)
Call it “mass” for consistency.
ryanpeach commentedon Feb 18, 2018
Because I can’t hardly think of a time when a pdf would be useful in the context of a histogram, whereas pmfs are very natural use cases for them, yet we have a flag for the pdf and not one for the pmf.
On top of that, weighting the values individually seems unintuitive, though I can see why it’s correct. It just seems like something the average user shouldn’t have to hack around. I’ll add it if that’s what it takes.
tacaswell commentedon Feb 18, 2018
This conversation has gotten more contentious than necessary.
This is pushing on a tension we have through out Matplotlib between methods with simple signatures that do one thing (and try to do it well) and methods with very complex signatures (and internals) that do many different things. Having kwargs that are mutually exclusive or have complex conditional dependencies between them quickly becomes hard to document and maintain (we already have things like this so this is not a theoretical concern). The signature and implementation of
histare already pretty complex, I am extremely wary of adding yet more.Matplotlib has a wide enough user base that we have to be careful about generalization and appeals to 'intuitive'. From my background (physics), the density normalization seems 'natural' and normalizing by mass seems like a corner case where I am fine using the 'weights' method (but definitely get that my experience is not universal).
As I said above, I am wary of adding complexity to
hist. If you do want to add amasskwarg, how will it interact withdensity,normed(which is still there, we did not set a deprecation version), and 'stacked' histograms?It may also be worth taking this question to numpy (a year or so ago, we had a request for optimal bin estimation in
histwhich got pushed to numpy and now we havebins='auto'onnp.histogram). Currently (I think) in the case of non-stacked histograms we just forward all of the computation down to numpy.@ImportanceOfBeingErnest
How is this different than passing in the weights?
17 remaining items
ryanpeach commentedon Feb 20, 2018
The issue about the pdf and the pmf is slightly pedantic anyway because there is no such thing as a continuous function in numpy/matplotlib. We are always dealing with discrete floating point numbers, and thus a distribution either with
bins=len(np.unique(x))(stem) orbins<len(np.unique(x))(bar). Then there is the independent option of density with or without scale invariance. Am I right?efiring commentedon Feb 20, 2018
No, this is not a matter of being pedantic. There is a big difference between looking at frequencies of discrete events (e.g., counts of Fords, Hondas, etc. going through a tollboth in a day) and quantities that are binned such that there are several or many different values being grouped together in each of several bins. The fact that there is a finite number of floating point numbers that can be represented in a computer with a single or double-precision float is irrelevant.
ryanpeach commentedon Feb 20, 2018
But these aren't mutually exclusive. For instance, my example and @ImportanceOfBeingErnest 's example are binned, continuous, and thus demand a bar plot. However, they do not require normalization via the absolute area under the curve, rather by the normalized xaxis area under the curve. A stem chart won't work for them, because they aren't a selection of a few discrete things, yet the density function won't work for them either, because it cares about the absolute area under the curve. Is that a pdf or a pmf? You tell me.
ryanpeach commentedon Feb 20, 2018
I think this inevitably comes down, on a technical level, to an argument over limiting choice within a function that was designed for convienience. All of this could be done by hand, plt.hist could be removed and let everyone just use bar plots or stem plots with np.hist till the end of time. But it's obvious that someone thought including plt.hist was important for convenience, but didn't take into account all the popular ways of using it. Now we are stuck looking at wikipedia definitions to mathematical terms that have at best a passing relation to the data visualization tequnique of a histogram, and the normalization constants commonly associated with one, and are arguing over how a user "should" do normalization, how they "should" visualize probabilities and frequencies, how they "should" bin things. Call those normalization options whatever you want, make pretty and "cannonical" defaults, but not having them because they don't fit a mold (playing favorites) even though plenty of people use them every day is what is pedantic.
ImportanceOfBeingErnest commentedon Feb 20, 2018
There is a good principle when it comes to plotting: Keep data aggregation and manipulation separate from data visualization.
Matplotlib provides functions to draw data as lines, points, shapes, bars etc. Once you have aggregated your data, you can decide on which kind of plot works best to visualize this data. This completely lies in the user's responsibility. If the user wants to draw a pmf of his data as a bar plot, he can do so.
So following this principle there should not even be a
histfunction in the first place - essentially it only saves some typing and is hence convenient to use. Once you start adding all possible use cases of this function as individual arguments you will loose much of the convenience. It may look as if this function is easier to use if some argument does exactly what you want, but you'll soon end up spending more time understanding all the arguments and their implications on the drawn result than you would have spend on aggregating the data in the way you need it and simply calling the line/bar/stem/scatter function.In view of this, I am currently not sure if there is actually anything matplotlib should do, other than the two points mentioned above: (1) improve documentation (2) decide upon adding an additional frequency normalization argument.
@ryanpeach Despite all the comments added from your side, we are still at exactly the point we were yesterday. Be assured that people did understand that there might be a need to do (1) and (2) and they will decide on that at some point in the future.
ryanpeach commentedon Feb 20, 2018
Ok, just let me know. I just hope you don't choose to ignore (2) while maintaining the density kwarg, that's just biased. I'll let the thread linger till you all make a decision but still offer my services in making the pull request.
afvincent commentedon Feb 20, 2018
FWIW, Numpy has a similar issue thread: numpy/numpy#9921 (but it got significantly less traction, to say the less).
anntzer commentedon Feb 20, 2018
I strongly believe
massshould not be added (unless numpy adds it, which I would deem a mistake but would begrundingly accept following).jklymak commentedon Feb 20, 2018
The two definitions are the same if you have equally spaced bins (which is the most usual situation). I’d agree that if numpy accepts mass we can revisit this and follow their lead. @ryanpeach if you want to continue discussion, you should chime in on the numpy issue linked above, or of course another dev is free to re-open this if they disagree...
afvincent commentedon Feb 21, 2018
A more or less recent decision was to make
plt.hista thinner (plotting) wrapper aroundnp.histogramthan it used to be (because the first one predated the latter IIRC), was it not? So I would suggest that if at some point we want to provide a way to plot PMF-like objects throughplt.hist, then it may be a reasonable idea to work with Numpy devs to see if it can make it innp.histogram.FWIW, after a bit of googling, it seems like some (and not most of those that I checked¹) plotting softwares offer the feature that is discussed here through a keyword argument like “norm”/“density”/etc. that can take string inputs among “pdf”, “cdf”, “counts”, “probability” (usually what is discussed here), “sf”, etc.
https://www.mathworks.com/help/matlab/ref/histogram.html
http://reference.wolfram.com/language/ref/Histogram.html
https://www.rdocumentation.org/packages/graphics/versions/3.4.3/topics/hist
https://www.originlab.com/doc/Origin-Help/Create-Histogram
Edit: burnt ^^.
ryanpeach commentedon Feb 21, 2018
Just posted on the numpy thread.
I don't think it's numpy's job to do this because normalization is easy to do mathematically post-hoc. It's just the graphing that's hard. It's matplotlibs wrapper that makes it difficult to do intermediate steps on the binning like normalization. But I digress.
nunocalaim commentedon Aug 29, 2018
So I came here because I have a histogram with log=True and density=True and it is giving me weird results. Note that I have logarithmic spaced bins

Note how on some histograms the bins, for which supposedly there are no counts, still have some height to them (is appearing above 0). Ideally this is not what I was looking for. I feel this discussion is related to this problem. And I am unsure if this is a bug or intended.
ImportanceOfBeingErnest commentedon Aug 29, 2018
@nunocalaim Thanks for reaching out. However I do not think your issue is in any way related. Would you mind opening a new issue and include a self-contained example (i.e. a code one can copy, paste and run) to allow people to reproduce your problem? This is the only way one can find out if the outcome is expected or whether there is a problem in the matplotlib code base.
jklymak commentedon Nov 24, 2020
Note that we now have
stairwhich allows folks to normalize whatever way is most appropriate for their field. #18275I'm pretty concerned with Matplotlib mediating multiple normalization conventions among fields. But if someone can come up with a clean way to accommodate everyone, or write their own wrapper, I imagine we could consider that.