Skip to content

plt.hist density argument does not function as described. #10398

Closed

Description

@ryanpeach

Bug report

Bug summary

plt.hist density argument does not function as described.

Code for reproduction

import matplotlib.pyplot as plt
import numpy as np
x = np.asarray([0,0,0,0,1,1,2,2,2,2,3,3,3,3,3,3,3], dtype="float")
n, bins, _ = plt.hist(x, density=1, bins=2)
print(n)
print(np.sum(n))

Actual outcome

[0.23529412 0.43137255]
0.6666666666666666

Expected outcome

import matplotlib.pyplot as plt
import numpy as np
x = np.asarray([0,0,0,0,1,1,2,2,2,2,3,3,3,3,3,3,3], dtype="float")
n, bins, _ = plt.hist(x, density=1, bins=2)
n /= np.sum(n)  # As advertised
print(n)
print(np.sum(n))
[0.35294118 0.64705882]
1.0

Matplotlib version

  • Operating system: OSX High Sierra
  • Matplotlib version: 2.1.1
  • Matplotlib backend: MacOSX
  • Python version: 3.6.4
  • Other libraries: numpy 1.14.0
  • Installed via: pip3

Activity

  1. ImportanceOfBeingErnest commented on Feb 8, 2018

    @ImportanceOfBeingErnest
    Member

    The histogram works as expected. The density argument is explained in the documentation

    density : boolean, optional
    If True, the first element of the return tuple will be the counts normalized to form a probability density, i.e., the area (or integral) under the histogram will sum to 1. [...]

    Hence if you calculate the integral,

    print (np.sum(n*np.diff(bins)))
    

    the result is indeed 1.0, as expected.

    You might want to search stackoverflow.com on questions where people got confused by the density or normed keyword argument.

  2. tacaswell commented on Feb 8, 2018

    @tacaswell
    Member

    @ryanpeach Which documentation did you find misleading?

  3. ryanpeach commented on Feb 15, 2018

    @ryanpeach
    Author

    I guess I don’t understand why np.diff(bins) is ever relevant for the summation, “the counts normalized to form a pdf, ie, the area under the histogram will sum to one” literally means “[the sum of] the first element of the tupple will sum to 1” as the number of bins in a histogram is more like a resolution for the integral than it is an xvalue.

    I don’t understand how to generate the desired functionality. Maybe there should be an option for this if it’s such a common misunderstanding.

  4. ryanpeach commented on Feb 15, 2018

    @ryanpeach
    Author

    I guess what I’m saying is, a pdf doesn’t include the xvalue in integrating to find the normalization constant. It’s the y value that must sum to 1.

  5. tacaswell commented on Feb 15, 2018

    @tacaswell
    Member

    The area of each bar is (height * width), “the counts normalized to form a pdf, ie, the area (or integral) under the histogram will sum to one” translate to np.sum(n*np.diff(bins)) == 1 (not np.sum(n) == 1).

    We changed the kwarg from 'normed', which could reasonably be either one, to 'density' to avoid this ambiguity.

    If you want your bars to sum to one (independent of the bin widths) then pass in weights=np.ones(len(data)) / len(data) and density=False.

  6. tacaswell commented on Feb 15, 2018

    @tacaswell
    Member

    @ryanpeach How would you re-write that docstring to be clearer to you?

  7. ImportanceOfBeingErnest commented on Feb 15, 2018

    @ImportanceOfBeingErnest
    Member

    The numpy.histogram documentation is a bit more verbose on this part:

    density : bool, optional

    If False, the result will contain the number of samples in each bin. If True, the result is the value of the probability density function at the bin, normalized such that the integral over the range is 1. Note that the sum of the histogram values will not be equal to 1 unless bins of unity width are chosen; it is not a probability mass function.

    It also directly provides an example case which makes this case clear:

    image

    Maybe the matplotlib documentation can link to the numpy docs or take this over?

    The comment above

    If you want your bars to sum to one (independent of the bin widths) then pass in weights=np.ones(len(data)) / len(data) and density=False.

    is also a good one which could be added to the docs.

    Alternatively, in order not to blow up the docstrings, what about a dedicated histogram example? It seems there is currently only the pyplot-text example around showing how to use the hist function?


    @ryanpeach "a pdf doesn’t include the xvalue in integrating to find the normalization constant" is not correct; any integral takes the independent variable into account, i.e. in ∫ f(x) dx you have dx, which is exactly the bin width.

  8. ryanpeach commented on Feb 17, 2018

    @ryanpeach
    Author

    Ok, so I just learned the distinction between a pdf and a pmf. Why not just add normed as a flag and make it the pmf? This is a very common feature, and shouldn’t require hacks to perform.

    diff = np.diff(bin_edge) if not normed else 1.

  9. ImportanceOfBeingErnest commented on Feb 17, 2018

    @ImportanceOfBeingErnest
    Member

    I guess it will be hard to add another functionality to an argument which has been there for years, even though it being deprecated by now. So instead of normed one could surely think of another name.

    However, one problem would be that the histogram can take unequal bin sizes; in this case normalizing to the sum of the counts is rather dangerous. Any such plot would essentially need a label "Attention, this plot has been normalized to the sum of the frequencies independent of the bin width" or so. Hence I'm not sure if adding this normalization will not lead to even more confusion.

    At the end why not stick to the above suggestion of

    weights=np.ones(len(data)) / len(data), density=False

  10. efiring commented on Feb 18, 2018

    @efiring
    Member

    For a PMF perhaps you should use a different calculation and plotting method entirely: instead of bins, find the counts of values for all unique (discrete) values in the array, and plot each count at each discrete value, either with a marker or with a vertical line up from the origin.

  11. ryanpeach commented on Feb 18, 2018

    @ryanpeach
    Author
  12. ryanpeach commented on Feb 18, 2018

    @ryanpeach
    Author

    However, one problem would be that the histogram can take unequal bin sizes

    Make those two operations mutually exclusive (illegal to use unequal bin sizes during pmf)

    what to call it

    Call it “mass” for consistency.

  13. ryanpeach commented on Feb 18, 2018

    @ryanpeach
    Author

    why not stick to the above suggestion

    Because I can’t hardly think of a time when a pdf would be useful in the context of a histogram, whereas pmfs are very natural use cases for them, yet we have a flag for the pdf and not one for the pmf.

    On top of that, weighting the values individually seems unintuitive, though I can see why it’s correct. It just seems like something the average user shouldn’t have to hack around. I’ll add it if that’s what it takes.

  14. tacaswell commented on Feb 18, 2018

    @tacaswell
    Member

    This conversation has gotten more contentious than necessary.

    Make those two operations mutually exclusive (illegal to use unequal bin sizes during pmf)

    This is pushing on a tension we have through out Matplotlib between methods with simple signatures that do one thing (and try to do it well) and methods with very complex signatures (and internals) that do many different things. Having kwargs that are mutually exclusive or have complex conditional dependencies between them quickly becomes hard to document and maintain (we already have things like this so this is not a theoretical concern). The signature and implementation of hist are already pretty complex, I am extremely wary of adding yet more.

    Because I can’t hardly think of a time when a pdf would be useful in the context of a histogram, whereas pmfs are very natural use cases for them, yet we have a flag for the pdf and not one for the pmf.

    On top of that, weighting the values individually seems unintuitive, though I can see why it’s correct. It just seems like something the average user shouldn’t have to hack around.

    Matplotlib has a wide enough user base that we have to be careful about generalization and appeals to 'intuitive'. From my background (physics), the density normalization seems 'natural' and normalizing by mass seems like a corner case where I am fine using the 'weights' method (but definitely get that my experience is not universal).

    I’ll add it if that’s what it takes.

    As I said above, I am wary of adding complexity to hist. If you do want to add a mass kwarg, how will it interact with density, normed (which is still there, we did not set a deprecation version), and 'stacked' histograms?

    It may also be worth taking this question to numpy (a year or so ago, we had a request for optimal bin estimation in hist which got pushed to numpy and now we have bins='auto' on np.histogram). Currently (I think) in the case of non-stacked histograms we just forward all of the computation down to numpy.

    @ImportanceOfBeingErnest

    However, one problem would be that the histogram can take unequal bin sizes; in this case normalizing to the sum of the counts is rather dangerous.

    How is this different than passing in the weights?

  15. 17 remaining items

  16. ryanpeach commented on Feb 20, 2018

    @ryanpeach
    Author

    The issue about the pdf and the pmf is slightly pedantic anyway because there is no such thing as a continuous function in numpy/matplotlib. We are always dealing with discrete floating point numbers, and thus a distribution either with bins=len(np.unique(x)) (stem) or bins<len(np.unique(x)) (bar). Then there is the independent option of density with or without scale invariance. Am I right?

  17. efiring commented on Feb 20, 2018

    @efiring
    Member

    No, this is not a matter of being pedantic. There is a big difference between looking at frequencies of discrete events (e.g., counts of Fords, Hondas, etc. going through a tollboth in a day) and quantities that are binned such that there are several or many different values being grouped together in each of several bins. The fact that there is a finite number of floating point numbers that can be represented in a computer with a single or double-precision float is irrelevant.

  18. ryanpeach commented on Feb 20, 2018

    @ryanpeach
    Author

    There is a big difference between looking at frequencies of discrete events (e.g., counts of Fords, Hondas, etc. going through a tollboth in a day) and quantities that are binned such that there are several or many different values being grouped together in each of several bins.

    But these aren't mutually exclusive. For instance, my example and @ImportanceOfBeingErnest 's example are binned, continuous, and thus demand a bar plot. However, they do not require normalization via the absolute area under the curve, rather by the normalized xaxis area under the curve. A stem chart won't work for them, because they aren't a selection of a few discrete things, yet the density function won't work for them either, because it cares about the absolute area under the curve. Is that a pdf or a pmf? You tell me.

  19. ryanpeach commented on Feb 20, 2018

    @ryanpeach
    Author

    I think this inevitably comes down, on a technical level, to an argument over limiting choice within a function that was designed for convienience. All of this could be done by hand, plt.hist could be removed and let everyone just use bar plots or stem plots with np.hist till the end of time. But it's obvious that someone thought including plt.hist was important for convenience, but didn't take into account all the popular ways of using it. Now we are stuck looking at wikipedia definitions to mathematical terms that have at best a passing relation to the data visualization tequnique of a histogram, and the normalization constants commonly associated with one, and are arguing over how a user "should" do normalization, how they "should" visualize probabilities and frequencies, how they "should" bin things. Call those normalization options whatever you want, make pretty and "cannonical" defaults, but not having them because they don't fit a mold (playing favorites) even though plenty of people use them every day is what is pedantic.

  20. ImportanceOfBeingErnest commented on Feb 20, 2018

    @ImportanceOfBeingErnest
    Member

    There is a good principle when it comes to plotting: Keep data aggregation and manipulation separate from data visualization.
    Matplotlib provides functions to draw data as lines, points, shapes, bars etc. Once you have aggregated your data, you can decide on which kind of plot works best to visualize this data. This completely lies in the user's responsibility. If the user wants to draw a pmf of his data as a bar plot, he can do so.
    So following this principle there should not even be a hist function in the first place - essentially it only saves some typing and is hence convenient to use. Once you start adding all possible use cases of this function as individual arguments you will loose much of the convenience. It may look as if this function is easier to use if some argument does exactly what you want, but you'll soon end up spending more time understanding all the arguments and their implications on the drawn result than you would have spend on aggregating the data in the way you need it and simply calling the line/bar/stem/scatter function.

    In view of this, I am currently not sure if there is actually anything matplotlib should do, other than the two points mentioned above: (1) improve documentation (2) decide upon adding an additional frequency normalization argument.
    @ryanpeach Despite all the comments added from your side, we are still at exactly the point we were yesterday. Be assured that people did understand that there might be a need to do (1) and (2) and they will decide on that at some point in the future.

  21. ryanpeach commented on Feb 20, 2018

    @ryanpeach
    Author

    Ok, just let me know. I just hope you don't choose to ignore (2) while maintaining the density kwarg, that's just biased. I'll let the thread linger till you all make a decision but still offer my services in making the pull request.

  22. afvincent commented on Feb 20, 2018

    @afvincent
    Contributor

    FWIW, Numpy has a similar issue thread: numpy/numpy#9921 (but it got significantly less traction, to say the less).

  23. anntzer commented on Feb 20, 2018

    @anntzer
    Contributor

    I strongly believe mass should not be added (unless numpy adds it, which I would deem a mistake but would begrundingly accept following).

  24. jklymak commented on Feb 20, 2018

    @jklymak
    Member

    The two definitions are the same if you have equally spaced bins (which is the most usual situation). I’d agree that if numpy accepts mass we can revisit this and follow their lead. @ryanpeach if you want to continue discussion, you should chime in on the numpy issue linked above, or of course another dev is free to re-open this if they disagree...

  25. afvincent commented on Feb 21, 2018

    @afvincent
    Contributor

    A more or less recent decision was to make plt.hist a thinner (plotting) wrapper around np.histogram than it used to be (because the first one predated the latter IIRC), was it not? So I would suggest that if at some point we want to provide a way to plot PMF-like objects through plt.hist, then it may be a reasonable idea to work with Numpy devs to see if it can make it in np.histogram.

    FWIW, after a bit of googling, it seems like some (and not most of those that I checked¹) plotting softwares offer the feature that is discussed here through a keyword argument like “norm”/“density”/etc. that can take string inputs among “pdf”, “cdf”, “counts”, “probability” (usually what is discussed here), “sf”, etc.

    https://www.mathworks.com/help/matlab/ref/histogram.html
    http://reference.wolfram.com/language/ref/Histogram.html
    https://www.rdocumentation.org/packages/graphics/versions/3.4.3/topics/hist
    https://www.originlab.com/doc/Origin-Help/Create-Histogram

    Edit: burnt ^^.

  26. ryanpeach commented on Feb 21, 2018

    @ryanpeach
    Author

    Just posted on the numpy thread.

    I don't think it's numpy's job to do this because normalization is easy to do mathematically post-hoc. It's just the graphing that's hard. It's matplotlibs wrapper that makes it difficult to do intermediate steps on the binning like normalization. But I digress.

  27. nunocalaim commented on Aug 29, 2018

    @nunocalaim

    So I came here because I have a histogram with log=True and density=True and it is giving me weird results. Note that I have logarithmic spaced bins
    captura de ecra 2018-08-29 as 18 59 22

    Note how on some histograms the bins, for which supposedly there are no counts, still have some height to them (is appearing above 0). Ideally this is not what I was looking for. I feel this discussion is related to this problem. And I am unsure if this is a bug or intended.

  28. ImportanceOfBeingErnest commented on Aug 29, 2018

    @ImportanceOfBeingErnest
    Member

    @nunocalaim Thanks for reaching out. However I do not think your issue is in any way related. Would you mind opening a new issue and include a self-contained example (i.e. a code one can copy, paste and run) to allow people to reproduce your problem? This is the only way one can find out if the outcome is expected or whether there is a problem in the matplotlib code base.

  29. jklymak commented on Nov 24, 2020

    @jklymak
    Member

    Note that we now have stair which allows folks to normalize whatever way is most appropriate for their field. #18275

    I'm pretty concerned with Matplotlib mediating multiple normalization conventions among fields. But if someone can come up with a clean way to accommodate everyone, or write their own wrapper, I imagine we could consider that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions