You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Automatic number of bins for matplotlib histograms #4316
Long story short, plt.hist has poor default behaviour as I have to constantly play around with the number of bins to get a good idea of what I'm looking at. The bins=10 works ok for up to 1000 points, but has poor performance for anything more, and doesn't account for variability either
R doesn't suffer from these problems and provides methods for use with it's hist method. I would like to provide similar functionality for matplotlib, to at least provide some kind of good starting point, as histograms are very useful for initial data discovery.
The notebook above provides an explanation of the problem as well as some proposed alternatives. Use different datasets (type and size) to see the performance of the other alternatives suggested.
This should probably go against numpy or scipy, we (try to) delegate all of the computation to them.
I am not a huge fan of mpl picking up what seems like domain specific heuristics which does lead to the question of where it should go and I don't have a good answer for that yet.
I do recognize many of those optimality measures, and my first paper
utilized Scott's Rule and Friedmen-Diaconis. But, I do think that this
belongs in numpy where everyone can benefit from it. We could then make
sure that our handling of "bins" is agnostic to it being a float or a
string.
Long story short, plt.hist has poor default behaviour as I have to
constantly play around with the number of bins to get a good idea of what
I'm looking at. The bins=10 works ok for up to 1000 points, but has poor
performance for anything more, and doesn't account for variability either
R doesn't suffer from these problems and in case provides methods for use
here. I would like to provide similar functionality for matplotlib, to at
least provide some kind of good starting point.
The notebook above provides an explanation of the problem as well as some
proposed alternatives. Use different datasets (type and size) to see the
performance of the other alternatives suggested.
I've put together an implementation to add this new functionality nayyarv@f000359,
but am hesitant to make a pull request as I would like some feedback from a
maintainer before doing so.
—
Reply to this email directly or view it on GitHub #4316.
Firstly, I can't believe I managed to spell Freedman as Freeman for basically everything I've done.
I was about to argue that this belongs in mpl as histograms are primarily a visual tool, but I didn't consider other libraries (such as bokeh or even ggplot) that may not use mpl's implementation of hist, that would not be able to benefit.
I'll raise this issue with the numpy team, and see if I can get it included there.
I want to push back a little bit on histograms being primarily visual. Some sections of image processing algorithms use histograms heavily and high energy physics experiments are built around measuring the difference between an observed histogram and a predicted one.
Nice, I hope this gets into numpy/scipy, as someone who has used the default hexbin (and written my own 3d spherical binning based of of hexbin, (I wanted to find the mode of my data)), I wonder if these techniques could apply to other binning strategies?
Oops, I completely forgot about image processing, though the idea of optimal bin numbers makes less sense as there is no distribution to measure against (and are usually pre-defined). The physics experiment however could benefit from something of the sort.
Either way I stand corrected. I've made the change to my numpy fork and I'm currently waiting for the scipy mailing lists to come back online so I can make a post.
If I get it accepted, I'll open the request to update matplotlib, but until then I'll close this.
Activity
[-]Automatic Bin Choices for matplotlib histograms[/-][+]Automatic number of bins for matplotlib histograms[/+]tacaswell commentedon Apr 6, 2015
This should probably go against
numpyorscipy, we (try to) delegate all of the computation to them.I am not a huge fan of mpl picking up what seems like domain specific heuristics which does lead to the question of where it should go and I don't have a good answer for that yet.
WeatherGod commentedon Apr 6, 2015
I do recognize many of those optimality measures, and my first paper
utilized Scott's Rule and Friedmen-Diaconis. But, I do think that this
belongs in numpy where everyone can benefit from it. We could then make
sure that our handling of "bins" is agnostic to it being a float or a
string.
On Mon, Apr 6, 2015 at 7:19 AM, Varun Nayyar notifications@github.com
wrote:
nayyarv commentedon Apr 7, 2015
Firstly, I can't believe I managed to spell Freedman as Freeman for basically everything I've done.
I was about to argue that this belongs in mpl as histograms are primarily a visual tool, but I didn't consider other libraries (such as bokeh or even ggplot) that may not use mpl's implementation of hist, that would not be able to benefit.
I'll raise this issue with the numpy team, and see if I can get it included there.
Thanks!
tacaswell commentedon Apr 7, 2015
I want to push back a little bit on histograms being primarily visual. Some sections of image processing algorithms use histograms heavily and high energy physics experiments are built around measuring the difference between an observed histogram and a predicted one.
OceanWolf commentedon Apr 7, 2015
Nice, I hope this gets into numpy/scipy, as someone who has used the default hexbin (and written my own 3d spherical binning based of of hexbin, (I wanted to find the mode of my data)), I wonder if these techniques could apply to other binning strategies?
nayyarv commentedon Apr 8, 2015
Oops, I completely forgot about image processing, though the idea of optimal bin numbers makes less sense as there is no distribution to measure against (and are usually pre-defined). The physics experiment however could benefit from something of the sort.
Either way I stand corrected. I've made the change to my numpy fork and I'm currently waiting for the scipy mailing lists to come back online so I can make a post.
If I get it accepted, I'll open the request to update matplotlib, but until then I'll close this.
Thanks!