Skip to content

Automatic number of bins for matplotlib histograms #4316

Closed

Description

@nayyarv

http://nbviewer.ipython.org/github/nayyarv/matplotlib/blob/master/examples/statistics/Automating%20Binwidth%20Choice%20for%20Histogram.ipynb

Long story short, plt.hist has poor default behaviour as I have to constantly play around with the number of bins to get a good idea of what I'm looking at. The bins=10 works ok for up to 1000 points, but has poor performance for anything more, and doesn't account for variability either

R doesn't suffer from these problems and provides methods for use with it's hist method. I would like to provide similar functionality for matplotlib, to at least provide some kind of good starting point, as histograms are very useful for initial data discovery.

The notebook above provides an explanation of the problem as well as some proposed alternatives. Use different datasets (type and size) to see the performance of the other alternatives suggested.

I've put together an implementation to add this new functionality, but am hesitant to make a pull request as I would like some feedback from a maintainer before doing so.

Activity

  1. changed the title [-]Automatic Bin Choices for matplotlib histograms[/-] [+]Automatic number of bins for matplotlib histograms[/+] on Apr 6, 2015
  2. tacaswell commented on Apr 6, 2015

    @tacaswell
    Member

    This should probably go against numpy or scipy, we (try to) delegate all of the computation to them.

    I am not a huge fan of mpl picking up what seems like domain specific heuristics which does lead to the question of where it should go and I don't have a good answer for that yet.

  3. WeatherGod commented on Apr 6, 2015

    @WeatherGod
    Member

    I do recognize many of those optimality measures, and my first paper
    utilized Scott's Rule and Friedmen-Diaconis. But, I do think that this
    belongs in numpy where everyone can benefit from it. We could then make
    sure that our handling of "bins" is agnostic to it being a float or a
    string.

    On Mon, Apr 6, 2015 at 7:19 AM, Varun Nayyar notifications@github.com
    wrote:

    http://nbviewer.ipython.org/github/nayyarv/matplotlib/blob/master/examples/statistics/Automating%20Binwidth%20Choice%20plt.hist.ipynb

    Long story short, plt.hist has poor default behaviour as I have to
    constantly play around with the number of bins to get a good idea of what
    I'm looking at. The bins=10 works ok for up to 1000 points, but has poor
    performance for anything more, and doesn't account for variability either

    R doesn't suffer from these problems and in case provides methods for use
    here. I would like to provide similar functionality for matplotlib, to at
    least provide some kind of good starting point.

    The notebook above provides an explanation of the problem as well as some
    proposed alternatives. Use different datasets (type and size) to see the
    performance of the other alternatives suggested.

    I've put together an implementation to add this new functionality
    nayyarv@f000359,
    but am hesitant to make a pull request as I would like some feedback from a
    maintainer before doing so.

    —
    Reply to this email directly or view it on GitHub
    #4316.

  4. nayyarv commented on Apr 7, 2015

    @nayyarv
    Author

    Firstly, I can't believe I managed to spell Freedman as Freeman for basically everything I've done.

    I was about to argue that this belongs in mpl as histograms are primarily a visual tool, but I didn't consider other libraries (such as bokeh or even ggplot) that may not use mpl's implementation of hist, that would not be able to benefit.

    I'll raise this issue with the numpy team, and see if I can get it included there.

    Thanks!

  5. tacaswell commented on Apr 7, 2015

    @tacaswell
    Member

    I want to push back a little bit on histograms being primarily visual. Some sections of image processing algorithms use histograms heavily and high energy physics experiments are built around measuring the difference between an observed histogram and a predicted one.

  6. OceanWolf commented on Apr 7, 2015

    @OceanWolf
    Member

    Nice, I hope this gets into numpy/scipy, as someone who has used the default hexbin (and written my own 3d spherical binning based of of hexbin, (I wanted to find the mode of my data)), I wonder if these techniques could apply to other binning strategies?

  7. nayyarv commented on Apr 8, 2015

    @nayyarv
    Author

    Oops, I completely forgot about image processing, though the idea of optimal bin numbers makes less sense as there is no distribution to measure against (and are usually pre-defined). The physics experiment however could benefit from something of the sort.

    Either way I stand corrected. I've made the change to my numpy fork and I'm currently waiting for the scipy mailing lists to come back online so I can make a post.

    If I get it accepted, I'll open the request to update matplotlib, but until then I'll close this.

    Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions