Skip to content

Animated histogram example is very misleading #18880

Closed

Description

@lodo1995

Problem

The example in https://matplotlib.org/gallery/animation/histogram.html supposedly shows how to make an animated histogram out of some data (in the example, random numbers from a gaussian distribution). However, the example generates a wrong animation.

This can be a big issue for people who do not have much experience tinkering with matplotlib and just plug their data into the example, thinking that it will work, and who might even end up going through all their data thinking that the histogram is correct and their data is bad (I just spent some time helping a friend to fix the histogram).

The problem lies in the fact that np.histogram is used to generate a completely new histogram, from completely new data, at each timestep. However, only the top coordinates of each bin are updated. The code assumes that bins returned by np.histogram will always have the same x coordinates and do not need to be completely redrawn. This is of course wrong. The presence of outliers may force the bins to change width and may cause them to "move" towards one of the sides.

Suggested Improvement

The simplest improvement is, of course, to update the animate function to completely redraw the bins, including inside the animate function more of the code that is already present outside it, to generate the patch in the first place:

def animate(i):
    # simulate new data coming in
    data = np.random.randn(1000)
    n, bins = np.histogram(data, 100)

    left = np.array(bins[:-1])
    right = np.array(bins[1:])
    top = bottom + n

    verts[0::5, 0] = left
    verts[0::5, 1] = bottom
    verts[1::5, 0] = left
    verts[1::5, 1] = top
    verts[2::5, 0] = right
    verts[2::5, 1] = top
    verts[3::5, 0] = right
    verts[3::5, 1] = bottom

    return [patch, ]

Note that an alternative solution, namely the use of np.digitize instead of np.histogram to fit the new data to the existing histogram, would not work, as the new data might cover a different range than the original bins.

This would make the code much better, but there is another hidden problem: set_xlim and set_ylim are currently called based on the first batch of data. This might cause two issues, which do not show up that much with gaussian-distributed data, but may arise in real-world usage:

  • if outliers force the bin distribution to be much to the left or much to the right, some bins might be outside of limits and not show up;
  • if some bins end up being taller, they will be "cut" at the top of the plot.

To solve this issue, it is necessary to know beforehand the left-most, right-most and tallest bin across all batches of data. However, the solution to this is not trivial and may change depending on the actual real-world scenario in which this example is being applied. So I am not sure which solution should be implemented in the example:

  • assume that it is not possible to analyze all data before starting the plotting; in this case, just hard-code reasonable values for left, right and top limits;
  • call np.histogram on each batch of data, find the left-most left bin edge, right-most right bin edge and tallest bin across all histograms computed, set those as the limits;
  • call np.histogram once on all the data, generating a single set of bins; set the left and right limits based on these bins; then use np.digitize to assign each data point from each batch to the correct bin; use these assignments to find the tallest bin, not the original values from np.histogram (those would be too high); the advantage of this solution over the previous one is that the bins themselves do not move; they only change height.

The second or third solution would require quite a big refactoring of the source code, which might be good in general, as the new version of animate shown above would have quite a lot of code duplication with the main script, which could be avoided with a refactoring.

Activity

  1. jklymak commented on Nov 3, 2020

    @jklymak
    Member

    I think you are correct that the example is poor. But the solution is to simply manually specify the bins rather than computing them automatically each time.

  2. ianhi commented on Nov 3, 2020

    @ianhi
    Contributor

    A few quick things on this:

    1. I also found this example confusing and opened: Use plt.stairs in animated histogram example #18570 suggesting using the plt.stairs for that example.
    2. There was recently a very nice simplification of the example: Simplification of animated histogram example #18821
      • Takes the manually specifying bins approach.
    3. I implemented a dynamically updating hist that also allows using bins='auto' by creating new Rectangle patches everytime + recalculating the xlim/ylim. I think this may also be simple enough to be an example, but I plt.stairs is almost certainly a better way to accomplish it.:
        heights, bins = np.histogram(arr, bins=bins, density=density, weights=weights)
        width = bins[1] - bins[0]
        new_patches = []
        for i in range(len(heights)):
            new_patches.append(Rectangle((bins[i], 0), width=width, height=heights[i]))
        xlims = (bins.min(), bins.max())
        ylims = (0, heights.max() * 1.05)

    https://github.com/ianhi/mpl-interactions/blob/99d6337b135b0f0299126061a0af70f00c210910/mpl_interactions/pyplot.py#L300-L308
    https://github.com/ianhi/mpl-interactions/blob/99d6337b135b0f0299126061a0af70f00c210910/mpl_interactions/pyplot.py#L329

  3. jklymak commented on Nov 3, 2020

    @jklymak
    Member

    OK, so this is already fixed in 3.4.0. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions