Memory issue when plotting large arrays with pcolormesh

Plotting large-ish arrays using pcolormesh with current (e2c5918) matplotlib master looks like it might have a memory leak. Running the following code on matplotlib v1.3.1 produces a QuadMesh in approximately a second. However running the same code on matplotlib master floods the memory until either the pcolormesh call is aborted or it runs out of memory.

import matplotlib.pyplot as plt
import numpy as np

data = np.random.rand(3000*3000).reshape(3000, 3000)
plt.pcolormesh(data)
plt.show()

Confirmed with agg backend, and simply ending with plt.draw(). I don't think this is a memory leak, but it is certainly a serious degradation of performance, probably a side effect of some enhancement.

The problem is that add_collection is calling get_datalim which is calling get_paths, which is constructing a huge list of Path objects. Pcolormesh is supposed to be handling its own datalim setting, precisely to avoid this. I don't have time now to look into this, but it needs to be fixed.

Fix regression in pcolormesh: don't generate a path list.

Use autolim kwarg in add_collection to prevent duplication of effort.

I'm using Version 1.4.2 (macports) and the problem still exists! Scripts that worked fine with version 1.3.1 now run out of memory.

Can you provide a (minimal) example which is still a problem?

Which backend are you using?

The example from the first comment perfectly illustrates the problem when using savefig instead of show:

import matplotlib.pyplot as plt
import numpy as np

data = np.random.rand(3000*3000).reshape(3000, 3000)
plt.pcolormesh(data)
plt.savefig("test.png",bbox_inches='tight')

the keyword box_inchesis the actual problem. Running this script with matplotlib 1.3.1 takes only seconds and a few hundred MB of memory. Running it with version 1.4.2 is not possible because it runs out of memory after a few minutes. Killing the script before it crashes by itself reveals that it tries to convert the mesh into a path:

Traceback (most recent call last):
  File "pcolormesh_bug.py", line 6, in <module>
    plt.savefig("test.png",bbox_inches='tight')
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/pyplot.py", line 577, in savefig
    res = fig.savefig(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/figure.py", line 1470, in savefig
    self.canvas.print_figure(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 2151, in print_figure
    bbox = a.get_window_extent(renderer)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/collections.py", line 207, in get_window_extent
    return self.get_datalim(transforms.IdentityTransform())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/collections.py", line 180, in get_datalim
    paths = self.get_paths()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/collections.py", line 1675, in get_paths
    self.set_paths()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/collections.py", line 1680, in set_paths
    self._meshWidth, self._meshHeight, self._coordinates)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/collections.py", line 1706, in convert_mesh_to_paths
    return [Path(x) for x in points]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/path.py", line 137, in __init__
    vertices = np.asarray(vertices, np.float_)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/numeric.py", line 462, in asarray
    return array(a, dtype, copy=False, order=order)
KeyboardInterrupt

I hope this helps to find the problem!

Re opening this so it does not get lost.

This would be a serious issue because I think ipython's inline backend
automatically does bbox_inches='tight'.

On Fri, Nov 7, 2014 at 9:07 AM, Thomas A Caswell notifications@github.com
wrote:

Re opening this so it does not get lost.

—
Reply to this email directly or view it on GitHub
#3095 (comment)
.

I can confirm that this happens and seems related to bbox_inches='tight'

This also might explain the issue that came up that was specific to the inline backend (#3508) which Phil already linked this way.

It looks like this came from 88b722f which removed an over-loading of get_datalim on Quadmesh. This causes the get_windowextent method from collections to try to make all of the paths for the quadmesh as python path objects which causes at least a 5x blow up in the memory used (just from the data, let alone the Path objects). Working on a PR to more efficiently determine the extent (and getting very confused with using offsets with QuadMesh).

BUG : provide get_window_extent for QuadMesh

closed by #4045

Navigation Menu

Repository navigation

Memory issue when plotting large arrays with pcolormesh #3095

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Memory issue when plotting large arrays with pcolormesh #3095

Description

Activity

efiring commented on May 30, 2014

efiring commented on May 30, 2014

r5r3 commented on Nov 6, 2014

tacaswell commented on Nov 6, 2014

r5r3 commented on Nov 7, 2014

tacaswell commented on Nov 7, 2014

WeatherGod commented on Nov 7, 2014

jenshnielsen commented on Nov 7, 2014

tacaswell commented on Nov 7, 2014

tacaswell commented on Jan 20, 2015

tacaswell commented on Jan 31, 2015

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions