Repository navigation
Description
Bug report
I've noticed that when running a matplotlib as a run-chart/long term plotting tool, memory use by matplotlib increases over the course of about a week, ultimately crashing the program. I'm continually updating my plot, but I am only plotting the most recent 100 data points (so that the data itself isn't contributing to the memory increase). See an overly simplified example below that still shows the issue. As you will see when running the code below, matplotlib\transforms.py just continually increases by around 5-15 kb on each turn of the loop, despite clearing my axes and using the object oriented version of matplotlib. This increase will continue to happen until it fills up the memory of my computer. Is this intentional? My thought was that in the object oriented program version of matplotlib, my code shouldn't increase in memory like this.
Most of the code below is simply just tracemalloc helper functions for keeping track of memory.
Code for reproduction
import matplotlib.pyplot as plt
import numpy as np
import os
import linecache
import sys
import tracemalloc
import time
def display_top(snapshot, key_type='lineno', limit=2):
'''
function for pretty printing tracemalloc output
'''
snapshot = snapshot.filter_traces((
tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
tracemalloc.Filter(False, "<unknown>"),
))
top_stats = snapshot.statistics(key_type)
print("Top %s lines" % limit)
for index, stat in enumerate(top_stats[:limit], 1):
frame = stat.traceback[0]
# replace "/path/to/module/file.py" with "module/file.py"
filename = os.sep.join(frame.filename.split(os.sep)[-2:])
print("#%s: %s:%s: %.1f KiB"
% (index, filename, frame.lineno, stat.size / 1024))
line = linecache.getline(frame.filename, frame.lineno).strip()
if line:
print(' %s' % line)
other = top_stats[limit:]
if other:
size = sum(stat.size for stat in other)
print("%s other: %.1f KiB" % (len(other), size / 1024))
total = sum(stat.size for stat in top_stats)
print("Total allocated size: %.1f KiB" % (total / 1024))
tracemalloc.start()
y = np.random.rand(100)
x = range(len(y))
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(x,y,'b-')
plt.show(block=False)
t0 = time.time()
while True:
try:
ax.clear()
ax.plot(x,np.random.rand(100),'b-')
plt.pause(0.0001)
snapshot = tracemalloc.take_snapshot()
print("Run Time:", time.time()-t0)
display_top(snapshot)
time.sleep(0.05)
except KeyboardInterrupt:
breakIf you'd prefer to use your own memory debugger, here's the simplest form of this issue:
import matplotlib.pyplot as plt
import numpy as np
import time
y = np.random.rand(100)
x = range(len(y))
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(x,y,'b-')
plt.show(block=False)
t0 = time.time()
while True:
try:
ax.clear()
ax.plot(x,np.random.rand(100),'b-')
plt.pause(0.0001)
print("Run Time:", time.time()-t0)
time.sleep(0.05)
except KeyboardInterrupt:
breakIn the console output, you'll see matplotlib\transforms.py quickly become the largest memory holder, and increase continually.
This issue also appears if I use a line object and line.set_xdata, etc., rather than ax.plot().
Using WinPython, Python 3.6.2
Matplotlib version 2.2.2
Windows 7 and Windows 10
Backend TkAgg
Activity
jklymak commentedon Aug 28, 2018
I can replicate this on master, at least using your test, and a
line.set_ydata. You have to wait a while for the transform.py code to dominate the memory, but it does after a while at about 3kb/iteration.jklymak commentedon Aug 28, 2018
child._parentskeeps growing without bound here. Not quite clear to me how the child parent relationship works here, or why the same child keeps accumulating new parents.WeatherGod commentedon Aug 28, 2018
tacaswell commentedon Aug 28, 2018
Does throwing in explicit gc (ex
gc.collect(2)) help?Is the issue that
clearis not getting enough things?How are we generating new transform objects when using
set_ydata? Can you reproduce this just by drawing repeatedly?mgeorgiadis commentedon Aug 28, 2018
Throwing in explicit gc did not appear to help the problem.
Based on my reading of transforms, it appears that the code creates some sort of additional "weak" reference between parents and children. Is this weak reference what is causing the size of the child._parents to grow continually? I don't know much about the inner workings of matplotlib or what this weakref module is.
Snippet from transforms.py
tacaswell commentedon Aug 28, 2018
Ah, so it is the weakref objects them selves which are leaking? Maybe we just need to every so often clean out
child._parents?weakrefis a built in python module (see https://docs.python.org/3/library/weakref.html).jklymak commentedon Aug 28, 2018
Child._parents keeps growing. Not really sure why other than each draw of line object does make a new transform.
anntzer commentedon Aug 28, 2018
Or just use a WeakSet? Something like
can someone see whether this fixes it?
jklymak commentedon Aug 28, 2018
Away from computer for a while but happy to try. Imagine this’ll fix the issue but a little confused why this happens in the first place.
jklymak commentedon Aug 29, 2018
@anntzer
jklymak commentedon Aug 29, 2018
WeakValueDictionaryseems to work, though I'm not sure if I understand the__get/settstate__stuff well enough to know if that is still working...anntzer commentedon Aug 29, 2018
Seems reasonable.