Skip to content

Memory leak with log scale in pcolorfast, pcolormesh, imshow ... #15474

New issue
New issue
Closed
Closed
Memory leak with log scale in pcolorfast, pcolormesh, imshow ...#15474
Labels
PerformanceRelease criticalFor bugs that make the library unusable (segfaults, incorrect plots, etc) and major regressions.status: confirmed bugtopic: transforms and scales
Milestone
v3.2.2

Description

@ibancg
ibancg
opened on Oct 22, 2019

matplotlib seems to leak memory when I switch to log scale on the Y axis, for the functions pcolorfast, pcolormesh, imshow and specgram

Please consider the following spectrogram animation (sorry for the extra dependencies to scipy)

from scipy.signal.spectral import spectrogram
from scipy.signal.waveforms import chirp

import matplotlib.animation as animation
import matplotlib.pyplot as plt
import numpy as np

fs = 8000 # sampling rate
t = np.arange(0, 5.0, 1/fs) # 5 seconds of time

fig, axes = plt.subplots(nrows=1)

def signal(t): # generates a "chirp" signal
    return chirp(t = t, f0 = 100.0,  f1 = 3000.0, t1 = 10.0)

def spect(x): # computes spectrogram
    freq, time, Sxx = spectrogram(x, fs = fs, nfft = 1024, noverlap = 512, nperseg = 1024)
    Z = 10 * np.log10(Sxx)
    return freq, time, Z

y = signal(t)
freq, time, Z = spect(y)
im = axes.pcolorfast(time, freq, Z)
axes.set_xlabel('t [s]')
axes.set_ylabel('f [Hz]')
axes.set_yscale('log') # <-- exposes the leak
axes.set_ylim(0, fs/2)

def animate(i):
    y = signal(t + i/30.0) # sliding temporal window
    freq, time, Z = spect(y)
    im.set_data(Z)
    return [ im ]

ani = animation.FuncAnimation(fig, animate, np.arange(1, 200), 
                              interval=0, blit=True)
plt.show()

Everything stays stable if I stick to linear scale on the Y axis, commenting out the line

axes.set_yscale('log')

I can reproduce it with matplotlib 3.1.1 (via pip) and previous versions (tested with 3.0.2 from system-wide Debian testing package). Also I am using Qt5Agg backend and python 3.7.5rc1

Activity

  1. jklymak commented on Oct 22, 2019

    @jklymak
    jklymak
    on Oct 22, 2019
    Member

    It’s possible a transform isn’t being cleaned up. How bad is the leak?

  2. ibancg commented on Oct 22, 2019

    @ibancg
    ibancg
    on Oct 22, 2019
    Author

    Using psutil I see that the heap usage in this particular example stays stable at 145 MB with linear scale, but it eats up 2 GB in less than 20 seconds for the log scale.

    Our application, with less frequent updates, takes a few minutes to claim all the memory.

  3. jklymak commented on Oct 22, 2019

    @jklymak
    jklymak
    on Oct 22, 2019
    Member

    Agreed, this leaks on my machine using master.

  4. jklymak
    added this to the v3.3.0 milestone on Oct 22, 2019
  5. jklymak
    added
    status: confirmed bug
    topic: transforms and scales
    on Oct 22, 2019

  6. espehannila commented on Mar 10, 2020

    @espehannila
    espehannila
    on Mar 10, 2020

    Memory leak is real, is there any resolution for this issue or any temporary solution to bypass before new release?

    ...\site-packages\matplotlib\image.py:431: size=141 MiB, count=79, average=1833 KiB
...\site-packages\matplotlib\image.py:466: size=141 MiB, count=52, average=2785 KiB

  7. jklymak commented on Mar 10, 2020

    @jklymak
    jklymak
    on Mar 10, 2020
    Member

    I guess a work around is to animate by saving pngs and then calling ffmpegs or other writers.

  8. tacaswell commented on May 18, 2020

    @tacaswell
    tacaswell
    on May 18, 2020
    Member

    We talked about this on the call and we suspect that the memory leak is in the c++ based resampling code for AxesImage.

  9. tacaswell
    added
    Release criticalFor bugs that make the library unusable (segfaults, incorrect plots, etc) and major regressions.
    on May 18, 2020

  10. anntzer commented on May 18, 2020

    @anntzer
    anntzer
    on May 18, 2020
    Contributor

    Upon inspection it looks like the issue may be(?) in _image_wrapper.cpp::image_resample, specifically at

        output_array = (PyArrayObject *)PyArray_FromAny(
        py_output_array, NULL, 2, 3, NPY_ARRAY_C_CONTIGUOUS, NULL);
    if (output_array == NULL) {

    where the problem is that we already allocated an array in python code and allocate another PyObject here(?); however the obvious

    diff --git i/src/_image_wrapper.cpp w/src/_image_wrapper.cpp
index de65e90c8..cb9a91ce0 100644
--- i/src/_image_wrapper.cpp
+++ w/src/_image_wrapper.cpp
@@ -146,9 +146,13 @@ image_resample(PyObject *self, PyObject* args, PyObject *kwargs)
         goto error;
     }
 
-    output_array = (PyArrayObject *)PyArray_FromAny(
-        py_output_array, NULL, 2, 3, NPY_ARRAY_C_CONTIGUOUS, NULL);
-    if (output_array == NULL) {
+    if (!PyArray_Check(py_output_array)) {
+        PyErr_SetString(PyExc_ValueError, "output array must be a numpy array");
+        goto error;
+    }
+    output_array = (PyArrayObject *)py_output_array;
+    if (!PyArray_IS_C_CONTIGUOUS(output_array)) {
+        PyErr_SetString(PyExc_ValueError, "output array must be a C-contiguous");
         goto error;
     }
 
@@ -330,11 +334,10 @@ image_resample(PyObject *self, PyObject* args, PyObject *kwargs)
 
     Py_DECREF(input_array);
     Py_XDECREF(transform_mesh_array);
-    return (PyObject *)output_array;
+    Py_RETURN_NONE;
 
  error:
     Py_XDECREF(input_array);
-    Py_XDECREF(output_array);
     Py_XDECREF(transform_mesh_array);
     return NULL;
 }

    (just getting the array from outside and making sure it has the right size and not messing with its refcounts at all) does not seem to fix the issue.

  11. QuLogic
    added a commit that references this issue on May 20, 2020

    Fix leak of input mesh during image resampling.

    8e7aaff

  12. QuLogic commented on May 20, 2020

    @QuLogic
    QuLogic
    on May 20, 2020
    Member

    It was right to look at the C++ code, but it wasn't output array that leaking. I included that change as part of general cleanup, but the real leak was the input mesh.

  13. QuLogic
    added
    Performance
    on May 20, 2020
  14. tacaswell
    modified the milestones: v3.3.0, v3.2.2 on May 20, 2020
  15. QuLogic
    added a commit that references this issue on May 20, 2020

    Fix leak of input mesh during image resampling.

    7977b5f
  16. tacaswell
    closed this as completedin 8e7aaffon May 21, 2020

  17. tacaswell commented on May 21, 2020

    @tacaswell
    tacaswell
    on May 21, 2020
    Member

    Thanks for reporting this @ibancg !

  18. ibancg commented on May 21, 2020

    @ibancg
    ibancg
    on May 21, 2020
    Author

    Great, thanks for the fix!.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    PerformanceRelease criticalFor bugs that make the library unusable (segfaults, incorrect plots, etc) and major regressions.status: confirmed bugtopic: transforms and scales

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions