[Bug]: Long audio files result in incomplete spectrogram visualizations

Bug summary

When using matplotlib to generate spectrogram visualizations of audio files, if the audio file is too long, the spectrogram portion of the plot becomes blank towards the latter half, while the waveform continues to be displayed properly.

Code for reproduction

import torchaudio
import torchaudio.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np
import scipy.interpolate
import librosa.display
import matplotlib.gridspec as gridspec

from tqdm import tqdm
from concurrent.futures import ProcessPoolExecutor, as_completed


def compute_spectrogram_per_channel(channel_waveform, sample_rate):
    # Create transformer to convert waveform to spectrogram
    spectrogram_transform = transforms.Spectrogram(n_fft=2048, hop_length=256)

    # Apply the transformer
    spectrogram = spectrogram_transform(channel_waveform.unsqueeze(0))

    amplitude_spectrogram = np.sqrt(spectrogram)
    db_spectrogram = librosa.amplitude_to_db(amplitude_spectrogram[0].numpy(), ref=np.max)
    db_spectrogram = np.clip(db_spectrogram, a_min=None, a_max=0)  # clip to 0dB

    # Set new log scale
    num_freqs, num_frames = db_spectrogram.shape
    min_freq = 1  # human hearing range in Hz
    max_freq = sample_rate / 2  # Nyquist frequency
    frequencies = np.linspace(min_freq, max_freq, num=num_freqs)

    # Create a new scale
    log_scale = np.log10(frequencies)
    linear_scale = np.linspace(np.log10(min_freq), np.log10(max_freq), num=num_freqs)
    scale_ratio = 0.75  # adjust this parameter to control the ratio of log scale and linear scale
    new_scale = scale_ratio * log_scale + (1 - scale_ratio) * linear_scale

    new_db_spectrogram = np.empty_like(db_spectrogram)

    # Apply interpolation for each frame
    for frame in tqdm(range(num_frames)):
        interpolator = scipy.interpolate.interp1d(log_scale, db_spectrogram[:, frame])
        new_db_spectrogram[:, frame] = interpolator(new_scale)

    return channel_waveform.t().numpy(), db_spectrogram, new_db_spectrogram


def plot_spectrogram(waveforms, new_db_spectrograms, audio_duration):
    num_channels = len(waveforms)

    # Create a plot, set the background to black and adjust the size based on audio duration
    plt.figure(figsize=(max(audio_duration * 2, 10), 8), facecolor="black")

    # Dynamically create subplots based on the number of channels
    gs = gridspec.GridSpec(2 * num_channels, 1, height_ratios=[1] * num_channels + [5] * num_channels)

    # Loop through each channel to plot the waveform and spectrogram
    for i in range(num_channels):
        # Plot the waveform
        ax_waveform = plt.subplot(gs[i])
        ax_waveform.plot(waveforms[i], color="#4BF2A7")
        nonzero_indices = np.where(waveforms[i] != 0)[0]  # Find indices of non-zero values
        ax_waveform.set_xlim(nonzero_indices[0], nonzero_indices[-1])  # Set x limit to range of non-zero values
        ax_waveform.axis("off")

        # Plot the spectrogram
        ax_spectrogram = plt.subplot(gs[i + num_channels])
        ax_spectrogram.imshow(new_db_spectrograms[i], origin="lower", aspect="auto")
        ax_spectrogram.axis("off")

    plt.subplots_adjust(left=0, right=1, top=1, bottom=0, wspace=0, hspace=0)  # Adjust to remove borders and gaps
    plt.savefig("spectrogram.jpeg", facecolor="black", bbox_inches="tight", pad_inches=0)  # Save the figure


def main(file):
    # Load the audio file
    waveform, sample_rate = torchaudio.load(file)

    num_channels = waveform.shape[0]
    audio_duration = waveform.shape[1] / sample_rate  # Calculate audio duration

    with ProcessPoolExecutor() as executor:
        futures = {
            executor.submit(compute_spectrogram_per_channel, waveform[ch], sample_rate): ch
            for ch in range(num_channels)
        }
        waveforms = [None] * num_channels
        db_spectrograms = [None] * num_channels
        new_db_spectrograms = [None] * num_channels
        for future in as_completed(futures):
            ch = futures[future]
            waveforms[ch], db_spectrograms[ch], new_db_spectrograms[ch] = future.result()

    # Adjust figure width based on audio duration
    plot_spectrogram(waveforms, new_db_spectrograms, audio_duration)


# Run the main function
if __name__ == "__main__":
    file = "long.wav"
    main(file)

Actual outcome

When visualizing an audio file of significant length, the spectrogram does not render correctly for the entire duration of the audio. The latter part of the spectrogram is blank and contains no information, which doesn't match with the waveform visualization, which continues to display normally.

long wave

short wave

Expected outcome

The spectrogram should be consistently rendered for the entire duration of the audio file, matching the waveform visualization. Regardless of the length of the audio file, the spectrogram should be complete and not become blank at any point.

Additional information

Other libraries involved: torchaudio, librosa, numpy, scipy

This issue occurs regardless of the audio file format or the specifics of the audio content. It seems directly related to the length of the audio file.

Operating system

Windows & Ubuntu

Matplotlib Version

3.7.2

Matplotlib Backend

TkAgg

Python version

Python 3.10.10

Jupyter version

No response

Installation

pip

Can you reproduce this without the dependency on other libraries (i.e. pure numpy and matplotlib, preferably)?

We tend not debug scripts that require additional libraries.

I'm suspicious that the facecolor (which is set twice) also ends at the same point the image ends, which may indicate some form of clip box.

You may also have problems with large images (>2**23 per dimension), which do get downsampled automatically, and not 100% sure how that handles "auto" aspect.

I'm also slightly curious if removing the bbox_inches="tight" affects the behavior, though I would rather doubt it

At some point you can't keep creating an image. Making a figure that is hundreds of inches wide can't render on any realistic screen anyway. Suggest you reduce your data somehow, perhaps by making the spectrogram windows wider in time.

At some point you can't keep creating an image. Making a figure that is hundreds of inches wide can't render on any realistic screen anyway. Suggest you reduce your data somehow, perhaps by making the spectrogram windows wider in time.

I need to create a very wide image to display on a web page. Users can drag to scroll the viewport. The data cannot be reduced or scaled, this is the only way.

Can you reproduce this without the dependency on other libraries (i.e. pure numpy and matplotlib, preferably)?

We tend not debug scripts that require additional libraries.

I'm suspicious that the facecolor (which is set twice) also ends at the same point the image ends, which may indicate some form of clip box.

You may also have problems with large images (>2**23 per dimension), which do get downsampled automatically, and not 100% sure how that handles "auto" aspect.

I'm also slightly curious if removing the bbox_inches="tight" affects the behavior, though I would rather doubt it

I tried removing bbox_inches="tight" and "facecolor='black'" at the plt.savefig function, but the result remained the same as before.

Here's the minimal test code that I've written which can reproduce this issue:

import numpy as np
import matplotlib.pyplot as plt

image = np.zeros((50000, 800, 3))

linspace = np.linspace(0, 1, 800)

image[:, :, 0] = 1 - linspace  # Red channel
image[:, :, 2] = linspace  # Blue channel

r, g, b = image[:, :, 0], image[:, :, 1], image[:, :, 2]
r_hist, r_bins = np.histogram(r, bins=256, range=[0, 1])
g_hist, g_bins = np.histogram(g, bins=256, range=[0, 1])
b_hist, b_bins = np.histogram(b, bins=256, range=[0, 1])

plt.figure(figsize=(500, 8), dpi=100)

plt.subplot(2, 1, 1)

plt.imshow(image, aspect="auto")

plt.subplot(2, 1, 2)

plt.plot(r_bins[:-1], r_hist, color="red", alpha=0.7, label="Red")
plt.plot(g_bins[:-1], g_hist, color="green", alpha=0.7, label="Green")
plt.plot(b_bins[:-1], b_hist, color="blue", alpha=0.7, label="Blue")

plt.tight_layout()
plt.savefig("gradient_image.jpeg", pad_inches=0)

@djkcyl

Thanks @djkcyl, I can reproduce your test code on my machine. Lowering the number of pixels and figsize by a factor of 10, I get this as the expected image:

I would definitely consider this a bug. The only justification I can think of for limiting the size of an image is preventing a user from running out of memory and crashing the shell, but if we want a hard limit then it should be made an rcparam. Beyond the use case above, images for print often need much-larger-than-screen resolutions, and if someone wants to do manipulations on a gigapixel image they should ideally only be limited by their machine's capability.

I'm pretty sure this is an agg limitation of 2^16 pixels in either direction?

2^16 = 65536 pixels would work for this example... if I eyeball it this looks more like this image cuts off around 2^14 = 16384 pixels (about a third of the 50000 we are trying to draw). That limit might come if they are indexing RGB subpixels instead of real pixels (3 * 2^14 fits in a uint16).

I'm pretty unfamiliar with AGG, but looking at the source I do see a note here that suggests that this limit should be able to be raised to 2^32 somehow: https://agg.sourceforge.net/antigrain.com/__code/include/agg_scanline_bin.h.html#scanline32_bin

// Adaptation for 32-bit screen coordinates ([scanline32_bin](https://agg.sourceforge.net/antigrain.com/__code/include/agg_scanline_bin.h.html#scanline32_bin)) has been sponsored by 
// Liberty Technology Systems, Inc., visit http://lib-sys.com

Maybe related to #23826?

We've had quite a few issues like this over the years. Maybe Cairo doesn't have this issue?

@djkcyl

Probably also related to #19276 .
Unlike #19276 (which was an image that was bigger than the limit that were failing to down sample), this is a big render failing.

At 50k pixels wide Agg appears to be breaking down. If you remove the image and just look at the bounding boxes in the Axes something is going very wrong.

I think the quickest fix for @djkcyl is to use the svg backend with interpolation='none' which will drop an (un-resampled) image in the SVG and defer the rendering problem to the browser:

import numpy as np
import matplotlib.pyplot as plt

N = 800
image = np.zeros((50_000, N, 3))

linspace = np.linspace(0, 1, N)

image[:, :, 0] = 1 - linspace  # Red channel
image[:, :, 2] = linspace  # Blue channel

r, g, b = image[:, :, 0], image[:, :, 1], image[:, :, 2]
r_hist, r_bins = np.histogram(r, bins=256, range=[0, 1])
g_hist, g_bins = np.histogram(g, bins=256, range=[0, 1])
b_hist, b_bins = np.histogram(b, bins=256, range=[0, 1])

fig = plt.figure(figsize=(500, 8), dpi=100, layout='constrained')

ax = fig.add_subplot(2, 1, 1)


ax.imshow(image, aspect="auto", interpolation='none')

ax2 = fig.add_subplot(2, 1, 2)

ax2.plot(r_bins[:-1], r_hist, color="red", alpha=0.7, label="Red")
ax2.plot(g_bins[:-1], g_hist, color="green", alpha=0.7, label="Green")
ax2.plot(b_bins[:-1], b_hist, color="blue", alpha=0.7, label="Blue")

fig.savefig("/tmp/gradient_image2.svg")

It may also be work checking if mplcairo can render these

We already have some checks in the c++ code to make sure we do not try to render an output that is so big we segfault.

Trying to render this with mplcairo fails hard:

[truncated]

File ~/.virtualenvs/bleeding/lib/python3.13/site-packages/mplcairo/base.py:365, in FigureCanvasCairo.print_png(self, path_or_stream, dryrun, metadata, pil_kwargs, **kwargs)
    362 def print_png(self, path_or_stream, *,
    363               dryrun=False, metadata=None, pil_kwargs=None, **kwargs):
    364     _check_print_extra_kwargs(**kwargs)
--> 365     img = self._get_fresh_straight_rgba8888()
    366     if dryrun:
    367         return

File ~/.virtualenvs/bleeding/lib/python3.13/site-packages/mplcairo/base.py:345, in FigureCanvasCairo._get_fresh_straight_rgba8888(self)
    344 def _get_fresh_straight_rgba8888(self):
--> 345     renderer = self.get_renderer()
    346     renderer.clear()
    347     with _LOCK:

File ~/.virtualenvs/bleeding/lib/python3.13/site-packages/mplcairo/base.py:221, in FigureCanvasCairo.get_renderer(self, cleared)
    219     renderer = last_renderer
    220 else:
--> 221     renderer = GraphicsContextRendererCairo(*args)
    222     self._last_renderer_call = args, renderer
    223 if cleared:  # matplotlib#22245 (<3.6).

File ~/.virtualenvs/bleeding/lib/python3.13/site-packages/mplcairo/base.py:63, in GraphicsContextRendererCairo.__init__(self, width, height, dpi)
     60 def __init__(self, width, height, dpi):
     61     # Hide the overloaded constructors used by from_pycairo_ctx and
     62     # _for_fmt_output.
---> 63     _mplcairo.GraphicsContextRendererCairo.__init__(
     64         self, width, height, dpi)
     65     RendererBase.__init__(self)

ValueError: invalid value (typically too big) for the size of the input (surface, pattern, etc.)
Error in callback <function _draw_all_if_interactive at 0x7f1d10094900> (for post_execute):

I suspect we need to push down the maximum output size in the Agg backends as "does not segfault" is not stringent enough.

I tried saving the result as an SVG, but I still encounter the issue when opening it in a browser.

I've created this other issue, that was closed as duplicate of this issue.

So I'm wondering what's the resolution status for this bug, since its been a long time without news.

I've tried some of the mentioned approaches above but it didn't work.

Agg: Use 32-bit scan line classes

Navigation Menu

Repository navigation

[Bug]: Long audio files result in incomplete spectrogram visualizations #26368

Bug summary

Code for reproduction

Actual outcome

Expected outcome

Additional information

Operating system

Matplotlib Version

Matplotlib Backend

Python version

Jupyter version

Installation

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: Long audio files result in incomplete spectrogram visualizations #26368

Description

Bug summary

Code for reproduction

Actual outcome

Expected outcome

Additional information

Operating system

Matplotlib Version

Matplotlib Backend

Python version

Jupyter version

Installation

Activity

ksunden commented on Jul 21, 2023

jklymak commented on Jul 21, 2023

djkcyl commented on Jul 21, 2023

djkcyl commented on Jul 21, 2023

djkcyl commented on Jul 21, 2023

scottshambaugh commented on Jul 24, 2023

jklymak commented on Jul 24, 2023

scottshambaugh commented on Jul 24, 2023

scottshambaugh commented on Jul 24, 2023

jklymak commented on Jul 24, 2023

tacaswell commented on Jul 24, 2023

djkcyl commented on Jul 25, 2023

estebanuri commented on Apr 30, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions