Skip to content

Speed up RGBA-stage image resampling - #32275

Merged
iccir merged 2 commits into
matplotlib:mainfrom
zekebarger:speed-up-premultiplied-alpha-resampling
Sep 3, 2026
Merged

iccir merged 2 commits into
matplotlib:mainfrom
zekebarger:speed-up-premultiplied-alpha-resampling

Conversation

@zekebarger

@zekebarger zekebarger commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

PR summary

Closes #32240.

Redrawing an imshow image got about 2x slower between 3.10.9 and 3.11. @ayshih traced this to #29776, which moved resampling into premultiplied alpha space. The premultiply and demultiply are both full-array NumPy passes, but in their current form, they can't use NumPy's fast paths.

Demultiply is the more expensive one:

np.divide(res[..., :3], res[..., 3:], out=res[..., :3], where=res[..., 3:] != 0)

This takes ~16 ms per draw in the code I used to reproduce the issue in #32240. Two reasons it's slow: out=res[..., :3] is a stride-4 view (so contiguous inner loops can't be used), and passing an array to where= is itself expensive. Running it on a 462x1560x4 float64 array takes 16.3 ms as written, 8.3 ms without where=, and only 2.8 ms operating on the whole contiguous array. An all-True mask array still costs 2.3-2.8x depending on whether the destination is strided, so the cost just comes from taking the masked path itself.

For both the demultiply and premultiply steps, this PR implements contiguous whole-array operations that save the alpha channel and then restore it.

  • Demultiply: save the alpha values, divide the whole array by a version of alpha where zeros have been replaced by ones, then put the original alpha values back. This is equivalent to the where= form, and while dividing alpha by itself involves more operations, the fact that it runs contiguously means it's much faster.
  • Premultiply: same idea. Making the multiply contiguous takes it from 13.4 ms to 4.6 ms per draw in my reproduction code. On top of that, when the alpha channel is all 1 (as with colormapped or opaque RGBA inputs), it can be skipped entirely for the cost of a 0.6 ms check, taking the line from 13.4 ms to 0.6 ms.

One positive side effect is that the line A = A.copy() is no longer needed. It existed to stop A[..., :3] *= A[..., 3:] from writing into the caller's array, but now A is not modified in place. When alpha is all 1 it's left untouched, and otherwise the multiplication just builds a new array.

Timings

Median of 15 repeats, same code from #32240 (macOS arm64, NumPy 2.5.2)

case 3.10.9 3.11.1 main this PR
scalar, interpolation='none' 26.9 ms 53.1 ms 51.2 ms 23.9 ms
scalar, interpolation='nearest' 27.1 ms 61.7 ms 56.7 ms 28.9 ms
RGBA, interpolation='none' 13.7 ms 34.0 ms 32.9 ms 17.2 ms
RGBA, interpolation='nearest' 13.9 ms 35.2 ms 33.7 ms 17.9 ms

Verification

A new test, test_rgba_stage_demultiplies_resampled_alpha, checks that RGB values survive the premultiply/demultiply round trip for several alpha values and that a fully transparent image never divides by zero.

To double-check that rendered output is unchanged, I rendered a large set of configurations and compared the RGBA canvas buffers against main and against the 3.11.1 wheel. No differences were observed.

Script used for the comparison

Run under each build, then compare:

import sys, numpy as np, matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

rng = np.random.default_rng(1234)
alpha_arr = rng.random((7, 11))
arrays = {
    "scalar_f":    rng.random((7, 11)),
    "scalar_u8":   (rng.random((7, 11)) * 255).astype(np.uint8),
    "scalar_mask": np.ma.masked_array(rng.random((7, 11)), rng.random((7, 11)) < .3),
    "rgb_f":       rng.random((7, 11, 3)),
    "rgba_f":      rng.random((7, 11, 4)),
    "rgba_opaque": np.dstack([rng.random((7, 11, 3)), np.ones((7, 11))]),
    "rgb_u8":      (rng.random((7, 11, 3)) * 255).astype(np.uint8),
    "rgba_u8":     (rng.random((7, 11, 4)) * 255).astype(np.uint8),
    "wide":        rng.random((60, 900)),
}

out = {}
for name, arr in arrays.items():
    for interp in ["none", "nearest", "bilinear", "bicubic", "lanczos", "antialiased"]:
        for alpha in [None, 0.4, "array"]:
            for stage in ["auto", "data", "rgba"]:
                for origin in ["upper", "lower"]:
                    if alpha == "array" and arr.shape[:2] != alpha_arr.shape:
                        continue
                    fig = plt.figure(figsize=(2.37, 1.63), dpi=100)
                    ax = fig.add_axes((0.13, 0.11, 0.7, 0.72))
                    ax.imshow(arr, interpolation=interp, origin=origin, aspect="auto",
                              alpha=alpha_arr if alpha == "array" else alpha,
                              interpolation_stage=stage)
                    fig.canvas.draw()
                    out[f"{name}|{interp}|{alpha}|{stage}|{origin}"] = \
                        np.asarray(fig.canvas.buffer_rgba()).copy()
                    plt.close(fig)

np.savez_compressed(sys.argv[1], **out)
print(f"{len(out)} configurations -> {sys.argv[1]}")
a, b = np.load("before.npz"), np.load("after.npz")
print(sum(np.array_equal(a[k], b[k]) for k in a.files), "of", len(a.files), "identical")

AI Disclosure

I used Claude Code in several places and thoroughly reviewed its output. It profiled 3.10.9 against 3.11.1 to locate the two affected lines, benchmarked the approach implemented here, wrote the patch and the test, and ran the verification. I wrote or revised all prose, reviewed all code, and re-ran the verification script.

PR quality check

  • Use an expressive title
  • New and changed code is tested
  • [N/A] Plotting related features are demonstrated in an example
  • [N/A] New features and API changes have release notes
  • [N/A] Documentation complies with general and docstring guidelines

Premultiplying and unpremultiplying around the resample step both operated
on the strided A[..., :3] view, and the unpremultiply additionally used the
`where=` form of np.divide, which is several times slower than the unmasked
form.  Operate on the whole contiguous array and restore the alpha channel
instead, and skip premultiplication altogether when alpha is uniformly 1.

The rendered output is unchanged.
@github-actions

Copy link
Copy Markdown

Thank you for opening your first PR into Matplotlib!

If you have not heard from us in a week or so, please leave a new comment below and that should bring it to our attention. Most of our reviewers are volunteers and sometimes things fall through the cracks. We also ask that you please finish addressing any review comments on this PR and wait for it to be merged (or closed) before opening a new one, as it can be a valuable learning experience to go through the review process.

You can also join us on discourse chat for real-time discussion.

For details on testing, writing docs, and our review process, please see the developer guide.
Please let us know if (and how) you use AI, it will help us give you better feedback on your PR.

We strive to be a welcoming and open project. Please follow our Code of Conduct.

@ayshih ayshih left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting! I wouldn't have expected there to be such a degree of speed-up by (incorrectly) multiplying the alpha channel by itself and then restoring it.

My primary comment is please change "unpremultiply" to "demultiply" in comments and test name. Aside from that, all of my comments are about cleaning up the writing of your unit test.

Comment thread lib/matplotlib/tests/test_image.py Outdated
axes_image.make_image(None)[0]


def test_rgba_stage_unpremultiplies_resampled_alpha():

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewrite this test using pytest.mark.parametrize() instead of the internal function

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

Comment thread lib/matplotlib/tests/test_image.py Outdated
fig, ax = plt.subplots()
axes_image = ax.imshow(data, interpolation='nearest',
interpolation_stage='rgba')
with np.errstate(divide='raise', invalid='raise'):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the RuntimeWarning would already get converted to an error by pytest, right?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's true, fixed

Comment thread lib/matplotlib/tests/test_image.py Outdated
Comment on lines +1879 to +1880
# Trim the partially covered pixels at the edges of the image.
return image[2:-2, 2:-2]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think your test as written will use nearest-neighbor interpolation, so I don't think trimming the edge is even necessary

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, fixed!

Comment thread lib/matplotlib/tests/test_image.py Outdated
opaque = render(1)
for alpha in (0.25, 0.5, 0.75):
partial = render(alpha)
assert_array_equal(partial[..., :3], opaque[..., :3])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since your arrays (both input and output) are constant in value, you can just check against the expected output value

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Use Agg's "demultiply" terminology, matching src/_image_resample.h, rather
than the invented "unpremultiply".

Rewrite the test with pytest.mark.parametrize, and drop the np.errstate
guard and the edge trimming.  The guard is redundant because the test
suite already turns the RuntimeWarning into an error, and nearest-neighbour
interpolation makes the output a single flat colour, so the edges need no
special handling and the expected value can be asserted directly.
@zekebarger

Copy link
Copy Markdown
Contributor Author

@ayshih Thank you for the review! All comments have been addressed.

@ayshih ayshih left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR looks good to me! I've checked it out on my Windows machine, but it'd be prudent for another reviewer to explicitly confirm the speed-up on a non-Windows x86 processor.

@ayshih ayshih added this to the v3.11.2 milestone Sep 2, 2026
@iccir

iccir commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

macOS 14 on an M2 Ultra (arm64):

main

  scalar array, interp='none'          73.58 ms
  scalar array, interp='nearest'       86.48 ms
  RGBA array,   interp='none'          44.44 ms
  RGBA array,   interp='nearest'       45.88 ms

This PR

  scalar array, interp='none'          32.14 ms
  scalar array, interp='nearest'       40.83 ms
  RGBA array,   interp='none'          22.55 ms
  RGBA array,   interp='nearest'       23.47 ms

The numbers look good, I'll review the code later today!

@iccir
iccir self-requested a review September 3, 2026 16:13

@iccir iccir left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good!

Long term, I think that we should investigate resampling in premultiplied-space (per the "TODO:" comment). This solution makes complete sense and works in the meantime.

Congrats on your first PR to Matplotlib, @zekebarger! We hope to hear from you again! 🥳🎉

@iccir
iccir merged commit c79b59e into matplotlib:main Sep 3, 2026
46 checks passed
@jklymak

jklymak commented Sep 3, 2026

Copy link
Copy Markdown
Member

This is great. I wonder if the speed tests could find its way into mpl_bench ??

QuLogic added a commit that referenced this pull request Sep 4, 2026
…275-on-v3.11.x

Backport PR #32275 on branch v3.11.x (Speed up RGBA-stage image resampling)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: AxesImage redraws ~2x slower in 3.11 than in 3.10

4 participants