Skip to content

imshow padding around NaN values #18735

Closed

Description

@benjimin

There seems to be a discrepancy in how matplotlib's imshow displays rasters which contain NaN areas. The NaN areas of the raster appear larger than they really are.

I assume the problem might be that the raw raster gets resampled/interpolated during display (but before converting to RGBA) and the NaN values propagate during that interpolation (since the ordinary mean of a list containing at least one nonfinite value is also not finite). If so, a fix would be to ensure conversion to RGBA always precedes any image resampling/rescaling.

Here is an example. The first subplot shows a float32 raster with values of zero and one. The second subplot exactly replaces the nonzero pixels with NaN, however the resulting shapes look different (making narrow non-NaN features invisible). The third subplot combines both, showing that the non-NaN area (red) is eroded, resulting in a black outline.

The expected correct behaviour would be that the red layer perfectly covers the black layer, leaving no border visible.

import matplotlib.pyplot as plt, numpy as np

raster = np.full(shape=[200]*2, fill_value=np.nan, dtype=np.float32) # example image
raster[100:150, 100:150] = 0 
for i in range(10):
    raster[20*i:20*i+i, :] = 0 # lines of varying thickness
    
plt.figure(figsize=(8,8)) # small figure

plt.subplot(1,3,1).imshow(np.isnan(raster).astype(np.float32)) # 0 vs 1
plt.subplot(1,3,2).imshow(raster)                              # 0 vs NaN
 
ax = plt.subplot(1,3,3)
ax.imshow(np.isnan(raster), cmap='gray') # black border/underlayer
ax.imshow(raster, cmap='autumn')         # red interior

image

Tested on matplotlib 3.2.1, in a jupyter 1.0.0 sandbox using ipykernel.pylab.backend_inline backend, ipykernel 5.3.4, python 3.6.9, Linux 4.14 amzn2 x86_64.

Activity

  1. jklymak commented on Oct 15, 2020

    @jklymak
    Member

    By default we apply an antialiasing filter if subsampling. One consequence is that NaNs will be smeared with the data. You can turn off the antialiasing or you can use more resolution so you don't need to subsample (i.e. in the above you cite a "small figure" so basically this problem goes away for a big figure).

    For more discussion, you can look at https://matplotlib.org/gallery/images_contours_and_fields/image_antialiasing.html

    You could argue that we should interpolate in RGB space, but that leads to colors that are not even in the colormap. I'm personally not sure thats a problem for the anti-aliasing filter, but it is disconcerting to see red in an image that is not supposed to have any red in it.

  2. benjimin commented on Oct 15, 2020

    @benjimin
    Author

    I strongly argue interpolation should always be in RGBA space.

    For example, consider categorical data. Interpolation in the raw data space results in colours that correspond to value ranges that do not exist in the dataset (such as orange in this image below). This causes misleading visualisations.

    def sample(n):
        rnd = np.random.random([n]*2)
        categorical = np.zeros_like(rnd, dtype=np.uint8)     # 49% category A (Blue)
        categorical[rnd > np.linspace(0, 1, n)[:,None]] = 10 # 46% category B (Grey)
        categorical[rnd > 0.95] = 5                          #  5% category C (Green)
        return categorical                                   #  0% category D (Orange)
    
    fig, (ax1, ax2) = plt.subplots(1, 2)
    im1 = ax1.imshow(sample(20), cmap='tab20c')  # coarse features
    im2 = ax2.imshow(sample(100), cmap='tab20c') # fine features
    plt.colorbar(im2, ax=(ax1, ax2), location='bottom')

    image

    Instead if we interpolate in RGBA space, the result will be the same as if we generated the image at a very high resolution, and then viewed that image from a distance. This is exactly what we should want to see.

    We can get the desired behaviour by using imsave to convert to RGBA in advance:

    import io, matplotlib.image
    array = sample(100)
    buffer = io.BytesIO()
    plt.imsave(buffer, array, cmap='tab20c') # pre-convert to colour space
    buffer.seek(0)
    
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(3, 3))  # arbitrarily small figure
    im1 = ax1.imshow(array, cmap='tab20c')            # broken interpolation
    im2 = ax2.imshow(matplotlib.image.imread(buffer)) # correct interpolation
    plt.colorbar(im1, ax=(ax1, ax2), location='bottom')

    image

    If we need to interpolate, we want the image to look the same as if we were looking at a high resolution version from closer or further away. Zooming in and out (or changing resolution) shouldn't create perceptually different colours.

  3. jklymak commented on Oct 15, 2020

    @jklymak
    Member

    ping @tacaswell who is the current custodian behind how we currently do it.

  4. tacaswell commented on Oct 15, 2020

    @tacaswell
    Member

    For example, consider categorical data. Interpolation in the raw data space results in colours that correspond to value ranges that do not exist in the dataset (such as orange in this image below).

    If you have categorical data use interpolation='nearest' so that we do not try to interpolate (as unlike continuous data categorical data can not be interpolated). Our default assumption is that the input data to imshow is continuous like (e.g. an image) even if it comes in as integers so the AA default interpolation makes sense.

    Zooming in and out (or changing resolution) shouldn't create perceptually different colours.

    Yes, that is why we resample in data space not in RGBA space. We used to do it that way and it lead to significant artifacts in the continious data case. This image was generated with mpl1.3 (iirc) using viridis (which by design contains no red)

    imshow_issue

    This effect will also happen in the categorical case but because it is getting in wrong in a less obvious way (the average of blue and gray stands out from blue and gray less than orange).

    Similarly, if data that is nan is participating in the computation to get the resampled value, then that value must also be poisoned to be "bad" which leads to the apparent erosion. The size of the eroded area is going to depend on the size of the interpolation kernel, to avoid it use "nearest" or "none" (which again, if you have categorical data you should be doing anyway).

    We can get the desired behaviour by using imsave to convert to RGBA in advance:

    You can also call the color map directly rather than having to go through disk:

    import matplotlib.cm as mcm
    my_cm = mcm.get_cmap('tab10', lut=10)
    rgba_catagorical = my_cm(data.astype(int))

    which is faster (no I/O) and does not do any resampling of any sort under the hood for you.

  5. brunobeltran commented on Oct 15, 2020

    @brunobeltran
    Contributor

    Just to put my opinion here, I would argue that antialiasing is an image-space operation. If it introduces artifacts that the user does not like, then they don't want antialiasing, they want data interpolation.

    That said, to my (naive) eyes, having not thought about this problem nearly as much as many other people here, the best default behavior would seem to be to alias in norm'd-but-not-cmap'd space.

  6. jklymak commented on Oct 15, 2020

    @jklymak
    Member

    OK, there are two conflicting desires here, but they both are correct. But if we don't delineate them we will continue to argue in circles.

    Data smoothing:

    The example in @tacaswell comment is smoothing the data - the user has coarse data and they would like it to not look "blocky". In this instance I agree that smoothing in data space is absolutely correct. Whether Matplotlib should do this, or whether the user should do this before they send to matplotlib is a reasonable question, but currently we do it, and so be it.

    The user may also want to smooth in data space to reduce subsampling artifacts if they know the data will be subsampled in their output image. i.e. they have 50,000x50,000 data points, they should probably consider smoothing the data in some way before trying to publish it. Again, I'm not really sure matplotlib should do this, but...

    Image resampling

    We currently smooth and resample in one step. However, all the above could be done without changing the size of the data, and a post-colormap resampling step be added that resamples an image in RGBA space. While this is "image processing" and could also be argued that it is not something Matplotlib should do, the fact of the matter is that we output raster data at certain dpi, and hence always need to resample.

    So, in the plots above, the red regions are egregious if we are smoothing blocky data. Of course we don't want that. However, if we are subsampling, and those are individual pixels that you have zoomed in on, then they are probably perceptually correct.

    So, a possible algorithm would be

    1. smooth the data with a smoothing kernel, but keep the data the same size.
    2. map to RGBA
    3. subsample the RGBA image with a different kernel that does anti-aliasing.

    i.e. imshow(X, interpolation='nearest', rgba_subsample='hanning') would give the user above what they want, and
    imshow(X, intepolation='sinc', rgba_subsample='nearest') would be what we currently do.

    The cost would be potentially making a very big RGBA image and two interpolation steps, and of course extra documentation.

  7. tacaswell commented on Oct 15, 2020

    @tacaswell
    Member

    To be very concrete, averaging / interpolating in RGB space produces colors that are not in the color map:

    averaging

    import matplotlib.cm as mcm
    import matplotlib.pyplot as plt
    from matplotlib.patches import Rectangle
    
    fig, ax = plt.subplots(figsize=(6.4, 4.8))
    
    my_cm = mcm.get_cmap("viridis")
    height = .75
    width = 1
    scale_height = .3
    
    for x_loc, colour in zip(
        np.arange(3) * width, [my_cm(0.0), np.mean(my_cm([0.0, 1.0]), axis=0), my_cm(1.0)]
    ):
        ax.add_patch(Rectangle((x_loc, height + scale_height), width, height, color=colour))
    
    for indx, colour in enumerate(my_cm(np.linspace(0, 1, 16))):
        x_loc = 1 + indx * width / 16
        ax.add_patch(Rectangle((x_loc, scale_height), width/16, .5, color=colour))
    
    for x_loc, colour in zip(np.arange(3) * width, my_cm([0, 0.5, 1])):
        ax.add_patch(Rectangle((x_loc, 0), width, height, color=colour))
    
    ax.set_xlim(0, 3)
    ax.set_ylim(-.3, 2*height + scale_height + .3)
    ax.axis('off')
    ax.axhline(height + scale_height / 2, color='k')
    ax.text(1.5, -.3, "Average in data space (correct)", ha='center', va='bottom', fontsize=20)
    ax.text(1.5, 2*height + scale_height + .3, "Average in RGB space (wrong)", ha='center', va='top', fontsize=20)
    
    fig.savefig('/tmp/averaging.png')

    If we had a checker-board of 0, 1, and then we down-sampled by averaging in 2x2 grids, and put a color bar on the image: should the image be a uniform reddish-brown or a uniform teal-ish?

    A version of this code needs to end up in #18487


    @brunobeltran We currently do the resampling on a (linearly normalized) version of the data, the then apply the user norm and then run the result of that through the colormap (because (log(a) + log(b))/2 != log((a + b) / 2) and it is not clear which which order is "right").

  8. jklymak commented on Oct 15, 2020

    @jklymak
    Member

    Great example.

    If we had a checker-board of 0, 1, and then we down-sampled by averaging in 2x2 grids, and put a color bar on the image: should the image be a uniform reddish-brown or a uniform teal-ish?

    Either is wrong with no context. If there is no data that is 0.5 in the dataset, whether its OK to make it be 0.5 depends on the application. Making it brown is equally valid because it is a visual mix of this blue and this yellow.

    Redo the above with RdBu. You get purple in between. Is white more "correct"? Like I'm sympathetic that it is in many situations, but from an antialiasing point of view, it will actually be a disaster.

    averaging

  9. tacaswell commented on Oct 15, 2020

    @tacaswell
    Member

    imshow when you are in the single channel input mode it has the assumption baked into it that the input data is a uniform rectangular sampling of a continuous function who's values are a field (in the math sense aka the Reals). To come back to @benjimin 's issue, categorical values do not satisfy that last condition (cat1 + cat2 is probably not defined and cat1 / 2 is definitely not), however because the values are being encoded as a numpy array, we happy do it (even though it is nonsense) because there is no way to express to Matplotlib that "these are categoricals, you can't add them or divide by the reals!" (being able to express this sort of thing is what @story645 is working on with the CZI funding).

    The whole reason we went down the perceptually uniform colormap path was so that the user gets a reliable understanding of the underlying data, doing RGB interpolation makes most of that moot. In these cases we are generating colors that are not in the color map so it is clear that "something is wrong", I suspect that there is a color map (maybe a pathological one) where the RGB averaging takes you to a valid, but wrong color.

  10. jklymak commented on Oct 15, 2020

    @jklymak
    Member

    @tacaswell, again two different use cases (#18735 (comment)) which should have different interpolation.

  11. anntzer commented on Oct 15, 2020

    @anntzer
    Contributor

    FWIW I agree with the following statement

    Either is wrong with no context. If there is no data that is 0.5 in the dataset, whether its OK to make it be 0.5 depends on the application.

    and not with the following one

    in the single channel input mode it has the assumption baked into it that the input data is a uniform rectangular sampling of a continuous function who's values are a field

  12. tacaswell commented on Oct 15, 2020

    @tacaswell
    Member

    Those are not different things.

    In general we get in a MxN array from the user and need to use that data to color a PxQ set of pixels. In all but the case where the user data size exactly matches the output size (which is the situation we fabricate when you use mimage.imsave or let fig.figimage resize its self) we have to do some resampling.

    To do that re-sampling you can not start on the user side and "push" forward to the result side (because depending on the transform and kernel etc you a naive loop over the input space will map to every point in the output space), you have to start from the result side and "pull" results forward. That is for each pixel in the target space you say "given my target position, the transform between the two spaces, and my kernel I need to pull the following pixels from the input space and do some math on them". In the case of "nearest", you sort out what pixel is nearest to where the output pixel projects back to and grab that value. In the case of the non-trivial kernels you sort out where the target pixel maps back to then grab the window of input pixels around that and produce a value. This process is mathematically identical if the target space is bigger or smaller than the source space (but in one case we call it upsampling and in the other smoothing).

    The default interpolation mode when down-sampling prevents getting aliases patterns when you have features in the data that are beating against the output pixels by smoothing at the size of the kernel (and we probably should scale the kernel radius based on the pixel size as even with the current version of AA you can construct a pathological dataset that will show the beating).


    If 0.5 is not a valid value in the data set, then the RGB average still maps to something that is not in the input data ( and is not in the colormap so it also is not valid for the output space).


    @anntzer If you do not agree that those are the assumptions baked into imshow, what do you think the assumptions on the data are? I do not disagree it gets used for things that do not quite match those assumptions, but that is how we treat the data inside.

  13. anntzer commented on Oct 15, 2020

    @anntzer
    Contributor

    Just to be clear, I am only talking about the downsampling case here; I think we are all in agreement in the upsampling case.

    • We explicitly support categorical colormaps: there is a large amount of code/complexity in colorbar.py mostly present to make them work. In the case of catgorical colormaps your assumptions completely break down (as you stated yourself).

    • Actually I personally rarely use categoricals, so let's look at the continuous case. Your argument seems to be that mapping to a value that is both not in the colormap nor in the input data is worse than mapping to a value not in the input data, but still in the colorbar. As stated by @jklymak above, I believe this is highly application-dependent and not always true (and it is incumbent on the user to pick the right colormap and sampling level to make the display suitably reflect the data). To reuse the example of the RdBu colormap used by @jklymak (assume white = zero), one could well be happy to be able to distinguish between white (a region where the values are really actually close to zero) and purple (a region where the values vary quickly between negative and positive values). Or perhaps one does want to treat such a mixture of values as if it was a flat region zero (white) -- it depends on the application.

  14. tacaswell commented on Oct 15, 2020

    @tacaswell
    Member

    colorbar applies to more than images (scatter, contours, pcolormesh, etc), my point is only about the assumptions on the input data to imshow not to all Colormappable objects. If the OP had used something in the pcolor family we would not have had this report :)

    import matplotlib.pyplot as plt, numpy as np
    
    raster = np.full(shape=[200]*2, fill_value=np.nan, dtype=np.float32) # example image
    raster[100:150, 100:150] = 0 
    for i in range(10):
        raster[20*i:20*i+i, :] = 0 # lines of varying thickness
        
    plt.figure(figsize=(8,8)) # small figure
    
    plt.subplot(1,3,1).pcolormesh(np.isnan(raster).astype(np.float32)) # 0 vs 1
    plt.subplot(1,3,2).pcolormesh(raster)                              # 0 vs NaN
     
    ax = plt.subplot(1,3,3)
    ax.pcolormesh(np.isnan(raster), cmap='gray') # black border/underlayer
    ax.pcolormesh(raster, cmap='autumn')         # red interior

    pcolormesh

    def sample(n):
        rnd = np.random.random([n]*2)
        categorical = np.zeros_like(rnd, dtype=np.uint8)     # 49% category A (Blue)
        categorical[rnd > np.linspace(0, 1, n)[:,None]] = 10 # 46% category B (Grey)
        categorical[rnd > 0.95] = 5                          #  5% category C (Green)
        return categorical                                   #  0% category D (Orange)
    
    fig, (ax1, ax2) = plt.subplots(1, 2)
    im1 = ax1.pcolormesh(sample(20), cmap='tab20c')  # coarse features
    im2 = ax2.pcolormesh(sample(100), cmap='tab20c') # fine features
    plt.colorbar(im2, ax=(ax1, ax2), location='bottom')

    pcolormesh2

    I would argue that in addition to dealing with the initial nan erosion, it also gives significantly crisper looking plots as the edges of the boxes are not rounded off and the halo around the edges of the first image is gone.


    BoundryNorm is also a bit of a red-herring because it is applying the categories after we have done the re-sampling and might be a better argument of why it needs to be resample -> norm -> colormap than LogNorm.


    Relying on a colormap and interpolation implementation details to detect noisy parts of your data seems cute but super brittle...

    averaging

  15. 2 remaining items

  16. jklymak commented on Oct 15, 2020

    @jklymak
    Member

    Again, great example! But the bottom right one is what imagemagic would give you if you gave it a high-res version, and it is the true perceptual blend of those colors:

    e.g. (I made a 1000-dpi 8x8 image and then ran convert Boo.png -resize 200x200 Magick50.png)
    Magick50

    imagemagick of course uses an anti-aliasing filter. I'm suggesting that we could add an anti-aliasing step that is equivalent.

  17. jklymak commented on Oct 15, 2020

    @jklymak
    Member

    At the minumium this whole issue could be added to the discussion at https://matplotlib.org/devdocs/gallery/images_contours_and_fields/image_antialiasing.html Certainly the trick of applying the norm and filtering RGBA is a cool one I didn't realize would work and could be documented there.

  18. tacaswell commented on Oct 16, 2020

    @tacaswell
    Member

    and it is the true perceptual blend of those colors:

    I would have called this "a data destroying artifact due to down-sampling the after rendering" ;)


    @benjimin Thank you for asking an interesting question, hopefully the high volume and intensity of this discussion has not been offputting ;)

  19. jklymak commented on Oct 16, 2020

    @jklymak
    Member

    I would have called this "a data destroying artifact due to down-sampling the after rendering" ;)

    All subsampling destroys data, by definition.

  20. jklymak commented on Oct 16, 2020

    @jklymak
    Member

    From the example in #18749, using RdBu colormap - obviously the purple colors are not in the colormap:

    sphx_glr_image_antialiasing_007

    PixelPeep

  21. brunobeltran commented on Oct 16, 2020

    @brunobeltran
    Contributor

    This thread seems to be full of people that agree with each other presenting increasingly clever examples of when one of antialiasing or data interpolation works better than the other.

    I largely agree with what @jklymak is saying, which, as I read it, is simply that we should just let the user choose whether to do any interpolation before cmap, after, neither, or both. I understand that there needs to be a default, but @tacaswell is there anything wrong with offering both options in principle? Obviously both are useful in different domains.

    My only point earlier was that if we do offer both options...only one of these should be called antialiasing....

  22. tacaswell commented on Oct 16, 2020

    @tacaswell
    Member

    We can not not interpolate (except for the vanishingly small case when the size of the image exactly matches the size on the screen). Interpolating to before color mapping leads to technically correct (based on the assumptions that imshow has baked into it) but in some cases undesirable effects. Interpolating after color mapping leads colors being in the image that are not in the color map, which I think violates a rather core aspect of using color mappables (that you can go from the color -> colorbar -> value).

    I posted an image in #18749 , if you use interpolation='nearest' with the disk example you get both no white ring and no purple pixels which seems strictly better to me.

    I am 👍 on documenting that you can do the color mapping yourself, hand imshow an RGBA array and then it will do the interpolation in RGBA space, however I am still very skeptical that you ever actually want this (because you can get the same or better effect some other way).

  23. jklymak commented on Oct 16, 2020

    @jklymak
    Member

    that you can go from the color -> colorbar -> value

    You lose that ability as soon as you subsample. You can't go back to the original data from the image.

    So you have two choices: do you want the "fake" pixel to look like it has a certain data value between the adjacent original data? Or do you want the "fake" pixel to look like its between the two (or more) colors from the upsampled image that make it up? Both are completely valid.

    In terms of looking things up in colormaps, If I have red/blue/red/blue, making the picture purple makes it pretty clear to me that the data is a combination of many red-blues all smeared together. If I make the field all white, I might naturally assume my data is all zero:

    fig, axs = plt.subplots(2, 2, figsize=(5.5, 4.3), sharex=False, sharey=False,
                            constrained_layout=True)
    
    aa = np.ones_like(a)
    aa[::2, :] = -1
    
    norm = mcolors.Normalize(vmin=-1.2, vmax=1.2)
    cmap = cm.RdBu_r
    a_rgba = cmap(norm(aa))
    
    axs = axs.flatten()
    
    axs[0].imshow(aa, interpolation='nearest', cmap='RdBu_r', vmin=-1.2, vmax=1.2)
    axs[0].set_title('Zoomed')
    axs[0].set_xlim([0, 65])
    axs[0].set_ylim([0, 45])
    
    axs[1].imshow(aa, interpolation='nearest', cmap='RdBu_r', vmin=-1.2, vmax=1.2)
    axs[1].set_title('No antialiasing')
    
    axs[2].imshow(aa, interpolation=interp, cmap='RdBu_r', vmin=-1.2, vmax=1.2)
    axs[2].set_title('Data antialiasing')
    pc = axs[3].imshow(a_rgba, interpolation=interp)
    axs[3].set_title('RGBA antialiasing')

    ZoomAntiAlias

    If you zoom on the white figure, imagine your surprise when solid blue and red stripes emerge, whereas its not at all surprising when they emerge on the purple figure. Which one tells you more about the (hidden) aspects of your subsampled image?

    however I am still very skeptical that you ever actually want this

    This is basic image anti-aliasing and downsampling, which every image-editing software package in the world does. It is a very common thing to want to do when subsampling an image. You can readily claim its not always advisable and I agree, particularly when dealing with quantitative data, but people clearly "actually want this".

  24. linked a pull request that will close this issueENH: allow image to interpolate post RGBA #18782on Oct 21, 2020
  25. jklymak commented on Oct 21, 2020

    @jklymak
    Member

    @benjimin #18782 is meant to allow you to get what you want directly.

  26. added this to the v3.5.0 milestone on Aug 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions