Skip to content

Coerce MxNx1 images into MxN images for imshow - #15090

Merged
tacaswell merged 9 commits into
matplotlib:masterfrom
kmader:patch-1
Nov 1, 2019
Merged

tacaswell merged 9 commits into
matplotlib:masterfrom
kmader:patch-1

Conversation

@kmader

@kmader kmader commented Aug 20, 2019 •

Copy link
Copy Markdown
Contributor

PR Summary

Causes matplotlib.pyplot.imshow to work with depth=1 images (common in deep learning and segmentation) as if they were grayscale rather than throwing an error message (so it should be backward compatible, unless someone expected it to throw an error)

PR Checklist

  • Has Pytest style unit tests
  • Code is Flake 8 compliant
  • New features are documented, with examples if plot related
  • Documentation is sphinx and numpydoc compliant
  • Added an entry to doc/users/next_whats_new/ if major new feature (follow instructions in README.rst there)
  • Documented in doc/api/api_changes.rst if API changed in a backward-incompatible way

Specific Tests

  • shape (n,m) -> colormapping
  • shape(n,m,1) -> colormapping, same as (n,m), newly introduced here.
  • shape(n,m,2) -> Error
  • shape(n,m,3) -> RGB image
  • shape(n,m,4) -> RGBA image
  • shape(n,m,5) -> Error

@kmader

kmader commented Aug 20, 2019

Copy link
Copy Markdown
Contributor Author

I could easily add a unit test to test_image.py, but wasn't sure if it was better to add a new one or piggy back it on another test that already does grayscale images.

@tacaswell tacaswell added this to the v3.3.0 milestone Aug 20, 2019
Comment thread lib/matplotlib/image.py Resolved
@tacaswell

Copy link
Copy Markdown
Member

This is projecting (N, M, 1) input down to (N, M) which means in goes through color mapping code is by default produces an image which is not grayscale. The other option would be to project it up to a (N, M, 3) array where each plane is identical which would run the input through the RGB(A) handling code path which would produce a gray-scale output.

My inclination would be towards the first case, but from your language motivating this change I expected it to be doing the second.

@tacaswell

Copy link
Copy Markdown
Member

Thank you for working on this!

Either way, we will definitely need a test and a whats_new or API change entry.

@jklymak

jklymak commented Aug 20, 2019

Copy link
Copy Markdown
Member

As mentioned in the issue I think it’s confusing for mxnx1 array to behave differently from an mxn array. If anything, I think we should just reshape the mxnx1 array to drop the redundant dimension.

adding @tacaswell input

Co-Authored-By: Thomas A Caswell <tcaswell@gmail.com>
@kmader

kmader commented Aug 20, 2019

Copy link
Copy Markdown
Contributor Author

@tacaswell @jklymak for the usual use cases with tensorflow and pytorch it would make more sense to treat it the same way as the N, M (rather than N, M, 3) since it is often scaled differently. For me the main point would be that all model outputs could be handled the same way rather than having add ifs around every statement
if x.ndim==3 and x.shape[-1]==1 elif x.ndim==3 and x.shape[-1]==3

@kmader kmader changed the title Assume grayscale for depth=1 images Assume scalar value for depth=1 images Aug 20, 2019
adding a test to make sure it works correctly with a 10x10x1 image
@tacaswell

Copy link
Copy Markdown
Member

For your current code, ax.imshow(img.squeeze(),...) should work in most cases (it would behave badly if you have shape (1, N) or (M, 1) input so we can't do that internally, but I suspect that in 99% of image based machine learning workflows that would be ok).

@kmader

kmader commented Aug 20, 2019

Copy link
Copy Markdown
Contributor Author

@tacaswell that is potentially a perfectly acceptable solution.Should I close the PR?
If we close it, I assume (maybe wrongly) that other people might run into this issue, would it make sense to give them more directed feedback on how to use imshow correctly when the input is MxNx1?

@jklymak

jklymak commented Aug 20, 2019

Copy link
Copy Markdown
Member

We reshape arrays in other contexts. I think coercing MxNx1 to MxN is reasonable in this case.

removing whitespace
@kmader kmader changed the title Assume scalar value for depth=1 images Coerce MxNx1 images into MxN images for imshow Aug 20, 2019
adding whats_new document
updating based on feedback from @ImportanceOfBeingErnest
@ImportanceOfBeingErnest

Copy link
Copy Markdown
Member

I think ultimately one would like to test each of the following cases

  1. shape (n,m) -> colormapping
  2. shape(n,m,1) -> colormapping, same as (n,m), newly introduced here.
  3. shape(n,m,2) -> Error
  4. shape(n,m,3) -> RGB image
  5. shape(n,m,4) -> RGBA image
  6. shape(n,m,5) -> Error

It doesn't look like those are all tested currently.

An image comparisson test may make sense for cases 1, 4 and 5. (All of those could be tested within the same figure) For case 2. one may rather compare the output to case 1., using the check_figures_equal decorator. (We do in general try to keep the number of image comparison tests as low as possible.)

@kmader

kmader commented Aug 21, 2019

Copy link
Copy Markdown
Contributor Author

@ImportanceOfBeingErnest I hadn't known about the check_figures_equal decorator that probably makes more sense for the cases. I already added the check that imshow gives the same result with MxNx1 and MxN (case 2). Should I add checks for all the other cases as well?

@ImportanceOfBeingErnest

Copy link
Copy Markdown
Member

That would be the ultimate goal. Up to you if you want to do that work and include it in this PR or not.

changing to check_figures_equal, adding 10x10x2 and 10x10x5 failure cases
@kmader

kmader commented Oct 4, 2019

Copy link
Copy Markdown
Contributor Author

The x3 and x4 cases seem to be already covered in other tests, so that should

@image_comparison(['interp_alpha.png'], remove_text=True)
def test_alpha_interp():
'Test the interpolation of the alpha channel on RGBA images'
fig, (axl, axr) = plt.subplots(1, 2)
# full green image
img = np.zeros((5, 5, 4))
img[..., 1] = np.ones((5, 5))
# transparent under main diagonal
img[..., 3] = np.tril(np.ones((5, 5), dtype=np.uint8))
axl.imshow(img, interpolation="none")
axr.imshow(img, interpolation="bilinear")

def test_imsave_color_alpha():
# Test that imsave accept arrays with ndim=3 where the third dimension is
# color and alpha without raising any exceptions, and that the data is
# acceptably preserved through a save/read roundtrip.
np.random.seed(1)
for origin in ['lower', 'upper']:
data = np.random.rand(16, 16, 4)
buff = io.BytesIO()
plt.imsave(buff, data, origin=origin, format="png")
buff.seek(0)
arr_buf = plt.imread(buff)
# Recreate the float -> uint8 conversion of the data
# We can only expect to be the same with 8 bits of precision,
# since that's what the PNG file used.
data = (255*data).astype('uint8')
if origin == 'lower':
data = data[::-1]
arr_buf = (255*arr_buf).astype('uint8')
assert_array_equal(data, arr_buf)

@kmader

kmader commented Oct 4, 2019

Copy link
Copy Markdown
Contributor Author

@ImportanceOfBeingErnest everything seems fine for me (but I am very inclined to say that), anything else I should add?

@ImportanceOfBeingErnest

Copy link
Copy Markdown
Member

Appart from the flake8 error this looks fine to me.
A general question would be if we want to have this for imshow only. It might be considered as inconsistent to let (n,m,1) arrays be handled by imshow, but not by pcolor[mesh].

removing whitespace
@kmader

kmader commented Oct 4, 2019

Copy link
Copy Markdown
Contributor Author

@ImportanceOfBeingErnest I can change it, it appears to be a completely different mechanism in matplotlib/axes/_axes.py but should be possible to change. We should then probably also add tests that explicitly fail for 10x10x2, 10x10x3, and 10x10x4 images because pcolormesh can only ever work on x,y,1 sized images.

@ImportanceOfBeingErnest

Copy link
Copy Markdown
Member

I guess that can also be another PR; depending on what is easiest. Approving this as it stands.

Comment thread lib/matplotlib/image.py Outdated
"float".format(self._A.dtype))

if (self._A.ndim == 3 and self._A.shape[-1] == 1):
# If just one dimension assume grayscale

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment isn't correct anymore - its not assuming greyscale, its just doing a mapping into the colormap?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as that slipped past me, thanks!

@Zac-HD Zac-HD left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm -0 on this change - I'd prefer to add a more helpful error message with the .squeeze(axis=-1) trick, because IMO building in fixes like this tends to make it harder to develop a mental model of what the arrays should be doing or what counts as valid input. This would be far from the only case though, so it's not a strong objection!

@kmader - add "Closes #15089" to your top-level description, and it will auto-close when this is merged 😄

Comment thread lib/matplotlib/image.py

if (self._A.ndim == 3 and self._A.shape[-1] == 1):
# If just one dimension assume scalar and apply colormap
self._A = self._A[:, :, 0]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self._A = self._A.squeeze(axis=-1)

@tacaswell tacaswell added the Release critical For bugs that make the library unusable (segfaults, incorrect plots, etc) and major regressions. label Oct 30, 2019
@tacaswell

Copy link
Copy Markdown
Member

Pre-emptively labeling as release critical to make sure it does not fall through the cracks.

The discussion at #7221 suggests that at least some users would expect us to go to RGB gray scale rather than color mapping.

@jklymak

jklymak commented Oct 30, 2019

Copy link
Copy Markdown
Member

@tacaswell are you suggesting we hold off on merging it? I'm not sure why MxNx1 would be considered greyscale, but MxN would be considered color mappable. Is this really a particular convention in machine learning? I'm not clear on why that would be the case; I can imagine having a 3-D array MxNxNsamps, but I'd have expected that to be accessed as img[:, :, ind] which is back to MxN.

@timhoffm

timhoffm commented Oct 30, 2019 •

Copy link
Copy Markdown
Member

+/-0 on accepting MxNx1. On the one hand, we should have clear APIs. On the other hand, it may be convenient to accept MxNx1 specifically, if people have this kind of data a lot.

However, if MxNx1 is accepted, it should be handled the same way as MxN. I don't think it's safe to assume that MxNx1 is a grayscale image ("In the face of ambiguity, refuse the temptation to guess."). Let's rather go for consistency than a subtle different behavior, which may be handy for some, but unexpected for others.

@kmader

kmader commented Oct 30, 2019

Copy link
Copy Markdown
Contributor Author

My main reason for adding it was my experience from teaching students newer to python where particularly with segmentation problems students were confused at why MxN and MxNx3 images worked but MxNx1 didn't. I can, however, appreciate that there are few easy decisions.

@jklymak

jklymak commented Oct 30, 2019

Copy link
Copy Markdown
Member

Right but how did they get an mxnx1 matrix in the first place?

@kmader

kmader commented Oct 30, 2019

Copy link
Copy Markdown
Contributor Author

PyTorch, Keras, and Tensorflow, and even simpler tools like https://github.com/Calysto/conx all output binary segmentations (the most common case) as MxNx1

@jklymak

jklymak commented Oct 30, 2019

Copy link
Copy Markdown
Member

Is there a good reason why they do this?

@kmader

kmader commented Oct 30, 2019

Copy link
Copy Markdown
Contributor Author

The main reason (AFAIK) they do this is to make convolutions in convolutional neural networks easier. So all the convolutions have MxNxQ to MxNxR format and binary is just when R=1

@jklymak

jklymak commented Oct 30, 2019

Copy link
Copy Markdown
Member

Cool, so from my point of view there is no special difference between MxNx1 and MxN, so I still think this PR that treats them as the same is a reasonable convenience. But I don't feel strongly about it if there are strong objections.

@NelleV

NelleV commented Oct 31, 2019

Copy link
Copy Markdown
Member

Looks good to me as well. I also asked @jni as someone who works on images if he thought there was a case where we wouldn't want this, and he couldn't find any.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Release critical For bugs that make the library unusable (segfaults, incorrect plots, etc) and major regressions.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Coerce MxNx1 images into MxN images for imshow

7 participants