Behavior of masked values in colormap plot has changed in 2.2.0

Bug report

Bug summary

Something changed the behavior of masked values in colormaps. Before, masked values on a 'jet' colormap would be drawn as white (and could be set to different colors with set_bad). This is no longer the case on matplotlib 2.2.0 - the masked values are being printed out as blue, the same color as valid values, and set_bad has no effect. Verified former behavior to be working on 2.1.1 and 2.0.2.

Not sure this is an issue or expected behavior, but I couldn't find a description of this behavior change anywhere. Please close if this is expected.

Changing interpolation from gaussian to none makes no difference (the sample code was originally used for another bug)

Code for reproduction

import matplotlib.pyplot as plt
import matplotlib.colors

heatmap = [[0] * 32 for _ in range(128)]
cnt = 0
last = 1
for lvl in range(127):
    if lvl % 20 < 10:
        for i in range(32):
            heatmap[lvl][i] = 40
    else:
        for i in range(32):
            heatmap[lvl][i] = 0
jet = plt.get_cmap('jet')
jet.set_bad('white', 1)

plt.imshow(heatmap, origin='lower', interpolation='gaussian', aspect='auto', cmap=jet,
                 norm=matplotlib.colors.LogNorm())
plt.show()

Actual outcome

Expected outcome

Matplotlib version

Operating system: Windows
Matplotlib version: 2.2.0
Matplotlib backend (print(matplotlib.get_backend())): TkAgg
Python version:3.6.3
Jupyter version (if applicable):
Other libraries:

Installed from pip

From #9961, I tried using LogNorm(vmin=1, vmax=5), and I got:

ln(1) -> 0
ln(.5) -> -0.5306 ...
ln(np.nan) -> nan
ln(0) -> masked

But with LogNorm()
ln(1) -> 0
ln(.5) -> 0
ln(np.nan) -> 0
ln(0) -> masked

Numpy version 1.13

To recap (some tests were done in #9961): If LogNorm sets vmin != vmax, the graph is plotted correctly:

If the vmin == vmax == None (i.e. LogNorm()), the shown effect of a block of blue with no stripes is seen.

class LogNorm(Normalize):
    """
    Normalize a given value to the 0-1 range on a log scale
    """
    def __call__(self, value, clip=None):
        if clip is None:
            clip = self.clip

        result, is_scalar = self.process_value(value)

        result = np.ma.masked_less_equal(result, 0, copy=False)

        self.autoscale_None(result)
        vmin, vmax = self.vmin, self.vmax
        if vmin > vmax:
            raise ValueError("minvalue must be less than or equal to maxvalue")
        elif vmin <= 0:
            raise ValueError("values must all be positive")
        elif vmin == vmax:
            result.fill(0)

The problem is coming from the last line above, which is throwing away the mask in the special case where there is only a single valid value in the array. I will submit a PR for it.

Wrong! Investigating...

@tacaswell

The problem is that all the complex rescaling in the resampling code in imshow probably should be skipped for the special case where there is only a single valid value in the array. @tacaswell, sorry, but I think I had better leave the implementation of that special case to you.

This issue has been marked "inactive" because it has been 365 days since the last comment. If this issue is still present in recent Matplotlib releases, or the feature request is still wanted, please leave a comment and this label will be removed. If there are no updates in another 30 days, this issue will be automatically closed, but you are free to re-open or create a new issue if needed. We value issue reports, and this procedure is meant to help us resurface and prioritize issues that have not been addressed yet, not make them disappear. Thanks for your help!

I cannot reproduce this problem with the specified versions. However I can reproduce on main, and a bisect points to the backport of #20511 in 3.4.3. That change does modify LogNorm to remove an accidental modification of the internal data. However, at this point in main, the LogNorm is created out of the LogScale and doesn't have a class-specific autoscale_None that can be overridden. But, having written that change, I still think it is correct.

So in the end, I do have to wonder if imshow is applying the norm parameter correctly.

@efiring

After some further investigation, I think @efiring was on the right track above. With the refactoring of LogNorm into a LogScale+Normalize derivative, the location of the problem has changed a bit, but it's still about the same:

In [1]: from matplotlib.colors import LogNorm

In [2]: LogNorm(vmin=10, vmax=11)([0, 9, 10, 11, 12, 13])
Out[2]: 
masked_array(data=[--, -1.1054487136015845, 0.0, 1.0, 1.912928473834252, 2.752741260231958],
             mask=[True, False, False, False, False, False],
       fill_value=1e+20)

In [3]: LogNorm(vmin=10, vmax=10)([0, 9, 10, 11, 12, 13])
Out[3]: 
masked_array(data=[0., 0., 0., 0., 0., 0.],
             mask=False,
       fill_value=1e+20)

When vmin != vmax, anything outside produces something outside the 0-1 range expected for a norm and negative numbers for LogNorm are masked. But when they're equal, then only zeros are output, due to:

matplotlib/lib/matplotlib/colors.py

Lines 1712 to 1713 in 5a89bdb

    
           if self.vmin == self.vmax: 
        
               return np.full_like(value, 0)

This can be fixed by something like:

diff --git a/lib/matplotlib/colors.py b/lib/matplotlib/colors.py
index 434bb54235..94a50e1d90 100644
--- a/lib/matplotlib/colors.py
+++ b/lib/matplotlib/colors.py
@@ -1709,8 +1709,6 @@ def _make_norm_from_scale(
                 self.autoscale_None(value)
             if self.vmin > self.vmax:
                 raise ValueError("vmin must be less or equal to vmax")
-            if self.vmin == self.vmax:
-                return np.full_like(value, 0)
             if clip is None:
                 clip = self.clip
             if clip:
@@ -1720,8 +1718,11 @@ def _make_norm_from_scale(
             if not np.isfinite([t_vmin, t_vmax]).all():
                 raise ValueError("Invalid vmin or vmax")
             t_value -= t_vmin
-            t_value /= (t_vmax - t_vmin)
-            t_value = np.ma.masked_invalid(t_value, copy=False)
+            if self.vmin == self.vmax:
+                t_value[t_value > 0] += 1
+            else:
+                t_value /= (t_vmax - t_vmin)
+                t_value = np.ma.masked_invalid(t_value, copy=False)
             return t_value[0] if is_scalar else t_value
 
         def inverse(self, value):

(though I'm not sure if the last line should be in or out of the else.)

However, fixing that does not fix the image. This appears to be due to this extra clipping applied at:

matplotlib/lib/matplotlib/image.py

Lines 454 to 473 in 5a89bdb

    
           # Clip scaled data around norm if necessary.  This is necessary 
        
           # for big numbers at the edge of float64's ability to represent 
        
           # changes.  Applying a norm first would be good, but ruins the 
        
           # interpolation of over numbers. 
        
           self.norm.autoscale_None(A) 
        
           dv = np.float64(self.norm.vmax) - np.float64(self.norm.vmin) 
        
           vmid = np.float64(self.norm.vmin) + dv / 2 
        
           fact = 1e7 if scaled_dtype == np.float64 else 1e4 
        
           newmin = vmid - dv * fact 
        
           if newmin < a_min: 
        
               newmin = None 
        
           else: 
        
               a_min = np.float64(newmin) 
        
           newmax = vmid + dv * fact 
        
           if newmax > a_max: 
        
               newmax = None 
        
           else: 
        
               a_max = np.float64(newmax) 
        
           if newmax is not None or newmin is not None: 
        
               np.clip(A_scaled, newmin, newmax, out=A_scaled)

Since a LogNorm will autoscale without the 0, it will tell the image code here that vmin==vmax==40, and then the image code clips to a minimum of 40, losing all the zeros before the norm is even called. I have not tested it, but do also wonder if that means that a norm with vmin/vmax within the array limits will not correctly plot under or over colours.

Is vmin == vmax something we want/need to support? There is ambiguity when calling norm(vmin), should we return 0, 0.5, or 1? It seems like disallowing the case where vmin == vmax would solve a lot of special-casing in other places.

We already apply a nonsingular transform in Colorbar to specifically avoid this case when the norm is on an axis, but I'm wondering if this special-casing should actually be moved upstream to the Normalize class and applied during the autoscaling?

This issue has been marked "inactive" because it has been 365 days since the last comment. If this issue is still present in recent Matplotlib releases, or the feature request is still wanted, please leave a comment and this label will be removed. If there are no updates in another 30 days, this issue will be automatically closed, but you are free to re-open or create a new issue if needed. We value issue reports, and this procedure is meant to help us resurface and prioritize issues that have not been addressed yet, not make them disappear. Thanks for your help!

The topic of singular norms (vmin == vmax) is not consistently solved. Not a high priority, but we should come up with a consistent approach. See also

Make singular colorbars consistent with single-value mappables. #26307 / Inconsistent plot with and without colorbar #15566
[Bug]: setting norm for imshow and not using a colorbar produces error #28555

Navigation Menu

Repository navigation

Behavior of masked values in colormap plot has changed in 2.2.0 #11039

Bug report

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	# Clip scaled data around norm if necessary. This is necessary
	# for big numbers at the edge of float64's ability to represent
	# changes. Applying a norm first would be good, but ruins the
	# interpolation of over numbers.
	self.norm.autoscale_None(A)
	dv = np.float64(self.norm.vmax) - np.float64(self.norm.vmin)
	vmid = np.float64(self.norm.vmin) + dv / 2
	fact = 1e7 if scaled_dtype == np.float64 else 1e4
	newmin = vmid - dv * fact
	if newmin < a_min:
	newmin = None
	else:
	a_min = np.float64(newmin)
	newmax = vmid + dv * fact
	if newmax > a_max:
	newmax = None
	else:
	a_max = np.float64(newmax)
	if newmax is not None or newmin is not None:
	np.clip(A_scaled, newmin, newmax, out=A_scaled)

Behavior of masked values in colormap plot has changed in 2.2.0 #11039

Description

Bug report

Activity

arthur00 commented on Apr 12, 2018

arthur00 commented on Apr 12, 2018

efiring commented on Jul 2, 2018

efiring commented on Jul 2, 2018

efiring commented on Jul 2, 2018

github-actions commented on May 4, 2023

QuLogic commented on May 4, 2023

QuLogic commented on May 4, 2023

greglucas commented on May 5, 2023

github-actions commented on Jul 12, 2024

timhoffm commented on Aug 13, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions