Skip to content

[Bug]: Matplotlib 3.5 breaks unyt integration of error bars #21669

Closed

Description

@neutrinoceros

Bug summary

Discovered this in unyt's daily CI jobs: https://github.com/yt-project/unyt/actions

Code for reproduction

import matplotlib.pyplot as plt
import unyt as un

un.matplotlib_support.enable()
fig, ax = plt.subplots()

x = un.unyt_array([1, 2], "cm")
y = un.unyt_array([3, 4], "kg")
yerr = un.unyt_array([0.1, 0.2], "kg")
ax.errorbar(x, y, yerr=(yerr, yerr))

Actual outcome

UnitOperationError: The <ufunc 'subtract'> operator for unyt_arrays with units "kg" 
(dimensions "(mass)") and "dimensionless" (dimensions "1") is not well defined.
detailed raceback
╭──────────────────────────── Traceback (most recent call last) ────────────────────────────╮
│                                                                                           │
│ /private/tmp/matplotlib/t.py:10 in <module>                                               │
│                                                                                           │
│    7 x = un.unyt_array([1, 2], "cm")                                                      │
│    8 y = un.unyt_array([3, 4], "kg")                                                      │
│    9 yerr = un.unyt_array([0.1, 0.2], "kg")                                               │
│ ❱ 10 ax.errorbar(x, y, yerr=yerr)                                                         │
│   11                                                                                      │
│ /private/tmp/matplotlib/lib/matplotlib/__init__.py:1348 in inner                          │
│                                                                                           │
│   1345 │   @functools.wraps(func)                                                         │
│   1346 │   def inner(ax, *args, data=None, **kwargs):                                     │
│   1347 │   │   if data is None:                                                           │
│ ❱ 1348 │   │   │   return func(ax, *map(sanitize_sequence, args), **kwargs)               │
│   1349 │   │                                                                              │
│   1350 │   │   bound = new_sig.bind(ax, *args, **kwargs)                                  │
│   1351 │   │   auto_label = (bound.arguments.get(label_namer)                             │
│                                                                                           │
│ /private/tmp/matplotlib/lib/matplotlib/axes/_axes.py:3474 in errorbar                     │
│                                                                                           │
│   3471 │   │   │   │   │   │   xo, yo, marker='|', **eb_cap_style))                       │
│   3472 │   │                                                                              │
│   3473 │   │   if yerr is not None:                                                       │
│ ❱ 3474 │   │   │   lower, upper = extract_err('y', yerr, y, lolims, uplims)               │
│   3475 │   │   │   barcols.append(self.vlines(                                            │
│   3476 │   │   │   │   *apply_mask([x, lower, upper], everymask), **eb_lines_style))      │
│   3477 │   │   │   # select points without upper/lower limits in y and                    │
│                                                                                           │
│ /private/tmp/matplotlib/lib/matplotlib/axes/_axes.py:3434 in extract_err                  │
│                                                                                           │
│   3431 │   │   │   │   │   f"'{name}err' (shape: {np.shape(err)}) must be a scalar "      │
│   3432 │   │   │   │   │   f"or a 1D or (2, n) array-like whose shape matches "           │
│   3433 │   │   │   │   │   f"'{name}' (shape: {np.shape(data)})") from None               │
│ ❱ 3434 │   │   │   return data - low * ~lolims, data + high * ~uplims  # low, high        │
│   3435 │   │                                                                              │
│   3436 │   │   if xerr is not None:                                                       │
│   3437 │   │   │   left, right = extract_err('x', xerr, x, xlolims, xuplims)              │
│                                                                                           │
│ /Users/robcleme/.pyenv/versions/3.9.7/envs/tmp_unit/lib/python3.9/site-packages/unyt/arra │
│ y.py:1764 in __array_ufunc__                                                              │
│                                                                                           │
│   1761 │   │   │   │   │   │   │   │   else:                                              │
│   1762 │   │   │   │   │   │   │   │   │   raise UnitOperationError(ufunc, u0, u1)        │
│   1763 │   │   │   │   │   │   else:                                                      │
│ ❱ 1764 │   │   │   │   │   │   │   raise UnitOperationError(ufunc, u0, u1)                │
│   1765 │   │   │   │   │   conv, offset = u1.get_conversion_factor(u0, inp1.dtype)        │
│   1766 │   │   │   │   │   new_dtype = np.dtype("f" + str(inp1.dtype.itemsize))           │
│   1767 │   │   │   │   │   conv = new_dtype.type(conv)                                    │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
UnitOperationError: The <ufunc 'subtract'> operator for unyt_arrays with units "kg" 
(dimensions "(mass)") and "dimensionless" (dimensions "1") is not well defined.

Expected outcome

This is the output figure with matplotlib 3.4.3
unyt_err

Additional information

The regression bisects to #19526 (8cd22b4)

I note that if replace the last line with

ax.errorbar(x, y, yerr=yerr)

(passing a single array instead of a tuple), it doesn't break on the main branch.

Operating system

Unbuntu

Matplotlib Version

3.5.0

Matplotlib Backend

not relevant (I get an identical error with the 'agg' backend)

Python version

tested on 3.7 to 3.9

Jupyter version

No response

Installation

pip

Activity

  1. added this to the v3.5.1 milestone on Nov 18, 2021
  2. neutrinoceros commented on Nov 18, 2021

    @neutrinoceros
    ContributorAuthor

    (updated because I saw that that my script was overly simplified to a point that it was actually not broken on the main branch)

  3. neutrinoceros commented on Nov 18, 2021

    @neutrinoceros
    ContributorAuthor

    I think I got a fix, will open a PR shortly !

  4. neutrinoceros commented on Nov 18, 2021

    @neutrinoceros
    ContributorAuthor

    After digging a bit I'm actually stuck on this comment:

    # Casting to object arrays preserves units.

    1. I don't know where this idea comes from
    2. it's not (currently ?) true for unyt arrays

    It seems crucial to understand why this assumption was made in matplotlib, and wether it complies with NEP 18 (array_function protocol).
    It is entirely possible at this stage that the actual issue is on the unyt side, because it's not up to date with NEP 18.
    Also note that unyt's integration is still considered experimental (to the best of my knowledge).

  5. anntzer commented on Nov 18, 2021

    @anntzer
    Contributor

    I think(??) I got this belief from pint, but not sure either. OTOH, is there any thing we can do to convert unitful arraylikes to actual arrays that 1) works across unit libs and 2) does not destroy units?
    Possibly, it may make sense for numpy to rule on an expected behavior here...

  6. ngoldbaum commented on Nov 18, 2021

    @ngoldbaum
    Contributor

    If you’re handed an ndarray subclass, I think casting to a base ndarray would work across libraries? Pint doesn’t implement an ndarray subclass so expecting all unit libraries to work like it does may not be a good expectation. Both unyt and astropy implement ndarray subclasses.

    It might also help to have mock unit libraries (or even directly testing extant unit libraries) in matplotlib’s test suite as this kind of change seems to happen relatively often and you’re not depending on downstream libraries to alert you to breakage.

  7. anntzer commented on Nov 18, 2021

    @anntzer
    Contributor

    The case pointed out by @neutrinoceros is already (should?) only done when the input is not a ndarray or subclass thereof.

  8. neutrinoceros commented on Nov 18, 2021

    @neutrinoceros
    ContributorAuthor

    Correct, but the way it's implemented currently behaves weirdly when the input is a tuple of arrays

  9. anntzer commented on Nov 18, 2021

    @anntzer
    Contributor

    Ah, fair enough.

  10. jklymak commented on Nov 18, 2021

    @jklymak
    Member

    It might also help to have mock unit libraries (or even directly testing extant unit libraries) in matplotlib’s test suite as this kind of change seems to happen relatively often and you’re not depending on downstream libraries to alert you to breakage.

    We do have a mock unit library. If downstream libraries want to add new ones (and tests) I'm sure we would entertain that. I'm not sure we will accept testing every possible downstream library that may have decided to add units. OTOH we should specify exactly what we will accept in terms of units so downstream libraries have consistent rules. This has been a problem for years, and no one has fixed it, so we play whack-a-mole. Hopefully there is light at the end of the tunnel with the upcoming NASA grant perhaps supporting a specification and interface overhaul. Please

  11. neutrinoceros commented on Nov 18, 2021

    @neutrinoceros
    ContributorAuthor

    Well unyt isn't new to this game, I think it's maybe the third biggest player after pint and astropy, but clearly we've missed an opportunity to catch this much earlier by testing only against final releases, so I'll try to fix this on our end to try and improve the situation for everyone in the future.

  12. ngoldbaum commented on Nov 18, 2021

    @ngoldbaum
    Contributor

    Ah then I guess a concrete improvement would be a test that directly exercises the tuple of unit arrays case for errorbar. Sorry for the off-the-cuff peanut gallery advice.

  13. anntzer commented on Nov 18, 2021

    @anntzer
    Contributor

    Going back to the specific problem here: is there a way to cast a 2-tuple of unyt arrays into a (2, N) unyt array? (... that is not unyt-specific...)

  14. ngoldbaum commented on Nov 18, 2021

    @ngoldbaum
    Contributor

    @anntzer, perhaps something like this?

    In [1]: import unyt as u
    
    In [2]: d = ([1, 2]*u.g, [1, 2]*u.g)
    
    In [3]: type(d[0])(d)
    Out[3]:
    unyt_array([[1, 2],
                [1, 2]], 'g')
    
    In [4]: import astropy.units as u
    
    In [5]: d = ([1, 2]*u.g, [1, 2]*u.g)
    
    In [6]: type(d[0])(d)
    Out[6]:
    <Quantity [[1., 2.],
               [1., 2.]] g>
    
  15. anntzer commented on Nov 18, 2021

    @anntzer
    Contributor

    Ah, thanks for the suggestion.

  16. neutrinoceros commented on Nov 18, 2021

    @neutrinoceros
    ContributorAuthor

    Note that this doesn't work with two unyt arrays unless they have the exact same unit (not just coercible ones), but other than that this was also my best guess, and it's probably good enough indeed.

  17. neutrinoceros commented on Nov 19, 2021

    @neutrinoceros
    ContributorAuthor

    Update: @ngoldbaum has a working patch for the problem I mentioned in my previous reply here yt-project/unyt#211, so now I have no reserves left to express on this approach

  18. jklymak commented on Nov 19, 2021

    @jklymak
    Member

    Going back to the specific problem here: is there a way to cast a 2-tuple of unyt arrays into a (2, N) unyt array? (... that is not unyt-specific...)

    Do we need the errors to be a 2, N array, versus two N-arrays? I assume the unyt 1-D arrays cast, so isn't the solution from our side to just have errlo and errhi?

  19. anntzer commented on Nov 19, 2021

    @anntzer
    Contributor

    Technically, the docs state that yerr can be a (2, N) array-like which I think means "can be cast to a (2, N) array" (and it should be possible to cast a tuple[1d_unyt_array, 1d_unyt_array] into that). Trying to unpack immediately into errlo, errhi is probably also possible, but just requires much more wrangling than a straightforward np.broadcast_to(err, (2, N))...

  20. added
    Release criticalFor bugs that make the library unusable (segfaults, incorrect plots, etc) and major regressions.
    on Dec 1, 2021
  21. added 2 commits that reference this issue on Dec 8, 2021
    783c822
    2024498
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Release criticalFor bugs that make the library unusable (segfaults, incorrect plots, etc) and major regressions.topic: units and array ducktypes

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions