Skip to content

plt.hist() fails with TensorFlow Numpy emulation #19574

Open

Description

@FlorinAndrei

Bug report

Bug summary

Generating np.random.randn(1000) values, visualizing them with plt.hist(). Works fine with Numpy.

When I replace Numpy with tensorflow.experimental.numpy, Matplotlib 3.3.4 fails to display the histogram correctly. Matplotlib 3.2.2 works fine.

Code for reproduction

import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
import tensorflow.experimental.numpy as tnp

# bad image
labels1 = 15 + 2 * tnp.random.randn(1000)
_ = plt.hist(labels1)

# good image
labels2 = 15 + 2 * np.random.randn(1000)
_ = plt.hist(labels2)

Actual outcome

np-bad

Expected outcome

np-good

Matplotlib version

  • Operating system: Windows 10
  • Matplotlib version (import matplotlib; print(matplotlib.__version__)): 3.3.4
  • Matplotlib backend (print(matplotlib.get_backend())): module://ipykernel.pylab.backend_inline
  • Python version: 3.8.7
  • Jupyter version (if applicable): see below
  • Other libraries: see below

TensorFlow 2.4.1

jupyter --version
jupyter core     : 4.7.0
jupyter-notebook : 6.1.6
qtconsole        : 5.0.1
ipython          : 7.20.0
ipykernel        : 5.4.2
jupyter client   : 6.1.7
jupyter lab      : not installed
nbconvert        : 6.0.7
ipywidgets       : 7.6.3
nbformat         : 5.0.8
traitlets        : 5.0.5

Python installed from python.org as an exe installer. Everything else is pip install --user

Bug opened with TensorFlow on this same issue:

tensorflow/tensorflow#46274

Activity

  1. jklymak commented on Feb 25, 2021

    @jklymak
    Member

    I think we are happy to help, but we make no guarantee to support all numpy-like objects that exist in the python universe. If you want help with this I suggest figuring out how to reproduce w/o tensorflow so we can advise.

  2. tacaswell commented on Feb 25, 2021

    @tacaswell
    Member

    This looks like a failure of the up-cast to 2D logic (and instead of 1 histogram of N elements we are seeing N histograms of 1 element).

  3. tacaswell commented on Feb 25, 2021

    @tacaswell
    Member

    Specifically this code:

    def _reshape_2D(X, name):
    """
    Use Fortran ordering to convert ndarrays and lists of iterables to lists of
    1D arrays.
    Lists of iterables are converted by applying `numpy.asanyarray` to each of
    their elements. 1D ndarrays are returned in a singleton list containing
    them. 2D ndarrays are converted to the list of their *columns*.
    *name* is used to generate the error message for invalid inputs.
    """
    # unpack if we have a values or to_numpy method.
    try:
    X = X.to_numpy()
    except AttributeError:
    try:
    if isinstance(X.values, np.ndarray):
    X = X.values
    except AttributeError:
    pass
    # Iterate over columns for ndarrays.
    if isinstance(X, np.ndarray):
    X = X.T
    if len(X) == 0:
    return [[]]
    elif X.ndim == 1 and np.ndim(X[0]) == 0:
    # 1D array of scalars: directly return it.
    return [X]
    elif X.ndim in [1, 2]:
    # 2D array, or 1D array of iterables: flatten them first.
    return [np.reshape(x, -1) for x in X]
    else:
    raise ValueError(f'{name} must have 2 or fewer dimensions')
    # Iterate over list of iterables.
    if len(X) == 0:
    return [[]]
    result = []
    is_1d = True
    for xi in X:
    # check if this is iterable, except for strings which we
    # treat as singletons.
    if (isinstance(xi, collections.abc.Iterable) and
    not isinstance(xi, str)):
    is_1d = False
    xi = np.asanyarray(xi)
    nd = np.ndim(xi)
    if nd > 1:
    raise ValueError(f'{name} must have 2 or fewer dimensions')
    result.append(xi.reshape(-1))
    if is_1d:
    # 1D array of scalars: directly return it.
    return [np.reshape(result, -1)]
    else:
    # 2D array, or 1D array of iterables: use flattened version.
    return result

    which recently got re-worked due to changes in how numpy auto-up-casts object / ragged arrays.

  4. FlorinAndrei commented on Feb 25, 2021

    @FlorinAndrei
    Author

    I suggest figuring out how to reproduce w/o tensorflow so we can advise

    The Tensorflow failure is the only case I know, and it's how I ended up reporting this bug. I was just trying to learn the Numpy emulation in Tensorflow.

  5. mwaskom commented on Feb 25, 2021

    @mwaskom

    I think the two problems are:

    1. Matplotlib preprocesses instances of np.ndarray so they are always 2D, with vectors gaining an outer dimension. The tensorflow object doesn't get this preprocessing. (Matplotlib should probably be using __array__ here?)

    2. The introspection of the elements in X to test for iterableness (which lets a normal sequence of numbers work as expected) fails because the elements of the tensorflow object are 0D but an instance of Iterable:

    from collections.abc import Iterable
    print(isinstance(labels1[0], Iterable))
    print(isinstance(labels2[0], Iterable))
  6. added this to the v3.4.1 milestone on Feb 25, 2021
  7. tacaswell commented on Feb 25, 2021

    @tacaswell
    Member

    It is reasonable to expect that we will ducktype TF arrays correctly here. For now I think there is a user-space work around to cant to numpy before passing to us. That said, none of the core developers are regular TF users so any help with duck-typing them correctly would be greatly appreciated.

    Internally we can not directly use np.asarray before handling units because that will break things like pint, but checking for __array__ may work better?

    Labeled this as good first issue because I think all of the work that needs to be done is in _reshape_2D and requires little understanding of the rest of Matplotlib internals.

  8. jklymak commented on Feb 25, 2021

    @jklymak
    Member

    I think it is desirable to ducktype what we can, but we need some rules for what that means. Random object x used to work, but now it doesn't is not a good argument that random x should ever have worked. It would be preferable to try and stipulate what properties x needs to have for successful duck typing.

  9. mwaskom commented on Feb 25, 2021

    @mwaskom

    Isn't that what obj.__array__ is for?

  10. tacaswell commented on Feb 25, 2021

    @tacaswell
    Member

    Long term we can rely on the behavior defined by https://data-apis.org/blog/array_api_standard_release/ but for now I suspect we are going to have to keep playing whack-a-mole with these.

  11. 21 remaining items

  12. mwaskom commented on Aug 21, 2021

    @mwaskom

    Yes but what does it print?

  13. sriakshata commented on Aug 22, 2021

    @sriakshata

    See above comment https://github.com/matplotlib/matplotlib/issues/19574#issuecomment-902075023. I have attached the output I got.

  14. modified the milestones: v3.5.0, v3.6.0 on Sep 25, 2021
  15. removed this from the v3.6.0 milestone on Jul 8, 2022
  16. github-actions commented on Nov 20, 2023

    @github-actions

    This issue has been marked "inactive" because it has been 365 days since the last comment. If this issue is still present in recent Matplotlib releases, or the feature request is still wanted, please leave a comment and this label will be removed. If there are no updates in another 30 days, this issue will be automatically closed, but you are free to re-open or create a new issue if needed. We value issue reports, and this procedure is meant to help us resurface and prioritize issues that have not been addressed yet, not make them disappear. Thanks for your help!

  17. mwaskom commented on Nov 20, 2023

    @mwaskom

    I don't know whether this is still an issue (I think so) but something is a little whacky about this bot: it has been a lot more than 365 days since the last comment here.

  18. rcomer commented on Nov 20, 2023

    @rcomer
    Member

    I guess if we wanted to be strict, the bot message could say "it has been over 365 days since the last update". It's basically just working through the issue backlog and checking update timestamps.

  19. jklymak commented on Nov 20, 2023

    @jklymak
    Member

    To clarify further, we set a limit of how many issues get checked and how often so that they don't all get closed at once with no chance for comment.

  20. added
    keepItems to be ignored by the “Stale” Github Action
    and removed
    status: inactiveMarked by the “Stale” Github Action
    🌱 Good first issueOpen a pull request against these issues if there are no active ones!
    on Nov 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions