Skip to content

[Bug]: MacOSX backend windows have no accessibility data #24608

Closed

Description

@jdtsmith

Bug summary

A very common way of interacting with windows on MacOS is through Apple's Accessibility Framework, which is used for all manner of screen manipulations, voice readers, etc. For example, sight-impaired people can use voice commands to select buttons or have control names read to them, users can use accessibility controls to move and arrange windows, etc.

I have discovered that the plotting windows produced in the "Python" app of the MacOSX backend are created without any accessibility information whatsoever. I used the "Accessibility Inspector" to check this. I've never encountered this before; in fact I didn't think it was possible. Is it a known issue?

Code for reproduction

Simply create a MacOSX backend window and attempt to inspect it with "Accessibility Inspector".

Actual outcome

A seemingly normal MacOS window is created, but it has no accessibility controls or data of any kind.

Expected outcome

Like all modern MacOS applications, Python windows come with at least basic accessibility controls.

Additional information

No response

Operating system

MacOS

Matplotlib Version

3.5.2

Matplotlib Backend

MacOSX

Python version

3.10.8

Jupyter version

n/a

Installation

pip

Activity

  1. greglucas commented on Dec 5, 2022

    @greglucas
    Contributor

    I don't think this is exclusive to the macosx backend. I can't pick up anything with tkagg or qt5agg backends either.

  2. jdtsmith commented on Dec 5, 2022

    @jdtsmith
    Author

    Thanks. With --matplotlib=qt5 in iPython, I do find accessibility info on the plot window:

    image

    It's much slower than macosx backend though, so it would be great to add accessibility data to that.

  3. greglucas commented on Dec 6, 2022

    @greglucas
    Contributor

    @jdtsmith, I've never used the Accessibility Inspector before, is there something I need to do to get it to capture the window properly? It doesn't register anything for me on any of the backends. I have been running with MPLBACKEND=qt5agg python test.py (running with ipython --matplotlib=qt5 and inputting interactively didn't work for me either.

    We are including the ApplicationServices header, but maybe there is some more explicit work we need to do in the macosx backend. PRs very welcome on this!

    #include <ApplicationServices/ApplicationServices.h>

  4. jdtsmith commented on Dec 6, 2022

    @jdtsmith
    Author

    You hit option-space and that activates the cursor to grab any elements (which is highlighted in green). Clicking on an element then "freezes" it so you can inspect the information. You'll notice that the macosx backend plot window is invisible to Accessibility Inspector.

    Unfortunately I've never implemented such a thing in Objective C; just a "user" of accessibility data via a tool called Hammerspoon.

  5. greglucas commented on Dec 6, 2022

    @greglucas
    Contributor

    You'll notice that the macosx backend plot window is invisible to Accessibility Inspector.

    That is what all of the Matplotlib backends look like to me :( The highlighted green selector gets all of the surrounding windows but doesn't ever see the Matplotlib ones as you mention.

    Unfortunately I've never implemented such a thing in Objective C; just a "user" of accessibility data via a tool called Hammerspoon.

    If you are interested in improving Matplotlib's accessibility in other ways or chiming in with advice, we have a few current issues surrounding alt-text and ARIA where your input would be welcome!
    #21328
    #24309

  6. jdtsmith commented on Dec 6, 2022

    @jdtsmith
    Author

    You'll notice that the macosx backend plot window is invisible to Accessibility Inspector.
    That is what all of the Matplotlib backends look like to me :( The highlighted green selector gets all of the surrounding windows but doesn't ever see the Matplotlib ones as you mention.

    Maybe that's good news? I.e. it means PyQT recently added it? Here are my versions:

    PyQt5                     5.15.7
    PyQt5-Qt5                 5.15.2
    PyQt5-sip                 12.11.0
    
  7. greglucas commented on Jun 4, 2023

    @greglucas
    Contributor

    I think the root of the problem is which compiled python we are using to run our programs. To get accessibility features we need a Framework build of Python.

    If I create a new virtual environment from a brew-installed Python 3.11 and use that, I get the accessibility information on all of the buttons and windows.

    python3.11 -m venv .env
    source .env/bin/activate
    pip3 install matplotlib==3.7.1
    python3 test_script.py

    However, if I use a conda environment's python (which is non-framework), then the window is invisible to the accessibility panel as you described. If you are in conda, you can choose a framework build of Python with "w" pythonw test_script.py which then also shows the accessibility information.

    Unfortunately, I don't think there is anything we can do on the Matplotlib side for this.

  8. jdtsmith commented on Jun 5, 2023

    @jdtsmith
    Author

    Note that QT backends offer accessibility with non-framework python versions. It's not clear how they manage that yet the MacOSX backend does not.

  9. greglucas commented on Jun 5, 2023

    @greglucas
    Contributor

    Note that QT backends offer accessibility with non-framework python versions. It's not clear how they manage that yet the MacOSX backend does not.

    I can't reproduce that with the same pyqt versions as you posted above. I need a framework python version to get QT and Tk windows to show up in the accessibility framework as well. How are you installing python and matplotlib for the different situations?

  10. jdtsmith commented on Jun 5, 2023

    @jdtsmith
    Author

    I take that back. I am indeed using a framework build (brew-installed), yet the MacOS backend does not provide accessibility information, unlike QT. This is Python 3.11.3, MPL 3.7.1.

  11. greglucas commented on Jun 5, 2023

    @greglucas
    Contributor

    Odd, and you are running the same python with each of the backends? Here is a quick screencap with a fresh install of matplotlib==3.7.1 bringing up a simple figure window with the macosx backend and then showing the accessibility options that appear on the right when clicking the pan button.
    image

    Some other thoughts for differences: M1/arm Mac (I'm on intel), XCode/clang versions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions