[ENH]: Possible improvements for out-of-the-box interactivity

Problem

Compared to other visualization frameworks, matplotlib has very limited capability for interacting with the plot. I think we could do a bit better here.

There are two obstacles related to interactivity in Matplotlib:

By design, we have a one-way processing pipeline from data to a visual representation (pixels or vector primitives). Backtracking which data or Artist is responsible for a visual output is possible, but not quite easy. Additionally, any desired visual change requires a complete re-rendering.
We have multiple backends and want to provide consistency. This means, we have non-interactive backends that just can't provide interactivity. The interactivity we provide should work across all interactive backends.

1 is an architecture limitation, that we cannot easily circumvent. This means, we won't be high-perfoming and won't be able to support complex interactions. 2 also limits the complexity of interactions we can practically implement. OTOH it does not mean we cannot provide any interactivity at all. For example, we do have panning and zooming in interactive backends.

Proposed solution

Let us consider which small improvements we can reasonably make to improve interactive experience. The following list should track interactivity ideas that seem prossible to implement under the current architecture with limited effort. Please feel free to add:

Data tooltip support [ENH]: Data tooltip support #23378
Support panning and zooming using mouse scroll wheel Support panning and zooming using mouse scroll wheel #20317
Toggling data via legend Add Checkboxes to Legend Controlling Subplot Visibility #16796

Compared to other visualization frameworks, matplotlib has very limited capability for interacting with the plot.

What other general scientific visualization frameworks are you referring to here?

What other general scientific visualization frameworks are you referring to here?

bokeh and plotly to name two popular examples. But basically you could take any javascipt or d3js based projects from https://pyviz.org/overviews/.

Compared to other visualization frameworks, matplotlib has very limited capability for interacting with the plot.

What other general scientific visualization frameworks are you referring to here?

Apart from matplotlib, I've only worked with pyqtgraph and plotly but matplotlib definitely lies behind here.

@jbednar

But basically you could take any javascipt or d3js based projects from https://pyviz.org/overviews/.

I think that's the problem though, in that what I've heard folks most ask for for improving our interactivity is better web/HTML integration. Which far as I know, (and I fully admit I can be totally wrong here), means a (better?) HTML/JS backend. And like I could be wrong but I think one way to get there is serialization.

But also tagging @jbednar b/c he may have suggestions/opinions on what sorta interactivity would be good for mpl to provide.

Better web/HTML integration is certainly on the wishlist. But IMHO it’s on the larger side of changes (write a new backend / improve ipympl). Additionally, serialization is partly orthogonal. It would improve communication between frontend and renderer, or move tasks to the frontend. Other interactive backends would benefit as well. Either way, it’s also a large-effort project (and I believe the data prototype is going that direction).

Both are out of scope for this collection. I‘m trying to find the low-hanging fruits that can be implemented in a reasonable one-PR effort. I believe the above mentioned topics fall into that category.

For sure the state of various matplotlib/d3 based projects indicates that leveraging d3 is hard to get the abstractions correct. D3 uses SVG and canvas, so perhaps using those primitives is the way to go. This GSOC from 2019 https://blog.pyodide.org/posts/canvas-renderer-matplotlib-in-pyodide/ looks like it was very close to a working backend.

However, agreed that this is more of a heavy lift than I think this issue is discussing.

@ianhi

Both are out of scope for this collection. I‘m trying to find the low-hanging fruits that can be implemented in a reasonable one-PR effort. I believe the above mentioned topics fall into that category.

hmm, also tagging @ianhi here then for ideas.

But also I think improving the docs isn't quite low-hanging fruit but can be scoped to one PR at a time and would probably go a long way to showing folks that Matplotlib has pretty rich interactivity support, here's how to hook the wires up. On my todo list/ a long standing idea was convert https://github.com/story645/pydata_nyc_2023 into a gui agnostic "how to build a dashboard" tutorial - or of course someone can write there own, but the idea is have an end to end tutorial. I've been thinking this way cause most of our current docs are either such small widget/gui examples it can be hard to fit them into larger context, or written in a gui programming build a desktop app context and the terminology isn't always the same when talking about viz.

Sure, documentation helps to discover what is available. However, „how to hook the wires up“ sounds like low-Level plumbing. Personally, my focus is out-of-the-box or simple configuration (e.g. add (static) tooltips via a tooltip argument)

However, „how to hook the wires up“ sounds like low-Level plumbing

I was thinking more like the "how to make a dashboard" level, where you wanna build up something with multiple linked panels & where zoom on one panel updates the other (inset indicator but make it a panel) & maybe make one panel with a hover tool and one with a span selector and that these things talk to each other.

Basically what I mean by "how to hook up the wires" is showing how to combine the various "out of the box" things.

I understood that. A tutorial for this may be nice, but it's not my scope here.

One a side-note: IMHO the "various out of the box things" are quite limited and low-level (basically, zoom/pan, picker, events/callbacks). We're missing a lot to make complex dashboarding easy (visual-to-data relation, layouting, good widgets, ...). There's a reason for additional explicit dashboarding libraries (e.g. panel or dash) built on top of the (already better interactive) js libraries. We will not be able to compete with those. Or to put it in another way: If you can reimplement medium-complexity dash or panel examples without significant more effort in Matplotlib, I'd be impressed and change my opinion.

So yes, showing that one can get a bit of interactivity and a simple dashboard is valuable. But as of now, I would not recommend matplotlib to anybody who wants to build more than the simplest of dashboards.

So my question is who are you trying to target w/ these out-of-the box interactivity improvements? Is it the dashboarding library authors? Like these are different classes of things and that's why my suggestions are also disjoint:

Maybe dashboard was the wrong word to use here, but using it in the broadest sense multi linked panel interactive tool. Though what do we talk about when we talk about dashboards? has me thinking that #19520 and a consistent .set_data interface (which yes prototype work) are probably the concrete improvements there.

And I'm hammering on docs b/c the folks I talk to tend to be surprised that we even do interactive, so I think cleaning up the docs would be as much of an improvement as new code.

who are you trying to target w/ these out-of-the box interactivity improvements

end-users, who just expect a little more interactivity.

my suggestions

these suggestions are infrastructure behind the scenes, which may or may not be necessary. I come from the user side:

data tooltip support: As written in the issue, I want that plt.plot(range(4), range(4), tooltip=['A', 'B', 'C', 'D']) "just works". A bit analogous to how we set additional info with the legend label.
Toggling data via legend: No configuration: Clicking legend entries could by default hide/show data (and fade the entry if data is hidden). This is a bit similar to the colorbar zooming we've implemented: We've added default interactive capability to existing elements that were previously static.
Support panning and zooming using mouse scroll wheel: Also no configuration. Just add the functionality.

these suggestions are infrastructure behind the scenes, which may or may not be necessary

I'm mostly suggesting better docs so that end users understand what comes out of the box vs what knobs we give them to make their own things.
As far as I remember, we don't have a clean "here are the out of the box interactive things we provide" doc.

"here are the out of the box interactive things we provide" doc

https://matplotlib.org/stable/users/explain/figure/interactive.html#

That doc does not cover things like sharex/sharey or auto update colorbar scroll or the data cursor (which yes gets a very brief mention). It's TOC tree is:

live updating a figure in a REPL
GUI toolbar explanation
keybindings
GUI backends kinda

That doc does not cover things like sharex/sharey or auto update colorbar scroll or the data cursor

Please open an issue or issues.

@jbednar

But also tagging @jbednar b/c he may have suggestions/opinions on what sorta interactivity would be good for mpl to provide.

I think I have the same concerns as mentioned by others above -- It will be very difficult to get full, nicely integrated interactivity with Matplotlib's current design, so it's important to focus on some limited subset that is better than what's available now, but is feasible. But I also agree with @story645 that very few users make use of what's already supported by Matplotlib.

So, if I were in charge of a small pot of money or developer time to spend, what I would personally direct that work to be is:

making basic cleanups to ipympl and particularly the icons and widgets to make what's already there be cleaner, nicely laid out, and "reasonable looking" in a web browser -- surely that's not a huge project! Then
documenting everything that's available and make it super easy for people to discover and use. Then
adding some limited additional interactivity as proposed in the first message here.

I.e., it seems like there's lots of low-hanging fruit just to make the interactive plotting look like it's from this millenium so that it doesn't feel quite so awkward to use, and then there's lots of outreach needed for most mpl users to even realize that this capability exists and is easy to invoke. At that point I'd add more interactivity, but for my part I think 1 and 2 are holding things back more than 3 is.

It will be very difficult to get full, nicely integrated interactivity with Matplotlib's current design,

I'm still curious what defines "nicely integrated interactivity" or if there is a package that exemplifies this? I understand the three wishes above, but none of those are hard to implement with Matplotlib's current design. But I'm curious what else would make things better?

Sure! Just try out ipympl as at https://matplotlib.org/ipympl/ and then compare it with even the basic interactivity at https://docs.bokeh.org/en/0.10.0/docs/user_guide/interaction.html or https://plotly.com/python/line-and-scatter/ or https://altair-viz.github.io/user_guide/interactions.html#filtering-data . All four libraries produce plots that are interactive to some degree, but Matplotlib looks and acts like something from the 1990s with only very basic operations supported, while the rest all integrate well with the browser, look like normal web pages, have fast, responsive interactivity, and allow you to interact directly with data points in a clean and straightforward way. Hence my suggested list above -- first get mpl looking like it's even in the same universe as those JS-based libraries, then make sure lots of people will realize that this level of support is even available, and then try to add specific interactive functionalities where feasible.

@ianthomas23

making basic cleanups to ipympl and particularly the icons and widgets to make what's already there be cleaner, nicely laid out, and "reasonable looking" in a web browser

Not sure how much of that is part of the plan for @ianthomas23, who has been working on various ipympl improvements.

Also to also be clear, also I know it's not an either or thing as different (unpaid) folks want to work on different things. I opened an issue to track doc improvements separately #28722, but originally I had just seen "interactivity improvements" and my second thought was DOCS!

Mostly cause I figure otherwise what happens is we have all these nice new things nobody will know about, b/c that's what's been happening.

But also in hindsight I get that this was supposed to be a tracking "what interactivity API can we add relatively easily" issue so I am sorry for taking the discussion very off track/suggest maybe opening a more tightly scoped issue? (or honestly GitHub tracking project for all the interactivity things).

but Matplotlib looks and acts like something from the 1990s with only very basic operations supported,

I don't think their GUI elements look particularly better than ipympl's. I agree our basic pyqt etc backends look a little long in the tooth, but I'm sure that if someone wanted to overhaul the toolbar widgets we'd be OK with that.

As for "basic operations" I agree that there is tooltip support that we are missing out of the box (though I'm not sure we want tooltip support on by default). Otherwise, these other libraries seem to have the same basic tools: zoom, pan, reset, save. Plotly has a "selection" tool that can do fancy things if put into a matrix with other plots, but the fancy things rely on underlying data relationships between the plots. And we have RectangleSelector, so Matplotlib can do this with a bit of plumbing. I actually prefer Matplotlib's forward/backward tools which I don't see in those other libraries.

the rest all integrate well with the browser, look like normal web pages, have fast, responsive interactivity,

Someone "just" needs to write an html5/canvas backend who wants to do this; I don't think it is a fundamental limitation of Matplotlib. (see https://summerofcode.withgoogle.com/archive/2019/projects/4683094261497856 for one attempt).

@timhoffm

Someone "just" needs to write an html5/canvas backend who wants to do this; I don't think it is a fundamental limitation of Matplotlib. (see https://summerofcode.withgoogle.com/archive/2019/projects/4683094261497856 for one attempt).

While I'm the one who brought it up first, I agree w/ @timhoffm that to do it properly relies on a re-architecture and I think he's right about it being out of scope here. The backend you link is the basis for the one in PyIodide, and as far as I know the improvements on the mpl side to make that work better are part of the proposed work in the 2nd ROSES proposal. I also think some of that proposed work could help in building a better HTML backend, but that's the scope of work involved.

though I'm not sure we want tooltip support on by default

I think @timhoffm's/#23378 proposal of enable via kwarg is a good compromise, to bring it back to the original purpose of this issue 😓

I actually prefer Matplotlib's forward/backward tools which I don't see in those other libraries.

Matplotlib can probably do most of what some of those libraries do (and can do more than some of them), but many users don't realize this b/c of the documentation (#28722) which is why I, and I think @jbednar, rate better docs so high for "possible improvements for out of the box interactivity".

@ianthomas23

making basic cleanups to ipympl and particularly the icons and widgets to make what's already there be cleaner, nicely laid out, and "reasonable looking" in a web browser

Not sure how much of that is part of the plan for @ianthomas23, who has been working on various ipympl improvements.

I don't have any plans for improving the appearance of ipympl, I'd much rather make it more resilient to upstream changes (in matplotlib, ipython and ipykernel).

IMO, docs come after out of the box interactivity. First you use it, then you customize it. Customization is where you need docs.

And that's because as you said, a lot of these things are already available, but just poorly documented or not as easy to implement as they should be. So 1) add these:

Data tooltip support [ENH]: Data tooltip support #23378

Support panning and zooming using mouse scroll wheel Support panning and zooming using mouse scroll wheel #20317

Toggling data via legend Add Checkboxes to Legend Controlling Subplot Visibility #16796

and then 2)

documenting everything that's available and make it super easy for people to discover and use.

And for the last point:

making basic cleanups to ipympl and particularly the icons and widgets to make what's already there be cleaner, nicely laid out, and "reasonable looking" in a web browser -- surely that's not a huge project!

From a user point of view, the widgets and buttons, although certainly feeling outdated, are a don't-care. As long as the plots look professional, and that I can interact with them, I don't care about what the buttons look like.

surely that's not a huge project!

It may not be a huge project, but as long as there is no canvas renderer, I don't see why anyone would use the web backend (the agg backend <-> web frontend is too slow).

I don't see why anyone would use the web backend (the agg backend <-> web frontend is too slow).

To set perspective: I believe it's typically fast enough for one-time actions (think mouse-click, hover), but not for continuous updates (mouse move).

making basic cleanups to ipympl and particularly the icons and widgets to make what's already there be cleaner, nicely laid out, and "reasonable looking" in a web browser -- surely that's not a huge project!

I'd expect this is mainly CSS, so could be done by almost anybody without having to know anything about matplotlib or ipympl.

and then 2)\n\ndocumenting everything that's available and make it super easy for people to discover and use.

See issue is we won't have a place to put the documentation for the new stuff b/c we don't have a place for the current stuff b/c it doesn't exist.

My worry as related to this tracking issue is that w/o a plan for where to put docs, new features get documented in the current haphazard fragmented poorly discoverable way as part of the PRs they get put in , which creates a bigger documentation burden b/c now more stuff added to a doc reorg.

ETA: yes I can work on docs separately/concurrently, but the way the project works the decision has to happen their first then.

See issue is we won't have a place to put the documentation

Google "Matplotlib interactivity" and the top hit takes you to https://matplotlib.org/stable/users/explain/figure/interactive.html. So we do indeed have a place to put the documentation.

See issue is we won't have a place to put the documentation

Google "Matplotlib interactivity" and the top hit takes you to https://matplotlib.org/stable/users/explain/figure/interactive.html. So we do indeed have a place to put the documentation.

I think adding one more section to that document so it's an even larger collection of related but disjoint topics would make that page even harder to maintain and the documentation even more undiscoverable so I would have major concerns on a PR that adds docs there that until we have a discussion and plan on interactivity docs.

But we're getting even more off topic here so we can discuss docs at #28722

Navigation Menu

Repository navigation

[ENH]: Possible improvements for out-of-the-box interactivity #28708

Problem

Proposed solution

new widget?

artist method?

event hook?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[ENH]: Possible improvements for out-of-the-box interactivity #28708

Description

Problem

Proposed solution

Activity

jklymak commented on Aug 12, 2024

timhoffm commented on Aug 12, 2024

ecstrema commented on Aug 12, 2024

story645 commented on Aug 12, 2024

timhoffm commented on Aug 12, 2024

jklymak commented on Aug 12, 2024

story645 commented on Aug 12, 2024

timhoffm commented on Aug 12, 2024

story645 commented on Aug 12, 2024

timhoffm commented on Aug 12, 2024

story645 commented on Aug 12, 2024

new widget?

artist method?

event hook?

timhoffm commented on Aug 12, 2024

story645 commented on Aug 12, 2024

jklymak commented on Aug 12, 2024

story645 commented on Aug 12, 2024

jklymak commented on Aug 12, 2024

jbednar commented on Aug 14, 2024

jklymak commented on Aug 14, 2024

jbednar commented on Aug 14, 2024

story645 commented on Aug 15, 2024

jklymak commented on Aug 15, 2024

story645 commented on Aug 15, 2024

ianthomas23 commented on Aug 15, 2024

ecstrema commented on Aug 15, 2024

timhoffm commented on Aug 15, 2024

timhoffm commented on Aug 15, 2024

story645 commented on Aug 15, 2024

jklymak commented on Aug 15, 2024

story645 commented on Aug 15, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions