DOC: plotting section for Users Guide - #29124

Given that this is clearly a lot of work, what's the motivation for this section/this new set of docs? What problem/questions from folks are you seeing that you're trying to solve here that's not currently being addressed in the gallery?

I'll ping for a review for this. Obviously this can just be considered scaffolding.

I believe I addressed why the gallery is not a replacement for explaining how to make plots in the User Guide in the PR description. The gallery will never be a cohesive introductory guide, and I think that is fine. The approach here is to provide a basic start with links to more examples in the appropriate section of the gallery, or elsewhere in the User Guide.

I'm slightly annoyed by the structural duplication of this page and "Plot types".

This was deliberately done so that they would present material in the same order but with more depth than in the quick visual index of "Plot Types". I'm not sure I understand the annoyance - particularly given that I made the Plot Types PR.

MHO this would work better if

you first create an issue describing the topic, and the idea for the stuctural solution (optionally/alternatively make a proof-> of-concept PR for the structure). We could then agree on a the structure first.
then make a PR that puts the desired structure in place
after that, add multiple PRs for the individual sections.
This would keep the PRs smaller and more focused and significantly ease reviewing.

I'd consider this the proof of concept PR. If you have serious objections to the structure, folks could propose alternates. If not, I'd suggest this PR is better than nothing, which is what we currently have. If folks disagree and feel it is worse than nothing, then we should close the PR and continue to have a large gap in the docs.

If folks disagree and feel it is worse than nothing, then we should close the PR and continue to have a large gap in the docs.

I'm not sure what gap it is you're trying to fill with these docs. The plot type gallery is already heavily curated, so I don't think that the narrative is adding much here.

Some of the examples feel so bogged down with extras that I'm not sure that readers will pick up the pattern that I think is what you're trying to convey, and information on how to use each specific function.

I think if there's a gap here, it's in understanding the artists that back the plots and I'd orient the user guide plotting section around those underlying artists.

The User guide has how to make a figure, how to make an axes, ...., a bunch of complicated stuff. I don't think "go look at the plot types gallery" is a good substitute for that middle ground, and I don't think the plot types gallery ties visualizations together in a hierarchy other than their ordering. Which is great - I look at the plot type gallery as a quick overview of what the library can do that a user can quickly scan to see if their chosen visualization is there, and see what we call it.

This new section of the User Guide should get into more details and connect the visualizations which is what I tried to start here. For sure, adding more detail about the underlying artists could be helpful, though I'd perhaps not ruin the flow so much as show a few examples of using the artists, and link out to the API docs or explanation elsewhere in the the Guide.

Some of the examples feel so bogged down with extras

I'm not sure what you are referring to here, but the examples are meant to be semi complete visualizations in most cases. If folks feel something got too far into the weeds, they are welcome to propose scaling a section back or breaking it into components.

I do somewhat feel you are expressing two contradictory ideas of what this section is meant for; "more details about underlying artists" and "fewer extras" are pulling in opposite directions. Obviously writing examples is a balancing act, and folks should feel free to edit this first pass all they want.

The User guide has how to make a figure, how to make an axes, ...., a bunch of complicated stuff.

That's because the user guide introduces the concept of a figure, axes, colormap, etc, all of which are backed by objects. There's not a concept of a plot in the same manner, the closest being the artists that underlie the plot. And it covers the basics of how to make a plot in getting started and in the discussions of the library components relevant for that plot.

I'm not sure what you are referring to here, but the examples are meant to be semi complete visualizations in most cases.

For example, this gets so bogged down in the user warning (which is better explained in the API docs) that it loses focus on how to use the function (if that's the goal)
https://github.com/matplotlib/matplotlib/blob/2a0455e40a990bec76f671ef2d9be0cf7f6ea4af/galleries/users_explain/plotting/gridded.py#L95C5-L118

For example, the focus here is on setting up the data, the actual usage of the function not changing from the usage above:

For example, the goal of this example is presumably to contrast the two, which a reader can pick up from the gallery thumbnails:

https://github.com/matplotlib/matplotlib/blob/2a0455e40a990bec76f671ef2d9be0cf7f6ea4af/galleries/users_explain/plotting/gridded.py#L150C5-L173

I do somewhat feel you are expressing two contradictory ideas of what this section is meant for; "more details about underlying artists" and "fewer extras" are pulling in opposite directions.

What I mean by focusing on the artist is more documents in the style of https://matplotlib.org/devdocs/users/explain/artists/imshow_extent.html. It unpacks a characteristic of ImageBase objects that once the reader understands it, they can then apply that understanding to every plotting method that takes an extent keyword.

That's because the user guide introduces the concept of a figure, axes, colormap, etc, all of which are backed by objects. There's not a concept of a plot in the same manner, the closest being the artists that underlie the plot. And it covers the basics of how to make a plot in getting started and in the discussions of the library components relevant for that plot.

From my point of view people need to be able to make a figure, make and axes, and then make visualizations in those figures and axes. The User Guide as it stands does not have that last step. They are all underscored by objects, so I'm not following what that has to do with explaining how to use the library.

For example, this gets so bogged down in the user warning (which is better explained in the API docs) that it loses focus on how to use the function (if that's the goal)

What I mean by focusing on the artist is more documents in the style of https://matplotlib.org/devdocs/users/explain/artists/imshow_extent.html.

I'd say both of these are somewhat asides that help the user better use the library. You could easily argue that the asides should be in a separate page or put in-line. But I don't think a deep-dive on every artist type and its quirks is the right level for the top-section of the user guide. From my point of view, if it can be explained quickly, inline is OK, if not then it should be linked out to a dedicated subpage.

I give one high-level comment, other than that I'm out of the details here, because the PR gives me a mental overload:

I definitively see a place for a more detailed usage description of (some) plotting functions.
Examples:

• plot() needs explanition at least on (i) plotting lines and/or markers, (ii) the use of the shorthand fmt string plot(x, y, 'r-') vs. keyword arguments; (iii) an explantion that and why the result is a list of Line2D and not a single Line2D plus the tuple-unpacking trick to put it into a single variable.
• imshow() should have a basic explanation how to use color limits and colormaps for simple cases (not going deep dive into the data->color pipeline).

The user guide is certianly the right place for this. (How to plot with lines / markers / images). I'm not clear though on the framing / perspective. I'm feeling that the focus on "plotting methods" (Document: Using matplotlib > Plotting methods, and using method names as the sections) is wrong for a user guide. I think it should instead be something like "Using matplotlib > Visualizing different kinds of data". Then the subsections "pairwise data", "statistical plots", ... in here are ok. But within these sections again, it should not be methods but "drawing lines, markers, bars, ..." etc.

From my point of view people need to be able to make a figure, make and axes, and then make visualizations in those figures and axes. The User Guide as it stands does not have that last step. They are all underscored by objects, so I'm not following what that has to do with explaining how to use the library.

Because those sections are focused on explaining what a figure (object) is and how to create it, what an axes (object) is and how to create it, etc.

@timhoffm is also suggesting by artist here, just implicitly:

But within these sections again, it should not be methods but "drawing lines, markers, bars, ..." etc.

Because each of those visual elements is backed by a different artist.

@timhoffm is also suggesting by artist here, just implicitly:

But within these sections again, it should not be methods but "drawing lines, markers, bars, ..." etc.
Because each of those visual elements is backed by a different artist.

To be precise, I'm suggesting by kind of visualization. There's a strong parallel between "kind of visualization" and Artists, but it's not exactly 1:1 (lines = Line2D, markers = Line2D). And my point is that the user guide should be written from the perspective what the user wants (e.g. a color-coded visualization of 2D data) not what we provide (a method imshow() or an AxesImage). We should structure by the former and explain how to realize it using the latter.

And my point is that the user guide should be written from the perspective what the user wants

Sure agreed. That is basically what the current version does. It explains the type of data to be visualized and then explains the different methods to accomplish that. We don't want these to be completely abstract. Users should learn out names for things. And they should also learn the differences between the methods.

Because those sections are focused on explaining what a figure (object) is and how to create it, what an axes (object) is and how to create it, etc.

I think you are making a distinction here that the original author of those sections didn't intend.

However, the practical difference between an Axes object and a Line2d object is that you need to call many methods on the Axes object to use the library at all (or use pyplot). On the other hand, you could make hundreds of complicated visualizations using Line2d artists and not know any of the methods on that artist, or that the artist object even exists.

Not at all to argue that these sections should avoid mentioning or using artists, just that I don't think that should be a focus at the top level.

That is basically what the current version does. It explains the type of data to be visualized and then explains the different methods to accomplish that.

But what @timhoffm (and I agree) is saying is it should be grouped by visualization task. I think some of this is fixable with subheadings saying things like "Plotting lines", etc...

I think what's getting jumbled up here is that method and task aren't a 1:1 either. Like scatter plots can be made using .plot or .scatter, line plots can be made using .plot, and for special types of lines .vlines, .hlines, .axline, .event and
.stem, .stair and .step (all of which I think can be accomplished with .plot)

But what @timhoffm (and I agree) is saying is it should be grouped by visualization task. I think some of this is fixable with subheadings saying things like "Plotting lines", etc...

The current headings are:

• Plotting Methods
  • Pairwise data
  • Statistical plots
  • Gridden plots
  • Unstructured data
  • 3d and volumetric data

The sections start with a short intro about the type of data to be visualized, and then are indeed subsectioned by method type. So taking "Pairwise data" , it has

• plot
• fill_between, fill_betweenx, and stackplot
• errorbar
• scatter
• bar and stem
• step and stairs

we could maybe expand the description of things you do with pairwise data a bit at the beginning, but this is pretty obvious thing to do with your data. We could also change the headings, but to what without being too verbose? "plot" -> "plotting lines and markers, one color and marker at a time" "fill_between"-> "indicating regions around a line"?

The first sentence of each of these sections basically says what the primary goal of the method is, so in my opinion it's clearest just to use the method name. New users may not know our names for things, but the hope is they can realize they want to plot pairwise data, and then skim the plots for the method most suited to their visualization and then read more.

@jklymak did you intend to close this? I mostly wasn't sure if this was ready for rereview b/c you had a bunch of close pushes together and then I got bogged down in paper submissions.

Yes - but anyone is welcome to pick it up and use it how they see fit. Thanks

Doing a final review and merging this is still on my todo list (I have just not found the hour I think it is going to take).

and merging

I still think the scope of this PR means it should get two approvals. This isn't a new paragraph but a whole new section - that's a lot of new docs to maintain, so there should be buy in from at least two maintainers.

ETA: I think a discussion on #32113 should happen first.

To be clear, I do not think we are going to reach consensus here and I intend to make an executive decision (to merge) as the project lead.

Not sure I have ever actually done this before, but cases like this are why we have that fallback in the process.

, I do not think we are going to reach consensus here

@jklymak has put in a ton of work making changes to try to meet me and Tim in the middle so I don't know that that's true w/o review. Also we always have a discussion on the call about things this contentious.

I'm not even saying don't merge, just wait til after Thursday. The reason we rarely merge by fiat (speaking as the community manager) is b/c it has high negative community costs.

I would like to pick up my request from #29124 (comment)

My only request would be that you add a documentation section next to https://matplotlib.org/devdocs/devel/document.html#examples-guidelines and https://matplotlib.org/devdocs/devel/document.html#plot-types-guidelines that describes intent and recommendations of this new part of the docs. Otherwise I'm afraid everybody will have their own mental model how this should look like and future iterations will not converge to a consistent or desired state.

with Jody's response:

Sure, done. I added a section on tutorials as well since that was missing, and updated the table to have a bit of info about the goal of each section of the galleries, made that a subsection, and added a link near the table to the subsection.

Note this is a separate commit: 2bb91bb

I think it's a good start, but still does not define define the scope of the plotting section in plotting data. For that I'd like to bring to attention #29124 (comment), which seems like having reached some consensus between Jody and me.

My preferred approach would be

1. cobble the above together in a separate PR, focussing on our intent. When that is agreed and merged
2. do the actual doc content changes and gauge them to our agreed-on intent. I would do this even in multiple smaller PRs to ease review. I believe we should favor quality over quantity and speed for these docs. So I'd rather incrementally add carefully reviewed content than throwing in a lot of stuff at once. Our docs and in general matplotlib usage instructions do not lack volume, they do lack targeted high-quality explanations.

As said in #29124 (comment)

I do have concerns on structure, content, style and quality. As originally said, I feel unable to properly review and address them due to the size of the PR.

cobble the above together in a separate PR, focussing on our intent.
When that is agreed and merged do the actual doc content changes and gauge them to our agreed-on intent.

So the reason I assigned myself to this PR was my plan was to do those PRs (using this as the base & w. Jody as coauthor & aiming for minimal content changes/keeping Jody's content as the initial commit) in a kinda stacked PR fashion (so everything gets merged together) over the next couple of weeks b/c I figure Jody doesn't have the bandwidth for more review cycles. ETA: And yes, since I'd be authoring the PR, I'd also be panhandling for reviews from other maintainers.

I think it's a good start, but still does not define define the scope of the plotting section in plotting data. For that I'd like to bring to attention #29124 (comment), which seems like having reached some consensus between Jody and me.

My preferred approach would be

1. cobble the above together in a separate PR, focussing on our intent. When that is agreed and merged
2. do the actual doc content changes and gauge them to our agreed-on intent. I would do this even in multiple smaller PRs to ease review. I believe we should favor quality over quantity and speed for these docs. So I'd rather incrementally add carefully reviewed content than throwing in a lot of stuff at once. Our docs and in general matplotlib usage instructions do not lack volume, they do lack targeted high-quality explanations.

As said in #29124 (comment)

I do have concerns on structure, content, style and quality. As originally said, I feel unable to properly review and address them due to the size of the PR.

Matplotlib has been discussing the state of the docs for over a decade. We could spend another decade trying to craft the perfect process for a flawless User's Guide, or we can merge an "okay" baseline today that our users can actually read, give us feedback on, and help us iterate on.

I am perfectly fine deferring to the project's decision on whether to merge or close this. However, given the history of the documentation gap, I remain skeptical that closing this will result in someone pushing through a substantive replacement anytime soon.

in someone pushing through a substantive replacement anytime soon

I think this PR is most of the way there (which is why I was planning to pick it up) but my (@ I think Tim's) concerns are mostly about scope & I don't think that's fixable in a follow up w/o us first agreeing on scope. Concretely, I expect that if I put in a PR deleting all the side quests (which apologies if you've already done on the changes I haven't reviewed), it will end up w/ all the same discussions on scope that aren't resolved here but we have a merged document with loose scoping that will invite more additions to the text w/ everyone's typical use case & that's how our docs grow unmanageable.
ETA: Or my PR deleting things gets merged and then you or someone else puts in a PR adding it back and we go in circles.

our users can actually read, give us feedback on, and help us iterate on.

For the most part, the content here doesn't add content that's not already in our docs but mostly just presents it in a different way. That's why Tim and I aren't treating it as an urgent documentation gap that needs filling.

ETA: also one of the things this PR is most stuck on here is the guidelines for iteration - and contributors aren't gonna feel empowered to help us iterate if they can't parse out how. That's why the examples and API currently get the most docs PRs - they're the best scoped and therefore easiest to understand what docs for that section looks like. Most folks don't touch userguide precisely b/c it's ill defined.

Which also I think the root disagreement here is a matter of perspective. I think you're coming at this as the person who writes a lot of the docs, and Tim and I are approaching this as the folks who clean up/consolidate/restructure the docs.

But also I think we've gone in so many circles here & this just needs a discussion on the call where other maintainers feel invited to chime in b/c I think the discussion here has gotten too contentious for that. If @jklymak can't make it anytime soon, this PR can be championed, which is the way we frequently handle PRs on the call where the author isn't present.