Removal of verbose not documented: AttributeError: module 'matplotlib' has no attribute 'verbose'

Bug report

Bug summary

In v2.2.0, the verbose class was removed (and replaced with logging) - see #9302. The API changes document has no mention of this change.

This was definitely part of the external API - the default matplotlibrc file contained the note:

# You can access the verbose instance in your code
#   from matplotlib import verbose.

This was just a straight-ahead error on my part. __init__.py had a line: verbose = Verbose() that I didn't realize had global significance. So

pretty sure you can do verbose = matplotlib.Verbose() in the meantime.
I'll open a PR to revert the dropped line.

Sorry about the inconvenience. The API was not meant to change.

This seems to have other consequences as well, namly PyCharm not producing plots in its toolwindow. See https://stackoverflow.com/questions/49146678/module-matplotlib-has-no-attribute-verbose

@ImportanceOfBeingErnest

A new stable version of matplotlib must be released as soon as feasible. Ideally, tonight or tomorrow. Why? Because the entire matplotlib ecosystem has been brought to its knees by this apocalyptic mistake.

As @ImportanceOfBeingErnest notes, the breakage extends to PyCharm itself. Our own multiphysics biology simulator is now incompatible with the most recent stable version of matplotlib. Functional and unit tests are failing everywhere. Dumpsters are belching fire. The streets are littered with bugs. An ominous red glow emanates from the ashen sky. In the deafening stillness, you can hear a codebase cry. I believe my delicate hair may even have been singed.

This is calamitous, frankly. For stability, downstream applications requiring matplotlib as a third-party dependency now need to either:

Blacklist matplotlib 2.2.0 and desperately pray for a hasty resolution.
Monkeypatch the top-level matplotlib module as follows (untested, but conceivably working):

try:
    from matplotlib import verbose
except:
    from matplotlib import Verbose
    matplotlib.verbose = Verbose()

Emergency blood ministration may now be in order.

Again, so very sorry this caused a major issue for folks. Obviously Matplotlib does not want to break things.

If a downstream package depends on Matplotlib, the maintainers are strongly encouraged to test out the release candidates . That way we can catch unanticipated bugs before the new release. Yes our testing should catch everything, but downstream packages use the API in ways the developers don't always anticipate.

Release candidates are announced via mailing lists

@jklymak

Again, so very sorry this caused a major issue for folks.

Thanks for being so forthright about this whole mishap. You're awesome, @jklymak.

Obviously Matplotlib does not want to break things.

Absolutely. That said, to prevent this from inevitably regressing, it'd be useful if a unit test exercising the existence of the matplotlib.verbose object could be added to matplotlib's test suite.

If a downstream package depends on Matplotlib, the maintainers are strongly encouraged to test out the release candidates.

That's fair enough, but... reality intrudes. When project deadlines conflict with best practices, project deadlines always win. Our project alone has five large-scale mandatory dependencies (including matplotlib, NumPy, SciPy, and Pillow) and a bevy of large-scale optional dependencies (including NetworkX, PyDot, and ruamel.yaml).

Consider PyCharm, then. How many mandatory and optional Python dependencies must that million-line codebase have? Effectively all of them.

Expecting project maintainers to actively test release candidates for all dependencies seems... egregious. With all due respect, that responsibility principally lies with the matplotlib development team. This is what continuous integration is for. Happily, you already do continuous integration. Assuming a test exercising this issue is added to your test suite and a new point release (e.g., matplotlib 2.2.1) is promptly issued, that's the best anyone can or should expect.

Thanks again, @jklymak and crew. We couldn't model head-regenerating flatworm planaria without you!

Well, just for some history. Before 2.2, we didn't need a unit test for import matplotlib.verbose because every module called it! Then no modules called it, and no one noticed. 😢 We probably won't add one now because its deprecated (i.e. verbose will be removed in 6 months).

There is already a PR in to re-instate verbose for now. I wouldn't hold my breath for a release this w/e, but it'll happen soon (beyond my pay grade, but the higher ups are keenly aware of the problem).

Completely understand about everyone's time, and the difficulties of maintaining package compatibility. No doubt downloading every RC for every dependency is a PITA. We'll work through this as quickly as we can. Thanks for your patience.

We probably won't add one now because its deprecated (i.e. verbose will be removed in 6 months).

Gotcha. If that's the case, we won't even bother monkey-patching verbose back in; instead, we'll:

Detect the current version of matplotlib at runtime.
If matplotlib < 2.2.0, then use matplotlib.verbose.
Else, defer to matplotlib's new logging integration.

Thanks for the heads up!

I wouldn't hold my breath for a release this w/e, but it'll happen soon (beyond my pay grade, but the higher ups are keenly aware of the problem).

Cheers. While I'm a Vim afficianado myself, PyCharm + matplotlib 2.2.0 is currently broken, which is a big deal. With any luck, both teams will roll out solutions as speedily as feasible.

In the meanwhile, there be scorpions here. 🦂 🦂 🦂

Out of curiousity what do you need verbose for?

Out of curiousity what do you need verbose for?

That's a good question, actually.

The answer, of course, is many, many things. Because matplotlib is so central to our simulator, we manipulate matplotlib at a dangerously lower level than one might expect or desire. I hope to eventually contribute most of this functionality back to matplotlib itself, but... time constraints.

In particular, our simulator:

Dynamically reduces matplotlib verbosity at runtime depending on the current logging level requested by the end user. Specifically, if the user requests:
- Any higher logging level, we forcefully inject --verbose-silent into sys.args in the manner detailed below.
- The logging.DEBUG level or lower, we forcefully inject --verbose-helpful into sys.args before importing from matplotlib. For obscure reasons I no longer recall, doing so required that we monkeypatch the matplotlib.verbose object at runtime as follows:

            # Prevent the verbose.set_level() method from reducing to a noop,
            # as occurs when this private attribute is *NOT* nullified. waat?
            verbose._commandLineVerbose = None      # yes, this is horrible

Defines a number of animation-specific MovieWriter subclasses, enabling end users to configure animation encoding to a finer-grained degree of detail. These include:
- NoopWriter, an animation writer ignoring rather than writing animation frames.
- ImageMovieWriter, an animation writer writing animation frames to still image files (e.g., PNG) rather than encoded videos.
Defines a wickedly complex submodule named mplvideo, whose principal claim to fame is the get_first_codec_name() function. As the name (hopefully) implies, this function dynamically queries both matplotlib and the external command-line video encoder (e.g., fmpeg, avconv, mencoder) for the optimal codec (e.g., libx264) supported by the passed container format (e.g., mkv, mp4). What all of this means is that end users no longer need to explicitly select codecs; instead, our simulator automatically selects the best codec for the requested video filetype. It'd be great to shift this ~~maintenance burden~~ intelligent logic into matplotlib itself. (Someday. Over the bug-filled rainbow.)

Needless to say, logging concerns frequently raised their ugly heads while implementing this deep voodoo. Ergo, matplotlib.verbose.

This functionality is all several years old now. It's probably long since been superseded by recent matplotlib improvements, but... it works, so that's nice.

Now that I've penned the above, I have growing concerns. In theory, matplotlib's shift to the canonical logging API is great, because matplotlib.verbose is terrible. In practice, I'm unclear how exactly to refactor our workflow. Documentation is fairly scant at the moment.

Since matplotlib logging is typically much too verbose for our purpose, we'd like to squelch this verbosity. How exactly would we go about that? I note this germane discussion from several months ago, which suggests that:

The --verbose-* family of CLI options (e.g., --verbose-helpful, --verbose-silent) are probably going away in the near-term future. Is that accurate? That's fine for us, if so. Matplotlib's coercive parsing of sys.argv is an unpleasant side effect that we try to ignore as much as possible.
The logging level for the matplotlib logger must be configured before importing from matplotlib, according to this erudite comment from you. Is that still accurate? Specifically, is the following now the recommended approach for reducing matplotlib verbosity at runtime?

import logging
logger = logging.getLogger('matplotlib')
logger.setLevel(logging.WARNING)
import matplotlib

Again, allow me to mention that you're awesome.

Basically, there are some log calls at DEBUG level that are triggered just by importing matplotlib(.pyplot). (I have put in a few PRs that try to make sure that the act of importing matplotlib(.pyplot) never logs anything at a level higher than DEBUG, but no guarantees there.) Given that in the absence of config, the default loglevel is WARNING, that means that if you are interested in matplotlib's import-time DEBUG logs, you need to configure the loglevel before importing matplotlib.

If you don't care about these, then the time where you configure logging doesn't matter. In other words: logging configuration needs to respect time causality.

@anntzer

Excellent. As you suggest, the default log level of WARNING should resolve all of the above concerns. Still, I'll probably manually configure logging in a manner respecting time causality... because I am pedantic and fear-based and want to use the phrase "in a manner respecting time causality" in a comment.

Thanks, @anntzer and @jklymak. We now return to your regularly scheduled development efforts.

@leycec

That's fair enough, but... reality intrudes. When project deadlines conflict with best practices, project deadlines always win. Our project alone has five large-scale mandatory dependencies (including matplotlib, NumPy, SciPy, and Pillow) and a bevy of large-scale optional dependencies (including NetworkX, PyDot, and ruamel.yaml).

@leycec Just as another point - it may be possible to test pre releases as part of your CI process, so you may not have to do it by hand. I'm no expert on CI, but I understand our nightly build does this with both python and numpy (i.e. it doesn't run for each PR). If more downstream developers added this, I think it'd be a huge help to us.

For PyCharm, it seems jet brains have a fix in: https://intellij-support.jetbrains.com/hc/en-us/community/posts/360000097690-AttributeError-module-matplotlib-has-no-attribute-verbose-?page=1#community_comment_360000089990

Navigation Menu

Repository navigation

Removal of verbose not documented: AttributeError: module 'matplotlib' has no attribute 'verbose' #10716

Bug report

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Removal of verbose not documented: AttributeError: module 'matplotlib' has no attribute 'verbose' #10716

Description

Bug report

Activity

jklymak commented on Mar 8, 2018

ImportanceOfBeingErnest commented on Mar 8, 2018

leycec commented on Mar 9, 2018

jklymak commented on Mar 9, 2018

leycec commented on Mar 9, 2018

jklymak commented on Mar 9, 2018

leycec commented on Mar 9, 2018

jklymak commented on Mar 9, 2018

leycec commented on Mar 9, 2018

leycec commented on Mar 9, 2018

anntzer commented on Mar 9, 2018

leycec commented on Mar 9, 2018

jklymak commented on Mar 9, 2018

jklymak commented on Mar 12, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions