You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
saving a figure triggers (very) excessive IO activity #1530
Saving a figure from the GUI under GTK triggers a lot of IO activity. Up to 60Mb/sec on my system.
The file /.local/share/recently-used.xbel gets written to each time the session history is scrolled through with cursor keys.
I am running matplotlib 1.2.0 but this issue is, I believe, much older than that.
I reported the issue to IPython (see ipython/ipython#2608) first thinking this was a purely IPy problem but I could then reproduce the bug in a normal python session.
I think you have provided the essential clue to a problem I was running into some time ago (ipython/ipython#1184). Unfortunately, I don't see any good solution. The problem seems to be deeply embedded. Although everything I have found indicates it is a gtk problem, using the qt4 backend does not solve it; my guess is that the qt4 toolkit is conforming to some standard for activity reporting such that the underlying gtk window manager is being notified and is accessing the infamous recently-used.xbel file. Using the tk backend seems to avoid this; please try that on your system and see if it provides a workaround for you.
Solutions suggested on the web all involve trying to make recently-used.xbel inaccessible in some way; the problem is that the attempted access still triggers a gtk warning message every time.
I tried the tk backend and it indeed circumvents the problem. But I was not able to use the GUI to save the figure, I got an exception:
Exception in Tkinter callback
Traceback (most recent call last):
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 1413, in call
return self.func(*args)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/backends/backend_tkagg.py", line 882, in save_figure
default_filetype_name = filetypes[default_filetype]
KeyError: 'auto'
Why on earth would we need the auto mode? Is a drop down list with e.g png on top not the best option?
If you set savefig.format in matplotlibrc, indeed, I can reproduce this bug. Is that a setting you entered yourself? I don't think savefig.format == 'auto' is supposed to have any special meaning.
I had a look at my matplotlibrc. Everything is default (i.e the commented version)m except the backend selection which is
backend : GTKAgg
the savefig part is
# the default savefig params can be different from the display params
# Eg, you may want a higher resolution, or to make the figure
# background white
#savefig.dpi : 100 # figure dots per inch
#savefig.facecolor : white # figure facecolor when saving
#savefig.edgecolor : white # figure edgecolor when saving
#savefig.extension : auto # what extension to use for savefig('foo'), or 'auto'
I am positive I did not edit this.
If I now set it to png, this is what I get
>>> import pylab as p
/usr/local/lib/python2.7/dist-packages/matplotlib/__init__.py:695: UserWarning: savefig.extension is deprecated and replaced with savefig.format; please use the latter.
warnings.warn(self.msg_depr % (key, alt))
The gui comes up with png extension though. I edited my matplotlibrc according to the new convention and warning is no longer raised. I guess installing the latest matplotlib did not overwrite my old matplotlibrc.
Now, as for the original bug, IO activity. I tried to reproduce the bug from script using savefig. And it seems I never trigger it (I created and saved some hundreds of pics in a loop). As soon as I use the gui, bingo...
The gui calls savefig too, that the gtk does some homekeeping and writes frequetly used once is ok, but why does the history browsing have to retrigger that? Looks more like a readline problem to me ...
(EDIT: @pelson fixed shouting, 😄, by adding code blocks code)
That's a pretty serious bug. What's happening here is that we deprecated savefig.extension -- if the user has a savefig.extension in their matplotlibrc, we set savefig.format to its value. Unfortunately the new savefig.format does not expect auto as a value. This is something one would only see if they had a matplotlibrc from an earlier version and then upgraded matplotlib without upgrading their matplotlibrc. I'll have to add a fix for this.
I'm still investigating why saving from Gtk+ has such high IO -- still unable to reproduce -- I might send you some further things to investigate if I think of anything.
Yes, please!
I did a test on a colleague's computer. He had matplotlib installed but has never used it (no comment).
I tried saving a fig with the gui and I could see that there was some IO when I scrolled the history (using iotop), nevertheless this did not really slow the shell down. I had a look at his /.local/share/recently-used.xbel and the file was brand new and only some kb. In comparison my recently-used was almost 1Mb. I cleared it and tried again. Now the bug is still to be detected but one could possibly not notice it without accurate monitoring because this does not slow down the machine much (still, this remains not overly nice).
So does that suggest that this isn't a bug with matplotlib, but instead with Gtk/Gnome's recently used handling? We don't ever touch or deal with that file directly, but simply use the standard dialog that Gtk provides.
Well, it seems even worse than I thought.
In order to reproduce the IO problem, it is enough to press the save button and open the FileChooserDialog. Even after canceling the operation before actually saving, the damn recently_used file gets written to at each cursor key press!
So I guess it is really a GTK problem...
I'll try my luck and submit a bug report there.
Thanks for your input.
Two of us here also seeing this issue. After one opening of the save figure dialog, the recently-used.xbel file is continuously written upon keyboard input in the terminal.
Unfortunately, it seems they 'solved' it by not using the new GTK file selector but they do report this little gem:
"This is because, when using the newer chooser, Gtk writes out the 'Recently Used Files' list everytime the Gtk main loop exits.". Would that also be true in IPython?
It seems it came up again later for vim, to which the gtk folks claimed they were misusing Gtk's main loop. If that is also true for IPython/PyGTK/matplotlib (I don't even understand what uses what), then we/they/you should perhaps weigh in on the debate here: https://bugzilla.gnome.org/show_bug.cgi?id=546691
With or without ipython, we are calling main_quit() only when the last figure is closed in non-interactive mode. If main_quit is getting called with every cursor motion, it must be getting done inside of gtk itself.
We still have a major bug here. To reproduce (on linux, ubuntu Precise virtual machine, mpl master):
Bring up two terminal windows. In the first:
strace -o testgtk ipython --pylab=gtk
In the second:
grep recently-used testgtk | wc
Now, you can periodically use the second window to see what is causing accesses to recently-used. It turns out that if you don't bring up the gtk savefig file chooser at all, there are no accesses. But if you do bring it up, even without saving, then forever after, each keystroke in the ipython window triggers eleven accesses! So evidently the gtk file chooser is leaving something nasty running in the background.
Activity
efiring commentedon Nov 24, 2012
I think you have provided the essential clue to a problem I was running into some time ago (ipython/ipython#1184). Unfortunately, I don't see any good solution. The problem seems to be deeply embedded. Although everything I have found indicates it is a gtk problem, using the qt4 backend does not solve it; my guess is that the qt4 toolkit is conforming to some standard for activity reporting such that the underlying gtk window manager is being notified and is accessing the infamous recently-used.xbel file. Using the tk backend seems to avoid this; please try that on your system and see if it provides a workaround for you.
Solutions suggested on the web all involve trying to make recently-used.xbel inaccessible in some way; the problem is that the attempted access still triggers a gtk warning message every time.
jfmoulin commentedon Nov 25, 2012
I tried the tk backend and it indeed circumvents the problem. But I was not able to use the GUI to save the figure, I got an exception:
Exception in Tkinter callback
Traceback (most recent call last):
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 1413, in call
return self.func(*args)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/backends/backend_tkagg.py", line 882, in save_figure
default_filetype_name = filetypes[default_filetype]
KeyError: 'auto'
Why on earth would we need the auto mode? Is a drop down list with e.g png on top not the best option?
efiring commentedon Nov 25, 2012
I can't reproduce that problem; for me, saving via the gui works fine, and the file selector comes up with a png default.
mdboom commentedon Nov 26, 2012
If you set
savefig.formatin matplotlibrc, indeed, I can reproduce this bug. Is that a setting you entered yourself? I don't thinksavefig.format == 'auto'is supposed to have any special meaning.jfmoulin commentedon Nov 27, 2012
I had a look at my matplotlibrc. Everything is default (i.e the commented version)m except the backend selection which is
backend : GTKAgg
the savefig part is
I am positive I did not edit this.
If I now set it to png, this is what I get
The gui comes up with png extension though. I edited my matplotlibrc according to the new convention and warning is no longer raised. I guess installing the latest matplotlib did not overwrite my old matplotlibrc.
Now, as for the original bug, IO activity. I tried to reproduce the bug from script using savefig. And it seems I never trigger it (I created and saved some hundreds of pics in a loop). As soon as I use the gui, bingo...
The gui calls savefig too, that the gtk does some homekeeping and writes frequetly used once is ok, but why does the history browsing have to retrigger that? Looks more like a readline problem to me ...
(EDIT: @pelson fixed shouting, 😄, by adding code blocks
code)jfmoulin commentedon Nov 27, 2012
oops, sorry I should have previewed... I did not intend shouting at you ;0)
mdboom commentedon Nov 27, 2012
That's a pretty serious bug. What's happening here is that we deprecated
savefig.extension-- if the user has asavefig.extensionin theirmatplotlibrc, we setsavefig.formatto its value. Unfortunately the newsavefig.formatdoes not expectautoas a value. This is something one would only see if they had a matplotlibrc from an earlier version and then upgraded matplotlib without upgrading their matplotlibrc. I'll have to add a fix for this.I'm still investigating why saving from Gtk+ has such high IO -- still unable to reproduce -- I might send you some further things to investigate if I think of anything.
jfmoulin commentedon Nov 27, 2012
Yes, please!
I did a test on a colleague's computer. He had matplotlib installed but has never used it (no comment).
I tried saving a fig with the gui and I could see that there was some IO when I scrolled the history (using iotop), nevertheless this did not really slow the shell down. I had a look at his /.local/share/recently-used.xbel and the file was brand new and only some kb. In comparison my recently-used was almost 1Mb. I cleared it and tried again. Now the bug is still to be detected but one could possibly not notice it without accurate monitoring because this does not slow down the machine much (still, this remains not overly nice).
mdboom commentedon Nov 27, 2012
So does that suggest that this isn't a bug with matplotlib, but instead with Gtk/Gnome's recently used handling? We don't ever touch or deal with that file directly, but simply use the standard dialog that Gtk provides.
jfmoulin commentedon Nov 29, 2012
Well, it seems even worse than I thought.
In order to reproduce the IO problem, it is enough to press the save button and open the FileChooserDialog. Even after canceling the operation before actually saving, the damn recently_used file gets written to at each cursor key press!
So I guess it is really a GTK problem...
I'll try my luck and submit a bug report there.
Thanks for your input.
If the user has "savefig.extension : auto" (deprecated) in their (old…
oliford commentedon May 17, 2013
Did this get reported at Gtk? I can't see a link.
Two of us here also seeing this issue. After one opening of the save figure dialog, the recently-used.xbel file is continuously written upon keyboard input in the terminal.
There seems to be an old related bug in vim ( http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=456897 ).
Unfortunately, it seems they 'solved' it by not using the new GTK file selector but they do report this little gem:
"This is because, when using the newer chooser, Gtk writes out the 'Recently Used Files' list everytime the Gtk main loop exits.". Would that also be true in IPython?
oliford commentedon May 17, 2013
It seems it came up again later for vim, to which the gtk folks claimed they were misusing Gtk's main loop. If that is also true for IPython/PyGTK/matplotlib (I don't even understand what uses what), then we/they/you should perhaps weigh in on the debate here: https://bugzilla.gnome.org/show_bug.cgi?id=546691
efiring commentedon May 17, 2013
With or without ipython, we are calling main_quit() only when the last figure is closed in non-interactive mode. If main_quit is getting called with every cursor motion, it must be getting done inside of gtk itself.
We still have a major bug here. To reproduce (on linux, ubuntu Precise virtual machine, mpl master):
Bring up two terminal windows. In the first:
strace -o testgtk ipython --pylab=gtkIn the second:
grep recently-used testgtk | wcNow, you can periodically use the second window to see what is causing accesses to recently-used. It turns out that if you don't bring up the gtk savefig file chooser at all, there are no accesses. But if you do bring it up, even without saving, then forever after, each keystroke in the ipython window triggers eleven accesses! So evidently the gtk file chooser is leaving something nasty running in the background.
jfmoulin commentedon May 23, 2013
Hello,
sorry for the late reaction.... yes I actually reported the bug at gnome under
https://bugzilla.gnome.org/show_bug.cgi?id=689292
but this does not seem to raise substantial interest.
backend_gtk: don't hide FileChooserDialog; closes matplotlib#1530
Merge pull request #2056 from efiring/gtk_filechooser_no_hide