Skip to content

disk IO activity on cursor press #2608

Closed

Description

@jfmoulin

Hi,

this is an exact duplicate of issue #396 which was closed without getting a real solution.

I frequently get a very slow IPython session with a lot of IO activity triggered by cursor actions (up/dowm history and also moving left/right).
IOtop shows writes up to 50-60Mb/s.
This is always correlated to working interactivelly with matplotlib (memory use is not noticeably higher as usual). Simple stuff, no monster object or very complex figure.

I use Ipython 13.1 and matplotlib (1.2.0), but this problem is older than both these versions...
Running on Linux Mint 13 64 bit wit readline5 5.2.11 and readline6 6.2.8 (could it be a problem to have both libraries???)

Thanks a lot in advance... this is litterally a show stopper!
JF

Activity

  1. takluyver commented on Nov 20, 2012

    @takluyver
    Member

    Is there any way to determine what file it's writing to? I can't think why IPython, matplotlib or readline should be doing a lot of IO on cursor movements.

  2. rkern commented on Nov 20, 2012

    @rkern
    Contributor

    lsof(1) will show what files are opened by the IPython process. strace(1) may give more real-time information about the reads and writes.

  3. jfmoulin commented on Nov 20, 2012

    @jfmoulin
    Author

    Thanks for the tips.
    I am now trying to reproduce the problem, and of course it does not want to appear.
    I'll post a lsof output as soon as I get it again...
    Thanks for the fast answers anyway!

  4. takluyver commented on Nov 23, 2012

    @takluyver
    Member

    Thanks, @ajhorton, any hint of a pattern is a step forwards. Unfortunately, I can't replicate the problem, though. This is what I tried:

    IPython master, matplotlib master, Ubuntu 12.10

    ipython --pylab
    
    ...
    
    In [2]: abc = 123  # This was testing responsiveness
    
    In [3]: plot(arange(10))
    Out[3]: [<matplotlib.lines.Line2D at 0xa3ac6ec>]
    
    # Clicked the save button in the matplotlib window, saved as PNG to the desktop
    
    # (Console remains responsive)
    
  5. jfmoulin commented on Nov 23, 2012

    @jfmoulin
    Author

    Hi!

    @ajhorton, you nailed it down! I thought I was trying everything but missed to save a pic.
    Using pylab.savefig is apparently not a problem but as soon as I save a figure from the GUI I trigger this monster IO activity. I tried several times and the bug always shows up.

    I produced a dump of lsof and strace (I'll mail it tgz (200kb) to the takluyver and rkern hoping that this is the right way to proceed... let me know if I could put it online somewhere...).
    JF

  6. jfmoulin commented on Nov 23, 2012

    @jfmoulin
    Author

    Well, me again... I guess the best way is to use something like filebin:
    http://filebin.ca/NdspC6wfTV3 links to the tar with the two dumps.
    JF

  7. bfroehle commented on Nov 24, 2012

    @bfroehle
    Contributor

    Reading through the strace log shows that the program is repeatedly re-creating /home/jfmoulin/.local/share/recently-used.xbel.

    The repeated loop appears to be the following. Now I'm not sure where the loop "starts" but I chose what I thought was a reasonable start.

    open("/home/jfmoulin/.local/share/recently-used.xbel.05HRNW", O_RDWR|O_CREAT|O_EXCL, 0666) = 8
    fcntl(8, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
    fstat(8, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
    mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5d2651d000
    lseek(8, 0, SEEK_CUR)                   = 0
    write(8, "<?xml version=\"1.0\" encoding=\"UT"..., 983040) = 983040
    write(8, "ions>\n          <bookmark:applic"..., 3396) = 3396
    fstatfs(8, {f_type=0xf15f, f_bsize=4096, f_blocks=53597515, f_bfree=9962170, f_bavail=7279047, f_files=13418496, f_ffree=13311717, f_fsid={975816597, -635617892}, f_namelen=143, f_frsize=4096}) = 0
    lstat("/home/jfmoulin/.local/share/recently-used.xbel", {st_mode=S_IFREG|0600, st_size=986436, ...}) = 0
    fsync(8)                                = 0
    close(8)                                = 0
    munmap(0x7f5d2651d000, 4096)            = 0
    rename("/home/jfmoulin/.local/share/recently-used.xbel.05HRNW", "/home/jfmoulin/.local/share/recently-used.xbel") = 0
    chmod("/home/jfmoulin/.local/share/recently-used.xbel", 0600) = 0
    rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
    read(0, "[", 1)                         = 1
    select(1, [0], NULL, NULL, {0, 100000}) = 1 (in [0], left {0, 99991})
    fstat(0, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0
    fcntl(0, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
    write(11, "\1\0\0\0\0\0\0\0", 8)        = 8
    recvfrom(10, 0x374eec4, 4096, 0, 0, 0)  = -1 EAGAIN (Resource temporarily unavailable)
    poll([{fd=11, events=POLLIN}, {fd=10, events=POLLIN}, {fd=5, events=POLLIN}, {fd=7, events=POLLIN}, {fd=0, events=POLLIN}], 5, 0) = 2 ([{fd=11, revents=POLLIN}, {fd=0, revents=POLLIN}])
    read(11, "\6\0\0\0\0\0\0\0", 16)        = 8
    write(11, "\1\0\0\0\0\0\0\0", 8)        = 8
    write(11, "\1\0\0\0\0\0\0\0", 8)        = 8
    write(11, "\1\0\0\0\0\0\0\0", 8)        = 8
    write(11, "\1\0\0\0\0\0\0\0", 8)        = 8
    write(11, "\1\0\0\0\0\0\0\0", 8)        = 8
    poll([{fd=10, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=10, revents=POLLOUT}])
    writev(10, [{"+\0\1\0", 4}, {NULL, 0}, {"", 0}], 3) = 4
    poll([{fd=10, events=POLLIN}], 1, -1)   = 1 ([{fd=10, revents=POLLIN}])
    recvfrom(10, "\1\2J#\0\0\0\0\5\0 \3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 4096, 0, NULL, NULL) = 32
    recvfrom(10, 0x374eec4, 4096, 0, 0, 0)  = -1 EAGAIN (Resource temporarily unavailable)
    recvfrom(10, 0x374eec4, 4096, 0, 0, 0)  = -1 EAGAIN (Resource temporarily unavailable)
    
  8. takluyver commented on Nov 24, 2012

    @takluyver
    Member

    Apparently recently-used.xbel is something used by GTK. I tried with ipython --pylab gtk, and I think I could just see a tiny slowdown after saving a plot. Is there anything notable about that file on your machine - like it being really big, or on a network drive?

    @bfroehle , I've had no experience of reading strace logs, but does that mean it's trying to write to the file and failing?

    I wonder if the matplotlib guys would have a better idea about debugging this.

  9. jfmoulin commented on Nov 24, 2012

    @jfmoulin
    Author

    /.local/share/recently-used.xbel is 967K no idea if this has to be considered big. It is on a local disk, which by the way is a ssd... so not the kind of disk you want rewritten in an endless loop.
    Some people apparently try to prevent this file to be used by making it read only, but some temporary files with a similar name might be created though.

    I could reproduce the problem in a normal python shell, so I guess I should post the report on Matplotlib's issue tracker

  10. takluyver commented on Nov 24, 2012

    @takluyver
    Member

    Yep, I think reporting it to matplotlib is a good idea. Keep us posted
    about it!

    On 24 November 2012 16:20, jfmoulin notifications@github.com wrote:

    /.local/share/recently-used.xbel is 967K no idea if this has to be
    considered big. It is on a local disk, which by the way is a ssd... so not
    the kind of disk you want rewritten in an endless loop.
    Some people apparently try to prevent this file to be used by making it
    read only, but some temporary files with a similar name might be created
    though.

    I could reproduce the problem in a normal python shell, so I guess I
    should post the report on Matplotlib's issue tracker

    —
    Reply to this email directly or view it on GitHubhttps://github.com//issues/2608#issuecomment-10679217.

  11. bfroehle commented on Nov 24, 2012

    @bfroehle
    Contributor

    Yes, please report it to Matplotlib, but they might send you upstream to PyGTK (or similar).

  12. devedetti commented on Aug 16, 2013

    @devedetti

    Hi, I have an SSD disk an every times that I press enter under ipython with dummy code (example: In [19]: a = 2), a write disk is maked. It is noticeable for a some lag between press enter and see the result!

  13. takluyver commented on Aug 16, 2013

    @takluyver
    Member

    @nachopro : Is this connected to matplotlib, as described in the discussion above? If so, see the matplotlib issue. If not, open a new issue.

  14. added this to the no action milestone on Mar 26, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions