[Bug]: 3.5.1 on mips64el: segfault when running on tests #21789

Activity

1. jklymak commented on Nov 29, 2021

   

   jklymak

   on Nov 29, 2021

   Member

   More actions

   This is dying on test_agg.py::test_large_single_path_collection which says:

       # Generates a too-large single path in a path collection that
       # would cause a segfault if the draw_markers optimization is
       # applied.
   

   Looks like we are causing a segfault, so therefore are applying the optimization? This logic looks like it was tweaked in 53d2549 by @tacaswell, so maybe he has some ideas if something was missed?
2. QuLogic commented on Nov 30, 2021

   

   QuLogic

   on Nov 30, 2021

   Member

   More actions

   That commit didn't change any C code, just made the limits bigger. It's possibly overflowing now.

   Can you get a complete backtrace with gdb?
3. tacaswell commented on Nov 30, 2021

   

   tacaswell

   on Nov 30, 2021

   Member

   More actions

   Due to our minor re-writing of history, github has lost the ability to point to the orign PR: #19343 . The new version of the commit that is likely at fault is d0b5550 . @sandrotosi Can you revert that patch on the broken system?
4. sandrotosi commented on Nov 30, 2021

   

   sandrotosi

   on Nov 30, 2021

   ContributorAuthor

   More actions

   @sandrotosi Can you revert that patch on the broken system?

   only on mips64el and not on any other architecture? i'm not sure if that's supported by our build tools, i'll investigate
5. QuLogic commented on Nov 30, 2021

   

   QuLogic

   on Nov 30, 2021

   Member

   More actions

   Have you tried 3.4.3 again? It does not appear to pass the tests either, so I'm not sure this is a regression in Matplotlib.
6. QuLogic commented on Nov 30, 2021

   

   QuLogic

   on Nov 30, 2021

   Member

   More actions

   The crash is here:
   

   matplotlib/src/_backend_agg.h

   Line 658 in 9765379

   agg::render_scanlines(sa, sl, rendererAA);

   
   and I did not look any further than that. Reverting d0b5550 does not fix the crash.
7. sandrotosi commented on Dec 1, 2021

   

   sandrotosi

   on Dec 1, 2021

   ContributorAuthor

   More actions

   (the git reorg makes it kinda tricky to navigate old prs and commit) i just had a look at the previous builds of mpl in Debian, and this was already happening in 3.3.4, then all the other builds from that version until the last one were fine.

   Even 3.5.0 built fine once and then failed in a subsequent upload that was basically a no-code change release (enable of sphinx-panels).

   It is very possible that the code causing the segfault has been there for a while, but now it looks like other packages have maybe started to stress the same faulty codepath that previously only mpl tests were using.
8. olebole commented on Dec 9, 2021

   

   olebole

   on Dec 9, 2021

   Last edited by olebole

   More actions

   FYI Debian#1001168 (comment) contains a backtrace for this issue, and also a minimal example code by @picca.
9. sandrotosi commented on Dec 24, 2021

   

   sandrotosi

   on Dec 24, 2021

   ContributorAuthor

   More actions

   is there something we (debian) can do to help you figure out what's going on? since we build all matplotlib reverse-dependencies on mips64el too, and with mpl segfaulting there, many of them are also failing, causing quite a bit of a backlog on our end.

   thanks!
10. picca commented on Dec 24, 2021

    

    picca

    on Dec 24, 2021 via email

    More actions

    for info, I rebuilt 3.5.0 on bullseye mips64el chroot and it works. numpy switched from 1.19.5 to 1.21, so maybe this is something related to this newer numpy. here the info I found during my day's of debuging https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1001168#57
11. tacaswell commented on Dec 24, 2021

    

    tacaswell

    on Dec 24, 2021

    Member

    More actions

    Try also reverting 26e00af ?

    Seems trivial, but previously copied paths lost of they should be simplified or not which could in turn be changing details of which c-code gets run which may have brought this out.

    Can someone who has access to a mips64 machine bisect based on @picca 's MWE?

    ---

    (the git reorg makes it kinda tricky to navigate old prs and commit)

    Sorry about that, we accidentally merged a whole virtualenv and its removal into our history.
12. sandrotosi commented on Dec 27, 2021

    

    sandrotosi

    on Dec 27, 2021

    ContributorAuthor

    More actions

    just tried reverting 26e00af (and only that commit) and the test suite failed in the same manner.
13. sandrotosi commented on Dec 27, 2021

    

    sandrotosi

    on Dec 27, 2021

    ContributorAuthor

    More actions

    i tested:

    3.5.0
    3.4.3
    3.3.4
    3.2.2

    all of them sigsegv on the same test.

    this becomes evident the problem is somewhere else than mpl, likely numpy as picca mentioned. i'll start looking into that now
14. sandrotosi commented on Dec 28, 2021

    

    sandrotosi

    on Dec 28, 2021

    ContributorAuthor

    More actions

    i've run again that single test in gdb and got

    (gdb) r -m pytest build/lib.linux-mips64-3.10/matplotlib/tests/test_agg.py::test_large_single_path_collection --verbose --full-trace
    Starting program: /usr/bin/python3.10 -m pytest build/lib.linux-mips64-3.10/matplotlib/tests/test_agg.py::test_large_single_path_collection --verbose --full-trace
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib/mips64el-linux-gnuabi64/libthread_db.so.1".
    ============================================================================================================================= test session starts ==============================================================================================================================
    platform linux -- Python 3.10.1, pytest-6.2.5, py-1.10.0, pluggy-0.13.0 -- /usr/bin/python3.10
    cachedir: .pytest_cache
    rootdir: /home/morph/matplotlib, configfile: pytest.ini
    collecting ... [Detaching after vfork from child process 7446]
    [Detaching after vfork from child process 7447]
    collected 1 item                                                                                                                                                                                                                                                               
    
    build/lib.linux-mips64-3.10/matplotlib/tests/test_agg.py::test_large_single_path_collection 
    Program received signal SIGSEGV, Segmentation fault.
    agg::pixfmt_alpha_blend_rgba<fixed_blender_rgba_plain<agg::rgba8T<agg::linear>, agg::order_rgba>, agg::row_accessor<unsigned char> >::blend_solid_hspan (covers=0x10000000000 <error: Cannot access memory at address 0x10000000000>, c=..., len=<optimized out>, y=534, 
        x=<optimized out>, this=<optimized out>) at extern/agg24-svn/include/agg_color_rgba.h:395
    395             static AGG_INLINE value_type multiply(value_type a, value_type b)
    (gdb) 
    

    that seems to point to something in extern/agg24-svn and not numpy? there's no different between 3.4.3 and 3.5.0 in that directory, so maybe it's another library matplotlib includes that we should be looking at, but which one?
15. sandrotosi commented on Dec 28, 2021

    

    sandrotosi

    on Dec 28, 2021

    ContributorAuthor

    More actions

    continuing with the investigation, i checked what packages were different between the last good build (-) and the first one that started SIGSEGVing (+) and here's an extract of the relevant ones (the full diff can be obtained with this command: diff -u <(curl -s 'https://buildd.debian.org/status/fetch.php?pkg=matplotlib&arch=mips64el&ver=3.3.4-2&stamp=1633211581&raw=1') <(curl -s 'https://buildd.debian.org/status/fetch.php?pkg=matplotlib&arch=mips64el&ver=3.3.4-2%2Bb1&stamp=1637459674&raw=1') , packages diff starts searching the string Buildinfo):

    - g++ (= 4:10.2.1-1),
    - g++-10 (= 10.3.0-11),
    - gcc (= 4:10.2.1-1),
    - gcc-10 (= 10.3.0-11),
    - gcc-10-base (= 10.3.0-11),
    - gcc-11-base (= 11.2.0-8),
    + g++ (= 4:11.2.0-2),
    + g++-11 (= 11.2.0-10),
    + gcc (= 4:11.2.0-2),
    + gcc-11 (= 11.2.0-10),
    + gcc-11-base (= 11.2.0-10),
    
    - libfreetype-dev (= 2.10.4+dfsg-1),
    - libfreetype6 (= 2.10.4+dfsg-1),
    - libfreetype6-dev (= 2.10.4+dfsg-1),
    + libfreetype-dev (= 2.11.0+dfsg-1),
    + libfreetype6 (= 2.11.0+dfsg-1),
    + libfreetype6-dev (= 2.11.0+dfsg-1),
    
    - libgcc-10-dev (= 10.3.0-11),
    - libgcc-s1 (= 11.2.0-8),
    + libgcc-11-dev (= 11.2.0-10),
    + libgcc-s1 (= 11.2.0-10),
    
    - libgl1-mesa-dri (= 21.2.2-1),
    - libglapi-mesa (= 21.2.2-1),
    - libglib2.0-0 (= 2.70.0-1+b1),
    - libglibmm-2.4-1v5 (= 2.66.1-1),
    + libgl1-mesa-dri (= 21.2.5-1),
    + libglapi-mesa (= 21.2.5-1),
    + libglib2.0-0 (= 2.70.1-1),
    + libglibmm-2.4-1v5 (= 2.66.2-1),
    
    (note: python 3.10 is added to the list of supported versions)
    - libpython3-all-dev (= 3.9.2-3),
    - libpython3-dev (= 3.9.2-3),
    - libpython3-stdlib (= 3.9.2-3),
    - libpython3.9 (= 3.9.7-4),
    - libpython3.9-dev (= 3.9.7-4),
    - libpython3.9-minimal (= 3.9.7-4),
    - libpython3.9-stdlib (= 3.9.7-4),
    - libqt5core5a (= 5.15.2+dfsg-12),
    - libqt5dbus5 (= 5.15.2+dfsg-12),
    + libpython3-all-dev (= 3.9.8-1),
    + libpython3-dev (= 3.9.8-1),
    + libpython3-stdlib (= 3.9.8-1),
    + libpython3.10 (= 3.10.0-5),
    + libpython3.10-dev (= 3.10.0-5),
    + libpython3.10-minimal (= 3.10.0-5),
    + libpython3.10-stdlib (= 3.10.0-5),
    + libpython3.9 (= 3.9.9-1),
    + libpython3.9-dev (= 3.9.9-1),
    + libpython3.9-minimal (= 3.9.9-1),
    + libpython3.9-stdlib (= 3.9.9-1),
    + libqt5core5a (= 5.15.2+dfsg-13),
    + libqt5dbus5 (= 5.15.2+dfsg-13),
      libqt5designer5 (= 5.15.2-5),
    
    - libstdc++-10-dev (= 10.3.0-11),
    - libstdc++6 (= 11.2.0-8),
    + libstdc++-11-dev (= 11.2.0-10),
    + libstdc++6 (= 11.2.0-10),
    
    - linux-libc-dev (= 5.14.6-3),
    + linux-libc-dev (= 5.14.16-1),
    
    - python3 (= 3.9.2-3),
    + python3 (= 3.9.8-1),
      python3-alabaster (= 0.7.12-1),
    - python3-all (= 3.9.2-3),
    - python3-all-dev (= 3.9.2-3),
    - python3-argon2 (= 18.3.0-2+b2),
    + python3-all (= 3.9.8-1),
    + python3-all-dev (= 3.9.8-1),
    + python3-argon2 (= 21.1.0-1),
    
    - python3-cffi (= 1.14.6-2),
    - python3-cffi-backend (= 1.14.6-2+b1),
    + python3-cffi (= 1.15.0-1),
    + python3-cffi-backend (= 1.15.0-1+b1),
    
    - python3-numpy (= 1:1.19.5-1),
    + python3-numpy (= 1:1.21.4-2),
    

    does any of them ring a bell to you guys?
16. sandrotosi

    changed the title [-][Bug]: 3.5.0 on mips64el: segfault when running on tests[/-] [+][Bug]: 3.5.1 on mips64el: segfault when running on tests[/+] on Jan 3, 2022
17. sandrotosi commented on Jan 3, 2022

    

    sandrotosi

    on Jan 3, 2022

    ContributorAuthor

    More actions

    I've upgraded matplotlib to 3.5.1 in Debian, and this is the confirmation the test suite still SIGSEGVs when running on mips64el; full build log here
18. picca commented on Jan 3, 2022

    

    picca

    on Jan 3, 2022

    More actions

    Hello I investigated a bit more and compiled with gcc-10 and no-lto

    https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1001168#72

    no segfault with gcc-10 and no-lto.
19. QuLogic commented on Jan 5, 2022

    

    QuLogic

    on Jan 5, 2022

    Member

    More actions

    - g++ (= 4:10.2.1-1),
    - g++-10 (= 10.3.0-11),
    - gcc (= 4:10.2.1-1),
    - gcc-10 (= 10.3.0-11),
    - gcc-10-base (= 10.3.0-11),
    - gcc-11-base (= 11.2.0-8),
    + g++ (= 4:11.2.0-2),
    + g++-11 (= 11.2.0-10),
    + gcc (= 4:11.2.0-2),
    + gcc-11 (= 11.2.0-10),
    + gcc-11-base (= 11.2.0-10),
    
    - libgcc-10-dev (= 10.3.0-11),
    - libgcc-s1 (= 11.2.0-8),
    + libgcc-11-dev (= 11.2.0-10),
    + libgcc-s1 (= 11.2.0-10),
    

    Maybe; probably, even?

    - libfreetype-dev (= 2.10.4+dfsg-1),
    - libfreetype6 (= 2.10.4+dfsg-1),
    - libfreetype6-dev (= 2.10.4+dfsg-1),
    + libfreetype-dev (= 2.11.0+dfsg-1),
    + libfreetype6 (= 2.11.0+dfsg-1),
    + libfreetype6-dev (= 2.11.0+dfsg-1),
    

    There is some text in the test, but that seems unlikely if everything else is fine.

    - libgl1-mesa-dri (= 21.2.2-1),
    - libglapi-mesa (= 21.2.2-1),
    - libglib2.0-0 (= 2.70.0-1+b1),
    - libglibmm-2.4-1v5 (= 2.66.1-1),
    + libgl1-mesa-dri (= 21.2.5-1),
    + libglapi-mesa (= 21.2.5-1),
    + libglib2.0-0 (= 2.70.1-1),
    + libglibmm-2.4-1v5 (= 2.66.2-1),
    

    Highly unlikely.

    (note: python 3.10 is added to the list of supported versions)
    - libpython3-all-dev (= 3.9.2-3),
    - libpython3-dev (= 3.9.2-3),
    - libpython3-stdlib (= 3.9.2-3),
    - libpython3.9 (= 3.9.7-4),
    - libpython3.9-dev (= 3.9.7-4),
    - libpython3.9-minimal (= 3.9.7-4),
    - libpython3.9-stdlib (= 3.9.7-4),
    - libqt5core5a (= 5.15.2+dfsg-12),
    - libqt5dbus5 (= 5.15.2+dfsg-12),
    + libpython3-all-dev (= 3.9.8-1),
    + libpython3-dev (= 3.9.8-1),
    + libpython3-stdlib (= 3.9.8-1),
    + libpython3.10 (= 3.10.0-5),
    + libpython3.10-dev (= 3.10.0-5),
    + libpython3.10-minimal (= 3.10.0-5),
    + libpython3.10-stdlib (= 3.10.0-5),
    + libpython3.9 (= 3.9.9-1),
    + libpython3.9-dev (= 3.9.9-1),
    + libpython3.9-minimal (= 3.9.9-1),
    + libpython3.9-stdlib (= 3.9.9-1),
    + libqt5core5a (= 5.15.2+dfsg-13),
    + libqt5dbus5 (= 5.15.2+dfsg-13),
      libqt5designer5 (= 5.15.2-5),
    

    Also doubtful.

    - libstdc++-10-dev (= 10.3.0-11),
    - libstdc++6 (= 11.2.0-8),
    + libstdc++-11-dev (= 11.2.0-10),
    + libstdc++6 (= 11.2.0-10),
    

    Agg is in C++, so maybe, but I don't think it uses very much of the standard library.

    - linux-libc-dev (= 5.14.6-3),
    + linux-libc-dev (= 5.14.16-1),
    

    Maybe, insomuch as libc affects everything.

    - python3 (= 3.9.2-3),
    + python3 (= 3.9.8-1),
      python3-alabaster (= 0.7.12-1),
    - python3-all (= 3.9.2-3),
    - python3-all-dev (= 3.9.2-3),
    - python3-argon2 (= 18.3.0-2+b2),
    + python3-all (= 3.9.8-1),
    + python3-all-dev (= 3.9.8-1),
    + python3-argon2 (= 21.1.0-1),
    
    - python3-cffi (= 1.14.6-2),
    - python3-cffi-backend (= 1.14.6-2+b1),
    + python3-cffi (= 1.15.0-1),
    + python3-cffi-backend (= 1.15.0-1+b1),
    

    Nope.

    - python3-numpy (= 1:1.19.5-1),
    + python3-numpy (= 1:1.21.4-2),
    

    Maybe.
20. sandrotosi commented on Jan 29, 2022

    

    sandrotosi

    on Jan 29, 2022

    ContributorAuthor

    More actions

    this turned out to be a bug in gcc-11 on mips64* architectures (-O1/-O2 may generated bad code, while -O0/-O3 appear to be fine), which in debian we worked around by passing -O3 to gcc
21. QuLogic

    added

    status: upstream fix required

    on Jan 31, 2022
22. QuLogic commented on Jan 31, 2022

    

    QuLogic

    on Jan 31, 2022

    Member

    More actions

    I'll close this then as an upstream issue.
23. QuLogic

    closed this as completedon Jan 31, 2022
24. yangfl commented on Mar 14, 2022

    

    yangfl

    on Mar 14, 2022

    More actions

    FYI following issue: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104914