Skip to content

Switching Compilers (Intel Related) #4518

Closed

Description

@frenchwr

Hello all, I have an intel-compiled version of Python 2.7.8 that I'm trying to build and install Matplotlib into. When I run "python setup.py build" everything builds fine until:

icc -pthread -fno-strict-aliasing -O3 -fPIC -fp-model strict -fomit-frame-pointer -DNDEBUG -g -O3 -Wall -fPIC -DMPL_DEVNULL=/dev/null -DPY_ARRAY_UNIQUE_SYMBOL=MPL_matplotlib__qhull_ARRAY_API -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/usr/local/python2/2.7.8/x86_64/intel14/nonet/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include -I/usr/include -I. -Iextern -I/usr/local/python2/2.7.8/x86_64/intel14/nonet/include/python2.7 -c src/qhull_wrap.c -o build/temp.linux-x86_64-2.7/src/qhull_wrap.o
In file included from src/qhull_wrap.c(10):
extern/qhull/qhull_a.h(106): warning #77: this declaration has no storage class or type specifier 
template <typename T> 
^

In file included from src/qhull_wrap.c(10):
extern/qhull/qhull_a.h(106): error: expected a ";"
template <typename T>
         ^

In file included from src/qhull_wrap.c(10):
extern/qhull/qhull_a.h(115): warning #12: parsing restarts here after previous syntax error
  void    qh_qhull(void);
                        ^

compilation aborted for src/qhull_wrap.c (code 2) 
error: command 'icc' failed with exit status 2

The issue here is that icc does not support templates. When I manually run this command with icpc rather than icc it compiles fine. Can someone advise me on how I can go about switching to icpc to compile qhull_wrap.c? I'm also happy to submit a pull request if this is something that can be generalized in the build process. Version info listed below.

[frenchwr@vmps11 matplotlib]$ uname -a
Linux vmps11 2.6.32-358.14.1.el6.x86_64 #1 SMP Tue Jul 16 23:51:20 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
[frenchwr@vmps11 matplotlib]$ lsb_release -a
LSB Version:    :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: CentOS
Description:    CentOS release 6.5 (Final)
Release:    6.5
Codename:   Final
[frenchwr@vmps11 matplotlib]$ python --version
Python 2.7.8
[frenchwr@vmps11 matplotlib]$ icc --version
icc (ICC) 14.0.2 20140120
Copyright (C) 1985-2014 Intel Corporation.  All rights reserved.
[frenchwr@vmps11 matplotlib]$ pip freeze
biopython==1.65
Cython==0.22
mock==1.0.1
nose==1.3.4
numpy==1.9.1
pandas==0.15.2
pyparsing==2.0.3
python-dateutil==2.4.0
pytz==2014.10
scikit-learn==0.15.2
scipy==0.15.1
seaborn==0.4.dev0
six==1.9.0
word-count==0.1.0
[frenchwr@vmps11 matplotlib]$ git describe
v1.4.3-1743-ga951b75

Activity

  1. tacaswell commented on Jun 12, 2015

    @tacaswell
    Member

    Taking a quick look at setupext.py it looks like we outsource this work to distutils (and maybe specifically distutils.sysconfig?). I suspect we are using it wrong and relying on gcc 'just working' if you throw c++ at it which we should fix, but that involves touching the build system which is a terrifying prospect.

    I think you can get it to behave by doing

    CC=icpc python setup.py install

    which gives me output like

    12:30 $ CC=icpc python setup.py develop
    ============================================================================
    Edit setup.cfg to change the build options
    
    BUILDING MATPLOTLIB
                matplotlib: yes [1.5.dev1]
                    python: yes [3.4.3 |Continuum Analytics, Inc.| (default, Mar
                            6 2015, 12:03:53)  [GCC 4.4.7 20120313 (Red Hat
                            4.4.7-1)]]
                  platform: yes [linux]
    
    REQUIRED DEPENDENCIES AND EXTENSIONS
                     numpy: yes [version 1.9.2]
                       six: yes [using six version 1.9.0]
                  dateutil: yes [using dateutil version 2.4.2]
                      pytz: yes [using pytz version 2015.4]
                   tornado: yes [using tornado version 4.1]
                 pyparsing: yes [using pyparsing version 2.0.3]
                    libagg: yes [pkg-config information for 'libagg' could not
                            be found. Using local copy.]
                  freetype: yes [version 2.5.2]
                       png: yes [version 1.5.13]
                     qhull: yes [pkg-config information for 'qhull' could not be
                            found. Using local copy.]
    
    OPTIONAL SUBPACKAGES
               sample_data: yes [installing]
                  toolkits: yes [installing]
                     tests: yes [using nose version 1.3.4 / using unittest.mock]
            toolkits_tests: yes [using nose version 1.3.4 / using unittest.mock]
    
    OPTIONAL BACKEND EXTENSIONS
                    macosx: no  [Mac OS-X only]
                    qt5agg: no  [PyQt5 not found]
                    qt4agg: yes [installing, Qt: 4.8.6, PyQt: 4.8.6; PySide not
                            found]
                   gtk3agg: no  [Requires pygobject to be installed.]
                 gtk3cairo: no  [Requires cairocffi or pycairo to be installed.]
                    gtkagg: no  [Requires pygtk]
                     tkagg: yes [installing, version not identified]
                     wxagg: no  [requires wxPython]
                       gtk: no  [Requires pygtk]
                       agg: yes [installing]
                     cairo: no  [cairocffi or pycairo not found]
                 windowing: no  [Microsoft Windows only]
    
    OPTIONAL LATEX DEPENDENCIES
                    dvipng: yes [version 1.14]
               ghostscript: yes [version 9.16]
                     latex: yes [version 3.14159265]
                   pdftops: yes [version 0.33.0]
    
    running develop
    running egg_info
    writing lib/matplotlib.egg-info/PKG-INFO
    writing requirements to lib/matplotlib.egg-info/requires.txt
    writing dependency_links to lib/matplotlib.egg-info/dependency_links.txt
    writing namespace_packages to lib/matplotlib.egg-info/namespace_packages.txt
    writing top-level names to lib/matplotlib.egg-info/top_level.txt
    reading manifest file 'lib/matplotlib.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'lib/matplotlib.egg-info/SOURCES.txt'
    running build_ext
    building 'matplotlib.ft2font' extension
    icpc -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPY_ARRAY_UNIQUE_SYMBOL=MPL_matplotlib_ft2font_ARRAY_API -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/home/tcaswell/.virtualenvs/dd_3k/lib/python3.4/site-packages/numpy/core/include -I/home/tcaswell/.virtualenvs/dd_3k/include/freetype2 -I/usr/local/include -I/usr/include -I. -I/home/tcaswell/.virtualenvs/dd_3k/include/python3.4m -c src/ft2font.cpp -o build/temp.linux-x86_64-3.4/src/ft2font.o
    unable to execute 'icpc': No such file or directory
    error: command 'icpc' failed with exit status 1
    
    

    which correctly fails as I don't have the intel compilers installed.

    Hope some of that is helpful!

  2. WeatherGod commented on Jun 12, 2015

    @WeatherGod
    Member

    I think you can do python setup.py build --compiler icpc or something to
    the effect?

    On Fri, Jun 12, 2015 at 12:22 PM, Will French notifications@github.com
    wrote:

    Hello all, I have an intel-compiled version of Python 2.7.8 that I'm
    trying to build and install Matplotlib into. When I run "python setup.py
    build" everything builds fine until:

    icc -pthread -fno-strict-aliasing -O3 -fPIC -fp-model strict -fomit-frame-pointer -DNDEBUG -g -O3 -Wall -fPIC -DMPL_DEVNULL=/dev/null -DPY_ARRAY_UNIQUE_SYMBOL=MPL_matplotlib__qhull_ARRAY_API -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/usr/local/python2/2.7.8/x86_64/intel14/nonet/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include -I/usr/include -I. -Iextern -I/usr/local/python2/2.7.8/x86_64/intel14/nonet/include/python2.7 -c src/qhull_wrap.c -o build/temp.linux-x86_64-2.7/src/qhull_wrap.o
    In file included from src/qhull_wrap.c(10):
    extern/qhull/qhull_a.h(106): warning #77: this declaration has no storage class or type specifier
    template
    ^

    In file included from src/qhull_wrap.c(10):
    extern/qhull/qhull_a.h(106): error: expected a ";"
    template
    ^

    In file included from src/qhull_wrap.c(10):
    extern/qhull/qhull_a.h(115): warning #12: parsing restarts here after previous syntax error
    void qh_qhull(void);
    ^

    compilation aborted for src/qhull_wrap.c (code 2)
    error: command 'icc' failed with exit status 2

    The issue here is that icc does not support templates. When I manually run
    this command with icpc rather than icc it compiles fine. Can someone advise
    me on how I can go about switching to icpc to compile qhull_wrap.c? I'm
    also happy to submit a pull request if this is something that can be
    generalized in the build process. Version info listed below.

    [frenchwr@vmps11 matplotlib]$ uname -a
    Linux vmps11 2.6.32-358.14.1.el6.x86_64 #1 SMP Tue Jul 16 23:51:20 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

    [frenchwr@vmps11 matplotlib]$ lsb_release -a
    LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
    Distributor ID: CentOS
    Description: CentOS release 6.5 (Final)
    Release: 6.5
    Codename: Final

    [frenchwr@vmps11 matplotlib]$ python --version
    Python 2.7.8

    [frenchwr@vmps11 matplotlib]$ icc --version
    icc (ICC) 14.0.2 20140120
    Copyright (C) 1985-2014 Intel Corporation. All rights reserved.

    [frenchwr@vmps11 matplotlib]$ pip freeze
    biopython==1.65
    Cython==0.22
    mock==1.0.1
    nose==1.3.4
    numpy==1.9.1
    pandas==0.15.2
    pyparsing==2.0.3
    python-dateutil==2.4.0
    pytz==2014.10
    scikit-learn==0.15.2
    scipy==0.15.1
    seaborn==0.4.dev0
    six==1.9.0
    word-count==0.1.0

    [frenchwr@vmps11 matplotlib]$ git describe
    v1.4.3-1743-ga951b75

    —
    Reply to this email directly or view it on GitHub
    #4518.

  3. frenchwr commented on Jun 12, 2015

    @frenchwr
    Author

    When I use CC=icpc python setup.py build the build makes a bit more progress but eventually bombs out with:

    building 'matplotlib._cntr' extension
    icpc -fno-strict-aliasing -O3 -fPIC -fp-model strict -fomit-frame-pointer -DNDEBUG -g -O3 -Wall -fPIC -DPY_ARRAY_UNIQUE_SYMBOL=MPL_matplotlib__cntr_ARRAY_API -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/usr/local/python2/2.7.8/x86_64/intel14/nonet/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include -I/usr/include -I. -I/usr/local/python2/2.7.8/x86_64/intel14/nonet/include/python2.7 -c src/cntr.c -o build/temp.linux-x86_64-2.7/src/cntr.o
    src/cntr.c(1846): error: a value of type "void *" cannot be assigned to an entity of type "char *"
          if (mpa) mask = PyArray_DATA(mpa);
                        ^
    
    src/cntr.c(1904): error: a value of type "void *" cannot be assigned to an entity of type "char *"
          data = PyArray_DATA(Cdata);
               ^
    
    compilation aborted for src/cntr.c (code 2)
    error: command 'icpc' failed with exit status 2

    This looks like the opposite problem as before, i.e. the C++ compiler is having trouble with C code. I then tried CC=icc CXX=icpc python setup.py build which triggers the original error. It does appear that in this case both icc and icpc are invoked, but it looks somewhat random...C files are occasionally compiled with icpc, C++ files occasionally compiled with icc, so I don't know what to make of that.

    I suspect there is a flag I can pass either icc to give it better C++ support, or icpc to give it better C support. I'll keep poking around.

  4. tacaswell commented on Jun 12, 2015

    @tacaswell
    Member

    https://mail.python.org/pipermail/python-dev/2012-March/117219.html <- is
    that helpful at all? I suspect you may get better help upstream talking to
    either the numpy/scipy folks or the distutils folks.

    Do you have similar issues compiling numpy/scipy/scikit-image ?

    On Fri, Jun 12, 2015 at 3:37 PM Will French notifications@github.com
    wrote:

    When I use CC=icpc python setup.py build the build makes a bit more
    progress but eventually bombs out with:

    building 'matplotlib._cntr' extension
    icpc -fno-strict-aliasing -O3 -fPIC -fp-model strict -fomit-frame-pointer -DNDEBUG -g -O3 -Wall -fPIC -DPY_ARRAY_UNIQUE_SYMBOL=MPL_matplotlib__cntr_ARRAY_API -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/usr/local/python2/2.7.8/x86_64/intel14/nonet/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include -I/usr/include -I. -I/usr/local/python2/2.7.8/x86_64/intel14/nonet/include/python2.7 -c src/cntr.c -o build/temp.linux-x86_64-2.7/src/cntr.o
    src/cntr.c(1846): error: a value of type "void *" cannot be assigned to an entity of type "char *"
    if (mpa) mask = PyArray_DATA(mpa);
    ^

    src/cntr.c(1904): error: a value of type "void *" cannot be assigned to an entity of type "char *"
    data = PyArray_DATA(Cdata);
    ^

    compilation aborted for src/cntr.c (code 2)
    error: command 'icpc' failed with exit status 2

    This looks like the opposite problem as before, i.e. the C++ compiler is
    having trouble with C code. I then tried CC=icc CXX=icpc python setup.py
    build which triggers the original error. It does appear that in this case
    both icc and icpc are invoked, but it looks somewhat random...C files are
    occasionally compiled with icpc, C++ files occasionally compiled with icc,
    so I don't know what to make of that.

    I suspect there is a flag I can pass either icc to give it better C++
    support, or icpc to give it better C support. I'll keep poking around.

    —
    Reply to this email directly or view it on GitHub
    #4518 (comment)
    .

  5. frenchwr commented on Jun 12, 2015

    @frenchwr
    Author

    Yes I battled with numpy for quite some time before getting it to build and pass all tests, but what I learned there does not help much in this case. I explored the use of different compiler flag options for both icc and icpc but I could not manage to get the matplotlib build to succeed.

    In the end, I decided to modify a few lines of code to remove Intel preprocessors and the template definition that triggered the original error. The build succeeded and no additional unit test errors or failures were introduced.

    See PR here: #4519

  6. tacaswell commented on Jun 14, 2015

    @tacaswell
    Member

    Reading this again, it looks like the errors you are getting from the icpc are finding sloppyness in our c code. See #4524

  7. tacaswell commented on Jun 19, 2015

    @tacaswell
    Member

    This has been fixed two different ways, closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions