Trouble installing from PyPI: error: ‘PyArray_Descr’ {aka ‘struct _PyArray_Descr’} has no member named ‘fields’ #169
Description
I'm having a time trying to build the PDAL library from source, and then install python-pdal with pip in a Docker container. This was working last week, so I strongly suspect my issues are related to the release of Numpy 2.0.0 and subsequent deprecation of some of their C APIs.
Strangely, I still can't get it to work when pinning Numpy to some version <2 (that still meets the python-pdal dependency requirements).
The whole error printout is:
------
> [6/8] RUN pip install pdal:
5.518 Collecting pdal
5.778 Downloading pdal-3.4.3.tar.gz (89 kB)
5.858 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.5/89.5 kB 1.0 MB/s eta 0:00:00
5.863 Installing build dependencies: started
14.25 Installing build dependencies: finished with status 'done'
14.25 Getting requirements to build wheel: started
14.31 Getting requirements to build wheel: finished with status 'done'
14.31 Preparing metadata (pyproject.toml): started
14.38 Preparing metadata (pyproject.toml): finished with status 'done'
14.38 Requirement already satisfied: numpy>=1.22 in /usr/local/lib/python3.11/site-packages (from pdal) (1.22.0)
14.38 Building wheels for collected packages: pdal
14.38 Building wheel for pdal (pyproject.toml): started
17.74 Building wheel for pdal (pyproject.toml): finished with status 'error'
17.74 error: subprocess-exited-with-error
17.74
17.74 × Building wheel for pdal (pyproject.toml) did not run successfully.
17.74 │ exit code: 1
17.74 ╰─> [42 lines of output]
17.74 *** scikit-build-core 0.9.6 using CMake 3.25.1 (wheel)
17.74 *** Configuring CMake...
17.74 loading initial cache file build/cp311-cp311-linux_x86_64/CMakeInit.txt
17.74 -- The C compiler identification is GNU 12.2.0
17.74 -- The CXX compiler identification is GNU 12.2.0
17.74 -- Detecting C compiler ABI info
17.74 -- Detecting C compiler ABI info - done
17.74 -- Check for working C compiler: /usr/bin/cc - skipped
17.74 -- Detecting C compile features
17.74 -- Detecting C compile features - done
17.74 -- Detecting CXX compiler ABI info
17.74 -- Detecting CXX compiler ABI info - done
17.74 -- Check for working CXX compiler: /usr/bin/c++ - skipped
17.74 -- Detecting CXX compile features
17.74 -- Detecting CXX compile features - done
17.74 -- Found Python3: /usr/local/bin/python (found version "3.11.9") found components: Interpreter Development.Module NumPy
17.74 -- Performing Test HAS_FLTO
17.74 -- Performing Test HAS_FLTO - Success
17.74 -- Found pybind11: /tmp/pip-build-env-m9teg40_/overlay/lib/python3.11/site-packages/pybind11/include (found version "2.12.0")
17.74 -- Configuring done
17.74 -- Generating done
17.74 -- Build files have been written to: /tmp/pip-install-_l8qrq6n/pdal_a380f1311cc54674ac4cc5d4e56ba631/build/cp311-cp311-linux_x86_64
17.74 *** Building project with Ninja...
17.74 [1/5] Building CXX object CMakeFiles/libpdalpython.dir/src/pdal/PyArray.cpp.o
17.74 FAILED: CMakeFiles/libpdalpython.dir/src/pdal/PyArray.cpp.o
17.74 /usr/bin/c++ -Dlibpdalpython_EXPORTS -I/tmp/pip-build-env-m9teg40_/overlay/lib/python3.11/site-packages/numpy/_core/include -isystem /usr/local/include/python3.11 -isystem /tmp/pip-build-env-m9teg40_/overlay/lib/python3.11/site-packages/pybind11/include -O3 -DNDEBUG -fPIC -fvisibility=hidden -flto -fno-fat-lto-objects -std=c++17 -MD -MT CMakeFiles/libpdalpython.dir/src/pdal/PyArray.cpp.o -MF CMakeFiles/libpdalpython.dir/src/pdal/PyArray.cpp.o.d -o CMakeFiles/libpdalpython.dir/src/pdal/PyArray.cpp.o -c /tmp/pip-install-_l8qrq6n/pdal_a380f1311cc54674ac4cc5d4e56ba631/src/pdal/PyArray.cpp
17.74 /tmp/pip-install-_l8qrq6n/pdal_a380f1311cc54674ac4cc5d4e56ba631/src/pdal/PyArray.cpp: In constructor ‘pdal::python::Array::Array(PyArrayObject*)’:
17.74 /tmp/pip-install-_l8qrq6n/pdal_a380f1311cc54674ac4cc5d4e56ba631/src/pdal/PyArray.cpp:100:29: error: ‘PyArray_Descr’ {aka ‘struct _PyArray_Descr’} has no member named ‘fields’
17.74 100 | int numFields = (dtype->fields == Py_None) ?
17.74 | ^~~~~~
17.74 /tmp/pip-install-_l8qrq6n/pdal_a380f1311cc54674ac4cc5d4e56ba631/src/pdal/PyArray.cpp:102:45: error: ‘PyArray_Descr’ {aka ‘struct _PyArray_Descr’} has no member named ‘fields’
17.74 102 | static_cast<int>(PyDict_Size(dtype->fields));
17.74 | ^~~~~~
17.74 /tmp/pip-install-_l8qrq6n/pdal_a380f1311cc54674ac4cc5d4e56ba631/src/pdal/PyArray.cpp:113:39: error: ‘PyArray_Descr’ {aka ‘struct _PyArray_Descr’} has no member named ‘fields’
17.74 113 | PyObject *names_dict = dtype->fields;
17.74 | ^~~~~~
17.74 [2/5] Building CXX object CMakeFiles/libpdalpython.dir/src/pdal/StreamableExecutor.cpp.o
17.74 [3/5] Building CXX object CMakeFiles/libpdalpython.dir/src/pdal/PyPipeline.cpp.o
17.74 [4/5] Building CXX object CMakeFiles/libpdalpython.dir/src/pdal/libpdalpython.cpp.o
17.74 ninja: build stopped: subcommand failed.
17.74
17.74 *** CMake build failed
17.74 [end of output]
17.74
17.74 note: This error originates from a subprocess, and is likely not a problem with pip.
17.74 ERROR: Failed building wheel for pdal
17.74 Failed to build pdal
17.74 ERROR: Could not build wheels for pdal, which is required to install pyproject.toml-based projects
------
Dockerfile:32
--------------------
30 |
31 | # Install Python pdal package
32 | >>> RUN pip install pdal
33 |
34 | # Clean up unnecessary files to reduce image size
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install pdal" did not complete successfully: exit code: 1
The Dockerfile being built looks like this (simplified from the original)
# Stage 1: Build PDAL
FROM python:3.11
# Install build dependencies
RUN apt-get update && apt-get install -y \
cmake \
build-essential \
ninja-build \
wget \
libgdal-dev \
&& apt-get clean
# I was sure that pinning Numpy would make it work, but no luck
RUN pip install "numpy==1.22"
# Download and extract PDAL source code
RUN wget https://github.com/PDAL/PDAL/releases/download/2.7.1/PDAL-2.7.1-src.tar.bz2 -O /tmp/PDAL-2.7.1-src.tar.bz2 \
&& mkdir -p /tmp/pdal-src \
&& tar -xjf /tmp/PDAL-2.7.1-src.tar.bz2 -C /tmp/pdal-src --strip-components=1 \
&& rm /tmp/PDAL-2.7.1-src.tar.bz2
# Build and install PDAL using Ninja
RUN cd /tmp/pdal-src \
&& cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/local . \
&& ninja \
&& ninja install
# Set environment variables for runtime
ENV LD_LIBRARY_PATH=/usr/local/lib:${LD_LIBRARY_PATH}
ENV CMAKE_PREFIX_PATH=/usr/local
# Install Python pdal package
RUN pip install pdal
# Clean up unnecessary files to reduce image size
RUN apt-get remove --purge -y \
wget \
build-essential \
cmake \
git \
&& apt-get autoremove -y && apt-get clean
# Verify installation
RUN ldconfig && python -c "import pdal; print('PDAL imported successfully')"Thanks in advance for any help!