Skip to content

Commit c58e6a3

Browse files
assignUserassignUser
authored
ARROW-17021: [C++][R][CI] Enable use of sccache in crossbow (apache#13556)
This is a first stab at enabling sccache, I used the R nightly job as an example as it compiles arrow on mac, win and linux. I want to open this for feedback specifically with an eye towards the cmake changes I implemented which enable this as long as the correct envvars (AWS creds + bucket) are set and sccache is available. My hope was to keep the changes required to activate sccache for existing jobs as minimal as possible. While working on this I also noticed that `ThirdpartyToolchain.cmake` is not build using (s)ccache, the flags had to be explicitly set. That fix alone should speed up CI across the board (in arrow and crossbow). For the R task this results in a build time reduction of 40-70% 🚀 Here a run of the same task without caching https://github.com/ursacomputing/crossbow/actions/runs/2563558583 cc: @kszucs @raulcd @kou Thanks for your input! Authored-by: Jacob Wujciak-Jens <jacob@wujciak.de> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
1 parent decddbb commit c58e6a3

23 files changed

Lines changed: 219 additions & 38 deletions

‎ci/docker/centos-7-cpp.dockerfile‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
FROM centos:centos7
1919

2020
RUN yum install -y \
21+
curl \
2122
diffutils \
2223
gcc-c++ \
2324
libcurl-devel \
@@ -31,8 +32,12 @@ ARG cmake=3.23.1
3132
RUN mkdir /opt/cmake-${cmake}
3233
RUN wget -nv -O - https://github.com/Kitware/CMake/releases/download/v${cmake}/cmake-${cmake}-Linux-x86_64.tar.gz | \
3334
tar -xzf - --strip-components=1 -C /opt/cmake-${cmake}
34-
ENV PATH=/opt/cmake-${cmake}/bin:$PATH
35-
ENV CC=/usr/bin/gcc
36-
ENV CXX=/usr/bin/g++
37-
ENV EXTRA_CMAKE_FLAGS="-DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX"
38-
ENV ARROW_R_DEV=TRUE
35+
36+
COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
37+
RUN bash /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin
38+
39+
ENV PATH=/opt/cmake-${cmake}/bin:$PATH \
40+
CC=/usr/bin/gcc \
41+
CXX=/usr/bin/g++ \
42+
EXTRA_CMAKE_FLAGS="-DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX" \
43+
ARROW_R_DEV=TRUE \

‎ci/docker/conda-cpp.dockerfile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,9 @@ RUN mamba install -q -y \
3838
COPY ci/scripts/install_gcs_testbench.sh /arrow/ci/scripts
3939
RUN /arrow/ci/scripts/install_gcs_testbench.sh default
4040

41+
COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
42+
RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin
43+
4144
ENV ARROW_BUILD_TESTS=ON \
4245
ARROW_DATASET=ON \
4346
ARROW_DEPENDENCY_SOURCE=CONDA \

‎ci/docker/conda.dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ FROM ${arch}/ubuntu:18.04
2121
# install build essentials
2222
RUN export DEBIAN_FRONTEND=noninteractive && \
2323
apt-get update -y -q && \
24-
apt-get install -y -q wget tzdata libc6-dbg gdb \
24+
apt-get install -y -q curl wget tzdata libc6-dbg gdb \
2525
&& apt-get clean \
2626
&& rm -rf /var/lib/apt/lists/*
2727

‎ci/docker/debian-10-cpp.dockerfile‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ RUN apt-get update -y -q && \
4040
ccache \
4141
clang-${llvm} \
4242
cmake \
43+
curl \
4344
g++ \
4445
gcc \
4546
gdb \
@@ -76,6 +77,9 @@ RUN apt-get update -y -q && \
7677
COPY ci/scripts/install_minio.sh /arrow/ci/scripts/
7778
RUN /arrow/ci/scripts/install_minio.sh latest /usr/local
7879

80+
COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
81+
RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin
82+
7983
ENV absl_SOURCE=BUNDLED \
8084
ARROW_BUILD_TESTS=ON \
8185
ARROW_DATASET=ON \

‎ci/docker/debian-11-cpp.dockerfile‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ RUN apt-get update -y -q && \
3737
ccache \
3838
clang-${llvm} \
3939
cmake \
40+
curl \
4041
g++ \
4142
gcc \
4243
gdb \
@@ -78,6 +79,9 @@ RUN /arrow/ci/scripts/install_minio.sh latest /usr/local
7879
COPY ci/scripts/install_gcs_testbench.sh /arrow/ci/scripts/
7980
RUN /arrow/ci/scripts/install_gcs_testbench.sh default
8081

82+
COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
83+
RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin
84+
8185
ENV absl_SOURCE=BUNDLED \
8286
ARROW_BUILD_TESTS=ON \
8387
ARROW_DATASET=ON \

‎ci/docker/fedora-35-cpp.dockerfile‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ RUN dnf update -y && \
3030
ccache \
3131
clang-devel \
3232
cmake \
33+
curl \
3334
curl-devel \
3435
flatbuffers-devel \
3536
gcc \
@@ -71,6 +72,9 @@ RUN /arrow/ci/scripts/install_minio.sh latest /usr/local
7172
COPY ci/scripts/install_gcs_testbench.sh /arrow/ci/scripts/
7273
RUN /arrow/ci/scripts/install_gcs_testbench.sh default
7374

75+
COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
76+
RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin
77+
7478
ENV absl_SOURCE=BUNDLED \
7579
ARROW_BUILD_TESTS=ON \
7680
ARROW_DEPENDENCY_SOURCE=SYSTEM \

‎ci/docker/ubuntu-18.04-cpp.dockerfile‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@ RUN apt-get update -y -q && \
6060
ca-certificates \
6161
ccache \
6262
cmake \
63+
curl \
6364
g++ \
6465
gcc \
6566
gdb \
@@ -100,6 +101,10 @@ RUN apt-get update -y -q && \
100101
# - s3 tests would require boost-asio that is included since Boost 1.66.0
101102
# ARROW-17051: this build uses static Protobuf, so we must also use
102103
# static Arrow to run Flight/Flight SQL tests
104+
105+
COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
106+
RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin
107+
103108
ENV ARROW_BUILD_STATIC=ON \
104109
ARROW_BUILD_TESTS=ON \
105110
ARROW_DATASET=ON \

‎ci/docker/ubuntu-20.04-cpp-minimal.dockerfile‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ RUN apt-get update -y -q && \
2828
build-essential \
2929
ccache \
3030
cmake \
31+
curl \
3132
git \
3233
libssl-dev \
3334
libcurl4-openssl-dev \
@@ -70,6 +71,9 @@ RUN /arrow/ci/scripts/install_minio.sh latest /usr/local
7071
COPY ci/scripts/install_gcs_testbench.sh /arrow/ci/scripts/
7172
RUN /arrow/ci/scripts/install_gcs_testbench.sh default
7273

74+
COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
75+
RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin
76+
7377
ENV ARROW_BUILD_TESTS=ON \
7478
ARROW_DATASET=ON \
7579
ARROW_FLIGHT=ON \

‎ci/docker/ubuntu-20.04-cpp.dockerfile‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@ RUN apt-get update -y -q && \
6868
ca-certificates \
6969
ccache \
7070
cmake \
71+
curl \
7172
g++ \
7273
gcc \
7374
gdb \
@@ -116,6 +117,9 @@ RUN /arrow/ci/scripts/install_gcs_testbench.sh default
116117
COPY ci/scripts/install_ceph.sh /arrow/ci/scripts/
117118
RUN /arrow/ci/scripts/install_ceph.sh
118119

120+
COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
121+
RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin
122+
119123
# Prioritize system packages and local installation
120124
# The following dependencies will be downloaded due to missing/invalid packages
121125
# provided by the distribution:

‎ci/docker/ubuntu-22.04-cpp-minimal.dockerfile‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ RUN apt-get update -y -q && \
2828
build-essential \
2929
ccache \
3030
cmake \
31+
curl \
3132
git \
3233
libssl-dev \
3334
libcurl4-openssl-dev \
@@ -70,6 +71,9 @@ RUN /arrow/ci/scripts/install_minio.sh latest /usr/local
7071
COPY ci/scripts/install_gcs_testbench.sh /arrow/ci/scripts/
7172
RUN /arrow/ci/scripts/install_gcs_testbench.sh default
7273

74+
COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
75+
RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin
76+
7377
ENV ARROW_BUILD_TESTS=ON \
7478
ARROW_DATASET=ON \
7579
ARROW_FLIGHT=ON \

0 commit comments

Comments
 (0)