Skip to content

Commit 70c6708

Browse files
assignUserjonkeane
assignUser
and
jonkeane
authored
ARROW-12590: [C++][R] Update copies of Homebrew files to reflect recent updates (apache#13769)
Building on apache#12157 Lead-authored-by: Jacob Wujciak-Jens <jacob@wujciak.de> Co-authored-by: Jonathan Keane <jkeane@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
1 parent 8474ee5 commit 70c6708

7 files changed

Lines changed: 189 additions & 51 deletions

File tree

‎LICENSE.txt‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1990,12 +1990,14 @@ for PyArrow. Ibis is released under the Apache License, Version 2.0.
19901990

19911991
This project includes code from the autobrew project.
19921992

1993-
* r/tools/autobrew and dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb
1994-
are based on code from the autobrew project.
1993+
The following files are based on code from the autobrew project:
1994+
* r/tools/autobrew
1995+
* dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb
1996+
* dev/tasks/homebrew-formulae/autobrew/apache-arrow-static.rb
19951997

19961998
Copyright (c) 2019, Jeroen Ooms
19971999
License: MIT
1998-
Homepage: https://github.com/jeroen/autobrew
2000+
Homepage: https://github.com/autobrew/
19992001

20002002
--------------------------------------------------------------------------------
20012003

‎dev/tasks/homebrew-formulae/autobrew/apache-arrow-static.rb‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
# Cloned from https://github.com/autobrew/homebrew-core/blob/high-sierra/Formula/apache-arrow.rb
19+
# Upstream any relevant changes (dependencies, build args ...)
20+
21+
class ApacheArrowStatic < Formula
22+
desc "Columnar in-memory analytics layer designed to accelerate big data"
23+
homepage "https://arrow.apache.org/"
24+
url "https://www.apache.org/dyn/closer.lua?path=arrow/arrow-8.0.0.9000/apache-arrow-8.0.0.9000.tar.gz"
25+
# Uncomment and update to test on a release candidate
26+
# mirror "https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-8.0.0-rc1/apache-arrow-8.0.0.tar.gz"
27+
sha256 "9948ddb6d4798b51552d0dca3252dd6e3a7d0f9702714fc6f5a1b59397ce1d28"
28+
head "https://github.com/apache/arrow.git"
29+
30+
bottle do
31+
sha256 cellar: :any, arm64_big_sur: "ef89d21a110b89840cc6148add685d407e75bd633bc8f79625eb33d00e3694b4"
32+
sha256 cellar: :any, big_sur: "6fcb9f55d44eb61d295a8020e039a0622bdc044b220cfffef855f3e3ab8057a1"
33+
sha256 cellar: :any, catalina: "bf71b17766688077fb9b4e679f309742c16524015287dd3633758c679752c977"
34+
end
35+
36+
depends_on "boost" => :build
37+
depends_on "cmake" => :build
38+
depends_on "aws-sdk-cpp-static"
39+
depends_on "brotli"
40+
depends_on "lz4"
41+
depends_on "snappy"
42+
depends_on "thrift"
43+
depends_on "zstd"
44+
45+
conflicts_with "apache-arrow", because: "both install Arrow"
46+
47+
def install
48+
ENV.cxx11
49+
# https://github.com/Homebrew/homebrew-core/issues/94724
50+
# https://issues.apache.org/jira/browse/ARROW-15664
51+
ENV["HOMEBREW_OPTIMIZATION_LEVEL"] = "O2"
52+
53+
args = %W[
54+
-DARROW_BUILD_SHARED=OFF
55+
-DARROW_BUILD_UTILITIES=ON
56+
-DARROW_COMPUTE=ON
57+
-DARROW_CSV=ON
58+
-DARROW_DATASET=ON
59+
-DARROW_FILESYSTEM=ON
60+
-DARROW_GCS=ON
61+
-DARROW_JEMALLOC=ON
62+
-DARROW_JSON=ON
63+
-DARROW_MIMALLOC=ON
64+
-DARROW_PARQUET=ON
65+
-DARROW_S3=ON
66+
-DARROW_VERBOSE_THIRDPARTY_BUILD=ON
67+
-DARROW_WITH_BROTLI=ON
68+
-DARROW_WITH_BZ2=ON
69+
-DARROW_WITH_LZ4=ON
70+
-DARROW_WITH_SNAPPY=ON
71+
-DARROW_WITH_ZLIB=ON
72+
-DARROW_WITH_ZSTD=ON
73+
-DLZ4_HOME=#{Formula["lz4"].prefix}
74+
-DPARQUET_BUILD_EXECUTABLES=ON
75+
-DTHRIFT_HOME=#{Formula["thrift"].prefix}
76+
]
77+
78+
mkdir "build"
79+
cd "build" do
80+
system "cmake", "../cpp", *std_cmake_args, *args
81+
system "make"
82+
system "make", "install"
83+
end
84+
end
85+
86+
test do
87+
(testpath/"test.cpp").write <<~EOS
88+
#include "arrow/api.h"
89+
int main(void) {
90+
arrow::int64();
91+
return 0;
92+
}
93+
EOS
94+
system ENV.cxx, "test.cpp", "-std=c++11", "-I#{include}", "-L#{lib}", \
95+
"-larrow", "-larrow_bundled_dependencies", "-o", "test"
96+
system "./test"
97+
end
98+
end

‎dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb‎

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
# under the License.
1717

1818
# https://github.com/autobrew/homebrew-core/blob/master/Formula/apache-arrow.rb
19+
1920
class ApacheArrow < Formula
2021
desc "Columnar in-memory analytics layer designed to accelerate big data"
2122
homepage "https://arrow.apache.org/"
@@ -25,15 +26,15 @@ class ApacheArrow < Formula
2526

2627
bottle do
2728
cellar :any
28-
sha256 "a55211ba6f464681b7ca1b48defdad9cfbe1cf6fad8ff9ec875dc5a3c8f3c5ed" => :el_capitan_or_later
29+
sha256 "9cd44700798638b5e3ee8774b3929f3fad815290d05572d1f39f01d6423eaad0" => :high_sierra
2930
root_url "https://autobrew.github.io/bottles"
3031
end
3132

3233
# NOTE: if you add something here, be sure to add to PKG_LIBS in r/tools/autobrew
3334
depends_on "boost" => :build
34-
depends_on "brotli"
3535
depends_on "cmake" => :build
3636
depends_on "aws-sdk-cpp"
37+
depends_on "brotli"
3738
depends_on "lz4"
3839
depends_on "snappy"
3940
depends_on "thrift"
@@ -49,24 +50,20 @@ def install
4950
-DARROW_DATASET=ON
5051
-DARROW_FILESYSTEM=ON
5152
-DARROW_GCS=ON
52-
-DARROW_HDFS=OFF
5353
-DARROW_JEMALLOC=ON
5454
-DARROW_JSON=ON
5555
-DARROW_MIMALLOC=ON
5656
-DARROW_PARQUET=ON
57-
-DARROW_PYTHON=OFF
5857
-DARROW_S3=ON
59-
-DARROW_USE_GLOG=OFF
6058
-DARROW_VERBOSE_THIRDPARTY_BUILD=ON
6159
-DARROW_WITH_BROTLI=ON
6260
-DARROW_WITH_BZ2=ON
6361
-DARROW_WITH_LZ4=ON
6462
-DARROW_WITH_SNAPPY=ON
6563
-DARROW_WITH_ZLIB=ON
6664
-DARROW_WITH_ZSTD=ON
67-
-DCMAKE_UNITY_BUILD=OFF
68-
-DPARQUET_BUILD_EXECUTABLES=ON
6965
-DLZ4_HOME=#{Formula["lz4"].prefix}
66+
-DPARQUET_BUILD_EXECUTABLES=ON
7067
-DTHRIFT_HOME=#{Formula["thrift"].prefix}
7168
]
7269

@@ -86,7 +83,8 @@ def install
8683
return 0;
8784
}
8885
EOS
89-
system ENV.cxx, "test.cpp", "-std=c++11", "-I#{include}", "-L#{lib}", "-larrow", "-o", "test"
86+
system ENV.cxx, "test.cpp", "-std=c++11", "-I#{include}", "-L#{lib}", \
87+
"-larrow", "-larrow_bundled_dependencies", "-o", "test"
9088
system "./test"
9189
end
9290
end

‎dev/tasks/macros.jinja‎

Lines changed: 24 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -226,6 +226,21 @@ on:
226226
done
227227
{% endmacro %}
228228

229+
{%- macro pin_brew_formulae(is_fork = false) -%}
230+
{#- This macro expects the cwd to be arrow/r -#}
231+
# Copy all *brew formulae
232+
cp ../dev/tasks/homebrew-formulae/autobrew/apache-arrow*.rb tools/
233+
234+
# Pin the git commit in the formula to match
235+
cd tools
236+
if [ "{{ is_fork }}" == "true" ]; then
237+
sed -i.bak -E -e 's/apache\/arrow.git"$/{{ arrow.github_repo.split("/") | join("\/") }}.git", :revision => "'"{{ arrow.head }}"'"/' apache-arrow*.rb
238+
else
239+
sed -i.bak -E -e 's/arrow.git"$/arrow.git", :revision => "'"{{ arrow.head }}"'"/' apache-arrow*.rb
240+
fi
241+
rm -f apache-arrow*.rb.bak
242+
{% endmacro %}
243+
229244
{%- macro github_change_r_pkg_version(is_fork, version = '\\2.\'\"$(date +%Y%m%d)\"\'' ) -%}
230245
- name: Modify version
231246
shell: bash
@@ -236,16 +251,8 @@ on:
236251
DESCRIPTION
237252
head DESCRIPTION
238253
rm -f DESCRIPTION.bak
239-
cp ../dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb tools/apache-arrow.rb
240254

241-
# Pin the git commit in the formula to match
242-
cd tools
243-
if [ "{{ is_fork }}" == "true" ]; then
244-
sed -i.bak -E -e 's/apache\/arrow.git"$/{{ arrow.github_repo.split("/") | join("\/") }}.git", :revision => "'"{{ arrow.head }}"'"/' apache-arrow.rb
245-
else
246-
sed -i.bak -E -e 's/arrow.git"$/arrow.git", :revision => "'"{{ arrow.head }}"'"/' apache-arrow.rb
247-
fi
248-
rm -f apache-arrow.rb.bak
255+
{{ pin_brew_formulae(is_fork) }}
249256
{% endmacro %}
250257

251258
{%- macro github_test_r_src_pkg() -%}
@@ -309,3 +316,11 @@ on:
309316
# Set envvar for later steps by appending to $GITHUB_ENV
310317
write(paste0("R_PROFILE_USER=", profile_path), file = Sys.getenv("GITHUB_ENV"), append = TRUE)
311318
{% endmacro %}
319+
320+
{# Detect if we are using a fork or the upstream repo #}
321+
{% set is_upstream_b = arrow.github_repo == 'apache/arrow' %}
322+
{# use filter to cast to string and convert to lowercase to match yaml boolean #}
323+
{% set is_fork = (not is_upstream_b)|lower %}
324+
325+
{% set r_release = '4.2' %}
326+
{% set r_oldrel = '4.1' %}

‎dev/tasks/r/github.macos.autobrew.yml‎

Lines changed: 39 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -16,44 +16,60 @@
1616
# under the License.
1717

1818
{% import 'macros.jinja' as macros with context %}
19+
{% set is_fork = macros.is_fork %}
1920

2021
{{ macros.github_header() }}
2122

2223
jobs:
2324
autobrew:
24-
name: "Autobrew"
25-
runs-on: macOS-10.15
25+
name: "Autobrew {{ "${{ matrix.platform }}" }}"
26+
runs-on: {{ "${{ matrix.platform }}" }}
27+
strategy:
28+
fail-fast: false
29+
matrix:
30+
platform:
31+
- macos-11
32+
- macos-10.13 # self-hosted
33+
r-version:
34+
- {{ macros.r_release }}
35+
- {{ macros.r_oldrel }}
2636
steps:
2737
{{ macros.github_checkout_arrow()|indent }}
28-
2938
- name: Configure autobrew script
3039
run: |
40+
# minio is pre-installed on the self-hosted 10.13 runer
41+
if [ {{ '${{ matrix.platform }}' }} != macos-10.13 ]; then
42+
# install minio for tests
43+
brew install minio
44+
fi
3145
cd arrow/r
32-
# Put the formula inside r/ so that it's included in the package build
33-
cp ../dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb tools/apache-arrow.rb
34-
# Pin the current commit in the formula to test so that we're not always pulling from master
35-
sed -i.bak -E -e 's@https://github.com/apache/arrow.git"$@{{ arrow.remote }}.git", :revision => "{{ arrow.head }}"@' tools/apache-arrow.rb && rm -f tools/apache-arrow.rb.bak
36-
# Sometimes crossbow gives a remote URL with .git and sometimes not. Make sure there's only one
37-
sed -i.bak -E -e 's@.git.git@.git@' tools/apache-arrow.rb && rm -f tools/apache-arrow.rb.bak
38-
# Get minio for S3 testing
39-
brew install minio
40-
- uses: r-lib/actions/setup-r@v1
41-
- name: Install dependencies
42-
run: |
43-
install.packages("remotes")
44-
remotes::install_deps("arrow/r", dependencies = TRUE)
45-
remotes::install_cran(c("rcmdcheck", "sys", "sessioninfo"))
46-
shell: Rscript {0}
47-
- name: Session info
46+
{{ macros.pin_brew_formulae(is_fork)|indent }}
47+
- uses: r-lib/actions/setup-r@v2
48+
if: matrix.platform != 'macos-10.13'
49+
with:
50+
r-version: {{ '${{ matrix.r-version }}' }}
51+
use-public-rspm: true
52+
- name: Setup R
53+
if: matrix.platform == 'macos-10.13'
4854
run: |
49-
options(width = 100)
50-
pkgs <- installed.packages()[, "Package"]
51-
sessioninfo::session_info(pkgs, include_base = TRUE)
52-
shell: Rscript {0}
55+
# rig is a system utility that allows for switching
56+
# between pre-installed R version on the self-hosted runners
57+
rig default {{ '${{ matrix.r-version }}' }}
58+
rig system setup-user-lib
59+
rig system add-pak
60+
- uses: r-lib/actions/setup-r-dependencies@v2
61+
with:
62+
working-directory: 'arrow/r'
63+
extra-packages: |
64+
any::rcmdcheck
65+
any::sys
66+
any::readr
5367
- name: Check
5468
env:
5569
_R_CHECK_CRAN_INCOMING_: false
70+
NOT_CRAN: true
5671
ARROW_USE_PKG_CONFIG: false
72+
ARROW_R_DEV: true
5773
run: arrow/ci/scripts/r_test.sh arrow
5874
- name: Dump install logs
5975
run: cat arrow/r/check/arrow.Rcheck/00install.out

‎dev/tasks/r/github.packages.yml‎

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,7 @@
2121
# crossbow submit --param custom_version=8.5.3 r-binary-packages
2222
# if the param is unset defaults to the usual Ymd naming scheme
2323
{% set package_version = custom_version|default("\\2.\'\"$(date +%Y%m%d)\"\'") %}
24-
# We need this as boolean and string
25-
{% set is_upstream_b = arrow.github_repo == 'apache/arrow' %}
26-
# use filter to cast to string and convert to lowercase to match yaml boolean
27-
{% set is_fork = (not is_upstream_b)|lower %}
28-
{% set is_upstream = is_upstream_b|lower %}
29-
24+
{% set is_fork = macros.is_fork %}
3025

3126
{{ macros.github_header() }}
3227

‎r/tools/autobrew‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,15 @@
1919
export HOMEBREW_NO_ANALYTICS=1
2020
export HOMEBREW_NO_AUTO_UPDATE=1
2121

22-
# Official Homebrew no longer supports El-Capitan
23-
UPSTREAM_ORG="autobrew"
22+
if [[ ${OSTYPE:6} -ge 20 ]]; then
23+
# We are on a modern enough macOS, we can use the real brew
24+
UPSTREAM_ORG="homebrew"
25+
PKG_BREW_NAME="$PKG_BREW_NAME-static"
26+
else
27+
# Official Homebrew no longer supports El-Capitan
28+
# so we need to use the forked autobrew version of brew that supports old macOSes
29+
UPSTREAM_ORG="autobrew"
30+
fi
2431

2532
if [ "$DISABLE_AUTOBREW" ]; then return 0; fi
2633
AUTOBREW=${TMPDIR-/tmp}
@@ -36,6 +43,13 @@ curl -fsSL https://github.com/$UPSTREAM_ORG/brew/tarball/master | tar xz --strip
3643
export HOMEBREW_CACHE="$AUTOBREW"
3744
LOCAL_FORMULA="tools/${PKG_BREW_NAME}.rb"
3845
if [ -f "$LOCAL_FORMULA" ]; then
46+
if [[ ${OSTYPE:6} -ge 20 ]]; then
47+
$BREW tap
48+
49+
# Tap https://github.com/autobrew/homebrew-cran so that we can get dependencies from there
50+
$BREW tap autobrew/cran
51+
fi
52+
3953
# Use the local brew formula and install --HEAD
4054
$BREW deps -n "$LOCAL_FORMULA" 2>/dev/null
4155
BREW_DEPS=$($BREW deps -n "$LOCAL_FORMULA" 2>/dev/null)

0 commit comments

Comments
 (0)