Skip to content

Commit b4e3017

Browse files
kszucskou
kszucs
authored and
kou
committed
ARROW-2836: [Packaging] Expand build matrices to multiple tasks
Author: Krisztián Szűcs <szucs.krisztian@gmail.com> Author: Kouhei Sutou <kou@clear-code.com> Closes apache#2259 from kszucs/taskpp and squashes the following commits: 7938cfc [Kouhei Sutou] [Packaging] Ensure recreating GTK-Doc related build files 5441623 [Kouhei Sutou] [Packaging] Fix a form e0408c1 [Kouhei Sutou] [Packaging] Fix a form 5bb2957 [Kouhei Sutou] [Packaging] Fix forms bc401a7 [Kouhei Sutou] [Packaging] update available task names a74b0e6 [Kouhei Sutou] [Packaging] linux-packages is invalid e96ae45 [Kouhei Sutou] [Packaging] pip doesn't have -y option 74dbf1c [Kouhei Sutou] [Packaging] Stop to use travis_wait b299c6f [Krisztián Szűcs] couple of comments; little readme update fc923b3 [Krisztián Szűcs] update package signing to conform the changes a8875c0 [Krisztián Szűcs] enable deb parallel builds c1de236 [Krisztián Szűcs] query tasks artifacts in job status 8ca72b9 [Kouhei Sutou] [Packaging] Fill artifacts for Ubuntu 17.10 and 18.04 784e85a [Kouhei Sutou] [Packaging] Stop to suppress build log a1320ea [Kouhei Sutou] [deb] Add missing Protocol Buffers related packages to Ubuntu 17.10 5892473 [Krisztián Szűcs] list expected linux package artifacts for el6, el7, xenial, trusty fb7d62b [Krisztián Szűcs] package format parameter for linux tasks bd233ae [Krisztián Szűcs] fix deploy tags in templates e391f3a [Krisztián Szűcs] fox osx rpaths for boost 6b735af [Krisztián Szűcs] re-export env variables for osx wheels e8fc1d5 [Krisztián Szűcs] linux tasks b7c71e0 [Krisztián Szűcs] fix validation order 7cf0e9c [Krisztián Szűcs] remove build matricies from ci configurations; expand to multiple tasks instead
1 parent 2c90eff commit b4e3017

16 files changed

Lines changed: 542 additions & 354 deletions

File tree

‎c_glib/Makefile.am‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ SUBDIRS = \
2626

2727
EXTRA_DIST = \
2828
README.md \
29+
autogen.sh \
2930
meson.build \
3031
meson_options.txt \
3132
test

‎c_glib/autogen.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,4 +23,4 @@ set -e
2323
mkdir -p m4
2424

2525
gtkdocize --copy
26-
autoreconf --install
26+
autoreconf --install --force

‎dev/tasks/README.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,7 @@ submission. The tasks are defined in `tasks.yml`
8585

8686
```bash
8787
# pygit2 requires libgit2: http://www.pygit2.org/install.html
88-
pip install -y jinja2 pygit2 click ruamel.yaml setuptools_scm github3.py python-gnupg
88+
pip install jinja2 pygit2 click ruamel.yaml setuptools_scm github3.py python-gnupg
8989
```
9090

9191
7. Try running it:
@@ -127,7 +127,7 @@ The script does the following:
127127
to build conda recipes on linux it will create a new branch:
128128
`crossbow@build-<id>-conda-linux`.
129129
3. Pushes the modified branches to GitHub which triggers the builds.
130-
For authentication it uses github oauth tokens described in the install
130+
For authentication it uses GitHub OAuth tokens described in the install
131131
section.
132132

133133

@@ -150,14 +150,14 @@ The script accepts a pattern as a first argument to narrow the build scope:
150150
Run multiple builds:
151151

152152
```bash
153-
$ python crossbow.py submit linux-packages conda-linux wheel-win
153+
$ python crossbow.py submit debian-stretch conda-linux-py36 wheel-win-py36
154154
Repository: https://github.com/kszucs/arrow@tasks
155155
Commit SHA: 810a718836bb3a8cefc053055600bdcc440e6702
156156
Version: 0.9.1.dev48+g810a7188.d20180414
157157
Pushed branches:
158-
- linux-packages
159-
- conda-linux
160-
- wheel-win
158+
- debian-stretch
159+
- conda-linux-py36
160+
- wheel-win-py36
161161
```
162162

163163
Just render without applying or committing the changes:
@@ -166,14 +166,14 @@ Just render without applying or committing the changes:
166166
$ python crossbow.py submit --dry-run task_name
167167
```
168168

169-
Run only `conda` package builds but on all platforms:
169+
Run only `conda` package builds and a Linux one:
170170

171171
```bash
172-
$ python crossbow.py submit conda-win conda-osx conda-linux
172+
$ python crossbow.py submit -g conda centos-7
173173
```
174174

175175
Run `wheel` builds:
176176

177177
```bash
178-
$ python crossbow.py submit wheel-osx wheel-linux wheel-win
178+
$ python crossbow.py submit --group wheel
179179
```

‎dev/tasks/conda-recipes/appveyor.yml‎

Lines changed: 17 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -17,37 +17,31 @@
1717

1818
environment:
1919
ARROW_VERSION: {{ arrow.version }}
20-
matrix:
21-
- CONFIG: variants\win_c_compilervs2015cxx_compilervs2015python3.5.yaml
22-
CONDA_INSTALL_LOCN: C:\\Miniconda35-x64
23-
24-
- CONFIG: variants\win_c_compilervs2015cxx_compilervs2015python3.6.yaml
25-
CONDA_INSTALL_LOCN: C:\\Miniconda36-x64
2620

2721
# We always use a 64-bit machine, but can build x86 distributions
2822
# with the TARGET_ARCH variable.
2923
platform:
30-
- x64
24+
- x64
3125

3226
install:
33-
# Cywing's git breaks conda-build. (See https://github.com/conda-forge/conda-smithy-feedstock/pull/2.)
34-
- cmd: rmdir C:\cygwin /s /q
27+
# Cywing's git breaks conda-build. (See https://github.com/conda-forge/conda-smithy-feedstock/pull/2.)
28+
- cmd: rmdir C:\cygwin /s /q
3529

36-
# Add path, activate `conda` and update conda.
37-
- cmd: call %CONDA_INSTALL_LOCN%\Scripts\activate.bat
38-
- cmd: conda.exe update --yes --quiet conda
30+
# Add path, activate `conda` and update conda.
31+
- cmd: call {{ conda_install_location }}\Scripts\activate.bat
32+
- cmd: conda.exe update --yes --quiet conda
3933

40-
- cmd: set PYTHONUNBUFFERED=1
34+
- cmd: set PYTHONUNBUFFERED=1
4135

42-
# Add our channels.
43-
- cmd: conda.exe config --set show_channel_urls true
44-
- cmd: conda.exe config --remove channels defaults
45-
- cmd: conda.exe config --add channels defaults
46-
- cmd: conda.exe config --add channels conda-forge
36+
# Add our channels.
37+
- cmd: conda.exe config --set show_channel_urls true
38+
- cmd: conda.exe config --remove channels defaults
39+
- cmd: conda.exe config --add channels defaults
40+
- cmd: conda.exe config --add channels conda-forge
4741

48-
# Configure the VM.
49-
- cmd: conda.exe install -n root --quiet --yes conda-forge-ci-setup=1
50-
- cmd: run_conda_forge_build_setup
42+
# Configure the VM.
43+
- cmd: conda.exe install -n root --quiet --yes conda-forge-ci-setup=1
44+
- cmd: run_conda_forge_build_setup
5145

5246
# Skip .NET project specific build phase.
5347
build: off
@@ -56,7 +50,7 @@ test_script:
5650
- git clone -b {{ arrow.branch }} {{ arrow.remote }} arrow || exit /B
5751
- git -C arrow checkout {{ arrow.head }} || exit /B
5852
- pushd arrow\dev\tasks\conda-recipes
59-
- conda.exe build --output-folder . -m %CONFIG% parquet-cpp arrow-cpp pyarrow
53+
- conda.exe build --output-folder . -m {{ variant_config_file }} parquet-cpp arrow-cpp pyarrow
6054
- pushd win-64
6155
- for %%f in (*.tar.bz2) do (
6256
set %%g=%%~nf
@@ -68,7 +62,7 @@ artifacts:
6862
- path: arrow\dev\tasks\conda-recipes\win-64\*.tar.bz2
6963

7064
deploy:
71-
release: {{ job.branch }}
65+
release: {{ task.tag }}
7266
provider: GitHub
7367
auth_token: "%CROSSBOW_GITHUB_TOKEN%"
7468
artifact: /.*\.tar\.bz2/

‎dev/tasks/conda-recipes/travis.linux.yml‎

Lines changed: 21 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -21,48 +21,38 @@ os: linux
2121
dist: trusty
2222

2323
env:
24-
matrix:
25-
- CONFIG=variants/linux_python2.7.yaml
26-
- CONFIG=variants/linux_python3.5.yaml
27-
- CONFIG=variants/linux_python3.6.yaml
2824
global:
29-
- TRAVIS_TAG={{ job.branch }}
25+
- TRAVIS_TAG={{ task.tag }}
3026
- ARROW_VERSION={{ arrow.version }}
3127
- PYTHONUNBUFFERED=1
3228

3329
install:
34-
# Install Miniconda.
35-
- echo `pwd`
36-
- |
37-
echo ""
38-
echo "Installing a fresh version of Miniconda."
39-
MINICONDA_URL="https://repo.continuum.io/miniconda"
40-
MINICONDA_FILE="Miniconda3-latest-Linux-x86_64.sh"
41-
curl -L -O "${MINICONDA_URL}/${MINICONDA_FILE}"
42-
bash $MINICONDA_FILE -b
30+
# Install Miniconda.
31+
- echo `pwd`
32+
- |
33+
echo ""
34+
echo "Installing a fresh version of Miniconda."
35+
MINICONDA_URL="https://repo.continuum.io/miniconda"
36+
MINICONDA_FILE="Miniconda3-latest-Linux-x86_64.sh"
37+
curl -L -O "${MINICONDA_URL}/${MINICONDA_FILE}"
38+
bash $MINICONDA_FILE -b
4339
44-
# Configure conda.
45-
- |
46-
echo ""
47-
echo "Configuring conda."
48-
source /home/travis/miniconda3/bin/activate root
49-
conda config --remove channels defaults
50-
conda config --add channels defaults
51-
conda config --add channels conda-forge
52-
conda config --set show_channel_urls true
53-
conda install --yes --quiet conda-build
40+
# Configure conda.
41+
- |
42+
echo ""
43+
echo "Configuring conda."
44+
source /home/travis/miniconda3/bin/activate root
45+
conda config --remove channels defaults
46+
conda config --add channels defaults
47+
conda config --add channels conda-forge
48+
conda config --set show_channel_urls true
49+
conda install --yes --quiet conda-build
5450
5551
script:
5652
- git clone -b {{ arrow.branch }} {{ arrow.remote }} arrow
5753
- git -C arrow checkout {{ arrow.head }}
5854
- pushd arrow/dev/tasks/conda-recipes
59-
- conda build --output-folder . -m $CONFIG parquet-cpp arrow-cpp pyarrow
60-
- pushd linux-64
61-
- |
62-
for file in *.tar.bz2; do
63-
mv "$file" "$(basename "$file" .tar.bz2)-linux-64.tar.bz2"
64-
done
65-
popd
55+
- conda build --output-folder . -m {{ variant_config_file }} parquet-cpp arrow-cpp pyarrow
6656

6757
deploy:
6858
provider: releases

‎dev/tasks/conda-recipes/travis.osx.yml‎

Lines changed: 29 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -21,60 +21,48 @@ os: osx
2121
osx_image: xcode6.4
2222

2323
env:
24-
matrix:
25-
- CONFIG=variants/osx_python2.7.yaml
26-
- CONFIG=variants/osx_python3.5.yaml
27-
- CONFIG=variants/osx_python3.6.yaml
2824
global:
29-
- TRAVIS_TAG={{ job.branch }}
25+
- TRAVIS_TAG={{ task.tag }}
3026
- ARROW_VERSION={{ arrow.version }}
3127
- PYTHONUNBUFFERED=1
3228

3329
before_install:
34-
# Remove homebrew.
35-
- |
36-
echo ""
37-
echo "Removing homebrew from Travis CI to avoid conflicts."
38-
curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/uninstall > ~/uninstall_homebrew
39-
chmod +x ~/uninstall_homebrew
40-
~/uninstall_homebrew -fq
41-
rm ~/uninstall_homebrew
30+
# Remove homebrew.
31+
- |
32+
echo ""
33+
echo "Removing homebrew from Travis CI to avoid conflicts."
34+
curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/uninstall > ~/uninstall_homebrew
35+
chmod +x ~/uninstall_homebrew
36+
~/uninstall_homebrew -fq
37+
rm ~/uninstall_homebrew
4238
4339
install:
44-
# Install Miniconda.
45-
- |
46-
echo ""
47-
echo "Installing a fresh version of Miniconda."
48-
MINICONDA_URL="https://repo.continuum.io/miniconda"
49-
MINICONDA_FILE="Miniconda3-latest-MacOSX-x86_64.sh"
50-
curl -L -O "${MINICONDA_URL}/${MINICONDA_FILE}"
51-
bash $MINICONDA_FILE -b
52-
53-
# Configure conda.
54-
- |
55-
echo ""
56-
echo "Configuring conda."
57-
source /Users/travis/miniconda3/bin/activate root
58-
conda config --remove channels defaults
59-
conda config --add channels defaults
60-
conda config --add channels conda-forge
61-
conda config --set show_channel_urls true
62-
conda install --yes --quiet conda-forge-ci-setup=1
63-
source run_conda_forge_build_setup
40+
# Install Miniconda.
41+
- |
42+
echo ""
43+
echo "Installing a fresh version of Miniconda."
44+
MINICONDA_URL="https://repo.continuum.io/miniconda"
45+
MINICONDA_FILE="Miniconda3-latest-MacOSX-x86_64.sh"
46+
curl -L -O "${MINICONDA_URL}/${MINICONDA_FILE}"
47+
bash $MINICONDA_FILE -b
6448
65-
# conda install --yes --quiet conda-build
49+
# Configure conda.
50+
- |
51+
echo ""
52+
echo "Configuring conda."
53+
source /Users/travis/miniconda3/bin/activate root
54+
conda config --remove channels defaults
55+
conda config --add channels defaults
56+
conda config --add channels conda-forge
57+
conda config --set show_channel_urls true
58+
conda install --yes --quiet conda-forge-ci-setup=1
59+
source run_conda_forge_build_setup
6660
6761
script:
6862
- git clone -b {{ arrow.branch }} {{ arrow.remote }} arrow
6963
- git -C arrow checkout {{ arrow.head }}
7064
- pushd arrow/dev/tasks/conda-recipes
71-
- conda build --output-folder . -m $CONFIG parquet-cpp arrow-cpp pyarrow
72-
- pushd osx-64
73-
- |
74-
for file in *.tar.bz2; do
75-
mv "$file" "$(basename "$file" .tar.bz2)-osx-64.tar.bz2"
76-
done
77-
popd
65+
- conda build --output-folder . -m {{ variant_config_file }} parquet-cpp arrow-cpp pyarrow
7866

7967
deploy:
8068
provider: releases

0 commit comments

Comments
 (0)