Skip to content

Commit ea0fb37

Browse files
trxcllntwesm
trxcllnt
authored and
wesm
committed
ARROW-1577: [JS] add ASF release scripts
@wesm does [this](apache@b270dba#diff-fc8acbd4f42fb5e6b0cad14928b68115R59) look good? Author: Paul Taylor <paul.e.taylor@me.com> Author: Wes McKinney <wes.mckinney@twosigma.com> Closes apache#1346 from trxcllnt/js-asf-release-scripts and squashes the following commits: ac246cd [Wes McKinney] Update npm-release.sh while integration testing from release tarball is hard 469beea [Wes McKinney] Add to READMEs about how to release and verify JS release 160f5c3 [Paul Taylor] remove tag-version-prefix from npmrc 854020d [Wes McKinney] Run unit tests without integration tests for now 9485922 [Wes McKinney] Add JavaScript release verification script dd06c46 [Wes McKinney] Create tag in source release script 2c6ce86 [Paul Taylor] number of npm-release.sh script arguments is 0 e1c3a96 [Paul Taylor] npm install before release, use npm version --git-tag-version with tag-version-prefix 5870043 [Wes McKinney] Number of arguments is 2 1133fb8 [Wes McKinney] Remove upstream Arrow version tag. Bump JS version to 0.2.0 30b602f [Paul Taylor] add dev/release/js-source-release.sh script be82c08 [Paul Taylor] add npm-release.sh script to publish npm modules f72fc83 [Paul Taylor] update test_js commands in verify-release-candidate.sh 5194951 [Paul Taylor] npm pack the js sources into the release tarball d73c47a [Paul Taylor] prepare js lib for release 7d9c84a [Paul Taylor] remove tslib patches as it doesn't work on windows
1 parent 1524ed7 commit ea0fb37

14 files changed

Lines changed: 292 additions & 35 deletions

‎ci/travis_script_integration.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ pushd $ARROW_JS_DIR
5151
npm run lint
5252
npm run build
5353
# create initial test data
54-
npm run test:createTestData
54+
npm run create:testdata
5555
# run once to write the snapshots
5656
npm test -- -t ts -u --integration
5757
# run again to test all builds against the snapshots

‎dev/README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,3 +113,11 @@ Studio 2015):
113113
```
114114
dev/release/verify-release-candidate.bat apache-arrow-0.7.0.tar.gz
115115
```
116+
117+
### Verifying the JavaScript release
118+
119+
For JavaScript-specific releases, use a different verification script:
120+
121+
```shell
122+
bash dev/release/js-verify-release-candidate.sh 0.7.0 0
123+
```

‎dev/release/README‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,3 +49,16 @@ Logon to the apache repository: https://repository.apache.org/#stagingRepositori
4949
Select the arrow staging repository you just just created: orgapachearrow-100x
5050
Click the "close" button
5151
Once validation has passed, click the "release" button
52+
53+
# Releasing JavaScript
54+
55+
* Make release branch then tag the release
56+
57+
git checkout -b release-js-X.Y.Z
58+
git tag -a apache-arrow-js-X.Y.Z
59+
60+
* Build the source release (requires NodeJS)
61+
62+
dev/release/js-source-release.sh X.Y.Z $RC_NUM
63+
64+
* After release vote push tag, and rebase master on release branch

‎dev/release/js-source-release.sh‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
#!/bin/bash
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
#
20+
set -e
21+
22+
SOURCE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
23+
24+
if [ "$#" -ne 2 ]; then
25+
echo "Usage: $0 <js-version> <rc-num>"
26+
exit
27+
fi
28+
29+
js_version=$1
30+
rc=$2
31+
32+
tag=apache-arrow-js-${js_version}
33+
tagrc=${tag}-rc${rc}
34+
35+
echo "Preparing source for tag ${tag}"
36+
37+
tarball=${tag}.tar.gz
38+
39+
# cd to $ARROW_HOME/js
40+
cd $SOURCE_DIR/../../js
41+
JS_SRC_DIR="$PWD"
42+
# npm pack the js source files
43+
npm install
44+
45+
npm version --no-git-tag-version $js_version
46+
git add package.json
47+
git commit -m "[Release] Apache Arrow JavaScript $js_version"
48+
git tag -a ${tag}
49+
50+
release_hash=`git rev-list $tag 2> /dev/null | head -n 1 `
51+
52+
if [ -z "$release_hash" ]; then
53+
echo "Cannot continue: unknown git tag: $tag"
54+
exit
55+
fi
56+
57+
echo "Using commit $release_hash"
58+
59+
cd $SOURCE_DIR
60+
61+
rm -rf js-tmp
62+
# `npm pack` writes the .tgz file to the current dir, so cd into js-tmp
63+
mkdir -p js-tmp
64+
cd js-tmp
65+
# run npm pack on `arrow/js`
66+
npm pack ${JS_SRC_DIR}
67+
# unzip and remove the npm pack tarball
68+
tar -xzf *.tgz && rm *.tgz
69+
# `npm pack` puts files in a dir called "package"
70+
cp $JS_SRC_DIR/../NOTICE.txt package
71+
cp $JS_SRC_DIR/../LICENSE.txt package
72+
# rename "package" to $tag
73+
mv package ${tag}
74+
tar czf ${tarball} ${tag}
75+
rm -rf ${tag}
76+
77+
${SOURCE_DIR}/run-rat.sh ${tarball}
78+
79+
# sign the archive
80+
gpg --armor --output ${tarball}.asc --detach-sig ${tarball}
81+
gpg --print-md MD5 ${tarball} > ${tarball}.md5
82+
sha1sum $tarball > ${tarball}.sha1
83+
sha256sum $tarball > ${tarball}.sha256
84+
sha512sum $tarball > ${tarball}.sha512
85+
86+
# check out the arrow RC folder
87+
svn co --depth=empty https://dist.apache.org/repos/dist/dev/arrow js-rc-tmp
88+
89+
# add the release candidate for the tag
90+
mkdir -p js-rc-tmp/${tagrc}
91+
cp ${tarball}* js-rc-tmp/${tagrc}
92+
svn add js-rc-tmp/${tagrc}
93+
svn ci -m 'Apache Arrow JavaScript ${version} RC${rc}' js-rc-tmp/${tagrc}
94+
95+
cd -
96+
97+
# clean up
98+
rm -rf js-tmp
99+
100+
echo "Success! The release candidate is available here:"
101+
echo " https://dist.apache.org/repos/dist/dev/arrow/${tagrc}"
102+
echo ""
103+
echo "Commit SHA1: ${release_hash}"

‎dev/release/js-verify-release-candidate.sh‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
#!/bin/bash
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
#
20+
21+
# Requirements
22+
# - nodejs >= 6.0.0 (best way is to use nvm)
23+
24+
case $# in
25+
2) VERSION="$1"
26+
RC_NUMBER="$2"
27+
;;
28+
29+
*) echo "Usage: $0 X.Y.Z RC_NUMBER"
30+
exit 1
31+
;;
32+
esac
33+
34+
set -ex
35+
36+
HERE=$(cd `dirname "${BASH_SOURCE[0]:-$0}"` && pwd)
37+
38+
ARROW_DIST_URL='https://dist.apache.org/repos/dist/dev/arrow'
39+
40+
download_dist_file() {
41+
curl -f -O $ARROW_DIST_URL/$1
42+
}
43+
44+
download_rc_file() {
45+
download_dist_file apache-arrow-js-${VERSION}-rc${RC_NUMBER}/$1
46+
}
47+
48+
import_gpg_keys() {
49+
download_dist_file KEYS
50+
gpg --import KEYS
51+
}
52+
53+
fetch_archive() {
54+
local dist_name=$1
55+
download_rc_file ${dist_name}.tar.gz
56+
download_rc_file ${dist_name}.tar.gz.asc
57+
download_rc_file ${dist_name}.tar.gz.md5
58+
download_rc_file ${dist_name}.tar.gz.sha512
59+
gpg --verify ${dist_name}.tar.gz.asc ${dist_name}.tar.gz
60+
gpg --print-md MD5 ${dist_name}.tar.gz | diff - ${dist_name}.tar.gz.md5
61+
if [ "$(uname)" == "Darwin" ]; then
62+
shasum -a 512 ${dist_name}.tar.gz | diff - ${dist_name}.tar.gz.sha512
63+
else
64+
sha512sum ${dist_name}.tar.gz | diff - ${dist_name}.tar.gz.sha512
65+
fi
66+
}
67+
68+
setup_tempdir() {
69+
cleanup() {
70+
rm -fr "$TMPDIR"
71+
}
72+
trap cleanup EXIT
73+
TMPDIR=$(mktemp -d -t "$1.XXXXX")
74+
}
75+
76+
setup_tempdir "arrow-js-$VERSION"
77+
echo "Working in sandbox $TMPDIR"
78+
cd $TMPDIR
79+
80+
VERSION=$1
81+
RC_NUMBER=$2
82+
83+
TARBALL=apache-arrow-js-$1.tar.gz
84+
85+
import_gpg_keys
86+
87+
DIST_NAME="apache-arrow-js-${VERSION}"
88+
fetch_archive $DIST_NAME
89+
tar xvzf ${DIST_NAME}.tar.gz
90+
cd ${DIST_NAME}
91+
92+
npm install
93+
# npx run-s clean:all lint create:testdata build
94+
# npm run test -- -t ts -u --integration
95+
# npm run test -- --integration
96+
npx run-s clean:all lint build
97+
npm run test
98+
99+
echo 'Release candidate looks good!'
100+
exit 0

‎dev/release/verify-release-candidate.sh‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -173,7 +173,16 @@ test_glib() {
173173
test_js() {
174174
pushd js
175175
npm install
176-
npm run validate
176+
# clean, lint, and build JS source
177+
npm run clean:all
178+
npm run lint
179+
npm run build
180+
# create initial integration test data
181+
npm run create:testdata
182+
# run once to write the snapshots
183+
npm test -- -t ts -u --integration
184+
# run again to test all builds against the snapshots
185+
npm test -- --integration
177186
popd
178187
}
179188

‎js/.npmrc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
save-prefix=
2-
package-lock=false
2+
package-lock=false

‎js/README.md‎

Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -183,22 +183,24 @@ The base `apache-arrow` package includes all the compilation targets for conveni
183183
The targets are also published under the `@apache-arrow` namespace:
184184

185185
```sh
186-
npm install @apache-arrow/es5-cjs # ES5 CommonJS target
187-
npm install @apache-arrow/es5-esm # ES5 ESModules target
188-
npm install @apache-arrow/es5-umd # ES5 UMD target
189-
npm install @apache-arrow/es2015-cjs # ES2015 CommonJS target
190-
npm install @apache-arrow/es2015-esm # ES2015 ESModules target
191-
npm install @apache-arrow/es2015-umd # ES2015 UMD target
192-
npm install @apache-arrow/esnext-esm # ESNext CommonJS target
193-
npm install @apache-arrow/esnext-esm # ESNext ESModules target
194-
npm install @apache-arrow/esnext-umd # ESNext UMD target
186+
npm install apache-arrow # <-- combined es5/CommonJS + UMD, es2015/ESModules + UMD, and TypeScript package
187+
npm install @apache-arrow/ts # standalone TypeScript package
188+
npm install @apache-arrow/es5-cjs # standalone es5/CommonJS package
189+
npm install @apache-arrow/es5-esm # standalone es5/ESModules package
190+
npm install @apache-arrow/es5-umd # standalone es5/UMD package
191+
npm install @apache-arrow/es2015-cjs # standalone es2015/CommonJS package
192+
npm install @apache-arrow/es2015-esm # standalone es2015/ESModules package
193+
npm install @apache-arrow/es2015-umd # standalone es2015/UMD package
194+
npm install @apache-arrow/esnext-esm # standalone esNext/CommonJS package
195+
npm install @apache-arrow/esnext-esm # standalone esNext/ESModules package
196+
npm install @apache-arrow/esnext-umd # standalone esNext/UMD package
195197
```
196198

197199
### Why we package like this
198200

199201
The JS community is a diverse group with a varied list of target environments and tool chains. Publishing multiple packages accommodates projects of all stripes.
200202

201-
If you think we missed a compilation target and it's a blocker for adoption, please open an issue. We're here for you ❤️.
203+
If you think we missed a compilation target and it's a blocker for adoption, please open an issue.
202204

203205
# License
204206

‎js/gulp/package-task.js‎

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -48,13 +48,7 @@ const createMainPackageJson = (target, format) => (orig) => ({
4848
module: `${mainExport}.mjs`,
4949
browser: `${mainExport}.es5.min.js`,
5050
[`browser:es2015`]: `${mainExport}.es2015.min.js`,
51-
[`@std/esm`]: { esm: `mjs` },
52-
// Temporary workaround until https://github.com/Microsoft/tslib/pull/44 is merged
53-
scripts: {
54-
postinstall: `npm i shx && npm run tslib_mjs && npm run tslib_pkg && npm r shx`,
55-
tslib_mjs: `shx cp $(node -e \"console.log(require.resolve('tslib/tslib.es6.js'))\") $(node -e \"var r=require,p=r('path');console.log(p.join(p.dirname(r.resolve('tslib')),'tslib.mjs'))\")`,
56-
tslib_pkg: `node -e \"var r=require,p=r('path'),f=r('fs'),k=p.join(p.dirname(r.resolve('tslib')),'package.json'),x=JSON.parse(f.readFileSync(k));x.main='tslib';f.writeFileSync(k,JSON.stringify(x))\"`
57-
}
51+
[`@std/esm`]: { esm: `mjs` }
5852
});
5953

6054
const createTypeScriptPackageJson = (target, format) => (orig) => ({

‎js/gulp/test-task.js‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,12 +73,13 @@ const CPP_STREAM_TO_FILE = path.join(CPP_EXE_PATH, 'stream-to-file');
7373
const CPP_FILE_TO_STREAM = path.join(CPP_EXE_PATH, 'file-to-stream');
7474

7575
const testFilesDir = path.join(ARROW_HOME, 'js/test/data');
76+
const snapshotsDir = path.join(ARROW_HOME, 'js/test/__snapshots__');
7677
const cppFilesDir = path.join(testFilesDir, 'cpp');
7778
const javaFilesDir = path.join(testFilesDir, 'java');
7879
const jsonFilesDir = path.join(testFilesDir, 'json');
7980

8081
async function cleanTestData() {
81-
return await del(`${testFilesDir}/**`);
82+
return await del([`${testFilesDir}/**`, `${snapshotsDir}/**`]);
8283
}
8384

8485
async function createTestJSON() {

0 commit comments

Comments
 (0)