Skip to content

Commit 9abf114

Browse files
xingao267alxhub
xingao267
authored and
alxhub
committed
feat(bazel): use rbe_autoconfig() and new container. (#29336)
After this PR is merged, maintainers no longer need to update .bazelrc file, toolchain and platform related flags for RBE builds and tests (unless there is a breaking change in Bazel related to those flags). Maintainers just need to update the pin of @bazel-toolchains repo regularly in the packages/bazel/package.bzl file according to https://releases.bazel.build/bazel-toolchains.html to include the latest checked-in toolchain configs. If rbe_autoconfig() cannot find appropriate toolchain configs for the version of Bazel in the version of @bazel_toolchains repo that is currently used by this project, it will pull down the container and generate the configs on the fly as the beginning of the build/test. PR Close #29336
1 parent 8d3365e commit 9abf114

8 files changed

Lines changed: 93 additions & 263 deletions

File tree

‎.bazelrc‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -116,16 +116,21 @@ build --define=compile=legacy
116116
# --config=remote
117117
###############################
118118

119-
# Load default settings for Remote Build Execution
120-
# When updating, the URLs of bazel_toolchains in packages/bazel/package.bzl
121-
# may also need to be updated (see https://github.com/angular/angular/pull/27935)
122-
import %workspace%/third_party/github.com/bazelbuild/bazel-toolchains/bazelrc/bazel-0.24.0.bazelrc
119+
# Load default settings for Remote Build Execution.
120+
import %workspace%/third_party/github.com/bazelbuild/bazel-toolchains/bazelrc/.bazelrc.notoolchain
123121

124122
# Increase the default number of jobs by 50% because our build has lots of
125123
# parallelism
126124
build:remote --jobs=150
127125

128-
# Point to our custom execution platform; see tools/BUILD.bazel
126+
# Toolchain and platform related flags
127+
build:remote --host_javabase=@rbe_ubuntu1604_angular//java:jdk
128+
build:remote --javabase=@rbe_ubuntu1604_angular//java:jdk
129+
build:remote --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8
130+
build:remote --java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8
131+
build:remote --crosstool_top=@rbe_ubuntu1604_angular//cc:toolchain
132+
build:remote --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1
133+
build:remote --extra_toolchains=@rbe_ubuntu1604_angular//config:cc-toolchain
129134
build:remote --extra_execution_platforms=//tools:rbe_ubuntu1604-angular
130135
build:remote --host_platform=//tools:rbe_ubuntu1604-angular
131136
build:remote --platforms=//tools:rbe_ubuntu1604-angular

‎.circleci/config.yml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -167,7 +167,8 @@ jobs:
167167
- *attach_workspace
168168
- *init_environment
169169
- *setup_circleci_bazel_config
170-
170+
# Enable remote/sibling docker which is needed by auto-selection of toolchain configs for RBE.
171+
- setup_remote_docker
171172
# Setup remote execution and run RBE-compatible tests.
172173
- *setup_bazel_remote_execution
173174
- run: yarn bazel test //... --build_tag_filters=-ivy-only --test_tag_filters=-ivy-only
@@ -180,6 +181,8 @@ jobs:
180181
- *attach_workspace
181182
- *init_environment
182183
- *setup_circleci_bazel_config
184+
# Enable remote/sibling docker which is needed by auto-selection of toolchain configs for RBE.
185+
- setup_remote_docker
183186
- *setup_bazel_remote_execution
184187

185188
# We need to explicitly specify the --symlink_prefix option because otherwise we would
@@ -409,6 +412,8 @@ jobs:
409412
- *attach_workspace
410413
- *init_environment
411414
- *setup_circleci_bazel_config
415+
# Enable remote/sibling docker which is needed by auto-selection of toolchain configs for RBE.
416+
- setup_remote_docker
412417
- *setup_bazel_remote_execution
413418

414419
- run: scripts/build-packages-dist.sh
@@ -435,6 +440,8 @@ jobs:
435440
- *attach_workspace
436441
- *init_environment
437442
- *setup_circleci_bazel_config
443+
# Enable remote/sibling docker which is needed by auto-selection of toolchain configs for RBE.
444+
- setup_remote_docker
438445
- *setup_bazel_remote_execution
439446

440447
- run: scripts/build-ivy-npm-packages.sh

‎WORKSPACE‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,3 +114,23 @@ sass_repositories()
114114
load("@io_bazel_skydoc//skylark:skylark.bzl", "skydoc_repositories")
115115

116116
skydoc_repositories()
117+
118+
load("@bazel_toolchains//rules:environments.bzl", "clang_env")
119+
load("@bazel_toolchains//rules:rbe_repo.bzl", "rbe_autoconfig")
120+
121+
rbe_autoconfig(
122+
name = "rbe_ubuntu1604_angular",
123+
# The sha256 of marketplace.gcr.io/google/rbe-ubuntu16-04 container that is
124+
# used by rbe_autoconfig() to pair toolchain configs in the @bazel_toolchains repo.
125+
base_container_digest = "sha256:677c1317f14c6fd5eba2fd8ec645bfdc5119f64b3e5e944e13c89e0525cc8ad1",
126+
# Note that if you change the `digest`, you might also need to update the
127+
# `base_container_digest` to make sure marketplace.gcr.io/google/rbe-ubuntu16-04-webtest:<digest>
128+
# and marketplace.gcr.io/google/rbe-ubuntu16-04:<base_container_digest> have
129+
# the same Clang and JDK installed.
130+
# Clang is needed because of the dependency on @com_google_protobuf.
131+
# Java is needed for the Bazel's test executor Java tool.
132+
digest = "sha256:74a8e9dca4781d5f277a7bd8e7ea7ed0f5906c79c9cd996205b6d32f090c62f3",
133+
env = clang_env(),
134+
registry = "marketplace.gcr.io",
135+
repository = "google/rbe-ubuntu16-04-webtest",
136+
)

‎packages/bazel/package.bzl‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -28,11 +28,10 @@ def rules_angular_dev_dependencies():
2828
_maybe(
2929
http_archive,
3030
name = "bazel_toolchains",
31-
sha256 = "67335b3563d9b67dc2550b8f27cc689b64fadac491e69ce78763d9ba894cc5cc",
32-
strip_prefix = "bazel-toolchains-cddc376d428ada2927ad359211c3e356bd9c9fbb",
31+
sha256 = "142bcbd8cb751ce1193a1d7fef4e328493cd0a69cc0555183ad237f81418ba40",
32+
strip_prefix = "bazel-toolchains-628224f6cf48e81116d0ee0bf65424eaa630d5b3",
3333
urls = [
34-
"https://mirror.bazel.build/github.com/bazelbuild/bazel-toolchains/archive/cddc376d428ada2927ad359211c3e356bd9c9fbb.tar.gz",
35-
"https://github.com/bazelbuild/bazel-toolchains/archive/cddc376d428ada2927ad359211c3e356bd9c9fbb.tar.gz",
34+
"https://github.com/xingao267/bazel-toolchains/archive/628224f6cf48e81116d0ee0bf65424eaa630d5b3.tar.gz",
3635
],
3736
)
3837

‎third_party/github.com/bazelbuild/bazel-toolchains/bazelrc/.bazelrc.notoolchain‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Copyright 2016 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# This .bazelrc file contains some of the flags required for the provided
16+
# toolchain with Remote Build Execution. Specifically, it includes all flags
17+
# except toolchain/platform flags.
18+
19+
# Depending on how many machines are in the remote execution instance, setting
20+
# this higher can make builds faster by allowing more jobs to run in parallel.
21+
# Setting it too high can result in jobs that timeout, however, while waiting
22+
# for a remote machine to execute them.
23+
build:remote --jobs=50
24+
25+
# Set various strategies so that all actions execute remotely. Mixing remote
26+
# and local execution will lead to errors unless the toolchain and remote
27+
# machine exactly match the host machine.
28+
build:remote --spawn_strategy=remote
29+
build:remote --strategy=Javac=remote
30+
build:remote --strategy=Closure=remote
31+
build:remote --strategy=Genrule=remote
32+
build:remote --define=EXECUTOR=remote
33+
34+
# Enable the remote cache so action results can be shared across machines,
35+
# developers, and workspaces.
36+
build:remote --remote_cache=remotebuildexecution.googleapis.com
37+
38+
# Enable remote execution so actions are performed on the remote systems.
39+
build:remote --remote_executor=remotebuildexecution.googleapis.com
40+
41+
# Enable encryption.
42+
build:remote --tls_enabled=true
43+
44+
# Set a higher timeout value, just in case.
45+
build:remote --remote_timeout=3600
46+
47+
# Enable authentication. This will pick up application default credentials by
48+
# default. You can use --auth_credentials=some_file.json to use a service
49+
# account credential instead.
50+
build:remote --auth_enabled=true

‎third_party/github.com/bazelbuild/bazel-toolchains/bazelrc/bazel-0.21.0.bazelrc‎

Lines changed: 0 additions & 127 deletions
This file was deleted.

‎third_party/github.com/bazelbuild/bazel-toolchains/bazelrc/bazel-0.24.0.bazelrc‎

Lines changed: 0 additions & 106 deletions
This file was deleted.

0 commit comments

Comments
 (0)