Skip to content

Commit 0a639eb

Browse files
author
chenlai
committed
Update on "More update on the guidance"
Differential Revision: [D34226823](https://our.internmc.facebook.com/intern/diff/D34226823/) [ghstack-poisoned]
2 parents 98a8e44 + 144c2af commit 0a639eb

File tree

116 files changed

+2409
-906
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

116 files changed

+2409
-906
lines changed

‎.github/README.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# pytorch/.github
2+
3+
This directory contains workflows and scripts to support our CI infrastructure that runs on Github Actions.
4+
5+
6+
## Workflows / Templates
7+
8+
Our current Github Actions setup uses templates written in [Jinja](https://jinja.palletsprojects.com/en/3.0.x/) that are located in the
9+
`.github/templates` directory to generate workflow files found in the `.github/workflows/` directory.
10+
11+
These templates contain a couple of utility templates used to discern common utilities that can be
12+
used amongst different templates.
13+
14+
### (Re)Generating workflow files
15+
16+
You will need `jinja2` in order to regenerate the workflow files which can be installed using:
17+
```bash
18+
pip install -r .github/requirements.txt
19+
```
20+
21+
Workflows can be generated / regenerated using the following command:
22+
```bash
23+
.github/regenerate.sh
24+
```
25+
26+
### Adding a new generated workflow
27+
28+
New generated workflows can be added in the `.github/scripts/generate_ci_workflows.py` script. You can reference
29+
examples from that script in order to add the workflow to the stream that is relevant to what you particularly
30+
care about.
31+
32+
Different parameters can be used to acheive different goals, i.e. running jobs on a cron, running only on trunk, etc.
33+
34+
#### ciflow (specific)
35+
36+
ciflow is the way we can get `non-default` workflows to run on specific PRs. Within the `generate_ci_workflows.py` script
37+
you will notice a multitude of `LABEL_CIFLOW_<NAME>` variables which correspond to labels on Github. Workflows that
38+
do not run on ``LABEL_CIFLOW_DEFAULT` can be triggered on PRs by applying the label found in `generate_ci_workflows.py`
39+
40+
Example:
41+
```python
42+
CIWorkflow(
43+
arch="linux",
44+
build_environment="periodic-linux-xenial-cuda10.2-py3-gcc7-slow-gradcheck",
45+
docker_image_base=f"{DOCKER_REGISTRY}/pytorch/pytorch-linux-xenial-cuda10.2-cudnn7-py3-gcc7",
46+
test_runner_type=LINUX_CUDA_TEST_RUNNER,
47+
num_test_shards=2,
48+
distributed_test=False,
49+
timeout_after=360,
50+
# Only run this on master 4 times per day since it does take a while
51+
is_scheduled="0 */4 * * *",
52+
ciflow_config=CIFlowConfig(
53+
labels={LABEL_CIFLOW_LINUX, LABEL_CIFLOW_CUDA, LABEL_CIFLOW_SLOW_GRADCHECK, LABEL_CIFLOW_SLOW, LABEL_CIFLOW_SCHEDULED},
54+
),
55+
),
56+
```
57+
58+
This workflow does not get triggered by default since it does not contain the `LABEL_CIFLOW_DEFAULT` label in its CIFlowConfig but applying
59+
the `LABEL_CIFLOW_SLOW_GRADCHECK` on your PR will trigger this specific workflow to run.
60+
61+
#### ciflow (trunk)
62+
63+
The label `ciflow/trunk` can be used to run `trunk` only workflows. This is especially useful if trying to re-land a PR that was
64+
reverted for failing a `non-default` workflow.
65+
66+
## Infra
67+
68+
Currently most of our self hosted runners are hosted on AWS, for a comprehensive list of available runner types you
69+
can reference `.github/scale-config.yml`.
70+
71+
Exceptions to AWS for self hosted:
72+
* ROCM runners
73+
74+
### Adding new runner types
75+
76+
New runner types can be added by committing changes to `.github/scale-config.yml`. Example: https://github.com/pytorch/pytorch/pull/70474
77+
78+
> NOTE: New runner types can only be used once the changes to `.github/scale-config.yml` have made their way into the default branch

‎.github/merge_rules.json‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,16 @@
11
[
22
{
33
"name": "ONNX exporter",
4-
"patterns": ["torch/onnx/**", "torch/csrc/jit/passes/onnx/**", "torch/csrc/jit/passes/onnx.*", "test/onnx/**", "docs/source/onnx.rst"],
4+
"patterns": [
5+
"torch/onnx/**",
6+
"torch/csrc/jit/passes/onnx/**",
7+
"torch/csrc/jit/passes/onnx.*",
8+
"test/onnx/**",
9+
"docs/source/onnx.rst",
10+
"torch/csrc/jit/serialization/export.*",
11+
"torch/csrc/jit/serialization/onnx.*",
12+
"torch/_C/__init__.pyi.in"
13+
],
514
"approved_by": ["BowenBao", "garymm"],
615
"mandatory_app_id": 12274
716
},

‎.github/requirements.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
jinja2

‎.github/templates/common.yml.j2‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ concurrency:
4343
run: .github/scripts/parse_ref.py
4444
{%- endmacro -%}
4545

46-
{%- macro upload_test_statistics(build_environment, when="always()", pytorch_directory="") -%}
46+
{%- macro upload_test_statistics(build_environment, when="always()", pytorch_directory="", needs_credentials=False) -%}
4747
- name: Display and upload test statistics (Click Me)
4848
{%- if pytorch_directory %}
4949
working-directory: !{{ pytorch_directory }}
@@ -59,6 +59,10 @@ concurrency:
5959
SHA1: ${{ github.event.pull_request.head.sha || github.sha }}
6060
TAG: ${{ steps.parse-ref.outputs.tag }}
6161
WORKFLOW_ID: '${{ github.run_id }}'
62+
{%- if needs_credentials %}
63+
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_OSSCI_METRICS_ACCESS_KEY_ID }}
64+
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_OSSCI_METRICS_SECRET_ACCESS_KEY }}
65+
{%- endif %}
6266
shell: bash
6367
run: |
6468
python3 -m pip install -r requirements.txt
@@ -185,10 +189,13 @@ concurrency:
185189
docker system prune -af
186190
{%- endmacro -%}
187191

188-
{%- macro checkout(submodules="recursive", deep_clone=True, directory="", repository="pytorch/pytorch", branch="") -%}
192+
{%- macro checkout(submodules="recursive", deep_clone=True, directory="", repository="pytorch/pytorch", branch="", checkout_pr_head=True) -%}
189193
- name: Checkout !{{ 'PyTorch' if repository == "pytorch/pytorch" else repository }}
190194
uses: zhouzhuojie/checkout@05b13c9a0d21f08f6d5e64a1d5042246d13619d9
191195
with:
196+
{%- if checkout_pr_head %}
197+
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
198+
{%- endif %}
192199
{%- if deep_clone %}
193200
# deep clone, to allow use of git merge-base
194201
fetch-depth: 0

‎.github/templates/linux_binary_build_workflow.yml.j2‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ jobs:
5353
steps:
5454
!{{ common.setup_ec2_linux() }}
5555
!{{ common.checkout(deep_clone=False, directory="pytorch") }}
56-
!{{ common.checkout(deep_clone=False, directory="builder", repository="pytorch/builder") }}
56+
!{{ common.checkout(deep_clone=False, directory="builder", repository="pytorch/builder", checkout_pr_head=False) }}
5757
{%- if config["gpu_arch_type"] == 'cuda' and config["gpu_arch_version"].startswith('11') %}
5858
- name: Set BUILD_SPLIT_CUDA
5959
run: |

‎.github/templates/macos_ci_workflow.yml.j2‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -147,7 +147,7 @@ jobs:
147147
!{{ common.render_test_results() }}
148148
!{{ common.upload_downloaded_files(name='macos', artifact_name="test-jsons", use_s3=False) }}
149149
!{{ common.upload_test_reports("macos", artifact_name="test-reports", use_s3=False) }}
150-
!{{ common.upload_test_statistics(build_environment) }}
150+
!{{ common.upload_test_statistics(build_environment, needs_credentials=True) }}
151151
{% endblock +%}
152152
{%- endif %}
153153

‎.github/workflows/generated-caffe2-linux-xenial-py3.7-gcc5.4.yml‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.github/workflows/generated-docker-builds.yml‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.github/workflows/generated-ios-12-5-1-arm64-coreml.yml‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.github/workflows/generated-ios-12-5-1-arm64-custom-ops.yml‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)