dataflow: new minimal gpu samples #6096

davidcavazos · 2021-06-07T22:01:40Z

Description

Adds new minimal examples to use GPU with tensorflow and pytorch.
Simplifies the landsat sample with newer patterns.
Fixes some resource leaks due to uncaught errors in some fixture's teardown code.

Checklist

I have followed Sample Guidelines from AUTHORING_GUIDE.MD
README is updated to include all relevant information
Tests pass: nox -s py-3.6 (see Test Environment Setup)
Lint pass: nox -s lint (see Test Environment Setup)
These samples need a new API enabled in testing projects to pass (let us know which ones)
These samples need a new/updated env vars in testing projects set to pass (let us know which ones)
Please merge this PR for me once it is approved.
This sample adds a new sample directory, and I updated the CODEOWNERS file with the codeowners for this sample

…amples into dataflow-gpu

snippet-bot · 2021-06-07T22:01:53Z

No region tags are edited in this PR.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

Refresh this comment

kurtisvg · 2021-06-07T22:54:42Z

@davidcavazos Would you mind pinging me for review once you have the tests passing?

dataflow/gpu-workers/pytorch-minimal/README.md

dataflow/gpu-workers/tensorflow-landsat/main.py

dataflow/gpu-workers/tensorflow-landsat/run.yaml

dataflow/gpu-workers/tensorflow-landsat/main.py

dataflow/gpu-workers/pytorch-minimal/main.py

…amples into dataflow-gpu

davidcavazos · 2021-07-01T19:51:01Z

@leahecole @tvalentyn, tests are passing, this should be ready to merge if there are no more comments.

leahecole

David this is so so so close and also it's a really fantastic PR. When you're back, let's sit down for like 15 min and just knock out my questions. (I sent you an email about this too in case your inbox is gross from being OOO)

dataflow/gpu-examples/conftest.py

dataflow/gpu-examples/pytorch-minimal/e2e_test.py

dataflow/gpu-examples/pytorch-minimal/noxfile_config.py

dataflow/gpu-examples/pytorch-minimal/run.yaml

…amples into dataflow-gpu

davidcavazos · 2021-07-13T01:37:56Z

@leahecole the new Beam version depends on google-cloud-profiler which requires a C++ compiler and Python build tools in the container image. Also the newer PyTorch image is quite larger so I had to adjust the timeout for building the image.

All tests are passing again.

leahecole · 2021-07-13T20:52:41Z

Used admin merge just because second approver didn't have correct permissions, but they are a Dataflow expert.

David Cavazos added 6 commits June 7, 2021 11:37

dataflow: minimal gpu examples

f2bacd1

Merge branch 'master' of github.com:GoogleCloudPlatform/python-docs-s…

7bb9e48

…amples into dataflow-gpu

added tests

d8bba19

Merge branch 'master' of github.com:GoogleCloudPlatform/python-docs-s…

2ebd8b7

…amples into dataflow-gpu

update dockerignore and cloudignore

ed2d91d

remove old config

7969369

davidcavazos requested a review from a team as a code owner June 7, 2021 22:01

product-auto-label bot added the samples Issues that are directly related to samples. label Jun 7, 2021

google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Jun 7, 2021

blunderbuss-gcf bot assigned kurtisvg Jun 7, 2021

update header year

d34a2ca

davidcavazos changed the title ~~Dataflow gpu~~ dataflow: new minimal gpu samples Jun 7, 2021

David Cavazos added 2 commits June 7, 2021 15:09

removed unused import

193ce0e

add test

abcbaae

kurtisvg marked this pull request as draft June 7, 2021 22:54

David Cavazos added 2 commits June 7, 2021 16:42

make image name unique

f6af338

fix image name

ba72e54

product-auto-label bot added the api: dataflow Issues related to the Dataflow API. label Jun 8, 2021