Skip to content

getting_started

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History
 
 

getting_started

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
src
src
 
 
README.md
README.md
 
 
metadata.json
metadata.json
 
 
pom.xml
pom.xml
 
 

README.md

Dataflow flex template: Getting started sample

Before you begin

Make sure you have followed the Dataflow setup instructions.

Create a Cloud Storage bucket

export BUCKET="your-bucket"
gcloud storage buckets create gs://$BUCKET

Create an Artifact Registry repository

export REGION="us-central1"
export REPOSITORY="your-repository"

gcloud artifacts repositories create $REPOSITORY \
    --repository-format=docker \
    --location=$REGION

Build the JAR file

mvn clean package

Build the template

export PROJECT="project-id"

gcloud dataflow flex-template build gs://$BUCKET/getting_started_java.json \
    --image-gcr-path "$REGION-docker.pkg.dev/$PROJECT/$REPOSITORY/getting-started-java:latest" \
    --sdk-language "JAVA" \
    --flex-template-base-image JAVA11 \
    --metadata-file "metadata.json" \
    --jar "target/flex-template-getting-started-1.0.jar" \
    --env FLEX_TEMPLATE_JAVA_MAIN_CLASS="com.example.dataflow.FlexTemplateGettingStarted"

Run the template

gcloud dataflow flex-template run "flex-`date +%Y%m%d-%H%M%S`"  \
    --template-file-gcs-location "gs://$BUCKET/getting_started_java.json" \
    --region $REGION \
    --parameters output="gs://$BUCKET/output-"

Clean up

To delete the resources that you created:

gcloud artifacts repositories delete $REPOSITORY --location $REGION --quiet
gcloud storage rm gs://$BUCKET --recursive

What's next?

For more information about building and running flex templates, see 📝 Use Flex Templates.