Skip to content

Dataflow runner options: disk type & streaming engine#906

Merged
feast-ci-bot merged 2 commits intofeast-dev:masterfrom
pyalex:extend-dataflow-options
Aug 1, 2020
Merged

Dataflow runner options: disk type & streaming engine#906
feast-ci-bot merged 2 commits intofeast-dev:masterfrom
pyalex:extend-dataflow-options

Conversation

@pyalex
Copy link
Copy Markdown
Collaborator

@pyalex pyalex commented Jul 29, 2020 •
edited
Loading

What this PR does / why we need it:

This PR extends execution parameteres for Dataflow Job according to official documentation.

Next parameter were added:

  • workerDiskType
  • enableStreamingEngine

Next parameter were updated:

  • zone -> workerZone (due to deprecation)

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:


new options introduced in application.yaml

feast:
  jobs:
    runners:
    - name: dataflow
      options:
        enableStreamingEngine: false
        workerDiskType: compute.googleapis.com/projects/asia-east1-a/diskTypes/pd-ssd
        workerZone: asia-east1-a  # renamed "zone" due to deprecation

@pyalex pyalex force-pushed the extend-dataflow-options branch from 595197e to 99b9385 Compare August 1, 2020 06:07
@pyalex pyalex added the kind/feature New feature or request label Aug 1, 2020
@pyalex
Copy link
Copy Markdown
Collaborator Author

pyalex commented Aug 1, 2020

/retest

@pyalex
Copy link
Copy Markdown
Collaborator Author

pyalex commented Aug 1, 2020

/test test-end-to-end-batch-dataflow

@pyalex
Copy link
Copy Markdown
Collaborator Author

pyalex commented Aug 1, 2020

/retest

woop
woop reviewed Aug 1, 2020
View reviewed changes
@woop
Copy link
Copy Markdown
Member

woop commented Aug 1, 2020

/lgtm

@feast-ci-bot
Copy link
Copy Markdown
Collaborator

feast-ci-bot commented Aug 1, 2020

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pyalex, woop

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@pyalex
Copy link
Copy Markdown
Collaborator Author

pyalex commented Aug 1, 2020

/test test-end-to-end-redis-cluster

@feast-ci-bot feast-ci-bot merged commit 294faa5 into feast-dev:master Aug 1, 2020
@mrzzy mrzzy mentioned this pull request Aug 2, 2020
Merged
pyalex pushed a commit that referenced this pull request Aug 2, 2020
Dataflow runner options: disk type & streaming engine (#906)
1bf2faf 
* diskType & streamingEnginer

* edit infra docs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@woop woop woop approved these changes

@davidheryanto davidheryanto Awaiting requested review from davidheryanto

@khorshuheng khorshuheng Awaiting requested review from khorshuheng

@zhilingc zhilingc Awaiting requested review from zhilingc

Assignees

@woop woop

Labels

approved kind/feature New feature or request lgtm size/M

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@pyalex @woop @feast-ci-bot