Determine this is the right repository
Summary of the feature request
The Dataproc API seems to support the Spark autotuning feature preview, as documented here:
https://cloud.google.com/dataproc-serverless/docs/concepts/autotuning#dataproc_serverless_autotuning-api
We'd like to make use of this feature through the Dataproc Python SDK.
Desired code experience
file: main.py
pyspark_batch = dataproc.types.PySparkBatch(
main_python_file_uri=...,
args=...,
python_file_uris=...,
jar_file_uris=...,
archive_uris=[...],
)
batch = dataproc.types.Batch(
pyspark_batch=pyspark_batch,
runtime_config=dataproc.types.RuntimeConfig(
version=...,
properties=...,
autotuning_config=dataproc.types.AutotuningConfig( ###
cohort="my-cohort", # new
scenarios=["SCALING", "BHJ", "MEMORY"], # config
), ###
),
)
request = dataproc.CreateBatchRequest(
parent=...,
batch=batch,
)
client = dataproc.BatchControllerClient()
op = client.create_batch(request=request)Expected results
A Dataproc Serverless batch is created with autotuning enabled, and the cohort and scenarios set as indicated.
API client name and version
google-cloud-dataproc
Determine this is the right repository
Summary of the feature request
The Dataproc API seems to support the Spark autotuning feature preview, as documented here:
https://cloud.google.com/dataproc-serverless/docs/concepts/autotuning#dataproc_serverless_autotuning-api
We'd like to make use of this feature through the Dataproc Python SDK.
Desired code experience
file: main.py
Expected results
A Dataproc Serverless batch is created with autotuning enabled, and the
cohortandscenariosset as indicated.API client name and version
google-cloud-dataproc