Beta ProductSQL Proxy is currently in beta. Features and APIs may change.
Explicit Routing
Control routing using @datafold: directives in SQL comments.
Warehouse Size
-- @datafold:warehouse_size=L
SELECT * FROM large_table
2XS, XS, S, M, L, XL, 2XL, 3XL, 4XL
Specific Warehouse
-- @datafold:warehouse=prod-analytics
SELECT * FROM sales.transactions
Jobs Compute
For long-running transformations. Classic compute with spot instances is typically more cost-effective than serverless (which charges by DBU hours).
Serverless
-- @datafold:jobs_compute
CREATE TABLE result AS SELECT ...
Classic (Recommended for Cost)
-- @datafold:jobs_compute type=classic node_type_id=i3.xlarge num_workers=4
CREATE TABLE result AS SELECT ...
Autoscaling
-- @datafold:jobs_compute type=classic node_type_id=i3.xlarge autoscale.min_workers=2 autoscale.max_workers=10
CREATE TABLE result AS SELECT ...
Using Cluster Policies
Databricks cluster policies auto-apply defaults (tagging, instance types, etc.):
-- @datafold:jobs_compute type=classic policy_id=0123456789ABCDEF apply_policy_default_values=true
CREATE TABLE result AS SELECT ...
-- @datafold:jobs_compute type=classic node_type_id=i3.xlarge num_workers=4 custom_tags.cost_center=12345 custom_tags.team=analytics
CREATE TABLE result AS SELECT ...
AWS Spot Instances
-- @datafold:jobs_compute type=classic node_type_id=i3.xlarge num_workers=4 aws_attributes.availability=SPOT_WITH_FALLBACK aws_attributes.spot_bid_price_percent=100
CREATE TABLE result AS SELECT ...
For the complete list of all supported parameters including AWS attributes, cluster logging, security modes, and more, see Jobs Compute Routing Directives. Smart Routing
When enabled, smart routing uses ML to predict optimal warehouse size for queries without explicit directives. Contact your Datafold administrator to enable.
Default Behavior
Queries without routing directives use the default warehouse configured for your connection.