Beta ProductSQL Proxy is currently in beta. Features and APIs may change.
| Mode | How it works | Best for |
|---|
| Passthrough | User credentials forwarded directly to Databricks | Users with existing Databricks credentials |
| Managed | Proxy stores Databricks credentials, users authenticate with proxy tokens | Centralized credential management |
Datafold Admin Account
Datafold requires an admin account to manage infrastructure on your behalf:
- Spin up/down SQL warehouses
- Log query metadata for routing optimization
Setup
Create a service principal or use an existing one with the following permissions:
| Resource | Permission | Purpose |
|---|
| SQL Warehouses | CAN MANAGE | Start/stop warehouses, modify settings |
| Unity Catalog | USE CATALOG, USE SCHEMA | Access metadata for routing |
datafold_sql_proxy schema | CREATE TABLE, INSERT, SELECT | Query logging and routing optimization |
Passthrough Authentication
With passthrough authentication, user credentials are forwarded directly to Databricks. Users authenticate with their own Databricks credentials.
PAT (Personal Access Token)
Use your existing Databricks PAT:
# dbt profiles.yml
my_project:
outputs:
prod:
type: databricks
host: sqlproxy.your-company.datafold.com
http_path: /sql/1.0/warehouses/proxy
token: "{{ env_var('DATABRICKS_TOKEN') }}"
M2M OAuth (Service Principal)
Use your existing Databricks service principal. This method requires tools that support OAuth token exchange (e.g., Databricks JDBC driver, direct API calls):
# Python example with databricks-sql-connector
from databricks import sql
connection = sql.connect(
server_hostname="sqlproxy.your-company.datafold.com",
http_path="/sql/1.0/warehouses/proxy",
credentials_provider=lambda: {
"Authorization": f"Bearer {get_oauth_token()}" # Your M2M OAuth token
}
)
The Python databricks-sql-connector used by dbt does not support M2M OAuth. To use a Databricks service principal with dbt, register the service principal via the Admin API and use a proxy token. Managed Authentication
With managed authentication, the proxy stores Databricks credentials for registered principals. Users authenticate with proxy tokens instead of Databricks credentials.
Setup
- Create a principal via the Admin API with their Databricks credentials
- Generate a proxy token for the principal via the Tokens API
- Distribute the token to users or configure in CI/CD
Using Proxy Tokens
Proxy tokens use the format sqlp_pcp_... (for principals) or sqlp_pm_... (for proxy managers). Use them in place of Databricks PATs:
# dbt profiles.yml
my_project:
outputs:
prod:
type: databricks
host: sqlproxy.your-company.datafold.com
http_path: /sql/1.0/warehouses/proxy
token: "{{ env_var('PROXY_TOKEN') }}" # Proxy token (sqlp_pcp_...)
Benefits
- Centralized credential management - Databricks credentials stored securely in proxy
- Token rotation - Revoke proxy tokens without changing Databricks credentials
- Audit trail - Track which principals executed which queries
- Simplified onboarding - Users don’t need individual Databricks credentials
Required Databricks Permissions
The Databricks credentials used (either your own with passthrough, or the principal’s with managed auth) need the same permissions they would need when connecting to Databricks directly.
SQL Warehouse Access
| Permission | Grants |
|---|
| CAN USE | Execute queries on the warehouse |
Jobs Compute Access
For @datafold:jobs_compute routing:
| Permission | Grants |
|---|
| CAN MANAGE RUN | Execute jobs |
| CAN MANAGE | Create and edit jobs |
Unity Catalog Access
Standard Unity Catalog privileges on catalogs, schemas, and tables:
- USE CATALOG
- USE SCHEMA
- SELECT (for reading data)
- MODIFY (for writing data)
