This exercise is to bootstrap a databricks / aws tenancy using stackql-deploy. It is an important use case for platform bootstrap and we are excited to perform it with the stackql toolchain. We hope you enjoy and find this valuable. Please drop us a note with your forthright opinion on this and check out our issues on github.
Please take the greatest care in performing this exercise; it will incur expenses, as it involves creating (and destroying) resources which cost money. Please be aware that you must cancel your databricks subscription after completing this exercise, otherwise you will incur ongoing expenses. That is, do not skip the section Cancel databricks subscription. We strongly advise that you verify all resources are destroyed at the conclusion of this exercise. Web pages and certain behaviours may change, so please be thorough in your verification. We will keep this page up-to-date on a best effort basis only. It is very much a case of owner onus applies.
Dependencies:
- aws Account Created.
- Required clickops to set up databricks on aws:
- Turn on aws Marketplace
databricksoffering, using the aws manage subscriptions page, per Figure S1. - Follow the suggested setup flow as directed, from this page. These clickops steps are necessary at this time for initial account setup. The way I followed this, it created a workspace for me at setup, per Figure S3. We shall not use this one and rather, later on we shall dispose of it; because we do not trust auto-created resources out of hand. In the process of creating the databricks subscription, a second aws account is created.
- Copy the databricks account id from basically any web page in the databricks console. This is done by clicking on the user icon at the top RHS and then the UI provides a copy shortcut, per Fugure U1. Save this locally for later use, expanded below.
- We need the aws account id that was created for the databricks subscription. It is not exactly heralded by the web pages, nor is it actively hidden. It can be captured in a couple of places, including the databricks storage account creatted in the subscription flow, per Figure XA1. copy and save this locally for later use, expanded below.
- Create a service principal to use as a "CICD agent", using the page shown in Figure S4.
- Grant the CICD agent account admin role, using the page shown in Figure S5.
- Create a secret for the CICD agent, using the page shown in Figure S6. At the time you create this, you will need to safely store the client secret and client id, as prompted by the web page. These will be used below.
- Turn on aws Marketplace
- Setup your virtual environment, from the root of this repository
cicd/setup/setup-env.sh.
Now, is is convenient to use environment variables for context. Note that for our example, there is only one aws account apropos, however this is not always the case for an active professional, so while DATABRICKS_AWS_ACCOUNT_ID is the same as AWS_ACCOUNT_ID here, it need not always be the case. Create a file in the path examples/databricks/serverless/sec/env.sh (relative to the root of this repository) with contents of the form:
#!/usr/bin/env bash
export AWS_REGION='us-east-1' # or wherever you want
export AWS_ACCOUNT_ID='<your aws account ID>'
export DATABRICKS_ACCOUNT_ID='<your databricks account ID>'
export DATABRICKS_AWS_ACCOUNT_ID='<your databricks aws account ID>'
# These need to be created by clickops under [the account level user managment page](https://accounts.cloud.databricks.com/user-management).
export DATABRICKS_CLIENT_ID='<your clickops created CICD agent client id>'
export DATABRICKS_CLIENT_SECRET='<your clickops created CICD agent client secret>'
## These can be skipped if you run on [aws cloud shell](https://docs.aws.amazon.com/cloudshell/latest/userguide/welcome.html).
export AWS_SECRET_ACCESS_KEY='<your aws secret per aws cli>'
export AWS_ACCESS_KEY_ID='<your aws access key id per aws cli>'
Now, let us do some sanity checks and housekeeping with stackql. This is purely optional. From the root of this repository:
source examples/databricks/serverless/convenience.sh
stackql shell
This will start a stackql interactive shell. Here are some commands you can run (I will not place output here, that will be shared in a corresponding video):
registry pull databricks_account v24.12.00279;
registry pull databricks_workspace v24.12.00279;
-- This will fail if accounts, subscription, or credentials are in error.
select account_id FROM databricks_account.provisioning.credentials WHERE account_id = '<your databricks account id>';
select account_id, workspace_name, workspace_id, workspace_status from databricks_account.provisioning.workspaces where account_id = '<your databricks account id>';For extra credit, you can (asynchronously) delete the unnecessary workspace with delete from databricks_account.provisioning.workspaces where account_id = '<your databricks account id>' and workspace_id = '<workspace id>';, where you obtain the workspace id from the above query. I have noted that due to some reponse caching it takes a while to disappear from select queries (much longer than disappearance from the web page), and you may want to bounce the stackql session to hurry things along. This is not happening on the stackql side, but session bouncing forces a token refresh which can help cache busting.
Time to get down to business. From the root of this repository:
source examples/databricks/serverless/convenience.sh
source venv/bin/activate
install stackql-deploy from https://github.com/stackql/stackql-deploy-rs/releasesalternatively set the
AWS_REGION,AWS_ACCOUNT_ID,DATABRICKS_ACCOUNT_ID,DATABRICKS_AWS_ACCOUNT_IDalong with provider credentialsAWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,DATABRICKS_CLIENT_ID,DATABRICKS_CLIENT_SECRET
Then, do a dry run (good for catching some environmental issues):
target/release/stackql-deploy build \
examples/databricks/serverless dev \
-e AWS_REGION=${AWS_REGION} \
-e AWS_ACCOUNT_ID=${AWS_ACCOUNT_ID} \
-e DATABRICKS_ACCOUNT_ID=${DATABRICKS_ACCOUNT_ID} \
-e DATABRICKS_AWS_ACCOUNT_ID=${DATABRICKS_AWS_ACCOUNT_ID} \
--dry-runYou will see a verbose rendition of what stackql-deploy intends to do.
Now, let use do it for real:
stackql-deploy build \
examples/databricks/serverless dev \
-e AWS_REGION=${AWS_REGION} \
-e AWS_ACCOUNT_ID=${AWS_ACCOUNT_ID} \
-e DATABRICKS_ACCOUNT_ID=${DATABRICKS_ACCOUNT_ID} \
-e DATABRICKS_AWS_ACCOUNT_ID=${DATABRICKS_AWS_ACCOUNT_ID} \
--show-queriesThe output is quite verbose, concludes in:
2025-02-08 12:51:25,914 - stackql-deploy - INFO - 📤 set [databricks_workspace_id] to [482604062392118] in exports
2025-02-08 12:51:25,915 - stackql-deploy - INFO - ✅ successfully deployed databricks_workspace
2025-02-08 12:51:25,915 - stackql-deploy - INFO - deployment completed in 0:04:09.603631
🚀 build complete
Success!!!
We can also use stackql-deploy to assess if our infra is shipshape:
stackql-deploy test \
examples/databricks/serverless dev \
-e AWS_REGION=${AWS_REGION} \
-e AWS_ACCOUNT_ID=${AWS_ACCOUNT_ID} \
-e DATABRICKS_ACCOUNT_ID=${DATABRICKS_ACCOUNT_ID} \
-e DATABRICKS_AWS_ACCOUNT_ID=${DATABRICKS_AWS_ACCOUNT_ID} \
--show-queriesAgain, the output is quite verbose, concludes in:
2025-02-08 13:15:45,821 - stackql-deploy - INFO - 📤 set [databricks_workspace_id] to [482604062392118] in exports
2025-02-08 13:15:45,821 - stackql-deploy - INFO - ✅ test passed for databricks_workspace
2025-02-08 13:15:45,821 - stackql-deploy - INFO - deployment completed in 0:02:30.255860
🔍 tests complete (dry run: False)
Success!!!
Now, let us teardown our stackql-deploy managed infra:
stackql-deploy teardown \
examples/databricks/serverless dev \
-e AWS_REGION=${AWS_REGION} \
-e AWS_ACCOUNT_ID=${AWS_ACCOUNT_ID} \
-e DATABRICKS_ACCOUNT_ID=${DATABRICKS_ACCOUNT_ID} \
-e DATABRICKS_AWS_ACCOUNT_ID=${DATABRICKS_AWS_ACCOUNT_ID} \
--show-queriesTakes its time, again verbose, concludes in:
2025-02-08 13:24:17,941 - stackql-deploy - INFO - ✅ successfully deleted AWS_iam_cross_account_role
2025-02-08 13:24:17,942 - stackql-deploy - INFO - deployment completed in 0:03:21.191788
🚧 teardown complete (dry run: False)
Success!!!
Now, let us do some sanity checks and housekeeping with stackql. This is purely optional. From the root of this repository:
source examples/databricks/serverless/convenience.sh
stackql shell
This will start a stackql interactive shell. Here are some commands you can run (I will not place output here):
registry pull databricks_account v24.12.00279;
registry pull databricks_workspace v24.12.00279;
select account_id, workspace_name, workspace_id, workspace_status from databricks_account.provisioning.workspaces where account_id = '<your databricks account id>';
This is very important.
Go to the aws Marketplace manage subscriptions page, navigate to databricks and then cancel the subscription.
Figure S1: Create aws databricks subscription.
Figure S2: Awaiting aws databricks subscription resources.
Figure S3: Auto provisioned workspace.
Figure U1: Capture databricks account id.
Figure XA1: Capture cross databricks aws account id.
Figure S4: Create CICD agent.
Figure S5: Grant account admin to CICD agent.
Figure S6: Generate secret for CICD agent.







