Skip to main content

Databricks

Databricks Prerequisites​

CloudCADI authenticates to Databricks via OIDC-based workload identity federation instead of a static client secret.

Before you start

Note down the OIDC values for CloudCADI's hosting environment - Issuer, Subject, and Audience - from the table below. You'll use these when creating the federation policy in Databricks.

Hosted onIssuerSubjectAudience
AWS (EKS)AWS Outbound Identity Federation issuer: https://<uuid>.tokens.sts.global.api.awsIAM Role ARN deployed on EKS: arn:aws:iam::<account-id>:role/<role-name>databricks
Azure (AKS)https://sts.windows.net/<tenant-id>/App Registration Object IDhttps://management.azure.com/
GCP (GKE)https://accounts.google.comService Account Unique IDdatabricks

Databricks Federation Setup Walkthrough​

This document describes the step-by-step procedure to federate a Databricks account with CloudCADI and connect its workspaces.

Step 1 - Create a service principal in Databricks​

  1. In the Databricks account console, go to User management → Service principals, then click Add service principal. Add service principal
  2. To create a new service principal, enter a service principal name (e.g. CloudCADI-service-principal) and click on Add service principal. New service principal
  3. Open the new principal's Principal information tab and note its UUID, which will be used as the Client ID in CloudCADI. Principal information
  4. Go to the Roles tab and enable Account admin for the service principal. Assign account admin role

Step 2 - Create a federation policy​

  1. On the service principal's Credentials & secrets tab, under Federation policies, click Create policy. Create federation policy
  2. Set Federated credential provider to Custom. Select custom provider
  3. Fill in the OIDC values for CloudCADI's hosting environment (see Databricks Prerequisites), then click Create policy:
    • Issuer URL
    • Subject
    • Audiences
    • Subject claim Federation policy details
  4. From the account menu (top-right), copy the Account ID - it is needed on the CloudCADI side. Copy Databricks account ID

Step 3 - Connect the account in CloudCADI​

  1. In CloudCADI, go to Settings → Databricks and select the Provider - AWS, GCP, or Azure. Every provider requires the same three fields, populated with that provider's identity details:

    • Account ID - paste the Account ID copied
    • Account Name - the Databricks account name
    • Client ID - paste the ID copied

    Click Done. CloudCADI Databricks settings

  2. Once the account shows as Connected, click Fetch Workspaces to Connect. Fetch workspaces

  3. After workspaces sync, click Connect on the workspace to link. Connect workspace

info

A metastore can be shared across multiple workspaces. For each metastore, CloudCADI creates a SQL warehouse to query system table data.

Step 4 - Grant catalog permissions to the service principal​

  1. From the Databricks account console Workspaces list, click Open to launch the workspace. Open workspace
  2. In the workspace, go to Catalog → system → Permissions, then click Grant. Grant permissions on system catalog
  3. Add the service principal as a Principal, check USE CATALOG, USE SCHEMA, and SELECT, then click Confirm. Grant privileges dialog
info

The service principal needs at least USE CATALOG, USE SCHEMA, and SELECT on the system catalog for CloudCADI to read Databricks usage and billing data.