> For the complete documentation index, see [llms.txt](https://docs.datalogz.io/guides/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.datalogz.io/guides/organizational-setup/connector-onboarding-guide/databricks.md). # Databricks This guide sets up the Databricks connector using an **account-level service principal** with OAuth machine-to-machine (M2M) authentication. The Databricks connector brings your Databricks account into Datalogz Control Tower: workspaces, Unity Catalog assets, lineage, dashboards, and Genie agents, including how people actually use those agents. Datalogz reads **metadata only**: names, owners, structure, lineage and usage of your Databricks assets. It never reads the data inside your tables, and it makes no changes to your Databricks account. {% hint style="warning" %} **Supported clouds:** Databricks on **AWS** and **Azure** (Azure Databricks). **Google Cloud** is in preview: contact before you set it up. Sovereign clouds (Azure Government, Azure China and AWS GovCloud) are not supported. {% endhint %} ### Before you begin You need people with these roles in Databricks: * **Account admin**, to create the service principal and give it the account admin role * **Workspace admin** on each workspace you want monitored, to assign the service principal Unity Catalog should be enabled on the workspaces you want monitored. In the SQL below, replace `` with the service principal's Client ID (Application ID). ### Step 1: Create a service principal in Databricks 1. Sign in to the Databricks **account console** for your cloud: | Cloud | Account console | | ------------ | --------------------------------------- | | AWS | `https://accounts.cloud.databricks.com` | | Azure | `https://accounts.azuredatabricks.net` | | Google Cloud | `https://accounts.gcp.databricks.com` | 2. Go to **User management** > **Service principals** > **Add service principal**. Give it a recognizable name, for example `datalogz-control-tower`. 3. Open the service principal, go to **Roles** and turn on **Account admin**. 4. Open **Credentials & secrets** (OAuth secrets) and click **Generate secret**. 5. Copy the **Client ID** and the **Secret** straight away. Databricks shows the secret only once. 6. Note your **Account ID**. You'll find it in the account menu at the top right of the account console. {% hint style="info" %} **Why account admin?** Datalogz needs it to list your workspaces and Unity Catalog metastores, and to show people's names instead of numeric IDs. Control Tower checks for this role when you connect. Assign it **directly** to the service principal: a role that only comes through a group may not be detected. {% endhint %} {% hint style="info" %} **On Azure:** you can use a Databricks-managed service principal (as above) or a Microsoft Entra ID-managed one. The first account admin of an Azure Databricks account is set up through Microsoft Entra ID, so you may need your Entra ID administrator to grant the account admin role. {% endhint %} OAuth secrets have an expiry date, which you set when you generate them. Rotate the secret before it expires. ### Step 2: Add the service principal to your Databricks workspace Datalogz discovers every workspace in your account and syncs the ones the service principal can access. 1. In the account console, go to **Workspaces**, select a workspace, then **Permissions** > **Add permissions**. Add the service principal with the **User** role. 2. In the workspace's admin settings (**Identity and access** > **Service principals**), confirm the service principal has the **Admin access**, **Databrics SQL access**, **Workspace access** and **consumer access** entitlements. 3. Repeat for every workspace you want monitored.
### Step 3: Grant Unity Catalog read access Run this in a SQL editor as a metastore admin or catalog owner, once for each catalog you want monitored: ```sql -- BROWSE exposes metadata only, never table data. GRANT BROWSE ON CATALOG TO ``; ``` Catalogs without this grant don't appear in Control Tower. New catalogs only show up once they've been granted, so add this grant to your process for creating catalogs. ### Step 4: Enable lineage (system tables and a SQL warehouse) Lineage is read from Databricks system tables using a SQL warehouse. A SQL warehouse is **not** needed for the rest of the inventory. 1. Make sure the `system.access` system schema is enabled on each metastore. You can check in Catalog Explorer under the **system** catalog. 2. Grant read access to the lineage and audit tables: ```sql GRANT USE CATALOG ON CATALOG system TO ``; GRANT USE SCHEMA ON SCHEMA system.access TO ``; GRANT SELECT ON TABLE system.access.table_lineage TO ``; GRANT SELECT ON TABLE system.access.audit TO ``; ``` 3. In each workspace, open **SQL Warehouses**, select a warehouse, open **Permissions** and give the service principal **Can use**. We recommend creating a **small, dedicated serverless** warehouse for Datalogz and granting Can use on that one only. {% hint style="info" %} **You don't select a warehouse in Control Tower.** Datalogz uses a warehouse the service principal has **Can use** on, preferring one that is already running and serverless. If the only warehouses it can use are stopped, it starts one during the sync. Granting Can use on a single small warehouse is how you control which warehouse is used and what it costs. Without a warehouse, lineage for that workspace is limited to the tables Genie agents are grounded on. {% endhint %} ### Step 5: Dashboards and notebooks * **AI/BI dashboards:** share the dashboards, or the folders that contain them, with the service principal at **Can view**. * **Notebooks:** give the service principal **Can read** on the folders you want inventoried, for example `/Workspace/Shared` and `/Workspace/Users`. ### Step 6: Genie agents Genie agents have their own access lists. The service principal only sees agents it has been granted access to, and **workspace admin doesn't override this**. 1. Add the service principal to a group, for example `datalogz-readers`. 2. Share your Genie agents with that group at **Can view**. This brings the agents into Control Tower: owner, warehouse, grounding tables, instructions, sample questions and access list. 3. **Optional, conversation content:** to collect questions, generated SQL and answers from **all** users, grant the group **Can manage** on the agents and turn on **Enable Genie conversation content** in Control Tower (Step 7). With only Can view, Databricks returns just the service principal's own conversations, so agents look unused. {% hint style="warning" %} **Can manage** also allows editing and deleting the agent. Datalogz only reads, but Databricks has no read-only permission for other users' conversations. Grant it only if you want conversation analytics. {% endhint %} On the first sync, Datalogz collects Genie conversations from the last 30 days. After that, it only collects what changed. To collect older history, contact . ### Step 7: Create the connector in Control Tower 1. In Control Tower, go to **Settings** > **Connectors** > **BI Connectors** and click **New Connector**. 2. In **Select a BI platform**, choose **Databricks**. 3. Enter: * **Cloud:** choose **AWS**, **Azure** or **Google Cloud**. Each option shows its account console host; pick the one you signed in to in Step 1. * **Account ID** * **Client ID** and **Client Secret** from Step 1 4. Click **Connect**. Control Tower checks the credentials and the account admin role. 5. Review the **Workspace Preflight**. It lists every workspace Datalogz discovered, with its system schema status and the number of catalogs the service principal can access. Warnings here don't block setup: you can fix them later using the troubleshooting table. 6. Enter a **Connector Name** and choose the **Asset Snapshots** schedule: every day or every week, at a time you choose. 7. Optionally, turn on **Enable Genie conversation content** (see Step 6). 8. Click **Next**, review the summary, then click **Confirm and Finish**. The first sync of a large estate can take a while, especially for workspaces with tens of thousands of notebooks. If a very large workspace can't be fully inventoried within one sync, Datalogz loads what it collected and marks that workspace as incomplete. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.datalogz.io/guides/organizational-setup/connector-onboarding-guide/databricks.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.