Agents for Databricks development

Where each level lives in Databricks

The levels from Part 1 map onto Databricks like this:

Level	Databricks feature	Notes
2. Assistant in your tool	Genie Code in notebooks, the SQL editor and the Lakeflow Pipelines Editor	Earlier docs call it the Databricks Assistant agent mode or Data Science Agent
4. Configured agent	Genie spaces	Instructions, example SQL, trusted assets
7. Agent loop	A local coding agent (Claude Code, Cursor, Codex CLI, GitHub Copilot and others) working in your repository	Optional extra to Genie Code
8. Shared tools	Agent skills (`databricks aitools`) and managed MCP servers	Skills teach, MCP acts
9. Multi-agent	Agent Bricks supervisor and Genie agents	Documentation is in transition, see below

Before you start: workspace and governance

From the Genie Code documentation:

Partner-powered AI features must be enabled for both the account and the workspace, and the workspace must be in a supported region. Genie Code is a Designated Service that uses Geos for data residency.
Its access follows your own permissions: it can only reach data you can access and do what you are allowed to do. Unity Catalog stays the control point.
It asks for approval before acting. Depending on the mode you click Allow or Skip, or choose Auto-approve to skip future prompts. For multi-step work you can Continue or Reject, and Stop halts it.
Multi-step tasks need the current tab to stay open.
It can generate and run code, so use it only with code and data you trust.
You can add custom instructions (code conventions, preferred libraries) and skills for domain-specific tasks.

A sensible policy: Auto-approve only in a personal development workspace, never anywhere that touches shared or production data.

1. Genie Code in notebooks and the SQL editor

In a notebook, Genie Code can create cells, generate text and code, run cells and read their output to interpret the result. In the SQL editor it plans and generates queries. With your approval it can search tables, edit a notebook, run cells and read outputs.

Example prompts from the documentation:

Analyze @sales_transactions from samples.bakehouse to identify the top-selling product.

Using the @incidents dataset, build a forecast of the daily number of incidents
for the next 2 weeks.

Give this notebook a relevant name.

Engineering prompts that work well (illustrative):

Profile @bronze.sales.orders: null rates, duplicate order_ids, and the min and
max of ingested_at. Summarise what looks wrong, and do not change any table.

Explain why this job is slow. Read the cell output, then suggest changes to
the join order and partitioning.

Notice the pattern: ask for analysis first and say what must not change. Let it propose edits only after you have read its findings.

2. Genie Code for Lakeflow pipelines

In the Lakeflow Pipelines Editor, Genie Code works at pipeline level. According to the Databricks documentation it can:

plan and generate a whole pipeline from one prompt, or extend an existing one,
search workspace tables to find the data,
create and edit multiple pipeline files and show diffs for review,
run a file, a dry run, a pipeline run or a full refresh,
explain transformations and flag unexpected row-count or schema changes,
diagnose and fix errors, and suggest query changes so materialized views can refresh incrementally,
migrate dbt and Informatica projects into Lakeflow pipelines (labelled Beta).

Databricks’ announcement also describes extending pipelines with AutoCDC flows, Auto Loader configuration and expectations, and working inside Declarative Automation Bundles (DABs, called Asset Bundles in older documentation) to add and validate job resources.

Prompts from the documentation:

Build and run a medallion architecture pipeline for fraud detection using the
table transactions and customers in my_catalog.my_schema.

create silver_sales_data in a new file that reads from bronze_sales_data and
cleans the data and adds useful quality expectations.

Fix the failure in this pipeline.

For the second prompt, the shape of output to expect and review (written by hand to illustrate):

CREATE OR REFRESH STREAMING TABLE silver_sales_data (
  CONSTRAINT valid_order_id EXPECT (order_id IS NOT NULL) ON VIOLATION DROP ROW,
  CONSTRAINT non_negative_amount EXPECT (amount >= 0)
)
AS SELECT
  order_id,
  CAST(amount AS DECIMAL(18, 2)) AS amount,
  to_date(order_ts) AS order_date,
  _ingested_at
FROM STREAM(bronze_sales_data);

What to check in the diff every time:

the grain and keys,
whether each expectation drops rows or only records violations,
that nothing is silently filtered,
whether the incremental behaviour is what you intended.

Before it runs code or updates a pipeline, Genie Code asks for approval: Allow, Decline, Allow in this thread, or Always allow. Use “Always allow” only where a mistake costs nothing.

3. Local coding agents with Databricks skills and MCP

Many engineers prefer a coding agent in their own editor or terminal, working on a repository. Databricks supports that with two separate things, and it is important not to confuse them.

Skills teach the agent how to work. The Databricks CLI installs task-specific instruction files for the coding agents it detects:

databricks aitools install --agents claude-code --scope project --skills bundles
databricks aitools list
databricks aitools update

--agents takes a list such as claude-code,cursor. Supported agents include Claude Code, Cursor, Codex CLI, GitHub Copilot, OpenCode, Antigravity, Gemini CLI, Goose, Kiro and Pi.
--scope project keeps the skills in the current project. The default for install is global.
Skill areas listed by Databricks include bundles, the Databricks CLI, Lakeflow jobs and pipelines, Unity Catalog, Structured Streaming, Genie, AI/BI dashboards, Agent Bricks and MLflow evaluation.
Installing skills does not set up MCP servers.
Skills from arbitrary third-party repositories are not vetted by anyone. Read them before installing, as you would any dependency.

MCP servers let it do the work. Databricks hosts managed MCP servers, governed by Unity Catalog:

Server	URL pattern	Status in the docs
Genie (Unity Gateway)	`https://<workspace-hostname>/ai-gateway/mcp-services/system.ai.genie_one_mcp`	Generally available
SQL (Unity Gateway)	`https://<workspace-hostname>/ai-gateway/mcp-services/system.ai.dbsql`	Beta, needs an account admin to enable
SQL (legacy)	`https://<workspace-hostname>/api/2.0/mcp/sql`	See the docs
Unity Catalog functions	`https://<workspace-hostname>/api/2.0/mcp/functions/{catalog}/{schema}/{function_name}`	See the docs

Databricks recommends the Genie server for read-only questions from external assistants. The Unity Gateway servers apply Unity Catalog grants and policies, while the legacy endpoints depend only on permissions on the underlying resources. In a client that accepts a remote MCP server URL, add the URL and sign in. The Databricks docs have setup guides for coding agents such as Claude Code, Codex and Cursor.

The workflow mirrors the Fabric one. Keep pipelines, jobs and configuration as code in a bundle, and give the agent only the development target:

# databricks.yml
bundle:
  name: sales_pipelines

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://<dev-workspace-hostname>
  prod:
    mode: production
    workspace:
      host: https://<prod-workspace-hostname>

databricks bundle validate -t dev    # the agent can run this
databricks bundle deploy -t dev      # and this, against dev only

The agent edits bundle and pipeline files on a feature branch and deploys to dev.
A pull request runs databricks bundle validate, tests, and a dry run of the pipeline in CI.
A human reviews and merges. A CI service principal, not the agent, deploys to prod.

Good first tasks: adding expectations and tests, converting a notebook job to a bundle resource, documenting tables, and reviewing pipeline code for anti-patterns. Avoid asking it to “optimise the cluster” in a production workspace.

4. Genie spaces: a governed Q&A agent

A Genie space lets people ask plain-language questions about a chosen set of tables. Its quality comes from what you configure:

Instructions: plain-text notes on business rules, terminology and date formats.
Example SQL queries: they teach Genie how to answer related questions.
Trusted assets: parameterized example queries or SQL functions. Genie marks an answer trusted only when one is used verbatim, so they are the right tool for questions that must always be answered the same way. Trusted assets complement instructions, they do not replace them. Check the current preview status in your workspace.

A trusted asset is a table function with a good comment:

CREATE OR REPLACE FUNCTION gold.genie.revenue_by_region(start_date DATE, end_date DATE)
RETURNS TABLE (region STRING, net_revenue DECIMAL(18, 2))
COMMENT 'Net revenue per sales region between two dates. Use for any question about revenue by region.'
RETURN
  SELECT c.region, SUM(f.net_revenue)
  FROM gold.sales.fct_orders f
  JOIN gold.sales.dim_customer c ON f.customer_key = c.customer_key
  WHERE f.order_date BETWEEN start_date AND end_date
  GROUP BY c.region;

Keep these functions in a dedicated schema so they are easy to share and review. Anyone with at least CAN EDIT on the space can add or remove trusted assets, so treat that permission as you would write access.

The same descriptions that help Genie help every other agent, so do the data modelling work from Part 1 first: documented grain, one definition per metric, and precomputed flags for awkward date logic. Then keep golden questions and run them after each change.

5. Orchestrating several agents

Agent Bricks is Databricks’ area for building and coordinating agents. A Supervisor Agent can orchestrate Genie agents, agent endpoints, Unity Catalog functions, MCP servers and custom agents, so one entry point can send a sales question to a Genie space and a “how do I…” question to a documentation agent.

The documentation is in transition: a newer page points low-code users to Genie agents and code-based agents to the Agent Bricks CLI, and describes the Supervisor API as deprecated, while an older page still presents the Supervisor Agent as current. Check which applies to your workspace before building.

Do not start here. Build and evaluate each specialist on its own first, and add an orchestrator only when one agent’s tool list has become too long to choose from reliably.

6. Rollout checklist for a Databricks team

Enable partner-powered AI features for a pilot workspace and check the region requirements.
Add team custom instructions for Genie Code: naming rules, layer rules, preferred libraries.
Put pipelines and jobs into bundles with separate dev and prod targets.
Install skills at project scope, and review what they contain.
Give coding agents access to dev only. CI service principals deploy to production.
Prefer the Genie MCP server for external assistants, and check its status and grants.
Document tables and columns in Unity Catalog before building a Genie space.
Keep golden questions for every space, and review trusted assets like code.
Keep approvals on, except in personal development workspaces.

Sources

Names, preview status and limits change, so the documentation is the source of truth. Code marked illustrative was written for this guide.