AI in practice
AI agents in data engineering
A practical guide in three parts: how to work with AI at different levels, how agents help data engineers in Microsoft Fabric, and how they help in Databricks.
Part 1
Levels of working with AI
From a chat window to autonomous agents: ten levels of interacting with AI, what each one needs, with code for the API, tool use and agent loops, and how to choose the right level.
Read →
Part 2
Agents in Microsoft Fabric for data engineers
Where AI assistants and agents fit into day-to-day Fabric engineering: notebooks, Data Factory, pipeline operations, repository workflows and business Q&A, with the real features, costs and limits.
Read →
Part 3
Agents for Databricks development
How to use Genie Code, Lakeflow, local coding agents with Databricks skills and MCP, and Genie spaces in everyday Databricks engineering, with the real features, commands and examples.
Read →
Product names and preview status in this space change quickly. Tool details in this guide were checked against the vendors’ documentation in October 2026, and each part lists its sources. Confirm what is enabled in your own workspace before relying on any of it.
Why this guide exists
“Agent” now covers everything from a chat window to a system that runs unattended overnight. For a data engineer that is confusing, because the right tool, the risk and the controls are completely different at each end of that range.
This guide separates the levels first, then shows where each one is useful in the two platforms most data teams on Azure use, Microsoft Fabric and Databricks. It focuses on what you can set up and try this week, with the real features, limits and configuration.
How to read it
- Start with Part 1 if you want a clear map of the options, from chat to API keys to tool use and agent loops.
- Go to Part 2 or Part 3 if you already know which platform you work on.
- Every part ends with a rollout checklist and links to the sources.
The short version
| If you want to… | Start at | Where to read |
|---|---|---|
| Write and debug code faster in a notebook | An assistant inside the tool | Part 2 (Fabric), Part 3 (Databricks) |
| Let analysts ask questions about governed data | A configured agent (Genie space, Fabric data agent) | Part 2, Part 3 |
| Automate a repetitive engineering task with your own code | The API with a key, then tool use | Part 1 |
| Have an agent change pipeline code in a repository | A coding agent on a branch, checked by CI | Part 2, Part 3 |
| Give many assistants the same governed tools | MCP servers and skills | Part 1, Part 3 |
One rule runs through all three parts: use the lowest level that does the job. Every step up buys more autonomy and costs more in risk, review effort and evaluation.