Databricks platform observability and automation

The problem

As an isolutions engineer, I currently work as a platform and data engineer on a client’s Databricks platform. A shared data platform needs to be observable and its routine operations repeatable, otherwise the team spends its time on manual checks and firefighting.

My part

• Observability for the platform with Prometheus for metrics and Grafana for dashboards.
• Recurring platform tasks automated as GitHub Actions workflows instead of manual steps.
• Databricks as the core data engineering environment.