Databricks Data + AI Summit 2025

Introduction

Now that the Databricks Data + AI Summit 2025 has wrapped up, I’m reflecting on the exciting updates and innovations showcased this year. Databricks has made notable strides in GenAI integration, enhanced the developer experience, and strengthened platform governance. These improvements build on the progress from previous summits while introducing new tools and capabilities. However, from the perspective of daily data engineering tasks, there are still opportunities for further growth. Here are some of the key features that stood out to me, along with what I’m hoping to see in future releases. 

Lakebase

Lakebase feels like a real turning point for operational databases in the AI era—finally bringing a fully managed, serverless Postgres that’s natively integrated with the lakehouse. Instead of wrestling with complex ETL pipelines or juggling separate stacks for analytics and transactions, Lakebase lets teams work with familiar Postgres tools and SQL, while syncing operational data directly into analytics and AI-driven apps. The decoupled compute and storage model means you can independently scale resources to deliver low-latency, high-throughput solutions for everything from customer-facing apps to dashboards and workflows.  

Features like point-in-time recovery, performance monitoring, and seamless workspace integration are built-in, and with support for ACID transactions, online feature stores, and Unity Catalog, Lakebase lets developers build, scale, and manage intelligent applications—without the operational headaches that come with traditional databases. It’s a major step forward for anyone looking to unify their operational and analytical workloads in a single, governed platform.

Advantages of Lakebase over traditional databases: 

Here are some of the advantages of Lakebase: 

1. Seamless Integration of Operational and Analytical Data 
2. Separation of Compute and Storage 
3. Serverless and Fully Managed Experience 
4. Branching and Copy-on-Write Capabilities 
5. Open Source Foundation and Ecosystem Support 
6. Unified Governance and Security 
7. Optimized for AI Agents and Automation

Lakeflow

Lakeflow is being considered as unified end-to-end data engineering approach by incorporating following components: 

1. Lakeflow connect 
2. Lakeflow Declarative Pipelines 
3. Lakeflow Jobs 

Lakeflow designer is the new AI powered visual builder for building production grade data-pipelines enabling no-code.  

Lakeflow really stands out as a game-changer for data engineering teams like ours. Instead of juggling a mess of disconnected tools and struggling with governance headaches, we finally have a unified solution built right into the Data Intelligence Platform. With Lakeflow handling everything from ingestion to orchestration—and integrating seamlessly with Unity Catalog for governance—we get full visibility and control across our pipelines. It’s made our workflows so much simpler, and we’re spending far less time on tool integration and more on actually delivering value from our data.

Lakeflow Connect: 

In addition to already established Lakeflow connect, more built-in connectors have been established for reliable and simple ingestion, which are optimized for efficient data extraction with enablement of CDC. A new feature Zerobus have been introduced as a Lakeflow Connect API which is used for near real time data ingestion with high throughput (100MB/S) and near real-time latency (<5 sec). This efficient ingestion framework delivers scalable performance and is seamlessly integrated with the Databricks Platform, enabling you to immediately tap into a wide range of analytics and AI capabilities.

Lakeflow Declarative Pipelines: 

This brings power of Spark Declarative Pipelines to Databricks Data Intelligence Platform. It is 100% source compatible with open standard which means we can develop pipeline once and can use it any time. One good thing is it also supports backward compatibility which makes it easier to use already existing DLT pipelines without any requirement of rewriting. These also offer hands-off serverless compute plus unified governance by connecting to Unity Catalogue.  

The newly introduced IDE for Data Engineering is a contemporary, unified workspace designed to simplify and accelerate the pipeline development process. It features 

1. Code and DAG side by side, with dependency visualization and instant data previews 
2. Context-aware debugging that surfaces issues inline 
3. Built-in Git integration for rapid development 
4. AI-assisted authoring and configuration 

Lakeflow Jobs: 

With the general availability of Lakeflow, Databricks is taking orchestration to the next level by evolving Workflows into Lakeflow Jobs—a unified, robust orchestrator that’s tightly integrated with the entire data engineering stack. Lakeflow Jobs empowers teams to manage everything from declarative pipelines and notebooks to SQL queries, dbt transformations, and even publishing AI or BI dashboards, all within a single platform. The addition of advanced control flow features like conditional execution, loops, and dynamic parameterization, along with flexible triggers such as file arrival and table updates, means jobs only run when new data is available—maximizing efficiency. Plus, serverless job execution brings automatic performance optimizations. What really stands out is the end-to-end observability: visual monitoring, detailed debugging, proactive alerts, historical insights, and built-in data quality expectations all come together to give data teams full control and confidence over every stage of their pipelines.

Lakeflow Designer

This is a no-code pipeline builder integrated with Databricks Database Intelligence platform and AI powered. This provides a visual canvas and built-in natural language interface with which we will be able to build pipelines and perform data analysis without a single piece of code.  

Current no-code tools live outside databricks which cause following issues: 

Siloed Workflows as every team builds in different tools and when need to change data platforms data engineers need to rebuild them entirely.  

Production challenges as external pipelines run without governance or observability which makes them difficult to maintain 

Limited AI productivity as dealing with different tools and data being present at different location it is difficult to get required results from AI and we will be generally getting generic answers to question when there is no access to data. 

Lakeflow designer helps overcome all these issues as we have everything in one place (i.e Databricks) where pipelines are created, managed and governed which helps in governance and scaling from day one with no rewrites. 

Lakeflow Designer is set to transform collaborative data engineering by offering a unified, visual platform where business users can easily build pipelines with no-code tools, while data engineers retain the ability to inspect, edit, and productionize these pipelines using the familiar Lakeflow Declarative Pipelines framework. This seamless handoff reduces redundant work and ensures everything is production-ready from the start, with built-in versioning, governance via Unity Catalog, and comprehensive observability. What truly sets Designer apart is its AI-driven development, which leverages deep integration with the Databricks Intelligence Platform to generate pipelines grounded in your organization’s unique data context and usage patterns. With Private Preview launching soon, Lakeflow Designer promises to empower more users to create reliable, governed pipelines—without adding complexity or risk. 

Agent Bricks

It is Databricks new platform designed to radically simplify and accelerate the development of high-quality, domain-specific AI agents for enterprises. Instead of requiring teams to manage the complexity of agent development, Agent Bricks lets users focus on defining the agent’s purpose and providing natural language feedback, while the platform handles the rest—automatically generating evaluation suites, optimizing performance, and continually improving quality through advanced techniques like Agent Learning from Human Feedback (ALHF). This automation eliminates the need for manual tuning, reduces development timelines from weeks to days, and ensures agents are both cost-efficient and production-ready. 

Agent Bricks is optimized for key industry use cases such as structured information extraction, knowledge assistance, custom text transformation, and orchestrated multi-agent systems. The platform leverages enterprise data to generate synthetic datasets and task-specific benchmarks, enabling robust, repeatable evaluation and optimization without manual labeling or extensive trial-and-error. Integrated with the Databricks Data Intelligence Platform, Agent Bricks brings together security, governance, and seamless deployment, making it easy for organizations to build, monitor, and scale AI agents that deliver consistent business value. 

Lakebridge

It is the free migration tool for automatic migration of data from traditional warehouses to Databricks. It automates 80% of migration process like profiling, SQL conversion, validation and reconciliation. Lakebridge offers following strategic advantages: 

1. Clear insight into migration scope and complexity
2. Automated, high-fidelity code conversion and validation
3. Easy reconciliation of migrated workloads for data accuracy
4. A proven path to accelerate your journey to Databricks SQL 

It also supports lift-and-shift and hybrid migration approaches. It delivers end-to-end migration through following three key components: 

Analyzer which performs detailed assessment of legacy warehouse environment. 

Converter helps intelligently convert legacy ETL workflows and SQL scripts into performant, compatible Databricks or Spark SQL code. 

Validator which ensures data accuracy and correctness. 

It also provides built-in dashboards and reports which allows teams to track progress, validate results and better understand their evolving data landscape, by speeding up adoption and delivering value faster. It currently support more than 10 legacy data warehouses and ETL tools—including Teradata, Snowflake, Oracle, SQL Server, and Informatica with more coming soon.  Leveraging the proven migration technology from BladeBridge, which is trusted by top system integrators like Accenture and Capgemini, Lakebridge offers advanced SQL parsing, code conversion, and validation. 

Databricks is dedicated to improving the migration experience by making it quicker, more intelligent, and highly predictable. In the near future, they will launch a next-generation migration solution powered by cutting-edge AI technology. Lakebridge is also planning to incorporate into Mosaic AI, creation of dedicated Data Migration module and a separate Graphic User Interface (GUI) in future. 

Databricks Free Edition 

The new Free Edition from Databricks which is a hot-topic is a fantastic way for users to dive into AI and data engineering, offering hands-on experience with Mosaic AI for building agents and applications, and step-by-step guidance on preparing, deploying, and governing AI systems. It’s perfect for collaborative learning, with shared notebooks supporting Python, SQL, and more—ideal for group projects and experimentation. Users can also create interactive dashboards, analyze real datasets using the built-in SQL editor, and master data pipeline development with Lakeflow. The AI-powered Databricks Assistant is always on hand to help with coding, making it easier to learn and troubleshoot as you go. Plus, the ability to invite friends and teammates for real-time collaboration truly mirrors a modern workplace environment, making learning both engaging and practical. 

Features that became tangible 

Here are updates on some of the anticipated features as per the previous blog Anticipating Databricks Data + AI Summit 2025: What Data Engineers Should Watch For 

Unity Catalog & Lakehouse Federation 

1. Delivered on unified governance across multiple data sources. 
2. Performance improvements with smart caching and enhanced pushdown optimizations reduced latency. 
3. Expanded native support for source-side transformations beyond just SQL. 

SQL Editor 

1. Progressed toward a truly federated workspace with live querying and transformation of external data sources without mandatory data registration.
2. Simplified hybrid analytics workflows for analysts and engineers. 

Mosaic AI 

1. Introduced Git-backed versioning and lineage tracking for reusable AI functions, addressing production-readiness concerns.
2. Improved UI for managing, searching, and understanding dependencies of AI components. 

Mosaic AI Agent Framework 

1. Broadened availability beyond private preview.
2. Integrated with third-party observability tools (e.g., Langfuse, Trulens) for enhanced monitoring.
3. Added automated evaluation pipelines and telemetry for robust GenAI production workloads.

AI/BI Dashboards (Genie) 

1. Enhanced multi-source join capabilities and dashboard interactivity (cross-page filtering, drill-through).
2. Introduced AI-driven visualization recommendations based on user behavior. 
3. Improved conversational analytics for more accurate, natural language insights. 

Closing Thoughts

If I had to sum it up, the 2025 Databricks Data + AI Summit truly marked a turning point for Databricks—especially for those of us eager to move faster without sacrificing reliability or governance. With Lakebase bringing managed, AI-ready Postgres to the lakehouse, Lakeflow Designer making collaborative pipeline development accessible to everyone, and Agent Bricks automating the creation and optimization of enterprise AI agents, we finally saw many long-anticipated features become reality. Add in the advancements in Unity Catalog, Mosaic AI, and no-code analytics with Databricks One, and it’s clear Databricks is doubling down on unification, openness, and AI-powered productivity for teams across the data spectrum.