Spark Declarative Pipelines, Structured Streaming or Lakeflow Jobs? A practical guide

In data engineering, choosing a technology is rarely about finding one tool that is simply the best overall. In practice, the better question is which approach fits a given pipeline, how much control the team needs and how expensive the solution will be to maintain over time. This issue becomes especially visible in Databricks, where teams often compare Spark Declarative Pipelines, Structured Streaming and Lakeflow Jobs. Although these solutions serve different purposes, the boundaries between them are not always obvious at first. In our own projects, this choice usually appears not at the beginning of the work but once the pipeline starts to grow and operational trade-offs become harder to ignore.

In practice, one of the most common mistakes is to treat these solutions as direct substitutes. Spark Declarative Pipelines and Structured Streaming partly overlap, because both can be used for batch and streaming workloads. However, they operate at different levels. Structured Streaming is the underlying stream-processing engine, while SDP builds on it and simplifies pipeline development. Lakeflow Jobs fits into the picture differently, because it handles orchestration, scheduling and task dependencies rather than the processing model itself. For that reason, the key question is not simply which tool to choose, but where each of these solutions fits in the broader data pipeline.

Common challenges in selecting the right approach

Choosing between Spark Declarative Pipelines, Structured Streaming and Lakeflow Jobs can be difficult because these solutions address different layers of a data platform.

One practical challenge is defining the project’s real priority. Some pipelines need to be delivered quickly and kept easy to maintain, while others require fine control over streaming logic, latency, or execution behavior. Another issue is that teams often focus on transformation logic first and only later realize that scheduling, dependencies, retries and operational monitoring are just as important in production. The decision is also harder because these tools are not purely competing options. In many cases, they complement one another. Spark Declarative Pipelines simplify pipeline development; Structured Streaming offers more control over streaming logic and Lakeflow Jobs manages multi-step workflow execution. In many projects, the wrong choice is not choosing the weaker tool but assigning the right tool to the wrong role.

A poor choice at this stage often leads to avoidable complexity, higher maintenance effort, and scaling issues later on. This is why the decision should be treated not only as a technical one, but also as an architectural and operational decision.

Overview of the three approaches

Spark Declarative Pipelines

Spark Declarative Pipelines is a Databricks framework for building batch and streaming pipelines in SQL and Python. In this model, developers describe the pipeline logic declaratively, while the platform takes care of execution planning, incremental processing and part of the operational overhead. Databricks documentation also highlights built-in support for data quality expectations and monitoring, which helps improve pipeline reliability. An important recent change is that Declarative Pipelines became open source in June 2025, following Databricks’ contribution of the project to Apache Spark. This makes the framework more open and potentially more relevant outside the Databricks ecosystem.

Structured Streaming

Structured Streaming is the Apache Spark engine typically used in Databricks for incremental and near-real-time processing. Its core idea is that the logic for incoming data can be expressed similarly to batch processing, while the engine executes computations continuously and incrementally. Databricks also emphasizes that Structured Streaming provides end-to-end fault tolerance and exactly-once processing guarantees.

Lakeflow Jobs

Lakeflow Jobs focuses on workflow automation rather than data processing itself. It enables teams to organize multiple tasks into a single workflow, manage dependencies, define schedules or retries, and react to triggers such as time events, table updates, or new files. In other words, it should be seen as an orchestration layer, not as an alternative to a processing engine.

Direct comparison

Case studies

While the perspective presented in this article is based on our own experience and architectural reasoning, it is also useful to look at selected industry examples provided by Databricks. They help illustrate how similar decisions are made in different organizations and confirm that the trade-offs discussed here appear in real production environments.

Case 1. From Kafka events to reliable Delta Tables

Block needed a way to make streaming data available in Delta Lake quickly, while still supporting change data capture, simple transformations and data quality checks. At the same time, the team wanted to avoid building and maintaining too much custom workflow logic. To address that, the company chose Spark Declarative Pipelines as a more managed and declarative way to build end-to-end streaming pipelines. In practice, Kafka events were ingested into bronze tables and then merged into higher-quality Silver tables. According to Databricks, this reduced pipeline development time from days to hours, improved development velocity, and lowered operational overhead. The example suggests that this approach works especially well when faster delivery and easier maintenance matter more than full implementation freedom.

In this kind of scenario, Spark Declarative Pipelines tends to be the better option when simplicity and speed of delivery are more important than implementation freedom.

Case 2. Rebuilding a real-time streaming application at scale

FreeWheel was running a large streaming application that processed ad events at very large scale, but the platform struggled with instability, heavy maintenance effort, and unwanted spikes in processing latency. In response, the company rebuilt this part of the platform around Spark Structured Streaming on Databricks, using it as the core processing model for a more stable real-time architecture. Databricks reports a sevenfold improvement in system performance, along with better data quality and lower maintenance effort. What stands out in this example is that Structured Streaming becomes especially valuable when continuous processing, low latency and tighter control over execution are more important than keeping the implementation as simple as possible.

In a setup like this, Structured Streaming is the more suitable choice when continuous processing, lower latency and execution control take priority over implementation simplicity.

Case 3. Orchestrating complex data workflows across multiple sources

The Rank Group was dealing with fragmented data spread across many systems and warehouses, which made orchestration, data quality and cost control increasingly difficult. The company also needed easier access to unified data and faster delivery of insights. To improve this setup, it adopted Lakeflow Jobs to orchestrate and manage data from multiple sources inside its Lakehouse environment. Databricks describes the solution as a pattern-based framework that handles inputs from APIs, databases, message queues, and files. According to the published results, the system supported nearly 20 million daily transactions, increased productivity in the data operations engineering team by 30% and contributed to more than £1.2 million in cost savings. In this case, the main value did not come from changing the processing model itself, but from improving the coordination of dependent tasks within a single production workflow.

In a case like this, Lakeflow Jobs is particularly useful when the main challenge is coordinating many dependent tasks, schedules and sources rather than designing the processing logic itself.

How to choose

Summary

The comparison shows that these three options should not be treated as direct substitutes. Spark Declarative Pipelines is best suited to projects that need fast delivery and lower maintenance overhead. Structured Streaming is more appropriate when a team needs high control, custom processing logic and low-latency execution. Lakeflow Jobs is most useful when the main need is coordinating multiple tasks within a larger workflow. Before making a final decision, you should verify the required latency, the level of implementation complexity, your team can support, the need for orchestration and the long-term maintenance cost. In practice, the best result often comes not from choosing one tool in isolation, but from combining the right processing approach with the right orchestration model.

***

All content in this blog is created exclusively by technical experts specializing in Data Consulting, Data Insight, Data Engineering, and Data Science. Our aim is purely educational, providing valuable insights without marketing intent.