Today · blog, October 23

Agents make 70% of Databricks SDP pipelines; base is unclear

Databricks self-reports that agents created more than 70% of the Spark Declarative Pipelines data pipelines on its platform. Spark Declarative Pipelines is Databricks' way of writing data pipelines: the author declares which tables are needed and which query produces each one, and Spark decides the execution order. The post reads: "over 70% of SDP pipelines on Databricks are now created by agents" (Databricks, 2026-10-23). The post doesn't define the denominator of the 70% or what "created by agents" means. Databricks had the same agent write pipelines in both declarative and procedural styles. The declarative code was 51% shorter than the procedural code. Databricks writes that a dry run catches only structural errors; logic errors still depend on testing and review. Our read: by Databricks' own account, a dry run won't catch an agent's logic errors. Teams adopting agent-written pipelines still need people to test and review what the pipelines compute. How we checked →

Concept illustration: on the right, a wooden easel holds a large white board with a red stacked-layer icon and the lettering "DATABRICKS" and "SDP PIPELINES", and below them a horizontal bar filled mostly in red with a small speech-bubble icon at its left end. On the left, on a wooden table, a hand reaching in from the left edge holds a magnifying glass over an open laptop. The laptop screen shows a row of blue squares joined by a line, with a grey box above and an arrow curving down from it to the last square, and a folded card lettered "AGENT" stands beside the laptop.

Source and checks

This item ran in the brief of Oct 25, 2026, alongside 8 other items. How we checked it, the quote checks and what was still missing that day are on that issue's How we checked page.

Cite this item

SecondSource (2026). "Agents make 70% of Databricks SDP pipelines; base is unclear." https://secondsource.io/en/product/2026-10-25-1/. Accessed access date.