DataFusion at LangChain: the Rust data engine moves into the agent stack

On August 13, 2026 the San Francisco Apache DataFusion Meetup met at LangChain's new office in SOMA. The premise, from the organizers' invitation, was an evening of talks about DataFusion "and its growing role in modern data infrastructure," with a spotlight on real-world applications, performance work, and integrations. What the evening actually showed was narrower and more interesting: the same Rust query engine now sits underneath agent observability, database provisioning for agents, a Spark replacement, and the connectivity layer that ties them together.

Five talks, four companion interviews, and a walk through the office before doors opened are all on FunctionalTV. The graph of who spoke, what they built, and how it connects lives on devreal.ai.

The arc of the evening

The agenda ran from the engine outward. It opened inside DataFusion itself, with adaptive query execution in the distributed runtime, then moved to a full Spark-compatible engine built on it, then to two products that ship it to customers, and closed with the connectivity standard that lets any of them talk to any other without copying rows.

• Alexander Bianchi (Datadog) opened with Adaptive Query Execution in DataFusion Distributed: the distributed runtime learning from partial results and re-planning as it goes.
• Shehab Amin (LakeSail) presented Inside Sail: Rebuilding the Spark Lakehouse Ecosystem in Rust, a Spark Connect-compatible engine with in-process Python, native Iceberg and Delta Lake, and unified batch, streaming, and AI workloads, all without the JVM.
• Ankush Gola (LangChain) presented Building SmithDB: The Database for Agent Observability, the DataFusion-powered database behind LangSmith's trace workloads: object storage, Vortex files, demanding trace-query patterns, and the production gains they delivered.
• Divya Ranganathan and Zac Farrell (Hotdata) presented Provisioning Millions of Databases with DataFusion and DuckLake, on batching and a Rust-native architecture that provisions hundreds of thousands of isolated databases quickly, on the way to a million in under a minute.
• Emil Sadek (Columnar) closed with Arrow End-to-End: Integrating DataFusion and ADBC, showing how ADBC drivers, table providers, and Flight SQL move Arrow data from server to client with no row-oriented detour.

Read in order, the five talks are one argument. DataFusion is fast enough to be the engine. Sail makes it Spark-shaped so existing code runs on it. SmithDB and Hotdata prove it in production for agent workloads. ADBC is how the pieces exchange data without paying the transpose tax.

The conversations

Between talks, Alexy Khrabrov sat down with three of the speakers for Structured Output, the struct.fm podcast.

• Ankush Gola, LangChain — Interview with Alexy: SmithDB, database extensibility, production performance, and where Rust fits in AI infrastructure. (episode)
• Romain Priour, AI Infra at LangChain — Interview with Alexy: deploying LangSmith for enterprise customers, Kubernetes, and a hybrid SaaS and self-hosted model on AWS. (episode)
• Zac Farrell, Hotdata — Interview with Alexy: building cloud databases for agents on Rust and DataFusion, isolated workspaces, and choosing Rust-native infrastructure over legacy stacks. (episode)

Before the doors opened, DataFusion at LangChain: Meet the Speakers walks through the office and checks in with the organizers and speakers previewing their talks.

Where it lives

Every talk, speaker, company, and project from the evening is a node in the developer community graph:

• Event: San Francisco Apache DataFusion Meetup, August 13, 2026
• Community: Apache DataFusion Meetup on Luma

The organizers said they have room for speakers at the December meetup. If you are building on DataFusion, that is the place to show it.