Spark NYC at Datadog: where Spark performance is headed

The Apache Spark NYC Meetup met on August 26, 2026 at Datadog's New York office. The organizers framed it as "an evening of practical technical sessions, real-world lessons from the field, and open discussions around the future of data infrastructure," for data engineers, platform teams, and, in their words, "anyone who's spent too many hours staring at the Spark UI." Two of the three talks were about where Spark performance is headed: one on modernizing what you already run, one on what's coming next in Spark itself.

The arc of the evening

The agenda moved from Spark's own roadmap, to running Spark at scale with agents watching over it, to replacing the runtime under Spark without touching the code above it.

• Meni Shmueli (DataFlint) opened with What's new in Spark 4.2, a tour of the next release from a company that builds Spark observability.
• Yarden Wolf (Wix) presented 850 Engineering Hours Back, Every Month: Context-Aware Agentic Platform for Airflow & Spark at Wix, on the internal agents that triage production alerts and root causes for a data platform serving hundreds of engineers, data scientists, and analysts.
• Shehab Amin (LakeSail) closed with Spark Rebuilt in Rust: Making the Data Engine AI-Native, presented at the meetup as Modernizing Spark: Performance Boost Without Rewrite. Sail keeps the Spark SQL and DataFrame interfaces and replaces the JVM runtime with an Arrow-native engine on Apache DataFusion: Python UDFs in-process over shared Arrow memory, an actor-based control plane with stateless workers, pipelined shuffle, native Delta Lake, Iceberg, and DuckLake, and SQL-based observability through a system catalog.

Together the three talks bracket the question the meetup was built around. You can wait for Spark 4.2, you can wrap the Spark you have in agents that read its logs for you, or you can keep the API and swap the engine.

The conversations

Alexy Khrabrov recorded two Structured Output interviews on location.

• Yarden Wolf, Wix — Interview with Alexy: how agentic workflows change the work of Wix's engineers, data scientists, and analysts while a human stays in the loop for production alerts; Base and Wix Harmony, which pairs vibe coding with drag-and-drop editing; authentication and AI gateways for internal agents; specialized agents like Airbot for Airflow and a root-cause-analysis agent working together; and Wix Headless, which lets a coding agent build a site's backend on Wix. (episode)
• Adam de Delva, DTR — Interview with Alexy: growing a network of open-source communities from 30,000 to over three million members, with new communities in Ghana and Latin America; open-source sustainability and getting companies to reinvest in the commons; digital public infrastructure and sovereign AI, from mapping workloads onto neoclouds to trustless container scheduling with the Linux Foundation Decentralized Trust and Hyperledger communities; and the UOR Foundation, where the specs and protocols for sovereign AI infrastructure are built. Recorded on a New York rooftop after the meetup. (episode)

Where it lives

• Event: Apache Spark NYC Meetup, August 26, 2026
• Community: Spark NYC on Luma, hosted by Geffen Fridman, Daniel Aronovich, Yuval Nash, and Datadog

Thanks to Datadog for the space, the pizza, and the view.