Matt Topol is trying to kill ODBC

"I'm trying to kill ODBC," Matt Topol said, when I asked him at PyData Amsterdam to explain ADBC in one line. It is a good line because it is literally the plan.

Matt is a co-founder of Columnar, a PMC member of Apache Arrow and Apache Iceberg, and, as of this year, on the initial PMC of Apache Magpie. He is the person who shows up wherever a database meets a client and points out that the two have been lying to each other about the shape of the data for thirty years. Our Structured Output conversation was recorded on September 11 on the floor of the NDSM Loods, the day after his talk on ADBC and DuckDB and a few weeks before he took the same argument to dbt Summit.

The transpose tax

Here is the problem as Matt states it. Nearly every analytical system is column-oriented. Nearly every data-frame library is column-oriented. But the connection between them, ODBC or JDBC, is row-oriented. So the database transposes columns into rows to send them, and pandas or Polars transposes them back into columns to use them. You pay the copy twice, on every query, for the privilege of a 1990s interface.

ADBC is a single client API with drivers behind it, exactly like ODBC, with one difference: data crosses the interface as Apache Arrow. If the source returns columns, they stay columns end to end. If the source is Postgres and returns rows, your application still gets a consistent columnar result, and you never own the type mapping. It is simpler and faster, which is rare.

The interface is part of the Arrow project, Apache-governed and open. Most of the drivers are open too.

dbc: uv for drivers

The part of the story that made the room at PyData laugh was about installing drivers. Anyone who has hunted a vendor site for the right ODBC package and then guessed where to put it knows the feeling. Columnar's answer is dbc, a package manager for ADBC drivers, "uv for drivers" in Matt's phrase:

`` dbc install bigquery dbc install flightsql dbc install spark ``

Binaries are built per platform, published to a CDN, signed by Columnar, and verified on install. A contributor is adding a platform flag so you can pull the driver for a Docker image's target rather than your laptop's.

DuckDB, dbt, and the rest of the ecosystem

Matt's PyData talk was about ADBC and DuckDB. Columnar contributed a community extension so DuckDB can attach to any ADBC driver: INSTALL adbc FROM community, then send a query to a remote system and pull the result set in as Arrow, zero-copy, and mix it with local data. The analytical part, the SQL, was always the easy part. Getting the data to where the SQL runs is the hard part, and this is the efficient way to do it.

Then there is dbt. At dbt Summit Matt spoke about dbt Core v2 building its database adapters on ADBC, and dbt Fusion using ADBC as well. When the most widely used transformation tool standardizes on a connectivity layer, the vendors follow. Which is Matt's other campaign: more systems should return Arrow, and more should accept Arrow. Today several major warehouses will hand you a whole table as Arrow but have no native way to ingest an Arrow stream, so bulk load means writing Parquet to object storage and running a copy. Agents produce data as well as consume it. That gap will close.

The ADBC Spark driver, and Sail everywhere

The detail that matters most to us at LakeSail is small. dbc install spark gives you the ADBC Spark driver. Point it at a Sail instance and add api=connect to the connection URI, and you are talking Spark Connect. Spark Connect is an Arrow protocol; Sail is built on DataFusion, which is Arrow-native. So the path from Sail to DuckDB, to Polars, to dbt, to Power BI, to anything with an ADBC driver manager, is Arrow the whole way, with no serialization and no transpose.

That is what "Sail everywhere" means in practice. We did not have to write a connector for each tool. We had to speak Spark Connect, and ADBC did the rest.

Magpie, and using AI without lying about it

We closed on Apache Magpie, which grew out of the Airflow maintainers' own tooling: a set of skills that help maintainers triage issues and pull requests, draft reviews, and mentor new contributors, with AI. Two rules stood out. Magpie asks before it posts anything to GitHub, and anything it posts is labeled as AI-assisted and signed off by a human. Matt's workflow on arrow-go and iceberg-go is to let Magpie sweep the open PRs, then review each of its reviews himself before deciding.

His position on AI in open source is the pragmatic one. Projects that ban AI contributions outright will lose contributors they need, and open source never has enough. The first time someone engages with your project may be with an LLM's help. The question is whether they will stay and learn, not what tool they typed with. You are responsible for what goes out under your name, and, as he put it, the AI is going to do stupid things anyway, so you had better know enough to steer it.

The next thing, we agreed, is a binary protocol for agent-to-agent data. MCP is JSON, which is why it cannot move much of it. Someone should build ADBC for agents. Matt said he would, if he had the time. Consider that a call for contributors.

• Interview: Matt Topol, Columnar, on ADBC — Interview with Alexy (episode and transcript)
• Related: Emil Sadek of Columnar on Arrow End-to-End: Integrating DataFusion and ADBC at the DataFusion meetup at LangChain
• On devreal.ai: Matt Topol, Columnar, ADBC, Apache Arrow