Ritchie Vink, Polars — Interview with Alexy

I'm Alexi Kraov, the organizer of Rust I'm Alexi Kraov, the organizer of Rust AI community. We just held our first AI community. We just held our first meet up in Amsterdam before the PI data meet up in Amsterdam before the PI data Amsterdam Italian office and we have Amsterdam Italian office and we have with us Richie the founder of Polers. with us Richie the founder of Polers. Yes. Yes. And you guys chose Rust for the And you guys chose Rust for the implementation of Polers. So tell us why implementation of Polers. So tell us why you selected Rust? How does it help to you selected Rust? How does it help to build the AI infra structure of build the AI infra structure of tomorrow? tomorrow? Well, when I started a polar, it was Well, when I started a polar, it was mostly because I was interested in ROS. mostly because I was interested in ROS. I was just very excited about the tech, I was just very excited about the tech, about going into more low-level about going into more low-level languages, uh, but also in a modern languages, uh, but also in a modern language. Uh, that was 6 years ago. language. Uh, that was 6 years ago. Nowadays, it's also proven to be a very Nowadays, it's also proven to be a very good language for AI. And, um, yeah, good language for AI. And, um, yeah, that's mostly a lucky coincidence, I that's mostly a lucky coincidence, I would say. would say. uh for a lot of people rusts were seen uh for a lot of people rusts were seen as the the complicated language the the as the the complicated language the the hard language and now this whole hard language and now this whole strictness this whole borrow checker is strictness this whole borrow checker is actually very powerful actually very powerful at compile time you can prove uh a lot at compile time you can prove uh a lot of invariance uh that your thread safe of invariance uh that your thread safe that your uh memor is safe um so you can that your uh memor is safe um so you can yeah you can uh let an agent code with yeah you can uh let an agent code with much more confidence much more confidence so basically now people do not really so basically now people do not really have to fully understand what's going on have to fully understand what's going on because if you have a general idea for a because if you have a general idea for a good engineer and you can unleash your good engineer and you can unleash your agent on Rust, agent on Rust, yeah, yeah, it will be better than equivalent blob it will be better than equivalent blob of Python. of Python. Yeah, in Rust I found it to be a lot Yeah, in Rust I found it to be a lot better than it is in Python. The the better than it is in Python. The the agent has much more information uh given agent has much more information uh given back from the compiler. I would still back from the compiler. I would still say you need to be a good engineer to say you need to be a good engineer to get them the best out of it and getting get them the best out of it and getting in the loop is still very important in the loop is still very important in my opinion. But uh yeah it was a a in my opinion. But uh yeah it was a a lucky shot in that sense. lucky shot in that sense. So uh polars right? So basically like So uh polars right? So basically like there is pandas there is polars there there is pandas there is polars there are data frames are data frames in spark. How did you come up with the in spark. How did you come up with the idea of polars? What makes it special? idea of polars? What makes it special? Uh you have a growing community Uh you have a growing community like what's what's special about polars like what's what's special about polars which makes it a good choice for data which makes it a good choice for data frames? frames? Well when I started polars you you only Well when I started polars you you only had pondos it's a def facto dataf frame had pondos it's a def facto dataf frame library. Um and pandas was very uh library. Um and pandas was very uh powerful in a sense with a very rich powerful in a sense with a very rich API. You could do a lot of stuff but it API. You could do a lot of stuff but it was not very predictable in the API. was not very predictable in the API. Um everything was quite local and API Um everything was quite local and API could could there was no consistent data model but there was no consistent data model but that never made a lot of sense to me. that never made a lot of sense to me. But also pandas But also pandas is in the business of of data is in the business of of data processing. However, processing. However, it didn't look as much to databases or it didn't look as much to databases or query engines in general. It didn't make query engines in general. It didn't make a query plan, didn't make didn't do any a query plan, didn't make didn't do any query optimization. Uh it utilized NumPy query optimization. Uh it utilized NumPy or Python itself for its data uh or Python itself for its data uh storage. Uh whereas there's a my there storage. Uh whereas there's a my there were sort of uh decades of of research were sort of uh decades of of research uh within databases that was sort of uh within databases that was sort of ignored. Uh I think it was Dundas was ignored. Uh I think it was Dundas was mostly uh originated. It feels like it mostly uh originated. It feels like it just came to be and whatever was out just came to be and whatever was out there was used but it wasn't uh there was used but it wasn't uh connected to database research and when connected to database research and when I started writing I wanted to be to be I started writing I wanted to be to be multi-threaded I wanted to be lazy and I multi-threaded I wanted to be lazy and I wanted to have very optimizer wanted to have very optimizer um and the more I learned about it the um and the more I learned about it the more I felt like this should exist for a more I felt like this should exist for a single node single node I want to get maximum performance out of I want to get maximum performance out of my laptop my laptop because pend was kind of the design way because pend was kind of the design way to slice and dice tables right so it to slice and dice tables right so it came from the need of data scientists came from the need of data scientists not uh from the background of database not uh from the background of database engineers who properly can populate the engineers who properly can populate the tables. Yeah, but there's a in the vend tables. Yeah, but there's a in the vend diagram you need both. Also, a data diagram you need both. Also, a data scientist was waiting for it was pretty scientist was waiting for it was pretty common to be waiting for a join in common to be waiting for a join in pandas and you you just saw one core in pandas and you you just saw one core in your your uh system monitor uh chugging your your uh system monitor uh chugging away for 10 minutes. away for 10 minutes. Why not use all the cores? Why not use all the cores? Right. Right. Right. Right. Right. Right. I'm very curious that you said about not I'm very curious that you said about not predictable because I've been you know predictable because I've been you know early organizer of scala communities and early organizer of scala communities and spark. So spark is effectively in my spark. So spark is effectively in my view and marski had a talk at my view and marski had a talk at my conference scale by way that basically conference scale by way that basically spark is distributed scola. So scola spark is distributed scola. So scola collection was the thing which gave collection was the thing which gave birth to this modern data processing birth to this modern data processing where you treat everything as a where you treat everything as a collection and so the the winning uh of collection and so the the winning uh of spark it you know had this moment spark it you know had this moment because everything you would think about because everything you would think about in the API map filter reduce you know in the API map filter reduce you know it's object-oriented so something map it's object-oriented so something map would map something filter would filter would map something filter would filter so do you mean predictability in that so do you mean predictability in that sense like you think of the method it sense like you think of the method it should exist on data frame and it should should exist on data frame and it should be there with a proper name be there with a proper name yes but um because we make a logical yes but um because we make a logical plan. Um we had to come up with a data plan. Um we had to come up with a data model and it should like relational model and it should like relational model. The relational data is a logical model. The relational data is a logical uh if you if you have an API and you uh if you if you have an API and you make a method on top of a data make a method on top of a data container, container, but that method but that method can just do local implementations and can just do local implementations and doesn't go through a a a doesn't go through a a a engine. engine. It's very hard to make that consistently It's very hard to make that consistently behaving. But if it is forced to go behaving. But if it is forced to go through the engine through a virtual through the engine through a virtual machine then you have to come up with a machine then you have to come up with a data model. You have to come up with data model. You have to come up with something consistent otherwise something consistent otherwise right right it will not work. it will not work. So you so you mean predictability in So you so you mean predictability in terms of consistent performance terms of consistent performance consistent performance but also consistent performance but also consistent behavior consistent types. consistent behavior consistent types. Yeah. Yeah. Um what Um what um pandas was non strict. So it used to um pandas was non strict. So it used to be that if you uh parsed a be that if you uh parsed a an example it had a column of strings an example it had a column of strings and it had to parse a date column out of and it had to parse a date column out of that you had to parse that as a date that you had to parse that as a date column. column. What it used to do was run Python What it used to do was run Python daytime parsing on every element daytime parsing on every element sequentially. sequentially. Uhhuh. Uhhuh. But if the first one was date month But if the first one was date month years and the other was years uh month years and the other was years uh month day. day. Yeah. It just swapped them around and Yeah. It just swapped them around and you had you had or the first one was milliseconds, the or the first one was milliseconds, the second was seconds. You had not a un you second was seconds. You had not a un you did not have a unified did not have a unified data type on a column. data type on a column. Mhm. Mhm. Uh you would have different data types Uh you would have different data types within. within. So you didn't really type the column So you didn't really type the column properly. properly. No, it was not consistent in its type No, it was not consistent in its type model model because Python is not really a type because Python is not really a type language and so people don't think in language and so people don't think in terms of types. terms of types. Yes. But it could have been in pandas. Yes. But it could have been in pandas. That's right. That's right. That's right. That's right. I mean Python is just the host language. I mean Python is just the host language. Correct. You can make anything out. Correct. You can make anything out. Correct. But because it comes from that Correct. But because it comes from that background, people who did this did not background, people who did this did not really think of imposing a column type. really think of imposing a column type. Yeah. And um my experience was also that Yeah. And um my experience was also that my types could change in production my types could change in production workloads depending on what kind of data workloads depending on what kind of data went in there. went in there. Okay, Okay, I had the same query, but I had a I had the same query, but I had a rolling window of data. And all of a rolling window of data. And all of a sudden, I was not indexing with a sudden, I was not indexing with a integer, but I was indexing with a integer, but I was indexing with a float. And my production pipeline broke float. And my production pipeline broke uh 40 minutes. I will be terrified as a uh 40 minutes. I will be terrified as a strongly typed person. Yes, strongly typed person. Yes, I will be terrified of this. I will be terrified of this. So these things, yeah, these are paper So these things, yeah, these are paper cuts that hurt me and I wanted that uh cuts that hurt me and I wanted that uh fixed in a data frame library. fixed in a data frame library. Interesting. Interesting. So basically, Interesting. Interesting. So basically, did you start with Rust right away out did you start with Rust right away out of the gate? of the gate? Yeah. Yeah. Okay. Okay. Yeah. The initial goal, so Paul's story Yeah. The initial goal, so Paul's story of just moving goalpost. The initial of just moving goalpost. The initial goal was, oh, I really like this this goal was, oh, I really like this this library or I really like this this library or I really like this this language in Rust. It doesn't have a data language in Rust. It doesn't have a data frame library. Let's make one. Uh I frame library. Let's make one. Uh I worked my first join. Uh worked my first join. Uh did a benchmark with pondas and my join did a benchmark with pondas and my join was super slow. So the the next goal was was super slow. So the the next goal was make it faster than pondas. make it faster than pondas. Mhm. Mhm. Then the next goal was okay make a Then the next goal was okay make a python API. Then initially I wanted to python API. Then initially I wanted to copy the pom API and then I learned copy the pom API and then I learned what what's the consistence behavior? I what what's the consistence behavior? I could not find it. could not find it. Mhm. Uh and then I learned about Mhm. Uh and then I learned about databases and about lazy programming and databases and about lazy programming and then at that point I learned about the then at that point I learned about the optimizer and I thought okay this should optimizer and I thought okay this should exist. exist. Mh. Mh. Uh but it was constantly moving Uh but it was constantly moving goalpost. So the initial goal was just goalpost. So the initial goal was just to across the library. to across the library. Currently I want polars to be the Currently I want polars to be the fastest engine for any scale. I also fastest engine for any scale. I also wanted to be the fastest on distributed. wanted to be the fastest on distributed. Mhm. Mhm. When I started this was never my goal. I When I started this was never my goal. I would have laughed in your face if you would have laughed in your face if you said we would be doing this. So um said we would be doing this. So um yeah. you are basically growing with the yeah. you are basically growing with the community and we're here at the community and we're here at the community conference. I'm super curious. community conference. I'm super curious. It's PI data. It's Python. We're in the It's PI data. It's Python. We're in the same boat at lake sale. We have you know same boat at lake sale. We have you know rust engine but we uh most of the rust engine but we uh most of the customers talk with through pi spark customers talk with through pi spark because support spark connect right. So because support spark connect right. So this is kind of so we say pip install pi this is kind of so we say pip install pi sale. So how do you interact with the sale. So how do you interact with the python ecosystem? uh and like I think python ecosystem? uh and like I think now a lot of folks you know in in Bay now a lot of folks you know in in Bay Area where you know I'm from basically Area where you know I'm from basically solentic and so like they all agents are solentic and so like they all agents are in Python. How how do you see uh polar in Python. How how do you see uh polar kind of being in this Python ecosystem kind of being in this Python ecosystem and specifically in the ecosystem which and specifically in the ecosystem which is all Python now? is all Python now? Yeah. Yeah. So Polar is a Python first Yeah. Yeah. So Polar is a Python first library. We focus mostly on our Python library. We focus mostly on our Python API. It's a Python data frame API. API. It's a Python data frame API. Um so in that sense we're good. uh we're Um so in that sense we're good. uh we're in the corpus of all the big the big in the corpus of all the big the big models. So models know how to write models. So models know how to write polars. polars. Uh a benefit is that at before we run Uh a benefit is that at before we run the query you can also do we can already the query you can also do we can already do a lot of type check. So we do a lot of type check. So we so you write a query you the agent gets so you write a query you the agent gets feedback on the types or errors before feedback on the types or errors before it runs a query. So the wrong query will it runs a query. So the wrong query will not go and reach the not go and reach the Yeah, there are still of course some Yeah, there are still of course some runtime checks still but we try to catch runtime checks still but we try to catch as many errors as possible up front. as many errors as possible up front. Um so that's really beneficial compared Um so that's really beneficial compared to um to um uh compared to something like pandas. Um uh compared to something like pandas. Um we're also focusing more on SQL now we're also focusing more on SQL now because agents took that friction away. because agents took that friction away. Humans Humans I mean humans don't like to write very I mean humans don't like to write very complicated SQL strings on typed an complicated SQL strings on typed an agent doesn't care. agent doesn't care. So um yeah the world is changing so uh So um yeah the world is changing so uh looking into that as well but uh I think looking into that as well but uh I think there's still merit in in a API there's still merit in in a API especially with regard to composability. especially with regard to composability. You can pass expressions to functions You can pass expressions to functions let them generate stuff. So the meta let them generate stuff. So the meta programming you can do in polars is very programming you can do in polars is very powerful powerful and it's all typed. Yeah, and it's all typed. Yeah, very cool. And so the last question over very cool. And so the last question over here at you know it's your hometown here at you know it's your hometown Amsterdam right where Polaro started Amsterdam right where Polaro started this is one of my favorite conferences this is one of my favorite conferences PI data I'm coming like year on year uh PI data I'm coming like year on year uh what's special about Amsterdam data what's special about Amsterdam data community like what's kind of you know community like what's kind of you know makes this conference you know good for makes this conference you know good for you and page in general you and page in general well it's a it's a home game of course well it's a it's a home game of course so that's very easy so that's very easy I think Amsterdam is great because of I think Amsterdam is great because of all the the international And it's a all the the international And it's a very it's one of the largest pas I've very it's one of the largest pas I've been to. I've been been to. I've been to multiple padas across the globe. But to multiple padas across the globe. But um um yeah, it's always a big a big yeah, it's always a big a big It's a big one and one of my favorites It's a big one and one of my favorites too. too. Yeah. Yeah. And uh just a lot of uh I Yeah. Yeah. And uh just a lot of uh I think different think different uh business areas, different different uh business areas, different different fields. So fields. So not all agents like they're actual not all agents like they're actual businesses using data for something, businesses using data for something, right? which is right? which is and uh them uh in the Amsterdam is a and uh them uh in the Amsterdam is a great city to uh to have fun. So uh great city to uh to have fun. So uh one of the best one of the best and is one of the best one of the best and is there like a lot of customers local there like a lot of customers local Netherlands customers who you see Netherlands customers who you see bowlers? bowlers? Uh definitely definitely a bit less than Uh definitely definitely a bit less than in the US. They our market adoption is in the US. They our market adoption is biggest in the US. I think the US is biggest in the US. I think the US is just just Europe is always a bit slower in Europe is always a bit slower in adoption. they look at okay what's adoption. they look at okay what's what's big in the US and then 10 years what's big in the US and then 10 years later it sort of later it sort of moves over moves over but we have a lot of photo users uh but we have a lot of photo users uh uh in Europe as well um nowadays luckily uh in Europe as well um nowadays luckily in the beginning it was always uh we in the beginning it was always uh we were much more uh well known in US were much more uh well known in US currently it's uh currently it's uh uh uh we're also big in Europe but we're also big in Europe but yeah yeah it's spread in the world yeah yeah it's spread in the world and you're one of the few remaining and you're one of the few remaining European companies doing European companies doing Yeah. Yeah. Yeah. Um Yeah. So, um I Yeah. Yeah. Yeah. Um Yeah. So, um I think after the think after the the DB acquisition, now we're uh the DB acquisition, now we're uh yeah, we're the European tech company. yeah, we're the European tech company. Yeah. The independent Amsterdam Yeah. The independent Amsterdam different company. different company. All right. Well, thank you so much. All right. Well, thank you so much. We're looking forward to hosting you at We're looking forward to hosting you at the Rust AI events in Bay Area. Cool. the Rust AI events in Bay Area. Cool. And I think like, you know, we'll have, And I think like, you know, we'll have, you know, like a system of people doing you know, like a system of people doing basically what they call AI infra 2.0 Z basically what they call AI infra 2.0 Z where we power the fast, you know, AI of where we power the fast, you know, AI of tomorrow with all this wonderful Rust tomorrow with all this wonderful Rust engineering. engineering. Yeah. Yeah. It would be great to uh to Yeah. Yeah. It would be great to uh to be there and and be in the Bay Area. I be there and and be in the Bay Area. I always like to have this whole um always like to have this whole um vibrant culture. vibrant culture. Looking forward. Cheers. Thank you. Looking forward. Cheers. Thank you. Bye, guys. All right.