SF Scala: Manasi Vartak Interview

[Music] hello everybody my name is Alexa crabber hello everybody my name is Alexa crabber from the founder and organizer of the from the founder and organizer of the era AI and we have a joint meet up era AI and we have a joint meet up tonight at D 2aq a well-known company tonight at D 2aq a well-known company basically doing data orchestration and I basically doing data orchestration and I think it's very fitting we have a jury think it's very fitting we have a jury made up of multiple meetups and the made up of multiple meetups and the topic of this meetup is machine learning topic of this meetup is machine learning models and specifically versioning and models and specifically versioning and monitoring we have two talks and the monitoring we have two talks and the monitoring is with hundreds upon monitoring is with hundreds upon pushkaryova and the versioning is with pushkaryova and the versioning is with mono severe talk who is the CEO of Verta mono severe talk who is the CEO of Verta III recent startup III recent startup so welcome mama C thank you thank you so welcome mama C thank you thank you for having me tell us a little bit about for having me tell us a little bit about yourself why did you decide to start yourself why did you decide to start Verta and what this is all about Verta and what this is all about absolutely so I am a technical founder I absolutely so I am a technical founder I did my PhD at MIT worked on data systems did my PhD at MIT worked on data systems all throughout grad school so one of the all throughout grad school so one of the systems I built was modeled TB we systems I built was modeled TB we started looking at how data scientists started looking at how data scientists work and what were challenges and this work and what were challenges and this was around the time one Alex net and all was around the time one Alex net and all of the deep learning models were really of the deep learning models were really becoming significant I realized that the becoming significant I realized that the things that we take granted for software things that we take granted for software I like get type versioning systems I like get type versioning systems deployment debugging were all missing deployment debugging were all missing for models and so we ended up building for models and so we ended up building model DB which became a popular open model DB which became a popular open source project and then further evolved source project and then further evolved from that nice and so I'm curious so you from that nice and so I'm curious so you were a researcher to me right was very were a researcher to me right was very soft regime before that how the specific soft regime before that how the specific topic came up yeah so I did I worked at topic came up yeah so I did I worked at Microsoft between undergrad and grad Microsoft between undergrad and grad school um my PhD was also in the school um my PhD was also in the database group from which came post database group from which came post grass and verdict also like we built grass and verdict also like we built systems and so it just we write software systems and so it just we write software all day it was a clear and natural all day it was a clear and natural direction direction nice but obviously machine learning nice but obviously machine learning model also very different beasts do you model also very different beasts do you have specific issues in machine learning have specific issues in machine learning how did this come about yeah absolutely how did this come about yeah absolutely so my interests are between data systems so my interests are between data systems nml my under nml my under with CS and math so I also write with CS and math so I also write algorithms and things like that I used algorithms and things like that I used to be part of the feed ranking team at to be part of the feed ranking team at Twitter also worked on dynamic ad Twitter also worked on dynamic ad targeting at Google so that was applied targeting at Google so that was applied ml and that gave a good balance to the ml and that gave a good balance to the systems building and advert up my CTO systems building and advert up my CTO actually comes from Twitter and NVIDIA actually comes from Twitter and NVIDIA we both work together Twitter is like it we both work together Twitter is like it also brings a lot of significant chops also brings a lot of significant chops in the production of asian of models in the production of asian of models very cool very cool so the you can have very cool very cool so the you can have brilliant this two worlds right song I'm brilliant this two worlds right song I'm know I'm curious right because you're know I'm curious right because you're clearly in the camp of software clearly in the camp of software engineers which most of our meetup engineers which most of our meetup members are right so we kind of so the members are right so we kind of so the the theory which kind of most I see the theory which kind of most I see working in in the area if you have working in in the area if you have working company you know machine working company you know machine learning is a commodity you take it off learning is a commodity you take it off itself you're not you know working Geoff itself you're not you know working Geoff Kimpton next deploring iteration you are Kimpton next deploring iteration you are a grunt deploying models making sure a grunt deploying models making sure that make money or don't lose money and that make money or don't lose money and and your job depends on them being you and your job depends on them being you know in production right and so like know in production right and so like versioning is beautiful it sounds like versioning is beautiful it sounds like music to our ears because if something music to our ears because if something bad happens you need to find out and so bad happens you need to find out and so can we understand get and and continuous can we understand get and and continuous integration like so but but a lot of integration like so but but a lot of data science development is done by data science development is done by folks who do Python and do it in a folks who do Python and do it in a notebook they're not in production so notebook they're not in production so how do you address those guys like how how do you address those guys like how do you interact with engineers versus do you interact with engineers versus the designers versus business owners the designers versus business owners right I will see all these metrics in right I will see all these metrics in the hundred person um so for us the the hundred person um so for us the fundamentals of our product don't change fundamentals of our product don't change for us a model version consists of as I for us a model version consists of as I will talk about the code the data will talk about the code the data configured environment together now if configured environment together now if I'm a data scientist I am in a Jupiter I'm a data scientist I am in a Jupiter notebook or our studio and the way that notebook or our studio and the way that they access our product is via very easy they access our product is via very easy to use api's or gooeys mm-hmm now if to use api's or gooeys mm-hmm now if you're a software engineer and or you're you're a software engineer and or you're a DevOps person then you care about your a DevOps person then you care about your SaaS throughput how's your model SaaS throughput how's your model behaving then you can access it directly behaving then you can access it directly through kubernetes sort of coop control through kubernetes sort of coop control use the yeah no files integrated into use the yeah no files integrated into Jenkins do the software route and then Jenkins do the software route and then if you're a business owner then you're if you're a business owner then you're only looking at our part of the product only looking at our part of the product so there's the same fundamental so there's the same fundamental very similar to say github if you only very similar to say github if you only look at the UI that maybe a business look at the UI that maybe a business owner or a project manager can owner or a project manager can understand it versus if you're a understand it versus if you're a software developer you have the command software developer you have the command line name you're doing merges and pulls line name you're doing merges and pulls all the time all the time so here's kind of an EVE scenario right so here's kind of an EVE scenario right so I'm a business owner and I run an so I'm a business owner and I run an online shop right now my data scientists online shop right now my data scientists are kind of doing the upselling are kind of doing the upselling customers on shopping carts right or customers on shopping carts right or they're basically like I'm watching it they're basically like I'm watching it they're coming back and so you know the they're coming back and so you know the the model basically does it and so I the model basically does it and so I have a metric how much do they upsell have a metric how much do they upsell each checkout and and and so so can you each checkout and and and so so can you kind of can we is the field this is kind of can we is the field this is necessary reach the point where I am the necessary reach the point where I am the shop owner I have no idea what that shop owner I have no idea what that assigns as the programmers are doing I assigns as the programmers are doing I wake up in the middle of the night I wake up in the middle of the night I wish how much every checkout is wish how much every checkout is upsetting and suddenly because the upsetting and suddenly because the previous night some somebody messed up previous night some somebody messed up and the models really bad and like I'm and the models really bad and like I'm something really plunged can I push a something really plunged can I push a button to rollback like is it at the button to rollback like is it at the level where you can give this tool to a level where you can give this tool to a business owner who which is a clear business owner who which is a clear business metric and he will be alerted business metric and he will be alerted or is it the job of an engineer I mean or is it the job of an engineer I mean are we gonna involve somebody who are we gonna involve somebody who understands the business metric should understands the business metric should it be a pet religion level thing for an it be a pet religion level thing for an engineer or can you let like reach the engineer or can you let like reach the point where a business owner will be point where a business owner will be able to to say no this doesn't work able to to say no this doesn't work because you know he cares he wakes up because you know he cares he wakes up and he does a rollback is it possible I and he does a rollback is it possible I would say that in this sense our models would say that in this sense our models are not different from existing software are not different from existing software if I'm a product owner and say it's my if I'm a product owner and say it's my mobile app and I see something wrong mobile app and I see something wrong with it with it I don't go to our dashboard and roll it I don't go to our dashboard and roll it back I hurt my engineer to do it yes so back I hurt my engineer to do it yes so I don't think ml models for any I don't think ml models for any different we would escalate it via page different we would escalate it via page or duty they can provide input into why or duty they can provide input into why they think a metric is off but we don't they think a metric is off but we don't think they're the right people to roll think they're the right people to roll back a production system okay all right back a production system okay all right cool and so I'm kind of very curious cool and so I'm kind of very curious about this specialization about this specialization right so not only we get a bunch of em right so not only we get a bunch of em elopes startups dealing with other BOTS elopes startups dealing with other BOTS now have you know monitoring versioning now have you know monitoring versioning and tuning as specialties of different and tuning as specialties of different companies so do you think and obviously companies so do you think and obviously if you if we go this route and if you if we go this route and standardization standardization when you can really agree what these when you can really agree what these models are yeah because right like if we models are yeah because right like if we really want be like do you think it's really want be like do you think it's going that way that you know we have or going that way that you know we have or like language used to describe it and like language used to describe it and models are we gonna go this route of models are we gonna go this route of kind of thousand flowers or isn't gonna kind of thousand flowers or isn't gonna be polite platforms which their own be polite platforms which their own clothes tools because like for if the a clothes tools because like for if the a standardization and open market them standardization and open market them it's really good for star laws if it's it's really good for star laws if it's all kind of inside things and then say all kind of inside things and then say sure Amazon and Google like what do you sure Amazon and Google like what do you think this is going I'm going to see my think this is going I'm going to see my specialization or not I would say we're specialization or not I would say we're going towards standardization and we going towards standardization and we have been involved in some of these have been involved in some of these efforts through model DB through verda efforts through model DB through verda where we're trying to find what does a where we're trying to find what does a model need to have with it what kind of model need to have with it what kind of descriptors doesn't need to have how descriptors doesn't need to have how does a model package get defined and does a model package get defined and things like that so I don't think just things like that so I don't think just as in software we now have reached a as in software we now have reached a conclusion on what a Jenkins file should conclusion on what a Jenkins file should look like so we are going to approach look like so we are going to approach that stage with models we're just that stage with models we're just getting there slowly and the startups I getting there slowly and the startups I think are providing a lot of impetus for think are providing a lot of impetus for that there's gonna be a bunch of llamó that there's gonna be a bunch of llamó [Laughter] so we can check in yeah that would be so we can check in yeah that would be wonderful wonderful very wonderful yes yes but I think very wonderful yes yes but I think Corinne that he says doesn't really got Corinne that he says doesn't really got a good job in popularizing yamo as well a good job in popularizing yamo as well as having api's and we've pulled some of as having api's and we've pulled some of those designs and sort of ideas from those designs and sort of ideas from there there we seem to be working well so what is we seem to be working well so what is the hardest challenge for you and you the hardest challenge for you and you know left onto the next year were like know left onto the next year were like what's the your current you know what what's the your current you know what keeps you up at night what do you want keeps you up at night what do you want to solve with this kind of coming year to solve with this kind of coming year for Virgil when you're a small start-up for Virgil when you're a small start-up there's lots of challenges as you might there's lots of challenges as you might imagine this move this field moves very imagine this move this field moves very fast as you might imagine and so there's fast as you might imagine and so there's always either a new algorithm you'd have always either a new algorithm you'd have of doing things or there are new of doing things or there are new companies out there so for us it's companies out there so for us it's always finding what is the thing that we always finding what is the thing that we can do especially well compared to other can do especially well compared to other companies and at the end of the day for companies and at the end of the day for us are we making a difference for the us are we making a difference for the customers that we work for if we're customers that we work for if we're moving the needle for them then that's moving the needle for them then that's a huge one for us and that's what we'll a huge one for us and that's what we'll focus on sounds awesome we wish you all focus on sounds awesome we wish you all the luck and they're looking forward to the luck and they're looking forward to my talk thank you very much thank you my talk thank you very much thank you [Music]