DevReal: Michael Goin Interview

hello everybody I'm Alexi krau the hello everybody I'm Alexi krau the founder and organizer of beera AI the founder and organizer of beera AI the longest running deepest technical longest running deepest technical biggest AI metup in the world running in biggest AI metup in the world running in the Bay Area continuously since 2015 the Bay Area continuously since 2015 live on location AWS ji LOF and here live on location AWS ji LOF and here with us we have Michael going who is the with us we have Michael going who is the engineering lead at neural magic uh and engineering lead at neural magic uh and uh they are the maintainers of VM uh uh they are the maintainers of VM uh welcome and please tell us what VM is VM welcome and please tell us what VM is VM is the is the most used open-source is the is the most used open-source inference server uh around the world inference server uh around the world it's uh loved for its you know uh it's uh loved for its you know uh support of many models inference support of many models inference optimizations uh quantization methods uh optimizations uh quantization methods uh and of course constant research that and of course constant research that gets cont contributed into gets cont contributed into it and uh why is it so widely used and it and uh why is it so widely used and why is it so Loft like it has a fiber why is it so Loft like it has a fiber Comm on it yeah I mean we have you know Comm on it yeah I mean we have you know sort of many companies and many academic sort of many companies and many academic institutions that build on to it uh you institutions that build on to it uh you know it's it's a really good thing when know it's it's a really good thing when you see uh you know dozens of papers you see uh you know dozens of papers every year based off of VM or comparing every year based off of VM or comparing to VM uh you know uh against some new to VM uh you know uh against some new method that they came off of uh which is method that they came off of uh which is usually made as a fork a VM with some usually made as a fork a VM with some new idea put in and so this is allowed new idea put in and so this is allowed for a lot of Rapid development and Rapid for a lot of Rapid development and Rapid integration because these you know integration because these you know promising research results can be promising research results can be quickly validated reproduced and quickly validated reproduced and integrated in um benefiting everybody integrated in um benefiting everybody and you're gu based out of Boston uh yes and you're gu based out of Boston uh yes yeah we're originally an MIT startup MIT yeah we're originally an MIT startup MIT start I wanted to ask you this uh so MIT start I wanted to ask you this uh so MIT and Sil Valley do you see any cultural and Sil Valley do you see any cultural differences is being kind of a little differences is being kind of a little bit removed from Sil and well does it bit removed from Sil and well does it help you to focus on help you to focus on science I think uh I think yeah there's science I think uh I think yeah there's definitely a lot less talk about tech uh definitely a lot less talk about tech uh over there and uh I think you know I I over there and uh I think you know I I haven't been over on the west coast too haven't been over on the west coast too much but uh I know from our perspective much but uh I know from our perspective it's made it pretty easy to uh you know it's made it pretty easy to uh you know hire and find people that are very hire and find people that are very grounded in uh high performance grounded in uh high performance Computing and uh you know math software Computing and uh you know math software libraries uh which has been really good libraries uh which has been really good for us as we're really focused on for us as we're really focused on performance um you know these these performance um you know these these models are are mostly Matrix models are are mostly Matrix multiplications so multiplications so uh this is really our Focus as a uh this is really our Focus as a inference performance company is it okay inference performance company is it okay if I reveal the secret of AI to a go if I reveal the secret of AI to a go ahead so oi is just Matrix ahead so oi is just Matrix multiplications all the way down and multiplications all the way down and matrices need numbers so first we can matrices need numbers so first we can basically convert reality to numbers put basically convert reality to numbers put them in matrices and then we run a bunch them in matrices and then we run a bunch of multiplications yeah through tensor of multiplications yeah through tensor course through tens right so this is all course through tens right so this is all you need to know about yeah uh and but you need to know about yeah uh and but this is a complex engineering problem this is a complex engineering problem right so you mentioned right so you mentioned uh some of the challenges what are the uh some of the challenges what are the kind of hardest problems VM kind of hardest problems VM solves actually one of the hardest solves actually one of the hardest problems we have is I would say CPU problems we have is I would say CPU overhead uh particularly because a lot overhead uh particularly because a lot of VM is surprisingly enough written in of VM is surprisingly enough written in Python M even though it's you know a Python M even though it's you know a very performant Library we choose to very performant Library we choose to continue writing the majority of it uh continue writing the majority of it uh the vast majority of it in Python and in the vast majority of it in Python and in pytorch to make it approachable and pytorch to make it approachable and accessible for new people uh of course accessible for new people uh of course this has required us to you know take this has required us to you know take advantage of uh sort of Cutting Edge uh advantage of uh sort of Cutting Edge uh technology you know probably from from technology you know probably from from pytorch you're familiar with torch pytorch you're familiar with torch compile this has been a great uh you compile this has been a great uh you know uh advancement that's allowed us to know uh advancement that's allowed us to sort of get back into traditional graph sort of get back into traditional graph compilation uh and and try to you know compilation uh and and try to you know uh get some larger graphs uh to to uh get some larger graphs uh to to prevent the amount of CPU GPU prevent the amount of CPU GPU communication we have to make but also communication we have to make but also it's required us to make sort of uh you it's required us to make sort of uh you know relatively complex know relatively complex multi-process uh you know python multi-process uh you know python architecture with interprocess architecture with interprocess communication proper communication proper asynchronicity um to keep these you know asynchronicity um to keep these you know uh big very fast gpus fed uh you know uh big very fast gpus fed uh you know this is essentially the goal we just this is essentially the goal we just want to keep these things at 100% want to keep these things at 100% utilization and anything we're not doing utilization and anything we're not doing on the GPU needs to happen while the GPU on the GPU needs to happen while the GPU is running not while the GP is waiting is running not while the GP is waiting uh so you know these are some of the uh so you know these are some of the biggest concerns uh that that that we biggest concerns uh that that that we have to do to keep the project uh have to do to keep the project uh successful as all of the new inference successful as all of the new inference optimizations and methods come in optimizations and methods come in through research and through uh uh you through research and through uh uh you know Enterprise and uh uh I mentioned know Enterprise and uh uh I mentioned that your title is engineering lead and that your title is engineering lead and so uh building V involves a lot of so uh building V involves a lot of engineering can you talk a little bit engineering can you talk a little bit about how this uh discipline of about how this uh discipline of engineering is software engineering uh engineering is software engineering uh knowledge is helping you scale V knowledge is helping you scale V yeah I mean uh I have maybe kind of a yeah I mean uh I have maybe kind of a vague title uh because I I work on a lot vague title uh because I I work on a lot of vague things I think uh to to to be of vague things I think uh to to to be in the in the to be really successful in in the in the to be really successful in the field of performance engineering uh the field of performance engineering uh I I believe you have to go down to the I I believe you have to go down to the assembly level the kernel level uh when assembly level the kernel level uh when you really need to make a really really you really need to make a really really tight Loop uh but then also go all the tight Loop uh but then also go all the way up to the the python the machine way up to the the python the machine learning level uh in order to you know learning level uh in order to you know understand and connect to the understand and connect to the applications that these engines are are applications that these engines are are ending up just being used for uh you ending up just being used for uh you know ultimately the worst thing you can know ultimately the worst thing you can do in performance engineering is get the do in performance engineering is get the wrong Benchmark it's very easy to delude wrong Benchmark it's very easy to delude yourself that you are performant if you yourself that you are performant if you create some micro Benchmark and you're create some micro Benchmark and you're like oh I did a great job I'm at 100% like oh I did a great job I'm at 100% utilization but then you get some real utilization but then you get some real world data and it's uh it's very world data and it's uh it's very heterogeneous you have some very heterogeneous you have some very uh you know long prompts coming in or or uh you know long prompts coming in or or lots of images all at once and uh you lots of images all at once and uh you have maybe uh you know a bunch of small have maybe uh you know a bunch of small ones piling up um you really need to be ones piling up um you really need to be sort of uh antagonistic to yourself and sort of uh antagonistic to yourself and uh play Devil's Advocate to uh you know uh play Devil's Advocate to uh you know understand all of the workloads uh and understand all of the workloads uh and and how you can optimize for them and and how you can optimize for them and also what things are worth optimizing also what things are worth optimizing for and not because you still want to for and not because you still want to keep the software uh maintainable and keep the software uh maintainable and and readable and accessible for new and readable and accessible for new people so uh you know it it really people so uh you know it it really requires I think not being afraid to requires I think not being afraid to understand how computers work uh but understand how computers work uh but also not being afraid of the also not being afraid of the applications that people are building on applications that people are building on top of the computers uh which I think top of the computers uh which I think has been a a problem since computers has been a a problem since computers have come around makes sense this is have come around makes sense this is great to hear and I come from you know great to hear and I come from you know strong typed programming background and strong typed programming background and at first you know I was a bit skeptical at first you know I was a bit skeptical about python succeeding in this space about python succeeding in this space but it's become more performant and what but it's become more performant and what I really like I like types so I like I really like I like types so I like things like pantic which gives a schema things like pantic which gives a schema and I'm super excited about moo Chris and I'm super excited about moo Chris lner language what what are your lner language what what are your thoughts about kind of type Machinery thoughts about kind of type Machinery help it by be more help it by be more performant I that's a really good performant I that's a really good question I mean So within neurom magic question I mean So within neurom magic and within my background uh we were and within my background uh we were heavy users of C++ heavy template meta heavy users of C++ heavy template meta programming modern features you know we programming modern features you know we we we love that so we did give up a lot we we love that so we did give up a lot of that going to python but within VM of that going to python but within VM everything is typed we use mypi and and everything is typed we use mypi and and linting to to you know verify this on linting to to you know verify this on oursel and it makes it much easier to uh oursel and it makes it much easier to uh you know crawl across the code base you know crawl across the code base um you know I I I think projects like uh um you know I I I think projects like uh like modular or or mojo as you said uh like modular or or mojo as you said uh and and Triton which are these uh you and and Triton which are these uh you know not these domain specific languages know not these domain specific languages um are you know for for efficient code um are you know for for efficient code generation are really interesting uh I I generation are really interesting uh I I mean we use Triton a lot pytorch uses mean we use Triton a lot pytorch uses Triton a lot a lot of the ml Community Triton a lot a lot of the ml Community does as well I think this is uh you know does as well I think this is uh you know unfortunately kind of the best thing we unfortunately kind of the best thing we have at the moment an open source uh I have at the moment an open source uh I uh while I really love the language and uh while I really love the language and I wish I could use it more it's it just I wish I could use it more it's it just it's kind of too high level and doesn't it's kind of too high level and doesn't have enough performance to truly have enough performance to truly saturate the you know h100s h20s the saturate the you know h100s h20s the these really uh you know heavy gpus and these really uh you know heavy gpus and I I do think something more like like I I do think something more like like Mojo that has a sort of progressive uh Mojo that has a sort of progressive uh deepening uh where you can get down to deepening uh where you can get down to actually defining you know the the width actually defining you know the the width of of registers or the layouts of things of of registers or the layouts of things in a little bit more careful uh uh in a little bit more careful uh uh pathway um is really interesting but one pathway um is really interesting but one of the things that we're actually one of of the things that we're actually one of the biggest users of for our most the biggest users of for our most performing kernels is a library by performing kernels is a library by Nvidia called cutless M uh which is Nvidia called cutless M uh which is essentially a code generation it's a essentially a code generation it's a it's a template meta programming library it's a template meta programming library in C+ Plus but it's a code generation in C+ Plus but it's a code generation library for Cuda that allows you to library for Cuda that allows you to Define these very complex asynchronous Define these very complex asynchronous pipelined uh Matrix pipelined uh Matrix multiplications um in a way that uh is multiplications um in a way that uh is is much easier to do manually than than is much easier to do manually than than than in Cuda um so uh we we definitely than in Cuda um so uh we we definitely are proponents of code generation we're are proponents of code generation we're just not a little sure uh we we we just not a little sure uh we we we definitely don't want to write definitely don't want to write everything in cut lless even though it's everything in cut lless even though it's the it's the best thing we have for the the it's the best thing we have for the best performance best performance uh but you know Triton and Mojo are uh but you know Triton and Mojo are maybe a little bit too high level um maybe a little bit too high level um where we we definitely want to invest where we we definitely want to invest more in Middle grounds uh but you know more in Middle grounds uh but you know we've uh We've made compilers before uh we've uh We've made compilers before uh within our company so you know we we within our company so you know we we want to encourage other people to work want to encourage other people to work on it sounds great well thank you so on it sounds great well thank you so much it's a lot of information and uh much it's a lot of information and uh we're really looking forward to your we're really looking forward to your talk and welcome to the community we talk and welcome to the community we hope to hear more from you guys as you hope to hear more from you guys as you develop your then yeah thank you so much develop your then yeah thank you so much it's a pleasure to contributing to open it's a pleasure to contributing to open source