SF Scala: Ulf Adams Interview

[Music] hello everybody I'm Alexi crabber of the hello everybody I'm Alexi crabber of the organizer of SF Scala and here on organizer of SF Scala and here on location at Twitter headquarters and we location at Twitter headquarters and we have this amazing meet up about have this amazing meet up about functional programming and machine functional programming and machine learning and it's about reproducibility learning and it's about reproducibility and one of the topics were never covered and one of the topics were never covered before is basil build system we learn before is basil build system we learn about it from Oscar Boykin who is a big about it from Oscar Boykin who is a big champion in the community and he's gonna champion in the community and he's gonna mention it and in his his man talk and mention it and in his his man talk and so we thought we'll we'll get somebody so we thought we'll we'll get somebody to talk about basil specifically and to talk about basil specifically and we're very fortunate to have we're very fortunate to have Adams who is the technical lead for Adams who is the technical lead for basil who is actually based in Munich basil who is actually based in Munich and he is visiting so it's all inside and he is visiting so it's all inside that beautifully welcome off we're that beautifully welcome off we're really happy to have you really happy to have you so in our corner of the woods we use so in our corner of the woods we use various aesthetic things to build things various aesthetic things to build things like we have our SBT which was supposed like we have our SBT which was supposed to be simple build tool and now like one to be simple build tool and now like one of the most complex build tools in the of the most complex build tools in the world and we have some other Twitter world and we have some other Twitter users pans which is its own mono repo or users pans which is its own mono repo or a gigantic build system you know using a gigantic build system you know using Python and then we have some newcomers Python and then we have some newcomers mill and CBT so we have programmers are mill and CBT so we have programmers are very fond of build systems so we love very fond of build systems so we love this and so it's exciting to hear about this and so it's exciting to hear about basil but we don't know much about it so basil but we don't know much about it so can you tell us a little bit briefly you can you tell us a little bit briefly you know what is it how many people use it know what is it how many people use it what is it good for but basil started what is it good for but basil started out as Google Earth system ensign a out as Google Earth system ensign a couple of years ago we decided to open couple of years ago we decided to open source it and you know you may know the source it and you know you may know the name of Google system is blaze and basil name of Google system is blaze and basil is sort of a rearrangement of the is sort of a rearrangement of the letters we couldn't use the term place letters we couldn't use the term place because of trademark reasons yeah and because of trademark reasons yeah and Google is using it for basically all of Google is using it for basically all of its entire projects not for everything its entire projects not for everything you know if you know Chrome or Android you know if you know Chrome or Android those aren't using basil yet we're those aren't using basil yet we're working on that so so it is basically a working on that so so it is basically a useful building very big for business useful building very big for business that yes you could say that so I think that yes you could say that so I think you folks you know in the community you folks you know in the community probably build this kind of stuff right probably build this kind of stuff right so so is it just us there is it is it so so is it just us there is it is it kind of as complex as the code base kind of as complex as the code base you're building is it much more complex you're building is it much more complex to make when I'm from one to ten right to make when I'm from one to ten right it's like if we make us three how it's like if we make us three how complex's is basil this is certainly a complex's is basil this is certainly a lot more complex than make and if you're lot more complex than make and if you're if you're dealing with such a big if you're dealing with such a big codebase everything becomes a challenge codebase everything becomes a challenge one of our biggest challenges is that one of our biggest challenges is that naive implementations of you know naive implementations of you know dependency data structures and dependency data structures and algorithms often run into a quadratic algorithms often run into a quadratic performance and memory consumption and performance and memory consumption and at Google scale that's already a problem at Google scale that's already a problem right we have to get it down from right we have to get it down from quadratic to linear and that requires quadratic to linear and that requires very very careful coding and using all very very careful coding and using all the right tricks and you know using all the right tricks and you know using all the tricks in in our repertoire to get the tricks in in our repertoire to get it down to linear okay it down to linear okay and so you see inherently parallel is it and so you see inherently parallel is it like basically a cluster by default so like basically a cluster by default so basil itself doesn't but it does it is basil itself doesn't but it does it is able to run individual actions in able to run individual actions in parallel if they are independent so parallel if they are independent so let's say you have you know it's called let's say you have you know it's called a program and it consists of two a program and it consists of two libraries and the libraries don't do libraries and the libraries don't do anything with each other then it can anything with each other then it can compile them in parallel and it can do compile them in parallel and it can do that on different machines so what we do that on different machines so what we do at Google is we have a machine cluster at Google is we have a machine cluster with a couple thousand machines or so with a couple thousand machines or so and then you can use as many of them as and then you can use as many of them as are currently free to to the Europe are currently free to to the Europe mm-hmm so what is the typical load you mm-hmm so what is the typical load you know how many machines are typically know how many machines are typically used for any kind of things I can't I used for any kind of things I can't I don't know tens hundreds thousands well don't know tens hundreds thousands well if you look if you look at basil if you look if you look at basil you know basil will run about a couple you know basil will run about a couple of hundred threads and if there is is of hundred threads and if there is is that much parallelism in your built then that much parallelism in your built then it will actually use a couple hundred it will actually use a couple hundred machines to do that okay so it's so for machines to do that okay so it's so for you it's not a problem to harness two you it's not a problem to harness two hundred machines and no process okay hundred machines and no process okay cool so and and how how long we should cool so and and how how long we should mastic is it right so I mean we're kind mastic is it right so I mean we're kind of in a weird is mostly scholar and this of in a weird is mostly scholar and this particular matter but if there are particular matter but if there are multilingual people I'm doing into multilingual people I'm doing into itself is multilingual so you know how itself is multilingual so you know how like language specific is it well it like language specific is it well it certainly has its roots in C++ and Java certainly has its roots in C++ and Java because those are the main languages at because those are the main languages at Google but we've been using it for Google but we've been using it for basically anything there are people basically anything there are people using it for Scala Kotlin Fortran using it for Scala Kotlin Fortran Haskell you name it we have it I there I Haskell you name it we have it I there I think there were even people who are think there were even people who are using for Pascal mm-hmm using for Pascal mm-hmm I'm not sure if I remember how path code I'm not sure if I remember how path code is written but you know it sort of diff is written but you know it sort of diff to the past right but yeah it's all to the past right but yeah it's all there it's all there so there so so is there it's all there so there so so is it is it hard to you know let's say it is it hard to you know let's say rough people suddenly learn about you rough people suddenly learn about you know like the like Python when they want know like the like Python when they want to do it so how hard is it for for a new to do it so how hard is it for for a new group to to use basil group to to use basil well it certainly requires some well it certainly requires some investment to learn how basil works you investment to learn how basil works you need to understand some of how basil need to understand some of how basil works in order to be able to extend it works in order to be able to extend it because because of the scale issues we because because of the scale issues we have developed very special data have developed very special data structures and algorithms to handle that structures and algorithms to handle that scale and so if you extend it you also scale and so if you extend it you also need to be involved in that that said need to be involved in that that said the community has been able to write the community has been able to write rust robots actually and Scala rules and rust robots actually and Scala rules and cottonwoods with fairly little help from cottonwoods with fairly little help from us us mm-hmm so it doesn't seem too hard oh mm-hmm so it doesn't seem too hard oh and I'm not a good judge for that okay and I'm not a good judge for that okay okay so you mentioned that basically okay so you mentioned that basically better kind of has its roots in Google's better kind of has its roots in Google's own build system right so so would you own build system right so so would you say that this is kind of it has almost say that this is kind of it has almost like a decade the more of experience so like a decade the more of experience so so is it is it really so bicycles tested so is it is it really so bicycles tested at all the scale right and so so it's at all the scale right and so so it's it's really mature about systems it's it's really mature about systems it's open-source recently but it's actually open-source recently but it's actually very well - yes okay so yeah it's it's very well - yes okay so yeah it's it's over ten years old now it was you know over ten years old now it was you know the first lines of code were written the first lines of code were written over that over ten years ago and it has over that over ten years ago and it has grown with Google Google wasn't you know grown with Google Google wasn't you know ten years ago Google wasn't actually as ten years ago Google wasn't actually as big as it is today big as it is today mm-hmm the code base wasn't as big and mm-hmm the code base wasn't as big and so we had a lot of problems we so we had a lot of problems we discovered as we were going along mm-hm discovered as we were going along mm-hm so what got you interested in in the so what got you interested in in the space well I dunno I applied at Google space well I dunno I applied at Google and so I went to the office for the and so I went to the office for the first time and the you know the sightly first time and the you know the sightly told me well there are two projects that told me well there are two projects that you could work on how about you talk to you could work on how about you talk to the respective leads and then you know the respective leads and then you know you can you can make a decision where you can you can make a decision where you want to go and so I went to the you want to go and so I went to the first guy and he was sitting at his first guy and he was sitting at his computer and I was like so what's your computer and I was like so what's your project about well you have to like it project about well you have to like it to work on it okay but what is it about to work on it okay but what is it about all right so that wasn't the word that I all right so that wasn't the word that I said and then I went over to the other said and then I went over to the other to the other guy and he said look let me to the other guy and he said look let me start this built here and we'll let the start this built here and we'll let the plant run in the background while I plant run in the background while I explain how the architectures is you explain how the architectures is you know how it works how its setup how its know how it works how its setup how its distributing builds on the data center distributing builds on the data center yeah well like whoa that's what I want yeah well like whoa that's what I want to work on this is so cool and then I to work on this is so cool and then I stuck with it and so I've been working stuck with it and so I've been working on basil for for nine years now wow on basil for for nine years now wow that's awesome not quite so what did you do build not quite so what did you do build systems before did you like make you systems before did you like make you know did you like to write micros not at know did you like to write micros not at all I just you know started to work on all I just you know started to work on it it cool it it cool can I grew on me interesting and so you can I grew on me interesting and so you mentioned that the team is not mentioned that the team is not distributed so if I folks in Europe pick distributed so if I folks in Europe pick up folks on this course but he said up folks on this course but he said there's almost nobody in the area where there's almost nobody in the area where Google started so what like was it an Google started so what like was it an evolution I mean did they migrate well evolution I mean did they migrate well the you know the manager of the group at the you know the manager of the group at the time wanted to have thought it was a the time wanted to have thought it was a good idea to have people working on it good idea to have people working on it in your neck or I don't know for what in your neck or I don't know for what reasons and so he started the group in reasons and so he started the group in Munich and then the group in Munich was Munich and then the group in Munich was actually the main group working on the actually the main group working on the boat system for a while and then Google boat system for a while and then Google hired you know a director in New York to hired you know a director in New York to work on developer tools mm-hmm and you work on developer tools mm-hmm and you know a second group started to work on know a second group started to work on it from from New York interesting yeah I it from from New York interesting yeah I mean I know many folks Google but this mean I know many folks Google but this is interesting to me like that heart or is interesting to me like that heart or not like the build system is not a not like the build system is not a little developed in the vein which is little developed in the vein which is interesting but maybe that has interesting but maybe that has advantages right because when wake up advantages right because when wake up you guys already did everything so so you guys already did everything so so this is three you know we're really this is three you know we're really happy you know to to learn more about happy you know to to learn more about this so I look forward to your main talk this so I look forward to your main talk and you know a portion to have you here and you know a portion to have you here so we know welcome back any time and so we know welcome back any time and tell us more about Basel you