Scale By the Bay 2023: Interview with RJ Nowling.

so my name is RJ noling I'm actually an so my name is RJ noling I'm actually an associate professor at a small school in associate professor at a small school in uh Milwaukee called the Milwaukee School uh Milwaukee called the Milwaukee School of Engineering so we have about 3,000 of Engineering so we have about 3,000 students mostly undergraduate programs students mostly undergraduate programs with some Master's programs uh our Focus with some Master's programs uh our Focus we're Polytechnic so all of our majors we're Polytechnic so all of our majors are related to engineering and so I'm a are related to engineering and so I'm a co-director of uh several graduate co-director of uh several graduate programs in machine learning and I teach programs in machine learning and I teach courses such as data science but also ml courses such as data science but also ml production systems and I had four years production systems and I had four years of Industry experience as an ml engineer of Industry experience as an ml engineer between my PhD and faculty position and between my PhD and faculty position and I've recently done some Consulting work I've recently done some Consulting work with Memphis uh who's developing a uh with Memphis uh who's developing a uh sort of message streaming platform uh sort of message streaming platform uh data streaming platform so um I I teach AI uh but you platform so um I I teach AI uh but you know my focus is is especially recently know my focus is is especially recently is on the ml production system so when I is on the ml production system so when I was uh an engineer in Industry I worked was uh an engineer in Industry I worked on massive systems that for ad click on massive systems that for ad click prediction um with hundreds of millions prediction um with hundreds of millions of of different cookies and so one of of of different cookies and so one of the things that you know we realized is the things that you know we realized is that or at my university is this that or at my university is this knowledge is kind of specialized and knowledge is kind of specialized and it's you learn it tribally you go met it's you learn it tribally you go met work under someone at a startup or work under someone at a startup or another company doing this and so my another company doing this and so my focus has been how do I standardize this focus has been how do I standardize this knowledge and package it in a format knowledge and package it in a format that's easily adap uh picked up by other people I think one of the the industry people I think one of the the industry changes I've seen is you know four or changes I've seen is you know four or five years ago when I was doing this it five years ago when I was doing this it was mostly batch and so we do a lot of was mostly batch and so we do a lot of sort of the backend pre-computer a lot sort of the backend pre-computer a lot of things for our known users and that's of things for our known users and that's very computationally expensive um you very computationally expensive um you know and it it wasn't very amendable to know and it it wasn't very amendable to you know figuring out which users had you know figuring out which users had updated data and which users didn't so updated data and which users didn't so you ended up recomputing everything it you ended up recomputing everything it was expensive very complicated pipelines was expensive very complicated pipelines hard to maintain hard to operate hard to hard to maintain hard to operate hard to keep running um and now I you know I keep running um and now I you know I love streaming because it allows us to love streaming because it allows us to do things like you know oh we have some do things like you know oh we have some history for these you know incoming data history for these you know incoming data for these particular set of users we can for these particular set of users we can update just for those users the update just for those users the information we have and update our key information we have and update our key value stores or whatever to use at value stores or whatever to use at prediction time so it the pipelines prediction time so it the pipelines become a lot smaller in some ways become a lot smaller in some ways because they only need to process the because they only need to process the changes and that makes them easier to changes and that makes them easier to manage and easier to implement uh makes manage and easier to implement uh makes the resources you know in terms of cloud the resources you know in terms of cloud compute costs and stuff easier to compute costs and stuff easier to predict um and then the other side of predict um and then the other side of that is is you know we're seeing some of that is is you know we're seeing some of that incremental computation moving from that incremental computation moving from the back end to the real time you know the back end to the real time you know so one of the other industry Trends is so one of the other industry Trends is you know this was kind of what my talk you know this was kind of what my talk was about was not just hey let's was about was not just hey let's pre-compute all of this information we pre-compute all of this information we need for the users because we don't want need for the users because we don't want to store things when you know only 1% of to store things when you know only 1% of our users are active users or something our users are active users or something like that or 10% that's costly so like that or 10% that's costly so instead how can we store information instead how can we store information such as say you know such as say you know uh the American Community survey data uh the American Community survey data like incomes and population size and like incomes and population size and stuff like that for zip codes that can stuff like that for zip codes that can be used across a wide range of objects be used across a wide range of objects so it's much smaller much more efficient so it's much smaller much more efficient to to store but then when you do the to to store but then when you do the predictions in real time you can pull predictions in real time you can pull those from the key value store and do those from the key value store and do your join online and so you basically your join online and so you basically just put everything assemble everything just put everything assemble everything as late as possible in the process to as late as possible in the process to avoid having to pre-compute it which avoid having to pre-compute it which again reduces cost and it also makes you again reduces cost and it also makes you more responsive right you can pull the more responsive right you can pull the latest data and and so you can make sure latest data and and so you can make sure that you know if you have new that you know if you have new information about the user that you information about the user that you incorporate that as quickly as possible in terms of some of the talks possible in terms of some of the talks I'm most interested in there's a talk by I'm most interested in there's a talk by Quin here about streaming graph so I had Quin here about streaming graph so I had some friends who tried to use graph some friends who tried to use graph databases in genomics and it didn't work databases in genomics and it didn't work very well because the data sizes were very well because the data sizes were just m massive and not in terms of like just m massive and not in terms of like raw gigabytes per se but just a number raw gigabytes per se but just a number of very small records with a lot of of very small records with a lot of interconnections and so I'm very interconnections and so I'm very interested in how Quin as a streaming interested in how Quin as a streaming graph engine uh allows that incremental graph engine uh allows that incremental comp again that theme of incremental comp again that theme of incremental computation allows that incremental computation allows that incremental computation um because I think it might computation um because I think it might make it more feasible instead of trying make it more feasible instead of trying to do giant batch on graph um I was also to do giant batch on graph um I was also very interested in the talk on very interested in the talk on converting a scallet pipeline to rust I converting a scallet pipeline to rust I don't personally use rust but I have don't personally use rust but I have some uh industry folks who I have some uh industry folks who I have connections with who do and they're just connections with who do and they're just amazed by like the the operational amazed by like the the operational efficiency of Building Systems in Rust efficiency of Building Systems in Rust and so I'm kind of curious as to sort of and so I'm kind of curious as to sort of going forward as a technology we know going forward as a technology we know it's great uh it's great uh how well will programmers and teams be how well will programmers and teams be able to adapt to it it's known as kind able to adapt to it it's known as kind of a hard language um and so I'm kind of of a hard language um and so I'm kind of curious as to how that's going to play curious as to how that's going to play out so my favorite conferences tend to be so my favorite conferences tend to be conferences that i' would say have like conferences that i' would say have like two or 300 people um because you get to two or 300 people um because you get to actually meet people and talk to them actually meet people and talk to them and you don't feel lost in the crowd and and you don't feel lost in the crowd and scale by the bay despite the name has scale by the bay despite the name has the appropriate scale of attendees and the appropriate scale of attendees and they're all practitioners right it's not they're all practitioners right it's not most of the talks here are not most of the talks here are not necessarily a companying company talking necessarily a companying company talking about their product that's an engineer about their product that's an engineer talking about a problem they learn they talking about a problem they learn they faced and how they solve that problem faced and how they solve that problem and their experiences so as another and their experiences so as another engineer I can take that and use that engineer I can take that and use that now that isn't to say there isn't great now that isn't to say there isn't great representation here of companies and representation here of companies and products there are but I love that products there are but I love that they're they're talked about in a very they're they're talked about in a very practical way of here's as an engineer practical way of here's as an engineer what you care about here's how it can what you care about here's how it can solve your problems um and so I just solve your problems um and so I just really enjoy the conversation and really enjoy the conversation and meeting I've been able to run into meeting I've been able to run into people that I worked with in people that I worked with in Industry you know four or five years ago Industry you know four or five years ago uh who haven't seen in four or five uh who haven't seen in four or five years and so that's really cool