scale.bythebay.io: Shiva Chaitanya Interview

I'm Shiva Chaitanya I work a Netflix for I'm Shiva Chaitanya I work a Netflix for depersonalization infrastructure teen depersonalization infrastructure teen I'm a senior software engineer and we I'm a senior software engineer and we help our machine learning teams help our machine learning teams accelerate a BTS [Music] [Music] one of the major challenges for us is one of the major challenges for us is collecting a lot of data for machine collecting a lot of data for machine learning we have more than hundred learning we have more than hundred million members using Netflix and there million members using Netflix and there is a lot of user behavior that we want is a lot of user behavior that we want to capture because really the more the to capture because really the more the data that we can feed to our machine data that we can feed to our machine learning models we can learn better learning models we can learn better about our users and organizing all the about our users and organizing all the data in a standard format and feeding data in a standard format and feeding into machine learning pipelines which into machine learning pipelines which are both open source and proprietary is are both open source and proprietary is a big challenge for us with scale a big challenge for us with scale especially that pipelines that are especially that pipelines that are real-time real-time that means not that need to finish with that means not that need to finish with a pretty strict SLA let's say we have a a pretty strict SLA let's say we have a streaming job that needs to work on streaming job that needs to work on micro batches of data and need to finish micro batches of data and need to finish that within 10 minutes one of the that within 10 minutes one of the challenges that we keep facing is challenges that we keep facing is because our jobs depend on so many because our jobs depend on so many external services even if one of them external services even if one of them goes down that tends to back up our goes down that tends to back up our processing and it takes a while for us processing and it takes a while for us to debug what's going on and because to debug what's going on and because it's real time we don't have the luxury it's real time we don't have the luxury to rerun it or backfill the data for to rerun it or backfill the data for ones that we didn't process earlier and ones that we didn't process earlier and we need to keep moving on so informing we need to keep moving on so informing the right teams debugging it in a fast the right teams debugging it in a fast manner as possible as far as it's hard manner as possible as far as it's hard for us and especially when the scale is for us and especially when the scale is large the problems get compounded [Music] [Music] as we keep growing as a company we just as we keep growing as a company we just need to make sure that things that we need to make sure that things that we take for granted at a smaller scale take for granted at a smaller scale continue to work and as we on board a continue to work and as we on board a lot more tests as we collect more data lot more tests as we collect more data about our users as the demographics keep about our users as the demographics keep changing we expand to newer countries changing we expand to newer countries where the infrastructure may not be the where the infrastructure may not be the same so ensuring the status quo for a same so ensuring the status quo for a much larger scale in low bandwidth much larger scale in low bandwidth countries making sure we collect all the countries making sure we collect all the data in a timely matter all that is data in a timely matter all that is going to be a significant challenge for going to be a significant challenge for us machine learning lends itself us machine learning lends itself naturally to functional programming naturally to functional programming because you've taken a lot of data then because you've taken a lot of data then you apply vector of operations and all you apply vector of operations and all of on that data and you typically if you of on that data and you typically if you look at tens of flow frameworks or spark look at tens of flow frameworks or spark machine learning lab library which is machine learning lab library which is written in Scala they taken a bunch of written in Scala they taken a bunch of data apply a lot of transforms using data apply a lot of transforms using vector algebra and functional vector algebra and functional programming let's you think about that programming let's you think about that more naturally than say purely more naturally than say purely imperative way of writing these programs imperative way of writing these programs so we use scholar quite a bit in Netflix so we use scholar quite a bit in Netflix for all of machine learning processing [Music] [Music] my favorite part was learning about my favorite part was learning about Scala because I'm a new adopter of Scala Scala because I'm a new adopter of Scala so going especially to the functional so going especially to the functional track and learning all the new ways of track and learning all the new ways of using Scala to meet my needs was using Scala to meet my needs was eye-opening to me and I would you know eye-opening to me and I would you know given a chance I will keep coming given a chance I will keep coming you