Rajesh Muppala Interview

hello everybody I'm Alexa Crum hello everybody I'm Alexa Crum founder on the organizer of Scala by the founder on the organizer of Scala by the way the yearly conference forskolin way the yearly conference forskolin scale and here we are on location scale and here we are on location Twitter and I have a dish map along with Twitter and I have a dish map along with me who is the founder and director of me who is the founder and director of director feature of index a company out director feature of index a company out of Chennai yes and I met him at Pacific of Chennai yes and I met him at Pacific no rest call a couple of years ago and no rest call a couple of years ago and I'm very happy to finally have him here I'm very happy to finally have him here with us he was on a data panel welcome with us he was on a data panel welcome Rajesh thank you how can I do a little Rajesh thank you how can I do a little bit about index and what is the bit about index and what is the trajectory and how you guys are related trajectory and how you guys are related to all this scala technologist we're to all this scala technologist we're machine yeah so so I tend X we are machine yeah so so I tend X we are trying to build a Google of products I trying to build a Google of products I mean what do I mean by that is you know mean what do I mean by that is you know you have you know Facebook and LinkedIn you have you know Facebook and LinkedIn having an index of people and Google having an index of people and Google having an index of locations so we want having an index of locations so we want to be the index of products all things to be the index of products all things so we do that by you know crawling so we so we do that by you know crawling so we have a you know crawling engine that is have a you know crawling engine that is built using akka cluster you know that's built using akka cluster you know that's about three years old and that's when we about three years old and that's when we actually started with Scala so over the actually started with Scala so over the course of time you know we've used course of time you know we've used started using a lot of color started using a lot of color technologies we use carding for our technologies we use carding for our machine learning data pipeline and more machine learning data pipeline and more recently we are using spark for our recently we are using spark for our end-user end-user platform that allows end-users to you platform that allows end-users to you know build data pipelines and get data know build data pipelines and get data out out of our systems mm-hmm so yeah so out out of our systems mm-hmm so yeah so we are we started as a Java shop but you we are we started as a Java shop but you know once you know we moved to Scala know once you know we moved to Scala about three years back and we've not you about three years back and we've not you know gone back since yes I remember know gone back since yes I remember vividly you showed me so you worked for vividly you showed me so you worked for parks before I see you using your OCD parks before I see you using your OCD and you've got this amazing dev boards and you've got this amazing dev boards all the socket processes go away and all all the socket processes go away and all this stuff crawling and be recognized this stuff crawling and be recognized and so I mean it's a fascinating problem and so I mean it's a fascinating problem right basically to collect and resolve right basically to collect and resolve all the products in the world yes so all the products in the world yes so what is what does call a platform give what is what does call a platform give you why did you choose it you why did you choose it yeah I think I mean see the collar it's yeah I think I mean see the collar it's already you know it called as a disputed already you know it called as a disputed system and you know I worked on go CD system and you know I worked on go CD and you know I built I mean that goes and you know I built I mean that goes into the disputed system and I we use into the disputed system and I we use Java there you know I didn't want to Java there you know I didn't want to write the low-level you know code of write the low-level you know code of handling concurrency and all because you handling concurrency and all because you know it's very difficult to reason out know it's very difficult to reason out what's you know what happens when things what's you know what happens when things go bad and with akka you know which you go bad and with akka you know which you know which are just you know started it know which are just you know started it felt like a you know nice nice fit fit felt like a you know nice nice fit fit for our use case also at the same time for our use case also at the same time you know we realized that for we were you know we realized that for we were writing a lot of Java Java MapReduce and writing a lot of Java Java MapReduce and it was not the right kind of abstraction it was not the right kind of abstraction for writing complex data pipelines and for writing complex data pipelines and scalding you know with its functional scalding you know with its functional abstractions in appointed as the right abstractions in appointed as the right kind of tool that we were looking at you kind of tool that we were looking at you know in people we also tried the pig but know in people we also tried the pig but you know scalding again was was the you know scalding again was was the right choice for us so so I think it was right choice for us so so I think it was when not just one reason I mean there when not just one reason I mean there multiple reasons I mean these libraries multiple reasons I mean these libraries and the ecosystem itself that you know and the ecosystem itself that you know help does I mean that made us choose help does I mean that made us choose color yes and I think there is a lot of color yes and I think there is a lot of interesting challenges there so how do interesting challenges there so how do you guys kind of connect data you guys kind of connect data engineering data science right so you do engineering data science right so you do machine learning and this products like machine learning and this products like what are the challenges of doing these what are the challenges of doing these two things together I mean when one two things together I mean when one challenge has been you know the data challenge has been you know the data that we get is not structured mm-hmm I'd that we get is not structured mm-hmm I'd you know a lot of data is unstructured you know a lot of data is unstructured yes so so I mean bulk of a time is spent yes so so I mean bulk of a time is spent in actually structuring this data and in actually structuring this data and making it you know useful for data making it you know useful for data science yes and you know to be frank you science yes and you know to be frank you know we made a few mistakes in hiding know we made a few mistakes in hiding itself you know where we went itself you know where we went invented and and hired just data invented and and hired just data scientist but we're looking for people scientist but we're looking for people who can take do things end-to-end you who can take do things end-to-end you know build models as well let's take know build models as well let's take them to production who can both be the them to production who can both be the data engineers rather that exactly yes data engineers rather that exactly yes this is great yeah we call this guys this is great yeah we call this guys through social engineers in some of the through social engineers in some of the companies I work for so so I this is a companies I work for so so I this is a fascinating problem and I mean in San fascinating problem and I mean in San Francisco Bay Area Francisco Bay Area there is nothing hotter than data there is nothing hotter than data science machine learning artificial science machine learning artificial intelligence right and obviously if you intelligence right and obviously if you can do this with data platforms which can do this with data platforms which scale right this is the Holy Grail scale right this is the Holy Grail right so and and you mentioned that kind right so and and you mentioned that kind of in not not everybody and you of in not not everybody and you community understand this and so if you community understand this and so if you were to explain like obviously we're the were to explain like obviously we're the epicenter here right so so how do you epicenter here right so so how do you see kind of the evolution of let's say I see kind of the evolution of let's say I know I take community in India right know I take community in India right oh how do you see this kind of wave of oh how do you see this kind of wave of machine learning big analytics taking a machine learning big analytics taking a hold there and why is this important to hold there and why is this important to really pay attention to this I mean in really pay attention to this I mean in India I think I mean the startup wave is India I think I mean the startup wave is pretty good there right now but I don't pretty good there right now but I don't think a lot of companies have are think a lot of companies have are focusing on the data side of things yet focusing on the data side of things yet I mean yes you know it's not their core I mean yes you know it's not their core business I mean it's you know they have business I mean it's you know they have apps or they have you know end-user apps or they have you know end-user applications or consumer apps but you applications or consumer apps but you know they use data science as a you know know they use data science as a you know as something that adds to their their as something that adds to their their business but it's not part of their core business but it's not part of their core business mm-hmm so I think I mean we'll business mm-hmm so I think I mean we'll we'll start seeing lot more data we'll start seeing lot more data companies I think I think they're one of companies I think I think they're one of the earliest and I mean at least I don't the earliest and I mean at least I don't know if any other company that deals you know if any other company that deals you know as much data as as we deal with so know as much data as as we deal with so I mean it's really hard to find you know I mean it's really hard to find you know people who have you know experience in people who have you know experience in some of the stuff that we do I mean we some of the stuff that we do I mean we don't get Hadoop engineers but you know don't get Hadoop engineers but you know they've not worked at the scale that they've not worked at the scale that that we were coming therefore they want that we were coming therefore they want some courses or you know yeah it's yeah some courses or you know yeah it's yeah this is a tough problem so so I remember this is a tough problem so so I remember right Oh metal you basically mentioned right Oh metal you basically mentioned that you know you went back to Chennai that you know you went back to Chennai right and you want to work in India you right and you want to work in India you want to employ Indian engineers and and want to employ Indian engineers and and I think I've seen kind of a wave with I think I've seen kind of a wave with you know Modi government kind of you know Modi government kind of technocratic focus I've seen a lot of technocratic focus I've seen a lot of young people actually and all people and young people actually and all people and all kind of people going back to India I all kind of people going back to India I saw Francis Pankaj Gupta was a Twitter saw Francis Pankaj Gupta was a Twitter guy right who did you know basically guy right who did you know basically whom to follow recommendation system and whom to follow recommendation system and now he went back there and so we see a now he went back there and so we see a lot of execs and developers so can you lot of execs and developers so can you tell a little bit you know how this tell a little bit you know how this works what do you see you know from that works what do you see you know from that other side yeah so I mean I think so so other side yeah so I mean I think so so if you see right I mean there are there if you see right I mean there are there are a few unicorns so to say in India I are a few unicorns so to say in India I mean there is there are the flip cards mean there is there are the flip cards yes you know we're going against Amazon yes you know we're going against Amazon you have the Ola cabs that's going to you have the Ola cabs that's going to going against uber you know then you going against uber you know then you have oh yeah so so there are a few few have oh yeah so so there are a few few companies like that companies like that and and I think you know they're you and and I think you know they're you know definitely you know having people know definitely you know having people from here in the Silicon Valley who have from here in the Silicon Valley who have done these kind of things you know done these kind of things you know scaling engineering teams back in India scaling engineering teams back in India would really help help is what what I would really help help is what what I would think yeah and a lot I think you would think yeah and a lot I think you know I think the the companies which know I think the the companies which hire kind of execs from this kind hire kind of execs from this kind I think the exclusive to say we need I think the exclusive to say we need guys who have experience with Big Data guys who have experience with Big Data machine learning right so I think the machine learning right so I think the court in the corporate level the pretty court in the corporate level the pretty smart they they understand that they smart they they understand that they need this kind of talent infusion right need this kind of talent infusion right because the scale here I think is because the scale here I think is leading will be capital big data capital leading will be capital big data capital of the world right and so from here kind of the world right and so from here kind of percolate through but so I'm curious of percolate through but so I'm curious about the community you mentioned the about the community you mentioned the committee is not as intense as developed committee is not as intense as developed but community is the main way of but community is the main way of learning about this technology so what learning about this technology so what you see here is completely organic right you see here is completely organic right people understand because there is a people understand because there is a demand for this technologists in the demand for this technologists in the market and this is the highest paid jobs market and this is the highest paid jobs in the industry they basically folks in the industry they basically folks learn oh this is this conference is learn oh this is this conference is learning vehicle right a lot of people learning vehicle right a lot of people come here they learn about the come here they learn about the technologies and then the K basically go technologies and then the K basically go to companies who need them so I to companies who need them so I personally think this is inevitable in personally think this is inevitable in in India as well right so I'm curious in India as well right so I'm curious you know how do you think this can be you know how do you think this can be fostered and you know if somebody helps fostered and you know if somebody helps with the community what rewards they can with the community what rewards they can reap by by being early kind of community reap by by being early kind of community organizers in India I mean I see I think organizers in India I mean I see I think gues there's definitely a need for gues there's definitely a need for building the community and you know I building the community and you know I think um think companies like index has think um think companies like index has to you know be part of it because you to you know be part of it because you know I think we are doing some some know I think we are doing some some really interesting stuff in this area I really interesting stuff in this area I mean unfortunately you know we've not mean unfortunately you know we've not been you know able to do in spend as been you know able to do in spend as much as as much time as we would like much as as much time as we would like but it's one of the things that we would but it's one of the things that we would like to do I mean you know I'm probably like to do I mean you know I'm probably when I go back you know we will start when I go back you know we will start this community I mean the challenge is this community I mean the challenge is also you know the maturity of folks also you know the maturity of folks there is when people are just learning there is when people are just learning about big data is getting their hands about big data is getting their hands dirty with Hadoop so how do we get them dirty with Hadoop so how do we get them from mere learners to you know from mere learners to you know practitioners I think that's that that practitioners I think that's that that gap is what we need to bridge is what I gap is what we need to bridge is what I think and and there is I mean and think and and there is I mean and there's work to be done and you know there's work to be done and you know hopefully you know we will do something hopefully you know we will do something about it about it yes and and there's also one more point yes and and there's also one more point I want to make I mean one you know this I want to make I mean one you know this is about what the work that we're doing is about what the work that we're doing at index so the goal of goal for me to at index so the goal of goal for me to add I mean the reason I am here is to add I mean the reason I am here is to probably you know because there are a probably you know because there are a lot of speakers you know who who are lot of speakers you know who who are actually building the stuff that we are actually building the stuff that we are using so I wanted to you know talk to using so I wanted to you know talk to them so probably you know 80% I wanted them so probably you know 80% I wanted to you know learn from them and probably to you know learn from them and probably 20% share what we are we are doing 20% share what we are we are doing endings but one thing I realized when I endings but one thing I realized when I when I came here is that do you know when I came here is that do you know actually you know it's 50% learning in actually you know it's 50% learning in 50% sharing in the sense that you know 50% sharing in the sense that you know whatever stuff we build at index you whatever stuff we build at index you know some of the stuff you are talking know some of the stuff you are talking about using go CD as a big data pipeline about using go CD as a big data pipeline yeah is something that you know folks yeah is something that you know folks from Apple from stripe I mean you know from Apple from stripe I mean you know they've actually said okay can you can they've actually said okay can you can you show this I mean can you know can you show this I mean can you know can you send can you you know tell me more you send can you you know tell me more about it I want to yes I want to know about it I want to yes I want to know more about it I think that's that's a more about it I think that's that's a little I mean it was surprising and it's little I mean it was surprising and it's also flattering yeah so yeah and yeah I also flattering yeah so yeah and yeah I mean I had the same reservation right I mean I had the same reservation right I thought but basically it's kind of what thought but basically it's kind of what really impressed me about in Dick's that really impressed me about in Dick's that you guys kind of quietly and you know you guys kind of quietly and you know built everything everybody's talking built everything everybody's talking about and so we often have talked to about and so we often have talked to people who build one half of this people who build one half of this probably presenting how they put two probably presenting how they put two things together you guys put like a things together you guys put like a million things together and they have a million things together and they have a dashboard right and it has been going at dashboard right and it has been going at scale right so I think it's really scale right so I think it's really interesting whether you have something interesting whether you have something very vailable have a proven engine which very vailable have a proven engine which is you know internet scale which right is you know internet scale which right like comes and finds all the products like comes and finds all the products and resolve them and it's a thing it's and resolve them and it's a thing it's kind of the modern startup worlds it's kind of the modern startup worlds it's an extremely valuable thing basically an extremely valuable thing basically this is the final result of all the this is the final result of all the fundraising and all the promises people fundraising and all the promises people start with right so it's like two beats start with right so it's like two beats having said that you know we still have having said that you know we still have a long way to way in the sense that I a long way to way in the sense that I mean one of the big challenges we're mean one of the big challenges we're dealing with is the quality of data dealing with is the quality of data itself because you get data from itself because you get data from multiple sources yeah so we actually and multiple sources yeah so we actually and I think we figure out the problem of I think we figure out the problem of ingesting that data you know the scaling ingesting that data you know the scaling of the pipelines yeah now the problem of the pipelines yeah now the problem that we're actually dealing with is that we're actually dealing with is actually how do we improve the quality actually how do we improve the quality of the data and make it more useful for of the data and make it more useful for our for our customers right in I mean our for our customers right in I mean again to me it's fascinating right again to me it's fascinating right because you know essentially data because you know essentially data science is 80/20 it's 80% data plumbing science is 80/20 it's 80% data plumbing there's veneering data wrangling right there's veneering data wrangling right so so once you have the efficient so so once you have the efficient structure to iterate upon then you can structure to iterate upon then you can improve the quality because you can pump improve the quality because you can pump the data through it quickly and that's the data through it quickly and that's the phase we are in right now all right the phase we are in right now all right so I'm kind of interest like let's say so I'm kind of interest like let's say you know I hope a bunch of folks in you know I hope a bunch of folks in India or C see us and how would you India or C see us and how would you describe your ideal candidate and you describe your ideal candidate and you know the young people who want know the young people who want to to of take courses and learn right what is of take courses and learn right what is your ideal research engineer and what your ideal research engineer and what would you suggest you know to them how would you suggest you know to them how can they build up the skills needed to can they build up the skills needed to do the job of research engineer now I do the job of research engineer now I think one is I mean then they should you think one is I mean then they should you know understand basic computer science know understand basic computer science concepts I mean they should you know concepts I mean they should you know they should understand algorithms you they should understand algorithms you know they should understand yeah under know they should understand yeah under the concept of computer science the the concept of computer science the second thing is I mean they should love second thing is I mean they should love data yes I mean they should you know data yes I mean they should you know they should you know they should not be they should you know they should not be afraid of you know taking the sleeves up afraid of you know taking the sleeves up and you're getting digging deeper into and you're getting digging deeper into data and and understanding what what data and and understanding what what what it is all about and then and I what it is all about and then and I think you know once they have these two think you know once they have these two things right I mean there is you know things right I mean there is you know enough of let's say coaching and enough enough of let's say coaching and enough of mentoring that we can do in our of mentoring that we can do in our company or or I mean you know they're company or or I mean you know they're enough resources that are available for enough resources that are available for them to you know be productive mm-hm and them to you know be productive mm-hm and kind of let's say you know they're good kind of let's say you know they're good enough and acquire them as junior enough and acquire them as junior musician engineers what will they learn musician engineers what will they learn on the job what can you give them back on the job what can you give them back what is the advantages of being exposed what is the advantages of being exposed to all these technologies you guys built to all these technologies you guys built for them no I think the the learning for them no I think the the learning rate I mean the you know you get to do rate I mean the you know you get to do what you know you get to do in a year what you know you get to do in a year you know what you would get to do in you know what you would get to do in another company in over five years right another company in over five years right when you're I mean so I I have this I when you're I mean so I I have this I mean I talked about this concept of you mean I talked about this concept of you know what is called an escape velocity know what is called an escape velocity right I mean once once the rocket reach right I mean once once the rocket reach an escape velocity you know it's it's on an escape velocity you know it's it's on its orbits on its own right yes it's its orbits on its own right yes it's very important for you to figure out you very important for you to figure out you know the right company in your in your know the right company in your in your start of your career so that you know start of your career so that you know you so that you're maximizing learning you so that you're maximizing learning yes and then once that happens I mean yes and then once that happens I mean you are you know you're all set in your you are you know you're all set in your career so I think you know in given that career so I think you know in given that what given the kind of stuff that we are what given the kind of stuff that we are doing I mean there's hard problems you doing I mean there's hard problems you are solving I think we can draw a desk a are solving I think we can draw a desk a velocity to developers is what I think velocity to developers is what I think nice so then you know my follow-up nice so then you know my follow-up questions will be how do you make sure questions will be how do you make sure that your most valuable developers not that your most valuable developers not escape all right because if you train escape all right because if you train all these guys and then satellite all these guys and then satellite Twitter and Facebook and Google see that Twitter and Facebook and Google see that they use the same technologist and they use the same technologist and there's not fully what they're doing there's not fully what they're doing right and they all kind of come into right and they all kind of come into India how how do you retain the top India how how do you retain the top talent I mean I I don't know I mean I talent I mean I I don't know I mean I think we are we are struggling on that think we are we are struggling on that front you know front you know I think it's very difficult for folks in I think it's very difficult for folks in you know in my company to appreciate the you know in my company to appreciate the kind of work we are doing because I mean kind of work we are doing because I mean it's it's a hard problem in trying to it's it's a hard problem in trying to create a new market I mean on a business create a new market I mean on a business side I mean we're not you know we're not side I mean we're not you know we're not having a trajectory that you would like having a trajectory that you would like because you know it's a hard problem to because you know it's a hard problem to solve so so I don't know I mean I one solve so so I don't know I mean I one think would be you know if they could think would be you know if they could come here and see for themselves you come here and see for themselves you know the kind of impact that you know know the kind of impact that you know what I could see here right I mean what I could see here right I mean people are coming and talking to me people are coming and talking to me about you know what is about the scale about you know what is about the scale about the idea pipe data pipelines mhm about the idea pipe data pipelines mhm so so I don't know I mean yeah let's see so so I don't know I mean yeah let's see yeah cool you know yeah we don't know yeah cool you know yeah we don't know that you sure but I think you know from that you sure but I think you know from what I can see I am really kind of happy what I can see I am really kind of happy about the trajectory of your garden about the trajectory of your garden right and I think it's definitely great right and I think it's definitely great to have index in our community it's to have index in our community it's great to finally have this conference great to finally have this conference will now hope you know next time you will now hope you know next time you come and do a full talk right and kind come and do a full talk right and kind of we collaborate on the meetups and of we collaborate on the meetups and community and we're help you know happy community and we're help you know happy to help you guys build this community to help you guys build this community yeah thanks a lot and I think you're yeah thanks a lot and I think you're doing a great job with the community doing a great job with the community here and good luck thank you thank you here and good luck thank you thank you yeah thanks