sfspark.org: Alexy Khrabrov interviews Sandy Ryza

hello everybody I'm electric robber of hello everybody I'm electric robber of the organizer of SF Sparkle friends give the organizer of SF Sparkle friends give it up about spark in its ecosystem such it up about spark in its ecosystem such as Sakha Kafka Cassandra Pasha blink and as Sakha Kafka Cassandra Pasha blink and you see simple men date that will hear you see simple men date that will hear about later today and we're very happy about later today and we're very happy to have here without send ariza who is to have here without send ariza who is the senior data scientist at caldera and the senior data scientist at caldera and the author of the new book from o'reilly the author of the new book from o'reilly called advanced analytical spark which called advanced analytical spark which we have multiple copies here in the we have multiple copies here in the office and they're very much enjoyment office and they're very much enjoyment great to hear and the topic of Senna's great to hear and the topic of Senna's talk about semantic analysis of formula talk about semantic analysis of formula documents the patches park is highly documents the patches park is highly relevant nitra which is a document relevant nitra which is a document productivity company we work with productivity company we work with hundreds of millions of pdfs want to hundreds of millions of pdfs want to understand what they are evolved so this understand what they are evolved so this is the technology we chose to use as is the technology we chose to use as well so welcome to india is great to well so welcome to india is great to have you here thanks Evan here so I did have you here thanks Evan here so I did some research on you and the cold air some research on you and the cold air has a nice blog post type of meeting the has a nice blog post type of meeting the data scientists and and we both know data scientists and and we both know Josh real pretty well so it's kind of Josh real pretty well so it's kind of expanding on Josh reals theory of what a expanding on Josh reals theory of what a data scientist is a little bit and that data scientist is a little bit and that but that was a thing a year ago so I'm but that was a thing a year ago so I'm wondering so it basically can appreciate wondering so it basically can appreciate that here the data scientist is a little that here the data scientist is a little janitor and and has also some janitor and and has also some interesting points but I wondering you interesting points but I wondering you know after that year hope they're wrong know after that year hope they're wrong and I'm special after writing this book and I'm special after writing this book your initial vision change or was there your initial vision change or was there some evolution of this and what what it some evolution of this and what what it is that's a great question yeah I would is that's a great question yeah I would say that probably if anything my vision say that probably if anything my vision then reinforced in that a lot of my job then reinforced in that a lot of my job is doing engagements at different is doing engagements at different customers oh yeah help you fellow come customers oh yeah help you fellow come something you get it set up for whatever something you get it set up for whatever days whatever people want um and very days whatever people want um and very rarely does it come down to actually i rarely does it come down to actually i would say applying machine learning would say applying machine learning algorithm um often people turn to people algorithm um often people turn to people were called data scientists to have were called data scientists to have someone to think about the data reason someone to think about the data reason about it and try that sort through do about it and try that sort through do the same common sense stuff that you the same common sense stuff that you know you might use another spheres but i know you might use another spheres but i don't know some people i'm confident in don't know some people i'm confident in doing that in relation to Lord data sets doing that in relation to Lord data sets um so yeah if anything I would say that um so yeah if anything I would say that uh that a Chandler Bing is still some uh that a Chandler Bing is still some huge percent of it and then another huge percent of it and then another percent of it before you even get to percent of it before you even get to doing math or statistics it's just sort doing math or statistics it's just sort of what makes sense here gorilla right of what makes sense here gorilla right over there I think there is a number over there I think there is a number falls in around that eighty percent of falls in around that eighty percent of identify his job spam Monching the data identify his job spam Monching the data do you think that's roughly the right do you think that's roughly the right ratio I think that's perf little better ratio I think that's perf little better ratio and so I mean in my theory I I ratio and so I mean in my theory I I kind of also have some kind of simple kind of also have some kind of simple way sore eyes were the scientists are so way sore eyes were the scientists are so I wonder I know or they kind of I wonder I know or they kind of exaggerated but I will run the Bayou so exaggerated but I will run the Bayou so one theory runs that no it's also a data one theory runs that no it's also a data scientist a small insight is a Python scientist a small insight is a Python programmer cook and you know sort and programmer cook and you know sort and ready right and and and because ready right and and and because typically i wonder if you agree that typically i wonder if you agree that most algorithms most results can be most algorithms most results can be obtained with the elder a delicious sort obtained with the elder a delicious sort uniq take top tab right which is super uniq take top tab right which is super easy to do a new leak stories because easy to do a new leak stories because you can fight these things right and you you can fight these things right and you can but you do explore it at once you can but you do explore it at once you can very easily as education whether can very easily as education whether you're sorting right so once you have you're sorting right so once you have something to soar it then it's probably something to soar it then it's probably was going to be here just trying to get was going to be here just trying to get things you know things you know in the state that they can be ordered in the state that they can be ordered that's right you some shorts and that's right you some shorts and transformed into pulse is sorting some transformed into pulse is sorting some field and then you get the red thread so field and then you get the red thread so so this code usually starts in spalding so this code usually starts in spalding so the big date as I this is somebody so the big date as I this is somebody who can sort the array loop or spark or who can sort the array loop or spark or something which spans more than one something which spans more than one machine and then of course it becomes machine and then of course it becomes much much harder because even getting much much harder because even getting air in the air and certain it and it's air in the air and certain it and it's become so so I wonder you know if you do become so so I wonder you know if you do this and kind of in their kind of the this and kind of in their kind of the question is do we have two kinds of data question is do we have two kinds of data scientists because machine learning is scientists because machine learning is harder and very different thing and harder and very different thing and usually it's explained through our usually it's explained through our presence and statistics right so so in presence and statistics right so so in reality I'm not just that a lot of reality I'm not just that a lot of people who can really properly sort people who can really properly sort distributed array are different from the distributed array are different from the people who can reason about you know people who can reason about you know gradient boosting and various optimizers gradient boosting and various optimizers and very special algorithms and the the and very special algorithms and the the letter kind of people use the letter kind of people use the development matlab and the top for development matlab and the top for instance vision sciences program in instance vision sciences program in matlab and and so they still have to matlab and and so they still have to integrate in some way so I wonder what integrate in some way so I wonder what you're taking this what do you see it you're taking this what do you see it caldera what you seen these are distinct caldera what you seen these are distinct merging they still kind of develop in a merging they still kind of develop in a parallel what's your views um yeah parallel what's your views um yeah certainly I mean I think in general certainly I mean I think in general people columns of data scientists or a people columns of data scientists or a great extent people that come from or great extent people that come from or math or physics background and people math or physics background and people who come from more programming who come from more programming background um I think the whole pieces background um I think the whole pieces of having like data science as a term at of having like data science as a term at all is like the idea that is useful for all is like the idea that is useful for these people to get together and for these people to get together and for single people to try to gain a little single people to try to gain a little bit of both of these but these skills um bit of both of these but these skills um you know in many situations you call the you know in many situations you call the latter group of computer science group latter group of computer science group data engineers mm-hmm that's the data engineers mm-hmm that's the breakdown that happens to be a lot of breakdown that happens to be a lot of companies but I don't really see that companies but I don't really see that division going away and so but grab this division going away and so but grab this kind of unicorns who do both right i kind of unicorns who do both right i think you're closer to think you're closer to uniform right i mean the dream is that uniform right i mean the dream is that you are able to build frameworks like you are able to build frameworks like spark for example that makes it easy to spark for example that makes it easy to dupe up right to work with you data sets dupe up right to work with you data sets to put things in production but also to to put things in production but also to easily let River knowledge you have easily let River knowledge you have about an asset istics hmm I'm in a way about an asset istics hmm I'm in a way it doesn't require putting up in Java or it doesn't require putting up in Java or a MapReduce yes so you're also a a MapReduce yes so you're also a contributor to purchase park so a year contributor to purchase park so a year ago you described the job is spending ago you described the job is spending half the time with customers and how the half the time with customers and how the time contribute into tools such as much time contribute into tools such as much respond uses steal the same ratio it's a respond uses steal the same ratio it's a good question yeah I think that in the good question yeah I think that in the very recent past I've moved a little bit very recent past I've moved a little bit away from sparking a little bit more away from sparking a little bit more towards the data science side of the job towards the data science side of the job um yeah I've been on a lot of customer um yeah I've been on a lot of customer engagements not doing a lot of Education engagements not doing a lot of Education for Clara talking and speaking for Clara talking and speaking engagements this kind of stuff careful engagements this kind of stuff careful with yeah not enough to do it and I with yeah not enough to do it and I think I might be coming back a bit more think I might be coming back a bit more towards Barca and near future like get towards Barca and near future like get tired of flying around them enough and tired of flying around them enough and when you are can you just park what a when you are can you just park what a race are going to get into um so the two race are going to get into um so the two main areas are darn and then statistics main areas are darn and then statistics functionality and ml is um we're functionality and ml is um we're probably a much more of an emphasis on probably a much more of an emphasis on yarn so I come from a Hadoop background yarn so I come from a Hadoop background huh committed on yarn mm-hmm so I've huh committed on yarn mm-hmm so I've been doing a lot of yarn scheduling been doing a lot of yarn scheduling stuff right now one of the big things stuff right now one of the big things I'm working on is reviewing a patch that I'm working on is reviewing a patch that will bring in the ability to do executor will bring in the ability to do executor replacement based on data locality replacement based on data locality preferences mm-hmm and this has been preferences mm-hmm and this has been something that sort of it missing its something that sort of it missing its park in a while maybe a place where park in a while maybe a place where MapReduce they'll have differentiation MapReduce they'll have differentiation but this guy sigh sigh Xiao at Intel but this guy sigh sigh Xiao at Intel eyes thankfully taking this up and we're eyes thankfully taking this up and we're trying to move that forward oh nice trying to move that forward oh nice that's something that's great to hear that's something that's great to hear because I think little college is really because I think little college is really important for performance so yeah I important for performance so yeah I think it's very interesting that the think it's very interesting that the other yarn so it's I think it's really other yarn so it's I think it's really kind of shocked eating the kind of shocked eating the can kind of folks who can actually make can kind of folks who can actually make real progress as far because you know real progress as far because you know yarn as a divorcee things right like how yarn as a divorcee things right like how the actual raleigh's doesn't it's not the actual raleigh's doesn't it's not obvious thing in you cover because usual obvious thing in you cover because usual star on amazon but then he got a star on amazon but then he got a production environment like natural once production environment like natural once the question is how the actual wrong the question is how the actual wrong with the 47 really lightly have a with the 47 really lightly have a liability of them is on what yarn or or liability of them is on what yarn or or mrs. right then the historically spark mrs. right then the historically spark was developed on VESA sexually messes was developed on VESA sexually messes was the project but they was working on was the project but they was working on that his bond was a cute ab demo right that his bond was a cute ab demo right then suddenly kind of it reversed and so then suddenly kind of it reversed and so messes with motive is the twitter but messes with motive is the twitter but now there is Mississippi here and a lot now there is Mississippi here and a lot of folks actually are running sparking of folks actually are running sparking messes so and I'm probably more familiar messes so and I'm probably more familiar with companies running messages bitter with companies running messages bitter and sigmoid only takes guys their and sigmoid only takes guys their bicycles very easy to come into a shop bicycles very easy to come into a shop and install messes alright if you don't and install messes alright if you don't have the full cdh there already so I'm have the full cdh there already so I'm wondering kind of kind of emphasize or wondering kind of kind of emphasize or highlight the differences are in between highlight the differences are in between erm SSN you know why they obviously by erm SSN you know why they obviously by contribute to yarn then you you bet on contribute to yarn then you you bet on the yarn or you're like you're so how the yarn or you're like you're so how would somebody new peak between yarn would somebody new peak between yarn messes yeah it's a great question um I messes yeah it's a great question um I mean in theory they did the same thing I mean in theory they did the same thing I personally wish that we could just personally wish that we could just standardized on one of them um may standardized on one of them um may supplement fine fine um but as it ended supplement fine fine um but as it ended up handing out we have two of these I up handing out we have two of these I think what is sort of ended up happening think what is sort of ended up happening is that people who are trying to use is that people who are trying to use clusters for scheduling much of services clusters for scheduling much of services I've got 10 more tomatoes and yarns I've got 10 more tomatoes and yarns become more of a framework or that high become more of a framework or that high throughput data processing mm-hmm um so throughput data processing mm-hmm um so most people i talked to run spark on most people i talked to run spark on yarn it could be a fluid in some sort of yarn it could be a fluid in some sort of insulated Cloudera situation but my insulated Cloudera situation but my hunch is that sport gets run a little hunch is that sport gets run a little bit more on your bit more on your it does on missus I can talk about sort it does on missus I can talk about sort of specific you know sells for yarn of specific you know sells for yarn ourselves for missus but in theory is ourselves for missus but in theory is you know their systems that do the same you know their systems that do the same thing okay I hope you know I would like thing okay I hope you know I would like for us I'll be able to pick one and move for us I'll be able to pick one and move forward with it but maybe to stuff with forward with it but maybe to stuff with a flipper interpretation Oh couldn't do a flipper interpretation Oh couldn't do a bit of altaf achieve myriad which I a bit of altaf achieve myriad which I think that's some kind of preparation of think that's some kind of preparation of this ah I actually not her that is that this ah I actually not her that is that something that everyone you're not something that everyone you're not Messrs that's what I you know what it Messrs that's what I you know what it does mommy know we don't have talked does mommy know we don't have talked about it um yeah I mean there's been about it um yeah I mean there's been quite a few of my proposal to run yarns quite a few of my proposal to run yarns and mrs. and mrs. inside of yarn um you and mrs. and mrs. inside of yarn um you know it seems a little weird to me like know it seems a little weird to me like within yarn you're already doing to within yarn you're already doing to level scheduling or are you already have level scheduling or are you already have you are doing scheduling and then you are doing scheduling and then sparking scheduling that's right we can sparking scheduling that's right we can hear on scheduling so why another layer hear on scheduling so why another layer is needed a nut sure um you know at is needed a nut sure um you know at least in Malaga maybe is a temporary least in Malaga maybe is a temporary solution yeah make sense yes yeah that solution yeah make sense yes yeah that was my question as well right because was my question as well right because obviously it's already the way to do obviously it's already the way to do performances to basically make the performances to basically make the schedule Kevin transparent like over schedule Kevin transparent like over assembly right between different assembly right between different frameworks so that kind of what my frameworks so that kind of what my personal was ah interesting so so months personal was ah interesting so so months definitely the kind of the Golden Age of definitely the kind of the Golden Age of spark right there is tremendous interest spark right there is tremendous interest we see IBM announcing spark technology we see IBM announcing spark technology center right here on market street and center right here on market street and get behind spark before us so this is get behind spark before us so this is something this is this is a fantastic something this is this is a fantastic time sourcing way some folks are saying time sourcing way some folks are saying maybe couple years ago that Hadoop is maybe couple years ago that Hadoop is going to go with it is actually going to going to go with it is actually going to be sparked a dupe name is going to be be sparked a dupe name is going to be there as a brand but in fact is going to there as a brand but in fact is going to be spark is going to be big data now do be spark is going to be big data now do you agree with this um so I think it you agree with this um so I think it partly comes from the original partly comes from the original completion with Hadoop and hadoop completion with Hadoop and hadoop mapreduce hmm so yeah I wonder doesn't mapreduce hmm so yeah I wonder doesn't believe believe that produce is going to go away nobody that produce is going to go away nobody knows how soon Burke has a lot of you to knows how soon Burke has a lot of you to catch up but ultimately matters can be catch up but ultimately matters can be totally gone I think spark and satisfy totally gone I think spark and satisfy all the use cases I'm not convinced with all the use cases I'm not convinced with the other parts of Hadoop are going to the other parts of Hadoop are going to go away not convinced that at HDFS is go away not convinced that at HDFS is going to go away or or you know even if going to go away or or you know even if it gets replaced by some other file it gets replaced by some other file system those Hadoop api's I've sold away system those Hadoop api's I've sold away mm-hmm I'm interacting you know of the mm-hmm I'm interacting you know of the processing layer to interact with the processing layer to interact with the storage layer and then the resource storage layer and then the resource management aspect yarn on top of that um management aspect yarn on top of that um and then of course there's the whole and then of course there's the whole broader ecosystem right boom broader ecosystem right boom century every sort of sort of project so century every sort of sort of project so of course those aren't technically had of course those aren't technically had you but I wouldn't say that spark spark you but I wouldn't say that spark spark has extremely important role but I has extremely important role but I wouldn't say that everything revolves wouldn't say that everything revolves around it yes and so we see I think you around it yes and so we see I think you see definitely everybody with also led see definitely everybody with also led to use in HDFS so I think there is not to use in HDFS so I think there is not really any other simple alternative to really any other simple alternative to just break into in stores and um txt just break into in stores and um txt files into it forever right or up arcade files into it forever right or up arcade or doing that so what we see a lot of or doing that so what we see a lot of places running down some somebody came places running down some somebody came up with the term smack which is spark up with the term smack which is spark mrs. acha Cassandra Kafka so so it's mrs. acha Cassandra Kafka so so it's kind of a new stack yes I think that's kind of a new stack yes I think that's it folks came up with this and I told it folks came up with this and I told you know this should be an extensional you know this should be an extensional it's my card which is how available it's my card which is how available resilient distributed is not in it all resilient distributed is not in it all right you just by default it is so I'm right you just by default it is so I'm not like a pirate that's right that's not like a pirate that's right that's right that's right about so um so do you right that's right about so um so do you think that running spark on top of think that running spark on top of Cassandra and the rest of small Cassandra and the rest of small suitcases ah you know people doing suitcases ah you know people doing industry or it's still necessary in most industry or it's still necessary in most cases to set up a HDFS and remember the cases to set up a HDFS and remember the full system I think a question I mean it full system I think a question I mean it depends what you're trying to optimize depends what you're trying to optimize for um what cassandra is great for is if for um what cassandra is great for is if you really want to have like random you really want to have like random writes and reads book both Cassandra in writes and reads book both Cassandra in space space no sequel store optimized for a huge no sequel store optimized for a huge data and then getting doing point data and then getting doing point lookups that kind of stuff if what you lookups that kind of stuff if what you really care about is top performance for really care about is top performance for scans and doing batch batch data flip of scans and doing batch batch data flip of stuff then I think issue if that's going stuff then I think issue if that's going to be what you want for a while mm-hmm to be what you want for a while mm-hmm oh yeah I mean it's also a little bit oh yeah I mean it's also a little bit less operationally complex to deal with less operationally complex to deal with a no sequel database for just until with a no sequel database for just until with the file system right so see i don't i the file system right so see i don't i don't see i cain't are taking over but don't see i cain't are taking over but they're certainly valid right and the they're certainly valid right and the kind of thing that relates to another kind of thing that relates to another development so we recently CA a passive development so we recently CA a passive link and I am trying the space of link and I am trying the space of streaming systems and today the second streaming systems and today the second talk which would have is on min data talk which would have is on min data which is a storm like very efficient which is a storm like very efficient streaming system so these systems put streaming system so these systems put streaming first on patch second I flink streaming first on patch second I flink make this point that there's a continuum make this point that there's a continuum and batch is just a limit version of a and batch is just a limit version of a stream right so so where are you know a stream right so so where are you know a third of storm and then spark was any third of storm and then spark was any initial page but responded with mini initial page but responded with mini patching now we have language basically patching now we have language basically do instrument first and beige being do instrument first and beige being eliminated right so I wonder right you eliminated right so I wonder right you guys because they're essentially serve guys because they're essentially serve your customers right how do you see the your customers right how do you see the space of all being do you see spark and space of all being do you see spark and flame kind of and similar systems flame kind of and similar systems co-evolving woofers is this kind of co-evolving woofers is this kind of steering first base second base first in steering first base second base first in a second dichotomy is it you know kind a second dichotomy is it you know kind of reduced oh this it's hard to change of reduced oh this it's hard to change for smaller essence or fall fling to for smaller essence or fall fling to achieve the same best performance right achieve the same best performance right i don't think it kind of relates to this i don't think it kind of relates to this edge defensive that because the normal edge defensive that because the normal setup right now she's done gigantic setup right now she's done gigantic amount of stuff into the APIs then read amount of stuff into the APIs then read some subset of it the membrane spark to some subset of it the membrane spark to beijing ready and ghetto streaming comes beijing ready and ghetto streaming comes out of that one in production nice you out of that one in production nice you put it into a stream in context but put it into a stream in context but essentially assume this batch model so i essentially assume this batch model so i think that's kind of why is it rather think that's kind of why is it rather than that right yeah i mean so i think than that right yeah i mean so i think first of all you know giving this to the first of all you know giving this to the continuum is great primarily because it continuum is great primarily because it allows you to have you know allows you to have you know ap is for these two things yeah what ap is for these two things yeah what realize maybe even a while ago if you realize maybe even a while ago if you can create platforms that make you able can create platforms that make you able to use the same code or fashion to use the same code or fashion pre-processing um in terms of support pre-processing um in terms of support for this fling so I would you know for this fling so I would you know always say if you can one system is always say if you can one system is better than two so if it turns out that better than two so if it turns out that spark into everything definitely can do spark into everything definitely can do um having that in the mix you know um having that in the mix you know additional overhead on the other hand if additional overhead on the other hand if it turns out that flink sort of you know it turns out that flink sort of you know this business a long way to go i believe this business a long way to go i believe have both spark and flank to be a full have both spark and flank to be a full training system whatever that means so training system whatever that means so if link is somehow way faster at getting if link is somehow way faster at getting there then the spark but i don't see any there then the spark but i don't see any reason why we wouldn't you know in cdh reason why we wouldn't you know in cdh let's say include that in our platform let's say include that in our platform mm-hmm okay and I guess my wife ambition mm-hmm okay and I guess my wife ambition will be force on Scola because in a way will be force on Scola because in a way of weeks Kyle shop we there's a real of weeks Kyle shop we there's a real difference of Scala and and sabari can difference of Scala and and sabari can and so natural use a scholar of the and so natural use a scholar of the spark and almost everything I will spark and almost everything I will simply play app and akka so ah sure simply play app and akka so ah sure that's correct so and we have this big that's correct so and we have this big dance call conference nagas where we're dance call conference nagas where we're emphasizing end-to-end data pipeline emphasizing end-to-end data pipeline scholar right so basically we say noisy scholar right so basically we say noisy yeah we don't want export and report and yeah we don't want export and report and low the rule to keep it everything in low the rule to keep it everything in very type of manner and move around so I very type of manner and move around so I wonder if you see this is a feasible wonder if you see this is a feasible vision in the industry which is vision in the industry which is experiences with Scala you know can we experiences with Scala you know can we move enough data science from Python to move enough data science from Python to scala what would it take where should we scala what would it take where should we you know ask our community focus on you know ask our community focus on making data science in Scala when I kind making data science in Scala when I kind of the cubicle um yeah great question so of the cubicle um yeah great question so at first used a la maybe a year year and at first used a la maybe a year year and a half ago when i started working on a half ago when i started working on this part i hadn't touched it before um this part i hadn't touched it before um since then I'm a total convert think since then I'm a total convert think it's you know the bee's knees best thing it's you know the bee's knees best thing ever I you know I would love for ever I you know I would love for everyone to use skull for everything everyone to use skull for everything whether that's a realistic goal of whether that's a realistic goal of course is a whole a whole other question course is a whole a whole other question um so I've started developing on top of um so I've started developing on top of spark this time series spark this time series library a lot of statistics library a lot of statistics functionality and stuff to make it easy functionality and stuff to make it easy to do concerts analysis on top of spark to do concerts analysis on top of spark and so that I start building in Scala and so that I start building in Scala and you know the hope there is that we and you know the hope there is that we can add a lot of functionality the can add a lot of functionality the currently exists and let's say MATLAB or currently exists and let's say MATLAB or pine one package it looks that's models pine one package it looks that's models on top of Scala so yeah I would say that on top of Scala so yeah I would say that I'm a low-grade Scala evangelist so I'm a low-grade Scala evangelist so obviously Python an R have a huge obviously Python an R have a huge amounts of momentum for data science I amounts of momentum for data science I mean Java has a huge amount of momentum mean Java has a huge amount of momentum just for Kate engineering stuff yes um just for Kate engineering stuff yes um so we'll see how things pan out in the so we'll see how things pan out in the next five to ten years all right well next five to ten years all right well thank you much we're looking forward thank you much we're looking forward you're talkin the is great to go again you're talkin the is great to go again thanks very much less a