sfspark.org: Alexy Khrabrov interviews Frank Nothaft

hello everybody I'm Alexa crabber the hello everybody I'm Alexa crabber the organizer of sf's park in France and organizer of sf's park in France and here we are allocation chartboost and we here we are allocation chartboost and we have a very special meet up tonight it's have a very special meet up tonight it's called open genomics skallon spark for called open genomics skallon spark for good how open source developers can help good how open source developers can help in genomics research and cancer research in genomics research and cancer research and it's a very important topic coming and it's a very important topic coming out in a personalized medicine we have out in a personalized medicine we have two great talks today from employed in two great talks today from employed in Berkeley and from driver genomics and Berkeley and from driver genomics and here we'll also have frank not huffed here we'll also have frank not huffed who is a PhD candidate at my pad who is a PhD candidate at my pad berkeley hello Frank it's great to have berkeley hello Frank it's great to have you great to be here so I think Adam and you great to be here so I think Adam and spark set up as the project from a blob spark set up as the project from a blob it's really well known by this point in it's really well known by this point in time and the community and so you're one time and the community and so you're one of the folks directly working this so of the folks directly working this so maybe you can tell us a little bit about maybe you can tell us a little bit about that project sure sure so you know we've that project sure sure so you know we've been working on the atom project for been working on the atom project for about two and a half years right now and about two and a half years right now and the atom project is an attempt to use the atom project is an attempt to use the spark and the Hadoop stack to try to the spark and the Hadoop stack to try to process genomic data in a faster easier process genomic data in a faster easier to work with way so you know when we to work with way so you know when we when we started looking at the problem when we started looking at the problem most of the tools that are out there in most of the tools that are out there in genomics right now are focused on a genomics right now are focused on a single node and they are you know single node and they are you know they're based heavily around some of the they're based heavily around some of the legacy file formats and legacy legacy file formats and legacy technologies that were very common when technologies that were very common when the next generation sequencing error the next generation sequencing error really started getting kicked off you really started getting kicked off you know two thousand six two thousand eight know two thousand six two thousand eight uh you know since then you know one of uh you know since then you know one of the big realizations that we had is the big realizations that we had is actually this this huge growth in actually this this huge growth in genomic sequencing has really paralleled genomic sequencing has really paralleled how you know the rise of all these how you know the rise of all these high-performance analytic technologies high-performance analytic technologies came up in the you know in this big data came up in the you know in this big data space so we saw a very good opportunity space so we saw a very good opportunity to go ahead and take some of the tools to go ahead and take some of the tools that were you know coming out on the that were you know coming out on the scene whether it's a tool like spark you scene whether it's a tool like spark you know know I'll format like Park hey you know data I'll format like Park hey you know data serialization method like Avro to just serialization method like Avro to just go ahead and make it easier to process go ahead and make it easier to process large amounts of genomic data at a high large amounts of genomic data at a high rate of speed hmm ah you know so over rate of speed hmm ah you know so over the past two years what we've been the past two years what we've been working on is trying to essentially working on is trying to essentially replicate and improve on some of the replicate and improve on some of the current standard practice in genomics so current standard practice in genomics so we've been trying to tackle some of the we've been trying to tackle some of the big problems that people have in the big problems that people have in the data management side of genomics you data management side of genomics you know whether it's making sure that the know whether it's making sure that the data can be easily processed by a wide data can be easily processed by a wide variety of tools can be easily accessed variety of tools can be easily accessed queried you know one of the big problems queried you know one of the big problems that we've run into is that the legacy that we've run into is that the legacy tools aren't very easy for people to run tools aren't very easy for people to run ad hoc queries about against so you know ad hoc queries about against so you know there's there's kind of this meme where there's there's kind of this meme where if I if I'm going to run some ad hoc if I if I'm going to run some ad hoc genomic experiment you know I'm going to genomic experiment you know I'm going to be doing this through command line tools be doing this through command line tools like awk grep said so on and so forth like awk grep said so on and so forth that's all right if you just want to that's all right if you just want to look at a single person's genome but you look at a single person's genome but you know when I'm scaling up to tens to know when I'm scaling up to tens to hundreds thousands of genomes it just hundreds thousands of genomes it just doesn't work well so you know we've been doesn't work well so you know we've been we've been working heavily on these we've been working heavily on these pipelines we're starting to focus more pipelines we're starting to focus more on the ad hoc kind of the data science on the ad hoc kind of the data science side of genomics so we're pretty excited side of genomics so we're pretty excited about where you know where the about where you know where the technology's heading so this is a technology's heading so this is a mention seven or cry because I think ten mention seven or cry because I think ten years ago I'm in like a previous year so years ago I'm in like a previous year so cut these books coming out from areli cut these books coming out from areli another publisher is no pearl football another publisher is no pearl football for my expired and photographer my for my expired and photographer my physics right so it's kind of physics right so it's kind of interesting maybe you can explain to interesting maybe you can explain to kind of folks who are not familiar with kind of folks who are not familiar with specific technologists why spark is a specific technologists why spark is a better vehicle for this kind of problems better vehicle for this kind of problems than you know 70 can Perl and Python than you know 70 can Perl and Python sure sure well you know there's a few sure sure well you know there's a few things and the the big first thing is if things and the the big first thing is if you look at the computational patterns you look at the computational patterns that are in play you know you have these that are in play you know you have these tasks that are processing very large tasks that are processing very large amounts of data you know a single human amounts of data you know a single human genome in you know non summarize and genome in you know non summarize and it's just raw as form is about 200 it's just raw as form is about 200 gigabytes of data mm-hmm I you know when gigabytes of data mm-hmm I you know when you start talking about these large file you start talking about these large file sizes moving to a parallel system like sizes moving to a parallel system like spark that's really optimized for making spark that's really optimized for making optimal use of data locality optimal use of data locality high-performance file systems is just high-performance file systems is just you know it's it's a good step in the you know it's it's a good step in the right direction right direction um additionally you know one of the big um additionally you know one of the big reasons that we think spark is a good reasons that we think spark is a good fit for this less so then you know fit for this less so then you know legacy Hadoop was is that a lot of these legacy Hadoop was is that a lot of these pipelines rely heavily on iteration no pipelines rely heavily on iteration no they may be a lot of the algorithms that they may be a lot of the algorithms that people are running are similar to people are running are similar to machine learning algorithms you know you machine learning algorithms you know you have lots of iterative algorithms but have lots of iterative algorithms but additionally even the non machine additionally even the non machine learning like algorithms tend to rely on learning like algorithms tend to rely on you know a long pipeline that is you you know a long pipeline that is you know job cascaded after job cascaded know job cascaded after job cascaded after jocks three matching right then after jocks three matching right then things like that exactly and it's you things like that exactly and it's you know that's a good fit for spark know that's a good fit for spark additionally the you know one of the big additionally the you know one of the big advantages that we see you know over advantages that we see you know over over a lot of these legacy tools where over a lot of these legacy tools where you know people are going ahead and you know people are going ahead and they're piping tools through command they're piping tools through command lines and so on and so forth is we've lines and so on and so forth is we've taken an approach that focuses a lot on taken an approach that focuses a lot on having a you know a schema as a having a you know a schema as a descriptor of data instead of some descriptor of data instead of some binary or text-based format mm-hmm and binary or text-based format mm-hmm and there's there's a few advantages to this there's there's a few advantages to this you know right now you know I can take you know right now you know I can take this you know I can take this ad hoc this you know I can take this ad hoc query that I'm running on the command query that I'm running on the command line and I can go ahead and I can move line and I can go ahead and I can move it to a tool like sparks equal or Impala it to a tool like sparks equal or Impala and I can run my you know I can run my and I can run my you know I can run my data analysis queries in that tool in a data analysis queries in that tool in a higher-level language then you know higher-level language then you know someone who's just piping to your piping someone who's just piping to your piping things around on the command line yes things around on the command line yes yes and that really reminds me right to yes and that really reminds me right to one of the themes of the upcoming it's one of the themes of the upcoming it's called by the bay conference we call it called by the bay conference we call it no et al right an etl basically means no et al right an etl basically means you dump something in the text and then you dump something in the text and then you par or par so you lose that the you par or par so you lose that the schema you lose the typing information schema you lose the typing information so your bicycle it's interesting that so your bicycle it's interesting that you're focusing the same advantage that you're focusing the same advantage that they are keeping the the information in they are keeping the the information in between the stages yeah definitely and I between the stages yeah definitely and I know that you use an average Bar K for know that you use an average Bar K for this purpose mm-hmm exactly and you know this purpose mm-hmm exactly and you know we've been doing a lot of optimizations we've been doing a lot of optimizations recently to you know that try to improve recently to you know that try to improve our performance you know with those our performance you know with those technologies you know there's a variety technologies you know there's a variety of yo we got very good performance right of yo we got very good performance right now but we found a couple of you know now but we found a couple of you know actually a glow hanging fruit that allow actually a glow hanging fruit that allow us to optimize things even further huh us to optimize things even further huh interesting I hope to hear more about it interesting I hope to hear more about it so one question that in a lot of so one question that in a lot of developers in this community ask developers in this community ask especially bc huge infuse especially bc huge infuse I'm about you know the tonight's event I'm about you know the tonight's event will have more than 200 people signed up will have more than 200 people signed up people comes with different companies people comes with different companies and the first reaction when we share and the first reaction when we share this will share this at night Ron the this will share this at night Ron the shares are the company says basically shares are the company says basically how we can help right so people even how we can help right so people even note in in biotech space 12 for instance note in in biotech space 12 for instance enable their purse or developers to work enable their purse or developers to work on this right they want to contribute on this right they want to contribute and obviously this is not as easy as and obviously this is not as easy as kind of picking up a how-to for a linux kind of picking up a how-to for a linux project right you you need to learn more project right you you need to learn more so since you've been dealing with this so since you've been dealing with this area for a long time can you summarize area for a long time can you summarize what is a good way to for somebody who what is a good way to for somebody who is open-source developer who is is open-source developer who is enthusiastic about that but does not enthusiastic about that but does not much biology right what's a good way to much biology right what's a good way to learn more about the space about the learn more about the space about the problem and maybe kind of then kind of problem and maybe kind of then kind of understand what algorithms need understand what algorithms need improvement how they can happen with the improvement how they can happen with the whole area sure sure that's that's a whole area sure sure that's that's a great question and you know I know that great question and you know I know that that's something that I found kind of that's something that I found kind of difficult when I came into the field difficult when I came into the field myself you know so we've been trying to myself you know so we've been trying to do some work in the you know in the atom do some work in the you know in the atom project try to reach out do more more project try to reach out do more more developer facing documentation you know developer facing documentation you know one of my colleagues we lost our one of my colleagues we lost our syndrome cut from cloudera had a chapter syndrome cut from cloudera had a chapter in a recent O'Reilly book on you know in a recent O'Reilly book on you know how to use how do you spark to do data how to use how do you spark to do data science he had a chapter talking about science he had a chapter talking about you know basic introduction to genomics you know basic introduction to genomics and showing how this maps into a spark and showing how this maps into a spark based pipeline my colleague Massey has a based pipeline my colleague Massey has a similar similar chapter coming out in a similar similar chapter coming out in a Hadoop focused book mm-hmm so we're you Hadoop focused book mm-hmm so we're you know we're working hard to try to put know we're working hard to try to put together some examples there we're together some examples there we're trying to put together a tutorial trying to put together a tutorial article as well you know otherwise I article as well you know otherwise I think you know part of the part of the think you know part of the part of the problem can just start with problem can just start with understanding where the data is coming understanding where the data is coming from and what some of the big challenges from and what some of the big challenges are um you know it's kind of daunting are um you know it's kind of daunting the literature in the field is you know the literature in the field is you know its vast and wide but one of the things its vast and wide but one of the things that is good in in biology is that we do that is good in in biology is that we do have a very large you know a large have a very large you know a large collection of review collection of review articles these are articles that try to articles these are articles that try to give more background you know on give more background you know on specific topic specific analysis needs specific topic specific analysis needs that try to give the history of the that try to give the history of the problem and current approaches to solve problem and current approaches to solve it mm-hmm so there's a variety of good it mm-hmm so there's a variety of good articles there that that are you know articles there that that are you know great for people are coming up in the great for people are coming up in the field the other thing and I think one of field the other thing and I think one of the you know one of the areas that's the you know one of the areas that's very good and actually you know an area very good and actually you know an area that I kind of eased myself in is that I that I kind of eased myself in is that I you know even if you're a computational you know even if you're a computational person who doesn't yet understand the person who doesn't yet understand the algorithms you know a lot of you a lot algorithms you know a lot of you a lot of how you can get started maybe around of how you can get started maybe around optimizing some of the performance you optimizing some of the performance you know for for a lot of us you know if you know for for a lot of us you know if you come from the data engineering side you come from the data engineering side you know you're used to running big Hadoop know you're used to running big Hadoop pipelines you understand you know how to pipelines you understand you know how to how to tweak the performance and how to tweak the performance and optimize the performance of these large optimize the performance of these large long-running you know analytics jobs and long-running you know analytics jobs and we've actually had several people who've we've actually had several people who've gotten into the project by saying hey gotten into the project by saying hey you know I kind of understand what this you know I kind of understand what this algorithm is doing kind of vaguely I'm algorithm is doing kind of vaguely I'm gonna measure its performance try to gonna measure its performance try to make it a bit faster and then you know make it a bit faster and then you know three months later they wind up you know three months later they wind up you know really understanding how everything really understanding how everything works in the system what some of these works in the system what some of these important algorithms are what the data important algorithms are what the data types look like mm-hmm um so you know types look like mm-hmm um so you know it's a bit more computationally friendly it's a bit more computationally friendly you know it's it's a type of thing you you know it's it's a type of thing you put your computer hat on you start to put your computer hat on you start to make some you know modifications the make some you know modifications the code base to see if you can improve code base to see if you can improve performance and just by doing that you performance and just by doing that you know you can understand a lot about know you can understand a lot about what's going on in the system no so what's going on in the system no so you're basically start kind of no way you're basically start kind of no way using your skills in optimization and using your skills in optimization and then can expand more into into the then can expand more into into the domain space and actual brings me to the domain space and actual brings me to the question what got interested you in like question what got interested you in like why you you know got into this field and why you you know got into this field and you know kind of what kind of people I you know kind of what kind of people I would say by generalization can be would say by generalization can be interested in in the biology problems interested in in the biology problems sure sure well you know the the big sure sure well you know the the big thing for me that that got me excited thing for me that that got me excited about it is just it's this opportunity about it is just it's this opportunity to really apply computational to really apply computational computational technologies to something computational technologies to something that's a greater social good mm-hmm um that's a greater social good mm-hmm um you know that's that's what something you know that's that's what something that really got me inspired about it I that really got me inspired about it I you know I was looking for you know I was looking for Dana get really start playing around Dana get really start playing around more with a big data stack came across more with a big data stack came across this application domain dived in and this application domain dived in and haven't regretted it since I you know I haven't regretted it since I you know I think one of the great things about think one of the great things about about this kind of quantitative area of about this kind of quantitative area of biology is that it's it's somewhere biology is that it's it's somewhere where a wide variety of people can come where a wide variety of people can come in and start working on problems you in and start working on problems you know if you're if you're a strong know if you're if you're a strong developer you know we always need people developer you know we always need people who can go through and put together who can go through and put together these algorithms get them running fast these algorithms get them running fast processing large amounts of data if processing large amounts of data if you're someone who has more of an you're someone who has more of an analytical mind you know more of an analytical mind you know more of an algorithms backgrounds a lot of these algorithms backgrounds a lot of these algorithms that we're using in genomics algorithms that we're using in genomics are fundamentally you know they're based are fundamentally you know they're based on really core theoretical computer on really core theoretical computer science you know maybe they're based on science you know maybe they're based on graph theoretical algorithms you know graph theoretical algorithms you know you're doing some sort of string edit you're doing some sort of string edit distance comparison mm-hmm so that's distance comparison mm-hmm so that's another area and then fundamentally a another area and then fundamentally a lot of what we're doing is some some lot of what we're doing is some some form of statistical inference yes so you form of statistical inference yes so you know we have this huge community in the know we have this huge community in the bay area that's developed this huge data bay area that's developed this huge data science community a lot of that is some science community a lot of that is some form of you know statistical inference form of you know statistical inference machine learning and a lot of those machine learning and a lot of those skills naturally map very well to the skills naturally map very well to the problems that people are tackling and problems that people are tackling and biodata jr. of course that's like all biodata jr. of course that's like all this things together exactly exactly this things together exactly exactly interesting so you know I'll probably interesting so you know I'll probably rip out by kind of hypothesizing um so rip out by kind of hypothesizing um so let's see you know personal medicine let's see you know personal medicine offices is gaining momentum let's say offices is gaining momentum let's say magically you have a thousand developers magically you have a thousand developers your disposal heretical building of your disposal heretical building of top-level skull and spiral developers top-level skull and spiral developers who are you know eager to help right and who are you know eager to help right and through the community effort effectively through the community effort effectively you have you know part-time have some of you have you know part-time have some of the spare time instead of building the spare time instead of building another pet store a little bit rails another pet store a little bit rails they will help with Adam right and they will help with Adam right and hopefully we'll get enough tutorials hopefully we'll get enough tutorials enough resources to explain them enough resources to explain them basically the space right so so in you basically the space right so so in you can put them on any tasks in this can put them on any tasks in this pipeline right so what would you do this pipeline right so what would you do this force where would you direct the effort force where would you direct the effort where you call the attention you know where you call the attention you know what should they do sure that's that's a what should they do sure that's that's a really interesting question and one you really interesting question and one you know one that I've been thinking about a know one that I've been thinking about a lot recently and you know what's kind of lot recently and you know what's kind of interesting is one of the one of the big interesting is one of the one of the big problems in genomics problems in genomics is a problem that anybody who works in a is a problem that anybody who works in a organization that has a big data lake or organization that has a big data lake or you know big data silos can can relate you know big data silos can can relate to and it's dealing with discovering to and it's dealing with discovering large data sets and figuring out how to large data sets and figuring out how to aggregate across those data sets well aggregate across those data sets well you know a lot of the algorithms that you know a lot of the algorithms that that we've been working on the first two that we've been working on the first two years of Adam have been focused on some years of Adam have been focused on some of these initial you know initial of these initial you know initial discoveries that you want to make you discoveries that you want to make you know you want to detect oh you know how know you want to detect oh you know how is your genome and my genome difference is your genome and my genome difference right so we've spent a lot of time in right so we've spent a lot of time in the last few years working on that but the last few years working on that but in the future what I'd like to see you in the future what I'd like to see you no more focus on is saying okay well you no more focus on is saying okay well you know now that I understand why you know know now that I understand why you know how your genome and my genome are how your genome and my genome are different how does that influence you different how does that influence you know how the biology of our bodies works know how the biology of our bodies works differently you know why does that differently you know why does that influence some of my personal disease influence some of my personal disease risk factors how does it change what risk factors how does it change what traits the two of us have and all of traits the two of us have and all of this fundamentally is a statistical this fundamentally is a statistical process that depends on having very process that depends on having very large data sets mmm um you know like large data sets mmm um you know like especially if you look there some especially if you look there some studies that have looked at for example studies that have looked at for example schizophrenia autism Alzheimer's disease schizophrenia autism Alzheimer's disease there you know if you're able to go there you know if you're able to go ahead and and look at you know not ten ahead and and look at you know not ten people's genomes not a hundred people's people's genomes not a hundred people's genomes but ten thousand people's genomes but ten thousand people's genomes and have a much better genomes and have a much better understanding of what genomic links understanding of what genomic links there are between that disease yo there are between that disease yo essentially what links there are between essentially what links there are between that disease and the genomics mm-hmm but that disease and the genomics mm-hmm but you know one of the problems right now you know one of the problems right now is even though you know even though we is even though you know even though we can build these algorithms to do these can build these algorithms to do these sorts of aggregations it's actually very sorts of aggregations it's actually very difficult to manage the data sharing you difficult to manage the data sharing you know and a lot of this is for good know and a lot of this is for good reason you know fundamentally whenever reason you know fundamentally whenever you're collecting information about you're collecting information about someone's genome or someone's you know someone's genome or someone's you know someone's health status someone's someone's health status someone's familial health there's a lot of familial health there's a lot of important personally identifiable important personally identifiable information there yes so we see a lot of information there yes so we see a lot of challenges coming up and you know kind challenges coming up and you know kind of one in the area of you know if I'm a of one in the area of you know if I'm a if I'm a doctor if I'm some clinician if I'm a doctor if I'm some clinician how can I discover who else has a cohort how can I discover who else has a cohort of Alzheimer's patients who else has of Alzheimer's patients who else has looked at people with myeloid you know looked at people with myeloid you know acute myeloid leukemia mm-hmm but then acute myeloid leukemia mm-hmm but then once they once they gone ahead and done that and that's gone ahead and done that and that's fundamentally a metadata search problem fundamentally a metadata search problem it's a search and indexing problem it's a search and indexing problem something that as computational people something that as computational people have thought a very long time about it have thought a very long time about it gets into this question of how can we do gets into this question of how can we do analytics in a way that you know manages analytics in a way that you know manages these privacy concerns right and I and I these privacy concerns right and I and I think that's a it's an interesting think that's a it's an interesting problem I don't think it's been looked problem I don't think it's been looked at a ton in the analytic space but I at a ton in the analytic space but I think it's a very natural extension on think it's a very natural extension on top of what people have doing there have top of what people have doing there have been doing in the analytic space yes I been doing in the analytic space yes I think the privacy preserve and data think the privacy preserve and data mining I think it's a whole field right mining I think it's a whole field right obviously you know with the volume of obviously you know with the volume of data tool you need to improve specialist data tool you need to improve specialist algorithms get better matter that algorithms get better matter that reminds me a problem we'll add one more reminds me a problem we'll add one more question to this so you mentioned question to this so you mentioned basically you know there are these tasks basically you know there are these tasks and then we'll multiple bodies obviously and then we'll multiple bodies obviously you know privacy is government or you know privacy is government or regulation but if the database and regulation but if the database and maintained by consortium so and for maintained by consortium so and for those who are not on the field can you those who are not on the field can you maybe briefly outline and what are the maybe briefly outline and what are the other force is what are the key groups other force is what are the key groups and we can supply the links to this and we can supply the links to this video all right but you know what are video all right but you know what are the key bodies in the accademia what are the key bodies in the accademia what are the key body is he kind of big data the key body is he kind of big data genomics that you think it's useful for genomics that you think it's useful for for folks to follow to learn more about for folks to follow to learn more about Brad because there is a lot of Sanders Brad because there is a lot of Sanders where this knowledge has been kept a lot where this knowledge has been kept a lot of different organizations and companies of different organizations and companies which basically want to solve these which basically want to solve these problems sure sure well you know there's problems sure sure well you know there's a there's a variety of variety of groups a there's a variety of variety of groups out there you know there's there's out there you know there's there's obviously our our kind of large open obviously our our kind of large open source group yes there's also a similar source group yes there's also a similar open source effort that's coming out of open source effort that's coming out of the genomics England project this is a the genomics England project this is a project to go ahead and sequence a project to go ahead and sequence a hundred thousand people in the UK hundred thousand people in the UK they're developing a similar open open they're developing a similar open open source stack called open open sea be so source stack called open open sea be so that's that's another resource for that's that's another resource for people to look at you know there's a people to look at you know there's a variety of other places you know there's variety of other places you know there's it's kind of hard to hard to say a it's kind of hard to hard to say a single place but there's a lot of single place but there's a lot of interesting a lot of interesting interesting a lot of interesting discussion on the privacy and data discussion on the privacy and data sharing side coming out of the Global sharing side coming out of the Global Alliance for genomics and health yes Alliance for genomics and health yes that's that's a big one and then that's that's a big one and then you know the one of the wonderful things you know the one of the wonderful things with with the field of genomics is that with with the field of genomics is that there's so many so many great people there's so many so many great people doing such valuable work that it's kind doing such valuable work that it's kind it's kind of hard to point to little it's kind of hard to point to little centers of excellence yeah but you know centers of excellence yeah but you know I know there's a there's a a burgeoning I know there's a there's a a burgeoning community that's really kind of coming community that's really kind of coming up in the in the open source and up in the in the open source and scalable tools for genomics world so I scalable tools for genomics world so I think you'll start to see more projects think you'll start to see more projects starting to starting to come up to speed starting to starting to come up to speed over the next year okay so looking over the next year okay so looking forward and for you guys will have more forward and for you guys will have more resources that the new committee side resources that the new committee side called open genomics at i/o which were called open genomics at i/o which were putting together to aggregate this putting together to aggregate this resource will have more links and and resource will have more links and and will definitely follow and playoffs will definitely follow and playoffs progress and you know hope to have you progress and you know hope to have you know Frank's research link from their know Frank's research link from their past present and future and looking past present and future and looking forward to your talk today great well forward to your talk today great well thank you for having me thanks right