Grant Ingersoll, Q&A with Alexy Khrabrov of SF Text @VigLink 20150210

hello everybody this is SF text this is hello everybody this is SF text this is our first inaugural metop we're on our first inaugural metop we're on location at viglink in San Francisco and location at viglink in San Francisco and uh one of our speakers tonight is gr uh one of our speakers tonight is gr anderol the CTO of lucid Works uh the anderol the CTO of lucid Works uh the company behind uh solar and Lucine and company behind uh solar and Lucine and as uh many of you know Lucine is mostly as uh many of you know Lucine is mostly widely used and deployed open- Source widely used and deployed open- Source search engine uh almost every company search engine uh almost every company which wants to have search on their site which wants to have search on their site has Lucen somewhere uh so uh I want to has Lucen somewhere uh so uh I want to ask uh first of all welcome Grant thank ask uh first of all welcome Grant thank you glad to be here uh it's a I think you glad to be here uh it's a I think search is a common theme in all kind of search is a common theme in all kind of text applications and our general theme text applications and our general theme is text everybody needs to find some is text everybody needs to find some some text there looking for first I want some text there looking for first I want to ask you uh how did you uh come to to ask you uh how did you uh come to Lucine and and solar in the first place Lucine and and solar in the first place and how this interest evolved over and how this interest evolved over several years to to become a company uh several years to to become a company uh what's the trajectory yeah no it's a what's the trajectory yeah no it's a it's a great question I mean I think it's a great question I mean I think you're dead on that Lucine and solar you're dead on that Lucine and solar have grown so much uh my my personal have grown so much uh my my personal history is I started I used to live up history is I started I used to live up in uh Syracuse New York and if anybody's in uh Syracuse New York and if anybody's familiar with that there they know familiar with that there they know Syracuse University which uh Dr Syracuse University which uh Dr Elizabeth lyy who who is a pretty big Elizabeth lyy who who is a pretty big name in the the text in natural language name in the the text in natural language processing field she had a company there processing field she had a company there called text wise which uh happened to called text wise which uh happened to have a job when I was looking for a job have a job when I was looking for a job I had never done text in college or I had never done text in college or anything like that uh got hired to build anything like that uh got hired to build uh a cross language search system uh a cross language search system between English French German Spanish between English French German Spanish and and Japanese uh we used Oracle context Japanese uh we used Oracle context search this is back in like search this is back in like 1999 I believe uh so that's kind of my 1999 I believe uh so that's kind of my first foray into search in NLP Flash first foray into search in NLP Flash Forward uh to 2004 I had left text wise Forward uh to 2004 I had left text wise went off and did some other stuff and went off and did some other stuff and then came back to work for Liz again at then came back to work for Liz again at the University this time in a place the University this time in a place called the center for natural language called the center for natural language processing I was hired to build a processing I was hired to build a English Arabic cross language search English Arabic cross language search system this time you can guess who was system this time you can guess who was paying for that right and uh I walked in paying for that right and uh I walked in the door my boss said hey we use this the door my boss said hey we use this thing called Lucine I said great let me thing called Lucine I said great let me go uh look up what that is and uh go uh look up what that is and uh started playing around with it as I as I started playing around with it as I as I started building out this system started building out this system realized that I needed some things in realized that I needed some things in Lucine that weren't there uh started Lucine that weren't there uh started submitting patches and kind of the rest submitting patches and kind of the rest is history if if you will uh Flash is history if if you will uh Flash Forward to Forward to 2007 founded Lucid Works around the 2007 founded Lucid Works around the premise of commercializing the premise of commercializing the capabilities around solar and Lucine and capabilities around solar and Lucine and providing product and all of that kind providing product and all of that kind of on top of it and of course it's all of on top of it and of course it's all exploded since then as well so a lot of exploded since then as well so a lot of luck uh right place right time and a luck uh right place right time and a bunch of hard work as well but that's bunch of hard work as well but that's great yeah I remember I think I first great yeah I remember I think I first saw your talk at Hado Summit in 2012 and saw your talk at Hado Summit in 2012 and that was already in the business context that was already in the business context and you basically outline big picture and you basically outline big picture and big compon components of big data so and big compon components of big data so it's clearly when people think big data it's clearly when people think big data solar Lucin comes up now and uh so I solar Lucin comes up now and uh so I wonder how do you go from Lucine which wonder how do you go from Lucine which runs on single Noe to solar and what is runs on single Noe to solar and what is the landscape in terms of uh uh the landscape in terms of uh uh distributed systems and how do you see distributed systems and how do you see that area evolve yeah no it's uh it's that area evolve yeah no it's uh it's really been interesting over the years really been interesting over the years because you I think with Lucine you know because you I think with Lucine you know obviously it's just a set of java apis obviously it's just a set of java apis and that provides search functionality and that provides search functionality things like obviously matching queries things like obviously matching queries and doing fuzzy matching and all that and doing fuzzy matching and all that kind of stuff as well as like language kind of stuff as well as like language analyzers and and and then a whole bunch analyzers and and and then a whole bunch of modules that kind of help you build of modules that kind of help you build you know a kit of parts to build a you know a kit of parts to build a Search application Along Came solar and Search application Along Came solar and really kind of said all right well let's really kind of said all right well let's put this in a service layer and let's put this in a service layer and let's make it easy for people to interact with make it easy for people to interact with it in the very early days of solar the it in the very early days of solar the scaling primarily was in a replicated scaling primarily was in a replicated model for query volume because model for query volume because essentially you know nobody really had essentially you know nobody really had really large scale problems other than really large scale problems other than you know the the internet search people you know the the internet search people like Google and and being in the like like Google and and being in the like because they were were and of course because they were were and of course they were building their own uh so you they were building their own uh so you know especially in the e-commerce know especially in the e-commerce environment you know it's just hey I've environment you know it's just hey I've got a lot of queries but I don't have a got a lot of queries but I don't have a big index so let's just replicate it and big index so let's just replicate it and then you know Flash Forward a few more then you know Flash Forward a few more years as hadoop's taking off which you years as hadoop's taking off which you know of course Hadoop started as part as know of course Hadoop started as part as the Lucine project as well uh was okay the Lucine project as well uh was okay well we now need to scale search to well we now need to scale search to match those kinds of needs as well so match those kinds of needs as well so not only do we need to scale for query not only do we need to scale for query volume but we also need to scale for a volume but we also need to scale for a number of documents and so uh you know a number of documents and so uh you know a lot of people had rolled their own using lot of people had rolled their own using solar and Lucine over the years and so solar and Lucine over the years and so you know one of the things that we you know one of the things that we really focused on as a company was let's really focused on as a company was let's make sure that solar just has really make sure that solar just has really good distributed search capabilities out good distributed search capabilities out of the box solar Cloud was born for of the box solar Cloud was born for whatever reason I don't know the naming whatever reason I don't know the naming of it but you know that was kind of the of it but you know that was kind of the way we attacked it added in first way we attacked it added in first distributed search capabilities you had distributed search capabilities you had to do the indexing yourself and then uh to do the indexing yourself and then uh with solar Ford auto uh which was I with solar Ford auto uh which was I don't know a year and a half two years don't know a year and a half two years ago now added the capability to do all ago now added the capability to do all the indexing and all of that as well the indexing and all of that as well such that you know I think I was at a such that you know I think I was at a meet up just recently up at Salesforce meet up just recently up at Salesforce in Seattle I think they announced their in Seattle I think they announced their their solar deployment somewhere 200 300 their solar deployment somewhere 200 300 billion uh documents in it you know so billion uh documents in it you know so people doing really large scale search people doing really large scale search you know across you know hundreds of you know across you know hundreds of billions of documents it's only a matter billions of documents it's only a matter of time we before I'm sure we're doing of time we before I'm sure we're doing trillions uh and so you know you're trillions uh and so you know you're seeing just this massive uptake in and I seeing just this massive uptake in and I think the other interesting thing is think the other interesting thing is people are asking more interesting people are asking more interesting questions of their search engine as well questions of their search engine as well it's not just hey find me keywords and it's not just hey find me keywords and get back results it's bring in you know get back results it's bring in you know all kinds of different data structures all kinds of different data structures numbers spatial enumerated types how do numbers spatial enumerated types how do I slice and dice that and bring me back I slice and dice that and bring me back what's important right ultimately you what's important right ultimately you know shifting from it just being kind of know shifting from it just being kind of traditional keyword search to being all traditional keyword search to being all about what's important in my data yes about what's important in my data yes and then you know shifting through you and then you know shifting through you know using that to then power your know using that to then power your Downstream applications yes this this Downstream applications yes this this actually uh connects very well with one actually uh connects very well with one of the themes of our Meetup because we of the themes of our Meetup because we see text as emerging again as a window see text as emerging again as a window into human intent right the through this into human intent right the through this little box uh and this is how we are little box uh and this is how we are used to do search but now if you go to used to do search but now if you go to Google and you enter sf4 BOS it will Google and you enter sf4 BOS it will give you tickets probably from uh San give you tickets probably from uh San Francisco to Boston uh or more Francisco to Boston uh or more structured data and uh uh essentially structured data and uh uh essentially search Eng ultimate goal is to figure search Eng ultimate goal is to figure out what was our original intent behind out what was our original intent behind the query so here of course I want to the query so here of course I want to ask how we open source people are going ask how we open source people are going to compete with the big guys right to compete with the big guys right because we have we have open source we because we have we have open source we have the community uh but the uh kind of have the community uh but the uh kind of structure data people want to ask uh may structure data people want to ask uh may have semanti content and um how do we have semanti content and um how do we kind of map this uh more uh with kind of map this uh more uh with structure data which you mentioned how structure data which you mentioned how do we kind of uncover more more behind do we kind of uncover more more behind you know more in our documents what do you know more in our documents what do you think is going to happen here in the you think is going to happen here in the open source the next few years wow open source the next few years wow that's a lot of good questions in there that's a lot of good questions in there I mean I think you know obviously I I mean I think you know obviously I can't predict the future but you know can't predict the future but you know what you're seeing in open source I what you're seeing in open source I think in the you know kind of the early think in the you know kind of the early days especially when I started with open days especially when I started with open source open source was primarily focused source open source was primarily focused on on commoditizing what people were seeing in commoditizing what people were seeing in the proprietary landscape or they just the proprietary landscape or they just got tired of building the same thing got tired of building the same thing every time they went to a different every time they went to a different company so they said you know the company so they said you know the Forward Thinking people were able to say Forward Thinking people were able to say let's open source this such that I can let's open source this such that I can carry it with me whereas nowadays I carry it with me whereas nowadays I think you're starting to see just a lot think you're starting to see just a lot more Innovation happening right in open more Innovation happening right in open source out of the box as opposed to source out of the box as opposed to having you know being cooked up in some having you know being cooked up in some lab somewhere it's now being done in lab somewhere it's now being done in open source that being said I think that open source that being said I think that especially on the NLP side of things especially on the NLP side of things it's still a little bit trailing it's still a little bit trailing obviously there's a lot of good new obviously there's a lot of good new projects out there the really the the projects out there the really the the biggest challenge for open source I biggest challenge for open source I think when it comes to n P not so much think when it comes to n P not so much search but NLP is just having the sheer search but NLP is just having the sheer amount of data and being able as an open amount of data and being able as an open source Community to deal with the source Community to deal with the pedabytes of data that can feed into you pedabytes of data that can feed into you know the the really large scale models know the the really large scale models that you're going to build around uh you that you're going to build around uh you know whether that's machine translation know whether that's machine translation or question answering or whatever it is or question answering or whatever it is that takes a lot of effort and a lot of that takes a lot of effort and a lot of compute capabilities and as like uh open compute capabilities and as like uh open source Community you kind of have to beg source Community you kind of have to beg and borrow to get that kind of stuff and borrow to get that kind of stuff together right so those are I think some together right so those are I think some of the big challenges on the open source of the big challenges on the open source side that being said you know I mean side that being said you know I mean there's just a lot of interesting there's just a lot of interesting projects out there with NLP such that projects out there with NLP such that you know for a long time I would always you know for a long time I would always say oh well NLP is great and I love say oh well NLP is great and I love working on it but put it in practice is working on it but put it in practice is really challenging and and you know it really challenging and and you know it demos very well but it doesn't always demos very well but it doesn't always perform very well whereas I think in you perform very well whereas I think in you know say the last I don't know 3 to 5 know say the last I don't know 3 to 5 years maybe even less you're really years maybe even less you're really starting to see really large scale starting to see really large scale practice iCal applications where people practice iCal applications where people are leveraging NLP on a daily basis and are leveraging NLP on a daily basis and it's it's becoming natural whether it's it's becoming natural whether that's on your phone or on a website that's on your phone or on a website whatever you're able to interact with whatever you're able to interact with the machine in a natural way speak the the machine in a natural way speak the language right um the other interesting language right um the other interesting thing I think is uh you know a lot of thing I think is uh you know a lot of people talk your question about people talk your question about structured and unstructured um it's one structured and unstructured um it's one of my pet peeve terms because I actually of my pet peeve terms because I actually hate the word unstructured when it comes hate the word unstructured when it comes to text and the reason why is if you to text and the reason why is if you think about it right text is one of the think about it right text is one of the most highly structured things we have most highly structured things we have right we're just not very good at right we're just not very good at telling the computer what that structure telling the computer what that structure is and and letting the computer take is and and letting the computer take advantage of those right the grammar and advantage of those right the grammar and deal with the grammars and and the deal with the grammars and and the relationships and and all of those kinds relationships and and all of those kinds of things the ambiguity um that's I of things the ambiguity um that's I think one of the cool things that's been think one of the cool things that's been happening as well is more and more happening as well is more and more capabilities are coming around that make capabilities are coming around that make that easier to deal with right and that easier to deal with right and obviously you know more universities are obviously you know more universities are teaching this kind of stuff more people teaching this kind of stuff more people are feeling comfortable dealing with are feeling comfortable dealing with probabilities and statistics as opposed probabilities and statistics as opposed to you know just kind of the pure to you know just kind of the pure relational models that you know probably relational models that you know probably you and I were trained on way back when you and I were trained on way back when we were undergrads uh and so I think you we were undergrads uh and so I think you know as you get that Comfort level know as you get that Comfort level raised as the open source gets better raised as the open source gets better that raises the bar for everybody and that raises the bar for everybody and that's kind of you know in my book that's kind of you know in my book taming text to try to address that kind taming text to try to address that kind of crowd of people who are getting who of crowd of people who are getting who are new to all this stuff how do I start are new to all this stuff how do I start to deal with the fact that I may not get to deal with the fact that I may not get the same answer every time I ask the the same answer every time I ask the question right because the DAT has question right because the DAT has changed or or because there's some fuzzy changed or or because there's some fuzzy fuzziness in there that you know I just fuzziness in there that you know I just I I haven't accounted for uh by the way I I haven't accounted for uh by the way you know uh Tam is a great book actually you know uh Tam is a great book actually Tom Morton and I were in the same PD Tom Morton and I were in the same PD program at pen and kind of he still is program at pen and kind of he still is there uh uh working for Comcast so this there uh uh working for Comcast so this is interesting connection and I actually is interesting connection and I actually enjoyed it uh I yeah I wonder uh it's enjoyed it uh I yeah I wonder uh it's it's a very useful book and it's a good it's a very useful book and it's a good primer uh on on on text processing uh so primer uh on on on text processing uh so I wonder I wonder um um so in terms of scale right uh so I um um so in terms of scale right uh so I was at the uh 2007 machine translation was at the uh 2007 machine translation Summit where Google uh translate talked Summit where Google uh translate talked for the first time and France o for the first time and France o presented the model and he said it's presented the model and he said it's very simple if you want to real like very simple if you want to real like your translation let's just collect your translation let's just collect several trillion words and let's just do several trillion words and let's just do seven grams right and essentially that's seven grams right and essentially that's how how you solve this problem so I how how you solve this problem so I wonder this a small this a small problem wonder this a small this a small problem that's right so I wonder uh given the that's right so I wonder uh given the scale you mentioned at sales force scale you mentioned at sales force deployment uh can I do a seven G model deployment uh can I do a seven G model of several trillion words uh with solar of several trillion words uh with solar Lucin uh uh we seven trillion words in Lucin uh uh we seven trillion words in um I mean I think you can well I mean um I mean I think you can well I mean it's obviously going to take a lot of it's obviously going to take a lot of Hardware there's a lot more involved in Hardware there's a lot more involved in than just the pure like the pure uh than just the pure like the pure uh programming side of that problem in programming side of that problem in solar right um you know there certainly solar right um you know there certainly people who you know you would have very people who you know you would have very very large indexes and and and you know very large indexes and and and you know we've worked really hard in the we've worked really hard in the community to make sure we can scale to community to make sure we can scale to very large sizes can we do 7even very large sizes can we do 7even trillion uh n seven grams uh or a trillion uh n seven grams uh or a trillion s war and um you know give me trillion s war and um you know give me enough Hardware I'll get you there I enough Hardware I'll get you there I don't know right off off hand I would don't know right off off hand I would think we could get in that range though think we could get in that range though that it would be that it would be interesting um you know and certainly interesting um you know and certainly when you bring in tools like Hadoop and when you bring in tools like Hadoop and Spark and and that kind of stuff you Spark and and that kind of stuff you know very complimentary you know go off know very complimentary you know go off and do the really large scale stuff in and do the really large scale stuff in in an approach like that and then you in an approach like that and then you know you're reducing it down into know you're reducing it down into something that is more manageable and something that is more manageable and then use search to kind of access that then use search to kind of access that and and you know query around it and and and and you know query around it and and slice and dice it in different ways in a slice and dice it in different ways in a more realtime approach right but yeah no more realtime approach right but yeah no I mean I don't know do you have a big I mean I don't know do you have a big enough budget for us to do I I'm just enough budget for us to do I I'm just interested in kind of scale because you interested in kind of scale because you know here in the Salesforce size I I know here in the Salesforce size I I think that's very I think it's getting think that's very I think it's getting really there uh where you know uh big really there uh where you know uh big guys are playing uh so uh so I'm I'm guys are playing uh so uh so I'm I'm curious um about your so I know that curious um about your so I know that your technology side is very rigorous your technology side is very rigorous and I was really impressed by the fact and I was really impressed by the fact that you guys are running your own that you guys are running your own Jepson tests right so japson is this Jepson tests right so japson is this distributed uh system uh testing uh Su distributed uh system uh testing uh Su which is basically integration test which is basically integration test which it tests you know all kinds of which it tests you know all kinds of failure scenarios so and and uh uh it failure scenarios so and and uh uh it was done uh as as as a series of of of was done uh as as as a series of of of studies uh but uh uh Lucid Works studies uh but uh uh Lucid Works maintains its own jabson uh test Suite maintains its own jabson uh test Suite so I I wonder and and apparently it so I I wonder and and apparently it shows that you guys are really uh paying shows that you guys are really uh paying attention to to all kinds of fa attention to to all kinds of fa scenaries I wonder how do you uh how do scenaries I wonder how do you uh how do you make sure that the customers are not you make sure that the customers are not losing data and and you know how do you losing data and and you know how do you like what kind of tools do you have and like what kind of tools do you have and and what kind of scenarios do you see and what kind of scenarios do you see around this right well this is you know around this right well this is you know I mean obviously the jeffson work happy I mean obviously the jeffson work happy to get into I mean I think this is one to get into I mean I think this is one of the beauties of the open source is of the beauties of the open source is not just commod commoditization of not just commod commoditization of functionality but commoditization of functionality but commoditization of test across a large number of people test across a large number of people when you're the most widely deployed when you're the most widely deployed search technology on the planet right search technology on the planet right you get a lot of people kicking the you get a lot of people kicking the tires and coming up with error scenarios tires and coming up with error scenarios and all of those kinds of things such and all of those kinds of things such that it gets to be you know you get a that it gets to be you know you get a lot of great feedback from the community lot of great feedback from the community at at the Quality Assurance level so you at at the Quality Assurance level so you know you've got that layer then we as a know you've got that layer then we as a company you know we look at where are company you know we look at where are areas that we can fund efforts around areas that we can fund efforts around making sure that you know we can making sure that you know we can leverage the community stuff and then leverage the community stuff and then feedback our own capabilities the Jepson feedback our own capabilities the Jepson thing uh was definitely an area of thing uh was definitely an area of Interest we wanted to we already knew Interest we wanted to we already knew like solar and solar Cloud was very like solar and solar Cloud was very robust and very mature and you know to robust and very mature and you know to some extent we needed to just simply some extent we needed to just simply prove it by going through you know what prove it by going through you know what Jepson does is simulate uh Network Jepson does is simulate uh Network partitions disc failure Hardware failure partitions disc failure Hardware failure all of these kinds of things in a nice all of these kinds of things in a nice self-contained way such that you can self-contained way such that you can test whether you have data loss because test whether you have data loss because one of the big things you know we have one of the big things you know we have companies who run their whole business companies who run their whole business off of solar right so data loss is a off of solar right so data loss is a huge problem for them if it happens huge problem for them if it happens right and so in fact in where they're right and so in fact in where they're actually using solar as the actually using solar as the authoritative store for their data right authoritative store for their data right so we've got to make sure that it's rock so we've got to make sure that it's rock solid under all conditions now obviously solid under all conditions now obviously you know no software is perfect but so you know no software is perfect but so we want to make sure that we take a lot we want to make sure that we take a lot of you know put in a lot of effort to of you know put in a lot of effort to make sure that we're doing those things make sure that we're doing those things the jebson example is a is a great one the jebson example is a is a great one of uh actually one of our Engineers uh of uh actually one of our Engineers uh you know one of the things that people you know one of the things that people often talk about with with solar cloud often talk about with with solar cloud is oh I've got this extra thing that's is oh I've got this extra thing that's called zookeeper and and that makes me called zookeeper and and that makes me have to do a little bit extra work one have to do a little bit extra work one of the things we wanted to do is show of the things we wanted to do is show why that was such an advantage because why that was such an advantage because okay got this extra process but the okay got this extra process but the thing that you get out of that now is thing that you get out of that now is you know okay it's a little bit harder you know okay it's a little bit harder to start but would you rather have to start but would you rather have harder to or easier to start or easier harder to or easier to start or easier to finish I don't know about you I would to finish I don't know about you I would rather have my weekends when the system rather have my weekends when the system is up and in production and all of that is up and in production and all of that kind of stuff then you know a little kind of stuff then you know a little extra work to set up zookeeper up front extra work to set up zookeeper up front because well by the way it's not that because well by the way it's not that hard anyway so we as a company wanted to hard anyway so we as a company wanted to Showcase why we chose zookeeper as the Showcase why we chose zookeeper as the distributor coordination service for distributor coordination service for solar and show and demonstrate very solar and show and demonstrate very cleanly especially compared to our cleanly especially compared to our competition of how you can avoid things competition of how you can avoid things like split bent brain scenarios Network like split bent brain scenarios Network partitions data loss by making a better partitions data loss by making a better decision about the robust you know about decision about the robust you know about the way you coordinate your distributed the way you coordinate your distributed activities so for people who don't care activities so for people who don't care about data loss then you know you don't about data loss then you know you don't necessarily care about zookeeper and necessarily care about zookeeper and don't get me wrong there are use cases don't get me wrong there are use cases for that you know if it's not your for that you know if it's not your authoritative store if you're just doing authoritative store if you're just doing uh you know some data that you can throw uh you know some data that you can throw away if it goes away or you can just away if it goes away or you can just reindex then you know maybe zookeeper reindex then you know maybe zookeeper isn't your thing but if you're going to isn't your thing but if you're going to run your business if you're doing you run your business if you're doing you know $2 billion dollar a year in know $2 billion dollar a year in e-commerce and you need to handle e-commerce and you need to handle massive scale at Christmas time or the massive scale at Christmas time or the holiday season you can't afford to have holiday season you can't afford to have data loss you can't afford to have the data loss you can't afford to have the kinds of problems that you know kinds of problems that you know zookeeper helps you avoid so gladly pay zookeeper helps you avoid so gladly pay that cost because it's going to save you that cost because it's going to save you you know in the long run on uh it's you know in the long run on uh it's great to hear this I mean uh you know I great to hear this I mean uh you know I uh come from a distributed systems uh come from a distributed systems Community where we use Scola and Spark Community where we use Scola and Spark and a and tools like that so and you and a and tools like that so and you know meses and Aurora of course and all know meses and Aurora of course and all these things and so it comes you know these things and so it comes you know goes without saying what kind of goes without saying what kind of benefits you derive from zuker but benefits you derive from zuker but apparently uh you know it's the apparently uh you know it's the knowledge kind of you know spreading knowledge kind of you know spreading through through communities different through through communities different speed uh so it's it's great to hear that speed uh so it's it's great to hear that you guys you know rely on that this is you guys you know rely on that this is proven technology right which is proven technology right which is designed to actually you know enable designed to actually you know enable this um so uh so you mention spark and this um so uh so you mention spark and you know since I also uh organize SF you know since I also uh organize SF scol mop and organize spark M up uh I scol mop and organize spark M up uh I got to ask about this uh so how do you got to ask about this uh so how do you guys interoperate with uh with Scala and guys interoperate with uh with Scala and Spark and given that spark now runs on Spark and given that spark now runs on top of almost any distributed system uh top of almost any distributed system uh you know should I be able to run spark you know should I be able to run spark on top of solar for instance and open up on top of solar for instance and open up you know a bunch of readers read a bunch you know a bunch of readers read a bunch of stuff in and kind of do something of stuff in and kind of do something there have you been uh listening in on there have you been uh listening in on our meetings at work at Lucid works no I our meetings at work at Lucid works no I mean that's a great question now we just mean that's a great question now we just recently actually published uh solar recently actually published uh solar spark integration it's up on our Lucid spark integration it's up on our Lucid Works GitHub page so if you go just do Works GitHub page so if you go just do Lucid Works GitHub you'll find uh a Lucid Works GitHub you'll find uh a spark solar integration there all open spark solar integration there all open source Apache License uh and so that's source Apache License uh and so that's kind of one level that's essentially kind of one level that's essentially about how do I interact with getting about how do I interact with getting data in and out of solar into spark and data in and out of solar into spark and vice versa right um so that's kind of vice versa right um so that's kind of the one level of thing the other level the one level of thing the other level is in in actually in our commercial uh is in in actually in our commercial uh product called Lucid Works Fusion we are product called Lucid Works Fusion we are in the process of bringing in so we you in the process of bringing in so we you know with Fusion we do a lot of things know with Fusion we do a lot of things about signal capture there things like about signal capture there things like click stream analysis all that and then click stream analysis all that and then turn that back into relevance Fe you turn that back into relevance Fe you know back into relevance gold if you know back into relevance gold if you will so if you take all the click will so if you take all the click information that you're getting then information that you're getting then turn that back around into better search turn that back around into better search results right so we do a lot of that in results right so we do a lot of that in solar right just out of the box and we solar right just out of the box and we can query solar in the right way such can query solar in the right way such that we can do things like that we can do things like recommendation engines right out of recommendation engines right out of solar collab of filtering all that kind solar collab of filtering all that kind of stuff where it then gets interesting of stuff where it then gets interesting is essentially those are all like is essentially those are all like aggregation problems right or building aggregation problems right or building up models and they often require large up models and they often require large scale batch you know style compute this scale batch you know style compute this is where Spark comes in as if in the is where Spark comes in as if in the truly large cases where we have you know truly large cases where we have you know uh you know you're doing millions and uh you know you're doing millions and millions of events an hour or you know a millions of events an hour or you know a day Etc and you get a really large day Etc and you get a really large essentially Matrix spark is a great essentially Matrix spark is a great place to come in and do that machine place to come in and do that machine learning work where you're building a learning work where you're building a model in spark and then you you plug it model in spark and then you you plug it back in Works seamlessly with fusion and back in Works seamlessly with fusion and then you can go and you know query that then you can go and you know query that model so if you're building a model so if you're building a classification model if you're building classification model if you're building a recommender etc we can tie that all in a recommender etc we can tie that all in together and so that's going to that's together and so that's going to that's going to be part of our product here going to be part of our product here very soon as well so so yeah we love very soon as well so so yeah we love spark uh you know we did map ruce for a spark uh you know we did map ruce for a long time saw the shortcomings uh you long time saw the shortcomings uh you know another project I'm involved with know another project I'm involved with Mout uh that's all been switching over Mout uh that's all been switching over to spark and Scala as well I'm not as to spark and Scala as well I'm not as deeply involved there as I once was uh deeply involved there as I once was uh but you know a lot of great work there but you know a lot of great work there about bringing more machine learning about bringing more machine learning capabilities into sculla and into spark capabilities into sculla and into spark uh you know around there's some really uh you know around there's some really nice uh there's a really nice DSL in nice uh there's a really nice DSL in there for building generalized linear there for building generalized linear models now uh right you know leverages models now uh right you know leverages skull underneath the hood and and makes skull underneath the hood and and makes it super easy for people to build out uh it super easy for people to build out uh those kinds of things build your models those kinds of things build your models and that's all spark and scol underneath and that's all spark and scol underneath the hood as I understand it so uh you the hood as I understand it so uh you know I myself am not a scholar person know I myself am not a scholar person but uh you know we we do see it in a lot but uh you know we we do see it in a lot of different places and have some of it of different places and have some of it within the company as well so that's within the company as well so that's great to hear you know because uh I can great to hear you know because uh I can tell you you know kind of a as a tell you you know kind of a as a developer who build systems uh we use developer who build systems uh we use both search and and NLP machine learning both search and and NLP machine learning and extraction I think uh uh solar and and extraction I think uh uh solar and Spark make an ideal combination because Spark make an ideal combination because for instance if I round up of a database for instance if I round up of a database I essentially have to think how I'm I essentially have to think how I'm representing my textual data in the representing my textual data in the database database right and you guys spend a lot of time right and you guys spend a lot of time thinking how you do that so so thinking how you do that so so essentially for a lot of applications essentially for a lot of applications we're going to see uh uh discuss SE text we're going to see uh uh discuss SE text this makes actually a very good this makes actually a very good combination so maybe uh since you combination so maybe uh since you mentioned you guys are developing this mentioned you guys are developing this maybe I can ask you to come back and and maybe I can ask you to come back and and talk about it when you guys are talk about it when you guys are developing it because that's certainly a developing it because that's certainly a great combination of Technologies so in great combination of Technologies so in the last question I want to ask you uh the last question I want to ask you uh so uh the I think you really present so uh the I think you really present interesting path in open source because interesting path in open source because you you you're the guy who interested in you you you're the guy who interested in specific technology Masters it very specific technology Masters it very deeply and specifically with Lucine and deeply and specifically with Lucine and then you you turn it into a successful then you you turn it into a successful business right which is basically used business right which is basically used by by a lot of people all over the world by by a lot of people all over the world so uh what kind of advice do you have to so uh what kind of advice do you have to open source people who who want to do open source people who who want to do the same there is a lot of uh folks the same there is a lot of uh folks especially in the uh kind of uh especially in the uh kind of uh technology area who develop tools uh technology area who develop tools uh what is you know what are the important what is you know what are the important things you have to pay attention to if things you have to pay attention to if you want to you know take an important you want to you know take an important open source technology and make a open source technology and make a business out of it wow uh that's a great business out of it wow uh that's a great question um you know I mean I think question um you know I mean I think there's a lot of lessons learned across there's a lot of lessons learned across so many things you know I mean building so many things you know I mean building a business is so much more than just the a business is so much more than just the technology right uh and open source as a technology right uh and open source as a business model has a lot of great things business model has a lot of great things and a lot of challenges as well right in and a lot of challenges as well right in terms of you know you've got uh you know terms of you know you've got uh you know for us it was interesting like uh very for us it was interesting like uh very early on you know we were like uh let's early on you know we were like uh let's just go we can just go put out our our just go we can just go put out our our welcome mat and people come to us for welcome mat and people come to us for solar support and you know the brake fix solar support and you know the brake fix hey I need insurance for instance off of hey I need insurance for instance off of my for my search engine right and we my for my search engine right and we still do a lot of that business but still do a lot of that business but what's interesting is that solar doesn't what's interesting is that solar doesn't break and so people are like hey that's break and so people are like hey that's great we love solar but you know come great we love solar but you know come renewal time hey I already know and it renewal time hey I already know and it works and I haven't had any trouble and works and I haven't had any trouble and so one of the interesting things that so one of the interesting things that we've evolved into is like you've got to we've evolved into is like you've got to have uh capabilities on top of that so have uh capabilities on top of that so when we think about how we engage with when we think about how we engage with our customers we've shifted very much to our customers we've shifted very much to what we call a customer success model what we call a customer success model which is hey you know we're going to be which is hey you know we're going to be there for the break fix that's one thing there for the break fix that's one thing but in reality what customers kept but in reality what customers kept coming to us for is help me understand coming to us for is help me understand the difficult problems better you guys the difficult problems better you guys are expert experts in search you know are expert experts in search you know machine learning you KN know natural machine learning you KN know natural language processing help me with product language processing help me with product in terms of tools that make my life in terms of tools that make my life easier to do those things but then also easier to do those things but then also hey how do I make this better how do I hey how do I make this better how do I have better relevance for have better relevance for how do I scale this how do I grow with how do I scale this how do I grow with my business how do I ask more my business how do I ask more interesting questions of my data and so interesting questions of my data and so it it what the customer success model it it what the customer success model that we've put in place you know like I that we've put in place you know like I said obviously has The Brak fixed but is said obviously has The Brak fixed but is also really geared towards making sure also really geared towards making sure customers can grow and be successful customers can grow and be successful together with us so you know open source together with us so you know open source is one part of the business decision is is one part of the business decision is should I put my technology out in the should I put my technology out in the open and and you know everybody's going open and and you know everybody's going to be different there uh but you know I to be different there uh but you know I think one of the things that you have to think one of the things that you have to think about is you know is this think about is you know is this technology something that has a big technology something that has a big enough base that I can actually build a enough base that I can actually build a company around obviously you know it's company around obviously you know it's got to be more than just your pet got to be more than just your pet project or you and your buddy's pet project or you and your buddy's pet project uh and it's not enough just just project uh and it's not enough just just show up and say okay well hey I've got show up and say okay well hey I've got uh you know kind of Open Source kind of uh you know kind of Open Source kind of goes through these phases of you know goes through these phases of you know it's a couple of guys or or a couple of it's a couple of guys or or a couple of people together working on something people together working on something that's of interesting to interest to that's of interesting to interest to them and then it usually evolves to them and then it usually evolves to where you can kind of support an where you can kind of support an ecosystem of one-off Consultants who can ecosystem of one-off Consultants who can come and say hey I can help you with come and say hey I can help you with your hard problem and then you know it your hard problem and then you know it gets to the likes of the dupes and the gets to the likes of the dupes and the Sparks and the the solar Etc where you Sparks and the the solar Etc where you can sustain multiple businesses around can sustain multiple businesses around it and you know and and they all it and you know and and they all contribute back to the core but they all contribute back to the core but they all have you know presumably their unique have you know presumably their unique differentiator and I think you know differentiator and I think you know depending on where that project is and depending on where that project is and its lifespan will determine how you can its lifespan will determine how you can go about uh commercializing uh open- go about uh commercializing uh open- Source technology um you know the other Source technology um you know the other thing is like you know you just always thing is like you know you just always have to keep investing in the open have to keep investing in the open source and there's always this kind of source and there's always this kind of delicate Balancing Act of you know at delicate Balancing Act of you know at the end of the day you have to you know the end of the day you have to you know quote unquote put food on the table you quote unquote put food on the table you have to pay salaries you got to keep the have to pay salaries you got to keep the lights on Etc so there's something that lights on Etc so there's something that you have to be uh you know that you have you have to be uh you know that you have to be able to monetize but at the same to be able to monetize but at the same time you know the community pressure is time you know the community pressure is often how do I you know I want all of often how do I you know I want all of this for free they want your time for this for free they want your time for free they want your code for free Etc so free they want your code for free Etc so like you have to kind of work together like you have to kind of work together to come up with a win-win situation for to come up with a win-win situation for everybody such that the community is everybody such that the community is benefiting as well as you know you're benefiting as well as you know you're able to uh to pay salaries and all that able to uh to pay salaries and all that kind good balance between contributions kind good balance between contributions from the outside community and the from the outside community and the company employees and so like for us you company employees and so like for us you know as a company we look at how uh we know as a company we look at how uh we can complement and extend solar we still can complement and extend solar we still invest a lot in core solar I think we're invest a lot in core solar I think we're the single largest contributor or two the single largest contributor or two solar in the community uh but then we solar in the community uh but then we also look at from a product standpoint also look at from a product standpoint is what things do we know customers want is what things do we know customers want that the community doesn't do so for that the community doesn't do so for instance you know solar itself doesn't instance you know solar itself doesn't do connectors to you know crawl the web do connectors to you know crawl the web or connect to Twitter or you know or connect to Twitter or you know SharePoint or whatever file system SharePoint or whatever file system repository there is out there so that's repository there is out there so that's a gap that we fill as a company right or a gap that we fill as a company right or solar is primarily made up of developers solar is primarily made up of developers who are Focus on the back end you know who are Focus on the back end you know algorithms and and core search algorithms and and core search functionality they don't really care functionality they don't really care about search UI so we come in and bring about search UI so we come in and bring user interfaces to the customers that user interfaces to the customers that allow them to work with Solar without allow them to work with Solar without having to necessarily write Java code having to necessarily write Java code right they probably care about the U but right they probably care about the U but not about building the US exactly and not about building the US exactly and then we you know like and then like the then we you know like and then like the the other part of how we are with solar the other part of how we are with solar really is how do we ask more interesting really is how do we ask more interesting questions of solar so we want to drive questions of solar so we want to drive more data into solar more data into solar such that we can ask more interesting such that we can ask more interesting questions of it so again it's kind of questions of it so again it's kind of that complement and extend model of Open that complement and extend model of Open Source if you will that I think uh you Source if you will that I think uh you know has worked well for us and and you know has worked well for us and and you know we're continuing to invest a lot in know we're continuing to invest a lot in in that way of doing things oh that's in that way of doing things oh that's great I think