sftext.org: Alexy Khrabrov interviews Ilya Kreymer about Internet Archiving

hello everybody I'm Alexi crabber of the hello everybody I'm Alexi crabber of the organizer of SF text meet up and hear organizer of SF text meet up and hear our location with Ilya Kramer our our location with Ilya Kramer our speaker today who talked about wayback speaker today who talked about wayback machine common crawl old web and his machine common crawl old web and his basically master for having the way the basically master for having the way the web looked at some point in time and web looked at some point in time and we're going to talk with Ilya about why we're going to talk with Ilya about why is this interesting Chloe up it's great is this interesting Chloe up it's great to have you here great to be here thank to have you here great to be here thank you for having me speak today so so you for having me speak today so so fascinating talk one question is why is fascinating talk one question is why is this interesting and why is this this interesting and why is this interesting to you well it is interesting to you well it is interesting to me and I think in general interesting to me and I think in general because so much so much of what happens because so much so much of what happens in our in our lives happens on the web in our in our lives happens on the web hmm and things on the web disappear hmm and things on the web disappear people don't realize that it may it may people don't realize that it may it may seem that that the web is permanent but seem that that the web is permanent but really it isn't and URLs and content on really it isn't and URLs and content on the web disappears all the time I don't the web disappears all the time I don't have stats on me but in a slide I had a have stats on me but in a slide I had a other informaton how it so it's sort of other informaton how it so it's sort of part of history it's I think that it's part of history it's I think that it's an incredibly important thing to do and an incredibly important thing to do and what i'm doing i think is building new what i'm doing i think is building new technology to try to keep up with with technology to try to keep up with with preserving the web as the web evolved so preserving the web as the web evolved so a lot of the tools that have been built a lot of the tools that have been built to preserve the web have know many years to preserve the web have know many years ago and the web is constantly changing ago and the web is constantly changing at a really fast pace and so new tools at a really fast pace and so new tools need to be built to in order to do keep need to be built to in order to do keep up with with the changes in the web and up with with the changes in the web and at the same time you know Charcot has at the same time you know Charcot has done a great job of saving the early web done a great job of saving the early web and I'm also interested in kind of and I'm also interested in kind of making sure that that's preserved using making sure that that's preserved using old browsers and that we can sort of old browsers and that we can sort of remember remember how the web was and then study it and how the web was and then study it and and refer back to it using the and refer back to it using the environments as close as we can to the environments as close as we can to the environments that existed in ok and so environments that existed in ok and so maybe you can explain to folks who are maybe you can explain to folks who are new to this as you mentioned multiple new to this as you mentioned multiple things which are horizontal under the things which are horizontal under the graph is a company's nonprofit conquer graph is a company's nonprofit conquer all is nonprofit we're back machine is all is nonprofit we're back machine is is a tool which internet archive was is a tool which internet archive was using so can we kind of explain we using so can we kind of explain we should these things are kind of should these things are kind of different entities and how historically different entities and how historically they appeared one after the other and they appeared one after the other and kind of what this field looking like kind of what this field looking like today well and yes and I think that the today well and yes and I think that the project I'm also working with a nine project I'm also working with a nine nonprofit called rhizome dot org and nonprofit called rhizome dot org and then they're a non-profit committed to then they're a non-profit committed to art preservation or digital art art preservation or digital art preservation and in German current preservation and in German current preservation and so Weber quarter is preservation and so Weber quarter is also a nonprofit project and I think also a nonprofit project and I think that's probably not a coincidence that's probably not a coincidence perhaps because I think that perhaps perhaps because I think that perhaps preservation of its well it'd be great preservation of its well it'd be great if it was a perhaps a for-profit if it was a perhaps a for-profit commercial entity i think that the commercial entity i think that the nonprofit aspect of it makes it more nonprofit aspect of it makes it more durable in a way and more more perhaps durable in a way and more more perhaps more difficult but also more durable in more difficult but also more durable in that will see that this is an that will see that this is an organization that is trying to preserve organization that is trying to preserve things for the longer term rather than things for the longer term rather than make a new product and you know make a make a new product and you know make a bunch of money in the short term with bunch of money in the short term with for investors for example and so I think for investors for example and so I think that the nonprofit approach definitely that the nonprofit approach definitely makes sense for longer term digital makes sense for longer term digital preservation and then perhaps is the preservation and then perhaps is the best approach I don't know if yeah then best approach I don't know if yeah then that that's sort of my feeling about it that that's sort of my feeling about it right now is that nonprofits are a great right now is that nonprofits are a great option for for for software development option for for for software development also I and in this in this field in also I and in this in this field in particular mhm and I guess my question particular mhm and I guess my question is also why do new organizations appear is also why do new organizations appear because the original profit so it would because the original profit so it would be logical for them to all unite but you be logical for them to all unite but you know we see Internet Archive then the know we see Internet Archive then the Sekhem on crawl right let me see this Sekhem on crawl right let me see this new generation so what kind of new generation so what kind of as kind of the creation of new archival as kind of the creation of new archival organizations well I think everyone's organizations well I think everyone's you know I think it's good that there's you know I think it's good that there's multiple organizations that are I don't multiple organizations that are I don't think that there should be just one think that there should be just one archive that that's dangerous you know archive that that's dangerous you know what if that archive but what if that what if that archive but what if that archive isn't is inaccessible so I think archive isn't is inaccessible so I think the more archiving organizations there the more archiving organizations there are the better enter archives been are the better enter archives been around for a long time they've been around for a long time they've been crawling the web and also other other crawling the web and also other other content that they have an archive of content that they have an archive of many different things available to to many different things available to to the public some things and then common the public some things and then common crawl for example is more focused i crawl for example is more focused i think on perhaps more more for think on perhaps more more for researchers in terms of crawling data researchers in terms of crawling data not necessarily full web sites but not necessarily full web sites but crawling data and making it publicly crawling data and making it publicly available for researchers hmmm and what available for researchers hmmm and what I'm trying to do with web reporter is I'm trying to do with web reporter is kind of distribute all of this so that kind of distribute all of this so that rather than have one one big archive I rather than have one one big archive I would like to build tools that enable would like to build tools that enable many smaller archives and that perhaps many smaller archives and that perhaps different than than than what the entire different than than than what the entire archives mission is in some sense so yes archives mission is in some sense so yes I think it's really excitable do you I think it's really excitable do you describe because we're recorder lets describe because we're recorder lets individuals see if that their own individuals see if that their own histories like and I think this is the histories like and I think this is the only feasible way to solve this dynamic only feasible way to solve this dynamic content problem right so because I think content problem right so because I think a few years ago there was this question a few years ago there was this question was behind the ? right because there are was behind the ? right because there are the parameters which can figure a URL to the parameters which can figure a URL to basically be an API to a database you basically be an API to a database you cannot crawl the database and so now cannot crawl the database and so now what's happening basically everything what's happening basically everything you know Facebook is just giant database you know Facebook is just giant database and you cannot crawl it and and so and and you cannot crawl it and and so and most JavaScript apps become single page most JavaScript apps become single page apps which basically produce content apps which basically produce content based on very clicking so it almost based on very clicking so it almost sounds like you cannot really have the sounds like you cannot really have the web but you can archive some snapshots web but you can archive some snapshots right and that'll be kind of discretized right and that'll be kind of discretized version of the web right right that's version of the web right right that's definitely yeah that that's that's a definitely yeah that that's that's a fundamental problem is that you only see fundamental problem is that you only see the client-side not not the server side the client-side not not the server side however there is a lot of stuff however there is a lot of stuff happening on the client side so as as happening on the client side so as as more client-side applications become more client-side applications become more sophisticated you know right now more sophisticated you know right now focusing on one specifically sort of focusing on one specifically sort of archiving HTTP traffic but you could archiving HTTP traffic but you could also archive JavaScript events and then also archive JavaScript events and then figure out ways to archive the figure out ways to archive the client-side more effectively as well client-side more effectively as well although it is true that that and although it is true that that and sometimes you're still creating sometimes you're still creating snapshots as far as the the server snapshots as far as the the server status is is concerned I think that status is is concerned I think that there's more more opportunity to to there's more more opportunity to to archive the client side as well mm-hmm archive the client side as well mm-hmm so what do you think about multimedia so what do you think about multimedia contents obviously a lot of these contents obviously a lot of these websites and radio stations and TV websites and radio stations and TV stations and so are you trying to stations and so are you trying to capture the sound and video or is it up capture the sound and video or is it up to the sites like YouTube which can to the sites like YouTube which can actually host them save the videos what actually host them save the videos what do we do about this mom text data well I do we do about this mom text data well I mean the idea with with web recorders mean the idea with with web recorders that it allows users to archive anything that it allows users to archive anything that that's you know what what their that that's you know what what their personal browsing experience know is so personal browsing experience know is so it was owned including set the sound and it was owned including set the sound and video so if you are browsing a YouTube video so if you are browsing a YouTube video it should be able to yeah I mean video it should be able to yeah I mean it should be able to our archive that it should be able to our archive that and that's that's up to that particular and that's that's up to that particular user as as with anything Allison in the user as as with anything Allison in the in their browser and of course you know in their browser and of course you know YouTube is certainly not should not be YouTube is certainly not should not be regarded as an archive in any way regarded as an archive in any way because YouTube videos get removed all because YouTube videos get removed all the time right for a variety of reasons the time right for a variety of reasons some valid some less valid and so if a some valid some less valid and so if a user needs to have an archive I mean the user needs to have an archive I mean the user could be a museum for example or user could be a museum for example or more or a library that I would like to more or a library that I would like to have an important video that was have an important video that was uploaded to YouTube now if YouTube uploaded to YouTube now if YouTube besides that that they take it down then besides that that they take it down then that video is lost to everyone including that video is lost to everyone including a memory institution that that should a memory institution that that should have a copy of it and so with with tools have a copy of it and so with with tools like Weber quarter like Weber quarter it gives institutions and individuals it gives institutions and individuals the ability to archive what they what the ability to archive what they what they see is important to them so they see is important to them so basically you say that I'm owning my basically you say that I'm owning my information consumption experience and information consumption experience and because you know I you know it was a because you know I you know it was a time in my life when I was looking this time in my life when I was looking this V at this video I all the time and have V at this video I all the time and have a record of this regardless of what you a record of this regardless of what you know external world is doing the source know external world is doing the source of this video I kind of in my private of this video I kind of in my private space I should be able to enjoy space I should be able to enjoy basically right like replay of that so basically right like replay of that so this is their interest of cop caption this is their interest of cop caption personal time so I'm just you know kind personal time so I'm just you know kind of just good to go back like it like it of just good to go back like it like it took a little bit about your career took a little bit about your career right because it's not a very common right because it's not a very common thing to join a non-profit or having the thing to join a non-profit or having the web in Silicon Valley weather is like web in Silicon Valley weather is like you know million start-up opportunities you know million start-up opportunities so can talk a little bit about you know so can talk a little bit about you know how I got involved with computers the how I got involved with computers the internet on kind of how did you arrive internet on kind of how did you arrive at this occupation well I've been a long at this occupation well I've been a long time ago I worked in video games that time ago I worked in video games that that was actually the first field i that was actually the first field i started in and I worked in video games started in and I worked in video games for a number of years and I still have for a number of years and I still have many friends in that industry and that's many friends in that industry and that's a then I decided to take a break from a then I decided to take a break from from some programming and I I didn't from some programming and I I didn't actually focused on the focus on playing actually focused on the focus on playing accordion for a number of years oh I accordion for a number of years oh I took a break and but then I felt I kind took a break and but then I felt I kind of decided that it was you know my of decided that it was you know my savings right ready at and that I was savings right ready at and that I was time to to look for to do something that time to to look for to do something that I found to be meaningful and I had a I found to be meaningful and I had a friend who ever she wasn't here tonight friend who ever she wasn't here tonight but a list a good friend of mine who I but a list a good friend of mine who I played music without who works at the played music without who works at the entire archive and he told me that they entire archive and he told me that they were hiring and psycho well I've always were hiring and psycho well I've always known about the Wayback Machine I known about the Wayback Machine I thought that there was a really awesome thought that there was a really awesome tool wouldn't be cool to work on the way tool wouldn't be cool to work on the way back machine back machine and until I applied and that's how i and until I applied and that's how i started working there in in 2011 this is started working there in in 2011 this is a serendipity yes and then meeting the a serendipity yes and then meeting the right people yes so it's sort of a right people yes so it's sort of a combination yeah it's an interesting combination yeah it's an interesting stuff I can't help asking you about the stuff I can't help asking you about the accordion because I entertained the idea accordion because I entertained the idea like I i I'd like to eat the idea of like I i I'd like to eat the idea of knowing how to play the accordion how knowing how to play the accordion how hard it's actually learned playing the hard it's actually learned playing the corner uh well with with practice it corner uh well with with practice it becomes easier like with anything else becomes easier like with anything else does it play a little bit of the piano does it play a little bit of the piano yeah if you play the piano that makes it yeah if you play the piano that makes it a lot easier but yeah I mean it that a lot easier but yeah I mean it that that helped me definitely with the with that helped me definitely with the with the right hand it's pretty much about the right hand it's pretty much about the same in the left hand is I think a the same in the left hand is I think a bit simpler even in the piano so I bit simpler even in the piano so I although lately i've been i've been so although lately i've been i've been so busy with web record i I'm playing as busy with web record i I'm playing as much so I need to get need to find much so I need to get need to find another balance you know and play some another balance you know and play some more interesting yeah so we know we kind more interesting yeah so we know we kind of try to combine art with data science of try to combine art with data science that our dated by the by events so we that our dated by the by events so we know it's really great to see musicians know it's really great to see musicians kind of a data scientist kind of the kind of a data scientist kind of the same person so we like who always would same person so we like who always would be well you know happy to see you play be well you know happy to see you play accordion that you know and talk give it accordion that you know and talk give it a can you talk like that that's a a can you talk like that that's a fantastic combination oh thank you I fantastic combination oh thank you I guess thank you again for for having me guess thank you again for for having me here yeah I think it's a yeah and I'm here yeah I think it's a yeah and I'm happy to have been able to present here happy to have been able to present here those hugely entertaining and very those hugely entertaining and very interesting I think like a lot of you interesting I think like a lot of you know all the data scientists and kind of know all the data scientists and kind of all the kind of text miners they need all the kind of text miners they need this data right like we have this record this data right like we have this record now we can go play with it so I think now we can go play with it so I think that's really give us a lot of his eye that's really give us a lot of his eye so thanks again and we will you know so thanks again and we will you know happily curvy all in all talk in the happily curvy all in all talk in the future great well thank you thanks yeah future great well thank you thanks yeah yeah thank you you