Scale By the Bay 2023: Interview with Nick Hill.

my name is Nick Hill I work for IBM my name is Nick Hill I work for IBM research I work on infrastructure for research I work on infrastructure for serving uh large language models so I don't work on the models so I don't work on the applications as such I work more on the applications as such I work more on the the the actual platform um so there AI the the actual platform um so there AI I'm supporting all of the other sort of I'm supporting all of the other sort of AI applications that are built on top um AI applications that are built on top um really looking to maximize the uh use of really looking to maximize the uh use of the the gpus and get as many um requests the the gpus and get as many um requests through as we can for a small footprint so I talked a little bit about footprint so I talked a little bit about this in in the talk I just gave but uh this in in the talk I just gave but uh there's a lot of change in this space of there's a lot of change in this space of uh llm inferencing right now like every uh llm inferencing right now like every other week it seems there's a new other week it seems there's a new project coming out and they're project coming out and they're incorporating the techniques of other incorporating the techniques of other projects and adding their own and saying projects and adding their own and saying they're much faster so um our sort of they're much faster so um our sort of main I guess ongoing project is to try main I guess ongoing project is to try and keep tabs and all these things and and keep tabs and all these things and figure out okay which ones can we uh figure out okay which ones can we uh leverage and where do we sort of add our leverage and where do we sort of add our own things in um so yeah really you know own things in um so yeah really you know looking at uh how do we keep up and um looking at uh how do we keep up and um and which which Tech Technologies do we and which which Tech Technologies do we choose out of this set that that keeps changing I really like the the keynote changing I really like the the keynote from Chris latner um I think the uh uh from Chris latner um I think the uh uh the new the work he's doing with the MOJ the new the work he's doing with the MOJ language is um is really interesting and language is um is really interesting and yeah looking forward to seeing how that yeah looking forward to seeing how that that pans out might be something that we that pans out might be something that we can use in our work uh at some point um can use in our work uh at some point um and uh I'm also interested in yeah other and uh I'm also interested in yeah other talks on llms there's one about pytorch talks on llms there's one about pytorch 2.1 new features tomorrow which we use a 2.1 new features tomorrow which we use a lot of those um and actually I uh quite lot of those um and actually I uh quite interested in some of the rust talks interested in some of the rust talks because I've been using rust a lot more because I've been using rust a lot more uh lately I'd say it has a different feel lately I'd say it has a different feel to a lot of other conferences it's sort to a lot of other conferences it's sort of much more sort of friendly developer of much more sort of friendly developer focused uh I really like the way it's a focused uh I really like the way it's a mix of different uh subject so like I mix of different uh subject so like I mentioned rust there's a you know rust mentioned rust there's a you know rust representation also distributed systems representation also distributed systems is something I'm involved in also AI is something I'm involved in also AI obviously so it's like the combination obviously so it's like the combination of these things I think is uh it's of these things I think is uh it's really nice to you know be able to see really nice to you know be able to see talks on each and somewhere they talks on each and somewhere they combined so