0:05
What's up, Brian? How's it going? Hi.
Search this video
0:07
Yeah, Brian and I both work on InferenceX, so it's called Inference Ice, all right, cuz we got sued by HBO Max.
0:12
Just kidding, that's a joke.
0:12
But yeah, I guess like let's just kind of talk through like what it is and like what we've been working on and and kind of its influence, I guess.
0:21
Today, you know, you kind of saw, you know, Jensen go and talk about it and declare himself the inference king.
0:28
I guess what does it mean to you to kind of be the inference king and like have the the highest throughput chip?
0:36
Like what What do you think is important about that in like today's kind I mean, yeah, Jensen's probably pretty pretty proud.
0:42
Uh across all of our benchmarks, Nvidia chips like get the highest scores, the lowest cost for each make per token, etc.
0:51
So it's really uh the king of inference, I guess.
0:57
I guess like Why do you think that I think a lot of people, right, like on on X and stuff say, "Oh, well, like Dylan and SemiAnalysis are just like, you know, kind of like praising Nvidia."
1:06
But like explain to people how this is truly like an independent third-party benchmark and it's not like other other suites where like we have like rules and stuff and people can hill climb.
1:17
Like kind of explain your thoughts on that.
1:21
Yeah, so I guess the biggest part that is that we are open source.
1:23
I mean, if there are people who think that AMD can do better, they have they can feel free to just submit runs or submit configs their configs and we can run it for them.
1:35
Uh but so far all the data has been showing that Nvidia does in fact surpass AMD in almost all aspects. Yeah.
1:41
Yeah, I think that's I think that's a good point.
1:45
Like the idea of InferenceX is sort of to provide a benchmark such that when people are hill climbing it, it's making the entire like ML community better and more efficient and stuff.
1:58
So they they allow I'd say like to add to your point, right?
2:01
You know, it's completely open source and anyone is welcome to kind of like contribute to it and just make the entire ecosystem better.
2:06
So yeah, we we we always want to be like really transparent about that, but yeah, I think I think that makes it even more like prominent when a when a chip provider can be like, "Yeah, we have the the fastest chip." Yeah. Yeah.
2:19
Actually, I actually I want to know more about like what's it like working with AMD and Nvidia.
2:23
Because like you've been here longer than me, so how how's the experience actually?
2:28
Yeah, actually it's been really good and like InferenceX would not be possible without their engineers.
2:34
Like they do they do a lot of the tuning of the configs and stuff.
2:39
And yeah, I think it's it's been really a joy to work with a lot of them.
2:42
Um you know, and yeah, it's been it's been really great.
2:47
Um I guess I guess we can talk about So like what do you think about the LPU stuff, right?
2:52
Cuz that's like kind of like a hot topic right now and like it directly kind of ties into like the Pareto frontier of that we show in InferenceX.
3:00
Like what are your thoughts on that just in general?
3:01
Oh yeah, Jensen did just announce that earlier, I guess.
3:05
I mean, Nvidia does knows the most about its customers.
3:08
Like of course the customers would directly ask Nvidia for stuff that would improve their their inference performance, so I guess in a way disaggregated inference is the future and to really optimize the each specific part of disaggregated inference is that this LPU does in fact uh surpass GPU in some aspects and that's why Nvidia developed it, already acquired Groq and uh use it instead. Yeah.
3:36
Yeah, I think it's really interesting that instead of just like using the the Groq rack to do inference like cross all curve, they're really just leaning into the the strengths of Groq and just saying like, "Okay, well, I can do really fast interactivity and and we're just going to do that like co-package with with the Rubin system."
3:57
And like I wonder if like you think we'll see that from other alternative chips like Matrox and and others or do you think they will this be a continuing trend?
4:08
I mean, maybe not with Nvidia specifically, but It may be interesting to see.
4:11
The idea is that like Amazon is using uh Nvidia 72 along with their own chips for disaggregated inference.
4:18
Like there's really a lot of space to optimize each part, especially since it is disaggregated. Yeah, yeah, I agree.
4:24
And especially with like new new architectures like obviously Mo picture of is a new, but as we see new new architectures, right?
4:32
Like I think inference really like Jensen was talking about today, there's a lot of opportunity for optimizations and like you can really disaggregate a lot of the the inference parts.
4:42
So I think I'm really excited to see like like what what comes up and what we can actually showcase on um on InferenceX, right?
4:49
Uh really I hope we can get like some alternative chip providers on it. Of course. Of course.
4:56
Yeah, and the trend is towards I guess towards like everyone's trying to develop their own chips.
5:01
Uh OpenAI has that a tight ending development and some of the trends are also like regarding the model specs.
5:07
We are seeing MOE models being more sparse with smaller experts.
5:14
And of course this plays into YEP.
5:14
So it's quite interesting to see that uh the researchers who are planning model architectures actually care more about inference optimizations.
5:26
Uh I used to think that this was only a deep sick thing.
5:28
Like only deep sick really cared about this uh inference optimizations, but yeah, seems like uh inference performance is getting more and more important and is uh being considered along with the model's like actual performance. Right.
5:42
Wait, so tell me tell tell me what what plans you have for InferenceX.
5:46
Yeah, yeah, that's that's at V3.
5:47
That's that's a good good question.
5:49
So yeah, right now with InferenceX, right, we're we're using random data for for all of our our benchmarks, which is actually a quite good for a baseline benchmark, right?
6:00
So when you think about it, um the way I like to explain it is in a way right now the benchmarks are kind of showing like the worst the chips will actually perform, right?
6:09
Because it's just a baseline.
6:10
There's no prefix caching uh because we don't do multi-turn data sets, it's all random.
6:16
And then speculative decoding acceptance rates are quite low cuz it's the random data.
6:20
data. Uh but for InferenceX V3, you know, we want to add agentic benchmarks like really really good like a representative multi-turn QA chat benchmarks where there are a ton of client sessions each with, you know,
6:33
multiple turns and we'll enable prefix caching and um like to your point, right, like Nvidia is beating AMD across the whole curve like like you said our most parts, but um like AMD MI355X versus B200, MI355 has one and a half times the HBM band capacity. So with prefix caching, right,
6:50
So with prefix caching, right, like we'd expect it to to do to do better possibly.
6:55
So I think it will help showcase uh the chips from a more objective Yeah, nuanced I was going to say like perspective, right, and like capturing all the all of the uh all the parts of each chip, but yeah, I think actually making any a good agentic benchmark is something that we're realizing is quite difficult and something that hasn't really been done publicly yet.
7:20
So that that's a lot of work, but yeah. Oh, yeah.
7:25
And I've heard like the doing GDC after about these or like or events I've been speaking to people and it seems like KV cache uh offloading and KV cache discount strategies are getting more and more popular. Yeah, yeah.
7:39
KV cache offloading is something we've been investigating a lot.
7:43
Like vLLM just uh basically in in vLLM 0.
7:46
16, I I think they made native KV cache offloading.
7:51
You have things like LM cache, uh distributed offloading like with LMD and stuff.
7:55
And I think you know, you know, we were talking to a lot of people that hacked Pong the other day like doing this as their project and stuff.
8:00
And yeah, I I think I think I mean, at the end of the day when you you when when you want long context inference, you need a big KV cache and you also need that to be extremely fast.
8:10
So there's not really enough room just in HBM.
8:11
You need to offload it somewhere.
8:13
Uh so yeah, I mean I don't know.
8:17
Like what do you what do you think about this topic?
8:18
Like do you think like basically the future is just going to be like how fast we can offload KV cache, how much we can offload?
8:25
Or do you think it's bigger HBM or what do you think?
8:28
I mean I'm very interested To be honest, I'm very interested to what see what the frontier labs are actually doing, but I mean no one knows and open source is probably behind those labs by like uh half a year maybe at best.
8:41
Yeah, but Yeah, I mean, that's a good point cuz like Opus 4.
8:44
6, you know, 1 million context is the the default now, right?
8:49
So like how are they serving like hundreds of thousands of users with a million context?
8:53
Like it's quite quite interesting.
8:56
So yeah, I mean I don't know.
8:58
There there are a lot of uh interesting topics to cover and like the goal of InferenceX, I think uh uh both of us have this like policy where we just want to make the most real-world benchmark possible and we don't want to force people to use certain uh optimizations and we just want to showcase like the the optimal Pareto frontier.
9:17
So whether that's using uh you know, AFD or uh you know, like KV cache offloading, like we just we just want to show it the most honest way possible.
9:29
Yeah, I think I think that about wraps up unless you have anything else to say. Yeah. So >> That's it.
9:33
I think anyone listening to this podcast should definitely uh check out inferencex.
9:38
com to see all these results for free, beautiful visualization.
9:41
We got a great front-end developer, Alex. Shout out Alex. Yeah. And yeah, that's it.
9:45
Thanks Thanks for talking, Brian. Appreciate it.