0:06
All right, we're here at GTC electrifying energy.
Search this video
0:10
All day there's incredible announcements, people to meet, and all night what did you do all night? Waleed. Um, I stayed home.
0:15
I was whipping with three different Claude code sections. No, no, no, I'm kidding.
0:20
Of course, obviously a bunch of fun stuff happening.
0:24
Well, so Waleed, you're you're here.
0:25
Tell me about your company, what are you working on? Yeah. Um, so we're Macaw. Formerly Mako. Uh, now we're in Macaw.
0:32
But no, we're >> So so you you had to change the name, but you also changed what you're working on. A little bit about that.
0:37
>> is moving up the stack.
0:37
So what we had been doing, you know, previously was um, a lot of work on improving quality of kernel generation and doing a lot of work on validating kernels that you do generate.
0:46
Uh, these days you can actually get pretty nice kernels out of Claude code and Codex and everything.
0:53
And I think the entire Silicon Valley, like 80% of the startups here are starting to rethink everything now that coding has come this far.
0:58
I remember when Sonnet came out, like Sonnet 3, when it for when we first saw models that were actually really good.
1:05
And I remember people were talking about they're saying, "Look, all right, it does front end, it does this, but does it do this the complicated stuff? Does it do kernels? Does it do compilers?"
1:14
And it turns out the answer is yes.
1:14
I think the funnest thing about the Opus 46 announcement was like make a C compiler.
1:20
And then it spent like 20k and it made a C compiler.
1:23
[laughter] I was like, "Wow."
1:24
No, and you can like nitpick and be like, "Oh, well, there's this bug, there's that bug, there's this bug." But I mean, come on. Yeah.
1:30
>> Like you kind of just like told it what to do.
1:31
Um, so what we've been doing at Macaw is you know, it kind of forces you to get into like the product aspect.
1:39
Like what do the people whose job it is to, you know, make things run fast?
1:41
What does that person do throughout their day?
1:44
And how does it change now with Claude code?
1:46
And how can you build a platform that makes their life easier?
1:50
So so before stepping into what you're doing now, I actually want to ask a bit more about the code gen side, right?
1:56
Everyone's so excited about code gen, it's been something that people have been investing in for, you know, some time.
2:03
Um, there's a numerous startups now in the space.
2:06
Why are you moving out of it?
2:08
I I actually don't think we're we're not moving out of it is the thing.
2:10
So we still a core part of this is that we use LLMs to write high performance low level code.
2:17
Um, and what we've done is the platform that we're building, we want Claude to get better.
2:22
We want Codex to get better.
2:24
We should use that as a tailwind.
2:25
And then how do you build the rest of a platform around this kind of a capability that you can just keep assuming is going to get better.
2:30
The point is that you cannot um, have code generation itself be your product.
2:34
I think we're going to see a lot of basically giving the taking the code gen capability and then making these kind of end-to-end platforms that just do a lot of the work for you.
2:48
Um, and what what what goes into this end-to-end platform then?
2:50
Is it is it you're making the most of validation engines or what exactly do you do?
2:53
>> That's definitely part of it and that's a really important part cuz again, you have to have that feedback.
2:56
But the kernel code gen we started with input PyTorch get kernel.
3:00
I think the next step is going to be input a Hugging Face link, get full VLLM deployment with whatever kernels you need in there.
3:08
And then like let's think about InferenceX for a second.
3:12
Well, I think the obvious next step for what's probably happening with you guys, you see this Pareto curve where there's a bunch of different curves.
3:19
Now, the question for everybody else is like, "I want the good curve. How do I get that?
3:22
How do I get that for my model, my use case, which is maybe not the exact one that's on here?"
3:26
And so I think a big one is going to be "Here are the GPUs that I have or here is the cloud access I have.
3:32
Here is the model I want.
3:34
Here are the SLAs that my customers expect.
3:36
What's the best way to deploy this thing?"
3:39
And that's going to need um, you know, a lot of testing, a lot of infrastructure, a lot of like hardware to figure out what that Pareto curve is for that specific use case.
3:48
>> That's that's actually something that we're doing in InferenceX, right?
3:51
>> Right, cuz the natural next step, I would expect that, yeah.
3:52
At least uh, initially we were three models cuz we were trying to implement things like evals and implement things like YDP, disagregated prefill decode.
4:00
We're implementing all these features.
4:02
Uh, but over the last month roughly, we've added we've triple I think we've tripled number of models or doubled number of models.
4:10
Um, and those have been like day zero, we got the Pareto curve basically just by telling Claude to um, generate all the different configs, right?
4:19
Um, pretty simple and then cut all the configs that make zero sense to run. Right?
4:23
Like you're not going to run PP8.
4:26
Uh, you're going to pipeline it eight times or you're not going to do certain configs that just don't make any sense.
4:30
So then then getting it to first from first principles just eliminate uh, a lot of the configs.
4:34
And then there's there's configs that you actually don't know the answer because like, well, this one theoretically is better from a roofline basis, but maybe the kernels aren't optimized there yet or whatever.
4:43
And then and then and then we run those and then we have the Pareto curve.
4:46
Um, without using too much compute and able to implement a new model at least in the current leading VLLM actually laying.
4:54
It's still very manual engineer in the loop process, but I can I can see what you mean.
4:57
Like that that's going to be something that's much sooner to have. Yeah.
5:01
And I mean, I feel like every 2 weeks like the limit of what is being automated changes, right?
5:07
So I I totally think that, you know, everything is going to end up being automated at some point, right?
5:10
So what's next for InferenceX on carriers?
5:14
Are you guys going to actually do the dynamic submit a model and all runs?
5:15
Um, I mean, generally we accept PRs.
5:20
If someone wants to submit a PR for a model, great.
5:24
Um, the goal is to do day zero support of models now.
5:27
>> Now, what's the GPU spend on that?
5:27
Uh, >> Or just capacity on there?
5:30
Thankfully, we have great partners that give us free GPUs including CoreWeave, Microsoft, um, Oracle, Nebius, Crusoe, um, TensorWave, AMD, Nvidia, and I'm going to kill my There's one I'm missing. I know I'm missing one.
5:47
Uh, but we have great partners that give us compute for free because this is a free open source project.
5:51
We actually make money off of InferenceX.
5:53
Um, obviously there's a lot of learnings that we get out of We then turn into other products, but um, InferenceX, I mean, we're going to have day zero support for all the new models.
6:03
Um, we're adding more and more evals cuz it turns out people's kernels cheat. Yeah. >> Right? Um, >> Yeah, evals, man.
6:08
I'm telling you, validation engineering.
6:10
These things are faster, but it's like we only implemented GSM8K so far, but even in GSM8K you see a 10% swing even on different points on a curve for the same hardware. >> Really?
6:22
>> Because something they did on the curve, something they did on the kernels, something they did somewhere >> kernels.
6:27
I thought you meant like the full exact same, but just Yeah, it's it's it's a long the sweep, right?
6:30
So when you sweep everything like the kernels for this optimization point that's great at throughput or this one for latency actually has different performance. Yeah.
6:39
>> Um, which which is sort of like makes me wonder like you know, people are always like, "Oh, OpenAI downgraded GPT.
6:45
Oh, Anthropic downgraded GPT."
6:48
Often times it's in the in the they're like, "It's because they've gotten so much more traffic."
6:54
And it's like, "Well, actually perhaps there's some kernel screwery here." >> Yeah.
6:58
Do you guys use open source models? No.
7:03
But you use open source models.
7:04
Everything on InferenceX is open source.
7:06
We love open source models.
7:09
But like stop being poor, just pay Claude. Yeah.
7:11
Pay Anthropic, pay daddy.
7:13
No, we we've done a bunch of RL, um, on some of the open source We've done it on GPT-5, too.
7:19
Um, but yeah, the closed source models are a step ahead. Yeah. Exactly.
7:21
It's It's like 3 months difference in model timing or 6 months difference in model comp capabilities ends up being a >> Do you think that is true?
7:31
>> year or more in terms of like when you're able to release a project and then get the traction and get the customer feedback and, you know, this this iterative cycle.
7:37
So it's like you know, and this is why we spend everyone I encourage everyone to use fast mode even though it breaks the bank. It's like use fast mode.
7:43
Yeah, more tokens per second is more more intelligence.
7:47
>> Yeah, more intelligence per second.
7:47
More intelligent per second is more revenue eventually even though it's mostly just a cost.
7:54
>> Yeah, sounds like Jensen, man. Buy more to save more.
7:57
>> Well, actually he sounds like me. Oh.
7:59
Yeah, but he sounds like Jensen, of course.
8:00
Did you see the I'm just saying. I'm just saying.
8:02
I thought the Apple was good.
8:04
Dude, he spent 5 minutes on our slide, said my name.
8:07
He only said two people's names in the entire keynote, mine and the open cloud bro.
8:11
And now the open cloud bro deserves it.
8:12
I don't know why he said my name.
8:15
>> [laughter] >> Yeah, Jensen called me.
8:16
I told him not this time, maybe next. Yeah, yeah.
8:17
It it it it was it was I knew it was going to happen.
8:22
I was I was I was unbothered.
8:23
Didn't matter to me, you know. >> Yeah, it was cool. No, seriously though. Um, big things, man. Yeah, big things.
8:28
I'm excited to hear what you guys are going to do forward in the future.
8:31
>> Yeah, we'll announce the fund raising tokens and then you'll uh, you'll you'll you'll you'll see. Awesome.
8:36
We we encourage every new startup to do this, announce their fund raising Claude 4. 6 fast mode tokens.
8:41
Thank you so much for joining us. All right, man. Yeah.