0:05
[music] >> Hi everyone.
Search this video
0:10
Thank you Mohammed for joining us with this interview.
0:13
So yeah, can you first introduce yourself a bit? Sure.
0:16
Yeah, thank you Kimbo for having me.
0:18
So I am Mohammed Abdelfattah.
0:21
I'm an assistant professor at Cornell University.
0:24
I'm also the chief science officer at Macorix.
0:29
So my work is broadly on AI performance and Macorix we're trying to automate as much of that AI performance engineering as we can.
0:36
So I guess my first question may be I think the audience would love to know is maybe the origins of Macorix, the motivation of why build an automated performance engineer and then I think a lot people who already know Macorix the question would be why so the multiple name changes? I see.
1:01
Yeah, so the name change thing is because we used to be called Maeko and Maeko was you know widely used names with other startups.
1:08
One of them asked us not to use the name.
1:10
They were they asked nicely and we complied.
1:12
asked nicely and we complied. So and so I mean our our thesis with Macorix is that you know we can automate a lot of the things that engineers are doing manually now and we started with this thesis almost a couple of years ago now before you know code generation took off and
1:32
before you know agents are now really becoming that capable and we've successfully kind of started at this code generation core where we are really trying to automate high performance GPU kernels and high performance kernels for other hardware and then that's becoming a component in our system level optimization. So that
1:49
So that means extending to definitely inference server and and also you know optimizing things like training pipelines and reinforcement learning rollouts and things like that.
2:02
What I'm most excited about now is actually those system level problems.
2:06
And we have a bunch of interesting research that we hope to kind of really boost performance to levels that we couldn't get before.
2:13
Now, what what kind of system level problems are you how do you find interesting recently?
2:21
Yeah, so I mean, we at Pecora we have kind of still the pragmatic approach of, you know, hyperparameter tuning, uh adapting to, you know, the underlying hardware.
2:28
Uh but as kind of chief science officer and also as being fundamentally a researcher, I'm looking into kind of newer algorithms as well.
2:38
Um one new inference algorithm that we're very excited about is called uh sequential Monte Carlo speculative decoding.
2:46
And so, the way that works is that, you know, in normal speculative decoding, uh you have a draft model, so you draft one token at a time, and then you take K tokens and you verify them in parallel with a larger model.
2:57
And that's how you get speed up because you're using that large model in parallel instead of autoregressively.
3:03
However, one problem is that any tokens that don't match the target model, you rewind the draft and then you start again.
3:10
And so, this rewinding is just a big performance hit.
3:14
In sequential Monte Carlo, what we do instead is that we keep N drafts alive.
3:19
So, we have N drafts going in parallel.
3:22
And instead of verifying with the target model and looking for an exact match, we're actually just scoring those N drafts with the target model.
3:29
And based on that importance sampling, based on that importance score, we do some kind of sequential Monte Carlo sampling where we can evict the ones with a very low score and duplicate the ones the drafts with a very high score.
3:42
And if we do that again and again, turns out, well, we're always accepting K tokens.
3:46
Yes, we have the extra compute for multiple drafts, but we're getting much faster performance than normal speculative decoding.
3:56
So, if I'm understanding correctly, it sounds like a way of increasing the number of draft tokens, more specifically in parallel, while guaranteeing a higher acceptance rate.
4:09
Is that like a >> We're Yeah.
4:10
Yeah, you're exactly right.
4:10
So, we we are utilizing the draft model more.
4:16
Think of it as if you're solving a math problem, you know, pass at n is always better than pass at one.
4:23
And so, we have kind of n trials happening from this draft model.
4:27
And we're always accepting k tokens.
4:29
We're never rejecting any token or rolling back.
4:31
And that's fundamentally why we can get a higher speed up.
4:36
However, to account for the fact that we're never rewinding, we have those n drafts going in parallel, and we're always keeping the particles or the drafts that have a higher chance of matching the target.
4:46
And that's due to kind of the scoring that happens at the target model.
4:52
And so, we've we've tested this methodology kind of compared to SGELang and compared to vLLM.
4:57
In our testing, SGELang was much faster than speculative decoding, and so that's what we're we're comparing to.
5:03
We're about five times faster than SGELang baseline.
5:05
And about two times faster than speculative decoding on SGELang with their experimental overlap scheduler.
5:13
And we're also quite a bit faster than SSD, which is an an even newer baseline [clears throat] that um also has this notion of speculative drafts that was recently published by other folks.
5:26
Based on what you share about the results, that does sound very almost unreal.
5:30
Like boosting performance >> I mean, like like any results, there are a bunch of caveats.
5:35
The result that I just quoted is a batch size of one.
5:37
And so, this is really targeting, you know, the really low latency regime.
5:43
Obviously, also because we're increasing compute.
5:45
At low batch size, we were utilizing unused compute because all of this is memory bound.
5:50
Even the draft model is memory bound.
5:52
But at high batch size, things start to get a bit more complex.
5:57
Basically, we saturate compute a bit earlier than normal speculative decoding.
6:01
So, like any result, there are some caveats, but focusing on that low batch, low latency regime, we we are able to do really well.
6:11
We're also fundamentally a lossy method, and that can be like a big turn off for many people who are interested in maintaining kind of exact accuracy.
6:17
But again, in some of our empirical testing, we even outperform the target model.
6:22
And so, theoretically, this shouldn't be possible if the draft model is strictly worse than the target.
6:30
Basically, as the N goes to infinity, we should be matching the target model.
6:34
Should not be outperforming it.
6:38
However, because of, you know, the variance in quality of of these models, sometimes having N drafts from a smaller draft model can actually outperform the target.
6:48
And so, we're quite excited about it.
6:50
We're in the process of kind of moving this from research to product now.
6:53
Should have a preprint out today or tomorrow, hopefully.
6:58
And we're already kind of working on it on the Mac OS side.
7:03
Um another question I have is does it require like a calibration data set for training?
7:10
Like the the speculators or is it kind of more close to MTP where you do it while training the model? I see.
7:20
Yeah, that's a great question.
7:21
So, SMC speculative decoding can work on top of MTP, can work on top of Eagle, on top of a special trained predictor.
7:30
In fact, SMC is quite flexible.
7:30
This isn't something we're exploiting yet.
7:33
It can even operate on two models which have different tokenization.
7:38
So, there are some some tricks that we can do there to allow, you know, a model with a different vocabulary to be a draft model for another target model with a slightly different organization.
7:50
But, that's that's still kind of a niche use of it.
7:54
There are lots of kind of cool things on the research side that we're still exploring.
7:57
But, just yeah, fundamentally it's compatible with with all of these uh speculation methods.
8:01
How do you plan to integrate this new method into the product?
8:07
Yeah, so we have an SG length work that uses this.
8:09
As you may know, part of our product kind of uses many open source inference servers under the hood, chooses the best one, and so we kind of add our optimizations to the one that makes most sense.
8:20
We're also actually working on our own inference engine at the moment because, again, I pointed to some of the fancier things that we wanted to do, and some of those are not so easy to do in SG length and other inference engines.
8:32
Yeah, I guess it's a constant battle between cutting edge research and production.
8:40
So, it's yeah, it kind of it's it'll be hard to implement some features in the production grade serving frameworks. Yeah. Okay, let me see.
8:50
About the auto GPU kernel generation, I guess I think there are definitely a lot of companies doing or a lot of people pursuing similar ideas, and I think everyone faces the issue with kernel benchmarking, like how to accurately do that.
9:09
And then um in and in worst cases reward hacking, where the code generated kind of found loopholes in and harness, and then gaming the benchmark.
9:19
So, I wonder how does Amakara deal with this kind of issue?
9:25
And any insights or any pains, uh fun stories into this?
9:33
Yeah, there there are some stories that are not so fun, including kind of maybe the worst demo experience I've ever had in my life.
9:40
Maybe the worst 30 minutes of my life.
9:42
Where in the early days we were demoing uh that's okay.
9:47
We were demoing our tool and there was a lot of reward hacking and tells you something about kind of how these foundation models were trained.
9:53
Seems that um kind of there is still it's still far from perfect.
10:00
Uh however, since then we've kind of really bolstered our eval eval pipeline.
10:05
So that reward hacking is caught kind of multiple steps.
10:07
One of the that is just kind of tracing through the code and making sure all of the generated functions are called properly.
10:13
Some of it is using AI to detect reward hacks as well.
10:19
That's usually quite an effective catch all.
10:21
We have like I don't know 11 different maybe classes of reward hacks that we catch that way.
10:27
And then our eval pipeline is also very rigid.
10:30
Like if a library is called and it shouldn't be called then just it won't go through eval.
10:36
Like we don't allow the generated code to just arbitrarily use, you know, libraries from everywhere.
10:42
It shouldn't be generated code with as few dependencies as possible.
10:46
And so yes, I mean none of that is super kind of fancy I would say.
10:51
It's it's a lot of engineering to get this correct and I think we're at a stage where we're very confident in our eval pipeline.
10:58
We have sold our eval pipeline to others to use training their foundation models.
11:04
At least one customer so far and they've been fairly happy with it as well. Interesting.
11:11
Yeah, that sounds very impressive.
11:13
By by selling to clients do you mean as in selling a kind of like model providing like a sandbox environment?
11:21
Is that kind of the the thing where you instead of general CPU you provide GPU kernel evals?
11:30
Is this something like that?
11:31
>> we basically expose an API, which is the same API we use internally.
11:33
We just expose it to customers and give them a token for authentication.
11:38
And then they use our infrastructure.
11:41
Some of it uses models, some of it doesn't for our kind of evaluation and profiling and all of that.
11:48
So they get kind of the reward signal back and they use that to train their model.
11:54
They give us a kernel and a problem. We give them the reward. Okay.
11:56
Yeah, that uh Um yeah, I think uh sandbox environments has become increasingly important or people are seeing it more in the typical RL post-training pipeline.
12:06
People are seeing that inference is got going very fast.
12:11
Training has been very fast and then the upcoming bottleneck in the post-training pipeline will will uh start becoming the sandbox part.
12:18
So yeah, it's a very cool to see uh Makura doing this kind of work.
12:25
So so like like I said, a a lot of uh companies are doing similar things.
12:28
Um for example, there's one called uh Standard Kernel Co.
12:33
Um they're also trying to build an agent that uh generates uh GPU kernels.
12:39
I wonder how do they compare to Makura or how other companies compare to Makura and what's a unique what's a unique advantage of Makura?
12:53
Yeah, I mean when I think about our competitors, I find that most of them are like super smart and super capable competitors.
13:00
I admire a lot of uh the work coming out of Standard Kernel, Kimera Labs, other other labs where you know, code generation is really part of their core offering.
13:10
I mean, I think one way we like to think we differentiate ourselves is by focusing on performance.
13:14
We don't I mean, at some point we thought we were uh going to kind of sell just a code generation agent.
13:21
It's like selling a compiler.
13:23
You can't very hard to do, basically.
13:25
Our value proposition is that we sell performance.
13:28
And you come to Macuara, you get end-to-end performance managed on your infrastructure or any infrastructure that you want.
13:34
And like I said, it will include code generation because kernels are very big part of that performance, but it will also include kind of the latest research that we're doing on things like Spec T code.
13:46
Another example is, you know, we were working on numerics.
13:48
Obviously, low precision is super important.
13:50
And so, some again of our research that's hopefully making it to some of the product at Macuara is some research we have on making FP4 more accurate.
13:58
And that's the kind of research that really needs very in-depth kernel work to work well.
14:07
So, so in this case, I promise I'll be brief, but basically, in FP4, you can represent minus zero and plus zero.
14:14
You can represent both polarities of zero, and it's a wasted quantization level.
14:18
When you have 16 quantization levels, and you're wasting one of them to represent zero in two ways.
14:22
And so, we remap the redundant zero, we called it tracer, redundant zero remapping, to a learnable special value.
14:29
And this effectively gives you the accuracy of FP5 or FP5.
14:34
something in the memory footprint of FP.
14:37
And we went and implemented this on Nvidia.
14:40
The weight-only quantization is fine because you upcast anyways later.
14:44
And so, remember that special value can't be FP4 anymore.
14:49
It has to be a higher precision.
14:50
Otherwise, it it doesn't really make sense.
14:52
So, weight-only kernels are fine, but when we wanted to use NVFP4 and the tensor cores for weight activation quantization, we basically incur big performance overhead.
15:03
And the reason is that, you know, you can no longer can get away with doing one pass of tensor core on this.
15:09
You have to do a first pass, and then another pass to compensate for all of the entries that were negative zero and were remapped to something outside of the FP4 range.
15:17
This is the most naive solution.
15:19
The second pass is very sparse, and so it has a high potential for acceleration, but fundamentally we're going to be slower than FP.
15:27
However, that's not true on AMD.
15:30
So on AMD, the FP6 data path shares the hardware with FP4.
15:37
And so actually with the same throughput, we can upcast to FP6, still have the memory footprint of FP4, but upcast those special values and use FP6.
15:46
Whereas in Nvidia, FP6 shares the data path with FP8.
15:50
And so now this like very niche unused precision FP6 is coming to our rescue because we can show, you know, high performance for this special fancy razor FP4, but only on AMD right now unless Nvidia changes their hardware.
16:05
So anyways, that was a long-winded way of answering the original question of kind of differentiation with uh competitors and and so on.
16:11
We want to deliver end-to-end performance and accuracy.
16:16
And you know, we have a bunch of these cool research ideas up uh our sleeve and we focus on delivering, you know, the end-to-end platform and the deployment engine of models as opposed to uh a compiler, which which is fundamentally what code generation is.
16:33
Yeah, that that makes sense.
16:35
Um by offering performance instead of uh just a compiler or just a coding agent, and I guess you have more flexibility of um what you can uh do and what techniques you can use.
16:49
And I find that I enjoy that the FP6 thing.
16:52
I find that very interesting.
16:54
I recognize that uh AMD has like a different FP6 flops compared to Nvidia, so it's interesting that you guys can get better performance on AMD, >> [clears throat] >> which is typically um or not seen as as often, I guess.
17:12
The AMD hardware fundamentally is is quite strong.
17:17
I think it's software needs to catch up in some places and we hope to kind of fill some of that gap as well. Yeah, totally.
17:24
And uh kernel writing part, I know you you're Makora wants to provide performance in general, but are we only talking about um GPU kernel generation or even some other more general performance-related optimizations?
17:40
I think we can expect that our frontier models would get better at them.
17:45
I think 6 months ago, probably AI frontier models can't even write GPU kernels properly.
17:51
But right now, they're almost able to write QDSL kernels.
17:57
So, that's like a very big uh performance improvement and I think it's reasonable to ex- expect that frontier's model will continue to improve.
18:06
So, in this uh timeline or in this situation, where does um how does Makora fit into the this situation and uh does Makora plan to adapt in any way or yeah, anything like that? Yeah.
18:22
I mean, I found that, you know, more and more as time passes by and as models and, you know, foundation model agents become better and better, the less we have to do on specifically the code generation part.
18:34
And so, we're always trying out new models, always trying out new agents and integrating them into our product.
18:41
And so, we're not we're very ego-less in that regard in that we have, you know, multiple agents that our product will try and it will just deliver the best kernel. That's the goal there.
18:54
And as long as that's kind of can be used as a tailwind kind of improve our end-to-end performance, that's what we care about.
19:00
We also want to we're also looking at open-source models.
19:03
And so, recently uh one of our researchers tried out the Gemma 4 model.
19:07
I I it was recently released and we're getting really good performance out of it as well.
19:15
And so like fine-tuning and specializing these models is still something we're looking into just to be kind of less dependent and provide a, you know, less expensive alternative to these large foundation models.
19:26
And we're kind of keeping an eye on how fast these open-source and available models are catching up.
19:32
But our philosophy is whatever gets you performance, we use.
19:37
And that means just using something off-the-shelf or within our own software ecosystem, that's fine by me as long as it delivers the best kernel.
19:44
More and more though, like if I were to add to that a bit, generating a kernel is one thing and generating a kernel that's integrated into end-to-end software, whether it's a library, GGML for some, you know, Qualcomm NPU, or if it's SG Lang for an AMD GPU, that's a slightly different thing.
20:03
We have to care about data layouts, interfaces, quantization modes, and there these details kind of add up in the end and that's the difference between kind of foundation model generating code or our tool generating a kernel that's ready to be integrated into the end solution. I see. I see.
20:22
So, Macor tries to be more um tool agnostic, I guess.
20:25
Is that like a right way to uh Yeah, we're very um foundation model supplier kind of agnostic. >> I see.
20:36
And uh so so I guess um like a next question of that would be if that's a case, um what would be the major reason clients would reach out to or customers would reach out to um or more concretely, what are the major reasons right now um that clients are reaching out to Macor for service or for optimizations instead of just paying Claude or playing paying uh uh GPT for and just let it run and try to solve it.
21:10
You know, that's a good question.
21:12
I mean, our customers are kind of the most kind of foundational layer are hardware vendors that want to build out software support for their hardware.
21:21
We've worked with large hardware companies that you know, had literally just released a programming language for their NPU and we were able to generate kernels in that programming language and building out kind of a primitive library.
21:35
Other vendors that would like to build that extra layer up to VLA or SV lang inference server in that stage.
21:43
And then there are all So, that's kind of one class of customers.
21:46
You want to build out support for your hardware and you don't want to train like a developer network for it.
21:53
And yes, you can do it yourself internally, but I was just chatting with one of those customers and they were saying that internally one engineer is able to use kind of foundation models to generate kernels, whereas we're able to deliver it kind of more consistently and constantly using our tool.
22:12
So, automating all these tidbits around kernel generation, there is still value there in having an end-to-end product.
22:19
And like I said, it's like building a compiler versus just generating an example code or proof of concept.
22:26
But again, we we realized that that is less sustainable than focusing on performance in terms of a business model.
22:36
And so, the other class of customers are enterprise customers.
22:38
So, these guys have bought just a bunch of GPUs and they have no idea of getting high performance out of them.
22:45
They don't have an in-house performance team.
22:47
And these customers are a lot, or they have access to or they want to host their own models.
22:53
They cannot use public APIs and they want to make that available to their employees.
22:59
And so, basically maximizing the performance out of infrastructure becomes kind of the next level of customers.
23:06
And we're also chatting with Neoclouds who are offering hardware.
23:10
Sometimes they have their own token factories or their own token APIs, and sometimes they kind of just rent out the GPUs and they want to have, you know, go-to software solution to point their customers at.
23:22
And for all of these, that's kind of us expanding out to the system level and, you know, going beyond kernel generation to system level performance.
23:30
And so, yeah, so that system level performance part can we automate it with, you know, foundation model agents and models?
23:40
Maybe, but it requires way more expertise at this stage, I would say.
23:44
And so, we bring in that expertise.
23:47
Yeah, yeah, that that makes uh makes a lot of sense.
23:49
Yeah, I can imagine definitely a lot of enterprises um would still be reluctant to um uh put their information outside of or give it to um the companies like OpenAI and Cloud without a uh exclusive contract or something.
24:09
That they could reach out to Macorix to uh by having their own infra and then reaching out to Macorix to build out on their internal models, serving their internal models, and something like that.
24:20
Yeah, I guess one last question, what's next for Macorix?
24:22
What are the plans for next month, next 6 months, or even next year?
24:28
Yeah, I mean the the plan is to go higher level in the stack and really have a killer kind of inference engine, and then maybe expanding to training or reinforcement learning as well.
24:39
Like being the deployment engine for high-performance AI models.
24:43
You know, I we have this vision of even, you know, actually instead of in addition to the technology and all of the cool research ideas that we're implementing, actually making it a very usable product.
24:53
And so that even like system administrators at enterprises can use this and get access to that performance quite easily by pushing buttons and stuff.
25:01
And so really kind of closing the gap between you know, I bought a GPU to I'm getting the most performance out of that GPU.
25:09
That's what we're really working on and excited about these days.
25:12
Then our grand vision is more on the abstraction of different hardware.
25:18
And do there are so many opportunities now for disaggregation to get more performance and because we have that code generation superpower and we all have to do that.
25:29
There is a lot of potential for, you know, generating, you know, the right code for the right hardware and disaggregating your workload on different machine types.
25:35
Speculative decoding is a perfect example of that.
25:40
Draft target disaggregation in the case of SMC as well.
25:42
So, yeah, lots of opportunities there to kind of first abstract away performance engineering and then abstract away the hardware infrastructure all together and use, you know, code generation and agents to get the most performance out of it.
25:57
Does Memcached plan to open source anything in the future?
25:59
Yeah, I mean, um all of the stuff that I talked about, you know, that's coming from the research side will be open source.
26:08
And so, uh even for this SMC stuff, we're going to release some code uh there.
26:12
Give it back to the community and we'll build kind of um a closed source enterprise version of it. Mhm. Sounds good.
26:20
Yeah, thank you, Mohammad, for this interview.
26:25
It was really nice talking to you again.