Ep. 038 - Is GLM 5.3 a Mythos-Class Cyber Model From China? (InferenceX)

0:06

Hello everyone, welcome back to SemiAnalysis Weekly.

0:08

This week I'm joined by Wen Yao and Kimbo.

0:10

We're going to talk about everything GLM 5. 3.

0:12

Guys put out a great article recently.

0:14

The model is about two months old and we've got everything from InferenceX performance results to model architecture post-training pipeline and just like what people are using it for that we can dig into.

0:25

Spoiler alert, lots of discussions around cybersecurity capabilities with this model.

0:29

Kimbo, Wen Yao, welcome to the show, guys.

0:31

Thanks for having me, Jordan. Excited to be here.

0:35

I can hear it in your voice, guys. Okay.

0:37

So the title of the article was How GLM 5.

0:37

3 sparse attention affects HBM and memory usage.

0:46

For those who don't know, maybe we can start Kimbo with you with just an introduction.

0:50

Like, what is sparse attention?

0:53

Why do we care about HBM memory usage when we talk about these new open models?

0:57

Especially ones that are coming from Chinese companies like Z. ai here.

1:02

Yeah, so sparse attention is a category of attention mechanism where in contrast to linear attention, where you try to modify the core operation of the attention, which is usually called scaled dot-product attention.

1:19

Linear attention changes tries to change the fundamental algorithm to linear time algorithm, whereas sparse attention chooses to change the way you select the number of tokens while keeping the core attention operation.

1:34

So the idea is to invent some sort of selects token selection algorithm.

1:41

So that regardless of how long the sequence, the context becomes, you will be attending to only the most important tokens, the most relevant tokens at every forward pass.

1:53

Yeah, so this makes the model a lot more efficient, right?

1:55

And a lot of different labs are actually pursuing things like this to make attention a little bit more efficient.

2:01

Can you maybe give a little comparison of what Z.

2:01

ai is doing with the GLM models to some other labs like DeepSeek Kimi?

2:08

Yeah, so like you said, it's almost in some sense inevitable because just physics, we just don't have that enough memory and that enough don't have that enmu enough compute.

2:18

And GLM uses since GLM 5, they use the DeepSeek sparse attention.

2:27

The attention mechanism DeepSeek invented actually for their V3. 2 model.

2:32

And they opted that instead.

2:35

And the core components are you have a lightning indexer, which is the token selection component, and then the sparse attention.

2:46

The sparse attention is based off of the more standard multi-latent attention, multi-head latent attention invented by DeepSeek at popularized on DeepSeek V3. So that's GLM.

3:01

And for DeepSeek Even though they invented DSA, DeepSeek sparse attention on V3.

3:05

2, they continued innovating, they continued changing their attention variants.

3:12

So for V4, the for the V4 series, they invented CSA, compressed sparse attention, and heavily compressed attention, which are most roughly speaking building on top of DSA and further adding more token compression modules and for V4.

3:28

1 Flash they simplified the HCA and CSA into like a slightly different design and yes Okay. Yeah, sorry.

3:34

I mean, don't mean to interrupt you, but let's say that the reason they're pursuing this is in the name of efficiency, right?

3:48

Like the I think people speculate as to why the Chinese labs may care about this more than the American labs due to restrictions that they have on hardware, but roughly the outcome from all of these pursuits of sparse attention is that the cost to serve tokens on this chart, a million tokens, goes way, way down.

4:08

Like it's really, really cheap to serve if you have the ability to use sparse attention or like additional ways to like HCA and stuff like that, right? Yeah, totally.

4:22

It's just infeasible to fit that many tokens with a dense like a dense attention anymore, given like our usage goes easily above a million token contexts for like a single session. So yeah.

4:46

Okay, Wen Yao, let me bring you in here.

4:47

When it comes to actually serving this model and trying to analyze the cost to serve a million tokens at 150 tokens per second, this obviously involves us running benchmarks that simulate a workload that you know a real provider like Z.

5:02

ai or somebody else who's downloading the model and serving it, like one of the inference endpoint providers, would actually be you know incurring.

5:13

In other words, they want to serve dozens or hundreds or even thousands of concurrent users on a single model endpoint.

5:21

Can you talk to me about that restriction that Kimbo just mentioned?

5:25

Like if you're gonna support all these users with a million token context, it's gonna have a lot of pressure on your system, like the resources that you have available, like memory, right? Yeah, absolutely.

5:35

So I think for serving one of the factors that people care the most is like one of the factors definitely aggregated token output because it represents how many total output it can produce for your users, right?

5:53

And there is actually a trade off here is that well we only have does fix its amount of memory in the GPU and in order to get more concurrency to serve more users. We need a More memory.

6:09

So there is a technique called HiCache, which it serves as prefix cache reuse and when there are more concurrencies the concurrency goes up.

6:23

It will offload the memory from HBM to DRAM and that ends up will make the percentage of host memory goes up as the number of concurrency goes up.

6:37

So if you see on the B200 charts that we have on the HiCache yeah right here the concurrency of 8 and 16 was very different on the percentage of host memory and this is actually even larger than B300 because B200 it has less memory per chip And yeah, it's a really interesting and observation from our side.

7:09

Kimbo, do you remember when HiSparse and HiCache first got introduced?

7:13

Like these are clearly optimizations that are meant to save people money when they're serving these models, which kind of implies that the people that are introducing them have a lot of users that are using their models that they need to worry about.

7:26

It's not a theoretical thing. Yeah.

7:29

So HiSparse is the sparse attention version of HiCache.

7:35

And the whole idea of HiCache is trying to offload KV cache from the HBM to a lower tier higher capacity but lower bandwidth memory and only load the most relevant set of tokens during computation.

7:55

And I think HiSparse is a good example of an explanation to I think an also a misnomer that sparse attention fixes or fixes all the problems of KV cache consuming too much memory or having high memory requirements.

8:15

So the concretely sparse attention doesn't mitigate the memory capacity bottleneck.

8:22

It's a technique that helps a lot with memory bandwidth.

8:26

But as for capacity, because during the sparse attention computation, you would still need to select the tokens.

8:33

So you still have to keep all the tokens in in a relevant context and you cannot discard them during computation.

8:40

So at some point it still has to be loaded into HBM.

8:44

So the memory capacity consumption will not be significantly reduced.

8:49

And that's where HiSparse becomes useful because it can apply the typical caching rules so that you could load the most relevant parts into HBM.

9:01

Okay, Wen Yao talk to me about about some of the initial performance results here.

9:04

I mean, like the ability to use HiSparse versus HiCache, I mean both I guess, depending on the definition of terms, on this model GLM 5.

9:13

3 is obviously giving some great performance results, especially for particular vendors when you just look like total tokens per one dollar of TCO forty-five million tokens at peak up here, like very, very efficient. It is, it is.

9:36

And I just wanna add up to what Kimbo just said.

9:40

I think one thing that people that we want to emphasize on article is that even though GLM is compute efficient, it doesn't mean that we have efficiency on the memory side and because the models still need to read through the relevant context history, right?

10:00

So HiSparse becomes very important and It's offloading from HBM to DRAM and offloading back is very techniques.

10:10

And we've also introduced in the article.

10:13

And as for the performance we have, I think generally Nvidia chips just get ahead of all the other chips at the very beginning and A few weeks later, when we about to ship the article, very surprisingly that the AMD, interactivity at a hundred and fifty tokens per second is relatively the same cost as NVIDIA chips on B200.

10:38

Yeah, let me actually show that animation as we go through it right now.

10:42

Like the models were released in early October, and we've got this nice feature on InferenceX where you can go back in time and see how things make progress.

10:50

So all the way through, sorry, it was released in early August. We're now in October.

10:54

As you kind of shift through and you start to see these results show up on the dashboard, you know, wow, you can see The red AMD curves overtake the green NVIDIA curves, which is quite impressive.

11:10

And now NVIDIA start to come back up.

11:15

But just explain to me your experience here, because you guys are running this stuff every day.

11:20

Lots of people have been anti-AMD for a long time, especially the AMD software ecosystem, including us.

11:29

But this is a relevant model that's not even two months old, and the best curves for the highest.

11:36

You know, throughput, even at a hundred tokens per second per user, which is a real interactivity range for that, you know, the inference endpoints I can see on OpenRouter are not approaching. So it's below that.

11:48

That's a lot of high performance from the MI355 from AMD. Good for them.

11:57

Yeah, they're definitely working hard overnight and trying to do a range of optimizations.

12:04

I think it shows that I guess there are two sides, right?

12:10

The first side is the hardware where hardware vendors like NVIDIA and they have the their chips ready and for their customers That chip's performance was fixed.

12:25

And on the other side is the inference engine side, where you have to be catching up and continuing doing optimization and iterated over again to making sure that your performance takes more cost efficient.

12:38

And this is a perfect example where we see that AMD engineers working hard on their engine, ATOM, and over the time its cost per million token just catch up to a lot of the Nvidia's chips. Yeah.

12:54

And we have a the live chart specifically looking at the prefix cache reuse that you guys were talking about earlier.

13:05

If you click this at the bottom, you can see for a given configuration like MI355 on FP four or flip over to GB300 for the given points on the curve, how much of the host cache, like the DRAM, is actually being used.

13:17

Maybe dig into the point you were just making about the Implications for DRAM.

13:25

Like if we go back to the first big open model release from a Chinese lab, which was like DeepSeek taking the world by storm almost two years ago now, right?

13:37

But anyway, V3 and R1, memory stocks sold off in NVIDIA, and everybody was really concerned that a more efficient model was gonna mean less use of memory.

13:46

But maybe the opposite is true where When you guys are describing as you make the models more efficient, you can therefore increase concurrency and serve more users, and you have just as many demand as much demand on the memory itself.

13:58

But could you put a little bit of a finer point on the capacity versus bandwidth thing?

14:02

Because a lot of our listeners maybe tuned in for the conversation with Myron a few weeks ago where we talked about the de-spec of the Rubin GPUs for Rubin Ultra from a terabyte of G of memory per GPU down to 192, matching the R200, that keeps the bandwidth the same but reduces the capacity.

14:20

Can you explain the trade-off in the context of like inference where HBM bandwidth and capacity versus host DRAM bandwidth and capacity are all used to serve users at a lot of points on this curve?

14:33

I guess I can first start off and you know on the inference side.

14:38

So if we think about memory offloading and capacity, we definitely want to serve more user with more concurrency, right?

14:46

Because that makes the aggregated output tokens maximum.

14:50

But on the other side, we still need to think about the trade-off, which is the downside of offloading to DRAM, which is longer latency, especially for long tail latency.

15:02

So I guess the thing here is that we will customize based on different use cases.

15:08

So for example, we've introduced another inference engine, TileRT in our article, where it's extremely focusing on low latency cases.

15:17

Essentially what it does is that it has a mega kernel for decode to reduce the launch overhead and in smaller batch cases such like batch batch size equal to one it's it just shipped with blazing fast decode speed which is up to four hundred and seventy tokens per second and the trade-off is that it cannot serve with large batch sizes.

15:45

But we've seen that more and more inference engines are trying to serve with this special use cases for general gpu rather than Cerebras and people who want high interactivity and low latency will definitely love it And I guess the trade-off is really depends on how much of the cost you're willing to pay and what kind of the use cases you're dealing with.

16:10

If you are Jane Street, you definitely want your request gets like several seconds to get the output, right?

16:15

But if you're just like users like me, I'm dealing with the normal agent task, I can just wait it.

16:20

I can just like maximize the output as long as the cost is affordable.

16:27

You're not using ultra fast mode Wen Yao? No, I'm not using it. It's too expensive.

16:30

Also, it's hard to me to feel the difference somehow. Kimbo, how about you?

16:37

Are using ultra fast on any of the coding agents right now?

16:41

Not really, I don't think I can't even use fast right now. So Yeah.

16:47

Okay, so what do think about TileRT and the trade offs there?

16:49

'Cause like Wen Yao said, you gotta pay significantly more for that faster token speed.

16:54

It's a fun engineering problem, right?

16:57

But maybe not a business thing that is realistic.

17:00

My interpretation of so back to like Jordan, back to your talking about the Rubin de-spec, keeping memory capacity, keeping memory bandwidth, prioritizing that over memory capacity.

17:13

I think that's a good example of all the design decisions that we s we saw as well that memory bandwidth is more important than capacity.

17:23

So concretely the way to think about it is like we said sparse attention doesn't help with memory capacity and you will still need memory bandwidth to sustain the performance that you would like.

17:37

And I guess in the ultimate speed of light performance, which we can see in some sense we can see it in TileRT or the ultra-low latency regime, is that you're fighting against how fast you can load your model or how fast you can load the relevant data.

17:54

To finish the very small, relatively small amount of computation you have.

17:58

So in the ultra-low latency regime, memory bandwidth is the most important part.

18:05

And the problem with memory capacity is that we still need a lot of them, but the I think dealing with insufficient memory capacity is a very standard problem in I think computer architecture.

18:19

That's why we have cache hierarchy.

18:21

And there are numerous ways of through algorithm to hardware through hardware to overlap the computation with lower tier lower bandwidth memory transfers.

18:32

So in Rubin de-spec case, I think they're betting keeping the memory bandwidth.

18:38

While trying to connect GPUs through networking.

18:41

So and again try to overlap them. Yeah.

18:44

Do you think if you had to forecast what Z.

18:44

ai's thinking about right now, when they keep the model architecture the same for 5. 2 to 5.

18:51

3 and get such a performance improvement strictly from their post training pipeline, in my view, do you think they're experimenting with even more efficiencies on future attention architectures, or you think they're just gonna double down on what DSA and just keep it the same for some future, you know, GLM 6 release or whatever comes next?

19:08

I think my guess is that they would probably continue.

19:12

My guess is probably continue the same architecture, probably pushing further for 5. X or like 5. 4 maybe.

19:20

But based on their release it seems like they will continue trying different architecture because not too long after 5.

19:28

3 actually they released 5.

19:28

3 Flash which is not really a flash model it's in my opinion almost like a Frankenstein model where they do KDA the Kimi Delta Attention with DSA as far as I remember.

19:43

So they're mixing all the efficient alternative attention mechanisms putting into a model and actually train it and actually the model is pretty good.

19:54

So I think that the success of that model should be able would probably translate into like a scale-up version of a model with even better capabilities and having a similar architecture. Yeah.

20:09

Let's talk about the post training pipeline.

20:12

I mean, maybe as a bit of a backdrop, actually before that, we can talk about the capabilities.

20:16

But the first time I remember people talking about this was I'll show a tweet from TR Taxes, which then got Elon to respond, which then got Jie Tang, the founder of Z. ai to respond.

20:30

You know, when do you think China's gonna get a Mythos-class model? This is back in June. When GLM 5.

20:35

2 had been out for a little while.

20:38

Elon responds probably Q1 and Jie Tang says won't take that long.

20:45

I made a comment on I made a comment on one of the episodes we recorded yesterday where I said to Dan, to me GLM 5.

20:55

3 is a frontier cybersecurity model because I get classified out of using the Anthropic or the OpenAI models even though we're in the CVP Red teaming program, and we're doing stuff with the neoclouds to help them do this because it's not even the classifiers, it's the stuff trained into the model that I can't get around.

21:12

And so I fall back to using GLM 5.

21:12

3, and I need to be more creative in my prompting if I want to get the frontier models to not refuse me and pollute the whole context window where you have to like rewind if you want to try and do anything again.

21:22

So anyway, all of this is accomplished on I can show benchmarks and whatever, but like It's an incredibly capable coding model, especially for cybersecurity tasks.

21:32

And it's open, available.

21:32

Anyone can download the weights.

21:34

This is not the biggest model on the market.

21:37

And all of the improvements from 5. 2 to 5.

21:37

3 were done through post-training.

21:42

So I mean, I kind of agree, just keep post-training going.

21:45

But what are the unique aspects of their post-training pipeline that you think are letting them do this?

21:49

Yeah, so yeah, just to comment on the tweet.

21:52

I think Jie Tang the CEO it's pretty vindicated, right?

21:57

'Cause it took only like a month, I think, after that.

22:00

So can't believe Elon got out-Elon'd by someone, like, out-Elon'd on timeline.

22:09

The only thing Elon is, well, not the only thing, but the thing that Elon is most known for, for aggressive timeline.

22:14

But but yeah, so I think GLM 5. 3.

22:17

I think it's a hot thing to say this is Mythos-class right now, but it's or Fable-class, but it's certainly like approaching there.

22:27

It's nipping on their heels, pretty getting pretty close, I would say, right? True.

22:32

So well, technically it's definitely not Mythos-class.

22:35

But I think it's still nontrivial to get Anthropic themselves to admit that 5.

22:35

3 is what do they call it? Fable-class, I think. Fable near Fable level.

22:46

For their Yeah, I'll go find that blog post from them.

22:52

Yeah, so yeah it's I think it's a nontrivial feat, all within like a couple months of post training on the same model.

23:00

And yeah, so the in terms of post training pipeline, I would say among all the Chinese model labs I think Z.

23:08

ai is the one that invests most heavily on post training and focus really laser focused on post training.

23:19

As you can see from other model labs where they change the architecture every like every once in a while.

23:28

GLM has been focused on this same architecture. Starting from GLM 5.

23:32

Despite you see DeepSeek changing already two iterations of this the since DSA.

23:40

Kimi for the past year keep releasing new versions of their attention and doing scale up experiments.

23:47

You say Ant Ling doing all kinds of linear attention as well.

23:51

StepFun doing Also, very cool things, MiMo being even more conservative on that side.

23:56

GLM has stuck with the architecture mostly while focusing on post training.

24:01

And I would say the techniques in general aren't too far off.

24:06

The basic techniques of their post-training aren't too far off from actually the DeepSeek V3, which is still GRPO based.

24:14

But I think GLM, the post training pipeline, they have a very clear steps and reasoning to how they adapt their algorithm along with the efficiency they need.

24:25

So for example, the first step, they use a typical GRPO for general post-training, the first step.

24:33

They do it in a synchronous version, in a synchronous way, so that they could get the most stability out of the training.

24:40

So that will be, yes, the first part.

24:45

And then after that, because they would need to further extract the system efficiency out of training and also because agentic workloads they are typically longer in the sequence length and GRPO has always I guess our RLVR in general has always have the weak training signal issue.

25:09

So as your sequence gets longer the reward signal the strength of the reward signal gets diluted through the sequence roughly speaking.

25:19

So they because of that they The second stage they changed to a more reinforced like slightly different variant of post-training algorithm where you could tolerate other issues of the system.

25:32

For example, like policy staleness, so that you could use older version of the policy or you could use just the most recent version of the policy instead of having to deal with a lot more policy staleness ratio related issues during calculation.

25:51

The give me a quick summary.

25:51

This is algorithmic related decisions that you're describing here.

25:56

These are maybe unique decisions that they've made, but nothing I've heard you say, you know, before this talk or now would lead me to believe that GLM 5.

26:02

3 is doing something so unique on the algorithm that it's so completely different to any of the other labs that they have a big advantage on in terms of like algorithmic work in the post training pipeline. Is that fair to say?

26:18

I guess what I've said so far is not too unique to Z. ai.

26:26

Actually, I wanna to say what I feel about GLM 5.

26:26

3 when it first came out, like too with the no, it's completely based on post-training with the previous models.

26:36

I feel like this is like a wake-up call for me.

26:41

So if you see the benchmark results of GLM 5.

26:41

3, it's a agentic results on the benchmark was improved it more than 2% by just doing post-training and you know it's more GPU efficient if you're doing post-training than pre-training model from scratch.

26:57

And I think that's a wake-up call for a lot of other companies as well that they don't have you know enough budget to train a model from scratch but they also want to you know improve their company's workflow for some specific area and they happen to have that proprietary and private data.

27:16

So they can just like do post training themselves and achieve a better result than some of the closed-source models.

27:25

If I remember correctly that GLM 5.

27:25

3 was focusing on agentic reasoning and general RL and we see all the three areas has improved quite a lot by looking through the benchmarking results.

27:38

So that's like the day that I saw the tweet where GLM 5.

27:44

3 comes out and all the benchmark improvement I was like, no way, just done this by post training.

27:51

That's quite amazing to me, yeah. Yeah.

27:53

So yeah, I totally agree with Wen Yao on how just how much juice there's left to squeeze by just post training. And yeah 5.

28:06

3's post training effort is definitely very impressive and also cyber security. 5.

28:10

2 In our article I think we also compared GLM 5.

28:14

3's cyber capabilities with 5. 2.

28:19

And it's like very notable.

28:22

It's like a night and day capability difference and that's all just post training.

28:27

So yeah, w the part unique about 5.

28:27

3, I would say to be fully honest, GLM hasn't released like a detailed technical report since GLM 5.

28:39

So working based off of talking with researchers and the blog announcements, I think there are two things that are most notable.

28:48

One is the one that I actually publish results about or research about the they're doing PPO style training.

28:56

They're reverting in some sense reverting back to the typical RL training format having like a critic model grading rollouts along the way and co-training the critic model along with the actor the the main model so that's they call it single rollout asynchronous optimization SAO and the second thing is they in their public speeches and in the conferences, they talk a lot about thank you for putting out the name meme by the way.

29:30

They talk a lot about thank you.

29:32

I don't know it's probably too Taiwanese too local.

29:39

Like even my like Chinese friends actually the slime authors and Just Chinese community.

29:45

They don't even understand that meme, unfortunately.

29:49

Maybe explain that a bit, Kimberly. It's all fun.

29:54

Wen Yao, have you watched SAO, Sword Art Online?

29:57

No, no, I haven't but I've saw your explanation on Slack. That was fun. Explained it. I thought I don't know.

30:05

Do they understand this meme, Kimbo?

30:05

Ten, fifteen, something like that. I don't know.

30:11

No one I haven't asked anyone about that yet, but that's Okay.

30:16

But I jumped in earlier and I said something that might get misconstrued here, which is like they didn't have necessarily super unique advancements in algorithms.

30:26

It doesn't and you just said the word slime, which I was waiting for.

30:28

Like the both the single rollout asynchronous optimization, but then also all the infrastructure seems to me like work related to the efficiency of the training that just allows them to run a bunch more experiments, potentially much more jobs, potentially just like much more compute efficient jobs.

30:44

Which is itself an advancement that sets them ahead of anybody else with like equivalent compute or even more compute who just like can't do as much research as them. Yeah, yeah. Definitely.

30:50

Slime is highly regarded in the community, in the open source community.

30:56

They're probably arguably one of the best, if not the best, design, architecture design of post-training framework.

31:05

And yeah the efficiency I'm sure they have like very good infrastructure to support that. They also GLM the Z.

31:09

ai they also released an article I think probably not too long ago, two weeks ago or something, about how they used their models to co kind of collaborate with models to further help them improve the system efficiency.

31:28

So that includes bringing up the verifying the GPU kernels for their crazy GLM 5.

31:30

3 Flash model on an unknown hardware.

31:36

So on the I think it's the I'm not sure what GPU exactly what hardware exactly, but not the GPUs.

31:43

So they have unknown hardware with different different architecture, they use their models to improve that and they have ways to improve the loop of improvement and also monitoring system level jobs how to further squeeze out more efficiency out of the whole infrastructure so So yeah, I think there slime is definitely algorithm-wise SAO is a clear innovation, which is applied in 5. 2 and 5. 3.

32:21

And then infrastructure based on even just on what they open sourced, it's a very good improvement and definitely would help with And let me throw one thing in here that I think a lot of people have missed or is at least something I've been focused on.

32:39

If we go back to the GLM 5 paper, I want to read a paragraph from this.

32:45

And remember this is like almost a year old at this point.

32:50

They say last one more technical contribution lies in the fact that from the first day.

32:56

GLM 5 is full stack adapted to Chinese GPU ecosystems.

32:59

We have successfully completed deep optimization, spanning from underlying kernels to upper-level inference frameworks across seven mainstream domestic chip platforms, including Huawei Ascend, Moore Threads, Hygon, Cambricon, Kunlunxin, MetaX, and Enflame.

33:17

When you're talking about the model helping bring up the infrastructure for serving GLM 5, it's not necessarily the case that this is just NVIDIA GPUs and kernels and the serving stack that we're talking about for SGLang on the AMD or NVIDIA GPUs that we test in InferenceX.

33:34

They're serving this model, at least they claim, for many months on Seven different domestic Chinese accelerators.

33:41

What do you guys make of that?

33:42

First of all they also publicly claimed that 5.

33:42

3 Flash, the when it's under the name aux alpha, they're fully served on Chinese hardware.

33:54

So that's because I remember when people tested it They saw aux alpha being very good while also being able to serve I think a hundred trillion or something.

34:03

Like a ridiculous amount of tokens compared to I guess based on the amount of tokens that they can serve for free, people thought that's it's impossible that's it's served on a GPU by a Chinese app because Chinese labs aren't that GPU rich enough.

34:21

But turns out they're using Chinese hardware.

34:24

So I think that assuming everyone's honest, I think it's they're able to serve it like plumb through all the blockers, like the basics of the kernels, the systems, the plumb through everything and while serving those models efficiently.

34:44

Yeah, here's I've got on screen aux alpha was a stealth model and OpenRouter.

34:49

It was only a month and a half ago, August twentieth, when this happened.

34:54

I it feels like a while ago now.

34:56

Much longer, but anyway, things move fast.

34:59

Yeah, they did it they did bring up I think one month or two I think. So the 5.

35:05

3 Flash is cause yeah they released 5.

35:07

3 on early August and then they released 5. 3 Flash in late August.

35:19

So that means that even if they decided to The problem is when they start developing the 5.

35:23

3 Flash model, if we have to like kinda upper bound, lower bound, it's probably won't be too long.

35:32

It's probably like two months or three months, as far as I can tell.

35:33

Because betting on KDA is still a big bet before Kimi K3 comes out because no one really knows whether KDA scales that well.

35:47

And KDA, I think Kimi K3 came out like mid July.

35:54

Probably the most aggressive timeline would be they trained 5.

35:54

3 Flash, like within two months.

36:00

If that's true, that's actually pretty crazy. Yeah.

36:02

One Yeah, it is really crazy.

36:05

One thing I want to add about the hardware side of things is that I think I also wrote it in the article. We can kind of guess.

36:11

So first of all, the arithmetic intensity of the attention operation during decode is typically matched with the arithmetic and intensity of the hardware.

36:22

So this is shown most notably in DeepSeek V3, which DeepSeek openly talks about it.

36:28

So that That means that there is some sort of hardware and software co-design in this sense where you try to change the model, the configurations of the model to fit the hardware properties of the GPUs that you serve.

36:43

So DeepSeek fitted for H800s.

36:46

But then so it got me curious, and then I try to calculate the number for GLM 5.

36:51

One of the most interesting changes they did is they changed the number of heads and also I can't remember the exact configured configurations they changed but because of the changes the arithmetic intensity is very different from the hardware the arithmetic intensity for the NVIDIA GPUs so Yeah, I'll bring this up here, Yes, thank you.

37:14

So I think the calculation I did was somewhat around a hundred and twenty for GLM 5 or the 5. X series models.

37:23

And so if I and then I did some like bare basic calculations, it somewhat fits with the Moore Threads GPUs. So it's somewhere there.

37:35

That does not prove that they're optimizing it.

37:40

That does not prove that Z.

37:40

ai is actually serving models on those hardware.

37:46

It is doesn't really prove that it's optimized for that, but I think at least it shows.

37:52

My conspiracy theory is that they're probably optimized for it, but I couldn't get a confirmation. So Z.

37:58

ai, if you're watching this, would love your thoughts. But We got alpha here. Leaked by Kimbo.

38:02

Yeah, but at least I think what we can yeah, it's just calculate the arithmetic intensity, right?

38:09

But at least I think it shows that they're definitely optimizing for something else.

38:14

Or they're sacrificing quite a lot of software hardware optimization for other reasons that are not shown in public.

38:23

'Cause it obviously wouldn't work well on not the optimal for let's say B200s or H200s even. Yeah.

38:31

It's a sign it's a sign of what's to come that somebody might make a software level co design decision based on hardware that is not NVIDIA GPUs.

38:40

I think that's a quite a big statement. Mm, yeah.

38:42

And I guess like adding on that, that with the capability improved of the models But we begin to see that the R&D cycle has been shortened drastically, right?

38:55

And take a kernel side as an example.

38:58

You know, now nowadays agent they can with the proper tools like kernel development agent or the cake just released recently with the right harness and the right tools, people get to write kernels that performs much better than human themselves.

39:13

And we also got like leaderboard.

39:15

That agent writes kernels as long as you have the budget, it will write faster kernel than humans.

39:21

And actually it's like pretty interesting to see how different model company going to you know doing compete with the chatter with the help of agent so it's like human plus AI to do the compete to do the compete yeah How much how many tokens do you think the AMD TileRT mega kernel for GLM 5. 3 cost them to produce? Be a lot, I think.

39:44

What do you think, Kimbo?

39:46

I don't have a good idea, honestly. Yeah.

39:49

I wonder, yeah, maybe we should ask. Maybe we should. Up with good ideas here.

39:53

Okay guys, anything you think that we've missed talking about this?

39:58

I think we went through InferenceX performance results, some high level discussion about the cyber capabilities of this model, what we're using it for, HiSparse, HiCache, you know, KV offloading, model architecture, post training pipeline. What else is left?

40:11

Rumors and vibes on what's to come with GLM 5. 4?

40:16

What You want to speculate as we sign off here?

40:21

Yeah, I think we briefly talked about it. The 5.

40:24

3 Flash is definitely a success for them.

40:28

They're able to mix DSA with KDA.

40:32

So while like a very new architecture while still having a very great model, having great capabilities.

40:39

So I suspect they can definitely scale up 5.

40:39

3 Flash, the model architecture of 5. 3 Flash to 5.

40:45

3 level size and probably even squeeze out even more performance or model capabilities. Yeah I think 5.

40:54

3 Flash has that crazy efficiency as well, right?

40:57

Because they are linear attention and sparse attention. So yeah.

41:04

I'm curious about what they're going to do next. Cause I think Z.

41:07

ai they definitely explore something else.

41:11

By post training they show us like you can improve model quickly but not only scale up but also squeeze up given your current parameter size even that even though you're not trillion model size you can still perform as well and for those companies who do inference for the model and they're heavily sensitive to the cost I think it will be definitely a good call for them.

41:35

Actually we've seen like in the Bay Area, like Harvey has their own post-trained agent based on Kimi model.

41:46

We'll also see a Periodic Labs.

41:48

Maybe Kimbo can extend that has done the post-training on AI for science.

41:52

So I'm really excited to see like how this trend goes and especially what GLM 5. 4 will bring us.

41:59

Yeah, one I guess one last thing, I think about post training that is very important but not very much discussed and Z.

42:07

ai also is not willing to clarify more either is human effort.

42:12

Last time in their public like in by conferences and things, they talked about they spent a lot of human effort into supposedly data when training 5. 2 and 5. 3.

42:25

And the problem is what kind of human effort, what kind of well human effort I would say I'm pretty sure it's something related to data labeling.

42:33

But the problem is what kind of data labeling and what kind of effort into data labeling.

42:37

And yeah, that's one like I would think it's also pretty important when it comes to post training that they're able to get quality data. Makes sense.

42:49

So what do you think comes next?

42:51

Do you think we get Space Bunny Alpha is MiniMax 3. 1 Flash? Do you think it's GLM?

42:56

Do you think Kimi comes out with something new? DeepSeek.

43:00

We got a big release from Reflection. I don't know.

43:02

What are you guys looking forward to most with the model releases coming out soon?

43:06

Some smaller I'm looking for some like efficient model actually Reflection's model is it's really efficient and I'm thinking like maybe in the future we have large size model right two trillion three trillion and it's capable enough for people to do powerful stuff but we also have models that offer good enough performance, but it's like extremely efficient.

43:31

That company won't care about its cost and still do some certain thing well like reflection that they're released their models. Makes sense, guys. Okay.

43:40

Well, thanks for coming on.

43:42

Whirlwind tour of everything GLM 5. 3.

43:45

And why we don't think model releases like this mean people should sell their memory stocks.

43:49

And why we expect Jevons Paradox to continue to result in more people consuming a lot more tokens and a lot more HBM and a lot more bandwidth over time.

43:59

Thanks for joining, guys. Great job. Thank you, Jordan. Thank you Wen Yao. Thank you, Jordan. Thanks everybody.