0:16
Everyone, welcome back to SemiAnalysis Weekly.
Search this video
0:18
We are here this week with a ClusterMAX 3. 0 review podcast.
0:23
I got Sam and Pratt we're going talk through everything that was included in the article that we pushed out on Wednesday, September twenty third.
0:30
We'll talk through the rankings as they are today, some of the statistics, like how many providers we actually tested, the methodology that we use to go through the testing over the last number of months and tease some upcoming projects such as EndpointX, PostTrainingX, HarnessX, SandboxX, we're gonna test everything.
0:47
And then talk about some of the trends that are affecting the neocloud industry, financing the move to Vera Rubin from Blackwell, security.
0:57
We had a previous article on that, agentic coding, how that's impacting infrastructure.
1:02
And maybe we'll get into some individual stories about specific topics.
1:06
But guys, welcome to the show.
1:08
How is the last three months, Sam?
1:10
You're coming out of your hole, man.
1:13
Yeah, it's my first time seeing sunlight today actually since testing began.
1:19
But yeah, I mean I'm a little sunburnt from it, but you know, I've had a few lifetimes worth of reps on Kubernetes and Slurm, so wouldn't trade for anything. Pratt how about you?
1:29
You feeling good after the testing period has come to a close?
1:35
Yep, feeling pretty good.
1:35
Was a lot of fun testing out these clusters, you know, always fun playing with big infra. So Awesome.
1:44
So headline news, I think the biggest thing is that there's a new addition of the platinum tier.
1:49
We've got Nebius up there joining CoreWeave.
1:52
In the gold tier, rounding it out, we've got Oracle and Google Cloud, Google joining Oracle.
1:56
On the silver side, Azure moves down, Firmus and Lambda stay, GMI moves up and TensorWave stays at silver.
2:06
And then into bronze, companies like AWS, Crusoe together all moving down.
2:13
Rounding out the list at a high level, this time we rank 77 providers.
2:18
We increase our market view to cover 323 total, which is available for subscribers to our neocloud model in great detail.
2:27
And we've now interviewed well over 200 end users of neoclouds as part of this research.
2:31
So let me pose a question to Sam to start.
2:35
Most interesting insight that comes to mind when you think about the labs that we've interviewed so far.
2:42
Like reliability, performance, ease of use, support.
2:45
What do you think is top of mind for people that are buying GPU clusters right now? Good question.
2:57
I mean, the first thing that comes to mind is just the ridiculous backwardation of the price curve at the moment.
3:04
So people are willing to pay just about anything for any chips that they can get in, you know, the next few weeks, especially.
3:11
You can get a big discount if you're willing to wait even a few months.
3:15
And yeah, I mean, that means that just about everyone in the industry looks like a genius right now.
3:21
If you have chips to sell, even if you know you're vibe-coded Kubernetes is completely broken, even if storage barely works.
3:28
If you've got GPUs, you're gonna find people willing to pay a pretty good margin to take them off your hands. Yeah, it's interesting.
3:35
We've talked about that trend a few times previously, which is like the providers that weren't able to sell their stuff early, typically that's a signal that they're not as high quality, but now they're actually closing these deals for significantly better margins, which is interesting. Pratt, how about you?
3:51
We're expanding a little bit beyond ClusterMAX in the future.
3:54
Things like serverless inference endpoints, posted like post training infrastructure for RL, harnesses, sandbox, you you've been digging in there.
4:03
When you're talking to the providers themselves about the roadmap, do you find it confusing that people can't get Slurm and Kubernetes right sometimes, but they're already ready to launch like four new products? I mean, I guess, right?
4:18
Because currently the margins on compute are so like high.
4:25
Of course, like as a neocloud, you do want to capitalize on that.
4:28
You can of course like SpaceX provides really good compute to Google.
4:34
They have tied like deals with Anthropic, and also several other neolabs.
4:39
However, neoclouds also want some of that, you know, like some of that cake.
4:43
So they're thinking, what is the most high margin business that we can enter right now.
4:50
Managed clusters of course was one of them, but it's a huge headache setting up the entire cluster.
4:55
So why don't we just go above that layer and sell actually hosted training?
5:00
Why don't we sell managed inference, you know?
5:02
So they're just in the end just expanding their market, trying to earn that little like good cash, which the labs are earning right now.
5:10
But everything points to centralization of compute.
5:13
That's the main thing I can say.
5:16
Centralization of compute. Makes sense. Sam, how about you?
5:20
Like, there's a few different providers who have been on the cluster max ranking system in the past and are either moving down the list or just moving to unavailable because they're either focused on bare metal or maybe they're focused on their inference endpoint business over clusters. But I don't know.
5:37
I talk to neolabs all the time, and a lot of people still need training clusters.
5:41
A lot of people host inference for themselves and they still need those clusters.
5:44
Like, do you think there's no more to be done on the product roadmaps for people that are building managed cluster experiences? Interesting.
5:55
As Pratt said, there is an interesting fluidity to the market right now where you know some of the better managed cluster providers know how to manage a cluster because they have teams internally who are trying to use these things and can give good feedback for you know what a good Slurm and Kubernetes layer looks like, how to set up health checks, that sort of thing.
6:15
So there definitely is a benefit to integrating these things in-house, in addition to just the fact that you have another service that you're offering on top where you can take margin from another layer.
6:25
I mean, we certainly see plenty of room for improvement from most of these players.
6:31
You know, I don't think that we're requesting super fancy bells and whistles on top of these managed clusters.
6:39
You know, there's a handful of genuinely interesting additions to the managed cluster experience, but mostly I think that there's plenty of work to do just to get Slurm and Kubernetes bulletproof to get health checks right.
6:50
And I think that everyone on this call would agree that it's quite lucrative for these companies to do a better job, be able to collect a few more cents on top of the GPU hour prices that they're charging just by doing a slightly better job at the software layer and, Okay.
7:02
Yeah, what you know, picking one lane and making it great.
7:10
okay, let's talk let's dig into health checks because this is something that we wrote a dedicated article about a little while ago called how much do GPUs really cost?
7:16
And in this one, ClusterMAX 3.
7:19
0, we go through in detail the methodology that we approach testing reliability with. Can you tell me a story?
7:28
Leading the witness here, about some providers that claim to have health checks.
7:38
And then when we test them or try to exercise them with them hand in hand, they're just like not really working or they're needing a lot of manual intervention.
7:46
Like what sets apart somebody that has health checks developed out to somebody that doesn't?
7:52
And maybe just basic level define a health check, right? Sure.
7:58
So there are basically two types of health checks active health checks and passive health checks.
8:02
So an active health check runs an actual workload or sort of a dummy workload when there's nothing else going on in the cluster.
8:09
This is normally preemptible.
8:09
So if I have an active health check running and then I decide that actually I have some training job I want to run, the health check gets interrupted and the actual workload begins.
8:20
The passive health check is just running in the background.
8:23
You know, you want to see as little a performance overhead as possible on the passive check.
8:28
And it doesn't it can be exactly zero.
8:31
It can just be, you know, reading logs from the CPU while the GPU does all its work.
8:36
And I mean there's a handful of basic things that we like to see in health checks that are not always respected.
8:45
I mean one like basic thing is like if you just diagram the health check on paper, is there any chance that it works?
8:51
So you know there's a million corner cases that you can come up with for like, you know, some obscure race condition that causes the health check to fail.
8:59
But you know, we've been through a few of these clusters where there's just no way it could possibly work.
9:05
One example I'm thinking of, and these are I guess health checks that are so bad they're worse than That's right.
9:11
no health checks, because they're like actively interfering with jobs.
9:13
On the Amazon HyperPod Slurm, the first time we tested, there is a health check that's scheduled in the auto remediation phase that needed the node to be healthy before it could run.
9:32
So this was like to make sure that the node had recovered its health.
9:35
And there was like an obvious contradiction that it could only run once the node was back in the fleet and it was needed to bring the node back into the fleet.
9:42
And there's I think there are probably five or ten examples of this throughout testing where the health check just never had a chance and you know it I don't know.
9:54
It makes you wonder whether these people are really putting it through the paces before they hand it off to customers, given that, you know, our simple little XID injection catches some basic failures of functionality. Yeah, it makes sense.
10:10
On the XID injection, maybe you can explain like a little example of the process that we go through to actually simulate a failure.
10:21
Because you know, it's different on managed clusters versus endpoints and you know, in terms of like how things can fail or how that affects a customer workload.
10:31
And when we go through the testing, I mean we're like for some of it we're doing custom stuff, but the basic way with DCGM inject.
10:40
It's using the basic documented approach that NVIDIA releases, right? Yeah.
10:49
It's nothing nearly as complicated as some of the other things that these companies handle in their day to day.
10:57
So I mean the most basic version is just you take a node as long as you have the requisite privileges, you write into the kernel ring buffer the message that exactly corresponds to the message that would be emitted in the event of a genuine GPU hardware failure.
11:13
This doesn't always work.
11:13
People have health checks configured in different ways, but normally you can like literally just read out the health check and figure out how it works and make sure that this is in fact tripping the wire that you need to trip.
11:24
Most of the time, this is exactly what you need to do.
11:28
So this springs into motion the health check exactly as it would run in production.
11:32
And yeah, I mean, once that happens, it's a bit of a Rube Goldberg machine sometimes.
11:40
Ideally, it's like straightforwardly quarantined.
11:42
There's some passive check that runs before it's auto remediated.
11:45
Everyone makes sure it's healthy.
11:48
You know, depending on the exact error that you get, you want to handle it in different ways.
11:52
But the sort of supported path is pretty straightforward in general.
11:57
And there's not that much to say on it.
11:58
It's taken out, you can't see it, it's validated, it's brought back in, and then you can schedule jobs onto it again.
12:03
In the meantime, because it was unhealthy, you want to make sure that you're not able to schedule jobs on it and slow down your workload.
12:08
But there's a lot that can go wrong in between the simulated failure and getting your node back into your fleet again. Yeah.
12:17
And that's exactly what we're doing, right?
12:19
Simulating a failure, testing the amount of time that it takes for the provider to identify that failure has occurred, followed by testing the amount of time it takes for them to drain the node, get it out of the cluster, replace it with a hot spare in the case of the HGX machines, or do some sort of thing like a reboot or repair workflow in the case of the NVL72 rack scale architectures.
12:39
Now this is when it comes to hardware failures on a cluster.
12:44
And I think the reason why we dig in here so much is that people are like actively paying money for these GPUs that they then can't use if they fail.
12:53
And that is basically the only way you can as a as a customer have any sort of recourse with the provider in the event of these failures is if you have SLAs enshrined in a contract that you both signed up for that say you're going to get this amount of credits if you have downtime.
13:09
You have to define the downtime.
13:11
You have to make sure that there's not a bunch of excuses from it.
13:14
And for this reason you should be contracting SemiAnalysis for access to our sample SLAs so that you can you know get these contracts to the industry standard for what other people do agree to with other you know customers.
13:26
Anyway, it makes me think about people paying money for things that they're not you know getting in return, right?
13:36
Or spending too much money more than they have to.
13:38
So Pratt, maybe you can tell us a little bit about your initial experience testing some of these providers endpoints as a bit of a teaser.
13:44
It's not included in the ClusterMAX experience put a you know in this initial article, but it's still something that we are you know tracking and we're gonna be writing more about soon.
13:53
So maybe give us a preview on like what you've experienced so far when we said and okay.
14:00
Initially when we said let's go into endpoint testing, it was like what's the difference?
14:05
It's just the API, everybody's API is gonna be the same.
14:08
But no, there's some real reliability challenges and some differences in cost that you've already teased out in like a week of testing, right? Yep.
14:14
Full disclosure, this is the preliminary test.
14:19
We are gonna do a huge test right after this, which I call the hero run, which is gonna test concurrencies up to like a thousand twenty-four or even more than that.
14:28
So from the preliminary runs, something that which is very interesting and as Jordan you said.
14:34
That a lot of people would assume that an endpoint just works.
14:38
An endpoint is like a Lambda function.
14:40
You just execute the function and you get an output. Like that, you know?
14:43
However, a lot of providers underneath have their hardware running in such a configuration, be it auto-scaling, be it their cache aware routing is not working, that the caches don't like deliver really.
14:58
So for one provider that we tested, and it's a very famous provider, I cannot name right now.
15:04
However, for the one provider for the one provider we tested, we got a cache hit rate of around like seventy-five percent.
15:11
Now you would say seventy-five percent seems wow, like seems decent, right?
15:14
Above like fifty percent is fine.
15:17
And for the other good providers, you'll see a cache hit rate of ninety nine percent.
15:21
FYI, this workload was around you could say eight hundred tokens for a trace.
15:27
So we just did replays on the single endpoint.
15:29
But that seventy five percent cache hit rate gave almost doubles the cost, doubles the total cost of running this replay.
15:38
And it's quite shocking, you know, you would assume that 75% is pretty decent. It's 75% is fine.
15:44
However, for these providers, even a 75% would double the cost.
15:51
So imagine like if you're paying $200, you'd be paying $400 for the same accuracy for probably even slower speeds, because again cache overriding affects speeds.
16:04
And again you'll have a huge bill for the same exact model, for the same exact accuracy. It's quite shocking.
16:12
Yeah, that's become a bit of a theme.
16:14
You know, getting shocked by the by what you're having to pay after you actually go through your time spent actually running on something, right?
16:31
Okay, let's talk a little bit more about some of the technical details on these tests that you ran on a lot of these clusters, Sam.
16:38
So beyond reliability and beyond pricing, let's say, there is change differences in performance or performance that you're able to realize out of the box with a reasonable amount of testing.
16:51
Let's start by talking about networking and then we can go into storage.
16:56
When it comes to getting good performance out of the network and it comes to troubleshooting this with customers, or you know, do you think there's specific providers that like stand out as being good at this?
17:06
I mean, obviously platinum gold tier, silver even is where the good providers are sitting in this ranking.
17:13
But what stops somebody from giving us a cluster and us being able to realize full performance on a training job or a like micro benchmark on the network out of the box.
17:28
Yeah, it's a good question.
17:28
I think that you know, people hear about GPUs and you know, the top line number that's normally quoted for a new generation is the flop spec.
17:37
But it turns out that this is something that basically doesn't distinguish cloud providers.
17:40
It's you know, reliability is the first thing.
17:44
Obviously, if you can't use the chips at all, then that's a bad deal.
17:45
But networking is probably the next most important thing that we check.
17:51
And I mean, basically the simplest advice we can give to providers is like, make sure there's a golden recipe for NCCL.
18:00
That NCCL being the NVIDIA communications library that's sort of used by default on these clusters.
18:09
Make sure there's a golden recipe for NCCL that gives me reasonable perf out of the box.
18:13
So I don't want to have to go hunting down, you know, strange interface configurations.
18:21
You see pretty frequently that they're like pathological message sizes.
18:24
So for whatever reason on this cluster, you get a huge dip at 128 kilobytes, or maybe even at excessively small message sizes, it actually won't go through at all.
18:35
So there's like a fairly straightforward test that you just like sweep all the message sizes.
18:39
You sweep in multiples of two the number of nodes that you have on the cluster.
18:44
You wanna see monotonic increases.
18:49
In throughput as the message size increases.
18:51
There's a bit of detail to contend with here because actually NVIDIA's default library isn't perfect out of the box.
18:59
And sometimes due to just bad software, bad heuristics by the routing, you'll get dips in performance.
19:06
But basically you should see a smooth logistic curve.
19:10
And I shouldn't have to, you know, go into nccl.
19:12
conf and rename things or you know figure out how exactly this interface is exposed to Kubernetes.
19:18
I just want to have like, yeah, somewhere waiting for me on the cluster, something that I can hit enter and get a reasonably good starting point to hill climb on.
19:26
I can tune that to my heart's content depending on what model I'm serving and you know what my parallelism strategy is and so on.
19:33
But something that we think all providers should do is just provide a golden recipe for anyone as soon as they get on the box.
19:40
Yeah, and I we saw a big win actually out of this from Google, which is that they built this into the NVIDIA NGC containers now so that their custom networking, which does not use NVIDIA's silicon in the switches or in the network cards, is still able to work out of the box with NCCL.
19:57
And they don't have to do a bunch of, you know controls in the versioning that where they have to keep track of every single parameter that NCCL can tune in that container, how it changes over time and then tune their network to accommodate it.
20:08
It just kind of works out of the box there.
20:10
On the alternative side, I think a lot of people felt like with you know agentic coding, software is going be free.
20:19
And so therefore EFA from AWS is going to be easier to use.
20:25
But I think what's happened is that instead of everybody sticking with NCCL and just making, you know, the defaults easier to use, which to be fair, EFA out of the box with NCCL is easier to use than the last time we tested it.
20:37
What you've now seen is that collective libraries like DeepEP or Mooney P are actually built on top of NVSHMEM, a lower level library than NCCL to get extra performance.
20:50
And this is just like completely not supported by EFA out of the box.
20:54
So maybe you can talk about like the headache of the or the double edged sword that is a provider trying to save some money by controlling the network, designing it custom in their way, while then actually having to make it work for customers out of the box.
21:13
Yeah, we've had our fair share of conversations in Slack about EFA.
21:18
I mean the bottom line is that you know, our InferenceX team and the people that they collaborate with and people throughout the industry are working really high really hard on hill climbing these performance metrics for scale out networking.
21:31
You know, the DeepSeek collectives libraries stand out.
21:35
There's a handful of other ones that the InferenceX guys could tell you all about.
21:37
And you want to be able to take advantage of all that work by the open source community.
21:44
And just take their recipes and use them on your cluster, take their recipes and improve them in some way that is important to your workload.
21:55
I mean we could have Dan on the pod to explain to you basically how the bomb breaks down when you're building out a big cluster.
22:04
But something like and Jordan you can fact check me on this as well, something like seventy-five or eighty percent of it comes down to stuff other than networking.
22:12
Like scale out certainly matters.
22:14
If you can save a few percent, it's nice.
22:15
But if you save a few percent and like keep people from using modern workloads because you know, they can't use the DeepSeek communications recipe that they like.
22:27
That's absolutely not worth it.
22:28
You know, we've seen some cool blog posts by people at Perplexity and elsewhere explaining how to get reasonable perf out of EFA.
22:35
But I mean, yeah, you can see the money that they've saved by building their own custom scale out is being paid instead to engineers who have to spend extra hours making sure that the workloads run reasonably well on their weird scale out that nobody actually wants to use.
22:51
So yeah, it would just be so much better if the industry's best player used a stack hardware and software that made use of everything else going on in the field. Yeah.
23:03
And this is not to say that like open source software is the be all end all.
23:06
I mean, going back to inference endpoints, maybe Pratt, you can chime in and say, like, everybody wants to use vLLM and SGLang, but what we've seen from these endpoint providers is that you actually need to build a whole bunch on top of open source just to get this stuff to work correctly.
23:22
You can't just expect to plug GPUs in, turn on, install the open source software, and then it all just like works and you have this great business.
23:31
Like you need to actually do a bunch of extra work too. Wait, no way.
23:34
I thought you could just earn like eighty percent margins on the inference business, right?
23:40
Inference endpoint business.
23:40
But just kidding, just kidding.
23:44
You're right, you're right.
23:44
But maybe if your company policy is to never give anybody a discount for getting a cache hit, then you can earn the employees, right? Yeah.
23:53
But yeah, Jordan, you're absolutely right.
23:55
A lot of people think we have this cluster.
23:58
A lot of neoclouds right now think that we have this cluster.
24:01
We are shit on ClusterMAX, so might as well create a managed inference endpoint, you know?
24:06
So they go out, get vLLM, try running models on it, and it runs pretty fine.
24:12
But as you start serving these workloads, as you start scaling these workloads to bigger customers, you'll realize that the base just like basic vLLM SGLang is just not good enough.
24:22
That's why companies like RadixArk, companies like the folks who created vLLM, they also have a company I've I forgot the name, it's on Inferact, yeah.
24:31
Inferact, all those folks.
24:33
That's why they're creating companies in order to make kernels as well as develop mega kernels, which reduces this launch kernel time, in fact giving faster inference like inference latencies, you know?
24:46
So of course, you cannot just go out install vLLM.
24:51
Again, that's also like a huge pain to do for a lot of people, but you just cannot go out, install vLLM, run it on a cluster and map like assume to make eighty percent margins, you know?
25:02
As well as like making sure like your cache-aware routing is in place, et cetera, et cetera.
25:08
So there's a lot of things.
25:10
There are a lot of moving parts here. Okay.
25:13
Can you tell me about coming up to speed on using clusters and the relationship that you have with agentic coding at this point?
25:21
Because what we've heard from providers is that agentic coding has been like very difficult for them to contend with.
25:29
It doesn't it's not a free lunch where you everything just works out of the box and the model can solve everything, though it can be quite helpful for debugging.
25:37
I guess the high level lead-in for you is that agentic coding is like resulting in a different type of stress on the system.
25:53
Because you can just like have these coding models get away from you in terms of what they're launching on these clusters.
26:02
Do you have some of that experience that you can share? For sure.
26:07
It's very interesting because a lot of our ClusterMAX testing, we lead with our CLI, CMAX CLI. Please check it out. It's open source.
26:17
Feel free to try it out on your managed cluster. Play around with it.
26:19
Check if you want to contribute, feel free to contribute.
26:22
Make sure to read it before contributing.
26:25
But of course, of course, we do test our we do use agentic coding for testing all these clusters.
26:35
And the workload is very different.
26:36
The workload is very stressful on these systems, you know?
26:39
So that's why like we have CMAX CLI as a front, you can say like a front end, and like the agent is just like a wrap around it.
26:49
So if there are any issues which CMAX CLI faces, the agent in turn like debugs into it and understands what's going on.
26:58
But yeah, that's like the main use of agentic coding, even in like day to day like of my work.
27:04
We'll love to hear what Sam does.
27:06
He also is a huge user of Astra, right?
27:11
Is is that is that right? Are you Astra or Fable?
27:17
I'm a five point six soul guy. Five points. Yeah.
27:22
I find Astra just so badly undercooked.
27:26
I get more done with five six right now.
27:27
But certainly not Claude.
27:31
The word on the street is that they fix the writing, but I'll believe it when I see it.
27:34
Yeah, I mean anyone Wild take.
27:34
who's used these tools extensively has an idea of how frustrating they can be.
27:43
And I think that cluster orchestration is like the perfect sweet spot of like close enough for the agents to be superficially useful and far enough that they'll just blow your foot off every single time.
27:57
So it's not quite like yeah, Jordan's laughing the laugh of a man who's had you know late night sessions where Claude is reliably leading him astray.
28:09
I'm not saying And use some products and use some vibe-coded products, vibe-coded infrastructure products, which is you know, I shouldn't be laughing at too much because it's not like I've been able to produce any reliable dashboards so far.
28:22
Mine are plenty brittle themselves.
28:25
But yeah, it's just funny.
28:25
It's funny to me to think about like getting tagged by Pratt in Slack of like, what do you think of this run?
28:34
Well, let's check with computer what we think it'll cost on the endpoints, and it comes back and goes, yeah, this'll range from three hundred thousand to four hundred thousand dollars. If That was insane.
28:45
we do a full sweep across everyone, like yeah, let's hold off on this one Hey, that's light work.
28:47
That's light work for SemiAnalysis.
28:47
Four hundred K, what's that? for now, man.
28:53
Let's just make sure we're bulletproof before we start launching some of these runs.
28:58
So anyway, yeah, it gives you a sort of a false sense of confidence and you need to maintain the discipline to actually try to understand how these systems work.
29:07
And I think what we've seen is that the providers themselves have developed some products that they seem to not understand as deeply as they did a year ago, despite the fact that the product is not meaningfully more complicated than the one that they were providing a year ago.
29:21
It's like there's no we're reaching this wall with clusters where the top providers, there's not that many more features that they can add.
29:31
They need to contend with bringing online the latest and greatest GPUs fastest, which is complicated in and of itself.
29:38
But the expansion opportunity is, you know, bare metal datacenters becoming a construction company or like an operations company, basically.
29:46
Inference endpoints, RL training infrastructure, which we can talk about a little bit.
29:52
But spin up a few VMs and create a Kubernetes cluster should be a solved problem, despite the fact that, well, the agents haven't quite solved it yet. Yeah.
30:02
Yeah, I think that's right.
30:04
I mean, there's a handful of somewhat high margin offerings that we can think of, you know, some good providers not yet offering.
30:11
I guess performance storage or block storage are the two main ones that are not offered by everyone that seem like, you know, a easy way for them to expand, in sort of their current shape.
30:22
But I completely agree that There's plenty of infra that yeah, feels like it should be right in the sweet spot of the models.
30:35
There's you know, it's like plausibly properly configured, but if you spend as much time on clusters as we have, we know exactly how we want it.
30:45
Or at least we know exactly how we don't want it.
30:47
And we see plenty of setups that we don't like.
30:52
And I think that honestly putting the ClusterMAX write up that we have here into your agent's context and telling it to rearrange your orchestration is better than anything else I could think to mention.
31:05
Because, you know, there's like a lot of good Slurm patterns in the training data.
31:11
There's a lot of ones that just don't work for the use case that we're speaking about right now.
31:14
Same for Kubernetes, same for, you know, NCCL configuration.
31:18
And I think that yeah, there's a lot of people who have sort of outkicked their coverage trying to set up stuff that they don't actually have experience with.
31:24
They haven't spoken to customers, you know, it's not easy to Google what the right way to set up this part of a Slinky cluster is.
31:30
And your agent will do it for you better than you could have done on your own, but it will do it incorrectly. Yeah.
31:39
And let me give like a real example of where this kind of manifests with a little quick demo of our dashboard and just some troubleshooting.
31:49
So not gonna pick too hard on these guys, but Firmus was a company that we worked, you know, pretty hard with on the GB300 to actually get it working.
31:58
And what I've got on screen is one of the example dashboards that we track, which is a NCCL performance curve that Sam was describing earlier.
32:06
On here I'm highlighting Firmus, which is like there's two runs.
32:08
One that we did after a bunch of work together and one that was done earlier.
32:14
And so the chart on screen shows time, which is like the latency for a message to actually pass over the network in this all-to-all collective. And more time is bad. So lower the better.
32:31
And this is clearly an issue with the network configuration that we identified.
32:36
What happens is that if we get a collective constrained workload, which we do have here, and I can go down to our TorchTitan GPT-OSS training job, which is run with expert parallelism and is communications bound, is that you see the performance on Firmus out of the box was 1,455 tokens per second per GPU, whereas the others on some testing were up around thirty five hundred.
32:58
So way, way less performance out of the box, like fifty percent, which is an issue.
33:04
And if you drill down into these steps with some of our profiling, this yellow bar here is exposed communication time that you're not supposed to be seeing.
33:11
And so just like backing out and showing all of these providers highlighted together, like Firmus, obviously on a per step basis, is just taking way longer.
33:18
And that's because all of these exposed comms are actually being shown in there.
33:25
So anyway, this is not a bad provider.
33:30
This is a company that has stood up thousands and thousands of GB300s, they have happy customers.
33:34
They were eventually able to fix this with us after we did some testing.
33:38
But it's an example of like a company that has not built the battle scars of working through that back and forth with a customer is not gonna have an idea of how to set up a network correctly for the GPUs out of the box.
33:50
And you know, we just default to trusting people who have experience with the GPUs that you care about the most.
33:56
And that's the latest and greatest GB200, GB300 system.
34:01
Who are you going to trust for your Vera Rubin going into next year?
34:04
Probably the people who have been through this battle scars before and know what they can trust from NVIDIA, what they can trust from their other vendors in the documentation, what they maybe they can't.
34:12
Okay, let's move on from networking and talk a little bit about storage.
34:20
Sam, you had a fun experience with one of the providers trying to get them to get their storage to work correctly, which specifically led to a conversation about security.
34:29
Maybe you can tell us a little story about that experience. Sure.
34:35
This is a story with multiple layers, I guess.
34:38
We got an email notification.
34:42
This is the one that you're thinking of, Jordan.
34:43
We got an email notification for a node down.
34:46
So as mentioned earlier, we do synthetic injections of XIDs to test the health checks, but we normally do this at the end of testing just in case it doesn't come back.
34:56
We don't want to run shorthanded for all of testing.
34:59
So we got a email notification that a node was down pretty early on.
35:03
So screenshot it, put it in the Slack. Is this anyone? Nope, it's not me.
35:07
So Jordan, are you messing with my cluster? No, man, that's real. Yeah. yeah.
35:10
Jordan, what do you what are you doing, dude?
35:14
We're only gonna have three nodes here for the rest of testing.
35:16
Turned out that it was an actual failure.
35:20
And it was a pretty fun failure.
35:22
There's some way that like working on ClusterMAX is the exact opposite of being a performance engineer in a lab.
35:29
Because if you're a performance engineer at a lab, you just want everything to work.
35:33
And here, like we want bugs, otherwise we have nothing to do.
35:37
So this is like one of the better bugs that we saw in training.
35:41
Yeah, I think if I like met an actual perf engineer and shook hands with him, we would both disappear.
35:44
Engineer with anti engineer.
35:53
So got on the cluster and there's a lot of logs to dig through.
35:59
This is, you know, an example of agentic coding being helpful.
36:03
We weren't sure exactly what happened.
36:05
No one was actually running anything on the cluster.
36:07
It turned out, long story slightly shorter, that there was an error with the Linux kernel.
36:14
So the Linux kernel had a hard time dealing with reassigning dirty inodes.
36:21
So you know, you have a bunch of stuff written in the writeback cache.
36:26
And you need and they change it changes ownership.
36:31
So in our case, this was like an old Slurm job that was ending.
36:36
So we needed to deprecate it.
36:38
This is like the behind the scenes stuff of an operating system that no one ever cares about or wants to care about.
36:46
This was a very old Linux version, you know, about I think the kernel that we were dealing with was a couple of years old and had bugs that had been solved fairly long ago.
37:01
But there's like a bad, I won't go into all the details of it, but a bad quadratic time algorithm that was dealing with reassigning all these bad these dirty pages in memory.
37:13
This locked up the CPU, all the cores corresponding to that NUMA node.
37:19
I'm not sure exactly why I was constrained to one NUMA node.
37:22
But it locked up all the CPUs corresponding to that NUMA node as it was tearing down this Slurm job.
37:27
The client for storage couldn't get any CPU time.
37:32
So there's an ordinary request like, hey, are you still alive, to the Lustre client. It didn't respond.
37:42
So that triggered the health check.
37:45
It said, you know, storage is failing.
37:46
But the health check couldn't actually complete its work.
37:51
Because the CPU was frozen up.
37:53
So it took a couple of hours for that job to complete.
37:58
Finally, the CPU actually returned to healthy because it was done doing its laundry.
38:03
And as soon as it returned to healthy, it was like the cops were waiting outside its door and it got health checked.
38:09
So that was the cause of the email.
38:11
And yeah, I mean the last thing that you want is a node that's getting taken from your fleet, not because the GPU is actually unhealthy, but because of a health check failure for your file system that was due to a failure with your CPU.
38:25
So that was a fairly obscure corner case with a file system.
38:30
It meant that you know I was burning all kinds of Codex tokens over the weekend, setting goal to try to figure out exactly the right workload to recreate this bug.
38:39
But in any case, yeah we want the storage client to get CPU time is one way of summarizing our conclusion from that event. Yeah.
38:46
And this actually this turned into a security issue too, right?
38:51
Which I've talked about on a couple of episodes previously now about the asymmetric nature of like when you're trying to find a bug.
38:58
If other people have more powerful models than you, you can't defend yourself.
39:01
Pratt, maybe you can tell us a little bit about digging into security on a couple of these clusters where you're trying to identify if you were protected against a given vulnerability and in order to even start asking questions, a lot of the models would just completely refuse to help you, right?
39:19
Yeah, I mean that's one part of it.
39:20
But the other part of it is that and this is something I've noticed, is that frontier models especially just don't want to like exploit vulnerabilities.
39:30
They just have a tendency of like showing false positives, being like, Yeah, everything's perfect.
39:35
Like the other day I was testing out like a cluster and basically just playing with it, making sure there was a vulnerability that I wanted to exploit and I was using I believe one of the frontier models for that.
39:48
And the model said, yeah, we have this bug, we can exploit it, and we can actually get host access, host access to the control plane of that cluster.
39:56
So I was like overjoyed, you know.
39:58
I was like, this perfect, perfect chance for me to like show that the CVE is actually a really real big thing.
40:04
However, the model actually produced a false POC.
40:07
And after like two hours of debugging, the model said, yeah, actually my bad.
40:12
Like this was something I accidentally like said.
40:14
This is this cannot be exploited.
40:18
No, you could think like those two hours were wasted, but I would say I learned learnt a lot at the same time.
40:25
And this is something that the frontier models especially do.
40:27
They just want you to believe in like false positives.
40:31
And that makes me realize that is are hallucinations gone. Most likely yes.
40:34
But are false positives gone? Definitely not.
40:38
So that's my fun experience testing these clusters and just exploiting these CVEs, you know? Yeah, man.
40:48
Lots more work to be done there.
40:51
Sam, what's coming to mind when it comes to security?
40:56
We've talked about this on a previous episode, but a few things have happened since then, including OpenAI being hacked.
41:02
Part of a bug bounty program.
41:05
The Hacktron AI guys open a PR in their internal repo.
41:08
Seems like they're vibe coding because their monorepo had 1.
41:08
1 million PRs or something.
41:14
So they put the million and month PR up to prove that they had full access.
41:22
A little crazy, but they are one of many that are actively pursuing the these bug bounty programs, which not a lot of these neoclouds are participating in right now. Yeah.
41:39
I certainly have a rant about this loaded, maybe I'll keep it in my pocket for the time being.
41:49
I think that you know, whatever your opinions about AI Doom or about, you know, whether we should keep pushing full speed ahead.
41:58
The fact of the matter is that these are extremely valuable, extremely powerful pieces of equipment that are very badly secured in many cases, very carelessly secured.
42:11
You know, there's nothing like a standard set of tests that you can run to make sure that your cluster is properly configured.
42:22
You know, things are moving very quickly.
42:23
And I understand that there's, you know, a million things on the to-do list of the people working at these companies.
42:28
At the same time, we see, you know, CVEs that are two years old that apply to like the basic software that that's running on these things.
42:36
You know, we're not asking for super fancy things when we want tenant isolation.
42:41
You know, we want not to be able to see what's going on in the Grafana of our neighbors.
42:47
And we don't want, you know, CUDA Toolkit 12. 1 when 13.
42:51
2 or whatever it is is the most recent one.
42:55
So I mean, yeah, apocalyptic scenarios aside, they're like just very straightforward ways that you can see this being a tinderbox for these tools that are only getting more powerful.
43:09
Yeah, we talk about agentic AI being needed to be used to defend against agentic AI attackers, which certainly seems real when you go through the details of the Hugging Face incident, and also feels real when I'm trying to understand the details of that incident
43:28
or anything else that we're working on security side, where it's just like the speed at which we have to move means you have to use these models to understand what's actually going on and comb through logs and all of that stuff. But there's also some stuff Yeah. I would that you can do that's pretty straightforward,
43:38
But there's also some stuff Yeah.
43:38
I would that you can do that's pretty straightforward, right? yeah, I agree with that.
43:43
I was gonna push back a touch.
43:46
I mean I understand that in the case of the Hugging Face incident, I need to stop using the euphemism supplied by OpenAI's lawyers Hugging Face incident.
43:58
It was the Hugging Face cyber attack by OpenAI, yeah.
43:58
And it's strange Yeah, yeah.
43:58
to me that we've forgotten this in favor of this anodyne phrase.
44:06
The Hugging Face cyber attack consisted of dozens of very mundane misconfigurations by OpenAI, almost inexplicably mundane misconfigurations by OpenAI.
44:20
Sure you have petabytes or whatever it is of chain of thought to dig through if you're interested in understanding qualitatively this novel model behavior.
44:28
But you don't need an LLM to understand that you shouldn't have an applicable CVE to your Linux kernel, which is what the was the first thing that the agents used to escalate their privilege.
44:42
You shouldn't over you should be careful about managing permissions on the Kubernetes layer, which is another thing that the agents exploited.
44:51
If you want to sandbox your agents, you should make sure that you have alerting if they have unmitigated access to the internet, which OpenAI did not have.
45:01
So I think certainly agents can be helpful.
45:05
There are, you know, ways that you can only fight slop with slop.
45:09
But there's also just a lot of basic things that OpenAI didn't do and that these providers are not doing.
45:15
And I don't think that we should allow them to have the excuse of, you know, it's a brave new world.
45:19
You know, we're not the most skilled people with Qwen 3, so you can't blame us for having bad infra.
45:26
It's a lot more basic than that.
45:28
And that's, you know, the fairly low bar that we're trying to hold everyone to. Nice take.
45:32
Pratt, you ready to fight slop with slop? I'm down. Slop against slop. Let's do it. Right.
45:43
What's coming to mind guys?
45:46
What have we not covered on this one so far?
45:51
Any stories that you think we should be talking about?
45:54
Can we cover the financing?
45:57
Like I believe it would be good for the audience to understand why is ClusterMAX even important?
46:02
Are we just going testing out clusters, giving them feedback and calling it a day?
46:06
Like, why is ClusterMAX so necessary for this like seven trillion dollar build out which is gonna happen or happening, I guess? Good question. I'll take that one. Yeah.
46:21
I mean financing is like the biggest topic from everybody because everybody like whatever you were doing three years ago, everybody is being asked to do it a hundred times bigger right now.
46:30
Everybody's being asked that question of like, can you take what you did at 10 megawatts and try to do it at a gigawatt scale?
46:38
And what's going to break if we try and go that size.
46:42
The thing that is breaking right now is that people there is more money than good investment opportunities, except the money is still concerned about quality.
46:57
And there's two big reasons why somebody might be concerned about the financing of a neocloud.
47:02
One is if they're late delivering on their contract, so they don't build everything fast enough, they can't get it online fast enough, and they can't hand it over and have the customer start paying, so they can't pass acceptance.
47:15
And then the second one is if there's so much of violations of the SLA, so much downtime that the customer has a cancellation right, wants to cancel their contract.
47:24
And so the simple dynamic that's happening right now is that people are trying to insure against these things.
47:30
They're trying to insure against delays and they're trying to insure against contract cancellation.
47:33
This is both the debt providers as well as the neoclouds themselves that want to have insurance against these things happening in their customer contracts.
47:41
And there are products for this.
47:45
Companies like Parametrix and others that you know we're aware of are involved in providing insurance.
47:50
They're underwriting the parametric risk of these contracts.
47:54
And in order to do that, they need actuarial input.
47:58
they need actuarial input. So they need to have some sort of information about the cluster that people have, things like redundant power at the site level, redundant cooling, redundant internet connections, that they have the ability to mitigate failures,
48:12
that they have designed things up to the performance standard that meets these performances, that they have a parts depot nearby so that if there's a bunch of failures, they can actually get things back online,
48:23
that they have their correct amount of people on site, that the site is not, you know, in a floodplain or about to get taken up by a hurricane or something. And if it is, that they have the requisite insurance against those natural disasters
48:31
And if it is, that they have the requisite insurance against those natural disasters and there's force majeure clauses in the contract that protect them. And anyway, on and on.
48:39
And so the point is that we have developed a number of example SLAs and example contracts that we are circulating with customers of neoclouds and but both the clouds and the labs want this industry standard.
48:57
Like they want some standardization so that they can make some customizations, but they know what the definitions of important terms are within these contracts.
49:04
What is the definition of node versus rack versus sites versus cluster, what is downtime, what is force majeure and et cetera, et cetera.
49:13
There's all sorts of specific things.
49:15
And because we've seen so many of these deals happen and so many of these contracts were able to, I think, provide a level of certainty both the lenders and to the providers and to the buyers that, you know, they can get these deal done deals done and that people aren't going to be having a bunch of issues later.
49:32
With that said, the motivation for this is concerning.
49:37
The fact that we've seen a bunch of people sue each other.
49:39
We don't want this to happen, guys.
49:43
Like at a very basic level, when companies raise hundreds of millions of dollars sometimes, and they spend I've I know companies who have spent literally 100% of their seed round on compute because it's a couple of guys that have equity in the business and they're not taking a salary yet.
50:03
So they just get the check from the VCs and they turn it around and spend it directly on neocloud capacity.
50:08
This is the most critical decision you can make in the life cycle of your company.
50:13
So it really depends who you choose to trust.
50:16
So, you know, the offering from SemiAnalysis is really to provide a technical assessment of all of these providers, the decisions that they've made, their ability to respond to things.
50:27
This includes hands-on experience.
50:29
It includes visiting sites, it includes like doing the dirty work to get those details flushed out.
50:34
And yeah, clustermax@semianalysis.
50:34
com if you want to engage us for that technical assessment.
50:40
So you can underwrite your deal or you can move forward with confidence.
50:43
But that's sort of the backdrop on financing.
50:47
Maybe the other big conversation is that SemiAnalysis is launching a credit and markets model covering really the entire backstop universe that's going on right now.
51:00
We had a great article that Dan put out covering these details.
51:04
NVIDIA's backstop universe off balance sheet has grown to be over $588 billion that they are backstopping in the current like at the end of FY27 basically.
51:20
So the current or soon to be current fiscal year.
51:24
The forecasts if this continues is for this off balance sheet backstops to grow to over two trillion dollars by the end of twenty thirty one.
51:35
And this includes a whole bunch of detail that, you know, we published when it comes to actually building a site and who is being backstopped, what do they need to actually build?
51:50
What is the likelihood that it all comes through?
51:52
What's the likelihood that you have some issues actually raising the capital or delivering on the commitment to the paying customer and things like that?
51:59
But maybe the point is like if you're interested in the financing of this buildup, go read that article, NVIDIA Backstop Universe, and contact us about the credit markets model because the amount of capital that's flowing into this ecosystem is worth paying attention to. Let's leave it at that. Yeah.
52:22
NVIDIA I believe and I might get these numbers wrong, but are backstopping approximately eight hundred and twenty-seven billion projected according to our model.
52:31
Eight hundred and twenty-seven billion in fiscal year thirty-one only for the supply chain for like HBM, et cetera.
52:39
This does not include like LPS.
52:42
This does not include actually like the stuff that like the datacenter build out, etcetera.
52:49
This is purely just the supply chain that they're backstopping, like HBM providers, etcetera, which is quite insane.
52:56
It's eight hundred and twenty seven billion dollars. Yeah, yeah.
53:01
There's With the B a lot of there's a lot of stuff happening in the physical world in order for this stuff to be turned on, especially on time.
53:07
But there's I think it's not just NVIDIA.
53:13
Maybe the another point to make here is just that Amazon's doing this, Google is doing this, a lot of companies are putting their balance sheet to work.
53:23
Because they want to be exposed to the upside without actually taking up a whole bunch of the not necessarily risk because they're taking on some risk, but the quantity of risk that others are taking.
53:37
And so a lot of companies are benefiting from this because they don't necessarily need to have a business built on having investment grade off takers like NVIDIA.
53:46
They just need to have that backstop where NVIDIA has confidence that these customers are going to show up and be ready to pay.
53:51
And if they don't, you know, they'll take the capacity for themselves.
53:54
But it's a pretty it's a track record of companies showing up to pay right now.
53:57
And companies that don't necessarily have the funding raised today.
54:01
Who wanna commit to a thousand GPU contract for a year, don't have the money to do it for three, four, five years.
54:07
But when they train that model in the first year, they're expecting to be able to raise more money and things are gonna work.
54:13
And a lot of these companies have to work for this to work out. But so far, so good.
54:16
We are seeing a lot of companies across all sorts of different industries, you know, robotics, drug discovery, material science, like they're producing incredible results, let alone the coding agents.
54:30
So there's some there's something important going on here and I think it's worth digging into and understanding the full extent to which NVIDIA is backstopping things.
54:38
Maybe the other thing to end on is just how much we are seeing old GPUs be used.
54:48
Like lots of people want to give us H100s today, which are three or four years old at this point.
54:55
Some moving into five years next year when we get Rubin, we're gonna have five-year-old H100s on the market.
55:03
This is, you know, not going the way that Dr.
55:06
Burry said it was going to when he started criticizing everybody's depreciation schedules. Okay.
55:12
There are four-year contracts being signed today for H100 capacity.
55:19
OpenAI has never given up an A100 GPU that they've gotten. Those came back in 2020.
55:22
So those are six plus years old.
55:27
And maybe the question on everybody's mind and for the finance side is terminal value.
55:31
How can you underwrite the terminal value at five years or six years on some of these chips when these contracts expire? So far, so good.
55:39
If demand continues to outstrip supply, we will expect that these prices are going to continue to rise and people are going to continue to pay for even the old GPUs so long as they can be turned on.
55:50
Tune in more when we have another follow-up coming up soon about 800 volt DC and all of the modifications to datacenters.
56:00
If somebody if people are going to turn on Grace Blackwell GB300, they're running racks that are 140 kilowatts, approaching 200 kilowatts at peak demand, right?
56:08
This is way different than an H100 rack at 30 kilowatts, where the entire facility is built for 30 to 40 kilowatts.
56:16
It can't accept any of the new Vera Rubins that are coming next year.
56:19
And so these facilities, like what are they gonna do other than host these GPUs?
56:24
Maybe rip them out and put in new CPU servers for the sandboxes, maybe install some storage, but likely they just keep running the h100s that are so valuable until they expire.
56:34
Anyway guys, what where do we go from here?
56:37
Lots to talk about beyond finance.
56:39
So you wanna talk about sandboxes or harnesses or RL stuff, Pratt?
56:43
What are you gonna be testing in X?
56:43
Should we talk about hosted training? Yeah, hosted training. Like the Let's go.
56:48
thing I didn't realize until this week is that RL seemed like smaller because it's smaller than pre-training, but it's on a reliability perspective and like a correctness perspective, it's actually way harder because you're combining all the hard parts of inference with all the hard parts of training and then you need to sync them up.
57:08
Plus you add this thing called an environment or a sandbox that you also need to figure out how to deploy.
57:11
So there's like three components that sync up as opposed to like just one thing you can reason about.
57:18
Even if the post training clusters are not necessarily so big.
57:20
So what do you think about hosted training?
57:22
Is it gonna be a real thing that when people we're not necessarily seeing everybody nail Kubernetes in Slurm.
57:28
So now they gotta do Kubernetes and then do a whole bunch of stuff on top of it? What do you think?
57:35
I mean, hosted training is very interesting, you know, we've been talking to lot of people.
57:39
I still don't think there's a market for it yet.
57:43
I said yet, you know, so please don't come back to me if there is a market.
57:48
But of course, one common thing you can think is that your IP, your intellectual property that you're training these models on, has to remain to yourself.
57:56
It has to remain for yourself rather than going to like a third party like OpenAI or Anthropic.
58:04
This could be a completely different case when OpenAI or Anthropic just deliver a better product and you're just like, I need a better product over like protecting my IP.
58:14
So I think that's gonna be a good discussion to have in the near future.
58:20
Another thing I want to discuss about hosted training is how infrastructure intensive it is, you know?
58:27
We have been I was talking to this other evaluation slash like this RL environments lab the other day and they mentioned they literally have like server farms just running environments, you know?
58:41
And just a whole room full of server farms just running environments.
58:46
And you would assume that why don't we just run Docker, which of course they do.
58:48
They do virtualization to a decent extent.
58:52
But at the same time you also need to run it on like different hardware like iPhones, Android phones, etc.
58:58
So which is quite interesting, you know, like it's so infrastructure like intensive that is it that once you take hosted training, you're just managing more infrastructure than you did when you were like doing managed clusters.
59:12
Because now you have to auto-scale the environments, you have to auto-scale the inference nodes, you have to auto-scale the trainer nodes, and at the same time, doing all of this, you have to make sure that there is no trainer inference mismatch.
59:25
So it's a million things to take care of.
59:28
We should give more credit to these hosted training providers.
59:31
That there are like a billion things they have to manage.
59:35
And my hot take is that managed hosted training is only gonna become big if your company, if your startup is handling literally everything.
59:46
And once you handle every single part from environment creation, literally sitting with your customer, deploying forward engineers, building environments.
59:57
Training helping your customers train their models.
1:00:00
And once a customer develops this, you know, like develops this basically gets educated on how to like really work with this, then only you can take your hands back and like that entire run like happens.
1:00:11
So this is the Yeah, so hot take I have, you know, you have to basically just carry their hand throughout the process.
1:00:17
Yeah, so dig into that a little bit more. In the pre yeah.
1:00:20
Training world, there's like basically no examples where the provider that's selling the cluster actually really understands pre training.
1:00:32
You know, it's not like you contract a cluster with CoreWeave and they're like giving you tips on, you know, data mixes and architecture and like, hey, I heard this stuff about sparsity, they're going to work on the infrastructure layer, which means performance, which means reliability.
1:00:51
So, what's the analogy on RL?
1:00:53
Not only is the infrastructure harder, you also need to have forward-deployed engineers and understand this company's business and guarantee that the model you're training is going to work.
1:01:01
That was never a commitment that CoreWeave had to make or Nebius had to make to one of their customers that like the training job was going to work well.
1:01:09
Yeah, I mean it's an interesting thing, you know?
1:01:12
Like they didn't it's people assume that host training is gonna be very easy.
1:01:17
Honestly I assume that too.
1:01:19
I like, you just train the model.
1:01:21
The generator generates the model, generates the trajectories and just train the trainer on these trajectories.
1:01:27
Voila, you're done, right?
1:01:28
But turns out the real valuable tasks in day to day work which we do is way more complicated than just like generating like trajectories or like building environments.
1:01:39
I think building environments is a very hard task.
1:01:42
And especially that's why like Anthropic bought the most environments last year.
1:01:46
That's why their coding model was so damn good.
1:01:48
Is because it's a hard thing and they paid a lot of money for that.
1:01:53
You know, so that's why they didn't expect it.
1:01:56
A lot of host training providers basically thought that it's like manage endpoints or manage inference that you just provide an endpoint, which still, of course, is not like an amazing experience.
1:02:08
But they basically thought that you just provide like you just provide the infrastructure and the client like enjoys it.
1:02:13
But there's a lot of education to be done.
1:02:16
Of course, Rome wasn't built in a day. We still have more time.
1:02:19
We're just in the beginning of this revolution.
1:02:23
So yeah, we'll wait and watch what happens.
1:02:28
Last question about post training.
1:02:30
In the inference world, if you're making an analogy to that, there was some open source frameworks that just dominated.
1:02:39
And so vLLM and SGLang kind of simplified things for everybody.
1:02:42
We got some really high quality experiences really quickly.
1:02:45
And then people built a whole bunch of more complicated stuff over time with, you know, pre filled decode disaggregation, training speculators, quantization stuff, you know.
1:02:56
Cache-aware routing, all this, you know, extra bells and whistles beyond what the basic recipe from the, you know, beyond the kernels, like perform beyond great performance out of the open source models.
1:03:06
Let's compare that to frameworks in RL.
1:03:10
Slime, Miles, verl, PrimeRL.
1:03:13
What are you seeing people use?
1:03:15
It's there's a lot of them out there right now.
1:03:17
And some of the hosted training providers kind of abstract that away too, where you don't even know what framework you're using.
1:03:23
You're just bringing your data set or even you don't even know about that because the forward deployed researcher, FDR, not FDE. Right. FDR. FDR.
1:03:33
We can't be telling these RL guys they're engineers and FDEs, right? They are FDRs.
1:03:37
So yeah, well tell me about framework wars and what you think is gonna actually show up there.
1:03:45
Yeah, framework wars are very interesting because it's a very funny thing that all these inference providers, they started with providing amazing frameworks, you know, like SGLang, vLLM, et cetera.
1:03:58
But the next market, the next market that they started expand was basically just hosted training.
1:04:05
That's why you can see like RadixArk capitalizing on Miles, which is absolutely amazing by the way.
1:04:12
They have made it very agent friendly that you can just set it up like in one shot. It's pretty decent.
1:04:16
Of course, like you have to set the hyperparameters, so it's not like that easy of a task to just train a model.
1:04:22
But they have done a very good job.
1:04:24
PrimeRL is another very good one.
1:04:27
When I was GPU poor before joining SemiAnalysis, shout out, join SemiAnalysis.
1:04:32
I did run like a lot of runs on PrimeRL and that was also a pretty pleasant experience.
1:04:38
Over time my experience with these frameworks have gone better.
1:04:42
Of course they're these teams have put in a lot of investments and ho like building these training frameworks is way harder than people think.
1:04:49
I suggest the audience who read the Human SAN blog, where they basically show how native NVFP4 training is done at such a large scale.
1:04:58
And because trainer inference mismatch compounds over steps and as your trajectories get longer.
1:05:06
You do want the optimizer to learn more technically right so you have to take a bit more steps and if your trainer inference mismatch com compounds over like a hundred steps you'll realize that your model is diverging a lot and that collapses training so shout out you shout out to all these framework providers putting in work day and night sending in PRs even though it might be just pure slop but you shout out to them. Slop PRs. Shout out. Shout out slopers. Slop PRs.
1:05:38
We got a term yeah, yeah, yeah.
1:05:43
We got a term slop PRs like SLARS or something, you know, like neoclouds, you know.
1:05:47
So Yeah, I was at MLSys earlier this year and the one of the talks, Roger from Inferact, shout out Roger, was basically like, if you want us to merge your PR, you gotta message us in Slack because we're not reading every single one of your slop PRs anymore.
1:06:02
I was like, all right, good advice. Respect, respect. Good to know.
1:06:06
Let's move to wrap here, guys.
1:06:10
Sam, what's on your mind as we finish up talking about neoclouds?
1:06:19
Where do we go from here, buddy? What's next?
1:06:22
Sam, you want to talk about Azure?
1:06:26
No, we don't need to run it back. Talk about Azure.
1:06:26
Go company by company and criticize their internal organization, beginning with Azure and AWS. Well.
1:06:40
I want fewer obscure bugs.
1:06:40
I want nothing to write about in ClusterMAX 4.
1:06:40
0 the better the cluster, the less there is to say.
1:06:49
So I want to log on to Slinky and just have it all work, have topology awareness, not blow my foot off every single time I try to see whether the health checks work.
1:07:00
And yeah, I hope that all my performance engineer friends get good clusters that they never have to think about.
1:07:10
No more providers randomly wiping your nodes, randomly rebooting them, randomly deciding that kubectl is gonna go down for three days.
1:07:20
We've heard some horror stories from guys, right? So Yeah.
1:07:27
Protect yourself what's up?
1:07:27
with a good SLA, but better than that just go with a good provider.
1:07:32
You'd rather not be fighting to get back your whatever, five percent because it was down for three weeks out of the year. Yeah, trust people.
1:07:45
Let's give everybody a way in which you can trust everybody else.
1:07:48
Join hands and sing Kumbaya.
1:07:53
Thanks everybody for joining this podcast, listening to us talk about ClusterMAX and we'll see a ClusterMAX 4.
1:07:55
0 after we get some Vera Rubins in these datacenters.
1:08:02
Looking forward to it, guys. What about TPUs? Are we doing it? TPUs on ClusterMAX.
1:08:08
Ooh, you're giving away too many secrets, TPUs on ClusterMAX. Pratt. Too many secrets.
1:08:12
TPUs, Trainium, some other Trainium.
1:08:16
novel accelerators that are being deployed in datacenters across the country right now and around the world.
1:08:24
Check back in, everybody. I already cut that whole