0:07
Hi everyone, welcome back to uh GTC 2026.
Search this video
0:09
I'm Jordan Anos uh on behalf of SemiAnalysis and Sale Media.
0:15
We're here with Kav Ul Shah of Pebble.
0:18
We're going to talk all about GPU optimization for data centers, energy efficiency, things like that. Welcome. >> Thanks. Thanks for having me.
0:25
>> All right, let's start with a little overview.
0:26
Can you uh tell everybody a little bit about Pebble and maybe what's interesting about GTC? >> Yeah.
0:32
Um so at Pebble, we work on uh data center energy optimizations.
0:35
Uh so what that means is we help uh data centers uh produce AI factories uh produce more tokens per watt.
0:46
Um and there's a few different ways we we do it.
0:48
Um and uh we've been able to like demonstrate uh uh our optimizations on like inference clusters and training clusters. So, yeah. >> Awesome.
0:58
So, why don't we walk through I guess the different ways and inference versus training?
1:04
Um maybe you can pick where to start, but at a high level uh you know, my understanding is that lots of people are are worried about uh provisioning enough GPUs to maximize all of their power because there's a very big difference between the power capacity of a data center versus the IT load versus the actual utilization, right? >> Yeah, that's right.
1:21
And uh that is what we uh discovered when we look at all the observability and telemetry of these uh GPU cluster is one of key insights from our work is uh most of these workloads uh running on like GPU clusters, they the relationship between power and performance is it's not linear.
1:42
Uh so often times like you you would crank up like you'll think you would crank up like more power in a in a GPU uh and produce like more tokens, but there is a point on on sort of like the power performance curve where it actually starts to like diminish or saturate uh where you're essentially consume your GPU clusters are consuming more power but not producing as many tokens.
2:03
And that's the inefficiency that we exploit through our work.
2:09
>> Can you talk about why that might be the case?
2:10
Are these you know, memory bandwidth constrained workloads or these long context, you know, prefill compute constrained workloads?
2:18
>> Yeah, so with inference there's there's a few few reasons I would say the first one being just like how sort of like power, voltage, and clock frequencies work in a in a GPU cluster.
2:29
They're like the relationship between like voltage, power, and clock frequencies is is non-linear. So that's that's one.
2:38
It's more like physics defined.
2:38
And then there is the workload itself, right?
2:42
Like inference workloads are memory bound.
2:46
So often times like the power that is being used is to read model weights into HBM.
2:56
And not not for like it may not be like using the SMs may not be be utilized by prefill as much.
3:07
So that is also like one of the reasons why we see these inefficiencies where the decode stage is consuming a lot of the power and the SMs on the prefill is just kind of waiting for the decode to be ready. >> Yeah. >> Yeah. >> Makes sense.
3:22
So maybe can you walk me through how Palpal helps solve that problem exactly?
3:27
>> Yeah, so one of the ways we do it is for every workload we essentially let's say running on like Kubernetes or Slurm cluster, we we look at the characteristics of the workload.
3:37
So what kind of workload it is, what is the load on that workload.
3:43
So some of this is also load dependent cuz inference workloads can be sort of like bursty, spiky, random load sometimes.
3:50
random load sometimes. so uh, some of like load is one of the factors that we look at and then, uh, we look at we essentially try and like plot the understand the relationship between the workload, the power, uh, and
4:06
performance, uh, in terms of like tokens or latency and, uh, try to sort of like find a point on that curve where we could adjust, uh, our levers are sort of like power and clock frequency where we can dynamically, uh, cap the power and the clock frequency of these workloads, so. >> Okay, so the scenario, I guess, is that
4:24
>> Okay, so the scenario, I guess, is that um, a given cluster is over-provisioned, it has more GPUs than is necessary to serve the workload at the, you know, steady state that you guys are provisioning for and then you're able to tune individual GPU nodes or maybe the PDU in data center facilities, like, how do you actually go about implementing the power capping that would help for this?
4:46
>> Yeah, so we, uh, we're able to do it at the GPU level, uh, so we, uh, essentially adjust, uh, like, uh, the power caps and the clock frequencies, uh, per GPU, uh, of the of that workload.
4:58
So, if it's running on like a single GPU, it we would just apply our power caps and clock frequency to that single GPU and the same with like a multi-GPU uh, workload as well, so.
5:11
Um, in terms of like, uh, most of the observability, uh, that we, uh, get, the telemetry data that is either coming from like say, vLLM metrics or SGI and whatever sort of the inference server is and then, uh, uh, the power consumption data is coming from like NVIDIA GPUs and their, uh, some of the exporters as well, so.
5:32
>> Okay, makes sense, yeah.
5:32
Can you talk about actually how this gets implemented?
5:35
So, would somebody who's running vLLM or SGI and need to be running a sidecar container?
5:39
Would they be using the exporter metrics and and nothing else?
5:44
Is there like some node level controller that you guys deploy?
5:50
>> Yeah, so it is essentially like a helm install on your Kubernetes cluster.
5:55
So it spins up a bunch of like Damon sets and Kubernetes objects on on your cluster so that uh uh it's able to like read some of the telemetry and essentially calculate what the optimal clock frequency and power cap is for that workload. So. >> Cool. Awesome.
6:15
I I was at a talk I I didn't read the actual paper but at NeurIPS this year where they were walking through how certain kernels would perform at different levels depending on the power cap. Uh-huh.
6:29
I'm wondering if you guys have explored how this might impact the actual performance of a given workload and if there's a way to tune you know, the workload itself, the the kernel based on the power cap.
6:41
>> Yeah, so that's something that we we certainly like we haven't uh gone down to like the actual code of the kernels uh specifically.
6:47
I think that is some some that is where we want to head eventually where we optimize some of these kernels for energy and what.
6:56
Um but it is something we're more at the systems level at this point but certainly want to dive deeper more into like the kernels as well. >> Cool.
7:06
Is it Is it dynamic in that the knobs or the levers that you guys are able to control would be able to be adjusted based on an inference workload changing, you know, some diurnal pattern through the day or you've got, you know, a bunch of uh batch jobs that are scheduled at night whereas the real-time jobs or people hitting the endpoint during the day or something like that.
7:30
>> Yeah, so we're we're constantly um sort of optimizing power caps and the clock frequencies.
7:35
So it is something that our uh system is sort of like learning uh throughout the day, so uh when when someone installs us in their cluster in their Kubernetes cluster, we uh we take maybe a few days to essentially learn the characteristics of that workload before we apply any power caps cuz we want to be we want to be sure like we don't want to uh uh mess up their SLAs or violate some of their um uh compliance SLO compliances, right? So >> That makes sense. >> yeah. >> Mhm.
8:08
Is there one thing we've been talking about is well, all that we've been talking about really is inference.
8:11
Now, we're talking about the cluster itself, so is there more to talk about maybe on the training side or maybe on the data center infrastructure side?
8:21
>> Yeah, so on the data center infrastructure side uh there is uh like uh something that uh we've been sort of we've also been exploring the idea of like flexible power uh I in the in the data centers where uh if right now like most of the data center designs uh designed are uh sort of static consumers of power from the grid.
8:43
Uh one of the ideas that we're exploring is like what if these data centers are sort of grid responsive and uh the whole idea of like flexibility uh as well.
8:54
Uh there's 100 gigawatts of flexible power uh out there uh in the US in the US in the US uh and uh uh if the data centers can demonstrate that they can be flexible and that is they can curtail power for certain peak periods in the entire year, um they can get access to this uh 100 gigawatts of flexible power for certain tradeoffs.
9:14
Um and uh one of the things where one of the products that we're working on is like how do we make these AI clusters or these uh AI workloads sort of like responsive to the grid where we can essentially lower their power consumption without impacting uh SLOs or with minimal impact on performance.
9:36
>> Yeah, it seems like a huge uh thing to decide if you guys to get working on that and uh yeah, see some of the results.
9:42
So, uh yeah, I appreciate you coming in today.
9:43
Uh we got to wrap here, but um again, this is Jordan Novalis for Cell Media and SemiAnalysis at GTC 2026.
9:50
And thanks a bunch for for coming in having a chat. >> Awesome.
9:53
Thanks so much for having me yet.