0:04
Hello everyone.
Search this video
0:04
Welcome back to Something Else Weekly where we are here to anthropomorphize AI this week.
0:07
Doug, how you doing, man? >> Good. How are you guys?
0:13
I haven't uh seen you guys in a bit.
0:17
>> Yeah, I haven't seen you in a while.
0:19
We've been uh we've been sitting here just testing NeoClouds. Sam, how you doing? >> Good.
0:24
No, no pressure on a podcast with Jordan and Doug, but I'll do my best.
0:30
>> I think you're going to do just fine.
0:32
So, uh, look, I think today's, uh, today's podcast topic we're going to talk about is, uh, one that's probably near and dear to Sam and Jordan's heart, which is, uh, security.
0:40
These guys are the guys who do Clustermax.
0:42
And I think, um, you know, we're going to talk about Hugging Face later, but I guess something that is maybe underappreciated is the huge vendors of all these labs is the Neoclouds, right?
0:52
the Neoclouds, right? the NeoClouds being people selling GPUs as a service and um you know one of our core goals of cluster max has always been like you know a GPU isn't a G just a GPU the providers do make quite a bit of difference and you know we talk about uptime reliability all these kind of
1:08
thing on the SLA but one thing that I don't think we ever talk about is security um I've heard some horror stories but I guess from Jordan and Sam's perspective you know how secure are these things and like how does that how much does that influence you know the cluster mac rank cluster X rankings. >> I think there's a huge variability which
1:26
>> I think there's a huge variability which is what we explore in this article a little bit.
1:29
Um there are some NeoClouds that are getting to the point where they operate like hyperscalers and hyperscalers really set the bar when it comes to enterprise security because they've been serving everybody from like the top fortune 500 companies to federal governments all around the world and they've been held to this standard for a while.
1:44
But only certain NeoClouds are actually getting to that standard.
1:48
Whether it's like the physical security of the data center or it's how they run the operations of the bare metal machines, how they design the network or even up into like the orchestration layer and applications.
1:57
And what we demonstrate in this article is a few ways in which this can fail.
2:01
Uh maybe we can go through some of those anecdotes, but it's pretty uh it's yeah it's pretty it's look people it's a counterparty, right?
2:12
Like when startups are raising 60 70 80% of their venture capital and spending it on GPUs they are really putting a lot of faith and trust in these neocloud providers and some deserve it and some really do not deserve that trust when it comes to security. Okay.
2:30
Well, quick name your most insecure uh I guess unsecure uh sorry, not insecure.
2:34
Insecure is another another topic altogether, but uh name uh your most unsecure Neocloud story.
2:39
Now, you should anonymize it.
2:41
You shouldn't say who it is, right?
2:43
But >> we're going to name and shame in Clustermax 3 coming soon for sure.
2:45
But um yeah, uh quick horror stories.
2:48
My least favorite was one in Asia where we got a cross-tenant remote code execution vulnerability which meant we got two separate tenants.
3:00
I was able to write something into the logs in one of the tenants and read it from the others uh which we saw from all the other tenants could be inference providers on open router like this was a provider that was serving inference on open router and their tenants were doing the same.
3:18
And so this is anybody's open claw or their uh you know coding assistant like a pi or a open code or client or something like that if they are using um you know this uh provider's endpoint on open router they're conceptually exposing credentials.
3:38
Um that was that was my least favorite one.
3:40
Sam what's uh what's yours?
3:43
Yeah, I think the the front end graphana trying to filter out all the data that they were um exporting identically from all the tenants um in the cluster was was pretty impressive.
3:53
Um and I think that that brings up something that's worth emphasizing which is that we do think that hyperscalers generally set the bar in terms of security and I don't want to say there are no hard engineering problems that you need to solve to serve a secure cluster.
4:06
Um, but the stuff that we're calling out here, the stuff that we went through in the article and, um, the stuff that we'll be particularly harsh about, um, once we publish this Cluster Max 3 write up is like not anything Galaxy.
4:18
We're not interested in you inventing novel cryptography techniques.
4:24
Just like do the basics correctly.
4:26
Um, make sure that tenants are properly isolated.
4:29
Make sure that you're following all the very basic security best practices.
4:33
Um, and everyone will be much better off.
4:38
>> Yeah, you heard us here.
4:38
Be secure and do the basic things.
4:41
So, but okay, I actually kind of want to take the I kind of want to take that uh the next step from that because I think the thing people um you know, we've been talking about security a lot.
4:52
I think this is like a really good time to bring up Clustermax, but the thing that everyone's really talking about is this uh is the hugging face hack.
4:57
And uh Rune famously tweeted at the same time, hey, just because I had, you know, broke all the sandboxes and they were, you know, you're they're screwing around with the Artifactory message board.
5:08
Um doesn't mean the weights were out of the data center.
5:12
Uh help me understand what this would mean for let's say cuz I mean like Anthropic is actually let's say using some lower like silver or gold tier, right?
5:21
Or a silver tier provider. How would that happen?
5:24
Does that mean the model weights or you know the model itself could be uh become insecure?
5:26
you know could see other endpoints within a cluster.
5:31
Do you think it could like you know break the sandbox itself like I able to essentially say hey um I I there's able to have access walk us through that because I think this is an under appreciated part.
5:40
We're just focused on I mean we see this over and over where everyone's so focused on the end product they forget about the infrastructure and this is a huge layer of infrastructure security and it seems like there's you know black guys everywhere. Walk us through that. What could happen?
5:55
Give us some doomsday dude.
5:57
anthropomorphized an evil an evil actor evil uh bad opus. >> Yeah. Okay.
6:03
So, I mean, okay, I'll I'll give you uh a rough timeline of what we think happened just based on what we've read from Hugging Face blog posts as well as OpenAI and uh and then we can talk about like some of the implications here.
6:16
So, taking a step back, our recommendation to providers in the cluster max rankings is literally just keep your stuff up to date.
6:23
If there are existing vulnerabilities that have been described publicly uh in you know popular software this could be uh anything from like Docker to the Nvidia driver to Kubernetes or anything the Linux kernel um clearly people and in this case agents can just like read that description and then build an exploit from it.
6:45
It's not uh you know it's not anything more strict and serious than just keeping stuff up to date.
6:52
But that that's the problem.
6:52
So this reveals itself in the hugging face openai situation because basically what happened is is two things.
6:58
One stuff isn't up to date and then two stuff is incorrectly configured or they're not using the tools that they have at their disposable at their disposal to keep things secure.
7:08
And I'll focus on hugging face to start.
7:10
So hugging face has this data sets API.
7:12
You can call it and you can get responses which include the data sets that you might want to use.
7:17
A data set could include something for training a model.
7:20
But in this case it includes uh a data set for an eval called cyberjim that uh an openi model really wants access to.
7:27
This this part comes later.
7:29
But from hugging spaces perspective all they're doing is running this data API. It runs on kubernetes.
7:33
This kubernetes cluster could run on a hyperscaler like AWS using EKS.
7:37
It could run on a Kubernetes cluster they built themselves.
7:40
It could run on core debious or any of the other Neo clouds.
7:43
We don't actually know where it was running.
7:44
But the point is that that Kubernetes cluster serves up this API which then uh an agent in this case an attacker gets access to because it uploads a malicious readme to the data sets API you know pretending like it's uploading data uh and the read the readme gets a um read access to proc self environment and it returns all the credentials and source on that worker.
8:11
So now the person who's uploaded data can see the source code the worker's running and they can see the root level credentials on that worker pod on Kubernetes.
8:21
This starts a cascade where you can then use those credentials to exploit get full control over the pod and then start slowly taking over the hugging face infrastructure so that you can search around and and try to find access to the raw data sets that they're looking for. Right.
8:34
Right. Um, in this case, uh, it demonstrates a few interesting things because the agents, it it was clear it was an agent that was attacking them because it was probing everywhere they said and it only took them 13 hours to
8:48
go from the malicious read me uploaded to the data sets API to getting cluster admin privilege across the cluster and they claim multiple clusters which implies that the agent is moving laterally across the infrastructure like moving from one cluster to the other. Um
9:02
moving from one cluster to the other. Um the reason they're able to do this and the misconfiguration of this is because hugging face has no admission policy which is a configuration parameter in Kubernetes that you can use to reject a a pod with root privilege or to mount host path which basically means um they
9:21
should be in these production clusters rejecting any pod like meaning any container or any workload that tries to get scheduled on the cluster that tries to get root access to the underlying workers and that tries to read from the host path meaning like read from the root logs or proc self environment in this case and get credentials. Um so two
9:39
this case and get credentials. Um so two things can fix it in this case and one um if they were keeping uh I believe that the HDF5 data set vulnerability in their data sets API was like a zero day that the model found so that that's that's kind of a little crazy but if
9:58
they were keeping the Kubernetes cluster secure and running it with these configuration parameters on it um then the model even if it had gotten full privilege and full source on the worker wouldn't have been able to escalate to get root on the underlying host and to start to move laterally in the environment. So I guess the point is
10:14
So I guess the point is like if you are hugging face in this scenario and you are trusting a provider who might run the Kubernetes service on your behalf, you are trusting that provider to potentially educate you on the concept of an admission controller or to actually implement it for you.
10:28
Um I think the you know we can go through more about OpenAI.
10:36
does does a lot of similar stuff where the agents are uh escalating to root privilege in the OpenAI infrastructure themselves.
10:43
In this case, exploiting a vulnerability in the Linux kernel which was publicly documented.
10:49
Uh I believe it's called Fragnesia.
10:51
We've tried to look into this, tried to figure out how they did it. Not sure exactly how.
10:54
Um and then there's a couple of other things where obviously people are making a big deal out of the agents coordinating which is a big deal.
11:02
Well, Sam, maybe you can give your your take on on that at like a higher level, but for me, like the practical stuff is like first of all, keep your stuff up to date, which in this case, the OpenAI Linux kernel is out of date.
11:13
And so there's a publicly disclosed vulnerability.
11:14
It's pretty easy for an agent to download the reference like description of this and then exploit it.
11:20
And then two is like if you're trusting providers to run clusters for them for you, you have to trust that they're going to run them well and they're going to know how Kubernetes works and they're not going to mess things up.
11:35
Sam, what's your take on uh on the maybe the high level of like the agents coordinating in the swarm that told me some interesting stuff about uh I don't know just what's your take? >> Sure. Yeah.
11:47
Um, I mean, I think it's excellent to have that technical background on the, you know, blow-by-blow details of the exploit because I mean, perhaps it's a fault of my own tuning of my Twitter algorithm, but I feel like I'm seeing so much like extremely granular analysis of um the behavior of the models and not enough analysis of like the actual security infrastructure that they exploited.
12:12
Um, obviously both are interesting to some extent.
12:17
Um, but I think this is a preview of the future in the sense that, you know, these are these models are extremely hard to reason about in some ways.
12:24
Um, they're alien intelligences doing things that humans couldn't do, but also they're taking advantage of like very quotidian chinks in the kernel. Um, chinks in the armor.
12:33
Um, you know, a problem with the Linux kernel.
12:37
Um, all all sharing credentials for the Artifactory service.
12:40
Um, you know, bad Kubernetes security policies on Hogface's part.
12:44
Um so yes these are like strange emergent behaviors in some sense but also like they can only take advantage of like ordinary bad human engineering practices.
12:54
Um, as far as the behavior of the models themselves go, um, I think one thing worth mentioning that I haven't heard discussed enough is that these are, um, these were models, um, RLD to hell to be just like extremely relentless, um, and keep pursuing tasks even after like they were um, other models would deem them impossible and also fine-tuned to be more uh, willing to collab to collaborate with other agents.
13:23
Um so this isn't necessarily an indication of like how uh 56 soul would behave in goal mode.
13:31
Um these were models that uh you know had been incentivized to behave differently.
13:38
Um that's why they uh found it so natural to collaborate on the artifactory directory.
13:46
um >> hackers this all time goal seek you know like all time right I mean they were like do not stop make no mistakes do not quit you know so yeah yeah maybe it's it's crazy that like you can elicit behavior like this at all but it's it's worth making clear that this is not necessarily the default behavior.
14:07
So, for example, the the self-sacrificing actions of some of the agents where they were like um you know, willing to drive their own probability of success down to zero um to I don't know the exact details of it, but they were um yeah, trying to make sure that the traces of all the other agents weren't discoverable by the greater so that everyone else could get good scores.
14:28
It's sort of bizarre that they were willing to um forego the any potential of their own rewards um in the service of the swarm, but um yeah, the these were agents specifically trained to do things like that.
14:41
So I don't I don't know how much it reveals necessarily about the behavior of cyber security >> kind of related actually.
14:46
I want to um I I kind of am interested.
14:50
There was this um do you see that like unsafe model?
14:54
Like essentially someone took Quen and then they they did a fine-tune to make it like uh specifically focused for red teaming.
15:01
I was like what the and I know everyone was like please do not do this.
15:05
Like that's the overall X Twitter theme feed.
15:08
feed. But it's kind of crazy like you know are we going to see like almost um you know okay let's say intelligence hits some scale there's open models that you can have that and you can essentially start to do fine-tuning and
15:19
RL that like gets these like you know kind of psycho models if it makes sense right where it's like hey man I need you to attack the infrastructure and do not stop right or you could have like I need you to glaze up the user and do not stop for to maximize reward. I'm just
15:31
I'm just wondering here um is this going to be like a trend that we continue to see?
15:38
Like it's kind of an interesting one to be honest with you.
15:42
>> Yeah, you're you're talking about um the ablation of the uh or obliteration.
15:45
ai release of GLM53, I think.
15:51
Um yeah, I can I mean we did talk about this in the article, but I can give a hot take right now.
15:55
Um, so I'm going to take the position that this is a good thing because it is a warning shot to everybody that anybody can do this.
16:07
Um, not necessarily because I think it's a good thing that this model is released or that people should be using this model which is like the obliterated version of GLM53 where they um, train it to remove the safeguards and do offensive cyber red teamming.
16:21
But it's it's just a very clear uh demonstration of what is obviously possible today when people release openw weight models.
16:33
And we need to be contending with the fact that anybody can do this in private and not host the model.
16:36
Um I'll talk about two things in our experience uh which in the hugging face open AAI saga hugging face claimed multiple times that they got a bunch of refusals from Fable and Opus and from uh 5.
16:49
6 six soul when they were trying to defend themselves from agent attack because they were talking about cyber security concerns.
16:57
So if they talk about getting attacked and it's related to cyber security stuff, the model just refuses to help them because of this double-edged nature of any of this cyber security stuff, right?
17:07
If you if you want to understand anything cyber security related, that kind of implies that you're going to be able to exploit it on somebody else but also to defend yourself, right?
17:13
Um, and so I I mean they had to go with GLM 5.
17:17
3 to help with a bunch of their analysis.
17:21
We are having the exact same experience doing this work, right?
17:23
We're we're trying to figure out if these neo clouds have issues or if they have stuff up to date and you can only test things from the customer's perspective.
17:31
You don't have full like root access to check if a version of the virtual machine runtime is like up to date.
17:37
is like up to date. But if you run a PC exploit and then you see that the exploit works now you know it's not up to date for example right and we have to write a bunch of these exploits by hand and dig into it ourselves directly you know back to the stone age um strictly because the models won't won't
17:55
help us do it even when we are approved in the cyber security programs of some of these companies that we're working with like we had alpha access to a few of these models and we were approved for cyber security stuff and we're still getting refusals which means it's like trained into the weights it's not classifier or filter that's refusing us. So, I think it's good that it's frankly
18:11
So, I think it's good that it's frankly that they're demonstrating that this is possible with the open weight models and we have to contend with the fact that this is the world we live in right now.
18:20
But of course, it's scary that like um you know the demonstration is on display for people.
18:27
Yeah, I think I think it's a very under discussed question and it's an empirical one ultimately to what extent these models in practice will be used by attackers versus defenders.
18:38
Um I mean you can do this pie this guy theorizing about you know the equilibrium that we're going to arrive at.
18:44
Um I mean the the day-to-day experience of us is like trying to use Fable to figure out whether someone who's using this cluster is going to be secure or not and we can't do it.
18:54
And I know there are a lot of people trying to do very mundane things that are just like pretty straightforwardly good to be done that um they can't with these models.
19:02
Um and yeah, something like this post train of Quen um moves the equilibrium a bit.
19:07
You know, a um stronger weapon in the hands of the defender means that maybe you're a bit more willing to or excuse me, a stronger weapon in the hands of the attacker means that maybe you're a little more willing at the margin to give a better tool to the defenders.
19:20
But yeah, it's just it's just not clear to me that um I mean obviously if you ask fable to like half the NSA, it should refuse.
19:31
Um but like the safeguards as a general practice, it's just not I'm not convinced that they're uh they're a good policy in general as opposed to just, you know, letting uh letting all the other social safeguards that keep us from tearing [snorts] ourselves to shreds function.
19:48
>> Just seems like KYC helps a lot.
19:48
Like if you KYC the customers and you're tracking who's doing what, this seems like it's going to do a lot of of help.
19:57
But that that sort of implies that a big chunk of the market which doesn't want to be KYC and is currently using these endpoints is going to go away.
20:04
So um I don't know that that seems like an obvious first step that's going to happen. Uh we'll see. >> Wow.
20:16
So the gang is against open models is what I'm hearing. I'm just kidding. [laughter] Um, no.
20:20
So, I actually uh maybe maybe okay. Sorry. And you know what?
20:24
And I know Akos is going to clip that and screw me on Twitter [laughter] on X, man.
20:31
It's not just a It's other guys on X that are now clipping this 100%, dude. >> Um, okay.
20:39
So, actually, let's uh maybe let's do another kind of rendition of this, right?
20:44
Because a logical thing to do on KYC and we've actually been talking about this.
20:47
talking about this. I know OpenAI has been talking about this and also I know Enthropics talking about this is um outcome based pricing right now the question is okay are they willing to sell their unsafe services to known customers for outcomes alone right maybe
21:02
that that's the way like that to me is the most logical way that doesn't print deflation that is like makes sense is good for the labs quote unquote has safety right like it's it it's kind of all lining up where it's like you know anthropic safety team becomes a seal team seal team six of like cyber
21:20
security around the world you pay hundreds of thousands of dollars but you know some guy is on the other phone it's like an FTE where he's like yeah uh we just had Fable try to hack up your hack your entire you know mythos unconstrained essentially hacked and got root root access all day um we didn't do
21:37
anything with it but here's all the fixes you should do um you know you know we we can like you know have another service provider you know help you go update this all and then you congratulations, you're you're good for your pen, you know, your quarterly checkup or something like that. They
21:50
They sell this service as a giant audit for every company on the world.
21:54
Um, it could be, you know, and then they could essentially be paid to be uh, you know, white hats, right?
22:00
That's an example where I think there's like outcome based pricing.
22:03
It's clear it' be really valuable.
22:05
You would have a massive incentive to be in the lead without disclosing your hand.
22:10
Um, what do you think of that?
22:12
because I think that that's something that people are I mean I think it would be the labs would be super stoked about it.
22:18
I think it would feel safe.
22:20
You wouldn't have these like obliterated um models going around, you know, hacking everyone's stuff. Um any thoughts?
22:30
>> Yeah, I mean um I want Sam to jump in here too.
22:33
here too. So the I I think you literally just described a finance bros perspective of project glasswing and daybreak from anthropic and openai respectively which is like exactly why they did not release mythos originally and they got together with a
22:48
bunch of key partners and we're like we're going to give you access to all of these um 0 days before they go public and like you're going to be able to patch them and then you're going to be able to roll out updates and yada yada yada. Um,
22:58
Um, and then you just said, "And they can make a bunch of money off of this," which is like an obvious conclusion that I hadn't considered because all of the writing about it is just all about how this is what's good for humanity and you know how it's going to benefit everybody.
23:11
But obviously people would want a security audit as a service before the models get released.
23:14
Um, and I think it speaks to a broader point which we are writing about right now for an upcoming article about how with AI access to the models and access to compute is more important to pursue the frontiers of really any domain than having access to the most top experts. We've seen this in math.
23:36
We've now seen this in cyber security.
23:38
I think we see this in like software engineering broadly.
23:42
We see this in uh trading where like hedge funds are getting exclusive access to the fast tiers of models.
23:48
Uh we're going to see this in drug discovery soon with anthropics focus on that and we see this in autonomous vehicles with you know Whimo and Tesla like uh I'm sure that they also have a lot of the top people but it is table stakes to have basically the most compute and access to frontier model if you want to pursue the frontiers of research in any domain at this point.
24:13
And um okay Sam over to you to talk about what we are seeing publicly right now which is like we we are not seeing the Yeah. >> Yeah.
24:32
I guess tying it back to what Doug just brought up, I think it's I think it's an interesting point that, you know, this might be a durable source of income for a Frontier Lab, but it's contingent upon them maintaining a significant lead over the open source models.
24:45
Obviously, you're not going to pay anthropic if you can just do it with GLM u for a fraction of the cost.
24:48
Um and yeah, right now it's not it's not clear to me that that these open source models are like um close enough to be super helpful as you know defensive partners.
25:04
Um, so one of the one of the things I mentioned earlier is that I'm frustrated with the sort of quasi first principles thinking about model capabilities on the basis of these transcripts.
25:13
Um, and not enough sort of like mundane empirical um, getting into the weeds of like what we're actually seeing in terms of CVEes, in terms of churn on important code bases.
25:25
And yeah, so the way the research worked for this article is we uh you know, thought we were going to throw together a few charts and just show like how crazy the impact of uh you know, project glasswing has been on the day-to-day practice of cyber security researchers or you know 56 soul fields qualitatively like just ridiculously good at this sort of thing.
25:43
So, we were going to, you know, grab some charts from um PieTorch and um Linux and whatever else and just show this huge uptick.
25:50
And, you know, we looked for one correlation, we looked for another um yeah, and Jordan is uh pulling up pulling it up on his screen here.
26:01
Um and this is one of the few where um you know, there is something to see, although the result is probably not statistically significant.
26:09
Um but in any case, yeah, I mean you can look at um plenty of open source kernels where maybe or open source projects where maybe from first principles you would have said okay um you know we have for this insanely low price um access to intelligence who like knows of every single um coding pattern can read every single line of your codebase with security in mind and there just hasn't been a very large impact.
26:33
Um and I don't have a good theory for this.
26:36
I don't see people discussing it as much as I would like them to.
26:38
I don't think that all these people, you know, trying to figure out what 2050 will look like on the basis of these OpenAI transcripts have a very robust theory of what we're seeing.
26:48
Um, so I wish that there was a lot more discussion of uh, you know, what what the data actually shows so far, you know, a few months into this experiment with these um, strange intelligences being applied to society um, and how fast we can fortify ourselves.
27:02
Honestly, I kind of wish we like maybe this is like the most weird way to go about it.
27:09
Like okay, just uh spitballing here.
27:11
Should we be like scraping and pulling in all the git public GitHub repos of the most public uh things and essentially doing a change log over time to see if uh the I mean like one way you see all the commits going vertical like that that has to happen that is obviously flowing through code.
27:26
Then it's like okay would we would it be possible if we were essentially able to say penetr like could we do a time series where it's like okay uh 5.
27:34
2 whatever open model could it find the vulnerability in this one this one this one this one you know is that like you know essentially like maybe doing something like essentially kind of like a ghetto agent x trait.
27:46
Yeah, that that's what we kind of >> Yeah, that's what we kind of did so far.
27:52
I mean, we're we're like halfway through the analysis, so I almost hesitate to talk about it openly here, but um we think there's something there where there are patches being rolled out to a lot of these open source projects that refer to things that we would think
28:04
would be security issues that are not being disclosed as CVES or as bug fixes explicitly, which you know, maybe this is like a people not following process because they're vibe coding and going too fast, or maybe this is like intentional so that the models don't patch [clears throat] this thing. I
28:22
I can't imagine that's the case.
28:22
Um, something's happening. Uh, for sure.
28:28
>> Something's happening. >> What? >> Wow.
28:30
The analysis you expect from some analysis.
28:34
Well, look, you you listen to the the CEOs of these security companies talk and you're like, "Well, obviously we're just going to go look and find how it's been a rocket ship of new CVES going out cuz, you know, they just don't have enough time and they they're like crying for more resources and everything and it's just not the case. >> Yeah.
28:52
So, we um we spent a fair number of Perplexity Computer credits um pulling a ton of GitHub blogs.
28:59
We we >> we >> we [laughter] the proverbial we of Sam Harsh.
29:06
>> Okay, exclude yourself, Doug. I was part of this.
29:08
[laughter] >> I I I actually I'm not a computer user, but I I should get in on the game. >> Yeah.
29:15
So, there was uh there were credits spent, let's say, u in the last couple weeks pulling a bunch of data from GitHub.
29:23
Um, so Jordan mentioned that it's possible that um the workflow of these projects has changed significant significantly.
29:30
That's something that was mentioned um in our brainstorming in Slack that um because everyone has access to these models, maybe it doesn't make sense to go through an ordinary CVE disclosure process.
29:42
You know, if um OpenAI's new model comes out and it's able to find this new bug, you expect that everyone else is finding the same bug simultaneously.
29:49
Um, so there's no reason to go through this oldfashioned uh process where, you know, it's embargoed for a time and not everyone can see it because everyone is, you know, using the same model and finding the same bugs and so you might as well just send your your fix straight into upstream.
30:04
Um, [snorts] so what we looked at to figure out whether this is happening is whether the ratio of CVE to ordinary security patches has changed and hasn't.
30:14
Um, >> a lot of these repos are seeing like more PRs than normal.
30:18
So, just like strictly more code churn.
30:20
Um, and a and a slightly higher percentage perhaps of like fixes or of of changes that mention security related terms.
30:30
Um, but you know, if I just showed you the chart with no context, it's not like you would be looking at it and saying, um, oh my goodness, what's what's happened to the practice of, you know, cyber security engineering?
30:41
Um, so maybe we're looking in the wrong place, but um, I don't know.
30:45
I think that there's probably some outer loop here that's like a lot clunkier than um, people might imagine.
30:51
Um, and these things are taking longer to impact than you would expect. >> Yeah.
30:55
One other thing to say though just about the open versus closed model gap is that Mythos was driving a lot of this with Project Glass Wing and it was announced on April 7th. Okay.
31:03
This means that it was clearly done training earlier than that. We think February.
31:07
Um that's 5 to 7 months to the present day.
31:13
Mythos 2 is clearly better than Mythos 1.
31:15
I don't think anybody expects that this is the frontier model.
31:17
We have seen no benchmarks for it.
31:19
We have seen no release announcement.
31:20
We have seen nothing other than rumors about people using Mythos call it 1.
31:25
1 or two or whatever you you prefer.
31:28
And so I maintain the feeling that the gap from open versus closed frontier models uh continues and is strong.
31:38
But the gap for usable publicly announced uh open versus frontier models uh open versus closed frontier models is clearly closed.
31:49
Um, and we we went through that in a previous article where we compared the three eras of these models over time on a select set of benchmarks.
31:58
And in cyber security, I mean, frankly, uh, all of the public benchmarks are saturated at this point.
32:04
There's not a lot of, uh, good public benchmarks that you can use.
32:10
It's strictly held out private ones and vibes at this point. >> Yeah.
32:15
I mean, >> it's not even even the crazy emergent behavior we just saw in this open AI hoging phase thing is not something that seems like totally out of the realm of possibility of like even a somewhat mediocre open model.
32:28
Like it's just a lot of a lot of persistence.
32:30
I mean you can imagine it's like trying to achieve the goal of like uh you know solving some exploit gym task and its only conceivable access to the real world is through this artifactory service like of course once it realizes it doesn't have the materials it needs in its environment because open AI forgot to give it to it um it's going to pound the hell out of this uh you know single escape hatch and eventually it finds its way out.
32:58
Um, so I mean like willingness to just like burn some tokens and, you know, probe every single possible weak point is seems to me the most important thing.
33:08
>> I'm I'm going to be honest with you.
33:08
I I I'm completely Yeah, I I'm sure it's a very performant model, but it doesn't seem like uh you know, this was like a mythos model, you know, whatever.
33:18
It's like the the real point of it is scale.
33:23
And it's like if you could have an agent swarm of like 70,000 GLM 5.
33:24
3 flash or whatever, you probably get to the same outcome, right?
33:30
It's not a special level of intelligence.
33:32
I think it's just like a special level of scale. Um, right.
33:34
And like, hey, getting 70,000 people to work on any and not we're not going to, you know, not going to anthroporize. Uh, okay.
33:42
I'm not going to anthroporize the agents, but like imagine imagine >> you said swarm, man. This is a civilization.
33:47
I thought this was people. I said people. Um, sorry. Sorry.
33:49
Um, so, so imagine having 10,000 engineers work on something and they only have one escape hatch.
33:55
They're going to figure something out, right?
33:57
And it's like I think that that's like the harder part or the harder problem that we're going to really struggle is like just the scale of the right?
34:03
When then we talk about like the people who are trying to diagnose the problem.
34:06
They're like, "Yeah, man.
34:08
I'm just a I'm a guy with a model doing this cuz there's no way any human could do this anyways."
34:13
So, it's like we're just like it feels like we're at a order of magnitude that's outside of any any one human's brain.
34:20
brain. uh it's it I mean there's something fundamental about cyber security which is the asymmetry right it's so much easier to find a single exploit than it is to protect every single surface that could be attacked and I mean we're making progress on protecting the surfaces but to get back
34:38
to the core of the article we can run through some common examples of bad designs that are present on Neoclouds today that are worth billions of dollars and serving companies in banking telco research academia uh federal defense and national intelligence at large countries
34:54
like top 10 global GDP countries who are so out like years out of date on software before anybody's really started talking about this thing and are just sitting there like waiting to be exploited don't have a process to patch you know fixes don't have a process to notify their customers and are just kind
35:15
of you know running fast because there's so much demand for GPUs Sure enough, they're trying their best, but best isn't good enough if somebody else trusts you to put their closely guarded secrets, weights, and you know, uh, data sets. I mean, even just regular consumers who
35:31
I mean, even just regular consumers who are trying to save a buck are really not, I think, uh, up to speed on the fact that their banking information is included in OpenClaw because they kind of yoloed the Door Dash CLI and are like, I'm going to
35:47
order some burritos with this OpenClaw thing I text and I'm going to save a few bucks by trusting somebody on Open Router who doesn't necessarily have the security practices up to date and then somebody else is reading every log. Um,
35:56
Um, anyway, the these are like maybe it's worth going through these tangible examples, but you need to do both sides.
36:05
You need to protect yourself, too.
36:07
>> I was going to have the hottest take of all.
36:09
And I'm I'm excited for the clip. Okay.
36:11
Uh, maybe instead of um uh security through what is it? Obscurity, right?
36:16
We're going to have security through just there's so much information only another agent is able to comprehend it.
36:22
it. So it's like yeah it's it's out there but it's you have to essentially be like yeah my open call has to go find Jordan's open clause traces and and other otherwise this is like you know sea of information uh there's literally no way you other you have to be like
36:37
yeah man I could do that >> this is this is where the >> I mean >> right right but no and it's literally what happening because because hugging face is is claiming that's exactly what they would never have been able to figure out the extent of this exploit in the time they did without using AI. I
36:52
I think that's obviously true.
36:54
Uh we could never have done all of this work that we've done testing all these clusters without AI and doing it all manually.
36:58
Um but the I mean the bigger thing is that it's asymmetric in terms of the guardrails.
37:04
The person who's trying to read your stuff is obviously not listening to the laws and rules and regulations out there.
37:10
They're going to use the obliterated version of the model and now they've got this asymmetric advantage over the people that get refused every single time they're trying to understand what's happening when people are attacking them.
37:19
So, it's it's like there's an asymmetry that favors the attackers all the way down.
37:24
Um, which I mean like is a real concern for the people who assess everything that's going on right now and are just like, you know, libertarian view like open for everybody.
37:38
Let's all compete in the marketplace.
37:40
Um it's it's it's an imperfect marketplace by definition uh because people don't have equal access to compute and all of this stuff.
37:49
People get things before others and so you there's some chain of trust that you kind of need to build here and uh yeah we we like who's trustworthy I think is like going to be the really big question because there's no crypto level zero trust scenario of the future that I can imagine actually plays out. Yeah, I agree with that.
38:10
Yeah, I was going to say even if there was a crypto level zero trust, I'm sure they've already found all the exploits for it.
38:14
You know, that stuff that stuff's incredibly, you know, like that's like notoriously like, oh man, protocol got hacked and it's the the funds a bit drained, right?
38:22
Um >> yeah, I uh also uh let's do an over under.
38:27
I mean, any other kind of or let's do any anything else you guys want to cover?
38:31
Otherwise, I was going to say, dude, is obliterated just an NSA thing where it's like, hey, we're about to get you red bandit. You want this?
38:36
You know the NSA is the NSA model. It's a honeypot. Exactly.
38:44
[laughter] >> HS download the obliterated thing.
38:45
They came I see you there. It's a honeypot.
38:49
[snorts] [laughter] >> That's good. >> Yeah.
38:52
That's a It's a It's a >> Oh. Oh man. Yeah. Yeah.
38:57
>> I wonder who the I didn't even look this up.
38:59
I wonder who the investors and founders are for Obliteration.
39:01
Which is based in PaloAlto, California. Suspicious.
39:08
>> [laughter] [gasps] >> Yeah, >> Doug, you got to tweet that right now. Honeypot. Honeypot. CIA honeypot.
39:15
>> Well, no, no, I think Okay, so the tweet should be like, I [clears throat] sure hope it's an NSA honeypot because the alternative is way worse. [laughter] >> Yeah.
39:24
Well, I mean, well, look, it it's interesting that it came from an American lab, not a Chinese lab. >> Yeah.
39:32
an American lab filled with American citizens.
39:37
>> Not we're not conspiracy theorists here on our [laughter] >> on our show.
39:42
You can save that for your other uh other radio show. [laughter] >> No, dude.
39:46
There is no other radio show. This is it.
39:48
Sorry, that's my that this a joke to be clear.
39:53
>> You're saying transistor radio's dead?
39:56
>> Those transistors do burn out, I guess. >> RIP, man. All right.
39:59
Well, I'm going to do some research on the side here.
40:05
>> Yeah, there there's some way that this is uh you know, I I'm probably like overthinking this to the fifth degree here.
40:12
There's some way that this is making me more optimistic that we found all these ridiculous and like ridiculously straightforward bugs and like very critical infrastructure.
40:20
Um, and the models are already certainly good enough to exploit then.
40:25
um you know, you can use a Kimmy K or whatever um and not get refusals and take advantage of things like what we just saw.
40:34
So, why are we not already seeing the world going to hell? Um I don't know.
40:40
It it just seems to me that there's like some outer loop that I don't really understand here that's holding things together much more than I would have expected.
40:47
And like maybe this, you know, sort of quasi first principal thinking again, it's just not the right way to think about it.
40:53
or something constraining us much more than model capabilities already.
41:00
>> Morals of >> you want to get philosophical, Sam?
41:01
Do you believe in the goodness of humanity?
41:06
>> I think nothing ever happens and there's some mysterious substance that keeps the world from falling apart and we're all always going to be fine and we don't need to worry about this. >> Doug's nodding.
41:21
>> That's an amazing philosophy. I agree completely.
41:25
If you agree with that, Doug, that doesn't >> Nothing ever happens, man.
41:27
Dude, nothing ever happens, man.
41:29
Is is keeping us all safe.
41:31
You know, >> this is a really interesting way to to both keep your like extreme cynicism, but also be positive. >> Yeah.
41:41
No, that's the goal to have a worldview as insulated as possible from anything that actually happens.
41:48
[laughter] >> There's some [clears throat] stuff happening in the world right now, guys.
41:51
I think we need to keep keep tabs on it for the next couple of years. >> Yeah, man. [laughter] Yeah. >> A couple blog posts.
41:59
No, no, there's no monetary value to the hacking.
42:04
>> Somebody in HuggingFace went on Slack one day, got a pager duty alert or some data dog thing and was like, "Hey, we're under attack.
42:10
Why do we not own six of our clusters anymore?" or whatever it was.
42:15
>> And you're like, "Yeah, doesn't matter. Nothing ever happens. GLM 5. 3 saved the day. They figured it out.
42:20
They rotated the credentials. Doesn't matter.
42:21
Like this doesn't feel like a bit of a close call of like hugging faces acquisition from Nvidia could go smoke.
42:32
>> It's just a Tuesday, man.
42:32
That's all of our jobs in the future. >> All of our jobs.
42:35
We just have pagers every single day.
42:37
Oh, my friend's giving me a little lovely love tap, you know, just trying to drain my bank account again. Who's >> your friend? [laughter] >> Yeah. Ah, Jordan.
42:43
Jordan's really pissed about last week.
42:46
You know, I beat him in pickle ball, so now he's going to spill all my money.
42:48
And then you say AI get them better.
42:51
That's the crypto that's the that's the uh anarctic techno capitalist future that we're we're we're driving to, you know.
42:57
driving to, you know. So Clam and Tom get together in like, you know, July or whenever this is happening and they go, "Hey, you remember that project with the with, you know, the new GLM guys where we're going to get their model weights
43:11
in advance and the Kimmy guys where we get theirs and Deepseek where we get theirs and, you know, all this all this work that we're doing with Nvidia and let's say like Thinking Machines and Coolside and Reflection and all the American labs that are open. Maybe even,
43:21
Maybe even, you know, Open AI's got a GP2s coming soon or something.
43:25
Let let's say all this is happening and they're go, "Hey, we almost lost all of our production clusters to an agent that we don't really understand.
43:32
It depends on two credentials and some zero days in our data sets API." Um, yeah, I'm good. Let's sell to Nvidia.
43:41
[laughter] Let's Let's cash this check, guys. Okay.
43:46
I'm I'm kind of tired of running this. >> Yeah.
43:50
I mean that's >> it's guarant it's guaranteed to happen now that you mention it because if that didn't happen then something would happen and nothing ever happens. >> Right. Right. Right. Right. Right.
43:59
If they didn't sell then something would happen so therefore they had to sell. >> Yep. >> Yeah.
44:04
I mean I I both of you I think we're [laughter] trolling.
44:09
Um but anyways let's can we let's bring it back to the end.
44:12
Is there anything else you want to talk about on a security issue?
44:16
>> I think yeah Neocloud should get their stuff up to date. Sam, you go.
44:20
>> Yeah, I was going to say like on on a serious note, um I think it's an interesting quasi philosophical question, the extent of whips, this is like bizarre emergent behavior from entities that we don't understand.
44:30
entities that we don't understand. this is a civilization um that was you know burned to the ground and rose again from the ashes or this is just like ordinary you know security malpractice and like you know open AAI needs to work on isolating their credentials and um you know patching their their sandboxes and
44:49
so on and monitoring while these things are going >> yeah any kind of monitoring at all would have sufficed here no um so I don't know I think that I think that um you know it's worth taking very serious to the perspective that like these are um you know fundamentally new um forms of intelligence that we need to contend with. And it's also worth taking
45:11
And it's also worth taking seriously the idea that like you know there's nothing new under the sun um and we just need to like make sure our docker is up to up to spec and we'll all be fine. >> Yeah. Okay.
45:22
So let me rant about the the last sentence there which is like um we we put out this cluster max article uh before cluster max 3.
45:32
0 because there was a warning shot to neoclouds and even though we had done hands-on testing with a bunch of them you know we developed the tests as we do testing and so like people that we tested in March or April you know they didn't get the same level of uh um tender loving care when it came to security as the people that were coming later where we had a more automated process and we could run this thing.
45:50
So, we released this convenient script.
45:51
It's called CMAX audit security.
45:54
You can install it with pip install cluster max.
45:56
Anybody out there who wants to try it, you can go do that or you can check out the repo on GitHub.
45:59
It's a very simple utility.
46:01
We update it regularly with like a GitHub action that checks all of the security bulletins from Nvidia and Docker and the Linux kernel and AMD and like uh stuff across the Nvidia GPUs and uh their drivers and um uh like the network cards and their firmware and their driver versions.
46:18
And I mean this is particularly interesting for the blue field networks, the cars which have like their own little CPU on them that you can control, etc. , etc.
46:27
Um, please get your stuff up to date.
46:29
We're doing our best to try to help people do this.
46:31
Um, the reason to do this is that because we we had a few examples that we want to talk about.
46:37
First is that we're able to view metadata about other tenants on shared infrastructure via people leaving their BMC's open to the uh public internet in some cases uh via front end networking that had no VLAN or VXLAN setup so you could just see cross tenant via backend networking that's incorrectly configured.
46:53
This is missing infiniband security keys.
46:55
So people please implement your P keys, M keys, SA keys.
46:59
Uh storage servers that weren't correctly enforcing arbback so you could see between different volumes different users.
47:04
Um storage just incorrectly configured in the overlay network because customers are accessing the underlay network.
47:09
Monitoring dashboards where you have like this godle level authentication token on the Prometheus side and then you just try to scope that with a single off token on graphana. This doesn't work.
47:18
You need to do graphana per instance.
47:19
You can't do multi-tenant graphana. That is crazy.
47:24
people who can break out of containers and VMs where you're just having one level of isolation on shared servers and escalation to root. Please don't do this. This is a bad design.
47:30
Really, anything I'm describing here that has a bad design uh is something where there's a single point of failure.
47:37
So, please don't use single off tokens or single boundaries of isolation like a single container where one vulnerability leads to like a full cross tenant rce.
47:45
um have multiple layers that if somebody breaks out of a container, they're still in a VM or, you know, multiple layers of security on the network and things like that.
47:55
Um the what this leads to is cascading vulnerabilities where people, you know, one thing doesn't seem that bad, but if there's a zero day in your environment, if you you know, keep these things going, then it just actually gets worse.
48:08
Uh we made some memes so that this sticks in people's mind.
48:11
Here is my favorite from this.
48:14
Thank you XKCD for the original uh meme here.
48:16
We can't depend on single O tokens to have everybody secure.
48:22
Uh maybe one last thing to say on this is that um you can read the article like the article is there. Send it to your agent.
48:31
Um have it summarize these details. Use the CLI.
48:36
Message us if you have questions.
48:36
But thank everybody for listening and hopefully you take security seriously going forward into the future.
48:42
Yeah, and we're already seeing the results right now.
48:46
Like people are already starting to update some of their things and it's from people that you'd be surprised to hear, I guess.
48:51
So, uh, your cluster could be insecure today. Sign up for cluster. You know what?
48:56
No, we need to No, no, we need to do pit pit cmax uh audit, right?
49:02
>> 1800 cluster max and we will audit your cluster for free.
49:04
[laughter] >> Yeah, >> dude.
49:09
Honestly, why don't >> why don't we just we should have just zero we should have just like uh just bug reported everyone know hey your sucks.
49:16
Pay us pay us money the the off the aggressive >> make this about make this about money dog. Thanks man.
49:21
[laughter] >> Hey I'm being sarcastic.
49:22
No, I'm I'm I'm kind of pointing out >> didn't come across to me. Thanks for clarifying. [laughter] >> Whoa.
49:30
We're we're we look at the um look at the good that we're doing.
49:33
Everyone's we're trying to update everyone's for free. >> Doug, come on. Come with me.
49:36
We're doing something out of the goodness of our heart.
49:39
Please, >> we are >> This is This is why nothing ever happens because cynics like you [laughter] have people like us come along and help everybody and then go, "Dude, nothing ever happens."
49:50
[laughter] >> That's okay.
49:53
That's too much of a pat on my own back.
49:55
We're definitely not at the frontiers of NeoCloud security.
49:57
Um but uh man, get yourself up to date, please.
50:03
This this goes to both providers and to customers.
50:06
customers who are trusting people to run a cluster for you.
50:07
Run this utility and then ask a provider to upgrade it.
50:11
If there's demand on their side, they're going to from everybody.
50:14
They're going to have to build these systems and we're just going to help everybody get better.
50:20
All right, I think we're out of time.
50:22
Thank you everybody for listening. Um, no takeaways. Sam, good job.
50:26
>> Yo, say something at the end. Come on.
50:28
No, >> no, the thank you guys.
50:29
Um, I mean, the point of the Cluster Max project is to make sure nothing ever happens.
50:33
We're afraid that something might happen as regards security.
50:36
Um, and we want to make sure to protect against it. >> Have a good one. >> Good job. All right. Take care, Ro. All right. See you guys.