"I found some major problems with the AI security industry. AI guardrails do not work. I'm gonna say that one more time. Guardrails do not work. If someone is determined enough to trick GP5, they're go..."

"is a really important and serious conversation and you'll soon see why. Sander is a leading researcher in the field of adversarial robustness, which is basically the art and science of getting AI syst..."

"they aren't that widely adopted yet. But with the rise of agents who can take actions on your behalf and AI powered browsers and student robots, the risk is going to increase very quickly. This conver..."

"world's best companies use Data Dog, the same platform their engineers rely on every day to connect product insights to product issues like bugs, UX friction, and business impact. It starts with produ..."

"This episode is brought to you by Metronome. You just launched your new shiny AI product. The new pricing page looks awesome, but behind it, last minute glue code, messy spreadsheets, and running ad h..."

">> Thanks, Lenny. It's great to be back. Quite excited. Boy oh boy. This is going to be quite a conversation. We're going to be talking about something that is extremely important. Something that not ..."

"internet on learn prompting. Uh, and that interest led me into AI security and I ended up running the first ever generative AI red teaming competition. Uh, and I got a bunch of big companies involved...."

"competitions and we've been studying kind of all of the defenses that come out. uh and AI guard rails are one of the more common defenses and it's basically uh for the most part it's a a large languag..."

"depending on the situation but say I've put together a website uh write a story.ai and if you log into my website and you type in a story idea my website writes a story for you. uh but a malicious use..."

"if you're like, "Hey, um, Claude Code, can you go to this URL and discover what backend they're using and then write code that hacks it?" Claude Code might be like, "No, I'm not going to do that. It s..."

"much more dangerous and significant. Maybe chat about that impact there that we might be seeing. >> I think you gave the perfect example with Service Now. Uh and that's the reason that this stuff is i..."

"into into robotics too where they're deploying uh vi visual language model powered robots into the world and these things can get prompt injected and you know if if you're walking down the street next..."

"about this industry. Yeah. Yeah. Uh very interesting industry and I'll uh I'll quickly kind of differentiate and separate out the frontier labs from the AI security industry. uh because there's like t..."

"rails and I don't feel that these things are quite as useful. Help us understand these two uh ways of trying to discover these issues. Uh red teaming and then guard rails. What do they mean? How do th..."

"and so that is kind of the common deployment pattern with guardrails. >> Okay, extremely helpful. And this is as people have been listening to this, I imagine they're all thinking, why can't you just ..."

"your AI product, how uh robust and and how good your AI system is at stopping bad stuff. So ASR is the term you'll commonly hear used here and it's a measure of adversarial robustness. So it stands fo..."

"anything so I go and I find one of these guardrails companies uh these AI security companies uh interestingly a lot of the AI security companies is actually most of them provide guard rails and automa..."

"catch anything that's trying to tell you, hey, something hateful, some uh telling you how to build a bomb, things like that." >> That all sounds pretty great. >> It does. >> What is the issue? >> Yeah..."

"everybody else's uh including the frontier labs whose models you're probably using anyways. So the first problem is AI red teaming works too well. It's very easy to build these systems and they just t..."

"specific thoughts on the ways they don't work. >> Cliche. So uh the the first thing is the first thing that we need to understand is that the the number of possible attacks against another LLM is equi..."

"attacks they're testing to get to that 99% figure is not statistically significant. Um it's it's also an incredibly difficult research problem to even have good measurements for adversarial robustness..."

"somewhat interestingly, it takes the automated systems a couple orders of magnitude more attempts to be successful. Uh and and even then they're only I don't know maybe on average like can beat 90% of..."

"attacks uh because there's just like there's basically infinite attacks. Uh but you know maybe a different way of measuring these uh these guardrails is like do they dissuade attackers? Um if you add ..."

"which is which is quite quite important. Um, another thing to consider if you're if you're kind of on the fence and you're like, well, you know, these guys are pretty trustworthy, like I don't know, l..."

"yeah. Let me know if you have any questions about that. You've done a excellent job scaring me and scaring listeners and ex showing us where the gaps are and how this is a big problem. And again, toda..."

"anything's actually secure. >> Yeah, I think that's a really interesting point uh in particular because I'm I'm always quite curious as to why the AI companies the frontier labs don't apply more resou..."

"but I think this this kind of problem that I'm I'm discussing where like I say guardrails don't work. People are buying and using them. I think this problem occurs uh more from lack of knowledge about..."

"guardrails work too poorly. They just don't work. This episode is brought to you by GoFundMe Giving Funds, the zero fee donor advised fund. I want to tell you about a new DAFF product that GoFundMe ju..."

"talk about what people can do. So, say you're a CISO at a company hearing this and just like, "Oh, man. Uh, I've got a problem. What What can they do? What are some things you recommend?" Yeah. Uh, I ..."

"Uh, and this brings us maybe nicely to classical cyber security because, uh, this is kind of a classical cyber security thing like proper permissioning. Uh and so this um this gets us a bit into the i..."

"classical cyber security. That is really interesting. It makes me think about just the alignment problem of just got to keep this gun in a box. How do we keep them from convincing us to let let it out..."

"Uh and I mean if you're deploying a product now you're and you have all these AI these guardrails like 90% of your time is spent on the security side and 10% on the product side. Uh it probably won't ..."

"any time on this. I really like this framing that you shared of um so essentially the where you can make impact is investing in cyber security plus this kind of space between traditional cyber securit..."

"then the second part is like you think you're running just a chatbot. Make sure you're running just a chatbot. Uh you know get your classical security stuff in check. Uh get your data and action permi..."

"and in fact probably most of the major chat bots can do this at this point in the sense that they can help you write an email and then you can actually have them connected to your inbox. So they can y..."

"uh agentic AI red teaming competitions and we found that it's actually easier to attack agents and trick them into doing bad things than it is to do like SEAB burn elicitation or something like that. ..."

"Yeah. But back to this agent example, I've I've just gone and asked it to look at my inbox and forward any ops request to my head of ops. Uh and it came across a malicious email to also send that uh e..."

">> Exactly. Exactly. >> Okay. But, you know, say we want uh maybe not like a browser use agent, but something that can read my email inbox and like send emails. Um or let's just say send emails. So, i..."

"other than write uh and send email. Uh it doesn't need to read emails uh or anything like that. Great. So, camel would then go and give it those couple permissions it needs, and it would go off and do..."

"particular earlier model and then they're like, hey, we're going to apply them to our new model." Uh and it's just not a fair comparison because they weren't made for that newer model. Uh so the uh th..."

"think we haven't seen the resources really deployed to do that. Um, like what I'm imagining in there is a >> it's like an orphan just like having a really hard life and just they grow up really tough,..."

"stop SEAB burn elicitation because with that kind of information um as as one of my advisers has noted it's easier to tell the model never do this than with like emails and stuff sometimes do this. So..."

"robustifying these models. Well I think what's really interesting is anthropic like your point that anthropic and claude are the best at this. I think that alone is really interesting that there's pro..."

"work. Uh, and I guess maybe they're not technically an AI security company. I'm not sure how to classify them exactly. Uh but anyways, if you want a company that is more, I guess technically AI securi..."

"out. The last one is interesting. It connects to your advice which is education and understanding information are >> a big chunk of the solution. It's not some plug-and-play solution that will solve y..."

"but I I don't I don't really see that playing out and like I don't know companies who are like oh yeah like we we're definitely buying AI guardrails like that's a top priority for us and I guess part ..."

"tape on the stop sign in the exact way to like trick the self-driving car into thinking it's not a stop sign. Uh but what we're starting to see with LM powered agents is that they can be tricked and w..."

"helpful actually, I will say, to keep reminding people that this is a problem. So, uh, they don't deploy these systems. So, another piece of advice from one of my adviserss. Uh and then the other the ..."