"Okay, today I have the pleasure of interviewing  Paul Christiano, who is the leading AI safety   researcher. He's the person that labs and  governments turn to when they want feedback   and advice on ..."

"economic powers, one world government. Why do you think that's the transition   that's likely to happen at some point. So again, at some point I'm imagining, or I'm   thinking of the very broad sweep ..."

"worlds or like feasible worlds is the one  that seems most desirable to me that is   sort of decoupling the social transition from this  technological transition. So you could say, like,   we're about..."

"not even all collectively engaged in this process.  And I think from the perspective of an AI company,   you kind of don't have this fast handoff option.  You kind of have to be doing the option value..."

"Yeah, I guess there's two aspects of that that  seem particularly challenging, or there's a   bunch of aspects that are challenging. All of  these are things that I personally like. I just   think abo..."

"that regime might last like a very long time. Does that regime have to be global? I guess   initially it can be only in the countries  in which there is AI or advanced AI,   but presumably that'll pro..."

"for avoiding a really bad situation. You would  really love to understand if you've built systems,   like if you had a system which resents the fact  it's interacting with humans in this way. This   i..."

"there's no contradiction in principle to building  cognitive tools that help humans do things without   themselves being like moral entities. That's like  what you would prefer. Do you'd prefer build ..."

"Yeah, where are you at on that? I do not right now want to just take some   random AI, be like, yeah, GPT Five looks pretty  smart, like, GPT Six, let's hand off the world to   it. And it was just som..."

"Okay, we can come back to this later. Let's get  more specific on what the timelines look for these   kinds of changes. So the time by which we'll have  an AI that is capable of building a Dyson spher..."

"40% by 2040. So I think that seems longer than  I think Dario, when he was on the podcast,   he said we would have AIS that are capable  of doing lots of different kinds of they'd   basically pass a T..."

"this sort of two poles of that discussion, and I  feel like I'm presenting it that way in Pakistan,   and then I'm somewhere in between with this nice,  moderate physician of only a 15% chance. But in..."

"Okay, so maybe instead of speaking in terms  of years, we should say, but by the way,   it's interesting that you think the distance  between can take all human cognitive labor   to Dyson sphere is tw..."

"Yeah, I mean, get us there is, again, a little  bit complicated. Like there's a system that's   a drop in replacement for humans and there's a  system which still requires some amount of schlep   befo..."

"And that schlep is what gets  you from 15% to 40% by 2040.  Yeah, you also get a fair amount of scaling  between you get less scaling is probably going   to be much, much faster over the next 4 or fiv..."

"things and there's a good chance it's not. Yeah,  that might be totally wrong. Like maybe just   making up numbers, I guess like 50 50 on that one. If it's 50 50 by in the next 4 years that it will   ..."

"it's more likely than not that you can be like  a drop in replacement for humans. I think if you   just literally say like you train on web text,  then the question is kind of hard to discuss   becaus..."

"you're like, hey, here's what I want. I want my  model to interact with some job over the course   of a month and then at the end of that month have  internalized everything that the human would have ..."

"How fast do you think the pace of algorithmic  advances will be? Because if by 2040,   even if scaling fails since 2012, since the  beginning of the deep learning revolution,   we've had so many new t..."

"I think you probably get a lot, though. You get a  bunch of orders of magnitude of total, especially   if you ask how good is a GPT five scale model or  GPT 4 scale model? I think you probably get lik..."

"I guess the way. That you could think of. This is  like, I think both analogies are reasonable. One   analogy being like, evolution is like a training  run and humans are like the end product of that ..."

"In what ways is human learning  superior to grading descent?  I mean, the most obvious one is just like, ask  how much data it takes a human to become like,   an expert in some domain, and it's like m..."

"Yeah, I mean, I think the way I would put it is  like the number of bits it takes to specify the   learning algorithm to train GPT 4 is like very  small. And you might wonder maybe a genome, like,   t..."

"then there's some resolution. Each of those  carries some bits, but let's say it carries   like ten bits or something. Just from timing  information at the resolution you have available,   then you're..."

"Right, yeah. So it's a little bit hard to say  exactly what the task definition is there like   you could say, like making a bone. We can't make a  bone, but you could try and compare a bow and the   ..."

"what stage does misalignment happen? So right  now, with something like GPT four, I'm not   even sure it would make sense to say that it's  misaligned because it's not aligned to anything   in particu..."

"like some of these abstractions seem like they do  apply to GPT Four. It seems like probably it's not   egregiously misaligned, it's not doing the kind  of thing that could lead to takeover, we'd gues..."

"to intimidate the humans into giving it a high  reward, et cetera. I think that doesn't really   require that much. This basically requires a  system which is like, in fact, looks at a bunch of   envi..."

"New situation to get reward, by the way, this  requires it to consciously want to do something   that it knows the humans wouldn't want it to  do. Or is it just that we weren't good enough   to specif..."

"it like less than one in 1000 for GPT five. Certainly if we deployed all these AI systems,   and some of them are reward hacking, some of  them are deceptive, some of them are just normal   whatever, ..."

"This seems like a much more gradual story  than the conventional takeover stories,   where you just like, you train it and then  it comes alive and escapes and takes over   everything. So you think th..."

"where something bad happens, I still do think  this is the best guess. Okay, so that's like,   part one of my answer. Part two of the answer was,  like, in this proximal situation where something   ba..."

"my best guess is because there are a bunch  of other AIS running around 2D or lunch.  So how much better a situation would we be in  if there was only one group that was pursuing   AI. No other countr..."

"That could be economic competition or military  competition or whatever. So I kind of think   ultimately most of the harm comes from the  fact that lots of people can develop AI.  How hard is a takeov..."

"are deploying AI everywhere, I think of this  competitive dynamic if people aren't deploying   AI everywhere so if countries are not happy,  deploying AI in. These high stakes settings. Then   as AI i..."

"probably is about I think it's not necessary. But  if you ask about the median scenario, it involves   a bunch of humans working with AI systems,  either being directed by AI systems, providing   comp..."

"I think you actually have  given that probability online.  I've certainly guessed. Okay, but it's not zero. It's   like a significant percentage. I gave like 50 50.  Okay. Yeah. Why is it tell me abou..."

"Okay, sure. Because there's just so much  raw resources elsewhere in the universe.  Yeah. Also, you can marginalize humans pretty  hard. Like, you could totally cripple human like,   you could cripple..."

"it's quite robust that you don't want to murder  them. That is, I think the weird decision theory   a causal trade stuff probably does carry the day. Oh, wait, that contributes more to that 50 50 of  ..."

"like, do I want to murder everyone? Your calculus  is like, if this is my real chance to murder   everyone, I get the tiniest bit of value. I get  like 1,000,000,000,000th of the value or whatever,   ..."

"It’s going to happen. So  you should have something.  Yeah, I think that in some sense, you’re always  going to face this trade off where alignment makes   it possible to deploy AI systems or it makes..."

"rather than delaying the Chat GBT wake up  thing by a year and then having chat GBT was.  Net negative or Rlhf was net negative. So here, just on the acceleration, it’s just like,   how is the press o..."

"right now we hit pause and you have ten years of  no alignment, no capabilities, but just people get   to talk about it for ten years. How much more does  that prepare people than we only have one yea..."

"there’s a question of during a pause, does Nvidia  keep making more? Like, that sucks if they do   if you do a pause. But in practice, if you did a  pause, nvidia probably couldn’t keep making more   ..."

"at these different benchmarks, we’re going to  make sure we have these safeguards, what happens?   I mean, there are other companies and other  countries which care less about this. Are you just   slo..."

"that you blur this out and then people want the  weights now because they know what they can do?  Yeah, I think the general discussion does  emphasize potential harms or potential. I mean,   some of t..."

"allow you to quickly build much more powerful  systems, or might, if you’re trying to hold   off on development just itself, be able to create  much more powerful systems. One question is how to   han..."

"So if we’re talking not only about a powerful  model but like a really broad deployment of   just something similar to the Open Eyes API where  people can do whatever they want with this model   and m..."

"but eventually that’s not the case. Eventually,  like, AIS will enable just like, totally   different ways of killing a billion people. But I think I interrupted you on the initial   question of, yeah..."

"those tests are indicative of the real world. So  we’ve tried to argue like, hey, actually the AI is   not very good at distinguishing situations  we produce in the lab as tests from similar   situati..."

"what is the model paying attention to. It gets  basically like a first line of if you get lucky,   what would work here? And then there’s a deeper  like, you probably have to do novel science at   som..."

"How do you create the optimal conditions  for it to want to be deceptive? Do you   fine tune it on mindcomp or what are you doing? Yeah, so for deceptive alignment, I mean, I think   it’s really compl..."

"Yeah, I think that at a meta level in terms of  what’s your protection like, I think what you   want to be saying is, we have these examples  in the lab of something bad happening. We’re   concerned a..."

"we not use them? Which I think is like that’s  sort of the good story. The good story is you   develop methods that address a bunch of existing  problems because they just are more principled   ways t..."

"You mean what is this kind of criterion? At the  end of the day, we kind of want some criterion.   And the way the criterion should work is like you  have your neural net, you have some behavior of   ..."

"Would it be useful to maybe motivate this  by explaining what the problem with normal   Mechanistic Interpretability is? So  you mentioned induction heads. This is   Anthropic found in two layer trans..."

"it’s clear why the explanation matters. Yeah. And for this purpose, it’s like the   thing that’s essential is kind of reasoning from  one property of your model to the next property   of your model. I..."

"even for small models to have a clearer sense. The point you made about as you automate   it is it because whatever work the  automated alignment researcher is doing,   you want to make sure you can v..."

"like, here’s a direction activation space,  and here’s how it relates to this direction   activation space. And so just pointing out  a bunch of stuff like that. Here’s these   various features constr..."

"Yeah, I mean, to be clear, I think you probably  wouldn’t be looking at a separate circuit,   which is part of why it’s hard. You’d be looking  at like, the model is always doing the same thing   on e..."

"job, the story is like if there’s a new input  where you expect the property to still hold,   that will be because you expect the explanation  to still hold. Like the explanation generalizes as   well..."

"distribution. And then on this test time, when  you run it on the new input, it’s like, does I   think I’m on the train distribution. It says,  no. You compare that against your explanation,   actuall..."

"reasons. It does something. Like there was some  earlier step, maybe you could think of it as like   at the first step where it’s like now I’m going  to try and do the sneaky thing to make my thoughts..."

"Yeah, I also want to maybe step back a tiny bit  and clarify that. I think this project is kind   of crazily ambitious and the main reason, the  overwhelming reason I think you should expect   it to b..."

"And you can imagine interpretability on simple  models where you’re just like by gradient descent,   finding feature directions that have desirable  properties. But then when you imagine like,   hey, ..."

"worked. Or you’re like, Why do these proteins?  You’re like, it’s just because these shape like,   this is a low energy configuration. And we’re very  open to in some cases, there’s not very much more..."

"training that demand explanation. I also, again,  want to emphasize here that when we’re talking   about searching for explanations, this is some  dream. We talk to ourselves. Like, why would   this b..."

"I think the mathematicians just wouldn’t be very  surprised or wouldn’t care that much. And this is   related to the motivation for the project. I think  just in a lot of domains, in a particular doma..."

"That’s the case in which it’s not aligned. Yeah, I mean, maybe one way of putting it is   just like, we can wait until we see this input, or  like, you can wait until you see a weird input and   say, ..."

"are slightly intention and how do we deal? Yeah, that seems super interesting. We’ll see   what other applications it has. I don’t know, like  computer security and code checking. If you can   actuall..."

"I think theoretical computer science is  an exception where I think this is like,   in some sense, like what the best of theoretical  computer science is like. So you have all this   reason you have t..."

"in the margins of the podcast. Yeah, with luck. Yeah. So I think it’s hard   to work on because it’s not clear what a success  looks like. It’s not clear if success is possible.   But I do think there..."

"Well, one question I think would be interesting  to ask, you know, I think people can talk vaguely   about the value of theoretical research and how  it contributes to real world applications and you ..."

"Carl Schulman laid out his picture of the  intelligence explosion in the seven hour episode.   I know you guys have talked a lot. What about his  basic is? Do you have some main disagreements? Is   th..."