"Today I'm chatting with Joe Carlsmith.  He's a philosopher and, in my opinion,   a capital-G great philosopher. You can  find his essays at joecarlsmith.com.  So we have GPT-4, and it doesn't seem lik..."

"Another thing to consider is the verbal behavior  of these models. When I talk about a model's   values, I'm referring to the criteria that end  up determining which plans the model pursues. A   model..."

"the actual factors that determine their  choices. They can lie. They might not   even know what they would do in a given  situation, all sorts of stuff like that.  It's interesting to think about this..."

"something that has this intricate representation  of human values and it doesn't seem that hard to   sort of lock it into a persona that we are  comfortable with. I don't know what changes.  Why is al..."

"You mentioned the idea of, "You are what you  pretend to be." Will these AIs, if you train   them to look nice, fake it till they make it?  You were saying we do this to kids. I think   it's better to..."

"genuinely at that point much, much more  sophisticated than you, and doesn't want   to reveal its true values for whatever reason.  Then when the children show some obviously fake   opportunity to def..."

"Eliezer on. I'm much more bullish on our ability  to solve this problem, especially for AIs that are   in what I think of as the "AI for AI safety sweet  spot." This is a band of capability where they..."

"or punishment a human gets. It's literally a  gradient update down to the parameter of how much   this would contribute to you putting this output  rather than that output. Each different parameter   ..."

"This is the basic reason for concern. Imagine  that we're going to transition to a world in   which we've created these beings that are just  vastly more powerful than us. We've reached the   point wh..."

"in some sense, humans voluntarily did that. Maybe there were competitive pressures,   but you intentionally handed off huge portions of  your civilization. At that point, it's likely that   humans hav..."

"this. With the non-Nazi being trained by Nazis,  it's not just that I have different values. I   actively despise their values. I don't expect this  to be true of AIs with respect to their trainers.  ..."

"efforts to change its values to be a certain way. Maybe it's worth saying a little bit here about   what actual values the AI might have. Would it  be the case that the AI naturally has the sort   of ..."

"its concern for reward has some sort of long  time horizon element. It not only wants reward,   it wants to protect the reward button  for some long period or something.  Another one is some kind of m..."

"can debate the probabilities of the worst-case  scenario, but what is the positive vision we're   hoping for? What is a future you're happy with? This is my best guess and I think this is probably   t..."

"A big part of this discourse, at least among  safety-concerned people, is there's a clear   trade-off between competition and race dynamics  and the value of the future, or how good the   future ends ..."

"it could go rogue. But if multiple people are  training AIs, they all end up rogue such that   the compromises between them don't end up with  humans not violently killed? It fails on Google's   run a..."

"is pretty good and gets into a bunch of the  weeds that might give a more concrete sense.  All right. Now on to part two, where  we discuss the “Otherness and Control   in the Age of AGI” series. Here..."

"vein. I just want to name it as a possible  mistake in this vicinity. We don't want to   engage in the following form of reasoning. Let's  say you have two entities. One is in the role of   creator. O..."

"I have a much better grip on what's going on  with Lewis than with Nietzsche there. Maybe   let's just talk about Lewis for a second. There's  a version of the singularity that's specifically   a hypo..."

"You also have a very interesting essay  about what we should expect of other humans,   a sort of extrapolation if they  had greater capabilities and so on.  There’s an uncomfortable thing about the  c..."

"Now that said, I do think many humans would  likely be nicer if they FOOMed than certain   types of AIs. But with the conceptual structure  of the argument, it's a very open question   how much it app..."

"As our scientific and technological power  advances, it’s plausible that more and more   stuff will likely be explicable this way. Why is  this man on the moon? How did that happen? Well,   there was ..."

"Yeah, I agree. There's a few things going  on there. Even if you're engaged in this   ontology of carving up the world into different  agencies, at the least you don't want to assume   that they're al..."

"on the right-wing side of the debate. They say  to themselves, “Traditionally we favor markets,   but now look where our society is headed. It's  misaligned in the ways we care about society   being a..."

"actual good epistemology. You need to actually  know, is this a real risk? What are the actual   stakes? You need to look at it case by case  and be like, “Is this warranted?” That's one   point on th..."

"as a thing that can even make sense. They'll  say things like, “Look, these are the kinds   of things that are going to be exploring  the black hole, the center of the galaxy,   the kinds of things th..."

"They might even be incomprehensible to us. I  don't think so... There's different types of   incomprehensible. Say I show up in the future  and this is all computers. I'm like, “Okay,   all right.” Th..."

"I agree. There's a few different things there.  What are you going for? Are you going for actively   good or are you going for avoiding certain stuff?  Then there's a different question which is,   wh..."

"If you look at the Soviet Union, the  collectivization of farming and the   disempowerment of the kulaks was not as a  practical matter necessary. In fact it was   extremely counterproductive and it a..."

"A kind of paradigm context in which we think  that is appropriate is if something is an active   aggressor against the boundaries and cooperative  structures that we've created as a civilization.   I ..."

"concern from this other concern about where the  future eventually goes. How much do we want to be   trying to steer that actively? I wrote the  series partly in response to the thing you're   talking..."

"what I'm trying to say is let's draw on the full  wisdom we have here, while obviously adjusting   for ways in which things are different. There’s one thing the ember analogy brings   up about getting..."

"Even if you bracket space, time is also very big.  We've got 500 million years, a billion years,   left on Earth if we don't mess with the  sun. Maybe you could get more out of it.   That's still a lo..."

"value systems. Correspondingly, we should be  really interested in doing that. I sometimes use   this heuristic in thinking about the future: We  should be aspiring to really leave no one behind.   Wh..."

"be applying the golden rule as we're thinking  about inventing these AIs. There’s some way in   which I'm trying to embody attitudes towards them  that I hope they would embody towards me. It's   uncl..."

"and the morality of gradient descenting on  their minds, which we can address later.  The thing that personally gives me the  most unease about alignment is that at   least a part of the vision here s..."

"Overall, we are going to need to stare hard at  it. Right now, the default mode of how we treat   AIs gives them no moral consideration at all.  We're thinking of them as property, as tools,   as prod..."

"stuff at stake in that kind of binary. 
 With respect to how we treat the AIs,   I have a couple of contradicting intuitions.  The difficulty with using intuitions in this   case is that obviously it'..."

"of wariness towards, in the context of the way you  end up talking about AI. It’s about maintaining   control over AI, making sure that it doesn't  rebel. We should be noticing the reference   class t..."

"That's one feature of the situation. The opposite perspective here is that you're   doing this sort of vibes-based reasoning of, "Ah,  that looks yucky," doing gradient descent on these   minds. In th..."

"could both be like moral patients—this sort of new  species in the sense that should conjure wonder   and reverence—and such that they will kill you. I have this example of the documentary Grizzly   M..."

"about what should be done, the big crux that  I have is the question of how weird things   end up default, how alien they end up. You made a  really interesting argument on your blog post that   if mo..."

"that's not rational. That's not the march  of reason." There's still empirical work you   can do to tell whether that's what's going on. On moral anti-realism, consider Aristotle and   us. Has there b..."

"anti-realism, but it doesn't move me very much. I don't know if moral realism is the right word,   but you mentioned the thing. There's something  that makes hearts converge to the thing we are   or t..."

"So the question is, do the worlds that matter  have this kind of convergent moral force,   whether metaphysically inflationary or not,  or are those the only ones that matter?  Maybe what I meant was,..."

"It's called "Against the normative realist's  wager." Here's the case that convinces me.  Imagine that a metaethical fairy appears  before you. This fairy knows whether there   is a Dao. The fairy say..."

"I'm also just low on this "everyone converges"  thing. You train a chess-playing AI. Or somehow   you have a real paperclipper and you’re like  "Okay, go and reflect." Based on my understanding   of h..."

"on this one thing. They're like, "Can't you see?  It's Zorgo. Zorgo is the thing.” and it’s all the   AIs. That would be interesting, very interesting. My personal prediction is that's not what we see..."

"blinder we should be really watching out for. I have enough credence on some sort of moral   realism. I'm hoping that if we just do the  anti-realism thing of just being consistent,   learning all the..."

"Here’s one big crux. You're training these models.  We were in this incredibly lucky situation where   it turns out the best way to train these models  is to just give them everything humans have ever..."

"getting for free are things like consciousness,  pleasure, other features of human cognition.  There are paperclippers and there are  paperclippers. If the paperclipper is an   unconscious kind of vor..."

"There's a different view, which is that  consciousness is something that's quite   structural. It's much more defined by functional  roles, like self-awareness, a concept of yourself,   maybe higher-o..."

"think that could show up quite robustly. Part of your day job is writing these   Section 2/2.5-type reports. Part of  it is like, “society is like a tree   that's growing towards the light.” What is i..."

"Could you explain the nature of the transfer  between the two, in particular from the literary   side to the technical side? Rationalists  are sort of known for having an ambivalence   towards great w..."

"You can fall off on both sides of the horse. I remember I was talking to somebody who at   least is familiar with rationalist discourse.  He was asking me what I was interested in these   days? I was ..."

"Some intellectually sincere individuals I know  who focus on the big picture also possess an   impressive command of a wide range of empirical  data. They're really interested in empirical   trends, n..."

"There can be people trying to draw many pieces  together to form an overall picture, people going   deep on specific pieces, and people doing more  generative work, throwing ideas out there to see   w..."

"Suppose we figure out that consciousness is just a  word we use for a hodgepodge of different things,   only some of which encompass what we care about.  Maybe there are other things we care about tha..."

"it is possible that it includes a bunch of stuff  that we don't normally ascribe consciousness to.  You say "a complete theory of mind," and  presumably after that, a more complete ethic. Even   the n..."

"competitive. It’s still important to keep  in mind. It's important to keep in mind in   the context of integrating AIs into our society.  We've been talking a lot about the ethics of this,   but there..."

"the thing I'm getting at in that quote. Okay. I think that's an excellent place   to close. Joe, thanks for coming on  the podcast. We discussed the ideas   in the series. People might not appreciate,..."