"Today I have the great pleasure of chatting with  Scott Alexander and Daniel Kokotajlo. Scott is   of course the author of the blog Slate Star  Codex, Astral Codex 10 now. It’s actually been,   as you..."

"So that’s the easy part. The hard part is  we also want to be right. So we’re trying   to forecast how things are going to go, what speed  they’re going to go at. We know that in general,   the median..."

"Another related thing too is that the original  thing was not supposed to end in 2026, it was   supposed to go all the way through the exciting  stuff, right? Because everyone’s talking about,   what ..."

"like, how can I say no to this person? Everyone  else on the team, also extremely impressive.   Eli Liflund, who’s a member of Samotsvety, the  world’s top forecasting team. He has won, like,   the to..."

"I’ve probably changed my mind towards, against,  towards, against, intelligence explosion three,   four times in the conversations I’ve had  in the lead-up in talking to you and then   trying to come ..."

"So we have the stats tracked on the site of the  story. Part of why we did it as a website is so   that you can have these cool gadgets and widgets.  And so as you read the story, the stats on the   s..."

"exist by this year. Like there’ll be some Twitter  thread about someone being like, “I plugged in   this agent to like run my party and it worked!” Our scenario focuses on coding in particular   becau..."

"like the call center workers haven’t been fired  yet. So why not just say look, at higher scale   it will probably get even more difficult. Wait a second, I’m a little confused to hear   you say that,..."

"where we’re not cherry picking things. Yeah. I had this interesting experience   yesterday. We were having lunch  with this senior AI researcher,   probably makes on the order of millions a  month or ..."

"What are you thinking of? Well, I asked this question where, as you say,   they know all this stuff. I don’t know if you  saw this. I asked this question where I said,   look, these models know all th..."

"I will actually disagree with this.  We know that humans can do like,   we have examples of humans doing this. I agree  that we don’t have logical omniscience because   there is a combinatorial explos..."

"I think if you were to force the AI to do this  in a reasonable way, or you were to train the AI   such that it itself could come up with the plan of  going through this in some kind of heuristic-lade..."

"able to overcome those gaps faster than they  normally would because the coding agents are   doing all the grunt work really fast for them. Yeah, I think it might be useful to think of   our timelines..."

"So if I look back at AI progress in the  past, if we were back in, say, 2017.   Suppose we had these superhuman coders in 2017;  the amount of progress we’ve made since then,   so where we are current..."

"researcher. And the way we did our forecast was,  for each of these milestones, we were like, what   is it going to take to make an AI that achieves  that milestone? And then once you do achieve that ..."

"One of my favorite meme images is this graph  showing world GDP over time. You’ve probably   seen it, it spikes up and then there’s a little  thought bubble at the top of the spike in 2010 or   someth..."

"transitions have been smoother  than the one you were anticipating.  We’re not sure about that, actually. So one of  these models is just a hyperbola. Everything   is along the same curve. Another mod..."

"Okay, now that we brought up the intelligence  explosion, let’s discuss that, because I’m kind   of skeptical. It doesn’t really seem to me  that a notable bottleneck to AI progress, or   the main bot..."

"explosion? And the answer is: combination of that  speed up and the speed up in serial thought speed.  And also the research taste thing. Here are  some important inputs to AI R&D progress today:   re..."

"core setup of our model. And when we get our 25x  multiplier, it’s starting from those premises.  Is there some intuition pump from history  where there’s been some output and because   of some really..."

"getting real world data. I just think it’s hard… So we are actually imagining online learning   happening. Oh really?  Yeah. But not so much real world as in… the thing  is that if you’re trying to tr..."

"what you’re suggesting with this hyper  efficient hive mind of AI researchers,   no human bureaucracy has just out of the gate  worked super efficiently, especially one where   they don’t have experie..."

"these sorts of large scale cooperative dynamics  of your moral maze, you have an institution,   but then it becomes like a moral maze and it sort  of collapses under its own weight and stuff like   th..."

"stuff. You’ll have time for sort of like empires  to rise and fall, so to speak, and all of that   to be added to the training data and yeah. But I  could see it taking longer than we depict. Maybe   ..."

"you got this superhuman thing at the end of  it and then it seems to just go through the   tech tree of mirror life and nanobots and  whatever crazy stuff. And maybe that part   I’m also really skepti..."

"and that sort of thing. I think that if you want  a contrast, some people in the past have proposed   much faster scenarios where they email some cloud  lab and start building nanotech right away by j..."

"thinking very fast, but they are better at  learning from each experiment than the best   human would be at learning from that experience. Yeah, I think the fact that there’s a million   of them or th..."

"We think it will potentially take less with  superintelligence, because first of all, if   you look at the history of this process, despite  this being the fastest anybody has ever done this,   it was..."

"They were able to get like I think a lot more  years of technological advance in than we would   have expected. Partly that’s because just Elon  is crazy and never sleeps. Like if you look   at the ex..."

"So you’re focusing on the nanobots, I want to  ask a couple questions. One, what about just   the regular robots? And then two, what would your  quantities be for all of these things? So first,   what..."

"indefinitely into the future without any humans. I think I would probably need to sit down and   just think about the numbers, but  maybe 2040 or something like that?  Ten years, basically, instead of..."

"than having to have experienced it. I do think  that a superintelligence back in the Roman era   could have guessed that eventually you could  get autonomous machines that burn something to   produce ..."

"This is an interesting rabbit hole, so let’s stick  with it before we get back to the intelligence   explosion. I think we’re treating this really  like all these technologies come out of this 1%   of..."

"I would also point out that even when we  make these random fortuitous discoveries,   it is usually an extremely smart professor  who’s been working on something vaguely related   for years in a first..."

"We’re just thinking that it would take  place faster than you might expect. You   were saying something like 10 years and  we’re saying something like one year. But   we are imagining this broad diffu..."

"competitors”, and they’re both going to do that. In fact, in our scenario, we have the AIs asking   for special economic zones where most of  the regulations are waived, maybe in areas   that aren’t s..."

"Scott, you reviewed Joseph Henrik’s book Secrets  of Our Success, and then I interviewed him   recently, and there the perspective is very much  AGI is not even a thing, almost. I know I’m being   a l..."

"fewer examples and this good research taste where  you can decide what things to look at to get these   examples, then you are still going to start off  much worse than an Australian Aborigine who has..."

"So I think that if you were to send a team of  ethnobotanists into Australia and ask them,   using all the top technology and all of their  intelligence to figure out which plants are safe   to eat no..."

"very fast. I think if we do get the world with  the Dyson sphere in five years, in retrospect,   it will look like everything was continuous and  everyone just tried things. Trying things can   be any..."

"different scenarios after some point. So,  yeah, what is a sort of crucial turning   point and what happens in these two scenarios? Right. So the crucial turning point is mid-2027,   when they’ve basi..."

"This is the August 2027 alignment crisis where  they notice some warning signs like this in   their hive mind, right? And in the branch  where they slow down and fix the issues,   then great, they slo..."

"in the same way that once the AI can do chess  and philosophy and all these other things,   eventually you have to admit it’s intelligent. So I think that each individual failure,   maybe it will make..."

"And that’s all just for the alignment stuff. But then there’s the concentration of power   stuff where somehow in the middle of doing all  of that, the powerful people who are involved   need to someh..."

"Maybe it just learns both of these things in the  right combination, I know there are many people   who say that’s very unlikely. I haven’t yet  had the discussion where that worldview makes   it into..."

"under control and the alignment plan they gave us  is actually correct, rather than something they’re   trying to use to deceive us? All of those  things I’m completely agnostic on, but that   leaves ..."

"are programmed to follow the commands of important  people rather than just doing things on their own.  If I may add to that. So by the government, I  think what Scott meant is the executive branch,  ..."

"votes on high level questions like “what goals  should we put into the superintelligences?”.  So, we were just getting lunch with a prominent  Washington, D.C. political journalist,   and he was makin..."

"but then also actually doing all the research. And part of why we think this wakeup happens is   because the company deliberately decides to wake  up the president. You could imagine running the   sce..."

"We also are pretty uncertain how much opposition  there’s going to be from civil society and how   much trouble that’s going to cause for the  companies. So people who are worried about job   loss, pe..."

"If the big bottleneck to the good future here  is just putting in, not this Eliezer-type galaxy   brain, high volatility, “there’s a 1% chance this  works, but we gotta come up with this crazy scheme ..."

"My answer, just based on this initial  discussion, seems to be nationalization.   Not only because it sort of deprioritizes the  people who want to think about safety and more   maybe prioritizes- the..."

"that it already exists. Yeah.  I flip flopped on this. I think I used to be  against, and then I became for, and then now I   think I’m still for, but I’m uncertain. So I  think if you go back in time..."

"I think traditionally in the AI safety community  there’s been this idea which I myself used to   believe, that it’s an incredibly high priority to  basically have way better information security.   A..."

"still do the hacking if you train against  it, they just won’t tell you about it”. You   can imagine very naive regulatory responses. It  doesn’t just have to be regulations, one might   be more optim..."

"On the other hand, if we just trust it to  the companies, they’re in a race with each   other and they’re full of people who have  convinced themselves that this is not a   big deal for various reason..."

"There’s other types of transparency too.  So transparency about capabilities and   transparency about the spec and the governance  structure. So for the capabilities thing,   that’s pretty simple. If ..."

"it doesn’t actually slow down the companies at  all. And it seems like a pretty decent ask to me.  If you told Madison and Hamilton and so forth  that- they knew that they were doing something   impor..."

"do the things that cause them to get reinforced. The thing I want to point out is that… Your   conclusions about where the world ends up as a  result of changing many of these parameters is   almost l..."

"So far these systems, as they become smarter, seem  to be more reliable agents who are more likely to   do the thing I expect them to do. So you have two  different stories, one with a slowdown where ..."

"exactly what they want from this interaction-  metaphorically, sorry, I’m anthropomorphizing-   which is the reinforcement. So we think that  this latter category of training failure is   going to get..."

"To be clear, we’re very uncertain about all of  this. So we have a supplementary page on our   scenario that goes over different hypotheses  for what types of goals AIs might develop in   training pro..."

"Yeah, we don’t know how this will work at the  limit of all these different training methods,   but we’re also not completely making this  up. We have seen a lot of these failure   modes in the AI age..."

"lie to your boss and keep lying to your boss, but  over time it’s just not going to work out with   you- or you become president or something, one  or the other. So if you believe in this extremely   ..."

"even if they are deceitful in some situations. I kind of want to call you out on the claim that   groups of humans don’t plot against other groups  of humans. I do think we are all descended from   th..."

"or in the president. Ideally, we want to have  it so that the legislature has a substantial   amount of power over the spec, for example. What do you think of the balance of power   idea of if there i..."

"approach than making some bespoke social program  where you make the same dialysis machine in the   year 2050 even though you’ve got ASI or something. I am also worried about UBI from a different   pe..."

"think might also help with this, I’m not sure.  But I think here’s a simple model. Let’s say   nine people out of ten don’t actually care and  would be fine with the factory farm equivalent   for the ..."

"liberalism in this sense. Like the United States  is so far a liberal country and we do ban slavery   and torture. I think it is plausible to imagine a  future society that works the same way. This ma..."

"you get some bonus if you agree. But whereas  what OpenAI did was unusual because it was like   your equity if you don’t. But non disparate  disagreements are actually somewhat common.  So basically i..."

"met up with him recently and he gives me, he made  a hundred translations of his favorite Greek poet,   Cavafy, and he gave me a copy. And it’s just this  thing he’s been doing on his side. It’s just ..."

"people write books and they write journal articles  and they write works in progress articles all   the time. So I’m back to not understanding this. No, I mean ChatGPT can write you a book. There’s   ..."

"What advice do you have to somebody who wants to  become good at it but isn’t currently good at it?  Do it every day, same advice as for everything  else. I say that I very rarely see new bloggers   w..."

"I have noticed two things in my own…  I mean, I don’t do that much writing,   but from the little I do: one, I actually was  very shallow and wrong when I started. I started   the blog in college. So ..."

"factor for so many of these people is courage  because everybody I talk to who blogs is within   1% of not having enough courage of blogging. That’s right. That’s right. And also “courage”   makes it ..."

"I wonder if this sends the wrong message or raises  expectations or raises concerns and anxieties. But   one idea I’ve been shooting around, and I’d be  curious about your take on this: I feel like th..."

"the old fashioned way, which is very long. Yeah. Okay. So you say that throwing money   at people to make them, to get them to blog at  least didn’t seem to work for the FTX folks.   If it was up to y..."