"Today I have the pleasure of speaking with  Carl Shulman. Many of my former guests,   and this is not an exaggeration, have told me  that a lot of their biggest ideas have come   directly from Carl es..."

"that is approaching human-level intelligence. The way to think about it is — we have a process   now where humans are developing new computer  chips, new software, running larger training runs,   and ..."

"But how much harder? There's a paper called  “Are Ideas Getting Harder to Find?" that was   published a few years ago. 10 years ago at MIRI,  I did an early version of this analysis using   data mainl..."

"the industry would be trivial. We're getting  more than one doubling of the effective labor   supply than we need for each doubling of the labor  requirement and in that data set, it's over four.   So..."

"and you wind up more restricted on this. Got it. The bloom paper said there was a 35%   increase in transistor density and there  was a 7% increase per year in the number   of researchers required to ..."

"run more instances of the existing AI, but to  train larger AIs. There's hardware technology,   how much you can get per dollar you spend on  hardware and there's software technology and   the softwar..."

"where at a place like DeepMind salaries were  still larger than compute for the experiments.   Although more recently tremendously more of the  expenditures were on compute relative to salaries.   If ..."

"the growth of those capabilities is that pretty  consistently the capabilities are doubling on a   shorter time scale than the people required to  do them are doubling. We talked about hardware   and ..."

"and growth in budgets are as follows — For  hardware, they're looking at a doubling of   hardware efficiency in like two years. It's  possible it’s a bit better than that when   you take into account ..."

"in which the “effective compute” would increase? Looking at it right now, it looks like you might   get two or three doublings of effective  compute for this thing that we're calling   software progre..."

"of improving chip design that those people  are working on now but get it done faster.   While that's one thing I think that's less  important for the intelligence explosion.   The reason being that w..."

"innovations, things like inventing the transformer  or discovering chinchilla scaling and doing your   training runs more optimally or creating flash  attention. If you move that from 8 months to 4   ..."

"the 168 hour work week, they can have much more  education than any human. A human, you got a PhD,   it's like 20 years of education, maybe longer  if they take a slow route on the PhD. It's   just no..."

"Yeah, what is the thing it starts with  and how close are we to that? Because   to be a researcher at OpenAI is not just  completing the hello world Prompt that Copilot   does right? You have to choos..."

"if you're going to get the good version and not  the catastrophic destruction of the human race   or some other disastrous outcome. In between  it's a question of — how risky and uncertain   is the ne..."

"it opens up new applications, that is you're  not just improving your search but maybe it   makes self-driving cars work, you replace bulk  software engineering jobs or if not replace them   amplify p..."

"redirecting existing fabs to produce more AI chips  and they're just actually using the AI chips that   these companies have in their cloud for the big  training runs. I think that that's enough to go..."

"Yes, you definitely make the hundred billion one.  As you go up to a trillion dollar run and larger,   it's going to involve more fab construction and  yeah, fabs can take a long a long time to build...."

"of dollars, the world's best software engineers  can earn millions of dollars today and maybe more   in a world where there's so much demand for AI.  And then times four for working all the time.   If..."

"over succeeding years software progress turns out  to stall. You lose the gains that you are getting   from moving researchers from other fields. Lots of  physicists and people from other areas of com..."

"we can't keep going with this rapid redirection  of resources into AI. That's a one-time thing.  If the current scale up works we're going to  get to AGI really fast, like within the next   10 years o..."

"in this area, so in the 2000s which was  before the deep learning revolution,   how would I think about timelines? How did  I think about timelines? And then how have I   updated based on what has bee..."

"really intensive massive brute  force search and things like   evolutionary algorithms can produce intelligence. Isn’t the fact that octopi and other mammals got   to the point of being pretty intelli..."

"a lot of what is different about us with a brute  force story which is that you expend resources   on having a bigger brain, keeping it in good  order, and giving it time to learn. We have an   unusua..."

"seeing supported from biology and from our  experience with AI where you can explain —   Yeah, in general, there are trade-offs where the  extra fitness you get from a brain is not worth it   and so c..."

"more of our attention and effort on learning  which pays off more when you have a bigger   brain and a longer lifespan in which to learn in. Many creatures are subject to lots of predation   or diseas..."

"where we can learn a lot from our groups. And that  means it pays off to just expand our investment   on these multiple fronts in intelligence. That's so interesting. Just for the audience   the calcu..."

"Evolution doesn't have foresight. The thing  in this generation that gets more surviving   offspring and grandchildren is the thing that  becomes more common. Evolution doesn't look ahead   and think ..."

"that brain. But yeah, primates will use sticks to  extract termites, Capuchin monkeys will open clams   by smashing them with a rock. But what they  don't have is the ability to sustain culture.   A p..."

"then Australia, and then you had smaller island  situations like Tasmania. Technological progress   seems to have been faster the larger the connected  group of people. And in the smallest groups,   l..."

"Okay. And the crucial point in relevance to  AI is that the selective pressures against   intelligence in other animals are not acting  against these neural networks because they're   not going to get..."

"we're explicitly training them to become more  intelligent. So we have good first principles   reason to think that if it was scaling that  made our minds this powerful and if the things   that preven..."

"kind of interested. You referenced Chinchilla  scaling at some point. For the audience this   is a paper from DeepMind which describes if  you have a model of a certain size what is   the optimum amou..."

"Chinchilla scaling would suggest that for  a brain of human size it would be optimal   to have many millions of years of education  but obviously that's impractical because of   exogenous mortality fo..."

"Yeah, if you actually look at it quantitatively  that's not true and even in recent history it   looks like a pretty close balance between  the costs and the benefits of having more   cognitive abilit..."

"And then there's a trade-off about just cleaning  mutational load. So every generation new mutations   and errors happen in the process of reproduction.  We know there are many genetic abnormalities t..."

"adapting to new circumstances. Since agriculture  people have been developing things like the   ability to have amylase to digest breads and milk.  If you're evolving for all of these things and if   ..."

"the average observed return in income is  still only one or two percent proportional   increase. There's more effects at the tail,  there's more effect in professions like STEM   but on the whole it's..."

"That is so interesting. Not to mention of  course that if you had 2x the brain size,   without c-section you or your mother or both would  die. This is a question I've actually been curious   about fo..."

"I guess that is similar to how planes were  inspired by the existence proof of birds   but jet engines don't flap. All right,  good reason to think scaling might work.   So we spent a hundred billion ..."

"to rapid-rapid progress on that problem. Somebody  might think that with humans the reason the amount   of population working on a problem is such a good  proxy for progress on the problem is that the..."

"measuring cumulative production in a field,  which is also going to be a measure of the   scale of effort and investment, and people have  used this correctly to argue that renewable   energy technolo..."

"over quite a lot of time. So I'm wondering what‘s  the nature of the deviation you're thinking of?  Maybe this is a good way to describe what happens  when more humans enter a field but does it even  ..."

"similar to the humans but running 100 times as  fast and so. You have to tell a story where no,   the AI can't really do the same things as the  humans and we're talking about what happens when   the ..."

"enough to replace large chunks of the electric  grid. Earlier you were dealing with very niche   situations like satellites, it’s very difficult  to refuel a satellite in place and in remote   areas. ..."

"of it going in. But this population dynamic  is pretty analogous. Humans invent farming,   they can have more humans, they  can invent industry and so on.  Maybe somebody would be skeptical that with ..."

"need to have an atmosphere there. You need a bunch  of supporting tools and resources and materials   and those supporting resources and materials  will do a lot more productively working with   AI an..."

"that progress then you don't have the loop, right? I mean this is why we're not there yet.  But then what is the reason  to think we'll be there?  The broad reason is the inputs are scaling up.  Epoch..."

"the next 10 years or so. And so if you started off  with a kind of vague uniform prior, you probably   can't make AGI with the amount of compute that  would be involved in a fruit fly existing for   a..."

"Okay, that's actually a really interesting point.  Now somebody might say, there's not some sense in   which AIs could universally speed up the progress  of OpenAI by 50 percent or 100 percent or 200 ..."

"When we think about the ways in which AI can  contribute, there are things we talked about   before like the AI setting up their own curriculum  and that's something that Ilya can't and doesn’t   do d..."

"Later on you get to the point where yeah, the AI  can do your job including the most difficult parts   and maybe it has to do that in a different way.  Maybe it spends a ton more time thinking about  ..."

"importance of key research employees and such.  I certainly think that some researchers add more   than 10 times the average employee, even much  more. And obviously managers can add an enormous   amo..."

"is where will the AI advantages and disadvantages  be? One AI advantage is being omnidisciplinary   and familiar with the newest things. I mentioned  before there's no human who has a million years   ..."

"AI knocked that down a lot faster than AI  researchers and students who had registered   forecasts on it. If you're getting top-notch  scores on graduate exams, creative problem   solving, it's not ob..."

"and weaknesses than the human students. Sure, sure. Would it be possible for this   intelligence explosion to happen without any  hardware progress? If hardware progress stopped   would this feedback ..."

"some economies of scale from just making so many  fabs. And this is applicable in general across   industries. When you produce a lot more, the costs  fall. ASML has many incredibly exotic suppliers  ..."

"Got it. And when you say there would be  a greater population of AI researchers,   are we using population as a thinking tool of  how they could be more effective? Or do you   literally mean that the ..."

"Okay, we accept the model and now  we've gone to something that is at   least as smart as Ilya Sutskever on all the  tasks relevant to progress and you can have   so many copies of it. What happens in..."

"then each doubling can come more rapidly and  we can talk about what are the spillovers?  As the models get more capable they can be  doing other stuff in the world, they can spend   some of their tim..."

"You can do all of these things and AIs wind up  being applied where the returns are highest.   Initially the returns are especially high in doing  more software and the reason for that is again,   if ..."

"and further software gains are running out.  They start to slow down again because you're   getting towards the limits. You can't do  any better than the best. What happens then?  By the time they're ..."

"and now I'm skipping ahead of the most important  months in human history. I can talk about   what it looks like if it's just the AIs took  over, they're running things as they like.   How do things e..."

"One of the most immediately  accessible things is where we   have large numbers of devices or artifacts  or capabilities that are already AI operable   with hundreds of millions equivalent researchers..."

"all the existing robotic equipment or remotely  controllable equipment that is wired for that,   the AIs can operate that quite well. I think some people might be skeptical   that existing robots give..."

"But looking around the body. There's legs to  move around, and not only that necessarily,   wheels work pretty well. Many factory jobs and  office jobs can be fully virtualized. But yeah,   some amoun..."

"ramped up military production by converting  existing civilian industry. And that was   without the aid of superhuman intelligence  and management at every step in the process   so yeah, part of that ..."

"billion human size robots a year. The value per  kilogram of cars is somewhat less than high-end   robots but yeah, you're also cutting out most  of the wage bill because most of the wage bill   is pa..."

"view if you ignore the delay of capital  adjustment of building new tools for the workers.   Just raise the typical productivity for workers  around the world to be more like rich countries   and get ..."

"can use for the initial robot construction. Cognitive tasks are being automated and the   production of them is greatly expanding and  then the physical tasks which complement them   are utilizing hum..."

"easy, really solid. There's an enormous  margin there. We were talking before about   skilled human workers getting  paid a hundred dollars an hour is   quite normal in developed countries for  very i..."

"that a thing does. How much do they contribute to  these tasks? And I'm using this as a start to try   and get back to the physical replication times. I guess I'm wondering what is the implication   o..."

"in say one year, if you have to build a whole new  factory to double everything, you don't have time   to amortize the cost of that factory. Right now  you might build a factory and use it for 10 year..."

"doubling time that is less than a year. I think  significantly less than a year as you get into it.  So in this first first phase you  have humans under AI direction   and existing robot industry and ..."

"to reproduce at least this fast. At the extreme  you have bacteria that are heterotrophic so   they're feeding on some abundant external food  source and ideal conditions. And there's some   that can ..."

"There would be some tendency to bandwagon.  You have some some small startup, even if   they make an algorithmic improvement, running  it on 10 times, 100 times or even two times,   if you're talking ..."

"some discussion by my Open Philanthropy  colleague, Ajeya Cotra, she has a piece called   default outcome of training AI without specific  countermeasures. Default outcome is a takeover.   But yes, we..."

"doesn't necessarily have to be that this AI  will survive because it probably won't. AIs   are constantly spawned and deleted on the  servers and the new generation proceed. But   if an AI that has a ..."

"If we wind up with this situation where  we were producing these millions of AI   instances of tremendous capability, they're all  doing their jobs very well initially, but if we   wind up in a situat..."

"What do you find most promising? General directions that people are   pursuing is one, you can try and make  the training data better and better   so that there's fewer situations where the  dishonest..."

"maybe we could and humans are trained with simple  reward functions. Things like the sex drive, food,   social imitation of other humans, and we wind up  with attitudes concerned with the external wor..."

"Yeah, so in the limit, if we have positive  reinforcement for certain kinds of food sensors   triggering the stomach, negative reinforcement  for certain kinds of nociception and yada yada,   in the l..."

"I don't want to do that because the heuristics and  predictors that my brain has learned don’t want   to short circuit that process of updating. They  want to not expose the dumber predictors in my   ..."

"they're relatively motivated to tell us if they're  having thoughts about, have you had dreams about   an AI takeover of humanity today? And it's a  standard practice that they're motivated to do to  ..."

"and of contributing to a larger legitimate  process. That's a goal you can aim for,   getting an AI that is aimed at doing that and has  strong guardrails against the ways it could easily   deviate fr..."

"And on the other hand, these AIs In  their spare time or in ways that you   don't perceive or monitor appropriately or  they're only supervised by other AIs who   conspire to make the AI takeover happ..."

"the job at the same time. That's a failure mode.  If humans still have the hard power though, if   you still have situations where humans are looking  at some of the actual outputs that are produced, ..."

"AI models has been altered in that way. My picture of aligned subhuman AI to   the superhuman AI being aligned is still murky.  If you can talk about that more concretely.  Eliezer’s claims were somet..."