"Today I have the pleasure of speaking with  Carl Shulman. Many of my former guests,   and this is not an exaggeration, have told me  that a lot of their biggest ideas have come   directly from Carl es..."

"that is approaching human-level intelligence. The way to think about it is — we have a process   now where humans are developing new computer  chips, new software, running larger training runs,   and ..."

"But how much harder? There's a paper called  “Are Ideas Getting Harder to Find?" that was   published a few years ago. 10 years ago at MIRI,  I did an early version of this analysis using   data mainl..."

"the industry would be trivial. We're getting  more than one doubling of the effective labor   supply than we need for each doubling of the labor  requirement and in that data set, it's over four.   So..."

"and you wind up more restricted on this. Got it. The bloom paper said there was a 35%   increase in transistor density and there  was a 7% increase per year in the number   of researchers required to ..."

"instances of them and then they can do two  different jobs, manage two different machines,   work on two different design problems. Now you  can get more gains than just what you would get   by having..."

"to those models. Where is the loop here? The work involved in designing new hardware   and software is being done by people now.  They use computer tools to assist them,   but computer time is not the..."

"other R&D, you get thousands or tens of thousands  of people who are working on AI research.  We would want to zoom in on those who are  developing new methods rather than narrow   applications. So in..."

"There's a group called Epoch, they've  received grants from open philanthropy,   and they do work collecting datasets that  are relevant to forecasting AI progress.   Their headline results for what's..."

"greater investment. Part of that will come from  them having better models.You get the same effect   of increasing it by 10x just from having a better  model. You can spend more money on it to train a..."

"It can help with the hardware design. NVIDIA  is a fab-less chip design company. They   don't make their own chips. They send files of  instructions to TSMC which then fabricates the   chips in their ..."

"At what point would it get to the point  where the AIs are helping develop better   software or better models for future AIs?  Some people claim today, for example,   that programmers at OpenAI are us..."

"innovations, things like inventing the transformer  or discovering chinchilla scaling and doing your   training runs more optimally or creating flash  attention. If you move that from 8 months to 4   ..."

"consider that the output of our existing fabs make  tens of millions of advanced GPUs per year. Those   GPUs if they were running AI software that was  as efficient as humans, it is sample efficient, ..."

"an intelligence explosion. Intelligence explosion  has to start with something weaker than that.  Yeah, what is the thing it starts with  and how close are we to that? Because   to be a researcher at ..."

"intense application of the ways in which AIs have  advantages partly offsetting their weaknesses.   AIs are cheap so we can call a lot of them to  do many small problems. You'll have situations   wher..."

"the internet. We're getting towards the end of  that but yeah, as the AIs get more sophisticated   they'll be better able to tell what is a useful  kind of skill to practice and to generate that.   We..."

"the enormous number of programming challenges. In LLMs themselves, there's a paper out of   Anthropic called Constitution AI where they  basically had the program just talk to itself   and say, "Is th..."

"at each step along the way, does it look  like it makes sense to do the next step?   From where we are right now seeing the results  with GPT-4 and ChatGPT companies like Google and   Microsoft are pr..."

"if you're going to get the good version and not  the catastrophic destruction of the human race   or some other disastrous outcome. In between  it's a question of — how risky and uncertain   is the ne..."

"it opens up new applications, that is you're  not just improving your search but maybe it   makes self-driving cars work, you replace bulk  software engineering jobs or if not replace them   amplify p..."

"redirecting existing fabs to produce more AI chips  and they're just actually using the AI chips that   these companies have in their cloud for the big  training runs. I think that that's enough to go..."

"Yes, you definitely make the hundred billion one.  As you go up to a trillion dollar run and larger,   it's going to involve more fab construction and  yeah, fabs can take a long a long time to build...."

"of dollars, the world's best software engineers  can earn millions of dollars today and maybe more   in a world where there's so much demand for AI.  And then times four for working all the time.   If..."

"over succeeding years software progress turns out  to stall. You lose the gains that you are getting   from moving researchers from other fields. Lots of  physicists and people from other areas of com..."

"we can't keep going with this rapid redirection  of resources into AI. That's a one-time thing.  If the current scale up works we're going to  get to AGI really fast, like within the next   10 years o..."

"in this area, so in the 2000s which was  before the deep learning revolution,   how would I think about timelines? How did  I think about timelines? And then how have I   updated based on what has bee..."

"really intensive massive brute  force search and things like   evolutionary algorithms can produce intelligence. Isn’t the fact that octopi and other mammals got   to the point of being pretty intelli..."

"a lot of what is different about us with a brute  force story which is that you expend resources   on having a bigger brain, keeping it in good  order, and giving it time to learn. We have an   unusua..."

"Yeah, in general, there are trade-offs where the  extra fitness you get from a brain is not worth it   and so creatures wind up mostly with small brains  because they can save that biological energy a..."

"more of our attention and effort on learning  which pays off more when you have a bigger   brain and a longer lifespan in which to learn in. Many creatures are subject to lots of predation   or diseas..."

"development of cubs that play with each other.  Why isn't this a hill on which they could have   climbed to human level intelligence themselves?  If it's something like language or technology,   human..."

"in most of these cases it's enough to invest  more but not invest to the point where a mind   can easily develop language and technology  and pass it on. You see bits of tool use in   some other prima..."

"And we can look at some human situations. There's  an old paper, I believe by the economist Michael   Kramer, which talks about technological growth  in the different continents for human societies.  ..."

"allowed us to expand our population. They created  additional demand for intelligence so our brains   became three times as large as chimpanzees and  our ancestors who had a similar brain size.  Okay...."

"should just continue to become very smart. Yeah, we are growing them in a technological   culture where there are jobs like software  engineer that depend much more on cognitive   output and less on t..."

"exponentially decay as you expend resources. Oh, that's a really good point.  Chinchilla scaling would suggest that for  a brain of human size it would be optimal   to have many millions of years of e..."

"Yeah, if you actually look at it quantitatively  that's not true and even in recent history it   looks like a pretty close balance between  the costs and the benefits of having more   cognitive abilit..."

"And then there's a trade-off about just cleaning  mutational load. So every generation new mutations   and errors happen in the process of reproduction.  We know there are many genetic abnormalities t..."

"adapting to new circumstances. Since agriculture  people have been developing things like the   ability to have amylase to digest breads and milk.  If you're evolving for all of these things and if   ..."

"the average observed return in income is  still only one or two percent proportional   increase. There's more effects at the tail,  there's more effect in professions like STEM   but on the whole it's..."

"That is so interesting. Not to mention of  course that if you had 2x the brain size,   without c-section you or your mother or both would  die. This is a question I've actually been curious   about fo..."

"I guess that is similar to how planes were  inspired by the existence proof of birds   but jet engines don't flap. All right,  good reason to think scaling might work.   So we spent a hundred billion ..."

"to rapid-rapid progress on that problem. Somebody  might think that with humans the reason the amount   of population working on a problem is such a good  proxy for progress on the problem is that the..."

"measuring cumulative production in a field,  which is also going to be a measure of the   scale of effort and investment, and people have  used this correctly to argue that renewable   energy technolo..."

"observed correlations, from ideas getting harder  to find, have held over a fair range of data and   over quite a lot of time. So I'm wondering what‘s  the nature of the deviation you're thinking of? ..."

"speed. And we ask well, could the human research  community have solved these algorithm problems, do   things like invent transformers over 100 years, if  we have AIs with a population effective popul..."

"Moore's law leads to more humans somehow  and more humans are becoming researchers.  You do actually have a version of that in the  case of solar. You have a small infant industry   that's doing thing..."

"is actually the long run growth of human  civilization itself and I know you had Holden   Karnofsky from the open philanthropy project on  earlier and discuss some of this research about   the long ru..."

"of it going in. But this population dynamic  is pretty analogous. Humans invent farming,   they can have more humans, they  can invent industry and so on.  Maybe somebody would be skeptical that with ..."

"really end loaded. Is that what you're saying? Yeah, we already see this in these really high   IQ companies or projects. Theoretically I guess  Jane Street or OpenAI could hire like a bunch of   medi..."

"There are lots of cases like that where partial  automation is in practice not very usable because   complementing other resources you're gonna  use those other resources less efficiently.   In a post..."

"have it wandering around your china shop. Yeah. Why is that not a good objection?  I think that that is one of the ways in which  partial automation can fail to really translate   into a lot of econom..."

"But then what is the reason  to think we'll be there?  The broad reason is the inputs are scaling up.  Epoch have a paper called compute trends across   three eras of machine learning and they  look a..."

"Okay, that's actually a really interesting point.  Now somebody might say, there's not some sense in   which AIs could universally speed up the progress  of OpenAI by 50 percent or 100 percent or 200 ..."

"When we think about the ways in which AI can  contribute, there are things we talked about   before like the AI setting up their own curriculum  and that's something that Ilya can't and doesn’t   do d..."

"Later on you get to the point where yeah, the AI  can do your job including the most difficult parts   and maybe it has to do that in a different way.  Maybe it spends a ton more time thinking about  ..."

"importance of key research employees and such.  I certainly think that some researchers add more   than 10 times the average employee, even much  more. And obviously managers can add an enormous   amo..."

"is where will the AI advantages and disadvantages  be? One AI advantage is being omnidisciplinary   and familiar with the newest things. I mentioned  before there's no human who has a million years   ..."

"AI knocked that down a lot faster than AI  researchers and students who had registered   forecasts on it. If you're getting top-notch  scores on graduate exams, creative problem   solving, it's not ob..."

"If we say that the technology is frozen, which  I think is not the case right now, Nvidia has   managed to deliver significantly better chips for  AI workloads for the last few generations. H100,   A1..."

"industries. When you produce a lot more, the costs  fall. ASML has many incredibly exotic suppliers   that make some bizarre part of the thousands of  parts in one of these ASML machines. You can't   ..."

"effective population or that you'd need this  many people to have this effect. We were talking   earlier about the trade-off between training  and inference in board games and you can get   the same p..."

"what we have is something doing things like making  a ton of calls to make up for being individually   less capable or something that’s able to drive  forward AI progress. That process is continuing, ..."

"we already have an eight months  doubling time for software progress   then by the time you get to that kind of a point,  it's maybe more like four months, two months,   going into one month. If the t..."

"The way I would see this playing out is as the AIs  get better and better at research, they can work   on different problems, they can work on improving  software, they can work on improving hardware,..."

"and further software gains are running out.  They start to slow down again because you're   getting towards the limits. You can't do  any better than the best. What happens then?  By the time they're ..."

"explosion look like? They have trouble going from,  well there's stuff on servers and cloud compute   and that gets very smart. But then how does what  I see in the world change? How does industry or ..."

"this software-based intelligence explosion that's  mostly happening on server farms then you have   minds that have been able to really perform on  a lot of digital-only tasks, they're doing great   o..."

"technology, possibly with quite different  technology. But just taking what we've got,   right now the industrial robot industry produces  hundreds of thousands of machines a year.   Some of the nicer..."

"Say you convert the auto industry to robot  production. If it can produce an amount of mass   of machines that is similar to what it  currently produces, that's enough for a   billion human size robot..."

"view if you ignore the delay of capital  adjustment of building new tools for the workers.   Just raise the typical productivity for workers  around the world to be more like rich countries   and get ..."

"so if you go to something that's like 10x the  size of the current car industry in terms of its   production, which would still be like a third of  our current economy and the aggregate productive   c..."

"that a thing does. How much do they contribute to  these tasks? And I'm using this as a start to try   and get back to the physical replication times. I guess I'm wondering what is the implication   o..."

"[unclear] That's not to say that's the  limit of the most that technology could do   because biology is able to reproduce at faster  rates and maybe we're talking about that in a   moment, but if we'r..."

"talking about how they could take over, is  it best to start at a subhuman intelligence   or should we just start at we have a human-level  intelligence and the takeover or the lack thereof?  Differen..."

"There would be some tendency to bandwagon.  You have some some small startup, even if   they make an algorithmic improvement, running  it on 10 times, 100 times or even two times,   if you're talking ..."

"some discussion by my Open Philanthropy  colleague, Ajeya Cotra, she has a piece called   default outcome of training AI without specific  countermeasures. Default outcome is a takeover.   But yes, we..."

"So this is a way in which we could have AIs train  that develop internal motivations such that they   will behave very well in this training situation  where we have control over their reward signal a..."

"What do you find most promising? General directions that people are   pursuing is one, you can try and make  the training data better and better   so that there's fewer situations where the  dishonest..."

"Yeah, so in the limit, if we have positive  reinforcement for certain kinds of food sensors   triggering the stomach, negative reinforcement  for certain kinds of nociception and yada yada,   in the l..."

"How it is that you wind up with some humans being  enthusiastic about the idea of wireheading and   others not. And you could do experiments with  AIs to try and see, well under these training   condi..."

"this is inside [unclear] for the audience, but  basically you can't cleanly infer what quality   a single neuron stands for. This neuron is  about Alexander the Great or this neuron   is about my desi..."

"you're in pretty good shape. And there are ways  for that situation to be relatively stable. We   can look ahead and experimentally see how changes  are altering behavior, where each step is a modest ..."

"if the AIs conspired and are able to coordinate  then they have the potential to just walk off   the job at the same time. That's a failure mode.  If humans still have the hard power though, if   you ..."

"AI models has been altered in that way. My picture of aligned subhuman AI to   the superhuman AI being aligned is still murky.  If you can talk about that more concretely.  Eliezer’s claims were somet..."