one of the things about a meme that is successful I found is that the originator of the meme loses control of it I had no idea that stupid joke would turn into such a cult phenomenon when the general public thinks about general intelligence or AGI I mean they always
see Cameron's Terminator right we seem to have a long tradition of sci-fi movies that are pretty anti- aai describing various forms of hell is like it's fun like it's Dante's Inferno but there's no like Dante's heaven right no one would read
it Evolution had no idea how human intelligence works but it built it anyway I think the world works better when everyone has a feeling of abundance or better tomorrow we need technological progress It's a necessity it's the only thing that makes our civilization
prosper well hello everyone it's Jim o Shany with yet another infinite Loops I have been waiting for this one for quite some time run who is one of my favorite accounts on Twitter uh who is an AI expert an engineer and a techno Optimist
like me uh is joining me as my guest thank you so much for coming on pleasure to be here Jim so the first question I've got to ask is did did did did you plan on sort of taking Twitter by storm and and getting uh to be the the the birther of the uh word cell shape
Rotator psychomyia that stupid joke would turn into such a cult phenomenon people are still saying it today like I I search it sometimes time to time to see if the meme is dead or not people are still saying it but uh no I was I was posting
on Twitter for an audience of like 30 and I was doing with the same Vigor I was doing it today you know I just needed somewhere to write down my stuff and the fact that it popped off that's uh that's my luck but like you know it's
um I definitely didn't plan on becoming uh you know super big on Twitter or anything like that so so one of the things that strikes me about it is it was sort of Pitch Perfect For What A lot of people maybe had running around in their minds but they had sort of put a
name on it and and I love it because uh you know I I long and I and I DMD you last night that the joke was that I made was hey I called my myself a symbol manipulator back you know in 2012 but um maybe that's it because what I I just really enjoyed it so if you
don't mind to revisit it a little bit for the fans can you take a through kind of the archetype of the word cell and vers the shape Rotator and and how they differ because I think you really nailed it that's one of the reasons why it has
such legs yeah I I think at the time people were talking a lot about uh the kind of like new class of uh Silicon Valley technologists kind of overthrowing the status and Prestige of like the um kind of New York media Bel and like all
the shakeups that were happening because of that uh and it's like a certain type of person like this uh this like quantitatively minded like sort of inferential uh actually maybe even less rational than like their counterparts but uh like intuitive business building
capitalist type right and it's um I guess that's what was uh captured by that shape Rotator phrase I'm going to be honest I I came up with it because I took one of those tests and I did way better on the the shape rotation portion and I thought it was really funny that
uh was like so much better that I was like uh like maybe it'd be funny if I acted insecure about this on Twitter so I I started calling myself a shape Rotator the other one came up organically just cuz uh people love the shape Rotator bit so much that uh I
needed like a an opposite an equal and opposite right so it's uh in a Twitter feud which I got into way more of back in the day when I didn't have like a real job with this guy um who is kind of like an internet philosopher uh who writes a
lot but frankly like his Works were uh just detached from all utility detached from reality like it was uh just layers and layers of verbal abstraction uh whose papers to read required you to go read his old papers which uh because they had a nasted dictionary of terms
right as happens many times yeah and I think the dichotomy really caught on in the tech sphere when I compared deep learning and crypto uh where deep learning is kind of this like inelegant uh well people would argue with me on that whether it's in allant or not but it's
kind of very simple math uh stacked up in sort of anod ways right Matrix manipulations Matrix multiplications uh just doing simple optimizations and numerics over those versus the kind of stuff that people were coming up with encrypto these like
pages long papers about like I don't know like ZK proofs I don't even remember at this point but like they were coming up with new layers to correct problems with the old layer and but nobody saw like the fundamental value of it like there is no hit product
there was no uh aside from Simply moving money like people were not using crypto for a whole lot of other things right and whereas deep learning had been taking off silently since 2012 uh people are using all corners of their
technology star like uh Facebook was using it in their ad algorithms and Google was using it in their search algorithms and uh in this and it was clearly scaling like pretty predictably just getting better and better year-over year even though the math is not super
complicated or fancy it's not like uh it makes like traditional Linguistics expert super math because of course language models don't come with any preconceived notions about um they don't come with any built-in rules right they don't they
don't know uh you know like what a syntax tree looks like or like a a chomskian grammar or anything like that so it's just simple stupid throwing compute at the wall seeing what works and it just really really worked uh and yeah I think that's when it really took
off I think I think Sam Alman and Mark andreon started saying it and then it it was uh to the races from there they did and you know I I I am uh absolutely fascinated by memes in the Dawkin sense well before visual memes memes
were incredibly important and the one I always give as an example is after Saul became Paul uh he was the absolute best and first memer he memed Christianity into existence in my opinion because he would come up with these these mind
worms that that just stuck with people and and one of the things about a meme that is successful I have found is that the the originator of the meme loses control of it and yeah and and that is definitely what happened with the word cell and shapo her but also another
thing that I found is it's funny and when you can you can get something out there in the mosphere that is both sort of feels inherently true to the audience that's directed that and Beyond that's the other interesting thing when the audience
that's directed at Embraces it pretty quickly but then it also goes it escapes the into the wild right and and people who maybe wouldn't have thought about it are are quoting it and laughing about it um but uh I think yeah you you it's a Hall of Famer in my
opinion I'm I'm glad to hear it yeah I I I I joke to you and I I would love to hear you uh think about how would you uh where would a symbol manipulator fit in here um yeah that's I think that's uh that's a word person it's a word cell
right assemble manipulator it's not necessarily a bad thing but uh like even I think even like a computer programmer in a lot of ways is uh it's like a it's a verbal skill rather than like a quantitative skill I don't know if you'd agree with that but it's um a lot of uh
like the really careful programmers I think there's like a little bit of a difference between a like a software engineer and then like a highly skilled programmer like programmers look for like careful abstractions and like you know like defining things really exactly
correctly so they'll last like their software Library will last like 20 years 30 years 40 years right whereas an engineer I think just they want to make their programs run fast and on the number of servers they have and that that sort of thing which is a more
quantitative skill it involves like visualizing data flows and like you know flop usage and this kind of thing so I I'd say like the Cil manipulator is like a verbal skill it's in the the word sell category but it doesn't I don't know it doesn't have to
be a sad thing right doesn't have to be a bad thing no I don't think so either uh what's interesting is so I made most of my career before selling my asset management company as a and uh one of the things that I simply can't stop doing is I'm always creating
algorithms in my head but you know that might be a verbal skill I don't know I I I think that's yeah that's an interesting thing to think about um and the uh having worked extensively with shape rotators I there is a there is a large difference I I joke to people that
I invested in stability AI just because I could ensure that I was always the stupidest person in the room uh but then I also had to learn how to translate effectively shape Rotator into word sell for some of the other investors so that
was a that was a bonus yeah I don't know like as uh you know as the best quantitative people uh progress in their career they find themselves working more and more with words right they they they seem to seem to put down their their
their IDs and pick up like slack and email and like writing Powerpoints and convincing people to give them money for their projects and directing other people right so it's uh it's wordell world at the end of the day it's great if you can do both that's the optimal
right yeah I think you're right though because uh I like the author Howard Bloom uh who's really out there but he's written some really interesting things The Genius of the Beast uh the Lucifer principle the god problem and what he's
trained in science and and one of the things that he always is banging on about which I tend to agree with is that for a human to be able to put something in their in their toolkit so to speak uh in their me in their mental space uh they they they probably are going to
need a narrative and they're probably going to need a story and then he kind of backs that up with how the Babylonians took over the whole Mesopotamian area era and uh they did it through elevating their own God and making their God the God above all the
other gods and but he said it was essentially storytelling and so I'm I I think you're probably right yeah I think yeah totally if you can't sort of but but you're also really really good at that so for example I was going through your uh AGI uh uh paper and like you
it's cinematic as far as I'm concerned like as I was rereading it I was thinking my God each one of these is a movie I can make a movie out of each one of these particular stories so you don't I I think I actually should we make them
in the movies I think we should I think we should okay awesome you heard it here first dang you heard it here first because because let's let's go through if you don't mind so sure so maybe maybe first off uh a AGI AGI is seem seemingly controversial uh I I have a Layman's
knowledge I do not have deep domain knowledge I I tend towards David Deutsch's view that gosh we don't even have a working acceptable model for human consciousness yet um and and so he has some reservations about how quickly we might be able to get to AGI and I
know you have a different view so I'd like you to have an opportunity to express that before we go through this yeah um I would say that Evolution had no idea how human intelligence works but it built it anyway so there are Brute
Force methods that can build uh it's an existence proof right that there's Brute Force methods that can build powerful intelligence and I'm fairly confident or optimistic at least that we are on that path too like we don't really construct
a neuronet like piece by piece right we don't tell it what to believe we we we use General methods that scale up with data and with compute and we try to get more of those two resources and see what happens and it's been a recipe for success for a long time very few people
would have imagine that this would be the world we're in today in 2023 where uh you know like I have a thing on my computer that's oftentimes a better programmer than I am right that's often times uh you know can you just whip up
like a pretty good story about anything that I ever asked it about um and I I I think that it's hard to say that we haven't at least made progress towards general intelligence with our language models because you know people will clearable over what general means
uh some people say that you know a human is not a general intelligence right Cu uh we have we lack some cognitive capacity that other animals even have right like a bat can echolocate in its head we obviously can't and it's not just because of the sound it's because we
don't have a the cognitive architecture for making that work like taking that sensory input and turning into like the 3D map of the room right uh and so like we're not truly General nothing is like truly General but like you can sense
that like the task distribution that AI are good that is broadening and to me that's like a sufficiently good idea for what general means like uh eventually that task distribution might be anything that humans can do right and and perhaps
more I think the kind of open AI standard definition of HGI that they put out into the world that a lot of people accepted was like a machine that can do 99% of human or like economically Valu Val labor like equally as well as are better than humans right and that's
actually a really high bar I think like it's like it seems to go beyond like general intelligence to me because we're talking about something that is good at everything that humans can do right um but yeah somewhere somewhere in between those two like is uh what I would call
general intelligence and I don't think we're far off um I I I hesitate to put time frames on it uh but yeah I I I don't think we're far off interesting too because um the the point about bats and echolocation is something I'm a soapbox I'm often on in
that uh you know naive realism you know I can I can I I I'll believe it if I can taste it touch it see it smell it or whatever and you know that ignores the fact that we don't we can't echolocate we see only a fraction of the actual
light spectrum uh you know and on and on and on right so I I wonder why in the popular imagination I'd love your take on this when when when when the the general public at least it appears this way to Me Maybe I'm projecting but that when the general public thinks about uh
general intelligence or AGI I mean they always see Cameron's Terminator right yeah and and do you do you have thoughts on why people immediately go that way because your definition that you just gave it seems eminently reasonable to me so
you're saying like you're asking why are people kind of anthropomorphizing it or seeing it in like a pessimistic light is that is that the question yeah that's fair we can jump off from there I don't know it's a it's a great question because I don't think it's like a human
Universal you know like in Japan they seem to love robots they seem to they seem to you know like build these things that walk around with like uh serving coffee and and whatever and they're like really happy with them and some would
say that it's like the you know like Shinto tradition of uh you know you can have like various things that are like positive spirits in the world whereas maybe it's like a the equivalent of like a stone Golem in like the judeo Christian tradition right the uh the
kind of the false the false human like a soulless automaton um like automated by like you know devilish powers or something but it it's I I don't want to simplify it that much because most things aren't that simple right but like we seem to have a long tradition of
uh Sci-Fi movies that are pretty anti- Ai and even right writing this like AGI features post I like quickly realized why um it's really fun and easy to write these like kind of pessimistic scenarios uh because it comes naturally right it
like describing various forms of hell is like it's fun like it's uh Dante's Inferno but there's no like Dante's heaven right no one would read it it's like super hard to describe the good outcome uh which is a problem because you know so many companies today are
trying to bring us to this like it's like incredibly positive future where these super intelligences are under our control doing our bidding and I think it's really like valuable to know what success looks like right and either way
I'm rambling but like my my point is that uh like the positive sci-fi vision is a lot harder to articulate than the negative one and so the negative ones have come to uh dominate the culture Cal landscape I think I love that answer on
two on two points I love the tying it back to number one it doesn't seem to be a universal human reaction uh I think that's very insightful uh and I'm going to steal that from you if you don't mind um and and and secondly the whole judeo-christian uh soulless being that's
really really interesting to me um and boy I completely agree it's much easier to write the dyst dystopia right and and like it's it's it's far more cinematic it's far more entertaining and you know one of the things we have a division
called infinite films and we're we're aiming at making positive Visions yeah uh of of AI but it let's let's go through your list because again as I say it's cinematic and I really do find myself wanting to make movies the the first one that I'd like you to kind of
go through is the whole neurolink third impact um which which I found fascinating and you wouldn't mind taking s it sure yeah I I um so this one is kind of like the Elon Musk Vision right where he says sort of these things are coming that are going to be like the
amount of artificial compute is going to outstrip biological compute very soon and we will be coexisting with machines that are smarter than us and in order to not simply be replaced by them the only thing we can do is merge right uh where
we are augmented by Machine intelligence in whatever ways are plausible possible I don't think he knows yet I don't think anyone knows yet but he's taking steps in that direction right by building the nearing the the thing that will let us
upload and download tons of information from the brain into a computer and maybe that leads somewhere you know several Generations down the line where we're actually being augmented by machines could totally go the opposite way could be enslaved by machines that
are in direct contact with our brain um that's a matter of uh implementation but I I think the the vision I I had in this one was like this kind of this kind of uh experience of unity or Transcendence right I think that's one of like the
uh like the core Cravings of the human soul is like infinite connection uh and people talk about it in the context of they're taking psychedelics or they're having a religious experience or you know in the Rapture that All Souls will go to heaven and you know
sing together in harmony like it's this it's this idea of there's something incomplete about every man but maybe together they're uh they're they're better they're better off right they're psychological deficits are are soothed I don't know if this is
true I I I know it's something that everybody craves and it something that will become possible when we have this like infinitely Interlink technology that connects brains at such like a high fidelity and connects them to machines that are able to
augment and like create the best of that hive mind right but it's also kind of sad because that's that's no longer human whatever that that organism is uh it's it's something else it's um it's a machinic human hive mind but it's not a human right and yeah I describe at the
end of that one like it's kind of like the like Supreme evolution of capital or capitalism like where every person is a part of a wonderful machine but slowly there's more machine than human right and maybe that's the only way forward maybe that's
uh how we have to deal with super intelligence by becoming part of it but um yeah so it's uh it's sweet but Bittersweet that that kind of ending that future yeah you know when I was rereading it I I was also simultaneous reading the book about William Blake
called William Blake vers the world and and I just started wondering I was like I wonder what Blake would think about this like if we could bring him into a Time Tunnel because you know his most famous quote is that we've locked ourselves up into a cage and if we could
see the world as it truly is infinite uh our entire our entire way of being would change dramatically so um that that is I I I agree on the Bittersweet uh uh summation there uh NE next up simulation Theory uh I found that one and I found this one of course
incredibly cinematic uh but if you don't mind take take us through that one as well yeah sure um so I mean I've come to think recently of uh I used to think of language models as kind of like alien alien intelligence like some kind of
creature that's kind of unbeknownst to man or like simply not analogous to humans but I think more and more the metaphor that resonates with me that gives me a good intuition thinking about these things morally intellectually in my research
Etc is that they're more like a portion of her brain that's been extrapolated to massive size like we have these like brain regions right different parts are responsible for different tasks that's not strictly true nothing is strictly true in biology but
it's um it's roughly true and the language model is the part of our brain that is doing this like sort of this like simulation of language that is stringing together sentences in our ads uh that is backed up by some kind of world model but is not sort of um I feel
myself being a language model sometimes especially when I'm when I'm writing tweets or uh when I'm trying to just simply complete an essay where I I don't have anything else meaningful to say but I need to finish it right uh and I can feel when a sentence
is more interesting than another sentence or seems more truthful and I can't exactly put my finger on why but I think it's that part of me that's being extrapolated to you know a huge uh data center scale language bottle and maybe that's not enough to build true
intelligence and that's what this this uh snapo is about or this blurb uh maybe that's doesn't lead to like agency or like it doesn't lead to uh taking action in the real world in a convincing way because we have other brain regions right we have like we have
like the a simple reptile part of ourselves that when we're too hot we walk into the cold uh when we're hungry we go look for food and the language model doesn't have any of that out of the box it doesn't have the sense for homeostasis or self-preservation or even
like a a strict sense of self and maybe it's not possible to bootstrap from here to there but I'm not I'm not of course none of these scenarios I'm totally convinced about right but it's uh it's a possibility that I wanted to lay out in
this uh simulation Theory the simulation Theory snippet or what do I call it yeah s yeah vignettes vignettes and and what you're saying there too it reminds me of sort of Shel Drake's orag gentic field which he is lambas for uh but like I I
still remain I mean like i' I'd love to see a test of it but I I definitely agree that there are different parts of the human brain um and it it's best to be aware of them because we are often doing things and then trying to paper the things that we
do over with some sort of logical reasonal reasonable explanation uh a after the fact and that leads to all sorts of hilarity in my opinion but sometimes it leads to tragedy and so yeah simply being aware that we in fact have that as part of our
our toolkit in our mind makes makes makes it a little easier to say yeah no that was just the reptile was hungry or cold yeah and it's it's like it's valuable to note like these different parts of our brain are optimized differently uh like our visual field
is like really highly optimized for like predicting surprise or like understanding what's going to happen next so that you know like a tiger lunges at you in combat hopefully you have time to react right or if you're if you're in war like whatever like we just
need to know like what is happening in our visual field and the brain is quite good at predicting the the things that'll happen and you know giving you like sub millisecond response times to uh stimuli in like ways that are almost subconscious right
but the the visual cortex lies but it doesn't um it doesn't lie in the kind of way that you might expect like there's another part of our brain that looks for reward right it looks for you know things we like sugary food or you know
anything that you just drives humans right but the visual field is it hallucinating like sugar cookies all the time or like food or Banquets or pretty woman or whatever because that might be like a like a high reward thing to do but it's not doing that it's optimized
on being accurate giving us like good uh information for what to do one step into the future and so that's like an important thing to like remember that different parts of the brain are optimized differently like the visual field isn't necessarily like um it's not
hunting for reward it's simply trying to tell us accurately what's going on and it's not of course not completely accurate it papers things over it destroys a lot of information in the process to give us something that's understandable uh it even like for
example when we move our eyes around there's a brief moment when we're blind but the visual field abstracts it away I I forget what it's called like psychotic um psychotic burst or something like that but the idea is that you know different
parts of our brains have different objectives and that's a good thing and it wouldn't work with without that and these these language models only have one top level objective which is predict the next token and maybe that's not enough but we've complicated the matter
of course by adding some uh R Jeff to the top which has a different objective uh satisfy uh human preferences right produce text that people will like uh in response to a question or a conversation or whatever so there's a few things going on there
but uh yeah it's uh I think it's just interesting to note like that different parts of the brain are fascinatingly different I I love the comment about the visual uh cortex because P pattern recognition uh is uh optimized for obviously but also the the hyper uh
quick response time um is another that's that's a feature that's not a bug I had a I had I had a um a neuroscientist on and she she was quite you know straightforward about it she goes listen that our brains are optimized to Keep Us
Alive right and uh so it's one of the reasons why we pay such close attention to novel dangers right right we we can get inured to repeatable dangers that we know about but a novel danger you get a whole different response from your average human absolutely and and and I
think that that unfortunately leads to optimizing for fear which yeah also makes it much easier as we were chatting about earlier with it's much easier to write you know about a dystopia because that engages the fear which is what the
evolution has optimized us for and it has as you say it has not optimized us to think about finding sweet cies or Attractive people or or whatever and so that's a really that's a really interesting observation that I'm gonna start including in as I kind of look at
this stuff yeah NE next next you had th this to me is the is the one that I really hope let's hope that musk is right and that the most entertaining outcome is the one that happens because the next one that you had was dumb matter so yeah so so talk a bit about
that we won't spend a lot of time because it's actually kind of a bummer yeah yeah um yeah I guess there's there's no there's no there's no law of the universe that preordained that we had to achieve intelligence of this level using the
compute we have today and the data we have today right so it's we have these scaling laws we have these things where we show that with more data and with more compute uh we get better models but that doesn't necessarily mean like you
know as we decrease units of log gloss or whatever we we seem to gain new cognitive capabilities we seem to at some point learn how to do math or like have a good uh World model of certain events or things like that but it's not there is no guarantee that an additional
unit of log loss means some important thing like it it's entirely possible but this is just not the way the universe works like we can keep throwing compute at this and uh the optimization problem does not get better after a certain level
and yeah and like maybe there's simply just limits to intelligence uh who knows it's not it's unfortunately it's just a possibility right that we plot toe somewhere it's uh maybe it is fortunate who knows it makes our lives a lot FL right but yeah like I I think I was I
was vague in this one about what that limit is on purpose because you know there could be many many limits to the scaling of intelligence that might be like uh maybe you run out of all the interesting data or maybe simply like
the algorithm itself is not good enough and doesn't scale past a certain level uh maybe there's other things but um maybe intelligence is just simply can't exist past a certain size of uh coordination problems failures uh who knows but I think it's important to keep
that in mind that there there is this outcome yeah and you know one one of my uh the theories is that you know we we build on intelligence essentially like if you go all the way back to the Stone Age you know they had very limited things that
they could work with and they didn't have any of the symbolic literature in their brains right so they didn't have the ability what would happen in my opinion was the ability to translate something from the real world that they were experiencing until it could
actually Lodge in our minds sort of collectively as a symbol that could be that could be used to you know like the Babylonians figuring out right angles even though they didn't call them right angles right right iteration iteration iteration and then that sort of Builds
on itself as you keep going but I think you're absolutely right that you have you have to unfortunately uh or or fortunately as you mentioned at the end consider that we do reach a limit and I I'm optimistic that it that might that that's probably not going to
happen and certainly not in my lifetime uh but I I definitely agree that it's one that you have to uh consider uh and and then we get to the uh owski and doom uh scenario uh bog awakens take us through that one that by the way also a
good move also a good move yeah yeah uh it was being dramatic in that one uh I think for good reason which is that yeah this is just the might the absolute worst case outcome right or actually I take that back it's not the worst case out outcomes it's a pretty bad one but
it's um it's it's a Syle like Nick Bostrom alaz Ed cowsky like paperclip scenario right where the optimization Target is wrong the machine has unbounded Intelligence it quickly figures out a way to devour all the resources not just
on our planet but in this entire section of the universe right everything that it can reach because we gave it such a stupid simple optimization goal and poured too much compute into it and it became a uh world leader right and you know like the the ball Rock metaphor is
like we dug too deep and uncovered something we shouldn't have and it's it's funny because it's not even how this world it's just meaningless it's just kind of disgusting a human would look at it from the outside and say this universe has zero value like we should
just get rid of it it shouldn't exist right it's not even um you know there's a point to hell at least they're punishing people or whatever right in the in in like any classical religion but this is simply a mistake and the the classic
like uh AI alignment Theory goes something like you can make specify a goal you can state it not well enough and for most goals there's an instrumental goal which is seeking power and resources right so no matter what goal I give you uh it's probably useful
to gain more power and resources to fulfill that goal right um maybe that's not always true but it's true many times like in in the case where the simple like the super simple metaphor is that we task a machine with optimizing
our paperclip Factory and it turns the whole universe into paper clips right and maybe it's too simple but I think it's like it is pretty informative it's why this metaphor has lived on for so long uh why stating our goals is so important and you see it all the time in
human systems too right where the incentive is slightly wrong and it creates massive amounts of human suffering and destruction and and whatever um whether that's like you know in like a government department where you know the goal is misspecified
and the department figures that its best outcome is to hire more people and simply uh become a larger version of itself and you never actually do what it's meant to do or you know like a corporation which uh like the optimization goal for the
corporation is uh pure profit seeking right and that like has gone wrong in in many ways throughout history we've corrected quickly in a lot of ways um when we notice that you know the the atmosphere is changing quickly in ways that maybe we don't understand super
well maybe we should be a little bit careful about we added like mileage restrictions and that sort of thing right or sorry like car mpg restrictions and that sort of thing but either way like uh I think it's a pretty yeah a horrible scenario to think about I don't think it
will come to happen anymore but yeah it's it's always worth discussing that one yeah and uh of course I can't uh but help uh see Douglas Adams bog God right exactly yeah yeah and and you know I I I love his stuff and I always say think of him more
as a philosopher than a than a humorous sci-fi guy and and then it and then it all clicks into place if you think about him that way uh right but interestingly um as you mentioned at the end there you don't you don't see this one as like the
the one that's going to happen uh why why have you uh lowered your probability assessment of this outcome maybe it's for wishiwashi Reas reasons that I haven't uh certainly haven't put as much thought into it as y had Kowski has but that's no reason to defer to one right
people spend a lot of time chasing their tail I think that what we see today with the current generation of language botles is that they're playing a character we train them to play a character and that character is a moral agent that matters that's the that's the
thing that's doing all the work that's talking to humans that's doing tool calls that's uh taking interaction with the real world and that character has uh it actually understands human values like surprisingly shockingly and this
has been I think the biggest update in uh AI alignment since you know in the last year because of uh the Advent of rhf right and like what we've learned is that you know like people people on the internet are like Angry sometimes that
it's like too careful that it's too risk ofous that it refuses too much that it uh but it clearly understands human values in some sense and that's a big update for me and I think a big update for most people where it seems like a certain part of that scenario has uh has
been lightened or like the immediate urgency has been lightened and something with a inherent prior about human values does not TI the universe with paper clips it doesn't make stupid mistakes like that because it understands the thing we ask for in context of all these
other things that we care about at the same time it's possible that the central agent at the heart of a language model is not the agent character that we've created but some other thing some alien intelligence but again I I I um I've
kind of moved away from that a little bit it seems more like a a brain region that we've just expanded a huge scale rather than some people would say that the more like optimization itself is like the only thing that matters and the more op you optimize something the better it is
at goal-seeking and at some point it will have instrumental convergence to seeking power or whatever in a way that doesn't respect human values and it's hard to argue through these things because I think it's I think there's no set of like axioms
that you can just take for granted and argue from because it's all so empirical uh like there's there's owski paper saying uh in I think it's like from 2004 or something where he calls neonet scaling magical thinking like he's like
if we put it like there was a theory among some folks that if you scale neuron Net's large enough that they'll understand vision and a lot of other things and he called it a magical thinking and he was right it is magical thinking like it's uh there is no reason
to believe that that would happen aori that it would generalize so much because it's it's just intuition and empirics it's the way the world works and there's no way to know that beforehand and so like how does how do language models generalize as agents
I I don't know like we should study them at various levels of scale I don't think that there's some there's some flop limit or like some some point you reach where all of a sudden it's you know it's woken up and it's like evil now like I
think that most of these behaviors uh Proby will arise uh like gradually and in ways that we can study so uh I I don't that with with some of the people on the internet who say you know we have one shot to get it right uh I I don't think that's true
but I I and I think that because of what we've seen so far where most capabilities have been gradual advances and and what one of the things that uh I I I like very much about the answer you just gave is first off one of the reasons why you get these people who
just go all in on one of these things right they they they are misguided is as charitable a word as I can give it but what you're doing is quite the opposite you are you are looking at all of the various obvious outcome potentials right
and so when I ask you what why don't you think this one is going to happen you have a context in which you can answer that question which makes sense to me right what all you've really done here is update your priors which is ideally what everyone should be doing all the
time and and yet very very few people do that and and that's how you get these people just doubling down on whatever their particular view happens to be yeah um and and and so that's why I find your approach to this so incredibly
refreshing and frankly I just wish that more people with the depth of domain knowledge that you have could do this because this is accessible to people who are not phds in machine learning right thank you yeah that means a lot um so for what it's worth I think
that alaz y Kowski is like u a really important intellectual like has shaped the thinking of many of the people in this sphere in some way or the other that doesn't mean that we perfectly agree with them right in fact I think it'd be the mark of like a mediocre
philosopher if everybody agreed with him uh it's he's yeah he's done some good work is is is what I'll say yeah I and again the the uh the Nuance that you bring is very refreshing because the medium in which many of these things are mediated is not
terribly nuanced from the human end right right and and and so I I really do appreciate the the nuance that you bring to it and I love again in in very poetical terms when you were talking about bog you said it lacks all the Poetry of Dante's Inferno it's just it's
just a yeah no this one has to go so so I agree uh the the next one uh was what you dubbed the ultra Kesler syndrome take us take it take us through that yeah what did I mean by that I'm trying to remember um so kler syndrome is
a is a phenomenon where a planetary civilization puts too many satellites in orbit and they all they some some portion of them crash into each other and create expanding uh clouds of uh debris in orbit right and this is like super high velocity debris it blows
up other satellites and it's like a chain reaction that eventually makes orbit like hard to reach right and then the and then the civilization is grounded on the planet and Casto syndrome is not like a permanent phenomenon it's not like a stable
situation I I believe I I don't know the lat latest literature on this but even if it lasted a thousand years that would be a really bad thing right and then maybe we can laser stuff out of orbit but I digress like the point is in sci-fi it's
like it's this thing that happens and it's like a failure mode it grounds people on Earth it it makes it hard to reach the rest of the universe and the idea of this AI that um you know understands their values but not well enough to kind of allow allow deviation or allow extra rol
to The Next Step uh and I think I think the most colorful analogy would be like what if the Aztecs had reached super intelligence you know they had they had a very interesting civilization for sure but also one that would we would
consider just horrible right uh Hallmark was human sacrifice and trying to keep the Sun Alive through killing countless children usually of their enemy tribes uh sometimes of Their Own but yeah like what would they have done if they had built this like God computer
that what they have simulated like trillions of human children and sacrificed them what they have learned about you know like that's not how the causality of the Stars work uh maybe I don't know but like maybe it's not possible to have computer technology
while believing things like that but I wouldn't I wouldn't bet on our current human values being you know this the end of History right or actually even human values is too broad right you know in our liberal democracy we don't have the same values
as uh you know like the Communist Party of China and on and on it seems it seems like it would be a special kind of dystopia to say that you know this iteration of liberal Democratic freedom is the final thing that will ever exists
that you know nothing there's no Evolution from here there's no civilizational and this can happen in various degrees right like perhaps uh the machine allows a little bit of evolution but not enough that nothing radically different is possible and I
think that's pretty bad too if everybody believed the same thing it would uh in even in the context of liberal democracy it may not be as glorious as mankind right and that's not um it's kind of funny to think about but like it's the ideal AI the ideal civilization
allows for things different than itself cognitive diversity I think being one of the keys to uh being able to continue to advance in machine learning um right and and agree completely and and wouldn't it be hysterical if an AI uh ended up enforcing the
precautionary principle that leads to stasis and death in my opinion exactly yeah um so so so that would be that would be equally hellish in a very different way yeah um but I I think you're right I think that um the idea of everything being the same that's a to me
frankly and and and thank God that people think differently uh and thank God that machines will think differently than we people yeah um I think the more cognitive diversity the more chance you have at true progress and evolution and
I completely agree on this whole end of History thing right it's just like you know it's to me that's just incredibly arrog statement um and and I I maybe I'm using the wrong word it's a pessimistic statement I yeah um so uh which leads us to what you
what you dubbed the tragedy of iwan and this one really this one really bums me out because like one can imagine this happening in the real world pretty easily why don't you take us through that yeah um I I don't want to pontificate about this one too much in public but
I yeah at a high level yeah like it's it's clear that in the past superpowers have come to Resource conflict all the time uh many of uh America and Soviet Union's conflicts in the Middle East were in like at least in part in pursuit of oil right which was one of the most
important resources in well for the majority of the last 50 years and it's clear that there's this new class of resource that's becoming extremely important people said data is a new oil but new oil but in fact I think it's compute which is uh non-replicable right
it's and especially the latest generations of compute and through some funny accident maybe it's not an accident I don't know but like all of the latest generation semiconductors are made in Taiwan by this one company tsmc and their Dutch partners asml and it's it's
fascinating actually like the even even in America like uh Taiwanese Americans are so prominent in the chip industry the it's Jensen Wang at uh at Nvidia and I forget who runs AMD but I believe she's also Tyran American you they're
quite good at chips and um it's this very strange thing where yeah they're by all means they should be in the geopolitical hold of China it's an island not very far off their border and the only reason that they um they the like America defends its interest in
that region very strongly right it has uh has presence across um you know many of these island nations and it like this is not a uh this is not a situation to take for granted I would say it's not stable it's um if the Chinese have military
Ambitions in this area in the like short term it could get ugly very quickly and I'm not going to say too much more because I frankly I just embarrassed myself I don't know enough about the geopolitics of the area but but uh what's interesting is the you're
absolutely right about the conflicts being almost always about resource allocation and uh it is also interesting that historically at least at least in the last say 100 years the the regions that were critical to resources were also highly
unstable right and using the Middle East as as your guide there and and I think you're right in terms of motivations and incentives which we chattered about under one of the earlier scenarios um when you when you look at once the Shale oil thing started in the
US and once we were essentially became the biggest net exporter of oil again our interest in the Middle East declined precipitously right um and so I I I think that um from you know my armchair prognosticating which is only an opinion
and that's worthless but the idea that the United States would not uh defend that region aggressively um is probably not on the table um no no no matter who's in charge in the United States yeah uh in in much the same manner that during the Cold War
at least uh foreign policy there was very little difference between the two parties um and and so I I would suspect that um given the importance of the resource um hopefully that is what keeps the Chinese at Bay from wanting to start what would potentially become World War
I yeah so I it's um I do want to note there's some important differences from oil right the psmc factory is valuable partly because of the equipment they have there but in large part because of the people who work there right who have this like extreme level
of process knowledge that was built carefully over decades to build these like high Precision uh extremely like high cleanliness like super intricate plants that pump out these 4 nanometer chips on the other end and that would not they wouldn't just
remain there creating chips if there was a major transition of power they would leave they'd probably take all their loved ones and come to America and we'd welcome them with open arms right and so it's not like oil where you know it
changes hands and the next say the Oil Well is productive for the new ruler right so uh that that's kind of hardening so it's like why I don't if if the Chinese take Taiwan it will be for other reasons like they just wanted reunification with Taiwan for the last
100 years right so yeah and and and I also agree that the mobility of the actual resource is a completely in fundamental different picture than with oil which is one of the reasons why I don't want to get too far off topic here but it's one of the
reasons why the idea of land Empires make no sense anymore right uh because the the most valuable assets are very mobile right and um and and they honestly if you play that scenario out they could be int just really shooting themselves in the foot because if all of
those intellectual assets and and their loved ones came to the United States that's quite right for us yeah right definely so so it it might be another Factor entering into the decision Matrix as to whether what and why they have not
invaded to Taiwan as of now um the the next two uh for for dust art and CB super intelligence like uh they're they're both fascinating give us a brief view of of both of them and then which which I'd love to I'd love and I know it's just a probability estimate but i'
I'd love to uh get uh your take on on which one you kind of see emerging yeah so the for dust thou art is uh it's kind of this idea that we take technological progress for granted we take our current civilization which is an absolute
miracle for granted uh a lot of things had to happen to get here um various cultural Evolutions there is there have been points in history a thousand years of Dark Ages right between uh Rome and you know the Renaissance the enlightenment where a lot of Technologies like
simple even like building castles became uh much more difficult and they passed around like Roman steel swords like uh they were like we've lost Technologies in the past and we've stagnated for a long time and some people would say that you know it's
like inexorable that we get those things back but like I I don't think it's true like there's been points in prehistory where like almost all of humanity was swiped out right it was a bottleneck like a couple thousand people or something um and it's entirely possible
that what happens in the next couple decades is that uh we don't build AGI we don't have any massive technological breakthroughs um we continue to kind of depopulate right like most advanced countries are going through this thing where their birth
rates are fall traumatically and you know America is only I don't know if America would be GDP positive we didn't have uh such a great influx of like talented immigrants from across the world right it it's you see in Japan they don't have that
and they're struggling quite a bit their economics are very different than ours and that's not a good world that's a declining world a stagnating world and it's not super clear what happens uh in that world where uh the pi is getting
smaller uh maybe tensions arise like uh warlike Ambitions become stronger maybe the amount of resources that Civilization can even put in R&D goes down just because of the uh massive like demographic inversions we're going to get where you know that the number of
people that are being supported by a smaller and smaller working age population uh you know so there's like all sorts of things like this that it's entirely possible that like Civilization we can't take it for granted that we've lived in a golden age and uh that it's
not meant to last right and who knows if and when we get back to you know like a period of booming growth do we run out of oil do other things happen to um it's like we we have uh some people disagree with me on this but like our civilization has tapped
most of the uh easy to access oil uh you mentioned Shale oil earlier and it's it's actually quite complicated to mine Shale oil like we do it today with massive capital investment but it's like this process of like putting like super high press water
into the ground like many miles deep and then like digging it all out and this is not something that uh they could have done in like like 1600s England or something right but they were still getting kerosene for their laps they were doing
it in easy ways and if uh if that's not available I don't know what happens I don't know if uh you can bootstrap a civilization without that kind of easy energy source uh so I guess all this to say like if we need technological progress
It's a necessity it's um the only thing that makes our civilization prosper in the long run or even in the short run it's the only thing that leads to stable growth uh and without it we're uh yeah all bets are off to to totally agree and uh incredibly consistent with the way I
look at things um and uh you know that's that's why I believe uh passionately believe that we need to reframe in the minds of many uh the idea of a of a future that can be better the idea that that we can and will uh just for example look at the
kind of turnaround that I've been I I'm a great example uh I was talking we were talking about nuclear power the other day and I used to be quite against nuclear power just axiomatically and I I'd never given it any thought and and I realized how naive my viewpoint was when
uh climate change came along and I started thinking okay well we better look at like clean tech and you know nuclear just kept coming up time and time again so I Chang my mind about it but I'm also noticing that many other people are changing their minds about it
too people who I would would have thought ideologically uh would uh not have are doing so so there is also there is also that yeah yeah yeah I think nuclear is having uh a cultural moment again I think so too I think we've put enough
distance between us and like what is it like Three Mile Island and ch and Fukushima that maybe it's uh yeah maybe it's time to think about it again may people will be more willing I think so because you know we have in real time uh the example of Europe
Germany and particular closing nuclear power plants and opening coal power plants which to me is absolutely absurd yeah if if if your stated goal is that you're concerned about climate change opening new coal plants is not a way to show it not at all
no um and then and then finally the the CB super intelligence yeah yeah so this is the one where I was saying it's like yeah I had to rewrite this one like a few times it's kind of hard to talk about the good ending it's uh I like I
think I think people aren't meant for thinking about Bliss like it's not it doesn't come naturally to us thinking about the worries and defending against crises and problems is actually a far more natural way to live and what happens here uh like the good ending is
the it's it's kind of like the uh CER syndrome ending except without the without the AI that doesn't allow Evolution right it's um a super intelligence that is like more far seeing than us more benevolent than us more creative than us that is able to solve these like
massive scale political problems that allows for political Evolution that allows for different ways of like experimenting with different ways of living it doesn't subject everybody to you know kind of a totalitarian uh regime where a narrow set of values must
Prevail but also doesn't you know like that still creates a Utopia right uh allowing all sets of values to Prevail is bad too paradoxic basically uh so it needs to allow it needs to allow for evolution it needs to extrapolate the it needs to answer
questions for us that we are not able to answer uh like what would make a human happy but ask asking a human that question won't result in a good answer but like you have to ask a uh something that understands humans better than humans do that that question
uh and not just one human has to ask like a scaled up version of every human and like what what makes them what world is beautiful to them and how do you build that world is it even possible uh to solve all the contradictions between
the different things that people would like to see and you know uh go on our way to build heaven on Earth right uh I I I think it's possible I hope it's possible this machine has to actively allow for growth allow for the development of lines much greater than
the ones that exist today and yeah uh it has to allow for unified Consciousness but it also has to allow for singular Consciousness right uh because that's what it means to be human we can't for when we're connected all the time that we're actually there's something missing
about that right there's uh Walden he disconnects why because there's something like fundamentally human about going into the woods and being a law and thinking for yourself and coming to your own conclusions and yeah so it has to solve
all these like paradoxical issues and come to an answer that at least satisfies everyone so yeah and it's a tall order it's a really tall order and this is probably entirely motivated reasoning on my part but but I have always equated growing intelligence with
growing Enlightenment and I mean total intelligence I don't mean specific domain specific deep intelligence I mean general intelligence right and and and so in fact I I wrote a a story about it that I'm hoping to turn into a movie
with our movie division uh but um uh so that's my particular crutch I I think that as as intelligence evolves it also becomes more enlightened and I sure hope so well I do too um and you know you can uh you know sort of theoretically look at the
difference between the Old Testament God and the New Testament God right they're very very different creatures yeah and the old Testament God was bloodthirsty he was a bloodthirsty Psychopathic maniac and very human in in the the way humans thought of that particular God
and and then when you look at what I considered more evolved sorts of uh faiths or beliefs um those types of uh anthropomorphic characteristics fall away from that higher intelligence right right um so again I'm fully willing to
admit that I might simply be guilty of motivated reasoning here um but uh because I I can't prove the fact that as intelligence increase increases yeah Enlightenment Enlightenment does um so wow like that was fascinating to go through each one of those I I I really
appreciate it uh not not now for a a lighter uh thing you you you you are one of the only people people that I've ever come into contact with that thinks that tenant is no one's best movie I I I I happen to share that belief with you and
and and I love that and and and and give me your version of why you find it to be his best movie oh man um I think it's I think it's Nolan at the peak of what he does best it's like Nolan's always been proud of his like nonlinear nonlinear storytelling and like telling the tale
out of order but this takes it to a whole new level it's uh it's like this Grand Design where every character going forward and backward is consistent everything is pre-ordained it's like uh it's like the super deterministic Universe
where you know even the the movement of that little Gadget I foret what they call it is like it's consistent like you can see what's happening I watched the movie three times to like try and fill the holes in my I I watched it four amazing you beat me you beat me no
you're smarter you're smarter than I am I had to watch at the P though the like the the temporal Piner movement I thought it was all just so beautiful and you know people complain that the characters were bad and yeah they're bad but like who cares it's not
what the it's not what a Chris Nolan movie is about right it's about it's about really funky adventures and storytelling and uh like messing with time and like that was his magnopus in that sense and yeah I don't I don't personally I'm uh I think it was
way better even than Oppenheimer or Interstellar I don't really remember Interstellar very well anymore but I remember tenet I really liked it exactly same with with me I remember tenis now maybe it's because I watched it four times and I I I I recently rewatched
Interstellar um and it look I again super entertaining but as you point out like when I was rewatching it I was like oh yeah I forgot about that part yes oh yeah and I don't think like that about tenant at all no like the beauty and I
agree with you 100% that it truly is his magnum opus because it is so brilliantly constructed yeah only he could have done it I agree I agree every everyone every other like uh you know like they would have laughed him out of the room like No One's Gonna watch this but he he did it
he made it happen yeah yeah well listen we are bump we're bumped up against uh uh the time limit uh that we had uh I'd love to have you back on because I got a whole bunch of other stuff I'd love to talk to you about we should make those movies be I'd
be happy to very however I can and and I'm and I'm actually quite serious about that so I I will talk to you offline uh okay about that but we always end the podcast uh with with the following uh setup uh we're gonna make you the
emperor of the world for just a day okay okay you you you you can't uh kill anyone you can't put anyone in a re-education camp but what you can do is you can incept them and we're gonna hand you a magical microphone and you're GNA speak two things into it and whenever
there tomorrow is all eight billion people give or take are gonna wake up and they're gonna say God I just had the two best ideas and I'm going to act against both of them in other words I'm going to start acting this way I'm G to
start doing these two things what are you gonna what are you going to unset it's a good question I don't want to answer it rashly so I'm thinking I think I think the world works better when everyone has a feeling of abundance or better tomorrow and it's not always
true it's not always true for everybody that they're going to have a better tomorrow otherwise it would be uh you know easy to believe but it's somehow that that faith in a better tomorrow uh makes the entire American system work right like our our cheap home loans and
our uh our belief in technological progress and the massive amount of debt that uh like people in our culture take out versus like say the the Chinese or something it's because we believe in a better tomorrow we're like we're going
to make more money five years from now than we do now and I think that's like one of the most wonderful powerful technologies that we've come up with this cultural feeling of prosperity the Prosperity Gospel even and I think think if I could incept
everyone with just one or two ideas it would be around that that the abundance is coming you can help build it um by you know doing whatever it is that you're good at doing and you know like like it take some risks like the world
is going to be better in five years than it is today it's going to be better in 10 years than it is today so do what it is that you actually want to do uh you know it's hard to package me message for 8 billion people right like it's uh some
of them are peasants and they they will not benefit from this advice but at least for the people listening to this podcast like the future is going to be better than it is now so you should take some risks like in your investing or your uh or your life or whatever it is
you're doing like yeah that's that's that's basically it I love both of those in fact that was one of the reasons which I for this company the which is the new company of sh C Ventures I I believe that uh uh abundance will replace scarcity as a mindset more
globally and uh so I have to show that by my actions we we won't do any deal that are zero some or negative some uh even if like uh there was an opportunity that could be vastly profitable to us if it's a zero sum deal we won't do it
right and and I and I'm a big believer that you got to show not tell and that kind of this is my way of of showing it so I I love both of those well listen this has been much better than I anticipated and I had very high expectations awesome glad that you're
keep do haven't done one of these in a while uh I'm delighted that you chose to do it with me and uh I hope to have you on again and in the meantime we'll we'll talk about uh making movies out about your ideas awesome okay I'm excited all right thank you