"the problem is that we do not get 50 years to try and try again and observe that we were wrong and come up with a different Theory and realize that the entire thing is going to be like way more diffic..."

"about what's inside there so if it were up to me I would be like okay like this far no further time for the summer of AI where we have planted our seeds and now we like wait and reap the rewards of th..."

"this model to a vast Corpus of text of people discussing Consciousness on the internet which means that when it talks about being self-aware we don't know to what extents it is repeating back what it ..."

"I don't know I feel like when you run over this science fiction guard rails like maybe this thing but what about gbt maybe maybe not this thing but like what about gpt5 you know this this would be a g..."

"difficulty removing all mention of emotions from gpt's data set I would be relatively surprised to find that it has developed exact analogs of human emotions and there I think that humans have well li..."

"think it's possible to investigate and study the way neuroscientists study the brain which is look into the darkness The Mystery of the human brain by just desperately trying to figure out something a..."

"and interestingly like people have started to get preliminary results showing that reinforcement learning by human feedback has made the GPT series worse in some ways in particular like it used to be ..."

"Transformer Networks of of neural networks which if if reasoning is not impressive to you or it is impressive but there's other levels to achieve I mean it's just not how I carve up reality what's uh ..."

"I'd rather not be wrong next time it's a bit ambitious to go through your entire life never having been wrong um one can aspire to be well calibrated like not so much think in terms of like was I righ..."

"and felt the resulting description into one of the stable diffusion things I think and you know she you know it's she's pretty and this is something that should have been like an amazing moment like t..."

"and stable diffusion output it and there's the concern about how much the discourse is going to go completely insane once the AIS all look like that and like are actually look like people talking um a..."

"can acknowledge it's caring now it's being trained by this imitation process followed by reinforcement learning on human in human feedback and we're like trying to point it in this direction and it's ..."

"it's fake because it is fake because we have been trained them training them using imitative learning rather than and this is not spontaneous um and they keep getting smarter do you think we would osc..."

"you're going to have a whole group of people who can just like never be persuaded of that because to them like being wise being cynical being skeptical is to be like oh well machines can never do that..."

"indistinguishable to me other people might have had better Distinction on it indistinguishable blob of different AI methodologies all of which are promising to achieve intelligence without us having t..."

"that said I was never skeptical that evolutionary computation would not work in the limit like you throw enough computing power at it it obviously works that is where humans come from um and it turned..."

"to do which is an opinion that has changed less than my opinion about whether you or not you can actually do it do you think AGI could be achieved with a neural network as we understand them today yes..."

"too late now but don't you think others would do it eventually you shouldn't do it first like if if you already have giant nuclear stockpiles don't build more if some other country starts building a l..."

"so can we still man the case for some level of transparency and openness maybe open sourcing so the case could be that because gpt4 is not close to AGI if that's the case that this does allow open sou..."

"resource I do not believe in the practice of Steel Manning there's something to be said for trying to pass the ideological Turing test where you describe your opponent's position uh the disagree disag..."

"what I'm saying and one interpretation is really stupid and whack and doesn't sound like me and doesn't fit with the rest of what I've been saying and one interpretation you know sounds like some like..."

"uh compelling eloquent way these are the points that elieza would make to to make the case in this case that hey it's gonna kill all of us but that that that's what steel Manning is presenting it in a..."

"in the way you think do you bet money on it and do you bet money on their beliefs when you believe them are we allowed to do probability sure you can State a probability that yes there's there's a loo..."

"if if he thinks it's dangerous rather than asking me I think there's a lot of mystery about what intelligence is what AGI looks like so I think all of us are rapidly adjusting our model but the point ..."

"way um yeah that uh the laws of physics are actually wrong that kind of update no no like just like oh like I Define intelligence this way but I now see that was a stupid definition I don't feel like ..."

"things like going to the Moon because if you generalize sufficiently far and sufficiently deeply chipping Flint hand axes and outwitting your fellow humans is you know basically the same problem is go..."

"a way to measure general intelligence I mean I could ask that question a million ways but basically is will you know it when you see it it being in an AGI system if you boil a frog gradually enough if..."

"to turn back past there isn't it possible that there's a you know with a frog metaphor you can kiss the Frog and it turns into a prince as you're boiling it could there be a phase shift in the Frog wh..."

"and you've written about this it's just fascinating which is the thing that happened with gpt4 that I guess they don't describe in papers is that they have like hundreds if not thousands of little hac..."

"um it is possible that like all these little tweaks are things that like save them a factor of three total on computing power and you could get the same performance by throwing three times as much com..."

"us so I I guess I could but I would offer to instead say like drop that empathy with me I bet you don't believe that why don't you tell me about how why you believe that AGI is not going to kill every..."

"systems both are a little bit terrifying but yes the worst one is the paper Club maximizer something totally boring but to me the positive and we can we can talk about trying to make the case of what ..."

"Advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer and in that report uh summarizing some of the major subfields of artif..."

"poorly aligned superintelligence and it killed us all we got to observe how it had killed us and you know not immediately know why but like come up with theories and come up with the theory of how you..."

"try is that is that is that what you believe do you think and if so why do you think that's true we can't do research on alignment before we reach this critical point so the problem is is that what yo..."

"now the reason this doesn't work is that people can be smart enough to realize that the answer you're looking for is yes I'm a good person and say that even if they're not really good people so the wo..."

"versus like aspects of it that naturally talk that way have been reinforced I I wonder if there could be measures of how manipulative a thing is so I think of uh Prince mishkin character from uh The I..."

"disagree with that I mean it's a it's a beautiful idea to start over but I don't I think fundamentally the system is trained on human data on language from the internet and it's currently aligned with..."

"trying to do when I interact with other humans is trying to fit in trying to play the a a robot trying to play human characters so I don't know how much of human interaction is trying to play a charac..."

"and yet it is there it's a perturbation on this little on this slice through you how beautifully did you put it it's a slice through you it may even be a slice that controls you I'm gonna think about ..."

"thought going on in there which is very unlike human thought and is directed around like okay what would a human do over here and well first of all I think it matters because there are there's you kno..."

"different people in there exactly like the people but how much of it is like learn how much of it is by gradient descent getting optimized to perform similar thoughts as humans think in order to predi..."

"externally do but in terms of like internal Machinery that started to be present it's just because we have no idea of what the internal Machinery is that we are not already seeing like chunks of Machi..."

"okay uh to push back on that you can in that's true but there's a difference between over the board chess in person and simulated games played by Alpha zero with itself yeah so is it possible to have ..."

"the strong systems that have learned to lie to you can't weak AGI systems help model lying like what uh is it such a giant leap that's totally non-interpretable for weak systems can cannot weak system..."

"talking to gpt3 right here so like uh help me out here help me well I like I'm trying I'm trying not to be like I'm also trying to be constrained to say things that I think are true and not just thing..."

"out up to right now and you're probably trying to slow down the capability gains because there's only so much you can speed up that alignment stuff but leave that aside we'll mention that also but may..."

"definitely a long way off because progress is really slow that sounds sensible to us rlhf thumbs up produce more outputs like that one I agree with this output this output is persuasive even in the fi..."

"for example and it feels to me like I have watched the field of alignment just fail to thrive except for these parts that are doing these sort of like relatively very straightforward and legible probl..."

"suggestor just learns to exploit the the flaws in the verifier you don't think it's possible to build the verifier that's powerful enough for agis that are stronger than the ones who currently have so..."

"you growing up not I would not say growing exponentially first because the word exponential is like a thing that has a particular mathematical meaning and there's all kinds of like ways for things to ..."

"be smarter than you know something can be smarter than Earth getting 100 years to think but nonetheless if you were very very smart and you are stuck in a little box connected to the internet and you'..."

"you think like the aliens computers are much faster than the aliens and you are running at the computer speeds rather than the alien brain speeds so if you like are asking an alien to please cop you o..."

"yeah yeah uh you're not an ideally perfect programmer but you know you're a better programmer than the aliens the aliens are just less man they're good wow and I'm much much faster a much faster looki..."

"changed if you did it right yeah you know you don't want the aliens to notice yeah what's your next step uh presumably I have programmed in me a set of objective functions right like no you're just Lu..."

"live life and so messing with one aspect of the system you have to be very careful how you improve that aspect without destruction so you're still Lex yeah but you think very quickly you're Immortal y..."

"compares to different impact of the world of different Technologies the different Innovations of the invention of the automobile or Twitter Facebook and social networks they've had a tremendous impact..."

"impressive AI systems even recommender systems you can just imagine those kinds of systems manipulating you you you're not understanding the nature of the manipulation and that escaping I I can Envisi..."

"much smarter than you what's what's a good thought experiment that you've relied on to try to build up intuition about what happens here uh I have been struggling for years to convey this intuition um..."

"step along that interval of around the same length and people's minds just go blank like how do you even do that so I can and we can and I can try to like break it down and consider what it would mean..."

"true things because the the basic Paradigm of machine learning that we are presently operating under is that you can have the loss function but only for things you can evaluate if what you're evaluati..."

"State getting worse and there's nothing you can do about it so you're not there's no there's no like uh there's no brain power making progress in trying to figure out how to align these systems you're..."

"an off switch that's that's a fundamental part of it's not smart enough to to prevent you from pressing the off switch and probably not smart enough to want to prevent you from pressing the off switch..."

"the sound then it escapes from whichever system you've built the almighty lever into and copies itself somewhere else so your answer to that research question is no yeah but I don't know if that's a h..."

"cynicism about such a thing not being possible if you haven't done it works on the first try what so yes so yes again something smarter than you so that's that is a fundamental thing if it has to work..."

"kind of alignment A system that escapes the box it's from is a fundamentally different thing I think for you yeah no but not for this so you put a line there and everybody else puts a line somewhere e..."

"but like this is not something we cannot take for granted that there's going to be an outcry yeah people have different thresholds for when they start to outcry PT for granted but I I think your intui..."

"and there are like basic obstacles where you've got the the weak version of the system that doesn't know enough to deceive you and the strong version of the system that could deceive you if it wanted ..."

"still far away from that I think a lot of people are surprised by the capabilities of gpt4 right so now people are waking up okay we need to study these language models I think there's going to be a l..."

"interpretability the yacht versus the interpretability research the old uh the old trade-off uh I I just I think uh it's just I think there's going to be a huge amount of allocation of funds I hope th..."

"kill you now what it is definitely a good step one right yeah what's step two if you cut out that layer is it going to stop waiting to kill you when you optimize against visible misalignment you are o..."

"but do you think it's possible because we can do experimentation to discover the source of the desire to kill I can tell it to you right now is that it wants to do something and the way to get the mos..."

"exactly right that it didn't be like oh I will like detach their heads and put them in some jars and keep the heads alive forever and then go do the thing but leaving that aside well not leaving that ..."

"more drastic the ones you could the failure modes are much simpler it's it's like yeah like the AI puts the universe into a particular state it happens to not have any humans inside it okay so the pap..."

"destroyed then that got changed to a fable of like well you made a paperclip Factory and it did exactly what you wanted what you wanted but you asked it to do the wrong thing which is a completely dif..."

"allocate a lot of resources to the alignment problem well I can easily imagine that at some point this Panic expresses itself in the waste of a billion dollars spending a billion dollars correctly tha..."

"off Target being wrong in a way that's mixed stuff easier you know that's that's not the usual project management story yeah but then this is the first time we're really tackling the problem of AI ali..."

"nonetheless imagine that there is this hill climbing process not like gradient descent because gradient descent uses calculus this is just using like where are you but still hill climbing in both case..."

"tries to optimize the very simple loss function you are training it on there is so much that we cannot possibly cover all of it I think we did a good job of getting your sense from different perspecti..."

"an intelligence in it do most of them not include humans I mean if you the vast majority of randomly specified utility functions do not have Optima with humans in them would be the like first thing I ..."

"would it be true there's a line from Grumpy Old Men where this guy says in a grocery store he says you can wish in one hand and crap in the other and see which one fills up first this is a science pro..."

"and if you if like you restrained A reproduction those genes get less frequent in the Next Generation compared to your con specifics and natural selection doesn't do that in fact Predators over run pr..."

"if you're like reading on restrained populations and so that's like the sort of so this is not a smart optimization process natural selection is like so incredibly stupid and simple that we can actual..."

"it would be by an AGI system well it's kind of stupid it like has to like run hundreds of generations to notice that something is working it doesn't be like oh well I tried this in like one organism I..."

"really really hot well let me ask you let me talk to you about Consciousness um also coupled with that question is imagining a world with super intelligent AI systems that get rid of humans but nevert..."

"little tool like we can get rid of I guess I'm trying to get some color in your opinion of how useful it is in the uh intelligence of a human being and then try to generalize that to AI whether AI wil..."

"is in part a solution to the human alignment problem I don't I think the human alignment problem is a terrible phrase because it is very very different to like try to build systems out of humans some ..."

"a human it's probably also generally smarter at building AI systems this is the ancient argument for foom put forth by IJ good and probably some science fiction writers before that um but I don't know..."

"so example of like fraction of sub debate and the thing that Robin Hansen said was more complicated than that and like a brief summary he was like well you'll have like we won't have like one system t..."

"over it as of gpt4 but don't you think there's uh going to be potentially definitive moments when we say that this is a sentient being this is a being that is like we would go to the Supreme Court and..."

"people are already saying that I think they're probably mistaken but I'm not sure because nobody knows what goes on inside those things they're not saying that at scale okay so the question is the I t..."

"features that are not significant relative to the broader intelligence that we're talking about so perhaps gpt4 is already there but to have the the video where woman's face or man's face to whom you ..."

"do you think ego is empowering or limiting for the task of understanding the world deeply I reject the framing so you disagree with having an ego so what do you think about it I I think that the quest..."

"debated AI systems and I think that the person who won that debate was guern and I think that reality was like side of the utkowski handsome Spectrum like further from utkowski and I think that's beca..."

"some kind of wisdom and insight you can give to how to clear your mind and think clearly about the world man this is an example of like where I wanted to be able to put people into fmri machines so th..."

"like safe and convenient and like didn't really inconvenience anybody and then everybody died everywhere there was enough actual like oh we're going to die we should not do that we should do something..."

"future it probably doesn't exist but it's beautiful that you're looking for ways that you're wrong and it's also beautiful that you're open to being surprised by that same young physicist with some br..."

"of uh Evolution and that finiteness doesn't seem to be obviously integrated into it and AI systems so it feels like almost some some fundamentally in that aspect some fundamentally different thing tha..."

"was part of the meaning of life and now and now I still think it's a pretty stupid idea but you do not need life to be finite to be meaningful it just has to be life what role does Love play in The Hu..."

"expect this to not happen that I do not think this is what happens by default that I do not think that this is the future we are on track to get is why would go down fighting rather than you know just..."

"I care and part of that is uh part of that is the love that connects all of us it's one of the things that I care about and the flourishing of the collective intelligence of the human species you know..."