the scope for like what we could achieve is like really extraordinarily large like maybe kind of larger than most people kind of typically entertain them having this kind of property of being like anti-fragile with respect to being wrong like really celebrating
and endorsing changing your mind in a kind of loud and public way but you can also do a thing which is try to make a lot of money and just you know make a useful product and then use the success of that first thing to then just think squarely like how do i just do the most
good if you have like 10 million in the bank and you make another 10 million does your life get twice as good obviously not right if on the other hand you just care about like making the world go well then the world's an extremely big place and so you basically don't run into these diminishing
returns like at all there's this question of like if the many worlds view is true what if anything could that mean with respect to questions about like what should we do or what's important today i have the pleasure of interviewing finn morehouse who is a research scholar at the oxford
university's future of humanity institute and he's also an assistant to toby ort and also the host of the hear this idea podcast um finn i know you've got a ton of other projects under a belt so do you want to talk about all the different things you're working on and how you got into ea
and this kind of research i think you nailed the broad strokes there i think yeah i've kind of failed to specialize in a particular thing and so i found myself just dabbling in projects that seem interesting to me trying to help get some projects off the ground and just doing research on
you know things which seem maybe underrated i probably won't bore you with the list of things and then yeah how do i get into ea actually also a fairly boring story unfortunately i really loved philosophy i really loved kind of pestering people by asking them all these questions you
know why are you not why are you still eating meat red's kind of peter singer and will mccaskill and i realized i just wasn't actually living things like these things out myself i think there are some just like force of consistency that pushed me into really getting involved i think the second
piece was just the people um i was lucky enough to have this student group where i went to university and i think there's some dynamic of realizing that this isn't just a kind of free-floating set of ideas but there's also just like a community of people i like really get on with and have
all these like incredibly interesting kind of personalities and interests so um those two things i think yeah and then so what was the process like uh i know a lot of people who are vaguely interested in ea but not a lot of them then uh very quickly transition to you know working on
research with top va researchers so uh yeah just walk me through how you ended up where you are yeah i think i got lucky with the timing of the pandemic which is not something i suppose many people can say i did my degree i was quite unsure about what i wanted to do there was some option of taking some kind of close to default path of maybe something like you know
consulting or whatever and then i was kind of i guess forced into this natural break where i had time to step back and i you know i guess i was lucky enough that i could afford to kind of spend a few months just like figuring out what i wanted to do with my life and that space was enough to
like maybe start like reading more about these ideas also to try kind of teaching myself skills i hadn't really tried yet so try to you know learn to code for a lot of this time and so on and then i just thought well i might as well wing it there are some things i can apply to
i don't really rate my chances but the cost to applying to these things is so low it seems worth it and then um yeah i guess i got very lucky and here i am awesome okay so let's talk about one of these things you're working on which is that you're you've set up and are going to be
helping judging these prizes about uh ea writing one is you're giving out uh five prizes for a hundred thousand dollars each for um blogs that discuss effective ultras related ideas another is uh five prizes of twenty thousand dollars each to criticize the ideas so talk more about these
prizes why is why uh why is it now an important time to be uh talking about and criticizing ea that is a good question um i wanna say i reluctant to frame this as like me personally like okay yeah i certainly kind of helped help set up these um initiatives well so i heard i i heard on the
inside that actually you've been uh you've been pulling a lot of the weight on these projects certainly yeah i've um find myself uh with the time to kind of like get these things over the line which i'm yeah i'm pretty happy with so yeah the criticism thing let's start with that
i want to say something like in general being receptive to criticism is just like obviously really important and if as a movement you want to succeed where succeed means not just like achieve things in the world but also like end up having just close to correct beliefs as you can
get and having this kind of property of being like anti-fragile with respect to being wrong like really celebrating and endorsing changing your mind in a kind of loud and public way that seems really important and so i know this is just like a kind of prima facie obvious case of wanting
to incentivize criticism but you might also ask like why now there's a few things going on there one is i think the effective altruism movement overall has reached its place where it's actually beginning to do like a lot of really incredible things there's a lot of like funders now kind
of excited to find find kind of fairly ambitious scalable projects and so it seems like if there's a kind of an inflection point you want to get the criticism out the door and you want to respond to it like earlier rather than later because you want to set the path in the right direction rather than
a just course which is more expensive later on well mccaskill made this point uh a few months ago you can also point to this dynamic and some other social movements where the kind of really exciting beliefs are kind of have this like period of plasticity in the early days they kind of ossify
and you end up with this like set of beliefs it's kind of like trendy um or socially rewarded to hold in some sense you feel like you need to hold certain beliefs in order to kind of get credit from you know certain people and the costs to like publicly questioning some
practices or beliefs become too high and that is just like a failure mode and it seems like one of the more salient failure modes for a movement like this so it just seems really important to like be quite quite proactive about celebrating this dynamic where you notice you're doing something
wrong and then you change track and then maybe that means shutting something down right you set our project the project seems really exciting you get some like feedback back from the world feedback looks more negative than you expected and so you stopped doing the project
and in some important sense that is like a success like you did the correct thing right it's important to celebrate that so i think these are some of the things that go through my through my head just like framing criticism in like this kind of positive way yeah it seems pretty important
right right i mean analogously it said that uh losses are as important as profit in terms of motivating uh economic incentives and it seems very similar here in a slack we were talking and you mentioned that uh maybe one of the reasons this is born now is if if a prize of
twenty thousand dollars can help somebody help us figure out how to or not me i don't have the money but like help svf figure out how to uh how to better allocate like 10 million dollars that that's a steal it's really impressive that effective altruism is a movement that is willing
to fund criticism of itself i i don't know is there any other example of a movement in history where that's been so interested in criticizing itself and becoming anti-fragile in this way i guess one thing i want to say is like the proof is in the pudding here like it's one thing to kind of
make noises to the effect you're like interested in being criticized and i'm sure lots of movements make that another thing to like really follow through on them and you know is a fairly young movement so i guess tai will tell whether it really does that well i'm very hopeful i also
want to say that this like particular prize is like you know one kind of part of a much um a much bigger thing hopefully that's a great question i actually don't know if i have good answers but that's not to say that they're right now i'm sure there are like political liberalism
as a strand of thought like in political philosophy comes to mind as maybe an example one other random thing i want to point out or mention you mentioned prophets and just like doing the maths and what's the like ev of like investing in just red teaming an idea like shooting an idea
down i think thinking about the difference between the for-profit and non-profit space is quite an interesting analogy here um you have this very obvious feedback mechanism in for-profit land which is you have an idea no matter how excited you are about the idea you can very quickly learn
whether the world is uh as excited which is to say you can just fail and that's like a tight useful feedback loop to figure out what your whether what you're doing is um worth doing those feedback loops don't by default exist if you don't expect to get anything back when you're doing
these projects and so that's like a reason to i want to implement those things like artificially like one way you can do this is with like charity evaluators which in some sense impose a kind of market-like mechanism where like now you have an incentive to actually be achieving the thing
that you're like ostensibly setting out to achieve because there's this third actor that's more party that's kind of assessing whether you're you're getting it but i think that that framing i mean we can try saying say more about it but that's like a really useful framing i think to me anyway
um and uh one other reason this seems important to me is if you have a movement that's about like 10 years old um like this you know we have we have like uh strains of ideas that are thousands of years old that have significant improvements made to them uh that that were that were missing
before so just on that uh that alone it seems to me that the reason to expect some mistakes either at a sort of like theoretical level or in the applications that that does seem like i do have a strong prior that there are such mistakes that could be identified in a reasonable amount of time
yeah i guess one framing that i like as well is not just thinking about here's a set of claims we have we want to like figure out what's wrong but some really good criticism can look like look you just missed this distinction which is like a really important distinction to make or
you miss this like addition to this kind of naive like conceptual framework you're using and it's really important to make that addition a lot of people are like skeptical about progress in in kind of non-empirical fields so like philosophy for instance it's like
oh we've been thinking about these questions for thousands of years but we're still kind of unsure and i think that miss is like a really important kind of progress which is something you might call like conceptual engineering or something which is finding these like really useful distinctions
and then like building structures on top of them and so it's not like you're making claims which are necessarily true or false but there are other kinds of useful criticism which include just like getting your kind of models like more more useful speaking of just making progress on questions
like these one thing that's really surprising to me and maybe this is just like my ignorance of the philosophical history here it's super surprising to me that the movement like long-term ism at least in its modern form it took thousands of years of philosophy before somebody
had the idea that oh like the future could be really big therefore the future matters a lot um and so maybe you could say like oh you know there's been lots of movements in history that have emphasized i mean existential risk maybe wasn't a prominent thing to think about before
nuclear weapons but that have emphasized that civilizational collapse is a very prominent factor that uh might be very bad for many centuries so we should try to make sure society's stable or something but do you have some sense of you have a philosophy background do you have some sense
what is the philosophical background here and to the extent that these are relatively new ideas how did it take so long yeah that's like such a good question i think one name that comes to mind straight away is this historian called tom moynihan who so you wrote this book about
something like the history of how people think about existential risk and then more recently he's been doing work on the question you are which is like what took people so long to reach this like what now seems like a fairly natural thought i think part of what's going on here is it it's really hard or easy i should say
to underrate just how much i guess is somewhat related to what i mentioned in the last question just how much kind of conceptual apparatus we have going on there's like a bit like the water we swim in now and so it's hard to notice so one example that comes to mind is thinking about
probability as this thing we can talk formally about this is like a shockingly new uh thought also the idea that human history might end and furthermore that that might be within our control that is to decide or to prevent that happening prematurely these are all like really surprisingly
new thoughts i think it just like requires a lot of imagination and effort to put your self into the shoes of people living earlier on who just didn't have the kind of yeah like i said the kind of tools for thinking that make these ideas pop out much more naturally and of course as soon as
those tools are in place then the like conclusions fall out pretty quickly but it's not easy and i agree that i appreciate that actually wasn't a very good answer um just because it's such a hard question um yeah so you know what's interesting is that more recently um maybe i'm unaware of
the full context of the argument here but uh i think i've heard uh holden kurnowski write somewhere that he thinks that there's more value in thinking about the issues that ea has already identified rather than identifying some sort of unknown risk that for example what ai might have
been like 10 20 years ago yeah 11 i mean given this historical experience that you can have some very fundamental tools for thinking about the world missing and consequently miss some very important moral implications uh does that imply that we should expect the space that ai alignment
occupies in terms of our priorities should we expect something as big or bigger coming up or just generally tools of thinking like you know expected value thinking for example yeah that's a good question um i think one thing i want to say there is it seems pretty likely that
the most important like kind of useful concepts for finding important things are also going to be the lowest hanging and i don't know i think it's like very roughly correct that we did in fact like over the course of building out kind of conceptual frameworks we picked them
like the most important ideas first and now we're kind of like refining things and adding maybe somewhat more peripheral things um that's at least if that like trend is roughly gonna hold and that's a reason for um not expecting to find like some kind of earth-shattering new
concept from left field although i think that's like a very weak and vague argument to be honest um also i guess it depends on what you think your time span is like if your time span is the entire span of time that humans have been thinking about things then maybe you would
think that actually it's kind of strange that it took like 3000 years before maybe even longer uh i guess it depends on when you define the start point it took you know 3 000 people to realize hey we should think in terms of probabilities and in terms of expected impact so in that sense maybe
it's like i don't know it took like 3 000 years of thinking to get to this very basic uh very basic idea what seems to us like a very important and basic idea i feel like maybe i have i want to say two things if you imagined lining up like every person who ever lived just like in a row
and then you kind of like walks along that line saw how much progress people have made across the line so you're going across people rather than across time then i think like progress in how people think about stuff looks a lot more like linear and in fact started earlier
then maybe you might think by just looking at like progress over time and if it was faster early on then if you're kind of following the very long run trend then maybe you should expect like um not to find these kind of again totally left field ideas soon but i think the second thing which is maybe more important is like i also buy this idea
that in some sense um progress about thinking in thinking about what's like most important is really kind of boundless like david deutsch talks about this kind of thought a lot when you come up with new ideas that just generates new problems new questions actually
some more ideas um that's very well and good i think there's some sense in which you know one priority now could just be framed as giving us time to like make that progress and even if you thought that like we have this kind of boundless capacity to come up with
a bunch of new important ideas it's pretty obvious that that's like a prerequisite and therefore in some sense that's like a robust argument for thinking that like um trying not to kind of throw humanity off course and um preventing mitigating some of these catastrophic risks is always just
going to shake out as like a pretty important thing to do maybe one of the most important things yeah i i i think that's reasonable um but but then there's a question of like even if you think that the existential risk is the most important thing um to what extent have you discovered all the uh
again that like x-risk uh argument and by the way i thought earlier what you said about you know trying to extrapolate what we might know from the limits of physical laws um you know if that can kind of constrain what we think might be possible i i think that's an interesting idea but i i
wonder like partly like one argument is just like we don't even know how to define those physical constraints and like before you had the theory of computation it wouldn't even make sense to say like oh uh like this much matter can sustain like so much so much uh flops uh floating point
operations per second and then second is like yeah if you know that number it still doesn't tell you like what you could do with it i you know what i think um uh an interesting thing that ho karnowski talks about is uh he has this article called this can't go on where he makes the argument that
listen if you just have a compounding economic growth at some point you'll get to the point where um you know like you'll have many or many many many times earth's economy per atom in the affectable universe and so it's hard to see like how you could keep having economic growth beyond
that point but that itself seems like um i don't know if that's true then there has to be like a physical law that's like the maximum gdp per atom is this right like if there's no such constant then you can like you should be able to surpass it i i guess it still leaves a lot to be
even desiring you could know such a number you don't know like how interesting or what what kinds of things could be done at that point yeah i guess first one is you know even if you think that like preventing these kind of very large scale risks that might like curtail human potential even if
you think that's just incredibly important you might miss some of those risks because you're you're just unable to articulate them or really like conceptualize them i feel like i just want to say at some point uh we have a pretty good understanding of kind of roughly what like what
looks most important like for instance if you kind of i know get stranded on a camping trip and you're like we need to just survive long enough to make it out and it's like okay what do we look out for i don't really know what the wildlife is here because i haven't been here before but probably
it's going to look a bit like this i can at least imagine you know the risk of dying of thirst even though i've never died at first before and then it's like what what if we haven't cons like even begun to think of like the other i was like yeah maybe but it's kind of there's just some like you
know table something practical reason for uh focusing on the things which are most salient and like definitely spending some time thinking about things we haven't thought of yet but um it's not like that list is just like completely endless and there's a kind of i guess a reason
for that and then you say the second thing which i don't actually know if i have like a ton of interesting things to say about although maybe you could try like kind of zooming in on what what you're interested in there i i i kind of think of it i don't think the second thing has a big
implications for this argument but the two um uh yeah we have like 20 other topics that are just as interesting but yeah but just as a i don't know as a closing note the the analogy is very interesting
to me that camping trevor you're trying to like do what needs to be done to survive i don't know okay so text an analogy it might be like i don't know somebody like the laser discovers oh that berry that we're all about to eat because we feel like that's the only way to get sustenance here while
we're you know just almost starving um don't eat that beer because that berries poisonous um and then then maybe somebody could point out okay so given the fact that we've discovered one poisonous food in this environment should we expect there to be other poisonous foods uh
that we don't know about uh i i don't know i don't know if there's anything more to say on that topic i mean one thing well like one i guess kind of angle you can put on this is you ask this question like we have precedent for a lot of things like we know now that uh igniting nuclear weapons does not
ignite the atmosphere which was a worry that some people had um so we at least have some kind of bounce on how bad certain things can be and so if you ask this question like what is worth worrying about most in terms of what kinds of risks might um reach this level of potentially posing an
existential risk well it's going to be the kinds of things we haven't done yet that we haven't like got some experience with and so you can ask this question like what is what things are there in this space of like kind of big seeming but totally novel precedent-free changes or events
and it actually does seem like you can kind of try generating that that list and getting at answers this is why maybe or at least one reason why ai sticks out because it's like fulfills his criteria of being pretty potentially big and transformative and also the kind of thing we
don't have any experience with yet but again it's not as if that list is like in some sense endless like there are only so many things we can do in the space of uh decades right okay yeah so moving on to another topic we're talking about the for-profit entrepreneurship as a a as a
potentially impactful thing you can do sorry maybe not in this conversation but like we separately we had uh one point yeah yeah um yeah so okay so to clarify this is not just for-profit in order to uh do earning to give so you become a billionaire and you give your wealth away to what extent can you
identify opportunities where you can just build a profitable company that solves uh an important problem area or you know makes people's uh lives better um one example of this is wave it's a company for example that uh helps with uh you know transferring money and banking services in africa
um probably has boosted people's uh uh well-being in all kinds of different ways um so to what extent can we expect just a bunch of for-profit opportunities for making people's lives better yeah that's a great question and there is really a sense in which some of the more like innovative
big for-profit companies just are like doing an incredibly useful thing for the world they're like providing a service that wouldn't otherwise exist and people are obviously using it because they are a successful pro for-profit company yeah so i guess the question is something like
you know you're stepping back you're asking how can i like have a ton of impact with what i do the question is are we like underwriting just starting a company so i feel like i want to throw a bunch of kind of disconnected observations we'll see if they like tied
together there is a reason why you might in general expect a non-profit route to do well and this is like obviously very naive and simple but where there is a for-profit opportunity you should just expect people to kind of take it like this is why we don't see 20 bills lying on the sidewalk
but the natural incentives for uh in some sense taking opportunities to like help people whether it isn't um a profit opportunity they're gonna be weaker and so if you're thinking about the like difference you make compared to whether you do something or whether you don't do it
in general you might expect that to be bigger where you're doing something non-profit and like in particular this is where there isn't a market for a good thing so it might be because the things you're helping like aren't humans maybe because they like live in the future so they can't
um pay for something it could also be because maybe you want to or get a really impactful technology off the ground in those cases you get a kind of free rider dynamic i think where there's less reason to like well you can't protect the ip and patent something but this reason to be the first mover and so this is like maybe it's not for profit but
starting a or helping kind of get a technology off the ground which could eventually be a space for a bunch of for-profit companies to make a lot of money that seems really exciting also creating markets where there aren't markets seems really exciting so for instance setting up
like amc's advanced market commitments um or prices or just giving yeah creating incentives where there aren't any so you get the like efficiency and competition kind of gains that you get from the for-profit space that seems great but that's not really answering your question because
the question is like what about actual for-profit companies i don't know what i have to say here like in terms of whether they're being underrated um yeah actually i'm just curious what you think okay so i i i think i have like four different reactions to um i've been remembering the number
four just in case i'm at three and i'm like i think another thing to say okay so um yeah so i i i had a draft uh an essay about this i didn't end up publishing but that led to a lot of interesting discussions between uh us so that that's why uh we might have uh i i don't know in case the audience
feels like they're interrupting a conversation that was already uh preceded the the beginning here uh so one is that um to what extent should we expect this market to be efficient so one thing you can think is listen the amount of potential startup ideas are so vast and the amount
of great founders is so small that you can have a situation where the the most profitable ideas are yeah that it's right that like somebody like elon musk will come up and like pluck up like all the maybe like the 100 billion ideas but if you have like a company like uh wave i'm sure they're doing
really well but uh you know if it's not obvious how it becomes the next google or something and i guess more importantly if it requires a lot of context for example you talked about like um neglected groups um i guess this doesn't solve for animals and um future people but if
you have somebody something in global health where you're like a neglected group is for example people living in africa right um the people who could be building companies don't necessarily have experience with the problems that these neglected groups have so if you have
it's very likely that or i guess it's possible that you could come upon an idea if you were specifically looking at how to help for example you know people suffering from poverty in the world in the the poorest parts of the world you could like identify a problem that just
like people who are programmers in silicon valley just wouldn't know about okay so a bunch of other ideas regarding uh the other things you said one is okay maybe maybe a lot of progress depends on fundamental new technologies and companies coming at the point where the technology is already
available and somebody needs to really implement and put all these ideas together yeah two things on that one is uh good like we don't need to go in rabbit hole on this one is the argument that actually the the invention itself not the invention the innovation itself is a very
important aspect and potentially a bottleneck neck aspect of this of getting an invention off the ground and scaled another is if you can build a 100 billion dollar company or a trillion dollar company or maybe not even just like a billion dollar company you have the resources to actually
invest in r d i mean think of a company like google right like how many billions of dollars have they basically poured down the drain on like hair brain schemes um uh you you can have like reactions to deep minds with regards to ai alignment but i mean just like uh other kinds of
research things that you've done seem to be like really interesting and uh really useful um and yeah all the other fan companies have like a program like this like microsoft research or i don't know what amazon's thing is and then another thing you can point out is with regards to setting
up a market that would make other kinds of ideas possible um and other kinds of businesses possible in some sense you could maybe make the argument that that's that maybe some of the biggest companies that's exactly what they've done right if you think of like uber
um it's not a market for companies or maybe maybe amazon is a much better example here where um you know like you theoretically who had an incentive before like a pandemic happens i'm gonna manufacture a lot of masks right but amazon provides uh makes the market so much more liquid
so that you can you can just start manufacturing masks and now immediately put them up on amazon so it seems in these ways actually maybe starting a company is a really uh is an effective way to deal with those kinds of problems yeah man we've gone so async here i should have just like
said one thing and then i'm sorry for throwing lots of things um as far as i can remember those are all great points yeah i think my like high level thought is i'm not sure how much we disagree but i guess one thing i want to say is again thinking about like in general what should you expect
the real biggest opportunities to typically be for like just having a kind of impact you know one thing you might think of is if you can optimize for two things separately that is optimize for the first thing and then use that to optimize for the second thing first is
trying to optimize for some like combination of the two at the same time you might expect to do better if you do the first thing so for instance you can do a thing which looks a bit like trying to do good in the world and also like make a lot of money um like social enterprise and often that goes very well but you can also do a thing which is try to
make a lot of money and just just you know make a useful product that is not directly aimed at you know improving humanity's prospects or anything but it's just kind of just great and then use the success of that first thing to then just think squarely like how do i just do the most
good um without worrying about whether there's some kind of profit mechanism i think often that strategy is gonna pan out well there's a thought about the kind of the tales coming apart if you've had this thought that at the extremes of like either kind of scalability in terms of
opportunity to make a lot of profit and at the extreme of doing like a huge amount of goods you might expect it's better to be like not such a strong correlation again one reason like in particular that you might think that is because you might think that like future really matters
like humanity's future and um sorry to be like a stock record but like because there's not really like a natural market there because these people don't haven't been born yet that is like a rambly way of saying that okay that's not always going to be true but i basically just agree that yeah
i would want to resist a framing of doing good which just leaves out like also doing some studying some successful for-profit company like there are just a ton of really excellent examples of where that's just been a huge success and yeah should be celebrated um so yeah i think
i disagree with the spirit um maybe we disagree somewhat on the like how much we should kind of relatively emphasize these different things but um doesn't seem like a kind of very deep disagreement yeah yeah maybe i've been spending too much time with brian kaplan or something
um so by the way the tail is coming apart i think is a very interesting um very interesting way to think about this uh scarlet xander's good article on this and like one thing he points out is like yeah generally you expect different parts of different types
of strength to correlate but the guy who has the strongest grip strength in the world is probably not the guy who has the biggest squad in the world right yeah okay so i think that's an interesting place to leave that idea oh yeah another thing i wanted to talk to you about was
back testing ea so if you have these basic ideas of we want to look at problems that are important neglected and tractable and applied them throughout history so like a thousand years back uh 2000 years back 100 years back is there a context in which applying these ideas um
would maybe lead to a perverse outcome an unexpected outcome um are there examples where i mean there's many examples in history where things you could have like easily made things much better but maybe it made it much better than even conventional morality or like present-day
uh ideas would have made them so react to the first part of the question which as i understand it is something like can we think about what some kind of effective altruism like movement or if these ideas were in the water like significantly earlier whether they might
have misfired sometimes or maybe they might have succeeded in that in fact how do we think about that at all i guess one thing i want to say is that very often the correct decision exante is a decision which might do really well in like some possible outcomes but you might still expect
to fail right the kind of mainline outcome is this doesn't really pan out but it's a it's a moonshot and if it goes well it goes really well this is i guess similar to certain kinds of investing where if that's the case then you should expect even if you follow the exact like correct strategy you
should expect to look back on the decisions you make uh made rather and uh see a bunch of failures um where failure is you know you just have very little impact and i think it's important not to to kind of resist the temptation to like really kind of negatively update on whether that was
the correct strategy because it didn't pan out and so i don't know if something like ea type thinking was in the water and was like thought through very well yep i think it would go wrong a bunch of times and that shouldn't be kind of terrible news when i say go wrong i mean like
not pan out rather than do harm if you did harm okay that's like a different thing um i think one thing that points to by the way is like you could take it you could choose to take a strategy which looks something like mini max regrets right so you have a bunch of options you can ask about the kind
of roughly worst case outcome um or just kind of like you know default air outcome on each option and one strategy is just like choose the option with the least bad kind of meh case and if you take this strategy you should expect to look back on the decisions you make and like
not see as many failures so that's one point in favor of it another strategy is just like do the best thing in expectation like if i made these decisions constantly what in the long run just ends up like making the world best and this looks a lot like just taking the highest ev
option maybe you don't want to like uh run the risk of causing harm so you know that's okay to include and you know i happen to think that like that kind of second strategy is very often going to be a lot better and it's really important not to be misguided by this kind of feature of the
mini max regret strategy where you look back and kind of feel a bit better about yourself in many cases if that makes sense yeah that's super interesting i mean if you think about uh back testing in terms of the the stock like you know models for the stock market one thing that
to analogize this one thing that uh tends to happen is that a strategy of just like trying to maximize returns from a given trade that results very quickly and you going bankrupt because like sooner or later there will be a trade where you lose all your money and so then there's something
called the cali criterion where you reserve a big portion of your money and you only bet with a certain part of it which sounds more similar to the minimized regret thing here um unless your expected value includes a possibility that i mean in this context like you know like losing all
your money is like an existential risk right so uh yeah maybe you like bake into the cake in the definition of expected value the odds of like losing all your money um yeah yeah yeah that's a great that's a really great point like i guess in some cases you want
to take something which looks a bit more like the the kelly bets um but if you add to your margins like relatively small margins compared to the kind of pot of resources you have then i think it often makes sense to take just do the best thing but not worry too much about what's the kind of like
size of the the kelly ben but um yeah that's a it's a great point and like i guess a naive version of doing this is just kind of losing your bankroll very quickly because you've like take it to your enormous bets and forgotten that um they might not pan out yeah so i appreciate that
what did you mean by add to the margins so if you think that there's a kind of a pool of resources from which you're drawing which is something like maybe philanthropic funding for the kind of work that you're interested in doing and you're only a relatively marginal actor um then that's unlike
being like an individual investor where you're more sensitive to the risk of just running out of money and um when you're more like an individual investor then you want to like pay attention to what the size of the kelly bet is if you're acting at margins then maybe that is less of like a big
consideration although it is obviously still a you know very important point well and then uh by the way i don't know if you saw my recent blog post about why i think there will be more billionaires yes okay yeah yeah uh i don't know what your reaction to any of the ideas there is but like
my claim is that we should expect the total funds dedicated to va to grow quite quite a lot um yeah i think i really liked it by the way i thought it was great one thing it made me think of is that there's quite an important difference between trying to maximize returns for yourself and then
trying to get the most returns just like for the world which is to say just doing the most good where one consideration we've just talked about which is a risk of just like losing your bankroll which is where like kelly betting becomes relevant um another consideration is that
as an individual just like trying to do the best for yourself you have like pretty steeply diminishing returns from money or just like how well your life goes with that extra money right so like if you have like 10 million in the bank and you make another 10 million does your
life get twice as good obviously not right and as such you should be kind of risk-averse when you're thinking about the possibility of like making a load of money if on the other hand you just care about like making the world go well um then the world's an extremely big place and so
you basically don't run into these diminishing returns like at all and for that reason like if you're making money at least in part to in some sense give it away or otherwise just like have a positive effect in some impartial sense then you're gonna be less risk averse which means
maybe um you fail more often but it also means that people who succeed like succeed really hard um yeah so i don't know if that's in some sense i'm just recycling what you said but um i think it's like a really kind of neat observation well and another interesting thing is that not only
is that true but then you're also you're also in a movement where everybody else's has a similar idea and not only is that true but also the movement is full of people who are young techy smart and as you said dress neutral so basically people who are going to be way over represented
in the in the ranks of future billionaires and they're all hanging out and they have this idea that you know we can become rich together and then make the world better by doing so um you would expect that this would be exactly the kind of situation that would lead to people
teaming up and starting billion dollar companies all right uh yeah so a bunch of other topics and effective authorism that i wanted to ask you about so one is should it impact our decisions in any ways if the many world's interpretation of quantum mechanics is true i know the argument that
oh you can just think of you can just translate amplitudes to probabilities and if it's just probabilities then decision theory doesn't change my problem with this is i i've gotten like very lucky in the last few months now i think it like changes my perception of that if i realize
actually most me's and okay i i know there's like problems with saying me's at to what extent they're fungible most branches of the of the multiverse uh like i i'm like significantly worse often that that makes it worse than oh i just got lucky um but like now i'm here and another
thing is if you think of existential risk um and do you think that even if like essential risk is very likely in some branch of the multiverse humanity survives i don't know that seems better in the end than oh the probability was really low but like it just resolved to we didn't survive
does that make sense okay all right there's there's a lot there i guess rather than doing a terrible job at trying to explain what this money was thing is about maybe it's worth just kind of pulling people towards you know just googling it i should also add this enormous caveat that i
don't really know what i'm talking about this is just kind of an outsider who's taking this kind of i know this just this stuff seems interesting yeah okay so there's this question of like what if the many worlds view is true what if anything could that mean uh with respect to questions about
like what should we do or what's important and one thing i want to say is just like without zooming into anything it just seems like a huge deal like every second every day i mean in some sense like just kind of dissolving into this like cloud of me's and like just kind
of unimaginably large number of of me's and that each of those me's is kind of in some sense of dissolving into more clouds um this is just like wild also seems somewhat likely um to be true as like far as i can tell okay so like what does this mean you yeah you point out that you can
talk about having a measure over worlds in some sense you can there's actually a problem of how you get like probabilities or how you make sense of probabilities on the many worlds view and there's a kind of neat way of doing that which like makes use of questions about how you should
make decisions that is you should just kind of weigh future use according to in some sense how likely they are but it's really the reverse you're like explaining what it means for them to be more likely in terms of how it's rational to weigh them um and then i think it's like a ton
of very vague things i can try saying so maybe i'll just try doing like a brain dump of things you might think that like many worlds being true could push you towards being more risk neutral in certain cases if you weren't before um because in some cases you're like translating from some
chance of this thing happening or it doesn't into some fraction of you know worlds this thing does happen and another fraction it doesn't that's kind of like i don't think it's worth reading too much into that because i think a lot of the like important uncertainties about the world
are still like subject to uncertainties about how most worlds will in fact turn out but it's kind of interesting and notable that you kind of like convert between overall uncertainty about how things turn out to like more certainty about the fraction of the ways things turn down
i think another like interesting feature of this is that so the question of like how you should act is no longer a question of like how should you kind of benefit this person who is you in the future who's one person it's more like how do you benefit this like cloud of people who are all
success of you that's just kind of like diffusing into the future and i think you point out that you could just like basically salvage a lot of basically all decisions even if that's true but the like picture of what's going on changes and in particular i think just intuitively like
it feels to me like the gap between acting in a self-interested way and then like acting in an impartial way where you're like helping other people it kind of closes a little in a in a way like just you're already benefiting many people by doing the thing that's kind of rational to benefit
you um which isn't so far from benefiting people who aren't like continuous with you in this special way so i kind of like that as a thing huh um it's so interesting yeah and then okay there is also this like slightly more out there thought which is here's the thing you
could say many worlds is true then there is at least a sense in which there are very very many more people in the future uh compared to the past like just unimaginably many more and even like the next second from now there are many more people so you might think that should like make us have a really steep negative discount rate
on the future which is to say we should like value future times much more than present times and like in a way which would just kind of it wouldn't like modify how we should act it just like explodes how we should think about this this definitely doesn't seem right maybe one
way to think about this is that if this thought was true or like looks kind of directionally true then that might also be a reason for being extremely surprised that we're both speaking at like an earlier time rather than a later time because if you think you're just like randomly
drawn from all the people who ever lived it's like absolutely mind-blowing that we get drawn from like today rather than tomorrow yeah given that it was like 10 to the something many more people tomorrow um so it's probably wrong and wrong for reasons i don't have a very good handle on because
i just like don't know what i'm talking about i mean i can kind of try parroting the reasons but like it's something i'm you know i'm interested in trying to really crock those reasons a bit more that that's really interesting um i i i i didn't think about that argument uh for uh this election
argument i think one resolution i've heard about this is that you can think of the proportion of you know hilbert space or like the proportion of all the the the like the universe's wave function that could be the like the probability rather than each different branch uh you know what i just
realized that the selection argument you can you made maybe that's an argument against bostrom's um boston's idea of we're living in a simulation because basically his argument is that there will be many more simulations than there are real copies of you therefore you're probably in
a simulation the like the thing about like saying that all the simulations plus you are your prior should be equally distributed among them seems similar to saying your prior of being distributed along each possible like branch of uh the wave function should be your prior across to
them should be the same whereas i think um in the context of the wave function you were arguing that maybe it should be like you shouldn't think about it that way you should think about like maybe a proportion of the total oh wait uh turtle hobart space um yeah does that make sense i i don't know
if i said wait say it again how how it links into simulation type stuff instead of thinking about each possible simulation as an individual thing a crotch which that is equally as likely each individual instance of a simulation is equally as likely as you're living in the real world
maybe simulation as a whole is equally likely to be living in the real world just as you being alive today rather than tomorrow is equally likely despite the fact that there will be many more branches and new branches of the wave function tomorrow yeah okay there's a lot i get a lot going
on i feel like there are people who like actually know what they're talking about here just tearing their hair out like give me this obvious thing um so you mentioned that you're having a podcast um you but by the way if you are one such person please do uh like email me or dm me or something
very interesting um so yeah you mentioned like it is obviously there is a measure over over worlds and this like lets you talk about things being sensible again also maybe like one minor thing to comment on is talking about probabilities is kind of hard because ever in many
worlds just everything happens that can happen and so it's like difficult to get the language exactly right but anyway so i totally get the point and then the question of how it maps on to simulation type thoughts here's a i don't know like maybe a thought which kind of connects to this um do
you know like sleeping beauty type problems um no no okay it's certainly a vaguely remembered example um but let's let's try it so in you like original sleeping beauty problem you go to sleep okay and then i flip a coin um or you know whoever someone flips a coin
if it comes up tails they wake you up once if it comes up heads they wake you up once and then they uh you go back to sleep and you know your memory is wiped and then you're working up again as if you're being woken up uh in the other world and um okay so you go to sleep
you wake up and you ask what is the chance that the coin came up heads up tails and it feels like they're kind of really intuitive reasons for both 50 and one-third here's a related question which is maybe a bit simpler at least in my head i flip a coin if it comes up heads
i like just make a world with one observer in it and if it comes up tails i make a world with a hundred observers in it maybe it could be like running simulations with 100 people you like wake up in one of these worlds you don't know how many other people are there in the
world you just know that like someone has flipped a coin and decided to make a world with either one or 100 people in it what is the chance that you're in the world with 100 people and like there's a reason for thinking it's half and there's a reason for thinking
that it's like i know 100 over 101. does that make sense so i i understand the logic behind the half what is the reason for thinking i mean regardless of where you ended up as the observer it seems like if the odds of like the coin coming up oh i guess is it because you'd expect there to be more
observers in the other universe like wait yeah so what is the logic we're thinking it might be uh 100 over one well you might think of it like this how should i reason about where i am well maybe it's something like this i'm just a random observer right of all the possible observers that could have come out of this and there are 101 possible observers
and you can just imagine that i've been like randomly drawn okay and if i'm randomly drawn from all the possible observers then it's overwhelmingly likely that i'm in the big world huh that's super interesting um i should say actually i i should i should plug someone who
does know what they're talking about on this which is joe carl smith who has like a series of like really excellent uh blog posts oh um he's coming on the podcast next week yes amazing okay he's like really gonna talk about it i don't wanna like okay i don't wanna scoop him but
one thought that comes from him which is just like really cool maybe just to kind of round this off is if you're like a 100 over 101 on examples like this and you think there's any chance that like the universe is infinite in size then you should think that the chance you're in a universe that is
infinite an extent is just like one or close to one uh if that makes sense um i see yeah yeah okay so in the end then does your awareness in many worlds is like a good explanation has that has that impacted your view of what should be done in any way yeah
so i don't really know if i have a good answer my best guess is that things just shake out to kind of where they started as long as you started off in this kind of like relatively risk neutral place i suspect that if many worlds is true this might have like this might make it much harder to
hold on to kind of intuitive views about personal identity for the reason that like there isn't this like one person who you're like continuous with throughout time and know other people it's how people tend to think about what it is to like be a person and then there's this kind of like vague
thing which is just occasionally i'd you know just like remember like every other month or so that maybe many walls is true and it's kind of like blows my mind i don't know what to do about it and i just like go on with my day that's about where i am okay um all right but other interesting
topics to uh talk about talent search what is the what is ea doing about identifying let's say more people like you basically right but maybe even like people like you who are not in like places where they're not next to oxford i don't know where you actually are from originally but um
like i if they're from like some like i don't know like china or india or something um well what what is ea doing to recruit more finns from uh from places where they might not otherwise work on ea yeah it's a great question and yeah to be clear i just won the lottery on things going right to kind
of um be lucky enough to do what i'm doing now so yeah in some sense the question is how do you like print more winning lottery tickets and then did like find those people who really deserve them but like it was currently not being identified i love this concert i just i read that book talent
um tyler cowan and daniel grace recently and yeah there's something really powerful about this fact that this like business of you know finding really like smart driven people and connecting them with opportunities to like do the things they really want to do this is like really kind of still
inefficient and there's just still like so many just people out there who like aren't kind of getting those opportunities i actually don't know if i have much more like kind of insight to add that other than this is just a big deal and it's like there's a sense of which it is
an important consideration for this like project of trying to do the most good like you really want to find people who can like put these ideas in practice and i think there's a special premium on that kind of person now given that there's like a lot of philanthropic kind of funding
ready to like be deployed there's also a sense in which this is just like in some sense like a cause in its own right it's kind of analogous to open borders in that sense at least in my mind i hadn't really like appreciated it in on some kind of visual level before i write that book and
then another thing you talked about in the book is you want to get them when they're young you can really shape somebody's ideas about what's worth doing if you and then also their ambition about what they can do if you catch them early um and you know but tyler khan also had an interesting
blog post a while back where he pointed out that a lot of people applying to his emerging ventures uh program a lot of young people flying um are heavily influenced by effective altruism which seems very like it's going to be a very important factor in uh in the in the long term i
mean eventually these people will be in positions of power yeah so maybe effective authorism is already succeeding to the extent that a lot of the most uh ambitious people in the world are uh identify that way or at least i mean given the selection effect that the telecollege program has
but yeah so um what is it that can be done to get people when they're young um yeah i mean it's a very good question and i think like what you point out there is is right there's some nick whitaker has this blog post just something like the it's called
the lamplight model of talent curation um he like draws this distinction between casting um like a very wide net that's just kind of very legibly prestigious and then you know filtering through thousands of of applications or in some sense like pulling out the bat signal that
in the first instance just like attracts the like really promising people and maybe actually drives away people who will be better fit for something else um so like an example is if you were to hypothetically write a quite quite a wonky economics blog like every day for
however many years and then run some furniture program you're just like automatically selecting for people who read that blog and that's like a pretty good kind of stunning population to begin with so i really like that that kind of thought of just like not needing to be like incredibly loud
and like prestigious sounding but rather just like being quite honest about what this thing is about so you just attract the people who who like really sorted out because that's just quite a good feature i think another thing that again this is like not a very interesting point to make but
something i've really realized the value of is like having physical um hubs and so there's this model of you know running like fellowships for instance where you just like find really promising people and then there's just so much to be said for like putting those people
in the same place and you know surrounding them with maybe people who are a bit more like senior and just kind of like letting this natural process happen where people just get really excited that there is this like community of people working on stuff that previously you just been kind of
reading about in your bedroom on like some blogs that like as a source of motivation i know it's like less tangible than other things but yeah just like so so powerful and like probably i know one of the reasons i'm like working here maybe yeah um it is one aspect of uh
working from home that you don't get that um regarding the first point so i think uh um well maybe maybe that should update in favor of not doing community outreach and community building like maybe that's negative marginal utility because like if i think about for example um my local so there was an effective altruism group at my college that didn't attend
um and there's also like an effective altruism group for the city as a whole um in austin that i don't attend um and the reason is just because i don't know the people who there is some sort of um adverse election here where the people who are leading organizations like this
are people who couldn't just like aren't directly doing the things that effective autism says they might be might consider doing um and are more interested in the social aspects of altruism um so i i don't know i i i'd be much less impressed with the movement if my first introduction to it was
the specific groups that like i've had the personal i've personally interacted with rather than i don't know just like hearing will mccaskill on a podcast um this is by the way the fourth ladder being my first introduction to effective altruism yeah interesting um i feel
like i really don't want to like underwrite the job that community builders are doing i think in fact it's turned out to have been like and still it's just like incredibly valuable especially just looking at the numbers of like what you can achieve as like a group organizer at
your university like maybe you could just change the course of like more than one person's career over the course of like a year of your time that's like pretty incredible but yeah i guess part of what's going on is that the difference between like going to your like local group or
like engaging with stuff online is that you get to kind of choose the stuff you engage with and like maybe one upshot here is that the like kind of set of ideas that might get associated with um ea it's like very big and you don't need to buy into all of it or just like be passionate about
all of it like if this kind of ai stuff just like really seems interesting but maybe other stuff is just like more peripheral then you know one yeah like this could push towards wanting to have like just a specific group for people who are just like you know this aisle seems cool other stuff not my
like cup of tea um so yeah i mean in the future is like things get scaled up as well as kind of scaling out i think also maybe having this like differentiation and kind of diversification of like different groups i mean it seems pretty good but just like more of everything also seems good
yeah yeah i mean i i'm i'm probably overfitting on my own experience and given the fact that i don't didn't uh didn't actively interact with any other communities i'm uh probably not even informed on those experiences of loves um but there was an interesting post on an effective autism forum
that somebody sent me where they were making the case that um at their college as well they got the sense that uh the ea community building stuff had the negative impact because people were kind of turned off by their peers and also uh there's a difference between like i don't
know somebody like saying baker great or woman haskell uh advising you and obviously virtually um to um uh to do these kinds of things versus like i don't know some some sophomore at your university starting philosophy right um there's no effect um yeah um i think my guess is that
like on that these these efforts are still just like overwhelmingly positive but um yeah i think it's like pretty interesting that people have the experience you describe as well yeah and interesting to think about ways to kind of like get around that so long reflection is a it seems
like a bad idea no i'm so glad you asked um yeah i want to say i want to say no i think in some sense i've like come around to it as an idea but yeah okay maybe it's worth like oh really interesting maybe it's worth i guess like trying to explain what's going on um
so if you like were to zoom out like really far over time and consider our place now like in history and you could like ask this question about suppose in some sense humanity just became like perfectly coordinated what's the plan like what what kind of in general should we be prioritizing and like in what stages
and um you might say something like this it looks like this moment in history which is to say maybe this century or so just looks kind of wildly and like unsustainably dangerous like or kind of so many things are happening at once it's really hard to know how things get a pan out
but it's like possible to imagine things panning out really badly and badly enough to just like more or less end history okay so before we can like worry about some kind of longer term considerations let's just get our act together and make sure we don't mess things up
so okay like that seems like a pretty good first priority but then okay suppose that you succeed in that and like we're in a significantly safer kind of time uh what then you might notice that the scope for like what we could achieve is like really extraordinarily large like maybe kind of larger than most people kind of like typically entertain
like we could just do a ton of really exceptional things but also there's this kind of feature that maybe in the future not and not especially long-term future we might more or less for the first time be able to embark on these like really kind of ambitious projects that are in some
important sense uh like really hard to reverse and that might make you think okay at some point it'd be great to like some like you know achieve our that potential that we have and just like like for instance the kind of lower band on this is lifting everyone out of poverty who remains in poverty and
then like going even further just making everyone even wealthier able to do more things that they want to do making more scientific discoveries whatever so we want to do that but maybe something should come in between these two things which is like figuring out what is actually good um
and okay why should we think this i think one thought here is it's very plausible i guess it's kind of links to what we're talking about earlier that the way we think about you know like really positive features like one of the best features it's just like really kind of incomplete
almost certainly we're just getting a bunch of things wrong by this kind of pessimistic induction on the past like a bunch of smart people thought really reprehensible things like 100 years ago so we're getting things wrong and then it's like second thought is i don't know seems possible to
actually make progress here in thinking about what's good there's this kind of interesting point that most like work and i guess you might call it like moral philosophy has focused on the negatives so you know avoiding doing things wrong fixing harms avoiding bad outcomes but this
idea of like studying the positive or studying like what we should do if we can kind of do like many different things this is just like super super early and so we should expect to be able to make a ton of progress and so okay again imagine that the world is like perfectly coordinated
would it be a good idea to like spend some time maybe a long period of time kind of deliberately holding back from embarking on these like huge irreversible projects which maybe involve like leaving earth in kind of certain you know scenarios or otherwise just like doing things
which are hard to undo should we spend some time thinking before then yeah sounds good and then i guess the very obvious response is okay that's a pretty huge assumption um that we can just like coordinate around that and i think the answer is yep it is but as a kind of directional ideal
should we push towards or away from the idea of like taking our time holding our horses kind of getting people together who haven't really like been part of this like conversation and like hearing them yeah definitely seems worthwhile okay so i i have another good abstract idea that i want
to entertain by you so you know it seems like kind of wasteful that we have these different companies that are building the same exact product uh but you know because they're really the same exact product they don't have economies of scale and they don't have coordination there's just a
whole bunch of uh loss that comes from that right wouldn't it be better if we could just coordinate and just like figure out the best person to produce something together and then just have them produce it and then we could also coordinate to figure out like what
what what is the right quantity and quality of for for them to produce i'm not trying to say this is this is like communism or something i i'm just saying it's ignoring what would be required like in this analogy you're ignoring like what kind of information gets lost and um what kinds of uh
what what it requires to do that so-called coordination um in the communism example um in this example it seems like that you're not uh whatever would be required to prevent somebody from realizing some like let's say somebody has a vision for like we want to colonize a star
system we want to like i don't know make some new technology right that's part of something that the long reflection would curtail maybe i'm getting this wrong but it seems like it would require almost a global panopticon uh totalitarian uh state to be able to like prevent people from
escaping the reflection okay so there's a continuum here and i basically agree that some kind of panopticon like thing not only is impossible but actually sounds pretty bad but something where you're just like pushing in the direction of being more coordinated on the international level about things that matter seems like desirable and possible
um and in particular like preventing really bad things rather than like trying to get people to like all do the same thing um so the biological weapons convention strikes me as an example which is like imperfect and underfunded but you know nonetheless kind of directionally good
and maybe an extra point here is that there's like a sense in which the long reflection option or i guess the better framing is like aiming for a bit more reflection rather than less that's like the conservative option that's like doing what we've already been doing um just a bit
longer rather than some like radical option so i i agree it's like pretty hard to imagine like you know some kind of super long period where everyone's like perfectly agreed on on doing this um but yeah i think framing it as like a directional ideal seems pretty worthwhile and i
guess i know maybe i'm kind of naively hopeful about the possibility of coordinating better around things like that uh there's two reasons why this seems like a bad idea to me one is okay first of all who is going to be deciding when we've come to a good consensus about uh okay so
we've decided like this is the way things should go um and now we're like ready to escape the long reflection and realize our vision for the rest of the lifespan of the universe who is going to be doing that it's the people who are presumably in charge of the long reflection almost by definition
it'll be the people who have an incentive in uh preserving whatever power uh well power balances exist at the end of the long reflection and then the second thing you'd ask is like um there's like a difference between i think uh having a consensus on not using biological weapons
or something like that where you're limiting a negative versus it seems like when we've had uh when we've required society-wide consensus on what we should aim towards achieving um the outcome has not been good in history uh it seems better that on the positive end to just leave it open-ended
and then just maybe um when necessary say that like the the very bad things we might want to restrict together yeah yeah okay i think i kind of just agree with a lot of what you said so i think the best like framing of this is the version when you're
preventing something which most people can agree is negative which is to say some actor unilaterally deciding to like do this huge irreversible or set out on this huge irreversible project like something you said was that the outcome is going to reflect the
like values of whoever is like in charge and then not just his values i mean also i mean just like think about how guilds work right it's like um if the whenever we for example an industry we let how the industry should progress we let those kinds of decisions be made collectively by the people
who are currently dominant in the industry um uh you know gills or something like that um or um or like industrial conspiracies uh as well it seems like the outcome is um uh outcome is just uh bad like uh and so like my prior would be that at the end of such a situation our ideas about what we
should do would actually be worse than uh going into the long reflection i i mean obviously uh the uh it really depends on how it's implemented right so i'm not saying that but i just just like broadly given all possible implementations and maybe the most likely implementation
given how government's right now yeah yeah yeah i should say that like i am in fact like pretty absurd i just kind of i know it's more more enjoyable to like give this thing it's hearing no no i i enjoy the uh the parts where we have the discretes yeah so one
thought here is if you're worried about the future like the course of the future being determined by some single actor i mean that that worry is just symmetrical with the worry of letting whoever wins some race first go and do you know go and do the thing the like project where they more or
less kind of determine what happens to the rest of humanity so the option where you like kind of deliberately wait and let people like have some uh like global conversation i know it seems like that is less worrying even if the worry is still there i should also say i can imagine the outcome
is not unanimity in fact it'd be like pretty wild if it was right but you want the outcome to be some kind of like stable friendly disagreement where now we can kind of like maybe reach some kind of cosine solution and we just like go and do our own things there's like a bunch of projects
which kind of go off at once um i don't know that feels like really great to me compared to whoever gets there first determining how things turn out but yeah i mean it's hard to talk about stuff right because it's like somewhat speculative but um i think it's like a useful
like north star or something to try pointing towards um okay so maybe to make it more concrete i i wonder if you're uh if your expectation that the the consensus view uh would be better than the first move review uh let's in like today's world maybe okay either we either we have the form of government uh and not just government but yet also the the i mean the
industrial and logistical organization that i don't know like elon musk has designed for mars either that is so if he's the first mover for mars would you prefer that or we have the un uh come to a consensus between all the different countries uh about like how
we should have the first mars colony organized or should would the marsh colony run better if after like 10 20 years of that the they're the ones who decide how the first mars colony goes global consensus views to be better than first member reviews yeah that's a good question and
i mean one obvious point is not always right like there are certainly cases where the consensus view is just like somewhat worse um i think you limit the downside with the consensus view right because you give people the space to express why they just don't think this like one idea is is bad
i don't know if it's an answer your question but like it's a really good one they're like you can imagine that kind of the un-led thing is going to be like way slower it's going to probably be way more expensive the international space station is a good example where um i know i think that turned
out pretty pretty well but a like private version of that would have happened like in sometimes a lot more effectively i guess i'm not like the elon example is is kind of a good one because it's not obvious why that's like super worrying the thing i have in mind in like long reflection
example is maybe like a bit more kind of wild um but it's really hard to make it concrete so i'm yeah somewhat floundering there's there's also another reason to like um to the extent that somebody has the resources i i don't know maybe this like just gets too irreconcilable um
question about your priors about other kinds of political things but um to the extent that somebody has been able to build up resources uh privately to be able to be a first mover in a way that is going to matter for the long term what do you think about what kind of views they're likely
to have and what kind of competencies they're likely to have versus assuming that the way governments work and function and the quality of their government governance doesn't change that much for the next hundred years what kind of outcomes you'll have from basically if you if you
think like the likelihood of leaders like donald trump or joe biden is like going to be similar for like the next hundred years and if you think like the richest people in the world are the first movers are going to need people that are similar to elon musk i i can see people having like
genuinely different uh reasonable views about who should like uh the elon musk have a hundred years from now or the joe biden over 100 years from now have the power to decide the long run course of humanity is that uh fulcrum and this debate that you think is important or is that maybe
is that not as relevant as i might think yeah i guess i'll try saying some things and maybe we'll like respond to that kind of two things are going through my head so one is something like you should expect these questions about like what should we do when we have the capacity to do like
a far larger range of things that we currently have the capacity to do that question is going to hinge like much more importantly on like theories people have and like world views and very kind of particular details much more than it does now and i'm going to do a bad job at trying to articulate
this but there's some kind of analogy here where if you're like fitting a curve to some points you can like overfit it in fact you can overfit it in various ways and they all look pretty similar but then if you like extend the axes so you like see what happens to the curves like beyond
the points those different ways of fitting it can you like go all over the place and so like there's some analogy here when you kind of expand the pos like the space of what we could possibly do um different views which look kind of
similar right now or at least come to similar conclusions they just like go all over the shop and so that is not responding to your point but i think it's like really worth saying like this is a reason for expecting reflecting on what the right view is to be quite important and like and then
i guess at least into a second thought which is something like i guess there's two things going on one is the thing you mentioned which is there are basically just a bunch of political dynamics where you can just like reason about where you should expect values to heads for like political reasons
in some sense is like now better than the defaults and what is that default and then there's a like a kind of different way of thinking about things which is like severally from political dynamics can we actually make progress in like thinking better about what's best to do in the
same way that we can like make progress in science like kind of separately from the fact that like people's views about science are influenced by like political dynamics and maybe like a disagreement here is a disagreement about like how much scope there is to just get better at thinking about these things i mean one
like reason i can give i guess i kind of mentioned this earlier is this project here of like thinking about what's best to do maybe kind of thinking better about ethics it's not the thing it's like maybe more relevant to think this is like on the order of kind of 30
years old rather than on the order of 2000 years old you might call it like secular ethics perfect writes about this right he's like talks about this kind of there are at least reasons for hope um we haven't ruled out that we can not make a lot of progress because the thing we were doing before
like we were trying to think systematically about what's best to do was just very unlike the thing that we should be interested in i'm sorry that was like a huge ramble but hopefully there's something
there yeah i i want to go back to what you were saying earlier about how um you can you think you can think of uh i i don't know global consensus as the reduced variance version of future views and you know so i i think that like to the extent that you think a downside is really bad um i think
that's a that's a good argument um and then yeah i mean it's like similar to my argument against like uh monarchist which is that like actually i think it is reasonable to expect that if you could like reasonable uh you could uh reliably have people like lee kuan yew who are in charge of your
country and you have a monarchy that things might be better than a democracy it's just that the the bad outcome is just so bad that it's like better just having like a low variance uh uh thing like democracy is if i want to talk about maybe one last kind of trailing thought on what you said
is um i think i guess pogba has this thought and also david deutsch like did a really good job at kind of explaining it about one like underrated value of democracy is not just in some sense having this function to like combine people's views into like some kind of you know optimal path
which is like some mishmash what everyone thinks it's also like having the ability for um people who are being governed to just like cancel this current experiment in governance and try again so it's something you know it's like we'll give you freedom to you know implement this kind of governance plan that seems really exciting and then we're just
gonna like pull the brakes when it goes wrong and that kind of that option to like start again in general just feels like really important as some kind of tool you want in your like tall kit when you're thinking about these like pretty big futures i guess my hesitation about
this is i i can't imagine as like any form of government where at the end of it i would expect that a consensus view from i mean not just like uh nerdy communities like ea but like an actual global consensus would be something that i think is a good path maybe maybe it's something like i
don't think it's like the worst possible path but i mean one thing about reducing variance is like if you think the far future can be really really good then by reducing variance you're like cutting off a lot of expected value right um and then you you can think like democracy works
much better in cases where the problem is like closer to something that the people can experience it's like i don't know if the democracies don't have famines because it's like if there's a famine you get voted out right or like you have major wars as well right but um if you're talking
about like demar some form of uh consensus way of deciding uh what should the far far future look like it's not clear to me why uh the consensus view on that would be it's likely to be correct yeah yeah yeah i think maybe some of what's going on here is i'd want to resist the
and it's my fault for i think like suggesting this framing that is like you just you spend a bunch of time thinking and like having this conversation and then you take this like international vote on what we should do um i think maybe another framing is something like let's just
give the time for the people who like want to be involved in this to like make the progress that could be possible on thinking about these things and then just like see where we end up where i know there's like a very weak analogy to progress in like other fields where we don't make
progress in like uh mathematics or science by like taking enormous votes on what's true um but we can by just like giving people who are interested in making progress there's space and time to do that and then at the end it's like often pretty obvious what turns out that's like very
begging the question because it's like way more obvious um what's right and wrong if you're like doing maths compared to doing this kind of thing but um no but also like what happens if this seems similar like the question about uh monarchy where it's like what happens if
they pick the wrong person or they were like the wrong paula bureau to pick of what the what the what the charter you take to the rest of the universes yeah it seems like a hard problem to ensure that you have uh the group of people who will be deciding this
either if it's a consensus or if it's a single person or anything in between like it has to be some decision maker right um i think you can just imagine there being no decision maker right so like the thing could be let's agree to have some time to reflect on what is best and we might
come to someone else and then at the end like you know one version of this is just let things happen like there's no final decision it's just like that time between doing the thing and thinking about it just like extending that time for a bit seems good i see okay okay uh yes sorry i missed that uh
okay so actually one of the major things we're gonna discuss this is like one all the things we discussed so far was one like quadrant of like of the conversation actually you know before we talk about space governance let's talk about uh podcasting so you have your own podcast
um i i have my own well what have you uh why did you started like what have you what have you been your like experiences so far what have you learned about the the joy and impact of podcasting so story is uh luca who's close friend of mine who i do this podcast with we're both at university
together and we're like both podcast nerds and i think i remember we were in our last year and we had this conversation like we're like surrounded by all these people who just seem like incredibly interesting like all these you know like academics we really love to talk about um i'll talk to and
if we just like email them saying we're doing a podcast and wanted to interview them that could be a pretty good excuse to talk to them so let's see how easy it is to do this turns out the startup costs on doing a podcast are like pretty low if you want to do like a scrappy version of it right
did that it turns out that like academics especially but just like tons of people really love being asked to talk about the things they think about all that all day right it's like a complete win-win where you're like you're trying to boost the ideas of someone or some
actual person um who you think deserves more airtime that person gets to like talk about their work and you know spread their ideas so it's like huh there's like no downsides to doing this other than the time also i should say that the kind of yes rate on our emails was like
considerably higher than we thought we were you know like two random undergrads with microphones um but this is a really nice like kind of snowball effect where if someone who is like well known is like gracious enough to say yes despite knowing not really knowing knowing what you're about
and then you do an interview and then like it's pretty good interview when you're emailing the next person you don't have to like sell yourself you can just be like hey i spoke to this other impressive person um and of course you get this like this kind of snowball so
no it's definitely a fantasy yeah it's great is that the best kind of ponzi scheme though podcasts as like a form of media are just like incredibly special um there's something about just the incentives between like guest and host just like aligned so much better than like i know
if this was like some journalistic interview it'd be like way kind of uncomfortable there's something about the fact that it's still kind of hard to like search transcripts so there's less of a worry about like forming all your words in the right way so it's like more relaxed
yeah recommended yeah i know it's it's and it's um it's such a natural form of you can think of writing as a sort of way of imitating conversation and audiobooks is a way of trying to imitate like a thing that's trying to imitate conversation that you're like audiobooks
seem like they're supposed to because like you get the writing is like yeah you're visually perceiving uh what was originally uh um an ability you had originally for understanding uh you know um uh uh you know audible ideas um but then audiobooks it's like you're going
through two layers of translation there where you don't have the natural repetition and the ability to gauge the other person's reaction uh and so on that um and the back and forth obviously that uh an actual conversation has and um yeah so that that's why it's like people um potentially listen
to like podcasts too much where i don't know that they're just like they have something in the areas the whole day which you can't imagine for like audiobooks right yeah few things makes me think of one is there's some experiment where i guess you can just do it yourself when if you
force people not to use disfluencies discloses sorry like um's and ours those people just get like much worse at uh reading words in some sense like disfluencies like help us because i'm using the word like right now communicate thought for some reason and then if you take it yeah
i guess i can speak for myself and then you word for word to transcribe what you're saying when i say you i mean me um it's like hot garbage it's like i've just learned how to talk yes um but that that pattern of speech like you point out is in fact easier to to digest or at least it requires less kind of stamina or effort no yeah and then the
same type of interesting point about this um in um antifragile i'm basically remembering this but he makes a point that um sometimes when a signal is distorted in some way it makes it you retain or absorb more of it because you have to go through extra effort to
understand it which is which is a reason for example i think his example was if um if i i don't know if somebody's like speaking but that they're like far away or something so their audio is muted you gotta like apply more concentration which makes it um which makes it actually which means
you more retain more of their content so if you like overlay or someone says with a bit of noise or you turn on the volume very often people have like better comprehension of it because because the thing you just said which is like you're paying more attention also i think maybe i was
misremembering the thing i mentioned earlier or maybe it's a different thing which is you can like take perfect speech like recordings and then you can like insert um and like make it worse and then you can do like a comprehension test where people listen to the different versions
and they do better with the versions which are like less perfect um is it just about having more space between uh words like or is it actually the um like if you just added space instead of um would that have the same effect or is that there's something
specific about it and there's a limit to how that can stretch like something like global ecology i'm in some global continent that of just like it's like or something like evokes evokes like absolute concentration um i'm curious to ask you like i know i want to know what you feel like you've learned from
doing podcasting so i don't know what maybe one question here is like yeah what's some kind of underappreciated difficulty of uh trying to ask good questions i mean you're like obviously are currently asking excellent questions so what have you learned that um one thing you um i think i've heard this advice that you want to do something where
a thing that seems um easier to you is difficult for other people like i have tried okay so one obvious thing you can do is like ask on twitter i'm interviewing this person what should i ask them and you'll observe that all the and like all the questions
that people will like propose are like terrible um and um so but maybe it's just like oh yeah there's there's adverse selection the people who actually could come up with good questions are not gonna spend the time to like reply to your tweet um but then i've even like um
hopefully they're not listening but i've i've even like tried to like hire like i don't know research partners or research assistants who can help me come up with questions more recently and the questions they come up with also seem um like it just left like uh yo how
did grover come in the midwest like change your views about blah blah blah it just like i have i i i it's just a question that's uh whose answer is not interesting it's not a question you would organically have if you at least i hope you would now organically i want to ask them if you were
only talking to them one-on-one so um it does seem like the skill is um harder than i would have uh it's rarer than i would have expected i don't know why i don't know if you have a good sense of because you you have an excellent the podcast were asked questions i i i don't know
what do you think have you observed this where the the the asking good questions is a rare skill than you might think certainly i've observed that it's really hard skill i still feel like kind of i still felt like it's really difficult i also at least like to think that we've got a
bit better first thing i thought there was this example you gave of like what was it like growing up in the midwest we always ask those kinds of questions so you know like how did you get into behavioral economics and why do you think it was so important these are
just like guaranteed to be kind of uninspiring answers so specificity seems like a really good or like kind of what what what is your book about yeah exactly exactly um yeah tell us about yourself um this is what i love conversations with tyler it's one of the many reasons i love him
he'll just like launch with you know like the first question will be like about some footnote in this person's like undergrad dissertation and that just sets the tone so well um also i think cutting off which i've made very difficult for you i guess cutting off answers once the interesting thing has been said and the elaboration or like the um
caveats on the like meat of the answer are often just like way less worth hearing i think i'm trying to ask questions which a person has no hope of knowing the answer to even though it'd be great if they knew the answer to like so what should we do about this policy is a pretty bad
move also um if you speak to people who are like familiar with asking questions about like their book for instance in some sense you need to like flush out the kind of pre-prepared like spiel that they have in their heads um like i know you could even just do this like before the interview right
and then like it gets to the good stuff where they're actually being made to think about things rob woodland has a really good um like list of interview tips which i think i know i guess every reason this is kind of nice to talk about other than the fact this is like good to
have some kind of like inside baseball talk is that you know like skills of interviewing feel pretty transferable to just asking people good questions which is like a generally useful skill um hopefully um so yeah i guess i've found that it's like really difficult i still get pretty
frustrated with how hard it is but um it's like a cool thing to realize that you are able to like kind of slowly learn yeah so okay so what what is how do you think about the value of your writing through your podcast and then what what advice do you have for somebody who might want to start
their own yeah so i know kind of one reason you might think podcasts are really useful in general is i guess the way i think about this is like you can imagine there's a kind of just stock of like ideas that seem really important like if you just have a conversation with i don't know
someone who's like researching some cool topic and they tell you all this cool stuff this like isn't isn't written up anyway like oh my god these are like kind of exist in the world i think in many cases this like stock of important ideas just grows faster than if you're able to like
in some sense pay it down and like put out into the world and that's just a bad thing so this is overhang you want to fix and then you can ask this question of okay what's the just like the most one of the most effective ways to like communicate ideas um relatively well put them out into the
world well i know just like having a conversation with that person is just like one of the most kind of efficient ways of doing it i think it's like interesting in general to consider like the kind of rate of information transfer for different kinds of like media and stuff like transmitting and receiving ideas so like on the like best
end of the spectrum right i'm sure you've had kind of conversations where you everyone you're talking with like shares a lot of context and so you can just kind of blur out this like slightly incoherent three minute like i just had this kind of thought in the shower
and they can fill in the gaps and basically just like get the idea and then at the kind of opposite end like maybe you want to write an article like a kind of prestigious outlet and so you're like kind of covering all your bases and making it like really well written
and then just like the information per kind of effort it's just like so much lower and i guess like academic certain kinds of academic papers are like way out on the the other side so yeah just like as a way of solving this kind of problem of there's this overhang of important
ideas podcasts just seemed like a really kind of good way to to do that i guess when you don't successfully put ideas out into the world you get these little kind of like clusters or like fogs of uh like contextual knowledge where everyone knows these ideas in the right circles but they're hard
to pick up from like legible sources and and it's like kind of maps onto this idea of like context being that thing which is scarce i remember like tyler cohen talking about that i kind of like eventually made sense in that context i will mention that um um it seems like kind of uh
uh the the thing you mentioned about either just head off on a podcast and explain your idea or you know take the time to uh do it in like a prestigious place it seems very much like a volvo strategy um whereas the middle ground spending like four or five hours writing a blog post where it's not going to be in
something that plays super prestigious you might as well just like either just put it up in a podcast if it's something you just want to get over with or you know and spend some time a little bit more time getting into more prestigious plays the argument against it i guess is that
the idea seems more accessible if it's in the form of a blog post for i don't know for posterity if you just uh want that to be like the canonical source uh for something but again if you wanted to be the connected source you should just make it a uh sort of like more official thing because
if it's just a youtube clip then it's it's a it's a little difficult for people to like reference to it you can kind of get the best of both worlds so you can put your recording into like there are you know the software that transcribed your podcast right you put it into that if you're
lucky enough to have someone to help you with this you can get someone or you could do it yourself like go through the podca the transcript to make sure it's kind of there aren't any like glaring mistakes and now you have this like artifact that is in text form that like lives on the internet
and it's just like way cheaper than writing it in the first place but yeah that's a great point and also people should read your uh barbells for life is that it bible strategies for life yeah yeah that's it yeah cool maybe one last thing that seems worth saying on this topic of
podcasting is like it's quite easy to start doing a podcast and um my guess is it's often worth at least trying right so i know i guess there are probably a few people listening to this who've like kind of entertained the idea uh one thing to say is it doesn't need to be the case
that if you just like stop doing it it doesn't really pan out after like five episodes or even fewer that it's a failure like you can frame it as i wanted to make like a small series it's just like a useful artifact to have in the world which is like i don't know here's this kind of bit of
history that things underrated and i'm going to tell the story in like four different hour long episodes if you like set out to do that then you have this like self-contained chunk of work uh see maybe that's like a useful framing and there's a bunch of resources which i'm sure
it might be possible to link to on just like how to set up a podcast i like tried writing like collecting some of those resources the thing to emphasize i think is that you i i think i've talked to like at least i don't know three or four people at this point who have told me like
oh i have this idea for a podcast um it's going to be about you know like architecture it's going to be about like vr or whatever they seem like good ideas i'm not thinking about the ideas themselves i but i just like i talked to them like six months later and it's like they haven't started
it yet and i i just tell them like literally just email somebody right now whoever you want me to be your first guest i mean i cold emailed uh brian kaplan he ended up being my first guest um yeah just email them and like set something on the calendar because i don't know what
it is maybe just about life in general i don't know if it's specific to podcasting but the amount of people i've talked to who have like vague plans of starting a podcast and have nothing scheduled uh or like no immediate like they i don't know what they're they're
expecting like some mp3 file to appear on their hard drive on some fine day um so yeah but yeah just do it like get it on the calendar now yeah that seems that seems good also there's like some way of thinking about this where you could just like if you just write off in advance that your
first i know let's say seven episodes are just gonna be like embarrassing to listen to um that is more freeing because it probably is the case um but you like need to go through the like the bad episodes before you start getting good at anything i guess it's like not even a podcast point um
yeah also there's if you're just like brief and polite there's like very little cost in being ambitious with the people you reach out to yeah um so yeah just go for it right there brian that's interesting uh he wrote an interesting argument about this somewhere where he was pointing out
that actually the cost of cool d milling are much lower if you're like an unknown quantity than if you are like somebody who was like has somewhat of a reputation because if you if you're just nobody then they're gonna forget you ever cold email them right they're just gonna ignore it in their inbox
if you ever run into them in the future they're just like not gonna register i've registered you the first time if you're like somebody who has like somewhat of a reputation then there's like a mystery of like why are we not getting introduced to somebody who should know both of us right if
you claim to be i don't know like a professor who wants to start a podcast um um yeah but anyways just reinforcing the point that the cost is really low all right cool okay uh let's talk about space space governance so this is an area where you've uh you've been writing about and researching
uh recently okay one concern uh uh you might have is you know that toby horde has that book a book about the uh the precipice about how we're in this time of peril um where we have like one in six odds of going extinct this century is there some reason to think that once we get to space this
will no longer be a problem or the the risk of extinction for humanity go you know asymptotes to zero i think one point here so actually maybe it's worth beginning with a kind of like naive case for thinking that like spreading through space is just like the ultimate hedge against extinction
um and this is you know you can imagine just like duplicating civilization or at least having kind of civilizational backup like things which are like in different places in space if the risk of any one of them like being hit by an asteroid or like otherwise
encountering some existential catastrophe if those risks are independent then you get this like it's exponent it's like a power law with every new um backup right it's like it's like having multiple kind of backups of some some data in different places in the world right so if those risks are independent then it is in fact the case that like going
to space is just like incredibly good strategy i think they're pretty compelling reasons to think that a lot of the most worrying risks are like really not independent at all um so one example is you can imagine very dangerous pathogens if there's any trouble between these
places then the pathogens can travel um but like maybe the more pertinent example is if you think it's worth being worried about artificial general intelligence that is like unaligned that goes wrong and like really relentlessly pursues really terrible goals then just having some like
some just physical space between two different places is really not going to work as a real kind of uh hedge so i'd say something like you know space seems kind of it's even that useful to like diversify go to different places but like absolutely not sufficient for like getting through
this kind of time of perils then yeah i guess there's kind of this follow-up question which is like okay well why expect that there is any hope of like getting the the risk down to sustainable levels if you're sympathetic to the possibility of like really just transformative artificial
general intelligence like arriving you might think that in some sense getting that transition right where the outcome is that now you have this thing on your side which like has your interests in mind or has like good values in mind but has this like general purpose kind of reasoning
capability that in some sense this just like tilts you towards being safe just like indefinitely long and one reason is if bad things pop up like some unaligned thing then you have this much better established safe and aligned thing which has this kind of defensive advantage
so that's one consideration and then if you're like less sympathetic to this ai story i think you'd also just tell a story um about like being optimistic for our kind of capacity to like catch up along some kind of wisdom
or coordination dimension if you like really zoom out and look at how like quickly we just invented all this like kind of insane technology that is like a roughly kind of exponential process you might think that that might kind of eventually like slow down but our like improvements and just
how well we're able to coordinate ourselves like continues to increase and so that it's like defensive advantage in the long run those are two pretty weak arguments so i think it's like actually just a very good question to think about and i i like you know also kind of acknowledge
it's like not a very kind of compelling uh answer i'm wondering if there are aspects that you can discern from first principles about the safety of space um which suggests that either i don't know either there's no good reason to think the time apparels ever ends um i mean because the thing
about ai is like that's true whether you go to space or not right like um if it's aligned then i guess it can indefinitely reduce existential risk i mean one thought you can have is maybe i i don't know contra the long reflection the thing we're talking about which is that if you think that
one of the bottlenecks to a great future could be um i don't know like some sort of tyrannical um a tyrannical is like a kind of a coded term in terms of like conventional political thought but you know what i mean then the diversity of political models that being spread out would have uh maybe
that is a positive thing um on the other hand agueron has this interesting blog post about um about space wars where he points out that the logic of a mutually short destruction goes away in space so maybe we should expect more conflict because it's hard to identify who the culprit
is if like an asteroid was redirected to your planet and you know if they can sweep it dust officially fast they can like basically destroy your above ground civilization um yeah so i mean is there something we can discern from first principles about how violent and how i don't
know how how how pleasant the time and space will be um yeah it's a really good question i will say that i think i have not like reflected on that question enough to like give a really authoritative answer instantly one person who absolutely has is um anders samberg who
has been thinking about almost exactly these questions for a very long time and in some point in the future i might have a book about this so um watch that space one consideration is that you can start at the end you can consider what happens very far out in the future and it turns out that because the universe is expanding
for any just like point in space so if you consider the next like cluster over or maybe even the next galaxy over there'll be a time in the future where it's impossible to reach that other point in space no matter how long you have to to get there so even if you sent out
like a signal in the form of lights it would never reach there because there'll always be a time in the future where you start expanding faster than the speed of light relative to that other place so okay like there's a small consolation there which is if you last long enough to get to
this kind of era of isolation then suddenly you become independent again in the like strict sense i don't think that's especially relevant when we're considering kind of more i guess i guess relatively speaking narrow term things um gordon's point is really nice so gwen starts by pointing
out that we have this like logic with nuclear weapons on earth or mutually assault destruction where the emphasis is on a second strike so if i receive a first strike from someone else i can identify the someone else that first strike came from and i can kind of like credibly commit
to retaliating um and the thought is that this like disincentivizes that person from launching the first strike in the first place which makes a ton of sense uh gwen's point i guess the thing you already mentioned is in space there are reasons for thinking it's gonna be much harder to like
attribute where a strike came from that means that you don't have like any kind of credible way to threaten a retaliation and so mutually sure destruction doesn't work and that's kind of like actually a bit of an uncomfortable thought because the alternative to mutually issue
destruction in some sense is just first strike which is if you're worried about some other actor being powerful and after destroy you then you should destroy their capacity to destroy you um so yeah it's like a slightly bleak blog post i think there are like a ton of other considerations
but some of which are a bit more hopeful ones that you might imagine is in general a kind of like defensive advantage in space over offensive one reason is that space is uh this like dark canvas in 3d where there's absolutely nowhere to hide and so you can't sneak up on anyone
um yeah i think there's like a lot of there's a lot of stuff to say here and a lot of a little bit i don't quite fully understand yet but i guess that makes it uh that makes it an interesting and important place to be uh subject to be studying if we uh we don't
know that much about how it's going to turn out so um if von neumann has this vision that you would have you would have set up a sort of virus-like probe that infests a planet and uses of its usable resources to build more probes which go on and infect more planets is uh is the long-run
future of the universe that the all the available low-hanging resources are burnt up in uh you know some sort of like fire out you know expanding fire of von neumann probes because it seems like as long as one person decides that this is something they want to do then you know yeah the
the low-hanging fruit in terms of uh spreading out will just be burned up by somebody who built something like this yeah that's a really good question um so okay maybe there's like an analogy here where we have on earth we have organisms which can like convert raw resources plus sunlight
into more of them and they replicate it's notable that they don't like blanket the earth although i do just as a tendon i remember someone mentioned there's a thought of like an alien you know arrived in earth and asked a question what is the most successful species it'd probably be grass
but okay the the reason that like particular organisms that just reproduce using sunlight don't just kind of have this like green goo dynamic is because there are competing organisms there are things like you know antivirals and so on um so i guess like you mentioned it's not as if as soon
as this thing gets seeded it's game over you can imagine trying to catch up with these things and stop them and i know what's the equilibrium here where you have things that are trying to catch things and things which are also spreading it's like pretty unclear but it's not clear that it's
everything gets burned down although i know it seems like worth having on the table as a possible outcome and then another thought is i guess something you also like basically mentioned robin hansen has this paper called i think um burning the cosmic commons i think the things he
says are like a little bit subtle but i guess to kind of like bastardize the overall point there's an idea that you should expect kind of selection effects on what you observe in the long run of like which kinds of things have won out in this kind of like race for different parts of space
and in particular the things you should expect to win out are these things which like burn resources very fast and are like greedy in terms of grabbing um as much space as possible and i don't know that seems like roughly correct he also has a more
recent bit of work called grabby aliens i think there's a website grabby aliens.com which kind of expands on this point and asks this question about what you know we should expect to see such um kind of grabby civilizations um yeah i mean one maybe kind of one like slightly fanciful
upshot here is you don't want this like greedy von neumann type probes to win out that also just like dead they have no kind of nothing of value um and so if you think you have something of value to spread maybe that is a reason to spread um more quickly than you otherwise like would have
planned once you've like figured out what that thing is if that makes sense yeah so then does this militate um towards the logical space race where similar to the first strike where if you are not sure that you can retaliate you want to do it for strike maybe there's a logic to as long as
you have like at least some what of a compelling vision of what the far future should look like you should try to make sure it's you who's the first actor that goes out into space even if you don't have everything sorted out even if you have like concerns about how uh yeah yeah yeah like
you'd ideally like to spend more time my guess is that the time scales on which these dynamics are relevant are like extremely long time scales compared to what we're familiar with so i don't think that any of this like straightforwardly translates into you know
wanting to speed up on the order of decades and in fact if any like delay on the order of decades i know presumably also centuries gives you like a marginal improvement in your like long run speed then just because of the like again the time scales and this distances involved you almost
always want to take that trade-off so yeah i guess i'd want i'd be wary of like reading too much into all this stuff in terms of like what we should expect for some kind of race in the near term it just turns out the space is like extremely big and there's like a ton of stuff there so in in
anything like the midterm i think this reasoning about like oh we'll run out of useful resources probably won't kick in but that's like that's just me speculating so i yeah i don't know if i have i have a kind of clear answer to that okay so if we're talking about space governance is there any
reason to think okay in the far future we can expect that space will be colonized either by you know like a fully artificial um uh intelligence or by simulations of humans like m's in either case it's not clear that these entities would feel that constrained by whatever
norms of space governance we detail now what is the reason for thinking that you know any sort of charter or constitution that the un might build regardless of how i don't know how sane it is will be the basis of which like the actual long run the fate of space uh is decided upon yeah yeah
so i guess i know the first thing i want to say is that it does in fact feel like an extremely long shot to expect that any kind of norms you end up agreeing on now even if they're good flow through to the point where they really matter um if they ever do but okay so you can ask like what
are the worlds in which this this like early thinking does end up being uh being good on the end point i don't know like i can imagine for instance the us constitution surviving in importance at least to some extent if digital people come along for the right it's not obvious
why there's some discontinuity there i guess the important thing is considering what happens after anything like kind of transformative artificial intelligence arrives my guess is that the worlds in which this is like even kind of remotely this like you know super long term
what norm should we have for the settling space the worlds in which this matters or does anything worthwhile uh whilst in which you know alignment goes well right and it goes well in the sense that there's a significant sense in which humans are still in the driving seat and when they're
looking for precedence they just look to existing like institutions and norms so i know that seems kind of there's like so many variables here that this seems like a fairly narrow kind of set of worlds but i know seems seems pretty possible and then it's also kind of like you know settling the
moon or mars where that is just like much easier to imagine how this stuff actually kind of ends up influencing or positively influencing how things turn up feels worth pointing out that there are things that really plausibly matter when we're thinking about space that aren't just like
thinking about these crazy kind of very long run sci-fi um scenarios although they are like pretty fun to think about one is that there's just a ton of like pretty important infrastructure kind of currently orbiting the earth and also anti-satellite weapons are being built and my
impression is well in fact i think it's the case that there is kind of worryingly small amount of agreement and regulation about the use of those weapons um maybe that puts you in a kind of analogous position to not having many agreements over the use of nuclear weapons although maybe
less worrying in certain respects but still seems worth taking that seriously and thinking about how to make progress there yeah i think it's just like a ton of other kind of near-term considerations there's this great graph actually um on our data which i guess i can tell you the link to after
this um which shows the number of objects launched into orbit um especially low earth orbit just over time and it's like perfect hockey stick and i know it's like quite a nice illustration of why you might might kind of pay to like think about how to make sure this stuff goes well and the story
behind that graph is kind of fun as well i was like messing around on some un website which has it's just like this database incredible database which has more or less every kind of officially recorded launch logged to like all this data about like how many objects would contain whatever
it was like the clunkiest api you've ever seen um you have to like manually like click through each page it takes like five seconds to load and you have to like scrape it so i was like okay this is great that this exists i am not like remotely sophisticated enough to know how to like make
use of it but i emailed the hour and dated people saying fyi this exists if you happen to have like you know a ton of time to burn then then have at it and um ed's ed matto from iowan data got back to me like a month later like hey i had a free day all done and it's like up on the website
it's so cool so cool cool okay i think that's that's my space rambling do i get i i'd quite like to ask you a couple questions if that's all right i realize i've been kind of hogging the airwaves yeah so here's one thing i've just been interested to know you're doing this like
blogging and podcasting right now but yeah what's next like 2024 201k what is he doing i think um i'll probably be i've just had i don't know the idea of building a startup has been very compelling to me and not necessarily from i think it's the most impactful thing uh that
could possibly be done although i think it is very impactful but it just i don't know it just like if you have a people tend to have like different things that are like i want to be a doctor or something that it's uh you know it's something that's uh stuck in your head so yeah i think
that's probably what i'll um be attempting to do in 2024 i don't i i think the situation in which i like remain a blogger and podcaster is if it just turns out to be like i don't know if i have if the podcast just becomes like really huge right at that point it might make more sense that oh
like actually this is a way currently i think the impact the podcast has is like like like 0.000001 and then the 0.01 is just me getting to learn about a lot of different things so i think for for it to have any um not necessarily that it has to be thought of in terms of impact but in
terms of like how useful is it i think it's only the case if it like really becomes much bigger nice that sounds great maybe this is getting right to start a conversation what about what about a non-profit startup all the same like excitement if you have a great idea
you kind of skip the fundraising stage more freedom because you don't need to like make well i thought you started smiling right sure but like if it's a great idea then i'm sure there'll be like support to make it happen yeah if there's something where i don't see a way to like profitably do it and i think it's very important to be done
yeah i definitely wouldn't be opposed to it but is that by the way where you're leaning like i asked you in 2024 what is finn doing uh do you have a non-profit startup i don't have something concrete in mind that kind of thing feels very exciting to me to at least at least try out
gosh i got a show yeah yeah i think um i i i guess my prior is that there are profitable ways to do many things if you're more creative about it there are obvious counter examples of so many different things where yeah i could not tell you how you could make that profitable right like if
you have like something like one day sooner where they're trying to you know speed up challenge trials it's like how is that a startup it's not clear um so yeah i i think that there's like a big branch of the decision tree where i think that's the most compelling thing i could do
nice and maybe a connected question is i'm curious what you think ea in general is underrating from your point of view also maybe another question you can answer said is like what do you think like i am personally getting wrong or got wrong but maybe the kind of the more general question
is the more interesting one for most people so i think when you have statements um which are somewhat ephemeral or ambiguous like let's say there's like a some historian like uh twenbee right he wrote uh study of history and one of the things he says in it is like civilizations die
when um the elites lose confidence in the norms that they're setting and they're uh in their it loses the confidence to rule that okay so i i i don't think that's like actually an s x risk right i'm just like trying to use that as an example like something that comes up off the top of my
head it's the kind of thing um it like it could be true i don't know how i would think about it in uh sort of i mean it doesn't seem tractable i i don't know i even analyze whether it's true or not using uh using the modes of analyzing importance of topics that we've been using throughout
this conversation i don't know what that implies for you because it's not clear to me like maybe ea shouldn't be taking things that are vague and ambiguous like that seriously to begin with right yeah if there is some interesting way to think about statements like that from a perspective
that eas could appreciate including myself from a perspective that i could appreciate um i'd be really interested uh to see what that would be because there does seem to be a disconnect where when i talk to my friends who are intellectually um inclined um who have a lot of interesting ideas
but requiring a sort of like translation layer almost like a compiler uh uh or like a transpiler that you know uh converts uh you know code from this language into like assembly here um it does create a little bit of inefficiency and potentially uh a loss of topics that could
be talked about nice that feels like a great answer i i just say something i'm kind of worried about as well especially in leading towards a more kind of speculative long-termist and um seems really important to like keep hold of like some real truth-seeking attitudes where those
kind of like what the obvious feedback of where you're whether you're getting things right or wrong is is much harder and often you don't have the luxury of having that so yeah i think just like keeping that attitude in mind seems like very important i like that what is your answer by the
way what do you think that you should improve on yeah i guess off the top of my head maybe i have two answers which like go in exact opposite directions so one answer is that one kind of something that looks a bit like a failure mode that i'm a bit worried about is as or if the
movement grows significantly then the ideas that kind of originally motivated it there were like quite new and um exciting and important ideas somewhat kind of dilute maybe because it's like i guess it's related to what you said you kind of lose these attitudes of like taking weird ideas
seriously like scrutinizing one another quite a lot and it becomes a bit like i don't know green washing or something where like the language stays but the real kind of like fire behind it of like taking impact really seriously rather than just saying the right things that kind of fades away
so i don't know if i don't think i want to say he is currently underrating that in any important sense but it's like something that seems worth having as a you know kind of worry on the radar and then like the roughly opposite thing that seems also worth worrying about is i think it's
just really worth paying attention or like uh it's worth considering best case outcomes when a lot of this stuff maybe grows um quite considerably you know thinking about how this stuff is like could become mainstream i think thinking about really scalable projects as well as just like
little fun kind of interventions on on margins there's at least some chance that becomes like very important um and so as such you know one part of that is like learning to make a lot of these fields legible and attractive to um people who could contribute who are like learning about it for the first time um and just yeah in general
planning for the best case which could mean just like being thinking in very ambitious terms thinking about things going very well that just also seems worth doing so i think it's a very vague answer but maybe that's um yeah maybe not worth keeping them but that's my answer perhaps
uh you know opposite to what you were saying about ea not taking weird ideas too seriously in the future it's maybe they're taking uh weird ideas too seriously now it could be the case that just following basic common sense morality kind of like what tyler crown talks about it stubborn
attachments is really the most effective ways to deal with many threats even weird threats if you have if you have areas that are more speculative like bio risk or ai where it's not even clear that the things you're doing to address them are necessarily making them better i know
there's concern in the movement like the initial grant that they gave to open ai might have like sped up um ai doom um maybe the best case scenario in cases where there's a lot of ambiguity is to just do more common sense things like and maybe this is also applicable to things like
global health where malaria nets are great but the way that hundreds of millions of people have been left out of poverty is just through implementing capitalism right it's not through uh targeted interventions like that again i don't know what this implies for the movement in general like
even if like just implementing like the neoliberal agenda is the best way to like decrease poverty like what does that mean that somebody should do if they're yeah what does that mean you should do with a marginal million dollars right so it's not clear to me it's something i hope i'll know
more about in like five to ten years is i'd be very curious to talk to future me about like what does he think about comments that's morality or sticking beard idea seriously i think like one way of thinking about quote-unquote weird ideas is that in some sense they are the result of like
taking a bunch of common sense starting points and then just like really reflecting on them hard and seeing what comes out yeah so i think maybe the question is like how much trust should we place on those like reflective processes versus like how much what should i probably be on like
weird ideas being true because they're because they're weird is that like good or bad and then like separately i know one thing that just seems kind of obvious and important is if you take these ideas like first of all you should ask yourself whether you like actually believe them or whether
they are like kind of fun to like say or like you're just kind of saying that you believe them and then sometimes i know it's like fun to say with ideas but it's like okay i actually don't have good grounds to believe this and then second of all if you do in fact believe something
it's like really valuable to ask if you think this thing is really important and true um why aren't you working on it if you have the opportunity to work on it this is like the hamming question right what's the most important in your fields and then um what's stopping you from from
working on it and obviously look many people have the luxury of like dropping out everything and and working on the things that they in fact believe are really important but if you do have that that opportunity that's a question which i know is maybe just valuable to to ask
maybe this like this is a meta objection to ea which is that i'm aware of a lot of potential objections to ea like the ones we were just talking about but there's so many other ones where people will identify yeah yeah that's an interesting point and then like nobody knows what
to do about it right it's like oh you know should we take common sense morality more seriously like weird ideas more seriously it's like oh that is an interesting debate and then but how do you resolve that i i don't know how to resolve that i don't know if somebody's come up with
a good way to resolve that uh i guess it kind of hooks into the long reflection stuff a little bit because one answer here is just time so i think the story of people raising concerns about ai is maybe instructive here where you know early on you get some real kind of just like radical out there
researchers or writers who are kind of raising this is a worry there's a lot of kind of like weird baggage attached to the well they write and then maybe get like a first book or two and then you get like more kind of prestigious or established um people expressing concerns i think one way to accelerate that process
when it's like worth accelerating is just to ask that question right like do i in fact see like can i go along with this argument do i see a hole in it and then if the answer is no like if it just kind of checks out even if you're obviously gonna always gonna be uncertain but if
it's like yeah it seems seems kind of reasonable then um by default you might just like spend a few years being like just kind of living like oh yeah this thing that i guess i think is true but i'm not really acting on you can't just like skip that step and be like well now
i'm not sure i agree i think um maybe maybe an analogy here is like i don't know you're in a relationship and you think like oh i don't see what's wrong with this relationship so um instead of just waiting a few years to like try to find something wrong with it
might as well just tie the knot now and get married i i think it's something similar with um i think of failure mode if you maybe not in this because we're easy wouldn't see it in ea but we all you can see generally in the world is that people just come
to conclusions about how the world works or how the world ought to work too early in life where when they don't seem to know that much about what is optimal and what is possible so yeah maybe they should just wait a little longer maybe just like integrate like these weird radical ideas
as things that exist in the world and wait until you're like late 20s until you decide actually this is the thing i should do with the rest of my career or with my political uh rights or whatever yeah i think that's actually that's just a really good point i think maybe i'd want to
kind of walk back what i said based on on that but i think there's some version of it which i'd still really endorse which is maybe like you know spent like some time reflecting on this such that i don't expect further reflection is gonna like radically change what i think um you can maybe
talk about uh this being the case like a group of people rather than a particular person and i could just like really see this thing playing out where i just like believe it's important for a really long time without acting on it and that's the thing which seems worth sure skipping i mean to be
to be a little tiny bit more concrete like if you really think some of this these potentially catastrophic risks just like are real and you think there are things that we can do about it then sure seems good to start working on this stuff um and you really want to like avoid that
regret of you know some years down the line like i really could have just started work on that earlier there are occasions where this kind of thinking is is useful or at least kind of asking this question like what would i do i do right now if i just like did what my kind of idealized self
would endorse doing um maybe that's useful so it seems that if you're trying to pursue i don't know a career related to ea there's like two steps where the first step is you have to get a position like the one you have right now where you're you know learning a lot and figuring out future steps
and then the one after that is where you actually lead or take ownership of a specific project like a you know nonprofit startup or something do you have any advice for somebody who's before step one huh that's a really good question i also will just do the annoying thing of saying
definitely other things you can do other than that that kind of like two-step trajectory but yeah what as then go directly to step two or never go to step two and just like be a really excellent researcher or communicator and like anything else i think like where you have the luxury of doing it
not kind of rushing into the most salient like career option and then retroactively justifying why it was the correct option i think is like quite a nice thing to bear in mind i suppose often it's quite uncomfortable obviously doing something like consulting
yeah something like that yeah i mean the kind of the obvious advice here is that there is a website designed to answer this question uh which is 80 000 hours oh yeah so maybe some there's a particular bit of advice from adk which i found very useful which was after i left uni
i was like really unsure what i wanted to do i was choosing between a couple options and i was like oh my god this is like such a big decision because i guess in this context it's not only not only do you have to answer the question of what might be a good fit for me
what i might enjoy but also like sometimes what is like actually most important maybe and how am i supposed to answer that given that like there's a ton of disagreement and so i just like found myself like bashing my head against the wall of trying to get to a point where i certain that like one option was better than the other and um the piece
of advice that i found useful was that often you should just write off the possibility of becoming fully certain about what option is best instead what you should do is you should reflect on the decision like proactively that is you know talk to people write down your like thoughts and just
like keep iterating on that until the like the dial stops moving backwards and forwards and just kind of settles on some particular uncertainty so it's like look i guess i'm i'm kind of 60 70 option a's better than b and that hasn't really changed having done like a bunch of extra thinking
that's roughly speaking the point where it might be best to make the decision rather than holding out for um certainty does that make sense yeah it's like kind of like gradient descent where if if the loss function hasn't changed in the last iteration you uh you call it yeah i like
it like it yeah that's super interesting though i i guess one problem maybe that somebody might face is that before they've actually done things it's hard to know yeah uh that like that's actually a like not that this is actually going to be my career but i would have like the podcast was just
something i did as in like i was bored during cove and i i i yeah the classes went online and i just didn't have anything else to do i don't think it's something i would have pursued if i ever thought of it well i never thought of it as a career right so it's like but just doing things like
that can potentially lead you down interesting um interesting outfits yeah yeah yeah i think that's a great point there was um i guess we're both involved with this uh blog prize and there was a uh like a kind of mini prize last month for people writing about like the idea of agency and what you
just said i think links into that really nicely there's this kind of property of going from realizing you can do something to doing it which just seems like both really valuable and learnable so yeah just like have it going from the idea of i could maybe do a little podcast series to like
actually testing it and like being open to the possibility that it fails but you learn something from it just really valuable also we were talking about sending cold emails in that same bit of the conversation right like um if there's someone you really look up to and you have you think it's like
very plausible that you might end up in their like line of research and you think there's a bunch of things you can learn from them as long as you're not like demanding a huge amount of their time or
attention then you can't just like ask to talk to them i think finding a mentor in places like this is uh just like so useful and just like asking people if they could fill that role like again in a kind of friendly way it's just um you know maybe it's a kind of a move people don't opt for
a lot of the time but yeah it's like taking the non-obvious options being proactive about connecting to other people seeing if you can like physically meet other people who are like interested in the same kind of weird things as you yeah this is all like extremely obvious but
i guess it's stuff i kind of would really have benefited from learning um earlier on yeah and the unfortunate thing is it's like uh not clear how you should apply that in your own circumstance when you're um when you're trying to decide what to do um okay so yeah let's uh let's close out by
talking about uh just like plugging the effective ideas uh the block prize you just mentioned and then the the red teaming ea uh a contest you want to talk we already mentioned that earlier if you just want to leave like links and just again it summarizes them for cool i appreciate that um
yeah so the criticism contest the deadline is the first of september the kind of canonical post that announces that is an ea forum post which um be very grateful if you could link to somewhere but i'm you know happy to do that and then price pool is um at least a a hundred
thousand dollars but possibly more if there's just like a lot of exceptional um entries um and then hopefully all the kind of relevant information is there and then yeah this block price as well which i've been kind of helping run i think you mentioned right at the start so the
like overall price is yeah hundred thousand dollars and up to five five of those prizes um but also there are these smaller monthly prizes that um i just mentioned so last month was the theme was agency and the theme uh this month is to write some response or some reflection on this uh
series of blog posts uh called the most important century uh blog post series by holden karnovsky which incidentally people should just read anyway i think it's just really like truly excellent and kind of remarkable that like one of the most affecting like series of blog posts i've basically
ever read was written by the co-ceo of his like enormous um like philanthropic organization in his spare time it's just kind of insane um yeah so uh the website is uh effective ideas.org yeah and then um obviously uh where can people find you so your website twitter handle and then
um yeah where can people find your podcast oh yeah so website is myname.com twitter is my name um and podcast uh is called hear this idea as in listen this idea so it's just that phrase.com um and i'm sure if you if you kind of google it it'll it'll come up um but by the way what is
your uh what is your probably distribution of how impactful these um criticisms end up uh being or just how good they end up being like if you had to guess what what is like your median outcome uh and then what is like your 99th or 90th percentile outcome of how good these end up being yeah
okay that's a good question i feel like i want to say that doing this stuff is really hard so um i don't like discourage posting by saying this but i think you know maybe the median submission is you know like really robustly useful absolutely worth writing and submitting that said maybe the
difference between the most valuable posts of this kind or work of this kind and the median kind of effort is probably very large which is to say that the the ceiling is is really high if you think you have a one percent chance of influencing 100 million dollars of philanthropic spending then
there is some sense in which a you know impartial philanthropy donor might be willing to spend uh roughly 1 of that amount to kind of find out that information right which is like a million dollars so yeah this stuff can be like really really important i think
uh yeah yeah okay excellent yeah so the stuff you're working on seems really interesting and the block prices seem up like they're gonna they might have a potentially very big impact i mean our our world views have been shaped so much by some of these bloggers we've talked about
so yeah if uh if this leads to one more of those uh that alone could be uh very valuable so finn thanks so much for coming on the podcast this was um the longest but also one of the most fun uh one of the most fun conversations i've gotten a chance to do the whole thing was so
much fun thanks so much for having me thanks for watching i hope you enjoyed that episode if you did and you want to support the podcast the most helpful thing you can do is share it on social media and with your friends other than that please like and subscribe on youtube and leave good reviews on podcast platforms cheers i'll see you next time