0:00
You got two things that determine how your life turns out.
Search this video
0:02
One is luck, which sorry, you can say I make my own luck all all day.
0:06
That literally is not a sentence of English that makes sense.
0:10
The other thing is decision quality.
0:10
At the core of every single decision is a forecast.
0:15
>> Everything is a bet, >> right?
0:16
Here's the set of possible outcomes.
0:18
There's a payoff associated with each of those outcomes.
0:20
That's how we calculate the expected value toward your goals.
0:23
But the explanation that we're jumping to is inaccurate.
0:25
And it's because we don't know how to interrogate the data.
0:35
>> This one is particularly special.
0:35
I just had an incredible conversation with my friend Annie Duke, the former worldclass professional poker player.
0:43
Most of our conversation was about her new book, but she's written several books on decisionmaking and I recommend all of them.
0:52
Please enjoy my incredible conversation with Annie Duke.
0:55
Annie Duke, welcome back to Infinite Loops.
1:02
It's so good to see you in person.
1:04
>> In person this amazing last time I was on Zoom.
1:07
>> I was reminiscing about Do you remember when we were at uh Friend of a Farmer and we were talking about your last book? >> Yeah.
1:13
>> And that turned out to be so great.
1:13
But before we started recording, you started telling me about your new book, which absolutely fascinates me.
1:20
We're not going to ask you the whole thing, but okay.
1:23
First off, tell our listeners and viewers a little bit about what it's about because I think boy is the time right for this book. >> Oh, well, thank you. Yeah.
1:31
So, a lot of what I think about, you know, obviously I'm trying to help people to make uh more effective decisions.
1:39
And part of the way that I think about that is that at the core of every single decision is a forecast.
1:48
So I mean if you think about any decision you make right like you're considering different options.
1:52
Should I take this way to work or this way to work?
1:55
Everything is a bet >> right?
1:56
Uh what you have to do for any of those options that you're considering is make a forecast of how you know how much that uh option that you select is going to on average help you gain ground toward your goals uh in alignment withever whatever your values are. Right?
2:12
So obviously when we get into the value thing we get into utility.
2:16
But so those forecasts though as you're thinking about that right like I'm thinking about this option.
2:22
Here's the set of possible outcomes.
2:24
There's a payoff associated with each of those outcomes.
2:28
There's a probabilities of those occurring.
2:30
That's how we calculate the expected value or ground gained toward your goals.
2:34
Those forecasts can only be as good as the inputs.
2:36
How do we think about those inputs?
2:40
Well, you know, we have experiences that we might learn from.
2:42
A lot of what my past writing has been about is how the way that we interpret those experiences uh will be biased in a way to cause us to uh come to inaccurate conclusions or or lessons learned about um the experiences that we have.
3:00
So for example, like if you think about the centerpiece of thinking in bats, it's really this resulting problem that makes it so that when we have an experience like things go poorly um that we draw the wrong conclusions about what the decision quality was and then that causes us to uh make bad worse decisions going forward.
3:19
If you think about that, that's really a forecasting problem that it's changing the way you think about the probability of a bad outcome occurring under the same decision circumstances, right?
3:29
So that's kind of like, you know, how are we learning from our own experience, right?
3:33
But there's this other source of, you know, source of input, right?
3:39
There's this other input into the decisions that we make, which is the information that we come across.
3:45
And we're going to come across that information or data either because uh it exists out in the wild like we find that data or that information somewhere else like a user on social media might offer it to us or we might self-generate right.
3:57
So, uh, we could be doing analyses of our portfolio performance, for example, or we might be looking over like all the founders we've ever invested in and uh trying to signal detect like what founders are good or what founders are bad or uh we're tracking how our sales are doing given like maybe different marketing strategies that we're trying.
4:19
And this this is all like data generation and we might be creating data visualizations and things like that.
4:25
And then we're trying to uh come to conclusions about what is causing us to observe those things that will then help us to make decisions about how to get better outcomes for ourselves in the future.
4:39
outcomes for ourselves in the future. So we might be looking at uh you know do we can we come to the conclusion is there an explanation for what we're seeing in the data that says that the marketing strategies that we're trying are actually improving sales and if we think
4:54
that that's true uh then we may invest more money in those marketing strategies as an example we might see data about the efficacy of vaccines and that might drive decisions for ourselves about whether we ought to get a vaccine or not uh depending on what we're looking for in terms of our own health outcomes as an example. Right. All right. So that's Right. All right.
5:14
So that's this other source of input um separate and apart from our own experience that we're that um is driving the types of decisions that we make.
5:25
So anyway, I was thinking about that particular problem and I said, you know, it's kind of interesting.
5:31
Everybody really seems to be very worried about misinformation in this world, but I come across um some uh work from Duncan Watts who's at Penn.
5:39
Um and he had actually been looking at let's divide it into two worlds kind of like misinformation, let's call it someone just making something up and lying to you, >> right?
5:51
Propaganda, >> propaganda, like you know, bots, right?
5:56
that kind of stuff >> versus information that is either uh offered in a way that's been misinterpret misinterpreted by the person who's offering it.
6:06
So this is they're not lying, right?
6:07
So the the core fact that they're offering you is true.
6:11
So that's the difference.
6:11
So in misinformation, the fact that's being offered is not true.
6:15
Someone says 17,000 people died in New York City last year.
6:19
It's you can go look that up. It's not true, right?
6:21
versus I offer a fact that's true.
6:24
Let's say it were true that 340 people were murdered in New York City last year.
6:30
Let's just say that's true.
6:31
Um and then I draw a conclusion from that number that is misinterpreted.
6:38
So it ends up being misleading, right?
6:40
And it could be that the person offering it uh is actually offering that interpretation that's misleading or misinterpreted or you could just misinterpret it yourself. Okay?
6:48
Okay, so let's sort of divide it in that those two worlds.
6:51
And what Duck and Watts found is that it's 41 to1.
6:53
The misleading thing, the misinterpretation is actually the bigger problem.
6:57
So when I saw that, this actually it aligned with something that had made me really mad.
7:05
Um, which was an article that I read in the Washington Post in 2022 where I really felt this problem of look, if you think about it, I don't think that the majority of people are trying to lie to you.
7:17
you. I really don't like yes do we all have our biases sure but I think what the biases are doing is kind of under the hood driving the types of explanations that we're jumping to when we see data right so so you have some sort of piece of data and that data is
7:34
merely and I think it's hard for people to wrap their heads around it but it's just a description it's just a description of something that's been observed so we can take like a simple bar chart it's a description there were this many at this time this many at this time this this time. It's just a It's just a description.
7:49
But like when we see like it's small this year and then it's bigger this year and it's bigger the next year, we think it's actually giving us why.
7:58
We we sort of jump to that why.
8:01
And the why that we're going to jump to is certainly under the hood going to be driven by bias.
8:05
But I don't think that people are like purposely lying to you.
8:09
I don't think they're purposely trying to mislead you.
8:10
And I think that when you're looking at data that you might have generated say with your in your like own investment firm or whatever that you're certainly not trying to lie to yourself there, right?
8:21
But we do end up in some sense lying in the sense that we're that the conclusion or the explanation that we're jumping to is inaccurate.
8:31
And it's because we don't know how to interrogate the data. Okay.
8:37
So that's this is what I'm trying to address and it aligned with this work that I had seen from Duncan Watts because I was already really itching sort of in this space uh to write this book because of this Washington Post article. So here's what happened.
8:50
Um in 2022, it was like October of 2022, I read this article in the Washington Post and at that time the Washington Post would have been considered liberal bias for sure.
9:06
Um and at that time 2022 uh certainly provaccine bias, right?
9:10
So uh the the headline of this article caught my eye because of that because it was basically like in in friction with what I knew their bias was and the title of the article was uh COVID is no longer a pandemic of the unvaccinated.
9:28
So I I mean I really perked up.
9:28
I was like what whoa what kind of data is in this?
9:35
Because the Washington Post is certainly not, you know, we're not talking about a paper that that's like antiax, right?
9:42
Where I would have all sorts of reasons to be skeptical of that.
9:48
>> Um, in this case, I'm like, "Wow, like something must have happened."
9:53
>> So, I read the article and the main data, and again, this is a description.
9:58
This is what we have to remember. It's just a description.
9:59
The main data that they're citing is that in August of that year, 58% of the people who had died of COVID were vaccinated. 42% were not. Okay.
10:09
So, the reporter having seen this and go fact check it. It's true.
10:17
You can fact check for August of 2022.
10:19
You'll see that that's true. And this is the problem.
10:22
>> But the old quantity me is just red flag.
10:24
>> Your head is exploding. Right?
10:28
>> So, so this is what happened.
10:28
that it's like you know this is bigger than this. 58% is more than 42.
10:33
So that means like more vaccinated people died of COVID than not vaccinated.
10:39
So vaccines must not work.
10:42
Clearly the editor had not seen that there might be an issue with this explanation.
10:48
Um so I said well okay wait I'm going to keep reading there must because they're going to tell me what percentage of the population are vaccinated right?
10:55
like obviously no they did not.
11:00
So I was like all right let me go look that up.
11:03
So I went and looked it up cuz it wasn't in the article which made me very upset.
11:07
So I went and looked it up and they had a definition in there and according to their definition at that time 80% of the population was vaccinated.
11:16
So I was like okay well wait just right let's just start here.
11:22
80% of the population is vaccinated.
11:26
58% of the people who died of COVID are vaccinated.
11:28
Let's look at it from the other frame.
11:30
20% of the population is unvaccinated.
11:33
And they accounted for 42% of the deaths. Right?
11:35
So I was like, okay, wait. It okay.
11:38
It I certainly think that this this explanation, which is the headline, is is not only unwarranted, but like >> dangerous. Yeah.
11:48
Um, so then I was like, well, okay, like if I really wanted to take a step further, I ought to age match, right?
11:54
Because obviously older people are both more likely to be vaccinated, more likely to die of COVID than younger people.
12:01
There was a statistician who had, somebody else had gotten upset about the article.
12:04
Um, and they did do the age matching and it turned out that you were five times more likely to die if you were unvaccinated of COVID in the month that the Washington Post declared that CO was no longer a pandemic of the unvaccinated.
12:20
And you know, I just said like I don't think this reporter is trying to lie.
12:23
I just think they don't know how to interrogate data.
12:29
>> So they see this data and they don't know the questions you ask.
12:31
know the questions you ask. most simple of which would be out of how many which was the first question I asked right well what wait how many people are vaccinated versus unvaccinate like I need to know that thing right before I can even interpret this on another
12:47
stream I've been seeing my clients making these type of errors too right so a very simple example would be I had a client who was uh trying to increase their inbound topunnel this was an investment client uh trying to increase their inbound top of funnel investment opport opportunities. Um, and they
13:02
Um, and they showed me a uh histogram.
13:04
Um, and it was just like here's the size that were in top of funnel in 2016, 2017, 2018, 2019, so on so forth.
13:14
Um, and all of a sudden, like around 2020, this thing hockey sticks and there's a little arrow there.
13:24
There's an arrow that says implemented experimental marketing strategies.
13:27
So they said, "Well, we just want you to look at this because like is this sort of part of my job, right?
13:32
We want we want we want you to look at this because what we really want to do is invest heavily in these marketing strategies because you can see that they cause this to go up."
13:41
So notice that again we're talking about this big jump, right?
13:43
So I sort of call it crossing the chasm, you know, from description.
13:48
Okay, you've shown me a description.
13:50
This is how many in each year to explanation.
13:52
These marketing strategies cause this thing to occur.
13:57
That's an explanation, but there's this big thing in between that you've just crossed the chasm, right? Interrogation.
14:03
How would we interrogate that data in order to understand that?
14:07
Now, I I'm guessing to a lot of people that when I sort of describe this situation where they're like, well, isn't that obvious?
14:12
But I can show you why it's not obvious pretty easily.
14:14
I could say, what if I had an arrow there and it said Russia invaded Ukraine? Would you fall for this?
14:21
Would you fall for this? And the answer of course is no because there's no there's this great concept um in cognitive psychology uh Tanya Lumbroso talks about this called explanatory satisfaction and it's when an explanation it's
14:39
exactly what the name is feels satisfying right and that could be because it confirms a bias that you have right so if I if I have some sort of bias and the the the um explanation confirms that bi bias or beliefs that I already have, I'm probably going to feel pretty satisfied by it. At which point,
14:55
At which point, just like when you're satisfied by your meal, you don't go look for dessert.
14:59
You you're not going to go and try to do other things with the data.
15:02
Another way that it can be satisfying, uh, is that uh, and this happens, I think, a lot in the investment world, it feels insightful and contrarian because then you feel like you discovered something that other people don't know. Right. >> Right. Right.
15:18
And so that could be another way that like or sometimes it's just simply that it makes sense. >> Yeah. >> Right.
15:23
That it kind of >> or that your reasoning is being motivated by some other external factor. >> Yeah. It it could be that.
15:28
I think the makes sense thing though is a good thing to sort of just linger on, which is we don't like randomness.
15:35
We want to know why things are occurring.
15:36
So when we kind of land on an explanation where it sort of solves that discomfort that we don't know why something happened, we'll often just stop just because ah that makes sense, right?
15:48
So that that actually can create explanatory satisfaction.
15:51
It's just that feeling of like, yeah, that makes sense.
15:52
So that would be like when you don't necessarily have like a a dog in the race, right?
15:57
Like it's like, okay, I' I've got no dog in this fight now.
15:59
I don't have to feel uncomfortable that I don't know why things are happening in the world.
16:02
Okay, so we can take this example and say, well, why are they jumping to that conclusion?
16:08
Well, doesn't it feel good that you experimented with some marketing strategies and then you're inbound top of funnel hockey stick?
16:12
So, anyway, I looked at this and I said, "Oh, okay, that's interesting.
16:18
Um, before you do that, could you just make me the literally an identical histogram that's just total opportunities in each year? That's all I want."
16:29
year? That's all I want." And so they did that and then then the marketing strategies never got implemented because it just turned out that at that there were just like a whole bunch of extra opportunities that have been created partly because of co right so um >> so it could have been that the
16:46
opportunities were flat and it really worked right like I didn't have an opinion and I want to make this clear I didn't have an opinion about whether those strategies did work or didn't work there there'd be no reason for me to have the the opinion my opinion was whether you could draw that conclusion from the data that I had been shown. That was my that was all that I cared
17:04
That was my that was all that I cared about at that time.
17:06
And I said, well, I don't know until I know how many total opportunities there were.
17:09
Uh, and if you can show me that, then you could say, well, what proportion of the total opportunities did you actually capture?
17:18
And if it's true that you were capturing a larger proportion of the total opportunities, then I'm going to support your conclusion.
17:22
So, I just want to make it clear, it's not just like, oh, the media is telling you things and not giving you context or whatever.
17:28
giving you context or whatever. which is a problem if you don't know how to take that into your own hands and actually ask the type of questions that I did of that article but it's also like within your own business particularly in in this world of like data insights and like AI is generating data for you and how how do you even know how to
17:46
interpret it right because the AI isn't going to interpret it for you you like you have to know how to ask the right questions whether you're asking the questions of an AI or your own data or your insights team or or whatever it might be you have to take that into your own hands because that's going to determine choices you make about like your health, right? It's going to where
18:03
It's going to where you live.
18:06
Um it's going to help you decide like if you're trying to compare the performance of one person versus another.
18:14
Uh you have to it's going to tell you what you think is causing the outcomes that you observe in the world.
18:20
And then sometimes that means that you're going to say this is signal for something to occur or sometimes it's if I do this I can cause a greater likelihood of X occurring or a lower likelihood of X occurring.
18:31
It really really matters.
18:34
So the book is really just trying to get people to say I have to approach data with enough skepticism that I'm always going back to the I'm never skipping the interrogation step and then I need to know what the things are that I need to interrogate.
18:51
And they're super simple things like out of how many what am I comparing it to? Is the sample any good?
18:57
Um, and so I'm just building that out through a, you know, fun narratives and like the COVID story which is in there obviously. Yeah.
19:05
Um, um, there's a lot of like sports stories in there and then there's, uh, stories from like my own clients and, um, trying to get people to understand >> you need to not allow information to happen to you, right?
19:19
You have to look at the information and say if I were to think about how I explain what I am observing because again data is just it's just a description of an observation something you've observed in the world.
19:32
If I'm going to explain it what are the questions that I need to ask of this in order to actually get to the closest to the right explanation that that I could. >> Yeah.
19:44
And you know, as as you were talking about it, like some data is just so consistent with people's priors that they just even if you tell them I I had an experience where uh I put it in my book, What Works on Wall Street.
20:00
For a five-year period, there was an investment strategy which was really simple. It was one line.
20:07
Buy the 50 stocks with the highest gain in sales year-over-year.
20:15
makes a lot of intuitive sense to people.
20:17
Wow, their sales are cocky sticking and they're and then you show them the actual results of that strategy over a 5-year period and it kills the market.
20:27
It like triples the S&P 500.
20:32
The problem was it was for five years and we were looking at a data set of nearly 50 years.
20:37
When you run that strategy over the entire data set, it's worse than tea bills.
20:43
And yet, I could have, and I've been duplicitous and and not honest, I could have gone out and raised a [ __ ] ton of money on that. >> That's exactly right.
20:55
>> And people just fall in love with that idea.
20:58
One of my favorite examples that I have always used is when someone says to you that this player has uh had a hit six out of their seven last at bats that you always know that on the eighth one they did not get a hit. >> Right.
21:21
And that's that problem like why are you showing me these five years? >> Yeah.
21:25
>> Because I think sometimes that can be disingenuous, right?
21:27
like some sometimes someone can literally be trying to sell you on something that isn't true.
21:30
Again, even in that case, it's up to you because if you look at those five years and you fact check it, it's going to be right.
21:38
So, it's up to you to say, well, wait, what's what else should I be asking about this?
21:42
Like, is this the right sample to be looking at, for example, or why are they showing me these exact five years or or whatever it might be, right?
21:49
Like, it's up to you to say, "Hey, hold on a second."
21:52
say, "Hey, hold on a second." But then if you're this and this is a very strong tendency within business you have to understand that you're actually doing that to yourself all the time because what people do a lot is they kind of like you know massage they don't massage the data they just data mine right they're just looking for which comparison is going to tell me that this
22:12
thing that I did is better and I think that most people don't understand that there's something wrong with that they think that they actually find the truth when they find out that like X is better than Y and they don't understand well if you try a hundred different ways you're just randomly going to find out that X is better than Y and it doesn't necessarily mean that that's going to hold going forward. Yeah. Yeah. >> Right.
22:31
So I don't really address so much of that in this book of of things like you know just hygiene around data you know like you have to segregate some of the data so you can do all the massaging you want on 60% of the data but then you have to see if it's predictive of the other 40% for example like I don't care if you massage 60%.
22:48
Like you just have to make sure it predicts the other 40% that got quarantined. >> Yeah.
22:53
You have to have a out of sample hold out. >> Exactly.
22:55
You know, you you can obviously within the uh social sciences now and cognitive science, you have to pre-register your hypotheses.
23:02
You have to pre-register the analyses that you're going to do that helps you with that type of hygiene.
23:07
I don't you know that's not the everyday person doesn't necessarily need to know that.
23:10
What the everyday person needs to know is well am I seeing the whole sample?
23:16
That's a good question, right?
23:18
Like you know what's the denominator?
23:20
Um that's my favorite one.
23:23
You know, just like what how how out of how many uh there's a famous example on hormone replacement therapy that now has just become quite well known uh partly through Peter's book outlive.
23:35
They looked at a large sample of people who were taking hormone therapy versus not.
23:39
These are women who are permenopausal, menopausal or post-menopausal.
23:42
And in the group of people who were taking hormone therapy, five ended up with cancers, reproductive cancers.
23:51
And in the group that weren't, four did.
23:54
Now, the way that that number was reported in the media was that you had a 20% a 25% increase in your risk of cancer.
24:00
That's true if you're looking at four versus five, right?
24:02
But you have to say, well, wait a minute, but out of how many people total?
24:07
Like, because I need to know what the absolute risk is before I compare the relative risk.
24:11
So, it turned out it was out of a thousand.
24:14
So, it was four out of a thousand versus five out of thousand.
24:18
Now, you're smiling because like immediately you're like, well, then that's not even a difference, >> right?
24:23
I think it ends up I can't remember exactly.
24:25
I h I have it in the book. I think it's under one. >> I think it's 0. 06.
24:30
I mean, it's really small. >> Yeah.
24:32
um it's not a 25% increase, right?
24:36
So, but because it was reported that way, a generation of women were not given hormone replacement therapy and suffered through what are for some people really debilitating symptoms of menopause.
24:50
Like, so these questions that you have to ask are like really high stakes.
24:55
I'm not just being like a nitpicky like I know the scientific method and I you know you need to know how to interpret data.
25:02
You as an everyday person in your life are making decisions about things like what medical treatments are going to work for me.
25:10
What are the things that I'm going to do?
25:12
What type of exercise am I going to do?
25:14
What type of marketing strategies am I going to invest in?
25:16
What type of signals are am I do I think that I'm detecting in terms of what makes a good investment versus a a not so good investment?
25:24
You know, so on so forth.
25:26
where you it could be as simple as like which baseball player is better than the other one.
25:30
Like I would I which I have an example of in the book. So these things matter. That's the thing.
25:34
They matter for government policy.
25:36
They matter for your own health issues.
25:38
They matter for your business.
25:41
Um and we have to start paying attention to how do we get good at this thing that feels like in in some ways like eggheheading and don't tell me about your math, right? Right.
25:51
>> But you don't need to think like these these questions are actually really not very you know as you know >> while I write all of my books are about mathematical concepts.
25:59
They're not what I would call mathy. No. >> Right.
26:02
It's just more of these general concepts of like well why don't you just ask out of how many?
26:06
Like >> and you're if you just if that was the only question you ever learned your life would be a lot better. >> Totally agree.
26:14
And uh the there's a couple of things right.
26:18
So, you know, I did this series on the great reshuffle, right?
26:22
And one of the things that I was thinking about as I was getting ready to chat with you was like the world we came from was kind of a clockwork world, right?
26:33
A implies B implies C implies D.
26:36
You know, you climb the ladder, you get your gold watch, you get a pension, you know, very simple, very deterministic, right?
26:43
And and what's a deterministic type game? Chess, right?
26:46
If you know the rules, uh, you can get pretty good at chess.
26:52
The world we're going into is a probabilistic world.
26:54
And what game is best for that?
26:58
High stakes poker, right?
27:01
You have to make decisions on incomplete information.
27:03
You have to understand that people might be lying to you, i. e. bluffing.
27:08
You you have to be able to update your priors immediately when the new card turns over. Right?
27:15
So, I just don't think your average person is built for that.
27:21
You know, it's something I've banged on and on and on about.
27:23
You know, we are deterministic thinkers living in a probabilistic world and hilarity or tragedy often ensue.
27:30
And so, my question to you as one who is like deeply in this domain, is it just part of our human nature that we don't grock this stuff?
27:43
So, I think like here's here's the thing.
27:48
Well, first of all, let me just a slight quibble.
27:50
There has been no time in our history where we weren't living in a probabilistic world.
27:54
The question is, how big an error was it to act like it was deterministic? >> Deterministic.
28:01
>> So, the slower the cycle of change, >> the less of an error it is to act like the world is deterministic.
28:07
But I can show you that even back then the world was probabilistic because uh we did have big sea changes like um you lived your life thinking that you were going to be an Agg, right?
28:19
And then you shifted to everything all of a sudden shifted to manufacturing. >> Yep. >> Right.
28:26
Uh and it's like whoa I my whole life just got turned upside down. Right.
28:31
Um, and that's where you got like for example the lite movement where they're saying, "Whoa, we don't like that this changed.
28:37
We thought the world was going to be this way.
28:39
We thought that was a good prediction to make and now like oo technology and stop that." Right?
28:42
So I think that now those the the rapidity with which it's shifting uh makes it so that it's much more obvious that it's an error to behave as if the world is deterministic.
29:01
You're going to get punished much more quickly for thinking that. Right.
29:05
that. Right. And I think that that's particularly true over say the last 25 years that that cycle has sped up because even back in 1985 or whatever um I would say even in 1990 the early 90s you you still could believe that you
29:21
know if I go do this then these things will happen and these opportunities will be available to me right so let me just kind of start there but um let's think about an evolving human who believes who lives their life not believing that things will be the same, right? That for example, so you have to
29:42
That for example, so you have to figure out your migration pattern, right?
29:48
If you're a nomad, um if you're not a nomad, you have to figure out like uh when are you planting, when are you harvesting, when are you doing don't you have to believe the world is deterministic in that case, right?
29:57
I mean that's why we have rituals, right?
30:01
That's where Thanksgiving or harvest comes from, right? Sure.
30:04
>> Um uh you know we're planting at a certain time during the year where now obviously they understood there were ups and downs in terms of rain and when that happened that was very uncomfortable and so they would do rain dances or in the worst cases they might sacrifice a virgin or two right which is like you know but they're where they're trying to sort of overcome some of the randomness that occurs.
30:24
that occurs. But of course like it they they must believe that because they have to plan they have to plant at a certain time they have to harvest at a certain time they have to and that they have to believe that that's going to be the same
30:39
over time right as as the world becomes more complex because uh we're more global our social groups are bigger and as technology advances at a faster pace then what happens is that our minds that we're built to be deterministic, I think for relatively good reason, right? To
30:57
To keep structure in society and structure in our day and structure in our year and structure in our lives.
31:03
Um, now all of a sudden is butdding up against the complexity and and and the speed of change, right?
31:11
Where it then becomes a much bigger error.
31:13
I can take that to a broader view of just kind of bias in general, right?
31:18
general, right? which is when we think about a lot of the sort of you know when we think about the heristic side right these shortcuts that we take they mostly work >> which is why we have them >> um but then when they don't work it's really bad and the more sort of complex
31:34
the world is the worse it is so you can think about something like the availability bias for example well if we're living in a very small territory in a social group that isn't more than 300 if we see things more they probably do occur more in the world it's actually like a pretty reasonable shortcut under those circumstances. But when the world
31:51
But when the world becomes global and it's not just what we're seeing in our area and we're looking at the news and they show fires but not drownings, right?
32:00
Or uh I live in America so I think John is a more popular name when it's actually globally not that popular, right?
32:08
Um now it becomes a problem, right?
32:11
So that's just that's a way to add complexity into the problem.
32:16
So most of the ways in which we think about sort of cognitive error, a lot of them developed because they were shortcuts that worked most of the time in in in the world in which we evolved, right?
32:27
And I think that that's true of this sort of like thinking about things deterministically in the world that we evolved.
32:33
I think that that was probably mostly okay and you mostly weren't get punishing for it.
32:39
Unless you were the person who got sacrificed because you didn't have enough rain that year that then you got super punished for you got super punished for that.
32:45
But >> I think another thing at play and by the way I agree with you the world was always probabilistic.
32:53
>> Um uh but in the current environment it's really much more important to understand that >> very high uncertainty >> and and and yet another thing is this illusion of certainty we seek.
33:05
We we prefer the candidate who is certain in his or her pronouncements.
33:14
>> And one of the things I was thinking about, you've seen the news on on the prediction markets getting doing deals with CNN, CNBC, and I was thinking how cool to be able to use prediction markets to spin them up on and have a ticker on the CEO says we're going to he says with great certainty, we are going to double sales next year.
33:35
Wouldn't it be great if you had a decently liquid because you were good to call out liquidity, a decently liquid prediction market where the bets could go right underneath the statement?
33:49
>> Well, I you know, I think that would be great as long as at the same time we were educating people about how to think probabilistically, right?
34:03
So, I mean, obviously this is part of the reason why um I co-founded the Alliance for Decision Education because I think trigonometry is such a silly thing to teach the average person.
34:11
Um I don't think we're sailing by seextant anymore.
34:18
So, I don't exactly know why why people need to know trigonometry like if you're raising a barn. I I don't know.
34:25
But I think that that's something that if you're actually going to be like a structural engineer, you can probably take as an elective later.
34:29
And it would probably be better to be teaching people statistics and probability as as kind of a core decision skill, right?
34:36
I mean, you can teach a lot of stuff a lot earlier than at which point that would be um appropriate.
34:41
But the reason why I mentioned that is because we all know um that you can go back to the very famous 2016 election where I think Hillary Clinton was like 60% to win, >> right?
34:54
>> According to the, you know, I mean Nate Silver was essentially sort of running something that would have looked like a prediction market.
35:00
I think that had you if you had done a prediction market it would have been 665 probably wouldn't have been relatively in line with that and when Trump won people are like you were wrong.
35:10
So there we go right so yes I think it would be wonderful to have that just as sort of for people in the no certainly and I think that if it went below 50% it might be helpful if someone was declaring it to be 100%.
35:25
Would it be helpful if it was like the market was saying it was 78% and the person was declaring it to be 100%.
35:30
I don't think it would because I think that people would sort of interpret that as like okay well that's definitely a sure thing. >> Yeah. Yeah. Yeah.
35:38
>> As opposed to like okay that's like 3 to oneish, right?
35:41
Like so I'm going to put one bullet in a fourchamber gun.
35:46
Russian roulette anybody? Right. Right.
35:49
And I think that most people would be like, whoa, no.
35:52
If you put it that way, no. Right.
35:56
But like you I think you have to get it into those concrete terms for people to actually get it.
36:00
And likewise, I think that there are things that people can say where they could even add some nuance to it.
36:09
And you know, the prediction market would say this is 35%.
36:14
But actually given what the payoff for that would be, it would be like you would be willing to say, "Okay, yeah, I would take the 35% because the payoff's going to be five to one or something like that."
36:24
And I think that these concepts are actually just very hard for people to grasp.
36:27
Obviously, as a poker player, that's the whole gig, right?
36:32
It's uh okay, I only I remember once I was actually live streaming way back, this was in the early days of it.
36:38
I was live streaming um uh playing in a poker tournament and I had a hand and um I was getting 11 to one from the pot and this was on a final bet so I wasn't going to have to invest any more money.
36:50
There was nothing implied that was going to happen in the future.
36:53
It was just a straight call on the end and I had terrible hand.
36:56
I was getting 11 to one from the pot and I said, "Well, like I'm profitable here at like 9%."
37:07
And I just better than that actually. It's like 8%.
37:09
And I was like, "Well, I've never been that sure about anything.
37:13
I guess I should just call, right?"
37:15
Because I haven't, right?
37:15
Like I have to I said I have to be I have to be well like I have to be over 90% sure that this is not the best hand.
37:24
>> And I just went I Okay.
37:24
I I'm like I'm not sure about anything.
37:27
Like I mean I'm sure about my own name, but about that I'm not.
37:30
And I called and I won the pot.
37:32
And it was literally like I I was trying to sort of take the magic out of it.
37:36
It's like I don't have a dead read on this person, right?
37:39
>> I'm not I I do not I would not I'm not sure that they're bluffing.
37:42
I'm not declaring anything with certainty.
37:44
I'm just saying like I don't know.
37:46
I've never been over 90% on anything. I'm calling. Right. And I won the pot.
37:51
And I think that that type of thinking, right, it's it's like you have to get to people early to start to get them to be able to view the world through that lens.
38:02
So I think it would be great as like what's this and why should I care?
38:07
And then if we back that up with really good education, I think that would be like the most amazing thing ever. >> Yeah.
38:15
So talk to me about education because that's another hobby horse of mine, right?
38:19
We we are using an antiquated educational system for a world that no longer exists, right?
38:24
It was designed for the industrial era and where essentially you wanted to be able to have people sit in a room for eight hours and take instruction, right?
38:35
Um if if you were if you were put in charge, if you were the new secretary, do we still have an education department?
38:43
I don't know whether they accidentally >> I don't know if they're doing anything at the moment.
38:47
But I don't think it's been I don't think it's been cancelled yet, but >> but but tell me what what it would look like because I agree with you by the way that early education especially >> would be able to mold the way that the younger people are thinking on their way up.
39:02
But like what what would you change about the curriculum?
39:05
I think that there's a lot of people who are working in ed reform and a lot of the work is, you know, like we're going to create this like special type of school where, you know, people are doing something completely different.
39:21
And um I think that that's fine and I I think it's great to have those kinds of things, but I think that we do need to live in the reality that whatever you're doing, 80% of kids are going to public school.
39:37
>> And I'm including charter schools in that, right?
39:38
Like 80% of kids are going in public schools or charter schools.
39:42
And so you need to be thinking about how do we change education within that system.
39:48
I I just think that's it's very important to acknowledge that because I could say all sorts of things about here's a kind of school that I would create, right?
39:54
But that's great, but you're not you're you're not getting to the you know, most of the population.
40:01
>> Thus is why I made you the head of the department of education. >> Well, thank you.
40:07
>> So keeping keeping that as stipulated, keeping that in mind for public >> school since 80% of kids go to those, >> right?
40:14
So basically what I would be doing is changing the curriculum out of something which is very factbased into something which is very decision-based.
40:29
This shouldn't surprise you.
40:31
Um so there's some interesting things that you can do actually.
40:35
So, at the alliance, we we've actually developed a tool um and it's it's AI.
40:42
It's an AI tool where you as a teacher can input a lesson plan.
40:44
So, let's let's imagine that you're teaching McBTH or or you're teaching about the water cycle or pick your pick your topic, right?
40:53
you can input it into this tool and all that the tool will do is embed decision education decision skills into into the um into the lesson plan.
41:05
So we could think about something like McBth for example or uh Eisenhower, right?
41:09
Or or that and let's think about a skill that we can teach people.
41:15
Let's explore the counterfactuals.
41:17
What if McBth had done this?
41:20
>> Oh, that's cool, >> right?
41:21
Like how do you think that this would have turned out in the water table?
41:23
We could say, okay, here's the average rainfall in this area, right, per year.
41:28
What if we put a mountain here, right?
41:30
Because what really what are you teaching people in a factbased?
41:34
You're like, uh, here's here's the rainfall in an area where this mountain is here, and mountains affect things this way, but you're not actually asking people to put that into an actual forecast of what would happen to the rain if you stuck a mountain in a place, right?
41:48
Where they're having to take into account all the other things about the climate in that area, as an example, right?
41:52
Going back to McBth, it's like, well, what were all the options?
41:55
Why do you think they chose this option?
41:57
How did that affect Lady McBth?
41:58
How was Lady McBth's decisions uh taking into account McBth's desires versus her desires versus so on so forth?
42:08
What's interesting about it is I think it actually makes the material stickier anyway because you're not just memorizing facts, right?
42:14
They're now thinking for themselves.
42:16
So whatever decision skill you want to think about like teaching a agency or optionality or forecasting or bias, right?
42:24
What were their different biases?
42:27
You can start to teach the cognitive biases through that lens.
42:30
For example, um this idea of counterfactual thinking, right?
42:33
You can start to embed that into any lesson.
42:35
And I think it just becomes much more interesting than fact-based.
42:39
On top of that, I think that you have to start skilling kids up on how how are you thinking about getting good inputs from AI into your decisions.
42:56
So just a concrete example of what we're doing uh we have a forecasting competition super fun.
43:00
So you can think about like uh good judgment project which is Phil Catlock super forecasters right and Barb Miller.
43:06
They they've got good judgment people go on there.
43:08
you as an adult can go on there and you can enter a forecasting competition. It's very fun.
43:12
Um so we said, well, why aren't we doing this with kids, right?
43:16
So we have a forecasting competition.
43:16
We actually are able to give some kids training and some not.
43:20
Um preliminary results are showing that the training is quite effective.
43:24
Um the training is actually based on the work from my dissertation which showed that training uh noviceses um on some simple concepts for forecasting is very effective.
43:33
Things like what's the base rate um is a really good thing to teach people.
43:36
Um, so we do that with middle schoolers and high schoolers and we encourage them to use AI >> because that's the world we're living in. >> Totally right. >> Could not agree more.
43:47
>> But what we're trying to teach them is AI is a big fat liar unless you know how to prompt it correctly.
43:53
So what we ask them to do is show your work if you're using AI to help you with try to figure out how you're going to forecast.
44:00
And we have kids forecast things they care about, right?
44:04
like uh in the first week of uh the Taylor Swift new album drops, how many copies do you think it will sell? Right?
44:11
So, they care about that kind of thing.
44:13
If you're going to use AI to help you with that, show your work and show the prompts.
44:17
And we're going to also give you training on how to prompt AI, which is a decision skill in and of itself, right?
44:25
How do you prompt it to get you to give you an answer that's more accurate?
44:30
How do you check your sources?
44:32
How do you get it to be skeptical?
44:33
How do you get it to give you counterarguments?
44:36
How do you get get the right whatifs?
44:39
And what they learn is that the better you are at prompting the AI, the better the forecast that you're going to be able to make because your inputs are going to be better and then you're more likely to win the prizes that we're offering along with that.
44:49
So, I think that there's just like starting in kindergarten, you can start to teach this stuff.
44:55
You can start to teach very simple probability and we used to by the way it's gotten dropped out of the education system until about 8th grade now.
45:01
We used to start teaching probability in kindergarten.
45:05
So let's bring that back in.
45:07
A kindergartener could learn the concept of agency.
45:09
They could learn the concept of options.
45:12
Well, what are the different choices that you have?
45:15
Um for those different choices, what would what do you think would happen if you chose this thing?
45:20
Why do you think that would happen?
45:22
Why is that a conversation that we're not having with kindergarteners? Right?
45:25
and instead we're teaching trigonometry.
45:30
I just it's so weird to me.
45:33
>> I again you had me at hello there because I agree.
45:38
Um it we we're living in a world where I I I think you know when I was coming up your memory was super important and oh my gosh your your ability to recall facts and when you're talking about Mc Beth I immediately went the lady doth protests too much >> I still I could there was a siloquy I had to learn from King Lear Cordelia.
45:59
It was one from Cordelia because we all had to memorize something.
46:03
I can still say it, >> right?
46:06
Like why is this living living right in my head? But whatever. >> Exactly.
46:10
But that that isn't the case anymore, right? We we have ready. >> Exactly.
46:16
Like we had a set of encyclopedias on our Same with me.
46:18
I mean, I'm sure that at the time they were like 25 years old.
46:21
I mean, I don't think any of the information was particularly correct anymore, but if I wanted to know something, I had to look it up in there. >> Yeah. Exactly.
46:29
And and so you it it's not like the world we were coming from was some bizarro world.
46:36
No, >> it's it's that's what we had available to us.
46:41
And so you had if you had a good memory, good for you.
46:45
You're going to go a lot further.
46:46
If you were a high agency person, sort of by your natural disposition, >> good for you.
46:52
You're going to go a lot further.
46:55
But I don't think that these things are necessarily well the memory might be innate but I don't think agency is innate.
47:02
I think that you can learn agency. >> Exactly. That's exactly right.
47:05
You can see actually there are socioeconomic differences in uh in the way that people approach agency.
47:13
So when you see people of higher socioeconomic status, you know, which generally means that you sort of just live in a world where you have more agency, right?
47:23
And you see how they're interacting with a doctor, for example, they're asking all they're ask they're acting like, you know, hey, I'm your customer, right?
47:31
And they're asking all sorts of questions and they're just not accepting what they say.
47:36
But when you see people of lower e socioeconomic status and generally, you know, what obviously correlates with that is uh the parents tend to be uh not as well educated, for example.
47:48
educated, for example. again 10 not always but um and they're living in a world where it's a little bit more happening to you or at least it feels like that when they go to the doctor the doctor says it that's it they don't even ask any questions and they're told not to don't be don't be rude to that doctor right like they're the authority in the
48:05
room whereas when you know when you've been living a life where you understand you know a lot of doctors like you're like well you're an idiot I mean like you're the you're the dumbass at the party right who like oh my god would you shut up you're just more likely to be asking questions and skeptical and getting second opinions and things like that, which of course is an act of agency. So, I I think it's incredibly
48:24
So, I I think it's incredibly important to teach very early on this idea that you're an agent of your own decisions.
48:31
Um, and we can get back to like this core idea that's really throughout all of my writing, which is look, you got two things, only two, that's it, that determine how your life turns out.
48:42
One is luck, which sorry, it's exogenous force that acts upon you.
48:47
You have you have no I don't you can say I make my own luck all all day.
48:49
It's like that literally is not a sentence of English that makes sense.
48:52
I mean it's grammatically correct but that's about it.
48:55
Uh it has words in it that have meanings but that's it.
48:57
Um you know it's it's not like you're reading a Louisis Carol poem or something like that.
49:01
Like it's it's actual word words of English but as a string of English it doesn't make any sense because you can't make your own luck.
49:07
Definitionally >> I actually want to push back.
49:09
I don't believe you make your own luck, but I believe that if you have your recticular activating system, your aperture set very wide, you might be able to always be scanning for opportunities that >> oh are lucky.
49:29
>> I'm 100% with you on that. Okay.
49:29
I think that when people say I make my own luck, what they mean is I make decisions that increase the probability that I'm going to with you, right?
49:39
That doesn't mean that you've made any luck. >> No.
49:44
>> And by the way, nobody who's had bad things happen to them has ever said, "I make my own luck." No.
49:49
I just want to point that out.
49:50
Which is why, you know, it's dumb.
49:53
>> But but also but also the person who is forever going on and on about how everything in the world is due to luck is probably one of the most unlucky people in the world. >> Well, of course.
50:04
I mean, that person obviously didn't have good outcomes. That is for sure.
50:08
And that's the point, right?
50:10
like we want to take credit.
50:10
So like you know it's it's always the most successful people who are like I make my own luck right >> and it's like no I mean you may make better than average decisions but you do not make your own luck and obviously at the at at sort of the start of anybody's life is like look what if I had been born in 1600 what my life would different >> you know actually I love playing that game.
50:35
Yeah, >> because it it totally immediately informs you of how I I I say this often.
50:44
I am so [ __ ] lucky that I was born in America, right, >> in 1960 >> because like if if it was 1870 salon >> Exactly.
50:56
>> My opportunity set is going to be a lot smaller. >> Exactly.
50:59
And the thing is people will be like, "Oh, Jim, you're being so humble."
51:02
It's like, no, it's just true. >> Yes. Right. >> Right.
51:04
It's like if Bill Gates had been born 30 years later, >> no Microsoft, no billions >> because the computer already would have happened.
51:13
>> I make this point all the time about my own book, What Works on Wall Street.
51:17
The only reason I got to write that book was because chance luck made it me born in 1960.
51:26
If Ben Graham had access to the computers that I had and the databases that I had, he would have written that book.
51:34
>> It would not have been an opportunity for me. >> That's exactly right.
51:36
So, and the thing is that that doesn't take away from the accomplishment. You saw the opportunity.
51:43
You put in the hard work. You did the research.
51:45
You wrote the great book.
51:45
That that is that's the thing. It's like a blend. Yeah. >> Right.
51:51
And and it doesn't discount your accomplishment to say, "Hey, there's there was a lot of luck in what happened."
51:58
I mean, LeBron James is tall.
52:03
I'm pretty sure he didn't have any control over that, right?
52:05
Like, he's a great basketball player, but he's tall. >> That's right.
52:10
>> What if he were 5'4, right?
52:10
I don't think he ends up being LeBron James.
52:13
I had a statistician on the podcast who wrote a book about the NBA and he said it may not surprise you that your chances of being selected for the NBA go drastically higher if you're over 7 feet tall. >> That's exactly right. Right.
52:29
And like a little coordinated and then you're pretty good.
52:33
But you don't even need to be that coordinated cuz you just go like this. I've got a basket.
52:38
>> So like you have to you have to be smart enough to know the rules, you know, like Yeah.
52:42
So, so that's so that's thing number one is luck which you do not make.
52:47
It's just the thing it happens. You have no control.
52:49
The other thing is decision quality.
52:52
When we think about it like I'm born into the world and when I'm born into the world to whatever parents I'm born to with whatever talents or lack of talent that I might have um physical characteristics so on so forth, there's some distribution of outcomes that's available to me. That's true.
53:11
Now the question is what's the probability that I I end up at the right tail?
53:16
And that's going to be determined by the decisions that I make.
53:20
The better decisions I make, the more that I can get to that right tail of the distribution on average.
53:28
And obviously that doesn't mean all right.
53:30
I can make the best decision.
53:32
I can go through a green light and I a car can hit me. >> Yep.
53:37
>> What am I going to do?
53:37
But if if I'm consistently going through green lights and not going through red lights, the probability that I have better outcomes is going to be higher than someone who's consistently going through red lights, right?
53:48
And that's all we're trying to do.
53:49
The person going through the red lights could just literally have that h for their whole life be unscathed, >> but it's a much lower probability that that will occur for them. >> Right?
53:58
And many times people will confuse in their minds probability verse possibility. >> Right. >> Right.
54:05
They're two very different things.
54:09
>> Could happen, >> right? Yeah. Exactly.
54:10
It's it's it's how they market the New York lotto, right? >> Hey, you never know.
54:17
>> Yeah, it could happen.
54:17
>> Well, you do kind of know. >> You do kind of know.
54:21
Look, every once in a while you're going to get more than a dollar back for the dollar you spend.
54:26
>> Yeah, >> it's rare, but it it occasionally will happen.
54:29
And even when you calculate that, it would be really good if you calculated in the fact that the more people play, the more likely you're sharing, which then makes it really hard.
54:39
>> Fran Lewood had said a great line, which was, uh, I I think my chances of winning the lottery are the same whether I buy a ticket or not.
54:49
>> Close to it, >> but it's it's it is one of those I I love this framing because you're absolutely right.
54:56
the quality of your decision making on average.
54:59
decision making on average. It does not it does not mean you could make all the best decisions in the world and just sadly you are in the cohort that bad [ __ ] happens to but the the two together why is it so hard for people >> well I mean let's go back to explanatory
55:20
satisfaction right like people are so uncomfortable with randomness >> they really hate it >> they really hate it and obviously in its worst form it becomes a conspiracy theory >> of course >> right uh and this concept of like ah we're sampling a lot of stuff all the time and sometimes things will happen at the same time together. There was a a
55:36
There was a a study where um it basically just like you're given sort of a toy example, right?
55:46
And it's like, "Hey Jim, you know, there's this museum and uh we just found out again data which is a description.
55:52
We just found out that um people who visit the portrait room, the portrait gallery are more likely to give a donation on their way out.
56:01
You're sort of asked why and people are kind of like a little bit uncomfortable like I don't know why and they're sort of making things up but they don't necessarily have a lot of confidence in what they make up.
56:09
But then when you show them a scientific paper and this is true that when people are being observed they're more pro pro-social.
56:18
Uh then all of a sudden they're like, "Oh, it's because there are eyes in the room and that's why they're doing it."
56:24
And they're like 100% confident because it's taken away this like I don't know why these two things are crying together which like even on that small little bit and that thing that has nothing to do with you still makes you uncomfortable.
56:37
Now, how do you get them out of that? Right?
56:39
Is you say to them, "Well, here are a whole bunch of other explanations.
56:45
How then all of a sudden they'll sort of get off that like this is 100%."
56:47
It's if you give them this one piece of scientific data, which is true, and you let them make the connection, they actually don't self-generate explanations.
56:59
>> But when you're like, you know, oh, did you know that the portrait room is shabier than the other galleries, right? As an example, right?
57:05
So you start to generate other things that could be true.
57:08
Uh people who visit the portrait gallery are of higher net worth, right?
57:13
Like so now I can start to give you a lot of other things and then you go back to not being you know you're you go back to that but you're sort of back to being uncomfortable and you kind of want to know.
57:21
>> But if I if I give you a pathway to certainty, you're going to take it.
57:23
And I would go back to like you're living, you know, in a small tribe in this, you know, small area and like how are you supposed to survive if you think the world is just random. >> Right. >> Right.
57:38
Like you have to sort of >> Well, that's the illusion of certainty. Right.
57:43
It it I think is very much an evolutionary aspect of our behavior >> and it it's one of I had an example uh that I put in a couple of my books which is uh this uh professor did this study where it was Mr. Smith and Mr.
57:59
Jones and he told them that the object of the study was to see how good by getting feedback uh they could get at judging whether a picture of a cell was cancerous or not.
58:16
>> What he didn't tell them was that Mr.
58:19
Smith was getting true feedback.
58:19
So in other words, if he guessed right, he got a green light.
58:23
If he guessed wrong, he got a red light. >> Mr.
58:26
Jones was getting feedback based on Mr. Smith's guesses. >> Right. Okay. So, if Mr.
58:30
Smith guessed wrong, but Jones guessed right, he'd he'd still get a red light. >> Yeah.
58:39
>> What's really And they did this with multiple participants.
58:43
>> And what is really really interesting is the guy getting the true feedback is pretty soon improving his batting average to about >> right.
58:52
>> He's getting seven out of 10. Right. Right.
58:54
This poor guy over here is not.
58:57
But what's really interesting is when they come and they ask them, well, how do you make your prediction, the guy who's getting real feedback offers concise, straightforward examples.
59:08
Well, if the cell seems to have a broken edge or if it's fuzzy over here, I guess that it's sick.
59:17
And if it doesn't have those things, I I guess that it's healthy. Mr.
59:22
Jones, who's getting the bad feedback, launches.
59:25
He looks at this guy and he's like, "That's the most pedestrian example I've ever heard in my life." No, no, no.
59:31
Sometimes it's this and sometimes it's that.
59:33
And he just weaves this tapestry.
59:36
Very complicated tapestry.
59:39
What's really interesting, all of the guys getting it right are in awe of the guy who is getting it wrong.
59:48
>> Oh my god, you're so smart.
59:48
you're and when they go and do it again, his performance materially declines. >> Declines. That's interesting.
59:57
>> And so it's it's almost like we overindex on people who, you know, >> wave their hands a lot.
1:00:07
I I really try to teach people that when you're communicating information, this is so hard because of the point that you just made, that your job is to communicate at the exact same time what you know and what you don't know, right?
1:00:24
>> At literally the at the exact same time.
1:00:26
Now, what we tend to want to do is be like, I know this thing for sure.
1:00:29
Uh which is a very bad representation of what you know and what you don't know. >> Very bad.
1:00:34
So it's very bad representation.
1:00:36
So what what I try to tell people to do do is like let's imagine that I'm working with somebody and you know their job is they're making sales forecasts, right? Um okay, that's fine.
1:00:47
Give me a point estimate.
1:00:47
That's great because there there I can der there's some useful information I can derive from your point estimate.
1:00:52
But then what I'd like you to do is give me a lower bound and an upper bound >> and I can set 90% confidence interval, 80 80% confidence interval.
1:00:59
And just for those who don't know, like a 90% confidence interval just means that the true answer if I were omnisient would land within that boundary 90% of the time.
1:01:11
>> Y >> okay so give me a lower bound and an upper bound.
1:01:15
Now what you've actually done is told me what you know and what you don't know.
1:01:18
So there's a very simple example that I do in my classes uh which is so I have a dog.
1:01:24
Uh the people in my class have not seen a picture of my dog.
1:01:28
So, I always start with, "What's the weight of my dog?"
1:01:30
Very often they'll be like, "I don't know."
1:01:32
And I'm like, "Okay, that's fine."
1:01:33
But like, "Give me your best guess." Okay.
1:01:34
So, I'm just trying to get them to give me their best guess.
1:01:38
And usually they'll say somewhere between 25 and 45, maybe 50 pounds, somewhere in there.
1:01:44
So, so let's imagine they say 35 pounds. Okay.
1:01:46
So, I'm like, "Okay, that's your best guess.
1:01:48
What do you think the smallest amount my dog weighs is?"
1:01:52
Um, and usually they'll say like 10 pounds cuz I didn't say puppy, right?
1:01:57
Um, so let's say 10 pounds.
1:01:57
I'll go, "What what do you think the biggest amount my dog weighs is?"
1:02:00
And they'll they'll often say something like around 120, but sometimes as high as 150.
1:02:04
I say, "Okay, that's interesting."
1:02:07
And then I start to ask them some question.
1:02:12
So I can ask you a question.
1:02:12
Why do you think they don't say 500 lb? >> Why do I think? Yeah. >> Yeah.
1:02:17
Because uh there would be a very rare dog that weighs 500 lb.
1:02:22
>> I think it would be dead actually.
1:02:25
It could be a prehistoric uh dog.
1:02:28
>> A dog that I own today, right?
1:02:30
>> Cuz dogs don't weigh that much. That's right.
1:02:32
>> So, when I say to them like your initial instinct was, well, I don't know what your dog weighs, but like, oh, actually, you know a lot about what my dog weighs because things can weigh >> 10,000 pounds or a million pounds.
1:02:43
And you're you just gave me like a a very small range in the in the context of what the possibilities are that things and what you know about you gave me 10 to 120 pounds, right?
1:02:56
So you know they don't they don't come in one pound sizes if I'm calling it a dog.
1:03:00
Y >> and you know it doesn't come in 500 lb sizes. That's amazing.
1:03:04
So now I ask them another question which I'll ask you.
1:03:06
You can pretend to be the person. All right.
1:03:09
So, it's interesting because you gave me 10 lb to 125 120 lb, but your point estimate was 35 lbs, which is a lot closer to 10 than it is to 120. Why is that?
1:03:22
>> I wouldn't make that estimate, though. >> Well, okay.
1:03:24
So, what they say, and I I would argue they're correct to do this, is that larger dogs are rarer.
1:03:30
So, there's skew in the distribution.
1:03:34
Mo the majority of dogs that people have are on the small end of the scale, right?
1:03:38
Think about people in New York City who have apartments.
1:03:42
They're going to tend to have smaller dogs.
1:03:44
>> Well, sorry to interrupt, but that was where my mind immediately went. >> Oh, yeah.
1:03:47
That you would actually >> where I would like say, "Where are you living?" >> Yes. >> Well, okay.
1:03:52
So, that's my next thing that I point out. Okay.
1:03:54
So, this is I'm refusing to give you any information except it's a dog. Yeah. >> Right. Okay.
1:03:58
So, I say, "Oh, that's interesting.
1:04:00
So the fact that the point estimate is closer to the lower end than the higher end actually tells me something about the shape of the distribution, right?
1:04:09
How common are small dogs compared to Wow, you really know a lot about dogs.
1:04:15
And then I say, okay, so let's imagine that I'm asking you to do this, right?
1:04:21
I'm saying like give me a point estimate lower bound and upper bound.
1:04:24
Tell me what what are the things that you want to ask me?
1:04:28
And they'll say things like, "Where do you live? What's the breed?" Right? And I say, "Oh, okay. So, that's interesting."
1:04:36
So, if I actually allow you to build the uncertainty in, it causes you to start to generate questions that will help you because it opens you.
1:04:44
But here's the even better thing.
1:04:46
If all I do is declare to you, you know, 40 pounds, if you say something different to me, you now are are at risk of me thinking you are wrong.
1:05:00
But if instead I say, "Well, I think Annie's dog is like 35 pounds, maybe like lowest 10, highest 120 pounds."
1:05:05
I'm I'm included in that is a request for information, right?
1:05:13
Included in that is, "Hey, if you know anything, can you help me?"
1:05:18
So now maybe you'll be like, "You know what?
1:05:21
Uh Annie has, you know, a toy poodle.
1:05:24
You know, I actually think you should lop off that topic.
1:05:26
I think it's probably going to be lower.
1:05:28
But the point is that it invites you into the conversation to help me.
1:05:31
And now when we're thinking about I want to express both what I know and what I don't know.
1:05:40
Now all of a sudden you find out, oh, but that doesn't mean I don't know anything. >> Right. That's what I love. Yeah.
1:05:46
Actually, that's what I love about the framing of this.
1:05:48
I'm more likely to end up knowing more at the end of the process because first of all I'm going to be more inquisitive but also other people are going to offer more freely offer me information.
1:05:58
So I if I could change the world that earnings estimate would be lower bound this upper bound this with some context here's the circumstances under which we would see the lower bound.
1:06:08
Um so like simple example with sales estimates like let's imagine that someone calls a number like uh a million net new ARR at the end of Q4 uh lower bound 800 upper bound 2 million.
1:06:21
You immediately know that there's a big account that might slip to Q1.
1:06:24
Like it it's it's like it's so easy to figure that out and you can just say to the person tell me why.
1:06:29
Now as someone who's trying to budget for the organization isn't that way better. >> It absolutely is.
1:06:34
And and yet I'm also thinking about another example that you're what you're by the way I love that exercise because what I love about it is you are showing people that they know a lot more intuitively than they think they do and they're learning that through asking questions. Yeah. >> Which is great.
1:06:55
>> Um but the there's an example of another professor.
1:07:00
I love what these guys get up to.
1:07:02
But it essentially it they they give you one base rate.
1:07:06
They say that there is a town of 100,000 people.
1:07:08
70,000 of them are lawyers. 30,000 are engineers.
1:07:16
And then your job is to guess from a random sampling that we take out of uh of a big bag full of names, how many of the 10 that I pull out are lawyers and engineers.
1:07:28
So, I pull out 10 names and they're just a name.
1:07:31
Annie, Jim, Jean Mark, uh, Richard, right? Just the name.
1:07:36
What do you think most people say when asked, "How many of these are lawyers and how many are engineers?" >> I don't know.
1:07:45
I mean, I would hope they just go, "Well, well, you gave me the base rate, so I guess I'm going to say seven of them." >> Right?
1:07:51
They either do that and this is actually again through iterative >> trials of this.
1:07:56
They use one of two strategies.
1:07:58
They either go with the base rate and say that seven are are lawyers and and three are engineers. >> Yeah.
1:08:07
Given that I literally don't know anything else. >> That's right. That's all you know.
1:08:09
And the other uh one they go with is they say they're all lawyers.
1:08:14
They just keep saying they're all lawyers to try because they know they got to do it multiple times, right?
1:08:22
But then they add a little twist.
1:08:24
>> Okay, >> the next round with a different group, of course.
1:08:28
>> The the next round adds descriptive but meaningless information.
1:08:32
Annie is 44 years old and wears glasses.
1:08:35
Jim is 65 years old and uh you know does this this and this.
1:08:42
When they add descriptive pe uh information, people begin to largely deviate from the base rate. Yeah.
1:08:50
>> Then you >> Well, I mean, obviously the guy who wears glasses is an engineer, >> right?
1:08:53
Well, but then they add stereotypical. >> Yes.
1:08:57
>> And when they add stereotypical, people completely ignore the base rate.
1:09:00
Frank is shy, likes mathematical puzzles, and wears glasses.
1:09:06
You could jack up the number of uh lawyers to 99. It doesn't matter. >> It doesn't matter.
1:09:12
That one's the >> and and and so how h how how would you solve for that?
1:09:19
Because that's such a natural tendency >> as a mathematician.
1:09:22
Uh you ought to just be ignoring that information and you should just go with the base rate as your as your first best guess. >> Right.
1:09:30
>> Uh unless there's something causal, right?
1:09:32
So obviously wearing glasses doesn't cause you to be anything. Right. >> Right. Okay.
1:09:38
If you're going to deviate from the base rate as your forecast, you have to believe that there's some sort of dislocation that's occurred.
1:09:43
So there has to be some causal difference.
1:09:45
So the the example I give is like don't if you have to predict the probability that a cat 3 or higher hurricane is going to make landfall in the US in a given year, don't use statistics from the 60s because something has changed, right?
1:10:01
So that that I'm really trying to sort of pound that home that your best guess should always be the base rate unless you have a really good reason to deviate from it.
1:10:09
That is actually part of the training from my dissertation work.
1:10:11
Um and we do get people we can get people to do that, right?
1:10:16
It's part of the training that we're giving to the high schoolers and the middle schoolers.
1:10:20
Um now I'm going to put on my linguistics app.
1:10:24
There's a part of linguistics which is called pragmatics.
1:10:26
Uh and it just has to do with like well we know what pragmatics are, right?
1:10:31
like what is pragmatically.
1:10:32
So um okay so let's imagine that there's only one pencil on the table.
1:10:42
There's only one and it's red.
1:10:46
Um there's a very big difference in how you think about the redness of the pencil if I say can you hand me that pencil versus can you hand me that red pencil.
1:10:56
So you can intuitit this immediately, right?
1:11:00
Because there's only one pencil on the table, right?
1:11:02
So it must be like a red pencil.
1:11:04
Like this must be what it's called.
1:11:06
This has the redness is a thing about it, right?
1:11:09
Like so it's a correcting pencil or something like that. Okay.
1:11:12
This is just a thing about language is that when we offer information to people that is outside of what needs to be offered in the situation, we automatically assume that that information is incredibly important.
1:11:27
So there's an old study that they did which was on similarity and it was basically how do you interpret North America is similar to China versus China is similar to North America.
1:11:42
And it turns out that people interpret those two things very differently because there are rules about what is being compared to what.
1:11:50
The son is like the father not the father is like the son as an example.
1:11:54
So, there's things that have to do with social status, size, whatever.
1:11:56
So, let's imagine that I said to you, um, the bicycle is next to the house.
1:12:05
I'm sure that what you're imagining is a normal bicycle and kind of a normal house, right?
1:12:12
But what if I said to you, the house is next to the bicycle, >> right?
1:12:19
It's like, you're like, what's the deal with that bicycle? Is it like a statue?
1:12:23
Like a big >> big statue or >> Right.
1:12:26
So you could think about this. I don't know.
1:12:27
>> Or is it a little model house? >> Right.
1:12:29
That you could because you're like you're like what's going on?
1:12:31
So like the a real life example of this is um in Philadelphia there's a very big clothes pin. >> Mhm. Yeah. And I've seen it. >> Right.
1:12:40
Which is a really big statue.
1:12:40
So if you're saying like I'm standing next to the clothes pin >> pin.
1:12:45
>> It makes sense in the context of that, right?
1:12:48
But it doesn't make any sense if it's a legit clothes pin, right? because you're bigger. >> Yes. >> Okay.
1:12:53
So, so I do think that when we're doing those types of things, we have to take into account prag pragmatics.
1:12:59
So, there's the classic base rate neglect, right?
1:13:04
Linda is mousy and she's quiet and she has whatever and like what's the chance that she's a librarian, right?
1:13:10
And uh and it has to do with this what they call the conjunction fallacy.
1:13:15
>> Yeah, that's what I was actually thinking of, right? Yep.
1:13:16
And but as a linguist what I say is but why are you offering me all this information about Linda? >> Right. >> Right.
1:13:24
Like so yes we don't want to fall for it because it is a way of trickery and you can see that trickery occurring in people who are actually trying to sell you or fool you or something like that.
1:13:36
So we would like not to fall for it.
1:13:38
But it's not that surprising that you are because when you're saying what are the chances that Linda is both right a D&D fan and a librarian or whatever it is right that when you're giving all of this extra detail as someone who speaks the language what's going through your head is why are you giving me all this detail?
1:14:01
It must be meaningful in the same way that you told me that the pencil was read.
1:14:04
my mind sort of automatically goes to if I'm being fed a ton of additional information that they're they're trying to sell me on something.
1:14:16
>> That is exactly right.
1:14:17
>> So, I immediately get a little more suspicious. >> Yes.
1:14:20
So, you intuitively know it. >> Yeah.
1:14:23
>> See, that's the thing because you're like, "What? Wait, what?
1:14:24
Why are you why are you telling me all these completely unnecessary details?"
1:14:28
I I love the overlaying linguistics and the and the different rules of language that I I had never really actually thought about that how they can make things that seem stupid like statistically they are. I love that.
1:14:49
I've never actually thought about that.
1:14:50
>> You know what I think is interesting about it I bet you didn't think the conversation was going to go here. >> I love it.
1:14:55
What I think is really interesting about it is that if you think about the framing effect, >> it's in the same vibe world, right? Yep.
1:15:08
>> And they do recognize the framing effect, right?
1:15:10
So we should understand that there's just like a grammatical effect also, right?
1:15:12
Or a pragmatic effect of like I'm offering you information. Why am I doing that? Right? Where people Yes.
1:15:19
In reality, you should just say 70%.
1:15:22
se a 70% chance it's a lawyer except why are you giving me all this information right >> right like weird so maybe I should be paying attention to it so we they even they understood it when it was like okay a thousand people you know will die of this cancer I've got a new experimental treatment if we give it to these thousand people 600 of them will live everybody's like yes >> if you say 400 will die they're like Oo, >> I don't know.
1:15:56
>> So, they're understanding that, right?
1:15:58
That okay, there's these issues of just sort of how things are framed.
1:16:01
But then, well, we also have to take into account the grammatical context in which you're saying them >> and why why are you being inefficient?
1:16:11
Because that's what you're doing when you're giving the extra information is that you're adding in efficiency.
1:16:15
>> You're being very inefficient. I love that.
1:16:17
I have to this is absolutely >> the strange twists and turns of the conversation that nobody was expecting. There you go.
1:16:27
>> I think that's actually great though because you know I think immediately about Claude Shannon and information theory, right?
1:16:32
And what he said was pretty straightforward, right?
1:16:37
Information is unexpected, right?
1:16:37
And so built into our processing of that is this natural why is Annie giving me this additional information. I love that. >> Right.
1:16:49
So should it change my probability?
1:16:51
Well like in most normal communications. Yes.
1:16:54
That is what I would say. Yeah.
1:16:58
>> Is because you are assuming that I am an honest broker >> who is trying to help you to understand the world in some way.
1:17:04
>> So I don't think it's unreasonable for you to say we're just having a conversation.
1:17:08
conversation. I just told you a whole bunch of extra stuff and it should probably change your forecast now when a politician is adding all this stuff in don't fall for it please right or a saleserson is adding in all of this extra context
1:17:22
>> well you know Shannon himself said the amount of information in a stump political speech is probably zero whereas a the amount of information in a short poem could be kind of off the charts right because of the an his notion the unexpected. But I love the
1:17:37
But I love the way that feeds into why are you giving me this information this linguistically?
1:17:45
Why are you adding red pencil?
1:17:45
There must be something about that particular type of pencil that is meaningful or you would not have been inefficient >> in a normal exchange.
1:17:54
I mean, again, going back to evolution, how could we as humans have survived in social groups that we needed in order to be able to overcome the fact that we're physical weaklings?
1:18:05
If we did not believe that people were generally telling us the truth, >> right?
1:18:09
Well, you have to for social cohesion.
1:18:12
>> Unless you're a psychopath or a sociopath, you default to truth. >> Exactly.
1:18:16
So, if we're just like normal people, I'm not a politician. I'm not a salesperson. I'm not whatever.
1:18:21
Like, we assume we're just normal people.
1:18:23
and I say, "Can you hand me that red pencil?"
1:18:28
when it's the only pencil on there, you're going to assume that I've added that in on for a reason because I'm trying to give give you some sort of information.
1:18:37
Like otherwise, I wouldn't do it, >> right?
1:18:40
>> The reason why I tell people in any training that I ever do, you have to start with the base rate and then you better have a damn good reason >> that you're deviating from it. It better be super good.
1:18:51
I mean, and obviously like in the market this is huge.
1:18:53
People are think dislocations are happening all the time, >> right?
1:18:58
And it's like, why are you assuming there's a dislocation? They're rare.
1:19:02
>> Like, they're really rare.
1:19:02
And you better have a really good reason for why you think that things are so different now. Words to live by, man. >> Yeah.
1:19:11
>> But >> I'm thinking of the uh cartoon from The New Yorker with the guy at the pay phone.
1:19:17
Uh so that dates the cartoon. >> Sure.
1:19:20
Um, and he's shouting into the pay phone, "I don't care what your earnings estimates are. Sell."
1:19:25
And in the background is a burning factory. >> Right. Right. Right. Right. Right. Right. Exactly.
1:19:32
So, you better have a good reason. >> He had a good reason. >> He had a good reason.
1:19:36
He was looking at the factory legit burning down.
1:19:38
So, I I think that you don't you don't want to live in one world or the other, right?
1:19:45
>> The one world is things will always stay the same.
1:19:47
And that goes back to the beginning of the conversation. Yeah, >> right.
1:19:50
Like there's a pace of change, right?
1:19:52
And we don't want to be one of those people who's just like it's all as it has been, it always will be.
1:19:56
That's going to be a really bad place to live.
1:20:01
But you also don't want to be like it's different now, it's different now, it's different now, it's different now, it's different now.
1:20:05
So that's why it's always start with the base rate, assume equilibrium, which just means that that base rate is going to hold.
1:20:12
Obviously, there's variation around it, but that should be your best guess, right?
1:20:15
should be your best guess, right? And this is one of the things that I always caution people about um particularly when we're thinking about for example valuations right like okay you're looking at a company that's valued you know three times higher in terms of
1:20:34
multiple which is a lot higher three times higher in terms of multiple to earnings than any company that's been like it before just because people are like yo we're so excited about this company you have understand that there's just a lot of contagion and social proof that could be happening there. You
1:20:49
You better be able to tell me why.
1:20:53
Why is this company different than all these other companies?
1:20:56
Why does it deserve a different valuation?
1:20:58
deserve a different valuation? And the example that I I give to try to illustrate that is if you were looking at the uh you know price toearnings ratio for Amazon and you were saying well okay it's a retail store you know Walmart is four to six times earnings and here's Amazon you know which is like
1:21:19
25 times earnings or whatever right yeah then you may say well that's just wrong right but if you're like well maybe there's something different about this But you have to understand that yes in that particular case I'm giving you an example where there was something
1:21:34
different right it's not bound by physical space and distribution is different it's online it can do all sorts of other stuff you know so on so forth that it may be actually deserving of that higher uh ratio in that case you'd be correct you also have to understand that under those circumstances you have a bias
1:21:51
toward thinking that it's different so whatever explanations you're coming up with for why it's different you should be suspicious of and what I would really recommend if you really want to get a good answer about it is to ask six very very smart people to answer that question independently. Don't all be in
1:22:07
Don't all be in a room together.
1:22:08
Just have them do it independently.
1:22:10
Uh why do you think this is different?
1:22:12
And then you should actually add some context onto it.
1:22:16
Imagine it's a year from now and the stock is cratered.
1:22:17
Uh why do you think that is?
1:22:20
Why do you think people got it wrong?
1:22:22
Imagine it's a year from now and that that price seemed to be reasonable, right?
1:22:28
Why do you think that is?
1:22:28
What do you think people got right about the price?
1:22:31
And have six really smart people independently answer that question.
1:22:34
And you're more likely to get to whether that's really a dislocation or not.
1:22:40
>> That's actually a good AI prompt, too, by the way.
1:22:42
by the way. rather than tell the AI no matter which model you're using say um what type of people should I ask about this problem >> and then it generates all the different characters attaches different probabilities to them that's another
1:23:00
little trick that is very helpful >> what please attach a probability to this and you get the different actors you get cognitive diversity and you get much closer to an interesting answer from the AI >> because it defaults to the dead middle, right? And hallucinates anything that it
1:23:17
And hallucinates anything that it is inconvenient.
1:23:23
>> Yeah, I would actually add a little color to that depending on the model that you're using.
1:23:28
Uh is that it defaults to whatever is going to get you to use it more. >> Right?
1:23:35
To be fair, that's not true of all of them, but um you know, for many of them. Yeah.
1:23:41
For the majority, for the majority of them, I think is it a perp perplexity that doesn't do that?
1:23:48
>> Perplexity generally does not.
1:23:48
We h we have our own AI lab.
1:23:51
Um and we have them all in there and perplexity generally doesn't do it. Yeah.
1:23:57
>> Um >> so I think that I think that's tends to be better. >> Terribly.
1:24:00
However, you can prompt even the most syncopantish.
1:24:04
>> But that's why we want to teach about prompts to kids, >> right?
1:24:07
>> Because that's the thing is like you have to take control over this. Absolutely.
1:24:10
>> What people don't understand and this is true of any social media use is you have to think about AI the same way uh as social media, right?
1:24:18
It wants you to engage with it, >> right?
1:24:22
It's maxi it's maximizing the objective function for you keeping doing it.
1:24:27
you keeping doing it, which means that it's gonna tell you whatever is going to keep you doing it, right?
1:24:32
So, in the same way that if I see some random like, you know, you see little clips of things all the time on social media where they're clipping out like 20 seconds of an interview, if it's not particularly high stakes, whatever, right?
1:24:48
But if you're gonna change your behavior or your opinion or your something based on that 20 second clip, I would really highly recommend you go look at the whole interview, right?
1:24:58
Like please go look at the whole interview because there is no context around this thing, right?
1:25:04
And I think that you have to view social media the same.
1:25:08
I mean, you have to view LLMs the same way, right?
1:25:10
When you ask it something and it tells you something, you better try to get the whole interview, right? Okay.
1:25:16
only site scientific journals.
1:25:20
Give me the citations with the links so that I can go look at them.
1:25:24
You know, show me exactly where you're drawing this conclusion from.
1:25:28
Uh what if this conclusion that you're giving, what if what you're telling me is completely wrong?
1:25:34
Why do you think that could happen?
1:25:36
I've actually just done this with like the Google AI where I've prompted it twice differently.
1:25:39
I actually um did it recently.
1:25:41
I was trying to find out if like a celebrity was still married to another celebrity and I asked it in a certain way and it was like yes they're still married and then I asked it in a different way and it was like they got divorced two years ago.
1:25:52
Um the actual answer was they were still married which which I actually knew but it was like amazing.
1:25:57
It was within like a second of each other >> you know that it gave me a completely different answer.
1:26:02
And so you you should just act like if I asked in a different way it would give me a different answer.
1:26:07
So what would be the way that I would ask it that might get me a different answer because I want to see the whole clip. >> Yeah. I love that.
1:26:11
The last thing that I want to talk about is another little bugaboo of mine and and that is self- selected samples.
1:26:20
When that book came out, The Millionaire Next Door PE people, this is the greatest book like this is and and I so I read it.
1:26:27
I'm like, well, his sample was completely self- selective.
1:26:36
>> Of course, there's like, you know, it's like how many books are like the five habits? Exactly. Exactly.
1:26:40
You know, this >> from good to great. >> Oh my god.
1:26:51
>> Literally, like I look at I'm not for burning books.
1:26:52
I'm just for Please don't read them.
1:26:56
>> But why does it work so well?
1:26:56
Why did they become best sellers?
1:27:01
>> Can I tell you something?
1:27:01
This is why I'm writing this book.
1:27:04
Because this problem is in the book, right?
1:27:07
And it's not just when people are trying to sell you.
1:27:11
So it's a natural proclivity of us as human beings to say if I want to have a good outcome, I should look at things that the the good outcomes and figure try to figure out why.
1:27:22
It goes to that explanation, right?
1:27:25
So I'm going to look at all these successful people.
1:27:27
I'm going to figure out the things they're doing and then if I do those things, I'll be successful, too.
1:27:32
Now, separate and apart from well, you have to understand there was a lot of luck in there.
1:27:37
Like we Bill Gates could do all the things he does today and if he was born 30 years later, I'm sorry, bud, >> right?
1:27:42
Like I'm sure you're a very smart and wonderful person, but you're not having that result, >> right?
1:27:48
>> And if I ran Bill Gates, if I Monte Carlo Bill Gates, what percent of the time?
1:27:53
>> Yeah, I love doing, >> right?
1:27:54
Like let's Monty Carlo Bill Gates when he's born at the exact same time.
1:27:59
So that it's really hard for people to wrap their head around them again because we're determinist.
1:28:02
So we think you do you're all you will always have the result that you have and that the things that you do will create that result. Right?
1:28:11
Let's go again looping right back to the beginning of the conversation.
1:28:15
So it's this natural thing.
1:28:17
It's this natural thing to look at the survivors and what people again compared to what what about people who do those exact same things >> and die >> who didn't s we have to look at the whole thing. >> Yeah. No, no, no, no.
1:28:30
I completely >> I want to look at 50 years, not five, >> right?
1:28:35
>> So, working with a client and they were like, well, okay, we generally don't like to hire people that we would call just like difficult to work with.
1:28:44
>> Okay, so let's think like you don't really like their ocean score, for example, right?
1:28:48
So, they were like, but you know what?
1:28:50
We looked at our most successful engineers and 80% of those engineers were difficult.
1:28:57
We were looking like our top 20% performers for engineers and 80% of them were difficult.
1:29:03
So they were like, we want to actually start changing sort of signal detection for hiring engineers and we want to add this to to sort of say like if if they're difficult, we want to be more likely to hire them, right? Okay.
1:29:15
So this is what I'm saying.
1:29:15
It's not just someone who's trying to sell a book. Okay.
1:29:17
So anyway, so they say this to me because you know that I I work on that was working on the hiring process with them and I I said, "Oh, that's interesting.
1:29:27
Did you ask the same question about how difficult your bottom 20% engineering hires were?"
1:29:35
And they were like, "What?"
1:29:37
And I said, "Well, I look, I just don't know whether this is a quality of successful engineers or just engineers.
1:29:49
So anyway, they went and did that and of course it was literally the same. >> Yeah.
1:29:53
>> Just 80% were had this quality, right?
1:29:56
If we actually get over our bias, remember I said that one of the things that brings explanatory satisfaction is you think, "Oh, I've learned something that other people don't know."
1:30:04
that other people don't know." if we get over our bias that someone who tends to be disagreeable um actually does better in this job and we should overcome our bias and they're trying really hard and they don't understand that they just wrote good to great right like they don't get it oh
1:30:19
that's the millionaire next door so you know I think that this is part of the problem is we want to believe that we can come up with an answer and the natural way to do that is to say well let's look at the successful ones >> right and you have to again say well I don't want to just look these five years. I want to look at all 50 and
1:30:35
I want to look at all 50 and that's actually going to help me because God forbid I start waking up at 4 in the morning because I think that that's going to make me a billionaire, right?
1:30:46
>> It is one of my hottest buttons.
1:30:46
I I I was sitting here on that couch and I saw one of those listicles.
1:30:53
This is like 2018 2019 and I just lost it. >> Yeah.
1:30:58
And and I like your head big thread on Twitter like don't please please please stop don't do this. >> Yeah.
1:31:07
There's there's a famous example that is like in this world about vitamin E.
1:31:11
So in the '9s it was like oh like people who take vitamin E are super healthy so you should start taking vitamin E.
1:31:19
And then that result was published right and more people were taking vitamin E.
1:31:23
Anyway, when you do a randomized control trial, it vitamin E, taking vitamin E supplements is net bad, >> right?
1:31:31
>> So, the question is why?
1:31:31
And it has to do with the sampling error problem, right?
1:31:34
Like, well, the people who are taking a vitamin E are self- selected.
1:31:37
Y >> So, if they're taking a vitamin supplement, it means that on average they're healthier because they are more interested in health.
1:31:47
So even if you reduce their health a tiny bit without knowing it, they're still going to appear healthier than the average person who's not taking vitamin E >> because of the self- selection bias. >> Exactly.
1:32:00
So you have to do a randomized control trial where you're like, you're taking vitamin E, you're not, >> right?
1:32:06
>> And then let's see if it's net good or bad, right?
1:32:07
And that gets you to the appropriate comparison.
1:32:09
So and then it turns out like vitamin E uh supplementation is actually net bad.
1:32:14
Um, but we only know that, right, if we do if we do a randomized control trial.
1:32:19
And this is how we end up like doing things like taking hydroxychloroquin because there's some rando small sample whatever without a control group, you know, and then people are like, I'm going for it.
1:32:33
And here's the thing is that I'm fine if it's a free roll, right?
1:32:37
I don't care if you're going off a small sample that is suggestive data.
1:32:43
if there are no negative side effects to the thing that you're doing, right? >> I'm really not.
1:32:48
But we know that that's not true of hydroxychloricquin.
1:32:52
Hydroxychloricquin has very very bad adverse effects.
1:32:56
>> Uh and so the people who do take hydroxychloricquin are basically saying I'm willing to bear those adverse effects in order for the benefit that this drug is going to offer me.
1:33:08
Right now there are some things that you can do that really just don't have any downside.
1:33:12
Um, one of them that is very very close to no downside are statins.
1:33:17
Although I just want to say for the record, I'm not a medical doctor.
1:33:20
So, please don't go take statins because I told you to go talk to your doctor about it.
1:33:24
But my understanding as a non-medical professional um is that hydroxychloricquin has very bad side effects.
1:33:31
And statins like can have some muscle weaknesses, some issues that are pretty rare.
1:33:35
It tends to be in people who are sicker.
1:33:37
So, uh, if you're a healthy individual, um, it's relative it's a relatively free.
1:33:44
>> And I' and I've read recently actually just on statins, the the newer class of statins has a lower incidence of the muscle weaken. >> Yes.
1:33:54
And you can take like >> Qanol and things like that in order to resolve it.
1:33:57
But so the point is that if you're on the fence, >> right, and you're like, oh, you know, my cholesterol is high.
1:34:02
Should I take this thing or not?
1:34:04
thing or not? Uh and this I would give this advice for any drug is say well what are the side effects right what are the things that can go wrong and when the doctor says not too much you can be like okay whatever >> right but if the doctor says oo a whole hell of a lot you should be like well can you show me the randomized control trial please
1:34:22
>> exactly >> because I would actually like to understand if this is effective and then I would like to understand if it is effective how effective is it what's the probability of the adverse side effect what's the probability you know that I'm going to get the benefits that I see and Now I can actually say am I lengthening my life or shortening it? Uh so you
1:34:36
Uh so you better go through all of that.
1:34:39
But when the answer is me not too much then I don't really care.
1:34:42
It's actually a similar problem to what you're asking about the millionaire next door.
1:34:45
Well what about all the people who weren't millionaires? >> Right. >> Right.
1:34:50
Did they have the same habits?
1:34:52
Did they do the same things? >> Yeah.
1:34:54
>> What's the probability given that you're doing those things that you is it better than average? Right.
1:34:58
Like there's all these questions that you have to ask of that and you you know you should have been asking that of that drug as well given that there were bad things that could occur from it. >> Yeah. >> Right.
1:35:09
So if somebody says uh if you work out uh let's say if you take a brisk 30 minute walk five times a week you're more likely to become a millionaire then uh go do it because actually independent of any millionaireness uh taking walks is good for you.
1:35:25
Um, but I would really like it if you didn't believe it was going to make you a millionaire.
1:35:28
That's what that's the only thing. But go and walk. Maybe it'll help. I don't know.
1:35:32
But it will help you in other ways.
1:35:35
>> I I I love both the theme.
1:35:35
And when is the book coming out?
1:35:39
>> Well, so the manuscript isn't even due until June.
1:35:41
So I it'll probably it's going to be out in like Q4 of >> 2026 or maybe Q1 of 2027 would be my guess.
1:35:53
So, you have a little bit to go.
1:35:56
>> I I really want you send me the manuscript.
1:35:58
I would love I would love to read it.
1:36:01
>> It sounds like I've loved all your books, as you know. >> Well, thank you.
1:36:05
>> But I love this book because I think it's incredibly necessary.
1:36:11
>> Well, particularly in the information environment that we live in. >> Exactly.
1:36:14
and and I I just think anything that can be helpful to people who are drowning uh that's a good thing with very little downside like your earlier >> I just want to be clear like there are no equations in the book it's literally just hey ask this question and it's it's just a checklist of questions that you can ask that will allow you to avoid um what I always say impaling yourself on of data.
1:36:44
Please don't impale yourself.
1:36:46
>> Well, like back to evolution though, we we we evolved to make stories click. >> That's right. >> It is.
1:36:52
I I you know, when I was selling quant investing, what did I do? I told stories about it.
1:36:58
I And I used to joke, I'm going to tell you stories about why you shouldn't listen to stories when making investment decisions.
1:37:05
Why you But I never put numbers like No.
1:37:08
because like people just like totally glaze over, >> right?
1:37:14
And I think that the important thing is that this is the mind wear that we're born with.
1:37:19
And the idea that you can overcome the types of errors that we're going to make by sheer, well, I know about it now and so I'm fine or whatever. I mean, it's hilarious.
1:37:33
And so you have to start to think about how do I build structure into the way that I'm thinking so that I can uh mitigate the errors that I might be making.
1:37:45
That's you know always been my approach in my writing.
1:37:47
Like there is nowhere in the book quit that I say oh you know about the sunk cost fallacy so you're fine right?
1:37:54
In fact I say quite the opposite.
1:37:58
Instead I say look you have to start making precommitment contracts.
1:38:00
You have to set uh kill criteria which are just stopping rules, right? You have to follow them.
1:38:07
Um you need to write them down on a piece of paper.
1:38:10
You've got to say if I observe this signal, I will stop doing the thing that I'm doing.
1:38:14
Um and that's going to really help you to get to these these decisions to stop things faster because you can't just, you know, the one that was always hilarious to me was investors who would say, "Oh yeah, I know about the sun cost fallacy.
1:38:28
So every morning I just imagine, what if I didn't own these things? Would I buy them again?
1:38:31
And I'm like, well, that's nice, but you do own them, so that doesn't work.
1:38:35
Uh, so if there were no transaction costs, then I would say if there were no transaction costs, I would just tell you, well, actually sell it, >> right?
1:38:46
>> Just sell everything and then decide whether you want to reby it.
1:38:47
Now, obviously, unfortunately, there are transaction costs.
1:38:50
So that becomes impractical.
1:38:51
But um but you can set rules around when would you sell and when would you not that are going to be much more helpful than your nice little thought experiment which is cute but it's not actually going to help you overcome anything.
1:39:04
And I think that with this idea of like you know this jumping to explanation that we do and then getting settled into that explanatory satisfaction.
1:39:12
I can give you this conceptually all I want.
1:39:14
I can tell you that these are errors that we're making left and right.
1:39:17
I can tell you really fun stories about baseball players and orthopedists and vitamin E and education policy.
1:39:26
I actually have one about like school size and things like that.
1:39:27
And I can tell you this all the time.
1:39:29
It's going to be obvious when I tell you this that these errors are being made.
1:39:32
And you're going to be like, "Cool, done. I'm fixed now."
1:39:36
And no, the the book is actually saying you need to literally go through this checklist and you these are the set of questions that you have to ask of any piece of information that you see.
1:39:46
And until you've asked all of these questions, you cannot draw a conclusion from it. >> I love it. I love it.
1:39:51
And can you program your personal AI to make go through everything for you?
1:39:57
>> Well, you should be able to.
1:39:57
And I think that that's actually something that AI ought to be helpful with. Yeah.
1:40:00
You know, you can input a checklist into the AI and then the AI can actually take it off for you and make sure that you're actually going through that.
1:40:08
Now, I wouldn't necessarily trust the AI to answer the question that you're asking.
1:40:13
Um, unless you're a very good prompter of AI and you do it, but it can at least get you to ask the question. >> Exactly. >> Right.
1:40:18
So, uh, which >> and and act as kind of a forcing function. Right.
1:40:21
It's like, >> why is it saying that? Right. >> Right.
1:40:25
Because when you were going earlier about getting people about learning about how much they actually do know about dogs, >> right? >> Right.
1:40:33
It's kind of the same thing, right?
1:40:35
If you have that on your AI, I just kind of think how cool would it be if I had that and I'm reading an article and it's running in the background.
1:40:45
>> Yeah, >> that would be really cool.
1:40:46
>> So, that's actually that type of sort of decision co-pilot kind of thing uh is something that we're actually working on on at the alliance. >> Oh, very cool.
1:40:55
Um, we're very lucky to have had Eric Corvettz um just join the board from Microsoft and he uh he actually uh is working on a he's been working on a tool like that that we're hoping to be able to translate for younger people as well. >> Fantastic.
1:41:12
Annie, I always have so much fun when I'm talking to you because you know so much about so many things.
1:41:17
I love the linguistic thing.
1:41:19
I'm gonna >> That was just That was >> I'm gonna be just thinking about that for the rest of the night. >> I like to surprise.
1:41:25
I like to I like to surprise.
1:41:29
>> Well, you get one last chance.
1:41:29
I don't know if you remember from the last time you were on because it was a long time ago, but we ask we're going to wave a wand.
1:41:36
We're going to live in magic world.
1:41:38
We're going to make you empress of the world. You can't kill anyone.
1:41:40
You can't put anyone in a re-education camp.
1:41:46
But what you can do is speak into a magic microphone and you can incept the entire population of the world.
1:41:51
Whenever their morning is, they're going to wake up with the two things you're about to incept for me and our audience that they're going to wake up and they're going to say, you know, I I' I've never really acted on these ideas, these morning ideas that I get, but this time I'm going to be different.
1:42:09
these two ideas that you've incepted in them, they're going to think are their own and they're going to act on them.
1:42:16
What two things are you going to incept into the >> Oh my gosh, this is so hard.
1:42:21
I don't know what I answered last time.
1:42:26
If I could actually get people to think probabilistically, like, please, that would be so great.
1:42:32
Imagine how much better it would be to watch the news, >> right? >> Right.
1:42:38
Like, oh my gosh, that'll be so much better.
1:42:42
Just view the world as it is.
1:42:44
Like, it's probable, isn't it?
1:42:48
>> First of all, you won't be so freaking surprised and probably not as angry all the time, I think, would be good.
1:42:52
And I think the first explanation that comes to your mind for what you see is not necessarily the right one, >> right?
1:43:01
Like, you have to be much more skeptical. >> Yeah.
1:43:03
Um, and I think that that would I I think that that's a really big one because we see that all this information is floating around and you just think you know what it means.
1:43:11
It's like unless you actually know how how to ask a question of that data, you just don't think about how many political arguments are based on the exact same piece of information and you're interpreting it one way and I'm interpreting it another.
1:43:23
And we never stop to think like, well, maybe we're just not interpreting it right.
1:43:29
>> Yeah, maybe we're wrong. >> Maybe we're wrong.
1:43:30
I'm just gonna I just want to add a third one in just but this is not me.
1:43:34
I'm I'm going to attribute this to somebody else. >> Okay.
1:43:37
>> I I had a conversation with Lorie Santos who's wonderful.
1:43:39
She she does all you know she's the happiness person over at Yale.
1:43:43
I asked her like so sort of what's the best thing you think that somebody can do um to be happier?
1:43:49
And her answer was, "When you're with other people, don't even have the phones anywhere near where anybody can see them because just having it here makes the conversations worse."
1:44:03
Um, which I haven't I've I've actually implemented in my life.
1:44:05
But the one that really changed my life was get out in nature.
1:44:10
I'll go out with my dog for two hours and I do not I literally don't touch my phone.
1:44:17
I don't think anything has had such a profound impact on my happiness more than that.
1:44:24
>> Now, you got to tell me what kind of dog do you have?
1:44:27
>> Oh, I was trying to hide it because maybe people will take my class on Maven and I know it's fine.
1:44:31
I have a mini Bernadoodle named Otis. >> Okay.
1:44:35
>> Who is uh one of the loves of my life.
1:44:35
I think that my children and husband might say the love of my life. I'm not sure.
1:44:43
But um >> Annie, thank you so much.
1:44:44
It's so great to see you and it's just so much fun to do in person.
1:44:49
>> Yeah, it's so nice to do it not on Zoom.
1:44:52
I was so happy that we could make this work. >> Yeah. Thanks for coming on.
1:44:55
>> Well, thank you for having me. It's always a joy.