today I am speaking to Philip tetlock who quite simply is one of the greatest social scientists in the world up until now we've been doing conversations with Tyler face-to-face but for obvious reasons Phillip is in Philadelphia he teaches at University of Pennsylvania and I'm here in Arlington Virginia let's just jump right into it first question Phillip with our forecasters do we want accuracy or do we want them to be a kind of portfolio to
make us more aware of extreme events and possibilities I think we want a lot of things from our forecasters and accuracy is often not the first thing I think that we want for we look to forecasters for ideological reassure me reassurance we look to forecasters for entertainment and we look to forecasters for minimizing regret functions of various sorts so that we would really regret not having anticipated X Y or Z so we want
to pump up the probabilities of those things but if we take say the corona virus if we had had a few more extreme nuts who were maybe wrong most of the time but insisting that we needed to fear the next pandemic wouldn't we have been better off with that kind of portfolio and thus we don't actually want more accuracy from our forecasters well in some sense we already did have that portfolio there were it was a main mainstream position among
epidemiologists for the last 20 years or so that you had a recipe for a disaster David Epstein the guy who recently wrote range there very interesting guy you may have had him on your show I don't know but he recently encoded himself from a 2007 newsletter that he wrote and and he said something like the the presence of a large reservoir of SARS like viruses among them horseshoe bats combined with the culture of eating exotic meats is um
it's a time bomb and that was in mind that was in microbiology books I was you know in the first decade of the 21st century that was it was common knowledge and indeed even before SARS one even before the first floor first SARS outbreak some that the D me ologist to caulk Oviatt nineteen stars too but even before SARS one epidemiologists were acutely aware of this so it's not as though we didn't have it in our portfolio we did
so those forecasters maybe weren't entertaining enough isn't then the margin we want to work on to make our better forecasters more entertaining and not more accurate yes No well this ratio isn't gonna be great they can assure you of that because there are plenty of people who are naturally more entertaining than epidemiologists but maybe the whole portfolio needs to be more vivid rather than trying to fine-tune the accuracy of
particular parts of it right well you have lots of people competing in the marketplace of ideas for attention and that that's a hard competition for scientists to beat what do you think of the argument that science only exists at
all because most scientists are overconfident that if they were rational Bayesian they would just latch on to the opinions of the smartest and best trained people before them that there's only progress precisely because people
are making forecasting mistakes right so you hear version two that argument I suppose on Wall Street as well sure I deal of truth to it I certainly have been guilty of overconfidence in many junctures of my career thinking I'm going to be able to take on things that looked impossible and often turned out to be impossible both most projects that most scientists embark on I think don't succeed does take a certain amount of
quasi irrational persistence as with Columbus right or the founding of the United States arguably was irrational to break away from the British Empire right which was doing pretty well back then it seemed like a risk seeking move but say I set up an alternate research program and I sought to take forecasters and a make them more entertaining and B maybe I'd give them uppers so they were more overconfident I mean would that do the
world's good well you could certainly have induced the epidemiologists who are worried about the horse Bath's in central China or other or other possible sources of zoonotic viruses you could certainly have induced them to pump up their probabilities and you would of course started to run into a problem of crying wolf if they'd been saying there's a 30 40 50 percent chance of of a viral leap into human beings each year and it did and it didn't
happen didn't happen didn't happen people would you get a crying wolf effect right right how do you think about financial markets in relation to your work on super predictors our financial markets in essence super predictors to begin with work in super predictors on average beat financial markets oh well you know we play with prediction markets in with the work with the intelligence community even going all the way back to Admiral Poindexter
in the original DARPA effort to launch prediction markets inside the intelligence community I think your colleague Robin Hanson was involved some of that work out of brown 2000 obviously 2003 2004 and that in that range we've also been working with the prediction markets in parallel with forecasting tournaments and there are pros and cons to each method of eliciting judgments but the IC the intelligence community
doesn't let us turn those markets make those markets deep and liquid the way they are on Wall Street people are essentially competing for reputational points the way they are in forecasting tournaments the monetary prizes are
either small or non-existent so most economists I think would not consider that to be a very robust test of the efficacy of prediction markets and then when prediction markets fall Stewart and they do typically not perform quite as
well as forecasting tournaments when they fall short it's hardly a decisive rebuke of the market mechanism for eliciting forecasts there are there are lots of very powerful institutional actors like Goldman Sachs and so forth that are continually trying to do exactly what you described it's a it's an it's an ongoing process I don't need to hear an economist I don't you tell you that okay there are implicit prediction markets in
say the coronavirus prices of airline stocks right and those are very liquid they've been very thickly traded lately they have indeed if you took your 10 best super forecasters and brought them into the hedge fund people at Goldman Sachs and you all sat down together who would be teaching whom it's it's an interesting experiment and and you know the person okay the my project manager from the first set of forecasting tournaments
teri Murray founded a company good judgment incorporated which does things like that so that's a proprietary venture and you know we probably want to talk to Terry about how successful or not successful they've been in doing
that I think they've had some success I think it's extremely hard to do that it's non-trivial i I think there's a good deal of similarity in the cognitive ability cognitive style profiles for super forecasters and the kinds of
people you see on the staffs of goldman sachs and it would be a tight race what about us the sports betting market do you think there are inefficiencies in that because they don't have enough super forecasters I'm not an expert on sports maybe I'm better to talk to Nate Silver about that but let me put the question more generally there are many markets out there which predict something sports betting markets are simply the most
obviously most explicit about prediction so if there was something the world didn't know about prediction already those markets should be inefficient yes or no why would you think the answer would be yes well let's say that people favored the home team too much so too many people might bet on the New York team's the Los Angeles teams and then the odds would be skewed so if there's a bias in people without super forecasting
techniques we would expect sports odds to somehow be off or at least they would have been off before your work was published well or well our arbitrage predate predated super forecasting well but isn't arbitrage itself super
forecasting right people are arbitrage in a forum yes on the basis of some set of information I think that's better so all these markets out there do you think they are without you already the best available super forecasters the term
best is that it's a term I'm just not comfortable with I I'd rather doubt that throughout the optimal forecasting frontier at the moment but they're other you know they're often probably fairly close and is there is there room to incentivize people to predict the market well that's one of those paradoxes that economists are written about aren't they when people predict out too many decimal places do you think that's absurd or do
you think it's useful what's the optimal level of granularity for different categories of forecasts I think for the kinds of things we were looking at in the IR original forecasting tournaments the geopolitical events like how long the Syrian civil war would last or Russia do in eastern Ukraine things of that sort yes it would be absurd to go to three or four decimals point originally the National Intelligence Council which
synthesizes a lot of intelligence analysis originally they only distinguished five degrees of uncertainty and they didn't they didn't put numbers on it more recently they've moved to seven degrees of uncertainty and they do put numerical ranges on so somewhat likely represents a certain probably the range now in our work we've explored how how granular was it's been the best forecasters are doing various rounding experiments where we round their
forecasts off to the nearest tenth that kind of thing and see whether or not if it's pseudo precision if when they adjust them and they move from point zero point 6 to point six live for example if on average that doesn't you know improve their accuracy we would conclude that that they can't achieve that level of granularity our best statistical estimates are that our forecasters for the types of questions the intelligence community often poses
can distinguish between ten and fifteen degrees of uncertainty which is considerably more than the seven they think they can now a lot more than the five they thought they used to be able to distinguish but how useful is it to be able to distinguish varying degrees of uncertainty is going to hinge on the kind of game you're playing if it's poker you know you might well when someone who's very adept at distinguishing
things to say three decimal points if you could take just a bit of time away from your research and play in your own tournaments are you as good as your own best super forecasters I don't think so I don't think I have the patience or the temperament for doing it I didn't make I did give it a try in the second year of the first set of forecasting tournaments back in 2012 and I monitored I wanted to the aggregates we meet we had an
aggregation algorithm that was performing very well at the time and it was outperforming ninety-nine point eight percent of the forecasters from whom the address the composite was derived so if I simply had predicted what the composite said at any at each point in time in that tournament I would have been a super super forecaster I would have been better than ninety-nine point eight percent of the super forecasters so even though I knew that
it would be it was unlikely that I could outperform the composite I did research some questions where I thought the the composite was excessively aggressive and I tried to second-guess it and the net result of my efforts instead of finishing in the top no point oh two percent or whatever I was I think I finished in the middle of the super forecaster pack so that doesn't mean I'm a super forecaster it just means that when I tried to make forecast
better than the composite I degraded the accuracy significant but what do you think is the kind of patience you're lacking because if I look at your career you've been working on these databases on this topic for what over thirty years that's incredible patience right more patience than most of your super forecasters have shown so is there some disaggregated notion of patients where they have it in you donut
yeah they have a skill set I mean and it mostly in the most recent Rena mess we've been working on with them that this becomes even more evident that they're there their willingness to delve into the details of really pretty
obscure problems for very minimal compensation it's quite quite a quite extraordinary they are intrinsically cognitively motivated in a way that it's quite remarkable what how am i different from that I guess I have a little bit of
attention deficit disorder and my attention tends to roam so I've not just worked on forecasting tournaments I mean I've been fairly persistent in pursuing this topic since the mid-1980s you know Mac would even before Gorbachev became general party secretary I was doing a little bit of this but I've been doing a lot of other things as well on the side and so my my my attention tends to roam I'm interested in taboo trade off some
interest accountability there various things I've studied that don't quite following this burger doesn't that make you more of a fox though you know something about many different areas I could ask you about antebellum American discourse before the Civil War and you would know who had the smart arguments and who didn't right well I would know who has arguments to take the more integrative Lee complex forms on the one
hand on the other hand and then synthesis whether you want to consider those arguments smarter or not is another matter but yes it is I said I suppose that's fair I mean I've always resonated a little more to the fox's end of the Hedgehog's but I think when you look at the great achievements in science they often come from Hedgehog if you look today at the on-going debates about coronavirus and what will happen and I mean now the debates
amongst the smart people the people you respect what is the mistake you see them making the biggest mistake oh you want me to be an amateur epidemiologist here no mistakin reasoning you don't have to give your numerical estimate procedurally what are they not getting right well is it a mistake if you're a public health expert who feels that there that one mistake is much worse than the other it's much better to
overestimate the threat of the virus and to underestimate it because you have to influence public opinion and public behavior there you're not forecast and you're engaged in manipulation social influence so you mean this thing comes back to original question about you know what do we want from our forecasters and accuracy it's only one of the things we want from them we look to forecasters for a lot of things basic to inspire our
confidence to inspire fear and so forth if we're trying to estimate how much people cut back on their risk-taking behavior because they're afraid of the virus or the group of people best suited to do that epidemiologists some other
social scientists or your super forecasters maybe even economists right we study elasticity's why should it be the epidemiologists I think you'd want an interdisciplinary team III think there's a good diversity is one of these
words has been reduced to a cliche but I I think we have found in our work that cognitive diversity helps and it helps in certain quite well-defined ways if you want to create a composite that out predicts the vast majority the super forecaster is a good way to do it is not only to take the most recent forecast of the best forecasters in a domain but it's also to extremize that forecast to the degree that people who normally
disagree agree with each other and when you have that when you have convergence among diverse observers that's a signal that the weighted average composite is probably too conservative and you should extreme lies does the team the diverse team have a CEO someone in charge not in this case no it's done purely statistically but would you put someone in charge and maybe the person in charge would implement that statistical algorithm
right but who's the person you would put in charge of the team an epidemiologist yourself your best super forecaster Bill Gates that's a matter of managerial skill and I would say you know going back to my old one of my old dissertation advisors at Yale 40-plus years ago or Janis on groupthink I would pick a leader who knows how to shut up and not reveal opinions at the beginning of the meeting and knows how to listen and which group
of people do you think that best describes I think a lot of good executives have the intuition that you get more out of a team of forecasters or problem solvers if you initial if you elicit independent judgments initially that are
uncontaminated by conformity pressure and then you create an environment which ideas can be freely critiqued before lifting the veil of anonymity and letting people see who's taking which positions do you think having machine
learning and artificial intelligence has made us much better at forecasting things right now socially range of things for sure sure at the micro level but social events whether it'll be a recession how many people will die from the corona virus will be settle Mars well I aren't they just ran a forecasting tournament called hybrid forecasting competition in which they patted pitted algorithmic broaches and human approaches and hybrid approaches
against each other and Wow you know I should love IR to speak for itself about how well as programs worker don't work I I don't I don't think there's a lot of evidence to support the claim that machine intelligence is well-equipped to take on the sorts of problems that the intelligence community wanted to have answered when it runs the forecasting tournaments has been running with with our research team the things
like the Syrian civil war or Russia Ukraine settlement of Mars though though these are these are events for which base rates are elusive it's not like you're you're screening credit card applicants from visa educate your
don't machine intelligence is going to dominate human intelligence totally machine intelligence dominates humans and go and chess it may now dominate human humans and poker I don't know that we're what the state of the art is quite
there yet but you know it is in their Starcraft or you know whatever the next thing that Dennis Thomas is going to conquer so but no I don't see evidence that those approaches work in the domains that we we we study with the intelligence community so do you think the hybrid man-machine proaches are overrated no I don't think it's like I think it's a matter of it's very domain-specific it's it sounds like a great idea who
could be against it but the devil lurks in the details and it doesn't deliver as automatically as you might hope its trench warfare do you think the world as a whole is becoming easier to predict or harder to predict and again I mean social events
I'm not sure there's a definite trend one way or the other are you hear a lot of talk a lot of claims that there are but you know if you look back in the 20th century there certainly were lots of major pockets of
unpredictability no it's not it's not clear what if I say again current events aside but it seems easier to predict there hasn't been a world war since 1945 there's been steady economic growth in most parts of the world more peace isn't that easier to predict you just predict 2 to 4% global economic growth and you pick up a fair amount of what's happened since 1950 indeed so well simple extrapolation algorithm is how or historically are
hard to beat and if you you know we just were running a Cove 819 mini forecasting tournament right now and they're proving to be hard to be the the skill of course is when when when to when to alter the trend the weather to accelerate it or to decelerate it or to change direction and you have a personal intuition on that III think humans have been repeatedly humbled in competitions against simple statistical algorithms going back to
Paul Neil a Mis little book on clinical versus actuarial approaches to predicting in medicine and psychiatry so I would say be humble now there's some of your early research that if I read it properly suggests that making people accountable leads to more evasion and self-deception on their part are you worried that you work with pundits by trying to make them more accountable will lead to more evasion and self-deception from them or how do you
square early tetlock and mid period too late that lock it's actually not not not too difficult in that particular case it really depends on the type of accountability tournaments create a very stark monistic type of accountability in which one thing and only one thing matters and that is a accuracy and you get you get no you get no points for playing to you it's for being an ideological cheerleader and pumping up
the probabilities of things that your team wants to be true or downplaying the probability these are the things your team doesn't want to be true you take a reputational hit so the incentives are very unusually tightly aligned to favor accuracy that's extremely unusual in the social world most forms of accountability occur in organizational settings in which there are lots of distortions at work and the rational
political response for a decision-maker located in most accountability matrices and organizations is to engage in a mixture or strategic attitude shifting toward the views of important others or as you put it the evasion
procrastination and so forth but given that when a pundit is on say the evening news there's not a little box at the bottom it gives the tetlock score of that pundit correct so most people don't know the actual record so given the out
there somewhere is a measure of how good bad the pundit is are you worried that in a sense you will make those pundits run further away from objective standards precisely because they do poorly by them no if we can make them
run much further away from objectives tangi said they already are but you know it that that's a very interesting interesting point about forecasting tournaments and I look at the kinds of people who are attracted to participate
in I mean we at the very outset I mean I invited lots of big shots to participate in forecasting tournaments and they turned me down they've repeatedly turned me down at a very interesting correspondence of William Safire in the 1980s about forecasting tournaments we could talk a little bit later but the upshot of this is that young people who are upwardly mobile see forecasting tournaments is an opportunity to rise
old people like me and aging baby boomers types who occupy a relatively high status and cite organizations see forecasting tournaments is a way to lose you know the best if I'm a senior analyst inside the intelligence agency and I'm on the National Intelligence Council an expert on China and the go-to guy for the President on China and some upstart R&D operation called lie orifice says hey we're gonna run these forecasting tournaments in which we
assess how well the analytic community community can put probabilities unless even thing is going to do next and I'll be on a level playing field competing against 25 year olds 65 year old how how am I likely to react to this proposal to this new method of doing business it doesn't take a lot of empathy or bureaucratic imagination to suppose I'm kind of try to nix this thing which nation's government in the world do you
think listens to you the most you know I may not know right I might actually say you know III suppose the the most prominent political fan I had with the mome is probably one of the senior advisers to Boris Johnson Dominic Cummings who recently caused a bit of a stir in the UK by appointing a super forecaster who had written some blogs that people interpreted as misogynist or racist or fascist or eugenicist or I don't know
some mixture of all those things but I think his name was an abuse of Biscay as a young man 24:25 but kind of the kind of person young people are attracted as I said before it's the forecasting Germans I'd say it's a fast track toward upward mobility music I have all the high status people making vague verbiage forecasts people like me but do you think Cummings actually is influenced by you because as I understand what he's doing correctly
or not he thinks he knows a bunch of things that other people do not and that seems somewhat non Pett Lockean right so maybe part of his portfolio a video logical Armour but maybe he's actually very non pet Lockean and it's the people in Singapore who are your true fans well there are some people in Singapore too and that's an interest it's an interesting place you should mention that interesting you to bring that one up but
I well mr. Cummings and mr. Goh both separately brought up my work at various points and during the brexit brexit debate and Michael Gove at least in the UK famously said that Britain has had enough of experts I don't know if you
remember that that of course particular quote but and he was thinking of well he invoked a support at least for that position x-rayed political judgment in which portions of that book you know compare subject matter experts to
minimalist statistical baselines like you know extrapolates translation algorithms and the answer was often no so he said the goal was raising the point though you know we're one of these guys where these guys get off making these confident predictions about the consequence of the brookside when the best empirical evidence would suggest are probably not materially more accurate than simple extrapolation algorithms so it was yeah that was
brought up it was brought up for a political reason he had a political point to make and I think that's and Dominic Cummings had to play as a political point to make us well I feel he fears that I think parts of the civil service are hostile toward brexit and want to undermine the the Boris Johnson administration objectives now I think this comes to something deeper now it's it's not just about the UK it's about the intellectual fissure
that exists between social science and conservatives that most social scientists are liberal and conservatives are weary of advice from social scientists I think we may have partly paid some price for that in this epidemiological and in this in them to put it kindly the slowness of the Trump administration responds to Co bid 19 yes now you brought up Britain if we look back at speeches in the British House of Commons who was giving the most
cognitively complex speeches I can't you know Bob Putnam collected those data that I reported a study reported that study in 1984 but Bob Putnam collected the original data and he reported in a book beliefs of politicians from which the 1970s and I think those data were based on interviews with members of the British House of Commons they were not they were not speeches they were interviews confidential interviews about
them got access to and put a lot of work into obtaining and he shared them with me and I I used them as grist for a small research program I was running in the 1980s on cognitive style and political ideology trying to tease apart the rigidity of the right versus the ideologue hypotheses and who sounds the smartest from that period in that period it was a mixture of moderate labor moderate conservative it was this
interest that did better it was slightly left left shifted do you think we can draw any inferences from that about politics should we have more faith in the people who sound smarter or not at all well I think that particular measure an integrative complexity is is is I think it does have some correlation with with forecasting accuracy but I think you're all you're picking up something more than just forecasting accuracy you're picking up what I
value pluralism you're picking up a tendency to endorse values that are often in conflict with each other so the more frequently that you as a political thinker confront cognitive business between your values your value
orientations the more pressure you are to engage in integrative lis complex synthetic thinking when you're valuating more lopsided it's easier to engage in what we call simpler modes of cognitive dissonance reduction like denial or
bolstering and spreading spreading of the alternative so we downplay one values push up the other value and make your life stress-free but some value at some ideological positions at some points in history require more tolerance for dissonance and some people are more inclined to fill those roles and you think those politicians are also likely to be better forecasters you know we're not talking about huge effect sizes here
integrative complexity I think that fluid intelligence is probably a more powerful predictor but integrative complex a combination of fluid intelligence an integrative complexity I think does does boost forecasting accuracy yes and if you were running the CIA those people who are pluralistic and the man are you just outlined would you promote them more rapidly now of course the CIA bear in mind is by statute supposed to be value neutral it's not
there they're supposed to be feeding impartial apolitical advice just just don't believe that right well that is supposed to be the division of labor here now is it is it and forecasting tournaments are one I think one reason they may be interested in forecasting tournaments it's because forecasting tournaments incentivize people to do one thing and only one thing and that's accuracy you don't you don't get points
for skewing or your judgments toward your favorite cause you take a hit on the long term by doing that if you were in charge of the CIA and had a free hand how would you reform it wool boy well there's a long history to efforts to reform the CIA you know going back to 1908 was founded in 1947 there been various efforts since then people have been unhappy with the CIA for many reasons over time Vietnam being a big one but but there
are lots of other reasons that people have expressed unhappiness most liberals and conservatives have the various points been unhappy with with the performance of the intelligence community in 2001 it came to a team to a kind of a crisis point I think and then there was a commission to reform intelligence analysis and and after W in WMD Fiasco in Iraq that the pressure grew even more one reason why we're even talking right now today is that the
intelligence community was forced essentially by the recommendations of the of the Reform Commission to take keeping score more seriously to take training for accuracy and and and monitoring of accuracy more seriously and I think that they create that's when they created the intelligence Advanced Research Projects activity which is the Rd branch housed within the Office of the Director of National Intelligence and its job is to support innovative
research that will improve the quality of intelligence analysis where you know that can be defined in various ways but what accuracy is certainly an important component of that but it's not just it's not accuracy with a liberal skew or conservative skew is supposed to be just playing just the facts ma'am interesting question so choose to predict right that reflects values obviously I'm sorry which questions they choose to predict
that reflects values whenever bureaucracy tells me the value free I start getting more suspicious very quickly right I never believed them it might be okay for them not to be value free but the cynical response is the correct one here well indeed you know there is the old expression there's no view from nowhere there is no such thing as pure value neutrality that doesn't mean it's not something worth aspiring to but your
what your right the the value is off even if you had a perfectly objective forecast in tournament system if you had people there generating the questions and have a promoting a political agenda you could skew the results I think
that's one of your points right now in the middle of all these discourses we have a segment called overrated versus underrated and I'll toss out a few names ideas and you tell me if you think they're overrated or underrated as
that okay Philadelphia the city of Philadelphia overrated or underrated I got endless grief when I my wife and I had decided to leave Berkeley and move to Philadelphia they people thought that we were borderline insane but we left for further for very personal reasons not and without going into a web with what those were I would say Philadelphia's been a moderately pleasant surprises it's a city that has many many many problems but it's not
it's not as bad as the people that people in Northern California thought it was Tolstoy you know I haven't I read a little bit of tall story and I've seen a number of films and I know some of the short end versions and I know that Isaiah Berlin had a hell of a time class of like tell stories Hedgehog or a fox but I don't think I'll pass on them John Cleese he's commented on you you're allowed to comment on him right well um
he gave me I had a wonderful time as a kid watching Monty Python so I think and also Fawlty Towers so I really enjoyed his enjoyed his comedy I haven't followed him since then but when I was younger I thought he was absolutely hilarious and brilliant and I appreciate the flattering things he said about my work the television show The Sopranos I fell for it and I James Gandolfini I fell in love with his performances and and then when
he in one of the last things he did is he played leon panetta in zero dark thirty and there he is you know eliciting forecasts from people in the CIA about whether Osama bin Laden isn't that compound and about about is he there or isn't he effing there great wonderful I didn't ever knew him but I think very highly of his work the threat of terrorism do we overrated or underrated in the United States that's a very difficult question because
of the tail risk aspect to it if you looked at the number of people who died from terrorism versus other causes it would seem that the amount of money we spend on suppressing terrorism would be disproportionate but the the tail risk complicates that a lot what is your favorite movie I don't have a hierarchy like that I'm sorry nothing comes to mind what's the movie you've seen the greatest number of times that you can
count over and over I did see myself coming back just very recently a lot last week and the quarantine period to west world it's not a movie it's a series of course but um III think that is quite quite I think the first two seasons are quite quite brilliant on historical counterfactuals by what year do you think the ascent of the West was more or less inevitable well I I have you inside information here we did it we did a survey of some
very prominent historians we reported it in that book I'm making the West we put a part of it anyway also in an article on science review III I think that the if you looked at the just the unweighted average of judgments for the median I think was probably around 1730 1740 and you think after that it was not very contingent that would that would have been you see I'm I'm not a historian of the West I mean I really no I'm simply
reporting the news here and is there any great hinge of contingency that you think about looking backwards like oh my goodness if there hadn't been a Reformation or if there hadn't been a Council of Trent or or what well I I you
know I'm I'm a fan of Steve Pinker I think the Enlightenment was a big deal because it's got people thinking more rationally yeah when terms of science and that had huge huge spillover effects there have been enlightenment in China
ago could the Chinese have created certain types of technologies without science what about if they hadn't scuttled their Navy around 1400 there all those sorts of counterfactuals and how yes have necessary or contingent contingent do you think it is that we keep on thinking in enlightenment like terms is it once you're locked into it it keeps on going or is it like good government that you have to renew it every generation or two well I see the
work I'm doing is very much in the spirit of the Enlightenment public transparent standards of evidence for judging subject matter expertise I think one of the great challenges of our time is it's striking the right balance between democracy and technocracy and I think the the fissures that have emerged it's not just you know a conservative or very skeptical social science and apparently some parts of biological
science too just I think very unfortunate but you see this kind of some reflexive skepticism toward science on the left as well so I mean how do you manage DeMott the relation between small d-- Democrats and technocrats in in a society in which expert guidances is increasingly crucial what are you learning from playing the game civilization 5 or at least watching others do so but that's a good example actually of why I'm not a super
forecaster I don't play civilization 5 but civilization 5 was one of the simulations that I our pitch owes to feature in his counterfactual forecasting tournaments under the rubric of focus and if any of your listeners are interested in signing up to be forecasters for focus we still have one more round round 5 and we will be recruiting people but but I'd again it reflects my temperament I don't have the patience for a game like sieve fine that
requires a lot you know there's a forecasting it's all inevitably a mixture of fluid and crystallized intelligence and you have to invest a lot of energy into mastering a game like like like civilization 5 and I suppose as people get older they may become less likely to make those kinds of cognitive investments I mean it becomes more and more essential as I get older I think to focus on the things where I have a have
a real comparative advantage the best chess players are all young right yes we know this clearly yeah I mean it's interesting the domains in which child prodigies emerge music and chess and math if we take super counterfactual lists and super forecasters to those two groups basically overlap or how do they differ I think they have to be rather intimately connected although disentangling this one is going to be really really hard and it's one of
the things I do want to dedicate a few years of my life to doing now obviously when you say someone a good counterfactual Iser it some people shrug their shoulders and they say well how are you possibly going to know whether you know you would have gotten that undo the assassination of the Archduke in 1914 you undo World War 1 undo Hitler do you undo World War two you make Kennedy grouchier during the Cuban Missile
Crisis you triggered World War three you've got these sorts of arguments that are essentially unresolvable you can't rerun history you can rerun civilization 5 but you can't really run history so that makes counterfactuals a place where we're ideologues can retreat they can make up the data that make up whatever facts they want to justify pretty much whatever would know no matter how about how bad thing no matter how bad the war
in Iraq went you can always argue that things would have been worse if Saddam Hussein had remained in power so you have these counterfactual you have these factual and counterfactual reference points that people use and debates
implicitly to to make to make rhetorical points so Kennedy so a lot of people a part of what attracted to me to counter for counterfactuals was hey how important they are in drawing any lessons from history be how important
they are and policy arguments and see how unresolvable they are now one of the things I think we're hoping to do in the focus program is to develop some objective metrics for identifying people and methods of generating probabilities that produce imperious superior counterfactual forecasts and simulated worlds in which you can rerun history and assess you know what the probability distributions of possible worlds are
well it turns out you know you get World War 1 37 percent of the time even if you undo the assassination of the Archduke and you get something like World War 2 you can you see where we're going so we're hoping that when one results of focus will be to help us identify people and methods that generate superior counterfactual forecast and domains where there is a ground truth the next task will be to connect superior
performance and simulated worlds to superior performance in the actual world and this is where things get tricky of course because in the actual world we don't have these some the ground truth so if I ask you a counterfactual question of the form you know if NATO hadn't expanded eastward as far as it did in 2004 into the Baltics NATO us NATO so nato-russia relations would would be considerably friendlier than they are now our chess players good
forecasters but do want to go with this for a second I'm sure sorry yeah no I don't wanna stop keep on going we've got a counterfactual there we can't rerun history we don't know how our relations with Russia would be if we had they don't hadn't gone into the Baltics right sure would have gobbled up the Baltics maybe maybe Russia would be friendlier and feel less threatened you know people have more hawkish or more double models
of Russia and those mental models predispose them to give you certain canned almost ideologically reflects that answers to those counterfactuals friend right now you so you can but you can measure what people's beliefs are in the counterfactual and then you can measure people's beliefs about conditional forecasts that are logically connected to the counterfactuals and kind of a Bayesian entrance network so you can measure you know what yeah if I
know that you you you think that the Russians would be every bit as nasty and snarly yeah even if we hadn't moved it moved into the in the Baltics it might even be nastier it's probably a fair bet that you're also likely to think it's a good idea to increase arms sales to the Ukraine ratchet up sanctions on cronies and so forth so we can identify the the counterfactual belief correlates of more or less accurate conditional
forecasting and in that sense you can indirectly validate or invalidate you can render more or less plausible certain counterfactual beliefs that's that that's the longer-term objective of this research program is to it's not just saying we're not playing civilization five for the sake of sharing getting better at civilization five we're their ultimate goal is to link the sophistication of kind of factual reasoning about the past to the
subtlety and the accuracy of conditional forecast going into the future another thing you should observe by the way if people are becoming better counterfactual reasoner's is you should observe less ideological polarization in their counterfactual beliefs yeah so yeah just the counterfactual belief should become as ideologically depolarized as conditional forecasts are just playing World of Warcraft a lot help you become a better forecaster or
playing chess I don't have any evidence bearing on either of those things but you know if there's a third variable problem they're there but you get used to a test right you know if you lose there's very little self-deception this is true I mean the people who do well in these sorts of things often like games like that venture capitalists when they try to spot talent in others do you interpret their behavior in terms of a
super forecaster model someone like Peter Thiel he found Mark Zuckerberg Reed Hoffman Elon Musk he's a kind of super forecaster how do you super forecast talent and other people well it really helps to be working in an environment in which super bright people are not super rare it's very very hard to forecast to it to identify talent when when when the base rate falls below one in a thousand one at ten thousand a
hundred thousand you're looking for a needle in the haystack but there's this the great advantage of antigen Chur capitalists in Silicon Valley have is that the talent pool is relatively rich so there's quite a few flaky people who come by seeking their money for sure but but their odds of success are significantly better then you then they would be if there's working from a population base rate and of course they can tolerate a lot of
mistakes because just a few hits what will pay for a lot of false positives but some do much much better than others right Mike Moritz Peter Thiel right the question is are they doing better because they have better social networks and doing better or they have better judgment do you think super forecasting as a technique also applies to super forecasting how people will do I'm sorry super forecasting as a technique for
predicting the future it also apply to predicting how successful people will be yes I I think everything we know from the overlap between super forecasting and intelligence and you overlap between intelligence and success in various in
many different professional lines of work which suggests that I was almost certainly true who first super forecasted euro and major success I don't know what my first major success was and some people whether there has
been a major success in my career but it's hard coming I think probably my advisor at university of british columbia peter suit fell who supported me and believed in me when we didn't seem to be very many good reasons for
doing so it's kind of a key confused canadian kid who was unsure what whether to become a lawyer in canada or go off to Oxford or go to graduate school in the US and who tipped me toward toward social science and us and what did he
see that other people had not seen god only knows well you if anyone would know it's you right you've lived with it for some time I think he probably saw I thought I was I was probably bright enough to do well and he probably thought that I was contrarian and weird enough today that there was some possibility of doing something distinctively well something distinctive and different and doing well do you think a kadai academic advisors
in general today undervalue weird students I think there's probably a tendency in that direction yes because weirdness is is a is a big word with weirdness takes lots of forums at you and I wouldn't we would not be embarrassed about trading away lots of weird people sir do you think people who grow up with the second culture or better forecasters oh you're thinking of my work with Carmi every time our course yeah I think it does
need to be some advantage there yeah and where does that come from how does it work I mean it has to tie ties into accountability accountability to conflicting audiences and value pluralism you you you have a richer internal dialogue you learn to balance conflicting perspectives more you have to be a better perspective taker perspective-taking is a very important part of super forecasting - how do we create more nate silver's and Philip
tetlock what should we change in the world to get more of you more they name they're probably very different creatures sure but we want more of you both right what should we do you know one thing I think that would be useful there is this tendency for different for training and universities to have become hyper professionalized and compartmentalized so I think it is harder for for people who have weird interests astraddle say
psychology in organizational science and political science and law history the people who have weird sets of interests it's hard for them to get traction in the current career environment certainly in in in in my home discipline of psychology I mean you had this kind of rampant publication inflation and we can you PhD students would be very lucky to get to get a job you'd have to have a ridiculous number of publications and
some of them in a journals and it's our sort of thing we didn't really expect maybe a 30 years ago we thought no that's a tenure case that's a junior higher so those kinds of pressures I think produce a focus and a narrowing of focus now you say well yes what you know the great I said earlier in the head a lot of the great advances in science come from hedge we're producing hedgehogs on an industrial scale here and I think
there's some advantage that make me a little bit more room for for the further four weirdo eclectic and the way the departments are carved up would you change that at all you know a number of academics in the past I get Margie
uc-irvine many decades ago tried to do something like that and that in a me government and president of the university Pennsylvania is trying to do that with you know and integrates knowledge which is the the the cherry professorships
that were used to among other things to hire to hire Barb and me and hire a number of other people and and we are sort of floaters you know we're not for not connected to one unit we've all multiple multiple connections so and
that's I so that's a very hope that's a very hospitable work environment my point of view but I don't see a lot of places doing a Jim March UC Irvine experiment Elissa integrate all social sciences together and Amy Gutmann Pik
Penn integrates knowledge kind of program you don't see too many of them yet it depends against the green I'm not even sure it'll survive at Penn beyond Amy I mean the natural tendency will be for departments to want to clawback the resources let's say someone comes up to you and they say Philip I would like to be more Fox like I'm not enough of a fox what actual advice would you give them to achieve that what should they do not
do wake up earlier in the morning exercise more to commit career suicide become more Fox like maybe they're not an academic they're a smart business person how do they do this I you know kind of obvious things like read a little bit more outside your field if you're a liberal read The Wall Street Journal if you're conservative you know you read the New York Times you expose yourself to distant points of view try
to cultivate some interest outside your field trying to connect them together I think there's an optimal distance I mean so for history for example is you know sounds ended quite very different from what I did when I started as an
experimental psychologist in history looks very different but they can be connected because historical judgment is something that psychologists study to some degree as I call it as interested in hindsight encounter of facts
and so forth so you can link the two so I think it's an optimal distance and when you go foraging as a fox you you probably don't wanna forage you know way way far away you want you want to forage far enough away that'll be stimulating but but but still possible to reconnect what kinds of people are best at adversarial collaboration rare beers are variable also that's but how do you spot really really hard to do personality
trade a cognitive trait gosh that's the hard one I mean that was a thing that Danny conned an economy I think coin determine he was you know dealing with his various critics over time and my wife barb actually was involved in an adversarial collaboration between the Cana man camp and a Giga render camp on the conjunction fallacy it took at least two or three years of her life it's a hard it's very hard to get people to it
mean lots of us in principle or pop Aryans we believe in stating our beliefs is falsifiable hypotheses and we also most of us believe their beliefs are probabilistic so we're gonna somewhat Bayesian but that's lip service there there's what we there's there's what we believe you know there's our formal set of epistemological formal self-concept which is kind of noble falsification ax stand probabilistic and and then there's
how we actually behave when our egos are at stake in particular controversies and those things are quite different I I think Danny Kahneman who's not known as an optimist you know did propose they did proposed it it's it sounds like an optimistic idea but I think he's not all that optimistic about what it can achieve on close inspection I might my efforts at adversarial collaboration have not been all that successful I'd like to jump start a few
of them again but it's it's a it's very hard to find the right dance partners let's try a question from the realm of the everyday in the mundane if I go around and I look at Mexican restaurants I'm very good at predicting which ones have excellent tacos what are you good at predicting I'm not good at predicting your questions good at predicting I think I was pretty good at anticipating the fragility of a lot of micro social
science knowledge prior to the replication crisis erupting you know I mean everyday life oh okay so not not not in my but social events not social science what in your life were you good at predicting when you're going to get tired and want to go to bed at night or when the dog wants to eat what is it well we have a pet free existence and you know we don't our lives are actually pretty pretty pretty simple guess we subscribe to that old
adage I should assume it should be you'll be more boring and stayed in your life so you can be violence and creative in your work we have a kind of a routine here so it's gnarly predictable maybe weed and now you know and then into
quarantine days I mean it used to be would be free quarantine you know he said oh I'm you know Liz let's this we're gonna go to Europe we're gonna go here there they were there are these little points of unpredictability that
spice up life but those do not exist right now and two more questions to close first what can you tell us about your next project well I think the next project is the one that I mentioned earlier it's linking historical counterfactual reasoning with conditional forecasting I think it'll be the second phase of the focus research tournaments I think that counterfactual reasoning has for too long been the last refuge of ideological
scoundrels and it'll be insofar as we can improve the stand standards of evidence and proof in judge counterfactual claims as well as conditional forecasts linking the two I think there's a potential for improving the quality of debates mom interested parties and finally what should a super forecaster predict about the future course of your influence oh to be very cautious because we know we're running against the green we're running against
a psychological green run against human nature we were any games of sociological green but the Enlightenment to stable you tell us right so if it keeps on cumulating and growing your influence should be enormous positions well there's a lot of there's a lot of cognitive resistance to treating one's beliefs is falsifiable testable falsifiable probabilistic propositions may people naturally are gravitates who are thinking of their beliefs as as ego
defining quasi sacred possessions and that that's one major source of us one major obstacle then you have the existing status hierarchies you have subject matter experts who are entrenched have influence why would they want to participate in exercises in which the best possible outcome is a tie which they really reaffirm that they deserve the status that they already have so that's not a psychological sociological resistance you know it
seems sociologists and economists have kind of different reactions to forecasting German sociologist the reactor you know why would why would anyone be naive enough to think that anyone would want to have a forecast in German organization their their status disruptive right yes and an economist to say well these things are so great how coming about everywhere and that Philip tetlock thank you very much it's please take care