"I'm very clear that I'm a big fan of test everything which is any code change that you make any feature that you introduce has to be in some experiment because again I've observed this sort of surpris..."

"welcome to Lenny's podcast where I interview world-class product leaders and growth experts to learn from their hard-winning experiences building and growing today's most successful products today my ..."

"after a short word from our sponsors this episode is brought to you by mixpanel get deep insights into what your users are doing at every stage of the funnel at a fair price that scales as you grow mi..."

"test you've run or maybe the most surprising result from an A B test that you run yeah so I think the the opening example that I use in my book and in my class is the most surprising public example we..."

"that he did he spent a couple of days implementing it and as is you know the common uh thing at Bing he launched the experiment uh and a funny thing happened we had an alarm big escalation something i..."

"idea in Bing's history and and rated properly right nobody gave this the the priority that in hindsight it deserves and that that's something that happens often I mean we are often humbled by how bad ..."

"shout out to Ricardo or mutual friend who helped make this conversation happen there's this like holy grail of experiments that I think people are always looking for of like one you know hour of work ..."

"many of them launched not every experiment maps to an idea so it's possible that when you have an idea your first implementation you start an experiment boom it's egregiously bad because you have a bu..."

"very humbling I know that every group that starts to run experiments they always start off by thinking that somehow they're different and their successor is going to be much much higher and they're al..."

"um but there's another more more accurate I would say uh resource that's useful that I recommend to people and it's a site called goodui.org and good ui.org is exactly the site that tries to do what y..."

"is institutional memory right which is can you document things well enough so that the organization remembers the successes and failures and learns from them I think one of the mistakes that some comp..."

"negative and that's interesting so we focus not just on the winners but also surprising losers things that people thought would be a no-brainer to run and then for some reason it was very negative and..."

"next iteration what advice do you have for people to actually remember these surprises you said that a lot of it is institutional what do you recommend people do so that they can actually remember thi..."

"this will get segue to something I wanted to touch on which is there's often a I guess a weariness of running too many experiments and being too data driven and the sense that experimentation just lea..."

"be successful over time if you just try enough but some experiments have you have to allocate sometimes to these high risk High reward ideas we're going to try something that's most likely to fail but..."

"them were a breakthrough and I remember sort of mailing uh chilu with some statistics showing that you know it's time to abort it's time to fail on this uh and you know he decided to continue more and..."

"ingredients to a b testing and I'll just say I'll write not every domain is a minimal thing to be testing right you can't a B test mergers and Acquisitions right it's something that happens once you e..."

"zero but when you're not there it's still expensive and there are there may be reasons why not to run AB thefts you talked about how you may be too small to run a b tests and this is a constant questi..."

"you're not degrading uh and getting value out of response so you ask for rule of thumb 200 000 users you're magical below that start building the culture start building the platform start integrating ..."

"those experiments and we were able to map out you know this number of ads causes this much increase to churn this number of ads causes this much increase to the time that users take to find a successf..."

"then you need to insert these other criteria and what am I doing through the user experience one way around it is to put this constraint another one is just to have these other metrics again something..."

"basically have a kind of a drag metric that makes sure you're not hurting something that's really important to the business and then being very clear on what's the long-term metric we care most about ..."

"about it one is you can run long term experiments for the goal of learning something so I mentioned that at Bing we did run these experiments where we increased the odds and decreased the odds so that..."

"then so then we backed up and then we said okay we can either phrase this as a constraint satisfaction problem you're allowed to send user and email every X days or which is what we ended up doing is ..."

"understand based on what users were unsubscribing from which ones are really beneficial I love the surprising results we all love them I mean this is this is The Humbling reality and you know people t..."

"launch this even though it's bad for the user no that's terrible yeah yeah so uh this is this is the other advantage of of recognizing this humble reality that most ideas fail right if if you believe ..."

"just as you said yourself they be ready to fail right I mean do you do you really want to work on something for six months or a year and then run the A B test and realize that you've hurt revenues or ..."

"was and this was you know credited to chilu and uh Sacha Nadella they were ones that says Ronnie you know you try to get office to run experiments we'll give you the air support uh and it was hard but..."

"is it ever worth just going let's just rethink this whole thing and just give it a shot to break out of a local Minima or local Maxima essentially so I think what you said is fair I mean I I do want t..."

"you know let's make sure that we understand that shipping this project has no value is complicating the code Base maintenance costs will go up you don't ship on flat unless it's a sort of a legal requ..."

"in the end from what I remember speaking of Airbnb I want to chat about Airbnb briefly I know there's and you're limited in what you can share but uh it's interesting that Airbnb seems to be moving in..."

"then roughly what's your sense of how things are going where it's going so as you as you know I'm restricted from talking about Airbnb I will say a few things that I am allowed to say one is in my tea..."

"counter factual we don't know it's a really interesting perspective yeah there may be such an interesting natural experiment of a way of doing things differently there's like de-emphasizing experiment..."

"could be better all right mysterious one more question Airbnb you were there during kovid which was quite a wild time for Airbnb we had sunshine on the podcast talking about all the craziness that wen..."

"experiment so sometimes it means that you might have to replicate them six months down when covid say uh is not as impactful as it is saying that you have to make decisions quickly to me I'll point yo..."

"yeah whatever another case study for the history books Airbnb experiences I want to shift a little bit and talk about your book which you mentioned a couple of times it's called trustworthy online con..."

"was translated to Chinese Korean Japanese and Russian and so it's it's great to see that we helped the world become more data driven with experimentation and I'm happy because of that and I was pleasa..."

"important in writing experiments so to me the experimentation platform is the safety net and it's an oracle so it serves really two purposes the safety net means that if you launch something bad you s..."

"and the nice thing is when we we built all these checks to make sure that the experiment is correct if there was something wrong with it we would stop and say hey something is wrong with the experimen..."

"error rate so what this led is that people that started using optimizely thought that the platform was telling them they're very successful but when they actually started to see while it told us this ..."

"trust because they built something that had very much inflated erroring that is uh pretty scary to think about you've been running all these experiments and they weren't actually telling you accurate ..."

"versus 49.8 split and therefore something is wrong with the experiment right now people I remember when we first implemented this check we were surprised to see how many experiments suffered from this..."

"mismatch so we blanked out the scorecard we have this button and then we started to see that people press the button is still presented the results of experiments with sample ratio of this method so w..."

"and I will say that nine out of ten when we call out time is law it is the case that we find some flaw in the experiment now there are obviously outliers right that first experiment that I shared wher..."

"right it's not five percent it's 26 so that's the number that you should have in your mind and that's why when I worked at Airbnb one of the things we did is we said okay if you're less than 0.05 but ..."

"say someone listening wants to start running experiments they say they have tens of thousands of users at this point what would be the first couple steps you'd recommend well so if they have somebody ..."

"understand the question of the oec is it clear what they're optimizing for right there are some groups where you can come up with a good oec some groups are harder you know I remember one funny exampl..."

"ends up being most important still I want to ask you about speed is there anything you recommend for helping people run experiments faster and get results more quickly that they can Implement yeah so ..."

"fewer users Ronnie is there anything else you want to share before we get to our very exciting lightning round uh no I think we've asked a lot of good questions uh hope people enjoy this I know they w..."

"dangerous half-truths and total nonsense uh by the Stanford professors from The Graduate School of Business very interesting to see many of the things that we grew up with as sort of well understood t..."

"but uh but yeah we were in the vicinity that's pretty scary my wife thinks I've yeah every every time something's wrong with me she's like oh that must be a Chernobyl Chernobyl thing okay next questio..."

"point we had a false alarm and the cops came in and had this amazing video of how they're entering the house uh and pulling the guns out you gotta share that on Tick Tock that's good content wow okay ..."

"another I think there's a book that's based on this like how to read a book well Ronnie the experiment of us recording a podcast I think is a hundred percent positive p-value 0.0 thank you so much for..."