All Segments
3 tips from one of the world’s leading experts in a/b testing
3 segments available
You can listen to the full pod with Ronny Kohavi here: https://www.youtube.com/watch?v=hEzpiDuYFoE #startup #founder #FAANG #softwareengineer #productmanager a/b test basics, experimentation in product, a/b test tips, how to do a/b tests, a/b testing
Segments Timeline
Speeding Up Experiment Results
Ronny Kohavi emphasizes the importance of having a quick scorecard after an A/B test concludes. He argues that a good platform should provide results within a day, rather than making teams wait for data scientists. This approach significantly accelerates the decision-making process in A/B testing.
"one is if your platform is good when the experiment finishes you should have a scorecard soon after maybe picks a day but it shouldn't be that you have to wait a week for the data scientist to me this..."
Reducing Metric Variance
In this segment, Kohavi discusses techniques to reduce the variance of metrics in A/B testing, allowing for faster results with fewer users. He provides an example from Airbnb, where they cap metrics like nights booked to ensure statistically significant results are achieved more quickly.
"things now in terms of using the data efficiently there are mechanisms that help you reduce the variance of metrics so that you need less users so that you get results faster some examples that you mi..."
Using Pre-Experiment Data
Kohavi introduces the technique called Cupid, which utilizes pre-experiment data to adjust results. This method helps achieve unbiased outcomes with lower variance, ultimately requiring fewer users to reach statistically significant conclusions in A/B testing.
"faster a third technique is called Cupid which uses the pre-experiment data to adjust the result and we can show that you get the result as unbiased but with lower variance hence it requires fewer Eng..."