0:00
i felt like it was a problem that was very solvable and we ended up renting a stadium to just hire like 60 000 drivers in a couple of weeks so i think looking back it was certainly a risk like when i got there it was in a house and i realized
Search this video
0:17
i've probably made a huge mistake but we were growing very quickly already even at that small scale of like 4 000 orders per day crystal wajaya has been leading product and growth teams at some of the largest
0:33
consumer businesses in southeast asia including kumu where she's currently the chief product officer and gojek where she built and led the growth team through the early years of what is now the largest super app in southeast asia
0:46
to put this in context gojek completes more rights per day than lyft and more food deliveries than grubhub ubereats and doordash combined and it's the number one mobile wallet in indonesia and southeast asia in my opinion american startups have a
1:00
lot to learn from startups in asia and crystal has been at the ground floor of some of the biggest successes there in our conversation we cover the biggest growth unlocks that crystal has seen across the company she's worked at
1:13
what growth investments usually pay off and which often don't we dig into growth models a bunch of pro tips for accelerating growth why most analytics efforts fail and how to avoid that how to hire and structure your growth team
1:26
and we also talk about the non-profit that crystal started that aims to help young women get into stem called generation girl crystal is such a star and i hope that you enjoyed this episode as much as i did and with that i bring you crystal or
1:40
jaya if you're setting up your analytics stack but you're not using amplitude what are you doing amplitude is the number one most popular analytics solution in the world used by both big companies like shopify
1:54
instacart and atlassian and also most tech startups amplitude has everything you need including a powerful and fully self-service analytics product an experimentation platform even an integrated customer data platform to help you understand your
2:09
users like never before give your teams self-service product data to understand your users drive conversions and increase engagement growth and revenue ditch your vanity metrics trust your data work smarter and grow your business
2:25
try amplitude for free just visit amplitude.com to get started [Music] hey ashley head of marketing at flat file how many b2b sas companies would you estimate need to import csv files from their customers at least 40
2:40
and how many of them screw that up and what happens when they do well based on our data about a third of people will consider switching to another company after just one bad experience during onboarding so if your csv importer doesn't work
2:53
right which is super common considering customer files are full of unexpected data and formatting they'll leave i am zero percent surprised to hear that i've consistently seen that improving onboarding is one of the highest leverage opportunities for both sign up conversion and increasing long-term retention getting people to your aha moment more quickly and reliably is so incredibly important totally it's incredible to see how our
3:19
customers like square spotify and zora are able to grow their businesses on top of flat file it's because flawless data onboarding acts like a catalyst to get them and their customers where they need to go faster if you'd like to learn more or get
3:35
started check out flat file at flat file dot com slash lenny crystal thank you so much for being here i've read a bunch of your stuff online we've exchanged a bunch of emails and tweets but that's the first time that
3:51
we're actually chatting for real and i'm really excited to yeah i'm excited to learn from you and for folks to learn about you we may have like cross paths on clubhouse and the audio forums or on the twitterverse so it's really cool to
4:03
see wow i just remember that that is so right i think we were talking about reforge and epo is that right yes that's right good times oh my god clubhouse days the days when cold does a thing they have a thank you to learn about
4:16
exponential decay oh man okay maybe we'll get to that and we're gonna we're gonna be chatting a lot about consumer growth and a bunch of stuff along those lines but before we get into that you have a fairly unusual path and also geography as compared to many of my other guests and so just to set a little context can you just kind of walk us through your your kind of career path and journey from i think as an investment banker initially and then
4:39
currently as chief product officer at kumu and then living in singapore also so yeah tell us all about your path yeah i think my path was certainly an a non-standard one while i grew up in san jose the bay area you could see
4:54
companies like lyft kind of emerging around the year that i was graduating college but really it was how do i graduate college as quickly as possible because this is very boring so i took a poli sci major i am not a math or a computer science major
5:12
i didn't know what a consultant was because i was just like trying to get out of college i didn't realize people start looking for jobs before they graduate so last two weeks of school i was looking on craigslist cause i was like
5:25
craigslist is how everyone gets a job right i'm like a first generation american student so my parents could not help me at all with like you should look into this company called mckinsey or hear all the life paths that you have
5:40
ahead of you so i ended up taking an investment banking research job and my job there was to figure out how to call startups and analyze their potential for vc financing or m a advisory and i barely knew what those words meant at
5:59
the time i ended up owning a huge excel database of like 130 000 rows 60 plus columns and because again i am very impatient i was like this is a terrible experience how would i create a customer database and so i ended up kind of google
6:19
throughing all of the work needed to build a mysql database i presented a plan and investment banking surprise surprise is not very tech forward so they looked at my plans and they're like what is this my sequel thing isn't that super expensive what is open source so i ended up leaving that job because i realized that if i wanted to get into something more tech it would probably not be at a investment bank so i took the investment banking
6:49
strategies that i had learned there and kind of applied the same pattern matching to companies in southeast asia so my family originally is from indonesia i thought i have a kind of safety net i must speak indonesian really well just by births so maybe that's a great country for me to look at so i took kind of the approach of let's find a company that makes a lot of sense that i feel i resonate with and i literally cold called emails and some
7:18
companies so gojek being on that list i literally emailed someone after googling hr at gojek and said i'm willing to move to indonesia take a bet on me um and they actually did so i got extremely lucky five years lie by insanely fast i went
7:35
through building out the data team from scratch when you have all of the data you know how much fraud you have in the system so then i ended up building out the fraud and risk team picked up performance marketing and then
7:47
it was like okay now we're ready to grow so you have all of this data now take on growth got it you kind of uh were very modest about go-jek and the success of that company and also kumu where you work now so just to kind of
8:03
set a little context for folks that aren't familiar with these companies can you share how big they are and how big of a deal they are in southeast asia yeah they are uh pretty massive so gojek is now called go 2 they just merged with
8:16
the largest e-commerce platform in indonesia so across southeast asia we had about 170 million users like it's southeast asia has scale if you ever wanted to work at scale you would go to southeast asia we had 20 plus different
8:33
services from transportation to food shopping medicine delivery bill pay movie tickets so it was like all of the startups in america in one app all being built at the same time with the same user base and so everything was tremendously
8:51
layered because you could fill all of these opportunity gaps in the market where a single app would probably not be as sustainable so gojek is massive across indonesia singapore thailand vietnam and then kumu is kind of a super app for
9:10
social so gojek was very transactional it was like here's a job to be done i want to pay for something and someone delivers it to me and with kumu it's more so of a i want to do clubhouse zoom google hangouts gather
9:23
round all in one app so we cover social feeds audio video multi-seats there's a ton of different use cases that we serve on kumu and kumu's primarily in the philippines but ranks top 10 in a bunch of countries as a top grossing mobile app
9:40
so with kuma you joined when they're already doing fairly well with gojek as you said he joined very early what did you see in that company that helped you decide to join such a risky early stage company for folks that are
9:53
maybe thinking about joining a startup what kind of things did you take away at what to look for honestly it's probably a lot of luck but also at that age i realized i have very little to lose so with go-jek i think i felt like it
10:06
was the right company because i was able to really clearly understand the value prop traffic in indonesia is crazy it takes you two hours to go 20 kilometers so of course you want to take a motorcycle taxi to beat that traffic
10:22
of course you don't want to go out and get food and then have to come back this long pathway of two hours so i think taking that warren buffett approach i knew that the product made sense the market made sense as well so
10:37
drivers there were already a thing but it was very hard to connect them to the consumer it was painful to haggle prices there were lots of restaurants scattered across indonesia so the valley prop and the market made sense and the channel by
10:51
which you would do it through this mobile app made a little bit less sense at the time because most drivers didn't have a mobile app but i felt like it was a problem that was very solvable and we ended up renting a stadium to just hire
11:04
like 60 000 drivers in a couple of weeks so i think looking back it was certainly a risk like when i got there it was in a house and i realized i've probably made a huge mistake but we were growing very quickly already
11:22
even at that small scale of like 4 000 orders per day i want to spend a lot of time talking about what you learned driving growth these companies but one quick question so gojek is kind of the super app where you do a lot of stuff in one app do you
11:35
have any insights into why a super app hasn't emerged in the us yeah i think the sentimentality of a conglomerate is very different in southeast asia so we've grown up with you know a specific conglomerate owning not just
11:51
the mall that you go to but also the apartment building that you live in and the school that you go to and so they're very well integrated and there's a sense of trust in a conglomerate whereas in america we already kind of shy away from
12:07
like does google know too much about me there's also i think the second aspect of it which is that in asia we've kind of leapfrogged to the computer era so everyone has a phone but you may not even have a computer in the entire
12:20
household and so when your phone is full are you going to delete a photo of your kid or are you going to delete this app you're probably going to delete the app so for anyone to really survive it has to be part of this super app concept oh wow i've never thought of it that way that you don't have a lot of space in your phone and so you want one app to do a lot of things that's right so there's a decision factor that you don't really
12:43
have in the us because the cloud storage and device capacity there is a little bit bigger hmm so interesting so in the u.s you could have different apps be like basically the super app doesn't have to be the best at everything the fact that it does enough and everything good enough wow amazing okay that's super interesting okay so transitioning a bit to growth and things you've learned along the way so you talked about how i think gojek
13:11
you said hired like tens of thousands of drivers really quickly are there things that startups in asia do that you think companies in the us should do and can learn from in terms of yeah so we did kind of crazy things right like
13:25
if someone told you in the us that they were gonna rent out a stadium pre-load a bunch of mobile devices market that driver should come here in mass for a job fair they're gonna give them a phone and send them on their way like some people would kind of
13:40
say no like that's kind of crazy won't we get in trouble and to an extent maybe that's true so maybe there are some limitations there but this concept of doing things that are somewhat crazy but validate a point
13:53
doing stuff that don't scale especially i think is really the bread and butter of what we did at gojek like we were insanely scrappy we would do things as simple as wanting to test a subscription feature which was just released in singapore a
14:10
couple weeks ago we ended up saying we have this voucher system that we can distribute vouchers in the back end we obviously know our driver's phone numbers why don't we just add them to a whatsapp group we'll add
14:23
100 drivers randomly to a whatsapp group we'll tell them every time you are on a ride with a customer try to sell them this pitch you are the only driver who can sell you know a subscription package have the customer give you ten dollars
14:39
text us when they say yes someone will be sitting by this phone all day every day we'll look up the customer that you were on a ride with in the back end we'll give them the vouchers in the back end and then we'll deduct 10 from your
14:54
balance like it works it's really this wizard of oz experience we don't have to build anything i coordinated with a bunch of interns and we were able to validate some of the value prop and conversion rates that we would expect in
15:08
a subscription service when we wanted to do a new onboarding screen but turns out we had lots of engineering work to do we took a screenshot of the screen as is and we just had our designer put what the onboarding flow might look like
15:24
if we had to overlay it on top of the screen and we just sent that as like an in-app message and then eventually i think finding stuff that does scale intuitively we knew that we were sending out lots of fake features through things
15:41
like type form surveys things like a personality quiz can be very easily done through type form and we realized that if we built in the in-app web page and we made it easier for us to do a website deployment on our backend side
15:56
we wouldn't have to wait for a mobile app elites to test some of these new features out that could be done on web so it's really just like what is the user experience that we want to create how do we manifest that as quickly as
16:10
possible let's just try that first going back to the stadium example i knew you said that you hired a stadium full of people i didn't realize it was actually a stadium it was literally a stadium that we rented like a football field a couple
16:23
football fields if i'm not wrong it's long lines boxes of phones and sim cards so it was a lot of just like doing really hard work to get to that scale wow i know you do a lot of advising too do you advise startups to
16:39
be more scrappy and do things that don't scale imagine because in the us the culture is a little different like the only thing better than know it like if you have data of what your customers are doing that is the best data you could ever get and so if you don't have a tested hypothesis if you can't think of a way to run an experiment then honestly that idea is pretty useless like maybe it makes sense to the market to the model
17:10
but you could have weird consumer sentiments like not everyone is a rational actor so testing the actual experience and seeing how people respond to it that's the best possible data pulling that thread a little bit for
17:23
startups experiments are often hard because they're just not enough data not enough users how do you think startups should approach that can you run experiments when you're really really early you should i mean even if you have a sample size of 30
17:38
the data you get back generally does not change but its precision will so mathematically speaking you're gonna get the same level of trends but the precision at which you understand those trends will become more deep if you have more data
17:55
but the underlying information that you're getting out of that won't be very different at larger scales so what's better than having 30 data points certainly having 100 but what's better than having zero is definitely 30.
18:10
fascinating so contrarian running experiments at 30 people i love that you have to i mean like every idea is so cheap at that scale like you could do things that don't scale dramatically better with 30 people than at 100 if
18:26
you're testing so when you just to kind of pull out a little bit when you're running an experiment 30 people what do you look for you're looking for like 20 of them to do something like a large percentage of that group does something right
18:39
so everyone wants to go on like retention they want to see that users are doing this thing and they want to get from step 0 to 100 really quickly but they don't realize that like users make decisions based on succeeding events so what's one step
18:57
before the user makes that decision what are the things that they have to do the things that have to be done so we're always looking for what is a specific reason that this user might have converted for things like go food it
19:12
would be things like when does a user try a new merchant if what people are ordering right now or just food that they already trust and know if you need to have trust in order to purchase food from a merchant how do we
19:26
generate that trust so we actually hacked it by connecting people's facebook connect login so we had already had permission to look at who they had connected with on facebook we actually looked at the food that their friends had purchased
19:42
and used that as a data set of hey here's food that lenny purchased and liked maybe you would like it too and so that was one way to hack the trust factor and we did find that when we told people this friend purchased from this merchant
20:01
you would be twice as likely to purchase from a brand new restaurant than users who did not have this feature and that increases gmv that eventually gets you to the conversion rate that you wanted but it solved a different problem before
20:16
and how do i convert it was how do i solve for trust how do i break the barrier of facilitating that decision making process that aha moment by fixing the setup moment which was trust and that's just a general kind of rule of thumb you have don't use retention as a goal i know you wrote about this somewhere is that kind of a rough rough rule of thumb you use i think a lot of people thought that i had meant like retention sucks don't care
20:44
about it at all but in reality it was really like when you think about retention like that's just not specific enough so there is this um mental model that i use from made to stick where they'll tell you like lenny think of everything in the world that is orange and you're like an orange what else and then if you change that structure with sandbox to think of everything orange that's in a construction site then you really start to realize
21:17
and grasp at concrete concepts and can actually action on them in real life got it speaking of retention where have you found products and companies have the most success increasing retention it's usually the step right before
21:34
conversion so if they aren't sure like why the user opens the app or they aren't sure why the user got to this checkout page it's often some like copy or the pack has been ineffective in some way i'd like to see founders think about
21:53
the user psych model that dorias contractor often talks about so you need some momentum in that user journey to get them over the hump of some of these very painful user processes like typing in a credit card that's a lot of
22:09
work how do you lower that friction and being able to sequence the right steps effectively and just moving around screens actually can do a lot going even deeper there so all the companies you've worked out the companies you've advised you're on
22:26
the boards of a couple companies i noticed what have you found to be really good uses of time in terms of growth investments like things that often work and then kind of a second question what do you find is rarely successful where
22:40
people invest a lot of time and ends up not being really useful for growth yeah i think i see a lot of founders like grasping at straws so they'll be like this brand new feature that does something kind of different from what
22:54
people are already doing on our app like this will make things work but they don't have any wizard of oz test they haven't proven that people want to do that they don't have any data of users currently trying to do that and that's a that's a sign of like why this instead of literally anything else that you could be doing i do find if you have a lot of people landing on a webpage or an app and then not doing anything then it's probably copy like
23:24
they haven't even experienced the product it's clearly not the product that's wrong so how can you change the copy and resonate with the pain point rather than the solution you are offering so that users understand how to
23:37
fit themselves into the use case so copy is a big one if i see conversion rates aren't landing between app launch to some first action but if there is conversion and they're just not as frequent i try to look at what the most painfully long conversion
23:59
events are so users who eventually check out or eventually completed the aha moment what are the user paths and what is the longest one that seems like it's the most painful are there enough people trying to do that and how do we shorten that cycle
24:15
so for kumu it was things like users wanted to sign up and find their friends on kumu and so they were using search frequently search was under utilized api it was kind of slow we sped that up conversion rates go from
24:31
60 to 90 percent like over the course of a few weeks of just optimizing that and putting more content there so looking at like where are people doing things and then failing like you already know this percent of people would
24:45
convert if you fix to this that's indefinite potential win so we try to layer these definite wins with like crazy bets of like brand new feature with no data at least run an experiment if you can but i always try to layer in these sure
25:00
wins when you talk about conversion being good and bad do you have a rule of thumb or kind of a mental model of like here's a rough range of like this is good and we should not really spend a lot of time on this and this is bad and
25:12
we should optimize so assuming that the frequency is correct so you have a weekly frequency if users are coming back if it's a free product 60 percent right it has to be at least 60 if it's a free product week over a week
25:25
if it's a paid product i usually look at that more as like maybe 20 to 30 percent and this is retention people coming back the next week exactly coming back in the second week or month or whenever your frequency ratio is and this is at scale
25:40
so if you are much smaller like your friends and family that better be near close to 80 no matter what because if you can't even convince the people who care about you to use the product it probably isn't going to solve
25:54
the job for anyone else very handy very concrete numbers and then your point is that when you're a startup it's only going to go down because your kind of early adopters are more excited and and they'll be more excited about coming back and so yeah so you want to start really high i mean don't make the same mistake that uh netflix and spotify have made which i guess is when they've launched they've started international expansion and they
26:18
see this very small percentage of users start to sign up for spotify or netflix there are very few people though in southeast asia or internationally that have the types of credit cards that spotify or netflix would accept and so
26:33
when they launch in these markets and they see a ton of uptick in the first week they're like this is only going to get better when in reality it's like you just pulled forward everyone who could have possibly subscribed to you now you're going to
26:45
have to work a lot harder to get everyone else the 60 number um so you're you're saying it's like that every week 60 of the previous week come back roughly is just a rule of thumb exactly is that kind of how you think about it
26:56
versus say cohort retention is that just because it's easier as just a simple rule of thumb i am actually thinking of it as cohorts so 60 should be your week one and then it should flatten i think i usually give teams like two to
27:09
three weeks or frequency periods to see things flatten but it better flatten around sixty percent for a free product that's kind of that's actually what we saw at gojek early days it was like 60 70 retention rates because people
27:26
were using this product that really solved a huge problem for them and i think that's when i knew we were gonna be fine if people keep coming back the product just needs to work wow so week one forty percent of people drop
27:39
off week two and beyond basically nobody drops off is kind of what you look for wow what a what a high bar but i like that cause yup well go jack isn't that good okay there we go if you want to be a decacorn there's your new benchmark
27:54
exactly amazing okay there's a bunch of other stuff i want to dig into one is it's just data modeling and thinking about growth strategy as a founder so say a startup is just trying to think about how do we drive growth where do we
28:07
invest do you have kind of a framework or a process i know this might be a really big question but just for founders to think about to think about how their growth works what their drivers might be how would a founder approach that problem
28:20
for sure so i thought that this was not an obvious process it wasn't like an explicit process until i worked with reforge to build my data for pms program gotta get that plug there go reform but i basically talked with the reforged
28:37
folks about like here's what i would do in all of these scenarios and they're like oh so you're mean you're doing this step one step two and i was like yes actually how did you figure that out so i don't really think in frameworks like this is just a logical process to me but i think what i've figured out is it's step one like you have constraints right similar to our sandbox example of like everything in the world that's orange versus
29:02
everything in a construction site you have to think about the physics of the current market the product the model and the channels that you're using so to use gojek as an example it would be market of indonesia here are the consumers in this market
29:18
the driver side supply side in this market here is the product mobile app we're able to connect drivers and consumers there is a allocation that we create model we charge per order channel we are able to do this through push
29:34
notifications or in acquiring new users it might be through facebook ads or and this was a really big insight for us it's the real world there's a physical conception of a driver in a jacket driving around the city who is marketing
29:49
gojek for us and word of mouth actually was primarily driven by i saw a driver on the street so i knew gojek was here and that actually was a huge driver of all of gojek's growth as it expanded to new cities so step one is what are the physics
30:07
step two is when you think about loops and growth funnels and the quantitative inputs to each loop does that fit into these physics or do you have to change like four or five different things so we were very careful about changing too many parameters and making too many bets on too many variables going our way so we would always change like one small thing at a time and make sure that it fit into the model this episode is brought to you by epo
30:41
epo is a next generation a b testing platform built by airbnb alums for modern growth teams companies like netlify contentful and cameo rely on apple to power their experiments wherever you work writing experiments is
30:55
increasingly essential but there are no commercial tools that integrate with a modern grow team stack this leads to wasted time building internal tools or trying to run your experiments through a clunky marketing tool when i was at airbnb
31:07
one of the things that i loved about our experimentation platform was being able to easily slice results by device by country and by user stage epo does all that and more delivering results quickly avoiding annoying prolonged analytic
31:21
cycles and helping you easily get to the root cause of any issue you discover epo lets you go beyond basic click-through metrics and instead use your north star metrics like activation retention subscriptions and payments
31:35
info supports tests on the front end the backend email marketing and even machine learning clients check out epo at getepo.com geteppo.com 10x your experiment velocity so step one just to cover this is figure out how you're growing in gojek's case
31:53
it was partly real world people just seeing seeing gojek riding around i think it's it's both figure out how you're growing and also the elements that you have at your disposal like what are the levers that you have that maybe
32:08
you've never tried using when we looked at our model this way we actually realized we had underutilized the driver's capacity to drive our growth pun definitely intended so in looking at the model this way we had thought through like what is our goal we want gopig to be much bigger than it really is it's an e-wallet service users are able to get access to this digital balance how do we drive adoption and so when we looked at the lever uh we
32:40
have a driver we actually created an incentive model so we built a very small service that would check when a driver got allocated to a customer again the product and the model we would then check in the database has this customer ever used our gopay product before did they have a digital balance and if the answer was no we would message the driver immediately hey this customer hasn't done a gopay top up before if you get them to give you cash
33:12
and we deposit it into their virtual wallet we'll give you extra money so using them as the salesperson you wouldn't believe how great of a salesperson someone can be when you were literally trapped in a car with them going somewhere and so you have this
33:28
captive audience captive attention you have someone who has the incentive to cross pay or cross sell someone and to go pay and customers were able to feel the benefit because the driver was explaining it to them directly
33:42
so it was one small there was no change to the physics it was a lever usage what a devious strategy it was huge it was like 60 of acquisition once we released that oh my god so for thinking through your potential evers and physics of your growth do you think about it like bottoms up here's all the things that are going on and here's areas we can invest or do you have kind of like a menu of options top down of like here's the 10 things it
34:10
could be it's looking like for joke go jack it's these four and let's focus on that yeah i think you you always have to start from the fact that like we are not wizards like it's very hard to move the physics of a universe when
34:25
you are trying these new things so start with what currently works and currently exists and where you think the biggest constraint is or the best lever is and then fix that one piece because the entire universe isn't exploding like you're not the world
34:43
isn't changing so dramatically that your physics change so i think rooted in reality is very important got it okay so see what's working find the constraint and then step two is basically what can you do to the product to optimize the funnel slash loop to make it go even exactly love that maybe um as another example if something comes to mind with kumu how do you think of kumu through this lens yeah i'm always very hesitant to talk
35:10
about kamuka as there's so much competition right now and we're like on the clasp of some very interesting interesting things but i think for kumu it's actually very complex because there's a lot of human emotion that is involved like with gojek
35:25
you knew if you got the job done you made a transaction with kumu how do you know if a consumer made a friend like felt like they had a genuine friendship so you almost have to create more friction to identify users who
35:39
really got past that barrier and are explicit with the activity that they did so we have features that tell us if the user is really searching for this job to be done if they really want to be part of a community how do they fill out
35:56
this do they fill out the form do they fill out a questionnaire of like many questions do they go through this friction just to get access to a community so we almost create this artificial friction to help differentiate how deeply a user
36:12
wants something or needs something and if the user doesn't fill out that questionnaire maybe they're actually looking for something else they were looking for entertainment they were looking for content or short form content and so
36:24
creating almost like hand raiser uh approaches for a user to say like i wanted this thing we leave a lot of breadcrumbs in the app to be able to identify those paths awesome while we're on the topic of these two companies
36:41
just maybe for inspiration to founders who are thinking of ways to drive growth what were a couple of the bigger unlocks growth-wise for these two companies or even any other company that you've worked with that's interesting
36:52
yeah definitely in the early days it was copy so i think if your product does something that's not super familiar you have to tie it to something that is so i talked about using drivers to sell go pay before that one thing that we did was to
37:10
actually take someone's virtual account number and put it onto a picture of a credit card like you know what a credit card is that's familiar to you a lot of people didn't know what a digital wallet was and so when they looked at this like okay i have this virtual thing that acts like a credit card it works like my debit account then they understood the concept a lot better and we actually saw top-ups increase based on us literally just
37:37
sending that picture with someone's virtual account number there so they could go to an atm and they would just type in the card number as they would a regular debit account and they realized that they could top up through that channel because that was something that was pretty interesting to us with just how do we tie the familiarity loop back into the consumer mental model of the product and drive acquisition that way and that was a go check
38:03
yep is there anything else maybe since you don't want to talk too much about kumu any other like advisorships or companies examples of something that ended up working really well to help them accelerate growth things that have worked really well so
38:17
for one of the companies i work with abn bev they run a lot of their d2c brands in south america and globally so one of the features that we were looking at was how do we ensure that subscriptions don't actually become a
38:36
cancelling point for a user so in the app you could cancel or you could resume your subscription but you couldn't pause it so when we looked at the cancellation reasons and we saw that their number one reason was i still have too
38:53
much fear we actually decided well let's just add a pause button then because like canceling the subscription is a permanent solution to having too much beer how do you make a temporary solution that solves the actual problem
39:08
adding in a pause button actually helped alleviate a lot of the churn that was becoming very hard to reacquire back so that was one fix where we looked at the again physics of the model like we're not going to create new changes to the product or create one-time buys or like reactivation emails will just solve the problem at that small constraint where everyone drops off wait so can you order beer a subscription is that a thing is this a
39:37
consumer product or is this like it was a thing yeah cool okay this also reminds me of airbnb this is actually one of the biggest wins is adding a snooze feature to your listing exactly the same thing yeah all right there we go awesome tip for
39:53
folks that have churn problems snooze pause i want to shift a little bit to a post that you wrote that maybe is one of the more popular posts you wrote on the reforge blog called why most analytics efforts fail
40:07
and yes i'd love to hear your broad overview why do most analytics efforts fail and then how do how do teams avoid this maybe what are like two or four things they get two to three things they can do yeah i'm actually pretty surprised at
40:19
how much noise that has generated because i guess it came from a place of frustration where i kept telling people like you're doing this wrong here's how you should probably be doing it but i think it resonated a lot with fultz
40:33
because they recognized all of those symptoms but they weren't sure why it was happening so to say like oh this is the thing instrumentation is what's wrong i think it's a very actionable thing like it's probably one of the most solvable problems out there it just takes some time and mental model ships to do it well so a lot of people look at tracking data as how do i track my okr how do i know if i'm going up or down but they don't use
41:02
it to track or identify insights so i will use the example of using twitter for quote unquote news when in reality they're actually using twitter for entertainment do not treat metric gathering as entertainment like it's not there for
41:20
you to be like oh that's interesting how novel and then not act on it so real news is information that changes what you do in the real world and if you don't change what you're doing what you are doing is just getting entertainment so
41:36
let's use that as a premise the next step in instrumentation is to look at the fact that measurements do not equate to insights a measurement would be an observation it's a data point in your database so the example being power users do four
41:54
times more bookings is an odd observed fact because your transactional database obviously says that that is the case but it's on an insight because it doesn't have context it doesn't give you information that lets you act on it and
42:08
better understand the problem so another example would be if i see my girlfriend hanging out with a guy i don't know that is an observed fact that you see in the real world your hypothesis could be that your girlfriend is cheating on you
42:24
but the insight uh the actual fact might be that she's not cheating on you it's her cousin and now your insight is i am paranoid and i need to change my behavior to be less crazy so the insight will provide value when
42:41
you have this why answered why is this person doing this thing here's why and then you are going to act differently so for our purposes if we look at a goat food user will transact and is more likely to use a voucher that's a fact that's an observation
43:05
but it's not an insight an insight would be something like go food users who are power users are more likely to use a free shipping discount on a high gmv basket versus non-power users and that actually tells you how to
43:24
change your marketing approach it tells you that in what circumstances does someone do this when it's a high gmd basket give power users the ability to get a free discount but do not do this for non-power users because they won't
43:38
convert any better than they normally would so that helps you change your marketing spend it helps you understand the decision points of power users versus non-power users the insight is instrumenting properties into an event
43:53
so that you can segment who is doing what behavior and make some pipeline hypotheses on that observation test that hypothesis and then you get some causal representation of whether or not that hypothesis was right so it sounds like a lot of the root of the issue is setting up the wrong metrics the wrong i guess there's the tracking element of just capturing the right information and then also just not focusing on insights versus just
44:22
having a bunch of information exactly what are signs that you're doing this like say someone's gonna go load up their dashboard and they're like am i am i failing or not what should they be looking for so i already know if a team is good at
44:36
instrumentation or not just by looking at the instrumentation spec the symptom of a bad data tracking approach is you have a ton of rows with a ton of events but every event has like one property or no property being tracked so
44:55
an example with gojek would be when a user lands on the map to select a drop-off point the event would be drop off or like map loaded let's say and the properties there should be things like how many drivers do they see
45:13
on the screen what is the pickup location is it what city is it in what latitude and longitude is it what is this is their surge pricing what is the current minimum fare do they have a voucher code all of these characteristics of the
45:30
experience and the context that can help you look at hey when a user only sees two drivers on the screen they're much less likely to convert than a user who sees five drivers on a screen now we can look at in what cities and in what
45:45
latitude and longitudes do we mostly only see two drivers versus five drivers like being able to do the second layer approach of the why and not just stop that that's weird when you have two drivers you are less likely to book but then you
45:59
never ask why like that drives me crazy or the inability to even know that like there were only two drivers on the screen like you're missing so much context of the user's experience that you're unable to make assumptions about
46:13
why the user didn't convert i love this is there is a resource like maybe your course is probably gonna be the answer but for folks that want to figure out how to do this sort of taxonomy and events well how do they go about doing
46:25
that so i think it's important to just like go through examples yes every product is different but everyone has the same sign up flow for the most part so look at the sign up flow examples that i have in the blog post or in i believe amplitude actually has a pretty good long-winded documentation on this on how to do event tracking but it's really a matter of like sitting down and thinking really deeply if i were to press this button why would i
46:55
and why would i not and am i tracking that in my in my user properties so it's really just like sitting down and mapping out the experience speaking of amplitude and other data tools do you have a default recommended
47:09
metric stack for founders just to start with and maybe a few other things as they evolve it really depends on how early they are so if they have a single data warehouse with all of their transactional data usually i say like you can probably get by with
47:23
google data studio it's free usually with whatever you're using if not metabase has a great open source free tool if you have someone who can write sql or if you have multiple databases then metabase is great if you need in-app
47:39
mobile device event tracking i usually recommend clevertap because mixpanel has unfortunately failed me a lot and amplitude doesn't have the crm components that i would need all in one space if i am much bigger and i need more
47:53
analytics juice maybe amplitude makes sense on top of this or something like that helps me pipe data into more dashboards and do less etl for me then i would get into segment and then once you get into experimentation obviously have to shout
48:08
out to epo i think they've really reinstrumented a lot of the dashboards that i would have normally had to do in experimentation projects so i usually look at something like epo to just automate the decision-making
48:20
flow awesome i think we're both small investors in appo uh big fans but a little bit biased but yeah it's a next airbnb team that built it so it's cool shifting a bit from metrics and data to just growth teams in general
48:36
maybe first question is just how do you recommend companies set up a growth team in the early days and then over time yeah so i can talk about how growth was set up at gojek as an example which i think is probably the best practice so
48:51
we didn't really know that what growth was at that time but we knew there were obvious gaps to fill so because we had grown so quickly the core product team was still making the core product features like as simple as
49:05
like phone number masking that wasn't a thing yet like you had access to your driver's phone number it's probably not a great thing it's probably part of the core functionality and we need to fill that gap at the same time growth was still
49:18
necessary because you have all these users trying to use the product that aren't quite getting there so things like figuring out what sms provider we should use to send the otp to this user who is signing up from this telco provider
49:37
that was a growth objective that like isn't necessarily core feature work but was a gap to fill given the onboarding and sms success delivery rates things like telling the driver if this was a brand new customer because at this
49:51
point in time drivers had taken thousands of rides and they assumed every single customer knew how go-jek worked when maybe they didn't and so the we knew that the protocol was that a power user would know they would make
50:05
an order and they would just wait they would wait somewhere they would keep an eye out for a driver and then they would get on the motorcycle and go but for a brand new user are you supposed to walk to the driver like are you supposed to find them it's
50:19
unclear to his brand new uneducated new user how to use the product and so first time user experience could have been a terrible one where they went and walked off and then the driver came to the pickup point and they couldn't find them
50:33
so it was all these like small acquisition adoption and engagement use cases that growth was filling the gap on and eventually we embedded our growth i would say product managers at the time into these teams and they
50:48
ended up kind of synthesizing what growth was as a full-time role eventually becoming pms who own specific parts of the product stack so in your experience and i hear this a lot is your first growth person shouldn't just
51:02
come in and figure out what to work on you should understand here's where we need growth help let's find somebody to tackle it versus come help us figure out what to do to drive growth is that how you've seen it exactly i think it's just
51:13
setting the bar like too high to expect someone to come in and model everything like again there are physics in place it's very hard to move everything so it's really about having someone who already has all of this data knows
51:28
where the biggest gaps are doesn't have to start from scratch and figure this out and then just picks some small space to work on that they know is workable do you have strong opinions about growth being integrated the way that you
51:41
described where growth pm is basically has a cross-functional team basically as the pm versus kind of a separate growth team that's off to the side yeah i think it can work as a separate growth team to the side if the company is truly
51:55
like head over heels tripping on insane product market fit like if there's insane product market fit and you are really scrambling to do core feature stacks then maybe a growth team to come and be clean up is fine like we really called
52:12
ourselves like we're the cleanup crew we pick up the pieces that were left behind we connect the dots like you forgot to plug this in we'll plug it in for you but we were a team of like lots of stats heavy people so a lot of my team were like statistics graduates we cared a lot about looking at numbers and odds and probabilities because it really is a numbers game at that scale you could work on anything and everything would probably do something
52:37
but what was the thing that would make the most impact now and unlock us for the future i was going to ask you folks to look for when they're hiring an early growth person is that what you find just stats data kind of person yep you have to have someone who knows how to run the numbers right if you're looking at ratios of conversion rates but you don't realize that this ratio is of a much smaller base size you're going to make the wrong decision
53:05
so someone who is intuitively good at statistics they know how to do sampling appropriately they know what selection bias is like the worst possible thing is to have a growth person who thinks they are doing the right thing and is measuring things wrong and then focusing on the wrong areas do you find that it's often easier or better to hire a young up-and-coming person or find someone that's got a bunch of experience for your first growth hire
53:34
i would hire someone who is willing to take intro to statistics course and it doesn't matter like if they've had the experience like go wild or not i think it really is like can they focus on the right opportunity rather than the most
53:50
flashy thing and i think both profiles can come under that got it and then what do you do in a hiring process for someone like this what kind of things do you suggest founders look for yep i actually look for that first
54:03
principle bias so i'll give people case studies of like here's what we see how do you know that this is true and then i have them set up an experiment design i want to see that they are sampling randomly not that
54:16
they're like i'm going to build this feature and launch it and of course it's going to work i want to see that they are taking a measured deliberate approach to considering like why someone might do this or what tools are available so a growth
54:30
team can go terribly wrong when they just try to like onboard a bunch of brand new tools that don't integrate well and it takes six months to integrate fully and then they get nothing done for six months like everything and growth is an opportunity
54:43
cost of time trade off with what you could have been doing to the product in that time so we buy biased towards like really quick hacky things like in the early days of gojek growth i think our our first real growth experiment we were
54:59
actually still the data team at this time was to connect a quick python script to the twilio api that we had access to and we smsed a bunch of drivers through a csv that we uploaded that said like hey your acceptance rate is really low
55:18
you're not supposed to do that please accept all the rides that you were getting and that actually increased acceptance rates by two percent across the board and it when we looked deeper into that data it did um even more so
55:29
for brand new drivers and so we then worked with the data driver onboarding team so they could better facilitate the onboarding experience for their drivers for the interview question that you described like an experiment design
55:41
question do you give that as a as a project where they can't have time to work on it or is it a live thing okay yeah i don't think live works really well uh for these case studies like i want to see people put in the time and the work
55:55
to do something to the best of their ability and of course we asked them like hey you have five days we expect you to spend probably like four hours on this so if you don't have four hours within these five days let us know
56:09
so we're pretty careful about giving them the appropriate amount of time to do it at the level of quality that we would have expected if they were to work here full time so give them those four hours we want to see like do they google if they can't
56:24
figure it out right now like let's see them google it we'll ask them like what approaches they took how do they figure this out and we like to hear people say that they literally had to google this and read a bunch of white papers like i do that as well for people trying to design one of these for themselves do you have a question that you've retired that you could share or something that would help somebody design their own kind of prompt
56:49
yeah i can give you a template after this fall amazing we'll include that in the show notes easy peasy amazing okay a last topic that i wanted to cover is a very cool thing that you're involved in it's a non-profit that you started
57:02
called generation girl and i think the mission is to help women and young girls get into stem so i'd love to hear about this program how you got into it what it's all about and then also just how listeners can help support what you're
57:14
doing absolutely yes generation girl is very near and dear to my heart so i co-founded this with a couple of amazing women who were also at gojek but are now full-time at generation girl so this really stemmed from us repeatedly getting
57:30
annoying comments about working in stem so things like you can't possibly be the engineer on this project like you look like you like makeup and stuff and we were like yes i absolutely love makeup but i also am badass at writing
57:45
swift code so step aside so having experienced a lot of the kind of misrepresentation of what an engineer should look like or should like i think we really look to um legally blonde is one of my favorite movies that represents you can take the powers that you have whether you like engineering or design or data and you can be whoever you want and still kick ass at it so a lot of the women that we support we're actually happy if they
58:20
go into one of our classes and they say actually i don't like engineering that's great that's agency and empowerment that they got to make that decision for themselves without any cultural biases or social
58:33
pressure telling them that they should feel this way so we offer free classes for girls 12 to 17. we have college classes we partner with teachers about how to teach stem topics especially in areas where they don't have laptops for every student
58:50
like how do you teach how to use figma and things like that so people can definitely support us and reach out to us we have a paypal on our website take a look can you share some of the impact that you've seen from this what
59:02
other numbers you can share anything that you can share around with what the organizations have done so we've already had uh several thousand students go through generation girl summer clubs and programs and classes so we have an event
59:15
every week we have a full summer club that's every single day for two weeks every summer and every winter we have partnerships with some of the biggest tech companies in indonesia where we partner students with engineers and they work on projects together and most recently we're part of the mit solve program with our new initiative class so plus we're creating a free to use site for teachers so right now we have partnered with a handful of
59:43
universities in indonesia both in rural and city of jakarta where teachers can now have the knowledge and material to explain newer concepts that maybe they're less familiar with because startup world changes rapidly how you develop changes
59:59
rapidly so this is one thing that we're most excited about because every teacher impacts thousands of students a year and being able to teach the teachers and give them the resources that they need is something that's really important
1:00:11
it's incredible it's currently just in southeast asia is that right only in indonesia because frankly this is where this is where everyone needs the most support i mean globally stem is not well received or welcoming at all to women i think it's
1:00:28
gotten worse over the past few decades like to below 18 of college graduates are women and computer science so we're really trying to reach the youngest generation because that's when you are told or informed that computer
1:00:44
science is for specific types of people it's really sad to hear that it's heading in the wrong direction what do you think is contributing to that i think there is still a lot of this mental model of what a computer scientist is able to do and how much support they're given so it's been shown in studies that at the youngest generation middle school high school you are more likely to be given introductory stem classes as a male than as a female
1:01:14
so women just aren't targeted for stem at that younger age and so when they enter the high school or college classes for computer science they're way behind and that does not feel good no one likes to be like the worst in the class and so
1:01:30
it's more likely that you'll drop out we've seen studies at carnegie mellon that actually would create introductory computer science classes before the college class starts and for the women who did join those classes they actually graduated at
1:01:47
similar rates as their male counterparts so it's really setting them up for success if folks want to help you said that there's a paypal page is there any other sort of action people can take yes enterprise software we love to teach
1:02:01
ios development licensed software we have hundreds of students a year so let us know awesome and they can reach you on generationgirl.com generationgirl.org crystal thank you so much for being here i've taken enough of
1:02:15
your time two last quick questions where can folks find you online if they want to reach out and then other than the generation girl chat we just had is there any other way folks can be helpful to you yes please find me at crystalwija.com you can reach out to me and my email is there listeners please do instrumentation correctly please don't trap your kpis please track your user journeys and experiences we'll have much funner things to talk
1:02:42
about if you do that amazing psa thank you so much crystal thanks lenny this was a blast thank you so much for listening if you found this valuable you can subscribe to the show on apple podcast spotify or your favorite podcast
1:02:55
app also please consider giving us a rating or leaving a review as that really helps other listeners find the podcast you can find all past episodes or learn more about the show at lennyspodcast.com see you in the next episode