0:00
From 2010 to 2020, we had the golden years of deterministic matching where it was very easy to run an ad and understand with precision who installed the app.
Search this video
0:06
Maybe you didn't know their name, but you actually would know their IDFA and you could tie that to their PII.
0:12
You can't do that anymore.
0:15
So, what that means is these ad networks are becoming more complex, sophisticated, and interesting, right at the same time that it's harder for marketers to really understand how they're spending money.
0:25
And so I am paying a lot of attention to how marketers make decisions with probabilistic data because most of the work that I'm doing now is actually saying, well, given that we don't have determinist data about a per certain audience or where somebody came from, how can I find other information that will create a model for 30% of the population and we can use that to extrapolate to a hundred.
0:48
Welcome to Lenny's Podcast, where I interview you world-class product leaders and growth experts to learn from their hardwood experiences building and growing today's most successful products.
0:57
Today my guest is Austin Hay.
1:02
Austin is one of the smartest people in the world on the field of MarTech, aka Marketing Technology.
1:08
He's advised companies like Notion, Airbnb, Walmart, Postmates, Robinhood, even Pete's Coffee and Mars on their MarTech strategy and tactics.
1:14
He's currently head of marketing technology at Ramp.
1:19
Before that, he was VP of business operations at Runway.
1:19
Before that, he was VP of growth at mParticle and the fourth employee at the Unicorn Branch Metrics.
1:24
He's also a teacher at Reforge on this very topic of MarTech.
1:30
In our conversation, Austin explains what exactly is MarTech, how it fits into your growth organization when you need to hire a MarTech person and what to look for plus his favorite interview questions.
1:41
Also, his favorite tools, frameworks, team structures, and emerging platforms that he's most excited about.
1:45
This episode is for anyone who's responsible for growth and is curious about ways to optimize your approach and how marketing technology fits into that.
1:55
Enjoy this episode with Austin Hay after a short word from our sponsors.
2:02
Today's episode is brought to you by OneSchema, the embeddable CSV importer for SaaS.
2:02
Customers always seem to want to give you their data in the messiest possible CSV file.
2:08
And building a spreadsheet importer becomes a never ending sync for your engineering and support resources.
2:16
You keep adding features to your spreadsheet importer, but customers keep running into issues.
2:21
Six months later, you're fixing yet another date conversion edge case bug.
2:21
Most tools aren't built for handling messy data, but OneSchema is.
2:25
Companies like Scale AI and PAVE are using OneSchema to make it fast and easy to launch delightful spreadsheet import experiences from embeddable CSV import to importing CSVs from an SFTP folder on a recurring basis.
2:42
Spreadsheet import is such an awful experience in so many products.
2:42
Customers get frustrated by useless messages like error on line 53 and never end up getting started with your product.
2:52
OneSchema intelligently corrects messy data so that your customers don't have to spend hours in Excel just to get started with your product.
2:56
For listeners of this podcast, OneSchema is offering a $1,000 discount. Learn more at oneschema. co/lenny.
3:08
This episode is brought to you by Mixpanel.
3:08
Get deep insights into what your users are doing at every stage of the funnel at a fair price that scales as you grow.
3:14
Mixpanel gives you quick answers about your users from awareness to acquisition through retention, and by capturing website activity, ad data, and multi-touch attribution, right in Mixpanel, you can improve every aspect of the full user funnel.
3:29
Powered by first party behavioral data instead of third party cookies.
3:34
Mixpanel is built to be more powerful and easier to use than Google Analytics.
3:38
Explore plans for teams of every size and see what Mixpanel can do for you at mixpanel. com/friends/lenny.
3:44
And while you're at it, they're also hiring.
3:44
So, check it out at mixpanel. com/friends/lenny.
3:51
Austin, thank you so much for being here. Welcome to the podcast. Lenny.
4:02
Thank you so much for having me.
4:02
We are going to get super nerdy today and we're going to dive deep into the very cool field of MarTech.
4:07
How excited are you about us chatting about MarTech? I'm so excited.
4:12
Because it seems like you might be one of the first people in product and growth to talk about MarTech. Wow, okay.
4:21
That makes me even more excited.
4:21
Yeah, it's something that I haven't fully understood and so I'm excited to dig real deep.
4:25
So, let's start with just the basics.
4:30
What exactly is MarTech and then what does someone who is in MarTech do? Such a good question.
4:35
Because marketing technology is like this very amorphous, cross-functional discipline that lives at the crossroads of product and growth and engineering and marketing.
4:45
It brings together processes and systems from a wide range of disciplines.
4:45
and marketing. It brings together processes and systems from a wide range of disciplines. And I think really the way to think about marketing technology is it's a product manager whose
4:58
specific role and focus is the system or the third party or first party platform because marketing technology can mean a collection of third party tools, which is a lot of people think, but as a company scales and grows actually it could include a collection of first party homegrown solutions that you build yourself with or in addition to third party. So, I like to think about marketing
5:14
So, I like to think about marketing technology more as one piece is people and process and the other is the system and the platform.
5:24
And that probably sounds pretty familiar to what a lot of product people think about their world as, and that's how I define MarTech.
5:28
And then you asked this other question around what exactly the role of somebody in MarTech, and maybe we'll talk about this a little later, but it's such a function of the size and the stage of the company that you're at.
5:36
At Airbnb, I would say Dmitri who you might've worked with was the MarTech guide.
5:41
He managed a lot of our Airbnb's, the first and third party tools.
5:45
Airbnb at that size was, I don't know, maybe 800 people or so.
5:50
And so it makes sense to have a function with product and engineering resources.
5:54
A small startup for example, when I was working with Siqi, we were just talking about this at Runway, there was no such thing as MarTech.
5:59
There was me and Tanner and Siqi standing up tools and using them because you just have to use the tools to get the job done.
6:04
And so I would say on the spectrum of what is MarTech, you really have to look at the size and the stages of the company and as you grow you start to see it become more refined or pronounced.
6:16
So, if someone listening to this that has done growth or has a growth PM may be like, oh, but this is sort of what I do.
6:21
What is the difference between someone that just runs growth or has a growth team versus someone that's specifically a MarTech person?
6:29
At some levels there's maybe no difference.
6:29
There's a lot of startups I would say are 30 people or less where you have a growth team and your growth acquisition person is using a CDP to send data to their ad network to run their ads because that's part of their job and maybe they are the MarTech person.
6:46
And actually you find a lot of people who consider themselves MarTech professionals now having started in growth or user acquisition roles because they had to just use tools in order to get their jobs done.
6:55
But what I would say is as a company grows and scales, it moves from being a community or village driven aspect of your products to being something that's centrally owned.
7:05
If you're a startup, again, like 30 to 40 people, everybody might chip in to manage your CDP or use Amplitude or build a first party solution on top of those.
7:16
It's a mixture of first and third party tools and engineering and product and marketing all work together on it.
7:20
That doesn't scale though.
7:20
As you cross a hundred to 200 people, somebody has to be responsible for knowing how data flows through tools, how it's worked, what's the schema.
7:29
And that's not even considering procurement and legal stuff.
7:29
You have infinite liability if you don't manage your contracts well.
7:34
And so usually around I would call it a hundred to 150 people is the critical mass where you can't just have a village approach to systems and tools much like in the IT org, if it was a village approach to SSO businesses would be in a lot of danger.
7:52
That's where you typically start to see the question of, all right, we need a systems and tools person.
7:56
We need somebody to manage these systems and manage that platform.
8:00
And there's a variety of ways it can go.
8:00
I've seen it go just into pure product that's with a product operations org and a product ops person actually will manage a lot of third and first party tools.
8:09
I've seen it go into the IT org, Walmart for example, at a really big scale.
8:14
They had a MarProd function which was marketing products.
8:14
It was product within the marketing function or product that was designed to serve marketing.
8:18
And then of course you can have more traditional routes like you can have marketing technology as a single standalone unit or business technology as a standalone unit.
8:27
Some of this depends too on whether the business is B2C versus B2B.
8:32
Classically in a B2B business you see it in rev ops or some types of systems role because you have to serve not only users coming into your funnel, but then the businesses that you're serving afterwards.
8:43
That's also where you typically see tools like Salesforce coming into play and more advanced CRMs.
8:48
In a B2C business, your user funnel is actually really simple acquiring users and you're getting them into your product and then product is taking them over.
8:57
There's no additional CRM, so usually your CDP is the source of truth and that's where you might actually see marketing technology fit in with growth a lot more.
9:03
Just some examples like at Postmates, I worked for them for a long time as a consultant.
9:08
Marketing technology was just part of growth.
9:12
We had a director of growth even before that, Siqi Chen who's the CEO of runway, and I guess you were his first manager as I just learned, he was the first VP of growth and marketing technology was just part of growth and product owned that as a system.
9:29
As a different example though, at Ramp we're big enough and we're a B2B company, but we have a B2C top of funnel where we try to acquire users and get them to fill out our application to get a credit card.
9:39
We have a distinct revenue operations team that's broken into business technology and marketing technology.
9:44
So, there's lots of flavors of how it can exist.
9:44
I think that's kind of the interesting and fun part of Marketing Tech is that it's not just one single version of the world that you apply to many companies, there's like a million variations that I've seen and they all kind of look to solve the same problem.
9:57
So, to make it even more specific and really simple for people to think about what someone in MarTech does, essentially it's using technology and tools to drive growth.
10:07
Is that a simple way of thinking about this one specific roles? Totally. That's exactly right.
10:11
And I have this adage I always say, which is tools are just meant to solve problems.
10:17
And the problem set for marketing technologists and business technologists is you focus on the tools.
10:21
And so when someone currently say listening doesn't have a MarTech person and they're thinking about, hey, is this a gap we have?
10:30
What is that slice of work that a MarTech person would take if they currently have say a growth team or a growth PM that's leading growth and a growth team around them?
10:40
This comes up all the time, by the way, I talk to businesses every year that have this problem of we have a growth team, we're growing pretty fast.
10:46
We have a guy that we hired, usually an engineer who stood up all these tools for us.
10:50
Or it could be gal too just to be clear, but this person has been here for two years and knows all of our systems really well, but now they're becoming overwhelmed.
11:00
They don't have enough time.
11:00
The systems are too complex.
11:06
This is the flavor of story that I hear so often around startups who have hired a great growth person and managed tools and systems, but at some point they reach that point in time where it's no longer manageable by one person or even a set of people.
11:16
And that slice of works looks like setting up new tools, building new tools on top of them because a lot of times you'll take a third party tool, call it like a segment or an amplitude, and you'll build tooling in your own stack behind it to power something much more advanced.
11:34
And everybody thinks that marketing technology is just the third party tools, but actually it's designing, architecting and building that stuff on top of your third party tools.
11:38
That's how you actually have a lot of velocity is thinking about not just build versus buy. It's build and buy now.
11:48
So, you buy the tool to get 90% of the way there and then you build the cool thing on top with the other 10%.
11:52
And so that architecting decision usually falls on this person.
11:52
The one really unsexy part of it, which I tend to love because it's really high leverage is the contract part.
12:05
When you start out as a business, you sign any contract you want with a third party because you're just trying to get going.
12:09
You have much bigger problems, product market fit, staying alive, runway.
12:13
But at some point as you scale and you're starting to make money, now you start to care more about not just how much money you're making but how much money you're losing usually from contrasting SaaS tools.
12:22
And so that's where you start to have more scrutiny around what types of deals are we signing, what are the terms?
12:26
Do we have liability exposure?
12:30
What's it going to cost us if we actually scale?
12:30
And it's great that we have this cool rate at 500 MTUs, what happens when we have a million MPUs?
12:35
So, I worked at mParticle, which was a CDP provider for a long time and I was their VP of growth and part of their SaaS vendor strategy is like, how can we design these cost structures in a way so that at the company scales we make more money?
12:51
That's just part of the business.
12:51
And so if you have that mindset of, well, I'm looking out for the business not just now, but two 30 years in the future, that's where you can also have a lot of value from a systems or marketing technologist.
13:05
Maybe a sign that you should start thinking about a MarTech person on a growth team is what I'm hearing is you're starting to accumulate all these different tools and maybe there's a sense that you could be a lot more efficient in connecting data and the backend infrastructure for how you think about growth and how you drive growth and measure growth.
13:24
Yeah, efficiency and pain.
13:24
I would say pain drives people more.
13:24
It's like, hey, we can't do something because nobody knows this thing.
13:29
We can't do something because we don't know the best way to set up these tools or to change these tools or we can't even move forward where a business plan because we're worried that changing our tools might have an impact.
13:44
And usually this is related to email marketing tools and data tools, so like CDPs and folks like Braze interval and just because a lot of times your email is the thing driving recurring customers to come back to your product and use it.
13:54
So, you can't actually sometimes make the changes you want without understanding how something was set up in the first place.
14:03
You talked about where this person would live in the organization.
14:03
There's all these different places.
14:07
I talked about revenue team, maybe the ops team, maybe growth team, marketing team.
14:11
What's your general advice for who should lead the hiring of this role and also just roughly who should they report to?
14:18
So, I have not to shamelessly plug my Reforge course in the fall, but I'm going to be shamelessly plug my Reforge course in the fall.
14:28
We have this awesome matrix that we built that shows where this person should live, what they do, who they should report into, and it's all part of the fall course if you want like the deep dive into it.
14:37
There's going to be a section on it, but just the gist of it is I first like to break it down into two dimensions.
14:41
First is a B2C company or B2B company.
14:46
And then the second dimension is how important is it to you that this person report into a specific function or not?
14:53
So, first with B2C and really maybe a simpler version of that is centralized versus decentralized.
14:58
So, we have B2C, B2B, centralized, decentralized.
15:04
In a B2C organization I think actually thinks it's quite simple.
15:08
Most of the time your tools, your marketing tools are intended to help the growth team.
15:14
The growth team has a job to be done, which is to spur user growth and tools are just meant to solve the problem.
15:18
So, marketing technology's job is to serve the growth team.
15:18
Now it obviously serves product and analytics and data, but its key stakeholder and customer is the marketing or growth function.
15:28
And so I think it makes a lot of sense that if you're designing an org under a CMO or a marketing person, you put marketing technology alongside your head of growth or maybe reporting into your head of growth depending on the seniority of the person.
15:36
And that works quite well.
15:40
The key thing there is you just want to make sure that this marketing technology person is a really strong technical architect or some type of technical operator because they're going to be your representation to the product in engineering orgs.
15:53
Now, some people take a little bit of a slight twist on that.
15:53
They say, hey, I have a product manager who manages growth that comes from the PM side.
15:57
You could have a platform PM that serves the same thing in MarTech and they're responsible for all internal platform systems.
16:06
And then you get into questions of does that belong in product ops or not?
16:06
And I'm not going to go there.
16:11
But for B2C, that's the centralized function.
16:11
For B2C decentralized, what you do instead is you just say like, hey, we're going to have one of these systems, people in every org.
16:21
Product is going to have a product ops person and growth is going to have a growth ops person, engineering will have engineering ops, and then we kind of divide the lines based on what tools they're managing.
16:31
I generally don't see that working very well just because as you add more operational people, it just creates more systems.
16:37
And so unless you're a massive company where you need that type of scale, I think most startups should avoid that decentralized model.
16:47
And then for B2B, I think B2B is really messy because not only do you have pure B2B where you're only selling to enterprises, but you have this concept of B2B2C, which is where you're actually selling to users and to businesses sometimes at the top of the funnel and the bottom, but also sometimes at the same time like notion.
17:00
Notion sells to users so, they have a little growth acquisition funnel at the top, but then they also sell to businesses.
17:09
And I find there's really, again, there's two ways decentralized or centralized actually at Ramp we've gone back and forth between the two models.
17:14
We started centralized with the Rev ops group, we decentralized it and put marketing technology into the CMO org and now we're rolling it back into the revenue operations org Largely has to do with who is our customer?
17:28
Whose problems are we solving and where are resources allocated?
17:28
Because if you have a decentralized model, then you run the risk of having to have lots of resources decentralized across the team.
17:37
And the question is, can that function actually get work done or resources spread too thin and the priorities on align that it makes it challenging to get work done.
17:46
And yeah, I would just say especially on B2B, for people out there listening, there is no right answer.
17:52
And I even think that marketing technology could live in product, it could also live in engineering.
17:58
Some of this has to do with who is the leader of this function.
17:58
If it blends more towards ops, meaning managing processes and systems, then yeah, maybe you want to decentralize it and keep it in its representative function.
18:10
If you have a really technical leader who was an architect or a PM that might indicate where that person should actually be leading their team.
18:15
So, it's very case specific, which I know is a terrible answer, but it's the way it is. Makes total sense.
18:24
If someone were to hire someone like in Austin, are you doing the work yourself?
18:28
Are you an IC for quite a while or do you end up building a team, say engineers that are building some of this infrastructure, how does that usually play out?
18:37
I think all marketing technologists at some level are ICs.
18:37
I think it's a great job personally, because I get to be an IC and a manager.
18:43
You have to be an IC in that, you are the most senior technical expert on all first party and third party systems.
18:47
So, you have to know really well how third party tools work and you don't know that without doing the work yourself.
18:53
So, I do find that some of the best marketing technologists have at least at some point in the last five years, been an operator and expert managing tools and systems.
19:03
And then usually the teams are small and super cross-functional.
19:07
So, what I would say is more important to look for than how many people has this person managed is how well can they manage upward, laterally and downward because they're going to have to go talk to the head of rev ops if they want to change something in Salesforce, they're going to have to talk to the VP of product if they want to make a big platform change that touches something else.
19:27
They're going to be relying constantly on data resources from their head of data.
19:27
So, I think that this person, the secret sauce is more of how good of a cross-functional team player are they.
19:36
I almost view them like a true quarterback every [inaudible] says people are quarterbacks.
19:41
But really marketing technology because it lives between so many departments, it plays that role of having to call plays and pull on different departments.
19:50
And because it sounds like you don't have a team to do some of these things and you need to convince people to help you out. Totally. Yeah.
19:54
It's a game of persuasion and salesmanship.
19:58
You have to convince people why the problems are big and especially as you get bigger, a lot of the decisions or problems of marketing technology are not about rapidly making a huge transformation.
20:09
It's slow transformation that can have big implications.
20:09
I'll just give you one example.
20:13
Like lots of big companies I talked to have two CDPs or two attribution tools and it's like there's the cost problem.
20:19
How do we get rid of this secondary tool to reduce the cost?
20:25
Maybe it's a million dollars, but there's also the complexity and decision-making problem.
20:30
How do we make people move and work faster by not having the complexity of asking, which tool do I use in such a simple decision?
20:35
And then you get to a really big scale at Walmart where your problem isn't even.
20:40
How do we consolidate the stack and make it so tools that are helpful for people, but how do we prevent from getting back to that state?
20:49
How do we put safeguards in place to make sure people actually have access to the tools that they want and can solve their problems?
20:53
But we're not introducing duplicative tech into our organization because a really well-known, sorry to put SaaS vendors on the spot here, but well-known SaaS vendor plays the land and expand motion.
21:02
You get in small and then you grow your business.
21:06
Well, that's a distinct problem for businesses that are trying to control costs and simplify the way the world works.
21:12
I want to talk about tools that you recommend and use most often, but I'm thinking maybe we start with a different question, which is around just what does your day look like as a MarTech person?
21:22
What are you doing day to day and from the lens of your growth PM listening or a leader listening and what could this person do for me and how much leverage can I get if I were to find a MarTech person?
21:36
There's half of marketing technology, which I would call somewhat administrative and high leverage.
21:42
It's managing PI requests and PI technology, managing administrative stuff like contracts and admittance to tools and permissions.
21:49
This is all at a big company scale.
21:49
You probably don't do this when you're a small company, but that stuff matters because give you an example, you give edit access to somebody who wants HubSpot and they send a fake email test to a million people and now you're on Twitter being embarrassed as a company.
22:07
It's like- Does that happen to you? It hasn't happen to me.
22:12
But I've gotten the emails from certain companies where it's like, this is a test and it came from an intern. Yeah, same.
22:20
You're like, that's just permissioning gone wrong.
22:21
So, I think a big part of the role is designing systems that are automated to handle that stuff because ideally you don't want to be sitting around in your computer all day clicking one conductor request to approve permissions.
22:30
You should look at the role, look at the experience of tenure and department and make a decision about which accesses you get.
22:36
So, automating that is a big part of my job.
22:40
The manual part of my job, which I feel like is actually really fun, is again the designing systems and contracts for the future.
22:44
So, it's about how do we design a system and create a vision and persuade people about what our system technology can look like over the course of one to two years, the time span that I usually look at.
22:59
And then how do you change state from then to now?
22:59
Some of that has to bring in financials and contracts.
23:05
That's where this plays a role.
23:05
What are our contract terms today?
23:05
What's the price we're paying?
23:10
What is our growth going to be?
23:10
Can we build a financial model to show how much it's going to cost us both in terms of operational efficiency and actual real fixed and variable costs to end up in that state?
23:21
And then how do I create a graceful argument to persuade people that we should spend engineering time and resources?
23:26
And usually it nets out pretty clear.
23:30
It's like if it's less than a certain amount, how do you justify spending any engineering time on it?
23:34
You have to wait for the problem to become big enough.
23:34
But then back to your other point around how do I give growth managers out there something useful.
23:38
I would say the big thing that people forget in an early stage of a company's lifetime is that the company will outlast you, hopefully.
23:49
You will not be the last growth manager unless the company fails.
23:49
So, I tend to take a little bit of a different approach than most, which is like I think you should always be thinking about the future.
24:00
That doesn't necessarily mean you should make design choices that over index towards the future so much that you miss product market fit or you make poor product decisions.
24:09
But when you set up tools and you pick tools and you implement them, you should be thinking, what's going to happen a year from now if I don't change anything?
24:15
And is this going to be a catastrophic situation or not?
24:20
And then try to take actions to mitigate that risk.
24:25
Some examples are like if it's $2,000 to get SSO and two days to set it up and that prevents you from having a security problem where somebody downloads all your users, it seems like a great investment. And guess what?
24:35
Over time, if you don't do that, you're going to eventually have to hire an IT person to go and set up SSF for all your tools.
24:42
So, some of this is more of just being a good steward about managing first and third party tools with an eye towards the future.
24:47
It's always a trade-off, right?
24:47
Because the more time you spend when you're building product early in a company's lifetime, that time could be spent on other things.
24:56
So, if you waste it managing third party tools or setting up correctly, then maybe you miss out on a key product feature.
25:00
So, I think it is a tough balance to strike.
25:05
Coming back to the different kind of roles within the growth umbrella, if someone has someone leading paid growth let's say, and they're just like a paid growth person, do you also find a MarTech person to work alongside this person?
25:16
How connected would you be to someone that's just responsible for paid growth?
25:22
Maybe a key differentiator too.
25:22
We didn't talk about this in the beginning, but there's marketing technology and marketing operations.
25:27
So, in my mind, this is just my own kind of mental framework is marketing technology has tech in it.
25:37
So, it's usually an engineer or somebody with an engineering background doing that function.
25:41
Marketing operations is usually not always technical.
25:41
Maybe a systems analyst or business analyst could be somebody really, really smart, but they may not have an engineering background.
25:52
So, I think that's a key distinction too.
25:52
And you typically see that in B2B where you'll have mar ops function, which is setting up campaigns, sending email blasts, debugging, doing analytics work, SQL queries, all semi-technical work but not engineering based.
26:02
So, in my mind when we talk about marketing technology, I'm really thinking it as an engineering based role and even by background, I'm not a software engineer, but I was a civil engineer and I learned how to program and I went through a bunch of coding to get there.
26:17
So, that's my way into the engineering world.
26:17
And you typically find that a lot with marketing technologists in particular is they either are software engineers or they've gotten enough experience to moonlight as software engineers.
26:33
And so we get to this problem set of a user acquisition person.
26:33
How would they rely on a marketing technologist?
26:39
Well, I think the most superhuman user acquisition people out there are engineers and they don't need a marketing technologist because they set up the tool themself.
26:48
They know how the paid campaign runs and they just do it all.
26:48
And you'll typically find these super humans at small startups where the engineer is just told by the co-founder, hey, go figure out how Facebook ads work. And superhuman is born.
26:58
More often though that doesn't happen.
26:58
Or those people once they do it once, they never want to do it again.
27:04
So, you'll typically find the role split and that's the natural thing that happens.
27:10
As you scale, you divide responsibility and you'll see you'll have the person who's responsible for bidding and acquiring users and paying down those campaign costs.
27:20
Then you have the person who's in charge of how does it all work?
27:20
How do we get this thing to actually run?
27:24
And that's very similar to what we have at Ramp.
27:24
We have an amazing user acquisition team.
27:29
I know Sri Batchu was on here a while back.
27:29
He hired a guy named Cody Morgan at Ramp who has a user acquisition team.
27:34
And the way to think of it is, my job is to help support them in running all their campaign needs and when they have a directive from the CEO that says we need to improve CAC or change any of our metrics, it's my job to partner with them to help them do that.
27:48
And actually one of the coolest and most fun projects that we worked on early when I joined Ramped is we were optimizing.
27:52
We're trying to get top of funnel data all the way down to the bottom of the funnel and tie it with opportunity data so we could send that back to the ad network so that rather than optimizing your campaign off of when a user clicks a button on the website, you're actually optimizing it off of did the opportunity occur and what was the kind of ideal value for that opportunity?
28:12
And you're sending that data as a synthetic event back to Facebook and all those guys.
28:16
So, it can be really cool and super advanced stuff depending how deep on the funnel you get and how complex your business is.
28:25
So, you're generally not running campaigns of your own unsafe Facebook or AdWords.
28:29
You're mostly as a MarTech person supporting people who are doing that. Yeah. Awesome.
28:35
Helping them use tools and technologies to do it. Great.
28:38
Do people give you goals?
28:38
Are you responsible for growth goals of your own?
28:45
And in general, are MarTech people, should they have goals and growth goals on their plate or are they just there to support people who do?
28:49
Oh, that's a great question and I would like, maybe this is at the end of the podcast, we ask people about this because I would love to know what is a better version of goaling?
28:59
So, there's two ways that I've thought of it.
28:59
One is my goals are directly tied to the people I'm serving.
29:04
So, if user acquisition has, I mean we do, we have a growth goal and we have a CAC goal at Ramp.
29:09
So, my goals are tied to them, so I'm going to help make sure that that is achieved, but then there's also a cost and efficiency goal that I internally think is valuable.
29:20
Whether or not the business thinks it is valuable, it doesn't really matter.
29:24
I come from a sales background and I like to run lean and efficient teams, and so I'm always thinking to myself, how much were the tools when I came in, how much are they now?
29:34
Have I set us up for success so that as we grow, our cost per user or cost per seat comes down and how much more efficient are we because of that?
29:40
The ideal world is that you actually are growing as a business making more money, hiring more people, acquiring more users, and your total cost of tooling per person goes down. That's like the dream.
29:50
And there's lots of ways you can build that financial model, but I mean that's what I think most marketing technology leaders should strive for is to make sure that they're controlling costs over time because most businesses don't.
30:04
There can be some goals that are discreet in nature that are not cost-efficient, but more like net capability related.
30:09
So, it's like, hey, we want to design a first party system that's world-class that achieves these three goals, right?
30:16
Maybe you want to incorporate artificial intelligence into some part of our product platform and incorporate third party tools.
30:25
And those are more like discrete product goals.
30:25
In the same way that a business might launch an external product goal to launch a feature, they sometimes also might have internal product goals, clean up our revenue operations systems, make our email marketing system better.
30:37
In particular, email marketing is one I see come often a lot with small businesses and even medium-sized businesses where they'll have picked a tool at the start of the company's lifecycle and as the company has grown, they've outgrown that tool.
30:47
They need to move to a Braze or Marketo.
30:53
And so there'll be a big six-month initiative to say, we just got to switch. That's the goal.
30:58
We have to safely get off this small tool to a much bigger, more complex tool that's going to cost us more.
31:03
It's a lot more complex, but we need to do it without losing money.
31:08
That's usually the job of a MarTech person in some type of change transformation effort.
31:14
Perfect segue to where I wanted to go, which is tooling and your recommendations and favorite tools.
31:18
And so maybe we start with just what do you find as a good starting tool stack for people starting to think about MarTech and basically growth, and then what does it end up being generally?
31:29
In terms of stack, again, we think about B2B and B2C.
31:34
B2C I would say the stack was largely solved from 2017 to 2020.
31:34
We've had like a renaissance of the data architecture, so what I'm going to do is I'm dig through B2C then and now and then we can go B2B then and now. Great. Okay.
31:52
So, B2C, if you back up to 2016, 2017, you have segment and the rise of the CDP.
31:52
Consumer based businesses have to collect a user and tie a bunch of data to them and then track their actions to send it out to performance ad networks and email marketing tools and product analytics tools.
32:11
And so you would see this very commoditized stack.
32:11
It would be like CDP in the middle bunch of tools connected.
32:16
The promise of the CDP was you integrate one SDK, your engineers don't hate you send all the data to the other tools you can create audiences. Great. Lasted for a long time.
32:25
The thing about it though that I think really changed around 2020 is that the cost of ownership of warehousing became much cheaper.
32:32
And so 2021, you start getting to the place where it actually makes a lot of sense and is really easy to store all your data in a warehouse model all your data in the warehouse, and to do it without needing a vast data team.
32:42
I would say Airbnb was probably doing all this well before anybody else was, but they had the main advantage of a lot of money and a lot of resources.
32:52
So, now come 2020, it's cost-efficient to have a data team with your own warehouse and to manage data centrally in something like Snowflake.
33:03
So, now this question is like, okay, well we got to get data into the warehouse, but how do we move data around is totally different.
33:08
And that's what really led to the rise of reverse ETLs.
33:13
So, now you can actually build your own CDP and lots of businesses already have, I'm consulting with a well known financial trading platform a couple of months back, and they have a CDP, they have all this internal data in their warehouse, but they have not been able to activate it because it's pretty old architecture.
33:30
Everything's batch based end of the day.
33:30
What they need is a reverse ETL.
33:35
They don't need to take that data and just get it out into the world.
33:35
So, they need the reverse ETL component or the transformation component of a CDP.
33:39
And so I'd say now today when we think about B2C businesses, you can either go to the traditional route, buy CDP, hook up all your tools, third party.
33:49
I think that's a great move if you do not have a lot of engineering resources because you're not spending a ton of time and energy on a warehouse and all the modeling that comes with it, you're just spending time to implement one SDK.
34:00
I think if simplicity is the name of the game for your business, CDP, Centralized Stack, great move.
34:10
If you are an advanced engineering culture and you are cutting edge and you're going to do a bunch of modeling in DBT and you already have Snowflake, you should move towards a model of using a reverse ETL.
34:21
What it means is that there's a way to get your data into the warehouse and then how you activate it is completely independent from the CDP.
34:25
And so what that means is actually you can have lots of different variations of the stack.
34:29
You could use Amplitude as your CDP, collect all your data, stream it into Snowflake.
34:35
They actually now have an integration with Snowflake that lets you feed data directly out of Snowflake, and then you could use a reverse ETL to just pipe that data wherever you want.
34:44
There's a really good section though, again, sorry to self aggrandize, but there's a really good section in the Reforge module this fall that talks about what happens when you have multiple ways to move data.
34:54
You buy amplitude for your CDP and you're moving data to your warehouse.
34:58
Amplitude is a bunch of integrations, but you also have reverse ETL and you can move data out of your warehouse. Where do you choose?
35:02
And I would say a lot of businesses get in trouble when they don't have a methodology or a system for how and when to move data from one place to the other, so they just do it haphazardly, right?
35:13
And the key in systems management is you want to design a process for doing it some type of waterfall or mental model for when it makes sense to move data directly from Amplitude, which is the ingestion point of your data stream or from the warehouse where you can model it and make it better.
35:27
I think the key is just having a philosophy and approach.
35:32
There's not really one answer, but that's all B2C. So, B2B I would say ... Yeah, go ahead.
35:39
Before we move on to that one, you mentioned reverse ETLs.
35:41
What are some examples of products that are reverse ETLs so that people can look them up?
35:45
Yeah, I personally think the reverse ETL is a capability.
35:50
It's the ability to move data from a warehouse to a tool.
35:50
So, technically speaking, you'll find reverse ETLs in CDPs and as standalone products.
35:55
Segment has a reverse ETL function they just launched, and Particle has a reverse ETL function they just launched.
36:01
Rudder Stack, which is a CDP has always had a reverse ETL function where you can take warehouse data and move it to different cloud infrastructure.
36:10
Then there are distinct standalone products.
36:10
Census, which was back Bay 16Z and Hightouch are the two standalone reverse ETLs.
36:16
And like I said, I'm an investor in Hightouch, love their work, we use them at Ramp.
36:21
At the end of the day, you should pick tools because they help solve problems, not because of anything else.
36:25
So, we can come back to that if you want. Wonderful. Great, great. Yeah, that was perfect. Keep going. Okay.
36:34
Yeah, so we talked about B2B, or sorry, B2C.
36:39
B2B, I probably don't have as much history as say people who survived the dot-com crash in 2008.
36:39
I started really my career in B2B in 2014, so I'll share a little bit of my experience and I'm just hopefully just saying this because listeners may chime in and be like, oh man, this guy doesn't know what the hell he's talking about, which is totally fair game.
36:57
So, 2014 though, I remember working at Branch, I was working for our COO Mike Molinet, who's now at this really cool company called Thena, but at the time I was working for Mike, and as we talked about before, oftentimes growth stacks just appear because you're given a challenge.
37:16
And I remember sitting in this tiny room with Mike.
37:21
We were over in Palo Alto, right off the fills in Palo Alto in this tiny room.
37:26
It was boiling in the room like so hot we were sweating and we were mapping out on a whiteboard how we would design our first version of our system, like how we capture leads, how we get them into Salesforce, how we would email them with a little tool called Outreach at the time, that was still a startup.
37:41
And I'll send it to you after this if you want to show them to viewers, but it's so MVP, but it still models what a lot of people have today.
37:47
There's some ingestion point for your data.
37:52
There's Salesforce, there's some type of outbounding tool, there's an enrichment tool, and then a lot of other Jerry rig stuff hooked up to Salesforce.
37:57
And for the most part, that's how B2B still exists today as you have Salesforce and then the whole world and the universe revolves around Salesforce.
38:07
You just have more advanced tools, you have Gom and stuff like that.
38:11
I think the big change though, and what is really fascinating and has been fun to watch is in the last two, three years, you now have this whole rise of B2B2C, which takes all the complexity of the top of funnel user acquisition system and stuffs it right alongside your CRM and how you build an elegant system there in that space, I think is one of the most complicated and intricate pieces of being a MarTech person today.
38:33
And some of it just has to do with the data language.
38:38
Like all these B2C tools were designed with two objects, a user and an event.
38:43
And so if you're not a technologist, it's like object orientation is how you kind of think about the world.
38:48
There's only two concepts for the world in a user acquisition based system.
38:52
A user who's a person either anonymous or known coming into your website and the things that they do on your website or application, and you use all that data to acquire them or model them.
39:03
In a B2B business you have all that complexity, but at the end of the day, you might not really need it if all the person is doing is it's just the company is signing the contract and then you don't really care what happens afterwards.
39:12
You might track users and events inside your application, but it's not for the acquisition, it's for the retention of the user.
39:22
B2B2C is fascinating because you have all the complexity at the top, but then how and when do you tie a user to a company or some type of entity object, and what tools do you need to do that and where do they live in the system?
39:33
And do those tools actually have competing priorities?
39:33
Let me give you the greatest example of this that happened at Notion when I was consulting to them at Chris when I was consulting to them and at Ramp is having both HubSpot and Salesforce.
39:45
Both are CRMs, both have the ability to track users and companies, neither are CDPs.
39:45
And how you actually map the data from HubSpot to Salesforce kind of determines how much hell you're in, and there's really no good solution.
39:58
It's just like you have to figure out for yourself, how do you want to acquire use at the top of the funnel?
40:01
How do you merge them into the bottom of the funnel of the Salesforce?
40:05
And again, there are lots of options or versions of the world.
40:05
You could use Amplitude only and collect all your user and event data and then merge that into Salesforce directly.
40:15
You could collect all your data in Amplitude or Segment and then post that to HubSpot, which then posts that to Salesforce.
40:19
But of course, as you make these decisions, your systems becomes more complicated and more than one person can manage.
40:24
So, there's this trade-off between complexity and resources that you always have to juggle.
40:32
Today's episode is brought to you by Brave Search and their newest product, the Brave Search API, an independent global search index you can use to power your search or AI apps.
40:38
If your work involves AI, then you know how important new data is to train your LLMs and to power your AI applications.
40:48
You might be building an incredible AI product, but if you're using the same data sets as your competitors to train your models, you don't have much of an advantage.
40:53
Brave Search is the fastest growing search engine since Bing, and it's 100% independent from the big tech companies.
41:04
Its index features billions of pages of high quality data from real humans, and it's constantly updated thanks to being the default search engine in the Brave browser.
41:10
If you're building products with search capabilities, you're probably experiencing soaring API costs or lack of viable global alternatives to Bing or Google.
41:20
It's only going to become harder to afford these challenges.
41:25
The Brave Search API gives you access to its novel web scale data with competitive features, intuitive structuring and affordable costs.
41:31
AI devs will particularly benefit from data containing thorough coverage of recent events.
41:36
Lenny's Podcast listeners can get started testing the API for free at brave. com/lenny. That's brave. com/lenny.
41:42
There's this big question within B2B and B2C around how to do attribution.
41:51
Well, it's a never ending struggle.
41:51
I'm curious if you have any pro-tips or best practices or tools that you use to improve the way attribution happens at a company.
42:07
Actually, I listened to your pod on multi-touch attribution.
42:11
I'm forgetting who you were with at this point, but it was like I was loving it because it talked about MMM and MTA specifically.
42:19
Yeah, that was a newsletter post actually, not even a podcast. Yes.
42:21
So, back to our conversation around division of responsibility.
42:21
I'm not always the person you should talk to create an MMM model.
42:28
I'm not a data scientist.
42:28
I know how to make MMM models and I know what they are.
42:34
Can you explain MMM briefly? Mixed Media Modeling.
42:37
And MTA stands for Multi-touch Attribution and it's these two ways of measuring the world and marketing to understand how you should allocate resources to campaign spend.
42:47
MTA and MMM though are both underpinned by how you collect data.
42:47
They're both informed by the user object and the event objects that you collect on your website or your application that then lead to the data that data scientists use for MTA and MMM.
42:58
That's the connection between data and MarTech is often the tools and systems that we build and stand up and manage are what are used for these very complicated growth, experimentation and attribution results at the end of it.
43:14
And one of the most discreet things you can do for MTA, because I get this question all the time around, hey, do we need MTA?
43:18
What should I do first touch or last touch? Should I do both?
43:22
And there's actually really, I can send you this guide, but there's six or seven things you can do to basically futureproof yourself from needing either one.
43:28
Because most businesses either start with first touch or last touch and then eventually want to move to a multi-touch attribution model.
43:38
And for those who don't know what that is, first touch is where you kind of collect the data about where somebody first came from.
43:46
Last touch is where you collect the data about where the person last came from.
43:46
So, an example, would this be like if I went to Lenny's Newsletter from a Google ad and that's all he has?
43:50
That would be my first touch and my last touch.
43:56
If I first came from a Google ad to Lenny's Podcast, but then later I came from a Facebook ad or I don't know direct, then that would be my last touch.
44:06
And so it's this question of does the Google original first Google Channel get credit or does the second one the Facebook or direct get credit?
44:12
And the first touch attribution model, a hundred percent goes to the first channel and the last touch attribution model, a hundred percent credit goes to the last touch.
44:19
And a mixed attribution model or multi-touch attribution model, you're trying to figure out how to split the difference.
44:23
And usually the evolution for businesses is they start with first touch or last touch, then they go to splitting it literally 50/50.
44:32
And then somebody gets angry because they're not getting enough credit and they say, we've got to go to MTA.
44:37
And there are both first party solutions for that and third party solutions for MTA.
44:42
But back to the main thing, the main point is if you think about what you're collecting, this is for website businesses, you're collecting the referrer, like in the URL where the person's coming from, and you need any UTMs associated with that person.
44:57
And you also need any parameters from the advertising networks that might give them the ability to counter a conversion.
45:03
Every ad network out there has little things they stuff into your URL that tell you that you came from them.
45:09
Facebook has FVP, FPID, they sometimes encode it.
45:15
Google has this thing called Google Click ID, which is just a really long string of characters that don't matter unless you know how to decode it.
45:19
But all advertisers, and for the longest time advertising worked by putting parameters in URLs, pushing somebody through to your website, collecting those parameters and then passing it back to the ad network so they could get credit for it.
45:35
And so in my mind, the best practice that everybody should stand up from day one is to basically design the system for MTA and then use whatever makes sense as you grow.
45:45
And so the way that I typically recommend to people is like imagine when a user comes to your website, you collect the URL, collect the referring URL, collect all the additional marketing parameters that you might want, [inaudible], TikTok ID, Microsoft ID, you should just make a list of them.
45:59
And if you don't have that list, I can give them to you.
46:03
And then you should collect all UTMs.
46:03
So, in the URL, you're going to have UTM campaign, UTM medium.
46:08
Most marketers use this to note what the campaign type was.
46:08
Now the thing is that UTM is only going to be specific to the moment in time that the person came to your website.
46:15
So, back to example about Lenny's podcast.
46:20
If I come to LE's podcast and I came from a Google ad, then my UTM is only for that Google ad.
46:25
So, I have a Google Click ID and I have a UTM.
46:31
So, what you're got to do is you've got to store those parameters locally on the device.
46:31
Either was a browser or whatever.
46:35
You got to store it as UTM first campaign, UTM last campaign.
46:43
And what you do is every single time that a person comes, you replace the last campaign or the last value with the one that's there.
46:49
So, say the last one was Facebook and then I come later from a direct mailer ad you replace the UTM last medium with the new one.
46:54
Now what's happening if you're using third party tools is that you're collecting this user information when the person's on the website, you're going to collect it both as a user attribute and as an event that way.
47:11
What's going to happen on the backend for your data warehouse team is they're going to see a user profile that has both the first attribution information and the last attribution information.
47:19
And then for all the stuff in the middle, you're firing off a page view event with first and last, where the last might deviate if there were multiple steps in the middle.
47:25
So, what they can do is they can just coalesce over all the last UTMs they've seen on all your events by user to get both their first one, all the ones in the middle and the last.
47:36
And so this isn't actually that complicated to set up.
47:41
Most people just don't do the work early on.
47:41
And then when they want to go back later and have MTA results, they don't have the data to do it.
47:46
So, one of the things I tell people who are debating this is let's just get the infrastructure right from the beginning.
47:54
Let's set up so that you have users, you have user attributes, you're collecting first and last UTM on users.
48:00
You're firing events with all those.
48:00
There's some other more complex things you can do too.
48:05
You can set them in first party cookies and you can also set them in your third party cookies for your tooling vendors.
48:10
At the end of the day though, what matters is you just are collecting this information from the beginning.
48:14
That way when you actually want to progress your attribution model, you don't have to wait a really long time to start gathering that data. Amazing.
48:24
I love the details that you're sharing.
48:24
I don't know where else people can find this sort of advice.
48:29
It sounds like a core part of this is one, just having a data warehouse where you just throw all this data into, and two, having a taxonomy that you can rely on and do multiple things with down the road. Is that roughly right?
48:38
Yeah, I think that's right.
48:38
The taxonomy though, I think what's interesting is it's very much guided by your third party tools.
48:45
And again, that's the reason why I think companies often miss the mark here is because they're not thinking about what can my tool actually allow me to put into it in the first place.
48:58
Just to make sure I understand what you're saying there, you're saying generally maybe third party tools limit what you can do, which set you up for hardship later.
49:03
And maybe what you're saying is do that yourself, that tracking piece, is that roughly what you're saying?
49:13
Yeah, I think that's right.
49:13
The way to think of it is if you build your own data warehouse, your schema is unlimited.
49:21
You can do whatever you want.
49:21
You can design product schema, you can design user schema and event schema, but most third party B2C tools don't allow you to control the schema.
49:33
There's only one CDP I know that does that, that's Snowplow.
49:36
The rest are there's a user object and an event object.
49:36
So, you can either stuff data as a user property onto the user object or you can stuff data into the event and fire it off as an event, but that's what you're working with.
49:48
So, what I'm saying is most people just don't think about the object orientation of the third party tools they think about and they don't design their website traffic or their app traffic.
49:59
We didn't talk about app, which is a whole different slew because doing attribution with Iowas 14 is much more difficult.
50:04
But even in the website version of the world, people will often just collect UTMs and think that their job is done and it's like actually it's more complex.
50:13
You have to think about first and last, think about the steps in the middle, design it so that you're putting it on the user profile and in the event.
50:18
And so this goes back to the main thing that we were talking about earlier, whereas the job of a marketing technologist is to think often one to two years down the road about what we're going to need to solve for and design systems in an elegant way, not to break the bank, but to at least be the minimum viable product to actually get there.
50:37
And a lot of my job, and I think the job of marketing technologists is trying to preserve that future state in the most minimally invasive engineering and resource way possible.
50:48
You've talked a bit about thinking ahead and a bunch of tools and platforms, and I'm wondering are there any new and emerging tools, platforms or even growth channels that you're keeping an eye on or excited about or finding more and more useful?
51:02
I'd be remiss if we didn't talk about Threads, right?
51:02
Threads is super interesting.
51:02
The question will be how quickly can they stand up an API for advertising and what does that look like or do they just blind it in with the existing meta and Facebook architecture?
51:14
One of the caveats that I'm sure a lot of performance marketers out there will agree with is Facebook has a conflict of interest in reporting, right?
51:25
They want you to spend money, so obviously they want to report the best results.
51:29
And that's the reason why attribution parties like Branch and AppsFlyer exist is to somewhat curtail that conflict of interest.
51:35
And so I'll be really interested just to see how attribution works, especially when you're moving from Instagram to Threads, from Facebook to Threads.
51:40
Will it be the same architecture, will be the same advertising platform?
51:47
Will they try to do something new?
51:52
So, I'm keeping my eyes on that.
51:52
Reddit is also a very interesting place to convert now.
51:52
They're opening up their conversions API, and I'm seeing a lot more investment in Reddit just because you can have embedded ads now that almost look like they can be posts that you can comment on.
52:01
I think it just speaks to the maturity of the advertising business.
52:07
What's happening in the background of all this is ad attribution from apps has become a lot more difficult and mostly aggregate.
52:11
From 2010 to 2020, we had the golden years of deterministic matching where it was very easy to run an ad and understand with precision who installed the app.
52:25
Maybe you didn't know their name, but you actually would know their IDFA and you could tie that to their PI.
52:30
You can't do that anymore. It's very challenging.
52:33
Even when you can do it, the results that you would get are pretty low because nobody's going to be opting into giving you their IDFA.
52:38
So, what that means is these ad networks are becoming more complex, sophisticated, and interesting right at the same time that it's harder for marketers to really understand how they're spending money.
52:51
And so I'm paying a lot of attention to how marketers make decisions with probabilistic data because most of the work that I'm doing now is actually saying, well, given that we don't have determinist data about a certain audience or where somebody came from, how can I find other information that will create a model for 30% of the population and we can use that to extrapolate to a hundred.
53:14
So, probabilistic matching and probabilistic attribution I feel like is a skillset that more marketing technologists and marketers should just get familiar with the way that we make decisions today.
53:24
Wow, I hadn't heard of this concept before.
53:29
And that's how people are starting, or at least you're suggesting that's how people should start thinking about growth results and impact is less, here's how much this ad drove, the likelihood that this ad did this had this sort of impact.
53:41
And it's not the case with all channels, but it's specific for apps that have mobile apps, they're going to be impacted by it because they just aren't going to be able to discreetly identify one-to-one the person that came from a campaign.
53:57
They'll know that a group of people came from a campaign, but they won't be able to make measurement with those people alongside other attributes.
54:01
For website, it's not the same, but there are lots of things that are making it more challenging.
54:08
One is browsers now are stripping out those URLs we talked about.
54:13
So, you're just seeing a bigger and bigger percentage of people being counted as organic that actually came from a paid advertisement because when they got redirected to your website, the browser truncated all those URL parameters.
54:22
The second thing is cookie blockers.
54:28
We talk about all these third parties before.
54:28
The way that third parties often collect information is they drop a cookie in your browser that tracks you, if you've heard of Segment, which is one of the most well-known CDPs of the last few years, is they implant a little third party cookie on the site that contains an anonymous user ID and all of your attributes as you're navigating the site.
54:48
And then once you log in, they convert that to a known or non-anonymous user ID.
54:53
Usually that's tied to some type of entity ID or a user record.
54:57
And at that moment in time, if you come back and they see your cookie, they kind of know who you are.
55:01
Now, if you're blocking cookies, that means you're basically remaining anonymous throughout the entire user journey until you log in.
55:06
Not to mention a lot of people have lead funnels where you need that information to actually understand what the user is doing before they convert.
55:12
So, if you're blocking third party cookies before they even get a chance to convert, you have no information about where the person came from.
55:20
You just saw that they signed up, and so it might as well be organic.
55:24
So, you talked about how many people are trying to get used to this new world of ATT and much harder to measure attribution and all that.
55:28
Is there anything you've learned that has worked well to help you recover from that a little bit in terms of measuring what's happening?
55:38
Is there any tips you can share or anything you've seen work?
55:43
Yeah, I mean I think a lot of people are just gravitating towards MMM now without really understanding when MMM is useful or not.
55:48
I don't know if there's a company called Recast.
55:48
I think you're an investor. Am I crazy? I am.
55:56
And that's who wrote that article that you mentioned actually. That's right.
55:59
It's Michael and it was somebody else, I can't remember the name besides- It was another Mike Taylor.
56:06
I'm not an expert on MMM, so I'm not going to be able to comment to quite the degree that they have.
56:10
But when I spoke with Michael, and when I think about MMM, a lot of my conversation is this actually really realistic for our business right now?
56:17
Do we have the data to run an MMM model and how is it going to change or chart the course of our performance ad marketing business in light of having this information?
56:29
And when I think about it through those lens, most of the time businesses are not ready for MMM.
56:34
They actually just be an MTA and they need better probabilistic modeling.
56:39
And I know that's not a super spicy take, but I'd just say at least at Ramp and what I'm seeing at other businesses right now that are operating, it's much more of like we're going back to the days where we understand in broad strokes how much each of our campaigns is driving in advertising revenue.
57:00
We're not able to tie that discreetly with the user journey.
57:00
And we know that some percentage of this user base might have been lost or organic.
57:04
So, in light of those, how do we make spend?
57:04
And then also you can be pretty smart.
57:09
You can do, for example, geo-based testing on billboards.
57:14
Try to isolate that as a factor if you withhold all other confounding factors so you can be smart.
57:20
Coordinating these types of campaigns though is really challenging, especially if you're a really big business, let's say runs online advertising throughout the US and you're trying to do targeted billboard tests in an isolated number of cities across the states coordinating to turn off demographics, make sure there's not isolating factors.
57:35
It can be really challenging.
57:35
So, there's not a silver bullet right now I don't think. Awesome.
57:40
Just a few more questions and then a broader question I want to ask.
57:46
So, say you want to start hiring the next Austin, first of all, what do you look for in the person?
57:52
What are signs that they're probably going to be worth chatting with?
57:55
And then what are some interview questions you'd like to ask to get a sense of how strong they are?
57:59
So, the first thing that I always gravitate towards is just intellectual curiosity.
58:04
And I know that's very, maybe a little bit overrated, but I think you can tell pretty quickly if somebody's just interested in the world and learning things.
58:14
And the thing about third party tools is you are constantly learning.
58:19
You'll never be an expert in everything because there's way too many tools to be an expert on, I forgot what publication, I think it's MarTech editor in chief or something.
58:24
There's a publication that I subscribed to and everything is classified as MarTech and the diagram is huge, like cover a wall.
58:35
Now I don't believe everything like that is MarTech, but even if a fraction is, there are way too many tools in technology to ever be an expert.
58:39
So, you have to be both very interested in learning and very willing to quickly learn if you want to be in the space.
58:50
And so I generally look for intellectual curiosity as the first sign.
58:50
The second thing that I think helps people a lot who have intellectual curiosity is they're scrappy in engineering.
58:56
They might not be the best engineer possible, but they know how to get around.
59:01
They know JavaScript, they know Python, they can read API documentation and make an API request.
59:06
They have enough base knowledge to basically understand how to solve a problem that an engineer might do even if they themselves are not an engineer.
59:18
Now obviously you can get lucky sometimes and you'll find the engineer who never wants to be an engineer again and decides to move into something less technical.
59:30
And in those cases they're super powerful, but I haven't met a lot of those people in my life.
59:34
And also there's just some business dynamics to it.
59:34
You could probably make more as a backend engineer than as a MarTech guy.
59:39
So, you probably just pursue the pathway that makes more money.
59:44
It's like a little bit of a utility function.
59:44
So, I look for intellectual curiosity.
59:50
I look for basic engineering scrappiness.
59:50
And as a side note, I would say lots of people out there, the advice that I give them is you don't have to go get a software engineering degree.
1:00:01
You can teach yourself, I am self-taught.
1:00:01
You can take a coding academy online.
1:00:01
I think you get enough knowledge through being able to do web programming or some type of backend programming.
1:00:13
So, I would say it's not more than a six-month investment for anybody to really get the skillset that's needed.
1:00:17
Obviously once you get the skillset, you can build upon it with years of experience afterwards.
1:00:21
But if you're new to the space and you're in marketing ops and you want to get more technical, or if you're a user acquisition manager who did paid performance, but you're like, I really want do things end to end.
1:00:30
You can just go pick up some software skills and you probably are going to be pretty dangerous from that.
1:00:35
And so those are the two things I gravitate towards.
1:00:39
There's obviously many more, but those are the first two.
1:00:39
The questions that I like to ask is what does I like to ask people how they prepared for the interview.
1:00:49
This is not, I can't take credit for this.
1:00:49
My wife told me about, gave me this idea and I loved it.
1:00:55
I think it was a16z partner.
1:00:55
But I love the question because when you ask, hey, how did you prepare?
1:00:59
You're really asking how does the person think? How did they plan?
1:01:04
How did they take things seriously or not? What did they read? What did they do?
1:01:09
And if you have to prompt them to tell you all the things they did, then they're just not a systems thinker.
1:01:14
But if they're like, hey, actually I read these things, I did this.
1:01:19
I woke up, I went for a run.
1:01:19
The more interesting complex the answer, the more interesting complex the candidate.
1:01:26
And so I love the question because it just gives you a really good understanding of the person on a whole, like right out the gate.
1:01:31
And then the other question I like to ask is I like to ask, so you're coming in tomorrow to our marketing tech system and by Friday you have to write up a report on all the things we should change. What do you do?
1:01:47
And I like to ask that question because it pretty much signals out people who are biased versus not.
1:01:51
People who have tooling biases will immediately just like, we should implement this tool because I used it before and I really like to hire people who are not tool specific, who are more tool agnostic and they think about tools as being things to solve problems as opposed to tools being things that you just solve because you've already solved it one way.
1:02:12
This isn't a gripe and it's certainly not intended to slice at PMs, but one of my observations of a lot of PMs is they just pick the tools they've already used before because it's easy and it's a shortcut for them, which I understand, but problems are not always the same.
1:02:27
So, tools shouldn't always be the same.
1:02:27
So, I like to pick people who think about the problem set and the solution space more and they ask questions about what problems you're trying to solve, which I think is much more of an actual PM mindset of trying to work backward from the problem as opposed to just taking the problem and regurgitating stuff that you already know.
1:02:45
Are there any flags you look for that tell you maybe this person isn't someone you want to be working with?
1:02:49
I answer that question on two spectrums.
1:02:52
One is if I'm hiring as somebody who's hiring an IC versus I'm getting hired.
1:02:59
So, one of the red flags whenever I'm approaching a company to work for them is I'll ask for their company financials and a company that's not willing to divulge their financials to a director level or above person, I don't want to do business with because that means they're hiding something.
1:03:14
Or they have a culture where they don't trust the most senior leaders of the organization.
1:03:19
Either is a bad choice in my perspective.
1:03:19
So, that's one of the questions I always ask when I'm going up for a job, when I'm hiring somebody.
1:03:24
Red flags, I feel like one of the false flags not a red flag is more like when there's a gap in somebody's job resume, everybody gravitates towards that and it's often really explainable.
1:03:36
A good example is I was hiring somebody once who had a two-year gap in their resume.
1:03:41
We didn't end up hiring the person, but they went through all the stages and we didn't hire them, not because of them, but because the job got removed and this person took two years to get a philosophy degree or maybe it was a poetry degree and then also taught himself to program.
1:03:58
So, it was a really enriching two years and there were lots of ways that I could see them bringing their past experience and the way that they took time off together to be a really well-rounded candidate.
1:04:09
So, I would say I look less for red flags and more for false identifiers on the resume application that may shortcut me towards the decision.
1:04:16
Another one is just school people just look at your school where you went to undergrad or grad and they kind of make a decision one way or the other.
1:04:29
And I feel like that's also can be a really bad shortcut because there's some amazing founders, for example, who went to school as you maybe had never heard of.
1:04:37
Yeah, I know that's not a good way to answer the question, but I don't have a good way of looking for red flags, but I do tend to spend a lot of time on netting out of false flags.
1:04:48
That was a great way to answer the question.
1:04:48
I want to move on to something totally different, and this isn't something I've been asking people, but I'm curious if there's something here then maybe if there is, I'll start asking this more regularly.
1:04:55
I'm curious if there's just any frameworks that you've found especially useful in your work or even life.
1:05:04
Does anything come to mind?
1:05:04
One thing that I want to build, so if I ever build this, maybe it'll be a newsletter for you, is just a one-page doc of the most useful life frameworks and they're just the words, and so you obviously have to know them, but I feel like I come across really good frameworks all the time and then I forget them.
1:05:20
So, I just want a one pager of Lenny's Life frameworks.
1:05:24
Okay, we're starting this right now. Okay, great. We'll have number one. All right. I like this.
1:05:34
Okay, so I've already said this and you've promised to put this at the top of the list, so I'm really excited.
1:05:38
It's just tools are meant to solve problems and I tell that to every person I hire.
1:05:43
I repeat it consistently at Ramp and all of my consulting gigs.
1:05:43
And it's not just the words, it's the spirit of it.
1:05:48
Tools are really just meant to solve problems.
1:05:48
You don't have to buy a tool to solve the problem.
1:05:54
You also don't have to buy a specific tool to solve the problem.
1:05:57
And I think it embodies so much of what marketing technology is trying to do.
1:05:57
It's trying to help people understand their problems and then actually take action on them using tools and technology that are most first party and third party.
1:06:09
And most people just focus on the tool part and focus on the buying and integration part.
1:06:14
And so I think if you consistently remind yourself that tools are just meant to solve problems, then you really get into a space where you as a systems' person can be an advocate for your marketer or your product people.
1:06:24
I think sometimes there is a little bit of a tendency for people to think that people who manage and set up tools are just interested in managing and setting up tools, but really at the end of the day, we're trying to help people actually do stuff.
1:06:37
Then there's this PPS framework that I talk about a lot, which is problem, people and system.
1:06:41
So, whenever there's a challenge that comes up like at Ramp or in a consulting gig, I like to first say what's the problem?
1:06:47
Who are the people involved and what system does it impact?
1:06:53
Usually because people just jump straight to the system.
1:06:53
They're like, hey, there's this problem, I just need to solve it with the tool.
1:06:58
Hey, I'm trying to do X, Y, and Z.
1:07:02
Can you just give me admin permission straight to the system?
1:07:06
So, if you back up though first you understand the problem like, hey, what is this person trying to solve?
1:07:10
What is their discreet issue?
1:07:10
A great example is I'm a sales manager and I want to make it so that every time I hire somebody, I don't have to go through this really tough process of onboarding my staff.
1:07:20
All right, so that's the problem.
1:07:20
Who are the people that involves, does the sales manager need permission from the CRO?
1:07:25
Do the sellers need to be trained?
1:07:25
Is there some other confounding factor that we're not aware of why we don't want to just automate this thing?
1:07:35
Once you have an understanding of the people and the problems that you're trying to solve, then it's really, really easy to design the system to solve that.
1:07:39
And so that's my number one framework for technologists in particular is like don't just jump to the system, think backwards, start with the people and the problem and then move to the system solution.
1:07:52
And then another one that I've already mentioned too is it's B and B as opposed to BVB.
1:07:52
So, build and buy as opposed to build versus buy.
1:07:59
People all the time just think the second that you're talking about implementing a tool or procuring a solution, it's, Hey, I want to build this thing or I want to buy this really expensive thing.
1:08:13
Build versus buy is a very narrowly constricting decision tree.
1:08:21
If it's only build versus buy, then you've already made the decision that you can only do one or the other, which means you're already fighting somebody at your organization.
1:08:30
Build and buy means that both of you can win and you can actually create a solution that is not only unique but saves the company time and resources and makes everybody happy.
1:08:36
It's more of a consensus driven approach.
1:08:41
Whenever I hear in a meeting or a call or some discussion about how we have a tool and it's really expensive and we want to build in herself, I try to just use the build and buy framework to tee people up and say, what about the problem? Can we buy?
1:08:59
What about the problem can we build?
1:08:59
And where does it make sense to invest our resources and our people accordingly to get the optimal outcome?
1:09:04
A great example is a company that I was consulting for was thinking about building their own AB testing tool.
1:09:10
And actually we had the same problem at Ramp recently, and they're like, well, we just think we should build ourself.
1:09:19
This is core to our technology.
1:09:19
We have the engineering resources to do it.
1:09:19
And they were evaluating it to build the entire system themselves or buy a third party, I think it was split.
1:09:29
io or something like that.
1:09:29
And the entire engagement was basically designing a financial model to show them that they could make a lot more money, save money, move faster if they just bought the third party tool at the lowest possible cost and spent all of their resources that they were going to spend building it, building around it and making it their own.
1:09:45
And there's lots of, I hate the word synergy, it's just so yucky. I don't mind it.
1:09:50
I think it communicates what you want to communicate, and I feel like people don't say it as often anymore, so maybe it's okay.
1:10:00
Yeah, because they're afraid.
1:10:00
There are mutual benefits is a better way of saying it, if you build a tool custom to yourself when you've bought a tool because the vendor at that point is committed to you and they want you to be successful, so you often can get accelerated outcomes if you build on top of a third party than if you just build it yourself.
1:10:19
A great example is say you buy one of these AB testing tools and you build around it and you're a large customer of them, but you've invested a lot of your own engineering resources to make this solution your own.
1:10:29
If they know that and they care about you, they're going to be willing to actually make you happy in the moments where you need a change from them, say some SDK change or a new feature or something like that.
1:10:44
A framework that I talk a lot about in my Reforge course is about building a stack.
1:10:44
Everybody asks, hey, how do I build my stack? What should I do? What tool should I use?
1:10:49
Even you earlier were like, what's the golden stack?
1:10:52
Tell me what five tools should I get? Just tell me. Yeah, yeah.
1:10:57
I'll tell you at the very end, I got to hold out.
1:11:00
Otherwise, you'll ditch this podcast. That's right. Yeah. Wait until the end.
1:11:04
Is there anything else you want to share before we get to our very exciting lightning round?
1:11:11
The only thing I had wanted to fit in, which I feel like is maybe a framework or maybe just a really good decision-making philosophy is this concept of thinking gray.
1:11:20
Have you heard of it before? No. Go on. Okay. So, Steven B.
1:11:25
Sample is a professor at USC.
1:11:31
He wrote a book called The Contrarian's Guide to Leadership, really great book.
1:11:37
It's one of his principles.
1:11:37
There's actually a lot of great principles in the book, but this is the one that I think has stuck with me the most in my career.
1:11:41
And the concept of thinking gray is so often in life and in our jobs, we are forced to make decisions very quickly.
1:11:47
We have to think black or white about a problem set or a solution, and then decide.
1:11:53
One of his tactics is this concept of thinking gray, which is actually to not decide for as long as you possibly can before you have to decide.
1:12:05
It's really challenging because it involves this little thing called patience, which I do not have a lot of, most of the time, and I know most people don't as well.
1:12:10
But it's particularly really relevant in systems thinking and product because so often we believe that we have to make a decision because our boss is telling us because there's an OPR, because we feel the pain because somebody's complaining to me.
1:12:27
But actually in reality, you don't have to make a decision at all.
1:12:32
You can just let it sit for a while.
1:12:32
And this also applies to, I think, how you move through the world and view people.
1:12:37
A lot of times we will meet somebody in a company setting or in a business setting, and we are quick to make decisions about them.
1:12:44
We even asked me questions about how I hire people very quickly.
1:12:49
We're looking for shortcuts to make decisions about evaluating people.
1:12:53
One of the best pieces of advice though about thinking gray is it gives you the grace to not decide about people until you have to decide.
1:12:58
So, obviously for an interview decision, you have to decide.
1:13:03
You have to decide yes or no.
1:13:03
But so often you'll come across people and you'll meet them once or twice.
1:13:07
And I feel like there's this tendency in the back of everybody's brain to be like, do I like this person?
1:13:13
Do I want to work with them?
1:13:17
And the question often is not that it's do you have to even make a decision right now?
1:13:17
And by leaving yourself to have space to decide, you actually open up the possibility that in the future you'll make a better decision.
1:13:29
So, I think that's a really good lesson for systems, and it's obviously a lesson that you can apply to the rest of your life too.
1:13:37
Austin, that was awesome.
1:13:37
And with that, we've reached our very exciting lightning round.
1:13:37
I've got six questions for you. Are you ready? I'm so ready, Lenny.
1:13:45
What are two or three books that you've recommended most to other people?
1:13:49
So, first book I already mentioned, but say it again.
1:13:49
It's The Contrarians Guide to Leadership awesome book.
1:13:54
Second book that would be really good is the Art and Adventure of Leadership by a guy name Warren Bennis.
1:13:59
And these are more philosophically leadership books.
1:13:59
They're less about technical specs on how to run a business.
1:14:05
So, you have to be into that.
1:14:10
Favorite recent movie or TV show.
1:14:10
I'm currently watching for the first time Suits, which I'd never seen before, which I think is pretty good because the story arc of every Suits episode is that there's a problem.
1:14:20
Then they solve the problem and then the problem is solved at the end.
1:14:24
So, it's very gratifying for anybody out there who's a high anxiety person who just wants to have this story arc resolved at the end of the episode.
1:14:28
But if that's not your jam and you like excitement, also watching Silo, Witcher.
1:14:34
For comic relief, there's Our Flag Means Death, which is hilarious. Have you seen that show?
1:14:41
No, not Our Flag Means Death.
1:14:45
Now you need to go watch it.
1:14:45
It's about black beard and gay pirate captain.
1:14:45
So, strongly recommend that.
1:14:52
And then for just really dumb comedy, What We Do in the Shadows is hilarious. What was that?
1:14:59
What We Do in the Shadows? Yeah. Okay.
1:15:01
Wow, a lot of recommendations. Thank you for that.
1:15:01
What is a favorite interview question you like to ask candidates?
1:15:07
So, I talked about what you did to prepare, but the other one that I think is really good because it forces people to get vulnerable is tell me about the most difficult or challenging thing you've overcome in the last year in your life.
1:15:21
It doesn't have to be work related, it could be personal.
1:15:21
And I think it's a great way to just reset the atmosphere, make people dig a little bit deeper into who they are and be more vulnerable.
1:15:32
And I find usually it also helps calm them down because if they shared them one of the most challenging, difficult, and hard parts of their life, then all the other questions just are pretty easy.
1:15:41
So, that's one of my favorites.
1:15:41
What is a favorite product that you've recently discovered that you really like? This sounds super dumb. It's called cal.
1:15:45
com, and I'll tell you the story first.
1:15:49
I've been a big Calendly user for a long time, but Calendly is pretty expensive.
1:15:53
If Calendly is listening, you want to give me promo, cool. But it's very expensive.
1:15:58
And then I also just found that it is not always graceful at syncing multiple calendars from both businesses and consulting gigs and personal, and I had trouble remembering my Calendly link. I don't know.
1:16:09
The interface is like circa 2016.
1:16:09
So, really looking for something a little bit more notion like with a Command K interface and just integrations that work. And cal. com has not failed me. It has been awesome.
1:16:21
So, if people are looking for new Calendly tools, strongly recommend.
1:16:27
Wow, I never heard of this.
1:16:27
What a great domain, cal. com. I know, right. Killer.
1:16:34
What is a favorite life motto that you often repeat to yourself or share with other people, either in work or in life?